betaFeedback

WorksComputer Science

DOCK: Dual-Stream-Inspired Self-Supervised Object-Centric Representations Grounded in Robot Kinematics

Abstract

Latent-space planners select actions by comparing the embedding of a predicted future image with that of a goal image. This assumes that distance in the embedding reflects how far the scene is from the goal. We profile this assumption with simulator-executed futures, which removes prediction error. For six frozen DINO and V-JEPA encoders, feature distance barely tracks the object's distance to its goal: the rank correlation is at most 0.11, against 0.87 for object-mask centroids. It is dominated by background and robot patches, and frozen-feature readouts capture the arm far better than the object. Inspired by the ventral (what) and dorsal (where/how) visual streams, we propose DOCK, which learns an object representation whose changes agree with measured physical motion. A grasped object moves with the gripper, so forward kinematics on the joint angles gives a proxy for the object's displacement without annotation. From 9,951 held-object transport trajectories in 21 simulated kitchen scenes, we mine 404K displacement pairs and train three components: an object token pooled from patch features (what), a spatial ruler that makes differences of readout coordinates equal to measured displacements (where), and a dynamics predictor over object tokens (how). Used as a planning cost, the learned representation places the object in 15 of 21 tasks, as many as an oracle that reads the true object position, while mask geometry reaches 10 and frozen feature distance 9. It still solves the most tasks when the unseen robot arm is visible, and its ranking transfers to held-out layouts with unseen object instances and categories without any adaptation.

Introduction

A world model lets an agent evaluate an action before taking it: the agent predicts the outcome and compares it with the outcome it wants [14, 20]. Self-supervised encoders such as DINO [6, 31, 37] and V-JEPA [4, 5] have made this practical without labels, and recent planners act directly in their

feature space. They encode a goal image, predict the features of candidate futures, and execute the action sequence whose outcome lies closest to the goal [4, 39, 43].

This pipeline assumes that the distance between two image embeddings measures progress toward the goal. For manipulation, current world models have two drawbacks that put this assumption in doubt. (Drawback 1) Task-agnostic relevance: JEPA-style methods [3], such as V-JEPA [4, 5], learn by predicting latent visual representations, aiming to retain meaningful information while suppressing irrelevant detail. However, what is relevant depends on the downstream task: object placement requires precise spatial localization of the object, yet the object covers only a few image patches and is easily ignored next to the much larger background and robot arm. (Drawback 2) Action-free pretraining: Encoder pretraining is action-free and does not explicitly model how actions affect future states, which leaves open whether the learned content and appearance features provide sufficient spatial precision. Exposing and removing these drawbacks raises two challenges. **(Challenge 1) Diagnosing the representation in isolation: A planner can fail because its predictor is wrong or because its cost does not measure the task, and closed-loop success rates do not tell the two apart; the arm also moves with the object it holds, so a cost may follow the arm rather than the object. (Challenge 2) Metric supervision without annotation: Placement needs a representation that singles out one small object and measures how far and in which direction an action moves it, but current representations largely ignore the object itself, and object-pose or depth labels are costly to collect.

To address these challenges, we present a series of profiling experiments that evaluate the representation alone, and a self-supervised training framework that teaches the representation where an object is and how actions move it, using only the robot’s own measured motion. Our contributions are twofold, as summarized below.

  • A profiling evaluation of current representations (Sec. 3). In a placement benchmark with exactly simulated futures, the goal-image feature distance of six frozen DINO and V-JEPA encoders is almost blind to where the manipulated object is (Figure 1): its rank correlation with the true object-to-goal distance is at most 0.11, against 0.87 for mask centroids. It is dominated by background and robot patches, and frozen-feature readouts capture the arm far better than the object.

Figure 1. Feature distance is not a ruler for object location. (a) Planners with pretrained encoders compare all patch features of the two frames, a distance dominated by background and robot. (b) We read the object’s coordinates from an object token and compare them in meters as the planning cost.

  • DOCK, a self-supervised dataset and training framework (Sec. 4). The framework is inspired by the dual-stream of the primate visual system [11, 27]: the ventral pathway for content (what) and the dorsal pathway for location and action (where/how). An object-centric representation of pretrained features supplies the what; a spatial ruler that reads out the object’s metric location adds the where, and dynamics prediction of the object’s motion under the robot’s actions adds the how (Figure 3). The ruler and the predictor are trained without annotation on 9,951 held-object transport trajectories, using only the gripper motion that forward kinematics computes from joint angles. As a planning cost, the learned representation places the object in 15 of 21 tasks, as many as an oracle that reads the true object position, against at most 10 for mask geometry and frozen feature distances.

Figure 3. Object-centric representation inspired by dual-stream. (a) Data collection. Held-object transport supplies displacement labels from joint angles and forward kinematics. (b) Joint training. Mask-pooled object tokens (what) support a displacement-supervised spatial ruler (where) and an action-conditioned dynamics predictor (how), with EMA-teacher targets from eight-frame windows. (c) Planning. With the encoder and readout fixed, ruler distance scores simulator rollouts against the goal.

Related Work

Visual representations for planning. Latent planners predict future features of self-supervised encoders such as DINO [6, 31, 37] and V-JEPA [4, 5, 28] and minimize their distance to a goal image [39, 43]; others learn visual rewards from value pretraining, temporal distance or demonstrations [15, 23, 24, 38]. We test whether feature distance tracks physical progress and learn a metric cost instead. Spatial and object-centric representations. Probes and 3D-aware fine-tuning study the geometry in pretrained features [7, 25, 42], and object-centric methods rely on slots, object dynamics, descriptors, keypoints or masks [8, 9, 19, 21, 26, 35, 36, 41]. We train a mask-pooled object token for metric location readout and action-conditioned prediction. Self-supervision from motion and interaction. Ego-motion [1, 18] and robot interaction [2, 17, 32] supervise visual learning, and measured camera velocity [13] resolves the scale ambiguity of self-supervised depth [10, 44]. We use gripper displacement from forward kinematics as a metric proxy for a held object’s displacement.

Profiling Current Representations

We test whether feature distance ranks physically better actions across six manipulation task families. For each of 360 states, we execute 300 candidate action sequences from the same saved simulator state and render their outcomes, removing learned-predictor rollout error from scoring. The full-grid cost is

cfeat(Ic,Ig)=1ND∑u=1N∥E(Ic)u−E(Ig)u∥22(1)c_{\mathrm{feat}}(I_c,I_g)=\frac{1}{ND}\sum_{u=1}^{N}\left\lVert E(I_c)_u-E(I_g)_u\right\rVert_2^2 \tag*{(1)}

where NN and DD are the patch count and feature dimension. We report mean within-state Spearman correlation ρ\rho: 1 indicates identical rankings and 0 indicates no monotonic agreement. Protocols appear in Appendix A.

Replacing the predictor with the simulator

Experiment. We compare feature costs on the world model’s predicted features (JwmJ_{\mathrm{wm}}) and on images rendered after executing the same actions in the simulator (JsimJ_{\mathrm{sim}}). As a physical reference, the oracle cost JoracleJ_{\mathrm{oracle}} sums task-specific normalized errors from true simulator states: hand–object distance for reach, object–goal distance for place, and hand–object distance plus the additional height needed to meet the 5 cm lift target for pick. Composite tasks sum the applicable terms (Appendix A.1). The two feature-cost rankings correlate at ρ(Jwm,Jsim)=0.711\rho(J_{\mathrm{wm}},J_{\mathrm{sim}})=0.711, but their correlations with JoracleJ_{\mathrm{oracle}} are only 0.006 and 0.026, respectively (Figure 2a). Thus, agreement with the predicted feature cost does not imply agreement with physical task progress. Binary-success AUC measures how often a successful candidate receives a lower cost than an unsuccessful one: 0.5 is chance and 1 is perfect discrimination. For JsimJ_{\mathrm{sim}}, the mean AUC is 0.514 over the 87 states containing both outcomes (95% interval: [0.483, 0.545]), indicating near-chance discrimination of successful candidates.

Figure 2. Profiling frozen representations on 360 states with 300 simulator-executed candidates each; (a–c) use DINOv3-L. (a) Feature-cost and oracle rank correlations. (b) Regional shares of pixel area, feature cost, and score variation. (c) Readout correlations and candidate-selection success; place compares separately decoded candidate/goal object positions. (d) Feature–oracle correlations across encoders. Whiskers: 95% bootstrap intervals.

Finding 1. Even with true simulator rollouts, goal-image feature distance does not reliably identify better actions.

Design implication. We evaluate the learned spatial cost on simulator-executed candidate rollouts to isolate scoring quality from predictor error (§ 5.1).

Spatial attribution of the cost

Experiment. We split each DINOv3 patch’s squared feature difference among background, robot, and object according to the fraction of its pixels in each region, measured by simulator segmentation masks. Pixel area is the fraction of image pixels occupied by a region; feature cost is its fraction of the summed patch cost. Score variation measures how the region’s cost changes with the total cost across candidates, using the signed covariance share defined in Appendix A.§ 6. Background and robot positions account for 64.12% and 33.09% of cost magnitude, leaving 2.80% at object positions. The object’s covariance share is only 0.31%, so the object hardly changes the score (Figure 2b).

Finding 2. Most of the full-image cost accumulates at background and robot token positions, motivating an object-focused representation.

Design implication. We pool features over the manipulated object’s mask to form the token used by both the spatial readout and dynamics predictor (§ 4.2).

Arm and object readouts

Experiment. We freeze DINOv3 and fit ridge and MLP readouts on whole-image mean-pooled features, and an attentive readout on the full token grid. Reach and pick probes predict hand–object distance and object lift from candidate images. For placement, we apply the same position readout separately to the candidate and goal images, then compute the Euclidean distance between predicted 3D object positions. The MLP attains within-state correlations of 0.940, 0.218, and 0.010 on pure reach, pick, and place; ridge and attention yield 0.006 and 0.013 on place (Figure 2c; protocols in Appendix A.§ 3). The large reach–place gap suggests an arm-based proxy: the readout may estimate approach from arm configuration without accurately localizing the object that has to be placed.

Finding 3. Strong reach decoding but weak placement decoding suggests that the readout captures arm configuration more readily than object position.

Design implication. We train the visual representation on arm-free images while deriving displacement supervision from robot kinematics (§ 4.1).

Candidate correlation across encoders

Experiment. On the same candidate banks, we compare DINOv2-S/B, DINOv3-L, V-JEPA 2-L, and R3M [29] by the rank correlation between feature distance and the task-specific oracle cost, using grid, pooled, and available CLS features. No tested variant exceeds ρ=0.144\rho= 0.144; Figure 2d shows the L2 comparisons. We also evaluate six frozen encoders on the separate 21-task placement benchmark used in § 5, with 300 simulator-generated candidates per task. In arm-free images, full-image and object-pooled costs reach at most 0.111 and 0.346, versus 0.868 for mask-centroid distance. On matched visible candidates, DINOv3 object pooling remains far below the mask-centroid distance (0.420 vs. 0.870 in rank correlation).

Finding 4. Across the tested encoders, appearance distance provides weak candidate ordering, motivating a displacement-grounded spatial cost.

Design implication. We train the encoder and readout with metric displacement supervision, then use readout distance as the planning cost (§ 4.2).

DOCK

Inspired by shared early processing and the specialized ventral and dorsal pathways in human vision [11, 27], we learn an object representation that plays the role of the ventral pathway (what), with a spatial ruler (where) and an action-conditioned dynamics predictor (how) that play the role of the dorsal pathway. Held-object transport supplies the motion supervision, and the learned ruler distance provides the metric planning cost at test time (Figure 3).

Dataset Collection

Kinematic supervision. A person carrying an object can feel how far the hand has moved. A robot has the same signal: while it holds an object, its joint angles determine the gripper’s motion, which gives us displacement supervision without annotation (Figure 3a). For arm joint angles qt∈R7\mathbf{q}_t \in\mathbb{R}^7, forward kinematics gives the gripper position gt=FK⁡(qt)∈R3\mathbf{g}_t = \operatorname{FK}(\mathbf{q}_t) \in\mathbb{R}^3 in the robot-base frame. The displacement label between frames ii and jj is

dij=gj−gi.(2)\mathbf{d}_{ij} = \mathbf{g}_j - \mathbf{g}_i. \tag*{(2)}

A grasped object moves with the gripper, so under a stable grasp with little gripper rotation, dij\mathbf{d}_{ij} serves as a proxy for the object’s displacement; metric scale comes from the robot’s link geometry.

Held-object transport. We collect 9,951 held-object transport trajectories in the 21 RoboCasa [30] scenes of the benchmark, yielding 3.09 million valid frames (Figure 3a). Each trajectory starts from a saved post-pickup state and moves the arm while maintaining the grasp, with the robot base and camera fixed. Trajectories include random walks, persistent and directed transport, reverse paths, expert paths, rotations and surface sweeps, with additional runs expanding spatial and motion coverage (Appendix B). Each step records a 256×256256 \times256 image, an object mask, joint angles and contact information. The same state is rendered with and without the robot; training uses the arm-free view to discourage decoding object position through the arm. From stable-grasp segments without object–environment contact, we construct two training sets. Displacement pairs,

labeled by Eq. (2), require a visible object in both frames, at least 5 mm of gripper translation, and at most 2∘2^\circ of rotation. Sampling across eight length bands yields 404,397 training pairs spanning 0.5 cm to beyond 90 cm. Temporal windows contain eight eligible frames at 0.25 s intervals, yielding 82,405 training windows; they permit gripper rotation within a stable grasp.

Whole trajectories are split 80/10/5/5 into training, validation, readout-fitting and test partitions. Transfer to unseen layouts, object instances and categories is evaluated separately (§ 5.4). Table 1 summarizes pair counts by displacement length and partition. Appendix B details the collection protocol, pair and window filtering, partitions and coverage statistics of the collected data.

Length (cm)TrainVal.FitFit-val.Test
[0.5, 2)63,000633188,68124,584197,537
[2, 5)63,000633149,36519,966160,875
[5, 10)63,000633107,30714,839114,698
[10, 20)63,00062873,20810,24477,478
[20, 35)63,00062737,5625,55639,891
[35, 60)56,45058816,7033,40320,055
[60, 90)31,2933542,9174745,556
≥ 901,65403000594
Total404,3974,096576,04379,066616,684

Table 1. Displacement pairs by length. All partitions pass the same pair filter. Training is capped per band and scene; the other partitions keep every eligible pair. Fit and Fit-val. are the two parts of the readout-fitting partition.

Training Framework

Object-centric representation (What). A pretrained ViT-L/16 encoder EθE_\theta, either DINOv3 [37] or V-JEPA 2 [4], maps the full image ItI_t to a 16×1616 \times16 grid of patch features. For V-JEPA 2, the image is repeated twice to form one tubelet. With wt,u∈[0,1]w_{t,u} \in[0, 1] denoting the fraction of patch uu covered by the simulator object mask MtM_t, the object token is

zt=∑uwt,uEθ(It)u∑uwt,u∈R1024.(3)\mathbf{z}_t = \frac{\sum_u w_{t,u} E_\theta(I_t)_u}{\sum_u w_{t,u}} \in\mathbb{R}^{1024}. \tag*{(3)}

The mask acts only at pooling, and no geometric augmentation changes the object’s image position. The same object token provides a compact visual input to both the location readout and dynamics predictor. This shared representation is analogous to early visual processing that supports specialized downstream pathways.

Spatial ruler based on object (Where). The spatial ruler learns from pairs of frames (Ii,Ij)(I_i, I_j) separated by a measured displacement dij\mathbf{d}_{ij}. A three-layer perceptron fϕ:R1024→R3f_\phi: \mathbb{R}^{1024} \to\mathbb{R}^3 maps their object tokens to coordinates whose difference should match this displacement:

Lruler=E(i,j)[hδ(∥fϕ(zj)−fϕ(zi)−dijℓ∥2)],(4)\mathcal{L}_{\mathrm{ruler}} = \mathbb{E}_{(i,j)}\left[h_{\delta}\left(\left\|f_{\phi}(z_j)-f_{\phi}(z_i)-\frac{d_{ij}}{\ell}\right\|_2\right)\right], \tag*{(4)}

Here ℓ=0.10 m\ell= 0.10\,\mathrm{m} sets the coordinate scale, and hδh_{\delta} is the Huber function [16] with δ=0.05\delta= 0.05, corresponding to 5 mm. The loss penalizes displacement error equally in all directions and updates both the readout and encoder. Only coordinate differences are supervised, so the readout learns a metric frame with a free origin and never needs an absolute object position as a label.

Dynamics prediction for object (How). A causal transformer predicts the next object token from visual history, proprioception and achieved actions:

z^t+1=Pψ(z≤t,s≤t,a≤t).(5)\hat{z}_{t+1} = P_{\psi}\left(z_{\leq t}, s_{\leq t}, a_{\leq t}\right). \tag*{(5)}

The seven arm joint angles qt\mathbf{q}_t shown in Figure 3a determine the gripper pose through forward kinematics. We concatenate its position (3 values) and orientation quaternion (4) with the two finger-joint positions to form the proprioceptive state st∈R9s_t \in\mathbb{R}^{9}. The achieved action at∈R7a_t \in\mathbb{R}^{7} contains the translation, rotation vector and change in the sum of finger-joint positions between frames. Targets zˉt+1\bar{z}_{t+1} come from a stop-gradient exponential-moving-average (EMA) teacher [3, 12]. For each eight-frame window, teacher forcing (TF) predicts the next token from object tokens encoded from the observed frames up to the current step. Autoregressive rollout (AR) starts from the first observed token and feeds each prediction back as the next visual input. Both modes use the same proprioceptive states and achieved actions and are supervised at all seven transitions:

Lm=17BK∑b=1B∑t=06∑k=1K(z^b,t+1,km−zˉb,t+1,kσk)2,(6)\mathcal{L}_{m} = \frac{1}{7BK}\sum_{b=1}^{B}\sum_{t=0}^{6}\sum_{k=1}^{K}\left(\frac{\hat{z}^{m}_{b,t+1,k}-\bar{z}_{b,t+1,k}}{\sigma_k}\right)^2, \tag*{(6)}
Ldyn=12(LTF+LAR),\mathcal{L}_{\mathrm{dyn}} = \frac{1}{2}\left(\mathcal{L}_{\mathrm{TF}}+\mathcal{L}_{\mathrm{AR}}\right),

Here m∈{TF,AR}m \in\{\mathrm{TF}, \mathrm{AR}\}, BB is the number of windows, K=1024K = 1024 is the token dimension, and σk\sigma_k is the batch standard deviation of teacher-token dimension kk.

Joint training. We jointly train the readout, predictor, and last eight encoder blocks with the final normalization layer. The losses are normalized by stop-gradient running means:

L=Lrulermruler+ϵ+Ldynmdyn+ϵ,(7)\mathcal{L} = \frac{\mathcal{L}_{\mathrm{ruler}}}{m_{\mathrm{ruler}}+\epsilon}+\frac{\mathcal{L}_{\mathrm{dyn}}}{m_{\mathrm{dyn}}+\epsilon}, \tag*{(7)}

The running means have a decay of 0.99. Each update uses 256 pairs and 64 windows with uniformly sampled scenes. For the main models, we use AdamW [22] for at most 20,000 updates and select checkpoints by validation loss; fixed-endpoint ablations and implementation details are described in § 5.3 and Appendix C, respectively.

Planning cost. For candidate and goal images of the same scene and object, the learned cost estimates their object-to-goal distance in meters:

c(Ic,Ig)=ℓ∥fϕ(zc)−fϕ(zg)∥2,(8)c(I_c,I_g)=\ell\left\|f_{\phi}(z_c)-f_{\phi}(z_g)\right\|_2, \tag*{(8)}

This replaces the feature distance of Eq. (1) in CEM [34] (§ 5.1). Candidate and goal are images of the same scene, so the free origin of the readout cancels in the cost. Our cost-isolation evaluations generate candidate futures with the simulator. We evaluate both the jointly trained (co-trained) readout and a fresh readout fitted on the readout-fitting partition with the encoder frozen, enabling a common readout protocol across training objectives.

Experiments

Setup

Benchmark and metrics. We evaluate on the 21 placement tasks, rendered arm-free (AF) and robot-visible (RV) from identical states; a task succeeds if the object ends within 15 cm of its goal. Decision-0 ranking scores a fixed bank of 300 simulated candidate futures per task and reports the Spearman correlation ρ\rho between cost and true object-to-goal distance, and the regret (true distance of the lowest-cost candidate minus that of the best). Closed-loop planning runs CEM [34] for six decisions of eight iterations each on simulator-generated futures [40, 45], which isolates the cost from prediction error, and reports the mean final distance and the number of solved tasks. Displacement error and further protocol details are in Appendix D. Models. We adapt the last eight blocks (Last 8) of DINOv3 and V-JEPA 2 ViT-L/16 on arm-free images only. Unless stated otherwise, the planning cost (Eq. (8)) uses the co-trained readout, trained jointly with the encoder, rather than a fresh one fitted afterwards with the encoder fixed. Each model has one training seed; reported deviations cover three planning repeats and measure execution variability only. Baselines. Frozen feature distance (Eq. (1)}) is computed over *all patches*; with patch differences weighted by the object’s coverage of each patch in the goal image (*goal mask*, where the object should be), in the candidate image (*candidate mask*, where it is) or by their sum (*both masks*); and between mask-pooled object features (*pooled object*, Eq. (3)`). We also compare with V-JEPA 2-AC planning with its released predictor, mask geometry (centroid pixel distance and its pinhole extension with mask-area depth), and an oracle that uses the true object-to-goal distance as its cost.

Main results

Planning. In arm-free images, the learned cost solves 15 of 21 tasks with both backbones, as many as the oracle (Table 2), at a mean final distance of 15.5 and 16.1 cm against the oracle’s 12.6 cm. Five of the six tasks our models fail also fail with the oracle, which points to the planner rather than the cost; the sixth has a hidden goal object, on which our cost abstains. Mask geometry solves at most 10 tasks, all of which our models also solve; frozen feature distance solves at most 9 and leaves the object 30 cm or more from the goal on average, and V-JEPA 2-AC with its own predictor solves only one of the 21 tasks.

CostClosed-loop planningDecision-0 ranking
Arm-freeRobot-visibleArm-freeRobot-visible
Dist. (cm)↓Succ.↑Dist. (cm)↓Succ.↑ρ\rho ↑Regret (mm)↓ρ\rho ↑Regret (mm)↓
Frozen feature distance
DINOv3-L, all patches41.238/2148.723/210.045127.360.200172.64
DINOv3-L, goal mask34.338/2149.402/21−0.002156.720.064188.35
DINOv3-L, candidate mask51.612/2147.713/21−0.078224.51−0.012235.34
DINOv3-L, both masks44.753/2148.371/21−0.011200.40−0.032180.96
DINOv3-L, pooled object30.019/2146.455/210.346146.030.114225.92
V-JEPA 2-AC-g, all patches47.506/2141.383/210.009219.930.153213.35
V-JEPA 2-AC-g, goal mask39.633/2140.923/210.018150.100.083187.51
V-JEPA 2-AC-g, candidate mask54.312/2149.831/21−0.045237.30−0.035244.94
V-JEPA 2-AC-g, both masks42.764/2147.353/21−0.009176.82−0.044210.75
V-JEPA 2-AC-g, pooled object33.538/2147.293/210.256141.860.153188.83
V-JEPA 2-AC-g, released predictor155.691/2154.670/21————
Mask geometry
Mask centroid19.439/2123.158/210.86850.210.45277.53
Mask centroid + size17.8610/2126.897/210.83722.590.21388.88
DOCK (ours)
DINOv3-L15.48±0.0515.48 \pm0.0515/2120.72±0.3620.72 \pm0.3612/210.91512.120.43586.97
V-JEPA 2-L16.13±0.5116.13 \pm0.5115/2124.01±0.2624.01 \pm0.269/210.89426.000.39585.82
Reference
True object distance (oracle)12.5615/2112.56∗12.56^{*}15/21∗15/21^{*}1.0000.001.0000.00

Table 2. Main results on the 21 placement tasks (protocol in § 5.1). ■ / ■: best and second-best value in each column, excluding the oracle; ■ / ■: best and second-best frozen-feature cost for each encoder.

Ranking. The learned costs also order the initial candidates best in arm-free images (ρ=0.915\rho= 0.915 and 0.894, against 0.868 for the mask centroid). The DINOv3 model halves the regret of the best baseline (12.1 against 22.6 mm for mask centroid + size), while the V-JEPA 2 model stays slightly above it (26.0 mm). Frozen feature distance stays far behind (ρ≤0.346\rho\le0.346, regret of at least 127 mm).

Mask pooling. Among the frozen-feature costs, pooling over the object mask, as our object token does (Eq. (3)), is the most effective use of the mask in arm-free images: it gives the highest correlation (ρ=0.346\rho= 0.346 and 0.256, against at most 0.045) and the best planning, whereas mask-weighted patch distances do not consistently improve on all patches; the other frozen encoders behave similarly (Tabs. 15 and 16). Pooling alone does not make the feature metric, however: with the same pooling, our trained encoder and readout raise the correlation ρ\rho from 0.346 to 0.915 for DINOv3 in arm-free images.

Robot-visible images. Our models never see the robot in training, yet with the arm rendered the DINOv3 model still solves 12 tasks at 20.7 cm and the V-JEPA 2 model 9, against 8 at 23.2 cm for the mask centroid. Decision-0 ranking is weak for every cost here, and the mask centroid even ranks the initial candidates slightly better than our models (ρ=0.452\rho= 0.452 against 0.435 and 0.395); yet over six decisions the DINOv3 model four more tasks. Frozen all-patch distance instead correlates better when the arm is visible (ρ\rho from 0.045 to 0.200 for DINOv3), consistent with a cost dominated by robot and background patches (§ 3.2).

Ablations

Ruler versus dynamics. Table 3 adapts the encoder with each loss alone and with both (ours), for 20,000 updates and with the same fresh-readout protocol. Test and Cube are the median displacement errors on held-out test pairs and on pairs with the object inside a 10 cm cube of each scene, which are withheld from training. Against the frozen encoder, our model lowers the test error from 24.0 to 15.2 mm @references】【。 One arm-free and two robot-visible goals of the instance-held-out tasks are hidden; those tasks abstain and count as failures, and ρ\rho is averaged over the other 9 and 8 tasks.

BackboneAdaptation lossError (mm)↓ TestError (mm)↓ CubeRanking, AF ρ ↑Ranking, AF Regret↓Ranking, RV ρ ↑Ranking, RV Regret↓Planning, AF Dist. (cm)↓Planning, AF Succ.↑Planning, RV Dist. (cm)↓Planning, RV Succ.↑
DINOv3-LNone (frozen encoder)23.9521.530.86265.070.341115.2818.78 ± 0.0610/2129.23 ± 0.057/21
DINOv3-LDynamics only25.4024.060.85066.730.348128.7923.75 ± 0.187/2129.90 ± 0.017/21
DINOv3-LRuler only14.8215.680.88826.860.42582.4717.05 ± 0.0913/2122.38 ± 0.0611/21
DINOv3-LRuler + dynamics15.1915.750.88523.150.41954.6515.75 ± 0.0915/2121.00 ± 0.1810/21
V-JEPA 2-LNone (frozen encoder)26.4024.470.84759.220.363116.8518.59 ± 0.0411/2132.27 ± 0.517/21
V-JEPA 2-LDynamics only28.1225.160.80481.720.350130.7826.86 ± 0.178/2127.34 ± 0.157/21
V-JEPA 2-LRuler only18.5118.200.87538.020.42975.0619.06 ± 0.0311/2121.81 ± 0.669/21
V-JEPA 2-LRuler + dynamics18.8516.950.87635.920.427100.2218.90 ± 0.0213/2126.53 ± 0.227/21

Table 3. Which loss adapts the encoder? Encoders adapted for 20,000 updates, each with a fresh readout. / : best and second-best value within each backbone. Regret in mm.

for DINOv3 and from 26.4 to 18.9 mm for V-JEPA 2, within 0.4 mm of ruler-only, with similar gains inside the cube. It gives the lowest arm-free regret for both backbones (23.2 and 35.9 mm) and solves the most arm-free tasks: 15 and 13, against 13 and 11 for ruler-only and at most 11 for the other two. For DINOv3 it also cuts the robot-visible regret of ruler-only from 82.5 to 54.7 mm and reaches the lowest final distance in both renderings, while for V-JEPA 2 the robot-visible results favor ruler-only.

Encoder adaptation. Table 4 compares our adapted encoder with a frozen one, for which only the readout and predictor are trained, and shows that training the encoder as well is necessary. With the co-trained readout, adaptation nearly halves the displacement error for DINOv3 (23.4 to 12.4 mm) and cuts it by a third for V-JEPA 2 (21.2 to 14.5 mm). It also improves the final distance in every planning setting and the ranking correlation in both renderings, for both backbones. The gain is largest when the unseen arm is visible, where adaptation raises the DINOv3 count from 8 to 12 tasks and lowers its mean final distance from 30.0 to 20.7 cm in closed-loop planning.

BackboneEncoderReadoutError (mm)↓ TestError (mm)↓ CubePlanning, AF Dist. (cm)↓Planning, AF Succ.↑Planning, RV Dist. (cm)↓Planning, RV Succ.↑Ranking, AF ρ ↑Ranking, AF Regret↓Ranking, RV ρ ↑Ranking, RV Regret↓
DINOv3-LFrozenCo-trained23.4224.1917.06 ± 0.0514/2129.96 ± 0.178/210.88319.710.314104.05
DINOv3-LFrozenFresh23.9521.5318.78 ± 0.0610/2129.23 ± 0.057/210.86265.070.341115.28
DINOv3-LLast 8Co-trained12.3915.2715.48 ± 0.0515/2120.72 ± 0.3612/210.91512.120.43586.97
DINOv3-LLast 8Fresh15.1715.8015.75 ± 0.0915/2121.00 ± 0.1810/210.88517.260.42050.34
V-JEPA 2-LFrozenCo-trained21.1824.0117.19 ± 0.1214/2126.79 ± 0.149/210.87725.880.363108.44
V-JEPA 2-LFrozenFresh26.4024.4718.59 ± 0.0411/2132.27 ± 0.517/210.84759.220.363116.85
V-JEPA 2-LLast 8Co-trained14.5419.1516.13 ± 0.5115/2124.01 ± 0.269/210.89426.000.39585.82
V-JEPA 2-LLast 8Fresh18.7916.8618.90 ± 0.0213/2126.53 ± 0.227/210.87635.920.427101.26

Table 4. Encoder adaptation and readout. Frozen: only the readout and predictor are trained; Last 8: the last eight encoder blocks are trained as well. / : best and second-best value within each backbone. Regret in mm.

Co-trained versus fresh readout. Table 4 also compares the co-trained readout with a fresh one fitted later with the encoder fixed, which tests whether the metric structure lies in the token or only in its jointly trained head. The co-trained readout is the better cost: in every configuration it has lower test error and arm-free regret and solves at least as many tasks. The gap depends on the encoder: on an adapted encoder a fresh readout solves 15 (DINOv3) and 13 (V-JEPA 2) arm-free tasks against 15 with the co-trained one, while on a frozen encoder it loses four and three tasks. This suggests that adaptation places the metric structure in the token itself, not in a particular readout.

Generalization to held-out instances and categories benchmark

We test transfer without any adaptation, readout fitting or calibration on two benchmarks whose target assets and kitchen layouts do not occur in the original 21 scenes: ten instance-held-out tasks, five of whose target categories still occur there, and nine category-held-out tasks whose target categories are new as well.

Unseen instances and layouts. The learned costs keep ranking candidates well in new kitchens (Table 5): in AF, ρ=0.697\rho= 0.697–0.8430.843 against at most 0.291 for frozen feature distance, at about half the regret (48–55 against 91–178 mm; Tab. 21). Planning gains are smaller and depend on the model: the best learned cost (V-JEPA 2, fresh readout) solves 5–6 of ten tasks against four for the best frozen baseline (pooled DINOv3), while the co-trained V-JEPA 2 readout solves only 2–3.

CostInstance-held-out (10 tasks) Arm-free Succ.↑Instance-held-out (10 tasks) Arm-free Ranking ρ\rho ↑Instance-held-out (10 tasks) Robot-visible Succ.↑Instance-held-out (10 tasks) Robot-visible Ranking ρ\rho ↑Category-held-out (9 tasks) Arm-free Succ.↑Category-held-out (9 tasks) Arm-free Ranking ρ\rho ↑Category-held-out (9 tasks) Robot-visible Succ.↑Category-held-out (9 tasks) Robot-visible Ranking ρ\rho ↑
Frozen feature distance
DINOv3-L, all patches2, 2, 20.0461, 1, 10.1842, 2, 20.0520, 0, 00.049
DINOv3-L, pooled object4, 4, 40.2911, 1, 10.0143, 3, 30.4822, 2, 20.228
V-JEPA 2-L, all patches1, 2, 20.1920, 0, 00.1041, 1, 10.0520, 0, 00.032
V-JEPA 2-L, pooled object1, 1, 10.0500, 0, 00.0011, 1, 10.1641, 1, 10.080
DOCK (ours), trained on the original 21 scenes only
DINOv3-L, co-trained readout5, 4, 40.7962, 2, 20.4885, 5, 50.7402, 2, 20.402
DINOv3-L, fresh readout4, 4, 50.8431, 1, 10.4406, 6, 60.7293, 3, 30.393
V-JEPA 2-L, co-trained readout3, 3, 20.6973, 3, 30.3204, 4, 40.6351, 1, 10.354
V-JEPA 2-L, fresh readout6, 5, 60.7832, 2, 20.4103, 3, 30.6563, 3, 30.353

Table 5. Transfer to held-out instances and categories, in unseen kitchen layouts. Succ.: tasks solved in each of the three repeats; ρ\rho: decision-0 ranking correlation. / /: best and second-best value in each column; / /: best and second-best frozen-feature cost. Final distance and regret are in Tabs. 21 and 22.

Unseen categories and layouts. The advantage holds for unseen categories: ρ=0.635\rho= 0.635–0.7400.740 in AF against at most 0.482, and DINOv3 with a fresh readout solves six of nine tasks, twice as many as pooled frozen DINOv3. Neither readout is consistently better on held-out tasks, and RV remains hard in both benchmarks: the learned costs solve 1–3 tasks against 0–2 for frozen features, so transfer does not remove the sensitivity to the visible robot.

Conclusion

Our profiling shows that feature distance, the cost of latent-space planners, fails for object placement: for six frozen encoders it is dominated by background and robot patches and barely tracks the object’s distance to its goal. Inspired by the ventral and dorsal visual streams, DOCK instead learns an object token (what), a spatial ruler (where) and a dynamics predictor (how) from held-object transport, supervised only by forward kinematics. With simulated futures, its metric cost solves 15 of 21 placement tasks, matching an oracle with the true object position, and still solves the most tasks with the unseen arm visible; its ranking transfers to unseen layouts, instances and categories.

LLM Usage Statement

We used large language models (Claude and Codex) to polish the writing of this paper, to write parts of the code, and to launch and schedule the experiments. The research ideas are our own, and all figures were drawn by the authors.

Impact Statement

This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

References

  1. [1]Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In ICCV, 2015.arxiv.org/abs/1505.01596
  2. [2]Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. In NeurIPS, 2016.arxiv.org/abs/1606.07419
  3. [3]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, 2023.arxiv.org/abs/2301.08243
  4. [4]Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025.
  5. [5]Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. TMLR, 2024.arxiv.org/abs/2404.08471
  6. [6]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.arxiv.org/abs/2104.14294
  7. [7]Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanchen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D awareness of visual foundation models. In CVPR, 2024.arxiv.org/abs/2404.08636
  8. [8]Stefano Ferraro, Pietro Mazzaglia, Tim Verbelen, and Bart Dhoedt. FOCUS: Object-centric world models for robotics manipulation. arXiv preprint arXiv:2307.02427, 2023.arxiv.org/abs/2307.02427
  9. [9]Peter R. Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In CoRL, 2018.arxiv.org/abs/1806.08756
  10. [10]Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self-supervised monocular depth estimation. In ICCV, 2019.arxiv.org/abs/1806.01260
  11. [11]Melvyn A. Goodale and A. David Milner. Separate visual pathways for perception and action. Trends in Neurosciences, 15(1):20–25, 1992.
  12. [12]Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020.arxiv.org/abs/2006.07733
  13. [13]Vitor Guizilini, Rares Ambrus, Sudeeep Pillai, Allan Raventos, and Adrien Gaidon. 3D packing for self-supervised monocular depth estimation. In CVPR, 2020.
  14. [14]David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NeurIPS, 2018.arxiv.org/abs/1809.01999
  15. [15]Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. In ICLR, 2020.
  16. [16]Peter J. Huber. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73–101, 1964.
  17. [17]Eric Jang, Coline Devin, Vincent Vanhoucke, and Sergey Levine. Grasp2Vec: Learning object representations from self-supervised grasping. In CoRL, 2018.arxiv.org/abs/1811.06964
  18. [18]Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In ICCV, 2015.
  19. [19]Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. In ICLR, 2020.
  20. [20]Yann LeCun. A path towards autonomous machine intelligence. OpenReview, version 0.9.2, 2022.
  21. [21]Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurIPS, 2020.arxiv.org/abs/2006.15055
  22. [22]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
  23. [23]Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. LIV: Language-image representations and rewards for robotic control. In ICML, 2023.arxiv.org/abs/2306.00958
  24. [24]Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value-implicit pre-training. In ICLR, 2023.arxiv.org/abs/2210.00030
  25. [25]Yunze Man, Shuhong Zheng, Zhipeng Bao, Martial Hebert, Liang-Yan Gui, and Yu-Xiong Wang. Lexicon3D: Probing visual foundation models for complex 3D scene understanding. In NeurIPS, 2024.arxiv.org/abs/2409.03757
  26. [26]Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake. kPAM: Keypoint affordances for category-level robotic manipulation. In ISRR, 2019.arxiv.org/abs/1903.06684
  27. [27]Mortimer Mishkin, Leslie G. Ungerleider, and Kathleen A. Macko. Object vision and spatial vision: two cortical pathways. Trends in Neurosciences, 6:414–417, 1983.DOI
  28. [28]Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-JEPA 2.1: Unlocking dense features in video self-supervised learning. arXiv preprint arXiv:2603.14482, 2026. 2
  29. [29]Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. In CoRL, 2022. 3arxiv.org/abs/2203.12601
  30. [30]Soroush Nasiriany, Abhiram Maddakuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. In RSS, 2024. 4arxiv.org/abs/2406.02523
  31. [31]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. TMLR, 2024. 1, 2
  32. [32]Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours. In ICRA, 2016. 2arxiv.org/abs/1509.06825
  33. [34]Reuven Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability, 1(2):127–190, 1999. 6DOI
  34. [35]Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. In ICLR, 2023. 2arxiv.org/abs/2209.14860
  35. [36]Junyao Shi, Jianing Qian, Yecheng Jason Ma, and Dinesh Jayaraman. Composing pre-trained object-centric representations for robotics from ”what” and ”where” foundation models. In ICRA, 2024. 2arxiv.org/abs/2404.13474
  36. [37]Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. DINOv3. arXiv preprint arXiv:2508.10104, 2025. 1, 2, 4arxiv.org/abs/2508.10104
  37. [38]Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. In ICML, 2018. 2arxiv.org/abs/1804.00645
  38. [39]Basile Terver, Tsung-Yen Yang, Jean Ponce, Adrien Bardes, and Yann LeCun. What drives success in physical planning with joint-embedding predictive world models? arXiv preprint arXiv:2512.24497, 2025. 1, 2arxiv.org/abs/2512.24497
  39. [40]Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In IROS, pages 5026–5033, 2012. 6DOI
  40. [41]Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. SlotFormer: Unsupervised visual dynamics simulation with object-centric models. In ICLR, 2023. 2arxiv.org/abs/2210.05861
  41. [42]Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, and Jan Eric Lenssen. Improving 2D feature representations by 3D-aware fine-tuning. In ECCV, 2024. 2arxiv.org/abs/2407.20229
  42. [43]Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. In ICML, 2025. 1, 2arxiv.org/abs/2411.04983
  43. [44]Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017. 2arxiv.org/abs/1704.07813
  44. [45]Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Kevin Lin, Abhiram Maddakuri, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020. 6arxiv.org/abs/2009.12293

Paper details

vlnc:W_k7cju6jg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude, Codex

View statement
Looking for related papers…
Contents