betaFeedback

WorksComputer Science

Frame First, Decode Once: Object-Centric Coordinates and Single-Pass Decoding for System-1 World Models

Abstract

Single-pass primitive decoders such as LocateAnything and VFlash predict all quantized coordinate tokens of a geometric primitive in one forward pass, enabling fast, System-1 visual planning. We study what matters when such a decoder serves as a System-1 world model: a fast action-conditioned predictor that ranks candidate pushes by their predicted outcome. On ManiSkill3 PushT-v1 (512 held-out test families, 8,192 rollouts) and on human demonstrations from lerobot/pusht (2,384 test windows), we compare decoders, training losses, rigid-body projections and coordinate frames, using matched checkpoints, hierarchical bootstrap confidence intervals and Holm correction. Two factors of similar size dominate. First, expressing inputs and outputs in the SE(2) frame of the object's initial pose lowers decision regret by 1.9-2.2x for each of five decoders, and the object-frame model trained on 128 families matches the world-frame model trained on 512. Second, single-pass decoding outperforms iterative decoding: in the object frame, single-pass heads reach 0.020-0.021 regret at 1.6-1.8 ms per decision, against 0.046 for 10-step DDIM diffusion and 0.059 for KV-cached autoregression at 10.3 and 12.9 ms. The choice among single-pass heads has no measurable effect on regret, and our registered hypothesis that the frame effect exceeds the decoder spread is not supported. Soft-token expectation with an end-to-end 2D Kabsch projection gives the lowest trajectory error (11.35 px on ManiSkill3, 3.78 px on lerobot/pusht), exact rigid-body consistency, and lower regret than diffusion at every budget from 4 to 50 DDIM steps, at 3-29x lower latency.

Introduction

CTION-CONDITIONED world models let an agent evaluate candidate actions by predicting their outcomes before acting [3], [18], [19]. A model that is queried inside the control loop, a fast “System-1” model in the sense of Kahneman [20], must do two things: run within a few milliseconds, and rank the candidate actions correctly. Vision-language and vision-language-action models represent geometry as sequences of quantized coordinate tokens ⟨0⟩…⟨1000⟩\langle0\rangle\ldots\langle1000\rangle [15]–[17]. Decoding such a trajectory token by token takes one sequential step per token, i.e. P×CP \times C steps for PP time steps of CC coordinates (100100 for P=C=10P=C=10). Even a coarser autoregression that emits one time step at a time, as in our baseline, needs about 12 ms per decision for a 3.8M-parameter model with a key-value cache, against about 1 ms for a single forward pass.

The author is with Sigmind.ai, Dhaka 1215, Bangladesh (e-mail: [email protected]).

LocateAnything [2] and VFlash [1] remove this bottleneck. Once the type of a primitive is known, its coordinate layout is fixed, so all of its coordinate slots can be predicted jointly in a single forward pass: slot positions attend to each other and to the preceding context. VFlash applies this flash primitive decoding (FPD) to visual plans such as robot trajectories and driving waypoints, and trains the slots with token-level cross-entropy.

We study single-pass slot decoders used as System-1 world models for contact-rich manipulation. The model predicts the trajectory of a pushed T-block under each of four candidate actions, and the agent picks the action with the highest predicted utility. Prior work leaves several design choices open, which we phrase as five questions:

  1. How does single-pass decoding compare with iterative decoders (DDIM diffusion, autoregression) built on the same backbone and data?

  2. Does supervising a soft-token expectation with a metric loss improve on token cross-entropy?

  3. Should rigid-body consistency come from direct SE(2) pose tokens or from redundant keypoints followed by a Procrustes projection?

  4. Should the projection be applied only at test time or trained end to end?

  5. Which matters more for decision quality, the coordinate frame or the decoder?

We answer these questions on ManiSkill3 PushT-v1 [4] and on the human demonstrations of lerobot/pusht [3], [5]. The study has three stages. A benchmark of twelve models was pre-registered before its test split was generated. Exploratory controls followed, including the object-frame models. Finally, a 2×52 \times5 frame ×\times decoder grid was registered after the exploratory object-frame result and run as a confirmatory replication on the same test families. Projected and unprojected variants of each head are evaluated on the same checkpoints, and each table of paired tests applies one Holm correction across all competitors it reports.

Our findings are:

  • The coordinate frame. Expressing inputs and outputs in the SE(2) frame of the object’s initial pose halves decision regret for every decoder we tested and gives roughly 4× sample efficiency. Controls on the RGB input show that the gain comes from the coordinate frame and not from removing the image.

  • Single-pass versus iterative decoding. On the same

backbone, single-pass heads have 2.3×–2.9× lower regret than a 10-step DDIM diffusion model and a KV-cached autoregressive model, and are 6×–8× faster. Which single-pass head is used makes no measurable difference to regret. The frame effect and the decoder effect are of comparable size, so our registered hypothesis that the frame dominates is not supported.

  • Procrustes-FPD. Soft-token expectation followed by a closed-form 2D Kabsch projection gives exact rigid-body consistency and the lowest trajectory error. Redundant keypoints with Procrustes fitting outperform direct SE(2) tokens in the world frame. We characterize when to apply the projection at test time and when to train through it, and we relate this trade-off to the rank of the projection’s Jacobian.

We also report an implementation error in our original diffusion baseline, its correction, and how it changed the results (Section V-D, Appendix A).

Throughout, “VFlash-style CE head” denotes our reimplementation of VFlash’s slot loss and decoding inside the same 2M-parameter backbone as every other model. We do not evaluate NVIDIA’s released model, and our results concern design choices for small action-conditioned predictors, not the VFlash system.

Parallel Primitive Decoding

LocateAnything [2] decodes each bounding box or point of a grounding answer as one unit in a single step, and VFlash [1] extends this idea to visual plans on top of a speculative drafter. Given the context of a frozen vision-language model, VFlash appends the BB positions of a primitive block. Positions inside the block attend to each other and to the context before the block, and all are predicted in one forward pass. Primitive positions are trained with token-level cross-entropy (Eq. (4) of [1], with unit weights on primitive positions):

LCE=−1PC∑t=1P∑c=1Clog⁡Pθ(zt,c∗∣x0,a).(1)\mathcal{L}_{\mathrm{CE}}=-\frac{1}{PC}\sum_{t=1}^{P}\sum_{c=1}^{C}\log P_{\theta}\left(z_{t,c}^{*}\mid x_{0},a\right). \tag*{(1)}

where, in our setting, zt,c∗=round⁡(1000⋅yt,c∗/512)∈{0,…,1000}z_{t,c}^{*}=\operatorname{round}(1000\cdot y_{t,c}^{*}/512)\in\{0,\ldots,1000\}. Eq. (1) removes the latency of autoregressive decoding. It is, however, indifferent to the distance between bins: predicting ⟨501⟩ or ⟨999⟩ for a target ⟨500⟩ incurs the same loss as long as both receive the same probability.

Soft-Bin Regression and Keypoint-Based Pose Estimation

Taking the expectation over discrete bins so that a continuous metric loss can be applied was introduced in integral human pose regression [6], differentiable spatial-to-numerical transforms [7], and Distribution Focal Loss for bounding-box edges [8]. In 6-DoF pose estimation and point-cloud registration, predicting redundant keypoints and fitting the pose in closed form with Kabsch [11] or orthogonal Procrustes [12] is known to outperform direct regression of angles or quaternions [9], [10], [13], [14]. We combine the two ideas inside a single-pass slot decoder and analyze the difference between applying the projection at test time and training through it.

Object-Centric and Equivariant Coordinate Frames

Expressing states and actions in an agent- or object-centric frame is standard in motion forecasting, e.g. VectorNet [21] and MTR [22], and equivariant architectures exploit the same symmetry in manipulation, e.g. Equivariant Diffusion Policy [23] and EquiBot [24]. Tokenized decoders, in contrast, usually express coordinates in the frame of the sensor: VFlash normalizes image coordinates to [0,1000][0,1000] and places 3D robot coordinates in the current camera frame (and driving waypoints in the ego frame), and general-purpose VLA models also tokenize image or robot-base coordinates [15]–[17]. We measure how much the frame matters relative to the choice of decoder, for five decoder families, and examine where the gain comes from.

Method

Problem Formulation

At decision time t0t_0 the world model receives an RGB image It0∈[0,1]3×128×128I_{t_0}\in[0,1]^{3\times128\times128}, a 4-step history Ht0−3:t0∈R4×10\mathbf{H}_{t_0-3:t_0}\in\mathbb{R}^{4\times10} of the K=4K=4 object keypoints Kt∈R4×2\mathbf{K}_t\in\mathbb{R}^{4\times2} and the pusher position et∈R2\mathbf{e}_t\in\mathbb{R}^{2}, the goal keypoints G∈R4×2\mathbf{G}\in\mathbb{R}^{4\times2}, and a candidate action a∈R4\mathbf{a}\in\mathbb{R}^{4}. It predicts the P=10P=10-step future Y1:P={[Kt,et]}t=1P∈[0,512]P×10\mathbf{Y}_{1:P}=\{[\mathbf{K}_t,\mathbf{e}_t]\}_{t=1}^{P}\in[0,512]^{P\times10}. Each candidate is evaluated with the task utility U(Y1:P,G,K0)∈[0,1]U(\mathbf{Y}_{1:P},\mathbf{G},\mathbf{K}_0)\in[0,1], which rewards final keypoints close to the goal and penalizes the final orientation error (Section IV-B), and the agent selects the candidate with the highest predicted utility.

Soft-Token Expectation

Let Lt,c∈R1001\mathbf{L}_{t,c}\in\mathbb{R}^{1001} be the logits of slot (t,c)(t,c), for time step t∈{1,…,P}t\in\{1,\ldots,P\} and coordinate c∈{1,…,10}c\in\{1,\ldots,10\}, and pt,c,b=softmax⁡b(Lt,c,b)p_{t,c,b}=\operatorname{softmax}_{b}(\mathbf{L}_{t,c,b}). Instead of decoding the most likely token, as the CE head does, we take the expectation [6], [8] in workspace units:

y^t,c=5121000∑b=01000b⋅pt,c,b.(2)\hat{y}_{t,c}=\frac{512}{1000}\sum_{b=0}^{1000}b\cdot p_{t,c,b}. \tag*{(2)}

Its gradient with respect to each logit is proportional to the distance between that bin and the current estimate,

∂y^t,c∂Lt,c,b=5121000pt,c,b(b−1000512y^t,c).(3)\frac{\partial\hat{y}_{t,c}}{\partial L_{t,c,b}}=\frac{512}{1000}p_{t,c,b}\left(b-\frac{1000}{512}\hat{y}_{t,c}\right). \tag*{(3)}

so a metric loss on y^t,c\hat{y}_{t,c} penalizes far-away bins more than near ones.

2D Procrustes (Kabsch) Projection

Even with a metric loss, the four keypoints are predicted independently in R8\mathbb{R}^{8}. Small inconsistencies between their predicted motions deform the block: the Soft-L1 head has a mean pairwise-distance error of 9.14 px on ManiSkill3, and every predicted trajectory violates rigidity (Table I).

To enforce rigidity, let K0={k0,i}i=1K\mathbf{K}_0=\{\mathbf{k}_{0,i}\}_{i=1}^{K} be the observed initial keypoints with centroid kˉ0\bar{\mathbf{k}}_0 and centered coordinates k~0,i=k0,i−kˉ0\tilde{\mathbf{k}}_{0,i}=\mathbf{k}_{0,i}-\bar{\mathbf{k}}_0. For the predicted keypoints K^t={k^t,i}i=1K\hat{\mathbf{K}}_t=\{\hat{\mathbf{k}}_{t,i}\}_{i=1}^{K} at step tt we find the rigid motion that best explains

(R^t,c^t)=arg⁡min⁡R∈SO(2), c∈R2∑i=1K∥c+Rk~0,i−k^t,i∥22.(4)(\hat{\mathbf{R}}_t,\hat{\mathbf{c}}_t)=\underset{\mathbf{R}\in\mathrm{SO}(2),\,\mathbf{c}\in\mathbb{R}^{2}}{\arg\min}\sum_{i=1}^{K}\left\|\mathbf{c}+\mathbf{R}\tilde{\mathbf{k}}_{0,i}-\hat{\mathbf{k}}_{t,i}\right\|_{2}^{2}. \tag*{(4)}
Procrustes-FPD architecture and supervision

Figure 1. Procrustes-FPD. Given a context x0x_0 and a candidate action aa, a 3-layer transformer (d=192d=192) with a block attention mask predicts the token distributions of all 100 trajectory coordinates in one forward pass. The soft-token expectations of the four object keypoints, Y^t∈R4×2\hat{Y}_t \in\mathbb{R}^{4 \times2}, are projected onto the rigid motions of the observed initial shape K0K_0 by a closed-form 2D Procrustes fit, ΠKabsch\Pi_{\mathrm{Kabsch}}. The projection is applied either at test time only (Pre-L1, α=1\alpha=1) or inside the training loss (E2E-L1, α=0\alpha=0). The predicted trajectory is scored with the task utility UU. Latency is for one decision over four candidate actions (Table I).

In 2D the minimizer has a closed form that needs neither an SVD nor an atan2 [11], [12]:

c^t=1K∑i=1Kk^t,i,k~t,i=k^t,i−c^t,(5)\hat{c}_t = \frac{1}{K}\sum_{i=1}^{K}\hat{k}_{t,i}, \qquad\tilde{k}_{t,i}=\hat{k}_{t,i}-\hat{c}_t, \tag*{(5)}
αt=∑i=1K(k~0,i,xk~t,i,x+k~0,i,yk~t,i,y),(6)\alpha_t=\sum_{i=1}^{K}\left(\tilde{k}_{0,i,x}\tilde{k}_{t,i,x}+\tilde{k}_{0,i,y}\tilde{k}_{t,i,y}\right), \tag*{(6)}
βt=∑i=1K(k~0,i,xk~t,i,y−k~0,i,yk~t,i,x),(7)\beta_t=\sum_{i=1}^{K}\left(\tilde{k}_{0,i,x}\tilde{k}_{t,i,y}-\tilde{k}_{0,i,y}\tilde{k}_{t,i,x}\right), \tag*{(7)}
cos⁡θ^t=αtαt2+βt2+ϵ,sin⁡θ^t=βtαt2+βt2+ϵ.(8)\cos\hat{\theta}_t=\frac{\alpha_t}{\sqrt{\alpha_t^2+\beta_t^2+\epsilon}}, \qquad\sin\hat{\theta}_t=\frac{\beta_t}{\sqrt{\alpha_t^2+\beta_t^2+\epsilon}}. \tag*{(8)}

The projected keypoints are ΠKabsch(K^t,K0)i=c^t+R^(θ^t)k~0,i\Pi_{\mathrm{Kabsch}}(\hat{K}_t,K_0)_i=\hat{c}_t+\hat{R}(\hat{\theta}_t)\tilde{k}_{0,i}. The map is differentiable wherever αt2+βt2>0\alpha_t^2+\beta_t^2>0.

Why keypoints rather than pose tokens. Suppose each keypoint slot had independent, isotropic error with variance σ2\sigma^2 per coordinate. Linearizing (5)--(8) around the true pose shows that the projection keeps only the component of the 2K2K-dimensional error that lies in the 3-dimensional tangent space of SE(2)\mathrm{SE}(2). It removes the 2K−3=52K-3=5 non-rigid components, which carries 5/85/8 of the error energy. Slot errors from a shared backbone are correlated, so this is a motivation, not a guarantee. A second consideration is that predicting the visible corners in image coordinates preserves the spatial correspondence between output and input, whereas a rotation token Δθt\Delta\theta_t turns an angular error into a position error scaled by the object’s moment arm ∥k~0,i∥2\lVert\tilde{k}_{0,i}\rVert_2. § 5 tests this choice directly.

Pre-L1 versus E2E-L1 Supervision

All trajectory decoders are trained with

L(α)=LCEls=0.02(L,z∗)+0.5 LCE(m,m∗)+15α LSL1(Y^,Y∗)+15(1−α)LSL1(ΠKabsch(Y^,K0),Y∗),(9)\mathcal{L}(\alpha)=\mathcal{L}_{\mathrm{CE}}^{\mathrm{ls}=0.02}(L,z^*)+0.5\,\mathcal{L}_{\mathrm{CE}}(m,m^*)+15\alpha\,\mathcal{L}_{\mathrm{SL1}}(\hat{Y},Y^*)+15(1-\alpha)\mathcal{L}_{\mathrm{SL1}}\left(\Pi_{\mathrm{Kabsch}}(\hat{Y},K_0),Y^*\right), \tag*{(9)}

where the first term is token cross-entropy with label smoothing 0.02, mm are the logits of an auxiliary contact-mode head, and LSL1\mathcal{L}_{\mathrm{SL1}} is the Smooth-L1L_1 loss on coordinates divided by 512. The CE head uses the first two terms only, without label smoothing: (1) plus the same auxiliary contact-mode term. With α=1\alpha=1 (Pre-L1) the Smooth-L1L_1 loss acts on the raw soft-token outputs and the projection is applied only at test time; Pre-L1 is therefore the Soft-L1 head followed by ΠKabsch\Pi_{\mathrm{Kabsch}}. With α=0\alpha=0 (E2E-L1) the loss acts on the projected trajectory, and its gradient flows through ΠKabsch\Pi_{\mathrm{Kabsch}}. Both variants have 2,050,733 parameters and the same inference cost.

Proposition 1 (Rank of the projection Jacobian). For K=4K=4 planar keypoints, y^t∈R8\hat{y}_t \in\mathbb{R}^{8}, the image of ΠKabsch(y^t,K0)\Pi_{\mathrm{Kabsch}}(\hat{y}_t,K_0) is the 3-dimensional manifold M(K0)⊂R8\mathcal{M}(K_0) \subset\mathbb{R}^{8} of rigid motions of K0K_0. At any y^t\hat{y}_t with αt2+βt2>0\alpha_t^2+\beta_t^2>0, the Jacobian JΠ=∂ΠKabsch(y^t,K0)/∂y^t∈R8×8J_{\Pi}=\partial\Pi_{\mathrm{Kabsch}}(\hat{y}_t,K_0)/\partial\hat{y}_t \in\mathbb{R}^{8\times8} has rank 3. Its 5-dimensional null space consists of the perturbations that leave (c^t,θ^t)(\hat{c}_t,\hat{\theta}_t) unchanged to first order.

Proof. ΠKabsch\Pi_{\mathrm{Kabsch}} factors as ψ∘ϕ\psi\circ\phi, where ϕ(y^t)=(c^t,c^t,θ^t)∈R2×S1\phi(\hat{y}_t)=(\hat{c}_t,\hat{c}_t,\hat{\theta}_t)\in\mathbb{R}^2\times S^1 and ψ(c,θ)i=c+R(θ)k0,i\psi(c,\theta)_i=c+R(\theta)k_{0,i}. The map ϕ\phi is a submersion where αt2+βt2>0\alpha_t^2+\beta_t^2>0: the centroid depends on all coordinates with full rank 2, and a rigid rotation of the centered prediction changes θ^t\hat{\theta}_t at unit rate. The map ψ\psi is an immersion for any non-degenerate shape K0K_0. Hence rank⁡(JΠ)=rank⁡(dψ dϕ)=3\operatorname{rank}(J_{\Pi})=\operatorname{rank}(d\psi\,d\phi)=3 and dim⁡ker⁡JΠ=8−3=5\dim\ker J_{\Pi}=8-3=5.

Proposition 1 is consistent with the trade-off we observe between the two variants. With Pre-L1, the Smooth-L1L_1 gradient constrains all 8 keypoint coordinates, so each slot is trained to track its own corner. With E2E-L1, the loss gives no gradient along the 5 non-rigid directions, so the slots are only constrained through the pose they jointly imply. On the continuous human demonstrations of lerobot/pusht this lowers trajectory error (11.76 →\to 10.17 px). On ManiSkill3, where the decision depends on small differences between candidate actions, it raises regret (0.040 →\to 0.048; § 5). The proposition does not by itself predict the sign of either effect.

Experimental Setup

Benchmarks

ManiSkill3 PushT-v1. We build a counterfactual benchmark on ManiSkill3 [4] (SAPIEN/PhysX on CPU, Panda arm with a stick tool). It has 512 training, 256 validation and 512 test families. A family fixes the initial T-block and pusher state; from it, 4 candidate actions are each rolled out under 4 hidden physics realizations that vary friction ([0.05, 2.5]), center-of-gravity offset, mass and inertia. This gives 16 rollouts per family and 8,192 test rollouts. The test split was generated after the benchmark was registered and was not used for any model selection. As a reference, an oracle that knows the true distribution over four contact modes and scores each action by its mean training utility per mode reaches 0.069 [0.059, 0.080] regret (47.1% Top-1). Ranking actions therefore requires predicting trajectory geometry, not only the contact mode. lerobot/pusht. The second benchmark uses the 206 human-teleoperated demonstrations of the simulated Push-T task released with Diffusion Policy [3] and distributed through LeRobot [5] (25,650 frames at 10 Hz). Splits are episode-disjoint: episodes 0–139 for training (7,663 windows), 140–165 for validation (1,487 windows) and 166–205 for testing (2,384 windows). The same test split was used in every stage of the study. The four T-block keypoints (junction, stem tip, left and right bar ends) are midpoints of the eight vertices given in lerobot/pusht_keypoints; their pairwise distances vary by less than 10−510^{-5} px over all frames. Utility is computed against the fixed goal pose [256,256,π/4][256,256,\pi/4] of the environment. Because every window has only one recorded outcome, this benchmark measures trajectory error, not decision regret.

Metrics

Coordinates are in workspace units: the plan workspace is mapped to [0,512][0,512], and we report distances in these units as px. The utility of a predicted outcome is

U=clip⁡(1−dfinalmax⁡(dinit,30),0,1)exp⁡(−(Δθ0.35)2),(10)U = \operatorname{clip}\left(1-\frac{d_{\mathrm{final}}}{\max(d_{\mathrm{init}},30)},0,1\right)\exp\left(-\left(\frac{\Delta\theta}{0.35}\right)^2\right), \tag*{(10)}

where dd is the mean keypoint distance to the goal and Δθ\Delta\theta is the wrapped orientation error of the junction-to-stem-tip vector.

  • Regret: for each test family, the true mean utility (over the four realizations) of the best action minus that of the action the model selects, averaged over families. Top-1: the percentage of families in which the selected action is the best one.

  • MAP-ADE: the mean Euclidean error over the 10 steps and 5 points (four keypoints and the pusher) of the model’s point prediction, averaged over realizations. MinADE16_{16}: the same for the best of 16 samples (stochastic decoders only).

  • RB-Err: the mean absolute deviation of the six pairwise keypoint distances from those of the true shape. RB-Viol: the percentage of predicted trajectories whose per-step deviation exceeds 2.5 px at any step.

  • Brier: the Brier score of the predicted contact-mode distribution. U-MAE: the absolute error of the predicted utility (lerobot/pusht).

Models

All models share the same context encoder: a small convolutional image encoder and a 3-layer transformer of width d=192d=192 over eight prefix tokens (image features, keypoint history, relative geometry, goal and action). They differ only in the output head and decoding procedure. The CE head (VFlash-style) is trained with Eq. (1). The Soft-L1 head adds the soft-token Smooth-L1L_1 loss (α=1\alpha=1), and Procrustes-FPD adds the Kabsch projection (Pre-L1 or E2E-L1). The MDN head predicts a 4-component mixture of trajectories, and Bi-FPD routes its components by predicted contact mode. The SE(2)-token variants of the CE, Soft-L1 and MDN heads predict the block pose (Δxt,Δyt,Δθt)(\Delta x_t,\Delta y_t,\Delta\theta_t) and the pusher with 50 slots instead of 100 keypoint slots. The iterative baselines are an autoregressive decoder that emits one time step (all 10 coordinates) per step, with a key-value cache (10 sequential steps), and a DDIM diffusion model over whole trajectories (10 steps unless stated). Both draw 16 samples per candidate. We also include a direct utility regressor and a contact-mode classifier that score actions without predicting a trajectory. Parameter counts range from 1.9M to 4.2M, mainly because of the output layers. The slot head shares one 1001-way output layer across slots (about 0.19M parameters), whereas the autoregressive and mixture heads predict all 10 coordinates of a step from one token (d×10×1001≈1.9d \times10 \times1001 \approx1.9M parameters per output layer). Latency is the median of 200 calls for one decision, i.e. a batch of four candidate actions with S=16S=16 samples where applicable, on one GPU. Latencies in Tables I and V come from separate measurement runs (the latter includes the frame transform) and should be compared only within a table. Table VI maps model names to identifiers in the released code.

Study Design and Statistics

  1. Pre-registered benchmark. Twelve models, including the CE, Soft-L1, MDN, Bi-FPD, autoregressive, diffusion, direct-regression and classifier models above, were registered with their hypotheses before the ManiSkill3 test split was generated. Learning rate, weight decay, dropout and loss weights were selected on the validation split only and then frozen. The remaining four registered models, a CE head with an in-block contact-mode token and three Bi-FPD ablations, address a question outside this paper; we omit them from the tables, and their results will be included in the released files.

  1. Exploratory controls. The Kabsch-projected variants, the SE(2)-token variants, E2E-L1 and all object-frame models were added afterwards. They reuse the frozen hyperparameters of their parent models without re-tuning, and we label them as exploratory.

  1. Matched checkpoints. The CE, Soft-L1 and MDN heads were retrained together with their projected and SE(2)-token variants, and each projected variant is evaluated on exactly the same checkpoints as its unprojected parent. Retraining the CE head with identical seeds (0–4) and configuration moved its mean test regret from 0.0457 to 0.0423, a difference of 0.0034 caused by nondeterministic CUDA kernels. The Soft-L1 head moved from 0.0424 to 0.0423, and the MDN head from 0.0440 to 0.0443. Family-cluster bootstrap intervals condition on the trained checkpoints, so regret differences below about 0.0035 in Tables I–IV are within run-to-run variation. Bi-FPD, the autoregressive model, the direct regressor and the classifier were not retrained and come from the original benchmark run. Their regret gaps to Pre-L1 (0.0082 and larger) exceed this retraining shift.

  1. Registered grid. After the exploratory object-frame results, we registered a 2×5 frame × decoder grid with hypotheses H1–H5, a corrected diffusion model and a data-scaling sweep, and ran it on the same test families. The grid uses a single training path for all decoders, removes the RGB input in both frames, and uses a seed × family hierarchical bootstrap that also resamples training seeds.

  1. Multiple comparisons. Each table of paired tests applies one Holm step-down correction across all competitors it reports against its reference, including weak and close ones, plus the superseded diffusion arm; family sizes are stated in each table. All bootstraps use B=2000B=2000 resamples. A reported p=0.0005p=0.0005 means that no resample crossed zero.

Results

ManiSkill3, World Frame

Table I, the top half of Table III and Fig. 2 report the matched world-frame comparison.

Trajectory prediction beats direct scoring. Procrustes-FPD (Pre-L1) has lower regret (0.040, 1.07 ms) than the KV-cached autoregressive model (0.076, 11.98 ms), the direct utility regressor (0.052) and the contact-mode classifier (0.060; all pHolm(14)=0.0070p_{\mathrm{Holm}(14)}=0.0070).

Soft-token expectation improves trajectory error. The Soft-L1 head lowers MAP-ADE from 25.6 px (CE head) to 20.9 px, with non-overlapping 95% CIs; regret is unchanged (0.042 vs. 0.042). Projecting the Soft-L1 output with ΠKabsch\Pi_{\mathrm{Kabsch}} (Pre-L1) removes all rigid-body error (from 9.14 px and 100% violations to zero) and lowers MAP-ADE by a further 0.91 px to 20.0 px (pHolm(13)=0.0065p_{\mathrm{Holm}(13)}=0.0065). Pre-L1 has the lowest MAP-ADE of all models, significantly below every other model in Table I (pHolm(13)=0.0065p_{\mathrm{Holm}(13)}=0.0065), e.g. 5.63 px below the CE head and 3.26 px below the CE head with the same projection.

The best decision regrets are statistically tied. Pre-L1 has significantly lower regret than the SE(2)-token Soft-L1 head (Δ=+0.0094\Delta=+0.0094), the SE(2)-token CE head (+0.0087+0.0087), Bi-FPD (+0.0082+0.0082) and E2E-L1 (+0.0076+0.0076, all pHolm(14)=0.0070p_{\mathrm{Holm}(14)}=0.0070). Its advantage over the CE head (+0.0024+0.0024, pHolm(14)=0.516p_{\mathrm{Holm}(14)}=0.516), the SE(2)-token MDN head (+0.0014+0.0014, pHolm(14)=0.531p_{\mathrm{Holm}(14)}=0.531), the unprojected Soft-L1 head (+0.0024+0.0024, unadjusted p=0.025p=0.025, pHolm(14)=0.125p_{\mathrm{Holm}(14)}=0.125) and the MDN head (+0.0044+0.0044, p=0.028p=0.028, pHolm(14)=0.125p_{\mathrm{Holm}(14)}=0.125) is not significant after correction. Most of these differences are also within the run-to-run variation of about 0.0035. In the world frame, therefore, the decoder head affects trajectory accuracy and rigidity much more than it affects decisions.

Keypoints with Procrustes fitting beat direct pose tokens. With identical backbones and losses, 100 keypoint slots followed by ΠKabsch\Pi_{\mathrm{Kabsch}} outperform 50 direct SE(2) slots in MAP-ADE (20.0 vs. 22.9 px, pHolm(13)=0.0065p_{\mathrm{Holm}(13)}=0.0065) and in regret (0.040 vs. 0.049, pHolm(14)=0.0070p_{\mathrm{Holm}(14)}=0.0070), consistent with findings in 6-DoF pose estimation [9, 10] (Fig. 4).

lerobot/pusht, World Frame

Table II, the bottom half of Table III and Fig. 3 report the 3-seed evaluation on the 40 test episodes.

Projection helps every Cartesian decoder. Applying ΠKabsch\Pi_{\mathrm{Kabsch}} at test time lowers MAP-ADE for the CE head (16.53 → 14.38 px), the Soft-L1 head (12.78 → 11.76 px), the MDN head (16.35 → 15.61 px), Bi-FPD (16.75 → 15.79 px), the direct regressor’s auxiliary keypoints (16.40 → 15.11 px) and the autoregressive model (25.43 → 24.11 px). As on ManiSkill3, the Soft-L1 head improves on the CE head (16.53 vs. 12.78 px raw, non-overlapping CIs).

Training through the projection gives the lowest trajectory error here. E2E-L1 reaches 10.17 px, lower than every other model in Table II (pHolm(9)=0.0045p_{\mathrm{Holm}(9)}=0.0045), including Pre-L1 (11.76 px, Δ=+1.59\Delta=+1.59 px) and the SE(2)-token Soft-L1 head (13.14 px, Δ=+2.97\Delta=+2.97 px). On utility error, the mode-routed Bi-FPD is best (0.059 U-MAE vs. 0.068 for E2E-L1). Together with the ManiSkill3 results, this shows that Pre-L1 and E2E-L1 trade places: E2E-L1 gives lower trajectory error on human demonstrations, while Pre-L1 gives lower regret on the counterfactual benchmark. Both were exploratory additions, so we report them as an ablation and do not select a single winner.

ModelMAP-ADE Raw ↓MAP-ADE Kabsch ↓MinADE16_{16} Raw ↓MinADE16_{16} Kabsch ↓RB-Err Raw ↓RB-Err Kabsch ↓U-MAE Kabsch ↓
Procrustes-FPD, E2E-L110.17 [9.33, 11.10]10.17 [9.33, 11.10]——0.00 [0.00, 0.00]0.000 [0.000, 0.000]0.068 [0.061, 0.076]
Soft-L1 head12.78 [11.90, 13.76]11.76 [10.88, 12.73]——6.50 [6.06, 6.92]0.000 [0.000, 0.000]0.086 [0.077, 0.095]
SE(2) tokens, Soft-L113.14 [12.27, 14.16]13.14 [12.27, 14.16]——0.00 [0.00, 0.00]0.000 [0.000, 0.000]0.071 [0.063, 0.080]
SE(2) tokens, MDN13.92 [12.81, 15.13]13.92 [12.81, 15.13]12.59 [11.53, 13.77]12.59 [11.53, 13.77]0.00 [0.00, 0.00]0.000 [0.000, 0.000]0.064 [0.056, 0.073]
CE head (VFlash-style)16.53 [15.46, 17.68]14.38 [13.43, 15.40]17.97 [16.75, 19.27]14.94 [13.83, 16.11]9.97 [9.25, 10.69]0.000 [0.000, 0.000]0.067 [0.060, 0.075]
MDN head (4 branches)16.35 [15.06, 17.86]15.61 [14.31, 17.11]15.04 [13.77, 16.53]14.21 [12.94, 15.68]6.33 [5.71, 7.03]0.000 [0.000, 0.000]0.080 [0.070, 0.090]
Direct utility regression16.40 [15.68, 17.13]15.11 [14.43, 15.81]——8.74 [8.27, 9.29]0.000 [0.000, 0.000]0.080 [0.072, 0.087]
Bi-FPD (mode-routed MDN)16.75 [15.44, 18.25]15.79 [14.55, 17.22]15.19 [14.05, 16.49]14.07 [12.99, 15.32]6.09 [5.44, 6.77]0.000 [0.000, 0.000]0.059 [0.051, 0.067]
Autoregressive (KV cache)25.43 [23.31, 27.79]24.11 [22.07, 26.42]24.68 [22.84, 26.73]21.34 [19.62, 23.26]6.84 [6.30, 7.45]0.000 [0.000, 0.000]0.069 [0.060, 0.079]

Table II. LEROBOT/PUSHT TEST SPLIT, WORLD FRAME (40 TEST EPISODES, 2,384 WINDOWS; 3 SEEDS; EPISODE-CLUSTER BOOTSTRAP, B=2000B=2000, 95% CIS; FIXED GOAL POSE [256,256,π/4][256, 256, \pi/4]). RAW: UNPROJECTED DECODER OUTPUT; KABSCH: AFTER PROJECTION ONTO THE RIGID-BODY MANIFOLD. FOR THE SOFT-L1 HEAD THE KABSCH COLUMN IS PROCRUSTES-FPD (PRE-L1). ERRORS IN PIXELS. THE SUPERSEDED DIFFUSION ARM IS REPORTED IN TABLE VII.

Figure 2. ManiSkill3 test split, world frame (512 families, 5 seeds, 95% family-cluster bootstrap CIs). (a) Decision regret and (b) trajectory error (MAP-ADE) against latency per decision (log scale).

The Object Frame

In Table 1, all eleven single-pass world-frame models have regret between 0.040 and 0.049. One possible reason is a noise floor: the regret targets are means over only four realizations, uˉf,a=14∑r=14uf,a,r\bar{u}_{f,a} = \frac{1}{4}\sum_{r=1}^{4} u_{f,a,r}. Bootstrapping the four realizations over the 512 test families, however, puts this floor at only 0.0047 regret (std 0.0009). A second possibility is overfitting, and a world-frame Soft-L1 model (seed 0) shows a large generalization gap:

ModelStatusRegret ↓\downarrowTop-1 % ↑\uparrowMAP-ADE ↓\downarrow (px)MinADE16_{16} ↓\downarrow (px)RB-Err ↓\downarrow (px)RB-Viol % ↓\downarrowLatency (ms)
Procrustes-FPD, Pre-L1Expl.0.040 [0.034, 0.046]60.2 [56.9, 63.6]20.0 [19.5, 20.5]—0.00 [0.00, 0.00]0.0%1.07
Soft-L1 headPre-reg.0.042 [0.036, 0.049]59.1 [55.9, 62.6]20.9 [20.4, 21.5]—9.14 [8.85, 9.42]100.0%1.07
Procrustes-FPD, E2E-L1Expl.0.048 [0.041, 0.054]57.8 [54.5, 61.1]20.7 [20.1, 21.3]—0.00 [0.00, 0.00]0.0%1.07
SE(2) tokens, Soft-L1Expl.0.049 [0.043, 0.056]56.0 [52.8, 59.4]22.9 [22.3, 23.4]—0.00 [0.00, 0.00]0.0%1.05
CE head (VFlash-style)Pre-reg.0.042 [0.037, 0.049]57.8 [54.5, 61.2]25.6 [24.9, 26.3]25.2 [24.5, 25.8]14.40 [14.00, 14.78]100.0%1.21
CE head + KabschExpl.0.044 [0.037, 0.051]57.5 [54.1, 60.9]23.2 [22.6, 23.9]21.5 [21.0, 22.2]0.00 [0.00, 0.00]0.0%1.21
SE(2) tokens, CEExpl.0.049 [0.042, 0.055]53.8 [50.5, 57.0]25.5 [24.9, 26.2]24.0 [23.4, 24.5]0.00 [0.00, 0.00]0.0%1.40
MDN head (4 branches)Pre-reg.0.044 [0.038, 0.051]57.0 [53.6, 60.4]25.0 [24.4, 25.7]19.8 [19.3, 20.4]7.16 [6.93, 7.39]100.0%1.18
MDN head + KabschExpl.0.046 [0.039, 0.053]56.3 [52.9, 59.6]24.6 [23.9, 25.2]19.2 [18.6, 19.7]0.00 [0.00, 0.00]0.0%1.18
SE(2) tokens, MDNExpl.0.041 [0.036, 0.047]57.1 [53.5, 60.7]24.4 [23.8, 25.0]18.3 [17.8, 18.8]0.00 [0.00, 0.00]0.0%1.18
Bi-FPD (mode-routed MDN)Pre-reg.0.048 [0.041, 0.056]55.7 [52.4, 59.1]24.3 [23.6, 25.0]20.0 [19.6, 20.5]6.74 [6.55, 6.94]99.9%1.52
Direct utility regressionPre-reg.0.052 [0.045, 0.059]52.0 [48.6, 55.5]34.7 [34.1, 35.3]—13.37 [13.13, 13.60]100.0%0.81
Contact-mode classifierPre-reg.0.060 [0.053, 0.067]48.9 [45.6, 52.5]————1.25
Autoregressive (KV cache)Pre-reg.0.076 [0.068, 0.085]44.1 [41.3, 47.1]37.2 [36.4, 38.1]27.3 [26.7, 28.0]7.83 [7.66, 8.00]99.1%11.98

Table 1. MaNiSkill3 PushT-v1 test split, world frame (512 families × 4 candidate actions × 4 physics realizations = 8,192 rollouts; 5 seeds; family-cluster bootstrap, B=2000B=2000, 95% CIs). The CE, Soft-L1 and MDN heads and their Kabsch-projected and SE(2)-token variants come from one training batch, with each projected variant evaluated on its parent’s checkpoints; the other models come from the original benchmark run. Pre-reg.: arm of the pre-registered benchmark; Expl.: exploratory control added afterwards. Latency: median over 200 calls for one decision (batch of 4 candidates, S=16S=16 samples where applicable). The pre-registered diffusion arm is omitted here because its implementation was faulty (Section V-D); its numbers are in Table VII and the corrected model is in Table V.

  • Training split (512 families): regret 0.0064, Top-1 82.6%, MAP-ADE 7.65 px.

  • Validation split (256 families): regret 0.0380, Top-1 58.2%, MAP-ADE 23.72 px.

  • Test split (512 families): regret 0.0484, Top-1 56.1%, MAP-ADE 22.66 px.

Planar pushing is equivariant to rigid motions of the scene, but a model that sees raw world coordinates from 512 families can fit the absolute initial poses (c0,θ0)(\mathbf{c}_0,\theta_0) instead of the pose-independent dynamics. We hypothesized that this explains the gap.

To test this, we wrap each decoder in a fixed change of frame. Let c0\mathbf{c}_0 be the centroid of the observed initial keypoints and R0\mathbf{R}_0 the orientation of the stem-to-bar axis. All 2D inputs p\mathbf{p} (keypoint history, goal keypoints, pusher positions and action endpoints) and all targets are mapped into the object frame before tokenization, and predictions are mapped back:

pOF=o+sR0T(p−c0),\mathbf{p}_{\mathrm{OF}} = \mathbf{o} + s\mathbf{R}_0^{\mathsf{T}}(\mathbf{p}-\mathbf{c}_0),
pworld=c0+s−1R0(pOF−o),(11)\mathbf{p}_{\mathrm{world}} = \mathbf{c}_0 + s^{-1}\mathbf{R}_0(\mathbf{p}_{\mathrm{OF}}-\mathbf{o}), \tag*{(11)}

with o=(256,256)T\mathbf{o}=(256,256)^{\mathsf{T}} and s=0.75s=0.75. With this scale, every object-frame coordinate in the dataset lies in [30,486][30,486], so no clipping occurs. Unless stated otherwise, object-frame models receive no RGB input. Table IV and Figure 5 report the results.

ManiSkill3, OF (5 seeds)Regret ↓\downarrowTop-1 % ↑\uparrowMAP-ADE ↓\downarrow (px)RB-Err ↓\downarrow (px)Brier ↓\downarrowGain ΔWF−OF\Delta_{\mathrm{WF-OF}} (Reg / ADE)pHolm(16)p_{\mathrm{Holm}(16)} (Reg / ADE)
Procrustes-FPD, E2E-L10.020 [0.016, 0.025]73.2 [69.8, 76.4]11.35 [10.95, 11.75]0.00 [0.00, 0.00]0.188 [0.173, 0.205]+0.027 / +9.4 pxReference
Procrustes-FPD, Pre-L10.022 [0.018, 0.027]71.3 [68.0, 74.6]11.42 [11.02, 11.81]0.00 [0.00, 0.00]0.202 [0.185, 0.219]+0.018 / +8.6 px0.465 / 0.0620
Soft-L1 head0.023 [0.019, 0.028]71.1 [67.9, 74.5]11.85 [11.46, 12.24]4.69 [4.57, 4.81]0.202 [0.185, 0.219]+0.019 / +9.0 px0.045 / 0.0080
SE(2) tokens, Soft-L10.023 [0.019, 0.028]72.5 [69.4, 75.6]11.84 [11.45, 12.22]0.00 [0.00, 0.00]0.189 [0.174, 0.206]+0.026 / +11.0 px0.128 / 0.0080
CE head + Kabsch0.021 [0.017, 0.026]70.2 [66.9, 73.6]11.70 [11.27, 12.12]0.00 [0.00, 0.00]0.214 [0.198, 0.231]+0.022 / +11.5 px1.000 / 0.0080
CE head (VFlash-style)0.021 [0.017, 0.026]70.5 [67.4, 73.8]12.38 [11.94, 12.82]5.40 [5.22, 5.56]0.214 [0.198, 0.231]+0.021 / +13.2 px1.000 / 0.0080
MDN head + Kabsch0.021 [0.017, 0.025]69.7 [66.4, 73.0]12.92 [12.47, 13.35]0.00 [0.00, 0.00]0.174 [0.160, 0.190]+0.025 / +11.6 px1.000 / 0.0080
MDN head (4 branches)0.021 [0.017, 0.026]69.5 [66.2, 72.9]13.09 [12.64, 13.52]2.63 [2.55, 2.70]0.174 [0.160, 0.190]+0.023 / +11.9 px1.000 / 0.0080
Bi-FPD + Kabsch0.025 [0.021, 0.030]66.7 [63.4, 70.1]14.28 [13.80, 14.75]0.00 [0.00, 0.00]0.230 [0.211, 0.250]+0.023 / +10.0 px0.128 / 0.0080
Bi-FPD (mode-routed MDN)0.025 [0.021, 0.030]67.0 [63.7, 70.3]14.40 [13.92, 14.86]2.19 [2.12, 2.25]0.230 [0.211, 0.250]+0.023 / +9.9 px0.168 / 0.0080
Autoregressive (KV cache)0.059 [0.052, 0.066]50.7 [48.1, 53.4]28.32 [27.58, 29.03]3.83 [3.67, 3.99]0.247 [0.229, 0.265]+0.017 / +8.9 px0.008 / 0.0080
Procrustes-FPD, E2E-L1 + warped RGB0.028 [0.024, 0.032]68.2 [65.0, 71.5]13.78 [13.31, 14.26]0.00 [0.00, 0.00]0.260 [0.238, 0.294]+0.020 / +6.9 px0.008 / 0.0080
Procrustes-FPD, Pre-L1 + warped RGB0.030 [0.025, 0.035]66.1 [62.8, 69.4]14.45 [13.97, 14.95]0.00 [0.00, 0.00]0.260 [0.241, 0.282]+0.010 / +5.5 px0.008 / 0.0080
Procrustes-FPD, Pre-L1 + unwarped RGB0.029 [0.024, 0.034]68.3 [65.2, 71.5]13.71 [13.24, 14.20]0.00 [0.00, 0.00]0.274 [0.253, 0.294]+0.011 / +6.3 px0.008 / 0.0080
WF control, no RGB: Procrustes-FPD, Pre-L10.043 [0.037, 0.050]59.1 [55.8, 62.8]19.31 [18.79, 19.84]0.00 [0.00, 0.00]0.320 [0.298, 0.344]−0.003 / +0.7 px0.008 / 0.0080
WF control, no RGB: Soft-L1 head0.042 [0.036, 0.049]60.4 [57.2, 63.7]20.52 [20.01, 21.04]14.6 [10.15, 10.77]0.320 [0.298, 0.344]−0.000 / +0.4 px0.008 / 0.0080
lerobot/pusht, OF (3 seeds)MAP-ADE Raw ↓\downarrowMAP-ADE Kabsch ↓\downarrowRB-Err Raw ↓\downarrowU-MAE Kabsch ↓\downarrowBrier ↓\downarrowGain ΔWF−OF\Delta_{\mathrm{WF-OF}} (Raw / Kab)pHolm(10)p_{\mathrm{Holm}(10)} (Raw / Kab)
SE(2) tokens, Soft-L13.68 [3.38, 4.00]3.68 [3.38, 4.00]0.00 [0.00, 0.00]0.021 [0.018, 0.025]0.220 [0.190, 0.250]+9.46 / +9.46 px0.0050 / 1.0000
Soft-L1 head3.92 [3.65, 4.21]3.71 [3.44, 4.00]1.43 [1.34, 1.51]0.022 [0.019, 0.025]0.205 [0.177, 0.233]+8.86 / +8.05 pxReference
Procrustes-FPD, E2E-L1 + rotated RGB3.72 [3.47, 3.98]3.72 [3.47, 3.98]0.00 [0.00, 0.00]0.022 [0.019, 0.025]0.220 [0.197, 0.244]+6.45 / +6.45 px0.0320 / 1.0000
Procrustes-FPD, E2E-L13.78 [3.52, 4.05]3.78 [3.52, 4.05]0.00 [0.00, 0.00]0.024 [0.021, 0.027]0.227 [0.196, 0.258]+6.39 / +6.39 px0.0050 / 0.3960
Soft-L1 head + rotated RGB5.70 [5.39, 6.02]5.47 [5.17, 5.79]1.66 [1.57, 1.75]0.031 [0.026, 0.037]0.272 [0.246, 0.295]+7.09 / +6.29 px0.0050 / 0.0500
CE head (VFlash-style)4.29 [3.95, 4.67]4.13 [3.79, 4.50]1.14 [1.02, 1.27]0.024 [0.020, 0.027]0.223 [0.194, 0.251]+12.24 / +10.25 px0.0050 / 0.0050
MDN head (4 branches)5.25 [4.86, 5.68]5.19 [4.80, 5.62]0.67 [0.61, 0.73]0.026 [0.022, 0.030]0.236 [0.201, 0.271]+11.10 / +10.42 px0.0050 / 0.0050
Bi-FPD (mode-routed MDN)5.32 [4.87, 5.79]5.27 [4.82, 5.73]0.59 [0.54, 0.64]0.026 [0.022, 0.029]0.228 [0.198, 0.259]+11.43 / +10.52 px0.0050 / 0.0050
Autoregressive (KV cache)5.58 [5.17, 6.03]5.43 [5.03, 5.87]1.05 [0.96, 1.15]0.031 [0.027, 0.035]0.233 [0.200, 0.266]+19.85 / +18.69 px0.0050 / 0.0050
WF control, no RGB: Soft-L1 head13.25 [12.42, 14.13]11.84 [11.04, 12.72]7.90 [7.43, 8.38]0.069 [0.060, 0.078]0.277 [0.242, 0.313]−0.46 / −0.08 px0.0050 / 0.0050

Table IV. Object-frame (OF) canonicalization. Each model is trained and evaluated in the frame of the initial object pose (c0,R0)(c_0, R_0) and its predictions are mapped back to the world frame. Unless marked, OF models receive no RGB input; “warped”, “rotated” and “unwarped” RGB variants are described in Section V-C. Gain: paired improvement over the same model in the world frame (Tables I and II). Top: ManiSkill3 (5 seeds); pHolm(16)p_{\mathrm{Holm}(16)} against OF Procrustes-FPD (E2E-L1). Bottom: lerobot/pusht (3 seeds); pHolm(10)p_{\mathrm{Holm}(10)} against the OF Soft-L1 head after Kabsch projection (Pre-L1). Holm families include the superseded OF diffusion arm (Table VII), which is not shown.

Figure 5. World frame (orange) against object frame (blue/green) for each decoder. (a) ManiSkill3 test regret (512 families, 5 seeds); the dashed line marks the best world-frame regret. (b) lerobot/pusht MAP-ADE after Kabsch projection (40 episodes, 3 seeds); the object frame lowers it by 63%–78%.

The object frame halves regret for every single-pass decoder. On ManiSkill3, E2E-L1 drops from 0.048 to 0.020 [0.016,0.025][0.016, 0.025] regret (Top-1 57.8% →\to 73.2%, MAP-ADE 20.7 →\to 11.35 px), Pre-L1 from 0.040 to 0.022 [0.018,0.027][0.018, 0.027] (MAP-ADE 11.42 px), the CE head with projection from 0.044 to 0.021 (MAP-ADE 23.2 →\to 11.70 px), and the MDN head with projection from 0.046 to 0.021 (MAP-ADE 24.6 →\to 12.92 px). On lerobot/pusht, the object frame lowers projected MAP-ADE by 63%–78% across decoders (Fig. 5b).

The gain comes from the frame, not from removing the image. Since the object-frame models receive no RGB, we trained world-frame controls without RGB as well. They reach 0.043 regret and 19.31 px MAP-ADE on ManiSkill3 (0.040 and 20.0 px with RGB) and 11.84 px on lerobot/pusht (11.76 px with RGB). Removing the image in the world frame does not help. Conversely, feeding the unwarped world-camera image to the object-frame model raises regret from 0.022 to 0.029 (Δ=+0.0066\Delta=+0.0066, unadjusted p=0.001p=0.001) and MAP-ADE from 11.42 to 13.71 px (p=0.0005p=0.0005). This is consistent with the image leaking the world pose back into the model. Rotating the image into the object frame with a differentiable warp removes most of this effect:

Figure 3. lerobot/pusht test split (40 episodes, 2,384 windows, 3 seeds, 95% episode-cluster bootstrap CIs): trajectory error of the raw decoder output (orange) and after 2D Kabsch projection (blue). The dashed line marks Procrustes-FPD (E2E-L1).

BenchmarkCompetitorRegret / Raw Δ\Delta95% CIpp (pHolmp_{\mathrm{Holm}})MAP-ADE / Kabsch Δ\Delta (px)95% CIpp (pHolmp_{\mathrm{Holm}})
ManiSkill3 (vs. Pre-L1)Soft-L1 head+0.0024[+0.0003, +0.0047]0.0250 (0.1250)+0.91[+0.86, +0.96]0.0005 (0.0065)
ManiSkill3 (vs. Pre-L1)Procrustes-FPD, E2E-L1+0.0076[+0.0040, +0.0114]0.0005 (0.0070)+0.74[+0.51, +0.98]0.0005 (0.0065)
ManiSkill3 (vs. Pre-L1)SE(2) tokens, Soft-L1+0.0094[+0.0054, +0.0138]0.0010 (0.0070)+2.91[+2.65, +3.16]0.0005 (0.0065)
ManiSkill3 (vs. Pre-L1)CE head (VFlash-style)+0.0024[−0.0020, +0.0064]0.2580 (0.5160)+5.63[+5.31, +5.97]0.0005 (0.0065)
ManiSkill3 (vs. Pre-L1)CE head + Kabsch+0.0037[+0.0000, +0.0073]0.0500 (0.1500)+3.26[+2.97, +3.57]0.0005 (0.0065)
ManiSkill3 (vs. Pre-L1)SE(2) tokens, CE+0.0087[+0.0043, +0.0130]0.0005 (0.0070)+5.55[+5.23, +5.88]0.0005 (0.0065)
ManiSkill3 (vs. Pre-L1)MDN head (4 branches)+0.0044[+0.0005, +0.0087]0.0280 (0.1250)+5.05[+4.72, +5.37]0.0005 (0.0065)
ManiSkill3 (vs. Pre-L1)MDN head + Kabsch+0.0061[+0.0021, +0.0103]0.0030 (0.0180)+4.59[+4.25, +4.92]0.0005 (0.0065)
ManiSkill3 (vs. Pre-L1)SE(2) tokens, MDN+0.0014[−0.0031, +0.0060]0.5310 (0.5310)+4.46[+4.09, +4.87]0.0005 (0.0065)
ManiSkill3 (vs. Pre-L1)Bi-FPD (mode-routed MDN)+0.0082[+0.0036, +0.0131]0.0005 (0.0070)+4.29[+3.96, +4.67]0.0005 (0.0065)
ManiSkill3 (vs. Pre-L1)Direct utility regression+0.0121[+0.0056, +0.0187]0.0005 (0.0070)+14.73[+14.32, +15.15]0.0005 (0.0065)
ManiSkill3 (vs. Pre-L1)Contact-mode classifier+0.0198[+0.0125, +0.0273]0.0005 (0.0070)———
ManiSkill3 (vs. Pre-L1)Autoregressive (KV cache)+0.0365[+0.0293, +0.0440]0.0005 (0.0070)+17.24[+16.62, +17.94]0.0005 (0.0065)
lerobot/pusht (vs. E2E-L1)Soft-L1 head+2.61 px[+2.37, +2.87]0.0005 (0.0045)+1.59 px[+1.35, +1.83]0.0005 (0.0045)
lerobot/pusht (vs. E2E-L1)SE(2) tokens, Soft-L1+2.97 px[+2.62, +3.32]0.0005 (0.0045)+2.97 px[+2.62, +3.32]0.0005 (0.0045)
lerobot/pusht (vs. E2E-L1)SE(2) tokens, MDN+3.74 px[+3.18, +4.29]0.0005 (0.0045)+3.74 px[+3.18, +4.29]0.0005 (0.0045)
lerobot/pusht (vs. E2E-L1)CE head (VFlash-style)+6.36 px[+5.81, +6.94]0.0005 (0.0045)+4.21 px[+3.77, +4.66]0.0005 (0.0045)
lerobot/pusht (vs. E2E-L1)MDN head (4 branches)+6.18 px[+5.53, +6.87]0.0005 (0.0045)+5.44 px[+4.81, +6.10]0.0005 (0.0045)
lerobot/pusht (vs. E2E-L1)Direct utility regression+6.23 px[+5.75, +6.69]0.0005 (0.0045)+4.94 px[+4.45, +5.42]0.0005 (0.0045)
lerobot/pusht (vs. E2E-L1)Bi-FPD (mode-routed MDN)+6.58 px[+5.87, +7.33]0.0005 (0.0045)+5.62 px[+4.96, +6.34]0.0005 (0.0045)
lerobot/pusht (vs. E2E-L1)Autoregressive (KV cache)+15.26 px[+13.80, +16.87]0.0005 (0.0045)+13.94 px[+12.55, +15.50]0.0005 (0.0045)

Table 3. PAIRED CLUSTER-BOOTSTRAP TESTS (BB=2000; TWO-SIDED pp, WITH HOLM-CORRECTED pHOLMp_{\mathrm{HOLM}} IN PARENTHESES). Δ\Delta IS COMPETITOR MINUS REFERENCE, SO Δ>0\Delta> 0 FAVORS THE REFERENCE. TOP: MANISKILL3, WORLD FRAME, AGAINST PROCRUSTES-FPD (PRE-L1), ON REGRET AND MAP-ADE. BOTTOM: LEROBOT/PUSHT, WORLD FRAME, AGAINST PROCRUSTES-FPD (E2E-L1), ON RAW AND KABSCH-PROJECTED MAP-ADE. EACH HOLM FAMILY CONTAINS ALL COMPETITORS SHOWN PLUS THE SUPERSEDED DIFFUSION ARM (NOT SHOWN; TABLE VII): 14 (REGRET) AND 13 (MAP-ADE) COMPARISONS ON MANISKILL3, 9 ON LEROBOT/PUSHT. REMOVING THE SUPERSEDED ARM COULD ONLY LOWER THE OTHER ADJUSTED pp-VALUES.

  • On the top-down view of lerobot/pusht, E2E-L1 with the rotated image reaches 3.72 px projected MAP-ADE. This matches the object-frame model without image (3.78 px, p=0.495p=0.495) and is 6.45 px lower than world-frame E2E-L1 (10.17 px, p=0.0005p=0.0005).

  • On the oblique (39°) camera of ManiSkill3, warping the image through the table-plane homography gives 0.028 regret and 13.78 px MAP-ADE. This is better than world-frame E2E-L1 (0.048, p=0.0005p=0.0005) but worse than the object-frame model without image (0.020, p=0.0005p=0.0005). A plausible explanation, which we have not tested, is that perspective and lighting in the oblique view still reveal part of the world pose.

These pp-values are unadjusted paired bootstrap tests of exploratory controls.

Projection still gives the lowest trajectory error in the object frame. On ManiSkill3, E2E-L1 (11.35 px) has lower MAP-ADE than the projected CE head (11.70 px), the projected MDN head (12.92 px), projected Bi-FPD (14.28 px) and the autoregressive model (28.32 px; all pHolm(16)=0.0080p_{\mathrm{Holm}(16)}=0.0080), and is tied with Pre-L1 (11.42 px, pHolm(16)=0.062p_{\mathrm{Holm}(16)}=0.062). On lerobot/pusht, Pre-L1 (3.71 px), E2E-L1 (3.78 px) and SE(2) tokens (3.68 px) are statistically tied. Pre-L1 is lower than the CE head (4.13 px), the MDN head (5.19 px), Bi-FPD (5.27 px) and the autoregressive model (5.43 px; all pHolm(10)=0.0050p_{\mathrm{Holm}(10)}=0.0050). Once the world rotation θ0\theta_0 is removed, SE(2) tokens lose most of their disadvantage (11.84 px on ManiSkill3), so their moment-arm penalty matters mainly when rotations are large. On regret, the single-pass heads in the object frame lie between 0.020 and 0.025, and most pairwise differences are not significant.

Figure 4. Direct SE(2) pose tokens (Δxt,Δyt,Δθt\Delta x_t, \Delta y_t, \Delta\theta_t) with 50 slots (purple) against 100 keypoint slots with Kabsch projection (blue), with identical backbones and losses, world frame. (a) MAP-ADE. (b) Regret on ManiSkill3.

Registered Frame × Decoder Grid, Diffusion Correction and Data Scaling

The comparisons above mix models from different stages, and the object-frame models differ from their world-frame counterparts in their RGB input. We therefore registered a 2×52 \times5 grid in which all decoders share one training path, no model receives RGB, and only the frame and the decoder vary (Table V, Figure 6). Its intervals come from a seed × family hierarchical bootstrap, which includes training-seed variation.

DecoderManiSkill3 WF RegretManiSkill3 OF RegretFrame Gain ΔWF−OF\Delta_{\mathrm{WF-OF}} (pHolm(9)p_{\mathrm{Holm}(9)})ManiSkill3 WF →\to OF ADE (px)vs. OF Procrustes-FPD: Reg / ADE pHolm(9)p_{\mathrm{Holm}(9)}PushT WF →\to OF ADE (px)Latency (ms)
Procrustes-FPD, E2E-L1 (single pass)0.0423 [0.0345, 0.0511]0.0203 [0.0157, 0.0253]+0.0220 (0.0045)19.1 →\to 11.3Reference12.00 →\to 3.781.62
CE head, VFlash-style (single pass)0.0450 [0.0367, 0.0544]0.0209 [0.0167, 0.0255]+0.0241 (0.0045)25.2 →\to 12.41.000 / 0.004513.66 →\to 4.131.76
MDN head, 4 branches (single pass)0.0425 [0.0352, 0.0507]0.0212 [0.0167, 0.0260]+0.0213 (0.0045)21.0 →\to 13.11.000 / 0.004514.74 →\to 5.191.68
Diffusion, DDIM-10, corrected (S=16S=16)0.0993 [0.0817, 0.1153]0.0460 [0.0353, 0.0584]+0.0532 (0.0045)59.7 →\to 27.80.005 / 0.004555.19 →\to 18.4510.32
Autoregressive, KV cache (S=16S=16)0.1110 [0.0968, 0.1272]0.0592 [0.0466, 0.0738]+0.0518 (0.0045)54.7 →\to 28.30.005 / 0.004524.12 →\to 5.4312.89
H2 (ManiSkill3)mean frame effect F=0.0345F=0.0345 regret (17.35 px ADE); five-decoder spread S=0.0389S=0.0389 regret (16.97 px ADE); F−S=−0.0044F-S=-0.0044 [−0.0226, +0.0074] regret, +0.37 [−5.67, +3.24] px ADE. Registered criterion (F−S>0F-S>0) not met.
H5: data scaling (Procrustes-FPD, E2E-L1; equal optimizer steps)H4: DDIM step sweep (corrected diffusion, OF)
Train Families NfamN_{\mathrm{fam}}WF Regret / ADEOF Regret / ADEFrame Gain Δ\Delta Reg / ADEDDIM StepsOF Regret ↓\downarrowOF MAP-ADE ↓\downarrow (px)Latency (ms)
N=64N=640.0714 / 40.9 px0.0538 / 22.5 px+0.0176 / +18.4 px4 steps0.0558 [0.0412, 0.0725]32.934.78
N=128N=1280.0663 / 32.6 px0.0435 / 19.4 px+0.0228 / +13.2 px10 steps0.0460 [0.0353, 0.0584]27.8410.32
N=256N=2560.0468 / 23.7 px0.0283 / 13.5 px+0.0185 / +10.3 px25 steps0.0419 [0.0324, 0.0534]25.9824.24
N=512N=5120.0423 / 19.1 px0.0203 / 11.3 px+0.0220 / +7.8 px50 steps0.0400 [0.0316, 0.0496]24.7747.02

Table V. Registered 2×52\times5 coordinate-frame ×\times decoder grid, DDIM step sweep and data-scaling study (seed ×\times family/episode hierarchical bootstrap, B=2000B=2000, 95% CIs). All ten grid models share one training path and receive no RGB input, so only the coordinate frame (WF: world frame; OF: object frame) and the decoder change. The diffusion model is the corrected version (§ 5-D). pHolm(9)p_{\mathrm{Holm}(9)}: Holm correction over the nine registered comparisons per metric. Latency: median over 200 calls for one decision (batch of 4 candidate actions, S=16S=16 samples where applicable), measured in the grid run.

Figure 6. Registered frame ×\times decoder grid and data scaling on ManiSkill3 (hierarchical seed ×\times family 95% CIs; no RGB input in either frame). (a) The object frame lowers regret by 1.9×\times–2.2×\times for all five decoders; in either frame, the iterative decoders (diffusion, autoregression) have 2.3×\times–2.9×\times the regret of the single-pass ones. (b) With equal optimizer steps, the object-frame model trained on 128 families matches the world-frame model trained on 512.

Diffusion baseline correction. While preparing the grid we found an error in our diffusion baseline. The model class defined a static method norm that shadowed the module’s final RMSNorm layer of the same name, so the original diffusion model ran without its final normalization. Its fixed target scaling also clipped object-frame targets at ±1.2\pm1.2. The corrected model restores the normalization and fits the target mean and 99.9th-percentile scale on the training split, in whichever frame the model is trained. The original numbers are superseded: we report them only in Appendix A (Table VII). They are excluded from the other tables and figures, but they remain in the Holm families, which is conservative. With the correction, the object frame lowers diffusion regret on ManiSkill3 by 2.16×2.16\times (0.0993 →\to 0.0460, pHolm(9)=0.0045p_{\mathrm{Holm}}(9)=0.0045), MAP-ADE by 2.15×2.15\times (59.73 →\to 27.84 px), and projected MAP-ADE on lerobot/pusht by 2.99×2.99\times (55.19 →\to 18.45 px).

H1 and H2: frame versus decoder. The object frame significantly lowers regret and MAP-ADE for all five decoders (Procrustes-FPD E2E-L1, CE head, MDN head, diffusion and autoregression; pHolm(9)<0.005p_{\mathrm{Holm}}(9)<0.005), so H1 is supported. H2 compared the mean frame effect FF (world-frame minus object-frame regret, averaged over decoders) with the decoder spread SS (worst minus best decoder in the object frame).

compared the mean frame effect FF (world-frame minus object-frame regret, averaged over decoders) with the decoder spread SS (worst minus best decoder in the object frame). For regret, F=0.0345F = 0.0345 [0.0283, 0.0415] and S=0.0389S = 0.0389 [0.0294, 0.0551], so F−S=−0.0044F-S = -0.0044 [−0.0226, +0.0074]. For MAP-ADE on ManiSkill3, F=17.35F = 17.35 and S=16.97S = 16.97 px, so F−S=+0.37F-S = +0.37 [−5.67, +3.24] px; on lerobot/pusht the values are 16.55 and 14.67 px. The registered criterion, a lower confidence bound of F−SF-S above zero, is not met: the two effects are statistically comparable. The spread comes from the iterative decoders, whose object-frame regret is 2.3×2.3\times (diffusion, 0.0460) and 2.9×2.9\times (autoregression, 0.0592) that of Procrustes-FPD. In an analysis that was not registered, the three single-pass heads are statistically indistinguishable on regret (0.0203, 0.0209, 0.0212; each against Procrustes-FPD, p>0.69p>0.69), while the frame changes each of them by 0.021–0.024 (frame effect minus single-pass spread: 0.0215 [0.0130, 0.0271]). We read the grid as two separable effects of similar size: the coordinate frame (about 2×2\times for every decoder) and single-pass versus iterative decoding (about 2×2\times–3×3\times). The choice of single-pass head is secondary.

H3 and H4: single-pass decoding against a DDIM step sweep. In the object frame, Procrustes-FPD (E2E-L1) reaches 0.0203 regret and 11.35 px MAP-ADE at 1.62 ms per decision. It ties the CE head (0.0209) and the MDN head (0.0212) on regret (pHolm(9)=1.0p_{\mathrm{Holm}}(9)=1.0) and has lower MAP-ADE than all four other decoders (pHolm(9)<0.005p_{\mathrm{Holm}}(9)<0.005). We ran the corrected object-frame diffusion model with 1, 2, 4, 10, 25 and 50 DDIM steps (Table V, bottom right). With 1 or 2 steps, sampling from this ϵ\epsilon-prediction model fails (0.1315 regret, MAP-ADE above 330 px), as expected when a single update must move from almost pure noise to the data. From 4 steps on, diffusion reaches 0.0558 regret at 4.78 ms, 0.0460 at 10.32 ms, and at best 0.0400 at 47.02 ms. Procrustes-FPD is both more accurate and faster than every setting, with 3×3\times–29×29\times lower latency than the 4- to 50-step settings, so H4 is supported. We did not test diffusion models trained or distilled for one or two steps, such as consistency models.

H5: data scaling. We trained Procrustes-FPD (E2E-L1) on nested subsets of Nfam∈{64,128,256,512}N_{\mathrm{fam}} \in\{64, 128, 256, 512\} training families with the same number of optimizer steps (Table V, bottom left; Figure 6b). The object-frame model trained on 128 families (0.0435 regret, 19.37 px) matches the world-frame model trained on 512 (0.0423 regret, 19.11 px), roughly a 4×4\times gain in sample efficiency. The gap between frames shrinks with data on MAP-ADE (18.39, 13.20, 10.26, 7.76 px) but not on regret (+0.0176, +0.0228, +0.0185, +0.0220). H5 is therefore only partly supported: the object frame improves data efficiency, and at 512 families its regret advantage has not yet narrowed.

Limitations

  • Scope. We study one task family, planar pushing of a single rigid object, with 2M- to 4M-parameter models. Whether the frame effect carries over to several objects, articulated or deformable objects, or full 3D pose is untested, as is its size for large pretrained backbones.

  • Simulation only. Both benchmarks are simulated. lerobot/pusht contains human demonstrations, but they were recorded in a 2D simulator. We do not evaluate on a physical robot, and we evaluate the world model by decision regret and trajectory error, not by closed-loop task success.

  • Reuse of test data. The ManiSkill3 test split was sealed for the pre-registered benchmark, but the exploratory controls and the registered grid were evaluated on the same test families after it was unsealed. The object-frame result was found in exploratory runs and then replicated in the registered grid. It was not predicted before the test data were seen.

  • Run-to-run variation. Identical retraining moved regret by up to 0.0035. Tables I–IV use family-cluster intervals that do not include this variation; only the grid (Table V) resamples seeds.

  • Baselines. Our iterative baselines are small models trained from scratch in our code base, and the diffusion baseline needed a correction. The corrected diffusion model inherits the hyperparameters selected for the original one, without re-tuning. Re-tuned, stronger or distilled diffusion models may narrow the gap. The VFlash-style head reimplements VFlash’s loss and decoding in our backbone; it is not the released VFlash model.

Conclusion

We studied single-pass slot decoders as System-1 world models, i.e. fast action-conditioned predictors, using a pre-registered benchmark, matched exploratory controls and a registered 68-run frame ×\times decoder and data-scaling grid on ManiSkill3 PushT-v1 and lerobot/push. Two factors largely determine how well such a predictor ranks candidate actions. The first is the coordinate frame: expressing inputs and outputs in the SE(2) frame of the object’s initial pose lowers test regret by about 2×2\times for every decoder we tested and gives roughly 4×4\times sample efficiency. The second is single-pass versus iterative decoding: in the object frame, single-pass heads have 2.3×\times–2.9×\times lower regret than corrected DDIM diffusion and KV-cached autoregression and are several times faster, while the choice among single-pass heads has no measurable effect on regret. Our registered hypothesis that the frame effect exceeds the decoder spread was not supported; the two are of comparable size. Soft-token expectation with an end-to-end 2D Kabsch projection gives the lowest trajectory error, rigid-body consistency by construction, and lower regret than diffusion at every budget from 4 to 50 DDIM steps. For practitioners, the recommendation is to choose the coordinate frame first and then decode in a single pass. Open questions are whether the frame effect holds for tasks with richer symmetry, such as several objects or full 3D pose, and how diffusion models distilled for one or two steps compare.

Acknowledgment

AI assistants were used to help write and debug the experiment code, to run and audit the experiments, and to draft and edit the text. The author designed the study, checked the results and the manuscript, and takes full responsibility for the content.

Availability. Code, frozen configurations, per-seed raw outputs and the script that generates every table and figure in this paper will be released publicly.

Superseded Diffusion Baseline

Table VII lists the results of the original diffusion baseline, whose final normalization layer was disabled (§ 5-D), next to the corrected model. The two sets of rows use different protocols: the superseded world-frame rows had RGB input, the corrected rows come from the grid without RGB, and their intervals are computed differently. They should therefore not be compared row by row. The correction mainly changed the object-frame behaviour. The superseded model became worse in the object frame (regret 0.084 →\rightarrow 0.091), whereas the corrected model improves by 2.16×\times, in line with the other decoders.

ModelSettingRegret ↓\downarrowMAP-ADE ↓\downarrow (px)
SupersededManiSkill3, WF0.084 [0.075, 0.095]29.9 [29.4, 30.4]
SupersededManiSkill3, OF0.091 [0.082, 0.102]38.6 [38.2, 39.0]
Supersededlerobot/pusht, WF—14.78 [14.16, 15.43]
Supersededlerobot/pusht, OF—18.26 [17.86, 18.68]
CorrectedManiSkill3, WF0.0993 [0.0817, 0.1153]59.7
CorrectedManiSkill3, OF0.0460 [0.0353, 0.0584]27.8
Correctedlerobot/pusht, WF—55.19
Correctedlerobot/pusht, OF—18.45

Table VII. Superseded diffusion baseline (final RMSNorm disabled by a bug) and the corrected model. Superseded rows follow the protocols of Tables I, II and IV (world-frame rows with RGB input); corrected rows are from the registered grid (no RGB input, hierarchical bootstrap; Table V). lerobot/pusht MAP-ADE is after Kabsch projection. 10 DDIM steps, S=16S=16.

NameFrameIdentifierStatusTables
Procrustes-FPD, Pre-L1WFA3_VFlashSoftL1_KabschExpl.I–III
Soft-L1 headWFA3_VFlashSoftL1Pre-reg.I–III
Procrustes-FPD, E2E-L1WFA3_VFlashSoftL1_E2EKabschExpl.I–III
SE(2) tokens, Soft-L1WFSE2_VFlashSoftL1Expl.I–III
CE head (VFlash-style)WFA2_VFlashPureCEPre-reg.I–III
CE head + KabschWFA2_VFlashPureCE_KabschExpl.I–III
SE(2) tokens, CEWFSE2_VFlashPureCEExpl.I–III
MDN head (4 branches)WFA6_GenericMDNPre-reg.I–III
MDN head + KabschWFA6_GenericMDN_KabschExpl.I–III
SE(2) tokens, MDNWFSE2_GenericMDNExpl.I–III
Bi-FPD (mode-routed MDN)WFA12_BiFPDPre-reg.I–III
Direct utility regressionWFA1_DirectScorePre-reg.I–III
Contact-mode classifierWFA8_ClassifierOnlyTunedPre-reg.I–III
Autoregressive (KV cache)WFA4_ARSeqSampledPre-reg.I–III
Diffusion, DDIM-10 (superseded)WFA5_DiffusionTrajPre-reg.I–III, VII
Procrustes-FPD, E2E-L1OFBF_A3_VFlashSoftL1_E2EKabschExpl.IV
Procrustes-FPD, Pre-L1OFBF_Equiv_ProcrustesFPDExpl.IV
Soft-L1 headOFBF_Equiv_RawSoftL1Expl.IV
SE(2) tokens, Soft-L1OFBF_SE2_VFlashSoftL1Expl.IV
CE head + KabschOFBF_A2_VFlashPureCE_KabschExpl.IV
CE head (VFlash-style)OFBF_A2_VFlashPureCEExpl.IV
MDN head + KabschOFBF_A6_GenericMDN_KabschExpl.IV
MDN head (4 branches)OFBF_A6_GenericMDNExpl.IV
Bi-FPD + KabschOFBF_A12_BiFPD_KabschExpl.IV
Bi-FPD (mode-routed MDN)OFBF_A12_BiFPDExpl.IV
Autoregressive (KV cache)OFBF_A4_ARSeqSampledExpl.IV
Diffusion, DDIM-10 (superseded)OFBF_A5_DiffusionTrajExpl.IV, VII
Procrustes-FPD, E2E-L1 + warped RGBOFBF_RotRGB_E2EKabschExpl.IV
Procrustes-FPD, Pre-L1 + warped RGBOFBF_RotRGB_ProcrustesFPDExpl.IV
Procrustes-FPD, Pre-L1 + unwarped RGBOFBF_WithRGB_ProcrustesFPDExpl.IV
WF control, no RGB: Procrustes-FPD, Pre-L1WFWF_NoRGB_ProcrustesFPDExpl.IV
WF control, no RGB: Soft-L1 headWFWF_NoRGB_RawSoftL1Expl.IV
Soft-L1 headOFBF_A3_VFlashSoftL1Expl.IV
Procrustes-FPD, E2E-L1 + rotated RGBOFBF_OrthoRGB_A3_VFlashSoftL1_E2EKabschExpl.IV
Soft-L1 head + rotated RGBOFBF_OrthoRGB_A3_VFlashSoftL1Expl.IV
Diffusion, DDIM-10 (corrected)WF / OFA5fix_DiffusionTrajGridV, VII

Table VI. Model names used in this paper and the corresponding identifiers in the released code and result files. Pre-reg.: pre-registered benchmark arm; Expl.: exploratory control; Grid: registered frame ×\times decoder grid, where every decoder is run as WF_NoRGB_⟨ID⟩ and BF_⟨ID⟩.

Appendix B: Model Names and Code Identifiers

Table VI maps the model names used in the text to the identifiers used in the released code and result files.

@page 11/12 starts_mid=0 ends_mid=1

NameFrameIdentifierStatusTables
Procrustes-FPD, Pre-L1WFA3_VFlashSoftL1_KabschExpl.I–III
Soft-L1 headWFA3_VFlashSoftL1Pre-reg.I–III
Procrustes-FPD, E2E-L1WFA3_VFlashSoftL1_E2EKabschExpl.I–III
SE(2) tokens, Soft-L1WFSE2_VFlashSoftL1Expl.I–III
CE head (VFlash-style)WFA2_VFlashPureCEPre-reg.I–III
CE head + KabschWFA2_VFlashPureCE_KabschExpl.I–III
SE(2) tokens, CEWFSE2_VFlashPureCEExpl.I–III
MDN head (4 branches)WFA6_GenericMDNPre-reg.I–III
MDN head + KabschWFA6_GenericMDN_KabschExpl.I–III
SE(2) tokens, MDNWFSE2_GenericMDNExpl.I–III
Bi-FPD (mode-routed MDN)WFA12_BiFPDPre-reg.I–III
Direct utility regressionWFA1_DirectScorePre-reg.I–III
Contact-mode classifierWFA8_ClassifierOnlyTunedPre-reg.I–III
Autoregressive (KV cache)WFA4_ARSeqSampledPre-reg.I–III
Diffusion, DDIM-10 (superseded)WFA5_DiffusionTrajPre-reg.I–III, VII
Procrustes-FPD, E2E-L1OFBF_A3_VFlashSoftL1_E2EKabschExpl.IV
Procrustes-FPD, Pre-L1OFBF_Equiv_ProcrustesFPDExpl.IV
Soft-L1 headOFBF_Equiv_RawSoftL1Expl.IV
SE(2) tokens, Soft-L1OFBF_SE2_VFlashSoftL1Expl.IV
CE head + KabschOFBF_A2_VFlashPureCE_KabschExpl.IV
CE head (VFlash-style)OFBF_A2_VFlashPureCEExpl.IV
MDN head + KabschOFBF_A6_GenericMDN_KabschExpl.IV
MDN head (4 branches)OFBF_A6_GenericMDNExpl.IV
Bi-FPD + KabschOFBF_A12_BiFPD_KabschExpl.IV
Bi-FPD (mode-routed MDN)OFBF_A12_BiFPDExpl.IV
Autoregressive (KV cache)OFBF_A4_ARSeqSampledExpl.IV
Diffusion, DDIM-10 (superseded)OFBF_A5_DiffusionTrajExpl.IV, VII
Procrustes-FPD, E2E-L1 + warped RGBOFBF_RotRGB_E2EKabschExpl.IV
Procrustes-FPD, Pre-L1 + warped RGBOFBF_RotRGB_ProcrustesFPDExpl.IV
Procrustes-FPD, Pre-L1 + unwarped RGBOFBF_WithRGB_ProcrustesFPDExpl.IV
WF control, no RGB: Procrustes-FPD, Pre-L1WFWF_NoRGB_ProcrustesFPDExpl.IV
WF control, no RGB: Soft-L1 headWFWF_NoRGB_RawSoftL1Expl.IV
Soft-L1 headOFBF_A3_VFlashSoftL1Expl.IV
Procrustes-FPD, E2E-L1 + rotated RGBOFBF_OrthoRGB_A3_VFlashSoftL1_E2EKabschExpl.IV
Soft-L1 head + rotated RGBOFBF_OrthoRGB_A3_VFlashSoftL1Expl.IV
Diffusion, DDIM-10 (corrected)WF / OFA5fix_DiffusionTrajGridV, VII

Table VI (PDF p. 11). Model names used in this paper and the corresponding identifiers in the released code and result files. Pre-reg.: pre-registered benchmark arm; Expl.: exploratory control; Grid: registered frame ×\times decoder grid, where every decoder is run as WF_NoRGB_⟨ID⟩ and BF_⟨ID⟩.

ModelSettingRegret ↓\downarrowMAP-ADE ↓\downarrow (px)
SupersededManiSkill3, WF0.084 [0.075, 0.095]29.9 [29.4, 30.4]
SupersededManiSkill3, OF0.091 [0.082, 0.102]38.6 [38.2, 39.0]
Supersededlerobot/pusht, WF—14.78 [14.16, 15.43]
Supersededlerobot/pusht, OF—18.26 [17.86, 18.68]
CorrectedManiSkill3, WF0.0993 [0.0817, 0.1153]59.7
CorrectedManiSkill3, OF0.0460 [0.0353, 0.0584]27.8
Correctedlerobot/pusht, WF—55.19
Correctedlerobot/pusht, OF—18.45

Table VII (PDF p. 11). Superseded diffusion baseline (final RMSNorm disabled by a bug) and the corrected model. Superseded rows follow the protocols of Tables I, II and IV (world-frame rows with RGB input); corrected rows are from the registered grid (no RGB input, hierarchical bootstrap; Table V). lerobot/pusht MAP-ADE is after Kabsch projection. 10 DDIM steps, S=16S=16.

the object-frame behaviour. The superseded model became worse in the object frame (regret 0.084 →\rightarrow 0.091), whereas the corrected model improves by 2.16×\times, in line with the other decoders.

Appendix B: Model Names and Code Identifiers

Table VI maps the model names used in the text to the identifiers used in the released code and result files.

References

  1. [1]Y.-C. Chou, S. Wang, J. Huang, C.-P. Huang, C.-A. Yang, X. Fan, J. Wang, Z. Li, Y. Kuang, R. Zheng, J. Bjorck, S. Liu, H. Su, A. Badki, L. Zhang, A. Juille, L. Fan, J. Kautz, and Z. Yu, “VFlash: Fast, accurate visual planning via flash primitive decoding,” arXiv preprint arXiv:2611.08449, 2026.arxiv.org/abs/2611.08449
  2. [2]S. Wang, S. Liu, Y. Kuang, X. Wei, Y. Liu, Z. Li, Y. Man, G. Chen, A. Tao, G. Liu, J. Kautz, L. Zhang, and Z. Yu, “LocateAnything: Fast and high-quality vision-language grounding with parallel box decoding,” arXiv preprint arXiv:2605.27365, 2026.arxiv.org/abs/2605.27365
  3. [3]C. Chi et al., “Diffusion policy: Visuomotor policy learning via action diffusion,” in Proc. Robotics: Science and Systems (RSS), 2023.
  4. [4]S. Tao et al., “ManiSkill3: GPU parallelized robotics simulation and rendering for generalized embodied AI,” arXiv preprint arXiv:2410.00425, 2024.arxiv.org/abs/2410.00425
  5. [5]R. Cadene et al., “LeRobot: State-of-the-art machine learning for real-world robotics in PyTorch,” GitHub repository, https://github.com/huggingface/lerobot, 2024.
  6. [6]X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei, “Integral human pose regression,” in Proc. ECCV, 2018, pp. 529–545.arxiv.org/abs/1711.08229
  7. [7]A. Nibali, Z. He, S. Morgan, and L. Prendergast, “Numerical coordinate regression with convolutional neural networks,” arXiv preprint arXiv:1801.07372, 2018.arxiv.org/abs/1801.07372
  8. [8]X. Li et al., “Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection,” in Proc. NeurIPS, 2020, pp. 21002–21012.arxiv.org/abs/2006.04388
  9. [9]S. Peng, Y. Liu, Q. Huang, X. Zhou, and H. Bao, “PVNet: Pixel-wise voting network for 6DoF pose estimation,” in Proc. IEEE/CVF CVPR, 2019, pp. 4561–4570.arxiv.org/abs/1812.11788
  10. [10]Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, “On the continuity of rotation representations in neural networks,” in Proc. IEEE/CVF CVPR, 2019, pp. 5745–5753.arxiv.org/abs/1812.07035
  11. [11]K. Kabsch, “A solution for the best rotation to relate two sets of vectors,” Acta Crystallographica Section A, vol. 32, no. 5, pp. 922–923, 1976.
  12. [12]P. H. Schönemann, “A generalized solution of the orthogonal Procrustes problem,” Psychometrika, vol. 31, no. 1, pp. 1–10, 1966.DOI
  13. [13]R. Brégier, “Deep regression on manifolds: A 3D rotation case study,” in Proc. Int. Conf. 3D Vision (3DV), 2021, pp. 166–174.arxiv.org/abs/2103.16317
  14. [14]C. Choy, W. Dong, and V. Koltun, “Deep global registration,” in Proc. IEEE/CVF CVPR, 2020, pp. 2514–2523.arxiv.org/abs/2004.11540
  15. [15]S. Bai et al., “Qwen2.5-VL technical report,” arXiv preprint arXiv:2502.13923, 2025.arxiv.org/abs/2502.13923
  16. [17]M. J. Kim et al., “OpenVLA: An open-source vision-language-action model,” in Proc. CoRL, 2024.arxiv.org/abs/2406.09246
  17. [18]D. Ha and J. Schmidhuber, “World models,” arXiv preprint arXiv:1803.10122, 2018.DOI
  18. [19]D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse domains through world models,” arXiv preprint arXiv:2301.04104, 2023.arxiv.org/abs/2301.04104
  19. [20]D. Kahneman, Thinking, Fast and Slow. New York, NY, USA: Farrar, Straus and Giroux, 2011.
  20. [21]J. Gao et al., “VectorNet: Encoding HD maps and agent dynamics from vectorized representation,” in Proc. IEEE/CVF CVPR, 2020, pp. 11525–11533.arxiv.org/abs/2005.04259
  21. [22]S. Shi, L. Jiang, D. Dai, and B. Schiele, “Motion transformer with global intention localization and local movement refinement,” in Proc. NeurIPS, 2022.arxiv.org/abs/2209.13508
  22. [23]D. Wang et al., “Equivariant diffusion policy,” in Proc. CoRL, 2024.arxiv.org/abs/2407.01812
  23. [24]J. Yang et al., “EquiBot: SIM(3)-equivariant diffusion policy for generalizable and data-efficient learning,” in Proc. CoRL, 2024.arxiv.org/abs/2407.01479

Paper details

vlnc:W_xrq9rzt4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

AI assistants

View statement
Looking for related papers…
Contents