betaFeedback

WorksComputer Science

HINT-WAM: Hindsight-Informed Training for World Action Models

Abstract

World action models (WAMs) have shown promising generalization and data efficiency in robot manipulation by coupling action generation with predictions of future visual dynamics. Translating these capabilities into reliable task execution, however, remains a challenge for existing post-training methods. Offline supervised fine-tuning provides dense supervision on expert trajectories, but its flow-matching action expert is trained only on noised demonstration actions and never on the intermediate states produced by its own sampler; task-specific adaptation can also compromise pretrained capabilities. On-policy reinforcement learning trains on the policy's own behavior, yet sparse task rewards make exploration and credit assignment difficult, while dense reward shaping still provides scalar evaluations rather than direct action targets. We introduce HINT-WAM, the first on-policy self-distillation framework that supervises the action expert on its own sampling trajectories using hindsight, the future video of the demonstration, as privileged information available to the WAM itself. For each demonstration observation, the student generates its action through its own denoising trajectory, and at the states along this trajectory the same model acts as a teacher conditioned additionally on this hindsight, providing a conditional action distribution as a dense training target, while the student sees only deployment-time inputs and needs no environmental rewards or simulator interaction. To transfer this supervision to a flow-matching action expert, HINT-WAM distills the teacher into the student with an energy score, a distribution-level objective that matches the teacher's action distribution from the student's own samples rather than regressing its velocity field. The project web is available in https://tuandattt.github.io/hintwam/.

Figure 1. HINT-WAM post-trains world action models with hindsight. Left: rollouts on two LIBERO-90 tasks from the same initial state, where SFT fails and HINT-WAM succeeds. Right: success rates (%) of HINT-WAM, the BC control, and full SFT on the new LIBERO-90 tasks, RoboTwin 2.0, and the four original LIBERO suites after post-training on LIBERO-90 without replay.

Introduction

World action models (WAMs) have demonstrated promising generalization and data efficiency for robot manipulation by coupling action generation with future visual prediction [[17, 21, 49]]. Recent models generate continuous action chunks through flow matching, providing an expressive interface between visual predictions and robot motion [[2, 21]]. Despite this progress, reliable execution on downstream tasks [[22, 23]] still requires adaptation to task-specific demands and to the deviations that arise during closed-loop interaction.

Supervised fine-tuning (SFT) offers a straightforward adaptation strategy by providing dense action targets from expert demonstrations. However, SFT supervises the model only on demonstration data. During training, the action expert sees only noised versions of the ground-truth actions and never the intermediate states produced by its own sampler, a form of exposure bias [33] and the sampler-level counterpart of the compounding errors at deployment [38], whose environment-level gap requires closed-loop interaction. Moreover, given only the current observation, a demonstrated action chunk is one of several plausible continuations, and the SFT target does not reveal which one the demonstrator followed. Finally, adapting a pretrained model to a narrow task distribution can degrade previously acquired capabilities [40].

On-policy reinforcement learning (RL) addresses the state-distribution mismatch by optimizing the policy on its own rollouts, motivating its use in VLA and WAM post-training [20, 32, 36]. However, many manipulation tasks offer only sparse success rewards, making exploration and credit assignment costly when successful trajectories are rarely sampled. Even with denser, reconstruction-based rewards [36], reward-based optimization only evaluates sampled behavior through scalar feedback, leaving the policy to infer how its actions should change. This motivates supplying structured action supervision directly on the states produced by the policy’s own action sampler.

Our key insight is that the future frames of a demonstration, which we call hindsight, carry privileged information about the intended task progression, connecting learning with privileged information [8, 43] to on-policy distillation [1]. We build on this insight in HINT-WAM: the same model serves as a teacher that additionally reads the hindsight, whereas the student observes only the inputs available at deployment and is supervised on the states produced by its own action sampler. Throughout this paper, on-policy refers to the generation of actions: as in on-policy distillation for language models [1], where prompts are drawn from a dataset and the student generates the response, observations are drawn from demonstrations and the student generates the denoising trajectory of its action chunk. We do not collect new environment states.

Transferring this supervision to a flow-matching action expert introduces a further challenge. Such experts generate continuous actions by integrating a velocity field, so the token-level probability matching used in autoregressive distillation does not apply directly. Regressing the teacher’s velocity field would drive the student toward the teacher’s conditional mean, which averages over modes that the student cannot distinguish without hindsight. We therefore distill the teacher with the energy score [14], a strictly proper scoring rule that matches the teacher’s action distribution from the student’s own samples and one teacher sample. Our contributions are summarized as follows:

  • We introduce HINT-WAM, a self-distillation framework for world action models in which the policy acts as its own teacher given hindsight, requiring neither a separately trained teacher nor environmental rewards or simulator interaction.

  • We propose a distribution-level distillation objective for flow-matching action experts based on the energy score, which avoids the mode averaging of velocity regression and requires no action-likelihood evaluation. The framework adapts the action expert while keeping the world-modeling backbone frozen.

  • Comprehensive experiments on LIBERO-90 and RoboTwin 2.0 demonstrate that HINT-WAM consistently improves post-training performance over SFT and matched baselines, while mitigating the catastrophic forgetting of pretrained skills.

Related Work

World Action Models

Vision-language-action (VLA) models combine large-scale multimodal pretraining with diverse robot demonstrations to translate camera observations and language instructions into robot commands [6, 18, 34, 50]. Rather than discretizing actions into tokens, many recent policies generate continuous action chunks with diffusion [10] or flow matching [5, 27]. World action models (WAMs) additionally reason about how the scene will evolve [39], and differ mainly in how tightly they tie prediction to control. Some extract predictive visual representations from a video model to condition a separate action head [17], while others generate future frames and actions jointly through a shared denoising process or a unified video–action backbone [16, 25, 49]. Autoregressive designs such as LingBot-VA interleave video and action prediction for closed-loop control [21], and RoboDreamer exploits imagined dynamics for planning [48]. Recent architectures pair a large world-modeling backbone with a lightweight flow-matching action expert [2]. While these works study WAM design and pretraining, HINT-WAM post-trains only the action expert on the states the policy visits.

On-Policy Self-Distillation

On-policy distillation builds on the observation, first formalized by DAgger [38], that a policy learns most effectively from the states it reaches on its own rather than from those recorded by a demonstrator. GKD [1] and MiniLLM [15] adopt this principle for compressing large language models: the student generates its own sequences, which the teacher then scores token by token, and MiniLLM additionally argues for the mode-seeking reverse KL as the distillation objective. A more recent line of work removes the need for a separate, larger teacher by letting a single model play both roles, where the teacher is the model itself given privileged information that the student does not see. Self-Distilled Reasoner [47] trains a reasoning model on its own on-policy samples under the supervision of a copy conditioned on reference solutions, dOPSD [11] extends this self-distillation scheme to diffusion language models, and PRISM [12] applies it to the design of communication topologies in multi-agent systems, guiding the designer with a privileged sibling. All of these methods operate on discrete tokens and rely on likelihood-based objectives. HINT-WAM brings on-policy self-distillation to embodied control, where the student is a flow-matching action expert that generates continuous actions without a tractable likelihood. Unlike prior flow-matching distillation methods, which regress the teacher’s velocity field [13, 24], we match action distributions with a proper scoring rule, the energy score [14], which needs only one teacher sample per state.

On-Policy Distillation for World Action Models

Recent works bring on-policy distillation to WAMs, supervising the student on its own rollouts with a separate, stronger task-specific teacher, either through joint video–action targets [44] or by replaying a fixed pool of student and teacher trajectories [29]. The former queries the teacher for paired video and action targets on student rollout histories, while the latter reweights each replayed decision with prefix-based importance weights to correct the shift between stored and current histories. These methods require an additional teacher model, making post-training computationally expensive. In contrast, HINT-WAM derives the teacher from the student itself by adding only a lightweight hindsight branch. To the best of our knowledge, it is the first self-distillation framework for world action models.

Method

This section presents HINT-WAM, which post-trains a pretrained WAM by letting the model act as its own teacher given hindsight. We first review the base model (§ 3.1) and outline the framework (§ 3.3), then describe the hindsight teacher (§ 3.4) and its distillation into the student (§ 3.5).

Preliminaries

Base world action model. We build on Fast-WAM [45], a world action model whose backbone is a Mixture-of-Transformers [26] with two experts that share a single attention operation in every layer. The video expert is a large pretrained video diffusion transformer [42] operating on video VAE latents, and the action expert is a lightweight diffusion transformer that generates an action chunk a∈RH×D\mathbf{a} \in\mathbb{R}^{H \times D} of HH control steps. The policy is conditioned on an observation o=(I,ℓ,s)\mathbf{o} = (I, \ell, s) consisting of the current camera frame II, the language instruction ℓ\ell, and the proprioceptive state ss. Action tokens attend only to the video tokens of the current frame, so the action expert never observes future frames.

Flow-matching action expert

Following the rectified-flow convention [27, 30], σ=0\sigma= 0 corresponds to data and σ=1\sigma= 1 to noise. An action chunk is aσ=(1−σ)e+σϵ\mathbf{a}_{\sigma} = (1-\sigma)\mathbf{e} + \sigma\boldsymbol{\epsilon} with ϵ∼N(0,I)\boldsymbol{\epsilon} \sim\mathcal{N}(0,I), and the action expert predicts a velocity vθ(aσ,σ,o)\mathbf{v}_{\theta}(\mathbf{a}_{\sigma},\sigma,\mathbf{o}) trained with

LFM(θ)=E(o,a),σ,ϵ[∥vθ(aσ,σ,o)−(ϵ−a)∥22].(1)\mathcal{L}_{\mathrm{FM}}(\theta) = \mathbb{E}_{(\mathbf{o},\mathbf{a}),\sigma,\boldsymbol{\epsilon}}\left[\left\|\mathbf{v}_{\theta}(\mathbf{a}_{\sigma},\sigma,\mathbf{o})-(\boldsymbol{\epsilon}-\mathbf{a})\right\|_{2}^{2}\right]. \tag*{(1)}

At inference, the policy integrates the probability-flow ODE from a1=ϵ\mathbf{a}_1 = \boldsymbol{\epsilon} to σ=0\sigma= 0 with NN Euler steps on a decreasing schedule 1=σ1>⋯>σN>σN+1=01 = \sigma_1 > \cdots> \sigma_N > \sigma_{N+1} = 0:

aσi+1=aσi+Δivθ(aσi,σi,o),Δi=σi+1−σi<0.(2)\mathbf{a}_{\sigma_{i+1}} = \mathbf{a}_{\sigma_i} + \Delta_i\mathbf{v}_{\theta}(\mathbf{a}_{\sigma_i},\sigma_i,\mathbf{o}), \qquad\Delta_i = \sigma_{i+1}-\sigma_i < 0. \tag*{(2)}

Since the video expert only processes the current frame, its keys and values are computed once per observation and cached, so every ODE step evaluates only the action expert. Finally, every point on a sampling trajectory yields an estimate of the clean action in a single forward pass,

a^θ(aσ)=aσ−σvθ(aσ,σ,o),(3)\hat{\mathbf{a}}_{\theta}(\mathbf{a}_{\sigma}) = \mathbf{a}_{\sigma} - \sigma\mathbf{v}_{\theta}(\mathbf{a}_{\sigma},\sigma,\mathbf{o}), \tag*{(3)}

which we call the implied action. For the optimal field it is the conditional mean E[a∣aσ]\mathbb{E}[\mathbf{a} \mid\mathbf{a}_{\sigma}] rather than the ODE endpoint, but at σN\sigma_N it coincides with the final Euler step, i.e., with the policy’s output.

Overview

Standard SFT minimizes Eq. (1) on the demonstrations and therefore suffers from exposure bias and target ambiguity (§ 1). Following on-policy self-distillation for language models [47], we let a single network serve as both teacher and student, where the teacher additionally receives hindsight [4]: the future video of the demonstration. Hindsight is available in every demonstration at no extra cost and resolves the ambiguity of the SFT target, while distilling on the student’s own sampling trajectories addresses the exposure bias. As shown in Figure 2, HINT-WAM proceeds in three stages:

Figure 2. Overview of HINT-WAM. Stage 0 (top left): the action expert is fine-tuned with flow matching on demonstrations, conditioned on the key/value cache of the frozen video expert, giving the student initialization. Stage A (top right): with the SFT model frozen, a hindsight encoder compresses the demonstration’s future observations into hindsight tokens, which the action expert reads through gated cross-attention adapters; only the encoder and adapters are trained. Stage B (bottom): the student integrates its own ODE without gradients, and its implied actions at selected denoising points are matched to a single sample from the frozen hindsight teacher with the energy score, together with a behavior-cloning anchor. Only the student is deployed.

  • Stage 0 (SFT). The action expert is fine-tuned on the demonstrations with Eq. (1), giving the student initialization θ0\theta_0.

  • Stage A (hindsight teacher). With θ0\theta_0 frozen, a lightweight hindsight encoder and gated adapters learn to read the future video, giving the teacher T=(θ0,ψ)T = (\theta_0,\psi).

  • Stage B (on-policy self-distillation). The student θ\theta, initialized from θ0\theta_0, is trained to match the teacher on its own sampling trajectories, with only the action expert updated.

The video expert stays frozen in every stage. At deployment, only the student is used, so the policy never requires access to the future.

Hindsight Teacher

The teacher is the student network θ0\theta_0 equipped with two small trainable modules: a hindsight encoder that summarizes the future of the demonstration, and gated cross-attention adapters that allow the frozen action expert to read this summary.

Hindsight encoder. Each training sample is a demonstration clip whose video window is encoded by the VAE into latents z∈RC×T×h×wz \in\mathbb{R}^{C \times T \times h \times w}. The first latent frame encodes the current observation, which the student already sees, while the remaining frames describe what happens next in the demonstration over the same horizon as the action chunk. The encoder drops the first latent frame so that the teacher gains no advantage from information shared with the student, mean-pools every remaining frame over space, and projects the resulting sequence with a linear layer followed by LayerNorm. A set of NϕN_{\phi} learned queries then pools this

sequence through multi-head attention [19]:

u=LN⁡(Wppool⁡hw(z1:)),ϕψ(z)=LN⁡(WoMHA⁡(Q,u))∈RNϕ×dϕ,(4)\begin{aligned} u &= \operatorname{LN}(W_p\operatorname{pool}_{hw}(z_{1:})),\\ \phi_{\psi}(z) &= \operatorname{LN}(W_o\operatorname{MHA}(Q,u)) \in\mathbb{R}^{N_{\phi} \times d_{\phi}}, \tag*{(4)} \end{aligned}

where uu is the projected sequence of future frames and QQ denotes the learned queries. Spatial pooling discards the scene layout, already visible in the current frame, and keeps the temporal axis that carries the trajectory’s future.

Gated cross-attention adapters. We attach adapters to the action expert after a subset of layers S\mathcal{S}. For the action-token hidden states hh leaving layer ℓ∈S\ell\in\mathcal{S}, the adapter computes

q=WdownLN⁡(h),k=Wkϕ,v=Wvϕ,h′=h+tanh⁡(αℓ)WupWoutAttn⁡(q,k,v),(5)\begin{aligned} q &= W_{\mathrm{down}}\operatorname{LN}(h), \qquad k = W_k\phi, \qquad v = W_v\phi,\\ h' &= h + \tanh(\alpha_{\ell})W_{\mathrm{up}}W_{\mathrm{out}}\operatorname{Attn}(q,k,v), \tag*{(5)} \end{aligned}

with a tanh gate in the style of Flamingo [3]. The adapter runs beside the frozen block rather than inside the shared attention, so that a zero branch is an exact identity, and we initialize Wup=0W_{\mathrm{up}} = 0 and αl=1\alpha_l = 1, so that the teacher is identical to θ0\theta_0 at the first step while the branch still receives gradients [35, 46]. Without ϕ\phi, the adapters are skipped, so student and teacher share all backbone weights.

Training objective. The entire backbone θ0\theta_0, including both experts, is frozen, and only the encoder and adapter parameters, collectively denoted ψ\psi, are trained with the flow-matching loss of Eq. (1) given hindsight:

LA(ψ)=E∥vθ0,ψ(aσ,σ,o,ϕψ(z))−(ϵ−a)∥22.(6)\mathcal{L}_A(\psi) = \mathbb{E}\left\lVert v_{\theta_0,\psi}\left(\mathbf{a}_\sigma,\sigma,o,\phi_\psi(\mathbf{z})\right) - \left(\boldsymbol{\epsilon}-\mathbf{a}\right)\right\rVert_2^2. \tag*{(6)}

Freezing θ0\theta_0 guarantees that any improvement of the teacher stems from the hindsight ϕ\phi rather than from additional fitting capacity. The teacher must be built on the same weights from which the student starts; otherwise it would pull the student toward a different backbone.

Distilling Hindsight on Student Trajectories

In Stage B, the student θ\theta is initialized from θ0\theta_0 and only its action expert is trained, while the teacher T=(θ0,ψ)T = (\theta_0,\psi) is kept frozen. Each training sample provides the current observation oo, the demonstrated chunk a\mathbf{a}, and the future latents z\mathbf{z}, seen only by the teacher through ϕ=ϕψ(z)\phi= \phi_\psi(\mathbf{z}).

On-policy states from the student’s sampler. Starting from fresh noise, the student integrates its own ODE in Eq. (2) with the deployed schedule and without gradients, and records each visited state aσi\mathbf{a}_{\sigma_i}. For the given observation, these are precisely the denoising states that the policy visits at inference. This rollout is on-policy with respect to the action sampler and needs no simulator; the observation itself is taken from the demonstration.

Where to supervise. The local error of an Euler step grows with Δi2\Delta_i^2, so we weight trajectory points by wi∝Δi2w_i \propto\Delta_i^2. On the shifted schedule we use, this weighting concentrates most of its mass on the last few points. Rather than sampling σ\sigma, we therefore enumerate the top-KK points and renormalize their weights to wˉk\bar{w}_k. Gradients enter only by reevaluating the student at these points, and we never backpropagate through the sequential rollout.

Velocity matching. A direct objective matches the teacher’s velocity at the selected states. With vθk=vθ(aσk,σk,o)v_\theta^k = v_\theta(a_{\sigma_k},\sigma_k,o) and vTk=vT(aσk,σk,o,ϕ)v_T^k = v_T(a_{\sigma_k},\sigma_k,o,\phi), we minimize

Lvel(θ)=∑k=1Kwˉk E∥vθk−sg⁡[vTk]∥22,(7)\mathcal{L}_{\mathrm{vel}}(\theta) = \sum_{k=1}^{K}\bar{w}_k\,\mathbb{E}\left\lVert v_\theta^k-\operatorname{sg}\left[v_T^k\right]\right\rVert_2^2, \tag*{(7)}

where sg⁡[⋅]\operatorname{sg}[\cdot] is the stop-gradient operator and padded steps of the chunk are masked out.

Distribution matching with the energy score. For a fixed student input, the minimizer of Eq. (7) is the teacher’s conditional mean given what the student observes. Because the student cannot see ϕ\phi, the teacher’s behavior is multimodal from the student’s perspective, and its mean may be an action that is correct under none of the modes. We therefore match distributions with the energy score [14, 41], a strictly proper scoring rule that requires only a single sample from the target. Concretely, the student draws MM independent trajectories and the teacher draws one action at σk\sigma_k by integrating its own ODE with ϕ\phi from fresh noise, both without gradients. At the selected point σk\sigma_k of the mm-th student trajectory, a differentiable student sample is obtained in a single forward pass through the implied action of Eq. (3), a^m,k=aσkm−σkvθ(aσkm,σk,o)\hat{\mathbf{a}}_{m,k}=\mathbf{a}_{\sigma_k}^{m}-\sigma_k v_\theta(\mathbf{a}_{\sigma_k}^{m},\sigma_k,o). The energy score over whole action chunks is

LES(θ)=∑k=1Kwˉk[1M∑m=1M∥a^m,k−aT∥2−12M2∑m,m′=1M∥a^m,k−a^m′,k∥2],(8)\mathcal{L}_{\mathrm{ES}}(\theta)=\sum_{k=1}^{K}\bar{w}_k\left[\frac{1}{M}\sum_{m=1}^{M}\left\lVert\hat{\mathbf{a}}_{m,k}-\mathbf{a}_T\right\rVert_2-\frac{1}{2M^2}\sum_{m,m'=1}^{M}\left\lVert\hat{\mathbf{a}}_{m,k}-\hat{\mathbf{a}}_{m',k}\right\rVert_2\right], \tag*{(8)}

where the norm is taken over the flattened H×DH \times D chunk. The first term pulls the student’s samples toward the teacher’s sample, and the second term pushes them apart, which prevents collapse onto the conditional mean. The population energy score is strictly proper, minimized only when the student’s distribution equals the teacher’s. Eq. (8) is its plug-in estimator, whose expectation weights the repulsion by (M−1)/M(M-1)/M (the unbiased estimator uses 1/[2M(M−1)]1/[2M(M-1)] over m≠m′m \ne m'); it thus slightly underweights dispersion, a bias vanishing as MM grows.

Behavior-cloning anchor. Pure distillation would also follow the teacher where the teacher is wrong. We bound this effect with a flow-matching term on the demonstrations, computed for the student without hindsight, LBC(θ)=LFM(θ)\mathcal{L}_{\mathrm{BC}}(\theta)=\mathcal{L}_{\mathrm{FM}}(\theta). The Stage-B objective is:

LB(θ)=λdistLdist(θ)+λBCLBC(θ),(9)\mathcal{L}_B(\theta)=\lambda_{\mathrm{dist}}\mathcal{L}_{\mathrm{dist}}(\theta)+\lambda_{\mathrm{BC}}\mathcal{L}_{\mathrm{BC}}(\theta), \tag*{(9)}

where Ldist∈{Lvel,LES}\mathcal{L}_{\mathrm{dist}}\in\{\mathcal{L}_{\mathrm{vel}},\mathcal{L}_{\mathrm{ES}}\} is the distillation objective, and λdist\lambda_{\mathrm{dist}} and λBC\lambda_{\mathrm{BC}} balance the two terms. Algorithm 1 summarizes one training step.

Experiments

Implementation

Datasets. We evaluate HINT-WAM on two simulation benchmarks, LIBERO [28] and RoboTwin 2.0 [9]. Our Algorithm 1 One training step of HINT-WAM (Stage B).

Require: batch {(o,a,z)}\{(o, a, z)\}, student θ\theta, frozen teacher T=(θ0,ψ)T = (\theta_0, \psi), KK, MM, NN, λdist\lambda_{\mathrm{dist}}, λBC\lambda_{\mathrm{BC}}

  1. C←C \leftarrow prefill video keys/values from the current frame in oo ▷ shared below

  2. ϕ←ϕψ(z)\phi\leftarrow\phi_\psi(z) ▷ hindsight, teacher only

  3. for m=1,…,Mm = 1, \ldots, M do

  4. {aσmi}i=1M←ROLLOUT⁡(vθ(⋅,⋅,o;C),ϵm)\{a_{\sigma_m^i}\}_{i=1}^{M} \leftarrow\operatorname{ROLLOUT}(v_\theta(\cdot,\cdot,o;C), \epsilon_m) ▷ no grad

  5. end for

  6. {(σk,wˉk)}k=1K←\{(\sigma_k,\bar{w}_k)\}_{k=1}^{K} \leftarrow top-KK points by Δi2\Delta_i^2

  7. aT←ROLLOUT⁡(vT(⋅,⋅,o,ϕ;C),ϵT).final⁡a_T \leftarrow\operatorname{ROLLOUT}(v_T(\cdot,\cdot,o,\phi;C), \epsilon_T).\operatorname{final} ▷ no grad

  8. a^m,k←aσk−σkvθ(aσk,σk,o;C)\hat{a}_{m,k} \leftarrow a_{\sigma_k} - \sigma_k v_\theta(a_{\sigma_k},\sigma_k,o;C) ▷ with grad

  9. Ldist←\mathcal{L}_{\mathrm{dist}} \leftarrow energy score of {a^m,k}\{\hat{a}_{m,k}\} against aTa_T ▷ Eq. (8)

  10. LBC←\mathcal{L}_{\mathrm{BC}} \leftarrow flow-matching loss of θ\theta on (o,a)(o,a) ▷ Eq. (1)

  11. update the action expert of θ\theta with ∇θ[λdistLdist+λBCLBC]\nabla_\theta[\lambda_{\mathrm{dist}}\mathcal{L}_{\mathrm{dist}}+\lambda_{\mathrm{BC}}\mathcal{L}_{\mathrm{BC}}]

main benchmark is LIBERO-90, whose 90 tasks share the robot and action space of the four standard LIBERO suites on which the released Fast-WAM checkpoint was trained, yet are unseen by it. We post-train on its 3,921 demonstrations and evaluate each task with 50 trials from the fixed initial states of LIBERO. To measure forgetting, we additionally evaluate the 40 tasks of the four original suites, which also provide a small replay memory for our continual variant. RoboTwin 2.0 covers dual-arm manipulation over 50 tasks. Since the released checkpoint was trained on these tasks, we evaluate on a hard subset of 25 tasks, selected before any training, with 50 episodes each in the clean and randomized settings. All methods share the same initial states, number of trials, and sampler.

Training details. We keep the architecture and sampler of the released Fast-WAM checkpoints unchanged, with 32-step action chunks generated by 10 Euler steps. The hindsight encoder and the adapters add about 13M parameters (0.2%), used only by the teacher. All stages use AdamW [31] with a cosine schedule and an effective batch size of 192 on 8 A5000 GPUs. Stage 0 fine-tunes the action expert for 5,000 steps (learning rate 10−410^{-4}), Stage A trains the teacher modules for 2,000 steps (3×10−43 \times10^{-4}), and Stage B trains the action expert for 2,000 steps (10−410^{-4}) with M=4M = 4, K=1K = 1, and λdist=λBC=1\lambda_{\mathrm{dist}} = \lambda_{\mathrm{BC}} = 1. Stages A and B start from SFT step 3,000, while we report the final step 5,000 as the SFT baseline. On RoboTwin, the released checkpoint serves as the SFT model, and all hyperparameters are reused without tuning. For continual post-training, the replay memory [7] is mixed into every stage, so that the teacher also learns to use hindsight on the old tasks. Further details are given in the supplementary material.

Comparison Results

We compare HINT-WAM with the following baselines, which share its starting checkpoint, data, trainable parameters, optimizer, and training budget unless stated otherwise:

  • Fast-WAM [45]: the released checkpoint.

  • SFT: fine-tunes the action expert with Eq. (1), without a teacher or on-policy states.

  • Full SFT: fine-tunes both the video and action experts.

  • BC control: continues SFT with the BC loss only (λdist=0\lambda_{\mathrm{dist}} = 0), testing the effect of additional training.

  • Self-imitation: adds successful rollouts of the SFT policy from randomized placements to the demonstrations and trains with behavior cloning.

  • Shuffled hindsight: runs HINT-WAM, but the teacher reads the future of another episode, removing only the privileged information.

On RoboTwin, we compare with Fast-WAM, full SFT, and the BC control.

Results on LIBERO-90. As shown in Table 1, the released Fast-WAM checkpoint solves only a small fraction of the unseen LIBERO-90 tasks, and SFT raises its success rate to 78.88%. HINT-WAM further improves it to 85.18%, a gain of 6.30% over SFT on average across the 90 tasks (paired t=3.58t = 3.58). It also surpasses full SFT, which fine-tunes both experts, by 3.94% while training only the action expert. The comparison baselines show where this gain comes from. First, it does not stem from additional training: continuing SFT with the same number of steps but without the teacher lowers the success rate to 71.56%. Second, it does not stem from training on the policy’s own data alone: adding the policy’s successful rollouts improves SFT by only 0.74%, and HINT-WAM outperforms self-imitation by 5.56% (paired t=2.96t = 2.96). Third, the gain is carried by the privileged information: when the teacher reads the future of another episode, the identical pipeline reaches only 80.28%, 4.90% below HINT-WAM (paired t=2.76t = 2.76).

MethodLIBERO-90RoboTwin 2.0 CleanRoboTwin 2.0 Random.
Fast-WAM [45]15.5686.4883.44
SFT78.88––
Full SFT81.2484.1283.24
BC control71.5685.3682.08
Self-imitation79.62––
Shuffled hindsight80.28––
HINT-WAM (Ours)85.1887.0483.68

Table 1. Post-training performance on LIBERO-90 and RoboTwin 2.0. All values are success rates (%). –: not applicable or not evaluated.

Results on RoboTwin 2.0. On RoboTwin 2.0, the released checkpoint was already trained on the same demonstrations, which leaves little headroom. Continued behavior cloning lowers success in both the clean and the randomized settings, by 1.12% and 1.36%, and full SFT also falls below the released checkpoint. In contrast, HINT-WAM stays at or slightly above the released checkpoint and outperforms the BC control by 1.68% in the clean setting (paired t=2.00t = 2.00) and by 1.60% in the randomized setting. The gains are smaller than on LIBERO-90 but show the same trend: given the same training budget, hindsight distillation improves the policy where additional imitation degrades it.

Continual Learning Results

Without replay. As shown in Table 2, post-training on LIBERO-90 alone causes severe forgetting on the four original suites for every method, reducing their average success from 96.55% to below 45%. HINT-WAM retains the most (44.05%, vs. 37.45% for SFT), while full SFT forgets the most (32.85%): updating the video expert as well helps on the new tasks but erodes considerably more of the pretrained skills. Distilled on its own sampling trajectories, the student stays closer to its current behavior than under off-policy imitation, reducing but not preventing forgetting.

MethodNew tasksOriginal suites
LIBERO-90SpatialObjectGoalLongAvg
Fast-WAM [45]15.5696.899.695.894.096.55
Without replay
SFT78.8858.633.253.44.637.45
Full SFT81.2454.430.244.62.232.85
BC control71.5655.425.647.04.033.00
Self-imitation79.6264.439.854.82.440.35
Shuffled hindsight80.2866.236.060.22.441.20
HINT-WAM (Ours)85.1870.838.458.88.244.05
With replay
SFT76.7097.295.892.081.291.55
Full SFT77.6497.896.094.483.492.90
HINT-WAM (Ours)81.5698.496.095.890.295.10

Table 2. Continual learning on LIBERO. Success rates (%) on the new LIBERO-90 tasks and the four original suites after post-training on LIBERO-90, without and with experience replay.

With replay. Experience replay largely closes the gap, and HINT-WAM composes with it. With replay, HINT-WAM outperforms SFT by 4.86% on the new tasks (paired t=2.66t = 2.66) and by 3.55% on the original suites, coming within 1.5% of the released checkpoint, with the largest gain on the long-horizon suite (9.0%). It also leads full SFT by 3.92% and 2.20%, respectively, showing that hindsight distillation is complementary to replay.

Computational Cost

Figure 3 separates the effect of HINT-WAM from that of additional training. SFT has converged: training from 3,000 to 5,000 steps costs 22 more hours and leaves success unchanged, and spending the same steps on BC lowers success, suggesting that further imitation overfits. HINT-WAM spends 4.4 hours on the hindsight teacher and 34 hours on distillation, 1.28× the training time of SFT, only 0.56× the 127 hours of full SFT (2.3× SFT), which still reaches lower success. A distillation step is 1.5× slower than a BC step, a cost paid only during training.

Success versus training time on LIBERO-90

Figure 3. Success vs. training time on LIBERO-90. HINT-WAM reaches the highest success at a fraction of the cost of full SFT, while further SFT or BC training brings no gain.

Ablation Study

Distribution matching. Figure 4 compares four distillation objectives under the same teacher, on-policy states, anchor, MM, and training budget. Velocity matching reaches 79.02%, essentially SFT (+0.14%, paired t=0.09t = 0.09), as regressing the teacher’s velocity drives the student toward its conditional mean. Kernel Maximum Mean Discrepancy (MMD) between the student samples and the teacher sample reaches 80.63%: although it matches distributions in principle, its Gaussian-kernel gradients vanish once samples lie farther apart than the bandwidth, as is common for high-dimensional action chunks. Action-space MSE on the implied actions reaches 82.93%, but it is minimized by collapsing every student sample onto the teacher’s mean, which discards the teacher’s uncertainty and conflicts with the flow-matching anchor; its gradient also grows with distance, so a single noisy teacher sample can dominate. The energy score, which is MMD with the distance kernel, avoids both issues: it matches the full distribution, needs no bandwidth, and has unit-norm gradients at every distance. It reaches 85.18%, 6.16% above velocity matching (paired t=3.62t = 3.62, 48 wins vs. 17 losses, sign-test p=0.0004p = 0.0004).

Figure 4. Ablation on the distillation objective. Gain in LIBERO-90 success over SFT with velocity matching (MSE), kernel MMD, action-space MSE, or the energy score.

Behavior-cloning anchor. Figure 5 shows that neither ingredient helps on its own. Continued imitation lowers success from 78.9% to 71.6%, and distillation without the anchor falls to 74.4%, below its own SFT initialization, since nothing ties the student to the demonstrations and errors of the teacher and of its own sampling can accumulate. Combined, the two reach 85.2%: while they change success by −7.3% and −4.5% alone, together they add +6.3%, a strong interaction rather than two additive effects. The anchor thus keeps the student on the demonstration distribution, while distillation corrects it on the states it visits.

Figure 5. Effect of the BC anchor on LIBERO-90.

Number of student samples. Figure 6 varies the number of student samples MM in Eq. (8). With M=1M = 1, the repulsion term vanishes, yet the variant still reaches 81.4% and outperforms velocity matching. Adding samples improves success monotonically to 85.2% at M=4M = 4, by 1.1–1.4% per sample. Action-space MSE with M=4M = 4, which also lacks repulsion, reaches only 82.9%, so the gain at M=4M = 4 stems from the repulsion that preserves the diversity of the student’s samples rather than from the number of samples alone. Since the curve has not saturated, larger MM may help further, at one more gradient-free rollout per sample.

Line chart showing the effect of the number of student samples $M$ on LIBERO-90 success

Figure 6. Effect of the number of student samples MM on LIBERO-90.

On-policy states. Figure 7 isolates where the student is supervised. Replacing the student’s sampler states with demonstrated actions noised to the same points σk\sigma_k, i.e., the states seen by SFT, reaches only 80.7%, whereas the student’s own sampling trajectories reach 85.2%. The shuffled-hindsight baseline (80.3%, Table 1), which keeps on-policy states but removes the privileged information, shows the converse. Hindsight and on-policy states thus each improve SFT by less than 2% alone but add 6.3% together: the teacher’s correction is most useful on the denoising states the student actually reaches at inference.

Figure 7. Effect of on-policy states on LIBERO-90.

Conclusion

We presented HINT-WAM, the first on-policy self-distillation framework for world action models, in which a hindsight teacher sharing the student’s backbone supervises the action expert on its own sampler states with the energy score. Without an external teacher, rewards, or a simulator, HINT-WAM outperforms SFT on LIBERO-90 and RoboTwin 2.0, forgets less, and composes with replay. Since its on-policy states come from the sampler rather than closed-loop interaction, closed-loop data collection is a natural next step.

This supplementary material provides additional details and results for HINT-WAM. Sec. A describes the implementation and hyperparameters, Sec. B the training cost, Sec. C the checks of the hindsight teacher, Sec. D the full paired statistics on LIBERO-90, Sec. E a per-task analysis, Sec. F further RoboTwin 2.0 results, Sec. G the LIBERO-90 evaluation protocol, Sec. H the self-imitation baseline, Sec. I the selection of the objective and learning rate, Sec. J a toy experiment on mode averaging, Sec. K qualitative rollouts, and Sec. L a discussion of the main design questions.

Implementation Details and Hyperparameters

Datasets. For LIBERO-90, we use the LeRobot-format release of the 90 tasks, comprising 3,921 demonstrations and 569K frames recorded at 20 Hz. The policy observes an agent-view and a wrist camera at 224×224224 \times224, concatenated horizontally into a single frame, and predicts 7-D actions (end-effector pose deltas and gripper). Forgetting is measured on the 40 tasks of the four original LIBERO suites (Spatial, Object, Goal, and Long) with 50 trials each. For the continual variant, the replay memory contains 5 demonstrations per original task (200 episodes) sampled once with a fixed seed; repeated four times, it accounts for about 19% of the training frames and is mixed into every training stage. On RoboTwin 2.0, the policy observes a head camera and two wrist cameras at 384×320384 \times320 and predicts 14-D actions (joint positions and grippers of both arms). We train on the demonstrations of the 50 tasks used by the released checkpoint and hold out 1% of the episodes. All methods use the normalization statistics of the released checkpoints.

Architecture. The Mixture-of-Transformers backbone of Fast-WAM has 30 layers. Its video expert is the Wan2.2-TI2V-5B diffusion transformer (5.00B parameters), and its action expert is a diffusion transformer with 1.02B parameters. Each action chunk spans H=32H = 32 control steps, and the video window covers 33 frames, of which every fourth frame is encoded by the VAE. Actions are generated with 10 Euler steps on a schedule with shift 5, both during training and at inference. Our modules are used only by the teacher. The hindsight encoder takes the VAE latents of the future frames, dropping the first latent frame, mean-pools them over space, and applies a linear layer (48→51248 \to512) followed by LayerNorm. Eight learned queries then pool the sequence with 8-head attention, followed by a linear layer and LayerNorm, so that ϕ∈R8×512\phi\in\mathbb{R}^{8 \times512}. The gated adapters are attached after layers {4,9,14,19,24,29}\{4, 9, 14, 19, 24, 29\} of the action expert with a bottleneck width of 512 and 8 heads, and are initialized with αl=1\alpha_l = 1 and Wup=0W_{\mathrm{up}} = 0, so that the teacher equals the student at initialization. Together, the two modules contain about 13M parameters.

Optimization. All stages use AdamW [31] with β=(0.9,0.95)\beta= (0.9, 0.95), a weight decay of 0.01, a cosine schedule with 5% linear warm-up, and gradient clipping at 1.0. Training runs in bfloat16 with DeepSpeed ZeRO-2 [37] on 8 GPUs with 24 GB of memory each, using seed 42. The video expert is frozen in every stage of every method except full SFT. Tab. 3 lists the stage-specific settings. In Stage 0, we use λBC=2\lambda_{\mathrm{BC}} = 2, which matches the total gradient scale of the distillation stage (λdist+λBC=2\lambda_{\mathrm{dist}} + \lambda_{\mathrm{BC}} = 2). In Stage B, the energy score uses M=4M = 4 student samples, K=1K = 1 supervised point, and one teacher sample, with λdist=λBC=1\lambda_{\mathrm{dist}} = \lambda_{\mathrm{BC}} = 1. At inference, only the student is used, conditioned on the current frame, the instruction, and the proprioceptive state; on LIBERO, it replans every 10 control steps. On RoboTwin, all hyperparameters are reused from LIBERO without any tuning.

Baselines. The BC control and full SFT use the same data, optimizer, schedule, and effective batch size as HINT-WAM (Tab. 3). The BC control is Stage B with the distillation term removed (λdist=0\lambda_{\mathrm{dist}} = 0, λBC=1\lambda_{\mathrm{BC}} = 1): it starts from SFT at step 3,000 and trains the action expert for 2,000 further steps with the flow-matching loss of (1), so it matches HINT-WAM in initialization, trainable parameters, and number of steps but has no teacher. Full SFT follows Stage 0 (λBC=2\lambda_{\mathrm{BC}} = 2, 5,000 steps from the released checkpoint), but trains both the video and the action expert (6.02B parameters); we report its final checkpoint. On RoboTwin, the BC control, full SFT, and HINT-WAM all start from the released RoboTwin checkpoint and train for 2,000 steps.

Training Cost

We measure the training time on 8 GPUs with 24 GB of memory from checkpoint timestamps, so the reported times include periodic evaluation and checkpoint saving. GPU-hours are computed as steps ×\times seconds per step ×8/3600\times8/3600. Tab. 4 lists the cost of every stage and pipeline on LIBERO-90.

HINT-WAM, consisting of SFT for 3,000 steps followed by Stages A and B, costs 1.28× the 5,000-step SFT pipeline and 0.56× full SFT. A full-SFT step costs 2.3× an action-expert step (91.4 vs. 40.2 s) because the 5B-parameter video expert is also trained. A Stage-B step costs 1.51× a BC step, since the student draws M=4M = 4 action chunks and the teacher one, all without gradients. Inference is unchanged: the deployed student is the unmodified Fast-WAM with 10 Euler steps, and the teacher and the hindsight encoder are discarded after training. On RoboTwin, Stage A takes 18.1 s per step (80 GPU-hours for 2,000 steps), Stage B 60.6 s (269 GPU-hours), and the BC control 37.8 s (168 GPU-hours).

HINT-WAMBaselines
Stage 0 (SFT)Stage A (teacher)Stage BBC controlFull SFT
Init.releasedSFT@3k, frozenSFT@3kSFT@3kreleased
Trainableaction expertψ\psi (enc.+adapt.)action expertaction expertvideo + action experts
#Params1.02B13M1.02B1.02B6.02B
LossEq. (1), λBC=2\lambda_{\mathrm{BC}} = 2Eq. (6)Eq. (9)Eq. (1), λBC=1\lambda_{\mathrm{BC}} = 1Eq. (1), λBC=2\lambda_{\mathrm{BC}} = 2
Learning rate10−410^{-4}3×10−43 \times10^{-4}10−410^{-4}10−410^{-4}10−410^{-4}
Steps5,0002,0002,0002,0005,000
s / step40.27.960.640.291.4

Table 3. Training settings of each stage and baseline on LIBERO-90. All runs share the optimizer, schedule, and effective batch size of 192.

Hindsight Teacher Checks

Before Stage B, we verify that hindsight lowers the teacher’s flow-matching loss on held-out frames, using 200 held-out batches and a paired comparison of the teacher with and without ϕ\phi. Table 5 shows that hindsight reduces the action loss of every teacher, and that the advantage of the LIBERO-90 teacher keeps growing from 1,000 to 2,000 training steps.

TeacherLoss change
LIBERO-90, built on SFT−3.28%-3.28\% (t=5.77t = 5.77)
LIBERO-90 + replay, built on SFT + replay−3.06%-3.06\% (t=4.68t = 4.68)
RoboTwin 2.0, built on released checkpoint−5.37%-5.37\% (t=3.43t = 3.43)
LIBERO-90, 1,000 vs. 2,000 steps−2.36%→−3.28%-2.36\% \to-3.28\%

Table 5. Hindsight teacher checks. Relative change in the held-out action loss of the teacher with versus without hindsight ϕ\phi.

The teacher must be built on the model being distilled. In a 20-task pilot with 10 trials per task, a teacher built on the weaker released checkpoint instead of the SFT model pulled the student down from 55.0% to 47.5%, below SFT

(with the velocity-matching objective). This supports the design choice in § 3.4 that the teacher shares the student’s initialization θ0\theta_0.

Full Paired Statistics on LIBERO-90

All tests are paired over the 90 LIBERO-90 tasks with 50 trials each. For every comparison, Table 6 reports the mean difference in success rate, its standard error, the paired tt-statistic, the numbers of tasks on which the first method wins, loses, or ties, and the exact two-sided sign test over the non-tied tasks. The standard error for shuffled hindsight is derived from its mean difference and tt-statistic. All LIBERO-90 numbers in the paper, including the released Fast-WAM and the BC control, use 50 trials per task. SFT reaches 78.88% at both step 3,000 and step 5,000.

ComparisonMean diff.SEttWins / losses / tiesSign test pp
HINT-WAM vs. SFT+6.301.763.5836 / 22 / 320.087
HINT-WAM vs. self-imitation+5.561.882.9647 / 23 / 200.0056
HINT-WAM vs. shuffled hindsight+4.901.782.7645 / 25 / 200.022
HINT-WAM vs. velocity matching+6.161.593.6248 / 17 / 250.00042
Velocity matching vs. SFT+0.141.520.0930 / 37 / 230.46
HINT-WAM + replay vs. SFT + replay+4.86–2.6644 / 23 / 230.014

Table 6. Paired statistics on LIBERO-90. Differences are in % over 90 tasks with 50 trials each.

Per-Task Analysis

Figure 8 shows how the gain is distributed over the 90 tasks. HINT-WAM improves 36 tasks, leaves 32 unchanged, and lowers 22. The gain is concentrated on the tasks where SFT is weak: +32.2% on the 13 tasks where SFT succeeds less than half of the time and +11.5% on the 15 tasks between 50% and 80%. On the 42 tasks that SFT already solves (≥95%\ge95\%), the change is small (−1.7%-1.7\%, i.e., less than one failed episode out of 50). The many ties and small losses on saturated tasks explain why the sign test is weaker than the paired tt-test (Table 6). Since tasks are grouped by the success rate of SFT itself, part of this trend may reflect regression to the mean.

Figure 8. Per-task gain of HINT-WAM over SFT on LIBERO-90. (a) Gain on each task, sorted. (b) Mean gain (±\pm standard error) with tasks grouped by the success rate of SFT.

Although HINT-WAM improves the average success rate, it does not improve every task. Table 7 lists the tasks with the largest drops for three pairs of methods. Most drops of HINT-WAM relative to SFT occur on pick-and-place tasks into a basket or a caddy compartment. Task 74 (“pick up the book and place it in the left compartment of the caddy”) is hurt by both distillation objectives, but much more by velocity matching (80% → 32%) than by the energy score (80% → 60%).

HINT-WAM vs. SFTMSE vs. SFTHINT-WAM vs. MSE
TaskSFTOursTaskSFTMSETaskMSEOurs
528058748032768660
536040899044535230
748060866028754830
510086588052219680
7810086610074309480

Table 7. Largest per-task drops on LIBERO-90. Success rates (%) over 50 trials per task.

Task instructions: 5, “put the chocolate pudding in the top drawer of the cabinet and close it”; 52, “pick up the milk and put it in the basket”; 53, “pick up the orange juice and put it in the basket”; 74, “pick up the book and place it in the left compartment of the caddy”; 78, “pick up the book and place it in the front compartment of the caddy”.

RoboTwin 2.0 Details

Hard subset. We keep the 25 tasks on which the released checkpoint scores below 95% in either the clean or the randomized setting, measured with 20 episodes per task before any of our training: blocks_ranking_size, handover_block, move_can_pot, move_stapler_pad, hanging_mug, open_microwave, pick_diverse_bottles, place_a2b_left, place_a2b_right, place_bread_basket, place_bread_skillet, place_can_basket, place_dual_shoes, place_mouse_pad, place_object_basket, place_object_scale, place_object_stand, put_bottles_dustbin, put_object_cabinet, rotate_qrcode, scan_object, stack_blocks_three, stack_bowls_three, stamp_seal, and turn_switch. Evaluation uses 50 episodes per task and setting. All methods start from the released checkpoint with the same data, 2,000 steps, a learning rate of 10−410^{-4}, and a batch size of 192.

Paired tests. Table 8 reports paired tests over the 25 tasks. HINT-WAM improves over the BC control in both settings, with borderline significance in the clean setting, while its difference from the released checkpoint is not significant. Continued full SFT also lowers success, by 2.36% in the clean and 0.20% in the randomized setting, and HINT-WAM exceeds it by 2.92 and 0.44%, respectively. Since the released checkpoint was already trained on this data, little headroom remains: continued training lowers success, whether by BC or by full fine-tuning, whereas HINT-WAM keeps it at or slightly above the level of the released checkpoint.

ComparisonCleanRandomized
HINT-WAM vs. BC control+1.68 (0.84, t=2.00t = 2.00), 12 / 9 / 4+1.60 (1.32, t=1.21t = 1.21), 14 / 9 / 2
HINT-WAM vs. released+0.56 (1.19, t=0.47t = 0.47), 11 / 9 / 5+0.24 (1.04, t=0.23t = 0.23), 13 / 9 / 3
BC control vs. released−1.12 (1.08, t=−1.04t = -1.04), 9 / 11 / 5−1.36 (1.43, t=−0.95t = -0.95), 8 / 15 / 2

Table 8. Paired tests on the RoboTwin 2.0 hard subset. Mean difference in % (SE, tt) and wins / losses / ties over 25 tasks.

Evaluation Protocol on LIBERO-90

Training and evaluation use the same 90 tasks but not the same episodes. Training uses the recorded human demonstrations (3,921 episodes, 569K frames), of which 10% (393 episodes, split once with a fixed seed) are excluded from training and used only for offline loss metrics. Evaluation runs 50 new closed-loop episodes per task from the fixed initial states provided by LIBERO, which are generated separately from the demonstrations. This is the standard LIBERO protocol, also used by OpenVLA [18], π0\pi_0 [5], and Fast-WAM [45]. We find no sign that the test placements are memorized: the SFT checkpoint at step 3,000 reaches 80.94% on 1,800 random placements, which never coincide with the test placements, and 78.88% on the fixed test placements. Our evaluation thus measures generalization to held-out initial configurations of the training tasks rather than to unseen tasks.

Self-Imitation Baseline

The SFT checkpoint at step 3,000 was rolled out from 20 random initial placements per task, none of which coincides with the test placements, yielding 1,800 episodes, of which 1,457 (80.94%) succeeded. We added 1,454 successful episodes (203K frames, about 26% of the training frames) to the LIBERO-90 demonstrations, excluding three episodes whose instruction does not appear in the training data. Training then followed Stage B of HINT-WAM with the behavior-cloning loss only, starting from SFT step 3,000 for 2,000 steps with a learning rate of 10−410^{-4} and a batch size of 192.

Selection of the Objective and Learning Rate

Most hyperparameters are inherited from Fast-WAM or fixed by design, and the loss weights, adapter placement, and hindsight size were not tuned. The few design decisions we made were based on the held-out LIBERO-90 episodes and a 20-task pilot on LIBERO-90 with 10 trials per task; no tuning was performed on RoboTwin.

  • Learning rate. An initial round with a learning rate of 10−510^{-5} for 500 steps left every method under-trained, at or below 16% on LIBERO-90. We therefore used 10−410^{-4}, the fine-tuning learning rate of Fast-WAM.

  • Objective. On the held-out episodes, the energy score reduced the error of the generated actions more than velocity matching (23.7% vs. 15.5% relative to the baseline). The closed-loop results confirm this choice (85.18% vs. 79.02%, Table 6).

  • Supervised points. With the Δi2\Delta_i^2 weighting on our 10-step schedule, the last point carries 63.6% of the mass and the last three points 91.1%. We therefore use K=1K=1 for the energy score and K=3K=3 for velocity matching. With K=1K=1, the selected point is σN\sigma_N, so the implied action a^m,1\hat{\mathbf{a}}_{m,1} is exactly the output of the mm-th student rollout.

Toy Experiment: Mode Averaging

The toy example in Figure 9 isolates the mode-averaging effect of velocity matching. A hidden binary variable cc decides which of two actions is correct; the teacher observes it, and the student does not. Distilled with the squared error, the student predicts the average of the two actions, which lands on neither mode (success 0.00). Distilled with the energy score from the same single teacher samples and M=4M=4, it places its samples on both modes (success 0.92). The same mechanism explains why velocity matching yields no gain on LIBERO-90, while the energy score improves success by 6.16% over it (Table 6).

Figure 9. Toy example of distillation under hidden information. A hidden variable cc selects one of two correct actions; the teacher observes it, the student does not. (a) Teacher samples. (b) Squared-error distillation collapses to the mean of the two modes. (c) The energy score recovers both.

Setup. The observation is x∼U(−1,1)x \sim\mathcal{U}(-1,1), and a hidden c∈{−1,+1}c \in\{-1,+1\}, drawn independently of xx with equal probability, sets the correct action a=c(1+0.5x)a=c(1+0.5x), so the student cannot infer cc and no single action is correct for both values. The teacher observes cc and outputs the correct action, and each training state provides one teacher sample. The student is an implicit sampler a=f(x,z)a=f(x,z) with z∼N(0,1)z \sim\mathcal{N}(0,1), implemented as an MLP with four hidden layers of 64 units and GELU activations. Both students are trained for 3,000 steps with Adam (learning rate 3×10−33 \times10^{-3}, batch size 256): the squared-error student regresses one sample onto the teacher sample, and the energy-score student matches M=4M=4 samples to it. Success is the fraction of student samples within 0.25 of either correct action, computed over 64 values of xx with 512 samples each and averaged over 3 seeds; a student that predicts the mean of the two actions therefore scores zero.

Qualitative Rollouts

Figs. 10 to 12 show closed-loop rollouts in which HINT-WAM succeeds where the baseline fails. Each pair of rows starts from the same initial state; the frames are evenly spaced over the episode, and the badge on the last frame marks success or failure.

New tasks. Figure 11 shows four LIBERO-90 tasks. The failures of SFT are typically errors in the final stage of the task rather than in grasping: the moka pot is not set down on the stove (task 19), the drawer is left open after the butter has been placed (task 3), the bowl is not placed on the cabinet (task 43), and the book is dropped into the wrong compartment of the caddy (task 81). HINT-WAM completes each of these tasks from the same initial state.

Rollouts on new LIBERO-90 tasks

Figure 11. Rollouts on new LIBERO-90 tasks. SFT (top row of each panel) fails, while HINT-WAM (bottom row) succeeds from the same initial state.

Retained skills. Figure 12 shows tasks from the original LIBERO suites, which the released Fast-WAM solves before post-training. After post-training on LIBERO-90 without replay, SFT fails these tasks, whereas HINT-WAM still solves them, consistent with its smaller forgetting in Table 2.

Rollouts on retained skills after post-training on LIBERO-90 without replay

Figure 12. Rollouts on retained skills after post-training on LIBERO-90 without replay. The released Fast-WAM (top) and HINT-WAM (bottom) succeed, while SFT (middle) fails, from the same initial state.

RoboTwin 2.0. Figure 10 shows four episodes of stack_bowls_three (clean setting). SFT moves single bowls but never completes the stack: it carries a bowl without placing it (a), loses track of the remaining bowls (b), flips a bowl (c), or stops after one lift (d), whereas HINT-WAM stacks all three. Both reach the same success rate on this task, so these episodes illustrate failure modes rather than a per-task gain.

Figure 10. Rollouts on RoboTwin 2.0 (stack_bowls_three, clean setting). SFT (top row of each panel) fails, while HINT-WAM (bottom row) stacks all three bowls from the same initial state.

Discussion

Why does a teacher that sees the future help? The student never receives ϕ\phi; the teacher supplies one action sample per state. Since the energy score is strictly proper, the student is driven towards the distribution of teacher actions given what it observes, i.e., the continuations consistent with the demonstrations, not one particular future. Hindsight thus resolves the ambiguity of the SFT target rather than giving information the student must recover. The gain requires the correct future: with the future of another episode, the same pipeline reaches only 80.28% (Table 1).

In what sense is HINT-WAM on-policy? The states are on-policy with respect to the action sampler: the student is supervised on the denoising states its own ODE visits; noised demonstrations instead reach only 80.74% (Figure 7). The observations still come from the demonstrations, so HINT-WAM needs neither rewards nor a simulator, but does not address compounding errors across control steps and is complementary to DAgger or reinforcement learning.

Is the gain due to extra compute or tuning? No. The same 2,000 extra steps without the teacher lower success to 71.56%, and HINT-WAM exceeds full SFT by 3.94% at 0.56× its cost (Table 4). Neither its loss weights nor its modules were tuned, and all hyperparameters are reused unchanged on RoboTwin.

StageStepss/stepHoursGPU-h
SFT (action expert)3k / 5k40.233.5 / 55.8268 / 447
Stage A (teacher)2k7.94.435
Stage B (distill.)2k60.633.7269
BC control2k40.222.3179
Full SFT5k91.41271,016
PipelineHoursGPU-hSuccess
SFT 5k55.844778.88
SFT 3k + BC 2k55.844771.56
Full SFT1271,01681.24
HINT-WAM71.657285.18

Table 4. Training cost on LIBERO-90. Top: individual stages. Bottom: complete pipelines and their LIBERO-90 success rates.

References

  1. [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In ICLR, 2024. 2arxiv.org/abs/2306.13649
  2. [2]A. Akbari et al. Flash-WAM: Modality-aware distillation for world action models. arXiv preprint arXiv:2606.05254, 2026. 1, 2arxiv.org/abs/2606.05254
  3. [3]Jean-Baptiste Alayrac et al. Flamingo: A visual language model for few-shot learning. In NeurIPS, 2022. 5arxiv.org/abs/2204.14198
  4. [4]Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Feng, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In NeurIPS, 2017. 3arxiv.org/abs/1707.01495
  5. [5]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. 2, 4arxiv.org/abs/2410.24164
  6. [6]Anthony Brohan et al. RT-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems (RSS), 2023. 2arxiv.org/abs/2212.06817
  7. [7]Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019. 6arxiv.org/abs/1902.10486
  8. [8]Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference on Robot Learning (CoRL), 2019. 2arxiv.org/abs/1912.12294
  9. [9]Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, et al. RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088, 2025. 5arxiv.org/abs/2506.18088
  10. [10]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), 2022. 2
  11. [11]Phuong Tuan Dat, Qi Li, and Xinchao Wang. dOPSD: On-policy self-distillation for diffusion language models. arXiv preprint arXiv:2607.04428, 2026. 3arxiv.org/abs/2607.04428
  12. [12]Phuong Tuan Dat, Qi Li, and Xinchao Wang. Prism: Privileged sibling-guided multi-agent topology design, 2026. 3arxiv.org/abs/2610.20628
  13. [13]Z. Fang et al. Flow-OPD: On-policy distillation for flow matching models. arXiv preprint arXiv:2605.08063, 2026. 3arxiv.org/abs/2605.08063
  14. [14]Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. 2, 3, 5
  15. [15]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In ICLR, 2024. 2DOI
  16. [16]Yanjiang Guo et al. Prediction with action: Visual policy learning via joint denoising process. In NeurIPS, 2024. 2arxiv.org/abs/2411.18179
  17. [17]Yucheng Hu et al. Video prediction policy: A generalist robot policy with predictive visual representations. In International Conference on Machine Learning (ICML), 2025. 1, 2arxiv.org/abs/2412.14803
  18. [18]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning (CoRL), 2024. 2, 4arxiv.org/abs/2406.09246
  19. [19]Juho Lee, Yoonhee Lee, Juhoon Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In International Conference on Machine Learning (ICML), 2019. 4arxiv.org/abs/1810.00825
  20. [20]Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, et al. SimpleVLA-RL: Scaling VLA training via reinforcement learning. arXiv preprint arXiv:2509.09674, 2025. 2arxiv.org/abs/2509.09674
  21. [21]L. Li et al. Causal world modeling for robot control. In Robotics: Science and Systems (RSS), 2026. 1, 2arxiv.org/abs/2601.21998
  22. [22]Qi Li, Xingyi Yang, and Xinchao Wang. Badwam: When world-action models dream right but act wrong. arXiv preprint arXiv:2607.15207, 2026. 1arxiv.org/abs/2607.15207
  23. [23]Qi Li, Bo Yin, Weiqi Huang, Ruhao Liu, Bojun Zou, Runpeng Yu, Jingwen Ye, Weihao Yu, and Xinchao Wang. Vision-language-action safety: Threats, challenges, evaluations, and mechanisms. arXiv preprint arXiv:2604.23775, 2026. 1arxiv.org/abs/2604.23775
  24. [24]Q. Li et al. DiffusionOPD: A unified perspective of on-policy distillation in diffusion models. arXiv preprint arXiv:2605.15055, 2026. 3arxiv.org/abs/2605.15055
  25. [25]Shuang Li, Yihua Gao, Dorsa Sadigh, and Shuran Song. Unified video action model. In Robotics: Science and Systems (RSS), 2025. 2arxiv.org/abs/2503.00200
  26. [26]Weixin Liang et al. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models. arXiv preprint arXiv:2411.04996, 2024. 3arxiv.org/abs/2411.04996
  27. [27]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In ICLR, 2023. 2, 3arxiv.org/abs/2210.02747
  28. [28]Bo Liu, Yifeng Zhu, Chungkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. In NeurIPS, 2023. 5arxiv.org/abs/2306.03310
  29. [29]Panjun Liu, Xiaohan Lei, Shiqi Zhang, Yikun Wang, Yongxin Zhang, Mingyi Hu, Shida Sun, Jiateng Shou, Wengang Zhou, Jiajun Deng, and Zhiwei Xiong. WAM-OPD: Sharpening world action models via on-policy distillation. arXiv preprint arXiv:2609.34250, 2026. 3arxiv.org/abs/2609.34250
  30. [30]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023. 3arxiv.org/abs/2209.03003
  31. [31]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 6, 1
  32. [32]Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. VLA-RL: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505.18719, 2025. 2
  33. [33]Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models. In International Conference on Machine Learning (ICML), 2023. 2arxiv.org/abs/2301.11706
  34. [34]Abby O’Neill et al. Open X-embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation (ICRA), 2024. 2arxiv.org/abs/2310.08864
  35. [35]William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 5arxiv.org/abs/2212.09748
  36. [36]Z. Qian, X. Chi, Y. Qi, H. Li, Z. Y. Chen, and S. Zhang. WAM-RL: World-action model reinforcement learning with reconstruction rewards and online video SFT. arXiv preprint arXiv:2606.17906, 2026. 2arxiv.org/abs/2606.17906
  37. [37]Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory optimizations toward training trillion parameter models. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2020. 1arxiv.org/abs/1910.02054
  38. [38]Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011. 2arxiv.org/abs/1011.0686
  39. [39]Qiuhong Shen, Shihua Zhang, Yue Liao, Qi Li, Zhenxiong Tan, Shizun Wang, Shuicheng Yan, and Xinchao Wang. World action models: A survey, 2026. 2
  40. [40]Idan Shenfeld, Jyotish Pari, and Pukit Agrawal. RL’s razor: Why online reinforcement learning forgets less. arXiv preprint arXiv:2509.04259, 2025. 2
  41. [41]Gábor J. Székely and Maria L. Rizzo. Energy statistics: A class of statistics based on distances. Journal of Statistical Planning and Inference, 143(8):1249–1272, 2013. 5DOI
  42. [42]Team Wan et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. 3arxiv.org/abs/2503.20314
  43. [43]Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Neural Networks, 22(5-6):544–557, 2009. 2DOI
  44. [44]Liuhaiwen Yang, Zhengyang Zhong, Hanshang Zhu, Ningwei Bai, Qichen Yin, Zhi Han, et al. WAM-OPD: Joint video-action supervision for world action model post-training with on-policy distillation. arXiv preprint arXiv:2608.22364, 2026. 3arxiv.org/abs/2608.22364
  45. [45]Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-WAM: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026. 3, 6, 7, 4arxiv.org/abs/2603.16666
  46. [46]Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. 5arxiv.org/abs/2302.05543
  47. [47]Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. In International Conference on Machine Learning (ICML), 2026. 2, 3arxiv.org/abs/2601.18734
  48. [48]Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. RoboDreamer: Learning compositional world models for robot imagination. In International Conference on Machine Learning (ICML), 2024. 2arxiv.org/abs/2404.12377
  49. [49]Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. In Robotics: Science and Systems (RSS), 2025. 1, 2arxiv.org/abs/2504.02792
  50. [50]Brianna Zitkovich et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL), 2023. 2arxiv.org/abs/2307.15818

Paper details

vlnc:W_xjpqf3ng
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents