WorksComputer Science
Beyond Validation Loss: Coverage and Reliability Dynamics in Robot Policy Pretraining
Abstract
In modern robot learning, generative robot policies are first pretrained through supervised learning from demonstrations, then refined through reinforcement learning, supervised finetuning, or other forms of posttraining. Yet conventional pretraining metrics, such as validation loss and average task success, provide an incomplete account of which policies support both effective direct deployment and subsequent posttraining. In this work, we study closed-loop pass@k, which evaluates the probability that k rollouts from the same initial condition attain at least one success. Thus, pass@1 measures mean task success; large-k pass@k measures whether success remains reachable under repeated sampling. Across a wide range of posttraining recipes, we show that large-k pass@k reveals posttraining differences not captured by pass@1. In particular, checkpoints with similar pass@1 but different large-k pass@k have different posttraining performance. Moreover, we show pass@k peaks at different checkpoints during supervised pretraining, revealing a coverage-reliability tradeoff over the course of training. Based on these observations, we propose two practical interventions: (1) a purely offline checkpoint-selection criterion that balances coverage and reliability without closed-loop evaluation, and (2) a generative data augmentation procedure that expands supervision beyond the finite demonstration set, preserving large-k pass@k while maintaining or improving pass@1 and raising the observed coverage-reliability Pareto frontier. Together, our results motivate evaluation, checkpoint selection, and training procedures that explicitly account for posttraining. Blog post, visualizations, and code are available at https://zhanyisun.github.io/robot-policy-learning-dynamics/.
Introduction
Robot policies are increasingly trained in stages: supervised behavior cloning (BC) or large-scale pretraining [13, 26, 7], followed by reinforcement learning (RL), supervised finetuning (SFT), or other forms of adaptation. Yet supervised pretraining and posttraining are usually studied as separate problems. The checkpoint handed to posttraining is typically chosen by validation loss, average task success, or simply taking the final checkpoint — criteria that evaluate the supervised pretraining stage in isolation. This leaves a central question unresolved: how do the learning dynamics of supervised pretraining shape both direct deployment and downstream finetunability, and what criterion can identify checkpoints that jointly support both?
Language-model research provides strong motivation to look beyond metrics and progress within a single stage of training in isolation. In particular, models with similar pretraining validation loss can perform very differently after posttraining, while prolonged pretraining or stronger SFT performance does not always carry over to subsequent posttraining [30, 44, 23, 51]. Recent work further highlights coverage of high-quality outputs as an important property for downstream posttraining [16, 10].
Robot control, however, presents a different set of challenges than language modeling: policies are densely conditioned on observations, actions are continuous and have far less distributional diversity [37], and success requires stable and precise closed-loop execution Simchowitz et al., 2025; Zhang et al., 2025). These demands may appear to favor increasingly reliable policies over broader behavioral coverage.
Contributions. Despite these differences, we find that supervised training of generative robot policies is governed by a similar tradeoff between broad behavioral coverage and reliable task execution. We study this tradeoff through closed-loop pass@* from a fixed initial condition (henceforth, fixed-start pass@). For each initial configuration, we rollout the stochastic policy times and ask whether at least one rollout succeeds, then average over the distribution of initial configurations (§ 2). Equivalently, pass@ is the expected probability, over the distribution of initial configurations, that at least one of independently sampled rollouts from the same configuration succeeds. At , this recovers the standard mean task-success rate; we refer to this single-rollout success as reliability. As increases, pass@ measures the fraction of initial conditions for which the policy can produce at least one successful rollout within attempts; we refer to this as coverage. The full pass@ profile therefore lets us study how reliability and coverage evolve during pretraining, and how these differences affect posttraining.
We establish two key findings (§ 3). First, large- pass@ tracks downstream posttraining outcomes that pass@1 alone doesn’t: checkpoints with similar pass@1 but different large- pass@ can yield substantially different performance after subsequent SFT or RL finetuning. Second, reliability and coverage diverge systematically during supervised pretraining. Large- pass@ peaks earlier and can decline while pass@1 continues to improve, revealing that further optimization can improve reliability at the expense of coverage. This tradeoff also matters for direct deployment: declining large- pass@ reflects the loss of low-probability successful behaviors on harder initial conditions, even as average task success improves.
Motivated by these findings, we propose a novel offline criterion for selecting policy checkpoints that best balance coverage and reliability (§ 4). Estimating pass@* requires many closed-loop rollouts from matched initial configurations, which is prohibitively expensive on real robots. Instead, our criterion is computed entirely offline from the quantile profile of held-out action error. This profile captures favorable-tail coverage and typical-case reliability, allowing us to select the checkpoint that best balances the two. Crucially, it requires no closed-loop policy evaluation, no deployment-specific choice of , and no tuned tradeoff weight, making it easy to integrate into modern robot policy training pipelines.
Lastly, we introduce a generative data-augmentation procedure that raises the Pareto frontier between mean task success and large- pass@* performance (§ 5). We train a joint observation–action generative model to expand supervision beyond the finite demonstration dataset. The resulting augmentation preserves favorable-tail coverage as pretraining proceeds while maintaining, and in some cases improving, reliability as well. By preserving coverage without sacrificing mean task success, it reduces the need for checkpoint selection and provides stronger initializations for subsequent posttraining.
Problem Setup
Generative robot policies. We study conditional generative policies over action chunks, focusing on diffusion-based policies. Under the noise-prediction parameterization, an expert action chunk is corrupted at noise level , and the denoiser is trained as
At inference, is iteratively denoised using to produce an action chunk.
Pretraining, posttraining, and checkpoint selection. We consider robot policies trained through an initial supervised pretraining phase followed by a subsequent posttraining phase. Along pretraining, optimization produces a sequence of checkpoints , where indexes training progress, measured in epochs or optimization steps depending on the training setting.1 The selected checkpoint may then be deployed directly or used to initialize posttraining. In standard practice, checkpoints are commonly selected using validation loss, held-out action MSE, task success rate from closed-loop evaluation, or simply by taking the final checkpoint. Our central question is how the choice of pretraining checkpoint affects performance under direct deployment and after subsequent posttraining, and how to identify checkpoints that support both.
Fixed-start pass@. In large language model evaluation on verifiable tasks such as code generation and mathematical reasoning, pass@ is the probability that at least one of independently sampled solutions is correct ([11]). For robot policies, we define the analogous closed-loop measure. For an initial condition , let denote the one-rollout success probability of policy checkpoint . Then, we define the pass@ for checkpoint as
That is, for each held-out initial condition, we reset the simulator and execute independent rollouts from checkpoint ; pass@ is the fraction of conditions for which at least one rollout succeeds. Pass@1 recovers the standard mean task-success rate.2
Later Checkpoints Trade Large- Pass@k for Mean Task Success; Pass@1 Can Miss Posttraining Differences
We begin by examining how pass@ curves across evolve over supervised pretraining, and whether posttraining from checkpoints with similar pass@1 but different large- pass@ leads to different performance.
Pretraining evaluation. We study supervised-pretraining dynamics in two settings: (1) single-task BC on MimicGen, and (2) multitask VLA pretraining on RoboCasa. For single-task BC, we train image-conditioned Diffusion Policies ([13]) from scratch on 200 demonstrations for each of five MimicGen tasks ([33]). For multitask VLA pretraining, we use the OpenPI implementation of ([21]) on 65 atomic RoboCasa365 tasks ([36]), with 100 demonstrations per task. Full evaluation details are provided in App. § 18.
For readability, Figure 1(a,Figure 1c) shows only two of the five selected MimicGen tasks together with the aggregate RoboCasa results. Main-text MimicGen experiments use 200 demonstrations per task. Later experiments (§ 5) show all five selected MimicGen tasks and the full RoboCasa evaluation.
Figure 1. Later checkpoints trade large- pass@ for mean task success; checkpoints with similar pass@1 can lead to different posttraining performance. Left: single-task BC followed by residual RL on two representative MimicGen tasks, with (a) showing pass@ during BC and (b) residual RL from different BC checkpoints. Right: multitask pretraining on RoboCasa tasks followed by SFT on target tasks, with (c) showing pass@ during pretraining and (d) downstream SFT. Across (a,c), larger- pass@ peaks and declines earlier than pass@1, so later checkpoints increasingly favor mean task success at the expense of large- pass@ performance. Orange marks the earlier checkpoint selected by our offline coverage–reliability criterion (§ 4), while blue marks the final supervised-training checkpoint; matching dashed lines in (a,c) indicate the initializations used in (b,d). Despite similar pass@1, finetuning from the orange checkpoint achieves stronger performance than finetuning from the blue checkpoint across both settings.
During supervised pretraining, pass@ peaks earlier as increases; mean task success favors later checkpoints. Figure 1(a,Figure 1c) shows a consistent pattern across single-task BC and multitask pretraining: the pass@ optimum shifts earlier as increases. Thus, later checkpoints can achieve higher mean task success even after large- pass@ has begun to decline. This divergence between mean task success and large- pass@ reflects a redistribution of success across conditions during supervised pretraining. Recall that is the one-rollout success probability from condition . Pass@1 depends only on its mean; for , we can write , where is concave in its argument . Shifting success probability from harder conditions toward already-reliable ones can therefore increase average success while decreasing larger- pass@. As supervised pretraining proceeds, condition-wise success can become increasingly polarized: already-solvable conditions become more reliable while harder conditions lose success probability. At the behavioral level, we interpret this redistribution as a tradeoff between reliability and coverage. See App. § 12 for corresponding condition-level decomposition.
Takeaway 1: Over supervised pretraining of robot policies, success can become increasingly polarized across conditions: mean task success can continue to improve or stabilize as large- pass@ declines.
Posttraining evaluation. We show that checkpoints with different pass@ profiles can attain substantially different posttraining performance. For each experiment, we compare the earlier checkpoint selected by our offline coverage–reliability criterion (§ 4), called the selected checkpoint, with the final checkpoint from the same supervised pretraining run. The two have similar pass@1, but the selected checkpoint retains substantially higher large- pass@.
We evaluate four posttraining settings. (1) Filtered BC: we collect the same number of rollouts from both checkpoint, retain successful episodes, append them to the original training set, and train a new BC policy. (2) steering with a learned critic: we train one Q-function per checkpoint on the original training data augmented with unfiltered rollouts from that checkpoint. At held-out state , both critics score the same eight candidate action chunks . We estimate each candidate’s value by branching 16 simulator rollouts from and measuring the fraction that succeed, then evaluate whether correctly ranks candidates by . (3) actor-critic residual RL with DICE-RL (Sun & Song, [45]), an online RL method for finite-horizon diffusion policies. (4) lastly, on RoboCasa365 only, we perform supervised finetuning (SFT) from different pretrained checkpoints using identical target-task finetuning data and optimization. Full experimental details are in App. § 18.
Together, these experiments provide complementary tests of how pretrained-policy coverage affects posttraining. In filtered BC, new supervision comes from successful rollouts (Mirchandani et al., [34]); under a fixed rollout budget, lower coverage can yield a similar number of successes from fewer conditions, reducing data diversity. In critic steering, broader coverage exposes the critic to a wider range of observation–action outcomes, which can improve generalization when ranking actions at unseen observation; such ranking is central to value-guided test-time steering (Nakamoto et al., [35]) and actor–critic RL. Actor-critic residual RL tests whether these differences persist when data collection, critic learning, and policy optimization occur together online. Lastly, SFT verifies the findings persist when target-task data are fixed rather than policy-generated rollouts.
Checkpoints with large- pass@ enjoy stronger posttraining performance despite similar pass@1 performance. Table 1 and Table 2 isolate this effect under online data collection. Across both tasks, retraining on successful rollouts from the selected (higher large- pass@) checkpoint produces a stronger filtered BC policy. Indeed, for matched rollout budgets, a greater chance at success on wider variety of initial conditions is more useful than repeated successes on already-solved conditions. Similarly, critic trained on the selected-checkpoint rollouts also rank counterfactual actions more accurately at held-out states, indicating better value-function generalization.
| Task | Selected | Final |
| Threading (D2) | 0.371 / 0.886 | 0.321 / 0.812 |
| Mug Cleanup (D1) | 0.558 / 0.914 | 0.502 / 0.866 |
Table 1. Filtered BC. Pass@1/pass@32 after retraining on successful rollouts generated by the selected or final checkpoint.
| Task | Selected | Final |
| Threading (D2) | 0.534 | 0.521 |
| Mug Cleanup (D1) | 0.589 | 0.547 |
Table 2. Critic action ranking accuracy. Within-state pairwise accuracy against simulator-estimated action values.
Figure 1(b,Figure 1d) shows that checkpoint selection also affects actor-critic RL finetuning. Under the same interaction budget, residual RL from the selected checkpoint, which has higher-large-, reaches higher task success than from the final checkpoint. In App. § 13.§ 2, we also compare different pretrained checkpoints under OGPO, a full-policy RL finetuning method ([38]). Unlike residual RL, OGPO directly finetunes the pretrained policy rather than learning a residual or steering head on top of a frozen policy. Nevertheless, the selected earlier checkpoint still outperforms the final checkpoint. Lastly, on RoboCasa, SFT from the selected checkpoint is also stronger under identical finetuning data and optimization, showing that the checkpoint effect is not limited to methods trained on policy-generated rollouts.
We note that large- pass@ does not globally rank checkpoints for finetuning: for much earlier checkpoints with lower pass@1, which checkpoint attains higher RL success can depend on finetuning budget and be task-specific. We report results on residual RL finetuning from more pretrained checkpoints in App. § 13.§ 1. We show that, on some tasks, a much earlier checkpoint with lower pass@1 but stronger large- pass@ can eventually outperform the selected checkpoint given a sufficiently large RL finetuning budget. Among checkpoints with comparable pass@1, however, large- pass@ is a better predictor of the RL posttraining performance.
Takeaway 2: Coverage–reliability dynamics during supervised pretraining affect posttraining: checkpoints with similar pass@1 can yield different posttraining performance. Our coverage–reliability criterion selects checkpoints that outperform later checkpoints favored by mean task success and held-out action MSE (App. § 15).
Loss of pass@ performance during training corresponds with overfitting on the empirical dataset, and concentrates on low noise levels. We next examine the learning dynamics behind the late-checkpoint loss of coverage. We find that pretraining initially improves generalization but later increasingly fits the finite training set. In particular, later checkpoints have higher held-out denoising loss near the low-noise end despite lower training loss, with large relative increases where observation support is sparse.
We first test the shift toward training-set overfitting. On Threading D2, we split 1,000 demonstrations into two disjoint 200-demo training sets, and , and a shared 600-demo held-out set, . We train separate and on and , respectively. Let denote the action chunk generated by from observation and initial noise . Figure 2(a) measures prediction disagreement . Lower means the two policies agree more on the same held-out inputs. This quantity initially decreases but increases again during later training, suggesting that each policy increasingly fits its own training split. This is corroborated by Figure 2(b), which measures the ratio between nearest neighbor distances from actions and , divided by those from (see App. § 14 for details). Here, means policy samples are closer to their own training split than held-out expert actions are. As pretraining proceeds,

Figure 2. Independent splits expose dataset-specific fitting. (a) Mean-squared prediction difference between two policies trained on disjoint splits, evaluated on the same held-out inputs with matched diffusion noise; predictions first converge, then diverge. (b) Ratio between the average nearest-neighbor distance from policy samples to their own training split and the average nearest-neighbor distance from held-out expert actions to that same training split; decreasing values indicate increasing bias toward the training split.
decreases for both policies. Taken together, these measurements support increasing fitting to the respective training splits during later training.
These split-data results are consistent with prior work showing that diffusion models shift from generalization toward training-set fitting and memorization under prolonged optimization ([17, 2, 50, 4, 9, 15]). In our image-conditioned robot policies, App. D further shows that later checkpoints have higher held-out denoising loss than earlier checkpoints near the low-noise end, despite having lower training loss at every noise level (Figure 11(a,Figure 11b)). The relative increase in held-out loss is also greater for observations with sparse training support and begins at higher noise (Figure 13). The intuition is that as the noise level decreases, the set of clean actions compatible with a noisy action shrinks; when observation support is sparse, fewer demonstrations also have nearby observations, leaving fewer training examples compatible with both the observation and the noisy action.
Figure 11. Later checkpoints exhibit higher held-out denoising loss near the low-noise end, and changes to the final denoising calls substantially affect pass@ profile. (a) On training demonstrations, -prediction MSE decreases with further training across all noise levels. (b) On unseen demonstrations, later checkpoints have higher denoising loss near the low-noise end than earlier checkpoints despite lower training loss. (c) In a 10-step DDIM sampler, replacing any one of the first four two-step bands (Q1–Q4) of the epoch-800 checkpoint with the corresponding epoch-200 weights has little effect on pass@1 or pass@10. Replacing Q5, the final two calls, restores most of the epoch-200 checkpoint’s pass@10 advantage while reducing pass@1.
Figure 13. Late-training held-out denoising loss increases more at lower diffusion noise levels and with sparser local conditioning support. Relative change in mean held-out -prediction MSE from epoch 200 to epoch 800, (Eq. D.6), for queries divided into equal thirds by local conditioning support. Negative values indicate lower mean loss at epoch 800; positive values indicate higher mean loss. At a fixed noise level, the relative loss change is generally larger for queries with sparser local conditioning support.
Takeaway 3: Supervised pretraining initially improves generalization, but later checkpoints increasingly fit the finite training set. Held-out denoising loss rises near the low-noise end, particularly for observations with sparse training support.
Diagnosing the Coverage–Reliability Tradeoff and Selecting Checkpoints Offline
Having characterized the transition toward finite-data overfitting, we next ask how to identify the coverage–reliability balance before overfitting, without explicitly evaluating pass@ performance through closed-loop policy deployment.
Best-of- action MSE captures the favorable tail but selects too early. Because diffusion denoising loss is often a poor indicator of closed-loop performance ([1]), we use held-out action MSE for offline evaluation throughout the paper ([1]). A tempting offline metric for approximating closed-loop pass@ is then held-out best-of- action MSE. Given a held-out dataset , we sample action chunks at each observation and retain the one closest to the expert action:
Figure 3 shows that larger shifts the optimum of earlier, as it does for pass@. For small , the two optima remain relatively close; as increases, the best-of- minimum increasingly precedes the corresponding closed-loop pass@ peak. This mismatch arises because measures only the favorable tail independently at each observation, whereas closed loop control requires reliable actions throughout an entire rollout trajectory. An early checkpoint can therefore attain low large-- while still assigning substantial probability to poor action samples that destabilize execution. This motivates us to move beyond the minimum over samples and examine the full held-out action MSE distribution.
Figure 3. Best-of- action MSE selects earlier checkpoints than closed-loop pass@. (a) Held-out best-of- action MSE across training checkpoints for different . (b) Closed-loop pass@ success rate across the same checkpoints. Matching colors and marker shapes denote the same value of across panels; hollow markers indicate the minimum MSE in (a) and maximum pass@ in (b). As increases, both optima shift earlier, but the best-of- MSE minimum precedes the corresponding pass@ peak, with a larger separation at higher .
Held-out action-MSE quantiles better reflect coverage and reliability. Best-of- MSE uses only the minimum error among sampled actions. Here, we now examine the full distribution of action MSE under policy samples. For held-out pair and checkpoint , draw , , and let be the -quantile of . We average each quantile across held-out observations,
Figure 4(a) shows a more systematic ordering: lower quantiles reach their minima earlier and then worsen, whereas higher quantiles continue improving later in supervised pretraining. Figure 4(b) illustrates this directly: the early checkpoint can produce lucky draws closer to the expert action, but its samples are substantially more dispersed; the late checkpoint produces more concentrated samples, but its best sample is farther from the expert.
Figure 4. Held-out action-MSE quantiles expose the coverage–reliability balance. (a) Held-out action-MSE quantiles across training, colored by percentile from 0.05 (red) to 0.95 (blue). (b) For a held-out observation, 64 matched-noise samples from ep50 and ep800, plus the expert action, are projected with PCA; rings mark the minimum-MSE sample. ep50 is more dispersed yet has lower best-of-64 MSE, whereas ep800 contracts but its best sample lies farther from the expert. Consistent with (a), lower quantiles favor earlier checkpoints, while higher quantiles improve later. (c) We summarize the lower and upper halves as and and select the checkpoint minimizing ; the selected ep350 matches the closed-loop oracle.
A distributional criterion identifies the coverage–reliability balance. Given the previous findings, we propose to use lower quantiles as an offline measure of coverage, i.e. whether low-MSE action chunks remain accessible under the policy, and higher quantiles as a measure of reliability, how consistently sampled action chunks remain close to the demonstrated action. Based on this, we define the following normalized quantile score for training budget :
Here, means checkpoint has the lowest observed MSE at percentile among checkpoints up to , while means the highest; intermediate values indicate its normalized position between these two extremes. Let denote the discrete set of evaluated quantile levels. To evaluate policy coverage and reliability, we average lower and upper halves of the quantile profile as
using to split the profile evenly without additional tuned threshold. We then select
The criterion therefore selects the checkpoint that minimizes the larger of and . Unlike best-of-, which uses only the minimum, it accounts for both lower and upper quantiles; unlike average validation MSE, it does not collapse the error distribution to a single average. By minimizing the worse of the two scores, the criterion avoids selecting a checkpoint with strong coverage but poor reliability, or vice versa. Lastly, because the criterion is defined relative to the quantile values, it also does not require task-specific calibration over absolute action-MSE thresholds.
Figure 4(c) illustrates the criterion on the 200-demonstration Threading D2 run. Using only offline held-out action MSE, the criterion recovers the closed-loop oracle at epoch 350. We define the closed-loop oracle by applying the same minimax balancing principle across the full pass@k profile. The criterion is also robust to the choice of training horizon once the coverage–reliability dynamics has stabilized: after lower quantiles begin to worsen and higher quantiles saturate, the selected checkpoint remains relatively stable as training continues. Extending training substantially beyond this point only marginally shifts the selected checkpoint (App. § 15). App. § 15 further validates the criterion across MimicGen tasks and dataset sizes, as well as multitask pretraining on RoboCasa.
Takeaway 4: Best-of- action MSE uses only the minimum-error sample and selects checkpoints too early, especially at large . Across the full held-out action-MSE distribution, lower quantiles favor earlier checkpoints while higher quantiles improve later, enabling an offline criterion that balances coverage and reliability without closed-loop evaluation.
Generative Augmentation Preserves Coverage for Downstream posttraining
Although our proposed coverage–reliability criterion (§ 4) selects a favorable compromise without closed-loop evaluation, it is constrained to the observed coverage–reliability Pareto frontier of that run. This raises a more fundamental question: is this tradeoff unavoidable? Can a training intervention give us the best of both worlds by preserving strong large- pass@ while continuing to improve reliability? If so, we can raise the coverage–reliability Pareto frontier.
§ 3 links the loss of coverage to overfitting to the finite demonstration data during policy training. In App. § 14, we show that the held-out denoising score-matching loss increases more sharply late in training for the observation-conditioned model than for its unconditional counterpart. Indeed, Gu et al. [17] show that even conditioning diffusion models on uninformative random labels can substantially exacerbate memorization. This suggests a direct intervention: expand supervision beyond the observed pairs using fresh synthetic observation–action pairs that reflect the population-level expert distribution.
Joint augmentation. To expand supervision beyond observed pairs while preserving observation–action consistency, we propose joint -augmentation (JointAug), which trains a generative model with checkpoints indexed by on the joint demonstration distribution and uses generated pairs for policy training. We save generator checkpoints at fixed intervals, and aim to identify a sampling window after the generator has learned population-level structure but before substantial overfitting. To this end we record each checkpoint’s ratio between generated actions’ nearest-neighbor distances to training expert actions and held-out expert actions (App. § 18.§ 3). A ratio of 1 indicates equal proximity to the two sets, and values below one indicate that generated samples are closer to the training data. We let be the first checkpoint where the ratio falls below one, when generated samples first become closer to training than held-out expert actions. We then sample synthetic pairs uniformly from and the four subsequent checkpoints spaced 100 epochs apart. Because checkpoints are saved relatively densely over this stage of generator training, the five-checkpoint window spans only the transition from population-level generalization toward early fitting of the empirical training set. We use five checkpoints as a simple default heuristic across tasks rather than tuning the window for each task. App. § 16.§ 2 shows that performance is robust to the exact generator checkpoint selection.
Then policy training uses equal amounts of real and generated supervision:
where is the uniform mixture of the five selected generator checkpoints. Real samples retain the clean expert targets; generated pairs expand supervision beyond the finite data. Because observations and actions are generated jointly, their correspondence is naturally preserved.
To evaluate JointAug, we compare against baseline Diffusion Policy and posterior behavior cloning (PostBC, Wagenmaker et al. (2025a)) across the full training horizon. PostBC aims to improve policy coverage by estimating observation-conditioned action covariance with a bootstrapped ensemble and perturbing action targets at clean observations . In that regime, the perturbed action is not constrained to remain compatible with the clean observation. In contrast, JointAug ensures is learned from the correct joint distribution (and thus, is consistent).
In the first five panels of Figure 5, we compare the methods across MimicGen tasks. JointAug achieves higher pass@10 than both PostBC and the baseline DP run and prevents the baseline’s late-training loss of coverage. Its pass@1 matches the baseline and sometimes improves it. PostBC also improves pass@10, but on some tasks lowers pass@1, likely because its perturbed actions can be incompatible with the clean observations. Moreover, unlike PostBC, which perturbs only action targets, JointAug generates new observation–action pairs to further mitigate the conditional overfitting observed in App. § 14. App. § 16.§ 3 analyzes the sampled action distributions. JointAug preserves the favorable low-error tail and reduces typical-sample error, helping explain its gains in both coverage and reliability. JointAug also provides checkpoints with stronger residual-RL initializations than the final baseline checkpoints (Figure 6), often matching or exceeding the checkpoint selected by our coverage–reliability criterion from baseline run. By preventing the late-training loss of coverage, JointAug also reduces the need for post hoc checkpoint selection. We further find that its performance is robust to the choice of generator checkpoint; more ablations and baseline comparisons are in App. § 16.

Figure 5. During pretraining, generative augmentation preserves coverage while maintaining, and in some tasks improving, reliability (pass@1). Across most MimicGen tasks and RoboCasa, Baseline DP loses pass@10 at later checkpoints. JointAug prevents this loss of coverage while maintaining or improving pass@1. PostBC also preserves pass@10 better than Baseline DP, but can do so at the cost of pass@1.
Figure 6. Generative augmentation produces stronger checkpoints for posttraining. Columns correspond to Figure 5. We compare posttraining from the baseline checkpoint selected by our coverage–reliability criterion (§ 4), the final Baseline DP checkpoint, and the final checkpoint from our generative augmentation run. Across MimicGen tasks, the final generative-augmentation checkpoint substantially outperforms the final baseline and often matches or exceeds the selected checkpoint. On RoboCasa, it matches the selected checkpoint on both pass@1 and pass@32.
A scalable action-only variant. For large-scale VLA pretraining, joint generation is expensive, so on RoboCasa we use a lightweight action-only variant, similar in spirit to PostBC but using the policy’s own action variation rather than a bootstrapped ensemble for covariance estimation, thereby avoiding the need to tune the perturbation scale. After a standard pretraining run, we select using our coverage–reliability criterion and estimate action covariance from multiple policy samples at each training observation. We then resume pretraining from with perturbed targets , where . This injects perturbations calibrated to the action spread at without shifting each training target in expectation, . Figure 5 and Figure 6 show that the variant preserves large- pass@k as pass@1 improves, and achieves SFT performance comparable to the selected early checkpoint.
Takeaway 5: Generative augmentation can preserve coverage while allowing reliability to improve, producing late-pretraining checkpoints with stronger pass@ profiles and better downstream posttraining than standard final checkpoints.
Coverage–Reliability Dynamics On a Real Robot
To test whether the coverage–reliability dynamics also arise in policies trained on real-robot data, we study a challenging variant of Push-T. We use only a wrist-mounted camera for visual observations and allow full 6-DoF end-effector control. The end-effector manipulates the T-block only through non-prehensile contact. We collect 740 human demonstrations, using 680 for training and 60 for held-out validation, and train a Diffusion Policy for 200 epochs with an action chunk length of 16. Experiment details are in App. H.8. The best-of- validation action MSE and error quantiles exhibit the same coverage–reliability tradeoff observed in simulation (Figure 7(b,c)). Applying our coverage–reliability balance criterion (§ 4) selects the checkpoint at epoch 46 (Figure 7(d)). We compare the selected checkpoint against the final checkpoint at epoch 200 using five rollouts from each of 20 fixed initial states. The selected checkpoint outperforms the final checkpoint by 10% on pass@1 and 20% on pass@5 (Figure 7(e)).
Figure 7. Real-robot Push-T. (a) Task setup. (b) Best-of- validation action MSE across training epochs. (c) Quantiles of validation action error across training epochs. (d) Normalized coverage and reliability scores with the selected checkpoint at epoch 46. (e) Pass@1 and pass@5 success rates of the selected (epoch 46) and final (epoch 200) checkpoints over 20 fixed initial states.
To investigate whether differences in policy coverage affect downstream model learning, we train two action-conditioned world models on the same demonstration data, augmented with 200 rollout trajectories from either the selected early checkpoint or the final checkpoint. We include all rollouts regardless of task success for world model training. For held-out validation, we collect another 100 rollout trajectories from intermediate checkpoints, with 20 trajectories each from epochs 75, 100, 125, 150, and 175.
The world model uses a frozen Stable Diffusion VAE to encode each image frame into a latent. We train a spatio-temporal DiT from scratch and inject action conditioning additively at each frame. As in policy training, we use an action chunk length of 16. Given two history frames and a 16-step action chunk, the model predicts the next four frames. At evaluation, we first predict four frames from two ground-truth history frames and the corresponding action chunk. We then predict the following four frames autoregressively from the model’s own predictions and the next action chunk. We average PSNR and LPIPS over the four predicted frames in each chunk. As shown in Table 3, the world model trained with selected-checkpoint rollouts achieves better prediction quality than the model trained with final-checkpoint rollouts. This result suggests that preserving policy coverage can benefit downstream world-model learning. Additional visualizations of the world model’s prediction are in App. H.8.
| Rollouts | PSNR↑ | LPIPS↓ |
| Chunk 1 | ||
| Selected | 27.9 | 0.112 |
| Final | 26.8 | 0.120 |
| Chunk 2 | ||
| Selected | 25.2 | 0.136 |
| Final | 24.1 | 0.145 |
Table 3. World-model prediction quality. Models are trained on demonstrations and rollouts from the selected or final checkpoint. Chunk 1 is predicted from two ground-truth context frames; Chunk 2 is predicted autoregressively from the model’s Chunk-1 output.
Abridged Related Work
Due to space constraints, we discuss key related work here and defer more comprehensive review to Appendix A. In language models, coverage of high-quality outputs has emerged as an important property for efficient posttraining and test-time scaling [16, 10]. Modern generative robot policies [13, 26, 7] are likewise followed by supervised finetuning, RL, or inference-time steering (Kim et al., 2025; Intelligence et al., 2025a; 2026), yet few works study how properties acquired during pretraining shape these later stages. PostBC and TMRL explicitly modify pretraining to improve subsequent RL by preserving action coverage or exploration [46, 19]. We instead study how coverage and reliability change during standard supervised pretraining itself and how they affect posttraining. Mirroring work showing that diffusion models can transition from generalization toward finite-sample fitting over training [9, 15], we characterize this learning dynamic in robot policies and connect it to closed-loop coverage and posttraining.
Conclusion and Future Work
This work studies how coverage–reliability dynamics during supervised robot-policy pretraining affect posttraining performance, and develops practical methods for offline checkpoint selection and coverage-preserving pretraining. An important direction for future work is to jointly develop principled data collection strategies and improved pretraining methods: demonstration datasets should have both broad coverage of task conditions and sufficient local density, and training objectives, architecture designs, and optimization strategies should improve reliability without sacrificing coverage or generalization. Another future direction is to develop a deeper theoretical understanding of how policy distributions evolve during pretraining and how these changes affect subsequent posttraining, which could guide both data collection and policy optimization. More broadly, our findings motivate a unified view of robot-policy pretraining and posttraining, in which pretrained policies are evaluated not only by their immediate performance but also by how effectively they can be improved through posttraining.
Acknowledgements
The authors thank all members of REALab and Huy Ha for their feedback on an earlier draft of this manuscript, and Jack Joler for data collection for the real-robot experiments. ZS is supported by the Stanford Graduate Fellowship. This work was supported in part by the NSF Award #2143601, #2037101, and #2132519. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
AI Use Statement
We used generative AI tools to polish the writing, search for relevant literature, and help with parts of the code implementation. We did not use them to generate data, derive mathematical results, develop research ideas, or interpret experimental results. The authors checked all AI-assisted work: we tested the code, verified every cited paper against its source, and confirmed the text accurately reflects our contributions. We take full responsibility for the content of this paper.
Related Work
Pretraining–posttraining Interactions in Robot Learning
Language-model research provides strong motivation for studying pretraining and posttraining jointly. Recent work shows that performance within one stage in isolation can poorly predict the outcomes after the next. Models with similar pretraining validation loss can differ substantially in their finetuning performance [30], and prolonged pretraining can reduce subsequent finetuning performance [44]. In SFT–RL pipeline, higher SFT performance does not always predict stronger post-RL performance; stronger SFT checkpoints can even perform worse after the same RL procedure [23, 51]. Recent theory instead highlights coverage, the probability mass assigned to high-quality outputs, as an important property for test-time sampling [16, 10]. Empirically, large- pass@ can predict post-RL performance better than SFT task performance [23]. TailSFT similarly improves posttraining by modifying SFT to preserve broader output coverage [31]. Robot learning increasingly follows a similar staged pipeline: BC or pretraining [13, 26, 7] followed by SFT [27, 6, 36], RL [21, 47], or inference-time steering [48, 12] (Intelligence et al. 2026). Yet few works study how properties learned during pretraining affect posttraining. PostBC proves that standard BC can fail to guarantee coverage of the demonstrator’s actions, which is necessary for RL finetuning to reach the demonstrator’s performance. It instead models the posterior over demonstrator action to guarantee such coverage and improve RL finetuning [46]. TMRL broadens coverage during pretraining by adding noise to the policy context, then lets RL control the noise level during finetuning [19]. Both methods redesign pretraining with noise injection specifically for downstream RL finetuning. We instead study how a policy’s finetunability changes over the course of standard supervised pretraining. We identify an evolving coverage–reliability tradeoff, show that it affects multiple forms of posttraining, and use this to design offline checkpoint selection criterion and training intervention.
Generalization and Memorization in Diffusion Models
Finite-data diffusion objectives admit closed form solutions that reproduce the training set, and several works characterize how model capacity, dataset size, conditioning, and optimization determine whether trained models approach these memorizing solutions [17, 2, 50, 5]. Recent papers further show a temporal separation in training dynamics: models first learn population-level structure that generalizes beyond the training set, while prolonged optimization can increasingly fit individual training examples [9, 15]. Related work on flow matching likewise studies why learned models generalize despite an empirical optimum that memorizes the data [4]. These findings have motivated interventions including early stopping, limiting model capacity [2, 50, 15], training from corrupted examples [14, 39], and perturbing conditioning input [43]. Most of this literature studies unconditional, class-conditioned, or text-conditioned generation of independent samples. Robot policies instead condition action generation on dense, high-dimensional observations and repeatedly sample the policy for multi-turn closed-loop interaction. He et al. [18] adopt a static view of trained diffusion policies and argue that, on small robot datasets, they largely retrieve actions from nearby training examples. We instead study how the policy distribution dynamically evolves over optimization, how this evolution changes closed-loop reliability and coverage and posttraining.
Robot Policy Evaluation and Model Selection
Closed-loop rollouts remain the standard for evaluating robot policies, but reliable comparisons require many trials and careful statistical treatment [28, 3, 42]. This cost motivates offline metrics for policy development and checkpoint selection. Mandlekar et al. [32] showed that validation loss can select substantially worse BC checkpoints than rollout evaluation. More recently, ABC found that diffusion validation loss was poorly correlated with real-world performance, whereas validation action error correlates more strongly with it [1]. Other work proposes manifold adherence [37] or task-critical action error [20] as more informative offline signals. Our setting differs because stochastic generative policies can have similar average error while differing substantially across repeated sampling. We therefore evaluate ate the full distribution of held-out action errors and select checkpoints by balancing coverage and reliability with a special focus for on their use in downstream posttraining.
Noise Injection in Robot Policies
Noise injection and local data augmentation methods have long been used to reduce covariate shift in behavior cloning. DART injects action noise during expert collection so demonstrations cover states reached after action perturbations [29]. Other work perturbs existing demonstrations and synthesizes corrective labels [24], uses noise augmentation to improve closed-loop stability [8], or generates corrective data under local continuity assumptions [25]. Recent theory shows that expert action noise can mitigate exponential compounding error in continuous-control imitation learning [52]. PostBC [46] and TMRL [19] similarly inject noise into actions or observation conditioning to broaden coverage for downstream RL. Our joint -augmentation instead learns the joint observation–action distribution from the demonstrations and generates new pairs together. Thus, the dataset itself determines both the observation variation and its compatible action, without additional expert collection, a hand-designed external noise distribution, or a local analytical dynamics model.
Success Polarization Across Evaluation Conditions
§ 3 showed that evolves asynchronously during supervised pretraining, and attributed this behavior to a redistribution of success across initial evaluation conditions. Here, we examine this redistribution directly at the condition level and make explicit why it causes larger- performance to decline first.
For the 200-demonstration Threading-D2 experiment, we evaluate every checkpoint on the same 500 evaluation conditions, with 32 rollout attempts per condition (). For each condition, we count the number of successful rollouts and track how the distribution of these counts changes over the course of supervised training.
Late training polarizes success across conditions. Figure 8 makes the redistribution described in § 3 explicit. Early in training, the improvement is largely uniform: the fraction of conditions with zero successes decreases and more conditions achieve higher success counts. Later, however, this trend reverses for the hardest conditions. Later in training, both the fraction of conditions with zero successes and the fraction with high success counts increase. Therefore, late training polarizes condition-wise success, improving reliability on a subset of conditions while reducing coverage on others.
Figure 8. Distribution of condition-level success counts over supervised pretraining. For each checkpoint, we run pass@32 on each of 500 evaluation conditions and bin each condition by its number of successful rollouts within 32 trials. The stacked bands show the fraction of conditions in each success-count bin; each vertical slice sums to one. During late training, the fraction of conditions with zero successes increases, while the fraction with high success counts also increases.
Why polarization produces asynchronous profiles. Let denote the one-rollout success probability for condition at checkpoint . Then
averages linearly. For , is concave. Thus, shifting success probability from a lower- condition to a higher- condition while holding mean success fixed leaves unchanged but lowers .
As increases, the factor suppresses changes on high-success conditions more strongly. Large- therefore depends more on whether low-success conditions remain solvable. Once those conditions begin to degrade, large- begins to decrease, even if small- continue to improve from improvements on already-reliable conditions.
Additional Results on RL Finetuning Across the Coverage–Reliability Tradeoff
Additional Results on Residual RL
Additional Results on Residual RL.
To examine more broadly how pretraining coverage and reliability affect residual RL finetuning (§ 3), we additionally finetune a substantially earlier checkpoint that retains more coverage but has lower than the checkpoint selected by our criterion (§ 4).
Figure 9 shows that the relative advantage of this much earlier checkpoint depends on the available online budget. Its lower can hurt early residual RL performance, while its greater coverage can become beneficial with more online budget; the exact tradeoff can be task dependent. While we do not claim that our criterion selects the optimal checkpoint for any online RL budget, it instead selects a checkpoint with substantially higher large- than the final checkpoint while maintaining comparable ; and across the evaluated budgets, this checkpoint consistently finetunes better than the final checkpoint from the same supervised run.
Figure 9. Residual RL finetuning across the coverage–reliability tradeoff. We additionally finetune an earlier baseline checkpoint (ep.200/100/100 for Threading/Hammer Cleanup/Mug Cleanup) with lower pass@1 and higher coverage than the checkpoint selected by our criterion. Its performance relative to later checkpoints depends on the available online budget.
The exact ordering of checkpoints after RL finetuning can depend on both the RL algorithm and the task dynamics. Different RL methods explore and use online experience differently, and task dynamics affect how quickly differences between pretrained policies are amplified or corrected during closed-loop interaction. Our experiments establish the online rollout budget-dependent ordering of checkpoints for DICE-RL; we believe similar effects could also arise in other residual- or steering-based RL methods. Large- should therefore be interpreted as an offline measure of how broadly a pretrained policy succeeds before finetuning, rather than as a universal predictor of checkpoint ordering after arbitrary RL finetuning procedures.
Full-policy Finetuning
Full-policy Finetuning. While DICE-RL directly depends on the quality of actions sampled from the pretrained policy, we further investigate whether checkpoint selection matters when RL can update the entire action generator. We adopt Off-Policy Generative Policy Optimization (OGPO) [38] and use its official implementation. We first pretrain flow-matching policies with behavior cloning (BC). During RL finetuning, OGPO learns action-value critics through off-policy TD learning and updates the full generative policy using a PPO-style objective over sampled action-generation trajectories. All experiments use low-dimensional state observations and 200 demonstrations per task. We increase the critic warmup budget to steps, since shorter warmup leads to substantially slower or stalled learning on MimicGen tasks. One possible explanation is that these tasks make accurate value estimation particularly challenging, although we do not fully isolate the cause of this sensitivity here.
Figure 10(a,Figure 10b) shows that flow-matching BC policies exhibit a similar coverage–reliability trade-off: best-of- validation action MSE increases for large and large- declines later in training. Notably, this observation occurs even with a relatively small five-layer MLP action network containing approximately 1.17M parameters, suggesting that the phenomenon is not limited to policies with high-capacity networks. Figure 10(c,Figure 10d) shows the evolution of validation error quantiles and the coverage–reliability criterion used for checkpoint selection.
Figure 10. Flow-matching policy pretraining and OGPO finetuning. Each row shows a MimicGen task with 200 demonstrations. (a) Best-of- validation action MSE. (b) pass@ during BC pretraining. (c) Validation error quantiles. (d) Coverage–reliability scores and selected checkpoint. (e) OGPO finetuning from the selected versus final BC checkpoint. The selected initialization generally achieves better downstream RL performance.
Figure 10(e) compares OGPO finetuning from the selected and final BC checkpoints. The selected checkpoint achieves better performance on four of the five tasks. This result suggests that the effects of policy coverage and checkpoint selection extend beyond RL methods that directly rely on sampling and steering: even when the entire action generator is finetuned, the selected checkpoint can provide a more favorable initialization for subsequent RL. For OGPO in particular, a possible explanation is that it computes policy updates from action-generation trajectories sampled under the current policy. An initialization that places greater probability mass on useful actions may therefore generate more informative advantage-weighted updates, whereas actions that are rarely sampled initially may be difficult to reinforce under a finite interaction budget.
Diagnosing Overfitting in Diffusion Policies
The main text establishes that extended supervised pretraining can improve (reliability) while reducing large- (coverage). Here, we examine how diffusion policies fit finite demonstrations over optimization time and across diffusion noise levels. We find that training denoising loss continues to decrease across noise levels, while held-out loss increases at later checkpoints toward the low-noise end. This deterioration is more pronounced in regions with sparse observation support. Furthermore, comparison with action-only diffusion trained on the same data shows that observation conditioning leads to greater late-training deterioration in held-out denoising loss, and such deterioration begins at higher diffusion noise levels. The final two low-noise denoising calls also appears to account for most of the difference in closed-loop performance between early and late checkpoints.
Held-out denoising deteriorates near the low-noise end. For checkpoint and diffusion noise level , define the -prediction MSE as
where , , and the expectation over is evaluated separately on training and unseen demonstrations. Figure 11(a,Figure 11b) show this loss across diffusion noise levels for checkpoints at different training epochs, evaluated on training and unseen demonstrations, respectively. On training demonstrations, later checkpoints have lower MSE than earlier checkpoints at every noise level (Figure 11(a)). On unseen demonstrations, however, this ordering reverses at the low noise end: later checkpoints have higher MSE than earlier ones at the same noise levels (Figure 11(b)). This train–held-out divergence indicates overfitting at the low noise end of the diffusion process. Since , expressing the error in clean-action MSE units preserves the checkpoint ordering at each fixed noise level, although it changes the absolute loss magnitudes across noise levels.
Independent data splits show fitting to the training split. Here we explain the data split experiment in main text § 3 in full details. We train two policies, and , on disjoint demonstration sets and , and evaluate both on the same held-out set with matched initial diffusion noise. Their predictions on first become more similar, then diverge as training continues. To test whether this late divergence reflects each policy moving toward its own training split, define the nearest-neighbor action distance
and the normalized ratio
Here is an action chunk sampled from given held-out observation and initial diffusion noise , and denotes the demonstrated action chunk of a training pair in . The numerator measures its distance to the policy’s training split ; the denominator measures the corresponding distance for held-out expert actions. Thus means that policy samples lie closer to than held-out expert actions do. A decrease in indicates increasing relative proximity to the policy’s training split. As shown in Figure 2(b), decreases later in training for both policies. Together with their increasing prediction divergence on in Figure 2(a), this supports increasingly split-specific fitting.
Observation conditioning exacerbates late-training overfitting. To examine the role of observation conditioning, we compare the image-conditioned denoiser with an action-only denoiser trained on the same demonstration action chunks. Here and denote the parameters at training epoch . The action-only model uses the same denoising backbone and training settings but receives no observation input. We evaluate both models on identical noisy actions from held-out demonstrations. Let denote the held-out -prediction MSE at noise level for . We measure the relative change from epoch 200 to 800 as
The ratio is unchanged if -prediction MSE is converted to clean-action MSE at the same noise level.
Figure 12 shows that changes from negative to positive as noise decreases for both models. Continued training therefore improves held-out denoising at high noise but worsens it near the clean end. The reversal occurs at higher noise for the conditional model, which also shows substantially greater deterioration toward low noise. One possible explanation is that as noise decreases, the noisy action becomes more informative about individual training actions. Observation provides an additional identifier for training examples, potentially enabling memorization when the action is more heavily corrupted.
Figure 12. Observation conditioning exacerbates the deterioration in held-out denoising during late training. Relative change in held-out -prediction MSE from epoch 200 to 800 for the image-conditioned and action-only models, evaluated on the same noisy action chunks. Negative values indicate lower held-out loss at epoch 800; positive values indicate higher loss. The conditional model exhibits held-out loss deterioration at higher diffusion noise levels.
We next examine how the conditional model’s late-training deterioration depends on the density of training observations near a held-out query. We approximate this density using the ground-truth simulator state because the policy is image-conditioned. For a query state and training episode , define
Let be the 10th-smallest across the 200 training episodes. We divide held-out queries into dense, medium, and sparse thirds according to , with larger values indicating sparser support.
For each group , let be the conditional model’s mean held-out -prediction MSE over queries in that group. Using the same relative-change measure as Eq. D.4, we compute
Figure 13 shows that the reversal from improvement to deterioration starts from higher noise levels for sparsely supported observations than for densely supported ones. Near the clean end, the relative increase in held-out loss is also generally larger in the less-supported groups. Thus, the noise range in which continued training harms generalization depends on how well the observation is covered by the training demonstrations.
Local-support analysis. To relate this pattern to finite-data support, we count training examples compatible with a denoising query in both observation conditioning and action space.
Assume the training windows are i.i.d. samples from a joint density that is continuous near the held-out query. Here is a conditioning representation and is the clean expert action. Take an independent held-out pair and one initial diffusion noise draw . At diffusion time ,
For a candidate clean training action , the forward diffusion process gives
Eq. D.9 shows that the likelihood depends on . An action at distance from has relative likelihood . Therefore, for any fixed relative-likelihood threshold, the radius of the corresponding set of clean actions is proportional to . As decreases, this set shrinks around .
We now introduce a simple local-support model. Fix a radius in conditioning space, and replace the smooth Gaussian weighting in Eq. D.9 with a hard action-space cutoff at . We then count training examples that satisfy both conditions:
Thus, an example is counted if its conditioning representation lies within distance of and its clean action lies within distance of . The cutoffs define an analytical neighborhood, not which examples the network actually uses.
Let be the probability that one i.i.d. training pair is counted by Eq. D.10. Then
It follows that and .
To evaluate , substitute and :
Near the low-noise end, and . In the joint limit , assuming is continuous and positive at , Eq. D.13 gives
where is the volume of the unit ball in .
Factoring gives
with . Sparser local conditioning support lowers the first factor; lower diffusion noise reduces and hence the second factor. Both lead to fewer locally compatible training examples. This reduced support is consistent with greater vulnerability to finite-data overfitting, but the calculation itself does not predict when a trained denoising network’s held-out loss will increase.
The measure used in Figure 13 is computed in simulator-state space, rather than the policy’s learned conditioning representation, and serves as a proxy for local conditioning support.
Low-noise changes affect closed-loop pass@. To test whether changes to low-noise denoising affect closed-loop success, we divide the 10 DDIM inference calls into five consecutive two-call bands, Q1–Q5, and replace the late checkpoint’s weights with the early checkpoint’s weights for one band at a time. Patching Q1–Q4 has little effect on pass@1 or pass@10, whereas replacing Q5 recovers most of the earlier checkpoint’s pass@10 advantage while reducing pass@1 (Figure 11(c)). Thus, for this checkpoint pair, the final two calls account for most of the observed differences in the pass@ profile.
Full Coverage–Reliability Results and Checkpoint Selection
We report the complete results across five MimicGen tasks with 200 and 500 demonstrations, together with multitask RoboCasa pretraining. For each setting, we show best-of- validation loss, closed-loop pass@, the full error-quantile profile, coverage–reliability checkpoint selection, and downstream residual RL or supervised finetuning.

Figure Hammer Cleanup D1, 200 demos. Hammer Cleanup D1 (200 demos). (a) Best-of- val error. (b) Pass@ success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.

Figure Hammer Cleanup D1, 500 demos. Hammer Cleanup D1 (500 demos). (a) Best-of- val error. (b) Pass@ success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.

Figure Mug Cleanup D1, 200 demos. Mug Cleanup D1 (200 demos). (a) Best-of- val error. (b) Pass@ success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.

Figure Mug Cleanup D1, 500 demos. Mug Cleanup D1 (500 demos). (a) Best-of- val error. (b) Pass@ success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.

Figure Square D2, 200 demos. Square D2 (200 demos). (a) Best-of- val error. (b) Pass@ success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.

Figure Square D2, 500 demos. Square D2 (500 demos). (a) Best-of- val error. (b) Pass@ success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.
The same learning dynamics hold across tasks and dataset sizes. Figure 14 and Figure 15 show a consistent pattern across all settings. Best-of- loss and pass@ favor increasingly earlier checkpoints as increases, while the error-quantile curves exhibit the same tradeoff between favorable-tail coverage and average-case reliability as described in § 4. The pattern holds for both single-task MimicGen BC and multitask RoboCasa pretraining. Importantly, pass@1 and =1 best-of- loss (mean held-out action MSE) generally favor later checkpoints after large- pass@ has begun to decline.
Figure 14. Full MimicGen results across tasks and dataset sizes. Results for all five tasks with 200 and 500 demonstrations. Each row reports best-of- validation loss, closed-loop pass@, the full error-quantile profile, coverage–reliability checkpoint selection, and downstream residual RL finetuning.
Figure 15. Full RoboCasa multitask pretraining results. Best-of- validation loss, pass@, error quantiles, coverage–reliability checkpoint selection, and downstream SFT.
Increasing the MimicGen dataset from 200 to 500 demonstrations generally attenuates, and in several cases delays, the late-stage loss of coverage. While two discrete dataset sizes are insufficient for a scaling-law claim, this trend is consistent with the finite-data overfitting diagnosis in Appendix § 14.
Coverage–reliability balance criterion agrees with a closed-loop oracle. We apply the coverage–reliability criterion from § 4 to every setting. For comparison, we define the closedloop oracle directly from the full pass@k table. Here we describe how we compute the checkpoint selected from oracle closed-loop pass@k runs. For each , we normalize pass@k across checkpoints as
where denotes pass@k at checkpoint . We then select
This oracle selects the checkpoint whose worst normalized gap from the best checkpoint is smallest across the full pass@k profile. Across tasks and dataset sizes, the checkpoint selected by our proposed offline criterion lies near the closed-loop oracle.
Selected checkpoints also support stronger finetuning. For all MimicGen settings, residual RL initialized from the selected checkpoint outperforms RL from the final BC checkpoint. RoboCasa shows the same pattern under SFT: initialization from the selected pretraining checkpoint yields stronger pass@1 and pass@32 throughout finetuning. These results extend the main-text finding that the coverage–reliability balance retained during supervised training leads to stronger finetuning performance compared to the final checkpoint from the same run.
Robustness to Generator Checkpoint Selection
Selection is robust to training horizon. We additionally extend the 200-demonstration Threading D2 run from 800 to 1600 epochs. Figure 16 shows that the same pass@k and error-quantile dynamics persist under substantially longer training, while the criterion continues to select an intermediate checkpoint.
Figure 16. Extended training on Threading D2. The coverage–reliability dynamics persist to 1600 epochs, and the offline criterion continues to select an intermediate checkpoint.
We further recompute both selectors using only checkpoints available up to each training horizon . Figure 17 shows that the offline-selected checkpoint closely tracks the corresponding closed-loop oracle as additional checkpoints become available, rather than simply moving toward the end of training. The selection criterion therefore does not depend sensitively on the chosen training horizon.

Figure 17. Robustness to training horizon. Offline selection tracks the closed-loop oracle as increases.
Additional Results on JointAug
Additional Task
We additionally evaluate JointAug on 3-Piece Assembly (D2) with 500 demonstrations. This task provides a more extreme example of the effect of pretraining coverage on residual RL finetuning. As shown in Figure 18, JointAug raises pass@10 to roughly 0.8 during BC pretraining, compared with below 0.1 for the baseline. Moreover, residual RL from the final JointAug checkpoint then reaches over 0.8 success, while finetuning from the baseline checkpoint remains near zero throughout the evaluated interaction budget. Thus, on this task, the pretraining difference changes residual RL from not working to successful finetuning.
Figure 18. JointAug enables residual RL finetuning on 3-Piece Assembly (D2). Left: JointAug substantially improves pass@1 and pass@10 during BC pretraining. Right: residual RL from the final JointAug checkpoint reaches over 0.8 success, while finetuning from the baseline checkpoint remains near zero.
Robustness to Generator Checkpoint Selection
By default, JointAug uses a uniform mixture of five generator checkpoints, spaced 100 epochs apart, around the transition identified by our train–held-out nearest-neighbor ratio (Appendix § 18.§ 3). We use this short checkpoint window to average supervision across the generator’s transition from population-level generalization toward increasing training-data bias.
To test how sensitive JointAug is to this heuristic choice, we compare the five-checkpoint mixture against distilling from individual generator checkpoints from the same region. Figure 19 shows that the resulting policy curves are similar across these choices on all four tasks. In particular, single generator checkpoints preserve most of the pass@10 improvement over the baseline. The five-checkpoint mixture performs similarly overall and gives slightly higher pass@1 on Mug Cleanup and Threading. Together, these results show that JointAug does not depend strongly on selecting one particular generator checkpoint or checkpoint mixture. Mixing several nearby checkpoints provides a simple way to reduce checkpoint sensitivity.
Figure 19. JointAug is relatively robust to generator checkpoint selection. We compare the five-checkpoint mixture used in our main experiments against individual generator checkpoints from the same run. Across four tasks, single-checkpoint variants have similar pass@10 gains over the baseline, while the mixture performs similarly overall and slightly improves pass@1 on Mug Cleanup and Threading.
Visualizing How JointAug Changes the Action Distribution
To understand why JointAug improves both coverage and reliability, we inspect the action distribution at held-out observations. For each state, we sample 64 action chunks from an early base line checkpoint (ep50), the final baseline checkpoint (ep800), and the final JointAug checkpoint (ep800), and jointly project the samples with PCA. We report both the mean action MSE and the best-of-64 MSE to the expert action. The former measures the accuracy of a typical sample, while the latter measures whether the sampled distribution still contains an action close to the expert.
Figure 20 shows that the early baseline checkpoint produces a broad action distribution: across the four states, best-of-64 MSE is very small, but mean MSE can go up to . Thus, lucky draw samples lie very close to the expert, while the majority of samples are much farther away. This provides coverage but poor reliability for stable closed-loop execution.
Figure 20. JointAug retains favorable-tail coverage while improving typical action accuracy. For four held-out Mug Cleanup states, we jointly project 64 action samples from baseline ep50, baseline ep800, and JointAug ep800 with PCA. Stars mark expert actions and rings mark the minimum-MSE sample. The baseline ep50 is dispersed with low best-of-64 but high mean MSE; baseline baseline ep800 contracts and lowers mean MSE but worsens best-of-64 MSE; JointAug ep800 achieves best of both worlds.
By ep800, the baseline distribution contracts substantially. Mean MSE improves a lot from the early checkpoint, but best-of-64 MSE worsens on all four states. The later baseline therefore produces more consistent samples, but its best samples are farther from the expert, thus losing coverage.
The final JointAug policy checkpoint combines the favorable properties of both checkpoints: its action clouds remain broader than the final baseline while being centered closer to the expert. Mean MSE falls further to –, and best-of-64 MSE decreases to –. Thus, JointAug achieves lower mean and best-of-64 error than the final baseline on every state, while nearly matching or improving the favorable tail of the early baseline.
Together, these visualizations provide a post-hoc explanation for the pass@1 and large pass@ improvement from JointAug. Jointly generated pairs expand supervision beyond the finite demonstrations while preserving observation–action correspondence. The resulting policy retains a broader set of low-error actions without the large average errors of the early baseline.
Additional Baseline - Context-Smoothed Pre-training in TMRL
We additionally compare against the context-smoothed pretraining part from TMRL [19], which is designed to increase action coverage by reducing the amount of information available in the policy context. During pretraining, TMRL applies a forward diffusion noise to the observation context,
then conditions the policy on both the corrupted context and its corresponding noise level . The intuition is that increasing aliases a wider neighborhood of contexts, so the corresponding action distribution can borrow actions from other demonstrations rather than fitting only the conditional distribution at the original observation. We tune the maximum context-noise level for each task and report the best-performing setting; full implementation details are in Appendix § 18.§ 5.
Figure 21 shows that context smoothing does not preserve coverage well in our setting. Across tasks, its pass@10 remains below JointAug and often declines substantially with further training; its pass@1 also tends to degrade
Figure 21. Context-Smoothed Pre-training from TMRL does not preserve coverage under prolonged training. Across five MimicGen tasks, TMRL generally underperforms JointAug in pass@10 and often loses both pass@10 and pass@1 at later checkpoints.
One likely reason is that context smoothing relies on local structure in the latent space where noise is applied. Isotropic Gaussian perturbations are useful when nearby context representations correspond to observations with compatible actions, since smoothing can then borrow coherent behaviors from neighboring contexts. In our image-based diffusion policies, however, this assumption may not hold. The observation encoder is trained jointly from task demonstrations and is not explicitly regularized so that Euclidean neighborhoods in its latent space preserve action similarity. Context corruption can therefore mix observations that require different actions. We note that differs from the image-based experiments in Hong et al. [19], which smooth pretrained VLM embeddings.
More generally, context smoothing increases overlap between conditional action distributions, but by design does not directly constrain the favorable tail of the action distribution as supervised training proceeds. It therefore does not guarantee preservation of large- pass@k.
Additional Results and Visualizations
Sampling Configuration Ablations
Our main experiments use DDIM sampling with 10 denoising steps and . The coverage–reliability tradeoff develops during diffusion-policy training rather than being induced by a particular sampling procedure. Figure 22 compares DDIM with 10 or 100 denoising steps, , and DDPM with 100 steps. Across all settings, pass@10 peaks earlier and declines with further training, while pass@1 continues to improve or remains stable. The same trend across sampling configurations shows that the late-training coverage loss is a property of the trained denoising model rather than the sampling configuration used for evaluation.
Figure 22. Pass@ training dynamics are stable across sampling configurations. Pass@1 and pass@10 are evaluated using DDIM with different numbers of denoising steps and , and using DDPM. All settings show the same late-training decline in pass@10.
Learning-rate Schedule Ablations
Our main experiments show that large- pass@k peaks earlier in supervised pretraining than pass@1, and can decline as pass@1 continues to improve. Because the learning-rate schedule controls how aggressively the policy is updated over training, it can affect when these changes occur. Related work on diffusion models likewise finds that optimization settings can shift the onset of finite-data fitting ([9, 49]). We therefore test whether changing the learning-rate schedule fundamentally changes the coverage–reliability tradeoff itself, or primarily shifts the timescales at which it appears. Across the schedules we evaluate, the same qualitative pattern remains: pass@10 peaks earlier and declines at later epochs, while pass@1 improves over a longer portion of training.
Setup. For single-task MimicGen, our main experiments follow the optimizer configuration of the public Diffusion Policy implementation [13]: AdamW with learning rate , , , weight decay , and 500 warmup steps followed by a single half-cosine decay from to zero over the full training horizon. We use this standard recipe throughout experiments included in the main text.
We repeat the 200-demonstration Threading 2D experiment with four alternative schedules while keeping all other training settings fixed. (1) Constant , which removes annealing while preserving the maximum learning rate of the default schedule; (2) constant to test whether reducing the learning rate shifts both the improvement in pass@1 and the decline in large pass@; (3) cosine decay from to which preserves the monotone decay of the default schedule but removes its near-zero learning rates late in training; (4) periodic cosine with four cycles to test whether repeatedly increasing and decreasing the learning rate changes the same trend or instead produces different behavior.
Together, these settings cover fixed, monotone-decay, and cyclic learning-rate schedules, while isolating the effects of learning-rate magnitude, annealing, and the final learning-rate floor. In Figure 23, each panel overlays one ablated schedule with the default single cosine schedule (gray); solid and dashed curves denote pass@10 and pass@1 respectively.
Figure 23. Learning-rate schedule ablation on Threading D2 Each panel overlays one ablated schedule (colored) with the default single-cosine schedule (gray). Solid and dashed curves denote pass@10 and pass@1, respectively. Across schedules, large- pass@ peaks earlier and declines as pass@1 improves later in training; changing the learning-rate schedule primarily shifts the timing of these changes.
Results. Removing learning-rate decay doesn’t change the decline in large pass@. With constant , pass@10 increases early and then decreases over later epochs, while pass@1 remains slightly below the default run through most of training. Keeping the learning rate fixed at its maximum value therefore does not preserve the higher late-training pass@10 without also changing pass@1. Likewise, changing the endpoint of the cosine schedule from zero to produces nearly the same pass@1 and pass@10 curves as the default schedule. These two comparisons show that the decline in large- pass@ is not specific to either learning-rate decay or the near-zero learning rates reached at the end of the default schedule.
With constant , both metrics change more slowly. Pass@10 rises more gradually, reaches its maximum later, and begins declining later than under the default schedule. Pass@1 also improves more gradually and remains lower at epoch 800. Thus, the higher pass@10 retained at later epochs is accompanied by lower pass@1; reducing the learning rate shifts both sides of the tradeoff later in training rather than preserving coverage at the same reliability.
Periodic cosine produces larger cross checkpoint fluctuations but does not change the longer-term trend. Pass@10 still reaches its highest values earlier and decreases later in training, while pass@1 remains higher at later epochs. Repeated learning-rate cycling therefore makes training less stable across checkpoints without removing the coverage–reliability tradeoff. Among the schedules tested here, the default single-cosine schedule also gives the smoothest learning curves and the highest late-training pass@1, supporting its use as the default schedule in our main experiments.
Across all schedules tested, large- pass@ still peaks earlier and declines as pass@1 improves later in training. Smaller constant learning rates shift both changes to later epochs, while periodic cosine produces larger fluctuations across checkpoints. Thus, the learning-rate schedule changes when the coverage–reliability tradeoff appears, but does not fundamentally escape it. This mirrors Bonnaire et al. [9], who find that changing the optimizer shifts the onset of generalization and memorization in diffusion models while preserving their ordering. More broadly, Wu et al. [49] shows that learning rate can directly regularize finite-data fitting of the diffusion score at small noise. Designing new optimization strategies that preserve large- coverage without hurting improvements in pass@1 is an exciting direction for future work.
Denoising Network Size Ablations
We next examine how denoiser network capacity affect the observed learning dynamics between coverage and reliability. We vary the U-Net channel widths while keeping the visual encoder, data, optimizer, learning-rate schedule, and sampler unchanged. We test four denoiser capacities contain 24M, 78M, 277M, and 1.0B parameters. As shown in Figure 24, pass@10 peaks between roughly epochs 200–300 for every U-Net model size and then eventually declines to a similar level by epoch 800. Therefore, the loss of coverage during late training is not specific to the capacity of the denoising network.

Figure 24. The late-training coverage loss persists across denoiser widths. We vary the U-Net from 24M to 1.0B parameters while keeping the encoder, data, optimizer, learning-rate schedule, and sampler fixed. All four models show a similar peak and subsequent decline in pass@10.
EMA schedule Ablations
EMA changes how quickly the network weights used for evaluation track the current denoiser weights. A possible explanation for the late-training drop in large pass@ is therefore that the EMA schedule forgets the earlier, higher-coverage weights too quickly. If so, increasing the EMA memory should substantially reduce or remove the decline. Let denote the current model parameters and the EMA parameters. The EMA weights are updated as
where the decay follows the Diffusion Policy warmup schedule
Our default uses , , , , and .
We ablate over two key hyperparameters, the power and , each controlling different parts of the EMA memory schedule. Increasing makes decay faster, so approaches one earlier in training. The EMA therefore puts less weight on the current network at each update and retains older weights for longer. In other words, larger gives the EMA a longer memory earlier in training. By contrast, becomes to matter once this growth reaches the cap value. Increasing allows to move even closer to one before being clipped, giving the EMA a longer maximum memory later in training.
We compare the default setting (, ) against and . The first comparison increases the EMA memory earlier in training. The second keeps fixed and increases the maximum memory later in training; for , the 0.9999 cap is reached around epoch 309, while the 0.99995 cap is reached around epoch 668.
Figure 25 shows comparisons between these different temporal averaging schedules. Holding fixed, increasing from 0.75 to 0.9 gives the EMA a longer memory earlier in training. Accordingly, the policy lags more while the underlying current model is improving: pass@10 rises more slowly and peaks later. Its decline after the peak is also more gradual because the EMA continues to place more weight on parameters from earlier checkpoints.
Figure 25. Longer EMA memory delays and smooths the pass@10 decline but does not fundamentally remove it. Holding fixed, increasing from 0.75 to 0.9 gives the EMA a longer memory earlier in training, causing pass@10 to peak and decline later. Holding fixed, increasing from 0.9999 to 0.99995 allows the EMA memory to continue growing later in training and further smooths the decline. Pass@10 still decreases under all schedules.
Holding fixed isolates the effect of . The two schedules are identical until the run reaches its cap; after that, allows the EMA memory to keep growing and therefore retains earlier parameters for longer. This further smooths the late-training change in pass@10 for a little bit, but does not fundamentally prevent its decline. Thus, increasing either the early growth or the maximum length of the EMA memory can delay and smooth the loss of coverage, but neither prevents it.
Isolating Coverage Loss to the Denoising Network
To determine how much of the late-training coverage loss can be attributed to the visual encoder or the denoising network, we run three complementary experiments. (1) Freeze the encoder: we freeze the visual encoder at epoch 150 and continue training only the denoiser, testing whether removing further encoder updates changes the subsequent coverage decline. (2) Fresh denoiser with a fixed encoder: we train a new denoiser from scratch while keeping either the epoch-150 or epoch-800 encoder fixed throughout training. This isolates whether denoiser optimization alone can reproduce the peak-then-decline pattern under a fixed representation. (3) Low-dimensional observations: we train diffusion policies directly from state observations, removing the visual encoder entirely. This tests whether the same coverage–reliability tradeoff arises without any learned visual representations.
Freezing the encoder does not stop the coverage decline. We choose epoch 150 according to a split-data representation comparison shows that the nearest-neighbor structure of independently trained encoders has stabilized by this point. Starting from the epoch-150 checkpoint, we replace the encoder with its EMA copy, freeze all encoder parameters, and continue training the denoiser till the end of the training horizon. Figure 26 shows that pass@10 still declines at later epochs, with a similar pass@1 trend to the original run. Continued encoder updates are therefore not responsible for the coverage loss.
Figure 26. Freezing the encoder does not prevent the late-training large pass@ decline. From epoch 150 onward, the encoder is fixed and only the denoiser is trained. Pass@10 still decreases at later epochs. The overall training dynamics is largely the same as baseline run.
A fresh denoiser reproduces the decline with a frozen learned visual representation. We next randomly initialize a new denoiser and train it from scratch for 800 epochs while keeping the encoder fixed throughout training. We run this experiment with both the epoch-150 and epoch-800 encoders, using the same denoiser initialization. As shown in Figure 27, pass@10 first increases and then declines for both frozen representations. The decline therefore reappears even when the encoder never changes and the denoiser is trained entirely from scratch.
Figure 27. Fresh denoisers reproduce the pass@10 decline with fixed encoder representations. New denoisers are trained from scratch with either the epoch-150 or epoch-800 encoder frozen throughout training. Both show a later decline in large pass@.
The same trend appears without a visual encoder. Finally, we train diffusion policies directly from low-dimensional state observations. Across all five tasks in Figure 28, baseline pass@10 again declines at later checkpoints, while pass@1 remains stable or improves. Thus, the late-training coverage loss happens in the absence of any visual encoder.
Figure 28. Late-training coverage loss also occurs with low-dimensional state observations. Across five tasks, baseline diffusion policies lose pass@10 at later checkpoints despite using no visual encoder. Joint augmentation preserves substantially more pass@10.
Together, these experiments show that visual-encoder overfitting cannot account for most of the late training coverage loss. Freezing the encoder does not eliminate the decline, a newly initialized denoiser reproduces it under a frozen representation, and the same trend appears in low-dimensional state-based policies with no visual encoder at all. The visual encoder can still affect the magnitude of the effect, but the peak-then-decline behavior arises from denoiser training itself.
This result matches the generalization–memorization mechanism identified in prior work on diffusion models. With finite training data, the empirical denoising objective is minimized by the empirical score rather than the population score; and prolonged optimization can drive the denoiser toward memorizing training examples [17, 2, 50]. Network training initially favors the smoother, population-like components of the score and fits the higher frequency, dataset-specific components later, particularly at low noise [9, 15]. Our results show the same transition in conditional diffusion policies and it persists after removing visual-representation learning and is concentrated in the denoising network.
Best-of-k Off-Manifold Error Reveals the Same Coverage–Reliability Tradeoff
We additionally ask whether the coverage–reliability tradeoff appears under a different offline metric of policy quality. Following the off-manifold metric of Pan et al. [37], we perturb expert actions at high-ambiguity training states, execute the perturbed actions to obtain OOD observations, and
| Training rollouts | |||
| Selected | 0.0800 | 0.1033 | 0.1186 |
| Final | 0.0815 | 0.1059 | 0.1219 |
| Training rollouts | |||
| Selected | 0.0992 | 0.1275 | 0.1458 |
| Final | 0.1003 | 0.1291 | 0.1484 |
measure the residual of each predicted action from the span of expert actions at nearby training states. We extend from the one policy prediction sample to best-of- over stochastic policy samples.
We compare epochs 400 and 800 of 3-Piece Assembly run. The later checkpoint has higher pass@1 (0.328 vs. 0.2988) but lower pass@10 (0.591 vs. 0.639). For each of 400 critical states, we generate 16 action-perturbed OOD observations. At each observation, we draw 64 policy action chunks and compute the expected minimum off-manifold error among samples; recovers the single-sample metric originally introduced by the paper.
Figure 29 shows the same tradeoff in this metric. At , the later checkpoint has slightly lower off-manifold error. However, the ordering reverses at , and the gap increases with . Thus, the later checkpoint produces a slightly better typical sample under the original single-sample metric, whereas the earlier checkpoint enjoys a substantially better favorable tail. Mean off-manifold error therefore does not reveal the coverage–reliability tradeoff across training checkpoints. It may be more useful for comparing different policy methods under matched training budget, as in the original study that introduced the metric. Here we show that best-of- off-manifold error reveals the same coverage–reliability tradeoff observed with held-out action error and closed-loop pass@.
Figure 29. Best-of- off-manifold error reveals the same coverage–reliability tradeoff. At , ep800 has slightly lower off-manifold error; from onward, ep400 is better and the gap grows with .
How Policy Coverage Affects World-Model Learning
We additionally test whether policy coverage affects world-model learning from policy rollouts. For each task, we construct two training sets that share the same 200 expert demonstrations and differ only in the policy used to collect 1,000 additional unfiltered rollout episodes. One uses an earlier, higher-coverage checkpoint selected by the coverage-reliability criterion and the other uses the final, lower-coverage checkpoint. We keep failed rollouts because they provide additional state–action–next-state outcomes for learning dynamics and counterfactual reasoning on unseen actions.
We train the same latent dynamics model on each dataset where images are encoded once with the same frozen DINOv3 ViT-B/16 encoder ([41]), so the comparison is not affected by representation learning. Each model receives two frames of history and predicts the next three latent states open loop using the predicted actions. We evaluate both models on the same held-out episodes collected across the full checkpoint family.
Higher-coverage rollout data produces better held-out prediction. Table 4 shows lower prediction error for the early-rollout model at every horizon on both tasks. These results mirror the critic-learning experiment. Under a fixed rollout budget, broader policy coverage provides a wider range of state–action outcomes for learning. Because the rollouts are unfiltered, this includes both successful and failed behavior; both are useful for learning how different actions lead to different future states.
| Threading (D2) | Mug Cleanup (D1) | ||||||
| Training rollouts | Training rollouts | ||||||
| Selected | 0.0800 | 0.1033 | 0.1186 | Selected | 0.0992 | 0.1275 | 0.1458 |
| Final | 0.0815 | 0.1059 | 0.1219 | Final | 0.1003 | 0.1291 | 0.1484 |
Table 4. Higher-coverage rollout data improves held-out world-model prediction. We report open-loop latent token MSE; lower is better. Each model uses the same 200 demonstrations plus 1,000 unfiltered rollouts from either the earlier selected or final policy checkpoint.
Implementation Details
Tasks Datasets, and Evaluation
Across both MimicGen and RoboCasa365, we use the demonstration datasets released with the official repositories. Unless otherwise noted, experiments in the main paper use image observations as policy inputs.
For MimicGen, we train policies on 200 demonstration trajectories for the main-text experiments. We additionally report results with 500 demonstrations in Appendix E. For RoboCasa pretraining, we use 65 atomic tasks with 100 human demonstrations per task. For SFT, we use 100 demonstrations per target task.
For MimicGen BC evaluation, pass@k is computed over 500 held-out task configurations. For residual RL finetuning, we report the training success rate measured as the fraction of successful episodes among the 100 most recently completed episodes.
For MimicGen BC evaluation, pass@k is computed over 500 held-out task configurations. For each configuration we run independent rollouts and record the number of successes ; for every we compute pass@k , i.e. the fraction of the -subsets of the rollouts that contain at least one success, averaged over configurations. Best-of- action error is computed analogously: for each held-out state we sample action chunks, sort their errors , and compute , the expected minimum error over a uniformly random -subset of the draws, averaged over states. Instead of using the first passes, we use use all samples to approximate both quantities, which gives the same expectation but reduces variance. For residual RL finetuning, we report the training success rate measured as the fraction of successful episodes among the 100 most recently completed episodes.
For RoboCasa evaluation, pass@k is computed over 50 held-out task configurations for each evaluated task.
For both task suites, we use the default task horizon specified by the official repos.
Baseline Single Task Diffusion Policy
Our baseline single-task BC policy is a CNN-based diffusion policy following the implementation of Chi et al. [13]: a 1D temporal U-Net denoiser conditioned on a global observation embedding via FiLM, with a ResNet-18 visual encoder trained from scratch jointly with the denoiser network. The policy consumes a single observation step (two 84×84 RGB views, stacked as a 6-channel input, plus the 9-d proprioceptive state) and predicts an 8-step action chunk with 6-d rotation representation; all 8 predicted actions are executed open-loop before the next inference. Training uses the standard denoising objective: a uniformly sampled diffusion step , forward noising with a squared-cosine schedule, and an MSE loss on the predicted noise (-prediction). At inference we sample with 10 DDIM steps (). All six MimicGen tasks share the identical architecture and optimization settings listed in Table 5.
| Hyperparameter | Value |
| Observation steps | 1 |
| RGB inputs | 2 views, (6-channel stack) |
| Proprioceptive state | 9-d |
| Predicted action chunk size | 8 |
| Executed action chunk size | 8 |
| Visual encoder | |
| Backbone | ResNet-18 |
| Normalization | GroupNorm |
| Encoder sharing | one encoder per camera (not shared) |
| Image augmentation | random crop to (center crop at inference) |
| Denoiser | |
| Architecture | 1D conditional U-Net |
| Channel dimensions | [512, 1024, 2048] |
| Kernel size / GroupNorm groups | 5 / 8 |
| Diffusion step embedding dim | 128 |
| Diffusion process | |
| Training diffusion steps | 100 |
| schedule | squared-cosine () |
| Prediction type / loss | -prediction |
| Inference sampler | DDIM, 10 steps, |
| Optimization | |
| Optimizer | AdamW (, ) |
| Learning rate | |
| Weight decay | |
| LR schedule | cosine, 500 warmup steps |
| Batch size | 512 |
| Training epochs | 800 |
| Demonstrations | 200 (500 for Three-Piece Assembly D2 only) |
| EMA | inv gamma 1.0, power 0.75, max decay 0.9999 |
| Evaluation weights | EMA |
Table 5. Baseline diffusion-policy configuration. All settings are shared across the six MimicGen tasks unless noted.
JointAug Implementation
JointAug augments policy training with samples from a joint generative model of the full observation–action tuple , trained on the same demonstrations as the policy. The generator is a pixel-space, dual-output stream diffusion model that denoises the two-camera frame, the 9-d proprioceptive states together with the action chunk under a single shared diffusion process. Image tokens and the low-dimensional tokens are concatenated, jointly self-attend, and are split again, so every modality is generated conditioned on the others rather than independently. All three modalities are noised at the same timestep and the model predicts the -target for each; the loss is a weighted sum of per-modality mean squared errors with equal weights, which prevents the image from drowning out the 80-d action.
Inference. Samples are drawn with the generator’s EMA weights using a DDIM sampler with 100 steps. Training uses 1000-step squared-cosine schedule.
Selecting the generator checkpoint window. We save generator checkpoints every 50 epochs. For each saved checkpoint , we draw 1024 joint samples and compare each generated action chunk with two equally sized sets of expert action chunks: the demonstrations used to train the generator, , and a held-out set of the same task that are used to train neither the generator nor the policy. Let be the nearest-neighbor distance on normalized action chunks. We define
means generated actions are equally close to training and held-out demonstrations. We set to the first saved checkpoint with , when generated samples first become closer to the training data than held-out data, and sample from and the four subsequent checkpoints spaced 100 epochs apart. Table 7 lists the resulting checkpoints for each task.
| Mug D1 | Square D2 | Threading D2 | Hammer D1 | 3-Piece D1 | 3-Piece D2 | |
| Demonstrations | 200 | 200 | 200 | 200 | 200 | 500 |
| Generator checkpoints (training epoch) | 100, 200, 300, 400, 500 | 100, 200, 300, 400, 500 | 100, 200, 300, 400, 500 | 100, 200, 300, 400, 500 | 100, 200, 300, 400, 500 | 50, 150, 250, 350, 450 |
Table 7. Per-task JointAug settings. Generator checkpoints are indexed by their training epoch; rows of each minibatch are drawn equally from the listed checkpoints
Distillation into policy training. During policy training, 50% of minibatch’s rows is replaced by freshly generated tuples; replaced and dataset rows are then trained with the policy’s ordinary behavior-cloning loss, so there’s no change to the policy architecture of Appendix § 18.§ 2.
PostBC Implementation
We follow the image instantiation of Algorithms 1 and 4 (Appendix § 14 of Wagenmaker et al. [46]). An ensemble of members is first fit to the policy’s own training split; each member is a freshly initialized copy of the policy’s observation encoder followed by a 3-layer MLP head that regresses the flattened action chunk by mean squared error, and each is fit on its own bootstrap resample of that split. The ensemble is trained once per task, before and independently of policy training, with the settings in Table 8.
Table 8: PostBC posterior-ensemble config and noise injection. All settings are shared across the five MimicGen tasks for which we report PostBC.
| Hyperparameter | Value |
| Ensemble members | |
| Ensemble size | 20 |
| Member architecture | policy observation encoder + MLP head |
| Head width / layers | 512 / 3 |
| Member output | flattened action chunk (8 × 10 = 80-d) |
| Ensemble optimization | |
| Optimizer | Adam |
| Learning rate | |
| Batch size (per member) | 128 |
| Training epochs | 200 |
| Checkpoing used | 40 |
| Noise injection | |
| Noise strength | 2.0 |
| Excluded dimensions | gripper (final action dimension, scale 0) |
Table 8.
Posterior uncertainty is estimated from the disagreement of the ensemble predictions, cached per training window offline. At every fetch we draw a fresh noise vector from the diagonal of that covariance, scale it by , and add it to the clean target action with the gripper dimension zeroed; policy training is otherwise unchanged from Appendix § 18.§ 2 or H.7. For MimicGen tasks, we use . For RoboCasa, we use .
Context-Smoothed Pre-training in TMRL Implementation
Context-smoothed pre-training Hong et al. [19] widens a BC policy’s support by corrupting its conditioning during pretraining, so that one network models from conditional action distribution at to the marginal action distribution at . We keep the paper’s corruption kernel and noise schedule: a discrete linear schedule over with context steps, the variance-preserving marginal , and drawn independently of the action timestep . We evaluate with a clean context, .
Multitask Training on Robocasa
| Hyperparameter | Value |
| Image | 2 views, 84×84 (6-channel stack) |
| Proprioceptive state | 9-d |
| Action chunk | 8 steps, absolute pose, 6-d rotation (10-d) |
| Observation steps | 1 |
| Image stream | |
| Architecture | conv U-Net |
| Channel dimensions | [96, 192, 384] |
| Bottleneck | self-attention at 21×21 |
| Low-dimensional stream | |
| Architecture | DiT over 9 tokens (1 state + 8 action) |
| Model width / layers / heads | 384 / 6 / 6 |
| Dropout | 0.1 |
| Cross-modal coupling | |
| Bridge | MM-DiT block at the bottleneck (joint self-attention) |
| Time embedding dim | 256 |
| Diffusion process | |
| Training diffusion steps | 1000 |
| schedule | squared-cosine () |
| Prediction type | -prediction |
| Loss | per-modality mean MSE, weights 1:1:1 (image:state:action) |
| Sampler | DDIM, 100 steps |
| Sampling weights | EMA |
| Optimization | |
| Optimizer | AdamW (, ) |
| Learning rate | |
| Weight decay | |
| LR schedule | cosine |
| Batch size | 256 |
| EMA | inv gamma 1.0, power 0.75, max decay 0.9999 |
| Demonstrations | 200 (500 for Three-Piece Assembly D2) |
Table 6. Joint generator configuration. All settings are shared across the six MimicGen tasks unless noted.
Residual RL Implementation.
We use DICE-RL ([45]) as an instantiation of residual RL finetuning. In DICE-RL, a frozen pretrained diffusion policy is finetuned by additive correction from a residual actor. Let be the observation embedding of the frozen policy and the initial DDIM noise; the base policy deterministically maps to a chunk of actions, and the executed chunk is , where only the residual MLP is trained. Since the chunk is executed open-loop, one transition spans the entire chunk and the critic scores the whole chunk. Because image features learned by BC on MimicGen tasks carry a weak value signal, we parameterize the residual actor and the critic with the low-dimensional state; the frozen pretrained policy still remains image-based. Episodes end at the first success step or at the task horizon, with truncation treated as terminal. Hyperparameters are identical across tasks (Table 9). For each residual RL experiment, we run with three random seeds.
Table 9: Residual RL finetuning configuration, identical across tasks; only the frozen base checkpoint and the environment horizon change.
| Hyperparameter | Value |
| Frozen base policy | EMA weights, DDIM 10 steps (), chunk |
| Residual actor | MLP [1024, 1024, 1024], GELU, LayerNorm |
| Critic ensemble | 10× MLP [1024, 1024, 1024], GELU, LayerNorm |
| BC weight | 200 |
| TD loss / -step | MSE / chunks |
| Discount / target update | 0.99 / 0.01 |
| Parallel environments | 64 |
| Gradient steps per iteration | 40 |
| Batch size | 256 |
| RLPD expert ratio | 0.5 → 0.1 over the first 10k training steps |
| Optimizer | Adam, lr , weight decay |
| Actor update frequency | 2 |
| BoN Exploration | over samples |
Table 9.
Multitask Training on Robocasa.
We use the RoboCasa release of openpi and follow the RoboCasa benchmark recipe unchanged, except for the pretraining learning rate (Table 10).
| Hyperparameter | Value |
| Peak learning rate | / |
| Schedule | 1k-step warm-up, cosine decay to peak |
| Training steps | 100k / 50k |
| Global batch size | 256 |
| Optimizer | AdamW (, , wd ), grad clip 1.0 |
| EMA decay (evaluated weights) | 0.99 |
| Action chunk / cameras / image size | 50 steps / 3 / |
Table 10. training configuration (pretraining / fine-tuning). All values follow the RoboCasa benchmark except the pretraining peak learning rate (benchmark: ).
Pretraining. Starting from pi05_base, the language model is frozen, the vision encoder is trained at the base learning rate, and the action expert at the base learning rate, for 100k steps at batch size 256 with AdamW, a 1,000-step warm-up, and cosine decay to the peak. The benchmark uses a peak learning rate of ; we instead uses (4× higher, decaying to ), which we find to work better, with the vision-encoder scaling unchanged.
Supervised fine-tuning. Each fine-tuning run initializes all weights from a pretraining checkpoint and uses the benchmark fine-tuning recipe as is: peak learning rate with the same warm-up and cosine schedule (50k steps), batch size 256, the same frozen/scaled parameter groups, and normalization statistics recomputed on the SFT data. Runs differ only in their initialization; checkpoints are saved every 2.5k steps and evaluated on the SFT tasks up to 20k steps.
Real Robot Experiments
BC Pretraining. We train a Diffusion Policy using one 224 × 224 image from each of the gripper-mounted main and ultrawide cameras and two timesteps of proprioceptive state, expressed relative to the current end-effector pose. Each camera has a separate CLIP-pretrained ViT-B/16 encoder with attention pooling, finetuned at the base learning rate. A 1D conditional U-Net predicts a 16-step action chunk at 20 Hz (0.8 s). Each 10D action contains relative end-effector translation (3D) and rotation (6D), plus gripper width. The diffusion process uses 50 training timesteps, a squared-cosine schedule, and -prediction; inference uses 16 DDIM steps. We train for 208 epochs with AdamW (lr , , weight decay ), batch size 64, cosine learning-rate decay with 5% warmup, gradient clipping at 0.5, and weight EMA with decay 0.9999. We apply random cropping (95%) and color jitter. Of the 740 demonstrations, 680 are used for training and 60 for held-out validation.
World Model. The world model is a 12-layer diffusion transformer with width 768 and 12 attention heads. It operates on latents from a frozen Stable Diffusion VAE. Each training sequence contains six ultrawide-camera frames at 200 ms intervals: two history frames and four target frames. We use causal attention and condition each predicted frame on the four actions executed since the previous frame by adding their embeddings to the frame tokens. We follow Diffusion Forcing and sample independent noise levels for each frame. Training uses -prediction, a 1000-step squared-cosine schedule. Inference uses 50 DDIM steps. We use AdamW (lr , weight decay 0.01), 1000 warmup steps followed by a constant learning rate, batch size 32, and bf16 mixed precision. Figure 30 compares ground-truth frames (top row) with predictions from world models trained on early-checkpoint rollouts (middle row) and late-checkpoint rollouts (bottom row). The early-rollout model’s predictions more closely match the ground truth.
Figure 30. Qualitative world-model predictions on real-robot Push-T. The top row shows ground truth; the middle and bottom rows show predictions from world models trained on rollouts from the selected early and final policy checkpoints, respectively. Each model predicts four frames from two history frames (Chunk 1), then autoregressively predicts four more frames (Chunk 2). The early-rollout model achieves higher PSNR in both chunks.
References
- [1]Arthur Allshire, Himanshu Gaurav Singh, Ritvik Singh, Adam Rashid, Hongsuk Choi, David McAllister, Justin Yu, Yiyuan Chen, Huang Huang, Pieter Abbeel, et al. Scalable behavior cloning with open data, training, and evaluation. arXiv preprint arXiv:2606.27375, 2026.arxiv.org/abs/2606.27375
- [2]Ricardo Baptista, Agnimitra Dasgupta, Nikola B Kovachki, Assad Oberai, and Andrew M Stuart. Memorization and regularization in generative diffusion models. arXiv preprint arXiv:2501.15785, 2025.arxiv.org/abs/2501.15785
- [3]Jose Barreiros, Andrew Beaulieu, Aditya Bhat, Rick Cory, Eric Cousineau, Hongkai Dai, Ching-Hsin Fang, Kunimatsu Hashimoto, Muhammad Zubair Irshad, Masha Itkina, et al. A careful examination of large behavior models for multitask dexterous manipulation. Science Robotics, 11 (113):eaea6201, 2026.DOI
- [4]Quentin Bertrand, Anne Gagneux, Mathurin Massias, and Rémi Emonet. On the closed-form of flow matching: Generalization does not arise from target stochasticity. Advances in neural information processing systems, 38:8522–8549, 2026.arxiv.org/abs/2506.03719
- [5]Giulio Biroli, Tony Bonnaire, Valentin De Bortoli, and Marc Mézard. Dynamical regimes of diffusion models. Nature Communications, 15(1):9957, 2024.DOI
- [6]Johan Bjorck, Fernando Castañeda, Nikita Chernіadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025.arxiv.org/abs/2503.14734
- [7]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024.arxiv.org/abs/2410.24164
- [8]Adam Block, Ali Jadbabaie, Daniel Pfrommer, Max Simchowitz, and Russ Tedrake. Provable guarantees for attentive reactive behavior cloning: Bridging low-level stability and high-level behavior. Advances in Neural Information Processing Systems, 36:48534–48547, 2023.
- [9]Tony Bonnaire, Raphaël Urfin, Giulio Biroli, and Marc Mézard. Why diffusion models don’t memorize: The role of implicit dynamical regularization in training. Advances in Neural Information Processing Systems, 38:141266–141286, 2026.
- [10]Fan Chen, Audrey Huang, Noah Golowich, Sadhika Malladi, Adam Block, Jordan Ash, Akshay Krishnamurthy, and Dylan Foster. The coverage principle: How pre-training enables post-training. In International Conference on Learning Representations, volume 2026, pp. 112225–112293, 2026a.arxiv.org/abs/2510.15020
- [11]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
- [12]William Chen, Jagdeep Singh Bhatia, Catherine Glossop, Nikhil Mathihalli, Ria Doshi, Andy Tang, Danny Driess, Karl Pertsch, and Sergey Levine. Steerable vision-language-action policies for embodied reasoning and hierarchical control. arXiv preprint arXiv:2602.13193, 2026b.arxiv.org/abs/2602.13193
- [13]Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684–1704, 2025.
- [14]Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, and Adam Klivans. Ambient diffusion: Learning clean distributions from corrupted data. Advances in Neural Information Processing Systems, 36:288–313, 2023.arxiv.org/abs/2305.19256
- [15]Alessandro Favero, Antonio Sclocchi, and Matthieu Wyart. Bigger isn’t always memorizing: Early stopping overparameterized diffusion models. ArXiv, abs/2505.16959, 2025. URL https://api.semanticscholar.org/CorpusID:28789454.arxiv.org/abs/2505.16959
- [16]Dylan J Foster, Zakaria Mhammedi, and Dhruv Rohatgi. Is a good foundation necessary for efficient reinforcement learning? the computational role of the base model in exploration. arXiv preprint arXiv:2503.07453, 2025.
- [17]Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:2310.02664, 2023.arxiv.org/abs/2310.02664
- [18]Chengyang He, Xu Liu, Gadiel Mark Sznaier Camps, Joseph Bruno, Guillaume Adrien Sartoretti, and Mac Schwager. Demystifying robot diffusion policies: Action memorization and a simple lookup table alternative. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2505.05787
- [19]Matthew M Hong, Jesse Zhang, Anusha Nagabandi, and Abhishek Gupta. Tmrl: Diffusion timestep-modulated pretraining enables exploration for efficient policy finetuning. arXiv preprint arXiv:2605.12236, 2026.arxiv.org/abs/2605.12236
- [20]Haoxu Huang, Tongsam Zheng, Yifan Chen, Jiacheng You, and Yang Gao. Critical interval mse: Toward reliable offline validation for robot manipulation policies. arXiv preprint arXiv:2606.29898, 2026.arxiv.org/abs/2606.29898
- [21]Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, et al. π0.6: a vla that learns from experience. arXiv preprint arXiv:2511.14759, 2025a. Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv preprint arXiv:2504.16054, 2025b.
- [23]Feiyang Kang, Michael Kuchnik, Karthik Padhye, Marin Vlastelica, Ruoxi Jia, Carole-Jean Wu, and Newsha Ardalani. Quagmires in sft-rl post-training: When high sft scores mislead and what to use instead. In International Conference on Learning Representations, volume 2026, pp. 56876–56918, 2026.arxiv.org/abs/2510.01624
- [24]Liyiming Ke, Jingqiang Wang, Tapomayukh Bhattacharjee, Byron Boots, and Siddhartha Srinivasa. Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine manipulation. In 2021 IEEE international conference on robotics and automation (ICRA), pp. 6185–6191. IEEE, 2021.arxiv.org/abs/2011.06719
- [25]Liyiming Ke, Yunchu Zhang, Abhay Deshpande, Siddhartha Srinivasa, and Abhishek Gupta. Cicil: Continuity-based data augmentation for corrective imitation learning. In International Conference on Learning Representations, volume 2024, pp. 35813–35831, 2024.arxiv.org/abs/2310.12972
- [26]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024.arxiv.org/abs/2406.09246
- [27]Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645, 2025.arxiv.org/abs/2502.19645
- [28]Hadas Kress-Gazit, Kunimatsu Hashimoto, Naveen Kuppuswamy, Paarth Shah, Phoebe Horgan, Gordon Richardson, Siyuan Feng, and Benjamin Burchfiel. Robot learning as an empirical science: Best practices for policy evaluation. arXiv preprint arXiv:2409.09491, 2024.
- [29]Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning, pp. 143–156. PMLR, 2017.arxiv.org/abs/1703.09327
- [30]Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In International Conference on Machine Learning, pp. 22188–22214. PMLR, 2023.arxiv.org/abs/2210.14199
- [31]Sadhika Malladi, Samy Jelassi, Dylan Foster, Jordan T Ash, and Akshay Krishnamurthy. Tailfst: Filtered fine-tuning improves post-training performance. arXiv preprint arXiv:2608.25756, 2026.arxiv.org/abs/2608.25756
- [32]Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298, 2021.arxiv.org/abs/2108.03298
- [33]Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17596, 2023.arxiv.org/abs/2310.17596
- [34]Suvir Mirchandani, Suneel Belkhale, Joey Hejna, Evelyn Choi, Md Sazzad Islam, and Dorsa Sadigh. So you think you can scale up autonomous robot data collection? arXiv preprint arXiv:2411.01813, 2024.arxiv.org/abs/2411.01813
- [35]Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance. arXiv preprint arXiv:2410.13816, 2024.arxiv.org/abs/2410.13816
- [36]Soroush Nasiriany, Sep Nasiriany, Abhiram Maddikuri, and Yuke Zhu. Robocasa365: A large-scale simulation framework for training and benchmarking generalist robots. In International Conference on Learning Representations, volume 2026, pp. 98643–98667, 2026.arxiv.org/abs/2603.04356
- [37]Chaoyi Pan, Giridharan Anantharaman, Nai-Chieh Huang, Claire Jin, Daniel Pfrommer, Chenyang Yuan, Frank Permenter, Guannan Qu, Nicholas Boffi, Guanya Shi, et al. Much ado about noising: Dispelling the myths of generative robotic control. In International Conference on Learning Representations, volume 2026, pp. 90575–90614, 2026.arxiv.org/abs/2512.01809
- [38]Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal, Shashwat Saxena, Jesse Zhang, Giri Anantharaman, Cleah Winston, Chaoyi Pan, Douglas Chen, Nai-Chieh Huang, et al. Ogpo: Sample efficient full-finetuning of generative control policies. arXiv preprint arXiv:2605.03065, 2026.arxiv.org/abs/2605.03065
- [39]Kulin Shah, Alkis Kalavasis, Adam R Klivans, and Giannis Daras. Does generation require memorization? creative diffusion models using ambient diffusion. arXiv preprint arXiv:2502.21278, 2025.
- [41]Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025.arxiv.org/abs/2508.10104
- [42]David Snyder, Asher James Hancock, Apurva Badithela, Emma Dixon, Patrick Miller, Rares Andrei Ambrus, Anirudha Majumdar, Masha Iktina, and Haruki Nishimura. Is your imitation learning policy better than mine? policy comparison with near-optimal stopping. arXiv preprint arXiv:2503.10966, 2025.
- [43]Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36:47783–47803, 2023.
- [44]Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, and Aditi Raghunathan. Overtrained language models are harder to fine-tune. arXiv preprint arXiv:2503.19206, 2025.
- [45]Zhanyi Sun and Shuran Song. From prior to pro: Efficient skill mastery via distribution contractive rl finetuning. arXiv preprint arXiv:2603.10263, 2026.arxiv.org/abs/2603.10263
- [46]Andrew Wagenmaker, Perry Dong, Raymond Tsao, Chelsea Finn, and Sergey Levine. Posterior behavioral cloning: Pretraining bc policies for efficient rl finetuning. arXiv preprint arXiv:2512.16911, 2025a.arxiv.org/abs/2512.16911
- [47]Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint arXiv:2506.15799, 2025b.
- [48]Yanwei Wang, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Pérez-D’Arpino, Dieter Fox, and Julie Shah. Inference-time policy steering through human interactions. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 15626–15633. IEEE, 2025.
- [49]Yu-Han Wu, Pierre Marion, GÊrard Biau, and Claire Boyer. Taking a big step: Large learning rates in denoising score matching prevent memorization. arXiv preprint arXiv:2502.03435, 2025.arxiv.org/abs/2502.03435
- [50]Zeqi Ye, Qijie Zhu, Molei Tao, and Minshuo Chen. Provable separations between memorization and generalization in diffusion models. In International Conference on Learning Representations, volume 2026, pp. 143375–143425, 2026.
- [51]Dylan Zhang, Yufeng Xu, Haojin Wang, Qingzhi Chen, and Hao Peng. Good sft optimizes for sft, better sft prepares for reinforcement learning. arXiv preprint arXiv:2602.01058, 2026.
- [52]Thomas T Zhang, Daniel Pfrommer, Chaoyi Pan, Nikolai Matni, and Max Simchowitz. Action chunking and exploratory data collection yield exponential improvements in behavior cloning for continuous control. arXiv preprint arXiv:2507.09061, 2025.

























