betaFeedback

WorksComputer Science

Beyond Validation Loss: Coverage and Reliability Dynamics in Robot Policy Pretraining

Abstract

In modern robot learning, generative robot policies are first pretrained through supervised learning from demonstrations, then refined through reinforcement learning, supervised finetuning, or other forms of posttraining. Yet conventional pretraining metrics, such as validation loss and average task success, provide an incomplete account of which policies support both effective direct deployment and subsequent posttraining. In this work, we study closed-loop pass@k, which evaluates the probability that k rollouts from the same initial condition attain at least one success. Thus, pass@1 measures mean task success; large-k pass@k measures whether success remains reachable under repeated sampling. Across a wide range of posttraining recipes, we show that large-k pass@k reveals posttraining differences not captured by pass@1. In particular, checkpoints with similar pass@1 but different large-k pass@k have different posttraining performance. Moreover, we show pass@k peaks at different checkpoints during supervised pretraining, revealing a coverage-reliability tradeoff over the course of training. Based on these observations, we propose two practical interventions: (1) a purely offline checkpoint-selection criterion that balances coverage and reliability without closed-loop evaluation, and (2) a generative data augmentation procedure that expands supervision beyond the finite demonstration set, preserving large-k pass@k while maintaining or improving pass@1 and raising the observed coverage-reliability Pareto frontier. Together, our results motivate evaluation, checkpoint selection, and training procedures that explicitly account for posttraining. Blog post, visualizations, and code are available at https://zhanyisun.github.io/robot-policy-learning-dynamics/.

Introduction

Robot policies are increasingly trained in stages: supervised behavior cloning (BC) or large-scale pretraining [13, 26, 7], followed by reinforcement learning (RL), supervised finetuning (SFT), or other forms of adaptation. Yet supervised pretraining and posttraining are usually studied as separate problems. The checkpoint handed to posttraining is typically chosen by validation loss, average task success, or simply taking the final checkpoint — criteria that evaluate the supervised pretraining stage in isolation. This leaves a central question unresolved: how do the learning dynamics of supervised pretraining shape both direct deployment and downstream finetunability, and what criterion can identify checkpoints that jointly support both?

Language-model research provides strong motivation to look beyond metrics and progress within a single stage of training in isolation. In particular, models with similar pretraining validation loss can perform very differently after posttraining, while prolonged pretraining or stronger SFT performance does not always carry over to subsequent posttraining [30, 44, 23, 51]. Recent work further highlights coverage of high-quality outputs as an important property for downstream posttraining [16, 10].

Robot control, however, presents a different set of challenges than language modeling: policies are densely conditioned on observations, actions are continuous and have far less distributional diversity [37], and success requires stable and precise closed-loop execution Simchowitz et al., 2025; Zhang et al., 2025). These demands may appear to favor increasingly reliable policies over broader behavioral coverage.

Contributions. Despite these differences, we find that supervised training of generative robot policies is governed by a similar tradeoff between broad behavioral coverage and reliable task execution. We study this tradeoff through closed-loop pass@kk* from a fixed initial condition (henceforth, fixed-start pass@kk). For each initial configuration, we rollout the stochastic policy kk times and ask whether at least one rollout succeeds, then average over the distribution of initial configurations (§ 2). Equivalently, pass@kk is the expected probability, over the distribution of initial configurations, that at least one of kk independently sampled rollouts from the same configuration succeeds. At k=1k = 1, this recovers the standard mean task-success rate; we refer to this single-rollout success as reliability. As kk increases, pass@kk measures the fraction of initial conditions for which the policy can produce at least one successful rollout within kk attempts; we refer to this as coverage. The full pass@kk profile therefore lets us study how reliability and coverage evolve during pretraining, and how these differences affect posttraining.

We establish two key findings (§ 3). First, large-kk pass@kk tracks downstream posttraining outcomes that pass@1 alone doesn’t: checkpoints with similar pass@1 but different large-kk pass@kk can yield substantially different performance after subsequent SFT or RL finetuning. Second, reliability and coverage diverge systematically during supervised pretraining. Large-kk pass@kk peaks earlier and can decline while pass@1 continues to improve, revealing that further optimization can improve reliability at the expense of coverage. This tradeoff also matters for direct deployment: declining large-kk pass@kk reflects the loss of low-probability successful behaviors on harder initial conditions, even as average task success improves.

Motivated by these findings, we propose a novel offline criterion for selecting policy checkpoints that best balance coverage and reliability (§ 4). Estimating pass@*kk requires many closed-loop rollouts from matched initial configurations, which is prohibitively expensive on real robots. Instead, our criterion is computed entirely offline from the quantile profile of held-out action error. This profile captures favorable-tail coverage and typical-case reliability, allowing us to select the checkpoint that best balances the two. Crucially, it requires no closed-loop policy evaluation, no deployment-specific choice of kk, and no tuned tradeoff weight, making it easy to integrate into modern robot policy training pipelines.

Lastly, we introduce a generative data-augmentation procedure that raises the Pareto frontier between mean task success and large-kk pass@*kk performance (§ 5). We train a joint observation–action generative model to expand supervision beyond the finite demonstration dataset. The resulting augmentation preserves favorable-tail coverage as pretraining proceeds while maintaining, and in some cases improving, reliability as well. By preserving coverage without sacrificing mean task success, it reduces the need for checkpoint selection and provides stronger initializations for subsequent posttraining.

Problem Setup

Generative robot policies. We study conditional generative policies πθ(a∣o)\pi_\theta(a \mid o) over action chunks, focusing on diffusion-based policies. Under the noise-prediction parameterization, an expert action chunk a0a_0 is corrupted at noise level tt, and the denoiser is trained as

at=αta0+σtϵ,ϵ∼N(0,I),LDP(θ)=E(o,a0),t,ϵ[∥ϵ−ϵθ(at,o,t)∥22].(1)a_t = \alpha_t a_0 + \sigma_t \epsilon,\quad\epsilon\sim\mathcal{N}(0,I),\qquad\mathcal{L}_{\mathrm{DP}}(\theta) = \mathbb{E}_{(o,a_0),t,\epsilon}\left[\left\lVert\epsilon- \epsilon_\theta(a_t,o,t)\right\rVert_2^2\right]. \tag*{(1)}

At inference, aT∼N(0,I)a_T \sim\mathcal{N}(0,I) is iteratively denoised using ϵθ\epsilon_\theta to produce an action chunk.

Pretraining, posttraining, and checkpoint selection. We consider robot policies trained through an initial supervised pretraining phase followed by a subsequent posttraining phase. Along pretraining, optimization produces a sequence of checkpoints πτ\pi_\tau, where τ\tau indexes training progress, measured in epochs or optimization steps depending on the training setting.1 The selected checkpoint may then be deployed directly or used to initialize posttraining. In standard practice, checkpoints are commonly selected using validation loss, held-out action MSE, task success rate from closed-loop evaluation, or simply by taking the final checkpoint. Our central question is how the choice of pretraining checkpoint affects performance under direct deployment and after subsequent posttraining, and how to identify checkpoints that support both.

Fixed-start pass@kk. In large language model evaluation on verifiable tasks such as code generation and mathematical reasoning, pass@kk is the probability that at least one of kk independently sampled solutions is correct ([11]). For robot policies, we define the analogous closed-loop measure. For an initial condition c∼pevalc \sim p_{\mathrm{eval}}, let qτ(c)q_{\tau}(c) denote the one-rollout success probability of policy checkpoint πτ\pi_{\tau}. Then, we define the pass@kk for checkpoint τ\tau as

Pk(τ)=Ec∼peval[1−(1−qτ(c))k].(2)P_k(\tau) = \mathbb{E}_{c \sim p_{\mathrm{eval}}}\left[1-\left(1-q_{\tau}(c)\right)^k\right]. \tag*{(2)}

That is, for each held-out initial condition, we reset the simulator and execute kk independent rollouts from checkpoint πτ\pi_{\tau}; pass@kk is the fraction of conditions for which at least one rollout succeeds. Pass@1 recovers the standard mean task-success rate.2

Later Checkpoints Trade Large-kk Pass@k for Mean Task Success; Pass@1 Can Miss Posttraining Differences

We begin by examining how pass@kk curves across kk evolve over supervised pretraining, and whether posttraining from checkpoints with similar pass@1 but different large-kk pass@kk leads to different performance.

Pretraining evaluation. We study supervised-pretraining dynamics in two settings: (1) single-task BC on MimicGen, and (2) multitask VLA pretraining on RoboCasa. For single-task BC, we train image-conditioned Diffusion Policies ([13]) from scratch on 200 demonstrations for each of five MimicGen tasks ([33]). For multitask VLA pretraining, we use the OpenPI implementation of π0.5\pi_{0.5} ([21]) on 65 atomic RoboCasa365 tasks ([36]), with 100 demonstrations per task. Full evaluation details are provided in App. § 18.

For readability, Figure 1(a,Figure 1c) shows only two of the five selected MimicGen tasks together with the aggregate RoboCasa results. Main-text MimicGen experiments use 200 demonstrations per task. Later experiments (§ 5) show all five selected MimicGen tasks and the full RoboCasa evaluation.

Figure 1. Later checkpoints trade large-kk pass@kk for mean task success; checkpoints with similar pass@1 can lead to different posttraining performance. Left: single-task BC followed by residual RL on two representative MimicGen tasks, with (a) showing pass@kk during BC and (b) residual RL from different BC checkpoints. Right: multitask pretraining on RoboCasa tasks followed by SFT on target tasks, with (c) showing pass@kk during pretraining and (d) downstream SFT. Across (a,c), larger-kk pass@kk peaks and declines earlier than pass@1, so later checkpoints increasingly favor mean task success at the expense of large-kk pass@kk performance. Orange marks the earlier checkpoint selected by our offline coverage–reliability criterion (§ 4), while blue marks the final supervised-training checkpoint; matching dashed lines in (a,c) indicate the initializations used in (b,d). Despite similar pass@1, finetuning from the orange checkpoint achieves stronger performance than finetuning from the blue checkpoint across both settings.

During supervised pretraining, pass@kk peaks earlier as kk increases; mean task success favors later checkpoints. Figure 1(a,Figure 1c) shows a consistent pattern across single-task BC and multitask pretraining: the pass@kk optimum shifts earlier as kk increases. Thus, later checkpoints can achieve higher mean task success even after large-kk pass@kk has begun to decline. This divergence between mean task success and large-kk pass@kk reflects a redistribution of success across conditions during supervised pretraining. Recall that qτ(c)q_{\tau}(c) is the one-rollout success probability from condition cc. Pass@1 depends only on its mean; for k>1k > 1, we can write Pk(τ)=Ec[fk(qτ(c))]P_k(\tau) = \mathbb{E}_c[f_k(q_{\tau}(c))], where fk(q):=1−(1−q)kf_k(q) := 1 - (1-q)^k is concave in its argument qq. Shifting success probability from harder conditions toward already-reliable ones can therefore increase average success while decreasing larger-kk pass@kk. As supervised pretraining proceeds, condition-wise success can become increasingly polarized: already-solvable conditions become more reliable while harder conditions lose success probability. At the behavioral level, we interpret this redistribution as a tradeoff between reliability and coverage. See App. § 12 for corresponding condition-level decomposition.

Takeaway 1: Over supervised pretraining of robot policies, success can become increasingly polarized across conditions: mean task success can continue to improve or stabilize as large-kk pass@kk declines.

Posttraining evaluation. We show that checkpoints with different pass@kk profiles can attain substantially different posttraining performance. For each experiment, we compare the earlier checkpoint selected by our offline coverage–reliability criterion (§ 4), called the selected checkpoint, with the final checkpoint from the same supervised pretraining run. The two have similar pass@1, but the selected checkpoint retains substantially higher large-kk pass@kk.

We evaluate four posttraining settings. (1) Filtered BC: we collect the same number of rollouts from both checkpoint, retain successful episodes, append them to the original training set, and train a new BC policy. (2) steering with a learned critic: we train one Q-function per checkpoint on the original training data augmented with unfiltered rollouts from that checkpoint. At held-out state ss, both critics score the same eight candidate action chunks {aj}\{a_j\}. We estimate each candidate’s value pjp_j by branching 16 simulator rollouts from (s,aj)(s,a_j) and measuring the fraction that succeed, then evaluate whether Q(s,aj)Q(s,a_j) correctly ranks candidates by pjp_j. (3) actor-critic residual RL with DICE-RL (Sun & Song, [45]), an online RL method for finite-horizon diffusion policies. (4) lastly, on RoboCasa365 only, we perform supervised finetuning (SFT) from different pretrained checkpoints using identical target-task finetuning data and optimization. Full experimental details are in App. § 18.

Together, these experiments provide complementary tests of how pretrained-policy coverage affects posttraining. In filtered BC, new supervision comes from successful rollouts (Mirchandani et al., [34]); under a fixed rollout budget, lower coverage can yield a similar number of successes from fewer conditions, reducing data diversity. In critic steering, broader coverage exposes the critic to a wider range of observation–action outcomes, which can improve generalization when ranking actions at unseen observation; such ranking is central to value-guided test-time steering (Nakamoto et al., [35]) and actor–critic RL. Actor-critic residual RL tests whether these differences persist when data collection, critic learning, and policy optimization occur together online. Lastly, SFT verifies the findings persist when target-task data are fixed rather than policy-generated rollouts.

Checkpoints with large-kk pass@kk enjoy stronger posttraining performance despite similar pass@1 performance. Table 1 and Table 2 isolate this effect under online data collection. Across both tasks, retraining on successful rollouts from the selected (higher large-kk pass@kk) checkpoint produces a stronger filtered BC policy. Indeed, for matched rollout budgets, a greater chance at success on wider variety of initial conditions is more useful than repeated successes on already-solved conditions. Similarly, critic trained on the selected-checkpoint rollouts also rank counterfactual actions more accurately at held-out states, indicating better value-function generalization.

TaskSelectedFinal
Threading (D2)0.371 / 0.8860.321 / 0.812
Mug Cleanup (D1)0.558 / 0.9140.502 / 0.866

Table 1. Filtered BC. Pass@1/pass@32 after retraining on successful rollouts generated by the selected or final checkpoint.

TaskSelectedFinal
Threading (D2)0.5340.521
Mug Cleanup (D1)0.5890.547

Table 2. Critic action ranking accuracy. Within-state pairwise accuracy against simulator-estimated action values.

Figure 1(b,Figure 1d) shows that checkpoint selection also affects actor-critic RL finetuning. Under the same interaction budget, residual RL from the selected checkpoint, which has higher-large-kk, reaches higher task success than from the final checkpoint. In App. § 13.§ 2, we also compare different pretrained checkpoints under OGPO, a full-policy RL finetuning method ([38]). Unlike residual RL, OGPO directly finetunes the pretrained policy rather than learning a residual or steering head on top of a frozen policy. Nevertheless, the selected earlier checkpoint still outperforms the final checkpoint. Lastly, on RoboCasa, SFT from the selected checkpoint is also stronger under identical finetuning data and optimization, showing that the checkpoint effect is not limited to methods trained on policy-generated rollouts.

We note that large-kk pass@kk does not globally rank checkpoints for finetuning: for much earlier checkpoints with lower pass@1, which checkpoint attains higher RL success can depend on finetuning budget and be task-specific. We report results on residual RL finetuning from more pretrained checkpoints in App. § 13.§ 1. We show that, on some tasks, a much earlier checkpoint with lower pass@1 but stronger large-kk pass@kk can eventually outperform the selected checkpoint given a sufficiently large RL finetuning budget. Among checkpoints with comparable pass@1, however, large-kk pass@kk is a better predictor of the RL posttraining performance.

Takeaway 2: Coverage–reliability dynamics during supervised pretraining affect posttraining: checkpoints with similar pass@1 can yield different posttraining performance. Our coverage–reliability criterion selects checkpoints that outperform later checkpoints favored by mean task success and held-out action MSE (App. § 15).

Loss of pass@kk performance during training corresponds with overfitting on the empirical dataset, and concentrates on low noise levels. We next examine the learning dynamics behind the late-checkpoint loss of coverage. We find that pretraining initially improves generalization but later increasingly fits the finite training set. In particular, later checkpoints have higher held-out denoising loss near the low-noise end despite lower training loss, with large relative increases where observation support is sparse.

We first test the shift toward training-set overfitting. On Threading D2, we split 1,000 demonstrations into two disjoint 200-demo training sets, D1D_1 and D2D_2, and a shared 600-demo held-out set, D3D_3. We train separate πτ(1)\pi^{(1)}_\tau and πτ(2)\pi^{(2)}_\tau on D1D_1 and D2D_2, respectively. Let aˉτ(i)(o,z)\bar{a}^{(i)}_\tau(o,z) denote the action chunk generated by πτ(i)\pi^{(i)}_\tau from observation oo and initial noise zz. Figure 2(a) measures prediction disagreement Dsplit(τ)=Eo∼D3,z[∥aˉτ(1)(o,z)−aˉτ(2)(o,z)∥22]D_{\mathrm{split}}(\tau) = \mathbb{E}_{o\sim D_3,z}\left[\left\|\bar{a}^{(1)}_\tau(o,z)-\bar{a}^{(2)}_\tau(o,z)\right\|_2^2\right]. Lower DsplitD_{\mathrm{split}} means the two policies agree more on the same held-out inputs. This quantity initially decreases but increases again during later training, suggesting that each policy increasingly fits its own training split. This is corroborated by Figure 2(b), which measures the ratio RiR_i between nearest neighbor distances from actions aˉτ(i)∼πτ(i)\bar{a}^{(i)}_\tau\sim\pi^{(i)}_\tau and DiD_i, divided by those from a⋆∼D3a^\star\sim D_3 (see App. § 14 for details). Here, Ri<1R_i < 1 means policy samples are closer to their own training split than held-out expert actions are. As pretraining proceeds, RiR_i

Two plots showing divergence between policies and proximity bias

Figure 2. Independent splits expose dataset-specific fitting. (a) Mean-squared prediction difference between two policies trained on disjoint splits, evaluated on the same held-out inputs with matched diffusion noise; predictions first converge, then diverge. (b) Ratio between the average nearest-neighbor distance from policy samples to their own training split and the average nearest-neighbor distance from held-out expert actions to that same training split; decreasing values indicate increasing bias toward the training split.

decreases for both policies. Taken together, these measurements support increasing fitting to the respective training splits during later training.

These split-data results are consistent with prior work showing that diffusion models shift from generalization toward training-set fitting and memorization under prolonged optimization ([17, 2, 50, 4, 9, 15]). In our image-conditioned robot policies, App. D further shows that later checkpoints have higher held-out denoising loss than earlier checkpoints near the low-noise end, despite having lower training loss at every noise level (Figure 11(a,Figure 11b)). The relative increase in held-out loss is also greater for observations with sparse training support and begins at higher noise (Figure 13). The intuition is that as the noise level decreases, the set of clean actions compatible with a noisy action shrinks; when observation support is sparse, fewer demonstrations also have nearby observations, leaving fewer training examples compatible with both the observation and the noisy action.

Figure 11. Later checkpoints exhibit higher held-out denoising loss near the low-noise end, and changes to the final denoising calls substantially affect pass@kk profile. (a) On training demonstrations, ϵ\epsilon-prediction MSE decreases with further training across all noise levels. (b) On unseen demonstrations, later checkpoints have higher denoising loss near the low-noise end than earlier checkpoints despite lower training loss. (c) In a 10-step DDIM sampler, replacing any one of the first four two-step bands (Q1–Q4) of the epoch-800 checkpoint with the corresponding epoch-200 weights has little effect on pass@1 or pass@10. Replacing Q5, the final two calls, restores most of the epoch-200 checkpoint’s pass@10 advantage while reducing pass@1.

Figure 13. Late-training held-out denoising loss increases more at lower diffusion noise levels and with sparser local conditioning support. Relative change in mean held-out ϵ\epsilon-prediction MSE from epoch 200 to epoch 800, Δrel,g(t)\Delta_{\mathrm{rel},g}(t) (Eq. D.6), for queries divided into equal thirds by local conditioning support. Negative values indicate lower mean loss at epoch 800; positive values indicate higher mean loss. At a fixed noise level, the relative loss change is generally larger for queries with sparser local conditioning support.

Takeaway 3: Supervised pretraining initially improves generalization, but later checkpoints increasingly fit the finite training set. Held-out denoising loss rises near the low-noise end, particularly for observations with sparse training support.

Diagnosing the Coverage–Reliability Tradeoff and Selecting Checkpoints Offline

Having characterized the transition toward finite-data overfitting, we next ask how to identify the coverage–reliability balance before overfitting, without explicitly evaluating pass@kk performance through closed-loop policy deployment.

Best-of-kk action MSE captures the favorable tail but selects too early. Because diffusion denoising loss is often a poor indicator of closed-loop performance ([1]), we use held-out action MSE for offline evaluation throughout the paper ([1]). A tempting offline metric for approximating closed-loop pass@kk is then held-out best-of-kk action MSE. Given a held-out dataset Dval\mathcal{D}_{\mathrm{val}}, we sample kk action chunks at each observation and retain the one closest to the expert action:

Bk(τ)=E(o,a∗)∼Dval[min⁡j≤kℓ(Aτ,j(o),a∗)].(3)B_k(\tau) = \mathbb{E}_{(o,a^*)\sim\mathcal{D}_{\mathrm{val}}}\left[\min_{j\leq k}\ell(A_{\tau,j}(o),a^*)\right]. \tag*{(3)}

Figure 3 shows that larger kk shifts the optimum of BkB_k earlier, as it does for pass@kk. For small kk, the two optima remain relatively close; as kk increases, the best-of-kk minimum increasingly precedes the corresponding closed-loop pass@kk peak. This mismatch arises because BkB_k measures only the favorable tail independently at each observation, whereas closed loop control requires reliable actions throughout an entire rollout trajectory. An early checkpoint can therefore attain low large-kk-BkB_k while still assigning substantial probability to poor action samples that destabilize execution. This motivates us to move beyond the minimum over samples and examine the full held-out action MSE distribution.

Figure 3. Best-of-kk action MSE selects earlier checkpoints than closed-loop pass@kk. (a) Held-out best-of-kk action MSE across training checkpoints for different kk. (b) Closed-loop pass@kk success rate across the same checkpoints. Matching colors and marker shapes denote the same value of kk across panels; hollow markers indicate the minimum MSE in (a) and maximum pass@kk in (b). As kk increases, both optima shift earlier, but the best-of-kk MSE minimum precedes the corresponding pass@kk peak, with a larger separation at higher kk.

Held-out action-MSE quantiles better reflect coverage and reliability. Best-of-kk MSE uses only the minimum error among sampled actions. Here, we now examine the full distribution of action MSE under policy samples. For held-out pair (oi,ai∗)(o_i,a_i^*) and checkpoint τ\tau, draw Aj∼πτ(⋅∣oi)A_j\sim\pi_\tau(\cdot\mid o_i), j=1,…,Mj=1,\ldots,M, and let Qi,τ(u)Q_{i,\tau}(u) be the uu-quantile of {ℓ(Aj,ai∗)}j=1M\{\ell(A_j,a_i^*)\}_{j=1}^M. We average each quantile across held-out observations,

Q‾τ(u)=1N∑i=1NQi,τ(u).(4)\overline{Q}_\tau(u)=\frac{1}{N}\sum_{i=1}^{N}Q_{i,\tau}(u). \tag*{(4)}

Figure 4(a) shows a more systematic ordering: lower quantiles reach their minima earlier and then worsen, whereas higher quantiles continue improving later in supervised pretraining. Figure 4(b) illustrates this directly: the early checkpoint can produce lucky draws closer to the expert action, but its samples are substantially more dispersed; the late checkpoint produces more concentrated samples, but its best sample is farther from the expert.

Figure 4. Held-out action-MSE quantiles expose the coverage–reliability balance. (a) Held-out action-MSE quantiles across training, colored by percentile uu from 0.05 (red) to 0.95 (blue). (b) For a held-out observation, 64 matched-noise samples from ep50 and ep800, plus the expert action, are projected with PCA; rings mark the minimum-MSE sample. ep50 is more dispersed yet has lower best-of-64 MSE, whereas ep800 contracts but its best sample lies farther from the expert. Consistent with (a), lower quantiles favor earlier checkpoints, while higher quantiles improve later. (c) We summarize the lower and upper halves as rcov∣H(τ)r_{\mathrm{cov}\mid H}(\tau) and rrel∣H(τ)r_{\mathrm{rel}\mid H}(\tau) and select the checkpoint minimizing max⁡{rcov∣H(τ),rrel∣H(τ)}\max\{r_{\mathrm{cov}\mid H}(\tau),r_{\mathrm{rel}\mid H}(\tau)\}; the selected ep350 matches the closed-loop oracle.

A distributional criterion identifies the coverage–reliability balance. Given the previous findings, we propose to use lower quantiles as an offline measure of coverage, i.e. whether low-MSE action chunks remain accessible under the policy, and higher quantiles as a measure of reliability, how consistently sampled action chunks remain close to the demonstrated action. Based on this, we define the following normalized quantile score for training budget HH:

rτ∣H(u)=log⁡Qˉτ(u)−min⁡s≤Hlog⁡Qˉs(u)max⁡τ′≤Hlog⁡Qˉτ′(u)−min⁡s≤Hlog⁡Qˉs(u)(5)r_{\tau\mid H}(u)=\frac{\log\bar{Q}_{\tau}(u)-\min_{s\leq H}\log\bar{Q}_{s}(u)}{\max_{\tau'\leq H}\log\bar{Q}_{\tau'}(u)-\min_{s\leq H}\log\bar{Q}_{s}(u)} \tag*{(5)}

Here, rτ∣H(u)=0r_{\tau\mid H}(u)=0 means checkpoint τ\tau has the lowest observed MSE at percentile uu among checkpoints up to HH, while rτ∣H(u)=1r_{\tau\mid H}(u)=1 means the highest; intermediate values indicate its normalized position between these two extremes. Let U\mathcal{U} denote the discrete set of evaluated quantile levels. To evaluate policy coverage and reliability, we average lower and upper halves of the quantile profile as

rcov∣H(τ)=mean⁡u∈U, u<0.5rτ∣H(u),rrel∣H(τ)=mean⁡u∈U, u≥0.5rτ∣H(u).(6)r_{\mathrm{cov}\mid H}(\tau)=\operatorname{mean}_{u\in\mathcal{U},\,u<0.5}r_{\tau\mid H}(u),\qquad r_{\mathrm{rel}\mid H}(\tau)=\operatorname{mean}_{u\in\mathcal{U},\,u\geq0.5}r_{\tau\mid H}(u). \tag*{(6)}

using u=0.5u=0.5 to split the profile evenly without additional tuned threshold. We then select

τH∗=arg min⁡τ≤H max⁡{rcov∣H(τ),rrel∣H(τ)}.(7)\tau_H^*=\underset{\tau\leq H}{\operatorname{arg\,min}}\ \max\left\{r_{\mathrm{cov}\mid H}(\tau),r_{\mathrm{rel}\mid H}(\tau)\right\}. \tag*{(7)}

The criterion therefore selects the checkpoint that minimizes the larger of rcov∣Hr_{\mathrm{cov}\mid H} and rrel∣Hr_{\mathrm{rel}\mid H}. Unlike best-of-kk, which uses only the minimum, it accounts for both lower and upper quantiles; unlike average validation MSE, it does not collapse the error distribution to a single average. By minimizing the worse of the two scores, the criterion avoids selecting a checkpoint with strong coverage but poor reliability, or vice versa. Lastly, because the criterion is defined relative to the quantile values, it also does not require task-specific calibration over absolute action-MSE thresholds.

Figure 4(c) illustrates the criterion on the 200-demonstration Threading D2 run. Using only offline held-out action MSE, the criterion recovers the closed-loop oracle at epoch 350. We define the closed-loop oracle by applying the same minimax balancing principle across the full pass@k profile. The criterion is also robust to the choice of training horizon once the coverage–reliability dynamics has stabilized: after lower quantiles begin to worsen and higher quantiles saturate, the selected checkpoint remains relatively stable as training continues. Extending training substantially beyond this point only marginally shifts the selected checkpoint (App. § 15). App. § 15 further validates the criterion across MimicGen tasks and dataset sizes, as well as multitask pretraining on RoboCasa.

Takeaway 4: Best-of-kk action MSE uses only the minimum-error sample and selects checkpoints too early, especially at large kk. Across the full held-out action-MSE distribution, lower quantiles favor earlier checkpoints while higher quantiles improve later, enabling an offline criterion that balances coverage and reliability without closed-loop evaluation.

Generative Augmentation Preserves Coverage for Downstream posttraining

Although our proposed coverage–reliability criterion (§ 4) selects a favorable compromise without closed-loop evaluation, it is constrained to the observed coverage–reliability Pareto frontier of that run. This raises a more fundamental question: is this tradeoff unavoidable? Can a training intervention give us the best of both worlds by preserving strong large-kk pass@kk while continuing to improve reliability? If so, we can raise the coverage–reliability Pareto frontier.

§ 3 links the loss of coverage to overfitting to the finite demonstration data during policy training. In App. § 14, we show that the held-out denoising score-matching loss increases more sharply late in training for the observation-conditioned model than for its unconditional counterpart. Indeed, Gu et al. [17] show that even conditioning diffusion models on uninformative random labels can substantially exacerbate memorization. This suggests a direct intervention: expand supervision beyond the observed (o,a)(o, a) pairs using fresh synthetic observation–action pairs that reflect the population-level expert distribution.

Joint (o,a)(o, a) augmentation. To expand supervision beyond observed (o,a)(o, a) pairs while preserving observation–action consistency, we propose joint (o,a)(o, a)-augmentation (JointAug), which trains a generative model Gγ(o,a)G_\gamma(o, a) with checkpoints indexed by γ\gamma on the joint demonstration distribution and uses generated pairs for policy training. We save generator checkpoints at fixed intervals, and aim to identify a sampling window after the generator has learned population-level structure but before substantial overfitting. To this end we record each checkpoint’s ratio between generated actions’ nearest-neighbor distances to training expert actions and held-out expert actions (App. § 18.§ 3). A ratio of 1 indicates equal proximity to the two sets, and values below one indicate that generated samples are closer to the training data. We let γ0\gamma_0 be the first checkpoint where the ratio falls below one, when generated samples first become closer to training than held-out expert actions. We then sample synthetic (o~,a~)(\tilde{o}, \tilde{a}) pairs uniformly from γ0\gamma_0 and the four subsequent checkpoints spaced 100 epochs apart. Because checkpoints are saved relatively densely over this stage of generator training, the five-checkpoint window spans only the transition from population-level generalization toward early fitting of the empirical training set. We use five checkpoints as a simple default heuristic across tasks rather than tuning the window for each task. App. § 16.§ 2 shows that performance is robust to the exact generator checkpoint selection.

Then policy training uses equal amounts of real and generated supervision:

Laug=12E(o,a)∼D[LDP(o,a)]+12E(o~,a~)∼Gmix[LDP(o~,a~)],(8)\mathcal{L}_{\mathrm{aug}} = \frac{1}{2}\mathbb{E}_{(o,a)\sim\mathcal{D}}\left[\mathcal{L}_{\mathrm{DP}}(o,a)\right] + \frac{1}{2}\mathbb{E}_{(\tilde{o},\tilde{a})\sim G_{\mathrm{mix}}}\left[\mathcal{L}_{\mathrm{DP}}(\tilde{o},\tilde{a})\right], \tag*{(8)}

where GmixG_{\mathrm{mix}} is the uniform mixture of the five selected generator checkpoints. Real samples retain the clean expert targets; generated pairs expand supervision beyond the finite data. Because observations and actions are generated jointly, their correspondence is naturally preserved.

To evaluate JointAug, we compare against baseline Diffusion Policy and posterior behavior cloning (PostBC, Wagenmaker et al. (2025a)) across the full training horizon. PostBC aims to improve policy coverage by estimating observation-conditioned action covariance with a bootstrapped ensemble and perturbing action targets a~\tilde{a} at clean observations oo. In that regime, the perturbed action is not constrained to remain compatible with the clean observation. In contrast, JointAug ensures (o~,a~)(\tilde{o}, \tilde{a}) is learned from the correct joint distribution (and thus, a~∣o~\tilde{a} \mid\tilde{o} is consistent).

In the first five panels of Figure 5, we compare the methods across MimicGen tasks. JointAug achieves higher pass@10 than both PostBC and the baseline DP run and prevents the baseline’s late-training loss of coverage. Its pass@1 matches the baseline and sometimes improves it. PostBC also improves pass@10, but on some tasks lowers pass@1, likely because its perturbed actions can be incompatible with the clean observations. Moreover, unlike PostBC, which perturbs only action targets, JointAug generates new observation–action pairs to further mitigate the conditional overfitting observed in App. § 14. App. § 16.§ 3 analyzes the sampled action distributions. JointAug preserves the favorable low-error tail and reduces typical-sample error, helping explain its gains in both coverage and reliability. JointAug also provides checkpoints with stronger residual-RL initializations than the final baseline checkpoints (Figure 6), often matching or exceeding the checkpoint selected by our coverage–reliability criterion from baseline run. By preventing the late-training loss of coverage, JointAug also reduces the need for post hoc checkpoint selection. We further find that its performance is robust to the choice of generator checkpoint; more ablations and baseline comparisons are in App. § 16.

Pass@10 and pass@1 success rates during pretraining across Threading (D2), Hammer Cleanup (D1), Mug Cleanup (D1), Square (D2), 3-Piece Assembly (D1), and RoboCasa (65 tasks)

Figure 5. During pretraining, generative augmentation preserves coverage while maintaining, and in some tasks improving, reliability (pass@1). Across most MimicGen tasks and RoboCasa, Baseline DP loses pass@10 at later checkpoints. JointAug prevents this loss of coverage while maintaining or improving pass@1. PostBC also preserves pass@10 better than Baseline DP, but can do so at the cost of pass@1.

Figure 6. Generative augmentation produces stronger checkpoints for posttraining. Columns correspond to Figure 5. We compare posttraining from the baseline checkpoint selected by our coverage–reliability criterion (§ 4), the final Baseline DP checkpoint, and the final checkpoint from our generative augmentation run. Across MimicGen tasks, the final generative-augmentation checkpoint substantially outperforms the final baseline and often matches or exceeds the selected checkpoint. On RoboCasa, it matches the selected checkpoint on both pass@1 and pass@32.

A scalable action-only variant. For large-scale VLA pretraining, joint generation is expensive, so on RoboCasa we use a lightweight action-only variant, similar in spirit to PostBC but using the policy’s own action variation rather than a bootstrapped ensemble for covariance estimation, thereby avoiding the need to tune the perturbation scale. After a standard pretraining run, we select c⋆c^{\star} using our coverage–reliability criterion and estimate action covariance Σc⋆(o)\Sigma_{c^{\star}}(o) from multiple policy samples at each training observation. We then resume pretraining from c⋆c^{\star} with perturbed targets a~=a+ϵ\tilde{a}=a+\epsilon, where ϵ∼N(0,Σc⋆(o))\epsilon\sim\mathcal{N}(0,\Sigma_{c^{\star}}(o)). This injects perturbations calibrated to the action spread at c⋆c^{\star} without shifting each training target in expectation, Eϵ[a~∣a,o]=a\mathbb{E}_{\epsilon}[\tilde{a}\mid a,o]=a. Figure 5 and Figure 6 show that the variant preserves large-kk pass@k as pass@1 improves, and achieves SFT performance comparable to the selected early checkpoint.

Takeaway 5: Generative augmentation can preserve coverage while allowing reliability to improve, producing late-pretraining checkpoints with stronger pass@kk profiles and better downstream posttraining than standard final checkpoints.

Coverage–Reliability Dynamics On a Real Robot

To test whether the coverage–reliability dynamics also arise in policies trained on real-robot data, we study a challenging variant of Push-T. We use only a wrist-mounted camera for visual observations and allow full 6-DoF end-effector control. The end-effector manipulates the T-block only through non-prehensile contact. We collect 740 human demonstrations, using 680 for training and 60 for held-out validation, and train a Diffusion Policy for 200 epochs with an action chunk length of 16. Experiment details are in App. H.8. The best-of-kk validation action MSE and error quantiles exhibit the same coverage–reliability tradeoff observed in simulation (Figure 7(b,c)). Applying our coverage–reliability balance criterion (§ 4) selects the checkpoint at epoch 46 (Figure 7(d)). We compare the selected checkpoint against the final checkpoint at epoch 200 using five rollouts from each of 20 fixed initial states. The selected checkpoint outperforms the final checkpoint by 10% on pass@1 and 20% on pass@5 (Figure 7(e)).

Figure 7. Real-robot Push-T. (a) Task setup. (b) Best-of-kk validation action MSE across training epochs. (c) Quantiles of validation action error across training epochs. (d) Normalized coverage and reliability scores with the selected checkpoint at epoch 46. (e) Pass@1 and pass@5 success rates of the selected (epoch 46) and final (epoch 200) checkpoints over 20 fixed initial states.

To investigate whether differences in policy coverage affect downstream model learning, we train two action-conditioned world models on the same demonstration data, augmented with 200 rollout trajectories from either the selected early checkpoint or the final checkpoint. We include all rollouts regardless of task success for world model training. For held-out validation, we collect another 100 rollout trajectories from intermediate checkpoints, with 20 trajectories each from epochs 75, 100, 125, 150, and 175.

The world model uses a frozen Stable Diffusion VAE to encode each image frame into a latent. We train a spatio-temporal DiT from scratch and inject action conditioning additively at each frame. As in policy training, we use an action chunk length of 16. Given two history frames and a 16-step action chunk, the model predicts the next four frames. At evaluation, we first predict four frames from two ground-truth history frames and the corresponding action chunk. We then predict the following four frames autoregressively from the model’s own predictions and the next action chunk. We average PSNR and LPIPS over the four predicted frames in each chunk. As shown in Table 3, the world model trained with selected-checkpoint rollouts achieves better prediction quality than the model trained with final-checkpoint rollouts. This result suggests that preserving policy coverage can benefit downstream world-model learning. Additional visualizations of the world model’s prediction are in App. H.8.

RolloutsPSNR↑LPIPS↓
Chunk 1
Selected27.90.112
Final26.80.120
Chunk 2
Selected25.20.136
Final24.10.145

Table 3. World-model prediction quality. Models are trained on demonstrations and rollouts from the selected or final checkpoint. Chunk 1 is predicted from two ground-truth context frames; Chunk 2 is predicted autoregressively from the model’s Chunk-1 output.

Abridged Related Work

Due to space constraints, we discuss key related work here and defer more comprehensive review to Appendix A. In language models, coverage of high-quality outputs has emerged as an important property for efficient posttraining and test-time scaling [16, 10]. Modern generative robot policies [13, 26, 7] are likewise followed by supervised finetuning, RL, or inference-time steering (Kim et al., 2025; Intelligence et al., 2025a; 2026), yet few works study how properties acquired during pretraining shape these later stages. PostBC and TMRL explicitly modify pretraining to improve subsequent RL by preserving action coverage or exploration [46, 19]. We instead study how coverage and reliability change during standard supervised pretraining itself and how they affect posttraining. Mirroring work showing that diffusion models can transition from generalization toward finite-sample fitting over training [9, 15], we characterize this learning dynamic in robot policies and connect it to closed-loop coverage and posttraining.

Conclusion and Future Work

This work studies how coverage–reliability dynamics during supervised robot-policy pretraining affect posttraining performance, and develops practical methods for offline checkpoint selection and coverage-preserving pretraining. An important direction for future work is to jointly develop principled data collection strategies and improved pretraining methods: demonstration datasets should have both broad coverage of task conditions and sufficient local density, and training objectives, architecture designs, and optimization strategies should improve reliability without sacrificing coverage or generalization. Another future direction is to develop a deeper theoretical understanding of how policy distributions evolve during pretraining and how these changes affect subsequent posttraining, which could guide both data collection and policy optimization. More broadly, our findings motivate a unified view of robot-policy pretraining and posttraining, in which pretrained policies are evaluated not only by their immediate performance but also by how effectively they can be improved through posttraining.

Acknowledgements

The authors thank all members of REALab and Huy Ha for their feedback on an earlier draft of this manuscript, and Jack Joler for data collection for the real-robot experiments. ZS is supported by the Stanford Graduate Fellowship. This work was supported in part by the NSF Award #2143601, #2037101, and #2132519. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.

AI Use Statement

We used generative AI tools to polish the writing, search for relevant literature, and help with parts of the code implementation. We did not use them to generate data, derive mathematical results, develop research ideas, or interpret experimental results. The authors checked all AI-assisted work: we tested the code, verified every cited paper against its source, and confirmed the text accurately reflects our contributions. We take full responsibility for the content of this paper.

Related Work

Pretraining–posttraining Interactions in Robot Learning

Language-model research provides strong motivation for studying pretraining and posttraining jointly. Recent work shows that performance within one stage in isolation can poorly predict the outcomes after the next. Models with similar pretraining validation loss can differ substantially in their finetuning performance [30], and prolonged pretraining can reduce subsequent finetuning performance [44]. In SFT–RL pipeline, higher SFT performance does not always predict stronger post-RL performance; stronger SFT checkpoints can even perform worse after the same RL procedure [23, 51]. Recent theory instead highlights coverage, the probability mass assigned to high-quality outputs, as an important property for test-time sampling [16, 10]. Empirically, large-kk pass@kk can predict post-RL performance better than SFT task performance [23]. TailSFT similarly improves posttraining by modifying SFT to preserve broader output coverage [31]. Robot learning increasingly follows a similar staged pipeline: BC or pretraining [13, 26, 7] followed by SFT [27, 6, 36], RL [21, 47], or inference-time steering [48, 12] (Intelligence et al. 2026). Yet few works study how properties learned during pretraining affect posttraining. PostBC proves that standard BC can fail to guarantee coverage of the demonstrator’s actions, which is necessary for RL finetuning to reach the demonstrator’s performance. It instead models the posterior over demonstrator action to guarantee such coverage and improve RL finetuning [46]. TMRL broadens coverage during pretraining by adding noise to the policy context, then lets RL control the noise level during finetuning [19]. Both methods redesign pretraining with noise injection specifically for downstream RL finetuning. We instead study how a policy’s finetunability changes over the course of standard supervised pretraining. We identify an evolving coverage–reliability tradeoff, show that it affects multiple forms of posttraining, and use this to design offline checkpoint selection criterion and training intervention.

Generalization and Memorization in Diffusion Models

Finite-data diffusion objectives admit closed form solutions that reproduce the training set, and several works characterize how model capacity, dataset size, conditioning, and optimization determine whether trained models approach these memorizing solutions [17, 2, 50, 5]. Recent papers further show a temporal separation in training dynamics: models first learn population-level structure that generalizes beyond the training set, while prolonged optimization can increasingly fit individual training examples [9, 15]. Related work on flow matching likewise studies why learned models generalize despite an empirical optimum that memorizes the data [4]. These findings have motivated interventions including early stopping, limiting model capacity [2, 50, 15], training from corrupted examples [14, 39], and perturbing conditioning input [43]. Most of this literature studies unconditional, class-conditioned, or text-conditioned generation of independent samples. Robot policies instead condition action generation on dense, high-dimensional observations and repeatedly sample the policy for multi-turn closed-loop interaction. He et al. [18] adopt a static view of trained diffusion policies and argue that, on small robot datasets, they largely retrieve actions from nearby training examples. We instead study how the policy distribution dynamically evolves over optimization, how this evolution changes closed-loop reliability and coverage and posttraining.

Robot Policy Evaluation and Model Selection

Closed-loop rollouts remain the standard for evaluating robot policies, but reliable comparisons require many trials and careful statistical treatment [28, 3, 42]. This cost motivates offline metrics for policy development and checkpoint selection. Mandlekar et al. [32] showed that validation loss can select substantially worse BC checkpoints than rollout evaluation. More recently, ABC found that diffusion validation loss was poorly correlated with real-world performance, whereas validation action error correlates more strongly with it [1]. Other work proposes manifold adherence [37] or task-critical action error [20] as more informative offline signals. Our setting differs because stochastic generative policies can have similar average error while differing substantially across repeated sampling. We therefore evaluate ate the full distribution of held-out action errors and select checkpoints by balancing coverage and reliability with a special focus for on their use in downstream posttraining.

Noise Injection in Robot Policies

Noise injection and local data augmentation methods have long been used to reduce covariate shift in behavior cloning. DART injects action noise during expert collection so demonstrations cover states reached after action perturbations [29]. Other work perturbs existing demonstrations and synthesizes corrective labels [24], uses noise augmentation to improve closed-loop stability [8], or generates corrective data under local continuity assumptions [25]. Recent theory shows that expert action noise can mitigate exponential compounding error in continuous-control imitation learning [52]. PostBC [46] and TMRL [19] similarly inject noise into actions or observation conditioning to broaden coverage for downstream RL. Our joint (o,a)(o,a)-augmentation instead learns the joint observation–action distribution from the demonstrations and generates new (o,a)(o, a) pairs together. Thus, the dataset itself determines both the observation variation and its compatible action, without additional expert collection, a hand-designed external noise distribution, or a local analytical dynamics model.

Success Polarization Across Evaluation Conditions

§ 3 showed that pass⁡@k\operatorname{pass}@k evolves asynchronously during supervised pretraining, and attributed this behavior to a redistribution of success across initial evaluation conditions. Here, we examine this redistribution directly at the condition level and make explicit why it causes larger-kk pass⁡@k\operatorname{pass}@k performance to decline first.

For the 200-demonstration Threading-D2 experiment, we evaluate every checkpoint on the same 500 evaluation conditions, with 32 rollout attempts per condition (pass⁡@32\operatorname{pass}@32). For each condition, we count the number of successful rollouts and track how the distribution of these counts changes over the course of supervised training.

Late training polarizes success across conditions. Figure 8 makes the redistribution described in § 3 explicit. Early in training, the improvement is largely uniform: the fraction of conditions with zero successes decreases and more conditions achieve higher success counts. Later, however, this trend reverses for the hardest conditions. Later in training, both the fraction of conditions with zero successes and the fraction with high success counts increase. Therefore, late training polarizes condition-wise success, improving reliability on a subset of conditions while reducing coverage on others.

Figure 8. Distribution of condition-level success counts over supervised pretraining. For each checkpoint, we run pass@32 on each of 500 evaluation conditions and bin each condition by its number of successful rollouts within 32 trials. The stacked bands show the fraction of conditions in each success-count bin; each vertical slice sums to one. During late training, the fraction of conditions with zero successes increases, while the fraction with high success counts also increases.

Why polarization produces asynchronous pass⁡@k\operatorname{pass}@k profiles. Let qτ(c)q_{\tau}(c) denote the one-rollout success probability for condition cc at checkpoint τ\tau. Then

Pk(τ)=Ec[1−(1−qτ(c))k].P_k(\tau) = \mathbb{E}_c\left[1-\left(1-q_{\tau}(c)\right)^k\right].

Pass⁡@1\operatorname{Pass}@1 averages qτ(c)q_{\tau}(c) linearly. For k>1k>1, fk(q)=1−(1−q)kf_k(q)=1-(1-q)^k is concave. Thus, shifting success probability from a lower-qq condition to a higher-qq condition while holding mean success fixed leaves pass⁡@1\operatorname{pass}@1 unchanged but lowers pass⁡@k\operatorname{pass}@k.

∂Pk∂τ=Ec[k(1−qτ(c))k−1∂qτ(c)∂τ].\frac{\partial P_k}{\partial\tau}=\mathbb{E}_c\left[k\left(1-q_{\tau}(c)\right)^{k-1}\frac{\partial q_{\tau}(c)}{\partial\tau}\right].

As kk increases, the factor (1−qτ(c))k−1\left(1-q_{\tau}(c)\right)^{k-1} suppresses changes on high-success conditions more strongly. Large-kk pass⁡@k\operatorname{pass}@k therefore depends more on whether low-success conditions remain solvable. Once those conditions begin to degrade, large-kk pass⁡@k\operatorname{pass}@k begins to decrease, even if small-kk pass⁡@k\operatorname{pass}@k continue to improve from improvements on already-reliable conditions.

Additional Results on RL Finetuning Across the Coverage–Reliability Tradeoff

Additional Results on Residual RL

Additional Results on Residual RL.

To examine more broadly how pretraining coverage and reliability affect residual RL finetuning (§ 3), we additionally finetune a substantially earlier checkpoint that retains more coverage but has lower pass⁡@1\operatorname{pass}@1 than the checkpoint selected by our criterion (§ 4).

Figure 9 shows that the relative advantage of this much earlier checkpoint depends on the available online budget. Its lower pass@1pass@1 can hurt early residual RL performance, while its greater coverage can become beneficial with more online budget; the exact tradeoff can be task dependent. While we do not claim that our criterion selects the optimal checkpoint for any online RL budget, it instead selects a checkpoint with substantially higher large-kk pass@kpass@k than the final checkpoint while maintaining comparable pass@1pass@1; and across the evaluated budgets, this checkpoint consistently finetunes better than the final checkpoint from the same supervised run.

Figure 9. Residual RL finetuning across the coverage–reliability tradeoff. We additionally finetune an earlier baseline checkpoint (ep.200/100/100 for Threading/Hammer Cleanup/Mug Cleanup) with lower pass@1 and higher coverage than the checkpoint selected by our criterion. Its performance relative to later checkpoints depends on the available online budget.

The exact ordering of checkpoints after RL finetuning can depend on both the RL algorithm and the task dynamics. Different RL methods explore and use online experience differently, and task dynamics affect how quickly differences between pretrained policies are amplified or corrected during closed-loop interaction. Our experiments establish the online rollout budget-dependent ordering of checkpoints for DICE-RL; we believe similar effects could also arise in other residual- or steering-based RL methods. Large-kk pass@kpass@k should therefore be interpreted as an offline measure of how broadly a pretrained policy succeeds before finetuning, rather than as a universal predictor of checkpoint ordering after arbitrary RL finetuning procedures.

Full-policy Finetuning

Full-policy Finetuning. While DICE-RL directly depends on the quality of actions sampled from the pretrained policy, we further investigate whether checkpoint selection matters when RL can update the entire action generator. We adopt Off-Policy Generative Policy Optimization (OGPO) [38] and use its official implementation. We first pretrain flow-matching policies with behavior cloning (BC). During RL finetuning, OGPO learns action-value critics through off-policy TD learning and updates the full generative policy using a PPO-style objective over sampled action-generation trajectories. All experiments use low-dimensional state observations and 200 demonstrations per task. We increase the critic warmup budget to 10610^{6} steps, since shorter warmup leads to substantially slower or stalled learning on MimicGen tasks. One possible explanation is that these tasks make accurate value estimation particularly challenging, although we do not fully isolate the cause of this sensitivity here.

Figure 10(a,Figure 10b) shows that flow-matching BC policies exhibit a similar coverage–reliability trade-off: best-of-kk validation action MSE increases for large kk and large-kk pass@kpass@k declines later in training. Notably, this observation occurs even with a relatively small five-layer MLP action network containing approximately 1.17M parameters, suggesting that the phenomenon is not limited to policies with high-capacity networks. Figure 10(c,Figure 10d) shows the evolution of validation error quantiles and the coverage–reliability criterion used for checkpoint selection.

Figure 10. Flow-matching policy pretraining and OGPO finetuning. Each row shows a MimicGen task with 200 demonstrations. (a) Best-of-kk validation action MSE. (b) pass@kk during BC pretraining. (c) Validation error quantiles. (d) Coverage–reliability scores and selected checkpoint. (e) OGPO finetuning from the selected versus final BC checkpoint. The selected initialization generally achieves better downstream RL performance.

Figure 10(e) compares OGPO finetuning from the selected and final BC checkpoints. The selected checkpoint achieves better performance on four of the five tasks. This result suggests that the effects of policy coverage and checkpoint selection extend beyond RL methods that directly rely on sampling and steering: even when the entire action generator is finetuned, the selected checkpoint can provide a more favorable initialization for subsequent RL. For OGPO in particular, a possible explanation is that it computes policy updates from action-generation trajectories sampled under the current policy. An initialization that places greater probability mass on useful actions may therefore generate more informative advantage-weighted updates, whereas actions that are rarely sampled initially may be difficult to reinforce under a finite interaction budget.

Diagnosing Overfitting in Diffusion Policies

The main text establishes that extended supervised pretraining can improve pass@1pass@1 (reliability) while reducing large-kk pass@kpass@k (coverage). Here, we examine how diffusion policies fit finite demonstrations over optimization time and across diffusion noise levels. We find that training denoising loss continues to decrease across noise levels, while held-out loss increases at later checkpoints toward the low-noise end. This deterioration is more pronounced in regions with sparse observation support. Furthermore, comparison with action-only diffusion trained on the same data shows that observation conditioning leads to greater late-training deterioration in held-out denoising loss, and such deterioration begins at higher diffusion noise levels. The final two low-noise denoising calls also appears to account for most of the difference in closed-loop pass@kpass@k performance between early and late checkpoints.

Held-out denoising deteriorates near the low-noise end. For checkpoint τ\tau and diffusion noise level tt, define the ϵ\epsilon-prediction MSE as

Lϵ,τ(t)=E(o,a0),ϵ[∥ϵ−ϵτ(at,o,t)∥22],(9)L_{\epsilon,\tau}(t)=\mathbb{E}_{(o,a_0),\epsilon}\left[\left\lVert\epsilon-\epsilon_\tau(a_t,o,t)\right\rVert_2^2\right], \tag*{(9)}

where at=αta0+σtϵa_t=\alpha_t a_0+\sigma_t\epsilon, ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I), and the expectation over (o,a0)(o,a_0) is evaluated separately on training and unseen demonstrations. Figure 11(a,Figure 11b) show this loss across diffusion noise levels for checkpoints at different training epochs, evaluated on training and unseen demonstrations, respectively. On training demonstrations, later checkpoints have lower MSE than earlier checkpoints at every noise level (Figure 11(a)). On unseen demonstrations, however, this ordering reverses at the low noise end: later checkpoints have higher MSE than earlier ones at the same noise levels (Figure 11(b)). This train–held-out divergence indicates overfitting at the low noise end of the diffusion process. Since Lϵ,τ(t)=SNR⁡(t)La0,τ(t)L_{\epsilon,\tau}(t)=\operatorname{SNR}(t)L_{a_0,\tau}(t), expressing the error in clean-action MSE units preserves the checkpoint ordering at each fixed noise level, although it changes the absolute loss magnitudes across noise levels.

Independent data splits show fitting to the training split. Here we explain the data split experiment in main text § 3 in full details. We train two policies, πτ(1)\pi_{\tau}^{(1)} and πτ(2)\pi_{\tau}^{(2)}, on disjoint demonstration sets D1D_1 and D2D_2, and evaluate both on the same held-out set D3D_3 with matched initial diffusion noise. Their predictions on D3D_3 first become more similar, then diverge as training continues. To test whether this late divergence reflects each policy moving toward its own training split, define the nearest-neighbor action distance

di(aˉ)=min⁡(o′,a′)∈Di∥aˉ−a′∥2(10)d_i(\bar{a})=\min_{(o',a')\in D_i}\left\lVert\bar{a}-a'\right\rVert_2 \tag*{(10)}

and the normalized ratio

Ri(τ)=Eo∼D3Ez[di(aˉτ(i)(o,z))]E(o,a⋆)∼D3[di(a⋆)].(11)R_i(\tau)=\frac{\mathbb{E}_{o\sim D_3}\mathbb{E}_{z}\left[d_i\left(\bar{a}_{\tau}^{(i)}(o,z)\right)\right]}{\mathbb{E}_{(o,a^\star)\sim D_3}\left[d_i(a^\star)\right]}. \tag*{(11)}

Here aˉτ(i)(o,z)\bar{a}_{\tau}^{(i)}(o,z) is an action chunk sampled from πτ(i)\pi_{\tau}^{(i)} given held-out observation oo and initial diffusion noise zz, and aˉ′\bar{a}' denotes the demonstrated action chunk of a training pair in DiD_i. The numerator measures its distance to the policy’s training split DiD_i; the denominator measures the corresponding distance for held-out expert actions. Thus Ri(τ)<1R_i(\tau)<1 means that policy samples lie closer to DiD_i than held-out expert actions do. A decrease in RiR_i indicates increasing relative proximity to the policy’s training split. As shown in Figure 2(b), Ri(τ)R_i(\tau) decreases later in training for both policies. Together with their increasing prediction divergence on D3D_3 in Figure 2(a), this supports increasingly split-specific fitting.

Observation conditioning exacerbates late-training overfitting. To examine the role of observation conditioning, we compare the image-conditioned denoiser ϵθτ(at,t,o)\epsilon_{\theta_\tau}(a_t,t,o) with an action-only denoiser ϵγτ(at,t)\epsilon_{\gamma_\tau}(a_t,t) trained on the same demonstration action chunks. Here θτ\theta_\tau and γτ\gamma_\tau denote the parameters at training epoch τ\tau. The action-only model uses the same denoising backbone and training settings but receives no observation input. We evaluate both models on identical noisy actions from held-out demonstrations. Let Lτm(t)L^m_\tau(t) denote the held-out ϵ\epsilon-prediction MSE at noise level tt for m∈{cond,uncond}m \in\{\mathrm{cond},\mathrm{uncond}\}. We measure the relative change from epoch 200 to 800 as

Δrelm(t)=L800m(t)−L200m(t)L200m(t),m∈{cond,uncond}.(12)\Delta_{\mathrm{rel}}^m(t)=\frac{L_{800}^m(t)-L_{200}^m(t)}{L_{200}^m(t)}, \qquad m\in\{\mathrm{cond},\mathrm{uncond}\}. \tag*{(12)}

The ratio is unchanged if ϵ\epsilon-prediction MSE is converted to clean-action MSE at the same noise level.

Figure 12 shows that Δrelm(t)\Delta_{\mathrm{rel}}^m(t) changes from negative to positive as noise decreases for both models. Continued training therefore improves held-out denoising at high noise but worsens it near the clean end. The reversal occurs at higher noise for the conditional model, which also shows substantially greater deterioration toward low noise. One possible explanation is that as noise decreases, the noisy action becomes more informative about individual training actions. Observation provides an additional identifier for training examples, potentially enabling memorization when the action is more heavily corrupted.

Figure 12. Observation conditioning exacerbates the deterioration in held-out denoising during late training. Relative change in held-out ϵ\epsilon-prediction MSE from epoch 200 to 800 for the image-conditioned and action-only models, evaluated on the same noisy action chunks. Negative values indicate lower held-out loss at epoch 800; positive values indicate higher loss. The conditional model exhibits held-out loss deterioration at higher diffusion noise levels.

We next examine how the conditional model’s late-training deterioration depends on the density of training observations near a held-out query. We approximate this density using the ground-truth simulator state because the policy is image-conditioned. For a query state ss and training episode ee, define

d(s,e)=min⁡s′∈e∥s−s′∥2.(13)d(s,e)=\min_{s'\in e}\lVert s-s'\rVert_2. \tag*{(13)}

Let r10(s)r_{10}(s) be the 10th-smallest d(s,e)d(s,e) across the 200 training episodes. We divide held-out queries into dense, medium, and sparse thirds according to r10r_{10}, with larger values indicating sparser support.

For each group g∈{dense,medium,sparse}g\in\{\mathrm{dense},\mathrm{medium},\mathrm{sparse}\}, let Lτ,gcond(t)L^{\mathrm{cond}}_{\tau,g}(t) be the conditional model’s mean held-out ϵ\epsilon-prediction MSE over queries in that group. Using the same relative-change measure as Eq. D.4, we compute

Δrel,g(t)=L800,gcond(t)−L200,gcond(t)L200,gcond(t).(14)\Delta_{\mathrm{rel},g}(t)=\frac{L^{\mathrm{cond}}_{800,g}(t)-L^{\mathrm{cond}}_{200,g}(t)}{L^{\mathrm{cond}}_{200,g}(t)}. \tag*{(14)}

Figure 13 shows that the reversal from improvement to deterioration starts from higher noise levels for sparsely supported observations than for densely supported ones. Near the clean end, the relative increase in held-out loss is also generally larger in the less-supported groups. Thus, the noise range in which continued training harms generalization depends on how well the observation is covered by the training demonstrations.

Local-support analysis. To relate this pattern to finite-data support, we count training examples compatible with a denoising query in both observation conditioning and action space.

Assume the training windows Dn={(Zi,Ai)}i=1nD_n = \{(Z_i, A_i)\}_{i=1}^{n} are i.i.d. samples from a joint density pZ,Ap_{Z,A} that is continuous near the held-out query. Here Zi∈RdzZ_i \in\mathbb{R}^{d_z} is a conditioning representation and Ai∈RdaA_i \in\mathbb{R}^{d_a} is the clean expert action. Take an independent held-out pair (Z⋆,A⋆)=(z⋆,a⋆)(Z^\star, A^\star) = (z^\star, a^\star) and one initial diffusion noise draw ϵ\epsilon. At diffusion time tt,

xt=αta⋆+σtϵ,rt:=σtαt,a~t:=xtαt=a⋆+rtϵ.(15)x_t = \alpha_t a^\star+ \sigma_t \epsilon,\qquad r_t := \frac{\sigma_t}{\alpha_t},\qquad\widetilde{a}_t := \frac{x_t}{\alpha_t} = a^\star+ r_t\epsilon. \tag*{(15)}

For a candidate clean training action AiA_i, the forward diffusion process gives

p(xt∣Ai)∝exp⁡(−∥xt−αtAi∥22σt2)(16)p(x_t \mid A_i) \propto\exp\left(-\frac{\lVert x_t-\alpha_t A_i\rVert^2}{2\sigma_t^2}\right) \tag*{(16)}
=exp⁡(−∥Ai−a~t∥22rt2).(17)= \exp\left(-\frac{\lVert A_i-\widetilde{a}_t\rVert^2}{2r_t^2}\right). \tag*{(17)}

Eq. D.9 shows that the likelihood depends on ∥Ai−a~t∥2/rt\lVert A_i-\widetilde{a}_t\rVert^2/r_t. An action at distance crtcr_t from a~t\widetilde{a}_t has relative likelihood e−c2/2e^{-c^2/2}. Therefore, for any fixed relative-likelihood threshold, the radius of the corresponding set of clean actions is proportional to rtr_t. As rtr_t decreases, this set shrinks around a~t\widetilde{a}_t.

We now introduce a simple local-support model. Fix a radius hh in conditioning space, and replace the smooth Gaussian weighting in Eq. D.9 with a hard action-space cutoff at rtr_t. We then count training examples that satisfy both conditions:

Nloc(t)=∑i=1n1{∥Zi−z⋆∥2≤h}1{∥Ai−a~t∥2≤rt}.(18)N_{\mathrm{loc}}(t) = \sum_{i=1}^{n} \mathbf{1}\{\lVert Z_i-z^\star\rVert_2\le h\}\mathbf{1}\{\lVert A_i-\widetilde{a}_t\rVert_2\le r_t\}. \tag*{(18)}

Thus, an example is counted if its conditioning representation lies within distance hh of z⋆z^\star and its clean action lies within distance rtr_t of a~t\widetilde{a}_t. The cutoffs define an analytical neighborhood, not which examples the network actually uses.

Let qtq_t be the probability that one i.i.d. training pair is counted by Eq. D.10. Then

qt=Pr⁡(∥Z−z⋆∥≤h,∥A−a~t∥≤rt)(19)q_t = \Pr(\lVert Z-z^\star\rVert\le h,\lVert A-\widetilde{a}_t\rVert\le r_t) \tag*{(19)}
=∫∥z−z⋆∥≤h∫∥a−a~t∥≤rtpZ,A(z,a) da dz.(20)= \int_{\lVert z-z^\star\rVert\le h}\int_{\lVert a-\widetilde{a}_t\rVert\le r_t}p_{Z,A}(z,a)\,da\,dz. \tag*{(20)}

It follows that Nloc(t)∼Binomial⁡(n,qt)N_{\mathrm{loc}}(t)\sim\operatorname{Binomial}(n,q_t) and E[Nloc(t)]=nqt\mathbb{E}[N_{\mathrm{loc}}(t)]=nq_t.

To evaluate qtq_t, substitute z=z⋆+huz = z^\star+ hu and a=a~t+rtva = \tilde{a}_t + r_t v:

qt=hdzrtda∫∥u∥≤1∫∥v∥≤1pZ,A(z⋆+hu,a~t+rtv) dv du.(21)q_t = h^{d_z} r_t^{d_a} \int_{\lVert u\rVert\le1} \int_{\lVert v\rVert\le1} p_{Z,A}(z^\star+ hu,\tilde{a}_t + r_t v)\,dv\,du. \tag*{(21)}

Near the low-noise end, rt→0r_t \to0 and a~t=a⋆+rtϵ→a⋆\tilde{a}_t = a^\star+ r_t\epsilon\to a^\star. In the joint limit h,rt→0h,r_t \to0, assuming pZ,Ap_{Z,A} is continuous and positive at (z⋆,a⋆)(z^\star,a^\star), Eq. D.13 gives

E[Nloc(t)]=VdzVdanpZ,A(z⋆,a⋆)hdzrtda[1+o(1)],(22)\mathbb{E}[N_{\mathrm{loc}}(t)] = V_{d_z}V_{d_a}np_{Z,A}(z^\star,a^\star)h^{d_z}r_t^{d_a}[1+o(1)], \tag*{(22)}

where VdV_d is the volume of the unit ball in Rd\mathbb{R}^d.

Factoring pZ,A(z⋆,a⋆)=pZ(z⋆)pA∣Z(a⋆∣z⋆)p_{Z,A}(z^\star,a^\star)=p_Z(z^\star)p_{A\mid Z}(a^\star\mid z^\star) gives

E[Nloc(t)]≈CnpZ(z⋆)hdz⏟local conditioning masspA∣Z(a⋆∣z⋆)rtda⏟local action mass at noise scale rt,(23)\mathbb{E}[N_{\mathrm{loc}}(t)] \approx Cn \underbrace{p_Z(z^\star)h^{d_z}}_{\text{local conditioning mass}}\underbrace{p_{A\mid Z}(a^\star\mid z^\star)r_t^{d_a}}_{\text{local action mass at noise scale }r_t}, \tag*{(23)}

with C=VdzVdaC = V_{d_z}V_{d_a}. Sparser local conditioning support lowers the first factor; lower diffusion noise reduces rt=σt/αtr_t = \sigma_t/\alpha_t and hence the second factor. Both lead to fewer locally compatible training examples. This reduced support is consistent with greater vulnerability to finite-data overfitting, but the calculation itself does not predict when a trained denoising network’s held-out loss will increase.

The r10r_{10} measure used in Figure 13 is computed in simulator-state space, rather than the policy’s learned conditioning representation, and serves as a proxy for local conditioning support.

Low-noise changes affect closed-loop pass@kk. To test whether changes to low-noise denoising affect closed-loop success, we divide the 10 DDIM inference calls into five consecutive two-call bands, Q1–Q5, and replace the late checkpoint’s weights with the early checkpoint’s weights for one band at a time. Patching Q1–Q4 has little effect on pass@1 or pass@10, whereas replacing Q5 recovers most of the earlier checkpoint’s pass@10 advantage while reducing pass@1 (Figure 11(c)). Thus, for this checkpoint pair, the final two calls account for most of the observed differences in the pass@kk profile.

Full Coverage–Reliability Results and Checkpoint Selection

We report the complete results across five MimicGen tasks with 200 and 500 demonstrations, together with multitask RoboCasa pretraining. For each setting, we show best-of-kk validation loss, closed-loop pass@kk, the full error-quantile profile, coverage–reliability checkpoint selection, and downstream residual RL or supervised finetuning.

Hammer Cleanup D1 results for 200 demonstrations

Figure Hammer Cleanup D1, 200 demos. Hammer Cleanup D1 (200 demos). (a) Best-of-kk val error. (b) Pass@kk success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.

Hammer Cleanup D1 results for 500 demonstrations

Figure Hammer Cleanup D1, 500 demos. Hammer Cleanup D1 (500 demos). (a) Best-of-kk val error. (b) Pass@kk success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.

Mug Cleanup D1 results for 200 demonstrations

Figure Mug Cleanup D1, 200 demos. Mug Cleanup D1 (200 demos). (a) Best-of-kk val error. (b) Pass@kk success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.

Mug Cleanup D1 results for 500 demonstrations

Figure Mug Cleanup D1, 500 demos. Mug Cleanup D1 (500 demos). (a) Best-of-kk val error. (b) Pass@kk success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.

Square D2 results for 200 demonstrations

Figure Square D2, 200 demos. Square D2 (200 demos). (a) Best-of-kk val error. (b) Pass@kk success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.

Square D2 results for 500 demonstrations

Figure Square D2, 500 demos. Square D2 (500 demos). (a) Best-of-kk val error. (b) Pass@kk success rate. (c) Full error quantiles. (d) Coverage–Reliability Balance. (e) FT from early vs. late ckpt.

The same learning dynamics hold across tasks and dataset sizes. Figure 14 and Figure 15 show a consistent pattern across all settings. Best-of-kk loss and pass@kk favor increasingly earlier checkpoints as kk increases, while the error-quantile curves exhibit the same tradeoff between favorable-tail coverage and average-case reliability as described in § 4. The pattern holds for both single-task MimicGen BC and multitask RoboCasa pretraining. Importantly, pass@1 and kk=1 best-of-kk loss (mean held-out action MSE) generally favor later checkpoints after large-kk pass@kk has begun to decline.

Figure 14. Full MimicGen results across tasks and dataset sizes. Results for all five tasks with 200 and 500 demonstrations. Each row reports best-of-kk validation loss, closed-loop pass@kk, the full error-quantile profile, coverage–reliability checkpoint selection, and downstream residual RL finetuning.

Figure 15. Full RoboCasa multitask pretraining results. Best-of-kk validation loss, pass@kk, error quantiles, coverage–reliability checkpoint selection, and downstream SFT.

Increasing the MimicGen dataset from 200 to 500 demonstrations generally attenuates, and in several cases delays, the late-stage loss of coverage. While two discrete dataset sizes are insufficient for a scaling-law claim, this trend is consistent with the finite-data overfitting diagnosis in Appendix § 14.

Coverage–reliability balance criterion agrees with a closed-loop oracle. We apply the coverage–reliability criterion from § 4 to every setting. For comparison, we define the closedloop oracle directly from the full pass@k table. Here we describe how we compute the checkpoint selected from oracle closed-loop pass@k runs. For each kk, we normalize pass@k across checkpoints as

gτ(k)=max⁡τ′Pk(τ′)−Pk(τ)max⁡τ′Pk(τ′)−min⁡τ′Pk(τ′),(24)g_{\tau}(k) = \frac{\max_{\tau'} P_k(\tau') - P_k(\tau)}{\max_{\tau'} P_k(\tau') - \min_{\tau'} P_k(\tau')}, \tag*{(24)}

where Pk(τ)P_k(\tau) denotes pass@k at checkpoint τ\tau. We then select

τoracle=arg⁡min⁡τ max⁡kgτ(k).(25)\tau_{\mathrm{oracle}} = \underset{\tau}{\arg\min}\ \max_k g_{\tau}(k). \tag*{(25)}

This oracle selects the checkpoint whose worst normalized gap from the best checkpoint is smallest across the full pass@k profile. Across tasks and dataset sizes, the checkpoint selected by our proposed offline criterion lies near the closed-loop oracle.

Selected checkpoints also support stronger finetuning. For all MimicGen settings, residual RL initialized from the selected checkpoint outperforms RL from the final BC checkpoint. RoboCasa shows the same pattern under SFT: initialization from the selected pretraining checkpoint yields stronger pass@1 and pass@32 throughout finetuning. These results extend the main-text finding that the coverage–reliability balance retained during supervised training leads to stronger finetuning performance compared to the final checkpoint from the same run.

Robustness to Generator Checkpoint Selection

Selection is robust to training horizon. We additionally extend the 200-demonstration Threading D2 run from 800 to 1600 epochs. Figure 16 shows that the same pass@k and error-quantile dynamics persist under substantially longer training, while the criterion continues to select an intermediate checkpoint.

Figure 16. Extended training on Threading D2. The coverage–reliability dynamics persist to 1600 epochs, and the offline criterion continues to select an intermediate checkpoint.

We further recompute both selectors using only checkpoints available up to each training horizon HH. Figure 17 shows that the offline-selected checkpoint closely tracks the corresponding closed-loop oracle as additional checkpoints become available, rather than simply moving toward the end of training. The selection criterion therefore does not depend sensitively on the chosen training horizon.

Robustness to training horizon

Figure 17. Robustness to training horizon. Offline selection tracks the closed-loop oracle as HH increases.

Additional Results on JointAug

Additional Task

We additionally evaluate JointAug on 3-Piece Assembly (D2) with 500 demonstrations. This task provides a more extreme example of the effect of pretraining coverage on residual RL finetuning. As shown in Figure 18, JointAug raises pass@10 to roughly 0.8 during BC pretraining, compared with below 0.1 for the baseline. Moreover, residual RL from the final JointAug checkpoint then reaches over 0.8 success, while finetuning from the baseline checkpoint remains near zero throughout the evaluated interaction budget. Thus, on this task, the pretraining difference changes residual RL from not working to successful finetuning.

Figure 18. JointAug enables residual RL finetuning on 3-Piece Assembly (D2). Left: JointAug substantially improves pass@1 and pass@10 during BC pretraining. Right: residual RL from the final JointAug checkpoint reaches over 0.8 success, while finetuning from the baseline checkpoint remains near zero.

Robustness to Generator Checkpoint Selection

By default, JointAug uses a uniform mixture of five generator checkpoints, spaced 100 epochs apart, around the transition identified by our train–held-out nearest-neighbor ratio (Appendix § 18.§ 3). We use this short checkpoint window to average supervision across the generator’s transition from population-level generalization toward increasing training-data bias.

To test how sensitive JointAug is to this heuristic choice, we compare the five-checkpoint mixture against distilling from individual generator checkpoints from the same region. Figure 19 shows that the resulting policy curves are similar across these choices on all four tasks. In particular, single generator checkpoints preserve most of the pass@10 improvement over the baseline. The five-checkpoint mixture performs similarly overall and gives slightly higher pass@1 on Mug Cleanup and Threading. Together, these results show that JointAug does not depend strongly on selecting one particular generator checkpoint or checkpoint mixture. Mixing several nearby checkpoints provides a simple way to reduce checkpoint sensitivity.

Figure 19. JointAug is relatively robust to generator checkpoint selection. We compare the five-checkpoint mixture used in our main experiments against individual generator checkpoints from the same run. Across four tasks, single-checkpoint variants have similar pass@10 gains over the baseline, while the mixture performs similarly overall and slightly improves pass@1 on Mug Cleanup and Threading.

Visualizing How JointAug Changes the Action Distribution

To understand why JointAug improves both coverage and reliability, we inspect the action distribution at held-out observations. For each state, we sample 64 action chunks from an early base line checkpoint (ep50), the final baseline checkpoint (ep800), and the final JointAug checkpoint (ep800), and jointly project the samples with PCA. We report both the mean action MSE and the best-of-64 MSE to the expert action. The former measures the accuracy of a typical sample, while the latter measures whether the sampled distribution still contains an action close to the expert.

Figure 20 shows that the early baseline checkpoint produces a broad action distribution: across the four states, best-of-64 MSE is very small, but mean MSE can go up to 2.32×10−32.32 \times10^{-3}. Thus, lucky draw samples lie very close to the expert, while the majority of samples are much farther away. This provides coverage but poor reliability for stable closed-loop execution.

Figure 20. JointAug retains favorable-tail coverage while improving typical action accuracy. For four held-out Mug Cleanup states, we jointly project 64 action samples from baseline ep50, baseline ep800, and JointAug ep800 with PCA. Stars mark expert actions and rings mark the minimum-MSE sample. The baseline ep50 is dispersed with low best-of-64 but high mean MSE; baseline baseline ep800 contracts and lowers mean MSE but worsens best-of-64 MSE; JointAug ep800 achieves best of both worlds.

By ep800, the baseline distribution contracts substantially. Mean MSE improves a lot from the early checkpoint, but best-of-64 MSE worsens on all four states. The later baseline therefore produces more consistent samples, but its best samples are farther from the expert, thus losing coverage.

The final JointAug policy checkpoint combines the favorable properties of both checkpoints: its action clouds remain broader than the final baseline while being centered closer to the expert. Mean MSE falls further to 0.180.18–0.64×10−30.64 \times10^{-3}, and best-of-64 MSE decreases to 0.060.06–0.23×10−30.23 \times10^{-3}. Thus, JointAug achieves lower mean and best-of-64 error than the final baseline on every state, while nearly matching or improving the favorable tail of the early baseline.

Together, these visualizations provide a post-hoc explanation for the pass@1 and large kk pass@kk improvement from JointAug. Jointly generated (o,a)(o, a) pairs expand supervision beyond the finite demonstrations while preserving observation–action correspondence. The resulting policy retains a broader set of low-error actions without the large average errors of the early baseline.

Additional Baseline - Context-Smoothed Pre-training in TMRL

We additionally compare against the context-smoothed pretraining part from TMRL [19], which is designed to increase action coverage by reducing the amount of information available in the policy context. During pretraining, TMRL applies a forward diffusion noise to the observation context,

q(c~t∣c)=N(αˉtc,(1−αˉt)I)q(\tilde{c}_t \mid c) = \mathcal{N}\left(\sqrt{\bar{\alpha}_t}c, (1-\bar{\alpha}_t)I\right)

then conditions the policy on both the corrupted context c~t\tilde{c}_t and its corresponding noise level tt. The intuition is that increasing tt aliases a wider neighborhood of contexts, so the corresponding action distribution can borrow actions from other demonstrations rather than fitting only the conditional distribution at the original observation. We tune the maximum context-noise level for each task and report the best-performing setting; full implementation details are in Appendix § 18.§ 5.

Figure 21 shows that context smoothing does not preserve coverage well in our setting. Across tasks, its pass@10 remains below JointAug and often declines substantially with further training; its pass@1 also tends to degrade

Figure 21. Context-Smoothed Pre-training from TMRL does not preserve coverage under prolonged training. Across five MimicGen tasks, TMRL generally underperforms JointAug in pass@10 and often loses both pass@10 and pass@1 at later checkpoints.

One likely reason is that context smoothing relies on local structure in the latent space where noise is applied. Isotropic Gaussian perturbations are useful when nearby context representations correspond to observations with compatible actions, since smoothing can then borrow coherent behaviors from neighboring contexts. In our image-based diffusion policies, however, this assumption may not hold. The observation encoder is trained jointly from task demonstrations and is not explicitly regularized so that Euclidean neighborhoods in its latent space preserve action similarity. Context corruption can therefore mix observations that require different actions. We note that differs from the image-based experiments in Hong et al. [19], which smooth pretrained VLM embeddings.

More generally, context smoothing increases overlap between conditional action distributions, but by design does not directly constrain the favorable tail of the action distribution as supervised training proceeds. It therefore does not guarantee preservation of large-kk pass@k.

Additional Results and Visualizations

Sampling Configuration Ablations

Our main experiments use DDIM sampling with 10 denoising steps and η=0\eta= 0. The coverage–reliability tradeoff develops during diffusion-policy training rather than being induced by a particular sampling procedure. Figure 22 compares DDIM with 10 or 100 denoising steps, η∈{0,0.5,1}\eta\in\{0, 0.5, 1\}, and DDPM with 100 steps. Across all settings, pass@10 peaks earlier and declines with further training, while pass@1 continues to improve or remains stable. The same trend across sampling configurations shows that the late-training coverage loss is a property of the trained denoising model rather than the sampling configuration used for evaluation.

Figure 22. Pass@kk training dynamics are stable across sampling configurations. Pass@1 and pass@10 are evaluated using DDIM with different numbers of denoising steps and η\eta, and using DDPM. All settings show the same late-training decline in pass@10.

Learning-rate Schedule Ablations

Our main experiments show that large-kk pass@k peaks earlier in supervised pretraining than pass@1, and can decline as pass@1 continues to improve. Because the learning-rate schedule controls how aggressively the policy is updated over training, it can affect when these changes occur. Related work on diffusion models likewise finds that optimization settings can shift the onset of finite-data fitting ([9, 49]). We therefore test whether changing the learning-rate schedule fundamentally changes the coverage–reliability tradeoff itself, or primarily shifts the timescales at which it appears. Across the schedules we evaluate, the same qualitative pattern remains: pass@10 peaks earlier and declines at later epochs, while pass@1 improves over a longer portion of training.

Setup. For single-task MimicGen, our main experiments follow the optimizer configuration of the public Diffusion Policy implementation [13]: AdamW with learning rate 10−410^{-4}, β1=0.95\beta_1 = 0.95, β2=0.999\beta_2 = 0.999, weight decay 10−610^{-6}, and 500 warmup steps followed by a single half-cosine decay from 10−410^{-4} to zero over the full training horizon. We use this standard recipe throughout experiments included in the main text.

We repeat the 200-demonstration Threading 2D experiment with four alternative schedules while keeping all other training settings fixed. (1) Constant 10−410^{-4}, which removes annealing while preserving the maximum learning rate of the default schedule; (2) constant 5×10−55 \times10^{-5} to test whether reducing the learning rate shifts both the improvement in pass@1 and the decline in large kk pass@kk; (3) cosine decay from 10−410^{-4} to 10−510^{-5} which preserves the monotone decay of the default schedule but removes its near-zero learning rates late in training; (4) periodic cosine with four cycles to test whether repeatedly increasing and decreasing the learning rate changes the same trend or instead produces different behavior.

Together, these settings cover fixed, monotone-decay, and cyclic learning-rate schedules, while isolating the effects of learning-rate magnitude, annealing, and the final learning-rate floor. In Figure 23, each panel overlays one ablated schedule with the default single cosine schedule (gray); solid and dashed curves denote pass@10 and pass@1 respectively.

Figure 23. Learning-rate schedule ablation on Threading D2 Each panel overlays one ablated schedule (colored) with the default single-cosine schedule (gray). Solid and dashed curves denote pass@10 and pass@1, respectively. Across schedules, large-kk pass@kk peaks earlier and declines as pass@1 improves later in training; changing the learning-rate schedule primarily shifts the timing of these changes.

Results. Removing learning-rate decay doesn’t change the decline in large kk pass@kk. With constant 10−410^{-4}, pass@10 increases early and then decreases over later epochs, while pass@1 remains slightly below the default run through most of training. Keeping the learning rate fixed at its maximum value therefore does not preserve the higher late-training pass@10 without also changing pass@1. Likewise, changing the endpoint of the cosine schedule from zero to 10−510^{-5} produces nearly the same pass@1 and pass@10 curves as the default schedule. These two comparisons show that the decline in large-kk pass@kk is not specific to either learning-rate decay or the near-zero learning rates reached at the end of the default schedule.

With constant 5×10−55 \times10^{-5}, both metrics change more slowly. Pass@10 rises more gradually, reaches its maximum later, and begins declining later than under the default schedule. Pass@1 also improves more gradually and remains lower at epoch 800. Thus, the higher pass@10 retained at later epochs is accompanied by lower pass@1; reducing the learning rate shifts both sides of the tradeoff later in training rather than preserving coverage at the same reliability.

Periodic cosine produces larger cross checkpoint fluctuations but does not change the longer-term trend. Pass@10 still reaches its highest values earlier and decreases later in training, while pass@1 remains higher at later epochs. Repeated learning-rate cycling therefore makes training less stable across checkpoints without removing the coverage–reliability tradeoff. Among the schedules tested here, the default single-cosine schedule also gives the smoothest learning curves and the highest late-training pass@1, supporting its use as the default schedule in our main experiments.

Across all schedules tested, large-kk pass@kk still peaks earlier and declines as pass@1 improves later in training. Smaller constant learning rates shift both changes to later epochs, while periodic cosine produces larger fluctuations across checkpoints. Thus, the learning-rate schedule changes when the coverage–reliability tradeoff appears, but does not fundamentally escape it. This mirrors Bonnaire et al. [9], who find that changing the optimizer shifts the onset of generalization and memorization in diffusion models while preserving their ordering. More broadly, Wu et al. [49] shows that learning rate can directly regularize finite-data fitting of the diffusion score at small noise. Designing new optimization strategies that preserve large-kk coverage without hurting improvements in pass@1 is an exciting direction for future work.

Denoising Network Size Ablations

We next examine how denoiser network capacity affect the observed learning dynamics between coverage and reliability. We vary the U-Net channel widths while keeping the visual encoder, data, optimizer, learning-rate schedule, and sampler unchanged. We test four denoiser capacities contain 24M, 78M, 277M, and 1.0B parameters. As shown in Figure 24, pass@10 peaks between roughly epochs 200–300 for every U-Net model size and then eventually declines to a similar level by epoch 800. Therefore, the loss of coverage during late training is not specific to the capacity of the denoising network.

Training success rate over training epochs for denoiser widths

Figure 24. The late-training coverage loss persists across denoiser widths. We vary the U-Net from 24M to 1.0B parameters while keeping the encoder, data, optimizer, learning-rate schedule, and sampler fixed. All four models show a similar peak and subsequent decline in pass@10.

EMA schedule Ablations

EMA changes how quickly the network weights used for evaluation track the current denoiser weights. A possible explanation for the late-training drop in large kk pass@kk is therefore that the EMA schedule forgets the earlier, higher-coverage weights too quickly. If so, increasing the EMA memory should substantially reduce or remove the decline. Let θt\theta_t denote the current model parameters and θˉt\bar{\theta}_t the EMA parameters. The EMA weights are updated as

θˉt=βtθˉt−1+(1−βt)θt,\bar{\theta}_t = \beta_t \bar{\theta}_{t-1} + (1-\beta_t)\theta_t,

where the decay follows the Diffusion Policy warmup schedule

βt=clip⁡(1−(1+stγ−1)−p,βmin⁡,βmax⁡),st=max⁡(0,t−t0−1).\beta_t = \operatorname{clip}\left(1-\left(1+\frac{s_t}{\gamma^{-1}}\right)^{-p},\beta_{\min},\beta_{\max}\right), \qquad s_t=\max(0,t-t_0-1).

Our default uses γ−1=1\gamma^{-1}=1, p=0.75p=0.75, t0=0t_0=0, βmin⁡=0\beta_{\min}=0, and βmax⁡=0.9999\beta_{\max}=0.9999.

We ablate over two key hyperparameters, the power pp and βmax⁡\beta_{\max}, each controlling different parts of the EMA memory schedule. Increasing pp makes (1+st)−p(1+s_t)^{-p} decay faster, so βt\beta_t approaches one earlier in training. The EMA therefore puts less weight on the current network at each update and retains older weights for longer. In other words, larger pp gives the EMA a longer memory earlier in training. By contrast, βmax⁡\beta_{\max} becomes to matter once this growth reaches the cap value. Increasing βmax⁡\beta_{\max} allows βt\beta_t to move even closer to one before being clipped, giving the EMA a longer maximum memory later in training.

We compare the default setting (p=0.75p = 0.75, βmax⁡=0.9999\beta_{\max} = 0.9999) against (0.9,0.9999)(0.9, 0.9999) and (0.9,0.99995)(0.9, 0.99995). The first comparison increases the EMA memory earlier in training. The second keeps pp fixed and increases the maximum memory later in training; for p=0.9p = 0.9, the 0.9999 cap is reached around epoch 309, while the 0.99995 cap is reached around epoch 668.

Figure 25 shows comparisons between these different temporal averaging schedules. Holding βmax⁡=0.9999\beta_{\max} = 0.9999 fixed, increasing pp from 0.75 to 0.9 gives the EMA a longer memory earlier in training. Accordingly, the p=0.9p = 0.9 policy lags more while the underlying current model is improving: pass@10 rises more slowly and peaks later. Its decline after the peak is also more gradual because the EMA continues to place more weight on parameters from earlier checkpoints.

Figure 25. Longer EMA memory delays and smooths the pass@10 decline but does not fundamentally remove it. Holding βmax⁡=0.9999\beta_{\max} = 0.9999 fixed, increasing pp from 0.75 to 0.9 gives the EMA a longer memory earlier in training, causing pass@10 to peak and decline later. Holding p=0.9p = 0.9 fixed, increasing βmax⁡\beta_{\max} from 0.9999 to 0.99995 allows the EMA memory to continue growing later in training and further smooths the decline. Pass@10 still decreases under all schedules.

Holding p=0.9p = 0.9 fixed isolates the effect of βmax⁡\beta_{\max}. The two schedules are identical until the βmax⁡=0.9999\beta_{\max} = 0.9999 run reaches its cap; after that, βmax⁡=0.99995\beta_{\max} = 0.99995 allows the EMA memory to keep growing and therefore retains earlier parameters for longer. This further smooths the late-training change in pass@10 for a little bit, but does not fundamentally prevent its decline. Thus, increasing either the early growth or the maximum length of the EMA memory can delay and smooth the loss of coverage, but neither prevents it.

Isolating Coverage Loss to the Denoising Network

To determine how much of the late-training coverage loss can be attributed to the visual encoder or the denoising network, we run three complementary experiments. (1) Freeze the encoder: we freeze the visual encoder at epoch 150 and continue training only the denoiser, testing whether removing further encoder updates changes the subsequent coverage decline. (2) Fresh denoiser with a fixed encoder: we train a new denoiser from scratch while keeping either the epoch-150 or epoch-800 encoder fixed throughout training. This isolates whether denoiser optimization alone can reproduce the peak-then-decline pattern under a fixed representation. (3) Low-dimensional observations: we train diffusion policies directly from state observations, removing the visual encoder entirely. This tests whether the same coverage–reliability tradeoff arises without any learned visual representations.

Freezing the encoder does not stop the coverage decline. We choose epoch 150 according to a split-data representation comparison shows that the nearest-neighbor structure of independently trained encoders has stabilized by this point. Starting from the epoch-150 checkpoint, we replace the encoder with its EMA copy, freeze all encoder parameters, and continue training the denoiser till the end of the training horizon. Figure 26 shows that pass@10 still declines at later epochs, with a similar pass@1 trend to the original run. Continued encoder updates are therefore not responsible for the coverage loss.

Figure 26. Freezing the encoder does not prevent the late-training large kk pass@kk decline. From epoch 150 onward, the encoder is fixed and only the denoiser is trained. Pass@10 still decreases at later epochs. The overall training dynamics is largely the same as baseline run.

A fresh denoiser reproduces the decline with a frozen learned visual representation. We next randomly initialize a new denoiser and train it from scratch for 800 epochs while keeping the encoder fixed throughout training. We run this experiment with both the epoch-150 and epoch-800 encoders, using the same denoiser initialization. As shown in Figure 27, pass@10 first increases and then declines for both frozen representations. The decline therefore reappears even when the encoder never changes and the denoiser is trained entirely from scratch.

Figure 27. Fresh denoisers reproduce the pass@10 decline with fixed encoder representations. New denoisers are trained from scratch with either the epoch-150 or epoch-800 encoder frozen throughout training. Both show a later decline in large kk pass@kk.

The same trend appears without a visual encoder. Finally, we train diffusion policies directly from low-dimensional state observations. Across all five tasks in Figure 28, baseline pass@10 again declines at later checkpoints, while pass@1 remains stable or improves. Thus, the late-training coverage loss happens in the absence of any visual encoder.

Figure 28. Late-training coverage loss also occurs with low-dimensional state observations. Across five tasks, baseline diffusion policies lose pass@10 at later checkpoints despite using no visual encoder. Joint (o,a)(o, a) augmentation preserves substantially more pass@10.

Together, these experiments show that visual-encoder overfitting cannot account for most of the late training coverage loss. Freezing the encoder does not eliminate the decline, a newly initialized denoiser reproduces it under a frozen representation, and the same trend appears in low-dimensional state-based policies with no visual encoder at all. The visual encoder can still affect the magnitude of the effect, but the peak-then-decline behavior arises from denoiser training itself.

This result matches the generalization–memorization mechanism identified in prior work on diffusion models. With finite training data, the empirical denoising objective is minimized by the empirical score rather than the population score; and prolonged optimization can drive the denoiser toward memorizing training examples [17, 2, 50]. Network training initially favors the smoother, population-like components of the score and fits the higher frequency, dataset-specific components later, particularly at low noise [9, 15]. Our results show the same transition in conditional diffusion policies and it persists after removing visual-representation learning and is concentrated in the denoising network.

Best-of-k Off-Manifold Error Reveals the Same Coverage–Reliability Tradeoff

We additionally ask whether the coverage–reliability tradeoff appears under a different offline metric of policy quality. Following the off-manifold metric of Pan et al. [37], we perturb expert actions at high-ambiguity training states, execute the perturbed actions to obtain OOD observations, and

Training rolloutsh=1h = 1h=2h = 2h=3h = 3
Selected0.08000.10330.1186
Final0.08150.10590.1219
Training rolloutsh=1h = 1h=2h = 2h=3h = 3
Selected0.09920.12750.1458
Final0.10030.12910.1484

measure the residual of each predicted action from the span of expert actions at nearby training states. We extend from the one policy prediction sample to best-of-kk over stochastic policy samples.

We compare epochs 400 and 800 of 3-Piece Assembly run. The later checkpoint has higher pass@1 (0.328 vs. 0.2988) but lower pass@10 (0.591 vs. 0.639). For each of 400 critical states, we generate 16 action-perturbed OOD observations. At each observation, we draw 64 policy action chunks and compute the expected minimum off-manifold error among kk samples; k=1k = 1 recovers the single-sample metric originally introduced by the paper.

Figure 29 shows the same tradeoff in this metric. At k=1k = 1, the later checkpoint has slightly lower off-manifold error. However, the ordering reverses at k=2k = 2, and the gap increases with kk. Thus, the later checkpoint produces a slightly better typical sample under the original single-sample metric, whereas the earlier checkpoint enjoys a substantially better favorable tail. Mean off-manifold error therefore does not reveal the coverage–reliability tradeoff across training checkpoints. It may be more useful for comparing different policy methods under matched training budget, as in the original study that introduced the metric. Here we show that best-of-kk off-manifold error reveals the same coverage–reliability tradeoff observed with held-out action error and closed-loop pass@kk.

Figure 29. Best-of-kk off-manifold error reveals the same coverage–reliability tradeoff. At k=1k = 1, ep800 has slightly lower off-manifold error; from k=2k = 2 onward, ep400 is better and the gap grows with kk.

How Policy Coverage Affects World-Model Learning

We additionally test whether policy coverage affects world-model learning from policy rollouts. For each task, we construct two training sets that share the same 200 expert demonstrations and differ only in the policy used to collect 1,000 additional unfiltered rollout episodes. One uses an earlier, higher-coverage checkpoint selected by the coverage-reliability criterion and the other uses the final, lower-coverage checkpoint. We keep failed rollouts because they provide additional state–action–next-state outcomes for learning dynamics and counterfactual reasoning on unseen actions.

We train the same latent dynamics model on each dataset where images are encoded once with the same frozen DINOv3 ViT-B/16 encoder ([41]), so the comparison is not affected by representation learning. Each model receives two frames of history and predicts the next three latent states open loop using the predicted actions. We evaluate both models on the same held-out episodes collected across the full checkpoint family.

Higher-coverage rollout data produces better held-out prediction. Table 4 shows lower prediction error for the early-rollout model at every horizon on both tasks. These results mirror the critic-learning experiment. Under a fixed rollout budget, broader policy coverage provides a wider range of state–action outcomes for learning. Because the rollouts are unfiltered, this includes both successful and failed behavior; both are useful for learning how different actions lead to different future states.

Threading (D2)Mug Cleanup (D1)
Training rolloutsh=1h = 1h=2h = 2h=3h = 3Training rolloutsh=1h = 1h=2h = 2h=3h = 3
Selected0.08000.10330.1186Selected0.09920.12750.1458
Final0.08150.10590.1219Final0.10030.12910.1484

Table 4. Higher-coverage rollout data improves held-out world-model prediction. We report open-loop latent token MSE; lower is better. Each model uses the same 200 demonstrations plus 1,000 unfiltered rollouts from either the earlier selected or final policy checkpoint.

Implementation Details

Tasks Datasets, and Evaluation

Across both MimicGen and RoboCasa365, we use the demonstration datasets released with the official repositories. Unless otherwise noted, experiments in the main paper use image observations as policy inputs.

For MimicGen, we train policies on 200 demonstration trajectories for the main-text experiments. We additionally report results with 500 demonstrations in Appendix E. For RoboCasa pretraining, we use 65 atomic tasks with 100 human demonstrations per task. For SFT, we use 100 demonstrations per target task.

For MimicGen BC evaluation, pass@k is computed over 500 held-out task configurations. For residual RL finetuning, we report the training success rate measured as the fraction of successful episodes among the 100 most recently completed episodes.

For MimicGen BC evaluation, pass@k is computed over 500 held-out task configurations. For each configuration we run KK independent rollouts and record the number of successes cc; for every k≤Kk \le K we compute pass@k =1−(K−ck)(Kk)= 1 - \frac{\binom{K-c}{k}}{\binom{K}{k}}, i.e. the fraction of the (Kk)\binom{K}{k} kk-subsets of the KK rollouts that contain at least one success, averaged over configurations. Best-of-kk action error is computed analogously: for each held-out state we sample KK action chunks, sort their errors e(1)≤⋯≤e(K)e_{(1)} \le\cdots\le e_{(K)}, and compute ∑r=1K−k+1(K−rk−1)(Kk)e(r)\sum_{r=1}^{K-k+1} \frac{\binom{K-r}{k-1}}{\binom{K}{k}}e_{(r)}, the expected minimum error over a uniformly random kk-subset of the KK draws, averaged over states. Instead of using the first kk passes, we use use all KK samples to approximate both quantities, which gives the same expectation but reduces variance. For residual RL finetuning, we report the training success rate measured as the fraction of successful episodes among the 100 most recently completed episodes.

For RoboCasa evaluation, pass@k is computed over 50 held-out task configurations for each evaluated task.

For both task suites, we use the default task horizon specified by the official repos.

Baseline Single Task Diffusion Policy

Our baseline single-task BC policy is a CNN-based diffusion policy following the implementation of Chi et al. [13]: a 1D temporal U-Net denoiser conditioned on a global observation embedding via FiLM, with a ResNet-18 visual encoder trained from scratch jointly with the denoiser network. The policy consumes a single observation step (two 84×84 RGB views, stacked as a 6-channel input, plus the 9-d proprioceptive state) and predicts an 8-step action chunk with 6-d rotation representation; all 8 predicted actions are executed open-loop before the next inference. Training uses the standard denoising objective: a uniformly sampled diffusion step t∈{0,…,99}t \in\{0,\ldots,99\}, forward noising with a squared-cosine β\beta schedule, and an MSE loss on the predicted noise (ϵ\epsilon-prediction). At inference we sample with 10 DDIM steps (η=0\eta= 0). All six MimicGen tasks share the identical architecture and optimization settings listed in Table 5.

HyperparameterValue
Observation steps1
RGB inputs2 views, 84×8484 \times84 (6-channel stack)
Proprioceptive state9-d
Predicted action chunk size8
Executed action chunk size8
Visual encoder
BackboneResNet-18
NormalizationGroupNorm
Encoder sharingone encoder per camera (not shared)
Image augmentationrandom crop to 76×7676 \times76 (center crop at inference)
Denoiser
Architecture1D conditional U-Net
Channel dimensions[512, 1024, 2048]
Kernel size / GroupNorm groups5 / 8
Diffusion step embedding dim128
Diffusion process
Training diffusion steps100
β\beta schedulesquared-cosine (β∈[10−4,0.02]\beta\in[10^{-4}, 0.02])
Prediction type / lossϵ\epsilon-prediction
Inference samplerDDIM, 10 steps, η=0\eta= 0
Optimization
OptimizerAdamW (β=(0.95,0.999)\beta= (0.95, 0.999), ϵ=10−8\epsilon= 10^{-8})
Learning rate1×10−41 \times10^{-4}
Weight decay1×10−61 \times10^{-6}
LR schedulecosine, 500 warmup steps
Batch size512
Training epochs800
Demonstrations200 (500 for Three-Piece Assembly D2 only)
EMAinv gamma 1.0, power 0.75, max decay 0.9999
Evaluation weightsEMA

Table 5. Baseline diffusion-policy configuration. All settings are shared across the six MimicGen tasks unless noted.

JointAug Implementation

JointAug augments policy training with samples from a joint generative model of the full observation–action tuple (o,a)(o, a), trained on the same demonstrations as the policy. The generator is a pixel-space, dual-output stream diffusion model that denoises the 6×84×846 \times84 \times84 two-camera frame, the 9-d proprioceptive states together with the 8×108 \times10 action chunk under a single shared diffusion process. Image tokens and the low-dimensional tokens are concatenated, jointly self-attend, and are split again, so every modality is generated conditioned on the others rather than independently. All three modalities are noised at the same timestep and the model predicts the vv-target for each; the loss is a weighted sum of per-modality mean squared errors with equal weights, which prevents the image from drowning out the 80-d action.

Inference. Samples are drawn with the generator’s EMA weights using a DDIM sampler with 100 steps. Training uses 1000-step squared-cosine schedule.

Selecting the generator checkpoint window. We save generator checkpoints every 50 epochs. For each saved checkpoint GγG_\gamma, we draw 1024 joint samples (o~,a~)∼Gγ(\tilde{o}, \tilde{a}) \sim G_\gamma and compare each generated action chunk with two equally sized sets of expert action chunks: the demonstrations used to train the generator, Dtrain\mathcal{D}_{\mathrm{train}}, and a held-out set Dheld\mathcal{D}_{\mathrm{held}} of the same task that are used to train neither the generator nor the policy. Let d(a~,D)=min⁡a′∈D∥a~−a′∥2d(\tilde{a},\mathcal{D}) = \min_{a' \in\mathcal{D}} \lVert\tilde{a} - a' \rVert_2 be the nearest-neighbor distance on normalized action chunks. We define

ρ(γ)=Ea~∼Gγ[d(a~,Dtrain)]Ea~∼Gγ[d(a~,Dheld)].(26)\rho(\gamma) = \frac{\mathbb{E}_{\tilde{a}\sim G_\gamma}[d(\tilde{a},\mathcal{D}_{\mathrm{train}})]}{\mathbb{E}_{\tilde{a}\sim G_\gamma}[d(\tilde{a},\mathcal{D}_{\mathrm{held}})]}. \tag*{(26)}

ρ(γ)=1\rho(\gamma) = 1 means generated actions are equally close to training and held-out demonstrations. We set γ0\gamma_0 to the first saved checkpoint with ρ(γ)<1\rho(\gamma) < 1, when generated samples first become closer to the training data than held-out data, and sample from γ0\gamma_0 and the four subsequent checkpoints spaced 100 epochs apart. Table 7 lists the resulting checkpoints for each task.

Mug D1Square D2Threading D2Hammer D13-Piece D13-Piece D2
Demonstrations200200200200200500
Generator checkpoints (training epoch)100, 200, 300, 400, 500100, 200, 300, 400, 500100, 200, 300, 400, 500100, 200, 300, 400, 500100, 200, 300, 400, 50050, 150, 250, 350, 450

Table 7. Per-task JointAug settings. Generator checkpoints are indexed by their training epoch; rows of each minibatch are drawn equally from the listed checkpoints

Distillation into policy training. During policy training, 50% of minibatch’s rows is replaced by freshly generated (o,a)(o, a) tuples; replaced and dataset rows are then trained with the policy’s ordinary behavior-cloning loss, so there’s no change to the policy architecture of Appendix § 18.§ 2.

PostBC Implementation

We follow the image instantiation of Algorithms 1 and 4 (Appendix § 14 of Wagenmaker et al. [46]). An ensemble of KK members is first fit to the policy’s own training split; each member fjf_j is a freshly initialized copy of the policy’s observation encoder followed by a 3-layer MLP head that regresses the flattened 8×108 \times10 action chunk by mean squared error, and each is fit on its own bootstrap resample of that split. The ensemble is trained once per task, before and independently of policy training, with the settings in Table 8.

Table 8: PostBC posterior-ensemble config and noise injection. All settings are shared across the five MimicGen tasks for which we report PostBC.

HyperparameterValue
Ensemble members
Ensemble size KK20
Member architecturepolicy observation encoder + MLP head
Head width / layers512 / 3
Member outputflattened action chunk (8 × 10 = 80-d)
Ensemble optimization
OptimizerAdam
Learning rate3×10−43 \times10^{-4}
Batch size (per member)128
Training epochs200
Checkpoing used40
Noise injection
Noise strength α\alpha2.0
Excluded dimensionsgripper (final action dimension, scale 0)

Table 8.

Posterior uncertainty is estimated from the disagreement of the ensemble predictions, cached per training window offline. At every fetch we draw a fresh noise vector from the diagonal of that covariance, scale it by α\alpha, and add it to the clean target action with the gripper dimension zeroed; policy training is otherwise unchanged from Appendix § 18.§ 2 or H.7. For MimicGen tasks, we use α=2.0\alpha= 2.0. For RoboCasa, we use α=1.0\alpha= 1.0.

Context-Smoothed Pre-training in TMRL Implementation

Context-smoothed pre-training Hong et al. [19] widens a BC policy’s support by corrupting its conditioning during pretraining, so that one network models pθ(a∣c~,tc)p_\theta(a \mid\tilde{c}, t_c) from conditional action distribution at tc=0t_c = 0 to the marginal action distribution at tc=1t_c = 1. We keep the paper’s corruption kernel and noise schedule: a discrete linear β\beta schedule over [10−4,0.02][10^{-4}, 0.02] with Tc=100T_c = 100 context steps, the variance-preserving marginal c~=αˉtcc+1−αˉtcϵ\tilde{c} = \sqrt{\bar{\alpha}_{t_c}}c + \sqrt{1 - \bar{\alpha}_{t_c}}\epsilon, and tc∼Unif⁡(0,1)t_c \sim\operatorname{Unif}(0,1) drawn independently of the action timestep tat_a. We evaluate with a clean context, tc=0t_c = 0.

Multitask Training on Robocasa

HyperparameterValue
Image2 views, 84×84 (6-channel stack)
Proprioceptive state9-d
Action chunk8 steps, absolute pose, 6-d rotation (10-d)
Observation steps1
Image stream
Architectureconv U-Net
Channel dimensions[96, 192, 384]
Bottleneckself-attention at 21×21
Low-dimensional stream
ArchitectureDiT over 9 tokens (1 state + 8 action)
Model width / layers / heads384 / 6 / 6
Dropout0.1
Cross-modal coupling
BridgeMM-DiT block at the bottleneck (joint self-attention)
Time embedding dim256
Diffusion process
Training diffusion steps1000
β\beta schedulesquared-cosine (β∈[10−4,0.02]\beta\in[10^{-4}, 0.02])
Prediction typevv-prediction
Lossper-modality mean MSE, weights 1:1:1 (image:state:action)
SamplerDDIM, 100 steps
Sampling weightsEMA
Optimization
OptimizerAdamW (β=(0.95,0.999)\beta= (0.95, 0.999), ϵ=10−8\epsilon= 10^{-8})
Learning rate1×10−41 \times10^{-4}
Weight decay1×10−41 \times10^{-4}
LR schedulecosine
Batch size256
EMAinv gamma 1.0, power 0.75, max decay 0.9999
Demonstrations200 (500 for Three-Piece Assembly D2)

Table 6. Joint (o,a)(o, a) generator configuration. All settings are shared across the six MimicGen tasks unless noted.

Residual RL Implementation.

We use DICE-RL ([45]) as an instantiation of residual RL finetuning. In DICE-RL, a frozen pretrained diffusion policy is finetuned by additive correction from a residual actor. Let ss be the observation embedding of the frozen policy and zz the initial DDIM noise; the base policy deterministically maps (s,z)(s, z) to a chunk πpre(s,z)\pi_{\mathrm{pre}}(s, z) of H=8H = 8 actions, and the executed chunk is a=πpre(s,z)+rθ(s,z)a = \pi_{\mathrm{pre}}(s, z) + r_\theta(s, z), where only the residual MLP rθr_\theta is trained. Since the chunk is executed open-loop, one transition spans the entire chunk and the critic Qϕ(s,a)Q_\phi(s, a) scores the whole chunk. Because image features learned by BC on MimicGen tasks carry a weak value signal, we parameterize the residual actor and the critic with the low-dimensional state; the frozen pretrained policy still remains image-based. Episodes end at the first success step or at the task horizon, with truncation treated as terminal. Hyperparameters are identical across tasks (Table 9). For each residual RL experiment, we run with three random seeds.

Table 9: Residual RL finetuning configuration, identical across tasks; only the frozen base checkpoint and the environment horizon change.

HyperparameterValue
Frozen base policyEMA weights, DDIM 10 steps (η=0\eta= 0), chunk H=8H = 8
Residual actorMLP [1024, 1024, 1024], GELU, LayerNorm
Critic ensemble10× MLP [1024, 1024, 1024], GELU, LayerNorm
BC weight λBC\lambda_{\mathrm{BC}}200
TD loss / nn-stepMSE / n=3n = 3 chunks
Discount γ\gamma / target update τ\tau0.99 / 0.01
Parallel environments64
Gradient steps per iteration40
Batch size256
RLPD expert ratio0.5 → 0.1 over the first 10k training steps
OptimizerAdam, lr 1×10−41 \times10^{-4}, weight decay 1×10−51 \times10^{-5}
Actor update frequency2
BoN Explorationarg⁡max⁡kmin⁡eQ\arg\max_k \min_e Q over K=16K = 16 samples

Table 9.

Multitask Training on Robocasa.

We use the RoboCasa release of openpi and follow the RoboCasa π0.5\pi_{0.5} benchmark recipe unchanged, except for the pretraining learning rate (Table 10).

HyperparameterValue
Peak learning rate1×10−41 \times10^{-4} / 2.5×10−52.5 \times10^{-5}
Schedule1k-step warm-up, cosine decay to 0.1×0.1\times peak
Training steps100k / 50k
Global batch size256
OptimizerAdamW (β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95, wd 10−1010^{-10}), grad clip 1.0
EMA decay (evaluated weights)0.99
Action chunk / cameras / image size50 steps / 3 / 224×224224\times224

Table 10. π0.5\pi_{0.5} training configuration (pretraining / fine-tuning). All values follow the RoboCasa π0.5\pi_{0.5} benchmark except the pretraining peak learning rate (benchmark: 2.5×10−52.5 \times10^{-5}).

Pretraining. Starting from pi05_base, the language model is frozen, the vision encoder is trained at 0.1×0.1\times the base learning rate, and the action expert at the base learning rate, for 100k steps at batch size 256 with AdamW, a 1,000-step warm-up, and cosine decay to 0.1×0.1\times the peak. The benchmark uses a peak learning rate of 2.5×10−52.5 \times10^{-5}; we instead uses 1×10−41 \times10^{-4} (4× higher, decaying to 1×10−51 \times10^{-5}), which we find to work better, with the vision-encoder scaling unchanged.

Supervised fine-tuning. Each fine-tuning run initializes all weights from a pretraining checkpoint and uses the benchmark fine-tuning recipe as is: peak learning rate 2.5×10−52.5 \times10^{-5} with the same warm-up and cosine schedule (50k steps), batch size 256, the same frozen/scaled parameter groups, and normalization statistics recomputed on the SFT data. Runs differ only in their initialization; checkpoints are saved every 2.5k steps and evaluated on the SFT tasks up to 20k steps.

Real Robot Experiments

BC Pretraining. We train a Diffusion Policy using one 224 × 224 image from each of the gripper-mounted main and ultrawide cameras and two timesteps of proprioceptive state, expressed relative to the current end-effector pose. Each camera has a separate CLIP-pretrained ViT-B/16 encoder with attention pooling, finetuned at 0.1×0.1\times the base learning rate. A 1D conditional U-Net predicts a 16-step action chunk at 20 Hz (0.8 s). Each 10D action contains relative end-effector translation (3D) and rotation (6D), plus gripper width. The diffusion process uses 50 training timesteps, a squared-cosine β\beta schedule, and ϵ\epsilon-prediction; inference uses 16 DDIM steps. We train for 208 epochs with AdamW (lr =7.5×10−5= 7.5 \times10^{-5}, β=(0.95,0.999)\beta= (0.95, 0.999), weight decay 10−610^{-6}), batch size 64, cosine learning-rate decay with 5% warmup, gradient clipping at 0.5, and weight EMA with decay 0.9999. We apply random cropping (95%) and color jitter. Of the 740 demonstrations, 680 are used for training and 60 for held-out validation.

World Model. The world model is a 12-layer diffusion transformer with width 768 and 12 attention heads. It operates on latents from a frozen Stable Diffusion VAE. Each training sequence contains six ultrawide-camera frames at 200 ms intervals: two history frames and four target frames. We use causal attention and condition each predicted frame on the four actions executed since the previous frame by adding their embeddings to the frame tokens. We follow Diffusion Forcing and sample independent noise levels for each frame. Training uses vv-prediction, a 1000-step squared-cosine schedule. Inference uses 50 DDIM steps. We use AdamW (lr =10−4= 10^{-4}, weight decay 0.01), 1000 warmup steps followed by a constant learning rate, batch size 32, and bf16 mixed precision. Figure 30 compares ground-truth frames (top row) with predictions from world models trained on early-checkpoint rollouts (middle row) and late-checkpoint rollouts (bottom row). The early-rollout model’s predictions more closely match the ground truth.

Figure 30. Qualitative world-model predictions on real-robot Push-T. The top row shows ground truth; the middle and bottom rows show predictions from world models trained on rollouts from the selected early and final policy checkpoints, respectively. Each model predicts four frames from two history frames (Chunk 1), then autoregressively predicts four more frames (Chunk 2). The early-rollout model achieves higher PSNR in both chunks.

References

  1. [1]Arthur Allshire, Himanshu Gaurav Singh, Ritvik Singh, Adam Rashid, Hongsuk Choi, David McAllister, Justin Yu, Yiyuan Chen, Huang Huang, Pieter Abbeel, et al. Scalable behavior cloning with open data, training, and evaluation. arXiv preprint arXiv:2606.27375, 2026.arxiv.org/abs/2606.27375
  2. [2]Ricardo Baptista, Agnimitra Dasgupta, Nikola B Kovachki, Assad Oberai, and Andrew M Stuart. Memorization and regularization in generative diffusion models. arXiv preprint arXiv:2501.15785, 2025.arxiv.org/abs/2501.15785
  3. [3]Jose Barreiros, Andrew Beaulieu, Aditya Bhat, Rick Cory, Eric Cousineau, Hongkai Dai, Ching-Hsin Fang, Kunimatsu Hashimoto, Muhammad Zubair Irshad, Masha Itkina, et al. A careful examination of large behavior models for multitask dexterous manipulation. Science Robotics, 11 (113):eaea6201, 2026.DOI
  4. [4]Quentin Bertrand, Anne Gagneux, Mathurin Massias, and Rémi Emonet. On the closed-form of flow matching: Generalization does not arise from target stochasticity. Advances in neural information processing systems, 38:8522–8549, 2026.arxiv.org/abs/2506.03719
  5. [5]Giulio Biroli, Tony Bonnaire, Valentin De Bortoli, and Marc Mézard. Dynamical regimes of diffusion models. Nature Communications, 15(1):9957, 2024.DOI
  6. [6]Johan Bjorck, Fernando Castañeda, Nikita Chernіadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025.arxiv.org/abs/2503.14734
  7. [7]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024.arxiv.org/abs/2410.24164
  8. [8]Adam Block, Ali Jadbabaie, Daniel Pfrommer, Max Simchowitz, and Russ Tedrake. Provable guarantees for attentive reactive behavior cloning: Bridging low-level stability and high-level behavior. Advances in Neural Information Processing Systems, 36:48534–48547, 2023.
  9. [9]Tony Bonnaire, Raphaël Urfin, Giulio Biroli, and Marc Mézard. Why diffusion models don’t memorize: The role of implicit dynamical regularization in training. Advances in Neural Information Processing Systems, 38:141266–141286, 2026.
  10. [10]Fan Chen, Audrey Huang, Noah Golowich, Sadhika Malladi, Adam Block, Jordan Ash, Akshay Krishnamurthy, and Dylan Foster. The coverage principle: How pre-training enables post-training. In International Conference on Learning Representations, volume 2026, pp. 112225–112293, 2026a.arxiv.org/abs/2510.15020
  11. [11]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
  12. [12]William Chen, Jagdeep Singh Bhatia, Catherine Glossop, Nikhil Mathihalli, Ria Doshi, Andy Tang, Danny Driess, Karl Pertsch, and Sergey Levine. Steerable vision-language-action policies for embodied reasoning and hierarchical control. arXiv preprint arXiv:2602.13193, 2026b.arxiv.org/abs/2602.13193
  13. [13]Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684–1704, 2025.
  14. [14]Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, and Adam Klivans. Ambient diffusion: Learning clean distributions from corrupted data. Advances in Neural Information Processing Systems, 36:288–313, 2023.arxiv.org/abs/2305.19256
  15. [15]Alessandro Favero, Antonio Sclocchi, and Matthieu Wyart. Bigger isn’t always memorizing: Early stopping overparameterized diffusion models. ArXiv, abs/2505.16959, 2025. URL https://api.semanticscholar.org/CorpusID:28789454.arxiv.org/abs/2505.16959
  16. [16]Dylan J Foster, Zakaria Mhammedi, and Dhruv Rohatgi. Is a good foundation necessary for efficient reinforcement learning? the computational role of the base model in exploration. arXiv preprint arXiv:2503.07453, 2025.
  17. [17]Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:2310.02664, 2023.arxiv.org/abs/2310.02664
  18. [18]Chengyang He, Xu Liu, Gadiel Mark Sznaier Camps, Joseph Bruno, Guillaume Adrien Sartoretti, and Mac Schwager. Demystifying robot diffusion policies: Action memorization and a simple lookup table alternative. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2505.05787
  19. [19]Matthew M Hong, Jesse Zhang, Anusha Nagabandi, and Abhishek Gupta. Tmrl: Diffusion timestep-modulated pretraining enables exploration for efficient policy finetuning. arXiv preprint arXiv:2605.12236, 2026.arxiv.org/abs/2605.12236
  20. [20]Haoxu Huang, Tongsam Zheng, Yifan Chen, Jiacheng You, and Yang Gao. Critical interval mse: Toward reliable offline validation for robot manipulation policies. arXiv preprint arXiv:2606.29898, 2026.arxiv.org/abs/2606.29898
  21. [21]Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, et al. π0.6: a vla that learns from experience. arXiv preprint arXiv:2511.14759, 2025a. Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv preprint arXiv:2504.16054, 2025b.
  22. [23]Feiyang Kang, Michael Kuchnik, Karthik Padhye, Marin Vlastelica, Ruoxi Jia, Carole-Jean Wu, and Newsha Ardalani. Quagmires in sft-rl post-training: When high sft scores mislead and what to use instead. In International Conference on Learning Representations, volume 2026, pp. 56876–56918, 2026.arxiv.org/abs/2510.01624
  23. [24]Liyiming Ke, Jingqiang Wang, Tapomayukh Bhattacharjee, Byron Boots, and Siddhartha Srinivasa. Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine manipulation. In 2021 IEEE international conference on robotics and automation (ICRA), pp. 6185–6191. IEEE, 2021.arxiv.org/abs/2011.06719
  24. [25]Liyiming Ke, Yunchu Zhang, Abhay Deshpande, Siddhartha Srinivasa, and Abhishek Gupta. Cicil: Continuity-based data augmentation for corrective imitation learning. In International Conference on Learning Representations, volume 2024, pp. 35813–35831, 2024.arxiv.org/abs/2310.12972
  25. [26]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024.arxiv.org/abs/2406.09246
  26. [27]Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645, 2025.arxiv.org/abs/2502.19645
  27. [28]Hadas Kress-Gazit, Kunimatsu Hashimoto, Naveen Kuppuswamy, Paarth Shah, Phoebe Horgan, Gordon Richardson, Siyuan Feng, and Benjamin Burchfiel. Robot learning as an empirical science: Best practices for policy evaluation. arXiv preprint arXiv:2409.09491, 2024.
  28. [29]Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning, pp. 143–156. PMLR, 2017.arxiv.org/abs/1703.09327
  29. [30]Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In International Conference on Machine Learning, pp. 22188–22214. PMLR, 2023.arxiv.org/abs/2210.14199
  30. [31]Sadhika Malladi, Samy Jelassi, Dylan Foster, Jordan T Ash, and Akshay Krishnamurthy. Tailfst: Filtered fine-tuning improves post-training performance. arXiv preprint arXiv:2608.25756, 2026.arxiv.org/abs/2608.25756
  31. [32]Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298, 2021.arxiv.org/abs/2108.03298
  32. [33]Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17596, 2023.arxiv.org/abs/2310.17596
  33. [34]Suvir Mirchandani, Suneel Belkhale, Joey Hejna, Evelyn Choi, Md Sazzad Islam, and Dorsa Sadigh. So you think you can scale up autonomous robot data collection? arXiv preprint arXiv:2411.01813, 2024.arxiv.org/abs/2411.01813
  34. [35]Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance. arXiv preprint arXiv:2410.13816, 2024.arxiv.org/abs/2410.13816
  35. [36]Soroush Nasiriany, Sep Nasiriany, Abhiram Maddikuri, and Yuke Zhu. Robocasa365: A large-scale simulation framework for training and benchmarking generalist robots. In International Conference on Learning Representations, volume 2026, pp. 98643–98667, 2026.arxiv.org/abs/2603.04356
  36. [37]Chaoyi Pan, Giridharan Anantharaman, Nai-Chieh Huang, Claire Jin, Daniel Pfrommer, Chenyang Yuan, Frank Permenter, Guannan Qu, Nicholas Boffi, Guanya Shi, et al. Much ado about noising: Dispelling the myths of generative robotic control. In International Conference on Learning Representations, volume 2026, pp. 90575–90614, 2026.arxiv.org/abs/2512.01809
  37. [38]Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal, Shashwat Saxena, Jesse Zhang, Giri Anantharaman, Cleah Winston, Chaoyi Pan, Douglas Chen, Nai-Chieh Huang, et al. Ogpo: Sample efficient full-finetuning of generative control policies. arXiv preprint arXiv:2605.03065, 2026.arxiv.org/abs/2605.03065
  38. [39]Kulin Shah, Alkis Kalavasis, Adam R Klivans, and Giannis Daras. Does generation require memorization? creative diffusion models using ambient diffusion. arXiv preprint arXiv:2502.21278, 2025.
  39. [41]Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025.arxiv.org/abs/2508.10104
  40. [42]David Snyder, Asher James Hancock, Apurva Badithela, Emma Dixon, Patrick Miller, Rares Andrei Ambrus, Anirudha Majumdar, Masha Iktina, and Haruki Nishimura. Is your imitation learning policy better than mine? policy comparison with near-optimal stopping. arXiv preprint arXiv:2503.10966, 2025.
  41. [43]Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36:47783–47803, 2023.
  42. [44]Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, and Aditi Raghunathan. Overtrained language models are harder to fine-tune. arXiv preprint arXiv:2503.19206, 2025.
  43. [45]Zhanyi Sun and Shuran Song. From prior to pro: Efficient skill mastery via distribution contractive rl finetuning. arXiv preprint arXiv:2603.10263, 2026.arxiv.org/abs/2603.10263
  44. [46]Andrew Wagenmaker, Perry Dong, Raymond Tsao, Chelsea Finn, and Sergey Levine. Posterior behavioral cloning: Pretraining bc policies for efficient rl finetuning. arXiv preprint arXiv:2512.16911, 2025a.arxiv.org/abs/2512.16911
  45. [47]Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint arXiv:2506.15799, 2025b.
  46. [48]Yanwei Wang, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Pérez-D’Arpino, Dieter Fox, and Julie Shah. Inference-time policy steering through human interactions. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 15626–15633. IEEE, 2025.
  47. [49]Yu-Han Wu, Pierre Marion, GÊrard Biau, and Claire Boyer. Taking a big step: Large learning rates in denoising score matching prevent memorization. arXiv preprint arXiv:2502.03435, 2025.arxiv.org/abs/2502.03435
  48. [50]Zeqi Ye, Qijie Zhu, Molei Tao, and Minshuo Chen. Provable separations between memorization and generalization in diffusion models. In International Conference on Learning Representations, volume 2026, pp. 143375–143425, 2026.
  49. [51]Dylan Zhang, Yufeng Xu, Haojin Wang, Qingzhi Chen, and Hao Peng. Good sft optimizes for sft, better sft prepares for reinforcement learning. arXiv preprint arXiv:2602.01058, 2026.
  50. [52]Thomas T Zhang, Daniel Pfrommer, Chaoyi Pan, Nikolai Matni, and Max Simchowitz. Action chunking and exploratory data collection yield exponential improvements in behavior cloning for continuous control. arXiv preprint arXiv:2507.09061, 2025.

Paper details

vlnc:W_grexauk9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents