Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data

Introduction

Offline pretraining should allow an agent to learn from a dataset once and then improve through interaction without carrying that dataset forever. Yet most offline-to-online RL methods continue sampling offline transitions during online fine-tuning. This assumption can be expensive or infeasible when the original data are constrained by storage, privacy, licensing, or deployment requirements. Worse, simply deleting the offline buffer can cause an agent to rapidly lose the capabilities acquired during pretraining. We ask how an offline-pretrained agent should be fine-tuned when the original dataset is no longer available, which hereafter we refer to as “no-retention fine-tuning”.

Previous work identifies the critic as an important source of instability in this setting. The transition to online interaction abruptly changes the state–action distribution, forcing value estimates learned from broad offline data to be recalibrated from a small online replay buffer. WSRL stabilizes this transition by collecting initial rollouts with the pretrained policy before beginning online updates [54]. This critic-centered view explains an important part of no-retention fine-tuning, but it leaves open what should happen to the behavioral model itself.

Behavioral regularization is central to many offline RL methods. By fitting or remaining close to dataset actions, the policy is discouraged from exploiting values assigned to unsupported actions [17, 50, 9, 46]. Recent approaches combine this principle with diffusion or flow-based policies that can represent complex and multimodal behavior distributions [12, 41, 24]. While the offline data remain available, continuing behavioral fitting can keep the policy anchored to the distribution learned during pretraining. When the offline dataset disappears, however, this regularization can become a liability. Behavioral cloning no longer preserves the offline distribution; it trains the behavioral model to imitate actions generated by the agent’s current decision rule. Early online experience is scarce, correlated, policy-dependent, and nonstationary. Consequently, behaviors that are not immediately revisited may be undervalued and underused before improved value estimates can (re)highlight their value. This motivates a simple principle: keep learning which actions are valuable, but stop relearning which actions are possible.

We instantiate this principle by stopping online behavior cloning (BC) to preserve the expressive prior learned offline. This does not freeze the entire agent: the critic and algorithm-specific improvement mechanisms continue learning from online experiences. IFQL makes this separation particularly clear: during the online stage a fixed flow policy proposes actions, while an updated critic improves which proposal is selected. In FQL, the flow policy remains fixed while the critic and a one-step actor continue learning, with the one step actor distilled toward the frozen flow policy. Action-chunked RL (QC) similarly preserves a prior over temporally extended behaviors while updating their values and selections.

Across seven OGBench domains (35 tasks), preserving the pretrained prior substantially improves no-retention fine-tuning for IFQL, FQL, and QC. However, our ablations reveal that stopping BC alone is insufficient without an expressive architecture: the frozen prior must support complex, multimodal distributions to succeed. When the prior is restricted to a simple Gaussian, performance collapses. These results complement critic-centered accounts with a clear policy-side design principle: freeze an expressive, multimodal behavioral prior, and ensure online value learning can directly exploit it.

No-Retention Fine-Tuning with Expressive Priors

Problem Setting We consider a discounted Markov decision process $\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$ and seek a policy maximizing $J(\pi) = \mathbb{E}_{\tau,P}\left[\sum_{t=0}^{\infty}\gamma^t r(s_t, a_t)\right]$. Offline pretraining uses $\mathcal{D}_{\mathrm{off}} = \{(s_i, a_i, r_i, s_i')\}_{i=1}^{N}$ to learn a critic $Q_{\phi_0}$ and a policy or behavioral model. During online fine-tuning, the agent accumulates $\mathcal{D}_{\mathrm{on}}^t$ from interaction. We study the no-retention setting, in which $\mathcal{D}_{\mathrm{off}}$ is unavailable after pretraining and all online updates use only $\mathcal{D}_{\mathrm{on}}^t$. Let $d_{\mathrm{off}}$ and $d_{\mathrm{on}}^t$ denote the empirical state–action distributions of the two buffers. Retained-data fine-tuning may train on $d^t_{\rho} = \rho_t d_{\mathrm{off}} + (1-\rho_t)d_{\mathrm{on}}^t$ for $\rho_t > 0$. No retention sets $\rho_t = 0$. This change matters because a behavioral loss optimized under $d^t_{\rho}$ remains partly supervised by offline behavior, whereas the same loss under $d_{\mathrm{on}}^t$ fits only the agent’s evolving online experience.

Behavioral Priors and Decision Policies. We distinguish a behavioral prior $b_{\xi}(a \mid s)$ from the decision policy used for interaction. The prior models the conditional action distribution represented in its training data. For a likelihood-based model, behavioral fitting can be written as $\mathcal{L}_{\mathrm{BC}}(\xi; \mathcal{D}) = -\mathbb{E}_{(s,a)\sim\mathcal{D}}[\log b_{\xi}(a \mid s)]$; deterministic, diffusion, and flow policies use the corresponding regression or generative objectives. The decision policy need not equal $b_{\xi}$. It may be represented by a separate value-optimized actor or constructed by critic-guided selection from the prior. We write this relationship as $\pi_t = \mathcal{I}_{\eta_t}(b_{\xi_t}, Q_{\phi_t})$, where $\mathcal{I}$ is a policy-improvement operator and $\eta_t$ denotes any additional trainable parameters. Thus, fixing the prior need not fix the decision rule, which can still adapt through $Q_{\phi_t}$ or $\eta_t$.

Flow-Based Behavioral Priors A diagonal Gaussian is efficient but unimodal and may poorly model datasets containing several plausible actions at the same state. Flow policies instead transport noise $z \sim\mathcal{N}(0, I)$ to the behavior distribution using a state-conditioned velocity field [28, 25, 41]. For a dataset action $a$, flow time $u \sim\mathcal{U}[0, 1]$, and interpolation $x_u = (1-u)z + ua$, conditional flow matching minimizes $\mathcal{L}_{\mathrm{FM}}(\xi; \mathcal{D}) = \mathbb{E}_{(s,a),z,u}[\lVert f_\xi(s, x_u, u) - (a-z)\rVert_2^2]$. Integrating $\mathrm{d}x_u/\mathrm{d}u = f_\xi(s, x_u, u)$ from $x_0 = z$ to $u = 1$ produces $F_\xi(s,z)$, whose induced distribution defines $b_\xi(\cdot\mid s)$. Flow matching therefore plays the role of BC: it fits the behavioral prior to actions represented in the current replay data. IFQL uses this flow directly as a proposal distribution. In the best-of-$K$ variant studied here, IFQL samples $a^{(i)} \sim b_\xi(\cdot\mid s)$ and executes $a^\star= \arg\max_i Q_\phi(s, a^{(i)})$. FQL instead retains $F_\xi$ as a behavioral reference and trains a one-step actor $\mu_\psi(s,z)$ by minimizing $\mathcal{L}_{\mathrm{FQL}}(\psi) = \mathbb{E}_{s,z}[-Q_\phi(s,\mu_\psi(s,z)) + \alpha\lVert\mu_\psi(s,z) - \operatorname{sg}[F_\xi(s,z)]\rVert_2^2]$. Thus, FQL separates flow fitting from value-guided actor distillation.

Preserving the Prior During Online Fine-Tuning After offline pretraining, we freeze the behavioral prior, $\xi_t = \xi_0$, while updating the critic and algorithm-specific improvement parameters on $\mathcal{D}_{\mathrm{on}}^t$. The decision policy is therefore $\pi_t = \mathcal{I}_{\eta_t}(b_{\xi_0}, Q_{\phi_t})$. We denote continued behavioral fitting on online replay by BC-ON and freezing the prior by BC-OFF. Stopping BC affects only behavioral-model fitting; value learning, action selection, and distillation toward the frozen prior remain active. Under BC-OFF, IFQL freezes its flow and updates the critic; FQL additionally updates its distilled one-step actor; and QC [24] freezes its flow-based chunk prior while updating the chunk-level critic used for best-of-$N$ selection.

Experimental Setup and Evaluation Protocol. Our primary benchmark spans seven OGBench manipulation and locomotion domains (cube-double-play, puzzle-4x4-play, scene-play, antsoccer-arena=navigate, antmaze-large-navigate, humanoidmaze-medium-navigate, humanoidmaze-large-navigate), evaluating all five single-task variants per domain. For targeted extensions, we evaluate on low-quality data using cube-double-noisy and antmaze-large-explore, and examine human teleoperation demonstrations using the Multi-Human (MH) Can and Lift tasks from Robomimic [29]. Algorithms are pretrained for 1M gradient steps and fine-tuned for 1M environment steps; under no retention, the offline buffer is discarded at the transition. We compare BC-ON versus BC-OFF across IFQL, FQL, and QC, focusing detailed diagnostics on IFQL and FQL. All evaluations use three independent seeds per task; we report mean success rates with shaded standard error (SE) bands. Full environment specifications, task selection rationale, hyperparameters, and per-task results are detailed in Sec. B.2.

Freeze the Prior, Improve the Decision Rule

We first show that freezing the behavioral prior improves no-retention fine-tuning with flow-based policies. We then test whether removing BC is sufficient without an expressive prior and whether freezing remains effective after pretraining on low-quality data.

While our primary evaluations include QC, we focus our detailed diagnostic ablations on IFQL and FQL. In addition to their strong performance across recent offline-to-online RL literature [41, 23], these two algorithms cleanly span the two fundamental mechanisms for utilizing an expressive prior: direct critic-guided action selection (IFQL) and actor distillation (FQL), while avoiding confounding factors from temporal abstraction.

Stopping BC improves no-retention fine-tuning. Figure 3 compares BC-ON and BC-OFF during no-retention fine-tuning. Across IFQL, FQL, and action-chunked RL, stopping behavioral fitting consistently improves online learning. The largest gains occur for IFQL: BC-OFF improves the aggregate mean and outperforms BC-ON across tasks. FQL and the action-chunked methods exhibit the same overall trend, although the magnitude of the improvement varies across algorithms and domains. The IFQL and QC results are particularly revealing. Neither directly optimizes their behavioral flow (or a distilled version of it) for return; the flow proposes candidate actions, and the critic selects among them. Under BC-OFF, the proposal distribution remains fixed while the critic continues learning from online experience. Performance can therefore improve without refitting the behavioral model. Under BC-ON, the same flow matching loss is continually optimized on online-only replay, and performance is substantially lower. These results show that an objective useful during offline pretraining need not remain useful after its original data disappear. In this setting, preserving the pretrained behavioral model is more effective than continually refitting it to the agent’s recent experience.

Attenuating the BC penalty cannot prevent collapse. To determine whether the failure of online behavioral cloning is merely an artifact of an overly aggressive regularization weight, we evaluate no-retention fine-tuning across varying BC penalty magnitudes (0.1× to 10× the default). Figure 2 demonstrates that scaling the BC weight, whether amplifying or aggressively attenuating it, fails to prevent catastrophic collapse in both IFQL and FQL. This confirms the failure mode is fundamentally tied to the objective itself rather than its hyperparameter value: any active optimization toward the agent’s narrowing online experience progressively erases the offline prior. Consequently, only the complete removal of the behavioral loss (BC-OFF) enables sustained online improvement.

Can a large offline warmup rescue continued BC?

Because the failure of BC-ON is directly tied to discarding offline data, we test whether a temporary data “warmup” can prevent the collapse. We pre-populate the online buffer using the pretrained checkpoint with 25,000 online transitions. As shown in Figure 4, Col 4 & Col 5, this intervention prevents the catastrophic failure of BC-ON; performance no longer flattens at near zero. However, across both IFQL and FQL on cube-double and ant-soccer-arena, combining this warmup with BC-OFF still yields vastly superior online adaptation. This confirms that continued BC can fundamentally restrict the ceiling of policy improvement even when provided with healthy data initialization.

Dissecting IFQL: Isolating the mechanisms of recovery. Our previous experiments reveal that IFQL achieves the most robust performance under the BC-OFF regime. To understand whether this success is driven by IFQL’s conservative value updates or its generative policy, we isolate these mechanisms by ablating both (Figure 4, Col 1). We first ablate the critic by replacing IFQL’s IQL-style expectile regression with a standard Bellman double Q-learning loss. Despite this modification, the agent still achieves continuous online improvement when offline data is discarded, and BC is removed. Next, we ablate the actor by replacing the expressive flow-matching model with a standard Gaussian policy. Under this configuration, online performance completely flatlines. This component isolation demonstrates that IFQL’s resilience is not strictly dependent on its specific value-update mechanism but rather on the expressive capacity of its prior.

Unimodal policies cannot be rescued by removing BC. If an expressive prior is truly the bottleneck, then stopping BC should similarly fail to rescue other algorithms restricted to a unimodal representation. To test this, we apply the same no-retention interventions to ReBRAC, an offline RL method utilizing a standard unimodal Gaussian actor. As shown in (Figure 4, Col 2), discarding the offline dataset causes ReBRAC to completely collapse. Crucially, disabling behavioral cloning (BC-OFF) fails to rescue the policy; performance flat lines at zero alongside BC-ON. This provides further evidence that successfully fine-tuning without offline data may require not only stopping BC but also a sufficiently expressive prior. Note that to ensure a fair evaluation, the ReBRAC plots feature the Puzzle domain instead of Ant Soccer, as ReBRAC’s offline performance on Ant Soccer was near zero.

Dissecting FQL: Distillation is not an optional regularizer. Under BC-OFF, FQL freezes the flow network but continues distilling its one-step actor toward it. If we sever this link by disabling the distillation loss—forcing the actor to train solely on the value objective—performance catastrophically collapses to zero (Table 5). Unlike IFQL, which samples directly from the flow, FQL relies entirely on distillation to transfer knowledge to the deployed policy. Disconnecting this pathway leaves the actor with no access to the offline manifold. This collapse validates our core

Task	BC-ON	BC-OFF	BC-OFF, no distillation
cube-double	0.000	0.977 ± 0.014	0.000
antsoccer-arena	0.000	0.303 ± 0.206	0.000

design principle: preserving a rich behavioral prior drives online success only when the architecture maintains an active pathway for the decision rule to exploit it.

Can BC-OFF recover performance from low-quality offline data? A potential concern with freezing the behavioral prior is that it permanently anchors the agent to the quality of the offline dataset. We test this by pretraining IFQL on low-quality, sub-optimal datasets (cube-double-noisy and antmaze-large-explore). In these settings, BC-OFF still demonstrates a promising capacity for online policy improvement, achieving returns above the initial offline baseline (Figure 4, Col 3). Because the flow-based policy is highly expressive, it models the broad, noisy distribution of the offline data without collapsing into a single suboptimal mode. Crucially, it appears to capture enough of the underlying valid action manifold to propose a diverse menu of actions that physically obey the environment dynamics, even if those actions originally led to low reward. We hypothesize that during online fine-tuning, this frozen prior continues to constrain the search space to these physically valid candidates. As the critic improves on new online experiences, it learns to consistently identify and select the higher-value actions from this broad distribution, yielding a better decision rule.

Generalization to Human Teleoperation Demonstrations

To confirm that prior collapse is not an artifact of synthetic OGBench datasets, we evaluate on the Robomimic benchmark using the Multi-Human (MH) datasets for Can and Lift. These datasets comprise 300 demonstrations collected across six operators of varying proficiency, introducing realistic suboptimality and behavioral multimodality. Details can be found in found in section B.3.2. As shown in Figure 5, the same failure mode emerges: continuing online behavioral cloning without offline data retention (BC-ON) induces severe degradation and task collapse across both IFQL and FQL. Conversely, discarding the behavioral objective while retaining the expressive prior (BC-OFF) completely prevents collapse and allows effective finetuning.

Stopping BC late still recovers part of the freezing benefit. Does preserving the behavioral prior require disabling BC from the beginning of online fine-tuning, or can the intervention remain effective after the prior has already been updated on online data? We evaluate intermediate behavioral cloning (BC) termination steps $\tau\in\{200\mathrm{k}, 500\mathrm{k}, 800\mathrm{k}\}$ during 1M online fine-tuning steps to determine if preserving the behavioral prior requires immediate freezing ($\tau= 0$). Table 1 demonstrates that immediate freezing consistently maximizes performance, whereas persistent BC ($\tau= 1\mathrm{M}$) yields the lowest success rates. Terminating BC at any intermediate $\tau$ recovers a substantial fraction of this gap. This demonstrates that removing the behavioral objective can act as a robust stabilizing intervention, even after the prior has begun to degrade.

$\tau$	cube-double	ant soccer-arena
0 (immediate)	0.85	0.71
200k	0.71	0.49
500k	0.53	0.50
800k	0.61	0.51
1M (never)	0.40	0.19

Generality Across Generative Priors

Having shown that turning BC off enables online fine-tuning in flow-matching algorithms in the no offline data retention regime, we examine whether this strategy extends to other expressive generative priors. To test this, we evaluate Implicit Diffusion Q-Learning (IDQL) (Hansen-Estruch et al., [11]), which pairs the implicit RL framework of IFQL with a diffusion objective (DDPM) rather than flow matching to parameterize its behavioral prior.

As shown in Figure 6, turning BC off enables consistent online adaptation for diffusion-based policies. In Cube Double, freezing the pretrained diffusion prior allows the policy to improve substantially beyond the offline checkpoint level, outperforming the standard buffer-retention baseline. In Ant Soccer, BC-OFF yields steady online progress, whereas continuing to train the prior even with full buffer retention degrades performance toward zero. These results indicate that the benefit of turning BC off is not specific to flow matching, and that a frozen pretrained prior supports effective online fine-tuning across distinct generative modeling paradigms.

Diagnosing Online Collapse

The previous section demonstrates that stopping BC can be important for successful no-retention fine-tuning for expressive flow-based policies. But why does continued behavioral cloning fail so catastrophically when offline data is discarded? In this section, we investigate the underlying mechanics of this phenomenon. We trace the failure of BC-ON to a specific behavioral causal chain, moving from catastrophic forgetting of the prior to representational collapse and ultimately critic degeneration. Beyond this feedback loop, we map the degradation of the actor network by tracking progressive layer wise parameter drift. We conclude by ruling out stale optimization statistics to confirm the failure is strictly objective-driven.

Catastrophic forgetting of the behavioral prior. When offline data is discarded, the online replay buffer initially contains only a narrow, correlated slice of recent experience. Continuing to optimize the BC loss on this restricted dataset forces the policy to rapidly overfit to its own early, suboptimal rollouts. This induces catastrophic forgetting of the broad, diverse behavior learned during pretraining. We can directly observe this erasure by tracking the flow policy’s action Mean Squared Error (MSE) on a held-out offline dataset (Figure 7, Left). For both IFQL and FQL, continuing BC without retention causes the offline action MSE to spike immediately and dramatically within the first 100k online steps. The actor rapidly unlearns the valid task manifold. In contrast, removing the behavioral objective (BC-OFF) completely preserves this foundational knowledge. This is also captured by the BC loss on the offline data (Figure 7, Right).

Online BC induces an actor collapse that drives data drift. This catastrophic forgetting of the prior thoroughly destabilizes the entire offline-to-online transition. To understand the structural impact of this failure, we isolate the actor’s intrinsic generative capacity by evaluating it post-hoc on a fixed set of diverse, held-out offline validation states. This ensures our measurements reflect the true model capacity rather than artifacts of a collapsed online replay buffer. Our empirical dynamics (Figure 8, Middle) reveal that as the flow network unlearns the prior, its internal representations rapidly degenerate, causing the rank of its features to plummet. This representational collapse directly destroys the policy’s generative capacity, leading to an immediate and sharp drop in both cross-state and within-state action diversity. While cross-state diversity captures whether the policy adapts to different states, within-state diversity ensures the generative model can produce multiple valid actions for a single state. This broad distribution preserves local exploration and, particularly for IFQL, provides the critic with a diverse menu of candidate actions to evaluate. Stripped of this exploratory power, the agent’s behavior undergoes a severe spatial collapse. PCA projections of online rollouts confirm this restriction, showing the actor trapped in repetitive failure modes with severely limited state-space coverage (Figure 9, Left). A concurrent drop in temporal coherency ($\mathbb{E}[\lVert x_{t+5} - x_t\rVert]$) for cube-double physically captures this collapse, demonstrating that broad exploration degenerates into localized jitter (Figure 9, Right).

Data starvation breaks the critic. This severely restricted coverage explains the downstream failure of value learning (Figure 8, Right). The BC objective does not directly alter the critic’s loss landscape, and the only mechanism through which BC can impact the critic is through the data the actor collects. Because the actor’s spatial exploration has collapsed, it feeds the critic a narrow, repetitive diet of transitions. Starved of diverse data, the critic’s own internal representations degenerate, leading to a severe collapse in its feature rank. We believe this representational collapse directly causes the decreasing critic separation gap, the difference in Q-values assigned to state-actions from top-20% versus bottom-20% return trajectories. The critic simply loses its ability to distinguish good states from bad ones because it no longer sees a sufficiently diverse state-action manifold to maintain rich, discriminative features. BC-OFF prevents this initial diversity collapse, supplying the critic with the rich, stable data necessary to preserve its feature rank and drive continuous policy improvement.

Drift accumulation and non-linear prior degradation. Table 1 shows that intermediate BC termination yields substantial, non-monotonic recovery. Performance drops sharply from the immediatefreeze optimum ($\tau= 0$), plateaus for hundreds of thousands of steps, and collapses under persistent BC ($\tau= 1\mathrm{M}$). This trajectory maps directly to the observed parameter drift in Figure 10, Left and Middle. The actor’s output-adjacent layer diverges within $10^3$–$10^4$ steps, rapidly degrading action quality and permanently lowering the performance ceiling. Deeper representational layers drift more slowly, creating a prolonged window where the prior is damaged but retains sufficient generative capacity to be rescued by removing BC. Persistent BC eventually overwrites these core representations entirely, triggering catastrophic failure.

Ruling out stale-optimizer statistics. The failure of online BC is visible at the level of gradient statistics long before downstream metrics degrade. Under no-retention BC-ON, Adam’s second-moment estimate $\hat{v}$ consistently plummets compared to BC-OFF across algorithms and tasks (Figure 10). Because $\hat{v}$ tracks the running magnitude of squared gradients, this drop indicates that the network is receiving weak, homogeneous gradient signals. Furthermore, because standard offline-to-online RL implementations routinely carry over optimizer statistics from pretraining into the online phase, this sharp divergence introduces a natural alternative hypothesis: the observed collapse might stem from stale Adam moment estimates failing to adapt to the online distribution shift, rather than the BC objective itself. To isolate this confounder, we clear the optimizer state at the transition boundary. Across all six Discard+BC-ON configurations, this reset yields exactly zero performance recovery, with final success rates remaining rigidly at 0.000 (Figure 12, Left). Under the functional BC-OFF regime, the reset produces only marginal and inconsistent fluctuations (Figure 12, Right). These results systematically rule out optimization artifacts as a failure mechanism.

Related Work

Behavior regularization in offline-to-online RL. Offline RL commonly regularizes policies toward dataset-supported behavior to limit extrapolation error [10, 17, 20]. Methods such as BCQ, BEAR, BRAC, advantage-weighted regression, TD3+BC, and ReBRAC differ in how they impose this constraint, but all rely on the dataset to define which behaviors should be preserved [10, 17, 50, 33, 16, 9, 46]. Most offline-to-online methods retain the offline dataset during fine-tuning, whereas considerably less work studies the setting in which that supervision disappears. WSRL discards the dataset but stabilizes the transition through pretrained-policy rollouts, while PORL considers a policy-only initialization with a newly learned critic [54, 51]. We retain the offline-pretrained actor and critic and study behavior regularization after its defining data become unavailable. Our results show that behavioral fitting should stop once that supervision disappears. See App. § 10 for a broader discussion.

Expressive policies for data-driven RL. Diffusion-QL combines an expressive diffusion actor with denoising-based BC and value maximization [48]. IDQL instead models the behavior distribution with diffusion and obtains an improved decision rule by critic-guided reweighting or selection [12]. FQL uses conditional flow matching to learn a behavior model and distills value-improved actions into an efficient one-step actor [41]. Q-chunking extends expressive behavior modeling to temporally extended action sequences [24]. Related work on policy-agnostic improvement also separates value improvement from the policy class used to represent the resulting actions [30]. Recent methods such as EXPO and FINO further investigate stable online optimization with expressive policies [8, 45]. We study a distinct question: whether an expressive behavioral model can remain useful as a fixed prior after both its training data and its behavioral-fitting objective are removed.

Lessons and Opportunities

Lessons. No-retention fine-tuning is fundamentally distinct from conventional offline-to-online RL. When the offline dataset is discarded, behavioral cloning ceases to act as a stabilizing regularizer; instead, it forces the network to imitate its own narrowing online experience. As shown, this rapidly cannibalizes the behavioral prior, causing a representational collapse that starves the rest of the learning process. The solution is not simply to turn off the objective, but to embrace a two-part design principle: successfully transitioning to online-only data requires (1) freezing an inherently expressive, multimodal behavioral prior, and (2) maintaining a decoupled decision mechanism (such as critic-guided selection or distillation) that allows online value learning to exploit that frozen manifold.

Opportunities. Our findings open several new avenues for data-restricted RL. While a strictly frozen prior is highly effective when pretraining and fine-tuning share the same underlying dynamics, extreme environmental distribution shifts may eventually require the prior to adapt. This motivates the need for mechanisms that bridge the gap between catastrophic online-only BC and hard freezing. Future work could explore generative replay, uncertainty-aware plasticity, or selective behavioral updates that preserve the broad, valid modes of the offline dataset while safely incorporating novel online behaviors. This is especially relevant to VLA fine-tuning, where expressive pretrained policies may need to support online improvement after demonstration data are discarded [6 b]. Ultimately, treating the pretrained model as a permanent, expressive foundation rather than a strictly plastic network offers a robust path forward for RL deployments with strict data constraints.

Acknowledgments

The authors would like to thank Walter Mayor for valuable discussions during the preparation of this work. We also thank Seohong Park for his helpful responses and assistance in clarifying questions regarding the methods and datasets. The research was enabled in part by computational resources provided by the Digital Research Alliance of Canada (https://alliancecan.ca) and Mila (https://mila.quebec). Mohammad Sami Nur Islam was supported by a Master’s research scholarship from the Fonds de recherche du Québec – Nature et technologies (FRQNT) under award number 369318. Aaron Courville acknowledges funding from the Institute of Information & Communications Technology Planning & Evaluation (IITP) through a grant funded by the Korean government (MSIT) (No. RS-2024-00457882, National AI Research Lab Project). Pablo Samuel Castro acknowledges funding from NSERC Discovery Grant. We want to acknowledge funding support from Google and CIFAR AI. We would also like to thank the Python community [47, 40] for developing tools that enabled this work, including NumPy [13], Matplotlib [14], Jupyter [15], and Pandas [32].

AI Use Statement

In this work, we used generative AI tools for aiding and polishing paragraph-level writing, discussing and refining the structure and flow of the introduction, finding related work, and editing and writing the code. We also used generative AI tools to assist with early drafts of some sections, which were later substantially rewritten and replaced by the authors. We have not used generative AI tools for research ideation, and the rest of the required disclosure tasks, such as generating synthetic datasets or proving mathematical claims, are not applicable to this work. We have reviewed the AI-assisted work. We reviewed LLM-generated code and read and incorporated suggested related work papers manually. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Additional Related Work

Offline reinforcement learning is vulnerable to extrapolation error because value functions and policies may be evaluated on actions that are poorly represented in the dataset [10, 17, 20]. A broad family of methods therefore constrains policy improvement toward the offline behavior distribution. BCQ restricts action selection using a learned behavior model [10]; BEAR and BRAC penalize divergence from the behavior policy [17, 50]; and advantage-weighted methods regress toward dataset actions according to their estimated value [43, 33, 16]. TD3+BC and ReBRAC combine value maximization with direct regression toward actions in the replay buffer [9, 46]. Although these methods differ in form, they share an important assumption: the data defining the behavioral constraint remain available during optimization. The behavioral objective is meaningful because its supervision continues to represent the desired action distribution. We study what happens when the objective remains active but the data that originally defined it do not.

Offline-to-online reinforcement learning initializes an online learner from offline pretraining and subsequently improves it through environment interaction [23]. AWAC uses offline data to initialize a policy and then performs advantage-weighted online updates [33]. Other methods retain offline transitions through balanced replay, large update-to-data ratios, or conservative value estimation [19, 2, 34]. Cal-QL explicitly calibrates the offline critic so that its values provide a more suitable initialization for online fine-tuning [34]. Additional methods manage the stability–plasticity trade-off using separate offline and online policies, adaptive behavioral constraints, or evolving reference policies [52, 53, 21]. These approaches show that offline supervision can accelerate online learning, but they generally assume continued access to the offline dataset or introduce an additional mechanism for preserving its influence.

A smaller body of work studies online adaptation after offline data are discarded. WSRL shows that offline data need not be retained when the online replay buffer is initialized with rollouts from the pretrained policy [54]. Its analysis emphasizes critic recalibration and value divergence under the distribution shift from offline data to online experience. PORL considers an even more restrictive policy-only setting in which the pretrained policy is available but the offline critic is not, and a new critic must be learned from interaction [51]. Reincarnating RL studies the broader problem of reusing previously learned policies and value functions to accelerate subsequent reinforcement learning rather than restarting from scratch [1]. These works establish that no-retention fine-tuning is possible, but primarily focus on preserving or reconstructing value estimates. We retain the offline-pretrained actor and critic and study a complementary actor-side question: should behavioral fitting continue once the data that defined its objective become unavailable?

Another line of work treats offline-to-online fine-tuning as a replay composition problem. Balanced replay and related methods control the relative contribution of offline and online transitions through sampling ratios, prioritization, or adaptive buffer construction [19, 2]. These methods primarily change the data seen by the critic and actor. Our intervention is different: we change whether the behavioral model continues to optimize an objective whose supervision has changed. This distinction is important in the no-retention setting. Once the offline buffer is unavailable, replay manipulation alone cannot restore the original behavioral distribution unless that distribution has been preserved by the model itself.

Diffusion- and flow-based policies provide expressive alternatives to unimodal Gaussian actors and can represent multimodal conditional action distributions [48, 28, 25]. Diffusion-QL combines behavior modeling with value maximization [48], while IDQL uses a diffusion behavior model to propose actions and a critic to reweight or select among those proposals [12]. FQL replaces iterative diffusion with flow matching and distills value-improved actions into an efficient one-step actor [41]. These methods separate the distribution used to propose actions from the mechanism used to improve or select them.

Recent work also studies how expressive policies should be optimized online. EXPO avoids directly propagating unstable value gradients through a long-horizon generative policy and instead uses a lightweight edit policy to modify actions sampled from an expressive base policy [8]. FINO adapts flow policies online through noise injection and entropy-guided sampling to promote exploration beyond the offline distribution [45]. These methods continue adapting the expressive policy while attempting to stabilize its optimization. We study a different intervention: preserving the pretrained expressive prior and continuing online improvement through the critic and decision rule.

Action chunking extends expressive behavior modeling from individual actions to temporally extended action sequences. Q-chunking applies this idea directly to reinforcement learning, using chunked actions to preserve temporally coherent behavior, improve exploration, and enable multi-step value backups [24]. This setting is especially relevant to no-retention fine-tuning because updating a chunk prior on a small online buffer can erase not only individual actions but also longer-horizon temporal structure. We therefore evaluate whether the prior-preservation principle extends from single-action flow policies to action-chunked RL.

Our work differs from these studies in its central intervention and diagnostic question. We do not introduce a new expressive policy class, a new critic-calibration procedure, or a new replay rule. Instead, we isolate the consequences of continuing versus stopping behavioral-model fitting after the offline data disappear. The resulting comparison reveals when an expressive prior remains useful, when removing BC is insufficient, and why the connection between the prior and the deployed decision rule is essential.

Experiment Details

Computational Resources

To optimize computational efficiency and ensure strictly matched starting conditions, we decoupled the offline pretraining phase from the online fine-tuning phase. For each algorithm, task, and random seed, we first executed the 1M offline gradient steps and saved the resulting pretrained checkpoints. All subsequent online fine-tuning setups were then initialized from these shared checkpoints.

Experiments were executed on NVIDIA H100 GPUs. To maximize hardware utilization, we multiplexed six individual experimental runs per GPU, with the underlying allocation providing 12 CPU cores and 124 GB of system memory under a 12-hour wall-clock budget. In total, the full study includes several hundred individual online training runs across the four evaluated algorithms (FQL, IFQL, ReBRAC, and QC).

Reusing pretrained checkpoints avoids redundant offline computation across fine-tuning configurations, while freezing the behavioral prior eliminates the need for continued behavioral-model optimization. These choices simplify controlled experimentation and may reduce computational overhead, supporting broader efforts toward accessible and reproducible deep RL research [4]. Such efficiency considerations are particularly relevant when conducting systematic investigations of architectural choices [49, 38, 36] and experimental configurations [35, 44].

Implementation and Hyperparameters

We base our implementations of FQL, IFQL, and ReBRAC on the official FQL repository, and Action-Chunked RL (QC) on the official QC repository. To ensure fair evaluations, we do not re-tune the base optimization hyperparameters. Instead, we retain the standard architecture and optimization settings from the reference implementations and apply their recommended task-specific hyperparameters. These values remain strictly fixed across all experiments so that our ablations isolate only the specific architectural or algorithmic components under study. Nevertheless, deep RL agents can be sensitive to hyperparameter selection (Ceron et al., 2024), and independently tuning each configuration may yield different results at substantial computational expense.

For FQL, the actor, flow model, and critic use four-layer MLPs with hidden dimensions (512, 512, 512, 512) and GELU activations. Optimization uses Adam with a learning rate of $3 \times10^{-4}$, minibatches of size 256, and a target-network smoothing coefficient $\tau= 0.005$. Flow policies use linear flow-matching paths, uniformly sampled flow times, and 10 Euler integration steps. FQL uses two Q-functions; their mean is used in the actor objective and, for the OGBench tasks considered here, also for the critic target.

IFQL uses the same flow-matching configuration as FQL and follows the default IQL value-learning setup, utilizing an IQL expectile of $\kappa= 0.9$. At evaluation time, IFQL samples $N$ candidate actions from the behavioral flow prior and executes the action with the highest estimated value. We use the task-specific values of $N$ detailed in Table 3.

Task	IFQL $N$	FQL $\alpha$	QC $N$	ReBRAC $(\alpha_1,\alpha_2)$
cube-double	32	300	32	(0.1, 0)
puzzle-4x4	32	1000	32	(0.3, 0.01)
antsoccer-arena	64	30	64	(0.01, 0.01)
humanoidmaze-medium	32	100	32	(0.01, 0.01)
humanoidmaze-large	32	100	32	(0.01, 0.01)

For QC, we preserve the flow-matching setup and maintain a fixed action chunking horizon of $H = 5$ across all tasks. Similar to IFQL, QC evaluates by sampling $N$ candidate action sequences from the frozen prior and selects the sequence that maximizes the chunk-level critic.

ReBRAC uses identical default architecture and optimization settings unless otherwise specified. Its primary method-specific hyperparameters are the actor behavioral-cloning coefficient $\alpha_1$ and the critic behavioral-cloning coefficient $\alpha_2$.

Hyperparameter	Value
Default MLP hidden dimensions	(512, 512, 512, 512)
Activation	GELU
Optimizer	Adam
Learning rate	$3 \times10^{-4}$
Batch size	256
Target-network update rate $\tau$	0.005
Discount $\gamma$	0.99 default; 0.995 for humanoidmaze/antsoccer
Number of Q-functions	2
FQL / IFQL / QC flow policy
Flow time sampling distribution	Unif([0, 1])
Euler integration steps	10
IFQL
IQL expectile $\kappa$	0.9
QC
Action chunking horizon $H$	5

Setting	Value
Offline pretraining steps	1,000,000
Online fine-tuning steps	1,000,000
Replay buffer capacity	2,000,000 transitions
Evaluation episodes per checkpoint	50
Evaluation interval	Every 100,000 steps
Seeds per configuration	3

Domain Details

OGBENCH

We cover both manipulation and locomotion domains from the OGBENCH benchmark using the single-task variants of each environment. We use all five tasks available for each domain. Under the single-task variant, the reward structure is semi-sparse: locomotion tasks receive a binary 0 or 1 reward upon completion, while manipulation tasks receive a negative reward corresponding to the number of unaccomplished subtasks.

• cube-double-play: A manipulation task to control a robot arm to arrange two cubes into designated configurations within a 500-step episode.

• puzzle-4x4-play: A manipulation task requiring the agent to press physical buttons to solve a $4 \times4$ “Lights Out” puzzle within a 500-step episode.

• scene-play: A manipulation task requiring a robot arm to manipulate diverse everyday objects, such as a drawer, window, and button locks, into a desired configuration within a 750-step episode.

• antsoccer-arena-navigate: A locomotion task where an Ant agent must dribble a soccer ball to a target location in an open arena within a 1000-step episode.

• antmaze-large-navigate: A locomotion task requiring an 8-DoF Ant agent to navigate to a goal location in a large maze layout within a 1000-step episode.

• humanoidmaze-medium-navigate: A locomotion task requiring a 21-DoF Humanoid agent to navigate to a goal location in a medium-sized maze within a 2000-step episode.

• humanoidmaze-large-navigate: A locomotion task requiring the 21-DoF Humanoid agent to navigate to a goal location in a larger and more complicated maze layout within a 2000-step episode.

ROBOMIMIC

To evaluate whether the collapse of the behavioral prior generalizes beyond synthetic benchmarks, we evaluate on the Robomimic benchmark [29] using human-collected teleoperation datasets. We adopt the Multi-Human (MH) datasets collected via RoboTurk, where demonstrations were gathered across six human operators stratified into three distinct proficiency levels (experienced, adequate, and inexperienced; 50 demonstrations per operator). Each task dataset contains 300 successful demonstrations featuring natural variance, suboptimal paths, and multimodal execution strategies. We evaluate on the following tasks:

• Lift: Requires a 7-DoF Franka Emika Panda arm to lift a small cube above a height threshold.

• Can: Requires the robot arm to grasp a coke can and place it inside a designated container bin.

Evaluation Across Additional Domains

We extend our evaluation to an additional two OGBench domains by incorporating Scene Play and Ant Maze Large as can be seen in Figure 11.

Mechanistic Analysis Metric Definitions

In section 4, we track several metrics to quantify the representational and behavioral collapse of the actor and critic. Prior work has highlighted the importance of representation quality, effective network capacity, and stable gradient propagation for deep RL [36, 36, 38, 3, 27]. Complementary studies have also shown that the quality and diversity of collected experience can substantially affect learning dynamics [31, 26]. To decouple the intrinsic capacity of the networks from the evolving quality of the online replay buffer, these metrics are evaluated post-hoc on a fixed, held-out offline validation set.

Actor Feature Rank. This metric measures the effective dimensionality of the actor network’s penultimate representation. We pass 1,024 offline validation states through the actor and extract the penultimate layer activations (dimension 512) to form a feature matrix $X$. We center the features ($X_{\mathrm{centered}} = X - \mu_X$) and compute the singular values $s_i$ using SVD. We compute the threshold stable rank [18] as the minimum number of singular values required to capture 99% of the total singular value energy: $\sum_{i=1}^{k} s_i / \sum_i s_i \ge0.99$. A higher rank indicates that the policy maintains rich feature representations without collapsing into a low-dimensional subspace. This metric is motivated by recent findings linking representation dimensionality and geometry to learning stability under non-stationarity [42].

Action Diversity (Within State). This metric measures the proposal stochasticity and sensitivity of the actor to input noise at a given state. For 128 validation states, we sample multiple action candidates per state using different random noise seeds. We compute the standard deviation of the actions at each state and report the mean across all states. High within-state diversity indicates the generative model can produce multiple distinct valid actions; low values indicate a deterministic collapse.

Action Diversity (Between States). This metric measures the spatial dispersion of predicted actions across different states. We predict the mean action $a_i = \pi(s_i)$ for $K = 128$ distinct validation states and compute the mean pairwise Euclidean distance across all state pairs: $\frac{1}{K(K-1)}\sum_{i\ne j}\lVert a_i-a_j\rVert_2$. If this metric collapses near zero, the actor is outputting a constant action regardless of the input state.

Critic Feature Rank. This metric measures the effective representation rank of the $Q$-function network. We pass 1,024 offline $(s,a)$ pairs through the critic, extract the penultimate layer activations, and compute the threshold stable rank exactly as done for the actor (with a 99% cumulative energy threshold). This measures whether the critic maintains expressive, high-capacity features capable of accurate value prediction. Prior work has demonstrated the importance of structured representations for value learning and actor-critic sample efficiency ([36, 36]), while studies of network capacity and optimization highlight additional factors governing representation quality and adaptability ([38, 3, 27]).

Critic Separation Gap This metric quantifies the ability of the critic to separate high-quality transitions from poor ones. We partition the offline validation dataset by trajectory return, isolating the top 20% of transitions ($(s_{\mathrm{top}}, a_{\mathrm{top}})$) and the bottom 20% ($(s_{\mathrm{bot}}, a_{\mathrm{bot}})$). We evaluate the ensemble minimum $Q$-values for both subsets and compute the difference: $\mathbb{E}[Q(s_{\mathrm{top}}, a_{\mathrm{top}})]-\mathbb{E}[Q(s_{\mathrm{bot}}, a_{\mathrm{bot}})]$. A large positive gap indicates the critic correctly assigns higher values to high-return states, whereas a gap collapsing to zero indicates severe value distortion.

Experiments

Optimizer-state reset. One possible explanation for the offline-to-online collapse is that the optimizer carries stale Adam statistics from offline pretraining into a different online data distribution. If this were the primary cause, resetting Adam’s first- and second-moment estimates at the offline-to-online transition should restore performance, particularly in the failing DISCARD+BC-ON condition.

We therefore repeat the corresponding experiments with the optimizer state reset immediately before online fine-tuning, while keeping the pretrained network parameters and all other hyperparameters unchanged. As shown in Figure 12, resetting Adam does not recover performance in any of the six DISCARD+BC-ON cells: all remain at zero change. Under BC-OFF, the reset produces only small and inconsistent differences. These results indicate that stale optimizer statistics do not explain the collapse; the dominant failure is associated with continuing behavioral fitting on online-only data.