betaFeedback

Reading v1View latest version

WorksComputer Science

Generating Critic-Guided Synthetic Experience in Latent World Models

Abstract

Learning from offline datasets is limited in scalability, as collecting new experience is often costly or unsafe in the real world. World models offer an alternative way to generate new experience through a learned dynamics model, yet learning policies in latent space is limited due to compounding errors in imagined rollouts. To this end, we propose using the world model as a local transition generator, leveraging critic guidance to choose actions inside that model to generate synthetic experiences. We study action refinement methodologies, introducing Critic-Guided Synthetic Experience (CGSE) generation in world models. Specifically, CGSE augments an existing offline dataset by sampling candidate actions using an offline policy and refining each to maximize a first-order approximation of the QQ-function within a trust region. We then use a frozen latent world model to imagine the future state given each updated action, and use the critic to choose the best one, appending this synthetic transition to the replay buffer. Our method requires no additional real-environment interaction or test-time planner, as it generates synthetic experience offline using the world model. On three OGBench manipulation tasks, CGSE improves an offline policy by a relative improvement of 7.1%7.1\% on the average success rate.

Introduction

World models learn to predict the consequences of actions, providing an internal simulator in which agents can imagine and learn from experience [11]. They have been applied across robotics [1] (Yang et al. 2023), autonomous driving [18], and video games [5], with the promise of serving as a scalable substitute for costly real-world interaction in reinforcement learning [12, 13]. The appeal is a limitless source of synthetic experience: an agent could practice inside its own learned simulator before deployment, potentially unlocking artificial superhuman intelligence [Hughes et al., 2024]. However, world models are statistical approximations learned from data, lacking the explicit rules and underlying dynamics that govern traditional simulators. Consequently, latent representations in high-dimensional visual domains are optimized for prediction rather than task completion, and proximity in latent space does not necessarily correspond to meaningful progress toward a goal. Moreover, errors accumulate over imagined rollouts, causing value estimates to drift toward states that are unlikely to occur in the real environment [20] (Yu et al. 2020).

In this work, we propose that world models need not be faithful simulators to be useful: they can instead serve as controlled generators of additional experience, enabling policy improvement. Rather than treating the world model as a simulator for long-horizon planning, we use it as a generator of synthetic experience for offline reinforcement learning. Starting from states grounded in the offline dataset, we optimize candidate actions within the learned dynamics model and score them with an offline-trained critic rather than by latent-space distance. Trust-region guidance constrains optimization to remain local to the data distribution, limiting model exploitation and error compounding over rollouts. The resulting synthetic transitions augment the offline dataset, while value estimation remains anchored to real experience.

We summarize our key contributions as follows:

  • We introduce CGSE, a method that uses a frozen latent world model as a critic-guided generator of synthetic transitions, improving an offline policy without real-environment interaction or a test-time planner.

  • We show that bootstrapping value estimates on imagined successors can degrade performance, while keeping the critic grounded in offline data supports actor-only improvement.

  • On three robotic manipulation tasks from OGBench [31], our method achieves a 7.1% relative improvement in average task success over offline FQL.

Preliminaries

Markov Decision Process. Markov Decision Process (MDP) is defined by the tuple (S,A,P,R,γ)(\mathcal{S}, \mathcal{A}, P, R, \gamma) [34]. The state space S\mathcal{S} consists of high-dimensional observations, while the transition dynamics are given by P(s′∣s,a)P(s^{\prime} \mid s, a). The policy π(a∣s)\pi(a \mid s) maps states to action distributions to maximize the expected discounted return. The action-value function encodes this objective for a given state-action pair:

Qπ(s,a)=Eπ[∑t=0∞γtr(st,at)∣s0=s, a0=a].(1)Q^{\pi}(s,a) = \mathbb{E}_{\pi}\left[\left.\sum_{t=0}^{\infty}\gamma^{t}r(s_t,a_t)\right|s_0=s,\ a_0=a\right]. \tag*{(1)}

To circumvent the intractability of planning directly in the high-dimensional observation space, all policy optimization, transition generation, and value estimation in our method are performed in a learned latent space Z\mathcal{Z}.

Latent World Model. We utilize the LeWM latent action-conditioned world model [28], a Joint-Embedding Predictive Architecture (JEPA) comprising an encoder EξE_{\xi} and a Transformer-based forward predictor fψf_{\psi}. The encoder maps observations to latent states z=Eξ(s)∈Zz = E_{\xi}(s) \in\mathcal{Z}. The predictor advances the latent state conditioned on a temporally extended action chunk a∈Aa \in\mathcal{A}:

z′=fψ(z,a).(2)z^{\prime} = f_{\psi}(z,a). \tag*{(2)}

The model is self-supervised on offline play data using a latent prediction loss and the SIGReg regularizer to maintain a well-scaled latent distribution. Crucially, EξE_{\xi} and fψf_{\psi} remain strictly frozen throughout our framework: the encoder provides a stable state space, while the predictor acts as a differentiable, one-step deterministic simulator.

Flow Q-Learning. Our base agent uses an action-chunked, best-of-NN variant of Flow Q-Learning (FQL) [32, 26]. FQL models complex action distributions with a conditional flow and, in its original formulation, distills a separate one-step actor using critic guidance. Our implementation uses the flow policy directly and selects actions with the critic. For a dataset action chunk aa, Gaussian noise ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I), and t∼U[0,1]t \sim\mathcal{U}[0,1], its velocity network vθv_{\theta} is trained by

LFlow(θ;D)=E(z,a)∼D,ϵ,t[∥vθ(z,(1−t)ϵ+ta,t)−(a−ϵ)∥22].(3)\mathcal{L}_{\mathrm{Flow}}(\theta;\mathcal{D}) = \mathbb{E}_{(z,a)\sim\mathcal{D},\epsilon,t}\left[\left\|v_{\theta}\left(z,(1-t)\epsilon+ta,t\right)-(a-\epsilon)\right\|_{2}^{2}\right]. \tag*{(3)}

Integrating this flow defines the proposal policy πθ\pi_{\theta}. During offline pretraining, the ensemble critic QϕQ_{\phi} is trained by chunk-level temporal-difference learning using recorded rewards and successors. At deployment, the agent selects the highest-valued of four flow samples using the critic. Both components are pretrained in the frozen LeWM latent space; subsequent actor-only fine-tuning applies Eq. (3) to mixed replay.

Critic-Guided Synthetic Generation in World Models

Instead of utilizing the world model for unconstrained, long-horizon simulation, we deploy it as a generator of synthetic transitions to augment offline policy learning. The offline critic provides a task objective for improving candidate actions, and short model rollouts expose their predicted consequences. We transfer the selected behavior into the actor while retaining value estimates learned from real offline experience.

Critic-guided Action Refinement

To generate high-quality synthetic experience, the generator requires an action selector M(z)M(z) that proposes an optimized action chunk for a given latent state zz. Its proposal actor π0\pi_0 and critic are fixed at their offline-pretrained values. Policy samples provide plausible starting actions, but need not maximize task value. Flow Map QQ-Guidance (FMQ) [Ziakas et al., 2026b] improves such proposals by maximizing a first-order approximation of the critic within a Euclidean trust region. This yields a single normalized gradient displacement, avoiding iterative action optimization. We adapt this guidance rule to a short model-based rollout score, using the offline flow policy to supply proposals.

For each candidate a0=a∼π0(⋅∣z)a_0 = a \sim\pi_0(\cdot\mid z), set z0=zz_0 = z and recursively compute zh+1=fψ(zh,ah)z_{h+1} = f_\psi(z_h, a_h) and ah+1∼π0(⋅∣zh+1)a_{h+1} \sim\pi_0(\cdot\mid z_{h+1}). The HH-step score is

J(a;z)=∑h=0H−1γhQϕ(zh+1,ah+1).(4)J(a; z) = \sum_{h=0}^{H-1} \gamma^h Q_\phi(z_{h+1}, a_{h+1}). \tag*{(4)}

Here QϕQ_\phi is the ensemble mean. Because each QQ-value already summarizes future rewards, JJ is an action-ranking surrogate, not an HH-step return estimate. With sampled continuation actions held fixed, let g=∇aJ(a;z)g = \nabla_a J(a; z). The linearized trust-region problem has the solution

Δ∗=arg⁡max⁡∥Δ∥2≤η [J(a;z)+gTΔ]=ηg∥g∥2,g≠0.(5)\Delta^* = \underset{\lVert\Delta\rVert_2 \le\eta}{\arg\max}\ \left[J(a; z) + g^\mathsf{T}\Delta\right] = \eta\frac{g}{\lVert g \rVert_2}, \qquad g \ne0. \tag*{(5)}

We refine each of NN proposals and clip to the action bounds:

a~=clip⁡(a+ηg∥g∥2,−1,1).(6)\tilde{a} = \operatorname{clip}\left(a + \eta\frac{g}{\lVert g \rVert_2}, -1, 1\right). \tag*{(6)}

Gradients pass through the latent predictions; continuation actions are detached and their sampling noise is fixed during differentiation. A fresh scoring pass evaluates the refined proposals, and a∗a^* is the candidate with the largest JJ. The trust-region optimum applies to the linearized surrogate before clipping, not to true environment return.

Algorithm 1 CGSE: synthetic transition generation in a learned world model (single-task).

Require: offline dataset D\mathcal{D}, frozen WM (Eξ,fψ)(E_{\xi}, f_{\psi}), fixed offline actor π0\pi_{0} and critic QϕQ_{\phi}

  1. buffer B←D\mathcal{B} \leftarrow\mathcal{D}; initialize πθ←π0\pi_{\theta} \leftarrow\pi_{0}; sample z∼Dz \sim\mathcal{D}

  2. for step =1,…,T=1,\ldots,T do

  3. a←M(z;π0,Qϕ,fψ)a \leftarrow\mathcal{M}(z;\pi_{0}, Q_{\phi}, f_{\psi}) ▷ FMQ-style refinement and rollout scoring

  4. z′←fψ(z,a)z' \leftarrow f_{\psi}(z,a) ▷ single model step

  5. append (z,a,z′)(z,a,z') to B\mathcal{B}

  6. sample minibatch from B\mathcal{B}; actor-only gradient step on πθ\pi_{\theta} (QϕQ_{\phi} frozen)

  7. z←z′z \leftarrow z' if not done, else resample z∼Dz \sim\mathcal{D} ▷ restart rule: Appendix § 9

  8. end for

Generating Synthetic Transitions

From a sampled empirical latent state z∼Dz \sim\mathcal{D}, the action selector M\mathcal{M} computes an optimized action chunk aa (using FMQ-style refinement). We then query the forward model for a single step, yielding z′=fψ(z,a)z' = f_{\psi}(z,a) and the synthetic transition (z,a,z′)(z,a,z'). Multi-step lookahead is reserved for action scoring; each stored transition represents a one-step prediction. Generation continues from z′z', so current states can accumulate prediction error along imagined chains. The real-state re-anchoring control instead samples an offline latent before every transition.

Policy Improvement with Synthetic Experience

The critical constraint of our framework is the treatment of the offline-trained critic QϕQ_{\phi} during data consumption. Generated transitions are appended to a replay buffer B\mathcal{B} initialized with the offline dataset D\mathcal{D}. As the FQL actor πθ\pi_{\theta} is updated using mixed batches from B\mathcal{B}, QϕQ_{\phi} remains frozen. The actor loss consumes only sampled states and actions, matching the behaviors proposed by M\mathcal{M} through Eq. (3) with D\mathcal{D} replaced by B\mathcal{B}. Neither synthetic rewards nor successor states enter this loss directly. In chained rollouts, however, generated successors become current states of later transitions, so model error can still affect the actor’s training distribution. Reward information guides action selection through the offline-trained critic. Figure 1 illustrates this generation process, and Algorithm 1 shows how it is interleaved with policy updates. Keeping the proposal actor, critic, and world model fixed maintains a stable action-supervision rule. This constrained, Dyna-style generation [36] adds no real-environment interaction and requires no world-model planner at test time.

CGSE transition generation in world model

Figure 1. CGSE transition generation in world model. Starting from an offline latent z0z_0, the fixed BC-flow actor samples NN candidate first actions and FMQ-style guidance applies a one-step trust-region critic-gradient refinement to each candidate. The refined candidates are rolled out under fψf_\psi, scored with QϕQ_\phi, and the best branch supplies only its first action a∗a^* and first model successor zψ′=fψ(z0,a∗)z'_\psi= f_\psi(z_0, a^*). The resulting synthetic transition (z0,a∗,zψ′)(z_0, a^*, z'_\psi) is appended to the replay buffer.

Experiments

Setup

We use OGBench [31] visual manipulation tasks (224×224224 \times224 RGB for our method, 5-D actions in chunks of h=5h=5). The world model and the base action-chunked FQL agent are trained on the same OGBench play data; the world model is trained self-supervised and then frozen, while the policy and critic form the offline checkpoint used to initialize our method. The FQL agent uses discount γ=0.99\gamma=0.99, a two-critic ensemble, and best-of-four flow action selection. We report task-1 results and selector ablations on cube-single, scene, and puzzle-3x3. The metric is native MuJoCo success. CGSE reports mean±\pmstd over three seeds, with 500,000 generation/update steps. Imagined trajectories restart from an offline latent after predicted goal attainment or a maximum of 40 model steps (Appendix § 9). We use N=32N=32, H=2H=2, and η=0.02\eta=0.02.

As the baseline, we train offline FQL from scratch for 1M updates on the same task-specific play data with three independent seeds, sharing each task’s frozen LeWM encoder. CGSE fine-tunes a checkpoint pretrained for 500,000 updates with 500,000 generation/update steps, so all methods in Table 1 receive the same 1M gradient steps. Table 2 additionally reports ablations on cube-single in which all variants start from the same offline checkpoint.

Table 1: Success rate (mean ±\pm std over three seeds, 250 episodes). Best per task in bold, second best underlined.

Methodcube-singlescenepuzzle-3x3Average
Offline FQL84.0 ±\pm 4.215.3 ±\pm 3.545.2 ±\pm 10.248.2 ±\pm 1.2
Synthetic experience generation
No critic guidance80.4 ±\pm 3.29.1 ±\pm 2.137.2 ±\pm 3.042.2 ±\pm 1.4
Best-of-NN86.3 ±\pm 3.113.2 ±\pm 3.750.5 ±\pm 2.750.0 ±\pm 1.3
GradPlan88.9 ±\pm 0.218.7 ±\pm 1.646.4 ±\pm 3.551.3 ±\pm 1.0
CGSE (Ours)90.7 ±\pm 0.917.1 ±\pm 2.446.9 ±\pm 3.651.6 ±\pm 1.8

Table 1.

Table 2: (a) Ablation on cube-single: all rows start from the same offline checkpoint and receive 500k additional actor updates; success (%), mean ±\pm std over three seeds. (b) Generation cost per transition.

(a) Ablation Study
TrainingSuccess (%)
No world model89.3 ±\pm 2.1
CGSE, re-anchoring91.2 ±\pm 3.2
CGSE, chaining90.9 ±\pm 1.3
Horizon Δ\Delta (frames)15102550
Mean drift1.773.406.1011.8016.54
(b) Time Complexity
SelectorTime (ms)
Best-of-NN2.52
GradPlan14.63
CGSE5.34

Table 2. Mean latent prediction drift ∥z^t+Δ−zt+Δ∥2\|\hat{z}_{t+\Delta}-z_{t+\Delta}\|_2 vs. rollout horizon on visual-cube-single.

For action-selection ablations, No critic guidance uses a single unrefined sample from the offline flow policy, Best-of-N ranks 32 unrefined proposals with Eq. (4), and GradPlan performs five gradient-ascent steps on the first chunk with step size 0.01. The model-based selectors share H=2H=2 lookahead.

Weaker learned dynamics on scene and puzzle-3x3 (§ 4.3) likely limit the benefit of model-guided generation on these tasks.

Results

Table 1 compares CGSE with offline FQL trained from scratch in the same frozen LeWM latent space under the same number of gradient steps. CGSE outperforms offline FQL by a relative improvement of 8.0% on cube-single (84.0% to 90.7%) and of 7.1% on the average success rate, with no additional real interaction or test-time planner. scene improves by 11.8% and puzzle-3x3 by 3.8%, both within the seed variability of the baseline. Table 2 isolates the contribution of the world model: critic-guided action refinement with and without model-based lookahead. All variants start from the same pretrained cube checkpoint, use the same frozen latent representation and critic, and receive 500,000 additional actor updates. tbl:2b reports the generation cost per transition in milliseconds on an NVIDIA A100 GPU, with N=32N=32 and H=2H=2 for the model-based selectors.

No world model applies the same critic-guided refinement as CGSE (normalized gradient step on Qϕ(z,a)Q_{\phi}(z,a) with η=0.02\eta=0.02, best of 32 proposals) but without any world-model rollout. Re-anchoring uses the model-based selector but samples an offline latent before every transition. CGSE improves over the no-world-model variant by 1.8% relative (95% paired interval on the difference: [−2.1, 5.6] points). Critic-guided refinement therefore accounts for most of the improvement; these experiments do not establish a significant additional benefit from model-based lookahead. Re-anchoring achieves similar success to chaining without a goal threshold.

Impact of Locality and Critic Grounding

The generator is useful only in a local operating regime. The first diagnostic is prediction drift: the latent error ∥z^t+Δ−zt+Δ∥2\lVert\hat{z}_{t+\Delta} - z_{t+\Delta} \rVert_{2} between rolled-out and true latents increases with horizon (Table 3). One model step (Δ=5\Delta=5 frames) has drift 3.40, and two steps (Δ=10\Delta=10) have drift 6.10, comparable to the ≈7.4\approx7.4 average distance between latent states; at longer horizons, drift exceeds the scale of meaningful state differences. Our generator therefore stores only single-step transitions and uses H=2H=2 only for action scoring. Empirically, generated successors remain close to offline latents (nearest-neighbour L2≈0.62L_{2} \approx0.62) and have norms on the same 7.8±1.27.8 \pm1.2 shell as real states.

The ablation rows of Table 1 compare action selection mechanisms and critic updates. Generating transitions with unrefined offline policy samples produces 80.4 ±\pm 3.2% final success on cube. Best-of-NN critic selection is already useful, reaching 86.3 ±\pm 3.1%. GradPlan provides an iterative optimization baseline, while CGSE provides our headline generator, reaching 90.7 ±\pm 0.9%. Their average success

rates are close (51.3% versus 51.6%), and these ablations do not establish a general performance advantage for CGSE. Its measured selector cost is lower: 5.34 ms versus 14.63 ms for GradPlan (Table 2b). When generated transitions also update the critic, success falls to 30.7 ±\pm 25.5%. This comparison changes rewards, successors, and actor-update conditions together and does not isolate their effects.

Related work

Model-based RL. Model-based reinforcement learning methods use rollouts from a learned dynamics model to augment policy or value learning [36, 20, 12, 13], and offline variants penalize or constrain model use in uncertain regions [30] (Yu et al. 2020; Yu et al. 2021). Separately, joint-embedding predictive architectures (JEPAs) predict future representations rather than reconstructing pixels [24, 3]. Latent world models have been used for planning and control [14, 15, 35, 4] (Zhang et al. 2026; Ziakas et al. 2026a). We build on a frozen action-conditioned latent world model [28], but use it as a source of synthetic experience rather than as a test-time planner.

Offline RL. Offline RL algorithms learn policies from fixed datasets [25]. Standard approaches include behavioral regularization [9, 29] (Wu et al. 2019), pessimistic value estimates [23, 2], and in-sample maximization [22, 10]. Diffusion and flow-matching policies [7, 33] are typically steered by the critic via weighted behavioral cloning [27, 21], reparameterized gradients [8] (Wang et al. 2023; Zhang et al. 2024), or best-of-NN rejection sampling [6, 16, 17]. Specifically, we leverage flow QQ-learning with action chunking [32, 26] and FMQ [Ziakas et al., 2026b], which provides a closed-form trust-region step along the critic gradient. In fact, we refine NN flow-policy proposals using the FMQ step, score the resulting sequences through latent world-model rollouts, and execute the best candidate.

Conclusion

We show that a latent world model can generate local, critic-guided synthetic interaction. Actions are selected by short-horizon search, successors are single model steps, and the resulting transitions train the actor while the offline critic remains fixed. Relative to fixed offline starting checkpoints, CGSE-generated transitions improve average task success by 7.1% relative without real-environment interaction or test-time planning, with gains concentrated on cube-single. Our matched controls show that zero-step critic-guided action refinement accounts for the majority of this improvement, while our diagnostics reveal that weaker action-contingent dynamics and frozen visual representations bottleneck performance on multi-object tasks like scene and puzzle-3x3. Future work should expand this evaluation to a broader range of environments using stronger world models. A more systematic study of compute-budget scaling and task-aware representation learning methods would further strengthen our work.

Acknowledgments

This work was carried out as part of the MEng thesis of Jorge de Freitas at Imperial College London. This work was supported by UKRI (EP/Y037111/1) as part of the ProSafe project (EU Horizon 2020, MSCA, grant no. 101119358).

Implementation details

The world models follow the LeWM architecture [28]: a ViT-Tiny encoder to R192\mathbb{R}^{192} and a Transformer latent predictor, with each step spanning h=5h=5 primitive actions (25-D action chunks), trained on each task family’s OGBench play data with a latent prediction loss plus SIGReg regularization and then frozen. The agent is action-chunked FQL [32, 26] with a best-of-NN flow actor (Nact=4N_{\mathrm{act}}=4), a two-critic ensemble, discount γ=0.99\gamma=0.99, and target Polyak τ=0.005\tau=0.005 during offline pretraining, which runs for 500,000 updates. Synthetic generation and actor training then run for 500,000 steps with batch size 256 and learning rate 10−410^{-4}. The replay buffer starts with 200k offline chunk transitions and appends one generated transition per update. Uniform sampling from all retained data gives an expected generated fraction of t/(200,000+t)t/(200{,}000+t) after tt updates, reaching 71.4% at 500k; the 2M capacity avoids eviction during these runs. CGSE uses FMQ-style refinement with N=32N=32 proposals, look-ahead H=2H=2, QQ-at-every-step scoring, and trust-region radius η=0.02\eta=0.02; imagined episodes truncate at 40 steps. Evaluation is the native OGBench success rate over 250 episodes with no world-model planner; we report mean±\pmstd over three seeds.

The offline FQL baseline uses 1M offline updates, batch size 256, and learning rate 3×10−43\times10^{-4}. Selector timing uses an NVIDIA A100 GPU and excludes compilation, rendering, and policy-update time; it measures generation cost rather than total training time.

Imagined trajectory restarts

In chained generation, a trajectory ends when ∥z′−zgoal∥2<2.0\lVert z^{\prime}-z_{\mathrm{goal}}\rVert_2<2.0 or after 40 model steps, then restarts from an offline latent. The distance check controls only trajectory termination; action scoring and actor training require no synthetic reward. Its sensitivity has not been tested. Real-state re-anchoring samples an offline latent before every transition and therefore requires no goal threshold.

References

  1. [1]Niket Agarwal et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025.arxiv.org/abs/2501.03575
  2. [2]Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436–7447, 2021.arxiv.org/abs/2110.01548
  3. [3]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture, 2023. URL https://arxiv.org/abs/2301.08243.
  4. [4]Marco Bagatella, Matteo Pirotta, Ahmed Touati, Alessandro Lazaric, and Andrea Tirinzoni. Td-jepa: Latent-predictive representations for zero-shot reinforcement learning, 2025. URL https://arxiv.org/abs/2510.00739.
  5. [5]Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, et al. Genie: Generative interactive environments. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2402.15391
  6. [6]Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating a trust region for offline reinforcement learning. Advances in Neural Information Processing Systems, 37: 50098–50125, 2024.arxiv.org/abs/2405.19690
  7. [7]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023.
  8. [8]Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309.16984, 2023.arxiv.org/abs/2309.16984
  9. [9]Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052–2062. PMLR, 2019.arxiv.org/abs/1812.02900
  10. [10]Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. In International Conference on Learning Representations, 2023.arxiv.org/abs/2301.02328
  11. [11]David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, volume 31, pages 2451–2463, 2018.arxiv.org/abs/1809.01999
  12. [12]Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.arxiv.org/abs/1912.01603
  13. [13]Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models, 2025. URL https://arxiv.org/abs/2509.24527.
  14. [14]Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control, 2022. URL https://arxiv.org/abs/2203.04955.
  15. [15]Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control, 2024. URL https://arxiv.org/abs/2310.16828.
  16. [16]Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. ldql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023.
  17. [17]Longxiang He, Li Shen, and Xueqian Wang. Aligniql: Policy alignment in implicit q-learning through constrained optimization. arXiv preprint arXiv:2405.18187, 2024.arxiv.org/abs/2405.18187
  18. [18]Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, et al. GAIA-1: A generative world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.arxiv.org/abs/2309.17080
  19. [20]Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization, 2019. URL https://arxiv.org/abs/1906.08253.
  20. [21]Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 67195–67212, 2023.arxiv.org/abs/2305.20081
  21. [22]Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.arxiv.org/abs/2110.06169
  22. [23]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 1179–1191, 2020.arxiv.org/abs/2006.04779
  23. [24]Yann LeCun. A path towards autonomous machine intelligence, version 0.9.2, 2022. URL https://openreview.net/pdf?id=BZ5a1r-kVsf.
  24. [25]Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.arxiv.org/abs/2005.01643
  25. [26]Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking, 2025. URL https://arxiv.org/abs/2507.07969.
  26. [27]Cheng Lu, Yang Hu, et al. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning. PMLR, 2023.arxiv.org/abs/2304.12824
  27. [28]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels, 2026. URL https://arxiv.org/abs/2603.19312.
  28. [29]Ashvin Nair and Murtaza reinforcement learning with offline datasets Dalal. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.arxiv.org/abs/2006.09359
  29. [30]Kwanyoung Park, Seohong Park, Youngwoon Lee, and Sergey Levine. Scalable offline model-based rl with action chunks, 2025a. URL https://arxiv.org/abs/2512.08108.
  30. [31]Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations, 2025b. URL https://arxiv.org/abs/2410.20092.
  31. [32]Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning, 2025c. URL https://arxiv.org/abs/2502.02538.
  32. [33]Tim Pearce, Tabish Rashid, Anssi Kanervisto, David Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hoffmann, and Sam Devlin. Imitating human behaviour with diffusion models. ArXiv, abs/2301.10677, 2023. URL https://api.semanticscholar.org/CorpusID:256231177.arxiv.org/abs/2301.10677
  33. [34]Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 2014.
  34. [35]Vlad Sobal, Wancong Zhang, Kyunghyun Cho, Randall Balestriero, Tim G. J. Rudner, and Yann LeCun. Learning from reward-free offline data: A case for planning with latent dynamics models, 2025. URL https://arxiv.org/abs/2502.14819.
  35. [36]Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bulletin, 2(4):160–163, 1991. doi: 10.1145/122344.122377. URL https://doi.org/10.1145/122344.122377.

Paper details

vlnc:W_y9bucc25
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 16

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents