WorksComputer Science
Generating Critic-Guided Synthetic Experience in Latent World Models
Abstract
Learning from offline datasets is limited in scalability, as collecting new experience is often costly or unsafe in the real world. World models offer an alternative way to generate new experience through a learned dynamics model, yet learning policies in latent space is limited due to compounding errors in imagined rollouts. To this end, we propose using the world model as a local transition generator, leveraging critic guidance to choose actions inside that model to generate synthetic experiences. We study action refinement methodologies, introducing Critic-Guided Synthetic Experience (CGSE) generation in world models. Specifically, CGSE augments an existing offline dataset by sampling candidate actions using an offline policy and refining each to maximize a first-order approximation of the -function within a trust region. We then use a frozen latent world model to imagine the future state given each updated action, and use the critic to choose the best one, appending this synthetic transition to the replay buffer. Our method requires no additional real-environment interaction or test-time planner, as it generates synthetic experience offline using the world model. On three OGBench manipulation tasks, CGSE improves an offline policy by a relative improvement of on the average success rate.
Introduction
World models learn to predict the consequences of actions, providing an internal simulator in which agents can imagine and learn from experience [11]. They have been applied across robotics [1] (Yang et al. 2023), autonomous driving [18], and video games [5], with the promise of serving as a scalable substitute for costly real-world interaction in reinforcement learning [12, 13]. The appeal is a limitless source of synthetic experience: an agent could practice inside its own learned simulator before deployment, potentially unlocking artificial superhuman intelligence [Hughes et al., 2024]. However, world models are statistical approximations learned from data, lacking the explicit rules and underlying dynamics that govern traditional simulators. Consequently, latent representations in high-dimensional visual domains are optimized for prediction rather than task completion, and proximity in latent space does not necessarily correspond to meaningful progress toward a goal. Moreover, errors accumulate over imagined rollouts, causing value estimates to drift toward states that are unlikely to occur in the real environment [20] (Yu et al. 2020).
In this work, we propose that world models need not be faithful simulators to be useful: they can instead serve as controlled generators of additional experience, enabling policy improvement. Rather than treating the world model as a simulator for long-horizon planning, we use it as a generator of synthetic experience for offline reinforcement learning. Starting from states grounded in the offline dataset, we optimize candidate actions within the learned dynamics model and score them with an offline-trained critic rather than by latent-space distance. Trust-region guidance constrains optimization to remain local to the data distribution, limiting model exploitation and error compounding over rollouts. The resulting synthetic transitions augment the offline dataset, while value estimation remains anchored to real experience.
We summarize our key contributions as follows:
We introduce CGSE, a method that uses a frozen latent world model as a critic-guided generator of synthetic transitions, improving an offline policy without real-environment interaction or a test-time planner.
We show that bootstrapping value estimates on imagined successors can degrade performance, while keeping the critic grounded in offline data supports actor-only improvement.
On three robotic manipulation tasks from OGBench [31], our method achieves a 7.1% relative improvement in average task success over offline FQL.
Preliminaries
Markov Decision Process. Markov Decision Process (MDP) is defined by the tuple [34]. The state space consists of high-dimensional observations, while the transition dynamics are given by . The policy maps states to action distributions to maximize the expected discounted return. The action-value function encodes this objective for a given state-action pair:
To circumvent the intractability of planning directly in the high-dimensional observation space, all policy optimization, transition generation, and value estimation in our method are performed in a learned latent space .
Latent World Model. We utilize the LeWM latent action-conditioned world model [28], a Joint-Embedding Predictive Architecture (JEPA) comprising an encoder and a Transformer-based forward predictor . The encoder maps observations to latent states . The predictor advances the latent state conditioned on a temporally extended action chunk :
The model is self-supervised on offline play data using a latent prediction loss and the SIGReg regularizer to maintain a well-scaled latent distribution. Crucially, and remain strictly frozen throughout our framework: the encoder provides a stable state space, while the predictor acts as a differentiable, one-step deterministic simulator.
Flow Q-Learning. Our base agent uses an action-chunked, best-of- variant of Flow Q-Learning (FQL) [32, 26]. FQL models complex action distributions with a conditional flow and, in its original formulation, distills a separate one-step actor using critic guidance. Our implementation uses the flow policy directly and selects actions with the critic. For a dataset action chunk , Gaussian noise , and , its velocity network is trained by
Integrating this flow defines the proposal policy . During offline pretraining, the ensemble critic is trained by chunk-level temporal-difference learning using recorded rewards and successors. At deployment, the agent selects the highest-valued of four flow samples using the critic. Both components are pretrained in the frozen LeWM latent space; subsequent actor-only fine-tuning applies Eq. (3) to mixed replay.
Critic-Guided Synthetic Generation in World Models
Instead of utilizing the world model for unconstrained, long-horizon simulation, we deploy it as a generator of synthetic transitions to augment offline policy learning. The offline critic provides a task objective for improving candidate actions, and short model rollouts expose their predicted consequences. We transfer the selected behavior into the actor while retaining value estimates learned from real offline experience.
Critic-guided Action Refinement
To generate high-quality synthetic experience, the generator requires an action selector that proposes an optimized action chunk for a given latent state . Its proposal actor and critic are fixed at their offline-pretrained values. Policy samples provide plausible starting actions, but need not maximize task value. Flow Map -Guidance (FMQ) [Ziakas et al., 2026b] improves such proposals by maximizing a first-order approximation of the critic within a Euclidean trust region. This yields a single normalized gradient displacement, avoiding iterative action optimization. We adapt this guidance rule to a short model-based rollout score, using the offline flow policy to supply proposals.
For each candidate , set and recursively compute and . The -step score is
Here is the ensemble mean. Because each -value already summarizes future rewards, is an action-ranking surrogate, not an -step return estimate. With sampled continuation actions held fixed, let . The linearized trust-region problem has the solution
We refine each of proposals and clip to the action bounds:
Gradients pass through the latent predictions; continuation actions are detached and their sampling noise is fixed during differentiation. A fresh scoring pass evaluates the refined proposals, and is the candidate with the largest . The trust-region optimum applies to the linearized surrogate before clipping, not to true environment return.
Algorithm 1 CGSE: synthetic transition generation in a learned world model (single-task).
Require: offline dataset , frozen WM , fixed offline actor and critic
buffer ; initialize ; sample
for step do
▷ FMQ-style refinement and rollout scoring
▷ single model step
append to
sample minibatch from ; actor-only gradient step on ( frozen)
if not done, else resample ▷ restart rule: Appendix § 9
end for
Generating Synthetic Transitions
From a sampled empirical latent state , the action selector computes an optimized action chunk (using FMQ-style refinement). We then query the forward model for a single step, yielding and the synthetic transition . Multi-step lookahead is reserved for action scoring; each stored transition represents a one-step prediction. Generation continues from , so current states can accumulate prediction error along imagined chains. The real-state re-anchoring control instead samples an offline latent before every transition.
Policy Improvement with Synthetic Experience
The critical constraint of our framework is the treatment of the offline-trained critic during data consumption. Generated transitions are appended to a replay buffer initialized with the offline dataset . As the FQL actor is updated using mixed batches from , remains frozen. The actor loss consumes only sampled states and actions, matching the behaviors proposed by through Eq. (3) with replaced by . Neither synthetic rewards nor successor states enter this loss directly. In chained rollouts, however, generated successors become current states of later transitions, so model error can still affect the actor’s training distribution. Reward information guides action selection through the offline-trained critic. Figure 1 illustrates this generation process, and Algorithm 1 shows how it is interleaved with policy updates. Keeping the proposal actor, critic, and world model fixed maintains a stable action-supervision rule. This constrained, Dyna-style generation [36] adds no real-environment interaction and requires no world-model planner at test time.

Figure 1. CGSE transition generation in world model. Starting from an offline latent , the fixed BC-flow actor samples candidate first actions and FMQ-style guidance applies a one-step trust-region critic-gradient refinement to each candidate. The refined candidates are rolled out under , scored with , and the best branch supplies only its first action and first model successor . The resulting synthetic transition is appended to the replay buffer.
Experiments
Setup
We use OGBench [31] visual manipulation tasks ( RGB for our method, 5-D actions in chunks of ). The world model and the base action-chunked FQL agent are trained on the same OGBench play data; the world model is trained self-supervised and then frozen, while the policy and critic form the offline checkpoint used to initialize our method. The FQL agent uses discount , a two-critic ensemble, and best-of-four flow action selection. We report task-1 results and selector ablations on cube-single, scene, and puzzle-3x3. The metric is native MuJoCo success. CGSE reports meanstd over three seeds, with 500,000 generation/update steps. Imagined trajectories restart from an offline latent after predicted goal attainment or a maximum of 40 model steps (Appendix § 9). We use , , and .
As the baseline, we train offline FQL from scratch for 1M updates on the same task-specific play data with three independent seeds, sharing each task’s frozen LeWM encoder. CGSE fine-tunes a checkpoint pretrained for 500,000 updates with 500,000 generation/update steps, so all methods in Table 1 receive the same 1M gradient steps. Table 2 additionally reports ablations on cube-single in which all variants start from the same offline checkpoint.
Table 1: Success rate (mean std over three seeds, 250 episodes). Best per task in bold, second best underlined.
| Method | cube-single | scene | puzzle-3x3 | Average |
| Offline FQL | 84.0 4.2 | 15.3 3.5 | 45.2 10.2 | 48.2 1.2 |
| Synthetic experience generation | ||||
| No critic guidance | 80.4 3.2 | 9.1 2.1 | 37.2 3.0 | 42.2 1.4 |
| Best-of- | 86.3 3.1 | 13.2 3.7 | 50.5 2.7 | 50.0 1.3 |
| GradPlan | 88.9 0.2 | 18.7 1.6 | 46.4 3.5 | 51.3 1.0 |
| CGSE (Ours) | 90.7 0.9 | 17.1 2.4 | 46.9 3.6 | 51.6 1.8 |
Table 1.
Table 2: (a) Ablation on cube-single: all rows start from the same offline checkpoint and receive 500k additional actor updates; success (%), mean std over three seeds. (b) Generation cost per transition.
| (a) Ablation Study | |||||
| Training | Success (%) | ||||
| No world model | 89.3 2.1 | ||||
| CGSE, re-anchoring | 91.2 3.2 | ||||
| CGSE, chaining | 90.9 1.3 | ||||
| Horizon (frames) | 1 | 5 | 10 | 25 | 50 |
| Mean drift | 1.77 | 3.40 | 6.10 | 11.80 | 16.54 |
| (b) Time Complexity | |
| Selector | Time (ms) |
| Best-of- | 2.52 |
| GradPlan | 14.63 |
| CGSE | 5.34 |
Table 2. Mean latent prediction drift vs. rollout horizon on visual-cube-single.
For action-selection ablations, No critic guidance uses a single unrefined sample from the offline flow policy, Best-of-N ranks 32 unrefined proposals with Eq. (4), and GradPlan performs five gradient-ascent steps on the first chunk with step size 0.01. The model-based selectors share lookahead.
Weaker learned dynamics on scene and puzzle-3x3 (§ 4.3) likely limit the benefit of model-guided generation on these tasks.
Results
Table 1 compares CGSE with offline FQL trained from scratch in the same frozen LeWM latent space under the same number of gradient steps. CGSE outperforms offline FQL by a relative improvement of 8.0% on cube-single (84.0% to 90.7%) and of 7.1% on the average success rate, with no additional real interaction or test-time planner. scene improves by 11.8% and puzzle-3x3 by 3.8%, both within the seed variability of the baseline. Table 2 isolates the contribution of the world model: critic-guided action refinement with and without model-based lookahead. All variants start from the same pretrained cube checkpoint, use the same frozen latent representation and critic, and receive 500,000 additional actor updates. tbl:2b reports the generation cost per transition in milliseconds on an NVIDIA A100 GPU, with and for the model-based selectors.
No world model applies the same critic-guided refinement as CGSE (normalized gradient step on with , best of 32 proposals) but without any world-model rollout. Re-anchoring uses the model-based selector but samples an offline latent before every transition. CGSE improves over the no-world-model variant by 1.8% relative (95% paired interval on the difference: [−2.1, 5.6] points). Critic-guided refinement therefore accounts for most of the improvement; these experiments do not establish a significant additional benefit from model-based lookahead. Re-anchoring achieves similar success to chaining without a goal threshold.
Impact of Locality and Critic Grounding
The generator is useful only in a local operating regime. The first diagnostic is prediction drift: the latent error between rolled-out and true latents increases with horizon (Table 3). One model step ( frames) has drift 3.40, and two steps () have drift 6.10, comparable to the average distance between latent states; at longer horizons, drift exceeds the scale of meaningful state differences. Our generator therefore stores only single-step transitions and uses only for action scoring. Empirically, generated successors remain close to offline latents (nearest-neighbour ) and have norms on the same shell as real states.
The ablation rows of Table 1 compare action selection mechanisms and critic updates. Generating transitions with unrefined offline policy samples produces 80.4 3.2% final success on cube. Best-of- critic selection is already useful, reaching 86.3 3.1%. GradPlan provides an iterative optimization baseline, while CGSE provides our headline generator, reaching 90.7 0.9%. Their average success
rates are close (51.3% versus 51.6%), and these ablations do not establish a general performance advantage for CGSE. Its measured selector cost is lower: 5.34 ms versus 14.63 ms for GradPlan (Table 2b). When generated transitions also update the critic, success falls to 30.7 25.5%. This comparison changes rewards, successors, and actor-update conditions together and does not isolate their effects.
Related work
Model-based RL. Model-based reinforcement learning methods use rollouts from a learned dynamics model to augment policy or value learning [36, 20, 12, 13], and offline variants penalize or constrain model use in uncertain regions [30] (Yu et al. 2020; Yu et al. 2021). Separately, joint-embedding predictive architectures (JEPAs) predict future representations rather than reconstructing pixels [24, 3]. Latent world models have been used for planning and control [14, 15, 35, 4] (Zhang et al. 2026; Ziakas et al. 2026a). We build on a frozen action-conditioned latent world model [28], but use it as a source of synthetic experience rather than as a test-time planner.
Offline RL. Offline RL algorithms learn policies from fixed datasets [25]. Standard approaches include behavioral regularization [9, 29] (Wu et al. 2019), pessimistic value estimates [23, 2], and in-sample maximization [22, 10]. Diffusion and flow-matching policies [7, 33] are typically steered by the critic via weighted behavioral cloning [27, 21], reparameterized gradients [8] (Wang et al. 2023; Zhang et al. 2024), or best-of- rejection sampling [6, 16, 17]. Specifically, we leverage flow -learning with action chunking [32, 26] and FMQ [Ziakas et al., 2026b], which provides a closed-form trust-region step along the critic gradient. In fact, we refine flow-policy proposals using the FMQ step, score the resulting sequences through latent world-model rollouts, and execute the best candidate.
Conclusion
We show that a latent world model can generate local, critic-guided synthetic interaction. Actions are selected by short-horizon search, successors are single model steps, and the resulting transitions train the actor while the offline critic remains fixed. Relative to fixed offline starting checkpoints, CGSE-generated transitions improve average task success by 7.1% relative without real-environment interaction or test-time planning, with gains concentrated on cube-single. Our matched controls show that zero-step critic-guided action refinement accounts for the majority of this improvement, while our diagnostics reveal that weaker action-contingent dynamics and frozen visual representations bottleneck performance on multi-object tasks like scene and puzzle-3x3. Future work should expand this evaluation to a broader range of environments using stronger world models. A more systematic study of compute-budget scaling and task-aware representation learning methods would further strengthen our work.
Acknowledgments
This work was carried out as part of the MEng thesis of Jorge de Freitas at Imperial College London. This work was supported by UKRI (EP/Y037111/1) as part of the ProSafe project (EU Horizon 2020, MSCA, grant no. 101119358).
Implementation details
The world models follow the LeWM architecture [28]: a ViT-Tiny encoder to and a Transformer latent predictor, with each step spanning primitive actions (25-D action chunks), trained on each task family’s OGBench play data with a latent prediction loss plus SIGReg regularization and then frozen. The agent is action-chunked FQL [32, 26] with a best-of- flow actor (), a two-critic ensemble, discount , and target Polyak during offline pretraining, which runs for 500,000 updates. Synthetic generation and actor training then run for 500,000 steps with batch size 256 and learning rate . The replay buffer starts with 200k offline chunk transitions and appends one generated transition per update. Uniform sampling from all retained data gives an expected generated fraction of after updates, reaching 71.4% at 500k; the 2M capacity avoids eviction during these runs. CGSE uses FMQ-style refinement with proposals, look-ahead , -at-every-step scoring, and trust-region radius ; imagined episodes truncate at 40 steps. Evaluation is the native OGBench success rate over 250 episodes with no world-model planner; we report meanstd over three seeds.
The offline FQL baseline uses 1M offline updates, batch size 256, and learning rate . Selector timing uses an NVIDIA A100 GPU and excludes compilation, rendering, and policy-update time; it measures generation cost rather than total training time.
Imagined trajectory restarts
In chained generation, a trajectory ends when or after 40 model steps, then restarts from an offline latent. The distance check controls only trajectory termination; action scoring and actor training require no synthetic reward. Its sensitivity has not been tested. Real-state re-anchoring samples an offline latent before every transition and therefore requires no goal threshold.
References
- [1]Niket Agarwal et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025.arxiv.org/abs/2501.03575
- [2]Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436–7447, 2021.arxiv.org/abs/2110.01548
- [3]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture, 2023. URL https://arxiv.org/abs/2301.08243.
- [4]Marco Bagatella, Matteo Pirotta, Ahmed Touati, Alessandro Lazaric, and Andrea Tirinzoni. Td-jepa: Latent-predictive representations for zero-shot reinforcement learning, 2025. URL https://arxiv.org/abs/2510.00739.
- [5]Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, et al. Genie: Generative interactive environments. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2402.15391
- [6]Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating a trust region for offline reinforcement learning. Advances in Neural Information Processing Systems, 37: 50098–50125, 2024.arxiv.org/abs/2405.19690
- [7]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023.
- [8]Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309.16984, 2023.arxiv.org/abs/2309.16984
- [9]Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052–2062. PMLR, 2019.arxiv.org/abs/1812.02900
- [10]Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. In International Conference on Learning Representations, 2023.arxiv.org/abs/2301.02328
- [11]David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, volume 31, pages 2451–2463, 2018.arxiv.org/abs/1809.01999
- [12]Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.arxiv.org/abs/1912.01603
- [13]Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models, 2025. URL https://arxiv.org/abs/2509.24527.
- [14]Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control, 2022. URL https://arxiv.org/abs/2203.04955.
- [15]Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control, 2024. URL https://arxiv.org/abs/2310.16828.
- [16]Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. ldql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023.
- [17]Longxiang He, Li Shen, and Xueqian Wang. Aligniql: Policy alignment in implicit q-learning through constrained optimization. arXiv preprint arXiv:2405.18187, 2024.arxiv.org/abs/2405.18187
- [18]Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, et al. GAIA-1: A generative world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.arxiv.org/abs/2309.17080
- [20]Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization, 2019. URL https://arxiv.org/abs/1906.08253.
- [21]Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 67195–67212, 2023.arxiv.org/abs/2305.20081
- [22]Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.arxiv.org/abs/2110.06169
- [23]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 1179–1191, 2020.arxiv.org/abs/2006.04779
- [24]Yann LeCun. A path towards autonomous machine intelligence, version 0.9.2, 2022. URL https://openreview.net/pdf?id=BZ5a1r-kVsf.
- [25]Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.arxiv.org/abs/2005.01643
- [26]Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking, 2025. URL https://arxiv.org/abs/2507.07969.
- [27]Cheng Lu, Yang Hu, et al. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning. PMLR, 2023.arxiv.org/abs/2304.12824
- [28]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels, 2026. URL https://arxiv.org/abs/2603.19312.
- [29]Ashvin Nair and Murtaza reinforcement learning with offline datasets Dalal. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.arxiv.org/abs/2006.09359
- [30]Kwanyoung Park, Seohong Park, Youngwoon Lee, and Sergey Levine. Scalable offline model-based rl with action chunks, 2025a. URL https://arxiv.org/abs/2512.08108.
- [31]Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations, 2025b. URL https://arxiv.org/abs/2410.20092.
- [32]Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning, 2025c. URL https://arxiv.org/abs/2502.02538.
- [33]Tim Pearce, Tabish Rashid, Anssi Kanervisto, David Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hoffmann, and Sam Devlin. Imitating human behaviour with diffusion models. ArXiv, abs/2301.10677, 2023. URL https://api.semanticscholar.org/CorpusID:256231177.arxiv.org/abs/2301.10677
- [34]Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 2014.
- [35]Vlad Sobal, Wancong Zhang, Kyunghyun Cho, Randall Balestriero, Tim G. J. Rudner, and Yann LeCun. Learning from reward-free offline data: A case for planning with latent dynamics models, 2025. URL https://arxiv.org/abs/2502.14819.
- [36]Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bulletin, 2(4):160–163, 1991. doi: 10.1145/122344.122377. URL https://doi.org/10.1145/122344.122377.