WorksComputer Science
MAC-v2: Long-horizon model-based value expansion from pixels
Abstract
We study whether long-horizon model-based value expansion can serve as an effective framework in pixel-based offline reinforcement learning (RL). Long-horizon model-based value expansion from pixels is substantially harder than state-based environments, as the agent must learn both the representation and its dynamics, and errors from either can accumulate recursively over imagined trajectories. We introduce MAC-v2, which enables long-horizon model-based expansion from pixels by training a reconstruction-free latent world model with action chunking. By learning representations from prediction and control objectives, MAC-v2 does not need to encode control-irrelevant visual details that can hinder the dynamics prediction. Action chunking further mitigates the prediction error by reducing the number of recursive model predictions required for value expansion. Across diverse visual benchmarks including visual OGBench, MMBench2, Robomimic, and LIBERO-10, MAC-v2 achieves strong performance against model-free and model-based offline RL baselines. Our analyses show that long-horizon model-based value expansion is critical for the performance of MAC-v2, and that both action chunking and latent learning objectives are essential for reliable long-horizon latent imagination.
Introduction
Many real-world decision-making problems, such as robotic manipulation, require agents to learn directly from high-dimensional visual observations. This is particularly challenging in offline reinforcement learning (RL), where policies must be learned entirely from a fixed dataset without additional environment interaction. Unlike online RL, offline RL cannot evaluate the policy through environment interaction and rely on off-policy value estimates learned from the fixed dataset. As the learned policy deviates from the behavior policy, these off-policy estimates can become inaccurate due to distribution shift. Model-based value expansion ([6]) offers an appealing alternative by learning a world model, where on-policy trajectories can be generated by using a world model without accessing the real environment ([59, 19, 53, 13] (Park & Lee 2025)).
Prior works have shown that model-based value expansion can be applied to offline RL from pixels, but typically only over short imagination horizons. For example, methods such as LOMPO ([46]), OfflineDV2 ([34]), and SeMOPO ([56]) learn latent world models from visual observations and mitigate model exploitation through uncertainty estimation, conservative penalties, or representation decomposition. While these approaches demonstrate that model-based value expansion can be effective, their model rollouts are generally kept short (e.g., LOMPO, OfflineDV2, and SeMOPO use an imagination horizon of 5 steps). In contrast, recent state-based offline RL methods have shown that substantially longer model-based value expansion can be effective for long-horizon tasks ([41, 4]).
Long-horizon model-based value expansion from pixels is substantially harder, as the agent must learn both the representation and its dynamics from the fixed offline data. Since pixel observations contain substantial control-irrelevant information ([7, 35]), training a world model directly on pixels becomes unnecessarily challenging. This motivates learning a compact latent representation for world modeling. Here, the key challenge is to learn a representation that preserves the information needed for control, and also be predictable by the learned dynamics [18, 52]. Otherwise, this can lead to inaccurate value estimation, as errors from either the representation or the dynamics can accumulate recursively over imagined trajectories, and this issue is further exacerbated in the offline setting, where the agent cannot collect corrective data when those trajectories reach poorly modeled regions.
We introduce MAC-v2, which enables long-horizon model-based value expansion from pixels by training a reconstruction-free latent world model with action chunking. By learning representations from prediction and control objectives, MAC-v2 does not need to encode control-irrelevant visual details that can hinder the dynamics prediction, while also preserving the information needed for control. Moreover, we utilize action chunking [61] for the policy and the world model, reducing the number of recursive model predictions required over long horizons [41]. Together, these design choices mitigate the compounding error during imagined rollouts and enable long-horizon model-based value expansion from visual observations.
We evaluate MAC-v2 across a broad suite of visual benchmarks and show that it matches or substantially outperforms prior model-free and model-based RL methods that rely on shorter-horizon value estimates. MAC-v2 substantially outperforms prior model-free and model-based baselines on 25 visual OGBench [42], and matches the performance of the best prior methods in 10 MMBench2 [14] tasks. We further evaluate MAC-v2 on robotic manipulation benchmarks, including Robomimic [37] and LIBERO-10 [32], where the offline datasets consist primarily of expert demonstrations and behavior cloning policy serves as a strong baseline. On Robomimic, MAC-v2 is the only RL method that matches or exceeds behavior cloning across all evaluated tasks, while remaining competitive with strong behavioral baselines on LIBERO-10. Our analyses show that long-horizon model-based value expansion is critical for this performance, and learning a reconstruction-free encoder with both world model and control objectives is important for reliable long-horizon latent rollouts. Overall, our results suggest that, with the design choices of MAC-v2, latent world models can support effective long-horizon model-based value expansion from pixels.
Related Works
Offline model-free RL. Offline RL aims to learn policies solely from pre-collected datasets without further environment interaction [24, 28]. A central challenge is distribution shift, as learned policies may select out-of-distribution states or actions for which value estimates are unreliable. Prior work addresses this issue through conservative objectives [23], behavior regularization [57, 45, 39, 8, 54, 44], in-sample value maximization [22, 58, 9], and sample-selection methods [2, 17]. More recently, action chunking has shown strong performance on long-horizon offline RL tasks [30, 50] by improving temporal consistency and reducing the effective decision horizon. However, these model-free approaches rely on short-horizon off-policy targets for value learning, which can limit performance due to inaccurate value propagation [43]. In contrast, MAC-v2 learns values from long-horizon on-policy targets generated by a model through action-chunked latent rollouts, reducing the error from the recursive bootstrapping.
Offline model-based RL. MAC-v2 is a model-based offline RL method. Such methods learn a dynamics model from the fixed dataset and use it to generate synthetic trajectories for policy evaluation and optimization. By evaluating the policy through the learned model, they can utilize on-policy value estimation instead of off-policy estimates. However, policies may exploit inaccuracies in the learned dynamics model, leading to unreliable value estimates. Prior work addresses this issue through conservative objectives [19, 60], uncertainty-aware penalties [59, 53, 49] (R|gter et al. 2022; Park & Lee 2025), and in-distribution policy optimization [41]. In particular, MAC [41] combines action-chunked dynamics with a rejection sampling policy, enabling long-horizon model-based value expansion on state-based tasks. MAC-v2 builds on this framework, but addresses the additional challenge of learning a representation that can support accurate recursive model predictions directly from visual observations.
Online RL from pixels. Learning from pixels requires representations that retain information relevant for control while remaining compact and predictable. Prior work has studied this problem extensively in online RL, including model-free approaches [38, 21, 26] and model-based approaches [11, 12, 15, 16]. Representation objectives include image reconstruction [11, 12 b], contrastive learning [25, 51], and reconstruction-free latent prediction [48, 16, 36]. MAC-v2 adopts the reconstruction-free latent prediction paradigm established by the prior works, and we do not propose a entirely new representation learning objective. Instead, we study how to utilize such representations for long-horizon model rollouts, required by offline model-based value expansion. We show that, when combined with action chunking, predictive latent representations can support long-horizon imagination and value estimation from a fixed visual dataset.
Offline model-based RL from pixels. Several prior works have specifically studied model-based offline RL from visual observations using reconstruction-based latent representations. LOMPO [46] learns a latent dynamics model and calculates model uncertainty in latent space, while OfflineDV2 [34] adapts DreamerV2 [10] to offline visual control with uncertainty-aware penalties. However, model imagination is typically kept relatively short (e.g., LOMPO and OfflineDV2 use imagination of 5 steps). We instead study whether predictive latent representations can support long-horizon model-based value expansion from pixels. By combining latent prediction with action chunking, MAC-v2 enables stable value expansion over more than 100 environment steps.
Preliminaries
Problem setting. We consider a block Markov decision process (MDP) defined as , where is the latent environment state space, is the observation space, and is the action space. The reward function is , is the initial state distribution, is the transition dynamics, and is the observation model. The agent does not observe directly and instead receives . We assume that the current observation is sufficient to identify the underlying state for decision making, so that reactive policies are sufficient. For a policy , we define , where is the discount factor. The goal of offline RL is to learn a policy maximizing using only a fixed dataset of previously collected trajectories, without additional environment interaction.
Flow Matching Policy. A flow matching policy [31, 1, 33] represents a noise-conditioned deterministic policy by transporting samples from a simple reference distribution to the action distribution conditioned on a state. Let denote a state-conditioned, time-dependent velocity field, where denotes the flow time. Given an initial noise sample , the corresponding flow is defined by the ordinary differential equation (ODE) [27]
The policy is defined as the endpoint of this flow,
which induces a conditional action distribution through .
Given state-action samples , flow matching trains the velocity field along linear interpolation paths between Gaussian noise and the target action . Specifically, for
the velocity field is trained by minimizing
After training, actions are sampled by drawing and numerically integrating the learned velocity field from to .
MAC-v2
We present MAC-v2, an offline model-based RL framework that enables long-horizon model-based value expansion from visual observations. MAC-v2 builds on two lines of work: model-based value expansion with action chunking from MAC [41], and latent world models trained with prediction and control objectives akin to reconstruction-free online model-based RL literatures [48, 15, 16]. Neither line of work alone enables long-horizon model-based value expansion in pixel-based offline RL, as MAC-v2 assumes low-dimensional state observations, while prior reconstruction-free latent world models have been limited to short-horizon rollouts.
Our key design choice is to learn a reconstruction-free latent dynamics with action chunking. With action chunking size of , an imagination horizon of environment steps requires only recursive model predictions, reducing the compounding of prediction errors. Moreover, we learn the representation with world modeling and behavior cloning objective, encouraging the representation to preserve the information for reward prediction and control, while being predictable under the learned dynamics model. We describe our world model objectives in Section 4.1, and the long-horizon model-based value expansion with the learned latent space in Section 4.2.
Learning Representations for Long-Horizon Imagination
We encode each image observation into a latent state using a deterministic encoder . Rather than reconstructing the input image, MAC-v2 learns the encoder through objectives relevant to latent prediction and control. In particular, gradients from the dynamics, reward, and behavior cloning objectives are propagated into the encoder, learning a compact representation that retains information for predicting future latent states, rewards, and control.
Concretely, given a latent state and an action chunk, we learn a deterministic dynamics model to predict the latent state after environment steps, by minimizing
where denotes the stop-gradient operator.
We also learn a distributional action-chunked reward model with two-hot encoding [12]. We regress the target return by minimizing
We parameterize the behavioral policy as a conditional flow model, and train a state-conditioned velocity field using the flow-matching objective
We denote by the endpoint at obtained by integrating the learned velocity field from the initial noise .
Long-Horizon Latent Value Expansion with Action Chunking
With the learned latent world model, we apply the action-chunked value-learning framework of MAC [41] in the latent space. First, we extract the policy from the behavioral policy by sampling behavioral action candidates from multiple noises , and extracting the best action among the candidates via rejection sampling [2]:
We train the value function purely from latent on-policy model rollout, generated as
where denotes the action chunk generated from the rejection sampling policy , and denotes the discounted sum of rewards predicted from the reward model Algorithm 1 MAC-v2
Input: Dataset , rollout length , action chunking size , rejection sampling size , encoder , dynamics / reward model / , value , Q-function , flow BC policy
// Training loop
while not converged do
▷ Sample action-chunked batch from the dataset (, )
Sample batch
▷ Train dynamics, reward model (and encoder) using dataset transitions
Update to minimize (Equation (5))
Update to maximize (Equation (6))
▷ Train BC policy (and encoder) using dataset transitions
Update with flow-matching loss (Equation (7))
▷ Generate model rollouts in latent space ()
for do
▷ Update value and critic using model rollouts
Update with -step targets from the rollout (Equation (10))
Update with the learned value function (Equation (11))
// Extract action from flow BC policy with rejection sampling
function
return
. Here, initial states is uniformly sampled from the dataset and encoded with , and subsequent actions, rewards, and next latent states are synthesized by our rejection-sampling policy, reward model, and dynamics model, respectively, where we denote with a hat notation.
After collecting the trajectory, we learn a distributional value function with the following loss:
where denotes exponentially averaged target parameters [38], , and the expectations are over and uniformly sampled from .
Finally, we train the action-chunked Q function for the rejection sampling with the following loss:
For both value learning, we use two-hot encoding for the distribution. We provide a pseudocode for MAC-v2 in Algorithm 1.
Experiments
We now empirically evaluate MAC-v2 and examine whether long-horizon model-based value expansion is effective for offline RL from pixels. First, we show that MAC-v2 can serve as an effective general-purpose offline RL algorithm from visual observations in § 5.1, by evaluating on offline RL benchmarks with diverse control problems and challenging long-horizon tasks where accurate value propagation is important. Second, we examine whether MAC-v2 can be also applied in robotic manipulation settings, where the offline datasets are closer to expert demonstrations and strong behavioral cloning policies already provide competitive performance, in § 5.2. Finally, we analyze the importance of long-horizon model-based value expansion of MAC-v2 and ablate the training objectives used for world model learning in § 5.3. Unless otherwise specified, all results are averaged over four random seeds.
Visual offline RL benchmarks
We first evaluate whether MAC-v2 can serve as a general-purpose offline RL method from pixels. We consider two complementary benchmark suites. Visual OGBench [42] contains structured manipulation problems with long temporal horizons, suited for evaluating whether model-based value expansion can propagate useful information over long sequences. MMBench2 [14] provides a more diverse visual control tasks and diverse source of data collection, testing whether it can be used for beyond the structured manipulation setting.
Experimental setting
Tasks and datasets. We evaluate on 25 visual single-task environments from OGBench [42] and report average success rates within each task family, following the evaluation protocol of Park et al. [44]. We additionally evaluate on 10 “unseen” tasks from MMBench2 [14], which cover a diverse set of visual control problems. We resize observations to and combine all the datasets (e.g., collected from curiosity-driven policy, expert policy, random policy) assigned to each task. For MMBench2, we report normalized return on a 0-100 scale using the minimum and maximum trajectory returns from the benchmark.
Methods. We compare MAC-v2 against strong model-free and model-based offline RL baselines. For model-free methods, we consider IQL [22], ReBRAC [54], IFQL, the flow-based variant of IDQL [17], and FQL [44]. For model-based methods, we focus specifically on prior offline RL methods that learn from visual observations. This literature is considerably smaller than the broader literature on online visual model-based RL methods, which are not directly comparable in our setting. For offline model-based methods, we include LOMPO [46], which learns a latent world model and apply uncertainty-aware policy optimization on top of the frozen representation, OfflineDV2 [34] (denoted as OffDV2), which combines uncertainty penalty [59] on top of the world model from DreamerV2 [10].
Results
Table 1 summarizes the results on the visual OGBench and MMBench2 tasks. On OGBench, MAC-v2 outperforms all baselines in 4 out of 5 task families, and matches QC on the remaining one (visual-cube-single). Notably, MAC-v2 shows 95% score in puzzle-3x3, where the best prior method scores 22%, while prior model-based RL methods score near-zero performance across all tasks. On MMBench2, MAC-v2 achieves the best performance on the ogbench and pygame domains, and remains competitive with strong model-free baselines such as FQL on the remaining tasks. Together, these results shows the effectiveness of long-horizon model-based value expansion across diverse visual offline RL problems.
| Environment | Model-Free | Model-Based | ||||||
| IQL | ReBRAC | IFQL | FQL | QC | LOMPO | OffDV2 | MAC-v2 | |
| OGBench (25 tasks) | ||||||||
| visual-cube-single (5 tasks) | ||||||||
| visual-cube-double (5 tasks) | ||||||||
| visual-scene (5 tasks) | ||||||||
| visual-puzzle-3x3 (5 tasks) | ||||||||
| visual-puzzle-4x4 (5 tasks) | ||||||||
| MMBench2 (10 tasks) | ||||||||
| dmcontrol (2 tasks) | ||||||||
| maniskill (1 task) | ||||||||
| ogbench (2 tasks) | ||||||||
| pygame (5 tasks) |
Table 1. Results on standard offline RL visual benchmark tasks. We report the success rate for OGBench tasks and normalized scores (0-100) for MMBench2 tasks across 4 seeds, averaged over 15 episodes. Best scores for each task are highlighted in magenta.
Robotic manipulation benchmarks
We next study the case where offline datasets are composed primarily of successful or near-expert demonstrations. In these datasets, behavioral cloning is already a strong baseline, which is challenging for offline RL algorithms to improve upon. This evaluation tests whether model-based value expansion can also be effective in these datasets, and whether it outperforms behavioral cloning policies.
Experimental setting
Tasks and datasets. We evaluate on three visual manipulation tasks from Robomimic (Mandlekar et al. [37]) and the ten long-horizon tasks from LIBERO-10 (Liu et al. [32]). For Robomimic, we use the multi-human (mh) datasets and observations from both the agent-view and wrist cameras. For LIBERO-10, we use the original observations and only use the agent-view camera. We report average task success rates on both benchmarks.
Methods. Because temporally coherent action sequences are particularly important in these robotic manipulation tasks, we compare against strong model-free baselines that utilizes action chunking. These include action-chunked flow-matching BC (Lipman et al. [31]), QC (Li et al. [30]), which learns an action-chunked value function and performs rejection sampling over a flow-based BC policy, and QC-IFQL, an action-chunked variant of IFQL (Hansen-Estruch et al. [17]). We also compare with model-based baselines, including LOMPO (Rafailov et al. [46]) and OfflineDV2 (Lu et al. [34]) (denoted as OffDV2).
Results
Table 2 summarizes the results on visual robotic manipulation. On Robomimic, MAC-v2 matches or exceeds BC on all three tasks and improves performance on the challenging square task. Notably, MAC-v2 is the only RL method that consistently matches or outperforms BC across the full benchmark. On LIBERO-10, MAC-v2 remains competitive with BC and strong action-chunked model-free methods. We found that LIBERO-10 consists largely of unimodal demonstration data, leaving relatively limited room for rejection sampling to improve the policy over the behavioral policy. These results suggest that MAC-v2 can retain the robustness of behavior cloning in robotic settings, while providing meaningful improvements when value-based policy selection has sufficient room to improve upon the dataset policy.
| Environment | Model-Free | Model-Based | ||||
| BC | QC | QC-IFQL | LOMPO | OffDV2 | MAC-v2 | |
| Robomimic (3 tasks) | ||||||
| lift | ||||||
| can | ||||||
| square | ||||||
| LIBERO (10 tasks) | ||||||
| LIBERO-10 (10 tasks) |
Table 2. Results on visual robotic benchmarks. We report the success rate across 4 seeds, averaged over 15 episodes. Best scores for each task are highlighted in magenta.
Analysis
In this section, we discuss and analyze the components of MAC-v2 through the following Q&As. Specifically, we examine whether long-horizon model-based value expansion drives its performance gains, and which components make such expansion effective.
Q: Is long-horizon model-based value expansion important for MAC-v2?
A: To answer this question, we compare the performance of MAC-v2 with various horizons, by varying the rollout horizon and action chunk length , summarized in Figure 1. First, we evaluate rollout horizons of environment steps with an action chunk size of 10 on puzzle-4x4-task5. Performance improves as the horizon increases, demonstrating the benefit of long-horizon value expansion. We also vary the action chunk size over on puzzle-4x4-task1, and the performance improves as the chunk length increases, consistent with the result with the rollout length. Together, these results support that long-horizon model-based value expansion with action chunking is important for the performance of MAC-v2.

Figure 1. Importance of long-horizon rollout in MAC-v2.
Q: Can prior methods also perform long-horizon value estimation?
A: To answer this question, we evaluate QC (Li et al. [30]) and OfflineDV2 (Lu et al. [34]) on puzzle-4x4-task5 with different value estimation horizons. Specifically, we vary the action chunk size in QC and the model rollout length in OfflineDV2 to control the horizon of its value estimation. While MAC-v2 benefits from extending the value estimation horizon, neither baseline shows comparable improvements as its horizon increases (Figure 2). These results suggest that simply extending the value estimation horizon is insufficient for the baselines, highlighting the importance of MAC-v2’s design choices in facilitating long-horizon model-based value expansion.

Figure 2. Prior works do not scale with longer horizons.
Q: Is each MAC-v2’s objective essential for long-horizon model rollouts?
A: We find that the both BC policy objective and WM objective plays an important role in retaining control-relevant and predictable information. Specifically, we remove the gradient from the BC objective and the world modeling objective to the encoder while keeping all other components unchanged on cube-double-task3. Figure 3 shows that removing the actor gradient or the world modeling gradient substantially degrades performance, suggesting that both dynamics prediction and policy objective alone are insufficient to learn a representation suitable for long-horizon latent rollouts. This highlights that MAC-v2’s design choice to incorporate policy objective and world modeling objective for representation learning is crucial for reliable long-horizon latent value expansion.
Figure 3. Objectives are essential.
Q: How about using other representation objectives instead?
A: Here, we replace the encoder learning objective of MAC-v2 with other representation objectives and examine whether the learned representation can support long-horizon value expansion. We consider pixel reconstruction (Wahlström et al. [55]), SimCLR (Chen et al. [3]), and LeWorldModel (Maes et al. [36]) and compare those methods in cube-double-task3. As shown in Figure 4, prior representation learning methods substantially degrades the performance. This highlights that MAC-v2’s objective is crucial for reliable long-horizon latent value expansion, and cannot be replaced with conventional representation learning objectives.
Figure 4. Other losses fall short.
Closing remarks
In this work, we introduced MAC-v2, an offline model-based RL framework for long-horizon model-based value expansion from visual observations. MAC-v2 combines reconstruction-free latent dynamics with action chunking, enabling long-horizon imagination with substantially fewer recursive world model predictions. Across 35 standard visual offline RL tasks and 13 robotic manipulation tasks, MAC-v2 achieves strong performance against both model-free and model-based baselines, with substantial gains on challenging OGBench and Robomimic tasks while remaining competitive with strong behavior-cloning baselines on LIBERO-10.
Several open questions remain for MAC-v2, suggesting directions for future work. First, while having larger action chunks reduces the number of recursive model predictions, but excessively long chunks reduce the policy’s reactivity. Developing adaptive or hierarchical action chunking could provide a better trade-off between long-horizon prediction and closed-loop control. Moreover, our experiments are primarily limited to simulated, single-task environments with relatively limited visual variation, leaving a substantial gap to realistic multi-task settings. Extending MAC-v2 to large-scale vision-language-action models (VLAs) or world action models (WAMs) could be a promising direction toward long-horizon model-based value expansion in more diverse and realistic domains.
AI use statement
In this work, we used generative AI tools for polishing the writing, and implementing the methods. We have not used generative AI tools for designing or providing feedback on research methodology or experiments, proposing or refining hypotheses, assisting with translation. Supporting qualitative and thematic data analysis and interpreting the results, and generating synthetic data sets, helping develop theoretical models or conceptual frameworks, formulating mathematical claims, providing critical ingredients for proving mathematical claims, assisting in the writing of proofs, cleaning and reformatting dataset, interpreting results, are not applicable to this work. The authors have manually reviewed the paper to prevent plagiarism, and read the LLM-generated code and verified and tested its correctness. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.
Reproducibility statement
We include implementation details needed for reproduction, such as hyperparameters (including the baselines), datasets, and evaluation protocols in Section A. We provide the full results of each tasks in Section B for better future reproducibility.
Acknowledgments
Kwanyoung Park is partly supported by the Korea Foundation for Advanced Studies (KFAS). This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley. This research was partly supported by AFOSR under FA9550-22-1-0273 and DARPA TIAMAT.
Experimental Details
We implement MAC-v2 on top of the codebase of Park et al. [43]. Each experiment takes approximately 9 hours for OGBench and MMBench2 benchmarks, 12 hours for Robomimic and 14 hours for LIBERO-10 benchmark on a single A5000 GPU.
Implementation Details
Network architecture for MAC-v2. For OGBench experiments, we use the impala_small encoder from Espeholt et al. [5], following the implementation of FQL (Park et al., [44]) to match the baselines. For LIBERO-10 experiments, we use the Resnet-18 backbone (trained from scratch) as the image is more complicated and higher resolution (128px). For other experiments, we use the convolutional encoder architecture (12M version) from DreamerV3 (Hafner et al., [12]). On top of the encoder, we use 4-layer MLP with hidden size 256 for actor, critic, and dynamics head. For critic and reward head, we use the two-hot encoding (Hafner et al., [12]).
Data augmentation. We apply random crop augmentation of size 4 for OGBench and 12 for LIBERO-10, following the implementation of Park et al. [44]. For model-free approaches, we use the same augmentation scheme as they follow the implementation of Park et al. [44]. For model-based RL baselines, we follow the setup of original papers.
Accelerating rejection sampling during training. Following Park et al. [41], we use different numbers of samples for rejection sampling during training and evaluation (which we call and ) to reduce the training time. Specifically, we use during training and at test time.
Implementation details for the compared methods. For the model-free baselines, we follow the implementation by Park et al. [42] and Li et al. [30]. For LOMPO (Rafailov et al., [46]) and OfflineDV2 (Lu et al., [34]), we use the official codebase released by the authors.
MAC-v2 with terminals. We predict the terminal signal from the terminal model , and calculate the discounted reward target . We use a hard threshold 0.5 and calculate . We train the value function from latent on-policy model rollout (i.e., imaginary trajectories), generated as
We calculate the value target as:
Finally, we train the action-chunked Q function for the rejection sampling with the following loss:
Hyperparameters
We group the configurations into three families: model-free baselines (Table 4), model-based baselines (Table 5), and MAC-v2 (Table 6). Image resolution, encoder and augmentation are set per benchmark, and are given in Table 3. Evaluation uses 15 episodes for each seed. We match the encoder for MAC-v2 and model-free methods, but we use the original codebase’s architecture for model-based baselines as their architecture (history-conditioned RSSM) significantly differs from our method (markovian CNN encoder).
| Benchmark | Resolution | Encoder | Aug. pad |
| OGBench | impala_small | 4 | |
| MMBench2 | 12M Conv. | 4 | |
| Robomimic | 12M Conv. | 4 | |
| LIBERO-10 | ResNet-18 | 12 |
Table 3. Per-benchmark input and encoder settings. These apply to MAC-v2 and the model-free baselines, so every comparison uses the same encoder within a benchmark. Please refer to Section A.1 for implementation details of the encoder.
| Hyperparameters | Value |
| Gradient steps | 500K |
| Learning rate | |
| Optimizer | Adam ([20]) |
| Batch size | 256 |
| MLP size | |
| Frame stack | 3 |
| Image augmentation | DrQ random shift, (tied) |
| Discount factor | 0.99 |
Table 4. Hyperparameters of the model-free baselines.
| Hyperparameters | Value |
| Gradient steps | 100K |
| Learning rate | |
| Optimizer | Adam ([20]) |
| Batch size | 256 |
| MLP size | |
| Image resolution | |
| Frame stack | 1 |
Table 5. Shared hyperparameters of model-based baselines.
| Hyperparameters | Value |
| Learning rate | |
| Optimizer | Adam |
| Layer normalization | True |
| Target network update rate | 0.02 |
| Discount factor | 0.999 |
| -return coefficient | 1.0 |
| Flow steps | 5 |
| 8 | |
| 32 | |
| Rollout length | 10 |
| Action chunk size | 10 |
| Latent dimension | 512 |
| Image augmentation | DrQ random shift (tied) |
| Value / reward loss | symexp two-hot, 255 bins on |
| Weight decay | |
| Gradient clipping | 1.0 |
| Frame stack | 1 |
Table 6. Hyperparameters of MAC-v2.
Shared hyperparameters for model-free baseline. IQL, ReBRAC, IFQL, FQL and QC are run from the reference FQL codebase ([44]) at its visual-task defaults. QC additionally uses best-of-32 sampling over 5-step action chunks, so it is chunked where the other four are single-step. We use following the original codebase.
Shared hyperparameters for model-based baseline. Both are run at their authors’ defaults, changing only the uncertainty coefficient . Both methods train their world model for 25K steps before the policy phase. Both methods are trained for 100k steps, as we found the performance starts to degrade after 100k steps. We note that the computational budget for 100k steps for these methods is larger than MAC-v2, taking around 16 hours in a single A5000 GPU.
MAC-v2 hyperparameters. We report the hyperparameters for our method in Table 6. MAC-v2 uses the same across all tasks and all four benchmarks, no per-task tuning is performed. We apply DrQ-style random-shift augmentation ((Kostrikov et al., 2021), using the same shift for an observation and the next observation. We normalize the value with the maximum return of the dataset, and train the value and reward heads with symexp two-hot categorical loss of Hafner et al. [12].
Hyperparameters for baselines. For LOMPO and Offline DV2, we sweep the uncertainty penalty coefficient . For FQL we search over the behavior-cloning coefficient , and for ReBRAC over . For FQL, IQL, ReBRAC, we use the reported hyperparameter from Park et al. [44] in OGBench. For QC and QC-IFQL, we search over the action chunking size . IFQL is run at a single fixed configuration. The hyperparameters are reported in Table 7, Table 8, Table 9, Table 10 for OGBench, MMBench2, Robomimic, LIBERO-10, respectively. We denote “N/A” in the tables if a method achieves zero performance across all hyperparameters tested in our sweep. If not specified, all other hyperparameters follow the defaults provided in the original papers.
| Environment | LOMPO () | OffDV2 () | FQL () | ReBRAC () | IQL () |
| visual-cube-single-play-v0 | 5 | 5 | 300 | (1.0, 0) | 1 |
| visual-cube-double-play-v0 | 5 | 10 | 100 | (0.1, 0) | 0.3 |
| visual-scene-play-v0 | 3 | 10 | 100 | (0.1, 0.01) | 10 |
| visual-puzzle-3x3-play-v0 | 3 | 10 | 300 | (0.3, 0.01) | 10 |
| visual-puzzle-4x4-play-v0 | 5 | 3 | 300 | (0.3, 0.01) | 3 |
Table 7. Baseline hyperparameters on the OGBench visual tasks. Uncertainty penalty for the model-based baselines, behavior-cloning coefficient for FQL, and for ReBRAC.
| Environment | LOMPO () | OffDV2 () | FQL () | ReBRAC () | IQL () |
| cup-catch-var1 | 1 | 10 | 0.3 | (0.01, 0) | 1 |
| finger-turn-easy-var1 | 30 | 1 | 0.3 | (0.01, 0.01) | 1 |
| ms-push-banana | 30 | 30 | 1.0 | (1.0, 0) | 1 |
| og-point-var1 | N/A | 10 | 0.3 | (0.01, 0.01) | 1 |
| og-point-var2 | 1 | 10 | 0.3 | (0.01, 0.01) | 1 |
| pygame-point-maze-var4 | 3 | 1 | 0.1 | (0.01, 0) | 1 |
| pygame-reacher-easy | 30 | 1 | 1.0 | (0.1, 0.01) | 1 |
| pygame-dungeon-explorer1 | 10 | 3 | 0.3 | (0.1, 0) | 1 |
| pygame-foraging | 30 | 10 | 0.1 | (1.0, 0) | 1 |
| pygame-whirlpool | 3 | 10 | 0.1 | (0.1, 0) | 1 |
Table 8. Baseline hyperparameters on MMBench2. Uncertainty penalty for the model-based baselines, behavior-cloning coefficient for FQL, and for ReBRAC, for IQL.
| Task | LOMPO () | OffDV2 () | BC () | QC () | QC-IFQL () |
| lift | 3 | 10 | 10 | 5 | 5 |
| can | N/A | N/A | 10 | 5 | 10 |
| square | N/A | N/A | 10 | 5 | 5 |
Table 9. Baseline hyperparameter on Robomimic. Uncertainty penalty for the model-based baselines, action chunk size for BC, QC and QC-IFQL.
| Task | LOMPO () | OffDV2 () |
| put the white mug on the left plate and put the yellow and white mug on the right plate | N/A | N/A |
| put the white mug on the plate and put the chocolate pudding to the right of the plate | N/A | N/A |
| put the yellow and white mug in the microwave and close it | 3 | N/A |
| turn on the stove and put the moka pot on it | N/A | N/A |
| put both the alphabet soup and the cream cheese box in the basket | N/A | N/A |
| put both the alphabet soup and the tomato sauce in the basket | 10 | N/A |
| put both moka pots on the stove | N/A | N/A |
| put both the cream cheese box and the butter in the basket | N/A | N/A |
| put the black bowl in the bottom drawer of the cabinet and close it | N/A | N/A |
| pick up the book and place it in the back compartment of the caddy | N/A | N/A |
Table 10. Per-task uncertainty coefficient of the model-based baselines on LIBERO-10. We sweep over on each tasks for model-based baselines. N/A marks tasks on which every scored zero.
Complete Numerical Results
For completeness, we provide the full per-task results for OGBench tasks in Table 11 (corresponding to Table 1). The results are averaged over 4 seeds and we report the standard deviations for each tasks. We highlight the numbers with the best performance.
| Environment | IQL | ReBRAC | IFQL | FQL | QC | LOMPO | OffDV2 | MAC-v2 |
| visual-cube-single-task1 | ||||||||
| visual-cube-single-task2 | ||||||||
| visual-cube-single-task3 | ||||||||
| visual-cube-single-task4 | ||||||||
| visual-cube-single-task5 | ||||||||
| visual-cube-double-task1 | ||||||||
| visual-cube-double-task2 | ||||||||
| visual-cube-double-task3 | ||||||||
| visual-cube-double-task4 | ||||||||
| visual-cube-double-task5 | ||||||||
| visual-scene-task1 | ||||||||
| visual-scene-task2 | ||||||||
| visual-scene-task3 | ||||||||
| visual-scene-task4 | ||||||||
| visual-scene-task5 | ||||||||
| visual-puzzle-3x3-task1 | ||||||||
| visual-puzzle-3x3-task2 | ||||||||
| visual-puzzle-3x3-task3 | ||||||||
| visual-puzzle-3x3-task4 | ||||||||
| visual-puzzle-3x3-task5 | ||||||||
| visual-puzzle-4x4-task1 | ||||||||
| visual-puzzle-4x4-task2 | ||||||||
| visual-puzzle-4x4-task3 | ||||||||
| visual-puzzle-4x4-task4 | ||||||||
| visual-puzzle-4x4-task5 |
Table 11. Complete results for OGBench visual benchmark tasks.
| Environment | IQL | ReBRAC | IFQL | FQL | QC | LOMPO | OffDV2 | MAC-v2 |
| cup-catch-var1 | ||||||||
| finger-turn-easy-var1 | ||||||||
| ms-push-banana | ||||||||
| og-point-var1 | ||||||||
| og-point-var2 | ||||||||
| pygame-point-maze-var4 | ||||||||
| pygame-reacher-easy | ||||||||
| pygame-dungeon-explorer1 | ||||||||
| pygame-foraging | ||||||||
| pygame-whirlpool |
Table 12. Complete results for MMBench2 tasks.
| Environment | Model-Free | Model-Based | ||||
| BC | QC | QC-IFQL | LOMPO | OffDV2 | MAC-v2 | |
| LIBERO (10 tasks) | ||||||
| turn on the stove and put the moka pot on it | ||||||
| put the black bowl in the bottom drawer of the cabinet and close it | ||||||
| put the yellow and white mug in the microwave and close it | ||||||
| put both moka pots on the stove | ||||||
| put both the alphabet soup and the cream cheese box in the basket | ||||||
| put both the alphabet soup and the tomato sauce in the basket | ||||||
| put both the cream cheese box and the butter in the basket | ||||||
| put the white mug on the left plate and put the yellow and white mug on the right plate | ||||||
| put the white mug on the plate and put the chocolate pudding to the right of the plate | ||||||
| pick up the book and place it in the back compartment of the caddy |
Table 13. Complete results for LIBERO tasks.
Training Curves
We provide the training curve of MAC-v2 for the OGBench, MMBench, robomimic, LIBERO-10 result of Table 1 and Table 2, in Figure 5 to Figure 8. We plot the mean and the standard deviation (across 4 seeds) by covering [mean - std, mean + std] area with a lighter color.
Figure 5. Training curve of MAC-v2 in OGBench environments. We report the success rate for 15 evaluation episodes across 4 seeds (total 60 episodes). Shaded region represents the [mean - std, mean + std].
Figure 8. Training curve of MAC-v2 in LIBERO-10 environments. We report the success rate for 15 evaluation episodes across 4 seeds (total 60 episodes). Shaded region represents the [mean - std, mean + std]. For the task instruction of each tasks for this graph, please refer to Table 2.
Figure 6. Training curve of MAC-v2 in MMBench2 environments. We report the normalized return for 15 evaluation episodes across 4 seeds (total 60 episodes). Shaded region represents the [mean - std, mean + std].
Figure 7. Training curve of MAC-v2 in Robomimic environments. We report the success rate for 15 evaluation episodes across 4 seeds (total 60 episodes). Shaded region represents the [mean - std, mean + std].
References
- [1]Michael S. Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2209.15571
- [2]Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2209.14548
- [3]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020.arxiv.org/abs/2002.05709
- [4]Hojun Chung, Junseo Lee, and Songhwai Oh. Offline reinforcement learning with universal horizon models. arXiv preprint arXiv:2605.15603, 2026.arxiv.org/abs/2605.15603
- [5]Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018.arxiv.org/abs/1802.01561
- [6]Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value expansion for efficient model-free reinforcement learning. In International Conference on Machine Learning (ICML), 2018.DOI
- [7]Minghao Fu, Fan Feng, Nicklas Hansen, and Biwei Huang. Back to parsimonious latents: Learning task-centric world models from visual foundations. arXiv preprint arXiv:2605.25620, 2026.arxiv.org/abs/2605.25620
- [8]Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2021.
- [9]Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2301.02328
- [10]Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020a.arxiv.org/abs/2010.02193
- [11]Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), 2020b.arxiv.org/abs/1912.01603
- [12]Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640:647–653, 2025a.DOI
- [13]Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models. arXiv preprint arXiv:2509.24527, 2025b.arxiv.org/abs/2509.24527
- [14]Nicklas Hansen and Xiaolong Wang. Hallucination in world models is predictable and preventable, 2026.arxiv.org/abs/2606.27326
- [15]Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In International Conference on Machine Learning (ICML), 2022.arxiv.org/abs/2203.04955
- [16]Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.16828
- [17]Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. ArXiv, abs/2304.10573, 2023.
- [18]Riashat Islam, Manan Tomar, Alex Lamb, Yonathan Efroni, Hongyu Zhang, Aniket Didolkar, Dipendra Misra, Xin Li, Harm Van Seijen, Remi Tachet des Combes, et al. Agent-controller representations: Principled offline rl with rich exogenous information. arXiv preprint arXiv:2211.00164, 2022.arxiv.org/abs/2211.00164
- [19]Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel : Model-based offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.
- [20]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.arxiv.org/abs/1412.6980
- [21]Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations (ICLR), 2021.
- [22]Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2110.06169
- [23]Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2006.04779
- [24]Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-art, pp. 45–73. Springer, 2012.
- [25]Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pp. 5639–5650. PMLR, 2020.arxiv.org/abs/2004.04136
- [26]Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.09754
- [27]John M Lee. Introduction to Smooth Manifolds. Springer, 2012.
- [28]Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv, abs/2005.01643, 2020.arxiv.org/abs/2005.01643
- [30]Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. arXiv preprint arXiv:2507.07969, 2025b.arxiv.org/abs/2507.07969
- [31]Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.02747
- [32]Bo Liu, Yifeng Zhu, Ch reen?ongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023a.arxiv.org/abs/2306.03310
- [33]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations (ICLR), 2023b.arxiv.org/abs/2209.03003
- [34]Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. Transactions on Machine Learning Research (TMLR), 2023.arxiv.org/abs/2206.04779
- [35]Hao Luo, Wanpeng Zhang, Yicheng Feng, Sipeng Zheng, Haiwen Xu, Chaoyi Xu, Ziheng Xi, Yuhui Fu, and Zongqing Lu. Being-h0-7: A latent world-action model from egocentric videos. arXiv preprint arXiv:2605.00078, 2026.arxiv.org/abs/2605.00078
- [36]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. Leworld-model: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026.arxiv.org/abs/2603.19312
- [37]Ajay Mandlekar, Danfei Xu, J. Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.arxiv.org/abs/2108.03298
- [38]Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.arxiv.org/abs/1312.5602
- [39]Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. ArXiv, abs/2006.09359, 2020.arxiv.org/abs/2006.09359
- [41]Kwangyoung Park, Seohong Park, Youngwoon Lee, and Sergey Levine. Scalable offline model-based rl with action chunks. International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2512.08108
- [42]Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Obgench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), 2025a.arxiv.org/abs/2410.20092
- [43]Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. arXiv, abs/2506.04168, 2025b.arxiv.org/abs/2506.04168
- [44]Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In International Conference on Machine Learning (ICML), 2025c.arxiv.org/abs/2502.02538
- [45]Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. ArXiv, abs/1910.00177, 2019.
- [46]Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for dynamics and control, pp. 1154–1168. PMLR, 2021.arxiv.org/abs/2012.11547
- [48]Julian Schrittwieser, Ioannis Antonoglou, T. Hubert, K. Simonyan, L. Sifre, Simon Schmitt, A. Guez, Edward Lockhart, Demis Hassabis, T. Graepel, T. Lillicrap, and D. Silver. Mastering Atari, Go, Chess and Shogi by planning with a learned model. Nature, 588 7839:604–609, 2020.
- [49]Anya Sims, Cong Lu, Jakob N Foerster, and Yee W Teh. The edge-of-reach problem in offline model-based reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2402.12527
- [50]Gwanwoo Song, Kwanyoung Park, and Youngwoon Lee. Chunk-guided q-learning. arXiv preprint arXiv:2603.13971, 2026.arxiv.org/abs/2603.13971
- [51]Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International conference on machine learning, pp. 9870–9879. PMLR, 2021.arxiv.org/abs/2009.08319
- [52]Ruixiang Sun, Hongyu Zhang, Xin Li, and Riashat Islam. Learning latent dynamic robust representations for world models. arXiv preprint arXiv:2405.06263, 2024.arxiv.org/abs/2405.06263
- [53]Yihao Sun, Jiaj Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman inconsistency for model-based offline reinforcement learning. In International Conference on Machine Learning, pp. 33177–33194. PMLR, 2023.
- [54]Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2305.09836
- [55]Niklas Wahlström, Thomas B Schön, and Marc Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXiv:1502.02251, 2015.arxiv.org/abs/1502.02251
- [56]Shenghua Wang, Ziyuan Chen, Le Gan, Shuai Feng, and De-Chuan Zhan. Semopo: learning high-quality model and policy from low-quality offline visual datasets. In Proceedings of the 41st International Conference on Machine Learning, pp. 49867–49889, 2024.
- [57]Yifan Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. ArXiv, abs/1911.11361, 2019.arxiv.org/abs/1911.11361
- [58]Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Chan, and Xianyuan Zhang. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2303.15810
- [59]Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2005.13239
- [60]Tianhe Yu, Ayiral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2102.08363
- [61]Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems (RSS), 2023.arxiv.org/abs/2304.13705





