betaFeedback

WorksComputer Science

MAC-v2: Long-horizon model-based value expansion from pixels

Abstract

We study whether long-horizon model-based value expansion can serve as an effective framework in pixel-based offline reinforcement learning (RL). Long-horizon model-based value expansion from pixels is substantially harder than state-based environments, as the agent must learn both the representation and its dynamics, and errors from either can accumulate recursively over imagined trajectories. We introduce MAC-v2, which enables long-horizon model-based expansion from pixels by training a reconstruction-free latent world model with action chunking. By learning representations from prediction and control objectives, MAC-v2 does not need to encode control-irrelevant visual details that can hinder the dynamics prediction. Action chunking further mitigates the prediction error by reducing the number of recursive model predictions required for value expansion. Across diverse visual benchmarks including visual OGBench, MMBench2, Robomimic, and LIBERO-10, MAC-v2 achieves strong performance against model-free and model-based offline RL baselines. Our analyses show that long-horizon model-based value expansion is critical for the performance of MAC-v2, and that both action chunking and latent learning objectives are essential for reliable long-horizon latent imagination.

Introduction

Many real-world decision-making problems, such as robotic manipulation, require agents to learn directly from high-dimensional visual observations. This is particularly challenging in offline reinforcement learning (RL), where policies must be learned entirely from a fixed dataset without additional environment interaction. Unlike online RL, offline RL cannot evaluate the policy through environment interaction and rely on off-policy value estimates learned from the fixed dataset. As the learned policy deviates from the behavior policy, these off-policy estimates can become inaccurate due to distribution shift. Model-based value expansion ([6]) offers an appealing alternative by learning a world model, where on-policy trajectories can be generated by using a world model without accessing the real environment ([59, 19, 53, 13] (Park & Lee 2025)).

Prior works have shown that model-based value expansion can be applied to offline RL from pixels, but typically only over short imagination horizons. For example, methods such as LOMPO ([46]), OfflineDV2 ([34]), and SeMOPO ([56]) learn latent world models from visual observations and mitigate model exploitation through uncertainty estimation, conservative penalties, or representation decomposition. While these approaches demonstrate that model-based value expansion can be effective, their model rollouts are generally kept short (e.g., LOMPO, OfflineDV2, and SeMOPO use an imagination horizon of 5 steps). In contrast, recent state-based offline RL methods have shown that substantially longer model-based value expansion can be effective for long-horizon tasks ([41, 4]).

Long-horizon model-based value expansion from pixels is substantially harder, as the agent must learn both the representation and its dynamics from the fixed offline data. Since pixel observations contain substantial control-irrelevant information ([7, 35]), training a world model directly on pixels becomes unnecessarily challenging. This motivates learning a compact latent representation for world modeling. Here, the key challenge is to learn a representation that preserves the information needed for control, and also be predictable by the learned dynamics [18, 52]. Otherwise, this can lead to inaccurate value estimation, as errors from either the representation or the dynamics can accumulate recursively over imagined trajectories, and this issue is further exacerbated in the offline setting, where the agent cannot collect corrective data when those trajectories reach poorly modeled regions.

We introduce MAC-v2, which enables long-horizon model-based value expansion from pixels by training a reconstruction-free latent world model with action chunking. By learning representations from prediction and control objectives, MAC-v2 does not need to encode control-irrelevant visual details that can hinder the dynamics prediction, while also preserving the information needed for control. Moreover, we utilize action chunking [61] for the policy and the world model, reducing the number of recursive model predictions required over long horizons [41]. Together, these design choices mitigate the compounding error during imagined rollouts and enable long-horizon model-based value expansion from visual observations.

We evaluate MAC-v2 across a broad suite of visual benchmarks and show that it matches or substantially outperforms prior model-free and model-based RL methods that rely on shorter-horizon value estimates. MAC-v2 substantially outperforms prior model-free and model-based baselines on 25 visual OGBench [42], and matches the performance of the best prior methods in 10 MMBench2 [14] tasks. We further evaluate MAC-v2 on robotic manipulation benchmarks, including Robomimic [37] and LIBERO-10 [32], where the offline datasets consist primarily of expert demonstrations and behavior cloning policy serves as a strong baseline. On Robomimic, MAC-v2 is the only RL method that matches or exceeds behavior cloning across all evaluated tasks, while remaining competitive with strong behavioral baselines on LIBERO-10. Our analyses show that long-horizon model-based value expansion is critical for this performance, and learning a reconstruction-free encoder with both world model and control objectives is important for reliable long-horizon latent rollouts. Overall, our results suggest that, with the design choices of MAC-v2, latent world models can support effective long-horizon model-based value expansion from pixels.

Related Works

Offline model-free RL. Offline RL aims to learn policies solely from pre-collected datasets without further environment interaction [24, 28]. A central challenge is distribution shift, as learned policies may select out-of-distribution states or actions for which value estimates are unreliable. Prior work addresses this issue through conservative objectives [23], behavior regularization [57, 45, 39, 8, 54, 44], in-sample value maximization [22, 58, 9], and sample-selection methods [2, 17]. More recently, action chunking has shown strong performance on long-horizon offline RL tasks [30, 50] by improving temporal consistency and reducing the effective decision horizon. However, these model-free approaches rely on short-horizon off-policy targets for value learning, which can limit performance due to inaccurate value propagation [43]. In contrast, MAC-v2 learns values from long-horizon on-policy targets generated by a model through action-chunked latent rollouts, reducing the error from the recursive bootstrapping.

Offline model-based RL. MAC-v2 is a model-based offline RL method. Such methods learn a dynamics model from the fixed dataset and use it to generate synthetic trajectories for policy evaluation and optimization. By evaluating the policy through the learned model, they can utilize on-policy value estimation instead of off-policy estimates. However, policies may exploit inaccuracies in the learned dynamics model, leading to unreliable value estimates. Prior work addresses this issue through conservative objectives [19, 60], uncertainty-aware penalties [59, 53, 49] (R|gter et al. 2022; Park & Lee 2025), and in-distribution policy optimization [41]. In particular, MAC [41] combines action-chunked dynamics with a rejection sampling policy, enabling long-horizon model-based value expansion on state-based tasks. MAC-v2 builds on this framework, but addresses the additional challenge of learning a representation that can support accurate recursive model predictions directly from visual observations.

Online RL from pixels. Learning from pixels requires representations that retain information relevant for control while remaining compact and predictable. Prior work has studied this problem extensively in online RL, including model-free approaches [38, 21, 26] and model-based approaches [11, 12, 15, 16]. Representation objectives include image reconstruction [11, 12 b], contrastive learning [25, 51], and reconstruction-free latent prediction [48, 16, 36]. MAC-v2 adopts the reconstruction-free latent prediction paradigm established by the prior works, and we do not propose a entirely new representation learning objective. Instead, we study how to utilize such representations for long-horizon model rollouts, required by offline model-based value expansion. We show that, when combined with action chunking, predictive latent representations can support long-horizon imagination and value estimation from a fixed visual dataset.

Offline model-based RL from pixels. Several prior works have specifically studied model-based offline RL from visual observations using reconstruction-based latent representations. LOMPO [46] learns a latent dynamics model and calculates model uncertainty in latent space, while OfflineDV2 [34] adapts DreamerV2 [10] to offline visual control with uncertainty-aware penalties. However, model imagination is typically kept relatively short (e.g., LOMPO and OfflineDV2 use imagination of 5 steps). We instead study whether predictive latent representations can support long-horizon model-based value expansion from pixels. By combining latent prediction with action chunking, MAC-v2 enables stable value expansion over more than 100 environment steps.

Preliminaries

Problem setting. We consider a block Markov decision process (MDP) defined as M=(S,O,A,r,μ,p,q)\mathcal{M} = (\mathcal{S}, \mathcal{O}, \mathcal{A}, r, \mu, p, q), where S\mathcal{S} is the latent environment state space, O\mathcal{O} is the observation space, and A=Rd\mathcal{A} = \mathbb{R}^{d} is the action space. The reward function is r(s,a):S×A→Rr(s,a) : \mathcal{S} \times\mathcal{A} \to\mathbb{R}, μ(s)∈Δ(S)\mu(s) \in\Delta(\mathcal{S}) is the initial state distribution, p(s′∣s,a):S×A→Δ(S)p(s' \mid s,a) : \mathcal{S} \times\mathcal{A} \to\Delta(\mathcal{S}) is the transition dynamics, and q(o∣s):S→Δ(O)q(o \mid s) : \mathcal{S} \to\Delta(\mathcal{O}) is the observation model. The agent does not observe sts_t directly and instead receives ot∼q(⋅∣st)o_t \sim q(\cdot\mid s_t). We assume that the current observation is sufficient to identify the underlying state for decision making, so that reactive policies π(at∣ot)\pi(a_t \mid o_t) are sufficient. For a policy π\pi, we define Vπ(s)=Eτ∼π[∑t=0∞γtr(st,at)∣s0=s]V^\pi(s) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t) \mid s_0=s\right], where γ∈(0,1)\gamma\in(0,1) is the discount factor. The goal of offline RL is to learn a policy maximizing Es0∼μ[Vπ(s0)]\mathbb{E}_{s_0 \sim\mu}[V^\pi(s_0)] using only a fixed dataset D={τ(i)}\mathcal{D} = \{\tau^{(i)}\} of previously collected trajectories, without additional environment interaction.

Flow Matching Policy. A flow matching policy [31, 1, 33] represents a noise-conditioned deterministic policy π=πθ(z,ϵ)\pi= \pi_\theta(z,\epsilon) by transporting samples from a simple reference distribution to the action distribution conditioned on a state. Let vθ:[0,1]×Z×A→Av_\theta: [0,1] \times\mathcal{Z} \times\mathcal{A} \to\mathcal{A} denote a state-conditioned, time-dependent velocity field, where u∈[0,1]u \in[0,1] denotes the flow time. Given an initial noise sample ϵ∼N(0,Id)\epsilon\sim\mathcal{N}(0,I_d), the corresponding flow ψθ(u;z,ϵ)\psi_\theta(u;z,\epsilon) is defined by the ordinary differential equation (ODE) [27]

dduψθ(u;z,ϵ)=vθ(u,z,ψθ(u;z,ϵ)),ψθ(0;z,ϵ)=ϵ.(1)\frac{\mathrm{d}}{\mathrm{d}u}\psi_\theta(u;z,\epsilon) = v_\theta(u,z,\psi_\theta(u;z,\epsilon)), \qquad\psi_\theta(0;z,\epsilon) = \epsilon. \tag*{(1)}

The policy is defined as the endpoint of this flow,

πθ(z,ϵ)=ψθ(1;z,ϵ),(2)\pi_\theta(z,\epsilon) = \psi_\theta(1;z,\epsilon), \tag*{(2)}

which induces a conditional action distribution πθ(a∣z)\pi_\theta(a \mid z) through ϵ∼N(0,Id)\epsilon\sim\mathcal{N}(0,I_d).

Given state-action samples (s,a)∼D(s,a) \sim\mathcal{D}, flow matching trains the velocity field along linear interpolation paths between Gaussian noise ϵ\epsilon and the target action aa. Specifically, for

au=(1−u)ϵ+ua,(3)a_u = (1-u)\epsilon+ ua, \tag*{(3)}

the velocity field is trained by minimizing

Lflow(θ)=E(s,a)∼D, ϵ∼N(0,Id), u∼Unif⁡([0,1])[∥vθ(u,s,au)−(a−ϵ)∥22].(4)\mathcal{L}^{\mathrm{flow}}(\theta) = \mathbb{E}_{(s,a)\sim\mathcal{D},\,\epsilon\sim\mathcal{N}(0,I_d),\,u\sim\operatorname{Unif}([0,1])}\left[\left\|v_\theta(u,s,a_u)-(a-\epsilon)\right\|_2^2\right]. \tag*{(4)}

After training, actions are sampled by drawing ϵ∼N(0,Id)\epsilon\sim\mathcal{N}(0,I_d) and numerically integrating the learned velocity field from u=0u=0 to u=1u=1.

MAC-v2

We present MAC-v2, an offline model-based RL framework that enables long-horizon model-based value expansion from visual observations. MAC-v2 builds on two lines of work: model-based value expansion with action chunking from MAC [41], and latent world models trained with prediction and control objectives akin to reconstruction-free online model-based RL literatures [48, 15, 16]. Neither line of work alone enables long-horizon model-based value expansion in pixel-based offline RL, as MAC-v2 assumes low-dimensional state observations, while prior reconstruction-free latent world models have been limited to short-horizon rollouts.

Our key design choice is to learn a reconstruction-free latent dynamics with action chunking. With action chunking size of nn, an imagination horizon of HnHn environment steps requires only HH recursive model predictions, reducing the compounding of prediction errors. Moreover, we learn the representation with world modeling and behavior cloning objective, encouraging the representation to preserve the information for reward prediction and control, while being predictable under the learned dynamics model. We describe our world model objectives in Section 4.1, and the long-horizon model-based value expansion with the learned latent space in Section 4.2.

Learning Representations for Long-Horizon Imagination

We encode each image observation into a latent state using a deterministic encoder fξ:O→Zf_{\xi} : \mathcal{O} \to\mathcal{Z}. Rather than reconstructing the input image, MAC-v2 learns the encoder through objectives relevant to latent prediction and control. In particular, gradients from the dynamics, reward, and behavior cloning objectives are propagated into the encoder, learning a compact representation that retains information for predicting future latent states, rewards, and control.

Concretely, given a latent state and an action chunk, we learn a deterministic dynamics model pψ:Z×An→Zp_{\psi} : \mathcal{Z} \times\mathcal{A}^{n} \to\mathcal{Z} to predict the latent state after nn environment steps, by minimizing

Ldyn(ψ,ξ)=E(ot,at,⋯ ,ot+n)∼D[∥pψ(fξ(ot),at)−sg⁡(fξ(ot+n))∥22],(5)\mathcal{L}^{\mathrm{dyn}}(\psi,\xi)=\mathbb{E}_{(o_t,a_t,\cdots,o_{t+n})\sim\mathcal{D}}\left[\left\|p_{\psi}(f_{\xi}(o_t),a_t)-\operatorname{sg}(f_{\xi}(o_{t+n}))\right\|_2^2\right], \tag*{(5)}

where sg⁡(⋅)\operatorname{sg}(\cdot) denotes the stop-gradient operator.

We also learn a distributional action-chunked reward model rψ:Z×An→Δ(R)r_{\psi} : \mathcal{Z} \times\mathcal{A}^{n} \to\Delta(\mathbb{R}) with two-hot encoding [12]. We regress the target return rt=∑i=0n−1γirt+ir_t=\sum_{i=0}^{n-1}\gamma^i r_{t+i} by minimizing

Lrew(ψ,ξ)=E(ot,at)∼D[−log⁡rψ(rt∣fξ(ot),at)].(6)\mathcal{L}^{\mathrm{rew}}(\psi,\xi)=\mathbb{E}_{(o_t,a_t)\sim\mathcal{D}}\left[-\log r_{\psi}(r_t\mid f_{\xi}(o_t),a_t)\right]. \tag*{(6)}

We parameterize the behavioral policy as a conditional flow model, and train a state-conditioned velocity field vθ:R×Z×An→Anv_{\theta} : \mathbb{R}\times\mathcal{Z}\times\mathcal{A}^{n}\to\mathcal{A}^{n} using the flow-matching objective

Lflow(θ,ξ)=Eϵ∼N(0,Ind), (ot,at)∼D,u∼Unif⁡([0,1]), aϵ=(1−u)ϵ+uat[∥vθ(u,fξ(ot),aϵ)−(at−ϵ)∥22].(7)\mathcal{L}^{\mathrm{flow}}(\theta,\xi)=\mathbb{E}_{\substack{\epsilon\sim\mathcal{N}(0,I_{nd}),\ (o_t,a_t)\sim\mathcal{D},\\u\sim\operatorname{Unif}([0,1]),\ a_{\epsilon}=(1-u)\epsilon+ua_t}}\left[\left\|v_{\theta}(u,f_{\xi}(o_t),a_{\epsilon})-(a_t-\epsilon)\right\|_2^2\right]. \tag*{(7)}

We denote by πθ(zt,ϵ)\pi_{\theta}(z_t,\epsilon) the endpoint at u=1u=1 obtained by integrating the learned velocity field from the initial noise ϵ∼N(0,Ind)\epsilon\sim\mathcal{N}(0,I_{nd}).

Long-Horizon Latent Value Expansion with Action Chunking

With the learned latent world model, we apply the action-chunked value-learning framework of MAC [41] in the latent space. First, we extract the policy from the behavioral policy by sampling behavioral action candidates a^(i)=πθ(z,ϵ(i))\hat{a}^{(i)}=\pi_{\theta}(z,\epsilon^{(i)}) from multiple noises ϵ(i)∼N(0,Ind)\epsilon^{(i)}\sim\mathcal{N}(0,I_{nd}), and extracting the best action among the candidates via rejection sampling [2]:

πθ∗(z)=argmax⁡a^(1),…,a^(M)Qφ(z,a^(i)).(8)\pi_{\theta}^{*}(z)=\underset{\hat{a}^{(1)},\ldots,\hat{a}^{(M)}}{\operatorname{argmax}}Q_{\varphi}(z,\hat{a}^{(i)}). \tag*{(8)}

We train the value function purely from latent on-policy model rollout, generated as

T=(zt,a^t,r^t,z^t+n,a^t+n,r^t+n,…,z^t+Hn),(9)\mathcal{T}=(z_t,\hat{a}_t,\hat{r}_t,\hat{z}_{t+n},\hat{a}_{t+n},\hat{r}_{t+n},\ldots,\hat{z}_{t+Hn}), \tag*{(9)}

where a^t\hat{a}_t denotes the action chunk a^t:t+n−1\hat{a}_{t:t+n-1} generated from the rejection sampling policy πθ∗(zt)\pi_{\theta}^{*}(z_t), and r^t\hat{r}_t denotes the discounted sum of rewards ∑i=0n−1γirt+i\sum_{i=0}^{n-1}\gamma^i r_{t+i} predicted from the reward model Algorithm 1 MAC-v2

Input: Dataset D\mathcal{D}, rollout length HH, action chunking size nn, rejection sampling size MM, encoder fξf_{\xi}, dynamics / reward model pψp_{\psi} / rψr_{\psi}, value VϕV_{\phi}, Q-function QϕQ_{\phi}, flow BC policy πθ\pi_{\theta}

// Training loop

while not converged do

▷ Sample action-chunked batch from the dataset (at=at:t+n−1a_t = a_{t:t+n-1}, rt=∑i=0n−1γirt+ir_t = \sum_{i=0}^{n-1} \gamma^i r_{t+i})

Sample batch {(ot,at,rt,ot+n)}∼D\{(o_t, a_t, r_t, o_{t+n})\} \sim\mathcal{D}

▷ Train dynamics, reward model (and encoder) using dataset transitions

Update pψ,fξp_{\psi}, f_{\xi} to minimize E[∥pψ(fξ(ot),at)−sg⁡(fξ(ot+n))∥22]\mathbb{E}[\|p_{\psi}(f_{\xi}(o_t), a_t) - \operatorname{sg}(f_{\xi}(o_{t+n}))\|_2^2] (Equation (5))

Update rψ,fξr_{\psi}, f_{\xi} to maximize E[log⁡(rt∣rψ(fξ(ot),at))]\mathbb{E}[\log(r_t \mid r_{\psi}(f_{\xi}(o_t), a_t))] (Equation (6))

▷ Train BC policy (and encoder) using dataset transitions

Update πθ,fξ\pi_{\theta}, f_{\xi} with flow-matching loss (Equation (7))

▷ Generate model rollouts in latent space (z^t=fξ(ot)\hat{z}_t = f_{\xi}(o_t))

for k=0,1,…,H−1k = 0, 1, \ldots, H - 1 do

a^t+kn←POLICY⁡(z^t+kn)\hat{a}_{t+kn} \leftarrow\operatorname{POLICY}(\hat{z}_{t+kn})

z^t+(k+1)n,r^t+kn∼pψ(⋅∣z^t+kn,a^t+kn),rψ(⋅∣z^t+kn,a^t+kn)\hat{z}_{t+(k+1)n}, \hat{r}_{t+kn} \sim p_{\psi}(\cdot\mid\hat{z}_{t+kn}, \hat{a}_{t+kn}), r_{\psi}(\cdot\mid\hat{z}_{t+kn}, \hat{a}_{t+kn})

▷ Update value and critic using model rollouts

Update VϕV_{\phi} with nHnH-step targets from the rollout (Equation (10))

Update QϕQ_{\phi} with the learned value function VϕV_{\phi} (Equation (11))

// Extract action from flow BC policy πθ\pi_{\theta} with rejection sampling

function POLICY⁡(z)\operatorname{POLICY}(z)

ϵ(i)∼N(0,I)\epsilon^{(i)} \sim\mathcal{N}(0, I)

a^(i)=πθ(z,ϵ(i))\hat{a}^{(i)} = \pi_{\theta}(z, \epsilon^{(i)})

return argmax⁡a^(1),…,a^(M)Qϕ(z,a^(i))\operatorname*{argmax}_{\hat{a}^{(1)}, \ldots, \hat{a}^{(M)}} Q_{\phi}(z, \hat{a}^{(i)})

rψ(⋅∣zt,at)r_{\psi}(\cdot\mid z_t, a_t). Here, initial states ztz_t is uniformly sampled from the dataset and encoded with fξf_{\xi}, and subsequent actions, rewards, and next latent states are synthesized by our rejection-sampling policy, reward model, and dynamics model, respectively, where we denote with a hat notation.

After collecting the trajectory, we learn a distributional value function Vϕ(zt):Z→Δ(R)V_{\phi}(z_t) : \mathcal{Z} \to\Delta(\mathbb{R}) with the following loss:

LV(φ)=E[−log⁡Vφ(v^t+kn∣z^t+kn)] ; v^t+kn=∑i=kH−1γ(i−k)nr^t+in+γ(H−k)nvφˉ(z^t+Hn)(10)\mathcal{L}^{V}(\varphi) = \mathbb{E}[-\log V_{\varphi}(\hat{v}_{t+kn}\mid\hat{z}_{t+kn})] \ ;\ \hat{v}_{t+kn} = \sum_{i=k}^{H-1}\gamma^{(i-k)n}\hat{r}_{t+in} + \gamma^{(H-k)n}v_{\bar{\varphi}}(\hat{z}_{t+Hn}) \tag*{(10)}

where φˉ\bar{\varphi} denotes exponentially averaged target parameters [38], vφ(⋅)=Ev∼Vφ(⋅)[v]v_{\varphi}(\cdot) = \mathbb{E}_{v\sim V_{\varphi}(\cdot)}[v], and the expectations are over T=(zt=z^t,a^t,r^t,…,z^t+Hn)\mathcal{T} = (z_t = \hat{z}_t, \hat{a}_t, \hat{r}_t, \ldots, \hat{z}_{t+Hn}) and kk uniformly sampled from {0,1,…,H−1}\{0, 1, \ldots, H - 1\}.

Finally, we train the action-chunked Q function Qφ(zt,at):Z×An→Δ(R)Q_{\varphi}(z_t, a_t) : \mathcal{Z} \times\mathcal{A}^{n} \to\Delta(\mathbb{R}) for the rejection sampling with the following loss:

LQ(φ)=Eot∼D[−log⁡Qφ(r^t+γnvφ(z^t+n)∣zt,a^t)](11)\mathcal{L}^{Q}(\varphi) = \mathbb{E}_{o_t\sim\mathcal{D}}[-\log Q_{\varphi}(\hat{r}_t + \gamma^n v_{\varphi}(\hat{z}_{t+n})\mid z_t, \hat{a}_t)] \tag*{(11)}

For both value learning, we use two-hot encoding for the distribution. We provide a pseudocode for MAC-v2 in Algorithm 1.

Experiments

We now empirically evaluate MAC-v2 and examine whether long-horizon model-based value expansion is effective for offline RL from pixels. First, we show that MAC-v2 can serve as an effective general-purpose offline RL algorithm from visual observations in § 5.1, by evaluating on offline RL benchmarks with diverse control problems and challenging long-horizon tasks where accurate value propagation is important. Second, we examine whether MAC-v2 can be also applied in robotic manipulation settings, where the offline datasets are closer to expert demonstrations and strong behavioral cloning policies already provide competitive performance, in § 5.2. Finally, we analyze the importance of long-horizon model-based value expansion of MAC-v2 and ablate the training objectives used for world model learning in § 5.3. Unless otherwise specified, all results are averaged over four random seeds.

Visual offline RL benchmarks

We first evaluate whether MAC-v2 can serve as a general-purpose offline RL method from pixels. We consider two complementary benchmark suites. Visual OGBench [42] contains structured manipulation problems with long temporal horizons, suited for evaluating whether model-based value expansion can propagate useful information over long sequences. MMBench2 [14] provides a more diverse visual control tasks and diverse source of data collection, testing whether it can be used for beyond the structured manipulation setting.

Experimental setting

Tasks and datasets. We evaluate on 25 visual single-task environments from OGBench [42] and report average success rates within each task family, following the evaluation protocol of Park et al. [44]. We additionally evaluate on 10 “unseen” tasks from MMBench2 [14], which cover a diverse set of visual control problems. We resize observations to 64×6464 \times64 and combine all the datasets (e.g., collected from curiosity-driven policy, expert policy, random policy) assigned to each task. For MMBench2, we report normalized return on a 0-100 scale using the minimum and maximum trajectory returns from the benchmark.

Methods. We compare MAC-v2 against strong model-free and model-based offline RL baselines. For model-free methods, we consider IQL [22], ReBRAC [54], IFQL, the flow-based variant of IDQL [17], and FQL [44]. For model-based methods, we focus specifically on prior offline RL methods that learn from visual observations. This literature is considerably smaller than the broader literature on online visual model-based RL methods, which are not directly comparable in our setting. For offline model-based methods, we include LOMPO [46], which learns a latent world model and apply uncertainty-aware policy optimization on top of the frozen representation, OfflineDV2 [34] (denoted as OffDV2), which combines uncertainty penalty [59] on top of the world model from DreamerV2 [10].

Results

Table 1 summarizes the results on the visual OGBench and MMBench2 tasks. On OGBench, MAC-v2 outperforms all baselines in 4 out of 5 task families, and matches QC on the remaining one (visual-cube-single). Notably, MAC-v2 shows 95% score in puzzle-3x3, where the best prior method scores 22%, while prior model-based RL methods score near-zero performance across all tasks. On MMBench2, MAC-v2 achieves the best performance on the ogbench and pygame domains, and remains competitive with strong model-free baselines such as FQL on the remaining tasks. Together, these results shows the effectiveness of long-horizon model-based value expansion across diverse visual offline RL problems.

EnvironmentModel-FreeModel-Based
IQLReBRACIFQLFQLQCLOMPOOffDV2MAC-v2
OGBench (25 tasks)
visual-cube-single (5 tasks)65±465 \pm474±174 \pm136±536 \pm562±462 \pm478±278 \pm20±00 \pm00±00 \pm074±174 \pm1
visual-cube-double (5 tasks)13±113 \pm14±34 \pm32±12 \pm15±35 \pm33±13 \pm10±00 \pm00±00 \pm053±353 \pm3
visual-scene (5 tasks)18±818 \pm833±433 \pm421±221 \pm241±241 \pm257±257 \pm22±12 \pm14±24 \pm272±1272 \pm12
visual-puzzle-3x3 (5 tasks)2±32 \pm319±119 \pm121±121 \pm122±122 \pm120±020 \pm03±23 \pm22±12 \pm195±495 \pm4
visual-puzzle-4x4 (5 tasks)0±00 \pm013±113 \pm14±54 \pm510±110 \pm120±220 \pm20±00 \pm01±01 \pm046±546 \pm5
MMBench2 (10 tasks)
dmcontrol (2 tasks)19±219 \pm239±239 \pm217±217 \pm253±153 \pm122±222 \pm214±414 \pm444±1244 \pm1231±431 \pm4
maniskill (1 task)5±25 \pm28±38 \pm311±411 \pm43±23 \pm23±23 \pm223±323 \pm30±00 \pm04±24 \pm2
ogbench (2 tasks)7±47 \pm415±215 \pm210±310 \pm330±430 \pm46±26 \pm20±00 \pm01±01 \pm033±433 \pm4
pygame (5 tasks)41±241 \pm250±550 \pm547±147 \pm158±158 \pm150±150 \pm131±331 \pm343±143 \pm166±266 \pm2

Table 1. Results on standard offline RL visual benchmark tasks. We report the success rate for OGBench tasks and normalized scores (0-100) for MMBench2 tasks across 4 seeds, averaged over 15 episodes. Best scores for each task are highlighted in magenta.

Robotic manipulation benchmarks

We next study the case where offline datasets are composed primarily of successful or near-expert demonstrations. In these datasets, behavioral cloning is already a strong baseline, which is challenging for offline RL algorithms to improve upon. This evaluation tests whether model-based value expansion can also be effective in these datasets, and whether it outperforms behavioral cloning policies.

Experimental setting

Tasks and datasets. We evaluate on three visual manipulation tasks from Robomimic (Mandlekar et al. [37]) and the ten long-horizon tasks from LIBERO-10 (Liu et al. [32]). For Robomimic, we use the multi-human (mh) datasets and 84×8484 \times84 observations from both the agent-view and wrist cameras. For LIBERO-10, we use the original 128×128128 \times128 observations and only use the agent-view camera. We report average task success rates on both benchmarks.

Methods. Because temporally coherent action sequences are particularly important in these robotic manipulation tasks, we compare against strong model-free baselines that utilizes action chunking. These include action-chunked flow-matching BC (Lipman et al. [31]), QC (Li et al. [30]), which learns an action-chunked value function and performs rejection sampling over a flow-based BC policy, and QC-IFQL, an action-chunked variant of IFQL (Hansen-Estruch et al. [17]). We also compare with model-based baselines, including LOMPO (Rafailov et al. [46]) and OfflineDV2 (Lu et al. [34]) (denoted as OffDV2).

Results

Table 2 summarizes the results on visual robotic manipulation. On Robomimic, MAC-v2 matches or exceeds BC on all three tasks and improves performance on the challenging square task. Notably, MAC-v2 is the only RL method that consistently matches or outperforms BC across the full benchmark. On LIBERO-10, MAC-v2 remains competitive with BC and strong action-chunked model-free methods. We found that LIBERO-10 consists largely of unimodal demonstration data, leaving relatively limited room for rejection sampling to improve the policy over the behavioral policy. These results suggest that MAC-v2 can retain the robustness of behavior cloning in robotic settings, while providing meaningful improvements when value-based policy selection has sufficient room to improve upon the dataset policy.

EnvironmentModel-FreeModel-Based
BCQCQC-IFQLLOMPOOffDV2MAC-v2
Robomimic (3 tasks)
lift100±0100 \pm0100±0100 \pm0100±0100 \pm027±4227 \pm4251±1951 \pm19100±0100 \pm0
can97±397 \pm397±397 \pm392±392 \pm30±00 \pm00±00 \pm095±395 \pm3
square68±2168 \pm2155±1355 \pm1355±1055 \pm100±00 \pm00±00 \pm072±1072 \pm10
LIBERO (10 tasks)
LIBERO-10 (10 tasks)80±180 \pm177±177 \pm178±378 \pm31±21 \pm20±00 \pm080±180 \pm1

Table 2. Results on visual robotic benchmarks. We report the success rate across 4 seeds, averaged over 15 episodes. Best scores for each task are highlighted in magenta.

Analysis

In this section, we discuss and analyze the components of MAC-v2 through the following Q&As. Specifically, we examine whether long-horizon model-based value expansion drives its performance gains, and which components make such expansion effective.

Q: Is long-horizon model-based value expansion important for MAC-v2?

A: To answer this question, we compare the performance of MAC-v2 with various horizons, by varying the rollout horizon HH and action chunk length nn, summarized in Figure 1. First, we evaluate rollout horizons of {10,50,100}\{10, 50, 100\} environment steps with an action chunk size of 10 on puzzle-4x4-task5. Performance improves as the horizon increases, demonstrating the benefit of long-horizon value expansion. We also vary the action chunk size over {1,5,10}\{1, 5, 10\} on puzzle-4x4-task1, and the performance improves as the chunk length increases, consistent with the result with the rollout length. Together, these results support that long-horizon model-based value expansion with action chunking is important for the performance of MAC-v2.

Importance of long-horizon rollout in MAC-v2

Figure 1. Importance of long-horizon rollout in MAC-v2.

Q: Can prior methods also perform long-horizon value estimation?

A: To answer this question, we evaluate QC (Li et al. [30]) and OfflineDV2 (Lu et al. [34]) on puzzle-4x4-task5 with different value estimation horizons. Specifically, we vary the action chunk size in QC and the model rollout length in OfflineDV2 to control the horizon of its value estimation. While MAC-v2 benefits from extending the value estimation horizon, neither baseline shows comparable improvements as its horizon increases (Figure 2). These results suggest that simply extending the value estimation horizon is insufficient for the baselines, highlighting the importance of MAC-v2’s design choices in facilitating long-horizon model-based value expansion.

Prior works do not scale with longer horizons

Figure 2. Prior works do not scale with longer horizons.

Q: Is each MAC-v2’s objective essential for long-horizon model rollouts?

A: We find that the both BC policy objective and WM objective plays an important role in retaining control-relevant and predictable information. Specifically, we remove the gradient from the BC objective and the world modeling objective to the encoder while keeping all other components unchanged on cube-double-task3. Figure 3 shows that removing the actor gradient or the world modeling gradient substantially degrades performance, suggesting that both dynamics prediction and policy objective alone are insufficient to learn a representation suitable for long-horizon latent rollouts. This highlights that MAC-v2’s design choice to incorporate policy objective and world modeling objective for representation learning is crucial for reliable long-horizon latent value expansion.

Figure 3. Objectives are essential.

Q: How about using other representation objectives instead?

A: Here, we replace the encoder learning objective of MAC-v2 with other representation objectives and examine whether the learned representation can support long-horizon value expansion. We consider pixel reconstruction (Wahlström et al. [55]), SimCLR (Chen et al. [3]), and LeWorldModel (Maes et al. [36]) and compare those methods in cube-double-task3. As shown in Figure 4, prior representation learning methods substantially degrades the performance. This highlights that MAC-v2’s objective is crucial for reliable long-horizon latent value expansion, and cannot be replaced with conventional representation learning objectives.

Figure 4. Other losses fall short.

Closing remarks

In this work, we introduced MAC-v2, an offline model-based RL framework for long-horizon model-based value expansion from visual observations. MAC-v2 combines reconstruction-free latent dynamics with action chunking, enabling long-horizon imagination with substantially fewer recursive world model predictions. Across 35 standard visual offline RL tasks and 13 robotic manipulation tasks, MAC-v2 achieves strong performance against both model-free and model-based baselines, with substantial gains on challenging OGBench and Robomimic tasks while remaining competitive with strong behavior-cloning baselines on LIBERO-10.

Several open questions remain for MAC-v2, suggesting directions for future work. First, while having larger action chunks reduces the number of recursive model predictions, but excessively long chunks reduce the policy’s reactivity. Developing adaptive or hierarchical action chunking could provide a better trade-off between long-horizon prediction and closed-loop control. Moreover, our experiments are primarily limited to simulated, single-task environments with relatively limited visual variation, leaving a substantial gap to realistic multi-task settings. Extending MAC-v2 to large-scale vision-language-action models (VLAs) or world action models (WAMs) could be a promising direction toward long-horizon model-based value expansion in more diverse and realistic domains.

AI use statement

In this work, we used generative AI tools for polishing the writing, and implementing the methods. We have not used generative AI tools for designing or providing feedback on research methodology or experiments, proposing or refining hypotheses, assisting with translation. Supporting qualitative and thematic data analysis and interpreting the results, and generating synthetic data sets, helping develop theoretical models or conceptual frameworks, formulating mathematical claims, providing critical ingredients for proving mathematical claims, assisting in the writing of proofs, cleaning and reformatting dataset, interpreting results, are not applicable to this work. The authors have manually reviewed the paper to prevent plagiarism, and read the LLM-generated code and verified and tested its correctness. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Reproducibility statement

We include implementation details needed for reproduction, such as hyperparameters (including the baselines), datasets, and evaluation protocols in Section A. We provide the full results of each tasks in Section B for better future reproducibility.

Acknowledgments

Kwanyoung Park is partly supported by the Korea Foundation for Advanced Studies (KFAS). This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley. This research was partly supported by AFOSR under FA9550-22-1-0273 and DARPA TIAMAT.

Experimental Details

We implement MAC-v2 on top of the codebase of Park et al. [43]. Each experiment takes approximately 9 hours for OGBench and MMBench2 benchmarks, 12 hours for Robomimic and 14 hours for LIBERO-10 benchmark on a single A5000 GPU.

Implementation Details

Network architecture for MAC-v2. For OGBench experiments, we use the impala_small encoder from Espeholt et al. [5], following the implementation of FQL (Park et al., [44]) to match the baselines. For LIBERO-10 experiments, we use the Resnet-18 backbone (trained from scratch) as the image is more complicated and higher resolution (128px). For other experiments, we use the convolutional encoder architecture (12M version) from DreamerV3 (Hafner et al., [12]). On top of the encoder, we use 4-layer MLP with hidden size 256 for actor, critic, and dynamics head. For critic and reward head, we use the two-hot encoding (Hafner et al., [12]).

Data augmentation. We apply random crop augmentation of size 4 for OGBench and 12 for LIBERO-10, following the implementation of Park et al. [44]. For model-free approaches, we use the same augmentation scheme as they follow the implementation of Park et al. [44]. For model-based RL baselines, we follow the setup of original papers.

Accelerating rejection sampling during training. Following Park et al. [41], we use different numbers of samples for rejection sampling during training and evaluation (which we call NtrainN_{\mathrm{train}} and NtestN_{\mathrm{test}}) to reduce the training time. Specifically, we use Ntrain=8N_{\mathrm{train}} = 8 during training and Ntest=32N_{\mathrm{test}} = 32 at test time.

Implementation details for the compared methods. For the model-free baselines, we follow the implementation by Park et al. [42] and Li et al. [30]. For LOMPO (Rafailov et al., [46]) and OfflineDV2 (Lu et al., [34]), we use the official codebase released by the authors.

MAC-v2 with terminals. We predict the terminal signal d^t=1−∏i=0n−1(1−d^t+i)\hat{d}_t = 1 - \prod_{i=0}^{n-1}(1 - \hat{d}_{t+i}) from the terminal model dψ(zt,at):Z×An→Rd_\psi(z_t, a_t) : \mathcal{Z} \times\mathcal{A}^{n} \to\mathbb{R}, and calculate the discounted reward target rt=∑i=0n−1γi(∏j=1i(1−dt+j))rtr_t = \sum_{i=0}^{n-1} \gamma^i \left(\prod_{j=1}^{i}(1-d_{t+j})\right)r_t. We use a hard threshold 0.5 and calculate d^t=1(dψ(zt,at)>0.5)\hat{d}_t = \mathbb{1}(d_\psi(z_t, a_t) > 0.5). We train the value function from latent on-policy model rollout (i.e., imaginary trajectories), generated as

T={(zt(i),a^t(i),r^t(i),d^t(i),z^t+n(i),a^t+n(i),r^t+n(i),d^t+n(i),…,z^t+Hn(i))}i=1M.(12)\mathcal{T} = \left\{\left(z_t^{(i)}, \hat{a}_t^{(i)}, \hat{r}_t^{(i)}, \hat{d}_t^{(i)}, \hat{z}_{t+n}^{(i)}, \hat{a}_{t+n}^{(i)}, \hat{r}_{t+n}^{(i)}, \hat{d}_{t+n}^{(i)}, \ldots, \hat{z}_{t+Hn}^{(i)}\right)\right\}_{i=1}^{M}. \tag*{(12)}

We calculate the value target as:

v^t+kn=∑i=kH−1γ(i−k)n(∏j=ki−1d^t+jn)r^t+in+γ(H−k)n(∏j=kH−1d^t+jn)vϕˉ(s^t+Hn),(13)\hat{v}_{t+kn} = \sum_{i=k}^{H-1} \gamma^{(i-k)n}\left(\prod_{j=k}^{i-1}\hat{d}_{t+jn}\right)\hat{r}_{t+in} + \gamma^{(H-k)n}\left(\prod_{j=k}^{H-1}\hat{d}_{t+jn}\right)v_{\bar{\phi}}(\hat{s}_{t+Hn}), \tag*{(13)}

Finally, we train the action-chunked Q function Qφ(st,at):Z×An→Δ(R)Q_{\varphi}(s_t, a_t) : \mathcal{Z} \times\mathcal{A}^{n} \to\Delta(\mathbb{R}) for the rejection sampling with the following loss:

LQ(φ)=Eot∼D[−log⁡Qφ(r^t+γn(1−d^t+jn)vψ(z^t+n)∣zt,a^t)].(14)L^{Q}(\varphi) = \mathbb{E}_{o_t\sim\mathcal{D}}\left[-\log Q_{\varphi}\left(\hat{r}_t + \gamma^n(1-\hat{d}_{t+jn})v_{\psi}(\hat{z}_{t+n}) \mid z_t,\hat{a}_t\right)\right]. \tag*{(14)}

Hyperparameters

We group the configurations into three families: model-free baselines (Table 4), model-based baselines (Table 5), and MAC-v2 (Table 6). Image resolution, encoder and augmentation are set per benchmark, and are given in Table 3. Evaluation uses 15 episodes for each seed. We match the encoder for MAC-v2 and model-free methods, but we use the original codebase’s architecture for model-based baselines as their architecture (history-conditioned RSSM) significantly differs from our method (markovian CNN encoder).

BenchmarkResolutionEncoderAug. pad
OGBench64×6464 \times64impala_small4
MMBench264×6464 \times6412M Conv.4
Robomimic84×8484 \times8412M Conv.4
LIBERO-10128×128128 \times128ResNet-1812

Table 3. Per-benchmark input and encoder settings. These apply to MAC-v2 and the model-free baselines, so every comparison uses the same encoder within a benchmark. Please refer to Section A.1 for implementation details of the encoder.

HyperparametersValue
Gradient steps500K
Learning rate3×10−43 \times10^{-4}
OptimizerAdam ([20])
Batch size256
MLP size[512,512,512,512][512, 512, 512, 512]
Frame stack3
Image augmentationDrQ random shift, paug=0.5p_{\mathrm{aug}} = 0.5 (tied)
Discount factor γ\gamma0.99

Table 4. Hyperparameters of the model-free baselines.

HyperparametersValue
Gradient steps100K
Learning rate3×10−43 \times10^{-4}
OptimizerAdam ([20])
Batch size256
MLP size[512,512,512,512][512, 512, 512, 512]
Image resolution64×6464 \times64
Frame stack1

Table 5. Shared hyperparameters of model-based baselines.

HyperparametersValue
Learning rate3×10−43 \times10^{-4}
OptimizerAdam
Layer normalizationTrue
Target network update rate τ\tau0.02
Discount factor γ\gamma0.999
λ\lambda-return coefficient1.0
Flow steps5
NtrainN_{\mathrm{train}}8
NtestN_{\mathrm{test}}32
Rollout length HH10
Action chunk size nn10
Latent dimension512
Image augmentationDrQ random shift (tied)
Value / reward losssymexp two-hot, 255 bins on [−5,5][-5,5]
Weight decay1×10−41 \times10^{-4}
Gradient clipping1.0
Frame stack1

Table 6. Hyperparameters of MAC-v2.

Shared hyperparameters for model-free baseline. IQL, ReBRAC, IFQL, FQL and QC are run from the reference FQL codebase ([44]) at its visual-task defaults. QC additionally uses best-of-32 sampling over 5-step action chunks, so it is chunked where the other four are single-step. We use paug=0.5p_{\mathrm{aug}} = 0.5 following the original codebase.

Shared hyperparameters for model-based baseline. Both are run at their authors’ defaults, changing only the uncertainty coefficient λ\lambda. Both methods train their world model for 25K steps before the policy phase. Both methods are trained for 100k steps, as we found the performance starts to degrade after 100k steps. We note that the computational budget for 100k steps for these methods is larger than MAC-v2, taking around 16 hours in a single A5000 GPU.

MAC-v2 hyperparameters. We report the hyperparameters for our method in Table 6. MAC-v2 uses the same (n,H,Ntrain,Ntest)=(10,10,8,32)(n, H, N_{\mathrm{train}}, N_{\mathrm{test}}) = (10, 10, 8, 32) across all tasks and all four benchmarks, no per-task tuning is performed. We apply DrQ-style random-shift augmentation ((Kostrikov et al., 2021), using the same shift for an observation and the next observation. We normalize the value with the maximum return of the dataset, and train the value and reward heads with symexp two-hot categorical loss of Hafner et al. [12].

Hyperparameters for baselines. For LOMPO and Offline DV2, we sweep the uncertainty penalty coefficient λ∈{1,3,10,30}\lambda\in\{1, 3, 10, 30\}. For FQL we search over the behavior-cloning coefficient α∈{0.1,0.3,1.0}\alpha\in\{0.1, 0.3, 1.0\}, and for ReBRAC over (αactor,αcritic)∈{0.01,0.1,1.0}×{0,0.01}(\alpha_{\mathrm{actor}}, \alpha_{\mathrm{critic}}) \in\{0.01, 0.1, 1.0\} \times\{0, 0.01\}. For FQL, IQL, ReBRAC, we use the reported hyperparameter from Park et al. [44] in OGBench. For QC and QC-IFQL, we search over the action chunking size {5,10}\{5, 10\}. IFQL is run at a single fixed configuration. The hyperparameters are reported in Table 7, Table 8, Table 9, Table 10 for OGBench, MMBench2, Robomimic, LIBERO-10, respectively. We denote “N/A” in the tables if a method achieves zero performance across all hyperparameters tested in our sweep. If not specified, all other hyperparameters follow the defaults provided in the original papers.

EnvironmentLOMPO (λ\lambda)OffDV2 (λ\lambda)FQL (α\alpha)ReBRAC (αactor,αcritic\alpha_{\mathrm{actor}}, \alpha_{\mathrm{critic}})IQL (α\alpha)
visual-cube-single-play-v055300(1.0, 0)1
visual-cube-double-play-v0510100(0.1, 0)0.3
visual-scene-play-v0310100(0.1, 0.01)10
visual-puzzle-3x3-play-v0310300(0.3, 0.01)10
visual-puzzle-4x4-play-v053300(0.3, 0.01)3

Table 7. Baseline hyperparameters on the OGBench visual tasks. Uncertainty penalty λ\lambda for the model-based baselines, behavior-cloning coefficient α\alpha for FQL, and (αactor,αcritic)(\alpha_{\mathrm{actor}}, \alpha_{\mathrm{critic}}) for ReBRAC.

EnvironmentLOMPO (λ\lambda)OffDV2 (λ\lambda)FQL (α\alpha)ReBRAC (αactor,αcritic\alpha_{\mathrm{actor}}, \alpha_{\mathrm{critic}})IQL (α\alpha)
cup-catch-var11100.3(0.01, 0)1
finger-turn-easy-var13010.3(0.01, 0.01)1
ms-push-banana30301.0(1.0, 0)1
og-point-var1N/A100.3(0.01, 0.01)1
og-point-var21100.3(0.01, 0.01)1
pygame-point-maze-var4310.1(0.01, 0)1
pygame-reacher-easy3011.0(0.1, 0.01)1
pygame-dungeon-explorer11030.3(0.1, 0)1
pygame-foraging30100.1(1.0, 0)1
pygame-whirlpool3100.1(0.1, 0)1

Table 8. Baseline hyperparameters on MMBench2. Uncertainty penalty λ\lambda for the model-based baselines, behavior-cloning coefficient α\alpha for FQL, and (αactor,αcritic)(\alpha_{\mathrm{actor}}, \alpha_{\mathrm{critic}}) for ReBRAC, α\alpha for IQL.

TaskLOMPO (λ\lambda)OffDV2 (λ\lambda)BC (nn)QC (nn)QC-IFQL (nn)
lift3101055
canN/AN/A10510
squareN/AN/A1055

Table 9. Baseline hyperparameter on Robomimic. Uncertainty penalty λ\lambda for the model-based baselines, action chunk size nn for BC, QC and QC-IFQL.

TaskLOMPO (λ\lambda)OffDV2 (λ\lambda)
put the white mug on the left plate and put the yellow and white mug on the right plateN/AN/A
put the white mug on the plate and put the chocolate pudding to the right of the plateN/AN/A
put the yellow and white mug in the microwave and close it3N/A
turn on the stove and put the moka pot on itN/AN/A
put both the alphabet soup and the cream cheese box in the basketN/AN/A
put both the alphabet soup and the tomato sauce in the basket10N/A
put both moka pots on the stoveN/AN/A
put both the cream cheese box and the butter in the basketN/AN/A
put the black bowl in the bottom drawer of the cabinet and close itN/AN/A
pick up the book and place it in the back compartment of the caddyN/AN/A

Table 10. Per-task uncertainty coefficient λ\lambda of the model-based baselines on LIBERO-10. We sweep over λ∈{1,3,10,30}\lambda\in\{1, 3, 10, 30\} on each tasks for model-based baselines. N/A marks tasks on which every λ\lambda scored zero.

Complete Numerical Results

For completeness, we provide the full per-task results for OGBench tasks in Table 11 (corresponding to Table 1). The results are averaged over 4 seeds and we report the standard deviations for each tasks. We highlight the numbers with the best performance.

EnvironmentIQLReBRACIFQLFQLQCLOMPOOffDV2MAC-v2
visual-cube-single-task173±773 \pm784±484 \pm444±644 \pm667±367 \pm388±588 \pm50±00 \pm00±00 \pm087±287 \pm2
visual-cube-single-task257±657 \pm671±671 \pm629±929 \pm970±470 \pm480±580 \pm50±00 \pm00±00 \pm072±372 \pm3
visual-cube-single-task379±479 \pm485±185 \pm159±1059 \pm1068±1368 \pm1390±490 \pm40±00 \pm00±00 \pm086±386 \pm3
visual-cube-single-task462±1862 \pm1869±469 \pm427±727 \pm751±1151 \pm1168±568 \pm50±00 \pm00±00 \pm064±364 \pm3
visual-cube-single-task554±454 \pm462±462 \pm422±422 \pm457±857 \pm862±662 \pm60±00 \pm00±00 \pm062±362 \pm3
visual-cube-double-task125±725 \pm75±35 \pm36±26 \pm215±1215 \pm126±26 \pm20±00 \pm01±11 \pm167±467 \pm4
visual-cube-double-task24±14 \pm11±01 \pm00±00 \pm00±00 \pm02±22 \pm20±00 \pm00±00 \pm047±1447 \pm14
visual-cube-double-task313±413 \pm44±34 \pm31±11 \pm12±22 \pm22±12 \pm10±00 \pm00±00 \pm052±652 \pm6
visual-cube-double-task41±11 \pm10±00 \pm00±10 \pm10±00 \pm00±10 \pm10±00 \pm00±00 \pm010±510 \pm5
visual-cube-double-task521±321 \pm310±1410 \pm142±12 \pm14±24 \pm24±14 \pm10±00 \pm00±00 \pm090±490 \pm4
visual-scene-task169±4069 \pm4097±297 \pm288±488 \pm499±199 \pm1100±0100 \pm07±57 \pm59±69 \pm6100±0100 \pm0
visual-scene-task20±00 \pm065±1665 \pm160±10 \pm188±588 \pm596±396 \pm35±65 \pm69±59 \pm5100±0100 \pm0
visual-scene-task317±217 \pm24±34 \pm315±615 \pm619±519 \pm589±989 \pm90±00 \pm00±00 \pm090±290 \pm2
visual-scene-task41±01 \pm00±10 \pm10±00 \pm01±11 \pm11±11 \pm10±00 \pm00±00 \pm042±2942 \pm29
visual-scene-task50±00 \pm00±00 \pm00±00 \pm00±00 \pm00±00 \pm00±00 \pm00±00 \pm029±2929 \pm29
visual-puzzle-3x3-task17±127 \pm1290±490 \pm4100±1100 \pm197±197 \pm1100±0100 \pm013±913 \pm99±59 \pm5100±0100 \pm0
visual-puzzle-3x3-task20±00 \pm00±00 \pm00±00 \pm00±00 \pm00±00 \pm00±00 \pm00±00 \pm0100±0100 \pm0
visual-puzzle-3x3-task30±10 \pm12±12 \pm11±11 \pm11±01 \pm00±00 \pm00±00 \pm00±00 \pm083±2283 \pm22
visual-puzzle-3x3-task40±00 \pm01±11 \pm11±01 \pm06±26 \pm20±00 \pm00±00 \pm00±00 \pm098±298 \pm2
visual-puzzle-3x3-task51±11 \pm10±10 \pm11±21 \pm27±37 \pm30±00 \pm00±00 \pm00±00 \pm096±696 \pm6
visual-puzzle-4x4-task10±00 \pm022±522 \pm511±1911 \pm1928±528 \pm557±757 \pm70±00 \pm02±12 \pm190±490 \pm4
visual-puzzle-4x4-task20±00 \pm09±29 \pm20±10 \pm11±01 \pm02±12 \pm10±00 \pm01±01 \pm021±821 \pm8
visual-puzzle-4x4-task30±00 \pm015±215 \pm22±32 \pm314±314 \pm331±1131 \pm110±00 \pm01±11 \pm182±1882 \pm18
visual-puzzle-4x4-task40±00 \pm010±110 \pm14±44 \pm47±27 \pm28±88 \pm80±00 \pm00±10 \pm120±720 \pm7
visual-puzzle-4x4-task50±00 \pm010±110 \pm10±00 \pm01±01 \pm01±11 \pm10±00 \pm01±11 \pm116±1616 \pm16

Table 11. Complete results for OGBench visual benchmark tasks.

EnvironmentIQLReBRACIFQLFQLQCLOMPOOffDV2MAC-v2
cup-catch-var113±213 \pm247±647 \pm611±111 \pm171±171 \pm122±422 \pm44±34 \pm345±2645 \pm2633±433 \pm4
finger-turn-easy-var124±524 \pm532±432 \pm423±423 \pm435±335 \pm323±223 \pm224±924 \pm943±143 \pm129±529 \pm5
ms-push-banana5±25 \pm28±38 \pm311±411 \pm43±23 \pm23±23 \pm223±323 \pm30±00 \pm04±24 \pm2
og-point-var18±58 \pm517±217 \pm211±311 \pm332±632 \pm67±27 \pm20±00 \pm01±01 \pm042±542 \pm5
og-point-var27±47 \pm414±314 \pm39±39 \pm328±528 \pm56±26 \pm20±00 \pm01±01 \pm024±424 \pm4
pygame-point-maze-var410±610 \pm625±1425 \pm1426±726 \pm745±445 \pm416±516 \pm52±32 \pm35±15 \pm150±250 \pm2
pygame-reacher-easy11±211 \pm211±111 \pm110±110 \pm112±212 \pm29±19 \pm15±15 \pm128±428 \pm422±322 \pm3
pygame-dungeon-explorer143±243 \pm257±457 \pm446±346 \pm365±565 \pm564±164 \pm127±827 \pm828±728 \pm778±378 \pm3
pygame-foraging65±065 \pm065±265 \pm265±165 \pm176±176 \pm168±168 \pm144±844 \pm852±352 \pm382±182 \pm1
pygame-whirlpool74±774 \pm789±889 \pm886±486 \pm491±191 \pm192±392 \pm379±1879 \pm1899±199 \pm199±199 \pm1

Table 12. Complete results for MMBench2 tasks.

EnvironmentModel-FreeModel-Based
BCQCQC-IFQLLOMPOOffDV2MAC-v2
LIBERO (10 tasks)
turn on the stove and put the moka pot on it96±396 \pm389±1089 \pm1085±985 \pm90±00 \pm00±00 \pm097±297 \pm2
put the black bowl in the bottom drawer of the cabinet and close it98±198 \pm186±986 \pm994±394 \pm30±00 \pm00±00 \pm095±395 \pm3
put the yellow and white mug in the microwave and close it84±384 \pm377±777 \pm783±883 \pm89±129 \pm120±00 \pm081±881 \pm8
put both moka pots on the stove76±476 \pm469±269 \pm264±1064 \pm100±00 \pm00±00 \pm075±1675 \pm16
put both the alphabet soup and the cream cheese box in the basket69±469 \pm467±1867 \pm1868±868 \pm80±00 \pm00±00 \pm075±675 \pm6
put both the alphabet soup and the tomato sauce in the basket77±577 \pm580±880 \pm875±875 \pm82±42 \pm40±00 \pm082±782 \pm7
put both the cream cheese box and the butter in the basket84±1084 \pm1090±690 \pm685±385 \pm30±00 \pm00±00 \pm086±586 \pm5
put the white mug on the left plate and put the yellow and white mug on the right plate83±683 \pm671±1171 \pm1183±383 \pm30±00 \pm00±00 \pm074±974 \pm9
put the white mug on the plate and put the chocolate pudding to the right of the plate49±749 \pm759±559 \pm568±1468 \pm140±00 \pm00±00 \pm048±948 \pm9
pick up the book and place it in the back compartment of the caddy84±584 \pm580±580 \pm578±778 \pm70±00 \pm00±00 \pm082±582 \pm5

Table 13. Complete results for LIBERO tasks.

Training Curves

We provide the training curve of MAC-v2 for the OGBench, MMBench, robomimic, LIBERO-10 result of Table 1 and Table 2, in Figure 5 to Figure 8. We plot the mean and the standard deviation (across 4 seeds) by covering [mean - std, mean + std] area with a lighter color.

Figure 5. Training curve of MAC-v2 in OGBench environments. We report the success rate for 15 evaluation episodes across 4 seeds (total 60 episodes). Shaded region represents the [mean - std, mean + std].

Figure 8. Training curve of MAC-v2 in LIBERO-10 environments. We report the success rate for 15 evaluation episodes across 4 seeds (total 60 episodes). Shaded region represents the [mean - std, mean + std]. For the task instruction of each tasks for this graph, please refer to Table 2.

Figure 6. Training curve of MAC-v2 in MMBench2 environments. We report the normalized return for 15 evaluation episodes across 4 seeds (total 60 episodes). Shaded region represents the [mean - std, mean + std].

Figure 7. Training curve of MAC-v2 in Robomimic environments. We report the success rate for 15 evaluation episodes across 4 seeds (total 60 episodes). Shaded region represents the [mean - std, mean + std].

References

  1. [1]Michael S. Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2209.15571
  2. [2]Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2209.14548
  3. [3]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020.arxiv.org/abs/2002.05709
  4. [4]Hojun Chung, Junseo Lee, and Songh​wai Oh. Offline reinforcement learning with universal horizon models. arXiv preprint arXiv:2605.15603, 2026.arxiv.org/abs/2605.15603
  5. [5]Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018.arxiv.org/abs/1802.01561
  6. [6]Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value expansion for efficient model-free reinforcement learning. In International Conference on Machine Learning (ICML), 2018.DOI
  7. [7]Minghao Fu, Fan Feng, Nicklas Hansen, and Biwei Huang. Back to parsimonious latents: Learning task-centric world models from visual foundations. arXiv preprint arXiv:2605.25620, 2026.arxiv.org/abs/2605.25620
  8. [8]Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2021.
  9. [9]Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2301.02328
  10. [10]Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020a.arxiv.org/abs/2010.02193
  11. [11]Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), 2020b.arxiv.org/abs/1912.01603
  12. [12]Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640:647–653, 2025a.DOI
  13. [13]Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models. arXiv preprint arXiv:2509.24527, 2025b.arxiv.org/abs/2509.24527
  14. [14]Nicklas Hansen and Xiaolong Wang. Hallucination in world models is predictable and preventable, 2026.arxiv.org/abs/2606.27326
  15. [15]Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In International Conference on Machine Learning (ICML), 2022.arxiv.org/abs/2203.04955
  16. [16]Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.16828
  17. [17]Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. ArXiv, abs/2304.10573, 2023.
  18. [18]Riashat Islam, Manan Tomar, Alex Lamb, Yonathan Efroni, Hongyu Zhang, Aniket Didolkar, Dipendra Misra, Xin Li, Harm Van Seijen, Remi Tachet des Combes, et al. Agent-controller representations: Principled offline rl with rich exogenous information. arXiv preprint arXiv:2211.00164, 2022.arxiv.org/abs/2211.00164
  19. [19]Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel : Model-based offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.
  20. [20]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.arxiv.org/abs/1412.6980
  21. [21]Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations (ICLR), 2021.
  22. [22]Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2110.06169
  23. [23]Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2006.04779
  24. [24]Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-art, pp. 45–73. Springer, 2012.
  25. [25]Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pp. 5639–5650. PMLR, 2020.arxiv.org/abs/2004.04136
  26. [26]Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.09754
  27. [27]John M Lee. Introduction to Smooth Manifolds. Springer, 2012.
  28. [28]Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv, abs/2005.01643, 2020.arxiv.org/abs/2005.01643
  29. [30]Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. arXiv preprint arXiv:2507.07969, 2025b.arxiv.org/abs/2507.07969
  30. [31]Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.02747
  31. [32]Bo Liu, Yifeng Zhu, Ch reen?ongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023a.arxiv.org/abs/2306.03310
  32. [33]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations (ICLR), 2023b.arxiv.org/abs/2209.03003
  33. [34]Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. Transactions on Machine Learning Research (TMLR), 2023.arxiv.org/abs/2206.04779
  34. [35]Hao Luo, Wanpeng Zhang, Yicheng Feng, Sipeng Zheng, Haiwen Xu, Chaoyi Xu, Ziheng Xi, Yuhui Fu, and Zongqing Lu. Being-h0-7: A latent world-action model from egocentric videos. arXiv preprint arXiv:2605.00078, 2026.arxiv.org/abs/2605.00078
  35. [36]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. Leworld-model: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026.arxiv.org/abs/2603.19312
  36. [37]Ajay Mandlekar, Danfei Xu, J. Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.arxiv.org/abs/2108.03298
  37. [38]Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.arxiv.org/abs/1312.5602
  38. [39]Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. ArXiv, abs/2006.09359, 2020.arxiv.org/abs/2006.09359
  39. [41]Kwangyoung Park, Seohong Park, Youngwoon Lee, and Sergey Levine. Scalable offline model-based rl with action chunks. International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2512.08108
  40. [42]Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Obgench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), 2025a.arxiv.org/abs/2410.20092
  41. [43]Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. arXiv, abs/2506.04168, 2025b.arxiv.org/abs/2506.04168
  42. [44]Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In International Conference on Machine Learning (ICML), 2025c.arxiv.org/abs/2502.02538
  43. [45]Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. ArXiv, abs/1910.00177, 2019.
  44. [46]Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for dynamics and control, pp. 1154–1168. PMLR, 2021.arxiv.org/abs/2012.11547
  45. [48]Julian Schrittwieser, Ioannis Antonoglou, T. Hubert, K. Simonyan, L. Sifre, Simon Schmitt, A. Guez, Edward Lockhart, Demis Hassabis, T. Graepel, T. Lillicrap, and D. Silver. Mastering Atari, Go, Chess and Shogi by planning with a learned model. Nature, 588 7839:604–609, 2020.
  46. [49]Anya Sims, Cong Lu, Jakob N Foerster, and Yee W Teh. The edge-of-reach problem in offline model-based reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2402.12527
  47. [50]Gwanwoo Song, Kwanyoung Park, and Youngwoon Lee. Chunk-guided q-learning. arXiv preprint arXiv:2603.13971, 2026.arxiv.org/abs/2603.13971
  48. [51]Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International conference on machine learning, pp. 9870–9879. PMLR, 2021.arxiv.org/abs/2009.08319
  49. [52]Ruixiang Sun, Hongyu Zhang, Xin Li, and Riashat Islam. Learning latent dynamic robust representations for world models. arXiv preprint arXiv:2405.06263, 2024.arxiv.org/abs/2405.06263
  50. [53]Yihao Sun, Jiaj Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman inconsistency for model-based offline reinforcement learning. In International Conference on Machine Learning, pp. 33177–33194. PMLR, 2023.
  51. [54]Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2305.09836
  52. [55]Niklas Wahlström, Thomas B Schön, and Marc Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXiv:1502.02251, 2015.arxiv.org/abs/1502.02251
  53. [56]Shenghua Wang, Ziyuan Chen, Le Gan, Shuai Feng, and De-Chuan Zhan. Semopo: learning high-quality model and policy from low-quality offline visual datasets. In Proceedings of the 41st International Conference on Machine Learning, pp. 49867–49889, 2024.
  54. [57]Yifan Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. ArXiv, abs/1911.11361, 2019.arxiv.org/abs/1911.11361
  55. [58]Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Chan, and Xianyuan Zhang. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2303.15810
  56. [59]Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2005.13239
  57. [60]Tianhe Yu, Ayiral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2102.08363
  58. [61]Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems (RSS), 2023.arxiv.org/abs/2304.13705

Paper details

vlnc:W_uj5m3vry
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 17

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents