VaDar: Implicit Goal Conditioning via Value Disaggregation

Introduction

Learning to complete long-horizon, multi-step tasks is a fundamental challenge across a number of core AI domains, from robotics [26], to language modeling [67], to chemical reaction design [39]. A popular approach, often referred to as hierarchical goal conditioning (HGC) in reinforcement learning literature, decomposes these complex, long-horizon tasks into a set of subgoals, each of which can be executed in sequence by a dedicated learned, goal-conditioned policy ([26, 45, 34, 42]). However, attempting to learn per-subtask policies introduces a new set of challenges. Myopic completion of a given sub-task might compromise performance on future sub-tasks. Moreover, explicit decompositions often require sub-tasks to be sequential in nature, and necessitate a dedicated sub-task planner to decide the current subtask to be attempted during policy deployment.

In this paper, we propose an alternative to HGC that we call Value-Disaggregation (VADAR), a framework for sample-efficient reinforcement learning on long-horizon multi-step tasks. We focus on continuous control robotic manipulation tasks that can be decomposed into sub-tasks $g^1,\ldots,g^{|G|}$. These subgoal evaluations are rapidly becoming highly reliable with today’s VLMs ([20]). While HGC learns policies $\pi(s,g)$ that take sub-goals $g$ as explicit input, VADAR instead learns a sub-goal agnostic policy $\pi(s)$, with a critic that estimates the values of all sub-tasks simultaneously. The policy is then trained to maximize a linear combination of critic values, optimizing for an aggregate performance on current-sub-goals as well as future tasks. We stress that VADAR is a generic algorithmic template, being compatible with any preferred choice of critic learning algorithm or policy extraction method; we depict VADAR in Figure 1.

VADAR removes the need for subgoal-evaluation at inference by design. Thus, compute-intensive VLM calls used to delineate subgoals, need only be made at training time, and not during policy rollouts. Moreover, VADAR is flexible across both task structure and reward constraints, enabling activations both to safety-constrained reinforcement learning (Appendix § 19.§ 1), and to non-sequential task orders which could admit a directed acyclic graph (DAG) structure. (Appendix § 15).

Across a variety of benchmarks for continuous control and policy extraction methods, we find that VADAR, (1) Consistently outperforms HGC (§ 4), hindsight experience replay (HER) ([4]), and dense preference-based rewards ([10, 37]); (2) Enables the first non-VLA BC-to-online RL method to converge on the evaluated long-horizon ROBOCASA365 tasks ([46]) under fixed initial-state conditions; (3) Enables multi-objective reinforcement learning by incorporating safety constraints (Appendix § 19.§ 1), and enables test-time steering (§ 6.3) by combining critic learning with best-of-$N$ sampling ([36]); and (4) Works seamlessly with VLM supervision (Appendix § 17) and reaches convergence when goals are specified by reasoning Vision-Language Models (VLMs). This allows sample efficient robot learning from high-dimensional visual observations.

To understand these gains, our ablations examine dense subgoal supervision and future-aware policy extraction (Section 6.2). We find that future-aware policy extraction reduces myopic behavior, while intermediate sub-goal supervision improves learning even when policy extraction uses only final-subgoal values. This suggests that subgoals can support long-horizon RL through critic learning without explicit policy goal conditioning. Taken together, our findings invert the traditional perspectives on overcoming the challenges of long horizon, valued-based reinforcement learning. In short, we find: Rather than myopically conditioning policies via decomposed long-horizon sub-goals, predicting temporally rich subgoal successes can provide critics with dense supervision to enhance the accuracy of long-reaching measurements of full-task success.

Related Work

Long-horizon reinforcement learning via temporal abstraction. Long-horizon reinforcement learning is commonly addressed by introducing temporal abstraction: instead of reasoning over primitive actions, the agent reasons over extended behaviors, skills, or subgoals. Classical formulations include the options framework, where temporally extended actions are specified by initiation sets, intra-option policies, and termination conditions ([62, 5]); hierarchical abstract machines ([51]); feudal reinforcement learning ([17, 65]); and MAXQ value decomposition ([18]). Subsequent work studies how such skills and hierarchies can be discovered from data ([22, 21, 11, 53]). While these methods differ in how subtasks are specified or learned, they use subgoal structure to organize policy execution: the learned policy must either act through a prescribed hierarchy of subtasks([18, 51]) or utilize temporally extended controllers ([5, 22]). In contrast, VADAR assumes that subgoals are part of the MDP design, but does not learn or execute a hierarchy over them; subgoals are used only to construct training-time value targets for a single flat policy.

Goal-conditioned reinforcement learning. Goal condition RL provide a natural design for conditioning on desired outcomes/sub-goals. Instead of learning a separate controller for each task, goal-conditioned methods learn policies or value functions indexed by a goal, enabling transfer across related goal-reaching problems. Classical dynamic-goal learning first studied transfer across changing goal-reaching tasks ([30]). This idea was later extended through universal value function approximators, which learn goal-indexed value functions with function approximation ([57]), and hindsight relabeling, which reuses failed rollouts as successful examples for alternative achieved goals ([4]). However, conditioning a flat policy on the final task goal does not by itself resolve long-horizon credit assignment. For this reason, hierarchical goal-conditioned RL (HGC) uses goal conditioning as a local control primitive: a high-level policy, planner, or task specification provides intermediate subgoals, and a low-level goal-conditioned policy learns to reach them ([42, 34, 27, 43]). This reduces the effective horizon, but introduce additional dependencies: they often assume a sequential subgoal structure, treat only one subgoal as active at a time, and require the deployed policy to be supplied with the correct current goal. These issues are costly and difficult when subgoal progress and completion must be evaluated by expert rules or large vision-language models during inference. VADAR avoids this test-time dependency by learning a subgoal-agnostic policy; goal information enters only through the critic during training.

Value decomposition and reward-factorized critics. MAXQ decomposes the value of a hierarchical policy into child-task values and completion functions, reflecting the procedural structure of a subroutine hierarchy ([18]). Successor representations and successor features instead factor value prediction into predictive components and reward-dependent weights ([16, 7]). VADAR follows this latter spirit of reward-factorized value estimation, but uses subgoal-completion signals to learn a vector-valued critic in an offline-to-online actor-critic pipeline. Closer to robotics, task-and-motion planning (TAMP) couples high-level symbolic task planners with low-level continuous motion planners over a fixed subtask vocabulary ([31, 25]). VADAR shares the assumption that the task admits a subgoal-level decomposition, but does not plan over symbolic actions or commit to a fixed sequencing at execution time: the decomposition enters only through the critic, and the deployed policy is a single flat, subgoal-agnostic controller.

Representation learning for value-based RL. Recent work has shown that auxiliary prediction objectives can improve value learning by shaping the representations used by the critic. MR.Q ([24]) learns state and state-action representations that predict rewards, termination, and future latent states, using model-based objectives to improve a model-free actor–critic without planning through the learned dynamics. DR.O ([40]) builds on this approach by encouraging the learned representations to retain information about future states through a mutual-information objective, together with a recency-aware replay scheme. VADAR is complementary to these approaches: rather than introducing an auxiliary dynamics or representation-learning objective, it obtains additional supervision directly from the RL problem by fitting subgoal value functions at multiple temporal resolutions. Thus, the shared critic representation is trained simultaneously by intermediate and long-horizon TD targets, which we find can improve final-task value estimation even when the intermediate values are not used for policy extraction.

Preliminaries

MDPs. We formulate our problem as a Markov Decision Process (MDP) $M := (S, A, P_0, P, R, \gamma)$, with states $s \in S$, actions $a \in A$, initial state distribution $P_0$, transition probabilities $P$, reward $R$, and discount factor $\gamma\in(0,1)$. At each timestep $t$, the agent (e.g., robot) observes the state $s_t \in S$, takes an action $a_t \sim\pi(a_t \mid s_t) \in A$, transitions to the next state $s_{t+1}$ according to $s_{t+1} \sim P(s_{t+1} \mid s_t, a_t)$ while receiving the reward $R(s_t, a_t)$. For the MDP $M_{\mathrm{ENV}}$, we let $\mathbb{E}^{\pi,M}$ (resp. $\mathbb{P}^{\pi,M}$) denote the expectation (resp. probability distribution) over trajectories $(s_0, a_0, \ldots, s_T, a_T)$ with length $T+1$, with initial state distribution $s_0 \sim P_0$ and transition operator $P$. The object of reinforcement learning (RL) is to train a policy to optimize the cumulative discounted return: $J^M(\pi_\theta) = \mathbb{E}^{\pi}\left[\sum_{t \ge0} \gamma^t R(s_t, a_t)\right]$. We also recall the Q-function $Q^{\pi,M}(s,a) := \mathbb{E}^{\pi}\left[\sum_{t \ge0} \gamma^t R(s_t,a_t) \mid(s_t,a_t) = (s,a)\right]$ and value function $V^{\pi,M}(s) := \mathbb{E}_{a \sim\pi(\cdot\mid s)}[Q^{\pi}(s,a)]$. When $M$ is clear from context, we can omit it from the subscript.

Value-based Reinforcement Learning. Value-based RL methods ([60]) simultaneously learn a policy $\pi_\theta$ and associated $Q_\phi$-function, maintaining (1) a replay buffer $\mathcal{B} = \{(s,a,s',r,\text{done})\}$ of past states $s$, actions $a$, successor states $s' \sim P(s,a)$, rewards $r$, and done indicators done; (2) a policy $\pi_\theta$, critic network $Q_\phi: S \times A \to\mathbb{R}$ trained from this buffer. The parameters of $Q_\phi$ are updated to approximate $Q^{\pi_\theta}$, and $\pi_\theta(s)$ is updated to maximize critic values. Specifically, updates are sampled $(s,a,s',r) \sim\mathcal{B}$, and new actions $a' \sim\pi_\theta(s')$. Then,

1. Critic parameters $\phi$ are updated via gradient descent on a given critic loss

which measures the squared Bellman residual to the future target $y^{\mathrm{target}} = r + \gamma Q_{\phi}(s', a')$, and which may also incorporate some additional regularization $L_{\mathrm{reg}}(\cdot)$ depending on $s'$, $a'$, $\pi_{\theta}$ (omitted as arguments in the critic loss for simplicity).

2. Policy parameters $\theta$ are updated via gradient descent on a given policy loss $L_{\mathrm{policy}}(\pi_{\theta} \mid s, Q)$ mapping from policies $\pi_{\theta}: \mathcal{S} \to\Delta(\mathcal{A})$, states $s \in\mathcal{S}$, and $Q$ to $\mathbb{R}$.

Representative Algorithms: SAC, FQL and OGPO. We instantiate VADAR with Soft Actor-Critic (SAC) (Haarnoja et al., [29]), Flow Q-Learning (FQL) (Park et al., [50]), and off-policy generative policy optimization (OGPO) (Patil et al., [52]). SAC uses a maximum-entropy policy objective; FQL and OGPO support expressive generative control policies. These backbones allow us to test value disaggregation across different policy-extraction methods. Their policy objectives and algorithm-specific critic updates are detailed in Appendix § 11.

Sparse-Reward RL. In many tasks, such as those encountered in robot manipulation, defining reward that densely measure progress towards goal completion is challenging ([3, 32]). Thus, we consider a sparse-reward defined via the done indicator $\mathrm{done} \in\{0, 1\}$ indicating task completion. Standard practice is to set $r \leftarrow r_{\mathrm{sparse}}(\mathrm{done})$, where

and where $r_{\mathrm{done}}$ is a small positive constant (e.g. $r_{\mathrm{done}} = 5$). In words, $r_{\mathrm{sparse}}$ penalizes each time-step until the task is not completed, and provides a small positive bonus for task success.

Offline-to-Online RL. In many robotics settings, environment interaction is expensive and a fixed dataset of demonstrations $\mathcal{D}_{\mathrm{off}}$ is available. We follow the now-standard offline-to-online recipe (Nair et al., [44]): pretrain a behavior-cloning policy $\pi_{\mathrm{BC}}$ on $\mathcal{D}_{\mathrm{off}}$, then initialize the online policy from $\pi_{\mathrm{BC}}$ and fine-tune online, optionally mixing $\mathcal{D}_{\mathrm{off}}$ into the off-policy updates at a fixed ratio (Ball et al., [6]). All algorithms in our experiments share the same BC-pretraining and warm-start protocol, so comparisons isolate the online fine-tuning method.

Dense Preference Reward. An alternative approach to long-horizon RL is to learn dense preference-based rewards. We describe the formulation and implementation in Appendix § 13 and compare against this approach in § 6 and Appendix § 14.§ 2.

Hierarchical Goal Conditioning and Its Limitations

Algorithm 1 Hierarchical Goal Conditioning RL (HGC-RL).

1: Parameters: discount factor $\gamma$; learning rate $\eta$ 2: Initialize critic networks $Q_{\phi}: \mathcal{S} \times\mathcal{A} \times G \to\mathbb{R}$, policy $\pi_{\theta}: \mathcal{S} \times G \to\Delta(\mathcal{A})$, buffer $\mathcal{B} \leftarrow\varnothing$ 3: for episode $= 1, 2, \ldots$ do 4:     while episode not done do 5:         $a_t \leftarrow\pi_{\theta}(s_t, g_t)$, transition $s_{t+1} \sim P(s_t, a_t)$         $\triangleright g_t = g^{\mathrm{curr}} := \min\{g : \mathrm{done}_g = 0\}$ 6:         $\mathcal{B} \leftarrow\mathcal{B} \cup\{(s_t, a_t, s_{t+1}, r_t, g_t)\}$, $r_t \leftarrow r_{\mathrm{sparse}}(\mathrm{done}_{g_t})$ 7:     for number of updates do 8:         Sample $(s, a, s', r, g) \sim\mathcal{B}$, $a' \sim\pi_{\theta}(s')$ 9:         $y_g^{\mathrm{target}} \leftarrow r_g + \gamma Q_{\phi}(s', a', g)$         $\triangleright$ Target for current goal 10:         $\phi\leftarrow\phi- \eta\nabla_{\phi} L_{\mathrm{critic}}(Q_{\phi}(\cdot,\cdot,g) \mid s, a, y_g^{\mathrm{target}})$ 11:         $\theta\leftarrow\theta- \eta\nabla_{\theta} L_{\mathrm{policy}}(\pi_{\theta}(\cdot,g) \mid s, a, Q_{\phi}(\cdot,\cdot;g))$

Long-horizon reinforcement learning tasks require many time steps to complete. Sparse-rewards make credit assignment - identifying which actions contribute to task success/failure - challenging. Thus, prior work introduces hierarchical goal conditioning (HGC)², which decomposes the problem into a sequence of subgoals $G = \{g\}_{g \in G}$. Traditional goal-conditioned RL assumes that the completion of individual subgoals, what we call subgoal-dones $\mathrm{done}_g(s) := \mathbb{I}\{g\text{ is completed at state }s\}$, can be evaluated at both training time and during evaluation. Traditionally, $\mathrm{done}_g$ have been hand-specified, but modern VLMs can now evaluate $\mathrm{done}_g$'s specified by natural language queries (Chen et al., [12]). For simplicity, in the body of this paper we assume these goals satisfy a linear order with $g^1 \prec g^2 \cdots\prec g^{|G|} \in G$, where $g^{|G|}$ denotes final task completion and $g^1$ is the first goal (Appendix § 15 easily removes this sequentiality). Under this sequentiality condition, we define the current goal $g^{\mathrm{curr}} = \min\{g : \mathrm{done}_g = 0\}$, where min is taken with respect to the order $\prec$. Algorithm 1 depicts how value-based RL methods can be modified to incorporate HGC. These modifications of standard value-based RL, depicted in blue, are: policies and critics take the current goal $g_t = g^{\mathrm{curr}}$ as an argument (Line 2), so that in particular, the policy needs $g_t$ to select actions (Line 5). This necessitates that $g_t$ is tracked in the replay buffer (Line 6), is used to determine the critic target (Line 9), and features in both the policy and critic loss (Lines 10, 11). However, we identify 3 limitations of HGC, outlined below.

Limitation 1: Accurate goals must be supplied to the policy at test-time, which is prohibitive if sub-goal completions requires expert engineering or larger models to evaluate. Limitation 2: Optimizing for the current goal can be overly myopic if future actions depend subtly on how the current goal is completed, such as in robotic pick and place tasks. Limitation 3: HGC often requires goals to have a sequential structure, with only one active goal at a time.

We examine these limitations in Appendix § 12. As a point of comparison, we introduce implicit hierarchical goal conditioning (iHGC), which retains goal-conditioned critics but uses a subgoal-independent policy $\pi(a \mid s)$, removing the need for test-time goal information (Limitation 1). Although iHGC can outperform traditional HGC as shown in Figure 10, both optimize the policy against the current-subgoal critic alone and remain susceptible to myopic behavior (Limitation 2). This motivates VADAR, which incorporates future-subgoal values during policy optimization.

The Value Disaggregation (red!50!black VaDar) Framework

We now introduce Value Disaggregation (VADAR, Algorithm 2), a framework for subgoal-driven long-horizon RL that addresses the limitations of goal conditioning described in § 4. VADAR combines three key insights: (1) Like implicit hierarchical goal conditioning (iHGC), goal-conditioned critics can train a single, subgoal-independent policy. (2) Leveraging weighted combinations of critics can balance current and future goal completion, thereby reducing the myopic overexploitation described in Limitation 2. Finally, (3) The extraction from multiple critics is naturally compatible with general goal topologies, addressing Limitation 3 by allowing more than one subgoal to be active at any given time.

Algorithm 2 Value Disaggregation (VADAR)

1:  Parameters: discount factor $\gamma$; learning rate $\eta$; TD clipping parameter $c_{\mathrm{clip}}$.
2:  Initialize goal-conditioned critic $Q_{\phi}: S \times A \times G \to \mathbb{R}$; policy $\pi_{\theta}: S \to \Delta(A)$; replay buffer $\mathcal{B} \leftarrow \varnothing$.
3:  for episode $e = 1, 2, \ldots$ do
4:      while episode not done do
5:          $a_t \leftarrow \pi_{\bar{\theta}}(s_t)$, transition $s_{t+1} \sim \mathcal{P}(s_t,a_t)$
6:          Evaluate subtask completion vector: $d_t = \{\operatorname{done}_g(s_{t+1})\}_{g \in G} \in \{0,1\}^{G}$
7:          $\mathcal{B} \leftarrow \mathcal{B} \cup \{(s_t,a_t,s_{t+1},\mathbf{r}_t,\mathbf{d}_t)\}$, where $r_t[g] \leftarrow r_{\mathrm{sparse}}(d_t[g])$
8:          for number of update steps do
9:              Sample $(s,a,s',r,d) \sim \mathcal{B}$, $a' \sim \pi_{\theta}(s')$
10:             $y^{\mathrm{target}}[g] = \min\{r[g] + \gamma Q_{\phi}(s',a',g), c_{\mathrm{clip}}\cdot r_{\mathrm{done}}\}$, for all $g \in G$
11:             $\phi \leftarrow \phi - \eta \nabla_{\phi}\sum_{g \in G}\mathcal{L}_{\mathrm{critic}}(Q_{\phi}(\cdot,\cdot,g), y^{\mathrm{target}}[g])$
12:             $Q_{\mathrm{agg}}(\cdot,\cdot) \leftarrow \sum_g w_g Q_{\phi}(\cdot,\cdot,g)$, where $w \in \mathbb{R}^{G} \leftarrow \operatorname{GetWeighting}(d)$
13:             $\theta \leftarrow \theta - \eta \nabla_{\theta}\mathcal{L}_{\mathrm{policy}}(\pi_{\theta} \mid s,a,Q_{\mathrm{agg}})$

Conveniently, VADAR requires minimal modification to standard off-policy RL pipelines, wherein only the critic architecture (conditioned on subgoals) and the corresponding aggregation strategies change. We now walk through Algorithm 2 in detail, highlighting the components unique to VaDAR (shown in color in the pseudocode and below). We give a detailed practitioners guide to using VaDAR in Appendix A. In what follows, we use brackets [g] to denote goal-wise indexing.

Policy update via value weighting. The key innovation in VADAR is optimizing for weighted contributions of different critics during policy extraction, which addresses the myopic behavior described in § 4 and illustrated in Appendix § 12. At each environment step, VADAR evaluates a vector of subgoal completion indicators $\mathbf{d}_t = \{\operatorname{done}_g(s_{t+1})\}_{g \in G} \in\{0,1\}^{G}$. It uses these to computing a weighting GETWEIGHTING (Line 12) for policy extraction:

The policy is then updated to maximize $Q_{\mathrm{agg}}$ via the policy loss $L_{\mathrm{policy}}(\pi_{\theta} \mid s,Q_{\mathrm{agg}})$. Because the policy $\pi_{\theta}: S \to\Delta(A)$ is never conditioned on $g$ (subgoal agnostic), subgoals are not needed at test time. A good weighting function provides strong training signal for completing current goals while improving behavior for future goals.

Weighting choices for sequential goals. As in § 3, first assume that goals are ordered as $1,\ldots,|G|$, with goal $g^i$ serving as a prerequisite for goal $g^{i+1}$, denoted $g^i \prec g^{i+1}$. (see Appendix § 15 for the general case. Given a vector of completion indicators $\mathbf{d}$, recall that $g^{\mathrm{curr}} = \inf\{g : d_g = 0\}$ denotes the current unfinished goal.

We also define the next goal as $g^{\mathrm{next}} := \min\{g^+,g^{|G|}\}$, where $g^+ := \inf\{g' : g' \succ g^{\mathrm{curr}}\}$; again, min/inf is taken with respect to the goal order $\prec$, and $\succeq$ in Table 1 denotes the negation of $\prec$. Table 1 summarizes four choices for the weighting function along two axes. The first axis distinguishes myopic weightings, which optimize only a single goal, from future-aware weightings, which include all relevant future goals. The second axis distinguishes current weightings, which include the current goal, from next weightings, which only optimize for goals after the current goal. These choices are illustrated in Figure 3. Detailed experiments can be found in Section 6.2.

All-Subgoal Critic Learning Importantly, the weighted extraction Eq. (3) requires that the critic values $Q(s,a,g)$ for each goal are well-estimated. Therefore, we design an all-subgoal critic update. Recall that, at each environment step, VADAR evaluates a vector of subgoal completion indicators $\mathbf{d}_t$, and computes a corresponding reward vector $\mathbf{r}_t = r_{\mathrm{sparse}}(\mathbf{d}_t)$ entrywise via the sparse reward function Eq. (3.2), Line 6). This converts a sparse task reward into $|G|$ per-subgoal rewards (Line 10), each of which provides signal as soon as its corresponding subgoal is completed. The tuple $(s_t,a_t,s_{t+1},\mathbf{r}_t,\mathbf{d}_t)$ is stored in the replay buffer. Given a sampled transition $(s,a,s',\mathbf{r},\mathbf{d})$ from the buffer and a new action $a' \sim\pi_{\theta}(s')$, the vector of TD targets is computed entrywise:

where the min is applied entrywise. The critic is updated by a gradient update on the critic loss across all subgoals: $\sum_{g \in G} L_{\mathrm{critic}}(Q_{\phi}(s,a,g) \mid s,a,\mathbf{y}^{\mathrm{target}}_g)$.

Why clipping? The critic update involves a “clipping” of the maximum critic value. This is because, after a subgoal is completed, the agent keeps collecting $r_{\mathrm{done}}$ at every step, so without the clip the Q-value would accumulate to $r_{\mathrm{done}}/(1-\gamma)$. The clip caps the target at $c_{\mathrm{clip}}\cdot r_{\mathrm{done}}$ (Line 10), capping critic values for completed subgoals.

VADAR: Key Design Decisions

• Subgoal-agnostic policy. Subgoal information enters only through the critic at training time; deployed policy $\pi_\theta(a \mid s)$ needs no goal selection/conditioning at inference.

• Weighted aggregation for policy extraction. Per-subgoal Q-values are aggregated into a single scalar signal $Q_{\mathrm{agg}}$; the next-future (VADAR-NF) weighting is our default method which balances current and future subgoal completion.

• Reward clipping at $c_{\mathrm{clip}} \cdot r_{\mathrm{done}}$. Removes the need for critic loss gating on completed subgoals.

VADAR benefits from both dense critic supervision and weighted policy extraction (Section 6.2 and Appendix § 23.§ 1). We provide further details on non-sequential tasks in Appendix § 15, multi-objective optimization in Appendix § 19, and alternative critic parameterizations in Appendix § 10.§ 1. Finally, Appendix § 23 shows that fixed-weight value aggregation is theoretically (but only theoretically) equivalent to reward shaping but we later show (Appendix § 23.§ 1) that this equivalence does not explain VADAR’s empirical success.

Experiments

We evaluate VADAR on a suite of long-horizon manipulation benchmarks. Each method is pretrained via behavior cloning on offline data, stopped early at a low success rate so all methods begin fine-tuning from a comparable starting point, and then fine-tuned online for a fixed number of environment steps. Per-task descriptions and hyperparameters are detailed in Appendix § 16.

Baselines. We study VADAR with two different weighting schemes (Table 1) - VADAR-NF, using next-future (plotted in a solid line) and VADAR-CF, which use current-future (plotted in a dotted line). We compare against 3 alternatives, each using the same base-algorithm per plot: (1) Hierarchical Goal Conditioning (HGC), where both the policy and critic are conditioned on the current subgoal $g$, where $g$ is set to the next incomplete subgoal at every step (Algorithm 1). (2) Implicit Hierarchical Goal Conditioning (iHGC), which retains the goal conditioned critic of HGC but does not condition the policy on the subgoal. (3) Dense Preference Reward, denoted Dense, where we adopt best-practices for a learning dense reward model via preferences (described in Appendix § 13), where the reward models are pretrained with additional rollouts on a richer dataset of successful and failed runs in order to steelman their performance. (4) The original base-algorithm that using sparse reward with no goal information, denoted as “Sparse”.

Config.	Weight $w[g]$	Interpretation
Current-myopic	$\mathbb{I}[g = g^{\mathrm{curr}}]$	Only the current subtask contributes.
Current-future	$\mathbb{I}[g \succeq g^{\mathrm{curr}}]$	All incomplete subtasks contribute.
Next-myopic	$\mathbb{I}[g = g^{\mathrm{next}}]$	Focus on the next subtask, gated by the current subtask.
Next-future	$\mathbb{I}[g \succeq g^{\mathrm{next}}]$	Incomplete subtasks with incomplete parents contribute.

Environments. We evaluate on four benchmark suites spanning different task horizons and difficulties. Robocasa365 (Nasiriany et al., [46]) provides our three longest-horizon tasks, while Robomimic (Mandlekar et al., [41]) tests contact-rich, medium-horizon manipulation on Transport and Tool-Hang. We also evaluate on three multi-stage kitchen tasks from Libero (Liu et al., [38]scene-play-single-task-task4 task from Ogbench ({citet}Park et al., 2025a}). Task descriptions, subgoals, and their dependency structures are given in Appendix § 16.

Base-algorithm instantiations. We instantiate all above baselines with three value-based RL algorithms (details given in Appendix § 11). On the most challenging long-horizon environments (Robocasa365), we use OGPO (Patil et al., [52]), a state-of-the-art offline-to-online RL algorithm for flow-based policies. On the medium-horizon Robomimic tasks, we evaluate on both OGPO and the popular, though less performant FQL (Park et al., [50]), demonstrating that the benefits of VADAR apply to both. Because OGPO is considerably more sample efficient that than FQL from Sparse reward when all BC data are used, we also test OGPO in a regime with limited BC data, denoted low data, on Robomimic; details are described in Table 10. Lastly, we demonstrate VADAR’s applicability to non-generative policies with the seminal Soft Actor Critic (SAC) algorithm (Haarnoja et al., [29]) the simpler on OGBench and Libero environments.

	Transport (low)	Transport (full)	Tool-Hang (low)	Tool-Hang (full)
dataset.proportion	0.1	1.0	0.30	1.0
offline_steps	300K	300K	300K	300K
online_steps	3M	3M	3M	3M
q_warmup_steps	20K	20K	20K	20K
start_training	40K	40K	10K	10K
$\bar{\tau}_{\pi}$	0.005	0.005	0.005	0.005
bc_coeff	0.1	0.1	0.1	0.1
clip_bc_threshold	0.30	0.30	0.30	0.30
success buffer	true	true	true	true
subsample_bon	true	true	true	true

VaDAR outperforms natural baselines

Figure 4 shows that VADAR improves various policy-extraction methods, with the largest gains on the most challenging long-horizon tasks. Not only does VADAR benefit a SOTA method like OGPO, but also enhances weaker methods like FQL and SAC with improved critic learning.

VADAR supplies useful values for aggressive policy extraction. OGPO aggressively optimizes critics using PPO-style policy extraction on the generative policy. We find that VADAR with OGPO matches or outperforms all baselines, with the largest gains on the most challenging long-horizon tasks (Figure 4). On Robocasa365, VADAR is the only method to achieve convergence on LoadDishwasher, while on PreSoakPan it reaches nearly double the success rate of the strongest baseline, iHGC. Similarly, on ArrangeTea, VADAR outperforms baselines with superior training stability. On ROBOMIMIC with OGPO, we find that VADAR, HGC and iHGC obtain similar performance in both the low- and full-data regimes.

The benefits extend to first-order policy extraction. VADAR also improves learning with FQL and SAC, which update the policy through action gradients of the critic. With FQL, VADAR learns faster on ROBOMIMIC TRANSPORT and TOOL-HANG, demonstrating that its benefits do not require OGPO’s policy update. With SAC, the advantage is most apparent on the longer-horizon LIBERO tasks, whilst performance remains comparable to goal-conditioned baselines on OGBENCH. Together, these comparisons support the use of value disaggregation across different policy-extraction methods. Even where performance is comparable, VADAR retains the practical benefit of requiring no goal selection at inference.

Additional baseline comparisons. In Appendix § 14, we compare to two additional algorithm families, both of which significantly underperform VADAR. First, we test hindsight experience replay (HER) (Andrychowicz et al., [4]) on contact-rich LIBERO manipulation, including variants supplied with the same subgoal information as HGC. Second, we compare against reward learning with access to subgoal information, training separate preference-based reward models for each subgoal and aggregating with next-future weighting. VLM-provided subgoal labels. On ROBOCASA365 PRESOAKPAN, VADAR also learns with VLM-provided subgoal labels, reaching similar final success to oracle supervision with only modest effects on sample efficiency (Appendix § 17.§ 1).

Understanding What Enables red!50!black VaDar's Performance

Future-Aware Policy Extraction Improves Performance To isolate the effect of policy extraction, we compare the current-myopic (VADAR-GM), current-future (VADAR-CF), and next-future (VADAR-NF) weightings described in Table 1. These ablations share all-subgoal critic supervision and goal-agnostic policies. Figure 5 shows that VADAR-GM does not outperform HGC and iHGC, while the future-aware variants improve sample efficiency and are the only methods to converge on LOAD DISHWASHER. Hence, dense critic supervision alone is insufficient when used with myopic policy extraction. Next, we test whether future-aware policy extraction also improves HGC while retaining policy goal conditioning. On ROBOCASA365 PRESOAKPAN with OGPO, we introduce three Steelman variants, namely Steelman 1 using current-myopic, Steelman 2 using current-future, and Steelman 3 using next-future, extraction respectively. Figure 6 shows that Steelman 1 remains close to HGC, while Steelman 2 and Steelman 3 approach VADAR-CF and VADAR-NF. These comparisons show that future-aware extraction benefits both policy formulations on this task, allowing VADAR to retain these benefits without requiring subgoal selection at inference.

Variant	Critic supervision	Policy weighting	Policy
HGC (reference)	current subgoal	current-myopic	$\pi(a \mid s,g)$
iHGC (§ 4)	current subgoal	current-myopic	$\pi(a \mid s)$
Steelman 1	all subgoals	current-myopic	$\pi(a \mid s,g)$
Steelman 2	all subgoals	current-future	$\pi(a \mid s,g)$
Steelman 3	all subgoals	next-future	$\pi(a \mid s,g)$
VADAR-GM	all subgoals	current-myopic	$\pi(a \mid s)$
VADAR-CF (ours)	all subgoals	current-future	$\pi(a \mid s)$
VADAR-NF (ours)	all subgoals	next-future	$\pi(a \mid s)$

Dense Subgoal Supervision Improves Critic Learning. To further test whether dense supervision across multiple temporal resolutions improves critic learning, we ask a stricter question: can intermediate subgoal supervision improve the final-task critic even when those intermediate values are never used for policy extraction? We introduce VADARLAST-Q, which trains the same multi-goal critic as VADAR (§ 5) but extracts the policy using only the final-subgoal value $Q_{\phi}(s,a,g_{\lvert G\rvert})$. Thus, VADARLAST-Q uses the same policy objective as the Sparse baseline, while additionally supervising the critic on intermediate subgoals. Figure 7 shows that VADARLAST-Q approaches VADAR-NF on several tasks and substantially outperforms Sparse on the ROBOCASA365 benchmarks. Because the policy objective is unchanged, these gains isolate the effect of denser critic supervision. Full results across all benchmarks and policy-extraction methods are provided in Appendix § 21. This suggests that intermediate subgoal targets improve the critic even when their values are not used directly for policy extraction.

Effect of the policy extraction method. While VADARLAST-Q sometimes matches VADAR-NF, the size of this gap depends on the policy-extraction method. VADARLAST-Q is generally closer to VADAR-NF under OGPO, where the critic-derived signal supervises every denoising step of the generative policy. In FQL, where the critic provides less dense supervision than OGPO, VADARLAST-Q alone is not sufficient, and Next-future weighting provided additional gains.

Further findings. We further compare against VADAR-SHAPED, which aggregates subgoal rewards before the Bellman backup and trains a scalar critic. It fails to learn on the harder ROBOCASA365 tasks and is less sample efficient than VADAR-NF elsewhere (Appendix § 23.§ 1). These and the VADARLAST-Q results suggest that reward shaping alone does not explain the gains from all-subgoal critic supervision. Multi-objective optimization. VADAR also applies when reward components represent distinct objectives. As a proof of concept, Appendix § 19.§ 1 shows how to train separate critics for task completion and obstacle avoidance improving performance and training stability over SAC trained on the aggregated rewards with optimal reward weighting.

Test-Time Steering with a Frozen Policy

We hypothesize that VADAR’s success comes from learning a better critic through denser supervision across multiple temporal resolutions. We test this directly by asking: does the learned critic better identify actions that lead to task success? To isolate critic quality from policy learning, we fix a behavior-cloned policy and update only the critic. Following Q-chunking [35], we use best-of-$N$ steering: a frozen behavior-cloned policy proposes $N$ candidate action chunks, and the critic selects the highest-valued candidate. We compare VADAR-NF and Sparse critics trained on the same demonstrations and frozen-policy rollouts. For this steering experiment, next-future aggregation uses subgoal-completion indicators at inference; the unconditional policy learned in our main experiments requires no such information.

Figure 8 shows that the VADAR critic improves action selection most clearly on ArrangeTea and ToolHang. At the largest critic-training budget with $N = 4$, success rises from 23% to 55% on ArrangeTea and from 32% to 77% on Tool-Hang, compared with 23% and 44% for the Sparse critic. Gains on PreSoakPan are smaller. These results support the benefit of subgoal supervision for critic learning independently of policy updates. Appendix § 20 provides the full protocol, seed variability, candidate-count sweep, and critic visualizations.

Hypothetical Mechanism: Implicit Bellman-Backup via Dense Critic Supervision

The frozen-policy experiments show that subgoal supervision improves action selection independently of policy updates. Together with the VADARLAST-Q results, they motivate a further question: how can intermediate subgoal targets improve value predictions for the final task? We hypothesize that shared representations allow supervision at shorter horizons to inform value prediction at longer horizons. We refer to this proposed mechanism as an implicit Bellman backup. For a fixed policy, the classical Bellman recursion is

With sparse final-task rewards, one-step TD learning propagates the reward signal backward through successive bootstrapped targets. Function approximation can generalize across states, but the final-task reward alone provides little direct supervision for learning representations of intermediate progress.

On the other hand, VADAR supervises $Q_{\phi}(s,a,g)$ for every subgoal $g \in G$ at each sampled transition Eq. (4). These targets span multiple temporal resolutions: a state far from task completion may already be close to an intermediate subgoal. We speculate that predicting these nearer outcomes trains useful features before the final-task value is accurate. Furthermore, because the subgoal predictions share network parameters (Figure 9), features learned from intermediate targets are also available to the final-subgoal prediction $Q_{\phi}(s,a,g^{|G|})$. We hypothesize that this transfer helps the critic learn dependencies across stages of the task that would otherwise have to be learned from final-task rewards alone.

The VADARLAST-Q comparison provides evidence of this hypothesized mechanism: intermediate targets improve learning even though only the final-subgoal value enters policy extraction (Section 6.2). These comparisons do not establish additional Bellman backups within a forward pass or a guaranteed reduction in the number of TD updates. Appendix § 25 develops a related interpretation for iterative critics, which explicitly refine a vector of value estimates over multiple integration steps.

Limitations and Conclusion

This work introduces VADAR, an approach to critic learning that utilizes subtask decomposition, which provides significant improvements across a wide range of policy extraction algorithms as well as challenging environment specifications such as long horizon or safety filtering. Still, a limitation of VADAR is that it defining a fixed DAG structure of subtasks, whereas other methods enable conditioning on a continuum of goal information ([8, 54]). We look forward to generalizing the design principles of VADAR to this continuous or combinatorially large goal spaces, such as those described by images or text, in future work. Moreover, while recent literature has focused on improved methods for policy extraction, particularly with modern expressive policy parameterizations ([56, 52, 50, 19]), VADAR demonstrates that there remains significant untapped potential for algorithmic improvements to critic learning, particularly in long-horizon tasks. Moreover, in light of the recent popularity of scaling test-time computation ([59]), our study suggests a complimentary scaling direction: scaling the density of training signals used to supervise learning, which not only improves learning efficacy, but also reduces the need for complex test-time computation (e.g. VLM queries for goal-selection) at deployment.

Acknowledgments

The authors would like to thank Nidhi Hegde for helping with the initial implementation. MS would like to thank Abhishek Gupta and Andrea Bajcsy for insightful discussions. SS would like to thank Manan Agarwal and Giri Anantharaman for their help with the baseline implementations and Shreyas Kowshik, Michael Chen, Pranav Vignesh, Elizabeth Terveen and other members of CMU MaxLab for the informative discussions and feedback on an earlier draft. This research used the Babel computational cluster resources provided by the School of Computer Science at Carnegie Mellon University. MS and SBP would like to acknowledge support from the Toyota Research Institute (TRI) University 3.0 program.

Practitioner's Guide to red!50!black VaDar

Here we give a short hands-on reference for applying VaDAR to a new task or base algorithm. Subgoal decomposition. Most of the long horizon tasks can be decomposed naturally into subtasks which should be taked as the subgoals for VaDAR. For robotics tasks, we believe that the subgoals defined by us for our environments (Appendix § 16) can serve as good examples for subgoal selections for a new task.

Weighting variant. From our experiments, we found VaDAR-NF to be the most performative variant for robotic manipulation tasks, and recommend it as the default. VaDAR-CF can train slightly faster early on when the first subgoal is hard to reach from BC initialization, at the cost of greater susceptibility to over-exploiting the current goal. The myopic variants are useful as ablations but not as defaults.

Adapting to a new base algorithm. VaDAR is a wrapper over any Value Based RL Algorithm. Given any value-based RL algorithm $\mathcal{A}$, the required modifications are:

1. Change critic output dim from 1 to $|\mathcal{G}|$ (backbone unchanged); ensemble aggregation becomes entrywise.

2. Store per-step completion vectors $\mathbf{d}_t$ and per-head rewards $r_t$ in the replay buffer.

3. Replace the critic loss with its summed-over-heads version; broadcast any $\mathcal{L}_{\mathrm{reg}}$ across heads (see Appendix § 11.§ 1 for more details).

4. Apply the entrywise reward clip in the TD target.

5. In the policy loss, replace $Q_\phi(s,a)$ with $Q_{\mathrm{agg}}(s,a)=\langle w(\mathbf{d}(s)),\langle Q_\phi(s,a,g),g\in\mathcal{G}\rangle\rangle$.

The policy is not conditioned on subgoals. We have applied this recipe unchanged to SAC, FQL, and OGPO (Appendix § 11), and these serve as good references for applying VaDAR to any new Value Based RL Algorithm.

Critic Parameterizations, Ensembling, and Target Networks

This section expands the brief mention in § 3 of critic ensembling and target exponential moving averages (EMAs), which are standard practices in modern off-policy value-based RL and which we adopt without modification in VaDAR. The recipes below apply uniformly across the base instantiations used in our experiments; specific details that interact with the policy update are covered in Appendix § 11.

Critic Parameterizations

Vector-valued multi-goal critic. As an alternative to the goal-conditioned critic $Q_\phi(s,a,g)$, VaDAR can use a multi-goal critic $Q_\phi:\mathcal{S}\times\mathcal{A}\to\mathbb{R}^{\mathcal{G}}$ that maps each state-action pair to a $|\mathcal{G}|$-dimensional vector of Q-values. The $g$-th entry $Q_\phi[g](s,a)$ estimates the value of the state-action pair under the MDP induced by the per-subgoal reward $r_g$. Crucially, all $|\mathcal{G}|$ entries share a single network backbone, enabling representation sharing across subgoals: features learned for predicting one subgoal’s completion are available to all others, Both parameterizations provide dense supervision across subgoals; Appendix A compares them empirically. Appendix § 22 shows the vector-valued critic can be interpreted simply as a structured instantiation of successor features (Barreto et al., [7]), where each feature corresponds to subgoal performance.

Critic Ensembles

Off-policy methods are known to suffer from overestimation bias: maximizing over a noisy approximation of $Q^\pi$ systematically inflates the resulting target ([23, 13]). The standard remedy is to train an ensemble of $M$ critic networks $Q_{\phi_1},\ldots,Q_{\phi_M}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ and aggregate their predictions when forming the TD target. We modify the classical scalar-valued critic in each ensemble member to be a goal-conditioned critic $Q_{\phi_i}:\mathcal{S}\times\mathcal{A}\times\mathcal{G}\to\mathbb{R}$. Concretely, given any aggregator $\operatorname{agg}_M:\mathbb{R}^M\to\mathbb{R}$ (e.g., mean, min), we extend it to act on the $M$ vector critics by applying it independently to each subgoal head: for every $g\in\mathcal{G}$.

We use Eq. (B.1) as the meaning of “entrywise aggregation” throughout the paper, and refer back to it whenever an aggregator is applied to the critic. The ensemble size $M$ is taken to be 10 ($M=10$) by default, unless otherwise specified.

Target Aggregation

The TD target $Q_{\bar{\phi}}(s', a')$ is computed by aggregating the predictions of $M$ target critics $Q_{\bar{\phi}_1}, \ldots, Q_{\bar{\phi}_M}: S \times A \times G \to\mathbb{R}$, whose parameters are an EMA of the online critics (Appendix § 10.§ 4). We use the ensemble mean as the aggregator $\operatorname{agg}_M$ throughout, applied entrywise per Eq. (B.1):

The aggregated scalar $Q_{\mathrm{targ}}(s', a') = \langle w, \langle Q_{\bar{\phi}}(s', a', g), g \in G \rangle\rangle$ used downstream by the policy update (Eq. (3)) is recovered by an inner product with the policy-extraction weights $w$ ($w(s) = \operatorname{GETWEIGHTING}(d(s))$ based on the weighting strategies we described in Table 1). Note that other standard alternatives like the full ensemble minimum, or a randomized two-sample minimum in the style of REDQ (Chen et al. [13]) can also be used for the curtailing overestimation bias, but we found the entrywise mean to work consistently well across all VADAR experiments and adopt it as the default.

Target Network EMAs

To stabilize the moving Bellman target, we maintain a separate set of target critic parameters $\bar{\phi}_1, \ldots, \bar{\phi}_M$ that lag the online parameters via Polyak averaging:

with decay rate $\tau\in(0,1]$ applied at every gradient step. We use $\tau= 0.005$ throughout. When the base algorithm requires a target policy, the same Polyak update is applied to the policy parameters:

Whether a target policy is maintained, and how it is used, is algorithm-specific; see Appendix § 11.

Critic Loss

Combining Eqs. (B.2) and (B.3) with VADAR’s formulation (Eq. (4)) yields the following per-ensemble-member critic loss, summed over subgoal heads:

where $\mathbf{y}^{\mathrm{target}}$ is the clipped vector TD target from Eq. (4).

Base-Algorithm Details: SAC, FQL, and OGPO

We position VADAR as a wrapper around any value-based RL algorithm that replaces each scalar critic with a goal-conditioned critic $Q_{\phi_i}: S \times A \times G \to\mathbb{R}$, evaluated once per subgoal, with a few variations in the $Q_{\mathrm{targ}}$ aggregation that can be plugged into the base algorithm’s policy extraction objective. Here we present the policy losses $L_{\mathrm{policy}}$ and any algorithm-specific implementation details for the three base algorithms used in our experiments: SAC, FQL, and OGPO. § 6.1 pairs each base algorithm with a benchmark suite (SAC with OGBENCH and LIBERO, FQL with ROBOMIMIC, and OGPO with both ROBOMIMIC and ROBOCASA365); coupling each policy extraction method to its correspondingly challenging benchmark(s). The descriptions below are restricted to the components that interact with VADAR.

Soft Actor-Critic (SAC)

SAC (Haarnoja et al. [28]) is a maximum-entropy off-policy actor-critic algorithm that augments the standard return with a per-step entropy bonus $\alpha\mathcal{H}(\pi(\cdot\mid s))$. The optimal policy is the one that maximizes

where $\alpha>0$ is a temperature controlling the entropy-reward trade-off. We use a Gaussian policy with a tanh-squashing transform that bounds actions in $[-1,1]^{\dim A}$ (Haarnoja et al. [28], Eq. (11)): $a_t=\tanh(\mu_{\theta}(s_t)+\sigma_{\theta}(s_t)\odot\epsilon_t)$, with $\epsilon_t\sim\mathcal{N}(0,I)$.

Critic loss. The soft Q-function is trained against an entropy-augmented Bellman target. Plugging into the value-based RL template of § 3 ($L_{\mathrm{critic}}(Q_{\phi}\mid s,a,y^{\mathrm{target}})=(Q_{\phi}(s,a)-y^{\mathrm{target}})^2+L_{\mathrm{reg}}$), the entropy bonus enters through the regularizer in the target:

so that $L_{\mathrm{reg}}(s',a',\pi_{\theta})=-\gamma\alpha\log\pi_{\theta}(a'\mid s')$ in the notation of § 3.

Policy loss. The actor minimizes the KL-divergence between the policy and the (exponentiated) soft Q-function. Using the reparameterization trick (Haarnoja et al. [29]), this reduces to:

where $\operatorname{sg}$ is the stopgrad operator. The expectation is estimated by a single reparameterized sample per state, and the gradient flows through the Squashed Gaussian into both the policy $\mu_{\theta}$ and $\sigma_{\theta}$ heads.

Temperature auto-tuning. We tune $\alpha$ to match a target entropy $\bar{H}=-\dim(A)$ via the auxiliary loss (Haarnoja et al. [29]):

Interaction with VADAR. The scalar critic $Q_{\phi}:S\times A\to\mathbb{R}$ is replaced by the goal-conditioned critic $Q_{\phi}:S\times A\times G\to\mathbb{R}$, which is evaluated once for every subgoal $g\in G$. The Bellman target in Eq. (C.2) is computed for all goals, with the same next action $a'$ and the same entropy regularizer shared across subgoals:

where $Q_{\bar{\phi}}$ is the target critic as defined in Appendix § 10.§ 4, and the per-ensemble-member critic loss is summed over subgoals (Appendix § 10):

The actor sees only the aggregated scalar $Q_{\mathrm{agg}}(s,a)=\sum_{g\in G}w[g]Q_{\phi}(s,a,g)$ of Eq. (3), so the policy loss in Eq. (C.3) becomes

The temperature loss remains unchanged.

Flow Q-Learning (FQL)

FQL (Park et al. [50]) is an off-policy method designed for expressive flow-matching policies. Its key idea is to avoid backpropagating through the multi-step ODE integration of a flow policy by training a separate, expressive one-step policy that is simultaneously (i) regularized toward a frozen behavioral-cloning (BC) flow policy via a distillation loss and (ii) optimized to maximize the critic.

Architecture. FQL maintains three networks: a critic $Q_{\phi}$, a BC flow policy $\mu_{\theta}(s,z)$ that defines a flow-matching velocity field $v_{\theta}(t,s,x^{t})$, and a one-step policy $\mu_{\omega}(s,z):S\times\mathbb{R}^{\dim A}\to A$ that maps Gaussian noise $z\sim\mathcal{N}(0,I)$ directly to an action.

BC flow loss. The BC flow policy is trained with the standard flow-matching objective on the offline dataset $\mathcal{D}$ (Park et al. [50]):

The corresponding stochastic policy $\pi_{\theta}(a\mid s)$ is the push-forward of $\mathcal{N}(0,I)$ under the ODE $\mu_{\theta}(s,\cdot)$ defined by $v_{\theta}$. Crucially, this loss is the only training signal seen by the flow policy: $\mu_{\theta}$ is never updated against $Q_{\phi}$.

Critic loss. The critic is trained with the standard TD residual but using the one-step policy $\mu_{\omega}$ in the bootstrap action (Park et al. [50]):

where $Q_{\mathrm{targ}}=Q_{\bar{\phi}}$ is the EMA target critic (Appendix § 10.§ 4). One-step policy loss. The one-step policy is trained against an actor loss with two terms: a Q-maximization term and a distillation term that pulls $\mu_\omega(s,z)$ toward the BC flow policy’s output $\mu_\theta(s,z)$ at the same noise $z$ [50 Eqs. (7),(9]:

Gradients of the distillation term are stopped at $\mu_\theta$, so the BC flow policy is not affected.

Interaction with VADAR. When applying VADAR on top of FQL, the scalar critic is replaced by the goal-conditioned critic $Q_\phi:\mathcal{S}\times\mathcal{A}\times\mathcal{G}\to\mathbb{R}$, evaluated once for every subgoal $g\in G$. The TD bootstrap is computed for all goals, using a single bootstrap action $\mu_\omega(s',z)$ shared across subgoals:

and the per-ensemble-member critic loss is summed over subgoals:

Similar to before, we define the online aggregate $Q_{\mathrm{agg}}(s,a):=\sum_{g\in G}\mathbf{w}[g]Q_\phi(s,a,g)$ (online critic, matching the FQL convention of Eq. (C.10)). The Q-loss term in Eq. (C.10) then becomes

The distillation term is unchanged. The deployed policy is the one-step $\mu_\omega$, which takes no goal as input and hence requires no goal information at inference (in line with VADAR’s subgoal-agnostic property).

Off-Policy Generative Policy Optimization (OGPO)

OGPO [52] is an off-policy actor-critic for generative control policies (GCPs), like flow matching and diffusion, that side-steps the instabilities of backpropagation through the flow ODE by applying a zero-order PPO-style update to the denoising chain itself. VADAR pairs with OGPO on our most challenging long-horizon benchmarks (ROBOMIMIC, and ROBOCASA365). We summarize the policy loss here and note that OGPO follows the same aforementioned critic ensembling and EMA strategies (Appendix § 10).

Setup. A flow-based GCP samples an action $a_{t,0}$ via a $K$-step denoising chain $a_{t,K}\to a_{t,K-1}\to\cdots\to a_{t,0}$, where each step samples $a_{t,k-1}\sim\pi_\theta(\cdot\mid s_t,a_{t,k})$. The likelihood of a full chain factorizes as

OGPO treats the denoising trajectory $a_{t,K:0}$ as a sequence in an inner “denoising MDP” embedded inside the environment MDP, and applies PPO at the level of this inner MDP only.

Annealed importance sampling ratio. The reference policy is the EMA target $\bar{\pi}_\theta$ (Appendix § 10.§ 4). The policy ratio is the product of per-step ratios over the entire chain:

OGPO Policy Updates OGPO performs policy extraction via a PPO-style zeroth order policy gradient update that considers the entire flow-denoising chain as an MDP. The terminal action $a_{t,0}$ is the only action evaluated against the target critic. OGPO draws a group of $N_{\mathrm{group}}$ denoising trajectories $a_{K:0}^{(i,j)}$ from $\bar{\pi}_\theta(\cdot\mid s^{(i)})$ per state and uses the group mean as a value baseline:

This GRPO-style baseline obviates the need for a separately learned value network.

Policy loss. The PPO-clipped surrogate loss is applied over the entire flow denoising chain as:

where $\epsilon$ is the PPO clip parameter. Because both $\omega_\theta$ and the denoising trajectories can be computed in parallel without the need for environmental interactions, the loss is fully parallelizable across the $N_{\mathrm{batch}} \times N_{\mathrm{group}}$ batch.

Noise injection for flow policies. Evaluating $\omega_\theta$ requires per-step likelihoods $\pi_\theta(a_{t,k-1} \mid s_t, a_{t,k})$ to be nonsingular. Following Zhang et al. [69], we replace the deterministic Euler step with a Gaussian-noised update $a_{t,k-1} \sim\mathcal{N}(v_\theta(a_{t,k}, k/K, s_t), \sigma_k^2 I)$, and retain the marginal path-preserving correction of Albergo et al. [2] so that the per-step SDE marginals match standard ODE flow matching rollouts.

Interaction with VADAR. The critic is the goal-conditioned $Q_\phi: \mathcal{S} \times\mathcal{A} \times\mathcal{G} \to\mathbb{R}$, trained on the all-goal targets $y^{\mathrm{target}}[g] = r[g] + \gamma Q_{\bar{\phi}}(s', a', g)$ for every $g \in G$, exactly as above. VADAR composes with OGPO by replacing the scalar $Q_{\mathrm{targ}}$ in the advantage computation with the aggregate of the target critic evaluated on every subgoal, following Eq. (3):

The same weights $w$ are used for all $N_{\mathrm{group}}$ samples of a given state, since they depend only on that state’s completion vector. The PPO surrogate, group baseline, and noise-injection scheme are otherwise unchanged. For our experiments, unless otherwise stated, we use the OGPO+ variant (due to its superior performance).

Limitations of Explicit and Implicit Hierarchical Goal Conditioning

We examine the limitations summarized in § 4 using implicit hierarchical goal conditioning (iHGC), which preserves goal-conditioned critics but removes goal conditioning from the policy.

Implicit Hierarchical Goal Conditioning

We run HGC using oracle dones, providing perfect subgoal selection for this comparison.

iHGC eliminates the first limitation described above, as now the policy need not incorporate goal information at deployment time. As an immediate example, Figure 10 compares iHGC and HGC on Robomimic Tool-hang with the FQL (Appendix § 11 for more details) policy extraction backbone. Removing goal information from the policy improves final performance and yields a smoother learning curve, suggesting that goal-conditioning the policy can introduce unnecessary instability even when goal-conditioned critics remain useful. We hypothesize that removing goal-conditioning inputs avoids aliasing at goal boundaries.

Both Explicit and Implicit Hierarchical Goal Conditioning Suffer from Myopic Behavior

However, we still find that implicit and explicit hierarchical goal conditioning run into a common failure mode: overly myopic policy exploitation. Both methods update the policy against the critic of the currently active subgoal alone. As a result, the policy is rewarded for completing the current subgoal without accounting for how its manner of completion affects subsequent subtasks. We illustrate this failure mode on ROBOCASA365 ARRANGETEA and PRESOAKPAN in Figure 11.

In Figure 11(a), the policy places the kettle on the tray in a position that obstructs the subsequent mug placement. The “placing the kettle” subgoal is complete, but attempting to place the mug causes a collision that knocks the kettle off the counter. In Figure 11(b), the policy places the pan in the sink in a position that prevents stable placement of the sponge. Both these failure cases are manifestations of over-optimizing for the current subtask (Limitation 2). The policy does not distinguish between merely completing the current subgoal and completing it in a way that facilitates future subgoals. Our method, VADAR, addresses this limitation by optimizing the policy for future-subgoal values (§ 5), providing a learning signal for actions that enable progress toward full-task completion rather than just myopically completing the current subtask.

Dense Preference Reward: Algorithms and Implementation Details

Background and Limitations

An alternative to HGC for solving the credit assignment problem is to craft a denser reward signal $r$. When oracle dense rewards are not available, the most popular data-dependent method has been preference-based dense rewards defined at the level of full trajectories $\tau= (s_1, a_1, \ldots, s_H, a_H)$. Given two trajectories $\tau_1, \tau_2$, one maximizes the Bradley-Terry-Luce (BTL) [9] likelihood, assuming the probability that $\tau_2$ is preferred to $\tau_1$ is given by

where $R_\psi(\tau) = \sum_{s \in\tau} r_\psi(s)$ is cumulative trajectory return, $r_\psi(s)$ is a learned, per-state dense reward, and $\beta$ is a temperature parameter. Preference based rewards have shown strong promise ([68, 55, 63]), but are subject to other limitations: (1) assigning consistent numerical progress values on very long, multi-step tasks is challenging; (2) statistical errors in reward learning can lead to reward hacking; and (3) effective reward learning can require large datasets. Dense rewards are imperfect proxies for desired behavior, and agents may exploit errors in the learned reward rather than complete the task ([3, 32]). For an empirical comparison with value disaggregation, see Appendix § 14.§ 2. We describe our implementation below.

Implementation

We implemented dense preference rewards for the same policy backbones as VADAR: OGPO, SAC, FQL. Our PBRL implementation was heavily inspired by [58, 66, 64]. We used trajectory-wise preferences to train a state-based reward model independently for each task. Trajectory preferences were autogenerated and fit using the Bradley–Terry [9] model. When training the reward model, we uniformly sample from the generated preference dataset. We found that the most reliable preference signal was binary trajectory success. To ensure we used the most accurate preference signal, we experimented with alternatives in Table 3, but found them to be inaccurate.

Signal	Observed issue
$\sum_{s\in\tau} V^{*}(s)$	Uses the value function $V^{*}$ from an optimal policy $\pi^{*}$, but produced noisy preferences because $V^{*}$ gave poor estimates outside the distribution induced by $\pi^{*}$.
$\sum_{t,s_{t}\in\tau} \gamma^{t}r_{\mathrm{sparse}}(s_{t})$	Monte Carlo returns depend on the policy used to generate the trajectory, making the induced reward signal noisy and not policy-invariant at the state level.
$\sum_{r_{\mathrm{dense}}\in\tau} r_{\mathrm{dense}}$	Uses $r_{\mathrm{dense}}$ as defined from the QIPM main algorithm, but it preferred longer trajectories, which introduced a harder and less reliable preference signal for the reward model to learn.

Offline Training PBRL was provided privileged information, a diverse set of trajectories to kick off it’s reward training from, 100 trajectories collected from $(\pi_{\mathrm{EARLY}},\pi_{\mathrm{MID}},\pi_{\mathrm{LATE}})$, where these were checkpoints collected during the training of an optimal policy, $\pi^{*}$. This dataset combined with the original optimal success dataset provided on average 500 trajectories to pre-train the reward model from, with different levels of optimality and a diverse range of failure modes. We saw that this was required for PBRL to show signs of success, as demonstrated in Figure 12. This is a significant setback compared to VADAR, which is able to operate on far more limited datasets.

Online Training After offline reward-model pretraining, we continue updating the dense reward model during online RL following [66, 33]. The reward model is trained using the following iterative loop: the current reward model is used to train the policy, the resulting policy collects new trajectories, these trajectories are converted into additional synthetic preference comparisons, and the reward model is fine-tuned on the offline and online preference dataset.

During online RL, the environment reward is replaced by the learned state-only reward model $\hat{r}_{\theta}(s)$. The underlying RL algorithm is otherwise unchanged: OGPO, SAC, and FQL use their standard actor and critic losses, but with rewards given by $\hat{r}_{\theta}$ instead of the environment reward.

Every $K_{\mathrm{int}}$ environment steps, the reward model is fine-tuned on the union of offline and online preference pairs using the same Bradley–Terry objective as in offline pretraining. The reward model gradually adapts to the on-policy state distribution induced by the improving policy, while still being regularized by the original offline preference data.

After each reward-model update, we relabel the live RL replay buffer with the updated reward model: $r_t \leftarrow\hat{r}_{\theta}(s_t)$. This step makes the online reward update immediately affect subsequent policy learning.

Hyper Parameters Table 4 mentions the primary hyper parameters used to train the model.

Hyperparameter	Value
Reward model
ensemble size	3
model architecture	MLP of (512, 512, 512, 512)
learning rate	$3 \times10^{-4}$
training epochs	50
segment size	episode
Online PBRL schedule
RM update interval	5,000 env steps
maximum online feedback	10,000 preference pairs
feedback buffer	FIFO

Additional Baseline Comparisons

Hindsight experience replay (HER) struggles with contact-rich control

An alternative approach to mitigating the credit assignment problem in goal-conditioned RL is hindsight experience replay (HER) [4], which relabels states reached during a rollout as goals, so that the policy receives learning signal even when the task reward is never observed. HER has largely been evaluated in reaching, pick-and-place and maze settings, where any state is a reasonable goal. To test whether it applies to contact-rich manipulation settings, we evaluate HER with SAC on LIBERO-Task 6, with goals given by object states and relabelled from future states of the same episode, and with the same BC pretraining, online step budget and networks as our other SAC baselines. We also steelman HER with two variants that provide it with the same subgoal information that HGC receives: an oracle that supplies a representative goal state for each subgoal in turn, and a learned goal embedding trained to recognize subgoal completions. In both the variants, we find HER to be incapable of performing multi stage complex manipulation tasks, with the eventual success rate remaining quite close to the initial BC success rate.

TD critic learning outperforms preference-based per-subgoal reward

The Dense baseline in § 6 learns rewards from the full trajectory preference, so it does not receive the same subgoal information as VaDAR. To steelman the comparison, we learn separate preference reward models for each of the subtasks, aggregate them using our Next-Future weighting same as VaDAR-NF, and then train a scalar critic on aggregated reward. We call this method Pref-Subgoal. Figure 13 shows that the Pref-Subgoal underperforms VaDAR-NF on all four tasks. On PreSoakPan, Pref-Subgoal appears to be more sample efficient in the early stages but later fails to reach the same final performance as VaDAR-NF. On ArrangeTea and LoadDishwasher, it does not improve over its BC initialization. With FQL on Tool-Hang, Pref-Subgoal trains more slowly and plateaus below VaDAR-NF.

These results suggest that directly fitting the critic on multiple temporal resolutions is more reliable than learning dense per-subgoal rewards from preferences. We conjecture that small reward-model errors accumulate over the long horizon, while VADAR directly supervises per-subgoal TD targets at multiple temporal resolutions while sharing representations across subgoals. This structure may benefit from the implicit Bellman-backup mechanism discussed in § 6.4.

DAG Extension for Non-Sequential Subtasks

Multiple subgoals may be active simultaneously, as in bimanual robotic manipulation. The value-weighting function in Table 1 extends naturally to this setting: a DAG can express both parallel subgoals and prerequisites that must be completed before a subsequent goal becomes available.

In the sequential setting, subtasks must be completed in a fixed linear order. However, many real-world tasks have more complex structures where multiple subtasks can be pursued simultaneously for optimal behavior. Moreover, some subtasks can form a precondition boundary for others, where policies can be rolled out in the future subtasks only if the preconditions are satisfied. VADAR can naturally be applied to a litany of these subtask dependencies often represented as directed acyclic graphs (DAGs) with each node corresponding to an individual Q-function. This provides the core benefits of value disaggregation to rich dependency structures dominant in long-horizon RL settings.

Illustrative Example: Robomimic Transport. To illustrate this, consider the robomimic Transport task, a multi-robot task where two robot arms must coordinate to transport a payload from one bin to another while managing trash removal. The task decomposes into 7 interdependent subtasks forming a DAG:

1. lid_removed: Lid removed from start bin

2. payload_grasped: Payload grasped from start bin (Robot 0)

3. payload_lifted: Payload lifted above bins (Robot 0)

4. trash_grasped: Trash grasped from target bin (Robot 1)

5. trash_in_bin: Trash placed in trash bin (Robot 1)

6. payload_handed_off: Payload transferred from Robot 0 to Robot 1

7. payload_in_target: Payload placed in target bin (SUCCESS)

The dependencies form two parallel execution paths: (1 → 2 → 3) for payload handling by Robot 0, and (4 → 5) for trash management by Robot 1. The handoff step (6) requires both paths to complete, and the final success (7) depends on the handoff, an illustration is provided in Figure 14.

DAG Formalism. Recall from § 4 that the subgoals $G$ are partially ordered, where $g^{i} \prec g^{j}$ means that $g^{i}$ must be completed before $g^{j}$. We represent this structure with a DAG $\mathcal{D} = (G,E)$, with

This graph is acyclic: a directed cycle would imply $g^{i} \prec g^{i}$ for some subgoal $g^{i}$, contradicting the prerequisite relation. The linear ordering from § 4 is the special case in which the DAG is a chain: every subgoal except the initial one has exactly one parent. We define the following graph functions:

Gate Weights for DAG Structures. For a DAG, the “current” subgoals are the incomplete goals whose prerequisites have all been completed:

We also define the terminal subgoals and the immediate successors of the current frontier as

For $g^{\mathrm{next}}$, we handle the edge case where an element from $g^{\mathrm{curr}}$ is a terminal node, putting those nodes into the set. Using these sets, the DAG value-weighting variants are given in Table 5, provided that we are given $\mathbf{d}$.

Config.	Weight $w_g$	Interpretation
Current-myopic	$\mathbb{I}[g \in G^{\mathrm{curr}}(\mathbf{d})]$	Only currently available subtasks contribute.
Current-future	$\mathbb{I}[g \in\operatorname{desc}(G^{\mathrm{curr}}(\mathbf{d}))]$	Current subtasks and their descendants contribute.
Next-myopic	$\mathbb{I}[g \in G^{\mathrm{next}}(\mathbf{d})]$	Only direct successors of current subtasks contribute.
Next-future	$\mathbb{I}[g \in\operatorname{desc}(G^{\mathrm{next}}(\mathbf{d}))]$	Next subtasks and their descendants contribute.

Environment Details: Per-Task Descriptions: Subgoals, Dones, and DAG Structure

We list the subgoals used by VADAR and HGC on each benchmark task, taken directly from the simulator wrappers in our codebase. For each task we list (i) the high-level objective, (ii) the per-subgoal completion indicators $\operatorname{done}_g(s)$ as evaluated from low-level simulator state (no VLM calls are used for done evaluation), and (iii) the prerequisite structure used to gate subgoal completion. Subgoal completion is evaluated with monotonic completion semantics: once $\operatorname{done}_g$ becomes true and all prerequisites are satisfied, $\operatorname{done}_g$ remains true for the rest of the episode. The default prerequisite structure is a sequential chain ($g^i$ requires $g^1,\ldots,g^{i-1}$); tasks that override this with an explicit DAG are noted below. See Appendix § 15 for an in-depth explanation of the DAG structure.

Robocasa365

Our longest-horizon tasks come from the ROBOCASA365 (Nasiriany et al., [46]) suite. Each task involves multiple sequential phases of long-horizon kitchen manipulation. For our experiments, we use the state-based environments of ROBOCASA365 composite tasks (having multiple subtasks to be done in order to achieve success), which do not involve the movement of the robot’s base. Concretely, we evaluate on the following ROBOCASA365 tasks:

PRESOAKPAN. The agent must place a pan in the sink, place a sponge inside the pan in the sink, and then turn on the faucet. This task is decomposed into 5 sequential subgoals:

1. $g^1 = \texttt{pan\_grasped\_lifted}$: pan (obj1) is grasped and lifted to height $z > 0.88$ m.

2. $g^2 = \texttt{pan\_in\_sink}$: pan is contained inside the sink basin (obj_inside_of(obj1, sink)).

3. $g^3 = \texttt{sponge\_grasped\_lifted}$: sponge (obj2) is grasped and lifted to height $z > 0.88$ m.

4. $g^4 = \texttt{sponge\_in\_pan\_and\_sink}$: sponge is inside the sink, with $xy$-distance to pan $< 0.25$ m and vertical offset $z_{\mathrm{sponge}} - z_{\mathrm{pan}} \in(-0.05, 0.20)$ m.

5. $g^5 = \texttt{water\_on}$: faucet is on, both pan and sponge are in the sink, and the gripper is far from both objects (matches the environment’s original success check).

Final task success is $\operatorname{done}_{g^5}$.

LOADDISHWASHER. The agent must load two dishes (a cup and a bowl) into the dishwasher rack and close the dishwasher. Decomposed into 5 sequential subgoals:

1. $g^1 = \texttt{cup\_grasped}$: cup (dish0) is grasped (check_obj_grasped).

2. $g^2 = \texttt{cup\_in\_dw}$: cup is in contact with the dishwasher rack.

3. $g^3 = \texttt{bowl\_grasped}$: bowl (dish1) is grasped.

4. $g^{4} = \texttt{bowl\_in\_dw}$: bowl is in contact with the dishwasher rack.

5. $g^{5} = \texttt{dw\_closed}$: cup in rack and bowl in rack and dishwasher door closed (angular tolerance 0.05).

Final task success is done$_{g^{5}}$.

ARRANGETEA: This is the third ROBOCASA365 task we evaluate on. In this task, the agent must pick up the kettle and place it on a tray on the kitchen counter, then pick up a mug from inside the kitchen cabinet and then place it next to the kettle on the tray, and then close the hinged cabinet that initially holds the mug. This task is decomposed into 6 natural subgoals:

1. $g^{1} = \texttt{kettle\_grasped}$: kettle is grasped (\texttt{check_obj_grasped(kettle)}).

2. $g^{2} = \texttt{kettle\_on\_tray}$: kettle is contained on the tray (\texttt{check_obj_in_receptacle(kettle, container)}).

3. $g^{3} = \texttt{kettle\_on\_tray \& mug\_grasped}$: kettle still on tray and mug is grasped.

4. $g^{4} = \texttt{kettle\_on\_tray \& mug\_on\_tray}$: kettle still on tray and mug is now on the tray.

5. $g^{5} = \texttt{both\_on\_tray \& right\_door\_closed}$: both objects on tray and the right cabinet door is closed.

6. $g^{6} = \texttt{both\_on\_tray \& cab\_closed}$: both objects on tray and the full hinged cabinet is closed.

Final task success is done$_{g^{6}}$ which matches the original success of the ARRANGETEA task.

Robomimic

ROBOMIMIC (Mandlekar et al. [41]) contributes two contact-rich, medium-horizon tasks.

TRANSPORT. A bimanual task in which two arms coordinate to lift a lid, transport a payload between bins, and dispose of trash. The codebase exposes 7 subgoal indicators; the user-facing description of the DAG is given in Appendix § 15. The 7 subgoal checks are:

1. $g^{1} = \texttt{lid\_removed}$: lid handle has slid leftward by more than 0.08 m from its initial $x$-position ($x_{\mathrm{lid}} < x_{\mathrm{lid}}^{\mathrm{init}} - 0.08$).

2. $g^{2} = \texttt{payload\_grasped}$: Robot 0’s gripper is in contact with the payload geoms (\texttt{_check_grasp}).

3. $g^{3} = \texttt{payload\_lifted}$: payload $z$-position exceeds the table height by $> 0.10$ m.

4. $g^{4} = \texttt{trash\_grasped}$: Robot 1’s gripper is in contact with the trash geoms.

5. $g^{5} = \texttt{trash\_in\_bin}$: \texttt{transport.trash_in_trash_bin} flag is true.

6. $g^{6} = \texttt{payload\_handed\_off}$: Robot 1 is grasping the payload and Robot 0 is not.

7. $g^{7} = \texttt{payload\_in\_target}$: \texttt{transport.payload_in_target_bin} flag is true.

Structure: two parallel chains ($g^{1} \to g^{2} \to g^{3}$) and ($g^{4} \to g^{5}$), both prerequisites for $g^{6}$, which is itself a prerequisite for $g^{7}$ (Figure 14). Final task success is done$_{g^{7}}$.

TOOL-HANG. A single-arm precision task: insert a hook frame into a base stand, then hang a wrench on it. Decomposed into 4 sequential subgoals:

1. $g^{1} = \texttt{frame\_grasped}$: gripper is grasping the frame and the frame is lifted $> 0.02$ m above the table (\texttt{_check_grasp_frame}).

2. $g^{2} = \texttt{frame\_inserted}$: frame is assembled into the stand (\texttt{_check_frame_assembled}).

3. $g^{3} = \texttt{frame\_inserted\_and\_wrench\_grasped}$: frame is still inserted and the wrench is grasped and lifted $> 0.02$ m above the table.

4. $g^{4} = \texttt{wrench\_attached}$: wrench is hung on the inserted frame (\texttt{_check_tool_on_frame}).

The final success is done$_{g^{4}}$ which matches the success of the original TOOL-HANG env.

OGBench

Next we use the scene-play-singletask-task4 (put_in_drawer) from OGBENCH [49]. In this task, the agent needs to first unlock the drawer (by pressing a button), then open the drawer, then lift the cube and place it inside the drawer and finally close the drawer. The subgoal decomposition is as follows:

1. $g^{1} =$press_button_0: button 0 state is set to pressed (_cur_button_states[0] == 1); this unlocks the drawer.

2. $g^{2} =$open_drawer: drawer slide $x$-position $< -0.12$ (slide range $[-0.16, 0]$).

3. $g^{3} =$block_in_drawer: block $xyz$-position satisfies the env’s _is_in_drawer predicate.

4. $g^{4} =$close_drawer: $\lvert x_{\mathrm{slide}}\rvert< 0.04$ (drawer back to nearly fully closed).

Final task success is done$_{g^{4}}$ which is coonsisitent with the origianl success.

Libero

We evaluate VADAR on 3 multi stage kitchen tasks from the LIBERO-10 [38] suite. These are: KITCHEN_SCENE3, KITCHEN_SCENE4, KITCHEN_SCENE6. We use the following subgoal decomposition for these tasks: KITCHEN_SCENE3 (turn on the stove and place the moka pot). BDDL goal: (Turnon flat_stove_1) $\land$ (On moka_pot_1 flat_stove_1_cook_region). Decomposed into 3 sequential subgoals:

1. $g^{1} =$stove_on: (turnon flat_stove_1).

2. $g^{2} =$moka_pot_grasped: gripper is grasping any geom containing moka_pot_1.

3. $g^{3} =$moka_pot_on_stove: (on moka_pot_1 flat_stove_1_cook_region).

Final task success is done$_{g^{3}}$.

KITCHEN_SCENE4 (place black bowl in bottom drawer and close it). The bottom drawer of white_cabinet_1 is open at reset. BDDL goal: (Close white_cabinet_1_bottom_region) $\land$ (In akita_black_bowl_1 white_cabinet_1_bottom_region). Decomposed into 3 sequential subgoals:

1. $g^{1} =$bowl_grasped: gripper is grasping any geom containing akita_black_bowl.

2. $g^{2} =$bowl_in_drawer: (in akita_black_bowl_1 white_cabinet_1_bottom_region).

3. $g^{3} =$drawer_closed: (close white_cabinet_1_bottom_region).

Final task success is done$_{g^{3}}$.

KITCHEN_SCENE6 (place mug in microwave and close it). The microwave is open at reset. BDDL goal: (In white_yellow_mug_1 microwave_1_heating_region) $\land$ (Close microwave_1). Decomposed into 3 sequential subgoals:

1. $g^{1} =$mug_grasped: gripper is grasping any geom containing white_yellow_mug.

2. $g^{2} =$mug_in_microwave: (in white_yellow_mug_1 microwave_1_heating_region).

3. $g^{3} =$microwave_closed: (close microwave_1).

Final task success is done$_{g^{3}}$.

PushT-Obstacle

The multi-objective PUSHT-OBSTACLE task does not use a sequential subgoal decomposition; instead, the two “heads” correspond to two independent per-step signals defined in Appendix § 19:

1. $g^{1} =$task_done: block-goal coverage (intersection-over-union proxy) $\ge0.95$. Ratchets up.

2. $g^{2} =$collision_occurred: continuous proximity cost $c_t \in[0, 1]$ around the obstacle boundary (1 inside, decaying linearly across a 40-px soft margin).

Neither head is a prerequisite for the other, so GETWEIGHTING collapses to a constant weighting $\mathbf{w} = (1, \lambda_{\mathrm{safety}})$.

Suite	Task	$\lvert G\rvert$	Structure
ROBOCASA365	PRESOAKPAN	5	sequential chain
ROBOCASA365	LOADDISHWASHER	5	sequential chain
ROBOCASA365	ARRANGETEA	6	sequential chain
ROBOMIMIC	TRANSPORT	7	DAG, two parallel chains, joint handoff
ROBOMIMIC	TOOL-HANG	4	sequential chain
LIBERO	KITCHEN_SCENE3	3	sequential chain
LIBERO	KITCHEN_SCENE4	3	sequential chain
LIBERO	KITCHEN_SCENE6	3	sequential chain
OGBENCH	scene-play-singletask-task4	4	sequential chain
–	PUSHT-OBSTACLE (multi-objective)	2	independent heads (no prerequisites)

Hyperparameters

VADAR is tested with three off policy value based instantiations. Hyperparameters for each instantiation follows the best practices recommended in the corresponding original papers; we tune only the small set of task-level knobs (training-step budget, discount, BC regularization strength) needed to keep each baseline competitive on the task at hand, and we use the same setting across all baselines (Sparse, Dense, HGC, VADAR) on a given task so that comparisons are not confounded by tuning.

Common Hyperparameters

Most of our hyper parameters remain the same for all are experiments, unless explicitly mentioned. The detailed list of these are listed in Table 8.

Hyperparameter	Value
Optimization
critic learning rate	$3 \times10^{-4}$
actor / BC learning rate	$3 \times10^{-4}$
batch size	256
replay buffer size	$2 \times10^{6}$
Networks
actor MLP hidden dims	(512, 512, 512, 512)
critic MLP hidden dims	(512, 512, 512, 512)
critic ensemble size $M$	10
ensemble aggregation	mean
target EMA $\tau$ (critic)	0.05
target EMA $\bar{\tau}_{\pi}$ (actor)	0.05
VADAR critic
critic shape	$S \times A \to\mathbb{R}^{\lvert G\rvert}$
advantage gating mode	grandparent_future (VADAR-NF) / goal_future (VADAR-CF)
Online schedule
UTD (warmup, online)	(1, 1)

Environment	Max episode length
ROBOCASA365 (all envs: ARRANGETEA, PRESOAKPAN, LOADDISHWASHER)	1200
ROBOMIMIC TRANSPORT	800
ROBOMIMIC TOOL-HANG	1000
LIBERO KITCHEN_SCENE3/4/6	500
OGBENCH scene-play-singletask-task4	100
PUSHT-OBSTACLE	300

OGPO specific hyperparameters

OGPO [52] is our flow-based backbone for the RoboCasa365 and RoboMimic suites. Table 9 lists the OGPO hyperparameters that are held fixed across all five OGPO-backed tasks; the per-task differences are then enumerated below.

Hyperparameter	Value
$h$ (action chunk length)	8
q_warmup_steps	20,000
flow_steps	10
best_of_n	8
critic target EMA $\tau$	0.05

The remaining hyper parameters vary per task. For the RoboCasa365 suite of tasks (PreSoakPan, LoadDishwasher, ArrangeTea), we use the bc data obtained from the official VLA checkpoint (NVIDIA et al., 2025). Since we deal with state-based environments and RoboCasa365 environments change the layout/scenes/fixtures for every environment reset, we fix them across resets to avoid partial observability in state-based settings. For RoboCasa365 environments, we set bc_coeff to 1 and offline_steps to 500, 000. We use around 150 demos collected from the VLA checkpoint. On RoboMimic, we do two sets of experiments: (1) with the full set of the offline demos present (2) reducing the offline demonstration set to a fraction of the original (the dataset.proportion parameter): only this fraction of the offline trajectories is used for BC pretraining. The full list of our hyper parameter settings for RoboMimic experiments on OGPO is present in Table 10.

FQL per-task hyperparameters

FQL [50] is the secondary flow-based backbone we use on RoboMimic Transport and Tool-Hang. Table 11 lists the values used. We use the same network architecture and learning rates as OGPO (i.e., Table 8); We do behavior cloning using all the available data a bit longer, due to FQL being less performative. The detailed list of hyperparameters used for FQL are listed in Table 11.

	TRANSPORT	TOOL-HANG
offline_steps	1M	500K
online_steps	6M	3M
q_warmup_steps	50K	50K
start_training	10K	10K
$h$ (chunk)	8	8
$\gamma$	0.999	0.999
$\bar{\tau}_{\pi}$	0.005	0.005
q_agg	mean	mean
$|G|$, structure	7, DAG	4, chain
group_num_samples	32	32
best_of_n, flow_steps	8, 10	8, 10
$\alpha_{\mathrm{FQL}}$ (distillation coeff.)	1000	1000
bc_coeff	1.0	1.0
clip_bc_threshold	0.45	0.45

Learning with VLM-Provided Subgoal Labels

Goal-conditioned policies are computationally prohibitive with frontier model feedback

For our main experiments, the subgoal completion indicators $\operatorname{done}_{g}$ are evaluated using privileged simulator state. In real-world settings, these intermediate subgoal labels may instead be obtained from a VLM given visual observations and natural-language descriptions of the subgoals [20].

We test whether VADAR can learn when these intermediate training signals are provided by a VLM rather than by the simulator. On ROBOCASA365 PRESOAKPAN, we replace the oracle subgoal completion indicators with predictions from a VLM judge. The environment provides only the final task-success signal, while all intermediate subgoal supervision used by VADAR comes from the VLM. The policy and critic remain state-based, and the VLM is used only to generate training-time subgoal labels. Full details and the judge prompt are provided in Appendix § 18.

Figure 15 compares VADAR-NF trained with oracle and VLM-predicted subgoal completions. VLM supervision is somewhat less sample efficient early in training, but reaches a similar final success rate. These results show that VLMs can serve as training-time evaluators for the subgoal rewards used by VADAR, without introducing any VLM dependence at deployment time, while HGC needs such subgoal completion information during inference time, adding to the cost and latency of querying VLMs during deployment.

VLM Judge Prompt

Protocol. We use gemini-3.6-flash as the VLM judge for the experiment in Appendix § 17.§ 1. For every 8-step action chunk, the judge receives the frames at step offsets [0, 2, 5, 7] from all three cameras, together with the prompt below, and returns a JSON list of $|G|$ booleans, which we use as the subgoal completion vector $\mathbf{d}_{t}$ for the final step of the chunk. The prompt used for ROBOCASA365 PRESOAKPAN is shown below.

VLM judge prompt for ROBOCASA365 PRESOAKPAN

You are a strict robot-manipulation subgoal evaluator.

Task: PreSoakPan

The input is one 8-step action chunk. It contains 4 chronological timestamps at step offsets [0, 2, 5, 7]. Every timestamp has exactly three synchronized camera views. Timestamps are ordered oldest to newest.

Judge whether each subgoal's completion criterion is satisfied at the FINAL timestamp. Use the earlier timestamps as visual history to disambiguate grasping, motion, placement, and occlusion. A transient condition such as 'gripper is holding' must still be true at the final timestamp. Do not infer completion only from the expected task order, and do not require criteria that are not stated below.

Subgoals: 0. pan_grasped_lifted Completion criterion: The gripper is holding the pan and the pan is lifted off the counter. 1. pan_in_sink Completion criterion: The pan is resting inside the sink basin. 2. sponge_grasped_lifted Completion criterion: The gripper is holding the sponge and the sponge is lifted up. 3. sponge_in_pan_and_sink Completion criterion: The sponge is inside the sink and is placed on or in the pan. 4. water_on Completion criterion: The faucet water is running, the pan and sponge are in the sink, and the gripper has moved away from both objects.

Return only strict JSON in this form: {"done":[true,false,...]}. The list must contain exactly 5 booleans in the listed order.

Multi-Objective Optimization on PushT-Obstacle: Full Details

This appendix gives the full environment, algorithm, and training details for the multi-objective experiment in Appendix § 19.§ 1, together with the reward-shaping ablation that controls for the safety signal’s smoothness.

Generalization to multi-objective losses. We observe that VADAR does not require that the disaggregated value functions correspond to rewards associated with subgoals of a long-horizon task. Any collection of independently valuable rewards, such as safety constraints or velocity penalties, can be included in the fitting of the multi-objective critics. We validate the utility of VADAR for such multi-objective problems in Appendix § 19.§ 1.

red!50!black VaDar enables stable multi-objective optimization

Next, we test VADAR’s sample efficient multi-objective optimization capability. Specifically, we consider a setting where a reward function is decomposed into $G$ groups $r = \sum_{g \in G} r^g$. While classical RL uses an single critic for the aggregate reward $r$, we propose VADAR for separate per-reward critics, which are aggregated after TD updates in the policy extraction step. Full reward and environment details are provided in Appendix § 19. Here we present a high level summary.

To test VADAR for multi-objective optimization, we design PUSHT-OBSTACLE, a variant of the popular env PushT [14] where a blue-dot pushes a gray “T” to the green goal configuration. We add a fixed circular obstacle and include both a sparse task reward for pushing the T-block to the goal pose $r_t^{\mathrm{task}}$, and a dense safety reward that penalizes proximity to the obstacle $r_t^{\mathrm{safety}}$. We instantiate VADAR with SAC and with $G = 2$ heads (one per signal) and use a vector-valued multigoal critic $(Q_\phi)$, using a constant weighting for the aggregated critic in policy extraction: $Q_{\mathrm{agg}} = Q_\phi[\mathrm{task}] + \lambda_{\mathrm{safety}} Q_\phi[\mathrm{safety}]$. We compare against standard RL with aggregated reward $r_t = r_t^{\mathrm{task}} + \lambda_{\mathrm{safety}} r_t^{\mathrm{safety}}$ as a baseline (AGGREGATED-SAC). See Appendix § 19 and Appendix § 19.§ 6 for more details on the env and comparison of VADAR and AGGREGATED-SAC on PUSHT-OBSTACLE. Figure 16 shows that VADAR outperforms aggregated rewards, even

when $\lambda_{\mathrm{safety}}$ is chosen to maximize baseline performance. Moreover, we observe added training stability with VADAR safety, compared to AGGREGATED-SAC.

Environment

PUSHT-OBSTACLE extends the standard PushT [15] planar block-pushing task with a static circular obstacle of radius 30 px placed at (320, 220) in the 512$\times$512 workspace. The agent (a circular pusher) must move a T-shaped block to a target pose while avoiding the obstacle. We use the keypoint observation variant (pusht-keypoints-v0) with a 5D state $(x_{\mathrm{agent}}, y_{\mathrm{agent}}, x_{\mathrm{block}}, y_{\mathrm{block}}, \theta_{\mathrm{block}})$ and a 2D continuous action setting the target position of the PD-controlled pusher. Episodes terminate after 300 steps.

Each environment step yields two scalar signals:

• Task signal: a sparse coverage indicator $r^{\mathrm{task}}_t \in\{-1, +5\}$, equal to +5 when the block-goal IoU exceeds 95% and −1 otherwise.

• Safety signal: a continuous soft-margin proximity cost $c_t \in[0, 1]$ that takes value 1 if either the pusher or the block penetrates the obstacle, decays linearly to 0 across a safety margin of 10 px around the obstacle boundary, and is 0 outside the margin. The corresponding per-step safety reward used by the critic is $r^{\mathrm{safety}}_t = -5c_t$.

VaDar Instantiation

We provide the full pseudocode in Algorithm 3. We instantiate VADAR on top of a SAC backbone with $G = 2$: head 0 tracks task completion and head 1 tracks the safety cost. The vector critic $Q_\phi: \mathcal{S} \times\mathcal{A} \to\mathbb{R}^2$ is a 4-layer MLP of width 512 with layer norm, ensembled over 10 members and aggregated by mean. The actor is a tanh-squashed Gaussian over a 4-step action chunk, also a 4-layer MLP of width 512. The TD targets follow Eq. (4): the task head uses the standard reward-clipped VADAR target with cap $c_{\mathrm{clip}} \cdot r_{\mathrm{done}} = 15$, while the safety head uses the backup $-5c_t + \gamma^H Q[\mathrm{safety}](s', a')$ with $\gamma= 0.99$ and chunk length $H = 4$. Because neither subtask is a prerequisite for the other, the policy weighting GETWEIGHTING collapses to a constant $\mathbf{w} = (1, \lambda_{\mathrm{safety}})$, yielding the aggregated objective

with $\lambda_{\mathrm{safety}} = 1.0$ in our experiments. (We also do a sweep over the values of $\lambda_{\mathrm{safety}}$ Appendix § 19.§ 6). Apart from the vector-valued critic and this aggregation, the agent is parameterized as a standard Squashed Gaussian policy with learnable temperature.

Algorithm 3 VADAR for Multi-Objective RL (instantiated on PUSHT-OBSTACLE with $G=2$).

1. Parameters: discount $\gamma$; learning rate $\eta$; safety weight $\lambda_{\mathrm{safety}}$; task-head TD clip $c_{\mathrm{clip}}$.

2. Initialize multi-objective critic $Q_{\phi}:S\times A\to\mathbb{R}^{2}$ with heads {task, safety}; policy $\pi_{\theta}:S\to\Delta(A)$; replay buffer $\mathcal{B}\leftarrow\varnothing$.

3. for episode $=1,2,\ldots$ do

4. &nbsp;&nbsp;while episode not done do

5. &nbsp;&nbsp;&nbsp;&nbsp;$a_t\leftarrow\pi_{\theta}(s_t)$, transition $s_{t+1}\sim P(s_t,a_t)$

6. &nbsp;&nbsp;&nbsp;&nbsp;Read both signals: task indicator $\operatorname{done}[\mathrm{task}]_t\in\{0,1\}$ and continuous proximity cost $c_t\in[0,1]$.

7. &nbsp;&nbsp;&nbsp;&nbsp;Form reward vector $r_t=(r_{\mathrm{sparse}}(\operatorname{done}[\mathrm{task}]_t),-5c_t)$.

8. &nbsp;&nbsp;&nbsp;&nbsp;$\mathcal{B}\leftarrow\mathcal{B}\cup\{(s_t,a_t,s_{t+1},r_t,\operatorname{done}[\mathrm{task}]_t)\}$

9. &nbsp;&nbsp;&nbsp;&nbsp;for number of updates do

10. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sample $(s,a,s',r,\operatorname{done}[\mathrm{task}])\sim\mathcal{B}$, $a'\sim\pi_{\theta}(s')$

11. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$y^{\mathrm{target}}[\mathrm{task}]=\min\{r[\mathrm{task}]+\gamma Q_{\phi}[\mathrm{task}](s',a'),c_{\mathrm{clip}}\cdot r_{\mathrm{done}}\}$ ▷ reward clipping

12. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$y^{\mathrm{target}}[\mathrm{safety}]=r[\mathrm{safety}]+\gamma Q_{\phi}[\mathrm{safety}](s',a')$

13. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\phi\leftarrow\phi-\eta\nabla_{\phi}\sum_{g\in\{\mathrm{task},\mathrm{safety}\}}\mathcal{L}_{\mathrm{critic}}(Q_{\phi}[g]\mid s,a,y^{\mathrm{target},g})$

14. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$Q_{\mathrm{agg}}(\cdot,\cdot)\leftarrow Q_{\phi}[\mathrm{task}](\cdot,\cdot)+\lambda_{\mathrm{safety}}Q_{\phi}[\mathrm{safety}](\cdot,\cdot)$ ▷ constant weighting

15. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L}_{\mathrm{policy}}(\pi_{\theta}\mid s,a,Q_{\mathrm{agg}})$

Baselines

To isolate the contribution of value disaggregation from the contribution of the underlying reward terms, we compare against a dense-reward baseline that shares VADAR’s base algorithm (SAC), critic ensemble, network sizes, BC pretraining, and online step budget, and that consumes the exact same per-step task and safety signals but combines them into a single scalar reward fitted by a scalar critic. We call this AGGREGATED-SAC (Algorithm 4), and it combines the task and safety signals term-for-term as $r_t=r_t^{\mathrm{task}}+\lambda_{\mathrm{safety}}r_t^{\mathrm{safety}}$, with the same $\lambda_{\mathrm{safety}}=1.0$. This is the most direct baseline: it differs from VADAR only in collapsing the two signals before the Bellman backup rather than after.

Algorithm 4 Dense-reward shaping baseline for PUSHT-OBSTACLE (AGGREGATED-SAC).

1. Parameters: discount $\gamma$; learning rate $\eta$; safety weight $\lambda_{\mathrm{safety}}$.

2. Initialize scalar critic $Q_{\phi}:S\times A\to\mathbb{R}$; policy $\pi_{\theta}:S\to\Delta(A)$; replay buffer $\mathcal{B}\leftarrow\varnothing$.

3. for episode $=1,2,\ldots$ do

4. &nbsp;&nbsp;while episode not done do

5. &nbsp;&nbsp;&nbsp;&nbsp;$a_t\leftarrow\pi_{\theta}(s_t)$, transition $s_{t+1}\sim P(s_t,a_t)$

6. &nbsp;&nbsp;&nbsp;&nbsp;Read both signals: task indicator $\operatorname{done}^{\mathrm{task}}_t\in\{0,1\}$ and continuous proximity cost $c_t\in[0,1]$

7. &nbsp;&nbsp;&nbsp;&nbsp;Combine before the Bellman backup into a single scalar reward:

8. &nbsp;&nbsp;&nbsp;&nbsp;$r_t\leftarrow r_{\mathrm{sparse}}(\operatorname{done}^{\mathrm{task}}_t)+\lambda_{\mathrm{safety}}(-5c_t)$ ▷ shared signals with Algorithm 3; same $\lambda_{\mathrm{safety}}$

9. &nbsp;&nbsp;&nbsp;&nbsp;$\mathcal{B}\leftarrow\mathcal{B}\cup\{(s_t,a_t,s_{t+1},r_t)\}$

10. &nbsp;&nbsp;&nbsp;&nbsp;for number of updates do

11. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sample $(s,a,s',r)\sim\mathcal{B}$, $a'\sim\pi_{\theta}(s')$

12. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$y^{\mathrm{target}}=r+\gamma Q_{\phi}(s',a')$ ▷ single scalar TD target

13. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\phi\leftarrow\phi-\eta\nabla_{\phi}\mathcal{L}_{\mathrm{critic}}(Q_{\phi}\mid s,a,y^{\mathrm{target}})$

14. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L}_{\mathrm{policy}}(\pi_{\theta}\mid s,a,Q_{\phi})$

Training and Evaluation Protocol

All methods are pre-trained for 150,000 behavior-cloning steps on a fixed dataset of transitions collected from a previously trained SAC policy on the obstacle-free task, and then fine-tuned online for 800,000 environment steps. We measure the success rate in terms of success: block-goal coverage $\geq95\%$ and no collision with the obstacle occurs the episode.

Parameter Sweep for safety

We sweep over values of $\lambda_{\mathrm{safety}}\in\{0.1,1,5,10\}$ (Figure 18) and observe two trends. First, VADAR is far less sensitive to the choice of $\lambda_{\mathrm{safety}}$: it reaches comparable final success rates at $\lambda=0.1$ and $\lambda=1$, degrades only moderately at $\lambda=5$, and still makes meaningful progress at $\lambda=10$. AGGREGATED-SAC matches VADAR at small $\lambda$ but collapses to near-zero success once $\lambda\geq5$, since the safety penalty starts dominating the aggregate reward. Second, at any fixed $\lambda_{\mathrm{safety}}$, VADAR trains more stably than AGGREGATED-SAC, with narrower seed bands and fewer fluctuations during training. We conjecture that this is due to ease of composition at the Qfunction level: each VADAR head fits a $\lambda$-independent target, while AGGREGATED-SAC absorbs $\lambda$ into the target itself.

Test-Time Policy Steering

This appendix gives the full protocol, critic training details, and results for the test-time steering experiment in § 6.3.

Setup

We evaluate a critic by using it to select among candidate actions of a frozen policy. The policy is the BC-pretrained flow policy of OGPO, trained once and never updated. At every step, the policy proposes $N \in\{1, 4, 16, 64\}$ candidate action chunks of length $h = 8$, the critic scores each candidate, and the highest-valued chunk is executed. With $N = 1$ the critic is not used, so this setting is the unsteered BC policy and is shared across critics and budgets, and we report a single $N = 1$ baseline per task. The Sparse critic is scored by its ensemble mean (Appendix § 10). The VADAR critic is scored by the next-future aggregate $Q_{\mathrm{agg}}(s, a) = \sum_{g \in G} w[g] Q_{\phi}(s, a, g)$ of Eq. (3), where the goal-conditioned critic $Q_{\phi}(s, a, g)$ is evaluated once per subgoal and $\mathbf{w} = \operatorname{GETWEIGHTING}(\mathbf{d}_{t})$ is computed from the environment’s subgoal completion indicators.

Critic Training

Starting from the frozen BC policy, only the critic is trained; the policy is never updated against the critic. The replay buffer is first seeded with 20,000 environment steps of frozen-policy rollouts. The critic is then updated with 4 gradient steps per environment step on batches drawn half from the offline demonstrations and half from frozen-policy rollouts, for up to 300,000 environment steps, and we evaluate checkpoints at 40,000, 100,000, 200,000 and 280,000 steps. The two critics share all hyperparameters (discount, value bounds, normalization and network sizes, Table 8) and differ only in the critic loss. We report the mean and standard deviation over 3 seeds, each evaluated on 96 episodes.

Results

Figure 19 and Table 12 give the full sweep, and we observe three trends. First, steering with the VADAR critic improves with the critic training budget at every $N$ on ARRANGETEA and TOOL-HANG, and at the largest budget on PRESOAKPAN. Second, the Sparse critic, trained on identical data with an identical budget, does not improve on the unsteered policy on ARRANGETEA at any budget or $N$, is flat in both budget and $N$ on PRESOAKPAN, and on TOOL-HANG improves on the unsteered policy at small budgets but plateaus well below the VADAR critic. The only setting in which the Sparse critic is ahead is TOOL-HANG at the smallest budget, and the VADAR critic overtakes it from 100,000 steps onward. Third, steering degrades with large $N$ for both critics, since the maximum over more candidates increasingly selects out-of-distribution actions, and the degradation is milder for the VADAR critic.

Task	Critic budget	Sparse critic $N = 4$	$N = 16$	$N = 64$	VADAR critic $N = 4$	$N = 16$	$N = 64$
ARRANGETEA	$N = 1$				0.23 ± 0.04
	40K	0.17 ± 0.03	0.05 ± 0.04	0.01 ± 0.00	0.20 ± 0.07	0.10 ± 0.06	0.02 ± 0.02
	100K	0.24 ± 0.12	0.11 ± 0.15	0.05 ± 0.08	0.33 ± 0.10	0.23 ± 0.11	0.05 ± 0.05
	200K	0.27 ± 0.08	0.09 ± 0.03	0.05 ± 0.06	0.48 ± 0.07	0.30 ± 0.11	0.16 ± 0.10
	280K	0.23 ± 0.04	0.09 ± 0.01	0.02 ± 0.02	0.55 ± 0.05	0.38 ± 0.05	0.22 ± 0.06
PRESOAKPAN	$N = 1$				0.22 ± 0.03
	40K	0.27 ± 0.02	0.29 ± 0.04	0.26 ± 0.04	0.25	0.28	0.28
	100K	0.29 ± 0.02	0.26 ± 0.05	0.26 ± 0.02	0.30 ± 0.10	0.30 ± 0.12	0.27 ± 0.16
	200K	0.22 ± 0.01	0.27 ± 0.03	0.20 ± 0.02	0.31 ± 0.02	0.31 ± 0.01	0.31 ± 0.04
	280K	0.28 ± 0.04	0.22 ± 0.05	0.25 ± 0.04	0.28 ± 0.05	0.32 ± 0.05	0.36 ± 0.02
TOOL-HANG	$N = 1$				0.32 ± 0.04
	40K	0.42 ± 0.01	0.23 ± 0.04	0.17 ± 0.01	0.25 ± 0.06	0.16 ± 0.05	0.03 ± 0.02
	100K	0.32 ± 0.08	0.21 ± 0.11	0.19 ± 0.10	0.47 ± 0.18	0.21 ± 0.11	0.12 ± 0.01
	200K	0.45 ± 0.12	0.28 ± 0.05	0.24 ± 0.12	0.62 ± 0.10	0.36 ± 0.12	0.21 ± 0.04
	280K	0.44 ± 0.17	0.38 ± 0.08	0.32 ± 0.05	0.77 ± 0.05	0.53 ± 0.15	0.40 ± 0.11

To understand this gap, we roll out the frozen policy and, at each step, score the same 64 candidate action chunks with both critics. Figure 20 shows the two critics at one such step on TOOL-HANG, just before subgoal $g^{2}$ (frame_inserted, Appendix § 16) is completed. The insertion is the step that demands the most precision, and hence one of its most important decision steps. We take the plane spanned by the two principal directions of the 64 candidates, evaluate each critic on a grid of actions in this plane, and plot its contours together with the direction of its action-gradient. As both critics predict returns in the same units, we use the same contour spacing for both, so that a critic with little variation over the plane shows few contours. The VADAR critic varies by an order of magnitude more than the Sparse critic over this plane. The executed chunk is the action taken by the successful trajectory, i.e. the correct action at this step. The VADAR critic identifies it: it assigns its highest values to the candidates near the executed chunk, and its gradient points towards the executed chunk from across the candidate set. The Sparse critic does not: it assigns nearly the same value to all candidates, and its gradient does not point towards the executed chunk. This suggests that VADAR learns

a better critic: one that correctly ranks the actions at the step that decides the task, whilst the Sparse critic trained on the same data cannot tell them apart. We conjecture that this difference in critic quality is what drives VADAR’s advantage in both the online RL experiments, where the critic supervises policy extraction, and the steering experiments, where it ranks candidate actions directly.

Last-$Q$ Extraction: Full Benchmark Results

Figure 21 gives the full comparison accompanying Section 6.2.

Connections to Successor Features and Reward Shaping

VADAR’s aggregation step (Eq. (3)) is structurally similar to a success feature decomposition of a $Q$-function. Classical success features assume reward is linear in a feature map $\phi: S \times A \to\mathbb{R}^{|G|}$ ([16, 7]), $r(s,a) = \langle w,\phi(s,a)\rangle$ for some fixed $w \in\mathbb{R}^{|G|}$. The $Q$-function then inherits the same decomposition $Q^\pi(s,a) = \langle w,\psi^\pi(s,a)\rangle$, where $\psi$ is the successor feature, which represents the expected discounted sum of returns under policy $\pi$. Crucially as motivated by Barreto et al. [7], this allows efficient policy extraction from arbitrary reward weightings without retraining policies across tasks.

VADAR instantiates the reward features $\phi(s,a)$ as per-subgoal rewards $\phi_g(s,a)=r_g(s,a)$ so the successor features are precisely the per-subgoal Q-values $Q_{\phi_g}(s,a)=\psi_g^\pi(s,a)$. The weight vector $\mathbf{w}(s)=\operatorname{GETWEIGHTING}(d(s))$ is a state-dependent current subgoal completion vector. This allows VADAR to recover sufficient statistics for evaluating the entire ancestral tree of weighted objectives in a single multi-step RL training run with $\mathbf{w}(s)$ as the contraction operator.

Fixed-Weight Value Aggregation and Reward Shaping

For fixed weights and true per-goal Q-functions under a common policy, value aggregation is equivalent to evaluating an aggregated reward. Consider family of per-goal MDPs $M_g=(S,A,P_0,P,R_g,\gamma)$ which share common dynamics and discount factor, but differ in their reward $R_g$. For example, in the goal-conditioned setting described in § 4, choosing $R_g(s,a)\leftarrow r_{\mathrm{sparse}}(\operatorname{done}_g(s))$ yields that $M_g$ is the MDP where rewards are associated with subgoal $g$. In principle, however, the theory accomodates any choices of rewards. Finally, we let $Q^{\pi,M_g}$ denote the true Q-function (as described in the preliminaries) associated with $\pi$ under MDP $M_g$.

We now show that the Q-function achieved by aggregating per-goal Q-functions is equivalent to the Q-function of a policy in an MDP whose reward aggregates per-goal rewards:

Proposition O.1. Fix a weighting vector $\mathbf{w}\in\mathbb{R}^{G}$. Then, for any goal-unconditioned policy $\pi:S\to\Delta(A)$, and any $s,a$, we have the equality

where above $R_{\mathbf{w}}:=\sum_g w[g]R_g$ and $M_{\mathbf{w}}$ is the MDP defined by $(S,A,P_0,P,R_{\mathbf{w}},\gamma)$

Proof. Observe that for any MDP $M_g$ with reward $R_g$, we have

where $\mathbb{E}^{\pi}$ is expectation under the dynamics of the MDP and policy $\pi$. As the dynamics are shared between all per-goal MDPs, we may leverage linearity of expectation to commute the expectation with the weighted summation.

This identity applies to fixed weights and exact Q-functions. It does not establish equivalence between VADAR’s state-dependent weighting and clipped, approximate critic updates and learning a scalar critic on an aggregated reward. The experiments in Section 6.2 and Appendix § 23.§ 1 distinguish reward aggregation from the benefits of dense critic supervision.

Reward shaping does not explain Vadar’s success

It might be tempting to attribute VADAR’s success to reward shaping. However, in Section 6.2, we find that even using the final subgoal critic can lead to strong performance in many benchmarks. We conclude that VADAR’s benefits come as much from representation learning induced by denser critic supervision as they do from explicit reward shaping during policy extraction.

Subgoal Reward Shaping Does Not Reliably Enable Long-Horizon Learning

Appendix O establishes that, for fixed weights and true Q-functions under a common policy, a weighted sum of subgoal values equals the value of the correspondingly weighted reward. This identity does not guarantee equivalent learning with state-dependent weights or clipped, approximate critics. This suggests a natural alternative to VADAR: aggregate the per-subgoal rewards before the Bellman backup, and train a standard scalar critic and policy on the resulting shaped reward. We call this variant VADAR-SHAPED.

Figure 22 shows that this equivalence does not translate to learning in practice. VADAR-SHAPED fails to learn on the harder ROBOCASA365 tasks and is substantially less sample efficient than VADAR-NF elsewhere. Since VADAR-SHAPED combines the subgoal rewards before the Bellman update, the critic is trained on single scalar target rather than learning each subgoal value separately. Together with the VADARLAST-Q results, this suggests that VADAR’s gains come from multi-resolution temporal supervision, rather than from reward shaping alone.

Alternative critic parameterizations yield limited additional benefit

An alternative to VADAR’s goal-conditioned critic $Q_{\phi}(s, a, g)$, which is evaluated once per subgoal to form the vector of values used for policy extraction, is a vector-valued critic $Q_{\phi}: S \times A \to\mathbb{R}^{|G|}$ that predicts all subgoal values in a single forward pass. We call this variant VECTOR-VADAR.

Figure 23 shows that it performs roughly the same as VADAR across tasks and weightings. This is consistent with the previous subsections: both critics are supervised on every subgoal at every transition, and it is this dense supervision, together with non-myopic policy extraction, that accounts for VADAR’s performance rather than the particular critic parameterization.

A second alternative is an iterative critic, which refines its subgoal values over several passes and can be viewed as simulating a Bellman update inside the critic (Appendix § 25). We instantiate this with MIQ (Appendix § 25.§ 2) and again find performance on par with VADAR’s monolithic critic, with the iterative critic training somewhat more stably (Appendix § 25.2.5).

Combining VaDAR with Iterative Critics

VaDAR is compatible with critics that estimate values through iterative computation [1]. Here we explore how iterative computation and multi-goal prediction could implement an implicit Bellman update during value prediction. Empirically, iterative critics provide limited additional benefit over the monolithic critic (Appendix § 24).

Simulating the Bellman Update via Iterative Critic Parametrizations

Standard RL critics compute Q-values in a single monolithic forward pass. In the context of VaDAR, a monolithic critic produces all $|G|$ heads in a single shot, so the subgoal heads never communicate at inference time. Here we argue that combining iterative critic parameterizations with VaDAR can be viewed as implementing a Bellman update at inference time.

An iterative VaDAR critic proceeds in steps $j = 0, \ldots, N_{\mathrm{iter}}$, producing an estimates $\hat{q}^{j} \in\mathbb{R}^{G}$ of the critic values per step. At each step, the critic updates its estimate $\hat{q}^{j+1} \leftarrow Q_{\phi}(s, a, \hat{q}^{j}, j)$. Therefore, each updated $q^{k+1}$ can depend on previous estimates of critic values $q^{k}$. Informally, this computation graph can express belief-propagation updates across critic time steps, such as the classical Bellman update [61] which defines the Q-values. We depict this intuition in Figure 24. For training an iterative critic, we introduce a two-step minimal iterative Q-network (MIQ) inspired by Pan et al. [48] described in Appendix § 25.§ 2, which we anecdotally find more stable to train than flow-based critics [1].

The MIQ Critic

VaDAR’s vector critic synergizes with iterative critic parameterizations: each iteration can propagate information across subgoal heads, so the second iteration’s prediction for one subgoal can be conditioned on the first iteration’s prediction for the others. We instantiate this with MIQ, a critic-side adaptation of the Minimal Iterative Policy of Pan et al. [48] that retains the two-step iteration with $t^{\star} = 0.9$.

Architecture

A MIQ critic is a single MLP $Q_{\phi}: \mathcal{S} \times\mathcal{A} \times\mathbb{R} \times[0, 1] \to\mathbb{R}$ that takes a state-action pair $(s, a)$, a scalar $z$ that plays the role of the noise variable in a flow-matching linear interpolant, and a time index $t \in[0, 1]$. To recall the flow-matching analogy: a flow model is trained to map the linear interpolant $I_t = t x + (1-t) z$ between Gaussian noise $z$ and a clean target $x$ at time $t \in[0, 1]$ back to $x$, so that at $t = 0$ the input is pure noise and as $t \to1$ it approaches the target. MIP [48] retains this interpolant but supervises the network with the same target twice, at $t = 0$ and at $t = t^{\star}$ (with $t^{\star} = 0.9$), using a regression loss instead of a velocity-matching loss. MIQ is the same construction with the supervision target $x$ instantiated as the scalar Q-target $y^{\mathrm{target}}$: the flow-matching interpolant is replaced by an interpolant in Q-value space, and $z \sim\operatorname{Unif}[-\sigma,\sigma]$ is the noise variable that lives on the noisy end of that interpolant. At inference, MIQ runs two autoregressive steps:

i.e., the second iteration substitutes its own iteration-zero output $\hat{q}^{0}$ in place of an interpolated input, since $\hat{q}^{0}$ has been trained to approximate $y^{\mathrm{target}}$.

Training Objective

Following Pan et al. [48], Eq. (4.4), MIQ supervises both iterations against the standard scalar TD target $y^{\mathrm{target}} = r + \gamma Q_{\mathrm{target}}(s', a')$. To train the iteration-one mapping, we cannot feed the network the iteration-zero output (it is a moving target during training); instead, we feed a noise-corrupted version of the supervision signal itself. Concretely, we sample $z \sim\operatorname{Unif}[-\sigma,\sigma]$ and form the linear interpolant

which is a noisy copy of $y^{\mathrm{target}}$ (smaller noise the larger $t^{\star}$ is, hence $t^{\star}$ near 1). The MIQ critic loss for a single ensemble member is

The first term trains the iteration-zero pathway to predict $y^{\mathrm{target}}$ from a fully random $z$; the second term trains the iteration-one pathway to denoise a near-correct estimate $z_{t}^{\star}$ back to $y^{\mathrm{target}}$. At inference we substitute the iteration-zero output $\hat{q}^{0}$ for $z_{t}^{\star}$, which is well-behaved because the iteration-zero pathway has already been trained to produce a good approximation of $y^{\mathrm{target}}$.

Ensembling

We use an explicit ensemble of $M$ independently parameterized MIQ networks $Q_{\phi_1},\ldots,Q_{\phi_M}$ (ValueMIPEnsemble in our codebase). At each update, a single noise draw $z$ is shared across the ensemble; each member produces its own $\hat{q}^{0,(i)}$ at iteration zero and uses that member’s own $\hat{q}^{0,(i)}$ as the iteration-one input. Aggregation across the ensemble (Eq. (B.2)) is unchanged from Appendix § 10: we average the iteration-one predictions to form $Q_{\mathrm{targ}}$.

MIQ with VaDAR

We have seen how MIQ can be used as a general critic. Adapting it to our VaDAR framework requires minimal modifications. The scalar $Q_{\phi}:S \times A \times\mathbb{R} \times[0,1] \to\mathbb{R}$ now becomes a vector critic: $Q_{\phi}:S \times A \times\mathbb{R}^{|G|} \times[0,1] \to\mathbb{R}^{|G|}$. The noise variable $z \sim\operatorname{Unif}[-\sigma,\sigma]^{|G|}$ also becomes a $|G|$ dimensional vector. And the TD target is the vector $y^{\mathrm{target}} \in\mathbb{R}^{|G|}$ from Eq. (4), and Eq. (Q.1) becomes

Because the second pass’ input is the full vector $\hat{q}^{0}$, head $g$’s second estimate depends on every head’s previous estimate.

Policy Extraction via MIQ + VaDAR

To validate the hypothesis that iterative critic parameterizations learn implicit Bellman updates over the subgoal DAG, we test MIQ+VaDAR on LIBERO ($|G| = 3$) with SAC and ROBOMIMIC TRANSPORT ($|G| = 7$) with OGPO as the backbone. We compare the VaDAR’s monolithic multi-head critic against MIQ with varying Q-function update-to-data (UTD) ratios.

Results. Across the four panels in Figure 25, success rates with MIQ are broadly comparable to the monolithic critic, with a minor improvement in policy extraction for the higher critic UTD ($q = 4$) ablation.

The more informative differences show up on the critic itself: in Appendix § 25.§ 2 (Figure 26), per-subgoal TD residuals on LIBERO KITCHEN_SCENE6 drop under MIQ across all three subtasks and stay flat throughout training, while the monolithic critic’s TD error drifts upward late in training. The mean-$Q$ trajectory under MIQ is more stable as hypothesized due to the implicit message passing down the subgoal DAG. We posit this being more beneficial to use-cases where critic stability even at subgoal level is a large concern.

This view connects to the VADARLAST-Q ablation of Section 6.2, where policy extraction uses only the final-subgoal value and approaches VADAR-NF on several tasks. As discussed in § 6.4, these results are consistent with intermediate targets improving the shared representations used for final-subgoal prediction. MIQ introduces an explicit dependence between successive value estimates: its second pass conditions every subgoal prediction on the full first-pass vector $\hat{\mathbf{q}}$. The comparison therefore tests the additional benefit of iterative refinement beyond shared subgoal supervision; comparable success rates do not establish that the non-iterative critic performs the same internal computation.