betaFeedback

WorksComputer Science

Horizon-Scalable Q-Learning

Abstract

Learning accurate off-policy state-action value functions (Q-functions) from offline datasets becomes harder as the task horizon grows. One-step temporal-difference (TD) learning bootstraps after every transition, so small per-step errors accumulate along the horizon, and early states can be badly misvalued even when the TD loss is small. Multi-step TD shortens this chain by following the dataset trajectory for several steps; however, since the behavior policy is often suboptimal, every followed action adds a bias, and no fixed number of steps suits a whole trajectory. In this paper, we propose Horizon-Scalable Q-Learning (HQL), a simple method that enables accurate long-horizon value learning by automatically balancing between one-step and multi-step returns. HQL extends a target through a dataset action only when the critic values it at least as highly as the best-of-N actions sampled from the BC policy, and otherwise falls back to a one-step target. We show that this rule is a principled off-policy correction that needs no action densities, which makes HQL compatible with expressive flow and diffusion policies. Experiments show that HQL avoids the failures of both one-step and multi-step TD on a combination-lock task across horizons, and outperforms prior horizon-reduction methods on long-horizon OGBench and OCBench tasks.

Introduction

Off-policy reinforcement learning (RL) learns the state-action value function (Q-function) of a target policy from an offline dataset collected by a different behavior policy, which enables RL to reuse logged experience in real-world domains such as robotics [28, 18] and industrial control [71]. However, off-policy value learning becomes harder as the effective horizon grows [27], i.e., the number of transitions across which the critic must carry value information before reaching a state whose value it already knows. Most off-policy actor-critic methods learn the critic with one-step temporal-difference (TD) backups [12, 26, 23, 67, 45]. A one-step target bootstraps after a single logged transition, which makes it robust to off-policy data but moves value back by only one transition per update. We show that the value error at a state is exactly the sum of the one-step Bellman residuals along the rest of the trajectory, so a critic can reach a small TD loss and still misvalue the start states by an amount that grows with the horizon.

Multi-step returns [62, 54] or Q-chunking [31, 72] shorten this chain by following the dataset trajectory for several steps before bootstrapping. Nevertheless, this imposes a trade-off between fast value propagation (longer backups) and low off-policy bias (shorter backups): we show that each spanned suboptimal action adds a systematic bias that grows toward a limit linear in HH. Since offline trajectories could be mixed with suboptimal actions, no fixed backup length is optimal for every TD update. Per-decision off-policy corrections such as Retrace [50, 37] make this choice at every transition with importance ratios. However, these ratios require policy densities, which flow or diffusion-based policies [45, 34, 29, 70] do not expose.

In this paper, we propose Horizon-Scalable Q-Learning (HQL), which shortens the effective horizon of Q-learning without inheriting the bias of suboptimal dataset actions. HQL achieves this by automatically balancing between one-step and multi-step returns at every transition (Figure 1). More specifically, HQL attaches a binary gate to every logged action, i.e., every action stored in the dataset: if the target critic values it at least as highly as the best-of-NN actions sampled from the BC policy, the gate is open and the target continues through it for one more step; otherwise, the target falls back to a one-step bootstrap. Our key observation is that this gate is principled without any density: for the optimal policy π∝μexp⁡(τQ)\pi\propto\mu\exp(\tau Q) that behavior-regularized RL approximates [49, 48, 69], the behavior density cancels from the importance ratio, and as τ→∞\tau\to\infty the largest gate that preserves QπQ^\pi becomes exactly the comparison of critic values. HQL uses an estimated behavior model only to sample action candidates and never evaluates its density. Since the gate depends only on the ordering of critic values, it is also invariant to the reward scale, and the only hyperparameter in HQL is the number of candidates NN.

Horizon-Scalable Q-Learning diagram

Figure 1. Horizon-Scalable Q-Learning (HQL). Purple arcs show the reach of one training target. One-step TD accumulates residuals, and multi-step TD inherits the bias of the suboptimal action it spans (yellow). HQL compares each logged action with the best-of-NN policy samples, continuing the trace where it passes (c=1c=1) and cutting it where it fails (c=0c=0, dashed), so reward travels far without inheriting off-policy bias.

We evaluate HQL on a combination-lock task, whose true QQ-values are known exactly, and on long-horizon OGBench and OCBench tasks [43] (Park and Levine 2026). On the combination lock, HQL keeps the lowest value estimation error across different horizon, whereas one-step TD accumulates error and nn-step TD fails on corrupted dataset. On OGBench and OCBench, at a matched budget of critic targets, HQL outperforms one-step TD, nn-step TD, Q-chunking [31], and Long-horizon Q-Learning [1], especially on the most challenging long-horizon tasks. Perhaps surprisingly, we find that most of HQL’s targets are one or few-step targets: HQL wins by choosing which targets to extend, not by extending them far.

Related Work

Error propagation in value learning. Error propagation through bootstrapping has been studied for approximate value and policy iteration [36, 10], TD with function approximation [57], and nn-step and TD(λ\lambda) updates [20]; its instability with off-policy data is the deadly triad [58], and the horizon governs the difficulty of evaluation and learning [32, 27].

Multi-step off-policy correction. Multi-step returns and eligibility traces shorten the backup chain [62, 54, 7], and a small fixed nn is standard in deep RL [17, 19, 16]. Per-decision importance sampling [50] and its clipped variants (Retrace [37], Tree[37]), off-policy Q(λ)Q(\lambda) [14], V-trace [8], trajectory-aware traces [6]) correct the mismatch step by step, and emphatic TD reweights updates [55]. Watkins’ Q(λ)Q(\lambda) cuts traces at non-greedy actions while Peng’s Q(λ)Q(\lambda) keeps them [47, 24, 21]; other work adapts the trace [63, 51], avoids ratios via action-dependent bootstrapping [33], gates traces at non-greedy actions [5], or adds multi-step inequalities [1]. However, these corrections need action probabilities or a discrete greedy target policy.

Horizon reduction. Instead of correcting within a backup, horizon-reduction methods change what a decision is. Hierarchical methods act over options or subgoals ([3, 59, 38, 39, 66]), and action chunking treats a short action sequence as one decision ((Seo and Abbeel, 2024; Li et al., 2025b,a; Song et al., 2026)). However, within each segment they use a fixed, uncorrected backup and inherit the bias; HQL is orthogonal and contains both as ungated special cases. Another line exploits the structure of goal-reaching tasks through quasimetric value functions ([60, 73]), graph search over subgoals ([9]), and goal-conditioned hierarchies ([42, 2, 74] (Park et al. 2026)). These methods rely on a goal-reaching reward and thus apply only to goal-conditioned RL; in contrast, HQL makes no assumption on the reward.

Policy extraction with expressive policies. Offline RL methods regularize the policy toward the data, either explicitly through divergence penalties ([64, 25, 65, 11]) or implicitly through in-sample value learning and weighted behavior cloning ([48, 40, 23, 66, 67, 68]). Recent work extracts expressive diffusion or flow policies by value-guided training ([61, 34, 45]), rejection sampling from a behavior model ([4, 13]), adjoint matching ([29]), or value gradient flow ([70]). These methods keep a one-step critic; HQL only changes the critic’s target and can serve any of them.

Preliminaries

We consider the problem presented as a finite-horizon Markov decision process (MDP) M=(S,A,P,r,H)\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, H), where S\mathcal{S} and A\mathcal{A} are the state and action spaces, P(s′∣s,a)P(s' \mid s, a) and r(s,a)r(s, a) denote the transition dynamics and reward function, and HH is the horizon; an episode that reaches its goal early enters a zero-reward absorbing state, so every episode has length HH. For a policy π(a∣s)\pi(a \mid s), the action value is Qπ(s,a)=Eπ[∑tr(st,at)∣s0=s,a0=a]Q^\pi(s, a) = \mathbb{E}_{\pi}[\sum_t r(s_t, a_t) \mid s_0 = s, a_0 = a], with Vπ(s)=Ea∼π(⋅∣s)[Qπ(s,a)]V^\pi(s) = \mathbb{E}_{a \sim\pi(\cdot\mid s)}[Q^\pi(s, a)] and advantage Aπ(s,a)=Qπ(s,a)−Vπ(s)A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s). We consider the off-policy setting, where one needs to estimate QπQ^\pi of a target policy π\pi from a dataset (or replay buffer) D\mathcal{D} collected by a possibly suboptimal behavior policy μ(a∣s)\mu(a \mid s).

Error Accumulation in One-Step and nn-Step TD

Error decomposition along a chain. To understand how critic error scales with the horizon, we derive one error law for one-step and nn-step backups (derivations in Appendices § 8.§ 1–§ 8.§ 2). Consider the deterministic chain s0→s1→⋯→sHs_0 \to s_1 \to\cdots\to s_H induced by a fixed deterministic target policy, with an exactly known terminal value. Write Qˉt=Qˉ(st,at)\bar{Q}_t = \bar{Q}(s_t, a_t), let rtr_t be the reward on the transition out of sts_t, and let Qˉ\bar{Q} be the target critic, with Vˉπ(s)=Ea∼π(⋅∣s)[Qˉ(s,a)]\bar{V}^\pi(s) = \mathbb{E}_{a \sim\pi(\cdot\mid s)}[\bar{Q}(s, a)]. The nn-step target is

Yt(n)=∑j=0n−1rt+j+Vˉπ(st+n),(1)Y_t^{(n)} = \sum_{j=0}^{n-1} r_{t+j} + \bar{V}^\pi(s_{t+n}), \tag*{(1)}

where a true terminal state replaces the last term by its known value. The loss at lookahead nn is LTD(n)=Et[(Qt−sg⁡(Yt(n)))2]L_{\mathrm{TD}}^{(n)} = \mathbb{E}_t[(Q_t - \operatorname{sg}(Y_t^{(n)}))^2], where sg⁡\operatorname{sg} stops gradients ([54]); for n=1n = 1 we write LTDL_{\mathrm{TD}}. With the one-step Bellman residual et=Qt−rt−Qt+1e_t = Q_t - r_t - Q_{t+1} and the value error ϵt=Qt−Qtπ\epsilon_t = Q_t - Q_t^\pi, we have et=ϵt−ϵt+1e_t = \epsilon_t - \epsilon_{t+1} because Qtπ=rt+Qt+1πQ_t^\pi= r_t + Q_{t+1}^\pi, and with an exact terminal value (ϵH=0\epsilon_H = 0) the recursion telescopes:

ϵt=∑τ=tH−1eτ.(2)\epsilon_t = \sum_{\tau=t}^{H-1} e_\tau. \tag*{(2)}

In other words, the value error is exactly the sum of the downstream residuals, for any critic at any point in training.

A unified error law. To see how the residuals add up, we model them as random noise across training runs: each residual has zero mean and variance σ2\sigma^2, and residuals at different steps are only weakly correlated (Appendix § 8.§ 1). Under this model, applying Eq. (2) to nn-step backups, one block of nn steps at a time, gives a single expression for the start-state error (Appendix § 8.§ 2):

RMSE⁡(ϵ0)2≈HnLTD(n)⏟residual accumulation+(1−1n)2H2Aˉμ2⏟behavior-action bias,(3)\operatorname{RMSE}(\epsilon_0)^2 \approx\underbrace{\frac{H}{n}L_{\mathrm{TD}}^{(n)}}_{\text{residual accumulation}} + \underbrace{\left(1-\frac{1}{n}\right)^2 H^2 \bar{A}_\mu^2}_{\text{behavior-action bias}}, \tag*{(3)}

where Aˉμ≤0\bar{A}_{\mu} \le0 is the average advantage of the logged actions that the targets span. Intuitively, the first term decreases with nn while the second increases with nn, so a longer backup trades residual accumulation for behavior-action bias.

One-step TD: residuals accumulate as Θ(H)\Theta(\sqrt{H}). A one-step target spans no logged action, so the bias term vanishes, and summing Eq. (2) under the residual model gives, as H−t→∞H-t \to\infty,

RMSE⁡(ϵt)=E[ϵt2]∼σeffH−t,(4)\operatorname{RMSE}(\epsilon_t) = \sqrt{\mathbb{E}[\epsilon_t^2]} \sim\sigma_{\mathrm{eff}}\sqrt{H-t}, \tag*{(4)}

with σeff2=σ2+2∑ℓ≥1Cov⁡(et,et+ℓ)\sigma_{\mathrm{eff}}^2 = \sigma^2 + 2\sum_{\ell\ge1} \operatorname{Cov}(e_t,e_{t+\ell}) the long-run variance of the residuals. This result is important as it shows that a small training loss does not certify a small value error: since LTD=E[et2]=σ2L_{\mathrm{TD}} = \mathbb{E}[e_t^2] = \sigma^2 measures only the per-step residual, Eq. (3) at n=1n=1 reads

RMSE⁡(ϵ0)=HLTD(5)\operatorname{RMSE}(\epsilon_0) = \sqrt{H L_{\mathrm{TD}}} \tag*{(5)}

up to the factor σeff/σ\sigma_{\mathrm{eff}}/\sigma. Appendix § 8.§ 1 discusses when this rate fails and how a discount γ<1\gamma< 1 caps it at a horizon of order 1/(1−γ)1/(1-\gamma).

nn-step TD: bias grows toward H∣Aˉμ∣H|\bar{A}_{\mu}|. Bootstrapping nn steps ahead divides the residual term by nn. However, along a logged trajectory Yt(n)Y_t^{(n)} now deviates from Qπ(st,at)Q^\pi(s_t,a_t) by ∑j=1n−1Aπ(st+j,at+j)\sum_{j=1}^{n-1} A^\pi(s_{t+j},a_{t+j}) plus the bootstrap error at the endpoint (Eq. (14)). For a near-optimal target policy every spanned advantage is at most zero, so this bias is systematic and does not average out; as nn grows, the square root of the second term of Eq. (3) approaches H∣Aˉμ∣H|\bar{A}_{\mu}|. This indicates that no fixed nn suits all logged actions: the best lookahead nn changes from step to step, which calls for a per-transition rule to minimize the total error.

Horizon-Scalable Q-Learning

In this section, we introduce Horizon-Scalable Q-Learning (HQL). We begin with a gated multi-step target and design the gate based on value comparison (§ 4.1). We then show that this gate is a limiting case of per-decision importance sampling yet needs no action density (§ 4.2). Finally, we provide several practical considerations for HQL and discuss its connection to previous work (§ 4.3).

Gated Multi-Step Targets

Our key idea is to let a backup continue through a logged action only when that action is at least as good as the actions the target policy would take, i.e., its advantage under the target policy is non-negative, which keeps it out of the bias term of Eq. (3) while reducing the horizon of TD learning via multi-step bootstrapping. To do so, we attach a gate ci∈{0,1}c_i \in\{0,1\} to every transition and define the HQL target backward along a logged segment,

YtHQL=rt+γV‾π(st+1)+γct+1(Yt+1HQL−Q‾(st+1,at+1)),(6)Y_t^{\mathrm{HQL}} = r_t + \gamma\overline{V}^{\pi}(s_{t+1}) + \gamma c_{t+1}\left(Y_{t+1}^{\mathrm{HQL}}-\overline{Q}(s_{t+1},a_{t+1})\right), \tag*{(6)}

with the correction set to zero at the end of the segment, and γ\gamma kept for generality. Intuitively, the first term is the usual one-step bootstrap from the target policy, and the second term carries back whatever the logged continuation has learned beyond the critic’s current estimate at (st+1,at+1)(s_{t+1},a_{t+1}). When the gate accepts at+1a_{t+1}, this correction flows back and the target extends one more step; when it rejects at+1a_{t+1}, the target is a plain one-step target, and the states after the cut still start their own targets (as shown in Figure 1, bottom). Unrolling the recursion, a run of accepted actions yields a multi-step target whose length is set by the data rather than fixed in advance. Note that c≡0c \equiv0 recovers one-step TD and c≡1c \equiv1 is the uncorrected multi-step return.

It remains to decide when the gate is open, i.e., whether the target policy would keep the logged action. Intuitively, a logged action is safe to follow if it is at least as good as what the policy itself would propose, which the critic can check by comparing the logged action with candidates drawn from the policy’s own sampler:

ci=I[Q‾(si,ai)≥max⁡1≤m≤NQ‾(si,ai(m))],ai(m)∼μ(⋅∣si),(7)c_i = \mathbb{I}\left[\overline{Q}(s_i,a_i) \ge\max_{1 \le m \le N}\overline{Q}(s_i,a_i^{(m)})\right], \qquad a_i^{(m)} \sim\mu(\cdot\mid s_i), \tag*{(7)}

where μ\mu is the behavior policy that can be estimated using any advanced generative models from D\mathcal{D}. In other words, the logged action is kept if it is at least as good as the best of NN policy proposals. Note that the gate depends only on an ordering of critic values, so it is invariant to the scale of QQ, needs no behavior density, and has NN as its only hyperparameter; it costs NN behavior samples and critic evaluations per transition, the same as the best-of-NN bootstrap already requires.

Algorithm 1 Horizon-Scalable Q-Learning (critic update)

Require: dataset D\mathcal{D}, window length HH, gate width NN, target budget BB

  1. Sample B/HB/H windows (st,at,rt,…,st+H)(s_t, a_t, r_t, \ldots, s_{t+H}) of HH consecutive transitions from D\mathcal{D}

  2. At every position ii: draw ai(1:N)∼μ(⋅∣si)a_i^{(1:N)} \sim\mu(\cdot\mid s_i) and compute the gate cic_i (Eq. (7))

  3. For j=H−1,…,0j = H - 1, \ldots, 0: compute Yt+jHQLY_{t+j}^{\mathrm{HQL}} by Eq. (6), with no correction at j=H−1j = H - 1

  4. Regress all BB pairs (st+j,at+j)(s_{t+j}, a_{t+j}) onto sg⁡(Yt+jHQL)\operatorname{sg}(Y_{t+j}^{\mathrm{HQL}}); update Q‾\overline{Q}

A Density-Free Gate via Per-Decision Importance Sampling

A remaining question is whether cutting multi-step backups with this gate is principled, since cutting changes what a multi-step backup converges to. Per-decision importance sampling answers this [50, 37]: in the tabular setting, any gate ci∈[0,1]c_i \in[0, 1] with ci≤π(ai∣si)/μ(ai∣si)c_i \le\pi(a_i \mid s_i)/\mu(a_i \mid s_i) preserves QπQ^\pi as the fixed point of Eq. (6), and we call such a gate valid. However, evaluating this bound requires the density ratio π/μ\pi/\mu, which is unavailable and hard to estimate [35, 68] when using generative policies, or when policies are implicit, e.g., μ\mu is a flow sampler and π\pi selects among its samples. Fortunately, the ratio has a closed form for the policy that behavior-regularized RL methods approximate, i.e., the exponential tilt of the behavior policy [49, 48, 67, 29],

πθ(a∣s)=μ(a∣s)exp⁡(τQ(s,a))Z(s),Z(s)=Ea∼μ(⋅∣s)[exp⁡(τQ(s,a))],(8)\pi_\theta(a \mid s) = \frac{\mu(a \mid s) \exp(\tau Q(s,a))}{Z(s)}, \qquad Z(s) = \mathbb{E}_{a \sim\mu(\cdot\mid s)}[\exp(\tau Q(s,a))], \tag*{(8)}

so the behavior density cancels and log⁡(π/μ)(ai∣si)=τQ(si,ai)−log⁡Z(si)\log(\pi/\mu)(a_i \mid s_i) = \tau Q(s_i, a_i) - \log Z(s_i). We estimate the normalizer from NN behavior samples as log⁡Z^(si)=LSE⁡m(τQ(si,ai(m)))−log⁡N\log\widehat{Z}(s_i) = \operatorname{LSE}_m(\tau Q(s_i, a_i^{(m)})) - \log N, where LSE⁡m(xm)=log⁡∑m=1Nexp⁡(xm)\operatorname{LSE}_m(x_m) = \log\sum_{m=1}^{N} \exp(x_m) is the log-sum-exp, and clipping the ratio at one gives the largest valid gate

ci=exp⁡(min⁡{0,τQ(si,ai)−log⁡Z^(si)}).(9)c_i = \exp\left(\min\left\{0, \tau Q(s_i,a_i) - \log\widehat{Z}(s_i)\right\}\right). \tag*{(9)}

This soft gate interpolates between two extremes. As τ→0\tau\to0, every gate equals one and the target is an uncorrected multi-step return. As τ→∞\tau\to\infty, the tilt concentrates on the best behavior-supported action, and because LSE⁡m(xm)≤max⁡mxm+log⁡N\operatorname{LSE}_m(x_m) \le\max_m x_m + \log N, the exponent is nonnegative at any τ\tau when no candidate beats the logged action and tends to −∞-\infty otherwise, so the soft gate becomes exactly the value comparison of Eq. (7). Therefore, HQL’s gate is the τ→∞\tau\to\infty limit of a valid gate, which is more scalable and robust since it avoids a temperature and can be computed without density. It also outperforms the soft gate at every τ\tau we tried (Appendix § 8.7).

Practical Considerations

Dense targets over a trajectory window. Since HQL is not restricted to a fixed nn-step backup, it does not need to compute a target only at the first state of a sampled segment. Instead, HQL samples a trajectory window (st,at,rt,…,st+H)(s_t, a_t, r_t, \ldots, s_{t+H}) of HH consecutive transitions and computes the gated multi-step target for every state–action pair in the window. More specifically, it computes the gates cic_i once at every position, evaluates Eq. (6) backward from the window end, and regresses each (st+j,at+j)(s_{t+j}, a_{t+j}) onto its own target Yt+jHQLY_{t+j}^{\mathrm{HQL}}, j=0,…,H−1j = 0, \ldots, H - 1, masking positions beyond an episode boundary (Algorithm 1). By doing so, a window yields HH targets whose maximal reach decreases from HH to 11, and the gate can shorten each of them further. To enable a fair comparison, we hold the number of critic targets per update fixed: with a budget of BB, one-step TD samples BB transitions and HQL samples B/HB/H windows.

Multi-step decisions via Chunked-HQL. So far the decision stride is fixed to 1, meaning each decision corresponds to a single environment step. To allow for more flexible decision-making, we let one decision span several environment steps. More specifically, with stride LL, a decision is a chunk of LL primitive actions that the critic scores as a unit, Q(st,at:t+L)Q(s_t, a_{t:t+L}), and a window of HH chunks reaches L⋅HL \cdot H environment steps while still producing HH targets. Note that the gate is unchanged, now comparing the logged chunk with the best-of-NN chunks drawn from the policy, and the backup of Eq. (6) holds with γL\gamma^L in place of γ\gamma and per-chunk reward sums.

Connections to existing methods. To summarize, in HQL or Chunked-HQL, LL sets how many environment steps one decision covers, and HH sets how many decisions one target may chain (Figure 2). With L=1L=1, H=1H=1 gives one-step TD, an ungated window of size HH gives uncorrected HH-step TD, and a gated window gives HQL. With L=kL=k, H=1H=1 is Q-chunking with chunks of kk actions ([31]), since a single chunk leaves nothing to gate, and H>1H>1 gives Chunked-HQL at stride kk. Note that the two variables extend the maximal reach of a target, L⋅HL \cdot H environment steps, in different ways. Increasing HH lets one target makes more decisions while the policy outputs less actions per step, and the gate decides for each target how many of these decisions it actually uses. In contrast, increasing LL makes each decision cover more environment steps, but each chunk then runs open loop and the policy must search a larger, L⋅dim⁡(A)L \cdot\dim(\mathcal{A})-dimensional action space.

HQL and Chunked-HQL

Figure 2. HQL and Chunked-HQL. HH sets how many decisions one target chains, and LL how many environment steps each decision covers.

Experiments

The goal of our experiments is to answer four questions. (Q1) Does HQL enable better value learning under different horizon lengths and dataset suboptimality? (Q2) How does HQL compare with TD and other baselines on OGBench and OCBench? (Q3) Does HQL work with different policy-extraction methods? (Q4) How do the different components of HQL affect its performance?

Experimental Setup

Combination lock. For Q1, we use a modified combination-lock chain ([44]) with HH linearly ordered states and two actions: the “correct” one advances to the next state and the other returns to the first state, with reward -1 per step and 0 at the terminal state. The offline dataset consists of length-n=64n=64 windows (64 transitions each): for every state–action pair we roll out M=100M=100 windows that start with that pair and then follow the correct action, truncating a window early if it reaches the terminal state. Following Park et al. [44], we construct datasets of nn-step trajectory windows with uniform coverage of starting state–action pairs. To study the effect of increasing action corruption, we independently replace each of the n−1=63n-1=63 subsequent actions with the non-optimal action with probability p∈{0,0.002,0.005,0.011,0.022}p \in\{0, 0.002, 0.005, 0.011, 0.022\}, chosen so that in expectation (1−p)63∈{100,88,73,50,25}%(1-p)^{63} \in\{100, 88, 73, 50, 25\}\% of windows are clean, i.e. contain no corrupted action. We use H∈{512,1024,2048,4096}H \in\{512, 1024, 2048, 4096\}, giving 2HM windows per dataset. Because the dynamics are deterministic, the ground-truth Q is available in closed form (Appendix § 8.§ 3).

OGBench. Q2–Q4 use six state-based OGBench configurations ([43]): cube-double, cube-triple, scene (sparse reward), humanoidmaze-medium, antmaze-giant and humanoidmaze-giant, whose task horizons last 500 to 4000 steps (Table 5); we report the mean score over each configuration’s five tasks. Runs train for 1M offline and then 1M online steps, with 50 evaluation episodes every 10510^5 steps; a run’s score is the mean of its last three evaluations.

Task groupDataset size (transitions)Episode lengthAction dimensionObservation dimensionDefault task
Locomotion (locomaze)
pointmaze-medium1M1000221
pointmaze-large1M1000221
pointmaze-giant1M1000221
pointmaze-teleport1M1000221
antmaze-medium1M10008291
antmaze-large1M10008291
antmaze-giant†1M10008291
antmaze-teleport1M10008291
humanoidmaze-medium†2M200021691
humanoidmaze-large2M200021691
humanoidmaze-giant†4M400021691
antsoccer-arena1M10008424
antsoccer-medium4M10008424
Manipulation (manipspace)
cube-single1M2005282
cube-double†1M5005372
cube-triple†3M10005462
cube-quadruple5M10005552
cube-octuple–15005–2
scene†1M7505402
puzzle-3x31M5005554
puzzle-4x41M5005834
puzzle-4x53M10005992
puzzle-4x6–10005–2

Table 5. OGBench state-based task characteristics. Episode length is the maximum number of environment steps per episode. Rows marked † are the groups used in this work.

OCBench. To test HQL on even longer horizons, we also use OCBench ((Park and Levine, 2026)), a robotic manipulation benchmark whose datasets come from scripted policies with properties similar to human demonstrations. We take eight full-frequency (50Hz) tasks from four domains (block, chamber, switch, hanoi), whose episodes last 1000 to 5000 steps. Following the RL baselines of OCBench, all methods use 25-step action chunks, 8 × 2048 residual MLPs, an HL-Gauss critic and γ=0.9999\gamma=0.9999, with 1024 critic targets per update. Runs train for 1M offline and 1.5M online steps with 500 evaluation episodes every 5×1045 \times10^4 steps (Appendix § 8.§ 6).

Baselines. We compare HQL (H=16H=16) with existing baselines under a matched compute budget: every method receives 512 critic targets per gradient step. We consider three TD baselines. 16-window TD uses the same windows and batch as HQL but removes the gate, so comparing the two isolates the gate’s effect. 8-step TD (n=8n=8 matches the average reach of a 16-step window) and 1-step TD each regress a single target per sequence with a batch of 512. We also compare with Q-chunking (QC) ([31]), which treats a chunk of 8 actions as a single decision and bears no off-policy bias, and Long-horizon Q-Learning (LQL) ([1]), which adds hinge penalties to one-step TD that enforce multi-step inequalities along the logged trajectory. All methods share the same critic architecture and use the same behavior flow policy with best-of-NN sampling.

Q1: A Toy Example with Corrupted Windows

Figure 3 supports both halves of the analysis. On clean data, the error of 1-step TD grows by three orders of magnitude from H=512H=512 to 40964096, while 64-step TD and HQL avoid this accumulation. As the corruption rate increases, 64-step TD degrades sharply, because each target sums the negative advantages of the actions it spans. HQL cuts the trace at those actions: its error stays below both 1-step TD and 64-step TD at every horizon.

Figure 3. Combination lock. HQL remains accurate under data corruption as the horizon grows. The figure illustrates the RMSE between the learned QθQ_\theta and the true Q∗Q^*, averaged uniformly over all 2H2^H state–action pairs of the corrupted combination lock, after 1M1M updates (log scale). Panels vary the corruption fraction of the dataset. n-step TD (n=64n = 64) diverges as soon as any corruption is present; 1-step TD is unaffected by corruption but its error grows sharply with the horizon HH.

Q2: Results on OGBench and OCBench

HQL is the strongest method across all six domains. HQL reaches a mean success rate of 0.74 against 0.49 for the best TD baseline, 8-step TD (Figure 4). Consistent with our analysis in § 3.1, the gap is largest on the four configurations with the largest horizons of 1000 steps or more (Table 5); on cube-triple, HQL reaches 0.54 while both multi-step baselines score 0. 16-window TD matches HQL in everything except the gate, yet it is the weakest method on average, this demonstrates the effectiveness of gating in HQL.

Success-rate curves for HQL and baseline TD methods across six configurations

Figure 4. HQL versus baseline TD methods at matched critic update budget. Success rate averaged over the five tasks of each configuration and over 5 seeds, with standard deviation shown as the band; gray marks the offline phase. HQL outperforms all baselines on all six configurations, especially on the challenging long-horizon ones.

Comparison with other baselines. At the same budget and sampling width (Table 1), HQL improves the mean success rate by 12% over LQL and by 29% over QC. QC is best on cube-double and scene but scores 0% on both humanoid mazes.

HQLChunked-HQLLQLQC
antmaze-giant1 → 87±20 → 86±10 → 44±40 → 46±1
cube-double67 → 87±173 → 100±045 → 83±228 → 100±0
cube-triple12 → 54±17 → 84±155 → 35±31 → 26±3
hmaze-giant10 → 28±10 → 29±212 → 26±10 → 0±0
hmaze-medium56 → 86±186 → 88±248 → 86±10 → 0±0
scene93 → 99±098 → 100±096 → 97±197 → 100±0
mean40 → 74±141 → 81±134 → 62±121 → 45±1

Table 1. HQL and Chunked-HQL against LQL and QC. Success rates (%) are averaged over five tasks and reported as offline → online with standard error over five seeds; we bold the best offline and the best online score at integer level. HQL and Chunked-HQL achieve the best results, especially on the long-horizon tasks.

Although Q-chunking avoids the off-policy bias by having the critic score the whole LL-sized action chunk Q(st,at:t+L)Q(s_t,a_{t:t+L}), it falls short because the critic must choose among L⋅dim⁡(A)L \cdot\dim(\mathcal{A})-dimensional chunks, and a fixed best-of-NN search covers that space increasingly sparsely as the chunk size LL grows. Second, each chunk runs open loop for LL-steps (we choose L=8L = 8 while running QC), so the agent cannot correct its action within a chunk. This also explains why QC performs well on cube-double and scene, whose chunks are low-dimensional (please refer to Table 5 for action dimensions of the different OGBench tasks), but trails on antmaze-giant and fails on both humanoid mazes.

We expect a wider best-of-NN search to help QC, since it covers the chunk space more densely, though at a cost in samples and critic evaluations that grows with NN. The gate is also complementary to chunking: adding it on top of chunked actions (Chunked-HQL) raises QC’s success rate from 45% to 81% (Table 1).

Why does HQL help? One might expect a multi-step method to win by sending reward information far back along the trajectory. To understand why HQL works better than the TD baselines, we log how many steps each target’s trace survives (Figure 6). Perhaps surprisingly, the actual traces are short: 77% of targets are exactly one-step targets while 17% extend to two steps and 6% to three or more steps, and none exceeds six of the sixteen available steps. This indicates that HQL helps not by making targets longer but by choosing which ones to extend: nn-step TD extends all of them and inherits the bias of every action spanned, 1-step TD extends none, and HQL extends a target only when its logged action passes the gate.

Per-target trace length

Figure 6. Per-target trace length with the memoryless prediction (dashed).

Chunked-HQL scales to the longer horizons of OCBench. On OCBench, we compare Chunked-HQL (L=25L=25, H=8H=8, i.e., a 200-step window) with nn-step TD at n∈{100,200}n \in\{100, 200\}, which regresses the chunk critic onto an uncorrected nn-step return and is the backup used by OCBench’s own RL baselines (n=200n=200). Motivated by § 5.5, we decouple the two widths of HQL: the gate uses N=2N=2, while the bootstrap and acting use best-of-16 as in the baselines. Chunked-HQL reaches a mean success rate of 0.82 at 2.5M steps against 0.53 for 100-step TD and 0.46 for 200-step TD (Figure 5). The gap is largest on switch-3x3 (0.91 vs. 0.15), switch-4x4 (0.76 vs. 0.17) and chamber-medium (0.53 vs. 0.12), where the nn-step baselines barely improve or even degrade during online training. Note that 200-step TD is worse than 100-step TD on seven of the eight tasks, which agrees with Eq. (3): a longer uncorrected return sums the negative advantages of more logged actions, while the gate lets HQL extend a target only through chunks that pass the value comparison.

Success-rate curves for Chunked-HQL and n-step TD across eight OCBench tasks

Figure 5. Chunked-HQL versus nn-step TD on OCBench. Success rate on eight 50 Hz tasks (one seed, smoothed over three evaluations; numbers are the mean of the last three). All methods use 25-step chunks and the same best-of-16 bootstrap. Chunked-HQL matches or outperforms both baselines on all eight tasks, especially on switch and chamber-medium.

Q3: Combining HQL with Other Policy Extraction Methods

cube-doublecube-tripleantmz-gianthmz-giant
QAM29 → 501 → 114 → 831 → 0
+ HQL38 → 990 → 800 → 980 → 63
VGF69 → 743 → 251 → 280 → 0
+ HQL80 → 894 → 370 → 900 → 4

Table 2. HQL as a drop-in critic for different policy extraction methods. Success rates (%) on task 2, offline → online; we bold the better score of each pair. HQL improves both QAM and VGF on all four tasks.

HQL with QAM. Q-learning with adjoint matching (QAM) [29] trains a flow policy to sample from the tilted distribution π∝μexp⁡(τQ)\pi\propto\mu\exp(\tau Q). To combine it with HQL, we keep the hard gate unchanged and only replace the best-of-NN policy with the QAM policy. With HQL as its critic, QAM improves on all four tasks at the end of online training, especially on cube-triple (1% → 80%) and humanoidmaze-giant (0% → 63%).

HQL with value gradient flow (VGF). VGF [70] can be thought of as an implicit policy that transports from behavior samples toward high-value actions with a few steps of QQ-gradient ascent. With HQL as its critic, VGF again improves on all four tasks, most on antmaze-giant (28% → 90%).

Q4: Ablations of HQL

Does the window length HH matter? We find the effect is small. At a fixed budget of 512 targets per update, H∈{8,16,32}H \in\{8, 16, 32\} (batch 64/32/16) gives similar performance: 0.607, 0.646, and 0.621 (Figure 7, left). This agrees with the trace length analysis in § 5.3: since empirically no target used more than six steps, a window of 16 already exceeds what the gate uses. HQL is therefore not sensitive to HH unless it is too small.

HQL ablations over window length H and best-of-N gate width N

Figure 7. HQL ablations over window length HH (left) and best-of-NN gate width NN (right). HH matters less than NN, and a smaller NN gives better results.

Does the best-of-NN width matter for HQL? We try best-of-NN width from {4,8,16}\{4, 8, 16\}. On OGBench datasets we find that a smaller NN is better: N∈{4,8,16}N \in\{4, 8, 16\} gives the mean value of 0.596, 0.521, and 0.461, three times the spread of HH (Figure 7, right). Intuitively, a logged action exchangeable with the NN proposals beats all of them with probability 1/(N+1)1/(N+1), so a larger NN accepts fewer logged actions and HQL approaches 1-step TD when NN is large. We provide more details on the ablation setup in Appendix § 8.8.

Conclusion and Limitations

In this paper, we propose HQL, which shortens the effective horizon of Q-learning without inheriting the bias of suboptimal logged actions. HQL gates each logged action with a density-free value comparison and achieves strong results on long-horizon OGBench tasks, both alone and as a drop-in critic for QAM and VGF. We believe that HQL represents a concrete step toward horizon-scalable off-policy RL. One limitation is that the gate relies on the critic’s own ranking. One future work is to apply HQL over subgoals in a hierarchical manner.

Acknowledgement

We thank Claas Voelcker and members from MIDI lab for discussions on the method and feedback on the early draft of the paper. HX is supported by the Amazon Fellowship. This research used the computational cluster resource provided by the Texas Advanced Computing Center at UT Austin.

Appendix

Derivation of the One-Step Error Accumulation

This appendix states the residual model behind Eq. (4) and derives the rate. For a family of chains with growing horizon, assume the one-step residuals et=Qt−rt−Qt+1e_t = Q_t - r_t - Q_{t+1} form a second-order stationary process whose statistics do not depend on HH:

E[et]=0,Var⁡(et)=σ2,C(ℓ)=Cov⁡(et,et+ℓ),(10)\mathbb{E}[e_t] = 0,\qquad\operatorname{Var}(e_t) = \sigma^2,\qquad C(\ell) = \operatorname{Cov}(e_t,e_{t+\ell}), \tag*{(10)}

with weak correlation in the form ∑ℓ=1∞∣C(ℓ)∣<∞\sum_{\ell=1}^{\infty} \lvert C(\ell)\rvert< \infty. Taking the mean and variance of the sum in Eq. (2) with n=H−tn = H-t terms gives

E[ϵt]=0,(11)\mathbb{E}[\epsilon_t] = 0, \tag*{(11)}
Var⁡(ϵt)=nσ2+2∑ℓ=1n−1(n−ℓ)C(ℓ).(12)\operatorname{Var}(\epsilon_t) = n\sigma^2 + 2\sum_{\ell=1}^{n-1}(n-\ell)C(\ell). \tag*{(12)}

By summability of ∣C(ℓ)∣\lvert C(\ell)\rvert, as n→∞n \to\infty,

Var⁡(ϵt)n⟶σeff2≔σ2+2∑ℓ=1∞C(ℓ),(13)\frac{\operatorname{Var}(\epsilon_t)}{n} \longrightarrow\sigma_{\mathrm{eff}}^2 \coloneqq\sigma^2 + 2\sum_{\ell=1}^{\infty} C(\ell), \tag*{(13)}

the long-run variance, which is necessarily nonnegative. If σeff2>0\sigma_{\mathrm{eff}}^2 > 0, then RMSE⁡(ϵt)∼σeffH−t\operatorname{RMSE}(\epsilon_t) \sim\sigma_{\mathrm{eff}}\sqrt{H-t}, which is Eq. (4); this is the nondegenerate random-walk regime.

When the rate does not apply. The random-walk rate is conditional on the residual model and is not a worst case. Negative correlation can give σeff2=0\sigma_{\mathrm{eff}}^2 = 0 and o(H−t)o(\sqrt{H-t}) error, while a systematic same-sign residual of horizon-independent size gives Θ(H)\Theta(H). Such a residual arises, for example, when the critic’s values are shrunk toward zero under all-negative rewards: the semi-gradient fixed point with shrinkage κ\kappa satisfies et=−κQt>0e_t = -\kappa Q_t > 0 at every step. Accumulation also requires persistent residuals: in a fully expressive tabular chain, exact convergence gives Q=QπQ = Q^\pi, et=0e_t = 0, and no accumulation, whereas function approximation, finite optimization, and target-network lag all leave residuals.

Discounting. With γ<1\gamma< 1 the telescoped sum is geometrically weighted, ϵt=∑k=0H−t−1γket+k\epsilon_t = \sum_{k=0}^{H-t-1}\gamma^k e_{t+k}, and for uncorrelated residuals Var⁡(ϵt)=σ2(1−γ2(H−t))/(1−γ2)\operatorname{Var}(\epsilon_t) = \sigma^2(1-\gamma^{2(H-t)})/(1-\gamma^2). The factor grows linearly in H−tH-t until H−tH-t is of order 1/(1−γ)1/(1-\gamma) and saturates at 1/(1−γ2)1/(1-\gamma^2) afterwards, so the discount caps the accumulation at its own effective horizon (≈100\approx100 steps at γ=0.995\gamma= 0.995) rather than removing it; the analysis applies with HH replaced by min⁡{H,1/(1−γ)}\min\{H,1/(1-\gamma)\} up to constants.

Derivation of the nn-Step Error Accumulation

Single-target decomposition. With Aπ(s,a)=Qπ(s,a)−Vπ(s)A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s) and endpoint error ϵV(s)=V‾π(s)−Vπ(s)\epsilon_V(s) = \overline{V}^\pi(s) - V^\pi(s), expanding the target of Eq. (1) over the behavior continuation gives

Eμ[Yt(n)−Qπ(st,at)∣st,at]=∑j=1n−1Eμ[Aπ(st+j,at+j)∣st,at]+Eμ[ϵV(st+n)∣st,at].(14)\mathbb{E}_\mu\left[Y_t^{(n)} - Q^\pi(s_t,a_t)\mid s_t,a_t\right] = \sum_{j=1}^{n-1}\mathbb{E}_\mu\left[A^\pi(s_{t+j},a_{t+j})\mid s_t,a_t\right] + \mathbb{E}_\mu\left[\epsilon_V(s_{t+n})\mid s_t,a_t\right]. \tag*{(14)}

The derivation uses only rt+j+Vπ(st+j+1)=Qπ(st+j,at+j)=Vπ(st+j)+Aπ(st+j,at+j)r_{t+j}+V^\pi(s_{t+j+1}) = Q^\pi(s_{t+j},a_{t+j}) = V^\pi(s_{t+j}) + A^\pi(s_{t+j},a_{t+j}) at each spanned transition (deterministic transitions given the action; the j=0j=0 term contributes no advantage because the target evaluates Qπ(st,at)Q^\pi(s_t,a_t) at the logged action itself). The first term is the behavior-action bias inside the segment, the second the endpoint bootstrap error.

Chaining blocks along the horizon. To turn the per-target decomposition into a start-state statement, partition the chain into H/nH/n blocks of length nn (assume n∣Hn\mid H for clarity) and define the nn-step training residual e~t=Qt−Yt(n)\tilde{e}_t = Q_t - Y_t^{(n)} at each block start. Adding and subtracting QπQ^\pi and applying Eq. (14),

et=e~t+bt(n)+et+n,bt(n)=∑j=1n−1Aπ(st+j,at+j),(15)e_t = \tilde{e}_t + b_t(n) + e_{t+n},\qquad b_t(n) = \sum_{j=1}^{n-1} A^\pi(s_{t+j},a_{t+j}), \tag*{(15)}

where we identify the endpoint state-value error with the action-value error at the next block start (exact on the deterministic evaluation chain). Iterating over blocks with ϵH=0\epsilon_H = 0,

ϵ0=∑j=0H/n−1[e~jn+bjn(n)].(16)\epsilon_0 = \sum_{j=0}^{H/n-1}\left[\tilde{e}_{jn} + b_{jn}(n)\right]. \tag*{(16)}

Model the training residuals e~jn\tilde{e}_{jn} as zero-mean, weakly correlated across blocks with per-block variance σn2=LTD\sigma_n^2 = L_{\mathrm{TD}} (the analogue of the model in Appendix § 8.§ 1); the bias terms bjn(n)b_{jn}(n), by contrast, share a sign for a near-optimal target policy and add coherently. Then

RMSE⁡(ϵ0)2≈Hnσn2+(∑j=0H/n−1bjn(n))2,(17)\operatorname{RMSE}(\epsilon_0)^2 \approx\frac{H}{n}\sigma_n^2 + \left(\sum_{j=0}^{H/n-1} b_{jn}(n)\right)^2, \tag*{(17)}

and since the bias sum covers all logged actions except the H/nH/n block starts, ∑jbjn(n)=(1−1n)HAˉμ\sum_j b_{jn}(n) = (1-\frac{1}{n})H\bar{A}_{\mu} with Aˉμ\bar{A}_{\mu} the average spanned advantage, which is Eq. (3). The result is schematic in the same sense as Appendix § 8.§ 1: it conditions on the residual model, ignores cross terms between residuals and biases, and uses block-aligned targets, but none of these affect the two scalings it exhibits—a H/n\sqrt{H/n} random-walk term and a bias term that grows with nn toward the horizon-linear ceiling H∣Aˉμ∣H|\bar{A}_{\mu}|.

Combination-Lock Dataset

In this didactic setup (Figure 8), the policy π\pi is the deterministic answer policy, and the dynamics are deterministic and acyclic under π\pi, so the action-value function QπQ^\pi has a closed form. We anchor at the terminal state, Vπ(sH)=0V^\pi(s_H)=0. The Bellman equation then reduces to the recursion Vπ(si)=−1+Vπ(si+1)V^\pi(s_i)=-1+V^\pi(s_{i+1}), which unrolls to

Figure 8. The combination lock dataset with linearly arranged H states.

Vπ(si)=−(H−i),i=0,…,H−1.(18)V^\pi(s_i)=-(H-i), \qquad i=0,\ldots,H-1. \tag*{(18)}

Taking the answer action costs one step and advances the chain. Any other action costs one step and resets to the start. Hence

Qπ(si,ai⋆)=−1+Vπ(si+1)=−(H−i),(19)Q^\pi(s_i,a_i^\star)=-1+V^\pi(s_{i+1})=-(H-i), \tag*{(19)}
Qπ(si,aˉi)=−1+Vπ(s0)=−(H+1).(20)Q^\pi(s_i,\bar{a}_i)=-1+V^\pi(s_0)=-(H+1). \tag*{(20)}

OGBench Benchmark: Experiment Details and Per-Task Results

This appendix documents the configuration behind the OGBench benchmark figure of § 5.3 (Figure 4) and reports the per-task curves that its task means aggregate over (Figure 9).

Per-task view of the OGBench benchmark

Figure 9. Per-task view of the OGBench benchmark (Figure 4). Reading down a column shows how one task index behaves across environments, and a row shows how one environment varies over tasks.

Tasks and datasets. We use six environment–discount configurations: cube-double, cube-triple, scene, and humanoidmaze-medium at γ=0.99\gamma=0.99, and antmaze-giant and humanoidmaze-giant at γ=0.995\gamma=0.995; Table 5 lists dataset sizes, episode lengths, and dimensionalities. Every group provides five single-task variants that share one dataset and differ only in the reward relabelling, and we run all five, so each panel of Figure 4 aggregates five tasks. scene uses the sparse reward relabelling.

Training runs. Each run trains for 10610^6 offline gradient steps followed by 10610^6 online steps, evaluating every 10510^5 steps with 50 episodes; the reported score is the mean success rate over the last three evaluations. Each method is run across the five different tasks corresponding to each of the six OGBench environments for five seeds. Curves show the mean over seeds of the task-averaged score, with bands of ±1\pm1 standard deviation across seeds; we average over tasks before computing seed statistics because per-task success is bimodal, so the across-task spread would swamp seed variance. Figure 9 gives the per-task breakdown.

HQL configuration. HQL uses a window of H=16H = 16 with 32 windows per batch, i.e., 32×16=51232 \times16 = 512 critic targets per update under the accounting of § 4.3. The gate is the best-of-NN indicator of Eq. (7) with N=4N = 4, computed from the target critic. The policy is a behavior-cloned flow sampler with best-of-NN action selection (N=4N = 4) used both for acting and for the bootstrap V‾π\overline{V}^{\pi}, so the gate, the bootstrap, and the actor all refer to the same target policy. Table 4 lists the full configuration.

ParameterValue
Window length HH16
Windows per batch32
Critic targets per update512 (=32×16= 32 \times16)
Gate width NN (Eq. (7))4
Gate critictarget network Q‾\overline{Q}
Discount factor0.99 (0.995 for the two giant mazes)
OptimizerAdam (Kingma and Ba, [22])
Learning rate3×10−43 \times10^{-4}
Target network update rate5×10−35 \times10^{-3}
Critic ensemble size2
Target QQ aggregationmean
Network width512
Network depth4
Activation functionGELU (Hendrycks and Gimpel, [15])
Critic layer normYes
Actor layer normNo
Number of flow steps (behavior sampler)10
Best-of-NN action samples (actor and bootstrap)4
Action chunk size1 (no action chunking)
UTD ratio1
Offline training steps10610^{6}
Online training steps10610^{6}
Evaluation interval / episodes10510^{5} / 50
Seeds per cell5 (seeds 0–4)

Table 4. Hyperparameters for the OGBench benchmark experiment (Figure 4 and Figure 9).

Baseline configurations. The three TD baselines share all networks and optimization settings with HQL (Table 4) and differ only in how their 512 critic targets per update are formed (Table 3).

MethodSeq. lengthSeq. per updateTargets per seq.Target
HQL163216gated, every position
16-window TD163216nn-step, every position
8-step TD851218-step, first state
1-step TD151211-step

Table 3. How each method forms its 512 critic targets per update.

Shared Implementation Settings

Unless a later subsection states otherwise, every experiment uses the networks, optimizer and behavior flow sampler of Table 4. All agents are implemented in one JAX codebase with shared replay, evaluation and logging, so two compared methods differ only in the settings named for that comparison.

Matched budget. All comparisons in § 5.3 and the window-length ablation of § 5.5 fix the number of critic targets per update at 512, following the accounting of Park et al. [44]: a batch of BB windows of length HH yields B⋅HB \cdot H targets for methods that regress every position, and BB targets for methods that regress only the first state. The number of transitions read per update therefore differs across methods; for example, 8-step TD reads 4096 transitions to form its 512 targets.

Compute. Runs of 2×1062 \times10^{6} steps were executed on 8-GPU A100 and H100 nodes, eight runs per node, at about 150 iterations/s per run (≈4\approx4–6 h per run).

OCBench Benchmark: Experiment Details

This appendix documents the configuration behind the OCBench results of § 5.3 (Figure 5). Unlike the OGBench experiments, we follow the RL baselines of OCBench (Park and Levine, 2026) for every setting other than the critic target, so all three methods share the same networks, critic loss, action space and action chunks (Table 6).

ParameterValue
Action chunk size LL25 (cut at episode boundaries)
Window length HH (Chunked-HQL)8 chunks (200 environment steps)
Windows per batch (Chunked-HQL)128
Critic targets per update1024 (= 128 ×\times 8)
Batch size of nn-step TD1024 (one target per sequence)
nn of nn-step TD100, 200
Gate width NN (Eq. (7))2
Gate critictarget network Q‾\overline{Q}
Best-of-NN (actor and bootstrap)16
Discount factor0.9999
OptimizerAdam [22]
Learning rate10−410^{-4}
Target network update rate10−310^{-3}
Critic ensemble size2
Target QQ aggregationmean
Network (actor and critic)residual MLP, 8 ×\times 2048
Activation functionGELU [15]
Critic layer normYes
Actor layer normYes
Critic lossHL-Gauss, 101 bins on [0, 1], σ=0.75\sigma= 0.75 bins
Number of flow steps (behavior sampler)10
Flow-time distributionBeta(1, 1.5)
Action normalizationpercentiles 1/99 ↦±1\mapsto\pm1, clip at ±5\pm5
UTD ratio1
Offline training steps10610^{6}
Online training steps1.5×1061.5 \times10^{6}
Evaluation interval / episodes5×1045 \times10^{4} / 500
Seeds per cell1 (seed 0)

Table 6. Hyperparameters for the OCBench benchmark experiment (Figure 5).

Tasks and datasets. We use eight full-frequency (50 Hz) tasks, block-single, block-double, chamber-easy, chamber-medium, switch-3x3, switch-4x4, hanoi-single and hanoi-double (all task1), whose episode limits range from 1000 to 5000 steps. We train on all episodes of each dataset without filtering for successful ones. Actions are normalized as in OCBench, i.e., the 1st and 99th percentiles of each action dimension are mapped to ±1\pm1 and the result is clipped at ±5\pm5.

Method configurations. Chunked-HQL uses chunks of L=25L = 25 actions and windows of H=8H = 8 chunks (200 environment steps) with 128 windows per batch, i.e., 128×8=1024128 \times8 = 1024 critic targets per update. Its gate compares the logged chunk with the best of N=2N = 2 policy chunks, while the bootstrap and acting take the best of 16 chunks, the same width as the baselines. The nn-step TD baselines (n∈{100,200}n \in\{100, 200\}) regress a single target per sequence with a batch of 1024. Each run is evaluated with 500 episodes every 5×1045 \times10^{4} steps, and the reported score is the mean success rate over the last three evaluations.

Soft Gate versus Hard Gate

§ 4.2 shows that the hard gate of Eq. (7) is the τ→∞\tau\to\infty limit of the soft gate of Eq. (9). A natural question is whether a finite τ\tau works better, since the soft gate keeps partial credit for logged actions that are close to the best candidate. To answer this, we compare the soft gate at τ∈{0,0.5,1,3,30}\tau\in\{0, 0.5, 1, 3, 30\} with HQL’s hard gate on six OGBench configurations with five tasks each. Note that τ=0\tau=0 sets every gate to one and recovers uncorrected 8-step TD. All runs use H=8H=8, a batch of 128 windows (1024 critic targets per update), 1M offline and 1M online steps, and one seed; the soft gate estimates log Z^\hat{Z} and acts with 16 behavior samples, and the hard gate uses N=4N=4 for both.

Figure 10 and Table 7 show that the hard gate achieves the best mean success rate (61%), and that the soft gate improves steadily as τ\tau increases (36% → 57%), i.e., as it approaches the hard gate. The gap is largest on the long-horizon mazes: on humanoidmaze-medium the hard gate reaches 91% against 77% for the best soft gate, and on antmaze-giant 65% against 58%. The soft gate is only better on cube-double at τ=1\tau=1, while scene and puzzle-3x3 end at the same value for every gate. This indicates that the hard gate is not only the limit that removes τ\tau as a hyperparameter, but also the strongest member of the family: a finite τ\tau keeps partial credit for suboptimal logged actions, so the multi-step target inherits part of their bias, which is exactly the failure mode of uncorrected nn-step TD.

Soft gate at different temperatures versus HQL’s hard gate

Figure 10. Soft gate at different temperatures versus HQL’s hard gate. Success rate averaged over the five tasks of each configuration, one seed; gray marks the offline phase. The soft gate improves as τ\tau grows toward the hard gate, and the hard gate is the best on average, especially on the long-horizon mazes.

soft gatehard gate
τ=0\tau=0τ=0.5\tau=0.5τ=1\tau=1τ=3\tau=3τ=30\tau=30(HQL)
cube-double758187848081
cube-triple163739464448
scene596059606060
puzzle-3x3212020202020
antmaze-giant241124365865
hmaze-medium193760727791
mean364148535761

Table 7. Soft gate versus hard gate. Success rate (%) at the end of online training, mean of the last three evaluations over five tasks, one seed. We bold the best score in each row at integer level.

Ablation Details

Both panels of Figure 7 use HQL with the shared settings of Appendix § 8.§ 5. Each panel averages only over the environment–task cells that were run for every value of the ablated setting, so that only that setting varies within a panel.

Window length HH. We fix the budget at 512 critic targets per update, so the number of windows per batch decreases as HH increases (64, 32 and 16 for H=8H=8, 16 and 32 respectively). The panel covers six environment–discount configurations with five tasks each, i.e., 30 cells.

Gate width NN. We run this panel at H=8H=8 with 128 windows per batch, i.e., 1024 critic targets per update, with two seeds per setting (we use 512 critic targets everywhere else in the paper). Here NN sets both the number of candidates in the gate and the number of best-of-NN samples used for acting, so the panel varies the two together.

References

  1. [1]Armaan A. Abraham, Lucy Xiaoyang Shi, and Chelsea Finn. Long-horizon Q-learning: Accurate value learning via n-step inequalities. arXiv preprint arXiv:2605.05812, 2026.arxiv.org/abs/2605.05812
  2. [2]Hongjoon Ahn, Heewoong Choi, Jisu Han, and Taesup Moon. Option-aware temporally abstracted value for offline goal-conditioned reinforcement learning. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.12737
  3. [3]Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI Conference on Artificial Intelligence, 2017.arxiv.org/abs/1609.05140
  4. [4]Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2209.14548
  5. [5]Brett Daley. Gated Q-learning: Add off-policy bias to taste. In Reinforcement Learning Journal, volume 7, 2026.arxiv.org/abs/2607.28916
  6. [6]Brett Daley, Martha White, Christopher Amato, and Marlos C. Machado. Trajectory-aware eligibility traces for off-policy reinforcement learning. In International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2301.11321
  7. [7]Kristopher De Asis, J. Fernando Hernandez-Garcia, G. Zacharias Holland, and Richard S. Sutton. Multi-step reinforcement learning: A unifying algorithm. In AAAI Conference on Artificial Intelligence, 2018.arxiv.org/abs/1703.01327
  8. [8]Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018.arxiv.org/abs/1802.01561
  9. [9]Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.arxiv.org/abs/1906.05253
  10. [10]Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems (NeurIPS), 2010.
  11. [11]Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2106.06860
  12. [12]Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (ICML), 2019.arxiv.org/abs/1812.02900
  13. [13]Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023.
  14. [14]Anna Harutyunyan, Marc G. Bellemare, Tom Stepleton, and Rémi Munos. Q(λ) with off-policy corrections. In International Conference on Algorithmic Learning Theory (ALT), 2016.DOI
  15. [15]Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016.
  16. [16]J. Fernando Hernandez-Garcia and Richard S. Sutton. Understanding multi-step deep reinforcement learning: A systematic study of the dqn target. arXiv preprint arXiv:1901.07510, 2019.
  17. [17]Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.arxiv.org/abs/1710.02298
  18. [18]Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning (CoRL), 2018.arxiv.org/abs/1806.10293
  19. [19]Steven Kapturowski, Georg Ostrovski, John Quan, Rémi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations (ICLR), 2019.
  20. [20]Michael Kearns and Satinder Singh. Bias-variance error bounds for temporal difference updates. In Conference on Learning Theory (COLT), 2000.
  21. [21]Byeongchan Kim and Min hwan Oh. Peng’s q(λ) for conservative value estimation in offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2605.14779
  22. [22]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.arxiv.org/abs/1412.6980
  23. [23]Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2110.06169
  24. [24]Tadashi Kozuno, Yunhao Tang, Mark Rowland, Rémi Munos, Steven Kapturowski, Will Dabney, Michal Valko, and David Abel. Revisiting Peng’s Q(λ) for modern reinforcement learning. In International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2103.00107
  25. [25]Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.arxiv.org/abs/1906.00949
  26. [26]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2006.04779
  27. [27]Cassidy Laidlaw, Stuart Russell, and Anca Dragan. Bridging RL theory and practice with the effective horizon. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2304.09853
  28. [28]Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016.arxiv.org/abs/1504.00702
  29. [29]Qiyang Li and Sergey Levine. Q-learning with adjoint matching. arXiv preprint arXiv:2601.14234, 2026.arxiv.org/abs/2601.14234
  30. [31]Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. In Advances in Neural Information Processing Systems, 2025b.arxiv.org/abs/2507.07969
  31. [32]Qiang Liu, Lihong Li, Ziyang Tang, and Danyang Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2018.arxiv.org/abs/1810.12429
  32. [33]Ashique Rupam Mahmood, Huizhen Yu, and Richard S. Sutton. Multi-step off-policy learning without importance sampling ratios. arXiv preprint arXiv:1702.03006, 2017.arxiv.org/abs/1702.03006
  33. [34]Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-DICE: In-sample diffusion guidance for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024a.
  34. [35]Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhang. ODICE: Revealing the mystery of distribution correction estimation via orthogonal-gradient update. In International Conference on Learning Representations (ICLR), 2024b.
  35. [36]Rémi Munos. Error bounds for approximate value iteration. In AAAI Conference on Artificial Intelligence, 2005.
  36. [37]Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.arxiv.org/abs/1606.02647
  37. [38]Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018.arxiv.org/abs/1805.08296
  38. [39]Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv:1909.10618, 2019.
  39. [40]Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
  40. [42]Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQL: Offline goal-conditioned RL with latent states as actions. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
  41. [43]Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Benchmarking offline goal-conditioned RL. In International Conference on Learning Representations (ICLR), 2025a.
  42. [44]Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes RL scalable. In Advances in Neural Information Processing Systems, 2025b.arxiv.org/abs/2506.04168
  43. [45]Seohong Park, Qiyang Li, and Sergey Levine. Flow Q-learning. In International Conference on Machine Learning (ICML), 2025c.arxiv.org/abs/2502.02538
  44. [47]Jing Peng and Ronald J. Williams. Incremental multi-step Q-learning. Machine Learning, 22:283–290, 1996.
  45. [48]Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
  46. [49]Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. In AAAI Conference on Artificial Intelligence, 2010.DOI
  47. [50]Doina Precup, Richard S. Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation. In International Conference on Machine Learning (ICML), 2000.
  48. [51]Mark Rowland, Will Dabney, and Rémi Munos. Adaptive trade-offs in off-policy learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.arxiv.org/abs/1910.07478
  49. [54]Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018.
  50. [55]Richard S. Sutton, A. Rupam Mahmood, and Martha White. An emphatic approach to the problem of off-policy temporal-difference learning. Journal of Machine Learning Research, 2016.
  51. [57]John N. Tsitsiklis and Benjamin Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42(5):674–690, 1997.DOI
  52. [58]Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648, 2018.arxiv.org/abs/1812.02648
  53. [59]Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal networks for hierarchical reinforcement learning. In International Conference on Machine Learning (ICML), 2017.
  54. [60]Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning (ICML), 2023a.arxiv.org/abs/2304.01203
  55. [61]Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2023b.arxiv.org/abs/2208.06193
  56. [62]Christopher J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, University of Cambridge, 1989.DOI
  57. [63]Martha White and Adam White. A greedy approach to adapting the trace parameter for temporal difference learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2016.
  58. [64]Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.arxiv.org/abs/1911.11361
  59. [65]Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. In NeurIPS Offline Reinforcement Learning Workshop, 2021.arxiv.org/abs/2110.07395
  60. [66]Haoran Xu, Li Jiang, Jianxiong Li, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2210.08323
  61. [67]Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2303.15810
  62. [68]Haoran Xu, Shouzhe Li, Harshit Sikchi, Scott Niekum, and Amy Zhang. An optimal discriminator weighted imitation perspective for reinforcement learning. In International Conference on Learning Representations (ICLR), 2025a.arxiv.org/abs/2504.13368
  63. [69]Haoran Xu, Liyuan Mao, Hui Jin, Weinan Zhang, Xianyuan Zhan, and Amy Zhang. Uni-RL: Unifying online and offline RL via implicit value regularization. In Advances in Neural Information Processing Systems (NeurIPS), 2025b.
  64. [70]Haoran Xu, Kaiwen Hu, Somayeh Sojoudi, and Amy Zhang. Reinforcement learning via value gradient flow. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2604.14265
  65. [71]Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. DeepThermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In AAAI Conference on Artificial Intelligence, 2022.
  66. [72]Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems (RSS), 2023.arxiv.org/abs/2304.13705
  67. [73]Bill Chunyuan Zheng, Vivek Myers, Benjamin Eysenbach, and Sergey Levine. Multistep quasimetric learning for scalable goal-conditioned reinforcement learning. In International Conference on Learning Representations, 2026.arxiv.org/abs/2511.07730
  68. [74]John L. Zhou and Jonathan C. Kao. Flattening hierarchies with policy bootstrapping. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2505.14975

Paper details

vlnc:W_d7na3cd8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents