In this paper
AbstractIntroductionRelated WorkPreliminariesError Accumulation in One-Step and -Step TDHorizon-Scalable Q-LearningGated Multi-Step TargetsA Density-Free Gate via Per-Decision Importance SamplingPractical ConsiderationsExperimentsExperimental SetupQ1: A Toy Example with Corrupted WindowsQ2: Results on OGBench and OCBenchQ3: Combining HQL with Other Policy Extraction MethodsQ4: Ablations of HQLConclusion and LimitationsAcknowledgementAppendixDerivation of the One-Step Error AccumulationDerivation of the -Step Error AccumulationCombination-Lock DatasetOGBench Benchmark: Experiment Details and Per-Task ResultsShared Implementation SettingsOCBench Benchmark: Experiment DetailsSoft Gate versus Hard GateAblation DetailsReferencesWorksComputer Science
Horizon-Scalable Q-Learning
Abstract
Learning accurate off-policy state-action value functions (Q-functions) from offline datasets becomes harder as the task horizon grows. One-step temporal-difference (TD) learning bootstraps after every transition, so small per-step errors accumulate along the horizon, and early states can be badly misvalued even when the TD loss is small. Multi-step TD shortens this chain by following the dataset trajectory for several steps; however, since the behavior policy is often suboptimal, every followed action adds a bias, and no fixed number of steps suits a whole trajectory. In this paper, we propose Horizon-Scalable Q-Learning (HQL), a simple method that enables accurate long-horizon value learning by automatically balancing between one-step and multi-step returns. HQL extends a target through a dataset action only when the critic values it at least as highly as the best-of-N actions sampled from the BC policy, and otherwise falls back to a one-step target. We show that this rule is a principled off-policy correction that needs no action densities, which makes HQL compatible with expressive flow and diffusion policies. Experiments show that HQL avoids the failures of both one-step and multi-step TD on a combination-lock task across horizons, and outperforms prior horizon-reduction methods on long-horizon OGBench and OCBench tasks.
Introduction
Off-policy reinforcement learning (RL) learns the state-action value function (Q-function) of a target policy from an offline dataset collected by a different behavior policy, which enables RL to reuse logged experience in real-world domains such as robotics [28, 18] and industrial control [71]. However, off-policy value learning becomes harder as the effective horizon grows [27], i.e., the number of transitions across which the critic must carry value information before reaching a state whose value it already knows. Most off-policy actor-critic methods learn the critic with one-step temporal-difference (TD) backups [12, 26, 23, 67, 45]. A one-step target bootstraps after a single logged transition, which makes it robust to off-policy data but moves value back by only one transition per update. We show that the value error at a state is exactly the sum of the one-step Bellman residuals along the rest of the trajectory, so a critic can reach a small TD loss and still misvalue the start states by an amount that grows with the horizon.
Multi-step returns [62, 54] or Q-chunking [31, 72] shorten this chain by following the dataset trajectory for several steps before bootstrapping. Nevertheless, this imposes a trade-off between fast value propagation (longer backups) and low off-policy bias (shorter backups): we show that each spanned suboptimal action adds a systematic bias that grows toward a limit linear in . Since offline trajectories could be mixed with suboptimal actions, no fixed backup length is optimal for every TD update. Per-decision off-policy corrections such as Retrace [50, 37] make this choice at every transition with importance ratios. However, these ratios require policy densities, which flow or diffusion-based policies [45, 34, 29, 70] do not expose.
In this paper, we propose Horizon-Scalable Q-Learning (HQL), which shortens the effective horizon of Q-learning without inheriting the bias of suboptimal dataset actions. HQL achieves this by automatically balancing between one-step and multi-step returns at every transition (Figure 1). More specifically, HQL attaches a binary gate to every logged action, i.e., every action stored in the dataset: if the target critic values it at least as highly as the best-of- actions sampled from the BC policy, the gate is open and the target continues through it for one more step; otherwise, the target falls back to a one-step bootstrap. Our key observation is that this gate is principled without any density: for the optimal policy that behavior-regularized RL approximates [49, 48, 69], the behavior density cancels from the importance ratio, and as the largest gate that preserves becomes exactly the comparison of critic values. HQL uses an estimated behavior model only to sample action candidates and never evaluates its density. Since the gate depends only on the ordering of critic values, it is also invariant to the reward scale, and the only hyperparameter in HQL is the number of candidates .

Figure 1. Horizon-Scalable Q-Learning (HQL). Purple arcs show the reach of one training target. One-step TD accumulates residuals, and multi-step TD inherits the bias of the suboptimal action it spans (yellow). HQL compares each logged action with the best-of- policy samples, continuing the trace where it passes () and cutting it where it fails (, dashed), so reward travels far without inheriting off-policy bias.
We evaluate HQL on a combination-lock task, whose true -values are known exactly, and on long-horizon OGBench and OCBench tasks [43] (Park and Levine 2026). On the combination lock, HQL keeps the lowest value estimation error across different horizon, whereas one-step TD accumulates error and -step TD fails on corrupted dataset. On OGBench and OCBench, at a matched budget of critic targets, HQL outperforms one-step TD, -step TD, Q-chunking [31], and Long-horizon Q-Learning [1], especially on the most challenging long-horizon tasks. Perhaps surprisingly, we find that most of HQL’s targets are one or few-step targets: HQL wins by choosing which targets to extend, not by extending them far.
Related Work
Error propagation in value learning. Error propagation through bootstrapping has been studied for approximate value and policy iteration [36, 10], TD with function approximation [57], and -step and TD() updates [20]; its instability with off-policy data is the deadly triad [58], and the horizon governs the difficulty of evaluation and learning [32, 27].
Multi-step off-policy correction. Multi-step returns and eligibility traces shorten the backup chain [62, 54, 7], and a small fixed is standard in deep RL [17, 19, 16]. Per-decision importance sampling [50] and its clipped variants (Retrace [37], Tree[37]), off-policy [14], V-trace [8], trajectory-aware traces [6]) correct the mismatch step by step, and emphatic TD reweights updates [55]. Watkins’ cuts traces at non-greedy actions while Peng’s keeps them [47, 24, 21]; other work adapts the trace [63, 51], avoids ratios via action-dependent bootstrapping [33], gates traces at non-greedy actions [5], or adds multi-step inequalities [1]. However, these corrections need action probabilities or a discrete greedy target policy.
Horizon reduction. Instead of correcting within a backup, horizon-reduction methods change what a decision is. Hierarchical methods act over options or subgoals ([3, 59, 38, 39, 66]), and action chunking treats a short action sequence as one decision ((Seo and Abbeel, 2024; Li et al., 2025b,a; Song et al., 2026)). However, within each segment they use a fixed, uncorrected backup and inherit the bias; HQL is orthogonal and contains both as ungated special cases. Another line exploits the structure of goal-reaching tasks through quasimetric value functions ([60, 73]), graph search over subgoals ([9]), and goal-conditioned hierarchies ([42, 2, 74] (Park et al. 2026)). These methods rely on a goal-reaching reward and thus apply only to goal-conditioned RL; in contrast, HQL makes no assumption on the reward.
Policy extraction with expressive policies. Offline RL methods regularize the policy toward the data, either explicitly through divergence penalties ([64, 25, 65, 11]) or implicitly through in-sample value learning and weighted behavior cloning ([48, 40, 23, 66, 67, 68]). Recent work extracts expressive diffusion or flow policies by value-guided training ([61, 34, 45]), rejection sampling from a behavior model ([4, 13]), adjoint matching ([29]), or value gradient flow ([70]). These methods keep a one-step critic; HQL only changes the critic’s target and can serve any of them.
Preliminaries
We consider the problem presented as a finite-horizon Markov decision process (MDP) , where and are the state and action spaces, and denote the transition dynamics and reward function, and is the horizon; an episode that reaches its goal early enters a zero-reward absorbing state, so every episode has length . For a policy , the action value is , with and advantage . We consider the off-policy setting, where one needs to estimate of a target policy from a dataset (or replay buffer) collected by a possibly suboptimal behavior policy .
Error Accumulation in One-Step and -Step TD
Error decomposition along a chain. To understand how critic error scales with the horizon, we derive one error law for one-step and -step backups (derivations in Appendices § 8.§ 1–§ 8.§ 2). Consider the deterministic chain induced by a fixed deterministic target policy, with an exactly known terminal value. Write , let be the reward on the transition out of , and let be the target critic, with . The -step target is
where a true terminal state replaces the last term by its known value. The loss at lookahead is , where stops gradients ([54]); for we write . With the one-step Bellman residual and the value error , we have because , and with an exact terminal value () the recursion telescopes:
In other words, the value error is exactly the sum of the downstream residuals, for any critic at any point in training.
A unified error law. To see how the residuals add up, we model them as random noise across training runs: each residual has zero mean and variance , and residuals at different steps are only weakly correlated (Appendix § 8.§ 1). Under this model, applying Eq. (2) to -step backups, one block of steps at a time, gives a single expression for the start-state error (Appendix § 8.§ 2):
where is the average advantage of the logged actions that the targets span. Intuitively, the first term decreases with while the second increases with , so a longer backup trades residual accumulation for behavior-action bias.
One-step TD: residuals accumulate as . A one-step target spans no logged action, so the bias term vanishes, and summing Eq. (2) under the residual model gives, as ,
with the long-run variance of the residuals. This result is important as it shows that a small training loss does not certify a small value error: since measures only the per-step residual, Eq. (3) at reads
up to the factor . Appendix § 8.§ 1 discusses when this rate fails and how a discount caps it at a horizon of order .
-step TD: bias grows toward . Bootstrapping steps ahead divides the residual term by . However, along a logged trajectory now deviates from by plus the bootstrap error at the endpoint (Eq. (14)). For a near-optimal target policy every spanned advantage is at most zero, so this bias is systematic and does not average out; as grows, the square root of the second term of Eq. (3) approaches . This indicates that no fixed suits all logged actions: the best lookahead changes from step to step, which calls for a per-transition rule to minimize the total error.
Horizon-Scalable Q-Learning
In this section, we introduce Horizon-Scalable Q-Learning (HQL). We begin with a gated multi-step target and design the gate based on value comparison (§ 4.1). We then show that this gate is a limiting case of per-decision importance sampling yet needs no action density (§ 4.2). Finally, we provide several practical considerations for HQL and discuss its connection to previous work (§ 4.3).
Gated Multi-Step Targets
Our key idea is to let a backup continue through a logged action only when that action is at least as good as the actions the target policy would take, i.e., its advantage under the target policy is non-negative, which keeps it out of the bias term of Eq. (3) while reducing the horizon of TD learning via multi-step bootstrapping. To do so, we attach a gate to every transition and define the HQL target backward along a logged segment,
with the correction set to zero at the end of the segment, and kept for generality. Intuitively, the first term is the usual one-step bootstrap from the target policy, and the second term carries back whatever the logged continuation has learned beyond the critic’s current estimate at . When the gate accepts , this correction flows back and the target extends one more step; when it rejects , the target is a plain one-step target, and the states after the cut still start their own targets (as shown in Figure 1, bottom). Unrolling the recursion, a run of accepted actions yields a multi-step target whose length is set by the data rather than fixed in advance. Note that recovers one-step TD and is the uncorrected multi-step return.
It remains to decide when the gate is open, i.e., whether the target policy would keep the logged action. Intuitively, a logged action is safe to follow if it is at least as good as what the policy itself would propose, which the critic can check by comparing the logged action with candidates drawn from the policy’s own sampler:
where is the behavior policy that can be estimated using any advanced generative models from . In other words, the logged action is kept if it is at least as good as the best of policy proposals. Note that the gate depends only on an ordering of critic values, so it is invariant to the scale of , needs no behavior density, and has as its only hyperparameter; it costs behavior samples and critic evaluations per transition, the same as the best-of- bootstrap already requires.
Algorithm 1 Horizon-Scalable Q-Learning (critic update)
Require: dataset , window length , gate width , target budget
Sample windows of consecutive transitions from
At every position : draw and compute the gate (Eq. (7))
For : compute by Eq. (6), with no correction at
Regress all pairs onto ; update
A Density-Free Gate via Per-Decision Importance Sampling
A remaining question is whether cutting multi-step backups with this gate is principled, since cutting changes what a multi-step backup converges to. Per-decision importance sampling answers this [50, 37]: in the tabular setting, any gate with preserves as the fixed point of Eq. (6), and we call such a gate valid. However, evaluating this bound requires the density ratio , which is unavailable and hard to estimate [35, 68] when using generative policies, or when policies are implicit, e.g., is a flow sampler and selects among its samples. Fortunately, the ratio has a closed form for the policy that behavior-regularized RL methods approximate, i.e., the exponential tilt of the behavior policy [49, 48, 67, 29],
so the behavior density cancels and . We estimate the normalizer from behavior samples as , where is the log-sum-exp, and clipping the ratio at one gives the largest valid gate
This soft gate interpolates between two extremes. As , every gate equals one and the target is an uncorrected multi-step return. As , the tilt concentrates on the best behavior-supported action, and because , the exponent is nonnegative at any when no candidate beats the logged action and tends to otherwise, so the soft gate becomes exactly the value comparison of Eq. (7). Therefore, HQL’s gate is the limit of a valid gate, which is more scalable and robust since it avoids a temperature and can be computed without density. It also outperforms the soft gate at every we tried (Appendix § 8.7).
Practical Considerations
Dense targets over a trajectory window. Since HQL is not restricted to a fixed -step backup, it does not need to compute a target only at the first state of a sampled segment. Instead, HQL samples a trajectory window of consecutive transitions and computes the gated multi-step target for every state–action pair in the window. More specifically, it computes the gates once at every position, evaluates Eq. (6) backward from the window end, and regresses each onto its own target , , masking positions beyond an episode boundary (Algorithm 1). By doing so, a window yields targets whose maximal reach decreases from to , and the gate can shorten each of them further. To enable a fair comparison, we hold the number of critic targets per update fixed: with a budget of , one-step TD samples transitions and HQL samples windows.
Multi-step decisions via Chunked-HQL. So far the decision stride is fixed to 1, meaning each decision corresponds to a single environment step. To allow for more flexible decision-making, we let one decision span several environment steps. More specifically, with stride , a decision is a chunk of primitive actions that the critic scores as a unit, , and a window of chunks reaches environment steps while still producing targets. Note that the gate is unchanged, now comparing the logged chunk with the best-of- chunks drawn from the policy, and the backup of Eq. (6) holds with in place of and per-chunk reward sums.
Connections to existing methods. To summarize, in HQL or Chunked-HQL, sets how many environment steps one decision covers, and sets how many decisions one target may chain (Figure 2). With , gives one-step TD, an ungated window of size gives uncorrected -step TD, and a gated window gives HQL. With , is Q-chunking with chunks of actions ([31]), since a single chunk leaves nothing to gate, and gives Chunked-HQL at stride . Note that the two variables extend the maximal reach of a target, environment steps, in different ways. Increasing lets one target makes more decisions while the policy outputs less actions per step, and the gate decides for each target how many of these decisions it actually uses. In contrast, increasing makes each decision cover more environment steps, but each chunk then runs open loop and the policy must search a larger, -dimensional action space.

Figure 2. HQL and Chunked-HQL. sets how many decisions one target chains, and how many environment steps each decision covers.
Experiments
The goal of our experiments is to answer four questions. (Q1) Does HQL enable better value learning under different horizon lengths and dataset suboptimality? (Q2) How does HQL compare with TD and other baselines on OGBench and OCBench? (Q3) Does HQL work with different policy-extraction methods? (Q4) How do the different components of HQL affect its performance?
Experimental Setup
Combination lock. For Q1, we use a modified combination-lock chain ([44]) with linearly ordered states and two actions: the “correct” one advances to the next state and the other returns to the first state, with reward -1 per step and 0 at the terminal state. The offline dataset consists of length- windows (64 transitions each): for every state–action pair we roll out windows that start with that pair and then follow the correct action, truncating a window early if it reaches the terminal state. Following Park et al. [44], we construct datasets of -step trajectory windows with uniform coverage of starting state–action pairs. To study the effect of increasing action corruption, we independently replace each of the subsequent actions with the non-optimal action with probability , chosen so that in expectation of windows are clean, i.e. contain no corrupted action. We use , giving 2HM windows per dataset. Because the dynamics are deterministic, the ground-truth Q is available in closed form (Appendix § 8.§ 3).
OGBench. Q2–Q4 use six state-based OGBench configurations ([43]): cube-double, cube-triple, scene (sparse reward), humanoidmaze-medium, antmaze-giant and humanoidmaze-giant, whose task horizons last 500 to 4000 steps (Table 5); we report the mean score over each configuration’s five tasks. Runs train for 1M offline and then 1M online steps, with 50 evaluation episodes every steps; a run’s score is the mean of its last three evaluations.
| Task group | Dataset size (transitions) | Episode length | Action dimension | Observation dimension | Default task |
| Locomotion (locomaze) | |||||
| pointmaze-medium | 1M | 1000 | 2 | 2 | 1 |
| pointmaze-large | 1M | 1000 | 2 | 2 | 1 |
| pointmaze-giant | 1M | 1000 | 2 | 2 | 1 |
| pointmaze-teleport | 1M | 1000 | 2 | 2 | 1 |
| antmaze-medium | 1M | 1000 | 8 | 29 | 1 |
| antmaze-large | 1M | 1000 | 8 | 29 | 1 |
| antmaze-giant† | 1M | 1000 | 8 | 29 | 1 |
| antmaze-teleport | 1M | 1000 | 8 | 29 | 1 |
| humanoidmaze-medium† | 2M | 2000 | 21 | 69 | 1 |
| humanoidmaze-large | 2M | 2000 | 21 | 69 | 1 |
| humanoidmaze-giant† | 4M | 4000 | 21 | 69 | 1 |
| antsoccer-arena | 1M | 1000 | 8 | 42 | 4 |
| antsoccer-medium | 4M | 1000 | 8 | 42 | 4 |
| Manipulation (manipspace) | |||||
| cube-single | 1M | 200 | 5 | 28 | 2 |
| cube-double† | 1M | 500 | 5 | 37 | 2 |
| cube-triple† | 3M | 1000 | 5 | 46 | 2 |
| cube-quadruple | 5M | 1000 | 5 | 55 | 2 |
| cube-octuple | – | 1500 | 5 | – | 2 |
| scene† | 1M | 750 | 5 | 40 | 2 |
| puzzle-3x3 | 1M | 500 | 5 | 55 | 4 |
| puzzle-4x4 | 1M | 500 | 5 | 83 | 4 |
| puzzle-4x5 | 3M | 1000 | 5 | 99 | 2 |
| puzzle-4x6 | – | 1000 | 5 | – | 2 |
Table 5. OGBench state-based task characteristics. Episode length is the maximum number of environment steps per episode. Rows marked † are the groups used in this work.
OCBench. To test HQL on even longer horizons, we also use OCBench ((Park and Levine, 2026)), a robotic manipulation benchmark whose datasets come from scripted policies with properties similar to human demonstrations. We take eight full-frequency (50Hz) tasks from four domains (block, chamber, switch, hanoi), whose episodes last 1000 to 5000 steps. Following the RL baselines of OCBench, all methods use 25-step action chunks, 8 × 2048 residual MLPs, an HL-Gauss critic and , with 1024 critic targets per update. Runs train for 1M offline and 1.5M online steps with 500 evaluation episodes every steps (Appendix § 8.§ 6).
Baselines. We compare HQL () with existing baselines under a matched compute budget: every method receives 512 critic targets per gradient step. We consider three TD baselines. 16-window TD uses the same windows and batch as HQL but removes the gate, so comparing the two isolates the gate’s effect. 8-step TD ( matches the average reach of a 16-step window) and 1-step TD each regress a single target per sequence with a batch of 512. We also compare with Q-chunking (QC) ([31]), which treats a chunk of 8 actions as a single decision and bears no off-policy bias, and Long-horizon Q-Learning (LQL) ([1]), which adds hinge penalties to one-step TD that enforce multi-step inequalities along the logged trajectory. All methods share the same critic architecture and use the same behavior flow policy with best-of- sampling.
Q1: A Toy Example with Corrupted Windows
Figure 3 supports both halves of the analysis. On clean data, the error of 1-step TD grows by three orders of magnitude from to , while 64-step TD and HQL avoid this accumulation. As the corruption rate increases, 64-step TD degrades sharply, because each target sums the negative advantages of the actions it spans. HQL cuts the trace at those actions: its error stays below both 1-step TD and 64-step TD at every horizon.
Figure 3. Combination lock. HQL remains accurate under data corruption as the horizon grows. The figure illustrates the RMSE between the learned and the true , averaged uniformly over all state–action pairs of the corrupted combination lock, after updates (log scale). Panels vary the corruption fraction of the dataset. n-step TD () diverges as soon as any corruption is present; 1-step TD is unaffected by corruption but its error grows sharply with the horizon .
Q2: Results on OGBench and OCBench
HQL is the strongest method across all six domains. HQL reaches a mean success rate of 0.74 against 0.49 for the best TD baseline, 8-step TD (Figure 4). Consistent with our analysis in § 3.1, the gap is largest on the four configurations with the largest horizons of 1000 steps or more (Table 5); on cube-triple, HQL reaches 0.54 while both multi-step baselines score 0. 16-window TD matches HQL in everything except the gate, yet it is the weakest method on average, this demonstrates the effectiveness of gating in HQL.

Figure 4. HQL versus baseline TD methods at matched critic update budget. Success rate averaged over the five tasks of each configuration and over 5 seeds, with standard deviation shown as the band; gray marks the offline phase. HQL outperforms all baselines on all six configurations, especially on the challenging long-horizon ones.
Comparison with other baselines. At the same budget and sampling width (Table 1), HQL improves the mean success rate by 12% over LQL and by 29% over QC. QC is best on cube-double and scene but scores 0% on both humanoid mazes.
| HQL | Chunked-HQL | LQL | QC | |
| antmaze-giant | 1 → 87±2 | 0 → 86±1 | 0 → 44±4 | 0 → 46±1 |
| cube-double | 67 → 87±1 | 73 → 100±0 | 45 → 83±2 | 28 → 100±0 |
| cube-triple | 12 → 54±1 | 7 → 84±1 | 55 → 35±3 | 1 → 26±3 |
| hmaze-giant | 10 → 28±1 | 0 → 29±2 | 12 → 26±1 | 0 → 0±0 |
| hmaze-medium | 56 → 86±1 | 86 → 88±2 | 48 → 86±1 | 0 → 0±0 |
| scene | 93 → 99±0 | 98 → 100±0 | 96 → 97±1 | 97 → 100±0 |
| mean | 40 → 74±1 | 41 → 81±1 | 34 → 62±1 | 21 → 45±1 |
Table 1. HQL and Chunked-HQL against LQL and QC. Success rates (%) are averaged over five tasks and reported as offline → online with standard error over five seeds; we bold the best offline and the best online score at integer level. HQL and Chunked-HQL achieve the best results, especially on the long-horizon tasks.
Although Q-chunking avoids the off-policy bias by having the critic score the whole -sized action chunk , it falls short because the critic must choose among -dimensional chunks, and a fixed best-of- search covers that space increasingly sparsely as the chunk size grows. Second, each chunk runs open loop for -steps (we choose while running QC), so the agent cannot correct its action within a chunk. This also explains why QC performs well on cube-double and scene, whose chunks are low-dimensional (please refer to Table 5 for action dimensions of the different OGBench tasks), but trails on antmaze-giant and fails on both humanoid mazes.
We expect a wider best-of- search to help QC, since it covers the chunk space more densely, though at a cost in samples and critic evaluations that grows with . The gate is also complementary to chunking: adding it on top of chunked actions (Chunked-HQL) raises QC’s success rate from 45% to 81% (Table 1).
Why does HQL help? One might expect a multi-step method to win by sending reward information far back along the trajectory. To understand why HQL works better than the TD baselines, we log how many steps each target’s trace survives (Figure 6). Perhaps surprisingly, the actual traces are short: 77% of targets are exactly one-step targets while 17% extend to two steps and 6% to three or more steps, and none exceeds six of the sixteen available steps. This indicates that HQL helps not by making targets longer but by choosing which ones to extend: -step TD extends all of them and inherits the bias of every action spanned, 1-step TD extends none, and HQL extends a target only when its logged action passes the gate.

Figure 6. Per-target trace length with the memoryless prediction (dashed).
Chunked-HQL scales to the longer horizons of OCBench. On OCBench, we compare Chunked-HQL (, , i.e., a 200-step window) with -step TD at , which regresses the chunk critic onto an uncorrected -step return and is the backup used by OCBench’s own RL baselines (). Motivated by § 5.5, we decouple the two widths of HQL: the gate uses , while the bootstrap and acting use best-of-16 as in the baselines. Chunked-HQL reaches a mean success rate of 0.82 at 2.5M steps against 0.53 for 100-step TD and 0.46 for 200-step TD (Figure 5). The gap is largest on switch-3x3 (0.91 vs. 0.15), switch-4x4 (0.76 vs. 0.17) and chamber-medium (0.53 vs. 0.12), where the -step baselines barely improve or even degrade during online training. Note that 200-step TD is worse than 100-step TD on seven of the eight tasks, which agrees with Eq. (3): a longer uncorrected return sums the negative advantages of more logged actions, while the gate lets HQL extend a target only through chunks that pass the value comparison.

Figure 5. Chunked-HQL versus -step TD on OCBench. Success rate on eight 50 Hz tasks (one seed, smoothed over three evaluations; numbers are the mean of the last three). All methods use 25-step chunks and the same best-of-16 bootstrap. Chunked-HQL matches or outperforms both baselines on all eight tasks, especially on switch and chamber-medium.
Q3: Combining HQL with Other Policy Extraction Methods
| cube-double | cube-triple | antmz-giant | hmz-giant | |
| QAM | 29 → 50 | 1 → 1 | 14 → 83 | 1 → 0 |
| + HQL | 38 → 99 | 0 → 80 | 0 → 98 | 0 → 63 |
| VGF | 69 → 74 | 3 → 25 | 1 → 28 | 0 → 0 |
| + HQL | 80 → 89 | 4 → 37 | 0 → 90 | 0 → 4 |
Table 2. HQL as a drop-in critic for different policy extraction methods. Success rates (%) on task 2, offline → online; we bold the better score of each pair. HQL improves both QAM and VGF on all four tasks.
HQL with QAM. Q-learning with adjoint matching (QAM) [29] trains a flow policy to sample from the tilted distribution . To combine it with HQL, we keep the hard gate unchanged and only replace the best-of- policy with the QAM policy. With HQL as its critic, QAM improves on all four tasks at the end of online training, especially on cube-triple (1% → 80%) and humanoidmaze-giant (0% → 63%).
HQL with value gradient flow (VGF). VGF [70] can be thought of as an implicit policy that transports from behavior samples toward high-value actions with a few steps of -gradient ascent. With HQL as its critic, VGF again improves on all four tasks, most on antmaze-giant (28% → 90%).
Q4: Ablations of HQL
Does the window length matter? We find the effect is small. At a fixed budget of 512 targets per update, (batch 64/32/16) gives similar performance: 0.607, 0.646, and 0.621 (Figure 7, left). This agrees with the trace length analysis in § 5.3: since empirically no target used more than six steps, a window of 16 already exceeds what the gate uses. HQL is therefore not sensitive to unless it is too small.

Figure 7. HQL ablations over window length (left) and best-of- gate width (right). matters less than , and a smaller gives better results.
Does the best-of- width matter for HQL? We try best-of- width from . On OGBench datasets we find that a smaller is better: gives the mean value of 0.596, 0.521, and 0.461, three times the spread of (Figure 7, right). Intuitively, a logged action exchangeable with the proposals beats all of them with probability , so a larger accepts fewer logged actions and HQL approaches 1-step TD when is large. We provide more details on the ablation setup in Appendix § 8.8.
Conclusion and Limitations
In this paper, we propose HQL, which shortens the effective horizon of Q-learning without inheriting the bias of suboptimal logged actions. HQL gates each logged action with a density-free value comparison and achieves strong results on long-horizon OGBench tasks, both alone and as a drop-in critic for QAM and VGF. We believe that HQL represents a concrete step toward horizon-scalable off-policy RL. One limitation is that the gate relies on the critic’s own ranking. One future work is to apply HQL over subgoals in a hierarchical manner.
Acknowledgement
We thank Claas Voelcker and members from MIDI lab for discussions on the method and feedback on the early draft of the paper. HX is supported by the Amazon Fellowship. This research used the computational cluster resource provided by the Texas Advanced Computing Center at UT Austin.
Appendix
Derivation of the One-Step Error Accumulation
This appendix states the residual model behind Eq. (4) and derives the rate. For a family of chains with growing horizon, assume the one-step residuals form a second-order stationary process whose statistics do not depend on :
with weak correlation in the form . Taking the mean and variance of the sum in Eq. (2) with terms gives
By summability of , as ,
the long-run variance, which is necessarily nonnegative. If , then , which is Eq. (4); this is the nondegenerate random-walk regime.
When the rate does not apply. The random-walk rate is conditional on the residual model and is not a worst case. Negative correlation can give and error, while a systematic same-sign residual of horizon-independent size gives . Such a residual arises, for example, when the critic’s values are shrunk toward zero under all-negative rewards: the semi-gradient fixed point with shrinkage satisfies at every step. Accumulation also requires persistent residuals: in a fully expressive tabular chain, exact convergence gives , , and no accumulation, whereas function approximation, finite optimization, and target-network lag all leave residuals.
Discounting. With the telescoped sum is geometrically weighted, , and for uncorrelated residuals . The factor grows linearly in until is of order and saturates at afterwards, so the discount caps the accumulation at its own effective horizon ( steps at ) rather than removing it; the analysis applies with replaced by up to constants.
Derivation of the -Step Error Accumulation
Single-target decomposition. With and endpoint error , expanding the target of Eq. (1) over the behavior continuation gives
The derivation uses only at each spanned transition (deterministic transitions given the action; the term contributes no advantage because the target evaluates at the logged action itself). The first term is the behavior-action bias inside the segment, the second the endpoint bootstrap error.
Chaining blocks along the horizon. To turn the per-target decomposition into a start-state statement, partition the chain into blocks of length (assume for clarity) and define the -step training residual at each block start. Adding and subtracting and applying Eq. (14),
where we identify the endpoint state-value error with the action-value error at the next block start (exact on the deterministic evaluation chain). Iterating over blocks with ,
Model the training residuals as zero-mean, weakly correlated across blocks with per-block variance (the analogue of the model in Appendix § 8.§ 1); the bias terms , by contrast, share a sign for a near-optimal target policy and add coherently. Then
and since the bias sum covers all logged actions except the block starts, with the average spanned advantage, which is Eq. (3). The result is schematic in the same sense as Appendix § 8.§ 1: it conditions on the residual model, ignores cross terms between residuals and biases, and uses block-aligned targets, but none of these affect the two scalings it exhibits—a random-walk term and a bias term that grows with toward the horizon-linear ceiling .
Combination-Lock Dataset
In this didactic setup (Figure 8), the policy is the deterministic answer policy, and the dynamics are deterministic and acyclic under , so the action-value function has a closed form. We anchor at the terminal state, . The Bellman equation then reduces to the recursion , which unrolls to
Figure 8. The combination lock dataset with linearly arranged H states.
Taking the answer action costs one step and advances the chain. Any other action costs one step and resets to the start. Hence
OGBench Benchmark: Experiment Details and Per-Task Results
This appendix documents the configuration behind the OGBench benchmark figure of § 5.3 (Figure 4) and reports the per-task curves that its task means aggregate over (Figure 9).

Figure 9. Per-task view of the OGBench benchmark (Figure 4). Reading down a column shows how one task index behaves across environments, and a row shows how one environment varies over tasks.
Tasks and datasets. We use six environment–discount configurations: cube-double, cube-triple, scene, and humanoidmaze-medium at , and antmaze-giant and humanoidmaze-giant at ; Table 5 lists dataset sizes, episode lengths, and dimensionalities. Every group provides five single-task variants that share one dataset and differ only in the reward relabelling, and we run all five, so each panel of Figure 4 aggregates five tasks. scene uses the sparse reward relabelling.
Training runs. Each run trains for offline gradient steps followed by online steps, evaluating every steps with 50 episodes; the reported score is the mean success rate over the last three evaluations. Each method is run across the five different tasks corresponding to each of the six OGBench environments for five seeds. Curves show the mean over seeds of the task-averaged score, with bands of standard deviation across seeds; we average over tasks before computing seed statistics because per-task success is bimodal, so the across-task spread would swamp seed variance. Figure 9 gives the per-task breakdown.
HQL configuration. HQL uses a window of with 32 windows per batch, i.e., critic targets per update under the accounting of § 4.3. The gate is the best-of- indicator of Eq. (7) with , computed from the target critic. The policy is a behavior-cloned flow sampler with best-of- action selection () used both for acting and for the bootstrap , so the gate, the bootstrap, and the actor all refer to the same target policy. Table 4 lists the full configuration.
| Parameter | Value |
| Window length | 16 |
| Windows per batch | 32 |
| Critic targets per update | 512 () |
| Gate width (Eq. (7)) | 4 |
| Gate critic | target network |
| Discount factor | 0.99 (0.995 for the two giant mazes) |
| Optimizer | Adam (Kingma and Ba, [22]) |
| Learning rate | |
| Target network update rate | |
| Critic ensemble size | 2 |
| Target aggregation | mean |
| Network width | 512 |
| Network depth | 4 |
| Activation function | GELU (Hendrycks and Gimpel, [15]) |
| Critic layer norm | Yes |
| Actor layer norm | No |
| Number of flow steps (behavior sampler) | 10 |
| Best-of- action samples (actor and bootstrap) | 4 |
| Action chunk size | 1 (no action chunking) |
| UTD ratio | 1 |
| Offline training steps | |
| Online training steps | |
| Evaluation interval / episodes | / 50 |
| Seeds per cell | 5 (seeds 0–4) |
Table 4. Hyperparameters for the OGBench benchmark experiment (Figure 4 and Figure 9).
Baseline configurations. The three TD baselines share all networks and optimization settings with HQL (Table 4) and differ only in how their 512 critic targets per update are formed (Table 3).
| Method | Seq. length | Seq. per update | Targets per seq. | Target |
| HQL | 16 | 32 | 16 | gated, every position |
| 16-window TD | 16 | 32 | 16 | -step, every position |
| 8-step TD | 8 | 512 | 1 | 8-step, first state |
| 1-step TD | 1 | 512 | 1 | 1-step |
Table 3. How each method forms its 512 critic targets per update.
Shared Implementation Settings
Unless a later subsection states otherwise, every experiment uses the networks, optimizer and behavior flow sampler of Table 4. All agents are implemented in one JAX codebase with shared replay, evaluation and logging, so two compared methods differ only in the settings named for that comparison.
Matched budget. All comparisons in § 5.3 and the window-length ablation of § 5.5 fix the number of critic targets per update at 512, following the accounting of Park et al. [44]: a batch of windows of length yields targets for methods that regress every position, and targets for methods that regress only the first state. The number of transitions read per update therefore differs across methods; for example, 8-step TD reads 4096 transitions to form its 512 targets.
Compute. Runs of steps were executed on 8-GPU A100 and H100 nodes, eight runs per node, at about 150 iterations/s per run (–6 h per run).
OCBench Benchmark: Experiment Details
This appendix documents the configuration behind the OCBench results of § 5.3 (Figure 5). Unlike the OGBench experiments, we follow the RL baselines of OCBench (Park and Levine, 2026) for every setting other than the critic target, so all three methods share the same networks, critic loss, action space and action chunks (Table 6).
| Parameter | Value |
| Action chunk size | 25 (cut at episode boundaries) |
| Window length (Chunked-HQL) | 8 chunks (200 environment steps) |
| Windows per batch (Chunked-HQL) | 128 |
| Critic targets per update | 1024 (= 128 8) |
| Batch size of -step TD | 1024 (one target per sequence) |
| of -step TD | 100, 200 |
| Gate width (Eq. (7)) | 2 |
| Gate critic | target network |
| Best-of- (actor and bootstrap) | 16 |
| Discount factor | 0.9999 |
| Optimizer | Adam [22] |
| Learning rate | |
| Target network update rate | |
| Critic ensemble size | 2 |
| Target aggregation | mean |
| Network (actor and critic) | residual MLP, 8 2048 |
| Activation function | GELU [15] |
| Critic layer norm | Yes |
| Actor layer norm | Yes |
| Critic loss | HL-Gauss, 101 bins on [0, 1], bins |
| Number of flow steps (behavior sampler) | 10 |
| Flow-time distribution | Beta(1, 1.5) |
| Action normalization | percentiles 1/99 , clip at |
| UTD ratio | 1 |
| Offline training steps | |
| Online training steps | |
| Evaluation interval / episodes | / 500 |
| Seeds per cell | 1 (seed 0) |
Table 6. Hyperparameters for the OCBench benchmark experiment (Figure 5).
Tasks and datasets. We use eight full-frequency (50 Hz) tasks, block-single, block-double, chamber-easy, chamber-medium, switch-3x3, switch-4x4, hanoi-single and hanoi-double (all task1), whose episode limits range from 1000 to 5000 steps. We train on all episodes of each dataset without filtering for successful ones. Actions are normalized as in OCBench, i.e., the 1st and 99th percentiles of each action dimension are mapped to and the result is clipped at .
Method configurations. Chunked-HQL uses chunks of actions and windows of chunks (200 environment steps) with 128 windows per batch, i.e., critic targets per update. Its gate compares the logged chunk with the best of policy chunks, while the bootstrap and acting take the best of 16 chunks, the same width as the baselines. The -step TD baselines () regress a single target per sequence with a batch of 1024. Each run is evaluated with 500 episodes every steps, and the reported score is the mean success rate over the last three evaluations.
Soft Gate versus Hard Gate
§ 4.2 shows that the hard gate of Eq. (7) is the limit of the soft gate of Eq. (9). A natural question is whether a finite works better, since the soft gate keeps partial credit for logged actions that are close to the best candidate. To answer this, we compare the soft gate at with HQL’s hard gate on six OGBench configurations with five tasks each. Note that sets every gate to one and recovers uncorrected 8-step TD. All runs use , a batch of 128 windows (1024 critic targets per update), 1M offline and 1M online steps, and one seed; the soft gate estimates log and acts with 16 behavior samples, and the hard gate uses for both.
Figure 10 and Table 7 show that the hard gate achieves the best mean success rate (61%), and that the soft gate improves steadily as increases (36% → 57%), i.e., as it approaches the hard gate. The gap is largest on the long-horizon mazes: on humanoidmaze-medium the hard gate reaches 91% against 77% for the best soft gate, and on antmaze-giant 65% against 58%. The soft gate is only better on cube-double at , while scene and puzzle-3x3 end at the same value for every gate. This indicates that the hard gate is not only the limit that removes as a hyperparameter, but also the strongest member of the family: a finite keeps partial credit for suboptimal logged actions, so the multi-step target inherits part of their bias, which is exactly the failure mode of uncorrected -step TD.

Figure 10. Soft gate at different temperatures versus HQL’s hard gate. Success rate averaged over the five tasks of each configuration, one seed; gray marks the offline phase. The soft gate improves as grows toward the hard gate, and the hard gate is the best on average, especially on the long-horizon mazes.
| soft gate | hard gate | |||||
| (HQL) | ||||||
| cube-double | 75 | 81 | 87 | 84 | 80 | 81 |
| cube-triple | 16 | 37 | 39 | 46 | 44 | 48 |
| scene | 59 | 60 | 59 | 60 | 60 | 60 |
| puzzle-3x3 | 21 | 20 | 20 | 20 | 20 | 20 |
| antmaze-giant | 24 | 11 | 24 | 36 | 58 | 65 |
| hmaze-medium | 19 | 37 | 60 | 72 | 77 | 91 |
| mean | 36 | 41 | 48 | 53 | 57 | 61 |
Table 7. Soft gate versus hard gate. Success rate (%) at the end of online training, mean of the last three evaluations over five tasks, one seed. We bold the best score in each row at integer level.
Ablation Details
Both panels of Figure 7 use HQL with the shared settings of Appendix § 8.§ 5. Each panel averages only over the environment–task cells that were run for every value of the ablated setting, so that only that setting varies within a panel.
Window length . We fix the budget at 512 critic targets per update, so the number of windows per batch decreases as increases (64, 32 and 16 for , 16 and 32 respectively). The panel covers six environment–discount configurations with five tasks each, i.e., 30 cells.
Gate width . We run this panel at with 128 windows per batch, i.e., 1024 critic targets per update, with two seeds per setting (we use 512 critic targets everywhere else in the paper). Here sets both the number of candidates in the gate and the number of best-of- samples used for acting, so the panel varies the two together.
References
- [1]Armaan A. Abraham, Lucy Xiaoyang Shi, and Chelsea Finn. Long-horizon Q-learning: Accurate value learning via n-step inequalities. arXiv preprint arXiv:2605.05812, 2026.arxiv.org/abs/2605.05812
- [2]Hongjoon Ahn, Heewoong Choi, Jisu Han, and Taesup Moon. Option-aware temporally abstracted value for offline goal-conditioned reinforcement learning. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.12737
- [3]Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI Conference on Artificial Intelligence, 2017.arxiv.org/abs/1609.05140
- [4]Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2209.14548
- [5]Brett Daley. Gated Q-learning: Add off-policy bias to taste. In Reinforcement Learning Journal, volume 7, 2026.arxiv.org/abs/2607.28916
- [6]Brett Daley, Martha White, Christopher Amato, and Marlos C. Machado. Trajectory-aware eligibility traces for off-policy reinforcement learning. In International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2301.11321
- [7]Kristopher De Asis, J. Fernando Hernandez-Garcia, G. Zacharias Holland, and Richard S. Sutton. Multi-step reinforcement learning: A unifying algorithm. In AAAI Conference on Artificial Intelligence, 2018.arxiv.org/abs/1703.01327
- [8]Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018.arxiv.org/abs/1802.01561
- [9]Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.arxiv.org/abs/1906.05253
- [10]Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems (NeurIPS), 2010.
- [11]Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2106.06860
- [12]Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (ICML), 2019.arxiv.org/abs/1812.02900
- [13]Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023.
- [14]Anna Harutyunyan, Marc G. Bellemare, Tom Stepleton, and Rémi Munos. Q(λ) with off-policy corrections. In International Conference on Algorithmic Learning Theory (ALT), 2016.DOI
- [15]Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016.
- [16]J. Fernando Hernandez-Garcia and Richard S. Sutton. Understanding multi-step deep reinforcement learning: A systematic study of the dqn target. arXiv preprint arXiv:1901.07510, 2019.
- [17]Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.arxiv.org/abs/1710.02298
- [18]Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning (CoRL), 2018.arxiv.org/abs/1806.10293
- [19]Steven Kapturowski, Georg Ostrovski, John Quan, Rémi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations (ICLR), 2019.
- [20]Michael Kearns and Satinder Singh. Bias-variance error bounds for temporal difference updates. In Conference on Learning Theory (COLT), 2000.
- [21]Byeongchan Kim and Min hwan Oh. Peng’s q(λ) for conservative value estimation in offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2605.14779
- [22]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.arxiv.org/abs/1412.6980
- [23]Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2110.06169
- [24]Tadashi Kozuno, Yunhao Tang, Mark Rowland, Rémi Munos, Steven Kapturowski, Will Dabney, Michal Valko, and David Abel. Revisiting Peng’s Q(λ) for modern reinforcement learning. In International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2103.00107
- [25]Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.arxiv.org/abs/1906.00949
- [26]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2006.04779
- [27]Cassidy Laidlaw, Stuart Russell, and Anca Dragan. Bridging RL theory and practice with the effective horizon. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2304.09853
- [28]Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016.arxiv.org/abs/1504.00702
- [29]Qiyang Li and Sergey Levine. Q-learning with adjoint matching. arXiv preprint arXiv:2601.14234, 2026.arxiv.org/abs/2601.14234
- [31]Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. In Advances in Neural Information Processing Systems, 2025b.arxiv.org/abs/2507.07969
- [32]Qiang Liu, Lihong Li, Ziyang Tang, and Danyang Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2018.arxiv.org/abs/1810.12429
- [33]Ashique Rupam Mahmood, Huizhen Yu, and Richard S. Sutton. Multi-step off-policy learning without importance sampling ratios. arXiv preprint arXiv:1702.03006, 2017.arxiv.org/abs/1702.03006
- [34]Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-DICE: In-sample diffusion guidance for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024a.
- [35]Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhang. ODICE: Revealing the mystery of distribution correction estimation via orthogonal-gradient update. In International Conference on Learning Representations (ICLR), 2024b.
- [36]Rémi Munos. Error bounds for approximate value iteration. In AAAI Conference on Artificial Intelligence, 2005.
- [37]Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.arxiv.org/abs/1606.02647
- [38]Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018.arxiv.org/abs/1805.08296
- [39]Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv:1909.10618, 2019.
- [40]Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
- [42]Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQL: Offline goal-conditioned RL with latent states as actions. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
- [43]Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Benchmarking offline goal-conditioned RL. In International Conference on Learning Representations (ICLR), 2025a.
- [44]Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes RL scalable. In Advances in Neural Information Processing Systems, 2025b.arxiv.org/abs/2506.04168
- [45]Seohong Park, Qiyang Li, and Sergey Levine. Flow Q-learning. In International Conference on Machine Learning (ICML), 2025c.arxiv.org/abs/2502.02538
- [47]Jing Peng and Ronald J. Williams. Incremental multi-step Q-learning. Machine Learning, 22:283–290, 1996.
- [48]Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
- [49]Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. In AAAI Conference on Artificial Intelligence, 2010.DOI
- [50]Doina Precup, Richard S. Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation. In International Conference on Machine Learning (ICML), 2000.
- [51]Mark Rowland, Will Dabney, and Rémi Munos. Adaptive trade-offs in off-policy learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.arxiv.org/abs/1910.07478
- [54]Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018.
- [55]Richard S. Sutton, A. Rupam Mahmood, and Martha White. An emphatic approach to the problem of off-policy temporal-difference learning. Journal of Machine Learning Research, 2016.
- [57]John N. Tsitsiklis and Benjamin Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42(5):674–690, 1997.DOI
- [58]Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648, 2018.arxiv.org/abs/1812.02648
- [59]Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal networks for hierarchical reinforcement learning. In International Conference on Machine Learning (ICML), 2017.
- [60]Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning (ICML), 2023a.arxiv.org/abs/2304.01203
- [61]Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2023b.arxiv.org/abs/2208.06193
- [62]Christopher J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, University of Cambridge, 1989.DOI
- [63]Martha White and Adam White. A greedy approach to adapting the trace parameter for temporal difference learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2016.
- [64]Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.arxiv.org/abs/1911.11361
- [65]Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. In NeurIPS Offline Reinforcement Learning Workshop, 2021.arxiv.org/abs/2110.07395
- [66]Haoran Xu, Li Jiang, Jianxiong Li, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2210.08323
- [67]Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2303.15810
- [68]Haoran Xu, Shouzhe Li, Harshit Sikchi, Scott Niekum, and Amy Zhang. An optimal discriminator weighted imitation perspective for reinforcement learning. In International Conference on Learning Representations (ICLR), 2025a.arxiv.org/abs/2504.13368
- [69]Haoran Xu, Liyuan Mao, Hui Jin, Weinan Zhang, Xianyuan Zhan, and Amy Zhang. Uni-RL: Unifying online and offline RL via implicit value regularization. In Advances in Neural Information Processing Systems (NeurIPS), 2025b.
- [70]Haoran Xu, Kaiwen Hu, Somayeh Sojoudi, and Amy Zhang. Reinforcement learning via value gradient flow. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2604.14265
- [71]Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. DeepThermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In AAAI Conference on Artificial Intelligence, 2022.
- [72]Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems (RSS), 2023.arxiv.org/abs/2304.13705
- [73]Bill Chunyuan Zheng, Vivek Myers, Benjamin Eysenbach, and Sergey Levine. Multistep quasimetric learning for scalable goal-conditioned reinforcement learning. In International Conference on Learning Representations, 2026.arxiv.org/abs/2511.07730
- [74]John L. Zhou and Jonathan C. Kao. Flattening hierarchies with policy bootstrapping. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2505.14975

