betaFeedback

WorksComputer Science

FOCUS: Active Critic Learning for Long-Horizon Language Model Agents

Abstract

Sparse outcome rewards make it difficult for long-horizon language-model agents to assign credit to earlier actions. Learned critics provide value estimates for intermediate states, but these estimates can be inaccurate, while estimating intermediate values with repetitive continuation samples from points within a trajectory incurs substantial costs. To mitigate these issues, we introduce FOCUS, an RL method that uses selective branching to help the critic network learn its value estimator. The rollout controller facilitates active learning for the critic by sampling continuations from states that are expected to most reduce the critic's estimation error. These refined value estimates then score the segment between consecutive states with a TD(λλ) advantage, while the sampled evidence supplies training targets for the critic. FOCUS supports long-horizon agentic reinforcement learning through a modular context-management interface that decouples context management from rollout allocation. Our experiments show that using compaction-induced segment boundaries for value estimation and credit assignment, FOCUS outperforms GRPO and VinePPO with matched per-step generation budgets on legal reasoning, deep search, and code localization benchmarks, improving held-out accuracy at Qwen3.5-9B scale by 7.2 points over the strongest baseline on BrowseComp-Plus (55.4 vs. 48.2) and by 4.1 points on DeonticBench (56.5 vs. 52.4).

Introduction

Language-model agents are increasingly trained to solve tasks that unfold over dozens or hundreds of tool calls: resolving an issue in a real repository [20, 7], completing a task on a live website [76], or answering a question that requires many rounds of search and reading [3, 24]. Reinforcement Learning with Verified Rewards (RLVR) is the standard way to improve such agents, but long trajectories mean both sampling and credit assignment are costly. A single verifiable reward must be distributed across every token within a rollout even when many had little impact on the final outcome. To learn efficiently, we need proper intermediate value estimates under a limited rollout budget.

These estimates can come from a learned critic network [27] or through Monte Carlo approximation from a given state [46]. A learned critic reuses information across trajectories, but its predictions can be inaccurate. Tree-structured rollouts branch at intermediate states to refine value estimates [23, 17, 33, 12, 19]. However, sampling additional language model continuations for each estimate incurs substantial costs, raising the question of how to allocate a rollout budget to improve value estimation most efficiently.

We treat refining value estimates via additional continuation branching as an active learning problem for a critic network: continuations should be sampled selectively from states where additional evidence is expected to improve value estimation the most. We introduce FOCUS: Focused Optimization on Critical Underperforming Segments, a rollout controller that runs each episode as a search tree. FOCUS combines the critic’s predictions with evidence from sampled continuations and allocates further rollouts according to their estimated marginal reduction in value-estimation error [15, 22]. These refined values provide stepwise credit for the policy, while the acquired evidence supplies training targets for the critic.

In long-horizon tasks, context compaction [6, 14, 49, 34], where the agent resumes from a compacted account of its previous actions and observations, provides a natural structure for applying selective rollouts: each segment between compactions becomes a step, at which step-level rewards and values can be calculated (Figure 1). However, FOCUS does not depend on a particular context-management protocol; its modular design separates rollout allocation, context management, and credit assignment.1 Appendix § 13 details the implementation, and § 4 discusses related work.

We evaluate FOCUS on DeonticBench [9], BrowseComp-Plus [3], and LocBench [2]. FOCUS outperforms GRPO and VinePPO on all three benchmarks under a matched per-step generation budget. With Qwen3.5-9B, FOCUS improves held-out accuracy over the strongest baseline by 7.2 points on BrowseComp-Plus (55.4 vs. 48.2) (Table tab:main-results). Diagnostics show more effective reduction of value-estimation error as the refinement budget increases, while ablations demonstrate that the gains are not tied to a single compaction implementation (§ 3.4 and § 3.5).

In summary, our contributions are:

  • We cast the combination of critic predictions and continuation evidence as an active critic learning problem (§ 2.2 and § 2.3).

  • We introduce FOCUS, a highly customizable rollout controller, which allocates a limited rollout budget according to estimated value-estimation error reduction and integrates the resulting estimates with credit assignment.

  • We show that FOCUS improves over critic-free baselines on long-horizon benchmarks (§ 3).

Figure 1. Overview of the proposed FOCUS rollout scheduler for a deep-research-style problem. FOCUS continues working on a rollout tree. At each iteration, FOCUS STEP 1: takes an expanded tree; STEP 2: scores each compaction-induced state with the critic; STEP 3: compares against outcomes to decide critic quality (§ 2.2); STEP 4: sources additional continuations that are expected to correct critic errors most efficiently (§ 2.3). With this active learning loop for rollout acquisition, FOCUS provides a more accurate value estimate with limited rollout budgets (§ 3.4).

Method

Problem Setup

We consider reinforcement learning of a language-model policy πθ\pi_\theta from outcome rewards. The policy generates responses or actions from its current context, and a critic VϕV_\phi, parameterized by ϕ\phi, estimates the expected remaining return. We assume that at least some of the intermediate states are easily forkable and that repeated sampling from them is trivial. An episode ξ\xi records the generated turns and rewards from one completed run.

Given a finite rollout budget, FOCUS selects from candidate states from which to sample additional continuations, refines their value estimates, and constructs training samples for both the critic and the policy. For tool-using agents, compaction boundaries provide natural candidate states. Appendix § 13 details the execution mechanics used in our experiments.

Critic Refinement and Learning

We first describe how collected continuations produce the learning signals for the critic and the policy. § 2.3 describes how we allocate the rollout budget.

States and rollout segments. For each prompt xx, we represent collected rollouts as a tree Tx\mathcal T_x. A nonterminal node denotes a state from which the policy can resume with the same context and environment state; an edge contains the policy-generated continuation and environment transitions until the next state. We write cˉ\bar c for the context of a node (state), leaving its associated environment state implicit.

A completed trajectory contains nonterminal nodes cˉ0=x,cˉ1,…,cˉM\bar c_0=x,\bar c_1,\ldots,\bar c_M and a terminal node cˉM+1\bar c_{M+1}, connected by edges ei=(cˉi−1→cˉi)e_i=(\bar c_{i-1}\to\bar c_i), i=1,…,M+1i=1,\ldots,M+1.

Refining state values. Let Vπ(cˉ)V^\pi(\bar c) denote the expected discounted return of a continuation from cˉ\bar c under the collection policy. The critic Vϕ(cˉ)V_\phi(\bar c) predicts this value from the state’s context. However, this prediction can have high variance. We refine its prediction using the outcome evidence from the n(cˉ)n(\bar c) continuations ch(cˉ)\mathrm{ch}(\bar c) that have been expanded from it, backed up one level as in TreeRPO [64]:

Gˉ(cˉ)  =  1n(cˉ)∑cˉ′∈ch(cˉ)(Rcˉ→cˉ′+γdcˉ→cˉ′ V~(cˉ′)),V~(cˉ′)=0 if cˉ′ is terminal,\begin{equation} \bar G(\bar c) \;=\; \frac{1}{n(\bar c)} \sum_{\bar c' \in \mathrm{ch}(\bar c)} \Big( R_{\bar c \to \bar c'} + \gamma^{d_{\bar c \to \bar c'}}\, \tilde V(\bar c') \Big), \qquad \tilde V(\bar c') = 0 \text{ if } \bar c' \text{ is terminal,} \tag*{(1)} \end{equation}

where V~(cˉ′)\tilde V(\bar c') is the continuation’s own refined value (Eq. (2) below). We prefer this to a raw average of descendant outcomes [33] because the allocator chooses which nodes to expand, so the leaves beneath cˉ\bar c are no longer an on-policy draw from cˉ\bar c; the continuation’s refined value carries its evidence back up instead, one level at a time. We combine the critic prediction and the recursive backup with a fixed pseudo-observation weight κ\kappa,

V~(cˉ)  =  κ Vϕ(cˉ)+n(cˉ) Gˉ(cˉ)κ+n(cˉ).\begin{equation} \tilde V(\bar c) \;=\; \frac{\kappa\, V_\phi(\bar c) + n(\bar c)\,\bar G(\bar c)}{\kappa + n(\bar c)}. \tag*{(2)} \end{equation}

For n(cˉ)=0n(\bar c)=0, we set V~(cˉ)=Vϕ(cˉ)\tilde V(\bar c)=V_\phi(\bar c) without evaluating Gˉ\bar G; terminal values are zero. As n(cˉ)n(\bar c) increases, the blend places more weight on the backed-up mean [47, 52].

Critic supervision and actor advantages. After collection and the final backup, we train the critic on the recursive target Gˉ\bar G at every non-terminal state:

Lcritic(ϕ)  =  E[∑cˉ∈Tx:  n(cˉ)>0(Vϕ(cˉ)−Gˉ(cˉ))2].\begin{equation} \mathcal{L}_{\mathrm{critic}}(\phi) \;=\; \mathbb{E}\Big[\sum_{\bar c \in \mathcal{T}_x:\; n(\bar c) > 0} \big(V_\phi(\bar c) - \bar G(\bar c)\big)^2\Big]. \tag*{(3)} \end{equation}

The target is held fixed during training. This update transfers the collected continuation evidence into the critic used in subsequent training. Within the current iteration, the refined values V~\tilde V supply the segment-level temporal-difference residual2

A^i=δi  =  Ri+γdiV~(cˉi)−V~(cˉi−1).\begin{equation} \hat A_i = \delta_i \;=\; R_i + \gamma^{d_i}\tilde V(\bar c_i) -\tilde V(\bar c_{i-1}). \tag*{(4)} \end{equation}

We apply each edge’s detached advantage to its policy-generated task and context-management tokens, denoted yeiy_{e_i}, masking environment observations and conditioning text. With πold\pi_{\mathrm{old}} the collection policy, ρt\rho_t the token-level importance ratio, and ϵ\epsilon the clipping threshold, the actor minimizes

Lactor(θ)  =  − E[∑i  ∑t∈yeimin⁡(ρtA^i,  clip(ρt,1−ϵ,1+ϵ) A^i)],ρt=πθ(yt∣y<t)πold(yt∣y<t).\begin{equation} \mathcal{L}_{\mathrm{actor}}(\theta) \;=\; -\,\mathbb{E}\Big[\sum_{i}\;\sum_{t \in y_{e_i}} \min\big(\rho_t \hat A_i,\; \mathrm{clip}(\rho_t, 1-\epsilon, 1+\epsilon)\,\hat A_i\big)\Big], \qquad \rho_t = \frac{\pi_\theta(y_t \mid y_{<t})}{\pi_{\mathrm{old}}(y_t \mid y_{<t})}. \tag*{(5)} \end{equation}

Appendix § 11.1 gives the general TD(λ\lambda) extension.

Branching continuations from candidate states supplies additional value evidence [23, 17, 33, 12], but uniform allocation does not account for differences in critic error or the expected benefit of another sample. Given an explored tree Tx\mathcal T_x and an additional rollout budget BB, FOCUS chooses which candidate node receives the next continuation. The aim is to acquire evidence where it is expected to reduce value estimation error the most [42, 38].

Outcome evidence for selection. While selection uses the same formula for mixing observed continuation outcomes and critic predictions, it does not use the continuation-refined backups of its children as in Eq. (1). Specifically, let N(cˉ)N(\bar c) be the number of continuation outcomes and Gˉobs(cˉ)\bar G_{\mathrm{obs}}(\bar c) their mean. A completed continuation contributes one outcome at its starting node and at each newly created node along that continuation. Later branching from a descendant does not add more observations to its strict ancestors. Thus N(cˉ)N(\bar c) counts observed continuation outcomes, whereas n(cˉ)n(\bar c) in Eq. (1) counts direct children.

For binary continuation outcomes, the selection model uses a Beta prior of strength κ\kappa centered on v=Vϕ(cˉ)v=V_\phi(\bar c). Its posterior mean and expected squared critic error are

μobs=κv+NGˉobsκ+N,e2^=(v−μobs)2+μobs(1−μobs)κ+N+1.\begin{equation} \mu_{\mathrm{obs}} = \frac{\kappa v + N\bar G_{\mathrm{obs}}}{\kappa+N}, \qquad \widehat{e^2} = (v-\mu_{\mathrm{obs}})^2 + \frac{\mu_{\mathrm{obs}}(1-\mu_{\mathrm{obs}})}{\kappa+N+1}. \tag*{(6)} \end{equation}

Notice how μobs\mu_{\mathrm{obs}} is distinct from the continuation-refined training value V~\tilde V. The predictive variance of the next outcome is σ2^=μobs(1−μobs)\widehat{\sigma^2}=\mu_{\mathrm{obs}}(1-\mu_{\mathrm{obs}}). In particular, we rely on μobs\mu_{\mathrm{obs}} to conduct the selection due to the observation outlined in Appendix § 10.

Expected benefit of a continuation. We score the expected utility of a continuation using the marginal MSE reduction of value estimation error at that candidate state. For critic squared error e2e^2 and outcome variance σ2\sigma^2, the acquisition function becomes MSE(N)=(κ2e2+Nσ2)/(κ+N)2\mathrm{MSE}(N)=(\kappa^2 e^2+N\sigma^2)/(\kappa+N)^2. Differentiating with e2e^2 and σ2\sigma^2 held fixed, then substituting the posterior estimates gives

g(cˉ)  =  2κ2e2^+(N−κ) σ2^(κ+N)3,\begin{equation} g(\bar c) \;=\; \frac{2\kappa^{2}\widehat{e^2} + (N-\kappa)\,\widehat{\sigma^2}} {(\kappa+N)^{3}}, \tag*{(7)} \end{equation}

where N=N(cˉ)N=N(\bar c) counts completed continuation outcomes.


\caption{Selective rollout allocation and value refinement, with continuation helper}
\label{alg:refine}
\small
\algrenewcommand{\algorithmiccomment}[1]{\hfill\(\triangleright\) {\scriptsize #1}}
\begin{algorithmic}
\Require tree $\mathcal T_x$; fixed actor $\pi_\theta$ and critic $V_\phi$; budget $B$; prior strength $\kappa$
\Require environment $\mathcal E$; branch predicate $\mathcal B$; compaction trigger $\mathcal G$ and operator $\mathcal C_\pi$
\Ensure tree $\mathcal T_x$ with critic targets $\bar G$ and refined values $\tilde V$
\State initialize $N,\bar G_{\mathrm{obs}}$ from completed suffixes; refresh $\mathcal F,g$
\algblocktitle{allocation}{\S\ref{sec:method-search}: Selective rollout allocation}
\While{$B>0\ \land\ \mathcal F\ne\emptyset$}
  \State $\bar c\gets\arg\max_{u\in\mathcal F}g(u)$; $\xi\gets\textsc{Continue}(\bar c,\pi_\theta)$
  \If{$\xi=\bot$}
    \State $B\gets B-1$ \Comment{execution failure}
  \Else
    \State $\mathcal T_x\gets\textsc{Attach}(\mathcal T_x,\bar c,\xi)$
    \State $B\gets B-d(\xi)$ \Comment{$d(\xi)$: new task transitions}
    \State $N(u),\bar G_{\mathrm{obs}}(u)\gets0$ for $u\in\mathrm{New}(\xi)$ \Comment{new nodes}
    \State $\textsc{Score}(\mathcal T_x,V_\phi)$ \Comment{cache unscored nonterminal values}
    \For{$u\in\{\bar c\}\cup\mathrm{New}(\xi)$} \Comment{$G_\xi(u)$: observed suffix return}
      \State $\bar G_{\mathrm{obs}}(u)\gets\dfrac{N(u)\bar G_{\mathrm{obs}}(u)+G_\xi(u)}{N(u)+1}$; $N(u)\gets N(u)+1$
    \EndFor
  \EndIf
  \State refresh $\mathcal F,g$
\EndWhile\algblockend{allocation}
\Statex
\algblocktitle{refinement}{\S\ref{sec:method-credit}: Value refinement and critic targets}
\State $\tilde V(u)\gets0$ for terminal nodes; $\tilde V(u)\gets V_\phi(u)$ for nonterminal leaves
\For{$u\in\mathcal T_x$ with $n(u)>0$, in reverse topological order}
  \State $\bar G(u)\gets\dfrac{1}{n(u)}\displaystyle\sum_{v\in\mathrm{ch}(u)}\bigl(R_{u\to v}+\gamma^{d_{u\to v}}\tilde V(v)\bigr)$
  \State $\tilde V(u)\gets\dfrac{\kappa V_\phi(u)+n(u)\bar G(u)}{\kappa+n(u)}$
\EndFor\algblockend{refinement}
\Statex
\State \Return $\mathcal T_x$
\Statex
\Function{Continue}{$\bar c,\pi$}
  \State $(c,s)\gets\textsc{Fork}(\bar c)$; $\xi\gets[\,]$; $\zeta\gets[\,]$ \Comment{$\zeta$: current segment}
  \While{$\neg\textsc{Terminal}(s)$}
    \algblocktitle[3em]{compaction}{Modular context management}
    \If{$\mathcal G(c)=1$}
      \State $c\gets\mathcal C_\pi(c)$\algblockend{compaction}
    \Statex
    \Else
      \State $\tau\sim\pi(\cdot\mid c)$; $(s,o,r)\sim\mathcal E(s,a(\tau))$; $c\gets c\,\|\,\tau\,\|\,o$
    \EndIf
    \State $\eta\gets\textsc{Trace}()$; $\zeta\gets\zeta\,\|\,\eta$ \Comment{task/compaction turns, rewards, transitions}
    \State $s\gets\textsc{Update}(s,\eta)$ \Comment{counters and termination flags}
    \State \textbf{if} $\textsc{Failed}(s)$ \textbf{then return} $\bot$
    \If{$\textsc{Terminal}(s)\ \lor\ \mathcal B(c,s)=1$} \Comment{$\mathcal B$ independent of $\mathcal G$}
      \State $\bar c'\gets c$; $\xi\gets\xi\,\|\,[(\bar c',\textsc{Snapshot}(s),\zeta)]$; $\zeta\gets[\,]$ \Comment{save resumption state}
    \EndIf
  \EndWhile
  \State \Return $\xi$
\EndFunction
\end{algorithmic}
\algblockbox{allocation}{AlgAllocation}
\algblockbox{refinement}{AlgRefinement}
\algblockbox{compaction}{AlgContext}

Algorithm alg:refine first acquires rollout evidence (§ 2.3), then refines values bottom-up (§ 2.2). Each continuation completes before the next allocation decision. The eligible set is F={cˉ∈Tx:cˉ nonterminal and resumable, g(cˉ)>0}\mathcal F=\{\bar c\in\mathcal T_x:\bar c\text{ nonterminal and resumable},\, g(\bar c)>0\}. The included \textsc{Continue}(\bar c,\pi) helper returns new nodes and connecting segments through termination. Since gg uses observed suffix statistics and fixed critic predictions, value refinement can follow collection. Appendix § 11 gives the full training loop.

Practical considerations. Total branching width at any node is uncapped. The allocation phase can also run asynchronously, with each completed continuation updating the acquisition statistics while others remain in flight. In this case, we replace NN in Eq. (7) by Neff(cˉ)=N(cˉ)+m(cˉ)N_{\mathrm{eff}}(\bar c)=N(\bar c)+m(\bar c), where mm counts pending continuations; posterior estimates still use completed outcomes only. We limit concurrent continuations per node to C=2C=2 and globally to worker capacity WW, restricting eligibility to m(cˉ)<Cm(\bar c)<C. Completed children do not consume concurrency slots. In practice, the two loops can run concurrently, each with multiprocessing. After the launch budget is exhausted, pending continuations finish and a final bottom-up refinement ensures that the training targets reflect the completed tree.

Experiments

We first compare FOCUS with PPO and GRPO in a preliminary evaluation on MATH [16] problems at difficulty levels 4–5, tracking accuracy, evaluation output length, and per-rollout reduction over training updates. We then compare FOCUS to GRPO and VinePPO on three long-horizon benchmarks at Qwen3.5-9B and Qwen3.5-27B [41] (LoRA), assess how MCTS refinement improves critic estimates, and ablate the compaction operator and refinement rollouts.

Preliminary Evaluation on Mathematical Reasoning

We compare FOCUS, PPO, and GRPO with Qwen3.5-4B [41] on MATH Level 4–5 problems for 32 completed updates, evaluating the base checkpoint and checkpoints at updates 8, 16, 24, and 32. We use 512 training questions from the official MATH training split and 256 validation questions from its official test split.3 The training set contains 226 Level 4 and 286 Level 5 questions; validation contains 123 Level 4 and 133 Level 5 questions.

All methods consume the fixed 512-question order for 32 training steps. PPO uses 64 questions per training step; GRPO and FOCUS each use 16 questions per training step. To keep the training compute roughly matched, GRPO uses group size 6, and FOCUS uses a per-question edge budget of 64, with each edge being a maximum of 256 tokens.

(a) Accuracy.

(b) Tokens.

(c) Per-rollout reduction.

Figure 2. Preliminary evaluation with Qwen3.5-4B on MATH Level 4–5. Accuracy and the mean number of generated tokens per evaluation question are shown at the base checkpoint (update 0) and after 8, 16, 24, and 32 completed updates. Markers denote recorded checkpoints. Panel (c) shows FOCUS’s mean g(c)g(c) over all pre-allocation score evaluations within each update from 1 to 32; shading denotes ±1\pm 1 standard deviation of those scores, computed as the square root of the reported variance.

FOCUS has the highest accuracy at updates 8, 16, and 24 (Figure 2), and is only one correct answer behind GRPO at update 32. FOCUS improves accuracy earlier by update count, hinting at a training-compute advantage. Also at update 32, FOCUS generates 2,782 tokens per evaluation problem on average, compared with 5,747 for GRPO and 7,207 for PPO. We hypothesize this results from FOCUS’s ability to refine value estimates more effectively.

Figure 2c shows that throughout training g(c)g(c) decreases by about 50% from update 1 to update 32, showing that as FOCUS gradually tunes its critic, the expected reduction in value estimation error decreases, suggesting that FOCUS gradually learns to predict the expected outcome more accurately.

Long-horizon datasets

DeonticBench [9] DeonticBench evaluates models on context-grounded deontic reasoning: deriving obligations, permissions, and prohibitions from real statutes and case facts. It comprises 6,232 tasks across U.S. federal tax, airline baggage policy, immigration administration, and state housing law. Applying a long statute to a specific case is a multi-step process, which makes the benchmark a setting for long-horizon agentic reasoning in which the model consults the statutes on demand [10] and a natural testbed for stepwise rollout. We focus on the airline, sara-numeric, and sara-binary subsets, whose statutes are reasonably long and involve non-trivial long-horizon tasks. The actor harness exposes (t) list;, (t) grep;, (t) read;, and (t) python;. We stratify the data into equally sized training and held-out splits.

BrowseComp-Plus [3] BrowseComp-Plus evaluates deep search agents on complex, multi-hop information-seeking queries over a fixed, human-verified corpus rather than a live web API. The resulting trajectories are long-horizon: iterative query formulation, retrieval, and reflection, which makes the benchmark a natural testbed for stepwise rollout. Following TRACE [51], the actor harness exposes (t) search;, (t) open;, and (t) find; over the fixed corpus. We randomly split the 830 test tasks into 498 train, 83 validation, and 249 held-out tasks.

LocBench [2] LocBench is a code-localization benchmark of 560 real GitHub issues (242 bug reports, 150 feature requests, 139 performance issues, and 29 security issues) from repositories such as transformers, scikit-learn, vllm, and ultralytics; in LocAgent’s construction, bug reports are issues created relatively recently, and security and performance instances are collected via GitHub keyword search. Given only the issue, the agent must locate the files and functions touched by the gold patch. We instantiate it using the harness described in Appendix § 13; the actor exposes (t) list;, (t) grep;, and (t) read; for repository navigation. We score submissions as in LocAgent: file-level and function-level Acc@kk is 11 when every gold location appears in the top kk (added functions are excluded); we report Acc@{1,5}\{1,5\} at the file level and train on file-level recall at k=5k{=}5. We split the instances into equal training and held-out halves, stratified by repository and issue category.

Main results

We compare GRPO, VinePPO, and FOCUS using Qwen3.5-9B on DeonticBench, BrowseComp-Plus, and LocBench, and additionally with Qwen3.5-27B (LoRA) on DeonticBench. Table tab:main-results reports held-out accuracy alongside untrained reference models. All methods train for one epoch under a matched per-step generation budget with compaction on every rollout; protocol, optimization, and compute details are in Appendix § 6.

ModelMethodDeonticBenchBrowseComp
-Plus & 2cLocBench
AirlineSARA-NSARA-BHardOverallFile@1File@5
Reference (no RL)
gpt-oss-20b (medium)2.78.076.814.133.718.654.057.5
Nemotron 3 Nano 30B A3B2.712.089.918.336.119.357.966.6
Qwen3.5-9B5.330.089.919.745.929.762.167.8
Qwen3.5-27B58.760.094.243.773.450.666.779.3
Qwen3.5-9BGRPO14.056.092.828.252.448.270.180.5
VinePPO10.050.092.024.049.446.667.879.3
FOCUS21.352.096.426.856.555.472.481.6
Δ\Deltavs. GRPO+7.3+0.67+14.00+7.3_{+0.67}^{+14.00}−4.0−16.00+8.00-4.0_{-16.00}^{+8.00}+3.6+0.00+7.25+3.6_{+0.00}^{+7.25}−1.4−8.45+5.63-1.4_{-8.45}^{+5.63}+4.1+0.59+7.69+4.1_{+0.59}^{+7.69}+7.2+2.41+12.05+7.2_{+2.41}^{+12.05}+2.3−1.1+5.7+2.3_{-1.1}^{+5.7}+1.1+0.0+3.4+1.1_{+0.0}^{+3.4}
Qwen3.5-27BGRPO60.760.094.243.774.3OmitFill
VinePPO62.758.094.236.674.9OmitFill black!55outside the 27B study
FOCUS62.762.096.443.776.3OmitFill
Δ\Deltavs. GRPO+2.0−4.67+8.67+2.0_{-4.67}^{+8.67}+2.0−6.00+10.00+2.0_{-6.00}^{+10.00}+2.2+0.72+4.35+2.2_{+0.72}^{+4.35}+0.0−7.04+7.04+0.0_{-7.04}^{+7.04}+2.0−1.18+5.33+2.0_{-1.18}^{+5.33}OmitFill

FOCUS ranks first among the trained methods on every benchmark in Table tab:main-results. At Qwen3.5-9B, it leads on DeonticBench overall (56.5 vs. 52.4 for GRPO and 49.4 for VinePPO), reports the strongest BrowseComp-Plus result (55.4 vs. 48.2 for GRPO and 46.6 for VinePPO, each the best held-out checkpoint over training), and localizes best on LocBench (72.4 / 81.6 File@1 / @5, ahead of 70.1 / 80.5 and 67.8 / 79.3). FOCUS also leads across scale: Qwen3.5-27B with LoRA on DeonticBench still gains from FOCUS fine-tuning (76.3 overall vs. 74.3 and 74.9), but the margin over the baselines shrinks. This could partially be because the untuned 27B model already starts much higher (73.4 overall).

Training dynamics. FOCUS attains the highest on-policy reward throughout training on DeonticBench and BrowseComp-Plus (Figure 3), and the highest held-out accuracy on DeonticBench. On DeonticBench and LocBench, the three methods track closely early, then FOCUS pulls ahead and keeps improving or holds a higher plateau, whereas GRPO on DeonticBench peaks mid-run and then regresses. Late in training, GRPO’s mean tool calls collapse on all three benchmarks while VinePPO and FOCUS stay near their initial levels: GRPO increasingly (t) submit;s right after compaction even when the statute or repository still needs reading (Appendix § 12). We hypothesize that trajectory-level group credit makes such premature submissions easy to reinforce, whereas VinePPO’s suffix branching and FOCUS’s refined checkpoint values preserve longer horizons.

Figure 3. Training dynamics. Each column is one benchmark, showing training reward against optimizer step (top), held-out evaluation (middle), and mean numbers of tool calls per episode on the held-out split (bottom), for GRPO, VinePPO, and FOCUS under a matched per-step generation budget. Top panels plot mean terminal reward from on-policy collection; LocBench held-out reward is File@5. BrowseComp-Plus results are the equivalence judgement of Qwen3-32B [40].

Value refinement

To assess how efficient the FOCUS correction process is in reducing the critic error for long-horizon tasks, as well as whether the critic can improve throughout training, we conduct two diagnostic analyses.

Critic Error Reduction We start by assuming two types of imperfect critics: a constant critic init and a random critic init, and we compare how different priority schedules affect the critic error reduction against a ground truth determined by the mean outcome of n=16n{=}16 MCTS continuations. Figure 4a shows that under both types of critics, FOCUS’s correction process is noticeably more efficient than random selection. Also, using the raw critic value residual performs poorly. This shows that FOCUS-style scheduling not only works well under degenerate critics but also reduces variance even when the critic provides almost no information.

Critic Improvement We also test whether the critic improves over training. At each LocBench evaluation checkpoint, we sample n=16n{=}16 MCTS continuations from a set of m=8m{=}8 completed traces. We then compute the Spearman correlation between the critic value and the MCTS ground truth. Figure 4b shows that the FOCUS critic improves steadily.

(a) Critic MSE vs. budget (DeonticBench).

(b) Critic Spearman ρ\rho (LocBench).

Figure 4. Value refinement for FOCUS. (a) MSE on fixed audit trees as MCTS continuations are allocated by each rule (solid: constant critic init; dashed: random critic init). (b) Rank agreement between checkpoint critic values and an oversampled, critic-blind MCTS target on fixed audit states.

Ablation studies

We ablate the compaction operator and the refinement rollouts to test that FOCUS is not tied to one compactor and that its gains come from active error correction, not the critic alone.

Compaction operator. On DeonticBench, we replace the in-context compaction operator, which writes everything into the new context window, with an in-memory workspace that provides its own (t) put;, (t) get;, (t) scan; and (t) delete;. The two perform similarly (Figure 5a), so FOCUS’s effectiveness extends to different compaction operators.

Refinement rollouts. On BrowseComp-Plus, we remove the refinement rollouts of § 2.3, which amounts to collecting independent continuations from the question prompt and estimating step-wise advantages from the critic alone. Figure 5b shows that refinement rollouts are crucial for FOCUS on BrowseComp-Plus; without them, FOCUS trains more slowly, especially early on.

Structural compaction. We also compare a free-form compaction prompt with a structural prompt that induces a more constrained compaction on LocBench (Figure 5c; both prompts are reproduced in Appendix § 9). The free-form prompt suggests headings and keeps the reply verbatim. The structural prompt requires a single summary with four sections (State, Evidence, Open questions, Next steps) and rejects replies that do not parse. The free-form prompt reaches a higher peak, while the structural prompt improves more steadily.

(a) Compaction operator (DeonticBench).

(b) Refinement rollouts (BrowseComp-Plus).

(c) Structural compaction (LocBench).

Figure 5. Ablations at evaluation checkpoints. (a) Varying the compaction operator on DeonticBench under a matched budget; (b) FOCUS with full refinement vs. FOCUS without refinement rollouts on BrowseComp-Plus; (c) free-form vs. structural compaction on LocBench (File Acc@5).

Related Work

Structured rollouts and rollout allocation Search over intermediate states began as an inference-time technique: LLM tree search [65, 13, 75] and verifier-guided test-time scaling [35, 48] improved model responses. It was later used offline to generate training supervision [54, 69, 11] and then moved into the RL loop. VinePPO [23], SPO [12], and TEMPO [55] re-sample continuations from intermediate reasoning steps to replace the value network with Monte Carlo estimates; TreeRL [17], TreePO [33], and DeepSearch [62] run the search inside RLVR training so that shared prefixes cut generation cost and the tree yields process-level signal. Other work proposes where to branch: at high-entropy tokens [73, 1], at attention peaks [36], where the model’s belief shifts [28], or, for tool-using agents, at the turn boundaries after an observation [8, 80]. Tree-GRPO [19] makes each agent step a node and shows that intra-tree group objectives amount to step-level preference learning, while PaTR [32] routes the branching budget by a process score, and IGRPO [71] and SIGHT [74] by information gain. T2^2PO [57] instead resamples turns that its token- and turn-level uncertainty signals flag as making no progress. TRACE [81] unifies prompt- and turn-level allocation with a learned predictor of conditional success, directing extra rollouts to the prefixes most likely to yield mixed terminal rewards. FOCUS allocates additional continuations according to their estimated reduction in value-estimation error, using the acquired evidence to refine state values and train a reusable critic.

Active learning and expected error reduction Expected-error-reduction methods select observations according to their predicted effect on the learner’s error. [42] approximate this objective through sampling, while [38] use Bayesian parameter samples to estimate the reduction in predictive loss over an unlabeled validation set without retraining for every candidate query. A related approach selects data to reduce predictive variance, connecting variance reduction to squared-error minimization under an approximately unbiased learner [5]. Influence-function approaches estimate acquisition utility by approximating how additional training data affect a fitted model’s evaluation loss, without retraining for each candidate [26, 63]. FOCUS applies the error-reduction principle to allocating continuation samples among intermediate states.

Critic learning and credit assignment Trajectory-level reward is known to fail for multi-turn agents as it cannot tell a useful turn from a redundant one, and training on it alone is unstable [60, 61, 50]. One response is to place the reward and critic at the level of a turn or step rather than a token (too fine-grained) or a whole trajectory (too coarse) [77, 78, 29, 70, 72]. A turn is the natural unit of decision for an agent; the horizon measured in turns is short enough for a critic to bootstrap over, and a per-turn value gives every turn its own signal from a single rollout. Large-scale training is also shifting from structural comparison to a learned value: SAO [18] replaces GRPO’s group sampling with one rollout per prompt scored by a value model, as full groups are slow and off-policy under asynchronous training. Some process reward models serve as fixed value estimators during policy optimization [58, 37, 31, 4, 44, 51]. More recently, Le Critique [56] mixes a learned baseline with a training-free group-relative one according to how accurate the critic is. FOCUS uses refined state values to construct segment-level actor advantages and recursive backups to supervise the critic, coupling credit assignment in the current iteration with critic learning for subsequent iterations.

Context management for long-horizon agents Compaction is becoming standard for agents that outrun their context window [39, 66, 49, 30], and there is a recent push to train it jointly with the task rather than treat it as a frozen preprocessor [6, 34]. State restoration is also central to identifying step-level credit [68]. In our long-horizon experiments, compaction boundaries provide resumable states for acquiring continuation evidence. The rollout controller operates on these states independently of the compaction operator; its objective is to refine value estimates and construct critic supervision through selective rollouts.

Conclusion

We present FOCUS, a rollout controller for language-model reinforcement learning that refines intermediate value estimates through active critic learning and selective rollouts. FOCUS naturally supports long-horizon agentic RL through flexible and modular context-management protocols such as compaction. Across DeonticBench, BrowseComp-Plus, and LocBench, FOCUS improves over GRPO and VinePPO under a matched per-step generation budget, with the largest held-out gains on deep search and deontic reasoning and consistent benefits on code localization. Training curves show that these improvements come with more stable exploration: FOCUS maintains higher collection reward and avoids the late-training collapse in tool use seen under trajectory-level group credit. Ablations indicate that gains are not tied to a single compaction implementation and that refinement rollouts are beneficial on BrowseComp-Plus. We release our modular rollout controller with interfaces for alternative harnesses, search policies, and credit estimators. Future work may scale FOCUS to larger models and longer budgets, tighten the link between compaction quality and training stability, and adopt stronger uncertainty quantification for value refinement.

Acknowledgments

This work was supported in part by the Defense Advanced Research Projects Agency (DARPA) Scientific Feasibility (SciFy) program under contract HR001125C0304; by Schmidt Sciences through the “Safely Confident Reasoning” project in its Science of Trustworthy AI program; and by the Office of Naval Research (ONR) under grant N00014-24-1-2089. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of DARPA, the Office of Naval Research, or Schmidt Sciences. We acknowledge the use of computational resources from the Johns Hopkins DSAI cluster.

Training setup

Tables tab:setup-bc-focus–Table 12 give the training setup of each method on each benchmark, one table per run configuration. Adam is from [25].

Shared FOCUS settings. Across all FOCUS experiments, we use γ=1\gamma=1, λ=0\lambda=0, and κ=1\kappa=1. Thus rewards are undiscounted, actor advantages use the local segment TD residual in Eq. (4), and the critic contributes one pseudo-observation to both the recursive value blend and the selection posterior. Total branching width is uncapped, with at most C=2C=2 concurrent continuations per node; completed children do not count against this concurrency limit. These settings are shared across tasks and model sizes. The selection estimators are defined in Eq. (6) and Appendix § 10; the full training loop and budget semantics are given in Appendix § 11.

Setting Value
Actor model Qwen3.5-9B
Critic model 9B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Seeding budget BseedB_{\mathrm{seed}} (turns/prompt)
Group size Adaptive search
Questions
Root trajectories Set by adaptive search
Compaction trigger (prompt tokens)336
Compaction trigger (tool calls)
Actor reply cap (tokens)384
Compaction reply cap (tokens)384
Agent turn limit (turns)
Context window (tokens)304
Parameterization Full
Optimizer Adam
Actor learning rate 1×10−61\times 10^{-6}
Critic learning rate 1×10−61\times 10^{-6}
Learning-rate schedule Constant
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs–6×\timesH100
Critic GPUs (refinement)×\timesH100

Table 2. GRPO on BrowseComp-Plus, Qwen3.5-9B.

Setting Value
Actor model Qwen3.5-9B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Step advantage Group advantage broadcast over steps
Group size
Questions
Root trajectories
Compaction trigger (prompt tokens)336
Compaction trigger (tool calls)
Actor reply cap (tokens)384
Compaction reply cap (tokens)384
Agent turn limit (turns)
Context window (tokens)304
Parameterization Full
Optimizer Adam
Actor learning rate 1×10−61\times 10^{-6}
Learning-rate schedule Constant
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs–6×\timesH100

Table 2. GRPO on BrowseComp-Plus, Qwen3.5-9B.

Setting Value
Actor model Qwen3.5-9B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Branching states Compaction points
Value-only continuations per state (k=1k{=}1)
Group size
Questions
Root trajectories
Compaction trigger (prompt tokens)336
Compaction trigger (tool calls)
Actor reply cap (tokens)384
Compaction reply cap (tokens)384
Agent turn limit (turns)
Context window (tokens)304
Parameterization Full
Optimizer Adam
Actor learning rate 1×10−61\times 10^{-6}
Learning-rate schedule Constant
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs–6×\timesH100

Table 4. FOCUS on DeonticBench, Qwen3.5-9B.

Setting Value
Actor model Qwen3.5-9B
Critic model 9B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Seeding budget BseedB_{\mathrm{seed}} (turns/prompt)
Group size Adaptive search
Questions
Root trajectories Set by adaptive search
Compaction trigger (prompt tokens)336
Compaction trigger (tool calls)
Actor reply cap (tokens)144
Compaction reply cap (tokens)096
Agent turn limit (turns)
Context window (tokens)768
Parameterization Full
Optimizer Adam
Actor learning rate 1×10−61\times 10^{-6}
Critic learning rate 5×10−65\times 10^{-6}
Learning-rate schedule Constant
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs–6×\timesH100
Critic GPUs (refinement)×\timesH100

Table 4. FOCUS on DeonticBench, Qwen3.5-9B.

Setting Value
Actor model Qwen3.5-9B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Step advantage Group advantage broadcast over steps
Group size
Questions
Root trajectories
Compaction trigger (prompt tokens)336
Compaction trigger (tool calls)
Actor reply cap (tokens)144
Compaction reply cap (tokens)096
Agent turn limit (turns)
Context window (tokens)768
Parameterization Full
Optimizer Adam
Actor learning rate 1×10−61\times 10^{-6}
Learning-rate schedule Constant
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs–6×\timesH100

Table 6. VinePPO on DeonticBench, Qwen3.5-9B.

Setting Value
Actor model Qwen3.5-9B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Branching states Compaction points
Value-only continuations per state (k=1k{=}1)
Group size
Questions
Root trajectories
Compaction trigger (prompt tokens)336
Compaction trigger (tool calls)
Actor reply cap (tokens)144
Compaction reply cap (tokens)096
Agent turn limit (turns)
Context window (tokens)768
Parameterization Full
Optimizer Adam
Actor learning rate 1×10−61\times 10^{-6}
Learning-rate schedule Constant
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs–6×\timesH100

Table 6. VinePPO on DeonticBench, Qwen3.5-9B.

Setting Value
Actor model Qwen3.5-9B
Critic model 9B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Seeding budget BseedB_{\mathrm{seed}} (turns/prompt)
Group size Adaptive search
Questions
Root trajectories Set by adaptive search
Compaction trigger (prompt tokens)768
Compaction trigger (tool calls) (counter rolls over independently)
Actor reply cap (tokens)384
Compaction reply cap (tokens)384
Agent turn limit (turns) task turns incl. compactions; final submission allowed afterward
Context window (tokens)536
Parameterization Full
Optimizer Adam
Actor learning rate 1×10−61\times 10^{-6}
Critic learning rate 1×10−61\times 10^{-6}
Learning-rate schedule Constant
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs–6×\timesH100
Critic GPUs (refinement)×\timesH100

Table 8. GRPO on LocBench, Qwen3.5-9B.

Setting Value
Actor model Qwen3.5-9B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Step advantage Group advantage broadcast over steps
Group size
Questions
Root trajectories
Compaction trigger (prompt tokens)576
Compaction trigger (tool calls)
Actor reply cap (tokens)192
Compaction reply cap (tokens)240
Agent turn limit (turns)
Context window (tokens)304
Parameterization Full
Optimizer Adam
Actor learning rate 1×10−61\times 10^{-6}
Learning-rate schedule Constant
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs–6×\timesH100

Table 8. GRPO on LocBench, Qwen3.5-9B.

Setting Value
Actor model Qwen3.5-9B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Branching states Compaction points
Value-only continuations per state (k=1k{=}1)
Group size
Questions
Root trajectories
Compaction trigger (prompt tokens)576
Compaction trigger (tool calls)
Actor reply cap (tokens)192
Compaction reply cap (tokens)240
Agent turn limit (turns)
Context window (tokens)304
Parameterization Full
Optimizer Adam
Actor learning rate 1×10−61\times 10^{-6}
Learning-rate schedule Constant
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs–6×\timesH100

Table 10. FOCUS on DeonticBench, Qwen3.5-27B.

Setting Value
Actor model Qwen3.5-27B
Critic model 4B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Seeding budget BseedB_{\mathrm{seed}} (turns/prompt)
Group size Adaptive search
Questions
Root trajectories Set by adaptive search
Compaction trigger (prompt tokens)336
Compaction trigger (tool calls)
Actor reply cap (tokens)144
Compaction reply cap (tokens)096
Agent turn limit (turns)
Context window (tokens)768
Parameterization LoRA
LoRA (r,α,dropout)(r, \alpha, \text{dropout}) (16, 32, 0)(16,\ 32,\ 0)
Optimizer Adam
Actor learning rate 3×10−53\times 10^{-5}
Critic learning rate 2×10−52\times 10^{-5}
Value clip
Learning-rate schedule Constant, no warmup
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs×\timesH200
Critic GPUs (refinement)×\timesH200

Table 10. FOCUS on DeonticBench, Qwen3.5-27B.

Setting Value
Actor model Qwen3.5-27B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Step advantage Group advantage broadcast over steps
Group size
Questions
Root trajectories
Compaction trigger (prompt tokens)336
Compaction trigger (tool calls)
Actor reply cap (tokens)144
Compaction reply cap (tokens)096
Agent turn limit (turns)
Context window (tokens)768
Parameterization LoRA
LoRA (r,α,dropout)(r, \alpha, \text{dropout}) (16, 32, 0)(16,\ 32,\ 0)
Optimizer Adam
Actor learning rate 3×10−53\times 10^{-5}
Learning-rate schedule Constant, no warmup
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs×\timesH200

Table 12. VinePPO on DeonticBench, Qwen3.5-27B.

Setting Value
Actor model Qwen3.5-27B
Training length epoch
Training-data order Identical across methods
Compaction operator On
Rollout temperature
Advantage cutoff
Branching states Compaction points
Value-only continuations per state (k=1k{=}1)
Group size
Questions
Root trajectories
Compaction trigger (prompt tokens)336
Compaction trigger (tool calls)
Actor reply cap (tokens)144
Compaction reply cap (tokens)096
Agent turn limit (turns)
Context window (tokens)768
Parameterization LoRA
LoRA (r,α,dropout)(r, \alpha, \text{dropout}) (16, 32, 0)(16,\ 32,\ 0)
Optimizer Adam
Actor learning rate 3×10−53\times 10^{-5}
Learning-rate schedule Constant, no warmup
Adam (β1,β2,ϵ)(\beta_1, \beta_2, \epsilon) (0.9, 0.95, 10−8)(0.9,\ 0.95,\ 10^{-8})
Weight decay
Gradient-norm clip
PPO clip ratio
KL coefficient
Entropy bonus
Training GPUs×\timesH200
Rollout GPUs×\timesH200

Table 12. VinePPO on DeonticBench, Qwen3.5-27B.

Bootstrap procedure for main-result differences

Table tab:main-results reports FOCUS minus GRPO for every metric. We keep each reported checkpoint fixed during resampling. For LocBench, we draw 100,000 bootstrap samples of the original case count with replacement, using the same sampled case indices for FOCUS and its comparator. We compute the accuracy difference in each replicate; the 5th and 95th percentiles define a two-sided 90% percentile interval. Differences use exact counts before rounding where available. We use seed 20261021. Numerical intervals appear only in the additional rows of Table tab:main-results.

DeonticBench, Qwen3.5-9B. We use the corrected evaluation breakdown and its supplied paired 90% intervals. The evaluation contains 150 Airline, 50 SARA-N, and 138 SARA-B cases (338 total); the 71 Hard cases overlap these domains. The corrected FOCUS accuracies correspond to 32, 26, and 133 correct answers in the three domains, respectively (191 overall), with 19 correct on Hard. GRPO has 21, 28, and 128 correct (177 overall), with 20 correct on Hard. The supplied intervals are preserved rather than recomputed from the historical per-case export, which does not represent these corrected results.

LocBench. We pair 87 held-out query identifiers. FOCUS uses the earlier-run checkpoint 35 for both File@1 and File@5. GRPO’s File@1 score comes from checkpoint 1, and its File@5 score from checkpoint 89. Checkpoint 96 ties checkpoint 89 on aggregate File@5; we use the earlier matching checkpoint. Thus the two columns compare different GRPO checkpoints, consistent with the separately reported metric scores.

BrowseComp-Plus. The point difference uses the reported FOCUS and GRPO scores on 249 questions. We use the supplied 90% paired interval for this comparison; the underlying per-question outcomes are not included in the local analysis export.

DeonticBench, Qwen3.5-27B. We report point differences from the rounded FOCUS and GRPO manuscript scores. For Overall, the supplied paired counts are 27 cases correct only for FOCUS, 20 correct only for GRPO, and 291 with concordant outcomes (338 total). For Hard, the corresponding counts are 5, 5, and 61 (71 total); for SARA-B, they are 3, 0, and 135 (138 total); and for SARA-N, they are 4, 3, and 43 (50 total). Subtracting the SARA-N and SARA-B counts from Overall gives Airline counts of 20, 17, and 113 (150 total), assuming these domain counts refer to the same evaluation. We bootstrap these paired differences using 1,000,000 multinomial resamples with seed 20261021 and take the 5th and 95th percentiles.

The intervals are marginal, without multiple-comparison adjustment, and conditional on the selected checkpoints. They do not account for held-out checkpoint selection, training-seed variation, or repeated stochastic decoding. Case resampling also does not model dependence among related problem families or repository issues. Reproduction inputs and checkpoint provenance are preserved with the analysis script.

BrowseComp-Plus cold-starting

We cold-start BrowseComp-Plus with supervised imitation before the matched RL runs in § 3. We first collect 200 successful, uncompacted trajectories from Qwen3.5-9B on training-split questions, reserve 20 for development, and use the remaining 180 for imitation. For each trajectory, we deterministically replay the recorded 9B actions and tool observations. Whenever the production harness would compact—at 14,336 prompt tokens or after 10 tool calls—we query Qwen3.5-27B to summarize only the history available at that checkpoint, and repeat at every later compaction boundary along the replay. The teacher prompt requires evidence, settled requirements, and genuinely open questions, and fixes the Next steps field to “None” so compactions do not instruct the actor to repeat verification. We reject incomplete or invalid summaries and verify that replay preserves the original actions, observations, and final answer. Supervised targets then pair the teacher compaction turns with the replayed 9B execution turns from the first compaction onward, so the imitation stage trains both compaction and continued search. Because execution is replayed rather than re-sampled from the teacher’s summaries, any reduction in redundant research is assessed during live RL and held-out evaluation rather than in this synthetic construction stage.

Compaction prompts

Figure 5c compares two compaction prompts on LocBench. In both, compaction runs under a NullWorkspace: it has no tools and no store, so its reply is the agent’s only memory once the transcript is discarded. The prompts differ in how the reply must be shaped, whether it is checked, and how it re-enters the agent’s context (Table 13). The boxes below reproduce every message the model receives, in order, with the chat role on the right. Line breaks are reflowed, (s) slots; mark values filled at run time, and small grey labels name the source of each segment. Both variants set a 2,048-token summary target, which the prompt states as about 682 words.

Free-form Structural
Compaction instruction LocBench-specific instruction with the NullWorkspace guidance inserted Generic instruction with the NullWorkspace guidance, followed by a LocBench schema and an output contract
Reply format Four suggested headings; the reply is kept verbatim Exactly one <summary> block with four non-empty sections: State, Evidence, Open questions, Next steps
Validation Rejected if empty, cut off at the reply cap, or inside unclosed reasoning Also parsed into the four sections and held to the 2,048-token target; one repair retry. Compaction turns are penalized −0.01-0.01 on failure and −0.002-0.002 when the wrapper is recovered
Next steps Always “None” A short next action; any reread must match an Open question
Re-entry Notes appended to the original request inside <yournotes> (user turn) Request left untouched; notes in their own <assistantmemory> turn, then a resume instruction
Execution state None Harness-rendered JSON of executed calls, counters, and submissions
Reread rule Re-establish from the task’s sources whatever the notes leave open Treat State and Evidence as settled; reread only items listed under Open questions
Compaction sampling Library default (temperature 0.70.7) Temperature 0.20.2, top-pp 0.950.95
Compaction trigger Prompt length or spent per-round call budget Prompt length only

Table 13. Free-form versus structural compaction on LocBench. Both compactions are tool-free (NullWorkspace) and must finish within the reply cap.

Free-form compaction

The compaction turn sees an instruction and the wrapped transcript. Its reply is spliced into the agent’s original request.

Structural compaction

This prompt was used by the robust-null Improved PPO runs in Figure 5c. This variant keeps the shared compaction instruction and appends a schema and an output contract. A reply is accepted only if it parses into the four sections. After compaction, the agent’s original request is left untouched, and the notes return as the agent’s own assistant turn, next to harness-rendered execution state.

Outcome evidence for selective rollout allocation

FOCUS uses continuation evidence in two distinct ways: to refine state values for learning (§ 2.2) and to select candidate states for additional continuations (§ 2.3). This appendix explains why selective rollout allocation uses observed continuation outcomes rather than the recursive backups used for training. Throughout a collection round, the policy πθ\pi_\theta and critic VϕV_\phi are fixed. As in the methodology, cˉ\bar c denotes a state’s context, with its associated environment state implicit, and v=Vϕ(cˉ)v=V_\phi(\bar c) denotes the critic prediction at that state.

For critic refinement, n(cˉ)n(\bar c) counts direct children in Tx\mathcal T_x. The recursive backup Gˉ(cˉ)\bar G(\bar c) averages their contributions Rcˉ→cˉ′+γdcˉ→cˉ′V~(cˉ′)R_{\bar c\to\bar c'}+ \gamma^{d_{\bar c\to\bar c'}}\tilde V(\bar c'), and the refined state value V~(cˉ)\tilde V(\bar c) combines this backup with the local critic prediction (Eqs. (1)–(2)). For allocation, N(cˉ)N(\bar c) counts observed continuation outcomes, whose mean is Gˉobs(cˉ)\bar G_{\mathrm{obs}}(\bar c). These observations determine the posterior mean μobs\mu_{\mathrm{obs}} and the estimates e2^\widehat{e^2} and σ2^\widehat{\sigma^2} used in the acquisition score g(cˉ)g(\bar c) (Eqs. (6)–(7)). In particular, μobs\mu_{\mathrm{obs}} does not use downstream refined state values.

Recursive backups attenuate the residual used for selection. Consider a completed continuation that forms a chain with one direct child at each nonterminal state, zero intermediate rewards, terminal outcome zz, and γ=1\gamma=1. Suppose the critic predicts the same value vv at every nonterminal state and κ>0\kappa>0 is fixed. Index these states by j≥1j\geq1, the number of edges remaining until termination, and write Gˉj\bar G_j and V~j\tilde V_j for their recursive backups and refined state values. Since n=1n=1 at each state, setting a=(κ+1)−1a=(\kappa+1)^{-1} gives

Gˉj=v+(z−v)aj−1,V~j=v+(z−v)aj,(v−Gˉj)2=(v−z)2a2(j−1).\begin{equation} \bar G_j=v+(z-v)a^{j-1}, \qquad \tilde V_j=v+(z-v)a^j, \qquad (v-\bar G_j)^2=(v-z)^2a^{2(j-1)}. \tag*{(8)} \end{equation}

For j=1j=1, the terminal edge gives Gˉ1=z\bar G_1=z and V~1=(κv+z)/(κ+1)\tilde V_1=(\kappa v+z)/(\kappa+1). For j>1j>1, the zero-reward edge gives Gˉj=V~j−1\bar G_j=\tilde V_{j-1}; substitution into Eq. (2) proves the identities by induction. Thus, using (v−Gˉj)2(v-\bar G_j)^2 as selection evidence would attenuate the same realized critic–outcome discrepancy with every additional recursive backup. Under the default κ=1\kappa=1, this squared residual decreases by a factor of four per edge, even though the critic prediction and observed continuation outcome are identical at every state.

For example, with v=0.95v=0.95 and z=0z=0, the squared residuals (v−Gˉj)2(v-\bar G_j)^2 for j=1,2,3,4j=1,2,3,4 are 0.90250.9025, 0.2256250.225625, 0.056406250.05640625, and 0.01410156250.0141015625, respectively. If this is the initial completed continuation, each state instead has N=1N=1 and Gˉobs=z\bar G_{\mathrm{obs}}=z. For a binary outcome and κ=1\kappa=1, Eq. (6) gives

μobs,j=v+z2,ej2^=(v−v+z2)2+v+z2(1−v+z2)3.\begin{equation} \mu_{\mathrm{obs},j}=\frac{v+z}{2}, \qquad \widehat{e_j^2}= \left(v-\frac{v+z}{2}\right)^2+ \frac{\frac{v+z}{2}\left(1-\frac{v+z}{2}\right)}{3}. \tag*{(9)} \end{equation}

This expected squared critic error equals 0.308750.30875 at every state in the example. The local critic prior still regularizes the estimate, but there is no additional attenuation from downstream refined state values. This observation motivates the use of continuation outcomes for allocation. It does not identify the true critic error from one outcome: the posterior retains sampling uncertainty. With γ<1\gamma<1 or nonzero intermediate rewards, observed continuation returns may also differ across states according to the return definition.

Continuation outcomes under selective rollout allocation. Let H\mathcal H denote the collection history immediately before a continuation is sampled, including the selected candidate state cˉ\bar c. Let G(ξ;cˉ)G(\xi;\bar c) denote the observed discounted return of that continuation. If the policy resumes from the state’s context and associated environment state and samples to termination without filtering by outcome, then

E ⁣[G(ξ;cˉ)∣H,cˉ]=Vπθ(cˉ).\begin{equation} \mathbb E\!\left[G(\xi;\bar c)\mid\mathcal H,\bar c\right] =V^{\pi_\theta}(\bar c). \tag*{(10)} \end{equation}

Conditioning on the realized prefix gives the same statement at each newly created nonterminal state along the continuation, before its remaining suffix is sampled. A completed continuation therefore supplies one observed outcome at its starting state and at each such new state, as specified in § 2.3.

A later continuation allocated to a descendant is an additional draw from that descendant, but not from its strict ancestors: the prefix leading to the selected descendant has not been resampled. Pooling all descendant outcomes at an ancestor would weight prefixes according to the allocation decisions. FOCUS therefore updates NN and Gˉobs\bar G_{\mathrm{obs}} only at the starting state and newly created nonterminal states of the completed continuation. Recursive backups may still update ancestor training values. Equation (10) concerns each conditional draw; it does not imply that a sample mean at an adaptively chosen stopping time is unbiased.

Pseudo-observation weight and initial continuation evidence. The default κ=1\kappa=1 gives the local critic prediction weight 1/(n+1)1/(n+1) in the refined state value V~\tilde V, and weight 1/(N+1)1/(N+1) in the selection posterior mean μobs\mu_{\mathrm{obs}}. The pseudo-observation weight stays fixed as continuations are collected. This is a regularization choice. For the simplified MSE model in § 2.3, with fixed N>0N>0, e2>0e^2>0, and σ2>0\sigma^2>0, minimizing (κ2e2+Nσ2)/(κ+N)2(\kappa^2e^2+N\sigma^2)/(\kappa+N)^2 over κ\kappa gives κ∗=σ2/e2\kappa^*=\sigma^2/e^2. Thus κ=1\kappa=1 is optimal in that model only when e2=σ2e^2=\sigma^2; interpreting the critic as one pseudo-observation does not establish this equality.

For the binary selection model, setting N=0N=0 and κ=1\kappa=1 in Eqs. (6) and (7) gives

μobs=v,e2^=v(1−v)2,σ2^=v(1−v),g(cˉ)=0.\begin{equation} \mu_{\mathrm{obs}}=v,\qquad \widehat{e^2}=\frac{v(1-v)}{2},\qquad \widehat{\sigma^2}=v(1-v),\qquad g(\bar c)=0. \tag*{(11)} \end{equation}

Consequently, the acquisition score does not reduce to uncertainty sampling at a state with no observed continuation outcomes. The initial completed episode supplies one outcome at every nonterminal state on its trajectory, so those states enter allocation with N≥1N\geq1. Additional seeding supplies further continuation evidence.

Recursive backups for critic supervision and actor advantages. Equation (1) assigns one contribution to each direct child, regardless of how many additional continuations are subsequently allocated below that child. New descendant evidence changes the child’s refined state value and propagates through that contribution, preserving the weighting over direct children. The recursive backup Gˉ\bar G supplies the critic supervision target in Eq. (3); the refined state values V~\tilde V supply the actor advantages in Eq. (4). This use of downstream refined values can reduce sampling variance, but also propagates errors in those values and does not guarantee a lower MSE.

Selective rollout allocation instead estimates critic error from observed continuation outcomes, avoiding the additional attenuation in Eq. (8). The acquisition score g(cˉ)g(\bar c) is derived from the simplified MSE model with e2e^2 and σ2\sigma^2 held fixed, then evaluated using their posterior estimates and NN. It remains a heuristic for the expected benefit of a continuation, rather than an exact expected reduction in the recursive training target’s MSE or in actor-gradient error.

Full reinforcement-learning procedure

Algorithm alg:full-rl embeds selective rollout allocation (Algorithm alg:refine) in the actor–critic training loop. All experiments use γ=1\gamma=1, λ=0\lambda=0, κ=1\kappa=1, and C=2C=2; optimizer settings are given in Appendix § 6.


\caption{FOCUS actor--critic training with selective rollout allocation}
\label{alg:full-rl}
\small
\algrenewcommand{\algorithmiccomment}[1]{\hfill\(\triangleright\) {\scriptsize #1}}
\begin{algorithmic}
\Require prompts $\mathcal D$; parameters $(\theta_0,\phi_0)$; rounds $K$; $\gamma,\kappa$
\Require $B_{\mathrm{init}},B_{\mathrm{seed}},B_{\mathrm{ref}}$ \Comment{separate transition budgets; failure costs one unit}
\Require configured $\textsc{Continue}$, collection schedules, and optimizers
\Ensure trained parameters $(\theta_K,\phi_K)$
\Statex \Comment{Finish started continuations; charge overshoot only to the current phase.}
\Statex \Comment{$\textsc{Collect}$ completes and attaches $\textsc{Continue}$ results (Algorithm~\ref{alg:refine}).}
\Statex \Comment{Use the same $\mathcal G$ and $\mathcal C_{\pi_{\mathrm{old}}}$ throughout collection.}
\For{$k=0,\ldots,K-1$}
  \State $(\pi_{\mathrm{old}},V_{\mathrm{old}})\gets(\pi_{\theta_k},V_{\phi_k})$ \Comment{fixed during collection}
  \State $\mathcal X_k\sim\mathcal D$; $\mathcal D_A\gets\emptyset$; $\mathcal D_V\gets\emptyset$
  \For{$x\in\mathcal X_k$}
    \State $(\mathcal T_x,\bar c_0)\gets\textsc{Init}(x)$ \Comment{fresh root and environment}
    \State $\mathcal T_x\gets\textsc{Collect}(\mathcal T_x,\{\bar c_0\},\pi_{\mathrm{old}},B_{\mathrm{init}})$ \Comment{initial episodes}
    \State $\textsc{Score}(\mathcal T_x,V_{\mathrm{old}})$ \Comment{cache unscored nonterminal values}
    \State $\mathcal S_0\gets\textsc{Candidates}(\mathcal T_x)$ \Comment{resumable nonterminal nodes}
    \State $\mathcal T_x\gets\textsc{Collect}(\mathcal T_x,\mathcal S_0,\pi_{\mathrm{old}},B_{\mathrm{seed}})$ \Comment{seed continuations}
    \State $\textsc{Score}(\mathcal T_x,V_{\mathrm{old}})$
    \State $\mathcal T_x\gets\textsc{AllocateAndRefine}(\mathcal T_x,\pi_{\mathrm{old}},V_{\mathrm{old}},B_{\mathrm{ref}},\kappa)$ \Comment{Algorithm~\ref{alg:refine}}
    \For{$e=(\bar c,\bar c')\in\mathcal E_A(\mathcal T_x)$} \Comment{each distinct trainable edge once}
      \State $\hat A_e\gets\operatorname{stopgrad}\!\left(R_e+\gamma^{d_e}\tilde V(\bar c')-\tilde V(\bar c)\right)$
      \Statex \Comment{$y_e,\mu_e,\ell_e$: collected tokens, actor mask, behavior log probabilities}
      \State $\mathcal D_A\gets\mathcal D_A\uplus\{(y_e,\mu_e,\ell_e,\hat A_e)\}$ \Comment{$\uplus$: preserve repeated samples}
    \EndFor
    \For{$\bar c\in\mathcal T_x$ nonterminal, $n(\bar c)>0$}
      \State $\mathcal D_V\gets\mathcal D_V\uplus\{(\bar c,\operatorname{stopgrad}(\bar G(\bar c)))\}$
    \EndFor
  \EndFor
  \State $\mathcal D_A\gets\textsc{Filter}(\mathcal D_A)$ \Comment{optional: drop edges with $|\hat A_e|\leq$ cutoff}
  \Statex \Comment{Except malformed compactions; preserve coefficients and original edge-count normalization.}
  \State $\theta_{k+1}\gets\textsc{ActorUpdate}(\theta_k,\mathcal D_A,\pi_{\mathrm{old}})$ \Comment{Eq.~\ref{eq:actor}; identity if $\mathcal D_A=\emptyset$}
  \State $\phi_{k+1}\gets\textsc{CriticUpdate}(\phi_k,\mathcal D_V)$ \Comment{Eq.~\ref{eq:critic}; identity if $\mathcal D_V=\emptyset$}
  \State $\textsc{Publish}(\theta_{k+1},\phi_{k+1})$ \Comment{synchronize collection workers}
\EndFor
\State \Return $(\theta_K,\phi_K)$
\end{algorithmic}

TD(λ\lambda) advantage extension

For a completed episode, the segment-level TD(λ\lambda) extension [43] of Eq. (4) is

A^i  =  ∑j=iM+1λ j−i (∏l=ij−1γdl) δj,\begin{equation} \hat A_i \;=\; \sum_{j=i}^{M+1} \lambda^{\,j-i}\, \Big(\prod_{l=i}^{j-1} \gamma^{d_l}\Big)\, \delta_j , \tag*{(12)} \end{equation}

where λ∈[0,1]\lambda\in[0,1] controls trace decay across segments. All experiments use λ=0\lambda=0, recovering A^i=δi\hat A_i=\delta_i; λ=1\lambda=1 gives the observed return minus V~(cˉi−1)\tilde V(\bar c_{i-1}).

GRPO submit-after-compaction behavior

The boxes below are held-out episodes from late GRPO checkpoints, rendered from their saved traces by scripts/grpo_premature_cases.py. Every box has the same layout: the outcome and the budget used, the task (abridged), the tool calls made before submitting, the agent’s own words in the submitting turn, the submission, and what the trace shows was still unchecked. Trace text is verbatim; text is evidence the agent had in context, and

text is the claim it submitted on without checking.

Execution Model and Harness Mechanics

This section specifies the tool-using instantiation of the learning setup in § 2.1. The harness manages execution and context, while the rollout controller chooses the states from which to sample additional continuations. The mathematical-reasoning evaluation uses prefix continuations without compaction.

Tool calls and environment state. An environment E\mathcal E exposes tools A\mathcal A, such as (t) grep; and (t) read; over a repository or (t) search; over a fixed corpus. Given context cc, the policy generates a task turn τ∼πθ(⋅∣c)\tau\sim\pi_\theta(\cdot\mid c) ending in a tool call a(τ)∈Aa(\tau)\in\mathcal A. The environment executes the call and returns an updated state, observation, and reward, (s′,o,r)∼E(s,a(τ))(s',o,r)\sim\mathcal E(s,a(\tau)). Here ss contains the task state, workspace, and execution information needed to resume the run; cc denotes the model context. The harness appends the generated turn and observation to the context and retains the updated environment state. In our agent benchmarks, a special (t) submit; call ends the episode with a final response; an episode also ends when its configured turn limit is reached.

Context management and compaction. The policy has a finite context window of LL tokens. When a configured trigger G\mathcal G fires, the harness invokes a compaction procedure Cπθ\mathcal C_{\pi_\theta} to replace the accumulated context with a shorter one. We write G(c)\mathcal G(c) for brevity, suppressing any dependence on execution counters. The triggers used in the experiments include context length and, where configured, a per-round call budget (Appendices § 6 and § 9). Following recent long-horizon training work [6, 14, 34], compaction itself uses policy-generated turns. The harness constructs the resumed context from the resulting notes and configured task instructions. Compaction changes the model’s context while preserving task state and workspace separately; it does not itself advance the task environment.

Starting from prompt xx, the context evolution is

c0=x,ck+1={Cπθ(ck)if G(ck)=1,ck ∥ τk ∥ okotherwise,\begin{equation} c_0=x,\qquad c_{k+1}=\begin{cases} \mathcal C_{\pi_\theta}(c_k) & \text{if }\mathcal G(c_k)=1,\\[2pt] c_k\,\|\,\tau_k\,\|\,o_k & \text{otherwise,} \end{cases} \tag*{(13)} \end{equation}

where τk∼πθ(⋅∣ck)\tau_k\sim\pi_\theta(\cdot\mid c_k) and (sk+1,ok,rk)∼E(sk,a(τk))(s_{k+1},o_k,r_k)\sim\mathcal E(s_k,a(\tau_k)) on task turns, and ∥\| denotes concatenation. This equation describes the visible context; environment transitions and execution counters are maintained separately. Compaction may contain multiple generated turns. An episode records both task and compaction turns, together with their rewards, until termination.

Resumption and segment construction. Here ckc_k denotes the evolving context at execution step kk, while cˉj\bar c_j denotes the context at the jjth selected branchable state, consistent with § 2.2. The \textsc{Continue} helper in Algorithm alg:refine writes the current context as cc and a newly recorded node context as cˉ′\bar c'. The branch predicate B(c,s)\mathcal B(c,s) is independent of the compaction trigger G\mathcal G: a selected state can follow a task transition or a compaction, and compaction need not create a branching node. The compaction update uses the same operator Cπθ\mathcal C_{\pi_\theta} as Eq. (13). Our compaction-based experiments select post-compaction states for branching. To sample a continuation, the runner restores that context together with the corresponding task, workspace, and execution state. It then samples under the fixed collection policy. Preserving the context alone would not suffice when previous actions have changed the environment. Under this experimental choice, an episode with MM compactions has M+1M+1 segments: each contains the task turns up to the next compaction and the compaction turns that construct the next context; the final segment ends at termination. Between context changes, task turns extend the preceding context, permitting sequence training with observations masked [21, 7]. Policy-generated task and compaction tokens receive actor credit; environment observations and fixed harness text are excluded from the actor loss. Appendix § 11 specifies the training procedure.

Modularity and experimental variants. The allocation rule depends on resumable states, observed suffix returns, and critic predictions. The context-management component determines how those states are represented. Setting G≡0\mathcal G\equiv0 disables the optional compaction block in Algorithm alg:refine; candidate nodes then follow the task’s resumption rule, as in the mathematical-reasoning evaluation. Our experiments vary compaction operators and prompts while retaining this separation. Appendix § 9 gives the free-form and structural LocBench prompts, validation and repair rules, auxiliary penalties, and the exact placement of notes and execution metadata in the resumed context. Appendix § 6 reports the context, generation, and episode limits.

The modular design provides interfaces for extensions to richer context management, such as recursive calls in Recursive Language Models [67], editable context files in Context Language Models [45], and indexed experience memory in MemexRL [59]. Our implementation provides asynchronous sampling adapters for Slime [79] and Tinker [53].

References

  1. [1]Cao, Ruan, Li, Chao, Ning, Song, Chen, & Li. (2026). Tree A dv: Tree-structured advantage redistribution for group-based RL. https://arxiv.org/abs/2601.03703
  2. [2]Chen, Tang, Deng, Wu, Wu, Jiang, Prasanna, Cohan, & Wang. (2025). Loc A gent: Graph-guided LLM agents for code localization. https://arxiv.org/abs/2503.09089
  3. [3]Chen, Ma, Zhuang, Nie, Zou, Sharifymoghaddam, Liu, Green, Patel, Meng, Su, Li, Hong, Shi, Liu, Oyarhoseini, Thakur, Zhang, Gao, … Lin. (2026). B rowse C omp-plus: A fair and disentangled evaluation benchmark for deep search agents. https://aclanthology.org/2026.acl-long.1023/aclanthology.org/2026.acl-long.1023
  4. [4]Choudhury. (2025). Process reward models for LLM agents: Practical framework and directions. https://arxiv.org/abs/2502.10325
  5. [5]Cohn, Ghahramani, & Jordan. (1996). Active learning with statistical models. https://arxiv.org/abs/cs/9603104
  6. [6]Cursor. (2026). Training C omposer for longer horizons. https://cursor.com/blog/self-summarizationcursor.com/blog/self-summarization
  7. [7]Cursor Research. (2026). Composer 2 technical report. https://arxiv.org/abs/2603.24477
  8. [8]Dong, Mao, Ma, Bao, Chen, Wang, Chen, Du, Wang, Zhang, Zhou, Zhu, Wen, & Dou. (2026). Agentic reinforced policy optimization. https://arxiv.org/abs/2507.19849
  9. [9]Dou, Brena, Deo, Jurayj, Zhang, Holzenberger, & Van Durme. (2026a). Deontic B ench: A benchmark for reasoning over rules. https://arxiv.org/abs/2604.04443
  10. [10]Dou, Jurayj, Holzenberger, & Van Durme. (2026b). DAR : Deontic reasoning with agentic harnesses. https://arxiv.org/abs/2606.05009
  11. [11]Guan, Zhang, Liu, Shang, Sun, Zhu, Yang, & Yang. (2025). r S tar- M ath: Small LLMs can master math reasoning with self-evolved deep thinking. https://arxiv.org/abs/2501.04519
  12. [12]Guo, Xu, Liu, Ye, & Qiu. (2025). Segment policy optimization: Effective segment-level credit assignment in RL for large language models. https://arxiv.org/abs/2505.23564
  13. [13]Hao, Gu, Ma, Hong, Wang, Wang, & Hu. (2023). Reasoning with language model is planning with world model.
  14. [14]Harvey. (2026). Harvey T enet research preview. https://www.harvey.ai/blog/post-training-update-harvey-tenetharvey.ai/blog/post-training-update-harvey-tenet
  15. [15]Hay, Russell, Tolpin, & Shimony. (2012). Selecting computations: Theory and applications.
  16. [16]Hendrycks, Burns, Kadavath, Arora, Basart, Tang, Song, & Steinhardt. (2021). Measuring mathematical problem solving with the MATH dataset. https://arxiv.org/abs/2103.03874
  17. [17]Hou, Hu, Li, Lu, Tang, & Dong. (2025). Tree RL : LLM reinforcement learning with on-policy tree search. https://arxiv.org/abs/2506.11902
  18. [18]Hou, Li, Tang, & Dong. (2026). Single-rollout asynchronous optimization for agentic reinforcement learning. https://arxiv.org/abs/2607.07508
  19. [19]Ji, Ma, Wang, Chen, Chu, & Wu. (2026). Tree search for LLM agent reinforcement learning. https://arxiv.org/abs/2509.21240
  20. [20]Jimenez, Yang, Wettig, Yao, Pei, Press, & Narasimhan. (2024). SWE -bench: Can language models resolve real-world GitHub issues?. https://arxiv.org/abs/2310.06770
  21. [21]Jin, Zeng, Yue, Yoon, Arik, Wang, Zamani, & Han. (2025). Search- R1 : Training LLMs to reason and leverage search engines with reinforcement learning. https://arxiv.org/abs/2503.09516
  22. [22]Kaufmann, & Koolen. (2017). M onte- C arlo tree search by best arm identification.
  23. [23]Kazemnejad, Aghajohari, Portelance, Sordoni, Reddy, Courville, & Le Roux. (2025). Vine PPO : Refining credit assignment in RL training of LLMs.
  24. [24]Kimi Team. (2026). Kimi K3 : Open frontier intelligence. https://arxiv.org/abs/2607.24653
  25. [25]Kingma, & Ba. (2015). Adam: A method for stochastic optimization.
  26. [26]Koh, & Liang. (2017). Understanding black-box predictions via influence functions. https://proceedings.mlr.press/v70/koh17a.htmlproceedings.mlr.press/v70/koh17a.html
  27. [27]Konda, & Tsitsiklis. (1999). Actor-critic algorithms.
  28. [28]Lei, Li, Choubey, Zhang, Peng, Ye, Narayan, Ding, Savarese, & Wu. (2026). Fork where the model changes its mind: Belief-shift branching for tree-structured reinforcement learning. https://arxiv.org/abs/2609.11061
  29. [29]Li, Zhou, Meng, Vadera, Li, & Li. (2026a). Turn- PPO : Turn-level advantage estimation with PPO for improved multi-turn RL in agentic LLMs. https://arxiv.org/abs/2512.17008
  30. [30]Li, Zhang, Jurayj, Wang, Jin, Farajtabar, Nalisnick, & Khashabi. (2026b). Self-compacting language model agents. https://arxiv.org/abs/2606.23525
  31. [31]Li, & Li. (2025). Process reward model with Q -value rankings. https://arxiv.org/abs/2410.11287
  32. [32]Li, Li, Zhang, Yu, Lin, Jin, Guan, Liu, Li, Yin, & Shang. (2026c). Process reward informed tree rollout for effective multi-turn RL. https://arxiv.org/abs/2607.15610
  33. [33]Li, Gu, Wen, Li, Xing, Guo, Zheng, Zhou, Qu, Zhou, Zhang, Shen, Liu, Lin, Yang, Zhang, & Huang. (2026d). Tree PO : Enhancing policy efficacy and inference efficiency with tree modeling. https://arxiv.org/abs/2508.17445
  34. [34]Li, Hou, Jing, Tang, & Dong. (2026e). Compaction RL : Reinforcement learning with context compaction for long-horizon agents. https://arxiv.org/abs/2607.05378
  35. [35]Lightman, Kosaraju, Burda, Edwards, Baker, Lee, Leike, Schulman, Sutskever, & Cobbe. (2024). Let's verify step by step.
  36. [36]Liu, Wang, Shi, Xie, An, Zhang, Zhao, Gu, Lin, Hu, Li, Zhang, Zhou, & Gai. (2026). Attention as a compass: Efficient exploration for process-supervised RL in reasoning models. https://arxiv.org/abs/2509.26628
  37. [37]Luo, Liu, Liu, Phatale, Guo, Lara, Li, Shu, Zhu, Meng, Sun, & Rastogi. (2024). Improve mathematical reasoning in language models by automated process supervision. https://arxiv.org/abs/2406.06592
  38. [38]Mussmann, Reisler, Tsai, Mousavi, O'Brien, & Goldszmidt. (2022). Active learning with expected error reduction. https://arxiv.org/abs/2211.09283
  39. [39]Packer, Wooders, Lin, Fang, Patil, Stoica, & Gonzalez. (2023). MemGPT : Towards LLMs as operating systems. https://arxiv.org/abs/2310.08560
  40. [40]Qwen Team. (2025). Qwen3 technical report, 2025. https://arxiv.org/abs/2505.09388
  41. [41]Qwen Team. (2026). Qwen3.5 : Towards native multimodal agents, February 2026. https://qwen.ai/blog?id=qwen3.5qwen.ai/blog?id=qwen3.5
  42. [42]Roy, & McCallum. (2001). Toward optimal active learning through sampling estimation of error reduction. https://groups.csail.mit.edu/rrg/papers/icml01.pdfgroups.csail.mit.edu/rrg/papers/icml01.pdf
  43. [43]Schulman, Moritz, Levine, Jordan, & Abbeel. (2016). High-dimensional continuous control using generalized advantage estimation.
  44. [44]Setlur, Nagpal, Fisch, Geng, Eisenstein, Agarwal, Agarwal, Berant, & Kumar. (2025). Rewarding progress: Scaling automated process verifiers for LLM reasoning. https://arxiv.org/abs/2410.08146
  45. [45]Shao, Shen, Yin, Li, Wang, Ivison, Poovendran, Lambert, Xiao, Lewis, Yih, Zettlemoyer, & Koh. (2026). Context language models. https://arxiv.org/abs/2609.37725
  46. [46]Shao, Wang, Zhu, Xu, Song, Bi, Zhang, Zhang, Li, Wu, & Guo. (2024). Deep S eek M ath: Pushing the limits of mathematical reasoning in open language models. https://arxiv.org/abs/2402.03300
  47. [47]Silver, Huang, Maddison, Guez, Sifre, van den Driessche, Schrittwieser, Antonoglou, Panneershelvam, & Lanctot. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484–489.
  48. [48]Snell, Lee, Xu, & Kumar. (2025). Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. https://arxiv.org/abs/2408.03314
  49. [49]Sun, Lu, Ling, Liu, Yao, Yang, & Chen. (2026). Scaling long-horizon agent via context folding. https://arxiv.org/abs/2510.11967
  50. [50]Tan, Yang, Chen, Shao, Wen, Shen, Luo, Du, Guo, & Li. (2026). Hindsight credit assignment for long-horizon LLM agents. https://arxiv.org/abs/2603.08754
  51. [51]Tao, Peng, Yao, Ge, Cheng, Wang, Gao, & Li. (2026). TRACE : Turn-level reward assignment via credit estimation for long-horizon agents. https://arxiv.org/abs/2607.13988
  52. [52]Tesauro, Rajan, & Segal. (2010). Bayesian inference in M onte- C arlo tree search.
  53. [53]Thinking Machines Lab. (2026). Tinker, 2026. https://thinkingmachines.ai/tinker/thinkingmachines.ai/tinker
  54. [54]Tian, Peng, Song, Jin, Yu, Han, Mi, & Yu. (2024). Toward self-improvement of LLMs via imagination, searching, and criticizing.
  55. [55]Tran, Yao, & Yu. (2025). Exploiting tree structure for credit assignment in RL training of LLMs. https://arxiv.org/abs/2509.18314
  56. [56]Venkatraman, Dinot, & Aitchison. (2026). Le critique: Privileged value functions for LLM reinforcement learning. https://arxiv.org/abs/2608.16739
  57. [57]Wang, Cui, Zhang, Liu, Jin, Geng, Zhang, Zalmout, Shi, & Sun. (2026a). T$^2$PO : Uncertainty-guided exploration control for stable multi-turn agentic reinforcement learning. https://arxiv.org/abs/2605.02178
  58. [58]Wang, Li, Shao, Xu, Dai, Li, Chen, Wu, & Sui. (2024). Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. https://arxiv.org/abs/2312.08935
  59. [59]Wang, Chen, Wang, & Wei. (2026b). Memex(RL) : Scaling long-horizon LLM agents via indexed experience memory. https://arxiv.org/abs/2603.04257
  60. [60]Wang, Wang, Wang, Zhang, Li, Yang, Jin, Yu, Nguyen, Liu, Gottlieb, Lu, Cho, Wu, Fei-Fei, Wang, Choi, & Li. (2025). RAGEN : Understanding self-evolution in LLM agents via multi-turn reinforcement learning. https://arxiv.org/abs/2504.20073
  61. [61]Wei, Zeng, Li, Wang, Brown, Frunza, Deng, Schneider, Nevmyvaka, Zhao, Garcia, & Hong. (2025). Reinforcing multi-turn reasoning in LLM agents via fine-grained reward structure and credit assignment. https://arxiv.org/abs/2505.11821
  62. [62]Wu, Xuan, Qi, Tu, Lu, Li, & Choi. (2026). Deep S earch: Overcome the bottleneck of reinforcement learning with verifiable rewards via tree-based search. https://arxiv.org/abs/2509.25454
  63. [63]Xu, & Kazantsev. (2019). Understanding goal-oriented active learning via influence functions. https://arxiv.org/abs/1905.13183
  64. [64]Yang, Guo, Huang, Liang, Wang, & Tang. (2025). Tree RPO : Tree relative policy optimization. https://arxiv.org/abs/2506.05183
  65. [65]Yao, Yu, Zhao, Shafran, Griffiths, Cao, & Narasimhan. (2023). Tree of thoughts: Deliberate problem solving with large language models.
  66. [66]Yu, Chen, Feng, Chen, Dai, Yu, Zhang, Ma, Liu, Wang, & Zhou. (2026). MemAgent : Reshaping long-context LLM with multi-conv RL -based memory agent. https://arxiv.org/abs/2507.02259
  67. [67]Zhang, Kraska, & Khattab. (2025). Recursive language models. https://arxiv.org/abs/2512.24601
  68. [68]Zhang. (2026). From reasoning to agentic: Credit assignment in reinforcement learning for large language models. https://arxiv.org/abs/2604.09459
  69. [69]Zhang, Zhoubian, Hu, Yue, Dong, & Tang. (2024). ReST-MCTS* : LLM self-training via process reward guided tree search.
  70. [70]Zhang, Wang, Jia, Shen, Ding, Viola, Schmidhuber, & Elhoseiny. (2026a). Hybrid advantage estimation with unified critic for VLM agentic reinforcement learning. https://arxiv.org/abs/2607.23605
  71. [71]Zhang, Xu, Ding, Xie, Gao, Ding, Zhang, Fu, & Wang. (2026b). Information gain-based rollout policy optimization: An adaptive tree-structured rollout approach for multi-turn LLM agents. https://arxiv.org/abs/2607.06223v1
  72. [72]Zhen, Yu, Guo, Cheng, & Deng. (2026). Hierarchical reinforcement learning with augmented step-level transitions for LLM agents. https://arxiv.org/abs/2604.05808
  73. [73]Zheng, Xing, Gu, Liang, Qu, Zhou, Li, Wen, Lin, Huang, Liu, Zhang, & Ma. (2025). First return, entropy-eliciting explore. https://arxiv.org/abs/2507.07017
  74. [74]Zhong, Yang, Wu, Liu, Yao, & Kuang. (2026). SIGHT : Reinforcement learning with self-evidence and information-gain diverse branching for search agent. https://arxiv.org/abs/2602.11551
  75. [75]Zhou, Yan, Shlapentokh-Rothman, Wang, & Wang. (2024a). Language agent tree search unifies reasoning, acting, and planning in language models. https://arxiv.org/abs/2310.04406
  76. [76]Zhou, Xu, Zhu, Zhou, Lo, Sridhar, Cheng, Ou, Bisk, Fried, Alon, & Neubig. (2024b). WebArena : A realistic web environment for building autonomous agents. https://arxiv.org/abs/2307.13854
  77. [77]Zhou, Zanette, Pan, Levine, & Kumar. (2024c). Ar CH er: Training language model agents via hierarchical multi-turn RL.
  78. [78]Zhou, Jiang, Tian, Weston, Levine, Sukhbaatar, & Li. (2025). SWEET-RL : Training multi-turn LLM agents on collaborative reasoning tasks. https://arxiv.org/abs/2503.15478
  79. [79]Zhu, Xie, Lv, & slime Contributors. (2025). slime: An LLM post-training framework for RL scaling, 2025. https://github.com/THUDM/slimegithub.com/THUDM/slime
  80. [80]Zong, Chen, Li, Yi, Zhou, Li, Qian, Chen, & Jiang. (2026). AT$^2$PO : Agentic turn-based policy optimization via tree search. https://arxiv.org/abs/2601.04767
  81. [81]Zou, Wang, Qu, Jiang, Cai, Mao, Peng, Xu, Liu, Yang, Yang, & Ji. (2026). TRACE : A unified rollout budget allocation framework for efficient agentic reinforcement learning. https://arxiv.org/abs/2606.11119

Paper details

vlnc:W_efuffeat
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents