WorksComputer Science
FOCUS: Active Critic Learning for Long-Horizon Language Model Agents
Abstract
Sparse outcome rewards make it difficult for long-horizon language-model agents to assign credit to earlier actions. Learned critics provide value estimates for intermediate states, but these estimates can be inaccurate, while estimating intermediate values with repetitive continuation samples from points within a trajectory incurs substantial costs. To mitigate these issues, we introduce FOCUS, an RL method that uses selective branching to help the critic network learn its value estimator. The rollout controller facilitates active learning for the critic by sampling continuations from states that are expected to most reduce the critic's estimation error. These refined value estimates then score the segment between consecutive states with a TD() advantage, while the sampled evidence supplies training targets for the critic. FOCUS supports long-horizon agentic reinforcement learning through a modular context-management interface that decouples context management from rollout allocation. Our experiments show that using compaction-induced segment boundaries for value estimation and credit assignment, FOCUS outperforms GRPO and VinePPO with matched per-step generation budgets on legal reasoning, deep search, and code localization benchmarks, improving held-out accuracy at Qwen3.5-9B scale by 7.2 points over the strongest baseline on BrowseComp-Plus (55.4 vs. 48.2) and by 4.1 points on DeonticBench (56.5 vs. 52.4).
Introduction
Language-model agents are increasingly trained to solve tasks that unfold over dozens or hundreds of tool calls: resolving an issue in a real repository [20, 7], completing a task on a live website [76], or answering a question that requires many rounds of search and reading [3, 24]. Reinforcement Learning with Verified Rewards (RLVR) is the standard way to improve such agents, but long trajectories mean both sampling and credit assignment are costly. A single verifiable reward must be distributed across every token within a rollout even when many had little impact on the final outcome. To learn efficiently, we need proper intermediate value estimates under a limited rollout budget.
These estimates can come from a learned critic network [27] or through Monte Carlo approximation from a given state [46]. A learned critic reuses information across trajectories, but its predictions can be inaccurate. Tree-structured rollouts branch at intermediate states to refine value estimates [23, 17, 33, 12, 19]. However, sampling additional language model continuations for each estimate incurs substantial costs, raising the question of how to allocate a rollout budget to improve value estimation most efficiently.
We treat refining value estimates via additional continuation branching as an active learning problem for a critic network: continuations should be sampled selectively from states where additional evidence is expected to improve value estimation the most. We introduce FOCUS: Focused Optimization on Critical Underperforming Segments, a rollout controller that runs each episode as a search tree. FOCUS combines the critic’s predictions with evidence from sampled continuations and allocates further rollouts according to their estimated marginal reduction in value-estimation error [15, 22]. These refined values provide stepwise credit for the policy, while the acquired evidence supplies training targets for the critic.
In long-horizon tasks, context compaction [6, 14, 49, 34], where the agent resumes from a compacted account of its previous actions and observations, provides a natural structure for applying selective rollouts: each segment between compactions becomes a step, at which step-level rewards and values can be calculated (Figure 1). However, FOCUS does not depend on a particular context-management protocol; its modular design separates rollout allocation, context management, and credit assignment.1 Appendix § 13 details the implementation, and § 4 discusses related work.
We evaluate FOCUS on DeonticBench [9], BrowseComp-Plus [3], and LocBench [2]. FOCUS outperforms GRPO and VinePPO on all three benchmarks under a matched per-step generation budget. With Qwen3.5-9B, FOCUS improves held-out accuracy over the strongest baseline by 7.2 points on BrowseComp-Plus (55.4 vs. 48.2) (Table tab:main-results). Diagnostics show more effective reduction of value-estimation error as the refinement budget increases, while ablations demonstrate that the gains are not tied to a single compaction implementation (§ 3.4 and § 3.5).
In summary, our contributions are:
We cast the combination of critic predictions and continuation evidence as an active critic learning problem (§ 2.2 and § 2.3).
We introduce FOCUS, a highly customizable rollout controller, which allocates a limited rollout budget according to estimated value-estimation error reduction and integrates the resulting estimates with credit assignment.
We show that FOCUS improves over critic-free baselines on long-horizon benchmarks (§ 3).
Figure 1. Overview of the proposed FOCUS rollout scheduler for a deep-research-style problem. FOCUS continues working on a rollout tree. At each iteration, FOCUS STEP 1: takes an expanded tree; STEP 2: scores each compaction-induced state with the critic; STEP 3: compares against outcomes to decide critic quality (§ 2.2); STEP 4: sources additional continuations that are expected to correct critic errors most efficiently (§ 2.3). With this active learning loop for rollout acquisition, FOCUS provides a more accurate value estimate with limited rollout budgets (§ 3.4).
Method
Problem Setup
We consider reinforcement learning of a language-model policy from outcome rewards. The policy generates responses or actions from its current context, and a critic , parameterized by , estimates the expected remaining return. We assume that at least some of the intermediate states are easily forkable and that repeated sampling from them is trivial. An episode records the generated turns and rewards from one completed run.
Given a finite rollout budget, FOCUS selects from candidate states from which to sample additional continuations, refines their value estimates, and constructs training samples for both the critic and the policy. For tool-using agents, compaction boundaries provide natural candidate states. Appendix § 13 details the execution mechanics used in our experiments.
Critic Refinement and Learning
We first describe how collected continuations produce the learning signals for the critic and the policy. § 2.3 describes how we allocate the rollout budget.
States and rollout segments. For each prompt , we represent collected rollouts as a tree . A nonterminal node denotes a state from which the policy can resume with the same context and environment state; an edge contains the policy-generated continuation and environment transitions until the next state. We write for the context of a node (state), leaving its associated environment state implicit.
A completed trajectory contains nonterminal nodes and a terminal node , connected by edges , .
Refining state values. Let denote the expected discounted return of a continuation from under the collection policy. The critic predicts this value from the state’s context. However, this prediction can have high variance. We refine its prediction using the outcome evidence from the continuations that have been expanded from it, backed up one level as in TreeRPO [64]:
where is the continuation’s own refined value (Eq. (2) below). We prefer this to a raw average of descendant outcomes [33] because the allocator chooses which nodes to expand, so the leaves beneath are no longer an on-policy draw from ; the continuation’s refined value carries its evidence back up instead, one level at a time. We combine the critic prediction and the recursive backup with a fixed pseudo-observation weight ,
For , we set without evaluating ; terminal values are zero. As increases, the blend places more weight on the backed-up mean [47, 52].
Critic supervision and actor advantages. After collection and the final backup, we train the critic on the recursive target at every non-terminal state:
The target is held fixed during training. This update transfers the collected continuation evidence into the critic used in subsequent training. Within the current iteration, the refined values supply the segment-level temporal-difference residual2
We apply each edge’s detached advantage to its policy-generated task and context-management tokens, denoted , masking environment observations and conditioning text. With the collection policy, the token-level importance ratio, and the clipping threshold, the actor minimizes
Appendix § 11.1 gives the general TD() extension.
Selective Rollout Allocation for Critic Learning
Branching continuations from candidate states supplies additional value evidence [23, 17, 33, 12], but uniform allocation does not account for differences in critic error or the expected benefit of another sample. Given an explored tree and an additional rollout budget , FOCUS chooses which candidate node receives the next continuation. The aim is to acquire evidence where it is expected to reduce value estimation error the most [42, 38].
Outcome evidence for selection. While selection uses the same formula for mixing observed continuation outcomes and critic predictions, it does not use the continuation-refined backups of its children as in Eq. (1). Specifically, let be the number of continuation outcomes and their mean. A completed continuation contributes one outcome at its starting node and at each newly created node along that continuation. Later branching from a descendant does not add more observations to its strict ancestors. Thus counts observed continuation outcomes, whereas in Eq. (1) counts direct children.
For binary continuation outcomes, the selection model uses a Beta prior of strength centered on . Its posterior mean and expected squared critic error are
Notice how is distinct from the continuation-refined training value . The predictive variance of the next outcome is . In particular, we rely on to conduct the selection due to the observation outlined in Appendix § 10.
Expected benefit of a continuation. We score the expected utility of a continuation using the marginal MSE reduction of value estimation error at that candidate state. For critic squared error and outcome variance , the acquisition function becomes . Differentiating with and held fixed, then substituting the posterior estimates gives
where counts completed continuation outcomes.
\caption{Selective rollout allocation and value refinement, with continuation helper}
\label{alg:refine}
\small
\algrenewcommand{\algorithmiccomment}[1]{\hfill\(\triangleright\) {\scriptsize #1}}
\begin{algorithmic}
\Require tree $\mathcal T_x$; fixed actor $\pi_\theta$ and critic $V_\phi$; budget $B$; prior strength $\kappa$
\Require environment $\mathcal E$; branch predicate $\mathcal B$; compaction trigger $\mathcal G$ and operator $\mathcal C_\pi$
\Ensure tree $\mathcal T_x$ with critic targets $\bar G$ and refined values $\tilde V$
\State initialize $N,\bar G_{\mathrm{obs}}$ from completed suffixes; refresh $\mathcal F,g$
\algblocktitle{allocation}{\S\ref{sec:method-search}: Selective rollout allocation}
\While{$B>0\ \land\ \mathcal F\ne\emptyset$}
\State $\bar c\gets\arg\max_{u\in\mathcal F}g(u)$; $\xi\gets\textsc{Continue}(\bar c,\pi_\theta)$
\If{$\xi=\bot$}
\State $B\gets B-1$ \Comment{execution failure}
\Else
\State $\mathcal T_x\gets\textsc{Attach}(\mathcal T_x,\bar c,\xi)$
\State $B\gets B-d(\xi)$ \Comment{$d(\xi)$: new task transitions}
\State $N(u),\bar G_{\mathrm{obs}}(u)\gets0$ for $u\in\mathrm{New}(\xi)$ \Comment{new nodes}
\State $\textsc{Score}(\mathcal T_x,V_\phi)$ \Comment{cache unscored nonterminal values}
\For{$u\in\{\bar c\}\cup\mathrm{New}(\xi)$} \Comment{$G_\xi(u)$: observed suffix return}
\State $\bar G_{\mathrm{obs}}(u)\gets\dfrac{N(u)\bar G_{\mathrm{obs}}(u)+G_\xi(u)}{N(u)+1}$; $N(u)\gets N(u)+1$
\EndFor
\EndIf
\State refresh $\mathcal F,g$
\EndWhile\algblockend{allocation}
\Statex
\algblocktitle{refinement}{\S\ref{sec:method-credit}: Value refinement and critic targets}
\State $\tilde V(u)\gets0$ for terminal nodes; $\tilde V(u)\gets V_\phi(u)$ for nonterminal leaves
\For{$u\in\mathcal T_x$ with $n(u)>0$, in reverse topological order}
\State $\bar G(u)\gets\dfrac{1}{n(u)}\displaystyle\sum_{v\in\mathrm{ch}(u)}\bigl(R_{u\to v}+\gamma^{d_{u\to v}}\tilde V(v)\bigr)$
\State $\tilde V(u)\gets\dfrac{\kappa V_\phi(u)+n(u)\bar G(u)}{\kappa+n(u)}$
\EndFor\algblockend{refinement}
\Statex
\State \Return $\mathcal T_x$
\Statex
\Function{Continue}{$\bar c,\pi$}
\State $(c,s)\gets\textsc{Fork}(\bar c)$; $\xi\gets[\,]$; $\zeta\gets[\,]$ \Comment{$\zeta$: current segment}
\While{$\neg\textsc{Terminal}(s)$}
\algblocktitle[3em]{compaction}{Modular context management}
\If{$\mathcal G(c)=1$}
\State $c\gets\mathcal C_\pi(c)$\algblockend{compaction}
\Statex
\Else
\State $\tau\sim\pi(\cdot\mid c)$; $(s,o,r)\sim\mathcal E(s,a(\tau))$; $c\gets c\,\|\,\tau\,\|\,o$
\EndIf
\State $\eta\gets\textsc{Trace}()$; $\zeta\gets\zeta\,\|\,\eta$ \Comment{task/compaction turns, rewards, transitions}
\State $s\gets\textsc{Update}(s,\eta)$ \Comment{counters and termination flags}
\State \textbf{if} $\textsc{Failed}(s)$ \textbf{then return} $\bot$
\If{$\textsc{Terminal}(s)\ \lor\ \mathcal B(c,s)=1$} \Comment{$\mathcal B$ independent of $\mathcal G$}
\State $\bar c'\gets c$; $\xi\gets\xi\,\|\,[(\bar c',\textsc{Snapshot}(s),\zeta)]$; $\zeta\gets[\,]$ \Comment{save resumption state}
\EndIf
\EndWhile
\State \Return $\xi$
\EndFunction
\end{algorithmic}
\algblockbox{allocation}{AlgAllocation}
\algblockbox{refinement}{AlgRefinement}
\algblockbox{compaction}{AlgContext}
Algorithm alg:refine first acquires rollout evidence (§ 2.3), then refines values bottom-up (§ 2.2). Each continuation completes before the next allocation decision. The eligible set is . The included \textsc{Continue}(\bar c,\pi) helper returns new nodes and connecting segments through termination. Since uses observed suffix statistics and fixed critic predictions, value refinement can follow collection. Appendix § 11 gives the full training loop.
Practical considerations. Total branching width at any node is uncapped. The allocation phase can also run asynchronously, with each completed continuation updating the acquisition statistics while others remain in flight. In this case, we replace in Eq. (7) by , where counts pending continuations; posterior estimates still use completed outcomes only. We limit concurrent continuations per node to and globally to worker capacity , restricting eligibility to . Completed children do not consume concurrency slots. In practice, the two loops can run concurrently, each with multiprocessing. After the launch budget is exhausted, pending continuations finish and a final bottom-up refinement ensures that the training targets reflect the completed tree.
Experiments
We first compare FOCUS with PPO and GRPO in a preliminary evaluation on MATH [16] problems at difficulty levels 4–5, tracking accuracy, evaluation output length, and per-rollout reduction over training updates. We then compare FOCUS to GRPO and VinePPO on three long-horizon benchmarks at Qwen3.5-9B and Qwen3.5-27B [41] (LoRA), assess how MCTS refinement improves critic estimates, and ablate the compaction operator and refinement rollouts.
Preliminary Evaluation on Mathematical Reasoning
We compare FOCUS, PPO, and GRPO with Qwen3.5-4B [41] on MATH Level 4–5 problems for 32 completed updates, evaluating the base checkpoint and checkpoints at updates 8, 16, 24, and 32. We use 512 training questions from the official MATH training split and 256 validation questions from its official test split.3 The training set contains 226 Level 4 and 286 Level 5 questions; validation contains 123 Level 4 and 133 Level 5 questions.
All methods consume the fixed 512-question order for 32 training steps. PPO uses 64 questions per training step; GRPO and FOCUS each use 16 questions per training step. To keep the training compute roughly matched, GRPO uses group size 6, and FOCUS uses a per-question edge budget of 64, with each edge being a maximum of 256 tokens.
Figure 2. Preliminary evaluation with Qwen3.5-4B on MATH Level 4–5. Accuracy and the mean number of generated tokens per evaluation question are shown at the base checkpoint (update 0) and after 8, 16, 24, and 32 completed updates. Markers denote recorded checkpoints. Panel (c) shows FOCUS’s mean over all pre-allocation score evaluations within each update from 1 to 32; shading denotes standard deviation of those scores, computed as the square root of the reported variance.
FOCUS has the highest accuracy at updates 8, 16, and 24 (Figure 2), and is only one correct answer behind GRPO at update 32. FOCUS improves accuracy earlier by update count, hinting at a training-compute advantage. Also at update 32, FOCUS generates 2,782 tokens per evaluation problem on average, compared with 5,747 for GRPO and 7,207 for PPO. We hypothesize this results from FOCUS’s ability to refine value estimates more effectively.
Figure 2c shows that throughout training decreases by about 50% from update 1 to update 32, showing that as FOCUS gradually tunes its critic, the expected reduction in value estimation error decreases, suggesting that FOCUS gradually learns to predict the expected outcome more accurately.
Long-horizon datasets
DeonticBench [9] DeonticBench evaluates models on context-grounded deontic reasoning: deriving obligations, permissions, and prohibitions from real statutes and case facts. It comprises 6,232 tasks across U.S. federal tax, airline baggage policy, immigration administration, and state housing law. Applying a long statute to a specific case is a multi-step process, which makes the benchmark a setting for long-horizon agentic reasoning in which the model consults the statutes on demand [10] and a natural testbed for stepwise rollout. We focus on the airline, sara-numeric, and sara-binary subsets, whose statutes are reasonably long and involve non-trivial long-horizon tasks. The actor harness exposes (t) list;, (t) grep;, (t) read;, and (t) python;. We stratify the data into equally sized training and held-out splits.
BrowseComp-Plus [3] BrowseComp-Plus evaluates deep search agents on complex, multi-hop information-seeking queries over a fixed, human-verified corpus rather than a live web API. The resulting trajectories are long-horizon: iterative query formulation, retrieval, and reflection, which makes the benchmark a natural testbed for stepwise rollout. Following TRACE [51], the actor harness exposes (t) search;, (t) open;, and (t) find; over the fixed corpus. We randomly split the 830 test tasks into 498 train, 83 validation, and 249 held-out tasks.
LocBench [2] LocBench is a code-localization benchmark of 560 real GitHub issues (242 bug reports, 150 feature requests, 139 performance issues, and 29 security issues) from repositories such as transformers, scikit-learn, vllm, and ultralytics; in LocAgent’s construction, bug reports are issues created relatively recently, and security and performance instances are collected via GitHub keyword search. Given only the issue, the agent must locate the files and functions touched by the gold patch. We instantiate it using the harness described in Appendix § 13; the actor exposes (t) list;, (t) grep;, and (t) read; for repository navigation. We score submissions as in LocAgent: file-level and function-level Acc@ is when every gold location appears in the top (added functions are excluded); we report Acc@ at the file level and train on file-level recall at . We split the instances into equal training and held-out halves, stratified by repository and issue category.
Main results
We compare GRPO, VinePPO, and FOCUS using Qwen3.5-9B on DeonticBench, BrowseComp-Plus, and LocBench, and additionally with Qwen3.5-27B (LoRA) on DeonticBench. Table tab:main-results reports held-out accuracy alongside untrained reference models. All methods train for one epoch under a matched per-step generation budget with compaction on every rollout; protocol, optimization, and compute details are in Appendix § 6.
| Model | Method | DeonticBench | BrowseComp | ||||||
| -Plus & 2cLocBench | |||||||||
| Airline | SARA-N | SARA-B | Hard | Overall | File@1 | File@5 | |||
| Reference (no RL) | |||||||||
| gpt-oss-20b (medium) | 2.7 | 8.0 | 76.8 | 14.1 | 33.7 | 18.6 | 54.0 | 57.5 | |
| Nemotron 3 Nano 30B A3B | 2.7 | 12.0 | 89.9 | 18.3 | 36.1 | 19.3 | 57.9 | 66.6 | |
| Qwen3.5-9B | 5.3 | 30.0 | 89.9 | 19.7 | 45.9 | 29.7 | 62.1 | 67.8 | |
| Qwen3.5-27B | 58.7 | 60.0 | 94.2 | 43.7 | 73.4 | 50.6 | 66.7 | 79.3 | |
| Qwen3.5-9B | GRPO | 14.0 | 56.0 | 92.8 | 28.2 | 52.4 | 48.2 | 70.1 | 80.5 |
| VinePPO | 10.0 | 50.0 | 92.0 | 24.0 | 49.4 | 46.6 | 67.8 | 79.3 | |
| FOCUS | 21.3 | 52.0 | 96.4 | 26.8 | 56.5 | 55.4 | 72.4 | 81.6 | |
| vs. GRPO | |||||||||
| Qwen3.5-27B | GRPO | 60.7 | 60.0 | 94.2 | 43.7 | 74.3 | OmitFill | ||
| VinePPO | 62.7 | 58.0 | 94.2 | 36.6 | 74.9 | OmitFill black!55outside the 27B study | |||
| FOCUS | 62.7 | 62.0 | 96.4 | 43.7 | 76.3 | OmitFill | |||
| vs. GRPO | OmitFill |
FOCUS ranks first among the trained methods on every benchmark in Table tab:main-results. At Qwen3.5-9B, it leads on DeonticBench overall (56.5 vs. 52.4 for GRPO and 49.4 for VinePPO), reports the strongest BrowseComp-Plus result (55.4 vs. 48.2 for GRPO and 46.6 for VinePPO, each the best held-out checkpoint over training), and localizes best on LocBench (72.4 / 81.6 File@1 / @5, ahead of 70.1 / 80.5 and 67.8 / 79.3). FOCUS also leads across scale: Qwen3.5-27B with LoRA on DeonticBench still gains from FOCUS fine-tuning (76.3 overall vs. 74.3 and 74.9), but the margin over the baselines shrinks. This could partially be because the untuned 27B model already starts much higher (73.4 overall).
Training dynamics. FOCUS attains the highest on-policy reward throughout training on DeonticBench and BrowseComp-Plus (Figure 3), and the highest held-out accuracy on DeonticBench. On DeonticBench and LocBench, the three methods track closely early, then FOCUS pulls ahead and keeps improving or holds a higher plateau, whereas GRPO on DeonticBench peaks mid-run and then regresses. Late in training, GRPO’s mean tool calls collapse on all three benchmarks while VinePPO and FOCUS stay near their initial levels: GRPO increasingly (t) submit;s right after compaction even when the statute or repository still needs reading (Appendix § 12). We hypothesize that trajectory-level group credit makes such premature submissions easy to reinforce, whereas VinePPO’s suffix branching and FOCUS’s refined checkpoint values preserve longer horizons.
(a) DeonticBench.
(b) BrowseComp-Plus.
(c) LocBench.
Figure 3. Training dynamics. Each column is one benchmark, showing training reward against optimizer step (top), held-out evaluation (middle), and mean numbers of tool calls per episode on the held-out split (bottom), for GRPO, VinePPO, and FOCUS under a matched per-step generation budget. Top panels plot mean terminal reward from on-policy collection; LocBench held-out reward is File@5. BrowseComp-Plus results are the equivalence judgement of Qwen3-32B [40].
Value refinement
To assess how efficient the FOCUS correction process is in reducing the critic error for long-horizon tasks, as well as whether the critic can improve throughout training, we conduct two diagnostic analyses.
Critic Error Reduction We start by assuming two types of imperfect critics: a constant critic init and a random critic init, and we compare how different priority schedules affect the critic error reduction against a ground truth determined by the mean outcome of MCTS continuations. Figure 4a shows that under both types of critics, FOCUS’s correction process is noticeably more efficient than random selection. Also, using the raw critic value residual performs poorly. This shows that FOCUS-style scheduling not only works well under degenerate critics but also reduces variance even when the critic provides almost no information.
Critic Improvement We also test whether the critic improves over training. At each LocBench evaluation checkpoint, we sample MCTS continuations from a set of completed traces. We then compute the Spearman correlation between the critic value and the MCTS ground truth. Figure 4b shows that the FOCUS critic improves steadily.
(a) Critic MSE vs. budget (DeonticBench).
(b) Critic Spearman (LocBench).
Figure 4. Value refinement for FOCUS. (a) MSE on fixed audit trees as MCTS continuations are allocated by each rule (solid: constant critic init; dashed: random critic init). (b) Rank agreement between checkpoint critic values and an oversampled, critic-blind MCTS target on fixed audit states.
Ablation studies
We ablate the compaction operator and the refinement rollouts to test that FOCUS is not tied to one compactor and that its gains come from active error correction, not the critic alone.
Compaction operator. On DeonticBench, we replace the in-context compaction operator, which writes everything into the new context window, with an in-memory workspace that provides its own (t) put;, (t) get;, (t) scan; and (t) delete;. The two perform similarly (Figure 5a), so FOCUS’s effectiveness extends to different compaction operators.
Refinement rollouts. On BrowseComp-Plus, we remove the refinement rollouts of § 2.3, which amounts to collecting independent continuations from the question prompt and estimating step-wise advantages from the critic alone. Figure 5b shows that refinement rollouts are crucial for FOCUS on BrowseComp-Plus; without them, FOCUS trains more slowly, especially early on.
Structural compaction. We also compare a free-form compaction prompt with a structural prompt that induces a more constrained compaction on LocBench (Figure 5c; both prompts are reproduced in Appendix § 9). The free-form prompt suggests headings and keeps the reply verbatim. The structural prompt requires a single summary with four sections (State, Evidence, Open questions, Next steps) and rejects replies that do not parse. The free-form prompt reaches a higher peak, while the structural prompt improves more steadily.
(a) Compaction operator (DeonticBench).
(b) Refinement rollouts (BrowseComp-Plus).
(c) Structural compaction (LocBench).
Figure 5. Ablations at evaluation checkpoints. (a) Varying the compaction operator on DeonticBench under a matched budget; (b) FOCUS with full refinement vs. FOCUS without refinement rollouts on BrowseComp-Plus; (c) free-form vs. structural compaction on LocBench (File Acc@5).
Related Work
Structured rollouts and rollout allocation Search over intermediate states began as an inference-time technique: LLM tree search [65, 13, 75] and verifier-guided test-time scaling [35, 48] improved model responses. It was later used offline to generate training supervision [54, 69, 11] and then moved into the RL loop. VinePPO [23], SPO [12], and TEMPO [55] re-sample continuations from intermediate reasoning steps to replace the value network with Monte Carlo estimates; TreeRL [17], TreePO [33], and DeepSearch [62] run the search inside RLVR training so that shared prefixes cut generation cost and the tree yields process-level signal. Other work proposes where to branch: at high-entropy tokens [73, 1], at attention peaks [36], where the model’s belief shifts [28], or, for tool-using agents, at the turn boundaries after an observation [8, 80]. Tree-GRPO [19] makes each agent step a node and shows that intra-tree group objectives amount to step-level preference learning, while PaTR [32] routes the branching budget by a process score, and IGRPO [71] and SIGHT [74] by information gain. TPO [57] instead resamples turns that its token- and turn-level uncertainty signals flag as making no progress. TRACE [81] unifies prompt- and turn-level allocation with a learned predictor of conditional success, directing extra rollouts to the prefixes most likely to yield mixed terminal rewards. FOCUS allocates additional continuations according to their estimated reduction in value-estimation error, using the acquired evidence to refine state values and train a reusable critic.
Active learning and expected error reduction Expected-error-reduction methods select observations according to their predicted effect on the learner’s error. [42] approximate this objective through sampling, while [38] use Bayesian parameter samples to estimate the reduction in predictive loss over an unlabeled validation set without retraining for every candidate query. A related approach selects data to reduce predictive variance, connecting variance reduction to squared-error minimization under an approximately unbiased learner [5]. Influence-function approaches estimate acquisition utility by approximating how additional training data affect a fitted model’s evaluation loss, without retraining for each candidate [26, 63]. FOCUS applies the error-reduction principle to allocating continuation samples among intermediate states.
Critic learning and credit assignment Trajectory-level reward is known to fail for multi-turn agents as it cannot tell a useful turn from a redundant one, and training on it alone is unstable [60, 61, 50]. One response is to place the reward and critic at the level of a turn or step rather than a token (too fine-grained) or a whole trajectory (too coarse) [77, 78, 29, 70, 72]. A turn is the natural unit of decision for an agent; the horizon measured in turns is short enough for a critic to bootstrap over, and a per-turn value gives every turn its own signal from a single rollout. Large-scale training is also shifting from structural comparison to a learned value: SAO [18] replaces GRPO’s group sampling with one rollout per prompt scored by a value model, as full groups are slow and off-policy under asynchronous training. Some process reward models serve as fixed value estimators during policy optimization [58, 37, 31, 4, 44, 51]. More recently, Le Critique [56] mixes a learned baseline with a training-free group-relative one according to how accurate the critic is. FOCUS uses refined state values to construct segment-level actor advantages and recursive backups to supervise the critic, coupling credit assignment in the current iteration with critic learning for subsequent iterations.
Context management for long-horizon agents Compaction is becoming standard for agents that outrun their context window [39, 66, 49, 30], and there is a recent push to train it jointly with the task rather than treat it as a frozen preprocessor [6, 34]. State restoration is also central to identifying step-level credit [68]. In our long-horizon experiments, compaction boundaries provide resumable states for acquiring continuation evidence. The rollout controller operates on these states independently of the compaction operator; its objective is to refine value estimates and construct critic supervision through selective rollouts.
Conclusion
We present FOCUS, a rollout controller for language-model reinforcement learning that refines intermediate value estimates through active critic learning and selective rollouts. FOCUS naturally supports long-horizon agentic RL through flexible and modular context-management protocols such as compaction. Across DeonticBench, BrowseComp-Plus, and LocBench, FOCUS improves over GRPO and VinePPO under a matched per-step generation budget, with the largest held-out gains on deep search and deontic reasoning and consistent benefits on code localization. Training curves show that these improvements come with more stable exploration: FOCUS maintains higher collection reward and avoids the late-training collapse in tool use seen under trajectory-level group credit. Ablations indicate that gains are not tied to a single compaction implementation and that refinement rollouts are beneficial on BrowseComp-Plus. We release our modular rollout controller with interfaces for alternative harnesses, search policies, and credit estimators. Future work may scale FOCUS to larger models and longer budgets, tighten the link between compaction quality and training stability, and adopt stronger uncertainty quantification for value refinement.
Acknowledgments
This work was supported in part by the Defense Advanced Research Projects Agency (DARPA) Scientific Feasibility (SciFy) program under contract HR001125C0304; by Schmidt Sciences through the “Safely Confident Reasoning” project in its Science of Trustworthy AI program; and by the Office of Naval Research (ONR) under grant N00014-24-1-2089. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of DARPA, the Office of Naval Research, or Schmidt Sciences. We acknowledge the use of computational resources from the Johns Hopkins DSAI cluster.
Training setup
Tables tab:setup-bc-focus–Table 12 give the training setup of each method on each benchmark, one table per run configuration. Adam is from [25].
Shared FOCUS settings. Across all FOCUS experiments, we use , , and . Thus rewards are undiscounted, actor advantages use the local segment TD residual in Eq. (4), and the critic contributes one pseudo-observation to both the recursive value blend and the selection posterior. Total branching width is uncapped, with at most concurrent continuations per node; completed children do not count against this concurrency limit. These settings are shared across tasks and model sizes. The selection estimators are defined in Eq. (6) and Appendix § 10; the full training loop and budget semantics are given in Appendix § 11.
| Setting | Value |
| Actor model | Qwen3.5-9B |
| Critic model | 9B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Seeding budget (turns/prompt) | |
| Group size | Adaptive search |
| Questions | |
| Root trajectories | Set by adaptive search |
| Compaction trigger (prompt tokens) | 336 |
| Compaction trigger (tool calls) | |
| Actor reply cap (tokens) | 384 |
| Compaction reply cap (tokens) | 384 |
| Agent turn limit (turns) | |
| Context window (tokens) | 304 |
| Parameterization | Full |
| Optimizer | Adam |
| Actor learning rate | |
| Critic learning rate | |
| Learning-rate schedule | Constant |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | –6H100 |
| Critic GPUs (refinement) | H100 |
Table 2. GRPO on BrowseComp-Plus, Qwen3.5-9B.
| Setting | Value |
| Actor model | Qwen3.5-9B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Step advantage | Group advantage broadcast over steps |
| Group size | |
| Questions | |
| Root trajectories | |
| Compaction trigger (prompt tokens) | 336 |
| Compaction trigger (tool calls) | |
| Actor reply cap (tokens) | 384 |
| Compaction reply cap (tokens) | 384 |
| Agent turn limit (turns) | |
| Context window (tokens) | 304 |
| Parameterization | Full |
| Optimizer | Adam |
| Actor learning rate | |
| Learning-rate schedule | Constant |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | –6H100 |
Table 2. GRPO on BrowseComp-Plus, Qwen3.5-9B.
| Setting | Value |
| Actor model | Qwen3.5-9B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Branching states | Compaction points |
| Value-only continuations per state | () |
| Group size | |
| Questions | |
| Root trajectories | |
| Compaction trigger (prompt tokens) | 336 |
| Compaction trigger (tool calls) | |
| Actor reply cap (tokens) | 384 |
| Compaction reply cap (tokens) | 384 |
| Agent turn limit (turns) | |
| Context window (tokens) | 304 |
| Parameterization | Full |
| Optimizer | Adam |
| Actor learning rate | |
| Learning-rate schedule | Constant |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | –6H100 |
Table 4. FOCUS on DeonticBench, Qwen3.5-9B.
| Setting | Value |
| Actor model | Qwen3.5-9B |
| Critic model | 9B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Seeding budget (turns/prompt) | |
| Group size | Adaptive search |
| Questions | |
| Root trajectories | Set by adaptive search |
| Compaction trigger (prompt tokens) | 336 |
| Compaction trigger (tool calls) | |
| Actor reply cap (tokens) | 144 |
| Compaction reply cap (tokens) | 096 |
| Agent turn limit (turns) | |
| Context window (tokens) | 768 |
| Parameterization | Full |
| Optimizer | Adam |
| Actor learning rate | |
| Critic learning rate | |
| Learning-rate schedule | Constant |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | –6H100 |
| Critic GPUs (refinement) | H100 |
Table 4. FOCUS on DeonticBench, Qwen3.5-9B.
| Setting | Value |
| Actor model | Qwen3.5-9B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Step advantage | Group advantage broadcast over steps |
| Group size | |
| Questions | |
| Root trajectories | |
| Compaction trigger (prompt tokens) | 336 |
| Compaction trigger (tool calls) | |
| Actor reply cap (tokens) | 144 |
| Compaction reply cap (tokens) | 096 |
| Agent turn limit (turns) | |
| Context window (tokens) | 768 |
| Parameterization | Full |
| Optimizer | Adam |
| Actor learning rate | |
| Learning-rate schedule | Constant |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | –6H100 |
Table 6. VinePPO on DeonticBench, Qwen3.5-9B.
| Setting | Value |
| Actor model | Qwen3.5-9B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Branching states | Compaction points |
| Value-only continuations per state | () |
| Group size | |
| Questions | |
| Root trajectories | |
| Compaction trigger (prompt tokens) | 336 |
| Compaction trigger (tool calls) | |
| Actor reply cap (tokens) | 144 |
| Compaction reply cap (tokens) | 096 |
| Agent turn limit (turns) | |
| Context window (tokens) | 768 |
| Parameterization | Full |
| Optimizer | Adam |
| Actor learning rate | |
| Learning-rate schedule | Constant |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | –6H100 |
Table 6. VinePPO on DeonticBench, Qwen3.5-9B.
| Setting | Value |
| Actor model | Qwen3.5-9B |
| Critic model | 9B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Seeding budget (turns/prompt) | |
| Group size | Adaptive search |
| Questions | |
| Root trajectories | Set by adaptive search |
| Compaction trigger (prompt tokens) | 768 |
| Compaction trigger (tool calls) | (counter rolls over independently) |
| Actor reply cap (tokens) | 384 |
| Compaction reply cap (tokens) | 384 |
| Agent turn limit (turns) | task turns incl. compactions; final submission allowed afterward |
| Context window (tokens) | 536 |
| Parameterization | Full |
| Optimizer | Adam |
| Actor learning rate | |
| Critic learning rate | |
| Learning-rate schedule | Constant |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | –6H100 |
| Critic GPUs (refinement) | H100 |
Table 8. GRPO on LocBench, Qwen3.5-9B.
| Setting | Value |
| Actor model | Qwen3.5-9B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Step advantage | Group advantage broadcast over steps |
| Group size | |
| Questions | |
| Root trajectories | |
| Compaction trigger (prompt tokens) | 576 |
| Compaction trigger (tool calls) | |
| Actor reply cap (tokens) | 192 |
| Compaction reply cap (tokens) | 240 |
| Agent turn limit (turns) | |
| Context window (tokens) | 304 |
| Parameterization | Full |
| Optimizer | Adam |
| Actor learning rate | |
| Learning-rate schedule | Constant |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | –6H100 |
Table 8. GRPO on LocBench, Qwen3.5-9B.
| Setting | Value |
| Actor model | Qwen3.5-9B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Branching states | Compaction points |
| Value-only continuations per state | () |
| Group size | |
| Questions | |
| Root trajectories | |
| Compaction trigger (prompt tokens) | 576 |
| Compaction trigger (tool calls) | |
| Actor reply cap (tokens) | 192 |
| Compaction reply cap (tokens) | 240 |
| Agent turn limit (turns) | |
| Context window (tokens) | 304 |
| Parameterization | Full |
| Optimizer | Adam |
| Actor learning rate | |
| Learning-rate schedule | Constant |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | –6H100 |
Table 10. FOCUS on DeonticBench, Qwen3.5-27B.
| Setting | Value |
| Actor model | Qwen3.5-27B |
| Critic model | 4B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Seeding budget (turns/prompt) | |
| Group size | Adaptive search |
| Questions | |
| Root trajectories | Set by adaptive search |
| Compaction trigger (prompt tokens) | 336 |
| Compaction trigger (tool calls) | |
| Actor reply cap (tokens) | 144 |
| Compaction reply cap (tokens) | 096 |
| Agent turn limit (turns) | |
| Context window (tokens) | 768 |
| Parameterization | LoRA |
| LoRA | |
| Optimizer | Adam |
| Actor learning rate | |
| Critic learning rate | |
| Value clip | |
| Learning-rate schedule | Constant, no warmup |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | H200 |
| Critic GPUs (refinement) | H200 |
Table 10. FOCUS on DeonticBench, Qwen3.5-27B.
| Setting | Value |
| Actor model | Qwen3.5-27B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Step advantage | Group advantage broadcast over steps |
| Group size | |
| Questions | |
| Root trajectories | |
| Compaction trigger (prompt tokens) | 336 |
| Compaction trigger (tool calls) | |
| Actor reply cap (tokens) | 144 |
| Compaction reply cap (tokens) | 096 |
| Agent turn limit (turns) | |
| Context window (tokens) | 768 |
| Parameterization | LoRA |
| LoRA | |
| Optimizer | Adam |
| Actor learning rate | |
| Learning-rate schedule | Constant, no warmup |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | H200 |
Table 12. VinePPO on DeonticBench, Qwen3.5-27B.
| Setting | Value |
| Actor model | Qwen3.5-27B |
| Training length | epoch |
| Training-data order | Identical across methods |
| Compaction operator | On |
| Rollout temperature | |
| Advantage cutoff | |
| Branching states | Compaction points |
| Value-only continuations per state | () |
| Group size | |
| Questions | |
| Root trajectories | |
| Compaction trigger (prompt tokens) | 336 |
| Compaction trigger (tool calls) | |
| Actor reply cap (tokens) | 144 |
| Compaction reply cap (tokens) | 096 |
| Agent turn limit (turns) | |
| Context window (tokens) | 768 |
| Parameterization | LoRA |
| LoRA | |
| Optimizer | Adam |
| Actor learning rate | |
| Learning-rate schedule | Constant, no warmup |
| Adam | |
| Weight decay | |
| Gradient-norm clip | |
| PPO clip ratio | |
| KL coefficient | |
| Entropy bonus | |
| Training GPUs | H200 |
| Rollout GPUs | H200 |
Table 12. VinePPO on DeonticBench, Qwen3.5-27B.
Bootstrap procedure for main-result differences
Table tab:main-results reports FOCUS minus GRPO for every metric. We keep each reported checkpoint fixed during resampling. For LocBench, we draw 100,000 bootstrap samples of the original case count with replacement, using the same sampled case indices for FOCUS and its comparator. We compute the accuracy difference in each replicate; the 5th and 95th percentiles define a two-sided 90% percentile interval. Differences use exact counts before rounding where available. We use seed 20261021. Numerical intervals appear only in the additional rows of Table tab:main-results.
DeonticBench, Qwen3.5-9B. We use the corrected evaluation breakdown and its supplied paired 90% intervals. The evaluation contains 150 Airline, 50 SARA-N, and 138 SARA-B cases (338 total); the 71 Hard cases overlap these domains. The corrected FOCUS accuracies correspond to 32, 26, and 133 correct answers in the three domains, respectively (191 overall), with 19 correct on Hard. GRPO has 21, 28, and 128 correct (177 overall), with 20 correct on Hard. The supplied intervals are preserved rather than recomputed from the historical per-case export, which does not represent these corrected results.
LocBench. We pair 87 held-out query identifiers. FOCUS uses the earlier-run checkpoint 35 for both File@1 and File@5. GRPO’s File@1 score comes from checkpoint 1, and its File@5 score from checkpoint 89. Checkpoint 96 ties checkpoint 89 on aggregate File@5; we use the earlier matching checkpoint. Thus the two columns compare different GRPO checkpoints, consistent with the separately reported metric scores.
BrowseComp-Plus. The point difference uses the reported FOCUS and GRPO scores on 249 questions. We use the supplied 90% paired interval for this comparison; the underlying per-question outcomes are not included in the local analysis export.
DeonticBench, Qwen3.5-27B. We report point differences from the rounded FOCUS and GRPO manuscript scores. For Overall, the supplied paired counts are 27 cases correct only for FOCUS, 20 correct only for GRPO, and 291 with concordant outcomes (338 total). For Hard, the corresponding counts are 5, 5, and 61 (71 total); for SARA-B, they are 3, 0, and 135 (138 total); and for SARA-N, they are 4, 3, and 43 (50 total). Subtracting the SARA-N and SARA-B counts from Overall gives Airline counts of 20, 17, and 113 (150 total), assuming these domain counts refer to the same evaluation. We bootstrap these paired differences using 1,000,000 multinomial resamples with seed 20261021 and take the 5th and 95th percentiles.
The intervals are marginal, without multiple-comparison adjustment, and conditional on the selected checkpoints. They do not account for held-out checkpoint selection, training-seed variation, or repeated stochastic decoding. Case resampling also does not model dependence among related problem families or repository issues. Reproduction inputs and checkpoint provenance are preserved with the analysis script.
BrowseComp-Plus cold-starting
We cold-start BrowseComp-Plus with supervised imitation before the matched RL runs in § 3. We first collect 200 successful, uncompacted trajectories from Qwen3.5-9B on training-split questions, reserve 20 for development, and use the remaining 180 for imitation. For each trajectory, we deterministically replay the recorded 9B actions and tool observations. Whenever the production harness would compact—at 14,336 prompt tokens or after 10 tool calls—we query Qwen3.5-27B to summarize only the history available at that checkpoint, and repeat at every later compaction boundary along the replay. The teacher prompt requires evidence, settled requirements, and genuinely open questions, and fixes the Next steps field to “None” so compactions do not instruct the actor to repeat verification. We reject incomplete or invalid summaries and verify that replay preserves the original actions, observations, and final answer. Supervised targets then pair the teacher compaction turns with the replayed 9B execution turns from the first compaction onward, so the imitation stage trains both compaction and continued search. Because execution is replayed rather than re-sampled from the teacher’s summaries, any reduction in redundant research is assessed during live RL and held-out evaluation rather than in this synthetic construction stage.
Compaction prompts
Figure 5c compares two compaction prompts on LocBench. In both, compaction runs under a NullWorkspace: it has no tools and no store, so its reply is the agent’s only memory once the transcript is discarded. The prompts differ in how the reply must be shaped, whether it is checked, and how it re-enters the agent’s context (Table 13). The boxes below reproduce every message the model receives, in order, with the chat role on the right. Line breaks are reflowed, (s) slots; mark values filled at run time, and small grey labels name the source of each segment. Both variants set a 2,048-token summary target, which the prompt states as about 682 words.
| Free-form | Structural | |
| Compaction instruction | LocBench-specific instruction with the NullWorkspace guidance inserted | Generic instruction with the NullWorkspace guidance, followed by a LocBench schema and an output contract |
| Reply format | Four suggested headings; the reply is kept verbatim | Exactly one <summary> block with four non-empty sections: State, Evidence, Open questions, Next steps |
| Validation | Rejected if empty, cut off at the reply cap, or inside unclosed reasoning | Also parsed into the four sections and held to the 2,048-token target; one repair retry. Compaction turns are penalized on failure and when the wrapper is recovered |
| Next steps | Always “None” | A short next action; any reread must match an Open question |
| Re-entry | Notes appended to the original request inside <yournotes> (user turn) | Request left untouched; notes in their own <assistantmemory> turn, then a resume instruction |
| Execution state | None | Harness-rendered JSON of executed calls, counters, and submissions |
| Reread rule | Re-establish from the task’s sources whatever the notes leave open | Treat State and Evidence as settled; reread only items listed under Open questions |
| Compaction sampling | Library default (temperature ) | Temperature , top- |
| Compaction trigger | Prompt length or spent per-round call budget | Prompt length only |
Table 13. Free-form versus structural compaction on LocBench. Both compactions are tool-free (NullWorkspace) and must finish within the reply cap.
Free-form compaction
The compaction turn sees an instruction and the wrapped transcript. Its reply is spliced into the agent’s original request.
Structural compaction
This prompt was used by the robust-null Improved PPO runs in Figure 5c. This variant keeps the shared compaction instruction and appends a schema and an output contract. A reply is accepted only if it parses into the four sections. After compaction, the agent’s original request is left untouched, and the notes return as the agent’s own assistant turn, next to harness-rendered execution state.
Outcome evidence for selective rollout allocation
FOCUS uses continuation evidence in two distinct ways: to refine state values for learning (§ 2.2) and to select candidate states for additional continuations (§ 2.3). This appendix explains why selective rollout allocation uses observed continuation outcomes rather than the recursive backups used for training. Throughout a collection round, the policy and critic are fixed. As in the methodology, denotes a state’s context, with its associated environment state implicit, and denotes the critic prediction at that state.
For critic refinement, counts direct children in . The recursive backup averages their contributions , and the refined state value combines this backup with the local critic prediction (Eqs. (1)–(2)). For allocation, counts observed continuation outcomes, whose mean is . These observations determine the posterior mean and the estimates and used in the acquisition score (Eqs. (6)–(7)). In particular, does not use downstream refined state values.
Recursive backups attenuate the residual used for selection. Consider a completed continuation that forms a chain with one direct child at each nonterminal state, zero intermediate rewards, terminal outcome , and . Suppose the critic predicts the same value at every nonterminal state and is fixed. Index these states by , the number of edges remaining until termination, and write and for their recursive backups and refined state values. Since at each state, setting gives
For , the terminal edge gives and . For , the zero-reward edge gives ; substitution into Eq. (2) proves the identities by induction. Thus, using as selection evidence would attenuate the same realized critic–outcome discrepancy with every additional recursive backup. Under the default , this squared residual decreases by a factor of four per edge, even though the critic prediction and observed continuation outcome are identical at every state.
For example, with and , the squared residuals for are , , , and , respectively. If this is the initial completed continuation, each state instead has and . For a binary outcome and , Eq. (6) gives
This expected squared critic error equals at every state in the example. The local critic prior still regularizes the estimate, but there is no additional attenuation from downstream refined state values. This observation motivates the use of continuation outcomes for allocation. It does not identify the true critic error from one outcome: the posterior retains sampling uncertainty. With or nonzero intermediate rewards, observed continuation returns may also differ across states according to the return definition.
Continuation outcomes under selective rollout allocation. Let denote the collection history immediately before a continuation is sampled, including the selected candidate state . Let denote the observed discounted return of that continuation. If the policy resumes from the state’s context and associated environment state and samples to termination without filtering by outcome, then
Conditioning on the realized prefix gives the same statement at each newly created nonterminal state along the continuation, before its remaining suffix is sampled. A completed continuation therefore supplies one observed outcome at its starting state and at each such new state, as specified in § 2.3.
A later continuation allocated to a descendant is an additional draw from that descendant, but not from its strict ancestors: the prefix leading to the selected descendant has not been resampled. Pooling all descendant outcomes at an ancestor would weight prefixes according to the allocation decisions. FOCUS therefore updates and only at the starting state and newly created nonterminal states of the completed continuation. Recursive backups may still update ancestor training values. Equation (10) concerns each conditional draw; it does not imply that a sample mean at an adaptively chosen stopping time is unbiased.
Pseudo-observation weight and initial continuation evidence. The default gives the local critic prediction weight in the refined state value , and weight in the selection posterior mean . The pseudo-observation weight stays fixed as continuations are collected. This is a regularization choice. For the simplified MSE model in § 2.3, with fixed , , and , minimizing over gives . Thus is optimal in that model only when ; interpreting the critic as one pseudo-observation does not establish this equality.
For the binary selection model, setting and in Eqs. (6) and (7) gives
Consequently, the acquisition score does not reduce to uncertainty sampling at a state with no observed continuation outcomes. The initial completed episode supplies one outcome at every nonterminal state on its trajectory, so those states enter allocation with . Additional seeding supplies further continuation evidence.
Recursive backups for critic supervision and actor advantages. Equation (1) assigns one contribution to each direct child, regardless of how many additional continuations are subsequently allocated below that child. New descendant evidence changes the child’s refined state value and propagates through that contribution, preserving the weighting over direct children. The recursive backup supplies the critic supervision target in Eq. (3); the refined state values supply the actor advantages in Eq. (4). This use of downstream refined values can reduce sampling variance, but also propagates errors in those values and does not guarantee a lower MSE.
Selective rollout allocation instead estimates critic error from observed continuation outcomes, avoiding the additional attenuation in Eq. (8). The acquisition score is derived from the simplified MSE model with and held fixed, then evaluated using their posterior estimates and . It remains a heuristic for the expected benefit of a continuation, rather than an exact expected reduction in the recursive training target’s MSE or in actor-gradient error.
Full reinforcement-learning procedure
Algorithm alg:full-rl embeds selective rollout allocation (Algorithm alg:refine) in the actor–critic training loop. All experiments use , , , and ; optimizer settings are given in Appendix § 6.
\caption{FOCUS actor--critic training with selective rollout allocation}
\label{alg:full-rl}
\small
\algrenewcommand{\algorithmiccomment}[1]{\hfill\(\triangleright\) {\scriptsize #1}}
\begin{algorithmic}
\Require prompts $\mathcal D$; parameters $(\theta_0,\phi_0)$; rounds $K$; $\gamma,\kappa$
\Require $B_{\mathrm{init}},B_{\mathrm{seed}},B_{\mathrm{ref}}$ \Comment{separate transition budgets; failure costs one unit}
\Require configured $\textsc{Continue}$, collection schedules, and optimizers
\Ensure trained parameters $(\theta_K,\phi_K)$
\Statex \Comment{Finish started continuations; charge overshoot only to the current phase.}
\Statex \Comment{$\textsc{Collect}$ completes and attaches $\textsc{Continue}$ results (Algorithm~\ref{alg:refine}).}
\Statex \Comment{Use the same $\mathcal G$ and $\mathcal C_{\pi_{\mathrm{old}}}$ throughout collection.}
\For{$k=0,\ldots,K-1$}
\State $(\pi_{\mathrm{old}},V_{\mathrm{old}})\gets(\pi_{\theta_k},V_{\phi_k})$ \Comment{fixed during collection}
\State $\mathcal X_k\sim\mathcal D$; $\mathcal D_A\gets\emptyset$; $\mathcal D_V\gets\emptyset$
\For{$x\in\mathcal X_k$}
\State $(\mathcal T_x,\bar c_0)\gets\textsc{Init}(x)$ \Comment{fresh root and environment}
\State $\mathcal T_x\gets\textsc{Collect}(\mathcal T_x,\{\bar c_0\},\pi_{\mathrm{old}},B_{\mathrm{init}})$ \Comment{initial episodes}
\State $\textsc{Score}(\mathcal T_x,V_{\mathrm{old}})$ \Comment{cache unscored nonterminal values}
\State $\mathcal S_0\gets\textsc{Candidates}(\mathcal T_x)$ \Comment{resumable nonterminal nodes}
\State $\mathcal T_x\gets\textsc{Collect}(\mathcal T_x,\mathcal S_0,\pi_{\mathrm{old}},B_{\mathrm{seed}})$ \Comment{seed continuations}
\State $\textsc{Score}(\mathcal T_x,V_{\mathrm{old}})$
\State $\mathcal T_x\gets\textsc{AllocateAndRefine}(\mathcal T_x,\pi_{\mathrm{old}},V_{\mathrm{old}},B_{\mathrm{ref}},\kappa)$ \Comment{Algorithm~\ref{alg:refine}}
\For{$e=(\bar c,\bar c')\in\mathcal E_A(\mathcal T_x)$} \Comment{each distinct trainable edge once}
\State $\hat A_e\gets\operatorname{stopgrad}\!\left(R_e+\gamma^{d_e}\tilde V(\bar c')-\tilde V(\bar c)\right)$
\Statex \Comment{$y_e,\mu_e,\ell_e$: collected tokens, actor mask, behavior log probabilities}
\State $\mathcal D_A\gets\mathcal D_A\uplus\{(y_e,\mu_e,\ell_e,\hat A_e)\}$ \Comment{$\uplus$: preserve repeated samples}
\EndFor
\For{$\bar c\in\mathcal T_x$ nonterminal, $n(\bar c)>0$}
\State $\mathcal D_V\gets\mathcal D_V\uplus\{(\bar c,\operatorname{stopgrad}(\bar G(\bar c)))\}$
\EndFor
\EndFor
\State $\mathcal D_A\gets\textsc{Filter}(\mathcal D_A)$ \Comment{optional: drop edges with $|\hat A_e|\leq$ cutoff}
\Statex \Comment{Except malformed compactions; preserve coefficients and original edge-count normalization.}
\State $\theta_{k+1}\gets\textsc{ActorUpdate}(\theta_k,\mathcal D_A,\pi_{\mathrm{old}})$ \Comment{Eq.~\ref{eq:actor}; identity if $\mathcal D_A=\emptyset$}
\State $\phi_{k+1}\gets\textsc{CriticUpdate}(\phi_k,\mathcal D_V)$ \Comment{Eq.~\ref{eq:critic}; identity if $\mathcal D_V=\emptyset$}
\State $\textsc{Publish}(\theta_{k+1},\phi_{k+1})$ \Comment{synchronize collection workers}
\EndFor
\State \Return $(\theta_K,\phi_K)$
\end{algorithmic}
TD() advantage extension
For a completed episode, the segment-level TD() extension [43] of Eq. (4) is
where controls trace decay across segments. All experiments use , recovering ; gives the observed return minus .
GRPO submit-after-compaction behavior
The boxes below are held-out episodes from late GRPO checkpoints, rendered from their saved traces by scripts/grpo_premature_cases.py. Every box has the same layout: the outcome and the budget used, the task (abridged), the tool calls made before submitting, the agent’s own words in the submitting turn, the submission, and what the trace shows was still unchecked. Trace text is verbatim; text is evidence the agent had in context, and
text is the claim it submitted on without checking.
Execution Model and Harness Mechanics
This section specifies the tool-using instantiation of the learning setup in § 2.1. The harness manages execution and context, while the rollout controller chooses the states from which to sample additional continuations. The mathematical-reasoning evaluation uses prefix continuations without compaction.
Tool calls and environment state. An environment exposes tools , such as (t) grep; and (t) read; over a repository or (t) search; over a fixed corpus. Given context , the policy generates a task turn ending in a tool call . The environment executes the call and returns an updated state, observation, and reward, . Here contains the task state, workspace, and execution information needed to resume the run; denotes the model context. The harness appends the generated turn and observation to the context and retains the updated environment state. In our agent benchmarks, a special (t) submit; call ends the episode with a final response; an episode also ends when its configured turn limit is reached.
Context management and compaction. The policy has a finite context window of tokens. When a configured trigger fires, the harness invokes a compaction procedure to replace the accumulated context with a shorter one. We write for brevity, suppressing any dependence on execution counters. The triggers used in the experiments include context length and, where configured, a per-round call budget (Appendices § 6 and § 9). Following recent long-horizon training work [6, 14, 34], compaction itself uses policy-generated turns. The harness constructs the resumed context from the resulting notes and configured task instructions. Compaction changes the model’s context while preserving task state and workspace separately; it does not itself advance the task environment.
Starting from prompt , the context evolution is
where and on task turns, and denotes concatenation. This equation describes the visible context; environment transitions and execution counters are maintained separately. Compaction may contain multiple generated turns. An episode records both task and compaction turns, together with their rewards, until termination.
Resumption and segment construction. Here denotes the evolving context at execution step , while denotes the context at the th selected branchable state, consistent with § 2.2. The \textsc{Continue} helper in Algorithm alg:refine writes the current context as and a newly recorded node context as . The branch predicate is independent of the compaction trigger : a selected state can follow a task transition or a compaction, and compaction need not create a branching node. The compaction update uses the same operator as Eq. (13). Our compaction-based experiments select post-compaction states for branching. To sample a continuation, the runner restores that context together with the corresponding task, workspace, and execution state. It then samples under the fixed collection policy. Preserving the context alone would not suffice when previous actions have changed the environment. Under this experimental choice, an episode with compactions has segments: each contains the task turns up to the next compaction and the compaction turns that construct the next context; the final segment ends at termination. Between context changes, task turns extend the preceding context, permitting sequence training with observations masked [21, 7]. Policy-generated task and compaction tokens receive actor credit; environment observations and fixed harness text are excluded from the actor loss. Appendix § 11 specifies the training procedure.
Modularity and experimental variants. The allocation rule depends on resumable states, observed suffix returns, and critic predictions. The context-management component determines how those states are represented. Setting disables the optional compaction block in Algorithm alg:refine; candidate nodes then follow the task’s resumption rule, as in the mathematical-reasoning evaluation. Our experiments vary compaction operators and prompts while retaining this separation. Appendix § 9 gives the free-form and structural LocBench prompts, validation and repair rules, auxiliary penalties, and the exact placement of notes and execution metadata in the resumed context. Appendix § 6 reports the context, generation, and episode limits.
The modular design provides interfaces for extensions to richer context management, such as recursive calls in Recursive Language Models [67], editable context files in Context Language Models [45], and indexed experience memory in MemexRL [59]. Our implementation provides asynchronous sampling adapters for Slime [79] and Tinker [53].
References
- [1]Cao, Ruan, Li, Chao, Ning, Song, Chen, & Li. (2026). Tree A dv: Tree-structured advantage redistribution for group-based RL. https://arxiv.org/abs/2601.03703
- [2]Chen, Tang, Deng, Wu, Wu, Jiang, Prasanna, Cohan, & Wang. (2025). Loc A gent: Graph-guided LLM agents for code localization. https://arxiv.org/abs/2503.09089
- [3]Chen, Ma, Zhuang, Nie, Zou, Sharifymoghaddam, Liu, Green, Patel, Meng, Su, Li, Hong, Shi, Liu, Oyarhoseini, Thakur, Zhang, Gao, … Lin. (2026). B rowse C omp-plus: A fair and disentangled evaluation benchmark for deep search agents. https://aclanthology.org/2026.acl-long.1023/aclanthology.org/2026.acl-long.1023
- [4]Choudhury. (2025). Process reward models for LLM agents: Practical framework and directions. https://arxiv.org/abs/2502.10325
- [5]Cohn, Ghahramani, & Jordan. (1996). Active learning with statistical models. https://arxiv.org/abs/cs/9603104
- [6]Cursor. (2026). Training C omposer for longer horizons. https://cursor.com/blog/self-summarizationcursor.com/blog/self-summarization
- [7]Cursor Research. (2026). Composer 2 technical report. https://arxiv.org/abs/2603.24477
- [8]Dong, Mao, Ma, Bao, Chen, Wang, Chen, Du, Wang, Zhang, Zhou, Zhu, Wen, & Dou. (2026). Agentic reinforced policy optimization. https://arxiv.org/abs/2507.19849
- [9]Dou, Brena, Deo, Jurayj, Zhang, Holzenberger, & Van Durme. (2026a). Deontic B ench: A benchmark for reasoning over rules. https://arxiv.org/abs/2604.04443
- [10]Dou, Jurayj, Holzenberger, & Van Durme. (2026b). DAR : Deontic reasoning with agentic harnesses. https://arxiv.org/abs/2606.05009
- [11]Guan, Zhang, Liu, Shang, Sun, Zhu, Yang, & Yang. (2025). r S tar- M ath: Small LLMs can master math reasoning with self-evolved deep thinking. https://arxiv.org/abs/2501.04519
- [12]Guo, Xu, Liu, Ye, & Qiu. (2025). Segment policy optimization: Effective segment-level credit assignment in RL for large language models. https://arxiv.org/abs/2505.23564
- [13]Hao, Gu, Ma, Hong, Wang, Wang, & Hu. (2023). Reasoning with language model is planning with world model.
- [14]Harvey. (2026). Harvey T enet research preview. https://www.harvey.ai/blog/post-training-update-harvey-tenetharvey.ai/blog/post-training-update-harvey-tenet
- [15]Hay, Russell, Tolpin, & Shimony. (2012). Selecting computations: Theory and applications.
- [16]Hendrycks, Burns, Kadavath, Arora, Basart, Tang, Song, & Steinhardt. (2021). Measuring mathematical problem solving with the MATH dataset. https://arxiv.org/abs/2103.03874
- [17]Hou, Hu, Li, Lu, Tang, & Dong. (2025). Tree RL : LLM reinforcement learning with on-policy tree search. https://arxiv.org/abs/2506.11902
- [18]Hou, Li, Tang, & Dong. (2026). Single-rollout asynchronous optimization for agentic reinforcement learning. https://arxiv.org/abs/2607.07508
- [19]Ji, Ma, Wang, Chen, Chu, & Wu. (2026). Tree search for LLM agent reinforcement learning. https://arxiv.org/abs/2509.21240
- [20]Jimenez, Yang, Wettig, Yao, Pei, Press, & Narasimhan. (2024). SWE -bench: Can language models resolve real-world GitHub issues?. https://arxiv.org/abs/2310.06770
- [21]Jin, Zeng, Yue, Yoon, Arik, Wang, Zamani, & Han. (2025). Search- R1 : Training LLMs to reason and leverage search engines with reinforcement learning. https://arxiv.org/abs/2503.09516
- [22]Kaufmann, & Koolen. (2017). M onte- C arlo tree search by best arm identification.
- [23]Kazemnejad, Aghajohari, Portelance, Sordoni, Reddy, Courville, & Le Roux. (2025). Vine PPO : Refining credit assignment in RL training of LLMs.
- [24]Kimi Team. (2026). Kimi K3 : Open frontier intelligence. https://arxiv.org/abs/2607.24653
- [25]Kingma, & Ba. (2015). Adam: A method for stochastic optimization.
- [26]Koh, & Liang. (2017). Understanding black-box predictions via influence functions. https://proceedings.mlr.press/v70/koh17a.htmlproceedings.mlr.press/v70/koh17a.html
- [27]Konda, & Tsitsiklis. (1999). Actor-critic algorithms.
- [28]Lei, Li, Choubey, Zhang, Peng, Ye, Narayan, Ding, Savarese, & Wu. (2026). Fork where the model changes its mind: Belief-shift branching for tree-structured reinforcement learning. https://arxiv.org/abs/2609.11061
- [29]Li, Zhou, Meng, Vadera, Li, & Li. (2026a). Turn- PPO : Turn-level advantage estimation with PPO for improved multi-turn RL in agentic LLMs. https://arxiv.org/abs/2512.17008
- [30]Li, Zhang, Jurayj, Wang, Jin, Farajtabar, Nalisnick, & Khashabi. (2026b). Self-compacting language model agents. https://arxiv.org/abs/2606.23525
- [31]Li, & Li. (2025). Process reward model with Q -value rankings. https://arxiv.org/abs/2410.11287
- [32]Li, Li, Zhang, Yu, Lin, Jin, Guan, Liu, Li, Yin, & Shang. (2026c). Process reward informed tree rollout for effective multi-turn RL. https://arxiv.org/abs/2607.15610
- [33]Li, Gu, Wen, Li, Xing, Guo, Zheng, Zhou, Qu, Zhou, Zhang, Shen, Liu, Lin, Yang, Zhang, & Huang. (2026d). Tree PO : Enhancing policy efficacy and inference efficiency with tree modeling. https://arxiv.org/abs/2508.17445
- [34]Li, Hou, Jing, Tang, & Dong. (2026e). Compaction RL : Reinforcement learning with context compaction for long-horizon agents. https://arxiv.org/abs/2607.05378
- [35]Lightman, Kosaraju, Burda, Edwards, Baker, Lee, Leike, Schulman, Sutskever, & Cobbe. (2024). Let's verify step by step.
- [36]Liu, Wang, Shi, Xie, An, Zhang, Zhao, Gu, Lin, Hu, Li, Zhang, Zhou, & Gai. (2026). Attention as a compass: Efficient exploration for process-supervised RL in reasoning models. https://arxiv.org/abs/2509.26628
- [37]Luo, Liu, Liu, Phatale, Guo, Lara, Li, Shu, Zhu, Meng, Sun, & Rastogi. (2024). Improve mathematical reasoning in language models by automated process supervision. https://arxiv.org/abs/2406.06592
- [38]Mussmann, Reisler, Tsai, Mousavi, O'Brien, & Goldszmidt. (2022). Active learning with expected error reduction. https://arxiv.org/abs/2211.09283
- [39]Packer, Wooders, Lin, Fang, Patil, Stoica, & Gonzalez. (2023). MemGPT : Towards LLMs as operating systems. https://arxiv.org/abs/2310.08560
- [40]Qwen Team. (2025). Qwen3 technical report, 2025. https://arxiv.org/abs/2505.09388
- [41]Qwen Team. (2026). Qwen3.5 : Towards native multimodal agents, February 2026. https://qwen.ai/blog?id=qwen3.5qwen.ai/blog?id=qwen3.5
- [42]Roy, & McCallum. (2001). Toward optimal active learning through sampling estimation of error reduction. https://groups.csail.mit.edu/rrg/papers/icml01.pdfgroups.csail.mit.edu/rrg/papers/icml01.pdf
- [43]Schulman, Moritz, Levine, Jordan, & Abbeel. (2016). High-dimensional continuous control using generalized advantage estimation.
- [44]Setlur, Nagpal, Fisch, Geng, Eisenstein, Agarwal, Agarwal, Berant, & Kumar. (2025). Rewarding progress: Scaling automated process verifiers for LLM reasoning. https://arxiv.org/abs/2410.08146
- [45]Shao, Shen, Yin, Li, Wang, Ivison, Poovendran, Lambert, Xiao, Lewis, Yih, Zettlemoyer, & Koh. (2026). Context language models. https://arxiv.org/abs/2609.37725
- [46]Shao, Wang, Zhu, Xu, Song, Bi, Zhang, Zhang, Li, Wu, & Guo. (2024). Deep S eek M ath: Pushing the limits of mathematical reasoning in open language models. https://arxiv.org/abs/2402.03300
- [47]Silver, Huang, Maddison, Guez, Sifre, van den Driessche, Schrittwieser, Antonoglou, Panneershelvam, & Lanctot. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484–489.
- [48]Snell, Lee, Xu, & Kumar. (2025). Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. https://arxiv.org/abs/2408.03314
- [49]Sun, Lu, Ling, Liu, Yao, Yang, & Chen. (2026). Scaling long-horizon agent via context folding. https://arxiv.org/abs/2510.11967
- [50]Tan, Yang, Chen, Shao, Wen, Shen, Luo, Du, Guo, & Li. (2026). Hindsight credit assignment for long-horizon LLM agents. https://arxiv.org/abs/2603.08754
- [51]Tao, Peng, Yao, Ge, Cheng, Wang, Gao, & Li. (2026). TRACE : Turn-level reward assignment via credit estimation for long-horizon agents. https://arxiv.org/abs/2607.13988
- [52]Tesauro, Rajan, & Segal. (2010). Bayesian inference in M onte- C arlo tree search.
- [53]Thinking Machines Lab. (2026). Tinker, 2026. https://thinkingmachines.ai/tinker/thinkingmachines.ai/tinker
- [54]Tian, Peng, Song, Jin, Yu, Han, Mi, & Yu. (2024). Toward self-improvement of LLMs via imagination, searching, and criticizing.
- [55]Tran, Yao, & Yu. (2025). Exploiting tree structure for credit assignment in RL training of LLMs. https://arxiv.org/abs/2509.18314
- [56]Venkatraman, Dinot, & Aitchison. (2026). Le critique: Privileged value functions for LLM reinforcement learning. https://arxiv.org/abs/2608.16739
- [57]Wang, Cui, Zhang, Liu, Jin, Geng, Zhang, Zalmout, Shi, & Sun. (2026a). T$^2$PO : Uncertainty-guided exploration control for stable multi-turn agentic reinforcement learning. https://arxiv.org/abs/2605.02178
- [58]Wang, Li, Shao, Xu, Dai, Li, Chen, Wu, & Sui. (2024). Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. https://arxiv.org/abs/2312.08935
- [59]Wang, Chen, Wang, & Wei. (2026b). Memex(RL) : Scaling long-horizon LLM agents via indexed experience memory. https://arxiv.org/abs/2603.04257
- [60]Wang, Wang, Wang, Zhang, Li, Yang, Jin, Yu, Nguyen, Liu, Gottlieb, Lu, Cho, Wu, Fei-Fei, Wang, Choi, & Li. (2025). RAGEN : Understanding self-evolution in LLM agents via multi-turn reinforcement learning. https://arxiv.org/abs/2504.20073
- [61]Wei, Zeng, Li, Wang, Brown, Frunza, Deng, Schneider, Nevmyvaka, Zhao, Garcia, & Hong. (2025). Reinforcing multi-turn reasoning in LLM agents via fine-grained reward structure and credit assignment. https://arxiv.org/abs/2505.11821
- [62]Wu, Xuan, Qi, Tu, Lu, Li, & Choi. (2026). Deep S earch: Overcome the bottleneck of reinforcement learning with verifiable rewards via tree-based search. https://arxiv.org/abs/2509.25454
- [63]Xu, & Kazantsev. (2019). Understanding goal-oriented active learning via influence functions. https://arxiv.org/abs/1905.13183
- [64]Yang, Guo, Huang, Liang, Wang, & Tang. (2025). Tree RPO : Tree relative policy optimization. https://arxiv.org/abs/2506.05183
- [65]Yao, Yu, Zhao, Shafran, Griffiths, Cao, & Narasimhan. (2023). Tree of thoughts: Deliberate problem solving with large language models.
- [66]Yu, Chen, Feng, Chen, Dai, Yu, Zhang, Ma, Liu, Wang, & Zhou. (2026). MemAgent : Reshaping long-context LLM with multi-conv RL -based memory agent. https://arxiv.org/abs/2507.02259
- [67]Zhang, Kraska, & Khattab. (2025). Recursive language models. https://arxiv.org/abs/2512.24601
- [68]Zhang. (2026). From reasoning to agentic: Credit assignment in reinforcement learning for large language models. https://arxiv.org/abs/2604.09459
- [69]Zhang, Zhoubian, Hu, Yue, Dong, & Tang. (2024). ReST-MCTS* : LLM self-training via process reward guided tree search.
- [70]Zhang, Wang, Jia, Shen, Ding, Viola, Schmidhuber, & Elhoseiny. (2026a). Hybrid advantage estimation with unified critic for VLM agentic reinforcement learning. https://arxiv.org/abs/2607.23605
- [71]Zhang, Xu, Ding, Xie, Gao, Ding, Zhang, Fu, & Wang. (2026b). Information gain-based rollout policy optimization: An adaptive tree-structured rollout approach for multi-turn LLM agents. https://arxiv.org/abs/2607.06223v1
- [72]Zhen, Yu, Guo, Cheng, & Deng. (2026). Hierarchical reinforcement learning with augmented step-level transitions for LLM agents. https://arxiv.org/abs/2604.05808
- [73]Zheng, Xing, Gu, Liang, Qu, Zhou, Li, Wen, Lin, Huang, Liu, Zhang, & Ma. (2025). First return, entropy-eliciting explore. https://arxiv.org/abs/2507.07017
- [74]Zhong, Yang, Wu, Liu, Yao, & Kuang. (2026). SIGHT : Reinforcement learning with self-evidence and information-gain diverse branching for search agent. https://arxiv.org/abs/2602.11551
- [75]Zhou, Yan, Shlapentokh-Rothman, Wang, & Wang. (2024a). Language agent tree search unifies reasoning, acting, and planning in language models. https://arxiv.org/abs/2310.04406
- [76]Zhou, Xu, Zhu, Zhou, Lo, Sridhar, Cheng, Ou, Bisk, Fried, Alon, & Neubig. (2024b). WebArena : A realistic web environment for building autonomous agents. https://arxiv.org/abs/2307.13854
- [77]Zhou, Zanette, Pan, Levine, & Kumar. (2024c). Ar CH er: Training language model agents via hierarchical multi-turn RL.
- [78]Zhou, Jiang, Tian, Weston, Levine, Sukhbaatar, & Li. (2025). SWEET-RL : Training multi-turn LLM agents on collaborative reasoning tasks. https://arxiv.org/abs/2503.15478
- [79]Zhu, Xie, Lv, & slime Contributors. (2025). slime: An LLM post-training framework for RL scaling, 2025. https://github.com/THUDM/slimegithub.com/THUDM/slime
- [80]Zong, Chen, Li, Yi, Zhou, Li, Qian, Chen, & Jiang. (2026). AT$^2$PO : Agentic turn-based policy optimization via tree search. https://arxiv.org/abs/2601.04767
- [81]Zou, Wang, Qu, Jiang, Cai, Mao, Peng, Xu, Liu, Yang, Yang, & Ji. (2026). TRACE : A unified rollout budget allocation framework for efficient agentic reinforcement learning. https://arxiv.org/abs/2606.11119

















