WorksComputer Science
Depth Before Breadth: Reverse-to-Forward Curriculum with Advantage Decomposition for LLM Agent
Abstract
Reinforcement learning (RL) has become a common approach for training large language model (LLM) agents on long-horizon tasks. However, learning from sparse terminal rewards makes credit assignment and exploration difficult. We approach these challenges through a policy-induced capability region, the set of states from which the current policy can reliably complete a task. Near its boundary, an agent may need only a few actions to reach a more solvable state, even when the goal remains distant. This reduces long-horizon learning to two coupled subproblems of reaching that state and completing the task from there. We address them with paired advantages and a progressive reverse-to-forward curriculum. The advantages provide separate signals for reaching more solvable states and completing the task from them, while the curriculum expands the capability region backward along successful reference trajectories and then into off-reference states through on-policy exploration. On ALFWorld and WebShop, our method outperforms competitive agentic RL methods at both model scales, achieving success rates of 90.1%/96.1% and 80.2%/82.6% with Qwen2.5-1.5B/7B-Instruct, respectively.
Introduction
Large language models (LLMs) [1, 22] (Grattafiori et al. 2024) are increasingly deployed as agents that interact with external environments on behalf of users [50, 42]. Moving beyond traditional single-turn generation, these agents operate through a stateful, sequential decision process where each action continuously adapts to evolving observations. Reinforcement learning (RL) has emerged as the dominant paradigm for training such interactive agents [54, 27, 43, 46], with group-based policy gradient methods, particularly Group Relative Policy Optimization (GRPO)[33], fast becoming the prevailing standard [9, 53]. However, scaling these algorithms to long-horizon agent tasks remains bottlenecked by reward sparsity [3, 18, 51]. All failed rollouts produce no learning signal at any step, and even successful ones deliver only a single terminal reward that has to be attributed across the entire trajectory. This sparse feedback makes credit assignment particularly challenging as LLM agents navigate vast action spaces and diverse tasks through partially observable environments with textual observations.
To address reward sparsity, a standard strategy is to reshape the training distribution through curriculum learning, which exposes the policy to simpler tasks before introducing progressively harder ones [5, 23]. Reverse curriculum strategies operationalize this principle by initializing the agent near the goal and incrementally pushing the starting state farther back [11]. Recent LLM agent work adopts a reference-anchored variant [52]. Instead of sampling start states by random walks, the start regresses backward along a successful reference trajectory, so the policy first masters local execution near the goal and is only later asked to complete the task from the original start [29, 38].
However, reverse curriculum leaves a structural credit assignment problem unresolved. Although initializing closer to the goal allows terminal feedback to be attributed more precisely to individual actions, moving the starting point farther back stretches the trajectory, forcing the same coarse feedback to be distributed over a longer sequence. Consequently, isolating early actions that genuinely drove the outcome remains difficult [4, 25]. A separate limitation is that reverse curriculum starts are anchored to reference trajectories, limiting exposure to the off-reference states encountered during the policy’s own exploration [30].
Our key insight is that moving the training start backward also expands the policy’s capability region toward the original start. As Figure 1 shows, later checkpoints in reverse curriculum achieve high success from states progressively farther from the goal along the reference trajectory. A newly introduced starting state near this boundary can still be far from the goal, yet only a few actions away from a state within the policy’s existing capability region. We exploit this structure by splitting credit assignment into two subproblems: (i) how to enter the capability region from a state just outside it, a short-horizon problem; and (ii) how to complete the task from the state reached inside it, a longer-horizon problem that earlier training has already partially addressed. The same decomposition can then be applied to states near the capability boundary encountered in the policy’s own rollouts, extending training beyond the reference trajectory.
Figure 1. The three curves show Pass@1 at checkpoints 20, 40, 60 in reverse curriculum training, as the training start moves backward along a successful reference trajectory from near the goal toward the original task start. Defining the capability region on this trajectory as starting positions with Pass@1 above 80%, we observe that this region also expands toward the original start. By ckp60, its new boundary reaches approximately 60% remaining steps.
We operationalize this decomposition through a Paired Advantage (PA) update. For each training group, we sample prefixes from a starting state near the capability boundary, group them by their endpoints, and sample multiple suffixes from each distinct endpoint. The left advantage compares estimated completion rates across prefix endpoints, crediting prefixes that reach states the policy can solve more reliably and thereby guiding it into the capability region. The right advantage compares suffix outcomes from the same endpoint to identify which continuations lead to task completion. The suffix outcomes thus provide both the endpoint estimates used for prefix credit and the comparisons used for suffix credit. We apply this update in two stages. The reverse stage establishes the capability region along successful reference trajectories, moving training starts backward from near the goal until the original starts become reliably solvable for most tasks. The forward stage then broadens the region beyond those trajectories by training on states reached through prefixes sampled by the policy from the original starts. Our contributions are threefold:
We introduce a policy-induced capability region view of reverse curriculum training. It reveals a short horizon problem of entering the region within a long horizon task.
We propose a paired advantage method with a reverse-to-forward curriculum. Using terminal rewards alone, it compares completion rates across prefix endpoints to guide entry into the capability region and compares suffix outcomes from the same endpoint to improve task completion.
Comprehensive experiments on ALFWorld[36] and WebShop [49] show that our method outperforms strong baselines and achieves a larger capability region under matched environment interaction budgets.
Preliminaries
Problem setup. We consider a distribution of tasks . For each task , an LLM agent interacts with an environment over multiple steps. At step , the agent receives observation and emits a textual action ; the next observation then arrives in response. A trajectory together with its terminal reward captures one full interaction, where if the task is completed successfully and otherwise. The agent’s behavior is governed by an LLM policy that conditions the interaction. We define a decision context at step , yielding an action according to . The learning objective is:
Group-based policy optimization. Group-based methods estimate relative advantages by comparing several rollouts of the same task. Given a task , a policy samples a group of trajectories , with terminal rewards . A GRPO-style update assigns each trajectory an advantage based on its reward relative to the group:
the policy update weights each trajectory’s log-probabilities by . The same trajectory-level signal is broadcast to every action inside the rollout.
Policy-induced capability region. Let denote the probability that policy completes the task when acting from decision context . Given a reliability threshold , we define the policy-induced capability region by:
this region contains the decision contexts from which the current policy completes the task with probability at least . The threshold layer around is the capability boundary. For a tolerance , the corresponding near-boundary set is:
where . Contexts in sit close to the current reliability threshold and are informative for expanding the capability region. The region and its boundary are induced by the current policy and need no geometric structure on the context space.
Methodology
This section develops a method that expands the capability region using terminal feedback alone. The method draws training contexts from near the capability boundary and separates the learning signal into two parts. One part pushes contexts from outside the region into it, the other sharpens completion from inside the region, as illustrated in Figure 2. We implement this expansion with a two-stage curriculum: the reverse stage uses successful reference trajectories to initialize the solvable region and expand it backward along trajectory depth, after which the forward stage broadens the capability region through natural rollouts.

Figure 2. Overview. Starting from a context , the policy samples prefixes and groups those that reach the same endpoint context , yielding the set of distinct endpoints . From each , it samples suffixes and observes their terminal rewards. The left advantage compares estimated completion rates across endpoints to credit prefixes for where they lead, while the right advantage compares suffix outcomes from the same endpoint to credit the actions taken after reaching it.
Paired advantage decomposition
For a single task, we sample a rollout group of trajectories from starting contexts near the capability boundary . Let denote the behavior policy used to collect the rollouts. Each rollout is divided into a prefix and a suffix. The prefix determines which decision context is reached, while the suffix determines whether the task can be completed from that context.
For the -th rollout, the prefix is , for , where is the starting decision context and is the prefix endpoint observation. The prefix is used to determine whether the sampled action sequence gets a context with a high probability of task completion.
The sampled prefixes are deduplicated by their endpoint context . Two prefixes land in the same class if their endpoint contexts coincide, and denotes the resulting set of unique endpoint contexts. Let be the indices of prefixes with endpoint , and let be the number of such prefixes. Although each LLM turn contains both a <think> segment and an executable <action> segment, we identify duplicate endpoints using only the sequence of <action> outputs.
For each endpoint context , the policy samples suffixes from , where is the same for all endpoint contexts. The -th suffix is denoted by , and let denote its outcome. The empirical task completion value at is estimated by:
We compare endpoint contexts using a prefix-weighted baseline and assign the left advantage:
Thus, measures whether prefixes reaching endpoint have a higher empirical task-completion value than the prefix-weighted average over all endpoints. The suffixes associated with the same endpoint context are compared separately. For suffix sampled from , define the right advantage by:
This signal evaluates a suffix relative to the other suffixes sampled from the same endpoint context. When all suffixes associated with have the same outcome, the right signal vanishes, while can still contribute through the left signal in (6).
The paired decomposition therefore separates two aspects of capability expansion. The left signal evaluates which prefix executions reach contexts with higher completion probabilities, whereas the right evaluates how the task should be completed after a particular context has been reached.
To realize this decomposition in the policy update, each prefix is completed jointly with a suffix sampled from its endpoint. For prefix , let denote its endpoint context and independently sample uniformly from . The selected suffix forms a completed trajectory , where denotes concatenation. With the estimated advantages held fixed, the policy optimization objective is:
where is the importance sampling ratio, is the clipping parameter, and is the number of actions in the completed trajectory . The expectation is over task sampling, rollout collection under , and suffix selection. The two terms apply clipped relative advantages to the prefix and suffix actions, respectively. The two-stage curriculum uses the same objective and differs in their starting contexts and prefix lengths used to construct these trajectories.
Two-stage curriculum: reverse then forward
The paired advantage update samples starting contexts from the near-boundary set . The curriculum constructs these contexts in two stages. The reverse stage expands backward along successful reference trajectories, while the forward stage extends it to contexts encountered in the policy’s own rollouts.
Reverse stage: expanding backward along references. The reverse stage operates on training tasks drawn from . For each task, we assume a successful reference trajectory . The reverse stage starts from a context near the end of , where the remaining effective horizon is short, and progressively moves toward the original task start.
Let denote the initial position of the reverse curriculum. At stage , we select the current near-boundary context , where and is the fixed backward stride along the reference trajectory. We sample prefixes of length from , group them by their endpoint contexts, and apply the paired advantage decomposition to the resulting groups.
The update continues until the estimated success probability at reaches the reliability threshold , after which is treated as belonging to . The curriculum then proceeds to the earlier reference context indexed by . The reverse stage terminates when the original task start is included in for most training tasks. This produces a policy that can reliably solve the training tasks from their original starts, while its capability region may still be concentrated around the reference trajectories.
Forward stage: extending into the policy’s own distribution. The forward stage uses the now-solvable original context as a rollout anchor and extends to contexts generated by the policy itself. At forward stage , prefixes are sampled from with length , initialized at to match the final reverse-stage horizon and then increased according to a monotone schedule, for example with . The resulting prefixes may reach contexts outside the reference trajectories, and these endpoint contexts are then trained with the paired advantage update and incorporated into the expanding capability region.
The two stages therefore use the same update but different prefix distributions. The reverse stage samples prefixes from successive contexts along successful references and moves the capability boundary toward earlier task contexts. The forward stage samples prefixes from policy-generated rollouts starting at and expands the boundary to contexts outside the support of the references.
Curriculum progression. In the reverse stage, successful reference trajectories provide an ordered sequence of candidate starting contexts. Once the current start reaches the reliability threshold , moving backward along the reference trajectory tends to place the next start near the expanding capability boundary, as illustrated in Figure 1. We therefore use these starts as practical near-boundary contexts without explicitly computing or specifying a numerical . In the forward stage, we monitor the mean number of distinct prefix endpoints per rollout group. When this mean stabilizes over successive updates, indicating that endpoint diversity at the current prefix length has plateaued, we advance the curriculum and increase the prefix length by . Increasing the prefix length lets the policy search for endpoint contexts that may lie across the capability boundary.
Experiments
Experimental Setup
Benchmarks. We evaluate on ALFWorld ([36]) and WebShop ([49]). ALFWorld comprises 3,827 embodied household tasks across six categories (PICK, LOOK, CLEAN, HEAT, COOL, and PICK2). Given a textual goal, the agent completes each task through multi-turn interaction with the environment. WebShop simulates online shopping through an HTML interface. in which the agent searches for products, navigates pages, and makes a purchase based on a user instruction. It contains over 1.1 million products and 12,000 user instructions.
| Type | Method | Pick | Look | Clean | Heat | Cool | Pick2 | All | Score | Succ. |
| Closed-Source Models | ||||||||||
| Prompting | GPT-4o | 75.3 | 60.8 | 31.2 | 56.7 | 21.6 | 49.8 | 48.0 | 31.8 | 23.7 |
| Prompting | Gemini-2.5-Pro | 92.8 | 63.3 | 62.1 | 69.0 | 26.6 | 58.7 | 60.3 | 42.5 | 35.9 |
| 1.5B | Prompting Qwen2.5 | 5.9 | 5.5 | 3.3 | 9.7 | 4.2 | 0.0 | 4.1 | 23.1 | 5.2 |
| Prompting ReAct | 17.4 | 20.5 | 15.7 | 6.2 | 7.7 | 2.0 | 12.8 | 40.1 | 11.3 | |
| Prompting Reflexion | 35.3 | 22.2 | 21.7 | 13.6 | 19.4 | 3.7 | 21.8 | 55.8 | 21.9 | |
| RL Training PPO | ||||||||||
| RL Training RLOO | ||||||||||
| RL Training GRPO | ||||||||||
| RL Training CRPS | 91.0 | 72.0 | 88.0 | 90.0 | 73.0 | 76.0 | 83.0 | 73.0 | 61.0 | |
| RL Training GiGPO | ||||||||||
| RL Training Ours | ||||||||||
| 7B | Prompting Qwen2.5 | 33.4 | 21.6 | 19.3 | 6.9 | 2.8 | 3.2 | 14.8 | 26.4 | 7.8 |
| Prompting ReAct | 48.5 | 35.4 | 34.3 | 13.2 | 18.2 | 17.6 | 31.2 | 46.2 | 19.5 | |
| Prompting Reflexion | 62.0 | 41.6 | 44.9 | 30.9 | 36.3 | 23.8 | 42.7 | 58.1 | 28.8 | |
| RL Training PPO | ||||||||||
| RL Training RLOO | ||||||||||
| RL Training GRPO | ||||||||||
| RL Training CRPS | 91.0 | 80.0 | 86.0 | 83.0 | 80.0 | 71.0 | 86.0 | 82.0 | 74.0 | |
| RL Training GiGPO | ||||||||||
| RL Training Ours |
Table 1. Performance on ALFWorld and WebShop. We report success rates (%) for ALFWorld and task scores and success rates (%) for WebShop. Subscripts denote standard deviations where available. For our method, they are computed over three evaluation seeds using a fixed checkpoint. Baseline results are taken from GiGPO [9] and CRPS [52]. Bold indicates the best mean in each column within a model scale.
Baselines. We compare against closed-source LLMs, prompting agents, and RL-trained agents as three families of baselines. Closed-source models are GPT-4o [1] and Gemini-2.5-Pro [40], representing strong general-purpose reasoning. Prompting agents include Qwen2.5-Instruct, ReAct [50] and Reflexion [35], relying on prompts without parameter updates. RL-based training methods cover PPO [32], RLOO [20, 2], and GRPO [33] under trajectory-level supervision, alongside methods that address long-horizon credit assignment through step-level grouping GiGPO [9] and reverse replay CRPS [52].
Implementation Details. We adopt Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct [28] as base models. All RL methods use the same backbone and training settings, with 16 tasks per update, 8 rollouts per task. Agents generate reasoning within <think>...</think> tags before emitting an action within <action>...</action> tags. Episodes are capped at 50 turns in ALFWorld and 15 in WebShop. Our method samples prefixes per task and 2 suffixes per unique endpoint. All training runs on 4 NVIDIA H100 GPUs. More training setups and hyperparameter details are provided in Appendix § 9.
Main Results
Table 1 reports the result of our method at both model scales, compared with prompting and representative reinforcement learning methods under the matched evaluation protocol. RL training improves substantially over prompting alone on both benchmarks, confirming the difficulty of learning long-horizon interaction from prompts alone. On ALFWorld, our method reaches 90.1% overall success with Qwen2.5-1.5B-Instruct and 96.1% with Qwen2.5-7B-Instruct. These results exceed GRPO baseline by 17.3 and 18.5 percentage points, respectively, and the competitive GiGPO by 3.4 and 5.3 points. It leads in five of the six task categories at each scale. The exceptions are Clean at 1.5B (84.2% versus GiGPO’s 94.8%) and Cool at 7B (85.9% versus 89.3%). The overall gains over GRPO and GiGPO indicate that the method remains effective across tasks with different interaction requirements, rather than relying on improvements in a single category.
On WebShop, our method achieves a score of 88.6 and a success rate of 80.2% at 1.5B, compared with 83.1 and 65.0% for GiGPO. At 7B, it reaches 92.5 and 82.6%. The score exceeds GiGPO by 8.1 points, while the success rate exceeds CRPS by 8.6 percentage points. Our method leads on both WebShop metrics at both scales. The larger gains in success rate than in task score are consistent with more agents completing the full sequence of search, navigation, and product-selection decisions. The ablation results below examine the contribution of paired credit assignment to these gains.
Analysis
PA decomposition expands capability region. Figure 3a analyzes the prefix endpoints reached during ALFWorld training with Qwen2.5-7B-Instruct. The bars show the mean number of distinct endpoint contexts per training step, grouped by value , and the curves show the mean number of observed suffixes per endpoint. PA has 0.1 fewer endpoints per step than GRPO in the first bin but 23.8 more in the final bin, reaching 93.8 versus 70.0 endpoints per step, including 68.0 versus 55.5 with . Meanwhile, PA maintains a fixed budget of suffixes per endpoint by design (Eq. (5)), in contrast to GRPO’s observed increase from 2.1 to 3.7. This pattern is consistent with a winner-takes-all effect under GRPO: trajectory-level advantages reinforce prefixes associated with successful rollouts, so sampled prefixes may converge on fewer endpoint contexts. PA instead evaluates each distinct endpoint with two suffixes and separates prefix selection from continuation quality, keeping more potentially solvable endpoints available for learning.
Figure 3. Prefix endpoint coverage during ALFWorld training with Qwen2.5-7B-Instruct. In two panels, the left bar of each pair represents PA and the right bar represents GRPO. Both methods are initialized from the same model checkpoint and trained on the same dataset. PA samples exactly two suffixes from each unique endpoint, whereas GRPO observes one continuation for each prefix reaching that endpoint, so its suffix count is determined by . (a) Bars show the mean number of distinct prefix endpoints per training step, grouped by value . Curves show the mean number of suffixes per endpoint. (b) Endpoint counts restricted to those reached by exactly one prefix, . PA’s two suffixes can reveal mixed outcomes at these singleton endpoints.
Figure 3b offers a clearer view of this divergence by isolating endpoints reached by exactly one prefix, . For these singleton endpoints, GRPO observes one binary outcome and cannot assign an intermediate empirical value, PA’s two suffixes produce when one succeeds and the other fails, enabling the within-endpoint comparison in its right advantage (Eq. (7)). From the first to the final bin, PA’s singleton coverage falls from about 70 to 48.7 endpoints per step, yet its value-1 count rises from about 25 to 32.2, while its value-0.5 count falls from about 13 to 8.1. GRPO loses both singleton coverage (about 72 to 24.6) and value-1 endpoints (about 34 to 17.9). These contrasting trends are consistent with PA’s within-endpoint comparisons helping partially solvable contexts become more reliably solvable, while GRPO’s observed endpoint distribution contracts in both breadth and high-value count.
Why reverse, then forward? Figure 4 reports the distribution of , the number of unique prefix endpoint contexts reached per training group, across four consecutive training stages on WebShop. The reverse stage drives the mean from 4.08 down to 2.21 with the fraction reaching 46% by late reverse, evidence that the policy has collapsed most rollouts onto a single reachable endpoint per group. Through this collapse, the reverse stage establishes the initial capability region, with tight concentration on the reference trajectory yielding a small but reliable solvable region. The forward stage takes this region as its anchor and pushes the mean back to 6.68 in early forward, with 66% of groups at , before settling at 4.92 in late forward. Read across the four stages, the capability region is first anchored then expanded. The reverse stage establishes it along the reference trajectory, and the forward stage continues expansion into the off-reference states the policy now visits.

Figure 4. Distribution of the number of unique prefix endpoint contexts reached per training on Webshop.
Ablation Study
We conduct an ablation study to evaluate the effectiveness of each component in our method. As illustrated in Table 2, the two curriculum stages and paired advantages both contribute to the performance on WebShop using Qwen2.5-1.5B-Instruct. The reverse-only configuration baseline without paired advantages achieves a success rate of 68.8%. Adding the forward stage increases performance to 74.7%. Incorporating paired advantages into the reverse stage further improves the success rate to 76.6%, and in both curriculum stages achieves the best performance of 80.20%, which is 11.4 points higher than the reverse-only baseline. The gains from applying PA in both stages support the benefit of separately crediting prefix actions for the endpoints they reach and suffix actions for their outcomes from those endpoints.
| Curriculum | Rev. PA | Fwd. PA | Succ. (%) |
| Reverse only | – | – | 68.8 |
| Reverse + Forward | – | – | 74.7 |
| Reverse + Forward | ✓ | – | 76.6 |
| Reverse + Forward | ✓ | ✓ | 80.2 |
Table 2. Ablation study on WebShop using Qwen2.5-1.5B-Instruct.
Training Efficiency and Cost Analysis
fig:5a shows that PA achieves higher validation success at matched environment-interaction budgets, despite reaching about 5.9 distinct prefix endpoints per group in Figure 4 and sampling two suffixes from each non-terminal endpoint. Figure 5b helps explain why the additional branching does not necessarily increase interaction cost. Reverse training starts near the ends of successful references, keeping the remaining rollouts short. By late reverse, eight sampled prefixes reach only 2.21 distinct endpoints per group on average, limiting the number of endpoints that require suffix sampling. In forward training, endpoint diversity increases as PA explores contexts beyond the references. The reverse-trained policy helps avoid long, unproductive continuations from these endpoints, allowing PA to explore more contexts without a proportional increase in environment steps. The occasional cost spikes are consistent with batches reaching more endpoints, but PA generally uses fewer environment steps per training step than the baselines.
Figure 5. (a) Validation success against cumulative model-generated environment steps. (b) Environment steps consumed per training step for each arm.
Related Work
Reinforcement Learning for Long-Horizon LLM Agents. LLM agents extend language models to long-horizon sequential decisions through repeated interaction with environments. The paradigm has evolved from static text generation, through single-turn tool use [31], to multi-turn sequential decision making, exemplified by ReAct and Reflexion [50, 35]. In LLM training, the RLHF pipeline paired PPO with a learned value function [24, 32], while RLOO and GRPO removed the need for a separate critic [2, 33], and AGILE and WebRL subsequently applied reinforcement learning to agent settings [10, 27]. Recent work has brought online RL to long-horizon agents through dedicated multi-turn environments and scalable training systems such as RAGEN, AgentGym-RL, and SkyRL-Agent [43, 46, 6]. These efforts extend RL training to complete agent episodes across search, software engineering, and mobile GUI tasks [19, 6, 48].
Credit Assignment. In sparse reward settings, long-horizon agents typically receive supervision only from a terminal outcome, which makes credit assignment over intermediate decisions difficult. Process reward models and search-based approaches introduce intermediate signals, but require additional manual annotations and expensive search computation, respectively [21, 14]. Later agentic RL methods such as GiGPO and HGPO derive finer grained comparisons from repeated states across rollouts without externally labeled process rewards [9, 15], whereas STAPO derives a selective step-level signal via normalized entropy [26]. More recent approaches modify the rollout structure itself by aggregating trajectories into transition graphs or introducing structured branching and information guided exploration, as in GraphGPO, SPARK, Tree GRPO, and AEPO [7, 45, 17, 8]. E3-TIR takes a complementary approach by reusing a common continuation to compare prefixes, allowing downstream outcomes to provide evidence about prefix quality [13]. Taken together, these studies suggest that long-horizon credit assignment depends on how rollout structure determines which decisions share, inherit, or receive separately estimated learning signals.
Curriculum Learning. Curriculum learning structures the training distribution so that competence acquired on simpler subproblems supports learning on more difficult ones [5]. In reinforcement learning and robot control, this principle has been implemented by adapting restart states or task sequences around near-goal and demonstrated states [39, 11, 29, 16, 44, 34]. In agentic RL, curricula modify task distribution, the available supervision, or the interaction environment to regulate the difficulty of multi-turn interaction [37, 41, 47]. Recent work derives curricula directly from interaction outcomes, using failed attempts to generate new tasks and successful trajectories to identify informative restart states [27, 52]. Our method builds on curriculum learning by first expanding the training start backward along successful reference trajectories and then broadening training to states generated by the policy itself.
Conclusion
We train long-horizon agents from sparse terminal rewards by progressively expanding the policy’s capability region. Paired Advantage separately credits prefixes for reaching more solvable contexts and suffixes for task completion with a reverse-to-forward curriculum moves from successful references to policy-generated contexts. The method outperforms strong baselines on ALFWorld and WebShop at both model scales.
Limitations. Reverse training requires successful references, and suffix sampling requires restarting from intermediate contexts. These requirements restrict use when references or reliable resets are unavailable. If all sampled suffixes fail, both advantages vanish. Starting near known solutions reduces this risk, but expansion may stall when solvable contexts remain beyond the reach of sampled prefixes.
AI USE STATEMENT
In this work, we used generative AI tools to assist with code implementation and to improve the grammar, clarity, and readability of the manuscript. We have not used generative AI tools for generating research proposals, methodological designs, experimental plans, or scientific conclusions. We have reviewed and tested all AI-assisted code, and all reported results come from running the verified implementation by 2 authors. We also reviewed all AI-assisted writing to ensure it accurately reflects the methods, results, and claims of this work. We take full responsibility for the final content of this paper, including all text, claims, code, and other artifacts produced with the assistance of generative AI.
Algorithm
Algorithm 1 summarizes the training procedure. Endpoint grouping depends on the full decision context, which includes both the task and the interaction history. As a result, the same observation does not by itself guarantee the same endpoint. For each distinct endpoint, we sample suffixes once and allow every prefix reaching that endpoint to reuse them. Each iteration thus samples suffixes instead of . In the left advantage, the factor preserves the number of prefixes reaching each endpoint in the group baseline. Deduplication then reduces suffix sampling without changing how prefixes are weighted. The reverse and forward stages share the same paired advantage update. The transition between the two stages is determined by the estimated completion reliability at the original task starts.
Algorithm 1 Training LLM agents with Depth Before Breadth
Require: Policy , tasks , successful references , group size , suffix count , reverse stride , threshold , forward horizons
Ensure: Updated policy
Initialize the curriculum: reverse;
for each training iteration do
Update the old policy model and sample task
// Select the curriculum anchor
if reverse then
Move along the reference: ;
else
Start from the original context:
end if
// Sample prefixes and group shared endpoints
for do
Sample a prefix: ;
end for
Group endpoints: ; ;
// Sample suffixes and compute paired advantages
for each do
Sample continuations: from under
Estimate endpoint value by Eq. (5)
Compute suffix advantages by Eq. (7)
end for
Compute prefix advantages by Eq. (6)
for do
Select a suffix:
Stitch the trajectory:
Assign to and to
end for
// Update the policy and curriculum
Update by maximizing in Eq. (8)
if reverse and then
if the original starts pass the threshold for most training tasks then
Switch to forward: forward;
else
Move the start backward:
end if
else if forward and the schedule advances then
Increase the prefix horizon:
end if
end for
Implementation Details
Details of Training
Hyperparameters for ALFWorld. The actor is optimized for one epoch per update using Adam with a constant learning rate of and gradient clipping at 1.0. Training rollouts use temperature 1.0 and top- 0.95. The terminal reward is binary and is scaled by 10 for training. The reverse curriculum stride and forward curriculum stride are both 3, and the reliability threshold is 0.6.
Hyperparameters for WebShop. WebShop shares ALFWorld’s training batch size of 32, eight samples per prompt, learning rate, rollout sampling settings, and terminal reward scale. Its maximum prompt and response lengths are 2,048 and 13,312 tokens, respectively. Both curriculum strides are 3, and the reliability threshold is 0.78. Table 3 lists the full configuration.
| Config | ALFWorld | WebShop |
train_batch_size | 32 | 32 |
n_samples_per_prompt | 8 | 8 |
ppo_mini_batch_size | 16 | 16 |
ppo_micro_batch_size_per_gpu | 2 | 1 |
ppo_max_token_len_per_gpu | 12288 | 15360 |
log_prob_max_token_len_per_gpu | 12288 | 15360 |
log_prob_micro_batch_size_per_gpu | 2 | 1 |
use_dynamic_bsz | False | False |
ulysses_sequence_parallel_size | 1 | 1 |
tensor_model_parallel_size | 1 | 1 |
temperature (rollout) | 1.0 | 1.0 |
top-p (rollout) | 0.95 | 0.95 |
temperature (val) | 0 | 0 |
top-p (val) | 1.0 | 1.0 |
actor_learning_rate | ||
| / weight_decay | 0.9, 0.999 / 0.01 | 0.9, 0.999 / 0.01 |
grad_clip | 1.0 | 1.0 |
max_epochs | 1 | 1 |
lr_scheduler | constant | constant |
max_prompt_length | 2048 | 2048 |
max_response_length | 10240 | 13312 |
max_model_len | 12288 | 15360 |
multi_turn_max_turns | 50 | 15 |
action_budget | 120 | 80 |
clip_ratio | 0.2 | 0.2 |
clip_ratio_c (dual-clip) | 3.0 | 3.0 |
use_kl_loss / kl_coef | False / 0 | False / 0 |
use_kl_in_reward | False | False |
entropy_coeff | 0 | 0 |
loss_agg_mode | seq-mean-token-mean | |
norm_adv_by_std_in_grpo | True | True |
| Reverse curriculum stride () | 3 | 3 |
| Forward curriculum stride () | 3 | 3 |
| Reliability threshold () | 0.6 | 0.78 |
| Reward type | Binary | Binary |
| Reward scale | 10 | 10 |
Table 3. Hyperparameters of ALFWorld and WebShop.
Computing Details. For ALFWorld and WebShop, both Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct models are trained on NVIDIA H100 GPUs. ALFWorld is trained for 100 iterations and is evaluated on the seen task categories and instance variants, containing 140 tasks. WebShop is trained for 150 iterations and is evaluated on the test split of 500 tasks. For both environments, we sample 128 tasks at random per run and average results over 3 matched random seeds.
Prompts
Figures 6 and 7 show the prompt templates for WebShop and ALFWorld. At the start of an episode, the system message establishes the agent’s task and output format. The {task_description} and {current_observation} denote information supplied by the environment. WebShop provides the shopping instruction and current page, while ALFWorld also provides the canonical TextWorld goal. Each environment message supplies the actions currently available and the remaining action count.
Vector (PDF)Prompt Template for WebShop
# System message
You are an autonomous agent operating the WebShop text environment.
Your goal is to find, configure, and buy the product that best satisfies the shopping instruction.
Every assistant turn is already prefixed with <think>. Continue with the reasoning, close it with </think>, then output exactly one listed action inside <action> ... </action>.
Never invent a click target that is not listed for the current page.
# User message at each turn
Shopping instruction: {task_description}
Current page: {current_observation}
Available actions:
- {action_1}
- {action_2}
- ...
Actions remaining: {max_steps - step}Figure 6. Prompt template used for the WebShop agent.
The agent then interacts within a continuing conversation. After each action, the resulting observation is added to the interaction history and the agent generates its next response from the updated context. The prompt therefore supports a structured trajectory of observations and actions rather than a separate, complete prompt for each decision. In both environments, the <think>...</think> block contains the agent’s reasoning, followed by exactly one listed action in the <action>...</action> block. WebShop supplies the opening <think> tag as an assistant prefix, whereas ALFWorld requires the model to generate both complete blocks.
Figure 7. Prompt template used for the ALFWorld agent.
Case Study
WebShop

Figure 8. Illustration of a WebShop shopping task through multi-turn interaction.
Figure 8 traces a WebShop shopping task from search to purchase. The prefix issues a search based on the user’s constraints and reaches a results page with a candidate product. The suffix opens its product page, selects the requested size and color, and completes the purchase. The example shows how a search action determines the context from which the remaining decisions are made.
ALFWorld
Figure 9 follows an agent tasked with placing a chilled apple on a counter. In the prefix, it locates and picks up the apple before reaching the fridge. The suffix cools the apple and returns it to the countertop. The recorded suffix also includes one turn in which the agent produces reasoning but takes no environment action.

Figure 9. Illustration of an ALFWorld household task through multi-turn interaction.
References
- [1]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.arxiv.org/abs/2303.08774
- [2]Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforcement-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248–12267, 2024.arxiv.org/abs/2402.14740
- [3]Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1707.01495
- [4]Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems, 32, 2019.arxiv.org/abs/1806.07857
- [5]Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41–48, 2009.DOI
- [6]Shiyi Cao, Dacheng Li, Fangzhou Zhao, Shuo Yuan, Sumanth R Hedge, Connor Chen, Charlie Ruan, Tyler Griggs, Shu Liu, Eric Tang, et al. Skyrl-agent: Efficient rl training for multi-turn llm agent. arXiv preprint arXiv:2511.16108, 2025.arxiv.org/abs/2511.16108
- [7]Xin Cheng, Shuo He, Lang Feng, HaiYang Xu, Ming Yan, Lei Feng, and Bo An. Beyond trajectory-level attribution: Graph-based credit assignment for agentic reinforcement learning. arXiv preprint arXiv:2605.26684, 2026.arxiv.org/abs/2605.26684
- [8]Guanting Dong, Licheng Bao, Zhongyuan Wang, Kangzhi Zhao, Xiaoxi Li, Jiajie Jin, Jinghan Yang, Hangyu Mao, Fuzheng Zhang, Kun Gai, et al. Agentic entropy-balanced policy optimization. arXiv preprint arXiv:2510.14545, 2025.arxiv.org/abs/2510.14545
- [9]Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for LLM agent training. In Advances in Neural Information Processing Systems, volume 38, 2025.arxiv.org/abs/2505.10978
- [10]Peiyuan Feng, Yichen He, Guanhua Huang, Yuan Lin, Hanchong Zhang, Yuchen Zhang, and Hang Li. Agile: A novel reinforcement learning framework of llm agents. volume 37, pp. 5244–5284, 2024.arxiv.org/abs/2405.14751
- [11]Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Proceedings of the 1st Annual Conference on Robot Learning, volume 78, pp. 482–495. PMLR, 2017.arxiv.org/abs/1707.05300
- [13]Weiyang Guo, Zesheng Shi, Liye Zhao, Jiayuan Ma, Zeen Zhu, Junxian He, Min Zhang, and Jing Li. E3-TIR: Enhanced experience exploitation for tool-integrated reasoning. 2026.
- [14]Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 8154–8173, 2023.arxiv.org/abs/2305.14992
- [15]Shuo He, Lang Feng, Xin Cheng, Lei Feng, Bo An, et al. Hierarchy-of-groups policy optimization for long-horizon agentic tasks. In International Conference on Learning Representations, volume 2026, pp. 27572–27593, 2026.arxiv.org/abs/2602.22817
- [16]Boris Ivanovic, James Harrison, Apoorva Sharma, Mo Chen, and Marco Pavone. Barc: Backward reachability curriculum for robotic reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 15–21. IEEE, 2019.arxiv.org/abs/1806.06161
- [17]Yuxiang Ji, Ziyu Ma, Yong Wang, Guanhua Chen, Xiangxiang Chu, and Liaoni Wu. Tree search for llm agent reinforcement learning. 2026:87362–87388, 2026.arxiv.org/abs/2509.21240
- [18]Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025.arxiv.org/abs/2503.09516
- [19]Vivek Kalyan and Martin Andrews. Reinforcement learning for long-horizon multi-turn search agents. arXiv preprint arXiv:2510.24126, 2025.arxiv.org/abs/2510.24126
- [20]Wouter Kool, Herke Van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! 2019.
- [21]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. 2024:39578–39601, 2024.arxiv.org/abs/2305.20050
- [22]Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.arxiv.org/abs/2412.19437
- [23]Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1–50, 2020.arxiv.org/abs/2003.04960
- [24]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. volume 35, pp. 27730–27744, 2022.arxiv.org/abs/2203.02155
- [25]Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning. arXiv preprint arXiv:2312.01072, 2023.arxiv.org/abs/2312.01072
- [26]Qiuyi Qi, Tian Liang, Mutian Bao, Jinjiang Zhang, Dongnan Liu, Wei Zhou, Linjian Mo, Ming Kong, Jie Liu, Feng Zhang, et al. Stapo: Selective trajectory-aware policy optimization for llm agent training. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 28371–28392, 2026.
- [27]Zehan Qi, Xiao Liu, Iat Long Long, Hanyu Lai, Xueqiao Sun, Jiadai Sun, Xinyue Yang, Yu Yang, Shuntian Yao, Wei Xu, et al. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. In International Conference on Learning Representations, volume 2025, pp. 79791–79821, 2025.arxiv.org/abs/2411.02337
- [28]Qwen, ., An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianghong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. 2025.arxiv.org/abs/2412.15115
- [29]Cinjion Resnick, Roberta Raileanu, Sanyam Kapoor, Alexander Peysakhovich, Kyunghyun Cho, and Joan Bruna. Backplay: “man muss immer umkehren”. arXiv preprint arXiv:1807.06919, 2018.arxiv.org/abs/1807.06919
- [30]Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627–635. JMLR Workshop and Conference Proceedings, 2011.arxiv.org/abs/1011.0686
- [31]Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. volume 36, pp. 68539–68551, 2023.arxiv.org/abs/2302.04761
- [32]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
- [33]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and Dayao Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [34]Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Autonomous reinforcement learning via subgoal curricula. In Advances in Neural Information Processing Systems, volume 34, pp. 18474–18486, 2021.arxiv.org/abs/2107.12931
- [35]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. volume 36, pp. 8634–8652, 2023.arxiv.org/abs/2303.11366
- [36]Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.arxiv.org/abs/2010.03768
- [37]Weiting Tan, Xinghua Qu, Ming Tu, Meng Ge, Andy T. Liu, Philipp Koehn, and Lu Lu. Process-supervised reinforcement learning for interactive multimodal tool-use agents. arXiv preprint arXiv:2509.14480, 2025.arxiv.org/abs/2509.14480
- [38]Stone Tao, Arth Shukla, Tse-kai Chan, and Hao Su. Reverse forward curriculum learning for extreme sample and demo efficiency. In International Conference on Learning Representations, volume 2024, pp. 46955–46976, 2024.arxiv.org/abs/2405.03379
- [39]Arash Tavakoli, Vitaly Levdik, Riashat Islam, Christopher M. Smith, and Petar Kormushev. Exploring restart distributions. In Proceedings of the 4th Multidisciplinary Conference on Reinforcement Learning and Decision Making, 2019.arxiv.org/abs/1811.11298
- [40]Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
- [41]Georgios Tzannetos, Parameswaran Kamalaruban, and Adish Singla. Curriculum design for trajectory-constrained agent: Compressing chain-of-thought tokens in llms. volume 38, pp. 19881–19907, 2026.arxiv.org/abs/2511.02690
- [42]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.arxiv.org/abs/2305.16291
- [43]Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025.arxiv.org/abs/2504.20073
- [44]Jan Wöhlke, Felix Schmitt, and Herke van Hoof. A performance-based start state curriculum framework for reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1503–1511, 2020.DOI
- [45]Jinyang Wu, Shuo Yang, Yuhao Shen, Shuai Zhang, Zhengqi Wen, and Jianhua Tao. Spark: Strategic policy-aware exploration via dynamic branching for long-horizon agentic learning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23981–24004, 2026.
- [46]Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, et al. Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning. arXiv preprint arXiv:2509.08755, 2025.arxiv.org/abs/2509.08755
- [47]Peng Xia, Kaide Zeng, Jiaqi Liu, Can Qin, Fang Wu, Yiyang Zhou, Caiming Xiong, and Huaxiu Yao. Agent0: Unleashing self-evolving agents from zero data via tool-integrated reasoning. arXiv preprint arXiv:2511.16043, 2025.arxiv.org/abs/2511.16043
- [48]Yifan Xu, Xiao Liu, Xinghan Liu, Jiaqi Fu, Jiayu Huang, Hanchen Zhang, Bohao Jing, Shudan Zhang, Yuting Wang, Yuxiao Dong, et al. Mobilerll: Online agentic reinforcement learning for mobile gui agents. In International Conference on Learning Representations, volume 2026, pp. 35282–35315, 2026.arxiv.org/abs/2509.18119
- [49]Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. volume 35, pp. 20744–20757, 2022.arxiv.org/abs/2207.01206
- [50]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.03629
- [51]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. DAP: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026.arxiv.org/abs/2503.14476
- [52]Zijing Zhang and Xiajie Yang. CRPS: Curriculum replay via progressive suffixes from successful trajectories for long-horizon LLM agents. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 13891–13904, 2026.DOI
- [53]Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization, 2025. URL https://arxiv.org/abs/2507.18071.
- [54]Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446, 2024.arxiv.org/abs/2402.19446



