betaFeedback

WorksComputer Science

AttnSR: Attention Step Reward for Long-Horizon Agentic RL

Abstract

Long-horizon LLM agents make multi-step decisions, yet outcome-reward-only reinforcement learning provides little direct supervision for individual steps. We introduce AttnSR, a step-level reward augmentation method that derives reward allocations from policy attention extracted during training replay, without a separately trained reward model or additional rollouts. AttnSR traces backward through the policy model attention graph to estimate the relevance of earlier steps to the final response. The terminal step, measured relative to a running baseline, determines whether these rewards reinforce or penalize the selected steps. In a controlled Qwen2.5-7B-Instruct study, AttnSR consistently outperforms the PPO baseline across all evaluated checkpoints on ALFWorld, raising final held-out success from 91.2% to 97.1%. The ablation experiment shows that the benefit depends on how intermediate rewards are assigned, rather than simple accumulation of step rewards over time. AttnSR also improves ScienceWorld at every checkpoint while cutting environment interactions by 29.9%. These results support policy attention as a practical prior for step-level reward allocation in long-horizon agentic RL. Code is available at https://github.com/fairyshine/AttnSR .

Introduction

Reinforcement learning (RL) with verifiable rewards [21, 4] is increasingly applied to agents that interleave reasoning and tool use [16, 30, 34]. In environments such as ALFWORLD, WEBSHOP, and SCIENCEWORLD, agents make sequences of decisions to complete a task [24, 33, 29]. Vanilla PPO uses the outcome reward to estimate value targets and policy advantages [19, 18], but says little about how intermediate actions contribute. Assigning credit to earlier decisions thus remains a core challenge in long-horizon RL [36].

Step-level rewards address this challenge by providing more localized training signals, but obtaining them can require step annotations, additional environment interactions, or a separately trained evaluator [13, 20, 11]. This motivates using signals already available within the policy model to guide intermediate rewards, without incurring the substantial training and environment-interaction costs.

Attention patterns in the policy model may provide such a signal, since they record how later decisions attend to earlier actions and observations. We hypothesize that these relationships provide a useful prior for allocating intermediate rewards, although attention alone does not establish causal contributions [10, 1]. The signal can be extracted during the existing forward pass of the policy model over collected trajectories. The method adds little computation and requires neither a separate attribution backward pass nor a learned progress estimator.

Building on this hypothesis, we introduce Attention Step Reward (ATTNSR), which uses the attention of the policy model to identify relevant earlier steps and turns the task outcome into intermediate training signals. Starting from the final step, it traces attention links backward to estimate the relevance of earlier actions and observations. These estimates determine which intermediate steps receive reward, while weaker or less consistent attention reduces their overall magnitude. Outcomes above the running average score yield positive intermediate rewards, and those below it yield negative rewards. These signals can in principle be used directly by step-level RL algorithms or placed at assistant-step endpoints for token-level RL. We study the latter integration in PPO, where the added rewards change value targets and policy advantages while retaining the critic architecture and losses.

We evaluate ATTNSR on ALFWorld, ScienceWorld, and WebShop using Qwen2.5-7B-Instruct with matched token-level PPO training. ALFWorld success rises from 91.2% to 97.1%, with larger gains on unseen tasks, whereas the positional ablation reaches 92.3%. On ScienceWorld, ATTNSR yields higher official scores at all four reported checkpoints and 29.9% fewer mean environment steps. These results suggest that policy attention can serve as a useful signal for intermediate reward allocation, offering a way to enrich outcome-based supervision without additional rollouts or a separately trained reward model.

Our contributions are summarized below.

  • We hypothesize that policy attention provides a relevance prior over interaction steps by indicating how later decisions attend to earlier actions and observations. Based on this hypothesis, we use attention-derived dependencies to distribute outcome information across steps for temporal credit assignment, treating attention as a heuristic relevance signal, not evidence of causal contribution.

  • We introduce ATTNSR, a step-level reward augmentation scheme that combines attention-based allocation, confidence gating, and outcome centering. The signals can be used directly as step-level rewards or aligned with assistant-step endpoints for token-level objectives, and we study the latter integration with PPO. It needs neither counterfactual rollouts nor a separately trained reward model.

  • We evaluate ATTNSR across three agentic benchmarks with ablation and trajectory analysis to characterize its performance gains and potential boundary conditions across tasks with different interaction lengths. The results are consistent with a stronger benefit on complex, long-horizon tasks, while suggesting more limited gains when trajectories are short.

Problem Formulation

We model each LLM-based agent interaction trajectory as a multi-turn conversation. Episode ii starts from a task query and contains KiK_i decision steps. Each Si,kS_{i,k} comprises an assistant message Ai,kA_{i,k} and its environment observation Oi,kO_{i,k}, when present, and the final message may have no observation. Let Ii,k\mathcal{I}_{i,k} be the assistant token indices at step kk, and ei,k∈Ii,ke_{i,k} \in\mathcal{I}_{i,k} its last trainable token, the step endpoint. We supply PPO with a task outcome reward Ri∈[0,1]R_i \in[0,1]. In the canonical runs, RiR_i is binary success for ALFWorld, the normalized official score for WebShop, and the normalized final score for ScienceWorld. Format-error trajectories receive Ri=0R_i = 0. Thus the binary success metric for WebShop and ScienceWorld is an evaluation threshold on the official score, not a replacement for the training reward. Table 16 in Appendix H summarizes the notation.

SymbolMeaning
ii, ttepisode index; collected training-batch index
KiK_inumber of assistant steps in episode ii
kk, jjstep indices, 1≤k<j≤Ki1 \le k < j \le K_i
Si,kS_{i,k}interaction step: assistant message and observation, when present
Ai,kA_{i,k}, Oi,kO_{i,k}assistant message; associated environment observation
Ii,kI_{i,k}token indices in the assistant span of step kk
uutoken index into the serialized trajectory
ei,ke_{i,k}endpoint: last trainable token of step kk
ri,ur_{i,u}reward written at token uu
RiR_i, Rˉt\bar{R}_toutcome reward; EMA updated using batch tt before shaping
MtM_tmean outcome reward in batch tt; initializes Rˉ1=M1\bar{R}_1 = M_1
ll, L\mathcal{L}, LLlayer index; selected layers; L=∣L∣=6L = |\mathcal{L}| = 6
Gi,AA(l)G^{(l)}_{i,\mathrm{AA}}, Gi,AO(l)G^{(l)}_{i,\mathrm{AO}}step attention to historical assistant tokens; to paired observations
Gi(l)G^{(l)}_imixed step attention, 0.5Gi,AA(l)+0.5Gi,AO(l)0.5G^{(l)}_{i,\mathrm{AA}} + 0.5G^{(l)}_{i,\mathrm{AO}}
ai,k(l)a^{(l)}_{i,k}backtracked relevance of step kk at layer ll, defined in Eq. (3)
Δi,k\Delta_{i,k}, di,kd_{i,k}steps from kk to the terminus, Ki−kK_i - k; proximity feature exp⁡(−(Δi,k−1)/κ)\exp(-(\Delta_{i,k} - 1)/\kappa)
pi,k(l)p^{(l)}_{i,k}, pi,kp_{i,k}per-layer allocation; geometric-consensus allocation, defined in Eq. (5)
mi(l)m^{(l)}_i, Hi(l)H^{(l)}_iterminal-step covered mass; normalized entropy of pi(l)p^{(l)}_i
CiC_iconfidence gate, Ci∈[0,1]C_i \in[0, 1]
BiB_i, si,ks_{i,k}signed shaping budget; step reward Bipi,kB_i p_{i,k}, defined in Eq. (13)
A^u\hat{A}_u, AueA^e_u, yuy_uunnormalized advantage; normalized actor advantage; critic target
VuoldV^{\mathrm{old}}_u, VϕV_\phipre-update critic value; critic network
sg, norm, TVstop-gradient; standardization; total-variation distance
ρ=0.85\rho= 0.85hop decay in the backtracking recurrence
η=−0.25\eta= -0.25, κ=4\kappa= 4recency-correction weight and proximity decay
T=1T = 1, ϵ=10−12\epsilon= 10^{-12}allocation softmax temperature; lower bound before logarithms
β=0.25\beta= 0.25budget scale, bounding ∣Bi∣|B_i|

Table 16. Notation. Hyperparameter values refer to the canonical AttnSR runs.

Our terminal-only PPO baseline aligns rewards with trajectory tokens, indexed by uu, placing the outcome at the final assistant endpoint as

ri,uPPO=RiI[u=ei,Ki],(1)r^{\mathrm{PPO}}_{i,u}=R_i\mathbb{I}[u=e_{i,K_i}], \tag*{(1)}

where I[⋅]\mathbb{I}[\cdot] is the indicator function. Observation tokens provide context but are masked from actor and critic losses. TD residuals and GAE compute value-function targets and policy advantages over valid assistant tokens, skipping masked positions (Schmalen et al., 2016; 2017). The outcome reward evaluates the whole trajectory without specifying contributions of individual actions.

Let si,ks_{i,k} denote a signed shaping reward assigned to intermediate step k<Kik<K_i. ATTNSR augments the baseline reward sequence as

ri,uAttnSR=ri,uPPO+∑k=1Ki−1si,kI[u=ei,k].(2)r^{\mathrm{AttnSR}}_{i,u}=r^{\mathrm{PPO}}_{i,u}+\sum_{k=1}^{K_i-1}s_{i,k}\mathbb{I}[u=e_{i,k}]. \tag*{(2)}

This retains RiR_i at the final endpoint and adds si,ks_{i,k} at intermediate assistant endpoints. Algorithms that consume step-level rewards can use these signals directly in principle, and Equation (2) specifies their token-level placement for the PPO implementation studied here. Our goal is to derive si,ks_{i,k} from actor attention and terminal outcomes. Section 3 defines its allocation, confidence gate, and sign.

AttnSR: Actor Attention for Step Credit

ATTNSR constructs intermediate rewards by tracing how later decisions attend to earlier interaction steps. It uses these relationships to allocate rewards, with their magnitude and sign determined by the reliability of the attention signal and the trajectory outcome relative to a running baseline.

Where to assign credit: a step dependency graph

For each selected transformer layer ll, we construct a directed graph over interaction steps. Its components Gi,AA(l)[j,k]G_{i,\mathrm{AA}}^{(l)}[j,k] and Gi,AO(l)[j,k]G_{i,\mathrm{AO}}^{(l)}[j,k] measure attention from the assistant message at step jj to the assistant and observation messages at step kk, respectively. We sum causal attention probabilities over each historical message key tokens, then average over step jj query tokens and heads. Only k<jk<j edges are retained, and missing observations contribute zero AO mass. The combined graph is

Gi(l)=0.5Gi,AA(l)+0.5Gi,AO(l).G_i^{(l)} = 0.5G_{i,\mathrm{AA}}^{(l)} + 0.5G_{i,\mathrm{AO}}^{(l)}.

During actor replay for old log-probabilities, we capture Q/K after rotary position embedding (Su et al., [25]) and before fused attention (Dao et al., [3]), reconstructing the packed tensor/context-parallel sequence. Query blocks use full causal-prefix normalization before message aggregation, and the historical-step submatrix is not renormalized, which preserves its mass relative to other context tokens. Figure 1 illustrates how token attention becomes a step graph for reward construction.

Figure 1. Overview of ATTNSR. Actor replay supplies attention for a step graph. Backtracking from the terminal step, recency correction, and layer fusion yield an allocation, and confidence and the centered outcome scale it into signed intermediate rewards. The outcome reward is retained. Graph edges link later steps to earlier dependencies, and episode indices are omitted.

Relevance then flows backward from the terminal step, independently within each layer. Writing ai,k(l)≥0a_{i,k}^{(l)} \ge0 for the relevance layer ll assigns to step kk,

ai,Ki(l)=1,ai,k(l)=ρ∑j>kai,j(l)Gi(l)[j,k],(3)a_{i,K_i}^{(l)} = 1,\qquad a_{i,k}^{(l)} = \rho\sum_{j>k} a_{i,j}^{(l)}G_i^{(l)}[j,k], \tag*{(3)}

where j≤Kij \le K_i. Each backward path contributes its edge-weight product with decay ρ=0.85\rho= 0.85 per hop, capturing direct and indirect dependencies. This graph-hop decay differs from the temporal-distance correction below. Only k<Kik < K_i scores enter reward allocation.

How much to trust attention: correction and consensus

To reduce recency preference, define terminal distance Δi,k=Ki−k\Delta_{i,k} = K_i - k and proximity di,k=exp⁡(−(Δi,k−1)/κ)d_{i,k} = \exp(-(\Delta_{i,k} - 1)/\kappa). For Ki>1K_i > 1, we standardize log relevance and proximity over intermediate steps as

zi,k(l)=norm⁡(log⁡max⁡(ai,k(l),ϵ))+ηnorm⁡(di,k),pi,k(l)=exp⁡(zi,k(l)/T)∑h<Kiexp⁡(zi,h(l)/T).(4)z_{i,k}^{(l)} = \operatorname{norm}\left(\log\max(a_{i,k}^{(l)},\epsilon)\right) + \eta\operatorname{norm}(d_{i,k}), \qquad p_{i,k}^{(l)} = \frac{\exp(z_{i,k}^{(l)}/T)}{\sum_{h<K_i}\exp(z_{i,h}^{(l)}/T)}. \tag*{(4)}

Here ϵ=10−12\epsilon= 10^{-12} and norm⁡(x)=(x−mean⁡(x))/std⁡(x)\operatorname{norm}(x) = (x - \operatorname{mean}(x))/\operatorname{std}(x) uses population standard deviation, returning zero for a single element or standard deviation at most 10−610^{-6}. We use T=1T = 1, κ=4\kappa= 4, and η=−0.25\eta= -0.25. The negative proximity term reduces allocation to recent steps but does not set the sign.

Because attribution varies across layers [1], we fuse their independently computed allocations by a geometric mean,

qi,k=exp⁡(1L∑l∈Llog⁡max⁡(pi,k(l),ϵ)),pi,k=qi,k∑h<Kiqi,h,(5)q_{i,k} = \exp\left(\frac{1}{L}\sum_{l\in\mathcal{L}}\log\max\left(p_{i,k}^{(l)},\epsilon\right)\right), \qquad p_{i,k} = \frac{q_{i,k}}{\sum_{h<K_i}q_{i,h}}, \tag*{(5)}

where L=∣L∣L = |\mathcal{L}| is the number of selected layers. Geometric fusion favors consistent support across layers, and the normalization runs over intermediate steps.

Confidence controls the strength of the attention-derived reward. For layer ll, define the terminal covered mass as

mi(l)=∑k<Ki(Gi,AA(l)[Ki,k]+Gi,AO(l)[Ki,k])∈[0,1].(6)m_i^{(l)} = \sum_{k<K_i}\left(G_{i,\mathrm{AA}}^{(l)}[K_i,k] + G_{i,\mathrm{AO}}^{(l)}[K_i,k]\right) \in[0,1]. \tag*{(6)}

This uses the summed AA and AO masses before the equal-weight graph mixture. For Ki>2K_i > 2, define the normalized entropy

Hi(l)=−∑k<Kipi,k(l)log⁡max⁡(pi,k(l),ϵ)log⁡(Ki−1).(7)H_i^{(l)} = -\frac{\sum_{k<K_i}p_{i,k}^{(l)}\log\max\left(p_{i,k}^{(l)},\epsilon\right)}{\log(K_i - 1)}. \tag*{(7)}

For Ki=2K_i = 2, the unique intermediate step has allocation one and we set Hi(l)=0H_i^{(l)} = 0. The confidence components are

Ci,mass(l)=clip⁡(mi(l)/0.20,0,1),(8)C_{i,\mathrm{mass}}^{(l)} = \operatorname{clip}(m_i^{(l)}/0.20,0,1), \tag*{(8)}
Ci,focus(l)=clip⁡((1−Hi(l))/0.15,0,1),(9)C_{i,\mathrm{focus}}^{(l)} = \operatorname{clip}((1-H_i^{(l)})/0.15,0,1), \tag*{(9)}
Ci,layer=1−mean⁡lTV⁡(pi(l),pi),(10)C_{i,\mathrm{layer}} = 1-\operatorname{mean}_{l}\operatorname{TV}(p_i^{(l)},p_i), \tag*{(10)}
Ci=mean⁡l(Ci,mass(l)Ci,focus(l))Ci,layer∈[0,1],(11)C_i = \operatorname{mean}_{l}\left(C_{i,\mathrm{mass}}^{(l)}C_{i,\mathrm{focus}}^{(l)}\right)C_{i,\mathrm{layer}} \in[0,1], \tag*{(11)}

where clip⁡(x,0,1)\operatorname{clip}(x,0,1) saturates to [0,1][0,1] and TV⁡(p,q)=12∑k∣pk−qk∣\operatorname{TV}(p,q) = \frac{1}{2}\sum_k|p_k-q_k|. Covered mass, concentration, and layer agreement measure historical attention, selectivity, and consistency, respectively. The values 0.20 and 0.15 are saturation thresholds. The resulting confidence is a shaping gate, not a calibrated probability of correct attribution.

Zero total intermediate relevance sets that layer focus confidence to zero. For Ki=1K_i = 1, we skip allocation, set Ci=0C_i = 0, and retain terminal-only rewards.

Whether to reinforce a step: a centered budget

Attention indicates computational relevance, not whether a step contributed positively to success. We determine reward sign relative to a running outcome baseline. For collected training batch tt with mean outcome reward MtM_t, we initialize or update the baseline before shaping as

Rˉt={M1,t=1,0.99Rˉt−1+0.01Mt,t>1,Bi=βCi(Ri−Rˉt).(12)\bar{R}_t = \begin{cases} M_1, & t=1,\\ 0.99\bar{R}_{t-1}+0.01M_t, & t>1, \end{cases} \qquad B_i = \beta C_i(R_i-\bar{R}_t). \tag*{(12)}

Algorithm 1 ATTNSR reward construction for one batch

Require: completed trajectory with endpoints ei,ke_{i,k} and rewards RiR_i, selected layers L\mathcal{L}, and the previous baseline if initialized Ensure: token rewards ri,ur_{i,u} and updated baseline Rˉt\bar{R}_t

  1. capture detached post-RoPE QQ, KK during actor replay for old log-probabilities

  2. initialize ri,u←Ri1[u=ei,Ki]r_{i,u} \leftarrow R_i\mathbb{1}[u=e_{i,K_i}] and Ci←0C_i \leftarrow0 for each turn

  3. compute batch mean MtM_t, then initialize or update Rˉt\bar{R}_t by Equation 12

  4. for each episode ii with Ki>1K_i > 1 do

  5. &nbsp;&nbsp;&nbsp;&nbsp;for each layer l∈Ll \in\mathcal{L} do

  6. &nbsp;&nbsp;&nbsp;&nbsp;aggregate causal attention into AA/AO step graphs and set Gi(l)←12(Gi,AA(l)+Gi,AO(l))G_i^{(l)} \leftarrow\frac{1}{2}(G_{i,\mathrm{AA}}^{(l)} + G_{i,\mathrm{AO}}^{(l)})

  7. &nbsp;&nbsp;&nbsp;&nbsp;backtrack relevance and apply recency correction to obtain pi(l)p_i^{(l)} ▷ Eqs. 3–4

  8. &nbsp;&nbsp;&nbsp;&nbsp;compute mass and focus confidence, setting focus to zero if intermediate relevance is zero

  9. &nbsp;&nbsp;&nbsp;&nbsp;fuse layer allocations into pip_i and compute CiC_i ▷ Eqs. 5–11

  10. &nbsp;&nbsp;&nbsp;&nbsp;add βCi(Ri−Rˉt)pi,k\beta C_i(R_i-\bar{R}_t)p_{i,k} at u=ei,ku=e_{i,k} for k<Kik<K_i ▷ retain outcome reward

  11. return ri,ur_{i,u}, Rˉt\bar{R}_t

where β=0.25\beta=0.25. The detached baseline is shared across the batch. Outcomes above it yield positive budgets, and those below it yield negative budgets. Thus sign reflects relative outcome quality, not binary success. For k<Kik<K_i, we assign

si,k=Bipi,k=βCi(Ri−Rˉt)pi,k,(13)s_{i,k}=B_i p_{i,k}=\beta C_i(R_i-\bar{R}_t)p_{i,k}, \tag*{(13)}

at endpoint ei,ke_{i,k}, retaining RiR_i at ei,Kie_{i,K_i}. For Ki>1K_i>1, intermediate rewards sum to BiB_i, and zero confidence yields no shaping. Format-error trajectories retain environment reward Ri=0R_i=0 and participate in the EMA update. Algorithm 1 summarizes reward construction for one batch.

Since CiC_i, RiR_i, Rˉt∈[0,1]\bar{R}_t \in[0,1], ∑k<Ki∣si,k∣=∣Bi∣≤β\sum_{k<K_i}|s_{i,k}|=|B_i|\leq\beta, so the upper bound is length-independent, while the realized budget depends on confidence and outcome. The undiscounted shaped return is Ri+BiR_i+B_i, so policy invariance is not guaranteed. This is a training heuristic distinct from return-preserving redistribution [2], and evaluation uses the original task metrics.

Complexity and integration

Blockwise aggregation into step buckets avoids storing full token attention matrices but retains quadratic query–key arithmetic and selected-layer buffers. It reuses the actor forward pass without additional rollouts or a separately trained reward model. Wall-clock and peak-memory overheads remain unmeasured. Rewards enter PPO before advantage estimation, keeping its critic and losses.

Actor attention is detached when constructing rewards, while policy probabilities remain differentiable during optimization. Given pre-update critic values VuoldV_u^{\mathrm{old}} and unnormalized GAE advantages A^u\hat{A}_u, the canonical PPO training signals are

A~u=sg⁡(Norm⁡(A^u)),yu=sg⁡(A^u+Vuold),(14)\widetilde{A}_u=\operatorname{sg}\left(\operatorname{Norm}(\hat{A}_u)\right), \qquad y_u=\operatorname{sg}\left(\hat{A}_u+V_u^{\mathrm{old}}\right), \tag*{(14)}

where Norm uses the response mask and sg denotes stop-gradient. The actor uses A~u\widetilde{A}_u in its clipped surrogate, and the critic fits returns yuy_u, constructed before advantage normalization, using the existing clipped value loss with reference VuoldV_u^{\mathrm{old}}. Both signals remain fixed within the batch, and neither loss backpropagates through GAE or reward construction.

Experiments

Experimental Setup

Agentic Tasks. We evaluate on ALFWorld (274 held-out tasks, 140 seen and 134 unseen), WebShop (500 test tasks), and ScienceWorld (1,819 test tasks) [24, 33, 29], covering household interaction, online shopping, and scientific procedures.

ALFWorld official score equals task success rate. WebShop and ScienceWorld additionally provide graded scores for partial progress, reported separately from binary success. Their evaluators mark success at normalized scores of at least 0.99 and 0.999, respectively.

RL Training. We use Qwen2.5-7B-Instruct [17] with a frozen base model and rank-64 LoRA adapters [9] for actor and critic. Both learning rates are 5×10−65 \times10^{-6}, the global batch size is 8, and the reference KL coefficient is 0.02. PPO uses one rollout per prompt and one optimization epoch per batch. Separate policies train on each benchmark training split, excluding held-out tasks. Table 7 in Appendix § 9 reports data sizes and update counts. Figure 2 shows how detached actor attention changes policy advantages and value targets through intermediate rewards, retaining the PPO losses. We evaluate after each of four data epochs. Training uses VeRL [22], Megatron [23], and vLLM [12] on 8×8 \times H200 GPUs (140 GB each), with a 32,768-token context and no additional per-turn token or turn-count cap. Attention extraction uses L=6L = 6 layers and 128-token query blocks. Appendix § 9 lists all hyperparameters.

Figure 2. Integration of ATTNSR into actor–critic RL. (a) The PPO integration of ATTNSR adds intermediate rewards while retaining the outcome reward. (b) Detached actor attention supplies rewards, and pre-update critic values support GAE and target construction. Normalized advantages guide the actor, while estimated returns supervise the critic as in Equation 14. Targets remain fixed during optimization. Solid arrows carry signals or targets, and dashed arrows denote updates.

BenchmarkTraining tasksUpdates/epochUpdates at epoch 4
ALFWorld3,5534441,776
WebShop10,5871,3235,292
ScienceWorld3,5924491,796

Table 7. Canonical training-set sizes and update budgets, per method. One update uses one collected batch of eight trajectories.

Baselines and Controls. Terminal-only PPO matches ATTNSR in model, rollouts, optimizer, and evaluation. Linear Step Reward, described in § 4.3, controls for positional shaping on ALFWorld. GRPO uses four rollouts per prompt for one epoch. Alongside the 7B GRPO reference, we compare 3B GRPO with an AttnSR token-advantage variant, as detailed in Appendix § 12.§ 2. PPO/GRPO optimization budgets differ. We additionally compare adaptations of TARL [27], AgentPRM [32], and FlowTracer [5] on ALFWorld after four epochs of Qwen2.5-7B-Instruct LoRA training, using their final checkpoints. Appendix § 13 gives the details.

Evaluation Protocol. We decode greedily once per held-out task and report full-set metrics. The WebShop validation split is reserved for training-time validation. The main comparison uses epoch 4 without selecting the best test-set checkpoint. Format errors violate the required tool-call protocol and receive outcome reward zero without an additional training penalty. For statistical comparisons, exactinput matching retains all 274 ALFWorld and 500 WebShop tasks and 1,488 of 1,819 ScienceWorld tasks, as described in Appendix § 11. A two-sided exact McNemar test (McNemar, [15]) assesses binary success differences, and a paired bootstrap (Efron, [6]) with 10,000 resamples estimates 95% percentile intervals for success-rate and official-score differences. These intervals quantify sampling uncertainty over the matched held-out tasks, conditional on the trained policies.

MethodSuccess
Raw26.64
GRPO†69.71
PPO91.24
TARL96.72
AgentPRM91.97
FlowTracer95.62
PPO + AttnSR97.08

Table 1b. ALFWorld

Raw is the instruction-tuned model before RL. †GRPO uses epoch 1 with n=4n = 4, and all other trained policies use epoch 4, with n=1n = 1 for PPO and AttnSR. Optimization budgets differ. ALFWorld score equals success, while other tasks report score and success separately. Panel (b) reuses the canonical GRPO, PPO, and AttnSR results, and Appendix § 13 details the other baselines.

Main Results

As shown in Table 1, at epoch 4 AttnSR shows the clearest benefit on ALFWorld, smaller numerical gains on ScienceWorld, and mixed outcomes on WebShop. ALFWorld success rises from 91.24% to 97.08% (266/274), reducing failures from 24 to 8. ScienceWorld official score increases from 44.53% to 46.35%, and success from 28.59% to 30.90%. WebShop success rises from 40.20% to 41.00%, but official score falls from 70.74% to 69.71%.

MethodALFWorld ScoreALFWorld SeenALFWorld UnseenWebShop ScoreWebShop SuccessScienceWorld ScoreScienceWorld Success
GRPO†69.7175.0064.1867.2137.8030.1916.66
PPO91.2492.8689.5570.7440.2044.5328.59
PPO + AttnSR97.0896.4397.7669.7141.0046.3530.90

Table 1. Task performance with Qwen2.5-7B-Instruct. (a) Main Result across three benchmarks. (b) ALFWorld success. All values are percentages (%). (a) Main Result

Additional ALFWorld Baselines. Table 1(b) compares final success with TARL (96.72%), AgentPRM (91.97%), and FlowTracer (95.62%). AttnSR attains the highest observed success, 97.08%, although its margin over TARL is only one task (266 versus 265 of 274). These single-run results are descriptive, and the paired tests below concern the canonical PPO/AttnSR comparison.

Statistical Evidence. Tests use exact-input-matched tasks rather than the full-set estimates in Table 1. On ALFWorld, 20 tasks are solved only by AttnSR and 4 only by PPO, giving p=0.002p = 0.002 and a 95% gain interval of [+2.55, +9.49] points. Only ALFWorld remains significant after Bonferroni correction across the three success metrics. ScienceWorld gains +2.49 points on 1,488 matched tasks, versus +2.31 on the full set, but its unadjusted p=0.040p = 0.040 does not survive correction. As reported in Appendix § 10, its score interval and both WebShop intervals include zero.

Ablation, Control, and Transfer Analysis

Positional Reward Control. Linear Step Reward tests whether a simple temporal allocation can reproduce the ALFWorld gain. It retains the outcome reward, outcome centering, and budget scale 0.25, but replaces attention allocation with increasing temporal weights and removes confidence gating. For Ki>1K_i > 1,

wi,klin=k∑j=1Ki−1j,si,klin=0.25(Ri−Rˉtlin)wi,klin.(15)w_{i,k}^{\mathrm{lin}} = \frac{k}{\sum_{j=1}^{K_i-1} j}, \qquad s_{i,k}^{\mathrm{lin}} = 0.25(R_i - \bar{R}_t^{\mathrm{lin}})w_{i,k}^{\mathrm{lin}}. \tag*{(15)}

Each run maintains its own outcome baseline. The control reaches 92.34% success, gaining 1.10 points over PPO versus 5.84 points for AttnSR, as shown in Table 2. The control thus recovers less than one fifth of the gain.

MethodSucc. (%)Δ\Delta (pp)
PPO91.24–
+Linear Step Reward92.34+1.10
+AttnSR97.08+5.84
PPO Asynchronous59.32–
+AttnSR63.99+4.67
Qwen3-4B-Instruct-2507
PPO87.96–
+AttnSR91.24+3.28
Qwen2.5-3B-Instruct
PPO79.56–
+AttnSR87.23+7.66
GRPO12.41–
+AttnSR20.07+7.66

Table 2. ALFWorld success (%). Δ\Delta is the gain over each block baseline in percentage points.

Table 5 in Appendix § 7 shows that Linear Step Reward leads at epoch 1 but trails both methods at epochs 2–3 and AttnSR at epoch 4. Joint changes to allocation, gating, and realized budgets prevent isolating attention contribution or ruling out other dense rewards. Differences are descriptive because per-task outputs were not retained for paired testing.

MethodEpoch 1Epoch 2Epoch 3Epoch 4
PPO36.5069.3489.4291.24
Linear Step Reward43.4359.8579.2092.34
AttnSR37.5970.4491.2497.08

Table 5. ALFWorld success (%) across checkpoints for the canonical positional control. All checkpoints use the full 274-task held-out set.

Supplementary Training Configuration. Single-trajectory asynchronous Qwen2.5-7B-Instruct training gains 4.67 points in Table 2. Changed collection, importance weighting, and critic updates preclude component attribution or direct comparison with canonical PPO, as discussed in Appendix § 12.

Cross-backbone Check. Qwen3-4B-Instruct-2507 gains 3.28 points, and Qwen2.5-3B-Instruct gains 7.66 points, rising from 218 to 239 successes in Table 2. As shown in Figure 4 in the appendix, the 3B model improves on both seen and unseen tasks at all four checkpoints. Both retain the main PPO and shaping hyperparameters, and the 3B model uses relative-depth attention layers.

Figure 4. Qwen2.5-3B-Instruct held-out ALFWorld success after each of four PPO training epochs, shown overall (274 tasks), seen (140), and unseen (134). Each point is a full-set greedy evaluation from one training run per method. Lines connect checkpoints without smoothing, and no estimate of training-seed uncertainty is available.

Extension to GRPO. GRPO uses four rollouts per prompt for one epoch, matching the rollout count of four-epoch PPO with one rollout per prompt, but with fewer policy updates. 3B GRPO reweights group-normalized outcome advantages across action tokens, preserving each trajectory’s mean advantage with α=0.25\alpha= 0.25, as detailed in Appendix § 12.§ 2. In Table 2, success rises from 34 to 55 tasks. Format-error rates of 36.86% and 34.67% and single runs limit interpretation.

Learning Dynamics and Task-level Analysis

Training and Checkpoints. As reported in Appendix § 7, AttnSR scores higher at all four ALFWorld and ScienceWorld checkpoints, with the largest gaps at epochs 4 and 3, respectively. WebShop gains at epochs 1–3 reverse slightly at epoch 4. The eight-epoch traces with learning rate 1e-6 in Figure 3 supplement these results, but neither establishes an asymptotic advantage.

Environment outcome reward during eight-epoch training for ALFWorld, WebShop, and ScienceWorld

Figure 3. Environment outcome reward during eight-epoch training (learning rate 1e-6), excluding shaping rewards. These traces supplement the four-epoch held-out comparison.

Generalization and Task Categories. ALFWorld success rises from 92.86% to 96.43% on seen tasks and 89.55% to 97.76% on unseen tasks, as shown in Table 1. Clean, Heat, Cool, and Pick2 gain 6.89–10.87 points, while Pick and Look remain unchanged, as listed in Table 6 in the appendix. Gains on object-processing and two-object tasks are consistent with benefits on dependent actions, but amount to only three to five successes per family. PPO already reaches 100%/96.77% on Pick/Look, so complexity is confounded with headroom.

PickLookCleanHeatCoolPick2Overall
nn593158394641274
PPO (%)100.0096.7786.2189.7486.9687.8091.24
AttnSR (%)100.0096.7793.10100.0097.8395.1297.08
Δ\Delta (pp)0.000.00+6.89+10.26+10.87+7.32+5.84

Table 6. Fine-grained ALFWorld task-family success at epoch 4 on the full held-out set. Columns use the compact family names Pick (pick and place simple), Look (look at object in light), Clean, Heat, Cool, and Pick2 (pick two objects and place). Counts sum to 274 tasks.

Inference-time Interaction Analysis

Table 3 covers 5,186 PPO/AttnSR and 2,593 GRPO trajectories. Steps and output length do not measure reasoning or generated-token compute. The paired analyses in Appendix § 11 cover only the PPO and AttnSR policies.

TaskMethodMeanMed.P90Out. (k)
ALFWorldGRPO†36.9311.0105.033.44
PPO19.289.038.716.91
+AttnSR15.518.031.013.32
WebShopGRPO†8.466.012.011.65
PPO5.935.09.08.31
+AttnSR8.037.012.011.08
ScienceWorldGRPO†27.1913.074.043.93
PPO30.6815.080.045.42
+AttnSR21.5110.074.031.99

Table 3. Full-set trajectory statistics. Steps count environment interactions, and Out. is output in thousands of characters, including observations. †Same GRPO reference and checkpoint budgets as Table 1.

Overall Interaction Cost. Mean steps fall from 19.28 to 15.51 on ALFWorld (−19.56%) and from 30.68 to 21.51 on ScienceWorld (−29.88%), with higher success. Serialized output falls by 21.22% and 29.56%, respectively. WebShop rises from 5.93 to 8.03 steps without a clear gain.

What Explains Shorter Trajectories? On 268 ScienceWorld tasks solved by both policies, mean steps fall from 11.06 to 7.40. The 246 joint ALFWorld successes instead increase from 11.19 to 11.99 steps, and the 157 joint WebShop successes from 5.11 to 6.19, as shown in Table 12 in the appendix. The aggregate ALFWorld saving accompanies fewer prolonged failures, since the 24 PPO failures average 103.83 steps versus 109.12 for the 8 AttnSR failures. As discussed in Appendix § 11, earlier unsuccessful stopping may also reduce aggregate length. These outcome-selected comparisons distinguish shorter solutions from fewer failures but do not establish a causal mechanism.

TaskEligibleBoth succeedPPOAttnSRΔ\Delta steps
ALFWorld27424611.1911.99+0.80
WebShop5001575.116.19+1.08
ScienceWorld148826811.067.40-3.66

Table 12. Environment steps on exact-input-matched episodes where both policies succeed. Eligible pairs require the same evaluation row and identical input text, unique within each benchmark set. All ALFWorld and WebShop episodes qualify, while ScienceWorld retains 1,488 of 1,819 episodes, excluding 331 with different or non-unique input text. Results are descriptive.

Scope of the Benefit. WebShop has shorter PPO trajectories (5.93 steps versus 19.28/30.68 on ALFWorld/ScienceWorld), relatively close PPO/GRPO scores (70.74%/67.21%), and no detectable AttnSR gain. This suggests less benefit when shorter trajectories reduce delayed-credit difficulties. However, as noted in Table 1, PPO and GRPO optimization budgets differ, so this comparison establishes neither equivalence nor convergence. Policy-dependent lengths also confound horizon and complexity. Greater benefit on complex, long-horizon tasks remains a hypothesis, not evidence that step rewards generally fail on short tasks.

Related Work

Agentic RL. Language agents interleave reasoning with actions [34, 16], each spanning multiple tokens. This creates a distinction between token-level optimization and step-level credit assignment. POAD decomposes Bellman backups within and across actions [31], ArCHer pairs a turn-level critic with a token-level actor [37], and GiGPO groups steps with matching anchor states to estimate critic-free relative advantages [7]. These methods address the granularity of value or advantage estimation. AttnSR instead intervenes at the reward interface. It assigns outcome-conditioned rewards to interaction steps, which then enter the target computation of RL algorithms. In our token-level PPO implementation, these rewards are placed at assistant-step endpoints and propagated through GAE [19, 18], retaining the critic architecture and optimization losses. We evaluate this reward interface in PPO and separately test attention-based token-advantage reweighting in GRPO.

Credit assignment. Long horizons and partial observability complicate credit assignment from terminal outcomes [36]. VinePPO estimates intermediate values through Monte Carlo rollouts [11], AgentPRM learns step-level promise and progress [32], TARL uses an LLM judge for turn-level process supervision [27], and SWEET-RL trains an evaluator with privileged information [38]. Process supervision also uses step annotations [13], estimated progress [20], or trajectory preferences [14]. Other approaches use counterfactual baselines [8], hindsight estimators [26], learned progress in SPA-RL [28], or gradient saliency in ACPO [35]. FlowTracer derives token credit from answer-targeted information flow on an attention-induced graph [5]. AttnSR draws its allocation prior from actor attention, avoiding additional environment rollouts, a separately trained large reward model, or an attribution backward pass. This signal expresses relevance rather than verified causal importance [10, 1]. Confidence gating moderates its strength, while centered outcome determines its sign. Unlike return-preserving redistribution in RUDDER [2], ATTNSR retains the outcome reward and adds intermediate rewards. It thus changes the return without guaranteeing policy invariance, and we evaluate on the original metrics.

Conclusion

ATTNSR allocates outcome-conditioned rewards using actor attention, without extra rollouts or a learned reward model. With PPO, ALFWorld success rises from 91.2% to 97.1%, exceeding the linear control. ScienceWorld shows numerical gains with fewer interactions, while WebShop has no statistically detectable gain. Supplementary experiments show gains under both PPO and a GRPO token-advantage reweighting variant, with one training run per method. Future work will extend the evaluation to longer context lengths and interaction horizons, examining how attention-based credit allocation handles action dependencies spanning more distant steps.

Ethics Statement

This work studies training methods for simulated text-based agents. The experiments use benchmark environments and do not involve human subjects or collection of personal data. As with other agent-training methods, improved planning and tool-use capability could be transferred to applications with unintended consequences. Deployment should thus include task-specific safeguards and monitoring.

Data Availability Statement

We use publicly available ALFWorld, WebShop, and ScienceWorld benchmark environments and do not introduce a new dataset. The evaluation splits, checkpoint-level summaries, and derived tables used in this paper are included with the supplementary artifacts.

Reproducibility Statement

The training configuration, reward equations, evaluation splits, and checkpoint-level results are specified in this paper and its appendix. An anonymized repository will provide the reward extraction and allocation implementation, launch configurations, evaluation summaries, experiment record, significance analysis, and cached per-episode inputs. The main experiments use Qwen2.5-7B-Instruct, LoRA rank 64, VERL/Megatron training, vLLM rollout, and the fixed held-out sets of § 4.1.

AI Use Statement

Generative AI tools were used to assist with the organization and language editing of this manuscript, as well as with the development, debugging, and refinement of experimental code. The authors reviewed and verified all AI-assisted content, including technical claims, equations, experimental results, implementation details, and code, against the repository artifacts and experimental records. The authors remain fully responsible for the correctness, integrity, and reproducibility of this work.

Full checkpoint results

Table 4: Official scores (%) at four checkpoints from the canonical held-out evaluations. ALFWorld official score equals binary success.

TaskMetricEpoch 1Epoch 2Epoch 3Epoch 4
ALFWorld PPOscore (%)36.5069.3489.4291.24
ALFWorld AttnSRscore (%)37.5970.4491.2497.08
WebShop PPOscore (%)63.7267.5366.7170.74
WebShop AttnSRscore (%)65.7468.3569.5869.71
ScienceWorld PPOscore (%)22.8931.4934.5944.53
ScienceWorld AttnSRscore (%)27.1235.5741.1546.35

Table 4. Official scores (%) at four checkpoints from the canonical held-out evaluations. ALFWorld official score equals binary success.

Table 5: ALFWorld success (%) across checkpoints for the canonical positional control. All checkpoints use the full 274-task held-out set.

ALFWorld category-level results

Table 6: Fine-grained ALFWorld task-family success at epoch 4 on the full held-out set. Columns use the compact family names Pick (pick and place simple), Look (look at object in light), Clean, Heat, Cool, and Pick2 (pick two objects and place). Counts sum to 274 tasks.

Implementation details

The selected layers are 8, 10, 12, 14, 16, and 18. We use full causal-prefix softmax attention and do not renormalize within the step submatrix. AA mass targets historical assistant tokens, while AO mass targets the observation paired with the historical step. Step rewards are placed at assistant-step end tokens, and tool observations are excluded from the actor loss. The main implementation uses query blocks of 128 tokens and reconstructs tensor/context-parallel shards before graph aggregation. Format errors are assigned the ordinary outcome reward of zero rather than an extra penalty.

Each benchmark is trained independently from the instruction-tuned checkpoint. Table 7 reports the training manifests and collected-batch updates. Epoch lengths use complete batches of eight, dropping the incomplete final batch when present. Training reward is binary success on ALFWorld, the official score in [0,1][0, 1] on WebShop, and the final ScienceWorld score clipped to [0,100][0, 100] and divided by 100. The native ScienceWorld step-reward increments are not supplied to PPO. Benchmark-specific expert-action and search bonuses are disabled.

Paired significance analysis

Table 10 reports the full uncertainty analysis summarized in § 4.2. Both policies decode greedily on identical held-out tasks, so the final-checkpoint comparison is paired at the task level over the exact-input-matched episodes defined in Appendix § 11. Wins and losses count tasks solved by exactly one policy, and the McNemar test is applied to those discordant pairs only. Bootstrap intervals resample the paired per-task differences 10,000 times.

ComponentConfiguration
Base modelQwen2.5-7B-Instruct
Training frameworkVeRL with Megatron-based training
Rollout enginevLLM
Hardware8×8 \timesH200 GPUs; 140 GB memory per GPU
Fine-tuningLoRA rank 64; frozen base model
Trainable modulesActor and critic LoRA adapters
Message formatQwen standard chat format with JSON tool calls
Maximum context32,768 tokens
Additional length limitsNo per-turn token cap or turn-count cap
Training parallelismMegatron TP=4, CP=2
Rollout parallelismTensor parallelism 4

Table 8. Framework and model configuration for the canonical runs.

Inference-trajectory diagnostics

Source and measurement. We re-read the six final-checkpoint generations/*_openai.jsonl files referenced by the canonical evaluation CSVs. The analysis includes 274 ALFWorld, 500 WebShop, and 1,819 ScienceWorld episodes per method. Recomputed means for official score, success, environment steps, parsed calls, serialized output length, and trace-parser events agree with the evaluation summaries. Per-episode numeric extracts and source-file hashes are retained with the analysis script, with no new training or evaluation.

Table 3 additionally reports the three historical one-epoch GRPO evaluations on the same full-set sizes (2,593 trajectories). We extract their environment steps and serialized output lengths using the same definitions, and verify score, success, format-error rate, natural termination, and mean steps against the original GRPO summaries. Source hashes and numeric extracts are retained separately. These GRPO statistics are excluded from the paired and significance analyses below.

Environment steps use the evaluator env_steps field. Tool calls are counted from openai_tool_calls. They are parseable calls, not necessarily valid or successful environment actions. In these exports, parsed assistant tool-call turns equal the call count and are not reported as an independent measure of reasoning depth. Output length counts characters in the serialized output, including tool-call syntax and observations. We do not interpret it as model-generated token count, internal reasoning length, or inference FLOPs. Medians and P90 values are computed from the episode-level observations, and P90 uses linear interpolation.

Outcome-conditioned and paired comparisons. Table 11 separates successful and unsuccessful episodes using the evaluator binary success flag. These two groups have different task membership under the two policies. The paired analysis in Table 12 additionally requires the same evaluation row, an exact input-text hash match, and an input unique within each benchmark set. This retains all 274 ALFWorld and 500 WebShop pairs. For ScienceWorld, 331 episodes are excluded because inputs differ or are non-unique, leaving 1,488 eligible pairs and 268 joint successes. No matching is inferred from row position alone.

TaskMethodnnMean stepsnnMean steps
ALFWorldPPO25011.1624103.83
AttnSR26612.698109.12
WebShopPPO2015.222996.41
AttnSR2056.622959.00
ScienceWorldPPO52011.32129938.43
AttnSR5628.00125727.55

Table 11. Full-set trajectory lengths on each benchmark. Success and failure groups differ across policies and are not matched task subsets. Table 12 provides the restricted paired comparison.

On ALFWorld, the verified pairs include 20 tasks solved only by AttnSR, 4 only by PPO, 246 by both, and 4 by neither. This yields the observed net gain of 16 successes without claiming that every previously solved task remains solved. The corresponding WebShop counts are 48, 44, 157, and 251. Within the restricted ScienceWorld eligible set, they are 173, 136, 268, and 911. These counts should not replace the full-test success totals. Successful-subset comparisons are descriptive and post-selected on outcomes of both policies, not estimates of a causal treatment effect.

TaskPairsWinsLossesMcNemar ppΔ\Delta Success (pp)Δ\Delta Score (pp)
ALFWorld2742040.002+5.84 [+2.55, +9.49]+5.84 [+2.55, +9.49]
WebShop50048440.755+0.80 [-3.00, +4.40]-1.04 [-3.36, +1.20]
ScienceWorld14881731360.040+2.49 [+0.20, +4.70]+1.85 [-0.37, +4.06]

Table 10. Paired evaluation-set uncertainty for the final-checkpoint comparison, computed over exact-input-matched episodes. Wins and losses count tasks solved by exactly one policy, and pp is the unadjusted two-sided exact McNemar test on those discordant pairs. Intervals are 10,000-resample percentile bootstrap intervals for the mean paired difference (AttnSR − PPO) in points. ALFWorld official score equals binary success. Intervals quantify sampling uncertainty over matched tasks.

Tool use and termination. Table 13 distinguishes exported trace-parser events from the evaluator terminal format-error flag. Most parser events are unmatched call/response pairs, not malformed JSON. Specifically, ALFWorld PPO has 23 missing-response events, while AttnSR has 8 missing-response and 4 orphan-response events. ScienceWorld PPO has 426 missing-response and 10 call-parsing events, while AttnSR has 245 missing-response events and no call-parsing events. WebShop has zero events under PPO and 3 missing-response events under AttnSR. An episode may contain multiple events, so the event count per episode differs from the fraction of affected episodes. We do not equate these exported trace diagnostics with runtime tool-execution failures.

TaskMethodParse/ep.Parse%Format%Natural%Repeat%
ALFWorldPPO0.0848.390.000.000.04
ALFWorldAttnSR0.0443.280.000.000.00
WebShopPPO0.0000.000.000.000.04
WebShopAttnSR0.0060.600.000.000.03
ScienceWorldPPO0.24023.970.6012.1515.35
ScienceWorldAttnSR0.13513.470.2218.4216.30

Table 13. Tool-trace and termination diagnostics at epoch 4. Parse/ep. counts exported trace-parser events, and Parse% is the fraction of episodes with any such event. Format% and Natural% are the evaluator terminal flags. Repeat% is the pooled fraction of adjacent parsed-call pairs with identical function names and arguments, and repetition need not imply an invalid or redundant action.

Natural termination is also not a success indicator. On ScienceWorld its rate increases from 12.15% to 18.42%, so earlier unsuccessful stopping may contribute to the full-set length reduction, which is why we separately report jointly successful trajectories. Exact adjacent-call repetition does not uniformly decrease either, as the pooled ScienceWorld repetition fraction increases from 15.35% to 16.30%. Repeated commands can be necessary in stateful environments and are not labeled invalid or redundant without inspecting the intervening state. On WebShop, the parsed-call totals are 824 search and 2,143 click calls for PPO, versus 1,088 search and 2,925 click calls for AttnSR. This confirms that the higher interaction count involves both search and navigation rather than one tool alone.

Scope of supplementary runs

Table 1 includes a one-epoch GRPO reference with four rollouts per prompt. Table 2 reports the canonical positional control, asynchronous and Qwen3 backbone comparisons, and the Qwen2.5-3B

Table 11: Full-set trajectory lengths on each benchmark. Success and failure groups differ across policies and are not matched task subsets. Table 12 provides the restricted paired comparison.

Table 12: Environment steps on exact-input-matched episodes where both policies succeed. Eligible pairs require the same evaluation row and identical input text, unique within each benchmark set. All ALFWorld and WebShop episodes qualify, while ScienceWorld retains 1,488 of 1,819 episodes, excluding 331 with different or non-unique input text. Results are descriptive.

PPO and GRPO comparisons. The supplementary results are interpreted within their respective settings and are not pooled with the primary comparison for component attribution.

Asynchronous configuration. Both variants use Qwen2.5-7B-Instruct, rank-64 actor and critic LoRA, batch size 8, actor/critic learning rates of 5×10−65 \times10^{-6}, a 32,768-token context, and four data epochs on ALFWorld. This configuration uses asynchronous collection with a replay queue and rollout-policy token log-probabilities. Direct token importance sampling masks ratios outside [0.7,6.0][0.7, 6.0]. Observation tokens are skipped in GAE, with actor λ=1−1/(1.5ni)\lambda= 1 - 1/(1.5n_i) for nin_i trainable assistant tokens and critic λ=1\lambda= 1. Two critic passes are performed per actor pass, and critic LoRA is restricted to MLP projections, with a trainable value head. The critic is initialized from the instruction-tuned model without additional value pretraining. The AttnSR variant retains the canonical attention layers and shaping hyperparameters. Thus the comparison changes more than collection timing relative to canonical PPO.

Qwen3 configuration. The paired launch configuration uses Qwen3-4B-Instruct-2507 with the same main hyperparameters as the canonical study, namely rank-64 LoRA, actor/critic learning rates of 5×10−65 \times10^{-6}, batch and PPO minibatch size 8, one rollout per prompt, one PPO epoch per batch, reference KL coefficient 0.02, and a 32,768-token context. Training uses TP=4 and CP=2, and rollout uses TP=4. AttnSR retains the selected layers and reward settings. Four ALFWorld data epochs give 1,776 updates. Greedy evaluation of the epoch-4 policies on all 274 held-out tasks yields 241 successes for PPO and 250 for AttnSR. This is supplementary evidence across backbones, not a test isolating model size or individual reward components.

Qwen2.5-3B PPO checkpoints

Both policies start from Qwen2.5-3B-Instruct with a frozen base and rank-64 actor/critic LoRA. The main settings are retained, including actor/critic learning rates 5×10−65 \times10^{-6}, batch and minibatch size 8, one rollout per prompt, one PPO epoch per batch, KL coefficient 0.02, and a 32,768-token context. Four data epochs give 1,776 updates, with checkpoints every 444 updates. AttnSR retains the reward budget scale 0.25, confidence targets, EMA, and attention aggregation settings in Table 9. To preserve relative depth in the 36-layer model, each zero-based reference index l∈{8,10,12,14,16,18}l \in\{8, 10, 12, 14, 16, 18\} is mapped to ⌊35l/27+0.5⌋\lfloor35l/27 + 0.5 \rfloor, yielding {10,13,16,18,21,23}\{10, 13, 16, 18, 21, 23\}.

ComponentConfiguration
Batch size8
PPO update epochs1
Rollouts per prompt1
Advantage estimatorGAE with trajectory-level agent advantage
Actor / critic learning rate5×10−6/5×10−65 \times10^{-6} / 5 \times10^{-6}
Reference-policy KL coefficient0.02
Checkpoint frequencyOnce per data epoch
DecodingGreedy; one rollout per held-out task
Evaluation setsALFWorld 274; WebShop 500; ScienceWorld 1,819
Attention layers8, 10, 12, 14, 16, 18
Attention aggregationFull causal-prefix softmax; no step-submatrix renormalization; AA/AO = 0.5/0.5
Query block size128 tokens
Step-reward budget scale0.25
Global reward meanFirst-batch mean initialization; stop-gradient EMA coefficient 0.99
Hop decay0.85
Temporal-bias correctionWeight −0.25; distance scale κ=4\kappa= 4
Target attention mass / focus0.20 / 0.15
Reward placementFinal token of each assistant step; outcome reward on final step

Table 9. PPO, evaluation, and AttnSR hyperparameters used for the main results.

Figure 4 reports greedy evaluation of every checkpoint on all 274 held-out tasks. Overall success for PPO/AttnSR is 4.74/22.26%, 33.21/61.31%, 61.68/72.26%, and 79.56/87.23% at epochs 1–4.

The final checkpoint is reported without best-checkpoint selection. At epoch 4, mean environment steps fall from 32.79 to 22.44, while format errors increase from 0.00% to 2.55%, so higher success does not imply uniform improvement in protocol compliance. Each method has one training run, so the curves do not estimate training-seed variability. The baseline and AttnSR source trees also contain critic/value-head compatibility differences beyond attention reward code, so these runs are a descriptive cross-backbone check rather than an isolated reward-component ablation.

Qwen2.5-3B GRPO extension

The two GRPO policies independently start from Qwen2.5-3B-Instruct and train for one data epoch with four rollouts per prompt. Both use a frozen base, LoRA rank/alpha 64/64, actor learning rate 5×10−65 \times10^{-6}, prompt batch/minibatch size 8, actor KL loss coefficient 0.02, and no critic. Training uses TP=4, CP=2, rollout TP=4, and a 32,768-token context with prompt/response limits 7,168/25,600. The final checkpoint is saved at update 444. Evaluation uses greedy decoding on the same 274 tasks, with no training-time attention signal. Four rollouts for one epoch match the nominal trajectory count of one rollout for four PPO epochs, but not the update count, generated tokens, or runtime.

This experiment uses the initial token-advantage reweighting AttnSR variant. Let AiA_i be the ordinary GRPO outcome advantage computed from original environment rewards within each prompt group. Intermediate-step weights are broadcast to their action tokens and normalized to mean one over valid tokens with positive weights, accounting for step lengths. Set w~i,u=1\tilde{w}_{i,u}=1 at other valid action tokens, including the final step, unparsed positions, and zero-weight positions. The actor uses

Ai,u′=Ai[(1−α)+αw~i,u],α=0.25.(16)A'_{i,u}=A_i\left[(1-\alpha)+\alpha\tilde{w}_{i,u}\right],\qquad\alpha=0.25. \tag*{(16)}

Observations and padding are masked. This preserves each trajectory’s mean advantage and its sign, while varying its magnitude across steps, and equal-reward groups retain zero advantage. The collector uses the same relative layers and attention allocation as the 3B PPO run. However, positive trajectory confidence is a common factor that cancels during token-weight normalization, and only zero confidence disables redistribution. Thus this variant does not retain PPO’s continuous confidence modulation, outcome EMA, or intermediate rewards and GAE.

Both methods use the same GRPO implementation overlay, with attention reweighting enabled only in the AttnSR arm. Table 2 shows final success of 34/274 for GRPO and 55/274 for GRPO+AttnSR. Seen success rises from 15.00% to 22.14% (21 to 31 of 140 tasks), and unseen success from 9.70% to 17.91% (13 to 24 of 134 tasks). Format-error counts remain high at 101/274 and 95/274. These single-run results are descriptive, with no repeated-seed or paired significance analysis.

Additional ALFWorld baselines

Our ALFWorld adaptations of TARL [27], AgentPRM [32], and FlowTracer [5] are trained independently from Qwen2.5-7B-Instruct with LoRA for four data epochs. We evaluate the final checkpoints at update 1,776 on all 274 held-out ALFWorld tasks (140 seen and 134 unseen), using one greedy rollout per task and a 32,768-token context. The implementation variants recorded in the experiment inventory are TARL-Turn, AgentPRM-Progress, and FlowTracer-PPO, respectively. Table 1(b) retains the canonical PPO and AttnSR results as references, and no paired significance comparison with the three additional methods is claimed.

Table 14 and Table 15 report the final-checkpoint details. Success counts are 265, 252, and 262 for TARL, AgentPRM, and FlowTracer. All three have zero terminal format errors, and AgentPRM has eight naturally terminated episodes. These diagnostic fields are distinct from task success and do not indicate whether training completed normally. The small number of failed tasks (9, 22, and 12) makes failure-conditioned step means sensitive to individual long trajectories. Category-level counts and outcome-conditioned lengths are descriptive, not causal evidence about credit assignment.

MethodOverall / 274Seen / 140Unseen / 134Format errorsNatural term.
TARL265 (96.72)135 (96.43)130 (97.01)00
AgentPRM252 (91.97)130 (92.86)122 (91.04)08
FlowTracer262 (95.62)135 (96.43)127 (94.78)00
MethodLook (31)Pick (59)Clean (58)Cool (46)Heat (39)Pick2 (41)
TARL305854443940
AgentPRM315654403437
FlowTracer305754453640

Table 14. Task outcomes of the additional ALFWorld baselines at the final checkpoint (update 1,776). (a) Successful tasks, with success rates (%) in parentheses. Format and termination columns count episodes. (b) Successful tasks by category, with task counts in the header. Each row of (b) sums to the overall success count in (a). (a) Overall success and termination. (b) Successful tasks by category.

MethodMeanMedianSuccess meanFailure mean(k chars)
TARL13.70810.8398.3311.77
AgentPRM14.18810.3358.1813.89
FlowTracer19.061416.0684.4217.26
MethodMeanMedianP90MeanMedianP90
TARL3,9832,6567,824348214641
AgentPRM4,5962,6739,4936483061,303
FlowTracer5,8374,20310,186480348768

Table 15. Trajectory lengths of the additional ALFWorld baselines at the final checkpoint. (a) Environment steps and mean serialized output length in thousands of characters. Successful and failed subsets differ across policies. (b) Retokenized evaluation-text lengths. Full trajectories include tool observations, while assistant-only text excludes tool replies. These are counts from saved text, not original sampled token IDs or exact generation costs. (a) Environment steps and output length. (b) Retokenized text length.

Text lengths are obtained by retokenizing saved evaluation text with Qwen2TokenizerFast, with add_special_tokens=False. Full trajectories include tool observations, while assistant-only counts exclude tool replies. These measurements do not recover the original sampled token IDs and should not be interpreted as exact generation costs. All results are from one training run per method, and no estimate of training-seed variability is available.

Notation

Table 16 lists the symbols of § 2–§ 3. Episode index ii is omitted where unambiguous.

References

  1. [1]Samira Abnar and Willem H. Z u idema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190–4197, 2020. doi: 10.18653/v1/2020.acl-main.385.DOI
  2. [2]Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1806.07857
  3. [3]Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2205.14135
  4. [4]DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  5. [5]Zhichen Dong, Yang Li, Yuhan Sun, Weixun Wang, Yijia Luo, Zinian Peng, Taiheng Ye, Chao Yang, Wenbo Su, Yu Cheng, Bo Zheng, and Junchi Yan. How does reasoning flow? tracing attention-induced information flow for targeted RL in LLMs. arXiv preprint arXiv:2606.10646, 2026. doi: 10.48550/arXiv.2606.10646. URL https://arxiv.org/abs/2606.10646.
  6. [6]Bradley Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1): 1–26, 1979.
  7. [7]Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.10978
  8. [8]Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, volume 32, 2018. doi: 10.1609/aaai.v32i1.11794.DOI
  9. [9]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
  10. [10]Sarthak Jain and Byron C. Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3543–3556, 2019.
  11. [11]Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Refining credit assignment in rl training of llms. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
  12. [12]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, 2023.
  13. [13]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024.arxiv.org/abs/2305.20050
  14. [14]Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li, Junge Zhang, and Jianbin Jiao. Online process reward learning for agentic reinforcement learning. arXiv preprint arXiv:2509.19199, 2025.
  15. [15]Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947.
  16. [16]Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199, 2024.
  17. [17]Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
  18. [18]John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2016.
  19. [19]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  20. [20]Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning. arXiv preprint arXiv:2410.08146, 2024.
  21. [21]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Li, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
  22. [22]Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys), 2025.DOI
  23. [23]Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.arxiv.org/abs/1909.08053
  24. [24]Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021.arxiv.org/abs/2010.03768
  25. [25]Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
  26. [26]Hui-Ze Tan, Xiao-Wen Yang, Hao Chen, Jie-Jing Shao, Yi Wen, Yuteng Shen, Weihong Luo, Xiku Du, Lan-Zhe Guo, and Yu-Feng Li. Hindsight credit assignment for long-horizon llm agents. arXiv preprint arXiv:2603.08754, 2026.arxiv.org/abs/2603.08754
  27. [27]Weiting Tan, Xinghua Qu, Ming Tu, Meng Ge, Andy T. Liu, Philipp Koehn, and Lu Lu. Process-supervised reinforcement learning for interactive multimodal tool-use agents. arXiv preprint arXiv:2509.14480, 2025. doi: 10.48550/arXiv.2509.14480. URL https://arxiv.org/abs/2509.14480.
  28. [28]Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. Spa-rl: Reinforcing llm agents via stepwise progress attribution. arXiv preprint arXiv:2505.20732, 2025a.
  29. [29]Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11279–11298, 2022. doi: 10.18653/v1/2022.emnlp-main.775.
  30. [30]Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025b.
  31. [31]Muning Wen, Ziyu Wan, Weinan Zhang, Jun Wang, and Ying Wen. Reinforcing language agents via policy optimization with action decomposition. arXiv preprint arXiv:2405.15821, 2024.
  32. [32]Zhiheng Xi, Chenyang Liao, Guanyu Li, Yajie Yang, Wenxiang Chen, Zhihao Zhang, Binghai Wang, Senjie Jin, Yuhao Zhou, Jian Guan, Wei Wu, Tao Ji, Tao Gui, Qi Zhang, and Xuanjing Huang. AgentPRM: Process reward models for LLM agents via step-wise promise and progress. arXiv preprint arXiv:2511.08325, 2025. doi: 10.48550/arXiv.2511.08325. URL https://arxiv.org/abs/2511.08325.
  33. [33]Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, volume 35, pp. 20744–20757, 2022.
  34. [34]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023.
  35. [35]Junxi Yin, Haisen Luo, Zhenyu Li, Yihua Liu, Dan Liu, Zequn Li, and Xiaohang Xu. Pinpointing crucial steps: Attribution-based credit assignment for verifiable reinforcement learning. arXiv preprint arXiv:2510.08899, 2025.arxiv.org/abs/2510.08899
  36. [36]Chenchen Zhang. From reasoning to agentic: Credit assignment in reinforcement learning for large language models. arXiv preprint arXiv:2604.09459, 2026.arxiv.org/abs/2604.09459
  37. [37]Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. In Proceedings of the 41st International Conference on Machine Learning, 2024.
  38. [38]Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks. arXiv preprint arXiv:2503.15478, 2025.

Paper details

vlnc:W_5dtd56s4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 13

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents