In this paper
AbstractIntroductionRelated WorkMethodsProblem Setting and PreliminariesPlug-and-Play Process VerificationLast-Mile RefinementResultsExperimental SetupEffectiveness Across Domains and Model ScalesAblation StudiesEffectiveness Across Policy Optimization AlgorithmsConclusionDisclose Use of LLMsEthics StatementReproducibility StatementTraining and Evaluation BenchmarksPolicy Optimization DetailsTraining DetailsRepresentative Verification Inputs and ChecksMulti-hop Question Answering: HotpotQACode Generation: LiveCodeBenchMultimodal Reasoning: MATH-VisionMathematical Reasoning: AIME 2026ReferencesWorksComputer Science
Plug-and-Play Verifier for Decoupling Process Reward in Stabilizing Long-Horizon Agentic Reasoning
Abstract
Stabilizing long-horizon reasoning requires intelligent agents to sustain useful progress across successive reasoning steps and tool interactions. Such agents are typically trained with reinforcement learning (RL), where actor-critic RL uses a learnable critic network to estimate step advantages, while critic-free methods such as GRPO eliminate critic models and rely mostly on the group variants of sparse outcome-only rewards. Both reward types leave partially useful failed trajectories either implicit or indistinguishable from entirely unproductive ones, and as reasoning horizons grow, it becomes difficult to maintain consistent reasoning trajectories. In this work, we propose a plug-and-play verifier framework for stabilizing long-horizon reasoning through explicit verification of intermediate reasoning artifacts. Specifically, the framework decouples process verification through a unified interface that extracts intermediate reasoning artifacts from agent running trajectories, evaluates them using domain-specific verifiers, and aggregates the verification scores as bounded process reward. Then, we combine our dense verified process rewards and the sparse outcome rewards through a two-stage post-training schedule. We showed that with plug-and-play verifiers, we further stabilize the reasoning steps as last-mile self-reflection and self-consistency. Typical evaluation on HotpotQA using Qwen3.5-4B achieves a superior accuracy (60.18%), outperforming the standard GRPO baseline (57.73%). Our results suggest that plug-and-play verifiers provide a modular approach to learn more stable and reliable reasoning behaviors over long horizons.
Introduction
Large language models are increasingly deployed as agents that solve tasks by interleaving reasoning with actions in external environments, following the reasoning, acting, and observing cycle proposed by ReAct Yao et al. [49]. Equipped with tools and environment feedback, these agents tackle information seeking, software engineering, and web navigation through sequences of interdependent decisions [30, 45, 57, 55]. Success on these tasks requires effective agentic reasoning, where agents must interpret newly acquired evidence, revise their plans, and carry stable intermediate results forward to guide subsequent actions [34, 56, 58]. However, sustaining this capability becomes increasingly challenging as the interaction horizon grows [6, 11]. For example, a hallucinated claim, calculation error, or misinterpreted tool result can misdirect subsequent actions as later decisions build on earlier reasoning and observations [49, 52]. Evaluations across interactive environments, software engineering, and general assistant tasks further highlight persistent difficulties in reliably completing multi-step tasks [24, 18]. These difficulties are exacerbated in multi-agent interactions, where an agent’s erroneous outputs are passed to others and incorporated into reasoning, allowing errors to propagate and compound across agents [8, 4]. Therefore, stabilizing long-horizon agentic reasoning is emergent [43].
Reinforcement learning provides an effective approach to improving agentic reasoning by allowing models to explore alternative strategies and learn from verifiable outcomes without requiring human annotations for every intermediate step [40, 13, 19, 9]. Both actor-critic methods and critic-free methods such as GRPO rely on the supplied rewards to guide policy optimization [31, 33]. However, outcome-only rewards provide limited information about intermediate progress, assigning identical feedback to trajectories that may differ substantially in the validity of their intermediate artifacts [46, 54] (Xu & Fang 2026). This motivates explicit process supervision that can distinguish the quality of intermediate artifacts in long-horizon agentic tasks [46, 54, 52].
In this work, we observe that process rewards can be obtained by checking intermediate artifacts against task conditions and evidence independently of reward composition and policy optimization. Specifically, assessing whether an artifact satisfies a verification criterion does not require deciding how the score is combined with outcome rewards or used to update the policy. Conversely, downstream training requires the verification scores and their source-step assignment from the verifier. This independence allows us to decouple process verification from reward composition and policy optimization. Therefore, we implement process verification in external modules that interact with training through an interface that passes verification scores and their source-step assignment. Moreover, agents in different domains produce different intermediate artifacts. To support agents without adapting the training procedure to each verifier, we propose a unified plug-and-play verification interface. As illustrated in Figure 1, every verifier module extracts intermediate artifacts, verifies them, and returns bounded weighted sums of verification scores assigned to source interaction steps. Since downstream reward composition and policy optimization depend on each verifier only through these outputs, verifier modules for different agents can be introduced or replaced in a plug-and-play manner. Inspired by the above, we propose a plug-and-play verifier framework for stabilizing long-horizon agentic reasoning through explicit verification of intermediate artifacts. Our framework makes the following contributions:
Figure 1. Overview of the plug-and-play verifier. Left: artifacts are extracted per interaction step. Middle: a unified interface routes them to domain-specific verifiers, which return verification scores assigned to source steps. Right: These final reward are aggregated into process rewards and combined with the outcome reward. Bottom: verifiers interact with RL post-training through the interface and can be used independently across agents.
Process rewards by verifying intermediate artifacts against task conditions and environment evidence, providing complementary supervision beyond outcome rewards.
A unified interface that decouples process verification from reward composition and policy optimization, enabling plug-and-play integration of verifiers.]
To evaluate it, we conduct experiments using Qwen3.5-4B and Qwen3.5-9B on the HotpotQA validation split, LiveCodeBench, MATH-Vision, and AIME 2026, covering multi-hop question answering, code generation, multimodal reasoning, and mathematical reasoning. Our experiments show that incorporating a plug-and-play verifier can improve the performance and stabilize the reasoning trajectories. Our ablation studies further examine verification mechanisms and support the contributions of reward composition. Our results suggest that explicit process verification can complement outcome rewards to improve agentic reasoning.
Related Work
Long-horizon reasoning and multi-agent interaction. Long-horizon agentic reasoning requires agents to make successive decisions based on previously generated information and observations from tools or environments. Early approaches support multi-step reasoning through explicit intermediate steps and exploration of alternative reasoning paths [42, 50]. Tool-augmented approaches connect reasoning with external information and executable actions, enabling agents to interact with environments and perform tasks [49, 30, 45]. Feedback and episodic memory can guide agents to revise their strategies across successive attempts [34]. Multi-agent approaches extend these capabilities through role-specific workflows and iterative debate [15, 8]. However, evaluations on WebArena and AgentBench reveal persistent difficulties in long-term planning, decision-making, and reliable task completion [57, 24]. Analyses of multi-agent systems identify failures in coordination and task verification [1]. Recent analysis of search agents further identifies failures in evidence propagation [29].
Agentic reinforcement learning. Reinforcement learning enables agents to learn reasoning and tool-use strategies from task feedback. Search-R1 and ReTool incorporate search and code execution into multi-turn trajectories and optimize interaction policies using outcome rewards [19, 9]. Actor-critic methods use learned value functions to estimate advantages; these functions predict returns rather than verify intermediate artifacts [31]. Critic-free methods such as GRPO normalize rewards within a group of trajectories, but under outcome-only supervision, each trajectory receives a single outcome reward [33]. Research on multi-turn agent training addresses training stability and temporal credit assignment (Wang et al., 2026a,b; He et al., 2026; Yang et al., 2026a). More recently, GiPO combines trajectory-level comparisons with step-level advantage estimation over actions taken at recurring environment states [10]. IGPO instead derives step-level rewards from increases in the policy’s probability of producing the correct answer and combines them with outcome supervision [35].
Process supervision and verifiable feedback. Process supervision provides dense feedback on intermediate reasoning beyond sparse final outcome correctness. A central question is how to obtain reliable supervision for these intermediate steps. Initial human-in-the-loop methods demonstrate the effectiveness of the supervision in mathematical reasoning [23], while Math-Shepherd and ReST-MCTS derive supervision automatically from sampled trajectories and tree search [37, 53]. Rewarding Progress further evaluates intermediate steps by their contribution to a prover policy’s likelihood of success [32]. Alongside these approaches, another line of work grounds feedback in tools, evidence, or explicit verification criteria. CRITIC uses tools to validate and revise outputs at inference time [12], while VPR and VPRMs incorporate symbolic or algorithmic checks and deterministic rules into process rewards for intermediate actions and structured reasoning trajectories [52, 27]. For agentic tasks, this raises a further question of how to attribute feedback to individual actions [11, 2]. STAMP attributes verified evidence to the actions that first exposed it, while ABSeeker constructs step-level supervision from answer-backtracked clues [43, 25].
Methods
Problem Setting and Preliminaries
We study long-horizon agentic reasoning tasks. Given a task domain , let denote an input consisting of a task instruction and any available context, including images for multimodal tasks. A policy , parameterized by , produces a solution through successive policy generations, and interactions with an external environment.
Policy and trajectories. At interaction step , the policy generates an action conditioned on the interaction history . Specifically, an action is a complete model-generated response that can contain reasoning text, a tool call, or a final answer. When a tool is invoked, the environment returns an observation , which is incorporated into the history for the next step. The interaction proceeds until the agent submits a final answer or reaches a stopping condition, producing a trajectory
where denotes the number of policy generations. Observations can include tool outputs or continuation prompts supplied by the interaction protocol, and are omitted when no such input is supplied. Each policy generation contain multiple artifacts. Verification is performed at the artifact level, with scores attributed to the artifacts’ source interaction steps.
Outcome and process rewards. Trajectories with the same outcome reward may differ in whether their intermediate reasoning artifacts satisfy explicit task conditions and are supported by recorded evidence. Thus, we distinguish outcome and process rewards:
The outcome reward evaluates the final answer against domain-specific correctness, and is zero if no valid final answer is submitted. Evaluation references are available to the evaluator and need not be included in the policy input. In contrast, the process reward instead aggregates verification scores for intermediate artifacts, as defined in § 3.2. Importantly, a passing local check establishes only the property tested by its rule and does not establish final-answer correctness.
Plug-and-Play Process Verification
Our framework decouples domain-specific artifact verification from reward composition and policy optimization through a unified interface.
Decoupled local verification. Verification assesses an artifact from explicitly specified inputs, without evaluating the full reasoning narrative. To this end, a domain-specific extractor identifies artifacts in and their source interaction steps. An artifact is an explicit object admitting a check, such as an arithmetic equality, a citation, or a code version associated with an execution record. For each extracted artifact , the verification score is computed as
where contains the task conditions and records required by the check. These can include earlier artifacts or specified submission fields. For fixed rules and evaluation conditions, the score depends on the trajectory only through . Consequently, changing unrelated reasoning text or the final correctness label does not change the score. Moreover, the verifier does not access policy parameters or hidden states, sample policy continuations, or estimate eventual task success.
This separation allows checks to be replayed after trajectory completion when their required inputs have been recorded. In our implementation, the QA verifier uses source-span grounding and action-certificate matching without annotated supporting facts; the code verifier uses version lineage and developer-test replay; the visual verifier checks crop replay and string-level answer-claim coupling; and the mathematics verifier checks extracted numeric equalities. These fixed rules verify their stated properties rather than all reasoning steps. Representative verification inputs and checks for the four domains are provided in Appendix D.
Plug-and-play integration. We expose local verification through adapters and a common verification interface. Specifically, Let be a group of checks assigned to source step , with disjoint groups and domain-specific nonnegative weights. For each group, is the weighted sum of verification scores assigned to source step :
Here, the weights encode domain-specific normalization and check applicability. In particular, interaction steps without applicable verification checks are excluded from step-level normalization, whereas applicable checks that fail verification contribute zero scores. Artifacts scored individually use singleton groups. Unlike the local verification score , the weights used to compute can depend on other artifacts in the trajectory. Thus, the common bound provides a shared reward range, while the verification criteria and normalization rules remain domain-specific.
Based on these aggregated scores, the complete verifier module returns a set of source-step assignments and verification scores:
Here, is the number of emitted records. Records with may be omitted, while full audits retain failed checks, including applicable checks with missing required evidence. An empty verifier output yields zero process reward. Importantly, downstream reward composition consumes only , allowing compatible verifier modules to be replaced without modifying reward composition or policy optimization. Reuse with a different agent requires compatible artifacts and an adapter for its records. Figure 2 summarizes this decoupled design.
Figure 2. The plug-and-play process verification. (a) Identical outcome rewards can mask differences in verification results. (b) Local checks are decoupled from policy reasoning and return weighted sums of verification scores assigned to source steps through a common interface. These are aggregated into process rewards and combined with outcome rewards for GRPO or PPO training.
Reward composition and optimizer interface. At interaction step , the process reward is , with empty sums equal to zero. To combine process and outcome supervision, we introduce an outcome-process weight coefficient and define the step reward as
Process rewards are retained for independently verifiable intermediate artifacts whether a valid final answer is submitted. The outcome reward is assigned to the terminal step, while process rewards are assigned to their source steps. With , the trajectory-level reward is . The final rewards can be used by GRPO or PPO without changing the verification rules. In practice, the training uses outcome-only rewards () for the first 50 updates. At each subsequent update, a weight is sampled reproducibly from for each prompt group by all rollouts in that group. More details are provided in Appendix § 10.
Last-Mile Refinement
Trajectories with identical outcome rewards can differ in the validity of their intermediate reasoning artifacts. This difference allows process rewards to provide last-mile refinement signal when sparse outcome-only supervision is identical [22].
Differentiate rewards for policy refinement. Outcome-only GRPO relies on reward differences within a group of rollouts to construct relative advantages. With binary outcome rewards, both all-incorrect and all-correct groups have zero reward variance [51]. All-incorrect groups can impede early learning when the initial policy rarely generates successful rollouts. Cold-start supervised fine-tuning (SFT) on reasoning demonstrations provides an initialization strategy for RL [13]. On the other hand, the same issue also happens in correct groups. For a fixed prompt and interaction step , we consider a group with outcome rewards and a shared weighted coefficient . All rollouts in receive identical outcome rewards yielding zero advantages. Let denote the remaining process return, and let . Using the reward composition in Eq. (6), the weighted return is . Subtracting the group of rollout mean removes the common outcome term, yielding
where ensures the numerical stability and std is the sample standard deviation. Thus, differences in remaining process returns yield nonzero group-relative advantages even if rewards on all rollouts are the same. These differences provide the last-mile refinement signal for refining intermediate artifacts when the group’s final answers are all correct. More details are in Appendix B
Verification-privileged self-consistency. The last-mile refinement signal is grounded in external verification of intermediate artifacts. The verifiers evaluate these artifacts against explicit recorded evidence. Therefore, the policy learns from process rewards derived from these checks rather than judgments from a black-box reward model. Here, we interpret self-consistency from the above external verification perspective. Specifically, verification-privileged self-consistency requires all applicable verification checks within a single trajectory to pass. Let index the full set of applicable checks under a fixed evaluation protocol. A check fails if its required evidence is missing. Let denote the set of indices of checks passed by the trajectory. We define
Thus, if and only if at least one check applies and every applicable check has verification score . This criterion essentially measures internal (within-trajectory) consistency that obviously differs from the most recent majority-vote-based (external) work via selecting the most frequent answers in sampled trajectories [39].
Verifier-induced self-reflection. Moreover, internal self-consistency motivates self-reflection, where we interpret self-reflection as the policy proactively checking and revising its intermediate reasoning artifacts. Intuitively, self-consistency encourages self-reflection by consistently reducing generated hallucinations at intermediate reasoning steps. After trajectory generation, the verifier checks recorded intermediate artifacts and assigns process rewards to their source steps. An earlier failed check remains in and keeps , even if the agent later corrects the error. Nevertheless, these later artifacts can receive positive process rewards when they passes verification. Therefore, this step-level supervision can encourage the policy to check and revise intermediate reasoning in future trajectories, potentially reducing error propagation.
Overall, our framework decouples the verification process via independently checking its intermediate reasoning artifacts. When reasoning processes have heterogeneous returns, our approach provides a last-mile supervision signal for refining intermediate artifacts. Further, the resulting training outcomes ensure self-consistency and self-reflection. Thus, our framework can mitigate error propagation across successive reasoning steps and tool interactions, ultimately supporting more stable long-horizon agentic reasoning (see Appendix § 12 for reasoning examples).
Results
Experimental Setup
Training Data and Evaluation Benchmarks. We evaluate our approach across four domains: multi-hop question answering, code generation, multimodal reasoning, and mathematical reasoning. For multi-hop question answering, we train on the training split of HotpotQA [48] and evaluate on its validation split. For code generation, we train on TACO [21] and evaluate on LiveCodeBench [17]. For multimodal reasoning, we use the Vision-R1 [16] for training and MATH-Vision [36] for evaluation. For mathematical reasoning, we train on DeepScaleR-Preview-Dataset [26] and evaluate on AIME 2026 [7]. Training is performed separately for each domain. Within each domain, all compared methods use the same training data and evaluation split.
Baselines. We compare our approach with ReAct, which uses the initial model without RL, and GRPO, which trains the same initial model using only the binary outcome reward. We additionally include GRPO + LLM-as-a-judge, which uses a frozen Qwen3.5-9B judge to provide training rewards for the Qwen3.5-4B policy. Across the four domains, the judge evaluates intermediate retrieved evidence, code development and testing, visual inspection, and mathematical reasoning using domain-specific criteria. The resulting process rewards are assigned to the corresponding interaction steps and combined with the same binary outcome reward used by GRPO. Our method obtains process rewards from domain-specific verifiers instead of an LLM judge.
Base Models and Training Framework. We use Qwen3.5-4B and Qwen3.5-9B [28] to examine the effectiveness of our approach at two model scales. Our implementation builds on Agent-R1 [3], with GRPO [33] and PPO [31] as the training algorithms. All methods are implemented using the verl package for reward computation, training, and generation. Within each domain and model scale, the RL methods use the same initialization, task interface, tools, fine-tuning method, and training budget. For all domains, we use the first 10,000 examples from each prepared training split, using global prompt batch sizes of 20 and 8 for Qwen3.5-4B and Qwen3.5-9B. Qwen3.5-9B uses LoRA with rank 64 and scaling factor . Domain-specific settings are provided in Appendix C.
Evaluation Protocol. We evaluate agents by their final task performance and report accuracy (Acc., %) across all four benchmarks. Accuracy is defined as the percentage of evaluation instances for which the agent produces a correct final answer. Correctness is assessed using benchmark-specific criteria: answer exact match for HotpotQA, passing all evaluation test cases for LiveCodeBench, and answer correctness for MATH-Vision, AIME 2025 and AIME 2026. For interactive tasks, each evaluation episode includes the complete sequence of agent feedback until termination, and correctness is determined from the final submitted answer. We use vLLM [20] for speed-up generation.
Effectiveness Across Domains and Model Scales
We began our experimental analysis with a compact yet effective Qwen3.5-4B model, which has been widely used in recent studies [47]. In Table 1, we observed that the Qwen3.5-4B model achieved the best performance on four of the five benchmarks (AIME 2025: 40.00, AIME 2026: 55.83, HotpotQA: 60.18, LiveCodeBench: 49.57) and remained competitive on MATH-Vision (34.54). Furthermore, compared with the baseline methods, our approach achieved the highest average accuracy across the five benchmarks (Ours: 48.02, vs. ReAct: 38.81, GRPO: 44.95, GRPO + LLM-as-a-judge: 46.44). These results provided initial evidence that our plug-and-play verifier could improve agentic reasoning accuracy by incorporating verified process rewards alongside outcome rewards.
| Methods | AIME 2025 | AIME 2026 | HotpotQA | LiveCodeBench | MATH-Vision | Avg. |
| Qwen3.5-4B | ||||||
| ReAct | 26.67 | 50.00 | 46.78 | 40.47 | 30.13 | 38.81 |
| GRPO | 33.33 | 52.50 | 57.73 | 47.96 | 33.22 | 44.95 |
| GRPO + LLM-as-a-judge | 36.67 | 53.33 | 58.28 | 48.72 | 35.20 | 46.44 |
| Ours (plug-and-play verifier) | 40.00 | 55.83 | 60.18 | 49.57 | 34.54 | 48.02 |
| Qwen3.5-9B | ||||||
| ReAct | 36.67 | 53.33 | 52.56 | 49.00 | 36.18 | 45.55 |
| GRPO | 43.33 | 56.67 | 63.20 | 56.59 | 37.01 | 51.36 |
| Ours (plug-and-play verifier) | 46.67 | 58.33 | 66.67 | 58.48 | 38.16 | 53.66 |
Table 1. Performance (%) across four domains using Qwen3.5-4B and Qwen3.5-9B. Each domain is trained separately on its corresponding training dataset. Bold numbers indicate the best.
Next, we further tested our plug-and-play verifier using the larger Qwen3.5-9B model (see Table 1, second block). First, the large models showed better performance in general. For example, our method achieved higher accuracy in HotpotQA with Qwen3.5-9B (66.67) than with Qwen3.5-4B (60.18). Second, compared with outcome-only GRPO, our method consistently achieved better performance across all five benchmarks (AIME 2025: 46.67 vs. 43.33, AIME 2026: 58.33 vs. 56.67, HotpotQA: 66.67 vs. 63.20, LiveCodeBench: 58.48 vs. 56.59, MATH-Vision: 38.16 vs. 37.01). Meanwhile, our method outperformed both GRPO (53.66 vs. 51.36) and ReAct (53.66 vs. 45.55) on average. This trend supported our assumption that sparse outcome rewards may overlook meaningful differences in intermediate artifacts, while explicit process verification can provide complementary learning signals for policy optimization. We provided an illustrative HotpotQA example in Figure 3, with more details in Appendix § 12, to demonstrate how our plug-and-play verifier evaluates intermediate artifacts.

Figure 3. An example of our plug-and-play verifier on HotpotQA.
Regarding the comparison with the LLM-as-a-judge baseline, while our plug-and-play verifier achieved higher average accuracy using Qwen3.5-4B (Ours: 48.02 vs. LLM-as-a-judge: 46.44), it slightly underperformed on MATH-Vision (34.54 vs. 35.20). One potential reason was that our visual verifier primarily relied on explicit checks, such as crop provenance, replay consistency, and answer-claim matching, without directly assessing the semantic correctness of visual reasoning. This might limit the effectiveness of process supervision for tasks requiring more complex visual understanding, where LLM-based verification could provide more flexible feedback. Figure 4 further compares the training accuracy of both methods across the four training datasets. One of our future research directions will explore incorporating semantic verification into the proposed plug-and-play framework to further improve its generality across multimodal reasoning tasks.
Figure 4. Training accuracy of Qwen3.5-4B with Ours and GRPO with LLM-as-a-judge.
Ablation Studies
Having established the above results, we examined the interaction between outcome and process rewards and compared different verification mechanisms. We focused on HotpotQA, which provided both reference answers and annotated supporting facts, allowing outcome supervision and process supervision to be examined separately. All results in this subsection used Qwen3.5-4B with GRPO. We reported accuracy based on normalized exact match (EM).
Effect of process rewards. To examine the interaction between outcome and process rewards, we first considered outcome-only GRPO, which improved performance from the ReAct of 46.78 to 57.73. We then used process rewards based on gold-evidence matching to isolate the effect of process rewards. In Table 2 (a), process-only training achieved 47.27, suggesting that only process rewards were insufficient to optimize the final task objective. In contrast, process and outcome rewards further improved performance to 61.40. Since outcome rewards directly supervise final-answer correctness, whereas process rewards provide feedback on the intermediate evidence produced along the trajectory. These results suggest that evidence-based process supervision is most effective as a complementary signal rather than a replacement for the final task objective. We next compared different sources of process rewards, including our verifier that does not rely on gold evidence.
| (a) Reward configurations | (b) Verification mechanisms | ||
| Configuration | Acc. (%) | Configuration | Acc. (%) |
| ReAct | 46.78 | GRPO | 57.73 |
| Process only | 47.27 | + LLM-as-a-judge | 58.28 |
| Outcome only | 57.73 | + plug-and-play verification (ours) | 60.18 |
| Outcome + process | 61.40 | + Gold evidence (upper bound) | 61.40 |
Table 2. Ablation studies on HotpotQA with Qwen3.5-4B. (a) We compare process rewards, outcome rewards, and their combination, with ReAct as a reference. Process rewards in (a) use verification against gold evidence. (b) Each verification mechanism adds process rewards to the outcome-only GRPO baseline.
Comparison of Verification Mechanisms. We compare three process verification mechanisms. As shown in Table 2, our verifier achieves an accuracy of 60.18, outperforming LLM-as-a-judge. Gold-evidence matching achieves the accuracy at 61.40. These results suggest that deterministic certificate verification can provide useful process rewards without gold evidence or an LLM judge.
Effectiveness Across Policy Optimization Algorithms
Finally, we evaluated our plug-and-play verifier framework with both GRPO and PPO on HotpotQA using Qwen3.5-4B to examine whether the benefits of process verification extended beyond GRPO. For each algorithm, we compared outcome-only training with training augmented by process rewards. We used the same verifier and reward schedule across the two algorithms, while keeping initialization, task interface, training budget, and evaluation settings fixed.
As shown in Table 3, we observed that incorporating process verification improved accuracy from 57.73 to 60.18 under GRPO and from 55.10 to 57.60 under PPO. We obtained these gains using the same verifier module for both algorithms. Our results indicate that our plug-and-play verifier decoupling process verification from policy optimization enabled the verifier to be reused across GRPO and PPO.
| Algorithm | Outcome-only | + Verification |
| GRPO | 57.73 | 60.18 (+2.45) |
| PPO | 55.10 | 57.60 (+2.50) |
Table 3. Effect of process verification under GRPO and PPO on HotpotQA with Qwen3.5-4B. We report outcome accuracy (%). “+ Verification” adds process rewards to outcome-only training. Values in parentheses denote absolute accuracy gains over outcome-only training within each algorithm, in percentage points.
Conclusion
We propose a plug-and-play verifier framework that enables process-supervised RL for language model agents by decoupling process verification from reward composition and policy optimization. We note that our framework shares a similar idea with recent works. Yuan et al. [52] convert symbolic and algorithmic oracles into turn-level process rewards. Xu et al. [43] attribute verified evidence to the search actions that first exposed it. However, our framework is unique in combining a unified contract of bounded weighted sums of verification scores assigned to source steps, deterministic verifiers that require neither gold evidence nor LLM judges, and group-shared reward weight. Our results illustrate the potential of explicit process verification as a modular complement to outcome rewards in agentic RL.
Limitations and future work. We evaluate our proposed verifiers by mostly checking per-turn reasoning steps, such as evidence grounding and execution replay, rather than the multiple-turn semantic correctness of intermediate steps. We will incorporate process reward models [23, 32] and the most recent calibrated decision models such as Jev for future expansion of the plug-and-play verifier. Second, our agents operate within short interaction horizons on public agentic benchmarks. We will extend our framework to much longer-horizon environments [57, 18] for production agentic post-training. Beyond text and image reasoning, we will explore extending our plug-and-play verifier framework to multimodal video generation models [5]. Finally, we will explore verification-guided RSI, where agents iteratively refine their reasoning strategies and tool-use workflows.
Disclose Use of LLMs
We used LLMs, such as Claude and GPT, for initial idea brainstorming and later to check grammar and spelling and improve the clarity of our manuscript’s language. All scientific contents, including figures, tables, experimental design, analysis, and interpretations, are entirely the original work.
Ethics Statement
Our research does not involve any potential conflicts of interest or sponsorship issues. We have carefully considered and addressed issues related to discrimination, bias, and fairness in our methodology. This study does not involve any privacy or security concerns, fully complies with legal requirements, and upholds the highest standards of research integrity. All experimental procedures and data processing follow established ethical guidelines for machine learning research.
Reproducibility Statement
To support reproducibility, we describe the proposed plug-and-play verification framework and decoupled reward composition. The experimental settings and evaluation protocol are presented in § 4.1. Dataset specifications are provided in Appendix § 9, reward construction and policy optimization in Appendix § 10, training hyperparameters in Appendix § 11, and representative verification inputs and checks in Appendix D. Code is available at https://github.com/momo1443/plug-and-play-verification.
Training and Evaluation Benchmarks
We conduct training and evaluation using benchmarks from four domains: multi-hop question answering, code generation, multimodal reasoning, and mathematical reasoning.
HotpotQA [48]: a large-scale multi-hop question answering dataset constructed from Wikipedia. The dataset contains 90,447 training and 7,405 validation examples, with questions requiring reasoning over information distributed across multiple documents. Each example is additionally annotated with sentence-level supporting facts,
enabling explicit evaluation of evidence identification and multi-hop reasoning. We use the validation examples of HotpotQA for evaluation to measure the generalization of multi-hop question answering.
TACO [21]: a corpus consisting of 25,443 competitive programming problems from the training split of the TACO dataset. Each problem is paired with a natural-language problem statement, input-output specifications, reference solutions, and executable test cases, spanning algorithmic topics and difficulty levels for code generation and reasoning.
LiveCodeBench [17]: a 1,055-problem code generation benchmark consisting of competitive programming problems collected from LeetCode, AtCoder, and Codeforces. We use the
code_generation_litesetting withrelease_v6, covering problems released between May 2023 and April 2025. Its temporally recent problems and hidden test cases make it suitable for evaluating coding ability. We use LiveCodeBench for evaluation to measure the generalization of code generation.Vision-R1-RL [16]: a multimodal mathematical reasoning dataset containing approximately 11.3K image-text problems designed for reinforcement learning with verifiable rewards. The dataset consists primarily of visually grounded mathematical and geometric reasoning problems with deterministic final answers.
MATH-Vision [36]: a multimodal mathematical reasoning benchmark consisting of 3,040 visual mathematics problems collected from real-world mathematical competitions. The benchmark spans 16 mathematical disciplines and 5 difficulty levels, covering diverse problems that require both visual understanding and reasoning. We use MATH-Vision for evaluation to measure the generalization of multimodal reasoning.
DeepScaleR-Preview-Dataset [26]: a curated training corpus of approximately 40,000 unique mathematics problem–answer pairs compiled from AIME problems (1984–2023), AMC problems (before 2023), the Omni-MATH dataset, and the Still dataset, spanning difficulty levels from high-school competition to advanced mathematical reasoning.
AIME 2026 [7]: a benchmark comprising 30 problems from the 2026 AIME I and AIME II examinations. We use this recent competition benchmark to assess the generalization of mathematical reasoning to challenging competition-level problems.
The downloading link for each dataset can be accessed from the Hugging Face website:
HotpotQA:
https://huggingface.co/datasets/hotpotqa/hotpot_qaTACO:
https://huggingface.co/datasets/BAAI/TACOLiveCodeBench:
https://huggingface.co/datasets/livecodebench/code_generation_liteVision-R1-RL:
https://huggingface.co/datasets/Osilly/Vision-R1-rlMATH-Vision:
https://huggingface.co/datasets/MathLLMs/MathVisionDeepScaleR-Preview-Dataset:
https://huggingface.co/datasets/agentica-org/DeepScaleR-Preview-DatasetAIME 2026:
https://huggingface.co/datasets/MathArena/aime_2026
Policy Optimization Details
We describe step-level reward construction and the advantage estimators used by GRPO and PPO. Both algorithms share the same actor objective and policy-token mask, but use different advantage estimators.
Step-level reward construction. Consider a training update and prompt group for input . The rollout policy generates trajectories , where indexes the trajectories within the prompt group. Let denote the number of interaction steps in , its outcome reward, and its step-level process reward. Following the reward composition in Eq. (6), we define
Here denotes the indicator function, is the number of outcome-only warmup updates, and is a reproducibly sampled weighted coefficient shared by all rollouts within prompt group . We omit the fixed update and group in for readability. The outcome reward is assigned only to the terminal interaction step, while process rewards are assigned to their source interaction steps. Because , summing these composed step-level rewards yields the trajectory-level reward . For incomplete trajectories, , while independently verified process rewards are retained.
Group-relative advantages. For GRPO, we use undiscounted returns:
Each reward contributes to the returns of its assigned step and all preceding steps, regardless of when verification is performed. The terminal outcome reward is therefore included in each valid step’s return, although it is assigned only once to the terminal step.
We normalize returns across trajectories from the same prompt that reach interaction step :
where and are the mean and sample standard deviation of the corresponding returns, respectively. For a group containing only one trajectory at step , we set and to retain its return. Outcome and process rewards are combined before computing returns and advantages, rather than normalized separately. The resulting advantages remain fixed during policy optimization.
Last-mile signal under outcome rewards. Consider trajectories within the same prompt group that share outcome reward and reward weight . Let denote the remaining process return. The composed return is . For groups with more than one trajectory, substituting this into the group-relative advantage gives
where and are the mean and sample standard deviation of the process returns in the group. This recovers Eq. (7). The shared outcome reward cancels under group centering, allowing differences in verified process returns to distinguish trajectories with identical outcomes. When , identical outcome rewards instead produce zero group-relative advantages.
Policy-token masking and actor optimization. We assign each step-level advantage to all policy-generated tokens within that step, including reasoning text, tool-call arguments, and final-answer tokens. Prompt tokens and environment observations are excluded from the actor and KL losses, while observations remain available as context for subsequent policy generations.
For a policy token with context , we define the importance ratio
Using its assigned step advantage , the clipped surrogate is
We additionally apply dual clipping with : when , and otherwise. Let denote averaging first over valid policy tokens within each interaction step and then over all valid interaction steps in the sampled prompt group. We maximize
where , and controls KL regularization. The reference policy remains fixed during training. KL regularization is applied only to the actor objective, not to advantage estimation.
Integration with PPO. PPO uses the same verifier outputs, composed step rewards, policy-token mask, and clipped actor objective as GRPO. The difference lies in advantage estimation: PPO learns a critic and uses generalized advantage estimation (GAE) instead of group-relative normalization.
With and zero terminal bootstrapping, step-level GAE simplifies to
where denotes the interaction history before policy generation at step in trajectory , and is the value function parameterized by the pre-update critic parameters . We standardize these advantages across all valid interaction steps in the rollout batch:
where and denote the batch mean and sample standard deviation, respectively. These normalized advantages replace GRPO’s group-relative advantages in the shared actor objective.
The critic target is the undiscounted return:
We train the critic using a clipped squared-error value loss with clipping range 0.5, evaluated once per interaction-step starting state. Actor and critic parameters are updated separately, without a critic-only warmup.
In the Qwen3.5-4B HotpotQA comparison, the actor and critic learning rates are and , respectively.
Training Details
Training infrastructure. All experiments are conducted on a single Kubernetes (K8s) node with NVIDIA A40 GPUs (48 GB each). We use PyTorch with Fully Sharded Data Parallelism (FSDP) for distributed training and vLLM in server mode for rollout generation. A run of 500 training updates takes approximately 80 hours for Qwen3.5-4B and 94 hours for Qwen3.5-9B.
Training data and rollout budgets. We select the first 10,000 examples from each prepared training split and train for 500 updates per domain. Both GRPO configurations sample four trajectories per prompt. The global prompt batch size and actor mini-batch size are 20 prompts for Qwen3.5-4B and 8 prompts for Qwen3.5-9B. Consequently, a 500-update run samples 10,000 and 4,000 prompt instances, respectively, before expanding each prompt into four trajectories.
| Hyperparameter | Value |
| Optimizer | AdamW |
| Learning rate | |
| Training updates | 500 |
| LR schedule | Constant; no warmup |
| Global prompt batch size | 20 |
| Trajectories per prompt | 4 |
| Mini-batch size | 20 prompts |
| Optimization epochs per rollout | 1 |
| Initial prompt limit | 2048 tokens |
| History / context limit | 8192 tokens |
| Maximum interaction steps | 5 |
| Sampling temperature | 1.0 |
| Top- / Top- | 1.0 / -1 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| KL coefficient | 0.001 |
| Precision | bfloat16 |
| Random seed | 42 |
Table 4. Hyperparameters for Qwen3.5-4B GRPO training on DeepScaleR.
| Hyperparameter | Value |
| Fine-tuning method | LoRA (, ) |
| Optimizer | AdamW |
| Learning rate | |
| Training updates | 500 |
| LR schedule | Constant; no warmup |
| Global prompt batch size | 8 |
| Trajectories per prompt | 4 |
| Mini-batch size | 8 prompts |
| Optimization epochs per rollout | 1 |
| Initial prompt limit | 2048 tokens |
| History / context limit | 8192 tokens |
| Maximum interaction steps | 5 |
| Sampling temperature | 1.0 |
| Top- / Top- | 1.0 / -1 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| KL coefficient | 0.001 |
| Precision | bfloat16 |
| LoRA target modules | q_proj, k_proj, v_proj, o_proj, up_proj, gate_proj, down_proj |
| Random seed | 42 |
Table 5. Hyperparameters for Qwen3.5-9B LoRA GRPO training on DeepScaleR.
| Hyperparameter | Value |
| Optimizer | AdamW |
| Learning rate | |
| Training updates | 500 |
| LR schedule | Constant; no warmup |
| Global prompt batch size | 20 |
| Trajectories per prompt | 4 |
| Mini-batch size | 20 prompts |
| Optimization epochs per rollout | 1 |
| Max prompt length | 8192 tokens |
| Max completion length | 1024 tokens |
| Maximum interaction steps | 4 |
| Sampling temperature | 1.0 |
| Top- / Top- | 1.0 / -1 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| KL coefficient | 0.001 |
| Precision | bfloat16 |
| Random seed | 42 |
Table 6. Hyperparameters for Qwen3.5-4B GRPO training on HotpotQA.
| Hyperparameter | Value |
| Fine-tuning method | LoRA (, ) |
| Optimizer | AdamW |
| Learning rate | |
| Training updates | 500 |
| LR schedule | Constant; no warmup |
| Global prompt batch size | 8 |
| Trajectories per prompt | 4 |
| Mini-batch size | 8 prompts |
| Optimization epochs per rollout | 1 |
| Max prompt length | 8192 tokens |
| Max completion length | 1024 tokens |
| Maximum interaction steps | 4 |
| Sampling temperature | 1.0 |
| Top- / Top- | 1.0 / -1 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| KL coefficient | 0.001 |
| Precision | bfloat16 |
| LoRA target modules | q_proj, k_proj, v_proj, o_proj, up_proj, gate_proj, down_proj |
| Random seed | 42 |
Table 7. Hyperparameters for Qwen3.5-9B LoRA GRPO training on HotpotQA.
| Hyperparameter | Value |
| Optimizer | AdamW |
| Learning rate | |
| Training updates | 500 |
| LR schedule | Constant; no warmup |
| Global prompt batch size | 20 |
| Trajectories per prompt | 4 |
| Mini-batch size | 20 prompts |
| Optimization epochs per rollout | 1 |
| Max prompt length | 8192 tokens |
| Max completion length | 2048 tokens |
| Maximum interaction steps | 5 |
| Sampling temperature | 1.0 |
| Top- / Top- | 1.0 / -1 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| KL coefficient | 0.001 |
| Precision | bfloat16 |
| Random seed | 42 |
Table 8. Hyperparameters for Qwen3.5-4B GRPO training on TACO.
| Hyperparameter | Value |
| Fine-tuning method | LoRA (, ) |
| Optimizer | AdamW |
| Learning rate | |
| Training updates | 500 |
| LR schedule | Constant; no warmup |
| Global prompt batch size | 8 |
| Trajectories per prompt | 4 |
| Mini-batch size | 8 prompts |
| Optimization epochs per rollout | 1 |
| Max prompt length | 8192 tokens |
| Max completion length | 2048 tokens |
| Maximum interaction steps | 5 |
| Sampling temperature | 1.0 |
| Top- / Top- | 1.0 / -1 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| KL coefficient | 0.001 |
| Precision | bfloat16 |
| LoRA target modules | q_proj, k_proj, v_proj, o_proj, up_proj, gate_proj, down_proj |
| Random seed | 42 |
Table 9. Hyperparameters for Qwen3.5-9B LoRA GRPO training on TACO.
| Hyperparameter | Value |
| Optimizer | AdamW |
| Learning rate | |
| Training updates | 500 |
| LR schedule | Constant; no warmup |
| Global prompt batch size | 20 |
| Trajectories per prompt | 4 |
| Mini-batch size | 20 prompts |
| Optimization epochs per rollout | 1 |
| Max prompt length | 8192 tokens |
| Max completion length | 2048 tokens |
| Maximum interaction steps | 3 |
| Sampling temperature | 1.0 |
| Top- / Top- | 1.0 / -1 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| KL coefficient | 0.001 |
| Precision | bfloat16 |
| Random seed | 42 |
Table 10. Hyperparameters for Qwen3.5-4B GRPO training on Vision-R1.
| Hyperparameter | Value |
| Fine-tuning method | LoRA (, ) |
| Optimizer | AdamW |
| Learning rate | |
| Training updates | 500 |
| LR schedule | Constant; no warmup |
| Global prompt batch size | 8 |
| Trajectories per prompt | 4 |
| Mini-batch size | 8 prompts |
| Optimization epochs per rollout | 1 |
| Max prompt length | 8192 tokens |
| Max completion length | 2048 tokens |
| Maximum interaction steps | 3 |
| Sampling temperature | 1.0 |
| Top- / Top- | 1.0 / -1 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| KL coefficient | 0.001 |
| Precision | bfloat16 |
| LoRA target modules | q_proj, k_proj, v_proj, o_proj, up_proj, gate_proj, down_proj |
| Random seed | 42 |
Table 11. Hyperparameters for Qwen3.5-9B LoRA GRPO training on Vision-R1.
We use the same experimental setup, verifier, and reward composition for PPO and GRPO. PPO instead uses step-level GAE and a separately optimized critic initialized from the same pretrained checkpoint as the actor. The outcome-only PPO baseline fixes , whereas verification PPO follows the same group-shared reward weight schedule as verification GRPO. Validation uses outcome-only rewards, and no critic-only warmup is performed. Both PPO variants use the hyperparameters in Table 12.
| Hyperparameter | Value |
| Actor / critic optimizer | AdamW |
| Actor learning rate | |
| Critic learning rate | |
| Actor / critic LR schedule | Constant; no LR warmup |
| Training updates | 500 |
| Global prompt batch size | 20 |
| Trajectories per prompt | 4 |
| Max prompt length | 8192 tokens |
| Max completion length | 1024 tokens |
| Top- / Top- | 1.0 / −1 |
| Discount factor / GAE parameter | 1.0 / 1.0 |
| Policy clipping threshold | 0.2 |
| Dual-clip constant | 3.0 |
| Value clipping range | 0.5 |
| Actor / critic weight decay | 0.01 |
| Actor / critic gradient clipping | 1.0 |
| Actor-loss KL coefficient | 0.001 |
| Random seed | 42 |
Table 12. Hyperparameters for Qwen3.5-4B PPO training on HotpotQA.
| Parameter | Value |
| Sampling temperature | 0.6 |
| Top- / Top- | 1.0 / -1 |
| Precision / Random seed | bfloat16 / 42 |
| Inference engine | vLLM |
Table 13. Evaluation settings shared by all benchmarks.
| Benchmark | Metric | Thinking mode | Steps | Per-step cap | Total budget | Prompt limit | Context |
| AIME 2025 | avg@1 | On | 5 | 2,048 | 10,240 | 2,048 | 13,312 |
| AIME 2026 | avg@4 | On | 5 | 2,048 | 10,240 | 2,048 | 13,312 |
| HotpotQA | avg@1 | Off | 4 | 1,024 | 4,096 | 8,192 | 12,288 |
| LiveCodeBench v6 | avg@4 | Off | 5 | 2,048 | 10,240 | 8,192 | 12,288 |
| MATH-Vision | avg@1 | Off | 3 | 2,048 | 6,144 | 8,192 | 12,288 |
Table 14. Evaluation protocol across all benchmarks. avg@ denotes accuracy averaged over independently sampled trajectories per problem. The same sampling protocol is applied to all compared methods.
Representative Verification Inputs and Checks
We present representative examples across four reasoning domains to illustrate how intermediate artifacts are verified independently of final-answer correctness.
Multi-hop Question Answering: HotpotQA
Question: What 2014 American 3D computer-animated musical fantasy adventure comedy film features a song recorded by British singer Leah McFall?
Reference answer: The Book of Life.
Illustrative Agent Trajectory. The following trajectory is constructed from the recorded question and retrieved evidence to illustrate the local verification process.
Step 1: Initial Search. The agent searches for “2014 American 3D computer-animated musical fantasy adventure comedy film song recorded by British singer Leah McFall”. The retrieved results include passage:9904 (The Book of Life (2014 film)) and passage:9898 (Home (Leah McFall song)). The latter contains the supporting sentence:
“This song also featured in the latest film, "The Book of Life".”
We denote this sentence by . As the initial search requires no certificate, this step is excluded from verification and normalization.
Step 2: Evidence-Grounded Search. The agent searches for “The Book of Life Leah McFall” and provides a certificate containing:
Source:
passage:9898;Evidence: Sentence ;
Target: "The Book of Life".
Step 3: Final Answer. The agent submits "The Book of Life" with a final certificate citing passage:9898, sentence , and the answer span "The Book of Life".
Verification Results. At Step 2, source-span grounding succeeds because appears in the cited passage, while action-certificate matching succeeds because the certified target is consistent with the search action. At Step 3, both checks also pass: the cited evidence is grounded in the source, and the certified answer span matches the submitted answer. Therefore, all applicable local checks pass, yielding under Eq. (8).
Code Generation: LiveCodeBench
Question: Given an integer array, find the maximum sum of a nonempty contiguous segment whose adjacent elements have opposite parity.
Example input–output test cases: ; ; .
This is a constructed illustration, not a logged training trajectory. The solution uses dynamic programming to maintain the maximum valid subarray sum, extending a segment only when adjacent elements have opposite parity.
best = current = a[0]
for i in range(1, len(a)):
if a[i] % 2 != a[i - 1] % 2:
current = max(a[i], current + a[i])
else:
current = a[i]
best = max(best, current)
Output: best. The algorithm runs in time and uses auxiliary space.
Verification inputs and local checks. On these three developer tests, the code returns 15, 17, and , respectively. All three tests pass, yielding passed=3, total=3, and all-passed=true. The recorded execution identifies the immutable code artifact and its digest, the developer-test suite and its digest, and the execution-result digest. The code verifier replays the recorded execution and checks whether the replayed outputs match the recorded execution results. The submission certificate identifies the final code artifact and its digest and cites this execution record. The verifier checks recorded execution lineage and replay consistency independently of final submission, assigning verified process rewards to their source execution steps. Submission-certificate checks are applied only when a final code artifact is submitted.
Multimodal Reasoning: MATH-Vision
Question: Using the balanced scale, determine how many plums have the same weight as one apple.
Reference answer: 3.
This is a constructed illustration, not a logged training trajectory. The left pan contains three apples and two plums, and the right pan contains one apple and eight plums. Let and denote the weights of an apple and a plum, respectively. The balance gives
Thus, one apple weighs as much as three plums.
Final answer: 3.
Verification inputs and local checks. For the original image, the left- and right-pan crop boxes are and , respectively, in (left, top, right, bottom) pixel coordinates. The visual certificate cites both crop identifiers and includes the final answer 3 in its claim. The verifier checks crop provenance and replay consistency against the original image, together with string-level coupling between the submitted answer and the certificate claim.
Mathematical Reasoning: AIME 2026
Question: An integer arithmetic progression starts at 4 and includes 24 and 34. Sum the tenth terms over all progressions satisfying these conditions.
Reference answer: 178.
This constructed three-step illustration fits within the five-step limit; it is not a logged training trajectory.
Step 1: Constraining the common difference. Let be the common difference. Since all terms are integers and the progression reaches values above 4, is a positive integer. The required terms give
Hence, divides both 20 and 30.
Step 2: Enumerating the possibilities. The admissible common differences are 1, 2, 5, and 10, with
Step 3: Final submission. Each progression has tenth term , so their sum is
Final answer: 178.
Verification inputs and local checks. Excluding the final-answer span, the four numeric-equality artifacts are grouped by source step:
Step 1: 24 - 4 = 20; 34 - 4 = 30 Step 2: 1 + 2 + 5 + 10 = 18 Step 3: 4 * 4 + 9 * 18 = 178
All four equalities hold, with source-step indices (1, 1, 2, 3). These checks verify arithmetic only, not the divisibility argument, completeness of enumeration, or tenth-term formula.
References
- [1]Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. Why do multi-agent llm systems fail? Advances in Neural Information Processing Systems, 38, 2026.arxiv.org/abs/2503.13657
- [2]Jiajun Chai, Guojun Yin, Zekun Xu, Chuhuai Yue, Yi Jia, Siyu Xia, Xiaohan Wang, Jiwen Jiang, Xiaoguang Li, Chengqi Dong, et al. Rlfactory: A plug-and-play reinforcement learning post-training framework for llm multi-turn tool-use. arXiv preprint arXiv:2509.06980, 2025.arxiv.org/abs/2509.06980
- [3]Mingyue Cheng, Shuo Yu, Daoyu Wang, Qingchuan Li, Xiaoyu Tao, Jie Ouyang, Yucong Luo, Yitong Zhou, Qi Liu, and Enhong Chen. Agent-rl: A unified and modular framework for agentic reinforcement learning. arXiv preprint arXiv:2511.14460, 2025.
- [4]Yu Cui, Hang Fu, Haibin Zhang, Licheng Wang, and Cong Zuo. Free-mad: Consensus-free multi-agent debate. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 31977–31997, 2026a.arxiv.org/abs/2509.11035
- [5]Zijun Cui, Xiulong Liu, Hao Fang, Mingwei Xu, Jiageng Liu, Zexin Xu, Weiguo Pian, Shijian Deng, Feiyu Du, Chenming Ge, et al. Do joint audio-video generation models understand physics? arXiv preprint arXiv:2605.07061, 2026b.arxiv.org/abs/2605.07061
- [6]Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx. Agent-rlvr: Training software engineering agents via guidance and environment rewards. arXiv preprint arXiv:2506.11425, 2025.arxiv.org/abs/2506.11425
- [7]Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger, Kári Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms. 2026. URL https://arxiv.org/abs/2605.00674.
- [8]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023.arxiv.org/abs/2305.14325
- [9]Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. In International Conference on Learning Representations, volume 2026, pp. 37909–37926, 2026a.arxiv.org/abs/2504.11536
- [10]Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. Advances in Neural Information Processing Systems, 38:46375–46408, 2026b.arxiv.org/abs/2505.10978
- [11]Shubham Gandhi, Saurabh Goyal, Kiran Kate, and Yara Rizk. Draco: Fine-grained credit assignment with dynamic rubrics for long-horizon agent training. arXiv preprint arXiv:2609.04094, 2026.arxiv.org/abs/2609.04094
- [12]Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Nan Duan, Weizhu Chen, et al. Critic: Large language models can self-correct with tool-interactive critiquing. In International Conference on Learning Representations, volume 2024, pp. 57734–57811, 2024.arxiv.org/abs/2305.11738
- [13]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
- [15]Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Steven Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations, volume 2024, pp. 23247–23275, 2024.arxiv.org/abs/2308.00352
- [16]Wenxuan Huang, Bohan Jia, Shaosheng Cao, Zheyu Ye, Zhe Xu, Yao Hu, Shaohui Lin, et al. Vision-r1: Incentivizing reasoning capability in multimodal large language models. In International Conference on Learning Representations, volume 2026, pp. 63794–63812, 2026.arxiv.org/abs/2503.06749
- [17]Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, volume 2025, pp. 58791–58831, 2025.arxiv.org/abs/2403.07974
- [18]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world gitbug issues? In International Conference on Learning Representations, volume 2024, pp. 54107–54157, 2024.arxiv.org/abs/2310.06770
- [19]Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025.arxiv.org/abs/2503.09516
- [20]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp. 611–626, 2023.arxiv.org/abs/2309.06180
- [21]Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhibong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852, 2023.arxiv.org/abs/2312.14852
- [22]Zhichong Li, Lingjie Jiang, Yulan Hu, Xingchen Zeng, Yixia Li, Xiangwen Zhang, Guanhua Chen, Zheng Pan, Xin Li, and Yong Liu. No more stale feedback: Co-evolving critics for open-world agent learning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12643–12660, 2026.
- [23]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, pp. 39578–39601, 2024.arxiv.org/abs/2305.20050
- [24]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, volume 2024, pp. 52989–53046, 2024.arxiv.org/abs/2308.03688
- [25]Yijun Lu, Rui Ye, Jiajun Wang, Yuwen Du, Tian Jin, Songhua Liu, and Siheng Chen. Abseeker: Training long-horizon search agents via answer-backtracked credit assignment. arXiv preprint arXiv:2608.05102, 2026.arxiv.org/abs/2608.05102
- [26]Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl. Notion Blog, 3(5), 2025.
- [27]Massimiliano Pronesti, Anja Belz, and Yufang Hou. Beyond outcome verification: Verifiable process reward models for structured reasoning. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 32187–32202, 2026.
- [28]Qwen Team. Qwen3.5: Towards native multimodal agents. February 2026. URL https://qwen.ai/blog?id=qwen3.5.
- [29]Jiacheng Sang, Mengyuan Li, Sanxing Chen, Yukun Huang, Yu Feng, and Bhuwan Dhingra. Searchatlas: Analyzing agentic search strategies via evidential query graphs. arXiv preprint arXiv:2609.10901, 2026.arxiv.org/abs/2609.10901
- [30]Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539–68551, 2023.arxiv.org/abs/2302.04761
- [31]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
- [32]Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning. In International Conference on Learning Representations, volume 2025, pp. 60808–60838, 2025.arxiv.org/abs/2410.08146
- [33]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [34]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Languague agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023.arxiv.org/abs/2303.11366
- [35]Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, and Zhenzhe Ying. Information gain-based policy optimization: A simple and effective approach for multi-turn llm agents. arXiv e-prints, pp. arXiv–2510, 2025a.arxiv.org/abs/2510.14967
- [36]Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169, 2024a.arxiv.org/abs/2402.14804
- [37]Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426–9439, 2024b.
- [39]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.arxiv.org/abs/2203.11171
- [40]Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025b.arxiv.org/abs/2504.20073
- [42]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.arxiv.org/abs/2201.11903
- [43]Ke Xu, Han Xu, Xinran Chen, Yuqian Wang, Zhixuan Li, Xiaojian Liu, Changwo Wu, Jianqiang Xia, and Yuchen Li. Stamp: Provenance-guided credit assignment for deep search agents. arXiv preprint arXiv:2607.11172, 2026.arxiv.org/abs/2607.11172
- [45]John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024.arxiv.org/abs/2405.15793
- [46]Kaibing Yang, Guangfeng Cai, Shengtian Yang, Shuo He, Yu Li, Mengyi Liu, Pengwei Chen, Jun Xu, and Lei Feng. Progress-conditioned group policy optimization for long-horizon agentic tasks. arXiv preprint arXiv:2607.22724, 2026a.
- [47]Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, et al. Skilllopt: Executive strategy for self-evolving agent skills. arXiv preprint arXiv:2605.23904, 2026b.arxiv.org/abs/2605.23904
- [48]Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2369–2380, 2018.arxiv.org/abs/1809.09600
- [49]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.arxiv.org/abs/2210.03629
- [50]Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822, 2023.arxiv.org/abs/2305.10601
- [51]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026.arxiv.org/abs/2503.14476
- [52]Huining Yuan, Zelai Xu, Huajie Wang, Xiangmin Yi, Jiaxuan Gao, Xiao-Ping Zhang, Yu Wang, Chao Yu, and Yi Wu. Verifiable process rewards for agentic reasoning. arXiv preprint arXiv:2605.10325, 2026.arxiv.org/abs/2605.10325
- [53]Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735–64772, 2024.arxiv.org/abs/2406.03816
- [54]Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 10495–10516, 2025.
- [55]Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 414–431, 2025.arxiv.org/abs/2504.03160
- [56]Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406, 2023.arxiv.org/abs/2310.04406
- [57]Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abhishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, pp. 15585–15606, 2024.arxiv.org/abs/2307.13854
- [58]Rongzhi Zhu, Xiangyu Liu, Yi Liu, Shuo Zhang, Ruirui Zhang, Rui Wu, Tao Jiang, Zequn Sun, Wenhao Xu, and Wei Hu. Dense process supervision for search agents via fact utility estimation. arXiv preprint arXiv:2609.00833, 2026.arxiv.org/abs/2609.00833


