WorksComputer Science
GPRM: Global Perspective Process Reward Model
Abstract
Process Reward Models (PRMs) are widely used for step-level evaluation of multi-step reasoning in large language models, but existing methods are often limited by weak historical dependency modeling and insufficient global awareness. They typically fail to leverage prior step evaluations for coherent judgment and lack visibility into subsequent reasoning, leading to suboptimal assessment of intermediate steps. To address this issue, we propose the Global Perspective Process Reward Model (GPRM), which jointly models the problem, previous reasoning steps and their evaluations, the current step, and future steps to enable history-aware and globally informed process evaluation. We further introduce a structured evaluation framework along four dimensions—look back, look ahead, self-check, and goal alignment—to assess prior consistency, future plausibility, step validity, and task alignment. Based on this framework, we construct datasets for supervised fine-tuning (SFT) and reinforcement learning (GRPO). Experiments show that, despite using less training data and smaller model sizes, GPRM outperforms strong baseline models by 7.8 and 5.4 points on PRMBench and ProcessBench, respectively. GPRM also generalizes well across tasks, surpassing the GPT-4.1 baseline on Agent Error Bench, improving strong baselines by around 4 points under test-time scaling (TTS), and achieving competitive performance on PPE.
Introduction
Main contact: [email protected]
Equal contribution. † Corresponding author.
Introduction
Large language models (LLMs) have achieved remarkable progress on multi-step tasks such as mathematical reasoning, code generation, and complex decision making, with further gains from chain-of-thought (CoT) prompting and test-time search [14] (Yang et al. 2024). However, LLMs still frequently make mistakes in intermediate reasoning steps, including logical errors, calculation mistakes, missing conditions, and deviations from the original objective. Compared with errors in the final answer, these process-level errors are often more subtle and more likely to be propagated and amplified by later steps, ultimately causing the entire reasoning trajectory to drift away from the correct solution. To address this issue, Process Reward Models (PRMs) have been introduced to provide fine-grained evaluation of intermediate steps. Unlike Outcome Reward Models (ORMs), which focus only on the final answer, PRMs can localize where errors occur and provide more precise supervision signals for model training (Xiong et al., 2025; Ye et al., 2025), inference-time search, and error correction. As a result, PRMs have become a key component for improving the robustness of multi-step reasoning and have been widely used in mathematical verification, test-time search [9], and process supervision [15, 19].
Despite recent progress, existing PRMs still suffer from two major limitations. First, they lack historical judgment inheritance: most methods evaluate the current step based only on the question, preceding step texts, and the current step itself, without explicitly incorporating judgments made on earlier steps. As a result, although the model can observe the history (Zhao et al., 2025), it often fails to consistently leverage previously established diagnoses across long reasoning chains. Second, they lack a global perspective (Zou et al., 2025): existing PRMs typically assess each step from local context alone, without considering how subsequent reasoning unfolds. However, when performing complex reasoning (e.g., test-time compute scaling), large language models (LLMs) often utilize reflection, backtracking, or experience ”aha-moments”. During these ”aha-moments”, seemingly flawed or highly exploratory steps are actually necessary turning points that eventually lead to correct reasoning. A local, linear PRM mistakes such a detour for the first error, whereas a future-step perspective can see that the step is taken up and validated by later reasoning. Figure 2 quantifies both limitations.

Figure 2. Two limitations of existing PRMs and the effect of adding the future-step perspective and historical judgment inheritance. (a) Without the future-step perspective, only 49.2% of cases locate the first error correctly (37.6% mis-localize, 13.2% miss); adding future steps cuts mis-localization from 37.6% to 15.7%, largely because “aha-moment” detours are no longer mispenalized. (b) Without historical judgment inheritance, 45.1% find the first error but break the chain and 29.7% miss it entirely; inheriting prior judgments cuts the broken-chain rate from 45.1% to 35.5% by keeping a coherent view of all preceding steps.
To address these challenges, we propose the Global Perspective Process Reward Model (GPRM). As shown in Figure 1, GPRM evaluates a target step by jointly conditioning on the question, the preceding steps together with their historical judgments, the current step, and the future steps. Unlike conventional PRMs that score each step only from local context, this design explicitly models dependencies across the history–current–future structure, enabling both historical inheritance and global diagnosis. In addition, we introduce a unified generative four-dimensional diagnostic framework for process-level errors: Look-back, Look-ahead, Self-check, and Problem-alignment. These dimensions respectively examine consistency, justification from subsequent reasoning, intrinsic logical and computational validity, and alignment with the original task objective. Under this framework, step evaluation is formulated not as an isolated local classification problem, but as a structured diagnostic task over the full trajectory.
Figure 1. (a) Conventional PRMs judge the current step from local context only, without explicit inheritance of historical judgments or future-step perspective. (b) GPRM evaluates each step under a history–current–future view, combining historical judgments and future reasoning, and aggregates four diagnostic signals—Look-back, Look-ahead, Self-check, and Problem-alignment—into the final judgment.
To equip the model with these capabilities, we further design a training pipeline tailored to GPRM. In the supervised fine-tuning stage, we construct high-quality mathematical reasoning data under the four diagnostic dimensions and use a strong teacher model to generate step-level analyses, enabling the model to learn structured diagnosis and explanation. On top of this, we mine hard examples from PRM800K, focusing on low-confidence, long-horizon, and late-error cases, and build GRPO training instances to further improve robustness in challenging reasoning scenarios.
Extensive experiments demonstrate the effectiveness of GPRM in process supervision, cross-task generalization, and test-time search. On PRMBench and ProcessBench, GPRM surpasses mainstream GPT-4o and Gemini-2.0-thinking with a smaller model scale. On Agent Error Bench, it generalizes from mathematical reasoning to agent trajectory diagnosis without task-specific adaptation, and surpasses GPT-4.1. Under test-time search settings, GPRM further improves strong baselines. These results highlight the importance of global perspective and historical judgment inheritance for process-level reasoning evaluation. Our contributions can be summarized as follows:
Identifying Key Limitations. We identify two critical weaknesses in existing PRMs—lack of historical judgment inheritance and absence of global perspective—and demonstrate through quantitative analysis (Figure 2) that these lead to locally plausible yet globally incorrect judgments in long-chain reasoning.
GPRM Framework. We propose GPRM, which evaluates each step by jointly conditioning on
Figure 3. Overall training framework of GPRM. Stage I performs supervised fine-tuning on teacher-constructed step-analysis data with injected errors to learn four-dimensional diagnosis. Stage II introduces global context by completing future trajectories, constructing history-aware inputs, mining hard samples, and optimizing the model with GRPO.
the question, historical judgments , current step, and future trajectory (Figure 1). We introduce a four-dimensional diagnostic framework (Look-back, Look-ahead, Self-check, Problem-alignment) and a progressive training pipeline combining SFT with hard-example-based GRPO.
Superior Performance. GPRM-4B achieves 73.9 on PRMBench and 74.3 on ProcessBench, outperforming 7B PRMs and proprietary models including GPT-4o (66.8, 61.9) and o1-mini (68.8). GPRM generalizes to agent error localization (54.0%) and improves test-time search by +5.6 points under Greedy Guided Search@8.
Related Work
Process Reward Models. Process reward models (PRMs) provide step-level supervision for multi-step reasoning, complementing outcome reward models [1, 4, 9, 11]. High-quality supervision is costly, so methods such as Monte-Carlo estimation, rule-based construction, and automatic annotation have been developed [8, 10, 19], with later work improving PRMs via LLM-as-a-judge, preference optimization, retrieval, and dynamic verification (Zhang et al., 2025b).
Discriminative vs. Generative PRMs. Early PRMs predict a correctness label or scalar reward per step [7, 10, 13, 19] (Yang et al. 2024; Zou et al. 2025), underutilizing LLM reasoning. Generative PRMs instead analyze the step before judging: R-PRM [15], GenPRM (Zhao et al., 2025), and ThinkPRM [6] each reason over the current step, but from scratch and without inheriting prior judgments.
Global and Future-Step Perspective. Several works use future or global information, but collapse it into a scalar rather than reasoning over the future text: BiPRM (Zhang et al. (2026a)) re-scores each step against following steps in a right-to-left pass, value/advantage-style PRMs let a step’s score depend on future success probability from rollout (Luo et al. [10]; Wang et al. [19]), and CRM (Zhang et al. (2026b)) links step rewards to the final outcome via the chain rule of conditional probabilities. R-PRM analyzes consistency with preceding steps, yet inspects their content rather than prior judgments, and no existing generative PRM conditions on the future trajectory text. PRMs also serve as online verifiers in test-time search (Liu et al. [9]; She et al. [15]; Zhao et al. (2025)), where this richer view is especially valuable.
Method
Problem Formulation
We study the step-level evaluation problem in multi-step reasoning with large language models. Given a question and its corresponding reasoning trajectory , where denotes the -th reasoning step, our goal is to predict a correctness label for each step: , where indicates correctness and indicates an error.
Traditional PRMs judge the -th step as , a local decision on only the question and current prefix. Yet a step’s correctness depends both on prior reasoning (via intermediate conclusions) and on how it is used later (some steps only prove useful in hindsight). We thus formulate step-level evaluation as structured prediction over the full trajectory: given the question , the historical trajectory , the current step , and the subsequent trajectory , the Global Perspective Process Reward Model (GPRM) performs structured diagnosis and label prediction for each step using complete contextual information.
Global Perspective Process Reward Model
Global Input Design. For the -th step, we organize previous steps and their predicted judgments into a historical state , where for the first step. In practice, is serialized as a text sequence where each entry is formatted as “Step : [step text] → [CORRECT/INCORRECT]”, maintaining the temporal order. We denote the subsequent trajectory as if , and if . The unified input of GPRM at step is then .
This global input design enables GPRM to:
Historical Judgments (): Earlier errors may effect later steps through error spread.
Future Trajectory judgments (): Some steps can only be validated by examining their usage in subsequent reasoning.
Maintain Problem Context (): Ensure alignment with the original objective.
Four-Dimensional Structured Analysis. Given the input , the model outputs a structured analysis and the final predicted label: , where is the parameterized process evaluation model. When viewed as a probability distribution, we write ; in the reinforcement learning context, we use to denote the policy.
To improve stability and interpretability, we decompose the step analysis into four fixed dimensions: , where the superscripts denote:
F (Forward Consistency): checks if the current step is consistent with prior .
| Method | ALFWorld | WebShop | GAIA | Avg. | ||||
| S | S+M | S | S+M | S | S+M | S | S+M | |
| GPT-4.1-based baselines | ||||||||
| Direct Prompting | 28.0 | 14.0 | 30.0 | 6.0 | 26.0 | 10.0 | 28.0 | 10.0 |
| Brute Force | 10.0 | 5.0 | 8.0 | 0.0 | 18.0 | 8.0 | 12.0 | 4.3 |
| Binary Search | 20.0 | 6.0 | 14.0 | 8.0 | 22.0 | 10.0 | 18.7 | 8.0 |
| AgentDebug | 35.0 | 28.0 | 42.0 | 22.0 | 58.0 | 44.0 | 45.0 | 31.3 |
| ★ GPRM-4B | 38.0 | 30.0 | 44.0 | 24.0 | 60.0 | 46.0 | 47.0 | 33.0 |
| ★ GPRM-14B | 46.0 | 37.0 | 51.0 | 29.0 | 67.0 | 51.0 | 54.0 | 39.0 |
Table 2. Performance on Agent Error Bench. S: earliest error step localization. S+M: step + module identification. ★ denotes our models.
P (Posterior Verification): checks if the current step is reasonably used in subsequent .
G (Goal Alignment): determines if the current step still serves the problem .
S (Self Validity): verifies if the current step is logically and computationally correct.
The final step label is extracted from the generated analysis via a parsing function. Specifically, the model is trained to output judgment token at the end of : if ends with “[CORRECT]”, and if it ends with “[INCORRECT]”.
Recursive Evaluation Process. After completing the judgment for step , the current prediction is written back into the historical state: . Therefore, GPRM is no longer a local model that scores each step independently; instead, it is a global evaluation process that unfolds recursively along the entire reasoning trajectory.
Because GPRM scores a step only in light of its future trajectory , evaluating a candidate step in online search requires that the step first be extended into a complete trajectory so that is available. In two of our three inference regimes this incurs no extra generation cost, since is already contained in the given trajectory (offline diagnosis) or in the sampled completions (Best-of-); only the step-wise online search pays for completing candidates. We quantify this cost in § 5.4.
Progressive Training Strategy
We adopt a two-stage progressive training strategy. Both stages use the complete global input , but differ in the training objective.
Stage I (Supervised Fine-Tuning). The model learns step-level structured analysis by supervised imitation of the teacher’s four-dimensional diagnosis.
Stage II (Reinforcement Learning). The model is further optimized by GRPO with an outcome-based reward, refining its global judgments under the same input.
Through this two-stage training framework, we have enabled GPRM to possess a four-dimensional step correction capability with a global perspective and historical dependencies.
Supervised Fine-Tuning for Step Analysis
We construct an SFT dataset based on OpenMathInstruct-2, aligned with the four analysis dimensions by injecting targeted errors and collecting teacher-verified analyses.
Error Injection. Given a seed sample with , we define a set of fine-grained error types :
Calculation errors: arithmetic mistakes.
Logical errors: invalid inference steps.
Goal deviation: drifting away from the objective.
Inconsistency errors: contradicting prior conclusions.
For an error type and a target position , a teacher model injects an error into step via , and we propagate its effect to obtain a perturbed trajectory .
Teacher Annotation and Quality Control. Given , the teacher produces a structured analysis . If the teacher’s judgment is inconsistent with the gold label , we prompt for correction and re-analysis to ensure alignment with .
Dataset Construction. We build the SFT dataset over the complete global input , where are the teacher’s gold judgments on prior steps and is the remainder of the perturbed trajectory, so the model learns four-dimensional analysis already under historical judgments and the future view.
Training Objective. We optimize the autoregressive maximum likelihood objective
Reinforcement Learning for Global Evaluation
While SFT already teaches the four-dimensional analysis under the full global input, it is trained only on teacher-verified trajectories. We therefore construct a GRPO dataset based on PRM800K. Since PRM800K truncates each trajectory at the first erroneous step, the future trajectory is unavailable for posterior verification, so we complete it during data construction. We focus on hard cases with late-occurring errors and refine the model’s judgments via an outcome-based reward. We apply GRPO on the same complete global input , involving three key stages:
Future Trajectory Completion. For an erroneous step , we sample candidate continuations from a teacher model, retaining only those yielding incorrect final answers: .
Hard Sample Mining. We construct , where is generated by the SFT model. We then sample predictions and retain inputs where
at least one prediction disagrees with the gold label, forming the hard dataset .
GRPO Optimization. Let be the current policy and the reference policy (initialized from SFT). We employ an outcome-based reward , where extracts the final correctness label. The GRPO objective is:
where and . This optimization on enables robust step-level judgments under complete global context.
Experiments
Experimental Setup
Models. We train GPRM based on Qwen3-4B-Instruct and Qwen3-14B (scalability validation). Qwen3-235B-Instruct serves as the teacher model for error injection and annotation.
Benchmarks. We evaluate on process supervision (PRMBench, ProcessBench), cross-task generalization (Agent Error Bench, PPE), and test-time search (Best-of-8, Greedy Guided Search@8). Complete benchmark descriptions are in Appendix § 7.
Training Data and Settings. We adopt a two-stage training pipeline with standard autoregressive training. Details are in Appendices § 8 and § 9.
Main Results
Process Supervision. GPRM-4B achieves an overall score of 73.9 on PRMBench and an average F1 of 74.3 on ProcessBench, outperforming Qwen2.5-Math-PRM-7B (65.5, –), R-PRM-7B-DPO (66.8, 70.4), and proprietary models including GPT-4o (66.8, 61.9) and o1-mini (68.8, –). GPRM-14B further improves to 74.6 and 75.8, respectively.
Agent Error Bench. GPRM-4B achieves 47.0% on S and 33.0% on S+M, surpassing AgentDebug (45.0%, 31.3%). GPRM-14B further improves to 54.0% and 39.0%, demonstrating effective transfer to agent scenarios. We use Agent Error Bench to demonstrate cross-task generalization of the four-dimensional global framework; evaluating the end-to-end task success rate when GPRM is integrated as a verifier in interactive environments such as ALFWorld and GAIA is left to future work. Beyond agent and mathematical reasoning, GPRM also shows strong generalization to reasoning tasks (see Appendix § 8.§ 4 for PPE benchmark results).
Downstream Test-Time Search Results
Test-Time Search. Table 3 shows results under Best-of-8 and Greedy Guided Search@8 using Qwen2.5-7B-Instruct. Under Best-of-8, GPRM-4B achieves an average of 50.1, outperforming R-PRM-7B-DPO (49.6). GPRM-14B further improves to 51.0. The gain is moderate as Best-of- is limited by the quality of the candidate pool. Under Greedy Guided Search@8, both GPRM-4B and GPRM-14B achieve an average of 55.0, significantly outperforming R-PRM-7B-DPO (49.4).
| Setting / Model | AIME24 | AMC23 | MATH | Olymp. Bench | College Math | Minerva MATH | Avg. |
| Reference Baselines | |||||||
| pass@1 | 11.2 | 47.8 | 73.0 | 38.0 | 38.6 | 37.2 | 41.0 |
| maj@8 | 20.0 | 57.5 | 79.6 | 47.0 | 41.5 | 42.7 | 48.0 |
| pass@8 | 33.3 | 82.5 | 88.8 | 58.5 | 47.5 | 57.7 | 61.4 |
| Best-of-8 | |||||||
| Math-PSA-7B | 20.0 | 55.0 | 80.8 | 47.5 | 39.5 | 40.1 | 47.2 |
| RLHFlow-DeepSeek-8B | 13.3 | 57.5 | 76.2 | 40.0 | 39.0 | 39.7 | 44.3 |
| Skywork-PRM-7B | 16.7 | 55.0 | 81.2 | 44.0 | 40.5 | 44.5 | 47.0 |
| ReasonEval-7B | 6.7 | 55.0 | 75.2 | 41.0 | 40.0 | 40.4 | 43.1 |
| Qwen2.5-Math-7B-PRM800K | 13.3 | 57.5 | 80.0 | 44.5 | 43.5 | 43.0 | 47.7 |
| Qwen2.5-Math-PRM-7B | 16.7 | 55.0 | 82.0 | 48.0 | 43.5 | 43.0 | 48.0 |
| R-PRM-7B-DPO | 20.0 | 62.5 | 82.2 | 48.0 | 41.0 | 44.1 | 49.6 |
| ★ GPRM-4B | 20.0 | 63.0 | 82.6 | 48.5 | 40.5 | 45.0 | 50.1 |
| ★ GPRM-14B | 20.0 | 64.2 | 83.1 | 50.3 | 42.6 | 45.8 | 51.0 |
| Greedy Guided Search@8 | |||||||
| Math-Shepherd-7B | 13.3 | 52.5 | 74.6 | 38.5 | 36.5 | 41.2 | 42.8 |
| RLHFlow-PRM-DeepSeek-8B | 13.3 | 52.5 | 74.8 | 39.5 | 37.0 | 40.8 | 43.0 |
| Llemma-PRM800K-7B | 13.3 | 57.5 | 73.8 | 40.0 | 36.5 | 38.2 | 43.2 |
| Skywork-PRM-7B | 10.0 | 57.5 | 77.8 | 41.5 | 39.0 | 43.4 | 44.9 |
| ReasonEval-7B | 3.3 | 55.0 | 73.0 | 37.5 | 35.5 | 37.9 | 40.4 |
| Qwen2.5-Math-7B-PRM800K | 23.3 | 45.0 | 78.2 | 42.0 | 35.5 | 38.6 | 43.8 |
| Qwen2.5-Math-PRM-7B | 16.7 | 60.0 | 81.0 | 43.5 | 39.0 | 40.4 | 46.8 |
| R-PRM-7B-DPO | 16.7 | 70.0 | 80.0 | 46.5 | 39.5 | 43.4 | 49.4 |
| ★ GPRM-4B | 23.3 | 85.0 | 80.0 | 48.0 | 45.0 | 48.8 | 55.0 |
| ★ GPRM-14B | 23.3 | 87.5 | 85.0 | 45.0 | 39.5 | 50.0 | 55.0 |
Table 3. Performance under Best-of-8 and Greedy Guided Search@8 using Qwen2.5-7B-Instruct as the policy model. ★ denotes our models.
Inference Cost Analysis
We analyze the inference cost introduced by the future-step view . We compare the identical GPRM-4B-SFT with and without (same Qwen3-4B base, same training, same 8×H20-96 GB) over three regimes (Table 5). The key distinction is in online search: without a standard greedy pass scores 8 single-step candidates directly, whereas with each of the 8 candidates must first be completed into a full trajectory before GPRM scores it—it is this completion, not the future text, that raises cost.
| Regime | Tok/q | Lat(s) | Tok+ (%) | Score | |
| Single-problem diag. (offline) | |||||
| GPRM-4B (w/o ) | 17555 | 12.5 | – | 61.6 | |
| GPRM-4B (ours) | 19253 | 17.8 | +9.6 | 72.6 | |
| Best-of-8 (offline TTS) | |||||
| GPRM-4B (w/o ) | 140442 | 15.5 | – | 47.8 | |
| GPRM-4B (ours) | 162755 | 18.7 | +15.9 | 50.0 | |
| Greedy Guided Search@8 (online TTS) | |||||
| GPRM-4B (w/o ) | 152327 | 17.7 | – | 48.6 | |
| GPRM-4B (ours) | 261532 | 25.2 | +71.7 | 55.0 |
Table 5. Inference cost with/without future trajectory . Measured on GPRM-4B-SFT (8×H20, 96GB). Tokens: input+output per problem. Latency: end-to-end per problem. Score: mean of PRMBench/ProcessBench (diagnosis) or avg. of six TTS benchmarks. “w/o ” is control; differences arise solely from whether is used.
The cost of is regime-localized. It is nearly free offline (+9.6% tokens, +11.1 pts) and in Best-of-8 (+15.9% tokens, +2.2 pts), since the future is already part of the given or sampled trajectory. Only online greedy search is costly (+71.7% tokens, +42.4% latency) because of candidate completion—but it buys +6.4 pts, consistent with the 16.0-pt drop when is removed (Table 4). The overhead is thus the price of the global view, confined to online search.
| Variant | PRM (Overall) | Proc (Avg. F1) |
| Performance Metrics | ||
| w/o Hist. & Future | 66.2↓6.7 | 57.0↓15.3 |
| w/o Hist. Judg. | 69.8↓3.1 | 61.1↓11.2 |
| w/o Future | 71.6↓1.3 | 56.3↓16.0 |
| w/o Self Check | 68.7↓4.2 | 62.9↓9.4 |
| w/o Goal Align. | 69.9↓3.0 | 64.7↓7.6 |
| Full GPRM | 72.9 | 72.3 |
Table 4. Ablation study on the impact of historical judgments, future trajectories, and analysis dimensions. Results are reported using GPRM-4B-SFT. Note: “PRM” denotes PRMBench, “Proc” denotes ProcessBench. “Hist.” denotes Historical Judgments.
Ablation Studies
Table 4 presents ablation results on PRMBench and ProcessBench using GPRM-4B-SFT to isolate component effects. As Table 1 shows, the SFT→GRPO transition consistently improves performance, suggesting these findings remain valid post-GRPO. We treat the two-stage pipeline as a design necessity rather than an ablation variable: direct GRPO application fails to converge due to the complexity of simultaneously learning four-dimensional diagnostics and structured formatting under high-dimensional global input . SFT establishes the foundational analysis skills and format, enabling GRPO to effectively refine judgments on hard cases.
Table 1 | Overall performance of GPRM on process-supervision benchmarks. Subtable (a) reports results on PRMBench, and subtable (b) reports results on ProcessBench. Green rows indicate model categories, pink rows highlight our models, and ★ marks the primary model. The last two rows of each subtable compare the SFT and GRPO stages of GPRM-4B: GRPO consistently improves over SFT on both benchmarks (+1.0 on PRMBench, +2.0 on ProcessBench), showing that the gains from the four-dimensional global framework are preserved—and amplified—after reinforcement learning.
(a) Model performance comparison on PRMBench.
| Model Name | NR. | NCL. | Avg. | ES | SC. | DC. | CI | Avg. | PS | DR. | MS. | Avg. | Overall |
| LLM-as-judge, Proprietary language models | |||||||||||||
| GPT-4o | 57.0 | 62.4 | 59.7 | 72.0 | 69.7 | 70.7 | 71.1 | 70.9 | 62.5 | 65.7 | 99.2 | 75.8 | 66.8 |
| o1-mini | 65.6 | 63.7 | 64.6 | 74.5 | 67.7 | 73.8 | 72.3 | 72.1 | 61.8 | 64.8 | 100.0 | 75.5 | 68.8 |
| Gemini-2.0-flash-exp | 66.0 | 67.2 | 58.1 | 62.7 | 70.4 | 65.7 | 66.0 | 67.3 | 61.8 | 66.2 | 98.2 | 75.4 | 66.9 |
| Gemini-2.0-thinking-exp-1219 | 68.8 | 68.5 | 63.8 | 66.2 | 72.9 | 71.3 | 71.0 | 71.8 | 60.3 | 65.7 | 99.8 | 75.3 | 70.3 |
| LLM-as-judge, Open-source language models | |||||||||||||
| Qwen2.5-Math-72B-Instruct (Yang et al., 2024) | 55.3 | 54.9 | 55.1 | 55.5 | 71.6 | 58.1 | 59.1 | 61.1 | 47.4 | 53.8 | 100.0 | 67.1 | 57.4 |
| Qwen2.5-72B-Instruct [12] | 57.6 | 54.9 | 56.3 | 61.4 | 65.5 | 65.8 | 61.1 | 63.4 | 48.8 | 54.1 | 100.0 | 67.6 | 57.5 |
| QwQ-32B-Preview [18] | 57.2 | 55.6 | 56.4 | 67.4 | 72.3 | 66.2 | 66.9 | 68.2 | 57.8 | 62.7 | 100.0 | 73.5 | 63.6 |
| PRMs | |||||||||||||
| Math-Shepherd-7B [19] | 44.0 | 50.3 | 47.1 | 49.4 | 44.5 | 41.3 | 47.7 | 45.7 | 47.2 | 48.6 | 86.1 | 60.7 | 47.0 |
| RLHFlow-Mistral-8B (Xiong et al., 2024) | 46.1 | 47.3 | 46.7 | 56.6 | 55.1 | 54.4 | 63.8 | 57.5 | 51.5 | 56.2 | 97.9 | 68.5 | 54.4 |
| RLHFlow-DeepSeek-8B (Xiong et al., 2024) | 46.4 | 48.9 | 47.6 | 55.7 | 55.0 | 53.2 | 66.2 | 57.5 | 49.0 | 55.4 | 99.8 | 68.1 | 54.2 |
| Llemma-PRM800K-7B [17] | 49.3 | 53.4 | 51.4 | 56.4 | 47.1 | 46.7 | 53.3 | 50.9 | 51.0 | 53.5 | 93.6 | 66.0 | 52.0 |
| Skywork-PRM-1.5B (He et al., 2024) | 52.0 | 56.4 | 54.2 | 64.8 | 64.9 | 63.3 | 65.5 | 64.9 | 57.5 | 63.3 | 91.1 | 70.7 | 61.1 |
| Skywork-PRM-7B (He et al., 2024) | 56.4 | 62.8 | 59.6 | 69.4 | 67.1 | 67.7 | 69.9 | 68.5 | 60.9 | 65.8 | 93.2 | 73.3 | 65.1 |
| Qwen2.5-Math-7B-PRM800K (Zhang et al., 2025a) | 48.6 | 47.8 | 48.2 | 62.1 | 59.4 | 58.7 | 68.5 | 62.2 | 52.9 | 64.0 | 99.8 | 72.2 | 58.3 |
| Qwen2.5-Math-PRM-7B [12] | 49.0 | 55.1 | 52.1 | 71.8 | 67.3 | 66.3 | 78.5 | 71.0 | 57.6 | 69.1 | 99.7 | 75.5 | 65.5 |
| R-PRM-7B-SFT [15] | 52.7 | 64.7 | 58.7 | 70.1 | 62.7 | 63.4 | 69.5 | 66.4 | 61.4 | 67.4 | 98.3 | 75.7 | 64.9 |
| R-PRM-7B-DPO [15] | 52.2 | 58.2 | 55.2 | 72.1 | 69.1 | 68.9 | 75.0 | 71.2 | 61.2 | 69.5 | 99.1 | 76.6 | 66.8 |
| GenPRM-7B (Zhao et al., 2025) | 53.0 | 59.1 | 56.1 | 72.6 | 69.8 | 69.3 | 75.6 | 71.8 | 61.7 | 70.0 | 99.2 | 77.0 | 67.4 |
| GPRM-4B-SFT | 67.0 | 62.9 | 65.0 | 78.2 | 74.3 | 73.5 | 80.8 | 75.2 | 63.8 | 73.5 | 99.2 | 78.8 | 72.9 |
| GPRM-4B-GRPO | 68.2 | 63.4 | 65.8 | 79.1 | 75.0 | 74.2 | 81.5 | 76.2 | 64.5 | 74.1 | 99.3 | 79.3 | 73.9 |
| ★ GPRM-14B-GRPO | 69.5 | 64.8 | 67.2 | 80.3 | 76.1 | 75.4 | 82.6 | 77.6 | 65.9 | 75.2 | 99.5 | 80.2 | 74.6 |
(b) Model performance comparison on ProcessBench.
| Model | GSM8K Err. | GSM8K Cor. | GSM8K F1 | MATH Err. | MATH Cor. | MATH F1 | OLYMPIADBENCH Err. | OLYMPIADBENCH Cor. | OLYMPIADBENCH F1 | OMNIMATH Err. | OMNIMATH Cor. | OMNIMATH F1 | Avg. | F1 |
| LLM-as-judge, Proprietary language models | ||||||||||||||
| GPT-4o* | 70.0 | 91.2 | 79.2 | 54.4 | 76.6 | 63.6 | 45.8 | 58.4 | 51.4 | 45.2 | 65.6 | 53.5 | 61.9 | |
| o1-mini* | 88.9 | 97.9 | 93.2 | 83.5 | 95.1 | 88.9 | 80.2 | 95.6 | 87.2 | 74.8 | 91.7 | 82.4 | 87.9 | |
| LLM-as-judge, Open-source language models | ||||||||||||||
| Llama-3.3-70B-Instruct* [3] | 72.5 | 96.9 | 82.9 | 43.3 | 83.2 | 59.4 | 31.0 | 94.1 | 46.7 | 28.2 | 90.5 | 43.0 | 58.0 | |
| Qwen2.5-Math-72B-Instruct* (Yang et al., 2024) | 49.8 | 96.9 | 65.8 | 36.0 | 94.3 | 52.1 | 19.5 | 97.3 | 32.5 | 19.0 | 96.3 | 31.7 | 45.5 | |
| Qwen2.5-72B-Instruct* [12] | 62.8 | 96.9 | 72.6 | 46.3 | 93.1 | 61.8 | 38.7 | 92.6 | 54.6 | 36.6 | 90.9 | 52.2 | 61.2 | |
| QwQ-32B-Preview* [18] | 81.6 | 95.3 | 88.0 | 78.1 | 79.3 | 78.7 | 61.4 | 54.6 | 57.8 | 55.7 | 68.0 | 61.3 | 71.5 | |
| PRMs | ||||||||||||||
| Math-Shepherd-7B [19] | 32.4 | 91.7 | 47.9 | 18.0 | 82.0 | 29.5 | 15.0 | 71.1 | 24.8 | 14.2 | 73.0 | 23.8 | 31.5 | |
| Math-PSA-7B (He et al., 2024) | 48.3 | 88.1 | 62.4 | 29.5 | 72.7 | 41.9 | 20.7 | 65.8 | 31.5 | 15.4 | 68.9 | 25.2 | 40.3 | |
| RLHFlow-DeepSeek-8B (Xiong et al., 2024) | 24.2 | 98.4 | 38.8 | 21.4 | 80.0 | 33.8 | 10.1 | 51.0 | 16.9 | 10.9 | 51.9 | 16.9 | 26.6 | |
| Llemma-PRM800K-7B [17] | 36.7 | 71.0 | 48.4 | 39.2 | 47.8 | 43.1 | 33.1 | 25.1 | 28.5 | 35.4 | 31.5 | 33.4 | 38.4 | |
| Skywork-PRM-7B (He et al., 2024) | 61.8 | 82.9 | 70.8 | 43.8 | 62.2 | 53.6 | 17.9 | 31.9 | 22.9 | 14.0 | 41.9 | 21.0 | 42.1 | |
| ReasonEval-7B [20] | 26.1 | 95.3 | 41.0 | 35.7 | 77.6 | 48.9 | 27.5 | 55.2 | 36.7 | 27.0 | 60.6 | 37.4 | 41.0 | |
| Qwen2.5-Math-7B-PRM800K* (Zhang et al., 2025a) | 53.1 | 95.3 | 68.2 | 48.0 | 90.1 | 62.6 | 35.7 | 87.3 | 50.7 | 29.8 | 86.1 | 44.3 | 56.5 | |
| R-PRM-7B-SFT [15] | 66.2 | 92.7 | 77.2 | 60.3 | 88.2 | 71.6 | 48.6 | 77.3 | 59.6 | 40.1 | 75.5 | 52.3 | 65.2 | |
| R-PRM-7B-DPO [15] | 72.0 | 91.7 | 80.7 | 71.2 | 83.5 | 76.9 | 60.2 | 67.8 | 63.8 | 55.5 | 65.6 | 60.1 | 70.4 | |
| GenPRM-7B (Zhao et al., 2025) | 61.4 | 92.7 | 73.7 | 70.7 | 86.6 | 77.9 | 68.2 | 75.8 | 71.8 | 68.8 | 79.7 | 73.8 | 74.3 | |
| GPRM-4B-SFT | 60.5 | 92.1 | 73.1 | 68.4 | 86.0 | 76.2 | 66.0 | 73.1 | 69.4 | 67.3 | 74.0 | 70.5 | 72.3 | |
| GPRM-4B-GRPO | 61.3 | 92.0 | 73.1 | 70.0 | 86.8 | 77.5 | 68.0 | 75.3 | 71.5 | 70.1 | 80.9 | 75.1 | 74.3 | |
| ★ GPRM-14B-GRPO | 62.3 | 93.4 | 74.7 | 72.2 | 88.0 | 79.3 | 70.8 | 77.4 | 73.9 | 72.5 | 78.3 | 75.3 | 75.8 |
Table 7.
Impact of Global Context. Removing both historical judgments and future trajectories causes significant drops of 6.7 (PRMBench) and 15.3 (ProcessBench) points. Isolating these components reveals that removing only historical judgments leads to drops of 3.1 and 11.2 points, while removing only future trajectories causes drops of 1.3 and 16.0 points, highlighting the critical role of future trajectories for early error localization in ProcessBench.
Impact of Analysis Dimensions. Removing Self Check and Goal Alignment results in performance drops of 4.2/9.4 and 3.0/7.6 points, respectively. These results confirm the contribution of all four dimensions. Independent expert review further validated the F/P/G/S axes as orthogonal and comprehensive in covering common practitioner-identified failure modes.
Conclusion
We propose GPRM, a global perspective process reward model that addresses two critical limitations of existing PRMs: the lack of historical judgment inheritance and the absence of a global perspective. By jointly conditioning on historical judgments, the current step, and future trajectories, GPRM enables robust step-level evaluation through a structured four-dimensional diagnostic framework. Our progressive training pipeline combines supervised fine-tuning with hard-example-based GRPO to enhance both foundational analysis capabilities and decision robustness in challenging scenarios. Extensive experiments demonstrate that GPRM achieves state-of-the-art performance on process supervision benchmarks, generalizes effectively to agent error localization and general reward modeling tasks, and significantly improves test-time search guidance.
Benchmark Descriptions
Process Supervision Benchmarks
PRMBench ([16]) is a fine-grained process supervision benchmark that evaluates models on three dimensions: Simplicity, Soundness, and Sensitivity. The benchmark requires identifying both the first erroneous step and all erroneous steps in a reasoning chain, focusing on continuous diagnosis over the entire trajectory.
ProcessBench ((Zheng et al., 2025)) focuses on earliest critical error localization in mathematical reasoning. It evaluates models across four datasets: GSM8K, MATH, OlympiadBench, and OmniMath. The benchmark uses F1 score to measure both error detection and correct step recognition.
Cross-Task Generalization Benchmarks
Agent Error Bench ((Zhu et al., 2025)) evaluates error localization in LLM-agent trajectories across three environments: ALFWorld, WebShop, and GAIA. The benchmark defines two tasks: S (identifying the earliest critical error step) and S+M (identifying both the error step and the responsible module).
PPE (Preference Proxy Evaluations) evaluates models’ ability to make preference judgments between candidate responses. We use the verifiable correctness subset, which includes: MMLU-Pro, MATH, GPQA, MBPP-Plus, and IFEval.
Test-Time Search Benchmarks
For test-time search evaluation, we use six mathematical reasoning benchmarks: AIME 2024, AMC 2023, MATH, OlympiadBench, College Math, and Minerva MATH.
We evaluate under two settings: Best-of-8 (sample 8 complete trajectories and select the top-ranked solution) and Greedy Guided Search@8 (step-wise 8-candidate search: at each step the policy generates 8 complete candidate trajectories, GPRM scores the current step of each under its future trajectory, and a step judged correct is committed before moving on). Because GPRM scores a step from its future trajectory, every candidate step must first be completed into a full trajectory; this completion cost is analyzed in § 5.4.
Data Construction Details
SFT Data Construction
We construct the SFT dataset based on OpenMathInstruct-2. The construction process consists of three steps:
Step 1: Seed Sample Selection. We select 80,000 hard samples from the train_1M subset of OpenMathInstruct-2.
Step 2: Targeted Error Injection. For each seed sample where , we use the teacher model Qwen3-235B-Instruct to inject errors targeting four analysis dimensions:
Forward Consistency (40K samples)
Self Validity (20K samples)
Goal Alignment (10K samples)
Posterior Verification (10K samples)
For each error type and target position , we generate an erroneous step and propagate its effect to subsequent steps, yielding a perturbed trajectory . The per-dimension counts (40K/20K/10K/10K) are set in proportion to the empirical error-type frequencies estimated from the PRM800K annotations, so that the injection distribution follows the natural occurrence of each diagnostic dimension; a more exhaustive mixture-ratio search is left to future work.
Step 3: Teacher Annotation. For each perturbed trajectory, we prompt the teacher model to generate a structured four-dimensional analysis for the target step . If the teacher’s initial judgment is inconsistent with the ground-truth label , we apply self-correction prompting and require re-analysis.
GRPO Data Construction
We construct the GRPO dataset based on PRM800K. The construction process consists of three steps:
Step 1: Difficult Sample Selection. We select 10,000 difficult samples from PRM800K based on three criteria: low confidence, long-horizon reasoning, and delayed-error positions.
Step 2: Future Trajectory Completion. For samples where the original trajectory terminates immediately after an error, we use the teacher model to generate the subsequent reasoning process. For an erroneous step , we condition on and generate candidate continuations using temperature and nucleus sampling .
Step 3: Hard-Example Mining. We construct the complete global input for each sample. We then perform samplings using the SFT model and retain samples where at least one prediction disagrees with the gold label. This yields approximately 20,000 GRPO training instances.
Generalization to General Reasoning Tasks
To assess whether GPRM’s capabilities extend beyond mathematical reasoning, we evaluate it on the verifiable correctness subset of the PPE (Process-level Preference Evaluation) benchmark [2], which covers diverse reasoning domains including multi-choice QA (MMLU-Pro), mathematics (MATH), science reasoning (GPQA), code generation (MBPP-Plus), and instruction following (IFEval).
As shown in Table 6, GPRM-4B achieves a mean score of 0.64, outperforming GPT-4o-mini (ArenaHard) (0.63) and Llama-3.1-70B (ArenaHard) (0.63), despite being significantly smaller in size. This performance approaches Claude 3.5 (ArenaHard) (0.68) and Athene-RM-70B (0.69), demonstrating that GPRM learns general trajectory evaluation capabilities that transfer effectively beyond mathematical domains. Notably, GPRM-4B shows strong performance across diverse tasks, achieving 0.71 on MATH, 0.67 on MMLU-Pro, 0.65 on MBPP-Plus, and 0.58 on GPQA, confirming its robustness as a general-purpose process reward model.
| Reward Model | MMLU-Pro | MATH | GPQA | MBPP-Plus | IFEval | Mean |
| Athene-RM-70B | 0.77 | 0.79 | 0.59 | 0.68 | 0.62 | 0.69 |
| Claude 3.5 (ArenaHard)† | 0.81 | 0.86 | 0.63 | 0.54 | 0.58 | 0.68 |
| Llama-3-OffsetBias-RM-8B | 0.62 | 0.68 | 0.55 | 0.74 | 0.62 | 0.64 |
| GPT-4o-mini (ArenaHard)† | 0.71 | 0.81 | 0.57 | 0.54 | 0.56 | 0.63 |
| Llama-3.1-70B (ArenaHard)† | 0.73 | 0.73 | 0.56 | 0.58 | 0.56 | 0.63 |
| internLM2-20B-Reward | 0.68 | 0.70 | 0.57 | 0.58 | 0.62 | 0.63 |
| Athene-RM-8B | 0.68 | 0.71 | 0.55 | 0.62 | 0.57 | 0.62 |
| ArmoRM-Llama3-8B-v0.1 | 0.66 | 0.71 | 0.57 | 0.54 | 0.58 | 0.61 |
| Skywork-Reward-Llama-3.1-8B | 0.64 | 0.70 | 0.57 | 0.52 | 0.61 | 0.61 |
| Nemotron-4-340B-Reward | 0.70 | 0.65 | 0.57 | 0.49 | 0.63 | 0.61 |
| internLM2-7B-Reward | 0.67 | 0.73 | 0.55 | 0.44 | 0.64 | 0.60 |
| Llama-3.1-70B (Alpaca)† | 0.66 | 0.66 | 0.56 | 0.52 | 0.56 | 0.59 |
| Claude 3.5 (Alpaca)† | 0.66 | 0.63 | 0.56 | 0.52 | 0.57 | 0.59 |
| Skywork-Reward-Gemma-2-27B | 0.54 | 0.63 | 0.53 | 0.59 | 0.54 | 0.56 |
| GPT-4o-mini (Alpaca)† | 0.57 | 0.64 | 0.53 | 0.52 | 0.56 | 0.56 |
| NaiveVerbosityModel | 0.48 | 0.50 | 0.48 | 0.31 | 0.52 | 0.46 |
| ★ GPRM-4B | 0.67 | 0.71 | 0.58 | 0.65 | 0.59 | 0.64 |
Table 6. Performance on the verifiable correctness subset of PPE. ★ denotes our model.
Training Details
SFT Training
Model Architecture. We initialize GPRM from Qwen3-4B-Instruct and Qwen3-14B.
Training Hyperparameters.
Learning rate: with cosine decay
Batch size: 128
Training epochs: 3
Optimizer: AdamW with ,
Weight decay: 0.1
Gradient clipping: 1.0
Warmup steps: 500
Training Objective. We use the standard autoregressive maximum likelihood objective:
GRPO Training
Training Hyperparameters.
Learning rate: with constant schedule
Batch size: 64
Training epochs: 2
Optimizer: AdamW with ,
Weight decay: 0.01
Gradient clipping: 1.0
Number of samples per input:
Clipping threshold:
Reward Function. We use a binary outcome-based reward:
GRPO Objective. The optimization objective is:
where is the importance sampling ratio, and is the group-relative advantage function.
Computational Resources
All experiments are conducted on NVIDIA A100 80GB GPUs. Training times:
GPRM-4B SFT: 12 hours on 8 GPUs
GPRM-4B GRPO: 8 hours on 8 GPUs
GPRM-14B SFT: 36 hours on 16 GPUs
GPRM-14B GRPO: 24 hours on 16 GPUs
Evaluation Protocols
Process Supervision Evaluation
PRMBench. We evaluate on all three dimensions and report the overall average score. For each problem, the model predicts step-level correctness labels, and we compute accuracy by comparing predictions with gold annotations.
ProcessBench. We report F1 scores for both error detection and correct step recognition on each of the four datasets. The average F1 is computed as the unweighted mean across all datasets.
Cross-Task Generalization Evaluation
Agent Error Bench. We evaluate on two tasks: S (accuracy of identifying the earliest critical error step) and S+M (accuracy of jointly identifying the error step and the responsible module). Results are reported separately for each environment and averaged.
PPE. For each prompt with multiple candidate responses, we apply GPRM to predict step-level correctness for each candidate trajectory, compute a trajectory score as the ratio of correct steps to total steps, and select the candidate with the highest score. We report the accuracy of preference judgments.
Test-Time Search Evaluation
Best-of-8. For each problem, we sample 8 complete trajectories, apply GPRM to compute a trajectory score for each candidate, and select the trajectory with the highest score. We report the accuracy of final answers.
Greedy Guided Search@8. For each problem, at each step the policy first generates 8 complete candidate trajectories that share the committed prefix but differ in how the current step continues into the future. GPRM then scores the current step of each candidate jointly with its future trajectory , and a step judged correct (one of the candidates scoring CORRECT) is committed to extend the trajectory; the committed step plus its prefix is fed back to the policy, and the next step is searched in the same way. The process repeats until the model emits a final answer. We report the accuracy of final answers. Because scoring a candidate step requires its future trajectory, the policy must complete each candidate before GPRM can evaluate it; the resulting compute overhead is analyzed in § 5.4.
References
- [1]X. Chen, G. Li, Z. Wang, B. Jin, C. Qian, Y. Wang, H. Wang, Y. Zhang, D. Zhang, T. Zhang, H. Tong, and H. Ji. Rm-rl: Reward modeling as reasoning, 2026. URL https://arxiv.org/abs/2505.02387.
- [2]E. Frick, T. Li, C. Chen, W.-L. Chiang, A. N. Angelopoulos, J. Jiao, B. Zhu, J. E. Gonzalez, and I. Stoica. How to evaluate reward models for RLHF. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=cbttLtO94Q.arxiv.org/abs/2410.14872
- [3]A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. V. Chern, C. Cauchetaux, C. Nayak, C. B. Ci, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Gefert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnston, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawlak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girradhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangam, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Groswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C.-H. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreym er, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E.-T. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspregen, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I.-E. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J.-B. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.
- [4]J. Guo, Z. Chi, L. Dong, Q. Dong, X. Wu, S. Huang, and F. Wei. Reward reasoning model, 2025. URL https://arxiv.org/abs/2505.14674.
- [6]M. Khalifa, R. Agarwal, L. Logeswaran, J. Kim, H. Peng, M. Lee, H. Lee, and L. Wang. Process reward models that think, 2025. URL https://arxiv.org/abs/2504.16828.
- [7]W. Li and Y. Li. Process reward model with q-value rankings, 2025. URL https://arxiv.org/abs/2410.11287.
- [8]H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step, 2023. URL https://arxiv.org/abs/2305.20050.
- [9]R. Liu, J. Gao, J. Zhao, K. Zhang, X. Li, B. Qi, W. Ouyang, and B. Zhou. Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling, 2025. URL https://arxiv.org/abs/2502.06703.
- [10]L. Luo, Y. Liu, R. Liu, S. Phatale, M. Guo, H. Lara, Y. Li, L. Shu, Y. Zhu, L. Meng, J. Sun, and A. Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024. URL https://arxiv.org/abs/2406.06592.
- [11]S. Park, X. Liu, Y. Gong, and E. Choi. Ensembling large language models with process reward-guided tree search for better complex reasoning, 2024. URL https://arxiv.org/abs/2412.15797.
- [12]Qwen, ; A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115.
- [13]A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, and A. Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning, 2024. URL https://arxiv.org/abs/2410.08146.
- [14]Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300.
- [15]S. She, J. Liu, Y. Liu, J. Chen, X. Huang, and S. Huang. R-prm: Reasoning-driven process reward modeling, 2025. URL https://arxiv.org/abs/2503.21295.
- [16]M. Song, Z. Su, X. Qu, J. Zhou, and Y. Cheng. Prmbench: A fine-grained and challenging benchmark for process-level reward models, 2025. URL https://arxiv.org/abs/2501.03124.
- [17]Z. Sun. Gpt-accelera: Simple and efficient pytorch-native transformer training and inference (batched). https://github.com/Edward-Sun/gpt-accelera, 2024.
- [18]Q. Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https://qwenlm.github.io/blog/qwq-32b-preview/.
- [19]P. Wang, L. Li, Z. Shao, R. X. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024. URL https://arxiv.org/abs/2312.08935.
- [20]S. Xia, X. Li, Y. Liu, T. Wu, and P. Liu. Evaluating mathematical reasoning beyond accuracy. arXiv preprint arXiv:2404.05692, 2024.arxiv.org/abs/2404.05692

