WorksComputer Science
Verifier-Guided Self-Consistency: Test-Time Reasoning with Imperfect Generators and Verifiers
Abstract
Large language models can solve many multi-step reasoning problems, but their reasoning remains brittle: plausible intermediate steps often lead to incorrect answers. Improving reasoning at test time requires balancing two imperfect signals: the generator, which provides diversity, and the verifier, which guides correctness. Existing approaches largely favor one side of this tradeoff. Verifier-weighted voting relies on trajectories sampled from the generator and cannot steer generation toward correct ones. By contrast, beam-style verifier-guided search can steer generation, but may over-concentrate on narrow trajectory sets and amplify verifier errors. We propose Verifier-Guided Self-Consistency (VGSC), a stochastic decoding framework that combines step-level correctness and future-outcome estimates to guide reasoning during generation. VGSC repeatedly proposes candidate next steps from the base language model and uses both signals in an acceptance/resampling procedure, while maintaining multiple trajectories for final answer-level self-consistency. We provide a probabilistic interpretation of VGSC as approximate sampling from a verifier-tilted trajectory distribution. Across multiple reasoning benchmarks, generator models, and verifier families, VGSC achieves higher average accuracy than representative baselines.
Introduction
Large language models (LLMs) have shown strong performance on multi-step reasoning tasks, especially when prompted to produce chain-of-thought (CoT) explanations [28] and when multiple reasoning trajectories are aggregated with self-consistency [27]. However, their reasoning remains brittle: models can produce plausible intermediate steps that lead to incorrect answers [3, 15].
A natural way to improve reliability is to augment the LLM generator with a learned verifier trained to estimate whether a solution, or an intermediate reasoning step, is correct [15, 26]. At inference time, such verifiers can guide search or rerank candidates without modifying the underlying LLM [13, 30, 19].
Existing inference-time uses of learned verifiers largely fall into two families. In post-hoc aggregation, multiple complete reasoning trajectories are sampled from the base model, and a verifier is used afterward to rank them or weight their votes [3, 27, 10]. This preserves diversity, but cannot help if correct trajectories are rarely generated. In contrast, verifier-guided decoding uses verifier information during decoding to guide search over partial reasoning traces [32, 13, 30, 35, 33]. Some methods rely on local signals that assess the next step [13, 30], while others use more future-looking signals that estimate whether a prefix is likely to admit a successful continuation [35, 33, 21].
Correspondence: [email protected], [email protected]. Our implementation is available at https://github.com/cant-access-rediska0123/vgsc .
These two families reflect a broader tension in test-time reasoning. The generator provides coverage and diversity over plausible trajectories, but may assign low probability to correct ones; the verifier provides guidance toward correctness, but is itself imperfect. Learned verifiers can be miscalibrated [20], generalize poorly across domains [34], or be over-optimized when used too aggressively [12]. Moreover, maximizing noisy scores inside beam search can introduce overestimation bias, so wider search may degrade reasoning performance rather than improve it [4]. Thus, post-hoc aggregation can fail when the generator rarely produces good trajectories, while beam-style verifier-guided search can fail when the verifier is too noisy for aggressive pruning.
In this paper, we propose Verifier-Guided Self-Consistency (VGSC), a decoding strategy for test-time reasoning with imperfect generators and verifiers. VGSC is motivated by the observation that neither generator likelihood nor verifier score is reliable enough to optimize in isolation. Instead of verifying only after complete generation or deterministically pruning to the highest-scoring partial chains, VGSC uses verifier information stochastically during generation. At each reasoning step, it samples candidate next steps from the base LLM and resamples among them using two complementary signals: (i) a step verifier estimating immediate correctness, and (ii) a future-outcome predictor estimating whether the resulting prefix is likely to admit a successful continuation. Repeating this process yields multiple verifier-guided but diverse trajectories, whose final answers are aggregated by self-consistency. Our contributions are as follows:
We introduce VGSC, a stochastic verifier-guided decoding algorithm that combines step-level correctness and future-outcome prediction through bounded rejection-resampling, while maintaining multiple trajectories for answer-level self-consistency.
We derive the exact local conditional of a verifier-tilted trajectory distribution and analyze the approximation error introduced by bounded rejection-resampling, establishing a finite-cap total variation bound under exact guidance.
We empirically show that VGSC improves accuracy over representative baselines across benchmarks, LLMs, and verifier families, while reducing LLM-generated tokens.
Background and Motivation
We study test-time search for chain-of-thought reasoning with a generator model and verifier guidance. Given input problem , let be a reasoning trajectory of steps. We denote prefixes by and , suffixes by , and prefix extension by .
The base language model induces a distribution over reasoning trajectories,
We assume access to a verifier that estimates whether a reasoning trajectory is correct. Concretely, let denote the event that trajectory is correct for problem , and define
Assuming a stepwise verifier, we factorize it as
where is the probability that step is correct given that the previous steps are correct. This interpretation is aligned with standard step-verifier supervision, where training prefixes are truncated at the first incorrect step ([15]).
Inference-Time Reasoning with Verifiers
Existing inference-time methods use verifier information in two main ways. The first is post-hoc aggregation: multiple complete reasoning trajectories are sampled from the base model and combined only after generation, for example by majority vote as in self-consistency (SC; [27]) or by verifier-weighted voting as in weighted self-consistency (WSC; [10]). These methods preserve diversity across reasoning paths, but the verifier does not guide generation itself, so they can fail when the generator rarely, or never, samples a correct trajectory.
The second family uses verifier information during decoding, for example through beam-style or tree-based search over partial reasoning traces ([30]; [32]). Such methods can allocate more computation to promising prefixes, but when verifier scores are noisy, they risk over-concentrating on a narrow region of the search space. Recent work shows that maximizing imperfect inference-time scores can induce overestimation bias, so increasing search width need not monotonically improve reasoning performance and can even degrade it ([4]).
These families expose a natural tradeoff: post-hoc aggregation preserves diversity but cannot steer generation toward missing correct trajectories, whereas verifier-guided search uses feedback earlier but can amplify local verifier errors and collapse diversity through aggressive deterministic pruning. Our goal is to combine the strengths of both families: generating diverse reasoning trajectories while biasing computation toward prefixes likely to lead to correct solutions. This motivates Verifier-Guided Self-Consistency (VGSC), which uses verifier guidance stochastically during decoding while still producing multiple independent trajectories for final self-consistency. Rather than optimizing any one imperfect signal in isolation, VGSC combines generator likelihood, immediate step correctness, and continuation value. Figure 1 compares VGSC with representative post-hoc aggregation and verifier-guided search approaches.

Figure 1. Illustration of inference-time reasoning with imperfect generators and verifiers. Left: Weighted Self-Consistency samples complete trajectories from and applies verifier weighting after generation, so it cannot recover from poor samples. Middle: Beam Search uses verifier scores during decoding, but deterministic selection can concentrate search in a narrow high- region; with a noisy verifier, chains may converge to the same false-positive region. Right: VGSC samples candidate next steps from and uses bounded acceptance-resampling with and , balancing verifier guidance and trajectory diversity.
Method: Verifier-Guided Self-Consistency
Verifier-tilted Target Distribution
We begin by defining the trajectory distribution that VGSC aims to approximate. Given a base LLM generator and a verifier , we define the verifier-tilted distribution
Here, controls the strength of verifier guidance: recovers the base generator, gives the standard verifier-tilted distribution [35], and larger concentrates more mass on high-verifier trajectories. Thus, favors trajectories that are both likely under the generator and highly rated by the verifier.
This distribution has a simple probabilistic interpretation at . If the verifier is calibrated so that , then
and therefore . Thus, is the posterior over reasoning trajectories conditioned on correctness, while is a tempered analogue. Weighted self-consistency method [10], which samples from and weights resulting texts with verifier scores after generation, can be viewed as importance sampling for the answer marginal under . VGSC instead uses verifier information during decoding to sample directly from .
Exact Local Conditional
For a prefix , the verifier-tilted trajectory distribution induces the following exact next-step conditional:
Proposition 1 Given a prefix , define expected future score as
Then the next-step conditional of the verifier-tilted distribution is
The proof is provided in Appendix § 10.§ 1. Proposition 1 shows that the ideal next-step distribution combines three signals: generator probability (), immediate step correctness (), and a continuation value () measuring how promising the resulting prefix is. This combination is central to VGSC, making the method less dependent on any single imperfect signal.
The main challenge is that is generally not available in closed form. Since the verifier is trained to approximate correctness probabilities, the future verifier factor can be viewed as estimating whether a sampled continuation completes the reasoning correctly from the current prefix. Thus, measures the expected verifier-rated success of continuations from . Estimating this expectation online with Monte Carlo rollouts at every decoding step would be prohibitively expensive. Instead, we use a learned future-outcome predictor to estimate the probability that a continuation from the current prefix reaches the correct final answer.
We train using Monte Carlo continuation labels: for each prefix in the training data, we sample multiple continuations and use the fraction that reach the correct final answer as the target. At decoding time, we apply the same verifier-trust exponent to both probability-valued signals:
VGSC Decoding Algorithm
Starting from an empty prefix, VGSC generates one trajectory by repeating a bounded rejection-resampling procedure. At step , given prefix , define the local acceptance weight
Since and take values in , . We then sequentially propose
and accept proposal with probability . If accepted, we set and stop sampling candidates for this step.
To keep computation bounded, we allow at most proposals per step. If all are rejected, we fall back to weighted selection among the cached proposals:
The selected step is appended to the prefix, and decoding continues until the trajectory is complete.
This procedure is adaptive: high-weight proposals can be accepted early, reducing generated tokens relative to always sampling candidates. The cap bounds the worst-case per-step budget, while fallback ensures decoding can always proceed.
We repeat this independently times to obtain guided trajectories , extract the final answer from each trajectory , and predict by self-consistency:
Theoretical Motivation
The following theorem quantifies the finite-cap bias introduced by bounded rejection-resampling.
Theorem 1 Fix a prefix , and define
Suppose and the practical local guidance weight is exact, so that
Let be the next-step distribution of the bounded VGSC decoder in § 3.3. Then the probability of reaching fallback is
Conditioned on not reaching fallback, the accepted proposal is distributed exactly as . Moreover, assume that . Then there exists a constant , independent of , such that
Discussion. Theorem 1 shows that VGSC approximates the ideal verifier-tilted local conditional with only finite-cap bias. Before the proposal cap is reached, rejection-resampling is exact; approximation arises only when all proposals are rejected. This event has probability , which decays exponentially in the proposal budget and the average acceptance mass . Thus, when candidates have non-negligible guidance weight, even a small cap makes fallback unlikely. With odds-weighted fallback, the resulting total variation error is of order . Increasing therefore quickly reduces bias, while early acceptance preserves adaptive compute savings. Empirically, using only proposals per step is already sufficient to outperform strong baselines (see § 4). We further measure in Appendix § 9.§ 3 that the fallback case is relatively infrequent: it occurs in about 46% of decoding steps for and remains below 71% even under much stronger verifier guidance, .
Appendix § 10 further extends this one-step result to trajectory-level distributions and analyzes stability under imperfect verifier scores. Overall, Theorem 1 shows that VGSC closely targets the verifier-tilted local conditional, and therefore inherits its balance of generator probability, step correctness, and continuation value. This balance reduces overconcentration on the failure modes of either the generator or the verifier.
Practical Motivation
The future-outcome term is practically important because a locally correct step may still fail to advance the solution. For example, a model may restate the problem or enumerate constraints without introducing a useful strategy; such steps can receive high step-verifier scores, while their prefixes remain unlikely to admit a correct continuation. Figure 2 illustrates this effect: step-level guidance alone can prefer locally plausible but stagnant continuations, whereas favors prefixes more likely to lead to a correct solution. This motivates combining immediate correctness with future-outcome prediction during decoding.

Figure 2. Simplified MATH example with Llama 3.2 3B Instruct and a PRM verifier. The future-outcome score favors prefixes likely to admit a correct continuation, steering VGSC toward the correct answer.
Experiments
Experimental setup
Datasets. We evaluate on five reasoning benchmarks: MATH ([7]) and OlympiadBench ([6]) for mathematics, text-only ScienceQA ([17]) for science, CommonsenseQA ([24]) for commonsense reasoning, and Knights&Knaves ([29]) for logic. We use the test splits of MATH and OlympiadBench. All tasks require reasoning with extractable final answers. Table 1 summarizes dataset statistics; prompts, answer extraction, and examples are provided in Appendix § 11.
| Dataset | Domain | # Samples | Mean # Steps | Answer format |
| MATH | Math | 500 | 8.4 | numeric |
| OlympiadBench | Math | 675 | 4.6 | numeric |
| ScienceQA (SciQA) | Science reasoning | 500 | 3.0 | multiple-choice |
| CommonsenseQA (CSQA) | Commonsense reasoning | 500 | 9.9 | multiple-choice |
| Knights&Knaves (K&K) | Logic | 100 | 10.5 | multiple-choice |
Table 1. Summary statistics of the evaluation datasets. Mean number of reasoning steps is computed for Qwen2.5-Math-1.5B-Instruct using our rule-based step splitting procedure.
LLMs. We use three instruction-tuned generators: Qwen2.5 Math 1.5B Instruct [31] in the main text, with additional results for Llama 3.2 3B Instruct [5] and Gemma3 4B Instruct (Kamath et al., 2025) in Appendix F.
Verifiers. Our primary step-level verifier is the frozen Qwen2.5 Math PRM 7B [34, 15]. For future-outcome prediction, we initialize a separate model from the same checkpoint and train using LoRA adapters only in its last four of 28 Transformer layers [8], yielding 46.1M trainable parameters. All remaining parameters are frozen. For each training prefix derived from PRM800K, we sample 10 continuations and train with binary cross-entropy against the fraction that reach the correct final answer. Because PRM800K is derived from the MATH training split, we treat MATH as in-domain for the verifier and the remaining benchmarks as out-of-domain.
In Appendix F, we also evaluate ReProbe [19], a lightweight (20M parameters) verifier trained on hidden states of a frozen LLM. We train separate ReProbe models for step correctness and Monte Carlo continuation prediction. Appendix D.4 provides more details on verifier training.
Evaluation Protocol. We report exact-match accuracy of extracted final answers, averaged over 5 seeds, and average generated tokens per example to measure the compute–accuracy tradeoff.
Baselines
We categorize baseline methods according to how they use verifier information.
No-verifier baselines. Single sample returns one sampled trajectory. Self-consistency (SC) samples trajectories and takes the majority-vote answer.
Post-hoc verifier aggregation. Weighted self-consistency (WSC) samples trajectories and weights each vote by the full-trajectory verifier score [10]. We also evaluate verifier-weighted DLE [14], which uses a specialized sampling procedure to generate diverse trajectories and then applies weighted self-consistency over the resulting set.
Verifier-guided search baselines. Beam search (BS) uses verifier guidance during generation. We evaluate BS-Step, which scores prefixes by the product of step scores , and BS-Future, which scores prefixes by . BS+SC majority-votes over completed beam candidates. Twisted Sequential Monte Carlo (TSMC) follows Zhao et al. [35], using for partial-generation twist estimates and as the final chain potential.
VGSC. VGSC jointly uses and during decoding, as described in § 3. We evaluate , where larger gives stronger verifier guidance.
We use ThinkBooster [23] implementations for all benchmarks and decoding strategies. Chain-of-thought traces are split into reasoning steps using a rule-based procedure; each step is capped at 2048 tokens and each trajectory at 50 steps. All experiments use temperature 1.0. For BS, BS+SC, and VGSC, we use chains and candidates per chain; for SC and WSC, we evaluate ; and for DLE and TSMC, we use . Appendix D provides full prompts, decoding settings, step-splitting details, and verifier-training hyperparameters. Appendix G describes the baseline methods, while Appendix A.1 reports additional baseline variants.
Main Results
Table 2 reports the main results for Qwen2.5 Math 1.5B Instruct with Qwen2.5 Math PRM 7B verifiers; Appendix F reports results for other LLMs (Llama 3.2 3B Instruct and Gemma3 4B Instruct) and verifier families (ReProbe).
| Method | MATH | Olympiad Bench | SciQA | CsQA | K&K | Average Accuracy | Average Tokens | Estimated TFLOPs |
| No-verifier baselines | ||||||||
| Single sample | 587 | 2.1 | ||||||
| SC | 2,928 | 9.1 | ||||||
| SC | 14,778 | 44.6 | ||||||
| Post-hoc verifier aggregation | ||||||||
| WSC | 2,964 | 52.0 | ||||||
| WSC | 14,791 | 253.1 | ||||||
| DLE | 14,655 | 248.8 | ||||||
| Verifier-guided search baselines | ||||||||
| BS-Step | 19,837 | 338.9 | ||||||
| BS-Future | 12,337 | 211.4 | ||||||
| BS+SC-Step | 19,911 | 340.1 | ||||||
| BS+SC-Future | 12,639 | 216.5 | ||||||
| TSMC | 15,045 | 287.7 | ||||||
| Verifier-guided self-consistency (ours) | ||||||||
| VGSC | 10,138 | 194.5 | ||||||
| VGSC | 13,839 | 264.8 | ||||||
| VGSC | 15,179 | 290.2 | ||||||
| VGSC | 16,093 | 307.6 |
Table 2. Accuracy across datasets and average generated tokens per sample for Qwen2.5 Math 1.5B Instruct generator with Qwen2.5 Math PRM 7B verifier. Results are averaged over 5 seeds. TFLOPs are estimated using Eq. (39). The best result in each column is bold and the second-best result is underlined.
Overall, VGSC achieves the strongest average performance among the evaluated decoding strategies. VGSC with achieves the highest average accuracy, 63.1%, exceeding the strongest non-VGSC baseline,
BS+SC-Future, by 2.4 percentage points and WSC with a similar nominal budget of by 3.7 points. Across its variants, VGSC achieves or ties the best result on all five benchmarks. The three settings achieve the three highest average accuracies, suggesting that moderate-to-strong verifier guidance is effective in this setting.
Table 2 also reports generated tokens and estimated inference compute per MATH example. VGSC can accept a proposed step before exhausting its candidate budget, reducing the number of generated tokens. For example, VGSC with achieves 81.2% accuracy using 13,839 tokens per example, compared with 77.4% accuracy and 19,837 tokens for BS-Step, a 30.2% token reduction. This saving is baseline-dependent: BS-Future generates fewer tokens but achieves lower accuracy.
The reported TFLOPs follow Eq. (39) under a conditional shared-prefix implementation. Because the two resulting PRMs differ only in their final four layers, this estimate counts a shared frozen prefix and two four-layer branches, corresponding to 8B effective verifier parameters under an equal-layer approximation. Section 4.4 further examines the accuracy–compute tradeoff across search budgets, and Appendix E details the compute estimates.
Ablations
We examine how VGSC scales with search budget, how it responds to corrupted verifier scores, and how its performance depends on the step-level verifier and future-outcome predictor . Additional appendix ablations examine score aggregation (Appendix A.1), the verifier-trust exponent (Appendix A.2), and allocation under a fixed nominal budget (Appendix A.3). Appendix B provides case studies, coverage statistics, and diversity statistics.
Scaling total budget with balanced search. We study budget scaling on random subsets of 100 examples from MATH, OlympiadBench, and SciQA. Figure 3 studies the compute-accuracy trade-off in a balanced regime ranging from 1 to 8, where increasing raises both the number of independent trajectories and verifier-guided candidate expansions per step. Beam-search baselines remain competitive on MATH, likely because the PRM verifier acts in-domain and is less prone to mis-scoring. On the verifier out-of-domain datasets, OlympiadBench and SciQA, BS variants degrade as the budget grows, starting around on OlympiadBench and on SciQA for BS-Step. This matches Dalal et al. [4]: increasing test-time compute can hurt when search aggressively maximizes noisy verifier scores. In contrast, sampling-based methods are more stable, and VGSC remains among the top-performing methods across nearly all budgets.
Figure 3. Accuracy–compute tradeoff on MATH, OlympiadBench, and SciQA with Qwen2.5-Math-1.5B and PRM-7B verifiers. For search methods, the budget increases as , jointly scaling cross-trajectory exploration and within-step candidate generation; SC and WSC use trajectories. The horizontal axis reports inference compute in TFLOPs per example.
Robustness to corrupted verifier scores. We evaluate robustness on the same random subsets of 100 examples from MATH, OlympiadBench, and SciQA. We perturb verifier logits with Gaussian noise of standard deviation before mapping them back to probabilities with a sigmoid, corrupting both and for methods that use both signals. Figure 4 shows that accuracy generally decreases as noise increases, although the effect varies across methods and datasets. VGSC remains comparatively stable, particularly on SciQA, where beam-search baselines degrade more substantially. These results suggest that joint verifier guidance combined with stochastic exploration can reduce sensitivity to corrupted scores.
Figure 4. Robustness to corrupted verifier scores. We perturb verifier outputs by adding Gaussian noise in logit space with standard deviation and then mapping back to probabilities.
Dependence on verifier components. We examine how VGSC’s performance depends on the step-level verifier and future-outcome predictor on the five main evaluation benchmarks. For VGSC, we compare joint guidance with alone and alone, keeping the remaining decoding settings unchanged. We also test whether combining the signals benefits deterministic search. For BS and BS+SC, we replace the step-only score with
A positive exponent applied to the entire joint score preserves its ranking and therefore does not affect beam selection.
Table 3 shows that joint guidance improves average accuracy for BS from 46.0% to 53.6% and for BS+SC from 57.0% to 60.8%. Both remain below full VGSC, which achieves 61.9% at and 63.1% at . Within VGSC, removing either component reduces average accuracy at both trust settings. These results suggest that performance depends both on the available verifier signals and on how they guide decoding.
| Method | Guidance | Avg. acc. | |
| BS | — | 46.0 | |
| BS | — | 53.6 | |
| BS+SC | — | 57.0 | |
| BS+SC | — | 60.8 | |
| VGSC | only | 5 | 61.0 |
| VGSC | only | 5 | 57.5 |
| VGSC | 5 | 61.9 | |
| VGSC | only | 10 | 61.8 |
| VGSC | only | 10 | 62.4 |
| VGSC | 10 | 63.1 |
Table 3. Effect of verifier components for Qwen2.5 Math 1.5B Instruct with Qwen2.5 Math PRM 7B. Accuracy is averaged across five benchmarks. Bold entries identify the best VGSC variant within each group.
Related Work
Verification for reasoning. Prior work has improved multi-step reasoning by training verifiers to assess either complete solutions or intermediate reasoning steps. These include outcome-level verifiers and process reward models (PRMs) for step-level supervision [3, 15, 26]. Recent work further studies how verifier quality depends on supervision, evaluation design, and verifier architecture [34, 19]. Our focus is complementary: rather than proposing a new verifier, we study how verifier signals should be used at inference time.
Continuation values and progress-based verification. Step correctness and continuation value provide different forms of guidance: a locally valid step need not make the remaining problem easier, while the probability of successful completion depends on both the prefix and the policy used to continue it. Setlur et al. [21] study process-advantage verifiers that assess progress toward successful completion, highlighting the importance of policy-dependent continuation estimates. VGSC similarly uses future-outcome information, but retains a separate step-correctness verifier. Our contribution concerns how these two signals enter a stochastic decoding rule, rather than a new value-estimation objective.
Inference-time verification. A common use of test-time compute is post-hoc aggregation: multiple reasoning trajectories are sampled from the base model and then combined by majority vote or verifier-weighted reranking [27, 3, 10]. These methods preserve diversity, but apply verification only after generation is complete. Other methods inject verifier information during decoding, including tree/search-based reasoning [32], step-level discriminator guidance [13], and self-evaluation guided stochastic beam search [30]. These approaches use local feedback to guide generation, but do not explicitly model the future success probability of continuing from a partial trajectory. Future-looking methods guide search using estimates of continuation quality. Value-guided search (VGS) uses a learned continuation-value model to guide block-wise search and aggregates completed trajectories through value-weighted majority voting [25]. This approach is closely related to our -only variant, which isolates future-outcome guidance within our decoding framework. VGSC additionally incorporates step-level correctness and uses stochastic acceptance/resampling rather than highest-value candidate selection. Twisted sequential Monte Carlo (TSMC) samples from a base language model tilted by a sequence-level potential using learned twists, particle reweighting, and resampling [35]. When this potential is a verifier score, TSMC targets a verifier-tilted distribution , but it treats the verifier as a terminal sequence-level signal and relies on learned twists for intermediate guidance. VGSC instead exploits intermediate process-verifier scores directly, combining and in each local next-step resampling decision while preserving multiple trajectories for final self-consistency.
Imperfect verifier signals. Learned verifiers are imperfect: PRMs can be miscalibrated, overestimate partial trajectory quality, and be exploited when optimized too aggressively at inference time [20, 12]. More broadly, search methods that rely heavily on verifier scores can suffer from over-optimistic selection, and wider verifier-guided search need not monotonically improve performance [4, 2]. This motivates our emphasis on stochastic guidance that balances generator exploration with verifier information, rather than deterministic maximization of noisy scores.
Conclusion
We introduced Verifier-Guided Self-Consistency (VGSC), a test-time reasoning method that combines generator sampling with verifier guidance during decoding. VGSC uses both step-level correctness estimates and future-outcome prediction to guide local sampling, while still producing multiple diverse reasoning trajectories for final self-consistency. This yields a simple alternative to both post-hoc verifier reranking and beam-style maximization.
Across multiple reasoning benchmarks, generator models, and verifier families, our results show that VGSC consistently improves over representative post-hoc aggregation baselines and achieves a stronger balance between exploration and verifier guidance than beam-style search. These findings suggest that inference-time reasoning benefits from stochastic guidance toward promising trajectories rather than aggressive deterministic pruning.
Limitations
VGSC requires auxiliary learned guidance signals at inference time, in particular the future-outcome predictor . In our implementation, is trained from Monte Carlo continuation outcomes for a fixed generator/verifier configuration, which adds some training and engineering overhead relative to simpler decoding schemes that only reuse an existing verifier at test time. We view this as a moderate limitation rather than an unusual requirement: Monte Carlo continuation correctness signals are already commonly used to train PRMs, since they can be obtained automatically without human step-level annotations. VGSC repurposes this same kind of automatically generated supervision to estimate continuation value, and in our experiments this additional signal provides a useful complement to step-level verification.
More broadly, VGSC depends on the quality and task alignment of the learned guidance signals. As with other verifier-guided methods, gains may be smaller when the verifier is weak, miscalibrated, or poorly matched to the target domain. Our results on CommonsenseQA with ReProbe illustrate this regime, where verifier-based guidance is less helpful than in mathematical and logical settings.
Finally, VGSC introduces the verifier-trust exponent as a method-specific hyperparameter. Although we find that a default range of works well across datasets, the best setting may vary with verifier quality, generator behavior, and task domain. This suggests that some validation or tuning may be needed when applying VGSC to new settings.
Ethical Considerations
VGSC aims to improve the reliability of language-model reasoning by using verifier guidance during test-time generation. However, it does not guarantee correctness: generated reasoning can still be flawed, and final answers can still be wrong.
The method depends on learned verifiers and future-outcome predictors, whose scores may be biased, miscalibrated, or poorly aligned with new domains. If these models assign high confidence to incorrect steps, VGSC may amplify such errors, especially under strong verifier guidance. Careful calibration, monitoring, and domain-specific validation are therefore important for practical deployment.
Additional Ablation Studies
This section presents supplementary ablation experiments that examine the robustness of our results to key design choices. In particular, we study alternative ways of aggregating step-level verifier signals and evaluate additional baseline variants that make use of verifier information differently.
Alternative Aggregations
We study whether verifier-guided methods depend on how step-level verifier scores are aggregated along a reasoning chain. All default method variants in the main paper use product aggregation; here, we additionally evaluate min and mean. Specifically, for the Qwen2.5 Math 1.5B Instruct + Qwen2.5 Math PRM 7B setting, we consider WSC, DLE, BS-Step, and BS+SC-Step variants in which the default product score is replaced by one of these alternatives. For a partial trajectory , we define the aggregated verifier score as
where . Product favors consistently high step scores, min is determined by the weakest step, and mean provides a softer notion of average prefix quality.
For VGSC, we analogously aggregate both the step-level verifier scores and future-outcome score into a single guidance signal,
where product is the default VGSC variant and min/mean are the alternatives studied here. We evaluate these variants with , the best setting from the main experiments.
Table 4 shows that aggregation choice has only a small effect on average accuracy. Product remains a strong default, while the comparable performance of min and mean suggests that verifier-guided decoding is not highly sensitive to this choice.
Table 4: Accuracy across datasets for Qwen2.5 Math 1.5B Instruct generator with Qwen2.5 Math PRM 7B verifier. Results are averaged over 5 random seeds. The best result in each column is bold and the second-best result is underlined. Product aggregation has overall the best performance.
| Method | Verifier Aggregation | MATH | OlympiadBench | SciQA | CsQA | K&K | Average Accuracy |
| Post-hoc verifier aggregation | |||||||
| WSC | product | 80.5 | 43.6 | 67.0 | 49.8 | 56.2 | 59.4 |
| WSC | min | 80.1 | 43.2 | 67.5 | 49.6 | 65.3 | 61.1 |
| WSC | mean | 77.5 | 39.3 | 64.0 | 48.0 | <u>55.7</u> | 56.9 |
| DLE | product | 78.6 | 43.8 | 67.5 | 52.8 | 60.4 | 60.6 |
| DLE | min | 79.2 | 42.7 | 67.2 | 52.4 | 58.0 | 59.9 |
| DLE | mean | 76.4 | 41.2 | 67.2 | 50.6 | 55.0 | 58.1 |
| Verifier-guided search baselines | |||||||
| BS-Step | product | 77.4 | 38.5 | 44.8 | 31.1 | 38.0 | 46.0 |
| BS-Step | min | 76.7 | 37.7 | 47.0 | 30.3 | 37.3 | 45.8 |
| BS-Step | mean | 77.5 | 38.2 | 46.3 | 31.0 | 40.7 | 46.7 |
| BS+SC-Step | product | 78.4 | 41.1 | 62.1 | 46.7 | 56.6 | 57.0 |
| BS+SC-Step | min | 77.7 | 41.2 | 63.1 | 48.5 | 56.3 | 57.4 |
| BS+SC-Step | mean | 78.6 | 41.5 | 61.5 | 47.1 | 54.3 | 56.6 |
| Verifier-guided self-consistency (ours) | |||||||
| VGSC | product | 81.1 | 45.0 | 70.3 | <u>54.5</u> | 64.5 | <u>63.1</u> |
| VGSC | min | <u>80.6</u> | <u>45.2</u> | <u>70.2</u> | 51.6 | 61.0 | 61.7 |
| VGSC | mean | 79.4 | 45.9 | <u>70.2</u> | 56.0 | <u>65.0</u> | 63.3 |
Table 4.
Verifier-Trust Parameter Sensitivity
Figure 5 examines the verifier-trust exponent , which controls the strength of verifier guidance during decoding. Across the evaluated datasets, increasing from small values generally improves accuracy, with the strongest performance typically attained around –. Further increases provide limited additional benefit and can reduce accuracy, suggesting that stronger guidance may amplify errors in the verifier signals. These results favor moderate verifier trust, although the best setting varies across datasets. We therefore treat as a task-dependent hyperparameter rather than recommend a single range for all tasks.
Figure 5. VGSC sensitivity to the verifier-trust exponent for 100 sample subsets of 3 datasets.
Compute Allocation under Fixed Budget
Figure 6 fixes the total search budget to and varies how it is split between cross-trajectory exploration, controlled by , and within-step verifier-guided resampling, controlled by . For reference, SC and WSC use 36 complete trajectories. VGSC remains competitive across almost all allocations, indicating that its gains do not depend on a narrowly tuned choice of and . The best performance is typically achieved at intermediate allocations, around , suggesting that both sources of diversity are useful: too few trajectories limits final self-consistency, while too few candidates per step weakens verifier-guided steering. This supports the balanced setting used in the main experiments.
Figure 6. Allocation of search compute under a fixed budget. We fix and vary how the budget is split between the number of reasoning chains and candidates .
Analysis and Examples
Analysis and Examples
In this section we provide a qualitative and quantitative analysis of the decoding strategies considered in the main paper.
Qualitative Examples
We first present qualitative examples illustrating the characteristic failure modes of baseline methods as well as VGSC. We analyze representative generations from OlympiadBench and SciQA using Qwen2.5 Math 1.5B Instruct and Llama 3.2 3B Instruct generators with the Qwen2.5 Math PRM 7B verifier for each baseline.
Example 1: VGSC vs WSC. A typical failure mode of WSC arises when all sampled trajectories are incorrect because the generator fails to produce a correct solution, even though the verifier can identify low-quality or erroneous intermediate steps. Since WSC applies verifier information only after complete trajectories have already been generated, it cannot use this signal to redirect decoding toward a more promising solution. In contrast, VGSC uses verifier guidance during decoding, which can steer generation away from erroneous continuations and toward a correct final answer. Figure 7 shows a representative example of this behavior.
Problem.
Let be a hexagon inscribed in a circle of radius 5. Given that and , compute .
Gold answer:
WSC trajectory (truncated).
Step 1 (): Since , these four equal chords subtend equal central angles; let each such angle be .
Step 2 (): The four equal angles account for half the circle, so , hence .
Step 3 (): Therefore, . ✗
VGSC trajectory (truncated).
Step 1 (): Let . Let each length-6 chord subtend central angle , and let each chord of length subtend central angle .
Step 2 (): Then , so .
Step 3 (): From , we get and , hence .
Step 4 (): So, . ✓
Figure 7. Truncated example from OlympiadBench where WSC fails but VGSC succeeds. WSC samples complete trajectories from the base generator and applies verifier information only after generation, so it cannot redirect decoding once it enters an incorrect line of reasoning. In this example, all WSC trajectories contain low-scored incorrect steps and end in incorrect answers. VGSC instead uses verifier guidance during decoding and is steered toward a correct solution.
Example 2: VGSC vs BS-Step. A common failure mode of BS-Step is that it overemphasizes local correctness, leading the search to prefer steps that are individually plausible but do not make meaningful progress toward the final solution. As a result, beam search can become trapped in locally high-scoring but unproductive reasoning paths. VGSC mitigates this issue by incorporating future-outcome guidance in addition to step-level verification, which favors trajectories that are both locally sound and globally promising. Figure 8 illustrates this pattern.

Figure 8. Problem. For a positive integer , define a sequence of integers by letting and for . Let . Determine the largest possible such that, for some positive integer , the numbers are all prime. Gold answer: 2. BS-Step trajectory (truncated). Step 1 (): We analyze the sequences and , where , , and V_{\phi}=1.000$): The first terms are and . Step 3 (): Next, and . Step 4 (): Next, and . Step 5 (): Next, and . Step 11 (): and . VGSC trajectory (truncated). Step 1 (): If $2^m-1$ is prime, then must be prime. Therefore, if and are both prime, then and must both be prime. Step 2 (): Taking gives , , and . Hence and are prime, while is composite. Thus . Step 3 (): It remains to show that cannot exceed 2. If is an odd prime and is prime, then is an odd prime with . Let . If is composite, then is composite. Step 4 (): If is prime, then , so . Hence , contradicting the primality of . Therefore is never prime, and the largest possible value is 2.
caption: Truncated example where BS-Step fails but VGSC succeeds. BS-Step ranks partial trajectories only by local step-verifier scores , so it keeps selecting steps that are individually correct but do not move the solution forward. In this example, beam search repeatedly expands the recurrence instead of reaching the key primality and divisibility argument. VGSC adds future-outcome guidance, which favors trajectories that make progress toward a complete proof and therefore reaches the correct answer.
Example 3: VGSC vs BS-Future. A common failure mode of BS-Future is over-concentration on a narrow region of the search space when the future-outcome predictor is imperfect. In such cases, beam search may continue expanding a flawed partial trajectory because its future score is somewhat inflated, even when that trajectory is ultimately unproductive. VGSC instead combines verifier guidance with generator sampling, which preserves exploration and reduces sensitivity to this kind of predictor noise. Figure 9 provides a representative example.

Figure 9. Problem. Hannah flips two fair coins, while Otto flips fair coins. Let be the probability that the number of heads on Hannah’s coins is greater than the number of heads on Otto’s coins. If , where and are relatively prime positive integers, compute . Gold answer: 17. BS-Future trajectory (truncated). Step 1 (): Let be the number of heads from Hannah’s two coins, , , and . Step 2 (): - number of heads from Otto’s coins, . Step 3 (): There are 3 cases when : ; or . Step 4 (): Case 1: . Step 5 (): Case 2: . Step 6 (): Case 3: . Step 7 (): Case 4: . VGSC trajectory (truncated). Step 1 (): Let be the number of heads from Hannah’s two coins and the number of heads from Otto’s five coins. Then , , and , while . Step 2 (): The event can occur only in two cases: or . Step 3 (): Therefore, . Step 4 (): Compute , and . Hence , so .
caption: Truncated example from OlympiadBench where BS-Future fails but VGSC succeeds. BS-Future ranks partial trajectories only by the future-outcome predictor , so moderately inflated future scores can keep the beam focused on a flawed branch. In this example, beam search commits to an incorrect case analysis and keeps expanding it without reaching other cases. VGSC preserves exploration due to independent sampling from verifier-tilted distribution and reaches the correct probability computation in majority of samples.
Example 4: VGSC failure. VGSC can also fail when the verifier is poorly aligned with the task. In this regime, the verifier may systematically prefer incorrect trajectories, so that guiding decoding toward the verifier-tilted distribution becomes harmful rather than helpful. This can make verifier-guided methods underperform simpler generator-only aggregation methods such as self-consistency. Figure 10 shows a representative failure case of this kind.

Figure 10. VGSC failure case from SciQA. In this example, plain self-consistency over base-model samples returns the correct answer 1, but the verifier is poorly aligned with the task and assigns higher scores to chains that confuse dictionary guide words with semantic relatedness. Consequently, WSC selects the incorrect answer 2, and VGSC is pulled toward the same incorrect region under verifier-guided decoding. This shows that misaligned verifier signals can make verifier-tilted sampling worse than using the base generator alone.
Coverage and chain diversity
To better understand the behavior of different methods, we complement the main accuracy results with simple statistics of candidate-set quality and whole-chain diversity. For each example, let denote the completed reasoning chains generated by a method. We report:
Accuracy: final exact-match accuracy after aggregation.
Pass@N: whether at least one of the generated chains is correct.
Diversity: one minus the mean pairwise ROUGE-L similarity [16] between completed chains from the same example:
Higher values of Pass@N and Diversity indicate better candidate coverage and greater diversity.
Table 6 reports the average values of the above metrics for the two datasets, highlighting the key differences among methods. SC and WSC often generate diverse reasoning chains; however, because they do not incorporate verifier guidance during generation, they tend to achieve lower accuracy. By contrast, BS-Step applies verifier guidance throughout decoding, which improves accuracy but often reduces diversity, as the search concentrates on a narrow set of trajectories. VGSC offers a middle ground: it generates multiple diverse chains while still leveraging verifier guidance during decoding, achieving strong Accuracy and Pass@N without suffering the same degree of diversity collapse as beam search.
| Method | MATH | OlympiadBench | ||||
| Accuracy ↑ | Pass@N ↑ | Diversity ↑ | Accuracy ↑ | Pass@N ↑ | Diversity ↑ | |
| No-verifier baselines | ||||||
| SC | 76 | 80 | 0.487 | 40 | 46 | 0.551 |
| Post-hoc verifier aggregation | ||||||
| WSC | 78 | 80 | 0.487 | 45 | 46 | 0.551 |
| Verifier-guided search baselines | ||||||
| BS-Step | 77 | 78 | 0.147 | 38 | 43 | 0.177 |
| BS-Future | 80 | 82 | 0.140 | 43 | 46 | 0.168 |
| Verifier-guided self-consistency (ours) | ||||||
| VGSC | 80 | 85 | 0.284 | 47 | 52 | 0.436 |
| VGSC | 79 | 84 | 0.277 | 45 | 49 | 0.429 |
Table 6. Accuracy, average pass@N and whole-chain diversity statistics on random 100-example subsets of MATH and OlympiadBench using Qwen2.5 Math 1.5B Instruct with the Qwen2.5 Math PRM 7B verifier.
Fallback Statistics
Theorem 1 shows that finite-cap bias occurs only when all proposed next steps are rejected and VGSC falls back to restricted selection. We therefore measure how often this fallback occurs under the default cap . Table 5 reports fallback rates for different verifier-trust exponents . As expected, fallback becomes more frequent as increases, since larger values make more selective. Nevertheless, even at , fewer than 70% of decoding steps reach fallback on average. On MATH, fallback is even rarer, occurring in only 10%–26% of steps across variants.
| Dataset | MATH | Olympiad Bench | SciQA | CsQA | K&K | Average | |
| 10.76% | 43.66% | 57.15% | 63.54% | 56.23% | 46.27% | ||
| 16.88% | 73.97% | 72.00% | 78.26% | 68.01% | 61.82% | ||
| 20.45% | 80.79% | 76.34% | 82.59% | 73.21% | 66.68% | ||
| 26.12% | 85.49% | 80.27% | 85.59% | 77.48% | 70.99% |
Table 5. Fraction of VGSC decoding steps that reach fallback for Qwen2.5 Math 1.5B Instruct + PRM.
Theoretical Results
This section proves Proposition 1 and Theorem 1. We also derive a trajectory-level implication of the one-step bound and analyze the stability of verifier-tilted distributions under imperfect verifier scores.
Proof of Proposition 1
By definition,
For a fixed prefix and candidate step , the conditional probability under is obtained by summing over all suffixes:
Using the chain rule for the generator and the stepwise verifier factorization, and absorbing all prefix-dependent terms into the proportionality constant, we get
This proves Proposition 1.
Proof of Theorem 1
Fix a prefix and write
By the exact-guidance assumption,
If , fallback has probability zero and the result is immediate. Thus assume .
Let be the event that all proposals are rejected. A single proposal is accepted with marginal probability , so
Conditioned on not reaching fallback, the output is exactly distributed as . Indeed, the probability that the first accepted proposal has value and occurs at attempt is . Therefore
Dividing by gives
It remains to bound the fallback contribution. Conditioned on rejection, each cached proposal is distributed as
The fallback uses odds weights
Hence
so the fallback step is a sampling-importance-resampling (SIR) approximation to from samples drawn from .
Let denote the marginal distribution of the fallback sample. Define
The boundedness assumption implies and
By the standard non-asymptotic bias bound for self-normalized importance sampling over bounded test functions [1 Theorem 2.1], and since the marginal law of the SIR fallback sample is the expectation of the self-normalized weighted empirical measure, there exists a constant , independent of , such that
Finally, the bounded VGSC transition decomposes as
Therefore,
Trajectory-level implication.
The one-step bound also gives a simple bound on complete trajectories. Assume that all generated trajectories have length at most . Let denote the ideal local conditional at step , and let denote the bounded VGSC transition. Define
By Theorem 1, for every reachable prefix,
Let be the trajectory distribution induced by bounded VGSC, and let be the trajectory distribution obtained by composing the ideal local conditionals . A standard step-by-step maximal coupling gives
Therefore,
In particular, if and for all reachable prefixes, then
When the unclipped exact weights are used, . When clipped weights are used, is the corresponding clipped verifier-guided trajectory distribution.
Stability under imperfect verifier scores.
The preceding results assume exact local guidance weights. We now note that the verifier-tilted target is stable to small verifier errors in log space. Let denote an ideal verifier and let denote an imperfect verifier. Define
Suppose that, for all trajectories with ,
Then
Indeed, writing
the log-error assumption implies . The density ratio between the two tilted distributions is proportional to , and a standard bounded-ratio argument gives the stated total-variation bound.
This bound highlights the role of : stronger verifier guidance can improve search when the verifier is informative, but it also amplifies verifier errors linearly for small . This supports treating as a verifier-trust parameter rather than always making it large.
Experimental Details
Datasets, Prompts, and Examples
We evaluate on five benchmarks covering open-ended mathematical reasoning and multiple-choice scientific, commonsense, and logical reasoning. MATH and OlympiadBench are evaluated as open-ended generation tasks, while filtered text-only subsets of ScienceQA, CommonsenseQA, and Knights&Knaves are evaluated as multiple-choice tasks. Table 1 reports subset sizes and mean reasoning-step counts for the Qwen2.5-Math-1.5B-Instruct generator.
For multiple-choice datasets, we map each gold answer to its answer-choice index and evaluate the predicted boxed integer. Table 7 summarizes the prompts. We keep prompt semantics fixed across generator families, changing only model-specific chat-template packaging when needed.
| Benchmark(s) | Task format | Prompt template | Expected final answer | |||
| MATH500, OlympiadBench | Open-ended | Please reason step by step, and put your final answer within \boxed{}. | \boxed{<answer>} | |||
| ScienceQA, CommonsenseQA | Multiple choice | Solve the problem carefully. Show your reasoning in short numbered steps. Then on the last line, output only the positive integer of the answer choice in exactly this format: `\boxed{(1 | 2 | 3 | ...)}`. | , |
| Knights&Knaves | Multiple choice (binary) | Solve the problem carefully. Show your reasoning in short numbered steps. Then on the last line, output only the positive integer of the answer choice in exactly this format: `\boxed{(1 | 2)}`. | , |
Table 7. Prompt templates used for each benchmark group. The semantic content of the prompts is kept fixed across generator families; only the model-specific chat-template packaging is changed when needed.
Table 8 provides example questions from each benchmark.
| Dataset | Example input question | Gold answer |
| MATH500 | Convert the point in rectangular coordinates to polar coordinates. Enter your answer in the form , where and . | |
| OlympiadBench | Xenia and Sergey play the following game. Xenia thinks of a positive integer not exceeding 5000. Then she fixes 20 distinct positive integers such that, for each , the numbers and are congruent modulo . By a move, Sergey tells Xenia a set of positive integers not exceeding 20, and she tells him back the set without spelling out which number corresponds to which index. How many moves does Sergey need to determine for sure the number Xenia thought of? | 2 |
| ScienceQA | Which figure of speech is used in this text? Sing, O goddess, the anger of Achilles son of Peleus, that brought countless ills upon the Achaeans. — Homer, The Iliad Options: (1) chiasmus (2) apostrophe | 2 |
| CommonsenseQA | A revolving door is convenient for two direction travel, but it also serves as a security measure at a what? Choices: (1) bank (2) library (3) department store (4) mall (5) new york | 1 |
| Knights&Knaves | A very special island is inhabited only by knights and knaves. Knights always tell the truth, and knaves always lie. You meet 2 inhabitants: Zoey, and Oliver. Zoey remarked, “Oliver is not a knight”. Oliver stated, “Oliver is a knight if and only if Zoey is a knave”. So who is Zoey? Choices: (1) knight (2) knave | 2 |
Table 8. Examples from the five evaluation datasets using the exact dataset paths and splits from our experiments. For multiple-choice datasets, we show the full prompted question including answer choices, since the model is instructed to return the boxed index of the selected option.
Answer Extraction and Evaluation
We use the default answer-extraction and evaluation logic implemented in the ThinkBooster library [23]. All reported results are exact-match accuracies computed on the extracted final answer.
Answer extraction. For each model output, we first extract the contents of the last complete \boxed{} expression. If no complete boxed answer is found, we fall back to the last parseable numeric span (integer or decimal) in the generated text. For the multiple-choice datasets (ScienceQA, CommonsenseQA, and
Knights&Knaves), we explicitly prompt the model to output the answer-choice index in boxed form, e.g. . This allows us to use a single extraction pipeline across both open-ended and multiple-choice benchmarks.
Answer evaluation. For ScienceQA, CommonsenseQA, and Knights&Knaves, exact-match is computed against the mapped gold answer-choice index. For MATH and OlympiadBench, we use a math-aware exact-match evaluator. The evaluator first normalizes the extracted answer and then applies a cascade of checks: normalized string match, numeric comparison, and symbolic comparison when both the prediction and the reference can be parsed as mathematical expressions. The symbolic comparison uses SymPy parsing utilities, so algebraically equivalent expressions are counted as correct [18].
Step Splitting and Decoding Details
Our decoding strategies operate over reasoning steps rather than individual tokens, following the step-based decoding framework implemented in ThinkBooster [23].
Rule-based step splitting. We use a rule-based detector to identify candidate step boundaries from discourse and structural markers such as first, next, then, finally, therefore, thus, let’s check, consider, paragraph breaks, and numbered lists. We additionally include a small set of custom markers, including for example, given that, and similarly, to improve consistency across model families.
Decoding settings. Unless otherwise noted, all main-paper experiments use the settings in Table 9 and Table 10. We enforce a minimum of 10 tokens per step, cap each step at 2048 tokens, and cap each trajectory at 50 reasoning steps.
| Setting | Value |
| Minimum tokens per step | 10 |
| Maximum tokens per step | 2048 |
| Maximum reasoning steps per trajectory | 50 |
| Total maximum tokens | 4096 |
| Temperature | 1.0 |
| Top-p, top-k | 1.0, 1000 |
| Repetition penalty | 1.2 |
| Random seeds |
Table 9. Shared decoding settings used in experiments.
| Method | Inference budget |
| SC | reasoning chains |
| WSC | reasoning chains |
| DLE | reasoning chains |
| TSMC | particles |
| BS | chains, candidate expansions |
| BS+SC | chains, candidate expansions |
| VGSC | chains, candidate expansions |
Table 10. Method-specific inference budgets used in the main experiments.
Verifier Training Details
We train two verifier families: a PRM-based verifier and a ReProbe verifier [19]. For each family, we train separate models for step correctness and step continuation.
Training data and supervision. Following [19], we construct verifier training data from the PRM800k corpus [15]. For each generator family, we first sample 10,828 questions, generate 3 reasoning trajectories per question, and split each trajectory into steps using the same rule-based splitter used at test time. From every intermediate step prefix , we then sample 10 Monte Carlo continuations using the same generation settings as at test time.
An LLM judge GPT 4.1 annotates each trajectory with the index of its first incorrect step. We derive two supervision targets from these annotations. For the continuation target, the label for prefix is the fraction of its sampled continuations that reach the correct final answer:
For the correctness target, if the first incorrect step is , then steps receive label 1, step receives label 0, and all later steps are ignored.
Training settings. For the PRM family, we use the original Qwen2.5 Math PRM 7B for the local correctness estimate , and for continuation values , we fine-tune LoRA adapters only in its last 4 layers (rank 128, scale 256), freezing the embedding and first 24 layers. For the ReProbe family, we train the local-correctness and continuation probes separately from scratch, using the same overall training setup and implementations from Shelmanov et al. [22]. The number of training epochs was selected based on the best validation-set accuracy. Table 11 summarizes the verifier training hyperparameters.
| Parameter | PRM verifier | ReProbe verifier |
| Architecture details | LoRA fine-tuning of Qwen2.5 Math PRM 7B; last 4 layers only; rank 128; alpha 256; dropout 0.05; attentions and MLP projections | Last hidden layer from LLM as features; ReProbe uses a 2-layer transformer head with dim. 768, 8 heads, and dropout 0.1 |
| Trainable parameters | 46.1M | 15m - 19m depending on base LLM |
| Maximum length | 2048 | 2048 |
| Batch size | 32 (1 per device, 32 accumulation steps) | 16 |
| Learning rate | ||
| Epochs | 5 | 6 |
| Validation split | 10% | 10% |
| Training seed | 42 | 42 |
| Regularization | — | Weight decay 0.1, max grad norm 1.0 |
Table 11. Hyperparameters for the two verifier families. For PRM, the step verifier is frozen and the future verifier uses last-layer QLoRA; ReProbe variants are trained separately for each supervision target.
Inference Compute Estimate
We estimate inference compute as model FLOPs. For a dense decoder-only Transformer, we approximate a forward pass over tokens with a -parameter model as FLOPs [11]. Let be the average generated-token count, is the mean prompt length, and the parameter count of counted component (generator or verifier). With cached incremental decoding, we estimate
Table 12 reports a shared-prefix inference estimate, not measured compute of our separate-PRM implementation. For a 28-layer PRM with only the last 4 layers adapted, two verifier branches count 8B effective parameters under the equal-layer approximation, instead of 14B. Including the generator gives 9.5B for Qwen and 11B for Llama. A single future PRM still requires the full 7B forward pass. Savings require identical input tokens and cached-prefix states for both branches; they do not follow from freezing weights alone.
| Generator + Verifier | Qwen2.5 Math 1.5B Instruct + Qwen2.5 Math PRM 7B | Llama 3.2 3B Instruct + Qwen2.5 Math PRM 7B | Qwen2.5 Math 1.5B Instruct + ReProbe | ||||||
| Method | Tokens | Params | TFLOPs | Tokens | Params | TFLOPs | Tokens | Params | TFLOPs |
| Single sample | 587 | 1.5B | 2.1 | 754 | 3.0B | 5.1 | 638 | 1.5B | 2.2 |
| SC () | 2,928 | 1.5B | 9.1 | 3,058 | 3.0B | 18.9 | 3,206 | 1.5B | 9.9 |
| SC () | 14,778 | 1.5B | 44.6 | 15,288 | 3.0B | 92.3 | 16,021 | 1.5B | 48.4 |
| WSC () | 2,964 | 8.5B | 52.0 | 3,029 | 10.0B | 62.5 | 3,243 | 1.519B | 10.1 |
| WSC () | 14,791 | 8.5B | 253.1 | 15,260 | 10.0B | 307.1 | 16,036 | 1.519B | 49.0 |
| BS-Step | 19,837 | 8.5B | 338.9 | 42,567 | 10.0B | 853.3 | 21,905 | 1.519B | 66.8 |
| BS-Future | 12,337 | 8.5B | 211.4 | 15,646 | 10.0B | 314.9 | 20,539 | 1.519B | 62.7 |
| VGSC () | 13,839 | 9.5B | 264.8 | 33,158 | 11.0B | 731.6 | 11,225 | 1.538B | 34.8 |
| VGSC () | 15,179 | 9.5B | 290.2 | 32,970 | 11.0B | 727.5 | 11,240 | 1.538B | 34.9 |
Table 12. Estimated inference compute. FLOPs are calculated using equation (39). PRM VGSC counts a shared frozen prefix and two last-layer branches (equal-layer estimate); ReProbe VGSC counts two 19M probes.
| Method | MATH | Olympiad Bench | SciQA | CsQA | K&K | Average Accuracy | Average Tokens | Estimated TFLOPs |
| No-verifier baselines | ||||||||
| Single sample | 36.2 ± 0.5 | 11.2 ± 0.3 | 51.7 ± 0.6 | 42.3 ± 0.3 | 29.0 ± 1.9 | 34.1 ± 0.4 | 754 | 5.1 |
| SC | 38.6 ± 0.9 | 10.0 ± 0.2 | 69.5 ± 0.7 | 59.8 ± 0.2 | 36.6 ± 2.0 | 42.9 ± 0.5 | 3,058 | 18.9 |
| SC | 50.9 ± 0.1 | 16.4 ± 0.3 | 82.7 ± 0.5 | 71.4 ± 0.2 | 49.4 ± 1.7 | 54.2 ± 0.4 | 15,288 | 92.3 |
| Post-hoc verifier aggregation | ||||||||
| WSC | 46.4 ± 0.5 | 13.7 ± 0.3 | 68.5 ± 0.7 | 61.1 ± 0.6 | 34.8 ± 1.3 | 44.9 ± 0.3 | 3,029 | 62.5 |
| WSC | 57.3 ± 0.5 | 19.6 ± 0.2 | 80.7 ± 0.5 | 71.8 ± 0.5 | 47.0 ± 2.2 | 55.3 ± 0.5 | 15,260 | 307.1 |
| DLE | 59.8 ± 0.4 | 28.3 ± 0.2 | 77.6 ± 0.1 | 72.2 ± 0.2 | 58.0 ± 1.0 | 59.2 ± 0.2 | 14,953 | 301.0 |
| Verifier-guided search baselines | ||||||||
| BS-Step | 52.3 ± 0.5 | 16.4 ± 0.3 | 53.0 ± 0.5 | 43.7 ± 0.6 | 14.0 ± 1.6 | 35.9 ± 0.4 | 42,567 | 853.3 |
| BS-Future | 64.8 ± 0.4 | 27.3 ± 0.4 | 76.1 ± 0.8 | 67.0 ± 0.4 | 41.0 ± 1.0 | 55.2 ± 0.3 | 15,646 | 314.9 |
| BS+SC-Step | 61.0 ± 0.3 | 22.3 ± 0.2 | 78.1 ± 0.5 | 65.6 ± 0.2 | 54.4 ± 1.8 | 56.3 ± 0.4 | 42,944 | 860.8 |
| BS+SC-Future | 65.7 ± 0.5 | 28.1 ± 0.4 | 79.3 ± 0.3 | 70.7 ± 0.6 | 61.5 ± 2.2 | 61.1 ± 0.5 | 15,270 | 307.3 |
| TSMC | 57.6 ± 0.2 | 21.4 ± 0.4 | 80.7 ± 0.4 | 69.0 ± 0.6 | 54.0 ± 1.8 | 56.5 ± 0.4 | 15,873 | 351.3 |
| Verifier-guided self-consistency (ours) | ||||||||
| VGSC | 63.6 ± 0.6 | 24.5 ± 0.3 | 83.7 ± 0.4 | 71.5 ± 0.2 | 60.5 ± 1.8 | 60.8 ± 0.4 | 28,715 | 633.9 |
| VGSC | 67.4 ± 0.4 | 28.7 ± 0.2 | 84.0 ± 0.4 | 72.2 ± 0.3 | 62.5 ± 2.1 | 63.0 ± 0.4 | 33,158 | 731.6 |
| VGSC | 67.0 ± 0.7 | 30.0 ± 0.2 | 83.4 ± 0.4 | 72.5 ± 0.5 | 59.5 ± 1.3 | 62.5 ± 0.3 | 32,970 | 727.5 |
| VGSC | 68.6 ± 0.2 | 28.3 ± 0.3 | 84.4 ± 0.3 | 71.1 ± 0.3 | 64.0 ± 2.9 | 63.3 ± 0.6 | 33,340 | 735.6 |
Table 13. Accuracy ↑ across datasets and average generated tokens per sample for Llama 3.2 3B Instruct generator with Qwen2.5 Math PRM 7B verifier. Results are averaged over 5 random seeds. The best result in each column is bold and the second-best result is underlined.
When the verifier is lightweight relative to the generator, VGSC’s generated-token savings can outweigh the cost of the additional future-outcome verifier. This is reflected in the ReProbe setting: counting one 19M probe for WSC/BS and two 19M probes for VGSC, VGSC has lower estimated FLOPs than the high-budget ReProbe baselines while using substantially fewer generated tokens. We view this lightweight-verifier regime as natural in practical deployments, where verifiers are often chosen to be smaller and cheaper than the generator they guide.
Additional Results
The main text reports results for the Qwen2.5 Math 1.5B Instruct model using the Qwen2.5 Math PRM 7B verifier. Here, we present additional results for Llama 3.2 3B Instruct and Gemma3 4B Instruct with the same verifier. For Qwen2.5 Math 1.5B Instruct, we also evaluate a second verifier family, ReProbe [19]. Tables 13, 14 and 15 show that the main trend is consistent across LLMs and verifiers: VGSC remains competitive and typically outperforms the corresponding baselines, suggesting that the benefits of verifier-guided self-consistency are not limited to a single LLM or verifier family.
Table 14: Accuracy across datasets and average generated tokens per sample for Gemma3 4B Instruct generator with Qwen2.5 Math PRM 7B verifier. Results are averaged over 5 random seeds. The best result in each column is bold and the second-best result is underlined.
| Method | MATH | Olympiad Bench | SciQA | CsQA | K&K | Average Accuracy | Average Tokens | Estimated TFLOPs |
| No-verifier baselines | ||||||||
| Single sample | 67.2 0.5 | 29.6 0.3 | 68.9 0.5 | 68.1 0.7 | 60.0 1.6 | 58.8 0.4 | 845 | 7.5 |
| SC | 74.7 0.4 | 36.7 0.2 | 75.1 0.7 | 70.4 0.4 | 66.8 1.0 | 64.7 0.3 | 4,315 | 35.3 |
| SC | 78.6 0.4 | 41.6 0.2 | 74.3 0.3 | 72.9 0.3 | 67.6 1.1 | 67.0 0.2 | 21,581 | 173.4 |
| Post-hoc verifier aggregation | ||||||||
| WSC | 76.0 0.4 | 37.9 0.4 | 79.3 0.4 | 72.8 0.3 | 68.4 1.0 | 66.9 0.3 | 4,308 | 96.9 |
| WSC | 80.8 0.3 | 43.4 0.3 | 80.8 0.3 | 74.8 0.2 | 71.4 1.1 | 70.2 0.2 | 21,573 | 476.7 |
| DLE | 77.4 0.5 | 42.0 0.4 | 80.6 0.4 | 74.4 0.3 | 65.2 1.7 | 67.9 0.3 | 13,285 | 294.4 |
| Verifier-guided search baselines | ||||||||
| BS-Step | 73.3 0.4 | 33.4 0.2 | 80.6 0.5 | 70.7 0.4 | 70.4 0.9 | 65.7 0.2 | 29,586 | 653.0 |
| BS-Future | 77.6 0.3 | 37.7 0.3 | 81.3 0.4 | 72.9 0.3 | 65.5 0.9 | 67.0 0.2 | 11,329 | 251.4 |
| BS+SC-Step | 75.6 0.4 | 36.9 0.2 | 79.6 0.5 | 71.2 0.5 | 73.5 1.2 | 67.4 0.3 | 29,562 | 652.5 |
| BS+SC-Future | 76.9 0.2 | 40.3 0.3 | 79.0 0.2 | 70.8 0.3 | 67.0 0.8 | 66.8 0.2 | 11,562 | 256.5 |
| TSMC | 79.5 0.3 | 42.3 0.3 | 81.1 0.6 | 71.6 0.2 | 71.0 1.3 | 69.1 0.3 | 16,295 | 393.4 |
| Verifier-guided self-consistency (ours) | ||||||||
| VGSC | 81.3 0.4 | 42.9 0.5 | 82.0 0.4 | 73.2 0.3 | 72.0 0.7 | 70.3 0.2 | 18,846 | 454.6 |
| VGSC | 81.5 0.3 | 43.6 0.5 | 82.3 0.4 | 73.4 0.3 | 70.5 0.9 | 70.3 0.2 | 19,553 | 471.6 |
| VGSC | 82.0 0.2 | 43.1 0.2 | 80.6 0.3 | 73.4 0.3 | 69.5 1.2 | 69.7 0.3 | 19,740 | 476.1 |
| VGSC | 81.0 0.3 | 43.0 0.1 | 81.0 0.3 | 73.6 0.2 | 73.5 1.0 | 70.4 0.2 | 19,696 | 475.0 |
Table 14.
Table 15: Accuracy across datasets and average generated tokens per sample for Qwen2.5 Math 1.5B Instruct generator with ReProbe verifier. Results are averaged over 5 random seeds. The best result in each column is bold and the second-best result is underlined.
| Method | MATH | Olympiad Bench | SciQA | CsQA | K&K | Average Accuracy | Average Tokens |
| No-verifier baselines | |||||||
| Single sample | 66.8 0.4 | 29.9 0.4 | 46.2 0.4 | 29.8 0.4 | 38.0 1.0 | 42.1 0.2 | 638 |
| SC () | 70.0 0.4 | 34.2 0.4 | 57.0 0.4 | 39.8 0.4 | 53.0 1.0 | 50.8 0.2 | 3,206 |
| SC () | 73.6 0.4 | 38.5 0.4 | 63.8 0.4 | 47.8 0.4 | 54.0 1.0 | 55.5 0.2 | 16,021 |
| Post-hoc verifier aggregation | |||||||
| WSC () | 72.7 0.4 | 35.3 0.4 | 56.8 0.4 | 38.0 0.4 | 51.2 1.0 | 50.8 0.2 | 3,243 |
| WSC () | 75.0 0.4 | 40.0 0.4 | 62.2 0.4 | 43.6 0.4 | 54.2 1.0 | 55.0 0.2 | 16,036 |
| DLE | 74.4 0.5 | 39.3 0.5 | 62.0 0.3 | 44.4 0.5 | 55.6 2.0 | 55.1 0.4 | 13,510 |
| Verifier-guided search baselines | |||||||
| BS-Step | 73.2 0.4 | 35.1 0.4 | 33.7 0.4 | 19.2 0.4 | 32.0 1.0 | 38.6 0.2 | 21,905 |
| BS-Future | 72.0 0.4 | 34.5 0.4 | 39.8 0.4 | 25.7 0.4 | 41.2 1.0 | 42.6 0.2 | 20,539 |
| BS+SC-Step | 71.7 0.4 | 36.2 0.4 | 56.7 0.4 | 38.2 0.4 | 54.2 1.0 | 51.4 0.2 | 21,891 |
| BS+SC-Future | 73.5 0.4 | 37.5 0.4 | 55.7 0.4 | 38.0 0.4 | 56.6 1.0 | 52.3 0.2 | 20,551 |
| TSMC | 68.8 0.4 | 33.8 0.4 | 56.8 0.4 | 38.0 0.4 | 50.0 1.0 | 49.5 0.2 | 12,827 |
| Verifier-guided self-consistency (ours) | |||||||
| VGSC () | 72.8 0.4 | 38.8 0.4 | 60.7 0.4 | 39.2 0.4 | 57.0 1.0 | 53.7 0.2 | 10,917 |
| VGSC () | 72.9 0.4 | 39.6 0.4 | 62.7 0.4 | 41.1 0.4 | 58.8 1.0 | 55.0 0.2 | 11,225 |
| VGSC () | 73.7 0.4 | 38.4 0.4 | 64.0 0.4 | 41.7 0.4 | 56.8 1.0 | 54.9 0.2 | 11,240 |
| VGSC () | 74.8 0.4 | 40.6 0.4 | 60.7 0.4 | 45.1 0.4 | 59.6 1.0 | 56.2 0.2 | 11,680 |
Table 15.
Baselines Description
This appendix briefly describes the baseline decoding strategies used in our experiments. Throughout, denotes the input problem, a reasoning trajectory consistent of reasoning steps, the base generator distribution, and the extracted final answer.
No-verifier baselines
Single sample. Single sample generates one reasoning trajectory
and returns the extracted final answer . This is the minimal-compute baseline and can be viewed as the special case of self-consistency with .
Self-consistency (SC). Self-consistency [27] samples independent reasoning trajectories
and predicts the majority answer:
SC does not use verifier information during generation or aggregation.
Post-hoc verifier aggregation
Weighted self-consistency (WSC). Weighted self-consistency [10] uses the same sampled trajectories as SC, but weights each vote by a verifier score for the full trajectory. Concretely, it predicts
Thus, WSC uses verifier information only after complete reasoning trajectories have already been generated, without affecting the decoding process itself.
DLE. Distinct Leaf Enumeration (DLE; Li et al. [14]) is a deterministic test-time procedure that explores a truncated next-token decoding tree based on sampling rules and returns up to distinct completed reasoning chains. In our experiments, DLE is used strictly as a post-hoc verifier baseline: the base generator runs the external DLE enumerator (leaf budget , temperature , top-, truncation parameter ). No verifier signal is used during tree expansion. Concretely, let denote the truncated next-token distribution with active set , and let be the induced sequence probability. Starting from a greedy rollout, DLE iteratively selects the unexplored branch with the largest path probability and completes it greedily,
where denotes the frontier of unexplored branch points in the truncated tree, until distinct leaves are produced. After decoding, each leaf is split into reasoning steps with the same rule-based step detector used elsewhere in this work, and the PRM assigns a correctness score to every step. Per-leaf verifier weights are obtained by aggregating the product of these step scores. Equivalent extracted answers are clustered, and the final prediction is obtained by WSC:
Verifier-guided search baselines
Beam search (BS). Beam search maintains the top partial reasoning chains during decoding. At step , suppose the current beam consists of prefixes
For each prefix , we sample candidate next steps
forming expanded prefixes
These expanded prefixes are scored, ranked globally, and only the top are kept in the beam.
We consider two alternative scoring rules. In BS-Step, we score a prefix by its accumulated step-level verifier score,
i.e., using as the beam score. In BS-Future, we score a prefix using the future-outcome predictor,
i.e., using as the beam score. Decoding continues until the beam trajectories are completed, and the final prediction is the answer extracted from the top completed trajectory under the corresponding scoring rule.
Beam search with self-consistency (BS+SC). Beam search with self-consistency uses the same beam expansion and prefix scoring procedure as BS, but aggregates the final answers of the completed beam trajectories by majority vote instead of returning only the top one. If denote the final completed beam trajectories, then
As with BS, we evaluate two variants: BS+SC-Step, which uses , and BS+SC-Future, which uses during beam construction.
TSMC. Twisted Sequential Monte Carlo (TSMC; [35]) performs verifier-guided decoding by maintaining particles, where each particle is a partial reasoning trajectory. At step , each unfinished particle is extended by sampling one next reasoning step from the base generator,
During decoding, TSMC uses the future-outcome predictor
to score partial trajectories and compute non-terminal importance weights
Unfinished particles are then resampled proportionally to , so computation is concentrated on prefixes predicted to lead to strong final solutions. Once a particle terminates, the terminal verifier is used only at the end through
The final prediction is obtained by weighted answer aggregation over completed particles:
Compared with the terminal-potential TSMC variant we evaluate, VGSC uses both local and long-horizon verifier information during decoding. TSMC guides intermediate search through and applies the terminal verifier only at completion, whereas VGSC also incorporates during intermediate decoding. This may improve robustness when local and future verifier signals capture complementary failure modes.
Computational Budget
For the main experiments, we evaluated 15 methods on 5 datasets using 5 random seeds across 4 model configurations (3 LLMs with PRM and 1 LLM with ReProbe). Each run required approximately 3 hours on 2 NVIDIA RTX 4090 GPUs (24 GB each). This results in a total computational cost of approximately 9,000 GPU-hours. The ablation studies required a comparable amount of computation, bringing the overall budget to approximately 18,000 GPU-hours.
Use of Large Language Models
This work investigates large language models as the main object of study. In addition, LLM-based tools, such as ChatGPT, were used in a limited and auxiliary capacity to assist with tasks including language polishing, grammar checking, and code debugging. All such uses were subject to human oversight and verification.
References
- [1]Sergios Agapiou, Omiros Papaspiliopoulos, Daniel Sanz-Alonso, and Andrew M Stuart. Importance sampling: Intrinsic dimension and computational cost. Statistical Science, pp. 405–431, 2017.arxiv.org/abs/1511.06196
- [2]Tristan Cinquin, Geoff Pleiss, and Agustinus Kristiadi. Limits of prm-guided tree search for mathematical reasoning with llms. arXiv preprint arXiv:2510.20272, 2025.arxiv.org/abs/2510.20272
- [3]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reihichiro Nakano, et al. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168, 9, 2021.
- [4]Gal Dalal, Assaf Hallak, Gal Chechik, and Yftah Ziser. More test-time compute can hurt: Overestimation bias in llm beam search. arXiv preprint arXiv:2603.15377, 2026.arxiv.org/abs/2603.15377
- [5]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
- [6]Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3828–3850, 2024.
- [7]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
- [8]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022.arxiv.org/abs/2106.09685
- [10]Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty. Advances in neural information processing systems, 38:19720–19745, 2026.arxiv.org/abs/2502.18581
- [11]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.arxiv.org/abs/2001.08361
- [12]Hadi Khalaf, Claudio Mayrink Verdun, Alex Oesterling, Himabindu Lakkaraju, and Flavio Calmon. Inference-time reward hacking in large language models. Advances in Neural Information Processing Systems, 38:61720–61760, 2026.arxiv.org/abs/2506.19248
- [13]Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. Grace: Discriminator-guided chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15299–15328, 2023.arxiv.org/abs/2305.14934
- [14]Xueyan Li, Johannes Zenn, Ekaterina Fadeeva, Guinan Su, Minrmaya Sachan, and Jonas Geiping. Efficient test-time inference via deterministic exploration of truncated decoding trees. arXiv preprint arXiv:2604.20500, 2026.arxiv.org/abs/2604.20500
- [15]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, pp. 39578–39601, 2024.arxiv.org/abs/2305.20050
- [16]Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74–81, 2004.
- [17]Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in neural information processing systems, 35:2507–2521, 2022.arxiv.org/abs/2209.09513
- [18]Aaron Meurer, Christopher P Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B Kirpichev, Matthew Rocklin, Amit Kumar, Sergiu Ivanov, Jason K Moore, Sartaj Singh, et al. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, 2017.
- [19]Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leipold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov, et al. Efficient test-time scaling of multi-step reasoning by probing internal states of large language models. arXiv preprint arXiv:2511.06209, 2025.arxiv.org/abs/2511.06209
- [20]Young-Jin Park, Kristjan Greenewald, Kaveh Alimohammadi, Hao Wang, and Navid Azizan. Know what you don’t know: Uncertainty calibration of process reward models. Advances in Neural Information Processing Systems, 38:38852–38895, 2026.arxiv.org/abs/2506.09338
- [21]Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning. In International Conference on Learning Representations, volume 2025, pp. 60808–60838, 2025.arxiv.org/abs/2410.08146
- [22]Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie, Igor Kiselev, Nico Daheim, Caiqi Zhang, Artem Vazhentsev, Minrmaya Sachan, et al. A head to predict and a head to question: Pre-trained uncertainty quantification head for hallucination detection in llm outputs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 35700–35719, 2025.
- [23]Vladislav Smirnov, Chieu Nguyen, Sergey Senichev, Minh Ngoc Ta, Ekaterina Fadeeva, Artem Vazhentsev, Daria Galimzianova, Nikolai Rozanov, Viktor Mazanov, Jingwei Ni, Tianyi Wu, Igor Kiselev, Minrmaya Sachan, Iryna Gurevych, Preslav Nakov, Timothy Baldwin, and Artem Shelmanov. Thinkbooster: A unified framework for seamless test-time scaling of llm reasoning, 2026.
- [24]Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149–4158, 2019.arxiv.org/abs/1811.00937
- [25]Kaiwen Wang, Jin-Peng Zhou, Jonathan D. Chang, Zhao-Lin Gao, Nathan Kallus, Kianté Brantley, and Wen Sun. Value-guided search for efficient chain-of-thought reasoning. ArXiv, abs/2505.17373, 2025. URL https://api.semanticscholar.org/CorpusID:278886339.arxiv.org/abs/2505.17373
- [26]Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426–9439, 2024.
- [27]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.arxiv.org/abs/2203.11171
- [28]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.arxiv.org/abs/2201.11903
- [29]Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pp. 2742–2785, 2025.
- [30]Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, MingSung Kan, Junxian He, and Qizhe Xie. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems 36, 2023.arxiv.org/abs/2305.00633
- [31]An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024.
- [32]Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822, 2023.arxiv.org/abs/2305.10601
- [33]Fei Yu, Anningzhe Gao, and Benyou Wang. Ovm, outcome-supervised value models for planning in mathematical reasoning. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 858–875, 2024.arxiv.org/abs/2311.09724
- [34]Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 10495–10516, 2025.
- [35]Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language models via twisted sequential monte carlo. arXiv preprint arXiv:2404.17546, 2024.arxiv.org/abs/2404.17546



