WorksComputer Science
Sequential Anytime Valid Tests for Efficient Inference-Time Alignment
Abstract
Inference-time alignment methods, such as Best-of-N (BoN), improve language model outputs by generating multiple candidates per prompt and selecting among them using a reward model. Much of this cost is wasted: a fixed budget is applied to every prompt, although the computation required by prompts varies widely. Adaptive methods halt once sampled answers appear settled, but their thresholds are tuned heuristics with no statistical control over early-stopping errors. We instead seek an anytime-valid stopping rule that takes an error tolerance as input and halts only once an answer is certified to hold a majority () under the generator's distribution. We aggregate candidates by final answer with a reward-weighted self-consistency (RWSC) score and treat stopping as a sequential test over the resulting clusters. Applying Wald's sequential probability ratio test to these clusters yields SPRT-RWSC, which performs well empirically, but its guarantee fails because the leading answer shifts during sampling, and the run is continuously monitored. Our main contribution, e-RWSC, replaces Wald's test with a betting supermartingale (an e-process) for every answer the policy produces, bounding the probability of ever certifying a non-dominant answer by , uniformly over time and across all answers. The RWSC scores drive the bets, so they affect only how quickly a dominant answer is certified, not validity; certification takes samples. Experiments across several language models, datasets, and reward models confirm the predicted scaling of cost and show that accuracy improves over baselines at substantially lower token cost.
Introduction
Inference-time alignment has emerged as the practical alternative to fine-tuning, in which one repeatedly generates responses from the existing language model (e.g., a policy) and selects among them. Best-of-N [2] selection scores each response with a reward model and returns the one with the highest score. Self-consistency [30] takes a majority vote over final answers. Both are effective, but expensive: their cost is full generations per prompt, and must be large enough for the hardest prompt in the workload. Since inference is the dominant cost of deployed language models, the token budget spent on selection should be minimized.
A fixed budget of is uniform across prompts, while the computation required by prompts varies enormously. On a prompt where the policy immediately concentrates on a single answer, ten samples are enough, whereas on a prompt on which the policy remains genuinely divided, even a thousand samples are not sufficient. Fixed-budget selection, therefore, results in both over- and underspending of tokens. Adaptive methods address this by halting generation once the sampled answers look settled, e.g., stopping when a posterior over answer frequencies concentrates [1, 28], when the marginal value of another sample falls below its cost [14], or when a budget estimated from a short burn-in is exhausted [22]. In every case, the halting rule is a tuned threshold with no control over the probability of stopping on the non-dominant answer. Moreover, they must be re-tuned for each new policy, reward model, and benchmark.
One would instead like an anytime-valid stopping rule that takes a formal error level as input and adapts the token budget accordingly, guaranteeing that the probability of erroneously certifying a non-dominant answer () is strictly bounded by . The rule must be defined based on a computable statistic of the generations. Since many distinct reasoning traces can yield the same final answer, we group responses by their final answers and frame the stopping rule at the answer level. We introduce Reward-Weighted Self-Consistency (RWSC), which scores each cluster by the softmax-weighted mass of its members’ reward scores, so that a cluster is favored both for containing many responses and for highly rewarded ones. Let denote the highest-scoring cluster (the leading answer) at the current step. Framing stopping as the question – is genuinely the dominant answer, or merely ahead by chance? – makes it a sequential hypothesis testing problem. Wald’s sequential probability ratio test (SPRT) is the classic solution: as each new response arrives, one accumulates evidence for or against the leader and stops once it crosses a boundary [27]. It performs strongly in practice, but remains a heuristic here, without any rigorous error-control guarantee.
Wald’s guarantee assumes hypotheses that are fixed in advance, tested by a single terminal decision, and none of these holds here. Writing for the probability that a fresh response yields answer , the rates defining the test are estimated from the same responses the test then examines, rather than fixed beforehand. The leading answer is itself a function of the sampled responses, so the parameter under test, , shifts whenever the leader does, forcing the test to restart. And the statistic is checked after every response, not at a single stopping point. Each of these voids the error control that Wald’s boundaries nominally provide. We resolve this with anytime-valid inference [23], whose guarantees hold under continuous monitoring and arbitrary stopping. We maintain a separate betting supermartingale for each final answer the policy produces, instantiating one when that answer first appears and assigning it a weight from a fixed, summable sequence. Since the weights sum to 1, they split a single error budget across answers, regardless of how many arise. Ville’s inequality [26] then bounds by the probability that a non-dominant answer () is ever certified, uniformly over time and over a growing set of answers. This addresses each failure of SPRT. A change in the leading answer is irrelevant, since every answer accumulates its own evidence. Continuous monitoring is an inherent property of Ville’s inequality, so checking the boundary after each response incurs no additional cost.
Validity requires only that the bets be predictable, each computed from earlier responses before the next one is sampled. The reward model sets each bet based on the current RWSC score of the answer cluster, which buys efficiency when the scores are accurate. The gain has a precise form: under the log-optimal Kelly betting [15], evidence for a dominant answer accrues at a rate , so certification takes about sampled responses. An accurate RWSC score bets near-optimally and approaches this rate, while an inaccurate or hacked score bets poorly and slows the certification. But it never crosses the boundary in expectation under the null, since validity does not depend on the estimate being correct.
The efficiency of the stopping rule, therefore, adapts to the prompt. The number of tokens spent scales with its difficulty: an answer the policy strongly favors () is certified in a few responses, whereas one near the decision boundary takes many, often more than the budget allows, in which case the procedure certifies nothing and falls back to standard aggregation. Either way, the practitioner fixes the error-control probability once, and the number of responses adjusts itself to each prompt. We summarize our contributions as follows.
(i) We cast early stopping as a sequential hypothesis-testing problem and introduce SPRT-RWSC, which applies Wald’s test to determine when the leading answer has converged. It operates on Reward-Weighted Self-Consistency (RWSC): an answer-level aggregation rule that scores answer clusters by reward-weighted mass, and recovers majority voting and Best-of- as extreme cases. It works well in practice, but Wald’s error guarantee does not hold in this setting (Section 2).
(ii) We propose E-RWSC, an anytime-valid stopping rule that accumulates evidence over all answers through separate e-processes. It controls the Type-I error (certifying an answer that is not dominant under the policy) at a user-specified level , uniformly over time, and over an unknown set of answers. The guarantee holds for any predictable choice of bets, so the reward model, which sets them through RWSC scores, governs only efficiency and never validity. Under a plug-in estimate of the log-optimal Kelly betting, certifying the dominant answer (when one exists) takes responses as . So, the number of tokens spent scales with the prompt’s difficulty (§ 3).
(iii) An empirical study on 5 Language Models (Qwen2.5-3B, Qwen2.5-7B, Llama-3-8B, Phi-3-Mini-4K, Mistral-24B) with 4 Reward Models (ArmoRM, FsfairX, Skywork V1, Skywork V2) over 4 Benchmarks (GSM8K, MATH500, MMLU, CruxEval) showing that realized certification error stays below the nominal level, that per-response cost follows the predicted scaling with prompt difficulty, and that E-RWSC matches or outperform several baselines at substantially lower average cost (§ 4). For example, with Qwen-2.5-3B as the language model, ArmoRM as the reward model, and a maximum generation budget capped at 512, on the MATH500 Benchmark, we achieve an 8.2% absolute accuracy gain over BoN with a 61.7% smaller budget.
Related work. A line of work improves which response is chosen from a pool: reward-weighted selection with a temperature [25], ensemble reward models normalized across members [8], and step-level search or process verification that refine how each candidate is generated and scored [21, 4, 5, 16]. RWSC draws on the first two but aggregates over final answers rather than individual candidates. Our contribution concerns the orthogonal question of how many candidates to generate. The adaptive stopping methods [1, 28, 14, 22, 24] share a common limitation: their halting rules are tuned thresholds resting on distributional assumptions, with no control over the probability of stopping on the wrong answer. ConSol [17], like SPRT-RWSC, applies Wald’s test to halt self-consistency generation, but compares only the top two answers and uses static priors. Two recent works give anytime-valid certificates for self-consistency: MMC [7] certifies the empirical-leader mode via paired e-processes and couples this with test-time training, and CITE [20] certifies a fixed target as the unique mode with a matching minimax lower bound. Unlike reward-agnostic mode certifiers (MMC, CITE), E-RWSC uses reward-informed betting to certify strict majority dominance (), achieving 94.5% accuracy on certified early stopped answers vs. 88.9% for MMC with a lower budget.
Answer-Level Aggregation and Sequential Stopping
Given a prompt , a language model samples responses from a fixed reference policy . Each response carries a chain of reasoning terminating in a final answer, extracted by a function that maps to a canonical answer string (e.g., the final numerical result of a math problem). A reward model scores how good a response is. Best-of-N (BoN) samples responses and returns the one with the highest reward. Self-consistency ignores the reward model and returns the answer produced by the majority of responses. Scoring responses individually, as BoN does, exposes selection to reward hacking at its worst: a single anomalously scored response wins outright. Yet discarding the reward signal entirely, as self-consistency does, throws away information about response quality. Since many distinct reasoning chains reach the same final answer, we group responses at the answer level, scoring each answer group by both the number of responses it contains and the total reward it receives.
Reward-Weighted Self-Consistency
Let be a pool of sampled responses, the set of responses yielding answer , and the set of observed answers. Reward-Weighted Self-Consistency (RWSC) assigns each group the softmax-weighted mass of its members’ rewards,
where is the reward normalized by the pool mean and standard deviation , and is a temperature parameter. The scores form a probability distribution over . As , every response contributes equally, and reduces to the fraction of responses in cluster , yielding self-consistency. As , the mass concentrates on the highest-reward response, recovering BoN. Intermediate allows the answer frequency and reward magnitude to jointly determine the winner.
RWSC returns the answer with the highest mass and, within its cluster, the highest-reward response:
RWSC, like BoN and self-consistency, draws a fixed responses per prompt. This is wasteful, since the computation a prompt requires varies by orders of magnitude, and a budget large enough for the hardest prompt overspends on easy ones. Instead, we let each prompt govern its own budget, framing the decision of when to stop as a sequential test of the convergence of RWSC clusters.
A Sequential Test over Answer Clusters
Fix a prompt . Let the responses be drawn sequentially from . At each step , let be the pool of sampled responses, the set of observed answers, the cluster for answer and its RWSC mass. Let be the current leader, the highest-RWSC-mass answer. For an answer , write for its generation probability under . Deciding when to stop amounts to determining whether the leading answer’s probability of generation is sufficiently high to declare it dominant. Each new response contributes to an agreement indicator
where denotes the sigma-algebra generated by first responses and their reward scores. The test asks whether the agreement rate is high, so that the leader has converged, or low, so that it has not. We instantiate it with Wald’s sequential probability ratio test (SPRT) [27] and call the resulting procedure SPRT-RWSC. Its pseudo-code is given in Appendix § 6.
Hypothesis test. Fix and at each step , we test two simple hypotheses,
where represents a non-convergent regime in which the leader holds only a minority of the mass and a regime in which it is dominant. The two rates are set per prompt from a short burn-in of responses (e.g. ), during which clusters are formed but no evidence is accumulated: we set and for a small , so that stopping always requires a strict majority. Under hypothesis , , each observation has mass , so the cumulative log-likelihood ratio after responses,
accumulates evidence for over . Given target error rates (accepting a non-dominant leader) and (rejecting a dominant one) in , Wald’s boundaries are and . If , the test returns the current leader as the final answer. If , it rejects the leader and resets the evidence . Otherwise, it draws another response, falling back to the current leader if the budget is reached.
Shifting leader. The RWSC leader is a function of the responses and can change as we draw . If the leader changes to , the hypotheses—defined relative to the previous leader—no longer concern the quantity being tested, and the accumulated measures evidence about the wrong answer. SPRT-RWSC resets and re-estimates for the new leader with updated answer set and leader’s RWSC mass . Thus, the hypotheses are repeatedly redefined based on the data, allowing the test to adapt to the prompt’s difficulty.
However, one cannot control the errors here as one can with Wald’s theorem. Wald’s classical guarantee is for two known hypotheses fixed in advance, with a single terminal decision. Here, the leader shifts during the run, changing the hypotheses, and the evidence is compared to the boundary after every response rather than at a single stopping point. SPRT-RWSC is thus a principled and effective heuristic, with tunable parameters , , with no guarantee on Type I and II error.
Anytime-Valid Certified Stopping
Wald’s guarantee is not designed for hypotheses that shift with the data or for a statistic inspected after every response. We therefore turn to a framework built for exactly these conditions. -processes ([23]) retain their validity under optional stopping, and predictable data-dependent construction. By building the test on e-processes, we control the probability of ever certifying a non-dominant answer at a user-specified level, uniformly over time.
Given a prompt , let be the final answer of response and the indicator that response agrees with answer . Let be the generation probability of answer . Then, , , are i.i.d. Bernoulli. For each answer , we test
where is threshold controlling dominance of an answer. Under , is not the dominant answer. Hence, rejecting certifies as dominant. When , at most one answer can do so. So the certified answer, if it exists, is unique.
Per-answer betting e-processes. Let be the indicator that response agrees with answer and the sigma-algebra generated by the first responses, their reward scores, and agreement indicators with answer . Let be a stochastic process predictable with respect to the filtration , i.e., is -measurable. Let the first appearance of answer . Define, for , the stochastic process
The above admits a betting interpretation ([23]): a bettor, starting with unit capital, bets repeatedly on the event that the next candidate agrees with the answer , at odds set by the null . Staking a fraction of current capital returns on agreement and forfeits otherwise, so is the bettor’s wealth after rounds.
Stopping rule. Let be the rank of answer in order of first appearance. Assign weights , so that . Given a certification level , we stop generating responses at
If some answer triggers the boundary, it is certified, and the returned candidate is the highest-reward trajectory in its cluster, . If the budget is exhausted first, the procedure falls back to uncertified RWSC selection (equation (2)).
Theorem 1 (Anytime-valid certification). Fix , , . For any answer distribution , any budget , and any predictable sequence of bets ,
Equivalently, with probability greater than , uniformly over all and over all answers such that is certified at time , satisfies .
Proof sketch. Certification of answer occurs when the bettor’s wealth reaches . For an answer that is not dominant (), is a nonnegative supermartingale, so by Ville’s inequality, it attains this level with probability at most . Summing the weights over answers bounds the overall probability of ever certifying a non-dominant answer by . A complete proof is given in Appendix § 7.
Remark 1. The bets enter Theorem 1 only through their range and predictability; they may depend arbitrarily on past responses and rewards, so the reward model cannot affect the certificate’s validity. A change in the RWSC leader is likewise immaterial: each answer accumulates evidence in its own e-process, so no statistic is invalidated or discarded when the leader shifts. Finally, Theorem 1 certifies dominance under the policy, , not correctness: a confidently mistaken policy is certified quickly on a wrong answer. What links certification to accuracy is the self-consistency premise that the majority vote is usually correct, which we verify empirically (§ 4).
Reward-informed betting
We now turn to that efficiency: given the freedom to choose any predictable bet, which choice certifies the dominant answer fastest, and how does the reward model inform that choice? Since the wealth compounds multiplicatively, the natural efficiency criterion is the growth rate , and the bet maximizing it is the classical Kelly bet [15, 3].
Lemma 1 (Kelly bet). Let and for . Let be the expected one-step log-increment in wealth. Then is strictly concave, and its unconstrained maximizer is the Kelly bet . When admissible, i.e. , it attains the growth rate .
Applied to answer , the numerator of the bet is the bettor’s edge over null, and the denominator is the variance of evidence under null. So the optimal stake is proportional to the signal-to-noise ratio of the evidence, and it is non-positive exactly when , where no favorable wager exists. Maximizing expected log wealth, rather than expected wealth, is necessary: expected wealth is maximized by the all-in bet , which collapses the wealth to zero on the first disagreement and cannot recover. Restricting to with rules this out: every step then multiplies the wealth by at least , so a disagreement shrinks but never collapses it, and the Kelly bet within this range maximizes the growth rate . When the bet is admissible, its growth rate is the KL-divergence between true and null agreement rates: evidence accrues at a rate equal to the statistical distinguishability of the two hypotheses. It matches the classical efficiency of Wald’s SPRT [27] while retaining validity under composite nulls and optional stopping.
Plug-in bets. The Kelly bet depends on the unknown generation probability and so cannot be played. Instead, we form a predictable estimate of from the responses and their rewards observed so far. At each step , where is the first appearance of answer , we play the bet
Here denotes the number of agreements it accrued since its appearance, and hence the empirical agreement rate serves as an estimate of . We use a regularized estimator, where the current RWSC mass of its cluster (equation 1) acts as a prior pseudo-observation of strength (we use ). The reward model enters the bet and, consequently, the test through the RWSC mass. We call this procedure E-RWSC. Its pseudo-code is given in Appendix A.
The clip keeps the bet in the admissible range . Informative reward scores steer toward early, sharpening the bets and accelerating certification; miscalibrated scores merely slow the growth of capital and cost efficiency, without affecting validity, since Theorem 1 holds for any predictable bet. Proposition 1 makes this precise, giving the asymptotic growth rate of the capital and the resulting certification time under the plug-in bet.
Proposition 1 (Growth rate and certification time). Let , for an answer and the Kelly bet for . Then, as , the plug-in bet , and the capital , almost surely. Let be the time at which is certified. Then and as , almost surely. Moreover, when , the stopping time of equation 8 coincides with except on an event of probability at most . Consequently in probability as .
Proposition 1 says that E-RWSC certifies a dominant answer in about responses: the cost grows only logarithmically in the confidence and decreases inversely with , which measures how far the answer’s mass sits above the threshold . Easy prompts, where one answer clearly dominates, are certified in a handful of responses; near-ties cost many, so computational cost adapts automatically to prompt difficulty. This rate is essentially the best attainable. Ota et al. [20] proves a lower bound of the same order for sequential mode certification under self-consistency, where the sample complexity is governed by the margin between the target answer and its closest competitor. Instead, E-RWSC certifies that the returned answer holds an absolute majority, – stronger than being the mode – with governing quantity . The rate transfers to the stopping time of the test (equation 8) in probability. Proof is in Appendix § 7.
Remark 2. Ota et al. [20] certify self-consistency by an intersection-union test (CITE), with a confidence-weighted variant in which each vote carries a weight. Here, the weight enters the certified object, so a poor weight changes what is certified. In E-RWSC, the reward model enters only the bets, leaving the certified event fixed, so a poor reward can only slow certification, never invalidate it. More importantly, CITE fixes the target in advance, whereas E-RWSC runs per-answer e-processes and certifies whichever answer crosses first, uniformly over the growing answer set. Empirically, E-RWSC achieves higher accuracy at a lower average token cost than CITE.
Experiments
To evaluate the efficacy and generalizability of our proposed E-SPRT framework, we conduct experiments across four datasets. We utilize GSM8K (Cobbe et al., [6]) and MATH500 (Hendrycks et al., [13]) to assess multi-step mathematical reasoning. To evaluate broad domain knowledge and academic reasoning, we benchmark on MMLU (Hendrycks et al., [12]). We test the framework’s capability in code reasoning and execution tracking using the CruxEval benchmark (Gu et al., [11]). The generation capabilities of our framework were analyzed across four distinct base language models: Qwen2.5-3B, Qwen2.5-7B, Llama-3-8B, and Phi-3-Mini-4K. To evaluate the generated reasoning paths, we utilized a diverse ensemble of four reward models: ArmoRM (Wang et al., [29]), Skywork V1 (Liu et al., [18]) and V2 (Liu et al., [19]), and FsfairX (Dong et al., [9]).
Comparison with Baselines. Figure 1 (a) compares RWSC against full-budget baselines and Figure 1 (b) compares our framework SPRT-RWSC and E-RWSC against early-stopping baselines with . Across all evaluated tasks, RWSC consistently outperforms all full-budget baselines, achieving up to a 6% accuracy improvement over Vanilla BoN on CruxEval. Among sequential methods, AdaBoN (Raman et al., [22]) maintains baseline accuracy but fails to meaningfully reduce compute, consistently requiring 18–19 candidates on average. In contrast, SPRT-RWSC and E-RWSC successfully cut the required budget nearly in half (11–17 candidates) while simultaneously matching or strictly exceeding the accuracy of full-budget baselines. As highlighted by the shaded dominance regions in Figure 1 (b), our methods, E-RWSC and SPRT-RWSC, consistently achieve the optimal Pareto frontier, outperforming CITE on both accuracy and budget reduction over 3 benchmarks and demonstrating a superior balance between alignment quality and computational efficiency. For a more detailed comparison with all the baselines, refer to Table 4 in the appendix.

Figure 1. Comprehensive comparison of all selection methods across GSM8K, MATH500, MMLU and CruxEval for Qwen2.5-3B-Instruct using ArmoRM reward model, with allowed maximum budget as 24. For E-Process methods, we use , , and . For SPRT we use , .
| Method | GSM8K Acc | GSM8K Budget | MATH500 Acc | MATH500 Budget | MMLU Acc | MMLU Budget | CruxEval Acc | CruxEval Budget |
| Oracle@24 (Hard Ceiling) | 96.5% | 24.00 | 90.2% | 24.00 | 93.4% | 24.00 | 76.5% | 24.00 |
| Full Budget Baselines | ||||||||
| Base Policy () | 81.1% | 24.00 | 63.6% | 24.00 | 65.8% | 24.00 | 44.0% | 24.00 |
| Vanilla BoN | 91.2% | 24.00 | 71.0% | 24.00 | 74.6% | 24.00 | 48.6% | 24.00 |
| Soft BoN | 89.1% | 24.00 | 69.4% | 24.00 | 70.0% | 24.00 | 46.8% | 24.00 |
| Answer-Level Aggregation | ||||||||
| Self-Consistency | 89.8% | 24.00 | 74.8% | 24.00 | 70.4% | 24.00 | 54.4% | 24.00 |
| RWSC | 91.4% | 24.00 | 74.2% | 24.00 | 74.7% | 24.00 | 54.6% | 24.00 |
| Early-stopping Baselines | ||||||||
| Pandora’s Box BoN | 88.1% | 68.6% | 73.4% | 49.4% | ||||
| AdaBoN | 91.3% | 71.4% | 74.8% | 48.8% | ||||
| BEACON | 91.3% | 70.8% | 74.6% | 48.5% | ||||
| ConSol | 88.0% | 75.4% | 71.8% | 54.1% | ||||
| CITE | 87.9% | 75.4% | 72.2% | 54.4% | ||||
| MMC | 87.9% | 75.4% | 72.2% | 54.4% | ||||
| Proposed Methods | ||||||||
| SPRT-RWSC | 91.1% | 74.6% | 74.5% | 54.8% | ||||
| E-RWSC | 91.2% | 74.4% | 76.7% | 54.6% |
Table 4. Comprehensive comparison of all selection methods across GSM8K, MATH500, MMLU and CruxEval for Qwen2.5-3B-Instruct using ArmoRM reward model, with allowed maximum budget as 24. For E-Process methods, we use , , and . For SPRT we use .
Cross-Model Generalization and Reward Model Sensitivity. We conduct a two-dimensional ablation on MATH500. Table 1 applies a fixed reward model (ArmoRM) across four generators (3B to 8B). While absolute performance scales with base policy strength (peaking at 81.8% for Qwen2.5-7B), SPRT-RWSC and E-RWSC consistently dominate BoN accuracy across all models, cutting inference budgets by up to 54% (less on the harder-to-converge Llama-3-8B and Phi-3-Mini-4K). Moreover, the stopping rule adapts to generator confidence, expanding budgets for uncertain models (22.45 for Llama-3-8B) and halting early for highly convergent ones (13.06 for Qwen2.5-7B). Table 2 fixes the generator (Qwen2.5-3B) to test RM sensitivity. BoN is sensitive to RM calibration, dropping to 62.6% with the weaker FsfairX RM and surging to 76.8% with Skywork V2. In contrast, RWSC and its sequential variants act as powerful regularizers against RM failures. E-RWSC maintains a robust 72.6% accuracy even with FsfairX and scales to 78.6% with Skywork V2.
| Method | Qwen2.5-3B Acc | Qwen2.5-3B Budget | Qwen2.5-7B Acc | Qwen2.5-7B Budget | Llama-3-8B Acc | Llama-3-8B Budget | Phi-3-Mini-4K Acc | Phi-3-Mini-4K Budget | Mistral-24B Acc | Mistral-24B Budget |
| (Base) | 63.6% | 1.00 | 75.6% | 1.00 | 26.2% | 1.00 | 39.2% | 1.00 | 66.2% | 1.00 |
| Vanilla BoN | 71.0% | 24.00 | 77.8% | 24.00 | 32.6% | 24.00 | 48.2% | 24.00 | 72.0% | 24.00 |
| RWSC | 74.2% | 24.00 | 81.4% | 24.00 | 39.6% | 24.00 | 56.4% | 24.00 | 76.8% | 24.00 |
| Pandora () | 68.6% | 78.0% | 30.0% | 44.2% | 70.8% | |||||
| AdaBoN | 71.4% | 78.0% | 32.2% | 47.8% | 70.8% | |||||
| BEACON | 70.8% | 77.6% | 32.0% | 47.8% | 72.2% | |||||
| ConSol | 75.4% | 80.8% | 38.6% | 53.4% | 76.8% | |||||
| CITE () | 75.4% | 81.8% | 39.0% | 53.6% | 77.4% | |||||
| MMC () | 75.4% | 81.6% | 39.0% | 53.6% | 77.4% | |||||
| Ours (Proposed) | ||||||||||
| SPRT-RWSC () | 74.6% | 81.6% | 39.6% | 56.6% | 76.8% | |||||
| E-RWSC () | 74.4% | 81.4% | 39.6% | 56.4% | 76.8% |
Table 1. Cross-Model Performance Comparison on MATH500 (, ArmoRM).
| Method | ArmoRM Acc | ArmoRM Budget | Skywork V1 Acc | Skywork V1 Budget | FsfairX Acc | FsfairX Budget | Skywork V2 Acc | Skywork V2 Budget |
| (Base) | 63.6% | 1.00 | 63.6% | 1.00 | 63.6% | 1.00 | 63.6% | 1.00 |
| Vanilla BoN | 71.0% | 24.00 | 68.6% | 24.00 | 62.6% | 24.00 | 76.8% | 24.00 |
| RWSC | 74.2% | 24.00 | 76.0% | 24.00 | 72.8% | 24.00 | 78.8% | 24.00 |
| Pandora () | 68.6% | 69.0% | 64.0% | 72.4% | ||||
| AdaBoN | 71.4% | 69.0% | 63.0% | 75.8% | ||||
| BEACON | 70.8% | 69.4% | 62.6% | 77.2% | ||||
| ConSol | 75.4% | 75.4% | 75.4% | 75.4% | ||||
| CITE () | 75.4% | 75.4% | 75.4% | 75.4% | ||||
| MMC () | 75.4% | 75.4% | 75.4% | 75.4% | ||||
| Ours (Proposed) | ||||||||
| SPRT-RWSC () | 74.6% | 75.8% | 72.6% | 78.2% | ||||
| E-RWSC () | 74.4% | 76.0% | 72.6% | 78.6% |
Table 2. Cross-RM Sensitivity Analysis on MATH500 (Qwen2.5-3B, ).
Stopping Rate and Reliability. Figure 3(a) shows that around 225 prompts out of 500 stop in less than 8 steps, whereas 174 extremely hard problems reach the maximum budget. At , E-Process early stops for only 65.2% of the prompts, CITE certifies 85.8% of the prompts, whereas other methods like our SPRT-RWSC, ConSol, and Pandora’s Box BoN stop 100% of the prompts early. But among these certified early-stopped answers, E-RWSC remains highly accurate. Figure 2a decouples the accuracy of the certified early stopped answer from hard answers that exhausted the maximum budget. E-RWSC achieve 94.5% accuracy on certified answers, which is more than 5% accuracy gap as compared to the closest competitor in certified accuracy, MMC, with 88.9% accuracy, and more than 10% gap as compared to CITE with 83.7% certified accuracy.

Figure 2. (a) Accuracy on stopped () vs. exhausted queries. (b) Decision reliability vs. overall token footprint. Figure 2: Stopping reliability and token footprint analysis on MATH500 () for Qwen2.5-3B with ArmoRM. (a) E-RWSC achieves 94.5% accuracy on certified early stops (). (b) Decision reliability on early stops plotted against overall token expenditure (-tokens)

Figure 3. Ablation Studies on MATH500. (a) E-RWSC stopping distribution for . (b) Sensitivity to Type-I error boundary (). (c) Sensitivity to the prior alignment threshold ().
Budget scaling and prevention of premature convergence. Table 3 illustrates the scaling behavior of selection methods as the maximum allowed budget expands from to . A critical failure mode, reward hacking, of BoN is exposed here: accuracy degrades at high budgets (dropping from 71.4% at to 69.8% at ). RWSC successfully regularizes this effect, monotonically scaling accuracy up to 77.8%. ConSol saturates early (halting at for ), resulting in stagnant accuracy even when the budget is much higher. In contrast, E-RWSC effectively bridges the gap between efficiency and inference scaling. It adaptively consumes more budget as increases (from 15.6 to 196.1), selectively spending compute on highly uncertain prompts. Crucially, the early-certified subset under E-RWSC exhibits remarkable precision across all scales: certified accuracy stays near-ceiling, scaling from 99.2% at (in just 8.1 steps) to 94.5% at (27.5 steps).
| Method | Acc | Budget | Acc | Budget | Acc | Budget | Acc | Budget | Acc | Budget | Acc | Budget |
| Vanilla BoN | 71.0% | 24.0 | 71.4% | 32.0 | 69.2% | 64.0 | 70.2% | 128.0 | 69.4% | 256.0 | 69.8% | 512.0 |
| RWSC | 74.2% | 24.0 | 76.0% | 32.0 | 77.0% | 64.0 | 77.6% | 128.0 | 77.6% | 256.0 | 77.8% | 512.0 |
| Pandora | 68.6% | 4.0 | 68.6% | 4.0 | 68.6% | 4.0 | 68.6% | 4.0 | 68.6% | 4.0 | 68.6% | 4.0 |
| AdaBoN | 71.4% | 70.6% | 70.0% | 70.2% | 69.8% | 69.4% | ||||||
| BEACON | 70.8% | 71.0% | 69.0% | 70.4% | 69.6% | 70.0% | ||||||
| CONSOL | 75.4% | 74.4% | 74.6% | 74.8% | 74.8% | 74.8% | ||||||
| CITE | 75.4% | 74.4% | 75.6% | 76.8% | 77.2% | 77.4% | ||||||
| MMC | 75.4% | 74.4% | 75.6% | 76.8% | 77.2% | 77.4% | ||||||
| SPRT-RWSC | 74.6% | 75.8% | 76.2% | 76.4% | 76.4% | 76.4% | ||||||
| E-RWSC | 74.4% | 76.0% | 77.0% | 77.8% | 77.8% | 78.0% |
Table 3. MATH500 Budget Sweep Results (Qwen2.5-3B, ArmoRM)
Sensitivity to e-process parameters ( and †). The parameters (Type-I error tolerance) and † (dominance threshold) govern the anytime-valid certification guarantee. Figure 3(b) demonstrates that E-RWSC is robust to . The average budget decreases smoothly from 34.22 to 30.11 as is relaxed from 0.05 to 0.4. This confirms that the practitioner can tune to trade off statistical confidence for computational efficiency without a significant drop in accuracy. Figure 3(c) sweeps the threshold †. Because † represents the hypothesized null-agreement rate, a higher † demands stronger empirical consensus before the plug-in Kelly bet yields positive capital growth. Consequently, the generation budget scales monotonically with † (from 15.56 at † = 0.1 to 60.83 at † = 0.9). Accuracies peak broadly in the middle range (0.3 † ), validating our default choice of † = 0.5 as a sound threshold for certifying a strict majority.
Compute Overhead. To test whether sequential monitoring introduces non-trivial algorithmic latency relative to raw candidate generation, we empirically benchmark the exact wall-clock latency of every pipeline component on an NVIDIA RTX 6000 Ada GPU and AMD host CPU for MATH500 (). Generating each candidate response ( tokens with Qwen2.5-3B) requires 7,084.5 ms, and reward scoring with ArmoRM requires 59.8 ms, totaling 7,144.3 ms of GPU compute per candidate. The CPU algorithmic overhead of E-RWSC for updating the e-process and evaluating the predictable Kelly bet across semantic clusters requires merely 13.96 s (0.014 ms) per step. Across an entire adaptive query run (), the cumulative algorithmic overhead is only 2.74 ms, representing less than 0.0002% of GPU computation time (14 faster than CITE’s 194.3 s confidence-interval check). Full hardware profiles and baseline comparisons are detailed in Appendix § 9.
Empirical validation of theoretical results.
We empirically validate E-RWSC theoretical guarantees in Figure 4 under dominance threshold (majority dominance). First, we validate the anytime-valid certification bound (Theorem 1) by sweeping the Type-I error rate and measuring the realized error rate of E-RWSC. Following Theorem 1, an error event occurs if an answer is certified at stopping time , but its underlying generation probability satisfies ; we evaluate this empirically using the candidate frequency across the full budget as an empirical estimate . As shown in Figure 4 (Top), the realized error rate remains strictly below the permitted threshold across all nominal levels (empirically , with for all and mean consensus ), directly validating the finite-sample anytime-valid boundary.

Figure 4. Empirical validation of E-RWSC theoretical bounds. (Top) Realized error rate respects the Type-I error () threshold. (Bottom) The inverse of signal strength versus stopping step shows a linear relationship.
Second, we validate Proposition 1, which governs the asymptotic growth rate of evidence. The theory states that the expected number of steps to stop scales with the inverse Kullback-Leibler divergence between the empirical answer distribution and the null hypothesis. As shown in the bottom panel, there is a linear relationship between the inverse of the signal strength and the stopping time , confirming that our thresholding effectively controls the early-stopping budget, which is proportional to the prompt’s inherent difficulty.
Conclusion
We introduced Reward-Weighted Self-Consistency (RWSC), an answer-level aggregation rule spanning Best-of- and self-consistency, and cast the question of how many candidates a prompt needs as a sequential test over its clusters: Wald’s SPRT applied to RWSC works well empirically but cannot guarantee its error rate under a shifting leader, so we introduce with E-RWSC, which maintains a betting e-process per answer and bounds the probability of ever certifying a non-dominant answer by for any predictable bets, at the minimax-optimal rate under Kelly betting. Across four models, four reward models, and four benchmarks, e-RWSC matches or exceeds Best-of- accuracy at substantially reduced cost (e.g., 8.2% higher than Vanilla BoN with 61.7% less budget on MATH500), with realized error and per-response cost tracking these bounds empirically. The main caveat is that certification guarantees policy dominance, , not factual correctness, so a confidently wrong policy is certified just as fast as a correct one.
Pseudocodes
Algorithm 1 SPRT-RWSC
Require: prompt ; policy ; reward model ; errors ; budget ; burn-in
;
Generate candidates ; ▷ Burn-in phase
Construct initial answer clusters and RWSC masses
▷ Identify initial leader
; set ▷ Initialize Log-Likelihood Ratio
for do
Sample ; ;
Update answer clusters and RWSC masses over
▷ Identify current leader
if then ▷ Leader Shift
; ; Update
else
▷ Check for agreement
;
if then ▷ Accept
return ▷ Stop early and return leader
else if then ▷ Reject
; Update ▷ Reset evidence and track new trend
end if
end if
end for
return ▷ Budget exhausted, return current leader
Algorithm 2 E-RWSC
Require: prompt ; policy ; reward model ; level ; threshold (default ); clip ; prior strength ; budget ; weights
; ▷ active answers; count of distinct answers seen
for do
for each do ▷ bets are predictable: computed before is drawn
▷ RWSC mass from (1) on
▷ plug-in Kelly bet, (10)
end for
sample ; score ;
recompute RWSC masses over
for each do ▷ settle all active bets against the new observation
; ;
end for
if then ▷ lazy instantiation; first observation consumed, not bet on
; ;
; ;
end if
if then ▷ Ville boundary, (8)
return ▷ certified: w.p. (Thm. 1)
end if
end for
return from equation 2 ▷ budget exhausted: uncertified fallback
Deferred Proofs
Proof of Theorem 1. Let denote the set of null answers, and for each answer let
be the event that is certified at some time. Note that , since is instantiated only once has been observed. We must bound .
Fix and work on the event , since outside this event there is no supermartingale—the process was never started, and isn’t even defined. The candidates are i.i.d. draws from and is a stopping time, so conditionally on the post- observations are i.i.d. Bernoulli and independent of ; starting the product at is what excludes the deterministic success implied by the definition of . Each factor is bounded below by , since worst case gives and , and the bets are -measurable, so
using and . Hence, under the conditional law given , the process is a nonnegative supermartingale started at 1.
Ville’s inequality bounds for any nonnegative supermartingale with . The rank is determined by the answers observed up to and including time , hence is -measurable; applying Ville at threshold gives
Since and , the tower property gives
The set of possible answer strings is countable, so countable subadditivity applies. Combining with equation (14) and exchanging sum and expectation by Tonelli’s theorem (all terms nonnegative),
the last inequality by dropping the restriction to , which only adds nonnegative terms.
Fix a sample path. Answers are enumerated in order of first appearance, so is injective on the observed answers ; the ranks are therefore distinct positive integers and
This holds for every sample path, not merely in expectation, so the random variable inside the expectation in equation (15) is bounded above by the constant ; hence its expectation is at most .
Substituting equation (16) into equation (15) gives . Truncation at only removes elements from each , so the bound holds for any budget; and since Ville’s inequality controls the supremum over all times, the guarantee is unaffected by when the analyst chooses to stop, which is the anytime-validity property. □
Proof of Lemma 1. Observe that . Strict concavity is immediate since on the admissible interval (a sum of two negative terms). Setting gives ; cross-multiplying and cancelling the terms linear in yields , which gives the bet. Substituting into and simplifying, using and , gives the growth rate. □
Proof of Proposition 1. Write . Throughout, “” denotes almost-sure convergence unless stated otherwise, and all statements are made on the event , which has probability one since . Recall the expected one-step log-increment
Abbreviate , , , so that .
(i) The plug-in bets converge. Conditionally on , the post-instantiation indicators are i.i.d. Bernoulli (as in the proof of Theorem 1). By the strong law of large numbers, the empirical agreement rate . The plug-in estimate (equation 10) is , a convex combination of the empirical rate and the bounded prior (RWSC mass) . The prior weight . Hence regardless of the prior sequence. The Kelly map is continuous (a clip of the continuous map ), so by the continuous mapping theorem, the plug-in bet when .
(ii) Convergence of capital. The bet is -measurable by construction, and is independent of with mean , so . The map is continuous on the compact interval (a sum of logarithms with arguments bounded below by ), so by continuity and Step (i), . The Cesàro lemma ([10]) gives
Let , so that is a martingale-difference sequence with respect to . Because , each lies in the bounded interval , so for a constant ; in particular . Then, the martingale strong law ([10]) yields . Adding this with equation 17, and noting the finitely many terms with index at most contribute and so do not affect the Cesàro limit, gives whenever .
(iii) Certification time. Fix a sample path in the probability-one event , on which . Since , we have and hence . Write for the log-boundary of . Note , hence , is fixed on this path and does not depend on , and as . Hence is finite for every since the walk diverges, and as . Now, by definition,
On the fixed path, the sequence converges to , so along the divergent integer sequence (and likewise , using ) both outer terms of equation 18, divided by , tend to . Hence , i.e. . Since , we may replace by , yielding a.s.
(iv) Stopping time. of equation 8 halts at the first answer to cross its boundary, so always, with equality when no other answer is certified first. When , is the unique answer with , so any other certified answer is non-dominant. By Theorem 1, this occurs with probability at most . Thus . Fix . Since almost surely, and hence in probability,
Combined with , a union bound gives
which completes the proof.
Comparison with baselines
Table 4 reports the full breakdown underlying the summary in § 4. The Oracle@24 row upperbounds what any selection rule can achieve from this candidate pool, and the gap to it is largest on CruxEval (76.5% vs. our best 54.8%) and smallest on GSM8K (96.5% vs. 91.4%), reflecting that code-execution answers are harder to concentrate reward mass on than short numeric ones. The base policy alone already reaches 81.1% on GSM8K without any selection, so the relative gains from BoN-style methods are naturally compressed there; on MATH500, MMLU, and CruxEval, where the base policy starts much lower (63.6%, 65.8%, 44.0%), selection contributes proportionally more of the final accuracy. Soft BoN (Verdun et al., [25]) underperforms Vanilla BoN on every benchmark, indicating that softening the arg-max selection with a temperature is not by itself sufficient to recover the consensus robustness that RWSC’s explicit answer-level aggregation provides. Among the baselines, BEACON (Wan et al., [28]) takes a very high budget, whereas for others, budget usage is highly Reward Model dependent or benchmark-dependent rather than a fixed property of the method: Pandora’s Box BoN (Kalayci et al., [14]) is highly dependent on the scale of output of reward modes, with very confident answers of ArmoRM, it stops at budget same as burn-in phase, although it changes as we change the reward model, as it can be seen in Table 2, ConSol’s budget ranges from 5.39 on MMLU to 11.99 on CruxEval, and CITE’s stays comparatively flat (21.60–23.31) across all four, consistent with its fixed-target certification not adapting its threshold to how concentrated a given benchmark’s answer distribution is. SPRT-RWSC and E-RWSC track prompt difficulty instead: both spend the most on CruxEval (17.17 and 20.40, respectively) and the least on GSM8K (11.48 and 14.81), mirroring the same base-policy-strength ordering that governs the oracle gap.
Computational Overhead
A common concern in sequential early-stopping mechanisms is whether the continuous algorithmic overhead of evaluating statistical bounds at each step negates the savings gained from generating fewer tokens. To rigorously quantify this trade-off, we conduct comprehensive wall-clock hardware benchmarks on the MATH500 dataset under using the Qwen2.5-3B-Instruct policy and ArmoRM-Llama3-8B-v0.1 reward model. All measurements are executed on dedicated hardware: an NVIDIA RTX 6000 Ada Generation GPU (48GB VRAM) for model inference and an AMD host processor for sequential decision logic.
Per-Candidate Latency Decomposition. Table 5 breaks down the exact execution latency per candidate response. On MATH500, the average candidate trajectory generated by Qwen2.5-3B contains 500.7 tokens. Autoregressive LLM decoding takes 14.15 ms per token, resulting in an average generation latency of 7,084.50 ms (≈7.08 s) per candidate. A batched forward pass of ArmoRM requires an additional 59.76 ms to evaluate the candidate’s scalar reward. Answer extraction via canonical regex parsing adds approximately 0.20 ms on CPU. Thus, generating and evaluating a single candidate takes 7,144.26 ms (≈7.14 s) of total GPU compute.
| Pipeline Component | Hardware Device | Complexity | Latency / Candidate |
| LLM Candidate Generation (Qwen2.5-3B) | GPU (RTX 6000 Ada) | 7,084.50 ms (≈7.08 s) | |
| RM Scoring (ArmoRM-8B forward pass) | GPU (RTX 6000 Ada) | 59.76 ms | |
| Answer Extraction & Normalization | CPU (Host) | regex | ≈ 0.20 ms |
| E-RWSC E-Process Update & Kelly Bet | CPU (Host) | scalar | 0.014 ms (13.96 s) |
| Total GPU Compute per Candidate | GPU | — | 7,144.26 ms (≈7.14 s) |
| Algorithmic Overhead Ratio | Host CPU / GPU | — | ≈ 0.00020% |
Table 5. Hardware Latency Breakdown per Candidate on MATH500 using Qwen2.5-3B-Instruct (avg 500.7 tokens/candidate) and ArmoRM-8B on an NVIDIA RTX 6000 Ada GPU and AMD host CPU.
In sharp contrast, the sequential stopping check of E-RWSC requires computing predictable Kelly bets across the set of distinct semantic answer clusters and updating scalar e-processes. Because empirical answer clustering produces at most distinct answers per prompt, this step consists of simple scalar arithmetic with time complexity , executing in just 13.96 s (0.014 ms) on CPU. The ratio of algorithmic decision overhead to raw GPU candidate compute is approximately 0.00020% (≈ 1 part in 500,000).
End-to-End Computational Profile Across Methods. Table 6 reports the full end-to-end computational expenditure across all baseline methods on MATH500 under . Key empirical takeaways include:
| Method | Avg Steps | Generated Tokens () | GPU Compute Time (s / query) | Algorithmic Check Latency (s / step) | Algorithmic Overhead (ms / query) | Overall Acc (%) | Certified Acc (%) |
| Fixed-Budget Baselines | |||||||
| Vanilla BoN | 512.0 | 256.3 | 3657.7 (≈61.0 min) | — | 0.000 | 69.8% | — |
| Self-Consistency | 512.0 | 256.3 | 3657.7 (≈61.0 min) | — | 0.000 | 77.4% | — |
| Adaptive Sequential Baselines | |||||||
| AdaBoN | 380.4 | 202.6 | 2717.9 (≈45.3 min) | 0.02 s | 0.008 | 69.4% | 76.4% |
| BEACON | 416.5 | 213.0 | 2755.5 (≈49.6 min) | 1.75 s | 0.729 | 70.0% | 71.8% |
| Pandora’s Box | 4.0 | 2.0 | 28.6 (≈0.5 min) | 0.17 s | 0.001 | 68.6% | 68.6% |
| ConSol | 12.3 | 9.1 | 88.1 (≈1.5 min) | 0.13 s | 0.002 | 74.8% | 74.8% |
| CITE | 124.9 | 86.5 | 892.2 (≈14.9 min) | 194.33 s | 24.269 | 77.4% | 83.7% |
| Proposed Frameworks | |||||||
| SPRT-RWSC (Ours) | 21.8 | 13.6 | 155.8 (≈2.6 min) | 5.83 s | 0.127 | 76.8% | 76.8% |
| E-RWSC (Ours) | 196.1 | 137.1 | 1400.8 (≈23.3 min) | 13.96 s | 2.737 | 78.0% | 94.5% |
Table 6. End-to-End Computational Profile and Algorithmic Overhead on MATH500 () for Qwen2.5-3B-Instruct with ArmoRM-8B. GPU generation and RM scoring latency measured on an NVIDIA RTX 6000 Ada GPU; algorithmic stopping-check overhead measured on the host CPU. Algorithmic overhead is under 0.0002% of generation time for E-RWSC, while saving 61.7% of total GPU compute time relative to fixed-budget BoN.
Massive GPU Savings without Latency Overhead: Fixed-budget methods (Vanilla BoN and Self-Consistency) consume 3,657.7 s (≈61.0 minutes) of dedicated GPU compute per query across the 512 candidates. By halting early on confident prompts (), E-RWSC slashes average GPU compute to 1,400.8 s (≈23.3 minutes)—saving 2,256.9 seconds (37.6 minutes) of GPU runtime per query (61.7% relative compute reduction). Over an entire query, the cumulative CPU stopping overhead of E-RWSC is just 2.737 ms, which is virtually unmeasurable in wall-clock user experience.
Efficiency Superiority over CITE: While CITE also performs certified sequential stopping, its stopping criterion requires inverting empirical-Bernstein confidence intervals over
dynamic subsets of candidates at every step. This inflates CITE’s per-step check latency to 194.33 s ( slower than E-RWSC), accumulating 24.269 ms of CPU compute per query. Despite this higher computational cost, CITE achieves only 83.7% certified accuracy, whereas E-RWSC achieves 94.5% certified accuracy with negligible algorithmic overhead.
Reward Hacking Regularization: Vanilla BoN suffers severe performance degradation at , dropping to 69.8% accuracy despite consuming over an hour of GPU compute. E-RWSC reaches 78.0% overall accuracy (and 94.5% on certified queries) while spending less than half the GPU compute, proving that anytime-valid stopping simultaneously optimizes computational efficiency and mathematical accuracy.
References
- [1]Pranjal Aggarwal, Aman Madaaan, Yiming Yang, et al. Let’s sample step by step: Adaptive-consistency for efficient reasoning and coding with llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12375–12396, 2023.arxiv.org/abs/2305.11860
- [2]Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander Nicholas D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=u3U8qzfVF7w.arxiv.org/abs/2401.01879
- [3]L Breiman. Optimal gambling systems for favorable games. The Kelly Capital Growth Investment Criterion: Theory and Practice, 3:47, 2011.
- [4]Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process. Advances in Neural Information Processing Systems, 37:27689–27724, 2024.arxiv.org/abs/2405.03553
- [5]Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, Jinsung Yoon, and Sercan O Arik. SETS: Leveraging self-verification and self-correction for improved test-time scaling. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=Wv9NMJoKww.arxiv.org/abs/2501.19306
- [6]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
- [7]Paula Cordero-Encinar and Andrew B Duncan. Certified self-consistency: Statistical guarantees and test-time training for reliable reasoning in llms. arXiv preprint arXiv:2510.17472, 2025.
- [8]Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In International Conference on Learning Representations, volume 2024, pp. 50905–50931, 2024.arxiv.org/abs/2310.02743
- [9]Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=a13aYUU9eU.arxiv.org/abs/2405.07863
- [10]Rick Durrett. Probability: theory and examples, volume 49. Cambridge university press, 2019.
- [11]Alex Gu, Baptiste Rozier, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. CRUXEval: A benchmark for code reasoning, understanding and execution. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 16568–16621. PMLR, 21–27 Jul 2024.arxiv.org/abs/2401.03065
- [12]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=d7KBjmI3GmQ.
- [13]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021b. URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/be83ab3ecd0db773eb2dc1b0a17836a1-paper-round2.pdf.
- [14]Yusuf Kalayci, Vinod Raman, and Shaddin Dughmi. Optimal stopping vs best-of-n for inference time optimization. arXiv preprint arXiv:2510.01394, 2025.arxiv.org/abs/2510.01394
- [15]John L. Kelly. A new interpretation of information rate. the bell system technical journal, 35(4): 917–926, 1956.
- [16]Hyunseok Lee, Seunghyuk Oh, Jaehyung Kim, Jinwoo Shin, and Jihoon Tack. Revise: Learning to refine at test-time via intrinsic self-verification. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267, pp. 33616–33634, 2025a.arxiv.org/abs/2502.14565
- [17]Jaeyeon Lee, Guantong Qi, Matthew Brady Neeley, Zhandong Liu, and Hyun-Hwan Jeong. Consol: Sequential probability ratio testing to find consistent llm reasoning paths efficiently. arXiv preprint arXiv:2503.17587, 2025b.arxiv.org/abs/2503.17587
- [18]Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451, 2024.arxiv.org/abs/2410.18451
- [19]Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. In International Conference on Learning Representations, volume 2026, pp. 133805–133838, 2026.arxiv.org/abs/2507.01352
- [20]Hirofumi Ota, Naoto Iwase, Yuki Ichihara, Junpei Komiyama, and Masaaki Imaizumi. Cite: Anytime-valid statistical inference in llm self-consistency. arXiv preprint arXiv:2605.05873, 2026.arxiv.org/abs/2605.05873
- [21]Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Chenhao Zhu, Xinzhe Juan, Ling Yang, Huazheng Wang, Kaixuan Huang, et al. Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling. In EMNLP (Findings), pp. 20894–20917, 2025.arxiv.org/abs/2410.16033
- [22]Vinod Raman, Hilal Asi, and Satyen Kale. Adabon: Adaptive best-of-n alignment, 2026. URL https://arxiv.org/abs/2505.12050.
- [23]Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4):576–601, 2023.arxiv.org/abs/2210.01948
- [24]Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=348hfcp rUs.arxiv.org/abs/2410.20290
- [25]Claudio Mayrink Verdun, Alex Oesterling, Himabindu Lakkaraju, and Flavio P Calmon. Soft best-of-n sampling for model alignment. In 2025 IEEE International Symposium on Information Theory (ISIT), pp. 1–6. IEEE, 2025.arxiv.org/abs/2505.03156
- [26]Jean Ville. Étude critique de la notion de collectif, volume 3. Gauthier-Villars Paris, 1939.
- [27]A. Wald. Sequential Tests of Statistical Hypotheses. The Annals of Mathematical Statistics, 16(2): 117 – 186, 1945. doi: 10.1214/aoms/1177731118.DOI
- [28]Guangya Wan, Zixin Stephen Xu, Sasa Zorc, Manel Baucells, Mengxuan Hu, Hao Wang, and Sheng Li. Beacon: Bayesian optimal stopping for efficient llm sampling. arXiv preprint arXiv:2510.15945, 2025.arxiv.org/abs/2510.15945
- [29]Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10582–10592, 2024.arxiv.org/abs/2406.12845
- [30]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2022.arxiv.org/abs/2203.11171