betaFeedback

WorksComputer Science

When Self-Consistency Hurts: The Hidden Cost of Majority Voting in Small Language Models

Abstract

We study prompted chain-of-thought (CoT) with self-consistency (SC), i.e. sampling N traces and plurality-voting, on small language models, a regime where per-query compute budgets are tight yet SC is commonly applied with a fixed sample budget inherited from large-model practice, regardless of instance difficulty. Across two model families (Qwen-2.5-0.5B, Gemma-3-1B) at FP16 and MLX-Q4 (4-bit quantisation in Apple's MLX framework), on GSM8K and MATH-500, we run 8 per-task and 4 task-pooled experiments at n=200-400 with 1,000x paired-bootstrap confidence intervals (CIs). Our first result is an upper bound: an oracle gate deciding per prompt whether to invoke SC reaches 109-174% of always-SC's accuracy at 23-30% of its compute in all 12/12 cases, so at N=5 always-SC is dominated everywhere on our grid. That margin is exactly the rate at which SC flips correct greedy answers to wrong, here 3-10.5% of prompts, which on Qwen-Q4 x MATH-500 exceeds the wrong-to-right rate and leaves always-SC 3 pt worse than always-greedy. Our second result is a deployable approximation: a gate over greedy-pass signals skips 45-99% of SC calls, with bootstrap CI half-widths under 5 pt, removing 36-79% of always-SC compute. Accuracy retention at those operating points has point estimates of 98.1-98.8%, which we report as point estimates because their CIs are wide (lower bounds 76-86%) over an SC accuracy of only 15-36%. The evidence covers N=5 on two <=1B families and two math benchmarks, which is the scope we claim. Code is available at https://github.com/heywanrong/SiftSC.

Introduction

Test-time compute (TTC) has become a standard component of modern language-model deployment. Starting from chain-of-thought (CoT) prompting [31, 18], a family of methods spanning self-consistency (SC) sampling [29], best-of-NN reranking with learned verifiers [9, 20], tree-structured search [34], budget-forced CoT [27] and tool-integrated self-verification [16] now underpins the strongest open and proprietary systems [24, 7]. The prevailing deployment recipe is simple: for every query, sample NN traces and plurality-vote. This always-SC policy is the default in every widely-used inference stack we are aware of, and it has been validated extensively on large (≥7\ge7B) models, where N=5N=5 to N=40N=40 reliably trades compute for accuracy [7, 27].

The economics look different for small language models, the segment where inference cost matters most: on-device deployment is bounded by memory and battery [30, 3], cloud cost is dominated by latency [4], privacy pressures favour local inference, and distilled sub-4B models now match or beat 7B+ reasoners on math [33]. The always-SC recipe was nonetheless inherited from large-model practice without much scrutiny of whether it remains the right default for prompted CoT at this scale. A parallel literature on efficient TTC has meanwhile developed around cutting the sample count [1, 28], adding confidence-aware stopping [2, 17], and predicting the budget up front [15]. All of it assumes that SC at some N>1N>1 is worth running. On the models we test, that premise is what fails.

We revisit two questions. (Q1) Is always-SC actually worth it on small models? The commonly used metric is SC’s accuracy relative to greedy, but the realistic deployment baseline is not plain greedy: it is “run cheap greedy first, and run SC only when the prompt requires it”. The fair comparison for always-SC is therefore an oracle gate that picks the better of greedy and SC per prompt. (Q2) If always-SC is dominated, can a cheap gate approximate that oracle? If so, selective SC becomes the natural default at this scale.

We answer both concretely. On Qwen-2.5-0.5B-Instruct [25] and Gemma-3-1B-it [12], each in FP16 and 4-bit quantisation in Apple’s MLX framework, across GSM8K [9] and MATH-500 [14], the oracle gate dominates always-SC in all 12/12 experiments, retaining 109–174% of its accuracy for 23–30% of its compute. What drives the dominance is a non-trivial harm rate: SC flips correct greedy answers to wrong on 3–10.5% of prompts, comparable to and sometimes larger than the rate at which it flips wrong to right. On Qwen-Q4×\timesMATH-500 the harm rate exceeds the help rate, and always-SC ends up 3 pt worse than always-greedy. A gate on greedy-pass signals recovers much of this margin. On the four pooled cells it skips 45–99% of SC calls, with 1000× bootstrap confidence intervals (CIs) whose half-widths on skip rate stay under 5 pt, and removes 36–79% of always-SC compute. Retention of always-SC accuracy at those operating points is 98.1–98.8% as a point estimate, and we report it only as such, because its bootstrap CIs are wide (§4).

Scope. Our evidence is narrower than the framing above may suggest, so we state the boundary here. Everything below is at N=5N=5, on two ≤\le1B families, at two precisions, on two math benchmarks, with one generation seed per cell. Within that grid the dominance of the per-prompt oracle gate over always-SC holds in every cell and the harm rate is measurable. We do not claim either for larger models, reasoning-distilled models, non-mathematical tasks, or larger NN (§7).

Relation to prior efficient-TTC work. Adaptive-SC methods [1, 28, 2, 17, 15, 22] reduce NN once SC has been invoked, whereas our gate acts on the invocation decision itself. Cascades and routers [35, 26, 23, 6, 8] choose which model answers, while our gate stays inside one model. § 2 compares it with both lines, and § 7 notes that we run no head-to-head comparison with either.

Contributions. (i) The first per-prompt measurement of prompted SC’s “helps vs. hurts” decomposition on small language models, exhibiting a non-negligible, and at one cell dominant, harm rate (§ 4); (ii) evidence against the “SC is free” premise of the efficient-TTC literature, bounded to N=5N=5 on these two families and two math benchmarks; (iii) a minimal 1–12 feature cross-validated TTC-gate that captures much of the oracle’s savings with tight skip-rate CIs, plus an ablation in which added capacity (a multi-layer perceptron, MLP) brings no benefit at this data scale (§ 5); (iv) a single-laptop reproduction path under MLX, included in the protocol.

Background and Related Work

Test-time compute. CoT prompting [31, 18] and SC [29] established the sample-and-aggregate recipe, [27] formalised its compute–accuracy scaling for large models, and [7] surveys the long-CoT literature that followed. [34] extends the recipe to structured search; [20, 9] train process- and outcome-reward verifiers for reranking. None of this line questions the assumption that SC is worth running on every prompt.

Adaptive self-consistency. Three families of methods make SC cheaper. Early-stopping rules halt sampling once the running majority looks stable, whether on count-based convergence [1], Bayesian and reliability-aware criteria [2, 17], or votes reweighted by per-sample reasoning quality [28]. Up-front budget prediction fixes NN before any sampling, as in Seer’s System-1 entropy estimate [15]. Cross-prompt allocation spends one global budget where it helps the most [22]. A fourth line provides statistical guarantees for sampling-based TTC [36, 10]. All of them take the invocation of SC as given and choose how many traces to draw, which is why the gate composes with them rather than competing with them: it answers the prior question of whether to draw any. The difference is quantitative too, since an early-stopping rule cutting NN from 5 to 2 saves at most 2.5× on a prompt, while skipping SC saves the whole SC budget. We do not run any of them head-to-head against the gate (§ 7).

The closest prior observation that more sampling can hurt is [5], who show majority-vote compound systems to be non-monotone in the number of model calls and trace this to heterogeneous query difficulty, easy queries improving while hard ones degrade. Their analysis picks one NN per task on large models. We measure the per-prompt help/harm decomposition on small models and act on it per prompt.

Small language models and quantisation. Distilled small language models [16, 33] make the small-language-model + TTC setting practically relevant. [19] report that 4-bit post-training quantisation (e.g. AWQ [21], GPTQ [11]) costs Llama-3 models an average of 11% and up to 32% of their math-reasoning accuracy. We measure a 5.5–27 pt FP16-vs-Q4 drop on our two families and show it interacting adversely with SC on at least one cell.

Calibration and verbalised confidence. Classical calibration work [13] and large-language-model uncertainty studies [32, 8] show token-level confidences to be useful though imperfect signals. We use only native greedy-pass logits and simple text features, with no fine-tuning and no verbaliser.

Routing and cascades. Cascade and router systems [35, 26, 23, 6] pick between models of different sizes or providers from logit-level or learned signals. Our gate is within-model and decides not which model runs but whether to run TTC at all, so it needs no cloud teacher, external verifier, or auxiliary model.

Experimental Setup

Models. We use Qwen-2.5-0.5B-Instruct [25] and Gemma-3-1B-it [12], each at two precisions: MLX-FP16 (native BF16 cast to FP16) and MLX-Q4 (affine weight-only quantisation, group size 64). On Apple Silicon the standard bitsandbytes-NF4 (4-bit NormalFloat) and auto-gptq stacks either lack GPU kernels or fall back to CPU at ≥10×\ge10\times latency, leaving MLX-Q4 as the practical Apple-GPU path. The combination of two families and two precisions gives four cells, each run on both datasets. A cell on one dataset is a per-task cell; a cell on both datasets is a pooled cell.

Datasets. We use the GSM8K test split [9] and MATH-500, the HuggingFaceH4 500-problem evaluation subset of MATH [14, 20]. Each cell runs the first 200 problems per dataset, giving 1 600 prompt-level observations across the per-task grid, and we pool the two tasks into a 400-per-cell aggregate to test whether the gate transfers across tasks.

Prompt and sampling protocol. One in-context exemplar is followed by the zero-shot CoT cue [18] “Let’s think step by step.” Sampled traces use temperature 0.7 and top-p 0.95; the deterministic pass uses T=0T=0. Per prompt we collect one greedy answer and N=5N=5 sampled answers, and SC@5 is plurality voting over the NN samples with a sum-log-probability tiebreak. Answer parsing is a layered regex over \boxed{}, “The answer is X”, “####”, and finally the last numeric expression in the output. All runs use seed 1 and deterministic MLX sampling.

Evaluation. Raw accuracies are reported as point estimates. FP16-vs-Q4 deltas use paired 1 000× bootstrap at the prompt level. Gate operating points at fixed threshold τ\tau use per-cell 1 000× bootstrap over prompts. Where we report retention R(τ)=acc⁡gate(τ)/acc⁡always-SCR(\tau)=\operatorname{acc}_{\mathrm{gate}}(\tau)/\operatorname{acc}_{\mathrm{always\text{-}SC}}, the denominator is the empirical SC@5 accuracy on the same cell. Two properties of these intervals matter. Every bootstrap resamples prompts while holding each prompt’s greedy and SC outcomes fixed, so an interval says how much a conclusion depends on which prompts were drawn, not how much it would move under a different generation seed. With one seed per cell, sampling variance in SC itself is not covered by any interval we report. Retention is also a ratio with a small denominator here (SC accuracy is 15–36%), so its intervals are wide even where the point estimate is stable. We therefore keep skip rate and retention as separate claims of different strengths throughout.

Cost accounting. We count compute in units of one generation pass, which keeps the accounting hardware-independent at the cost of ignoring prompt-length, batching and memory effects. Always-greedy costs 1 and always-SC costs NN. A gate always pays for the greedy pass, which provides both the fallback answer and the gate features, so a skipped prompt costs exactly 1 pass and nothing more. When the gate invokes SC we charge NN, treating the greedy trace as one of the NN votes and drawing N−1N-1 further samples:

C=1+(N−1)pinvoke,(1)C = 1 + (N-1)p_{\mathrm{invoke}}, \tag*{(1)}

where pinvokep_{\mathrm{invoke}} is the fraction of prompts on which the gate invokes SC. Our measured SC@5 accuracies come from five temperature-sampled traces rather than four plus the greedy one, so (1) is mildly idealised: an implementation requiring five fresh samples pays 1+N1+N on invoked prompts, and the compute removed from always-SC is then 25–79% rather than the 36–79% we report.

Hardware and compute budget. Everything runs on one MacBook Pro M4 Pro (24 GB unified memory, 16-core GPU, MLX 0.31, mlx-lm 0.31, PyTorch 2.11 with Metal Performance Shaders (MPS), Python 3.13), with no CUDA and no cloud. Generation takes ∼6.7\sim6.7 GPU-hours across the four cells, result consolidation under three minutes, and the released code, cached records and seeds reproduce every table and figure end-to-end.

The Oracle-Dominance Finding

Structural Decomposition

For any prompt x∈Xx \in\mathcal{X}, let g(x),s(x)∈{0,1}g(x),s(x) \in\{0,1\} indicate whether greedy and SC@5 are correct, and define the help and harm indicators y(x)=1[s(x)=1∧g(x)=0]y(x)=\mathbf{1}[s(x)=1 \land g(x)=0] and h(x)=1[g(x)=1∧s(x)=0]h(x)=\mathbf{1}[g(x)=1 \land s(x)=0]. Writing π\pi for the test distribution and E\mathbb{E} for Eπ\mathbb{E}_{\pi}, the accuracies of greedy, always-SC, and a per-prompt oracle that picks the better of the two are

αg=E[g],αsc=E[g]+E[y]−E[h],αo=E[g]+E[y],(2)\alpha_g = \mathbb{E}[g], \qquad\alpha_{\mathrm{sc}} = \mathbb{E}[g] + \mathbb{E}[y] - \mathbb{E}[h], \qquad\alpha_o = \mathbb{E}[g] + \mathbb{E}[y], \tag*{(2)}

where αsc=E[s]\alpha_{\mathrm{sc}}=\mathbb{E}[s] and αo=E[max⁡(g,s)]\alpha_o=\mathbb{E}[\max(g,s)]. We write Δsc=E[y]−E[h]=αsc−αg\Delta_{\mathrm{sc}}=\mathbb{E}[y]-\mathbb{E}[h]=\alpha_{\mathrm{sc}}-\alpha_g for SC’s net gain over greedy. The oracle’s gain over always-SC is then exactly

αo−αsc=E[h],(3)\alpha_o-\alpha_{\mathrm{sc}} = \mathbb{E}[h], \tag*{(3)}

so always-SC is strictly dominated whenever E[h]>0\mathbb{E}[h] > 0. The community’s view of SC implicitly assumes E[h]≪E[y]\mathbb{E}[h] \ll\mathbb{E}[y], which holds for large models but, as we show below, not for small ones.

Empirical Measurement

Figure 1 plots the oracle-retention ratio ρ=αo/αSC\rho= \alpha_o/\alpha_{\mathrm{SC}} across all 12 experiments (4 model-precision cells × (2 datasets + pooled)). Every bar lies at or above the 100% always-SC reference, with the 174% outlier at Qwen-0.5B-Q4 on MATH-500, where always-SC leaves 7 pt of reachable accuracy unused. The rest fall between 109% (Gemma-FP16 on GSM8K) and 146% (Qwen-Q4 pooled), and pooled retention (110–146%) tracks per-task retention, as is desirable for a gate meant to transfer across tasks. The oracle runs SC only on prompts where SC helps, so by Eq. (1) its compute is 1+(N−1)E[y]1 + (N - 1)\mathbb{E}[y]. With the help rates of Table 1, this is 23–30% of always-SC’s cost in all 12 experiments.

Figure 1. Oracle TTC-gate retention ρ=αo/αSC\rho= \alpha_o/\alpha_{\mathrm{SC}} across all 12 experiments (4 model-precision cells × 2 datasets + pooled). The annotated Qwen-0.5B×Q4×MATH-500 bar is the worst cell, where the oracle is 7 pt more accurate than always-SC, a retention of 174%. The dashed line at 100% marks always-SC parity, and retention lies in [109%, 174%] everywhere.

CellDatasetαg\alpha_gαsc\alpha_{\mathrm{sc}}E[y]\mathbb{E}[y]E[h]\mathbb{E}[h]Δsc\Delta_{\mathrm{sc}}
Qwen-FP16GSM8K39.0%40.5%12.0%10.5%+1.5
Qwen-Q4GSM8K20.5%24.5%12.5%8.5%+4.0
Gemma-FP16GSM8K37.5%44.5%11.0%4.0%+7.0
Gemma-Q4GSM8K15.0%17.5%7.0%4.5%+2.5
Qwen-FP16MATH-50018.0%19.5%6.0%4.5%+1.5
Qwen-Q4MATH-50012.5%9.5%4.0%7.0%−3.0
Gemma-FP16MATH-50022.5%28.0%9.0%3.5%+5.5
Gemma-Q4MATH-50010.0%12.0%5.0%3.0%+2.0

Table 1. Per-cell accuracies and the per-prompt decomposition of SC’s outcomes, n=200n=200 per row. αg\alpha_g and αsc\alpha_{\mathrm{sc}} are greedy and SC@5 accuracy, E[y]\mathbb{E}[y] the “SC helps” rate, E[h]\mathbb{E}[h] the “SC hurts” rate, and Δsc=E[y]−E[h]\Delta_{\mathrm{sc}} = \mathbb{E}[y] - \mathbb{E}[h] the net SC-minus-greedy gain (pt). On Qwen-Q4×MATH-500, E[h]>E[y]\mathbb{E}[h] > \mathbb{E}[y] and SC is net-negative.

Table 1 lists the underlying accuracies. FP16 greedy accuracies span 18–39% and Q4 greedy accuracies 10–20.5%. The paired FP16-vs-Q4 drop runs from 5.5 to 27 pt across greedy and SC@5, and all eight 95% paired-bootstrap CIs exclude 0 (half-widths 4.8–7.5 pt). The SC@5-minus-greedy gain is +1.5 to +7 pt on seven of the eight per-task cells and −3 pt on Qwen-Q4×MATH-500. Even the weakest claim for SC, that it is never worse than greedy, fails at this scale.

The same table gives the prompt-level outcome decomposition. The help rate E[y]\mathbb{E}[y] runs from 4% (Qwen-Q4 on MATH-500) to 12.5% (Qwen-Q4 on GSM8K) and the harm rate E[h]\mathbb{E}[h] from 3% (Gemma-Q4 on MATH-500) to 10.5% (Qwen-FP16 on GSM8K), so harm is non-trivial in every cell, and on Qwen-Q4×MATH-500 its 7% against 4% of help inverts the sign of ΔSC\Delta_{\mathrm{SC}}. The oracle margin in Figure 1 then behaves as Eq. (3) requires: on that cell the observed margin is 7 pt, matching E[h]\mathbb{E}[h] to the percentage point, which confirms that the decomposition is measuring what we think it is.

When Does SC Hurt?

A harm event requires a wrong answer to win a plurality over a correct one in five samples, which points to the shape of the model’s distribution over candidate answers, so we measured that shape on the cached traces. Pooling both datasets, the five samples yield on average 3.8 distinct answers per prompt for Qwen-FP16 and 3.0 for Gemma-FP16, and the winner carries only 42% and 58% of the five votes. The plurality is thin, at most two votes of five, on 72% of Qwen-FP16 and 47% of Gemma-FP16 prompts. Harm concentrates in that thin regime: across the four pooled cells, 80–90% of harm events were decided by a winner holding no more than two votes, and on 42–67% of them the correct answer was among the five samples and lost the vote anyway. Much of the harm is thus an aggregation failure rather than a sampling failure, which is why drawing more traces is not the obvious solution.

Quantisation moves every one of these quantities in the wrong direction. Under Q4 the mean number of distinct answers rises (Qwen 3.8 to 4.2, Gemma 3.0 to 3.5), the winning vote share falls (42% to 35%, 58% to 48%), and thin pluralities become more common (72% to 84%, 47% to 62%). Flattening the answer distribution is precisely what weakens a plurality vote, and the direction matches the numerical-computation and planning errors that low-bit quantisation is reported to induce on math reasoning [19]. The outcomes follow the same pattern: on Qwen-Q4×MATH-500 the help rate falls 2 pt relative to Qwen-FP16 while the harm rate rises 2.5 pt, inverting Δsc\Delta_{\mathrm{sc}}. Four cells at one seed cannot separate quantisation from what correlates with it, so we read this as a consistent association rather than an established mechanism, though it does rule out the harm being an artefact of one model or dataset. The two rates are simply both small and of the same order here, and a plurality vote cannot distinguish that case from the large-model one.

The TTC-Gate

Problem Formulation

We want a scalar score function σ:X→[0,1]\sigma:\mathcal{X}\to[0,1] that maps each prompt, given its greedy-pass statistics, to a “run-SC” propensity. For threshold τ∈[0,1]\tau\in[0,1] the gate policy is

πτ(x)={accept SC@5if σ(x)≥τ,accept greedyif σ(x)<τ,(4)\pi_{\tau}(x)= \begin{cases} \text{accept SC@5} & \text{if }\sigma(x)\geq\tau,\\ \text{accept greedy} & \text{if }\sigma(x)<\tau, \end{cases} \tag*{(4)}

with aπ(x)∈{0,1}a_{\pi}(x)\in\{0,1\} denoting the policy’s correctness. Its compute is Eq. (1) at pinvoke=Pr⁡[σ(x)≥τ]p_{\mathrm{invoke}}=\Pr[\sigma(x)\geq\tau], written C(τ)C(\tau), and its accuracy retention is R(τ)=Eπ[aπ]/αSCR(\tau)=\mathbb{E}_{\pi}[a_{\pi}]/\alpha_{\mathrm{SC}}. The goal is to maximise the skip rate κ(τ)=Pr⁡[σ(x)<τ]\kappa(\tau)=\Pr[\sigma(x)<\tau] (equivalently, to minimise C(τ)C(\tau)) subject to R(τ)≥R0R(\tau)\geq R_0.

Gate Instances

We instantiate three gates, ordered by model capacity:

CONF_ONLY. A single hand-crafted feature: the mean greedy-pass token margin confidence mc(x)=1T∑t=1T(pt(1)−pt(2))\mathrm{mc}(x)=\frac{1}{T}\sum_{t=1}^{T}(p_t^{(1)}-p_t^{(2)}), where pt(1),pt(2)p_t^{(1)},p_t^{(2)} are the two largest probabilities of the softmax over the top-5 logits at greedy-step tt. The gate score is σconf(x)=1−mc(x)\sigma_{\mathrm{conf}}(x)=1-\mathrm{mc}(x), min–max rescaled to [0,1][0,1] over the cell’s prompts, a monotone map that only sets the scale on which τ\tau is defined. Nothing is learned and τ\tau is the only parameter.

LOGISTIC. A 12-feature L2L_2-regularised logistic regression. The feature vector ϕ(x)∈R12\phi(x)\in\mathbb{R}^{12} concatenates four greedy-pass statistics (mean margin confidence mc(x)\mathrm{mc}(x), mean top-5 entropy, mean full-vocab entropy, greedy token count) with eight prompt-text counts (characters, words, digits, arithmetic operators, question marks, commas, lines, and multi-step connectives such as “then” or “given”). Writing ϕ~(x)=Σ−1/2(ϕ(x)−μ)\tilde{\phi}(x)=\Sigma^{-1/2}(\phi(x)-\mu) for the train-fold standardisation, the gate score is

σlog(x)=(1+e−(wTϕ~(x)+b))−1,(5)\sigma_{\mathrm{log}}(x)=\left(1+e^{-(w^{\mathsf{T}}\tilde{\phi}(x)+b)}\right)^{-1}, \tag*{(5)}

fitted by class-weighted cross-entropy with L2L_2 penalty λ=1\lambda=1, on the training label yi=y(xi)y_i=y(x_i), i.e. “SC flips a wrong greedy answer to right”. Class weights ωyi∝1/Pr⁡(y=yi)\omega_{y_i}\propto1/\Pr(y=y_i) counter the label imbalance, and scikit-learn defaults are used otherwise.

MLP. A two-hidden-layer MLP with widths 32×1632\times16 over the same ϕ~(x)\tilde{\phi}(x), ReLU activations, L2L_2 regularisation on all weights, trained on the same labels by unweighted cross-entropy, as scikit-learn’s MLP supports no class weights.

Training and Threshold Selection

Within each cell we run 5-fold cross-validation over the nn prompts. Per fold, the standardisation (μ,Σ\mu, \Sigma) and the classifier are fit on the four training folds and applied to the held-out fold, so each prompt is scored by a model that never saw it. Pooling the five held-out folds gives one out-of-fold score per prompt, {σ(xi)}i=1n\{\sigma(x_i)\}_{i=1}^{n}, on which the Pareto curves and operating points of § 5.4 are computed. The operating threshold is

τ⋆=arg⁡max⁡τ∈[0,1] κ(τ)s.t.R(τ)≥R0,(6)\tau^\star= \underset{\tau\in[0,1]}{\arg\max}\ \kappa(\tau) \quad\text{s.t.}\quad R(\tau) \ge R_0, \tag*{(6)}

with R0=0.98R_0=0.98, asking for the largest compute saving compatible with near-lossless accuracy. We then fix τ⋆\tau^\star and bootstrap-resample prompts 1 000 times for 95% CIs on (κ,C,R)(\kappa, C, R).

One consequence of this protocol should be stated before the results. The scores are out of fold, but τ⋆\tau^\star is chosen on the same out-of-fold pool on which we then report (κ,C,R)(\kappa, C, R), with no outer split around the selection, so the reported retention is mildly optimistic by an amount n=400n=400 cannot measure without a separate threshold set. Eq. (6) also places τ⋆\tau^\star exactly where R(τ)R(\tau) has just crossed R0R_0, the least favourable point for a reused estimate.

Pooled-Cell Operating Points

Figure 2 traces each gate’s Pareto curve through the (C,R)(C,R) plane on the four pooled cells (n=400n=400), with gold stars at the oracle and hollow circles at the best τ⋆\tau^\star under R≥0.98R \ge0.98. On Qwen-Q4 the gate collapses to always-greedy (C≈1.03C \approx1.03), the right response when SC adds only 0.5 pt over greedy; on the other cells it lies at C≈1.6C \approx1.6–3.2. Logistic wins 3/4 pooled cells and conf_only the fourth, with mlp best on none.

Figure 2. TTC-gate Pareto curves on the four pooled cells (n=400n=400 per cell). Each curve traces τ\tau across the (C,R)(C,R) plane. Gold star: oracle upper bound (annotated value is the true retention, plot is clipped at 115% for readability). Hollow circle: best operating point at R≥0.98R \ge0.98. Italic labels mark always-greedy (C=1C=1) and always-SC (C=5C=5). Logistic beats CONF_ONLY on 3/4 pooled cells, and MLP is best on none.

Table 2 reports the numerical operating points. On Qwen-Q4, conf_only skips 99.3% at C=1.03C=1.03: pooled SC accuracy is only 0.5 pt above greedy, so the gate essentially disables SC, driven by the net-negative Qwen-Q4×\timesMATH-500 cell. Skip rates across the four cells are 44.7–99.3% with CI half-widths under 5 pt, removing 36–79% of always-SC compute. Retention is 98.1–98.8% as a point estimate but carries a bootstrap lower bound of 76–86%, so we do not state the compute and accuracy claims as a single claim.

CellMethodτ⋆\tau^\starskip% [CI]CC [CI]RR [CI]
Qwen-FP16logistic0.62586.0 [82.5, 89.3]1.56 [1.43, 1.70]98.1 [82.5, 112.5]
Qwen-Q4conf_only0.91599.3 [98.5, 100.0]1.03 [1.00, 1.06]98.7 [76.5, 120.6]
Gemma-FP16logistic0.4644.7 [40.0, 49.5]3.21 [3.02, 3.40]98.8 [86.2, 112.4]
Gemma-Q4logistic0.51557.9 [53.2, 62.7]2.68 [2.49, 2.87]98.5 [77.9, 122.0]

Table 2. Table 2. Pooled (n=400n=400) TTC-gate operating points. τ⋆\tau^\star is chosen by Eq. (6) with R0=0.98R_0=0.98. The best gate skips 45–99% of SC calls, which removes 36–79% of always-SC compute under Eq. (1). Bootstrap (1 000×\times) 95% CIs in brackets. The skip-rate CIs are tight, with half-widths under 5 pt. The retention CIs are not tight, with lower bounds of 76–86%, because retention divides by an αsc\alpha_{\mathrm{sc}} of 0.15–0.36 on these cells. The 98.1–98.8% retention figures are therefore point estimates, not intervals, and τ⋆\tau^\star was selected on this same out-of-fold pool (§ 5).

Feature and Capacity Ablation

We compare CONF_ONLY (1 feature), LOGISTIC (12 features) and MLP (12 features, 32 × 16 hidden) at both the n=200n=200 per-task scale and the n=400n=400 pooled scale.

At the n=200n=200 per-task scale the training signal is sparse: each fold trains on 160 prompts whose positive rate is only 4–12.5%, i.e. 6–20 positive examples. At that size MLP collapses to τ=0\tau=0 (“always run SC”) on six of the eight per-task cells, and between the two usable gates the split is clear, with CONF_ONLY winning three of the four GSM8K cells and LOGISTIC all four MATH-500 cells: one confidence feature suffices where prompts are homogeneous, and the extra features help where they are not. Pooling to n=400n=400 roughly doubles the positives per fold, to 19–32, and LOGISTIC then overtakes CONF_ONLY on 3/4 cells.

The MLP is the informative negative result. It never wins at the pooled scale, and across all twelve experiments it is the best gate exactly once, on Qwen-FP16×GSM8K, where its 82% skip against 78% for CONF_ONLY lies inside the bootstrap CIs of both. Elsewhere it collapses to always-SC or falls behind a linear gate. The decision surface is evidently close to linear in the scaled features, so the extra capacity can only fit fold-specific noise.

In the pooled Qwen-FP16 logistic gate the five largest standardised coefficients are four correlated prompt-length and layout counts (commas, lines, words, characters, ∣wj∣=0.84|w_j| = 0.84–0.910.91) plus greedy full-vocab entropy (+0.810.81). Collinearity splits the length features’ joint signal across opposite signs, so the informative quantity is their weighted combination and not any single coefficient. A surface this close to linear is consistent with the MLP’s failure to improve on it. Figure 3 shows why we do not rank the gates at n=200n=200 at all: the per-task intervals overlap heavily between methods, and only the pooled points are clearly separated.

Figure 3. Operating points in the (skip rate, retention) plane with 1 000× bootstrap 95% CI bars on both axes, one panel per model-precision cell. Shape marks GSM8K (circles), MATH-500 (squares) and pooled (stars) evaluations, colour marks the gate, and the dashed line marks always-SC parity. Pooled points carry the tightest intervals; the per-task ones overlap between methods.

Discussion

Why is always-SC dominated? The guarantees available for sampling-based TTC [10, 36] assume mild concentration around the correct mode, and §4 is a measurement of that assumption failing. It also explains why methods that only shrink NN [1, 2, 17] cannot repair it: fewer samples change the variance of the vote, not the fact that the vote is taken over a distribution whose mode is wrong.

Gate capacity and data scale. The ablation of §5.5 shows that capacity is not the limiting factor at this data scale, not that nonlinearity is useless in principle: what the pooled logistic gate gains over CONF_ONLY is a weighted combination of prompt-length features and greedy entropy that the confidence margin alone misses, and the MLP’s single win lies inside the bootstrap noise.

Why is Gemma’s gate weaker than Qwen’s? Across Qwen cells the best gate skips 59–100% of prompts, and across Gemma cells only 10–58%. Part of that gap is not a property of the gate. Gemma’s answer distributions are the more concentrated ones (§ 4) and its pooled harm rate is roughly half Qwen’s (3.8% against 7.5–7.8%), so SC is more often worth running and there is less to skip. That explains the direction but not the size of the gap. Gemma-3’s post-training pipeline [12] and choices such as sliding-window attention could each compress the per-prompt spread of the features the gate reads, and establishing which distributions actually differ would take a study we have not run.

Composition and deployment. Adaptive-NN methods [1, 28] act after the invocation decision and the gate acts before it, so the savings multiply: a skipped prompt costs one pass instead of five, while cutting NN from 5 to 2 saves at most 2.5×2.5\times on a prompt that still runs. The gate is otherwise drop-in, with no retraining, no verifier, no sampler change, and one scalar τ∗\tau^* re-tunable from a deployment’s own (C,R)(C, R) curve. We should be precise about what 36–79% means, though: it counts generation passes removed (Eq. (1)), not latency or energy, and how much of it a phone obtains depends on batching, prefill reuse and memory pressure our accounting does not model [30, 3, 4]. The gate comes before any cross-model cascade [35, 23, 6], which makes selective SC a plausible first stage in a multi-level TTC stack rather than a substitute for one.

Broader implications. The efficient-TTC literature treats SC as a monotone gain whose only open question is how many samples to draw, but on the models we test the prior question of whether to sample at all is the one that changes accuracy. Two consequences follow, both scoped to this regime. Benchmark tables reporting SC@NN accuracy without the help/harm split can conceal a set of prompts on which SC degrades an answer greedy already had. And Eq. (3) is not specific to plurality voting: any aggregation step that can turn a correct base answer wrong has an oracle margin of exactly its own harm rate, so the same check is worth running for best-of-NN reranking and self-refinement.

Limitations

Our grid is two model families, both ≤1B\le1\mathrm{B} parameters, at two precisions, on two math benchmarks, at N=5N=5, with one generation seed per cell. Code generation, commonsense question answering, open-ended tasks, larger models, and reasoning-distilled or reinforcement-learning-trained small models are untested. We would expect distilled reasoners to concentrate the correct mode and so shrink E[h]\mathbb{E}[h], weakening the case for a gate, and larger NN to shrink it as well, but neither expectation is measured here. What we take to be transferable beyond the grid is the decomposition and the practice of reporting a harm rate, not the numeric ranges.

The statistical caveats add up. Every CI resamples prompts with each prompt’s greedy and SC outcomes held fixed, so it describes sensitivity to the prompt sample and says nothing about run-to-run sampling variance. The cell carrying the most weight in our framing, the negative Δsc\Delta_{\mathrm{sc}} on Qwen-Q4×\timesMATH-500, rests on n=200n=200 prompts at one seed, and two or three seeds on the two Qwen cells is the first experiment we would add. The 98.1–98.8% retention figures are also point estimates whose CIs reach down to 76–86% (Table 2), since retention divides by an SC accuracy of only 15–36%. Skip rate and compute are the well-supported claims, with CI half-widths under 5 pt. Retention is less well supported, especially because τ∗\tau^* is selected on the pool it is then evaluated on (§ 5).

Two further gaps are worth noting. Costs are counts of generation passes under Eq. (1), with the greedy trace treated as one of the NN votes. We report no wall-clock latency, energy, or peak-memory measurement, and an implementation unable to reuse the greedy trace would have 25–79% rather than 36–79% of always-SC compute removed. And while the gate is orthogonal to adaptive-NN methods by construction, we neither implement Adaptive-Consistency [1], confidence-guided stopping [2, 17], budget prediction [15] or cross-prompt allocation [22] as baselines, nor measure the gate composed with them. A deployment choosing among these options needs exactly that comparison, and its absence is the clearest gap here. An adaptive τ\tau and a three-way greedy / SC / SC-plus-verifier gate are likewise untested.

Conclusion

For prompted CoT with SC on the small models we study, “always run SC” is dominated by a per-prompt oracle gate in every one of our 12 experiments. The oracle’s margin equals the rate at which SC flips correct greedy answers to wrong, and at this scale that rate is large enough to matter: 3–10.5% of prompts, and on one cell enough to leave SC worse than greedy overall. A gate on greedy-pass signals recovers much of the available saving, skipping 45–99% of SC calls and removing 36–79% of always-SC compute, with retention point estimates of 98.1–98.8% whose intervals are wide enough that we report the compute and accuracy claims separately. The gate is model-local, verifier-free, and cheap enough to run on the device that hosts the model. None of this resolves the question for larger models, larger NN, or non-mathematical tasks, but it does suggest a cheap practice: before reporting that SC@NN improves accuracy on a small model, measure the harm rate and report it next to the gain.

Acknowledgments. This work was supported by the UK Engineering and Physical Sciences Research Council (EPSRC) under grant EP/X03688X/1. This work was also supported by the EPSRC Centre for Doctoral Training in Distributed Algorithms. The authors are grateful for the support of Stephen Pasteris at The Alan Turing Institute.

Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.

References

  1. [1]Aggarwal, P., Madaan, A., Yang, Y., Mausam: Let’s sample step by step: Adaptive-consistency for efficient reasoning and coding with LLMs. arXiv:2305.11860 (2023)arxiv.org/abs/2305.11860
  2. [2]Aghazadeh, E., Ghasemi, A., Beyhaghi, H., Pishro-Nik, H.: CGES: Confidence-guided early stopping for efficient and accurate self-consistency. arXiv:2511.02603 (2025)arxiv.org/abs/2511.02603
  3. [3]Bsharat, S.M., Ranjan, M., Myrzakhan, A., Liu, J., Guo, B., Tang, S., Liu, Z., Li, Y., Shen, Z.: Mobile-MMLU: A mobile intelligence language understanding benchmark. arXiv:2503.20786 (2025)arxiv.org/abs/2503.20786
  4. [4]Cao, J., Ma, Y., Li, X., Ren, Q., Chen, X.: Task-specific efficiency analysis: When small language models outperform large language models. arXiv:2603.21389 (2026)arxiv.org/abs/2603.21389
  5. [5]Chen, L., Davis, J.Q., Hanin, B., Bailis, P., Stoica, I., Zaharia, M., Zou, J.: Are more LLM calls all you need? towards scaling laws of compound inference systems. arXiv:2403.02419 (2024)arxiv.org/abs/2403.02419
  6. [6]Chen, L., Zaharia, M., Zou, J.: FrugalGPT: How to use large language models while reducing cost and improving performance. In: Transactions on Machine Learning Research (TMLR) (2024)arxiv.org/abs/2305.05176
  7. [7]Chen, Q., Qin, L., Liu, J., Peng, D., Guan, J., Wang, P., Hu, M., Zhou, Y., Gao, T., Che, W.: Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv:2503.09567 (2025)arxiv.org/abs/2503.09567
  8. [8]Chuang, Y.N., Sarma, P.K., Gopalan, P., Boccio, J., Bolouki, S., Hu, X., Zhou, H.: Learning to route LLMs with confidence tokens. arXiv:2410.13284 (2024)arxiv.org/abs/2410.13284
  9. [9]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., Schulman, J.: Training verifiers to solve math word problems. arXiv:2110.14168 (2021)arxiv.org/abs/2110.14168
  10. [10]Cordero-Encinar, P., Duncan, A.B.: Certified self-consistency: Statistical guarantees and test-time training for reliable reasoning in LLMs. arXiv:2510.17472 (2025)
  11. [11]Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D.: GPTQ: Accurate post-training quantization for generative pre-trained transformers. In: International Conference on Learning Representations (ICLR) (2023)arxiv.org/abs/2210.17323
  12. [12]Gemma Team: Gemma 3 technical report. arXiv:2503.19786 (2025)
  13. [13]Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: International Conference on Machine Learning (ICML) (2017)arxiv.org/abs/1706.04599
  14. [14]Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., Steinhardt, J.: Measuring mathematical problem solving with the MATH dataset. In: Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track (2021)
  15. [15]Ji, S., Wang, Y., Liu, Y., Zhu, Q., Che, W.: Seer self-consistency: Advance budget estimation for adaptive test-time scaling. arXiv:2511.09345 (2025)arxiv.org/abs/2511.09345
  16. [16]Kang, M., Jeong, J., Cho, J.: T1: Tool-integrated self-verification for test-time compute scaling in small language models. arXiv:2504.0718 (2025)
  17. [17]Kim, J., Yang, N., Min, K., Jung, K.: Reliability-aware adaptive self-consistency for efficient sampling in LLM reasoning. arXiv:2601.02970 (2026)arxiv.org/abs/2601.02970
  18. [18]Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. In: Advances in Neural Information Processing Systems (NeurIPS) (2022)arxiv.org/abs/2205.11916
  19. [19]Li, Z., Su, Y., Yang, R., Xie, C., Wang, Z., Xie, Z., Wong, N., Yang, H.: Quantization meets reasoning: Exploring LLM low-bit quantization degradation for mathematical reasoning. arXiv:2501.03035 (2025)arxiv.org/abs/2501.03035
  20. [20]Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., Cobbe, K.: Let’s verify step by step. arXiv:2305.20050 (2023)arxiv.org/abs/2305.20050
  21. [21]Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.M., Wang, W.C., Xiao, G., Dang, X., Gan, C., Han, S.: AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In: Proceedings of Machine Learning and Systems (MLSys) (2024)DOI
  22. [22]Liu, Z., Qu, H., Yin, X., Sun, H., Han, Y., Chen, T., Deng, Z.: PETS: A principled framework towards optimal trajectory allocation for efficient test-time self-consistency. arXiv:2602.16745 (2026)arxiv.org/abs/2602.16745
  23. [23]Ong, I., Almahairi, A., Wu, V., Chiang, W.L., Wu, T., Gonzalez, J.E., Kadous, M.W., Stoica, I.: RouteLLM: Learning to route LLMs with preference data. arXiv:2406.18665 (2024)arxiv.org/abs/2406.18665
  24. [24]OpenAI: OpenAI o1 system card. arXiv:2412.16720 (2024)
  25. [25]Qwen Team: Qwen2.5 technical report. arXiv:2412.15115 (2024)
  26. [26]She, J., Zheng, W., Liu, Z., Wang, H., Xing, E., Yao, H., Ho, Q.: Token level routing inference system for edge devices. arXiv:2504.07878 (2025)arxiv.org/abs/2504.07878
  27. [27]Snell, C., Lee, J., Xu, K., Kumar, A.: Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv:2408.03314 (2024)
  28. [28]Wang, G., Wu, Y., Chen, J., Li, S.: Reasoning aware self-consistency: Leveraging reasoning paths for efficient LLM sampling. arXiv:2408.17017 (2024)
  29. [29]Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. In: International Conference on Learning Representations (ICLR) (2023)arxiv.org/abs/2203.11171
  30. [30]Wang, Z., Yang, J., Qian, X., Xing, S., Jiang, X., Lv, C., Zhang, S.: MNN-LLM: A generic inference engine for fast large language model deployment on mobile devices. arXiv:2506.10443 (2025)DOI
  31. [31]Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. In: Advances in Neural Information Processing Systems (NeurIPS) (2022)arxiv.org/abs/2201.11903
  32. [32]Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., Hooi, B.: Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. arXiv:2306.13063 (2023)arxiv.org/abs/2306.13063
  33. [33]Xu, H., Peng, B., Awadalla, H., Chen, D., Chen, Y.C., Gao, M., Kim, Y.J., Li, Y., Ren, L., Shen, Y., Wang, S., Xu, W., Gao, J., Chen, W.: Phi-4-Mini-Reasoning: Exploring the limits of small reasoning language models in math. arXiv:2504.21233 (2025)arxiv.org/abs/2504.21233
  34. [34]Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T.L., Cao, Y., Narasimhan, K.: Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems (NeurIPS) (2023)arxiv.org/abs/2305.10601
  35. [35]Zheng, W., Chen, Y., Zhang, W., Kundu, S., Li, Y., Liu, Z., Xing, E.P., Wang, H., Yao, H.: CITER: Collaborative inference for efficient large language model decoding with token-level routing. arXiv:2502.01976 (2025)arxiv.org/abs/2502.01976
  36. [36]Zhou, Z., Tan, Y., Li, Z., Yao, Y., Guo, L.Z., Li, Y.F., Ma, X.: A theoretical study on bridging internal probability and self-consistency for LLM reasoning. arXiv:2510.15444 (2025)arxiv.org/abs/2510.15444

Paper details

vlnc:W_r8sc3tyg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents