WorksComputer Science
Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding
Abstract
Speculative decoding typically characterizes speedup by draft tokens accepted per verification round. Lossless verification preserves the target model's output distribution and thus expected completion length, making this metric a meaningful efficiency proxy. Relaxed (lossy) verification accepts tokens that lossless verification would reject, potentially altering that distribution. We identify a length inflation phenomenon: relaxed verification lengthens completions, primarily by extending reasoning traces, largely offsetting apparent gains in accepted tokens per round. We evaluate existing training-free relaxed verification rules on twelve target-drafter pairs against lossless verification on identical prompts, measuring completion length, verifier rounds, wall-clock time, and task accuracy. At each rule's most relaxed evaluated setting, relaxed verification on average accepts as many tokens per round but produces longer completions. For rules with higher acceptance lengths, tokens accepted only under relaxed verification tend to have low target-model probability, contributing to length inflation. For rules that reduce verifier rounds, accuracy typically drops substantially. Further analysis traces some failures to continued reasoning that exhausts the budget needed to deliver a final answer. Overall, accepted tokens per round alone is an unreliable measure of decoding speed under relaxed verification. Completion length and verifier rounds must also be reported to assess actual efficiency.
Introduction
Large language models generate text autoregressively: every new token requires a full forward pass over the prefix. Speculative decoding accelerates this loop by separating proposal from verification. A cheap draft model proposes several tokens ahead, and the target model scores all of them in one forward pass, committing the longest accepted prefix [29, 18 Chen]
et al., 2023). The technique is now standard in production inference stacks, with learned drafters such as the EAGLE family [21, 22] shipped in vLLM [17], SGLang [37], and TensorRT-LLM [26]. We call such drafters dedicated: a dedicated drafter computes its proposals from the target’s own hidden states, so it is trained for one target and cannot run without it.1 For example, the EAGLE-3 head of Llama-3.1-8B-Instruct is a single transformer layer that drafts from features of a low, a middle, and a high layer of the target. A standalone drafter is instead a complete small language model that drafts from the tokens alone and shares the target’s tokenizer, such as Llama-3.2-1B-Instruct for the same target.
In lossless speculative decoding the acceptance rule preserves the target distribution exactly, so the speedup is capped by how closely the drafter tracks the target [18]. Because the output distribution is unchanged, completion length is unchanged too, and a gain in accepted tokens per round is a gain in speed.
Lossy speculative decoding relaxes the acceptance rule to admit more draft tokens, trading exactness of the output distribution for speed. It also gives up the guarantee on length: a relaxed rule decodes a different distribution, and its completions can be longer or shorter than the target’s. A growing family of training-free methods does this by replacing the target distribution inside verification with a relaxed distribution controlled by a relaxation parameter [33, 19]. These methods report higher acceptance and throughput at what their evaluations describe as negligible quality cost.
Those gains, however, are measured almost entirely in accepted tokens per round and tokens per second [31]. A relaxed rule changes which tokens are committed, and therefore how long its completions run and whether they terminate cleanly. If a method accepts more tokens per round but also inflates completion length, the end-to-end gain shrinks and can invert. We call this length inflation under relaxed verification. The effect is invisible to both metrics. This paper measures it directly (§ 3.3 and § 3.4), characterizes it across rules, tasks, targets, drafters, and settings (§ 3.5 and § 3.6), and traces it to the low-probability tokens that relaxation commits (§ 3.7). The effect matters for reasoning models, whose thinking traces run to thousands of tokens. The added length buys no accuracy: forcing the reasoning trace to end near the generation budget recovers far more correct answers than it loses (§ 4).
Differences with prior work. Prior work has not sufficiently characterized the extent and consequences of length inflation under relaxed verification. Xia et al. [33] discuss longer, rambling responses only in weak-drafter settings. Wang et al. [31] study quality degradation but do not measure completion length. By systematically treating length inflation as a central measurement of relaxed verification, we identify it as a consistent phenomenon across lossy speculative decoding methods, whose final outputs are longer than their lossless counterparts. We further investigate what the additional generation accomplishes and whether its consequences are recoverable. These findings support a concrete explanation for some unsuccessful responses: continued reasoning exhausts the budget needed to deliver a final answer. In summary, we uncover a new prevalent phenomenon in lossy speculative decoding and identify its cause.
We evaluate five training-free relaxed verification rules, namely CACTUS (Hao and Mou, [13]), mentored decoding (Tran-Thien, [30]), reducible fuzzy speculative decoding (Holsman et al., [14]), and the two speculative cascade rules (Narasimhan et al., [25]), implemented as published, on six benchmarks covering math, code, chat, and long context, with twelve target-drafter pairs that combine four targets (GPT-OSS-20B, Qwen3-8B, Llama-3.1-8B-Instruct, and DeepSeek-R1-Distill-Llama-8B) with eight dedicated drafters and three standalone language models (360 cells in all). Our contributions are:
a token-efficiency accounting for relaxed speculative decoding: completion length, verifier rounds, the condition under which a relaxed rule saves decoding time, and the largest length inflation any relaxation of a given drafter can absorb (§ 2.2), with a protocol that compares methods at matched accepted length rather than matched ;
a cross-method, benchmark, target, and drafter measurement over 360 cells (§ 3), with paired bootstrap intervals for every cell, showing that relaxation inflates completion length, that the inflation cancels most of the promised reduction in verifier rounds and in measured time, that this holds at gentle settings as well as loose ones, and that the accuracy cost is large at loose settings and task dependent;
the identification of where the extra length comes from: relaxation admits draft tokens the target rates as unlikely, and the extra text goes into the reasoning trace where the model has one (§ 3.7);
evidence that the added length is waste: on grade-school math it follows a correct answer the model has already written, on competition math it delays the answer, and forcing the reasoning trace to end near the generation budget recovers far more correct answers (§ 4).
Relaxed Speculative Decoding
This section presents lossless and lossy speculative decoding in one framework and states every training-free method as a choice of relaxed target distributions in it; following Xia et al. [33] we also call the two cases strict and relaxed. Appendix § 6 gives each method’s rule and our implementation.
Strict speculative decoding
Let be the target model to accelerate and a cheaper drafter, with next-token distributions and over a vocabulary of size , and a drafted token. In one round of speculative decoding (Algorithm 1 in Appendix § 6, with ), the drafter samples a block of tokens and the target scores all of them, plus the position after the block, in a single forward pass. Each draft token is then accepted with probability ; at the first rejection the token is replaced by a sample from the residual distribution and the round ends. If every draft token is accepted, a bonus token is sampled from at the position after the block. One target forward pass thus commits between one and tokens, and the output is distributed exactly as ([18, 5]).
Let , , and be the wall-clock times of one decoding step of the target , one decoding step of the drafter , and one verification pass over the positions, and let denote the mean number of draft tokens accepted per round. A round costs and commits tokens on average. Under memory-bound conditions , so with the speedup is
The accepted length is bounded by drafter–target agreement: a draft token drawn from is accepted with probability
so strict verification cannot accept more than the two distributions share ([18]). Every relaxed method below is an attempt to lift this cap.
Relaxed speculative decoding
A relaxed method keeps the round of § 2.1 intact and replaces the target distribution at up to three points with relaxed target distributions : in the acceptance distribution, so that a draft token is accepted with probability ; in the residual sampled after a rejection; and at the bonus position. Each is a function of the two distributions, the drafted token, and a method-specific relaxation parameter . Strict speculative decoding is the special case at all three points. The parameter means something different in every method (a divergence budget, a divergence threshold, a deferral cost, a mixture weight), so values of are never comparable across methods; § 3 therefore compares methods at matched rather than matched .
Scope. We restrict attention to training-free rules that relax verification per token, i.e. methods fully specified by a choice of ; rules that train a component, such as a judge head ([2, 11]) or a learned ensemble weight ([32]), expose no to sweep.
Length-corrected speedup. Because a relaxed rule samples from rather than , it can change the length of the generated response, and Equation (1) no longer measures end-to-end speed. We can correct for this with
where denotes mean completion length and is the acceptance-based speedup of Equation (1). This study measures that factor directly: a method wins end to end only if its acceptance gain exceeds its length inflation.
Verifier rounds. The same trade-off can be measured without a hardware model. A completion of tokens generated in verifier rounds (target forward passes) commits tokens per round on average, so
Every round costs drafter steps and one verification pass for every rule in the sweep, so at fixed and drafter the decoding times of two rules are in the ratio of their round counts. Writing for the length inflation relative to strict verification,
and a relaxed rule reduces decoding time only if . The gain on the left is bounded by the draft length: a round accepts at most draft tokens, so no relaxation of a given drafter can beat . § 3 reports , , measured time, and task accuracy directly, at matched .
Training-free methods
Table A1 states every method as a choice of . The five swept rules fall into two groups: divergence-budgeted rules that boost the drafted token while bounding the distance from (CACTUS, mentored decoding), and switch rules that trust the drafter outright when a per-position test passes (fuzzy speculative decoding, speculative cascades).
| Method (arm) | controls | swept | |||
| CACTUS (cactus) (Hao and Mou, [13]) | reverse-KL budget | if , else ; | 0.03, 0.08, 0.18, 0.35 | ||
| Mentored decoding (mentored_dec) (Tran-Thien, [30]) | forward-KL budget | 0.15, 0.35, 0.55, 0.75 | |||
| Fuzzy, reducible variant (r_fuzzy) (Holsman et al., [14]) | divergence threshold | if , else | 0.03, 0.08, 0.15, 0.25 | ||
| Speculative cascade, optimal rule (spec_casc_opt) (Narasimhan et al., [25]) | deferral cost | if , else | |||
| Speculative cascade, token rule (spec_casc_tok) (Narasimhan et al., [25]) | trusted-set width | if , else ; , | 0.15, 0.35, 0.55, 0.8 | ||
| Ensemble (ens) (Wang et al., [32]) | mixture weight | not swept | |||
| Speculative contrastive decoding (spec_cont_dec) (Yuan et al., [35]) | contrast strength | , on the top-/top- mask of , outside | not swept |
Table A1. Training-free relaxed verification rules as instances of Algorithm 1. Strict speculative decoding is throughout. Every rule except the two cascade rules recovers strict verification at ; spec_casc_opt recovers it for (since ) and spec_casc_tok for . On the second sampler path (§ 3.1), spec_casc and spec_casc_tok use . Last column: the four values of the pilot (§ 3.2), used for every pair.
The cascade token rule (spec_casc_tok) is not strict verification even at : draft tokens inside a trusted top set of are accepted outright, and tokens outside it are judged against a scaled-down copy of , so it is the only rule in the sweep that can reject what lossless verification would accept.
Experimental Evaluation
We measure what relaxed verification does to the generated text: five training-free rules, mentored_dec, cactus, spec_casc_opt, r_fuzzy, and spec_casc_tok, run on six datasets with twelve target-drafter pairs, each compared with lossless (strict) speculative decoding on the same pair. One (pair, dataset, rule) combination is a cell, and there are 360. For every cell we report completion length, verifier rounds (target forward passes), measured time, and task accuracy.
Setup
Targets and drafters. We use four targets in twelve target-drafter pairs (Table 3): GPT-OSS-20B (OpenAI, [27]) with two EAGLE-3 heads (Li et al., [22]); Qwen3-8B (Yang et al., [34]) with an EAGLE-3 head, P-EAGLE (Hui et al., [15]), DSpark (Cheng et al., [8]), and the standalone Qwen3-0.6B and Qwen3-1.7B; Llama-3.1-8B-Instruct (Grattafiori et al., [12]) with EAGLE-3, EAGLE-1 (Li et al., [21]), and the standalone Llama-3.2-1B-Instruct; and DeepSeek-R1-Distill-Llama-8B (DeepSeek-AI, [10]) with EAGLE-3 and Llama-3.2-1B-Instruct (checkpoints in Appendix § 9). The EAGLE-1, EAGLE-3, P-EAGLE, and DSpark heads are the eight dedicated drafters; Qwen3-0.6B, Qwen3-1.7B, and Llama-3.2-1B-Instruct are the three standalone ones. Two pairs, GPT-OSS-20B with the nebius EAGLE-3 head and Qwen3-8B with the RedHatAI one,2 are measured in more depth (more seeds, timing on one server, run-log analyses, and the controls of Appendices § 7, C, and E); we call them the primary pairs. Every pair runs on vLLM 0.26.0 on a single NVIDIA H100 with (within 10% of the fastest lossless draft length; Appendix § 8), sampled drafts, temperature 1.0, top- 1.0, and seed 0; the bundled sampling defaults of Qwen3-8B and the Llama models are overridden (Appendices § 8 and D).
Implementation. vLLM exposes no relaxed acceptance rule, so each method is a patch to its rejection sampler that implements the formulas of nebius/EAGLE3-gpt-oss-20b and RedHatAI/Qwen3-8B-speculator.eagle3 on the Hugging Face Hub. Table A1, applied one at a time. Each patch ships with tests that check the formula against closed-form values.3 Code, patches, prompts, and per-run outputs are released.4
Datasets. Six datasets cover four regimes: mathematics (GSM8K (Cobbe et al., [9]), AIME24 (Mathematical Association of America, [23])), code generation (HumanEval (Chen et al., [6]), LiveCodeBench (Jain et al., [16])), open-ended chat (MT-Bench (Zheng et al., [36])), and long-context multiple-choice question answering (LongBench-v2 (Bai et al., [3])). GSM8K, AIME24, and LongBench-v2 are graded by exact match on the final answer; HumanEval and LiveCodeBench by executing the generated code against the problem’s tests. A run that never produces an answer is scored wrong. MT-Bench is scored by an LLM judge (Table B4). Case counts, generation budgets, and lossless references are listed in Table B1 and Table D3. A run that ends at its budget is counted as cut off.
| Dataset | Acc. | cut off | ||||
| GPT-OSS-20B | ||||||
| GSM8K | 150 | 2.57 | 322 | 94 | 0.960 | 2 |
| AIME24 | 30 | 2.19 | 8 717 | 2 953 | 0.767 | 7 |
| HumanEval | 150 | 2.53 | 912 | 272 | 0.960 | 0 |
| LiveCodeBench | 90 | 2.16 | 3 456 | 1 151 | 0.889 | 2 |
| MT-Bench | 80 | 2.34 | 1 264 | 414 | 7.29 | 4 |
| LongBench-v2 | 150 | 1.91 | 1 509 | 520 | 0.560 | 1 |
| Qwen3-8B | ||||||
| GSM8K | 150 | 1.51 | 1 292 | 510 | 0.800 | 25 |
| AIME24 | 30 | 1.10 | 19 141 | 9 475 | 0.700 | 17 |
| HumanEval | 150 | 1.42 | 3 670 | 1 535 | 0.827 | 11 |
| LiveCodeBench | 90 | 1.21 | 8 029 | 3 703 | 0.700 | 32 |
| MT-Bench | 80 | 1.25 | 2 073 | 913 | 6.94 | 14 |
| LongBench-v2 | 150 | 0.70 | 2 887 | 1 676 | 0.527 | 3 |
Table B1. Lossless (strict) speculative decoding on the evaluation cases, with no relaxation, for the two primary pairs (§ 3.1). : mean accepted draft tokens per verification round; : mean completion length in tokens; : mean verifier rounds; Acc.: pass rate; for MT-Bench, the mean judge score out of 10 (Table B4); cut off: share of runs (%) that end at the generation budget. Generation budgets (tokens): GSM8K 2 048, AIME24 32 768, HumanEval 9 000, LiveCodeBench 12 000, MT-Bench 4 096, LongBench-v2 8 192. Every comparison in § 3 is against the lossless run of the same pair, not against autoregressive decoding.
| Target, drafter | GSM8K | GSM8K | GSM8K acc. | AIME24 | AIME24 | AIME24 acc. | HumanEval | HumanEval | HumanEval acc. | LiveCodeBench | LiveCodeBench | LiveCodeBench acc. | MT-Bench | MT-Bench | MT-Bench acc. | LongBench-v2 | LongBench-v2 | LongBench-v2 acc. |
| GPT-OSS-20B, EAGLE-3 (RedHatAI) | 2.13 | 353 | .97 | 1.34 | 9 650 | .77 | 2.04 | 911 | .98 | 1.57 | 3 445 | .83 | 1.82 | 1 157 | 7.31 | 0.05 | 1 403 | .54 |
| Qwen3-8B, P-EAGLE | 2.16 | 1 303 | .81 | 2.26 | 18 758 | .73 | 2.05 | 3 863 | .85 | 1.84 | 8 072 | .73 | 1.57 | 2 112 | 6.96 | 1.38 | 3 023 | .55 |
| Qwen3-8B, DSpark | 2.77 | 1 335 | .75 | 2.51 | 19 550 | .70 | 2.50 | 3 756 | .84 | 2.31 | 8 063 | .74 | 2.12 | 2 133 | 6.74 | 1.88 | 2 948 | .47 |
| Qwen3-8B, Qwen3-0.6B | 3.16 | 1 310 | .77 | 3.58 | 17 807 | .67 | 2.90 | 3 814 | .84 | 3.01 | 8 076 | .72 | 2.38 | 1 997 | 6.80 | 2.51 | 2 989 | .46 |
| Qwen3-8B, Qwen3-1.7B | 3.82 | 1 316 | .79 | 4.15 | 19 469 | .73 | 3.71 | 3 876 | .84 | 3.80 | 8 098 | .69 | 3.04 | 2 098 | 6.80 | 3.07 | 2 896 | .45 |
| Llama-3.1-8B-Instruct, EAGLE-3 | 2.34 | 290 | .81 | 0.77 | 3 661 | .03 | 2.70 | 547 | .56 | 2.05 | 770 | .14 | 2.08 | 472 | 5.85 | 0.06 | 525 | .37 |
| Llama-3.1-8B-Instruct, EAGLE-1 | 1.57 | 279 | .79 | 0.77 | 4 842 | .03 | 1.76 | 529 | .58 | 1.42 | 830 | .19 | 1.32 | 468 | 5.95 | .92 | 705 | .35 |
| Llama-3.1-8B-Instruct, Llama-3.2-1B | 4.04 | 269 | .78 | 3.56 | 5 380 | .00 | 3.97 | 544 | .55 | 3.48 | 764 | .16 | 3.26 | 430 | 5.53 | 3.11 | 665 | .37 |
| R1-Distill-Llama-8B, EAGLE-3 | 3.07 | 623 | .77 | 0.60 | 12 527 | .27 | 1.89 | 3 739 | .86 | 0.77 | 8 882 | .58 | 2.13 | 1 384 | 5.40 | 0.03 | 1 911 | .29 |
| R1-Distill-Llama-8B, Llama-3.2-1B | 2.71 | 440 | .71 | 1.83 | 11 025 | .43 | 1.69 | 3 857 | .89 | 1.56 | 8 742 | .56 | 1.64 | 1 373 | 5.30 | 1.62 | 1 802 | .34 |
Table D3. Lossless references of the ten pairs beyond the primary ones (primary pairs: Table B1; same cases and budgets), seed 0: mean accepted length , mean completion tokens , and accuracy (MT-Bench: judge score out of 10).
Metrics. We report five quantities: , the mean accepted length (draft tokens accepted per round); , the mean completion length in tokens; , the mean number of verifier rounds; the mean per-request wall-clock time, measured at the client at batch size 1 and excluding server start; and accuracy, the pass rate over the dataset’s cases (for MT-Bench, the mean judge score). Each case is run once per arm. Every ratio to lossless is computed on the same cases for both arms, and its uncertainty is a 95% paired bootstrap interval over cases (10 000 resamples). A cell on a graded dataset (all but MT-Bench) loses accuracy when it falls two or more points below its lossless reference.
Matching on mean accepted length
Because is not comparable across methods (§ 2.2), we compare methods at matched . A short pilot runs each method’s four values (Table A1) on three cases per dataset; we then fix three target values of and run the full dataset at the closest to each. The five methods never share a reachable , so the match is approximate. The four pairs with a standalone drafter and the P-EAGLE pair run each rule at its loosest only. The tables report each method at its most relaxed setting; Sections 3.3 and 3.6 check that the conclusions do not depend on this choice.
Completion length
Relaxation lengthens completions in 248 of the 360 cells (Table 1 and 3): has median 1.08 and mean 1.54, with the 95% interval above 1 in 172 cells and below 1 in 67. Its median varies more with the rule (0.97 for spec_casc_tok to 1.39 for cactus) and the drafter (0.94 for standalone drafters to 1.23 for EAGLE heads) than with the task (1.04 on GSM8K to 1.14 on HumanEval), and cut-off runs make it a lower bound (§ 3.7).
| Dataset | mentored | cactus | casc-opt | r-fuzzy | casc-tok | mentored | cactus | casc-opt | r-fuzzy | casc-tok |
| GSM8K | 1.16 (.93) | 1.61 (.91) | 1.46 (.94) | 1.70 (.87) | 0.96° (.97) | 1.03° (.77) | 1.04° (.77) | 1.33 (.49) | 0.95° (.55) | 0.99° (.79) |
| AIME24 | 1.69 (.63) | 1.78 (.60) | 2.48 (.37) | 2.13 (.50) | 1.48 (.70) | 1.01° (.73) | 1.26 (.23) | 1.42 (.30) | 1.29 (.40) | 0.98° (.70) |
| HumanEval | 1.30 (.96) | 1.79 (.87) | 1.73 (.82) | 1.64 (.63) | 1.05° (.94) | 1.08 (.85) | 1.14 (.75) | 1.72 (.49) | 1.65 (.16) | 1.08 (.85) |
| LiveCodeBench | 1.25 (.84) | 1.54 (.61) | 1.64 (.49) | 1.65 (.34) | 1.16 (.88) | 1.05 (.63) | 1.15 (.44) | 1.27 (.33) | 1.07° (.02) | 1.00° (.71) |
| MT-Bench | 1.10° (6.41) | 1.18 (4.51) | 1.09° (5.74) | 1.20 (4.58) | 0.96° (7.49) | 1.04° (6.40) | 1.16 (2.99) | 1.22 (3.95) | 1.12 (3.05) | 1.04° (7.21) |
| LongBench-v2 | 1.59 (.51) | 1.77 (.36) | 1.79 (.47) | 1.69 (.40) | 1.13° (.53) | 1.06° (.51) | 2.10 (.29) | 1.16 (.37) | 1.08° (.44) | 1.00° (.49) |
Table 1. Length inflation on the primary pairs (§ 3.1) at each method’s most relaxed tested setting: completion length relative to lossless (; > 1 is inflation), with the accuracy at that setting in parentheses (MT-Bench: judge score out of 10). °: the 95% interval for includes 1 (Table B5). All twelve pairs: Table D1.
| Dataset | Method | [95% CI] | time [95% CI] | cut off | [95% CI] | time [95% CI] | cut off | ||
| GSM8K | mentored_dec | 0.75 | 1.16 [1.02, 1.33] | 0.96 [0.86, 1.08] | 4 | 0.75 | 1.03 [0.98, 1.07] | 0.95 [0.91, 1.00] | 26 |
| cactus | 0.35 | 1.61 [1.41, 1.83] | 1.19 [1.06, 1.33] | 5 | 0.35 | 1.04 [1.00, 1.08] | 0.94 [0.90, 0.97] | 27 | |
| spec_casc_opt | 0.05 | 1.46 [1.28, 1.65] | 1.18 [1.06, 1.31] | 5 | 0.05 | 1.33 [1.26, 1.40] | 1.12 [1.07, 1.18] | 58 | |
| r_fuzzy | 0.25 | 1.70 [1.49, 1.94] | 1.29 [1.15, 1.45] | 5 | 0.25 | 0.95 [0.86, 1.05] | 0.81 [0.74, 0.90] | 35 | |
| spec_casc_tok | 0.8 | 0.96 [0.88, 1.04] | 0.91 [0.85, 0.98] | 1 | 0.8 | 0.99 [0.95, 1.02] | 0.96 [0.92, 1.00] | 23 | |
| AIME24 | mentored_dec | 0.75 | 1.69 [1.34, 2.21] | 1.36 [1.07, 1.81] | 20 | 0.75 | 1.01 [0.91, 1.14] | 0.95 [0.83, 1.10] | 23 |
| cactus | 0.18 | 1.78 [1.37, 2.44] | 1.33 [1.00, 1.87] | 23 | 0.35 | 1.26 [1.10, 1.44] | 0.66 [0.56, 0.77] | 57 | |
| spec_casc_opt | 0.05 | 2.48 [1.89, 3.45] | 2.01 [1.51, 2.83] | 43 | 0.05 | 1.42 [1.21, 1.69] | 1.14 [0.94, 1.39] | 63 | |
| r_fuzzy | 0.25 | 2.13 [1.64, 2.92] | 1.64 [1.26, 2.30] | 30 | 0.25 | 1.29 [1.08, 1.53] | 1.18 [0.96, 1.45] | 37 | |
| spec_casc_tok | 0.8 | 1.48 [1.09, 2.04] | 1.33 [0.96, 1.85] | 20 | 0.8 | 0.98 [0.86, 1.11] | 0.93 [0.81, 1.07] | 13 | |
| HumanEval | mentored_dec | 0.75 | 1.30 [1.15, 1.46] | 1.09 [0.97, 1.21] | 1 | 0.75 | 1.08 [1.01, 1.16] | 1.02 [0.95, 1.09] | 13 |
| cactus | 0.35 | 1.79 [1.61, 1.99] | 1.35 [1.23, 1.49] | 1 | 0.35 | 1.14 [1.07, 1.23] | 1.01 [0.94, 1.09] | 20 | |
| spec_casc_opt | 0.05 | 1.73 [1.52, 1.97] | 1.43 [1.28, 1.60] | 2 | 0.05 | 1.72 [1.57, 1.89] | 1.46 [1.34, 1.60] | 40 | |
| r_fuzzy | 0.25 | 1.64 [1.46, 1.84] | 1.32 [1.18, 1.47] | 1 | 0.25 | 1.65 [1.50, 1.82] | 1.41 [1.28, 1.56] | 38 | |
| spec_casc_tok | 0.8 | 1.05 [0.96, 1.15] | 0.98 [0.91, 1.07] | 0 | 0.8 | 1.08 [1.02, 1.14] | 1.05 [0.99, 1.11] | 11 | |
| LiveCodeBench | mentored_dec | 0.75 | 1.25 [1.14, 1.36] | 1.01 [0.93, 1.11] | 9 | 0.75 | 1.05 [1.01, 1.09] | 0.95 [0.92, 0.99] | 34 |
| cactus | 0.18 | 1.54 [1.33, 1.77] | 1.14 [0.99, 1.32] | 11 | 0.35 | 1.15 [1.08, 1.22] | 0.82 [0.76, 0.89] | 54 | |
| spec_casc_opt | 0.05 | 1.64 [1.46, 1.84] | 1.32 [1.17, 1.48] | 19 | 0.05 | 1.27 [1.20, 1.36] | 1.09 [1.02, 1.16] | 61 | |
| r_fuzzy | 0.25 | 1.65 [1.45, 1.89] | 1.27 [1.12, 1.45] | 16 | 0.25 | 1.07 [0.96, 1.20] | 0.92 [0.82, 1.03] | 61 | |
| spec_casc_tok | 0.8 | 1.16 [1.03, 1.31] | 1.04 [0.92, 1.17] | 7 | 0.8 | 1.00 [0.97, 1.04] | 0.96 [0.92, 0.99] | 30 | |
| MT-Bench | mentored_dec | 0.75 | 1.10 [0.98, 1.22] | 0.87 [0.78, 0.97] | 1 | 0.75 | 1.04 [0.98, 1.11] | 0.93 [0.87, 0.99] | 21 |
| cactus | 0.35 | 1.18 [1.04, 1.35] | 0.82 [0.72, 0.95] | 6 | 0.35 | 1.16 [1.03, 1.29] | 0.75 [0.68, 0.84] | 32 | |
| spec_casc_opt | 0.05 | 1.09 [0.98, 1.21] | 0.87 [0.78, 0.97] | 2 | 0.05 | 1.22 [1.14, 1.32] | 1.01 [0.95, 1.09] | 28 | |
| r_fuzzy | 0.25 | 1.20 [1.07, 1.35] | 0.92 [0.81, 1.04] | 4 | 0.25 | 1.12 [1.04, 1.21] | 0.92 [0.85, 0.99] | 26 | |
| spec_casc_tok | 0.8 | 0.96 [0.88, 1.05] | 0.90 [0.84, 0.98] | 1 | 0.8 | 1.04 [0.97, 1.11] | 0.99 [0.93, 1.05] | 14 | |
| LongBench-v2 | mentored_dec | 0.75 | 1.59 [1.38, 1.83] | 1.04 [1.02, 1.07] | 3 | 0.75 | 1.06 [0.99, 1.14] | 0.99 [0.92, 1.07] | 5 |
| cactus | 0.35 | 1.77 [1.52, 2.05] | 1.03 [1.00, 1.05] | 2 | 0.35 | 2.10 [1.86, 2.37] | 1.05 [0.94, 1.16] | 57 | |
| spec_casc_opt | 0.05 | 1.79 [1.54, 2.07] | 1.07 [1.04, 1.10] | 14 | 0.05 | 1.16 [1.05, 1.28] | 0.90 [0.83, 0.98] | 17 | |
| r_fuzzy | 0.25 | 1.69 [1.45, 1.98] | 1.05 [1.02, 1.07] | 5 | 0.25 | 1.08 [0.99, 1.18] | 0.99 [0.90, 1.07] | 11 | |
| spec_casc_tok | 0.8 | 1.13 [0.98, 1.29] | 1.00 [0.98, 1.02] | 1 | 0.8 | 1.00 [0.92, 1.07] | 0.97 [0.90, 1.04] | 4 |
Table B5. The primary pairs’ cells at each method’s most relaxed tested setting: ; length ratio and per-request time ratio to lossless, each with its 95% paired bootstrap interval over cases (10 000 resamples); cut off: share of runs (%) that end at the generation budget (lossless: Table B1). Round ratios: Table 2; accuracy: Table 1; acceptance gains and round-ratio intervals: released tables.
The inflation persists at each rule’s gentlest setting (on the seven pairs run with the matched protocol, median 1.07 against 1.22 at the loosest, and more rounds than lossless in 97 cells against 94; Table D4) and across seeds (Appendix § 8). At Qwen3-8B’s recommended sampler (temperature 0.6), mentored_dec, cactus, and spec_casc_tok stop inflating on GSM8K and LiveCodeBench, but spec_casc_opt still inflates by 1.5× (Appendix § 8).
| Pair | same | acc. loss | acc. loss | ||||||
| GPT-OSS-20B + EAGLE-3 (nebius) | 0 | 1.07 | 0.98 | 9 | 14 | 1.57 | 1.03 | 15 | 22 |
| GPT-OSS-20B + EAGLE-3 (RedHatAI) | 14 | 1.23 | 1.09 | 22 | 15 | 1.35 | 1.07 | 20 | 17 |
| Qwen3-8B + EAGLE-3 (RedHatAI) | 0 | 1.04 | 0.99 | 12 | 13 | 1.08 | 0.94 | 7 | 18 |
| Qwen3-8B + DSpark | 8 | 1.01 | 0.89 | 1 | 12 | 1.06 | 0.85 | 2 | 18 |
| Llama-3.1-8B-Instruct + EAGLE-3 | 7 | 1.84 | 1.57 | 24 | 18 | 1.69 | 1.35 | 21 | 18 |
| Llama-3.1-8B-Instruct + EAGLE-1 | 12 | 1.23 | 1.13 | 18 | 18 | 2.42 | 1.42 | 21 | 19 |
| R1-Distill-Llama-8B + EAGLE-3 | 14 | 1.07 | 0.95 | 11 | 15 | 1.11 | 0.93 | 8 | 16 |
| All (210 cells) | 55 | 1.07 | 0.99 | 97 | 105 | 1.22 | 0.97 | 94 | 128 |
Table D4. Gentlest against most relaxed tested setting on the seven pairs run with the matched protocol (30 cells each; the settings with the lowest and the highest measured ): median and median over the pair’s cells, cells that need more rounds than lossless (), and graded cells two or more points below lossless (acc. loss, of 25; All: of 175). same: cells whose two settings are the same run.
Verifier rounds and time
The extra acceptance buys few rounds (Table 2 and 3, Figure 1 and Figure 2): over the 360 cells has median 0.93 and mean 1.10, and 114 cells need more rounds than lossless (67 significantly). The round ratio follows length, not acceptance (Spearman 0.78 with , −0.12 with the acceptance gain), and the length-corrected model of Equation (3), which at fixed reduces to Equation (5), predicts it from the measured length within 2% at the median.
| Dataset | ment. | cactus | c-opt | r-fuzzy | c-tok | ment. | cactus | c-opt | r-fuzzy | c-tok |
| GSM8K | 0.83 | 0.99 | 1.07 | 1.14 | 0.83 | 0.93 | 0.90 | 1.05 | 0.76 | 0.95 |
| AIME24 | 1.18 | 1.07 | 1.70 | 1.37 | 1.22 | 0.94 | 0.55 | 1.06 | 1.16 | 0.93 |
| HumanEval | 0.97 | 1.12 | 1.29 | 1.16 | 0.93 | 0.99 | 0.96 | 1.36 | 1.33 | 1.04 |
| LiveCodeBench | 0.89 | 0.94 | 1.16 | 1.11 | 0.98 | 0.93 | 0.72 | 1.03 | 0.87 | 0.94 |
| MT-Bench | 0.75 | 0.62 | 0.76 | 0.76 | 0.88 | 0.89 | 0.64 | 0.94 | 0.85 | 0.96 |
| LongBench-v2 | 1.11 | 0.87 | 1.31 | 1.10 | 0.97 | 0.98 | 0.80 | 0.83 | 0.96 | 0.96 |
Table 2. Verifier rounds relative to lossless () on the primary pairs at each method’s most relaxed tested setting (ment., c-: mentored, casc-). Bold: the 95% interval for lies entirely above 1 (more rounds than lossless); underline: entirely below 1. All twelve pairs: Table D2.

Figure 1. Every tested setting of the primary pairs (151 runs; filled markers GPT-OSS-20B, hollow Qwen3-8B): length inflation against acceptance gain. A relaxed rule saves verifier rounds only below the dashed line (Equation (5)). All twelve pairs, one panel per rule: Figure 2.

Figure 2. All 360 cells at each rule’s most relaxed tested setting, one panel per rule (log scales; outlined: the primary pairs). A cell saves verifier rounds only below the dashed line (Equation (5)). Above it: 21, 31, 31, 38, and 11 of each rule’s 72 cells (mean gain 1.21, 1.74, 1.41, 1.22, 1.11), and 111 of 210 cells with an EAGLE-3, EAGLE-1, or P-EAGLE head, 3 of 30 with DSpark, and 18 of 120 with a standalone drafter.
Time is worse than rounds on the primary pairs (median ratio 1.01 against 0.96; eleven cells save rounds but not time) because on their server a round’s cost grows with the tokens it commits; on a second server they agree (Appendix § 8), so rounds are the hardware-independent measure.
Accuracy retention
Relaxation costs accuracy in 198 of the 300 graded cells (lossless references: Table B1 and Table D3; MT-Bench: Appendix § 7). The three aggressive rules lose in 143 of their 180 graded cells, by 0.19 on average for cactus, 0.14 for spec_casc_opt, and 0.21 for r_fuzzy; mentored_dec loses in 33 of 60 cells (mean 0.03) and spec_casc_tok in 22 of 60 (mean 0.005).
The losses follow the inflation: 132 of the 144 graded cells whose interval lies above 1 lose accuracy (mean loss 0.21), against 66 of the other 156 (mean 0.03). By task they are largest on code generation, with median losses of 0.09 on HumanEval and 0.11 on LiveCodeBench against 0.03 to 0.04 on the other graded datasets.
Analysis
Settings that need fewer verifier rounds lose accuracy. With an EAGLE head, the three aggressive rules need fewer rounds than lossless in 40 of their 105 graded cells, and 35 of these lose accuracy, most where the cut is deepest: cactus cuts rounds to 0.44× on R1-Distill-Llama-8B/LiveCodeBench and to 0.55× on Qwen3-8B/AIME24, while accuracy falls from 0.58 to 0.02 and from 0.70 to 0.23. Of the 86 graded cells that save rounds and keep accuracy, 68 use spec_casc_tok, DSpark, or a standalone Qwen3 drafter, the choices closest to the target (below). On the primary pairs, tuning for accuracy saves a few percent of the rounds at best: only 29 of the 50 graded cells have a tested setting within two points of lossless, and the best of these have a median round ratio of 0.96 (Table B3).
Keeping accuracy means staying close to the target. spec_casc_tok, the only rule that can reject what lossless verification accepts (§ 2.3), is also the only one that keeps accuracy on average: over its 60 graded cells it loses 0.005, against 0.03 to 0.21 for the other rules, and its interval lies above 1 in 5 of its 72 cells, against 28 to 51. Its acceptance gain is the smallest (1.11×), yet because its completions do not grow (mean 0.96) it needs 0.86× the rounds of lossless on average.
The inflation is a property of the drafter, not of the target. The same target inflates very differently with each drafter (Table 3, Figure 2): Llama-3.1-8B-Instruct by 3.30× on average with its EAGLE-3 head and 1.10× with Llama-3.2-1B, Qwen3-8B by 1.18× with its EAGLE-3 head and 0.90× with Qwen3-1.7B. Every EAGLE head inflates ( interval above 1 in 132 of 210 cells, below 1 in 8), and Llama-3.1-8B-Instruct with its EAGLE head is the worst case: its lossless answers run 280 to 4 840 tokens, and cactus multiplies them by 4.0 to 13.2. Drafters that sample close to the target inflate little: DSpark has median 1.06 and a round interval below 1 in 22 of its 30 cells, and the standalone Qwen3 drafters lengthen completions significantly in none of their 60 cells and cut rounds by 10% to 46%, losing accuracy in 10 of 50 graded cells against 18 and 24 of 25 for DSpark and P-EAGLE. A standalone drafter from another model’s distribution inflates again: Llama-3.2-1B on R1-Distill-Llama-8B, up to 3.28× on LongBench-v2.
| Target, drafter | gain | ment. | cactus | c-opt | r-fuzzy | c-tok | all | ment. | cactus | c-opt | r-fuzzy | c-tok | all | acc. loss | |
| GPT-OSS-20B | |||||||||||||||
| EAGLE-3 (nebius) | 2.28 | 1.38 | 1.35 | 1.61 | 1.70 | 1.67 | 1.12 | 1.49 | 0.96 | 0.93 | 1.22 | 1.10 | 0.97 | 1.04 | 22/25 |
| EAGLE-3 (RedHatAI) | 1.49 | 1.35 | 1.31 | 2.32 | 1.36 | 1.84 | 1.07 | 1.58 | 1.14 | 1.01 | 1.16 | 1.52 | 1.05 | 1.18 | 17/25 |
| Qwen3-8B | |||||||||||||||
| EAGLE-3 (RedHatAI) | 1.20 | 1.27 | 1.05 | 1.31 | 1.35 | 1.20 | 1.01 | 1.18 | 0.94 | 0.76 | 1.04 | 0.99 | 0.96 | 0.94 | 18/25 |
| P-EAGLE (RedHatAI) | 1.88 | 1.12 | 1.03 | 1.10 | 1.07 | 1.35 | 1.00 | 1.11 | 0.95 | 0.93 | 0.97 | 1.15 | 0.96 | 0.99 | 24/25 |
| DSpark (DeepSeek) | 2.35 | 1.23 | 1.03 | 1.03 | 1.23 | 1.11 | 1.00 | 1.08 | 0.85 | 0.83 | 0.87 | 0.90 | 0.87 | 0.86 | 18/25 |
| Qwen3-0.6B | 2.92 | 1.24 | 0.92 | 0.90 | 0.92 | 0.97 | 0.94 | 0.93 | 0.75 | 0.67 | 0.78 | 0.67 | 0.87 | 0.75 | 6/25 |
| Qwen3-1.7B | 3.60 | 1.25 | 0.89 | 0.89 | 0.91 | 0.89 | 0.91 | 0.90 | 0.69 | 0.69 | 0.69 | 0.69 | 0.78 | 0.71 | 4/25 |
| Llama-3.1-8B-Instruct | |||||||||||||||
| EAGLE-3 (yuhuili) | 1.67 | 1.57 | 1.94 | 8.85 | 2.70 | 2.15 | 0.84 | 3.30 | 1.51 | 3.96 | 1.01 | 3.00 | 0.70 | 2.03 | 18/25 |
| EAGLE-1 (yuhuili) | 1.29 | 1.42 | 1.95 | 6.23 | 3.35 | 2.58 | 0.78 | 2.98 | 1.60 | 3.16 | 1.57 | 2.60 | 0.70 | 1.93 | 19/25 |
| Llama-3.2-1B | 3.57 | 1.32 | 0.94 | 1.59 | 0.83 | 1.30 | 0.85 | 1.10 | 0.70 | 1.08 | 0.66 | 0.90 | 0.74 | 0.82 | 16/25 |
| R1-Distill-Llama-8B | |||||||||||||||
| EAGLE-3 (yuhuili) | 1.42 | 1.41 | 1.04 | 2.11 | 1.19 | 1.19 | 0.98 | 1.30 | 0.92 | 0.80 | 0.84 | 1.05 | 0.93 | 0.91 | 16/25 |
| Llama-3.2-1B | 1.84 | 1.48 | 1.07 | 2.14 | 1.99 | 1.24 | 1.04 | 1.50 | 0.80 | 0.95 | 1.34 | 1.10 | 0.83 | 1.01 | 20/25 |
| All twelve pairs | — | 1.34 | 1.21 | 2.51 | 1.55 | 1.46 | 0.96 | 1.54 | 0.98 | 1.31 | 1.01 | 1.31 | 0.86 | 1.10 | 198/300 |
Table 3. Twelve target-drafter pairs at each rule’s most relaxed tested setting: for each rule, the mean of and of over the pair’s datasets; all: over all the pair’s cells. Bold (underline): the 95% interval lies above (below) 1 on every dataset. : mean lossless accepted length; gain: mean ; acc. loss: graded cells two or more points below lossless. Every cell: Table D1 and D2. ment., c-: mentored, casc-.
Why completions get longer
For all twelve pairs the run logs record whether a run hit its budget; for the primary pairs they also record the split between thinking trace and answer and, for 15 traced GPT-OSS-20B cases per dataset and arm, every drafted token’s target probability and both rules’ verdicts. Four measurements follow, at the loosest settings.
The extra text is thinking, not answer. On the primary pairs, GPT-OSS-20B’s analysis channel receives at least 92% of the extra characters in 25 of its 30 cells. Qwen3-8B puts more than 90% of the extra characters inside its <think> blocks at the median, with an exception: cactus on AIME24 inflates the answer (9.4×) because 11 of its 30 outputs reopen a closed think block. Llama-3.1-8B-Instruct has no thinking trace, so its inflation, the largest measured, lengthens the answer itself.
A broad shift with a heavy tail. On the primary pairs the per-case ratio has a median above 1 in 47 of the 60 cells and a lower quartile below 1 in 34. Its 90th percentile has median 2.2 over the cells, and the tenth of cases with the largest increase holds 59% of the net extra tokens at the median. This is not exact looping: on GPT-OSS-20B at most 2.9% of a cell’s tokens lie inside a repeated 50-gram (lossless: at most 1.9%). Qwen3-8B does loop under spec_casc_opt: 15% of its AIME24 tokens and 13% of its LongBench-v2 tokens (lossless 0.4% and 1.3%), with a mean longest repeat of 2 728 tokens on AIME24.
Cut-offs understate inflation. Relaxation raises the share of runs that end at the generation budget in 208 of the 360 cells (median 11% against 4% under lossless, where Qwen3-8B already reaches 25% on GSM8K; per-cell shares in the released tables). On the primary pairs, keeping only the cases that finished under both rules moves by at most 0.1 in 45 of the 60 cells, so the inflation is not an artifact of the cap.
Relaxation admits tokens the target rates as unlikely. We split the committed draft tokens of the traced runs into those both rules accept and those only the relaxed rule accepts (Table B2). The relaxed-only tokens are 7% to 24% of the committed draft tokens. For the four rules other than spec_casc_tok they have a median target probability of 0.01 to 0.09 (shared tokens: 0.78 to 0.93) and a mean target rank of 37 to 234. spec_casc_tok admits the fewest extra tokens (7%), and they are the target’s first or second choice (mean rank 0.9, median probability 0.29). Lossless verification accepts any token with , so a relaxed rule can only add tokens with : the overshoot tokens that Wang et al. [31] tie to quality loss. In our traces they also carry the extra length. A lossless control agrees: raising the sampling temperature from 1.0 to 1.2, which also commits lower-probability tokens, lengthens GPT-OSS-20B completions by 1.27× on GSM8K and 1.07× on LiveCodeBench, and inflates GSM8K by 3.9× with accuracy 0.31 (Appendix § 8). The logs show two things: relaxation commits tokens the target would rarely sample, and the thinking trace then gets longer.
| Method | committed draft tokens only | target prob. (median) only | target prob. (median) both | rank only | entropy (mean) only | entropy (mean) both |
| mentored_dec | 16% | 0.09 | 0.93 | 37 | 1.8 | 0.8 |
| cactus | 24% | 0.01 | 0.78 | 234 | 2.1 | 1.1 |
| spec_casc_opt | 16% | 0.07 | 0.92 | 88 | 1.8 | 0.8 |
| r_fuzzy | 21% | 0.01 | 0.83 | 134 | 1.8 | 1.0 |
| spec_casc_tok | 7% | 0.29 | 0.99 | 0.9 | 1.5 | 0.5 |
Table B2. What relaxation admits (GPT-OSS-20B, traced runs, loosest settings): committed draft tokens that only the relaxed rule accepts (“only”) against those lossless verification would also accept (“both”). Share of committed draft tokens, median target probability, mean target rank (0 is the argmax), mean target entropy (nats).
| Dataset | Qwen3-8B forced | correct | + | − | GPT-OSS-20B forced | correct | + | − |
| GSM8K | 307 | 56→225 | 171 | 2 | 32 | 4→10 | 6 | 0 |
| AIME24 | 60 | 2→6 | 4 | 0 | 36 | 3→7 | 7 | 3 |
| HumanEval | 205 | 16→52 | 37 | 1 | 9 | 2→0 | 0 | 2 |
| LiveCodeBench | 226 | 11→81 | 72 | 2 | 62 | 7→16 | 13 | 4 |
| LongBench-v2 | 157 | 17→33 | 18 | 2 | 38 | 8→9 | 7 | 6 |
| Total | 955 | 102→397 | 302 | 7 | 177 | 24→42 | 33 | 15 |
caption: Forcing relaxed runs to close their reasoning trace at a threshold near the generation budget, summed over the five rules at their loosest tested settings. forced: runs still reasoning at the threshold; correct: how many of them are correct without → with forcing; +: forced runs that go from wrong to correct; −: from correct to wrong. Per rule and thresholds: Table F1.
| Dataset, rule | Qwen3-8B forced | correct | + | − | cut off | GPT-OSS-20B forced | correct | + | − | cut off |
| GSM8K | ||||||||||
| mentored_dec | 44 | 6→38 | 32 | 0 | 4 | 6 | 2→2 | 0 | 0 | 3 |
| cactus | 56 | 15→45 | 30 | 0 | 11 | 9 | 1→2 | 1 | 0 | 6 |
| spec_casc_opt | 102 | 18→60 | 44 | 2 | 27 | 7 | 1→2 | 1 | 0 | 4 |
| r_fuzzy | 62 | 8→47 | 39 | 0 | 13 | 8 | 0→3 | 3 | 0 | 3 |
| spec_casc_tok | 43 | 9→35 | 26 | 0 | 6 | 2 | 0→1 | 1 | 0 | 1 |
| AIME24 | ||||||||||
| mentored_dec | 6 | 0→1 | 1 | 0 | 3 | 6 | 0→0 | 0 | 0 | 3 |
| cactus | 17 | 1→1 | 0 | 0 | 14 | 2 | 0→0 | 0 | 0 | 0 |
| spec_casc_opt | 19 | 0→1 | 1 | 0 | 17 | 13 | 0→4 | 4 | 0 | 9 |
| r_fuzzy | 13 | 1→2 | 1 | 0 | 9 | 9 | 1→1 | 1 | 1 | 5 |
| spec_casc_tok | 5 | 0→1 | 1 | 0 | 2 | 6 | 2→2 | 2 | 2 | 3 |
| HumanEval | ||||||||||
| mentored_dec | 22 | 2→12 | 10 | 0 | 5 | 1 | 0→0 | 0 | 0 | 0 |
| cactus | 32 | 1→10 | 9 | 0 | 16 | 3 | 0→0 | 0 | 0 | 2 |
| spec_casc_opt | 67 | 7→18 | 11 | 0 | 39 | 4 | 1→0 | 0 | 1 | 0 |
| r_fuzzy | 65 | 4→3 | 0 | 1 | 38 | 1 | 1→0 | 0 | 1 | 0 |
| spec_casc_tok | 19 | 2→9 | 7 | 0 | 8 | no forced runs | ||||
| LiveCodeBench | ||||||||||
| mentored_dec | 33 | 2→25 | 23 | 0 | 10 | 10 | 1→3 | 3 | 1 | 2 |
| cactus | 51 | 3→18 | 16 | 1 | 33 | 1 | 0→0 | 0 | 0 | 0 |
| spec_casc_opt | 58 | 3→14 | 12 | 1 | 30 | 21 | 1→6 | 5 | 0 | 10 |
| r_fuzzy | 55 | 0→0 | 0 | 0 | 42 | 20 | 0→4 | 4 | 0 | 10 |
| spec_casc_tok | 29 | 3→24 | 21 | 0 | 5 | 10 | 5→3 | 1 | 3 | 1 |
| LongBench-v2 | ||||||||||
| mentored_dec | 10 | 2→0 | 0 | 2 | 2 | 4 | 3→1 | 0 | 2 | 1 |
| cactus | 89 | 8→19 | 11 | 0 | 75 | 3 | 0→0 | 0 | 0 | 0 |
| spec_casc_opt | 31 | 3→5 | 2 | 0 | 24 | 22 | 5→4 | 3 | 4 | 7 |
| r_fuzzy | 19 | 3→7 | 4 | 0 | 6 | 8 | 0→3 | 3 | 0 | 3 |
| spec_casc_tok | 8 | 1→2 | 1 | 0 | 3 | 1 | 0→1 | 1 | 0 | 0 |
| Total | 955 | 102→397 | 302 | 7 | 442 | 177 | 24→42 | 33 | 15 | 73 |
Table F1. Table 4 by dataset and rule, each rule at its loosest tested setting. Columns as in Table 4, plus cut off: forced runs that still end at the budget. Thresholds in tokens (Qwen3-8B / GPT-OSS-20B): GSM8K 1 684 / 2 002; AIME24 31 260 / 31 605; HumanEval 8 150 / 8 497; LiveCodeBench 11 004 / 9 294; LongBench-v2 6 894 / 8 133.
:::
They do not show why an unlikely token makes a reasoning model think more rather than less.
What the Extra Length Buys
Reasoning models often answer better when they think longer ([24]), so the length that relaxation adds could be useful thinking. Per-token guards that force lossless verification do not shorten completions beyond their tuning cases (Appendix § 11), so we act on whole runs.
Closing the reasoning trace recovers answers. We rerun every relaxed run that passes a token threshold near the generation budget without closing its reasoning trace, and at the threshold force the token that closes the trace, the budget forcing of Muennighoff et al. [24]; runs that close their trace earlier are unchanged (Appendix § 11). Over 1 132 such runs (five rules, both primary pairs, five graded datasets; Table 4), correct answers rise from 126 to 439: 335 runs that failed now answer correctly and 22 that succeeded now fail, 302 and 7 of them on Qwen3-8B.
| Method | score [95% CI] | no ans. | answered | |
| GPT-OSS-20B | ||||
| lossless | — | 7.29 [6.75, 7.80] | 0 | 7.29 |
| mentored_dec | 0.75 | 6.41 [5.73, 7.09] | 1 | 6.48 |
| cactus | 0.35 | 4.51 [3.70, 5.34] | 6 | 4.80 |
| spec_casc_opt | 0.05 | 5.74 [5.04, 6.45] | 2 | 5.86 |
| r_fuzzy | 0.25 | 4.58 [3.94, 5.23] | 3 | 4.71 |
| spec_casc_tok | 0.8 | 7.49 [6.96, 7.99] | 0 | 7.49 |
| Qwen3-8B | ||||
| lossless | — | 6.94 [6.31, 7.55] | 10 | 7.79 |
| mentored_dec | 0.75 | 6.40 [5.70, 7.06] | 13 | 7.45 |
| cactus | 0.35 | 2.99 [2.40, 3.64] | 22 | 3.74 |
| spec_casc_opt | 0.05 | 3.95 [3.43, 4.49] | 22 | 5.07 |
| r_fuzzy | 0.25 | 3.05 [2.61, 3.51] | 20 | 3.73 |
| spec_casc_tok | 0.8 | 7.21 [6.59, 7.80] | 9 | 8.00 |
Table 4. MT-Bench quality of the primary pairs at the loosest settings, judged by Claude (claude-fable-5-1) with FastChat’s single-answer grading [36] of the first turn (the GPT-4 reference answer is supplied for math, reasoning, and coding questions). score: mean over the 80 questions, 1 to 10, with a bootstrap interval; no ans.: runs that never produced an answer (scored 1 without a judge call); answered: mean over the other runs.
The extra tokens follow the answer on grade-school math and precede it on competition math. For cases both arms answer correctly, we split the extra tokens at the first point where the output writes the reference answer (Table F2). On GSM8K, 50% (GPT-OSS-20B) and 82% (Qwen3-8B) of the extra tokens come after it, and most runs cut off at the budget already contain the answer. On AIME24, relaxed runs reach the answer later (median token 3 865 against 1 721 for GPT-OSS-20B), at least 59% of the extra tokens precede it, and few cut-off runs contain it.
| Target | Dataset | Pairs | First answer (token) lossless | First answer (token) relaxed | Length (tokens) lossless | Length (tokens) relaxed | Extra tokens after answer (%) | Cut-off runs with the answer relaxed | Cut-off runs with the answer lossless |
| GPT-OSS-20B | GSM8K | 486 | 102 | 126 | 183 | 264 | 50 [37, 59] | 15/26 | 0/2 |
| Qwen3-8B | GSM8K | 304 | 325 | 350 | 877 | 989 | 82 [65, 96] | 147/201 | 28/33 |
| GPT-OSS-20B | AIME24 | 36 | 1 721 | 3 865 | 3 822 | 7 020 | 41 [28, 54] | 8/33 | 1/2 |
| Qwen3-8B | AIME24 | 33 | 4 221 | 6 031 | 11 160 | 14 230 | −16 [−177, 49] | 9/53 | 1/5 |
Table F2. Where the extra tokens sit (§ 4): the four rules other than spec_casc_tok, each at its loosest setting. Pairs: (case, rule) pairs in which both arms answer correctly and neither is cut off; first answer and length: medians over pairs; extra tokens after answer: share of the summed extra length that falls after the point where each run first writes the answer, with a 95% bootstrap interval over cases (negative: the part after the answer shrinks while the part before it grows); cut-off runs with the answer: runs that end at the budget without a correct answer, and how many of them contain it.
The extra length is waste. On problems within the model’s reach, relaxation makes the model continue past an answer it already holds, as lossless reasoning models already do ([7]); on hard problems, it lengthens the route to the answer. In neither case do the extra tokens buy accuracy.
Conclusion
We evaluated five training-free relaxed verification rules on six benchmarks and twelve target-drafter pairs, measuring completion length, verifier rounds, time, and accuracy rather than acceptance alone. Relaxation does raise the number of draft tokens accepted per round, but the model spends that gain on longer outputs: across the 360 cells the mean length inflation (1.54×) exceeds the mean acceptance gain (1.34×). Where a rule does decode faster, accuracy drops; the one rule that keeps accuracy close to lossless, spec_casc_tok, is also the one that relaxes least.
This paper identifies length as what decides the speed of relaxed verification: once our measured length is supplied, the length-corrected model of Xia et al. [33] predicts our verifier rounds within about 2%, and the clock departs from it only through the cost of a round, which the model does not contain. Two conclusions follow. Acceptance gain is not a measure of speed for a relaxed rule, because relaxation changes what the model generates and not only how fast it is verified; completion length and verifier rounds have to be reported next to it. And the promising direction is not accepting more of the drafter’s tokens but choosing which ones to admit, since the only rule that worked trusted the target more.
Section 4 also yields a simple mitigation: closing the reasoning trace before the generation budget runs out recovers the answers that inflation cuts off, raising correct answers from 126 to 439 over the 1,132 runs it touches, under all five rules. It does not recover the speed, since these runs still spend most of their budget; closing the trace earlier, where the answer is already written, is the natural next step.
Limitations
Matching and settings. No is reachable by every method, so the matching of § 3.2 is best-effort and the tables compare methods at different , each at its most relaxed setting. Each method is tested at four published or reference values; other values could change its profile, but the conclusions also hold at the gentlest and the best accuracy-preserving settings (§ 3.3; Tables D4 and B3).
Budget censoring. Relaxed runs hit the per-dataset token budget more often, so lengths are right-censored and the inflation is, if anything, understated; part of the accuracy loss is unfinished runs. The GSM8K budget (2 048 tokens) is tight for Qwen3-8B: 25% of its lossless runs hit it (Table B1).
Sampling and timing. Each case runs once per arm (seeds repeated on the primary pairs only, § 3.3), and paired bootstrap intervals over cases are wide on AIME24 (30 cases) and MT-Bench (80), so we interpret only effects that recur across datasets or targets. Times are client-side at batch size 1 and analyzed on the primary pairs only; their ratio to lossless depends on the server’s cost per committed token (§ 3.4) and is noisy because each benchmark’s first cases ran on a fresh server process and the rest on a reused one, while round counts depend on neither. MT-Bench quality is one judge’s first-turn score, not an accuracy.
Scope. We use one serving stack (vLLM 0.26.0) and, for the main results, one sampler (temperature 1.0, top-p 1.0); on Qwen3-8B the gentle rules’ inflation depends on the sampler (§ 3.3). Every relaxed arm drafts tokens; draft length is swept only under lossless verification (Appendix § 8). The rules are our reimplementations in vLLM’s rejection sampler, checked at the strict limit but not against the authors’ code; the ensemble rule, speculative contrastive decoding (Table A1) and typical acceptance (Cai et al. [4]) are not swept. Lossless decoding at a higher temperature also inflates completions (§ 3.7), so the inflation is not specific to relaxed acceptance, and we do not explain why an unlikely token makes a reasoning model think longer.
Method Details
Algorithm 1 gives one round of speculative decoding with the three relaxed target distributions in place, and Table A1 states each training-free rule as a choice of them. The paragraphs below give our implementation of the five swept rules; Table A2 collects what the source papers measure.
CACTUS. Hao and Mou [13] choose, for the drafted token , the relaxed target that maximizes the acceptance probability under a reverse-KL budget ; a Taylor expansion of the constraint gives the closed form of Table A1, which boosts as a function of and shrinks every other token proportionally. The paper does not specify ; we use , as the released code does. A draft token outside the target’s top-/top- support is rejected regardless of .
Mentored decoding. Tran-Thien [30] maximizes the expected acceptance probability subject to a forward-KL budget . We implement the simplified form used by Narasimhan et al. [25] and Xia et al. [33]: every token receives the same multiplicative boost, capped at one, and the residual after a rejection is the unrelaxed .
Reducible fuzzy speculative decoding. Holsman et al. [14] trust the drafter outright when the Jensen–Shannon divergence between and , computed over the full vocabulary, is below . In their main variant (FSD) a failed test samples from directly; we implement the reducible variant rFSD (their Sec. 4.3), which falls back to the standard acceptance coin and so reduces to strict verification at . Xia et al. [33] prove that rFSD samples from the same per-token distribution as FSD with weakly higher acceptance.
Speculative cascades. For spec_casc_opt we implement the plug-in deferral rule of Narasimhan et al. [25], which trusts the drafter when ; at it still trusts the drafter wherever , so the pilot includes negative . For spec_casc_tok we implement the token-specific rule (their Sec. 4.4) in the form of Xia et al. [33]: draft tokens in the trusted set are accepted with probability one, and the others are judged against , which is stricter than lossless verification. Since the argmax of is always trusted, this rule is not strict even at .
What the source papers measure
Table A2 summarizes the results behind each method. The reported quantities are acceptance or accepted length, tokens per second or relative latency, and task quality, on short-form benchmarks: maximum output lengths of a few hundred tokens where they are stated at all, and timing budgets that fix the number of generated tokens in one case. Only Xia et al. [33] treat response length explicitly, as the correction term of Equation (3) and as a finding that relaxation with weak drafters can lengthen responses enough to reduce speedup. CACTUS flags the same effect as a caveat and adds a length-sensitive proxy, but does not measure length.
Full Results
Table B1 gives the lossless reference of the primary pairs, and Table B5 their cells at the most relaxed setting with 95% paired bootstrap intervals. Results at every tested setting, with plots of completion length, rounds, and accuracy against , are in the released tables.
Gentler settings. Restricting each method to its gentlest tested setting instead of its loosest gives with median 1.05 and mean 1.08 over the 60 cells of the primary pairs, a median round ratio of 0.98, and 21 cells that need more rounds than lossless; 27 of the 50 graded cells still lose accuracy (mean drop 0.03), against 40 (0.17) at the loosest. Table D4 (Appendix § 9) compares the two settings on all seven pairs run with the matched protocol.
Best accuracy-preserving setting. Table B3 gives, per method and primary pair, how many graded datasets have a tested setting within two accuracy points of lossless and the lowest round ratio among those settings (§ 3.6); for mentored_dec and spec_casc_tok every dataset Algorithm 1 One round of speculative decoding with relaxed target distributions. Lossless (strict) speculative decoding is the special case ; a lossy (relaxed) method is a choice of the three (Table A1).
Require: drafter , target , context , draft length , relaxation parameter , relaxed target distributions
▷ DRAFT: sample tokens autoregressively from
for to do
end for
▷ VERIFY: score all positions with one forward pass of
for
▷ Construct the relaxed targets (strict decoding: )
for
▷ Accept or reject each draft token; is the first rejection
for ;
if then
▷ Rejection: resample the rejected position from the residual
else
▷ All accepted: bonus token (an extra drafter step gives if needs it)
end if
return (between 1 and tokens)
has all four values, the 41 settings missing from the sweep having been run on the second and third servers against a lossless reference on the same server. For these two rules we reran the setting with the lowest time ratio among those within two points (MT-Bench: with a round ratio below 1) with a new seed on the second server (some Qwen3-8B arms on the third), paired with lossless at the same seed on the same server: it again has a time ratio below 1 and accuracy within two points in 15 of the 24 cases (released tables).
MT-Bench. MT-Bench (Table B4; Appendix § 9) separates the rules the same way: across the twelve pairs spec_casc_tok scores to points against lossless and mentored_dec to , while the three aggressive rules lose a median 2.3 points and up to 4.0. It is also where relaxation saves the most rounds (median 0.84; elsewhere 0.90 to 0.97), so the largest saving comes where the aggressive rules
| Method | Models and tasks | Speed metric | Length | Rounds | Long-form |
| CACTUS (Hao and Mou, [13]) | Qwen3, Gemma, R1-Distill, Llama pairs; GSM8K, IFEval, GPQA | AL, rejected-token count, vLLM wall-clock | (✓) | (✓) | (✓) |
| Mentored decoding (Tran-Thien, [30]) | T5-large/small; WMT15 En–Fr | tokens per second | × | × | × |
| Fuzzy (Holsman et al., [14]) | Llama-3.1, Gemma-2, Qwen2.5 pairs; GSM8K, CSQA, MMLU, HumanEval | tokens per second, accepted length | × | × | × |
| Speculative cascades (Narasimhan et al., [25]) | T5, Gemma pairs; translation, summarization, GSM8K, MBPP, QA | batch-one relative latency, rejection rate | × | × | × |
| Ensemble (Wang et al., [32]) | Llama-3.1, Qwen3, Gemma-3 pairs; GSM8K, CNN/DM, XSum, MBPP | acceptance rate, wall-clock at fixed token count | × | × | × |
| Spec. contrastive (Yuan et al., [35]) | Llama-2 7B/70B; WikiText, HumanEval, AlpacaEval, GSM8K | expected acceleration from acceptance rate | × | × | × |
| Xia et al. [33] | Qwen3-32B with 0.6B/8B drafters, Qwen3.5 MTP; AIME24, GPQA, LCB | at several , modeled , Eq. (3) | ✓ | × | ✓ |
Table A2. Results reported by the source papers (Appendix A.1). Length: completion length reported as a measured quantity; Rounds: verifier rounds or target forward passes reported; Long-form: evaluation on long reasoning traces. (✓) marks indirect or partial coverage: CACTUS reports rejected-token counts, which grow with length and with rounds ending in rejection, on reasoning-mode models, and discusses length only as a caveat and one qualitative case.
Seeds, Draft Length, Temperature, and Hardware
Hardware and compute. Seed 0 ran on an H100 PCIe server (the first server), seeds 1 and 2 on an H100 SXM server (the second); the Qwen3-8B additions of Appendices B and E (except the first 40 SPEED-Bench prompts per arm, run on the second), the Qwen3-8B controls and AIME24 seeds of this appendix, and the pairs of Appendix D ran on a third server, also H100 SXM. On the first server, the first cases of each benchmark ran right after a fresh server start and the rest on a process reused across requests. On the 28 GPT-OSS-20B cells with three seeds, and the round ratio replicate
lose judged quality. across the first two servers (mean 1.49, 1.44, 1.48 and mean round ratio 1.04, 1.01, 1.03 for seeds 0, 1, 2); the time ratio does not (1.17, 1.01, 1.03): it is lower on the second server in 23 and 24 of the 28 cells. Fitting per-run time per round as (tokens committed per round) locates the difference. On the first server is 12 to 14 ms and is 2.5 to 3.2 ms per token on HumanEval, MT-Bench, LiveCodeBench, and AIME24 ( 0.39 to 0.87; GSM8K rounds are too few and too short to fit, and LongBench-v2 is dominated by the prefill), so a relaxed arm’s time per round is 1.12× lossless on average and above 1.05× in 26 of the 30 GPT-OSS-20B cells. On the second server is 7.1 to 7.6 ms and at most 0.08 ms per token (), and the time-per-round ratio is 1.00: at seeds 1 and 2, 14 and 12 cells save rounds, the same cells save time, and the round and time ratios are never more than 0.06 apart. The third server’s nodes are not uniform (one node ran rounds about 11% slower than its neighbours on one day and not on the next), so every Qwen3-8B time ratio from that server is reported over the case pairs whose arm and lossless reference ran on the same node, or marked when fewer than ten pairs did; round ratios are unaffected. The experiments used about 750 H100 GPU-hours: 441 logged on the second and third servers and about 300 on the first.
| Method | GPT-OSS-20B cells | GPT-OSS-20B best | Qwen3-8B cells | Qwen3-8B best |
| mentored_dec | 5/5 | 0.86 | 5/5 | 0.88 |
| cactus | 1/5 | 0.98 | 2/5 | 0.97 |
| spec_casc_opt | 2/5 | 0.93 | 2/5 | 0.97 |
| r_fuzzy | 2/5 | 1.13 | 0/5 | — |
| spec_casc_tok | 5/5 | 0.83 | 5/5 | 0.93 |
Table 3 (PDF p. 14). The best accuracy-preserving setting. For each graded (dataset, method) cell we take, among the method’s tested values, the one with the fewest verifier rounds whose accuracy is within two points of lossless. “cells” counts the graded datasets (of five) on which any tested setting qualifies; “best ” is the lowest round ratio among them. The median best round ratio over the 29 qualifying cells is 0.96.
Seeds. To test whether the paired bootstrap over cases captures run-to-run variation, we repeated the lossless reference on GPT-OSS-20B with seeds 1 and 2 on five datasets: on every dataset the three runs’ mean completion lengths lie inside the 95% bootstrap interval of the seed-0 mean, and moves by at most 0.10. We then repeated 28 of the 30 GPT-OSS-20B loosest-setting cells and 22 of the 30 Qwen3-8B cells, with their lossless reference, with seeds 1 and 2 on the second server (Qwen3-8B AIME24 on the third; per-seed values in the released tables). On GPT-OSS-20B, exceeds 1 in all three seeds in 24 cells; the four exceptions are all spec_casc_tok, whose crosses 1 between seeds. The seed-to-seed standard deviation has median 0.08 for (maximum 0.27, on AIME24 with 30 cases), 0.06 for the round ratio, and 0.09 for the time ratio. The seed-0 interval of contains both new seeds in 21 of the 28 cells; five of the seven exceptions miss it by at most 0.04 and the other two are on AIME24. In 20 cells the round ratio lies on the same side of 1 in all three seeds; the eight that cross it are three AIME24 cells and five cells within 0.13 of 1 in every seed. On Qwen3-8B the seeds agree more closely: the standard deviation has median 0.03 for (maximum 0.10), 0.03 for the round ratio, and 0.04 for the time ratio; exceeds 1 in all three seeds in 16 of the 22 cells, the six exceptions being cells whose seed-0 is at most 1.12; and the round ratio lies on the same side of 1 in all three seeds in 21 cells (the exception, HumanEval spec_casc_tok, stays within 0.04 of 1). The GPT-OSS-20B LongBench-v2 cells of cactus and spec_casc_tok and the Qwen3-8B LongBench-v2 cells were run once; the Qwen3-8B AIME24 cells of mentored_dec, cactus, and spec_casc_tok have five seeds instead (below).
AIME24 over five seeds.
Because AIME24 has 30 cases, we repeated its GPT-OSS-20B arms with seeds 1 to 4 on the second server. Accuracy over the five seeds, with a two-level bootstrap interval over problems and seeds: lossless 0.79 [0.65, 0.91]; mentored_dec 0.68 [0.53, 0.82]; spec_casc_tok 0.69 [0.54, 0.82]; cactus 0.51 [0.36, 0.67]; r_fuzzy 0.48 [0.33, 0.63]; spec_casc_opt 0.39 [0.25, 0.53]. The seed-to-seed standard deviation is 0.04 to 0.06, one to two problems. Every rule scores below lossless on every seed: the three aggressive rules by 27 to 39 points, mentored_dec and spec_casc_tok by 11 and 10 points with overlapping intervals. On 30 problems varies widely across seeds (spec_casc_tok 0.96 to 1.60, mentored_dec 1.28 to 2.15), so the single-seed AIME24 values of Table 1 carry that spread. On Qwen3-8B (seeds 1 to 4 on the third server) the same five seeds give lossless 0.72 [0.57, 0.85]; mentored_dec 0.72 [0.56, 0.87]; spec_casc_tok 0.73 [0.58, 0.86]; cactus 0.21 [0.09, 0.35]; spec_casc_opt 0.32 [0.19, 0.45]; r_fuzzy 0.35 [0.21, 0.50]: the two gentle rules match lossless on average and the three aggressive rules lose 37 to 51 points.
Draft length. We ran lossless decoding at on GSM8K and LiveCodeBench, paired with on the same server and cases (GPT-OSS-20B on the second server, Qwen3-8B on the third; released tables). On GPT-OSS-20B shorter drafts save 5 to 9% of the time (every interval touches 1), longer drafts cost 6 to 24%, completion length is unchanged ( 0.96 to 1.06), and accuracy moves by at most five points, within the binomial error of the case counts. On Qwen3-8B, whose head accepts fewer tokens per round ( 1.50 and 1.13 at ), shorter drafts change the time by to and longer ones cost 8 to 16%, with 0.99 to 1.03. The reference of the body is therefore within 9% and 10% of the fastest lossless draft length on the two primary pairs, and no lossless draft length changes the length distribution.
Temperature. We ran lossless decoding on GPT-OSS-20B at sampling temperature 1.2 and 1.5 on the second server, paired with on the same server and cases (released tables). At , GSM8K completions are 1.27 longer ([1.10, 1.47]), rounds 1.37 and time 1.35, with accuracy unchanged (0.96 against 0.97); on LiveCodeBench ([0.99, 1.17]) and time . At , GSM8K inflates by with 48% of runs at the budget and accuracy 0.31, and LiveCodeBench by with accuracy 0.08. Acceptance falls with temperature ( 2.58, 2.36, 1.79 on GSM8K), so a hotter lossless sampler pays in length and in rounds per token at once. A change of +0.2 in temperature inflates GSM8K about as much as mentored_dec at its loosest setting does at (), without any relaxation. Qwen3-8B behaves differently: its completions barely lengthen at or 1.5 ( 1.02 to 1.04, accuracy within three points on GSM8K and down nine on LiveCodeBench at ), but its acceptance falls faster ( 1.50, 1.28, 0.82 on GSM8K), so rounds and time still rise by 12 to 49%.
Qwen3-8B at its recommended sampler. The body runs every target at temperature 1.0 and top- 1.0. Qwen3-8B’s bundled sampling defaults are overridden so that every target uses the same sampler, and its context window is extended to 65 536 tokens with YaRN (Peng et al., [28]) (factor 1.6) to match GPT-OSS-20B. Table C1 repeats the five rules at their loosest settings on Qwen3-8B with its recommended thinking-mode sampler (temperature 0.6, top- 0.95, top- 20) on GSM8K and LiveCodeBench, on the third server, paired with lossless at the same sampler. Lossless acceptance rises ( 1.68 and 1.36 against 1.50 and 1.13 at temperature 1.0 on the same server) and the picture changes for the gentle rules: mentored_dec, cactus, and spec_casc_tok no longer lengthen completions ( 1.01 to 1.04), save 4 to 7% of the rounds, and keep accuracy. r_fuzzy keeps the length but loses 15 and 57 accuracy points, and spec_casc_opt inflates by 1.52× and 1.48× with 87% and 99% of its runs at the budget and accuracy 0.20 and 0.00; its round ratio of 0.75 on LiveCodeBench is an acceptance gain of 2.0× against an inflation that the budget cuts short, not a usable setting.
| Dataset | Method | [95% CI] | [95% CI] | Acc. | cut off | |
| GSM8K | mentored_dec | 1.85 | 1.01 [0.97, 1.04] | 0.94 [0.91, 0.98] | 0.81 | 23% |
| cactus | 1.81 | 1.01 [0.97, 1.05] | 0.96 [0.92, 1.00] | 0.81 | 23% | |
| spec_casc_opt | 3.34 | 1.52 [1.43, 1.62] | 0.95 [0.89, 1.02] | 0.20 | 87% | |
| r_fuzzy | 2.05 | 1.01 [0.92, 1.09] | 0.89 [0.81, 0.96] | 0.64 | 39% | |
| spec_casc_tok | 1.80 | 1.01 [0.97, 1.04] | 0.96 [0.92, 1.00] | 0.78 | 26% | |
| LiveCodeBench | mentored_dec | 1.57 | 1.04 [1.00, 1.09] | 0.94 [0.90, 0.99] | 0.68 | 34% |
| cactus | 1.53 | 1.01 [0.98, 1.04] | 0.93 [0.90, 0.97] | 0.72 | 29% | |
| spec_casc_opt | 3.73 | 1.48 [1.36, 1.63] | 0.75 [0.68, 0.83] | 0.00 | 99% | |
| r_fuzzy | 1.70 | 1.08 [0.98, 1.19] | 0.97 [0.87, 1.07] | 0.10 | 52% | |
| spec_casc_tok | 1.51 | 1.02 [0.98, 1.06] | 0.95 [0.91, 0.99] | 0.68 | 32% |
Table C1. Qwen3-8B at temperature 0.6, top-p 0.95, top-k 20 (seed 0, third server), loosest settings, against lossless at the same sampler and cases: lossless 1.68 (GSM8K) and 1.36 (LiveCodeBench), accuracy 0.79 and 0.67, cut off 25% and 33%.
Other Targets and Drafters
Table 3 summarizes the twelve pairs. Table D1 and Table D2 list their 360 cells at each rule’s most relaxed tested setting, Table D3 gives the lossless references of the ten pairs beyond the primary ones, and Table D4 compares the gentlest and the most relaxed tested setting on the seven pairs run with the matched protocol. The released tables give every tested setting of every cell with , the cut-off share, the wall-clock ratio, and 95% intervals. The targets are openai/gpt-oss-20b, Qwen/Qwen3-8B, meta-llama/Llama-3.1-8B-Instruct, and deepseek-ai/DeepSeek-R1-Distill-Llama-8B. The drafters beyond the primary ones are RedHatAI/gpt-oss-20b-speculator.eagle3 for GPT-OSS-20B; RedHatAI/Qwen3-8B-speculator.peagle, deepseek-ai/dspark_qwen3_8b_block7, Qwen/Qwen3-0.6B, and Qwen/Qwen3-1.7B for Qwen3-8B; yuhuili/EAGLE3-LLaMA3.1-Instruct-8B, yuhuili/LLaMA3.1-Instruct-8B, and meta-llama/Llama-3.2-1B-Instruct for Llama-3.1-8B-Instruct; and yuhuili/EAGLE3-DeepSeek-R1-Distill-LLaMA-8B and meta-llama/Llama-3.2-1B-Instruct for DeepSeek-R1-Distill-Llama-8B. The standalone drafters run through vLLM’s draft-model path. The dedicated drafters other than P-EAGLE ran the matched protocol of § 3.2; P-EAGLE and the standalone drafters ran the loosest setting of each rule. These ten pairs ran on the third server, one persistent vLLM process per arm, each arm paired case by case with its own pair’s lossless run. The Llama-3.1-8B-Instruct and Llama-3.2-1B-Instruct weights come from public mirrors byte-identical to Meta’s files, and the Llama models’ bundled sampler defaults are overridden as for Qwen3-8B. The published EAGLE heads for the Llama models cap their position embeddings at 2 048; we extend the configuration to 65 536 positions, and do the same on LongBench-v2 for DSpark, P-EAGLE, and the standalone Qwen3 drafters, whose configurations stop at 40 960 (identical below it). The head for R1-Distill-Llama-8B stops drafting usefully beyond about 2 048 positions of a trace (lossless 2.2 to 3.4 within the first 2 048 positions, 0.2 to 1.2 beyond), which is why its lossless on LiveCodeBench, AIME24, and LongBench-v2 is below 1; the Llama-3.2-1B drafter shares R1-Distill’s vocabulary except for seven special tokens. On LongBench-v2, whose prompts run 10 000 to 45 000 tokens, the second GPT-OSS-20B head and Llama-3.1-8B-Instruct’s EAGLE-3 head also nearly stop drafting (lossless 0.05 and 0.06; R1-Distill’s 0.03); the rules either change little or, cactus above all, produce runaway answers, and we report these cells as measured. Relaxation raises in 347 of the 360 cells and lowers it in 13, 12 of them r_fuzzy and 8 on the Llama EAGLE heads (gain 0.80 to 0.995). HumanEval is graded by executing the last code block of the answer, as for the primary pairs; Llama-3.1-8B-Instruct often ends its answer with a usage example, so its HumanEval accuracies are lower bounds (executing instead the last block that defines the function raises lossless accuracy by at most 4 points and a relaxed arm’s by at most 10, and changes no conclusion). A Medusa head for Llama-3.1-8B-Instruct serves in vLLM but exposes no draft distribution to the verifier, so four of the five rules reduce to lossless verification on it, and it is not reported.
| Dataset | mentored | cactus | casc-opt | r-fuzzy | casc-tok | mentored | cactus | casc-opt | r-fuzzy | casc-tok |
| GSM8K | 1.16 (.93) | 1.61 (.91) | 1.46 (.94) | 1.70 (.87) | 0.96° (.97) | 1.03° (.77) | 1.04° (.77) | 1.33 (.49) | 0.95° (.55) | 0.99° (.79) |
| AIME24 | 1.69 (.63) | 1.78 (.60) | 2.48 (.37) | 2.13 (.50) | 1.48 (.70) | 1.01° (.73) | 1.26 (.23) | 1.42 (.30) | 1.29 (.40) | 0.98° (.70) |
| HumanEval | 1.30 (.96) | 1.79 (.87) | 1.73 (.82) | 1.64 (.63) | 1.05° (.94) | 1.08 (.85) | 1.14 (.75) | 1.72 (.49) | 1.65 (.16) | 1.08 (.85) |
| LiveCodeBench | 1.25 (.84) | 1.54 (.61) | 1.64 (.49) | 1.65 (.34) | 1.16 (.88) | 1.05 (.63) | 1.15 (.44) | 1.27 (.33) | 1.07° (.02) | 1.00° (.71) |
| MT-Bench | 1.10° (6.41) | 1.18 (4.51) | 1.09° (5.74) | 1.20 (4.58) | 0.96° (7.49) | 1.04° (6.40) | 1.16 (2.99) | 1.22 (3.95) | 1.12 (3.05) | 1.04° (7.21) |
| LongBench-v2 | 1.59 (.51) | 1.77 (.36) | 1.79 (.47) | 1.69 (.40) | 1.13° (.53) | 1.06° (.51) | 2.10 (.29) | 1.16 (.37) | 1.08° (.44) | 1.00° (.49) |
| GSM8K | 1.31 (.95) | 2.49 (.83) | 1.35 (.93) | 2.40 (.78) | 0.87 (.97) | 1.02° (.79) | 1.04 (.77) | 1.02° (.77) | 1.28 (.51) | 1.00° (.73) |
| AIME24 | 1.24° (.80) | 1.35° (.20) | 1.44 (.63) | 1.68 (.60) | 1.28 (.77) | 1.04° (.70) | 1.20 (.60) | 1.08° (.67) | 1.50 (.37) | 0.98° (.73) |
| HumanEval | 1.48 (.95) | 3.03 (.63) | 1.62 (.84) | 3.11 (.23) | 1.07° (.97) | 1.03° (.80) | 1.06 (.78) | 1.14 (.71) | 1.67 (.25) | 1.00° (.81) |
| LiveCodeBench | 1.43 (.81) | 2.55 (.16) | 1.46 (.73) | 1.28 (.39) | 1.09° (.91) | 1.01° (.67) | 1.10 (.56) | 1.04° (.51) | 1.31 (.03) | 1.04° (.64) |
| MT-Bench | 1.36 (6.36) | 1.56 (3.52) | 1.31 (5.40) | 1.58 (3.29) | 1.11° (7.56) | 1.05° (6.33) | 1.14 (4.14) | 1.02° (5.38) | 1.22 (3.35) | 1.02° (6.88) |
| LongBench-v2 | 1.07° (.57) | 2.92 (.24) | 1.01° (.49) | 0.98° (.57) | 1.02° (.51) | 1.04° (.51) | 1.07° (.47) | 1.10 (.48) | 1.14 (.45) | 0.99° (.47) |
| GSM8K | 0.94 (.81) | 0.93 (.79) | 1.10 (.64) | 1.04° (.69) | 0.91 (.83) | 0.94 (.83) | 0.93 (.83) | 0.98° (.79) | 1.01° (.76) | 0.95 (.83) |
| AIME24 | 1.13 (.63) | 1.09° (.70) | 1.52 (.37) | 1.20 (.60) | 1.10° (.67) | 0.98° (.70) | 0.91° (.77) | 0.95° (.70) | 1.06° (.50) | 0.97° (.67) |
| HumanEval | 1.10 (.80) | 1.11 (.82) | 1.43 (.61) | 1.24 (.64) | 1.05° (.81) | 0.92 (.87) | 0.94° (.87) | 0.95° (.83) | 1.03° (.85) | 0.95° (.87) |
| LiveCodeBench | 1.02° (.66) | 1.02° (.68) | 1.04 (.62) | 1.07 (.41) | 1.01° (.68) | 0.92 (.72) | 0.95 (.76) | 0.95 (.70) | 1.02° (.61) | 0.94 (.79) |
| MT-Bench | 1.01° (6.67) | 1.06° (5.81) | 1.16 (4.47) | 1.06° (5.35) | 0.96° (6.92) | 0.94 (6.95) | 0.91 (6.33) | 0.87 (6.74) | 0.89 (6.10) | 0.97° (7.17) |
| LongBench-v2 | 0.95° (.50) | 0.97° (.47) | 1.10 (.42) | 1.06° (.39) | 0.94° (.50) | 0.83 (.44) | 0.76 (.43) | 0.80 (.45) | 0.81 (.39) | 0.85 (.51) |
| GSM8K | 0.93 (.83) | 0.93 (.83) | 0.96 (.82) | 0.92 (.85) | 0.94 (.81) | 1.59 (.69) | 5.48 (.16) | 1.58 (.48) | 1.21° (.69) | 0.83° (.83) |
| AIME24 | 0.83 (.80) | 0.85 (.70) | 0.92° (.73) | 0.79 (.77) | 0.89° (.80) | 2.86 (.03) | 4.10 (.00) | 0.81° (.00) | 1.06° (.03) | 0.40 (.00) |
| HumanEval | 0.91 (.84) | 0.89 (.83) | 0.92 (.85) | 0.94° (.82) | 0.92 (.85) | 1.28 (.45) | 13.23 (.05) | 1.90 (.35) | 2.77 (.27) | 1.02° (.53) |
| LiveCodeBench | 0.91 (.73) | 0.91 (.77) | 0.92 (.80) | 0.92 (.76) | 0.93 (.71) | 1.03° (.14) | 12.77 (.00) | 2.58 (.02) | 1.68 (.10) | 0.87° (.17) |
| MT-Bench | 0.91 (7.01) | 0.92 (6.80) | 0.93 (6.66) | 0.90 (7.33) | 0.93 (7.09) | 1.71 (4.53) | 4.29 (2.00) | 1.08° (3.09) | 3.73 (2.77) | 0.84° (6.42) |
| LongBench-v2 | 0.83 (.51) | 0.81 (.44) | 0.80 (.33) | 0.85 (.44) | 0.86 (.42) | 3.18 (.37) | 13.22 (.25) | 8.26 (.27) | 2.44 (.32) | 1.06° (.37) |
| GSM8K | 1.63 (.72) | 5.84 (.06) | 2.87 (.35) | 3.59 (.26) | 0.86° (.79) | 0.86° (.75) | 1.44 (.60) | 0.81 (.72) | 1.17° (.61) | 0.83 (.86) |
| AIME24 | 2.08 (.43) | 4.41 (.00) | 2.70 (.00) | 1.10 (.00) | 0.46° (.04) | 0.35 (.03) | 0.56 (.00) | 0.30 (.07) | 0.55 (.00) | 1.07° (.03) |
| HumanEval | 1.03° (.53) | 9.37 (.01) | 4.88 (.08) | 1.39 (.49) | 1.03° (.54) | 1.05° (.40) | 1.15 (.38) | 1.07 (.41) | 1.20 (.23) | 0.94 (.57) |
| LiveCodeBench | 3.43 (.10) | 4.05 (.09) | 3.52 (.02) | 3.86 (.01) | 0.80 (.11) | 1.01° (.07) | 1.49 (.01) | 0.91° (.13) | 1.37 (.22) | 0.87° (.22) |
| MT-Bench | 2.13 (4.40) | 5.60 (1.96) | 1.88 (2.29) | 4.35 (2.11) | 0.83 (6.21) | 1.35° (5.34) | 2.45 (3.86) | 1.07° (5.14) | 1.60 (4.13) | 0.92° (6.54) |
| LongBench-v2 | 1.41° (.35) | 8.13 (.22) | 4.24 (.21) | 1.19° (.36) | 0.73° (.36) | 1.06° (.33) | 2.45 (.24) | 0.85° (.29) | 1.93 (.21) | 0.59 (.36) |
| GSM8K | 1.11 (.69) | 1.46 (.55) | 1.11 (.68) | 1.51 (.47) | 1.01° (.75) | 1.02° (.73) | 2.21 (.44) | 3.17 (.51) | 2.03 (.49) | 0.97° (.77) |
| AIME24 | 0.87 (.33) | 2.60 (.00) | 1.32 (.13) | 0.84 (.37) | 0.96° (.27) | 1.00° (.20) | 2.43 (.00) | 1.45 (.17) | 0.66 (.03) | 1.37 (.37) |
| HumanEval | 1.08 (.84) | 1.97 (.16) | 1.28 (.59) | 1.19 (.51) | 1.02° (.81) | 1.26 (.76) | 1.88 (.26) | 1.40 (.64) | 1.51 (.45) | 1.00° (.77) |
| LiveCodeBench | 1.07 (.43) | 1.33 (.02) | 1.03° (.38) | 1.12 (.17) | 0.95 (.59) | 1.18 (.32) | 1.28 (.04) | 1.09 (.30) | 0.88 (.14) | 1.04° (.42) |
| MT-Bench | 1.09 (5.09) | 1.31 (2.82) | 1.14 (4.83) | 1.51 (2.99) | 0.94° (5.39) | 1.02° (4.95) | 2.42 (1.90) | 1.58 (3.50) | 1.52 (3.03) | 0.92° (5.62) |
| LongBench-v2 | 1.03° (.35) | 4.00 (.29) | 1.23 (.33) | 0.99° (.29) | 1.01° (.32) | 0.95° (.31) | 2.64 (.25) | 3.28 (.34) | 0.83 (.33) | 0.94° (.33) |
Table D1. Length inflation at each method’s most relaxed tested setting, for all twelve pairs (first block: the primary pairs of Table 1): completion length relative to lossless ( is inflation), with the accuracy at that setting in parentheses (MT-Bench: judge score out of 10). °: the 95% interval for includes 1 (primary pairs: Table B5; other pairs: released tables).
| Dataset | GPT-OSS-20B + EAGLE-3 (nebius) ment. | cactus | c-opt | r-fuzzy | c-tok | Qwen3-8B + EAGLE-3 (RH) ment. | cactus | c-opt | r-fuzzy | c-tok | GPT-OSS-20B + EAGLE-3 (RH) ment. | cactus | c-opt | r-fuzzy | c-tok |
| GSM8K | 0.83 | 0.99 | 1.07 | 1.14 | 0.83 | 0.93 | 0.80 | 1.05 | 0.76 | 0.95 | 0.98 | 1.38 | 1.08 | 1.59 | 0.83 |
| AIME24 | 1.18 | 1.07 | 1.70 | 1.37 | 1.22 | 0.94 | 0.55 | 1.06 | 1.16 | 0.93 | 1.23 | 0.46 | 1.18 | 1.67 | 1.33 |
| HumanEval | 0.97 | 1.12 | 1.29 | 1.16 | 0.93 | 0.99 | 0.96 | 1.36 | 1.33 | 1.04 | 1.30 | 1.80 | 1.43 | 2.70 | 1.02 |
| LiveCodeBench | 0.89 | 0.94 | 1.16 | 1.11 | 0.98 | 0.93 | 0.72 | 1.03 | 0.87 | 0.94 | 1.34 | 1.08 | 1.38 | 1.28 | 1.06 |
| MT-Bench | 0.75 | 0.62 | 0.76 | 0.76 | 0.88 | 0.89 | 0.64 | 0.94 | 0.85 | 0.96 | 0.95 | 0.74 | 0.99 | 0.93 | 1.03 |
| LongBench-v2 | 1.11 | 0.87 | 1.31 | 1.10 | 0.97 | 0.98 | 0.80 | 0.83 | 0.96 | 0.96 | 1.03 | 0.58 | 0.92 | 0.97 | 1.02 |
| Dataset | Qwen3-8B + P-EAGLE ment. | cactus | c-opt | r-fuzzy | c-tok | Qwen3-8B + DSpark ment. | cactus | c-opt | r-fuzzy | c-tok | Qwen3-8B + Qwen3-0.6B ment. | cactus | c-opt | r-fuzzy | c-tok |
| GSM8K | 0.95 | 0.94 | 0.96 | 1.08 | 0.98 | 0.80 | 0.78 | 0.83 | 0.86 | 0.82 | 0.80 | 0.75 | 0.87 | 0.72 | 0.90 |
| AIME24 | 0.95 | 1.04 | 0.98 | 1.30 | 0.91 | 0.90 | 0.84 | 0.99 | 0.96 | 0.93 | 0.78 | 0.69 | 0.80 | 0.77 | 0.88 |
| HumanEval | 0.97 | 0.98 | 1.08 | 1.42 | 0.97 | 0.93 | 0.93 | 1.09 | 1.01 | 0.95 | 0.79 | 0.76 | 0.86 | 0.71 | 0.90 |
| LiveCodeBench | 0.92 | 0.94 | 0.95 | 1.10 | 0.98 | 0.86 | 0.88 | 0.89 | 0.93 | 0.87 | 0.73 | 0.69 | 0.83 | 0.68 | 0.87 |
| MT-Bench | 0.94 | 0.84 | 0.89 | 1.01 | 0.97 | 0.79 | 0.79 | 0.77 | 0.82 | 0.84 | 0.73 | 0.60 | 0.70 | 0.58 | 0.88 |
| LongBench-v2 | 0.95 | 0.82 | 0.95 | 0.98 | 0.95 | 0.82 | 0.74 | 0.67 | 0.83 | 0.81 | 0.65 | 0.54 | 0.65 | 0.56 | 0.77 |
| Dataset | Qwen3-8B + Qwen3-1.7B ment. | cactus | c-opt | r-fuzzy | c-tok | Llama-3.1-8B-I. + EAGLE-3 ment. | cactus | c-opt | r-fuzzy | c-tok | Llama-3.1-8B-I. + EAGLE-1 ment. | cactus | c-opt | r-fuzzy | c-tok |
| GSM8K | 0.76 | 0.76 | 0.76 | 0.75 | 0.83 | 1.25 | 2.67 | 0.82 | 1.32 | 0.63 | 1.39 | 3.08 | 1.35 | 3.18 | 0.73 |
| AIME24 | 0.66 | 0.67 | 0.74 | 0.61 | 0.77 | 1.70 | 1.08 | 0.21 | 1.08 | 0.30 | 1.45 | 1.64 | 0.80 | 0.76 | 0.41 |
| HumanEval | 0.75 | 0.73 | 0.74 | 0.75 | 0.82 | 1.37 | 8.47 | 1.34 | 5.16 | 0.89 | 1.00 | 4.98 | 3.33 | 1.59 | 0.94 |
| LiveCodeBench | 0.70 | 0.69 | 0.72 | 0.69 | 0.80 | 0.94 | 6.21 | 1.55 | 2.24 | 0.65 | 2.59 | 1.80 | 1.68 | 4.81 | 0.69 |
| MT-Bench | 0.67 | 0.67 | 0.67 | 0.66 | 0.77 | 1.59 | 2.12 | 0.61 | 6.01 | 0.67 | 1.79 | 3.22 | 0.93 | 4.01 | 0.78 |
| LongBench-v2 | 0.62 | 0.61 | 0.54 | 0.65 | 0.70 | 2.19 | 3.22 | 1.53 | 2.18 | 1.04 | 1.37 | 4.26 | 1.35 | 1.25 | 0.62 |
| Dataset | Llama-3.1-8B-I. + Llama-3.2-1B ment. | cactus | c-opt | r-fuzzy | c-tok | R1-Distill-Llama-8B + EAGLE-3 ment. | cactus | c-opt | r-fuzzy | c-tok | R1-Distill-Llama-8B + Llama-3.2-1B ment. | cactus | c-opt | r-fuzzy | c-tok |
| GSM8K | 0.68 | 1.06 | 0.68 | 0.86 | 0.74 | 0.87 | 1.00 | 0.89 | 1.11 | 0.90 | 0.87 | 1.37 | 2.30 | 1.99 | 0.89 |
| AIME24 | 0.26 | 0.40 | 0.23 | 0.39 | 0.80 | 0.80 | 0.75 | 0.72 | 0.79 | 0.94 | 0.76 | 0.98 | 0.99 | 0.69 | 1.00 |
| HumanEval | 0.83 | 0.87 | 0.90 | 0.90 | 0.84 | 1.01 | 0.95 | 1.01 | 1.22 | 0.97 | 0.93 | 0.80 | 0.97 | 1.26 | 0.81 |
| LiveCodeBench | 0.74 | 1.01 | 0.71 | 0.93 | 0.69 | 0.96 | 0.44 | 0.72 | 1.00 | 0.92 | 0.84 | 0.50 | 0.78 | 0.74 | 0.78 |
| MT-Bench | 0.94 | 1.55 | 0.81 | 1.05 | 0.83 | 0.85 | 0.67 | 0.89 | 1.18 | 0.82 | 0.75 | 0.96 | 0.99 | 1.19 | 0.77 |
| LongBench-v2 | 0.77 | 1.58 | 0.65 | 1.26 | 0.56 | 1.01 | 0.99 | 0.83 | 0.99 | 1.01 | 0.68 | 1.11 | 2.05 | 0.73 | 0.72 |
Table D2. Verifier rounds relative to lossless () at each method’s most relaxed tested setting, for all twelve pairs (first two: the primary pairs of Table 2; RH: RedHatAI; Llama-3.1-8B-I.: Llama-3.1-8B-Instruct; ment., c-: mentored, casc-). Bold: the 95% interval for lies entirely above 1 (more rounds than lossless); underline: entirely below 1; intervals are in the released tables.
SPEED-Bench
SPEED-Bench (Abramovich et al., [1]) has 880 prompts in eleven categories, built to measure speculative decoding across request types, and no accuracy grading; it is the benchmark closest to a serving workload, and Xia et al. [33] do not use it. We run the five rules at their loosest settings and lossless verification on both primary pairs (seed 0, budget 8 192 tokens) on the 672 prompts not drawn from the gated HLE set: eight categories are complete (80 prompts each), and Humanities, Math, and STEM keep 8, 18, and 6 prompts. Lossless completions are short here, and the acceptance gain of every rule but one exceeds its inflation, so rounds fall (Table E1); the exception is spec_casc_opt on Qwen3-8B, which inflates by 1.40× against a gain of 1.32× and needs 1.02× the rounds. Equation (5) predicts each of the ten round ratios within 7% and its direction in all ten. Per category (released tables), the round ratio’s interval lies below 1 in 31 of the 55 (rule, category) cells on GPT-OSS-20B and in 24 on Qwen3-8B, and above 1 in 3 and 1, all four with . On GPT-OSS-20B the rounds verdict and the time verdict agree in every cell; on Qwen3-8B the time ratios carry the third server’s node effect (Appendix § 8) and are reported over same-node pairs where possible. Quality is not graded on this benchmark, so the rules that save the most rounds here, cactus above all, are the ones that lose the most accuracy and judged quality in § 3.5.
| Rule | gain | gain | ||||||
| mentored_dec | 1.15 | 1.44 | 0.76 | 0.75 | 1.07 | 1.14 | 0.93 | 0.91† |
| cactus | 1.26 | 1.89 | 0.63 | 0.62 | 1.35 | 1.91 | 0.67 | 0.68‡ |
| spec_casc_opt | 1.23 | 1.41 | 0.82 | 0.82 | 1.40 | 1.32 | 1.02 | 1.06† |
| r_fuzzy | 1.20 | 1.52 | 0.77 | 0.75 | 1.20 | 1.28 | 0.95 | 0.96 |
| spec_casc_tok | 1.09 | 1.13 | 0.95 | 0.92 | 1.01 | 1.05 | 0.95 | 0.95† |
Table E1. SPEED-Bench over all 672 prompts, loosest settings, seed 0 (GPT-OSS-20B on the second server; Qwen3-8B’s first 40 prompts per arm on the second, the rest on the third). Lossless: 1 266 and 2 332 tokens on average, 1% and 5% cut off. , , : ratios of completion length, verifier rounds, and per-request time to lossless; gain: . Bold (underline): 95% paired interval above (below) 1. †: over the prompt pairs that share a node; ‡: fewer than ten do, all-pair ratio.
Forcing the Reasoning Trace to End
Guard search. A coding agent edited one function that returns, for each verification round, the draft positions to verify losslessly; at a marked position spec_casc_tok reduces its to limit, which is lossless verification. After each edit the agent replayed eight AIME24 cases on GPT-OSS-20B (, one fresh server per case) and kept the edit if mean completion length fell without losing more than one correct answer. Over eleven iterations, the best guard (lossless verification for eight positions after clusters of hesitation and discourse markers, and for whole rounds after 14 000 tokens without a final channel) cut the eight cases from 13 364 to 10 534 tokens at 6 of 8 correct; on all 30 cases it moved the mean from 9 270 to 10 363 tokens and accuracy from 22 to 23. On the eight cases, two runs that reached the 32 768-token budget without closing their reasoning trace held 61% of all output tokens. A guard judges one draft token at a time and cannot see whether a run will ever close its trace, so § 4 acts on the run instead.
Forcing. For Qwen3-8B, a hook in vLLM’s sampler writes </think> at the first draft position of a round once the request has generated the threshold number of tokens without closing its trace; the hook does not depend on the acceptance rule, so one hook serves all five rules. For GPT-OSS-20B, a patch to each rule injects the tokens that open the final channel. The threshold is the budget minus the 95th percentile of the answer length after the closing token in the pair’s lossless runs (characters converted at 3.25 per token); it was chosen after a first pass at 0.9 of the budget left too little room for the answer, so it was not fixed in advance. The forced runs are the relaxed runs that reached the budget or passed the threshold without closing their trace; the hook is inactive below the threshold, so every other run is unchanged. The baselines of spec_casc_tok and mentored_dec were rerun in one session; the other three rules use the runs of the sweep and its pilot (see § 3.2). A run cut off before closing its reasoning trace is scored wrong. Qwen3-8B reruns reproduce earlier runs (127 of 127 LongBench-v2 runs of spec_casc_tok below the threshold keep their exact token count); GPT-OSS-20B reruns on a server reused across requests do not (59 of 146), so its rescued and lost counts include trajectory noise, and the fresh-server checks below are its clean measurement.
Forcing results. GPT-OSS-20B, at its recommended sampler, gains much less than Qwen3-8B (33 rescued and 15 lost of 177 runs). The gentle rules recover most: on Qwen3-8B, 66% and 68% of forced mentored_dec and spec_casc_tok runs end correct, against 28% to 38% for the three aggressive rules (Table 4 and Table F1). Forcing does not always end the run: 515 forced runs still reach the budget (23% on GSM8K, 50% to 68% on the other datasets), as the model resumes reasoning or rewrites its solution after the forced token. Two paired checks on fresh servers, where reruns are reproducible byte for byte, show that the intervention is surgical: on GPT-OSS-20B AIME24, at thresholds of 28 000 and 30 000 tokens, it leaves 27 and 23 of 30 outputs unchanged, removes every run that reached the budget (3 and 7), and keeps accuracy at 22 of 30 for spec_casc_tok () while raising it from 16 to 19 for mentored_dec, with no correct answer lost.
Answer timing
For each output we find the first occurrence of the reference answer as a standalone number (thousands separators allowed) and convert its character offset to tokens in proportion to the output’s token count. Cases whose answer appears in the prompt are excluded (12 of 150 on GSM8K, none on AIME24). A pair is a case and a rule for which both arms answer correctly and neither is cut off; spec_casc_tok is left out because its completions do not grow. Intervals are 95% bootstrap intervals over cases (5 000 resamples; a resampled case keeps all its rules). The first occurrence is a proxy: an intermediate result can equal the answer. In one Qwen3-8B case under spec_casc_opt, both arms write the answer near token 300; lossless verification answers within 1 292 tokens, while the relaxed run writes the answer 23 more times and is cut off at 2 048 tokens with no answer.
References
- [1]Talor Abramovich, Maor Ashkenazi, Izzy Putterman, Benjamin Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein, and Yonatan Geifman. 2026. SPEED-Bench: A unified and diverse benchmark for speculative decoding. In International Conference on Machine Learning (ICML). ArXiv:2604.09557.arxiv.org/abs/2604.09557
- [2]Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas Kohler. 2025. Judge decoding: Faster speculative sampling requires going beyond model alignment. In International Conference on Learning Representations (ICLR). ArXiv:2501.19309.arxiv.org/abs/2501.19309
- [3]Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazhen Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). ArXiv:2412.15204.arxiv.org/abs/2412.15204
- [4]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning (ICML). ArXiv:2401.10774.arxiv.org/abs/2401.10774
- [5]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318.arxiv.org/abs/2302.01318
- [6]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.arxiv.org/abs/2107.03374
- [7]Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025. Do NOT think that much for 2+3=? On the overthinking of long reasoning models. In International Conference on Machine Learning (ICML). ArXiv:2412.21187.
- [8]Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, et al. 2026. DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv preprint arXiv:2607.05147.arxiv.org/abs/2607.05147
- [9]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.arxiv.org/abs/2110.14168
- [10]DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948.DOI
- [11]Roman Garipov, Fedor Velikониvtsev, Ivan Ermakov, Ruslan Svirschevski, Vage Egiazarian, and Max Ryabinin. 2025. AutoJudge: Judge decoding without manual annotation. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:2504.20039.arxiv.org/abs/2504.20039
- [12]Aaron Grattafori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.
- [13]Yongchang Hao and Lili Mou. 2026. Cactus: Accelerating auto-regressive decoding with constrained acceptance speculative sampling. In International Conference on Learning Representations (ICLR). ArXiv:2604.04987.arxiv.org/abs/2604.04987
- [14]Maximilian Holsman, Yukun Huang, and Bhuwan Dhingra. 2025. Fuzzy speculative decoding for a tunable accuracy-runtime tradeoff. In Findings of the Association for Computational Linguistics (ACL). ArXiv:2502.20704v4.arxiv.org/abs/2502.20704
- [15]Mude Hui, Xin Huang, Jaime Campos Salas, Yue Sun, Nathan Pemberton, Xiang Song, Ashish Khetan, and George Karypis. 2026. P-EAGLE: Parallel-drafting EAGLE with scalable training. arXiv preprint arXiv:2602.01469.arxiv.org/abs/2602.01469
- [16]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations (ICLR). ArXiv:2403.07974.arxiv.org/abs/2403.07974
- [17]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP).arxiv.org/abs/2309.06180
- [18]Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning (ICML). ArXiv:2211.17192.arxiv.org/abs/2211.17192
- [19]Ying Li, Zhaode Wang, Zhiwen Chen, Chengfei Lv, and Huan Wang. 2026. ARC-Decode: Accelerated decoding with risk-bounded acceptance. In International Conference on Machine Learning (ICML).
- [21]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024b. EAGLE: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning (ICML). ArXiv:2401.15077.arxiv.org/abs/2401.15077
- [22]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840.arxiv.org/abs/2503.01840
- [23]Mathematical Association of America. 2024. American invitational mathematics examination (AIME) 2024.
- [24]Niklas Muennighoff, Zitong Yang, Weijia Shi, Xi-ang Lisa Li, Fei-Fei Li, Hannah Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393.
- [25]Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. 2025. Faster cascades via speculative decoding. In International Conference on Learning Representations (ICLR). ArXiv:2405.19261.arxiv.org/abs/2405.19261
- [26]NVIDIA. 2023. TensorRT-LLM. https://github.com/NVIDIA/TensorRT-LLM.
- [27]OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925.
- [28]Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient context window extension of large language models. In International Conference on Learning Representations (ICLR). ArXiv:2309.00071.arxiv.org/abs/2309.00071
- [29]Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:1811.03115.arxiv.org/abs/1811.03115
- [30]Vivien Tran-Thien. 2023. An optimal lossy variant of speculative decoding. Blog post, https://huggingface.co/blog/vivien/optimal-lossy-variant-of-speculative-decoding.
- [31]Tianyu Wang, Yuxuan Zhou, Wenbin Wang, Heng Li, Zikai Xiao, and Junyuan Shang. 2026a. Revisiting lossy verification in speculative decoding: Mechanisms, trade-offs, and failure modes. arXiv preprint arXiv:2607.26627.arxiv.org/abs/2607.26627
- [32]Ziyi Wang, Siva Rajesh Kasa, Ankith M S, Santhosh Kumar Kasa, Jiaru Zou, Sumit Negi, Ruqi Zhang, Nan Jiang, and Qifan Song. 2026b. DIVERSED: Relaxed speculative decoding via dynamic ensemble verification. In International Conference on Artificial Intelligence and Statistics (AISTATS). ArXiv:2604.07622.arxiv.org/abs/2604.07622
- [33]Guoxuan Xia, Luka Ribar, and Paul Balanca. 2026. A practical investigation of training-free relaxed speculative decoding. arXiv preprint arXiv:2607.08690.arxiv.org/abs/2607.08690
- [34]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenguang Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.arxiv.org/abs/2505.09388
- [35]Hongyi Yuan, Keming Lu, Fei Huang, Zheng Yuan, and Chang Zhou. 2024. Speculative contrastive decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Short Papers). ArXiv:2311.08981.arxiv.org/abs/2311.08981
- [36]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. ArXiv:2306.05685.arxiv.org/abs/2306.05685
- [37]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyong Liu, Ying Sheng, Ruilin Wang, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:2312.07104.arxiv.org/abs/2312.07104