WorksComputer Science
Adaptive Multi-Token Prediction: How far should we go?
Abstract
Multi-token prediction (MTP) can accelerate language-model decoding by drafting several tokens and verifying them in a single target-model pass. However, the best speculation horizon changes with context: longer drafts offer greater amortization when reliable, but can waste compute on suffixes discarded after a rejection. Unlike autoregressive drafting, where a stopping rule can decide token by token, MTP must fix the horizon before drafting, as all heads fire in the same forward pass. We therefore formulate horizon selection as an online control problem, using verification feedback to learn how far the draft can be trusted in each context, and introduce Adaptive Multi-Token Prediction (AMTP): a lightweight controller that requires no retraining and leaves the exact verifier unchanged. To guide horizon selection, we characterize the expected number of tokens committed at each horizon via acceptance survival, the probability that a draft prefix is fully accepted. Before each round, AMTP predicts how far the draft is likely to survive and picks the horizon with the highest expected tokens per second, learning freely from every verification pass. Theoretically, we show that any online learner's prediction error translates into throughput regret against the best horizon for each context, and in turn into a bound on wall-clock completion time. With online gradient descent, this regret is sublinear. Across five draft-target pairings built on Llama-3.1-8B-Instruct with Medusa heads, Meta's MTP models, and Qwen3-4B-Inst-2507-MTP, evaluated on AIME 2025 and GSM8K, AMTP decodes up to 4.1x faster than standard decoding and matches or outperforms the fixed window in 8 of 10 settings, with throughput gains of up to 10%.
Introduction
Autoregressive decoding generates each new token through an evaluation of a target language model. Speculative decoding reduces this sequential bottleneck by using a cheaper draft mechanism to propose several tokens and evaluating them in parallel with the target model [33, 20, 9]. With an appropriate acceptance-and-correction rule, this procedure preserves the target distribution while committing multiple tokens in a single round [20, 9, 34]. Multi-token prediction (MTP) provides a natural proposal mechanism by predicting several future tokens from a shared representation [14, 7, 5], and have witnessed enormous success in practice: for instance, MTP modules are now shipped in frontier models [12], and pretrained next-token models can be converted into MTP drafters without architectural changes [32, 18, 39]. These methods create an inference-time decision that is distinct from learning a good draft model: how many tokens should be proposed before the next verification step? We refer to this quantity as the speculation horizon.
The speculation horizon directly affects decoding efficiency. A longer horizon amortizes verification over more committed tokens when the draft is reliable, but after an early rejection the discarded suffix adds latency and no output. Horizon selection therefore balances the expected gain from additional accepted tokens against the risk of speculating too far. We evaluate each horizon by its conditional throughput: the ratio of its expected number of committed tokens to its expected round latency. Even for a fixed draft-target pair, the throughput-maximizing horizon varies with context, because acceptance probabilities at each draft position depend on context. For MTP, this choice has a structure that isn’t exploited in current methods [4]: an autoregressive drafter can decide token by token whether to keep drafting, but MTP heads produce all draft tokens in a single forward pass, so the horizon must be fixed before drafting, from information available at that point.
Existing adaptive methods do not make this decision without additional training. Most stop an autoregressive drafter token by token, using a trained classifier or acceptance head [30, 17], entropy-based bounds [1, 38], or parallel drafting and verification [28]. Others shape draft trees from confidence after drafting [22, 23, 8], prune verification prefixes with predicted survival probabilities [10], or train reinforcement-learning policies for draft depth [35]. Concurrent work trains a router that selects the span length before decoding, but requires new prediction heads and verifies against greedy decoding only [11]. In fact, to our knowledge, AMTP is the first training-free method that selects the MTP horizon before drafting at every round. A complementary line of work uses verification feedback online to adapt the draft model [29, 31] or to select among drafters and hyperparameter configurations with regret guarantees against the best fixed choice [27, 16]. However, in this work, we compare against the stronger benchmark of the best horizon at each visited context. We study how learning acceptance behavior from verification feedback affects horizon-selection performance, and, for a given target model and draft mechanism, we develop an online controller and bound its conditional throughput regret relative to the best horizon at each visited context in terms of cumulative prediction error. § 2 discusses related work in detail.
To relate prediction error to horizon-selection, we first identify the acceptance statistics that influence the expected number of committed tokens for each horizon. A draft token can be committed as an accepted proposal only if all prior proposals are also accepted. We characterize the joint acceptance event via acceptance survival, the probability that a draft prefix is fully accepted, which further determines the expected number of committed tokens exactly. We express them in terms of conditional rejection hazards, where the hazard at a position is its rejection probability given acceptance of all prior proposals. Our analyses holds for general causal proposals, including independent and correlated prediction heads [7, 5]. Estimating the hazards requires distinguishing between proposed and surviving prefixes: verification provides draft-target discrepancies along sampled proposal prefixes, whereas the hazards average these discrepancies under the distribution of prefixes that survive verification. So, we derive a survival-weighted regression objective that targets these conditional hazards, and its excess population loss is the survival-weighted hazard-prediction error that enters our throughput-regret bound, mirroring decision-focused learning, where prediction losses are judged by the induced decisions (Elmachtoub and Grigas, 2022).
We use this objective to develop Adaptive Multi-Token Prediction (AMTP), a training-free horizon controller equipped with an online adaptive learning algorithm. Before drafting, AMTP combines hazard-based forecasts of committed tokens with estimates of round latency to evaluate the throughput of each horizon. Short horizons, however, conceal verification feedback at later positions. To learn under this partial feedback, AMTP combines greedy throughput-based selection with randomized full-horizon exploration [19] and uses inverse-probability weighting to obtain unbiased estimates of the survival-weighted regression loss.
Our contributions are as follows:
Acceptance survival. We characterize horizon selection through accepted-prefix survival and round cost for general causal proposals, showing how survival determines the expected number of committed tokens for each horizon (§ 3).
AMTP. We introduce a training-free online controller that chooses the MTP horizon before each round, by maximizing predicted tokens per second, learning from partial feedback of short horizons via randomized exploration and inverse-probability weighting (§ 4).
Prediction error to wall-clock time. We show that the prediction error of any online learner translates into throughput regret against the best horizon for each context, and in turn into a bound on expected completion time. With online gradient descent Zinkevich, [40] under partial feedback, the regret is , which is sublinear (§ 5).
Experiments. Across five draft–target pairings and two reasoning benchmarks, AMTP decodes up to faster than standard decoding and matches or outperforms the fixed window in 8 of 10 settings, with throughput gains of up to (§ 6).
Related Work
Speculative decoding and multi-token proposals. Speculative decoding drafts tokens with a cheap proposal and verifies them in parallel with the target model, using an acceptance-and-correction rule that leaves the target distribution unchanged [20, 9]. Most work uses a separate next-token draft model, or, as in EAGLE, a lightweight head that drafts autoregressively at the feature level [21]. Multi-token prediction (MTP) instead trains several future-token heads on a shared representation [14]; Medusa adds such heads to an existing model and verifies candidate continuations with tree attention [7], and recent work obtains MTP models from pretrained next-token models by self-distillation [18]. Joint multi-token prediction studies how to learn representations that support predicting several tokens at once [2]. We keep the exact verifier and study a different question: given a fixed proposal, how many tokens should we draft in each round?
Adaptive draft length for autoregressive drafters. A substantial line of work adapts the draft length of a next-token drafter. DISCO trains a classifier that decides after each drafted token whether to continue [30]; SpecDec++ adds an acceptance-prediction head to the draft model and stops once the predicted probability of a rejection crosses a threshold [17]; AdaEDL stops when an entropy-based lower bound on the acceptance probability falls below a threshold [1]; and SVIP derives a similar entropy-based stopping policy for long-form generation [38]. EAGLE-2 instead grows a draft tree whose shape follows context-dependent confidence [22], and Learning to Draft trains reinforcement-learning policies for draft depth and verification size with throughput rewards [35]. These methods share a structural assumption: drafting is sequential, so the decision to stop can be made token by token, after each drafted token is observed and before the next one is paid for. However, MTP removes this option. All heads fire in the same forward pass, so the cost of drafting is incurred up front and the horizon must be fixed before any draft token is sampled. Truncating a complete MTP draft afterwards saves verification cost but not drafting cost, and choosing the cut point after inspecting sampled tokens can break exactness (Appendix 3.1). AMTP is designed for this setting: it chooses the horizon from pre-draft information only, and our statistical and online-optimization analysis of horizon selection and acceptance survival applies to any causal proposal, including both independent and correlated heads.
Online learning for speculative decoding. Verification feedback has also been used to adapt speculative decoding online. OnlineSpec updates an evolving draft model and relates its dynamic regret to accepted length at a fixed candidate length [31]. Not-a-Bandit gives no-regret guarantees for selecting among draft models [27]. BanditSpec treats speculative-decoding hyperparameters, including the speculation length, as bandit arms and bounds regret against the best fixed configuration [16]. Our guarantee uses a stronger comparator, the best horizon for each context along the realized decoding trajectory, and is learner-agnostic: any online regression method’s prediction error transfers to throughput regret, with online gradient descent [40] as one instantiation. The broader strategy of translating prediction or optimization error into decision quality is related to perturbation-based model predictive control [24] and online adaptive policy selection [25, 26], which rely on assumptions about the controlled dynamics, such as contractive perturbations, that we do not impose on the language-model prefixes.
Speculation as a Survival Problem
We first describe one round of speculation, and then show that a single sequence of rejection hazards determines the expected gain of every horizon.
One Round of Speculation: Draft, Verify, Commit
At round , let be the committed prefix and let contain all information available before the next horizon is chosen, including , system state, and the controller parameters. For , we define the target and proposal block laws
where conditioning on and the fixed proposal parameters is suppressed. We let and . In this lens, independent MTP heads form the special case , where recurrent or otherwise correlated heads (Ankner et al., [5]) keep the dependence on , and our analytic framework covers both of these cases.
The horizon (or action) is chosen before the current draft is sampled. Moreover, every horizon truncates the same causal law , so changing does not change the distribution of earlier draft tokens: it only decides how many of them are drafted and sent to the verifier. A round proceeds as in Figure 1: the drafter samples , and a single target pass scores all positions. Exact speculative verification (Leviathan et al., [20]; Chen et al., [9]) then checks the draft from left to right, accepting with probability

Figure 1. MTP with exact verification. Proposal heads condition on earlier sampled tokens. One target pass scores , and circles denote prefix-ordered acceptance. Rejection emits a residual correction . The dashed bonus branch requires full acceptance.
where on a sampled proposal. At the first rejection, the verifier draws a correction token from the normalized positive part of . If all proposals are accepted, it draws a bonus token from the target conditional at the end of the draft. So, a round with horizon commits tokens, where is the length of the accepted draft prefix, and the committed tokens follow the target law exactly (Leviathan et al., [20]; Chen et al., [9]). We analyze a fixed output-token budget where truncating the final output preserves its target law.
Exactness requires choosing the horizon before drafting. Exactness continues to hold when is chosen adaptively, provided it depends only on and on randomness independent of the current draft. Earlier acceptance decisions depend only on earlier proposals and independent coins, so the verifier reaches position with the proposal still distributed as , and the standard one-step argument applies at every position and every round, however poorly the horizon is chosen. However, choosing the horizon after seeing the draft can break this, even for a perfect drafter. To see this, let be independent fair bits, draw , and keep if and otherwise. Every drafted token is accepted, so the second emitted token is on the long branch and a new fair bonus token on the short one. It equals with probability rather than . AMTP therefore fixes before sampling the draft. Sampling transformations such as temperature or top- truncation must be built into the and the verifier uses.
Moreover, how likely a draft token is to pass depends on how far the proposal is from the target at that position. For a proposed prefix , we can measure this by the local discrepancy
which is exactly the probability that position is rejected given that the verifier reaches it along prefix : equivalently, .
Importantly, accurate token marginals do not guarantee an accurate multi-token proposal. For instance (as in Appendix § 10), independent MTP heads can fit every position perfectly while missing dependencies among future tokens. The block KL divergence splits into a marginal fitting error and a dependence term that remains even for exact marginals. In a simple repeated-bit example, the block mismatch approaches its maximum while the expected number of committed tokens per round stays below three for every horizon, so longer windows add a verification cost with little additional token gain. The horizon should therefore be chosen from how long draft prefixes actually survive verification, and from what each round costs, rather than from marginal prediction accuracy alone.
Survival and the Rejection Hazard
We now make the the notion of “how long draft prefixes survive” precise, and formally define the acceptance survival and rejection hazard. For a virtual full draft , we define the prefix acceptance weight via and , which is the probability that the verifier accepts all of given the sampled tokens.
Definition 1 (Acceptance survival). The acceptance survival is the probability of accepting at least draft tokens, conditional on , and is given by with .
Definition 2 (Rejection hazard). The rejection hazard at position is the probability of rejection there, given that the preceding proposals were accepted, and is given (conditional on ) by
When , we set , as later survival probabilities are zero regardless of this convention. Since , the survival and expected gain follow from the hazard:
where quantities indexed by an action refer to the potential outcome of choosing that horizon, even if the policy assigns it zero probability. To justify why hazards are the right prediction target, we point to (5): a single vector determines the expected gain of all horizons at once, so a controller that predicts it can score every horizon before drafting. Indeed, the identities need no independence between draft positions, so they hold for correlated heads as well as independent ones.
Why the hazard is survival-weighted. The numerator of (4) weights each prefix by its likelihood of surviving to position . Replacing with the unweighted mean discrepancy is generally incorrect since prefixes that survive verification are distributed differently from arbitrary draft prefixes. For instance, consider a binary vocabulary . The first head proposes or with probability each, while the target places mass on , so on , on , and . Then, suppose the second head matches the target exactly after but has disjoint support from it after , so and . The unweighted mean discrepancy at position 2 is , but , since a draft that starts with the well matched token is less likely to reach position 2 at all. Here, the true gain is , and plugging this in the unweighted mean gives 2.125. Section 4 shows how to build this change of prefix distribution into the learning objective.
Online Horizon Control
Each round of speculative decoding poses a one-step control problem: before drafting, choose the horizon that maximizes throughput. If the rejection hazards were known, Section 3 would solve this problem exactly, since (5) gives the expected gain of every horizon. However, since they are not known, AMTP uses certainty equivalence: it plugs in predicted hazards, acts on them, and updates the predictor via online gradient descent (OGD) from the feedback that verification provides. Algorithm 1 gives the full loop and Figure 2 illustrates it.

Figure 2. AMTP’s online control loop. Before drafting, it picks the horizon with the highest predicted throughput. Verification feedback at every drafted position, including those after a rejection, enters the weighted loss in Equation 8, and an OGD step updates the hazard predictor.
The Control Problem. Let be the conditional expected cost of a round with horizon , including drafting, verification, synchronization and all controller overhead, with . We compare horizons at the same learner state and under the same update protocol, where the controller’s objective is its throughput, and the optimal horizon at the current history is
Algorithm 1 Adaptive Multi-Token Prediction
Require: Maximum horizon ; parameter set and ; exploration rate ; step sizes ; initial cost model
for each speculative round do
Observe pre-draft features ; predict hazards and costs for
Set using equation 7 ▷ plug-in control,
With probability set ; otherwise ▷ exploration
Draft and verify all positions in one target pass
Commit the accepted prefix and one correction or bonus token
For : record , , and
Update with from equation 8 ▷ OGD
Record the measured round cost and update the cost model
end for
AMTP predicts the rejection hazards from pre-draft features and position , and estimates rounds costs , where the predictions are -measurable. It converts the hazard predictions into survival probabilities and expected token gains and selects a candidate horizon:
with a fixed tie-breaking rule. A running product and a running sum score all horizons in time, and the controller does not assume that throughput is unimodal in , and so the only part of (7) that must be learned is the hazard forecaster .
Learning the hazards via Online Gradient Descent (OGD)
OGD needs a loss whose error is reflective of the error attributed to the control problem. Following the change of prefix distribution in Section 3.2, we use the survival-weighted population loss
conditional on , with the proposal law and features held fixed. Let be the same expression with the true hazards in place of . Then, represents the minimum loss over unrestricted pre-draft hazard predictions. We then show the following calibration identity:
Lemma 1 (Calibration identity). For every -measurable parameter ,
Expanding the weighted square around yields the identity, and the cross term vanishes by the hazard definition in (4); so, the loss is minimized by the true hazards, and its excess is a hazard error weighted by how often each position is reached: it is weighted less on positions that are less likely to survive verification. Section 5 shows that this excess controls the throughput lost.
The ideal learner. If verification revealed and at all positions of a draft , then the realized loss would be unbiased for , and the controller would simply run projected OGD [40, 15]:
where is the projection onto the (convex) set of parameters . For an affine forecaster , the gradient is . This learner always acts greedily, and § 5 shows that its throughput regret is sublinear in the number of rounds. However, this is an idealization as the discrepancy compares the draft with the target distribution , and the target computes only at positions it verifies. So, a round that drafts tokens gives no feedback at positions , and observing every position at each round would mean drafting and verifying the full window each time, which defeats the purpose of choosing the horizon. AMTP avoids this: it drafts the full window only in a small random fraction of rounds and re-weights the feedback so that the loss stays unbiased (§ 4).
Learning from partial feedback
A round with horizon reveals feedback only at positions . If a greedy controller under-estimates late-position survival, its drafts might be short, it may never observe late positions, and it might never corrects the estimate. To address this, AMTP makes two changes to the ideal learner.
Exploration. With probability the controller drafts , and with probability it drafts the full window [19]. The randomness here is independent of the draft given , so position is observed with known probability . Other exploration schemes, such as Thompson sampling for contextual bandits [37, 3], could replace uniform full-horizon exploration. We use -greedy exploration because it gives known inclusion probabilities, which is needed for inverse-probability weighting.
Inverse-probability weighting. After verification, and are available for every . Then, reweighting each observed term by its inclusion probability gives
AMTP runs (10) with in place of , holding and fixed in line 8 of Alg 1. However, relative to the ideal learner, exploration costs some throughput in rounds drafting , and the factor makes gradients noisier: we quantify both in § 5.
From Prediction Error to Decoding Efficiency
We now ask how much decoding efficiency AMTP loses by acting on predicted rather than true hazards. First, we show that, for any online learner, the throughput regret against the best horizon at each visited history is controlled by the learner’s excess prediction loss. We then bound that loss for OGD, first for the ideal full-feedback learner and then for AMTP’s partial-feedback learner, and then we bound the price of partial feedback. Finally, we show how to convert the throughput regret into bounds on expected completion time and speedup. We give complete proofs in Appendix B.
Prediction error to throughput regret
We allow the controller to maximize predicted throughput only approximately, with a selection error that accommodates exploration:
For a comparator sequence , define
Here is the learner’s regression regret against the comparator, is the comparator’s approximation error, and is the throughput regret against the best horizon at each history the learner visits. This comparator is stronger than the best fixed horizon: it may change from round to round with the context. By Lemma 1, . Write .
Theorem 1 (Dynamic regret transfer). Under the preceding assumptions, for every comparator sequence and every realization of the pre-round histories and selected horizons, we have
The key idea is to connect calibration to throughput prediction, and then to horizon selection. Telescoping the survival products and applying Lemma 1 bounds the error in predicted throughput uniformly over horizons. Combining this with Equation (12) bounds the per-round regret:
Summing over rounds and applying Cauchy–Schwarz gives Equation (13). A regression guarantee therefore becomes a throughput guarantee through the learner’s total excess prediction loss, including the comparator’s approximation error. Importantly, this does not depend on how is produced, which allows us to use any online learning algorithm and transfer their guarantees to our setting.
Remark 1 (Sharpness). The square-root dependence in Eq. (13) cannot be improved uniformly. Prop. 2 with independent proposals and known costs incurs one-round throughput regret .
Remark 2 (Estimated costs). Suppose instead that the controller uses , and let . Then, measuring relative to gives
Learning the hazards with OGD
We now bound for the two learners of § 4. Both use an affine forecaster on a nonempty closed convex set of diameter at most . The features are -measurable with , and forecasts lie in for all . Initialize . The comparator is -measurable, and its path length is bounded almost surely by a deterministic quantity, , as is standard for dynamic regret ([40, 6, 36]). We assume known costs throughout this subsection.
Ideal full feedback. The ideal learner of § 4 observes all positions and always acts greedily, so . Its gradients are unbiased for and satisfy .
Corollary 1 (OGD with full feedback). Under these conditions, projected OGD (Equation (10)) on the full-feedback loss with constant step size satisfies
Under realizability, optimizing , for fixed , , , , gives .
Equation 10 is the standard projected-OGD argument with a moving comparator. Theorem 1 and Jensen’s inequality then give the final rate. We give the result for any unbiased gradient with bounded second moment in Equation (10). As § 4 explains, this learner needs the target distribution at all positions in every round, so this is only our benchmark for an implementable controller.
Partial feedback. AMTP replaces with the observed loss. Here, the gradients remain unbiased, but each observed term is scaled by , so for we have , where exploration rounds contribute to the selection error.
Corollary 2 (OGD with partial feedback). Under these conditions, Algorithm 1 with known costs, exploration rate and constant step size satisfies
Under realizability, optimizing , for fixed , , , , gives .
The projected-OGD bound with the second-moment estimate above gives equation (16), and Theorem 1 with Jensen’s inequality and a balancing argument gives the final rate. This tuning uses the number of rounds and a known bound on comparator variation.
The price of partial feedback. Equations 15 and 16 differ only by the factor in the gradient-noise term, and the rates differ only via the selection-error term . Here, exploration must be frequent enough to keep gradient noise in check, but it also drafts the full window when a shorter one may be better. Balancing these effects moves the rate from to , which are both sublinear when . So, the average throughput approaches that of the best horizon at each history, and AMTP pays a vanishing gap for learning from the feedback of its horizon choices.
From throughput regret to speedup
We measure the throughput regret per round, as the goal is to minimize the time to produce an output. We fix an integer output budget , and compare AMTP with a baseline under matched prompts, the same target output law and the same timing scope. Let denote the baseline’s completion time and define its effective rate , assuming .
Corollary 3 (Completion time and speedup). Given at each visited history, for some constant , let the realized round cost satisfy , and let
Extend the same policy to rounds and suppose that . Then, the expected completion time and the speedup, defined as the ratio of expected completion times, satisfy
In particular, whenever .
To see why, let . The assumption on gives . Each round costs at least one token, so , and the final round overshoots the budget by at most tokens. Since is determined before round , conditional expectations gives
which is the bound in (18). Dividing the baseline’s completion time by this gives the claim.
Speedup guarantee of AMTP. Corollary 2 with gives , so the penalty in Equation (18) is . If and , the penalty vanishes and . Acceptance and round cost thus set the achievable speedup , and learning the horizon costs only a vanishing fraction of it.
Experiments
We evaluate AMTP on three self-speculative multi-token-prediction families (self-distilled Qwen3-4B, Meta’s facebook/multi-token-prediction 7B, and Medusa heads on Llama-3.1-8B-Instruct) and two cross-model pairings with a separate, larger target (self-distilled 4B → Qwen3-14B; Meta MTP 7B → Llama-2-13B), on AIME 2025 and GSM8K (full setup in Table 4, Appendix § 13). All runs disable KV caching, so verification recomputes the full context each round; each condition is a single unseeded run, and we compare conditions only within a model since hardware differs across runs. The fixed-window baseline uses the largest window each draft supports (, , ); the adaptive controller is the affine hazard predictor and cost model of § 4 with -bounded exploration, updated by one gradient step per round (hyperparameters in Table 5, Appendix § 13).
| # | Family | Pairing | Draft | Target | Plain baseline | Max |
| 1 | Self-distilled | Self-speculative | Qwen3-4B-Inst-2507-MTP | Same checkpoint | Same checkpoint | 16 |
| 2 | Meta MTP | Self-speculative | multi-token-prediction 7B | Same checkpoint | Llama-2-7B | 4 |
| 3 | Medusa | Self-speculative | Llama-3.1-8B-Instruct + Medusa heads | Same checkpoint | Llama-3.1-8B-Instruct | 6 |
| 4 | Self-distilled | Cross-model | Qwen3-4B-Inst-2507-MTP | Qwen3-14B | Qwen3-14B | 16 |
| 5 | Meta MTP | Cross-model | multi-token-prediction 7B | Llama-2-13B | Llama-2-13B | 4 |
Table 4. MTP draft/target experiment matrix, run without KV-caches. Rows 1-3 are the core self-speculative pairings; rows 4-5 are the cross-model scaling pairings. Each row is evaluated on GSM8K and AIME 2025 with a fixed window, AMTP, and plain (next-token) decoding of the target.
| Quantity | Value |
| Maximum window | 4 / 6 / 16 (Meta / Medusa / self-dist.) |
| Hazard model | affine (linear, no sigmoid link), 5 inputs |
| Hazard optimiser, learning rate | Projected OGD, 0.01 |
| Hazard projection radius | 10.0 |
| Exploration rate | 0.1 |
| Cost optimiser, learning rate | Adam, 0.01 |
| Start-up updates | 20 |
| Context scale , cost scale | 100, 1000 |
| Checkpoint interval | every 50 updates |
Table 5. AMTP configuration used in all experiments. No value was tuned per model or per task.
| Model | Decoding | AIME 2025 tok/s | AIME 2025 Speed-up | GSM8K tok/s | GSM8K Speed-up |
| Self-distilled Qwen3-4B | Plain (next-token) | 18.8 | 1.00× | 16.2 | 1.00× |
| Fixed | 34.0 | 1.81× | 38.6 | 2.39× | |
| Adaptive (ours) | 34.5 | 1.83× | 38.3 | 2.37× | |
| Meta MTP 7B | Plain (next-token) | 12.5 | 1.00× | 4.7b | 1.00× |
| Fixed | 37.7 | 3.01× | 19.2b | 4.09× | |
| Adaptive (ours) | 31.9 | 2.55× | 19.4b | 4.13× | |
| Meta MTP 7B → Llama-2-13B | Plain (next-token) | 6.2 | 1.00× | 2.7b | 1.00× |
| Fixed | 7.6 | 1.23× | 4.7b | 1.75× | |
| Adaptive (ours) | 28.2 | 4.55× | 11.6b | 4.30× | |
| Medusa, Llama-3.1-8B | Plain (next-token) | 10.7 | 1.00× | 13.0 | 1.00× |
| Fixed | 23.6 | 2.20× | 20.0 | 1.54× | |
| Adaptive (ours) | 25.3 | 2.37× | 19.4 | 1.50× | |
| Self-dist. 4B → Qwen3-14B | Plain (next-token) | 10.4 | 1.00× | 6.4a | 1.00× |
| Fixed | 22.5 | 2.17× | 21.3 | 3.33× | |
| Adaptive (ours) | 24.7 | 2.38× | 21.5 | 3.36× |
Table 1. Decoding throughput (tokens/s) and speed-up over plain decoding for each draft/target pairing on AIME 2025 and GSM8K, with AMTP in bold if it exceeds the best fixed window by more than 1%. aPlain decoding run reached 1,166 of 1,319 GSM8K problems before its timeout: speed-up computed over that subset. bMeta MTP’s base checkpoints do not emit an end-of-sequence token after answering, so GSM8K runs for this pairing are capped at 100 problems, 100 rounds each.
Discussion. AMTP helps most when drafting and verification differ in cost and when reasoning traces are long and variable in length. The cross-model pairings, where draft and target are separate models, show the largest gains over the fixed window: Meta MTP 7B →Llama-2-13B improves throughput by 268% on AIME and 147% on GSM8K (over its best fixed window), and self-distilled 4B →Qwen3-14B by 10% on AIME while staying within 1% of the best fixed window on GSM8K. In the self-speculative pairings, draft and target share one forward pass, so a longer horizon costs little extra and there is less to gain: self-distilled 4B and Medusa improve by 2–7% on AIME and stay within 3% of the fixed window on GSM8K. The one regression is Meta MTP 7B on AIME (−15%). Meta’s base checkpoint does not reliably stop after answering, and every round of a longer generation costs more. We note, however, that its acceptance rate was higher under adaptive control throughout. Across all pairings, AMTP drafts fewer tokens per round while committing a similar number, allowing it to accept a higher fraction of what it drafts than any fixed window, converting wasted draft computation into completed tokens even where the net throughput effect is small.
Conclusion and Future Work
In this work, we study how far MTP should speculate. Since all MTP heads fire in a single forward pass, the horizon must be chosen before drafting, and the right choice changes with context. We showed that acceptance survival determines the expected number of committed tokens at every horizon for any causal proposal, and built AMTP, an online controller that learns survival from verification feedback and picks the horizon with the highest predicted throughput, without retraining any model or changing the exact verifier. We show that any online learner’s prediction error translates into throughput regret against the best horizon for each context, and in turn into a bound on wall-clock completion time. Empirically, AMTP decodes up to faster than standard decoding and matches or outperforms the fixed window in 8 of 10 settings.
| Model | Eval | Accept. (%) Fixed | Accept. (%) Adapt. | Tokens/round Fixed | Tokens/round Adapt. | Accuracy (%) Plain | Accuracy (%) Fixed | Accuracy (%) Adapt. |
| Self-distilled 4B | GSM8K | 23 | 32 | 4.67 | 4.63 | 89.4 | 85.0 | 85.7 |
| Medusa 8B | GSM8K | 33 | 48 | 2.99 | 2.90 | 83.2 | 79.6 | 79.2 |
| Self-dist. 4B → 14B | GSM8K | 21 | 29 | 4.33 | 4.33 | 92.8ᵃ | 92.9 | 93.5 |
| Self-distilled 4B | AIME | 18 | 34 | 3.84 | 3.83 | 20.0 | 30.0 | 26.7 |
| Meta MTP 7B | AIME | 50 | 57 | 3.00 | 3.03 | 0.0 | 0.0 | 0.0 |
| Meta MTP 13B | AIME | 44 | 52 | 2.76 | 2.51 | 0.0 | 0.0 | 0.0 |
| Medusa 8B | AIME | 22 | 46 | 2.34 | 2.28 | 0.0 | 0.0 | 0.0 |
| Self-dist. 4B → 14B | AIME | 14 | 28 | 3.23 | 3.20 | 20.0 | 20.0 | 23.3 |
Table 2. Draft acceptance rate, tokens committed per verification round, and accuracy for plain decoding, the best fixed window and AMTP on AIME 2025 and GSM8K. ᵃPlain decoding run reached 1,166 of 1,319 GSM8K problems before its time limit; accuracy computed over that subset.
| Model | Eval | Mean | Std. | Range | Distinct |
| Self-distilled 4B | GSM8K | 11.21 | 2.73 | 1–16 | 16 |
| Meta MTP 7B | AIME | 3.54 | 0.69 | 1–4 | 4 |
| Meta MTP 13B | AIME | 2.93 | 0.99 | 1–4 | 4 |
| Medusa 8B | GSM8K | 3.95 | 1.02 | 1–6 | 6 |
Table 3. Distribution of the draft window chosen by AMTP over all verification rounds of a run. We remark here that the window is not constant: it varies across rounds within a single generation and across generations.

Figure 3. Draft window chosen by AMTP, averaged over generations in bins of 10 verification rounds (points; bars show one standard deviation across rounds in the bin). The window adapts as generation proceeds instead of staying at a single value.
Several limitations point to future work. Our experiments decode one sequence at a time without a KV-cache: batched serving with caching changes the cost of each horizon and is a natural direction. Comparing hazard-based and direct survival prediction, and extending horizon control to tree-structured drafts, are promising directions. Finally, it would be interesting to extend our work to adaptive diffusion-based speculative methods.
AI Use Statement
Generative AI tools assisted with editing the manuscript, generating code for the experiments, identifying and summarizing related literature, and proof-checking the mathematical formulations and proofs. The authors are responsible for checking the final claims, proofs, and citations.
Acknowledgements
We thank Adam Wierman, Jacob Abernethy, Matthew Fahrbach, and Sarah Liaw for helpful comments. EA’s research is supported by NSF Grant CCF 2338816.
Outline of the Appendices.
Section A provides some motivation for adaptive horizon selection;
Section B presents the proof of our main results;
Section C shows how to estimate the costs and throughput measurements;
Section D explains our experimental setup and provides further results.
Why Longer Drafts Degrade
Draft qualities can deteriorate with the horizon even if every head is accurate on its own. For independent heads, the block mismatch has two sources: marginal fitting error, and dependence among future target tokens that the heads cannot represent, neither of which individually determines the throughput-optimal horizon, which depends on the verifier’s acceptance profile and the round cost.
Fix a round, condition on its committed context, and suppress the round index. Let and be the target and proposal laws over the next tokens, assumed to be prefix marginals of common laws and , and let be the position- target marginal. The vocabulary is finite and logarithms are natural; KL identities use the usual extended-value convention.
Proposition 1 (Dependence and marginal fit). For independent heads , let be the total correlation of the target block. Then,
and, at each position ,
Proof. Insert into the likelihood ratio:
and take expectations under . Inserting it in the conditional ratio gives Eq. (20).
Even exact marginal heads, , therefore maintain the total correlation as their block KL divergence. The term is the information about earlier target tokens that an independent head cannot use, and heads that condition on earlier proposed tokens can reduce it.
We now explain how the block mismatch grows with the horizon. Let . Because marginalization contracts total variation, we have
Similarly, block mismatch caps survival: with the conditional kernels defined along a path, the probability that all sequential checks accept is
since each product of minima is at most the minimum of the two products.
Example 1 (Perfect marginals, bounded gain). Let the target draw one fair bit and repeat it. for all , so . Independent heads with exact marginals draw a fresh fair bit at each position, so . Then
The first proposal is always accepted. Given that the prefixes have been accepted, each later target conditional is a point mass at the first bit while the proposal is uniform, so each later position is accepted with probability . Therefore, we have that , and , showing that exact marginals can coexist with block mismatch approaching and an expected gain that never reaches tokens, no matter how long the draft.
Proofs of the Efficiency Guarantees
This section proves the results stated in the main text, in the order they appear. Throughout, write and . Then, by Lemma 1, we have .
Predictor class. We use an affine forecaster . For instance, consider simplex weights and features : here, the loss is convex in and the forecasts stay in .
Proof of Lemma 1. Write and expand each square. By the definition of the hazard equation 4, the cross term vanishes:
The first square summed over is , and the second is multiplied by .
Proof of Theorem 1. First, telescoping the survival products gives
Every factor in the product lies in , so summing over , applying Cauchy–Schwarz, and using and Lemma 1,
Dividing by and using gives . Combining this with the selection rule in equation 12 gives
Then, by Cauchy–Schwarz, we have .
A general projected-OGD bound. Both learning corollaries follow from the standard projected-OGD argument ([40]) with a moving comparator and stochastic gradients.
Lemma 2 (Projected OGD with a moving comparator). Let be a nonempty closed convex set of diameter at most , let , and let each be convex and differentiable on . Suppose with and . Let be -measurable with almost surely, for a deterministic . Then
With in the nondegenerate case, Theorem 1 and Jensen’s inequality give
Proof. Nonexpansiveness of the projection gives
By convexity and conditional unbiasedness, the expectation of the left-hand side is at least . To telescope against a moving comparator, we use
Summing over proves equation (23). Substituting the chosen gives , and equation (24) follows from Theorem 1 and Jensen’s inequality.
Proof of Corollary 1 (full feedback). Here, is unbiased for , and since , and , we have . Apply Lemma 2 with and . Under realizability, for fixed , , , .
Proof of Corollary 2 (partial feedback). Unbiased loss and gradient. Fix . Because the exploration coin is independent of the draft given , for each position
This discrepancy is needed only when the indicator is one. Summing over shows that the observed loss is conditionally unbiased, and differentiating shows this for the gradient of the affine predictor.
Second moment. Using , and ,
Substituting into equation (23) gives the regression bound.
Selection error. On a greedy round . On an exploration round, , so . Combining these bounds gives the final rates.
Proof of Corollary 3. Let . Since and ,
Every round commits at least one token, so , and the last round overshoots the budget by at most tokens: . The event is determined before round , so taking conditional expectations round by round and summing gives
where the last step uses and . Dividing by gives the completion-time bound, and dividing the baseline’s expected completion time by it gives the speedup bound.
The square-root dependence is tight. The transfer in Theorem 1 scales with the square root of the excess loss. The next result shows that this cannot be improved for the forecast-greedy controller, even with independent heads, known costs and .
Proposition 2. (Square-root dependence is unavoidable.) For every there is an exact speculative-decoding instance with , costs and , and a pre-draft hazard forecast with excess loss , such that forecast-greedy selection incurs
Repeating the construction over rounds gives
Hence no uniform one-round bound holds under the assumptions of the transfer theorem.
Proof. Instance. Take the vocabulary and a target that emits 0 with probability one after every prefix. The independent proposal heads are and . The first token is always accepted, so and . At the second position : a proposed 0 is always accepted and a proposed 1 always rejected, and both the correction and the bonus token are 0. Hence , and .
Forecast and excess loss. Take and . The discrepancies are deterministic and , so and , giving . Both the forecast and the true hazards lie in the affine class on .
Regret. Counting the correction or bonus token, , and . With the stated costs, , and . So and are unique, and the regret is . Since and the costs stay fixed as , no bound of the form , and in particular none of the form , can hold uniformly.
Many rounds. Use the same models, costs and forecasts at every round. All committed tokens are 0 and the models ignore the history, so these are consistent rounds of one decoding process, with and .
The construction above places the two horizons arbitrarily close to a decision boundary. When every suboptimal horizon is separated from the best one, the transfer becomes linear in the excess loss.
Proposition 3. (Transfer under a throughput gap.) Under the known-cost assumptions of Theorem 1, suppose such that for every round and horizon . Then
Proof. Let and . Step 2 of the proof of Theorem 1 gives . If there is nothing to prove. Otherwise , and Young’s inequality gives, after rearranging,
Summing over completes the proof.
Predicting survival directly. Hazards are one way to parameterize acceptance survival. The same verification feedback also supports predicting directly, using either of the labels
Both have conditional mean given . The second equals : it integrates out the current proposed token, so it is a Rao–Blackwellization of the first and has no larger conditional variance. Both are available at every drafted position under the feedback assumptions of § 4.
Let be a pre-draft survival forecast, let be either label, and define . As in Lemma 1, its excess loss is
With the same exploration scheme, the observed loss is conditionally unbiased. With , Cauchy–Schwarz gives
so with known costs the one-round throughput regret is at most .
Round Costs and Measured Throughput
The marginal value of an extra position makes the tradeoff explicit. If , then
The gain in expected committed tokens must cover the extra round cost. This comparison is local; the controller scores every candidate and does not assume that is unimodal.
Estimated latency costs. Write as in equation (9). If the controller instead uses , assume and define
Then
Thus Theorem 1 becomes
where is now defined relative to the estimated-cost objective.
Connection to measured acceptance. Over complete rounds, define the pooled draft acceptance rate, average window length, and average realized round cost by
Since , the measured throughput satisfies the exact identity
Consequently, acceptance improvements translate into throughput improvements through both the average window length and the round cost. Acceptance alone is insufficient: for a fixed context and draft law,
is nonincreasing in , so shortening the window increases acceptance without increasing throughput.
The identity in equation (31) applies to complete rounds under the one-correction-or-bonus convention. An output cutoff or early end-of-sequence token requires recording the actual committed count; the pooled identity should not be applied blindly to such truncated rounds.
Experimental Setup and Further Results
This section describes our experimental setup and provides further results. We use the following checkpoints that can be found on HuggingFace:
jwkirchenbauer/Qwen3-4B-Inst-2507-MTP,facebook/multi-token-prediction(7B_1T_4),meta-llama/Llama-3.1-8B-Instructwith Medusa heads,Qwen/Qwen3-14B,meta-llama/Llama-2-\{7b, 13b\}-hf.
Hardware and Software. Self-distilled Qwen3-4B, its Qwen3-14B cross-model pairing and Medusa runs use a NVIDIA H100 on a university SLURM cluster; Meta MTP runs use a single NVIDIA H100 SXM on a rented cloud pod. Every run occupies one GPU exclusively and processes one sample at a time, so measured wall time is not distorted by co-located jobs. Throughput is therefore comparable between conditions on the same hardware but not across the two hardware groups. Models are loaded in the checkpoint’s native precision through Hugging Face transformers; evaluations are driven by Inspect AI, using a custom model provider that wraps our speculative-decoding engine. The exact package versions are pinned in the released code’s lock file.
Tasks and Caps. AIME 2025 uses all 30 problems with a limit of 1,000 verification rounds and 3,000 generated tokens per problem. GSM8K uses the standard ten-shot prompt (fixed few-shot seed) with a limit of 300 rounds per problem for the speculative conditions. The self-distilled runs (self-speculative and paired with the Qwen3-14B target) use all 1,319 test problems; the Medusa runs use the first 250. The Meta MTP checkpoints are base models that do not emit an end-of-sequence token after answering, so they generate until the round cap; for the Meta 13B pair on GSM8K we therefore cap each of the first 100 problems at 100 rounds (300 tokens for plain decoding) and use identical caps across conditions, and we re-analyze the earlier 300-round Meta 7B fixed run on the same prefix. These runs measure throughput and acceptance only, since a 7–13B base model’s GSM8K accuracy is not a meaningful check of lossless decoding. The Qwen3-14B plain-decoding baseline for the self-distilled cross-model pairing reached 1,166 of 1,319 GSM8K problems before its time limit; its throughput and accuracy are computed over that completed subset, and the fixed and adaptive conditions for this pairing still cover all 1,319. Sampling draws from the unmodified softmax of each model (no temperature scaling), without a fixed random seed, so runs are not bit-for-bit repeatable.
Metrics. For a run with rounds , let be the number of tokens committed in round (accepted drafts plus the one token sampled from the target), the window used, and the measured wall time of the round, including drafting, verification and synchronization.
For plain next-token decoding, a round produces exactly one token (, no drafting). Sums run over all problems of an evaluation, so throughput is a token-weighted, not a per-problem, average. We report accuracy from the task’s own scorer.
Baselines. Plain decoding uses the same target model and prompt with the standard autoregressive loop, also without a KV-cache, so speed-ups isolate the effect of speculation. For the self-speculative pairings (self-distilled Qwen3-4B, Medusa on Llama-3.1-8B-Instruct) the plain-decoding baseline is the same checkpoint decoding without drafting. Meta’s 7B multi-token-prediction checkpoint has no separately released plain next-token model, so its plain-decoding baseline is Llama-2-7B, the base architecture it was fine-tuned from; for the 13B pairing, the target itself, Llama-2-13B, is the baseline. For the self-distilled cross-model pairing, the baseline is the target alone, Qwen3-14B.
Draft-Model Implementations
All draft models implement one interface: given the current context, return sampled tokens together with the distributions they were sampled from. The engine is agnostic to how the draft produces them, so AMTP and the verification step are shared across families.
Self-speculative and cross-model pairings. In a self-speculative pairing the same weights draft and verify and are loaded once. In a cross-model pairing a small multi-token-prediction draft proposes tokens for a larger target of the same vocabulary: self-distilled Qwen3-4B for Qwen3-14B, and Meta’s 7B multi-token-prediction model for Llama-2-13B. The engine checks that the draft and target share a tokenizer and vocabulary; the self-distilled model declares extra mask tokens.
Self-distilled Qwen3-4B (mask-token drafting). The checkpoint drafts tokens in one forward pass by appending reserved mask tokens (ids 151669–151700) to the context and reading the logits at the last real position and at each mask position. It supports . It ships without its own verification scheme, so verification is performed by our engine with the standard accept/reject rule.
Meta MTP (independent parallel heads). The 7B checkpoint has four output heads that read the trunk’s final hidden state and predict tokens independently. The checkpoint is not in Hugging Face format, so we load it with a self-contained implementation (transformer trunk, four heads, SentencePiece tokenizer). A single trunk forward produces all 4 distributions.
Medusa on Llama-3.1-8B-Instruct. Six residual-block heads, each with its own output projection, predict from the last hidden state of the base model. The base model’s own next-token distribution supplies position one; head then predicts position . Head weights are loaded from a separate repository, and the base model is the unmodified instruct checkpoint.
No KV-cache. All runs recompute the full context on every forward pass. This makes per-round cost depend visibly on context length, which AMTP’s cost model represents (Appendix D.2), and it removes cache-rollback logic that differs between attention variants and would confound the comparison across draft families. Absolute throughput is consequently lower than with an optimised cached implementation; the relative comparisons between conditions are what we report.
Adaptive Multi-Token Prediction Details
Hazard targets. The hazard label at drafted position is , the probability that a token drawn from is rejected. It is computed for every drafted position, including those after the first rejection, via the target distributions already produced by the verification.
Cost target. The cost model is fitted to the whole round’s measured wall time, as AMTP’s decision only needs the total. Since every run disables the KV-cache (Appendix § 13.§ 1), the cost model includes context length as a feature alongside the window ; a model fit with context length excluded explains almost none of the round-to-round cost variance under this setting.
Checkpoint lineage. Control-block state is saved every 50 updates under a key made of the draft checkpoint, target checkpoint and an implementation tag. The tag changes whenever the cost regime changes, and the current tag identifies runs without a key–value cache and with the rescaled cost model, so weights fitted under a different regime are never loaded.
Start-up. For the first 20 updates the window is drawn with probability proportional to , so the cost model observes several window sizes. All conditions labelled “adaptive” begin with this phase; the numbers we report include it.
Complete Results
Tables 7 and 8 list our final results behind the main-text tables, and Table 6 gives the distribution of windows chosen by AMTP. Tables 1–3 in the main text are derived from the same records.
| Model / eval | 2 | 3 | 4 | 5–6 | ||
| Meta MTP 7B, AIME | 1.4 | 7.3 | 27.0 | 64.3 | 0.0 | 0.0 |
| Meta MTP 13B, AIME | 9.5 | 24.6 | 29.5 | 36.5 | 0.0 | 0.0 |
| Meta MTP 7B, GSM8K* | 0.3 | 1.4 | 19.0 | 79.2 | 0.0 | 0.0 |
| Meta MTP 13B, GSM8K* | 0.5 | 2.9 | 16.7 | 79.9 | 0.0 | 0.0 |
| Medusa 8B, GSM8K | 0.1 | 4.7 | 29.2 | 43.5 | 22.5 | 0.0 |
| Self-dist. 4B, GSM8K | 0.0 | 0.0 | 0.0 | 0.2 | 2.9 | 96.9 |
| Self-dist. 4B → 14B, AIME | 0.3 | 1.1 | 4.7 | 10.4 | 29.7 | 53.8 |
| Self-dist. 4B → 14B, GSM8K | 0.0 | 0.0 | 0.2 | 0.6 | 4.0 | 95.1 |
Table 6. Share of verification rounds (%) in which the control block chose each window size, over the full round population. The distribution is wide for every model, and the mode differs by model and by task. *Capped runs (first 100 problems, first 100 rounds).
| Draft → target | Condition | Rounds | Tok./round | Accept. (%) | Tok./s | Gen. (h) | Acc. (%) |
| Self-dist. Qwen3-4B | Plain | 81,791 | 1.00 | – | 18.8 | 1.21 | 20.0 |
| Self-dist. Qwen3-4B | Fixed | 25,549 | 3.84 | 17.7 | 34.0 | 0.80 | 30.0 |
| Self-dist. Qwen3-4B | Adaptive | 25,927 | 3.83 | 33.9 | 34.5 | 0.80 | 26.7 |
| Self-dist. 4B → Qwen3-14B | Qwen3-14B | 60,091 | 1.00 | – | 10.4 | 1.61 | 20.0 |
| Self-dist. 4B → Qwen3-14B | Fixed | 18,244 | 3.23 | 14.0 | 22.5 | 0.73 | 20.0 |
| Self-dist. 4B → Qwen3-14B | Adaptive | 16,525 | 3.20 | 28.2 | 24.7 | 0.60 | 23.3 |
| Meta MTP 7B | Llama-2-7B | 13,190 | 1.00 | – | 12.5 | 0.29 | 0.0 |
| Meta MTP 7B | Fixed | 7,009 | 3.00 | 50.0 | 37.7 | 0.16 | 0.0 |
| Meta MTP 7B | Adaptive | 11,268 | 3.03 | 57.2 | 31.9 | 0.30 | 0.0 |
| Meta MTP 7B → Llama-2-13B | Llama-2-13B | 14,309 | 1.00 | – | 6.2 | 0.64 | 0.0 |
| Meta MTP 7B → Llama-2-13B | Fixed | 5,370 | 2.76 | 43.9 | 7.6 | 0.54 | 0.0 |
| Meta MTP 7B → Llama-2-13B | Adaptive | 2,772 | 2.51 | 51.6 | 28.2 | 0.07 | 0.0 |
| Medusa Llama-3.1-8B | Plain | 53,294 | 1.00 | – | 7.5 | 1.97 | 0.0 |
| Medusa Llama-3.1-8B | Fixed | 19,760 | 2.34 | 22.4 | 23.6 | 0.55 | 0.0 |
| Medusa Llama-3.1-8B | Adaptive | 18,395 | 2.28 | 46.2 | 25.3 | 0.46 | 0.0 |
Table 7. AIME 2025 evaluations ( problems in each row). Rounds: total verification rounds over all problems (one round is one token for plain decoding). Tok./s: total committed tokens per total measured round wall time. Gen. (h): summed round wall time. Acc.: fraction of problems answered correctly; it is 0 for every 7–13B model (Meta and Llama), so it carries no signal there.
Worked Examples. We provide real unedited excerpts from logged telemetry of a self-distilled Qwen3-4B adaptive run on AIME 2025: *Let A be the set of positive integer divisors of 2025. Let B be a randomly selected subset of A. The probability that B is a nonempty set with the property that the least common multiple of its elements is 2025 is , where and are relatively prime positive integers. Find .
| Draft → target | Condition | Problems | Rounds | Tok./round | Accept. (%) | Tok./s | Gen. (h) | Acc. (%) |
| Self-dist. Qwen3-4B | Plain | 1319 | 279,050 | 1.00 | – | 16.2 | 4.80 | 89.4 |
| Fixed | 1319 | 64,008 | 4.67 | 23.0 | 38.6 | 2.15 | 85.0 | |
| Adaptive | 1319 | 65,183 | 4.63 | 32.4 | 38.3 | 2.19 | 85.7 | |
| Self-dist. 4B → Qwen3-14B† | Qwen3-14B† | 1166 | 179,720 | 1.00 | – | 6.4 | 7.80 | 92.8 |
| Fixed | 1319 | 47,643 | 4.33 | 20.8 | 21.3 | 2.69 | 92.9 | |
| Adaptive | 1319 | 47,529 | 4.33 | 28.5 | 21.5 | 2.66 | 93.5 | |
| Meta MTP 7B* | Llama-2-7B | 100 | 24,440 | 1.00 | – | 4.7 | 1.44 | – |
| Fixed | 100 | 8,913 | 3.18 | 54.5 | 19.2 | 0.41 | – | |
| Adaptive | 100 | 9,065 | 3.21 | 58.6 | 19.4 | 0.42 | – | |
| Meta MTP 7B → Llama-2-13B* | Llama-2-13B | 100 | 23,660 | 1.00 | – | 2.7 | 2.46 | – |
| Fixed | 100 | 8,108 | 2.72 | 42.9 | 4.7 | 1.31 | – | |
| Adaptive | 100 | 8,206 | 2.75 | 46.6 | 11.6 | 0.54 | – | |
| Medusa Llama-3.1-8B | Plain | 250 | 44,877 | 1.00 | – | 13.0 | 0.96 | 83.2 |
| Fixed | 250 | 15,619 | 2.99 | 33.1 | 20.0 | 0.65 | 79.6 | |
| Adaptive | 250 | 16,739 | 2.90 | 48.2 | 19.4 | 0.69 | 79.2 |
Table 8. Complete GSM8K results. Self-distilled 4B and Qwen3-14B target rows use all 1,319 problems where finished; Medusa uses the first 250 problems; *Meta MTP runs are capped (first 100 problems, first 100 rounds; ∼300 tokens for plain decoding) because the Meta base models do not terminate after answering, so their scored accuracy (1–4%, driven almost entirely by truncation) is not meaningful and is omitted. †The Qwen3-14B plain run reached 1,166 of 1,319 problems before its time limit; its accuracy is over those completed problems.
Example 1: a difficult stretch, seven consecutive rounds. Roughly 650 verification rounds into its own generated solution, the model is deriving the number of qualifying subsets. Table 9 lists seven consecutive rounds verbatim. In each round the control block first chooses a window from its own hazard forecast, then drafts exactly tokens; the accepted column gives how many of those were accepted before the first rejection (or all , if none were rejected), and correction gives the resampled token that replaced the first rejected draft token. Draft tokens past the rejection point are never verified and contribute nothing to the committed output.
The committed text across these seven rounds, concatenated, reads: “2768 - 4864 = 27904\n\nYes,\n\nSo, the number of nonempty subsets$ such that LCM” with those visible self-corrections (Yes, So) where the model’s own verification caught a drafted continuation it would not have produced itself. The chosen window swings from k_t=7k_t=16$ round to round, and every round in this stretch needed at least one correction; this is a difficult patch of the derivation, not representative of the easier, longer-window stretches that dominate most of a generation.

(a) (a) Decoding throughput (tokens/second) for plain decoding, the best fixed window, and AMTP, across all five pairings and both evaluation tasks. Every multi-token-prediction condition beats plain decoding; the Meta MTP cross-model pairing (facebook/multi-token-prediction draft to Llama-2-13B target) shows the largest absolute gap between fixed and adaptive.

(b) (b) Throughput of AMTP, relative to the best fixed window, self-speculative pairings and the modest-effect cross-model pairing (self-distilled 4B→14B).

(c) (c) The cross-model Meta MTP pairing (draft/target cost asymmetry), shown on its own axis given the scale of the effect relative to every other condition.
Figure 4. Top: decoding throughput of plain decoding, the best fixed window and AMTP. Every multi-token-prediction condition is faster than plain decoding. Middle: throughput of AMTP relative to the best fixed window, for self-speculative pairings and the self-distilled cross-model pairing — effects are modest in both directions, within about 15% either way. Bottom: the Meta MTP cross-model pairing, shown separately because its gain (+268% AIME, +147% GSM8K) is an order of magnitude larger than every other condition — draft/target cost asymmetry combined with long, variable-length reasoning produces the largest effect in this validation.

Figure 5. Four consecutive real verification rounds (648–651) from one adaptive-control-block run on AIME 2025 (self-distilled Qwen3-4B). Green tokens were accepted, the red token was rejected and replaced by the correction shown on the right (resampled from the residual), and grey tokens were drafted but never verified because the round already stopped at the rejection. The control block re-selects the window every round (shown on the left). All tokens, windows, and corrections are taken directly from logged telemetry.
Example 2: full acceptance with a bonus token. Later in the same run, round 101 is more typical of an easy stretch. The context ends “. . . we want to count”, following directly from the probability setup. The control block forecasts rejection hazards , , for the next three positions, giving a predicted gain expected committed tokens, and the controller selects . Table 10 lists the single round. Since the draft is verified in full with no rejection, the engine draws one additional bonus token directly from the target’s own distribution rather than discarding the round.
| Round | Draft | Accepted | Bonus | |
| 101 | 3 | ▁the ▁number ▁of | 3/3 (all) | ▁non |
Table 10. One real verification round (101) with full acceptance, from the same adaptive-control-block run as Table 9. The bonus token is sampled from the target’s distribution at the position immediately after the draft, since no rejection occurred.
| Round | Draft | Accepted | Correction | |
| 648 | 13 | 2 7 6 8 – _ 4 8 6 4 = _ 2 | 13/13 (all) | — |
| 649 | 12 | 9 0 4 $ $ Yes But _ _ of _of | 4/12 | Yes |
| 650 | 14 | . _So , _$ _number _of 1 _subsets _subsets _with _is _** $ _elements | 1/14 | So |
| 651 | 14 | , _the _number $ _of _≤ $ _the 4 0 4 _with _^ \ | 3/14 | _of |
| 652 | 16 | _non _with B B _have _B (S _are _2 0 ^ $ = 2 2 | 1/16 | empty | ||
| 653 | 8 | _subsets $ $ _⊂ CM _= _ 2 | 3/8 | $ |
| 654 | 7 | _such _that _⊂ _⊂ _S $ | 3/7 | CM |
Table 9. Seven consecutive real verification rounds (648–654) from one adaptive-control-block run. _ marks a leading space in a token; $ and \ are literal draft tokens (the model was mid-LaTeX). All data is taken directly from logged round telemetry.
The committed text for the round is therefore “ the number of non”, four tokens from a single target forward pass, with no correction needed.
References
- [1]Sudhanshu Agrawal, Wonseok Jeon, and Mingu Lee. AdaEDL: Early draft stopping for speculative decoding of large language models via an entropy-based lower bound on token acceptance probability. In Proceedings of the 4th NeurIPS Efficient Natural Language and Speech Processing Workshop, volume 262 of Proceedings of Machine Learning Research, 2024. URL https://proceedings.mlr.press/v262/agrawal24a.html.arxiv.org/abs/2410.18351
- [2]Kwangjun Ahn, Alex Lamb, and John Langford. Efficient joint prediction of multiple future tokens. arXiv preprint arXiv:2503.21801, 2025. URL https://arxiv.org/abs/2503.21801.
- [3]Emile Anand and Sarah Liaw. Feel-good Thompson sampling for contextual bandits: a Markov chain Monte Carlo showdown. In Advances in Neural Information Processing Systems, volume 38, 2025.arxiv.org/abs/2507.15290
- [4]Emile Anand, Abdullah Ateyeh, Archer Wang, and Marin Soljačić. Smat-attention: Structured long-context sequence modeling, 2026. URL https://arxiv.org/abs/2609.36062.
- [5]Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. Hydra: Sequentially-dependent draft heads for medusa decoding, 2024. URL https://arxiv.org/abs/2402.05109.
- [6]Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations Research, 63(5):1227–1244, 2015.
- [7]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5209–5235. PMLR, 2024. URL https://proceedings.mlr.press/v235/cai24i8.html.arxiv.org/abs/2401.10774
- [8]Carrie Chen. EntMTP: Accelerating LLM inference with entropy guided multi token prediction. arXiv preprint arXiv:2606.27550, 2026.arxiv.org/abs/2606.27550
- [9]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023. URL https://arxiv.org/abs/2302.01318.
- [10]Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqì Zhu, Shirong Ma, Xi-aokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huangqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, and Wenfeng Liang. DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv preprint arXiv:2607.05147, 2026. URL https://arxiv.org/abs/2607.05147.
- [11]Sai Sreenivas Chintha, Chidera Biringa, Rishab Sharma, Anusha Agrawal, Prachi Badarayani, Gemma Zhang, Nigel Boachie Kunkamunah, Marco Ayala, and Sulaiman Vesal. Next token or vector? learning how far to decode. arXiv preprint arXiv:2610.22788, 2026.arxiv.org/abs/2610.22788
- [12]DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024.arxiv.org/abs/2412.19437
- [14]Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 15706–15734. PMLR, 2024. URL https://proceedings.mlr.press/v235/gloeckle24a.html.arxiv.org/abs/2404.19737
- [15]Elad Hazan. Introduction to online convex optimization. Foundations and Trends in Optimization, 2(3–4):157–325, 2016.
- [16]Yunlong Hou, Fengzhuo Zhang, Cunxiao Du, Xuan Zhang, Jiachun Pan, Tianyu Pang, Chao Du, Vincent Y. F. Tan, and Zhuoran Yang. BanditSpec: Adaptive speculative decoding via bandit algorithms. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 2025. URL https://proceedings.mlr.press/v267/hou25h.html.
- [17]Kaixuan Huang, Xudong Guo, and Mengdi Wang. SpecDec++: Boosting speculative decoding via adaptive candidate lengths. arXiv preprint arXiv:2405.19715, 2024. URL https://arxiv.org/abs/2405.19715.
- [18]John Kirchenbauer, Abhimanyu Hans, Brian Bartoldson, Micah Goldblum, Ashwinee Panda, and Tom Goldstein. Multi-token prediction via self-distillation. arXiv preprint arXiv:2602.06019, 2026. URL https://arxiv.org/abs/2602.06019.
- [19]John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in Neural Information Processing Systems, volume 20, 2007.
- [20]Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR, 2023. URL https://proceedings.mlr.press/v202/leviathan23a.html.
- [21]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 28528–28948. PMLR, 2024a. URL https://proceedings.mlr.press/v235/li24bt.html.
- [22]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421–7432. Association for Computational Linguistics, 2024b. doi: 10.18653/v1/2024.emnlp-main.422. URL https://aclanthology.org/2024.emnlp-main.422/.
- [23]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. In Advances in Neural Information Processing Systems, 2025.
- [24]Yiheng Lin, Yang Hu, Guannan Qu, Tongxin Li, and Adam Wierman. Bounded-regret MPC via perturbation analysis: Prediction error, constraints, and nonlinearity. In Advances in Neural Information Processing Systems, volume 35, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/eadeef7c51ad86989cc3b311cb49ec89-Abstract-Conference.html.
- [25]Yiheng Lin, James A. Preiss, Emile Anand, Yingying Li, Yisong Yue, and Adam Wierman. Online adaptive policy selection in time-varying systems: No-regret via contractive perturbations. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/a7a7180fe7f82ff98eee0827c5e9c141-Abstract-Conference.html.
- [26]Yiheng Lin, James A. Preiss, Fengze Xie, Emile Anand, Soon-Jo Chung, Yisong Yue, and Adam Wierman. Online policy optimization in unknown nonlinear systems. In Shipra Agrawal and Aaron Roth, editors, Proceedings of Thirty Seventh Conference on Learning Theory, volume 247 of Proceedings of Machine Learning Research, pages 3475–3522. PMLR, 30 Jun–03 Jul 2024. URL https://proceedings.mlr.press/v247/lin24a.html.arxiv.org/abs/2404.13009
- [27]Hongyi Liu, Jiaji Huang, Zhen Jia, Youngsuk Park, and Yu-Xiang Wang. Not-a-bandit: Probably no-regret drafter selection in speculative decoding for LLMs. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2510.20064.
- [28]Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, and Xiao Sun. Pearl: Parallel speculative decoding with adaptive draft length, 2025. URL https://arxiv.org/abs/2408.11850.
- [29]Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 2024.arxiv.org/abs/2310.07177
- [30]Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, and Roy Schwartz. Dynamic speculation lookahead accelerates speculative decoding of large language models. In Proceedings of the 4th NeurIPS Efficient Natural Language and Speech Processing Workshop, volume 262 of Proceedings of Machine Learning Research, pages 456–467. PMLR, 2024. URL https://proceedings.mlr.press/v262/mamou24a.html.arxiv.org/abs/2405.04304
- [31]Yu-Yang Qian, Hao-Cong Wu, Yichao Fu, Hao Zhang, and Peng Zhao. When drafts evolve: Speculative decoding meets online learning. arXiv preprint arXiv:2603.12617, 2026. URL https://arxiv.org/abs/2603.12617.
- [32]Mohammad Samragh, Arnav Kundu, David Harrison, Kumari Nishu, Devang Naik, Minsik Cho, and Mehrdad Farajtabar. Your llm knows the future: Uncovering its multi-token prediction potential, 2025. URL https://arxiv.org/abs/2507.11851.
- [33]Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems, volume 31, 2018.arxiv.org/abs/1811.03115
- [34]Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding, 2024. URL https://arxiv.org/abs/2401.07851.
- [35]Jiebin Zhang, Zhenghan Yu, Liang Wang, Nan Yang, Eugene J. Yu, Zheng Li, Yifan Song, Dawei Zhu, Xingxing Zhang, Furu Wei, and Sujian Li. Learning to draft: Adaptive speculative decoding with reinforcement learning. arXiv preprint arXiv:2603.01639, 2026. URL https://arxiv.org/abs/2603.01639.
- [36]Lijun Zhang, Shiyin Lu, and Zhi-Hua Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems, volume 31, 2018.
- [37]Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning, 2021. URL https://arxiv.org/abs/2110.00871.
- [38]Ziyin Zhang, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Rui Wang, and Zhaopeng Tu. Draft model knows when to stop: Self-verification speculative decoding for long-form generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 16685–16697. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.emnlp-main.844. URL https://aclanthology.org/2025.emnlp-main.844/.
- [39]Guoliang Zhao, Ruobing Xie, An Wang, Shuaipeng Li, Huaibing Xie, and Xingwu Sun. Self-distillation for multi-token prediction. arXiv preprint arXiv:2603.23911, 2026.arxiv.org/abs/2603.23911
- [40]Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning, 2003. URL https://www.martin.zinkevich.org/publications/ICML03.pdf.