WorksComputer Science
Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter
Abstract
Speculative decoding (SD) speeds up large language model (LLM) serving, but its gain depends on the workload's shape, such as batch size and context length, and on its content. A model-based drafter proposes a short block on any content, while a model-free drafter such as n-gram lookup proposes long copies at almost no cost when the output repeats its input, as in code editing. Adaptive verification spends a batch-wide verification budget on the draft tokens most likely to be accepted, but it supports only one model-based drafter. Model-free drafters provide no per-token confidence, so they cannot take part in budget allocation, and deployments must either fix one drafter or split their GPUs between drafters. We present \sys, a framework that runs a model-based and a model-free drafter on every step and gives each request the better of their chains, within adaptive verification and entirely on the GPU, which keeps asynchronous scheduling intact. The key idea is a price per verified draft token: the goodput that slot would earn elsewhere in the batch. Once the model-free drafter's tokens carry calibrated acceptance estimates, fit once on general text, this price puts both drafters' chains on one scale. Instantiated with \dspark and an n-gram drafter in vLLM, \sys achieves 1.10–1.62\x the output throughput of \dspark on code editing at batch size~4 and 1.04–1.27\x at batch size~128, while staying at 0.97–1.01\x of it on chat at temperature~0. A single deployment can therefore serve both workloads.
Introduction
Decoding dominates the cost of serving large language models (LLMs). At each step, the model reads its weights and KV cache to produce one token per request, leaving much of the available compute idle in memory-bound regimes (Pope et al., [36]). Speculative decoding (SD) (Leviathan et al., [22]; Chen et al., [5]) exploits this idle compute: a cheap drafter proposes several tokens, and the target model verifies them in a single forward pass while preserving the output distribution. The benefit depends on how many tokens are accepted relative to the cost of drafting and verification.
This tradeoff changes with both batch size and content. At small batch sizes, verifying additional tokens is relatively cheap, and longer drafts can substantially improve throughput. As the batch grows, verification becomes more expensive, and verifying too many tokens can make SD slower than vanilla decoding (Liu et al., [32]; 2024). Adaptive verification, as in TurboSpec (Liu et al., [31]), DSpark (Cheng et al., [7]), and D-cut (Liu et al., [30]), addresses this problem by adjusting the verification budget at each step and allocating it to draft tokens most likely to be accepted. The budget also responds to content: low-entropy domains such as coding and math tend to accept longer drafts than roleplay or creative writing (Abramovich et al., [1]).
Yet adaptive verification assumes that the drafter has already been chosen. It decides how much of a draft to verify, but not which drafter should produce it. This distinction matters because different drafters exploit different kinds of predictability. Model-based drafters learn to predict the target’s continuation and work across general text. Model-free drafters instead exploit repetition, copying tokens from the prompt or previously generated text. The best drafter therefore depends not only on how predictable the next tokens are, but also on where that predictability comes from.
Code editing makes this difference particularly clear. Given an existing program and a request to modify it, the model often reproduces large portions of the original code, changing only a few regions. A model-based drafter can predict these tokens, but its draft length is limited. An n-gram drafter can instead copy much longer continuations directly from the prompt, at little cost. On the NextCoder code-editing workload with Qwen3-4B at batch size (BS) 4, DSpark, which drafts seven tokens per step, achieves 4.17–5.00× speedup over vanilla decoding across edit distances from 0% to 70%. An n-gram drafter that proposes up to 15 tokens achieves 7.42× speedup when the output is nearly identical to the input, but falls to 1.80× on heavy rewrites (Figure 1). Neither drafter is uniformly better: copying wins when the output repeats the input, while model-based drafting is more robust when the output changes substantially.
Figure 1. The best drafter depends on content, and Hybrid Spark tracks the better one. Output-throughput speedup over vanilla decoding on NextCoder (Qwen3-4B), by edit distance between the input program and the model’s output (higher is better). The n-gram drafter outperforms DSpark only on near-copies, and verifying all of its drafts (n-gram) at BS 128 is slower than vanilla decoding from 35% edit distance up. N-gram + AV gives it adaptive verification with our estimator. Table 11 in the appendix lists the values.
| Edit distance | BS 4 DSpark | BS 4 n-gram + AV | BS 4 Hybrid Spark | BS 128 DSpark | BS 128 n-gram | BS 128 n-gram + AV | BS 128 Hybrid Spark |
| 0–5% | 5.00 | 7.42 | 7.98 | 2.88 | 3.14 | 3.68 | 3.57 |
| 5–10% | 4.87 | 5.07 | 7.11 | 2.69 | 2.20 | 3.00 | 3.30 |
| 10–15% | 4.78 | 4.06 | 6.52 | 2.48 | 1.60 | 2.54 | 2.84 |
| 15–20% | 4.61 | 3.31 | 5.96 | 2.49 | 1.47 | 2.51 | 2.91 |
| 20–25% | 4.54 | 2.88 | 5.65 | 2.35 | 1.12 | 1.99 | 2.54 |
| 25–30% | 4.53 | 2.64 | 5.43 | 2.41 | 1.11 | 2.06 | 2.65 |
| 30–35% | 4.42 | 2.34 | 5.09 | 2.39 | 1.00 | 1.96 | 2.57 |
| 35–40% | 4.40 | 2.22 | 5.03 | 2.45 | 0.95 | 1.88 | 2.58 |
| 40–45% | 4.29 | 2.10 | 4.83 | 2.29 | 0.84 | 1.73 | 2.42 |
| 45–50% | 4.34 | 2.06 | 4.87 | 2.50 | 0.89 | 1.80 | 2.63 |
| 50–55% | 4.32 | 1.97 | 4.81 | 2.27 | 0.81 | 1.72 | 2.50 |
| 55–60% | 4.17 | 1.88 | 4.53 | 2.37 | 0.78 | 1.63 | 2.50 |
| 60–65% | 4.30 | 1.89 | 4.66 | 2.48 | 0.79 | 1.69 | 2.53 |
| 65–70% | 4.47 | 1.80 | 4.75 | 2.68 | 0.83 | 1.74 | 2.86 |
| Pooled | 4.53 | 2.70 | 5.52 | 2.49 | 1.07 | 2.04 | 2.75 |
Table 11. Output-throughput speedup over vanilla decoding on NextCoder by edit-distance bucket (Qwen3-4B, mean of 3 repeats; higher is better), the values plotted in Figure 1. n-gram: every draft verified, plotted at BS 128 only. The last row pools all buckets and is not plotted.
This creates a practical problem for serving systems. A deployment serving both editing and general requests must either choose one drafter for all traffic or maintain separate deployments for different workloads. The former leaves performance on the table: using DSpark everywhere misses the gains from copying on light edits. The latter fragments GPU capacity across pools whose traffic varies over time, requiring each to be provisioned for its own peak. Ideally, a single deployment would use the better drafter for each request, and adapt that choice as generation progresses.
Combining drafters, however, is not as simple as running both and taking the longer draft. The two drafters produce different kinds of predictions, compete for the same verification budget, and must be compared at every decoding step without introducing overhead. This raises three challenges:
(C1) Estimating acceptance for model-free drafts. Model-based drafters provide token confidences, but an n-gram match only identifies a possible continuation, not how likely the target is to accept it. Without acceptance estimates, model-free drafts cannot participate in adaptive verification or be compared with model-based drafts.
(C2) Choosing between competing draft chains. Each request now has two candidate chains that may agree, diverge, or differ entirely. Verifying both requires tree attention or increases verification cost, while choosing one requires a common scale and a price that reflects the rest of the batch. Existing
hybrids [35] select the model-free chain when its score exceeds a fixed threshold, ignoring both the changing cost of verification and what other requests could gain from the same slots.
(C3) Making the decision practical at serving scale. Drafter selection must run at every decoding step, entirely on the GPU and at negligible cost, without host-device synchronization. Otherwise, it undermines asynchronous scheduling and CUDA graph execution. Existing CPU-based hybrid approaches [35] incur this synchronization. The system must also preserve the target model’s output distribution.
Our key observation is that adaptive verification already prices each draft token: a verified slot costs the marginal goodput it could earn elsewhere in the batch. If every proposed token carries an acceptance estimate on a common scale, chains from different drafters can be compared by their expected benefit at this price, which makes drafter selection and budgeting one decision. Building on this observation, Hybrid Spark combines a model-based and a model-free drafter within one adaptive-verification system. The model-free drafter receives calibrated acceptance estimates from cheap features, fit once by replaying target-model outputs on general text. At each step, Hybrid Spark selects a chain for each request at the common price and allocates the batch’s budget across the selected chains, all on the GPU and without changing the output distribution.
We instantiate Hybrid Spark with DSpark and a GPU n-gram drafter in vLLM [21], and evaluate it across five models, seven workloads, and three batch sizes. Across five code-editing workloads, Hybrid Spark achieves 1.10–1.62× the output throughput of DSpark at BS 4 and 1.04–1.27× at BS 128, while remaining at 0.97–1.01× of DSpark on chat at . The gains follow how much of the output can be reused: on NextCoder, light edits (15–20% edit distance) achieve 1.28–1.33× at BS 4, while heavy edits (60–70%) achieve 1.03–1.12× (Figure 5). A single deployment thus gains from copying when the output repeats its input and remains robust on general text.
Figure 5. Hybrid Spark raises output throughput over DSpark on almost every editing cell, and the gain shrinks with batch size and with edit distance. On chat (MT-Bench), it stays within 0.97–1.01. Speedup in output throughput (higher is better), mean of 3 repeats. First five panels: by batch size, with 95% CIs. Last three panels: NextCoder per edit-distance bucket. Missing points are buckets with fewer than 10 samples.
The same split between learned prediction and reuse appears in reinforcement learning (RL) post-training. Rollout generation dominates training time, and generations from the same prompt often overlap across adjacent epochs [27, 15]. As short rollouts finish, a few long generations at small effective batch size determine step time (Shao et al., 2026), which is where cheap copying helps most. History-based drafters exploit this overlap [15] (Shao et al. 2026), while a model-based drafter can go stale as the policy changes (Shao et al., 2026). Hybrid Spark’s model-free slot can hold such a drafter, and we leave RL rollout generation to future work.
Our contributions are: (1) a characterization of how drafter choice depends on content and batch size (§ 3); (2) a recipe for calibrating model-free drafts using offline replay and a per-position logistic estimator, instantiated for n-gram drafting and shown to transfer across datasets and models (§ 4.2 and § 5.5); (3) a drafter-agnostic chain-selection rule based on the marginal goodput of verification slots, integrated with adaptive verification entirely on the GPU, including an analysis of stale budget sizing (§ 4.3, § 4.4 and § 5.7); and (4) lossless alternative policies for probabilistic drafting and an evaluation of the DSpark + n-gram instantiation across models, workloads, batch sizes, and temperatures (§ 4.5, § 5 and H).
Background
Speculative decoding and goodput. In each step, a drafter proposes a draft chain of up to tokens per request. The target model scores all of them in one forward pass and accepts the longest prefix that passes rejection sampling [22, 5], plus one token of its own. The acceptance length (AL) is the mean number of tokens emitted per request per step. We write for the conditional acceptance rate of draft position given that positions were accepted, and for the survival of position , the probability that it is accepted. Goodput is the expected number of tokens emitted per unit of step time. Output throughput, our end-to-end metric, is total generated tokens over the duration of a benchmark run.
Drafters. Model-based drafters run a neural network, a small model or heads on the target (§ 6). DSpark ([7]) drafts a -token block semi-autoregressively and has a confidence head that predicts each draft token’s conditional acceptance. Model-free drafters (§ 6) copy continuations from the context or a datastore, such as earlier rollouts of the same prompt in RL post-training ([15] (Shao et al. 2026)). They cost almost nothing to run and can draft long chains when the output copies its input, but they give no calibrated confidence. Our instantiation uses the simplest of them, an n-gram match against the request’s own tokens.
Adaptive verification (AV). Verifying one more draft token, a slot, is nearly free at small batch size but costly at large batch size, once the verification pass becomes compute bound ([32]). With long contexts, loading the KV cache keeps large batches memory bound longer, so SD can still help there ([37]). DSpark’s adaptive verification (AV) treats all slots in the batch as one verification budget . It models the step cost from profiled tables and the expected emitted tokens for requests, and chooses the budget that maximizes goodput . It gives the slots to the (request, position) pairs with the highest survival, so each request keeps a prefix of its chain.
Asynchronous scheduling. Modern engines overlap the host’s preparation of step with the GPU’s execution of step . The host must fix the number of tokens in the next verification pass before the GPU has produced the confidences for it. DSpark therefore sizes the budget on the host from confidences produced two steps prior, one step older than the ones the GPU uses to allocate it ([7 §5.2]). Any per-step decision that Hybrid Spark adds must respect this split: a host read of a GPU value mid-step would serialize the pipeline.
Motivation
We measure the two drafters on Qwen3-4B with NextCoder code edits ([2]), bucketed by normalized edit distance between the input program and the model’s output, and on the general-text SPEED-Bench ([1]). § 5.1 gives details.
Observation 1: The better drafter depends on the request’s content. Figure 1 shows that with AV, the n-gram drafter outperforms DSpark at BS 4 only below 10% edit distance, and above that DSpark’s speedup is up to 2.5 times the n-gram drafter’s. Within a request, the drafter that accepts more also alternates from step to step: Liu et al. [32] show this for an n-gram drafter and EAGLE, and an oracle that picks the better drafter per step gains beyond either alone ([32], Figs. 11 and 12). This motivates choosing a drafter per request and per step (§ 4.3).
Observation 2: N-gram drafts are weakest at the first token and strongest deep in the chain. SD analyses often summarize a drafter by one acceptance rate ([22]), but the n-gram drafter does not fit this summary (Figure 2). Its first token is its least reliable, because a match shows only that the context repeats, not that the output will copy it. Given a match, this token is accepted at 0.88 on
Figure 2. The two drafters have opposite acceptance profiles. Conditional acceptance by draft position (Qwen3-4B; higher is better). Given a match, the n-gram drafter’s first token is its weakest and its deep tokens are accepted more often than any of DSpark’s. The n-gram curve counts only steps with a match.
NextCoder (0.76 on SPEED-Bench). Once it is accepted, the output has started to copy the context and tends to continue: acceptance rises to 0.97 by the fifteenth token (0.96), and positions 8–15, which DSpark never drafts, are accepted at 0.96–0.97 (0.92–0.96). DSpark’s acceptance is flatter, at 0.92–0.96 (0.69–0.74). Because AV ranks individual slots, these per-position differences determine which slots it allocates. The estimator therefore predicts acceptance per position, with separate terms for the first token (§ 4.2). When the n-gram chain begins with DSpark’s tokens, Hybrid Spark keeps its tail beyond DSpark’s seventh token, the most reliable tokens in either chain (§ 4.3).
Observation 3: The n-gram drafter needs the budget at large batch size but loses throughput to it at small batch size. Verifying every n-gram draft token is cheap at BS 4, but at BS 128 it is slower than vanilla decoding on every NextCoder bucket from 35% edit distance up, down to 0.78× (Figure 1). Giving the same drafter AV with our estimator (§ 4.2) yields a 1.90× output-throughput speedup over verifying every draft on pooled NextCoder at BS 128, from 1.17× at 0–5% to 2.15× at 60–65%. At BS 4, where verification is nearly free, the budget has little to gain, and the same AV costs the n-gram drafter 4.2% throughput. One cause is that the budget is sized from estimates produced two steps prior (§ 2). Because n-gram matches appear and vanish from step to step, these estimates are often wrong. Against a budget sized from fresh survivals, this loses 7–9% of predicted goodput at BS 4 (§ 5.7 and O). The n-gram drafter must therefore join the shared budget with calibrated confidences (§ 4.2 and § 4.4), and Hybrid Spark must keep the cost of stale sizing small at small batch size (§ 5.7).
Algorithm 1 One Hybrid Spark drafting step with DSpark and an n-gram drafter.
Require: context , DSpark chain with confidences , coefficients ; host scalars and
for all requests in parallel do
if and then
keep ; for , then continue with
else if matched and then
keep with
else
keep with
end if
end for
AV: give the top- (request, position) slots by ; each request keeps a prefix of its chain
copy to the host asynchronously for the next budget sizing
Design
Overview
Hybrid Spark addresses the three challenges through three design choices (Figure 3). (C1) It estimates acceptance probabilities for the model-free drafter, putting both chains on a common scale of prefix survival probabilities (§ 4.2). (C2) It selects one chain per request by comparing their expected value against the opportunity cost of verification slots elsewhere in the batch (§ 4.3). (C3) It makes all token-dependent decisions on the GPU, using budget parameters from the previous step to avoid host-device synchronization while preserving lossless decoding (§ 4.4).

Figure 3. Hybrid Spark overview, instantiated with DSpark and an n-gram drafter. Every per-step decision runs on the GPU (blue), while the CPU host (orange) sizes the budget and sets the slot price from survivals produced two steps prior. Solid arrows stay on the GPU, and dashed arrows carry host scalars or asynchronous copies.
Interfaces. Hybrid Spark requires each drafter to provide a candidate chain and per-position acceptance estimates. The model-based drafter returns a chain with calibrated confidences , while the model-free drafter returns a chain and features from which an estimator predicts acceptance probabilities . The model-free chain is fixed before verification and independent of the model-based drafter’s sampled tokens. Both drafters run on the GPU without a host read. Selection, budget allocation, and lossless policies depend only on , , , and , so they do not change when either drafter is replaced.
One step. Algorithm 1 illustrates one Hybrid Spark step. DSpark proposes tokens with confidences , while the n-gram drafter proposes tokens following the longest earlier occurrence of the request’s last 5–15 tokens (line 2). The estimator predicts acceptance probabilities (line 3), and both chains’ prefix survivals are computed as running products of their estimates. Hybrid Spark selects the chain with the larger net value at slot price (lines 5–7), and AV allocates the verification slots across the selected chains by survival (line 13). All decisions run on the GPU, and the host computes and from survivals produced two steps prior (line 14), so no mid-step synchronization is needed.
Beyond the evaluated DSpark and n-gram pair, the interface admits datastore-based drafters, such as the suffix trees of SuffixDecoding ([35]) or suffix trees over earlier RL rollouts ((Shao et al., 2026)), and more than two drafters, which we leave to future work. Table 2 in the appendix summarizes the notation.
| Symbol | Meaning |
| batch size (number of running requests) | |
| , | DSpark chain length (7); maximum n-gram chain length (15) |
| , , | n-gram match length (5–15); runner-up agreement (0/1); copy length (capped at 256) |
| , | conditional acceptance of position : DSpark’s confidence; Hybrid Spark’s n-gram estimate |
| survival, of the conditional acceptances | |
| , | verification budget (slots per step); budget chosen by AV |
| , , | step cost; base cost; slot cost |
| , | expected emitted tokens; goodput |
| slot price | |
| chain value |
Table 2. Notation.
Calibrated acceptance estimates for n-gram drafts
Challenge. Unlike model-based drafters, n-gram drafters produce tokens without confidence scores (C1). AV ranks slots across requests and drafters by survival and sizes the budget from it, so an n-gram draft needs a calibrated survival estimate to take part. We obtain these estimates from features already available during n-gram matching.
Mechanism. We predict the conditional acceptance probability at each draft position using a logistic regression:
where is the logistic function, distinguishes the first token from later tokens, and gives the prefix survival probability.
The estimator has one intercept per draft position, which captures the rising acceptance profile of Observation 2. It also uses four match features:
Match length .
Agreement , which is 1 if the runner-up match predicts the same next token or there is no runner-up. Agreement matters because two matches that continue differently leave open which one the output copies.
Full-match bonus , applied when the match reaches the 15-token cap.
Copy length , how far a full match extends backwards, capped at 256 tokens to bound the search.
Match length, the full-match bonus and copy length each have separate first-token and later-token coefficients (, and , indexed by ).
As Figure 4 shows, full matches and long copies improve acceptance most strongly at the first position. When a full match extends beyond 30 tokens, acceptance exceeds 99% at every position. For shorter copies, first-token acceptance is 86%, rising to 89–96% at later positions. The scan already computes , , and , so the estimate adds only one fused elementwise step per request.
Figure 4. Full matches and long copies raise acceptance most on the first token. Conditional acceptance by draft position, grouped by match length and, for full matches, copy length (Qwen3-4B, the half of SPEED-Bench the estimator is fit on; higher is better).
Fitting once on general data. We fit the estimator once per target model using offline acceptance labels from greedy decoding. Since the target output is deterministic at , replaying each generated sequence and rerunning the n-gram scan on its prefixes gives exact acceptance labels without online verification. This approach applies to any model-free drafter whose proposals can be reconstructed from the context and datastore.
We generate up to 512 tokens for 1,530 prompts from SPEED-Bench’s throughput split [1], using an 8k-token input limit. Each of the three entropy categories is split approximately in half, yielding 764 fitting prompts and 766 held-out prompts. We fit the regression using Newton’s method with ridge regularization and select the logarithmic copy-length feature using held-out log-loss. None of the editing benchmarks (Blazedit, CanItEdit, NextCoder, EditBench) are used for fitting or feature selection, so all editing results are out of sample. Section D reports the coefficients and fitting data.
Choosing one chain per request under one budget
Challenge. Each request has two candidate chains, but verifying both requires tree attention or extra slots (C2). Choosing one must account for the rest of the batch: a slot spent on one request cannot be used by another. Two simple policies fall short (§ 5.4): always preferring an n-gram match discards DSpark’s chain even for unreliable matches, while choosing the chain with more expected accepted tokens, , favors long chains with low-value tails.
Slot price. We assign each verification slot a price based on its opportunity cost elsewhere in the batch. AV selects the budget that maximizes goodput , where is the expected number of emitted tokens and is the step cost with slot cost (Section F). Adding a slot with survival changes goodput by
where . Thus, a slot improves goodput only when , with . We use this single slot price across the batch. The price rises with batch size (Section G), so chain selection matters most at BS 128.
Chain value. We score each chain by the expected benefit of its slots after subtracting their price:
Hybrid Spark selects the chain with larger , breaking ties in favor of the n-gram chain (line 7). Since survival decreases along a chain, the slots with form a prefix, matching the slots AV would verify. Unlike , this value ignores low-survival tail slots, so a longer chain wins only when its extra slots justify their cost (Section G).
Contains rule. When the n-gram chain is longer and begins with all tokens from DSpark, it is accepted at least as far as DSpark’s chain on every target draw. We therefore keep the n-gram chain and merge their survival estimates: the first positions take the larger survival, and the rest use the n-gram estimates (lines 5 and 6). This combines DSpark’s stronger estimates at the start with the n-gram tail, where the n-gram drafter is strongest (Observation 2).
Sharing the verification budget
Challenge. AV assumes fixed-length drafts from one drafter. With Hybrid Spark, chain lengths and survival curves vary across requests and steps, while selection and allocation must remain on the GPU without host synchronization (C3).
Mechanism. Hybrid Spark keeps AV’s budget sizing and allocation rules unchanged. The selection kernel writes the chosen chains, their valid lengths, and merged survivals into fixed-size GPU buffers. AV allocates verification slots and constructs the verification layout directly on the GPU, supporting variable-length chains without a host round trip.
The host computes the next budget and slot price from survivals produced two steps prior, one step older than the ones the GPU allocates from (line 14). Both are passed to the GPU as scalar launch parameters, avoiding synchronization within the current step. Section F describes the cost model.
Losslessness
Hybrid Spark is lossless at any target sampling temperature when drafting is greedy, but its value-based selection is not generally lossless with probabilistic drafting. With greedy drafting, both chains and all selection and allocation decisions depend only on the context, and the target verifies each draft token as a point mass. The same argument applies to D-cut pruning [30 App. A].
With probabilistic drafting, the contains and value rules depend on DSpark’s sampled tokens. Whether a slot is verified can then depend on the token sampled there, biasing the output distribution [30 App. B].
Two simpler policies remain lossless: n-gram-first, which selects the n-gram chain whenever a match exists, and extension, which verifies DSpark’s chain unchanged and appends the n-gram tail only when the entire DSpark chain is its prefix. Section H provides the arguments and summarizes the guarantees in Table 9.
| Target | Drafting | Proposed | N-gram-first | Extension |
| any | yes | yes | yes | |
| greedy | yes | yes | yes | |
| probabilistic | no | yes | yes |
Table 9. Where each chain-selection policy preserves the target’s output distribution exactly.
Evaluation
We evaluate Hybrid Spark along seven questions: (Q1) How much does it improve editing, and how does the gain change with batch size and edit distance? (Q2) Does it hurt performance on other workloads? (Q3) Would a simpler policy perform as well? (Q4) Which components matter, and does the estimator transfer across settings? (Q5) What is the runtime overhead of the additional drafter? (Q6) How much goodput is lost by sizing the budget from stale survival estimates? (Q7) How does it behave at ?
Section § 16 separately examines models and contexts where DSpark’s drafter is weak.
Setup
Implementation, hardware and serving. We evaluate the instantiation of § 4 (DSpark and the n-gram drafter), implemented in vLLM and referred to as Hybrid Spark below. Every arm runs on vLLM’s V1 engine with Model Runner V2 (MRv2). All runs use NVIDIA H100 80 GB (HBM3) GPUs. BS 4 and 32 run on one GPU and BS 128 on two or four GPUs with tensor parallelism, with the same layout for every arm of a comparison. All runs enable asynchronous scheduling and disable prefix caching. Every arm that uses AV enables it at every batch size.
Models and drafters. We evaluate five target models, each with its own DSpark drafter (): Qwen3-4B and Qwen3-8B (Yang et al., 2025), Gemma 4 12B [11], North-Mini-Code-1.0 [8], a 30B-total, 3B-active mixture-of-experts code model served with 4-bit weights, and Llama-3.1-8B-Instruct [13]. Table 3 lists the checkpoints. The n-gram drafter matches 5–15 tokens (longest match wins) and drafts up to 15. Each model uses its own estimator coefficients, fit as in § 4.2.
| Target model | DSpark drafter |
Qwen/Qwen3-4B | deepseek-ai/dspark_qwen3_4b_block7 |
Qwen/Qwen3-8B | deepseek-ai/dspark_qwen3_8b_block7 |
google/gemma-4-12B-it | deepseek-ai/dspark_gemma4_12b_block7† |
CohereLabs/North-Mini-Code-1.0-w4a16 | sdougbrown/North-Mini-Code-1.0-dspark |
meta-llama/Llama-3.1-8B-Instruct | weifanjiang/llama3.1-8b-it.speculators.dspark-ce01tv09-bs8-conf-swa |
Table 3. Target models and their DSpark drafters (). †Loaded with two rope_parameters keys removed from its configuration, which the installed Transformers version rejects. The weights are unchanged.
Workloads. Table 4 in Section C lists their sizes, prompts per run and output limits. For NextCoder and EditBench (Chi et al., 2026), we bucket requests by the normalized Levenshtein distance between the input program and the longest code block of the target model’s own greedy output without SD, in 5% buckets. We keep inputs above 1,000 characters and buckets with at least 10 samples. Buckets are therefore model specific. EditBench requests with more than 6,000 tokens of context (prompt plus output) are out of distribution for the DSpark drafters and are excluded from the main results (§ 16). Blazedit 5k and 10k [9] and CanItEdit [4] keep inputs above 1,000 characters with reference edit distance 5–25%. We use InstructCoder [23] as a mixed workload and MT-Bench (Zheng et al., 2023) first turns as general chat.
| Workload | Size | Prompts per run | Max out |
| Blazedit | 5k / 10k | 21 / 19 | 64 / 256 / 512 |
| CanItEdit | 30 | 64 / 256 / 512 per prompt | |
| NextCoder | 4,000 | 14 buckets, 0–70% | 4,096 |
| EditBench | 520 | 5 buckets, 0–25% | 4,096 |
| InstructCoder | 3,200 | 200 / 800 / 3,200 | 200 |
| MT-Bench | 80 | 160 / 640 / 1,280 | 256 |
Table 4. Workloads. Prompts per run at BS 4/32/128; small sets are oversampled to that count. Bucketed sets run each bucket with at least that many prompts.
Metric and statistics. We use vllm bench serve with the chat-completions API, a temperature of 0 unless otherwise stated, and client concurrency equal to the batch size (BS). Output throughput is the total number of output tokens divided by the run duration. Speedup is Hybrid Spark’s output throughput divided by the baseline’s, so values above 1 indicate that Hybrid Spark is faster. For bucketed workloads, we pool throughput across buckets: total output tokens divided by the total duration, treating the buckets as if they ran back to back. Each cell reports the mean over 3 independent runs (5 for the ablation), each launched on a fresh server. We report 95% confidence intervals computed by bootstrap resampling over the repeats. If a speedup’s confidence interval includes 1, we call the two systems statistically indistinguishable, or level.
Baselines. (1) Vanilla decoding. (2) DSpark with its own AV and the released block size , the primary baseline. (3) The n-gram drafter with AV, using Hybrid Spark’s estimator for its confidences (n-gram + AV). (4) The n-gram drafter with every draft verified. (5) Simpler hybrid policies and ablations in § 5.4 and 5.5.
Code editing (Q1)
Hybrid Spark’s gain is largest at small batch size and shrinks at large batch size. Across the five editing workloads and the three in-distribution models, Hybrid Spark’s speedup over DSpark is at BS 4 and at BS 128 (Figure 5; Table 12 in the appendix lists every cell). Every one of these speedups has a CI that excludes 1, and the only level editing cell is Gemma 4 12B on CanItEdit at BS 32. NextCoder gains least because it includes buckets up to 70% edit distance, where little of the output is copied.
Table 12. Output-throughput speedup of Hybrid Spark over DSpark (mean of 3 repeats; ◦: 95% CI includes 1, level). NextCoder and EditBench are pooled over their edit-distance buckets; EditBench excludes requests with more than 6,000 tokens of context. North-Mini-Code’s DSpark drafter is weak (§ 16), so it is not part of the headline ranges.
| Model | BS | Blazedit 5k | Blazedit 10k | CanItEdit | NextCoder | EditBench | InstructCoder | MT-Bench |
| Qwen3-4B | 4 | 1.34 | 1.50 | 1.35 | 1.24 | 1.62 | 1.14 | 0.99◦ |
| Qwen3-4B | 32 | 1.16 | 1.31 | 1.16 | 1.11 | 1.37 | 1.06 | 0.99 |
| Qwen3-4B | 128 | 1.13 | 1.26 | 1.12 | 1.10 | 1.27 | 0.99 | 0.98 |
| Qwen3-8B | 4 | 1.31 | 1.45 | 1.33 | 1.19 | 1.59 | 1.11 | 1.00◦ |
| Qwen3-8B | 32 | 1.13 | 1.28 | 1.11 | 1.08 | 1.26 | 1.02 | 1.00◦ |
| Qwen3-8B | 128 | 1.08 | 1.24 | 1.08 | 1.05 | 1.19 | 0.98 | 0.99 |
| Gemma 4 12B | 4 | 1.30 | 1.40 | 1.26 | 1.10 | 1.52 | 1.08 | 1.01 |
| Gemma 4 12B | 32 | 1.16 | 1.25 | 1.02◦ | 1.03 | 1.15 | 1.03 | 1.01 |
| Gemma 4 12B | 128 | 1.08 | 1.13 | 1.05 | 1.04 | 1.12 | 0.98 | 1.00◦ |
| North-Mini-Code | 4 | 1.92 | 2.50 | 1.56 | 1.28 | 2.69 | 1.16 | 0.99 |
| North-Mini-Code | 32 | 1.68 | 2.06 | 1.34 | 1.28 | 2.05 | 1.10 | 1.00◦ |
| North-Mini-Code | 128 | 1.44 | 1.62 | 1.27 | 1.23 | 1.69 | 1.15 | 1.00◦ |
Table 12.
Hybrid Spark’s gains are largest where DSpark’s drafter is weak: on North-Mini-Code (1.28–2.50 at BS 4), on Llama-3.1-8B, and on EditBench requests with more than 6,000 context tokens (up to 7.58). There DSpark’s drafter accepts few tokens per step, which leaves more room for the n-gram drafter. A better-trained drafter would likely reduce these gains, so we report them separately in § 16.
The gain comes from longer accepted chains. On Blazedit and CanItEdit at BS 4, DSpark accepts 6.1–6.9 tokens per step for the three models, and Hybrid Spark 8.4–10.0 (Figure 7 in the appendix shows AL per model and workload). The gain shrinks with batch size because grows, so fewer n-gram tail slots are admitted to the budget.
Figure 7. Hybrid Spark accepts more tokens per step than DSpark on every editing workload and the same on MT-Bench. Acceptance length (tokens per verification step, higher is better) at BS 4, mean of 3 repeats.
The gain tracks how much the output copies. The last three panels of Figure 5 break NextCoder down by edit distance: as less of the output is copied from the input, the gain falls. At BS 4 it falls from 1.61–1.63 for the two Qwen3 models at 0–5% to 1.03–1.12 at 60–70%, and at BS 128 to 1.01–1.07. EditBench shows the same trend (Figure 8 in the appendix).
Figure 8. Output-throughput speedup of Hybrid Spark over DSpark per EditBench bucket, without requests over 6,000 tokens of context (higher is better). Mean of 3 repeats.
Workloads where the n-gram drafter rarely helps (Q2)
Hybrid Spark is 0.97–1.01 of DSpark on chat. On MT-Bench at with greedy drafting, this range covers every model and batch size (Figure 5 and Table 1). AL is the same for both arms to within 0.1 (Figure 7) because the n-gram drafter rarely finds a match worth its price. On InstructCoder, where outputs partly copy short inputs, Hybrid Spark achieves 1.08–1.14 at BS 4 and 1.02–1.06 at BS 32, and is 0.98–0.99 at BS 128 (Gemma 4 12B: 0.98 [0.98, 0.99]).
| BS | Policy | NextCoder 15–20% T=0 | NextCoder 15–20% T=1 | NextCoder 40–45% T=0 | NextCoder 40–45% T=1 | MT-Bench T=0 | MT-Bench T=1 |
| 4 | Proposed | 1.30 | 1.28 | 1.13 | 1.13 | 0.99 | 0.98 |
| 4 | N-gram-first | 1.27 | 1.23 | 1.07 | 1.06 | 0.98 | 1.01 |
| 4 | Extension | 1.27 | 1.22 | 1.13 | 1.10 | 1.00° | 1.02 |
| 4 | DSpark (prob.) | 0.98 | 0.97 | 1.03 | |||
| 128 | Proposed | 1.16 | 1.18 | 1.05 | 1.07 | 0.97 | 0.97 |
| 128 | N-gram-first | 1.10 | 1.09 | 0.98 | 0.96° | 0.97 | 0.96 |
| 128 | Extension | 1.11 | 1.07 | 1.06 | 1.02° | 0.95 | 0.98° |
| 128 | DSpark (prob.) | 0.96° | 0.98° | 0.99° |
Table 1. Output-throughput speedup of each policy over DSpark with greedy drafting at the same temperature (Qwen3-4B, 3 repeats; °: CI includes 1). T=1 shows each policy where it is lossless: the proposed policy with greedy drafting, the others with probabilistic drafting.
Simpler policies (Q3)
The n-gram drafter alone is not a substitute. With AV, the n-gram drafter reaches 2.70 over vanilla decoding on pooled NextCoder at BS 4 and 2.04 at BS 128 (Qwen3-4B, Figure 1). Hybrid Spark reaches 5.52 and 2.75, a speedup of 2.04 [2.02, 2.07] and 1.34 [1.33, 1.36] over the n-gram drafter.
Always taking a matching n-gram chain gives up much of the gain at large batch size. Table 1 compares the proposed policy with n-gram-first and the extension policy on two NextCoder buckets and MT-Bench. At , n-gram-first’s throughput is 2.5–5.3% lower than the proposed policy’s at BS 4 and 5.1–7.3% lower at BS 128, and on 40–45% at BS 128 it is slower than DSpark alone (0.98). Its AL there is 6.5, against 7.2–7.4 for the proposed and extension policies and 6.1 for DSpark: it discards DSpark’s chain on short, unreliable matches.
Both the slot price and the contains rule improve throughput. In the ablation in § 5.5, replacing the value rule with the expected accepted length, , reduces throughput by 1.0%, and removing the contains rule costs 2.4%, the largest single effect (Figure 6). Without the contains rule, AL falls from 7.4 to 6.9, and the share of requests on the n-gram chain falls from 26.5% to 15.5%.
Figure 6. The contains rule and the slot price matter most for throughput, and copy length matters most for calibration. Left: output throughput with one component removed, relative to full Hybrid Spark (Qwen3-4B, NextCoder 40–45%, BS 128, 5 repeats, 95% CIs; higher is better). Right: increase in log-loss of the estimator per matched step when the feature is removed (lower is better). The contains rule and the slot price are selection rules, not estimator features, so calibration does not apply to them (n/a).
Estimator features and transfer (Q4)
No single estimator feature matters much for throughput, but together they do. We remove one component at a time from Hybrid Spark (Qwen3-4B). On NextCoder 40–45% at BS 128 (Figure 6, left), besides the two selection rules above, only copy length, the first-token terms and match length have a measurable effect (0.6–1.1%). In three more cells, every ablation stays within 0.992–1.003 of full Hybrid Spark. A small leave-one-out cost does not make a feature redundant: the match features overlap, so the refit shifts weight onto the rest. The agreement term shows no leave-one-out effect, yet the first token is accepted 24% of the time when the two matches disagree, against 81% otherwise. Removing all match features at once from the n-gram drafter with AV costs it 4.1% (0.959× [0.951, 0.967]) on pooled NextCoder at BS 128, whereas removing any one has no measurable effect. Adding terms one at a time (Section E) shows that most of the gain comes from giving the n-gram drafter any calibrated survival: with one shared intercept and no match features, it reaches 0.973× of the full estimator’s throughput on two NextCoder buckets at BS 128. Copy length closes most of the rest. AL is unchanged, so the estimator helps by choosing which slots to verify.
Calibration differs (Figure 6, right). Copy length dominates on EditBench, where long copies are common: without it, the expected calibration error (ECE) ([14]) rises from 0.032 to 0.132. On held-out SPEED-Bench, the position intercepts matter most (ECE 0.012 to 0.029).
The estimator transfers across datasets and models. The estimator is calibrated only on SPEED-Bench, a general-text benchmark, so all editing results are out of sample. Even so, its calibration error on EditBench is only 0.032. It also transfers across models. Serving Qwen3-4B with coefficients fitted on Qwen3-8B or Llama-3.1-8B changes pooled NextCoder throughput by at most 0.7% at any batch size (0.996–1.007× relative to model-specific coefficients), and AL by at most 0.08 in every bucket.
Overhead (Q5)
Running the n-gram drafter alongside DSpark costs at most 1.2%. To isolate this overhead, we configure Hybrid Spark never to select the n-gram chain, so it performs the scan, estimation and selection but verifies exactly DSpark’s chains. Its pooled throughput relative to DSpark at BS 128 is 0.988× on both NextCoder and EditBench for Qwen3-4B, and 0.993× and 0.996× for Qwen3-8B.
Stale budget sizing (Q6)
Sizing the budget from stale survivals costs at most 0.64% of goodput at BS 128. We compare the goodput of the budget sized from stale survivals with that of a budget sized from fresh ones (Section O). Under asynchronous scheduling, the host sizes the budget from survivals produced two steps prior, one step older than the fresh ones the GPU allocates it from (§ 2). This split is DSpark’s design (Cheng et al. [7], §5.2), and Hybrid Spark keeps it. The n-gram drafter suffers more from it than DSpark, because a match appears and vanishes abruptly as the output enters and leaves a region it copies. The per-request miscount of expected accepted drafts therefore has a standard deviation of 2.2–3.4 for the n-gram drafter and Hybrid Spark, against 0.85–1.74 for DSpark. Requests miscount independently, so the batch’s shortfall grows only as while its expected accepted drafts grow as , and the relative shortfall falls as . On Qwen3-4B, stale sizing costs the n-gram drafter 7.2–8.9% of goodput at BS 4 and Hybrid Spark 2.3–2.6%, but at most 0.64% and 0.30% at BS 128 on NextCoder, where AV matters most. Every throughput number in this paper already includes this loss. Section O derives the law, shows that it predicts the measured shortfall, and explains why a shortfall costs much less goodput than its size.
Sampling at T=1 (Q7)
At , the fastest lossless choice depends on the content. At , each policy must be paired with a drafting mode under which it stays lossless (§ 4.5): greedy drafting for the proposed policy, either mode for n-gram-first and the extension. On code editing, the proposed policy with greedy drafting is the fastest in three of the four NextCoder cells (Table 1 and Table 13; CIs exclude 1 in all four), 2.5–4.1% ahead of the better probabilistic alternative at BS 4 and 5.1–8.2% at BS 128. In the fourth cell, 40–45% at BS 128, the extension with greedy drafting is 1.6% faster. Of the two alternatives, the extension, which appends the n-gram tail to DSpark’s unchanged chain, keeps most of the proposed policy’s gain (Section N). On chat the order reverses at BS 4: the proposed policy is 3.9% slower than the extension and 4.3% slower than DSpark, both drafting probabilistically, while at BS 128 all three are level.
| BS | Policy | NextCoder 15–20% T0g | NextCoder 15–20% T0p | NextCoder 15–20% T1g | NextCoder 15–20% T1p | NextCoder 40–45% T0g | NextCoder 40–45% T0p | NextCoder 40–45% T1g | NextCoder 40–45% T1p |
| 4 | Proposed | 1.301 | 1.304 | 1.282 | – | 1.127 | 1.123 | 1.126 | – |
| 4 | N-gram-first | 1.269 | 1.274 | 1.244 | 1.231 | 1.067 | 1.067 | 1.070 | 1.056 |
| 4 | Extension | 1.269 | 1.271 | 1.241 | 1.222 | 1.128 | 1.131 | 1.124 | 1.098 |
| 4 | DSpark (prob.) | 0.994° | 0.976 | 0.994° | 0.974 | ||||
| 128 | Proposed | 1.163 | 1.159 | 1.178 | – | 1.053 | 1.051 | 1.067 | – |
| 128 | N-gram-first | 1.104 | 1.106 | 1.119 | 1.089 | 0.976 | 0.976 | 0.970° | 0.962° |
| 128 | Extension | 1.109 | 1.111 | 1.077° | 1.065 | 1.061 | 1.062 | 1.084 | 1.015° |
| 128 | DSpark (prob.) | 0.996° | 0.964° | 0.993 | 0.984° |
Table 13. Output-throughput speedup over DSpark with greedy drafting at the same temperature, all four sampling regimes (Qwen3-4B, NextCoder, 3 repeats), as in Table 1. T0p: with probabilistic drafting; T1g: with greedy drafting; DSpark (prob.): DSpark with probabilistic drafting. °CI includes 1. –: not lossless, not run.
Probabilistic drafting is beneficial only where the next token is uncertain. A greedy draft token is accepted with probability under the target’s distribution , while a token sampled from DSpark’s distribution is accepted with probability , which is close to 1 whenever is close to . On code editing, probabilistic drafting does not raise DSpark’s AL (6.1–6.7 in either mode), so it only adds its cost: DSpark is 1.6–3.6% slower with it. On chat it raises DSpark’s AL from 3.5 to 3.6 at BS 4, and DSpark is 2.9% faster with it. The drafting mode is a server setting, so a deployment at should choose it according to its traffic.
Related Work
Model-based drafters. Model-based drafters, surveyed by Xia et al. (2024), include separate small models (Leviathan et al. [22]), heads on the target such as Medusa (Cai et al. [3]), token trees (Miao et al. 2024), multi-token prediction heads [12] and EAGLE [24, 26]. Parallel drafters such as DFlash [6] and Domino [19] emit a whole block per forward pass. Any of them could fill the model-based slot given calibrated confidences.
Model-free and retrieval drafters. Prompt Lookup Decoding (Saxena, 2023) and LLMA (Yang et al., 2023) copy spans from the prompt; REST [16] retrieves from a datastore; SuffixDecoding [35] and SAM Decoding [18] index prior outputs; Lookahead decoding [10] collects n-grams from the target’s own parallel decoding. Blazedit [9] speeds up a draft model with prompt lookup for code editing, and Double (Shen et al., 2026b) adds retrieval to parallel speculative decoding. Any of these could fill Hybrid Spark’s model-free slot given features and a fitted estimator § 4.2. Drafters that run on the host would also need a GPU or asynchronous implementation (C3). Graft (Shen et al., 2026a) fills pruned tree slots with retrieved candidates within each request, whereas Hybrid Spark chooses between whole chains across the batch at a shared price.
Speculative decoding for RL rollouts. SPEC-RL [27] verifies the previous epoch’s rollout of the same prompt under the current policy, with a lenience parameter that relaxes acceptance and so changes the output distribution. RhymeRL [15] drafts from reward-aware suffix trees over each prompt’s recent responses and sets the draft length with an AIMD-inspired rule. DAS (Shao et al., 2026) drafts from per-problem suffix trees over a sliding window of recent rollouts and gives long trajectories larger draft budgets. Each uses one history drafter with a heuristic draft length. Hybrid Spark would instead combine such a drafter with a model-based drafter under one calibrated, priced budget.
Adaptive verification. TurboSpec [31] tunes the draft length per step for goodput, and SpecDec++ [20] and DISCO [33] set it per request from predicted acceptance. PEARL [28] and SpecBranch (Shen et al., 2026c) overlap drafting with verification, and TALON [29] grows a tree within a per-request token budget. Across a batch, TETRIS (Wu et al., 2025) selects draft tokens under a fixed verification capacity; DSpark [7] and D-cut [30] prune them under a profiled cost model; and ECHO [17] schedules tree depth and width under a global cap. All decide how much of one drafter’s chain to verify. Hybrid Spark extends this allocation to two drafters with different confidence sources.
Hybrid drafting. Liu et al. [32] bound the gain of an oracle that picks between an n-gram drafter and EAGLE per step, a choice that Hybrid Spark makes with a deployable policy. SuffixDecoding [35] likewise pairs a model-free drafter with a model-based one: per request, it takes the suffix-tree draft when its score, a sum of continuation frequencies in the tree, exceeds a fixed threshold. This rule is incompatible with adaptive verification in three ways. First, the score counts how often a continuation appeared in past outputs, not how often the target accepted it, so it cannot be ranked against DSpark’s survival in a batch-wide budget. Hybrid Spark instead fits acceptance probabilities (Section 4.2). Second, the fixed threshold does not rise with batch size as verification becomes compute bound, while Hybrid Spark prices slots by the goodput-optimal budget (Section 4.3). Third, its host-side suffix tree forces a CPU–GPU synchronization every step, which breaks the zero-synchronization design of vLLM’s Model Runner V2 for asynchronous scheduling (vLLM Team, 2026). Its vLLM implementation runs on the V1 model runner instead. Hybrid Spark keeps every per-step decision on the GPU (C3).
Conclusion
Model-based drafters perform well on general text, while n-gram drafters are faster on near-copy edits. Yet most deployments fix a single drafter, and adaptive verification can rank only drafts that carry confidences. Hybrid Spark runs both drafters on every step and gives n-gram drafts calibrated acceptance estimates, fit once on general text. It prices each verified slot at the goodput it would earn elsewhere in the batch: each request keeps the chain whose slots are worth their price, and adaptive verification allocates the budget over the chosen chains. Every per-step decision runs on the GPU, which keeps asynchronous scheduling intact.
Instantiated with DSpark and an n-gram drafter in vLLM, Hybrid Spark achieves 1.10–1.62 the output throughput of DSpark on code editing at BS 4 and 1.04–1.27 at BS 128, while staying at 0.97–1.01 of it on chat at . A single deployment can therefore serve both editing and general traffic, without partitioning GPUs into drafter-specific pools or forgoing the editing speedup. Either slot can in principle hold other drafters. A natural next step is RL post-training, where a history-based drafter built from earlier rollouts could fill the model-free slot. Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummins, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernogenuz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy,
Notation
Models and drafters
Table 3 lists the Hugging Face checkpoints of every target model and its DSpark drafter.
Workloads
Table 4 lists the size of each workload, the prompts per run and the output limit.
Estimator coefficients
The agreement weight is large and positive ( for Qwen3-4B), and a 256-token copy adds logits to the first position of a full match.
Fitting data (Qwen3-4B / Qwen3-8B). 342,593 / 328,854 output tokens; 749 / 709 requests with at least one match; 19,699 / 17,131 matched steps, of which 4,904 / 4,022 are full matches; 135,518 / 110,853 graded positions, of which 120,958 / 97,981 were accepted. The median copy length is 85 / 106 tokens, and 20% / 26% of copies reach the 256-token cap.
Building the estimator up one term at a time
Giving the n-gram drafter any calibrated survival is what lets it use AV, and that alone captures most of the gain. With one shared intercept fitted on SPEED-Bench and no match features, the n-gram drafter with AV reaches 0.973 [0.966, 0.980] of the full estimator’s output throughput on pooled NextCoder at BS 128 (Table 6). Verifying every draft is 1.89 slower than the full estimator on the same buckets (separate runs, 3 to 5 repeats), so one shared intercept keeps about 94% of the full estimator’s gain over verifying every draft.
| Terms | 15–20% | 40–45% | Pooled [95% CI] |
| 1 | 0.973 | 0.973 | 0.973 [0.966, 0.980] |
| 2 | 0.988 | 0.971 | 0.978 [0.971, 0.985] |
| 3 | 0.976 | 0.990 | 0.984 [0.978, 0.990] |
| 4 | 0.999 | 0.993 | 0.996 [0.987, 1.003] |
| 5 | 0.995 | 0.998 | 0.996 [0.991, 1.001] |
| 6 all (Equation (1)) | 1 | 1 | 1 |
Table 6. A single shared intercept already reaches 97% of the full estimator’s throughput, and copy length closes most of the rest. Output throughput of the n-gram drafter with AV relative to the full estimator (higher is better), Qwen3-4B, NextCoder, BS 128, 8 repeats, 95% CI for the pooled buckets. Estimators 3 and 4 each add one feature to estimator 2. Symbols as in Equation (1): one shared intercept, per-position intercepts, match length, copy length, agreement.
Setup. We serve the n-gram drafter with AV, where every survival AV uses comes from the estimator, with six estimators of increasing richness: (1) one intercept shared by all positions; (2) per-position intercepts ; (3) and match length ; (4) and copy length ; (5) , and agreement ; and (6) the full estimator of (1). Estimators 3 and 4 each add one feature to estimator 2. Each estimator is refit on the same SPEED-Bench fitting half (§ 4.2), with the omitted terms fixed at zero; in estimators 3–5 each feature has one coefficient shared by the first and later tokens. Table 7 lists the coefficients of estimators 1–5, and Table 5 those of estimator 6. Even the single shared intercept is a fitted estimator: predicts an acceptance of 0.893 at every position, so survival still decays with position and AV still trims long chains. Qwen3-4B, NextCoder 15–20% and 40–45%, BS 128, 8 repeats, each on a fresh server, with the six estimators interleaved within each repeat.
Fitted estimator coefficients (Equation (1)), fit on the SPEED-Bench fitting half. Position intercepts .
| Qwen3-4B | |||||||||||||||
| Qwen3-8B | |||||||||||||||
| Gemma 4 12B | |||||||||||||||
| North-Mini-Code | |||||||||||||||
| Llama-3.1-8B | |||||||||||||||
| Model | |||||||||||||||
| Qwen3-4B | |||||||||||||||
| Qwen3-8B | |||||||||||||||
| Gemma 4 12B | |||||||||||||||
| North-Mini-Code | |||||||||||||||
| Llama-3.1-8B |
Table 5. Fitted estimator coefficients (Equation (1)), fit on the SPEED-Bench fitting half. Position intercepts .
| 1 | 2.117 at every position | – | – | – | ||||||||||||||
| 2 | 1.158 | 1.702 | 2.104 | 2.325 | 2.466 | 2.420 | 2.536 | 2.544 | 2.515 | 2.605 | 2.757 | 2.854 | 2.877 | 3.036 | 3.046 | – | – | – |
| 3 | 0.793 | 1.293 | 1.658 | 1.845 | 1.962 | 1.882 | 1.960 | 1.923 | 1.841 | 1.876 | 1.981 | 2.025 | 1.992 | 2.098 | 2.053 | 0.196 | – | – |
| 4 | 0.860 | 1.345 | 1.700 | 1.880 | 1.987 | 1.987 | 1.967 | 1.922 | 1.834 | 1.859 | 1.956 | 1.994 | 1.950 | 2.045 | 1.988 | – | 2.070 | – |
| 5 | −0.247 | 0.144 | 0.495 | 0.671 | 0.776 | 0.685 | 0.754 | 0.709 | 0.618 | 0.646 | 0.749 | 0.791 | 0.750 | 0.850 | 0.799 | – | 2.014 | 1.261 |
Table 7. Coefficients of estimators 1–5 in Table 6 (Qwen3-4B, SPEED-Bench fitting half); estimator 6 is the full estimator in Table 5. Estimators 3–5 use one coefficient for the first and later tokens. Copy length enters only for full matches, as in Equation (1). A dash marks a term the estimator omits.
Copy length adds the most, and the other terms add little. Measured from per-position intercepts (0.978), copy length adds 1.8% (0.996 [0.987, 1.003]), and every estimator with it is level with the full one. Match length adds 0.6% (0.984), and per-position intercepts add 0.5% over one shared intercept. These two steps are within noise of each other, and their order differs between buckets: per-position intercepts help on 15–20% (0.973 to 0.988) but not on 40–45% (0.973 to 0.971), and match length does the reverse (0.976 and 0.990×). Agreement, the full-match bonus and the first-token split add nothing measurable on top of copy length (0.996× [0.991, 1.001] with agreement added). The match features still matter for calibration: copy length lowers the estimator’s calibration error on EditBench from 0.132 to 0.032, and agreement separates first-token acceptance of 24% from 81% (§ 5.5).
The estimator chooses which slots to verify; it does not raise acceptance. AL stays within 3.65–3.68 on 15–20% and 2.29–2.33 on 40–45% across all six estimators, so the throughput differences come from which slots AV verifies, not from how many drafts are accepted.
Scope. On these two buckets, intercepts alone cost 2.2%, less than the 4.1% pooled over all NextCoder buckets in § 5.5. Inside Hybrid Spark the same estimators stay level with the full one (intercepts alone: 0.985× [0.959, 1.007], 5 repeats).
Coefficients. Estimator 2’s intercepts rise from 1.158 at the first position to 3.046 at the fifteenth, an acceptance of 0.76 rising to 0.95, which is the profile of Observation 2 (Table 7). Adding a feature lowers the intercepts, because the feature then explains part of the acceptance. Fitted alone, match length has a positive weight (): longer matches are accepted more often. In the full estimator, its first-token weight is negative (, Table 5), because the full-match bonus and copy length take over the effect of long matches, so the two signs do not conflict.
Cost model
For reproducibility we describe how AV’s step-cost model (§ 4.3 and § 4.4) is built. It is profiled once at startup. The step cost is , and only the ratio of to affects decisions.
The slot cost for requests is the time difference between a verification pass with every query slot
filled and a pass with one token per request, divided by the number of slots added. It is tabulated per request count.
The base cost is a token-count term for the non-draft tokens plus a KV-read term proportional to the total context. Both come from two profiling passes, with and without synthetic context.
Above the CUDA graph capture limit, a step runs eagerly. The jump from graphed to eager execution is not linear in , so for above the limit the cost is
where is the profiled cost of an eager step with tokens.
All of this is host-side arithmetic over arrays the scheduler already holds.
Chain value: a worked example
The slot price (Equation (2)) depends on the batch size and the hardware through the step costs. For Qwen3-8B on NextCoder on an H100 GPU (§ 5.1), it is about 0.04 at BS 4 and about 0.3 at BS 128.
Table 8 shows a case where ranking chains by expected accepted tokens, , keeps the n-gram chain and the value (Equation (3)) keeps DSpark’s. Survivals are averaged over steps for Qwen3-8B on NextCoder 45–50% at BS 128, where . The n-gram chain’s lead in comes entirely from its 6 slots below , which add 1.52 expected tokens, about 0.25 per slot, while each slot would earn 0.309 elsewhere in the batch. On the 15–20% bucket (), the n-gram chain is stronger and also has the higher value, 4.36 against 4.04. The value rule thus drops the n-gram chain only where its slots are worth less than their price, not at every large batch size.
| Chain | slots | their | ||
| n-gram | 5.87 | 9 | 4.35 | 1.57 |
| DSpark | 5.58 | 7 | 5.58 | 3.42 |
Table 8. Chain value at BS 128 (Qwen3-8B, NextCoder 45–50%, ). prefers the n-gram chain, whereas prefers DSpark’s.
Losslessness
This appendix expands § 4.5.
Greedy drafting is lossless at any temperature. At target temperature 0, any draft only changes speed. At higher temperatures, with greedy drafting, both chains and every selection and allocation decision are functions of the context alone. The target verifies each draft token as a point mass with standard rejection sampling, so the output distribution is exactly the target’s. D-cut makes the same argument for pruning deterministic drafts ([30 App. A]).
Probabilistic drafting needs a restricted policy. When DSpark samples its drafts from its own distribution , verifying slot against preserves the target’s distribution only if the decision to verify slot , and how, is independent of the token drawn at slot given the tokens before it ([7]). Point-mass drafts fixed before verification, independently of the sampled tokens, are exempt. D-cut gives a one-token example of the bias when this fails ([30 App. B]). Our proposed policy (§ 4.3) violates the condition in two ways. First, the contains rule compares DSpark’s sampled tokens with the n-gram chain, so whether DSpark’s chain is kept depends on the tokens DSpark drew. Second, the value rule scores DSpark’s chain from its confidences , and DSpark’s confidence head computes from the token it sampled at position . A sampled token that makes DSpark confident about the next positions raises its chain’s value and keeps the chain, while one that makes it unsure lowers and hands the slots to the n-gram chain. Whether DSpark’s token at slot is verified then depends on that token itself, which skews the output toward tokens after which DSpark is confident.
Two lossless alternatives. For probabilistic drafting we therefore consider two simpler policies. (1) N-gram-first takes the n-gram chain whenever it matched and DSpark’s chain otherwise. A model-free chain is fixed before verification and does not depend on DSpark’s sampled tokens (§ 4.1), so the choice never reads DSpark’s samples. (2) The extension policy always verifies DSpark’s chain against and appends the n-gram tail only when DSpark’s whole chain is a prefix of the n-gram chain. Slots 1–7 are unchanged. A tail slot is reached only if all 7 DSpark tokens are accepted, so the extension decision reads only tokens before it, and a per-slot flag tells the rejection sampler to verify tail slots as point masses. Tail survivals continue from DSpark’s last survival, so a tail slot never outranks DSpark’s own slots. Table 9 summarizes where each policy is lossless. Since greedy drafting is lossless at any temperature, a deployment that samples at T=1 can either keep the proposed policy with greedy drafting or switch to an alternative with probabilistic drafting. Which is faster depends on the workload (§ 5.8).
Where DSpark’s drafter is weak
In these cases DSpark’s drafter accepts few tokens per step, so DSpark itself is slow and the speedups of Hybrid Spark over it are large partly for that reason. A better-trained drafter would shrink them, so we keep them out of our headline claims.
North-Mini-Code. On Blazedit and CanItEdit at BS 4, DSpark accepts – tokens per step, against – for Hybrid Spark (Figure 7). Hybrid Spark achieves – over DSpark on Blazedit and CanItEdit at BS 4 and – at BS 128 (Figure 5), and on pooled NextCoder, and and on InstructCoder at BS 4 and 128.
Llama-3.1-8B and long contexts. Table 10 reports Llama-3.1-8B with a community-trained DSpark drafter (AL 2.0–3.2 on editing) and EditBench requests with more than 6,000 tokens of context, where the in-distribution drafters’ AL falls (from about 6.5 below 6k tokens to 2.7 above 12k for Qwen3-4B and Qwen3-8B). Hybrid Spark achieves up to over DSpark at BS 4 and at BS 128, and Llama on MT-Bench is level (–).
| Model | Workload | Speedup BS 4 | Speedup BS 32 | Speedup BS 128 | AL, DSpark/ Hybrid Spark BS 4 | AL, DSpark/ Hybrid Spark BS 32 | AL, DSpark/ Hybrid Spark BS 128 |
| Llama-3.1-8B | Blazedit 5k | 1.97 | 1.71 | 1.41 | 2.4 / 5.1 | 2.4 / 5.0 | 2.4 / 4.6 |
| Llama-3.1-8B | Blazedit 10k | 2.96 | 2.32 | 1.96 | 2.1 / 6.3 | 2.0 / 6.3 | 2.0 / 6.0 |
| Llama-3.1-8B | CanItEdit | 1.84 | 1.49 | 1.30 | 3.2 / 6.2 | 3.2 / 5.9 | 3.0 / 5.2 |
| Llama-3.1-8B | NextCoder | 1.69 | 1.60 | 1.46 | 2.9 / 5.5 | 2.9 / 5.4 | 2.8 / 4.9 |
| Llama-3.1-8B | InstructCoder | 1.22 | 1.12 | 1.07 | 3.3 / 4.1 | 3.1 / 3.9 | 2.8 / 3.6 |
| EB-LC | Qwen3-4B | 3.99 | 3.26 | 2.76 | 3.3 / 14.4 | 3.3 / 14.3 | 3.2 / 14.2 |
| EB-LC | Qwen3-8B | 4.07 | 2.77 | 2.49 | 3.2 / 14.6 | 3.2 / 14.6 | 3.2 / 14.4 |
| EB-LC | Gemma 4 12B | 2.24 | 1.73 | 1.51 | 5.8 / 14.2 | 5.6 / 14.0 | 5.4 / 13.9 |
| EB-LC | Llama-3.1-8B | 7.58 | 5.09 | 3.87 | 1.6 / 14.3 | 1.6 / 14.3 | 1.6 / 14.1 |
Table 10. Out of distribution for the DSpark drafter: large gains that reflect weak drafters. Left: output-throughput speedup of Hybrid Spark over DSpark. Right: acceptance length (tokens per verification step) of DSpark/ Hybrid Spark; NextCoder’s is the mean over its edit-distance buckets. EB-LC: EditBench requests with more than 6,000 tokens of context.
Values of Figure 1
Speedup per workload
Table 12 lists the output-throughput speedup of Hybrid Spark over DSpark for every model, workload and batch size in Figure 5, including the pooled NextCoder and EditBench cells.
Acceptance length per workload
Figure 7 shows the AL of DSpark and Hybrid Spark at BS 4 on the five single-file workloads of Figure 5.
EditBench by edit distance
Figure 8 breaks EditBench down by edit distance, as the last three panels of Figure 5 do for NextCoder. As in the main results, requests with more than 6,000 tokens of context are excluded from every bucket. The available DSpark drafters were likely trained on little data this long, and their AL falls from about 6.5 below 6k tokens to 2.7 above 12k (§ 16). DSpark is then slow and the gains of Hybrid Spark over it are inflated, so we leave these requests out. As on NextCoder, the gain falls with edit distance for the three in-distribution models: at BS 4 from 1.82–1.87× at 0–5% to 1.34–1.41× at 20–25%, and at BS 128 from 1.18–1.36× to 1.07–1.14×.
All policy cells
Of the alternatives, the extension keeps most of the proposed policy’s gain. At it is within 5% of the proposed policy’s throughput, and on 40–45% it is within 0.2% of it or above it in every regime. N-gram-first is 0.8% and 2.3% faster than the extension on 15–20% at but slower than DSpark on 40–45% at BS 128. Table 13 and Table 14 list all four regimes for NextCoder and MT-Bench, each over DSpark with greedy drafting, as in Table 1.
| BS | Policy | T0g | T0p | T1g | T1p |
| 4 | Proposed | 0.989 | 0.981 | 0.985 | – |
| 4 | N-gram-first | 0.980 | 0.976 | 0.982 | 1.015 |
| 4 | Extension | 1.025 | |||
| 4 | DSpark (prob.) | 1.029 | |||
| 128 | Proposed | 0.968 | 0.964 | 0.971 | – |
| 128 | N-gram-first | 0.970 | 0.964 | 0.965 | |
| 128 | Extension | 0.953 | 0.974 | ||
| 128 | DSpark (prob.) | 0.983 |
Table 14. As Table 13, on MT-Bench (Qwen3-4B, 3 repeats).
SPEED-Bench at T=1. Table 15 repeats the T=1 regimes on the 1k and 2k SPEED-Bench prompts of Section P, with the same serving setup. The largest speedups are on low-entropy prompts at BS 4: 1.06× for the proposed policy, 1.07× for n-gram-first and 1.04× for the extension. With the better drafting mode per policy, the worst case is 0.97× for the proposed policy, 0.98× for the extension and 0.96× for n-gram-first, all on mixed or high-entropy prompts. Outside low-entropy prompts at BS 4, the fastest lossless choice is 1.00–1.02×, and in 6 of these 10 cells it is DSpark itself. Figure 9 gives the proposed policy at T=0 on the same prompts.
| BS | Policy | 1k context: Low T1g | 1k context: Low T1p | 1k context: Mixed T1g | 1k context: Mixed T1p | 1k context: High T1g | 1k context: High T1p | 2k context: Low T1g | 2k context: Low T1p | 2k context: Mixed T1g | 2k context: Mixed T1p | 2k context: High T1g | 2k context: High T1p |
| 4 | Proposed | 1.063 | – | 0.989 | – | 0.972 | – | 1.055 | – | 0.993 | – | – | |
| 4 | N-gram-first | 1.069 | 1.042 | 0.980 | 0.989 | 0.981 | 1.008 | 1.047 | 1.013 | 0.960 | 0.977 | 1.013 | 1.016 |
| 4 | Extension | 1.044 | 1.017 | 1.010 | 1.009 | 1.038 | 1.017 | 1.012 | |||||
| 4 | DSpark (prob.) | 0.978 | 1.013 | 0.974 | 1.013 | ||||||||
| 128 | Proposed | – | 0.975 | – | 0.981 | – | – | 0.979 | – | 0.989 | – | ||
| 128 | N-gram-first | 0.978 | 0.949 | 0.965 | 0.951 | 0.974 | 0.956 | 0.978 | 0.961 | 0.954 | 0.982 | 0.971 | |
| 128 | Extension | 0.970 | 0.988 | 0.970 | 0.968 | 0.982 | 0.985 | 0.970 | 0.975 | ||||
| 128 | DSpark (prob.) | 0.955 | 0.966 | 0.980 | 0.980 | 0.977 | 0.986 |
Table 15. As Table 13, at T=1 on held-out SPEED-Bench prompts with 1k and 2k tokens of context, per category (low entropy, mixed, high entropy; Qwen3-4B, FP8 KV cache, 3 repeats).
Figure 9. Hybrid Spark’s gain over DSpark grows with context length on low-entropy SPEED-Bench prompts, where DSpark’s AL falls at 8k and Hybrid Spark’s rises, and stays level on mixed ones. Top: output-throughput speedup of Hybrid Spark over DSpark (higher is better), with 95% CIs. Bottom: AL of Hybrid Spark (bars) and DSpark (black ticks). Qwen3-4B, , FP8 KV cache, prompts not used to fit the estimator, mean of 3 repeats.
Stale budget sizing
This appendix expands § 5.7.
Where staleness comes from. AV splits its work across the boundary of asynchronous scheduling (§ 2). The scheduler on the host fixes the size of the next verification pass, the budget , before the GPU has produced the survivals for that step, so it sizes the budget from survivals one step old. The GPU then decides which (request, position) pairs get the slots from the fresh survivals. This split is DSpark’s design [7 §5.2], and Hybrid Spark keeps it: only the budget’s size and the slot price are stale, never the allocation. A stale budget is too small or too large by however much the batch’s survivals changed in one step.
Why the n-gram drafter suffers more than DSpark. DSpark proposes a chain on every step, and its confidences drift only in magnitude from one step to the next. An n-gram match instead appears and vanishes abruptly as the output enters and leaves a region it copies from its input. A request that has just entered a copy region has stale survivals with no match, so the host sizes the budget as if the request were worth nothing and misses its 15-slot chain. A request that has just left one is counted for a chain it no longer has. We measure the per-request miscount , the expected accepted drafts under the stale survivals minus under the fresh ones. For the n-gram drafter, a match is gained or lost between the previous step and the current one for 5.7–9.2% of requests in each direction, and these requests are miscounted by −4.4 to −5.3 and +5.4 to +7.1 drafts. In Hybrid Spark a request’s chosen chain switches between n-gram and DSpark instead, for 8.8–13.7% of requests in each direction, with miscounts of −3.0 to −5.0 and +4.6 to +7.5. The standard deviation of is 0.85–1.74 for DSpark, 2.2–3.0 for the n-gram drafter and 2.4–3.4 for Hybrid Spark. Hybrid Spark’s is the largest because it has both sources of error: chain switches, as in the n-gram drafter, and drift in the confidences of the DSpark chains that the other requests carry.
The error grows as . The batch miscount is the sum of the per-request miscounts, . Two measured properties make its size predictable. First, does not depend on the batch size. For each model, arm and workload, it varies by at most 0.10 across BS 4, 32 and 128, since how often a request’s match comes and goes is a property of the request, not of the batch. Second, requests miscount independently: the standard deviation of is 0.96–1.11 times in every run. therefore has standard deviation and, as a sum of many independent terms, is close to normal with mean near zero. The budget is too small when , and on those steps the shortfall is the mean of a half-normal distribution:
The expected accepted drafts of the batch grow as , so the shortfall relative to them falls as (column Rel. in Table 16).
Methodology. We instrument the engine to record, at every full-batch step, both the stale survivals that sized the budget and the fresh ones. We re-pick the budget from each with AV’s own rule, which reproduces the budget the engine chose on 100% of decisions, and score both budgets on the fresh survivals. Goodput lost is , averaged over steps. It uses the estimator’s predicted acceptance, so only the difference between stale and fresh is meaningful. We run DSpark, n-gram + AV and Hybrid Spark on Qwen3-4B and Llama-3.1-8B, on InstructCoder and NextCoder 15–20% and 40–45%, at BS 4, 32 and 128, each on one GPU (54 runs).
The prediction matches the measurement. Table 16 compares Equation (5) with the measured shortfall. At BS 32 and 128 the observed shortfall is 0.88–1.14 times the prediction on every NextCoder run of both models, and 0.77–1.14 times including InstructCoder. At BS 4 a sum of four terms is far from normal, and the ratio spreads from 0.80 to 1.36. For every arm, the relative shortfall falls roughly as , from 8.1% to 1.4% for DSpark, 39.4% to 8.5% for the n-gram drafter and 13.5% to 2.8% for Hybrid Spark.
| Arm | BS | Pred. | Obs. | Rel. | Lost | |
| DSpark | 4 | 1.37 | 2.16 | 1.94 | 8.1% | 0.29 |
| DSpark | 32 | 1.36 | 6.11 | 5.63 | 3.0% | 0.16 |
| DSpark | 128 | 1.37 | 12.24 | 11.03 | 1.4% | 0.06 |
| n-gram + AV | 4 | 2.98 | 4.74 | 4.23 | 39.4% | 7.17 |
| n-gram + AV | 32 | 3.00 | 13.50 | 13.11 | 16.0% | 1.82 |
| n-gram + AV | 128 | 2.96 | 26.60 | 28.04 | 8.5% | 0.35 |
| Hybrid Spark | 4 | 3.39 | 5.35 | 4.72 | 13.5% | 2.63 |
| Hybrid Spark | 32 | 3.42 | 15.26 | 14.78 | 5.3% | 0.80 |
| Hybrid Spark | 128 | 3.43 | 30.66 | 30.73 | 2.8% | 0.30 |
Table 16. The half-normal model predicts the stale budget’s shortfall, and the relative shortfall and its goodput cost fall with batch size. Qwen3-4B, NextCoder 15–20%. : standard deviation of the per-request miscount. Pred.: Equation (5), from the measured and the number of requests with previous-step survivals (3.9–4.0, 31.3–31.7 and 125.5–127.0 on average). Obs.: mean shortfall on steps where the stale budget is too small. Both in expected accepted drafts. Rel.: Obs. divided by the batch’s expected accepted drafts. Lost: goodput lost (%).
The error is largest where AV matters least. On Qwen3-4B across the three workloads, stale sizing costs the n-gram drafter 7.2–8.9% of goodput at BS 4 and Hybrid Spark 2.3–2.6%, but at BS 128 it costs them at most 0.64% and 0.30% on NextCoder. At BS 4 verification is nearly free and AV has little to gain. At BS 128, where AV gives the n-gram drafter its 1.90× over verifying every draft (§ 3), staleness costs almost nothing. Every throughput number in this paper runs AV at every batch size, so it already includes this loss.
A shortfall costs much less goodput than its size. At BS 4, Hybrid Spark’s budget falls short by 13.5% of the batch’s expected accepted drafts but loses 2.6% of goodput, and the n-gram drafter’s falls short by 39.4% and loses 7.2%. Three effects shrink the cost. First, the allocation is fresh, so a short budget drops the lowest-ranked slots, whose survivals are closest to and which by Equation (2) add the least goodput. A long budget likewise adds slots just below . Second, verifying fewer slots also costs less, so the goodput ratio partly corrects itself. Third, the budget is too small on only about half the steps. Hybrid Spark also loses less than the n-gram drafter alone. When a request finds a match that its stale survivals miss, the host still budgets for the request’s DSpark chain rather than for nothing. The budget therefore falls short by less, and the request receives 72–80% of its due slots at BS 4, against 29–36% for the n-gram drafter alone.
Context length on SPEED-Bench
On low-entropy prompts, Hybrid Spark’s gain over DSpark grows with context length, while on mixed prompts the two are level. Figure 9 shows the speedups and ALs for Qwen3-4B at on SPEED-Bench prompts of 1k, 2k and 8k tokens.
Prompts. SPEED-Bench’s throughput split has three categories, which we run as separate workloads: low entropy (code completion and sorting), mixed (question answering and needle-in-a-haystack) and high entropy (book continuation and creative writing). The 8k split has 1,536 prompts, 512 per category (1,530 distinct). The estimator is fit on 764 of them (§ 4.2), so we use only the other 766: 256 low-entropy, 256 mixed and 254 high-entropy. At 1k and 2k we drop prompts drawn from the same source document as a fitting prompt and sample the same number per category as at 8k. Each request generates up to 512 tokens. BS 4 and 32 send each prompt once, BS 128 twice.
Serving. Every arm runs on two GPUs with tensor parallelism at every batch size, with an FP8 KV cache, so that 128 requests with 8k-token prompts fit in memory, and prefix caching off. FP8 lowers DSpark’s AL slightly (2.7 against 2.8 with the BF16 KV cache on the 8k high-entropy prompts at BS 128), so these speedups are not directly comparable with the rest of the paper. Each cell is the mean of 3 repeats, each on a fresh server.
Low entropy. Hybrid Spark’s speedup over DSpark rises from 1.07× at 1k to 1.40× at 8k at BS 4, and from 1.02× to 1.13× at BS 128. As with EditBench requests over 6,000 tokens of context (§ 16), DSpark’s AL falls with context, from 4.1 at 1k to 2.9 at 8k, while Hybrid Spark’s rises from 4.6 to 5.0 at BS 4. The n-gram drafter copies from the longer prompt what DSpark misses. At 8k, Hybrid Spark is 2.35× faster than vanilla decoding at BS 4 and 1.36× at BS 128.
Mixed and high entropy. On mixed prompts the AL of both systems is 3.6–4.0 at every length, and Hybrid Spark is level with DSpark at BS 4 and 32 and 0.98–0.99× at BS 128. On high-entropy prompts the gain is small but also grows with context, from 1.01× to 1.08× at BS 4 and from 0.99× (level) to 1.02× at BS 128.
References
- [1]Talor Abramovich, Maor Ashkenazi, Izzy Putterman, Benjamin Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein, and Yonatan Geifman. SPEED-Bench: A unified and diverse benchmark for speculative decoding. arXiv preprint arXiv:2604.09557, 2026.arxiv.org/abs/2604.09557
- [2]Tushar Aggarwal, Swayam Singh, Abhijeet Awasthi, Aditya Kanade, and Nagarajan Natarajan. NextCoder: Robust adaptation of code LMs to diverse code edits. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of PMLR, pages 616–638, 2025.
- [3]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2401.10774
- [4]Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, and Arjun Guha. Can it edit? Evaluating the ability of large language models to follow code editing instructions. arXiv preprint arXiv:2312.12450, 2023.arxiv.org/abs/2312.12450
- [5]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.arxiv.org/abs/2302.01318
- [6]Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block diffusion for flash speculative decoding. In Proceedings of the 43rd International Conference on Machine Learning (ICML), volume 306 of PMLR, 2026. arXiv:2602.06036.arxiv.org/abs/2602.06036
- [7]Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanghuai Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, and Wenfeng Liang. DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv preprint arXiv:2607.05147, 2026. Wayne Chi, Valerie Chen, Ryan Shar, Aditya Mittal, Jenny Liang, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Ion Stoica, Graham Neubig, Ameet Talwalkar, and Chris Donahue. EDIT-Bench: Evaluating LLM abilities to perform real-world instructed code edits. In International Conference on Learning Representations (ICLR), 2026. arXiv:2511.04486.
- [8]Cohere Labs. North-Mini-Code-1.0. https://huggingface.co/CohereLabs/North-Mini-Code-1.0, 2026.
- [9]Vijay Daita, Xinyu Lian, Lingming Zhang, and Jiawei Liu. Blazedit: Blazing-fast code editing via multi-layer speculation. https://huggingface.co/blog/ganler/blazedit, February 2025. Hugging Face blog.
- [10]Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of LLM inference using lookahead decoding. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of PMLR, pages 14060–14079, 2024.arxiv.org/abs/2402.02057
- [11]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Aditya Chawla, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, Clément Farabet, Johan Ferret, Filippo Galgani, Sertan Girgin, Jean-Bastien Grill, Maarten Grootendorst, Jiaxian Guo, Cassidy Hardin, Yanzhang He, Steven M. Hernandez, Omri Hornburger, Léonard Hussenot, Juyeong Ji, Armand Joulin, Aishwarya Kamath, Parnian Kassraie, Olivier Lacombe, Preethi Lahoti, Gaël Liu, Gus Martins, Luciano Martins, Tatiana Matejovicova, Ramona Merhej, Nikola Momchev, Sneha Mondal, Ryan Mullins, Sindhu Raghuram Panyam, Shreya Pathak, Sarah Perrin, André Susano Pinto, Etienne Pot, Angéline Pouget, Alexandre Ramé, Sabela Ramos, Douglas Reid, David Rim, Morgane Rivière, Karsten Roth, Louis Rouillard, Omar Sanseviero, Pier Giuseppe Sessa, Shane Settle, Danila Sinopalnikov, Sara Smoot, Piotr Stanczyk, Andreas Steiner, Lawrence Stewart, Ilya Tolstikhin, Michael Tschannen, Anton Tsitsulin, Nino Vieillard, Renjie Wu, Pingmei Xu, Haichuan Yang, Edouard Yvinec, Biao Zhang, Li Zhang, Joe Zou, Nicolas Aagnes, Abdelrahman Abdelhamed, Jakub Adamek, Shivani Agrawal, Shubham Agrawal, Ibrahim Alabdulmohsin, Jean Baptiste Alayrac, Uri Alon, Chandramouli Amarnath, Ankesh Anand, Chrysovalantis Anastasiou, Setareh Ariafar, François-Xavier Aubet, Kyriakos Axiotis, Federico Barbero, Joelle Barral, Alexei Bendebury, Urs Bergmann, Stanley Bileschi, Kat Black, Mathieu Blondel, Sebastian Borgeaud, Arthur Bražinskas, Ryan Burnell, Robert Busa-Fekete, Mu Cai, Daniele Calandriello, Glenn Cameron, Charlotte Cauchetaux, Rahma Chaabouni, Garima Chadha, Jetha Chan, Blake Jianhang Chen, Jesse Chen, Lin Chen, Xu Chen, Derek Cheng, Tzu hsiang Chien, Nikolai Chinaev, Yi Chou, Zhaohui Chu, Benjamin Coleman, Pooja Consul, Sam Conway-Rahman, Scott Crowell, Dylan Cutler, Vivek Dani, Samira Daruki, Anil Das, Daniel Deutsch, Nishanth Dikkala, Li Ding, Qiuhan Ding, Shenil Dodhia, Konstantin Donhauser, Tulsee Doshi, Anca Dragan, Alex Druinsky, Sahil Dua, Zoltan Egyed, Danielle Eisenbud, Daniel Eppens, Cindy Fan, Bahare Fatemi, Yassir Fathullah, Vlad Feinberg, Milen Ferev, Sebastiaan Fleernehag, Takumi Fujimoto, João Gabriel Oliveira, Isaac Galatzer-Levy, João Gante, Simon Geisler, Soham Ghosal, Antonious M. Girgis, Tamara von Glehn, Alec Go, Alhaad Gokhale, Alex Grills, Yiming Gu, Mayank Gupta, Pramod Gupta, Guru Guruganesh, Raia Hadsell, Hamza Harkous, Jitendra Harlalka, Demis Hassabis, Anja Hauth, Joe Heyward, Arian Hosseini, Chih-Yang Hsia, I-Hung Hsu, Xiaopeng Huang, Yangsibo Huang, Kevin Hui, Adrian Hutter, Te I, Fotis Iliopoulos, Advait Jain, Ganesh Jawahar, Ziwei Ji, Qilin Jin, Melvin Johnson, Kandarp Joshi, Arun Kandoor, Wang-Cheng Kang, Koray Kavukcuoglu, Mehran Kazemi, Kathleen Kenealy, Amr Khalifa, Phoebe Kirk, Ivan Korotkov, Suraj Kothawade, Vitaly Kovalev, Neel Kovelamudi, Adam Kraft, Ravin Kumar, Vivek Kumar, Harish Kuppam, Justin Lannin, Chen-Yu Lee, Seungji Lee, Dmitry Lepikhin, Alon Levkovitch, Dongdong Li, Qijia Li, Valentin Liévin, Ethan Lin, Ziqian Lin, Casper Liu, Tianlin Liu, Tianqi Liu, Xin Liu, Ivan Lobov, Mayank Lunayach, Min Ma, Gagan Madan, Andrii Maksai, Eric Malmi, Michal Matuszak, Daniel McDuff, Gaurav Menghani, Maciej Mikuła, Daniil Mirylenka, Karolis Misiunas, Vedant Misra, Andreea Mitran, Kareem Mohamed, Maksim Mukha, Eric Noland, James O’Donnell, Brendan O’Donoghue, Kate Olszewska, Bernett Orlando, Wanqiong Pan, Rina Panigrahy, Unnati Parekh, Nicolas Perez-Nieves, Chunjong Park, Eric Paskie, Liqian Peng, Bryce Petrini, Slav Petrov, Jonas Pfeiffer, Bilal Piot, Martyna Plomecka, Siim Poder, Octavio Ponce, Arijit Pramanik, David Racz, Anish Rajan, Michelle Ramanovich, Anand Rao, Marvin Ritter, Vitor Rodrigues, Evan Rosen, Mikołaj Rybiński, Noveen Sachdeva, Michaël E. Sander, Rohit Sathyanarayana, Sagar Savla, Samuel Schmidgall, Tal Schuster, George Scrivener, Benoit Seguin, Andrew Sellergren, Aliaksei Severyn, Izhak Shafran, Dhruv Shah, Bobak Shahriari, Yuan Shangguan, Ashish Shenoy, Pradeep Shenoy, Rakesh Shivanna, Pauline Sho, Lucas Spangher, Wojciech Stokowiec, Tim Strother, Yao Su, Yinghao Sun, Mukund Sundararajan, Andrea Tacchetti, Mor Hazan Taege, Pouya Tafti, Jean Tarbouriech, Chetan Tekur, Shantanu Thakoor, Rahul Thapa, Madeleine Traverse, Lenart Treven, Tao Tu, Chien Te Tung, Çağlar Ünlü, Petar Veličković, Malini Pooni Venkat, Sagar Gubbi Venkatesh, Vidya Venkiteswaran, Francesco Visin, Alex Vitvitskyi, Kiran Vodrahalli, Weiyi Wang, Xin Wang, Tris Warkentin, Jan Wassenberg, John Wieting, Cindy Wu, Lechao Xiao, Hao Xu, Yuhui Xu, Fuzhao Xue, Arun Yadav, Jun Yan, Antoine Yang, Lin Yang, Ming-Hsuan Yang, Ziyu Ying, Jae Hyeon Yoo, Morteza Zadimoghaddam, Sajjad Zafar, Fred Zhang, Jiageng Zhang, Jianyi Zhang, Xiaofan Zhang, Chao Zhao, David Zhou, and Chen Zou. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026.
- [12]Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of PMLR, pages 15706–15734, 2024.arxiv.org/abs/2404.19737
- [13]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravanakumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozier, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vraneš, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnston, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheshwari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernogoruz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
- [14]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.arxiv.org/abs/1706.04599
- [15]Jingkai He, Tianjian Li, Erhu Feng, Dong Du, Qian Liu, Tao Liu, Yubin Xia, and Haibo Chen. History rhymes: Accelerating LLM reinforcement learning with RhymeRL. arXiv preprint arXiv:2508.18588, 2025.arxiv.org/abs/2508.18588
- [16]Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D. Lee, and Di He. REST: Retrieval-based speculative decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024.
- [17]Xinyi Hu, Yuhao Shen, Baolin Zhang, Hengxin Zhang, Jun Dai, Shuang Ge, Lei Chen, Yue Li, and Mingcheng Wan. ECHO: Elastic speculative decoding with sparse gating for high-concurrency scenarios. arXiv preprint arXiv:2604.09603, 2026.arxiv.org/abs/2604.09603
- [18]Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, and Jing Zhang. SAM decoding: Speculative decoding via suffix automaton. arXiv preprint arXiv:2411.10666, 2024.arxiv.org/abs/2411.10666
- [19]Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, Hanlin Xu, and Linfeng Zhang. Domino: Decoupling causal modeling from autoregressive drafting in speculative decoding. arXiv preprint arXiv:2605.29707, 2026.arxiv.org/abs/2605.29707
- [20]Kaixuan Huang, Xudong Guo, and Mengdi Wang. SpecDec++: Boosting speculative decoding via adaptive candidate lengths. In Conference on Language Modeling (COLM), 2025. arXiv:2405.19715.arxiv.org/abs/2405.19715
- [21]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023.arxiv.org/abs/2309.06180
- [22]Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2211.17192
- [23]Kaixin Li, Qisheng Hu, Xu Zhao, Hui Chen, Yuxi Xie, Tiedong Liu, Qizhe Xie, and Junxian He. InstructCoder: Instruction tuning large language models for code editing. arXiv preprint arXiv:2310.20329, 2023.arxiv.org/abs/2310.20329
- [24]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024a.arxiv.org/abs/2401.15077
- [26]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2503.01840.arxiv.org/abs/2503.01840
- [27]Bingshuai Liu, Ante Wang, Zijun Min, Liang Yao, Haibo Zhang, Yang Liu, Xu Han, Peng Li, Anxiang Zeng, and Jinsong Su. SPEC-RL: Accelerating on-policy reinforcement learning with speculative rollouts. arXiv preprint arXiv:2509.23232, 2025a.arxiv.org/abs/2509.23232
- [28]Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, and Xiao Sun. PEARL: Parallel speculative decoding with adaptive draft length. In International Conference on Learning Representations (ICLR), 2025b. arXiv:2408.11850.arxiv.org/abs/2408.11850
- [29]Tianyu Liu, Qitan Lv, Yuhao Shen, Xiao Sun, and Xiaoyan Sun. TALON: Confidence-aware speculative decoding with adaptive token trees. arXiv preprint arXiv:2601.07353, 2026a.arxiv.org/abs/2601.07353
- [30]Tianyu Liu, Yuhao Shen, Rui Cen, Junhan Shi, Jiebin Zhang, Guangshuo Qin, Hong Liu, Song Liu, Guanghua Yu, and Jianchen Zhu. D-cut: Adaptive verification depth pruning for batched speculative decoding. arXiv preprint arXiv:2607.14647, 2026b.arxiv.org/abs/2607.14647
- [31]Xiaoxuan Liu, Jongseok Park, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Chen Zhang, Kuntai Du, Xiangxi Mo, Kaichao You, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. TurboSpec: Closed-loop speculation control system for optimizing LLM serving goodput. arXiv preprint arXiv:2406.14066, 2024.arxiv.org/abs/2406.14066
- [32]Xiaoxuan Liu, Jiaxiang Yu, Jongseok Park, Ion Stoica, and Alvin Cheung. Speculative decoding: Performance or illusion? In Proceedings of Machine Learning and Systems (MLSys), 2026c. arXiv:2601.11580.arxiv.org/abs/2601.11580
- [33]Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, and Roy Schwartz. Dynamic speculation lookahead accelerates speculative decoding of large language models. arXiv preprint arXiv:2405.04304, 2024.arxiv.org/abs/2405.04304
- [35]Gabriele Oliaro, Zhihao Jia, Daniel Campos, and Aurick Qiao. SuffixDecoding: Extreme speculative decoding for emerging AI applications. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2411.04975
- [36]Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heck, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems (MLSys), 2023.arxiv.org/abs/2211.05102
- [37]Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisht Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. MagicDec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. In International Conference on Learning Representations (ICLR), 2025. arXiv:2408.11049.arxiv.org/abs/2408.11049







