WorksComputer Science
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models
Abstract
Speculative decoding (SD) accelerates autoregressive inference by leveraging the memory bound nature of decoding, verifying multiple draft tokens in a single forward pass. For mixture-of-experts (MoE) models, prior work predicts a concave speedup-batch size relationship, with gains peaking at intermediate batch sizes due to expert saturation. We evaluate this across 10 sparse MoE models (30B–1T parameters, 8–512 experts) and find recent fine-grained models break this trend, showing higher speedup at low batch size. We find verification cost is governed not only by activated-expert bytes, but also by temporal overlap across draft tokens and GPU execution dynamics. At low batch size, MoE layers in fine-grained models account for a much smaller share of runtime than of weights; on H100, kernel profiling attributes this mainly to higher bandwidth utilization of MoE kernels relative to attention, with L2 absorbing repeated reads of co-routed experts. Across draft tokens, routing overlap further reduces the unique experts activated during verification relative to independence assumptions. We model this effect using a maximum-entropy Iterative Proportional Fitting (IPF) framework that predicts unique expert counts from routing-overlap statistics with less than 4% error, and substantially narrows the gap to real-model curves and captures the qualitative low-batch trend from synthetic routing.
Introduction
Sparse MoE architectures are increasingly used in high-capacity LLMs because they scale model capacity without proportionally increasing per-token compute [24, 11, 55, 16]. At the same time, long-horizon agentic workloads, test-time scaling, RL rollouts with long-tailed completions, and smaller memory budgets increasingly push inference into low-batch regimes where token latency matters most [29, 42, 26, 48, 46, 12, 27, 14]. Speculative decoding (SD) is a lossless way to reduce autoregressive latency: a draft model proposes tokens, and the target model evaluates the corresponding positions in a single forward pass, amortizing the cost of a target-model invocation across positions [31, 5]. Prior work suggests that SD is less effective for sparse MoE targets in this regime because verification may activate many additional experts and erode the expected gains [23, 34, 43].
For a fixed draft model, SD speedup is governed by the acceptance length and the verification cost. Most prior work focuses on improving acceptance length [34, 36, 4], while verification cost is often treated as a simple function of architecture and batch size [42]. For sparse MoEs, however, verification cost depends on how many unique experts are activated across the verification positions. MoESD [23] models this cost under an activated-expert-bytes view: in a memory-bandwidth-bound regime [52], cost scales with the total expert-weight bytes loaded during verification, approximated as the unique-expert count multiplied by the per-expert byte size. For the MoE models studied there, this view aligns with the expected concave speedup curve and limited low-batch gains. @referencesиаа
We ask whether that conclusion continues to hold for recent fine-grained sparse MoE models on modern hardware, and evaluate it across 10 models spanning 30B–1T parameters and 8–512 experts. For each model, we use the supported SD mechanism (EAGLE, EAGLE-3, MTP, or a separate draft model; details in Appendix § 14). Most of these models do not exhibit the predicted concave curve. Instead, many remain unusually efficient at low batch sizes and peak at or near batch size 1, precisely where the activated-expert-bytes view predicts limited SD gains for MoE targets.
This discrepancy points to two mechanisms missing from the activated-expert-bytes view:
Temporal overlap in MoE routing across verification positions. In fine-grained MoE models [8, 37], adjacent positions share 20–45% of their activated experts, far above the 4–12% predicted by an independence baseline (which assumes experts are selected independently across positions given routing probabilities). This temporal overlap reduces the unique expert count during -position verification by 21–34% relative to the independence baseline, and accounts for most of the overlap beyond the uniform baseline (which assumes experts are chosen uniformly at random across the expert pool; 69–92%) in high-sparsity models.
GPU execution dynamics. The activated-expert-bytes view ignores hardware-level effects that decouple parameter size from runtime, including memory hierarchy behavior (e.g., L2 cache reuse), kernel efficiency, and achieved FLOPs/bandwidth. For example, at batch size 1, activated-expert bytes are a poor proxy for latency: for Qwen3-30B FP8 on H100, MoE kernels reach 45–49% of peak HBM bandwidth while attention kernels reach only 22–25%, so MoE layers account for only 33% of forward-pass time despite representing 67% of model weights. When co-routed positions re-read the same small expert shard, L2 serves the repeated reads: in the same setting, the MoE L2 hit rate rises from 5.72% in AR decoding to 27.79% under SD verification. The low MoE runtime share also holds on B200 (35%), although the L2 hit rate there does not rise under SD: the specific mechanisms depend on the hardware and kernel, but the failure of activated-expert bytes as a cost proxy persists.
Taken together, these results show that SD performance in sparse MoE models is governed not only by activated-expert bytes, but also by temporal routing overlap and GPU execution dynamics. This predictive capability is useful in practice because evaluating SD-relevant target and draft architecture choices by direct training-and-benchmarking is expensive, especially at frontier scales. We therefore introduce a maximum-entropy framework based on Iterative Proportional Fitting (IPF) that uses empirical pairwise overlap statistics to predict unique expert counts within 4% error and reproduce observed speedup curves under synthetic routing. The same framework enables train-free screening of target and draft design tradeoffs before full training, including sparser targets and MoE-versus-dense draft models.
Contributions.
We provide a comprehensive empirical study of SD on 10 sparse MoE models spanning 30B–1T parameters and 8–512 experts, and show that fine-grained MoEs remain unusually efficient at low batch sizes and might even peak near batch size 1, contrary to the concave-curve prediction of the activated-expert-bytes view.
We quantify temporal routing overlap across draft positions, tasks, and languages, and show that consecutive positions in fine-grained MoE models reuse substantially more experts than an independent-routing baseline predicts. This overlap persists for rejected draft tokens and reduces the unique expert count during SD verification (§6).
We show that activated-expert bytes are a poor proxy for runtime at batch size 1 on both H100 and B200, and use detailed GPU profiling to identify the H100 mechanisms, HBM-bandwidth utilization gaps between MoE and attention kernels and L2 cache reuse for small expert shards, and to show that these mechanisms are kernel-dependent (§7).
We introduce a maximum-entropy IPF framework that predicts unique expert counts within 4% error and generates realistic token-to-expert assignments. Synthetic-routing experiments reproduce the observed speedup curve, and the same framework enables train-free screening of target and draft architecture tradeoffs, including sparser targets and MoE-versus-dense draft layers, before additional training (§8, Appendix M.3).
Related Work
MoE architectures and routing locality. Sparse MoE architectures trace back to sparsely-gated MoE [44] and later scaling and routing designs such as GShard [30], BASE Layers [32], Switch Transformers [17], GLaM [15], and Expert Choice Routing [57]. More recent language-model variants increasingly use finer-grained and shared-expert designs [28, 8, 24, 11, 55, 16]. Prior work [40, 3] shows that expert co-activation is influenced by semantic and positional similarity, and ExpertFlow [20] exploits this locality for memory-constrained deployment. We build on this line of work by quantifying how temporal locality across consecutive draft tokens changes speculative-verification cost.
Speculative decoding. Early work [47] introduced a prefix-validation approach for parallel autoregressive decoding. Modern speculative decoding methods differ mainly in how they construct and verify proposals: draft-model and seq2seq variants [31, 5, 53], retrieval- or tree-based proposals [21, 49, 38, 50], extra decoding heads and multi-token prediction [4, 2, 19, 11, 55], and higher-capacity or parallel drafting variants [34, 35, 36, 18, 25, 22].
Recent work has shown that SD can remain effective outside the traditional small-batch dense regime, but for different reasons. MagicDec [42] shows that in long-context serving, large batches can again favor SD because KV-cache traffic dominates verification cost. The closest prior work for MoE is MoESD [23], which models verification under an activated-expert-bytes view and accordingly predicts concave speedup curves with limited gains at low batch sizes. Because memory constraints typically force batch size down as sequence length grows, this low-batch regime is increasingly relevant for the longer-context, long-tailed RL rollout, and agentic workloads emerging in practice. Our work revisits this question for modern sparse MoEs and identifies a distinct regime in which fine-grained MoEs remain effective even at low batch sizes: temporal routing overlap and GPU execution dynamics yield higher speedup than the activated-expert-bytes view predicts.
Background and Experimental Setup
Each SD iteration has two phases: (1) a draft model proposes tokens, and (2) the target model verifies these draft tokens together with the next-token continuation in a single pass over positions. SD helps Time per output token (TPOT) only when the savings from fewer target-model calls exceed the overhead of verification—i.e., when acceptance is high and verification is cheap. In MoE models, this verification cost depends not only on model size but also on routing behavior, because verifying multiple tokens may activate overlapping sets of experts across the positions.
Because different SD mechanisms have different acceptance behavior, we isolate the target-side verification cost using target efficiency (TE) [23],
where denotes the latency of a single-token autoregressive (AR) decoding step on the target model and denotes the latency of the target model’s verification pass over positions in speculative decoding (SD). A higher TE indicates that the SD verification pass is closer in cost to a standard AR decoding step. Because TE is independent of acceptance length, it lets us compare target-side verification efficiency across draft mechanisms and model architectures.
The 10 evaluated MoE models span sparsity ratios (k/N, top-k over total experts) from 0.25 (Mixtral) to 0.020 (Qwen3-Next-80B), and per-expert weights from 4.7 MB to 176 MB in FP8, a 37× range that proves critical for GPU execution efficiency (§5); Appendix Table 6 lists the full architectural details and SD methods. All experiments use draft tokens, vLLM v1 (commit 7c080dd3c) [29], and MT-Bench [56] as the base dataset; Appendix § 14 provides details on the sweep schedule, token output lengths, and disabled prefix caching, § 9.1 and § 9.2 evaluate larger and long-context serving, and Appendix § 13 discusses the remaining scope of the and hardware assumptions. All models run on NVIDIA H100 SXM GPUs except for Kimi-K2.5 and DeepSeek-V3 which use NVIDIA B200 GPUs.
Speedup Curve Regimes
We begin by characterizing the observed speedup regimes. Figure 1 shows TPOT speedup () versus batch size for four representative models, where TPOT denotes wall-clock time per output token. Appendix § 15 shows all the 10-grid. MoESD [23] predicts a concave speedup curve that peaks at intermediate batch sizes, where expert activation saturates. Under this view, speedup at low batch sizes remains limited because verification must load additional experts. We observe a concave pattern in less sparse MoEs with larger per-expert weights (e.g., Qwen2-57B, Mixtral-8x7B). In contrast, sparser models with smaller experts (e.g., Qwen3-235B, Kimi-K2.5) exhibit non-concave curves, often peaking at batch size 1. The non-concave behavior is also not specific to MT-Bench: Appendix § 16 shows that Qwen3-30B and Qwen3-235B retain qualitatively similar speedup curves across multiple datasets (Figure 7).

Figure 1. TPOT speedup versus batch size for four representative MoE models. Mixtral-8x7B exhibits the predicted concave curve, whereas the other three are non-concave and peak at or near batch size 1.


Figure 7. (a) Qwen3-30B (b) Qwen3-235B
Figure 7: TPOT speedup versus batch size across four datasets for two representative non-concave models. Each curve corresponds to InstructCoder [33], MT-Bench [56], ShareGPT [6], or SpecBench [54]. Although the absolute speedups vary by dataset, both models retain the same qualitative non-concave, batch-size-1-peaked behavior.
We find that the speedup-curve shape closely tracks target efficiency at batch size 1, which measures how expensive it is for the target model to verify tokens instead of decoding a single token. Lower target efficiency leaves more headroom for the classic concave rise, whereas higher target efficiency leads to non-concave behavior, as shown in Figure 6. Qwen3.5-35B (bf16, TP2) is a notable exception: it exhibits non-concave behavior despite having lower target efficiency than GLM4.7-Flash (bf16, TP1). Table 1 reports the measured target efficiency for the main-text configurations alongside the activated-expert-bytes prediction; Appendix Table 10 reports the full cross-model measurements across SD methods, TP settings, and precisions. We find that target efficiency varies substantially even within a single model family as we change SD method, TP, and precision. In general, increasing TP or using lower precision increases target efficiency at batch size 1 and shift peak speedup toward smaller batch sizes. We analyze these effects in more detail in §5.
| Model | (MB/GPU) | (MB/GPU) | MoE wt% | TE pred. | TE observed | Gap | |
| Mixtral-8x7B (bf16, TP2) | 168.0 | 40.0 | 89.4 | 4.99 | 0.428 | 0.505 | +0.08 |
| Mixtral-8x7B (fp8, TP1) | 168.0 | 40.0 | 89.4 | 4.99 | 0.428 | 0.488 | +0.06 |
| Qwen3-30B (bf16, TP1) | 9.0 | 36.0 | 66.7 | 19.49 | 0.511 | 0.630 | +0.12 |
| Qwen3-30B (bf16, TP2) | 4.5 | 18.0 | 66.7 | 19.49 | 0.511 | 0.697 | +0.19 |
| Qwen3-30B (fp8, TP1) | 4.5 | 18.0 | 66.7 | 19.49 | 0.511 | 0.818 | +0.31 |
| Qwen3-30B (fp8, TP2) | 2.25 | 9.0 | 66.7 | 19.49 | 0.511 | 0.860 | +0.35 |
| Qwen3-235B (bf16, TP8) | 4.5 | 18.0 | 66.7 | 19.01 | 0.522 | 0.786 | +0.26 |
| Qwen3-235B (fp8, TP4) | 4.5 | 17.0 | 67.9 | 19.01 | 0.517 | 0.889 | +0.37 |
Table 1. TE predicted by the activated-expert-bytes model vs. measured TE at batch size 1 using per-GPU weights.

Figure 6. TPOT speedup versus batch size for all 10 MoE models ().
Limits of the Activated-Expert-Bytes View
At low batch size , decoding processes a single token per sequence, making the linear layers largely memory-bandwidth-bound in the roofline sense [52]: execution time is therefore dominated by loading weights from HBM rather than by compute. A first-principles estimate therefore assumes GPU time is proportional to weight bytes:
where is the per-GPU dense weight bytes (attention + shared expert), is the per-expert weight bytes, and is the empirical unique expert count during SD verification. Even when using the empirical measured from actual MT-Bench routing, this activated-expert-bytes model systematically underestimates target efficiency (see Table 1). The gap ranges from +0.06 (Mixtral, large experts) to +0.37 (Qwen3-235B, small experts). Weight bytes alone do not explain the GPU execution time at batch size 1. This activated-expert-bytes estimate omits critical model behavior and hardware interactions which we will discuss in detail in the following sections.
Temporal Overlap in MoE Expert Routing
To explain why verification cost stays low, we analyze expert overlap across consecutive tokens using three levels that isolate the contribution of temporal overlap; formal definitions and derivations are given in Appendix § 19:
Uniform baseline (): assumes all experts equally likely and tokens independent.
Independence baseline: uses the empirical per-layer expert probabilities while assuming tokens are routed independently. This captures savings from non-uniform expert popularity.
Empirical: measured from consecutive tokens on a real dataset, preserving temporal overlap.
We report routing-overlap statistics for the models in our SD setup; details in Appendix § 20. Appendix § 21 shows overlap for tokens separated by positions, where step denotes a separation of tokens. Across the eight measured models, mean empirical overlap declines from 35.1% at step 1 to 26.4% at step 3, while remaining , , and above the mean independence baseline of 10.5% at steps 1–3, respectively.
Table 2 reports step-1 overlap ratios, defined as the fraction of top- experts shared between adjacent tokens along with its decomposition, for four representative models; a broader model analysis is provided in Appendix § 19, Table 13. All models shows empirical overlap far above both the uniform baseline and the independence baseline. The temporal overlap beyond the independence baseline dominates in 6 of 8 models, contributing 69–92% of total overlap. Mixtral () and Qwen2-57B () are exceptions: their large combinatorial baseline leaves less room for temporal effects. We defer per-layer patterns to Appendix § 22.
| Model | Uniform | Indep | Empirical | Temporal | Temp/Total | |
| Mixtral-8x7B | 0.250 | 0.250 | 0.253 | 0.355 | 0.102 | 28.7% |
| Qwen3-30B | 0.063 | 0.063 | 0.119 | 0.431 | 0.312 | 72.4% |
| Qwen3-235B | 0.063 | 0.063 | 0.098 | 0.447 | 0.349 | 78.2% |
| Kimi-K2.5 | 0.021 | 0.021 | 0.040 | 0.296 | 0.256 | 86.4% |
Table 2. Step-1 expert overlap decomposition. Temporal = empirical − independence; Temporal/Total = fraction of overlap attributable to temporal overlap.
| Model | Uniform | Indep | Empirical | Temporal | Temp/Total | |
| Mixtral-8x7B | 0.250 | 0.250 | 0.253 | 0.355 | 0.102 | 28.7% |
| Qwen2-57B | 0.125 | 0.125 | 0.140 | 0.204 | 0.064 | 31.5% |
| GLM-4.7-Flash | 0.063 | 0.063 | 0.097 | 0.319 | 0.222 | 69.4% |
| Qwen3-30B | 0.063 | 0.063 | 0.119 | 0.431 | 0.312 | 72.4% |
| Qwen3-235B | 0.063 | 0.063 | 0.098 | 0.447 | 0.349 | 78.2% |
| Qwen3.5-35B | 0.031 | 0.031 | 0.055 | 0.332 | 0.277 | 83.4% |
| Kimi-K2.5 | 0.021 | 0.021 | 0.040 | 0.296 | 0.256 | 86.4% |
| Qwen3-Next-80B | 0.020 | 0.020 | 0.036 | 0.420 | 0.384 | 91.6% |
Table 13. Step-1 expert overlap decomposition for all measured models. Temporal = empirical − independence; Temporal/Total = fraction of overlap attributable to temporal overlap.
Applicability to draft tokens.
The overlap measurements above use accepted continuations. If rejected draft tokens were routed to substantially different experts, accepted-token overlap would mischaracterize the routing patterns encountered during SD verification. In aggregate MT-Bench, however, rejected tokens share a similar fraction of top- experts with the preceding token, and in some cases slightly more. The deconfounded accepted-vs.-first-rejection comparison in Appendix H shows the same pattern for Qwen3-235B at step 1, where first-rejection tokens have higher overlap than accepted tokens (0.488 vs. 0.447; Table 14). We therefore treat accepted-token overlap as a conservative proxy for aggregate verification cost.
| Model | Step | Acc | 1st Rej | ||
| Qwen3-30B | 1 | 0.433 | 0.441 | −0.008 | 4991 |
| Qwen3-30B | 2 | 0.409 | 0.448 | −0.039 | 3021 |
| Qwen3-30B | 3 | 0.392 | 0.420 | −0.028 | 1374 |
| Qwen3-235B | 1 | 0.447 | 0.488 | −0.040 | 8420 |
| Qwen3-235B | 2 | 0.422 | 0.475 | −0.053 | 5936 |
| Qwen3-235B | 3 | 0.407 | 0.458 | −0.051 | 3752 |
| Qwen3-Next-80B | 1 | 0.430 | 0.455 | −0.025 | 5088 |
| Qwen3-Next-80B | 2 | 0.419 | 0.449 | −0.031 | 5376 |
| Qwen3-Next-80B | 3 | 0.401 | 0.427 | −0.026 | 4932 |
| Kimi-K2.5 | 1 | 0.303 | 0.308 | −0.005 | n/a |
| Kimi-K2.5 | 2 | 0.292 | 0.305 | −0.013 | n/a |
| Kimi-K2.5 | 3 | 0.280 | 0.292 | −0.012 | n/a |
| Mixtral-8x7B | 1 | 0.366 | 0.359 | +0.007 | 6076 |
| Mixtral-8x7B | 2 | 0.366 | 0.347 | +0.020 | 4342 |
| Mixtral-8x7B | 3 | 0.369 | 0.339 | +0.031 | 2368 |
Table 14. Deconfounded consecutive overlap (accepted vs. first-rejection). Negative means rejected tokens have higher overlap.
Sub-Linear Verification Cost
The practical consequence for SD is that verification loads far fewer unique experts than the worst case predicts. Table 3 compares the per-layer unique expert count when verifying tokens under all three baselines for four representative models. For Qwen3-30B (, ), verifying four tokens loads only unique experts per layer, rather than the predicted under uniform random routing. This 33% reduction directly lowers the MoE forward-pass cost during SD verification.
| Model | Unif | Indep | Emp | Emp/ | Uniform IPF | Err | Skewed IPF | Err | ||
| Mixtral-8x7B | 2 | 8 | 5.5 | 5.5 | 5.0 | 2.5× | 4.99 | +0.1% | 4.99 | +0.1% |
| Qwen2-57B | 8 | 64 | 26.5 | 26.0 | 24.6 | 3.1× | 24.72 | +0.5% | 24.63 | +0.1% |
| Qwen3-30B | 8 | 128 | 29.1 | 26.9 | 19.5 | 2.4× | 20.08 | +3.1% | 19.72 | +1.2% |
| Kimi-K2.5 | 8 | 384 | 31.0 | 30.2 | 23.3 | 2.9× | 23.88 | +2.7% | 23.66 | +1.7% |
Table 3. Unique experts () and IPF prediction accuracy. Emp = empirical; Unif = uniform.
Two sources contribute to the savings. The step from the uniform to the independence baseline, which captures expert popularity, is modest: at most 8% (typically 2–5%) across the measured models (Table 20). The step from the independence baseline to the empirical measurement, which captures temporal overlap beyond independent routing, is dominant: a further 21–34% reduction for the six high-sparsity models (). Models with coarse routing (Mixtral, Qwen2-57B) see smaller temporal savings because the independence baseline is already close to the empirical value.
| Model | Unif | Indep | Emp | Emp/ | Uniform IPF | Err | Skewed IPF | Err | ||
| Mixtral-8x7B | 2 | 8 | 5.5 | 5.5 | 5.0 | 4.99 | +0.1% | 4.99 | +0.1% | |
| Qwen2-57B | 8 | 64 | 26.5 | 26.0 | 24.6 | 24.72 | +0.5% | 24.63 | +0.1% | |
| GLM-4.7-Flash | 4 | 64 | 14.6 | 13.9 | 11.0 | 11.16 | +1.6% | 11.07 | +0.8% | |
| Qwen3-30B | 8 | 128 | 29.1 | 26.9 | 19.5 | 20.08 | +3.1% | 19.72 | +1.2% | |
| Qwen3-235B | 8 | 128 | 29.1 | 27.7 | 19.0 | 19.67 | +3.5% | 19.45 | +2.3% | |
| Qwen3.5-35B | 8 | 256 | 30.5 | 29.5 | 21.9 | 22.68 | +3.4% | 22.48 | +2.5% | |
| Kimi-K2.5 | 8 | 384 | 31.0 | 30.2 | 23.3 | 23.88 | +2.7% | 23.66 | +1.7% | |
| Qwen3-Next-80B | 10 | 512 | 38.8 | 37.9 | 25.2 | 26.40 | +4.7% | 26.16 | +3.7% |
Table 20. Unique experts (, four-token window) and IPF prediction accuracy for the measured models.
This property appears largely data-agnostic for Qwen3-30B: across 13 Spec-Bench [54] task categories, the step-1 overlap ratio varies by only around the MT-Bench value of 0.431, and across seven Aya multilingual splits the full decay curve shifts upward slightly rather than weakening (Appendix I.1). These results suggest that sub-linear scaling holds across a broad range of realistic workloads.
Predicting Unique Experts and Validating the Framework
We develop a maximum-entropy Iterative Proportional Fitting (IPF) [13, 7] framework that predicts unique expert counts from pairwise routing-overlap statistics alone, without running inference. Each expert’s activation across consecutive tokens is modeled as correlated binary variables , with constraints
where denotes the empirical probability that expert is selected on a token (or in the uniform variant), assumed to be position-independent across the four-token window for simplicity, and denotes the fitted joint distribution over that expert’s activation pattern across the four-token window. IPF solves for the maximum-entropy joint distribution over states, from which the expected unique count follows as .
We use uniform IPF to denote the variant that assumes uniform expert probabilities while matching pairwise overlap across consecutive tokens. We use skewed IPF to denote the variant that uses empirical per-layer expert probabilities together with the same overlap ratios; this is the predictor we use throughout. Skewed IPF predicts empirical unique-expert counts within 4% for every measured model (maximum +3.7%, Qwen3-Next-80B), and uniform IPF within 5% (maximum +4.7%) (Appendix Table 20; Table 3 shows four representative models). Details of the IPF formulation and synthetic routing generation are in Appendix § 24.
Interaction of GPU execution of sparse MoE on Speculative Decoding
The preceding analysis shows that SD verification activates substantially fewer unique experts than an independence-based routing model would predict. However, temporal overlap alone does not fully explain the gap between activated-expert-bytes estimates and measured verification cost at batch size 1. The realized cost of loading experts also depends on how sparse MoE kernels execute on GPU hardware. We therefore turn to GPU execution dynamics, which govern how expert loading translates into observed latency through bandwidth utilization and cache behavior.
We profile Qwen3-30B and Mixtral at batch size 1 on H100 using Nsight Systems (kernel timing) and Nsight Compute (hardware counters), and repeat the profiling for Qwen3-235B on H100 and for Qwen3-30B and Mixtral on B200 (Appendix § 18). We focus on FP8 variants because they exhibit a larger gap and better expose hardware-level effects.
We summarize the metrics and hardware terms below.
| Metrics | Hardware terms | ||
| SD scaling | CTA | cooperative thread array (thread block) | |
| MoE scaling | SM | streaming multiprocessor (H100: 132 SMs) | |
| Attention scaling | HBM BW | high-bandwidth memory bandwidth | |
| L2 hit rate | fraction of accesses served by L2 (H100: 50MB) |
Table 4. Metrics and hardware terms used in profiling.
On H100, Table 5 summarizes the key measurements and reveals two mechanisms behind the gap: small experts (Qwen3-30B) create large disparities in bandwidth utilization and L2 cache behavior between attention and MoE kernels, while large experts (Mixtral) largely eliminate both.
| Metric | Qwen3-30B | Mixtral-8x7B |
| Expert size (FP8) | 4.5 MB | 168 MB |
| MoE share of weight | 66.7% | 89.4% |
| MoE share of GPU time | 33% | 75% |
| Attn CTAs / 132 SMs | 16 (12%) | 96 (73%) |
| Attn HBM BW (% peak) | 22–25% | 62% |
| MoE CTAs / 132 SMs | 96–128 | 132 |
| MoE HBM BW (% peak) | 45–49% | 79% |
| Attn SD scaling | 1.02 | 1.01 |
| MoE SD scaling | 1.57 | 2.19 |
| L2 hit rate (AR→SD) | 5.72%→27.79% | 1.68%→1.7% |
| TE pred. | 0.51 | 0.43 |
| TE observed | 0.82 | 0.49 |
| Weight model gap | +0.31 | +0.06 |
Table 5. GPU profiling comparison at batch size 1 on H100.
L2 cache reuse across co-routed experts
The activated-expert-bytes model assumes that MoE cost scales linearly with the number of unique experts loaded. At batch size 1, the default H100 MoE kernel processes each (token, expert) pair in its own thread block, so when multiple verification tokens are routed to the same small expert (Section 6), its weights are read once per token and the repeated reads are served from L2 rather than HBM. For Qwen3-30B, the L2 hit rate rises from 5.72% to 27.79% for 4.5 MB experts, whose working set partially fits within the H100’s 50 MB L2 cache. For Mixtral, the kernel groups co-routed tokens by expert under SD, so each expert is read once and there are no repeated reads to absorb (1.68% in both modes); its 168 MB experts would far exceed L2 in any case. Without L2, these repeated reads would make HBM traffic grow with the number of (token, expert) pairs rather than with the number of unique experts; L2 brings it back close to the unique-expert bytes that the activated-expert-bytes model assumes. The lower of Qwen3-30B (1.57 vs. 2.19 for Mixtral, despite a similar 2.4–2.5× unique-expert multiple; Section 6) therefore likely stems mainly from bandwidth utilization, discussed below.
Factors affecting L2 cache reuse. Table 1 shows that, within a model family, higher tensor parallelism and lower-precision weights tend to increase target efficiency. Qwen3-30B illustrates the effect concretely: target efficiency rises from 0.630 to 0.697 when TP increases from 1 to 2 at BF16, and from 0.630 to 0.818 when weights are quantized from BF16 to FP8 at TP1; the combined FP8, TP2 configuration reaches 0.860. One contributing mechanism is that tensor parallelism reduces the per-GPU expert shard size, while quantization further shrinks each shard, allowing more of the co-routed expert working set to stay in L2 between repeated reads. By contrast, Mixtral’s experts are so large that even a single expert does not fit in L2, so the scope for additional reuse is limited and the improvement remains small across the available EAGLE configurations. Appendix Figure 5 shows the corresponding Qwen3-30B configuration sweep.
HBM bandwidth utilization disparity. At batch size 1, FP8 attention GEMM kernels launch far fewer CTAs than FP8 MoE kernels, leaving many SMs idle and therefore achieving substantially lower HBM bandwidth (Table 5). This pattern is consistent with prior work on attention kernels, which emphasizes IO-aware scheduling and improved work partitioning [10, 9]. As a result, MoE accounts for less GPU time than its weight share would suggest. For Qwen3-30B, the roughly bandwidth advantage of MoE GEMMs over attention GEMMs compresses the MoE contribution from 67% of weights to 33% of runtime. For Mixtral, MoE and attention GEMMs achieve more similar bandwidth, so the MoE runtime share remains closer to its weight share. Verifying tokens barely increases attention cost () because KV-cache reads are reused across consecutive draft tokens, making MoE the principal source of SD slowdown in this regime. Consequently, the smaller MoE runtime share in Qwen3-30B makes SD verification faster than the activated-expert-bytes model would predict. For Mixtral, larger matrices enable more CTAs for attention, narrowing both the bandwidth gap and the activated-expert-bytes-model error.
Expert size as the unifying factor. On H100, both mechanisms are governed primarily by expert size. Large experts achieve high bandwidth utilization and exceed L2 capacity, so execution time remains approximately proportional to weight bytes and the activated-expert-bytes model is relatively accurate. Small experts leave bandwidth underutilized, making execution time substantially smaller than the activated-expert-bytes model would predict, while L2 absorbs repeated reads of co-routed experts. A full three-step correction chain that raises the estimate from 0.51 to 0.872 for Qwen3-30B is provided in Appendix § 17. Qwen3-235B (FP8, TP4) reproduces the same mechanisms on H100, with the largest weight-model gap among the profiled models (+0.38; Appendix § 18).
End-to-end validation via synthetic routing.
Section 6 and Section 7 show that both temporal overlap and GPU execution dynamics shape the SD speedup curves of MoE models. We therefore conduct synthetic experiments that combine these two factors to predict the speedup curves without using trained weights. The only assumption is access to reasonable expert-overlap statistics, obtained either from a model in the same family or from the averages in Table 17. To evaluate the IPF framework from § 6.3 and its synthetic-routing construction from Appendix § 24, we replace the model’s learned routing with IPF-generated synthetic routing while keeping all other inference settings fixed (dummy weights, same hardware). Because this method runs the real kernels on the target hardware, it captures hardware- and kernel-specific execution effects, such as those that differ between H100 and B200, without modeling them explicitly. We define the proxy target efficiency as , where = forward pass + logits + sampling, and = forward pass + logits + rejection sampling + draft.
| Model | Step 1 | Step 2 | Step 3 | Step 4 | Indep |
| Mixtral-8x7B | 0.355 | 0.303 | 0.298 | 0.288 | 0.253 |
| Qwen2-57B | 0.204 | 0.150 | 0.149 | 0.142 | 0.140 |
| GLM-4.7-Flash | 0.319 | 0.277 | 0.245 | 0.233 | 0.097 |
| Qwen3-30B | 0.431 | 0.356 | 0.328 | 0.312 | 0.119 |
| Qwen3-235B | 0.447 | 0.378 | 0.354 | 0.337 | 0.098 |
| Qwen3.5-35B | 0.332 | 0.272 | 0.243 | 0.225 | 0.055 |
| Kimi-K2.5 | 0.296 | 0.223 | 0.194 | 0.179 | 0.040 |
| Qwen3-Next-80B | 0.420 | 0.335 | 0.302 | 0.281 | 0.036 |
Table 17. Overlap ratio by step distance for the measured models. The independence baseline (last column) indicates the overlap expected in the absence of temporal effects.
We measure the proxy target efficiency under three routing assumptions: uniform random routing, uniform IPF routing with the same routing-overlap statistics in every MoE layer, and per-layer skewed IPF routing. Figure 2 compares these three settings. The uniform-random baseline consistently underestimates the target efficiency at small batch sizes, and the error is larger for sparser MoE models. Per-layer IPF synthetic routing closely tracks the real-model curves at small batch sizes and remains closer than uniform random routing across the full batch-size range. For Qwen3-30B (Figure 2b), we show all three routing variants for comparison. Although the skewed IPF variant is more accurate, the uniform IPF variant still captures the qualitative batch-size trends of the real-model SD speedup curves.

Figure 2. (a) Mixtral-8x7B bf16 (b) Qwen3-30B fp8 (c) Qwen3-235B fp8. Target efficiency under different routing strategies.
Residual gaps between the IPF-generated synthetic routing and the real-model curves are expected because the synthetic routing does not exactly reproduce the full routing distribution of the real model. In particular, higher-order statistics not captured by the pairwise routing-overlap constraints and position-independent per-expert marginals in Eq. (2) may still affect performance. Nevertheless, predicting these curves without training a model requires only uniform IPF together with routing-overlap statistics taken either from the same model family or from the averages in Table 17. These results show that routing-overlap statistics captured by the IPF framework substantially reduce the gap between synthetic and real-model curves relative to the uniform-random baseline, while preserving the relevant hardware-execution dynamics.
Discussion
Effect of the Speculative Window
Our main experiments fix , a common production setting [51, 39, 41]. To test whether the observed curve shapes depend on this choice, we repeat the end-to-end sweep at for one model from each regime: Mixtral-8x7B (FP8, TP1, EAGLE), Qwen3-30B (FP8, TP2, EAGLE-3), and Qwen3-235B (FP8, TP4, EAGLE-3), all on H100 with MT-Bench (Figure 3). For all three models, the curve family is the same at every : Qwen3-30B and Qwen3-235B remain non-concave and peak at batch size 1, whereas Mixtral remains concave, with its best speedup near batch size 16. Our curve taxonomy (§4) is therefore not an artifact of .

Figure 3. (a) Mixtral-8x7B (FP8, TP1) (b) Qwen3-30B (FP8, TP2) (c) Qwen3-235B (FP8, TP4). SD speedup vs. batch size for on MT-Bench (H100). Each model keeps its curve shape across ; larger mostly shifts the curve down, except for Qwen3-235B, whose stronger drafter keeps competitive.
Long-Context Inference
Our main sweeps use short MT-Bench prompts. To test the long-context regime, where KV-cache traffic becomes increasingly dominant, we repeat the end-to-end sweep for Qwen3-30B (FP8, TP2, EAGLE-3, ) on the throughput_32k split of SPEED-Bench [1], which has prompts of about 32k tokens, with 4096-token outputs (Figure 4). The KV cache limits the maximum batch size to 24. Target efficiency is higher than on MT-Bench at every batch size. This is consistent with KV-cache reads, which both vanilla decoding and SD verification incur, diluting the relative cost of the extra expert weights loaded during verification. The curve also remains non-concave: target efficiency is highest at batch size 1, dips at batch size 4, and recovers at larger batch sizes. Long context thus makes verification relatively cheaper for the MoE target; MagicDec [42] studies this regime in depth.

Figure 4. Target efficiency vs. batch size for Qwen3-30B (FP8, TP2, EAGLE-3, ) on H100: MT-Bench vs. SPEED-Bench 32k.
Implications for MoE Draft Layers
Current SD drafters commonly use a dense transformer layer. In Appendix § 25, we use the framework from Section 8 to analyze the effect of an MoE drafter when the target model is also an MoE model. The analysis suggests that MoE draft layers may become attractive in sufficiently sparse regimes. At low-to-medium batch sizes, an MoE drafter does not improve target efficiency, but if trained effectively it could still improve end-to-end speedup through longer acceptance lengths enabled by its additional parameters. At larger batch sizes, the MoE drafter becomes more attractive because it remains less compute-bound than a dense drafter when the target model itself is MoE. To probe a sparser setting than Qwen3-30B, we also simulate a 896-expert architecture (<1% activations) and show in Appendix § 25.3 that the MoE drafter becomes more favorable as the target model remains bandwidth-bound. Because this analysis is synthetic and does not measure acceptance-length gains from trained MoE drafters, we interpret it as a design study rather than direct evidence for trained MoE drafters.
Routing Inertia in Rejected Draft Tokens
Expert routing does not cleanly separate accepted from rejected draft tokens in a way that would support early termination. In the aggregate MT-Bench analysis, fine-grained models show comparable or slightly higher overlap for rejected tokens, and held-out Spec-Bench and Aya [45] slices preserve the same tendency in median, with some task- and language-specific reversals (Appendix § 20.8). Even in the largest stable aggregate effect (Qwen3-235B, Cohen’s ), the accepted and rejected distributions still overlap by , and most held-out slices remain negligible or small in effect size. These results suggest that routing-based rejection predictors are unlikely to be useful in practice: routing inertia is statistically robust but offers limited leverage for early-exit decisions.
Conclusion
As sparse MoE models move into increasingly memory-constrained inference regimes, an important question is whether speculative decoding remains worthwhile at low batch size. Across 10 recent sparse MoE models (30B–1T parameters, 8–512 experts), we find that the previously expected concave SD speedup curve does not characterize most modern sparse models: many attain high speedup at or near batch size 1. We show that verification cost in sparse MoE models is not determined by activated-expert bytes alone, but is also shaped by temporal overlap across consecutive tokens and by GPU execution dynamics, whose specific mechanisms vary across hardware and kernels. A maximum-entropy IPF framework predicts unique expert counts from routing-overlap statistics with less than 4% error and, under synthetic routing with fixed execution conditions, substantially narrows the gap to real-model curves while capturing the qualitative low-batch trend, thereby providing a train-free way to study target and draft architecture tradeoffs. Taken together, these results suggest that SD performance on sparse MoE models should be modeled using both routing-overlap statistics and GPU execution dynamics, and that fine-grained MoE designs may be more compatible with low-batch speculative inference than previously believed.
Acknowledgments and Disclosure of Funding
This work was funded by Cohere, where all authors were employed. We thank Zewen Shen for their helpful feedback on the paper.
References
Limitations
Our synthetic-routing analysis is approximate. The IPF construction matches pairwise overlap statistics, but does not model higher-order routing structure, position dependence in expert marginals, or how temporal overlap itself varies with architectural choices. In practice, one may use overlap statistics from a closely related model family, or aggregate statistics from the models we study, as a better train-free estimate than a uniform baseline when exploring target and draft design tradeoffs before training. However, understanding which architectural factors determine temporal overlap, and how reliably overlap transfers across model families and sparsity patterns, remains an important direction for future work.
Experimental Setup
All reported experiments use vLLM v1 (commit 7c080dd3c) with draft tokens (except the -sweep in § 9.1, which uses ) and MT-Bench as the base dataset. For the routing analyses (Section 6 and Appendix § 20), we build on this commit and extend its enable_return_routed_experts routing capture to also record target-model routing at rejected draft positions after each verification step. For the end-to-end SD sweep, the benchmark client oversamples the 80 MT-Bench prompts with replacement to match the configured number of requests at each concurrency setting, uses fixed 256-token outputs, and runs with prefix caching disabled. Every end-to-end speedup measurement was repeated three times in independent server sessions, and the coefficient of variation (CV, std/mean) of the speedup is below 5% at every batch size for every configuration (e.g., at most 1.4% for Mixtral-8x7B FP8 TP1 and Qwen3-30B FP8 TP2, and at most 3.3% for Qwen3-235B FP8 TP4). Because the acceptance length is nearly constant across runs, target efficiency (speedup divided by acceptance length) inherits the same low variability; figures and tables report a single run. Table 6 summarizes the evaluated MoE models and their SD methods, and Table 7 summarizes the benchmark schedule. All models run on NVIDIA H100 SXM GPUs except for Kimi-K2.5 and DeepSeek-V3 which use NVIDIA B200 GPUs. Tables 8 and 9 list the Hugging Face model identifiers, draft weights, SD method, tensor parallelism, and GPU used for each reported configuration. All quantized configurations (FP8, INT4, and MXFP4) use the quantization recipe stored in the corresponding published Hugging Face checkpoint, loaded as-is by vLLM; we apply no additional quantization.
| Model | Hidden | Layers | TP | SD Method | |||
| Mixtral-8x7B | 2/8 | 14336 | n/a | 4096 | 32 | 2 | EAGLE |
| Qwen2-57B | 8/64 | 2560 | 20480 | 3584 | 28 | 2 | Draft model (0.5B) |
| GLM-4.7-Flash | 4/64 | 1536 | 10240 | 2048 | 40 | 1 | MTP |
| Qwen3-30B | 8/128 | 768 | n/a | 2048 | 48 | 1 | EAGLE-3 |
| Qwen3-235B | 8/128 | 1536 | n/a | 4096 | 94 | 4 | EAGLE-3 |
| GPT-OSS-120B | 4/128 | 2880 | n/a | 2880 | 56 | 4 | EAGLE-3 |
| DeepSeek-V3 | 8/256 | 2048 | 18432 | 7168 | 61 | 8 | MTP |
| Kimi-K2.5 | 8/384 | 2048 | 18432 | 7168 | 61 | 8 | EAGLE-3 |
| Qwen3.5-35B | 8/256 | 512 | 512 | 2048 | 64 | 2 | MTP |
| Qwen3-Next-80B | 10/512 | 512 | 512 | 2048 | 80 | 4 | MTP |
Table 6. MoE models evaluated. Here denotes the total number of experts, the number of experts per token, the expert intermediate size, the shared-expert intermediate size (n/a if none), and TP the tensor parallelism.
| Batch size | Num prompts | Batch size | Num prompts |
| 1 | 80 | 16 | 320 |
| 2 | 80 | 32 | 640 |
| 4 | 80 | 64 | 1280 |
| 8 | 160 | 128 | 2560 |
Table 7. Benchmark schedule for the MT-Bench end-to-end SD sweep. The 80 base prompts are oversampled with replacement when the configured number of requests exceeds 80; all runs use fixed 256-token outputs and disable prefix caching.
Consolidated Target Efficiency Measurements
Qwen3-30B configuration and hardware sweep. Figure 5 compares Qwen3-30B speedup curves across BF16/FP8 and TP1/TP2 settings on H100, together with an FP8, TP2 run on B200. The comparison complements Table 10 by showing how the same model family’s batch-size trajectory changes as per-GPU expert shards shrink and as the hardware memory hierarchy changes.
| Model | SD method | Target eff. | Model | SD method | Target eff. |
| Qwen2-57B (bf16, TP2) | Separate draft | 0.270 | Qwen3-30B (fp8, TP2) | EAGLE-3 | 0.860 |
| Mixtral-8x7B (bf16, TP2) | Separate draft | 0.192 | Qwen3-235B (bf16, TP8) | EAGLE-3 | 0.786 |
| Mixtral-8x7B (bf16, TP2) | EAGLE | 0.505 | Qwen3-235B (fp8, TP4) | EAGLE-3 | 0.889 |
| Mixtral-8x7B (fp8, TP1) | EAGLE | 0.488 | Qwen3-Next-80B (bf16, TP4) | MTP | 0.615 |
| GLM-4.7-Flash (bf16, TP1) | MTP | 0.554 | Qwen3-Next-80B (fp8, TP2) | MTP | 0.707 |
| GLM-4.7-Flash (bf16, TP2) | MTP | 0.568 | Qwen3.5-35B (bf16, TP2) | MTP | 0.546 |
| Qwen3-30B (bf16, TP1) | EAGLE-3 | 0.630 | Kimi-K2.5 (int4, TP8) | EAGLE-3 | 0.752 |
| Qwen3-30B (bf16, TP2) | EAGLE-3 | 0.697 | DeepSeek-V3 (fp8, TP8) | MTP | 0.785 |
| Qwen3-30B (fp8, TP1) | EAGLE-3 | 0.818 | GPT-OSS-120B (mxfp4, TP4) | EAGLE-3 | 0.810 |
Table 10. Target efficiency at batch size 1 for . Higher values indicate non-concave speedup curves as observed in Figure 6.

Figure 5. Qwen3-30B TPOT speedup versus batch size across TP/precision settings and hardware. Panels show BF16 TP1 H100, BF16 TP2 H100, FP8 TP2 H100, and FP8 TP2 B200 (126 MB L2).
Full 10-Model Speedup Grid
Figure 6 plots TPOT speedup versus batch size for all 10 models, complementing the four representative examples in Figure 1.
| Model | Target weights | TP | GPU |
| Mixtral-8x7B | mistralai/Mixtral-8x7B-Instruct-v0.1 | 2 | H100 |
| Qwen2-57B | Qwen/Qwen2-57B-A14B-Instruct | 2 | H100 |
| GLM-4.7-Flash | zai-org/GLM-4.7-Flash | 1 | H100 |
| Qwen3-30B | Qwen/Qwen3-30B-A3B-FP8 | 1 | H100 |
| Qwen3-235B | Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 | 4 | H100 |
| GPT-OSS-120B | openai/gpt-oss-120b | 4 | H100 |
| DeepSeek-V3 | deepseek-ai/DeepSeek-V3-0324 | 8 | B200 |
| Kimi-K2.5 | moonshotai/Kimi-K2.5 | 8 | B200 |
| Qwen3.5-35B | Qwen/Qwen3.5-35B-A3B | 2 | H100 |
| Qwen3-Next-80B | Qwen/Qwen3-Next-80B-A3B-Instruct | 4 | H100 |
Table 8. Target-weight identifiers and runtime configuration used in the reported experiments.
| Model | Draft weights | Method |
| Mixtral-8x7B | yuhuili/EAGLE-mixtral-instruct-8x7B | EAGLE |
| Qwen2-57B | Qwen/Qwen2-0.5B-Instruct | Draft model |
| GLM-4.7-Flash | (built-in MTP heads) | MTP |
| Qwen3-30B | RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3 | EAGLE-3 |
| Qwen3-235B | RedHatAI/Qwen3-235B-A22B-Instruct-2507-speculator.eagle3 | EAGLE-3 |
| GPT-OSS-120B | nvidia/gpt-oss-120b-Eagle3-long-context | EAGLE-3 |
| DeepSeek-V3 | (built-in MTP heads) | MTP |
| Kimi-K2.5 | AQ-MedAI/Kimi-K25-eagle3 | EAGLE-3 |
| Qwen3.5-35B | (built-in MTP heads) | MTP |
| Qwen3-Next-80B | (built-in MTP heads) | MTP |
Table 9. Draft-weight identifiers and SD methods used in the reported experiments.
Cross-Dataset Stability of Non-Concave Speedup Curves
Figure 7 shows that the non-concave, batch-size-1-peaked speedup pattern persists across InstructCoder [33], MT-Bench [56], ShareGPT [6], and Spec-Bench [54] for Qwen3-30B and Qwen3-235B. This robustness indicates that the low-batch speedup advantage is not an artifact of the MT-Bench prompt distribution used in the main paper. In turn, it supports the interpretation that the favorable low-batch behavior of recent fine-grained MoEs reflects stable target-side execution properties rather than a dataset-specific acceptance effect. Although the absolute speedups vary by dataset, the qualitative curve shape remains the same, so the main conclusion is about regime structure rather than any single benchmark’s exact TPOT values.
Three-Step Correction: From Weight Model to Measured Target Efficiency
For Qwen3-30B at batch size 1 on H100, the activated-expert-bytes model predicts , whereas the measured value is 0.87–0.90. The gap decomposes into three independently measured corrections using the identity
where denotes the fraction of vanilla forward GPU time and denotes the SD scaling factor for each component.
Step 0: Activated-expert-bytes baseline (). This baseline uses weight bytes as a cost proxy: , (linear in the unique expert count), and .
Step 1: Correcting the MoE fraction (: 0.51 0.68, +0.168, accounting for 44% of the gap). Kernel-level profiling measures the MoE pipeline at 1629/4957 = 32.9% of vanilla target forward GPU time, rather than the 66.7% predicted by weight bytes [7].
Step 2: Correcting the MoE scaling (: 0.68 0.84, +0.164, accounting for 43% of the gap). Kernel-level profiling measures the MoE pipeline scaling at rather than the weight-linear value of , due to L2 cache reuse (5% → 31% hit rate) and dilution of pipeline overhead (routing , activation ).
Step 3: Correcting non-MoE scaling (: 0.84 → 0.87, +0.029, accounting for 8% of the gap). Non-MoE target components are slightly cheaper under SD: attention GEMMs have , FlashAttention [10] has , but framework overhead is amortized (), yielding a collective .
Table 11 summarizes the correction chain. Each step uses independently measured data from different profiling methods. The remaining gap from 0.872 (kernel-level sum) to 0.896 (sweep wall-clock) arises from inter-kernel scheduling gaps within the CUDA graph.
| Step | Correction | Target efficiency | % gap closed | |
| 0 | Weight model baseline | 0.511 | n/a | n/a |
| 1 | MoE fraction: 67% → 33% (GPU time weight) | 0.679 | +0.168 | 44% |
| 2 | MoE scaling: (L2 cache + dilution) | 0.843 | +0.164 | 43% |
| 3 | Non-MoE: (amortization) | 0.872 | +0.029 | 8% |
| Measured (kernel-level sum) | 0.872 | |||
| Measured (sweep wall-clock) | 0.896 |
Table 11. Three-step correction for Qwen3-30B at batch size 1 on H100.
Additional GPU Profiling: Qwen3-235B and B200
Table 12 extends the H100 profiling of Table 5 to a second fine-grained model on H100 (Qwen3-235B, FP8, TP4) and to Qwen3-30B and Mixtral on B200 (FP8, TP1), using the same Nsight Systems and Nsight Compute methodology at batch size 1. Each configuration uses the MoE kernel that vLLM selects by default. The main result is that the activated-expert-bytes model fails as a cost model for sparse MoE verification in every configuration we profile: it underpredicts target efficiency by 0.18–0.38 across all three configurations in Table 12, in addition to the H100 gaps of Table 5, and the error is largest for fine-grained models with small experts. The mechanisms behind the error differ across hardware and kernels, but the error itself persists. Qwen3-235B reproduces both H100 mechanisms: MoE kernels achieve higher bandwidth than attention kernels, and the L2 hit rate rises under SD.
| Metric | Qwen3-235B (H100, TP4) | Qwen3-30B (B200, TP1) | Mixtral-8x7B (B200, TP1) |
| MoE weight reads (SD) | per (token, expert) pair | once per expert | once per expert |
| Expert size (per GPU) | 4.5 MB | 4.5 MB | 168 MB |
| MoE share of weight | 67.9% | 66.7% | 89.4% |
| MoE share of GPU time | 34% | 35% | 69% |
| Attn CTAs / SMs | 18 (14%) | 16 (11%) | 48 (32%) |
| Attn HBM BW (% peak) | 19–26% | 12–13% | 27–37% |
| MoE CTAs / SMs | 48–256 | 96–128 | 148 |
| MoE HBM BW (% peak) | 34–46% | ∼15% | ∼23% |
| Attn SD scaling | 1.00 | 0.97 | 0.98 |
| MoE SD scaling | 1.51 | 1.90 | 1.91 |
| L2 hit rate (AR→SD) | 5.70%→28.66% | 2.49%→2.21% | 14.78%→14.75% |
| TE pred. | 0.51 | 0.51 | 0.43 |
| TE observed | 0.89 | 0.76 | 0.61 |
| Weight model gap | +0.38 | +0.25 | +0.18 |
Table 12. GPU profiling at batch size 1 for Qwen3-235B on H100 and for Qwen3-30B and Mixtral on B200 (all FP8). H100 has 132 SMs and B200 has 148 SMs.
B200. On B200, the phenomenon reproduces: MoE accounts for 35% and 69% of GPU time for Qwen3-30B and Mixtral (vs. 33% and 75% on H100), and the activated-expert-bytes model still underpredicts target efficiency (0.76 and 0.61 observed vs. 0.51 and 0.43 predicted), with the larger gap again for the smaller experts. The H100 mechanisms, however, do not carry over. The default MoE kernel for Qwen3-30B on B200 groups co-routed tokens by expert and reads each expert’s weights once, so there are no repeated reads for L2 to absorb and the L2 hit rate does not rise under SD (2.49%→2.21%); attention and MoE kernels both run at only 12–15% of peak bandwidth, leaving no attention-vs.-MoE disparity. The L2 hit rate thus reflects kernel design rather than hardware: running the default H100 MoE kernel on B200 with the same IPF routing gives an SD L2 hit rate of 27.19% for Qwen3-30B FP8, close to the 27.79% measured on H100. This suggests that at batch size 1 on B200, step time is dominated by costs that do not scale with expert bytes, likely fixed per-step overheads. The common principle is that runtime depends on the bandwidth each kernel achieves and on fixed per-step costs, whereas the activated-expert-bytes model assumes weight bytes at a single constant bandwidth: on H100 achieved bandwidth differs sharply between kernels, and on B200 fixed costs dominate. Verification cost should therefore be measured on the target hardware rather than derived from weight bytes, which is what the synthetic-routing method in Section 8 does.
Three-Level Overlap Decomposition Formulas
We decompose expert overlap into three levels with increasing realism. Each token selects exactly experts from total experts per layer; denotes the empirical selection probability of expert ().
Level 1: Uniform baseline. Assumes for all experts and that tokens are routed independently.
Level 2: Independence baseline. Uses the empirical per-layer expert probabilities while assuming that tokens are routed independently across timesteps. This captures overlap from non-uniform expert popularity.
Level 3: Empirical. Measured directly from consecutive tokens while preserving temporal overlap:
Decomposition. The empirical overlap can be decomposed as:
The temporal-overlap term is the largest source of overlap in 6 of the 8 studied models, accounting for 69–92% of total overlap, as shown in Table 2 of the main text.
Routing Inertia in Rejected Draft Tokens
Section 6.2 showed that, in the aggregate MT-Bench analysis, rejected draft tokens in fine-grained models exhibit equal or higher expert overlap than accepted tokens, supporting the sub-linear verification-cost argument for SD. Here we present the full statistical analysis underlying that claim and summarize the held-out Spec-Bench and Aya results.
Methodology
We record per-token expert assignments for every verification step during SD inference with an EAGLE draft model and . Because vLLM’s enable_return_routed_experts API returns routed experts only for accepted tokens by default, we extended the instrumentation to record routed experts for rejected tokens as well. Each step is labeled as accepted or rejected by the target model. For every pair of steps , we compute the consecutive overlap ratio: the fraction of top- experts shared between them.
A naive accepted-versus-rejected comparison conflates two distinct populations: when step is accepted, step is always accepted, whereas when step is rejected, step may be either accepted or rejected. We resolve this confound with a three-way partition:
Accepted: step accepted, step accepted (normal pair).
First-rejection: step rejected, step accepted (clean comparison).
Cascade: step rejected, step rejected (successive rejection).
The accepted vs. first-rejection comparison is the cleanest test: both populations have an accepted predecessor, isolating the effect of token correctness on expert routing.
Deconfounded Results
Table 14 reports deconfounded consecutive overlap for the reported models at steps 1–3. In the aggregate MT-Bench analysis, across all models with fine-grained routing (), first-rejection tokens exhibit equal or higher overlap than accepted tokens (). The sole exception is Mixtral (, ), whose coarse expert granularity (25% selection ratio) yields the opposite pattern.
Effect Sizes (Cohen’s )
Table 15 reports Cohen’s for the deconfounded comparison. In the aggregate MT-Bench analysis, the strongest effects (–) occur for Qwen3-235B, yet the accepted and rejected overlap distributions still overlap by approximately 85%. The mean difference (–) is 3–7× smaller than the per-sample standard deviation (), yielding a signal-to-noise ratio too low for reliable per-token classification despite .
| Model | Step | Cohen’s | Size | ||
| Qwen3-30B | 1 | −0.008 | −0.06 | 2.3e-3 | negligible |
| 2 | −0.039 | −0.26 | 5.9e-22 | small | |
| 3 | −0.028 | −0.19 | 5.4e-7 | negligible | |
| Qwen3-235B | 1 | −0.040 | −0.29 | 6.2e-111 | small |
| 2 | −0.053 | −0.38 | 8.2e-129 | small | |
| 3 | −0.051 | −0.37 | 5.3e-79 | small | |
| Qwen3-Next-80B | 1 | −0.025 | −0.20 | 7.3e-39 | small |
| 2 | −0.031 | −0.24 | 1.2e-56 | small | |
| 3 | −0.026 | −0.20 | 4.2e-30 | negligible | |
| Mixtral-8x7B | 1 | +0.007 | +0.05 | 9.5e-4 | negligible |
| 2 | +0.020 | +0.15 | 1.6e-12 | negligible | |
| 3 | +0.031 | +0.24 | 1.9e-16 | small |
Table 15. Cohen’s for consecutive overlap (accepted vs. first-rejection, deconfounded). Negative = rejected tokens have higher overlap.
Implication: expert routing is unlikely to predict draft rejection reliably. In the aggregate MT-Bench comparison, the 85% distribution overlap (at ) suggests that any classifier based on expert overlap would operate well below useful accuracy. The held-out Spec-Bench and
Aya slices summarized below are mostly negligible or small in effect size, with the largest outliers concentrated in low-acceptance multilingual settings, so they do not change this practical conclusion.
Cascade Amplification
When step is also rejected (cascade), consecutive overlap increases by an additional 1–5 percentage points over the first-rejection case. As a sequence drifts further from the correct continuation, routing becomes increasingly locally repetitive. At step 3 for Qwen3-30B, cascade overlap is 0.453 vs. first-rejection 0.420; for Qwen3-235B, 0.482 vs. 0.458.
Depth-Dependent Effect
The routing-inertia effect intensifies in deeper MoE layers. For Qwen3-235B at step 3, 71 out of 94 layers (76%) show , all in the negative direction; individual layers reach . Figure 8 visualizes the per-layer gradient across models.

Figure 8. Per-layer (rejected − accepted) consecutive overlap at step 1. Red indicates rejected tokens have higher overlap (routing inertia); blue indicates the opposite.
Coarse Routing Exception
Mixtral-8x7B (, , selection ratio 25%) is the sole model showing the opposite pattern: rejected tokens route to fewer shared experts (, to ). Its coarse expert granularity leaves insufficient room for routing inertia to manifest. This pattern suggests that the phenomenon is strongest when .
Statistical Summary Plots
Here anchor overlap denotes the fraction of top- experts shared between step 0 and step , where step 0 is the first token in the -token verification window. Figure 9 shows standard-error-of-the-mean (SEM) bar charts with Cohen’s annotations for anchor overlap across models and steps.

Figure 9. Anchor overlap (step 0 step ) with SEM error bars and Cohen’s annotations, by acceptance status and step. Whereas consecutive overlap is consistently higher for rejected tokens, anchor overlap tends to be similar or slightly lower, indicating that the inertia is local rather than global.
Held-Out Task and Language Slices
We repeated the deconfounded accepted-vs. first-rejection comparison on held-out Spec-Bench categories and Aya multilingual splits for Qwen3-30B and Qwen3-235B. Table 16 reports medians across slices. For both models, the median consecutive delta is near zero at step 1 and negative at steps 2–3, matching the local-inertia pattern in the aggregate MT-Bench analysis. The main exceptions are slice-level sign reversals in a handful of held-out settings, especially low-acceptance Aya splits.
| Model | Data | Anchor drop | |||
| Qwen3-30B | Spec-Bench | +0.000 | −0.017 | −0.017 | +0.085 |
| Qwen3-30B | Aya | +0.002 | −0.018 | −0.015 | +0.095 |
| Qwen3-235B | Spec-Bench | −0.018 | −0.021 | −0.028 | +0.070 |
| Qwen3-235B | Aya | −0.002 | −0.020 | −0.015 | +0.106 |
Table 16. Held-out deconfounded routing summary. Entries report median across held-out slices, where negative values mean rejected tokens have higher consecutive overlap, together with the median anchor-drop mean.
Chinese is the clearest case: for Qwen3-30B, the Aya zho split has acceptance length AL , leaving only 28 accepted positions at step 3; for Qwen3-235B, the largest multilingual outliers occur in zho and tel, where AL and 1.12. This behavior may also reflect an issue specific to model performance on the zho slice rather than a general language-level trend, and we do not investigate that failure mode further here. These low-acceptance slices can produce medium-sized effects, but they are too unstable to support a useful routing-based rejection predictor. By contrast, the Spec-Bench medians remain tightly aligned with the MT-Bench aggregate and show that dataset shift does not change the practical conclusion.
Overlap Decay
Table 17 shows the overlap-decay profile for the measured models. The main text focuses on to match our speculative-decoding setting, but we additionally report here to show that the decay trend persists one step beyond the verification horizon emphasized in the paper.
At step 4, Qwen3-235B retains overlap 3.4× above baseline (0.337 vs. 0.098), while Qwen3-Next-80B reaches 7.8× (0.281 vs. 0.036), indicating the strongest sustained temporal overlap among the measured models. The decay is monotonic though is well above the independence baseline even at Step 4.

Figure 10. Per-layer step-1 overlap ratio for Qwen3-30B (). The empirical overlap (blue) far exceeds the independence baseline (orange) at every layer except the first and last, with mid-network layers peaking at above baseline.
Held-Out Task and Language Slices
Table 18 compares Qwen3-30B’s MT-Bench overlap-decay profile with the mean over 13 held-out Spec-Bench categories and seven Aya multilingual splits. The Spec-Bench mean is nearly unchanged, while Aya shifts the full curve upward by roughly 0.03–0.05 at each step and raises the empirical verification ratio from 0.415 to 0.439. The new held-out data therefore strengthens, the sub-linear verification-cost claim: multilingual prompts slightly increase expert reuse, but the shape of the decay curve is unchanged.
| Data slice | Step 1 | Step 2 | Step 3 | Step 4 | Indep | Verify ratio |
| MT-Bench | 0.431 | 0.356 | 0.328 | 0.312 | 0.119 | 0.415 |
| Spec-Bench mean | 0.438 (0.407–0.468) | 0.358 (0.324–0.391) | 0.330 (0.289–0.366) | 0.313 (0.274–0.348) | 0.172 (0.122–0.207) | 0.418 (0.400–0.434) |
| Aya mean | 0.477 (0.447–0.515) | 0.396 (0.364–0.458) | 0.364 (0.333–0.424) | 0.345 (0.310–0.404) | 0.176 (0.148–0.228) | 0.439 (0.423–0.462) |
Table 18. Held-out overlap decay for Qwen3-30B. Parentheses report the range across held-out slices.
Per-Layer Overlap Analysis
Overlap varies systematically with depth: mid-network layers reach the independence baseline, whereas early and final layers show weaker correlation.
Table 19 shows step-1 overlap ratios for representative early, middle, and late layers in Qwen3-30B and Qwen3-235B. The depth-dependent pattern is consistent between the two models: early layers have overlap near the independence baseline, mid-network layers reach – the baseline, and final layers decline.
| Qwen3-30B | Qwen3-235B | ||
| Layer | Overlap (indep) | Layer | Overlap (indep) |
| 0 | 0.094 (0.089) | 0 | 0.110 (0.078) |
| 1 | 0.265 (0.099) | 1 | 0.168 (0.082) |
| 2 | 0.306 (0.118) | 2 | 0.247 (0.085) |
| 3 | 0.389 (0.121) | 3 | 0.328 (0.088) |
| 22 | 0.433 (0.111) | 45 | 0.399 (0.090) |
| 23 | 0.405 (0.103) | 46 | 0.365 (0.092) |
| 24 | 0.434 (0.116) | 47 | 0.505 (0.101) |
| 25 | 0.366 (0.114) | 48 | 0.474 (0.090) |
| 44 | 0.423 (0.134) | 90 | 0.392 (0.103) |
| 45 | 0.445 (0.135) | 91 | 0.381 (0.104) |
| 46 | 0.368 (0.133) | 92 | 0.345 (0.105) |
| 47 | 0.231 (0.137) | 93 | 0.219 (0.103) |
| avg | 0.431 (0.118) | avg | 0.445 (0.097) |
Table 19. Per-layer step-1 overlap ratio for representative layers in Qwen3-30B (48 layers) and Qwen3-235B (94 layers).
The independence baseline varies within a narrow band (0.078–0.137) across all layers, suggesting that the depth variation in overlap is driven primarily by differences in temporal-overlap strength rather than by changes in expert popularity. Layer 0 in both models has overlap only slightly above the independence baseline, suggesting that temporal overlap is still weak in the earliest layer. The last layer drops below the model average (0.231 and 0.219), suggesting weaker temporal persistence in expert routing near the output layer.
Unique Expert Counts and IPF Prediction
Table 20 extends Table 3 to the measured models. Both IPF variants (uniform and skewed) are reported; the skewed variant is consistently more accurate.
Both variants slightly overestimate the empirical unique count. The residual arises because the maximum-entropy model captures only pairwise correlations; higher-order correlations, in which three or more tokens simultaneously share experts, provide additional overlap not captured by the pairwise constraints. For coarse-routing models (Mixtral and Qwen2-57B), both variants are nearly exact (less than 0.5% error) because the small expert pool leaves little room for higher-order effects.
Synthetic Routing Generation via IPF
The IPF framework described in § 6.3 models each expert’s activation across consecutive tokens as a set of correlated binary variables . The constraints are:
Marginals: (per-expert probability, or in the uniform variant).
Pairwise joints: for steps .
IPF iteratively rescales the distribution over the states to satisfy all four marginal and six pairwise constraints. The predicted unique expert count is , where denotes the fitted joint distribution for expert .
Uniform vs. skewed variants. The uniform variant uses a single marginal for all experts and a single IPF solve. The skewed variant bins experts by their empirical probability (20 bins per layer), solves a separate IPF problem for each bin, and sums the per-expert contributions. The skewed variant captures both temporal overlap and non-uniform expert popularity, reducing the maximum prediction error from +4.7% (uniform IPF, under 5%) to +3.7% (skewed IPF, under 4%).
Synthetic routing generation. Beyond prediction, the same framework can generate synthetic token-to-expert assignments: (1) sample independent 4-bit patterns from the IPF distribution and (2) repair cardinality to enforce exactly active experts per token via single-expert swaps that minimize overlap error. In the skewed mode, the resulting per-layer expert probabilities are matched, achieving correlation with the real routing distributions. These assignments are used in the dummy-weight experiment (§6.3, Figure 2).
MoE Draft Layer Analysis
§ 9.3 raised the possibility that MoE draft layers may become attractive in sufficiently sparse regimes. This appendix addresses a narrower question: under the routing and hardware conditions of the measured models, when would an MoE drafter become cheaper than the dense drafters used in current SD systems?
Analytical Cost Model
Current SD drafters such as EAGLE-3 and MTP use a dense transformer layer. For Qwen3-30B, a one-layer MoE drafter has the same effective width as the dense drafter (top- expert intermediate = dense intermediate = 6144), and therefore approximately the same per-step cost at batch size 1, while exposing all expert weight matrices. If training could exploit this additional capacity, it might improve acceptance length without increasing draft latency; however, we do not train such drafters here and treat this only as a synthetic design study.
For an -layer MoE target with dense intermediate , a single MoE draft layer has the same effective width as a dense layer at batch size 1. We predict the MoE draft cost at batch size by anchoring at batch size 1 and adding the measured incremental per-layer cost:
where is the measured AR forward-pass time at batch size . captures any fixed framework overhead. captures the incremental per-layer MoE cost. Naively using as the drafter cost would underestimate the true cost, because some layerwise latency is hidden when multiple layers execute in a pipelined stack but becomes exposed when only a single layer is used. This anchoring avoids double-counting attention costs while capturing both the transition from memory-bandwidth-bound to compute-bound regimes and any fixed framework overhead.
Target Efficiency and Draft Cost for Qwen3-30B
Figure 11 shows the measured target efficiency of a dense draft layer and a synthetic MoE draft layer, together with the prediction from Equation (11). Figure 11b shows that the MoE drafter is slower at small batch sizes, but the dense drafter becomes compute-bound earlier and becomes more expensive at larger batch sizes. This behavior is consistent with the lower arithmetic intensity of MoE layers relative to dense layers [23]. Because the target model also contains MoE layers, its scaling remains closer to that of an MoE drafter than to that of a dense drafter. As a result, the dense drafter yields lower target efficiency at larger batch sizes, making the MoE drafter more attractive in that regime. This gap widens further for multi-layer drafters, which are increasingly common in recent SD work [25].

Figure 11. (a) Target efficiency (b) Total draft time (all steps). Dense draft cost for Qwen3-30B (, ). The dense drafter becomes more expensive at large batch sizes.
With one layer, the MoE drafter has only a modest effect on target efficiency, but if trained effectively it could still improve end-to-end speedup by increasing acceptance length through its additional parameters. The gap magnifies for multi-layer drafts (e.g., five layers): the total cost of five dense layers grows superlinearly at high batch sizes, whereas five MoE layers would track the target’s scaling more closely.
Synthetic Experiment with
For Qwen3-30B with experts, the MoE drafter’s advantage over the dense drafter appears only at very large batch sizes, beyond the regime where SD is most effective. To test whether this advantage broadens in sparser architectures, we extrapolate to experts (keeping , ) using dummy weights and uniform IPF routing with the overlap statistics from Qwen3-30B () in every layer.
Figure 12 shows the results. With , the MoE draft layer remains bandwidth-bound to much larger batch sizes, consistent with prior analysis [23]. Even at moderate to large batch sizes, the MoE drafter has a similar effect on the speedup curve as the dense drafter, unlike the case where the MoE advantage, assuming the same acceptance length, emerges only at very large batch sizes. The target-efficiency benefit is correspondingly more pronounced for larger number of layers, consistent with the trend in the setting. These synthetic results suggest that MoE draft layers may become attractive over a broader batch-size range in sufficiently sparse architectures.

Figure 12. (a) Draft / forward ratio (). (b) Total draft time (). Synthetic MoE draft analysis for (extrapolated from Qwen3-30B with dummy weights). The MoE layer remains bandwidth-bound to larger batch sizes, keeping the draft ratio flat where the dense drafter’s ratio rises.
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [N/A].
[N/A] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide a short (1–2 sentence) justification right after your answer (even for [N/A]).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will also be asked to include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While [Yes] is generally preferable to [No], it is perfectly acceptable to answer [No] provided a proper justification is given (e.g., error bars are not reported because it would be too computationally expensive” or “we were unable to find the license for the dataset we used”). In general, answering [No] or [N/A] is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading “NeurIPS Paper Checklist”,
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: The claims made in abstract and introduction are theoretically motivated and experimentally validated in Sections 4, 6, 7, 8, and M.3.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes] Justification: The paper discusses the limitations of the work in Section A.
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Please refer to § 6.3 and Appendix § 19 and Appendix § 24 for the full set of assumptions and proofs.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose our experimental setup needed to reproduce results in Appendix § 14.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will provide open access to the data and code during final submission.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (
https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (
https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: We disclose our experimental setup in Appendix § 14.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Every end-to-end speedup measurement was repeated over 3 runs in independent server sessions, and the coefficient of variation (std/mean) of the speedup is below 5% at every batch size for every configuration (Appendix § 14). Within each run, output token latency is measured as the median over 20k–655k output tokens during the benchmark client run. We also repeat the measurement for speedup curve on different datasets (MT-Bench, ShareGPT, InstructCoder, SpecBench (13 categories)) to ensure robustness in Figure 7. We also repeat the measurement of difference in temporal overlap between accepted and rejected hypotheses in Appendix § 20.8 on SpecBench (13 categories) and Aya (7 languages) to ensure robustness beyond MT-Bench. We report standard error and Cohen’s d for the difference in temporal overlap between accepted and rejected hypotheses in Appendix § 20 to quantify the statistical significance of the difference.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.).
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes]
Justification: We mention the required compute resources in our experimental setup in Appendix B.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: We have read and understood the NeurIPS Code of Ethics and made every effort to comply with it in our research.
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [N/A]
Justification: Our work has no societal impact.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification: Our paper poses no such risks.
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use vLLM (an open source LLM inference framework) in our work and cite it in the paper.
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets,
paperswithcode.com/datasetshas curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [N/A]
Justification: Our work does not introduce new assets.
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: Our work does not involve research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: Our work does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.
References
- [1]Talor Abramovich, Maor Ashkenazi, Izzy Putterman, Benjamin Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein, and Yonatan Geifman. SPEED-Bench: A unified and diverse benchmark for speculative decoding. In International Conference on Machine Learning, 2026. arXiv:2604.09557.arxiv.org/abs/2604.09557
- [2]Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. Hydra: Sequentially-dependent draft heads for medusa decoding. Conference on Language Modeling, 2024.arxiv.org/abs/2402.05109
- [3]Daniel Bershatsky and Ivan Oseledets. On the spatial structure of mixture-of-experts in transformers. ICLR 2025 Workshop on Sparsity in LLMs, 2025.arxiv.org/abs/2504.04444
- [4]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. International Conference on Machine Learning, 2024.arxiv.org/abs/2401.10774
- [5]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.arxiv.org/abs/2302.01318
- [6]Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhangzhao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
- [7]I. Csiszar. i-divergence geometry of probability distributions and minimization problems. The Annals of Probability, 1975.DOI
- [8]Damai Dai, Chengqi Deng, Chenggang Zhao, R.X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.70. URL https://aclanthology.org/2024.acl-long.70/.DOI
- [9]Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. International Conference on Learning Representations, 2024.arxiv.org/abs/2307.08691
- [10]Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. Neural Information Processing Systems, 2022.arxiv.org/abs/2205.14135
- [11]DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024.arxiv.org/abs/2412.19437
- [12]DeepSeek-AI. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 2025.DOI
- [13]W. Edwards Deming and Frederick F. Stephan. On a least squares adjustment of a sampled frequency table when the expected marginal totals are known. The Annals of Mathematical Statistics, 1940.DOI
- [14]Dong Du, Shulin Liu, Tao Yang, Shaohua Chen, and Yang Li. UloRL: An ultra-long output reinforcement learning approach for advancing large language models’ reasoning abilities. arXiv preprint arXiv:2507.19766, 2025.arxiv.org/abs/2507.19766
- [15]Nan Du et al. GLaM: Efficient scaling of language models with mixture-of-experts. International Conference on Machine Learning, 2022.arxiv.org/abs/2112.06905
- [16]Yulun Du et al. Kimi K2.5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026.arxiv.org/abs/2602.02276
- [17]William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 2022.arxiv.org/abs/2101.03961
- [18]Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of LLM inference using lookahead decoding. International Conference on Machine Learning, 2024.arxiv.org/abs/2402.02057
- [19]Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. International Conference on Machine Learning, 2024.arxiv.org/abs/2404.19737
- [20]Xin He, Shunkan Zhang, Kaijie Tang, Shaohui Shi, Yuxin Wang, Zihao Zeng, Zhenheng Tang, Xiaowen Chu, Haiyan Yin, Ivor W. Tsang, and Yew Soon Ong. ExpertFlow: Efficient mixture-of-experts inference via predictive expert caching and token scheduling. ACM/IEEE Design Automation Conference, 2026.
- [21]Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. REST: Retrieval-based speculative decoding. North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024.
- [22]Xin Huang et al. P-EAGLE: Parallel-drafting EAGLE with scalable training. arXiv preprint arXiv:2602.01469, 2026.arxiv.org/abs/2602.01469
- [23]Zongle Huang, Lei Zhu, Zong Yuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, and Tianyu Zhang. MoESD: Unveil speculative decoding’s potential for accelerating sparse MoE. Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.19645
- [24]Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.arxiv.org/abs/2401.04088
- [25]Guoqing Jiang et al. Scaling laws for speculative decoding. arXiv preprint arXiv:2505.07858, 2025.arxiv.org/abs/2505.07858
- [26]Sanjay Kariyappa and G. Edward Suh. SideQuest: Model-driven KV cache management for long-horizon agentic reasoning. arXiv preprint arXiv:2602.22603, 2026.arxiv.org/abs/2602.22603
- [27]Kimi Team. Kimi k1.5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501.12599, 2025.arxiv.org/abs/2501.12599
- [28]Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. International Conference on Learning Representations, 2023.arxiv.org/abs/2212.05055
- [29]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. ACM SIGOPS Symposium on Operating Systems Principles, 2023.arxiv.org/abs/2309.06180
- [30]Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. International Conference on Learning Representations, 2021.
- [31]Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. International Conference on Machine Learning, 2023.arxiv.org/abs/2211.17192
- [32]Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE layers: Simplifying training of large, sparse models. International Conference on Machine Learning, 2021.arxiv.org/abs/2103.16716
- [33]Kaixin Li, Qisheng Hu, James Xu Zhao, Hui Chen, Yuxi Xie, Tiedong Liu, Michael Shieh, and Junxian He. Instructcoder: Instruction tuning large language models for code editing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 473–493, Bangkok, Thailand, 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.acl-srw.52.arxiv.org/abs/2310.20329
- [34]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. International Conference on Machine Learning, 2024.arxiv.org/abs/2401.15077
- [35]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.arxiv.org/abs/2406.16858
- [36]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2503.01840
- [37]Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, and Sebastian Jaszczur. Scaling laws for fine-grained mixture of experts. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 33270–33288. PMLR, 2024. URL https://proceedings.mlr.press/v235/ludziejewski24a.html.arxiv.org/abs/2402.07871
- [38]Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhijao Jia. SpecInfer: Accelerating large language model serving with tree-based speculative inference and verification. ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2024.
- [39]NVIDIA. nvidia/gpt-oss-120b-eagle3-long-context, 2025. URL https://huggingface.co/nvidia/gpt-oss-120b-Eagle3-long-context. Hugging Face model card.
- [40]Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck, Man Luo, Sungduk Yu, Chendi Xue, and Vasudev Lal. Probing semantic routing in large mixture-of-expert models. Findings of the Association for Computational Linguistics: EMNLP 2025, pages 18263–18278, 2025.arxiv.org/abs/2502.10928
- [41]Red Hat Developer. Performance improvements with speculative decoding in vllm for gpt-oss, 2026. URL https://developers.redhat.com/articles/2026/04/16/performance-improvements-speculative-decoding-vllm-gpt-oss#get_started. Technical article.
- [42]Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. MagicDec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. International Conference on Learning Representations, 2025.arxiv.org/abs/2408.11049
- [43]Anish Saxena, Po-An Tsai, Hritvik Taneja, Aamer Jaleel, and Moinuddin Qureshi. Utility-driven speculative decoding for mixture-of-experts. IEEE/ACM International Symposium on Microarchitecture, 2025.arxiv.org/abs/2506.20675
- [44]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. International Conference on Learning Representations, 2017.arxiv.org/abs/1701.06538
- [45]Shivalika Singh, Freddie Vargus, Daniel D’Souza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11521–11567, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.620. URL https://aclanthology.org/2024.acl-long.620/.DOI
- [46]Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In International Conference on Learning Representations, 2025.
- [47]Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. Neural Information Processing Systems, 2018.
- [48]Weiwei Sun, Miao Lu, Zhan Ling, Kang Liu, Xuesong Yao, Yiming Yang, and Jiecao Chen. Scaling long-horizon LLM agent via context-folding. arXiv preprint arXiv:2510.11967, 2025.
- [49]Zितeng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. SpecTr: Fast speculative decoding via optimal transport. Neural Information Processing Systems, 2023.
- [50]Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zihao Jia, and Max Ryabinin. SpecExec: Massively parallel speculative decoding for interactive LLM inference on consumer devices. Neural Information Processing Systems, 2024.
- [51]Bangsheng Tang, Carl Chengyan Fu, Fei Kou, Grigory Sizov, Haoci Zhang, Jason Park, Jiawen Liu, Jie You, Qirui Yang, Sachin Mehta, et al. Efficient speculative decoding for Llama at scale: Challenges and solutions. arXiv preprint arXiv:2508.08192, 2025.
- [52]Samuel Williams, Andrew Waterman, and David Patterson. Roofline: An insightful visual performance model for multicore architectures. Communications of the ACM, 2009.
- [53]Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. Findings of the Association for Computational Linguistics: EMNLP 2023, 2023.
- [54]Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. Findings of the Association for Computational Linguistics: ACL 2024, 2024. URL https://arxiv.org/abs/2401.07851.
- [55]An Yang, Baosong Yang, Beichen Zhang, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
- [56]Lianmin Zheng et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Neural Information Processing Systems, 2023.
- [57]Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Zhifeng Chen, Quoc V Le, and James Laudon. Mixture-of-experts with expert choice routing. Neural Information Processing Systems, 2022.