betaFeedback

WorksComputer Science

Efficient and Robust Drafting with Multi-Token Prediction

Abstract

Speculative decoding has emerged as a critical technique for accelerating Large Language Model (LLM) inference by decoupling generation into a drafting phase and a parallel verification phase. A crucial challenge is therefore designing a drafter that is both lightweight and accurate enough to approximate the predictions of the target model. We present a highly optimized Multi-Token Prediction (MTP) architecture that extends existing backbone LLMs to efficiently produce high-quality drafts. Built around a single shared drafting head, our approach features three core architectural optimizations: target activation injection to anchor semantic planning, zero-copy KV-cache cross-attention to bypass prefill compute and reduce memory overhead, and an efficient clustered vocabulary projection to reduce the cost of full-vocabulary decoding. Testing these improvements with the Gemma 4 model family, our G-MTP shows out-of-the-box speedups for both large server-side setups (up to 3.1×\times) and smaller models targeting memory-constrained on-device environments (up to 2.3×\times on H100 and up to 2.2×\times on a smartphone GPU). In addition, we present parameter-efficient fine-tuning recipes for G-MTP that keep the drafter aligned with the backbone LLM under domain shift, requiring up to 1.83×\times fewer target forward passes than tuning only the backbone.

Introduction

Deploying Large Language Models (LLMs) at scale — across both server clusters and edge devices — faces a strict bottleneck due to the autoregressive nature of standard transformers. Generating a single token requires a full forward pass, which under-utilizes modern hardware and strains memory bandwidth. Speculative decoding mitigates this using a smaller draft model to predict multiple tokens ahead, which the target model verifies in parallel. However, standalone draft models must independently process the entire context, introducing additional prefill computations and separate memory overheads. This added friction degrades throughput in server clusters and completely prohibits deployment in memory-constrained edge devices like mobile phones.

To address this, we introduce an optimized Multi-Token Prediction head (G-MTP) that can be fully integrated with any modern LLM, such as Gemma 4. Instead of a separate model, we append a lightweight, shared 4-layer Transformer stack directly to the final layers of the main model. This architecture maximizes token acceptance while minimizing latency and memory overhead through several key innovations:

  • Shared Drafting Head: A single, lightweight transformer stack drafts multiple future tokens autoregressively. Although trained for a 3-step lookahead, the head generalizes to longer draft sequences during inference.

  • Target Activation Injection: The drafter takes the final activation of the target backbone as input, anchoring its planning in deep semantic history, as done by Li et al. (2024).

  • Zero-Copy KV-Cache Cross-Attention: The drafter cross-attends directly to the target model’s KV cache using a constant positional encoding, completely eliminating drafter prefill overhead.

  • Clustered Vocabulary Projection: A two-stage, cluster-based vocabulary projection that drastically reduces the computational matrix-multiplication overhead of projecting draft tokens onto the full vocabulary.

  • Robust Domain Adaptation: A computationally efficient distillation regime that prevents the drafter from suffering severe distributional decoupling when the verification backbone is fine-tuned for specialized downstream tasks.

We trained G-MTP drafters for the full Gemma 4 model family (Gemma Team et al., 2026) and release the checkpoints.1 To support reproducibility, we also release the complete domain adaptation training pipelines described in § 5.2

Related Work

Speculative decoding accelerates inference via rapid drafting and parallel verification (Chen et al., 2023; Leviathan et al., 2023). Foundational works like blockwise parallel decoding (Stern et al., 2018), Medusa (Cai et al., 2024), and multi-token prediction (MTP) (Gloeckle et al., 2024) train multiple independent non-autoregressive heads to achieve self-speculative speedups. More recently, DFlash (Chen et al., 2026) further optimizes this parallel drafting paradigm to maximize drafting speed. In contrast to parallel heads, EAGLE (Li et al., 2024), alongside recurrent and sequentially dependent drafters [3] (Cheng et al. 2024), introduced sequential autoregressive drafting conditioned on prior hidden states and sampled tokens. Similarly, several frontier models adopted a sequential MTP approach, including DeepSeek-V3 (DeepSeek-AI, 2024), Gemma 4 (Gemma Team et al., 2026), Qwen3 (Yang et al., 2025), GLM-4.5 (GLM-4.5 Team et al., 2025), and MiMo (Xiaomi LLM-Core Team et al., 2025). While some prior MTP designs assign separate modules for each lookahead step, our architecture, similar to FastMTP (Cai et al., 2025), maximizes memory efficiency by reusing a single shared 4-layer block.

A persistent challenge in developing sequential, feature-based draft models is managing the inherent uncertainty of autoregressive generation. As highlighted by EAGLE (Li et al., 2024), continuous feature sequences exhibit significant ambiguity because multiple valid token branches can naturally emerge from a single hidden state. To explicitly resolve this randomness, EAGLE incorporates the token sequence advanced by one time step alongside the feature sequence, anchoring the next-feature prediction with the actual sampled token. Our architecture integrates this insight by dynamically combining the main model’s current hidden state with the newly sampled next token’s embedding, without requiring complex auxiliary feature-pooling layers.

To bypass the standalone drafter prefill bottleneck, recent architectures loosely couple the drafter to the target model’s Key-Value (KV) cache [2] (Du et al. 2024) or redesign decoders into minimal cross-attention structures (Zhong et al., 2025). However, our integrated G-MTP layers directly share the primary KV cache via zero-copy cross-attention, eliminating standalone prefill overhead entirely. Finally, because fine-tuned backbones can cause drafters to undergo severe distributional decoupling (Zhou et al., 2024), we build on distillation-based MTP training and post-training adaptation methods [5] (Kirchenbauer et al. 2026; Teoh 2026) to demonstrate how G-MTP drafters survive domain shifts via robust, parameter-efficient adaptation strategies.

Methodology and Architecture

At the core of our speculative decoding framework is a shared 4-layer Multi-Token Prediction (G-MTP) Transformer block that outputs candidate tokens autoregressively. To initiate the drafting phase, the target backbone’s final layer hidden activations are combined with the current token’s embedding from the target backbone’s embedding table, and fed into the MTP head. Figure 1 highlights the construction of this input alongside the cross-attention flow. Rather than building an isolated context representation, the drafter actively queries the primary model’s pre-computed Key-Value (KV) cache via Zero-Copy Cross-Attention (Figure 2). This zero-copy memory integration completely bypasses the traditional prefill overhead associated with standalone draft models. Because the same 4-layer stack is recursively applied to predict every subsequent candidate, the architecture naturally accommodates arbitrary lookahead horizons. Following generation, the target backbone evaluates the entire draft trajectory concurrently in a single parallel verification step.

Figure 1. Architecture of the optimized Multi-Token Prediction (G-MTP) module. The input to the drafter is constructed by concatenating the token embedding (e(x)e(x)) with the backbone hidden state (h(L)h^{(L)}), both of dimension dtargetd_{\mathrm{target}}, and reducing to the drafter dimension ddraftd_{\mathrm{draft}}. The draft tokens are then propagated through the MTP layers, utilizing cross-attention against the backbone’s frozen KV Cache to bypass prefill computations before recursively triggering the next forward step. For subsequent drafting steps, the drafter’s output state is up-projected (WupW_{\mathrm{up}}), concatenated with the embedding of the sampled draft token, and down-projected (WdownW_{\mathrm{down}}) to form the next input (Equation (3)).

Figure 2. The autoregressive G-MTP drafter (blue blocks) is initialized with activations from the main model (gray blocks) and cross-attends to its KV cache. Across successive drafting steps, the positional encoding (p1p_1) remains constant, as the drafter repeatedly queries a static, pre-computed KV snapshot.

Target Activation Injection

Standalone draft models frequently suffer from lower acceptance rates because they generate predictions based solely on text history, remaining “blind” to the rich internal semantics computed by the primary target model. To provide better semantic context, our framework integrates directly with the target model. The drafting process is initialized by extracting the high-dimensional activations from the final processing layers of the target model.

Let e(xt+1)∈Rdtargete(x_{t+1}) \in\mathbb{R}^{d_{\mathrm{target}}} denote the (scaled) backbone embedding of the newly sampled token and ht(L)∈Rdtargeth_{t}^{(L)} \in\mathbb{R}^{d_{\mathrm{target}}} denote the final-layer hidden state of the target backbone, taken after its final RMSNorm. The initial input to the drafter, z1z_1, is constructed by concatenating these vectors and projecting them down to the smaller internal dimension of the drafter (ddraftd_{\mathrm{draft}}):

z1=Wdown[e(xt+1)∥ht(L)](1)z_1 = W_{\mathrm{down}}\left[e(x_{t+1}) \mathbin{\Vert} h_{t}^{(L)}\right] \tag*{(1)}

where Wdown∈Rddraft×2dtargetW_{\mathrm{down}} \in\mathbb{R}^{d_{\mathrm{draft}} \times2d_{\mathrm{target}}}. For any lookahead step k≥1k \ge1, given the projected input vector zk∈Rddraftz_k \in\mathbb{R}^{d_{\mathrm{draft}}}, our shared 4-layer drafting block processes zkz_k by cross-attending directly to the target backbone’s prefilled Key-Value caches, producing the next hidden state representation ht+k(draft)∈Rddrafth_{t+k}^{(\mathrm{draft})} \in\mathbb{R}^{d_{\mathrm{draft}}} (the output of the drafter’s final RMSNorm):

ht+k(draft)=Transformer⁡draft(zk,Kbackbone,Vbackbone)(2)h_{t+k}^{(\mathrm{draft})} = \operatorname{Transformer}_{\mathrm{draft}}(z_k, K_{\mathrm{backbone}}, V_{\mathrm{backbone}}) \tag*{(2)}

Because the target model has not yet verified the draft sequence, the MTP head acts recurrently for subsequent steps (k>1k > 1). Instead of maintaining an auxiliary growing KV-cache for its own predictions, the drafter functions similarly to a Recurrent Neural Network (RNN) over the draft trajectory, explicitly propagating state via its previous hidden activation. Formally, the drafter up-projects its previous hidden state ht+k−1(draft)h_{t+k-1}^{(\mathrm{draft})} back to the full target dimension using an up-projection matrix Wup∈Rdtarget×ddraftW_{\mathrm{up}} \in\mathbb{R}^{d_{\mathrm{target}}\times d_{\mathrm{draft}}}, concatenates it with the shared embedding of its newly predicted draft token e(x~t+k)e(\tilde{x}_{t+k}), and projects it down again to construct the subsequent input zkz_k (Figure 1):

zk=Wdown[e(x~t+k)∥Wupht+k−1(draft)](3)z_k = W_{\mathrm{down}}\left[e(\tilde{x}_{t+k}) \mathbin{\Vert} W_{\mathrm{up}}h_{t+k-1}^{(\mathrm{draft})}\right] \tag*{(3)}

This recursive projection natively maintains the autoregressive causal chain across the drafted sequence without requiring further forward passes through the backbone or storing additional keys and values for the drafted tokens.

Zero-Copy KV-Cache Cross-Attention

A primary inefficiency in standard speculative decoding is the computational cost of prefilling the draft model’s KV cache. To completely eliminate this prefill latency, our architecture employs a zero-copy KV cache sharing mechanism, where the G-MTP block cross-attends directly to the already-computed KV cache of the target model.

Mirroring the backbone’s interleaved local and global attention with a Local-Local-Local-Global topology, the drafter cross-attends directly to the target’s last prefilled local and global layers. Specifically, the drafter’s three local layers all cross-attend to the same last prefilled backbone local layer, while its single global layer cross-attends to the last prefilled backbone global layer. When the backbone employs cross-layer KV sharing ((Brandon et al., 2024)), as in E2B and E4B, these “last prefilled” caches do not strictly align with the absolute ultimate layer of the backbone. When the drafter cross-attends to the frozen KV cache, it utilizes a constant base positional encoding (p1p_1) for all queries; the forward temporal progression across drafting steps is instead implicitly encoded within the recurrently injected hidden states (Equation (3)) rather than through explicit absolute position IDs. Although multiple sequential drafting layers cross-attend to identically shared static keys and values, they are not redundant; each successive drafting layer dynamically updates its query vectors to extract progressively refined semantic relationships.

This direct cross-attention mechanism achieves a dual benefit: first, it provides the lightweight drafter with deep, full-context semantic history, drastically improving the acceptance rate of drafted tokens on complex tasks. Second, it prevents data duplication in memory. By querying the backbone’s KV cache directly, the drafter needs no KV cache of its own, which is especially relevant for memory-constrained on-device deployments.

Clustered Vocabulary Projection

In smaller transformers, like drafters, the final decoding step — multiplying the hidden state by a large embedding matrix to compute logits over the full vocabulary — constitutes a significant latency overhead. In the case of the Gemma 4 architecture, projecting a hidden state to a large vocabulary of ∣V∣=262,144|V| = 262{,}144 tokens requires O(∣V∣×ddraft)O(|V| \times d_{\mathrm{draft}}) operations. For a lightweight drafter, this overhead slows every sequential drafting step, eroding wall-clock speedups gained from speculative decoding.

To mitigate this latency overhead, we introduce a clustered vocabulary projection (Zhang et al., 2026). During an offline phase, the drafter’s own un-embedding matrix (∣V∣×ddraft|V| \times d_{\mathrm{draft}}) is clustered into K=2048K = 2048 distinct groups of size 128, using a balanced KK-means clustering, yielding a representative centroid in Rddraft\mathbb{R}^{d_{\mathrm{draft}}} for each cluster.

The drafter performs a two-stage sparse routing: it first produces cluster logits (ddraft×2048d_{\mathrm{draft}} \times2048 operation, with ddraft=256d_{\mathrm{draft}} = 256 for E2B and E4B), then computes exact token logits only for the top-32 most probable clusters (ddraft×4096d_{\mathrm{draft}} \times4096, i.e., 32 clusters of 128 tokens). This aggressive matrix compression reduces the token-logit computation from ddraft×262,144d_{\mathrm{draft}} \times262{,}144 to ddraft×(2,048+4,096)d_{\mathrm{draft}} \times(2{,}048 + 4{,}096) multiply-accumulates per step (≈43×\approx43\times fewer), while maintaining a >97%> 97\% token hit rate. On the Edge TPU of a Google Pixel 11, this reduces the drafting time of E2B and E4B by about 20% (§ 4).

Experimental Results

We trained G-MTP drafters for the full Gemma 4 model family (Gemma Team et al., 2026). All the drafters have the same architecture as described in this paper, differing only in the model dimension of the drafter and in the vocabulary projection: the Transformer block has model dimension 256 for Gemma 4 E2B and E4B, and 1024 for 26B MoE, 12B and 31B. All have three local and one global attention layers and a feed-forward expansion ratio of 8. The E2B and E4B drafters use the efficient clustered vocabulary projection while the larger models’ drafters use a standard full vocabulary projection.

To demonstrate that the theoretical block efficiency gains driven by G-MTP speculative decoding translate to wall-clock acceleration, we evaluate the end-to-end generation throughput of the Gemma 4 model family. Table 1 presents the autoregressive decoding speed (tokens/s) measured on NVIDIA H100 GPUs (a single H100 GPU for E2B, E4B and 12B, 2×H100 for 26B and 31B) utilizing bfloat16 and the vLLM serving engine (v0.26), with batch size 1, measured on the 880 prompts in the NVIDIA SPEED-Bench quality dataset [1]. See Appendix § 8.§ 1 and Table 5 for a full ablation of γ\gamma.

ModelDrafter DimensionDrafter ParamsOptimal γ\gammaAvg BEBaseline ThroughputThroughput with G-MTPSpeedup
Gemma 4 E2B25676M42.71278.8550.11.97×1.97\times
Gemma 4 E4B25677M93.45178.0415.52.33×2.33\times
Gemma 4 12B1024400M63.6993.5249.72.67×2.67\times
Gemma 4 26B MoE1024430M43.27248.9334.01.34×1.34\times
Gemma 4 31B1024500M84.2673.2230.13.14×3.14\times

Table 1. Parameter counts and generation speedup across the Gemma 4 family (throughput measured on H100, in tokens per second).

Modelγ\gammaAvg BEBaseline (tokens/s)G-MTP Speculative (tokens/s)Throughput Speedup
Gemma 4 E2B22.15278.8504.71.81×
Gemma 4 E2B32.47278.8531.51.91×
Gemma 4 E2B42.71278.8550.11.97×
Gemma 4 E2B52.88278.8550.11.97×
Gemma 4 E2B63.02278.8540.81.94×
Gemma 4 E2B73.11278.8533.91.92×
Gemma 4 E2B83.19278.8528.51.90×
Gemma 4 E2B93.24278.8505.81.81×
Gemma 4 E2B103.29278.8494.31.77×
Gemma 4 E2B113.33278.8482.51.73×
Gemma 4 E2B123.35278.8462.21.66×
Gemma 4 E4B22.20178.0347.01.95×
Gemma 4 E4B32.56178.0382.72.15×
Gemma 4 E4B42.82178.0402.02.26×
Gemma 4 E4B53.02178.0409.02.30×
Gemma 4 E4B63.18178.0410.22.30×
Gemma 4 E4B73.30178.0407.92.29×
Gemma 4 E4B83.39178.0401.62.26×
Gemma 4 E4B93.45178.0415.52.33×
Gemma 4 E4B103.49178.0409.02.30×
Gemma 4 E4B113.52178.0401.92.26×
Gemma 4 E4B123.53178.0390.22.19×
Gemma 4 12B22.4093.5200.42.14×
Gemma 4 12B32.8693.5225.82.41×
Gemma 4 12B43.2193.5240.72.57×
Gemma 4 12B53.4893.5246.72.64×
Gemma 4 12B63.6993.5249.72.67×
Gemma 4 12B73.8593.5248.22.65×
Gemma 4 12B83.9793.5248.22.65×
Gemma 4 12B94.0493.5249.02.66×
Gemma 4 12B104.1293.5245.52.63×
Gemma 4 12B114.1893.5238.42.55×
Gemma 4 12B124.2193.5233.42.50×
Gemma 4 26B MoE22.42248.9324.01.30×
Gemma 4 26B MoE32.90248.9332.11.33×
Gemma 4 26B MoE43.27248.9334.01.34×
Gemma 4 26B MoE53.56248.9318.91.28×
Gemma 4 26B MoE63.79248.9301.81.21×
Gemma 4 26B MoE73.97248.9282.31.13×
Gemma 4 26B MoE84.10248.9266.21.07×
Gemma 4 26B MoE94.20248.9248.51.00×
Gemma 4 26B MoE104.27248.9235.90.95×
Gemma 4 26B MoE114.36248.9222.50.89×
Gemma 4 26B MoE124.39248.9209.50.84×
Gemma 4 31B22.4573.2166.52.27×
Gemma 4 31B32.9673.2192.82.63×
Gemma 4 31B43.3573.2208.82.85×
Gemma 4 31B53.6773.2219.93.00×
Gemma 4 31B63.9273.2225.03.07×
Gemma 4 31B74.1373.2226.03.09×
Gemma 4 31B84.2673.2230.13.14×
Gemma 4 31B94.4173.2229.43.13×
Gemma 4 31B104.5073.2225.83.08×
Gemma 4 31B114.5673.2221.63.03×
Gemma 4 31B124.6273.2221.33.02×

Table 5. Throughput across lookahead horizons γ\gamma (Figure Figure 3 extends to γ=15\gamma= 15).

While smaller models like the E4B naturally possess high baseline generation speeds (178 tokens/s), deploying the ultra-lightweight G-MTP drafter accelerates it further to 415 tokens/s. This is particularly exciting because the E2B and E4B parameter bounds fall directly within the capacities of modern smartphones, making them strong candidates for on-device speedups. On-device measurements support this: with γ=3\gamma= 3, G-MTP accelerates decoding by 1.69×1.69\times for E2B and 2.18×2.18\times for E4B on the GPU of a Samsung Galaxy S26 Ultra (Appendix § 8.§ 2), and by 1.92×1.92\times and 1.98×1.98\times on the Edge TPU of a Google Pixel 11 (Table 3). More notably, as the verification backbone scales in parameter count (e.g., 12B and 31B models), the relative wall-clock cost of the verification pass amortizes efficiently against the drafter, yielding profound ∼2.7×\sim2.7\times to 3.1×3.1\times accelerations in end-to-end generation throughput, mitigating the memory-bandwidth bottlenecks traditionally associated with autoregressive sampling on large models.

Table 3 | Full (no-CE) vs. clustered (CE) vocabulary projection on the Edge TPU of a Google Pixel 11 (SPEED-Bench qualitative, 456 prompts, γ=3\gamma= 3, greedy decoding). Values are medians over prompts; speedups are ratios of median decoding throughput. CE vs. no-CE columns are ratios of decoding speedup and of total drafting time.

Modelno-CE Speedup vs. BaselineCE Speedup vs. BaselineCE vs. no-CE SpeedupCE vs. no-CE Drafter Time
Gemma 4 E2B1.89×1.92×1.01×0.81×
Gemma 4 E4B1.96×1.98×1.01×0.79×

Table 3.

For the 26B Mixture-of-Experts (MoE) architecture, verification carries a higher overhead per token at small batch sizes due to the potential activation of distinct experts across draft tokens, as evidenced by the high BE translating into a lower speedup. However, when operating at larger batch sizes, these expert activation costs are efficiently amortized across concurrent requests, yielding higher relative throughput speedups that plateau between 1.55×1.55\times and 1.71×1.71\times for batch sizes of 3 and above (Table 6).

ModelBatch Sizeγ\gammaAvg BEBaseline ThroughputThroughput with G-MTPSpeedup
Gemma 4 26B MoE143.27248.9334.01.34×
Gemma 4 26B MoE243.27442.8625.91.41×
Gemma 4 26B MoE343.27590.0923.71.57×
Gemma 4 26B MoE443.27747.91180.11.58×
Gemma 4 26B MoE543.27849.91409.01.66×
Gemma 4 26B MoE643.28982.21524.61.55×
Gemma 4 26B MoE743.271053.31797.41.71×
Gemma 4 26B MoE843.281183.01965.71.66×
Gemma 4 26B MoE943.271244.22023.41.63×
Gemma 4 26B MoE1043.271366.12259.31.65×
Gemma 4 26B MoE1143.271419.72297.01.62×
Gemma 4 26B MoE1243.271527.22518.01.65×
Gemma 4 26B MoE1343.271571.02601.71.66×
Gemma 4 26B MoE1443.271635.32622.11.60×
Gemma 4 26B MoE1543.271720.82818.51.64×

Table 6. Batch size scaling performance: Gemma 4 26B MoE (γ\gamma = 4, measured on 2×2 \timesH100)

Finally, in Table 2, we compare the measurements obtained for Gemma 4 31B to other speculative drafting frameworks. To ensure a rigorous comparison and eliminate software confounders, all alternative drafters were evaluated under identical conditions to our primary baselines (vLLM serving engine, bfloat16, 2×H100). This comparison is focused on the 31B variant as alternative drafters like EAGLE-33 ((Li et al., 2025)) and DFlash4 ((Chen et al., 2026)}) are readily available, while standard implementations are lacking for the smaller, edge-optimized variants. As shown, G-MTP consistently outperforms both EAGLE-3 and DFlash across all evaluated horizons, achieving superior block efficiency and output TPS. While DFlash can draft faster thanks to its parallel design, it achieves lower block efficiency, leading to lower overall speedups compared to G-MTP.

ModelModeγ\gammaBlock Eff.Output TPSSpeedup
Gemma 4 31BBaselineN/A1.0073.21.00×1.00\times
Gemma 4 31BDFlash22.21142.21.94×1.94\times
Gemma 4 31BDFlash42.76173.62.37×2.37\times
Gemma 4 31BDFlash83.15190.52.60×2.60\times
Gemma 4 31BEAGLE-322.08136.41.86×1.86\times
Gemma 4 31BEAGLE-342.52153.82.10×2.10\times
Gemma 4 31BEAGLE-382.80151.52.07×2.07\times
Gemma 4 31BG-MTP22.45166.52.27×2.27\times
Gemma 4 31BG-MTP43.35208.82.85×2.85\times
Gemma 4 31BG-MTP84.26230.13.14×3.14\times

Table 2. Comparison of speedup for three different Gemma 4 31B drafters (2×H100, utilizing FlashAttention 4 kernels, batch size 1) on SPEED-Bench quality dataset.

On-device evaluation on a mobile NPU. We also evaluate G-MTP on the Edge TPU of a Google Pixel 11, using 4-bit quantized Gemma 4 E2B and E4B, greedy decoding, γ=3\gamma= 3 and at most 600 output tokens, on the 456 single-turn prompts of the SPEED-Bench qualitative set ([1]). To measure the on-device effect of the clustered vocabulary projection (§ 3.3), we compare drafters with the full vocabulary projection (no-CE) and with the clustered projection (CE). As shown in Table 3, G-MTP accelerates decoding by 1.89× (E2B) and 1.96× (E4B) without the clustered projection and by 1.92× and 1.98× with it. The clustered projection reduces the drafting time to 0.81× (E2B) and 0.79× (E4B) of the full-projection drafting time, but it also lowers the median acceptance rate (from 0.49 to 0.48 for E2B and from 0.52 to 0.50 for E4B), which offsets most of this saving at γ=3\gamma= 3: the net end-to-end gain is 1.01× for both models.

Speculative Decoding Adaptation via Multi-Token Prediction Distillation

To investigate the dynamics of speculative decoding during domain adaptation, we conduct experiments using the Gemma 4 E4B model as our target verification backbone (Mtarget\mathcal{M}_{\mathrm{target}}). For speculative drafting, we utilize the open-sourced G-MTP drafter (Mdrafter\mathcal{M}_{\mathrm{drafter}}).

To test our fine-tuning strategies across diverse structural shifts, we evaluate on two different output domains: (1) orkg/SciQA ([4]): a semantic parsing dataset that maps natural language questions to complex SPARQL database queries over the Open Research Knowledge Graph; and (2) GSM8K ((Cobbe et al., 2021)): a grade-school mathematical reasoning benchmark.

We formalize and compare three distinct domain adaptation regimes for speculative decoding systems, using the original pre-trained target and drafter without any fine-tuning as the reference (No SFT reference, row 1 of Table 4). The regimes below are numbered according to their rows in Table 4:

DatasetConfigurationRegimeNTP AccuracyAcceptance Rateβ\betaTarget Steps
SciQA1. No SFT reference68.40%22.36%2.5112,334
SciQA2. Target Fine-Tuned Only (SFT)99.23%8.96%1.6019,423
SciQA3. Joint Fine-Tuning (SFT + MTP)99.37%28.64%2.9610,595
SciQA4. Two-Stage Distillation99.23%26.48%2.8011,155
GSM8K1. No SFT reference74.83%40.17%3.738,414
GSM8K2. Target Fine-Tuned Only (SFT)88.01%28.67%2.9510,435
GSM8K3. Joint Fine-Tuning (SFT + MTP)88.00%40.66%3.778,143
GSM8K4. Two-Stage Distillation88.01%37.90%3.588,591

Table 4. Speculative Decoding Domain Adaptation Performance (Gemma 4 E4B). Fine-tuning the model without updating the MTP (row 2) can lead to regression in Block Efficiency (β\beta). Joint fine-tuning (row 3) matches or exceeds the β\beta of the no SFT reference (row 1), and two-stage distillation (row 4) recovers about 96% of it.

  1. Target Fine-Tuned Only (SFT): Only the target backbone Mtarget\mathcal{M}_{\mathrm{target}} is fine-tuned via standard autoregressive cross-entropy on the downstream dataset. The speculative drafter Mdrafter\mathcal{M}_{\mathrm{drafter}} remains frozen in its pre-trained state.

  1. Joint Fine-Tuning (SFT + MTP): Both Mtarget\mathcal{M}_{\mathrm{target}} and Mdrafter\mathcal{M}_{\mathrm{drafter}} are fine-tuned simultaneously from base checkpoints using a composite objective that combines SFT with drafter distillation.

  1. Two-Stage Distillation: The domain-adapted target model from regime 2 is kept frozen. The speculative drafter Mdrafter\mathcal{M}_{\mathrm{drafter}} is then trained via probability distillation to align with the target model’s updated representations.

During training (γtrain=3\gamma_{\mathrm{train}} = 3 lookahead), our objective combines the backbone’s cross-entropy with an MTP head loss that interpolates between next-token prediction and probability distillation (Total Variation Distance); in all reported experiments we use pure distillation (wdistill=1.0w_{\mathrm{distill}} = 1.0, Appendix § 8.§ 3). For inference evaluation (γ=7\gamma= 7 lookahead), we log every verification step across a test set of N=500N = 500 samples to record NTP Accuracy, Acceptance Rate, Block Efficiency (β\beta), and total Target Steps. All model updates are performed via Low-Rank Adaptation (LoRA) (Hu et al., 2022), with full details provided in Appendix § 8.§ 3.

Table 4 details the performance of all domain adaptation regimes evaluated across both domains.

When the target backbone is fine-tuned alone (row 2), Next-Token Prediction (NTP) accuracy jumps significantly on both domains (+30.83 percentage points on SciQA, +13.18 on GSM8K). However, the frozen pre-trained drafter suffers severe distributional decoupling. Consequently, Block Efficiency drops from 2.51 to 1.60 on SciQA (-36.3%), and from 3.73 to 2.95 on GSM8K (-20.9%).

Both Joint Fine-Tuning (row 3) and Two-Stage Distillation (row 4) successfully realign the drafter with the backbone. As shown in Table 4, both adaptation strategies recover most or all of the block efficiency of the no SFT reference (row 1). Given the small difference between the two drafter adaptation methods, we conclude that both domain adaptation frameworks are highly effective mechanisms for maintaining speculative drafting efficiency under target distribution shift. Despite the drafter being trained to predict only γtrain=3\gamma_{\mathrm{train}} = 3 future steps, our architecture empowers it to recursively extrapolate its predictions beyond the training horizon (e.g., to an extended γ=7\gamma= 7) and successfully reach high acceptance rates.

Discussion and Conclusion

We introduced an optimized Multi-Token Prediction (G-MTP) architecture for efficient speculative drafting directly integrated into the main model. By leveraging target activation injection and zero-copy KV-cache cross-attention, our approach bypasses the standalone drafter prefill latency and maintains high acceptance rates. The clustered vocabulary projection further reduces the per-step drafting cost, enabling competitive and robust wall-clock generation speedups across the Gemma 4 family (up to 3.14×). Furthermore, our domain adaptation analysis highlights that simple distillation effectively realigns G-MTP drafters under distribution shift, preserving high block efficiencies where fine-tuning only the backbone degrades them. To ensure reproducibility and facilitate ongoing research, the code and distillation recipes for these domain adaptation experiments are made publicly available.5

Limitations and Future Work While our architectural innovations—specifically the KV-cache sharing and clustered vocabulary projection—were designed to bypass on-device constraints, our comparisons across the full model family were conducted on server GPUs (H100) to standardize our evaluation baselines. Our on-device measurements cover only E2B and E4B on two devices (a smartphone GPU without the clustered projection, and a mobile NPU with and without it). A natural direction for future work is a broader on-device study across devices, backends and batch sizes that also quantifies the memory and battery benefits of each component. Finally, this paper focuses on the drafter architecture and its adaptation to fine-tuned backbones; the pre-training recipe of the released G-MTP drafters is out of scope. All domain adaptation experiments in § 5 start from the publicly released drafter checkpoints and can therefore be reproduced without it.

Acknowledgments

We wish to thank Eden Cohen, Niv Giladi, Ethan Baron, Alon Levkovitch, Omer Vandsburger, Eliya Nachmani, Nadav Bar, Pauline Sho, Joe Zou, Ryan Mullins, Sindhu Raghuram, Mayank Chaturvedi, and Mohammadreza Heydary for their help, support, and feedback on the research.

Appendix

Gamma Selection and Batch Size Scaling

To determine the optimal lookahead horizon (γ\gamma) for the main generation speedup comparisons presented in Table 1, we empirically evaluated the Block Efficiency (BE), resulting speculative throughput (tokens/s), and overall wall-clock speedup across different values of γ\gamma. Table 5 details these measurements for each model in the Gemma 4 family.

As γ\gamma increases, the theoretical block efficiency monotonically rises due to longer draft sequences; however, wall-clock throughput eventually degrades as the penalty of drafting and verifying longer, lower-probability draft tokens outweighs the accepted generation lengths. The optimal γ\gamma varies heavily depending on the model’s structural capacity to parallelize verification (Figure 3).

Block Efficiency and speculative throughput versus lookahead horizon

Figure 3. Empirical scaling of theoretical Block Efficiency (BE) and speculative wall-clock throughput across varying lookahead horizons (γ\gamma). While longer draft sequences monotonically increase token acceptance, throughput degrades for some models (e.g., 26B MoE) as drafting and verification costs exceed generation gains.

While optimal γ\gamma dictates the length of the draft sequence, the batch size heavily influences the computational amortization of the verification phase, particularly for Mixture-of-Experts (MoE) architectures running highly concurrent workloads. As detailed in Table 6, we evaluate throughput scaling for the Gemma 4 26B MoE at γ=4\gamma= 4 across batch sizes 1 through 15. At low batch settings, the relative verification overhead of routing discrete draft tokens through parallel experts yields moderate speedups of 1.34× (batch size 1) and 1.41× (batch size 2). As the batch size increases, the cost of expert activation is amortized across sequences, and the speedup plateaus between 1.55× and 1.71× for batch sizes of 3 and above.

On-Device Measurements

Table 7 complements the server-side evaluation with on-device measurements of the G-MTP drafters for Gemma 4 E2B and E4B, deployed with the LiteRT-LM runtime on the GPU of a Samsung Galaxy S26 Ultra. These runs use the quantized LiteRT-LM releases of the models, greedy decoding, a lookahead of γ=3\gamma= 3 (matching the training lookahead) and the full vocabulary projection, i.e., the clustered vocabulary projection is not enabled. The benchmark contains 76 short prompts (median 29 input tokens) from four task types (8 free-form generation, 30 tone rewriting, 10 summarization and 28 code generation prompts), with generation capped at 512 tokens; every prompt is run with and without G-MTP.

ModelAcceptance RateBaseline (tokens/s)G-MTP (tokens/s)SpeedupFree-formRewriteSummarizeCode
Gemma 4 E2B0.5450.585.41.69×1.36×1.73×1.78×1.67×
Gemma 4 E4B0.5921.647.12.18×1.71×2.22×2.07×2.25×

Table 7. On-device decoding throughput of Gemma 4 E2B and E4B with LiteRT-LM on the GPU of a Samsung Galaxy S26 Ultra (γ\gamma = 3, greedy decoding, full vocabulary projection). Acceptance rate and throughput values are medians over the 76 prompts; the acceptance rate is the draft acceptance rate reported by the runtime. Speedups are ratios of median throughput with and without G-MTP, over all prompts and per task type.

G-MTP accelerates decoding by 1.69× for E2B and 2.18× for E4B. These speedups are in the same range as those measured on H100 at the same lookahead (1.91× and 2.15×; Table 5), although the prompt sets differ, and, as on H100, E2B obtains a smaller speedup than E4B. For both models, free-form generation has the lowest acceptance rate and the smallest speedup.

Details on Domain Fine-Tuning

Parameter-Efficient Fine-Tuning

To ensure computational efficiency and prevent catastrophic forgetting during domain specialization, all model updates are performed via Low-Rank Adaptation (LoRA). We inject rank-r=8r = 8 adapters (α=16\alpha= 16, dropout = 0.05, no bias) into:

  • Target Backbone (Mtarget\mathcal{M}_{\mathrm{target}}): Attention projections (q_proj, k_proj, v_proj, o_proj). This introduces 5,914,624 trainable parameters (∼\sim0.07% of the target backbone).

  • Speculative Drafter (Mdrafter\mathcal{M}_{\mathrm{drafter}}): Attention and MLP projections (q_proj, o_proj, gate_proj, up_proj, down_proj), as well as internal state mapping layers (pre_projection corresponding to WdownW_{\mathrm{down}}, and post_projection corresponding to WupW_{\mathrm{up}} from § 3.1). This introduces 385,024 trainable parameters (∼\sim0.5% of the drafter head). During fine-tuning and evaluation, the drafter keeps its clustered vocabulary projection with the original centroids: logits are computed only for tokens in the top-32 clusters, and all other tokens receive a constant minimum logit; the TVD loss is computed over the full 262K vocabulary of the resulting distribution. Since the output projection is not updated, the offline clusters remain valid after fine-tuning.

Under the Joint Fine-Tuning regime, the total overall system capacity mapped to trainable adapters is bounded to 6,299,648 parameters. All models are trained for 3 epochs using AdamW (β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, weight decay = 0.01) with a peak learning rate of η=2×10−4\eta= 2 \times10^{-4} for SciQA and η=1×10−4\eta= 1 \times10^{-4} for GSM8K (using bfloat16 mixed precision, effective batch size = 16, linear warmup/cosine decay, gradient clipping at ∥g∥2=0.3\lVert g \rVert_2 = 0.3).

Speculative G-MTP Objective & Domain Adaptation

For a training lookahead horizon of γtrain\gamma_{\mathrm{train}} steps (γtrain=3\gamma_{\mathrm{train}} = 3 in our experiments), the speculative drafter takes the target model’s hidden representation ht(L)h_t^{(L)} at step tt and predicts candidate distributions across t+2,…,t+γtrain+1t + 2, \ldots, t + \gamma_{\mathrm{train}} + 1. The unified training objective is:

Ltotal=Lbackbone_LM+λMTP∑k=1γtrainLhead_k(4)\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{backbone\_LM}} + \lambda_{\mathrm{MTP}} \sum_{k=1}^{\gamma_{\mathrm{train}}} \mathcal{L}_{\mathrm{head\_}k} \tag*{(4)}

where Lbackbone_LM\mathcal{L}_{\mathrm{backbone\_LM}} is the standard next-token cross-entropy and λMTP=0.4\lambda_{\mathrm{MTP}} = 0.4. For each lookahead step kk (i.e., the kk-th recursive step of the single shared drafter), the head loss Lhead_k\mathcal{L}_{\mathrm{head\_}k} interpolates between ground-truth cross-entropy (LNTP\mathcal{L}_{\mathrm{NTP}}) and Probability Distillation (LTVD\mathcal{L}_{\mathrm{TVD}}):

Lhead_k=(1−wdistill)LNTP_k+wdistillLTVD_k(5)\mathcal{L}_{\mathrm{head\_}k} = (1 - w_{\mathrm{distill}})\mathcal{L}_{\mathrm{NTP\_}k} + w_{\mathrm{distill}}\mathcal{L}_{\mathrm{TVD\_}k} \tag*{(5)}

Here, LNTP_k=−log⁡Pdrafter_k(xt+k+1∣ht(L),x≤t,xt+1:t+k)\mathcal{L}_{\mathrm{NTP\_}k} = -\log P_{\mathrm{drafter\_}k}(x_{t+k+1} \mid h_t^{(L)}, x_{\leq t}, x_{t+1:t+k}) computes the standard autoregressive cross-entropy. Note that during training the drafter is teacher-forced: the tokens xt+1:t+kx_{t+1:t+k} fed back as drafter inputs are the ground-truth tokens, whereas at inference time the drafter consumes its own predictions x~t+k\tilde{x}_{t+k} (Equation (3)). For all speculative adaptation experiments reported, we set wdistill=1.0w_{\mathrm{distill}} = 1.0, dedicating the drafter entirely to teacher mimicking. The TVD term minimizes the distance against the target’s updated distribution:

LTVD_k=12∑v∈V∣Pdrafter_k(v∣ht(L),x≤t,xt+1:t+k)−Ptarget(v∣x≤t+k+1)∣(6)\mathcal{L}_{\mathrm{TVD\_}k} = \frac{1}{2} \sum_{v \in\mathcal{V}} \left| P_{\mathrm{drafter\_}k}(v \mid h_t^{(L)}, x_{\leq t}, x_{t+1:t+k}) - P_{\mathrm{target}}(v \mid x_{\leq t+k+1}) \right| \tag*{(6)}

To strictly isolate the G-MTP updates from warping the target backbone’s representations during Joint Fine-Tuning, we apply a stop-gradient over the connection spanning the backbone’s initial hidden state ht(L)h_t^{(L)} and the shared KV cache.

Evaluation Metrics

Given accepted tokens (MsM_s) and proposed tokens (PsP_s) during forward pass s∈{1,…,Stotal}s \in\{1, \ldots, S_{\mathrm{total}}\}:

  • NTP Accuracy (ANTP\mathcal{A}_{\mathrm{NTP}}): The teacher-forced next-token prediction accuracy, defined as the frequency with which the target model’s top-1 prediction matches the ground-truth token.

  • Acceptance Rate (α\alpha): α=∑Ms∑Ps\alpha= \frac{\sum M_s}{\sum P_s}.

  • Block Efficiency (β\beta): β=1+∑MsStotal\beta= 1 + \frac{\sum M_s}{S_{\mathrm{total}}}.

  • Target Steps: Total target forward passes StotalS_{\mathrm{total}} required to decode the entire test set (lower indicates faster generation).

References

  1. [1]T. Abramovich, M. Ashkenazi, C. Putterman, B. Chislett, T. Mitra, B. D. Rouhani, R. Zilberstein, and Y. Geifman. SPEED-Bench: A unified and diverse benchmark for speculative decoding, 2026. URL https://arxiv.org/abs/2604.09557.
  2. [2]P. S. Aishwarya, P. A. Nair, Y. Samaga, T. Boyd, S. Kumar, P. Jain, and P. Netrapalli. Tandem transformers for inference efficient LLMs, 2024. URL https://arxiv.org/abs/2402.08644.
  3. [3]Z. Ankner, R. Parthasarathy, A. Nrusimha, C. Rinard, J. Ragan-Kelley, and W. Brandon. Hydra: Sequentially-dependent draft heads for Medusa decoding, 2024. URL https://arxiv.org/abs/2402.05109.
  4. [4]S. Auer, D. A. C. Barone, C. Bartz, E. G. Cortes, M. Y. Jaradeh, O. Karras, M. Koubarakis, D. Mouromtsev, D. Pliukhin, D. Radyush, I. Shilin, M. Stocker, and E. Tsalapati. The SciQA scientific question answering benchmark for scholarly knowledge. Scientific Reports, 13(1):7240, May 2023. ISSN 2045-2322. doi: 10.1038/s41598-023-33607-z. URL https://doi.org/10.1038/s41598-023-33607-z.
  5. [5]P. Badarayani, A. Jay, C. Zhou, D. Julienne, Y. Gao, T. Chen, G. Zerveas, I. Zabir, X. Zhou, C. Quirk, and X. Song. Match the distribution, not the compute: Post-training multi-token prediction heads, 2026. URL https://arxiv.org/abs/2610.00888.

Paper details

vlnc:W_uf3ghw9h
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents