WorksComputer Science
ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction
Abstract
Existing Multi-token prediction (MTP) methods are auxiliary by design: the multi-horizon loss mostly trains modules attached to the backbone, and each added horizon costs another module and another pass. We propose Efficient Causal Self-Recursion (ECHO), which removes both costs. The backbone's own final layer predicts every horizon, so the supervision lands on the parameters that generate the next token, and Dynamic Depth Sampling (DDS) keeps deeper horizons affordable by drawing the training depth with probabilities that decay along depth. Future prediction thus becomes part of the autoregressive objective rather than a task attached to it, and the gains follow on both sides of it. In pretraining, added depth converts into downstream accuracy where auxiliary modules saturate, concentrating on reasoning and code at half the MTP training overhead of training every depth. At inference, the pretrained model is its own drafter: it drafts beyond the depth it was trained for, and keeps its draft-target alignment on the open-ended, high-entropy generation where post-hoc drafters lose theirs. The advantage survives a doubled token budget, a repeated pretraining run, and supervised finetuning.
https://github.com/ernie-research/ECHO
Introduction
Large language models (LLMs) [1, 52, 3, 37] are predominantly trained with the Next-Token Prediction (NTP) objective. While NTP has demonstrated remarkable scalability and effectiveness, it provides supervision only at the immediate next-token horizon, leaving potentially useful future information unexploited. Moreover, autoregressive decoding remains inherently sequential, resulting in limited inference throughput.
To better utilize future information, recent work has explored Multi-Token Prediction (MTP) as an auxiliary objective on top of NTP. By introducing additional future-token prediction targets, MTP provides denser supervision during training and can generate draft tokens for speculative decoding, enabling inference acceleration.
Early MTP methods predict multiple future positions independently from the same hidden state Gloeckle et al. [20], providing denser supervision but ignoring causal dependencies among future tokens. Subsequent causal MTP methods such as DeepSeek-V3 Liu et al. [37] address this limitation by introducing autoregressive dependencies among future prediction steps, significantly improving prediction quality and improving the backbone model’s overall performance [37, 7, 58].
However, both causal and parallel MTP remain auxiliary by construction: future-token prediction is carried out by dedicated modules attached to the backbone, rather than being integrated into the autoregressive generation process itself. This leads to two limitations:
Indirect Multi-Horizon Supervision MTP losses primarily supervise the auxiliary prediction modules. Consequently, the backbone model receives multi-horizon learning signals only indirectly, potentially limiting their impact on long-horizon planning.
Costly Long-Horizon Supervision Denser supervision requires deeper prediction horizons, yet each additional horizon adds a prediction module in memory and another pass in compute. This cost limits the prediction horizon in large-scale causal MTP pretraining, typically to a single draft token.
To address the above limitations, we introduce Efficient Causal Self-Recursion (ECHO), a causal self-recursive framework that reuses the backbone’s final layer for future prediction and samples per-step prediction depths with decaying probabilities to reduce the amortized MTP training overhead.
Final-layer parameter sharing ensures that multi-horizon losses directly shape the backbone, rather than being absorbed by horizon-specific auxiliary modules. As shown in Figure 3a, this allows ECHO to benefit from deeper supervision: its downstream performance continues to improve at depth 3, whereas DeepSeek’s MTP [37] does not, consistent with observations in MiMo [41]. More fundamentally, training the same parameters responsible for next-token generation to predict multiple future tokens encourages the backbone to anticipate how the sequence unfolds over subsequent steps, as evidenced by ECHO’s substantial gains on reasoning tasks.
Figure 3. What final-layer sharing buys, in pretraining and at inference. (a) Mean per-task score difference from MTP-1 within each task category, over the fourteen benchmarks of Table 1, with the category size below each label and the fourteen-task averages in the legend. (b) Accepted length at with rejection sampling, against the target’s predictive entropy; the warm-start EAGLE-3 drafter is trained from DeepSeek’s MTP baseline, following Kimi Team [30].
To make long-horizon supervision computationally efficient, we introduce Dynamic Depth Sampling (DDS), which samples prediction depths with decaying probabilities. This simultaneously reduces the amortized MTP training overhead and mitigates the impact of increasing prediction errors and supervision noise at greater depths. After supervised finetuning, ECHO’s advantage grows further.
Beyond its pretraining benefits, ECHO entirely eliminates the need for post-hoc drafter finetuning. DDS makes longer-horizon supervision affordable, while final-layer parameter sharing enables depth extrapolation: since the same shared layer is reused at every draft step, ECHO can extend drafting beyond the trained prediction depth while maintaining high acceptance rates, at an amortized MTP training overhead of only 1.5 depth-1 steps (see § 4.1). By drafting directly with its pretrained weights, ECHO avoids the overfitting to academic benchmarks associated with post-hoc drafter finetuning [38], yielding substantial acceptance-length gains in realistic, high-entropy generation settings as shown in Figure 3b.
Our main contributions are summarized as follows:
We introduce Efficient Causal Self-Recursion (ECHO), which predicts every future horizon by recursively applying the backbone’s own final layer, with the per-step prediction depth drawn from a distribution decaying along depth.
ECHO imposes direct multi-horizon supervision, beating auxiliary MTP on reasoning and code at matched depth while holding on QA & multiple choice, and widening its lead on both classes
after supervised finetuning.
ECHO makes long-horizon prediction cheap in compute and free in parameters, and keeps gaining acceptance when extrapolating in depth, where auxiliary MTP saturates. It eliminates drafter finetuning entirely, yet outperforms post-hoc drafters, especially on open-ended generation.
Related Work
Multi-Token Prediction
Multi-Token Prediction (MTP) augments next-token prediction with future-token targets, improving language modeling while providing draft tokens for speculative decoding. It inherits the architecture of speculative draft models and differs in the training regime: draft models are finetuned post-hoc against a frozen backbone on a limited dataset, whereas MTP modules are optimized jointly with the backbone throughout pretraining.
Parallel MTP predicts multiple future positions independently from several output heads on a shared backbone Gloeckle et al. [20], following Medusa Cai et al. [9]. Causal MTP instead follows EAGLE Li et al. [35]: DeepSeek-V3 Liu et al. [37] feeds dedicated prediction modules the previous hidden state together with shifted token embeddings, preserving autoregressive dependencies among future steps, and GLM-5 Zeng et al. [58] shares those parameters across all depths, as EAGLE does, to reduce memory cost.
Post-hoc Drafter
Post-hoc drafters are trained on top of a frozen target model by finetuning on target-generated responses, and differ mainly in how a block of draft tokens is produced. Medusa Cai et al. [9] attaches parallel heads that draft every position from the final hidden state, while EAGLE-3 Li et al. [36] re-enters a lightweight decoder once per drafted token, conditioned on drafted states, preserving token-to-token dependency. Recent parallel drafters build on DFlash Chen et al. [11], which predicts the whole block in one pass, with DSpark Cheng et al. [13] restoring intra-block dependency through a lightweight causal head.
Two limitations are nonetheless inherent to post-hoc drafting. First, the drafter is narrowly aligned with the target, being finetuned on a dataset orders of magnitude smaller than the pretraining corpus. Second, scaling that training to the pretraining corpus is compute-heavy from the multi-step or multi-layer overhead, which has not been shown to improve the backbone’s downstream performance and may even degrade training. ECHO overcomes both: joint pretraining aligns it with the target, and its supervision improves the backbone at modest overhead.
Looped Transformers
Looped transformers reuse a shared block of layers across iterations, buying computational depth without new parameters: parameter efficiency ([31]), adaptive computation ([16]), reasoning at LLM scale ([48, 61]), test-time compute scaling ([18]), and depth allocated per token ([2]). In all of them recursion refines the same token position under a next-token objective; ECHO instead recurses along the horizon, each step conditioning on the next future token to predict one position further ahead.
Yet both buy structure rather than recall, and for looping that is a trade: at matched parameters and compute it leads on reasoning but trails on perplexity and closed-book QA ([48]), and at scale its gain is credited to knowledge manipulation rather than capacity, measured against external dense families rather than a same-recipe baseline ([61]). ECHO pays no such price: mathematics, reasoning and code gain while multiple choice and closed-book QA hold (Figure 3a), and both classes gain after finetuning (Figure 4b). Looping, latent chain-of-thought ([21]) and ECHO thus price latent computation differently: repeated passes over one position, annotated chains, and extra positions — the one price that accepted tokens can repay.
Figure 4. (a) Deeper sampling trails ECHO-1.5 before finetuning. (b) Finetuning widens the lead on both classes. (c) The lead against the spread two matched runs measure. The advantage persists under deeper sampling, supervised finetuning and a doubled token budget. Mean per-task gain over the matched DeepSeek baseline, by task class: MTP-1 before finetuning, MTP-1-Instruct after it. (c) shows the gap at step 78k and at step 156k, and the error bars are the run-to-run standard deviation from an independent repeat of the 78k recipe, 0.08 on reasoning & code and 0.38 on QA & multiple choice. Doubling the budget moves the reasoning & code gap by 0.01, a sixth of that spread, whereas the repeat moves the QA & multiple choice gap by 0.54, more than its own. The reasoning & code advantage therefore survives both conditions and the run-to-run spread, while the QA & multiple choice difference stays inside it.
ECHO: Efficient Causal Self-Recursion
We first show that final-layer sharing benefits the backbone through direct multi-horizon supervision, and then dynamic depth sampling makes it affordable at greater depth.
Direct Multi-Horizon Supervision via Final-Layer Sharing
We show that reusing the backbone’s final layer turns multi-horizon supervision into a direct training signal: downstream performance improves over auxiliary MTP, and shows further gains at greater depth.
Causal Self-Recursion: we follow the autoregressive design of DeepSeek-V3 ([37]) to preserve causal dependency, where each additional horizon is predicted from the previous final-layer hidden state and the embedding of the previously predicted token. As shown in Figure 2, rather than attaching a dedicated module for each depth, we recursively apply the backbone’s final decoder layer and output head at every horizon, connected to the previous step by a lightweight fusion layer :

Figure 2. Overview of ECHO. The backbone’s final decoder layer and output head are reused to predict every future horizon. Dynamic Depth Sampling (DDS) draws one prediction depth per optimizer step, with probabilities that decay along depth. Green blocks denote layers shared with the backbone.
apart from the lightweight fusion layer, every parameter used for future prediction is also used for next-token generation. Supervision from every horizon shapes the backbone directly, and the same layer can be applied repeatedly to draft beyond the trained depth.
We show two benefits of the resulting direct multi-horizon supervision:
Sharing improves downstream performance, especially in reasoning capability. To isolate the effect of sharing, we compare ECHO with DeepSeek’s MTP (Liu et al., [37]) at matched prediction depth (Figure 3a). Under this comparison, sharing the final layer gives ECHO the lead at depth 1, most clearly on math, reasoning and code (+0.79, +0.57 and +1.35), and it reaches +2.63 on reasoning and +2.23 on code at depth 3. The difference lies in where multi-horizon supervision mainly lands. With auxiliary modules, the signal is largely absorbed by the modules dedicated to individual horizons, leaving the backbone’s hidden layers specialized for the next token (Mehra et al., [43]). Sharing removes this bottleneck by construction: since the final layer predicts every horizon, supervision from all horizons is concentrated on the parameters that generate the next token.
Sharing promotes longer-horizon supervision. To examine how each design uses added depth, we compare each family with itself from depth 1 to depth 3. Averaged over all fourteen benchmarks, deeper supervision improves ECHO by 0.63 points, whereas DeepSeek’s MTP gains only 0.16, and ECHO’s advantage on reasoning and code persists as depth increases (Figure 3a). This divergence follows from where each new horizon’s loss goes: in ECHO it reaches the backbone directly, whereas in DeepSeek’s MTP it mostly trains the newly added module, as discussed above. This explains why auxiliary MTP rarely goes deeper in practice: DeepSeek-V3 trains with a single MTP module (Liu et al., [37]), and MiMo finds that adding MTP layers brings no further improvement (LLM-Core Xiaoumi, 2025).
Affordable Long-Horizon Supervision via Dynamic Depth Sampling (DDS)
We show that dynamically sampling the training depth lowers the cost of long-horizon supervision without sacrificing its gains in downstream performance and drafting quality.
Dynamic Depth Sampling. At each optimizer step, DDS draws a prediction depth over and unrolls the recursion of equation (1) for depths 1 through (Figure 2), so the MTP training overhead scales with the expected depth
In practice, we use with , giving , half the overhead of training all three depths. Throughout, this MTP training overhead is reported in units of the depth-1 module, that is, unrolls of the shared layer per optimizer step; it is the increment on top of the backbone, not the end-to-end cost of a training step.
DDS lowers the MTP training overhead while preserving downstream benefit. ECHO-1.5 matches ECHO-3 on the benchmark average (36.34 vs. 36.36, Figure 3a) at half the MTP training overhead. This holds because DDS changes how often each depth is trained, not which depths are trained. Since depths are sampled from , ECHO-1.5 still receives depth-3 supervision, only not at every step, so it keeps the benefit of long-horizon supervision without unrolling every depth in every batch. Because deeper horizons condition on longer chains of recursive hidden states, their supervision grows noisier with depth, so DDS concentrates training on reliable shallow positions and protects the shared backbone from deep-horizon noise, much as stochastic depth trains deeper layers less often [25]. At this cost, ECHO-1.5 also outperforms MTP-1 on math, reasoning and code (+1.00, +1.31, +2.37) and still leads MTP-3 at twice its budget (+0.55, +2.31, +1.67; Figure 3a and Table 1).
| Category | Benchmark | ECHO-1 | ECHO-3 | ECHO-1.5(3) | MTP-1 | MTP-3 | ECHO-2(7) |
| Mathematics | GSM8K | 61.49 | 59.82 | 63.31 | 61.64 | 63.08 | 61.33 |
| MATH | 36.04 | 36.01 | 34.64 | 34.31 | 33.78 | 33.56 | |
| Reasoning | BBH | 46.44 | 50.02 | 48.55 | 46.91 | 40.18 | 48.45 |
| BBEH | 6.63 | 6.41 | 6.09 | 5.33 | 6.96 | 5.43 | |
| PIQA | 77.57 | 78.26 | 78.38 | 77.68 | 77.97 | 78.09 | |
| WinoGrande | 63.85 | 64.01 | 64.40 | 62.27 | 63.06 | 62.59 | |
| Multi-subject MC | MMLU | 55.93 | 57.10 | 56.93 | 56.39 | 58.59 | 56.95 |
| MMLU-Pro | 26.02 | 26.37 | 27.58 | 26.60 | 29.37 | 27.90 | |
| Code | HumanEval+ | 26.22 | 29.88 | 29.27 | 23.17 | 25.00 | 26.83 |
| MBPP+ | 47.62 | 48.68 | 47.62 | 47.62 | 47.88 | 47.88 | |
| LiveCodeBench | 7.00 | 7.00 | 7.00 | 6.00 | 6.00 | 5.25 | |
| Closed-book QA | PreciseWikiQA | 7.88 | 8.30 | 8.13 | 7.48 | 7.52 | 7.54 |
| HotpotQA | 21.35 | 21.29 | 20.65 | 21.18 | 20.86 | 20.57 | |
| PopQA | 16.23 | 15.83 | 16.16 | 16.44 | 15.09 | 15.57 | |
| Overall Average (14 tasks) | 35.73 | 36.36 | 36.34 | 35.22 | 35.38 | 35.57 |
Table 1. Per-benchmark scores at step 78,000 (base models, no EMA).
Pushing the Frontier of Speculative Decoding under Realistic Serving
We analyze how ECHO’s end-to-end paradigm intrinsically aligns the drafter with the target, entirely eliminating the need for post-hoc drafter finetuning.
Training Shallow, Drafting Deep
We first show how DDS aligns training with the acceptance dynamics of speculative decoding, then demonstrate how last-layer sharing enables depth extrapolation beyond the trained depth.
DDS aligns training with speculative decoding. At inference, the drafter proposes tokens and the target verifies them in parallel, but draft tokens are committed only up to the first rejection [32]. With acceptance rate at position , position is reached with probability [56, 60], so the expected accepted length is
where is the accepted length, the bonus token plus the accepted drafts. A rejection at depth thus truncates the drafts at and after it, so shallow depths dominate accepted length. DDS effectively trains depth with a frequency that decays along depth,
so gives . Under lossless verification, the rejection rate is [32]. LK losses [47] exploit this identity to optimize acceptance directly, but treat each position equally. Instead, relating the rejection rate to the data distribution and accounting for truncation across depths gives Section B
where bounds the MTP loss. The last inequality compares DDS with the uniform depth supervision () and holds for schedules that decay toward the optimum (Theorem B.1). The uniform depth supervision weights every depth’s error equally, whereas decaying tightens the bound, so reducing the training loss translates more directly into accepted length. DDS thus concentrates target-draft alignment on the shallow depths that determine acceptance, preserving accepted length despite reduced supervision at greater depths. Existing drafters encode this decay by reweighting the loss while still training every depth at every step [11, 13]; DDS instead encodes it in how often each depth is trained, so the training compute follows the decay.
Extrapolating Beyond the Training Horizon. We refer to drafting beyond the maximum depth seen during training as depth extrapolation. As shown in Figure 1b, beyond the trained depth (), ECHO keeps gaining at a nearly constant rate, and ECHO-1.5 retains this gain at half the MTP training overhead of ECHO-3, whereas the gains of MTP-3 shrink with each step and saturation. Sharing also makes extrapolation the natural complement to DDS: training concentrates supervision on shallow depths, and recursion carries it to deeper ones at inference.
Figure 1. Multi-horizon supervision on the backbone’s own final layer buys downstream performance after pre-training and supervised fine-tuning (a) and drafting depth beyond what was trained (b).
Staying Aligned under realistic setting
We then show that ECHO outperforms existing post-hoc drafters, with larger gains on high-entropy, realistic, and open-ended tasks, thanks to two complementary factors: causal drafting ensures consistency among draft tokens, while joint pretraining aligns the draft distribution with the target.
Causal Drafting Matters in Open-Ended Generation. As shown in Figure 3b, ECHO and other causal drafters outperform parallel drafters on high-entropy tasks, consistent with Liu et al. [38]; Figure 8 shows the same ordering in decode speedup. When the next tokens are uncertain, the target’s choice is hard to anticipate, so the probability of acceptance falls off steeply with depth and most of a long draft is discarded. A causal drafter adapts to this by extending its draft step by step, while a parallel drafter emits a fixed-length block whose deeper tokens are mostly rejected at verification. The balance shifts on simple code and mathematics benchmarks such as HumanEval, MBPP, and GSM8K, where long stretches of output are determined by syntax or derivation, and block-parallel drafting becomes worthwhile (Liu et al., [38]), whereas realistic software engineering tasks such as SWE-bench remain high-entropy and still favor causal drafting (Figure 3b). Such simple tasks, however, are a small fraction of real-world usage, account for under 8% of consumer ChatGPT messages, while open-ended practical guidance, information seeking, and writing account for nearly 80% (Chatterji et al., [10]).

Figure 8. Decode speedup at with rejection sampling, over the same cells as Figure 3b. The warm-start EAGLE-3 drafter is trained from DeepSeek’s MTP baseline, following Kimi Team [30], and is normalized by the baseline of its own run.
Joint Pretraining Closes the Alignment Gap. Acceptance depends on how closely the draft distribution matches the target’s (Leviathan et al., [32]; Zhou et al., [59]), and a drafter can better match the target where it has seen the target’s data. ECHO is trained with the target end-to-end and sees its full pretraining corpus. In contrast, auxiliary drafters are fit to a frozen target on a small finetuning set, so their acceptance drops under distribution shift (Liu et al., [40]; Hong et al., [24]). The EAGLE-3 pair in Figure 3b isolates this effect: with architecture and finetuning data held fixed, warm-started EAGLE-3 achieves a longer accepted length than its cold-start counterpart, consistent with recent findings that pretrained initialization strengthens drafters, most on out-of-domain inputs (Bie et al., [5]). Yet on open-ended tasks, warm-starting still falls short of ECHO, with the largest gap on SWE-bench. ECHO is instead optimized jointly with the target on every pretraining update, so its alignment is maintained throughout training, in line with the high acceptance rates reported for MTP modules trained jointly during pretraining (Liu et al., [37]; LLM-Core Xiaomi, [41]).
The Advantage Holds across Data, Depth and Runs
We finally examine whether ECHO’s advantage holds along three axes: more pretraining data, deeper sampling, and repeated runs.
Scaling with Data
ECHO’s advantage persists with more pretraining data. We extend pretraining of ECHO-1 and DeepSeek’s MTP-1 from step 78k to step 156k, doubling the token budget. Both models improve, and ECHO-1 keeps its lead over DeepSeek’s MTP-1 measured at the same step: +0.52 at step 78k and +0.50 at step 156k, and on reasoning & code it is +0.88 and +0.89 (Figure 4c, Table 2). This indicates that ECHO’s advantage is stable under data scaling rather than an early-training effect that more data would erase.
| Benchmark | Baseline ECHO, run 1, step 78k | Baseline DeepSeek, run 1, step 78k | Repeat run ECHO, run 2, step 78k | Repeat run DeepSeek, run 2, step 78k | Twice the budget ECHO, run 1, step 156k | Twice the budget DeepSeek, run 1, step 156k |
| GSM8K | 61.49 | 61.64 | 60.05 | 60.73 | 64.82 | 65.05 |
| MATH | 36.04 | 34.31 | 34.58 | 35.37 | 38.49 | 36.65 |
| BBH | 46.44 | 46.91 | 48.15 | 47.79 | 48.31 | 48.79 |
| BBEH | 6.63 | 5.33 | 5.83 | 5.59 | 6.18 | 5.29 |
| PIQA | 77.57 | 77.68 | 78.67 | 76.64 | 79.71 | 79.59 |
| WinoGrande | 63.85 | 62.27 | 64.96 | 62.27 | 65.67 | 63.30 |
| MMLU | 55.93 | 56.39 | 56.08 | 56.30 | 59.23 | 59.36 |
| MMLU-Pro | 26.02 | 26.60 | 27.04 | 24.21 | 26.23 | 27.28 |
| HumanEval+ | 26.22 | 23.17 | 28.05 | 25.00 | 29.88 | 26.83 |
| MBPP+ | 47.62 | 47.62 | 49.21 | 48.68 | 50.00 | 50.26 |
| LiveCodeBench | 7.00 | 6.00 | 6.75 | 5.25 | 7.25 | 6.50 |
| PreciseWikiQA | 7.88 | 7.48 | 8.12 | 8.00 | 8.56 | 9.18 |
| HotpotQA | 21.35 | 21.18 | 22.11 | 22.10 | 23.23 | 21.90 |
| PopQA | 16.23 | 16.44 | 15.48 | 16.20 | 16.38 | 16.97 |
Table 2. One recipe under two perturbations, both against the same baseline. Baseline is the run reported in Table 1. Repeat run is an independent pretraining run of the same recipe at the same step. Twice the budget is the baseline run continued to step 156k.
Scaling with Depth
ECHO’s advantage holds across sampling depths. We next train ECHO-2(), which samples over with , at 4/3 the MTP training overhead of ECHO-1.5(). It trails ECHO-1.5() in pretraining but overtakes it after SFT by 0.49, reaching the largest lead over DeepSeek’s MTP-Instruct of any configuration (+2.10; Table 3). The gap in pretraining may be due to errors accumulating at deeper positions, which makes their supervision noisier and draws training away from the reliable shallow depths. After SFT, however, the deeper supervision appears to pay off, possibly because finetuning makes better use of what the backbone learned.
Stability across Runs
ECHO’s lead is robust to training and evaluation noise. A reported gap can come from noise rather than the method itself. We eliminate evaluation noise by decoding greedily, so the remaining source is training. To measure it, we pretrain ECHO-1 and DeepSeek’s MTP-1 a second time under the same recipe and compare the two runs at the same step. Across the two runs, ECHO leads DeepSeek’s MTP-1 by +0.52 and +0.78, and its lead on reasoning & code shifts by only 0.11 (Figure 4c; Section D). ECHO’s overall and reasoning & code leads are therefore reproducible across runs.
Experiment Setup
Full architecture, prompt, scoring and speculative decoding details are given in Section A.
Models and Baselines: Every configuration shares a 10B-A1B Latent MoE backbone (Blakeman et al., [7]) with 24 layers, multi-head latent attention (Liu et al., [37]), and 256 routed experts with eight active per token (Dai et al., [15]), trained under the same recipe so that the comparison isolates the MTP design.
Pretraining: 600B tokens at sequence length 4,096 and batch size 1,920 in bfloat16 with Muon (Jordan et al., [28]), on 24 nodes of eight H800 GPUs.
Dynamic Depth Sampling: ECHO-1.5() samples over with , giving and ECHO-2() samples over with , giving .
Supervised Finetuning: Every configuration is finetuned with the same recipe, 5,000 steps over one pass of an in-house chat corpus at context length 32,768, keeping its pretrained MTP depths.
Computational Cost: Each pretraining run takes approximately 3 days on 192 NVIDIA H800 GPUs with a global batch size of 1,920, followed by 5 hours of SFT on the same hardware. Post-hoc drafter training uses 24 H800 GPUs with a global batch size of 48, requiring about 22 hours for EAGLE-3, 19 hours for DFlash, and 55 hours for DSpeaker.
Benchmark Evaluation: Fourteen few-shot (Brown et al., [8]) benchmarks decoded greedily, grouped by task form into mathematics, reasoning, multi-subject multiple choice, code and closed-book question answering, with CoT prompting (Wei et al., [54]) on MATH, GSM8K and BBH.
Speculative decoding: We adopt the same temperature setting () for both the target and draft models, following Bebop (Li et al., [34]GLM Team, 2026). This matters for RL training: rejection sampling with draft temperature $> 0$ keeps acceptance stable as target entropy rises, whereas greedy drafting degrades sharply at higher temperatures ({citet}Li et al., 2026b`). All drafters are served with SGLang on a single node of eight H800 GPUs and budget-matched at seven proposed positions plus one bonus token per verification step. We report decode-only throughput and mean accepted length on ten tasks, split by the target model’s predictive entropy into predictable and open-ended groups.
Conclusion
We presented ECHO, which turns multi-token prediction from an auxiliary task into part of the autoregressive objective. By reusing the backbone’s own final layer at every horizon, ECHO lets multi-horizon supervision shape the parameters that generate the next token, and Dynamic Depth Sampling makes deeper horizons affordable by training them less often. Unlike auxiliary MTP such as DeepSeek, ECHO improves with added depth, and ECHO-1.5 matches the downstream gain of ECHO-3 at half the MTP overhead.
At inference, the same shared layer makes the pretrained model its own drafter with close draft-target alignment: it drafts beyond its trained depth where auxiliary modules saturate, and it outperforms post-hoc drafters on open-ended, high-entropy generation. This advantage holds under a doubled token budget, a repeated pretraining run, and supervised finetuning.
Limitations
Due to compute constraints, pretraining was repeated once and finetuning was run with a single seed. How far the gains from added depth and from depth extrapolation extend, how DDS behaves at greater depths, and why deeper sampling pays off only after finetuning are left to future work.
Experiment Setup
Pretraining
Model Architecture: The backbone is a 10B-A1B Latent Mixture-of-Experts transformer ([7]), that is 10B parameters in total with 1B activated per token, stacked as 24 layers of hidden size 1,024 with 16 attention heads. Attention is multi-head latent ([37]), with 512-dimensional query and key–value latents, and we add gated attention ([44]), per-head query–key normalization and rotary positional embeddings ([50]). Every layer is a mixture of experts in a latent space: the expert block projects the hidden state down to 512 dimensions, runs 256 fine-grained routed experts ([15]) of intermediate size 1,024 there, and projects back, with the router reading the hidden state rather than the latent. A sigmoid gate sends each token to eight routed experts, balanced by a sequence-level auxiliary loss, plus one shared expert. The SwiGLU ([49]) feed-forward width is 5,120 and the vocabulary holds 201,216 tokens.
Training Settings: We train the model on 600B tokens with a context sequence length of 4,096 and a global batch size of 1,920. Training uses bfloat16 precision and the Muon optimizer ([28]), with AdamW retained for the embeddings, the biases, the output head and the expert router, a learning rate of , a warmup of 1,000 steps, weight decay of 0.1, and gradient clipping at 1.0. Following Liu et al. [37], we set the MTP loss scaling factor to 0.3. Pretraining runs on 24 nodes of eight H800 GPUs each, with eight-way expert parallelism, stage-one optimizer sharding, no tensor or pipeline parallelism, and full activation recomputation.
Computational Cost: Each pretraining run is conducted on 24 nodes with eight NVIDIA H800 80GB GPUs per node (192 GPUs in total), taking approximately three days to complete. Supervised finetuning uses the same hardware configuration and takes approximately five hours per run. For the post-hoc speculative decoding baselines, we train the draft models on three nodes (24 H800 80GB GPUs), using a global batch size of 48. EAGLE-3 (cold-start and warm-start) and DFlash use a per-GPU batch size of 2 without gradient accumulation, whereas DSpark uses a per-GPU batch size of 1 with two gradient accumulation steps due to GPU memory constraints. EAGLE-3 requires approximately 22 hours per run in measured wall-clock time. Based on the recorded compute time and an estimated 1.27× overhead for startup, checkpointing, and cluster variability, DSpark requires approximately 55 hours, while DFlash requires approximately 19 hours.
Methods Compared: All six configurations are trained under the recipe above, so the comparison isolates the MTP design. ECHO reuses the backbone’s last layer as the body of every MTP depth; DeepSeek’s MTP baselines follow Liu et al. [37] and keep an independent module per depth, sharing only the embedding and the output head. ECHO-1 and MTP-1 train one depth, ECHO-3 and MTP-3 train three at every step, ECHO-1.5 samples depths one to three with , and ECHO-2 extends dynamic depth sampling to depths one through seven with . The two cost-matched comparisons are ECHO-1 against MTP-1 at equal depth-one cost, and ECHO-1.5 against MTP-3 at half the MTP cost.
Dynamic Depth Sampling: The sampled depth is , where places probabilities 0.6, 0.3 and 0.1 on depths 1, 2 and 3, so . Each optimizer step draws one and trains depths 1 through , and every micro-batch on the step shares the same , which keeps per-step compute constant. Averaging the loss over the trained depths gives depth the weight , and these weights sum to one: ECHO reweights depths by how often it samples them, not by attaching a per-depth loss coefficient. is a deterministic function of the seed and the step index, so every rank derives it without communication and a resumed run continues the same sequence.
Supervised Finetuning: All configurations adopt an identical recipe: 5,000 steps over one pass of an in-house chat SFT corpus at a context length of 32,768, with eight-way context parallelism. Each keeps the MTP depths it was pretrained with, so the finetuned comparison isolates the same design difference as the pretrained one.
Benchmark Evaluation
Evaluation Setup: We evaluate the pretrained checkpoints on fourteen few-shot benchmarks, grouped by the form of the task following Liu et al. [37]: mathematics with MATH ([23]) and GSM8K ([14]); reasoning with BBH ([51]), BBEH ([29]), WinoGrande ([46]) and PIQA ([6]); multi-subject multiple choice with MMLU ([22]) and MMLU-Pro ([53]); code with HumanEval+ and MBPP+ ([39]) and LiveCodeBench ([27]); closed-book question answering with PopQA ([42]), PreciseWikiQA ([4]) and HotpotQA ([57]). We report unweighted averages within each group and across all fourteen benchmarks. Benchmarks such as GPQA [45], SimpleQA [55] and HMMT are left out for lack of resolution rather than for any defect in their scoring: every configuration scores near their floor, so the row moves more than it resolves.
Prompts and Decoding: All benchmarks are evaluated in the few-shot in-context format [8]. Chain-of-thought prompting [54] is used on MATH, GSM8K and BBH, with eight, eight and four demonstrations respectively. The remaining benchmarks are prompted to answer directly, with five demonstrations for the multiple-choice benchmarks, ten for closed-book question answering and two for BBEH. BBH is released under both a chain-of-thought and an answer-directly prompt over the same items for examining the chain-of-thought effect [51]; we report its chain-of-thought form in the main results. Decoding is greedy throughout. Generation is capped per benchmark: two tokens for MMLU and eleven for MMLU-Pro, which are answered with a single option label; 65 for the closed-book sets and 101 for PopQA; 401 for MATH, 512 for GSM8K, BBEH and HumanEval+, 601 for BBH, 768 for LiveCodeBench and 8192 for MBPP+. WinoGrande and PIQA involve no generation. Stop sequences are applied when the answer is extracted rather than during decoding, so the cap is what bounds each output.
Scoring and Averaging: Each benchmark’s score is an unweighted macro-average over its sub-tasks rather than a per-item micro-average. Four scoring rules are used, one per group of benchmarks. GSM8K, MATH, BBH, MMLU and MMLU-Pro are scored by rule from the extracted answer. HumanEval+, MBPP+ and LiveCodeBench are scored as pass@1 by execution [12]. WinoGrande and PIQA are scored by length-normalized log-likelihood over the candidate continuations rather than by generation, as in Liu et al. [37], with the normalization of Gao et al. [17]. PreciseWikiQA, HotpotQA, PopQA and BBEH are graded against the reference answer by an LLM judge. In Table 1 the three factual-knowledge sets are graded by gpt-oss-120b and BBEH by Claude Opus 4.8; in Table 3 all four are graded by Claude Opus 4.8. Of these, the three factual-knowledge sets report the F1 of overall accuracy and accuracy among attempted items, with abstentions in the denominator of the former only. Category averages and the overall figure are unweighted means over benchmarks.
Speculative Decoding under Serving
Serving Stack: All serving measurements use SGLang on one node of eight H800 GPUs, with tensor parallelism 8 and FlashAttention-3.
Drafters: All drafters are budget-matched at seven proposed positions plus one target bonus token, i.e. eight verification slots per request per step. ECHO reuses the target’s final layer and full output head. EAGLE-3 [36] uses one draft layer with a reduced 32,000-entry head scattered back to target width, in a cold-start and a warm-start variant; following Kimi Team (2026), the warm start initializes the draft from DeepSeek’s pretrained MTP weights, and is always normalized by the baseline of its own run. DSpark [13] and DFlash [11] are block-parallel with five draft layers, the former adding a low-rank Markov transition bias.
Serving Metric: We report decode-only throughput, decoded tokens over summed per-request decode latency, which excludes prefill; metrics that include prefill confound the comparison across tasks with differing prompt lengths. Speedup is recomputed against the baseline of the same run, and accepted length is decoded tokens per verification step.
Tasks: Serving results use ten tasks in two groups, split by predictive entropy as in Liu et al. [38]. Predictable, following the commonly adopted SpecForge benchmarks [33]: HumanEval, MBPP, MATH-500, GSM8K, LiveCodeBench. Open-ended: SWE-bench, Arena-Hard, MT-Bench, summarization, Alpaca. Question answering, retrieval-augmented generation, translation and AIME-25 are excluded, all having median outputs of twenty to thirty tokens, too few verification steps for a stable decode-only ratio. The entropy axis is the target’s own predictive entropy, measured separately.
Verification Budget: DSpark is evaluated with its confidence scheduler disabled, verifying the full draft block at every step, as in the diagnostic baseline of Cheng et al. [13]. The scheduler is a calibrated cost model rather than a threshold: it needs a latency curve re-profiled for each GPU and backend [13, 38], and a confidence head calibrated on the serving distribution, since the survival probabilities it multiplies must be absolute rather than merely well-ranked. That calibration does not transfer, because the mechanism assumes acceptance varies sharply across domains in the first place. With the head enabled but neither artifact available, per-step cost rose 5–14% and nothing was pruned.
Depth Extrapolation: The depth sweep of Figure 1b uses a separate harness. We report the accepted length obtained at serving depth , , for conditional acceptance rate at depth . The expression is exact rather than a fit: the draft loop reaches depth only after depth is accepted, so is the probability that the first drafted tokens are all accepted, and summing those probabilities gives their expected number. Runs use draft depth , greedy on both sides, 256 new tokens, and pool GSM8K, HumanEval and MATH-500 weighted by rounds.
Decayed Depth Supervision and Accepted Length
Let , , and be the target, draft, and data next-token distributions at depth , each conditioned on the current prefix. Let , , and , the drafter distributional loss at depth . We assume the drafter is trained on target-generated data, so and . All quantities are averaged over prefixes, which we assume follow the training data distribution at inference; each inequality below holds pointwise for every prefix and therefore after averaging, with Jensen’s inequality moving the average inside square roots where needed.
Derivation of Eq. (5). The full chain is
with .
(i) Union bound. By Eq. (3), , where each term is the probability that draft is not accepted. The probability of at least one rejection among depths is at most , a bound tight up to second-order terms in . Summing over , each is counted once for each of the values , and under lossless verification [32].
(ii) Pinsker’s inequality. Since , Pinsker’s inequality gives .
(iii) Cauchy–Schwarz. Writing ,
with equality if and only if . Since for any , schedules with different compute budgets can be compared directly. For MTP-, , with .
(iv) Decayed schedules. This is Theorem B.1(c).
Proposition B.1. Suppose and , and let . Then (a) is strictly decreasing in ; (b) ; (c) for , is nonincreasing in , from at to at .
Proof. (a) is strictly decreasing and is nonincreasing, both positive. (b) By (a), is not proportional to , so equality in Eq. (7) fails. (c) With , the function is a sum of log-sum-exps of affine functions, hence convex, and is minimized at by the equality condition of Eq. (7); a convex function is nonincreasing to the left of its minimizer.
The assumption reflects that acceptance decays with depth. Theorem B.1 concerns how tightly the training loss bounds lost length; that DDS attains comparable at lower cost is verified empirically.
Data not generated by the target. If , step (ii) uses the triangle inequality and Pinsker on both terms, which adds to every bound in Eq. (6). This term depends on neither the drafter nor the schedule, so steps (iii), (iv), and Theorem B.1 are unchanged.
Full Benchmark Results
Table 1 gives the per-benchmark scores behind the category gains of Figure 3a, for all six pretrained configurations over the fourteen benchmarks of the reported scope.
Provenance of the cells: nine of the fourteen rows are scored by us under the protocol of Section A. The other five are reported by an internal evaluation platform and are reproduced here as it returned them: PreciseWikiQA, HotpotQA and PopQA, graded by gpt-oss-120b over its full sets of 5,000, 7,405 and 14,256 items, together with MATH (4,739 items, scored by rule) and MBPP+ (378 items, scored by execution). That platform’s answer-extraction and judging rules are not published with this paper, so these five rows cannot be reproduced exactly from outside it. Our own runs of them exist but measure a different quantity — a different few-shot prefix, an answer-extraction rule that does not match on MATH, a weaker test bar on MBPP+, and a 600-item sample under a different judge on the three closed-book sets — so they are not offered as a second reading.
Stability across Pretraining Runs
Two of the configurations were pretrained a second time under the same recipe, and the comparison survives it. Over the fourteen benchmarks of the reported scope, ECHO MTP-1 leads DeepSeek’s MTP-1 by 0.52 points in the baseline run and by 0.78 in the repeat. Reasoning (+0.57 then +1.33) and code (+1.35 then +1.69) both grow, multi-subject multiple choice turns in ECHO’s favor (−0.52 then +1.30), closed-book QA stays within a fifth of a point of zero in both, and mathematics is the one category that moves the other way (+0.79 then −0.73). Two evaluations of the same weights differ by 0.27 points per benchmark, which puts the noise at 0.13 on a fourteen-benchmark gap and 0.30 on a category gap. Table 2 gives the three column groups, and Figure 4c draws the two classes with the spread these two runs measure.
Twice the training budget. The baseline run was then carried to step 156,000 on both sides, twice its token budget. Over the fourteen rows ECHO’s lead holds, +0.52 at step 78k against +0.50 at step 156k, a change of −0.02 against a spread of 0.14 across rows, with seven rows in its favor at each step, and every bucket is reproduced to within 0.2 points. Doubling the budget therefore leaves both the overall gap and the category ordering where they were. Repeating the run reproduces the overall gap to 0.26 points but not its split across categories, which moves by 0.3 to 1.8 points.
Next-Token Loss Does Not Separate the Two Objectives
Figure 5 separates the objective into its two terms. On the backbone’s next-token loss the two runs are indistinguishable: the gap averages 0.004 over the 78k steps and stays between 0.004 and 0.005 in every 10k-step window, with ECHO-1 marginally above MTP-1 throughout. The MTP loss carries a larger gap in the same direction, 0.011 at step 78k. On the same token budget the two objectives therefore arrive at the same next-token likelihood, and neither margin is one the method is meant to win.
Figure 5. Training loss of ECHO-1 and MTP-1 over the first 78k steps of the baseline run, separated into the backbone’s next-token loss (left) and the MTP loss (right). The logged objective is the weighted sum , and the left panel is recovered by subtracting the weighted MTP loss from it. Each curve is an exponential moving average over all 78,000 logged steps, with no subsampling; the vertical range is clipped to what the curves occupy after step 12k, so the first few thousand steps run above the frame.
This is the expected outcome rather than a negative result. ECHO’s multi-horizon objective adds no capacity to the next-token head; it changes what the shared representation is required to support. In predicting the next token the model is held responsible for several future positions at once, which constrains the hidden state towards features that stay predictive beyond one step. The effect of such a constraint is regularization, and regularization does not surface in the quantity it constrains: next-token cross-entropy is the term both objectives already optimize, so matching it is the premise of the comparison rather than its outcome. The two configurations separate downstream instead, in the benchmark gaps of Tables 1 and 2 and in the draft quality that § 4 converts into decode speedup.
The same holds along depth. Figure 6 repeats the split for all five configurations, ECHO-1, ECHO-1.5 and ECHO-3 against MTP-1 and MTP-3, over the steps all the runs share. Within a family the backbone term barely moves with depth: the three ECHO configurations end within 0.002 of each other, and MTP-3 sits 0.0017 below MTP-1. Across all five it spans 0.006, and the ordering is not ours to like — MTP-3 holds the lowest backbone loss of any configuration, 1.676 against ECHO-3’s 1.682, while ECHO-3 leads it by 0.97 points over the fourteen benchmarks of Table 1. What does move is the MTP term, and it moves the way the weighting predicts: the per-step loss is averaged over the depths trained at that step, so adding depths takes weight off depth 1, and the depth-1 term rises by 0.027 from ECHO-1.5 to ECHO-3 and by 0.048 from MTP-1 to MTP-3. At matched depth 3 the two designs divide the horizons: reusing the final layer fits the nearest one better (1.902 against 1.911) and the separate modules fit the far ones better (1.992 and 2.045 against 2.002 and 2.083 at depths 2 and 3). ECHO-1.5’s deeper terms are logged only on the steps that sample them, 39.5% and 9.8% of steps for depths 2 and 3, which is the that prescribes in Equation (4). At matched step those sampled depths are where the saving is paid for: Figure 7 puts ECHO-1.5’s depth-2 term 0.051 above ECHO-3’s at step 78k and its depth-3 term 0.132 above, with the curves running parallel rather than converging. That is the shape of the trade, and it stops there. Depth 1, which both train at every step, costs nothing at all (1.874 against 1.902, in ECHO-1.5’s favour); the fourteen-task average is within 0.02 (36.34 against 36.36, Table 1); and the accepted length holds beyond the trained depth (Figure 1b). None of these margins is where the comparison is settled, which is this section’s point: the terms of the training objective do not rank the two designs.
Figure 6. The same two terms across both families, over the steps all five runs share. ECHO-1.5’s event files start at step 60k on our side and the first thousand of those go to the moving average’s warm-up, which fixes the window at 61k–78k; the full-run view of the depth-1 pair is Figure 5. On the backbone term (left) the five lie within 0.006 and the ordering follows the family rather than the depth; the depth-1 term (right) is where added depth shows up.
Figure 7. The depth-2 term at matched pretraining step. ECHO-1.5 samples depth 2 on 39.5% of steps where ECHO-3 trains it on all of them, so the sampled series carries fewer points over this window, 6,721 against 17,000. Fewer is not biased: at depth 1, which both configurations train every step, the loss on the steps that also sample depth 2 differs from the loss on the steps that do not by +0.0002, against a standard error of 0.0004.
After Supervised Finetuning
The pretrained checkpoints were finetuned on the chat SFT corpus, and Table 3 reports the fourteen in-scope benchmarks for the two ECHO configurations against DeepSeek’s MTP baseline finetuned the same way. ECHO keeps its lead after finetuning, +1.61 and +2.10 points over the fourteen rows, and the lead is again larger on reasoning and code than on QA and multiple choice. Three caveats apply. The cells are decoded greedily, so repeating an evaluation does not move the rule- and execution-graded rows, and no error bars are drawn for them. The four judge-graded rows, BBH and the three closed-book sets, do carry grader variability. BBH is read in its chain-of-thought form: under the answer-directly prompt the finetuned output degenerates to a constant option label.
| Category | Benchmark | MTP-Instruct | ECHO-1.5-Instruct | ECHO-2-Instruct |
| Mathematics | GSM8K | 61.87 | 62.70 | 65.20 |
| MATH | 31.05 | 38.54 | 38.27 | |
| Reasoning | BBH | 54.41 | 54.24 | 58.06 |
| BBEH | 11.30 | 10.33 | 10.00 | |
| PIQA | 74.49 | 74.38 | 74.38 | |
| WinoGrande | 61.17 | 61.09 | 61.56 | |
| Multi-subject MC | MMLU | 49.17 | 52.46 | 55.53 |
| MMLU-Pro | 29.06 | 32.25 | 31.28 | |
| Code | HumanEval+ | 47.56 | 48.78 | 51.83 |
| MBPP+ | 44.97 | 51.59 | 46.30 | |
| LiveCodeBench | 8.75 | 11.25 | 9.75 | |
| Closed-book QA | PreciseWikiQA | 4.22 | 4.90 | 5.06 |
| HotpotQA | 19.04 | 16.82 | 19.16 | |
| PopQA | 10.69 | 10.94 | 10.77 | |
| Overall Average (14 tasks) | 36.27 | 37.88 | 38.37 |
caption: tbl:3: The fourteen in-scope benchmarks after one round of supervised finetuning. Every cell is scored under the protocol of Table 1.
:::
Decode Speedup under Serving
Figure 8 reports the cells of Figure 3b as decode speedup instead of accepted length; the ordering of the drafters is the one discussed in § 4.
Comparability of Reported Baselines in Inference
The evaluation scales differ systematically. EAGLE-3 reports results across a wide range of target sizes, from Vicuna-13B up to LLaMA-3.3-70B, whereas the parallel drafters cluster at the small end: Domino (Huang et al., [26]) evaluates only Qwen3-4B and Qwen3-8B, DSpark extends to Qwen3-14B, and the largest target any of them reports is Qwen3-Coder-30B-A3B, a sparse model with roughly 3B active parameters, so its per-token cost is closer to a 4B dense model than to a 30B one. No paper in this family evaluates a dense target above 14B (Table 4).
| Model | Qwen3-4B | LLaMA-3.1-8B | Qwen3-8B | Gemma4-12B | Vicuna-13B | Qwen3-14B | Qwen3-Coder-30B-A3B | LLaMA-3.3-70B |
| EAGLE-3 | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ |
| DFlash | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
| Domino | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| DSpark | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
Table 4. Target models evaluated by each method, ordered by model size.
Beyond scale, the EAGLE-3 checkpoints these papers evaluate were never fairly trained (Table 5). DFlash and Domino used public checkpoints released by AngelSlim with the different training datasets and fewer samples. Drafter capacity is uncontrolled as well: at every target, the block-parallel drafter carries – the parameters of the EAGLE-3 head.
| Method | Model | Training datasets | Samples | Draft params |
| DFlash | Qwen3-4B / 8B | Nemotron V2, CodeAlpaca, ShareGPT, UltraChat, LongAlign | 800K | 459M |
| EAGLE-3 | Qwen3-4B / 8B | ShareGPT, UltraChat | 530K | 218M |
| Domino | Qwen3-8B | Open-PerfectBlend | 1.42M | 1049M |
| EAGLE-3 | Qwen3-8B | ShareGPT, UltraChat | 530K | 400M |
| DSpark | Qwen3-4B / 8B / 14B | Open-PerfectBlend | 1.3M | 459M / 1049M / 1573M |
| EAGLE-3 | Qwen3-4B / 8B / 14B | Open-PerfectBlend | 1.3M | 218M / 400M / 609M |
Table 5. Training data and drafter capacity in the DFlash-family papers’ comparisons against EAGLE-3. Drafter parameter counts are computed from released configs.
Although DSpark controls the data axis: it retrains EAGLE-3 and DSpark within a single framework on identical training dataset, however, its evaluation restricts EAGLE-3 to chain drafting instead of the original tree drafting. The released code proposes one token per step and returns a single trajectory with no tree construction, top- expansion, or candidate reranking. The baseline is therefore matched in data but evaluated in a degraded configuration, and its acceptance length understates what the method achieves as published.
Artifact Licensing
Table 6 summarizes the licenses and usage terms of the training and evaluation datasets used in our experiments. The pretraining corpus and in-house SFT dataset are proprietary and subject to restricted redistribution. All fourteen evaluation benchmarks are used solely for research evaluation and are not redistributed. The use of these artifacts follows their applicable licenses and access conditions.
| Type | Artifact | License / Terms of Use |
| Training Data | ||
| Pretraining | 600B-token corpus | Proprietary (redistribution restricted) |
| SFT | In-house chat corpus | Proprietary (redistribution restricted) |
| Model Weights and Checkpoints | ||
| Weights | DeepSeek-V3 pretrained weights | DeepSeek Model License |
| Model | gpt-oss-120b | Apache-2.0 |
| Evaluation Datasets | ||
| Mathematics | MATH | MIT (repository) |
| Mathematics | GSM8K | MIT |
| Reasoning | BBH | MIT |
| Reasoning | BBEH | CC BY 4.0 (data) |
| Reasoning | WinoGrande | Apache-2.0 |
| Reasoning | PIQA | Not specified |
| Multi-subject | MMLU | MIT |
| Multi-subject | MMLU-Pro | MIT |
| Code | HumanEval+ | Apache-2.0 (EvalPlus) |
| Code | MBPP+ | Apache-2.0 (EvalPlus) |
| Code | LiveCodeBench | MIT (code); data terms unclear |
| QA | PopQA | MIT (repository) |
| QA | PreciseWikiQA | CC BY-NC 4.0 (HalluLens) |
| QA | HotpotQA | CC BY-SA 4.0 |
| Additional Serving Benchmarks | ||
| Code | HumanEval | MIT |
| Code | MBPP | CC BY 4.0 |
| Mathematics | MATH-500 | MIT |
| Software | SWE-bench | MIT (repository) |
| Chat | Arena-Hard | Apache-2.0 (repository) |
| Chat | MT-Bench | Not confirmed (prompt dataset) |
| Summarization | Summarization prompts | Not specified |
| Instruction | Stanford Alpaca | CC BY-NC 4.0 |
| Software and Implementations | ||
| Serving | SGLang v0.4 | Apache-2.0 |
| Attention | FlashAttention-3 | BSD-3-Clause |
| Drafter | EAGLE-3 | Apache-2.0 |
| Drafter | DFlash | MIT |
| Drafter | DSpark | MIT |
| Evaluation | EvalPlus | Apache-2.0 (repository) |
Table 6. Licenses and terms of use for the scientific artifacts used in our experiments, including training data, pretrained models, evaluation benchmarks, and third-party software. Repository-level licenses are distinguished from dataset-specific and model-weight licenses where applicable.
AI Use Disclosure
During the preparation of this manuscript, the authors used ChatGPT (OpenAI) to assist with language editing, grammatical corrections, and improving the clarity of author-written text. The AI tool was not used to generate research findings or experimental results. All AI-assisted modifications were reviewed and verified by the authors, who take full responsibility for the content of this manuscript.
References
- [1]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.arxiv.org/abs/2303.08774
- [2]Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, and Se-Young Yun. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2507.10524
- [3]Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.arxiv.org/abs/2309.16609
- [4]Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. HalluLens: LLM hallucination benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 24128–24156, 2025.
- [5]Fengxiang Bie, Yuqing Jian, Yifan Yu, Zhongzhu Zhou, Zelei Shao, Ben Athiwaratkun, Shuaiwen Leon Song, Chenfeng Xu, Xiaoxia Wu, and Tianyi Zhang. Opsprey: Target-agnostic pre-training makes stronger drafters in speculative decoding. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, 2026. arXiv:2609.09338.
- [6]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.arxiv.org/abs/1911.11641
- [7]Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khattar, Adi Renduchintala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, et al. NVIDIA Nemotron 3: Efficient and open intelligence. arXiv preprint arXiv:2512.20856, 2025.arxiv.org/abs/2512.20856
- [8]Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2005.14165
- [9]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. MEDUSA: Simple LLM inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning, 2024.arxiv.org/abs/2401.10774
- [10]Aaron Chatterji, Thomas Cunningham, David J. Deming, Zoë Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. How people use ChatGPT. Working Paper 34255, National Bureau of Economic Research, 2025.
- [11]Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block diffusion for flash speculative decoding, 2026. URL https://arxiv.org/abs/2602.06036.
- [12]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
- [13]Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, and Wenfeng Liang. DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation, 2026. URL https://arxiv.org/abs/2607.05147.
- [14]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
- [15]Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1280–1297, 2024.arxiv.org/abs/2401.06066
- [16]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019.arxiv.org/abs/1807.03819
- [17]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPoFi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Hanani Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation. Zenodo, version v0.4.3, 07 2024. URL https://zenodo.org/records/12608602.
- [18]Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2502.05171
- [20]Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In International Conference on Machine Learning, 2024.arxiv.org/abs/2404.19737
- [21]Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. In Conference on Language Modeling, 2025.arxiv.org/abs/2412.06769
- [22]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a.
- [23]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b.
- [24]Fenglu Hong, Ravi Raju, Jonathan Lingjie Li, Bo Li, Urmish Thakker, Avinash Ravichandran, Swayambhoo Jain, and Changran Hu. Training domain draft models for speculative decoding: Best practices and insights. arXiv preprint arXiv:2503.07807, 2025.arxiv.org/abs/2503.07807
- [25]Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), pp. 646–661, 2016.arxiv.org/abs/1603.09382
- [26]Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, Hanlin Xu, and Linfeng Zhang. Domino: Decoupling causal modeling from autoregressive drafting in speculative decoding, 2026. URL https://arxiv.org/abs/2605.29707.
- [27]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Jon Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, 2025.arxiv.org/abs/2403.07974
- [28]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/.
- [29]Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K Jain, Virginia Aglietti, Disha Jindal, Yuanzhu Peter Chen, et al. BIG-Bench extra hard. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 26473–26501, 2025.
- [30]Kimi Team. Kimi K3: Open frontier intelligence, 2026. URL https://arxiv.org/abs/2607.24653.
- [31]Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.arxiv.org/abs/1909.11942
- [32]Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 19274–19286, 2023.arxiv.org/abs/2211.17192
- [33]Shenggui Li, Chao Wang, Yikai Zhu, Yubo Wang, Fan Yin, Shuai Shi, Yefei Chen, Xiaomin Dong, Qiaoling Chen, Jin Pan, Ji Li, Laixin Xie, Yineng Zhang, Lei Yu, Yonggang Wen, Ivor Tsang, and Tianwei Zhang. SpecForge: A flexible and efficient open-source training framework for speculative decoding, 2026a. URL https://arxiv.org/abs/2603.18567.
- [34]Yuchen Li, Huiqiang Jiang, Yang Xu, Jianxin Yang, Yi Zhang, Yizhong Cao, Yuhao Shen, Fan Zhou, Rui Men, Jianwei Zhang, An Yang, Bowen Yu, Bo Zheng, Fei Huang, Junyang Lin, Davihe ng Liu, and Jingren Zhou. Breaking entropy bounds: Accelerating rl training via mtp with rejection sampling, 2026b. URL https://arxiv.org/abs/2606.12370.
- [35]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, 2024.arxiv.org/abs/2401.15077
- [36]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. In Advances in Neural Information Processing Systems, 2025.
- [37]Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bohao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024a.arxiv.org/abs/2412.19437
- [38]Hong Liu, Rui Cen, Junhan Shi, Guangshuo Qin, Jiebin Zhang, Tianyu Liu, Runzhi Fan, Guoliang Zhao, Ruobing Xie, Kai Zhang, Song Liu, Guanghua Yu, and Jianchen Zhu. AngelSpec: Towards real-world high performance inference with speculative decoding, 2026. URL https://arxiv.org/abs/2607.25852.
- [39]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. Advances in neural information processing systems, 36:21558–21572, 2023.
- [40]Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 2024b.arxiv.org/abs/2310.07177
- [41]LLM-Core Xiaomi. MiMo: Unlocking the reasoning potential of language model – from pretraining to posttraining. arXiv preprint arXiv:2505.07608, 2025.
- [42]Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 9802–9822, 2023.arxiv.org/abs/2212.10511
- [43]Somesh Mehra, Javier Alonso Garcia, and Lukas Mauch. On multi-token prediction for efficient LLM inference. In ICLR 2025 Workshop on Sparsity in LLMs (SLLM), 2025. arXiv:2502.09419.arxiv.org/abs/2502.09419
- [44]Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, et al. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. Advances in Neural Information Processing Systems, 38:100092–100118, 2025.arxiv.org/abs/2505.06708
- [45]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level Google-Proof Q&A benchmark. In First Conference on Language Modeling, 2024.arxiv.org/abs/2311.12022
- [46]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021.arxiv.org/abs/1907.10641
- [47]Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, and Alexander Golubev. KV losses: Direct acceptance rate optimization for speculative decoding. In Proceedings of the 43rd International Conference on Machine Learning, 2026.
- [48]Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. In International Conference on Learning Representations, 2025.arxiv.org/abs/2502.17416
- [49]Noam Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.arxiv.org/abs/2002.05202
- [50]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
- [51]Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051, 2023.arxiv.org/abs/2210.09261
- [52]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.arxiv.org/abs/2307.09288
- [53]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024.arxiv.org/abs/2406.01574
- [54]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2201.11903
- [55]Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024.arxiv.org/abs/2411.04368
- [56]Tianyu Wu, Yu Yao, Zhenting Qi, Han Zheng, Zhuohan Wang, Haoran Ma, Lawrence Liao, Himabindu Lakkaraju, Ju Li, and Yilun Du. D-PACE: Dynamic position-aware cross-entropy for parallel speculative drafting. arXiv preprint arXiv:2605.18810, 2026.arxiv.org/abs/2605.18810
- [57]Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2369–2380, 2018.arxiv.org/abs/1809.09600
- [58]Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. GLM-5: From vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026.arxiv.org/abs/2602.15763
- [59]Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. DistillSpec: Improving speculative decoding via knowledge distillation. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.08461
- [60]Yuxuan Zhou, Fei Huang, Heng Li, Fengyi Wu, Tianyu Wang, Jianwei Zhang, Junyang Lin, and Zhi-Qi Cheng. Overcoming joint intractability with lossless hierarchical speculative decoding, 2026. URL https://arxiv.org/abs/2601.05724.
- [61]Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, et al. Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741, 2025.arxiv.org/abs/2510.25741





