DStill: Trajectory Consistency Distillation for Accelerated One-Step Drafting in Block Diffusion Speculative Decoding

Introduction

Speculative decoding accelerates Large Language Model (LLM) inference by using a small, inexpensive draft model to propose future tokens and verifying those proposals against the target model in a single parallel forward pass (Leviathan et al., [24]; Chen et al., [5]). Under the standard rejection-sampling verifier, this procedure is lossless as it preserves the output distribution of autoregressive sampling from the target model. An effective drafter must achieve a high acceptance length $\tau$, the expected number of draft tokens accepted per verification step, while keeping drafting latency low.

Target-conditioned¹ block-diffusion drafters, such as DFlash (Chen et al., [6]), predict an entire block in parallel. Each block starts in the root state: the latest verified token is visible as an anchor, and all future-token positions are masked. A one-pass drafter predicts each masked position without conditioning on the other masked positions. Independent predictions can combine tokens from incompatible continuations (Figure 2a) (Lee et al., [23]). Refining the block over multiple denoising passes progressively commits high-confidence tokens, fixing their values for subsequent passes. The remaining positions can then condition on these commitments and improve their agreement with the target. However, the additional drafter passes increase latency, and the benefit of longer accepted prefixes may not offset this cost.

We introduce DSTILL, a two-stage trajectory consistency distillation framework that transfers the benefits of multi-step refinement to one-step speculative drafting. In the first stage, we train a target-conditioned drafter to complete partially masked blocks in a single forward pass. After training, we use this drafter as a teacher to collect multi-step denoising trajectories, recording the intermediate partially unmasked blocks and final completions which are fully unmasked which we refer to as endpoints. In the second stage, a student initialized from the teacher learns to predict each trajectory’s final completion directly from its recorded states. At inference time, the student drafts a block in one forward pass, followed by standard target-model verification (Figure 1).

During distillation, we sample a state from the teacher’s recorded trajectory and use it as input to the student. We resample this state across epochs while keeping the trajectory’s final completion fixed as the training target. This exposes the student to different partial commitments for the same block, with varying amounts of context for predicting the remaining tokens. To supervise one-step drafting, we use a curriculum that gradually increases root-bias, in which we increase the probability of sampling the root state through a schedule as we progress through training.

Our procedure adapts the theoretical foundation of Consistency Models ([40]) to discrete diffusion by learning to map any intermediate point on a masking trajectory directly to the same endpoint. Just as these continuous models enforce a self-consistency property where outputs remain invariant for arbitrary points along a given probability flow trajectory, we train the student to map partially revealed intermediate blocks to a constant, fully refined target sequence. By internalizing this discrete refinement trajectory into its hidden states, the student learns to sharpen the blurry averaging of independent marginal probabilities (Figure 2b,Figure 2c), mirroring how continuous consistency models learn to convert pure noise into coherent data samples using a single step. We make three main contributions:

Transferring Multi-Step Refinement to One-Step Drafting. We introduce DSTILL, a framework that distills a drafter’s own multi-step refinement process into a one-step speculative drafter. By training the student to predict the teacher’s refined completion from various states along its trajectory, we successfully transfer multi-step capabilities to a single-step process. This approach recovers 35.8% and 28.6% of the multi-step teacher’s acceptance length improvements on GPT-OSS-120B and on Nemotron-3.5-30B-A3B respectively.

Zero-Overhead Efficiency Gains Across Models and Domains. We evaluate DSTILL across three target-model families on benchmarks spanning reasoning, math, code, and chat. Compared to DFlash, our method achieves up to 28% higher acceptance length and up to 25% wall-clock speedup. Crucially, after offline distillation, these gains are realized using the exact same drafter architecture, avoiding test-time latency penalties associated with additional correction heads or tree-search decoding.

Generalization to Block Diffusion Speculators. We also apply trajectory distillation to DSpark’s block diffusion backbone ([9]), showing a 4.5% improvement in acceptance length (Appendix § 13) demonstrating its general applicability and motivating extensions to other block diffusion speculators.

Related Work

Speculative Decoding. Speculative decoding ([24, 5]) accelerates autoregressive LLM generation by having a cheap draft model propose several future tokens, which the target model then verifies in a single forward pass; a rejection-sampling scheme guarantees that the resulting output distribution exactly matches that of the target model. The approach builds on earlier parallel-decoding ideas such as blockwise parallel decoding ([41]). Follow-up work extends verification from a single chain to a candidate tree ([32, 8]), while draft-free variants such as Jacobi-style lookahead decoding ([12]) remove the draft model entirely at the cost of a weaker acceptance guarantee.

Feature-based and Multi-head Drafting. EA-GLE ([27]) and its successors ([26, 28]) draft tokens autoregressively using target-model features and embeddings of preceding tokens. Medusa ([4]) instead attaches multiple prediction heads to the target backbone, predicting future tokens independently in parallel. Hydra ([1]) extends this design by conditioning each draft head on preceding draft tokens, introducing sequential dependencies within the proposed continuation while retaining lightweight prediction heads.

Parallel and Diffusion Drafters. DFlash ([6]) replaces sequential autoregressive drafting with a lightweight block-diffusion model that predicts an entire block in one pass while conditioning on hidden features from the target model. DFlare ([43]) scales this design by equipping each draft layer with a learned mixture over a broader set of target-layer features, increasing draft capacity. Other work restores intra-block dependencies through lightweight auxiliary modules: Domino ([17]) applies a causal head to the preliminary distributions produced by a parallel backbone, and DSpark ([9]) couples a parallel backbone with a small sequential markov head while adapting the verification length based on confidence estimates. DDTree ([38]) keeps predictions parallel but uses their per-position distributions to construct a draft tree for joint verification by the target model.

Masked and discrete diffusion language models. Absorbing-state (masked) diffusion formulates generation as iterative unmasking with weights derived from an evidence lower bound (ELBO) ([2, 31, 39]), and large masked-diffusion language models such as LLaDA ([34]) and Dream ([42]) trade sampling steps for generation quality via confidence-based remasking.

Consistency and few-step distillation. Consistency models learn a map from any point on a diffusion trajectory to its endpoint, collapsing many-step sampling into one or a few forward passes ([40]), later generalized to arbitrary source-target pairs along the trajectory ([19]). In the discrete/masked setting, self-distillation through time (SDTT) transfers multi-step unmasking distributions to fewer steps ([11, 43]) distill a many-step masked-diffusion model through generative-trajectory matching, and Kim et al. [20] combine consistency-guided distillation with cache-friendly block causality to accelerate masked-diffusion inference. Consistency LLMs fine-tune autoregressive models to predict fixed points from states along Jacobi-decoding trajectories, with an auxiliary ground-truth loss ([21]). These methods accelerate generation by the language model being distilled.

DStill

Figure Figure 1 describes DStill. In Stage I, we train a target-conditioned drafter to complete partially revealed reference blocks from sequences generated by the autoregressive target model. The trained drafter then serves as the teacher. We freeze its parameters and record its multi-step refinement trajectories offline. In Stage II, a student initialized from the teacher learns to predict the teacher’s final completions from states along these trajectories. At inference, the student drafts a block in a single forward pass, and the target LLM verifies the proposed tokens.

Setup and notation

Let $a$ be the latest verified token, and let $H_c$ be the target model’s cached hidden states for the tokens before $a$. The teacher and student operate on blocks of $B$ positions, with the fixed anchor $a$ at position 0 and future tokens at positions $1,\ldots,B-1$. In the root state, $x_1 = (a, [\mathrm{MASK}], \ldots, [\mathrm{MASK}])$, all future-token positions are masked. Following DFlash ([6]), the drafter $D_\theta$ attends bidirectionally within each block $x$ and to target context $H_c$ strictly before the anchor. It predicts a vocabulary distribution $p_{\theta,i}(\cdot\mid x, H_c)$ at each masked position $i$. Independently sampling the future tokens from the root predictions gives the proposal distribution

Greedy drafting instead takes each position’s top-1 prediction. Figure 2a illustrates how factorized sampling can combine incompatible continuations.

Multi-step refinement allows later predictions to condition on tokens selected earlier. The drafter completes a block over $K$ passes, selecting the highest-confidence masked positions and committing their predicted tokens at each pass. We measure confidence by the largest token probability at each position. Once committed, a token remains fixed for the remaining passes.

We denote the inputs to these passes by $x_1,\ldots,x_K$, where $x_1$ is the root state. The completed block after the final pass is denoted by $x_0$ and called the endpoint. Appendix § 7 gives the reveal schedule and verification procedure.

Stage I: Partial-mask teacher training

In Stage I, we train the drafter to complete partially revealed blocks in a single forward pass. This prepares the drafter for multi-step refinement, where later passes use previously committed tokens to predict the remaining tokens.

For a reference block $z=(a,z_1,\ldots,z_{B-1})$, we sample a reveal probability $r$ uniformly from a discrete set. We choose these probabilities so that training blocks contain, on average, as many visible tokens as the teacher sees at each refinement pass.

Given $r$, we reveal each future-token position independently with probability $r$. Revealed positions contain their reference tokens, and the remaining positions contain [MASK], producing the input block $x^r$. The expected fraction of masked future positions, or corruption level, is therefore $1-r$. The anchor remains visible, and $r=0$ gives the root state. We resample the reveal probability and positions across epochs.

Given this partially revealed block, we train the drafter to recover the reference tokens at the remaining masked positions. Let $M$ denote these positions. For $|M|>0$, the unweighted training objective is

The reveal probabilities and optional loss weights are specified in Appendix § 8.§ 1. After this training, we use the drafter’s own predictions to collect the refinement trajectories needed for student distillation.

Trajectory collection

We freeze the trained drafter as the teacher and run its confidence-guided refinement offline, once per training block. Each rollout starts from the root state and progressively reveals the teacher’s predictions. These predictions may differ from the reference tokens used in Stage I. We use greedy token selection for all models: at each selected position, the teacher commits its highest-probability token (Appendix § 11).

We record each trajectory so that its states and predictions can be reused during student training. Each record contains the final block $x_0$ and the pass at which each token was committed. At each pass, we also store the teacher’s top-$k$ token IDs and probabilities at the masked positions.

Because committed tokens are never revised, the endpoint and commit passes are sufficient to reconstruct every input state. To reconstruct a state, we reveal endpoint tokens committed before that pass and mask the remaining future positions. The anchor remains visible throughout.

The reconstructed states serve as inputs to the student. We supervise the remaining masked positions using the teacher’s committed tokens and the probability distributions recorded when those tokens were committed. Record formats, reconstruction, and the collection algorithm are presented in Appendix § 8.§ 2.

Stage II: One-step trajectory consistency distillation

The student $D_\phi$ shares the teacher’s architecture and is initialized from its weights. During distillation, we sample a state from the teacher’s recorded trajectory and use it as input to the student. The student is trained to predict the trajectory’s final completion at the positions that remain masked.

Different input states from the same trajectory share the same final completion. For each position that remains masked, the target token and teacher distribution stay fixed across these states. We refer to this shared supervision as trajectory consistency. Figure 2b,Figure 2c illustrates how this supervision can increase the probabilities of tokens from the teacher’s selected completion. Incompatible combinations can then become less likely while the student’s output distribution remains factorized. Each supervised position receives two training signals: the token committed by the teacher and the teacher’s probability distribution from the pass that committed it. Cross-entropy (CE) trains the student to predict the committed token, while knowledge distillation (KD) transfers the teacher’s uncertainty.

Root-biased state sampling. To expose the student to different amounts of visible context, we train on both root and intermediate states. Intermediate states provide committed tokens that help predict the remaining positions, while root states match the input used for one-step drafting. With probability $p_{\mathrm{root}}$, we sample the root state $x_1$; otherwise, we uniformly sample a recorded intermediate state. Blocks without intermediate states use root.

As training progresses, we increase $p_{\mathrm{root}}$ from 0.5 to 0.85 using a cosine schedule, establishing a curriculum that places greater emphasis on the root state later in training (Appendix § 8.§ 3). The input state is resampled across epochs, while trajectory’s final completion and teacher distributions remain fixed.

Endpoint supervision. After selecting an input state, we supervise the student’s predictions at the positions that remain masked. The training loss mask restricts supervision to response-token positions, and the anchor is excluded.

To compute the KD term, we use the teacher’s retained top-$k$ tokens and group the remaining probability mass into a single category. Let $Q_{\mathrm{end},i}$ denote this distribution for position $i$. Let $P_{\phi,i}$ denote the student’s distribution given $(x_t, H_c)$, using the same teacher-selected token IDs and the same aggregate category.

For each supervised position $i$, we combine CE and KD with weight $\alpha\in[0,1]$ on the KD term:

We use $\alpha= 0.2$ and $k = 16$, and average over the positions included in the loss.

We optimize this objective while keeping the teacher fixed. Both drafters reuse the target’s token embeddings and language-model (LM) head, which projects hidden states to vocabulary logits. These shared components also remain frozen, so Stage II updates only the remaining student parameters. Appendix § 8.§ 4 gives the exact probability construction, optional trajectory weights, batch aggregation, and training algorithm.

Inference

At inference, the student receives the target’s cached hidden states $H_c$ and the root state $x_1$. A single forward pass produces the proposed block $\hat{x}_0=(a,\hat{y}_1,\ldots,\hat{y}_{B-1})$.

For independently sampled one-pass proposals, the student’s per-position distributions provide the proposal conditionals required by the standard rejection-sampling verifier. This verifier preserves the target model’s sampling distribution. Greedy verification accepts tokens that match the target’s top-1 predictions.

After a rejection, the target supplies a correction token. If all draft tokens are accepted, it supplies a bonus token. This target-generated token becomes the next anchor, starting another drafting and verification step. At fixed one-pass drafting cost, longer accepted prefixes can improve end-to-end speedup; § 4 evaluates resulting wall-clock trade-off.

Experiments

Table 1 presents the main comparison across target models and benchmarks, reporting generation speedup and acceptance length. We also measure how much of the teacher’s eight-pass refinement gain transfers to one-pass drafting and examine the teacher objective, student supervision, and state-sampling curriculum.

Model	Method	Params (M)	GSM8K $S$	GSM8K $\tau$	MATH-500 $S$	MATH-500 $\tau$	HumanEval $S$	HumanEval $\tau$	MBPP $S$	MBPP $\tau$	MT-Bench $S$	MT-Bench $\tau$	SWE-Bench $S$	SWE-Bench $\tau$	AVERAGE $S$	AVERAGE $\tau$
GPT-OSS-120B	DFlash	785	4.81×	6.3	4.01×	5.17	3.53×	4.58	3.57×	4.63	2.42×	3.09	2.33×	2.96	3.45×	4.45
	DStill (Ours)	785	5.17×	6.79	4.27×	5.54	3.7×	4.83	3.8×	4.97	2.6×	3.34	2.53×	3.25	3.68×	4.79
Gemma-4-31B	DFlash	1536	5.85×	6.78	5.9×	6.84	5.2×	6.03	4.3×	5.00	3.16×	3.67	2.74×	3.17	4.53×	5.25
	DStill (Ours)	1536	6.45×	7.48	6.28×	7.28	5.59×	6.49	4.76×	5.52	3.47×	4.03	2.92×	3.38	4.91×	5.70
Gemma-4-26B-A4B	DFlash	430	2.86×	4.57	2.98×	4.65	2.58×	4.09	2.23×	3.55	1.78×	2.81	1.62×	2.54	2.34×	3.70
	DStill (Ours)	430	3.42×	5.57	3.53×	5.59	3.11×	5.02	2.79×	4.53	2.08×	3.38	1.75×	2.81	2.78×	4.48
Nemotron-3.5-30B-A3B	DFlash	481	2.62×	4.81	2.66×	4.86	2.31×	4.27	2.08×	3.79	1.77×	3.18	1.62×	2.89	2.18×	3.97
	DStill (Ours)	481	2.98×	5.62	3.02×	5.64	2.57×	4.88	2.33×	4.36	1.94×	3.6	1.66×	3.04	2.42×	4.52

Experimental Setup

Models and benchmarks. We evaluate GPT-OSS-120B ([37]), Gemma-4-31B and Gemma-4-26B-A4B ([13]), and Nemotron-3.5-30B-A3B ([35]), covering dense, transformer mixture-of-experts (MoE), and hybrid Mamba–attention MoE architectures. We run the benchmarks on GSM8K ([10]), MATH-500 ([16] (Lightman et al. 2024)), HumanEval ([7]), MBPP ([3]), SWE-bench ([18]), and MT-Bench ([44]). We evaluate 128 prompts each from GSM8K, MATH-500, and MBPP; 164 each from

HumanEval and SWE-bench; and 80 from MT-Bench. Further evaluation details are provided in Appendix § 11.

Training. Each target regenerates assistant responses in OpenPerfectBlend ((mlabonne, n.d.); its hidden-state cache is shared across both training stages and trajectory collection. Our DStill drafters use block size $B = 16$, comprising one anchor and 15 proposed tokens. We collect teacher trajectories using $K = 8$ refinement passes with greedy token selection. Unless otherwise stated, student distillation uses $\alpha= 0.2$, top-$k = 16$, and a cosine root-probability schedule from 0.5 to 0.85. The student is initialized from the teacher, and the target embeddings and LM head remain frozen throughout training. Drafter architectures are listed in Appendix Table 10, with further training details in Appendix § 11.

Comparisons. Our primary baseline is a draft model trained with the DFlash objective [6]. Within each target, this baseline and DSTILL share the drafter architecture, training corpus, block size, and one-pass drafting budget. For GPT-OSS-120B and Nemotron-3.5-30B-A3B, we evaluate the anchor-only baseline and partial-mask teacher with one and eight passes to measure the effect of iterative refinement and its transfer to the student (Table 4). We also apply trajectory distillation to a DSpark drafter (Appendix Table 11). Autoregressive target decoding provides the speedup reference.

Target Model	Drafter	Denoising Steps	GSM8K $\tau$	MATH-500 $\tau$	HumanEval $\tau$	MBPP $\tau$	MT-Bench $\tau$	SWE-Bench $\tau$	Average $\tau$	Recovery (%)
GPT-OSS-120B	DFlash	1	6.30	5.17	4.58	4.63	3.09	2.96	4.45	–
GPT-OSS-120B	DFlash	8	4.97	4.06	3.75	3.76	2.88	2.79	3.70	–
GPT-OSS-120B	Teacher	1	6.44	5.10	4.63	4.65	3.19	2.97	4.50	–
GPT-OSS-120B	Teacher	8	7.58	6.14	5.48	5.62	3.60	3.41	5.31	–
GPT-OSS-120B	DSTILL	1	6.79	5.54	4.83	4.97	3.34	3.25	4.79	35.80
Nemotron-3.5-30B-A3B	DFlash	1	5.84	4.84	5.98	4.44	2.90	2.92	4.49	–
Nemotron-3.5-30B-A3B	DFlash	8	4.94	4.19	4.70	3.90	2.84	2.78	3.89	–
Nemotron-3.5-30B-A3B	Teacher	1	6.77	5.43	6.85	5.02	3.18	3.31	5.10	–
Nemotron-3.5-30B-A3B	Teacher	8	7.80	6.48	8.57	6.17	3.51	3.52	6.01	–
Nemotron-3.5-30B-A3B	DSTILL	1	6.94	5.65	7.46	5.40	3.32	3.35	5.36	28.57

Inference setup. Table 1 reports results at request concurrency $C = 1$ and temperature 0, with reasoning enabled and reasoning effort set to medium for GPT-OSS-120B. For GPT-OSS-120B and Nemotron-3.5-30B-A3B, we additionally evaluate $C \in\{1, 8, 16, 32\}$ at temperatures 0 and 1; Nemotron is evaluated with reasoning both enabled and disabled (Table 2 and Table 3). Prompts, generation settings, and hardware are matched across methods within each comparison.

Dataset	Method	Temperature $= 0$ Avg $\tau$	Temperature $= 0$ Speedup $S$ $(\times)$ $C=1$	Temperature $= 0$ Speedup $S$ $(\times)$ $C=8$	Temperature $= 0$ Speedup $S$ $(\times)$ $C=16$	Temperature $= 0$ Speedup $S$ $(\times)$ $C=32$	Temperature $= 1$ Avg $\tau$	Temperature $= 1$ Speedup $S$ $(\times)$ $C=1$	Temperature $= 1$ Speedup $S$ $(\times)$ $C=8$	Temperature $= 1$ Speedup $S$ $(\times)$ $C=16$	Temperature $= 1$ Speedup $S$ $(\times)$ $C=32$
GSM8K	DFlash	6.3	4.81	3.01	3.04	3.59	5.54	4.17	2.05	2.55	3.3
	DStill	6.86	5.17	3.26	3.29	3.92	5.93	4.54	2.21	2.75	3.42
MATH-500	DFlash	5.16	4.01	2.47	2.53	3.3	4.37	3.29	2.15	2.09	2.49
	DStill	5.51	4.27	2.67	2.62	3.06	4.78	3.69	2.34	2.16	2.71
HumanEval	DFlash	4.54	3.53	2.15	2.19	2.61	4.11	3.12	1.93	1.9	2.39
	DStill	4.88	3.7	2.31	2.38	2.69	4.4	3.33	2.07	2.05	2.46
MBPP	DFlash	4.7	3.57	2.26	2.29	2.65	4.25	3.25	2	1.97	2.4
	DStill	4.98	3.8	2.39	2.42	2.69	4.58	3.48	2.15	2.12	2.56
MT-Bench	DFlash	3.1	2.42	1.53	1.52	1.86	3.05	2.25	1.44	1.37	1.76
	DStill	3.31	2.6	1.64	1.63	1.98	3.05	2.39	1.47	1.42	1.77
SWE-Bench	DFlash	2.9	2.33	1.43	1.41	1.62	2.66	2.06	1.29	1.24	1.5
	DStill	3.19	2.53	1.52	1.55	1.77	2.84	2.2	1.37	1.33	1.55

Dataset	Method	Temperature = 0 Avg $\tau$	Temperature = 0 Speedup $S$ ($\times$), $C=1$	Temperature = 0 Speedup $S$ ($\times$), $C=8$	Temperature = 0 Speedup $S$ ($\times$), $C=16$	Temperature = 0 Speedup $S$ ($\times$), $C=32$	Temperature = 1 Avg $\tau$	Temperature = 1 Speedup $S$ ($\times$), $C=1$	Temperature = 1 Speedup $S$ ($\times$), $C=8$	Temperature = 1 Speedup $S$ ($\times$), $C=16$	Temperature = 1 Speedup $S$ ($\times$), $C=32$
Reasoning = ON
GSM8K	DFlash	4.81	2.62	1.42	2.48	1.98	4.18	2.42	1.16	2.04	1.7
	DStill	5.71	2.98	1.67	2.79	2.19	4.73	2.64	1.3	2.34	1.9
MATH-500	DFlash	4.77	2.66	1.36	2.35	2.06	4.06	2.3	1.16	1.97	1.78
	DStill	5.52	3.02	1.52	2.69	2.39	4.53	2.47	1.28	2.19	1.95
HumanEval	DFlash	4.29	2.31	1.19	2.05	1.89	4.21	2.35	1.17	1.99	1.81
	DStill	4.89	2.57	1.33	2.33	2.13	4.74	2.63	1.27	2.24	2
MBPP	DFlash	3.78	2.08	1.01	1.77	1.64	3.76	2.11	1.02	1.7	1.57
	DStill	4.37	2.33	1.14	2.03	1.9	4.29	2.33	1.15	1.94	1.79
MT-Bench	DFlash	3.15	1.77	0.88	1.51	1.47	3.1	1.78	0.86	1.47	1.43
	DStill	3.59	1.94	0.96	1.7	1.68	3.45	1.93	0.95	1.62	1.58
SWE-Bench	DFlash	2.9	1.62	0.79	1.35	1.3	2.86	1.64	0.79	1.3	1.23
	DStill	3.03	1.66	0.82	1.4	1.35	2.97	1.67	0.81	1.33	1.3
Reasoning = OFF
GSM8K	DFlash	5.65	3.1	2.27	3.33	2.22	4.88	2.71	1.85	2.69	2.01
	DStill	7.02	3.75	2.66	3.83	2.47	5.9	3.37	2.13	3.04	2.24
MATH-500	DFlash	4.95	2.69	1.4	2.47	2.09	4.43	2.54	1.27	2.23	1.96
	DStill	5.51	2.91	1.56	2.77	2.31	4.92	2.68	1.44	2.42	2.19
HumanEval	DFlash	5.99	3.2	2.15	3.07	2	5.32	2.88	1.89	2.74	1.83
	DStill	7.46	3.88	2.64	3.43	2.28	6.52	3.46	2.26	3.22	1.95
MBPP	DFlash	4.42	2.42	1.21	2.02	1.65	4.2	2.35	1.18	1.92	1.56
	DStill	5.33	2.82	1.49	2.4	1.95	5.05	2.77	1.41	2.24	1.85
MT-Bench	DFlash	2.91	1.7	0.89	1.45	1.42	2.76	1.61	0.84	1.36	1.21
	DStill	3.3	1.91	0.98	1.67	1.5	3.13	1.78	0.94	1.52	1.4
SWE-Bench	DFlash	2.98	1.68	0.88	1.46	1.35	2.83	1.65	0.84	1.36	1.24
	DStill	3.22	1.77	0.93	1.54	1.45	3.11	1.77	0.87	1.51	1.32

Evaluation metrics. Mean acceptance length $\tau$ is the average number of tokens in the accepted block prefix, including the anchor token. We report speculative decoding speedup over autoregressive target decoding as $S = t_{\mathrm{AR}}/t_{\mathrm{spec}}$, where $t_{\mathrm{AR}}$ and $t_{\mathrm{spec}}$ denote generation time per output token under autoregressive and speculative decoding, respectively. Averages reported in the tables are unweighted means across benchmarks or concurrency levels, as indicated.

Main Results

Benchmark comparison. We compare DSTILL with DFlash on reasoning, mathematics, code generation, and conversation benchmarks for three model families across temperature and reasoning settings in Table 1, Table 2, and Table 3. We train iso-parameter models for both the methods using the OpenPerfectBlend dataset.

Performance across reasoning modes. With reasoning enabled, DSTILL consistently outperforms DFlash by 6.7%-18.8% on average speedups over autoregressive decoding and 7.6%-21.1% on acceptance length as shown in Table 1. On GPT-OSS-120B, DSTILL shows speed up improvements over DFlash across temperature and concurrency settings as in Table 2. In non-thinking mode, Nemotron achieves speedups of up to $3.88\times$ and $3.46\times$ at temperatures 0 and 1, respectively, at concurrency 1 (Table 3).

Transfer of refinement gains. Table 4 compares DFlash, the teacher, and DSTILL across refinement steps. Eight teacher passes improve acceptance length by 18.0% on GPT-OSS-120B and 17.9% on Nemotron. One-pass DSTILL recovers 35.8% and 28.6% of these gains, outperforming DFlash by 7.6% and 19.4%, respectively. The largest teacher–student gap is on Nemotron HumanEval (8.57 vs. 7.46; 1.11 tokens). Eight-pass DFlash even underperforms the one-pass teacher, likely due to a mismatch between its root-only training and partially revealed refinement states, motivating Stage I.

Serving performance. DSTILL improves throughput across all reported settings, achieving speedups of up to 5.2× for GPT-OSS-120B and 3.0× for Nemotron-3.5-30B-A3B over autoregressive decoding. At concurrency 32, speedups remain 2.7–3.9× and 2.1–2.4×, respectively, showing that the decoding gains persist under concurrent serving.

Ablation Studies

We examine the effects of continued baseline training and student state sampling. Evaluations use mean acceptance length with greedy decoding and one drafting pass on $8 \times$ H100 GPUs. Unless otherwise stated, student distillation runs for 3 epochs with $\alpha= 0.2$ and a cosine $p_{\mathrm{root}}$ schedule from 0.5 to 0.85. Additional ablations are reported in Appendix § 9.

Continued one-step training. Table 5 compares DFlash trained for 10 epochs with DSTILL in its default setting on Gemma-4-31B. DSTILL achieves higher acceptance length across all six benchmarks, with improvements ranging from 0.21 tokens on SWE-bench to 0.7 tokens on GSM8K. These results suggest that trajectory distillation provides gains beyond longer DFlash training.

Dataset	DFlash (5 epochs)$\uparrow$	DFlash (10 epochs)$\uparrow$	DSTILL (default)$\uparrow$
GSM8K	6.78	6.54	7.48
MATH500	6.84	6.67	7.28
HumanEval	6.03	5.96	6.49
MBPP	5.00	5.00	5.52
MT-Bench	3.67	3.57	4.03
SWE-Bench	3.17	3.13	3.38

State sampling. We compare root-only training, which samples exclusively from the fully masked block state ($p_{\mathrm{root}} = 1$), with our cosine schedule, which progressively increases the probability of sampling this state from 0.5 to 0.85. We trained the drafters for 16 epochs, and conduct this ablation on one-eighth of the training dataset, with all other training settings held fixed. Table 6 shows the cosine schedule improves $\tau$ across all six benchmarks, with the largest gains on GSM8K (+0.22) and MBPP (+0.10). These results suggest that progressively increasing the probability of sampling the fully masked state, while retaining exposure to intermediate trajectory states, improves student acceptance length.

Dataset	$p_{\mathrm{root}} = 1 \uparrow$	Cosine$\uparrow$	$\Delta$
GSM8K	4.77	4.99	+0.22
HumanEval	4.58	4.60	+0.02
MATH500	4.90	4.97	+0.07
MBPP	4.11	4.21	+0.10
MT-Bench	3.16	3.19	+0.03
SWE-Bench	2.57	2.58	+0.01
Average	4.01	4.09	+0.08

Conclusion

We presented a trajectory consistency distillation procedure for block-diffusion speculative decoding: a two-stage training pipeline that trains a multi-step masked-diffusion drafter with partial-mask supervision and then distills its confidence-guided denoising trajectory into a one-step student via an offline trajectory-replay pipeline. Distillation trains the student to map any state along the teacher’s trajectory, including the root state used at inference time, directly to the teacher’s endpoint, under a curriculum that shifts weight toward the root state as training progresses. We show that this procedure recovers 28.6%–35.8% of the acceptance length improvements of a multi-step teacher in a single step student with zero test-time overheads. Across models and domains, we demonstrate improvements of 28% in acceptance length and 25% in speedup. Finally, we demonstrate our procedure’s general applicability to diffusion speculators by improving DSpark acceptance lengths by 4.5%.

Limitations

DSTILL has a higher one-time training cost than direct DFlash training, comprising partial-mask teacher training, offline trajectory collection and storage, and student distillation. These additional costs can be amortized over subsequent generation, but only at a large enough inference workload. The student learns the teacher’s selected continuations and inherits errors in those endpoints, and the student’s one-pass outputs also remain factorized. Our experiments only cover the target models, drafter configurations, and benchmarks reported here. Transfer to other block sizes, architectures beyond DFlash and DSpark, and target-feature layouts requires further evaluation. Acceptance-length gains must be considered together with measured latency under the inference workload. Actual latency reductions depend on the specific inference workload, hardware serving kernels, and concurrency limits.

Drafting and Verification Details

Drafting Setup

Let $\mathcal{T}$ denote the target language model and $(u_{1:S}, a)$ the current verified prefix, where $a$ is its most recent token. We cache the target’s hidden states $H_c$ at a fixed set of layers over $u_{1:S}$. A draft block contains $B$ positions: position 0 holds the fixed anchor $a$, and positions $1,\ldots,B-1$ are the future tokens to be proposed. Uncommitted positions contain [MASK], so the initial block is $x_1 = (a, [\mathrm{MASK}],\ldots,[\mathrm{MASK}])$.

The drafter $D_\theta$ receives embeddings of the current block state $x$ together with $H_c$, attends bidirectionally within the block and to context strictly before its anchor, and produces a vocabulary distribution $p_{\theta,i}(\cdot\mid x,H_c)$ at each masked position $i$. Independently sampling all non-anchor positions from the root predictions gives the factorized proposal in Equation (1). Greedy drafting selects the most probable token at each position. In both cases, all positions are predicted from the same root state, yielding the one-pass factorization discussed in § 1.

The target scores the proposal in one causal forward pass. Standard lossless speculative sampling uses the target’s prefix-conditional probabilities and the proposal’s actual left-to-right conditionals, together with rejection correction (Leviathan et al., [24]; Chen et al., [5]). Under greedy verification, acceptance is determined by matching the target’s top-1 predictions. Let $L \in\{0,\ldots,B-1\}$ be the length of the accepted draft prefix. We denote mean acceptance length by $\tau= 1 + \mathbb{E}[L]$.

Denoising Trajectories

A multi-step drafter resolves the block over $K$ sequential passes. Let $x_t \equiv x^{(t)}$ denote the state before pass $t \in\{1,\ldots,K\}$, starting from the anchor-only root $x_1$. The set of positions already committed is $c^{(t)} \subseteq\{0,\ldots,B-1\}$, with $c^{(1)} = \{0\}$ and cardinality $|c^{(t)}|$. Thus the root contains the given anchor but zero committed future tokens. The remaining masked positions are $M_t = \{1,\ldots,B-1\} \setminus c^{(t)}$; we write $M$ when the pass index is unnecessary.

At pass $t$, the drafter predicts from $(x_t,H_c)$ and assigns each masked position confidence $\operatorname{conf}_i = \max_v p_{\theta,i}(v \mid x_t,H_c)$. It commits the

highest-confidence positions, writing their greedy or sampled tokens into the block and leaving earlier commitments unchanged. The updated state $x^{(t+1)}$ is the input to the next pass. The schedule allocates the remaining masked positions across the remaining passes, committing all positions after $K$ passes. The rollout starts from the root predictions in Equation (1); later passes condition on tokens committed earlier.

We reserve $x_0 \equiv x^{(K+1)} = (a,y_1,\ldots,y_{B-1})$ for the fully committed teacher endpoint. Here, $t$ indexes forward passes and the subscript 0 identifies the completed block. The student’s proposed block is $\hat{x}_0 = (a,\hat{y}_1,\ldots,\hat{y}_{B-1})$. Figure 1 uses these state conventions. Lossless verification of a sampled multi-step proposal requires its actual left-to-right conditionals, which can differ from the distributions recorded at commitment time.

This rollout defines the intermediate states $x_t$ and endpoint $x_0$ used in § 3: Stage I trains the teacher across intermediate corruption levels, and Stage II uses recorded teacher trajectories to supervise a one-pass student.

Training and Trajectory Details

Reveal schedule and teacher objective

During training, we cache $H_c$ from teacher-forced sequences and construct blocks around sampled anchor positions. Each block’s attention is restricted to its own positions and target context strictly before its anchor, with bidirectional attention within the block. This attention mask prevents cross-block leakage and allows multiple anchored blocks to be scored in parallel. Let $z = (a, z_1, \ldots, z_{B-1})$ denote a reference training block.

We derive corruption levels from the number of positions committed before each pass of Equation (4). Let $m_t$ include the anchor. The corresponding reveal probabilities for non-anchor positions are

For each block, we sample $r$ uniformly from $\mathcal{R}$ and independently reveal each non-anchor position with probability $r$; the anchor is always revealed. Revealed positions use their reference-token embeddings, and the others use [MASK], yielding the corrupted block $x^r$. The reveal level and Bernoulli mask are sampled anew when a block is revisited across epochs. The root level is $r=0$. Each level matches the expected number of revealed positions to a rollout state. Bernoulli sampling varies the revealed counts and position subsets. Fully committed levels are excluded from $\mathcal{R}$. Any Bernoulli outcome with no masked positions contributes no loss and is excluded from the training average.

For a nonempty masked set $M$, the per-block objective is

Training minimizes the arithmetic mean of this objective over nonempty blocks; batches with no nonempty blocks are skipped. The block weight scales each block’s normalized position loss and controls its contribution to the training average. Both weights are optional and default to 1.

Mask-count weighting. To emphasize heavily masked blocks relative to nearly complete ones, we use

Setting $\alpha_m=0$ gives $w_{\mathrm{block}}=1$, while $\alpha_m>0$ increases the relative weight of heavily masked blocks. The denominator normalizes against a uniform reference distribution over mask counts. The mean weight under the training sampler depends on its mask-count distribution.

Position decay. Following DFlash [6], we optionally give greater weight to early positions because acceptance depends on a prefix. We use $w_{\mathrm{pos}}(i)=\exp(-(i-1)/\gamma)$ for $\gamma>0$. With position decay disabled, $w_{\mathrm{pos}}(i)=1$.

The discrete levels emphasize corruption depths associated with the teacher’s refinement schedule, while random reveal subsets broaden the training distribution. This aligns expected reveal counts; the positions and reference tokens differ from the confidence-selected predictions exposed during a teacher rollout. Stage-I examples reveal reference tokens independently; they are not connected rollouts through the drafter’s own predictions. They therefore do not supply the trajectory endpoints needed for distillation. We obtain these by running the frozen Stage-I checkpoint through the full refinement schedule. Equivalent trajectories already recorded from that checkpoint under the same rollout policy could be reused; no additional teacher-training stage is needed.

Trajectory records and reconstruction

We run the frozen teacher’s confidence-guided rollout offline, once per training block, and record the tokens it commits. Each record contains

• $x_0\in\{1,\ldots,V\}^B$: the fully committed end-point, where $V$ is the vocabulary size;

• $\kappa\in\{0,\ldots,K\}^B$: each position’s commit pass, with $\kappa_0=0$;

• $(\operatorname{ids}_i^{(t)},\log p_i^{(t)})$: the top-$k$ token ids and their full-softmax log-probabilities at each position while it remains masked.

Because commitments are never revised, $x_0$ and $\kappa$ reconstruct every intermediate block without storing a separate $K\times B$ sequence of block states:

For example, a token committed at pass 2 is masked in the input to that pass and visible in the next state. The revealed token is the teacher-committed token $x_{0,i}$, which may differ from the reference token $z_i$.

For each non-anchor position $i$, let $S_i=\operatorname{ids}_i^{(\kappa_i)}$ be the retained ids at its committing pass. For $v\in S_i$, recover the original teacher probability as $q_{i,v}=\exp(\log p_i^{(\kappa_i)}(v))$, where the stored log-probability corresponds to id $v$. These probabilities Algorithm 1 Offline Trajectory Collection

Input: Frozen teacher $D_\theta$, context $H_c$, anchor $a$, block size $B$, steps $K$, top-$k$

Output: Endpoint $x_0$, commit passes $\kappa$, top-$k$ records

1: $x^{(1)} \leftarrow(a, [\mathrm{MASK}], \ldots, [\mathrm{MASK}])$

2: $c^{(1)} \leftarrow\{0\}$; $\kappa_0 \leftarrow0$

3: for $t = 1$ to $K$ do

4: $M_t \leftarrow\{1,\ldots,B-1\} \setminus c^{(t)}$

5: $\mathrm{logits} \leftarrow D_\theta(x^{(t)}, H_c)$

6: $P_i^{(t)} \leftarrow\operatorname{softmax}(\mathrm{logits}_i^{(t)}), i \in M_t$

7: $\mathrm{conf}_i \leftarrow\max_v P_{i,v}^{(t)}, i \in M_t$

8: Store top-$k$ ids in $\mathrm{ids}^{(t)}$

9: Store their $\log P_{i,v}^{(t)}$ in $\mathrm{logp}^{(t)}$

10: $n_t \leftarrow\left\lceil(B - |c^{(t)}|)/(K - t + 1) \right\rceil$

11: $J_t \leftarrow$ top $n_t$ indices in $M_t$ by confidence

12: $x^{(t+1)} \leftarrow x^{(t)}$

13: Fill $J_t$ with greedy or sampled teacher tokens

14: $\kappa_i \leftarrow t$, $i \in J_t$

15: $c^{(t+1)} \leftarrow c^{(t)} \cup J_t$

16: end for

17: $x_0 \leftarrow x^{(K+1)}$

18: return $x_0$, $\kappa$, $\{(\mathrm{ids}^{(t)}, \mathrm{logp}^{(t)})\}_{t=1}^{K}$

retain their full-vocabulary normalization. Define $q_{i,\mathrm{other}} = 1 - \sum_{v \in S_i} q_{i,v}$ and let $Q_{\mathrm{end},i}$ denote the resulting distribution over $S_i \cup\{\mathrm{other}\}$. We write $Q_{\mathrm{end}} = (Q_{\mathrm{end},i})_{i=1}^{B-1}$. Thus, the record supports both exact state reconstruction and committing-pass distribution supervision.

State-sampling schedule

With probability $p_{\mathrm{root}}$, we sample the anchor-only root $x_1$ ($t = 1$). Otherwise, we sample $t$ uniformly from $\{2,\ldots,\kappa_{\max}\}$, where $\kappa_{\max} = \max_i \kappa_i$ is the last commit pass. Blocks with no intermediate state fall back to $t = 1$. We resample $t$ on each visit to a trajectory across epochs, reconstructing the corresponding recorded state each time. The endpoint $x_0$ and committing-pass distributions remain fixed. Thus, different partial commitments provide supervision toward the same completed block; the student learns to predict all remaining scored positions in one pass. This combines the root state with intermediate states that provide additional endpoint supervision. Teacher-committed tokens in intermediate states provide additional context for predicting the remaining masked positions early in training. We ramp the root probability with training progress $\rho\in[0,1]$:

Here $w,c \ge0$ are warmup and cooldown fractions with $w+c<1$. We use $p_{\mathrm{start}}=0.5$, $p_{\mathrm{end}}=0.85$, $w=0.10$, and $c=0.30$. The schedule gives intermediate states more weight early and increasingly emphasizes the root state. In particular, the root receives direct supervision throughout training, with sampling probability 0.5 initially and 0.85 late in training. At the root, every scored non-anchor position is masked and receives endpoint supervision. Root behavior is therefore trained explicitly rather than inferred only from partially committed inputs.

Distillation distributions and loss aggregation

We reconstruct $x_t$ using Equation (8) and restrict $M_t$ to still-masked, scored response positions, excluding the anchor. Scored positions are response-token positions selected for supervision by the training loss mask. For $v \in S_i$, let $p_{i,v} = p_{\phi,i}(v \mid x_t, H_c)$ and $p_{i,\mathrm{other}} = 1 - \sum_{v \in S_i} p_{i,v}$. Define $P_{\phi,i}$ over the same $k+1$ categories as $Q_{\mathrm{end},i}$. The per-position objective is Equation (3). The CE term targets the teacher’s committed token, while KD transfers its committing-pass probability mass, including the omitted tail:

The KL divergence is computed over the retained token ids and a single aggregate tail category. Storing the teacher’s top-$k$ probabilities is sufficient to evaluate this coarsened loss. We use $\alpha= 0.2$; § 4 reports its ablation.

Trajectory-quality weighting. Optionally, we favor positions where the teacher’s endpoint agrees with the reference training block $z$. Let $d$ be the first scored non-anchor position with $x_{0,d} \ne z_d$, or

Algorithm 2 Trajectory Consistency Distillation
Input: Student $D_{\phi}$, training batch with stored trajectories, contexts $H_c$, progress $\rho$
Output: Updated $\phi$, unless the batch has zero weight
1: $\rho_{\mathrm{root}} \leftarrow$ {xref}`Equation 9` at $\rho$
2: lossSum $\leftarrow 0$; weightSum $\leftarrow 0$
3: for each block in the batch do
4: Resample $t$ from the root-biased schedule
5: Reconstruct $x_t$ using {xref}`Equation 8`
6: $M_t \leftarrow \{i : \kappa_i \ge t,\ i\ \text{scored},\ i \ne 0\}$
7: if $M_t \ne \emptyset$ then
8: logits $\leftarrow D_{\phi}(x_t, H_c)$
9: Recover $Q_{\mathrm{end},i}$ at pass $\kappa_i$, $i \in M_t$
10: Compute $\mathcal{L}_i$ using {xref}`Equation 3`, $i \in M_t$
11: Set $w_i$ using {xref}`Equation 11`, or 1 if disabled
12: lossSum $\leftarrow$ lossSum $+ \sum_{i\in M_t} w_i\mathcal{L}_i$
13: weightSum $\leftarrow$ weightSum $+ \sum_{i\in M_t} w_i$
14: end if
15: end for
16: if weightSum $> 0$ then
17: Backpropagate lossSum/weightSum
18: Update student parameters $\phi$
19: end if

$d = B$ if there is no disagreement. We use

with $0 < w_{\mathrm{first}} \le1$ and $0 < \beta< 1$; a flat tail weight applies if $\beta$ is unset. The weights provide a reference-agreement heuristic that reduces supervision after a mismatch while retaining the trajectory. A teacher token that differs from a sampled reference continuation can still be accepted under lossless speculative sampling.

For a batch of blocks indexed by $b$, we aggregate over scored masked positions using

When trajectory weighting is disabled, $w_{b,i} = 1$, recovering the ordinary mean token loss. Only blocks with scored masked positions enter the batch loss; batches with no such positions are skipped. Algorithm 2 summarizes one update.

Ablation Results

Student supervision. To investigate the effect of the supervision objective, we compare endpoint CE ($\alpha= 0$), KD ($\alpha= 1$), and their combination, CE+KD ($\alpha= 0.2$), in Equation (3). As shown in Table 7, the three objectives achieve broadly comparable acceptance lengths across benchmarks. CE and KD provide complementary supervision signals, combining endpoint-level targets with distributional information from the teacher.

Dataset	$\alpha= 0$	$\alpha= 0.2$	$\alpha= 1$
GSM8K	5.6	5.4	5.5
HumanEval	5.0	5.1	5.1
MATH500	5.7	5.6	5.6
MBPP	4.5	4.4	4.5
MT-Bench	3.3	3.4	3.3
SWE-Bench	2.8	2.8	2.8

Trajectory-quality weighting. The distillation loss can optionally down-weight supervision at positions after the teacher’s recorded endpoint first diverges from the reference continuation, following the reference-agreement heuristic in Equation (11). We compare a student distilled with uniform position weights against one that enables this weighting with $w_{\mathrm{first}} = 0.4$ and tail decay $\beta= 0.5$. Table 8 reports student acceptance length. Weighting increases $\tau$ on all six benchmarks, by 0.04 to 0.12 tokens, raising the average from 4.72 to 4.80 (+1.7%). Concentrating supervision on the agreed prefix, rather than on lower-agreement suffix positions that are less likely to be accepted, improves one-step drafting at no additional inference cost.

Dataset	Unweighted†	Weighted†	$\Delta$
GSM8K	6.96	7.01	+0.05
MATH-500	5.35	5.47	+0.12
HumanEval	4.77	4.88	+0.11
MBPP	4.95	4.99	+0.04
MT-Bench	3.26	3.32	+0.06
SWE-Bench	3.06	3.15	+0.09
Average	4.72	4.80	+0.08

Additional Experimental Results

Table 4 compares drafting cost and acceptance for the anchor-only baseline, the teacher evaluated with one or eight passes, and the distilled one-pass student. The teacher comparison isolates the refinement gain available for distillation.

Experimental Details

Verifier models. Table 9 lists the target-model specifications. The $A$ suffix denotes nominal active parameters for the Gemma and Nemotron MoE models.

Target Model	Size	Architecture	Precision	Context	# Layers
GPT-OSS-120B ([36])	120B	MoE	BF16	131,072 (128K)	36
Gemma-4-31B ([15])	31B	Dense	BF16	262,144 (256K)	60
Gemma-4-26B-A4B ([14])	26B-A4B	MoE	BF16	262,144 (256K)	30
Nemotron-3.5-30B-A3B ([35])	30B-A3B	Hybrid MoE	BF16	1,048,576 (1M)	52

Training data. OpenPerfectBlend (mlabonne, n.d.) supplies an instruction mixture spanning general conversation, mathematics, and code. For GPT-OSS-120B, assistant turns are regenerated through SGLang with reasoning effort set to medium and temperature 0.7. The regenerated conversations are chat-templated and converted to Arrow. A teacher-forced pass through the completed sequences caches input ids, the training loss mask, and target hidden states at target_layer_ids, using sequence length 24,576. The cache is reused across teacher training, trajectory collection, and student distillation.

Drafter architecture. The GPT-OSS-120B drafter is an injection-conditioned Qwen3-style transformer with $B = 16$, mask token id 200,000, and target_layer_ids = [1, 9, 17, 25, 33]. The backbone has 8 decoder layers, hidden size 2880, intermediate size 7680, and grouped-query attention with 64 query heads and 8 key/value heads. At each layer, a linear projection of the concatenated target features (input dimension $5 \times2880$), followed by RMSNorm, supplies additional attention keys and values. The query stream carries the current block’s token embeddings, including mask embeddings at unrevealed positions. Target embeddings and the LM head are frozen and shared; only draft-specific parameters are optimized. The configuration estimate is approximately 0.8B drafter parameters; the trainable parameter count is 785M. Drafter hidden dimensions and selected target layers for Gemma-4-31B, Gemma-4-26B-A4B, and Nemotron-3.5-30B-A3B are respectively 5376 ([1, 12, 23, 35, 46, 57]), 2816 ([1, 6, 11, 17, 22, 27]), and 2688 ([1, 11, 20, 30, 39, 49]).

Training configuration. Our training implementation builds on SpecForge ([25]). For GPT-OSS-120B, Stage I trains the teacher for 10 epochs on 16 A100 nodes, and Stage II distills the student for 3 epochs on 4 H100 nodes. Both stages use a learning-rate warmup ratio of 0.03 and gradient-norm clipping at 1.0. Trajectory collection uses $K = 8$ passes with greedy token selection and retains the top-16 probabilities. Trajectory-quality weighting is disabled except in its dedicated ablation.

Benchmarks and generation settings. We evaluate 128 prompts each from GSM8K, MATH-500, and MBPP; 164 each from HumanEval and SWE-bench; and 80 from MT-Bench. Generation uses a maximum of 4096 new tokens, with temperatures and reasoning settings specified in the corresponding tables. GPT-OSS-120B uses reasoning effort medium. Autoregressive and speculative runs use the same requests and generation settings. Benchmark averages are unweighted arithmetic means across the six datasets, computed separately for speedup and acceptance length. In the concurrency tables, Avg $\tau$ averages acceptance length over $C \in\{1, 8, 16, 32\}$.

Inference implementation. We use 8 H100 GPUs with vLLM version 0.30.0 ([22]), tensor-parallel size 8, and GPU memory utilization 0.8. One-pass DFlash and DSTILL evaluations use num_denoising_steps=1 and num_speculative_tokens=15. The refinement comparison additionally evaluates drafters with eight denoising passes. DFlash and DSpark use the corresponding dflash and dspark speculative configurations.

Measurement. Each inference configuration is evaluated once after four warmup runs, each allowing up to 128 generated tokens. Generation time per output token is computed by dividing the measured generation time by the actual number of generated tokens.

Teacher-gain recovery and ablation controls. We compute recovery from acceptance lengths averaged across the six benchmarks:

Recovery is reported when the denominator is positive. The state-sampling ablation uses Gemma-4-26B-A4B, with both variants trained for 16 epochs on one-eighth of the training corpus and evaluated on all six benchmarks at concurrency 16. All other training settings are held fixed. Root-only sampling uses $p_{\mathrm{root}} = 1$. The cosine schedule increases this probability from 0.5 to 0.85, with warmup and cooldown fractions of 0.10 and 0.30, respectively. Its training-averaged root probability is approximately 0.71.

Reproducibility

Appendices A and B detail the drafting, verification, and training procedures. § 4.1 and Appendix Appendix § 11 describe the datasets, model configurations, hyperparameters, hardware, and evaluation settings.

Model	Method	Params (M)	Hidden Layers	Target Hidden Layers	Attention
GPT-OSS-120B	DFlash	785	2880	8	1, 9, 17, 25, 33	Full
GPT-OSS-120B	DStill (Ours)	785	2880	8	1, 9, 17, 25, 33	Full
Gemma-4-31B	DFlash	1536	5376	5	1, 12, 23, 35, 46, 57	$\mathrm{SWA}_{2048}+1$ Full
Gemma-4-31B	DStill (Ours)	1536	5376	5	1, 12, 23, 35, 46, 57	$\mathrm{SWA}_{2048}+1$ Full
Gemma-4-26B-A4B	DFlash	430	2816	5	1, 6, 11, 17, 22, 27	$\mathrm{SWA}_{2048}$
Gemma-4-26B-A4B	DStill (Ours)	430	2816	5	1, 6, 11, 17, 22, 27	$\mathrm{SWA}_{2048}$
Nemotron-3.5-30B-A3B	DFlash	481	2688	6	1, 11, 20, 30, 39, 49	$\mathrm{SWA}_{2048}$
Nemotron-3.5-30B-A3B	DStill (Ours)	481	2688	6	1, 11, 20, 30, 39, 49	$\mathrm{SWA}_{2048}$

Application to DSpark

We apply trajectory consistency distillation to DSpark ([9]), jointly training its backbone and auxiliary heads. We collect $K = 8$ teacher refinement trajectories and distill a one-pass student, retaining the Markov and confidence losses in both training stages. Table 11 reports the per-benchmark acceptance lengths and generation speedups.

Dataset	Acceptance length $\tau$		Speedup $S$
	DSpark	+DSTILL	DSpark	+DSTILL
GSM8K	6.74	6.89	4.23	4.29
HumanEval	5.28	5.68	3.42	3.66
MATH-500	4.84	4.97	3.14	3.22
MBPP	5.01	5.11	3.25	3.29
MT-Bench	3.31	3.51	2.19	2.33
SWE-Bench	3.03	3.28	2.02	2.19
Average	4.70	4.91	3.04	3.16