WorksComputer Science
Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding
Abstract
In speculative decoding, a lightweight draft model proposes tokens that the target model verifies in parallel, speeding up generation without changing the target distribution. D-PACE (Wu et al., 2026) trains such drafters with a loss that weights each position by its contribution to the expected accepted length. Both its weights and its cross-entropy (CE) loss use only the draft's probability of the recorded target token, treating the target as a point mass. Under rejection sampling, however, a sampled draft token is accepted with probability equal to the overlap of the draft and target distributions, one minus their total variation (TV). We derive D-PAL (Dynamic Position-wise Acceptance Loss) from the exact expected accepted length, using this overlap as the acceptance signal. This yields overlap-based position weights with log acceptance (LA) as the per-position loss, and the objective reduces to D-PACE for a point-mass target. Since the weights are detached, any per-position loss can replace LA in practice. We show that forward KL divergence and Rényi divergence of order 1/2 match or exceed LA, and that Rényi-1/2 provably bounds the true acceptance rate from above and below. Replacing CE with Rényi-1/2 improves acceptance length under both D-PACE's weights and ours. On six benchmarks with a Qwen3-4B target, D-PAL raises mean acceptance length over D-PACE from 4.50 to 4.57 at T=0 and from 4.20 to 4.28 at T=1, and consistently achieves higher throughput at both temperatures.
Introduction
Speculative decoding accelerates autoregressive generation without changing the target model’s output distribution [21, 6]: a lightweight draft model proposes tokens, and the target model verifies them in one parallel pass. DFlash drafts a whole block in one forward pass [7]. Since verification stops at the first rejection, early block positions matter most. DFlash trains with cross-entropy (CE) and a fixed position decay, and DSpark and H-Spec add a total variation (TV) term to CE under the same decay [9, 19]. LK Losses optimizes acceptance directly, with the negative log acceptance or an adaptive mix of Kullback–Leibler (KL) divergence and TV, but still uses fixed decay [33]. D-PACE (Dynamic Position-Aware Cross-Entropy) instead derives position weights from a surrogate of the expected accepted length [41]. SpecForge [35], Speculators [36], and TorchSpec [38] use it to train DFlash drafters.
D-PACE treats the target as a point mass on the recorded token : it estimates acceptance by the draft’s confidence and fits that token with CE. Under standard rejection sampling, however, acceptance depends on the full target distribution through the overlap $\tilde{\alpha}_t = \sum_v \min(p_{t,v}, q_{t,v}) = (Leviathan et al., [21]). When the target is uncertain, confidence can underestimate this overlap by a wide margin. For example, if is uniform over tokens, the verifier always accepts a sampled draft token (), yet . CE likewise fits only the recorded token.
We propose D-PAL (Dynamic Position-wise Acceptance Loss), which we derive by differentiating the expected accepted length with the true acceptance signal . The derivation yields overlap-based position weights paired with a log acceptance (LA) loss, , as the local loss, which reduces to D-PACE when is a point mass. Because D-PAL detaches the weights, we can swap LA for any local loss in practice. We write D-PAL[X] for D-PAL with local loss X. We show overlap bounds the acceptance of the argmax proposals that deployed DFlash uses (Proposition 1). Replacing CE with a distributional loss, Rényi divergence of order , raises mean acceptance length under both confidence and overlap weights. In our experiments, we show that LA is not the best local loss. It is not differentiable where a draft probability equals the target probability, and its gradient with respect to the draft logits does not shrink near , whereas the KL and Rényi gradients vanish there. Forward KL and Rényi-1/2 match or exceed LA, and Rényi-1/2, unlike KL, bounds overlap from both sides (Lemma 1). We also test TV-based losses, where pure TV collapses and CE+TV falls short, and analyze their gradients.
Our contributions are:
Derivation from expected accepted length. Differentiating the exact expected accepted length with the overlap as the acceptance signal gives overlap-based weights paired with LA. Proposition 1 connects the overlap to the argmax verification used in deployment. We show D-PAL[LA] improves on D-PACE, raising mean acceptance length on Qwen3-4B from 4.50 to 4.54 at and from 4.20 to 4.24 at (Table 1).
| Method | GSM8K SR | GSM8K | MATH-500 SR | MATH-500 | HumanEval SR | HumanEval | MBPP SR | MBPP | MT-Bench SR | MT-Bench | Alpaca SR | Alpaca | Mean SR | Mean |
| Baselines — Temperature | ||||||||||||||
| DFlash | 3.20 | 4.50 | 3.94 | 5.46 | 3.33 | 4.67 | 3.06 | 4.28 | 1.97 | 3.15 | 1.77 | 2.71 | 2.88 | 4.13 |
| D-PACE | 3.50 | 4.96 | 4.35 | 6.06 | 3.60 | 5.09 | 3.34 | 4.65 | 2.05 | 3.36 | 1.83 | 2.87 | 3.11 | 4.50 |
| D-PAL weights | ||||||||||||||
| D-PAL[LA] | 3.50 | 5.03 | 4.32 | 6.06 | 3.65 | 5.15 | 3.38 | 4.70 | 2.04 | 3.41 | 1.84 | 2.90 | 3.12 | 4.54 |
| D-PAL[KL] | 3.50 | 5.06 | 4.43 | 6.11 | 3.68 | 5.19 | 3.38 | 4.71 | 2.08 | 3.42 | 1.87 | 2.92 | 3.16 | 4.57 |
| D-PAL[Rényi] | 3.53 | 5.07 | 4.39 | 6.10 | 3.66 | 5.17 | 3.31 | 4.68 | 2.08 | 3.44 | 1.86 | 2.92 | 3.14 | 4.56 |
| D-PAL[TV] | 0.75 | 1.01 | 0.76 | 1.02 | 0.77 | 1.04 | 0.77 | 1.03 | 0.77 | 1.05 | 0.77 | 1.07 | 0.77 | 1.04 |
| D-PAL[CE+TV] | 3.39 | 4.89 | 4.24 | 5.86 | 3.50 | 4.92 | 3.27 | 4.54 | 1.94 | 3.23 | 1.77 | 2.86 | 3.02 | 4.39 |
| D-PAL[KL+TV] | 3.49 | 5.03 | 4.33 | 6.05 | 3.64 | 5.14 | 3.33 | 4.68 | 2.04 | 3.40 | 1.85 | 2.88 | 3.11 | 4.53 |
| Baselines — Temperature | ||||||||||||||
| DFlash | 2.99 | 4.29 | 3.25 | 4.80 | 3.07 | 4.40 | 2.85 | 4.09 | 1.84 | 3.04 | 1.71 | 2.71 | 2.62 | 3.89 |
| D-PACE | 3.24 | 4.76 | 3.54 | 5.31 | 3.30 | 4.75 | 3.03 | 4.41 | 1.90 | 3.18 | 1.75 | 2.82 | 2.79 | 4.20 |
| D-PAL weights | ||||||||||||||
| D-PAL[LA] | 3.28 | 4.78 | 3.52 | 5.26 | 3.37 | 4.81 | 3.05 | 4.48 | 1.91 | 3.26 | 1.76 | 2.85 | 2.82 | 4.24 |
| D-PAL[KL] | 3.28 | 4.81 | 3.49 | 5.27 | 3.36 | 4.82 | 3.05 | 4.47 | 1.91 | 3.25 | 1.78 | 2.85 | 2.81 | 4.25 |
| D-PAL[Rényi] | 3.34 | 4.84 | 3.52 | 5.35 | 3.37 | 4.82 | 3.10 | 4.48 | 1.94 | 3.28 | 1.81 | 2.89 | 2.85 | 4.28 |
| D-PAL[TV] | 0.75 | 1.01 | 0.76 | 1.02 | 0.77 | 1.04 | 0.77 | 1.03 | 0.77 | 1.05 | 0.77 | 1.06 | 0.76 | 1.04 |
| D-PAL[CE+TV] | 3.18 | 4.64 | 3.48 | 5.19 | 3.23 | 4.61 | 2.96 | 4.32 | 1.85 | 3.11 | 1.69 | 2.78 | 2.73 | 4.11 |
| D-PAL[KL+TV] | 3.25 | 4.77 | 3.58 | 5.34 | 3.37 | 4.88 | 3.07 | 4.49 | 1.91 | 3.30 | 1.77 | 2.84 | 2.83 | 4.27 |
Table 1. D-PAL[Rényi], D-PAL[LA], and D-PAL[KL] beat D-PACE; D-PAL[TV] and D-PAL[CE+TV] fall short. Qwen3-4B target, three-layer DFlash drafts, , 100K training examples, for D-PAL and 0.5 for D-PACE. SR is speedup over autoregressive decoding, is acceptance length, and bold marks the best value.
Local loss beyond log acceptance. The derivation points to LA, but LA is not differentiable where the draft matches the target. Keeping the same weights, we replace it with forward KL and with Rényi-1/2, and both reach the same or higher acceptance length (Table 1). We also show Rényi-1/2 bounds overlap from above and below (Lemma 1).
Consistent gains over D-PACE and DFlash. In experiments across temperatures, block sizes, draft depths, target models at the 4B and 8B scale, and thinking modes, we show that D-PAL consistently achieves longer acceptance lengths and higher throughputs than both D-PACE (up to 5%) and DFlash (up to 11%).
Background
Speculative decoding with DFlash
In speculative decoding (Leviathan et al., [21]; Chen et al., [6]), the target model scores a block of draft proposals in one forward pass and accepts a contiguous prefix of length .
DFlash (Chen et al., [7]) drafts with a small block-diffusion model that projects hidden states from a few target layers into the keys and values of every draft layer. A block of positions starts with an anchor, the last verified token, followed by mask tokens. In one forward pass, the drafter predicts distributions for all masked positions. Each thus depends on the prefix but not on earlier proposals in the block, while the target’s conditions on all tokens before .
In training, DFlash samples anchor positions in responses generated by the frozen target and predicts the next recorded tokens from each anchor context. Since a proposal counts only if every earlier one is accepted, DFlash fits with CE and weights positions by a fixed exponential decay,
with for . The frozen target output head also gives the full target distribution at each position, but DFlash does not use it.
Expected accepted length
Let count the accepted draft tokens, and let be the conditional acceptance probability at position . The tail-sum identity gives the exact expected accepted length [23, 41]
Each appears in every accepted prefix ending at . For positive , its relative contribution and the gradient of are
Every term is positive, so decreases with .
Position weights and local losses
Equation (3) suggests drafter objectives for a block of the form , with weights computed on target-generated training contexts. DFlash [7] uses a fixed exponential decay to emphasize early positions (Equation (1)). D-PACE [41] instead uses the draft’s confidence as a proxy for , smooths it to with , and builds detached weights paired with CE:
With , an early low-confidence position cannot zero out all later weights.
D-PAL: Acceptance in the Weights, Fit in the Loss
Acceptance-derived weights from overlap
D-PACE’s unsmoothed confidence is overlap with a point mass on the recorded target token: . Standard rejection sampling instead compares the draft with the full target distribution. At a fixed context with target and draft distributions , the verifier accepts with probability . Averaging over gives the one-token acceptance [21, 6]
D-PAL applies the weighting rule of Equation (3) to this overlap. On target-generated sequences, the product of overlaps through position approximates the chance that sampled proposals survive to . As in D-PACE, we smooth only inside the products, , recompute the weights from the current draft, and detach them:
where is the local loss chosen in Section 3.2. Equivalently,
the smoothed acceptance through times the continuation value after (Figure 1).

Figure 1. D-PAL uses the full target distribution for position weighting and, with KL or Rényi, for local fitting. Top: D-PACE weights by confidence and fits one token with CE; faded bars are target mass CE ignores. Bottom: D-PAL weights by overlap and, with a distributional loss such as KL or Rényi, fits to all of . Each weighted term is the product of the acceptance through (left), the local loss at (center), and the continuation value after (right).
The SGLang DFlash backend proposes the draft argmax at both temperatures, so proposals are never sampled from . Overlap still bounds the acceptance of this proposal.
Proposition 1. (i) Let be the gap between the largest and second-largest target probabilities. If , then . (ii) .
At , the verifier accepts if it matches the target argmax, which (i) guarantees whenever the target margin exceeds . At , SGLang’s target-only verification accepts it with probability (SGLang Contributors [34]), which (ii) bounds from below (proof in Appendix § 8.§ 2).
Log acceptance loss and beyond
D-PAL[LA]. Equation (3) pairs each weight with , so gradient ascent on matches gradient descent on the log-acceptance (LA) loss , which gives D-PAL[LA]. For a point-mass target, , so LA becomes CE, as LK Losses notes (Samarin et al. [33]), and D-PAL[LA] reduces to D-PACE. At , gradient descent on D-PAL[LA] follows gradient ascent on the overlap-based prefix-sum surrogate (Appendix § 8.§ 3); as in D-PACE, we keep the floor as a regularizer.
However, LA is not differentiable where a draft probability equals the corresponding target probability. Its gradient is the TV gradient scaled by . With respect to the draft logits, the TV gradient is with (Samarin et al. [33]). It does not vanish as approaches , and changes sign where crosses . On the other hand, KL and Rényi logit gradients vanish at (Appendix § 8.§ 1). Figure 2 shows a binary example. Because is detached, any local loss can replace LA without changing the weight formula; in the following, we consider D-PAL[KL] and D-PAL[Rényi] as smooth alternatives to D-PAL[LA].

Figure 2. Near , LA’s slope stays near and flips sign, while KL and Rényi-1/2 flatten out. Binary example with target and draft ; slopes are with respect to , and all three losses are zero at .
D-PAL[KL]. Cross-entropy, the local loss of D-PACE, is smooth but fits one sampled token only. If , then , and the logit gradient has mean . Up to the constant , CE is thus a one-sample estimate of forward KL, with label noise at every position. Our labels are sampled at positive temperature, so CE also fits a sharper target than the distribution that sets acceptance. D-PAL[KL] fits this forward KL directly instead.
D-PAL[R ényi]. KL fits the full distribution, but it bounds overlap only from below and can be arbitrarily large while overlap is close to one. D-PAL[Rényi] instead replaces the minimum in overlap with a geometric mean, the Bhattacharyya coefficient , whose scaled negative logarithm is the Rényi divergence of order [30]:
Its logit gradient is with (Appendix § 8.§ 1), which varies continuously with the target–draft mismatch. The Bhattacharyya coefficient bounds overlap from above and below.
Lemma 1 (Overlap bounds). For categorical distributions , satisfies
This bound is the classical inequality between TV and the Bhattacharyya coefficient [12] (proof in Appendix § 8.§ 2). Taking negative logarithms gives . D-PAL[Rényi] minimizes the smooth quantity that controls both bounds, so driving it to zero drives LA to zero. This lower bound on overlap is never below the Bretagnolle–Huber bound from KL [39, 5] (Appendix § 8.§ 1).
D-PAL Algorithm
Algorithm 1 shows how to compute the D-PAL loss. Any of the previously discussed losses (LA, KL, Rényi, etc.) can be plugged in for .
Algorithm 1: D-PAL loss computation
Input: target , draft , valid mask , smoothing , and chosen local loss .
for each nonempty valid block :
Select proposals with in order; index them by .
for all
for : ▷ Smoothed prefix product
for : ▷ Position weights
Detach .
return Mean of within each sequence, then across sequences.
Experiments
Experimental setup
Models and training. We train DFlash drafters in SpecForge with frozen Qwen3-4B, Qwen3-8B, and Llama-3.1-8B-Instruct targets [7, 22, 42, 13]. Within each comparison, methods share target-generated data, optimizer, schedule, and decoding (Appendix § 7). Table 1 uses the Qwen3-4B-Instruct-100K dataset released by the DFlash authors. Following D-PACE, we use three-layer drafts and block size [7, 41]. Request grouping uses these same checkpoints. For the ablations and other targets, we use SpecForge to sample a 36K-example ShareGPT subset; ablations use Qwen3-4B and unless varied.
Evaluation. We evaluate on GSM8K [10] and MATH-500 [15, 27] for math, HumanEval and MBPP [8, 3] for code, and first-turn MT-Bench (Zheng et al., 2023) and Alpaca [37] for chat. We run the six-benchmark evaluations with SGLang DFlash on one NVIDIA L40S, one request at a time, at and . Our two primary evaluation metrics are speedup ratio (SR) (throughput divided by autoregressive throughput) and acceptance length . Six-benchmark means weight benchmarks equally, and every result averages three evaluation seeds. Across seeds, the standard deviation of each mean is 0 at (greedy decoding) and at most 0.02 at .
Objectives. DFlash and D-PACE are the baselines. All D-PAL objectives share the weight rule of Equation (6) and differ only in the local loss. We compare D-PAL[LA], D-PAL[KL], and D-PAL[Rényi] with three losses that contain an explicit TV term: D-PAL[TV], D-PAL[CE+TV], and D-PAL[KL+TV] (see Appendix § 7.§ 2 for details). Table 1 uses for D-PAL, the best Qwen3-4B value in Table 7, and for D-PACE, the default that D-PACE reports as stable over [41]. Ablations use unless varied.
| Qwen3-4B, , | Qwen3-4B, , | Qwen3-4B, , | Qwen3-4B, , | Llama-3.1-8B, , | Llama-3.1-8B, , | |
| 0 | 1.51 | 1.50 | 1.03 | 1.03 | 1.60 | 1.51 |
| 0.3 | 3.51 | 3.37 | 3.59 | 3.43 | 3.29 | 2.69 |
| 0.5 | 3.48 | 3.34 | 3.58 | 3.41 | 3.31 | 2.73 |
| 0.7 | 3.43 | 3.30 | 3.50 | 3.35 | 3.23 | 2.65 |
| 0.9 | 3.37 | 3.25 | 3.36 | 3.24 | 3.09 | 2.58 |
Table 7. Without the floor, D-PAL[Rényi] collapses; works best. Mean over six benchmarks as the floor varies.
D-PAL beats D-PACE
Table 1 compares all objectives on Qwen3-4B. D-PAL[Rényi] improves on D-PACE at both temperatures: mean rises from 4.50 to 4.56 at and from 4.20 to 4.28 at , and mean SR from 3.11 to 3.14 and from 2.79 to 2.85. The gain holds across benchmarks, since D-PAL[Rényi] has the higher on all 12 benchmark–temperature pairs and the higher SR on 10. The other two D-PAL objectives, D-PAL[LA] and D-PAL[KL], also beat D-PACE on both metrics at both temperatures. When we batch requests with the same checkpoints, D-PAL[Rényi] leads in in all ten settings and in throughput in nine (Table 2). In the separate 36K ablations at D-PACE’s own , D-PAL[Rényi] still leads D-PACE by 0.08 in mean at and 0.05 at (Table 3 and Table 7).
| Method | TPS | TPS | TPS | TPS | TPS | |||||
| DFlash | 310.24 | 4.82 | 423.17 | 5.48 | 607.93 | 5.46 | 990.60 | 5.49 | 1628.15 | 5.49 |
| D-PACE | 320.13 | 6.06 | 466.27 | 6.06 | 669.64 | 6.04 | 1120.19 | 6.06 | 1714.12 | 6.06 |
| D-PAL[Rényi] | 339.73 | 6.10 | 477.46 | 6.13 | 686.75 | 6.10 | 1094.14 | 6.13 | 1797.82 | 6.11 |
| DFlash | 252.70 | 4.82 | 342.24 | 4.86 | 487.59 | 4.83 | 788.09 | 4.82 | 1155.27 | 4.83 |
| D-PACE | 274.51 | 5.21 | 362.02 | 5.25 | 515.48 | 5.23 | 786.06 | 5.29 | 1215.91 | 5.27 |
| D-PAL[Rényi] | 276.68 | 5.30 | 370.04 | 5.32 | 530.42 | 5.33 | 826.42 | 5.35 | 1263.31 | 5.30 |
Table 2. D-PAL[Rényi] has the highest at every batch size and the highest throughput in 9 of 10 settings. 128 MATH-500 prompts on Qwen3-4B with the 100K checkpoints; is the inference batch size (number of requests) and TPS is output tokens per second. Temperature for the first three rows and for the last three rows.
| Loss | Signal | |||
| CE | Confidence | 0.5 | 3.50 | 3.36 |
| Rényi | Confidence | 0.5 | 3.58 | 3.42 |
| CE | Overlap | 0.3 | 3.50 | 3.36 |
| Rényi | Overlap | 0.3 | 3.59 | 3.43 |
Table 3. Rényi beats CE under both confidence and overlap weights. Each pair fixes the weight signal and floor and changes only the loss. Confidence with CE is D-PACE; overlap gives D-PAL[CE] and D-PAL[Rényi]. Mean over six benchmarks.
LA is not the best local loss. Under the same D-PAL weights, KL and Rényi reach the same or higher mean than LA at both temperatures (Table 1). KL leads by 0.01 at and Rényi by 0.03 at ; Rényi is our default because it also bounds overlap from above (Lemma 1).
Pure TV and CE+TV fall short. D-PAL[TV] collapses to , consistent with the small TV gradient at initialization that KL Losses reports (Samari et al., 2026). Adding CE restores training, but D-PAL[CE+TV], a DSpark- and H-Spec-style mixture, trails D-PAL[KL] by 0.18 at and 0.14 at and falls below D-PACE. Its fixed TV term keeps the nondifferentiable points at (§ 3.2), and the toy example in Figure 3 (Appendix § 7.§ 2) shows that its CE gradient can raise TV to first order. D-PAL[KL+TV], using the adaptive mixture of losses proposed in LK Losses, shifts weight to KL at low-acceptance positions and stays within 0.04 of D-PAL[LA], D-PAL[KL], and D-PAL[Rényi].
Figure 3. A CE+TV descent step can raise TV. Negative weighted gradients of TV () and CE () at one parameter point of the three-token example, on one shared scale and rotated for display. The two gradients have a negative inner product, and the mixture step raises TV to first order ().
Ablations separate the loss from the weights
Distribution fitting beats cross-entropy. Table 3 changes only the local loss, from CE to Rényi, under each weight signal. The switch raises mean by 0.08 at and 0.06 at under D-PACE’s confidence weights (), and by 0.09 and 0.07 under overlap weights (). With Rényi at the same floor , the two weight signals give similar mean (3.58 for both at ; 3.42 and 3.41 at ; Table 3 and Table 7). We keep overlap as the weight signal because it matches rejection-sampling acceptance, reduces to D-PACE’s weights for a point-mass target, and bounds argmax acceptance (Proposition 1).
D-PAL weights beat fixed decay on baseline losses. In Table 4, we compare DFlash’s static weight decay against D-PAL’s weight schedule across four losses from the literature: CE, used in Chen et al. [7]; LA and KL+TV, used in Samarin et al. [33]; and CE+TV, used in Cheng et al. [9]. At , D-PAL weights raise mean over DFlash’s schedule by 0.26 for CE, 0.29 for LA, 0.33 for KL+TV, and 0.08 for CE+TV. Gains are smaller at but hold for every loss.
| Local loss | Static | D-PAL | Static | D-PAL |
| CE | 3.24 | 3.50 | 3.11 | 3.36 |
| LA | 3.27 | 3.56 | 3.15 | 3.40 |
| CE+TV | 3.40 | 3.48 | 3.27 | 3.34 |
| KL+TV | 3.20 | 3.53 | 3.08 | 3.37 |
Table 4. D-PAL weights beat fixed decay for all losses from baselines. KL+TV follows the adaptive mixture of LK Losses with per-position . Static uses DFlash’s decay ; D-PAL uses (0.5 for CE). Mean over six benchmarks.
D-PAL[Rényi] leads at every block size and depth. Larger blocks help up to : the mean of D-PAL[Rényi] rises from to and levels off at (Table 5). Deeper drafts help both methods, and with five layers D-PAL[Rényi] still leads D-PACE by 0.06 at and 0.07 at $T = 1 ({xref}Table 6`).
| D-PACE | D-PAL[Rényi] | |||
| 8 | 3.41 | 3.28 | 3.48 | 3.34 |
| 12 | 3.51 | 3.36 | 3.57 | 3.42 |
| 16 | 3.50 | 3.36 | 3.58 | 3.41 |
Table 5. D-PAL[Rényi] has the highest at every block size. Mean over six benchmarks; three-layer drafts.
The floor prevents collapse. Without the floor (), mean falls to 1.03–1.60, likely because an early low-overlap position drives every later prefix product toward zero (Table 7). Any floor in {0.3, 0.5} works: the two stay within 0.04 of each other on every configuration.
Gains hold on other targets and inference settings
Gains transfer to 8B targets. D-PAL[Rényi] has the highest mean SR and on both 8B targets at both temperatures (Table 8). The gain is largest on Llama, where D-PAL[Rényi] leads D-PACE by 0.13 at and 0.12 at , while D-PACE falls below Static[CE+TV].
| Method | Metric | Llama-3.1-8B, , | Llama-3.1-8B, , | Qwen3-8B, , | Qwen3-8B, , |
| D-PACE | SR | 2.47 | 1.77 | 2.59 | 2.37 |
| 3.18 | 2.61 | 3.54 | 3.34 | ||
| D-PAL[Rényi] | SR | 2.57 | 1.84 | 2.65 | 2.40 |
| 3.31 | 2.73 | 3.63 | 3.39 |
Table 8. D-PAL[Rényi] leads on both 8B targets in SR and . Means over six benchmarks; three-layer drafts.
Gains hold in thinking mode. With thinking enabled, all three D-PAL losses match or improve D-PACE in mean ; D-PAL[Rényi] leads at and D-PAL[KL] at (Table 9).
| Method | GSM8K | MATH-500 | HumanEval | MBPP | MT-Bench | Alpaca | Mean |
| Temperature | |||||||
| DFlash | 3.64 | 3.53 | 3.92 | 3.55 | 3.12 | 3.04 | 3.47 |
| D-PACE | 3.94 | 3.82 | 4.14 | 3.96 | 3.28 | 3.17 | 3.72 |
| D-PAL[LA] | 3.96 | 3.91 | 4.21 | 3.85 | 3.47 | 3.22 | 3.77 |
| D-PAL[KL] | 3.98 | 3.90 | 4.20 | 3.96 | 3.46 | 3.23 | 3.79 |
| D-PAL[Rényi] | 4.09 | 3.87 | 4.05 | 4.02 | 3.52 | 3.31 | 3.81 |
| Temperature | |||||||
| DFlash | 3.21 | 3.10 | 3.08 | 3.01 | 2.69 | 2.62 | 2.95 |
| D-PACE | 3.42 | 3.34 | 3.29 | 3.20 | 2.84 | 2.73 | 3.14 |
| D-PAL[LA] | 3.44 | 3.35 | 3.29 | 3.18 | 2.86 | 2.73 | 3.14 |
| D-PAL[KL] | 3.47 | 3.38 | 3.33 | 3.23 | 2.87 | 2.79 | 3.18 |
| D-PAL[Rényi] | 3.47 | 3.37 | 3.31 | 3.21 | 2.87 | 2.77 | 3.17 |
Table 9. With thinking enabled, D-PAL[Rényi] leads mean at and D-PAL[KL] at . DFlash is the reference checkpoint (protocol in Appendix § 7.§ 1).
Related Work
Parallel drafters. Medusa and DFlash propose every block position in parallel ([4, 7]). Their per-position marginals ignore tokens drafted earlier in the block, which LiLiCorr calls the marginals problem ([31]). EAGLE, Hydra, Domino, and DSpark condition later proposals on earlier tokens ([24, 26, 2, 17, 9]). LiLiCorr reranks adjacent top- candidates with a head co-trained with DFlash, DFlash2 adds a predecessor-conditioned selector, and DFlow carries verifier
sad representations between rounds [31, 45] (Inco AI 2026). H-Spec replaces the drafter-side KV cache with a hybrid Mamba–attention drafter trained with CE+TV and fixed decay [19]. These methods change what the drafter conditions on; orthogonally, D-PAL changes the objective that trains the marginals and can be combined with them.
| D-PACE | D-PAL[Rényi] | |||
| Layers | ||||
| 3 | 3.50 | 3.36 | 3.58 | 3.41 |
| 5 | 3.75 | 3.56 | 3.81 | 3.63 |
Table 6. Deeper drafts help both methods; D-PAL[Rényi] leads at both depths. Mean over six benchmarks.
Acceptance-aware position weighting. Several methods adapt position weights to acceptance. D-PACE weights CE by a surrogate of expected length built from draft confidence [41], following EAGLE-2’s observation that draft confidence tracks acceptance [25]. PARD2 instead uses the product of the target’s probabilities of the preceding recorded tokens [1]. Other methods tie supervision to where the draft first fails: GRIFFIN propagates top- alignment masks [16], Spec-AUF keeps CE through the first predicted failure [43], and Whalen et al. [40] study first-error supervision with an expected-length chain reward. VAT keeps full weight up to the first simulated rejection, restarts the decay there, and trains a verification head [14]. It also finds that weighting by the exact derivative of expected length, computed from confidence, trails D-PACE’s weights. D-PAL weights keep D-PACE’s weighting rule and floor and use overlap, , in place of confidence.
Local losses for acceptance. Other works change the local loss. DistillSpec finds that the best divergence depends on the task and decoding strategy (Zhou et al., 2024), and DART fits target-to-draft KL with fixed decay [28]. Draft-QPD and AdaFlash train on draft-generated contexts with forward and reverse KL or clipped reverse KL [20, 29]. Closer to our setting, Bebop optimizes a normalized expected length built from TV acceptance [23]. LK Losses minimizes the negative log acceptance, the same term as our LA, or an adaptive KL–TV mixture, and notes that the negative log acceptance reduces CE for a point-mass target [33]. It keeps fixed position decay and leaves learnable or data-dependent weighting across draft heads to future work. DSpark and Uno mix CE or distillation with TV terms [9, 32]. D-PAL puts acceptance into the weights, so the loss can be a smooth distributional fit (§ 4).
Conclusion
We derived D-PAL from the expected accepted length in self-speculative decoding by replacing the point-mass confidence from D-PACE [41] with the full-distribution overlap. The derivation gives acceptance-derived position weights with log acceptance as the local loss. Furthermore, we show that other losses, such as KL and Rényi-1/2, match or exceed the derived log acceptance in average acceptance, while enjoying theoretical guarantees and smoothness, and show they can be combined with the detached D-PAL weight schedule. Experiments across six benchmarks show up to 5% longer acceptance lengths and higher throughput than D-PACE, and up to 11% higher than DFlash [7].
Limitations and future work. D-PAL weights assume that proposals within a block do not condition on each other, as in DFlash; drafters such as DSpark or EAGLE would need weights that model this dependence [9, 24, 26]. D-PAL weights could also train the parallel logits of DFlash2 or LiLiCorr [31] (Inco AI 2026).
Reproducibility Statement
Appendix § 7 gives training and evaluation settings. § 2.2–§ 3 and Algorithm 1 specify the objectives, with supporting derivations in Appendices § 8.§ 1 and § 8.§ 3. We provide the training code at https://github.com/Lucas-TY/D-PAL-SpecForge.
AI Use Statement
Generative AI tools assisted with drafting and revising text. Target language models generated the training responses as described in § 4. The authors are responsible for the final manuscript.
Experimental Settings and Loss Definitions
Training and evaluation
Table 1 uses the Qwen3-4B-Instruct-100K dataset released by the DFlash authors, which D-PACE also uses for its main Qwen3-4B experiments [7, 41]. Of its 99,987 rows, 99,171 are valid training sequences. We randomly sample ShareGPT prompts with SpecForge for the roughly 36K-example runs. All training responses in this paper are generated by the frozen target at temperature 0.7; Qwen3 uses non-thinking mode. Methods share data within each comparison (settings in Table 10). Results in Table 1–Table 2 average evaluation seeds 980406, 980407, and 980408. Across evaluation seeds, the standard deviation of each six-benchmark mean is 0 at and at most 0.02 at . Every configuration uses training seed 42.
| Training | |
| Optimizer / epochs | AdamW / 6 |
| Learning rate / schedule | ; cosine decay, 4% warm-up |
| Global batch / seed | 8 (Table 1 and Table 2), 4 (other runs); seed 42 |
| Sequence / anchor limits | 3072 tokens / 512 anchors |
| Default block size | Qwen: ; Llama: |
| Loss temperature | 1 (target and draft) |
| Evaluation | |
| Backend | SGLang 0.5.19 DFLASH |
| Evaluation temperatures | , |
| Generation limit | 2048 new tokens |
Table 10. Training and evaluation settings. Each block includes one anchor token excluded from the loss. Both draft depths use the same three target feature layers.
Compute. We estimate that running all training experiments on NVIDIA H200 GPUs would require approximately 840 GPU-hours, including 138 GPU-hours for the eight 100K runs. We run all evaluations on NVIDIA L40S GPUs.
Target and draft distributions. The frozen target output head maps cached final hidden states to full-vocabulary logits; softmax gives . At each recorded position, the draft predicts from the anchor context and a masked block. It does not see the full prefix available to the target at later positions. Both loss softmaxes use temperature one; we generate labels at temperature 0.7.
Inference protocol. SGLang 0.5.19 DFLASH uses argmax draft proposals at both temperatures. At , verification retains the target-argmax prefix; at , it applies target-only stochastic acceptance and rejection correction, which accepts the deterministic proposal with probability [34]. We disable top- and top- truncation and set SGLang’s single-token and accumulated acceptance thresholds to one. We measure through verification; overlap serves as a training proxy.
The grouped-request experiment in Table 2 uses 128 fixed MATH-500 prompts and one L40S. All inference batch sizes, including , use grouped-request inference, so its throughput is not directly comparable with the single-request results in Table 1. It evaluates the 100K checkpoints of DFlash, D-PACE, and D-PAL[Rényi] from Table 1.
Evaluation subsets. We use 128 examples each from GSM8K, MATH-500, and MBPP, 164 from HumanEval, and 80 first-turn MT-Bench prompts, with subset seed 42. Alpaca uses a fixed list of 128 single-turn prompts, supplied as alpaca_128.jsonl. We use the same prompts across methods and temperatures within each target.
Thinking-mode comparison. Table 9 uses a separate set of three-layer, checkpoints on the same six benchmark subsets, with thinking enabled. The D-PACE and D-PAL checkpoints use . The context limit is 16384 tokens and the generation limit is 8192; both reasoning and final-answer tokens count toward the metrics. Each run uses one L40S and one active request. We retain responses that reach the generation limit.
Local losses and position weights
Table 11 lists the local fitting terms, with recorded target token and training overlap . All D-PAL objectives use the detached weights in Equation (6) and differ only in their local loss. Divergences and overlap use the full vocabulary.
| Objective | Local loss |
| Cross-entropy | |
| D-PAL[LA] | |
| D-PAL[KL] | |
| D-PAL[Rényi] | |
| D-PAL[TV] | |
| D-PAL[CE+TV] | |
| D-PAL[KL+TV] |
Table 11. Local losses compared in this paper. D-PAL[KL+TV] computes for each block and position and detaches it during optimization.
DSpark’s has twice the TV coefficient of our CE+TV loss. D-PAL[KL+TV] computes at each position, whereas LK Losses aggregates acceptance across sequence and batch before setting its hybrid coefficient (Samarin et al., [33]). Static weights use , with for . D-PACE smooths confidence with . D-PAL smooths with in Table 1 and with in the ablations unless varied; D-PAL[CE] uses . We average blocks within sequences, then sequences, without normalizing by the weight sum.
Implementation. The two weight scans cost ; vocabulary sums cost . We compute the Rényi loss in 32-bit floating point (FP32) using log-softmax and logsumexp.
Gradient conflict in TV mixtures. A fixed positive TV coefficient leaves a mixture nondifferentiable, and the auxiliary gradient can oppose the TV gradient when the terms share parameters (Du et al., [11]; Yu et al., [44]). Let and . At differentiable points, a stochastic gradient descent step of size on a TV changes TV by
so the mixed step raises TV to first order when . For target and draft , vary two draft logits and fix the third at zero. With hard-target CE for token 1 and , : descent on the mixed loss raises TV to first order (Figure 3). KL and TV gradients can also oppose, which can affect D-PAL[KL+TV]. For and , and at
: the KL and TV gradients oppose. With detached position weights, Equation (11) applies to the weighted sum of these gradients over positions.
Additional Derivations
Rényi fitting gradient
Fix and let . Using ,
Here is the normalized geometric mean.
Comparison with KL. Rényi divergence is nondecreasing in order, so [39]. Bretagnolle–Huber gives [5]. For the same , Equation (10) gives an overlap lower bound at least as high as the KL bound.
Proofs of Proposition 1 and Lemma 1
Proof of Lemma 1. The upper bound follows from . For the lower bound, set and . Both have unit norm, and :
Hence .
Proof of Proposition 1. Write , , , , and .
(i) Suppose , so . Then
which contradicts . (ii) Since , we have .
TV gradients and position weights
TV coefficients. Fix the target model and target-generated training contexts. Differentiating the prefix sum of training overlaps gives
Multiplying this coefficient by gives the relative-contribution form in Equation (3), evaluated at training overlaps.
Gradient of the smoothed prefix sum. For positive , direct differentiation of the smoothed prefix sum gives
At , and gradient descent on D-PAL[LA] follows this direction; for , the factor replaces , and we treat the floor as a regularizer, as D-PACE does.
References
- [1]Zihao An, Taichi Liu, Ziqiong Liu, Dong Li, Ruofeng Liu, and Emad Barsoum. PARD-2: Target-aligned parallel draft model for dual-mode speculative decoding. arXiv preprint arXiv:2605.08632, 2026. URL https://arxiv.org/abs/2605.08632.
- [2]Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. Hydra: Sequentially-dependent draft heads for Medusa decoding. arXiv preprint arXiv:2402.05109, 2024. URL https://arxiv.org/abs/2402.05109.
- [3]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. URL https://arxiv.org/abs/2108.07732.
- [4]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024. URL https://arxiv.org/abs/2401.10774.
- [5]Clément L. Canonne. A short note on an inequality between KL and TV. arXiv preprint arXiv:2202.07198, 2023. URL https://arxiv.org/abs/2202.07198.
- [6]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023. URL https://arxiv.org/abs/2302.01318.
- [7]Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block diffusion for flash speculative decoding. arXiv preprint arXiv:2602.06036, 2026. URL https://arxiv.org/abs/2602.06036.
- [8]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. URL https://arxiv.org/abs/2107.03374.
- [9]Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaq i Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishui Zhang, Dongyan Zhao, and Wenfeng Liang. DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv preprint arXiv:2607.05147, 2026. URL https://arxiv.org/abs/2607.05147.
- [10]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168.
- [11]Yunshu Du, Wojciech M. Czarnecki, Siddhant M. Jayakumar, Mehrdad Farajtbar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:1812.02224, 2018. URL https://arxiv.org/abs/1812.02224.
- [12]Christopher A. Fuchs and Jeroen van de Graaf. Cryptographic distinguishability measures for quantum-mechanical states. IEEE Transactions on Information Theory, 45(4):1216–1227, 1999. doi: 10.1109/18.761271. URL https://arxiv.org/abs/quant-ph/9712042.
- [13]Aaron Grattafiori, Abhinanyu Dubey, Abhinav Jauhri, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783.
- [14]Geonmo Gu, Byeongho Heo, HeeJae Jun, Yoohoon Kang, Sangmin Lee, Sangdoo Yun, and Dongyoon Han. Verification-aware training for speculative decoding. arXiv preprint arXiv:2608.30135, 2026. URL https://arxiv.org/abs/2608.30135.
- [15]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2021. URL https://arxiv.org/abs/2103.03874.
- [16]Shijing Hu, Jingyang Li, Xingyu Xie, Zhihui Lu, Kim-Chuan Toh, and Pan Zhou. GRIFFIN: Effective token alignment for faster speculative decoding. arXiv preprint arXiv:2502.11018, 2025. URL https://arxiv.org/abs/2502.11018.
- [17]Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, Hanlin Xu, and Linfeng Zhang. Domino: Decoupling causal modeling from autoregressive drafting in speculative decoding. arXiv preprint arXiv:2605.29707, 2026. URL https://arxiv.org/abs/2605.29707.
- [19]Weifan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn, Reed Meyerson, Zhenting Qi, Tianyu Wu, Eldar Kurtic, Minlan Yu, and Alexandre Marques. H-Spec: Parallel speculative decoding without a drafter-side KV cache. arXiv preprint arXiv:2609.24197, 2026. URL https://arxiv.org/abs/2609.24197.
- [20]Haodi Lei, Yafu Li, Haoran Zhang, Shunkai Zhang, Qianjia Cheng, Xiaoye Qu, Ganqu Cui, Bowen Zhou, Ning Ding, Yun Luo, and Yu Cheng. Draft-OPD: On-policy distillation for speculative draft models. arXiv preprint arXiv:2605.29343, 2026. URL https://arxiv.org/abs/2605.29343.
- [21]Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, 2023. URL https://arxiv.org/abs/2211.17192.
- [22]Shenggui Li, Chao Wang, Yikai Zhu, Yubo Wang, Fan Yin, Shuai Shi, Yefei Chen, Xiaomin Dong, Qiaoling Chen, Jin Pan, Ji Li, Laixin Xie, Yineng Zhang, Lei Yu, Yonggang Wen, Ivor Tsang, and Tianwei Zhang. SpecForge: A flexible and efficient open-source training framework for speculative decoding. arXiv preprint arXiv:2603.18567, 2026a. URL https://arxiv.org/abs/2603.18567.
- [23]Yucheng Li, Huiqiang Jiang, Yang Xu, Jianxin Yang, Yi Zhang, et al. Breaking entropy bounds: Accelerating RL training via MTP with rejection sampling. arXiv preprint arXiv:2606.12370, 2026b. URL https://arxiv.org/abs/2606.12370.
- [24]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024a. URL https://arxiv.org/abs/2401.15077.
- [25]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858, 2024b. URL https://arxiv.org/abs/2406.16858.
- [26]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840, 2025. URL https://arxiv.org/abs/2503.01840.
- [27]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2305.20050.
- [28]Fuliang Liu, Xue Li, Ketai Zhao, Yinxi Gao, Ziyan Zhou, Zhonghui Zhang, Zhibin Wang, Wanchun Dou, Sheng Zhong, and Chen Tian. DART: Diffusion-inspired speculative decoding for fast LLM inference. arXiv preprint arXiv:2601.19278, 2026. URL https://arxiv.org/abs/2601.19278.
- [29]Yu-Yang Qian, Hao-Cong Wu, Chen Chen, Jiacheng Sun, Zhenhua Dong, Peng Zhao, and Zhi-Hua Zhou. AdaFlash: Adaptive speculative decoding via on-policy distilled diffusion drafters. arXiv preprint arXiv:2607.19223, 2026. URL https://arxiv.org/abs/2607.19223.
- [30]Alfréd Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, 1961.
- [31]Matan Rusanovsky, Yoav Miron, Roy Uziel, Omer Belhasin, Ran Zilberstein, Maor Ashkenazi, and Michael Elad. LiLiCorr: Lightweight likelihood correlation of parallel drafts for speculative decoding. arXiv preprint arXiv:2608.20530, 2026. URL https://arxiv.org/abs/2608.20530.
- [32]Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham, Jonathan Geuter, Chaitanya Dwivedi, Varad Pimpalkhute, Yash Akhauri, Alexander Moreno, Mikhail Yurochkin, Zhenting Wang, Mostafa Elhoushi, Nolan Dey, Shane Bergsma, Joel Hestness, John Thickstun, Eric Xing, and Zhengzhong Liu. Unlocking lossless speedups in LLMs via discrete diffusion. arXiv preprint arXiv:2609.04010, 2026. URL https://arxiv.org/abs/2609.04010.
- [33]Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skortsov, Filipp Fišin, and Alexander Golubev. LK Losses: Direct acceptance rate optimization for speculative decoding. arXiv preprint arXiv:2602.23881, 2026. URL https://arxiv.org/abs/2602.23881.
- [34]SGLang Contributors. SGLang v0.5.19 DFlash speculative verification. https://github.com/sgl-project/sglang/blob/v0.5.19/python/sglang/srt/speculative/dflash_utils.py, 2026. Source code, lines 775–995.
- [35]SpecForge Contributors. SpecForge unified training recipe catalog. Software documentation, 2026. URL https://github.com/sgl-project/SpecForge/blob/main/examples/configs/README.md. Accessed September 23, 2026.
- [36]Speculators Contributors. Speculators training command reference. Software documentation, 2026. URL https://docs.vllm.ai/projects/speculators/en/stable/cli/train/. Accessed September 23, 2026.
- [37]Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models, 2023. URL https://crfm.stanford.edu/2023/03/13/alpaca.html.
- [38]TorchSpec Contributors. TorchSpec DFlash training guide. Software documentation, 2026. URL https://github.com/lightsеekorg/TorchSpec/blob/main/docs/dflash/README.md. Accessed September 23, 2026.
- [39]Tim van Erven and Peter Harremoës. Rényi divergence and Kullback-Leibler divergence. IEEE Transactions on Information Theory, 60(7):3797–3820, 2014. doi: 10.1109/TIT.2014.2320500. URL https://arxiv.org/abs/1206.2459.
- [40]Lexington Whalen, Yuki Ito, and Ryo Sakamoto. Teaching diffusion to speculate left-to-right. arXiv preprint arXiv:2606.11552, 2026. URL https://arxiv.org/abs/2606.11552.
- [41]Tianyu Wu, Yu Yao, Zhenting Qi, Han Zheng, Zhuohan Wang, Haoran Ma, Lawrence Liao, Himabindu Lakkaraju, Ju Li, and Yilun Du. D-PACE: Dynamic position-aware cross-entropy for parallel speculative drafting. arXiv preprint arXiv:2605.18810, 2026. URL https://arxiv.org/abs/2605.18810.
- [42]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388.
- [43]Tianjian Yang and Meng Li. Spec-AUF: Accept-until-fail training under train-inference misalignment for masked block drafters. arXiv preprint arXiv:2607.01893, 2026. URL https://arxiv.org/abs/2607.01893.
- [44]Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, volume 33, pages 5824–5836, 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/3fe78a8acf5fda99de95303940a2420c-Paper.pdf.arxiv.org/abs/2001.06782
- [45]Yaohjie Zhang, Linfeng Zhang, Bin Cui, and Xupeng Miao. DFlow: Enabling verifier information flow in block diffusion speculative decoding. arXiv preprint arXiv:2609.06498, 2026. URL https://arxiv.org/abs/2609.06498.
