WorksComputer Science
One Call Ahead: Packed Self-Speculative Decoding with Diffusion Drafting
Abstract
Autoregressive (AR) language models generate tokens sequentially, so generating long outputs is costly. Speculative decoding mitigates this by using a lightweight drafter to propose multiple tokens for the target AR model to verify in parallel, while a diffusion-based drafter can produce an entire draft block in one forward pass. In models with AR and diffusion modes, one network both drafts and verifies, but drafting still waits for verification, so the standard schedule needs two sequential model calls per iteration. Existing one-call schedules avoid the wait but incur input size quadratic in the speculative width. Motivated by this efficiency bottleneck, we introduce Packed-SS, a self-speculative decoding schedule that verifies and drafts in a single model call with input size linear in the speculative width, while preserving the target model's distribution. The key idea is to let the diffusion drafter run one call ahead: while verifying the current draft, the model simultaneously constructs the next draft from an older verified prefix and keeps its unresolved suffix for the next iteration. Under a mild monotonicity assumption on the drafter, we further show that Packed-SS requires no more model calls than the standard two-call decoding in the greedy setting. Experiments on Nemotron 3B/8B and SetDLM 1.7B show higher throughput and lower mean wall time than both baselines under greedy decoding.
https://github.com/cyx78/One-Call-Ahead-Packed-Self-Speculative-Decoding-with-Diffusion-Drafting
Introduction
Modern language models (LMs) increasingly generate long sequences of intermediate reasoning before producing a final answer, making inference latency a central bottleneck. Standard autoregressive (AR) models [6] generate one token at a time from left to right. Producing an -length sequence therefore requires at least sequential model calls, where the computation for each new token depends on all previously generated tokens.
Speculative decoding [30, 10, 51] accelerates AR inference by using a lightweight draft model that proposes multiple tokens for the full-size target model to verify in parallel. A rejection-sampling correction step ensures that the resulting sequence follows exactly the target model’s distribution. The target model can thus output several tokens from a single verification call. However, when the drafter is AR, creating a draft block remains sequential. Therefore, speculative decoding shortens the expensive serial chain, but does not eliminate it.
Masked diffusion language models (MDLMs) provide a natural way to parallelize drafting [42, 45, 24, 40] (Ye et al. 2025). By learning to reverse a corruption process that transforms data into noise, an MDLM can predict multiple tokens simultaneously. Therefore, when used as a speculative drafter, it can draft an entire block in one forward pass [17, 31, 11, 55]. Some LMs support both causal AR prediction and parallel masked diffusion prediction in the same network [37] (Fu et al. 2026). They provide a natural form of self-speculative decoding, with the drafter and verifier sharing the same model weights.
However, shared weights do not eliminate the sequential dependency between drafting and verification. The standard self-speculative schedule, referred to as Linear-SS, uses two sequential calls: one AR call verifies the current draft and one diffusion call constructs the next draft from the newly verified prefix. Drafting cannot begin until verification finishes to determine where the verified prefix ends.
Quadratic-SS [37] eliminates this dependency by preparing a continuation for every possible verification outcome. With speculative width , it pre-drafts separately for the cases in which draft tokens are accepted. Whichever outcome occurs, the next draft is already available. This reduces each iteration to one call, but the number of evaluated positions grows quadratically with . Thus, Linear-SS evaluates positions in two calls, whereas Quadratic-SS evaluates in one.
At small batch sizes, a substantial fraction of a call’s latency can come from loading model weights from memory, making it attractive to evaluate more positions per call [8] (Fu et al. 2024). To illustrate this tradeoff, suppose a call evaluating positions costs , where is a fixed overhead, mostly weight loading, and is the marginal cost per position. An iteration costs approximately for Linear-SS and for Quadratic-SS. Verifying and drafting in one call with approximately evaluated positions would cost only per iteration, avoiding both the second-call overhead and the quadratic branching cost.
This raises the central question of this work:
Can we verify and draft in one call, while only evaluating positions linear in the speculative width?
Contributions. We answer affirmatively with Packed-SS, a plug-and-play self-speculative decoding schedule for such models. The idea is simple: the next draft does not need to wait for the newest verified prefix. Instead, we let the diffusion drafter run one call ahead. At iteration , one forward pass simultaneously (i) verifies the draft, and (ii) proposes a new draft block using the prefix available at the beginning of iteration . If verification advances by positions, the first positions of this new draft have already been resolved and are discarded. The remaining suffix becomes the draft for the next iteration. Drafting therefore no longer waits for verification. Figure 1 illustrates the difference between Packed-SS and existing schedules.

Figure 1. Where drafting happens in one call. Outlined boxes are model calls. Linear-SS must finish verifying before it can draft, so an iteration costs two calls. Quadratic-SS fits in one call by drafting a continuation for every place verification might stop, so its input grows quadratically with the speculative width . Packed-SS also uses one call, building a single draft from the previously verified prefix alone. The right column gives total draft and verifier prediction counts per iteration, including predictions that may later be discarded.
The two key observations behind this design are as follows. First, a diffusion drafter predicts tokens independently given its conditioning prefix, so a rejection at one position does not invalidate later draft tokens. Second, draft freshness affects speed rather than correctness: standard acceptance and residual correction remain exact when they use the proposal distribution that generated each draft token. Thus, an older draft may be accepted less often, but does not bias the output.
These observations remove the synchronization between drafting and verification. With an appropriate attention mask, they can be packed into one forward pass. Each iteration thus requires only one model call, while the number of evaluated positions remains linear in .
Our contributions are summarized as follows:
We introduce Packed-SS, which simultaneously verifies the previous draft and constructs the next draft in a single model call, with input size linear in the speculative width, while preserving the target model’s distribution.
We theoretically prove that, assuming that correct draft predictions remain correct as verified context grows, Packed-SS requires no more model calls than Linear-SS to generate any prescribed output length.
We evaluate Packed-SS on Nemotron 3B/8B (Fu et al., 2026) and SetDLM 1.7B [2]. Across the primary greedy evaluations, Packed-SS achieves higher throughput than Linear-SS and Quadratic-SS. Under temperature-1 sampling, it consistently outperforms Linear-SS while remaining competitive with Quadratic-SS.
Background
Notation
We write for the vocabulary and for model parameters. For a sequence we write , and denotes the sequence followed by item . We use to denote the positive part of . A complete list of variables can be found in Appendix § 6.
Preliminaries
Speculative decoding. Given prefix , the drafter proposes draft tokens , drawing each from a drafter distribution , where counts positions after . In one model call, the target model uses and to return target distributions for every . The draft tokens are then verified from left to right, where each draft token is accepted with probability . At the first rejection, the decoder replaces that draft token with a correction token from the residual distribution . If all tokens in a draft are accepted, the target model yields one extra bonus token drawn from . The rejection-sampling correction ensures that the generated text follows the target distribution exactly, so a low-quality drafter costs speed but never quality.
In practice, we use the target model to sample at a certain temperature, with temperature 1 being the model’s original distribution and lower temperature yielding a sharper distribution. At temperature 0, the output distribution mass is on its arg max token. This is also called greedy decoding.
Masked diffusion language models. A masked diffusion language model (MDLM) generates text by repeatedly filling missing token positions, represented by a special mask token Mask [42, 45]. Following the denoising framework of discrete diffusion [29, 3, 9, 38], the model is trained to predict original tokens from sequences in which randomly selected tokens have been replaced by Mask. During inference, let denote the visible tokens and their positions, and the remaining masked positions. One model call predicts, for each position , a distribution that approximates the conditional marginal of the data distribution . Sampling each position independently from its prediction approximates the joint distribution by the product of marginals . This approximation incurs error whenever the tokens in are not conditionally independent given , introducing intrinsic trade-off between sampling efficiency and accuracy [31, 14, 56]. To keep this error small, the decoder fills only a subset of the positions in each call, often those with the most confident predictions [40, 7] (Ben-Hamou et al. 2025), and then repeats the model call with the updated context to predict the remaining positions. Block diffusion applies this procedure within blocks generated from left to right, completing each block before starting the next [1]. A block size of one recovers autoregressive generation, while a single block covering the full sequence recovers full-sequence masked diffusion.
Shared-weight drafting and verification. A model with shared-weight AR and diffusion modes can perform both roles in speculative decoding: its masked mode serves as the drafter , proposing multiple tokens after the verified prefix in parallel, while its causal mode serves as the verifier . When these modes differ only in their attention masks, Packed-SS can combine them in one forward pass using a custom attention mask [39, 46] (Fu et al. 2024). Nemotron-Labs-Diffusion supports both modes through joint training on autoregressive and diffusion objectives (Fu et al., 2026). SetDLM also supports this construction through its any-order formulation: single-position conditionals in left-to-right order provide AR verification, while parallel predictions of unresolved positions provide diffusion drafting [2]. See Appendix § 9 for model-specific input layouts and attention-mask constructions.
Related work
Sequential draft–verify decoding. Most speculative decoding methods improve drafting within a sequential draft–verify schedule [30, 10]. Self-speculative methods reuse the target network: Draft & Verify [54] skips selected layers without retraining, whereas LayerSkip [19] trains intermediate layers for early-exit drafting. EAGLE [31] (Li et al. 2024b) variants instead train auxiliary drafters using target features. Parallel drafting reduces sequential work, from blockwise parallel decoding [47] and SpecDec [51] to diffusion-based drafters such as SpecDiff [17] and DiffuSpec [31]. DEER [16] aligns prefix-conditioned drafts with an AR teacher, while DFlash [11] uses a shallow block-diffusion drafter conditioned on target features. DSpark [15] adds confidence-based verification-length scheduling, while DFlow [55] reuses verifier features from rejected suffixes to improve subsequent drafts.
Other speculative decoding schedules. SpecInfer [39] and Medusa [8] use tree attention to verify multiple candidate continuations, while EAGLE-2 (Li et al., 2024a) adaptively constructs draft trees using confidence estimates. Tree-structured verification covers more candidate continuations, but does not by itself remove the dependency between drafting and verification. Jacobi decoding [44] takes a different approach, updating multiple tentative positions in parallel through fixed-point iteration. Lookahead Decoding (Fu et al., 2024) turns these trajectories into cached -gram candidates and combines a lookahead branch with a verification branch in one forward pass. For diffusion decoding, SSD [23] and FreeDaVe (Wu and Zhang, 2025) batch multiple speculative decoding states, with FreeDaVe also reusing accepted-state predictions for subsequent drafting. Within shared-weight models, the quadratic schedules in Samragh et al. (2025), TiDAR [37], and Nemotron (Fu et al., 2026) prepare continuations at every possible acceptance boundary, combining drafting and verification at the cost of evaluated positions for speculative width . Apple’s linear decoding [43] and Introspective DLMs (Yu et al., 2026) discard continuations conditioned on tentative tokens after rejection, which frequently fails to simultaneously yield a draft. Speculative Jacobi Decoding [48] reuses predictions after rejection, but its proposals also condition on tentative tokens.
Our Method
In this section, we first present the high-level description of Packed-SS. We then describe the procedure under greedy decoding in § 3.1, where the main idea is clearest. § 3.2 describes the general procedure.
At a high level, iteration of Packed-SS reads the verified prefix and the current draft , and performs verification and drafting simultaneously and independently in one model call. The verifier accepts tokens from the standard speculative decoding process (including one correction or bonus token). These tokens are extended to to form the next prefix . For speculative width , the drafter reads only and generates a new proposal for the next positions. The first of those positions have just been filled by the verifier, so their proposal tokens are discarded, and the rest become the draft that iteration verifies. See Figure 2 for an illustration of one iteration under greedy decoding.

Figure 2. One iteration of greedy Packed-SS. Columns are token positions. The old draft token , the verifier prediction , and the new proposal token are in the same position. A single forward pass takes the verified prefix and draft as input and produces the two middle rows: the AR verifier’s next-token predictions , conditioned on and , and the diffusion drafter’s proposal , conditioned on alone. Here, one matching draft token () and the correction token yield output tokens. Proposals at these two positions are discarded, leaving as the next draft.
Packed self-speculative procedure under greedy decoding
We now state the procedure under greedy decoding, and summarize it in Algorithm 1.
Initialization. Given a prompt , start with the prefix and draft .
Iteration . Given the prefix and draft following , iteration evaluates the verifier and drafter in one forward pass.
For verification, the AR mode predicts the token at each position of the previous draft plus one position after it, conditioned on the prefix and the draft tokens before that position:
Let be the number of leading draft tokens that match the verifier’s predictions:
Verification yields the accepted draft prefix followed by , which corrects the first mismatch or supplies a bonus after full acceptance. Thus, a total of tokens are accepted.
In the same call, the diffusion mode drafts the positions that follow the previous prefix :
It is important to note that the draft tokens are generated conditioned on the previous prefix , rather than in Linear-SS the new prefix . This distinction allows drafting and verification to run in parallel while Linear-SS needs to be evaluated sequentially. On nonterminal iterations, the suffix of after its first tokens becomes the new draft:
Algorithm 1 Packed-SS with LTR greedy decoding
Require: prompt P, speculative width W, output length limit L, verifier pθ, drafter qθ
1: P₀ ← P, D₀ ← ∅, t ← 1
2: while |Pₜ₋₁| − |P| < L and (t = 1 or the last token of Pₜ₋₁ is not EOS) do
In one model call:
3: vₜ⁽ⁱ⁾ ← arg maxₓ pθ(x | Pₜ₋₁, dₜ₋₁⁽<ⁱ⁾) for i = 1, . . . , |Dₜ₋₁| + 1 ▷ Verifier predictions
4: zₜ⁽ⁱ⁾ ← arg maxₓ qθ(x | Pₜ₋₁) for i = 1, . . . , W ▷ Drafter proposals
end
5: hₜ ← max{0 ≤ h ≤ |Dₜ₋₁| : dₜ₋₁⁽ⁱ⁾ = vₜ⁽ⁱ⁾ for all i ≤ h} ▷ Count matches
6: Cₜ ← (dₜ₋₁⁽¹⁾, . . . , dₜ₋₁⁽ʰᵗ⁾, vₜ⁽ʰᵗ⁺¹⁾), Aₜ ← hₜ + 1 ▷ Add verifier token
7: a ← min{Aₜ, L + |P| − |Pₜ₋₁|} ∪ {1 ≤ i ≤ Aₜ : Cₜ⁽ⁱ⁾ = EOS} ▷ Ending condition
8: Cₜ ← (Cₜ⁽¹⁾, . . . , Cₜ⁽ᵃ⁾) ▷ Prune by stopping index
9: Pₜ ← (Pₜ₋₁, Cₜ) ▷ Prepare next prefix, append accepted tokens
10: Dₜ ← (zₜ⁽ᵃ⁺¹⁾, . . . , zₜ⁽ᵂ⁾) ▷ Prepare next draft, zₜ⁽ⁱ⁾ is generated in line 4
11: t ← t + 1
12: end while
13: Output: Pₜ₋₁
Termination. Stop at the first accepted end-of-sequence (EOS) token or after new tokens.
Comparison with Linear-SS and Quadratic-SS. Linear-SS alternates between drafting-only and verification-only calls, whereas Packed-SS performs both in each call. By retaining the unresolved suffix of as the next draft, Packed-SS avoids a separate drafting call. This overlap comes at the cost of a shorter draft conditioned on less verified context: on nonterminal iterations of Algorithm 1, Packed-SS retains tokens by equation 4, whereas Linear-SS drafts tokens conditioned on more verified tokens. In addition, both Packed-SS and Quadratic-SS draft and verify simultaneously in one model call. However, Packed-SS evaluates positions per iteration rather than . The trade-off is conditioning context: Quadratic-SS’s selected proposal excludes only the final verifier token from , whereas Packed-SS conditions on , excluding all newly appended tokens. Conceptually, Packed-SS can be viewed as Quadratic-SS with only the branch for zero accepted draft tokens.
Theoretical guarantees. Under greedy decoding and the monotonicity assumption below, Packed-SS requires no more forward passes than Linear-SS to generate a prescribed number of tokens from the same prompt.
Fix a prompt and speculative width , and let denote the target model’s greedy continuation. Let denote the deterministic drafter’s prediction at position , conditioned on the prompt and .
Assumption 3.1 (Monotonicity of draft correctness). For each output position , there exists a threshold such that for every evaluated by either schedule.
Here, indicates that position is never correctly predicted before it is verified. Thus, additional verified context cannot turn a correct prediction into an incorrect one. The thresholds need not increase with : a later position may become correctly predictable with less context than an earlier one.
Theorem 3.2 (Pathwise dominance). Suppose Packed-SS and Linear-SS use the same target model, drafter, and speculative width , and start from a common verified prefix. Under Assumption 3.1, Packed-SS’s verified prefix after forward passes is at least as long as Linear-SS’s after iterations, for every .
Let denote the number of forward passes schedule requires to generate tokens after . Since each Linear-SS iteration uses two forward passes, for every , one has
Since both schedules follow the same greedy continuation, the bound also holds when stopping at the first emitted EOS token. See Appendix § 7.§ 1 for the proof.
General packed self-speculative decoding
We now present Packed-SS for general sampling. Compared with the greedy procedure in Algorithm 1 from § 3.1, the general sampling requires two minor modifications.
Consider iteration . First, we store beyond each draft token the distribution from which it was sampled, . At temperature 0, collapses to a point mass on . So storing the draft token alone is sufficient as in Algorithm 1.
Second, when verifying each draft token , we replace the deterministic match test in Equation (2) with the standard rejection-sampling acceptance rule. With the target conditional at that position and its stored drafter distribution, we accept the draft token with probability . Crucially, the denominator is the stored drafter distribution from which was originally sampled, even though it may have been computed from an older prefix. At the first rejection, we draw a correction from the normalized residual distribution . If the entire draft is accepted, we draw a bonus token from .
We summarize the resulting procedure in Algorithm 2 in Appendix § 7.§ 2. The following theorem, proved in Appendix § 7, states that Packed-SS preserves the distribution of the AR mode. Since greedy decoding is recovered at temperature 0, it also implies the correctness of Algorithm 1.
Theorem 3.3 (Distributional correctness). The continuation generated by Packed-SS after the prompt has the same distribution as sampling from the AR model , stopped at the first EOS or at the output length limit .
Finally, storing the drafter distributions need not require a probability for every vocabulary token at each draft position. With top- ([20]) or top- truncation ([28]), we need to store only the nonzero entries of . For instance, Qwen3 recommends top- for a vocabulary of 151,936 tokens ([41]), and the reference implementation of Llama 3 uses top- by default ([25]).
Experiments
Models and datasets. We evaluate Packed-SS on Nemotron-Labs-Diffusion ((Fu et al., 2026)) and SetDLM ([2]). Nemotron 3B and 8B are evaluated on five datasets: GSM8K ([18]) and MATH-500 ([27]) for mathematical reasoning, HumanEval ([13]) and MBPP ([4]) for code generation, and WritingBench ((Wu et al., 2025)) for generation speed on writing prompts. For SetDLM, we evaluate two separately fine-tuned checkpoints on GSM8K released by ([2]).
Experimental setup. All experiments use batch size one on a single A100 80GB GPU, with Nemotron in BF16 and SetDLM in FP32. Generations are capped at 512 tokens for code and writing and 1,024 tokens for mathematics, to avoid repetitive responses dominating the comparison. Because output lengths can differ across schedules, our primary efficiency metric is aggregate token throughput, which decomposes as
Tokens per NFE (Tokens/NFE) and decoder time per NFE (Time/NFE) are recorded to separate token yield from cost per NFE. Detailed setup and results are provided in Appendices § 10 and § 11.
Greedy decoding
Nemotron. Linear-SS and Packed-SS use diffusion block size , while Quadratic-SS uses . Each queried block includes the final token of the verified prefix, so the effective speculative width is . This gives Packed-SS and Quadratic-SS comparable maximum input sizes of 64 and 72 token rows, respectively.
These block sizes also give the highest throughput among the settings tested in the small-scale experiment in Appendix § 11.§ 1.
Results are summarized in Figure 3, with numerical results in Table 6. Packed-SS achieves the highest measured throughput in all ten Nemotron settings. Compared with Linear-SS, throughput improves by 3.5–9.3%, with all ten 95% confidence intervals for the throughput ratio above one. The improvement comes from a 13.4–20.3% increase in tokens per NFE, which more than offsets the 8.9–10.1% increase in decoder time per NFE. Compared with Quadratic-SS, throughput improves by 3.4–32.3% at the evaluated block sizes. Accuracy differences between schedules are observed, but they do not provide a direct comparison of decoding quality because they arise from finite-precision effects.

Figure 3. Greedy decoding efficiency and accuracy. Efficiency metrics are ratios to Linear-SS; accuracy differences are in percentage points. Error bars show 95% paired bootstrap confidence intervals. Native is the released non-speculative SetDLM diffusion decoder, shown for reference. WritingBench measures speed only.
Table 6: Nemotron absolute measurements at temperature 0. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each model and task.
| Task | Method | Tokens/s | NFE | Time/NFE (ms) | Accuracy (%) | Tokens | Wall (s) | NFE |
| Nemotron 3B | ||||||||
| MBPP | Linear-SS | 173.46 | 3.757 | 21.658 | 74.87 | 65.8 | 0.379 | 17.50 |
| Quadratic-SS | 164.38 | 4.288 | 26.089 | 74.34 | 65.9 | 0.401 | 15.36 | |
| Packed-SS | 181.10 | 4.312 | 23.812 | 74.60 | 67.0 | 0.370 | 15.53 | |
| HumanEval | Linear-SS | 237.45 | 5.158 | 21.722 | 79.27 | 90.5 | 0.381 | 17.55 |
| Quadratic-SS | 211.48 | 5.536 | 26.177 | 79.88 | 91.1 | 0.431 | 16.46 | |
| Packed-SS | 249.75 | 5.971 | 23.908 | 78.05 | 90.0 | 0.361 | 15.08 | |
| GSM8K | Linear-SS | 190.05 | 4.142 | 21.795 | 88.40 | 177.1 | 0.932 | 42.76 |
| Quadratic-SS | 180.93 | 4.750 | 26.253 | 88.25 | 178.9 | 0.989 | 37.66 | |
| Packed-SS | 201.55 | 4.812 | 23.873 | 88.10 | 179.1 | 0.888 | 37.22 | |
| MATH-500 | Linear-SS | 222.40 | 4.928 | 22.160 | 55.40 | 489.0 | 2.199 | 99.23 |
| Quadratic-SS | 193.10 | 5.229 | 27.081 | 58.20 | 488.5 | 2.530 | 93.41 | |
| Packed-SS | 230.08 | 5.586 | 24.280 | 58.40 | 492.9 | 2.142 | 88.24 | |
| WritingBench | Linear-SS | 84.56 | 1.864 | 22.038 | — | 502.1 | 5.938 | 269.43 |
| Quadratic-SS | 89.39 | 2.468 | 27.613 | — | 502.0 | 5.616 | 203.39 | |
| Packed-SS | 92.44 | 2.242 | 24.253 | — | 502.9 | 5.441 | 224.34 | |
| Nemotron 8B | ||||||||
| MBPP | Linear-SS | 151.96 | 4.421 | 29.095 | 85.45 | 56.9 | 0.375 | 12.87 |
| Quadratic-SS | 134.86 | 4.631 | 34.335 | 85.19 | 59.3 | 0.440 | 12.81 | |
| Packed-SS | 158.00 | 5.041 | 31.902 | 85.19 | 58.3 | 0.369 | 11.56 | |
| HumanEval | Linear-SS | 195.71 | 5.695 | 29.099 | 80.49 | 94.2 | 0.481 | 16.54 |
| Quadratic-SS | 155.30 | 5.335 | 34.353 | 82.32 | 91.5 | 0.589 | 17.15 | |
| Packed-SS | 205.46 | 6.546 | 31.859 | 81.71 | 92.8 | 0.452 | 14.18 | |
| GSM8K | Linear-SS | 127.39 | 3.744 | 29.389 | 90.52 | 313.2 | 2.459 | 83.66 |
| Quadratic-SS | 122.93 | 4.319 | 35.134 | 91.21 | 313.9 | 2.553 | 72.68 | |
| Packed-SS | 134.29 | 4.300 | 32.021 | 90.67 | 318.0 | 2.368 | 73.94 | |
| MATH-500 | Linear-SS | 147.42 | 4.281 | 29.037 | 60.40 | 640.4 | 4.344 | 149.61 |
| Quadratic-SS | 133.78 | 4.698 | 35.116 | 60.20 | 630.7 | 4.715 | 134.26 | |
| Packed-SS | 155.61 | 4.922 | 31.632 | 61.80 | 634.7 | 4.079 | 128.94 | |
| WritingBench | Linear-SS | 66.12 | 1.898 | 28.701 | — | 501.7 | 7.587 | 264.35 |
| Quadratic-SS | 67.59 | 2.476 | 36.633 | — | 501.5 | 7.420 | 202.55 | |
| Packed-SS | 71.46 | 2.255 | 31.559 | — | 502.6 | 7.033 | 222.87 |
Table 6. Nemotron absolute measurements at temperature 0. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each model and task.
SetDLM. The checkpoints with training horizons and are evaluated at speculative widths and , respectively. All three speculative schedules use the same width within each checkpoint.
Packed-SS improves throughput over Linear-SS by 6.0% at and 1.5% at , as in Figure 3 and Table 8. The additional token yield outweighs the higher cost per NFE: tokens per NFE increase by 8.5–8.9%, while decoder time per NFE increases by 2.7–7.0%. Packed-SS also has higher throughput than Quadratic-SS in both settings. The improvements extend to confidence-based verification order; details are provided in Appendix § 11.§ 5.
| Window | Method | Tokens/s | Tokens/NFE | Time/NFE (ms) | Accuracy (%) | Tokens | Wall (s) | NFE |
| Linear-SS | 94.93 | 2.610 | 27.496 | 68.39 | 257.2 | 2.709 | 98.53 | |
| Quadratic-SS | 86.25 | 3.414 | 39.583 | 68.39 | 257.1 | 2.981 | 75.31 | |
| Packed-SS | 100.60 | 2.842 | 28.247 | 68.39 | 257.2 | 2.556 | 90.49 | |
| Native | 77.21 | 2.119 | 27.442 | 59.67 | 252.4 | 3.269 | 119.11 | |
| Linear-SS | 98.85 | 2.792 | 28.243 | 66.57 | 259.9 | 2.629 | 93.09 | |
| Quadratic-SS | 41.60 | 4.074 | 97.934 | 66.57 | 260.4 | 6.260 | 63.92 | |
| Packed-SS | 100.30 | 3.030 | 30.209 | 66.57 | 259.9 | 2.591 | 85.77 | |
| Native | 80.59 | 2.220 | 27.552 | 59.51 | 253.6 | 3.147 | 114.21 |
Table 8. SetDLM absolute measurements on GSM8K. LTR verification and native decoding on all 1,319 test questions. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each window.
We also test the native SetDLM decoding schedule, which is diffusion decoding instead of speculative decoding. At and , Packed-SS achieves higher throughput than Native, with accuracy higher by 8.72 and 7.05 percentage points, respectively.
Decoding at temperature 1
We further evaluate the sampling procedure of § 3.2 at temperature 1, where draft tokens are verified probabilistically. We use the same five datasets and both Nemotron scales as in greedy decoding, with block size for Linear-SS and Packed-SS and for Quadratic-SS.
Scope and baselines. Linear-SS with speculative sampling is the standard stochastic setting ([30, 10]). However, prior empirical evaluations of Quadratic-SS are reported under greedy decoding ([37, 43] (Fu et al. 2026)). We therefore use Linear-SS as the primary baseline at temperature 1. For completeness, we extend Quadratic-SS to rejection sampling by applying the same probabilistic verification to its selected branch as a secondary comparison. More details are in Appendix § 10.§ 4.
Results. Results are presented in Figure 4, with numerical results in Table 7. Compared with Linear-SS, Packed-SS achieves higher throughput in all ten settings, improving throughput by 2.0–6.3%, with all ten 95% confidence intervals for the throughput ratio above one. The improvement again comes from increased token yield: Packed-SS generates 14.8–21.5% more tokens per NFE, which outweighs the 11.9–15.4% increase in decoder time per NFE.

Figure 4. Efficiency and accuracy at temperature 1. Efficiency metrics are ratios to Linear-SS, accuracy differences are in percentage points. Error bars show 95% paired bootstrap confidence intervals. WritingBench measures speed only.
| Task | Method | Tokens/s | Tokens/NFE | Time/NFE (ms) | Accuracy (%) | Tokens | Wall (s) | NFE |
| Nemotron 3B | ||||||||
| MBPP | Linear-SS | 120.38 | 2.508 | 20.834 | 68.14 | 102.7 | 0.853 | 40.95 |
| Quadratic-SS | 129.01 | 3.328 | 25.800 | 69.69 | 108.7 | 0.843 | 32.67 | |
| Packed-SS | 125.06 | 2.933 | 23.454 | 68.78 | 101.8 | 0.814 | 34.72 | |
| HumanEval | Linear-SS | 169.58 | 3.526 | 20.792 | 66.89 | 96.9 | 0.571 | 27.49 |
| Quadratic-SS | 170.63 | 4.348 | 25.484 | 67.29 | 95.6 | 0.561 | 22.00 | |
| Packed-SS | 173.20 | 4.048 | 23.374 | 68.35 | 97.7 | 0.564 | 24.12 | |
| GSM8K | Linear-SS | 137.38 | 2.977 | 21.670 | 84.15 | 197.7 | 1.439 | 66.41 |
| Quadratic-SS | 140.65 | 3.776 | 26.843 | 84.84 | 199.2 | 1.416 | 52.76 | |
| Packed-SS | 140.07 | 3.428 | 24.471 | 83.47 | 198.4 | 1.416 | 57.88 | |
| MATH-500 | Linear-SS | 150.13 | 3.266 | 21.753 | 53.80 | 518.2 | 3.451 | 158.66 |
| Quadratic-SS | 149.05 | 4.102 | 27.521 | 52.00 | 515.6 | 3.459 | 125.68 | |
| Packed-SS | 155.49 | 3.841 | 24.701 | 52.80 | 523.9 | 3.369 | 136.41 | |
| WritingBench | Linear-SS | 74.96 | 1.575 | 21.012 | — | 498.8 | 6.654 | 316.68 |
| Quadratic-SS | 78.20 | 2.141 | 27.382 | — | 497.0 | 6.355 | 232.09 | |
| Packed-SS | 78.04 | 1.890 | 24.222 | — | 500.1 | 6.409 | 264.58 | |
| Nemotron 8B | ||||||||
| MBPP | Linear-SS | 84.45 | 2.316 | 27.426 | 76.76 | 129.9 | 1.538 | 56.08 |
| Quadratic-SS | 91.22 | 3.062 | 33.572 | 74.98 | 130.4 | 1.429 | 42.58 | |
| Packed-SS | 89.56 | 2.752 | 30.722 | 75.36 | 133.6 | 1.491 | 48.54 | |
| HumanEval | Linear-SS | 107.78 | 2.944 | 27.316 | 70.70 | 120.3 | 1.116 | 40.87 |
| Quadratic-SS | 111.48 | 3.714 | 33.317 | 71.28 | 120.6 | 1.082 | 32.48 | |
| Packed-SS | 114.17 | 3.490 | 30.567 | 71.34 | 118.7 | 1.040 | 34.01 | |
| GSM8K | Linear-SS | 85.66 | 2.341 | 27.324 | 87.34 | 351.0 | 4.098 | 149.97 |
| Quadratic-SS | 91.86 | 3.109 | 33.840 | 86.50 | 348.1 | 3.790 | 111.99 | |
| Packed-SS | 91.08 | 2.794 | 30.678 | 85.29 | 345.0 | 3.787 | 123.46 | |
| MATH-500 | Linear-SS | 90.91 | 2.576 | 28.338 | 54.20 | 657.3 | 7.230 | 255.14 |
| Quadratic-SS | 95.73 | 3.391 | 35.417 | 55.20 | 658.5 | 6.879 | 194.22 | |
| Packed-SS | 95.47 | 3.047 | 31.919 | 54.20 | 664.6 | 6.962 | 218.10 | |
| WritingBench | Linear-SS | 52.63 | 1.433 | 27.225 | — | 500.9 | 9.518 | 349.59 |
| Quadratic-SS | 53.34 | 1.940 | 36.370 | — | 496.5 | 9.308 | 255.94 | |
| Packed-SS | 55.39 | 1.741 | 31.424 | — | 498.8 | 9.006 | 286.58 |
Table 7. Nemotron absolute measurements at temperature 1. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each model and task.
Compared with Quadratic-SS, Packed-SS shows comparable performance in terms of throughput. Its throughput point estimate is higher in four settings and lower in six. The paired 95% confidence intervals lie above one in three settings, below one in two, and include one in the remaining five.
Discussion and Limitations
We presented Packed-SS, which overlaps drafting and verification in one forward pass per iteration while evaluating only positions. Under greedy decoding and a monotonicity assumption, we showed that Packed-SS requires no more forward passes than Linear-SS to generate a prescribed number of tokens from the same prompt at the same width. Experiments across multiple tasks and model families demonstrate that Packed-SS can improve throughput over existing self-speculative decoding schedules.
Several limitations and directions for future work remain. Our single-call implementation requires shared-weight models supporting parallel drafting and autoregressive verification. Combining this schedule with drafters based on intermediate-layer representations remains unexplored. The current efficiency analysis relies on an idealized assumption; extending it to weaker conditions and stochastic sampling would broaden its applicability. Finally, preparing only a subset of the continuation branches used by Quadratic-SS could provide intermediate designs between Packed-SS and Quadratic-SS, potentially improving upon the current algorithm.
Acknowledgements
C. Cai is supported by the NSF CAREER Award CCF-2541600 and the NSF grant DMS-2515333.
Notation
Table 1 collects the symbols used in Section 3. In left-to-right (LTR) decoding, is indexed after , while and are indexed after . In the any-order procedure, position indices are fixed output positions after the prompt.
| Sizes and indices | |
| decoding iteration, starting at 1 | |
| token position: relative in LTR, fixed in any-order decoding | |
| rank in the verification order (any-order setting) | |
| speculative width: the fixed number of future positions proposed by the LTR drafter, , and the size of the active window in the any-order setting. The schedule and Theorem 3.2 are stated in alone | |
| Nemotron’s block size, : the last prefix token and draft positions | |
| number of unresolved positions in the active window, (any-order setting) | |
| available draft-prefix length in verification order , with (any-order setting) | |
| number of consecutive accepted draft tokens in verification order | |
| number of tokens accepted before applying the stopping rule; for the next-token verifier and for the any-order same-position verifier (Appendix C) | |
| emitted token count after applying the stopping rule, (LTR) | |
| maximum number of generated tokens, excluding the prompt | |
| last output position to generate in any-order decoding; initially , reduced to the position of an accepted EOS | |
| LTR prefix length, including the prompt | |
| Decoder state | |
| prompt; verified prefix (LTR) or prompt followed by accepted position–token pairs (any-order), with | |
| draft available for verification at iteration , with ; indexed after in LTR, with . In any-order decoding, a position–token map on | |
| new proposal: positions after the input prefix in LTR, or positions in in any-order decoding | |
| verifier token at position | |
| active unresolved positions; positions retained for the next draft, (any-order) | |
| verification order used by iteration | |
| generation order of an any-order model | |
| tokens accepted in iteration after applying the stopping rule; tokens in LTR or ordered position–token pairs in any-order decoding | |
| Distributions | |
| vocabulary; shared weights | |
| target token distribution given the prompt, accepted tokens, and earlier draft tokens | |
| drafter conditional at position : relative to its conditioning prefix in LTR, fixed in any-order decoding | |
| fresh drafter conditional , before prefix advancement (LTR) | |
| distribution stored with draft ; after retaining the LTR suffix | |
| verifier conditional at position after (LTR) | |
| Analysis (Theorem 3.2) | |
| correctness threshold for output position ; denotes no correct draft prediction before verification (Assumption 3.1) | |
| target greedy token at output position ; drafter’s prediction from the first verified output tokens | |
| verified output lengths after Linear iterations and Packed forward passes, respectively (Appendix B.1) | |
| verified output length after fresh drafting and verification from verified output tokens, before stopping (Lemma B.1) | |
| total model calls used by schedule to generate tokens after the prompt |
Table 1. Symbols. Sizes and indices, decoder state, and distributions.
Proofs
Pathwise dominance
We prove Theorem 3.2 using two lemmas. The first describes the progress of fresh drafting and verification; the second compares this progress with two forward passes of Packed-SS. The comparisons below concern verified output lengths, excluding the prompt, before applying EOS or length truncation; stopping is handled at the end of the theorem’s proof.
Lemma B.1 (Monotonicity of verified output length). Count verified output tokens after the prompt by , and define
with the minimum of an empty set read as . Then is nondecreasing. Under Assumption 3.1, is the verified output length after freshly drafting tokens from a context of verified output tokens and verifying that draft, for any such context evaluated by either schedule. Thus, counts the newly accepted tokens, including the correction or bonus token.
Proof. We first identify the number of tokens accepted by fresh drafting and verification. Fix a context of verified output tokens. If the draft contains an incorrect proposal, let be its first incorrect position. By Assumption 3.1, this is the first position in with . Verification accepts the earlier proposals and adds the corrected token at , giving a verified output length of . If all proposals are correct, verification accepts them and adds a bonus token at . These two cases give the expression for in Equation (6).
We next prove monotonicity. Take . If , then
It remains to consider . For every position satisfying , the definition of gives
Thus, the first position with cannot occur before . The other term in the minimum defining also satisfies
Both terms are therefore at least , proving .
Lemma B.2 (Verified output length after two forward passes). *Under Assumption 3.1, if a Packed-SS iteration begins with verified output tokens and the next iteration ends with , then . *
Proof. Let be the verified output length after the first forward pass. During this pass, Packed-SS verifies its stored draft and produces new proposals for positions , using the context of length . Every pass accepts at least one token, so .
If , the desired inequality already holds, since the next pass can only increase the verified output length. Otherwise, consider the positions
The new proposals at these positions are correct by the definition of . They also belong to the newly drafted block, because . Since these positions lie beyond the updated prefix, their proposals are retained for the next pass.
The next pass therefore accepts all of these retained proposals. Its correction or bonus token is added at a position at least , so its final verified output length satisfies . This argument also covers an empty range of retained positions, since the next pass still accepts at least one token.
Proof of Theorem 3.2. We first compare the two schedules without EOS or length truncation. Let be the number of verified output tokens after iterations of Linear-SS, and let be the corresponding number after forward passes of Packed-SS. Both counts exclude the prompt, and . A Linear-SS iteration drafts from its current verified prefix and then verifies that draft, so
We show by induction that for every . The claim holds at . Suppose it holds for some . Then
The first inequality follows from Lemma B.2; the second follows from the induction hypothesis and the monotonicity in Lemma B.1. This proves the induction step. The argument includes initialization: the first Packed-SS pass starts with an empty draft and generates one target token, so Lemma B.2 applies from .
Now fix , and let be the first Linear-SS iteration for which . Since each Linear-SS iteration uses two forward passes, its total cost is . The inequality shows that Packed-SS reaches the same output length within those passes. Therefore,
For , neither schedule makes a call.
Finally, both schedules produce the same greedy continuation. If that continuation contains an EOS token, its first occurrence is therefore at the same position for both schedules. Apply the same comparison to that position or the length limit , whichever is reached first. Truncating the final block changes the returned output but does not remove the forward pass that produced it. The same bound thus holds with EOS and length stopping. □
Exact sampling
Algorithm 2 gives the full LTR sampling procedure corresponding to § 3.2. Compared with Algorithm 1, the decoder stores the sampling distribution with each draft token and replaces greedy matching with rejection sampling.
Algorithm 2 Packed-SS with LTR rejection sampling
Require: prompt , speculative width , output length limit , verifier , drafter
while and or the last token of is not EOS) do
In one model call:
for ▷ Verifier distributions
for ▷ Drafter proposals
for ▷ Acceptance tests
▷ Count accepted tokens
▷ Correction or bonus
▷ Add verifier token
▷ Ending condition
▷ Prune by stopping index
▷ Append to prefix
for ▷ Sample retained draft
▷ Prepare next draft
end while
Output:
We now prove Theorem 3.3. First, we show that the acceptance and correction steps produce the correct target distribution for one token. Then, we show that each draft token still follows its stored sampling distribution when it is checked.
Conditioning information. Fix the prompt. Let collect the information used for conditioning immediately before sampling the draft . It includes the verified prefix at the start of call , the draft checked in that call, and the distributions used to sample that draft. It also includes the tokens accepted after applying the stopping rule and the new drafter distributions , computed from . Define
Conditional on , the advance , the updated prefix , and the stored distributions for the draft tokens are fixed. All draft sampling, acceptance tests, and correction draws use independent random numbers.
Write for the uniform random number used to test draft token in call . For , add the earlier draft tokens and the random numbers used to test them:
Conditional on , the preceding acceptance decisions are fixed, so whether verification reaches position is also fixed.
Lemma B.3 (One-token correction using the original sampling distribution). Consider the -th draft token reached by left-to-right verification in call . Let be the verified prefix, including the draft tokens already accepted in this call. Let be the target distribution for the next token. Let be the draft token and the distribution used to sample it. Assume for every token . Accept with probability , and otherwise emit an independent draw from the normalized positive part of . Then, given , the emitted token has distribution .
Proof. Condition on , so that and are fixed. Let denote the emitted token and the event that the draft token is accepted. We compute the probability of emitting a given token through each of the two possible branches.
The acceptance branch contributes
For , this follows by multiplying the proposal probability by the acceptance probability . For , both sides are zero. It follows that the total acceptance probability is
The rejection probability is
When , this is exactly the normalizing constant of the correction distribution. The rejection branch therefore contributes
Adding the two contributions yields
If , then and rejection has probability zero, so the same conclusion holds without a correction draw. Also, with probability one because is sampled from , so the acceptance test never divides by zero. Neither distribution needs to assign positive probability to every token. □
This is the standard speculative-sampling rule ([30, 10]). The next lemma shows that each draft token follows its stored sampling distribution when checked, as required by Lemma B.3.
Lemma B.4 (Distribution of a draft token when checked). For each draft position reached by verification in call , and every token ,
Proof. Condition on . The advance , the positions to retain, and the stored distributions are fixed. Algorithm 2 samples the tokens in independently from these distributions. Thus, has distribution and is independent of the earlier draft tokens given .
By definition, adds only the earlier draft tokens and the random numbers used to test them to . These random numbers are independent of . Conditioning on this additional information therefore leaves the distribution of unchanged, proving Equation (7). Whether verification reaches position is determined by , so the same identity holds at every position that is checked.
Proof of Theorem 3.3. Fix the prompt and consider Algorithm 2 without EOS or length stopping. Let be its -th output token after the prompt. We prove by induction on that
Since , the first call draws directly from , proving the claim for .
Suppose the identity holds for , where . Let be the call that produces . Its position within that call is . Thus, , with denoting the bonus position. Let collect , , , the earlier draft tokens, and the random numbers used to test them:
The first draft tokens have been accepted, so
First suppose , so the algorithm checks a draft token. For fixed and , determines whether the preceding draft tokens are accepted and whether . Hence Lemma B.4 gives . Applying Lemma B.3 with target and proposal yields
If , the algorithm draws a bonus token directly from , so the same equality holds.
Since is determined by , the law of total expectation gives
Multiplying by and applying the induction hypothesis proves the identity for .
Now restore stopping. Let
Using the same random numbers, the stopped algorithm produces the same tokens through position and returns . Taking in the identity above shows that has the AR distribution. Applying the same stopping rule therefore gives the same distribution of returned continuations.
Greedy decoding. For greedy decoding, let each model assign probability one to the argmax token chosen by its fixed tie-breaking rule, and zero to all other tokens. Then Theorem 3.3 gives the greedy correctness of Algorithm 1, including EOS and length stopping.
Any-Order Extension
A later token can be easier to predict than an earlier one. In LTR verification, the first mismatch ends acceptance for that call, so an uncertain early token can prevent later, easier tokens from being accepted. Checking tokens that are more likely to be accepted first may therefore let a call accept more draft tokens.
An any-order AR verifier enables this change: it conditions on tokens earlier in the chosen verification order, even when they occur later in the output sequence. For a permutation , write for the tokens earlier in that order. The model defines
The order specifies the conditionals, so different orders can produce different greedy outputs.
Self-speculation with separate draft and verification passes already appears in Any-Subset Speculative Decoding (ASSD) [26]. ASSD uses the same model to draft tokens in parallel and verify them in a second pass, matching the two-call schedule we call Linear-SS. Prism also describes self-speculative decoding and, separately, choosing the next position by confidence [31 Chapter 5]. Here we extend the packed schedule to such verification orders.
Procedure. In LTR decoding, accepted tokens form a prefix, and all unresolved positions lie after it. With any-order verification, a later position can be accepted while an earlier one remains unresolved. We therefore keep each accepted or draft token associated with its output position, numbered from 1 after the prompt. At the start of iteration , contains the prompt and accepted position–token pairs in acceptance order, and maps positions to available draft tokens. Position superscripts now refer to these fixed output positions rather than offsets after a prefix.
Each iteration starts its active window at the leftmost unresolved position and includes up to consecutive output positions. Positions whose tokens have already been accepted are skipped, leaving the set of unresolved positions to process. Some positions in may have no draft token yet; in particular, in the first iteration.
The order policy then chooses a permutation of using information available before the model call, and this order remains fixed throughout the call. Here, is the output position checked at rank ; changing the order does not move a draft token to a different output position. For example, confidence ordering checks positions in decreasing order of scores computed in an earlier iteration (see Appendix § 10.6 for the scores used in our experiments).
Verification follows , using draft tokens at earlier ranks as context. Write for the positions with available draft tokens, and let be the number of consecutive entries of in this set before the first missing draft token. A same-position verifier predicts a token at each queried output position. Writing for its distribution at position under the chosen order, its prediction at rank is
for . Accept matching draft tokens in this order until the first mismatch or missing draft token, then accept the verifier’s prediction at that position. If all positions have draft tokens and all match, accept those tokens without an extra token outside the window. Thus, if draft tokens match consecutively, the number accepted before applying the stopping rule is .
In the same call as verification, Packed-SS drafts every position in from . After acceptance, predictions at positions that remain unresolved become the next draft; predictions at newly accepted positions are discarded. For Quadratic-SS, the proposal branches follow instead of textual order, and the branch selected after verification supplies the next draft at the remaining unresolved positions.
Algorithm 3 gives the formal procedure for Packed-SS with any-order greedy decoding. Let be the last output position to generate. It starts at the length limit ; accepting EOS at position changes it to . The algorithm fills all remaining unresolved positions up to before returning. For an example with confidence-ordered verification on SetDLM, see Appendix § 11.§ 5.
In Algorithm 3, denotes the output positions with accepted tokens in , excluding the prompt, and is the token stored at position . The restriction removes output pairs beyond , preserving the prompt and the order of the remaining pairs. The completed sequence is returned with the prompt included and its continuation restored to textual order.
Algorithm 3 Packed-SS with any-order greedy decoding
Require: prompt $P$, speculative width $W \ge 1$, output length limit $L > 0$, verifier $p_\theta$, drafter $q_\theta$, order policy
1: $P_0 \leftarrow P$, $D_0 \leftarrow \varnothing$, $E \leftarrow L$, $t \leftarrow 1$
2: while $\{1,\ldots,E\} \nsubseteq \operatorname{dom}(P_{t-1})$ do
3: $\ell \leftarrow \min(\{1,\ldots,E\} \setminus \operatorname{dom}(P_{t-1}))$ ▷ Leftmost unresolved position
4: $U_t \leftarrow \{\ell,\ldots,\min(E,\ell+W-1)\} \setminus \operatorname{dom}(P_{t-1})$, $u_t \leftarrow |U_t|$ ▷ Active positions
5: Choose $\pi_t$ over $U_t$ using the order policy and information available before this call
6: $\kappa_t \leftarrow \max\{0 \le k \le u_t : \pi_t(r) \in \operatorname{dom}(D_{t-1}) \text{ for all } r \le k\}$ ▷ Available draft prefix
In one model call:
7: Compute $v_t^{(\pi_t(r))}$ by {xref}`Equation (8)` for $r=1,\ldots,\min(\kappa_t+1,u_t)$ ▷ Verifier predictions
8: $z_t^{(j)} \leftarrow \arg\max_x q_\theta^{(j)}(x \mid P_{t-1})$ for $j \in U_t$ ▷ Drafter proposals
end
9: $h_t \leftarrow \max\{0 \le h \le \kappa_t : d_{t-1}^{(\pi_t(r))}=v_t^{(\pi_t(r))} \text{ for all } r \le h\}$ ▷ Count matches
10: $A_t \leftarrow \min(h_t+1,u_t)$, $C_t \leftarrow \bigl((\pi_t(r),v_t^{(\pi_t(r))})\bigr)_{r=1}^{A_t}$ ▷ Collect accepted tokens
11: $E \leftarrow \min(\{E\}\cup\{j:(j,\mathrm{EOS})\in C_t\})$ ▷ Ending condition
12: $C_t \leftarrow C_t|_{\le E}$, $P_t \leftarrow P_{t-1}|_{\le E}$ ▷ Discard positions after $E$
13: $P_t \leftarrow (P_t,C_t)$ ▷ Append accepted tokens
14: $J_t \leftarrow \{j\in U_t\setminus\operatorname{dom}(P_t):j\le E\}$ ▷ Retained draft positions
15: $D_t \leftarrow \{j\mapsto z_t^{(j)}:j\in J_t\}$ ▷ Prepare next draft
16: $t \leftarrow t+1$
17: end while
18: Output: $(P,P_{t-1}(1),\ldots,P_{t-1}(E))$ ▷ Textual order
Proposition C.1 (Greedy correctness under the chosen verification orders). Fix a run of Algorithm 3. Consider a serial reference that starts from the same prompt and, in iteration , generates tokens greedily at positions , conditioning each prediction on its own and the tokens generated earlier in that group. The reference uses Algorithm 3’s verifier, tie-breaking rule, and rules for updating the accepted tokens and handling EOS. Then the two procedures return the same output sequence.
Proof of Proposition C.1. We show by induction that the two procedures have the same and after every iteration. The claim holds initially, since both start from the prompt with .
Suppose the claim holds before iteration . Let be the serial reference’s token at position in this iteration. We prove, in verification order, that
At rank 1, both predictions condition only on , so they agree. Now take and suppose agreement holds at all earlier ranks. Since , every satisfies . These earlier draft tokens match their verifier predictions by the definition of . Thus,
Substituting these equalities into Equation (8) gives
This proves agreement at every rank up to , including the final verifier token when .
The two procedures therefore generate the same position–token pairs before applying the stopping rule. They find EOS at the same positions, update identically, and remove the same pairs beyond . The resulting values of agree, completing the induction step.
Each iteration accepts at least one token at a previously unresolved position at or before the updated , while never increases. Hence the number of unresolved positions up to strictly decreases, and both procedures terminate. They return the same tokens in the same output-position order.
Remark C.2 (Sampling under a specified generation order). For a fixed generation order , the argument in Appendix § 7.§ 2 gives samples from . Choose the order before sampling the draft tokens, for example using proposal entropy or maximum proposal probability. Sorting by the probabilities of sampled draft tokens can change their distributions when checked.
Attention Mask Layouts
Packed-SS combines verification of the current draft and generation of the next proposal in one model call. This is achieved by a custom attention mask. Below we show the packed inputs, masks, and cache updates for Nemotron and SetDLM.
Nemotron: sequence packing and padding
Let . Each call after initialization processes the last token of the verified prefix, , followed by draft slots; earlier prefix tokens are read from the cache. The first-call is slightly different and discussed in Appendix § 10.§ 2.
Physical layout. Let be followed by enough masks to occupy slots. The Packed-SS input is
concatenated along the sequence axis. Earlier tokens are stored in KV cache and are not part of the input. Its shape is .
Let be the lower-triangular Boolean matrix including its diagonal, and the all-ones matrix. The following matrix specifies which rows in the current input may attend to each other:
and every row additionally sees the cached prefix . The causal region yields the shifted next-token verifier predictions (Fu et al., 2026). The proposal region contains another copy of the last prefix token, whose output is unused, and mask queries, whose outputs are the entries of . Padded verifier rows are evaluated but cannot extend acceptance, since verification stops at the first absent draft token, represented physically by a mask.
Cache update. After each call, the cache contains the verified prefix except its last token, which is processed in the next call. Keys and values from the proposal region and unaccepted verifier inputs are discarded.
SetDLM: position-aligned queries and ordered context
Duplicate masks. SetDLM predicts a token at the position of its input mask. Its training construction represents each position twice: once with the token and once with a mask, both using the same position ID (Arriola and Kuleshov, [2], Section 5.3 and Appendix K). The mask can attend to tokens earlier in the generation order but cannot see its own target token or later tokens, allowing all conditional predictions to be computed in one pass. We use the same construction for verification, placing draft tokens in the token entries and comparing them with predictions from the corresponding masks. Packed-SS adds a second mask at each position to generate the next draft from the prompt and accepted tokens.
Packed input and attention. Let contain the unresolved positions in the active window, and write . Let contain the correction token, if any, that has not yet been processed as an input. Let contain the previous draft , with masks for missing entries; contain masks for verifier predictions; and contain masks for new draft predictions. The input is
All regions attend to the cached keys and values of the prompt and accepted tokens. The regions , , also attend to , if present, and the row in attends to itself.
Write , , for the rows of , , at verification rank . Let contain the earlier ranks in the order that have a stored draft token. Using to mean that row may attend to the rows in , the remaining attention connections are
The three rows at rank share the textual position ID . The physical call evaluates all verifier queries, while the logical algorithm needs predictions only through the first missing draft token or disagreement.
Cache update. Only keys and values from and accepted draft tokens in are added to the cache. A mask’s keys and values do not represent its predicted token, so a correction is processed in the next call through .
Input-size comparison
Table Table 2 compares the input sizes of these layouts with those of Linear-SS and Quadratic-SS.
| Method | NFE/iteration | Nemotron | SetDLM general | ||
| Linear-SS | 2 | 25 | 49 | ||
| Quadratic-SS | 1 | 81 | 289 | ||
| Packed-SS (Ours) | 1 | 25 | 49 |
Table 2. Execution cost per iteration with cached prompt context. Evaluated uncached rows count token representations processed by the transformer, including duplicates and evaluated padding, summed across calls. Nemotron uses block size . In SetDLM, positions remain unresolved in the active window, and counts the correction token processed in this call; maximum row counts take and . Prompt rows are processed once and depend on prompt length.
For SetDLM, Linear-SS processes rows for drafting and for verification, caching between calls. Packed-SS processes the same total in one call. Quadratic-SS uses rows for verification and proposal regions of rows each.
Experimental Details
Hardware, models, and tasks
All runs use batch size one on a single A100 80GB GPU. Nemotron is evaluated in bfloat16 (BF16), and SetDLM in float32 (FP32). Nemotron uses left-to-right AR verification in every main setting. Both scales run the same four tasks—HumanEval, MBPP, MATH-500, and GSM8K ([13, 4, 27, 18])—under greedy decoding and under rejection sampling. Each cell draws on a fixed prompt set: 164 HumanEval problems, 378 MBPP problems, 500 MATH-500 problems, and all 1,319 GSM8K test questions, and these counts are the number of examples behind every mean and interval reported for that cell. Both scales additionally run all 555 English WritingBench prompts ((Wu et al., 2025)) at both temperatures. The SetDLM experiments allocate 1,024 output positions and use active windows of size 8 and 16; because its two windows come from separately adapted checkpoints, comparing them is not a same-weights window-size ablation. At both temperatures, Linear-SS and Packed-SS use Nemotron block size , while Quadratic-SS uses , with speculative width .
Training horizon versus inference window. The SetDLM training construction uses a desired prediction budget and permits schedules reaching future positions [2]. Checkpoints are named by their maximum horizon. Hence smax32 is the , maximum-horizon-32 model, and its matched active evaluation window is rather than 32.
Initialization and stopping
Initial calls. In our Nemotron implementation, the drafting part of a regular call’s input contains the last accepted token followed by masks. During initialization, the drafting part contains masks, giving one extra draft position. For SetDLM, Packed-SS builds the prompt cache in one counted call before entering the empty-draft loop of Algorithm 3.
At temperature 1, our implementation first processes and caches the prompt, then samples the first output token from its last prediction. A second call proposes draft tokens conditioned on the prompt and this first output token. Both calls count toward NFE and supply a full draft before verification, whereas Algorithm 2 starts from an empty draft.
Output caps. Generations are capped at 512 tokens for code and writing and 1,024 tokens for mathematics.
SetDLM stop sequences. An end marker in the SetDLM evaluation can consist of several tokens. We recognize it only after all of those tokens have been accepted. We set , the last output position, to the marker’s final position and fill any earlier unresolved positions before returning. If several markers are recognized, we use the one that ends earliest in the output.
Prompts and scoring
Nemotron uses zero-shot prompts: function completion for HumanEval, visible original test cases for MBPP, and requests for boxed final answers on the mathematics tasks. MBPP uses the 378-problem EvalPlus subset and its original base tests. MATH-500 uses boxed-answer exact match, and SetDLM GSM8K scoring compares normalized final answers. The code and mathematics runs use the same prompts and graders at both temperatures.
For temperature-1 WritingBench runs, each complete query uses the checkpoint’s user-message chat template with thinking disabled; prompts are not truncated. Scored-task tables report accuracy or pass@1 at temperature 0 and mean per-response correctness at temperature 1. WritingBench measures speed only, without judge-based scoring.
Sampling and repetitions
At temperature 1, rejection sampling can produce different generation trajectories for the same prompt, varying output length and decoding cost. To reduce this source of variance on the code tasks, each method draws 20 responses per HumanEval prompt and 7 per MBPP prompt. MATH-500, GSM8K, and WritingBench use one response per prompt and method. Sampling uses float32 probability normalization and the full vocabulary, without top- or top- filtering. For Quadratic-SS, the verification result selects the branch matching the accepted draft prefix before its next draft tokens are sampled.
Timing and confidence intervals
Decoder wall time is CUDA-synchronized and includes prefill and all decoding work, excluding model loading, prompt preparation, tokenization, output-text decoding, and scoring. Each model forward pass counts as one NFE, including prompt processing. Generated-token counts include stop and special tokens and any generated reasoning text.
We compute 95% percentile confidence intervals from 20,000 bootstrap resamples within each checkpoint, task, decoding mode, and verification order. Each resample draws prompt indices with replacement, uses the same indices for all methods, and recomputes the reported statistics. We bootstrap over prompts, using all repeated responses for each sampled prompt. The intervals describe variation across prompts rather than timing noise for a fixed generation, and are not adjusted for multiple comparisons.
SetDLM decoding settings
Confidence scores. For confidence ordering, Packed-SS and Quadratic-SS score each unresolved position by the probability assigned to its draft token when that token was proposed; Quadratic-SS uses the selected proposal branch. Linear-SS uses the probability of the verifier’s most likely token at each position in the previous iteration. Positions are ranked by decreasing score, with ties broken by textual position. Positions without a draft token or a finite score come last in textual order, so the first iteration uses LTR order. Each order is fixed before the next model call; for Linear-SS, this is the drafting call.
Native baseline. Native SetDLM uses the released sliding-window decoder with argmax token selection and confidence threshold 0.9. It uses the same GSM8K prompts, number of output positions, window sizes, and stopping convention as the speculative schedules.
Additional Experiments and Tables
This appendix collects additional experiments, absolute measurements, and numerical values supporting the main figures.
Nemotron block size and execution cost
To select the block size for each schedule, we run a small probe on Nemotron 8B using the first ten GSM8K test questions (Table 3). We choose the size with the highest token throughput (tokens/s) among those tested. All runs use a total output limit of 512 new tokens.
| Method | Tokens/s | Rows/call | Wall (s) | Total NFE | Tokens | Accuracy (%) | |
| Linear-SS | 8 | 94.9 | 8 | 2.824 | 98.2 | 267.9 | 100 |
| Linear-SS | 16 | 113.4 | 16 | 2.448 | 84.6 | 277.5 | 100 |
| Linear-SS | 32 | 121.9 | 32 | 2.237 | 77.4 | 272.8 | 100 |
| Packed-SS | 8 | 94.8 | 16 | 2.798 | 89.0 | 265.2 | 100 |
| Packed-SS | 16 | 123.5 | 32 | 2.315 | 73.3 | 285.9 | 100 |
| Packed-SS | 32 | 128.5 | 64 | 2.333 | 75.4 | 299.7 | 90 |
| Quadratic-SS | 8 | 122.3 | 72 | 2.232 | 64.9 | 272.9 | 100 |
| Quadratic-SS | 16 | 84.6 | 272 | 3.246 | 57.6 | 274.7 | 100 |
| Quadratic-SS | 32 | 25.4 | 1056 | 12.398 | 64.1 | 314.6 | 80 |
Table 3. Nemotron 8B GSM8K block-size probe. Each row uses the same ten questions. Tokens/s is calculated from the displayed means. Wall is mean total decoder time per response, including prompt processing; total NFE and output-token counts are also per-response means. Accuracy is reported for comparison. Rows/call is the maximum number of input rows per call after initialization.
Proposal-only LoRA accounting
We evaluate the proposal-only low-rank adapter trained by the Nemotron authors (Fu et al., 2026) on Nemotron 8B and all GSM8K test questions, changing diffusion drafting while preserving the AR verifier. The base and LoRA measurements in this study use a total output cap of 512 new tokens, including thinking and answer tokens. This cap differs from the 1,024-token GSM8K setting in § 4, so absolute timings are not directly comparable with Table 6. Packed-SS reduces NFE by 12.0% relative to Linear-SS but takes 1.749 rather than 1.698 seconds per example, a 3.0% slowdown (Table 4).
| Method | Tokens | Total NFE | Model (s) | Non-model (s) | Wall (s) | Tokens/s | Accuracy (%) |
| Linear-SS | 272.2 | 55.04 | 1.561 | 0.137 | 1.698 | 160.32 | 85.90 |
| Quadratic-SS | 273.3 | 54.44 | 1.781 | 0.343 | 2.123 | 128.72 | 86.35 |
| Packed-SS | 274.2 | 48.44 | 1.570 | 0.179 | 1.749 | 156.78 | 85.82 |
Table 4. Nemotron 8B GSM8K with proposal-only LoRA (). Means; model execution time and non-model decoder time sum to inclusive decoder wall time. Emitted length and throughput accompany latency, and accuracy is reported for comparison.
In calls with the adapter enabled, the implementation computes an update for every input row but applies it only to proposal rows. Packed therefore computes 64 updates but applies only 32, whereas Linear places its 32 proposal rows in a separate call and disables the adapter for its 32-row verifier. Quadratic applies updates to 64 proposal rows but computes them for all 72 input rows.
| Method | Query rows | Selected | Evaluated | Base ms | LoRA ms | Increase |
| Linear-SS | 32 + 32 | 32 | 32 | 54.41 | 56.77 | 4.3% |
| Quadratic-SS | 72 | 64 | 72 | 28.90 | 32.69 | 13.1% |
| Packed-SS | 64 | 32 | 64 | 28.58 | 32.37 | 13.3% |
Table 5. Nemotron 8B GSM8K model execution time per iteration. Linear’s iteration contains both calls. “Selected” counts rows receiving adapter updates; “evaluated” counts rows for which those updates are computed.
With LoRA, mean model execution time is nearly equal for Packed-SS and Linear-SS (1.570 versus 1.561 seconds). After the first call, Packed-SS averages 47.44 iterations versus Linear-SS’s 27.02, but its model time per iteration is 32.37 versus 56.77 milliseconds, so these differences almost cancel. Non-model decoder time adds 0.179 seconds for Packed-SS against 0.137 for Linear-SS, and the Packed-SS-minus-Linear-SS wall-time difference is 0.0513 seconds with paired 95% interval [0.0243, 0.0788]. Token throughput is also lower: Packed-SS emits 274.2 tokens on average against Linear-SS’s 272.2, and its throughput ratio is 0.978 with paired interval [0.969, 0.987]. Shared, densely fine-tuned weights for drafting and verification would eliminate these extra LoRA computations and could reduce the latency penalty. The Nemotron authors have not released such a checkpoint.
Model execution and non-model decoder time
To separate model evaluation from decoder overhead, we normalize each task’s component times by its Linear-SS total and then average tasks equally within each model scale, so that long-output tasks do not dominate the decomposition. The decomposition covers GSM8K and MATH-500 only.

Figure 5. Normalized Nemotron timing decomposition on GSM8K and MATH-500. Greedy decoding with a 1,024-token cap. Each task’s component times are divided by that task’s Linear-SS total, then the two tasks are averaged with equal weight within each model scale. The dashed line marks Linear-SS’s total. Non-model decoder time covers draft token construction, selection, and state updates.
Packed-SS cuts model execution time to 0.860 and 0.854 of Linear-SS’s total at 3B and 8B while raising non-model decoder time by about 40%, for totals of 0.964 and 0.951. Quadratic-SS cuts model execution time comparably, to 0.898 and 0.876, but its non-model decoder time is 2.9 and 2.7 times Linear-SS’s, so its totals exceed one at 1.106 and 1.062.
Absolute measurements
Confidence-ordered verification on SetDLM
The training horizons are and 32 for and 16, respectively. For and 16, respectively, maximum recurrent rows/call are (16, 32) for Linear-SS, (81, 289) for Quadratic-SS, (25, 49) for Packed-SS, and (16, 28) for Native. Native commits at confidence threshold 0.9 and has no verification order (Appendix § 10.6).
Under confidence order, Packed-SS remains fastest at both windows (Table 9), reducing Linear-SS’s wall time by 9.5% at and 9.7% at and its total NFE by 12.4% and 16.5%. Its throughput gains over Linear-SS are 10.2% and 12.8%, and at it reaches 2.20× Quadratic-SS’s throughput; all eight SetDLM paired intervals favour Packed-SS (Table 10). Two effects should be kept apart, because only one of them is a cost. The three LTR schedules have identical per-example correctness labels within each checkpoint, so Packed-SS’s LTR speedup incurs no observed accuracy cost. Changing the order can affect accuracy. Confidence ordering lowers Packed-SS’s latency by a further 4.1% at and 13.1% at relative to LTR, but accuracy falls from 68.39% to 65.35% and from 66.57% to 59.06%; the second drop is large enough to give up the advantage over SetDLM’s own threshold decoder, which scores 59.51% there. Adaptive position choices alter the greedy trajectory even in exact arithmetic, and single-step confidence is in any case a weak criterion for choosing which tokens to accept ([12]). We therefore report the ordering as a speed–accuracy trade-off of the decoding policy rather than a property of the schedule: Packed-SS gives the lowest latency in each regime.
| Window | Method | Tokens/s | Tokens/NFE | Time/NFE (ms) | Accuracy (%) | Tokens | Wall (s) | NFE |
| LTR verification and native decoding | ||||||||
| Linear-SS | 94.93 | 2.610 | 27.496 | 68.39 | 257.2 | 2.709 | 98.53 | |
| Quadratic-SS | 86.25 | 3.414 | 39.583 | 68.39 | 257.1 | 2.981 | 75.31 | |
| Packed-SS | 100.60 | 2.842 | 28.247 | 68.39 | 257.2 | 2.556 | 90.49 | |
| Native | 77.21 | 2.119 | 27.442 | 59.67 | 252.4 | 3.269 | 119.11 | |
| Linear-SS | 98.85 | 2.792 | 28.243 | 66.57 | 259.9 | 2.629 | 93.09 | |
| Quadratic-SS | 41.60 | 4.074 | 97.934 | 66.57 | 260.4 | 6.260 | 63.92 | |
| Packed-SS | 100.30 | 3.030 | 30.209 | 66.57 | 259.9 | 2.591 | 85.77 | |
| Native | 80.59 | 2.220 | 27.552 | 59.51 | 253.6 | 3.147 | 114.21 | |
| Confidence-ordered verification | ||||||||
| Linear-SS | 93.93 | 2.586 | 27.535 | 65.05 | 254.5 | 2.709 | 98.39 | |
| Quadratic-SS | 90.83 | 3.470 | 38.200 | 65.58 | 254.6 | 2.803 | 73.38 | |
| Packed-SS | 103.52 | 2.943 | 28.431 | 65.35 | 253.8 | 2.451 | 86.21 | |
| Linear-SS | 99.06 | 2.794 | 28.206 | 53.07 | 247.0 | 2.493 | 88.40 | |
| Quadratic-SS | 50.90 | 4.462 | 87.664 | 60.80 | 252.1 | 4.954 | 56.51 | |
| Packed-SS | 111.76 | 3.408 | 30.491 | 59.06 | 251.5 | 2.251 | 73.81 |
Table 9. SetDLM measurements on GSM8K. All 1,319 test questions. Top: LTR verification and native decoding; bottom: confidence-ordered verification. Tokens, inclusive wall time, and total NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each window and order.
| Setting | Baseline | wall (s) [95% CI] | Throughput ratio [95% CI] |
| LTR | Linear-SS | [, ] | 1.060 [1.058, 1.062] |
| Quadratic-SS | [, ] | 1.166 [1.164, 1.169] | |
| Confidence | Linear-SS | [, ] | 1.102 [1.098, 1.107] |
| Quadratic-SS | [, ] | 1.140 [1.136, 1.143] | |
| LTR | Linear-SS | [, ] | 1.015 [1.013, 1.017] |
| Quadratic-SS | [, ] | 2.411 [2.399, 2.424] | |
| Confidence | Linear-SS | [, ] | 1.128 [1.123, 1.134] |
| Quadratic-SS | [, ] | 2.196 [2.185, 2.207] |
Table 10. Paired SetDLM efficiency comparisons. Wall-time differences are Packed-SS minus baseline; throughput ratios are Packed-SS/baseline. Brackets give paired 95% confidence intervals.
Accuracy and output trajectories
We report paired accuracy differences to assess variation in task scores At temperature 1, the schedules draw different random samples, so their continuations and measured accuracies can differ. Under greedy decoding, small floating-point differences can also change token choices and subsequent continuations. Table 11 reports the Nemotron comparisons in percentage points, with paired 95% confidence intervals.
| Setting | Packed-SS − Linear-SS | Packed-SS − Quadratic-SS | Linear-SS − Quadratic-SS |
| Temperature 0 | |||
| 3B MBPP | -0.26 [-1.32, +0.79] | +0.26 [-0.79, +1.32] | +0.53 [-0.53, +1.59] |
| 3B HumanEval | -1.22 [-3.66, +1.22] | -1.83 [-4.27, +0.00] | -0.61 [-1.83, +0.00] |
| 3B GSM8K | -0.30 [-1.14, +0.53] | -0.15 [-0.99, +0.68] | +0.15 [-0.68, +0.99] |
| 3B MATH-500 | +3.00 [+0.80, +5.20] | +0.20 [-1.60, +2.00] | -2.80 [-4.60, -1.00] |
| 8B MBPP | -0.26 [-1.59, +0.79] | +0.00 [-0.79, +0.79] | +0.26 [-0.53, +1.06] |
| 8B HumanEval | +1.22 [-1.22, +3.66] | -0.61 [-3.05, +1.22] | -1.83 [-4.27, +0.00] |
| 8B GSM8K | +0.15 [-0.83, +1.14] | -0.53 [-1.59, +0.53] | -0.68 [-1.82, +0.38] |
| 8B MATH-500 | +1.40 [-0.60, +3.60] | +1.60 [-0.40, +3.80] | +0.20 [-2.00, +2.40] |
| Temperature 1 | |||
| 3B MBPP | +0.64 [-0.98, +2.31] | -0.91 [-2.61, +0.83] | -1.55 [-3.17, +0.08] |
| 3B HumanEval | +1.46 [+0.06, +2.90] | +1.07 [-0.34, +2.47] | -0.40 [-1.74, +0.92] |
| 3B GSM8K | -0.68 [-2.65, +1.36] | -1.36 [-3.34, +0.61] | -0.68 [-2.73, +1.36] |
| 3B MATH-500 | -1.00 [-4.60, +2.60] | +0.80 [-2.80, +4.40] | +1.80 [-1.80, +5.40] |
| 8B MBPP | -1.40 [-2.95, +0.15] | +0.38 [-1.32, +2.12] | +1.78 [+0.11, +3.44] |
| 8B HumanEval | +0.64 [-0.91, +2.20] | +0.06 [-1.49, +1.59] | -0.58 [-2.13, +0.95] |
| 8B GSM8K | -2.05 [-4.02, -0.08] | -1.21 [-3.26, +0.83] | +0.83 [-1.06, +2.73] |
| 8B MATH-500 | +0.00 [-4.00, +4.00] | -1.00 [-4.60, +2.80] | -1.00 [-4.80, +2.80] |
Table 11. Paired Nemotron accuracy differences.
For SetDLM, the LTR schedules have identical correctness labels despite different outputs on 5 of 1,319 smax16 questions and 13 of 1,319 smax32 questions. Their accuracy gains over Native are 8.72 points at (95% CI [6.37, 11.07]) and 7.05 points at (95% CI [4.78, 9.33]).
Paired efficiency comparisons
Table 12 and Table 13 report Packed-SS minus baseline wall time and Packed-SS/baseline throughput, with 95% paired intervals. Negative differences and ratios above one favor Packed-SS.
| Setting | Baseline | wall (s) [95% CI] | Throughput ratio [95% CI] |
| 3B MBPP | Linear-SS | -0.0092 [-0.0268, +0.0123] | 1.044 [1.007, 1.075] |
| Quadratic-SS | -0.0308 [-0.0563, -0.0027] | 1.102 [1.079, 1.127] | |
| 3B HumanEval | Linear-SS | -0.0207 [-0.0341, -0.0100] | 1.052 [1.033, 1.072] |
| Quadratic-SS | -0.0703 [-0.0886, -0.0539] | 1.181 [1.149, 1.213] | |
| 3B GSM8K | Linear-SS | -0.0434 [-0.0605, -0.0243] | 1.061 [1.049, 1.071] |
| Quadratic-SS | -0.1003 [-0.1187, -0.0798] | 1.114 [1.103, 1.125] | |
| 3B MATH-500 | Linear-SS | -0.0565 [-0.1145, +0.0040] | 1.035 [1.021, 1.048] |
| Quadratic-SS | -0.3872 [-0.4531, -0.3221] | 1.192 [1.174, 1.209] | |
| 3B WritingBench | Linear-SS | -0.4969 [-0.5335, -0.4604] | 1.093 [1.087, 1.099] |
| Quadratic-SS | -0.1753 [-0.2068, -0.1435] | 1.034 [1.029, 1.040] | |
| 8B MBPP | Linear-SS | -0.0058 [-0.0222, +0.0156] | 1.040 [1.024, 1.056] |
| Quadratic-SS | -0.0711 [-0.0921, -0.0520] | 1.172 [1.144, 1.200] | |
| 8B HumanEval | Linear-SS | -0.0294 [-0.0509, -0.0077] | 1.050 [1.022, 1.078] |
| Quadratic-SS | -0.1372 [-0.1634, -0.1121] | 1.323 [1.274, 1.376] | |
| 8B GSM8K | Linear-SS | -0.0908 [-0.1499, -0.0318] | 1.054 [1.042, 1.066] |
| Quadratic-SS | -0.1856 [-0.2433, -0.1277] | 1.092 [1.082, 1.104] | |
| 8B MATH-500 | Linear-SS | -0.2656 [-0.3560, -0.1778] | 1.056 [1.040, 1.071] |
| Quadratic-SS | -0.6361 [-0.7332, -0.5367] | 1.163 [1.145, 1.181] | |
| 8B WritingBench | Linear-SS | -0.5538 [-0.6101, -0.4958] | 1.081 [1.073, 1.089] |
| Quadratic-SS | -0.3866 [-0.4433, -0.3300] | 1.057 [1.050, 1.065] |
Table 12. Paired Nemotron efficiency comparisons at temperature 0.
| Setting | Baseline | wall (s) [95% CI] | Throughput ratio [95% CI] |
| 3B MBPP | Linear-SS | -0.0389 [-0.0678, -0.0107] | 1.039 [1.026, 1.052] |
| Quadratic-SS | -0.0287 [-0.0832, +0.0248] | 0.969 [0.956, 0.984] | |
| 3B HumanEval | Linear-SS | -0.0076 [-0.0250, +0.0093] | 1.021 [1.005, 1.038] |
| Quadratic-SS | +0.0033 [-0.0121, +0.0189] | 1.015 [0.997, 1.034] | |
| 3B GSM8K | Linear-SS | -0.0227 [-0.0832, +0.0387] | 1.020 [1.001, 1.037] |
| Quadratic-SS | +0.0000 [-0.0640, +0.0652] | 0.996 [0.979, 1.013] | |
| 3B MATH-500 | Linear-SS | -0.0820 [-0.2440, +0.0791] | 1.036 [1.012, 1.060] |
| Quadratic-SS | -0.0895 [-0.2481, +0.0661] | 1.043 [1.021, 1.066] | |
| 3B WritingBench | Linear-SS | -0.2455 [-0.3181, -0.1699] | 1.041 [1.034, 1.048] |
| Quadratic-SS | +0.0536 [-0.0187, +0.1283] | 0.998 [0.990, 1.006] | |
| 8B MBPP | Linear-SS | -0.0467 [-0.0970, +0.0024] | 1.061 [1.045, 1.076] |
| Quadratic-SS | +0.0619 [+0.0072, +0.1164] | 0.982 [0.969, 0.995] | |
| 8B HumanEval | Linear-SS | -0.0767 [-0.1104, -0.0443] | 1.059 [1.042, 1.077] |
| Quadratic-SS | -0.0423 [-0.0786, -0.0059] | 1.024 [1.007, 1.042] | |
| 8B GSM8K | Linear-SS | -0.3104 [-0.4833, -0.1363] | 1.063 [1.046, 1.081] |
| Quadratic-SS | -0.0025 [-0.1731, +0.1618] | 0.992 [0.976, 1.007] | |
| 8B MATH-500 | Linear-SS | -0.2686 [-0.5450, +0.0071] | 1.050 [1.029, 1.073] |
| Quadratic-SS | +0.0831 [-0.1779, +0.3394] | 0.997 [0.978, 1.018] | |
| 8B WritingBench | Linear-SS | -0.5121 [-0.6381, -0.3880] | 1.052 [1.043, 1.061] |
| Quadratic-SS | -0.3029 [-0.4490, -0.1578] | 1.039 [1.030, 1.047] |
Table 13. Paired Nemotron efficiency comparisons at temperature 1.
All ten paired throughput intervals against Linear-SS lie above one (Table 13), supporting a throughput advantage in the measured settings. Against the Quadratic-SS extension, three intervals lie above one, two lie below it, and five include it; the last group is inconclusive rather than evidence of equivalence.
Reference schedules. At temperature 0, Quadratic-SS has lower throughput than Linear-SS on all eight code and mathematics settings but higher throughput on WritingBench; at temperature 1, it has higher throughput on seven of the eight code and mathematics settings. Shorter accepted prefixes require more iterations, so eliminating a second call can outweigh the cost of its fixed 72-row pass (Appendix § 11.§ 3). On SetDLM, Linear-SS and Packed-SS are faster than Native at both windows; Quadratic-SS is faster only at (Table 8).
References
- [1]Arriola, M., Gokaslan, A., Chiu, J., Yang, Z., Qi, Z., Han, J., Sahoo, S., and Kuleshov, V. (2025). Block diffusion: Interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations.
- [2]Arriola, M. and Kuleshov, V. (2026). Set diffusion: Interpolating token orderings between autoregression and diffusion for fast and flexible decoding. In Proceedings of the 43rd International Conference on Machine Learning.arxiv.org/abs/2607.01775
- [3]Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. (2021a). Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, volume 34.arxiv.org/abs/2107.03006
- [4]Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. (2021b). Program synthesis with large language models. arXiv preprint arXiv:2108.07732.arxiv.org/abs/2108.07732
- [6]Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33.arxiv.org/abs/2005.14165
- [7]Cai, C. and Li, G. (2026). Confidence-based decoding is provably efficient for diffusion language models. arXiv preprint arXiv:2603.22248.arxiv.org/abs/2603.22248
- [8]Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. (2024). Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5209–5235.arxiv.org/abs/2401.10774
- [9]Campbell, A., Benton, J., De Bortoli, V., Rainforth, T., Deligiannidis, G., and Doucet, A. (2022). A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems.
- [10]Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. (2023). Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318.arxiv.org/abs/2302.01318
- [11]Chen, J., Liang, Y., and Liu, Z. (2026a). DFlash: Block diffusion for flash speculative decoding. In Proceedings of the 43rd International Conference on Machine Learning.arxiv.org/abs/2602.06036
- [12]Chen, K., Liu, Z., Tao, X., Liu, H., Fu, X., Zhang, S., Tu, D., Kong, L., Liu, R., and Li, H. (2026b). Beyond confidence: Adaptive and coherent decoding for diffusion language models. In Proceedings of the 43rd International Conference on Machine Learning.arxiv.org/abs/2512.02044
- [13]Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.arxiv.org/abs/2107.03374
- [14]Chen, S., Cong, K., and Li, J. (2025). Optimal inference schedules for masked diffusion models. arXiv preprint arXiv:2511.04647.arxiv.org/abs/2511.04647
- [15]Cheng, X., Yu, X., Shao, C., Li, J., Xiong, Y., Qian, Y., Zhu, J., Ma, S., Zhang, X., Ye, J., et al. (2026). DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv preprint arXiv:2607.05147.arxiv.org/abs/2607.05147
- [16]Cheng, Z., Yang, G.-W., Li, J., Deng, Z., Guo, M.-H., and Hu, S.-M. (2025). DEER: Draft with diffusion, verify with autoregressive models. arXiv preprint arXiv:2512.15176.arxiv.org/abs/2512.15176
- [17]Christopher, J. K., Bartoldson, B. R., Ben-Nun, T., Cardei, M., Kailkhura, B., and Fioretto, F. (2024). Speculative diffusion decoding: Accelerating language generation through diffusion. arXiv preprint arXiv:2408.05636.arxiv.org/abs/2408.05636
- [18]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.arxiv.org/abs/2110.14168
- [19]Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., Aly, A., Chen, B., and Wu, C.-J. (2024). LayerSkip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12622–12642.
- [20]Fan, A., Lewis, M., and Dauphin, Y. (2018). Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898.arxiv.org/abs/1805.04833
- [23]Gao, Y., Ji, Z., Wang, Y., Qi, B., Xu, H., and Zhang, L. (2025). Self speculative decoding for diffusion large language models. arXiv preprint arXiv:2510.04147.arxiv.org/abs/2510.04147
- [24]Gong, S., Agarwal, S., Zhang, Y., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Han, J., Peng, H., and Kong, L. (2025). Scaling diffusion language models via adaptation from autoregressive models. In International Conference on Learning Representations.arxiv.org/abs/2410.17891
- [25]Grattafiori, A. et al. (2024). The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.arxiv.org/abs/2407.21783
- [26]Guo, G. and Ermon, S. (2026). Self-speculative decoding accelerates lossless inference in any-order and any-subset autoregressive models. In International Conference on Learning Representations.
- [27]Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
- [28]Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2020). The curious case of neural text degeneration. In International Conference on Learning Representations.
- [29]Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. (2021). Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Information Processing Systems, volume 34.
- [30]Leviathan, Y., Kalman, M., and Matias, Y. (2023). Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR.arxiv.org/abs/2211.17192
- [31]Li, A. C. (2025). Generalization Through Richer Supervision. PhD thesis, Machine Learning Department, Carnegie Mellon University. CMU-ML-25-103; Prism is Chapter 5.DOI
- [37]Liu, J., Dong, X., Ye, Z., Mehta, R., Fu, Y., Singh, V., Kautz, J., Zhang, C., and Molchanov, P. (2025). TiDAR: Think in diffusion, talk in autoregression. arXiv preprint arXiv:2511.08923.arxiv.org/abs/2511.08923
- [38]Lou, A., Meng, C., and Ermon, S. (2024). Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning.arxiv.org/abs/2310.16834
- [39]Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., Shi, C., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. (2024). SpecInfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3.DOI
- [40]Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. (2025). Large language diffusion models. arXiv preprint arXiv:2502.09992.arxiv.org/abs/2502.09992
- [41]Qwen Team (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388.
- [42]Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., and Kuleshov, V. (2024). Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems.arxiv.org/abs/2406.07524
- [43]Samragh, M., Kundu, A., Harrison, D., Nishu, K., Naik, D., Cho, M., and Farajtabar, M. (2025). Your LLM knows the future: Uncovering its multi-token prediction potential. arXiv preprint arXiv:2507.11851.arxiv.org/abs/2507.11851
- [44]Santilli, A., Severino, S., Postolache, E., Maiorca, V., Mancusi, M., Marin, R., and Rodola, E. (2023). Accelerating transformer inference for translation via parallel decoding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12336–12355. Association for Computational Linguistics.
- [45]Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. (2024). Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems.arxiv.org/abs/2406.04329
- [46]Spector, B. and Ré, C. (2023). Accelerating LLM inference with staged speculative decoding. arXiv preprint arXiv:2308.04623.arxiv.org/abs/2308.04623
- [47]Stern, M., Shazeer, N., and Uszkoreit, J. (2018). Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems, volume 31.arxiv.org/abs/1811.03115
- [48]Teng, Y., Shi, H., Liu, X., Ning, X., Dai, G., Wang, Y., Li, Z., and Liu, X. (2024). Accelerating auto-regressive text-to-image generation with training-free speculative jacobi decoding. arXiv preprint arXiv:2410.01699.arxiv.org/abs/2410.01699
- [51]Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., and Sui, Z. (2023). Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP.arxiv.org/abs/2203.16487
- [54]Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. (2024). Draft & verify: Lossless large language model acceleration via self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
- [55]Zhang, Y., Zhang, L., Cui, B., and Miao, X. (2026). DFlow: Enabling verifier information flow in block diffusion speculative decoding. arXiv preprint arXiv:2609.06498.arxiv.org/abs/2609.06498
- [56]Zhao, Y. and Cai, C. (2026). Adaptation to intrinsic dependence in diffusion language models. arXiv preprint arXiv:2602.20126.arxiv.org/abs/2602.20126