betaFeedback

WorksComputer Science

One Call Ahead: Packed Self-Speculative Decoding with Diffusion Drafting

Abstract

Autoregressive (AR) language models generate tokens sequentially, so generating long outputs is costly. Speculative decoding mitigates this by using a lightweight drafter to propose multiple tokens for the target AR model to verify in parallel, while a diffusion-based drafter can produce an entire draft block in one forward pass. In models with AR and diffusion modes, one network both drafts and verifies, but drafting still waits for verification, so the standard schedule needs two sequential model calls per iteration. Existing one-call schedules avoid the wait but incur input size quadratic in the speculative width. Motivated by this efficiency bottleneck, we introduce Packed-SS, a self-speculative decoding schedule that verifies and drafts in a single model call with input size linear in the speculative width, while preserving the target model's distribution. The key idea is to let the diffusion drafter run one call ahead: while verifying the current draft, the model simultaneously constructs the next draft from an older verified prefix and keeps its unresolved suffix for the next iteration. Under a mild monotonicity assumption on the drafter, we further show that Packed-SS requires no more model calls than the standard two-call decoding in the greedy setting. Experiments on Nemotron 3B/8B and SetDLM 1.7B show higher throughput and lower mean wall time than both baselines under greedy decoding.

https://github.com/cyx78/One-Call-Ahead-Packed-Self-Speculative-Decoding-with-Diffusion-Drafting

Introduction

Modern language models (LMs) increasingly generate long sequences of intermediate reasoning before producing a final answer, making inference latency a central bottleneck. Standard autoregressive (AR) models [6] generate one token at a time from left to right. Producing an LL-length sequence therefore requires at least LL sequential model calls, where the computation for each new token depends on all previously generated tokens.

Speculative decoding [30, 10, 51] accelerates AR inference by using a lightweight draft model that proposes multiple tokens for the full-size target model to verify in parallel. A rejection-sampling correction step ensures that the resulting sequence follows exactly the target model’s distribution. The target model can thus output several tokens from a single verification call. However, when the drafter is AR, creating a draft block remains sequential. Therefore, speculative decoding shortens the expensive serial chain, but does not eliminate it.

Masked diffusion language models (MDLMs) provide a natural way to parallelize drafting [42, 45, 24, 40] (Ye et al. 2025). By learning to reverse a corruption process that transforms data into noise, an MDLM can predict multiple tokens simultaneously. Therefore, when used as a speculative drafter, it can draft an entire block in one forward pass [17, 31, 11, 55]. Some LMs support both causal AR prediction and parallel masked diffusion prediction in the same network [37] (Fu et al. 2026). They provide a natural form of self-speculative decoding, with the drafter and verifier sharing the same model weights.

However, shared weights do not eliminate the sequential dependency between drafting and verification. The standard self-speculative schedule, referred to as Linear-SS, uses two sequential calls: one AR call verifies the current draft and one diffusion call constructs the next draft from the newly verified prefix. Drafting cannot begin until verification finishes to determine where the verified prefix ends.

Quadratic-SS [37] eliminates this dependency by preparing a continuation for every possible verification outcome. With speculative width WW, it pre-drafts separately for the cases in which 0,1,…,W0, 1, \ldots, W draft tokens are accepted. Whichever outcome occurs, the next draft is already available. This reduces each iteration to one call, but the number of evaluated positions grows quadratically with WW. Thus, Linear-SS evaluates O(W)O(W) positions in two calls, whereas Quadratic-SS evaluates O(W2)O(W^2) in one.

At small batch sizes, a substantial fraction of a call’s latency can come from loading model weights from memory, making it attractive to evaluate more positions per call [8] (Fu et al. 2024). To illustrate this tradeoff, suppose a call evaluating nn positions costs τ+δn\tau+ \delta n, where τ\tau is a fixed overhead, mostly weight loading, and δ\delta is the marginal cost per position. An iteration costs approximately 2τ+2δW2\tau+ 2\delta W for Linear-SS and τ+δW2\tau+ \delta W^2 for Quadratic-SS. Verifying and drafting in one call with approximately 2W2W evaluated positions would cost only τ+2δW\tau+ 2\delta W per iteration, avoiding both the second-call overhead and the quadratic branching cost.

This raises the central question of this work:

Can we verify and draft in one call, while only evaluating positions linear in the speculative width?

Contributions. We answer affirmatively with Packed-SS, a plug-and-play self-speculative decoding schedule for such models. The idea is simple: the next draft does not need to wait for the newest verified prefix. Instead, we let the diffusion drafter run one call ahead. At iteration tt, one forward pass simultaneously (i) verifies the draft, and (ii) proposes a new draft block using the prefix available at the beginning of iteration tt. If verification advances by AA positions, the first AA positions of this new draft have already been resolved and are discarded. The remaining suffix becomes the draft for the next iteration. Drafting therefore no longer waits for verification. Figure 1 illustrates the difference between Packed-SS and existing schedules.

Diagram comparing Linear-SS, Quadratic-SS, and Packed-SS drafting

Figure 1. Where drafting happens in one call. Outlined boxes are model calls. Linear-SS must finish verifying before it can draft, so an iteration costs two calls. Quadratic-SS fits in one call by drafting a continuation for every place verification might stop, so its input grows quadratically with the speculative width WW. Packed-SS also uses one call, building a single draft from the previously verified prefix alone. The right column gives total draft and verifier prediction counts per iteration, including predictions that may later be discarded.

The two key observations behind this design are as follows. First, a diffusion drafter predicts tokens independently given its conditioning prefix, so a rejection at one position does not invalidate later draft tokens. Second, draft freshness affects speed rather than correctness: standard acceptance and residual correction remain exact when they use the proposal distribution that generated each draft token. Thus, an older draft may be accepted less often, but does not bias the output.

These observations remove the synchronization between drafting and verification. With an appropriate attention mask, they can be packed into one forward pass. Each iteration thus requires only one model call, while the number of evaluated positions remains linear in WW.

Our contributions are summarized as follows:

  1. We introduce Packed-SS, which simultaneously verifies the previous draft and constructs the next draft in a single model call, with input size linear in the speculative width, while preserving the target model’s distribution.

  1. We theoretically prove that, assuming that correct draft predictions remain correct as verified context grows, Packed-SS requires no more model calls than Linear-SS to generate any prescribed output length.

  1. We evaluate Packed-SS on Nemotron 3B/8B (Fu et al., 2026) and SetDLM 1.7B [2]. Across the primary greedy evaluations, Packed-SS achieves higher throughput than Linear-SS and Quadratic-SS. Under temperature-1 sampling, it consistently outperforms Linear-SS while remaining competitive with Quadratic-SS.

Background

Notation

We write V\mathcal{V} for the vocabulary and θ\theta for model parameters. For a sequence xx we write x⟨i⟩=(x(1),…,x(i−1))x^{\langle i\rangle} = (x^{(1)}, \ldots, x^{(i-1)}), and (x,y)(x,y) denotes the sequence xx followed by item yy. We use [u]+=max⁡{u,0}[u]_+ = \max\{u,0\} to denote the positive part of uu. A complete list of variables can be found in Appendix § 6.

Preliminaries

Speculative decoding. Given prefix PP, the drafter proposes WW draft tokens D=(d(1),…,d(W))D = (d^{(1)}, \ldots, d^{(W)}), drawing each d(i)d^{(i)} from a drafter distribution q(i)q^{(i)}, where ii counts positions after PP. In one model call, the target model pθp_\theta uses PP and DD to return target distributions p(i)=pθ(⋅∣P,d(<i))p^{(i)} = p_\theta(\cdot\mid P, d^{(<i)}) for every i≤W+1i \le W + 1. The draft tokens are then verified from left to right, where each draft token d(i)d^{(i)} is accepted with probability min⁡{1,p(i)(d(i))/q(i)(d(i))}\min\{1, p^{(i)}(d^{(i)})/q^{(i)}(d^{(i)})\}. At the first rejection, the decoder replaces that draft token with a correction token from the residual distribution ∝[p(i)−q(i)]+\propto[p^{(i)} - q^{(i)}]_+. If all tokens in a draft are accepted, the target model yields one extra bonus token drawn from p(W+1)=pθ(⋅∣P,D)p^{(W+1)} = p_\theta(\cdot\mid P, D). The rejection-sampling correction ensures that the generated text follows the target distribution exactly, so a low-quality drafter costs speed but never quality.

In practice, we use the target model to sample at a certain temperature, with temperature 1 being the model’s original distribution and lower temperature yielding a sharper distribution. At temperature 0, the output distribution mass is on its arg max token. This is also called greedy decoding.

Masked diffusion language models. A masked diffusion language model (MDLM) generates text by repeatedly filling missing token positions, represented by a special mask token Mask [42, 45]. Following the denoising framework of discrete diffusion [29, 3, 9, 38], the model is trained to predict original tokens from sequences in which randomly selected tokens have been replaced by Mask. During inference, let CC denote the visible tokens and their positions, and UU the remaining masked positions. One model call predicts, for each position i∈Ui \in U, a distribution qθ(i)(⋅∣C)q_\theta^{(i)}(\cdot\mid C) that approximates the conditional marginal p(i)(⋅∣C)p^{(i)}(\cdot\mid C) of the data distribution pp. Sampling each position independently from its prediction approximates the joint distribution p(zU∣C)p(z_U \mid C) by the product of marginals qθ(zU∣C)=∏i∈Uqθ(i)(zi∣C)q_\theta(z_U \mid C) = \prod_{i \in U} q_\theta^{(i)}(z_i \mid C). This approximation incurs error whenever the tokens in UU are not conditionally independent given CC, introducing intrinsic trade-off between sampling efficiency and accuracy [31, 14, 56]. To keep this error small, the decoder fills only a subset of the positions in each call, often those with the most confident predictions [40, 7] (Ben-Hamou et al. 2025), and then repeats the model call with the updated context to predict the remaining positions. Block diffusion applies this procedure within blocks generated from left to right, completing each block before starting the next [1]. A block size of one recovers autoregressive generation, while a single block covering the full sequence recovers full-sequence masked diffusion.

Shared-weight drafting and verification. A model with shared-weight AR and diffusion modes can perform both roles in speculative decoding: its masked mode serves as the drafter qθq_\theta, proposing multiple tokens after the verified prefix in parallel, while its causal mode serves as the verifier pθp_\theta. When these modes differ only in their attention masks, Packed-SS can combine them in one forward pass using a custom attention mask [39, 46] (Fu et al. 2024). Nemotron-Labs-Diffusion supports both modes through joint training on autoregressive and diffusion objectives (Fu et al., 2026). SetDLM also supports this construction through its any-order formulation: single-position conditionals in left-to-right order provide AR verification, while parallel predictions of unresolved positions provide diffusion drafting [2]. See Appendix § 9 for model-specific input layouts and attention-mask constructions.

Related work

Sequential draft–verify decoding. Most speculative decoding methods improve drafting within a sequential draft–verify schedule [30, 10]. Self-speculative methods reuse the target network: Draft & Verify [54] skips selected layers without retraining, whereas LayerSkip [19] trains intermediate layers for early-exit drafting. EAGLE [31] (Li et al. 2024b) variants instead train auxiliary drafters using target features. Parallel drafting reduces sequential work, from blockwise parallel decoding [47] and SpecDec [51] to diffusion-based drafters such as SpecDiff [17] and DiffuSpec [31]. DEER [16] aligns prefix-conditioned drafts with an AR teacher, while DFlash [11] uses a shallow block-diffusion drafter conditioned on target features. DSpark [15] adds confidence-based verification-length scheduling, while DFlow [55] reuses verifier features from rejected suffixes to improve subsequent drafts.

Other speculative decoding schedules. SpecInfer [39] and Medusa [8] use tree attention to verify multiple candidate continuations, while EAGLE-2 (Li et al., 2024a) adaptively constructs draft trees using confidence estimates. Tree-structured verification covers more candidate continuations, but does not by itself remove the dependency between drafting and verification. Jacobi decoding [44] takes a different approach, updating multiple tentative positions in parallel through fixed-point iteration. Lookahead Decoding (Fu et al., 2024) turns these trajectories into cached nn-gram candidates and combines a lookahead branch with a verification branch in one forward pass. For diffusion decoding, SSD [23] and FreeDaVe (Wu and Zhang, 2025) batch multiple speculative decoding states, with FreeDaVe also reusing accepted-state predictions for subsequent drafting. Within shared-weight models, the quadratic schedules in Samragh et al. (2025), TiDAR [37], and Nemotron (Fu et al., 2026) prepare continuations at every possible acceptance boundary, combining drafting and verification at the cost of O(W2)O(W^2) evaluated positions for speculative width WW. Apple’s linear decoding [43] and Introspective DLMs (Yu et al., 2026) discard continuations conditioned on tentative tokens after rejection, which frequently fails to simultaneously yield a draft. Speculative Jacobi Decoding [48] reuses predictions after rejection, but its proposals also condition on tentative tokens.

Our Method

In this section, we first present the high-level description of Packed-SS. We then describe the procedure under greedy decoding in § 3.1, where the main idea is clearest. § 3.2 describes the general procedure.

At a high level, iteration tt of Packed-SS reads the verified prefix Pt−1P_{t-1} and the current draft Dt−1D_{t-1}, and performs verification and drafting simultaneously and independently in one model call. The verifier accepts AtA_t tokens from the standard speculative decoding process (including one correction or bonus token). These AtA_t tokens are extended to Pt−1P_{t-1} to form the next prefix PtP_t. For speculative width WW, the drafter reads only Pt−1P_{t-1} and generates a new proposal Zt=(zt(i))i=1WZ_t = (z_t^{(i)})_{i=1}^{W} for the next WW positions. The first AtA_t of those positions have just been filled by the verifier, so their proposal tokens are discarded, and the rest become the draft DtD_t that iteration t+1t+1 verifies. See Figure 2 for an illustration of one iteration under greedy decoding.

Diagram of one iteration of greedy Packed-SS

Figure 2. One iteration of greedy Packed-SS. Columns are token positions. The old draft token dt−1(i)d_{t-1}^{(i)}, the verifier prediction vt(i)v_t^{(i)}, and the new proposal token zt(i)z_t^{(i)} are in the same position. A single forward pass takes the verified prefix Pt−1P_{t-1} and draft Dt−1D_{t-1} as input and produces the two middle rows: the AR verifier’s next-token predictions vt(i)v_t^{(i)}, conditioned on Pt−1P_{t-1} and dt−1(1),…,dt−1(i−1)d_{t-1}^{(1)}, \ldots, d_{t-1}^{(i-1)}, and the diffusion drafter’s proposal Zt=(zt(i))i=1WZ_t = (z_t^{(i)})_{i=1}^{W}, conditioned on Pt−1P_{t-1} alone. Here, one matching draft token (ht=1h_t = 1) and the correction token vt(2)v_t^{(2)} yield At=2A_t = 2 output tokens. Proposals at these two positions are discarded, leaving Dt=(zt(3),zt(4))D_t = (z_t^{(3)}, z_t^{(4)}) as the next draft.

Packed self-speculative procedure under greedy decoding

We now state the procedure under greedy decoding, and summarize it in Algorithm 1.

Initialization. Given a prompt PP, start with the prefix P0=PP_0 = P and draft D0=∅D_0 = \varnothing.

Iteration t≥1t \ge1. Given the prefix Pt−1P_{t-1} and draft Dt−1=(dt−1(1),dt−1(2),…)D_{t-1} = (d_{t-1}^{(1)}, d_{t-1}^{(2)}, \ldots) following Pt−1P_{t-1}, iteration tt evaluates the verifier and drafter in one forward pass.

For verification, the AR mode predicts the token at each position of the previous draft plus one position after it, conditioned on the prefix Pt−1P_{t-1} and the draft tokens before that position:

vt(i)=arg⁡max⁡x∈V pθ(x∣Pt−1,dt−1(<i)),1≤i≤∣Dt−1∣+1.(1)v_t^{(i)} = \underset{x \in\mathcal{V}}{\arg\max}\ p_\theta(x \mid P_{t-1}, d_{t-1}^{(<i)}), \qquad1 \le i \le\lvert D_{t-1}\rvert+ 1. \tag*{(1)}

Let hth_t be the number of leading draft tokens that match the verifier’s predictions:

ht=max⁡{0≤h≤∣Dt−1∣:dt−1(i)=vt(i) for every i≤h}.(2)h_t = \max\{0 \le h \le\lvert D_{t-1}\rvert: d_{t-1}^{(i)} = v_t^{(i)}\ \text{for every } i \le h\}. \tag*{(2)}

Verification yields the accepted draft prefix followed by vt(ht+1)v_t^{(h_t+1)}, which corrects the first mismatch or supplies a bonus after full acceptance. Thus, a total of At=ht+1A_t = h_t + 1 tokens are accepted.

In the same call, the diffusion mode drafts the WW positions that follow the previous prefix Pt−1P_{t-1}:

Zt=(zt(1),…,zt(W)),zt(i)=arg⁡max⁡x∈V qθ(i)(x∣Pt−1).(3)Z_t = (z_t^{(1)}, \ldots, z_t^{(W)}), \qquad z_t^{(i)} = \underset{x \in\mathcal{V}}{\arg\max}\ q_\theta^{(i)}(x \mid P_{t-1}). \tag*{(3)}

It is important to note that the draft tokens ZtZ_t are generated conditioned on the previous prefix Pt−1P_{t-1}, rather than in Linear-SS the new prefix PtP_t. This distinction allows drafting and verification to run in parallel while Linear-SS needs to be evaluated sequentially. On nonterminal iterations, the suffix of ZtZ_t after its first AtA_t tokens becomes the new draft:

Dt=(zt(At+1),…,zt(W)).(4)D_t = (z_t^{(A_t+1)}, \ldots, z_t^{(W)}). \tag*{(4)}
Algorithm 1 Packed-SS with LTR greedy decoding
Require: prompt P, speculative width W, output length limit L, verifier pθ, drafter qθ
1: P₀ ← P, D₀ ← ∅, t ← 1
2: while |Pₜ₋₁| − |P| < L and (t = 1 or the last token of Pₜ₋₁ is not EOS) do
    In one model call:
3:     vₜ⁽ⁱ⁾ ← arg maxₓ pθ(x | Pₜ₋₁, dₜ₋₁⁽<ⁱ⁾) for i = 1, . . . , |Dₜ₋₁| + 1 ▷ Verifier predictions
4:     zₜ⁽ⁱ⁾ ← arg maxₓ qθ(x | Pₜ₋₁) for i = 1, . . . , W ▷ Drafter proposals
    end
5: hₜ ← max{0 ≤ h ≤ |Dₜ₋₁| : dₜ₋₁⁽ⁱ⁾ = vₜ⁽ⁱ⁾ for all i ≤ h} ▷ Count matches
6: Cₜ ← (dₜ₋₁⁽¹⁾, . . . , dₜ₋₁⁽ʰᵗ⁾, vₜ⁽ʰᵗ⁺¹⁾), Aₜ ← hₜ + 1 ▷ Add verifier token
7: a ← min{Aₜ, L + |P| − |Pₜ₋₁|} ∪ {1 ≤ i ≤ Aₜ : Cₜ⁽ⁱ⁾ = EOS} ▷ Ending condition
8: Cₜ ← (Cₜ⁽¹⁾, . . . , Cₜ⁽ᵃ⁾) ▷ Prune by stopping index
9: Pₜ ← (Pₜ₋₁, Cₜ) ▷ Prepare next prefix, append accepted tokens
10: Dₜ ← (zₜ⁽ᵃ⁺¹⁾, . . . , zₜ⁽ᵂ⁾) ▷ Prepare next draft, zₜ⁽ⁱ⁾ is generated in line 4
11: t ← t + 1
12: end while
13: Output: Pₜ₋₁

Termination. Stop at the first accepted end-of-sequence (EOS) token or after LL new tokens.

Comparison with Linear-SS and Quadratic-SS. Linear-SS alternates between drafting-only and verification-only calls, whereas Packed-SS performs both in each call. By retaining the unresolved suffix of ZtZ_t as the next draft, Packed-SS avoids a separate drafting call. This overlap comes at the cost of a shorter draft conditioned on less verified context: on nonterminal iterations of Algorithm 1, Packed-SS retains ∣Dt∣=W−At≤W−1|D_t| = W - A_t \le W - 1 tokens by equation 4, whereas Linear-SS drafts WW tokens conditioned on AtA_t more verified tokens. In addition, both Packed-SS and Quadratic-SS draft and verify simultaneously in one model call. However, Packed-SS evaluates O(W)O(W) positions per iteration rather than O(W2)O(W^2). The trade-off is conditioning context: Quadratic-SS’s selected proposal excludes only the final verifier token from PtP_t, whereas Packed-SS conditions on Pt−1P_{t-1}, excluding all AtA_t newly appended tokens. Conceptually, Packed-SS can be viewed as Quadratic-SS with only the branch for zero accepted draft tokens.

Theoretical guarantees. Under greedy decoding and the monotonicity assumption below, Packed-SS requires no more forward passes than Linear-SS to generate a prescribed number of tokens from the same prompt.

Fix a prompt and speculative width WW, and let x1,x2,…x_1, x_2, \ldots denote the target model’s greedy continuation. Let x^k(c)\hat{x}_k(c) denote the deterministic drafter’s prediction at position kk, conditioned on the prompt and x1,…,xcx_1, \ldots, x_c.

Assumption 3.1 (Monotonicity of draft correctness). For each output position kk, there exists a threshold ck∗≤kc_k^* \le k such that 1{x^k(c)=xk}=1{c≥ck∗}\mathbb{1}\{\hat{x}_k(c)=x_k\}=\mathbb{1}\{c\ge c_k^*\} for every c<kc<k evaluated by either schedule.

Here, ck∗=kc_k^*=k indicates that position kk is never correctly predicted before it is verified. Thus, additional verified context cannot turn a correct prediction into an incorrect one. The thresholds need not increase with kk: a later position may become correctly predictable with less context than an earlier one.

Theorem 3.2 (Pathwise dominance). Suppose Packed-SS and Linear-SS use the same target model, drafter, and speculative width W≥1W \ge1, and start from a common verified prefix. Under Assumption 3.1, Packed-SS’s verified prefix after 2t2t forward passes is at least as long as Linear-SS’s after tt iterations, for every t≥0t \ge0.

Let NFES(L)\mathrm{NFE}_S(L) denote the number of forward passes schedule SS requires to generate LL tokens after PP. Since each Linear-SS iteration uses two forward passes, for every L≥0L \ge0, one has

NFEPacked-SS(L)≤NFELinear-SS(L).(5)\mathrm{NFE}_{\mathrm{Packed\text{-}SS}}(L) \le\mathrm{NFE}_{\mathrm{Linear\text{-}SS}}(L). \tag*{(5)}

Since both schedules follow the same greedy continuation, the bound also holds when stopping at the first emitted EOS token. See Appendix § 7.§ 1 for the proof.

General packed self-speculative decoding

We now present Packed-SS for general sampling. Compared with the greedy procedure in Algorithm 1 from § 3.1, the general sampling requires two minor modifications.

Consider iteration tt. First, we store beyond each draft token dt(i)d_t^{(i)} the distribution from which it was sampled, Qt(i)=qt(a+i)=qθ(a+i)(⋅∣Pt−1)Q_t^{(i)} = q_t^{(a+i)} = q_\theta^{(a+i)}(\cdot\mid P_{t-1}). At temperature 0, Qt−1(i)Q_{t-1}^{(i)} collapses to a point mass on dt−1(i)d_{t-1}^{(i)}. So storing the draft token alone is sufficient as in Algorithm 1.

Second, when verifying each draft token dt−1(i)d_{t-1}^{(i)}, we replace the deterministic match test dt−1(i)=vt(i)d_{t-1}^{(i)} = v_t^{(i)} in Equation (2) with the standard rejection-sampling acceptance rule. With pt(i)=pθ(⋅∣Pt−1,dt−1(<i))p_t^{(i)} = p_\theta(\cdot\mid P_{t-1},d_{t-1}^{(<i)}) the target conditional at that position and Qt−1(i)Q_{t-1}^{(i)} its stored drafter distribution, we accept the draft token dt−1(i)d_{t-1}^{(i)} with probability min⁡{1,pt(i)(dt−1(i))/Qt−1(i)(dt−1(i))}\min\{1,p_t^{(i)}(d_{t-1}^{(i)})/Q_{t-1}^{(i)}(d_{t-1}^{(i)})\}. Crucially, the denominator is the stored drafter distribution Qt−1(i)Q_{t-1}^{(i)} from which dt−1(i)d_{t-1}^{(i)} was originally sampled, even though it may have been computed from an older prefix. At the first rejection, we draw a correction from the normalized residual distribution ∝[pt(i)−Qt−1(i)]+\propto[p_t^{(i)}-Q_{t-1}^{(i)}]_+. If the entire draft is accepted, we draw a bonus token from pt(∣Dt−1∣+1)p_t^{(|D_{t-1}|+1)}.

We summarize the resulting procedure in Algorithm 2 in Appendix § 7.§ 2. The following theorem, proved in Appendix § 7, states that Packed-SS preserves the distribution of the AR mode. Since greedy decoding is recovered at temperature 0, it also implies the correctness of Algorithm 1.

Theorem 3.3 (Distributional correctness). The continuation generated by Packed-SS after the prompt PP has the same distribution as sampling from the AR model pθp_\theta, stopped at the first EOS or at the output length limit LL.

Finally, storing the drafter distributions Qt−1(i)Q_{t-1}^{(i)} need not require a probability for every vocabulary token at each draft position. With top-kk ([20]) or top-pp truncation ([28]), we need to store only the nonzero entries of Qt−1(i)Q_{t-1}^{(i)}. For instance, Qwen3 recommends top-k=20k = 20 for a vocabulary of 151,936 tokens ([41]), and the reference implementation of Llama 3 uses top-p=0.9p = 0.9 by default ([25]).

Experiments

Models and datasets. We evaluate Packed-SS on Nemotron-Labs-Diffusion ((Fu et al., 2026)) and SetDLM ([2]). Nemotron 3B and 8B are evaluated on five datasets: GSM8K ([18]) and MATH-500 ([27]) for mathematical reasoning, HumanEval ([13]) and MBPP ([4]) for code generation, and WritingBench ((Wu et al., 2025)) for generation speed on writing prompts. For SetDLM, we evaluate two separately fine-tuned checkpoints on GSM8K released by ([2]).

Experimental setup. All experiments use batch size one on a single A100 80GB GPU, with Nemotron in BF16 and SetDLM in FP32. Generations are capped at 512 tokens for code and writing and 1,024 tokens for mathematics, to avoid repetitive responses dominating the comparison. Because output lengths can differ across schedules, our primary efficiency metric is aggregate token throughput, which decomposes as

Throughput=total generated tokenstotal decoder wall time=Tokens/NFETime/NFE\mathrm{Throughput} = \frac{\text{total generated tokens}}{\text{total decoder wall time}} = \frac{\text{Tokens/NFE}}{\text{Time/NFE}}

Tokens per NFE (Tokens/NFE) and decoder time per NFE (Time/NFE) are recorded to separate token yield from cost per NFE. Detailed setup and results are provided in Appendices § 10 and § 11.

Greedy decoding

Nemotron. Linear-SS and Packed-SS use diffusion block size B=32B = 32, while Quadratic-SS uses B=8B = 8. Each queried block includes the final token of the verified prefix, so the effective speculative width is W=B−1W = B - 1. This gives Packed-SS and Quadratic-SS comparable maximum input sizes of 64 and 72 token rows, respectively.

These block sizes also give the highest throughput among the settings tested in the small-scale experiment in Appendix § 11.§ 1.

Results are summarized in Figure 3, with numerical results in Table 6. Packed-SS achieves the highest measured throughput in all ten Nemotron settings. Compared with Linear-SS, throughput improves by 3.5–9.3%, with all ten 95% confidence intervals for the throughput ratio above one. The improvement comes from a 13.4–20.3% increase in tokens per NFE, which more than offsets the 8.9–10.1% increase in decoder time per NFE. Compared with Quadratic-SS, throughput improves by 3.4–32.3% at the evaluated block sizes. Accuracy differences between schedules are observed, but they do not provide a direct comparison of decoding quality because they arise from finite-precision effects.

Greedy decoding efficiency and accuracy across Nemotron 3B, Nemotron 8B, and SetDLM settings

Figure 3. Greedy decoding efficiency and accuracy. Efficiency metrics are ratios to Linear-SS; accuracy differences are in percentage points. Error bars show 95% paired bootstrap confidence intervals. Native is the released non-speculative SetDLM diffusion decoder, shown for reference. WritingBench measures speed only.

Table 6: Nemotron absolute measurements at temperature 0. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each model and task.

TaskMethodTokens/sNFETime/NFE (ms)Accuracy (%)TokensWall (s)NFE
Nemotron 3B
MBPPLinear-SS173.463.75721.65874.8765.80.37917.50
Quadratic-SS164.384.28826.08974.3465.90.40115.36
Packed-SS181.104.31223.81274.6067.00.37015.53
HumanEvalLinear-SS237.455.15821.72279.2790.50.38117.55
Quadratic-SS211.485.53626.17779.8891.10.43116.46
Packed-SS249.755.97123.90878.0590.00.36115.08
GSM8KLinear-SS190.054.14221.79588.40177.10.93242.76
Quadratic-SS180.934.75026.25388.25178.90.98937.66
Packed-SS201.554.81223.87388.10179.10.88837.22
MATH-500Linear-SS222.404.92822.16055.40489.02.19999.23
Quadratic-SS193.105.22927.08158.20488.52.53093.41
Packed-SS230.085.58624.28058.40492.92.14288.24
WritingBenchLinear-SS84.561.86422.038—502.15.938269.43
Quadratic-SS89.392.46827.613—502.05.616203.39
Packed-SS92.442.24224.253—502.95.441224.34
Nemotron 8B
MBPPLinear-SS151.964.42129.09585.4556.90.37512.87
Quadratic-SS134.864.63134.33585.1959.30.44012.81
Packed-SS158.005.04131.90285.1958.30.36911.56
HumanEvalLinear-SS195.715.69529.09980.4994.20.48116.54
Quadratic-SS155.305.33534.35382.3291.50.58917.15
Packed-SS205.466.54631.85981.7192.80.45214.18
GSM8KLinear-SS127.393.74429.38990.52313.22.45983.66
Quadratic-SS122.934.31935.13491.21313.92.55372.68
Packed-SS134.294.30032.02190.67318.02.36873.94
MATH-500Linear-SS147.424.28129.03760.40640.44.344149.61
Quadratic-SS133.784.69835.11660.20630.74.715134.26
Packed-SS155.614.92231.63261.80634.74.079128.94
WritingBenchLinear-SS66.121.89828.701—501.77.587264.35
Quadratic-SS67.592.47636.633—501.57.420202.55
Packed-SS71.462.25531.559—502.67.033222.87

Table 6. Nemotron absolute measurements at temperature 0. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each model and task.

SetDLM. The checkpoints with training horizons Smax⁡=16S_{\max} = 16 and Smax⁡=32S_{\max} = 32 are evaluated at speculative widths W=8W = 8 and W=16W = 16, respectively. All three speculative schedules use the same width within each checkpoint.

Packed-SS improves throughput over Linear-SS by 6.0% at W=8W = 8 and 1.5% at W=16W = 16, as in Figure 3 and Table 8. The additional token yield outweighs the higher cost per NFE: tokens per NFE increase by 8.5–8.9%, while decoder time per NFE increases by 2.7–7.0%. Packed-SS also has higher throughput than Quadratic-SS in both settings. The improvements extend to confidence-based verification order; details are provided in Appendix § 11.§ 5.

WindowMethodTokens/sTokens/NFETime/NFE (ms)Accuracy (%)TokensWall (s)NFE
W=8W = 8Linear-SS94.932.61027.49668.39257.22.70998.53
Quadratic-SS86.253.41439.58368.39257.12.98175.31
Packed-SS100.602.84228.24768.39257.22.55690.49
Native77.212.11927.44259.67252.43.269119.11
W=16W = 16Linear-SS98.852.79228.24366.57259.92.62993.09
Quadratic-SS41.604.07497.93466.57260.46.26063.92
Packed-SS100.303.03030.20966.57259.92.59185.77
Native80.592.22027.55259.51253.63.147114.21

Table 8. SetDLM absolute measurements on GSM8K. LTR verification and native decoding on all 1,319 test questions. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each window.

We also test the native SetDLM decoding schedule, which is diffusion decoding instead of speculative decoding. At W=8W = 8 and W=16W = 16, Packed-SS achieves higher throughput than Native, with accuracy higher by 8.72 and 7.05 percentage points, respectively.

Decoding at temperature 1

We further evaluate the sampling procedure of § 3.2 at temperature 1, where draft tokens are verified probabilistically. We use the same five datasets and both Nemotron scales as in greedy decoding, with block size B=32B = 32 for Linear-SS and Packed-SS and B=8B = 8 for Quadratic-SS.

Scope and baselines. Linear-SS with speculative sampling is the standard stochastic setting ([30, 10]). However, prior empirical evaluations of Quadratic-SS are reported under greedy decoding ([37, 43] (Fu et al. 2026)). We therefore use Linear-SS as the primary baseline at temperature 1. For completeness, we extend Quadratic-SS to rejection sampling by applying the same probabilistic verification to its selected branch as a secondary comparison. More details are in Appendix § 10.§ 4.

Results. Results are presented in Figure 4, with numerical results in Table 7. Compared with Linear-SS, Packed-SS achieves higher throughput in all ten settings, improving throughput by 2.0–6.3%, with all ten 95% confidence intervals for the throughput ratio above one. The improvement again comes from increased token yield: Packed-SS generates 14.8–21.5% more tokens per NFE, which outweighs the 11.9–15.4% increase in decoder time per NFE.

Efficiency and accuracy at temperature 1

Figure 4. Efficiency and accuracy at temperature 1. Efficiency metrics are ratios to Linear-SS, accuracy differences are in percentage points. Error bars show 95% paired bootstrap confidence intervals. WritingBench measures speed only.

TaskMethodTokens/sTokens/NFETime/NFE (ms)Accuracy (%)TokensWall (s)NFE
Nemotron 3B
MBPPLinear-SS120.382.50820.83468.14102.70.85340.95
Quadratic-SS129.013.32825.80069.69108.70.84332.67
Packed-SS125.062.93323.45468.78101.80.81434.72
HumanEvalLinear-SS169.583.52620.79266.8996.90.57127.49
Quadratic-SS170.634.34825.48467.2995.60.56122.00
Packed-SS173.204.04823.37468.3597.70.56424.12
GSM8KLinear-SS137.382.97721.67084.15197.71.43966.41
Quadratic-SS140.653.77626.84384.84199.21.41652.76
Packed-SS140.073.42824.47183.47198.41.41657.88
MATH-500Linear-SS150.133.26621.75353.80518.23.451158.66
Quadratic-SS149.054.10227.52152.00515.63.459125.68
Packed-SS155.493.84124.70152.80523.93.369136.41
WritingBenchLinear-SS74.961.57521.012—498.86.654316.68
Quadratic-SS78.202.14127.382—497.06.355232.09
Packed-SS78.041.89024.222—500.16.409264.58
Nemotron 8B
MBPPLinear-SS84.452.31627.42676.76129.91.53856.08
Quadratic-SS91.223.06233.57274.98130.41.42942.58
Packed-SS89.562.75230.72275.36133.61.49148.54
HumanEvalLinear-SS107.782.94427.31670.70120.31.11640.87
Quadratic-SS111.483.71433.31771.28120.61.08232.48
Packed-SS114.173.49030.56771.34118.71.04034.01
GSM8KLinear-SS85.662.34127.32487.34351.04.098149.97
Quadratic-SS91.863.10933.84086.50348.13.790111.99
Packed-SS91.082.79430.67885.29345.03.787123.46
MATH-500Linear-SS90.912.57628.33854.20657.37.230255.14
Quadratic-SS95.733.39135.41755.20658.56.879194.22
Packed-SS95.473.04731.91954.20664.66.962218.10
WritingBenchLinear-SS52.631.43327.225—500.99.518349.59
Quadratic-SS53.341.94036.370—496.59.308255.94
Packed-SS55.391.74131.424—498.89.006286.58

Table 7. Nemotron absolute measurements at temperature 1. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each model and task.

Compared with Quadratic-SS, Packed-SS shows comparable performance in terms of throughput. Its throughput point estimate is higher in four settings and lower in six. The paired 95% confidence intervals lie above one in three settings, below one in two, and include one in the remaining five.

Discussion and Limitations

We presented Packed-SS, which overlaps drafting and verification in one forward pass per iteration while evaluating only O(W)O(W) positions. Under greedy decoding and a monotonicity assumption, we showed that Packed-SS requires no more forward passes than Linear-SS to generate a prescribed number of tokens from the same prompt at the same width. Experiments across multiple tasks and model families demonstrate that Packed-SS can improve throughput over existing self-speculative decoding schedules.

Several limitations and directions for future work remain. Our single-call implementation requires shared-weight models supporting parallel drafting and autoregressive verification. Combining this schedule with drafters based on intermediate-layer representations remains unexplored. The current efficiency analysis relies on an idealized assumption; extending it to weaker conditions and stochastic sampling would broaden its applicability. Finally, preparing only a subset of the continuation branches used by Quadratic-SS could provide intermediate designs between Packed-SS and Quadratic-SS, potentially improving upon the current algorithm.

Acknowledgements

C. Cai is supported by the NSF CAREER Award CCF-2541600 and the NSF grant DMS-2515333.

Notation

Table 1 collects the symbols used in Section 3. In left-to-right (LTR) decoding, dt(i)d_t^{(i)} is indexed after PtP_t, while zt(i)z_t^{(i)} and vt(i)v_t^{(i)} are indexed after Pt−1P_{t-1}. In the any-order procedure, position indices are fixed output positions after the prompt.

Sizes and indices
ttdecoding iteration, starting at 1
i,ji, jtoken position: relative in LTR, fixed in any-order decoding
r,sr, srank in the verification order (any-order setting)
WWspeculative width: the fixed number of future positions proposed by the LTR drafter, W>1W > 1, and the size of the active window in the any-order setting. The schedule and Theorem 3.2 are stated in WW alone
BBNemotron’s block size, B=W+1B = W + 1: the last prefix token and WW draft positions
utu_tnumber of unresolved positions in the active window, ut=∣Ut∣≤Wu_t = \lvert U_t\rvert\le W (any-order setting)
κt\kappa_tavailable draft-prefix length in verification order πt\pi_t, with 0≤κt≤ut0 \le\kappa_t \le u_t (any-order setting)
hth_tnumber of consecutive accepted draft tokens in verification order
AtA_tnumber of tokens accepted before applying the stopping rule; At=ht+1A_t = h_t + 1 for the next-token verifier and At=min⁡(ht+1,ut)A_t = \min(h_t + 1, u_t) for the any-order same-position verifier (Appendix C)
aaemitted token count after applying the stopping rule, a≤Ata \le A_t (LTR)
LLmaximum number of generated tokens, excluding the prompt
EElast output position to generate in any-order decoding; initially LL, reduced to the position of an accepted EOS
nt=∣Pt∣n_t = \lvert P_t\rvertLTR prefix length, including the prompt
Decoder state
P,PtP, P_tprompt; verified prefix (LTR) or prompt followed by accepted position–token pairs (any-order), with P0=PP_0 = P
DtD_tdraft available for verification at iteration t+1t + 1, with D0=∅D_0 = \varnothing; indexed after PtP_t in LTR, with ∣Dt∣≤W\lvert D_t\rvert\le W. In any-order decoding, a position–token map on JtJ_t
ZtZ_tnew proposal: WW positions after the input prefix in LTR, or positions in UtU_t in any-order decoding
vt(i)v_t^{(i)}verifier token at position ii
Ut,JtU_t, J_tactive unresolved positions; positions retained for the next draft, Jt⊆UtJ_t \subseteq U_t (any-order)
πt\pi_tverification order used by iteration tt
σ\sigmageneration order of an any-order model
CtC_ttokens accepted in iteration tt after applying the stopping rule; tokens in LTR or ordered position–token pairs in any-order decoding
Distributions
V,θ\mathcal{V}, \thetavocabulary; shared weights
pθp_\thetatarget token distribution given the prompt, accepted tokens, and earlier draft tokens
qt(i)q_t^{(i)}drafter conditional at position ii: relative to its conditioning prefix in LTR, fixed in any-order decoding
qt(j)q_t^{(j)}fresh drafter conditional qt(j)(⋅∣Pt−1)q_t^{(j)}(\cdot\mid P_{t-1}), before prefix advancement (LTR)
Qt(i)Q_t^{(i)}distribution stored with draft dt(i)d_t^{(i)}; Qt(i)=qt(i+ai)Q_t^{(i)} = q_t^{(i+a_i)} after retaining the LTR suffix
pt(i)p_t^{(i)}verifier conditional at position ii after Pt−1P_{t-1} (LTR)
Analysis (Theorem 3.2)
ek∗e_k^*correctness threshold for output position kk; ek∗=ke_k^* = k denotes no correct draft prediction before verification (Assumption 3.1)
xk,xˉk(c)x_k, \bar{x}_k(c)target greedy token at output position kk; drafter’s prediction from the first cc verified output tokens
ℓt,ct\ell_t, c_tverified output lengths after tt Linear iterations and tt Packed forward passes, respectively (Appendix B.1)
F(c)F(c)verified output length after fresh drafting and verification from cc verified output tokens, before stopping (Lemma B.1)
NFE⁡S(L)\operatorname{NFE}_{\mathcal{S}}(L)total model calls used by schedule S\mathcal{S} to generate LL tokens after the prompt

Table 1. Symbols. Sizes and indices, decoder state, and distributions.

Proofs

Pathwise dominance

We prove Theorem 3.2 using two lemmas. The first describes the progress of fresh drafting and verification; the second compares this progress with two forward passes of Packed-SS. The comparisons below concern verified output lengths, excluding the prompt, before applying EOS or length truncation; stopping is handled at the end of the theorem’s proof.

Lemma B.1 (Monotonicity of verified output length). Count verified output tokens after the prompt by cc, and define

F(c)=min⁡{c+W+1,min⁡{k>c:ck⋆>c}},(6)F(c) = \min\{c + W + 1,\min\{k > c : c_k^\star> c\}\}, \tag*{(6)}

with the minimum of an empty set read as +∞+\infty. Then FF is nondecreasing. Under Assumption 3.1, F(c)F(c) is the verified output length after freshly drafting WW tokens from a context of cc verified output tokens and verifying that draft, for any such context evaluated by either schedule. Thus, F(c)−cF(c) - c counts the newly accepted tokens, including the correction or bonus token.

Proof. We first identify the number of tokens accepted by fresh drafting and verification. Fix a context of cc verified output tokens. If the draft contains an incorrect proposal, let kk be its first incorrect position. By Assumption 3.1, this is the first position in c+1,…,c+Wc + 1,\ldots,c + W with ck⋆>cc_k^\star> c. Verification accepts the earlier proposals and adds the corrected token at kk, giving a verified output length of kk. If all WW proposals are correct, verification accepts them and adds a bonus token at c+W+1c + W + 1. These two cases give the expression for F(c)F(c) in Equation (6).

We next prove monotonicity. Take c′≥cc' \ge c. If c′≥F(c)c' \ge F(c), then

F(c′)≥c′+1>F(c).F(c') \ge c' + 1 > F(c).

It remains to consider c′<F(c)c' < F(c). For every position kk satisfying c′<k<F(c)c' < k < F(c), the definition of F(c)F(c) gives

ck⋆≤c≤c′.c_k^\star\le c \le c'.

Thus, the first position with ck⋆>c′c_k^\star> c' cannot occur before F(c)F(c). The other term in the minimum defining F(c′)F(c') also satisfies

c′+W+1≥c+W+1≥F(c).c' + W + 1 \ge c + W + 1 \ge F(c).

Both terms are therefore at least F(c)F(c), proving F(c′)≥F(c)F(c') \ge F(c).

Lemma B.2 (Verified output length after two forward passes). *Under Assumption 3.1, if a Packed-SS iteration begins with cc verified output tokens and the next iteration ends with c′′c'', then c′′≥F(c)c'' \ge F(c). *

Proof. Let c′c' be the verified output length after the first forward pass. During this pass, Packed-SS verifies its stored draft and produces new proposals for positions c+1,…,c+Wc + 1,\ldots,c + W, using the context of length cc. Every pass accepts at least one token, so c′>cc' > c.

If c′≥F(c)c' \ge F(c), the desired inequality already holds, since the next pass can only increase the verified output length. Otherwise, consider the positions

c′+1,…,F(c)−1.c' + 1,\ldots,F(c) - 1.

The new proposals at these positions are correct by the definition of F(c)F(c). They also belong to the newly drafted block, because F(c)−1≤c+WF(c) - 1 \le c + W. Since these positions lie beyond the updated prefix, their proposals are retained for the next pass.

The next pass therefore accepts all of these retained proposals. Its correction or bonus token is added at a position at least F(c)F(c), so its final verified output length satisfies c′′≥F(c)c'' \ge F(c). This argument also covers an empty range of retained positions, since the next pass still accepts at least one token.

Proof of Theorem 3.2. We first compare the two schedules without EOS or length truncation. Let ℓt\ell_t be the number of verified output tokens after tt iterations of Linear-SS, and let ctc_t be the corresponding number after tt forward passes of Packed-SS. Both counts exclude the prompt, and c0=ℓ0=0c_0 = \ell_0 = 0. A Linear-SS iteration drafts from its current verified prefix and then verifies that draft, so

ℓt+1=F(ℓt).\ell_{t+1} = F(\ell_t).

We show by induction that c2t≥ℓtc_{2t} \ge\ell_t for every t≥0t \ge0. The claim holds at t=0t = 0. Suppose it holds for some tt. Then

c2t+2≥F(c2t)≥F(ℓt)=ℓt+1.c_{2t+2} \ge F(c_{2t}) \ge F(\ell_t) = \ell_{t+1}.

The first inequality follows from Lemma B.2; the second follows from the induction hypothesis and the monotonicity in Lemma B.1. This proves the induction step. The argument includes initialization: the first Packed-SS pass starts with an empty draft and generates one target token, so Lemma B.2 applies from c0=0c_0 = 0.

Now fix L>0L > 0, and let TT be the first Linear-SS iteration for which ℓT≥L\ell_T \ge L. Since each Linear-SS iteration uses two forward passes, its total cost is 2T2T. The inequality c2T≥ℓT≥Lc_{2T} \ge\ell_T \ge L shows that Packed-SS reaches the same output length within those 2T2T passes. Therefore,

NFE⁡Packed-SS(L)≤2T=NFE⁡Linear-SS(L).\operatorname{NFE}_{\mathrm{Packed\text{-}SS}}(L) \le2T = \operatorname{NFE}_{\mathrm{Linear\text{-}SS}}(L).

For L=0L = 0, neither schedule makes a call.

Finally, both schedules produce the same greedy continuation. If that continuation contains an EOS token, its first occurrence is therefore at the same position for both schedules. Apply the same comparison to that position or the length limit LL, whichever is reached first. Truncating the final block changes the returned output but does not remove the forward pass that produced it. The same bound thus holds with EOS and length stopping. □

Exact sampling

Algorithm 2 gives the full LTR sampling procedure corresponding to § 3.2. Compared with Algorithm 1, the decoder stores the sampling distribution Qt−1(i)Q_{t-1}^{(i)} with each draft token and replaces greedy matching with rejection sampling.

Algorithm 2 Packed-SS with LTR rejection sampling

Require: prompt PP, speculative width WW, output length limit LL, verifier pθp_\theta, drafter qθq_\theta

  1. P0←P, D0,Q0←∅, t←1P_0 \leftarrow P,\ D_0,Q_0 \leftarrow\varnothing,\ t \leftarrow1

  2. while ∣Pt−1∣−∣P∣<L|P_{t-1}| - |P| < L and (t=1(t = 1 or the last token of Pt−1P_{t-1} is not EOS) do

    In one model call:

  3. pt(i)←pθ(⋅∣Pt−1,dt−1(<i))p_t^{(i)} \leftarrow p_\theta(\cdot\mid P_{t-1},d_{t-1}^{(<i)}) for i=1,…,∣Dt−1∣+1i = 1,\ldots,|D_{t-1}|+1 ▷ Verifier distributions

  4. qt(j)←qθ(j)(⋅∣Pt−1)q_t^{(j)} \leftarrow q_\theta^{(j)}(\cdot\mid P_{t-1}) for j=1,…,Wj = 1,\ldots,W ▷ Drafter proposals

  5. ui∼Unif⁡(0,1), αi←min⁡{1,pt(i)(dt−1(i))/Qt−1(i)(dt−1(i))}u_i \sim\operatorname{Unif}(0,1),\ \alpha_i \leftarrow\min\{1,p_t^{(i)}(d_{t-1}^{(i)})/Q_{t-1}^{(i)}(d_{t-1}^{(i)})\} for i=1,…,∣Dt−1∣i = 1,\ldots,|D_{t-1}| ▷ Acceptance tests

  6. ht←max⁡{0≤h≤∣Dt−1∣:ui<αi for all i≤h}h_t \leftarrow\max\{0 \le h \le|D_{t-1}|:u_i < \alpha_i\text{ for all }i \le h\} ▷ Count accepted tokens

  7. c∼R,R∝{[pt(ht+1)−Qt−1(ht+1)]+,ht<∣Dt−1∣,pt(∣Dt−1∣+1),ht=∣Dt−1∣.c \sim R,\quad R \propto\begin{cases}[p_t^{(h_t+1)}-Q_{t-1}^{(h_t+1)}]_+,&h_t<|D_{t-1}|,\\p_t^{(|D_{t-1}|+1)},&h_t=|D_{t-1}|.\end{cases} ▷ Correction or bonus

  8. Ct←(dt−1(1),…,dt−1(ht),c), At←ht+1C_t \leftarrow(d_{t-1}^{(1)},\ldots,d_{t-1}^{(h_t)},c),\ A_t \leftarrow h_t+1 ▷ Add verifier token

  9. a←min⁡{At,L+∣P∣−∣Pt−1∣}∪{1≤i≤At:Ct(i)=EOS}a \leftarrow\min\{A_t,L+|P|-|P_{t-1}|\}\cup\{1\le i\le A_t:C_t^{(i)}=\mathrm{EOS}\} ▷ Ending condition

  10. Ct←(Ct(1),…,Ct(a))C_t \leftarrow(C_t^{(1)},\ldots,C_t^{(a)}) ▷ Prune by stopping index

  11. Pt←(Pt−1,Ct)P_t \leftarrow(P_{t-1},C_t) ▷ Append to prefix

  12. Qt(j)∼qt(a+j), dt(j)∼Qt(j)Q_t^{(j)} \sim q_t^{(a+j)},\ d_t^{(j)} \sim Q_t^{(j)} for j=1,…,max⁡(0,W−a)j=1,\ldots,\max(0,W-a) ▷ Sample retained draft

  13. Dt←(dt(1),…,dt(max⁡(0,W−a)))D_t \leftarrow(d_t^{(1)},\ldots,d_t^{(\max(0,W-a))}) ▷ Prepare next draft

  14. t←t+1t \leftarrow t+1

  15. end while

  16. Output: Pt−1P_{t-1}

We now prove Theorem 3.3. First, we show that the acceptance and correction steps produce the correct target distribution for one token. Then, we show that each draft token still follows its stored sampling distribution when it is checked.

Conditioning information. Fix the prompt. Let Gt\mathcal{G}_t collect the information used for conditioning immediately before sampling the draft DtD_t. It includes the verified prefix Pt−1P_{t-1} at the start of call tt, the draft Dt−1D_{t-1} checked in that call, and the distributions Qt−1Q_{t-1} used to sample that draft. It also includes the tokens CtC_t accepted after applying the stopping rule and the new drafter distributions qt(1),…,qt(W)q_t^{(1)},\ldots,q_t^{(W)}, computed from Pt−1P_{t-1}. Define

Gt:=(Pt−1,Dt−1,Qt−1,Ct,(qt(j))j=1W).\mathcal{G}_t := \left(P_{t-1},D_{t-1},Q_{t-1},C_t,\left(q_t^{(j)}\right)_{j=1}^{W}\right).

Conditional on Gt\mathcal{G}_t, the advance a=∣Ct∣a = |C_t|, the updated prefix PtP_t, and the stored distributions Qt(i)Q_t^{(i)} for the draft tokens are fixed. All draft sampling, acceptance tests, and correction draws use independent random numbers.

Write ut+1,ju_{t+1,j} for the uniform random number used to test draft token dt(j)d_t^{(j)} in call t+1t+1. For 1≤i≤∣Dt∣1 \le i \le|D_t|, add the earlier draft tokens and the random numbers used to test them:

Gt,i:=(Gt,(dt(j),ut+1,j)1≤j<i).\mathcal{G}_{t,i} := \left(\mathcal{G}_t, (d_t^{(j)},u_{t+1,j})_{1 \le j < i}\right).

Conditional on Gt,i\mathcal{G}_{t,i}, the preceding acceptance decisions are fixed, so whether verification reaches position ii is also fixed.

Lemma B.3 (One-token correction using the original sampling distribution). Consider the ii-th draft token reached by left-to-right verification in call t+1t+1. Let H=(Pt,dt(<i))H = (P_t,d_t^{(<i)}) be the verified prefix, including the draft tokens already accepted in this call. Let P(x)=p(x∣H)P(x) = p(x \mid H) be the target distribution for the next token. Let D=dt(i)D = d_t^{(i)} be the draft token and Q=Qt(i)Q = Q_t^{(i)} the distribution used to sample it. Assume Pr⁡(D=x∣Gt,i)=Q(x)\Pr(D = x \mid\mathcal{G}_{t,i}) = Q(x) for every token xx. Accept DD with probability min⁡{1,P(D)/Q(D)}\min\{1,P(D)/Q(D)\}, and otherwise emit an independent draw from the normalized positive part of P−QP-Q. Then, given Gt,i\mathcal{G}_{t,i}, the emitted token has distribution PP.

Proof. Condition on Gt,i\mathcal{G}_{t,i}, so that PP and QQ are fixed. Let XX denote the emitted token and AA the event that the draft token is accepted. We compute the probability of emitting a given token xx through each of the two possible branches.

The acceptance branch contributes

Pr⁡(X=x,A∣Gt,i)=min⁡{P(x),Q(x)}.\Pr(X = x, A \mid\mathcal{G}_{t,i}) = \min\{P(x),Q(x)\}.

For Q(x)>0Q(x) > 0, this follows by multiplying the proposal probability Q(x)Q(x) by the acceptance probability min⁡{1,P(x)/Q(x)}\min\{1,P(x)/Q(x)\}. For Q(x)=0Q(x) = 0, both sides are zero. It follows that the total acceptance probability is

β:=Pr⁡(A∣Gt,i)=∑xmin⁡{P(x),Q(x)}.\beta:= \Pr(A \mid\mathcal{G}_{t,i}) = \sum_x \min\{P(x),Q(x)\}.

The rejection probability is

1−β=∑x(P(x)−min⁡{P(x),Q(x)})=∑x[P(x)−Q(x)]+.1-\beta= \sum_x\left(P(x)-\min\{P(x),Q(x)\}\right) = \sum_x [P(x)-Q(x)]_+.

When β<1\beta< 1, this is exactly the normalizing constant of the correction distribution. The rejection branch therefore contributes

Pr⁡(X=x,Ac∣Gt,i)=(1−β)[P(x)−Q(x)]+1−β=[P(x)−Q(x)]+.\Pr(X = x, A^c \mid\mathcal{G}_{t,i}) = (1-\beta)\frac{[P(x)-Q(x)]_+}{1-\beta} = [P(x)-Q(x)]_+.

Adding the two contributions yields

Pr⁡(X=x∣Gt,i)=min⁡{P(x),Q(x)}+[P(x)−Q(x)]+=P(x).\Pr(X = x \mid\mathcal{G}_{t,i}) = \min\{P(x),Q(x)\} + [P(x)-Q(x)]_+ = P(x).

If β=1\beta= 1, then P=QP=Q and rejection has probability zero, so the same conclusion holds without a correction draw. Also, Q(D)>0Q(D) > 0 with probability one because DD is sampled from QQ, so the acceptance test never divides by zero. Neither distribution needs to assign positive probability to every token. □

This is the standard speculative-sampling rule ([30, 10]). The next lemma shows that each draft token follows its stored sampling distribution when checked, as required by Lemma B.3.

Lemma B.4 (Distribution of a draft token when checked). For each draft position 1≤i≤∣Dt∣1 \le i \le|D_t| reached by verification in call t+1t+1, and every token xx,

Pr⁡(dt(i)=x∣Gt,i)=Qt(i)(x).(7)\Pr(d_t^{(i)} = x \mid\mathcal{G}_{t,i}) = Q_t^{(i)}(x). \tag*{(7)}

Proof. Condition on Gt\mathcal{G}_t. The advance aa, the positions to retain, and the stored distributions Qt(j)=q(a+j)Q_t^{(j)} = q^{(a+j)} are fixed. Algorithm 2 samples the tokens in DtD_t independently from these distributions. Thus, dt(i)d_t^{(i)} has distribution Qt(i)Q_t^{(i)} and is independent of the earlier draft tokens given Gt\mathcal{G}_t.

By definition, Gt,i\mathcal{G}_{t,i} adds only the earlier draft tokens and the random numbers used to test them to Gt\mathcal{G}_t. These random numbers are independent of (Gt,Dt)(\mathcal{G}_t,D_t). Conditioning on this additional information therefore leaves the distribution of dt(i)d_t^{(i)} unchanged, proving Equation (7). Whether verification reaches position ii is determined by Gt,i\mathcal{G}_{t,i}, so the same identity holds at every position that is checked.

Proof of Theorem 3.3. Fix the prompt PP and consider Algorithm 2 without EOS or length stopping. Let XkX_k be its kk-th output token after the prompt. We prove by induction on mm that

Pr⁡(X1:m=x1:m∣P)=∏k=1mpθ(xk∣P,x<k).\Pr(X_{1:m}=x_{1:m}\mid P)=\prod_{k=1}^{m}p_\theta(x_k\mid P,x_{<k}).

Since D0=∅D_0=\varnothing, the first call draws X1X_1 directly from pθ(⋅∣P)p_\theta(\cdot\mid P), proving the claim for m=1m=1.

Suppose the identity holds for m=k−1m=k-1, where k≥2k\ge2. Let t+1t+1 be the call that produces XkX_k. Its position within that call is i=k−(∣Pt∣−∣P∣)i=k-(|P_t|-|P|). Thus, 1≤i≤∣Dt∣+11\le i\le|D_t|+1, with i=∣Dt∣+1i=|D_t|+1 denoting the bonus position. Let Hk\mathcal{H}_k collect tt, ii, Gt\mathcal{G}_t, the earlier draft tokens, and the random numbers used to test them:

Hk:=(t,i,Gt,(dt(j),ut+1,j)1≤j<i).\mathcal{H}_k := \left(t,i,\mathcal{G}_t,\left(d_t^{(j)},u_{t+1,j}\right)_{1\le j<i}\right).

The first i−1i-1 draft tokens have been accepted, so

(P,X<k)=(Pt,dt(<i)),pt+1(i)=pθ(⋅∣P,X<k).(P,X_{<k})=(P_t,d_t^{(<i)}),\qquad p_{t+1}^{(i)}=p_\theta(\cdot\mid P,X_{<k}).

First suppose i≤∣Dt∣i\le|D_t|, so the algorithm checks a draft token. For fixed tt and ii, Gt,i\mathcal{G}_{t,i} determines whether the preceding draft tokens are accepted and whether ∣Pt∣−∣P∣+i=k|P_t|-|P|+i=k. Hence Lemma B.4 gives Pr⁡(dt(i)=x∣Hk)=Qt(i)(x)\Pr(d_t^{(i)}=x\mid\mathcal{H}_k)=Q_t^{(i)}(x). Applying Lemma B.3 with target pt+1(i)p_{t+1}^{(i)} and proposal Qt(i)Q_t^{(i)} yields

Pr⁡(Xk=x∣Hk)=pθ(x∣P,X<k).\Pr(X_k=x\mid\mathcal{H}_k)=p_\theta(x\mid P,X_{<k}).

If i=∣Dt∣+1i=|D_t|+1, the algorithm draws a bonus token directly from pt+1(i)p_{t+1}^{(i)}, so the same equality holds.

Since (P,X<k)(P,X_{<k}) is determined by Hk\mathcal{H}_k, the law of total expectation gives

Pr⁡(Xk=x∣P,X<k)=E[Pr⁡(Xk=x∣Hk)∣P,X<k]=pθ(x∣P,X<k).\begin{aligned} \Pr(X_k=x\mid P,X_{<k})&=\mathbb{E}\left[\Pr(X_k=x\mid\mathcal{H}_k)\mid P,X_{<k}\right]\\ &=p_\theta(x\mid P,X_{<k}). \end{aligned}

Multiplying by Pr⁡(X<k=x<k∣P)\Pr(X_{<k}=x_{<k}\mid P) and applying the induction hypothesis proves the identity for m=km=k.

Now restore stopping. Let

τ:=min⁡({1≤k≤L:Xk=EOS}∪{L}).\tau:=\min\left(\{1\le k\le L:X_k=\mathrm{EOS}\}\cup\{L\}\right).

Using the same random numbers, the stopped algorithm produces the same tokens through position τ\tau and returns (P,X1:τ)(P,X_{1:\tau}). Taking m=Lm=L in the identity above shows that X1:LX_{1:L} has the AR distribution. Applying the same stopping rule therefore gives the same distribution of returned continuations.

Greedy decoding. For greedy decoding, let each model assign probability one to the argmax token chosen by its fixed tie-breaking rule, and zero to all other tokens. Then Theorem 3.3 gives the greedy correctness of Algorithm 1, including EOS and length stopping.

Any-Order Extension

A later token can be easier to predict than an earlier one. In LTR verification, the first mismatch ends acceptance for that call, so an uncertain early token can prevent later, easier tokens from being accepted. Checking tokens that are more likely to be accepted first may therefore let a call accept more draft tokens.

An any-order AR verifier enables this change: it conditions on tokens earlier in the chosen verification order, even when they occur later in the output sequence. For a permutation σ\sigma, write xσ(<r)=(xσ(1),…,xσ(r−1))x_{\sigma(<r)} = (x_{\sigma(1)}, \ldots, x_{\sigma(r-1)}) for the tokens earlier in that order. The model defines

pθ(x1:L∣P)=∏r=1Lpθ(xσ(r)∣P,xσ(<r);σ).p_\theta(x_{1:L} \mid P) = \prod_{r=1}^{L} p_\theta(x_{\sigma(r)} \mid P, x_{\sigma(<r)}; \sigma).

The order specifies the conditionals, so different orders can produce different greedy outputs.

Self-speculation with separate draft and verification passes already appears in Any-Subset Speculative Decoding (ASSD) [26]. ASSD uses the same model to draft tokens in parallel and verify them in a second pass, matching the two-call schedule we call Linear-SS. Prism also describes self-speculative decoding and, separately, choosing the next position by confidence [31 Chapter 5]. Here we extend the packed schedule to such verification orders.

Procedure. In LTR decoding, accepted tokens form a prefix, and all unresolved positions lie after it. With any-order verification, a later position can be accepted while an earlier one remains unresolved. We therefore keep each accepted or draft token associated with its output position, numbered from 1 after the prompt. At the start of iteration tt, Pt−1P_{t-1} contains the prompt and accepted position–token pairs in acceptance order, and Dt−1D_{t-1} maps positions to available draft tokens. Position superscripts now refer to these fixed output positions rather than offsets after a prefix.

Each iteration starts its active window at the leftmost unresolved position and includes up to WW consecutive output positions. Positions whose tokens have already been accepted are skipped, leaving the set UtU_t of ut=∣Ut∣u_t = |U_t| unresolved positions to process. Some positions in UtU_t may have no draft token yet; in particular, D0=∅D_0 = \varnothing in the first iteration.

The order policy then chooses a permutation πt\pi_t of UtU_t using information available before the model call, and this order remains fixed throughout the call. Here, πt(r)\pi_t(r) is the output position checked at rank rr; changing the order does not move a draft token to a different output position. For example, confidence ordering checks positions in decreasing order of scores computed in an earlier iteration (see Appendix § 10.6 for the scores used in our experiments).

Verification follows πt\pi_t, using draft tokens at earlier ranks as context. Write dom⁡(Dt−1)\operatorname{dom}(D_{t-1}) for the positions with available draft tokens, and let κt\kappa_t be the number of consecutive entries of πt\pi_t in this set before the first missing draft token. A same-position verifier predicts a token at each queried output position. Writing pθ,jp_{\theta,j} for its distribution at position jj under the chosen order, its prediction at rank rr is

vt(πt(r))=arg⁡max⁡x  pθ,πt(r)(x∣Pt−1,(πt(s),dt−1(πt(s)))s<r),(8)v_t^{(\pi_t(r))} = \underset{x}{\arg\max}\; p_{\theta,\pi_t(r)}\left(x \mid P_{t-1}, \left(\pi_t(s), d_{t-1}^{(\pi_t(s))}\right)_{s<r}\right), \tag*{(8)}

for r=1,…,min⁡(κt+1,ut)r = 1, \ldots, \min(\kappa_t + 1, u_t). Accept matching draft tokens in this order until the first mismatch or missing draft token, then accept the verifier’s prediction at that position. If all utu_t positions have draft tokens and all match, accept those tokens without an extra token outside the window. Thus, if hth_t draft tokens match consecutively, the number accepted before applying the stopping rule is At=min⁡(ht+1,ut)A_t = \min(h_t + 1, u_t).

In the same call as verification, Packed-SS drafts every position in UtU_t from Pt−1P_{t-1}. After acceptance, predictions at positions that remain unresolved become the next draft; predictions at newly accepted positions are discarded. For Quadratic-SS, the proposal branches follow πt\pi_t instead of textual order, and the branch selected after verification supplies the next draft at the remaining unresolved positions.

Algorithm 3 gives the formal procedure for Packed-SS with any-order greedy decoding. Let EE be the last output position to generate. It starts at the length limit LL; accepting EOS at position jj changes it to min⁡(E,j)\min(E,j). The algorithm fills all remaining unresolved positions up to EE before returning. For an example with confidence-ordered verification on SetDLM, see Appendix § 11.§ 5.

In Algorithm 3, dom⁡(Pt)\operatorname{dom}(P_t) denotes the output positions with accepted tokens in PtP_t, excluding the prompt, and Pt(j)P_t(j) is the token stored at position jj. The restriction ↾≤E\mathord{\upharpoonright}_{\le E} removes output pairs beyond EE, preserving the prompt and the order of the remaining pairs. The completed sequence is returned with the prompt included and its continuation restored to textual order.

Algorithm 3 Packed-SS with any-order greedy decoding
Require: prompt $P$, speculative width $W \ge 1$, output length limit $L > 0$, verifier $p_\theta$, drafter $q_\theta$, order policy
1: $P_0 \leftarrow P$, $D_0 \leftarrow \varnothing$, $E \leftarrow L$, $t \leftarrow 1$
2: while $\{1,\ldots,E\} \nsubseteq \operatorname{dom}(P_{t-1})$ do
3: $\ell \leftarrow \min(\{1,\ldots,E\} \setminus \operatorname{dom}(P_{t-1}))$ ▷ Leftmost unresolved position
4: $U_t \leftarrow \{\ell,\ldots,\min(E,\ell+W-1)\} \setminus \operatorname{dom}(P_{t-1})$, $u_t \leftarrow |U_t|$ ▷ Active positions
5: Choose $\pi_t$ over $U_t$ using the order policy and information available before this call
6: $\kappa_t \leftarrow \max\{0 \le k \le u_t : \pi_t(r) \in \operatorname{dom}(D_{t-1}) \text{ for all } r \le k\}$ ▷ Available draft prefix
In one model call:
7: Compute $v_t^{(\pi_t(r))}$ by {xref}`Equation (8)` for $r=1,\ldots,\min(\kappa_t+1,u_t)$ ▷ Verifier predictions
8: $z_t^{(j)} \leftarrow \arg\max_x q_\theta^{(j)}(x \mid P_{t-1})$ for $j \in U_t$ ▷ Drafter proposals
end
9: $h_t \leftarrow \max\{0 \le h \le \kappa_t : d_{t-1}^{(\pi_t(r))}=v_t^{(\pi_t(r))} \text{ for all } r \le h\}$ ▷ Count matches
10: $A_t \leftarrow \min(h_t+1,u_t)$, $C_t \leftarrow \bigl((\pi_t(r),v_t^{(\pi_t(r))})\bigr)_{r=1}^{A_t}$ ▷ Collect accepted tokens
11: $E \leftarrow \min(\{E\}\cup\{j:(j,\mathrm{EOS})\in C_t\})$ ▷ Ending condition
12: $C_t \leftarrow C_t|_{\le E}$, $P_t \leftarrow P_{t-1}|_{\le E}$ ▷ Discard positions after $E$
13: $P_t \leftarrow (P_t,C_t)$ ▷ Append accepted tokens
14: $J_t \leftarrow \{j\in U_t\setminus\operatorname{dom}(P_t):j\le E\}$ ▷ Retained draft positions
15: $D_t \leftarrow \{j\mapsto z_t^{(j)}:j\in J_t\}$ ▷ Prepare next draft
16: $t \leftarrow t+1$
17: end while
18: Output: $(P,P_{t-1}(1),\ldots,P_{t-1}(E))$ ▷ Textual order

Proposition C.1 (Greedy correctness under the chosen verification orders). Fix a run of Algorithm 3. Consider a serial reference that starts from the same prompt and, in iteration tt, generates AtA_t tokens greedily at positions πt(1),…,πt(At)\pi_t(1),\ldots,\pi_t(A_t), conditioning each prediction on its own Pt−1P_{t-1} and the tokens generated earlier in that group. The reference uses Algorithm 3’s verifier, tie-breaking rule, and rules for updating the accepted tokens and handling EOS. Then the two procedures return the same output sequence.

Proof of Proposition C.1. We show by induction that the two procedures have the same PtP_t and EE after every iteration. The claim holds initially, since both start from the prompt PP with E=LE=L.

Suppose the claim holds before iteration tt. Let yry_r be the serial reference’s token at position πt(r)\pi_t(r) in this iteration. We prove, in verification order, that

yr=vt(πt(r)),r=1,…,At.y_r=v_t^{(\pi_t(r))},\qquad r=1,\ldots,A_t.

At rank 1, both predictions condition only on Pt−1P_{t-1}, so they agree. Now take r>1r>1 and suppose agreement holds at all earlier ranks. Since r≤At=min⁡(ht+1,ut)r\le A_t=\min(h_t+1,u_t), every s<rs<r satisfies s≤hts\le h_t. These earlier draft tokens match their verifier predictions by the definition of hth_t. Thus,

dt−1(πt(s))=vt(πt(s))=ys,s<r.d_{t-1}^{(\pi_t(s))}=v_t^{(\pi_t(s))}=y_s,\qquad s<r.

Substituting these equalities into Equation (8) gives

yr=arg⁡max⁡xpθ,πt(r)(x∣Pt−1,(πt(s),ys)s<r)=vt(πt(r)).\begin{aligned} y_r&=\arg\max_x p_{\theta,\pi_t(r)}(x\mid P_{t-1},(\pi_t(s),y_s)_{s<r})\\ &=v_t^{(\pi_t(r))}. \end{aligned}

This proves agreement at every rank up to AtA_t, including the final verifier token when ht<uth_t<u_t.

The two procedures therefore generate the same AtA_t position–token pairs before applying the stopping rule. They find EOS at the same positions, update EE identically, and remove the same pairs beyond EE. The resulting values of PtP_t agree, completing the induction step.

Each iteration accepts at least one token at a previously unresolved position at or before the updated EE, while EE never increases. Hence the number of unresolved positions up to EE strictly decreases, and both procedures terminate. They return the same tokens in the same output-position order. □\square

Remark C.2 (Sampling under a specified generation order). For a fixed generation order σ\sigma, the argument in Appendix § 7.§ 2 gives samples from pθσp_\theta^\sigma. Choose the order before sampling the draft tokens, for example using proposal entropy or maximum proposal probability. Sorting by the probabilities of sampled draft tokens can change their distributions when checked.

Attention Mask Layouts

Packed-SS combines verification of the current draft and generation of the next proposal in one model call. This is achieved by a custom attention mask. Below we show the packed inputs, masks, and cache updates for Nemotron and SetDLM.

Nemotron: sequence packing and padding

Let nt−1=∣Pt−1∣n_{t-1} = \lvert P_{t-1}\rvert. Each call after initialization processes the last token of the verified prefix, Pt−1(nt−1)P_{t-1}^{(n_{t-1})}, followed by B−1B - 1 draft slots; earlier prefix tokens are read from the cache. The first-call is slightly different and discussed in Appendix § 10.§ 2.

Physical layout. Let Dˉt−1\bar{D}_{t-1} be Dt−1D_{t-1} followed by enough masks to occupy B−1B - 1 slots. The Packed-SS input is

[Pt−1(nt−1),Dˉt−1∣Pt−1(nt−1),Mask⁡B−1],[P_{t-1}^{(n_{t-1})}, \bar{D}_{t-1} \mid P_{t-1}^{(n_{t-1})}, \operatorname{Mask}^{B-1}],

concatenated along the sequence axis. Earlier tokens are stored in KV cache and are not part of the input. Its shape is [1,2B][1, 2B].

Let TBT_B be the B×BB \times B lower-triangular Boolean matrix including its diagonal, and JBJ_B the B×BB \times B all-ones matrix. The following matrix specifies which rows in the current input may attend to each other:

(TB00JB)\begin{pmatrix} T_B & 0 \\ 0 & J_B \end{pmatrix}

and every row additionally sees the cached prefix Pt−1(<nt−1)P_{t-1}^{(<n_{t-1})}. The causal region yields the shifted next-token verifier predictions (Fu et al., 2026). The proposal region contains another copy of the last prefix token, whose output is unused, and B−1B - 1 mask queries, whose outputs are the entries of ZtZ_t. Padded verifier rows are evaluated but cannot extend acceptance, since verification stops at the first absent draft token, represented physically by a mask.

Cache update. After each call, the cache contains the verified prefix except its last token, which is processed in the next call. Keys and values from the proposal region and unaccepted verifier inputs are discarded.

SetDLM: position-aligned queries and ordered context

Duplicate masks. SetDLM predicts a token at the position of its input mask. Its training construction represents each position twice: once with the token and once with a mask, both using the same position ID (Arriola and Kuleshov, [2], Section 5.3 and Appendix K). The mask can attend to tokens earlier in the generation order but cannot see its own target token or later tokens, allowing all conditional predictions to be computed in one pass. We use the same construction for verification, placing draft tokens in the token entries and comparing them with predictions from the corresponding masks. Packed-SS adds a second mask at each position to generate the next draft from the prompt and accepted tokens.

Packed input and attention. Let UtU_t contain the unresolved positions in the active window, and write ut=∣Ut∣u_t = \lvert U_t\rvert. Let NtN_t contain the correction token, if any, that has not yet been processed as an input. Let OtO_t contain the previous draft Dt−1D_{t-1}, with masks for missing entries; MtM_t contain masks for verifier predictions; and RtR_t contain masks for new draft predictions. The input is

[Nt∣Ot∣Mt∣Rt],∣Nt∣=p∈{0,1},∣Ot∣=∣Mt∣=∣Rt∣=ut.[N_t \mid O_t \mid M_t \mid R_t], \qquad\lvert N_t\rvert= p \in\{0,1\}, \qquad\lvert O_t\rvert= \lvert M_t\rvert= \lvert R_t\rvert= u_t.

All regions attend to the cached keys and values of the prompt and accepted tokens. The regions OtO_t, MtM_t, RtR_t also attend to NtN_t, if present, and the row in NtN_t attends to itself.

Write OrO_r, MrM_r, RrR_r for the rows of OtO_t, MtM_t, RtR_t at verification rank rr. Let Kr\mathcal{K}_r contain the earlier ranks s<rs < r in the order πt\pi_t that have a stored draft token. Using X←YX \leftarrow Y to mean that row XX may attend to the rows in YY, the remaining attention connections are

Or←{Os:s∈Kr}∪{Or},Mr←{Os:s∈Kr}∪{Mr},Rr←{Rr}.O_r \leftarrow\{O_s : s \in\mathcal{K}_r\} \cup\{O_r\}, \qquad M_r \leftarrow\{O_s : s \in\mathcal{K}_r\} \cup\{M_r\}, \qquad R_r \leftarrow\{R_r\}.

The three rows at rank rr share the textual position ID πt(r)\pi_t(r). The physical call evaluates all utu_t verifier queries, while the logical algorithm needs predictions only through the first missing draft token or disagreement.

Cache update. Only keys and values from NtN_t and accepted draft tokens in OtO_t are added to the cache. A mask’s keys and values do not represent its predicted token, so a correction is processed in the next call through Nt+1N_{t+1}.

Input-size comparison

Table Table 2 compares the input sizes of these layouts with those of Linear-SS and Quadratic-SS.

MethodNFE/iterationNemotronW=8W = 8W=16W = 16SetDLM general
Linear-SS22B2B2549p+3utp + 3u_t
Quadratic-SS1B(B+1)B(B + 1)81289p+2ut+ut2p + 2u_t + u_t^2
Packed-SS (Ours)12B2B2549p+3utp + 3u_t

Table 2. Execution cost per iteration with cached prompt context. Evaluated uncached rows count token representations processed by the transformer, including duplicates and evaluated padding, summed across calls. Nemotron uses block size B=W+1B = W + 1. In SetDLM, ut≤Wu_t \le W positions remain unresolved in the active window, and p=∣Nt∣∈{0,1}p = \lvert N_t\rvert\in\{0, 1\} counts the correction token processed in this call; maximum row counts take ut=Wu_t = W and p=1p = 1. Prompt rows are processed once and depend on prompt length.

For SetDLM, Linear-SS processes p+utp + u_t rows for drafting and 2ut2u_t for verification, caching NtN_t between calls. Packed-SS processes the same total in one call. Quadratic-SS uses p+2utp + 2u_t rows for verification and utu_t proposal regions of utu_t rows each.

Experimental Details

Hardware, models, and tasks

All runs use batch size one on a single A100 80GB GPU. Nemotron is evaluated in bfloat16 (BF16), and SetDLM in float32 (FP32). Nemotron uses left-to-right AR verification in every main setting. Both scales run the same four tasks—HumanEval, MBPP, MATH-500, and GSM8K ([13, 4, 27, 18])—under greedy decoding and under rejection sampling. Each cell draws on a fixed prompt set: 164 HumanEval problems, 378 MBPP problems, 500 MATH-500 problems, and all 1,319 GSM8K test questions, and these counts are the number of examples behind every mean and interval reported for that cell. Both scales additionally run all 555 English WritingBench prompts ((Wu et al., 2025)) at both temperatures. The SetDLM experiments allocate 1,024 output positions and use active windows of size 8 and 16; because its two windows come from separately adapted checkpoints, comparing them is not a same-weights window-size ablation. At both temperatures, Linear-SS and Packed-SS use Nemotron block size B=32B = 32, while Quadratic-SS uses B=8B = 8, with speculative width W=B−1W = B - 1.

Training horizon versus inference window. The SetDLM training construction uses a desired prediction budget S∈{4,8,16}S \in\{4, 8, 16\} and permits schedules reaching 2S2S future positions [2]. Checkpoints are named by their maximum horizon. Hence smax32 is the S=16S = 16, maximum-horizon-32 model, and its matched active evaluation window is W=16W = 16 rather than 32.

Initialization and stopping

Initial calls. In our Nemotron implementation, the drafting part of a regular call’s input contains the last accepted token followed by WW masks. During initialization, the drafting part contains W+1W + 1 masks, giving one extra draft position. For SetDLM, Packed-SS builds the prompt cache in one counted call before entering the empty-draft loop of Algorithm 3.

At temperature 1, our implementation first processes and caches the prompt, then samples the first output token from its last prediction. A second call proposes draft tokens conditioned on the prompt and this first output token. Both calls count toward NFE and supply a full draft before verification, whereas Algorithm 2 starts from an empty draft.

Output caps. Generations are capped at 512 tokens for code and writing and 1,024 tokens for mathematics.

SetDLM stop sequences. An end marker in the SetDLM evaluation can consist of several tokens. We recognize it only after all of those tokens have been accepted. We set EE, the last output position, to the marker’s final position and fill any earlier unresolved positions before returning. If several markers are recognized, we use the one that ends earliest in the output.

Prompts and scoring

Nemotron uses zero-shot prompts: function completion for HumanEval, visible original test cases for MBPP, and requests for boxed final answers on the mathematics tasks. MBPP uses the 378-problem EvalPlus subset and its original base tests. MATH-500 uses boxed-answer exact match, and SetDLM GSM8K scoring compares normalized final answers. The code and mathematics runs use the same prompts and graders at both temperatures.

For temperature-1 WritingBench runs, each complete query uses the checkpoint’s user-message chat template with thinking disabled; prompts are not truncated. Scored-task tables report accuracy or pass@1 at temperature 0 and mean per-response correctness at temperature 1. WritingBench measures speed only, without judge-based scoring.

Sampling and repetitions

At temperature 1, rejection sampling can produce different generation trajectories for the same prompt, varying output length and decoding cost. To reduce this source of variance on the code tasks, each method draws 20 responses per HumanEval prompt and 7 per MBPP prompt. MATH-500, GSM8K, and WritingBench use one response per prompt and method. Sampling uses float32 probability normalization and the full vocabulary, without top-kk or top-pp filtering. For Quadratic-SS, the verification result selects the branch matching the accepted draft prefix before its next draft tokens are sampled.

Timing and confidence intervals

Decoder wall time is CUDA-synchronized and includes prefill and all decoding work, excluding model loading, prompt preparation, tokenization, output-text decoding, and scoring. Each model forward pass counts as one NFE, including prompt processing. Generated-token counts include stop and special tokens and any generated reasoning text.

We compute 95% percentile confidence intervals from 20,000 bootstrap resamples within each checkpoint, task, decoding mode, and verification order. Each resample draws prompt indices with replacement, uses the same indices for all methods, and recomputes the reported statistics. We bootstrap over prompts, using all repeated responses for each sampled prompt. The intervals describe variation across prompts rather than timing noise for a fixed generation, and are not adjusted for multiple comparisons.

SetDLM decoding settings

Confidence scores. For confidence ordering, Packed-SS and Quadratic-SS score each unresolved position by the probability assigned to its draft token when that token was proposed; Quadratic-SS uses the selected proposal branch. Linear-SS uses the probability of the verifier’s most likely token at each position in the previous iteration. Positions are ranked by decreasing score, with ties broken by textual position. Positions without a draft token or a finite score come last in textual order, so the first iteration uses LTR order. Each order is fixed before the next model call; for Linear-SS, this is the drafting call.

Native baseline. Native SetDLM uses the released sliding-window decoder with argmax token selection and confidence threshold 0.9. It uses the same GSM8K prompts, number of output positions, window sizes, and stopping convention as the speculative schedules.

Additional Experiments and Tables

This appendix collects additional experiments, absolute measurements, and numerical values supporting the main figures.

Nemotron block size and execution cost

To select the block size for each schedule, we run a small probe on Nemotron 8B using the first ten GSM8K test questions (Table 3). We choose the size with the highest token throughput (tokens/s) among those tested. All runs use a total output limit of 512 new tokens.

MethodBBTokens/sRows/callWall (s)Total NFETokensAccuracy (%)
Linear-SS894.982.82498.2267.9100
Linear-SS16113.4162.44884.6277.5100
Linear-SS32121.9322.23777.4272.8100
Packed-SS894.8162.79889.0265.2100
Packed-SS16123.5322.31573.3285.9100
Packed-SS32128.5642.33375.4299.790
Quadratic-SS8122.3722.23264.9272.9100
Quadratic-SS1684.62723.24657.6274.7100
Quadratic-SS3225.4105612.39864.1314.680

Table 3. Nemotron 8B GSM8K block-size probe. Each row uses the same ten questions. Tokens/s is calculated from the displayed means. Wall is mean total decoder time per response, including prompt processing; total NFE and output-token counts are also per-response means. Accuracy is reported for comparison. Rows/call is the maximum number of input rows per call after initialization.

Proposal-only LoRA accounting

We evaluate the proposal-only low-rank adapter trained by the Nemotron authors (Fu et al., 2026) on Nemotron 8B and all GSM8K test questions, changing diffusion drafting while preserving the AR verifier. The base and LoRA measurements in this study use a total output cap of 512 new tokens, including thinking and answer tokens. This cap differs from the 1,024-token GSM8K setting in § 4, so absolute timings are not directly comparable with Table 6. Packed-SS reduces NFE by 12.0% relative to Linear-SS but takes 1.749 rather than 1.698 seconds per example, a 3.0% slowdown (Table 4).

MethodTokensTotal NFEModel (s)Non-model (s)Wall (s)Tokens/sAccuracy (%)
Linear-SS272.255.041.5610.1371.698160.3285.90
Quadratic-SS273.354.441.7810.3432.123128.7286.35
Packed-SS274.248.441.5700.1791.749156.7885.82

Table 4. Nemotron 8B GSM8K with proposal-only LoRA (n=1,319n = 1{,}319). Means; model execution time and non-model decoder time sum to inclusive decoder wall time. Emitted length and throughput accompany latency, and accuracy is reported for comparison.

In calls with the adapter enabled, the implementation computes an update for every input row but applies it only to proposal rows. Packed therefore computes 64 updates but applies only 32, whereas Linear places its 32 proposal rows in a separate call and disables the adapter for its 32-row verifier. Quadratic applies updates to 64 proposal rows but computes them for all 72 input rows.

MethodQuery rowsSelectedEvaluatedBase msLoRA msIncrease
Linear-SS32 + 32323254.4156.774.3%
Quadratic-SS72647228.9032.6913.1%
Packed-SS64326428.5832.3713.3%

Table 5. Nemotron 8B GSM8K model execution time per iteration. Linear’s iteration contains both calls. “Selected” counts rows receiving adapter updates; “evaluated” counts rows for which those updates are computed.

With LoRA, mean model execution time is nearly equal for Packed-SS and Linear-SS (1.570 versus 1.561 seconds). After the first call, Packed-SS averages 47.44 iterations versus Linear-SS’s 27.02, but its model time per iteration is 32.37 versus 56.77 milliseconds, so these differences almost cancel. Non-model decoder time adds 0.179 seconds for Packed-SS against 0.137 for Linear-SS, and the Packed-SS-minus-Linear-SS wall-time difference is 0.0513 seconds with paired 95% interval [0.0243, 0.0788]. Token throughput is also lower: Packed-SS emits 274.2 tokens on average against Linear-SS’s 272.2, and its throughput ratio is 0.978 with paired interval [0.969, 0.987]. Shared, densely fine-tuned weights for drafting and verification would eliminate these extra LoRA computations and could reduce the latency penalty. The Nemotron authors have not released such a checkpoint.

Model execution and non-model decoder time

To separate model evaluation from decoder overhead, we normalize each task’s component times by its Linear-SS total and then average tasks equally within each model scale, so that long-output tasks do not dominate the decomposition. The decomposition covers GSM8K and MATH-500 only.

Normalized Nemotron timing decomposition on GSM8K and MATH-500

Figure 5. Normalized Nemotron timing decomposition on GSM8K and MATH-500. Greedy decoding with a 1,024-token cap. Each task’s component times are divided by that task’s Linear-SS total, then the two tasks are averaged with equal weight within each model scale. The dashed line marks Linear-SS’s total. Non-model decoder time covers draft token construction, selection, and state updates.

Packed-SS cuts model execution time to 0.860 and 0.854 of Linear-SS’s total at 3B and 8B while raising non-model decoder time by about 40%, for totals of 0.964 and 0.951. Quadratic-SS cuts model execution time comparably, to 0.898 and 0.876, but its non-model decoder time is 2.9 and 2.7 times Linear-SS’s, so its totals exceed one at 1.106 and 1.062.

Absolute measurements

Confidence-ordered verification on SetDLM

The training horizons are Smax⁡=16S_{\max} = 16 and 32 for W=8W = 8 and 16, respectively. For W=8W = 8 and 16, respectively, maximum recurrent rows/call are (16, 32) for Linear-SS, (81, 289) for Quadratic-SS, (25, 49) for Packed-SS, and (16, 28) for Native. Native commits at confidence threshold 0.9 and has no verification order (Appendix § 10.6).

Under confidence order, Packed-SS remains fastest at both windows (Table 9), reducing Linear-SS’s wall time by 9.5% at W=8W = 8 and 9.7% at W=16W = 16 and its total NFE by 12.4% and 16.5%. Its throughput gains over Linear-SS are 10.2% and 12.8%, and at W=16W = 16 it reaches 2.20× Quadratic-SS’s throughput; all eight SetDLM paired intervals favour Packed-SS (Table 10). Two effects should be kept apart, because only one of them is a cost. The three LTR schedules have identical per-example correctness labels within each checkpoint, so Packed-SS’s LTR speedup incurs no observed accuracy cost. Changing the order can affect accuracy. Confidence ordering lowers Packed-SS’s latency by a further 4.1% at W=8W = 8 and 13.1% at W=16W = 16 relative to LTR, but accuracy falls from 68.39% to 65.35% and from 66.57% to 59.06%; the second drop is large enough to give up the advantage over SetDLM’s own threshold decoder, which scores 59.51% there. Adaptive position choices alter the greedy trajectory even in exact arithmetic, and single-step confidence is in any case a weak criterion for choosing which tokens to accept ([12]). We therefore report the ordering as a speed–accuracy trade-off of the decoding policy rather than a property of the schedule: Packed-SS gives the lowest latency in each regime.

WindowMethodTokens/sTokens/NFETime/NFE (ms)Accuracy (%)TokensWall (s)NFE
LTR verification and native decoding
W=8W = 8Linear-SS94.932.61027.49668.39257.22.70998.53
Quadratic-SS86.253.41439.58368.39257.12.98175.31
Packed-SS100.602.84228.24768.39257.22.55690.49
Native77.212.11927.44259.67252.43.269119.11
W=16W = 16Linear-SS98.852.79228.24366.57259.92.62993.09
Quadratic-SS41.604.07497.93466.57260.46.26063.92
Packed-SS100.303.03030.20966.57259.92.59185.77
Native80.592.22027.55259.51253.63.147114.21
Confidence-ordered verification
W=8W = 8Linear-SS93.932.58627.53565.05254.52.70998.39
Quadratic-SS90.833.47038.20065.58254.62.80373.38
Packed-SS103.522.94328.43165.35253.82.45186.21
W=16W = 16Linear-SS99.062.79428.20653.07247.02.49388.40
Quadratic-SS50.904.46287.66460.80252.14.95456.51
Packed-SS111.763.40830.49159.06251.52.25173.81

Table 9. SetDLM measurements on GSM8K. All 1,319 test questions. Top: LTR verification and native decoding; bottom: confidence-ordered verification. Tokens, inclusive wall time, and total NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each window and order.

SettingBaselineΔ\Delta wall (s) [95% CI]Throughput ratio [95% CI]
W=8W = 8 LTRLinear-SS−0.1529-0.1529 [−0.1588-0.1588, −0.1470-0.1470]1.060 [1.058, 1.062]
Quadratic-SS−0.4248-0.4248 [−0.4377-0.4377, −0.4124-0.4124]1.166 [1.164, 1.169]
W=8W = 8 ConfidenceLinear-SS−0.2581-0.2581 [−0.2953-0.2953, −0.2217-0.2217]1.102 [1.098, 1.107]
Quadratic-SS−0.3521-0.3521 [−0.3836-0.3836, −0.3198-0.3198]1.140 [1.136, 1.143]
W=16W = 16 LTRLinear-SS−0.0382-0.0382 [−0.0432-0.0432, −0.0330-0.0330]1.015 [1.013, 1.017]
Quadratic-SS−3.6686-3.6686 [−3.7869-3.7869, −3.5590-3.5590]2.411 [2.399, 2.424]
W=16W = 16 ConfidenceLinear-SS−0.2428-0.2428 [−0.2757-0.2757, −0.2091-0.2091]1.128 [1.123, 1.134]
Quadratic-SS−2.7033-2.7033 [−2.7766-2.7766, −2.6335-2.6335]2.196 [2.185, 2.207]

Table 10. Paired SetDLM efficiency comparisons. Wall-time differences are Packed-SS minus baseline; throughput ratios are Packed-SS/baseline. Brackets give paired 95% confidence intervals.

Accuracy and output trajectories

We report paired accuracy differences to assess variation in task scores At temperature 1, the schedules draw different random samples, so their continuations and measured accuracies can differ. Under greedy decoding, small floating-point differences can also change token choices and subsequent continuations. Table 11 reports the Nemotron comparisons in percentage points, with paired 95% confidence intervals.

SettingPacked-SS − Linear-SSPacked-SS − Quadratic-SSLinear-SS − Quadratic-SS
Temperature 0
3B MBPP-0.26 [-1.32, +0.79]+0.26 [-0.79, +1.32]+0.53 [-0.53, +1.59]
3B HumanEval-1.22 [-3.66, +1.22]-1.83 [-4.27, +0.00]-0.61 [-1.83, +0.00]
3B GSM8K-0.30 [-1.14, +0.53]-0.15 [-0.99, +0.68]+0.15 [-0.68, +0.99]
3B MATH-500+3.00 [+0.80, +5.20]+0.20 [-1.60, +2.00]-2.80 [-4.60, -1.00]
8B MBPP-0.26 [-1.59, +0.79]+0.00 [-0.79, +0.79]+0.26 [-0.53, +1.06]
8B HumanEval+1.22 [-1.22, +3.66]-0.61 [-3.05, +1.22]-1.83 [-4.27, +0.00]
8B GSM8K+0.15 [-0.83, +1.14]-0.53 [-1.59, +0.53]-0.68 [-1.82, +0.38]
8B MATH-500+1.40 [-0.60, +3.60]+1.60 [-0.40, +3.80]+0.20 [-2.00, +2.40]
Temperature 1
3B MBPP+0.64 [-0.98, +2.31]-0.91 [-2.61, +0.83]-1.55 [-3.17, +0.08]
3B HumanEval+1.46 [+0.06, +2.90]+1.07 [-0.34, +2.47]-0.40 [-1.74, +0.92]
3B GSM8K-0.68 [-2.65, +1.36]-1.36 [-3.34, +0.61]-0.68 [-2.73, +1.36]
3B MATH-500-1.00 [-4.60, +2.60]+0.80 [-2.80, +4.40]+1.80 [-1.80, +5.40]
8B MBPP-1.40 [-2.95, +0.15]+0.38 [-1.32, +2.12]+1.78 [+0.11, +3.44]
8B HumanEval+0.64 [-0.91, +2.20]+0.06 [-1.49, +1.59]-0.58 [-2.13, +0.95]
8B GSM8K-2.05 [-4.02, -0.08]-1.21 [-3.26, +0.83]+0.83 [-1.06, +2.73]
8B MATH-500+0.00 [-4.00, +4.00]-1.00 [-4.60, +2.80]-1.00 [-4.80, +2.80]

Table 11. Paired Nemotron accuracy differences.

For SetDLM, the LTR schedules have identical correctness labels despite different outputs on 5 of 1,319 smax16 questions and 13 of 1,319 smax32 questions. Their accuracy gains over Native are 8.72 points at W=8W = 8 (95% CI [6.37, 11.07]) and 7.05 points at W=16W = 16 (95% CI [4.78, 9.33]).

Paired efficiency comparisons

Table 12 and Table 13 report Packed-SS minus baseline wall time and Packed-SS/baseline throughput, with 95% paired intervals. Negative differences and ratios above one favor Packed-SS.

SettingBaselineΔ\Delta wall (s) [95% CI]Throughput ratio [95% CI]
3B MBPPLinear-SS-0.0092 [-0.0268, +0.0123]1.044 [1.007, 1.075]
Quadratic-SS-0.0308 [-0.0563, -0.0027]1.102 [1.079, 1.127]
3B HumanEvalLinear-SS-0.0207 [-0.0341, -0.0100]1.052 [1.033, 1.072]
Quadratic-SS-0.0703 [-0.0886, -0.0539]1.181 [1.149, 1.213]
3B GSM8KLinear-SS-0.0434 [-0.0605, -0.0243]1.061 [1.049, 1.071]
Quadratic-SS-0.1003 [-0.1187, -0.0798]1.114 [1.103, 1.125]
3B MATH-500Linear-SS-0.0565 [-0.1145, +0.0040]1.035 [1.021, 1.048]
Quadratic-SS-0.3872 [-0.4531, -0.3221]1.192 [1.174, 1.209]
3B WritingBenchLinear-SS-0.4969 [-0.5335, -0.4604]1.093 [1.087, 1.099]
Quadratic-SS-0.1753 [-0.2068, -0.1435]1.034 [1.029, 1.040]
8B MBPPLinear-SS-0.0058 [-0.0222, +0.0156]1.040 [1.024, 1.056]
Quadratic-SS-0.0711 [-0.0921, -0.0520]1.172 [1.144, 1.200]
8B HumanEvalLinear-SS-0.0294 [-0.0509, -0.0077]1.050 [1.022, 1.078]
Quadratic-SS-0.1372 [-0.1634, -0.1121]1.323 [1.274, 1.376]
8B GSM8KLinear-SS-0.0908 [-0.1499, -0.0318]1.054 [1.042, 1.066]
Quadratic-SS-0.1856 [-0.2433, -0.1277]1.092 [1.082, 1.104]
8B MATH-500Linear-SS-0.2656 [-0.3560, -0.1778]1.056 [1.040, 1.071]
Quadratic-SS-0.6361 [-0.7332, -0.5367]1.163 [1.145, 1.181]
8B WritingBenchLinear-SS-0.5538 [-0.6101, -0.4958]1.081 [1.073, 1.089]
Quadratic-SS-0.3866 [-0.4433, -0.3300]1.057 [1.050, 1.065]

Table 12. Paired Nemotron efficiency comparisons at temperature 0.

SettingBaselineΔ\Delta wall (s) [95% CI]Throughput ratio [95% CI]
3B MBPPLinear-SS-0.0389 [-0.0678, -0.0107]1.039 [1.026, 1.052]
Quadratic-SS-0.0287 [-0.0832, +0.0248]0.969 [0.956, 0.984]
3B HumanEvalLinear-SS-0.0076 [-0.0250, +0.0093]1.021 [1.005, 1.038]
Quadratic-SS+0.0033 [-0.0121, +0.0189]1.015 [0.997, 1.034]
3B GSM8KLinear-SS-0.0227 [-0.0832, +0.0387]1.020 [1.001, 1.037]
Quadratic-SS+0.0000 [-0.0640, +0.0652]0.996 [0.979, 1.013]
3B MATH-500Linear-SS-0.0820 [-0.2440, +0.0791]1.036 [1.012, 1.060]
Quadratic-SS-0.0895 [-0.2481, +0.0661]1.043 [1.021, 1.066]
3B WritingBenchLinear-SS-0.2455 [-0.3181, -0.1699]1.041 [1.034, 1.048]
Quadratic-SS+0.0536 [-0.0187, +0.1283]0.998 [0.990, 1.006]
8B MBPPLinear-SS-0.0467 [-0.0970, +0.0024]1.061 [1.045, 1.076]
Quadratic-SS+0.0619 [+0.0072, +0.1164]0.982 [0.969, 0.995]
8B HumanEvalLinear-SS-0.0767 [-0.1104, -0.0443]1.059 [1.042, 1.077]
Quadratic-SS-0.0423 [-0.0786, -0.0059]1.024 [1.007, 1.042]
8B GSM8KLinear-SS-0.3104 [-0.4833, -0.1363]1.063 [1.046, 1.081]
Quadratic-SS-0.0025 [-0.1731, +0.1618]0.992 [0.976, 1.007]
8B MATH-500Linear-SS-0.2686 [-0.5450, +0.0071]1.050 [1.029, 1.073]
Quadratic-SS+0.0831 [-0.1779, +0.3394]0.997 [0.978, 1.018]
8B WritingBenchLinear-SS-0.5121 [-0.6381, -0.3880]1.052 [1.043, 1.061]
Quadratic-SS-0.3029 [-0.4490, -0.1578]1.039 [1.030, 1.047]

Table 13. Paired Nemotron efficiency comparisons at temperature 1.

All ten paired throughput intervals against Linear-SS lie above one (Table 13), supporting a throughput advantage in the measured settings. Against the Quadratic-SS extension, three intervals lie above one, two lie below it, and five include it; the last group is inconclusive rather than evidence of equivalence.

Reference schedules. At temperature 0, Quadratic-SS has lower throughput than Linear-SS on all eight code and mathematics settings but higher throughput on WritingBench; at temperature 1, it has higher throughput on seven of the eight code and mathematics settings. Shorter accepted prefixes require more iterations, so eliminating a second call can outweigh the cost of its fixed 72-row pass (Appendix § 11.§ 3). On SetDLM, Linear-SS and Packed-SS are faster than Native at both windows; Quadratic-SS is faster only at W=8W = 8 (Table 8).

References

  1. [1]Arriola, M., Gokaslan, A., Chiu, J., Yang, Z., Qi, Z., Han, J., Sahoo, S., and Kuleshov, V. (2025). Block diffusion: Interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations.
  2. [2]Arriola, M. and Kuleshov, V. (2026). Set diffusion: Interpolating token orderings between autoregression and diffusion for fast and flexible decoding. In Proceedings of the 43rd International Conference on Machine Learning.arxiv.org/abs/2607.01775
  3. [3]Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. (2021a). Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, volume 34.arxiv.org/abs/2107.03006
  4. [4]Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. (2021b). Program synthesis with large language models. arXiv preprint arXiv:2108.07732.arxiv.org/abs/2108.07732
  5. [6]Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33.arxiv.org/abs/2005.14165
  6. [7]Cai, C. and Li, G. (2026). Confidence-based decoding is provably efficient for diffusion language models. arXiv preprint arXiv:2603.22248.arxiv.org/abs/2603.22248
  7. [8]Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. (2024). Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5209–5235.arxiv.org/abs/2401.10774
  8. [9]Campbell, A., Benton, J., De Bortoli, V., Rainforth, T., Deligiannidis, G., and Doucet, A. (2022). A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems.
  9. [10]Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. (2023). Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318.arxiv.org/abs/2302.01318
  10. [11]Chen, J., Liang, Y., and Liu, Z. (2026a). DFlash: Block diffusion for flash speculative decoding. In Proceedings of the 43rd International Conference on Machine Learning.arxiv.org/abs/2602.06036
  11. [12]Chen, K., Liu, Z., Tao, X., Liu, H., Fu, X., Zhang, S., Tu, D., Kong, L., Liu, R., and Li, H. (2026b). Beyond confidence: Adaptive and coherent decoding for diffusion language models. In Proceedings of the 43rd International Conference on Machine Learning.arxiv.org/abs/2512.02044
  12. [13]Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.arxiv.org/abs/2107.03374
  13. [14]Chen, S., Cong, K., and Li, J. (2025). Optimal inference schedules for masked diffusion models. arXiv preprint arXiv:2511.04647.arxiv.org/abs/2511.04647
  14. [15]Cheng, X., Yu, X., Shao, C., Li, J., Xiong, Y., Qian, Y., Zhu, J., Ma, S., Zhang, X., Ye, J., et al. (2026). DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv preprint arXiv:2607.05147.arxiv.org/abs/2607.05147
  15. [16]Cheng, Z., Yang, G.-W., Li, J., Deng, Z., Guo, M.-H., and Hu, S.-M. (2025). DEER: Draft with diffusion, verify with autoregressive models. arXiv preprint arXiv:2512.15176.arxiv.org/abs/2512.15176
  16. [17]Christopher, J. K., Bartoldson, B. R., Ben-Nun, T., Cardei, M., Kailkhura, B., and Fioretto, F. (2024). Speculative diffusion decoding: Accelerating language generation through diffusion. arXiv preprint arXiv:2408.05636.arxiv.org/abs/2408.05636
  17. [18]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.arxiv.org/abs/2110.14168
  18. [19]Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., Aly, A., Chen, B., and Wu, C.-J. (2024). LayerSkip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12622–12642.
  19. [20]Fan, A., Lewis, M., and Dauphin, Y. (2018). Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898.arxiv.org/abs/1805.04833
  20. [23]Gao, Y., Ji, Z., Wang, Y., Qi, B., Xu, H., and Zhang, L. (2025). Self speculative decoding for diffusion large language models. arXiv preprint arXiv:2510.04147.arxiv.org/abs/2510.04147
  21. [24]Gong, S., Agarwal, S., Zhang, Y., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Han, J., Peng, H., and Kong, L. (2025). Scaling diffusion language models via adaptation from autoregressive models. In International Conference on Learning Representations.arxiv.org/abs/2410.17891
  22. [25]Grattafiori, A. et al. (2024). The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.arxiv.org/abs/2407.21783
  23. [26]Guo, G. and Ermon, S. (2026). Self-speculative decoding accelerates lossless inference in any-order and any-subset autoregressive models. In International Conference on Learning Representations.
  24. [27]Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
  25. [28]Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2020). The curious case of neural text degeneration. In International Conference on Learning Representations.
  26. [29]Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. (2021). Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Information Processing Systems, volume 34.
  27. [30]Leviathan, Y., Kalman, M., and Matias, Y. (2023). Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR.arxiv.org/abs/2211.17192
  28. [31]Li, A. C. (2025). Generalization Through Richer Supervision. PhD thesis, Machine Learning Department, Carnegie Mellon University. CMU-ML-25-103; Prism is Chapter 5.DOI
  29. [37]Liu, J., Dong, X., Ye, Z., Mehta, R., Fu, Y., Singh, V., Kautz, J., Zhang, C., and Molchanov, P. (2025). TiDAR: Think in diffusion, talk in autoregression. arXiv preprint arXiv:2511.08923.arxiv.org/abs/2511.08923
  30. [38]Lou, A., Meng, C., and Ermon, S. (2024). Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning.arxiv.org/abs/2310.16834
  31. [39]Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., Shi, C., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. (2024). SpecInfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3.DOI
  32. [40]Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. (2025). Large language diffusion models. arXiv preprint arXiv:2502.09992.arxiv.org/abs/2502.09992
  33. [41]Qwen Team (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  34. [42]Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., and Kuleshov, V. (2024). Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems.arxiv.org/abs/2406.07524
  35. [43]Samragh, M., Kundu, A., Harrison, D., Nishu, K., Naik, D., Cho, M., and Farajtabar, M. (2025). Your LLM knows the future: Uncovering its multi-token prediction potential. arXiv preprint arXiv:2507.11851.arxiv.org/abs/2507.11851
  36. [44]Santilli, A., Severino, S., Postolache, E., Maiorca, V., Mancusi, M., Marin, R., and Rodola, E. (2023). Accelerating transformer inference for translation via parallel decoding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12336–12355. Association for Computational Linguistics.
  37. [45]Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. (2024). Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems.arxiv.org/abs/2406.04329
  38. [46]Spector, B. and Ré, C. (2023). Accelerating LLM inference with staged speculative decoding. arXiv preprint arXiv:2308.04623.arxiv.org/abs/2308.04623
  39. [47]Stern, M., Shazeer, N., and Uszkoreit, J. (2018). Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems, volume 31.arxiv.org/abs/1811.03115
  40. [48]Teng, Y., Shi, H., Liu, X., Ning, X., Dai, G., Wang, Y., Li, Z., and Liu, X. (2024). Accelerating auto-regressive text-to-image generation with training-free speculative jacobi decoding. arXiv preprint arXiv:2410.01699.arxiv.org/abs/2410.01699
  41. [51]Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., and Sui, Z. (2023). Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP.arxiv.org/abs/2203.16487
  42. [54]Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. (2024). Draft & verify: Lossless large language model acceleration via self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  43. [55]Zhang, Y., Zhang, L., Cui, B., and Miao, X. (2026). DFlow: Enabling verifier information flow in block diffusion speculative decoding. arXiv preprint arXiv:2609.06498.arxiv.org/abs/2609.06498
  44. [56]Zhao, Y. and Cai, C. (2026). Adaptation to intrinsic dependence in diffusion language models. arXiv preprint arXiv:2602.20126.arxiv.org/abs/2602.20126

Paper details

vlnc:W_pv34xpdv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents