betaFeedback

WorksComputer Science

Fast at the Frontier, Wrong at the Fallback: A Commit-Level Analysis of Threshold Decoding in Diffusion Language Models

Abstract

Diffusion language models (DLMs) can write several tokens per forward pass. Open-weight DLMs commit, in any order, every masked position whose confidence passes a threshold. If none does, a fallback commits the most confident one, and the commits are never revised. Wide windows let the decoder commit far ahead of the text, which should speed up decoding but often costs accuracy. It is unclear where the model is confident and which commits cause errors. We trace every commit of four off-the-shelf DLMs on math, code, translation and summarization, and rerun failing answers with one step changed. Most tokens become confident only near the frontier, the leftmost masked position, so committing only there, still several tokens per step, is about as fast as committing anywhere. The accuracy loss comes from committing tokens before the text to their left, and committing one token per step does not reduce it. The worst case is an early end-of-sequence token that cuts the answer short. Fallback commits are 11-35% of commits but start 61-72% of the failures that left-to-right decoding avoids. Changing only the step where such a failure starts repairs it more often than changing a similar earlier step. Committing only at the frontier recovers most of the lost accuracy on all four models. Our code is available at https://github.com/rmakarovv/fast-at-the-frontier.

Introduction

Diffusion language models (DLMs) write an answer by filling in blanks. The answer starts as a row of masked positions. At each step, one forward pass predicts a token for every masked position, and the decoder fills some of them [38, 1] (Sahoo et al. 2024). An autoregressive (AR) model fills one position per pass, from left to right; a DLM can fill several, so it may decode faster. Open-weight DLMs such as LLaDA [38], Fast-dLLM v2 (Wu et al., 2026a) and Nemotron-Labs-Diffusion [12] use a confidence-based decoding policy (Wu et al., 2026b). The decoder works inside a range of answer positions, its window. At each step, it fills every masked position in the window whose most likely token has probability above a threshold. If none does, a fallback fills the single most confident position. We call each filled token a commit. These decoders never change a commit, and they may commit in any order. A wider window therefore looks like a cheap way to commit more tokens per step.

Following Yeom et al. (2026), we call the leftmost masked position the frontier. An ahead commit fills a position while a position to its left is still masked. For example, “miles” in Figure 1 is committed before the “15” to its left. Since commits are never revised, an ahead commit is a guess that the policy makes before the context to its left exists. A particularly harmful case is an ahead commit of an end-of-sequence (EOS) token, which forces the answer to finish before the chosen position.

Figure 1. One real 32-token block (NLD-3B, GSM8K, VANILLA); each box shows a token and its commit step. Blue: commits at the frontier; orange: ahead commits, made while a position to their left was still masked (“miles” at step 0, before “15”); dashed: fallback commits.

Wide windows can cost a lot of accuracy: on GSM8K, LLaDA scores 77% with 32-token windows but 14% with one 256-token window [30]. Recent policies keep commits near the frontier and recover accuracy. Representative examples include PARD [18], Fork-dLLM [11], RPD [14], and frontier gating (Yeom et al., 2026). However, by limiting where a DLM can commit, these policies give up the freedom to commit anywhere, which may slow down generation. Whether this freedom is worth keeping decides how DLMs should be decoded and what future models should be trained for. For the standard decoding policy in these models, which we call the vanilla policy, it is unclear whether the model becomes confident about masked positions far from the frontier or only about those near it, and which of its commits cause the errors. Answering these questions requires following single commits, because a bad commit may hurt the answer many tokens later, and averages over all commits, such as tokens per step or how closely the order follows left to right (Zhong et al., 2026), mix good and bad commits together. We ask three questions:

  • RQ1 (confidence). How does the model’s confidence change with distance from the frontier, and does committing only at the frontier slow decoding?

  • RQ2 (accuracy). Why do wide windows lose accuracy, and can a simple frontier policy recover it without slowing decoding?

  • RQ3 (commits). Which individual commits start the failures?

To answer these questions, we trace every commit of four models: Nemotron-Labs-Diffusion (NLD) 3B and 8B, LLaDA-8B and Fast-dLLM v2 7B. We use math and code benchmarks, plus translation and summarization on the two models that lose the most on math. For each commit, we record how far it is from the frontier and whether the fallback made it.

We compare decoding policies that differ in one part only, such as where the fallback commits. Our main test is a replay. Once two decodes differ at one step, all later steps differ too, so we take a failing decode, change only the step where it first departs from a left-to-right decode of the same problem, and continue decoding. As a control, we change a similar earlier step of the same decode. If changing the first departure step repairs the answer more often, that step more likely caused the failure. We also use two diagnostic policies, each a controlled ablation of VANILLA that removes one suspected cause of errors: (1) EOS-AT-FRONTIER, which allows the EOS token only at the frontier, and (2) the distance cap, which allows commits only within kk positions of the frontier (an extension of Yeom et al. (2026)).

We find that:

  • Most tokens become confident only near the frontier (RQ1). So a policy that commits only there runs as fast as VANILLA. Commits ahead of the frontier add almost no speed, making wider windows inefficient, especially at large batch sizes (§ 4).

  • The order of commits causes the accuracy loss (RQ2). Committing one token per step barely helps if the decoder still picks tokens by confidence. The loss comes from early EOS and early content tokens, in a mix that depends on the model, and committing only at the frontier recovers most of it on all four models (§ 5).

  • Most failures start when the fallback commits ahead of the frontier (RQ3). The fallback makes only 11–35% of all commits, yet these commits start 61–72% of the failures that left-to-right decoding avoids. When we replay a failing decode and change only that step, the answer is fixed more often than when we change a similar earlier step (§ 6).

Decoders should therefore commit only at the frontier, and faster decoding should come from committing more tokens at the frontier in each step rather than from wider windows.

Preliminaries

Masked diffusion language models. Masked diffusion language models (MDLMs) generate token sequences by gradually recovering tokens from masked inputs ([3, 38] (Sahoo et al. 2024; Shi et al. 2024)). Let x=(x1,…,xN)∈VNx=(x_1,\ldots,x_N)\in\mathcal{V}^N be a sequence of NN tokens from vocabulary V\mathcal{V}, and let mm be a special mask token. The forward diffusion process replaces each token with mm independently, according to a predefined noise schedule. With a linear noise schedule, the forward process at time t∈[0,1]t\in[0,1] is

qt(zt,i∣xi)=(1−t)δxi(zt,i)+tδm(zt,i),(1)q_t(z_{t,i}\mid x_i)=(1-t)\delta_{x_i}(z_{t,i})+t\delta_m(z_{t,i}), \tag*{(1)}

where δa\delta_a denotes a point mass at aa. A denoising model pθp_\theta learns to recover the original tokens from the corrupted sequence ztz_t. For the linear schedule, the standard continuous-time training objective reduces to a weighted masked-token prediction loss:

L=−Ex,t,zt[1t∑i=1NI[zt,i=m]⋅log⁡pθ(xi∣zt)](2)\mathcal{L}=-\mathbb{E}_{x,t,z_t}\left[\frac{1}{t}\sum_{i=1}^{N}\mathbb{I}[z_{t,i}=m]\cdot\log p_\theta(x_i\mid z_t)\right] \tag*{(2)}

where t∼U(0,1)t\sim\mathcal{U}(0,1) and zt∼qt(⋅∣x)z_t\sim q_t(\cdot\mid\mathbf{x}).

Blockwise masked diffusion. Blockwise diffusion combines autoregressive dependencies across blocks with parallel denoising within each block [1]. Given a block size BB, we partition the sequence into J=⌈N/B⌉J=\lceil N/B\rceil consecutive blocks, x=(x(1),…,x(J))\mathbf{x}=(\mathbf{x}^{(1)},\ldots,\mathbf{x}^{(J)}). The sequence distribution factorizes as

pθ(x)=∏b=1Jpθ(x(b)∣x(<b)),(3)p_\theta(\mathbf{x})=\prod_{b=1}^{J}p_\theta\left(\mathbf{x}^{(b)}\mid\mathbf{x}^{(<b)}\right), \tag*{(3)}

where x(<b)\mathbf{x}^{(<b)} denotes blocks before the bb-th block. Each block is generated through a masked diffusion process conditioned on the completed prefix. The denoising model predicts masked tokens using bidirectional dependencies within the current block while preserving causal dependencies across blocks. Training follows the masked diffusion objective in Eq. (2), with the unmasked preceding blocks provided as conditioning context.

At inference, the decoder generates blocks from left to right. It starts block bb with all positions masked, z0(b)=(m,…,m)z_0^{(b)}=(m,\ldots,m). At each step ss, the model predicts token distributions for all masked positions in parallel. The decoder selects a subset CsC_s to commit, using a fixed rule, such as a confidence threshold, or a learned policy. Let x^s,i\hat{x}_{s,i} be the predicted token at position ii. The decoder updates the state as follows:

zs+1,i(b)={x^s,i,i∈Cs,zs,i(b),otherwise.(4)z_{s+1,i}^{(b)}= \begin{cases} \hat{x}_{s,i}, & i\in C_s,\\ z_{s,i}^{(b)}, & \text{otherwise}. \end{cases} \tag*{(4)}

In this paper, we study irreversible decoding: once the decoder commits a token, it does not change it. We evaluate Nemotron-Labs-Diffusion (NLD), LLaDA, and Fast-dLLM v2 [12, 38] (Wu et al. 2026a). The set of masked positions can only shrink until the block is complete. Each evaluation can commit multiple tokens.

Confidence-based decoding. The threshold policy is defined as follows: at each step, the decoder commits every masked position in the window whose confidence is at least a threshold (0.9 here; Wu et al., 2026b). If none does, the fallback commits the most confident one. We call this policy VANILLA.

Key terms. We describe each committed token by where it is relative to the frontier and by whether the fallback made it (Figure 1). We define these terms below. Positions after answer’s first EOS hold EOS padding; all other tokens are called content tokens.

Terms for commits
Frontier: the leftmost masked position in the window. Left-to-right decoding commits only the frontier.
Ahead commit: a commit which has a position to its left that is still masked.
Fallback commit: a commit made by the fallback when no position reaches the threshold; all other commits are confident.

Table 16.

Experimental setup

Models. NLD 3B/8B [12] are block diffusion models trained with B=32B=32. LLaDA-8B-Instruct [38] is trained without blocks. At inference, we decode its 256-token answer either as one block (B=256B=256) or with B=32B=32. Fast-dLLM v2 7B (Wu et al., 2026a) adapts Qwen2.5 (Qwen Team, 2024) to block diffusion with B=32B=32, where each forward pass sees the whole block, but the decoder commits only inside the current sub-block of 8 tokens. A step commits only inside its window: the current block on NLD and LLaDA, and the current sub-block on Fast-dLLM v2. For Fast-dLLM v2 at B=64B=64 and 128, we turn sub-blocks off, so its window is also the whole block. We call the block size used in training (B=32B=32 for NLD and Fast-dLLM v2) the training block. Most results use each model’s wide window: B=128B=128 or 256 for NLD, B=256B=256 (one block) for LLaDA, and B=128B=128 for Fast-dLLM v2.

Decoding policies. Table 1 lists the policies: VANILLA, two one-token baselines (ONE-PER-STEP and LEFT-TO-RIGHT), four policies from recent work, and two diagnostic policies, each of which removes one suspected cause of errors.

PolicyTokens committed per step
VANILLAOne or more. Commit all tokens with confidence ≥0.9\ge0.9, otherwise the most confident one.
ONE-PER-STEPOne. Commit the most confident masked position.
LEFT-TO-RIGHTOne. Commit the frontier position (left-to-right decoding).
FRONTIER-ONLYOne or more. Commit the confident positions in a row, starting at the frontier. If none exists, commit the frontier position (PARD; Hou and Kwok, [18]).
Variants of VANILLA: one or more tokens per step
FRONTIER-FALLBACKUse the frontier position as the fallback (Fork-dLLM; Frković et al., [11]).
FRONTIER-GATINGLimit commits to the first 16 positions from the frontier (Yeom et al., 2026).
ENTROPY-BUDGETApply RPD’s 4-nat entropy budget to masked positions on the left (He et al., [14]).
EOS-AT-FRONTIERAllow the end-of-sequence token only at the frontier (ours).
Cap kkLimit commits to within kk positions of the frontier (ours, extends Yeom et al., 2026).

Table 1. Decoding policies. All policies use the same model, prompt, and confidence threshold (0.9). Commits are restricted to the decoding window. The first four vary position order and token count. The remaining policies modify VANILLA.

Tasks. We use math (GSM8K, Cobbe et al., [8]; MATH-500, Hendrycks et al., [15]; Lightman et al., [31]) and code (HumanEval, Chen et al., [7]; MBPP, Austin et al., [4]) benchmarks, plus translation (WMT16, Bojar et al., [6]) and summarization (CNN/DailyMail, Hermann et al., [16]) on NLD-3B and LLaDA.

Implementation details. Main comparisons use greedy decoding. In addition, we sample three seeds at temperature 0.7. The quality of DLM decoders can depend on the prompt (Zhang et al., 2026a), so we run LLaDA with two prompts (A: step by step; B: the CD4LM evaluation prompt, Liang et al., [30]). For every commit, we log its position, step, confidence, distance and whether the fallback made it (Figure 1). Tokens per forward (TPF) is the number of committed tokens per forward pass. Appendix § 9 gives the tasks, the settings of the baseline policies and how we grade answers.

Statistics. We compare accuracies with paired exact McNemar tests (McNemar, [35]), Holm-corrected (Holm, [17]), and report a difference as its estimate ±\pm half the width of its 95% bootstrap interval over problems. Accuracy differences are in percentage points (points).

Most tokens become confident only near the frontier

Under threshold decoding, a step commits every position whose confidence passes the threshold, so the number of committed tokens depends on how many positions are confident at the same step. To analyze how the model’s confidence varies across masked positions (RQ1), we compare VANILLA with FRONTIER-ONLY, which commits only at the frontier.

Most tokens become confident only near the frontier. In left-to-right decodes, for each answer position, we log the first step at which its confidence passes the threshold, and its distance to the frontier at that step. Specifically, for a commit at position jj and let ff be the frontier, we define the distance d=j−fd = j - f. On all four models, 15–24% of positions pass the threshold only once they are at the frontier, and 60–64% within seven positions of it. At a distance of 16 positions, the share of positions that pass the threshold for the first time falls below 1.5% (Figure 2a). The tokens a step can commit are therefore mostly within a few positions of the frontier, and a wider window mainly adds positions that become confident only later.

Figure 2. Most tokens become confident only near the frontier, so wider windows add little speed. (a) How far ahead of the frontier answer positions first pass the threshold in left-to-right decodes (GSM8K; NLD B=128B = 128, LLaDA B=256B = 256 prompt A, Fast-dLLM v2 B=128B = 128), in % of answer positions per distance. (b) Tokens per step by BB (NLD-3B, VANILLA, blocks inside the answer). (c) TPF of FRONTIER-ONLY over TPF of VANILLA at the same window, against VANILLA’s ahead share; one marker per model, task and window (34 in all). The two high points are settings where VANILLA cuts answers short.

Large blocks do not commit more per step. On NLD-3B, a position at a given offset in its block is committed at about the same step for every BB from 8 to 128, so the number of committed tokens per step stops growing near the training block (Figure 2b). LLaDA and Fast-dLLM v2 behave the same way, and only NLD-8B, and NLD-3B on MATH-500, still gain 10–17% tokens per step from B=32B = 32 to 128. Hence, the speed of threshold decoding is limited by how many tokens are confident near the frontier. Appendix § 10.§ 1 gives the per-position curves.

Removing ahead commits keeps the speed. With wide windows, about half of VANILLA’s commits on NLD and about three quarters on LLaDA are ahead commits (Figure 2c, horizontal axis). Yet FRONTIER-ONLY, which makes no ahead commits, has TPF at least 0.92 times that of VANILLA across the 34 settings in Figure 2. In other words, almost all of the speed comes from commits at the frontier, and ahead commits can be dropped without slowing decoding down.

Wider windows cost compute. Each forward pass processes the whole block, so a wider window adds compute per answer token. On NLD-3B, this is nearly free at batch size 1, but at batch size 32, B=128B = 128 is 2.4–2.5 times slower than B=32B = 32. Thus, wider windows make each forward pass more expensive, but save few passes, making decoding slower at larger concurrency. Appendix B.2 gives TPF and throughput for every setting.

Summary (RQ1). Most tokens become confident only near the frontier, ahead commits add little speed, and wider windows mainly add compute. Thus, faster decoding should come from committing more tokens at the frontier in each step. NLD’s self-speculation does this ([12] (Sahoo et al. 2026)) and is the fastest decoding policy we measured.

The order of commits causes the accuracy loss

VANILLA commits several tokens per step and picks them by confidence alone, ignoring their position. Errors could come from either choice: committing tokens in parallel, or ignoring where they are. To find which choice costs accuracy (RQ2), we change one at a time (Table 2). If committing in parallel were the problem, policies that commit one token per step would recover the accuracy. If ignoring position were the problem, policies that commit only at the frontier would recover it.

ModelBBconfidence orderfrontier ordereffect (points)EOS only at the frontier
several VANILLAone ONE-PER-STEPone LEFT-TO-RIGHTseveral FRONTIER-ONLYorderparallelismseveral EOS-AT-FRONTIERone ONE-PER-STEPorder
NLD-3B12879.179.788.087.6+8.4−0.586.287.3+1.1
NLD-3B25654.555.088.088.1+33.3−0.287.288.0+0.5
NLD-8B12887.386.790.290.1+3.2+0.291.091.0−0.8
NLD-8B25673.873.388.988.8+15.3+0.290.490.2−1.4
LLaDA, A25650.649.971.971.6+21.5+0.254.053.9+17.8
LLaDA, B25635.536.282.282.3+46.4−0.357.457.5+24.8
Fast-dLLM v212879.482.086.685.4+5.3−1.979.582.0+5.3

Table 2. Table 2: Frontier order beats confidence order at every wide window; committing one token per step does not help. Allowing EOS only at the frontier removes the gap on NLD, but not on LLaDA or Fast-dLLM v2. GSM8K accuracy (%) at each model’s wide window. Order: frontier minus confidence order, averaged over one and several tokens per step, 12\frac{1}{2}[(FRONTIER-ONLY − VANILLA) + (LEFT-TO-RIGHT − ONE-PER-STEP)]; parallelism: several minus one token per step, averaged over both orders. EOS only at the frontier: the confidence-order policies rerun with this restriction and the order effect recomputed; near zero means that early EOS tokens cause the order effect. Intervals in Appendix § 11.§ 1.

Order costs more accuracy than parallelism

On GSM8K, policies that commit tokens only at the frontier (LEFT-TO-RIGHT, FRONTIER-ONLY) beat policies that commit confident tokens anywhere in the window (ONE-PER-STEP, VANILLA) on every model by 3 to 46 points, while committing one token per step instead of several changes accuracy by at most two points (Table 2). What matters is which tokens the decoder commits: confidence order commits tokens ahead of the frontier, before the text to their left exists. A decoder can therefore keep the accuracy and still commit several tokens per step, as long as it commits them at the frontier. Appendix C.1 repeats the comparison on every task; parallelism costs a few points only on MATH-500.

Early EOS tokens cut answers short

When most of the tokens are still masked in a wide window, often no position is confident enough to pass the threshold. In that case the fallback commits the most confident position anywhere in the window, which is often an EOS token closer to the end of the block. We call such a commit an early EOS. We hypothesize that it happens due to the EOS padding that the model sees during the training. If the fallback commits EOS at position jj while the frontier is at position i<ji < j, the rest of the answer must fit into j−ij - i positions, which is often too few. On NLD-3B, answers with an early EOS from the fallback are 48% correct at B=128B = 128, and 26% at 256, while 85–89% of the same problems are solved with EOS-AT-FRONTIER decoding. In smaller windows (B≤64B \le64), EOS usually marks the right length. Appendix § 11.§ 2 gives the full counts.

Two causes, mixed differently by model. To separate the EOS effect from the commit order of the other tokens, we rerun the confidence-order policies with EOS allowed only at the frontier (Table 2, right). This splits each wide-window loss into an EOS cost (EOS-AT-FRONTIER minus VANILLA) and a content-order cost (FRONTIER-ONLY minus EOS-AT-FRONTIER). On NLD, almost all of the loss is EOS cost, on Fast-dLLM v2 almost all is content-order cost, while LLaDA has both. Keeping EOS at the frontier is therefore enough for NLD on GSM8K, but not for LLaDA, Fast-dLLM v2, or NLD-3B on HumanEval, so a fix found on one model has to be checked on the others. Appendix § 11.§ 1 gives this split for every task.

The reported collapse of LLaDA is mostly an EOS problem. In CD4LM’s released pipeline (Liang et al., [30]), LLaDA-8B falls from 77.7% on GSM8K with 32-token blocks to 13.2% with one 256-token block. Allowing EOS only at the frontier restores 64.5%, and committing the frontier at every step 79.3%, with the same number of forward passes, so a decoding policy fixes most of the collapse without retraining. Appendix § 11.§ 2 describes the pipeline and each change.

Frontier policies recover the accuracy

Which part of the policy matters. All policies share the decoder and threshold (Table 3) and differ in which ahead commits they allow: FRONTIER-FALLBACK any confident one, FRONTIER-GATING any within 16 positions, and ENTROPY-BUDGET confident ones with little entropy to their left. On NLD and LLaDA, all three recover most of FRONTIER-ONLY’s gain, so the loss there comes mostly from fallback commits far ahead, often EOS. On Fast-dLLM v2, they recover about half of it or less: confident commits a few positions ahead also cause errors. A policy that works on all four models must therefore also stop confident ahead commits, as FRONTIER-ONLY does.

PolicyNLD 3BNLD 8BLLaDA ALLaDA BFast-dLLM v2
VANILLA79.187.350.635.579.4
FRONTIER-FALLBACK85.789.667.975.482.6
FRONTIER-GATING87.390.972.680.780.8
ENTROPY-BUDGET87.191.473.280.880.3
EOS-AT-FRONTIER86.291.054.057.479.5
FRONTIER-ONLY87.690.171.682.385.4
VANILLA, default87.692.073.280.384.8

Table 3. Table 3: Frontier policies recover most of the loss; only FRONTIER-ONLY does so on all four models. GSM8K accuracy (%) at the wide window (NLD B=128B = 128; LLaDA B=256B = 256, prompts A and B; Fast-dLLM v2 B=128B = 128). Last row: VANILLA at each model’s default window (NLD and LLaDA B=32B = 32; Fast-dLLM v2, 8-token sub-blocks).

How far ahead is safe? A distance cap kk lets commits reach at most kk positions past the frontier (k=0k = 0 is FRONTIER-ONLY). Small caps (k≤2k \le2) keep FRONTIER-ONLY’s accuracy in most settings, and no cap value increases TPF by more than 18%, so there is little-to-no reason to commit ahead of the frontier at all. Appendix § 11.§ 3 gives the full sweep.

Robustness. At the wide window, FRONTIER-ONLY beats VANILLA at every threshold from 0.7 to 0.95, under sampling, and in most code, transTable 4: Most harmful departures start at fallback commits. Wide windows (NLD B=128B = 128, four tasks; Fast-dLLM v2 B=128B = 128; LLaDA B=256B = 256, prompts A and B). Fallback share: of all commits, and of the roots of harmful departures. Rate ratios per commit: harmful roots at fallback vs confident commits, and departures (harmful and helpful) at distance ≥1\ge1 vs 00 at the same confidence (range over distance bins; the reference commits the frontier, which lowers the frontier rate).

ModelFallback share (%)Rate ratio
commitsharmful rootsfallback / confidentahead / frontier
NLD-3B146120.29.6–14.6
NLD-8B357231.59.0–14.3
Fast-dLLM v2117019.17.8–12.3
LLaDA176810.31.3–1.9

Table 4. Most harmful departures start at fallback commits.

lation and summarization settings (Appendix § 11.§ 1). The gain grows with the window: on GSM8K, VANILLA at each model’s default window is within two points of FRONTIER-ONLY at the wide window (last row of Table 3). On Fast-dLLM v2, VANILLA already loses 4 points on GSM8K when its window grows from the default 8-token sub-block to the 32-token training block, and FRONTIER-ONLY recovers them (Appendix § 11.§ 4).

Summary (RQ2). Wide windows lose accuracy because of the order of commits. Early EOS tokens and early content tokens are the two causes, and their mix depends on the model, so the cheapest fix does too (EOS-AT-FRONTIER for NLD, FRONTIER-FALLBACK for LLaDA). Only FRONTIER-ONLY works on all four models at about the same speed, which makes it the safe default.

Most failures start when the fallback commits ahead of the frontier

To find which commits start the failures (RQ3), we decode each problem twice with the same model and prompt: once with VANILLA and once left-to-right with LEFT-TO-RIGHT (the reference). An error can show up many tokens after the commit that caused it, so we look at the first commit where the two decodes differ, which we call the root.

Root: the first commit where VANILLA differs from LEFT-TO-RIGHT.

A harmful departure is a problem that LEFT-TO-RIGHT solves and VANILLA fails; a helpful departure is the reverse.

Fallback commits start most failures, mainly ahead of the frontier. At wide windows, fallback commits are only 11–35% of all commits, yet they start 61–72% of the harmful departures (Table 4). Fallback commits also start most helpful departures, but net of these, they account for 56–69% of the accuracy gap on three of the four models. In addition, most of these fallback roots lie ahead of the frontier. Appendix § 12.§ 1 breaks the roots down by kind, distance and model.

Distance adds risk at the same confidence. Ahead commits have lower confidence than frontier commits, so their risk could come from confidence alone. To separate causes, we model the number of departures each commit starts by its distance and confidence (Poisson regression). On the three block-diffusion models, an ahead commit still starts departures several times as often as a frontier commit at the same confidence (Table 4). Thus, a confidence threshold alone cannot flag the risky commits, so a decoding policy should use a token’s distance from the frontier as well.

Changing one step repairs failing answers. To test whether the root commit is the one that causes the failure, we replay each failing decode with a single change: at the root’s step, the decoder commits tokens only at the frontier. As controls, we instead change up to four earlier steps of the same answer, preferring steps that make the same kind of ahead commit as the root (fallback or confident) and come just before it. We replay NLD-3B and Fast-dLLM v2 on GSM8K and MATH-500 at B=128B = 128, and NLD-3B on GSM8K at B=256B = 256. In all five cases, changing the root’s step repairs 7–39 points more failing answers than changing a control step (Appendix § 12.§ 2), so the root’s step contributes significantly to the failure. FRONTIER-ONLY makes the same change at every step, without knowing which answers fail.

Summary (RQ3). Failing decodes deviate from left-to-right generations mostly at fallback commits ahead of the frontier. On the block-diffusion models, distance from the frontier adds risk even at the same confidence, and changing only the root’s step repairs the answer more often than changing a matched control step. This is why FRONTIER-ONLY works: it never lets the fallback, or an EOS token, commit ahead of the frontier.

Related work

Decoding policies. Confidence and entropy are commonly used for decoding policies to decide how many tokens to commit [5, 23, 20] (Wu et al. 2026b; Hong et al. 2026). Analyses relate errors from independent parallel commits to the conditional total correlation of the committed positions (Wen et al., 2026; Zhang et al., 2026b). This quantity measures dependence among the committed tokens given their context. Building on this work, we ask where to commit instead of how many and find that order has the larger effect on GSM8K accuracy.

Prior studies report nearly left-to-right generation with 32-token blocks [18, 13]. The benefit of decoding easier positions first can grow with model error (Zhang et al., 2026b). Adaptive generation order also helps on Sudoku and planning tasks [25] (Ye et al. 2025). PC-Sampler uses position weights, providing a softer restriction than a fixed distance cap [19]. Other policies use stability across steps to gate commits [10] (Wang et al. 2026). Ni et al. [37] find that arbitrary order can reduce diversity and weaken pass@kk scaling, while Jeon et al. [22] attribute much of this loss to low-confidence remasking. Zhong et al. (2026) measure parallelism and generation order in aggregate across complete decodes. Different from these works, we trace individual commits and replay selected steps to test how they affect the final answer.

Several concurrent decoding policies restrict commit positions. These include PARD [18], Fork-dLLM [11], RPD [14], and frontier gating (Yeom et al., 2026). We compare policies based on these methods and examine why their restrictions affect speed and accuracy.

Effect of block size. Early termination and accuracy loss with wide-block decoding windows have been reported in several studies (Liang et al., 2026; Park et al., 2026; Kim et al., 2026b,a; Liu et al., 2026; Jazbec et al., 2026; Li et al., 2026). Proposed remedies change EOS probabilities, predict the answer length (Kobyzev et al., 2026), or retrain the model. We instead trace the early EOS token to the fallback and fix it in the decoding policy.

Revision and speculation. There also exist decoding policies that allow remasking previously committed tokens [2, 36] (Yu et al. 2026; Ye et al. 2026). A supplementary experiment on DiffusionGemma [9] finds that tokens first committed ahead of the frontier are revised more often than tokens committed at the frontier (Appendix E). There are also drafting methods that reuse spans to accelerate generation [28] (Yang et al. 2023; Wei et al. 2026; Wu 2026; Sahoo et al. 2026). Peng et al. (2025) study inference efficiency under load. We find that wide-window decoding is especially expensive at large batch sizes (Section 4).

Conclusion

In this paper, we conduct a commit-level analysis by tracing every commit of four open-weight DLMs to find where their speed and their errors come from. The speed comes from the tokens committed at the frontier, so faster decoders should increase the number of tokens committed there in each decoding step instead of widening the window. The errors start mostly where the fallback commits ahead of the frontier, and an early EOS token is the worst case. Committing only at the frontier keeps the speed and recovers most of the accuracy, even with windows eight times the training block. This gives a simple recipe for decoding: commit only at the frontier, or at least keep the fallback and EOS there. For future models, a wider window will make decoding faster only if the model is confident further ahead of the frontier, so this is what training should aim for.

Limitations

Four limitations bound our claims. (i) Decoders. We study threshold decoding that never revises a commit. Decoding policies that revise commits appear only in a side experiment on DiffusionGemma [9] (see Appendix E). (ii) Tasks and models. Our tasks are math, code, translation and summarization, where left-to-right order is natural. Tasks where other orders can help, such as Sudoku, planning [25] (Ye et al. 2025) and infilling, are out of scope. Our four main models have at most 8B parameters, and we do not study larger models. (iii) Window size. The accuracy effects are large mainly with windows wider than the training block (NLD is trained with B=32B = 32, and our main results use B=128B = 128 and 256). At the training block, VANILLA loses less accuracy and FRONTIER-ONLY gains less (Section 5.3). Our accuracy findings therefore matter most when a model is decoded with a wider window than it was trained with, as in recent evaluations of LLaDA with one 256-token block ([30]. Fast-dLLM v2 is an exception, as it loses accuracy as its window widens even inside its training block (Appendix C.4). (iv) Roots and replays. We pick roots after we know the outcome, and the control steps reduce this selection effect but do not remove it. Our Poisson model does not account for how hard each problem is, and we ran replays only on NLD-3B and Fast-dLLM v2. The roots therefore show where failures start, but the evidence that the root’s step causes the failure is limited to these two models.

Setup details

Table 5 lists the tasks.

TaskSource and splitSizeMetricModelsAnswer length
GSM8K ([8])openai/gsm8k, test1319accuracyall four179 tokens
MATH-500 ([15, 31])HuggingFaceH4/MATH-500500accuracyNLD, Fast-dLLM v2591 tokens
HumanEval ([7])EvalPlus ([34]), base tests164pass@1all four552 tokens
MBPP ([4])EvalPlus ([34]), base tests378pass@1all four461 tokens
WMT16 de→en ([6])wmt/wmt16, test, 500 random sentences500chrF ((Popović, 2015))NLD-3B, LLaDA18 words
CNN/DailyMail ([16] (See et al. 2017))abisee/cnn_dailymail 3.0.0, test, 500 random articles500ROUGE-L ([32])NLD-3B, LLaDA72 words

Table 5. Tasks. Size: number of problems. Answer length: mean over the VANILLA answers of NLD-3B at B=32B = 32 (tokens for math and code, words for translation and summarization). The CD4LM experiment (§ 5.2) uses that work’s own 440-problem GSM8K subset.

Baseline policies. FRONTIER-GATING applies VANILLA only to the masked positions j<f+16j < f + 16, where ff is the frontier (Yeom et al., 2026, default width). ENTROPY-BUDGET keeps RPD’s entropy budget (β=4\beta= 4 nats, summed over the masked positions to the left that are not committed in the same step) and its fallback (the most confident of the first 32 masked positions). It leaves out RPD’s layer-stability test, which adds candidates with confidence between 0.6 and 0.9 (He et al., [14]). Appendix C.1 compares it with RPD’s released decoder.

Grading. We grade math answers with math-verify and code with the EvalPlus base tests (Liu et al., [34]). NLD-8B and Fast-dLLM v2 often add an example code block after the function, so we grade the block that defines the function instead of the last block. This matters only on NLD-8B MBPP, where grading the last block would put FRONTIER-ONLY and LEFT-TO-RIGHT about 5 points below VANILLA instead of about 2 points above it. With the stricter EvalPlus tests, every difference larger than 1.1 points keeps its sign.

Hardware. All runs used NVIDIA A100-40GB GPUs.

Confidence and speed

Confidence by distance and tokens per step

First confidence by distance. Figure 2a uses the left-to-right decodes of Table 2 (LEFT-TO-RIGHT; on Fast-dLLM v2, sub-block 1), where the forward pass predicts a token and a confidence for every masked position but the decoder commits only the frontier. We leave out the positions that VANILLA’s fallback would have picked before they ever passed the threshold (0–25%), and average the shares per problem. Across models, tasks and windows, 52–76% of positions first pass the threshold within seven positions of the frontier. In these GSM8K runs, 60–99% of the positions that become confident 8–31 positions ahead, by model and distance, predict the token that the model writes there later, and 25–84% of those further ahead.

Tokens per step by block size. On NLD-3B, a position at a given offset in its block is committed at about the same step for every BB from 8 to 128: the mean commit step grows by 0.24–0.34 steps per position on GSM8K and 0.19–0.24 on MATH-500 (Figure 3). A larger block therefore mainly adds positions, and tokens per step (commits divided by steps) stop growing. From B=32B = 32 to 128, they change by a factor of 0.99–1.03 on NLD-3B (MATH-500: 1.17; Figure 2b), while NLD-8B still gains 10–17% on GSM8K and MATH-500.

Mean commit step by position for block sizes 8–128

Figure 3. A larger block mainly adds positions. Mean commit step by position for B=8B = 8–128128 (NLD-3B, GSM8K, VANILLA); the curves nearly coincide.

Other models. LLaDA commits 2.7 tokens per step at B=8B = 8 and 3.0–3.3 at B=32B = 32 to 128 (GSM8K, prompt A; prompt B: 2.4–2.7), and Fast-dLLM v2’s TPF stops growing at $B = 64 (Table 6). FRONTIER-ONLY behaves the same way: from B=32B = 32 to the wide windows, its TPF changes by a factor of 0.96–1.24, and by more than 1.10 only on NLD-8B.

ModelTaskWindowAheadVANILLAONE-PER-STEPFRONTIER-ONLYTPF VANILLATPF FRONTIER-ONLY
NLD-3BGSM8K4688.688.9–1.51–
NLD-3BGSM8K81389.189.2–2.06–
NLD-3BGSM8K162387.988.8–2.46–
NLD-3BGSM8K323287.688.388.12.752.61
NLD-3BGSM8K644187.088.988.42.722.74
NLD-3BGSM8K1284979.179.787.62.272.77
NLD-3BGSM8K2565254.5–88.11.242.51
NLD-3BMATH-5004677.075.4–1.56–
NLD-3BMATH-50081172.876.0–2.30–
NLD-3BMATH-500161870.075.2–2.84–
NLD-3BMATH-500322874.475.675.23.183.18
NLD-3BMATH-500643571.075.674.83.603.35
NLD-3BMATH-5001284369.673.873.23.543.38
NLD-3BMATH-5002564864.0–72.82.813.31
NLD-3BHumanEval4776.876.2–1.46–
NLD-3BHumanEval81572.072.6–1.81–
NLD-3BHumanEval162478.076.8–2.04–
NLD-3BHumanEval323173.275.079.32.122.12
NLD-3BHumanEval644070.775.678.72.122.19
NLD-3BHumanEval1284564.670.181.11.982.21
NLD-3BMBPP4772.572.8–1.44–
NLD-3BMBPP81672.572.5–1.71–
NLD-3BMBPP162572.274.6–1.89–
NLD-3BMBPP323471.274.674.91.952.05
NLD-3BMBPP644066.470.471.22.032.22
NLD-3BMBPP1284963.564.372.51.822.20
NLD-8BGSM8K4592.0––1.57–
NLD-8BGSM8K81092.2––2.25–
NLD-8BGSM8K161692.5––2.86–
NLD-8BGSM8K322692.092.091.33.173.26
NLD-8BGSM8K643591.1––3.17–
NLD-8BGSM8K1283787.386.790.13.373.78
NLD-8BGSM8K2564473.8–88.83.504.05
NLD-8BMATH-5004587.0––1.57–
NLD-8BMATH-50081085.2––2.25–
NLD-8BMATH-500161787.0––2.73–
NLD-8BMATH-500322584.085.885.63.243.20
NLD-8BMATH-500643485.4––3.37–
NLD-8BMATH-5001284181.282.485.03.633.40
NLD-8BMATH-5002564679.4–82.43.613.63
NLD-8BHumanEval4686.0––1.54–
NLD-8BHumanEval81284.1––2.08–
NLD-8BHumanEval162283.5––2.22–
NLD-8BHumanEval322883.582.383.52.742.52
NLD-8BHumanEval643882.9––2.68–
NLD-8BHumanEval1285386.086.084.82.152.64
NLD-8BMBPP4885.7––1.45–
NLD-8BMBPP81984.9––1.51–
NLD-8BMBPP162884.7––1.64–
NLD-8BMBPP322983.183.982.82.152.04
NLD-8BMBPP643782.3––2.12–
NLD-8BMBPP1283882.381.783.92.012.33
LLaDA (A)GSM8K323573.2––3.32–
LLaDA (A)GSM8K2567450.6–71.62.953.13
LLaDA (B)GSM8K323630.3––2.85–
LLaDA (B)GSM8K2567235.5–82.31.762.72
Fast-dLLM v2GSM8K4686.4––2.28–
Fast-dLLM v2GSM8K81284.8–86.72.852.78
Fast-dLLM v2GSM8K162081.8––3.19–
Fast-dLLM v2GSM8K323080.8–86.13.383.20
Fast-dLLM v2GSM8K644079.2–85.73.563.34
Fast-dLLM v2GSM8K1285079.482.085.43.323.14
Fast-dLLM v2MATH-5004661.4––2.28–
Fast-dLLM v2MATH-50081160.4–66.42.922.78
Fast-dLLM v2MATH-500161954.6––3.15–
Fast-dLLM v2MATH-500322752.6–62.03.343.11
Fast-dLLM v2MATH-500643752.2–62.23.523.32
Fast-dLLM v2MATH-5001284652.2–62.03.413.27

Table 6. All model ×\times task ×\times window settings (greedy). Ahead: ahead commits among the commits of VANILLA (%). Accuracy (%) of VANILLA, ONE-PER-STEP and FRONTIER-ONLY; TPF: tokens per forward. Window: BB for NLD and LLaDA (256 = one block); for Fast-dLLM v2, the sub-block for 4–16, the whole 32-token training block at 32, and BB for 64 and 128. A dash marks a setting we did not run.

Speed and compute

Ahead commits and TPF. Table 6 gives the ahead share and the TPF of every setting. Both count only answer tokens, including the EOS token that ends the answer, and leave out the EOS padding after it, which VANILLA also commits: on NLD-3B GSM8K, padding is 9%, 32% and 55% of its commits at B=32B = 32, 128 and 256. Over the 34 settings of Figure 2c, FRONTIER-ONLY reaches 0.92–2.03 times the TPF of VANILLA; with Fast-dLLM v2’s code runs added (40 settings), 0.88–2.03, with the lowest value on MBPP at B=32B = 32. The largest ratios come from answers that VANILLA cuts short: on problems whose two answers differ in length by at most 10%, the range is 0.88–1.27.

Compute. Each NLD forward pass processes all BB positions of the block, so it processes BB/TPF Table 5: Tasks. Size: number of problems. Answer length: mean over the VANILLA answers of NLD-3B at B=32B = 32 (tokens for math and code, words for translation and summarization). The CD4LM experiment (§ 5.2) uses that work’s own 440-problem GSM8K subset.

positions per answer token (Table 6). On NLD-3B GSM8K, this is 12 at B=32B = 32 for both policies, 46 for FRONTIER-ONLY and 57 for VANILLA at 128, and 102 and 207 at 256. Across NLD settings, it grows 3.4–3.8 times for FRONTIER-ONLY from B=32B = 32 to 128, and 6.4–8.3 times to 256. LLaDA processes the prompt and the whole 256-token answer in every forward pass: 112–149 positions per answer token for FRONTIER-ONLY and 118–231 for VANILLA.

Wall-clock time. Table 7 gives NLD-3B throughput. At batch size 1, the GPU has spare capacity, so the extra positions of a wider window cost almost nothing: B=128B = 128 is even 6–11% faster than B=32B = 32. At batch size 32, the GPU is fully used, and every extra position costs time. On Fast-dLLM v2 at batch size 1, FRONTIER-ONLY takes 0.77–1.12 times as long as VANILLA per answer token (two runs shared each GPU).

Decoderbatch 1batch 8batch 32
self-speculation4.723.421.93
+ LoRA drafter4.833.602.00
threshold decoding
VANILLA, B=32B = 322.951.570.63
FRONTIER-ONLY, B=32B = 322.851.540.62
VANILLA, B=128B = 1283.120.880.25
FRONTIER-ONLY, B=128B = 1283.160.910.26
VANILLA, B=256B = 2562.760.470.14
FRONTIER-ONLY, B=256B = 2562.870.460.13

Table 7. NLD-3B throughput relative to AR decoding on the same GPU. Steady-state throughput over the first 512 tokens of 64 long MATH-500 answers (A100-40GB, eager PyTorch). The self-speculation and B=32B = 32 rows come from one benchmark. The larger-block rows come from a second benchmark of two jobs run in opposite order, which agree within 3%.

Accuracy

Every task, threshold and policy

Order and parallelism on every task. Table 9 repeats the comparison of Table 2, with intervals, in every setting where we ran all four policies. Table 10 repeats it with EOS allowed only at the frontier in every cell, which needed new runs of ONE-PER-STEP only.

Model, windowtaskVANILLAONE-PER-STEPLEFT-TO-RIGHTFRONTIER-ONLYorderparallelisminteraction
NLD-3B, B=128B = 128GSM8K79.179.788.087.6+8.4 ± 1.7−0.5 ± 0.8+0.2 ± 1.5
NLD-3B, B=128B = 128MATH-50069.673.876.673.2+3.2 ± 2.9−3.8 ± 2.0+0.8 ± 3.3
NLD-3B, B=128B = 128HumanEval64.670.180.581.1+13.4 ± 6.3−2.4 ± 2.7+6.1 ± 5.2
NLD-3B, B=128B = 128MBPP63.564.373.872.5+9.3 ± 4.4−1.1 ± 1.9−0.5 ± 3.6
NLD-3B, B=256B = 256GSM8K54.555.088.088.1+33.3 ± 2.6−0.2 ± 1.0+0.6 ± 2.0
NLD-8B, B=128B = 128GSM8K87.386.790.290.1+3.2 ± 1.5+0.2 ± 0.9−0.6 ± 1.8
NLD-8B, B=128B = 128MATH-50081.282.483.885.0+2.6 ± 2.6+0.0 ± 2.1+2.4 ± 3.9
NLD-8B, B=128B = 128HumanEval86.086.085.484.8−0.9 ± 4.0−0.3 ± 1.8−0.6 ± 3.7
NLD-8B, B=128B = 128MBPP82.381.784.183.9+2.0 ± 2.6+0.1 ± 0.7−0.8 ± 1.7
NLD-8B, B=256B = 256GSM8K73.873.388.988.8+15.3 ± 2.3+0.2 ± 1.0−0.6 ± 2.1
LLaDA, A, B=256B = 256GSM8K50.649.971.971.6+21.5 ± 2.8+0.2 ± 0.9−1.1 ± 1.9
LLaDA, B, B=256B = 256GSM8K35.536.282.282.3+46.4 ± 2.9−0.3 ± 0.8+0.9 ± 1.7
Fast-dLLM v2, B=128B = 128GSM8K79.482.086.685.4+5.3 ± 1.9−1.9 ± 0.9+1.4 ± 1.7
Fast-dLLM v2, B=128B = 128MATH-50052.259.666.062.0+8.1 ± 3.1−5.7 ± 2.2+3.4 ± 4.2

Table 9.

ModelwindowtaskVANILLAONE-PER-STEPLEFT-TO-RIGHTFRONTIER-ONLYwithoutwithpar.int.kept
NLD-3BB=128B = 128GSM8K86.287.388.087.6+8.4 ± 1.7+1.1 ± 1.3−0.8 ± 0.6+0.6 ± 1.113%
NLD-3BB=128B = 128MATH-50070.874.276.673.2+3.2 ± 2.9+2.4 ± 2.8−3.4 ± 1.8+0.0 ± 2.775%
NLD-3BB=128B = 128HumanEval70.173.280.581.1+13.4 ± 6.3+9.1 ± 5.3−1.2 ± 2.4+3.7 ± 5.268%
NLD-3BB=128B = 128MBPP69.872.273.872.5+9.3 ± 4.4+2.1 ± 4.0−1.9 ± 1.7+1.1 ± 3.023%
NLD-3BB=256B = 256GSM8K87.288.088.088.1+33.3 ± 2.6+0.5 ± 1.3−0.4 ± 0.6+0.9 ± 1.11%
NLD-8BB=128B = 128GSM8K91.091.090.290.1+3.2 ± 1.5−0.8 ± 1.2−0.0 ± 0.8+0.1 ± 1.5−25%
NLD-8BB=128B = 128MATH-50082.684.483.885.0+2.6 ± 2.6+0.9 ± 2.5−0.3 ± 1.7+3.0 ± 3.535%
LLaDA, AB=256B = 256GSM8K90.490.288.988.8+15.3 ± 2.3−1.4 ± 1.3+0.0 ± 0.8−0.3 ± 1.6−9%
LLaDA, prompt AB=256B = 256GSM8K54.053.971.971.6+21.5 ± 2.8+17.8 ± 2.7−0.1 ± 0.8−0.4 ± 1.683%
LLaDA, prompt BB=256B = 256GSM8K57.457.582.282.3+46.4 ± 2.9+24.8 ± 2.6−0.0 ± 0.8+0.3 ± 1.753%
Fast-dLLM v2B=128B = 128GSM8K79.582.086.685.4+5.3 ± 1.9+5.3 ± 1.9−1.8 ± 0.9+1.3 ± 1.799%
Fast-dLLM v2B=128B = 128MATH-50052.658.866.062.0+8.1 ± 3.1+8.3 ± 3.2−5.1 ± 2.2+2.2 ± 4.2102%

Table 10.

Frontier policies. Table 8 compares the policies on the four NLD-3B tasks. FRONTIER-FALLBACK recovers 78–82% of FRONTIER-ONLY’s GSM8K gain on NLD, but on NLD-3B only 52% on HumanEval, 20% on MBPP and none on MATH-500, and on HumanEval FRONTIER-GATING and ENTROPY-BUDGET also stay 10 points below FRONTIER-ONLY. RPD’s released decoder, which keeps the layer st…

TaskVANILLAFRONTIER-ONLYFRONTIER-FALLBACKFRONTIER-GATINGENTROPY-BUDGETEOS-AT-FRONTIERVANILLA, B=32B = 32
GSM8K79.187.685.787.387.186.287.6
MATH-50069.673.268.871.872.270.874.4
HumanEval64.681.173.270.771.370.173.2
MBPP63.572.565.372.071.769.871.2

Table 8.

ENTROPY-BUDGET also stays 10 points below FRONTIER-ONLY. RPD’s released decoder, which keeps the layer-stability test that our ENTROPY-BUDGET leaves out, matches FRONTIER-ONLY on LLaDA (73.2 and 81.1 against 71.6 and 82.3 for prompts A and B) at 15% more TPF.

Code and the training block. On code at the wide window, FRONTIER-ONLY gains 9–17 points over VANILLA on NLD-3B and Fast-dLLM v2 and 1–3 points on LLaDA, and stays within two points on NLD-8B. At the training block (B=32B = 32), it changes accuracy by at most 1.6 points on NLD math and NLD-8B code, but still gains 4–6 points on NLD-3B code.

Thresholds. Table 11 varies the threshold. At every threshold from 0.7 to 0.95, the gap between FRONTIER-ONLY and VANILLA at the wide window is significant (the lowest bound of its 95% interval is 0.2 points, on MATH-500); at B=32B = 32, it is significant only at 0.7 and 0.8. A lower threshold commits more tokens per step, so we also compare the two policies at the same speed. On NLD-3B GSM8K at threshold 0.8, the two policies have about the same TPF and FRONTIER-ONLY is 9.6 points more accurate, but on MATH-500 the gap can vanish: VANILLA at 0.8 and FRONTIER-ONLY at 0.7 both reach 4.4 TPF and 67%.

Setting0.70.80.90.95
NLD-3B, B=128B = 12873.9 / 85.5 (+11.6)77.3 / 86.9 (+9.6)79.1 / 87.6 (+8.5)79.5 / 87.9 (+8.4)
TPF4.0 / 3.83.0 / 3.22.3 / 2.81.8 / 2.5
NLD-3B, B=128B = 128, MATH-50061.0 / 67.2 (+6.2)67.0 / 70.8 (+3.8)69.6 / 73.2 (+3.6)70.2 / 75.0 (+4.8)
TPF5.2 / 4.44.4 / 4.13.5 / 3.43.1 / 3.0
NLD-3B, B=32B = 3281.0 / 85.4 (+4.4)84.7 / 87.1 (+2.4)87.6 / 88.1 (+0.5)88.6 / 88.2 (−0.4)
TPF4.0 / 3.43.3 / 3.12.7 / 2.62.4 / 2.3
LLaDA, B=256B = 25639.2 / 80.7 (+41.5)35.4 / 81.8 (+46.4)35.5 / 82.3 (+46.9)35.9 / 82.3 (+46.3)
TPF4.2 / 4.02.7 / 3.31.8 / 2.71.3 / 2.3
Fast-dLLM v2, B=128B = 12868.6 / 81.2 (+12.6)75.1 / 85.0 (+9.9)79.4 / 85.4 (+6.1)79.8 / 86.5 (+6.7)
TPF5.1 / 4.44.1 / 3.83.3 / 3.12.8 / 2.7

Table 11.

Sampling. With three seeds at temperature 0.7, FRONTIER-ONLY beats VANILLA on GSM8K at the wide windows by 7.2 ± 1.6 points on NLD-3B (B=128B = 128), 19.5 on NLD-8B (B=256B = 256), 44.5 on LLaDA and 5.3 on Fast-dLLM v2.

Translation and summarization. On NLD-3B at B=128B = 128, FRONTIER-ONLY beats VANILLA by 5.7 ± 1.3 chrF and 7.5 ± 1.0 ROUGE-L (paired

differences of sentence-level scores), and ONE-PER-STEP changes either score by at most 0.2. Under VANILLA, 83% of the summaries are less than half as long as at the narrow window; under FRONTIER-ONLY, none are. On LLaDA at B=256B = 256, FRONTIER-ONLY raises ROUGE-L from 2.4 to 24.6, because every VANILLA summary ends almost at once, but changes translation little (+0.8±\pm1.1 chrF): the reference translations are short (median 17 words), and only 2% of VANILLA translations are cut short, so there is little early EOS to fix.

Early EOS

Early EOS on NLD-3B. An EOS token is early when it is committed while a position to its left is still masked. In NLD’s decoder, a block is finished once no position before its first EOS is masked, and a block that holds an EOS ends the answer, so an early EOS fixes the remaining length before the reasoning is written. On GSM8K at B=256B = 256, 80% of answers commit their EOS early, about half of them through the fallback. Answers with a fallback early EOS are 26% correct on GSM8K at B=256B = 256, 48% at B=128B = 128 and 37% on MATH-500 at B=256B = 256, against 89%, 85% and 69% of the same problems under EOS-AT-FRONTIER (Figure 4a), so a fallback early EOS costs 32–63 points. A confident early EOS costs 3–14 points.

Figure 4. Early EOS commits cause most of the wide-window loss on NLD and most of LLaDA’s reported collapse. (a) NLD: accuracy of the answers whose EOS token the fallback committed early, and of the same problems under EOS-AT-FRONTIER. (b) LLaDA-8B, GSM8K, in the released CD4LM evaluation pipeline (decoder, prompt and grader; 440 problems). Each of the three middle bars changes one setting of the released one-block setup; committing the leftmost masked position first removes the drop.

Where the fallback puts EOS. Fallback EOS commits have confidence of about 0.64–0.70 and lie a median 22–31 positions ahead of the frontier, which fits the padding explanation of § 5.2: in training, positions far beyond the written text are often padding, so EOS is the model’s best guess there, though not a confident one. The worst case is an EOS 1–4 positions after the frontier, which the fallback commits in 73–89% of cases: on GSM8K at B=256B = 256, 139 answers have one, and 1% of them are correct, against 84% under EOS-AT-FRONTIER.

Smaller windows. At B≤64B \le64, 18–75% of answers still commit their EOS ahead of the frontier, a median 5–20 positions ahead, but their lengths and accuracy match EOS-AT-FRONTIER (median length ratio 0.97–1.00; accuracy within one point on GSM8K and MATH-500 at B=32B = 32). In a small window, EOS can only be placed a few positions ahead, close to where the answer ends anyway.

LLaDA in the CD4LM pipeline. CD4LM reports LLaDA-8B at 13.8% on GSM8K with one 256-token block and 77.4% with 32-token blocks [30], and LLaDA’s authors trace such losses to the EOS padding in its fine-tuning data [38 Sec. 3.2]. In the released evaluation pipeline (440 problems), whose decoder commits the single most confident position per step, we get 13.2% and 77.7%. Figure 4b changes one setting at a time: allowing EOS only at the frontier gives 64.5%, committing the frontier at every step 79.3% (a gain of 66.1 points, p=3×10−83p = 3 \times10^{-83}, with the same number of forward passes), and changing only the chat template 27.5%. Most of the collapse therefore comes from where EOS is placed, and about 15 more points from the order of the content tokens.

Several leftmost positions per step. With two or four tokens per step, committing the leftmost positions still beats committing the most confident ones on GSM8K by 36–60 points, but without a confidence check it costs accuracy on code: on MBPP, the two leftmost positions per step give 22.0%, against 43.1% for the two most confident (p=6×10−14p = 6 \times10^{-14}); HumanEval: 28.7% against 30.5%, p=0.76p = 0.76, as PARD found for a fixed leftmost policy [18]. FRONTIER-ONLY, which keeps the confidence check, reaches 82.3% on GSM8K at B=256B = 256 with 2.7 TPF in our own pipeline (prompt B).

Distance cap

The cap policy first commits what FRONTIER-ONLY would commit: the confident positions in a row, starting at the frontier. It then also commits each confident masked position jj with j−f≤kj - f \le k, where ff is the leftmost position still masked after that. If no position qualifies, the fallback picks only among the masked positions within kk of the Table 8: NLD-3B accuracy (%) by task and policy. Policies at B=128B = 128, and VANILLA at B=32B = 32 for comparison; Table 9 gives ONE-PER-STEP and LEFT-TO-RIGHT.

Table 9: Order × parallelism in every setting with all four policies (accuracy, %; effects in points, ± half the width of the 95% problem-bootstrap interval; Section 5.1). VANILLA: confidence order, several tokens per step; ONE-PER-STEP: confidence order, one; LEFT-TO-RIGHT: frontier order, one; FRONTIER-ONLY: frontier order, several.

Table 10: Order × parallelism with EOS allowed only at the frontier in every cell: VANILLA becomes EOS-AT-FRONTIER, and ONE-PER-STEP is rerun with the same restriction; LEFT-TO-RIGHT and FRONTIER-ONLY commit only at the frontier, where the restriction changes nothing. Accuracy (%) and effects (points, 95% problem bootstrap). Par. and int.: parallelism and interaction effects. Kept: the order effect with the restriction as a share of the order effect without it; it is negative where the effect changes sign.

Table 11: Threshold sweep. Accuracy (%) of VANILLA / FRONTIER-ONLY with the gap (FRONTIER-ONLY minus VANILLA, points), and below it the TPF of VANILLA / FRONTIER-ONLY. NLD-3B; LLaDA B=256B = 256 (prompt B); Fast-dLLM v2 B=128B = 128; GSM8K unless stated.

frontier. We decode greedily at the wide windows of Table 2 (Table 12).

Model, taskk=0k = 012481632∞\infty
NLD-3B, GSM8Kacc.87.687.987.987.887.687.186.479.1
Δ\Delta0+0.4+0.4+0.2+0.1−0.5−1.1−8.5
±\pm±1.3±1.4±1.5±1.4±1.4±1.5±2.0
TPF2.772.812.822.852.872.862.802.27
NLD-3B, MATH-500acc.73.273.273.672.472.069.869.269.6
Δ\Delta0+0.0+0.4−0.8−1.2−3.4−4.0−3.6
±\pm±3.2±3.6±3.3±3.5±3.4±3.5±3.3
TPF3.383.633.703.993.663.823.633.54
NLD-3B, HumanEvalacc.81.175.676.869.567.768.969.564.6
Δ\Delta0−5.5−4.3−11.6−13.4−12.2−11.6−16.5
±\pm±4.6±5.2±7.0±6.7±6.1±5.8±6.4
TPF2.212.222.192.222.232.222.181.98
NLD-3B, MBPPacc.72.573.371.273.372.069.871.263.5
Δ\Delta0+0.8−1.3+0.8−0.5−2.6−1.3−9.0
±\pm±3.3±3.7±3.7±4.4±4.4±4.2±4.8
TPF2.202.232.112.042.142.122.001.82
NLD-8B, GSM8Kacc.90.191.691.392.091.791.490.587.3
Δ\Delta0+1.4+1.1+1.9+1.6+1.2+0.4−2.9
±\pm±1.2±1.2±1.3±1.3±1.4±1.4±1.8
TPF3.783.613.673.453.633.603.593.37
NLD-8B, MATH-500acc.85.081.884.483.082.280.680.081.2
Δ\Delta0−3.2−0.6−2.0−2.8−4.4−5.0−3.8
±\pm±3.1±3.0±3.0±2.9±3.1±3.3±3.0
TPF3.403.723.783.833.753.663.703.63
LLaDA (B), GSM8Kacc.82.382.782.182.081.080.680.835.5
Δ\Delta0+0.4−0.2−0.3−1.4−1.7−1.5−46.9
±\pm±1.7±1.9±1.9±2.0±2.0±2.0±3.0
TPF2.722.782.812.862.922.972.971.76
Fast-dLLM v2, GSM8Kacc.85.484.585.283.581.479.579.679.4
Δ\Delta0−0.9−0.2−1.9−4.0−6.0−5.8−6.1
±\pm±1.7±1.7±1.9±2.1±2.1±2.1±2.3
TPF3.143.223.223.263.273.293.313.32
Fast-dLLM v2, MATH-500acc.62.057.057.055.252.651.051.852.2
Δ\Delta0−5.0−5.0−6.8−9.4−11.0−10.2−9.8
±\pm±3.5±3.7±4.0±4.1±4.1±3.9±3.8
TPF3.273.263.333.353.283.463.403.41

Table 12. Distance cap: accuracy (%), its difference to k=0k = 0 (Δ\Delta, points, ±\pm half the width of its paired 95% problem-bootstrap interval) and tokens per forward when confident commits and the fallback may reach at most kk positions past the frontier; k=0k = 0 is FRONTIER-ONLY, k=∞k = \infty VANILLA.

Caps of one or two positions keep FRONTIER-ONLY’s accuracy in six of nine model–task settings. How far ahead is safe depends on the model: Fast-dLLM v2 loses accuracy within a few positions, and LLaDA only beyond 32. A larger cap also adds little speed: TPF rises by at most 18%, and in tokens per second the caps we timed (k=2k = 2 and 88) are 2–12% slower than FRONTIER-ONLY at batch sizes 1, 8 and 32 (NLD-3B, B=128B = 128, setup of Table 7).

Fast-dLLM v2 window sweep

Fast-dLLM v2 commits only inside 8-token sub-blocks of its 32-token training block, which limits how far ahead it can commit. Under VANILLA, widening the window lowers accuracy step by step (Table 6): from the default sub-block 8 to B=128B = 128, GSM8K drops from 84.8 to 79.4 and MATH-500 from 60.4 to 52.2 (Holm p≤0.002p \le0.002), and most of the drop is already there at the whole 32-token training block, while FRONTIER-ONLY stays at 85.4–86.7 and 62.0–66.4. On code, VANILLA falls from 63.4 to 50.0 on HumanEval and from 64.6 to 57.1 on MBPP at window 128 (Holm p≤0.003p \le0.003), while FRONTIER-ONLY keeps 66.5 and 68.8 at the same TPF.

Inside the default decoder, FRONTIER-ONLY gains 6.0 points on MATH-500 (Holm p=0.018p = 0.018) and 7.1 on MBPP, as PARD reports for this model (Hou and Kwok [18]). Strict left-to-right decoding (sub-block 1) reaches 87.1% on GSM8K and 67.2% on MATH-500, and FRONTIER-ONLY in the default decoder comes within 0.4 and 0.8

points of it at 2.8 times the TPF. The Fast-dLLM v2 paper already noted that wider sub-blocks hurt its 1.5B model even at one token per step (Wu et al. (2026a)).

Roots and replays

Roots

Definition. The reference is the model’s own LEFT-TO-RIGHT decode at the same window (on Fast-dLLM v2, sub-block 1). Both decodes are greedy with the same prompt, so they agree token for token up to the first block that holds a differing answer position, and inside that block the root is the differing position that VANILLA committed first. Distances are measured after each step: a token committed in a row with the frontier in the same step has d=0d = 0, like the frontier itself, so only commits with a masked position still to their left count as ahead. Inside the root block, we count only the commits made up to the root’s step; the later commits are mostly at the frontier, and counting them would make frontier commits look safer than they are. Roots in the EOS padding after an VANILLA answer that ends early are a separate kind; they are 20% of the harmful roots on NLD-3B and 19% on NLD-8B.

Poisson model. Figure 5 shows the rate ratios of confident commits by distance (a) and the shares of commits and roots by kind (b): most fallback roots lie ahead of the frontier. For the model, we put every diffusion commit in a cell of task ×\times distance bin (0, 1, 2–3, 4–7, 8–15, ≥16\ge16) ×\times confidence bin (nine bins from ≥0.999\ge0.999 to <0.50< 0.50, four of them below the threshold) ×\times answer position (0–63, 64–255, ≥256\ge256) ×\times EOS token. Per model, we fit

Figure 5. Fallback commits start most departures, and on the block-diffusion models, distance from the frontier adds risk at the same confidence. (a) Departures (harmful or helpful) started per confident commit, by distance to the frontier, relative to a frontier commit at the same confidence (Poisson model with answer position, EOS token and task; log scale; 95% intervals). (b) Shares of all commits, of harmful roots and of helpful roots by kind, at wide windows (NLD B=128B = 128, Fast-dLLM v2 B=128B = 128, LLaDA B=256B = 256). Reference: each model’s own LEFT-TO-RIGHT decode; commits counted up to the root’s step.

log⁡E[roots]=log⁡(commits)+β0+βtask+βdist+βconf+βpos+βEOS\log\mathbb{E}[\mathrm{roots}] = \log(\mathrm{commits}) + \beta_0 + \beta_{\mathrm{task}} + \beta_{\mathrm{dist}} + \beta_{\mathrm{conf}} + \beta_{\mathrm{pos}} + \beta_{\mathrm{EOS}} to all roots, to harmful roots (with the two top confidence bins merged, since harmful roots at ≥0.999\ge0.999 are rare) and to confident commits only (Table 13).

TermNLD-3BNLD-8BFast-dLLM v2LLaDA
all departures
distance 110.9−1.5+2.010.9^{+2.0}_{-1.5}9.0−1.0+1.29.0^{+1.2}_{-1.0}8.7−1.4+1.68.7^{+1.6}_{-1.4}1.6−0.3+0.41.6^{+0.4}_{-0.3}
distance 2–312.7−1.9+2.212.7^{+2.2}_{-1.9}10.7−1.2+1.710.7^{+1.7}_{-1.2}11.9−1.9+1.811.9^{+1.8}_{-1.9}1.9−0.3+0.21.9^{+0.2}_{-0.3}
distance 4–714.6−2.2+2.914.6^{+2.9}_{-2.2}11.9−1.5+2.011.9^{+2.0}_{-1.5}12.3−1.9+2.412.3^{+2.4}_{-1.9}1.3±0.21.3 \pm0.2
distance 8–1510.2−2.0+2.410.2^{+2.4}_{-2.0}14.3−2.1+2.414.3^{+2.4}_{-2.1}10.8±2.210.8 \pm2.21.4−0.4+0.31.4^{+0.3}_{-0.4}
distance ≥16\ge169.6−1.9+2.69.6^{+2.6}_{-1.9}12.0−2.3+2.712.0^{+2.7}_{-2.3}7.8−1.5+2.07.8^{+2.0}_{-1.5}1.5−0.4+0.31.5^{+0.3}_{-0.4}
harmful departures
distance 19.5−3.6+8.89.5^{+8.8}_{-3.6}7.7−3.2+5.67.7^{+5.6}_{-3.2}11.4−4.7+9.411.4^{+9.4}_{-4.7}0.9±0.30.9 \pm0.3
distance 2–313.3−4.9+11.013.3^{+11.0}_{-4.9}11.1−4.7+7.911.1^{+7.9}_{-4.7}13.7−5.5+11.013.7^{+11.0}_{-5.5}0.7±0.20.7 \pm0.2
distance 4–715.2−6.3+13.115.2^{+13.1}_{-6.3}10.3−5.2+7.910.3^{+7.9}_{-5.2}14.7−6.1+12.314.7^{+12.3}_{-6.1}0.5−0.2+0.10.5^{+0.1}_{-0.2}
distance 8–159.6−4.5+9.89.6^{+9.8}_{-4.5}14.6−7.2+13.214.6^{+13.2}_{-7.2}11.2−5.5+9.711.2^{+9.7}_{-5.5}0.4−0.2+0.30.4^{+0.3}_{-0.2}
distance ≥16\ge168.0−4.2+8.38.0^{+8.3}_{-4.2}4.0−4.0+7.04.0^{+7.0}_{-4.0}9.4−4.5+9.19.4^{+9.1}_{-4.5}0.9−0.3+0.20.9^{+0.2}_{-0.3}
confident commits only
distance 110.2−2.7+4.110.2^{+4.1}_{-2.7}7.9−4.8+13.87.9^{+13.8}_{-4.8}8.0−2.5+3.98.0^{+3.9}_{-2.5}3.3−1.1+1.53.3^{+1.5}_{-1.1}
distance 2–37.2−2.0+2.87.2^{+2.8}_{-2.0}13.0−6.0+19.813.0^{+19.8}_{-6.0}13.7−3.5+5.313.7^{+5.3}_{-3.5}0.9−0.3+0.40.9^{+0.4}_{-0.3}
distance 4–77.9−2.1+3.17.9^{+3.1}_{-2.1}5.0−2.6+7.95.0^{+7.9}_{-2.6}9.8−2.6+3.89.8^{+3.8}_{-2.6}0.7±0.30.7 \pm0.3
distance 8–155.8−1.7+2.65.8^{+2.6}_{-1.7}6.3−3.0+8.96.3^{+8.9}_{-3.0}8.6−2.7+4.08.6^{+4.0}_{-2.7}0.9−0.5+0.60.9^{+0.6}_{-0.5}
distance ≥16\ge166.2−1.9+3.06.2^{+3.0}_{-1.9}5.2−2.6+6.65.2^{+6.6}_{-2.6}8.2−2.8+3.68.2^{+3.6}_{-2.8}2.2−0.7+1.02.2^{+1.0}_{-0.7}
ahead (d≥1d \ge1) vs frontier, within each task
per task12.3; 12.5; 17.0; 8.210.4; 10.8; 37.5; 9.211.0; 8.93.9; 0.8

Table 13. Poisson model (reference: each model’s own LEFT-TO-RIGHT decode with the same prompt): rate ratios of roots per commit against a frontier commit at the same confidence (nine bins), answer position, EOS token and task; wide windows (NLD B=128B = 128, four tasks; Fast-dLLM v2 B=128B = 128, GSM8K and MATH-500; LLaDA B=256B = 256, prompts A and B); 95% intervals from 500 problem resamples within tasks, given as the distance to their upper (superscript) and lower (subscript) end, or ±\pm when the two are equal. Per-task values in the order GSM8K, MATH-500, HumanEval, MBPP (LLaDA: prompts A, B).

Confidence and distance. Confidence matters most: fallback bins start departures 90–1500 times as often as the ≥0.999\ge0.999 bin, and the position and EOS terms are small. At the same confidence, ahead commits still start harmful departures 4.0–15.2 times as often as frontier commits on the block-diffusion models, and 0.4–0.9 times as often on LLaDA, whose far-ahead predictions nevertheless disagree more often with the text written there later (Early predictions below).

Other references. Two other references give the same result. With NLD’s AR mode as the reference, fallback commits are 14% and 41% of the commits of NLD-3B and NLD-8B but start 65% and 88% of their harmful departures, and ahead commits start 1.6–3.6 times as many departures as frontier commits at the same confidence. With ONE-PER-STEP as the reference, ahead commits start 2.2–19.5 times as many roots per commit as frontier commits, over 24 settings.

Helpful departures. A helpful departure, where the VANILLA answer is right and the reference wrong, has a root defined the same way (Table 14). Fallback commits start 69–90% of the helpful roots, against 61–72% of the harmful ones, and per commit they start helpful departures 11–22 times as often as confident commits do: the fallback is where decoding leaves the left-to-right path in either direction, and a guess is sometimes right. EOS padding roots are almost only harmful (NLD-3B: 20% of harmful and 2% of helpful roots).

Modelroots harmfulroots helpfulfallback share of commitsfallback share of roots harmfulfallback share of roots helpfulfallback / confident per commit harmfulfallback / confident per commit helpfulfallback share of the net gap
NLD-3B2998114%61% ± 674% ± 920.2−5.7+7.320.2^{+7.3}_{-5.7}19.1−6.7+13.419.1^{+13.4}_{-6.7}56% ± 8
NLD-8B16310535%72% ± 790% ± 531.5−14.0+52.431.5^{+52.4}_{-14.0}19.9−8.8+27.119.9^{+27.1}_{-8.8}38%−69+2338\%^{+23}_{-69}
Fast-dLLM v22397511%70% ± 673% ± 1019.1−4.3+6.819.1^{+6.8}_{-4.3}22.3−8.9+14.222.3^{+14.2}_{-8.9}69% ± 9
LLaDA89412517%68% ± 369% ± 810.3−1.4+1.510.3^{+1.5}_{-1.4}10.7−3.1+5.710.7^{+5.7}_{-3.1}68% ± 4

Table 14. Harmful and helpful roots at wide windows (the runs and reference of Table 4, commits counted up to the root). Helpful: the VANILLA answer is right and the reference answer wrong. Shares ±\pm half the width of the 95% interval; rate ratios (roots per commit, fallback vs confident) and the fallback share of the net accuracy gap to the reference (harmful minus helpful roots; a decomposition of the gap, not a causal estimate) with the distance to the upper (superscript) and lower (subscript) end of their 95% interval, or ±\pm when the two were equal; 500 problem resamples within tasks.

Net share of the gap. Harmful roots outnumber helpful ones 1.6 to 7 times, and net of the helpful ones, departures that start at fallback commits make up 56–69% of the accuracy gap to the reference on NLD-3B, Fast-dLLM v2 and LLaDA. On LLaDA, the share counts only failures with a root; counting its 127 cut-short failures, which have none, lowers it to about 58%. On NLD-8B, where VANILLA loses few answers, the interval includes zero. This share says where departures start; the replays of Appendix § 12.§ 2 test whether changing that step changes the answer.

Which policy repairs which roots. On the harmful problems of the wide-window runs, with NLD’s AR mode as the reference on NLD, EOS-AT-FRONTIER solves 82–96% of the problems whose answer is cut short or whose root is EOS padding, but only 3–37% of the others. FRONTIER-ONLY solves 70–100% of them for every kind of root, on all four models.

Narrow windows. Fast-dLLM v2 at its default window (sub-block 8) and with the whole 32-token block shows the same pattern. Fallback commits are 12% of all commits but start 69% of the harmful departures, and confident commits at distance 1 start departures 33 times as often as frontier commits.

Early predictions. Roots depend on the reference decode. A second measure does not: whether a prediction made ahead of the frontier matches the token the model writes there later. In the left-to-right decodes of Appendix § 10.§ 1, we take at every step the positions that VANILLA would commit from the same forward pass, without committing them. For each position, we keep the first step at which VANILLA would commit it, and call the prediction wrong if its token differs from the one the model commits when the frontier reaches it. Confident predictions in a row with the frontier are wrong 0.1–0.5% of the time on all four models, confident ahead predictions 6.7–47% and fallback ahead predictions 8–32% (NLD and Fast-dLLM v2 at B=128B = 128, four tasks on NLD; LLaDA has only 39 fallback predictions ahead of the frontier). In a logistic model with confidence bins, answer position and task, the odds of a wrong prediction grow with distance on every model, LLaDA included: 3.6–20 times the odds at d=0d = 0 for distance 1, and 71–4157 times for distance 16 or more. This measure covers only positions inside the answer. Most far-ahead commits in the VANILLA decodes are EOS padding, which agrees with the reference and lowers the root rates at large distances (Figure 5a).

Replays

Design. For each problem, VANILLA decodes once at batch size 1, and we choose the steps to change from this decode. A replay repeats the decode with one change at one step: at that step, the decoder commits only at the frontier, as FRONTIER-ONLY would. For a fallback step, this commits the frontier instead of the fallback’s pick. At batch size 1, the replay computes exactly what the original decode computed up to that step, so any difference in the final answer comes from the change and from what the decoder does after it. With larger batches, bf16 results depend on which answers share a batch.

Which steps we change. Root: the root’s step, on

harmful departures whose root is an ahead commit. Matched controls: up to four earlier steps of the same answer. For each earlier step, we take its ahead commit that is most like the root, and rank the steps by four criteria in order: same kind as the root (fallback or confident; content or EOS padding), same block, fewest steps before the root, and closest distance. Sampled steps: four steps per failing answer, drawn uniformly from all steps that commit ahead of the frontier, without using the root. Only afterwards do we label each sampled step as the root’s step, before it or after it. Helpful roots: on answers that VANILLA gets right and the reference gets wrong, the root’s step and two sampled steps.

Selection. We pick the root after we know the outcome, so the root’s step could look special only because of how we choose it. The matched controls are still somewhat earlier than the roots (NLD-3B GSM8K, B=128B = 128: step 9.3 against 12.6 within the block) and less often fallback (47% against 65%). The sampled steps do not use the root at all, and a logistic regression adjusts for the remaining differences: it predicts repair on all replayed steps of the failing answers from whether the step is the root’s, where it lies relative to the root, its kind, distance, step, block and answer position, and the setting. The root effect, as an average marginal effect with a problem bootstrap, is +7.3 ± 3.1 points on NLD-3B (555 answers) and +30.8 ± 7.7 on Fast-dLLM v2 (239 answers).

Results. Against the matched controls, the root’s step repairs more in all five settings (Table 15). On NLD-3B GSM8K, changing earlier steps also often repairs the failure, so there the failure is not tied to one step. Sampled steps (Table 16) repair 4–24% of failing answers, and fallback steps repair

Table 15. At the root’s step, committing only at the frontier repairs more failing answers than at matched control steps. Repaired share (%) of failing answers when the root’s step or a matched earlier control step (ctrl.) is changed, and their paired difference (points, ± half the width of the 95% problem-bootstrap interval); nn: failing answers replayed. Table 16 gives sampled steps and helpful roots.

ModelTaskBBnnRootCtrl.Root − ctrl.
NLD-3BGSM8K1287744.229.2+14.9 ± 9.0
NLD-3BGSM8K25632529.522.3+7.2 ± 3.2
NLD-3BMATH1283324.26.3+17.9 ± 14.9
Fast-dLLM v2GSM8K12811447.48.2+39.2 ± 8.7
Fast-dLLM v2MATH1287328.810.6+18.2 ± 8.7

more than confident ones in every setting. The two model families differ in where the useful steps lie. On NLD-3B GSM8K, sampled steps before the root repair 42% (B=128B = 128, 36 steps) and 27% (B=256B = 256, 303 steps) of the answers, against 22% and 17% after it, so these failures can often be undone before the first departure; on Fast-dLLM v2, steps before the root repair 6% and 0%, and the root’s step matters most.

Helpful roots. On helpful departures, changing the root’s step breaks the correct answer 24–37% of the time, against 2–18% at two sampled steps of the same answers: a paired difference of +21 to +29 points on the three GSM8K settings, with intervals above zero, and not significant on MATH-500. The root’s step therefore matters in both directions, as the helpful roots of Table 14 suggest.

DiffusionGemma

DiffusionGemma-26B ([9]), a mixture-of-experts model with 3.8B active parameters, can re-sample accepted tokens, so it can undo an early guess. We test whether it uses this to fix ahead tokens. Its released sampler decodes 256-token answers block by block. Each step samples all positions, accepts the lowest-entropy set within an entropy bound of 0.1 and re-noises the rest with uniform random tokens, so an accepted token can still change later (temperature 0.8 to 0.4, at most 48 steps). We freeze accepted tokens at their first acceptance, and the same rule keeps choosing new positions, so the acceptance budget stays the same. We freeze either every accepted token, or only the tokens first accepted ahead of the frontier, at the frontier or at the first step. As a control, we freeze as many tokens at the same steps, chosen at random among the newly accepted ones. We run three seeds and average them within each problem (Table 17).

Frozen tokensacc.vs defaultshare of loss
none (default)94.3––
frontier tokens93.8−0.4±0.4-0.4 \pm0.415%
step-0 tokens93.6−0.7±0.4-0.7 \pm0.424%
as many random (control)92.6−1.6±0.5-1.6 \pm0.560%
ahead tokens92.0−2.2±0.5-2.2 \pm0.581%
all accepted tokens91.5−2.7±0.5-2.7 \pm0.5100%

Table 17. DiffusionGemma-26B with frozen tokens. Accepted tokens are frozen at their first acceptance, with the same acceptance budget. Three seeds averaged within each problem; four tasks pooled (2361 problems); HumanEval graded leniently. Differences to the default sampler ±\pm half the width of the 95% interval from re-sampling problems. Share of loss: each drop as a share of the 2.7-point drop from freezing all accepted tokens. The control freezes as many tokens at the same steps, at random positions.

Freezing every accepted token costs 1.8–3.8 points across the three seeds, and tokens first accepted ahead of the frontier are revised 1.3–6.2 times as often as frontier tokens. Freezing only the ahead tokens reproduces 81% of that loss, and freezing as many random tokens 60%. Ahead tokens are 57–79% of all tokens, so their share of the loss is about 1.1 times their share of the tokens; for frontier tokens, it is about 0.5 times.

References

  1. [1]Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. 2025. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models. In The Thirteenth International Conference on Learning Representations (ICLR).
  2. [2]Ali Asaria, Tony Salomone, and Deep Gandhi. 2026. Neither Parallel Nor Sequential: How DiffusionGemma Actually Commits Tokens. arXiv:2606.14620. Preprint, arXiv:2606.14620.arxiv.org/abs/2606.14620
  3. [3]Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021a. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, volume 34.arxiv.org/abs/2107.03006
  4. [4]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021b. Program Synthesis with Large Language Models. arXiv:2108.07732. Preprint, arXiv:2108.07732.arxiv.org/abs/2108.07732
  5. [5]Heli Ben-Hamu, Itai Gat, Daniel Severo, Niklas Nolte, and Brian Karrer. 2025. Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking. In Advances in Neural Information Processing Systems, volume 38.
  6. [6]Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, and 2 others. 2016. Findings of the 2016 Conference on Machine Translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131–198.DOI
  7. [7]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374. Preprint, arXiv:2107.03374.arxiv.org/abs/2107.03374
  8. [8]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. Preprint, arXiv:2110.14168.
  9. [9]DiffusionGemma Team. 2026. DiffusionGemma Technical Report. arXiv:2608.00146. Preprint, arXiv:2608.00146.arxiv.org/abs/2608.00146
  10. [10]Xingyou Fang, Jingxing Zhong, Xiaosong Yuan, and Xiaofeng Zhang. 2026. Information-Guided Frontier Decoding: Contextual Utility-Driven Commitment in dMLLMs. arXiv:2608.26641. Preprint, arXiv:2608.26641.arxiv.org/abs/2608.26641
  11. [11]Stipe Frković, Metod Jazbec, and Christian A. Naesseth. 2026. Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Models. arXiv:2609.39859. Preprint, arXiv:2609.39859.arxiv.org/abs/2609.39859
  12. [12]Yonggan Fu, Lexington Whalen, Abhinav Garg, Chengyue Wu, Maksim Khadkevich, Nicolai Oswald, Enze Xie, Daniel Egert, Sharath Turuvekere Sreenivas, Shizhe Diao, Chenhan Yu, Ye Yu, et al. 2026. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding. arXiv:2607.05722. Preprint, arXiv:2607.05722.arxiv.org/abs/2607.05722
  13. [13]Shanshan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. 2026. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation. In The Fourteenth International Conference on Learning Representations (ICLR).arxiv.org/abs/2506.20639
  14. [14]Zhenghao He, Bohan Liu, Guangzhi Xiong, and Aidong Zhang. 2026. Reliable Parallel Decoding in Masked Diffusion Language Models. arXiv:2609.36452. Preprint, arXiv:2609.36452.arxiv.org/abs/2609.36452
  15. [15]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
  16. [16]Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In Advances in Neural Information Processing Systems, volume 28, pages 1693–1701.arxiv.org/abs/1506.03340
  17. [17]Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70. Feng Hong, Geng Yu, Yushi Ye, Haicheng Huang, Huangjie Zheng, Ya Zhang, Yanfeng Wang, and Jiangchao Yao. 2026. Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs. In The Fourteenth International Conference on Learning Representations.
  18. [18]Kai Syun Hou and James Kwok. 2026. Rethinking the General Order of Block Diffusion Language Models. arXiv:2607.24306. Preprint, arXiv:2607.24306.arxiv.org/abs/2607.24306
  19. [19]Pengcheng Huang, Shuhao Liu, Zhenghao Liu, Yukun Yan, Shuo Wang, Zulong Chen, and Tong Xiao. 2025. PC-Sampler: Position-Aware Calibration of Decoding Bias in Masked Diffusion Models. arXiv:2508.13021. Preprint, arXiv:2508.13021.
  20. [20]Daniel Israel, Guy Van den Broeck, and Aditya Grover. 2025. Accelerating Diffusion LLMs via Adaptive Parallel Decoding. In Advances in Neural Information Processing Systems, volume 38.arxiv.org/abs/2506.00413
  21. [22]Moongyu Jeon, Dongjae Jeon, Bumjun Kim, Mingyu Kim, and Albert No. 2026. Low-Confidence Remasking Traps Flexibility: Realizing Arbitrary-Order Potential for Diverse Rollouts in Diffusion LLMs. arXiv:2609.34509. Preprint, arXiv:2609.34509.arxiv.org/abs/2609.34509
  22. [23]Siva Rajesh Kasa, Yasong Dai, Sumit Negi, and Hongdong Li. 2026. Fast-dLLM++: Fréchet Profile Decoding for Faster Diffusion LLM Inference. arXiv:2606.02955. Preprint, arXiv:2606.02955.arxiv.org/abs/2606.02955
  23. [25]Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. 2025. Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions. In Proceedings of the 42nd International Conference on Machine Learning (ICML). arXiv:2502.06768.arxiv.org/abs/2502.06768
  24. [28]Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. 2024. CLLMs: Consistency Large Language Models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 25426–25440. PMLR.arxiv.org/abs/2403.00835
  25. [30]Yihao Liang, Ze Wang, Hao Chen, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Emad Barsoum, Zicheng Liu, and Niraj K. Jha. 2026. CD4LM: Consistency Distillation and Adaptive Decoding for Diffusion Language Models. arXiv:2601.02236. Preprint, arXiv:2601.02236.DOI
  26. [31]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s Verify Step by Step. In The Twelfth International Conference on Learning Representations (ICLR).arxiv.org/abs/2305.20050
  27. [32]Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81.
  28. [34]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems, volume 36.
  29. [35]Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157.pubmed.ncbi.nlm.nih.gov/20254758
  30. [36]Mojtaba Nafez and James Henderson. 2026. Know When to Hold ’em: Correct-Token Retention in Uniform-State Diffusion Language Models. arXiv:2610.01275. Preprint, arXiv:2610.01275.arxiv.org/abs/2610.01275
  31. [37]Zanlin Ni, Shenzhi Wang, Yang Yue, Tianyu Yu, Weilin Zhao, Yeqing Hua, Tianyi Chen, Jun Song, Cheng Yu, Bo Zheng, and Gao Huang. 2026. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models. In Proceedings of the 43rd International Conference on Machine Learning (ICML).arxiv.org/abs/2601.15165
  32. [38]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Large language diffusion models. In Advances in Neural Information Processing Systems, volume 38, pages 56354–56392.arxiv.org/abs/2502.09992

Paper details

vlnc:W_jm7wm32z
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents