betaFeedback

WorksComputer Science

DUET: Dual-Sparse Inference for Block Diffusion Language Models

Abstract

Diffusion language models (DLMs) enable parallel token generation through iterative denoising, yet their inference remains costly due to repeated attention over a growing historical prefix and repeated processing of the entire current generation block. Historical attention becomes increasingly expensive as generation progresses, while all current-block positions are propagated through the Transformer at every denoising step even though only a small fraction are typically resolved at that step. In this paper, we introduce Duet, a training-free dual-sparse inference framework that addresses these challenges through prefix Key-Value (KV) sparsity and early query sparsity, respectively. Specifically, Duet exploits the concentration and cross-step stability of historical attention within each generation block to construct a compact historical KV subset that is reused across denoising steps. For query sparsity, Duet predicts update positions from shallow representations and propagates only the corresponding queries through the suffix Transformer layers, avoiding deep computation at positions unlikely to be updated in the current step. Experiments show that Duet maintains competitive accuracy while achieving 1.34×\times/2.83×\times end-to-end/per-step speedups on LLaDA2.1-mini and 2.41×\times/4.72×\times on SDAR-8B-Chat-b32 at a 32K context length.

Introduction

DLMs have recently emerged as an alternative paradigm for text generation, replacing strictly left-to-right autoregressive decoding with iterative denoising over multiple token positions [19, 18, 24]. By updating several positions in parallel, DLMs offer a promising path toward more flexible and potentially faster generation [20]. Recent advances have demonstrated competitive performance across general language understanding, mathematical reasoning, and code generation tasks [4], further motivating interest in efficient DLM inference.

However, the iterative nature of denoising introduces substantial computational redundancy. In block DLM (BDLM) decoding, a current generation block is iteratively refined over multiple denoising steps while conditioning on an generated prefix [1]. This inference pattern repeatedly incurs two major sources of unnecessary computation. First, every denoising step attends over the full historical prefix, making attention increasingly expensive as the context grows [16]. Second, all positions in the current block are typically propagated through the full Transformer stack at every denoising step, although only a subset of unresolved positions are ultimately selected for the current update [14]. These two forms of redundancy jointly make BDLM inference increasingly expensive.

Motivated by these inefficiencies, we introduce Duet, a training-free dual-sparse inference framework for BDLMs. Duet addresses the two sources of redundant computation with complementary sparsity mechanisms. First, current-block queries concentrate much of their historical attention on a compact prefix subset that remains stable across denoising steps within the same generation block. Duet exploits this structure through historical prefix KV sparsity, selecting a compact historical KV subset once for each generation block and reusing it throughout the block. Second, the positions selected by the final decoding update become predictable from early Transformer representations. Duet leverages this predictability through early query sparsity within block, applying the model’s original decoding policy at an early selection layer to identify active query rows and propagating only these rows through the remaining layers. Together, the two mechanisms reduce both historical attention and deep token-wise computation, including attention projections, FFN/MoE modules, and other token-wise operations. We evaluate Duet on LLaDA2.1-mini [5] and SDAR-8B-Chat-b32 [7] across long-context and standard reasoning benchmarks. Duet consistently reduces inference cost while preserving competitive generation quality. At a 32K context length, Duet achieves 1.34×/2.83× end-to-end/per-step speedups on LLaDA2.1-mini and 2.41×/4.72× on SDAR-8B-Chat-b32. We further analyze the two empirical observations underlying Duet, validating that compact historical prefix subsets remain stable within a generation block and that final update positions become highly predictable at early Transformer layers.

Figure 1. Illustration of different sparsity patterns in BDLM inference. FOCUS only reduces computation along the query dimension, whereas LoSA only sparsifies the historical prefix. Our method is designed around the two orthogonal computation dimensions, selectively reducing both the query rows propagated through deep layers and the historical prefix KV states they attend to.

Our contributions are summarized as follows:

  • We identify two complementary sources of redundancy in BDLM inference and show that historical-prefix relevance remains stable within a generation block, while final update positions can be predicted from early Transformer representations.

  • We propose Duet, a training-free dual-sparse inference framework that combines prefix KV sparsity with early query sparsity to reduce both historical attention and deep token-wise computation.

  • We conduct extensive experiments on two representative BDLMs across long-context, reasoning, code generation, and knowledge benchmarks, demonstrating competitive generation quality and substantial inference speedups.

Preliminaries

BDLM Inference. DLMs generate text through iterative denoising rather than left-to-right autoregressive decoding. We consider BDLM inference, where generation proceeds one block at a time. Let PP denote the length of the currently generated prefix and BB the size of the current generation block. At denoising step ss, the input sequence is

x(s)=(x1,…,xP,xP+1(s),…,xP+B(s)).(1)\mathbf{x}^{(s)} = \left(x_1,\ldots,x_P,x_{P+1}^{(s)},\ldots,x_{P+B}^{(s)}\right). \tag*{(1)}

After the final layer, the model applies its decoding policy to the current-block representations and resolves a subset of positions, while the unresolved positions are processed again in subsequent denoising steps. Once all positions in the block are resolved, the decoded block is appended to the historical prefix and generation proceeds to the next block.

Within a generation block, the historical prefix remains fixed while the current-block representations evolve across denoising steps. Therefore, historical prefix KV states can be cached and reused, whereas the BB current-block positions are repeatedly passed through the Transformer [22 a]. At layer ll and step ss, we denote the current-block queries by Q(l,s)∈RB×Hq×dh\mathbf{Q}^{(l,s)} \in\mathbb{R}^{B \times H_q \times d_h} and the cached historical keys and values by KP(l),VP(l)∈RP×Hkv×dh\mathbf{K}^{(l)}_P,\mathbf{V}^{(l)}_P \in\mathbb{R}^{P \times H_{kv} \times d_h}. Each current-block query attends to the PP cached prefix positions and the BB positions in its own block. Suppressing head counts and constant factors, the per-layer, per-step costs of dense attention and the feed-forward (FFN) or mixture-of-experts (MoE) module are

Tattn,dense(l)=O(B(P+B)dh+Bcproj(l)),Tff,dense(l)=O(Bcff(l)),(2)T_{\mathrm{attn,dense}}^{(l)} = O\left(B(P+B)d_h + B c_{\mathrm{proj}}^{(l)}\right), \qquad T_{\mathrm{ff,dense}}^{(l)} = O\left(B c_{\mathrm{ff}}^{(l)}\right), \tag*{(2)}

where cproj(l)c_{\mathrm{proj}}^{(l)} is the per-row attention projection cost and cff(l)c_{\mathrm{ff}}^{(l)} is the per-row cost of the FFN or activated MoE experts, including routing. The first attention term accounts for query–key interactions and value aggregation. These costs expose two complementary reduction opportunities: attending to fewer historical KV states and propagating fewer current-block query rows through deep Transformer layers.

Prefix-Sparse Attention. We first consider sparsity along the historical-prefix dimension. For clarity, consider a single attention head and omit layer and denoising step superscripts. Let qi∈Rdh\mathbf{q}_i \in\mathbb{R}^{d_h} be a current-block query and KP,VP∈RP×dh\mathbf{K}_{P},\mathbf{V}_{P} \in\mathbb{R}^{P \times d_h} and KB,VB∈RB×dh\mathbf{K}_{B},\mathbf{V}_{B} \in\mathbb{R}^{B \times d_h} be the historical-prefix and current-block KV states, respectively. Dense attention computes

oi=softmax⁡(qi[KP;KB]Tdh)[VP;VB].(3)\mathbf{o}_i = \operatorname{softmax}\left(\frac{\mathbf{q}_i[\mathbf{K}_{P};\mathbf{K}_{B}]^{\mathsf{T}}}{\sqrt{d_h}}\right)[\mathbf{V}_{P};\mathbf{V}_{B}]. \tag*{(3)}

Prefix-sparse attention instead retains only a subset SP⊆{1,…,P}S_{P} \subseteq\{1,\ldots,P\} of historical positions and computes

o~i=softmax⁡(qi[KP[SP];KB]Tdh)[VP[SP];VB],(4)\widetilde{\mathbf{o}}_i = \operatorname{softmax}\left(\frac{\mathbf{q}_i[\mathbf{K}_{P}[S_{P}];\mathbf{K}_{B}]^{\mathsf{T}}}{\sqrt{d_h}}\right)[\mathbf{V}_{P}[S_{P}];\mathbf{V}_{B}], \tag*{(4)}

where the semicolon concatenates along the KV sequence dimension and [SP][S_{P}] gathers the indexed historical rows. The original attention mask, when present, applies to both expressions. Only the historical KV portion changes; prefix sparsity does not remove the current-block KV states. Prefix sparsity therefore targets the context-length-dependent component of BDLM inference, which becomes increasingly important as PP grows.

Query-Sparse Deep Computation. BDLM inference also repeatedly propagates all BB current-block positions through the Transformer, even though only a subset may require deep computation at a given denoising step. We denote the active block positions at step ss by SQ(s)⊆{1,…,B}S_{Q}^{(s)} \subseteq\{1,\ldots,B\} and their number by Qs=∣SQ(s)∣Q_s = \lvert S_{Q}^{(s)} \rvert. Query sparsity omits the inactive query rows from subsequent layers: these rows do not incur deep attention projections or query-side attention, nor the corresponding FFN/MoE computation. The current block still provides its KV states to the active queries. Thus, prefix sparsity reduces the historical KV context, while query sparsity reduces the number of current-block positions receiving deep computation. The next section describes how Duet combines them during BDLM decoding.

Method

In this section, we present Duet, a training-free dual-sparse inference framework that reduces redundant computation in BDLM decoding through prefix KV sparsity and early query sparsity. We first provide an overview of Duet, and then describe the two sparsity mechanisms in detail.

Overview

As shown in Figure 2, Duet applies two complementary forms of sparsity to a generation block of BB positions conditioned on a prefix of length PP. The first denoising step (s=0s = 0) runs densely, providing the block queries and the full historical KV states needed for prefix selection. Duet then constructs a compact prefix cache from these states without an additional dense model forward. In subsequent steps, it processes the block densely through an early query-selection layer lQl_Q and propagates only active positions through the remaining layers. We use zero-based indices for layers and denoising steps.

Figure 2. Overview of Duet. (A) Block-level prefix selection: After the first dense step, Duet selects a compact historical prefix subset for each sparse suffix layer and reuses the resulting KV cache within the generation block. (B) Step-wise query selection: in subsequent denoising steps, Duet identifies active query rows at an early selection layer and propagates only these rows through the remaining Transformer layers, reducing redundant computation in both attention and FFN/MoE. Active and bypassed states are then recombined for the next denoising step.

For each suffix layer l>lQl > l_Q, prefix KV sparsity uses block queries and historical keys to construct a subset SP(l)⊆{1,…,P}S_P^{(l)} \subseteq\{1,\ldots,P\}. This subset remains fixed within the current block; at the next block, selection is repeated over the full historical KV cache. Layers 0,…,lQ0,\ldots,l_Q retain the full prefix KV states. At each subsequent step s≥1s \geq1, early query sparsity identifies an active set SQ(s)⊆{1,…,B}S_Q^{(s)} \subseteq\{1,\ldots,B\} at layer lQl_Q. Only these QsQ_s rows traverse layers lQ+1,…,L−1l_Q+1,\ldots,L-1, while the other rows bypass the suffix computation.

Within the sparse suffix, active queries attend to the selected historical KV states together with the current block KVs. Restricting the suffix to SQ(s)S_Q^{(s)} also avoids attention projections and FFN/MoE computation for inactive rows. The resulting active states are scattered back to their original block positions and combined with the bypassed states. The process repeats until the block is resolved, after which it is appended to the historical prefix.

Prefix KV Sparsity

Design intuition. Within a generation block, the historical prefix remains fixed while the current block is iteratively refined. Empirically, we observe that block queries concentrate a large fraction of their prefix-attention mass on a compact subset of historical tokens, and that this subset remains largely stable throughout the denoising steps within the same block. This motivates selecting a compact historical context once at the beginning of each generation block and reusing the resulting KV cache throughout that block. We validate this block-level stability in § 4.3.

Query–prefix similarity. Prefix selection is performed for each suffix layer l>lQl > l_Q, using queries captured during the first dense step. Let qh,i(l,0)∈Rdh\mathbf{q}_{h,i}^{(l,0)} \in\mathbb{R}^{d_h} be the attention-space query at position ii and query head hh, and let kg(h),j(l)∈Rdh\mathbf{k}_{g(h),j}^{(l)} \in\mathbb{R}^{d_h} be the historical key at position jj in its corresponding KV head g(h)g(h) under grouped-query attention. We measure their similarity by

ahij(l)=⟨qh,i(l,0),kg(h),j(l)⟩.(5)a_{hij}^{(l)} = \left\langle\mathbf{q}_{h,i}^{(l,0)}, \mathbf{k}_{g(h),j}^{(l)} \right\rangle. \tag*{(5)}

Block-shared prefix selection. Rather than constructing a separate historical context for each query, Duet selects one prefix subset shared across the block and query heads. Given a target budget BPB_P, each of the BHqB H_q query-head rows selects its m=max⁡(1,⌈BP/(BHq)⌉)m = \max(1,\lceil B_P/(B H_q)\rceil) highest-scoring historical positions, and the candidates are merged. We score each historical position across all block queries and query heads by

rj(l)=max⁡1≤i≤B1≤h≤Hqahij(l),(6)r_j^{(l)} = \max_{\substack{1 \leq i \leq B \\ 1 \leq h \leq H_q}} a_{hij}^{(l)}, \tag*{(6)}

which captures the strongest match to any query-head row. If the candidate union contains fewer than min⁡(BP,P)\min(B_P,P) distinct positions, we fill it with the highest-scoring remaining positions according to rj(l)r_j^{(l)}; if it contains more, we retain the min⁡(BP,P)\min(B_P,P) highest-scoring candidates. Thus, under a strict prefix budget, the shared set has size Kl=∣SP(l)∣=min⁡(BP,P)K_l = |S_P^{(l)}| = \min(B_P,P). Ranking by the unscaled dot product is equivalent to ranking by the scaled attention logit for a fixed head dimension. Scoring the full prefix costs O(BPdh)O(B P d_h) per suffix layer, suppressing head counts, and is performed once per generation block.

The selected indices gather the cached keys and values, K~P(l)=KP(l)[SP(l)]\widetilde{\mathbf{K}}_P^{(l)} = \mathbf{K}_P^{(l)}[S_P^{(l)}] and V~P(l)=VP(l)[SP(l)]\widetilde{\mathbf{V}}_P^{(l)} = \mathbf{V}_P^{(l)}[S_P^{(l)}]. This compact cache is reused throughout the current block; the full historical cache remains available for selection at the next block.

Early Query Sparsity

Design intuition. At each denoising step, standard BDLM inference propagates all positions in the current generation block through every Transformer layer, even though only a subset of unresolved positions are ultimately selected for the current update. We observe that these update positions can already be predicted from early Transformer representations. This suggests that query selection can be performed before the final layer, allowing Duet to avoid redundant deep computation on positions unlikely to be updated. Accordingly, Duet designates an early Transformer layer lQl_Q as the selection layer and propagates only the predicted active query rows through the remaining deep layers. We empirically validate this early predictability in § 4.3.

Early prediction of query positions. At a subsequent denoising step s≥1s \ge1, Duet evaluates layers 0,…,lQ0,\ldots,l_Q densely over the current block, including attention to the full historical prefix, and obtains the intermediate representations H(lQ,s)\mathbf{H}^{(l_Q,s)}. The model’s original decoding rule is then applied to H(lQ,s)\mathbf{H}^{(l_Q,s)} to estimate which masked positions are likely to be selected for the current update. Let U(s)\mathcal{U}^{(s)} denote the unresolved positions at step ss. The query retention ratio ρQ\rho_Q sets a nominal target of ⌈ρQ∣U(s)∣⌉\lceil\rho_Q|\mathcal{U}^{(s)}| \rceil active unresolved queries, selected according to the model’s decoding policy (confidence-based for LLaDA and in block order for sequential SDAR); previously resolved positions are always retained as well. Together, these positions form SQ(s)S_Q^{(s)}, whose QsQ_s rows are propagated through the suffix layers. In this way, the retention ratio controls the amount of deep computation devoted to unresolved positions, while reusing the original decoding rule keeps early query selection consistent with the underlying BDLM inference procedure without introducing an additional learned predictor.

Sparse suffix computation. For s≥1s \ge1 and l>lQl > l_Q, only the QsQ_s active query rows attend to the compact historical KV cache indexed by SP(l)S_P^{(l)} and the BB current-block KV positions. The first dense step initializes the current-block KV cache at all BB positions. Subsequent sparse steps update the selected entries and reuse the others, so active queries still attend to all BB block positions. Using the cost conventions of (2), the per-layer sparse costs are

Tattn,sparse(l,s)=O(Qs(Kl+B)dh+Qscproj(l)),Tff,sparse(l,s)=O(Qscff(l)).(7)T_{\mathrm{attn,sparse}}^{(l,s)} = O\left(Q_s(K_l+B)d_h+Q_sc_{\mathrm{proj}}^{(l)}\right), \qquad T_{\mathrm{ff,sparse}}^{(l,s)} = O\left(Q_sc_{\mathrm{ff}}^{(l)}\right). \tag*{(7)}

Compared with dense computation, the suffix attends over Kl+BK_l+B rather than P+BP+B KV positions and processes QsQ_s rather than BB rows through attention projections and FFN/MoE. These expressions apply only after the first dense step and beyond lQl_Q; the first step and the preceding layers retain the dense costs of (2). Because selection at the next block revisits the full prefix, the shorter attended KV sequence does not imply a reduction in the stored full historical KV cache.

Position-preserving update. Sparse suffix computation preserves the original block indices of all active query rows, allowing their updated representations to be written back to the corresponding positions after the deep layers. At the final decoding stage, the original decoding rule is applied only to the active unresolved positions SQ(s)∩U(s)S_Q^{(s)} \cap\mathcal{U}^{(s)}. Unselected unresolved positions bypass the suffix computation, remain masked at this step, and are reconsidered at the next step. Thus, the per-step update set may differ from dense decoding. The updated active rows, bypassed unresolved rows, and previously resolved rows are recombined to reconstruct the full block state for the next denoising step. Query selection is repeated until all positions are resolved. Once the current block is fully decoded, it is appended to the historical context, and Duet proceeds to the next generation block with an updated prefix.

Experiments

In this section, we evaluate Duet in terms of generation quality and inference efficiency across multiple models and benchmarks. We then analyze the two empirical observations underlying its design: the stability of prefix sparsity and the early predictability of query positions.

Experimental Setup

Models. We evaluate Duet on two representative BDLMs, LLaDA2.1-mini [5] and SDAR-8B-Chat-b32 [7]. Because the predictability of decoding positions emerges at different depths across models, we use model-specific query-sparsity configurations. For LLaDA2.1-mini, we set the query retention ratio to ρQ=0.7\rho_Q = 0.7 and the selection layer to lQ=1l_Q = 1, while retaining its original decoding policy. For SDAR-8B-Chat-b32, we set ρQ=0.5\rho_Q = 0.5 and lQ=5l_Q = 5, and use sequential decoding with 16 denoising steps. At the selection layer, Duet applies the same decoding policy used for the final update to predict active query rows.

Benchmarks. We evaluate accuracy on both long- and shorter-context tasks. For long-context evaluation, we use LongBench [3], which spans a diverse set of long-document understanding and generation tasks. We additionally evaluate mathematical reasoning on GSM8K [8] and MATH-500 [11], code generation on HumanEval [6], and knowledge-intensive question answering on MMLU [10].

Baselines and implementation. We compare Duet against dense inference and two sparse-inference baselines, FOCUS [14] and LoSA [23]. FOCUS primarily sparsifies computation within the current generation block, whereas LoSA focuses on sparsifying attention over the historical prefix. Both baselines use their default configurations. To ensure a fair systems comparison, we implement all methods within the same inference framework and preserve the algorithmic behavior described in the original papers. We further optimize the core sparse operators with Triton kernels and use the same runtime stack across methods. Since the original FOCUS implementation is optimized for shorter-context, larger-batch settings, we adapt its execution path to long-context decoding while keeping its selection and sparse-computation logic unchanged.

Efficiency evaluation. We evaluate inference efficiency on 80 NarrativeQA samples at 8K, 16K, and 32K context lengths with a fixed generation length of 256 tokens on an NVIDIA H800 GPU. We report end-to-end and stable per-step speedups in the main text, with detailed results provided in the appendix; stable per-step latency excludes the first denoising step to remove initialization overhead.

Main Results

Generation quality. Figure 3 shows the LongBench quality–throughput trade-off, Table 1 reports the corresponding Macro-F1 scores, and Table 2 reports short-context quality. At nominal budget 512, Duet matches dense Macro-F1 on LLaDA2.1-mini (51.35 vs. 51.33) with a 1.33× end-to-end speedup. On SDAR-8B, it reaches 49.15 versus 49.69 with a 2.42× speedup. LoSA also preserves quality, but at nominal budget 256 its measured prefix KV counts average 434 and 405 tokens on the two models, respectively, exceeding the configured budget. Full task-wise scores and KV counts are in Appendix Table 4.

Figure 3. LongBench quality–throughput trade-off. Five-task Macro-F1 relative to each model’s dense score (100%) versus 32K end-to-end throughput. Absolute Macro-F1 scores are reported in Table 1; task-wise results appear in Appendix Table 4.

ModelDenseFOCUSLoSADuet (Ours)
25651210242565121024
LLaDA2.1-mini51.3350.1751.24 (434)51.48 (833)51.29 (1566)51.1351.3551.34
SDAR-8B49.6949.6549.45 (405)49.65 (794)49.73 (1507)48.7749.1549.25

Table 1. LongBench Macro-F1 (%). Column headings give nominal prefix budgets. For LoSA, parentheses give measured prefix KV counts, which can exceed those budgets. Higher is better.

ModelMethodBudgetGSM8KMMLUHumanEvalMATH-500
LLaDA2.1-miniDense–85.8267.7978.6691.80
LoSA25684.61 (216)66.99 (873)70.73 (417)88.80 (512)
FOCUS–86.0566.7679.8892.60
Duet (Ours)25684.3166.9080.4987.60
SDAR-8BDense–88.2574.5261.5963.60
LoSA25687.11 (188)74.55 (458)54.88 (332)58.40 (294)
FOCUS–86.8874.5264.6365.80
Duet (Ours)25687.6474.5362.2063.60

Table 2. Generation quality on short-context benchmarks. We report accuracy (%) on GSM8K, MMLU, and MATH-500, and pass@1 (%) on HumanEval. For LoSA, parentheses give the average number of selected KV tokens per task. Higher is better.

Duet also maintains comparable quality on shorter-context benchmarks. Across GSM8K, MMLU, HumanEval, and MATH-500, both LLaDA2.1-mini and SDAR-8B closely track their dense counterparts, with LLaDA2.1-mini additionally outperforming Dense on HumanEval. These results show that Duet preserves generation quality across diverse tasks while substantially reducing inference cost, as evaluated next.

Inference efficiency. As shown in Figure 4 and Table 3, Duet provides increasingly larger end-to-end gains as the context grows. On LLaDA2.1-mini, Duet with a prefix budget of 256 achieves a 1.12×1.12 \times speedup at 16K and a 1.34×1.34 \times speedup at 32K, while FOCUS provides only a 1.04×1.04 \times speedup and LoSA remains slower than dense inference at 32K. The same trend is more pronounced on SDAR-8B: Duet reaches 1.53×1.53 \times, 2.05×2.05 \times, and 2.41×2.41 \times end-to-end speedups at 8K, 16K, and 32K, respectively. These results show that Duet remains effective as the historical prefix grows, where repeatedly attending to dense historical KV states becomes increasingly expensive.

Figure 4. Inference speedup across context lengths. End-to-end and stable-step speedups over dense inference on LLaDA2.1-mini and SDAR-8B at 8K, 16K, and 32K context lengths. Stable-step latency averages all denoising steps except the first step, highlighting the execution efficiency of sparse decoding. Duet achieves increasingly larger speedups as the context grows and consistently provides the strongest stable-step acceleration among the compared methods.

MethodBudget8K context E2E (s)8K context Step (ms)8K context Steps16K context E2E (s)16K context Step (ms)16K context Steps32K context E2E (s)32K context Step (ms)32K context Steps
LLaDA2.1-mini
Dense–0.84124.0316.061.44735.4716.003.48658.1225.05
FOCUS–1.07220.6524.051.46220.5825.153.36121.2364.53
LoSA2561.14832.2516.651.76737.7816.183.67240.5427.81
5121.17133.1216.591.77438.0016.183.67640.8928.06
10241.18834.2516.511.78939.0916.103.71240.7128.81
Duet(Ours)2560.86020.2615.601.29520.3215.882.60420.5523.29
5120.84019.5415.581.29220.2215.942.62020.5724.08
10240.87120.2415.481.30320.3615.702.63320.4624.16
SDAR-8B
Dense–5.35035.421289.27956.8812818.41699.53128
FOCUS–4.06225.801284.93425.181288.04625.80128
LoSA2564.99233.301286.36036.051289.92040.26128
5125.25235.371286.61638.3312810.22742.46128
10245.68438.761287.05041.7712810.79846.64128
Duet(Ours)2563.50821.121284.52321.121287.65621.09128
5123.52421.211284.56321.461287.62420.84128
10243.52621.051284.58820.961288.10020.58128

Table 3. Detailed inference efficiency results. E2E is end-to-end latency in seconds; Step is stable-step latency in milliseconds; Steps is the mean number of decoding steps per sample. Lower latency is better.

The stable-step results in Table 3 further isolate the computation saved during sparse decoding. At 32K, Duet reduces the stable-step latency from 58.12 ms to 20.55 ms on LLaDA2.1-mini, corresponding to a 2.83×2.83 \times speedup, and from 99.53 ms to 21.09 ms on SDAR-8B, corresponding to a 4.72×4.72 \times speedup. In comparison, LoSA reaches only 1.43×1.43 \times and 2.47×2.47 \times speedups on the two models at 32K, respectively, indicating that its prefix-selection overhead substantially limits the benefit of sparse attention. FOCUS achieves stable-step speedups closer to Duet—2.74×2.74 \times on LLaDA2.1-mini and 3.86×3.86 \times on SDAR-8B at 32K—but this advantage translates less effectively to end-to-end generation. FOCUS changes the decoding trajectory and requires more denoising steps to generate the same number of output tokens; we quantify this effect in Table 3. As a result, its end-to-end speedup is lower than what its per-step latency alone would suggest.

Analysis

We analyze stability of prefix sparsity and early predictability of query positions on LLaDA2.1-mini.

Stability of prefix sparsity. We examine whether a compact prefix subset selected at the beginning of a generation block remains representative throughout the subsequent denoising process. Take LLaDA2.1-mini as the For each block, we keep the selected historical prefix subset fixed and measure the fraction of prefix-attention mass covered by this subset as the block is progressively resolved. Figure 5a shows that even a prefix budget of 256 consistently captures around 85% of the historical attention mass, while increasing the budget to 512 raises the coverage to nearly 90%. More importantly, the covered attention mass remains stable throughout block decoding: across all evaluated budgets, the variation over the denoising trajectory is within approximately 2.5 percentage points. These results indicate that the current-block queries maintain a largely stable focus on a compact portion of the historical prefix, supporting our design of selecting prefix KV states once before decoding each block and reusing the resulting cache throughout its denoising steps.

Figure 5. Two sparsity opportunities in BDLM inference. (a) A compact prefix subset selected at the beginning of a generation block continues to cover most of the prefix-attention mass throughout denoising. (b) Intermediate representations can identify, with high recall, the positions selected by the final-layer decoding policy. Results are shown on LLaDA2.1-mini as a representative example.

Early predictability of query positions. We analyze whether the positions selected by the final decoding update can already be predicted from early Transformer representations. At each denoising step, we apply the model’s original decoding policy to hidden states from different layers and measure the recall of the positions selected by the final-layer decoding result. Figure 5b shows that the final update positions become highly predictable at surprisingly early layers. At layer 1, retaining 70% of the masked positions already recovers approximately 80% of the final update positions, while a 90% retention ratio reaches about 95% recall. The recall remains high through most of the network and approaches 100% near the final layers. These results demonstrate that the locations of the final decoding updates can be identified well before the full Transformer depth is evaluated, providing direct support for selecting active query rows at an early layer and avoiding unnecessary deep computation on the remaining positions.

Related Work

DLMs. DLMs generate text by iteratively denoising discrete token sequences. Prior work explored discrete-state diffusion, continuous diffusion in embedding space, and diffusion with pretrained masked language models [2, 13, 9]. Masked discrete diffusion provided a scalable formulation, later demonstrated at scale by LLaDA and Dream [19, 18, 24]. Fully parallel masked DLMs predict multiple positions per step, enabling bidirectional context modeling and parallel generation. However, repeatedly processing the full sequence incurs redundant computation on largely unchanged tokens, while bidirectional attention prevents direct reuse of the append-only KV cache used by autoregressive models [22, 15].

From DLMs to BDLMs. Fast-dLLM demonstrates the potential of block-level organization for DLM acceleration through block-wise approximate KV caching and confidence-aware parallel decoding [22]. Subsequent methods explore adaptive cache reuse at different granularities, including segment-level updates, delayed updates, and token-level state selection [15, 17, 12]. BDLMs make this structure part of the generation paradigm: Block Diffusion generates blocks autoregressively while refining tokens through diffusion within each block [1]; SDAR adapts pretrained autoregressive models to block-wise diffusion [7]; and LLaDA2.0 and LLaDA2.1 extend the scale and token-editing capabilities of this model family [4, 5]. Fast-dLLM v2 further combines block-diffusion model design with hierarchical caching and parallel decoding [21].

Inference optimization for existing BDLMs. Recent work optimizes existing BDLMs, including SDAR and LLaDA2.x, by reducing current-block computation or historical-prefix attention [14, 23]. FOCUS estimates token-wise decodability to skip positions unlikely to be updated, but under confidence-based decoding this may increase the number of denoising steps, limiting end-to-end gains [14]. LoSA sparsifies long-context prefix attention by recomputing attention for active queries and reusing outputs for stable ones, while repeated query-dependent KV selection still incurs nontrivial overhead [23]. As a result, the reduction in attention computation does not always translate into substantial end-to-end speedup.

Conclusion

We presented Duet, a training-free dual-sparse framework for BDLM inference that combines block-wise prefix KV selection with early query selection to avoid repeated historical attention and deep processing of inactive positions. The selected prefix is reused across denoising steps, while only predicted active queries traverse the remaining Transformer layers. Duet maintains competitive accuracy across long-context and standard benchmarks, achieving 1.34×/2.83×1.34 \times/2.83 \times end-to-end/per-step speedups on LLaDA2.1-mini and 2.41×/4.72×2.41 \times/4.72 \times on SDAR-8B-Chat-b32 at 32K. Stable prefix relevance within blocks and early predictability of update positions support both sparsity mechanisms.z

Acknowledgments

This work is supported by Jiangsu Science and Technology Major Project (#BG2025005), the Collaborative Innovation Center of Novel Software Technology and Industrialization, and NanJing Kunpeng&Ascend Center of Cultivation. Jingwei Xu ([email protected]) is the corresponding author.

AI use statement

The core idea and algorithmic design of Duet were developed by the authors. We used generative AI tools primarily to modify and optimize portions of the implementation and evaluation code, monitor experimental runs and inspect logs, and edit the manuscript for clarity and presentation. The tools also assisted in reviewing the presentation of mathematical claims and interpreting benchmark results; they did not originate the core method. The authors reviewed the AI-assisted work and take full responsibility for the final code, results, and claims in this paper.

Reproducibility statement

We release the anonymized source code and testing procedures for Duet at the repository linked in § 4. The release includes the implementation and evaluation scripts for the short- and long-context benchmarks and inference-speed measurements. § 3 specifies the inference algorithm, while § 4 describes the models, decoding settings, baselines, benchmarks, and evaluation protocol. The appendix provides task-wise LongBench results together with detailed latency and decoding-step measurements. Together, these materials support reproduction of the reported quality and efficiency comparisons.

Efficiency Evaluations

We evaluate inference efficiency on 80 NarrativeQA samples at 8K, 16K, and 32K context lengths, generating 256 tokens per sample on an NVIDIA H800 GPU. Table 3 reports three complementary measurements: mean end-to-end (E2E) generation latency, stable-step latency excluding the first denoising step, and the mean number of decoding steps per sample. End-to-end latency includes prefill and generation overheads, so its speedup need not equal the stable-step speedup.

At 32K context length, Duet with a prefix budget of 256 reduces LLaDA2.1-mini’s stable-step latency from 58.12 to 20.55 ms and its end-to-end latency from 3.486 to 2.604 s. Its mean step count decreases from 25.05 to 23.29. FOCUS reaches a similar stable-step latency of 21.23 ms but takes 64.53 steps on average, partly explaining its smaller end-to-end gain. SDAR-8B uses its default sequential decoding policy, which fixes the number of decoding steps for a given generation length. Thus, all methods take 128 steps in this experiment. On SDAR-8B, Duet reduces stable-step latency from 99.53 to 21.09 ms and end-to-end latency from 18.416 to 7.656 s at 32K.

Detailed LongBench Results

Table 4 reports the full task-wise F1 results underlying the LongBench Macro-F1 values in Figure 3. Each task contributes equally to Macro-F1. LoSA’s parenthesized values are measured averages of attended prefix KV tokens, not its configured budget; the parenthesized value in the Macro-F1 column is the mean of the five task-level KV counts.

MethodBudgetHotpotQATriviaQANarrativeQAQasperMultiFieldQAMacro-F1
LLaDA2.1-mini
Dense–54.6085.9124.7142.1549.2651.33
FOCUS–55.2384.7421.9641.8947.0350.17
LoSA25654.74 (430)85.51 (426)23.87 (447)43.20 (435)48.90 (433)51.24 (434)
LoSA51256.13 (833)85.13 (820)23.23 (873)43.32 (813)49.60 (825)51.48 (833)
LoSA102455.60 (1594)85.86 (1549)23.02 (1677)42.80 (1483)49.15 (1529)51.29 (1566)
Duet (Ours)25654.0284.4024.6943.2749.2951.13
Duet (Ours)51255.3685.1124.9842.2949.0151.35
Duet (Ours)102455.1385.4523.9742.9049.2551.34
SDAR-8B
Dense–52.1183.1525.0334.8553.2949.69
FOCUS–51.5583.6524.7834.9353.3449.65
LoSA25651.96 (412)83.12 (405)25.25 (398)34.38 (403)52.56 (406)49.45 (405)
LoSA51251.92 (809)83.12 (784)25.31 (804)34.49 (785)53.42 (786)49.65 (794)
LoSA102451.93 (1546)83.20 (1490)25.37 (1571)34.47 (1461)53.66 (1467)49.73 (1507)
Duet (Ours)25650.3483.3724.9134.3750.8548.77
Duet (Ours)51251.1183.1224.7634.7452.0149.15
Duet (Ours)102450.8782.8725.8734.7851.8649.25

Table 4. Task-wise results on LongBench. We report F1 (%) on five long-context tasks and Macro-F1, the unweighted average across tasks. For LoSA, parentheses give the measured average number of attended prefix KV tokens per task; the Macro-F1 column gives the mean of these five KV counts. Higher F1 is better.

References

  1. [1]Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models. In International Conference on Learning Representations, 2025. URL https://mlanthology.org/iclr/2025/arriola2025iclr-block/.
  2. [2]Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393.arxiv.org/abs/2107.03006
  3. [3]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.172. URL https://aclanthology.org/2024.acl-long.172.
  4. [4]Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, Chengxi Li, Chongxuan Li, Jianguo Li, Zehuan Li, Huabin Liu, Lin Liu, Guoshan Lu, Xiaocheng Lu, Yuxin Ma, Jianfeng Tan, Lanning Wei, Ji-Rong Wen, Yipeng Xing, Xiaolu Zhang, Junbo Zhao, Da Zheng, Jun Zhou, Junlin Zhou, Zhanchao Zhou, Liwang Zhu, and Yihong Zhuang. Llada2.0: Scaling up diffusion language models to 100b, 2025. URL https://arxiv.org/abs/2512.15745.
  5. [5]Tiwei Bie, Maosong Cao, Xiang Cao, Bingsen Chen, Fuyuan Chen, Kun Chen, Lun Du, Daozhuo Feng, Haibo Feng, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jian Guan, Kaiyuan Guan, Hongliang He, Zenan Huang, Juyong Jiang, Zhonghui Jiang, Zhenzhong Lan, Chengxi Li, Jianguo Li, Zehuan Li, Huabin Liu, Lin Liu, Guoshan Lu, Yuan Lu, Yuxin Ma, Xingyu Mou, Zhenxuan Pan, Kaida Qiu, Yuji Ren, Jianfeng Tan, Yiding Tian, Zian Wang, Lanning Wei, Tao Wu, Yipeng Xing, Wentao Ye, Liangyu Zha, Tianze Zhang, Xiaolu Zhang, Junbo Zhao, Da Zheng, Hao Zhong, Wanli Zhong, Jun Zhou, Junlin Zhou, Liwang Zhu, Muzhi Zhu, and Yihong Zhuang. Llada2.1: Speeding up text diffusion via token editing, 2026. URL https://arxiv.org/abs/2602.08676.
  6. [6]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Pappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.arxiv.org/abs/2107.03374
  7. [7]Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, et al. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation. arXiv preprint arXiv:2510.06303, 2025.arxiv.org/abs/2510.06303
  8. [8]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  9. [9]Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. DiffusionBERT: Improving generative masked language models with diffusion models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4521–4534, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.248. URL https://aclanthology.org/2023.acl-long.248/.
  10. [10]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
  11. [11]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
  12. [12]Yuchu Jiang, Yue Cai, Xiangzhong Luo, Jiale Fu, Jiarui Wang, CHONGHAN LIU, and xu yang. d²cache: Accelerating diffusion-based llms via dual adaptive caching. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors, International Conference on Learning Representations, volume 2026, pages 137511–137531, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/file/de4f2e3b8263aff39f0dd7571a8dc4b1-Paper-Conference.pdf.
  13. [13]Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-lm improves controllable text generation. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.arxiv.org/abs/2205.14217
  14. [14]Kaihua Liang, Xin Tan, An Zhong, Hong Xu, and Marco Canini. FOCUS: DLLMs know how to tame their compute bound. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=40fUEdvvH3.arxiv.org/abs/2601.23278
  15. [15]Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zhou, Qingyan Wei, Shaobo Wang, Yichen Zhu, and Linfeng Zhang. dLLM-cache: Accelerating diffusion large language models with adaptive caching, 2026. URL https://openreview.net/forum?id=DriG3hgh42.arxiv.org/abs/2506.06295
  16. [16]Lingkun Long, Yushi Huang, Shihao Bai, Ruihao Gong, Jun Zhang, Ao Zhou, and Jianlei Yang. Focus-dLLM: Accelerating long-context diffusion LLM inference via confidence-guided context focusing. In Maria Liakata, Vivian R. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14216–12160, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.556. URL https://aclanthology.org/2026.acl-long.556/.DOI
  17. [17]Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 149009–149033. Curran Associates, Inc., 2025. doi: 10.52202/085713-4984. URL https://proceedings.neurips.cc/paper_files/paper/2025/file/db0ee27cb50dd9087b133f6e7d28a90e-Paper-Conference.pdf.DOI
  18. [18]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Juh Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models. In Advances in Neural Information Processing Systems, 2025. URL https://mlanthology.org/neurips/2025/nie2025neurips-large/.arxiv.org/abs/2502.09992
  19. [19]Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9798331314385.arxiv.org/abs/2406.07524
  20. [20]Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, Kai Yu, and Zhijie Deng. Diffusion LLMs can do faster-than-AR inference via discrete diffusion forcing. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=t5uLZSrJgHF.arxiv.org/abs/2508.09192
  21. [21]Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, and Enze Xie. Fast-dllm v2: Efficient block-diffusion llm. In C. Vondrick, B. Hariharan, C. Rafel, L. Pinto, D. Yang, and A. Faust, editors, International Conference on Learning Representations, volume 2026, pages 128353–128370, 2026a. URL https://proceedings.iclr.cc/paper_files/paper/2026/file/d0865ceb51d35ec322f9af9db7806fcf7-Paper-Conference.pdf.arxiv.org/abs/2509.26328
  22. [22]Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dLLM: Training-free acceleration of diffusion LLM by enabling KV cache and parallel decoding. In The Fourteenth International Conference on Learning Representations, 2026b. URL https://openreview.net/forum?id=Z3Is6hnOT.arxiv.org/abs/2505.22618
  23. [23]Haocheng Xi, Harman Singh, Yuezhou Hu, Coleman Richard Charles Hooper, Rishabh Tiwari, Aditya Tomar, Minjae Lee, Wonjun Kang, Michael W Mahoney, Chenfeng Xu, et al. Losa: Locality aware sparse attention in diffusion language models. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=SMhHOupeRe.arxiv.org/abs/2604.12056
  24. [24]Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025.arxiv.org/abs/2508.15487

Paper details

vlnc:W_ktchrvjg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents