betaFeedback

WorksComputer Science

SemMask: Semantic-Prioritized Progressive Masking for Diffusion Language Models

Abstract

Diffusion Language Models (DLMs) are designed to enable globally parallel generation. However, in practice, they often exhibit pseudo-autoregressive behavior and a function-word-first bias, which undermines their performance on tasks such as text writing and code generation that benefit from global planning and parallel decoding. To address this, we propose SemMask, a novel fine-tuning paradigm that leverages a semantic-prioritized progressive masking strategy to encourage the model to decode high-information semantic tokens early and globally, thereby enhancing its global planning and parallel generation capabilities. Specifically, we first use the gradients of the original DLM with respect to each token to identify semantic tokens. During masking, these semantic tokens are preserved with priority, establishing a progressive generation order in which semantic tokens are decoded prior to function tokens. Meanwhile, we adjust the loss weight for semantic tokens to further reinforce their early prediction. Experimental results show that SemMask achieves quality comparable to strong autoregressive baselines on both text writing and code generation tasks, while providing superior semantic coherence. Notably, SemMask achieves higher accuracy with substantially fewer decoding steps, indicating that the early establishment of semantic structure improves the efficiency–accuracy trade-off.

Introduction

In recent years, Diffusion Language Models (DLMs) have emerged as a promising alternative to traditional autoregressive (AR) models, enabling parallel and efficient text generation [3, 14, 46] (Li et al. 2022). Unlike AR models that generate tokens sequentially from left to right, DLMs iteratively generate all tokens in parallel, enabling more global generation planning [34] (Ye et al. 2025).

However, a significant gap remains between the architectural potential of DLMs and their practical decoding behavior [9, 12] (Xu et al. 2025). While DLMs are designed to exploit bidirectional context and decode tokens in parallel, they exhibit two systematic decoding phenomena in practice: pseudo-autoregressive behavior and a function-word-first bias. As shown in Figure 1, under the default pure diffusion sampling strategy, DLMs exhibit a pseudo-autoregressive decoding behavior. This behavior compromises content completeness, resulting in fragmented and structurally disorganized generation. To empirically examine the function-word-first bias, we conduct a series of controlled experiments, with the results shown in Figure 2. As shown in Figure 2(c), the bottom 10% gradient-score tokens, shown in blue, are consistently decoded at earlier diffusion steps than the top 10% tokens, shown in orange. Figure 2(a) and (Figure 2b) show that the top 10% gradient-score tokens are primarily semantic tokens, such as nouns and verbs, whereas the bottom 10% are mostly function tokens, including pronouns, prepositions, and articles. These observations indicate a clear decoding bias toward decoding low-information function tokens earlier while delaying high-information semantic tokens.

Comparison of default diffusion decoding and SemMask decoding

Figure 1. Default diffusion decoding exhibits pseudo-autoregressive behavior, resulting in incomplete and structurally disorganized content, whereas SemMask prioritizes semantically important tokens to establish a global semantic scaffold and generate more coherent and complete text.

Figure 2. Analysis of the function-word-first bias on HANNA using LLaDA-8B-Instruct, with both the generation length and the number of sampling steps set to 3,000.

These phenomena arise from the confidence-guided remasking strategy employed during inference, which encourages DLMs to decode higher-confidence tokens while remasking lower-confidence ones [34, 22]. For DLMs trained with a random masking strategy, these higher-confidence tokens are often either early tokens in the sequence or function words, leading to a pseudo-autoregressive, function-word-first generation order. This generation order undermines the inherent global planning advantage of DLMs, resulting in suboptimal performance on tasks that rely on such global planning, such as text writing and code generation.

To address these issues, we propose SemMask, a novel fine-tuning paradigm designed to encourage the model to decode high-information semantic tokens early and globally, thereby activating its global planning and parallel generation capabilities. As illustrated in Figure 1 and Figure 2(d), our approach dissolves the pseudo-autoregressive behavior and function-word-first bias observed in the original model. To achieve this, we focus on solving two core problems: (1) how to identify the high-information semantic tokens in a text; and (2) how to guide the model to perform semantic-prioritized generation. For the first problem, we design a Gradient-Based Semantic Token Scoring strategy, which utilizes the gradients of the original DLM with respect to each token to identify semantic tokens. For the second problem, we design a Semantic Anchor Preservation Masking strategy for mask generation and a Semantic-Prioritized Loss Re-Weighting strategy for objective adjustment. During training, the identified semantic tokens are preserved with priority, establishing a progressive generation order in which semantic tokens are decoded prior to function tokens. During objective adjustment, the weights of semantic tokens in the final loss are also adjusted to further reinforce their early prediction.

To verify the effectiveness of our proposed SemMask, we conduct extensive experiments on both text writing and code generation benchmarks. The experimental results show that SemMask achieves performance comparable to strong autoregressive baselines on both tasks, while providing superior semantic coherence and inference efficiency. Notably, SemMask maintains strong performance with substantially fewer decoding steps, confirming that the early establishment of a semantic scaffold restores genuine parallel decoding capability.

Our contributions are summarized as follows:

  • We propose a novel training paradigm that strengthens the DLM’s ability to predict semantic tokens early in the generation process, thereby activating its global planning and parallel generation capabilities.

  • We introduce Gradient-Based Semantic Token Scoring, where loss gradients are used to quantify token importance, enabling the model to prioritize hard-to-learn semantic tokens over function tokens.

  • We introduce the Semantic Anchor Preservation Masking, a sparsity-constrained masking strategy that preserves the spatial integrity of semantic tokens during high-noise states. This mechanism enforces a global generation approach by ensuring that the global structure of the sequence is reconstructed before local syntactic details are filled in.

Diffusion Language Models

Diffusion language models (DLMs) have recently emerged as a promising alternative to traditional autoregressive models for text generation. While early approaches ([10, 13] (Li et al. 2022)) focused on exploring continuous embedding spaces, the field has increasingly shifted towards Masked Diffusion Models (MDMs), which directly model language in the categorical state space by predicting masked tokens through iterative denoising, enabling fast and parallel decoding ([3, 35, 18]). Building upon this discrete formulation, recent advancements have demonstrated that DLMs can rival autoregressive baselines at scale. A notable example is LLaDA, trained from scratch, which achieves competitive performance against strong baselines such as LLaMA 3-8B ([34]). Similarly, Dream presents a family of scalable diffusion models that match state-of-the-art autoregressive performance across general, mathematical, and coding tasks (Ye et al. 2025). Beyond general-purpose generation, recent research has expanded DLMs into various multimodal domains ([42, 44] (Li et al. 2026)) and specialized reasoning tasks, including code generation and mathematical inference ([21] (Xie et al. 2025)).

Despite these rapid advances, the prevailing paradigm in discrete DLMs remains uniform corruption, where tokens are masked uniformly at random. This agnostic masking policy treats all tokens identically, overlooking the hierarchical information density of natural language (Chao et al. 2026; Yu, Li, and Wang 2025), which often causes models to struggle with capturing global semantic dependencies in the early stages of generation. Furthermore, this homogeneous nature fails to provide explicit guidance on the optimal generation order, rendering naive decoding strategies highly sensitive to local context [19], thereby limiting the practical realization of true non-autoregressive modeling.

Non-Autoregressive Generation and Token Ordering

Non-Autoregressive generation aims to circumvent the high inference latency inherent in Autoregressive models by decoding multiple tokens in parallel. Seminal works like Insertion Transformers [37] and FELIX [33] explored permutation and dynamic insertion to break rigid sequential dependencies. In the context of MDMs, the noise schedule effectively serves as an implicit generation order. However, despite the theoretical capability for global parallel planning, position bias causes tokens adjacent to the left context to exhibit lower entropy and higher confidence, leading to a left-to-right convergence pattern [17, 7]. Consequently, early prediction errors propagate through subsequent tokens, often resulting in semantic drift and reduced coherence in longer sequences, akin to the error accumulation commonly observed in autoregressive decoding [32, 9, 25].

To restore true parallel reasoning, recent research has focused on optimizing token ordering through hybrid architectures, trajectory alignment, and adaptive inference. Structurally, methods like Block Diffusion [2], Seed Diffusion [36], and ReFusion (Li et al. 2025a) introduce semi-autoregressive block or slot-based mechanisms to enforce coarse-grained consistency. Beyond architectural constraints, several works optimize the generation trajectory directly during training: dParallel [7] employs distillation to enforce deterministic convergence, while DiffuCoder [15] and Trac-eRL [38] align training objectives with inference paths via reinforcement learning. Additionally, approaches like Prophet (Li et al. 2025b) and KLASS [24] exploit confidence gaps and stability metrics for early exit, whereas Dilated Scheduling [31] and adaptive sub-problem selection [23] optimize unmasking sequences to mitigate entropy accumulation. However, these approaches largely remain within the paradigm of parallelization of independent sampling rather than parallelization of semantic association. Therefore, our approach internalizes semantic priority decisions within the model, guiding it to first establish a global semantic scaffold by predicting semantic tokens with high information content across the sequence and then progressively fill in the remaining local function tokens.

Methodology

Our approach, SemMask, consists of three stages: (a) Gradient-Based Semantic Token Scoring estimates the semantic importance of each token, (b) Semantic Anchor Preservation Masking ranks tokens by their importance scores and progressively preserves them according to these scores, and (c) Semantic-Prioritized Loss Re-Weighting assigns greater optimization weights to tokens with higher importance scores to encourage their early reconstruction. Together, these components guide the model to first establish a global semantic scaffold and then progressively complete the remaining tokens, as illustrated in Figure 3.

Figure 3. Overview of the SemMask framework. (a) Gradient-Based Semantic Token Scoring estimates the semantic importance of each token. (b) Semantic Anchor Preservation Masking ranks tokens by their importance scores and progressively preserves high-scoring tokens as semantic anchors. (c) Semantic-Prioritized Loss Re-Weighting assigns larger optimization weights to more important tokens to encourage their early reconstruction.

Algorithm 1: Semantic Anchor Preservation Masking

Input: Clean sequence x0x_0, semantic scores SS, masking ratio tt Parameter: Anchor distance δ\delta, perturbation strength λ\lambda Output: Partially masked sequence xtx_t

  1. Initialize anchor set A←∅A \leftarrow\varnothing and forbidden set F←∅F \leftarrow\varnothing

  2. Set the target anchor count K←⌊L(1−t)⌋K \leftarrow\lfloor L(1-t)\rfloor

  3. Set the anchor distance δt←⌊δt⌋\delta_t \leftarrow\lfloor\delta t\rfloor

  4. for i=1i = 1 to LL do Sample zi∼Gumbel⁡(0,1)z_i \sim\operatorname{Gumbel}(0,1) Compute qi←Si+λziq_i \leftarrow S_i + \lambda z_i end for

  5. Let OO be the indices sorted by qiq_i in descending order

  6. for ii in OO do

  7. if ∣A∣<K\lvert A\rvert< K and i∉Fi \notin F then

  8. A←A∪{i}A \leftarrow A \cup\{i\}

  9. F←F∪{j∈{1,…,L}:∣j−i∣<δt}F \leftarrow F \cup\{j \in\{1,\ldots,L\}: \lvert j-i\rvert< \delta_t\}

  10. end if

  11. end for

  12. for i=1i = 1 to LL do

  13. xti←x0ix_t^i \leftarrow x_0^i if i∈Ai \in A, otherwise xti←[M]x_t^i \leftarrow[\mathrm{M}]

  14. end for

  15. return xtx_t

Gradient-Based Semantic Token Scoring

Default masking strategies fail to account for the varying importance of tokens in text. To address this limitation, we assume that the semantic importance of a token is reflected in the sensitivity of the model’s predictions to changes in that token. Accordingly, we use the magnitude of its gradient to quantify token importance.

Given a text sequence x=(x1,…,xn)x = (x_1,\ldots,x_n), we use the frozen parameters of the DLM to compute the gradient gig_i for each token xix_i. To ensure a consistent scale for token importance scores across different text sequences, we normalize the raw scores to obtain the final score SiS_i:

Si=gi−min⁡(G)max⁡(G)−min⁡(G)+ϵ(1)S_i = \frac{g_i-\min(\mathcal{G})}{\max(\mathcal{G})-\min(\mathcal{G})+\epsilon} \tag*{(1)}

where G={gi}i=1n\mathcal{G} = \{g_i\}_{i=1}^{n} denotes the set of gradient values for all tokens in the sequence xx, min⁡(G)\min(\mathcal{G}) and max⁡(G)\max(\mathcal{G}) denote the minimum and maximum values in G\mathcal{G}, respectively, and ϵ\epsilon is a small constant introduced for numerical stability.

Semantic Anchor Preservation Masking

Based on the token importance scores computed in the previous section, we use these scores to guide token masking during training, prioritizing tokens with higher scores as semantic tokens.

As shown in Algorithm 1, we first rank all tokens according to their importance scores and retain the highest scoring tokens as semantic anchors in the masked sequence, while the remaining tokens stay masked. Given the normalized semantic scores S=(S1,…,Sn)S = (S_1,\ldots,S_n), we select the semantic anchors to be retained by maximizing the following objective:

A∗=arg max⁡A⊆{1,…,L}∑i∈A(Si+λzi),zi∼Gumbel⁡(0,1)(2)A^* = \underset{A\subseteq\{1,\ldots,L\}}{\operatorname{arg\,max}}\sum_{i\in A}(S_i+\lambda z_i), \qquad z_i\sim\operatorname{Gumbel}(0,1) \tag*{(2)}

where A∗A^* is the optimal anchor set, LL is the sequence length, ziz_i introduces stochasticity, and λ\lambda controls the perturbation strength to balance semantic preference and anchor-selection diversity.

However, relying solely on importance scores for anchoring may lead to anchor clustering and insufficient contextual coverage, thereby increasing the difficulty of solving the generation task. Therefore, during the stages of training, we impose the constraints defined in Eq. (3) to ensure sufficient contextual coverage.

{∣A∣=⌊L(1−t)⌋,Anchor Cardinality∣i−j∣≥δt,∀i,j∈A, i≠j,Sequence-Space Coverage(3)\left\{ \begin{aligned} \lvert A\rvert&= \lfloor L(1-t)\rfloor, && \text{Anchor Cardinality}\\ \lvert i-j\rvert&\ge\delta_t,\quad\forall i,j\in A,\ i\ne j, && \text{Sequence-Space Coverage} \end{aligned} \right. \tag*{(3)}

where ii and jj denote anchor positions in the text sequence, t∼U(0,1)t\sim\mathcal{U}(0,1) denotes the masking ratio independently sampled for each training example and applied independently to all tokens, and δt\delta_t specifies the minimum distance between anchors. As t→1t\to1, the spatial constraint encourages the retained tokens to form a global semantic scaffold. As t→0t\to0, δt\delta_t decays to zero, allowing the remaining tokens to be progressively filled into the established global semantic scaffold.

Semantic-Prioritized Loss Re-Weighting

We propose a re-weighted loss function that adjusts the learning signal according to semantic scores during training, encouraging the model to prioritize the prediction of semantic tokens with higher scores.

As shown in Eq. (4), we introduce Ωi\Omega_i into the loss function by mapping each token’s semantic score to its corresponding importance weight. Here, α\alpha controls the overall weighting magnitude, while β\beta determines the emphasis placed on semantic tokens.

Ωi=1+α⋅exp⁡(βSi)(4)\Omega_i = 1+\alpha\cdot\exp(\beta S_i) \tag*{(4)}

The resulting loss function is defined as:

L(θ)=−Et,x0,xt[1t⋅ZΩ∑i=1LI[xti=[M]]⋅Ωi⋅ℓi(θ)](5)\mathcal{L}(\theta)=-\mathbb{E}_{t,x_0,x_t}\left[\frac{1}{t\cdot\mathcal{Z}_{\Omega}}\sum_{i=1}^{L}\mathbb{I}[x_t^i=[\mathrm{M}]]\cdot\Omega_i\cdot\ell^i(\theta)\right] \tag*{(5)}

where ℓi(θ)=log⁡pθ(x0i∣prompt⁡,xt)\ell^i(\theta)=\log p_{\theta}(x_0^i\mid\operatorname{prompt},x_t) denotes the token-level log-likelihood, I[⋅]\mathbb{I}[\cdot] is the mask indicator, and ZΩ=∑i∈maskedΩi\mathcal{Z}_{\Omega}=\sum_{i\in\text{masked}}\Omega_i normalizes the weighted objective. Accordingly, this training objective reinforces the progressive generation order induced by the masking strategy.

Experiments

Experimental Setups

Datasets. We constructed two high-quality training datasets tailored for text writing and code generation tasks. For the text writing task, we sourced data from Writing-Prompts [11] and applied GPT-5.1-based quality filtering, ultimately obtaining a subset of

10,000 samples. For code generation task, we applied the same filtering strategy to OpenCodeInstruct [1], resulting in 10,000 high-quality training samples.

Baselines. We benchmarked our approach against Llama-3-8B-Instruct [16] and Qwen2.5-7B-Instruct [41] as state-of-the-art autoregressive references. For diffusion baselines, we evaluated LLaDA-8B-Base, LLaDA-8B-Instruct, Dream-7B-Base, and Dream-7B-Instruct. We also evaluated two model variants, LLaDA-8B-SFT and Dream-7B-SFT, obtained by directly fine-tuning their respective instruction-tuned models on our curated datasets using LoRA (Hu et al. 2022).

Benchmarks and Metrics

We evaluate model performance on both tasks using multiple evaluation metrics. For the text writing task, we evaluate the model on two benchmarks: HANNA [8], which focuses on narrative continuation, and CommonGen [30], which focuses on keyword-constrained generation. We used BLEU and ROUGE-L to measure lexical overlap and surface similarity, BERT-Score and BLEURT to assess semantic consistency, and perplexity and GPT-5.1-Score to evaluate overall linguistic quality. Additionally, we introduce the GOD metric [17] to evaluate the model’s non-autoregressive generation behavior. For the code generation task, we used HumanEval [6] and MBPP [4] to evaluate functional correctness using accuracy, with CodeBLEU and BLEURT assessing structural and semantic similarity.

Main Results

Text Writing

As illustrated in Table 1, SemMask consistently improves semantic coherence across both LLaDA-based and Dream-based architectures. On HANNA, Dream7B-SemMask achieves the strongest overall performance with 0.4613 BLEU and 0.8254 BLEURT, outperforming the autoregressive baselines on most metrics, while Llama-3-8B-Instruct retains the highest BERT-Score of 0.8051. LLaDA-8B-SemMask also substantially improves over LLaDA-8B-SFT in both BLEU and GPT-5.1-Score. On CommonGen, SemMask obtains markedly higher GOD scores and lower perplexity, with Dream-7B-SemMask achieving the best GOD score of 7.4544 and the lowest perplexity of 21.6630. These results indicate that SemMask restores the parallel generation capability of diffusion language models and generates more coherent text.

ModelBLEU↑BLEURT↑ROUGE-L↑BERT-Score↑GPT-5.1-Score↑GOD↑Perplexity↓
Autoregressive Models
Llama-3-8B-Instruct0.31090.72080.33770.80513.0833–24.5731
Qwen2.5-7B-Instruct0.38930.71290.31530.80303.1042–22.9951
Diffusion Language Models
LLaDA-8B-Base0.33740.69360.21010.75411.95831.562449.1982
LLaDA-8B-Instruct0.38830.70140.21250.69322.86464.343033.2763
LLaDA-8B-SFT0.33520.72180.30530.75922.77084.821030.4352
LLaDA-8B-SemMask (Ours)0.39210.73010.31010.77463.01046.318722.5286
Dream-7B-Base0.36400.74720.32150.76642.01041.670440.9942
Dream-7B-Instruct0.40300.68280.37620.74152.91673.886132.6030
Dream-7B-SFT0.40860.73640.39300.75263.02083.900429.1478
Dream-7B-SemMask (Ours)0.46130.82540.41810.77623.12507.454421.6630

Table 1. Main results of text writing evaluation on HANNA and CommonGen. The best results in each category are highlighted in bold, and the second-best results are marked with underlines.

Code Generation. We further evaluate SemMask on code generation to assess its generalization in highly structured domains. As shown in Table 2, Dream-7B-SemMask achieves the best HumanEval accuracy of 0.7085, outperforming Dream-7B-Instruct at 0.5366 and the autoregressive Qwen2.5-7B-Instruct at 0.6182. LLaDA-8B-SemMask likewise improves HumanEval accuracy from 0.4000 to 0.6438 and surpasses Llama-3-8B-Instruct in CodeBLEU, indicating stronger syntactic and structural consistency. Although Qwen2.5-7B-Instruct remains the strongest model on MBPP accuracy, LLaDA-8B-SemMask raises MBPP CodeBLEU from 0.1768 to 0.2303, while Dream-7B-SemMask also improves BLEURT. These results demonstrate that SemMask enhances both functional correctness and structural integrity by better aligning natural-language instructions with executable code.

ModelAccuracy↑CodeBLEU↑BLEURT↑Accuracy↑CodeBLEU↑BLEURT↑
Autoregressive Models
Llama-3-8B-Instruct0.50270.18100.50980.46540.12320.4937
Qwen2.5-7B-Instruct0.61820.28360.41820.64440.25150.4231
Diffusion Language Models
LLaDA-8B-Base0.32120.13230.40630.38910.19960.4300
LLaDA-8B-Instruct0.40000.19440.43000.43630.17680.4489
LLaDA-8B-SFT0.45100.16660.44250.44140.16940.4737
LLaDA-8B-SemMask (Ours)0.64380.26280.44340.45210.23030.4500
Dream-7B-Base0.49630.11720.45170.46320.16510.4184
Dream-7B-Instruct0.53660.21370.42900.51410.18990.4252
Dream-7B-SFT0.62680.15080.49820.54400.15030.4664
Dream-7B-SemMask (Ours)0.70850.24990.51780.55680.24020.4547

Table 2. Main results of code generation evaluation on HumanEval and MBPP. The best results in each category are highlighted in bold, and the second-best results are marked with underlines.

Ablation Study

Validation of Gradient-Based Semantic Token Scoring. To validate gradient norms as a proxy for semantic importance, we compare SemMask with Random, IDF-based, POS-based, and Reverse Gradient strategies. For the POS baseline, we use spaCy to identify semantic tokens, including nouns, verbs, adjectives, and proper nouns, and prioritize their preservation. Reverse Gradient inverts our proposed selection rule by retaining tokens with the lowest gradient norms as semantic anchors. As shown in Table 3, SemMask achieves the best semantic performance on both backbones, increasing BLEU over the POS baseline from 0.3641 to 0.3921 on LLaDA and from 0.4154 to 0.4613 on Dream. Although Random achieves a marginally higher GOD score, its substantially weaker semantic metrics indicate that generation-order deviation alone does not guarantee coherent generation.

Effectiveness of Semantic Anchor Preservation Masking. To evaluate Semantic Anchor Preservation Masking, we compare SemMask with an unconstrained strategy and a static constraint strategy using two fixed anchor distances on CommonGen. The unconstrained strategy yields the weakest overall performance, suggesting that clustered semantic anchors provide insufficient contextual coverage. The two static constraint strategies improve performance, but their fixed anchor distances remain suboptimal across different masking ratios and sequence structures. As shown in Table 4, SemMask achieves the highest GOD and lowest perplexity on both backbones, reaching 6.3187 and 22.5286 on LLaDA

MethodGOD↑BLEU↑BLEURT↑ROUGE-L↑BERT-S↑
LLaDA-8B-Instruct
Random1002.40.36670.70160.21940.7092
IDF994.20.35320.70540.24450.7612
POS997.50.36410.71180.26160.7659
Reverse Gradient985.30.21050.58410.15520.5845
SemMask (Ours)998.90.39210.73010.31010.7746
Dream-7B-Instruct
Random1005.10.39400.69720.33150.7664
IDF996.50.39150.77540.35490.7690
POS998.10.41540.79220.37540.7710
Reverse Gradient988.40.25100.65550.22450.7085
SemMask (Ours)999.80.46130.82540.41810.7762

caption: tbl:3: Ablation study of semantic token scoring strategies on HANNA. Both the generation length and the number of sampling steps are set to 3,000. The best result for each metric under each setting is highlighted in bold.

:::

MethodStrategyGOD↑Perplexity ↓
LLaDA-8B-Instruct
Unconstrained Strategyδ=0\delta= 03.821036.4215
Static Constraint Strategyδ=2\delta= 23.452026.1240
Static Constraint Strategyδ=4\delta= 44.956427.5412
SemMaskδt←⌊δt⌋\delta_t \leftarrow\lfloor\delta_t \rfloor6.318722.5286
Dream-7B-Instruct
Unconstrained Strategyδ=0\delta= 02.900434.1200
Static Constraint Strategyδ=2\delta= 25.150025.1801
Static Constraint Strategyδ=4\delta= 45.281026.9499
SemMaskδt←⌊δt⌋\delta_t \leftarrow\lfloor\delta_t \rfloor7.454421.6630

caption: tbl:4: Ablation study of anchor distance constraint strategies on CommonGen. Both the generation length and the number of sampling steps are set to 30. The best result for each metric under each setting is highlighted in bold.

:::

and 7.4544 and 21.6630 on Dream, respectively. These results indicate that adapting the anchor distance to the masking ratio better balances non-autoregressive generation behavior and linguistic fluency.

Efficiency–Accuracy Trade-off Analysis

To assess inference efficiency, we report the per-sample runtime and average number of decoding steps for each method on HumanEval, using the configuration described in Appendix B.2 and a sequence length of 512. All runtimes are measured on 4 × V100 GPUs. As shown in Table 5, SemMask consistently achieves higher accuracy with fewer decoding steps and shorter inference time across both backbones. Compared with LLaDA-8B-SFT, LLaDA-8B-SemMask reduces the average decoding steps from 214 to 166 and the inference time from 11.12 s to 8.73 s while improving accuracy from 0.4510 to 0.6438; Dream-7B-SemMask similarly achieves an accuracy of 0.7085 using only 169 steps and 7.87 s.

Method#Steps ↓#Time ↓Acc ↑
LLaDA-8B
Base44123.54s0.3212
Instruct32316.49s0.4000
SFT21411.12s0.4510
SemMask (Ours)1668.73s0.6438
Dream-7B
Base43920.08s0.4963
Instruct34015.52s0.5366
SFT2019.51s0.6268
SemMask (Ours)1697.87s0.7085

Table 5. Comparison of generation efficiency and accuracy.

Case Study and Consistency with Human Judgments

Figure 4 presents two representative MBPP cases, where our method, SemMask, preserves critical arithmetic operations and boundary conditions that are incorrectly generated by the Instruct baselines. The visualized decoding orders further indicate that SemMask prioritizes semantically important tokens and establishes coherent program structures earlier, thereby improving functional correctness and semantic coherence. Figure 5 shows a strong correlation between GPT-5.1 scores and human judgments (r=0.9966r = 0.9966), confirming the reliability of the automatic evaluation and the superior performance of SemMask.

Case study on MBPP comparing generated programs and token decoding orders

Figure 4. Case study on MBPP comparing the generated programs and token decoding orders of the Instruct baselines and SemMask.

Automatic evaluation score vs. human rating

Figure 5. Automatic evaluation score vs. human rating.

SemMask substantially reduces both the average number of decoding steps and inference time while achieving the highest accuracy. The best result for each metric within each model family is highlighted in bold.

Conclusion

In this work, we introduce SemMask, a training paradigm designed to enhance the parallel generation capabilities of DLMs by mitigating pseudo-autoregressive behavior. Its Gradient-Based Semantic Token Scoring and Semantic Anchor Preservation Masking encourage important semantic tokens to be generated earlier, establishing a global semantic scaffold for coherent generation. Experiments on text and code generation tasks demonstrate consistent improvements in semantic coherence, generation efficiency, and task performance over standard DLM baselines. These results establish SemMask as an effective and robust approach for scalable non-autoregressive sequence generation.

Appendix Overview

This appendix includes the following sections:

  • Section A introduces the preliminaries of masked diffusion language models, including the forward masking process, training objective, and inference process.

  • Section B provides additional implementation details, including data processing and filtering, training and inference settings, gradient score computation, and the human evaluation protocol for HANNA.

  • Section C presents additional experimental results, including computational overhead analysis, the adaptation of SemMask to continuous diffusion models, and generalization to mathematical reasoning tasks.

  • Section D provides additional case studies illustrating how SemMask prioritizes semantically important tokens and establishes the core semantic structure early in the decoding process.

Preliminaries

Masked Diffusion Language Models. Unlike autoregressive language models that generate tokens in a strict left-to-right order, masked diffusion language models formulate generation as a probabilistic process consisting of a forward masking corruption process and a reverse denoising recovery process. Given a clean sequence x0=(x01,…,x0L)x_0 = (x_0^1, \ldots, x_0^L) and a masking ratio tt, the forward process independently replaces each token with [M][\mathrm{M}] with probability tt:

q(xt∣x0)=∏i=1L[(1−t)δ(xti=x0i)+tδ(xti=[M])](6)q(x_t \mid x_0) = \prod_{i=1}^{L} \left[(1-t)\delta(x_t^i=x_0^i)+t\delta(x_t^i=[\mathrm{M}])\right] \tag*{(6)}

The reverse process is parameterized by a mask predictor pθp_\theta, which takes the corrupted sequence xtx_t as input and jointly predicts the original tokens at all masked positions:

pθ(x0∣xt)=∏i:xti=[M]pθ(x0i∣xt)(7)p_\theta(x_0 \mid x_t) = \prod_{i:x_t^i=[\mathrm{M}]} p_\theta(x_0^i \mid x_t) \tag*{(7)}

The model is optimized using a weighted cross entropy loss restricted to the masked positions:

L(θ)=−Et,x0,xt[1t∑i=1LI[xti=[M]]log⁡pθ(x0i∣prompt,xt)](8)\mathcal{L}(\theta) = -\mathbb{E}_{t,x_0,x_t}\left[\frac{1}{t}\sum_{i=1}^{L}\mathbb{I}[x_t^i=[\mathrm{M}]]\log p_\theta(x_0^i\mid\text{prompt},x_t)\right] \tag*{(8)}

where t∼U(0,1)t \sim U(0,1), x0x_0 is sampled from the training data, and xtx_t is sampled from the forward masking process. During supervised fine tuning, the prompt tokens remain unchanged, while only the response tokens are independently masked and included in the training objective.

Inference Process. The inference process consists of a sequence of denoising steps. At each step, the model predicts all currently masked tokens in parallel, keeps reliable predictions, and remasks uncertain positions for further refinement. This parallel update mechanism allows multiple tokens to be generated simultaneously, offering greater flexibility and potentially higher efficiency than autoregressive decoding.

More Implementation Details

Data Processing and Filtering

Text Writing Training Data.

To construct a high-quality training dataset for text writing, we collected pairs of prompts and stories from the WritingPrompts dataset. To prevent potential data leakage, we used scripts based on predefined rules to remove samples overlapping with the HANNA benchmark. We then applied heuristic filtering to eliminate invalid or mixed-language text, nonstandard special characters, and samples exceeding the length limit of 4,096 tokens. We further employed GPT-5.1 to evaluate grammatical fluency, narrative coherence, logical consistency, content completeness, and alignment between each prompt and its corresponding story, removing samples with awkward expressions, abrupt plot transitions, contradictory events, or incomplete narratives. The resulting dataset contains 10,000 high-quality samples for training for the text writing task.

Code Generation Training Data. For the code generation task, we collected code solution pairs from the OpenCode-Instruct dataset. We removed samples overlapping with the HumanEval and MBPP benchmarks to prevent potential data leakage. We also excluded samples exceeding the length limit of 4,096 tokens and employed GPT-5.1 to evaluate the logical correctness and functional consistency of each code solution. This processing pipeline resulted in a training dataset containing 10,000 code solution pairs.

Training and Inference Settings

As shown in Table 6, we present the training configurations for both backbone models, with a context length of 4,096 tokens and LoRA applied to all linear projection layers. The semantic importance scores are computed using the frozen weights of each backbone, and SemMask uses λ=0.1\lambda= 0.1, δ=4\delta= 4, α=10.0\alpha= 10.0, and β=2.0\beta= 2.0. During inference, we adopt pure diffusion sampling with low-confidence remasking for LLaDA and a maximum threshold of 0.6 for Dream, setting both the generation length and sampling steps to 30 for CommonGen, 3,000 for HANNA, and 512 for HumanEval and MBPP.

Base ModelLoRA RankLoRA AlphaLearning RateLR ScheduleBatch SizeEpochs
LLaDA-8B-Instruct64642×10−42 \times10^{-4}cosine323
Dream-7B-Instruct32322×10−42 \times10^{-4}cosine323

Table 6. Training configurations for SemMask across the two backbone models.

Gradient Score Computation

For each sample, we sample multiple masking ratios t∼U(0,1)t \sim U(0,1) and independently mask the response tokens with probability tt, while keeping the prompt unchanged. Using the frozen backbone, we compute the original denoising objective only over the masked response tokens in each corrupted version. For every visible response token, we calculate the gradient of the denoising loss with respect to its input embedding and project this gradient onto the direction from the original token embedding to the mask token embedding. This projection measures how strongly masking the visible token would affect the reconstruction loss over the other masked tokens; a larger value indicates a greater contextual contribution and thus a higher semantic importance score. We average the scores across multiple masked versions to reduce randomness, such that tokens contributing more to the reconstruction of the masked sequence receive higher scores.

Human Evaluation Details

To evaluate text generation quality on the HANNA benchmark, three evaluators with master’s degrees independently rated each generated text on a five point scale. The evaluation considers five dimensions: content completeness, relevance, coherence, fluency, and factual consistency. The scoring criteria are defined as follows:

  • 1: The text fails to address the prompt or is largely incomprehensible, with severe deficiencies in at least four evaluation dimensions.

  • 2: The text partially addresses the prompt but contains major problems in at least three dimensions, such as substantial content omissions, logical contradictions, frequent grammatical errors, or factual inconsistencies.

  • 3: The text addresses the main topic and remains understandable, but contains one or two major deficiencies or several noticeable problems in completeness, coherence, fluency, relevance, or factual consistency.

  • 4: The text fully addresses the prompt and contains no major deficiencies, with only minor issues in one or two evaluation dimensions that do not substantially affect overall quality.

  • 5: The text fully satisfies the prompt across all five evaluation dimensions, demonstrating complete content, strong relevance, coherent organization, fluent language, and factual consistency without notable errors.

For each test item ii and evaluation dimension dd, the mean score assigned by the three evaluators is calculated as:

Sˉi(d)=Si,1(d)+Si,2(d)+Si,3(d)3(9)\bar{S}_i^{(d)} = \frac{S_{i,1}^{(d)} + S_{i,2}^{(d)} + S_{i,3}^{(d)}}{3} \tag*{(9)}

where Si,1(d)S_{i,1}^{(d)}, Si,2(d)S_{i,2}^{(d)}, and Si,3(d)S_{i,3}^{(d)} denote the scores assigned by the three evaluators.

The overall score for item ii is computed by averaging across the five evaluation dimensions:

Sˉi=15∑d=15Sˉi(d)(10)\bar{S}_i = \frac{1}{5} \sum_{d=1}^{5} \bar{S}_i^{(d)} \tag*{(10)}

The final human evaluation score for a model over all NN HANNA test samples is:

Sfinal=1N∑i=1NSˉi(11)S_{\mathrm{final}} = \frac{1}{N} \sum_{i=1}^{N} \bar{S}_i \tag*{(11)}

More Experimental Results

Computational Overhead Analysis

To evaluate the additional computational cost of SemMask, we compare its preprocessing and training overhead with standard SFT on 10,000 code training samples using 4×4 \timesV100 GPUs. As shown in Table 7, computing semantic importance scores requires only 392.56,s for LLaDA and 428.33,s for Dream. During training, SemMask increases the total runtime by only 0.55% for LLaDA and 0.81% for Dream compared with standard SFT, while maintaining comparable computational throughput. These results demonstrate that the additional cost of SemMask is mainly confined to the one-time preprocessing stage, with only marginal overhead during model training.

ModelPhaseTotal Time (s)Time per Step (s)Total PFLOPsThroughput (TFLOPs/s)
LLaDA-8B-SFTTraining21,502.7950.00158.427.37
LLaDA-8B-SemMask (Ours)Preprocessing392.56–34.2187.15
LLaDA-8B-SemMask (Ours)Training21,620.8850.00159.907.40
Dream-7B-SFTTraining27,180.1162.50138.625.10
Dream-7B-SemMask (Ours)Preprocessing428.33–36.2984.72
Dream-7B-SemMask (Ours)Training27,399.2866.66138.625.06

Table 7. Comparison of preprocessing and training costs between standard SFT and SemMask on 10,000 code training samples using 4×4 \timesV100 GPUs.

Adapting SemMask to Continuous Diffusion Models

We adapt SemMask to two representative continuous diffusion models, DiffuSeq and DiffuSeq-v2, by applying semantic guidance only to the target sequence. During training, high-importance target tokens receive weaker corruption and larger denoising weights, while the original model architectures and sampling mechanisms are otherwise preserved. During inference, high-scoring target tokens are fixed earlier to establish a global semantic structure before function words and local details are progressively completed. As shown in Table 8 and Table 9, SemMask consistently improves all evaluation metrics on both HANNA and HumanEval, with DiffuSeq-v2-SemMask achieving 0.3106 BLEU and 0.2549 accuracy. These results demonstrate that SemMask can be effectively extended beyond masked discrete diffusion models to continuous diffusion frameworks.

ModelBLEUBLEURTROUGE-LBERT-SGPT-5.1-S
DiffuSeq-SFT0.25460.43680.10010.53891.2658
DiffuSeq-SemMask (Ours)0.29120.44900.14240.59181.9644
DiffuSeq-v2-SFT0.27920.45670.19680.56741.5349
DiffuSeq-v2-SemMask (Ours)0.31060.50040.21650.59772.0116

Table 8. Performance of SemMask adapted to continuous diffusion models on the HANNA benchmark.

ModelAccuracyCodeBLEUBLEURT
DiffuSeq-SFT0.16440.09210.2397
DiffuSeq-SemMask (Ours)0.19470.11660.2963
DiffuSeq-v2-SFT0.20950.09900.2978
DiffuSeq-v2-SemMask (Ours)0.25490.13690.3259

Table 9. Performance of SemMask adapted to continuous diffusion models on HumanEval.

Generalization to Mathematical Reasoning Tasks

We further evaluate the generalization of SemMask to mathematical reasoning tasks using the GSM8K and MATH-500 benchmarks. As shown in Table 10, SemMask improves GSM8K accuracy from 70.09 to 73.20 on LLaDA and from 76.89 to 78.28 on Dream. On MATH-500, SemMask achieves performance comparable to the corresponding SFT baselines, with only marginal differences on both backbones. These results suggest that semantic-prioritized generation is particularly beneficial for longer reasoning trajectories, while offering limited advantages for tasks with shorter outputs.

ModelGSM8K ↑MATH-500 ↑
LLaDA-8B-Base67.8428.65
LLaDA-8B-Instruct68.2130.47
LLaDA-8B-SFT70.0931.93
LLaDA-8B-SemMask (Ours)73.2031.96
Dream-7B-Base74.3736.55
Dream-7B-Instruct75.0837.10
Dream-7B-SFT76.8939.66
Dream-7B-SemMask (Ours)78.2839.58

Table 10. Performance comparison on the GSM8K and MATH-500 mathematical reasoning benchmarks. The best result within each model family is highlighted in bold.

More Cases

We present additional case studies in Figure 6, Figure 7 and Figure 8. These cases further demonstrate that SemMask prioritizes semantically important tokens, enabling the model to establish a global semantic scaffold early and progressively complete the remaining tokens.

Case studies of SemMask on the HANNA benchmark

Figure 6. Case studies of SemMask on the HANNA benchmark.

Figure 7. Case studies of SemMask on the CommonGen benchmark.

Case studies of SemMask on the HumanEval benchmark
def how_many_times(string: str, substring: str) -> int:
    """
    Find how many times a given substring can be
    found in the original string. Count overlapping cases.

    >>> how_many_times('', 'a')
    0
    >>> how_many_times('aaa', 'a')
    3
    >>> how_many_times('aaaa', 'aa')
    3
    """
def how_many_times(string: str, substring: str) -> int:
    count = 0
    start = 0
    while start < len(string):
        start = string.find(substring, start)
        if start == -1:
            break
        count += 1
        start += 1
    return count
def int_to_mini_roman(number):
    """
    Given a positive integer, obtain its roman
    numeral equivalent as a string,
    and return it in lowercase.
    Restrictions: 1 <= num <= 1000

    Examples:
    >>> int_to_mini_roman(19) == 'xix'
    >>> int_to_mini_roman(152) == 'clii'
    >>> int_to_mini_roman(426) == 'cdxxvi'
    """
def int_to_mini_roman(number):
    roman_numerals = {
        1000: 'm', 900: 'cm', 500: 'd',
        400: 'cd', 100: 'c', 90: 'xc', 50: 'l',
        40: 'xl', 10: 'x', 9: 'ix', 5: 'v',
        4: 'iv', 1: 'i'
    }
    result = ''
    for value, numeral in roman_numerals.items():
        while number >= value:
            result += numeral
            number -= value
    return result

Figure 8. Case studies of SemMask on the HumanEval benchmark.

References

  1. [1]Ahmad, W. U.; Ficek, A.; Samadi, M.; Huang, J.; Noroozi, V.; Majumdar, S.; and Ginsburg, B. 2025. Opencompute instruct: A large-scale instruction tuning dataset for code llms. arXiv preprint arXiv:2504.04030.
  2. [2]Arriola, M.; Gokaslan, A.; Chiu, J.; Yang, Z.; Qi, Z.; Han, J.; Sahoo, S.; and Kuslesh­ov, V. 2025. Block diffusion: In­terpolating between autoregressive and diffusion language models. In International Conference on Learning Representa­tions, volume 2025, 50726–50753.
  3. [3]Austin, J.; Johnson, D. D.; Ho, J.; Tarlow, D.; and Van Den Berg, R. 2021a. Structured denoising diffusion mod­els in discrete state-spaces. Advances in neural information processing systems, 34: 17981–17993.
  4. [4]Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Micha­lew­ski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. 2021b. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  5. [6]Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. D. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  6. [7]Chen, Z.; Fang, G.; Ma, X.; Yu, R.; and Wang, X. 2025. dpar­allel: Learnable parallel decoding for dllms. arXiv preprint arXiv:2509.26488.
  7. [8]Chhun, C.; Colombo, P.; Suchanek, F.; and Clavel, C. 2022. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. In Proceedings of the 29th International Conference on Computational Linguis­tics, 5794–5836.
  8. [9]Deschenaux, J.; and Gulcehre, C. 2024. Promises, out­looks and challenges of diffusion language modeling. arXiv preprint arXiv:2406.11473.
  9. [10]Dieleman, S.; Sartran, L.; Roshannai, A.; Savinov, N.; Gani­n, Y.; Richemond, P. H.; Doucet, A.; Strudel, R.; Dyer, C.; Durkan, C.; et al. 2022. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089.arxiv.org/abs/2211.15089
  10. [11]Fan, A.; Lewis, M.; and Dauphin, Y. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 889–898.arxiv.org/abs/1805.04833
  11. [12]Feng, G.; Geng, Y.; Guan, J.; Wu, W.; Wang, L.; and He, D. 2026. Theoretical benefit and limitation of diffusion language model. Advances in Neural Information Processing Systems, 38: 24415–24459.
  12. [13]Gao, Z.; Guo, J.; Tan, X.; Zhu, Y.; Zhang, F.; Bian, J.; and Xu, L. 2024. Empowering diffusion models on the embedding space for text generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 4664–4683.
  13. [14]Gong, S.; Agarwal, S.; Zhang, Y.; Ye, J.; Zheng, L.; Li, M.; An, C.; Zhao, P.; Bi, W.; Han, J.; et al. 2025a. Scaling dif­fusion language models via adaptation from autoregressive models. In International Conference on Learning Representa­tions, volume 2025, 5046–5073.
  14. [15]Gong, S.; Zhang, R.; Zheng, H.; Gu, J.; Jaitly, N.; Kong, L.; and Zhang, Y. 2025b. Diff­ocoder: Understanding and im­proving masked diffusion models for code generation. arXiv preprint arXiv:2506.20639.
  15. [16]Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Vaughan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  16. [17]Gwak, D.; Jung, M.; Park, J.; Park, M.; Park, C.; Hyung, J.; and Choo, J. 2025. Reward-weighted sampling: Enhanc­ing non-autoregressive characteristics in masked diffusion llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 34562–34582.arxiv.org/abs/2509.00707
  17. [18]Hersche, M.; Moor-Smith, S.; Hofmann, T.; and Rahimi, A. 2025. Soft-masked diffusion language models. arXiv preprint arXiv:2510.17206.arxiv.org/abs/2510.17206
  18. [19]Hong, C.; An, S.; Kim, M.-S.; and Ye, J. C. 2025. Improving discrete diffusion unmasking policies beyond explicit reference policies. arXiv preprint arXiv:2510.05725.
  19. [21]Huang, Z.; Chen, Z.; Wang, Z.; Li, T.; and Qi, G.-J. 2026. Reinforcing the diffusion chain of lateral thought with diffusion language models. Advances in Neural Information Processing Systems, 38: 152677–152710.
  20. [22]Huang, Z.; Wang, Y.; Chen, Z.; and Qi, G.-J. 2025. Don’t Settle Too Early: Self-Reflective Remasking for Diffusion Language Models. arXiv preprint arXiv:2509.23653.
  21. [23]Kim, J.; Shah, K.; Kontonis, V.; Kakade, S.; and Chen, S. 2025. Train for the worst, plan for the best: Understand­ing token ordering in masked diffusions. arXiv preprint arXiv:2502.06768.
  22. [24]Kim, S. H.; Hong, S.; Jung, H.; Park, Y.; and Yun, S.-Y. 2026. Klass: Kl-guided fast inference in masked diffusion models. Advances in Neural Information Processing Sys­tems, 38: 92267–92301.
  23. [25]Kowsher, M.; Prathna, N. J.; Bhat, P.; Yu, C.-N.; Soltana­lian, M.; Garibay, I.; Garibay, O.; Chen, C.; and Yousefi, N. 2025. Predicting through generation: Why generation is bet­ter for prediction. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 26845–26871.
  24. [30]Lin, B. Y.; Zhou, W.; Shen, M.; Zhou, P.; Bhagavatula, C.; Choi, Y.; and Ren, X. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, 1823–1840.arxiv.org/abs/1911.03705
  25. [31]Luxembourg, O.; Permuter, H.; and Nachmani, E. 2025. Plan for speed: Dilated scheduling for masked diffusion language models. arXiv preprint arXiv:2506.19037.arxiv.org/abs/2506.19037
  26. [32]Majdidi, E.; and Crick, C. 2023. A Survey on Neural Text Generation and Detection. International Journal of Computational Linguistics (IJCL), 14(2): 19–35.
  27. [33]Mallinson, J.; Severyn, A.; Malmi, E.; and Garrido, G. 2020. FELIX: Flexible text editing through tagging and insertion. In Findings of the Association for Computational Linguistics: EMNLP 2020, 1244–1255.arxiv.org/abs/2003.10687
  28. [34]Nie, S.; Zhu, F.; You, Z.; Zhang, X.; Ou, J.; Hu, J.; Zhou, J.; Lin, Y.; Wen, J.-R.; and Li, C. 2026. Large language diffusions in Neural Information Processing Systems, 38: 50608–50646.
  29. [35]Sahoo, S. S.; Arriola, M.; Schiff, Y.; Gokaslan, A.; Marroquin, E.; Chiu, J. T.; Rush, A.; and Kuleshov, V. 2024. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37: 130136–130184.arxiv.org/abs/2406.07524
  30. [36]Song, Y.; Zhang, Z.; Luo, C.; Gao, P.; Xia, F.; Luo, H.; Li, Z.; Yang, Y.; Yu, H.; Qu, X.; et al. 2025. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.02193.arxiv.org/abs/2508.02193
  31. [37]Stern, M.; Chan, W.; Kiros, J.; and Uszkoreit, J. 2019. Insertion transformer: Flexible sequence generation via insertion operations. In International Conference on Machine Learning, 5976–5985. PMLR.arxiv.org/abs/1902.03249
  32. [38]Wang, Y.; Yang, L.; Li, B.; Tian, Y.; Shen, K.; and Wang, M. 2025. Revolutionizing reinforcement learning framework for diffusion language models. arXiv preprint arXiv:2509.06949.arxiv.org/abs/2509.06949
  33. [41]Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; Lin, H.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Lin, J.; Dang, K.; Lu, K.; Bao, K.; Yang, K.; Yu, L.; Li, M.; Xue, M.; Zhang, P.; Zhu, Q.; Men, R.; Lin, R.; Li, T.; Xia, T.; Ren, X.; Ren, X.; Fan, Y.; Su, Y.; Zhang, Y.; Wan, Y.; Liu, Y.; Cui, Z.; Zhang, Z.; and Qiu, Z. 2024. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115.arxiv.org/abs/2412.15115
  34. [42]Yang, L.; Tian, Y.; Li, B.; Zhang, X.; Shen, K.; Tong, Y.; and Wang, M. 2026. Mmada: Multimodal large diffusion language models. Advances in Neural Information Processing Systems, 38: 138867–138907.arxiv.org/abs/2505.15809
  35. [44]You, Z.; Nie, S.; Zhang, X.; ZHOU, J.; Lu, Z.; Wen, J.-R.; and Li, C. 2026. Llada-v: Large language diffusion models with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10093–10105.arxiv.org/abs/2505.16933
  36. [46]Zhang, S.; Zhao, Y.; Geng, L.; Cohan, A.; Tuan, L. A.; and Zhao, C. 2025. Diffusion vs. autoregressive language models: A text embedding perspective. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 4273–4303.arxiv.org/abs/2505.15045

Paper details

vlnc:W_5737rhcb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents