WorksComputer Science
DTCA: Denoising Token Credit Assignment for Reinforcement Learning in Diffusion Language Models
Abstract
Diffusion language models (DLMs) generate text through iterative denoising, enabling bidirectional context modeling and parallel token refinement. Recent studies have applied reinforcement learning (RL) to DLMs and improved their reasoning performance. However, the iterative denoising process makes credit assignment more challenging, as final outcome rewards provide limited information about how different tokens contribute to the final output. In this paper, we propose Denoising Token Credit Assignment (DTCA), a framework for token-level credit assignment in reinforcement learning for diffusion language models. DTCA estimates token contributions by measuring the improvement in prediction confidence of remaining masked positions after token commitment and groups these signals according to part-of-speech (POS) categories. These contribution weights are used to select informative denoising timesteps and assign token-level learning signals during RL training. DTCA further calibrates these learning signals according to token-level entropy changes induced by policy updates. Experiments on LLaDA-8B-Instruct show that DTCA consistently improves mathematical reasoning performance across different generation budgets, with gains of up to 9.0 points on MATH500 and 5.3 points on GSM8K over the base model.
Introduction
Diffusion language models (DLMs) have recently attracted increasing attention as an emerging paradigm for language modeling due to their ability to model bidirectional context and perform parallel token refinement [42, 65, 54], offering an alternative to autoregressive generation [5, 6, 51, 58]. Recent studies have explored reinforcement learning for DLMs by leveraging the generation process, mainly constructing learning signals from denoising trajectories or intermediate states during training [3, 21, 2, 7, 34]. However, DLMs generate sequences through multiple denoising steps, where tokens are gradually committed and each commitment can affect the refinement of remaining masked positions [14]. Therefore, RL optimization in DLMs requires finer-grained credit assignment that considers how different tokens affect subsequent refinement.
Figure 1. Preliminary analysis of POS-level differences in DLM denoising using rollouts generated from the GSM8K [30] training set. (a) Commitment timing of different POS categories across denoising steps. (b) Improvement in the prediction confidence of remaining masked positions after individual token commitment, averaged within each POS category. POS tags denote NOUN (noun), NUM (numeral), PUNCT (punctuation), PROPN (proper noun), VERB (verb), ADP (adposition), DET (determiner), and ADJ (adjective).
Achieving finer grained credit assignment requires estimating the contribution of individual tokens during DLM denoising. Specifically, we measure token contribution by how much a committed token improves the model’s prediction confidence on the remaining masked positions. However, directly estimating such token-level confidence improvements during RL training is challenging because multiple tokens are committed at the same denoising step and interact through the shared denoising process, making their individual effects difficult to isolate. Moreover, the estimated contribution can vary substantially across different rollout samples. To obtain more reliable estimates, we categorize tokens by part-of-speech (POS) categories and aggregate their refinement effects within each category. POS categories provide a natural grouping criterion because they capture different linguistic functions, with nouns and verbs typically carrying lexical information while determiners and adpositions mainly providing grammatical structure [31].
The denoising behaviors of different POS categories are further analyzed using rollouts generated from the GSM8K [30] dataset. As shown in Figure 1(a), different POS categories exhibit distinct commitment trajectories across denoising steps. NOUN maintains a relatively high commitment ratio, whereas VERB has a high ratio at the beginning that declines substantially over the denoising process. Other categories, such as ADP, DET, and ADJ, exhibit relatively lower commitment ratios with more stable variations. Figure 1(b) further shows the confidence gain associated with different POS categories. Clear differences emerge across categories: VERB achieves the largest confidence gain, followed by NOUN and PROPN, whereas categories such as DET and ADP show relatively smaller gains. These observations indicate that POS categories capture different denoising behaviors and refinement effects, making them a useful basis for grouping tokens and estimating contribution weights.
Based on these observations, we propose Denoising Token Credit Assignment (DTCA), a token-level credit assignment framework for reinforcement learning in diffusion language models. DTCA estimates token contributions from denoising refinement signals and aggregates them into POS-level contribution weights to obtain more stable and interpretable learning signals. These contribution signals are then used to select informative denoising timesteps and derive token-level learning signals during RL optimization. DTCA further incorporates an entropy-based credit calibration module inspired by OPEFO [4] to calibrate the learning signals according to token-level entropy changes induced by policy updates. Experiments on LLaDA-8B-Instruct show that DTCA consistently improves mathematical reasoning performance across different generation budgets, with improvements of up to 9.0 percentage points on MATH500 [29] and 5.3 percentage points on GSM8K [30] over the base model.
The main contributions are summarized as follows:
We analyze different denoising behaviors across POS categories in diffusion language models and use POS categories to estimate the effect of token commitments on subsequent refinement.
We propose DTCA, a token-level credit assignment framework that aggregates denoising refinement signals by POS categories and uses the resulting contribution weights for denoising timestep selection and token credit assignment. DTCA further adapts entropy-based credit calibration from OPEFO [4] to the diffusion language model setting for improved RL optimization.
Extensive experiments demonstrate the effectiveness of DTCA on mathematical reasoning benchmarks and validate the benefits of its key components.
Preliminaries
Diffusion Language Model Denoising
Masked diffusion language models generate responses through iterative denoising [5, 6]. Given a prompt , let denote a sampled trajectory, where is the partially masked response at denoising step and is the completed response. Under the monotonic unmasking schedule used in this work, each step reveals a subset of masked positions as predicted tokens; we call this token commitment. Committed tokens remain visible in subsequent states and can influence predictions at the remaining masked positions. The distribution below is the response distribution induced by this iterative sampling procedure, not an autoregressive token factorization.
Reinforcement Learning for Diffusion Language Models
Given prompts and sampled responses , reinforcement learning seeks to maximize expected sequence-level reward:
Group-relative policy optimization converts rewards from multiple responses to the same prompt into response advantages [40]. In masked diffusion RL, these advantages can be applied to predictions at selected denoising states [3]. Let be the selected steps, their optimization target positions, and the target token at position . A schematic token-level surrogate is
where and denotes a penalty relative to a frozen reference policy [40, 3]. The exact target set, credit , KL estimator, and update loss used by DTCA are specified below and in Appendix § 7. A sequence-level reward alone does not identify which commitments help subsequent denoising; this motivates assigning credit at the denoising-state and target-token levels.
Method
We propose Denoising Token Credit Assignment (DTCA), a token credit assignment framework for reinforcement learning in diffusion language models. As illustrated in Figure 2, DTCA estimates token contributions from denoising refinement dynamics and uses the resulting contribution weights for informative denoising state selection and token-level credit assignment. DTCA further calibrates the token-level learning signals according to entropy changes induced by policy updates.
Figure 2. Overview of Denoising Token Credit Assignment (DTCA). DTCA estimates token contributions from denoising refinement dynamics and uses the resulting contribution weights for denoising state selection and token-level credit assignment. The obtained learning signals are further calibrated according to entropy changes induced by policy updates for reinforcement learning optimization.
Token Contribution Estimation by POS Category
As illustrated in Figure 2, DTCA estimates token-level refinement contributions from denoising dynamics and aggregates these estimates into category-level contribution weights. Given a denoising trajectory , different token commitments may have different effects on subsequent refinement. DTCA therefore estimates the refinement contribution of each committed token by measuring the improvement in prediction confidence of the remaining masked positions after the commitment.
Specifically, for each individual token commitment at denoising step , let denote the response positions that remain masked both before and after the commitment, excluding the committed position . Given a fixed prompt , we define the aggregate prediction confidence of a denoising state over as the sum of the maximum prediction probabilities at those positions:
where indexes a masked position, is the vocabulary, and is the probability assigned to token at position conditioned on prompt and denoising state . For a committed token , its refinement contribution estimate is computed as:
where denotes the denoising state after committing token individually. Negative total changes are set to zero; the estimate is not divided by the number of remaining masks. A larger indicates a stronger estimated refinement effect of the committed token on subsequent denoising.
Obtaining reliable token-level estimates directly during RL training is challenging because evaluating individual token effects requires re-evaluating token commitments with additional forward passes, introducing substantial computational overhead during policy optimization. Moreover, these estimates can vary across different rollout samples. Therefore, DTCA constructs contribution estimates from offline denoising rollouts before RL training and groups them according to part-of-speech (POS) categories to obtain more stable category-level contribution weights.
During offline estimation, DTCA uses a denoising budget equal to half of the maximum generation length. At each denoising step, two tokens are committed, and DTCA evaluates each token separately through additional forward passes based on the confidence improvement of remaining masked positions. These estimates are aggregated at the POS level to reduce variance across rollouts.
For each committed token , we assign its POS category as . The contribution of each POS category is obtained by averaging the refinement contribution estimates of tokens belonging to that category:
where denotes the number of committed tokens in POS category . The resulting category-level contribution represents the average refinement effect estimated for tokens in each category.
POS Contribution Weight Construction.
The estimated POS contributions are normalized into bounded contribution weights:
where denotes the set of POS categories. The weight lies in the range and represents the scaling factor derived from the estimated refinement contribution of category . We constrain this range to prevent excessive credit scaling while preserving the original RL objective. These weights are used to adjust token-level credits during RL optimization.
Contribution-Based Timestep Selection
The category contribution weights are also used to identify informative denoising timesteps for RL training. Following RLDF [3], DTCA optimizes only a subset of denoising timesteps from each rollout. For a denoising timestep , let denote the set of tokens committed at this step. We compute the importance score of each timestep as:
where denotes the confidence of token when it is committed and denotes the contribution weight of its POS category. The term assigns higher scores to timesteps containing lower-confidence token commitments, while incorporates the estimated refinement contribution of the committed tokens.
DTCA ranks denoising timesteps according to and selects the smallest set of timesteps whose cumulative score reaches a predefined fraction of the total score. The number of selected timesteps therefore adapts to the distribution of contribution signals in each rollout. Lower and upper bounds are applied to the number of selected timesteps for stable and efficient training.
Contribution-Based Credit Assignment
The category contribution weights are used to assign different credits to tokens within each selected denoising timestep. For a selected timestep , let denote the tokens used for optimization, including the committed tokens at timestep and the retained masked positions selected by the threshold-based filtering strategy in RLDF [3]. POS categories for all tokens in are obtained from the completed rollout, where retained masked positions are associated with their final predicted tokens. The credit weight of token is determined by the contribution weight of its POS category:
where is the POS category of token . The normalization keeps the average credit within each timestep equal to one, so that the overall scale of the timestep advantage remains unchanged. Given the original timestep advantage , DTCA computes the token-level learning signal as
Tokens from categories with larger contribution weights receive stronger learning signals, while tokens from categories with smaller contribution receive weaker signals. In this way, DTCA transforms the original response-level feedback into more fine grained token credit during RL training.
Entropy-Based Credit Calibration
Although DTCA provides token-level learning signals according to denoising contribution, different denoising states may induce different entropy changes during policy updates. To further calibrate the token-level learning signals, DTCA incorporates an entropy-based credit calibration module inspired by OPEFO [4]. Unlike the original formulation for autoregressive generation, we adapt the token-level entropy change estimation to selected denoising states and their remaining target positions in diffusion language models.
For each response , selected denoising state , and remaining target position , the entropy change induced by the policy update is estimated as:
where denotes the probability assigned by the current policy to the target token and denotes the entropy of the full prediction distribution at this position. This formulation follows the first-order entropy change estimation of OPEFO [4] while replacing autoregressive decoding steps with selected denoising states and target positions.
Within each response, the entropy-increasing and entropy-decreasing updates are separated according to the sign of :
Following OPEFO [4], we compute a separate entropy flow balancing coefficient for each response:
The entropy balancing coefficient is then used to rescale token-level learning signals to balance entropy-increasing and entropy-decreasing updates. When , entropy-increasing updates are amplified and entropy-decreasing updates are attenuated. When , the adjustment is reversed. When , the credits remain unchanged:
Experiments
Experimental Setup
Model and data. We initialize RL training from LLaDA-8B-Instruct [5] without additional SFT. Following RLDF [3], we use the MATH [28] training split with the same data processing pipeline for RL optimization. To construct the fixed POS contribution table, DTCA collects offline denoising rollouts from the training data of MATH and GSM8K and builds a shared POS contribution table without updating model parameters.
Training. We train LLaDA-8B-Instruct with reinforcement learning without an intermediate SFT stage. For each rollout, DTCA selects denoising states with the highest POS-weighted uncertainty scores until covering 70% of the total score mass, with the number of selected states constrained between 2 and 32. We perform policy optimization using group-relative advantages and regularize the policy against a frozen reference model with the k3 KL estimator [64]. Rollouts are generated using block-wise decoding with KV cache, following Fast-dLLM [37]. Full implementation details and hyperparameters are provided in Appendix § 8.
Evaluation and baselines. We evaluate mathematical reasoning performance on MATH500 [29] and GSM8K [30] using final-answer accuracy. Following prior work, we evaluate all models under generation budgets of 128, 256, and 512, with denoising steps matched to the corresponding maximum generation lengths. We compare DTCA with LLaDA-8B-Instruct, Uniform SFT, LLaDA-1.5, and representative reinforcement learning methods for diffusion language models, including d1 [1], ESPO [17], TraceRL [21], Coupled-GRPO [16], and RLDF [3]. Additional details on training configurations and baseline evaluation settings are provided in Appendix § 8.2 and Appendix § 8.4.
Main Results
Table 1 compares DTCA with supervised and reinforcement learning baselines on MATH500 and GSM8K. DTCA consistently improves over LLaDA-8B-Instruct across both benchmarks and all evaluated generation budgets. On MATH500, DTCA improves the base model by 9.0, 6.8, and 4.8 percentage points at budgets of 128, 256, and 512, respectively. On GSM8K, the corresponding improvements are 5.3, 2.1, and 2.5 percentage points. These results show that DTCA provides consistent gains under different denoising budgets rather than relying on a specific inference setting. Compared with supervised optimization and existing diffusion language model reinforcement learning methods, DTCA achieves competitive performance across the reported settings, demonstrating the effectiveness of the proposed token-level credit assignment framework.
The largest improvements on both benchmarks occur at the smallest evaluated generation budget of 128, while gains remain positive at budgets of 256 and 512. Although the relative improvements become smaller at larger generation budgets, DTCA consistently improves over the base model, highlighting its stronger gains under limited denoising computation.
| MATH500 | GSM8K | |||||
| 2-4(lr)5-7 Method | 128 | 256 | 512 | 128 | 256 | 512 |
| LLaDA-8B-Instruct [5] | 30.8 | 36.4 | 39.2 | 75.3 | 83.2 | 82.9 |
| LLaDA-1.5 [33] | 32.6 | 39.4 | 39.8 | 76.2 | 81.6 | 83.0 |
| Uniform SFT | 24.6 | 30.0 | 32.6 | 74.8 | 80.7 | 78.6 |
| d1 [1] | 33.6 | 37.6 | 37.2 | 75.8 | 82.4 | 81.8 |
| ESPO [17] | 37.6 | 38.0 | 40.2 | 78.1 | 83.2 | 84.0 |
| TraceRL [21] | 32.6 | 34.2 | 34.0 | 76.6 | 80.4 | 81.8 |
| Coupled-GRPO [16] | 35.2 | 40.2 | 40.2 | 78.0 | 83.6 | 83.2 |
| RLDF [3] | 36.2 | 41.2 | 42.6 | 77.2 | 83.4 | 82.7 |
| DTCA w/o Entropy Calibration | 37.4 | 41.8 | 39.8 | 78.3 | 83.9 | 81.9 |
| DTCA w/o Contribution Credit | 36.2 | 39.4 | 41.2 | 77.1 | 82.9 | 82.3 |
| DTCA (ours) | 39.8 | 43.2 | 44.0 | 80.6 | 85.3 | 85.4 |
| vs. Base | +9.0 | +6.8 | +4.8 | +5.3 | +2.1 | +2.5 |
Table 1. Mathematical reasoning accuracy (%) on MATH500 and GSM8K. The last rows report DTCA, its ablation variants, and the improvement over the base model. Boldface indicates the best result in each column, and denotes the improvement over LLaDA-8B-Instruct.
Figure 3. Optimization dynamics of DTCA variants during RL training. (a) Response reward, (b) gradient norm, and (c) mean KL divergence. DTCA is compared with variants without Contribution Credit and Entropy Calibration.
Ablation Study
Table 1 reports ablation results for different DTCA components. DTCA w/o Contribution Credit replaces the contribution-based token credits with uniform credits while keeping the same timestep selection strategy. DTCA w/o Entropy Calibration removes the entropy-based credit calibration module while preserving contribution-based credit assignment and timestep selection.
Removing either component decreases performance compared with the full DTCA model across both benchmarks and different generation budgets. At the 256 generation budget, removing Contribution Credit leads to decreases of 3.8 and 2.4 percentage points on MATH500 and GSM8K, respectively, while removing Entropy Calibration results in smaller drops of 1.4 and 1.4 points. At the 512 generation budget, removing Entropy Calibration causes larger performance decreases, with drops of 4.2 and 3.5 points on MATH500 and GSM8K. These results show that both components contribute to the overall performance of DTCA.
Analysis of Optimization Dynamics
To understand how different DTCA components influence RL optimization, we analyze training dynamics in terms of reward, gradient norm, and KL divergence. Figure 3 compares DTCA with variants without Contribution Credit and Entropy Calibration during training.
Removing Contribution Credit results in similar reward trajectories to DTCA, but leads to substantially different update behaviors. In particular, the variant with uniform credit assignment consistently exhibits larger gradient norms throughout training while achieving comparable reward improvements. This suggests that contribution-based credit assignment mainly affects the quality of token-level learning signals, allowing the model to optimize with different update dynamics rather than simply changing the final reward trend. By considering token-level contribution differences, DTCA provides a more structured learning signal than uniform credit assignment.
Removing Entropy Calibration mainly affects the stability of later-stage policy updates. Although its reward trajectory initially follows a similar trend, the variant without entropy calibration shows a faster increase in KL divergence and lower reward in later training stages. This indicates that entropy calibration helps control policy changes by adjusting learning signals according to entropy changes, leading to more stable optimization during RL training.
Figure 4. Analysis of commitment dynamics on MATH500. DTCA is compared with the base model across three denoising stages. Each cell shows the difference in (a) token commitment ratio, (b) confidence at commitment, and (c) entropy at commitment.
Analysis of Token Commitment Dynamics
To understand whether contribution-based credit assignment changes the denoising behavior of diffusion language models, we analyze token commitment patterns after RL training. Token commitment reflects when tokens become fixed during denoising, while the commitment distribution and prediction states provide insights into the learned denoising behavior. Figure 4 compares DTCA with the base model on MATH500 across early, middle, and late denoising stages. Each cell reports the difference between DTCA and the base model in token commitment ratio, prediction confidence at commitment, and target entropy at commitment.
DTCA changes the commitment distribution across token categories rather than uniformly shifting all commitments. For example, DTCA consistently reduces the commitment ratio of NUM tokens while increasing the commitment ratio of PUNCT tokens across different denoising stages. These changes suggest that DTCA learns different commitment patterns across token categories instead of simply accelerating or delaying all token predictions. Notably, changes in commitment frequency do not always correspond to changes in commitment states. For example, VERB shows limited changes in commitment ratio but substantial increases in confidence and reductions in entropy at commitment. This indicates that DTCA influences not only which tokens are committed, but also the model states when commitments occur. The observed changes are consistent with leveraging denoising refinement signals to improve token-level credit assignment during reinforcement learning.
Analysis of Entropy-Based Credit Calibration
We study how entropy calibration changes token credits and prediction entropy across POS categories. We compare DTCA with a variant that keeps the same contribution-based credit assignment and timestep selection but removes entropy calibration. Figure 5 shows the entropy changes during training. Positive values mean increased entropy, and negative values mean decreased entropy. Values in parentheses show the relative change in assigned credit after calibration. We compare these credit changes with the entropy curves for each category.
In Figure 5, the two variants show similar entropy changes early in training and gradually diverge later. For most of the shown POS categories, DTCA has a lower curve late in training. Each curve shows changes relative to that run’s early training average. For example, without calibration, the NUM curve rises above zero after about 200 updates and stays positive for most of the remaining steps. DTCA stays near or below zero for most of this period. For SYM, the curve without calibration returns toward zero late in training, while DTCA remains negative. AUX shows the opposite pattern. Its curve without calibration drops sharply below zero near the end, while the DTCA curve stays close to zero. The differences between the two variants therefore vary across POS categories, rather than following the same direction for all categories.
Credit changes and entropy changes do not always follow the same direction. For example, SYM receives more credit but stays below its early entropy average late in training. AUX receives less credit but ends above its early average. Credit changes the strength of the existing policy update. Whether entropy rises or falls also depends on the advantage signal, the current prediction distribution, and the target token. Compared with OPEFO [4], DTCA applies entropy flow balancing after assigning POS-based credits. Within each response, updates in the same entropy-change set share a scaling factor. This factor and the final normalization preserve the credit ratios within that set. Thus, entropy flow balancing still leaves different credits for tokens with different POS contribution weights. The two runs also see different denoising states and different mixes of target tokens as training progresses. The curves therefore show overall training trends, rather than measuring the same tokens immediately before and after a single policy update.
Figure 5. Analysis of entropy-based credit calibration. Each panel shows the target entropy change of a POS category during RL training. The blue solid and orange dashed curves represent DTCA and DTCA without entropy calibration, respectively. The vertical axis shows the difference between the mean predictive entropy over the most recent 25 updates and the mean over the first 100 updates of the same run. Percentages in parentheses show the relative change in assigned credit after entropy calibration, and is the number of tokens counted.
Conclusion
We presented Denoising Token Credit Assignment (DTCA), a framework for fine-grained token-level credit assignment in reinforcement learning for diffusion language models. DTCA estimates token contributions from the confidence improvement of remaining masked positions after token commitment, aggregates these signals by POS categories, and uses the resulting contribution weights for denoising timestep selection and token-level credit assignment. DTCA further calibrates contribution credits according to entropy changes induced by policy updates, balancing entropy-increasing and entropy-decreasing updates. On LLaDA-8B-Instruct, DTCA improves the reproduced base model by 6.8 percentage points on MATH500 and 2.1 points on GSM8K under the 256-step evaluation setting. Training dynamics and component ablations show that contribution-based credit assignment improves token-level learning signals, while entropy-based calibration helps maintain more stable policy updates. These results demonstrate that denoising dynamics provide useful information for fine-grained credit assignment in reinforcement learning for diffusion language models.
Related Work
Diffusion Language Models
Building on the denoising paradigm of diffusion models [59], diffusion language models (DLMs) generate text through iterative denoising [55, 56, 57] rather than autoregressive next-token prediction [12, 11, 44]. Early discrete diffusion language models explore masked generation objectives and iterative refinement for language modeling [10, 6, 13]. LLaDA scales this paradigm to large language models with a masked diffusion objective and demonstrates competitive language generation performance [5]. Subsequent works further study improved training objectives, generation strategies [52, 53], scaling behaviors [15, 16, 37], and data efficiency under limited-data and multi-epoch training settings [62, 61, 60]. When applying reinforcement learning to these models, their iterative generation process requires particular consideration. Unlike autoregressive models, DLMs maintain multiple intermediate denoising states, where tokens are progressively committed and remaining masked positions are refined. This generation process introduces new challenges and opportunities for reinforcement learning in diffusion models.
Reinforcement Learning for Diffusion Language Models
Reinforcement learning for diffusion language models requires optimization methods that account for the iterative denoising process [8, 45]. Existing approaches mainly adapt RL objectives [43, 40, 63, 49] or exploit intermediate denoising information [9] to construct effective learning signals [32, 18, 33]. d1 introduces diffu-GRPO [1], Coupled-GRPO reduces likelihood estimation variance through complementary masks [16], and ESPO uses evidence lower bound as a proxy for sequence likelihood during RL optimization [17]. Other methods leverage denoising trajectories [48] and intermediate states to improve RL training, including TraceRL, RLDF, and DiSPO [21, 3, 2]. These methods demonstrate how diffusion generation characteristics can be incorporated into RL optimization [47]. However, they mainly focus on optimization objectives [19, 20], likelihood estimation [46], or denoising trajectories, while how to assign fine-grained credit to individual tokens during the denoising process remains underexplored.
Fine-grained Credit Assignment
Sequence-level rewards in language model reinforcement learning [41, 39] provide limited information about which tokens or reasoning steps contribute to the final outcome, motivating recent studies on fine-grained credit assignment [27, 50]. Existing approaches improve reward granularity through process supervision [35], step-level feedback, and token-level reward redistribution [36] or importance estimation [29, 38, 22, 23, 24, 25, 26]. However, these methods are mainly developed for autoregressive language models, where tokens are generated sequentially. For diffusion language models, DACA-GRPO provides an initial exploration of token-level credit assignment by weighting committed tokens according to denoising progress and changes in the predicted probabilities of remaining masked positions [7]. Different from DACA-GRPO, DTCA estimates token contributions from confidence improvements after token commitment, aggregates these signals according to POS categories, and uses the resulting contribution structure for denoising timestep selection and token-level credit assignment.
Method Supplement
Notation and boundary cases.
In Eq. (7), contains the positions committed at denoising step and is used for timestep selection. The optimization targets may differ and include both committed positions and retained masked positions selected by threshold-based filtering. POS categories for all optimization targets are obtained from the corresponding tokens in the completed rollout. In Eq. (10), denotes the policy update step size used in the first-order entropy change approximation, and and collect targets with positive and negative estimated entropy changes in response .
All 18 POS categories have entries in the contribution table. A POS tag outside this inventory, or a model token that cannot be aligned to a POS span, is assigned to the X category and uses its computed weight . If all category means coincide, we set for every category. If the total magnitude of estimated entropy change is zero, we set . A zero response advantage yields zero token credits without applying Eq. (14). We exclude empty commitment sets from offline statistics and states with no optimization targets from training.
\caption{One RL Update with Denoising Token Credit Assignment (DTCA)}
\label{alg:dtca}
\small
\begin{algorithmic}[1]
\State \textbf{Input:} Policy $\pi_\theta$, reference $\pi_{\mathrm{ref}}$, prompts $\mathcal{X}$, optimizer $O$
\Statex \hspace{\algorithmicindent} Fixed offline POS weights $\{w_c\}$, threshold $\mu$
\State $\tau\leftarrow\pi_\theta(\mathcal{X})$ \Comment{Generate denoising rollouts}
\State Compute rewards, advantages $\{A^{(b)}\}$, and token POS categories
\For{each response $b$ in $\tau$}
\For{each denoising timestep $t$ in response $b$}
\State $q_t^{(b)}\leftarrow\sum_{i\in\mathcal{A}_t^{(b)}}w_{c_i}(1-p_i)$
\EndFor
\State Select the highest-scoring steps covering mass $\mu$; apply count bounds to obtain $\mathcal{T}^{(b)}$
\For{each selected timestep $t\in\mathcal{T}^{(b)}$}
\State Reconstruct $x_t^{(b)}$ and form the retained targets $\mathcal{B}_t^{(b)}$
\State $a_{t,i}^{(b)}\leftarrow w_{c_i}/\mathrm{Mean}_{k\in\mathcal{B}_t^{(b)}}(w_{c_k})$ for $i\in\mathcal{B}_t^{(b)}$
\State $A_{t,i}^{(b)}\leftarrow a_{t,i}^{(b)}A^{(b)}$
\State Estimate $\Delta H_{t,i}^{(b)}$ from $A_{t,i}^{(b)}$, current target probabilities, and entropies
\EndFor
\EndFor
\For{each response $b$ in $\tau$}
\State Form $S_b^+$ and $S_b^-$; compute $\lambda_b$ and clip it to $[-0.95,0.95]$
\State Scale $A_{t,i}^{(b)}$ by $1+\lambda_b$ if $\Delta H_{t,i}^{(b)}>0$, otherwise by $1-\lambda_b$, to obtain $\hat{A}_{t,i}^{(b)}$
\State Normalize $\hat{A}_{t,i}^{(b)}$ to obtain $\widetilde{A}_{t,i}^{(b)}$ \Comment{Preserve total response credit}
\EndFor
\State $\theta\leftarrow O(\theta,\nabla_\theta\mathcal{L}_{\mathrm{DTCA}})$
\end{algorithmic}
Training objective.
Let be response ’s group-normalized advantage, used as in Eq. (9) at each selected step . The targets include masked positions with predicted token probability at least and positions committed at step . Since the completed rollout is available after generation, POS categories can be assigned to all targets in . Eqs. (8)–(13) yield for each target. We restore the response’s total credit:
This response-wide factor restores the overall credit scale after entropy calibration, while clipping may leave a residual entropy imbalance. The update loss is
where and . The reference policy is frozen. With one policy iteration, PPO ratio clipping is inactive. Entropy calibration modifies rather than adding an entropy loss.
Experimental Setup and Implementation Details
Base Models and Datasets
The policy and frozen reference model both start from LLaDA-8B-Instruct, without an intermediate SFT stage. RL training uses the MATH training set processed for RLDF. Offline POS contribution estimates are constructed from denoising rollouts over the MATH and GSM8K training splits without parameter updates to build a shared fixed POS contribution table. We evaluate on MATH500 and the GSM8K test split; neither evaluation split is used for RL training or POS table construction.
Training Configuration
We reproduce the compared methods under a unified experimental setup wherever applicable. RL methods start from LLaDA-8B-Instruct and use the same RLDF-processed MATH training split and mathematical correctness reward. All methods are evaluated using the common protocol in Appendix § 8.4. Each method retains its own objective and method-specific settings: we follow its original paper first, and use its released open-source code for details not specified in the paper.
For DTCA and the locally reproduced RLDF baseline, we use full-parameter bfloat16 training with AdamW, a constant learning rate of , and zero weight decay. Each prompt produces four responses, whose rewards are standardized within the group to form response advantages. Both methods use the same low-probability threshold of 0.2 for training-target filtering, token-level loss normalization, and a frozen-reference KL penalty with coefficient .
Their training rollouts use a maximum response length of 256 tokens and 256 denoising steps. Decoding uses a block size of 32 and a dynamic confidence threshold of ; when no position crosses the threshold, the highest-confidence position is committed. This decoding threshold is distinct from the training-target filter. KV caching is used during rollout generation.
Implementation Details of DTCA
We tag completions with spaCy’s en_core_web_sm POS model and align its word spans with model token positions. The 18 native POS categories share one contribution table. Before RL training, we map the offline category means to using Eq. (6). These weights are used for both timestep selection and token credit assignment and remain fixed throughout RL training.
DTCA changes the timestep-selection and credit-assignment rules relative to the locally reproduced RLDF baseline. For state selection, we rank observed denoising steps by and cover at least of the total priority mass, selecting between 2 and 32 steps. At each selected step, training targets are the remaining masked positions retained by the shared low-probability filter, plus the positions committed at that step even if they fall below the filter threshold. POS credits are normalized to preserve the mean response advantage within each selected step. Entropy-flow calibration limits the absolute balancing coefficient to and restores each response’s total credit after rescaling.
Evaluation Settings
We evaluate each checkpoint at maximum response lengths of 128, 256, and 512 tokens, using the same number of denoising steps as the response length. For every setting, we use block size 32, dynamic confidence threshold , temperature , and one generated response per question. The same math prompt template and decoding settings are used for DTCA and all locally reproduced baselines.
We evaluate only whether the final answer is correct, without scoring the intermediate reasoning steps. Accuracy is the fraction of questions with correct final answers on the complete test split. All baseline scores in Table 1 are obtained from our local reproductions under this evaluation protocol.
Additional Experiments and Analysis
Additional Ablation Studies
Contribution-based timestep selection.
Table 2 compares selection rules on a shared dataset using our fixed POS weights in . DTCA uses and selects 2–32 states. RLDF samples without replacement from a temperature-1 softmax of state mean token surprisal, using 16 or 32 states or DTCA’s per-response count. RLDF top- takes the highest-surprisal states, or all available states if fewer than . Coverage is the fraction of committed tokens, confidence , low-confidence mass , or entropy in selected states. Ratios are averaged over responses. Sampled RLDF uses 2,000 independent selections.
| Method | State ratio | Token cov. | Conf. cov. | Low conf. cov. | Entropy cov. |
| DTCA () | 16.82 | 19.99 | 17.72 | 40.59 | 37.81 |
| RLDF | 23.31 | 21.33 | 20.79 | 26.52 | 25.56 |
| RLDF (same count, ) | 16.82 | 15.01 | 14.34 | 19.93 | 19.22 |
| RLDF top-16 | 23.31 | 10.01 | 7.33 | 42.46 | 35.88 |
| DTCA () | 31.81 | 36.02 | 33.24 | 62.81 | 59.31 |
| RLDF | 45.42 | 42.61 | 41.73 | 49.91 | 48.73 |
| RLDF (same count, ) | 31.81 | 28.94 | 28.06 | 36.46 | 35.21 |
| RLDF top-32 | 45.42 | 28.75 | 24.99 | 66.39 | 60.66 |
Table 2. Coverage of selected denoising states. State ratio is the selected fraction per response. RLDF (same count) matches DTCA’s count at each . All entries are mean percentages.
At a matched state ratio of 16.82%, DTCA () increases low-confidence coverage from 19.93% to 40.59% and entropy coverage from 19.22% to 37.81% relative to RLDF, while retaining broader token coverage. RLDF top-16 achieves slightly higher low-confidence coverage (42.46%) but covers fewer tokens (10.01% versus 19.99%).
At and a matched state ratio of 31.81%, DTCA increases low-confidence coverage from 36.46% to 62.81% and entropy coverage from 35.21% to 59.31%. It also achieves higher coverage than RLDF while selecting fewer states. RLDF top-32 obtains comparable uncertainty coverage but selects more states and covers fewer tokens. These results show that DTCA selects denoising states with different uncertainty distributions, while the impact of state selection on final performance is evaluated through the main results and ablations.
Analysis of Denoising Dynamics
Policy entropy changes across POS categories.
Figure 6 and the upper panel of Figure fig:appendix_entropy_training compare current-commit token entropy by POS. Over updates 401–500, NUM entropy is 0.063 nats below its early baseline in DTCA but 0.208 nats above it without calibration. Late positive shifts in the uncalibrated run also appear for PROPN, X, and SPACE, rather than being confined to NUM.
Figure 6. Current-commit token entropy changes in DTCA. Each 20-update bin subtracts the POS-specific mean over updates 1–100. The color scale matches the upper panel of Figure fig:appendix_entropy_training.
Additional Results
Training dynamics against RLDF.
The lower panels of Figure fig:appendix_entropy_training compare the training dynamics of DTCA and RLDF over 500 updates. The figure reports response reward, policy entropy, and mean KL divergence, with curves smoothed by a trailing 15-update mean. A higher response reward indicates that more sampled responses are correct.
Figure 7. Additional entropy and training-dynamics analyses. Top: current-commit token entropy changes in DTCA without entropy calibration, using the same early-training baseline, 20-update bins, and color scale as Figure 6. Bottom: DTCA versus RLDF in (a) response reward, (b) policy entropy, and (c) mean KL divergence, using trailing 15-update means.
The two methods start with similar reward trajectories, with average rewards of 0.464 for DTCA and 0.467 for RLDF over updates 1–100. Over updates 401–500, DTCA achieves a higher average reward of 0.537, compared with 0.482 for RLDF. During the same period, DTCA maintains higher policy entropy (2.00 vs. 1.78 nats) and lower mean KL divergence (0.36 vs. 0.89). These results show that DTCA and RLDF follow different optimization trajectories, with DTCA achieving higher reward while maintaining a smaller deviation from the reference policy.
The observed differences are consistent with DTCA’s design, where contribution-based credit assignment provides token-dependent learning signals and entropy-based calibration adjusts updates according to entropy changes. This comparison reflects the combined effect of DTCA components and does not isolate individual module contributions.
Full POS entropy analysis.
Figure 8 shows the nine POS categories most often retained as training targets in DTCA. Together with Figure 9, it extends Figure 5 to all 18 POS categories. These curves cover all retained targets at selected states, rather than only current-commit tokens. With run-specific early baselines, positive values indicate entropy increases, and curves compare changes rather than absolute levels.
Figure 8. Predictive entropy changes for DTCA’s nine most frequent target POS categories. Blue solid: DTCA. Orange dashed: without entropy calibration. Curves show trailing 25-update token-weighted means relative to each run’s POS-specific baseline over updates 1–100.
The frequent categories show different trends. Over updates 401–500, NUM is 0.051 nats above its early baseline in DTCA, compared with 0.209 nats without calibration. For X, the shifts are 0.447 and 0.184 nats. SPACE, ADJ, and VERB generally move below their early baselines in both runs. Calibration does not uniformly lower entropy across POS.
Figure 9. Predictive entropy changes for the nine remaining POS categories, ordered by their frequency among DTCA’s retained targets. Colors, line styles, baseline, and smoothing match Figure 8.
The remaining categories also differ. Over updates 401–500, SYM falls by 0.192 nats from its early baseline in DTCA and by 0.014 nats without calibration. AUX instead rises by 0.072 nats in DTCA and falls by 0.133 nats without calibration. INTJ fluctuates sharply, warranting caution in interpreting its trend.
Entropy-flow calibration.
Figure 8 and Figure 9 summarize POS-level entropy trajectories over training. These are within-run trends rather than changes in matched tokens, since the models visit different states and retain different targets. Figure 10 complements these trajectories by examining the predicted entropy changes induced by individual policy updates, separated into entropy-increasing and entropy-decreasing parts.
Figure 10. Predicted entropy-increasing (solid) and entropy-decreasing (dashed) components for DTCA (blue) and DTCA without entropy calibration (orange). Curves use trailing 25-update means. The vertical axis is broken, with values in nats.
Solid curves show the total predicted increase, while dashed curves show the total predicted decrease. The broken vertical axis keeps both signs visible despite their similar magnitudes. Equal magnitudes imply a near-zero net predicted change.
DTCA keeps the two parts closer together. Over updates 1–100, its average net predicted change is nats, compared with nats without calibration. Over updates 401–500, the corresponding values are and nats. Thus, the predicted increase and decrease nearly cancel under DTCA in both windows. These values are first-order estimates based on the training signals, not direct measurements of entropy after a parameter update.
Limitations
While DTCA provides an effective approach for fine-grained credit assignment in diffusion language model reinforcement learning, several aspects remain open for future exploration. DTCA uses POS categories to aggregate token-level confidence improvements and obtain stable contribution estimates. This design relies on the quality of POS alignment and annotation, and more accurate token-level labeling strategies may further improve the estimation process. Moreover, DTCA currently defines token contribution based on the confidence improvement of remaining masked positions after token commitment. Developing more principled theoretical formulations for contribution estimation in diffusion language models remains an interesting direction. Finally, our experiments focus on mathematical reasoning with LLaDA-8B-Instruct. Applying DTCA to other tasks, such as code generation, may require adapting the token grouping strategy to different token distributions and generation patterns. Further evaluation on diverse diffusion language model architectures is also needed to understand the general applicability of DTCA.
References
- [1]Zhao, S., Gupta, D., Zheng, Q., & Grover, A. (2025). d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning. Advances in Neural Information Processing Systems, 38, Main Conference, 56729–56762. https://doi.org/10.52202/085713-1898
- [2]Oba, D., Furuta, H., & Okazaki, N. (2026). Diffusion-State Policy Optimization for Masked Diffusion Language Models. https://arxiv.org/abs/2602.06462
- [3]He, Q., Chen, H., Guo, Y., Zhu, H., Fung, Y. R., & Zhou, B. (2026). Reinforcement Learning from Denoising Feedback. https://arxiv.org/abs/2605.25638
- [4]Xu, H., Zhao, S., Wu, X., & Luu, A. T. (2026). Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization. https://arxiv.org/abs/2605.11491
- [5]Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., & LI, C. (2025). Large Language Diffusion Models. Advances in Neural Information Processing Systems, 38, Main Conference, 50608–50646. https://doi.org/10.52202/085713-1689
- [6]Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., & Kuleshov, V. (2024). Simple and Effective Masked Diffusion Language Models. Advances in Neural Information Processing Systems, 37, 130136–130184. https://doi.org/10.52202/079017-4135
- [7]Monsefi, A. K., Culver, D., Bhendawade, N., Boominathan, L., Ciosici, M. R., Zhang, Y., & Belousova, I. (2026). DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models. https://arxiv.org/abs/2605.16342
- [8]Raajesh, H., Shah, K., Klivans, A., & Krähenbühl, P. (2026). Mask-Aware Policy Gradients for Diffusion Language Models. https://arxiv.org/abs/2607.15200
- [9]Xie, S., Kong, L., Song, X., Dong, X., Chen, G., Xing, E. P., & Zhang, K. (2026). Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 42703–42720. https://doi.org/10.18653/v1/2026.acl-long.1978
- [10]Austin, J., Johnson, D. D., Ho, J., Tarlow, D., & van den Berg, R. (2021). Structured Denoising Diffusion Models in Discrete State-Spaces. Advances in Neural Information Processing Systems, 34, 17981–17993. https://proceedings.neurips.cc/paper_files/paper/2021/file/958c530554f78bcd8e97125b70e6973d-Paper.pdfproceedings.neurips.cc/paper_files/paper/2021/file/958c530554f78bcd8e97125b70e6973d-Paper.pdf
- [11]Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., & Hashimoto, T. B. (2022). Diffusion-LM Improves Controllable Text Generation. Advances in Neural Information Processing Systems, 35, 4328–4343. https://doi.org/10.52202/068431-0313
- [12]Lou, A., Meng, C., & Ermon, S. (2024). Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. https://arxiv.org/abs/2310.16834
- [13]Shi, J., Han, K., Wang, Z., Doucet, A., & Titsias, M. (2024). Simplified and Generalized Masked Diffusion for Discrete Data. Advances in Neural Information Processing Systems, 37, 103131–103167. https://doi.org/10.52202/079017-3277
- [14]Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., & Li, C. (2025). Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data. International Conference on Learning Representations, 2025, 64972–65009. https://proceedings.iclr.cc/paper_files/paper/2025/file/a365e37c18fb91af547a2f0012a89e98-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2025/file/a365e37c18fb91af547a2f0012a89e98-Paper-Conference.pdf
- [15]Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., & Kong, L. (2025). Dream 7B: Diffusion Large Language Models. https://arxiv.org/abs/2508.15487
- [16]Gong, S., Zhang, R., Zheng, H., Gu, J., Jaitly, N., Kong, L., & Zhang, Y. (2026). DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation. International Conference on Learning Representations, 2026, 15723–15750. https://proceedings.iclr.cc/paper_files/paper/2026/file/1a5b00e59818e4be1c89855a8b921903-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2026/file/1a5b00e59818e4be1c89855a8b921903-Paper-Conference.pdf
- [17]Ou, J., Han, J., Xu, M., Xu, S., Xie, J., Ermon, S., Wu, Y., & Li, C. (2026). Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective. International Conference on Learning Representations, 2026, 4941–4966. https://proceedings.iclr.cc/paper_files/paper/2026/file/0907335ecf28faf15be54485dbcbe70e-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2026/file/0907335ecf28faf15be54485dbcbe70e-Paper-Conference.pdf
- [18]Rojas, K., Lin, J., Rasul, K., Schneider, A., Nevmyvaka, Y., Tao, M., & Deng, W. (2026). Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization. International Conference on Learning Representations, 2026, 1938–1973. https://proceedings.iclr.cc/paper_files/paper/2026/file/0428b72cab14bc38db5b2bad183e1987-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2026/file/0428b72cab14bc38db5b2bad183e1987-Paper-Conference.pdf
- [19]Zhong, J., Wang, K., Ding, D., Feng, Z., Bai, H., Xiang, Y., Sun, J., & Xu, Q. (2026). Stabilizing Reinforcement Learning for Diffusion Language Models. https://arxiv.org/abs/2603.06743
- [20]Jiang, H., Feng, H., Abbeel, P., Jiao, J., Kanazawa, A., & Haghtalab, N. (2026). DiPOD: Diffusion Policy Optimization without Drifting Apart. https://arxiv.org/abs/2606.13795
- [21]Wang, Y., Yang, L., Li, B., Tian, Y., Shen, K., & Wang, M. (2025). Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models. https://arxiv.org/abs/2509.06949
- [22]Li, J., Li, L., Chang, T.-W., Kuang, K., Chen, L., Zhou, J., & Yang, C. (2025). RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 4993–5022. https://doi.org/10.18653/v1/2025.emnlp-main.252
- [23]Cui, G., Yuan, L., Wang, Z., Wang, H., Zhang, Y., Chen, J., Li, W., He, B., Fan, Y., Yu, T., Xu, Q., Chen, W., Yuan, J., Chen, H., Zhang, K., Lv, X., Wang, S., Yao, Y., Han, X., … Ding, N. (2025). Process Reinforcement through Implicit Rewards. https://arxiv.org/abs/2502.01456
- [24]Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., & Roux, N. L. (2025). VinePPO: Refining Credit Assignment in RL Training of LLMs. https://arxiv.org/abs/2410.01679
- [25]Li, S., Kallidromitis, K., Gokul, A., Kyuragi, Y., & Grover, A. (2026). Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization. https://arxiv.org/abs/2605.29198
- [26]Zhang, K., Wu, W., & Lin, Y. (2026). DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards. https://arxiv.org/abs/2605.21467
- [27]Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X.-H., Yang, J., Zhang, Z., Liu, Y., Yang, A., Zhao, A., Yue, Y., Song, S., Yu, B., Huang, G., & Lin, J. (2025). Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. Advances in Neural Information Processing Systems, 38, Main Conference, 115452–115486. https://doi.org/10.52202/085713-3850
- [28]Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. https://arxiv.org/abs/2103.03874
- [29]Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., & Cobbe, K. (2024). Let s Verify Step by Step. International Conference on Learning Representations, 2024, 39578–39601. https://proceedings.iclr.cc/paper_files/paper/2024/file/aca97732e30bcf1303bc22ac3924fd16-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2024/file/aca97732e30bcf1303bc22ac3924fd16-Paper-Conference.pdf
- [30]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). Training Verifiers to Solve Math Word Problems. https://arxiv.org/abs/2110.14168
- [31]de Marneffe, M.-C., Manning, C. D., Nivre, J., & Zeman, D. (2021). Universal Dependencies. Computational Linguistics, 47(2), 255–308. https://doi.org/10.1162/coli_a_00402
- [32]Tang, X., Dolga, R., Yoon, S., & Bogunovic, I. (2026). wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models. International Conference on Learning Representations, 2026, 150357–150385. https://proceedings.iclr.cc/paper_files/paper/2026/file/f381114cf5aba4e45552869863deaaa7-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2026/file/f381114cf5aba4e45552869863deaaa7-Paper-Conference.pdf
- [33]Zhu, F., Wang, R., Nie, S., Zhang, X., Wu, C., Zhou, J., Lin, Y., Wen, J.-R., & Li, C. (2026). LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 11425–11460. https://doi.org/10.18653/v1/2026.acl-long.524
- [34]Huang, Z., Chen, Z., Wang, Z., Li, T., & Qi, G.-J. (2025). Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models. Advances in Neural Information Processing Systems, 38, Main Conference, 152677–152710. https://doi.org/10.52202/085713-5105
- [35]Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., & Higgins, I. (2022). Solving math word problems with process- and outcome-based feedback. https://arxiv.org/abs/2211.14275
- [36]Tan, H., Wang, Z., Pan, J., Lin, J., Wang, H., Wu, Y., Chen, T., Zheng, Z., Tang, Z., & Yang, H. (2026). GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy. https://arxiv.org/abs/2508.04349
- [37]Wu, C., Zhang, H., Xue, S., Liu, Z., Diao, S., Zhu, L., Luo, P., Han, S., & Xie, E. (2026). Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding. International Conference on Learning Representations, 2026, 57027–57051. https://proceedings.iclr.cc/paper_files/paper/2026/file/5d8d4e6061c3ba96c240b7fa1ae3471d-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2026/file/5d8d4e6061c3ba96c240b7fa1ae3471d-Paper-Conference.pdf
- [38]Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., & Sui, Z. (2024). Math-shepherd: Verify and reinforce llms step-by-step without human annotations. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9426–9439.
- [39]Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081), 633–638.
- [40]Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., & Guo, D. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. https://arxiv.org/abs/2402.03300
- [41]Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744. https://doi.org/10.52202/068431-2011
- [42]Labs, I., Khanna, S., Kharbanda, S., Li, S., Varma, H., Wang, E., Birnbaum, S., Luo, Z., Miraoui, Y., Palrecha, A., Ermon, S., Grover, A., & Kuleshov, V. (2025). Mercury: Ultra-Fast Language Models Based on Diffusion. https://arxiv.org/abs/2506.17298
- [43]Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. https://arxiv.org/abs/1707.06347
- [44]Ye, J., Gong, S., Chen, L., Zheng, L., Gao, J., Shi, H., Wu, C., Jiang, X., Li, Z., Bi, W., & Kong, L. (2024). Diffusion of Thought: Chain-of-Thought Reasoning in Diffusion Language Models. Advances in Neural Information Processing Systems, 37, 105345–105374. https://doi.org/10.52202/079017-3343
- [45]He, H., Renz, K., Cao, Y., & Geiger, A. (2025). MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models. https://arxiv.org/abs/2508.13148
- [46]Wang, C., Rashidinejad, P., Su, A. (DiJia) ., Jiang, S., Wang, S., Zhao, S., Zhou, C., Shen, S., Chen, F., Jaakkola, T., Tian, Y., & Liu, B. (2026). SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models. International Conference on Learning Representations, 2026, 24623–24660. https://proceedings.iclr.cc/paper_files/paper/2026/file/2a2de7a4dcda5c6aac2db8cd771e6a47-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2026/file/2a2de7a4dcda5c6aac2db8cd771e6a47-Paper-Conference.pdf
- [47]Zhao, H., Liang, D., Tang, W., Yao, D., & Kallus, N. (2026). DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning. https://arxiv.org/abs/2510.02212
- [48]Yang, J., Chen, G., Hu, X., & Shao, J. (2025). Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Step. https://arxiv.org/abs/2509.23924
- [49]Ni, Z., Wang, S., Yue, Y., Yu, T., Zhao, W., Hua, Y., Chen, T., Song, J., Yu, C., Zheng, B., & Huang, G. (2026). The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models. https://arxiv.org/abs/2601.15165
- [50]Cheng, D., Huang, S., Zhu, X., Dai, B., Zhao, X., Zhang, Z., & Wei, F. (2026). Reasoning with exploration: An entropy perspective. Proceedings of the AAAI Conference on Artificial Intelligence, 40(36), 30377–30385.
- [51]Ye, J., Gao, J., Gong, S., Zheng, L., Jiang, X., Li, Z., & Kong, L. (2025). Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning. International Conference on Learning Representations, 2025, 77875–77898. https://proceedings.iclr.cc/paper_files/paper/2025/file/c1b3d1e2cf53bb28cabd801bd58b3521-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2025/file/c1b3d1e2cf53bb28cabd801bd58b3521-Paper-Conference.pdf
- [52]Wu, C., Zhang, H., Xue, S., Diao, S., Fu, Y., Liu, Z., Molchanov, P., Luo, P., Han, S., & Xie, E. (2026). Fast-dLLM v2: Efficient Block-Diffusion LLM. International Conference on Learning Representations, 2026, 128353–128370. https://proceedings.iclr.cc/paper_files/paper/2026/file/d0865cbe51d35ec322f9af9db7806fc7-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2026/file/d0865cbe51d35ec322f9af9db7806fc7-Paper-Conference.pdf
- [53]Xie, Z., Ye, J., Zheng, L., Gao, J., Dong, J., Wu, Z., Zhao, X., Gong, S., Jiang, X., Li, Z., & Kong, L. (2025). Dream-Coder 7B: An Open Diffusion Language Model for Code. https://arxiv.org/abs/2509.01142
- [54]Song, Y., Zhang, Z., Luo, C., Gao, P., Xia, F., Luo, H., Li, Z., Yang, Y., Yu, H., Qu, X., Fu, Y., Su, J., Zhang, G., Huang, W., Wang, M., Yan, L., Jia, X., Liu, J., Ma, W.-Y., … Zhou, H. (2025). Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference. https://arxiv.org/abs/2508.02193
- [55]Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., & Welling, M. (2021). Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions. Advances in Neural Information Processing Systems, 34, 12454–12465. https://proceedings.neurips.cc/paper_files/paper/2021/file/67d96d458abdef21792e6d8e590244e7-Paper.pdfproceedings.neurips.cc/paper_files/paper/2021/file/67d96d458abdef21792e6d8e590244e7-Paper.pdf
- [56]Dieleman, S., Sartran, L., Roshannai, A., Savinov, N., Ganin, Y., Richemond, P. H., Doucet, A., Strudel, R., Dyer, C., Durkan, C., Hawthorne, C., Leblond, R., Grathwohl, W., & Adler, J. (2022). Continuous diffusion for categorical data. https://arxiv.org/abs/2211.15089
- [57]Gulrajani, I., & Hashimoto, T. B. (2023). Likelihood-Based Diffusion Language Models. Advances in Neural Information Processing Systems, 36, 16693–16715. https://doi.org/10.52202/075280-0730
- [58]Yu, R., Li, Q., & Wang, X. (2025). Discrete Diffusion in Large Language and Multimodal Models: A Survey. https://arxiv.org/abs/2506.13759
- [59]Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems, 33, 6840–6851. https://proceedings.neurips.cc/paper_files/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdfproceedings.neurips.cc/paper_files/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf
- [60]Gao, Z., Luo, H., Chen, L., Liu, J. K., Tao, R., Zhou, J., & Dai, B. (2025). What Makes Diffusion Language Models Super Data Learners?. https://arxiv.org/abs/2510.04071
- [61]Ni, J., Liu, Q., Dou, L., Du, C., Wang, Z., Yan, H., Pang, T., & Shieh, M. Q. (2025). Diffusion Language Models are Super Data Learners. https://arxiv.org/abs/2511.03276
- [62]Prabhudesai, M., Wu, M., Zadeh, A., Fragkiadaki, K., & Pathak, D. (2025). Diffusion Beats Autoregressive in Data-Constrained Settings. Advances in Neural Information Processing Systems, 38, Main Conference, 10581–10606. https://doi.org/10.52202/085713-0354
- [63]Zekri, O., & Boulle, N. (2025). Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods. Advances in Neural Information Processing Systems, 38, Main Conference, 152868–152906. https://doi.org/10.52202/085713-5112
- [64]Schulman, J. (2020). Approximating KL Divergence. In http://joschu.net/blog/kl-approx.html. http://joschu.net/blog/kl-approx.htmljoschu.net/blog/kl-approx.html
- [65]Google DeepMind. (2025). Gemini Diffusion. In https://deepmind.google/models/gemini-diffusion. https://deepmind.google/models/gemini-diffusiondeepmind.google/models/gemini-diffusion










