WorksComputer Science
Empowering Masked Diffusion Models to Self-Correct with Leave-One-Out Transformers
Abstract
Masked diffusion models (MDMs) have emerged as a promising paradigm for language modeling, enabling parallel token generation with competitive performance. Yet standard MDMs suffer from two fundamental limitations: (1) inability to revise tokens, and (2) sparse training signal. We revisit hollow transformers, which address both limitations by predicting each token from its surrounding context, independently of the input at that position. We introduce the Leave-One-Out Transformer (LOOT), a more efficient and expressive architecture that advances hollow transformers through two complementary advances. First, we redesign the merging mechanism to enable frequent interaction between left and right contexts, substantially improving predictive performance over prior hollow transformers. Second, we derive a lower-variance estimator of the MDM NELBO that supervises all tokens and improves validation perplexity. Together, these advances yield a competitive standalone diffusion language model capable of effective self-correction during inference. On LM1B, LOOT matches the strongest dense-supervision MDM baseline in validation perplexity. On OpenWebText, LOOT achieves higher MAUVE and better generation quality-diversity tradeoffs than prior remasking methods, using fewer forward evaluations and fewer inference FLOPs.
Introduction
Diffusion language models (DLMs) offer a fundamentally different approach to text generation from autoregressive (AR) models. Rather than generating tokens sequentially from left to right, DLMs iteratively denoise entire sequences, allowing multiple tokens to be predicted in parallel. Masked diffusion models (MDMs) have emerged as a leading approach to diffusion for language, with recent models scaling up to 100B parameters [4]. Yet standard MDMs lose a defining appeal of diffusion: the ability to iteratively refine earlier predictions. MDMs generate text by progressively replacing [MASK] tokens with model predictions. Once a token is unmasked, it remains fixed, so errors cannot be corrected and propagate through the rest of the generation process. The standard MDM training objective is similarly restrictive. The training loss is computed only on masked positions, leaving visible tokens without direct supervision. Thus, standard MDMs face two limitations we aim to address: (1) inability to self-correct during inference, and (2) sparse training signal.
Addressing both limitations is challenging because standard MDMs use bidirectional transformers, allowing each prediction to depend on all input tokens. Consequently, we cannot extend the training loss to visible tokens, since the model could simply copy the correct answer rather than learning to predict it from context. Instead, we overcome this obstacle by modeling the conditional distribution at every position , ensuring that each prediction is independent of its corresponding input token. This leave-one-out property (Figure 1) has two key consequences: (1) every token, masked or visible, becomes a genuine prediction task over possible replacements that can be supervised during training, and (2) previously generated tokens can be revised during inference. Hollow transformers [31, 35] enforce this property by design, but have struggled to achieve competitive performance as standalone diffusion language models. We revisit hollow transformers and introduce two complementary advances in architecture and training to bridge this performance gap.
Figure 1. Dependencies of output tokens on input tokens in causal, bidirectional, and leave-one-out models. Unlike standard bidirectional MDMs, the output of LOOT at index does not depend on the input at index , enabling loss optimization even on unmasked tokens.
To this end, we introduce the Leave-One-Out Transformer (LOOT), a more expressive and computationally efficient hollow architecture. First, we redesign the merging mechanism of hollow transformers to enable more frequent interaction between left and right contexts (§ 3.2). Our ablations show that more frequent merging substantially improves language modeling performance, suggesting that repeated interaction allows the model to progressively refine its representations of the surrounding context. LOOT makes frequent merging efficient by avoiding the expanded hidden representations used in prior architectures, enabling full parameter sharing and sliding-window attention compatible with standard FlashAttention kernels. At the largest model size we evaluate, this allows LOOT to merge left and right contexts at every layer, whereas HollowDiff [35] can afford only two merges at comparable FLOPs. Second, we exploit the leave-one-out property to derive a new, lower-variance, all-position estimator of the MDM NELBO that improves the training of hollow transformers (§ 3.1).
Together, these advances yield better validation perplexity than prior hollow transformers at matched parameter counts and FLOPs, while establishing LOOT as a competitive standalone diffusion language model capable of effective self-correction. On OpenWebText, LOOT surpasses standard MDMs and ReMDM [33] in generation quality, as measured by generative-perplexity–entropy frontiers and MAUVE, while requiring fewer sampling steps than ReMDM.
Background
Diffusion models learn to reverse a pre-defined noising process that gradually transforms data into noise. D3PM extended diffusion to discrete data [3], and later work extended discrete diffusion to continuous time [5, 15, 24]. Two common choices of noising process are absorbing-state diffusion, which replaces token with a special [MASK] token, and uniform diffusion, which corrupts tokens by replacing them with random tokens from the vocabulary. Uniform diffusion allows previously generated tokens to be revised, but absorbing-state diffusion has proved particularly effective for language modeling, owing in part to its simple training objective and empirical success [24, 34, 17].
Masked Diffusion Models
Consider a sequence of tokens drawn from a vocabulary . Masked diffusion defines a forward noising process that independently replaces each token with [MASK] with probability , where and is a monotonically decreasing noise schedule satisfying and . Throughout this paper, we use to denote the set of masked positions at time , and to denote the resulting partially masked sequence. The forward process thus interpolates between clean data at and a fully masked sequence at . Crucially, once a token is masked in the forward process, it remains masked.
Training. The MDM training objective is a continuous-time NELBO that reduces to a weighted cross-entropy over masked positions [24, 30]:
where and . Since the optimal is independent of given the masked sequence [36, 18, 13], we follow standard practice and drop explicit time conditioning, writing the denoiser as . Importantly, the MDM loss in Eq. (1) is evaluated only at masked positions, leaving visible tokens without direct supervision.
Sampling. Generation starts from a fully masked sequence and progressively reveals tokens. Given a partially masked sequence , MDMs predict a distribution over tokens at each masked position. In practice, multiple positions can be unmasked in parallel through -leaping [15, 5], allowing generation in fewer steps than the sequence length. However, tokens that are unmasked in the same step are sampled independently conditioned on , and may be mutually inconsistent [14]. Previously unmasked tokens cannot be revised, so errors may propagate through subsequent generation steps.
Hollow Transformers
Hollow transformers [31, 35, 19] offer an architectural means of addressing both limitations of MDMs. They satisfy the leave-one-out property by design: the prediction at position depends only on and not on the input token . A hollow transformer therefore computes the conditional distributions at every position in a single forward pass. As a result, both masked and visible tokens can be supervised during training, and previously generated tokens can be revised during inference.
To achieve this property, Sun et al. [31] use two causal streams: a forward stream whose representation at position depends only on earlier tokens , and a backward stream whose representation at position depends only on later tokens . To recover a bidirectional representation at position that satisfies the leave-one-out property, hollow transformers merge the representations of left context and right context . Sun et al. [31] only merge at the final layer, while HollowDiff [35] introduces a third output stream that merges at intermediate layers.
Despite these advances, hollow transformers have struggled to match the performance of bidirectional transformers. For one, HollowDiff reports weaker performance than a standard MDM and achieves its best results by deploying the hollow model as a corrector for a separate MDM predictor model. This predictor-corrector setup incurs additional training and inference costs. We aim to bridge this gap through two complementary improvements. First, we redesign the merging mechanism to enable efficient, frequent interaction between left and right contexts. Our ablations show that merging at every layer substantially improves language modeling performance, suggesting that repeated interaction allows the model to progressively refine its representations of the surrounding context. Second, we derive a lower-variance estimator of the MDM NELBO that improves training efficiency.
Leave-One-Out Language Modeling
We introduce the Leave-One-Out Transformer (LOOT), a hollow architecture that computes the conditional distributions at every position in a single forward pass (Figure 2). By predicting each token independently of the input at that position, LOOT enables supervision of both masked and visible tokens during training, as well as self-correction during inference. We address the limitations of prior hollow transformers through two complementary advances: a new, lower-variance estimator of the MDM NELBO that supervises every position (§ 3.1), and a redesigned merging mechanism that efficiently integrates left and right contexts at every layer (§ 3.2 and § 3.3). Finally, we show how LOOT’s leave-one-out conditionals enable classical sampling techniques, including Gibbs-style and Metropolis–Hastings-style updates, allowing effective self-correction during generation (§ 3.4).
Figure 2. Overview of the Leave-One-Out Transformer (LOOT). Our goal is to predict every token from its surrounding context , independently of the input token at position . LOOT maintains three streams: a left-to-right stream , a right-to-left stream , and an output stream that merges the left context and right context at every layer. A stack of LOOT layers followed by a final merge produces the conditionals at every position in a single forward pass. This enables supervision at every token during training and self-correction during inference.
Training Objective
We exploit the leave-one-out property to derive an all-position estimator of the MDM NELBO. Unlike standard MDMs, which supervise only masked positions, LOOT can predict every token from its surrounding context. Formally, we require the following property.
Definition 1 (Leave-one-out property). A denoiser satisfies the leave-one-out (LOO) property if its prediction at position depends on the input only through . Equivalently, for any two inputs and differing only at position ,
Crucially, the LOO property ensures that the model’s prediction at position is the same whether the input at that position is masked or visible. For any mask set and position , the following holds
This invariance allows us to analytically marginalize the masking probability at each position in the standard MDM NELBO, yielding the following objective:
The full derivation is provided in Appendix § 9. This objective is a valid MDM likelihood bound whenever satisfies the LOO property. Every token contributes to the LOOT loss during training, yielding a lower-variance estimator of the MDM NELBO. Under the standard log-linear noise schedule used by MDLM [24] and LLaDA [17], the LOOT NELBO weights reduce to a constant for all . This eliminates the divergence of the standard MDLM NELBO weight , which diverges as .
Relation to dense MDM objectives. HollowDiff [35] and concurrently PGM [8] proposed the half-and-half loss, , which averages the MDM losses on masked and unmasked positions. While this provides supervision at every position, its NELBO weights diverge at both endpoints and . Our objective is a Rao-Blackwellization of the half-and-half loss, analytically marginalizing the masking probability at each position and eliminating the weight divergence (Appendix § 9.§ 2). In our ablations, we show that our proposed NELBO improves the validation perplexity on LMB1 by 3.1 for a LOOT model compared to the half-and-half loss (Table 5).
| Model | Merge | PPL (↓) |
| DiT + MDLM loss | n/a | 39.80 |
| Hollow transformer [31] | final layer | 41.07 |
| HollowDiff* () | every 3 layers | 37.91 |
| HollowDiff* () | every 2 layers | 37.23 |
| LOOT () | every layer | 36.23 |
| LOOT + half-and-half loss | every layer | 39.35 |
Table 5. Hollow transformer ablations on LM1B. Validation perplexity (↓) at , batch size 256, after 200K training steps. Hollow variants use the LOOT NELBO unless stated. ∗LOOT with merges every layers, following HollowDiff [35], isolating merging frequency.
The Leave-One-Out Transformer
Next, we propose a model architecture that efficiently evaluates at every position in a single forward pass, while preserving the LOO property (Def. 1) across layers.
Challenge. A single attention layer with mask , ones everywhere except on the diagonal, satisfies Definition 1, but the property is lost under composition: a neighboring position attends to in one layer, and position attends to ’s updated representation in the next, routing back into its own prediction. Hollow transformers resolve this with two causal streams (§ 2.2) which preserve the LOO property under composition.
Three Data Streams. LOOT represents each sequence with three parallel streams of hidden states, illustrated in Figure 2. (1) Forward stream : encodes left context. At each layer, position attends to positions via a standard left-to-right causal mask. (2) Backward stream : encodes right context, mirrored: position attends to positions via a right-to-left causal mask. These are the two causal transformers of Sun et al. [31]. (3) Output stream : computes the leave-one-out conditional. At each layer, position attends to forward keys and backward keys , merging bidirectional context that excludes . This third stream follows HollowDiff [35], with one important difference. LOOT merges at every layer rather than once or every few layers: bidirectional context accumulates with depth because each layer’s contribution to is added to the residual stream, and perplexity improves monotonically with merge frequency (Table 5).
Initialization. The forward and backward streams are initialized using the partially masked sequence , while the output stream is initialized to the [MASK] token embedding to ensure that predictions satisfy the LOO property. The output stream could alternatively be initialized using a separate learned embedding, which we did not explore in this work.
An Efficient Implementation of Loot
Parameterization. Aside from the attention masks, LOOT follows the standard DiT-style MDM architecture [20, 24], with adaptive layer norms, MLPs, and residual connections (full details in Appendix § 10.§ 1). The three streams share all attention projection matrices (, , , ) at each layer, so the parameter count of LOOT matches that of a standard DiT of the same depth and width. The forward and backward streams additionally share layer norms; the output stream has its own. This minor parameter overhead gives the output stream, which solves a structurally different task than the directional streams, additional expressivity at low cost.
Efficient Merging with FlashAttention. The merge block’s attention pattern admits a clean reformulation as sliding-window attention. By concatenating the backward and forward streams along the sequence dimension to form the key-value tensor , each output query attends to a contiguous length- window of the sequence Figure 3. This reduces the merge to a standard sliding-window attention pattern, executable in a single FlashAttention call [7]. The forward and backward streams have identical mask structure (causal in their respective directions) and are batched into a single FlashAttention call, so each LOOT layer requires only two FlashAttention calls in total, with no custom kernels.
Figure 3. Efficient Merger using sliding-window attention.
Computational Overhead. Counting matmul and attention FLOPs at , a LOOT forward pass costs that of a standard MDM of the same size (Appendix § 14). This is well below the naive of three streams: LOOT doubles rather than triples the attention work, its final merge block is cheaper than a full layer, and the vocabulary head is shared. With torch.compile, the forward+backward cost ratio ranges from at small batch sizes, where both models are kernel-launch-bound, to at large batch sizes and , where both are compute-bound. This per-pass overhead is repaid at inference, where a single LOOT pass returns all leave-one-out conditionals (§ 3.4).
Self-correction and sampling
A trained LOOT model predicts the full set of leave-one-out conditionals in a single forward pass. These conditionals provide a distribution over replacements at every position, enabling the use of classical sampling algorithms, such as Gibbs- and Metropolis-style methods. We outline some of the self-correction procedures, which enable efficient and effective self-correction during inference.
Two-stage sampling. We generate a sample in two stages. First, we produce an initial sequence using standard unmasking steps. Second, we run self-correction steps: at each step, one LOOT forward pass returns for every position , and a correction policy selects positions to edit and resamples them. The total number of forward passes is .
Correction policies. LOOT’s leave-one-out conditionals admit several principled correction policies; we summarize the main ones here and defer full details to Appendix § 10.§ 4.
The simplest is blocked Gibbs sampling: partition positions into groups, and at each correction step, resample one entire group in parallel from .
A second policy is Metropolis-Hastings, which proposes a replacement and accepts it with probability , comparing the model’s preference for the proposed token against its preference for the current one. If the proposal is more compatible with the surrounding context than the current token, it is accepted with probability one; if less compatible, the edit is accepted only stochastically, preventing the sampler from overwriting good tokens just because a sample happened to land on something different.
Finally, improvement-based policies score each position by the log-likelihood gap between the best alternative replacement and the current token. Positions with are locally improvable: under the model, some other token fits the surrounding context better than the current one. Greedy variants edit the position with the largest ; stochastic variants sample positions with probability proportional to .
Comparison to remasking. Remasking-based methods such as ReMDM [33] require multiple forward passes per correction (Figure 4): one to remask a token, and another to resample. Worse, choosing which token to remask requires the up-to-date conditionals , which a standard MDM cannot evaluate efficiently: recovering all would take forward passes, masking each unmasked position individually. ReMDM falls back to heuristics like confidence at the time of unmasking, which grow stale as context fills in. LOOT returns all conditionals in a single forward pass, so each correction step is one model evaluation regardless of how many tokens are revised.
Figure 4. LOOT enables efficient self-correction.
Experiments
Language Modeling
Setup. We evaluate LOOT against discrete diffusion baselines (Table 1) on two standard language modeling datasets: LM1B [6], with sequence length and 1M training steps, and OpenWebText [10] (OWT), with and 250K training steps. We follow the training setup of Sahoo et al. [26], matching model parameter counts across diffusion baselines within each benchmark. Following standard practice [24], we report diffusion perplexities using the NELBO upper bound on negative log-likelihood. Full training details are provided in Appendix § 10.
| Category | Method | LM1B () Params | LM1B () PPL () | OWT () Params | OWT () PPL () |
| Autoregressive | Transformer + AdaLN [26] | 132M | 21.86 | 162M | 17.78{sup}† |
| Diffusion | SEDD Absorb{sup}‡ [15] | 139M | 32.71{sup}¶ | 170M | 26.81 |
| Diffusion | MDLM{sup}‡ [24] | 139M | 170M | ||
| Diffusion | UDLM{sup}‡ [28] | 139M | 36.71 | 170M | 30.52 |
| Diffusion | Duo{sup}‡ [25] | 139M | 33.68 | 170M | 27.14 |
| Diffusion | PGM{sup}‡ [8] | 142M | 171M | ||
| Ours | LOOT | 139M | 170M |
Table 1.
Likelihood evaluation. On LM1B, LOOT improves over MDLM by 0.78 perplexity and matches PGM [8], the strongest prior dense-supervision MDM baseline, with differences between the two within evaluation-seed variation. On OWT at , LOOT achieves a perplexity of 26.83, compared with 25.20 for MDLM and 26.57 for PGM. Thus, while LOOT is competitive with standard MDMs, a modest performance gap remains at longer sequence lengths.
LOOT self-correction outperforms ReMDM with fewer sampling steps. (a) Quality–diversity frontiers on OpenWebText across sampling temperatures. At 256 and 512 NFEs, LOOT matches or surpasses ReMDM at 2048 NFEs across the frontier, while using and fewer inference FLOPs, respectively. (b) MAUVE () across NFE budgets at matched entropy. LOOT-Blocked-Gibbs at 256 NFEs surpasses ReMDM at 2048 NFEs.
We conduct two ablations to investigate the likelihood gap at longer sequence lengths and evaluate our improvements over prior hollow transformers (Appendix § 10.§ 8).
Sequence length ablations. We train three models on OWT at sequence lengths and , holding all other settings fixed (Table 4): standard MDLM, the LOOT architecture trained with the MDLM loss, and LOOT trained with our all-position objective. Fixing the LOOT architecture, our objective improves perplexity by 2.9 and 2.8 at and , respectively. This consistent improvement allows LOOT to outperform MDLM at on both OWT and LM1B. However, holding the MDLM loss fixed reveals a gap: the leave-one-out architecture incurs a perplexity penalty of 2.2 at , increasing to 4.4 at . Thus, while our objective more than compensates for this penalty at shorter sequences, it cannot fully close the gap at longer sequences. These findings suggest a length-dependent modeling capacity gap between hollow and standard bidirectional transformers. Nevertheless, LOOT achieves better perplexity than Duo [25] and UDLM [28], two diffusion language models that also support native self-correction.
| Model | ||
| MDLM (DiT) | 35.23 | 25.19 |
| LOOT + MDLM loss | 37.40 | 29.63 |
| LOOT + LOOT loss | 34.51 | 26.83 |
Table 4. Sequence length ablations. Validation perplexity (↓) on OpenWebText at (1M training steps) and (250K training steps), with batch size 512. We compare standard MDLM against the LOOT architecture trained with either the standard MDLM loss or our LOOT objective.
Hollow transformer ablations. Table 5 evaluates our architectural and training improvements using shorter LM1B runs. First, more frequent merging of left and right contexts consistently improves perplexity, with per-layer merging achieving the best performance. Second, our LOOT NELBO provides substantial gains independently of merging frequency: it improves perplexity over the half-and-half loss used by prior works [35, 8]. Notably, even the merge-once architecture of Sun et al. [31] trained with our objective approaches the bidirectional DiT baseline (41.07 vs. 39.80), while per-layer merging surpasses it (36.23 vs. 39.80). These results demonstrate that both our objective and architecture contribute to making hollow transformers competitive language models.
Table 1: Validation perplexity (PPL) on LM1B (, 1M steps) and OWT (, 250K steps). MDLM, PGM, and LOOT report mean std over 5 evaluation seeds. {sup}‡Reported in Sahoo et al. [26]. {sup}‡Reproduced from checkpoints provided by Sahoo et al. [24, 25] and Schiff et al. [28] in [26]. {sup}†Trained for 1M OWT steps (not 250K). {sup}¶No sentence packing.
Sampling and Self-correction
Setup. We evaluate unconditional generation on OpenWebText (OWT) at sequence length . We compare LOOT and MDLM trained from scratch for 250K steps on the same data, with global batch size 512 and the same log-linear noise schedule (Appendix § 10). We evaluate MDLM [24], ReMDM-loop [33], and LOOT with and without self-correction. Compute is measured in number of forward evaluations (NFE): for MDLM and ReMDM-loop, the number of reverse denoising steps; for LOOT, , the sum of initial denoising steps and refinement steps.
Evaluation protocol. We follow Pynadath et al. [22] and report the trade-off between generative perplexity (lower is better, a quality measure) and sample entropy (higher is better, a diversity measure). We use GPT-2 Large [23] to evaluate generative perplexity and generate 512 samples of length per configuration for the frontiers. We additionally report MAUVE [21], which captures both generation quality and cross-sample diversity, using 5,000 samples at temperatures matched to the entropy of OWT validation text. Following ReMDM [33], we use nucleus sampling with , but include quality–diversity frontiers for in Appendix § 10.§ 6. The full sampling evaluation details are provided in Appendix § 10.§ 5.
Compute efficiency. Figure 5 highlights our main result: LOOT achieves favorable generation quality with substantially less inference compute. Figure 5(a) shows that LOOT at 256 and 512 NFEs matches or surpasses ReMDM at 2048 NFEs across the quality–diversity frontier. More strikingly, Figure 5(b) shows that LOOT-Blocked-Gibbs at just 256 NFEs achieves a MAUVE score of 0.408, exceeding ReMDM’s 0.348 at 2048 NFEs, with matched sample entropy. These results correspond to fewer forward evaluations and fewer inference FLOPs, accounting for the cost of a LOOT forward pass. At 512 NFEs, LOOT further improves both metrics while requiring fewer inference FLOPs. In wall-clock time, LOOT at 256 NFEs is approximately faster than ReMDM at 2048 NFEs on the same hardware (Table 9; Appendix § 14).
Figure 5. (a) Unequal-compute frontier. (b) MAUVE across NFE budgets.
| Method | Steps (init + corr.) | TFLOPs / sample | Seconds / sample |
| MDLM | 75.0 | 11.76 [11.45, 12.08] | |
| LOOT, no correction | 167.9 | 14.54 [14.38, 14.70] | |
| LOOT-Blocked-Gibbs | 167.9 | 11.46 [11.15, 11.77] | |
| LOOT-Metropolis-Hastings | 167.9 | 10.46 [10.17, 10.75] | |
| ReMDM | 583.3 | 95.54 [87.82, 103.26] |
Table 9. Inference cost per generated sequence at (NVIDIA A6000, bf16). FLOPs are counted matmul and attention FLOPs; seconds are measured end-to-end, with 95% intervals over 3 runs.
Quality–diversity frontier. Figure 6 shows that LOOT with self-correction consistently outperforms ReMDM across the full quality–diversity frontier at every evaluated NFE budget. This demonstrates that LOOT achieves a better tradeoff between generation quality and diversity.
Figure 6. LOOT Self-Correction Improves the Sampling Frontier. Generative perplexity versus sample entropy on OpenWebText across NFE budgets. LOOT self-correction improves the frontier relative to MDLM, ReMDM-loop, and the uncorrected LOOT baseline.
MAUVE. MAUVE confirms these improvements at the distribution level (Table 2). At just 256 NFEs, LOOT-Blocked-Gibbs achieves MAUVE 0.408, surpassing ReMDM’s 0.348 at 2048 NFEs, with matched entropy. Crucially, these gains come from self-correction: LOOT-Baseline, which uses standard MDM ancestral sampling without self-correction, achieves only 0.010 MAUVE at 256 NFEs, even below MDLM (0.014). Applying Blocked Gibbs or Metropolis–Hastings correction to the same model raises MAUVE to 0.408 and 0.312, respectively. However, not all correction strategies perform equally well: Best Improvement reaches only 0.035. These results highlight the importance of the sampling algorithm and suggest substantial room for improvement beyond the three classical correction methods explored here. Full MAUVE results are provided in Appendix § 10.§ 7.
| MAUVE () | Gen. PPL () | Entropy | ||||||||||
| NFE = | 256 | 512 | 1024 | 2048 | 256 | 512 | 1024 | 2048 | 256 | 512 | 1024 | 2048 |
| MDLM | 0.014 | 0.015 | 0.017 | 0.018 | 63.1 | 60.8 | 59.4 | 58.5 | 5.53 | 5.52 | 5.51 | 5.50 |
| ReMDM | 0.040 | 0.091 | 0.242 | 0.348 | 53.9 | 44.9 | 37.6 | 31.2 | 5.54 | 5.50 | 5.45 | 5.40 |
| LOOT-Baseline | 0.010 | 0.010 | 0.012 | 0.012 | 55.4 | 50.8 | 48.8 | 47.6 | 5.47 | 5.45 | 5.43 | 5.42 |
| LOOT-Blocked-Gibbs | 0.408 | 0.549 | 0.580 | 0.602 | 20.1 | 20.0 | 21.1 | 17.5 | 5.40 | 5.45 | 5.51 | 5.46 |
| LOOT-Best-Improvement | 0.035 | 0.072 | 0.201 | 0.401 | 36.3 | 32.3 | 29.2 | 24.3 | 5.44 | 5.48 | 5.52 | 5.48 |
| LOOT-Metropolis-Hastings | 0.312 | 0.366 | 0.432 | 0.478 | 21.2 | 22.8 | 22.1 | 21.8 | 5.43 | 5.49 | 5.47 | 5.47 |
Table 2. Unconditional generation quality on OpenWebText. We compare LOOT sampling methods against MDLM and ReMDM across different numbers of function evaluations (NFEs). LOOT-Blocked-Gibbs achieves better generation quality than ReMDM with fewer NFEs (256 vs. 2048) at matched entropy.
Discussion
Disentangling architecture, training, and self-correction. Our ablations show that the LOOT architecture improves likelihood over prior hollow transformers (Table 5), while the LOOT loss further improves validation perplexity over both the standard MDM loss (Table 4) and the half-and-half loss (Table 5). We further observe that the LOOT architecture enables effective self-correction independently of the training loss (Appendix § 10.9). Even when trained with the standard masked-only MDM loss, LOOT substantially improves generation quality through self-correction, matching or surpassing ReMDM despite performing considerably worse without correction (Table 6). Together, these results establish the distinct contributions of our architecture and loss: the architectural redesign improves hollow language modeling and enables effective self-correction, while the LOOT loss further improves language modeling performance and training efficiency.
| Gen. PPL () | Entropy | ||||||||||||
| NFE = | 64 | 128 | 256 | 512 | 1024 | 2048 | 64 | 128 | 256 | 512 | 1024 | 2048 | |
| MDLM | 49.9 | 43.2 | 39.6 | 37.8 | 37.6 | 37.4 | 5.42 | 5.37 | 5.35 | 5.32 | 5.34 | 5.33 | |
| ReMDM | 56.8 | 40.6 | 33.1 | 27.3 | 22.4 | 19.0 | 5.46 | 5.40 | 5.36 | 5.31 | 5.24 | 5.20 | |
| LOOT + MDM loss | Baseline | 90.6 | 70.6 | 62.3 | 58.2 | 56.9 | 55.6 | 5.55 | 5.51 | 5.49 | 5.47 | 5.46 | 5.45 |
| Blocked Gibbs | 38.1 | 25.9 | 19.9 | 16.7 | 14.5 | 12.9 | 5.53 | 5.44 | 5.36 | 5.32 | 5.26 | 5.21 | |
| LOOT + LOOT loss | Baseline | 91.1 | 65.4 | 55.0 | 49.5 | 47.5 | 47.5 | 5.59 | 5.52 | 5.47 | 5.45 | 5.42 | 5.42 |
| Blocked Gibbs | 47.6 | 28.3 | 19.9 | 16.2 | 14.0 | 12.4 | 5.56 | 5.48 | 5.40 | 5.33 | 5.29 | 5.24 |
Table 6. Self-correction with the standard MDM loss. Generative perplexity () and sample entropy on OpenWebText (), using , top-p = 0.9, and 512 samples per setting. Ancestral sampling uses standard MDM unmasking without correction; Blocked Gibbs allocates half of the NFEs to initial sampling and half to refinement ().
Compatibility with pretrained checkpoints. An important practical question is whether LOOT can leverage existing pretrained models. Starting from an MDM checkpoint pretrained for 200K steps, we compare continuing standard MDM training for 50K steps against converting the checkpoint to LOOT and finetuning with our objective for the same duration (Appendix § 13.§ 1). Both models are trained on the same number of tokens, but the LOOT finetune incurs approximately the per-step FLOPs, resulting in the total training FLOPs. We find that finetuned LOOT retains comparable generation quality without correction, while self-correction yields substantial improvements over continued MDM training. This demonstrates that existing MDM checkpoints can be adapted for self-correcting generation without retraining from scratch.
Hybrid training and inference. Full parameter sharing allows LOOT to recover standard bidirectional attention (Appendix § 13). This opens the possibility of hybrid training and inference, using the cheaper MDM mode for highly masked sequences and switching to leave-one-out prediction for dense supervision and self-correction as more tokens become visible. The causal streams also suggest potential initialization from pretrained autoregressive models, particularly for block diffusion [2].
Limitations and future directions. Despite these improvements, LOOT retains a likelihood gap relative to standard bidirectional MDMs at longer sequence lengths and incurs approximately the FLOPs per forward pass. Moreover, the substantial differences between our three self-correction strategies demonstrate that generation quality depends strongly on the sampling algorithm. Understanding the source of the length-dependent likelihood gap, developing more effective self-correction methods, and exploring hybrid training and inference remain promising directions for future work.
Related Work
Diffusion Language Models. Masked diffusion language models were introduced by D3PM [3] and refined by SEDD [15], MDLM [24], and MD4 [30]. Subsequent work explored hybrid AR–diffusion factorizations [2, 16, 26, 1] and large-scale pretraining [17, 4, 34, 9]. A parallel line of work revisits noising processes that permit token revision during denoising [28, 32, 25]. While these objectives expose self-correction directly through the noise process, they remain empirically harder to optimize in parameter-matched settings [27]. LOOT instead retains the standard absorbing-state objective and introduces self-correction through the architecture.
Hollow transformers and leave-one-out denoisers. Hollow transformers were introduced by Sun et al. [31] to compute the singleton conditionals required by ratio matching in a single forward pass. HollowDiff [35] extended this architecture for diffusion-based correctors, while related constructions have been explored in discrete neural samplers [19]. We discuss these architectures and their limitations in § 2.2. In concurrent work, Gourevitch et al. [11] show that leave-one-out prediction arises naturally in uniform diffusion and can be obtained through a parameterization without architectural changes. In contrast, under masked noising, standard bidirectional transformers receive no supervision for leave-one-out predictions at visible positions. LOOT therefore enforces the property architecturally, allowing every position to be predicted from its surrounding context.
Inference-time self-correction. Several methods seek to address the inability of standard MDMs to revise previously generated tokens. ReMDM [33] enables self-correction through remasking, but relies on heuristics to determine which tokens to remask. Other approaches introduce auxiliary correction objectives that explicitly train models to recognize and correct errors [12, 29], departing from the standard variational likelihood objective. In contrast, LOOT directly models the leave-one-out conditionals at every position in a single forward pass, enabling principled Gibbs-style and Metropolis-style self-correction without heuristic remasking or auxiliary correction losses, while retaining training under the MDM NELBO.
Dense supervision of MDMs. PGM [8] improves training and sampling efficiency by partitioning tokens into masked and unmasked subsets, using restricted attention to prevent information leakage between them. However, PGM, like standard MDMs, cannot efficiently revise previously generated tokens. In contrast, LOOT models the leave-one-out conditionals at every position, enabling both dense supervision during training and self-correction during generation. Our Rao–Blackwellized NELBO estimator further reduces training variance while preserving the standard absorbing-state diffusion objective. We compare directly against PGM in our experiments, and provide a more detailed discussion of the loss in Appendix § 9.§ 2.
Conclusion
We introduce the Leave-One-Out Transformer (LOOT), a more expressive and efficient hollow architecture. By predicting every token from its surrounding context, LOOT admits a lower-variance training objective that supervises all positions, as well as Gibbs-style sampling methods for self-correction. A single LOOT forward pass computes all leave-one-out conditionals, whereas a standard MDM would require separate forward passes to obtain these predictions for visible tokens. Although each LOOT forward pass requires the FLOPs of a standard MDM, it enables high-quality, few-step generation. On OpenWebText, LOOT achieves better generation quality than remasking-based methods with fewer forward evaluations and fewer inference FLOPs. Future work may explore more effective self-correction algorithms and hybrid training and inference schemes that combine the efficiency of standard MDMs with the refinement capabilities of LOOT.
Acknowledgments
We thank Eric Enouen and Linxi Zhao for insightful discussions, help with the figures, and valuable feedback on the manuscript. This work was funded by NSF CAREER Award No. 2046760 and NSF grant IIS-2530143. This material is based on work supported by the AI Research Institutes program supported by the NSF and Intel Corporation under NSF award DMR-2433348.
Derivation of the LOOT Training Objective
This appendix derives the LOOT training objective from the masked diffusion NELBO under the leave-one-out (LOO) property of the denoiser (Definition 1). The argument is direct: under the LOO property, the per-position expectation in the MDM NELBO admits a Rao-Blackwellization that absorbs the weighting and yields a sum over all positions rather than a sum over the random subset . We then discuss the connection to the half-and-half objective used by HollowDiff [35] and PGM [8], which arises as an intermediate form and exhibits known numerical instability.
Setup. Let be a sequence of length over a vocabulary that includes a dedicated [MASK] token. The continuous-time forward process samples a mask at time , where each position is masked independently with probability . We write for the sequence with positions in replaced by [MASK], and assume a time-independent denoiser throughout, following standard practice in MDM [24, 30]. The schedule satisfies and , so .
The standard MDM continuous-time NELBO is
The loss is supported only on positions , and the factor combined with yields a positive loss.
The LOOT NELBO
Under the LOO property (Definition 1), masking position does not change the prediction at that position:
for all . We exploit this invariance to rewrite the MDM NELBO as a sum over all positions and marginalize each position’s own masking event.
Rewriting as a sum over all positions. The masked-position sum in can be expressed using indicator variables:
Substituting into (4) and exchanging summation and expectation gives
Rao–Blackwellization. Fix a position and let . We Rao–Blackwellize the -th summand by conditioning on :
By the LOO property, the prediction at position is unchanged whether that position is masked or visible. Consequently, depends only on and can be pulled out of the inner expectation:
where the final equality follows from the independence of per-position masking.
Cancellation and final form. Substituting into (7), the factor obtained by marginalizing the mask indicator cancels the inverse masking-probability weight:
Applying the LOO property once more, the log-probability is unchanged whether position is masked or visible. We can therefore replace the expectation over with an expectation over the full mask set , yielding
This is the objective used in the main text (Equation (3)). Unlike the standard MDM loss, the LOOT loss supervises every position while preserving the same expected NELBO.
Properties. The LOOT NELBO has three useful properties:
Equality with the MDM NELBO. The Rao–Blackwellization preserves the expected value, so for any denoiser satisfying Definition 1. Therefore, the LOOT objective is a valid likelihood bound.
Lower variance than LMDM. Fix and write and . By the LOO property, is independent of , so every term of the MDM estimator has larger variance than the corresponding LOOT term, at every noise level:
The excess diverges as . Consequently, under the log-linear schedule and with bounded log-probabilities, the variance of the single-sample MDM estimator grows logarithmically as the time-clipping threshold approaches zero whenever the loss does not vanish at low noise, whereas the variance of the LOOT estimator remains bounded.
Simplified NELBO weights. The LOOT estimator is weighted only by , rather than . Under the log-linear schedule , the remaining weight is constant, , eliminating the endpoint-divergent weighting of the standard MDM estimator.
Connection to the Half-and-Half Loss
A natural alternative to Rao–Blackwellization is to retain the masked-position estimator and add a symmetric supervision term for visible positions:
This is the half-and-half objective used by HollowDiff [35]. Concurrently, PGM [8] uses related complementary partition supervision with inverse-noise weighting, but predicts tokens in one partition from the opposite partition rather than satisfying the LOO property.
Like , the half-and-half objective places loss on every position. Under the LOO property, it also equals the standard MDM NELBO in expectation. Each position is masked with probability , so marginalizing its own masking event gives an expected weight of
Thus, applying the same per-position Rao–Blackwellization to the half-and-half estimator also yields the LOOT loss.
Numerical instability. Although the half-and-half and LOOT objectives agree in expectation, the former retains inverse-probability weights that diverge at both ends of the noise schedule. Writing , its per-position weight has variance
By (12), this contributes to each position’s loss variance. Under the log-linear schedule , bounded log-probabilities and nonvanishing expected squared losses near low noise imply logarithmically divergent variance for the complete half-and-half estimator as the time-clipping threshold approaches zero; an analogous divergence arises near high noise under the corresponding nondegeneracy condition. In contrast, LOOT eliminates both inverse-probability weights and retains bounded estimator variance.
PGM [8] report occasional training-loss spikes under complementary masking, consistent with the numerical sensitivity of inverse-noise weighting. The LOOT loss therefore preserves the expected MDM NELBO while eliminating the endpoint-divergent inverse-probability weights of both the standard MDM and half-and-half estimators.
Experimental details
We follow the experimental protocol of Sahoo et al. [24] closely, with modifications as required by the LOOT architecture (§ 3). Across both benchmarks, we optimize with Adam (learning rate , betas , ) without weight decay, apply gradient clipping at 1.0, and use a constant learning rate schedule with 2,500 steps of linear warmup. We maintain an exponential moving average (EMA) of the weights with decay 0.9999 and use this EMA copy for evaluation. Models are trained in bf16 mixed precision with a dropout rate of 0.1. All runs use four NVIDIA B200 GPUs on a single node, with DDP across GPUs and gradient accumulation with a global batch size of 512. Our implementation builds on the official MDLM codebase.1
Architectures
MDM baseline. The MDM baseline is a Diffusion Transformer (DiT) with 12 transformer blocks, 12 attention heads, and a hidden dimension of 768.
LOOT. LOOT uses the same per-block dimensions and depth as the MDM baseline. It maintains three streams (forward , backward , and output ) at each layer (§ 3.2) under full weight sharing: the three streams share a single set of attention projections (QKV and output), feedforward weights, and adaLN modulation parameters per block; the forward and backward streams further share their pre-attention and pre-MLP layer norms, while the output stream keeps its own layer norms. The trainable parameter count therefore matches MDM. When initializing from an MDM checkpoint, we copy each pretrained DiT block’s QKV projection, attention output projection, MLP, and adaLN modulation into the shared parameters, and copy its two layer norms into both the (shared) forward/backward layer norms and the output-stream layer norms.
LM1B
For LM1B [6], we use the bert-base-uncased tokenizer with a context length of 128 tokens and apply sequence packing, concatenating documents with the tokenizer’s separator ([SEP]) as the end-of-document marker. We train with a global batch size of 512 for 1M steps.
OpenWebText
For OpenWebText (OWT) [10], we use the GPT-2 tokenizer with a context length of 1024 tokens and apply sequence packing, concatenating documents with the GPT-2 <|endoftext|> token as the end-of-document marker. Since OWT does not provide an official validation split, we reserve the last 100K documents for validation following Sahoo et al. [24]. We train with a global batch size of 512 for 250K steps (per-GPU micro-batch of 32 with 4 gradient accumulation steps).
Self-Correction Policy Details
All self-correction policies use the same LOOT primitive: given a visible sequence , one forward pass returns for every position . The policies differ only in how they choose editable positions and replacements.
Algorithm 1 LOOT Self-Correction
Require: Initial steps , correction steps , policy
Generate with reverse denoising steps
for do
Compute for all
for do
if then
else
end if
end for
for all
end for
return
In Algorithm 1, is the proposed replacement token and indicates whether the edit is accepted.
LOOT-Blocked-Gibbs. Partition positions into groups . At step , choose . For every , sample , with . Thus all positions in the active block are resampled in parallel. In our experiments, we use groups.
LOOT-Best-Improvement. For each position, compute
Let be the positions with largest positive . For each , set
This greedily applies the locally best token replacements.
LOOT-Metropolis-Hastings. For each selected position , propose and accept the replacement with probability
Equivalently, draw and set .
LOOT-Blocked-Gibbs (guarded). Without nucleus truncation, high-temperature Gibbs updates occasionally insert low-probability tokens that corrupt the context of neighboring positions, and the chain can drift into repetitive fixed points (Appendix B.6). The guarded variant uses the same blocks and proposals as blocked Gibbs but accepts a draw only if it does not lower the leave-one-out likelihood of its position,
This deterministic acceptance rule prevents the collapse at every budget and temperature we tested, at a modest cost in generative perplexity where plain blocked Gibbs is stable. We use it only in the experiments without nucleus truncation. All policies use only the trained LOOT model.
Sampling evaluation
We evaluate unconditional generation on OpenWebText at sequence length . Each frontier point uses 512 independently generated samples. All samplers use nucleus sampling with , sweeping temperatures .
We measure sampling compute in number of forward evaluations (NFE). For MDM, ReMDM, and the uncorrected LOOT baseline, NFE counts denoising steps. For LOOT with self-correction, NFE , where and denote initial denoising and refinement steps, respectively. Unless otherwise stated, we allocate half the budget to each: . Correction policies are detailed in Appendix § 10.§ 4.
We evaluate generation quality using generative perplexity (GenPPL), computed as the exponential of the mean negative log-likelihood under GPT-2 Large [23], with teacher forcing after decoding and retokenizing generated samples. We measure diversity using sample entropy , defined as the average empirical unigram entropy of each generated sequence’s token IDs, before detokenization. Lower GenPPL indicates higher generation quality, while higher entropy indicates greater token diversity. We report quality–diversity frontiers across sampling temperatures.
Sampling without nucleus truncation
Figure 7a and Figure 7b repeat the comparison of § 4.2 with top-, i.e., sampling from the full categorical distribution at every step. Without truncation, MDLM and ReMDM exceed a generative perplexity of 100 at temperature 1.0 and above, so the perplexity axis is logarithmic. Plain blocked Gibbs collapses into repetition at temperature 1.0 and above: high-temperature draws occasionally insert low-probability tokens, the corrupted context then favors copying neighboring tokens, and the chain drifts into a low-entropy fixed point. We cut each sweep where entropy stops increasing with temperature. The guarded variant of Appendix § 10.§ 4 does not collapse at any budget, and together with the Metropolis-style corrector it dominates ReMDM at 2048 NFE from 128 NFE onward over the whole entropy range.
Figure 7. Generation quality without nucleus truncation (top-). (a) Quality–diversity frontiers using the protocol of Figure 6, with a logarithmic perplexity axis. Temperature sweeps are truncated where entropy stops increasing; guarded Blocked Gibbs remains stable across budgets. (b) LOOT self-correction at 256 and 512 NFEs versus ReMDM at 2048 NFEs, as in Figure 5a.
Full MAUVE results
Table 3 reports MAUVE for every sampler at every NFE budget, on 5000 samples per setting, together with generative perplexity and entropy on the same samples. We follow the MAUVE evaluation protocol of Wang et al. [33], ensuring direct comparability with their reported results. Blocked Gibbs achieves the highest MAUVE at every budget from 128 NFE onward.
| NFE | Method | genPPL | Entropy | MAUVE | |
| 64 | MDLM | 1 | 50.4 | 5.42 | 0.011 |
| ReMDM | 1 | 57.8 | 5.47 | 0.009 | |
| LOOT-Baseline | 0.95 | 58.7 | 5.42 | 0.007 | |
| LOOT-BG | 1 | 47.8 | 5.56 | 0.015 | |
| LOOT-BI | 0.95 | 71.2 | 5.42 | 0.006 | |
| LOOT-MH | 1 | 29.7 | 5.43 | 0.017 | |
| 128 | MDLM | 1.05 | 68.0 | 5.56 | 0.010 |
| ReMDM | 1 | 41.6 | 5.41 | 0.035 | |
| LOOT-Baseline | 1 | 66.1 | 5.52 | 0.008 | |
| LOOT-BG | 1 | 27.9 | 5.48 | 0.118 | |
| LOOT-BI | 1 | 59.8 | 5.54 | 0.010 | |
| LOOT-MH | 1.05 | 24.7 | 5.48 | 0.095 | |
| 256 | MDLM | 1.05 | 63.1 | 5.53 | 0.014 |
| ReMDM | 1.05 | 53.9 | 5.54 | 0.040 | |
| LOOT-Baseline | 1 | 55.4 | 5.47 | 0.010 | |
| LOOT-BG | 1 | 20.1 | 5.40 | 0.408 | |
| LOOT-BI | 1 | 36.3 | 5.44 | 0.035 | |
| LOOT-MH | 1.05 | 21.2 | 5.43 | 0.312 | |
| 512 | MDLM | 1.05 | 60.8 | 5.52 | 0.015 |
| ReMDM | 1.05 | 44.9 | 5.50 | 0.091 | |
| LOOT-Baseline | 1 | 50.8 | 5.45 | 0.010 | |
| LOOT-BG | 1.05 | 20.0 | 5.45 | 0.549 | |
| LOOT-BI | 1.05 | 32.3 | 5.48 | 0.072 | |
| LOOT-MH | 1.1 | 22.8 | 5.49 | 0.366 | |
| 1024 | MDLM | 1.05 | 59.4 | 5.51 | 0.017 |
| ReMDM | 1.05 | 37.6 | 5.45 | 0.242 | |
| LOOT-Baseline | 1 | 48.8 | 5.43 | 0.012 | |
| LOOT-BG | 1.1 | 21.1 | 5.51 | 0.580 | |
| LOOT-BI | 1.1 | 29.2 | 5.52 | 0.201 | |
| LOOT-MH | 1.1 | 22.1 | 5.47 | 0.432 | |
| 2048 | MDLM | 1.05 | 58.5 | 5.50 | 0.018 |
| ReMDM | 1.05 | 31.2 | 5.40 | 0.348 | |
| LOOT-Baseline | 1 | 47.6 | 5.42 | 0.012 | |
| LOOT-BG | 1.1 | 17.5 | 5.46 | 0.602 | |
| LOOT-BI | 1.1 | 24.3 | 5.48 | 0.401 | |
| LOOT-MH | 1.1 | 21.8 | 5.47 | 0.478 |
Table 3. Full sampling results on OpenWebText. Generative perplexity, sample entropy, and MAUVE across NFE budgets, with top- = 0.9 and the indicated sampling temperature . Best genPPL and MAUVE values at each NFE are bolded. BG, BI, and MH denote Blocked Gibbs, Best Improvement, and Metropolis–Hastings, respectively.
Ablations
This section presents two ablations complementing the language modeling results in § 4.1. First, we disentangle the effects of architecture and training objective across sequence lengths on
OpenWebText (Table 4), investigating the likelihood gap between LOOT and standard MDMs at longer sequences. Second, we evaluate the effects of merging frequency and training objective on LM1B (Table 5), demonstrating how both improvements contribute to the performance of LOOT.
Self-correction with the MDLM loss
To disentangle the benefits of self-correction from dense supervision, we evaluate the LOOT architecture trained with the standard masked-only MDLM loss. We use the OWT model from Table 4, selecting its best-validation checkpoint after 250k steps, trained with the standard MDM loss. Although visible tokens receive no direct supervision, the architecture still provides leave-one-out conditionals for self-correction. Table 6 compares ancestral sampling with Blocked Gibbs correction using the same model weights. Self-correction reduces generative perplexity by 2.4×–4.3× across NFE budgets. At 256 NFEs, Blocked Gibbs reduces genPPL from 62.3 to 19.9, matching the corrected model trained with our all-position objective. Even at 2048 NFEs, it achieves genPPL 12.9, compared with 19.0 for ReMDM and 37.4 for MDLM. These results demonstrate that effective self-correction is enabled by the leave-one-out architecture, even without dense supervision, while our objective independently improves likelihood and uncorrected generation quality.
Additional sampling experiments
Robustness. Figure 8 shows that the sampling gains are not isolated to a single temperature or policy. Across entropy buckets, all LOOT correction policies improve log-PPL over ReMDM, with the largest gains in the high-entropy regime where ReMDM degrades most sharply.
(a) (a) Best improvement in .
(b) (b) Mean improvement by entropy bucket.
Figure 8. LOOT gains are robust across entropy and compute. We report log-PPL improvement over ReMDM, where positive values indicate lower generative perplexity. Improvements persist across NFE budgets and entropy buckets.
Temperature response. Figure 10 and Figure 9 show that LOOT self-correction maintains higher entropy than MDLM and ReMDM baselines, across NFE budgets.

Figure 9. LOOT improves quality without collapsing entropy. Each point is matched to the uncorrected LOOT baseline at the same temperature and NFE. Positive vertical values indicate lower log-PPL than the baseline; horizontal values near zero indicate little entropy loss. LOOT correction policies concentrate in the high-improvement, low-entropy-shift region.

Figure 10. Sample entropy response across NFE budgets.
Qualitative OpenWebText generations
We present qualitative generations from models trained on OpenWebText. All samples use length 1024, temperature 1.0, top- = 0.9, and 1024 model evaluations.
LOOT-Blocked-Gibbs, = 4.
The Bitcoin community is moving strongly toward altcoins right now, with the latest altcoin being Monero. If you ask me, I say the most important thing to know about mining and whether altcoins are about money. One of the most exciting aspects of mining is getting an overview of what is going on in the Bitcoin ecosystem and getting to know everything going on along the way. The Ultimate Guide to Bitcoin Mining tells you everything that you really need to know about the basics of mining in the Bitcoin ecosystem. “Bitcoin.com’s Mining Guide is a comprehensive and comprehensive guide that covers all the basics of mining.” Want more? Do not forget to join the Bitcoin Forum mailing list, where we can answer your questions.
One of our latest and greatest 3D printing products is 3D printing. Our new ASUS 3D printer is innovative. It is not made straight out of cardboard or paper but it is made with a 3D printing system that is designed for delivering high quality products. So that sounds simple, but while we are trying to accomplish things like this, we have much more work to do and much more time than is needed to come up with a high quality product. We also have a strong commitment and commitment to delivering quality products to people who want to be our customers. To deliver quality products and services, we need to make sure customer satisfaction is our number one priority.
“It has made such a difference in my life,” Mo said. “I now have a better life and a better life than I ever had. We are a family. We have always wanted our dad to come back to Scotland.” Mo, 28, still hopes he will find work. Ever since he was released, he has given up looking for work and has been out of work for years. But now he wants to take care of himself a bit more. Now that one of Mo’s neighbours has come knocking on his door, he has two rental properties lying empty in a desperate attempt to find a job.
LOOT-Best-Improvement.
The biggest challenge for us is we have to improve working conditions and the quality of our work,” he said. “The problem is the fact that when you have the backing of the big companies, the same thing is creating a much bigger and riskier challenge to working conditions.” World Economic Outlook 2016 The United States and China receive 40% of India’s investment and trade-related return on investment, while Saudi Arabia, Chinese and Indian companies are also receiving higher returns than the US. Last year, the OECD reported that 60% of US companies, and 40% were as big as the UK and Germany. The US and China are the only two countries where the past year has brought bad news.
The video footage from the Clifton Ballet show was released Wednesday, after Greenpeace and the Wilberforce Foundation filed a lawsuit in support of the pesticide ban. “This is the largest smidgen of pesticide use that has ever had an adverse effect on health, and the court of appeals has allowed a large number of chemicals that cause cancer to be splashed into soil and water,” Greenpeace said. “The debate over pesticides also sends a message that they should be banned in order to reduce the harm caused by them, such as air pollution, and other harmful bioactive substances,” Greenpeace said in a statement.
McGuidley’s report on stock prices in late August, just a week after he resigned, said he is convinced there is evidence that many traders and service providers believe that a market correction will occur, and that even very high stock prices will benefit from it. “BnB Capital Markets probably would have had a similar effect if the trading volume fell and prices were rising seasonally,” he said. Black Friday, after all, rose 5 percent in August, and posted a few big gains in stock prices since then.
LOOT-Metropolis-Hastings.
In the area where the park was last shut down in 2010, the Forest Service has scrubbed the fallen trees from the park’s Grote and other mulch trees, a property donated to President Obama by climate-change activists. That has led to increased logging, for which the Forest Service continues to collect data every year. “The U.S. Forest Service has been able to collect that data and know that on the basis of that data, many trees were lost in the park,” said Michael Moriarty, of the National Resource Center for Rainforest. “The data for every tree in the park is collected and compared with data from previous years. Many of those are trees that were lost that have not been collected by the federal Forest Service yet.”
It is the latest development in a year where BlackBerry has found it nearly impossible to roll out a smartphone globally and to expand its reach beyond the United States. The company plans to launch about one-hundred new smartphones around the world, with most of them being expected to roll out sooner than the batch of more than 1,000 that are expected to launch. The company says it will be the first BlackBerry in the world and the first smartphone to be officially launched this year.
Disney, which owns Pixar’s stake in the film’s parent company. How big is the Disney deal for Pixar? Disney’s estimates of the total outlay were incorrect as of Wednesday morning, per the company’s website. An earlier report said that Disney bought Disney Animation from Pixar after Pixar acquired it for about $100 million. The deal allows Pixar to be part of Disney for five years. The Disney deal is one of the reasons the company has had to undergo a major restructuring because of the reorganization.
ReMDM.
Opponents say it runs the huge risk of market consolidation, which puts broadband and other telecoms-based services at risk. Meanwhile, wireless technology deployment costs more than US$12bn and requires major investment in innovation. Asked about Verizon’s proposal, Wheeler fired back, saying the company’s proposal would leave millions of customers with overpriced services. “It is driving up the prices of providers, especially for the biggest providers to consumers,” he said. “That is what we have created.” Verizon declined to comment Thursday. Wheeler’s proposal is set for next year’s hearings on telecom, where the other major carriers, such as Verizon and Charter, are being hammered. Verizon’s proposal would open up the possibility of an overhaul of the Code of Federal Regulations, which governs the wireless industry.
Ian Desmond could have made the team in the 2015 playoffs and suited up for plate duty. Instead, he has received some much needed help in the walk-out department. “I was always trying to make the contact and take advantage of the movement ability of opposing hitters,” Anderson said after his Feb. 18 loss to the White Sox over the Nationals. “I did not do that at the plate because I felt like I had to. Now, there is the thing about being on the field as the hitter and getting the walks.”
Since then, George Martin and I have done some great stuff, and we were both able to create the world and get to be a part of the story. Were there influences in the casts and sets? Did you meet in one scene? Did you meet any of the different creatives? We met a lot of fans. I am really excited about what George has done and what I have written. There is a fascinating book that I think can be read in its own way. How fun is it still exploring that little background that you know about Game of Thrones?
Checkpoint Compatibility and Hybrid Modeling
Recovering bidirectional attention. An interesting consequence of LOOT’s full parameter sharing is that the architecture can recover standard bidirectional MDM computation without introducing additional parameters. Recall that LOOT maintains three streams: a forward stream , a backward stream , and an output stream . To recover bidirectional attention, we copy the output stream into both directional streams before every layer:
We additionally allow the output stream to attend to its own position during merging. Since all three streams now share the same representations and attention parameters, their contributions can be combined to recover standard bidirectional self-attention exactly.
To see this, let , , and denote the query, key, and value projections of the shared representations, and define the attention scores . We partition the attention computation at position into left, right, and self contributions, with normalization factors
Let and denote the softmax-normalized attention outputs over the respective contexts. The merged output, including self-attention, is
Thus, the three-way merge is mathematically identical to bidirectional softmax attention. With shared parameters, consistent positional encodings, and matching attention masks, applying this construction at every layer recovers a standard bidirectional transformer. In particular, the directional-stream computations can be omitted entirely in this mode, reducing LOOT to the computational cost of a standard MDM forward pass.
Self-attention in the output stream. We also explored incorporating the self contribution into the output stream during standard leave-one-out training. This preserves the LOO property because the output representation is already independent of the input token . However, we found no consistent improvement in language modeling performance, with some configurations performing slightly worse. Furthermore, combining self-attention with our efficient sliding-window merging mechanism complicates backpropagation through the attention normalization, requiring FlexAttention or a custom differentiable implementation. We therefore omit this contribution from our default LOOT architecture.
Hybrid training and inference. The equivalence to bidirectional attention suggests a hybrid approach that combines the computational efficiency of standard MDMs with the dense supervision and self-correction capabilities of LOOT. At high masking rates, standard MDM training already supervises most positions, so the additional benefit of leave-one-out prediction may be limited despite its approximately forward-pass FLOP overhead. As more tokens become visible, however, leave-one-out prediction becomes increasingly valuable, providing supervision at previously unmasked positions and enabling correction of earlier predictions.
A promising direction is therefore to train a single model in both modes, using standard bidirectional MDM computation at high masking rates and switching to leave-one-out computation at lower masking rates. The same strategy could be applied at inference time, using efficient MDM sampling for initial generation followed by LOOT self-correction as more tokens become visible. Because both modes share the same parameters, this approach would not require separate predictor and corrector networks. We leave the design and evaluation of such hybrid training and sampling strategies for future work.
Finetuning from an MDM checkpoint
Because the three streams of LOOT share their attention and MLP weights (§ 3.3), a pretrained MDM checkpoint can initialize all of them directly. We test whether a short finetune is enough to equip an existing MDM with self-correction. Starting from the MDLM checkpoint of Appendix § 10.§ 3 after 200K steps on OWT, we either continue MDLM training for 50K steps, which gives the 250K-step MDLM used throughout § 4.2, or finetune the same checkpoint with the LOOT objective for 50K steps (sqrt noise schedule, learning rate with warmup and linear decay, global batch size 512, no beginning-of-sequence token). Both models see the same training data; the LOOT finetune costs the FLOPs per step, i.e., the total training FLOPs of the 250K-step MDLM.
Figure 11 repeats the comparison of Figure 6 with this finetuned model. Without correction, the finetuned LOOT tracks the continued-MDLM frontier at every budget, so 50K steps of leave-one-out training cost nothing in ancestral sampling quality. With correction, LOOT achieves a favorable generative perplexity–entropy tradeoff compared to ReMDM across all NFEs. An existing MDM checkpoint can thus be given self-correction with a finetune of a quarter of its pretraining steps.
Figure 11. Quality-diversity frontiers for LOOT finetuned from an MDM checkpoint. Same protocol as Figure 6 (top-); here the LOOT model is initialized from the 200K-step MDLM checkpoint and finetuned for 50K steps, while MDLM and ReMDM use the same checkpoint continued for 50K steps.
Computational cost
FLOPs per forward pass. Counting matmul and attention FLOPs at , a LOOT forward pass costs that of a DiT of the same size. The ratio is below the naive of three streams because LOOT doubles rather than triples the attention work (the forward and backward streams are batched into one causal attention call and the merge adds one sliding-window call per layer), uses a cheaper final merge block, and shares the vocabulary head. Since forward and backward passes scale together, training LOOT from scratch for the same number of steps as MDLM, as in § 4.2, costs the training FLOPs.
Measured throughput. Table 7 reports single-sequence latency without compilation. Table 8 reports the forward+backward throughput ratio with torch.compile across batch sizes: at small batch both models are kernel-launch-bound and compilation fuses LOOT’s merge operations, so the overhead is –; at large batch and both models are compute-bound and the ratio converges to , matching the analytic FLOP ratio.
Table 7: Latency of forward and backward passes on an NVIDIA B200 in bf16 at batch size 1, without compilation. We report the median of 200 timed iterations.
| Dataset | Architecture | Fwd (ms) | Fwd+Bwd (ms) |
| LM1B () | MDM | ||
| LM1B () | LOOT (ours) | ||
| OWT () | MDM | ||
| OWT () | LOOT (ours) |
Table 7. Latency of forward and backward passes on an NVIDIA B200 in bf16 at batch size 1, without compilation. We report the median of 200 timed iterations.
At this operating point LOOT runs at MDM latency on LM1B () and on OWT (); Table 8 shows that this single point understates the overhead at training batch sizes.
| Dataset | |||||||
| LM1B () | |||||||
| OWT () |
Table 8. Forward+backward throughput ratio MDM/LOOT (higher means larger LOOT overhead) on an NVIDIA B200 with torch.compile, mean std over timed iterations.
Table 8: Forward+backward throughput ratio MDM/LOOT (higher means larger LOOT overhead) on an NVIDIA B200 with torch.compile, mean std over timed iterations.
Inference cost per sample. Table 9 reports counted FLOPs and measured wall-clock time per generated sequence at on an NVIDIA A6000 (95% intervals over 3 runs). All methods run at their common reliable batch size: MDLM and LOOT at batch 4, ReMDM at its memory ceiling of batch 3 (LOOT at batch 3 is essentially unchanged, at 11.52 and 10.48 seconds for the two correctors). LOOT with self-correction at 256 NFE uses fewer FLOPs (71%) and about less wall-clock time per sample than ReMDM at 2048 NFE; at 512 NFE the FLOP saving is (42%). Against MDLM at the same number of forward passes, corrected LOOT is end-to-end no slower despite the FLOPs per pass; the uncorrected row shows this is a property of the corrected samplers, not of the architecture alone. The correction budget is also compressible: 8 Metropolis correction steps already capture 85% of the full 128-step improvement in generative perplexity.
Table 9: Inference cost per generated sequence at (NVIDIA A6000, bf16). FLOPs are counted matmul and attention FLOPs; seconds are measured end-to-end, with 95% intervals over 3 runs.
References
- [1]Marianne Arriola and Volodymyr Kuleshov. Set diffusion: Interpolating token orderings between autoregression and diffusion for fast and flexible decoding. arXiv preprint arXiv:2607.01775, 2026.arxiv.org/abs/2607.01775
- [2]Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models, 2025. URL https://arxiv.org/abs/2503.09573.
- [3]Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces, 2023. URL https://arxiv.org/abs/2107.03006.
- [4]Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, Chengyi Li, Chongxuan Li, Jianguo Li, Zehuan Li, Huabin Liu, Lin Liu, Guoshan Lu, Xiaocheng Lu, Yuxin Ma, Jianfeng Tan, Lanning Wei, Ji-Rong Wen, Yipeng Xing, Xiaolu Zhang, Junbo Zhao, Da Zheng, Jun Zhou, Junlin Zhou, Zhanchao Zhou, Liwang Zhu, and Yihong Zhuang. Llada2.0: Scaling up diffusion language models to 100b, 2025. URL https://arxiv.org/abs/2512.15745.
- [5]Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models, 2022. URL https://arxiv.org/abs/2205.14987.
- [6]Ciprian Chelba, Tomás Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Philipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. In Haizhou Li, Helen M. Meng, Bin Ma, Engsiong Chng, and Lei Xie, editors, 15th Annual Conference of the International Speech Communication Association, INTERSPEECH 2014, Singapore, September 14-18, 2014, pages 2635–2639. ISCA, 2014. doi: 10.21437/INTERSPEECH.2014-564. URL https://doi.org/10.21437/Interspeech.2014-564.
- [7]Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2205.14135
- [8]Justin Deschenaux, Lan Tran, and Caglar Gulcehre. Partition generative modeling: Masked modeling without masks. arXiv preprint arXiv:2505.18883, 2025.arxiv.org/abs/2505.18883
- [9]Yonggan Fu, Lexington Whalen, Abhinav Garg, Chengyue Wu, Maksim Khadkevich, Nicolai Oswald, Enze Xie, Daniel Egert, Sharath Turuvekere Sreenivas, Shizhe Diao, et al. Nemotron-labs-diffusion: A tri-mode language model unifying autoregressive, diffusion, and self-speculation decoding. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 54868–54887, 2026.arxiv.org/abs/2607.05722
- [10]Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019.
- [11]Samson Gourevitch, Yazid Janati, Dario Shariatian, Umut Simsekli, Eric Moulines, Eric P Xing, and Alain Durmus. Uniform diffusion models revisited: Leave-one-out denoiser and absorbing state reformulation. arXiv preprint arXiv:2605.22765, 2026.arxiv.org/abs/2605.22765
- [12]Jaeyeon Kim, Seunggeun Kim, Taekyun Lee, David Z Pan, Hyeji Kim, Sham Kakade, and Sitan Chen. Fine-tuning masked diffusion for provable self-correction. arXiv preprint arXiv:2510.01384, 2025.arxiv.org/abs/2510.01384
- [13]Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions, 2025. URL https://arxiv.org/abs/2502.06768.
- [14]Anji Liu, Oliver Broadrick, Mathias Niepert, and Guy Van den Broeck. Discrete copula diffusion. arXiv preprint arXiv:2410.01949, 2024.arxiv.org/abs/2410.01949
- [15]Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, Proceedings of Machine Learning Research, pages 32819–32848. PMLR / OpenReview.net, 2024. URL https://proceedings.mlr.press/v235/lou24a.html.arxiv.org/abs/2310.16834
- [16]Justin Lovelace, Christian Belardi, Sofian Zaluk, Adhitya Polavaram, Srivatsa Kundurthy, and Kilian Q Weinberger. Stop-think-autoregress: Language modeling with latent diffusion planning. arXiv preprint arXiv:2602.20528, 2026.arxiv.org/abs/2602.20528
- [17]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models, 2025. URL https://arxiv.org/abs/2502.09992.
- [18]Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In International Conference on Learning Representations, volume 2025, pages 64972–65009, 2025.arxiv.org/abs/2406.03736
- [19]Zijing Ou, Ruixiang Zhang, and Yingzhen Li. Discrete neural flow samplers with locally equivariant transformer. Advances in Neural Information Processing Systems, 38:127197–127237, 2026.
- [20]William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. URL https://arxiv.org/abs/2212.09748.
- [21]Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816–4828, 2021.arxiv.org/abs/2102.01454
- [22]Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang. Generative frontiers: Why evaluation matters for diffusion language models, 2026. URL https://arxiv.org/abs/2604.02718.
- [23]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- [24]Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.07524
- [25]Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin T. Chiu, and Volodymyr Kuleshov. The diffusion quality. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. URL https://proceedings.mlr.press/v267/sahoo25a.html.
- [26]Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, and Arash Vahdat. Esoteric language models. CoRR, abs/2506.01928, 2025. doi: 10.48550/ARXIV.2506.01928. URL https://doi.org/10.48550/arXiv.2506.01928.
- [27]Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang, Justin Deschenaux, Jingyu Liu, John Thickstun, and Ante Jukic. Scaling beyond masked diffusion language models. arXiv preprint arXiv:2602.15014, 2026.arxiv.org/abs/2602.15014
- [28]Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla-torre, Bernardo P. de Almeida, Alexander M. Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=15MrJ6g5G1.arxiv.org/abs/2412.10193
- [29]Yair Schiff, Omer Belhasin, Roy Uziel, Guanghan Wang, Marianne Arriola, Gilad Turok, Michael Elad, and Volodymyr Kuleshov. Learn from your mistakes: Self-correcting masked diffusion models, 2026. URL https://arxiv.org/abs/2602.11590.
- [30]Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data, 2025. URL https://arxiv.org/abs/2406.04329.
- [31]Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. arXiv preprint arXiv:2211.16750, 2022.arxiv.org/abs/2211.16750
- [32]Dimitri Von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hofmann. Generalized interpolating discrete diffusion. arXiv preprint arXiv:2503.04482, 2025.arxiv.org/abs/2503.04482
- [33]Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.00307, 2025.arxiv.org/abs/2503.00307
- [34]Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models, 2025. URL https://arxiv.org/abs/2508.15487.
- [35]Yixiu Zhao, Jiaxin Shi, Feng Chen, Shaul Druckmann, Lester Mackey, and Scott Linderman. Informed correctors for discrete diffusion models. arXiv preprint arXiv:2407.21243, 2024.arxiv.org/abs/2407.21243
- [36]Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling, 2025. URL https://arxiv.org/abs/2409.02908.









