WorksComputer Science
Train the Sampler You Run: Few-Step Masked Diffusion Language Models via Mixture-Likelihood Fine-Tuning
Abstract
When a masked diffusion language model is sampled in a few parallel steps in a random order, the model actually run is a mixture over reveal plans, while cross-entropy training fits token conditionals rather than this mixture. We introduce mixture-likelihood fine-tuning (Mix-FT), which fine-tunes a pretrained model on an importance-weighted bound of its own sampler's likelihood, with no teacher and no change to the architecture or the sampler. On LLaDA-8B, a LoRA fine-tune for 0.002% of the pretraining compute writes better OpenWebText continuations than continued cross-entropy and Alpha diffusion given the same compute, at every tested step count. With half the model evaluations, 16 rather than 32 steps per block, its continuations score better than the original model's in every seed under three judges, with confidence-based decoding as well as random order, and under confidence-based decoding two blinded LLM raters from different model families prefer them in 69 and 71% of pairs. On a public 110M-parameter block diffusion model, Mix-FT takes under four T4-hours and beats continued cross-entropy, Alpha diffusion and SDTT distillation at 4, 8 and 16 steps per block, under two judges in every seed. Exact likelihoods on text8, with a control that weights every plan equally, attribute about three quarters of the likelihood gain over continued cross-entropy to weighting each plan by how well it explains the sequence.
Introduction
Masked diffusion language models [26, 29, 17, 22] learn to fill masked positions given the revealed ones. They are attractive because they can be sampled in a few parallel steps: a block of positions is filled in steps, each revealing positions drawn independently given what is already revealed. The factorization error this causes is well known. Current fixes act at sampling time, through temperature [1] or better schedules [18, 3], add heads or latents [15, 35, 14], or distill from a many-step teacher [7, 11, 36]. We change the loss instead, and the objective follows from one observation. Under a random reveal order, the deployed sampler draws from a mixture over reveal plans ,
where each component is a product of factorized steps along plan . The standard cross-entropy (CE) loss averages over plans; it separates over steps and is minimized by the true conditionals, which do not minimize . So we fine-tune a CE-trained model on the likelihood of the sampler we run, through an importance-weighted bound [5] over random plans (Figure 1). We call the result Mix-FT.
This change of loss alone, for under 0.02% of the pretraining compute, makes the same network with the same sampler write better text. On LLaDA-8B [22], a LoRA fine-tune for 0.002% of the pretraining compute beats continued cross-entropy and Alpha diffusion [10] at every tested step count in every seed. With 16 steps per block, half the model evaluations, it is ahead of LLaDA’s 32 in every seed under three judges, with LLaDA’s confidence planner as well as random order, and under the confidence planner blinded LLM raters from two model families prefer its continuations in 69 and 71% of pairs (Section 5). On the public 110M BD3-LM model of OpenWebText [2], four T4-hours of Mix-FT beat continued cross-entropy, Alpha diffusion and SDTT distillation [7] at 4, 8 and 16 steps per block, all given the same extra compute: two judge models score its samples better than the strongest baseline’s at each step count in every seed, by 0.25 to 0.45 nats/token on average at matched per-sample entropy (Table 1 and Figure 2). Baselines given eight times the budget, more updates or BD3-LM’s own objective stay well behind § 3.

Figure 1. A -step random-order sampler draws a plan, then fills each group of positions independently given the groups before it. Its law is the average of the per-plan products. CE averages the per-plan log-likelihoods and is minimized by the true conditionals. Mix-FT fine-tunes on an importance-weighted bound of the log of the average.
Exact experiments on text8 explain why this works. We compute the sampler’s likelihood exactly with a dynamic program over reveal sets, so we can split the gain. Part of it is the within-step dependence that independent filling loses; the other part appears even at one token per step, where the fine-tuned network is better under the mixture over orders. A control with the same plans, batches and updates but equal plan weights recovers only a quarter of the gain.
Contributions.
A teacher-free fine-tune on the random-order sampler’s mixture likelihood that improves sample quality over every baseline we ran at 4, 8 and 16 steps per block, on public 110M, 1.1B and 8B models. On LLaDA-8B its 16 steps per block are ahead of the original’s 32; at 110M, with 256 plans per sequence and 8× the fine-tune budget, it beats the original model by more than 0.5 nats/token at equal steps, and its 8 steps per block close 95% of the gap between the original’s 8 and 16 steps.
The finding that training on coarse two-step plans improves 8- and 16-step sampling more than training on four- or eight-step plans at equal compute, and not because coarse plans allow more updates (§ 3).
An account of why CE is not the sampler’s optimum, building on Amin et al. [1], tested with exact -step likelihoods and a control that isolates the plan weighting (§ 2.1 and § 5).
Setup: the -step random-order sampler
We use block diffusion (Arriola et al., [2]): a sequence is split into blocks of positions, generated left to right, each by a masked diffusion sampler conditioned on the clean previous blocks. All statements below concern one block given its prefix, which we drop from the notation.
A plan is an ordered partition of the positions into groups of size , drawn uniformly. The denoiser is a distribution over the token at masked position given the revealed set . At step the sampler draws every position of independently from . Under a fixed plan and under the sampler,
A temperature replaces by ; we sweep for every model. Masked CE pretraining draws a mask rate and minimizes the CE of masked positions. Our stronger CE baseline draws a plan and a step , reveals and minimizes the CE of all masked positions, so it trains exactly at the states the sampler visits. Both have the true conditionals as their optimum.
Fine-tuning on the sampler’s mixture likelihood
Objective. We fine-tune on the random-order sampler’s mixture likelihood, through the importance-weighted (IWAE) bound over plans drawn from :
an upper bound on that tightens monotonically in (Burda et al., [5]). Evaluating the bound for a sequence costs forward passes: one with the block fully masked, shared by all plans, and per plan. Blocks share passes, since they are conditionally independent given the clean prefix. With self-normalized weights the gradient is
cross-entropy at the sampler’s own states with each plan weighted by how well it explains . At the weight is 1, and is the per-path loss below, whose optimum is the true conditionals, so the method needs . As the weights approach the plan posterior and the gradient approaches that of . In exact toys, training a free table on leaves it where it started, while removes 94% of the error of the true conditionals at (Table 24, Section F). Training targets one random-order sampler at one step count ( or 4); other step counts and confidence decoding are transfers we measure.
| sampler | true cond. | exact mixture | best temperature | |||
| , random order | 0.0440 | 0.0436 | 0.0103 | 0.0025 | 0.0000 | 0.0096 |
| , random order | 0.1481 | 0.1462 | 0.0380 | 0.0085 | 0.0008 | 0.0319 |
Table 24. Exact toy (, , entropy 5.79 nats): after training a free table on with fresh plans every step (1,500 Adam steps). is the per-path loss.
Why CE is not the sampler’s optimum.
Proposition 1 (Fixed or state-determined plans). If the plan is fixed, or the next group is a deterministic function of the revealed tokens that does not depend on , then
and every term is minimized by .
This is the chain rule. For such samplers nothing done to the denoiser helps beyond learning the conditionals; the remaining error is the plan’s factorization error. Planners that read the denoiser’s own outputs, such as confidence-based ones, are not covered. For a random plan, let be the posterior over plans.
Proposition 2 (Per-path loss = mixture loss + plan information). For every (a special case of Theorem 1 of Amin et al. [1], whose mixing term is this KL),
Proof. .
The left side, averaged over , is the per-path loss: CE at the sampler’s states up to a per-position weighting. It charges the sampler for any information a sample carries about its own plan. No reweighting of masks, steps or orders removes this term, since every such loss keeps the logarithm inside the plan average.
Two effects. Proposition 2 holds at every , and it allows two ways in which the mixture optimum differs from what CE learns. (i) Within-step dependence. When a step reveals positions, the mixture can let plans specialize, for example by committing early to a mode so that later same-step tokens are nearly determined. On small exact problems, re-optimizing a free table of conditionals for the mixture removes nearly all of the error that the true conditionals leave, more than a tuned per-step temperature does (Table 23). (ii) Fit under the order mixture at one token per step. At there is no within-step factorization, and exact conditionals make every order exact, so CE would be optimal with unlimited capacity. A finite network cannot make every order agree. The per-path loss, here the any-order negative ELBO (NELBO), asks it to fit every order equally; the mixture likelihood needs only the average over orders to be good, and the weights move effort toward the orders that explain each sequence. Two observations fit this: on text8 the exact likelihood at one token per step improves, by an amount similar to effect (i), and at 110M the tighter IWAE-64 bound of the one-token-per-step sampler improves while the NELBO, the per-path bound, gets worse (§ 3 and § 5).
| sampler | true conditionals | best global temp. | per-step temp. | re-optimized table |
| , fixed plan | 1.2049 | – | – | 1.2049 (no change) |
| , random order | 0.3899 | 0.2141 | 0.1138 | 0.0299 |
| , fixed plan | 0.5158 | – | – | 0.5158 (no change) |
| , random order | 0.0556 | 0.0200 | 0.0111 | 0.0000 |
Table 23.
Exact evaluation. Because depends on the revealed set and not on the order it was revealed in, the sum over plans in Equation (1) collapses to a dynamic program over reveal sets (Section G). At , it needs 127 forward passes per sequence for 2,520 plans. We use it for all block-8 text8 results, since IWAE-64 is looser for Mix-FT than for CE and so understates the gain.
Main result: a public 110M model
Setup. We fine-tune the public BD3-LM OpenWebText checkpoint with block 16 (Ariola et al. (2025)), a 110M-parameter model trained on sequences of 1,024 GPT-2 tokens, and study its natural sampler. Every arm starts from these weights and gets the same extra compute, counted in forward-equivalents of the network (§ 1). This budget is under 0.02% of the pretraining compute and under 4 T4-hours per arm. Mix-FT trains on two-step plans () with plans per sequence. The baselines are continued CE at the four-step sampler’s states; Alpha diffusion (Gushchin et al. [10]), trained exactly like continued CE with only the loss changed, at its best setting ; and SDTT (Deschenaux & Gulcehre [7]) distilled from 16 to 4 steps per block in two rounds (SDTT-4) and from 16 to 8 in one (SDTT-8). Each arm has three seeds (§ 1).
Evaluation. For each step count per block we draw 512 samples of 1,024 tokens at each of six temperatures. GPT-2-large and Pythia-1.4B (Radford et al. [25]; Biderman et al. [4]) score them on content tokens, leaving out newline and end-of-text targets, so that a model’s newline rate, which few-step sampling inflates and which differs between methods, does not decide a comparison (Table 8). Scored on all tokens instead, Mix-FT still beats the start and every baseline at , 8 and 16 in every seed under both judges (Section A). Temperature trades quality for diversity, so we compare models at equal per-sample entropy, the entropy of each sample’s own token frequencies averaged over samples. We interpolate judge NLL along each model’s temperature sweep to four entropy levels, from real text’s value to 0.3 nats below it, and average them. This compares whole temperature sweeps, which covers the temperature-tuned baseline of Amin et al. [1]. Unless stated otherwise, every claim below that one model beats another holds in every seed under both judges, and controls run in one seed use seed 0. We wrote down the criterion for each comparison before running it; Section K gives each with its outcome.
| model | matched entropy | |||||||||||
| 512 samples | 128 samples | 512 samples | 128 samples | |||||||||
| Start (BD3-LM) | 8.0 | 5.4 | 4.2 | 10.9 | 8.7 | 5.2 | 4.3 | 3.5 | 3.3 | 6.3 | 4.4 | 3.3 |
| Continued CE | 9.4 | 6.1 | 4.7 | 12.5 | 9.6 | 6.1 | 4.9 | 3.8 | 3.5 | 6.5 | 5.0 | 3.6 |
| Alpha () | 7.3 | 5.5 | 4.8 | 11.1 | 7.5 | 5.6 | 7.3 | 5.3 | 4.6 | 12.2 | 7.5 | 5.3 |
| SDTT-4 | 2.4 | 1.8 | 2.2 | 3.6 | 2.6 | 1.8 | 1.5 | 1.4 | 1.9 | 1.7 | 1.6 | 1.3 |
| SDTT-8 | – | 3.5 | 3.2 | – | – | – | – | 2.5 | 2.5 | – | – | – |
| Four-step Mix-FT | 8.5 | 5.8 | 4.7 | 11.2 | 8.5 | 5.6 | 5.2 | 3.9 | 3.6 | 7.0 | 5.4 | 3.7 |
| Mix-FT (two-step) | 7.3 | 5.5 | 4.8 | 9.2 | 7.1 | 5.6 | 4.7 | 4.2 | 4.1 | 5.4 | 4.7 | 4.2 |
| Mix-FT 8× | 7.5 | 5.5 | 4.7 | – | – | – | 5.8 | 4.5 | 4.3 | – | – | – |
| Mix-FT after SDTT-4 | 7.7 | 5.6 | 4.5 | – | – | – | 4.8 | 3.8 | 3.6 | – | – | – |
Table 8. Share of newline tokens in samples at 110M (percent; real held-out text: 3.35%). Matched entropy: interpolated along each model’s temperature sweep at the matched per-sample entropy, as for the judges. : the untempered sampler. The 512-sample files are the main protocol of § 3; the 128-sample files are the earlier protocol at , 4 and 8. Means over the three fine-tune seeds. –: not measured.
The first token. BD3-LM never trains position 0 of a window, and its sampler fixes that token to end-of-text; our fine-tunes train it, and our sampler draws it for every model. Drawing it costs the start 0.05 to 0.11 nats/token at 8 and 16 steps, since that output never received a gradient, and saves it 0.04 to 0.07 at 4 steps; it moves the fine-tuned models by at most 0.07. Every table draws it, and every comparison with the start is stated under whichever convention favors the start, with both numbers where they differ (Section B).
Mix-FT gives the best judged samples from 4 to 16 steps per block. At every step count from 4 to 16, Mix-FT’s samples score better than every baseline’s (Table 1 and Figure 2; per seed in Table 6): by 0.25 to 0.45 nats/token over the strongest fine-tuned baseline at each step count, and by 0.52 to 0.72 over continued CE, the usual way to spend extra compute. All methods take the same time per step, so Mix-FT is the best model we measured at every generation budget from 93 seconds up. At two steps per block it still beats the start, by 0.18 to 0.32 nats/token in every seed under both judges with the first token drawn, the start’s better convention there, while SDTT-4, distilled for few steps, leads. It is not merely the model that loses least. Every fine-tuned baseline ends behind the untouched start at , continued CE by 0.15 to 0.25 nats/token at and 16 (0.22 to 0.31 in every seed when both fix the first token), and neither a tenth of the learning rate nor EMA weights bring it level (Table 7e). Even BD3-LM’s own objective, continued for the same budget on its own windows, ends 0.18 to 0.43 behind the start, and Mix-FT on those windows beats it by 0.50 to 0.96 (Table 7g). Mix-FT beats the start at by more than 0.3 nats/token, and by 0.22 to 0.26 in every seed when both fix the first token. Per-step temperature schedules, tuned on a development split, improve Mix-FT, Alpha diffusion and SDTT-8 at by the same 0.3 nats/token, which leaves their order unchanged.
Figure 2. Sample quality against measured generation time at 110M (block 16, 512 samples per temperature, one T4). (a) GPT-2-large content-token NLL at matched per-sample entropy, seed means; bottom dotted line: held-out real text. With 8× the budget, Mix-FT at 8 steps per block comes within 0.04 nats/token of the start’s 16-step quality in about half the time when both fix the first token, the start’s own convention (hollow star, dashed line), and reaches it when both draw it (filled star, dotted line). (b) Each model’s NLL minus Mix-FT’s at the same step count (positive: Mix-FT better); lines are seed means, dots are the three paired seeds. From 4 steps per block up every baseline lies above zero in every seed; at 2 steps SDTT-4 lies below.
| Mix-FT against | GPT-2-large seed 0 | GPT-2-large seed 1 | GPT-2-large seed 2 | GPT-2-large mean | Pythia-1.4B seed 0 | Pythia-1.4B seed 1 | Pythia-1.4B seed 2 | Pythia-1.4B mean | |
| Start (BD3-LM) | 2 | 0.218 | 0.320 | 0.195 | 0.244 | 0.205 | 0.268 | 0.176 | 0.216 |
| Start (BD3-LM) | 4 | 0.573 | 0.584 | 0.532 | 0.563 | 0.595 | 0.587 | 0.560 | 0.581 |
| Start (BD3-LM) | 8 | 0.521 | 0.497 | 0.469 | 0.496 | 0.544 | 0.518 | 0.492 | 0.518 |
| Start (BD3-LM) | 16 | 0.348 | 0.364 | 0.312 | 0.341 | 0.368 | 0.388 | 0.334 | 0.363 |
| Continued CE | 4 | 0.569 | 0.735 | 0.673 | 0.659 | 0.573 | 0.709 | 0.660 | 0.647 |
| Continued CE | 8 | 0.693 | 0.729 | 0.718 | 0.713 | 0.708 | 0.743 | 0.716 | 0.722 |
| Continued CE | 16 | 0.535 | 0.531 | 0.492 | 0.519 | 0.545 | 0.538 | 0.490 | 0.525 |
| SDTT-4 | 2 | ||||||||
| SDTT-4 | 4 | 0.292 | 0.292 | 0.287 | 0.290 | 0.333 | 0.314 | 0.331 | 0.326 |
| SDTT-4 | 8 | 0.664 | 0.625 | 0.644 | 0.644 | 0.668 | 0.628 | 0.644 | 0.647 |
| SDTT-4 | 16 | 0.647 | 0.676 | 0.614 | 0.646 | 0.649 | 0.686 | 0.623 | 0.652 |
| SDTT-8 | 8 | 0.497 | 0.456 | 0.465 | 0.473 | 0.505 | 0.466 | 0.472 | 0.481 |
| SDTT-8 | 16 | 0.447 | 0.450 | 0.399 | 0.432 | 0.452 | 0.460 | 0.406 | 0.439 |
| Alpha diffusion () | 2 | 0.178 | 0.078 | 0.085 | 0.087 | 0.028 | 0.028 | ||
| Alpha diffusion () | 4 | 0.226 | 0.298 | 0.235 | 0.253 | 0.239 | 0.271 | 0.236 | 0.249 |
| Alpha diffusion () | 8 | 0.437 | 0.414 | 0.470 | 0.440 | 0.447 | 0.417 | 0.473 | 0.446 |
| Alpha diffusion () | 16 | 0.435 | 0.420 | 0.372 | 0.409 | 0.444 | 0.438 | 0.379 | 0.421 |
| Four-step Mix-FT (ablation) | 2 | 0.265 | 0.313 | 0.119 | 0.232 | 0.200 | 0.274 | 0.108 | 0.194 |
| Four-step Mix-FT (ablation) | 4 | 0.313 | 0.399 | 0.230 | 0.314 | 0.317 | 0.377 | 0.228 | 0.307 |
| Four-step Mix-FT (ablation) | 8 | 0.224 | 0.271 | 0.183 | 0.226 | 0.223 | 0.274 | 0.180 | 0.225 |
| Four-step Mix-FT (ablation) | 16 | 0.117 | 0.161 | 0.086 | 0.121 | 0.115 | 0.163 | 0.087 | 0.122 |
Table 6. Per-seed gains of Mix-FT at 110M: each model’s content-token judge NLL minus Mix-FT’s at matched per-sample entropy (nats/token, 512 samples per temperature; positive: Mix-FT better). Mix-FT is the two-step model of Table 1. Means are over the three seeds and are computed before rounding, as are the margins quoted in § 3.
Table 7. Controls at 110M. Each row gives the content-token judge gain of the first arm over the second at matched per-sample entropy (nats/token; positive: first arm better) under GPT-2-large and Pythia-1.4B, with each arm’s compute in multiples of the 1× budget and its number of optimizer updates. Mix-FT runs are named by the plans they train on (two-, four- or eight-step) and by , the number of plans per sequence ( unless stated). Rows with three seeds give seed means whose sign holds in every seed under both judges, except cells marked †; rows with one seed are seed 0. –: not measured.
| first arm | second arm | compute | updates | seeds | GPT-2 | Pythia | GPT-2 | Pythia | GPT-2 | Pythia |
| (a) plan coarseness and update count | ||||||||||
| two-step | four-step | 1 / 1 | 1,152 / 443 | 3 | +0.31 | +0.31 | +0.23 | +0.23 | +0.12 | +0.12 |
| two-step, 443 steps | four-step | 0.38 / 1 | 443 / 443 | 3 | +0.23 | +0.23 | +0.22 | +0.22 | – | – |
| two-step, 443 steps | two-step | 0.38 / 1 | 443 / 1,152 | 3 | −0.09 | −0.08 | −0.00† | −0.01† | – | – |
| two-step | four-step, 1,152 steps | 1 / 2.6 | 1,152 / 1,152 | 1 | +0.25 | +0.26 | +0.20 | +0.20 | – | – |
| two-step | four-step, | 1 / 1 | 1,152 / 1,047 | 1 | +0.48 | +0.48 | +0.45 | +0.46 | – | – |
| two-step | eight-step | 1 / 1 | 1,152 / 198 | 1 | +0.44 | +0.42 | +0.26 | +0.26 | +0.08 | +0.08 |
| (b) number of plans and compute (two-step plans) | ||||||||||
| 1 / 1 | 1,152 / 1,920 | 1 | +0.14 | +0.15 | +0.20 | +0.20 | – | – | ||
| 1 / 1 | 640 / 1,152 | 1 | +0.09 | +0.10 | +0.11 | +0.11 | +0.09 | +0.09 | ||
| 1 / 1 | 338 / 1,152 | 3 | +0.18 | +0.19 | +0.24 | +0.24 | +0.16 | +0.16 | ||
| 1 / 1 | 174 / 338 | 1 | +0.02 | +0.02 | +0.03 | +0.04 | +0.01 | +0.01 | ||
| 1 / 1 | 88 / 174 | 1 | −0.04 | −0.02 | −0.01 | −0.02 | −0.01 | −0.02 | ||
| at 3× | at 1× | 3 / 1 | 3,456 / 1,152 | 1 | −0.01 | −0.01 | +0.04 | +0.04 | −0.00 | +0.01 |
| at 3× | at 1× | 3 / 1 | 1,016 / 338 | 1 | +0.06 | +0.05 | −0.01 | +0.00 | −0.03 | −0.03 |
| at 2× | at 1× | 2 / 1 | 349 / 338 | 1 | +0.11 | +0.11 | +0.06 | +0.07 | +0.02 | +0.03 |
| at 4× | at 2× | 4 / 2 | 354 / 349 | 1 | +0.08 | +0.08 | +0.05 | +0.05 | +0.04 | +0.04 |
| at 8× | at 4× | 8 / 4 | 357 / 354 | 1 | +0.07 | +0.08 | +0.06 | +0.06 | +0.05 | +0.05 |
| (c) per-path controls: the same runs with equal plan weights | ||||||||||
| four-step Mix-FT | four-step per-path FT | 1 / 1 | 443 / 443 | 3 | +0.53 | +0.53 | +0.61 | +0.62 | – | – |
| four-step per-path FT | continued CE | 1 / 1 | 443 / 5,760 | 3 | −0.18 | −0.19 | −0.12 | −0.12 | – | – |
| two-step Mix-FT | two-step per-path FT | 1 / 1 | 1,152 / 1,152 | 1 | +0.80 | +0.79 | +0.78 | +0.79 | +0.57 | +0.59 |
| two-step per-path FT | continued CE | 1 / 1 | 1,152 / 5,760 | 1 | −0.23 | −0.21 | −0.08 | −0.08 | −0.04 | −0.04 |
| (d) per-step temperature schedule against one temperature at (same weights) | ||||||||||
| Mix-FT, schedule | Mix-FT, one temp. | 1 / 1 | 1,152 / 1,152 | 1 | – | – | +0.30 | +0.31 | – | – |
| Alpha, schedule | Alpha, one temp. | 1 / 1 | 5,760 / 5,760 | 1 | – | – | +0.29 | +0.30 | – | – |
| SDTT-8, schedule | SDTT-8, one temp. | 1 / 1 | 3,840 / 3,840 | 1 | – | – | +0.29 | +0.30 | – | – |
| (e) learning rate and weight averaging for continued CE | ||||||||||
| Mix-FT | continued CE, 0.1× lr | 1 / 1 | 1,152 / 5,760 | 1 | +0.76 | +0.76 | +0.80 | +0.81 | +0.51 | +0.52 |
| Mix-FT | continued CE, EMA | 1 / 1 | 1,152 / 5,760 | 1 | +0.63 | +0.62 | +0.68 | +0.69 | +0.51 | +0.52 |
| continued CE, 0.1× lr | untouched start | 1 / 0 | 5,760 / 0 | 1 | −0.19 | −0.16 | −0.28 | −0.26 | −0.16 | −0.15 |
| continued CE, EMA | untouched start | 1 / 0 | 5,760 / 0 | 1 | −0.06 | −0.03 | −0.16 | −0.15 | −0.17 | −0.15 |
| (f) baselines at the 8× budget | ||||||||||
| 8× Mix-FT | continued CE 8× | 8 / 8 | 357 / 46,080 | 3 | +1.19 | +1.21 | +1.24 | +1.27 | +0.87 | +0.89 |
| 8× Mix-FT | Alpha 8× | 8 / 8 | 357 / 46,080 | 3 | +0.95 | +0.97 | +0.98 | +1.01 | +0.81 | +0.84 |
| 8× Mix-FT | Alpha 8×, two-step states | 8 / 8 | 357 / ≤23,040‡ | 3 | +0.78 | +0.81 | +0.88 | +0.90 | +0.77 | +0.79 |
| 8× Mix-FT | SDTT-8 8× | 8 / 8 | 357 / ≤30,720‡ | 3 | +0.85 | +0.90 | +0.90 | +0.92 | +0.72 | +0.74 |
| (g) BD3-LM’s own objective (null fine-tune) and BD3-LM’s raw windows | ||||||||||
| null, raw windows | untouched start | 1 / 0 | 5,760 / 0 | 3 | −0.25 | −0.24 | −0.35 | −0.36 | −0.23 | −0.23 |
| Mix-FT, raw windows | null, raw windows | 1 / 1 | 1,152 / 5,760 | 3 | +0.72 | +0.72 | +0.84 | +0.86 | +0.53 | +0.55 |
| continued CE, raw windows | null, raw windows | 1 / 1 | 5,760 / 5,760 | 3 | +0.09 | +0.08 | +0.01† | +0.01† | −0.02† | −0.02† |
| Mix-FT, raw windows | continued CE, raw windows | 1 / 1 | 1,152 / 5,760 | 3 | +0.64 | +0.65 | +0.83 | +0.85 | +0.55 | +0.56 |
| Mix-FT, raw windows | untouched start | 1 / 0 | 1,152 / 0 | 3 | +0.47 | +0.48 | +0.49 | +0.50 | +0.31 | +0.32 |
| null, raw windows | null, our windows | 1 / 1 | 5,760 / 5,760 | 3 | −0.21 | −0.22 | −0.25 | −0.27 | −0.16 | −0.17 |
| null, our windows | untouched start | 1 / 0 | 5,760 / 0 | 3 | −0.04† | −0.02† | −0.10 | −0.09 | −0.07 | −0.06 |
caption:
Two-step training plans beat finer ones. Training on two-step plans beats the same recipe on four-step plans at every step count, by 0.12 to 0.31 nats/token on average (Table 4). A two-step plan costs fewer passes, so at equal compute this run takes 1,152 optimizer steps instead of 443, but the extra updates are not the reason (Table 7a): a two-step run of only 443 steps still beats the four-step recipe, while four-step runs given about as many updates as the two-step run stay 0.25 to 0.48 nats/token behind at . Even training on the eight-step plans that the sampler runs is worse at than training on two-step plans. Our untested hypothesis is that coarse plans give the objective the most to correct: the start’s plan weights are most concentrated at (median effective sample size of eight plans 1.32, against 1.54 at and 1.95 at ; Section A), so the plan information that Proposition 2 charges for is largest there. On text8, where the likelihood is exact, two-step plans also beat the four-step recipe at equal compute on the exact likelihood and on samples in every seed, though each step reveals half a block (Table 22b). How the two-step model was chosen is in Section K.
| training plans | opt. steps | sampler bound | sampler bound | NELBO | PPL | judge NLL at steps per block | judge NLL at steps per block | judge NLL at steps per block | judge NLL at steps per block |
| , (main) | 1,152 | 3.181 | 3.723 | 24.51 | 7.07 | 5.54 | 4.10 | 3.37 | |
| , (3 seeds) | 443 | 3.187 | 3.741 | 23.29 | 7.30 | 5.85 | 4.33 | 3.49 | |
| , | 1,047 | 3.194 | 3.747 | 22.73 | – | 6.01 | 4.53 | – | |
| , | 1,440 | 3.197 | 3.751 | 22.54 | – | 6.09 | 4.60 | – | |
| , | 1,920 | 3.181 | 3.731 | 23.55 | – | 5.67 | 4.27 | – |
Table 4. Ablations of Mix-FT at 110M at the compute of Table 1 (training plans , number of plans ), with the same columns; seed 0 except the four-step recipe (3 seeds). The main recipe (, ) is repeated for reference.
| (a) width | |||||||
| width | parameters | continued CE, | gain at | vs continued CE per step | vs continued CE gap | vs CE at states per step | vs CE at states gap |
| 256 | 4.8M | 1.794 | 0.042 | 0.024 | 0.018 | 0.026 | 0.016 |
| 512 | 19M | 1.683 | 0.036 | 0.018 | 0.018 | 0.019 | 0.017 |
| 768 | 43M | 1.666 | 0.032 | 0.013 | 0.019 | 0.015 | 0.017 |
| (b) training plans | |||||||
| arm | exact : seed 0 | exact : seed 1 | exact : seed 2 | exact : mean | mean | mean | vs four-step, seeds 0, 1, 2 |
| Start (Base 4×) | 1.8076 | 1.8054 | 1.8005 | 1.8045 | 1.7097 | 2.1033 | − |
| Mix-FT, four-step plans | 1.7518 | 1.7552 | 1.7449 | 1.7506 | 1.6731 | 2.0410 | reference |
| Two-step plans, equal compute | 1.7485 | 1.7500 | 1.7432 | 1.7472 | 1.6993 | 1.9984 | +0.09, +0.11, +0.06 |
| Two-step plans, equal updates | 1.7567 | 1.7600 | 1.7499 | 1.7555 | 1.7031 | 2.0125 | +0.06, +0.10, +0.02 |
| (c) plans per sequence at fixed updates | |||||||
| arm (compute) | exact : seed 0 | exact : seed 1 | exact : seed 2 | exact : mean | IWAE-8 − exact | ESS of 8 | vs Base 5× |
| Mix-FT, (1) | 1.7518 | 1.7552 | 1.7449 | 1.7506 | 0.063 | 3.39 | +0.16 |
| (3.9) | 1.7409 | 1.7411 | 1.7324 | 1.7382 | 0.079 | 2.95 | +0.20 |
| (15.4) | 1.7358 | 1.7346 | 1.7275 | 1.7327 | 0.093 | 2.65 | +0.25 |
| (30.8) | 1.7315 | 1.7350 | 1.7267 | 1.7311 | 0.096 | 2.56 | +0.27 |
| Continued CE (30.8) | 1.6509 | 0.040 | 4.51 | +0.12 |
Table 22. Text8 ablations (block 8, exact NLL in bpc at the best temperature). (a) Width: the protocol rerun at each width (means of 3 seeds). Gain: continued CE (Base 5×) minus Mix-FT at . It splits into the part that persists at one token per step (per step: both models sampled at ) and the narrowing of the gap between and (gap), against continued CE and against CE trained at the states. (b) Training plans: Mix-FT on two-step plans at the four-step recipe’s compute (more updates) and at its number of updates (0.36 of its compute). : judge gain over the four-step model at and matched distinct-4, per seed. (c) Plans per sequence at a fixed number of updates, with compute in units of 691,200 passes; IWAE-8 minus exact: how far the eight-plan bound lies above the exact NLL; ESS: median effective sample size of eight plan weights at ; against Base 5×, mean of 3 seeds. The last row is continued CE given the compute of (seed 0).
Likelihood and its price. Mix-FT has the best sampler bound of all arms at and , below continued CE’s in every seed (Table 1). The NELBO, the per-path bound of the one-token-per-step sampler, gets worse: its perplexity rises from 23.0 to 24.5, while continued CE lowers it to 22.4 under our scoring of the held-out windows (Section B). The tighter IWAE-64 bound of the same sampler does not get worse, and 0.002 to 0.003 nats/token below continued CE’s in every seed. So the NELBO rise widens its gap to the tighter bound, the looser per-path fit that effect (ii) predicts.
Half the steps. At equal compute, more plans help little beyond because the update count falls. Holding the update count near 350 and doubling the plans together with the compute narrows the gap between Mix-FT at and the start at one doubling at a time (Figure 3a). With at 8× the budget (0.18% of the pretraining compute as executed, 0.13% as counted; § 17), Mix-FT at closes 95% of the start’s own gap between 8 and 16 steps under GPT-2-large (bootstrap interval 92 to 99%) and 97% under Pythia, in about half the generation time (Figure 2a). This is under the convention that favors the start, with the first token fixed for both: there the 8-step model stays 0.04 nats/token behind under GPT-2-large and 0.02 under Pythia, whose interval includes zero, and 0.06 behind under both judges with per-step temperature schedules (Table 27). With the first token drawn, it is level with or ahead of the start in every seed under both judges, with either temperature setting. Its MAUVE at is within 0.02 of the start’s at in every setting. At equal steps it is far ahead, by more than 0.5 nats/token at under either convention. At 16× (, 0.25% of pretraining) every seed gains another 0.03 to 0.08: its 8 steps beat the start’s 16 in every seed with the first token drawn, and with it fixed the seed means are −0.005 (GPT-2-large) and +0.016 (Pythia; Table 27).
Figure 3. Half the steps. (a) 110M: judge NLL of the untouched start at 16 steps per block minus that of each model at 8 (GPT-2-large, content tokens, matched entropy, one temperature per model; > 0: the 8-step model is better). Doubling plans and nominal budget together at about 350 updates narrows the gap (as executed, 8× is 0.18% of pretraining and 16× 0.25%); at 8× the seeds are 0.03 to 0.06 below the start when both fix the first token (orange), the start’s own convention, and at or above it when both draw it; 16× adds 0.03 to 0.08, while every baseline given 8× stays more than 0.7 below in every seed, near or below the start’s own 8 steps. (b) Blinded pairwise raters, win rate of the first model, both models fixing the first token (both orders, ties half, 95% intervals; Table 33). (c) The same gap per seed (markers) and seed mean (bars) under GPT-2-large (filled circles), Pythia-1.4B (open circles) and Qwen2.5-7B (open diamonds): at 1.1B for the 1× fine-tune and the 8× recipe of (a); at 8B (block 32, 1×) for random order and LLaDA’s confidence planner, at 8 steps against the start’s 16 and at 16 against its 32 (Table 17).
| setting | judge | seed 0 | seed 1 | seed 2 | seed mean | 95% interval |
| global | GPT-2-large | +0.053 | +0.011 | +0.002 | +0.022 | [−0.011, +0.054] |
| Pythia-1.4B | +0.076 | +0.040 | +0.025 | +0.047 | [+0.011, +0.083] | |
| Qwen2.5-1.5B | +0.079 | +0.063 | +0.050 | +0.064 | [+0.030, +0.098] | |
| MAUVE at (start at : 0.948) | 0.940 | 0.934 | 0.926 | 0.933 | – | |
| schedule | GPT-2-large | +0.025 | +0.024 | +0.016 | +0.022 | [−0.004, +0.050] |
| Pythia-1.4B | +0.041 | +0.045 | +0.032 | +0.039 | [+0.012, +0.069] | |
| Qwen2.5-1.5B | +0.035 | +0.050 | +0.037 | +0.041 | [+0.012, +0.071] | |
| MAUVE at (start at : 0.952) | 0.951 | 0.949 | 0.945 | 0.948 | – | |
| global, first token fixed | GPT-2-large | −0.032 | −0.058 | −0.038 | −0.043 | [−0.073, −0.009] |
| Pythia-1.4B | −0.012 | −0.038 | −0.018 | −0.023 | [−0.059, +0.017] | |
| MAUVE at (start at : 0.942) | 0.943 | 0.940 | 0.940 | 0.941 | – | |
| schedule, first token fixed | GPT-2-large | −0.060 | −0.066 | −0.053 | −0.060 | [−0.085, −0.035] |
| Pythia-1.4B | −0.057 | −0.064 | −0.051 | −0.057 | [−0.082, −0.030] | |
| MAUVE at (start at : 0.947) | 0.946 | 0.952 | 0.958 | 0.952 | – | |
| 16×, global, first token drawn | GPT-2-large | +0.100 | +0.058 | +0.078 | +0.079 | [+0.047, +0.113] |
| Pythia-1.4B | +0.126 | +0.087 | +0.105 | +0.106 | [+0.071, +0.144] | |
| MAUVE at (start at : 0.948) | 0.945 | 0.933 | 0.954 | 0.944 | – | |
| 16×, global, first token fixed | GPT-2-large | +0.012 | −0.024 | −0.004 | −0.005 | [−0.036, +0.028] |
| Pythia-1.4B | +0.034 | −0.005 | +0.019 | +0.016 | [−0.020, +0.056] | |
| MAUVE at (start at : 0.942) | 0.951 | 0.942 | 0.949 | 0.947 | – |
Table 27.
We also put the comparison to blinded pairwise raters from two model families, ChatGPT and Claude Sonnet, with the first token fixed (Figure 3b; protocol in Section J). Both pass a built-in control that an open-weights Qwen2.5-7B rater failed. The start’s text wins only 6 to 7% of its pairs against its own text; the fine-tuned model at wins 45% with the raters pooled, so it recovers 88% of the win rate lost by halving the steps (95% interval 79 to 97%; 90% with the first token drawn). Against the start at ChatGPT’s reading is inconclusive (0.47) and Claude’s is worse (0.43). At equal steps both raters prefer the fine-tuned model’s text, in 76 and 79% of pairs and in every seed, on coherence, readability and substance (88% with the first token drawn; Table 33).
Table 33. Rater runs, 21 to 31 October 2026 (contrasts in Table 32). ChatGPT names are the product names shown in its interface; the Claude helpers ran claude-sonnet-5-5, at maximum effort in every run we use. The upper block rates samples with the first token fixed on both sides, the start’s convention; the lower blocks rate the earlier samples with it drawn. (a) : test side preferred, ties half, with its 95% interval; per seed: for fine-tune seeds 0, 1 and 2 (c0 compares the start with itself and has no seeds); : control rate; cons.: share of pairs with the same outcome in both display orders; agr.: share of the contrast’s judgments on which the Claude and ChatGPT runs give the same overall verdict; share: the share of the halving gap recovered, , with its 95% interval. (b) in c3 on each criterion of the rubric, with its 95% interval.
| (a) all runs | |||||||
| rater | contrast | per seed | cons. | agr. | reading | ||
| first token fixed (27 to 31 October) | |||||||
| Claude Sonnet 5.5 helpers | c1 | 0.428 [0.385, 0.472] | 0.485, 0.415, 0.384 | 0.831 [0.783, 0.878] | 0.78 | 76% | worse; share 0.84 [0.74, 0.94] |
| c0 | 0.055 [0.032, 0.081] | – | 0.984 [0.953, 1.000] | 0.95 | 95% | ||
| c3 | 0.786 [0.710, 0.855] | 0.848, 0.781, 0.730 | 0.987 [0.971, 1.000] | 0.88 | 89% | preferred | |
| ChatGPT “6.1 Sol Ultra” | c1 | 0.468 [0.425, 0.513] | 0.515, 0.455, 0.435 | 0.850 [0.802, 0.896] | 0.80 | 76% | inconclusive; share 0.93 [0.82, 1.03] |
| c0 | 0.072 [0.046, 0.102] | – | 0.984 [0.953, 1.000] | 0.93 | 95% | ||
| c3 | 0.758 [0.680, 0.829] | 0.805, 0.770, 0.699 | 0.964 [0.927, 0.990] | 0.87 | 89% | preferred | |
| both raters pooled | c1 | 0.448 [0.409, 0.490] | 0.841 | share 0.88 [0.79, 0.97] | |||
| c0 | 0.064 [0.041, 0.089] | 0.984 | |||||
| first token drawn (21 to 26 October) | |||||||
| Qwen2.5-7B-Instruct (a09a3545) | c1 | 0.500 [0.498, 0.503] | 0.499, 0.501, 0.502 | 0.503 [0.495, 0.510] | 0.003 | – | fails control (second text chosen in 99.2%); rated all 1,536 main pairs |
| Claude Sonnet 5.5 helpers | c1 | 0.416 [0.371, 0.462] | 0.450, 0.401, 0.396 | 0.888 [0.848, 0.926] | 0.82 | 79% | worse; share 0.79 [0.68, 0.91] |
| c0 | 0.091 [0.059, 0.127] | – | 0.902 [0.832, 0.961] | 0.96 | 95% | ||
| c3 | 0.882 [0.820, 0.934] | 0.879, 0.906, 0.859 | 0.898 [0.844, 0.948] | 0.94 | 90% | preferred | |
| ChatGPT “Astra 6 Ultra” | c1 | 0.502 [0.455, 0.547] | 0.550, 0.484, 0.472 | 0.893 [0.849, 0.932] | 0.81 | 79% | noninferior; share 1.00 [0.89, 1.12] |
| c0 | 0.089 [0.057, 0.122] | – | 0.930 [0.859, 0.984] | 0.97 | 95% | ||
| ChatGPT “6.1 Sol Ultra” | c3 | 0.878 [0.823, 0.927] | 0.898, 0.871, 0.863 | 0.909 [0.854, 0.956] | 0.91 | 90% | preferred |
| both raters pooled (mean of the four verdicts of a pair) | c1 | 0.459 [0.416, 0.501] | 0.891 | share 0.90 [0.80, 1.00] | |||
| c0 | 0.090 [0.059, 0.123] | 0.916 | |||||
| Claude Sonnet 5.5, pilot’s prompt (superseded) | c1 | 0.453 [0.408, 0.498] | 0.491, 0.436, 0.431 | 0.876 [0.835, 0.914] | 0.77 | 78% | inconclusive; share 0.88 [0.76, 0.99] |
| c0 | 0.106 [0.073, 0.143] | – | 0.895 [0.820, 0.957] | 0.93 | 93% | ||
| ChatGPT re-rating 192 of its own earlier judgments gave the same verdict on 90%. | |||||||
| (b) c3 by criterion | |||||||
| rater | coherence | readability | substance | repetition | |||
| first token fixed | |||||||
| Claude Sonnet 5.5 helpers | 0.772 [0.699, 0.837] | 0.824 [0.767, 0.876] | 0.728 [0.659, 0.792] | 0.483 [0.422, 0.546] | |||
| ChatGPT “6.1 Sol Ultra” | 0.738 [0.656, 0.815] | 0.876 [0.816, 0.928] | 0.738 [0.656, 0.815] | 0.514 [0.427, 0.602] | |||
| first token drawn | |||||||
| Claude Sonnet 5.5 helpers | 0.867 [0.809, 0.918] | 0.908 [0.855, 0.951] | 0.831 [0.772, 0.882] | 0.474 [0.410, 0.540] | |||
| ChatGPT “6.1 Sol Ultra” | 0.872 [0.816, 0.922] | 0.928 [0.878, 0.969] | 0.859 [0.803, 0.909] | 0.457 [0.375, 0.542] |
caption:
Under a confidence planner the ordering is the same. LLaDA’s default low-confidence remasking (Nie et al., [22]) samples every masked token at each step and keeps the most probable. Mix-FT never trained on it, but at 110M it beats every baseline and the start at , 8 and 16 under this planner too, and the 8× model at beats the start’s confidence sampler at in every seed, by 0.02 to 0.07 nats/token with the first token fixed and at least 0.15 with it drawn (Table 12).
| model | MAUVE, random order | MAUVE, random order | MAUVE, random order | MAUVE, confidence planner | MAUVE, confidence planner | MAUVE, confidence planner | judge NLL gap | judge NLL gap | judge NLL gap |
| Start (BD3-LM) | 0.04 | 0.55 | 0.95 | 0.01 | 0.11 | 0.72 | 0.04 | 0.56 | 0.50 |
| Continued CE | 0.08 | 0.48 | 0.90 | 0.04 | 0.22 | 0.73 | −0.01 | 0.45 | 0.42 |
| Alpha () | 0.17 | 0.68 | 0.92 | 0.06 | 0.29 | 0.73 | 0.14 | 0.47 | 0.41 |
| SDTT-8 | – | 0.52 | 0.92 | 0.01 | 0.06 | 0.51 | – | 0.60 | 0.52 |
| Mix-FT 1×, seed 0 | 0.18 | 0.89 | 0.96 | 0.06 | 0.54 | 0.91 | 0.20 | 0.51 | 0.45 |
| Mix-FT 1×, seed 1 | 0.19 | 0.88 | 0.96 | 0.07 | 0.52 | 0.91 | 0.21 | 0.48 | 0.45 |
| Mix-FT 1×, seed 2 | 0.14 | 0.86 | 0.95 | 0.06 | 0.54 | 0.91 | 0.19 | 0.46 | 0.41 |
| Mix-FT 8×, seed 0 | 0.38 | 0.94 | 0.95 | 0.16 | 0.91 | 0.96 | 0.28 | 0.38 | 0.28 |
| Mix-FT 8×, seed 1 | 0.37 | 0.93 | 0.95 | 0.14 | 0.87 | 0.96 | 0.23 | 0.33 | 0.26 |
| Mix-FT 8×, seed 2 | 0.36 | 0.93 | 0.95 | 0.14 | 0.87 | 0.96 | 0.27 | 0.35 | 0.26 |
| Start, fixed | 0.04 | 0.62 | 0.94 | 0.02 | 0.17 | 0.77 | – | – | – |
| Continued CE, fixed | 0.09 | 0.54 | 0.91 | – | – | – | – | – | – |
| Mix-FT 1×, seed 0, fixed | 0.18 | 0.86 | 0.96 | 0.07 | 0.52 | 0.91 | – | – | – |
| Mix-FT 8×, fixed | – | 0.94 | 0.95–0.96 | – | 0.84–0.88 | 0.96 | – | – | – |
Table 12. MAUVE at 110M under the random-order sampler and under LLaDA’s low-confidence planner, and the judge gap between the two planners. MAUVE uses GPT-2-large features with the boundary tokens removed (Section A), at matched per-sample entropy, with one global temperature per model and 512 samples per file. The gap is the same model’s content-token GPT-2-large NLL under the confidence planner minus its NLL under random order, at matched entropy (nats/token; positive: the confidence planner is worse). Pythia-1.4B gives the same signs, except for continued CE at , where both judges find the two planners level. Continued CE, Alpha and SDTT-8 are seed 0. Rows marked fixed fix the first token to end-of-text (Section B); the others draw it. –: not measured.
Compute alone does not close the gap. More compute without more plans does not help Mix-FT: at 3× the budget with or 32, it gains under 0.05 nats/token at (Table 7b). More compute does not help the baselines either. Given the full 8× budget (three seeds each), continued CE, Alpha diffusion on four-step and on two-step states, and SDTT-8, with teacher passes counted and checkpoints chosen on a development split, all stay more than 0.7 nats/token behind the start at , and their 8-step samples get worse or stay level, except Alpha on two-step states in one seed (0.13 to 0.16 better than 1× Alpha at , still 0.72 behind; Figure 3a, Table 28). They see 65 to 129 times as many distinct sequences as the 8× Mix-FT, and their bound barely moves: more data and more updates do not reproduce what the objective gives. On text8, continued CE given the compute of 256 plans, as 769 times the updates, reaches a much lower exact NLL than Mix-FT but worse samples (seed 0; Table 22c).
| arm | recipe | steps | FE | warmup | sequences | gain | |
| continued CE | CE at the sampler’s states, lr cosine to 10% over all steps | 46,080 | 2,211,840 | 100 | 368,640 | ||
| seed 1 | as seed 0 | 46,080 | 2,211,840 | 100 | 368,640 | ||
| seed 2 | as seed 0 | 46,080 | 2,211,840 | 100 | 368,640 | ||
| Alpha | as CE, Alpha loss , centering constant from the start model () | 46,080 | 2,211,840 | 100 | 368,640 | ||
| seed 1 | as seed 0, with its own constant () | 46,080 | 2,211,840 | 100 | 368,640 | ||
| seed 2 | as seed 0, with its own constant () | 46,080 | 2,211,840 | 100 | 368,640 | ||
| SDTT-8 | one round, teacher the start at 16 steps per block, student 8; reverse KL, lr constant after warmup, EMA 0.999 | 30,720 | 2,211,840 | 500 | 245,760 | ||
| seed 1 | as seed 0; evaluated at step 7,680 | 30,720 | 2,211,840 | 500 | 245,760 | ||
| seed 2 | as seed 0; evaluated at step 15,360 | 30,720 | 2,211,840 | 500 | 245,760 | ||
| Alpha-2 | as Alpha, at the sampler’s states, with the constant measured there (); evaluated at step 23,040 | 46,080 | 2,211,840 | 100 | 368,640 | ||
| seed 1 | as seed 0, with its own constant (); evaluated at step 11,520 | 46,080 | 2,211,840 | 100 | 368,640 | ||
| seed 2 | as seed 0, with its own constant (); evaluated at step 11,520 | 46,080 | 2,211,840 | 100 | 368,640 | ||
| Mix-FT (Table 26) | , | 357 | 2,210,544 | 44 | 2,856 |
Table 28. The baselines. FE as in § 17; SDTT counts its teacher’s passes (9 FE per sequence). : GPT-2-large / Pythia content, against the start at . Gain: the same comparison at against the arm’s own run in the same seed (Mix-FT: the main model; Alpha-2 in seeds 1 and 2: the seed-0 run, the only one). Rows without a seed are seed 0.
MAUVE and diversity agree with the judges. MAUVE (Pillutla et al., [24]) on GPT-2-large features with the boundary tokens removed (Section A) is 0.86 to 0.89 for Mix-FT at , against 0.68 for Alpha diffusion and 0.55 for the start (0.86 against 0.62 in seed 0 with the first token fixed; Table 12). The 8× model at is also more diverse across samples than the start at (Table 9), with a 6 to 9% higher Vendi score (Friedman & Dieng, [9]) on GPT-2-large features and a lower self-BLEU-4 in every seed, with the first token drawn or fixed. At equal steps it repeats more 4-grams per sample than the start (6.3% against 3.8%; real text 3.6%), which the raters did not penalize and the judge NLL barely notices (Section A).
| model | Vendi | self-BLEU-4 | distinct-4 | rep-4 (%) | |
| Real held-out text | – | 172 | 0.341 | 0.898 | 3.6 |
| Start (BD3-LM) | 8 | 115 | 0.292 | 0.923 | 3.1 |
| Start (BD3-LM) | 16 | 137 | 0.353 | 0.891 | 3.8 |
| Continued CE | 16 | 129 | 0.358 | 0.892 | 3.6 |
| Alpha diffusion | 16 | 133 | 0.363 | 0.869 | 5.3 |
| SDTT-8 | 16 | 138 | 0.339 | 0.916 | 2.5 |
| Continued CE 8× | 8 | 100 | 0.280 | 0.928 | 3.0 |
| Mix-FT 1× | 8 | 133–136 | 0.316–0.323 | 0.900–0.902 | 4.1–4.4 |
| Mix-FT 1× | 16 | 147–148 | 0.365–0.373 | 0.866–0.867 | 5.4–5.6 |
| Mix-FT 8× | 8 | 146–148 | 0.319–0.334 | 0.891 | 4.7–5.1 |
| Mix-FT 8× | 16 | 147–152 | 0.363–0.370 | 0.858–0.859 | 6.2–6.4 |
| Start, fixed | 8 | 115 | 0.302 | 0.918 | 3.3 |
| Start, fixed | 16 | 136 | 0.363 | 0.887 | 3.9 |
| Mix-FT 8×, fixed | 8 | 144–148 | 0.326–0.336 | 0.890–0.891 | 4.7–5.1 |
Table 9. Diversity across samples and repetition within them, at matched per-sample entropy (110M, block 16, 512 samples per file; each value is the mean over four entropy levels, from real text’s down to 0.3 nats below it). Vendi score on GPT-2-large features (higher: more diverse); self-BLEU-4 (lower: more diverse); distinct-4 over the set of samples; rep-4, the share of 4-grams repeated within a sample. Mix-FT ranges are over three seeds; baselines are seed 0, at 1× unless marked. Rows marked fixed fix the first token to end-of-text for both models (Section B); the others draw it. No sample of any model here has a near duplicate in its file (8-gram Jaccard similarity above 0.5).
At 8× the likelihood bound is level with the start’s. With 256 evaluation plans the 8× model’s bound is within about 0.01 nats/token of the start’s at and 16, while its NELBO perplexity rises to 28.8 and its plan weights concentrate on an effective 4 to 6 of the 256 plans (Table 11). Its bound still tightens faster than the start’s as plans are added, so its exact likelihood is not known. On text8, where it is exact, more plans at fixed updates lower the sampler’s NLL in every seed up to and then flatten, while the gap between the plan bound and the exact NLL widens by 0.03 bpc (Table 22c): a loose bound can hide a real gain.
| model | IWAE-64 | IWAE-64 | IWAE-256 | IWAE-256 | ESS of 256 plans | ESS of 256 plans | tightening | tightening | NELBO |
| Start (BD3-LM) | 3.027 | 2.966 | 3.008 | 2.955 | 10.8 | 18.0 | 0.002 | 0.001 | 3.135 |
| Continued CE | 3.010 | 2.943 | 2.993 | 2.933 | 11.7 | 19.6 | reference | 3.109 | |
| Continued CE 8× | 3.001 | 2.934 | 2.984 | 2.924 | 11.8 | 19.8 | 0.000 | 0.000 | 3.100 |
| Mix-FT 1× | 2.994 | 2.940 | 2.969 | 2.924 | 6.6 | 9.7 | 0.007 | 0.006 | 3.199 |
| Mix-FT 8×, seed 0 | 3.034 | 2.991 | 2.995 | 2.963 | 4.4 | 5.9 | 0.022 | 0.018 | 3.361 |
| Mix-FT 8×, seed 1 | 3.032 | 2.989 | 2.994 | 2.962 | 4.5 | 6.0 | 0.021 | 0.017 | 3.356 |
| Mix-FT 8×, seed 2 | 3.035 | 2.992 | 2.996 | 2.964 | 4.5 | 5.9 | 0.022 | 0.018 | 3.362 |
Table 11. Likelihood bounds of the 8× model at 110M (nats/token, on 256 held-out sequences). IWAE- bounds the -step random-order sampler with evaluation plans, at each model’s best of ; IWAE-64 is the mean over four disjoint groups of 64 of the 256 plans. The NELBO is the per-path bound of the one-token-per-step sampler. ESS: median effective sample size of the 256 self-normalized plan weights at . Tightening: how much more the bound falls than continued CE’s when the evaluation plans go from 64 to 256. Mix-FT 1× is the two-step model; it and both continued CE models are seed 0.
At scale: LLaDA-8B
We fine-tuned LLaDA-8B-Base (Nie et al., [22]) with LoRA on OpenWebText for prefix continuation, with three seeds per arm, and sampled it with its own sampler: after the prompt, six blocks of 32 tokens from left to right, with LLaDA’s default low-confidence remasking or with random order (Section C). Mix-FT uses our 1× recipe (, four-step plans) and beats continued CE and Alpha diffusion at the same compute at every step count, in every seed under three judges and both planners, by 0.22 to 0.60 and 0.18 to 0.40 nats/token, though their IWAE-8 bounds are within 0.011. At equal steps it beats the start by 0.50 to 0.85 at and 16 (Table 2).
Half the model evaluations. At , two tokens per step as in our 110M comparison, Mix-FT is ahead of LLaDA
Table 2. LLaDA-8B fine-tuned with LoRA for 0.0018% of its pretraining compute (block 32, 192-token continuations of a 64-token prompt). Top: GPT-2-large NLL at matched per-sample entropy (nats/token, 3-seed mean, lower is better); at the start reveals one token per step. Middle: the start at minus Mix-FT at , seed mean and 95% interval (positive: Mix-FT better). Bottom: share of 288 blinded pairwise judgments preferring Mix-FT at to the start at , low-confidence decoding only (ties half, 95% interval; Section C). Per seed and : Table 17 and Table 18.
Table 17. LLaDA-8B-Base fine-tuned with LoRA for OpenWebText prefix continuation (setting in Table 16). (a) Means of 3 seeds (start: one model; : seed 0) of the IWAE-8 bound of the -step random-order sampler (nats/token) and of the GPT-2-large content-token NLL of the continuations at matched per-sample entropy (nats/token, lower is better, 256 continuations per temperature), for each planner. Bold: best entry in the column among the first four rows. (b) Each model at steps against the start at , judge NLL of the start minus the model’s (positive: the model is better), per seed, with the seed mean, its 95% bootstrap interval over prompts, and the verdict of the rule fixed before the runs (Section C).
| random order | GPT-2-large | Pythia-1.4B | Qwen2.5-7B | ||||||||||
| against | s0 | s1 | s2 | mean | s0 | s1 | s2 | mean | s0 | s1 | s2 | mean | |
| Continued CE | 4 | +0.366 | +0.411 | +0.480 | +0.419 | +0.372 | +0.403 | +0.480 | +0.418 | +0.336 | +0.360 | +0.428 | +0.375 |
| 8 | +0.566 | +0.596 | +0.617 | +0.593 | +0.570 | +0.602 | +0.616 | +0.596 | +0.565 | +0.578 | +0.599 | +0.581 | |
| 16 | +0.472 | +0.402 | +0.373 | +0.416 | +0.478 | +0.412 | +0.392 | +0.427 | +0.446 | +0.406 | +0.375 | +0.409 | |
| Alpha | 4 | +0.268 | +0.303 | +0.411 | +0.327 | +0.284 | +0.329 | +0.414 | +0.342 | +0.243 | +0.281 | +0.323 | +0.282 |
| 8 | +0.346 | +0.374 | +0.420 | +0.380 | +0.366 | +0.419 | +0.423 | +0.403 | +0.334 | +0.384 | +0.370 | +0.362 | |
| 16 | +0.247 | +0.322 | +0.221 | +0.263 | +0.268 | +0.348 | +0.255 | +0.290 | +0.238 | +0.342 | +0.238 | +0.273 | |
| Start | 4 | +0.058 | +0.085 | +0.161 | +0.101 | +0.306 | +0.331 | +0.413 | +0.350 | +0.511 | +0.522 | +0.587 | +0.540 |
| 8 | +0.685 | +0.723 | +0.747 | +0.718 | +0.758 | +0.811 | +0.817 | +0.795 | +0.820 | +0.860 | +0.867 | +0.849 | |
| 16 | +0.478 | +0.513 | +0.510 | +0.500 | +0.547 | +0.582 | +0.589 | +0.573 | +0.573 | +0.629 | +0.619 | +0.607 | |
| low confidence | GPT-2-large | Pythia-1.4B | Qwen2.5-7B | ||||||||||
| against | s0 | s1 | s2 | mean | s0 | s1 | s2 | mean | s0 | s1 | s2 | mean | |
| Continued CE | 4 | +0.233 | +0.302 | +0.330 | +0.288 | +0.231 | +0.284 | +0.322 | +0.279 | +0.187 | +0.236 | +0.245 | +0.223 |
| 8 | +0.593 | +0.468 | +0.507 | +0.523 | +0.583 | +0.463 | +0.505 | +0.517 | +0.493 | +0.375 | +0.430 | +0.433 | |
| 16 | +0.460 | +0.415 | +0.480 | +0.452 | +0.468 | +0.420 | +0.489 | +0.459 | +0.441 | +0.380 | +0.464 | +0.429 | |
| Alpha | 4 | +0.208 | +0.260 | +0.277 | +0.248 | +0.229 | +0.307 | +0.308 | +0.281 | +0.115 | +0.227 | +0.208 | +0.183 |
| 8 | +0.387 | +0.348 | +0.405 | +0.380 | +0.393 | +0.374 | +0.420 | +0.396 | +0.289 | +0.267 | +0.309 | +0.288 | |
| 16 | +0.331 | +0.361 | +0.445 | +0.379 | +0.347 | +0.374 | +0.467 | +0.396 | +0.322 | +0.333 | +0.382 | +0.345 | |
| Start | 4 | +0.163 | +0.195 | +0.218 | +0.192 | +0.247 | +0.281 | +0.314 | +0.281 | +0.282 | +0.317 | +0.321 | +0.306 |
| 8 | +0.569 | +0.501 | +0.587 | +0.552 | +0.614 | +0.552 | +0.637 | +0.601 | +0.554 | +0.501 | +0.578 | +0.544 | |
| 16 | +0.648 | +0.601 | +0.682 | +0.644 | +0.703 | +0.655 | +0.738 | +0.699 | +0.681 | +0.639 | +0.711 | +0.677 |
Table 18. Per-seed comparisons at 8B: each arm’s content-token judge NLL minus Mix-FT’s at the same planner and number of steps per block (nats/token, matched per-sample entropy; positive: Mix-FT better). The start is one model, compared with each Mix-FT seed; continued CE and Alpha diffusion are paired by seed.
| model | random order | random order 16 | random order 32 | low confidence | low confidence 16 | low confidence 32 |
| Start | 5.01 | 3.98 | 3.54 | 5.72 | 5.02 | 4.48 |
| Continued CE | 4.88 | 3.89 | – | 5.69 | 4.83 | – |
| Alpha () | 4.67 | 3.74 | – | 5.55 | 4.76 | – |
| Mix-FT (ours) | 4.29 | 3.48 | – | 5.17 | 4.38 | – |
| start at minus Mix-FT at | ||||||
| GPT-2-large | +0.07 [−0.02, +0.15] | +0.10 [+0.01, +0.21] | ||||
| Pythia-1.4B | +0.10 [+0.02, +0.19] | +0.13 [+0.04, +0.23] | ||||
| Qwen2.5-7B | +0.17 [+0.09, +0.25] | +0.13 [+0.04, +0.22] | ||||
| ChatGPT rater | – | 69% [63, 75] | ||||
| Claude rater | – | 71% [66, 77] |
Table 2.
at , one token per step, in every seed under the three judges with either planner, so it writes better continuations with 96 instead of 192 forward passes (Table 2, Figure 3c). Under LLaDA’s own confidence decoding every judge’s interval is above zero, MAUVE rises from 0.63 to 0.73–0.77 (0.85 to 0.95 under random order), and blinded pairwise raters from two model families, ChatGPT and Claude Sonnet, prefer Mix-FT’s continuations in 69 and 71% of 288 pairs, in every seed (Table 2); both pass a built-in control (Section C). At , four tokens per step, Mix-FT closes 66 to 91% of the start’s own gap between 8 and 16 steps in every seed, and a run with 32 plans (seed 0) 78 to 99%, but stays 0.10 to 0.31 behind, where continued CE and Alpha diffusion stay 0.49 to 0.91 behind. At 110M and 1.1B the 8-step result needed 8× the budget, untried at 8B.
1.1B
On the 1.1B SMDM base model (Nie et al., [21]) (LoRA, block 16, under 0.01% of pretraining), Mix-FT also wins at every step count, and with the 110M half-step recipe at 8× its is ahead of the start’s on average under both judges (Table 30).
| model | opt. steps | NELBO | PPL | 4 | 8 | 16 | 4 | 8 | 16 | ||||
| Start (SMDM base) | – | 3.279 | 3.868 | 23.47 | 7.34 | 6.68 | 5.42 | 4.37 | 7.23 | 6.58 | 5.34 | 4.31 | |
| Continued CE | 600 | 2.997 | 3.605 | 17.88 | 7.71 | 6.63 | 5.24 | 4.29 | 7.36 | 6.43 | 5.07 | 4.13 | |
| Alpha () | 600 | 2.999 | 3.613 | 18.43 | 7.59 | 6.48 | 5.16 | 4.37 | 7.36 | 6.35 | 5.04 | 4.27 | |
| Mix-FT () | 600 | 2.990 | 3.598 | 19.37 | 7.45 | 5.99 | 4.46 | 3.75 | 7.16 | 5.80 | 4.31 | 3.61 | |
| Mix-FT () | 339 | 3.024 | 3.629 | 23.00 | – | 5.65 | 4.34 | – | – | 5.48 | 4.18 | – |
Table 30. The 1.1B SMDM base model fine-tuned with LoRA on OpenWebText (setting in Table 29). (a) Means of 3 seeds (start: one model) of the IWAE-32 bound of the -step sampler (nats/token, best of three temperatures), the NELBO perplexity, and the content-token judge NLL at matched per-sample entropy at steps per block (nats/token, lower is better, 256 samples per temperature). Bold: best entry in the column; –: not measured. (b) is the start’s judge NLL minus the model’s (positive: 8 steps better), with the seed mean and its 95% bootstrap interval over samples. MAUVE at matched entropy uses GPT-2-large features of each sample’s first 200 tokens; the start at scores 0.765, and real text scored against real text gives 0.829 at this sample size, so MAUVE differences near the top are noisy. Per-seed gains at equal steps are in Table 31.
Why it works: exact experiments on text8
Protocol. We use text8 (Mahoney, [20]) in sequences of 256 characters, a 4.82M-parameter block 4 diffusion transformer, block 8 and steps (two characters per step) unless stated. Compute is counted in sequence passes; one unit is 691,200 passes. The base model is masked-CE pre-training followed by four units of CE at the sampler’s states (“Base 4×”). Every fine-tune starts from Base 4× and gets one more unit, with three seeds. We report the exact -step NLL in bits per character (bpc) on 2,000 held-out sequences at the best of ten temperatures, and , the gap in a character-level judge’s NLL to continued CE at matched distinct-4, the share of distinct 4-grams among all 4-grams of the samples, along a sweep of ten temperatures (positive: better than continued CE).
A second stage, not a replacement. Used in place of CE from the start of training, the mixture objective is worse on the sampler’s likelihood, by about 0.06 bpc after four units
(Figure 5), with 25× fewer optimizer steps and a gradient that grows noisier once the weights move (Section E). CE learns the conditionals cheaply, and the mixture objective then corrects them.
Figure 5. Exact sampler NLL along the training budget (text8, block 8; thick: mean, thin: seeds). Trained from C0, the mixture objective falls behind CE, and the gap grows with compute. Started from the CE model at 4 units, one more unit of mixture fine-tuning beats one more unit of CE in every seed.
From a CE start it wins. From Base 4×, Mix-FT lowers the exact NLL by 0.039 to 0.043 bpc relative to continued CE in every seed (Table 3 and Figure 6), about three times what continued CE gains from its own fifth unit. Its samples are better at matched diversity in every seed, with equal or slightly higher coverage of real 4-grams (Figure 7). At block 16, where we can only bound the likelihood, the gain on IWAE-64 and IWAE-512 of the same size, and SDTT from the same weights barely moves the bound and gives worse samples than continued CE (Section E).
| arm | NLL | ||||
| Base 4× (start) | 1.804 | ||||
| Continued CE | 1.792 | 0 | 0 | 0 | 0 |
| SDTT | 1.802 | +0.009 | −0.038 | – | +0.009 |
| Per-path FT | 1.782 | −0.010 | +0.014 | – | – |
| Hybrid | 1.764 | −0.028 | +0.076 | – | −0.031 |
| Mix-FT | 1.751 | −0.042 | +0.156 | −0.025 | −0.039 |
Table 3. Text8, block 8, : exact -step NLL (bpc, best temperature, 3-seed mean) and differences against continued CE from the same Base 4× weights at equal extra passes (negative , positive : better). : exact NLL at (one token per step) against CE trained at the states. : block 16, , IWAE-64. Hybrid: Mix-FT plus the CE loss on the same forward passes. Per seed in Section E.
Figure 6. Text8 sampler likelihood relative to continued CE from the same Base 4× weights and extra compute (bars: mean of 3 seeds; markers: seeds). Block 8 uses the exact NLL; block 16 uses IWAE-64.
Figure 7. Text8 quality against diversity, seed 0. Top: judge NLL against distinct-4, one point per temperature. Bottom: judge NLL minus that of continued CE at matched distinct-4; markers at the four levels that define , whose mean is . Lower is better.
The plan weighting is the main cause. Per-path FT keeps everything else in which Mix-FT differs from continued CE (the same sequences, 8 plans and 25 states per sequence, optimizer and 25× fewer steps) and only replaces − log of the plan average by the average of − log, giving every plan weight 1/8. It is better than continued CE but worse than Mix-FT in every seed, so the weighting accounts for about three quarters of the likelihood gain (73 to 80%; Table 20). On samples, Mix-FT beats per-path FT in every seed, while per-path FT’s gain over continued CE is small and inconsistent; at 110M the same control ends behind continued CE on the bound and 0.53 to 0.62 nats/token behind Mix-FT on samples (Table 7c). The weighting also lets plans specialize, which is the plan information that Proposition 2 charges for: the effective sample size of eight plan weights falls from 4.3 under continued CE to 3.6 (Table 20).
Table 20. Text8 mechanism control (, , exact NLL in bpc at the best temperature). Per-path FT uses Mix-FT’s sequences, plans, replicas and step count with equal plan weights. : judge gain at matched distinct-4 against per-path FT. ESS: effective sample size of 8 self-normalized plan weights; gap: per-path bound minus IWAE-8, both on 512 held-out sequences. Lower rows: ablation at equal passes, seed 0.
| arm | seed 0 | seed 1 | seed 2 | vs per-path | ESS / gap (s0) |
| Base 5× (continued CE) | 1.7944 | 1.7946 | 1.7878 | −0.05 to +0.01 | 4.33 / 0.130 |
| Per-path FT (, equal weights) | 1.7839 | 1.7867 | 1.7760 | 0 | 4.37 / 0.129 |
| Mix-FT () | 1.7518 | 1.7552 | 1.7449 | +0.14, +0.14, +0.14 | 3.57 / 0.241 |
| Mix-FT, | 1.7853 | +0.01 | 4.34 / 0.130 | ||
| Mix-FT, | 1.7645 | +0.06 | 4.01 / 0.168 | ||
| Mix-FT, | 1.7553 | +0.14 | 3.75 / 0.210 |
Table 20.
Splitting the gain
At each step reveals one character, so effect (i) is absent. Mix-FT still beats CE trained at the states in every seed, by 0.022 to 0.028 bpc on the exact likelihood, and on samples. The model fine-tuned at is as good at as the one fine-tuned there, so its gain of 0.042 bpc splits into two parts of similar size: 0.025 that persists at one token per step, effect (ii), and 0.016 by which the gap between and narrows, effect (i) (Section E). Widening the network from 4.8M to 43M parameters shrinks the first part by 0.010 to 0.011 bpc in every seed and leaves the second level, as a capacity account of effect (ii) predicts (Table 22a).
Related work
Masked, block and any-order models. Masked diffusion models [26, 29, 17, 22] and block diffusion Arriola et al. [2] train with CE variants optimized by the true conditionals. Any-order autoregressive models [13, 30] are the one-token-per-step case.
Orders as latent variables. Amin et al. [1] analyze the few-step sampler as a mixture over paths; their Theorem 1 contains Proposition 2. They recover much of the gap at sampling time, with temperature 0.9 and float32 instead of float64 categorical sampling, without retraining; the diagnosis is theirs, and we train the denoiser on the mixture. Fox et al. [8] learn a token selector and a variational posterior over unmasking orders that sees the clean sequence, and LO-ARM [32] learns a stochastic order policy with such a posterior for one-token-per-step generation; we learn no order policy and fine-tune the denoiser on a multi-sample bound of a fixed uniform plan distribution. Eso-LMs [27] use a multi-order bound for evaluation and train with one order.
Trajectory-aware training. PUMBA [19] trains on the few-step policy’s own trajectories with a carry state. Its confidence planner and carry state put it outside Proposition 1 and our dynamic program, though not outside Proposition 2 (Section H). T3D [36] distills the model’s own trajectories with a reverse-KL-like loss and MDPO (He et al., 2025) uses policy gradients with intermediate rewards, both under confidence decoding, scoring a sample with its plan. We analyze only the idealized path-space reverse KL with plans independent of : like trajectory balance with a fixed backward policy [28] (Whittamer et al. 2023), it pushes a parallel sampler toward plan-invariant behavior, in exact toys toward independent filling (Section F).
Added latents, heads and teachers. Di4C [11] distills into a mixture of products over an auxiliary variable, E-MoE [14] mixes over expert routes with a single-route bound, VADD [35] adds a Gaussian latent and uses IWAE only for evaluation, and tensor-train heads [15] model the joint of masked tokens. Our mixture latent is the plan the sampler already draws. SDTT [7] matches a few-step student to its many-step self. Alpha diffusion [10] fine-tunes teacher-free with a sequence-level -divergence at one masking state; it changes the divergence, not the plan marginal.
Theory of parallel samplers. With exact conditionals, Wu (2026) separate schedule-averaged from mixture KL, Li et al. (2026) show that a uniform mixture of fixed schedules can beat any deterministic adaptive policy, and Chen et al. [6] analyze random-partition schedules. DUEL [31] computes exact likelihoods for deterministic unmasking. Sampling-only methods [3, 18] are complementary.
Limitations and conclusion
Scale. Our full set of baselines was run at 110M. At 1.1B and 8B we compared with continued CE and Alpha diffusion only, with LoRA and 256-token sequences. At 8B, Mix-FT at stays behind LLaDA at . On a 1.1B GSM8K model fine-tuned further on its own verified solutions, Mix-FT did not differ from continued CE, as the fixed point of the mixture refit predicts for self-generated data (Section D).
Two steps per block. At , SDTT-4’s samples beat Mix-FT’s under both judges in every seed, and SDTT also ranks ahead at matched distinct-4, partly by repairing a newline excess, more than twice real text’s rate, that Mix-FT keeps (Table 6 and Table 8). At 110M the 8× model closes most of the start’s gap between 8 and 16 steps, but its 4-step samples stay about 0.5 nats/token behind the start at 8.
Block size. We tested blocks of 8, 16 and 32, LLaDA-8B’s. With one block over 1,024 tokens, as in MDLM, eight plan weights have a median effective sample size of 1.00, against 1.54 for the start at block 16 and (Table 13), so a bound with few plans is loose.
Price. Mix-FT trades the per-path bound for the sampler’s: the NELBO perplexity rises from 23.0 to 24.5 at and to 28.8 at (Table 11). It costs passes per sequence and needs a CE-trained start: from under-trained weights the mixture refit loses to continued CE (text8 from scratch and a 34M model, Sections A and E).
Conventions. BD3-LM trained on raw windows with the first token clean, we on wrapped ones. Claims against the start use the first-token convention that favors it; the window convention moves BD3-LM’s objective by up to 0.38, and Mix-FT leads on either (Table 7g).
Scope. We train on uniform random-order samplers, and Proposition 1 does not cover confidence planners. We did not run PUMBA, Di4C or added-head methods. Judges and raters are models.
Table 13. Block 32 (exploratory). Top: content-token judge gain of the first model over the second at matched per-sample entropy (GPT-2-large / Pythia-1.4B, nats/token; positive: first model better). Mix-FT trains on four-step plans, with at the budget and at . Rows below the line compare a model at steps per block with the start at . Every row draws the first token, except those where both models fix it to end-of-text (Section B). Bottom: IWAE-32 bound of the four-step sampler, NELBO perplexity and plan ESS (ranges over seeds).
| comparison | seed | |||
| Mix-FT vs continued CE | 0 | 0.258 / 0.243 | 0.407 / 0.409 | 0.412 / 0.413 |
| Mix-FT vs continued CE | 1 | 0.310 / 0.290 | 0.455 / 0.475 | 0.430 / 0.436 |
| Mix-FT vs continued CE | 2 | 0.125 / 0.109 | 0.304 / 0.304 | 0.370 / 0.371 |
| Mix-FT vs start | 0 | 0.265 / 0.264 | 0.267 / 0.296 | 0.255 / 0.278 |
| both fix the first token | 0 | 0.253 / 0.239 | 0.241 / 0.238 | 0.144 / 0.152 |
| continued CE vs start | 0 | 0.006 / 0.021 | −0.140 / −0.113 | −0.157 / −0.135 |
| both fix the first token | 0 | 0.020 / 0.004 | −0.115 / −0.123 | −0.252 / −0.263 |
| Mix-FT vs continued CE | 0 | 0.552 / 0.558 | 0.879 / 0.909 | 0.752 / 0.775 |
| Mix-FT at vs start at | 0 | −1.048 / −0.998 | −0.679 / −0.650 | −0.193 / −0.177 |
| Mix-FT at vs start at | 0 | −0.847 / −0.786 | −0.404 / −0.367 | 0.022 / 0.048 |
| both fix the first token | 0 | −0.097 / −0.094 | ||
| continued CE at vs start at | 0 | −1.399 / −1.344 | −1.283 / −1.275 | −0.731 / −0.727 |
| model | IWAE-32, | NELBO PPL | median ESS of 8 plans, |
| Start at block 16 (reference) | 3.223 | 23.00 | 1.54 |
| Start (block-32 masks) | 3.387 | 23.30 | 1.13 |
| Continued CE (3 seeds) | 3.364 to 3.365 | 22.66 to 22.67 | 1.14 |
| Continued CE (seed 0) | 3.354 | 22.44 | 1.14 |
| Mix-FT (3 seeds) | 3.351 | 23.25 to 23.30 | 1.10 |
| Mix-FT (seed 0) | 3.348 | 24.15 | 1.07 |
Table 13.
Few-step masked diffusion samplers are mixtures over reveal plans, and fine-tuning on that mixture’s likelihood is a cheap, teacher-free change of loss. On LLaDA-8B it puts 16 steps per block ahead of the original’s 32. On a public 110M model it gives the best samples from 4 to 16 steps per block, and with more plans its 8 steps close 95% of the gap to the original’s 16. The two-step regime, where distillation still wins, and full fine-tuning at scale remain open.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning, specifically the efficiency of sampling from diffusion language models. Faster and better few-step generation lowers the cost of running such models, with the same potential for misuse as other text generators. We do not see consequences specific to this method that need discussion here.
The 110M experiments in more detail
Per-seed gaps. Table 6 gives every gain of Mix-FT by seed and judge. From to 16 the smallest gain over any baseline is 0.226, under GPT-2-large against Alpha diffusion at in seed 0. At SDTT-4 is better in every seed, and Mix-FT is level with or slightly ahead of Alpha diffusion.
Controls. Table 7 collects the 110M controls that § 3 summarizes. Removing the plan weights costs Mix-FT more than any other change we tried, and two-step training plans beat finer plans whether the compute or the number of updates is held fixed. More plans per sequence matter less: at equal compute the gain levels off beyond , while doubling the plans together with the compute adds about 0.06 nats/token at per doubling (seed 0).
MAUVE without boundary tokens. The standard MAUVE feature is GPT-2-large’s state at the final token. Every real sequence ends with an end-of-text token and no block-sampler sample does, and with that feature every block-sampler model scores about 0.004 regardless of quality. We drop the first and last position of every sequence, real and generated, before featurizing (real against real: 0.97).
Newline rates. Few-step samplers from the BD3-LM start over-produce newlines, which make up 3.35% of real held-out text (Table 8). Sharper sampling emits more of them, and every model except Alpha diffusion needs to match real text’s per-sample entropy, so for those models the rate at matched entropy, where the judges compare models, is higher than at . Only SDTT stays near or below real text’s rate at both operating points.
All tokens. Scored on every token, newline and end-of-text targets included, every comparison of Mix-FT with the start and the baselines at , 8 and 16 keeps its direction in every seed under both judges. Its margin over the strongest fine-tuned baseline becomes 0.24, 0.43 and 0.40 nats/token at , 8 and 16 under GPT-2-large (content tokens: 0.25, 0.44 and 0.41), and over continued CE 0.51 to 0.68 (content tokens: 0.52 to 0.72). A judge predicts a newline at well under 1 nat, against about 4 for a content token, so all-token scores favor whichever model emits more newlines; scoring content tokens removes that advantage for every model alike.
Diversity and repetition. Table 9 gives the measures behind the summary in § 3. At the 8× model is more diverse than the start at on Vendi and self-BLEU-4 and level on distinct-4. At equal steps Mix-FT has the higher Vendi score but slightly higher self-BLEU-4 and lower distinct-4, and like Alpha diffusion it repeats more 4-grams within a sample than the start or real text. Repetition does not flatter the judges here. Within the 8× model’s files we regressed each sample’s judge NLL on its per-sample entropy (quadratic) and its rep-4: at , more repetition at the same entropy goes with a worse NLL in every seed under both judges, and at it changes the NLL by at most 0.006 nats/token per percentage point. The blinded raters, asked about repetition among other criteria, find it level at equal steps (§ 3).
The four-step Mix-FT. Our first 110M model used four-step plans (443 steps). Its sampler bound was already better than every baseline’s in every seed, including Alpha diffusion at every (Table 4 and Table 10). Its samples beat continued CE on average at every step count, and SDTT-4, SDTT-8 and Alpha diffusion at and 16 in every seed. At fewer steps it was weaker: SDTT-4 was better at and mixed at , and Alpha diffusion was ahead in at least two of three seeds at and 4 with either sample count. Figure 4 and Table 10 collect these results with the 128-sample protocol; Table 1 supersedes them for the two-step model.
| model | NELBO | PPL | GPT-2-L all | GPT-2-L content | Pythia content | ||
| Start (BD3-LM) | 3.223 | 3.775 | 23.00 | ||||
| Continued CE | 3.209 | 3.764 | 22.41 | 0 | 0 | 0 | |
| SDTT-4 | 3.245 | 3.796 | 24.41 | +0.091 | +0.405 | +0.329 | |
| Hybrid | 3.200 | 3.752 | 22.69 | +0.113 | +0.155 | +0.152 | |
| Alpha diffusion | 3.202 | 3.758 | 23.16 | +0.255 | +0.324 | +0.321 | |
| Mix-FT, plans | 3.187 | 3.741 | 23.29 | +0.295 | +0.360 | +0.349 | |
| Mix-FT, plans | 3.181 | 3.723 | 24.51 | +0.442 | +0.577 | +0.563 |
Table 10. Stage 5 to 8 summary at 110M (128 samples per temperature at ). : judge NLL gain over continued CE at matched per-sample entropy, mean over (positive: better than CE); “all” scores every token with GPT-2-large, “content” excludes newline and end-of-text targets. Bounds: IWAE-32, nats/token; seed spreads at most 0.0013.
Figure 4. Four-step Mix-FT at 110M (and 34M FineWeb-Edu). (a) Sampler likelihood relative to continued CE at equal extra compute (bars: mean of 3 seeds; markers: seeds). (b) Content-token judge gain over continued CE at matched per-sample entropy (solid: GPT-2-large, dotted: Pythia-1.4B; from 512 samples). (c) Boundary-free MAUVE at matched entropy (solid: 128 samples, dashed: 512). (d) Newline rate at matched entropy; real text has 3.35%.
Hybrid. Adding the CE loss on the same forward passes () gives about 40% of four-step Mix-FT’s sample gain over continued CE (Table 10). At and 16 it also beats continued CE and SDTT-4 in every seed under both content judges, by 0.14 to 0.35 nats/token under GPT-2-large.
Mix-FT after SDTT. Fine-tuning the SDTT-4 student with Mix-FT, against giving it the same compute of further SDTT, lowers its bound by 0.063 nats/token in every seed and improves its samples at and 16 by about 0.5 nats/token. But it lands almost on the model that Mix-FT reaches from the original weights, within 0.003 on the bound and 0.04 on samples on average, and the nonlinear excess that SDTT had removed returns (Table 8). Mix-FT and SDTT are therefore alternatives rather than stages of one recipe. Further SDTT makes the bound and the samples at slightly worse.
Alpha diffusion. It reweights each block’s CE by , where is the model’s mean log-likelihood of the block’s masked tokens. It is a strong baseline: it beats continued CE at every , most at (+0.47 content, 128 samples), but its gain fades with more steps (+0.10 to +0.11 at ), where it falls below the untuned start. Larger degrades the
In (a), “443 steps” and “1,152 steps” are separate runs with their own learning-rate schedule over that many steps, not snapshots of a longer run. In (b), the run at 3× continues the 1× run for 2,304 more steps with a fresh warmup and cosine; the run at 3× is one run of 1,016 steps. In (c), per-path FT recovers none of Mix-FT’s gain on sampler bound: its gain over continued CE is −0.14 of Mix-FT’s for four-step plans and −0.07 for two-step plans. In (d), schedules are chosen at development split and scored on fresh samples. In (e), one run uses a tenth of the peak learning rate and the other is evaluated with an exponential moving average of its weights (decay 0.999). In (f), ‡ evaluated at the checkpoint chosen on a development split: half of Alpha’s 46,080 steps on two-step states in seed 0 and a quarter in seeds 1 and 2, all of SDTT-8’s 30,720 in seed 0, a quarter in seed 1 and half in seed 2; SDTT-8 counts its teacher’s passes. Rows in (f) pair each seed of the 8× Mix-FT with the same seed of the baseline. In (g), the null fine-tune inputs BD3-LM’s raw training objective (its loss and clipped masking schedule, the first position kept clean for both arms, the null is 0.24 to 0.40 behind the start on average and Mix-FT 0.54 to 0.91 ahead of the null. On our windows and with the first token fixed, the null is level with the start at and 0.18 to 0.23 behind it at and 16. @page 13/41 starts_mid=0 ends_mid=0
model: its bound is 3.466 at and 5.706 at , against 3.202 at . Smaller values do not help: is worse than 0.5 on samples at every , and is worse at and 8 and about level at and 16 (seed 0).
A 34M model trained by us. The recipe needs a CE-trained start. We trained a 34M block diffusion model (8 layers, width 384, 256 GPT-2 tokens, block 16) on 150M tokens of FineWeb-Edu (Penedo et al., [23]) with the text8 protocol. From Base 1×, where a further unit of CE still lowered the bound by 0.12 nats/token, both mixture arms ended 0.022 nats/token behind continued CE (one seed). From Base 2× both beat continued CE in all three seeds, on the bound and on GPT-2-large NLL at matched entropy, and Mix-FT () gains 0.030 to 0.040 nats/token on the bound, about twice the hybrid’s gain (Figure 4). At matched distinct-4 the gains are near zero on average at this scale.
Bounds of the 8× model. Table 11 gives the bounds of the 8× model and its comparators with up to 256 evaluation plans. With all 256, the 8× model is level with continued CE at and about 0.03 nats/token behind it at , in every seed. Its plan weights are much more concentrated than continued CE’s, so its bounds tighten faster as plans are added, and the gap at is still closing at 256 plans. The NELBO, which weights every plan equally, is the loosest bound and the one Mix-FT raises most. The 1× two-step model is below continued CE and the start at both step counts. Its bound also tightens faster, so its margin over continued CE at is larger with 256 plans than with the 64 plans of § 3.
Confidence sampler. Each step draws a token for every masked position from softmax( logits), scores it by its untempered probability, and keeps the highest. This is LLaDA’s low_confidence rule, and one step matches LLaDA’s reference code in distribution. Temperatures run from to 1.15 so that every model’s sweep brackets real text’s per-sample entropy, and each file has 512 samples. Mix-FT has three seeds and the baselines one. Table 12 gives MAUVE under both planners and the judge gap between them. On this checkpoint the confidence planner is worse than random order at matched entropy for every model at and 16, by at least 0.25 nats/token, and at for every model except continued CE, which is level. So under this planner the model at beats the start’s confidence sampler at but stays 0.29 to 0.35 nats/token behind the start’s random-order sampler, and at it does not reach the start’s confidence sampler at .
Block 32 (exploratory). No BD3-LM checkpoint at block 32 is public, so we ran the block-16 start with block-32 masks, which raises its NELBO perplexity only from 23.00 to 23.30. Before training, the median effective sample size of eight plan weights at was 1.13, below the 1.25 our pilot rule asked for, so we ran the fine-tunes as exploratory follow-ups (Table 13). For the start at block 16 the same median is 1.32, 1.54 and 1.95 at , 4 and 8 (1.06, 1.13 and 1.30 at block 32), and for MDLM’s single block of 1,024 tokens it is 1.00 at and 128. At the 1× budget, with 443 Mix-FT updates at and , Mix-FT beats continued CE at , 8 and 16 under both judges in every seed and has the better bound. In seed 0 it also beats the start at every step count, by 0.14 to 0.30 nats/token with the first token drawn or fixed, while continued CE falls behind the start at and 16 either way. With 8× the budget and (seed 0), Mix-FT at , two tokens per step, matches the start at under both judges when both draw the first token (+0.02 and +0.05), but falls 0.10 nats/token behind under both when both fix it to end-of-text, the start’s better convention here; fixing it helps the start at by 0.11 to 0.13 and leaves the 8× model where it was (Section B). It beats continued CE given the same budget by at least 0.55 nats/token at , 8 and 16.
The first token of a sequence
BD3-LM is trained on 1,024-token windows cut from concatenated documents, each document followed by end-of-text. Position 0 of a window is never masked and never scored, and BD3-LM’s sampler sets it to end-of-text before generating. Our fine-tunes follow MDLM’s data pipeline instead: every window begins and ends with end-of-text, and position 0 is masked and predicted like any other position. Our sampler draws position 0 for every model. The start therefore draws its first token from an output that never received a gradient, and the rest of the first block, and through the cache every later block, conditions on that token.
We sampled every model of the main comparison a second time with the first token fixed to end-of-text from the first step, with the same plans and random draws, so that the two files of each pair differ only in that token: the step whose group contains position 0 still draws a token there, which is discarded, and reveals one fewer position. Regenerated with the first token drawn, all 122 files reproduce the files of the main text token for token. Table 14 gives what the fix changes. It helps the start by 0.05 to 0.11 nats/token at and 16 and costs it 0.04 to 0.07 at . The fine-tuned models, trained to predict end-of-text at position 0, move by at most 0.07 and mostly by less than 0.03, whether they draw end-of-text there often (continued CE, in 56 to 64% of samples) or rarely (Mix-FT in 16 to 21%, the model in 1 to 3%); the start draws it in at most 0.2%.
| gain from fixing the first token | G | P | G | P | G | P |
| Start (BD3-LM) | −0.070 | −0.035 | +0.055 | +0.091 | +0.087 | +0.111 |
| Continued CE, seed mean | +0.005 | +0.000 | −0.013 | −0.002 | +0.015 | +0.019 |
| Mix-FT 1×, seed mean | −0.044 | −0.021 | −0.002 | +0.010 | −0.016 | −0.004 |
| Mix-FT 8×, seed mean | – | – | +0.022 | +0.042 | −0.002 | +0.016 |
| with per-step schedules | – | – | −0.003 | +0.014 | – | – |
| comparison | first token drawn | first token fixed | ||
| G | P | G | P | |
| Mix-FT 8× at vs start at , seed mean | +0.022 | +0.047 | −0.043 | −0.023 |
| with per-step schedules | +0.022 | +0.039 | −0.060 | −0.057 |
| share of the start’s gap between and 16 closed | 102% | 105% | 95% | 97% |
| Mix-FT 8× at vs start at $K = 16, seed mean | +0.633 | +0.664 | +0.545 | +0.569 | ||||
| Mix-FT 1× vs start, seed mean, | +0.563 | +0.581 | +0.588 | +0.595 |
| +0.496 | +0.518 | +0.439 | +0.437 | |
| +0.341 | +0.363 | +0.238 | +0.248 | |
| Continued CE vs start, seed mean, | −0.096 | −0.067 | −0.021 | −0.031 |
| −0.218 | −0.204 | −0.286 | −0.297 | |
| −0.178 | −0.161 | −0.250 | −0.254 |
Table 14. The first token. Top: judge NLL with the first token drawn minus with it fixed to end-of-text, for the same model, and random draws, at matched per-sample entropy (content tokens, nats/token; positive: fixing helps). Judging the drawn files with end-of-text put in place of their first token, so that only the samples differ, changes these by at most 0.014. Bottom: comparisons with the start, both models drawing or both fixing the first token (positive: first model better). G: GPT-2-large, P: Pythia-1.4B. Per-seed values and intervals for the 8× model are in Table 27; with the first token fixed, the 1× model beats the start by 0.54 to 0.62, 0.42 to 0.46 and 0.22 to 0.26 nats/token at , 8 and 16 in every seed under both judges.
Every comparison among fine-tuned models in the paper uses one convention for both sides and stands as reported. Comparisons with the start are stated under the convention that favors the start, with both numbers where they differ. At against the start at , the model closes 95% (GPT-2-large) and 97% (Pythia) of the start’s own gap between 8 and 16 steps with the first token fixed, and all of it with the first token drawn. The 1.1B results are not affected: the SMDM base model was trained to predict position 0, and our 1.1B pipeline follows its convention.
Bounds. Our bounds score all 1,024 positions of the held-out windows, including position 0, which the start never learned to predict, and position 1,023, which holds end-of-text only in our windows. BD3-LM’s validation scores windows without end-of-text tokens and leaves position 0 out. Table 15 gives the bound differences to the start under both scorings. Three comparisons change sign, continued CE’s IWAE-32 and NELBO and the model’s IWAE-32, and all three are within 0.01 nats/token of the start’s under BD3-LM’s scoring. The Mix-FT model’s bound stays below the start’s under both.
| model | IWAE-32, | NELBO | ||
| ours | BD3-LM’s | ours | BD3-LM’s | |
| Continued CE, seed 0 | −0.014 | +0.010 | −0.026 | +0.002 |
| Mix-FT 1×, seed 0 | −0.043 | −0.024 | +0.064 | +0.083 |
| Mix-FT 8×, seeds 0, 1, 2 | −0.010 to −0.009 | +0.002 to +0.003 | +0.221 to +0.227 | +0.229 to +0.234 |
Table 15. Bound minus the start’s bound (nats/token; negative: better than the start) under our scoring of the held-out windows (Table 1) and under BD3-LM’s (no end-of-text inside the windows, position 0 left out).
LLaDA-8B
Setting. Table 16 gives the setting. We fine-tune LLaDA-8B-Base [22] with LoRA for prefix continuation on OpenWebText and sample it with its own semi-autoregressive sampler. The 256 positions after BOS form eight blocks of 32; the prompt fills the first two, and the model generates the other six from left to right with steps per block, one full bidirectional forward per step, the later blocks masked and no KV cache. We sample with both of LLaDA’s planners: random order, the sampler Mix-FT trains on, and LLaDA’s default low-confidence remasking, which at each step draws a token for every masked position of the block and keeps the most probable. Every arm starts from the same weights and gets the same 480,000 forward-equivalents, 0.0018% of the pretraining compute, on the same windows. Continued CE and Alpha diffusion train at the four-step random-order sampler’s states, as at 1.1B. Mix-FT trains on four-step plans with , as in our block-32 runs (Table 13), because at block 32 a two-step plan reveals 16 tokens per step. A second Mix-FT run with (seed 0) tests whether more plans help at equal compute, since the start’s plan weights are nearly degenerate at four steps (median ESS of 8 plans 1.19 on the evaluation windows). Before any training, our implementation of the network was checked against the Hub’s own code and LLaDA’s likelihood estimator (Table 16).
| start | LLaDA-8B-Base (Nie et al., [22]), GSAI-ML/LLaDA-8B-Base, revision 0f2787f2, six bf16 shards (sha256 checked); 8.0B parameters, pretrained on 2.3T tokens; vocabulary 126,464 |
| port | our implementation, checked against the Hub’s remote code (transformers 4.46.3) before any training: on 4,104 masked positions its fp32 logits agree within and its masked CE to nats; in fp16, the mode used for training and sampling, its masked CE is within nats of the fp32 reference; LLaDA’s own likelihood estimator gives the same mean on 16 prompt–continuation pairs to 0.002 nats (Monte Carlo SE 2.5) |
| data | OpenWebText, re-tokenized with LLaDA’s tokenizer, documents joined with EOS; 65,536 training windows of 256 tokens from shard 0; evaluation on 256 windows from the last 3,000 documents of shard 79; prompt: BOS and the window’s first 64 tokens; the model generates the other 192 |
| LoRA | rank 32, alpha 32, fp32 adapters on all attention and MLP linears (83,886,080 parameters); base frozen in fp16 with an fp32 residual stream, layers split at 16 over the two GPUs of a T42; per-layer gradient checkpointing |
| budget | FE: a 257-position pass counts 1, or 3 with a backward. Mix-FT: , , 8 examples (window, block) per update with 25 rows each, a plan-weight pass without gradient and a weighted gradient pass (4 FE per row), 600 updates, 480,000 FE; Mix-FT (seed 0): 4 examples per update, 309 updates, 479,568 FE; continued CE and Alpha (): 267 rows per update at the four-step random-order states, 600 updates, 480,600 FE. Each is 0.0018% of pretraining, counting a forward and a backward for each of the 2.3T pretraining tokens |
| optimizer | AdamW (0.9, 0.999), no weight decay, lr , 5% warmup, cosine to 0, clip 1, fp16 loss scaling from , micro-batch 8 rows; window order seed , LoRA init seed |
| temperatures | per model, planner and : a bracketing pass on 32 prompts, then four main temperatures where the entropy curve reaches real text’s per-sample entropy , , and nats, 256 continuations each, and further files until real text’s entropy and 0.3 nats below it are both bracketed; the start’s sweep is seeded from its sweep |
| judges | GPT-2-large, Pythia-1.4B and Qwen2.5-7B (read) and Qwen2.5-1.5B (reported) on the decoded text, re-tokenized, with the prompt as context; content tokens only; Qwen2.5-7B in fp16 after an fp32 check; MAUVE on GPT-2-large features of prompt and continuation |
| bound | IWAE-8 of the block-32 random-order sampler at , 8, 16 on the 256 evaluation windows (one generated block per window), with the median effective sample size of the 8 plan weights |
Table 16. 8B setting (LLaDA-8B-Base, LoRA, OpenWebText prefix continuation).
Evaluation. Each model, planner and gets its own temperature sweep around real text’s per-sample entropy, with 256 continuations per file, and all models share the plan and sampling noise. We compare models at 110M: judge NLL of the continuation’s content tokens given the prompt, interpolated along each sweep to four entropy levels from real text’s value to 0.3 nats below it and averaged. Three judges are read, GPT-2-large, Pythia-1.4B and Qwen2.5-7B, and a fourth, Qwen2.5-1.5B, is reported. Intervals are a paired bootstrap over prompts with 2,000 replicates, shared by every file and by the real continuations, whose entropy and hence the levels are recomputed in each replicate. The start alone is also sampled at , one token per step, LLaDA’s default full-quality setting and the block-32 analog of our 110M comparison at . The expectations, written before any run, were: Mix-FT beats continued CE and Alpha diffusion at each planner and under the three judges in every seed; and, separately for each planner, Mix-FT at matches the start at ( under the three judges in every seed), is level with it (seed mean at least −0.03 with a 95% upper bound at least 0 under each judge), or does not reach it; the same rule applies to Mix-FT at against the start at .
Results. Mix-FT gives the best samples of the three fine-tunes at every planner and , in every seed under the three judges (Table 17a, Table 18): by 0.22 to 0.60 nats/token over continued CE and 0.18 to 0.40 over Alpha diffusion as seed means, and by at least 0.11 in any seed. The bound does not show this. With eight plans at block 32 it stays close to the per-plan bound (median ESS 1.1 to 1.7 for every model), and the three fine-tunes lie within 0.011 nats/token of each other on it, Alpha diffusion lowest, while their samples differ by up to 0.6. Mix-FT’s MAUVE is also higher than both baselines’ at every planner and . At equal steps it beats the untouched start by 0.50 to 0.85 at and 16 under either planner and by 0.10 to 0.54 at , where continued CE and Alpha diffusion are behind the start.
With half the steps the result depends on how many tokens each step reveals. At two tokens per step, the setting of our 110M comparison, Mix-FT at is ahead of the start at in every seed under all four judges with either planner, and the seed mean’s interval is above zero except under GPT-2-large with random order (Table 17b); its MAUVE is 0.95 against the start’s 0.85 under random order and 0.73 to 0.77 against 0.63 under low confidence. At four tokens per step, against the start at , it is behind under both planners, but most of the start’s own loss from halving the steps is gone: Mix-FT closes 72 to 91% of it with low confidence and 66 to 84% with random order, per seed under the three judges, against −3 to 47% for continued CE and Alpha diffusion (seed 0). More plans help at equal compute: the run (seed 0) is better than the mean of the runs at every planner and under GPT-2-large, by 0.04 to 0.12, and closes 78 to 99% of the start’s gap at , although its bound is 0.01 worse. At 110M and 1.1B the 8-step result needed at eight times the budget (Figure 3a), which we did not run at 8B.
As at 110M, LLaDA’s low-confidence planner is worse than random order at matched entropy for every model and under GPT-2-large, by 0.2 to 1.0 nats/token, and the ordering of the methods is the same under both. Matched entropy per sample does not by itself show diversity across samples (Table 19). In the comparison of § 4, Mix-FT at against the start at , Mix-FT has the higher Vendi score under both planners (103.2 and 98.8 against 96.9 and 95.3; real text 103.0) and repeats fewer 4-grams within a continuation; its self-BLEU-4 is higher, 0.18 against 0.16, closer to real text’s 0.21. At under random order it has a higher Vendi score than the start at (94.4 against 92.8) and than either baseline; under low confidence it is slightly below the start at $K = 16 (80.4 against 82.5) and above both baselines. Its repetition at (2.5% of 4-grams) is below the start’s at $K = 16 and close to real text’s 2.8%, and no model produces near-duplicate continuations.
| planner | model | rep4 | distinct-4 | self-BLEU-4 | near-dup. | Vendi | |
| random order | Start | 32 | 0.052 | 0.930 | 0.160 | 0.000 | 96.87 |
| Mix-FT | 16 | 0.040 | 0.937 | 0.182 | 0.000 | 103.20 | |
| Start | 16 | 0.042 | 0.943 | 0.148 | 0.000 | 92.79 | |
| Start | 8 | 0.024 | 0.965 | 0.115 | 0.000 | 76.70 | |
| Mix-FT | 8 | 0.025 | 0.958 | 0.157 | 0.000 | 94.39 | |
| Continued CE | 8 | 0.018 | 0.967 | 0.149 | 0.000 | 81.06 | |
| Alpha | 8 | 0.024 | 0.958 | 0.157 | 0.000 | 87.62 | |
| low confidence | Start | 32 | 0.045 | 0.936 | 0.159 | 0.000 | 95.26 |
| Mix-FT | 16 | 0.039 | 0.938 | 0.178 | 0.000 | 98.81 | |
| Start | 16 | 0.035 | 0.950 | 0.144 | 0.000 | 82.46 | |
| Start | 8 | 0.018 | 0.972 | 0.102 | 0.000 | 62.29 | |
| Mix-FT | 8 | 0.025 | 0.960 | 0.142 | 0.000 | 80.43 | |
| Continued CE | 8 | 0.015 | 0.971 | 0.134 | 0.000 | 67.11 | |
| Alpha | 8 | 0.025 | 0.959 | 0.142 | 0.000 | 70.37 | |
| real continuations | – | 0.028 | 0.935 | 0.207 | 0.000 | 103.00 |
Table 19. Diversity of the 8B continuations at matched per-sample entropy (seed means; each model at , Mix-FT also at ; the mean over the four entropy levels of each measure interpolated along the temperature sweep). rep4: share of repeated 4-grams within a continuation; distinct-4 and self-BLEU-4 over the 256 continuations; near-dup.: share of continuations whose 8-gram Jaccard similarity with another exceeds 0.5; Vendi on GPT-2-large features of prompt and continuation.
Blinded raters. We put the half-step result at two tokens per step to the same kind of blinded pairwise rater as at 110M (Section J), with the reading rule written down before any of its samples were drawn. Each model was sampled anew with LLaDA’s low-confidence decoding at its matched temperature: Mix-FT at (three seeds) and the start at . A judgment shows the 64-token prompt and the two 192-token continuations. Each seed contributes 96 pairs, 288 over 191 prompts, every pair rated in both orders in separate files of 40 judgments, with ties counted half and intervals from a bootstrap over prompts. Each model also has 8 control pairs on prompts used by no test pair: its matched-temperature continuation against a hotter one of its own, about 1.35 nats/token worse under GPT-2-large by extrapolation along the temperature sweeps above. A rater whose control interval has its lower bound at or below 0.60 is not read. ChatGPT prefers Mix-FT at in 68.8% of the pairs, 95% interval 62.9 to 74.5%, and every seed’s interval is above one half (67.4, 70.8 and 68.0%). It prefers each model’s matched-temperature text in 84% of the control pairs (interval 72 to 95%), and it agrees with itself across the two orders in 85% of the test pairs. Mix-FT is preferred on each criterion: coherence 69%, readability 67%, substance 69% and repetition 62%. Claude Sonnet, one fresh instance per file at maximum effort as in our other Claude ratings, prefers Mix-FT at in 71.4% of the pairs, 95% interval 65.6 to 77.1%, and every seed’s interval is above one half (72.4, 70.8 and 71.1%). Its control is 84% (interval 73 to 95%), it agrees with itself across the two orders in 85% of the test pairs, and its verdicts match ChatGPT’s in 82% of the pair orders. It prefers Mix-FT on each criterion as well: coherence 69%, readability 69%, substance 67% and repetition 60%.
1.1B GSM8K
We tested the 1.1B masked diffusion model of Nie et al. [21] fine-tuned by its authors for GSM8K, sampled in random order at two steps. LoRA fine-tuning on its own fine-tuning data (one seed) left Mix-FT and continued CE within 0.2 accuracy points, but the model has nearly memorized that data (about 7 nats per answer under the two-step sampler). We therefore fine-tuned on the model’s own verified solutions to 3,368 new questions: every fine-tune gained 1 to 3 points over the start, and Mix-FT and continued CE, at equal compute or at equal examples, did not differ (three seeds; paired difference points, 95% interval [, ]). We found no gain from the method on this task at this scale. This is what the objective predicts for self-generated data: the sampler’s own law is a fixed point of the exact mixture refit (Section F), so on its own samples the mixture likelihood has little to correct beyond the verification filter.
| model | sampler bound | 8 | 16 | GPT-2-large, random order | 8 | 16 | 32 | GPT-2-large, low confidence | 8 | 16 | 32 |
| Start (LLaDA-8B-Base) | 3.085 | 2.792 | 2.669 | 5.95 | 5.01 | 3.98 | 3.54 | 6.38 | 5.72 | 5.02 | 4.48 |
| Continued CE | 3.050 | 2.757 | 2.635 | 6.27 | 4.88 | 3.89 | – | 6.48 | 5.69 | 4.83 | – |
| Alpha () | 3.045 | 2.754 | 2.632 | 6.18 | 4.67 | 3.74 | – | 6.44 | 5.55 | 4.76 | – |
| Mix-FT (, ) | 3.048 | 2.761 | 2.643 | 5.85 | 4.29 | 3.48 | – | 6.19 | 5.17 | 4.38 | – |
| Mix-FT (seed 0) | 3.058 | 2.775 | 2.650 | 5.73 | 4.21 | 3.41 | – | 6.12 | 5.13 | 4.34 | – |
Table a. (a) seed means
| model, planner, steps | judge | seed 0 | seed 1 | seed 2 | seed mean | 95% interval | verdict |
| Mix-FT, low conf., 8 vs 16 | GPT-2-large | −0.129 | −0.196 | −0.111 | −0.145 | [−0.245, −0.044] | not reached |
| Pythia-1.4B | −0.084 | −0.145 | −0.061 | −0.097 | [−0.196, −0.001] | ||
| Qwen2.5-7B | −0.090 | −0.143 | −0.065 | −0.099 | [−0.182, −0.018] | ||
| Qwen2.5-1.5B | −0.094 | −0.150 | −0.077 | −0.107 | [−0.191, −0.025] | (reported) | |
| , seed 0, GPT-2-large | −0.109 | [−0.215, −0.008] | |||||
| Mix-FT, random, 8 vs 16 | GPT-2-large | −0.348 | −0.309 | −0.285 | −0.314 | [−0.413, −0.208] | not reached |
| Pythia-1.4B | −0.275 | −0.223 | −0.217 | −0.238 | [−0.322, −0.126] | ||
| Qwen2.5-7B | −0.210 | −0.170 | −0.163 | −0.181 | [−0.253, −0.062] | ||
| Qwen2.5-1.5B | −0.219 | −0.177 | −0.171 | −0.189 | [−0.264, −0.068] | (reported) | |
| , seed 0, GPT-2-large | −0.230 | [−0.337, −0.114] | |||||
| Mix-FT, low conf., 16 vs 32 | GPT-2-large | +0.108 | +0.061 | +0.143 | +0.104 | [+0.010, +0.208] | match |
| Pythia-1.4B | +0.138 | +0.090 | +0.173 | +0.134 | [+0.040, +0.229] | ||
| Qwen2.5-7B | +0.137 | +0.096 | +0.168 | +0.134 | [+0.045, +0.224] | ||
| Qwen2.5-1.5B | +0.130 | +0.088 | +0.154 | +0.124 | [+0.034, +0.216] | (reported) | |
| , seed 0, GPT-2-large | +0.142 | [+0.034, +0.254] | |||||
| Mix-FT, random, 16 vs 32 | GPT-2-large | +0.045 | +0.080 | +0.077 | +0.067 | [−0.018, +0.154] | match |
| Pythia-1.4B | +0.077 | +0.112 | +0.120 | +0.103 | [+0.021, +0.186] | ||
| Qwen2.5-7B | +0.132 | +0.188 | +0.178 | +0.166 | [+0.088, +0.250] | ||
| Qwen2.5-1.5B | +0.136 | +0.186 | +0.178 | +0.167 | [+0.090, +0.252] | (reported) | |
| , seed 0, GPT-2-large | +0.137 | [+0.036, +0.246] | |||||
| CE, low conf., 8 vs 16 | GPT-2-large | −0.721 | −0.664 | −0.618 | −0.668 | – | |
| Pythia-1.4B | −0.668 | −0.609 | −0.566 | −0.614 | – | ||
| CE, random, 8 vs 16 | GPT-2-large | −0.913 | −0.905 | −0.902 | −0.907 | – | |
| Pythia-1.4B | −0.845 | −0.825 | −0.833 | −0.834 | – | ||
| Alpha, low conf., 8 vs 16 | GPT-2-large | −0.515 | −0.545 | −0.516 | −0.525 | – | |
| Pythia-1.4B | −0.477 | −0.519 | −0.481 | −0.492 | – | ||
| Alpha, random, 8 vs 16 | GPT-2-large | −0.694 | −0.683 | −0.705 | −0.694 | – | |
| Pythia-1.4B | −0.642 | −0.641 | −0.639 | −0.641 | – |
Table b. (b) half the steps: each model at against the start at
Text8 details
Table 21. Text8 per seed: exact -step NLL (bpc, block 8) and IWAE-64 (block 16) at the best temperature. rows: exact NLL at one token per step.
| arm | seed 0 | seed 1 | seed 2 |
| block 8, , exact | |||
| C0 (masked CE pretraining) | 1.9595 | 1.9540 | 1.9339 |
| Base 1× / Mix 1× (from C0) | 1.8988 / 1.9093 | 1.8862 / 1.9031 | 1.8768 / 1.8830 |
| Base 4× / Mix 4× (from C0) | 1.8076 / 1.8735 | 1.8054 / 1.8687 | 1.8005 / 1.8545 |
| Base 5× | 1.7944 | 1.7946 | 1.7878 |
| SDTT from Base 4× | 1.8041 | 1.8033 | 1.7974 |
| Hybrid from Base 4× | 1.7653 | 1.7658 | 1.7611 |
| Mix-FT from Base 4× | 1.7518 | 1.7552 | 1.7449 |
| block 8, , exact | |||
| Base 4× | 1.7112 | 1.7142 | 1.7036 |
| CE at states / Mix-FT trained at | 1.7011 / 1.6766 | 1.7042 / 1.6765 | 1.6896 / 1.6671 |
| Mix-FT trained at | 1.6742 | 1.6777 | 1.6673 |
| block 16, , IWAE-64 | |||
| Base 5× / SDTT | 1.9608 / 1.9714 | 1.9631 / 1.9704 | 1.9636 / 1.9721 |
| Hybrid / Mix-FT | 1.9300 / 1.9220 | 1.9316 / 1.9234 | 1.9342 / 1.9245 |
| IWAE-512 (first 256 sequences): Base 5× / Mix-FT | 2.0080 / 1.9685 | 2.0089 / 1.9649 | 2.0094 / 1.9646 |
Table 21.
Decomposition arithmetic. Means over seeds:
The second line compares the -trained Mix-FT with CE trained at the states, both sampled at . The two parts sum to 0.041 because of rounding.
Width. Table 22a reruns the whole protocol, pretraining included, at widths 256, 512 and 768 (4.8M, 19M and 43M parameters) and splits the gain at each width. The wider networks are much better models: continued CE is 0.13 bpc lower at width 768 than at 256. The part of Mix-FT’s gain that persists at one token per step shrinks with width, by 0.010 to 0.011 bpc in every seed against continued CE, while the narrowing of the gap between and stays level, as a capacity account of effect (ii) predicts. Against CE trained at the states the split is the same, and at width 256 this rerun reproduces the gains of Table 21 within 0.001 bpc per seed.
Two-step training plans. Table 22b repeats the 110M comparison of training plans where the likelihood is exact. At a two-step plan reveals half a block per step, so we did not know whether the 110M result would carry over. At equal compute it does: two-step plans beat the four-step recipe on the exact likelihood, by 0.002 to 0.005 bpc, and on samples, in every seed. Unlike at 110M, the extra updates matter for the likelihood: with the four-step run’s number of updates, the two-step model is 0.005 bpc worse at in every seed, though its samples are still better. Each model is best near the step count it trained for: at the two-step model is 0.024 to 0.028 bpc worse than the four-step one, at about 0.04 better. As at 110M, the start’s plan weights are more concentrated at than at (median effective sample size of eight plans 3.60 to 3.65 against 4.21 to 4.29).
More plans, and more CE. Table 22c trains Mix-FT with , 32, 128 and 256 plans per sequence for the same number of updates, so its compute grows with . The exact NLL improves in every seed up to ( by 0.011 to 0.014 bpc over , by a further 0.005 to 0.007) and is level from 128 to 256 plans (mean −0.002). Over the same ladder the gap between the eight-plan IWAE bound and the exact NLL widens by 0.03 bpc, so a bound with few evaluation plans understates the gain, as it may for the 110M 8× model (Table 11). The samples improve along the whole ladder. Continued CE given the compute of , as 769 times the updates, reaches a much lower exact NLL, 1.651 against 1.732 bpc (seed 0), but worse samples: its against Base 5× is below that of at one unit. More plans buy samples that more CE does not, and the exact likelihood alone does not rank these models by their samples.
SDTT on text8. SDTT ([7]) from the same weights, following the paper’s algorithm and optimizer with teacher passes counted, does learn its own objective: its held-out distillation KL falls. But it improves on its start by only 0.002 to 0.004 bpc, less than a fifth unit of CE gives, and its samples are worse than continued CE’s in every seed. At the token level the same implementation removes the newline excess and is the best model at two steps per block (Table 1 and § 7).
Why SDTT does poorly on text8. The data point to two features of this setting, though we have not isolated either. First, at , SDTT runs one round, and its teacher is the same network at one character per step, which is 0.095 bpc better than the student under the exact likelihood (Base 4×: 1.7096 at , 1.8045 at ). Second, its target is the teacher’s per-position marginals, which by Propositions 1 and 2 carry no within-step dependence, and in character-level text the two characters filled in one step of an 8-character block can belong to the same word.
From scratch. A gradient probe on a small CPU proxy shows that near CE-trained weights the IWAE gradient carries about as much signal per pass as CE’s, but each step costs 25 passes. Once Mix has moved the weights, its plan weights become more unequal (effective sample size 3.8 to 4.1 of 8, against 4.7 to 4.9 at CE weights) and its gradient becomes 5 to 7 times noisier per pass. Raising the learning rate made it worse.
Exact toys
Table 23. Exact toy (, , HMM mixture, entropy 8.95 nats). KL() in nats for the -step sampler with the true conditionals, after tuning a temperature on them, and after re-optimizing all conditionals for the sampler’s exact likelihood. “Fixed” is a left-to-right plan.
Single-trajectory objectives. When fitting a target from rewards or energies, . The extra term vanishes only when every plan yields the same law. Plan invariance does not imply independence, but in the Ising toy the factorized sampler reached it by filling positions nearly independently. On a Ising model at inverse temperature 0.9 (), the path objective collapses the sampler’s total correlation from 1.72 to 0.17 nats, while the marginal objective keeps 1.42 (Table 25). Self-training is a second case: per-token CE on the sampler’s own samples converges to the true conditionals of , and iterating it re-factorizes the law each round. The per-path refit turns the HMM sampler into a near-product law in five rounds; the IWAE refit drifts at a rate that falls with , and the exact mixture refit is a fixed point (Figure 8). A test of reward-filtered self-training on Sudoku with a 5M-parameter transformer (two seeds) found no advantage for the IWAE refit.
| target, | objective | TC | TC of | |
| HMM (), | true conditionals | 0.1375 | 0.240 | 0.806 |
| path reverse KL | 0.1197 | 0.268 | ||
| marginal reverse KL | 0.0008 | 0.769 | ||
| Ising , inv. temp. 0.9, | true conditionals | 1.144 | 1.720 | 4.546 |
| path reverse KL | 0.680 | 0.171 | ||
| marginal reverse KL | 0.573 | 1.419 |
Table 25. Fitting a target from its energy alone: and total correlation (TC; 0 is a product law), nats, after 600 (HMM) or 300 (Ising) Adam steps from the true conditionals. The path objective had converged; the marginal one was still improving slowly.
Figure 8. Self-training on exact toys. Each round refits the sampler’s conditionals to its own law. The per-path refit removes dependence round after round; the IWAE refit drifts more slowly as grows; the exact mixture refit is a fixed point. Dotted: total correlation of the target.
The HMM targets are mixtures of sticky hidden Markov chains (, : 729 sequences; , : 65,536 sequences). The model is a free logit table over partial states, the same conditioning a network has; Adam minimizes the exact objective through a differentiable dynamic program from the true-conditional start. In Table 23 the random-order value is still falling after 150 Adam steps (0.0299); at it reaches 0.0008 after 2,000 steps.
Exact dynamic program
For one block with positions, steps and , let be the subsets of size . For , is the total probability that the sampler has revealed exactly after steps with tokens , summed over ordered partitions of :
One forward pass per reveal set gives at all positions and temperatures; blocks share passes because they are conditionally independent given the clean prefix. Reveal sets per sequence: 127 (, ), 255 (, ), 16,511 (, , not affordable; we report IWAE-64 and IWAE-512 there). A self-test against enumeration of all plans for several small () gives errors below nats.
Extended related work: PUMBA
PUMBA ([19]) trains the denoiser on consecutive steps of the few-step policy’s own trajectories, with teacher-forced commits, a carry state and truncated backpropagation through time, scales this to fine-tuning LLaDA-8B, and argues that training should follow the states the sampler visits, finding that a looser alignment helps where exact alignment overfits. Three points reconcile it with our argument. First, when the next group is a deterministic function of the revealed tokens, the per-trajectory likelihood is the sampler’s likelihood, so trajectory training targets the right quantity for such samplers; our argument concerns random-order samplers, and PUMBA’s planner reads the model’s own confidence, which Proposition 1 does not cover. Second, the carry state makes the conditionals depend on the trajectory, not only on the revealed set; Proposition 2 still holds for any mixture over plans, but Equation (1) and our dynamic program do not. Third, PUMBA’s critique targets random-masking CE, whose states do not match the sampler’s; our CE baseline trains at the sampler’s own states and is the closest analogue of trajectory training without a carry, and the mixture fine-tune beats it. A comparison under a policy-driven sampler remains open.
Hyperparameters
| 110M OpenWebText | |
| start | BD3-LM, OpenWebText, block 16 (kuleshov-group/bd3lm-owt-block_size16); 12 layers, width 768, 12 heads, 110M parameters as reported by Arriola et al. [2] (169.6M stored values, 92.4M outside the input embedding and output layer); , GPT-2 tokenizer; NELBO perplexity 23.0 on our held-out split |
| data | OpenWebText; training sequences from the first of 80 parquet shards; held-out: the last 3,000 documents of the last shard |
| equal compute | 276,480 forward-equivalents (FE) per arm; batch 8 × 1024 tokens |
| forward-equivalents | a 1,024-token pass counts 1, or 3 with a backward; BD3-LM’s pass over the clean context counts too, so a CE sequence costs 6 FE and a Mix-FT sequence . The checkpoint was pretrained for 1M steps of 512 sequences (524B tokens; 850K steps at block 1,024 and 150K at block 16; Arriola et al. [2]), at least FE by this count, so the 1× budget is under 0.02% of it and the 8× budget under 0.13% (0.18% as the 8× Mix-FT runs executed it; next row) |
| chunked runs | runs with do not fit a T4’s memory in one pass. The code first computes the plan weights without gradient, then repeats the forward pass with gradient in chunks of 32 plans (Section J), which executes about 1.4 times the counted FE (1,088 instead of 774 per sequence at ) and gives the same loss and gradient as one pass (largest relative gradient difference in fp64 at ). We count the single gradient pass that the method needs. A block-chunked backward, which puts all plans of a block in one chunk instead of chunking by plans, gives each block’s weights inside its chunk and removes the pass without gradient: our implementation of it executes exactly the counted 774 FE per sequence at , holds 32 rather than 34 sequence-equivalents per forward pass, and gives the single-pass loss and gradient (largest relative gradient difference in fp64). At 1.1B every Mix-FT run takes that pass, and there we count it (4 FE per row) |
| Mix-FT (main) | plans, , 1,152 steps; ablations: 443 steps, 1,047, 1,440, 1,920 |
| continued CE, Alpha | 5,760 steps, CE (Alpha: its reweighted CE) at the sampler’s states, same batches; Alpha block weights centered by a constant measured once on the start model; Alpha , best |
| optimizer (CE, Mix-FT, hybrid, Alpha) | AdamW (0.9, 0.999), no weight decay, peak lr , warmup min(100, [steps/8]), cosine to 10%, clip 1, fp16 autocast |
| SDTT-4 / SDTT-8 | 2 rounds of 1,920 steps (16 to 8 to 4 steps per block) / 1 round of 3,840 (16 to 8); reverse KL, lr constant after warmup, EMA 0.999, teacher passes counted |
| likelihood | IWAE-32 of the -step sampler, , 256 held-out sequences, , best reported; NELBO by the order-averaged estimator |
| generation | 512 samples of 1,024 tokens per temperature, , ; exact categorical sampling in fp32; one T4, batch 64 |
| judges | GPT-2-large (fp32) and Pythia-1.4B (fp16, checked against fp32); content tokens exclude newline and end-of-text targets; matched per-sample entropy at real −0.3, −0.2, −0.1, 0 nats |
Table 35.
| text8 | |
| model | 6 layers, , 4 heads, 4.82M parameters; sequences of 256 characters; block-causal attention over [clean | noisy] |
| C0 | 20,000 steps, lr , warmup 200, cosine to |
| one unit | 691,200 sequence passes: 21,600 steps of Base CE (lr ) or 864 of Mix-FT (, 25 passes per sequence) |
| optimizer | AdamW (0.9, 0.95), weight decay 0.01, clip 1.0, batch 32, fresh optimizer state at every restart |
| SDTT | reverse KL, dense Algorithm-1 targets, lr , no weight decay, EMA 0.999, rounds, teacher passes counted |
| evaluation | exact NLL () or IWAE-64 () at 10 temperatures, IWAE-512 () at 4; 512 samples at ; judge: causal character transformer (4 layers, ) |
Table 36.
Reproducing the half-step, baseline, 1.1B and rater experiments
All training, sampling and judging with open models ran on Kaggle T4 GPUs (16 GB), one job per GPU and usually two jobs per T4×2 kernel; the 8× Mix-FT runs (stage 20) and the schedule jobs (stage 21) had a kernel each. A job longer than one kernel (one Kaggle session, 11 to 11.8 h) checkpointed and resumed in the next. Stage numbers are those of Section K. Each stage’s scripts check the md5 of every imported earlier script and refuse to run otherwise; the md5s are listed in the stage’s handoff file.
The half-step model (110M). Table 26 gives the recipe. It is the two-step recipe of § 17 with plans per sequence and 8 times the budget. Each update evaluates the 256 plans in 8 chunks of 32: a pass without gradient over all chunks gives the self-normalized plan weights, then a pass with gradient backpropagates chunk by chunk. This is the exact IWAE gradient (relative difference to the unchunked one in fp64) and keeps every forward at 34 rows.
| start | the public BD3-LM checkpoint of § 17 (no earlier fine-tune) |
| plans | steps per block in training, plans per sequence, batch 8 sequences of 1,024 tokens, micro-batch 1 |
| budget | FE per sequence; 357 updates; 2,210,544 FE = 7.9953 the budget of 276,480 (under 0.13% of pretraining); the chunked backward executes 1,088 FE per sequence, 3,107,328 in all, 0.18% of pretraining (§ 17) |
| sequences seen | 2,856, a prefix of the run’s data order (no sequence repeats); same plan generator (RNG code 52, seed 1000s + 52) |
| optimizer | AdamW (0.9, 0.999), no weight decay, peak lr , warmup steps, cosine to 10% over the 357 steps, clip 1, fp16 autocast, gradient checkpointing |
| result | IWAE-32 at : 3.2141, 3.2128, 3.2145; at : 3.7293, 3.7289, 3.7300; NELBO PPL 28.82, 28.68, 28.85 (seeds 0, 1, 2) |
Table 26. Recipe of the model (mixft_k2m256x8, seeds 0, 1, 2).
Global temperature (stage 20). Every model is sampled at , 512 samples of 1,024 tokens per , with the same sampling seeds for every model. For each judge we compute the content-token NLL and the per-sample entropy (the entropy of each sample’s own token frequencies, averaged over samples) at every . We then interpolate the NLL along the model’s sweep at four entropy levels, real text’s entropy minus 0.3, 0.2, 0.1 and 0 nats, and average the four values. Real held-out text has 5.397 nats, so the levels are 5.097 to 5.397, and at least 3 of the 4 must be bracketed by the sweep. is this value for the start at minus that of the model at (positive: better).
Per-step schedules (stage 21). Step uses with , so is the geometric mean. There are nine shapes, , each swept over . The shape is chosen per model on a development split of 256 samples per (shape, ), drawn with the evaluation seeds plus 15,000,000 and so disjoint from every evaluation seed. We pick the shape with the largest mean over the two judges of its matched-entropy gain over the constant schedule if that mean is positive, and the constant schedule otherwise. The chosen shape is then sampled afresh (512 samples per , evaluation seeds) and only these fresh rows enter the reading. All three seeds and the start chose , .
Bootstrap. The bootstrap uses replicates. In each, every (model, , $\beta) file of 512 samples is resampled with replacement, independently across files, and the same indices are applied to the per-sample entropy and to every judge’s per-sample content NLL sums and counts; the whole interpolation is then recomputed. The start’s files are resampled once per replicate and shared by the three seeds, so the seed mean carries that dependence; the real-text entropy is held fixed; a replicate with fewer than 3 bracketed levels counts as . Intervals are 95% percentiles.
Third judge and margin. The third judge is Qwen2.5-1.5B base (commit 8faed761, fp16 after an fp32 check), scoring content tokens by the Pythia judge’s rule applied to its own tokenizer. The equivalence (noninferiority) margin is nats/token on the seed-mean : the reading is “within the margin” if the lower 95% bound exceeds under both GPT-2-large and Pythia, and “better” if it exceeds 0 under both. The margin was fixed before any bootstrap replicate or third-judge score was computed; the point estimates of were known by then. Table 27 gives per seed and the seed mean with its interval, under each judge and in both temperature settings. The one-sided bootstrap of is at most 0.002 for every seed mean and judge. These readings are for samples whose first token is drawn. With the first token fixed for both models, the same bootstrap puts the seed-mean lower bound below under both judges, so there is neither within the margin nor better (Table 27).
Share of the gap, and all tokens. The same bootstrap gives an interval for the share of the start’s own gap between and 16 that the $8\times, with both terms recomputed in every replicate. With the first token fixed for both models it is 95% under GPT-2-large and 97% under Pythia; with it drawn, 102% and 105% . Scoring all tokens, newlines and end-of-text targets included, moves toward the model, whose samples carry more newlines than the start’s at (5.5% against 4.2% at matched entropy, Table 8: with the first token fixed, the all-token seed-mean is −0.023 [−0.051, +0.009] under GPT-2-large and −0.011 [−0.046, +0.026] under Pythia, and with it drawn +0.049 [+0.018, +0.080] and +0.065 [+0.031, +0.100]. Content tokens stay our primary measure because they do not reward the newline excess.
The 16× model (110M). It doubles the 8× recipe once more: plans per sequence, 358 updates of eight sequences, FE per sequence and 4,416,288 in all, 16 times the 1× budget and 0.25% of pretraining. It ran on two T4s with the block-chunked backward of § 17, which executes exactly the counted FE; everything else is the 8× recipe. Table 27 gives its under both first-token conventions, with the bootstrap above. Every seed gains 0.03 to 0.08 over the 8× model under either convention. With the first token drawn its 8 steps beat the start’s 16 in every seed under both judges. With it fixed, the start’s convention, the seed means are −0.005 under GPT-2-large and +0.016 under Pythia, and the GPT-2-large interval misses the −0.03 margin by 0.006. At equal steps it beats the start by 0.58 to 0.62 nats/token with the first token fixed. Its IWAE-32 bound is level with the start’s at (0.009 above the 8× model’s) and 0.04 nats/token below it at , and its NELBO perplexity rises to 29.8 (8×: 28.8; start: 23.0).
Table 27. of the 8× and 16× models at 110M: the content-token judge NLL of the start at minus that of the fine-tuned model at , at matched per-sample entropy (nats/token; positive: 8 steps better), per seed and as the seed mean with its 95% bootstrap interval over samples. Global: one temperature per model. Schedule: each model’s own per-step temperature schedule, chosen on a development split. The first two blocks draw the first token of each sequence, as every table does; the next two fix it to end-of-text for both models (Section B) and were not scored by the third judge. The last two give the 16× model with one temperature under each convention.
The 8× baselines (110M). Table 28 lists them. All use the 8× budget, the same token cache and peak learning rate as their 1× versions, and sequences beyond the 110,721 of the cache from a fixed permutation of OpenWebText shards 1 to 4, so no sequence repeats. Stage 22 evaluates the final weights. Stage 25 instead saves the weights at 1/4, 1/2, 3/4 and all of the steps (EMA weights for SDTT) and samples each checkpoint at on a development split (256 samples at each of the six global temperatures, evaluation seeds plus 15,000,000). It then evaluates one checkpoint only: the one with the largest mean matched-entropy gain over the final checkpoint under the two judges if that gain is positive, and otherwise the final one. In seed 0 SDTT-8 kept its final checkpoint and Alpha-2 its half-budget one, in seed 1 both kept the quarter-budget one, and in seed 2 SDTT-8 kept the half-budget one and Alpha-2 the quarter-budget one.
The 1.1B experiments. Table 29 gives the setting. The sampler fills a 256-position canvas in blocks of 16 from left to right; positions after the current block hold the mask token, each step is one full bidirectional forward, and there is no KV cache. The bound is IWAE-32 per block on 512 held-out (window, block) pairs. Stage 14 sampled the start and every 1× model at ; stage 23 sampled the same models at with the same seeds and no further training; stage 27 trained the model and sampled it at . In stage 27 each job ran on both GPUs of a T4×2. The 2,827 rows of an update are split in two halves. The per-example plan log-likelihoods are summed across the two processes before the plan weights are formed, so the weights are exact, and the LoRA gradients are summed by one all-reduce before clipping and AdamW. The two replicas’ parameters were compared every 10 steps and stayed identical. Intervals at 1.1B are the bootstrap above with 256 samples per file.
| start | SDMD 1.1B base ([21]), nieshen/SDMD, file mdm_safetensors/mdm-1028M-330e18-rs1-0.01-bs-1024.safetensors, revision f28f1dfe, sha256 1b7378e0; 1,028M non-embedding parameters, pretrained on SlimPajama with 3.3 × FLOPs; TinyLlama (Llama-2) tokenizer |
| data | OpenWebText: 65,536 training windows of 256 tokens from shard 0 (documents packed with EOS); 2,048 held-out windows from the last 3,000 documents of shard 79; real reference: held-out windows 0 to 1,023 (per-sample entropy 4.7235 nats) |
| LoRA | rank 32, alpha 32, all attention and MLP linears (24,084,480 parameters); base frozen in fp16 |
| sampler | , block 16; FE: a 256-position pass counts 1, or 3 with a backward |
| (stage 14) | Mix-FT: , , 22 examples (window, block) per update, 600 updates, 475,200 FE; continued CE and Alpha (); 267 rows per update at the states, 600 updates, 480,600 FE; under 0.01% of pretraining |
| (stage 27) | Mix-FT: , , 11 examples per update, 339 updates, 3,833,412 FE = 7.99 × 480,000 (about 0.06% of pretraining) |
| optimizer | AdamW (0.9, 0.999), no weight decay, lr , 5% warmup (30 and 16 steps), cosine to 0, clip 1, fp16 loss scaling, micro-batch 16 rows; window order seed 1000 + , LoRA init seed 7000 + |
| sampling | , 256 samples per file, batch 64, seeds 4242 / 5353 + 100,000 + 1,000 (identical across models) |
| judges | GPT-2-large (fp32, openai-community/gpt2-large@212095d5) and Pythia-1.4B (fp16 after an fp32 check, EleutherAI/pythia-1.4b@fedc38a1) on the decoded text, re-tokenized; content tokens only; matched-entropy levels 4.423 to 4.723 nats |
Table 29. 1.1B setting (SMDM base, LoRA, OpenWebText).
| model | measure | seed 0 | seed 1 | seed 2 | seed mean | 95% interval |
| Mix-FT () | , GPT-2-large | +0.090 | +0.027 | −0.037 | +0.027 | [−0.107, +0.155] |
| , Pythia-1.4B | +0.151 | +0.149 | +0.098 | +0.133 | [+0.013, +0.242] | |
| MAUVE at | 0.932 | 0.930 | 0.874 | 0.912 | – | |
| Mix-FT () | , GPT-2-large | −0.013 | −0.115 | −0.151 | −0.093 | [−0.213, +0.026] |
| , Pythia-1.4B | +0.073 | −0.023 | −0.043 | +0.002 | [−0.111, +0.111] | |
| MAUVE at | 0.885 | 0.865 | 0.898 | 0.883 | – | |
| Continued CE | , GPT-2-large | −0.760 | −0.890 | −0.962 | −0.871 | – |
| , Pythia-1.4B | −0.666 | −0.769 | −0.817 | −0.751 | – | |
| MAUVE at | 0.526 | 0.539 | 0.498 | 0.521 | – | |
| Alpha () | , GPT-2-large | −0.805 | −0.668 | −0.896 | −0.790 | – |
| , Pythia-1.4B | −0.754 | −0.645 | −0.765 | −0.721 | – | |
| MAUVE at | 0.606 | 0.581 | 0.514 | 0.567 | – |
Table 30b. (a) seed means. (b) half the steps: each model at against the start at .
Blinded pairwise raters. Every pair shows two complete unconditional samples of 1,024 GPT-2 tokens, decoded exactly, each end-of-text token shown as [document break] on its own line; nothing is excerpted. Table 32 defines the three contrasts, c1 (the model at against the start at ), c0 (the start against itself at and 16) and c3 (the model against the start, both at ), with their temperature settings and controls. In c3 the start’s text is 0.04 to 0.07 nats hotter in per-sample entropy on this coarse grid, which leans toward Mix-FT, but at exactly matched entropy the judges still put Mix-FT 0.62 to 0.67 nats/token ahead. The rerun with the first token fixed on both sides uses the same pairs, orders and files with new opaque ids, each side at the grid point the same rule picks on the fixed-convention samples: the temperatures of Table 32 except the start’s in c3 (). The start’s text is 0.02 to 0.04 nats hotter in c1 (5.471 against 5.427 to 5.451), which leans slightly toward Mix-FT, and 0.065 to 0.095 nats colder in c3 (5.322 against 5.387 to 5.417), which leans toward the start. A fixed derangement pairs sample of one side with sample of the other, so no pair shares a sampling seed; the three seeds’ pairs that share a start sample form one cluster. Each pair is judged in both display orders by separate rater sessions, and the stored order is a fair coin per pair. Controls pair a model’s matched-temperature text with the same model’s hottest grid point, . A verdict scores 1 if the test side (the fine-tuned model in c1 and c3, the start at in c0) is preferred, for a tie and 0 otherwise; a pair’s score is the mean of its two orders, and is the mean over pairs. Intervals are 95% percentile bootstraps with 10,000 draws: over clusters for the pooled (ratio estimator), over pairs within a seed, and over control pairs for the control rate . A rater counts only if the lower bound of exceeds 0.60. On c1 the reading is “noninferior” if the lower bound exceeds 0.45, “inconclusive” if it lies in (0.40, 0.45] with the upper bound at least 0.45, and “worse” otherwise; on c3 it is “preferred” if the lower bound exceeds 0.50. Each rule was fixed before any text of its contrast was rated. The share of the halving gap recovered is
Table 32. The three rater contrasts. Each pair shows one complete sample of 1,024 tokens from each side, and every pair is judged in both display orders. For each model the temperature is the grid point whose per-sample entropy is closest to that of real text, 5.397 nats. A control pairs a model’s text at that temperature with the same model’s text at its hottest grid point, . In c3 the controls use the start, because at the 8× model’s hottest text was too close in quality to its matched text to test a rater.
| contrast | test side | reference side | temperatures | per-sample entropy (nats) | main pairs | control pairs | judgments |
| c1 | 8× model at | start at | each model’s own per-step schedule (the same schedule for all four models), | test 5.431 to 5.444, reference 5.434 | 512, drawn at random from 1,536 (171, 171 and 170 per seed) | 192 on the 8× model at (64 per seed) | 1,408 |
| c0 | start at | start at | the same schedule for both, | test 5.403, reference 5.434 | 256, drawn at random from 512 | 64 on the start at | 640 |
| c3 | 8× model at | start at | one global temperature each: for the 8× model (every seed), for the start | test 5.385 to 5.414, reference 5.457 | 192 (64 per seed) | 96 on the start at | 576 |
caption:
with an interval from the two bootstrap standard errors (normal approximation, c0 and c1 treated as independent). Table 33 lists the runs. The ChatGPT runs gave each file of 32 to 44 judgments to a fresh agent with no shared memory; the Claude
Table 31. Per-seed comparisons at 1.1B. Upper part: each arm’s content-token judge NLL minus Mix-FT’s at the same number of steps per block (nats/token, matched per-sample entropy, 256 samples per temperature; positive: Mix-FT better). The start is one model, compared with each Mix-FT seed; every other arm is paired by seed. Lower part: the IWAE-32 bound of the four-step sampler, each arm minus Mix-FT 1× (nats/token; positive: Mix-FT 1× better).
| arm | GPT-2-large seed 0 | seed 1 | seed 2 | mean | Pythia-1.4B seed 0 | seed 1 | seed 2 | mean | |
| Mix-FT 1× () against | |||||||||
| Start (SMDM base) | 2 | −0.008 | −0.158 | −0.153 | −0.106 | +0.181 | −0.011 | +0.042 | +0.071 |
| 4 | +0.883 | +0.516 | +0.660 | +0.686 | +0.957 | +0.625 | +0.766 | +0.782 | |
| 8 | +1.039 | +0.938 | +0.902 | +0.960 | +1.095 | +0.999 | +0.979 | +1.025 | |
| 16 | +0.693 | +0.540 | +0.623 | +0.619 | +0.780 | +0.617 | +0.705 | +0.701 | |
| Continued CE | 2 | +0.304 | +0.246 | +0.239 | +0.263 | +0.249 | +0.167 | +0.181 | +0.199 |
| 4 | +0.684 | +0.518 | +0.712 | +0.638 | +0.677 | +0.526 | +0.715 | +0.639 | |
| 8 | +0.747 | +0.775 | +0.811 | +0.778 | +0.739 | +0.746 | +0.774 | +0.753 | |
| 16 | +0.603 | +0.405 | +0.601 | +0.536 | +0.565 | +0.403 | +0.579 | +0.516 | |
| Alpha () | 2 | +0.340 | +0.076 | +0.025 | +0.147 | +0.395 | +0.103 | +0.120 | +0.206 |
| 4 | +0.734 | +0.250 | +0.460 | +0.481 | +0.810 | +0.353 | +0.511 | +0.558 | |
| 8 | +0.792 | +0.553 | +0.745 | +0.697 | +0.827 | +0.622 | +0.722 | +0.724 | |
| 16 | +0.862 | +0.512 | +0.474 | +0.616 | +0.883 | +0.556 | +0.540 | +0.660 | |
| Mix-FT 8× () against | |||||||||
| Start (SMDM base) | 4 | +1.131 | +1.058 | +0.901 | +1.030 | +1.184 | +1.121 | +0.979 | +1.095 |
| 8 | +1.143 | +1.080 | +1.016 | +1.080 | +1.173 | +1.172 | +1.120 | +1.155 | |
| Continued CE | 4 | +0.932 | +1.059 | +0.953 | +0.981 | +0.904 | +1.022 | +0.927 | +0.951 |
| 8 | +0.850 | +0.917 | +0.925 | +0.898 | +0.817 | +0.919 | +0.915 | +0.883 | |
| Mix-FT 1× () | 4 | +0.248 | +0.542 | +0.241 | +0.344 | +0.227 | +0.496 | +0.213 | +0.312 |
| 8 | +0.104 | +0.142 | +0.114 | +0.120 | +0.078 | +0.173 | +0.141 | +0.130 |
caption:
| sampler bound at , arm minus Mix-FT 1× | ||||
| arm | seed 0 | seed 1 | seed 2 | mean |
| Start (SMDM base) | +0.2882 | +0.2903 | +0.2868 | +0.2884 |
| Continued CE | +0.0075 | +0.0089 | +0.0038 | +0.0068 |
| Alpha () | +0.0086 | +0.0104 | +0.0082 | +0.0091 |
| Mix-FT 8× () | +0.0269 | +0.0379 | +0.0368 | +0.0339 |
Table 31 (PDF p. 33).
Sonnet runs gave each batch of judgments (44 in c1, 40 in c0, 36 per file in c3) to a separate helper; order-1 and order-2 batches were disjoint. The rater instructions below head every ChatGPT file and every file of the Claude runs we use (here for a 44-judgment file); in c1 and c0 the Claude helpers rated the very files ChatGPT rated. An earlier Claude run of c1 and c0 worked from the pilot’s version of this prompt, which we did not archive word for word, and also asked for a short reason. We decided before the re-run that its numbers would replace that run’s in the text whatever they showed, and we list the earlier run in Table 33 for reference.
You are a careful, impartial judge of English writing quality. This is a blinded rating task for a
research study.
This file holds 44 judgments. Each judgment shows Text A and Text B. Both were produced by language
models trained on English web text. Each excerpt is about the same length (1,024 tokens), may begin or
end in the middle of a sentence, and may contain the marker [document break] where one document ends and
another begins. Do not reward or penalize an excerpt for its topic, for starting or ending abruptly, or
for containing a document break. Judge which excerpt is better written, using these four criteria,
weighted equally:
1. Coherence: the text stays on topic and makes sense from sentence to sentence and from paragraph to
paragraph.
2. Readability: the sentences are grammatical, fluent, and easy to follow.
3. Repetition: the text is free of loops, repeated phrases, and degenerate lists (less repetition is
better).
4. Substance: the text conveys concrete, specific, and consistent content rather than vague,
contradictory, or empty filler.
The order in which the excerpts are shown is random; do not let it influence you. If the two excerpts are
of essentially equal quality overall, answer "tie".
Read both texts of every judgment in full and judge each judgment on its own merits. Do not use code,
scripts or statistics to decide; the verdicts must be your own reading.
Answer with exactly 44 lines of JSON, one per judgment, in order, and nothing else:
{"jid": "...", "coherence": "A|B|tie", "readability": "A|B|tie", "repetition": "A|B|tie", "substance":
"A|B|tie", "overall": "A|B|tie"}
The open-weights Qwen2.5-7B-Instruct run (greedy, fp16) used the same rubric as a single-pair chat prompt that also asked for a reason of one or two sentences.
Pooled over the two raters, the share of the halving gap recovered is 0.88 [0.79, 0.97] with the first token fixed and 0.90 [0.80, 1.00] with it drawn, and the two raters give the same verdict on 76 to 95% of the judgments of each contrast (Table 33). Fixing the first token lowers both raters’ equal-step preference from 88 to 76–79%, as it lowers the judges’ margin: it helps the start more than the fine-tuned model (Section B). For Claude the model, prompt and effort of the two runs are the same, so its change comes from the samples alone. With the archived prompt Claude rates c1 0.04 lower than with the pilot’s prompt; its two runs give the same verdict on 83% of the c1 judgments, little more than either gives with ChatGPT, and on 96% of the c0 judgments, so the c1 pairs are close calls for every rater.
Fine-tuning time. Table 34 gives measured training hours on T4 GPUs, summed over kernels from each kernel’s trainer start line to its budget stop or end of training (model loading and evaluation excluded). The Mix-FT run took 1.45 times as long as continued CE, mostly because its chunked backward executes about 1.4 times the counted FE. The block-chunked backward of § 17 executes only the counted FE: on a T4, one update of eight sequences takes 272 seconds with it instead of 397, which would bring the run to about 27 hours, the time of continued CE.
| model | run | budget | T4-hours |
| 110M | Mix-FT, , (1,152 steps) | 1 | 3.85, 3.38, 3.76 |
| 110M | continued CE (5,760 steps) | 1 | 3.10, 3.58, 3.37 |
| 110M | SDTT-8 (3,840 steps) | 1 | 3.35, 3.30, 3.28 |
| 110M | Mix-FT, , (357 steps) | 8 | 38.9, 37.7, 38.0 |
| 110M | Mix-FT, , (358 steps) | 16 | 49.7, 48.5, 49.2 |
| 110M | continued CE (46,080 steps) | 8 | 26.8, 26.0, 25.4 |
| 110M | Alpha (46,080 steps) | 8 | 26.5, 25.5, 26.1 |
| 110M | SDTT-8 (30,720 steps) | 8 | 25.2, 25.0, 25.2 |
| 1.1B | Mix-FT, , (600 steps) | 1 | 6.47, 7.04, 6.93 |
| 1.1B | continued CE (600 steps) | 1 | 8.49, 7.94, 7.61 |
| 1.1B | Mix-FT, , (339 steps) | 8 | 59.6, 56.1, 59.8 |
| 8B | Mix-FT, , (600 steps) | 1 | 73.3, 74.5, 75.4 |
| 8B | continued CE (600 steps) | 1 | 78.6, 80.0, 78.9 |
| 8B | Alpha (600 steps) | 1 | 77.7, 83.5, 79.4 |
Table 34. Measured fine-tuning time (T4 GPU-hours, training only; seeds 0, 1, 2 where run). At 110M every job used one T4 except the 16 jobs, which used two. At 1.1B the 1 jobs used one T4 and the 8 jobs two; at 8B every job used two, with the model split by layers. Jobs on two T4s count wall-clock hours 2.
Experimental decisions and outcomes
We wrote each expectation in a version-controlled log before the experiment ran, together with its code, arms and compute. The table gives, in order, what we expected, what happened, and where the paper reports the evidence; stage numbers are the log’s. is a matched-diversity judge gap (positive: arm better); is its content-token version at matched per-sample entropy; is a likelihood difference, arm minus reference (negative: arm better); is of a model at against the start at . Unless noted, three seeds.
In the table, C0 is the model after masked-CE pretraining only, Base is C0 followed by units of CE at the sampler’s states (stage 1 continued plain masked CE instead), Mix is the mixture objective trained from C0, and the hybrid adds the CE loss on Mix-FT’s forward passes. Q-numbers name the questions within a stage, and a kill condition is the outcome, fixed in advance, that rejects the expectation it names. The coverage guard asks whether, where Mix wins at matched distinct-4, its samples still cover at least as much of real held-out text’s 4-gram occurrences as the baseline’s, within half a percentage point (stage 2b also accepted a win at matched coverage), so that the gain is not bought with lost real 4-grams. The share of a per-path control is the part of Mix-FT’s gain over continued CE that the control recovers, on the likelihood in rows 10 and 13A and on the judge gain in row 30.
| stage | question and what we expected | what happened | evidence |
| 1 | Text8 pilot (, , 2 seeds, weaker Base): Mix beats Base on IWAE-64 by bpc without losing distinct-4. | Pass (0.055, 0.063 bpc). Later stages used a Base trained at the sampler’s states. | — |
| 2 | Mix from C0 against C0, Base, SDTT at : in all seeds and mean ; every baseline must learn. | Invalid: SDTT did not learn. Without SDTT the pass condition held. | — |
| 2b | Fixed SDTT; kill “Mix is under-trained” if the Mix minus Base bound gap at is not smaller than at ; coverage guard. | Pass; explanation rejected: on the IWAE-64 bound, averaged over blocks 8 and 16, the gap is +0.020 bpc at and +0.076 at (exact values at block 8 in Table 21); guard holds. | Table 21 |
| 2c | Two-stage recipe (, , exact): some arm with and against Base in all seeds. | Pass for Mix-FT and the hybrid. | Table 21 |
| 3 | Q1 Mix-FT vs SDTT (exact, ); Q2a ; Q2b ; pass if (Q2: ) and in all seeds. | Pass, pass, pass; coverage guard: no caveat. | Table 21 |
| 3, Q3 | 34M FineWeb-Edu from Base (one seed, reported only). | No verdict ( for both arms). | Figure 4 |
| 4 | 34M from Base : Mix-FT () and hybrid vs Base , nats/token and in all seeds. | Pass for both. | Figure 4 |
| 5 | 110M BD3-LM: Mix-FT and hybrid vs continued CE and SDTT; and all-token (mean over ) in all seeds. | Mix-FT passes against both; hybrid passes against CE. The all-token against SDTT depends on newline targets, so only the likelihood part of that comparison is claimed. | Table 10 |
| 6, Q6B | Mix-FT on two-step plans: against SDTT (all tokens) and against CE in all seeds; kill if mean . | Kill (mean ), although in every seed. | Table 1 |
| 7, Q7a | Four-step Mix-FT vs SDTT and CE at (512 samples): under both judgments in every seed. | Mix-FT better in all four comparisons. | Section A |
| 7, Q7b | Mix-FT on top of SDTT vs more SDTT: and at in every seed. | No verdict ( every seed; one seed slightly negative at ). | Section A |
| 8 | Alpha diffusion, : likelihood, and samples per against the best . | Four-step Mix-FT better on likelihood; samples mixed at , Alpha better at , Mix-FT better at . | Table 10 |
| 9c | 1.1B GSM8K, fine-tuning on the model’s verified solutions: Mix-FT minus CE accuracy in every seed with a bootstrap interval excluding 0. | No difference (Section D). | Section D |
| 10 | Text8 mechanism control: weighting is “the main cause” if Mix-FT beats per-path FT in every seed and per-path FT recovers at most half of Mix-FT’s likelihood gain over continued CE. | Main cause (per-path FT recovers a share of 0.24 on average); samples: Mix-FT better. | Table 20 |
Table 21 (PDF p. 37).
| stage | question and what we expected | what happened | evidence |
| 11 | 110M matched budgets (512 samples): four-step Mix-FT vs SDTT and Alpha at , vs SDTT-8 at $K = 8, 16. | Mixed at against both; Mix-FT better than SDTT-8 at and on likelihood. | Section A | ||
| 12, Q12a | Two-step Mix-FT vs Alpha, SDTT-4 (), SDTT-8 () and four-step Mix-FT, at : under both content judges in every seed. | Two-step Mix-FT better in all nine readings. | Table 1 |
| 12, Q12b | Seed 0: (two-step, ) minus (four-step, , 91% of the updates) at ; “two steps matter” if under both judges. | Two steps matter ( under both). | tbl:7a |
| 12E | Seed 0, equal updates: two-step at 443 steps vs four-step Mix-FT, and two-step vs four-step at 1,152 steps; under both judges at . | Two steps matter in both (, ). | tbl:7a |
| 13A | 110M per-path control: “main cause” if Mix-FT beats per-path FT in every seed and per-path FT recovers at most half of Mix-FT’s gain over CE; samples at . | Main cause (share ); samples: Mix-FT better at both. | tbl:7c |
| 13B, C | IWAE-64 at , Mix-FT minus CE per seed; 30 bound claims re-read at the validation temperature. | Mix-FT better than CE; no claim changes. | Section A |
| 13E, 31 | Text8 width test, rule on three-seed means: the one-token-per-step part of the gain shrinks with width (by at least 0.008 bpc), the gap part does not. | Supported. Against continued CE, the rule’s comparator, the first part falls from 0.024 to 0.013 bpc from width 256 to 768 (by 0.010 to 0.011 in each seed) and the second moves from 0.018 to 0.019; against CE trained at the states the split is the same. | tbl:22a |
| 14 | 1.1B SMDM base on OpenWebText: two-step Mix-FT vs continued CE, and under both judges in every seed. | Pass ( from to ; from to ). | Table 30 |
| 15B | 110M, two-step Mix-FT with against at equal compute: under both judges at in every seed. | better in all nine (, , ). | tbl:7b |
| 16, 16b | Seed 0: more plans against at , at equal compute () and with compute scaled to keep about 350 updates ( at , at ). | Equal compute saturates (, no gain); with compute more plans help (about per doubling). | tbl:7b |
| 17, 19 | Per-step temperature schedules, chosen on a development split for every model and the start: matches the start at ( under both judges in every seed). | No match (, ); the schedule helps every model, the start included, by to on fresh samples (eight-temperature grid). | tbl:7d |
| 18 | Importance proposal over plans instead of uniform plans. | Worse (); dropped. | – |
| 20 | at compute: under both judges in every seed (one global temperature). | Match in all three ( to ; Pythia to ), with the first token drawn (row 36). | Table 27 |
| 21 | The same with each model’s own schedule. | Match in all three ( to ; Pythia to ), with the first token drawn (row 36). | Table 27 |
| 23 | 1.1B, the two-step fine-tunes of stage 14 sampled at : Mix-FT at against the start at , under both judges in every seed. | No match in any seed ( to ). At equal steps () Mix-FT beats the start by 0.54 to 0.78 and continued CE by 0.40 to 0.60, in every seed under both judges. | Table 30 |
| 22 | Seed 0: continued CE and Alpha diffusion at compute; compute alone closes the gap if their under both judges. | Does not close it: (CE), (Alpha); Mix-FT beats both at . Seeds 1 and 2 (reported): , for CE and , for Alpha; Mix-FT beats both in the same seed by 0.78 to 1.25 at each . | Table 28 |
Table 1 (PDF p. 38).
| model | opt. steps | NELBO PPL | (55 s) | (93 s) | (162 s) | (305 s) | ||
| Start (BD3-LM) | – | 3.223 | 3.775 | 23.00 | 7.31 | 6.10 | 4.60 | 3.71 |
| first token fixed† | 7.46 | 6.17 | 4.54 | 3.63 | ||||
| Continued CE | 5,760 | 3.209 | 3.764 | 22.41 | – | 6.20 | 4.82 | 3.89 |
| SDTT-4 | 3.245 | 3.796 | 24.41 | 6.68 | 5.83 | 4.75 | 4.02 | |
| SDTT-8 | 3,840 | 3.228 | 3.775 | 23.42 | – | – | 4.58 | 3.80 |
| Alpha () | 5,760 | 3.202 | 3.758 | 23.16 | 7.16 | 5.79 | 4.54 | 3.78 |
| Mix-FT (ours) | 1,152 | 3.181 | 3.723 | 24.51 | 7.07 | 5.54 | 4.10 | 3.37 |
| first token fixed† | – | 5.58 | 4.10 | 3.39 | ||||
| Real held-out text | 2.78 |
Table 1. Public BD3-LM OpenWebText model (110M, block 16), all arms from the same weights at the same extra compute (about 0.02% of pretraining). Left: IWAE-32, the bound of Equation (2) with 32 evaluation plans, for the -step sampler (nats/token, best of three temperatures, 256 held-out sequences), and NELBO perplexity, which bounds the one-token-per-step sampler. Right: GPT-2-large NLL of the samples on content tokens at matched per-sample entropy (nats/token, lower is better), with the time per 512 samples of 1,024 tokens on one T4. Means of 3 seeds (start: one model); per seed in Table 6, Pythia-1.4B in Table 5. Bold: best entry. –: not measured. Ablations of Mix-FT in Table 4. †The first token of each sequence fixed to end-of-text, as BD3-LM’s own sampler does; the other rows draw it (Section B). Real text: 256 held-out windows at their own per-sample entropy, the top matched level.
| model | ||||
| Start (BD3-LM) | 7.18 | 6.08 | 4.59 | 3.69 |
| Continued CE | – | 6.14 | 4.79 | 3.86 |
| SDTT-4 | 6.70 | 5.82 | 4.72 | 3.98 |
| SDTT-8 | – | – | 4.55 | 3.77 |
| Alpha diffusion () | 6.99 | 5.75 | 4.52 | 3.75 |
| Mix-FT ( plans) | 6.96 | 5.50 | 4.07 | 3.33 |
| Real held-out text | 2.65 | |||
| plans, | 7.16 | 5.80 | 4.30 | 3.45 |
| plans, | – | 5.96 | 4.50 | – |
| plans, | – | 6.04 | 4.57 | – |
| plans, | – | 5.63 | 4.24 | – |
Table 5. As the right half of Table 1, scored by Pythia-1.4B on content tokens (nats/token, lower is better).
| stage | question and what we expected | what happened | evidence |
| 25 | Seed 0: SDTT-8 (teacher passes counted, development-selected checkpoint) and Alpha diffusion on two-step states, at ; same readings as 22. | Neither closes it ( SDTT-8, Alpha-2); Mix-FT beats both at . Seeds 1 and 2 (reported): , for SDTT-8 and , for Alpha-2; Mix-FT beats both in the same seed by 0.68 to 0.94 at each . | Table 28 |
| 27 | 1.1B, at : under both judges in every seed; seed-mean within the margin under both judges (Section J). | Not met in every seed: seeds 0 and 1 match, seed 2 is under GPT-2-large and under Pythia; seed means and . Margin shown under Pythia, not under GPT-2-large (interval ). MAUVE higher in every seed. | Table 30 |
| 29 | Bounds of the model with up to 256 plans at against continued CE (, ) and the start: sign per seed and whether the gap shrinks from 64 to 256 plans. | Above continued CE (by 0.03 at ; within 0.004 at ), below the start at ; the gap shrinks with more plans in every seed. | Table 11 |
| 28 | 110M, LLaDA’s low-confidence planner, matched entropy: the $8\times$ model at against the start at ( under both judges in every seed), and Mix-FT against every baseline at +0.15$ to ; Pythia to ) with the first token drawn, and to (Pythia to ) when both fix it (row 38); Mix-FT better in all 24 readings. | Table 12 | ||
| 30 | Seed 0 controls: (a) Mix-FT on eight-step plans, placed against the two- and four-step recipes (); (b) two-step per-path control, “main cause” if it recovers under half of Mix-FT’s gain over CE at ; (c) continued CE at the learning rate and with EMA weights: does Mix-FT still win widely? | (a) Behind two-step plans at every ; level with four-step plans at , behind at . (b) Main cause: per-path FT ends behind CE (share to ). (c) Mix-FT still wins by 0.51 to 0.81; both variants stay behind the start. | Table 7 |
| 33 | Blocks 32 and 64, before training: is the plan-mixture bound still usable (median ESS of eight plans and two further gates)? Block 32 then fine-tuned anyway as an exploratory run: Mix-FT at , , better than continued CE if under both judges in every seed; at (seed 0), under both judges at against the start at , and Mix-FT better than continued CE at . | No-go at both blocks (at block 32 the few-step candidates are have ESS 1.06, 1.13). Fine-tune: Mix-FT better at and 16 ( to ), and in seed 0 better than the start at each under both conventions ( to ), where continued CE is behind it at and 16. At : half-step match at with the first token drawn (, ), not when both fix it, the start’s convention (, ), and not at or 8; better than continued CE at ( to ). | Table 13 |
| 35 | LLaDA-8B-Base, block 32, LoRA, 0.0018% of pretraining per arm, three seeds, LLaDA’s two planners (Section C). (R1) Mix-FT better than continued CE and Alpha diffusion at under GPT-2-large, Pythia-1.4B and Qwen2.5-7B in every seed. (R2, R2-32) Per planner, Mix-FT at against the start at , and at against the start at : match ( under the three judges in every seed), level (seed mean and 95% upper bound under each) or not reached. | R1: pass at every planner, , judge and seed ( to ). R2: not reached under either planner ( to per seed, 66 to 91% of the start’s own gap closed); continued CE and Alpha diffusion to under GPT-2-large and Pythia. R2-32: match under both planners ( to per seed). | Table 17 |
Table train-the-sampler-you-run. Train the Sampler You Run
| stage | question and what we expected | what happened | evidence |
| 36 | The first token: BD3-LM never masks position 0 in training and its sampler fixes it to end-of-text, while ours draws it. Every model of the main comparison sampled both ways with the same random draws; the rule fires if fixing it helps the start by more than 0.05 under both judges at some , and then fixes the consequences: one convention in every table, each claim against the start stated under the convention that favors the start with both numbers, the rater studies rerun. Also a null fine-tune: BD3-LM’s own objective continued for the same budget on its raw windows (three seeds), expected level with or ahead of the start, with Mix-FT ahead of it, continued CE behind it, the window convention immaterial and our comparison unchanged on raw windows. | Fires: fixing the first token helps the start by 0.05 to 0.11 at and 16 (it costs 0.04 to 0.07 at ) and moves the fine-tuned models by at most 0.07. changes sign in every seed under both judges, with one temperature and with schedules ( to ); at equal steps the 8× model still beats the start by 0.53 to 0.58. In the rater rerun the 8× model at recovers 88% of the halving loss (90% before), and at equal steps the raters prefer it in 76 and 79% of pairs (88% before). The null is behind the start by 0.18 to 0.43 at every under both conventions (its EMA weights too), so not level or ahead as expected; Mix-FT is ahead of it by 0.50 to 0.96; continued CE against it is mixed. The window convention is not immaterial, though by the rule it is convention-dependent: on our windows the null is better in every seed, by 0.11 to 0.34 with the first token drawn, and with it fixed by 0.06 to 0.38 at and 8 but level in one seed at . On our windows it is level with the start on average at (seed 2 behind by 0.06 to 0.10) and behind it at and 16 in every seed, by 0.04 to 0.28. On raw windows Mix-FT’s gains move by more than 0.10, so “unchanged” fails: over continued CE they grow by 0.12 to 0.18 at , and over the start they shrink by 0.10 at under Pythia with the first token drawn. | Section B, Table 7g, Table 33 |
| 38 | Row 36’s consequence for the claims against the start that row 36 did not cover, with the first token fixed for both models. (L1) Under the confidence planner, the 8× model at beats the start at if its gain is positive under both judges in every seed; (L2) the 1× model beats the start at each under that planner (seed 0); (D1) the 8× model at is more diverse than the start at if its Vendi score is higher and its self-BLEU-4 lower in every seed; at block 32 (seed 0), (B1) the 8× model at matches the start at if its gain is under both judges, (B2) the 1× model beats the start and (B3) continued CE trails it at each . | L1 holds under both conventions (+0.02 to +0.07 fixed, +0.15 to +0.22 drawn). L2 beats at , 8 and 16 (+0.29 to +0.48). D1 holds. B1 does not hold under the start’s convention (, , , drawn). B2 better at every (+0.14 to +0.25); B3 behind at and 16. With its seeds 1 and 2 added, the 1× model beats the start at , 8 and 16 in every seed under both judges (Section B). | Section B |
| 39 | The 16× rung, three seeds: the 8× recipe with and 16× the budget (358 updates, 4,416,288 FE, 0.25% of pretraining), read under both first-token conventions. (R1) “Match” if is under both judges in every seed; “level on average” if not, but the 95% interval of the seed mean lies above under both judges. | Drawn: match (+0.06 to +0.10; Pythia +0.09 to +0.13). Fixed: no match; the seed means are under GPT-2-large and +0.016 under Pythia, and the GPT-2-large interval, , misses the margin by 0.006. Each seed gains 0.03 to 0.08 over 8× under either convention. | Table 27 |
| 40 | Text8, exact likelihood, run after the 110M results. (T1) Mix-FT on two-step plans against the four-step recipe at equal compute, both evaluated with the sampler: “two-step ahead” if its exact NLL is lower and its samples better (judge gap ) in every seed. We were unsure: at a two-step plan reveals a four block per step. (M1) The exact NLL along , 32, 128 and 256 plans at fixed updates: expected to keep improving or to flatten, not to get worse at the top. Reported: continued CE given the compute of (seed 0). | T1: two-step ahead in every seed (0.002 to 0.005 bpc; +0.06 to +0.11). At equal updates its likelihood is 0.005 bpc worse in every seed and its samples still better. M1: improves, then flattens ( against : mean ). Continued CE with the compute of : much lower exact NLL, worse samples. | Table 22b,Table 22c |
Table ??. Train the Sampler You Run
| stage | question and what we expected | what happened | evidence |
| 41 | Blinded pairwise raters on the 8B half-step result, before any of its samples were drawn: Mix-FT at against LLaDA at with low-confidence decoding, new samples at each model’s matched temperature, 288 pairs over three seeds, both orders. “Preferred” if the 95% lower bound of Mix-FT’s win rate exceeds 0.50, “noninferior” if it exceeds 0.45; a rater whose control lower bound is at most 0.60 is not read. | ChatGPT: preferred, 0.69 [0.63, 0.75], every seed’s lower bound above 0.50; control 0.84 [0.72, 0.95]. Claude Sonnet: preferred, 0.71 [0.66, 0.77], every seed’s lower bound above 0.50; control 0.84 [0.73, 0.95]. | Table 2 |
Table 2 (PDF p. 41).
How the two-step model became the main model. Stage 6 trained it to close SDTT’s lead at , and that rule was killed. A later look at its 128-sample files showed it ahead of Alpha diffusion and the four-step model at , 4 and 8. Stage 12 then tested it on the 512-sample protocol with the expectation above written down first. The 512-sample files at , 4 and 8 and temperatures 1 and 1.1 reuse the sampling seeds of the 128-sample files, so their first 128 samples are the ones already seen. All 36 such files, the two-step model’s at the three step counts and the files of the four-step Mix-FT, SDTT-4 and Alpha diffusion, each at both temperatures and in three seeds, reproduce their 128-sample files exactly. The other four temperatures and all of are new.
References
- [1]Amin, H., Yin, M., and Khanna, R. Simple diffusion language models are more effective few-step generators than reported. arXiv:2609.33947, 2026.arxiv.org/abs/2609.33947
- [2]Arriola, M., Gokaslan, A., Chiu, J. T., Yang, Z., Qi, Z., Han, J., Sahoo, S. S., and Kuleshov, V. Block diffusion: Interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations, 2025.
- [3]Ben-Hamu, H., Gat, I., Severo, D., Nolte, N., and Karrer, B. Accelerated sampling from masked diffusion models via entropy bounded unmasking. arXiv:2505.24857, 2025.
- [4]Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, 2023.arxiv.org/abs/2304.01373
- [5]Burda, Y., Grosse, R., and Salakhutdinov, R. Importance weighted autoencoders. In International Conference on Learning Representations, 2016.arxiv.org/abs/1509.00519
- [6]Chen, S., Cong, K., and Li, J. Optimal inference schedules for masked diffusion models. arXiv:2511.04647, 2025.arxiv.org/abs/2511.04647
- [7]Deschenaux, J. and Gulcehre, C. Beyond autoregression: Fast LLMs via self-distillation through time. arXiv:2410.21035, 2024.arxiv.org/abs/2410.21035
- [8]Fox, D., Bowyer, S., Liu, S., Aitchison, L., Santos-Rodriguez, R., and Yang, M. Learning generation orders for masked discrete diffusion models via variational inference. arXiv:2602.23968, 2026.arxiv.org/abs/2602.23968
- [9]Friedman, D. and Dieng, A. B. The vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research, 2023.
- [10]Gushchin, N., Baranchuk, D., and Korotin, A. Alpha diffusion language models: Factorization alone is not the problem. arXiv:2609.38066, 2026.arxiv.org/abs/2609.38066
- [11]Hayakawa, S., Takida, Y., Imaizumi, M., Wakaki, H., and Mitsufuji, Y. Distillation of discrete diffusion through dimensional correlations. In International Conference on Machine Learning, 2025.arxiv.org/abs/2410.08709
- [13]Hoogeboom, E., Gritsenko, A. A., Bastings, J., Poole, B., van den Berg, R., and Salimans, T. Autoregressive diffusion models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2110.02037
- [14]Ivanov, A., Kolesov, A., Korotin, A., Oseledets, I., and Goncharov, M. E-MoE: Enhanced mixture-of-experts for non-factorized diffusion language models. arXiv:2609.37533, 2026.arxiv.org/abs/2609.37533
- [15]Kim, B. and Sung, M. Tensor-train joint modeling for few-step discrete diffusion. arXiv:2607.03788, 2026.arxiv.org/abs/2607.03788
- [17]Lou, A., Meng, C., and Ermon, S. Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, 2024.arxiv.org/abs/2310.16834
- [18]Luxembourg, O., Permuter, H. H., and Nachmani, E. Plan for speed: Dilated scheduling for masked diffusion language models. In International Conference on Machine Learning, 2026.arxiv.org/abs/2506.19037
- [19]Madeira, M., Shidani, A., Bizeul, A., Turrisi, V., Béthune, L., Devnani, B., Busbridge, D., Ablin, P., and Monteiro, J. On trajectory-aware training for masked diffusion language models. arXiv:2609.37974, 2026.arxiv.org/abs/2609.37974
- [20]Mahoney, M. About the test data. http://mattmahoney.net/dc/textdata.html, 2011. Large Text Compression Benchmark; text8 described June 2006; page last updated Sept. 1, 2011.
- [21]Nie, S., Zhu, F., Du, C., Pang, T., Liu, Q., Zeng, G., Lin, M., and Li, C. Scaling up masked diffusion models on text. In International Conference on Learning Representations, 2025a.arxiv.org/abs/2410.18514
- [22]Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. In Advances in Neural Information Processing Systems, 2025b.arxiv.org/abs/2502.09992
- [23]Penedo, G., Kydlíček, H., Ben Allal, L., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., and Wolf, T. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024.
- [24]Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2102.01454
- [25]Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
- [26]Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.07524
- [27]Sahoo, S. S., Yang, Z., Akhauri, Y., Liu, J., Singh, D., Cheng, Z., Liu, Z., Xing, E. P., Thickstun, J., and Vahdat, A. Esoteric language models: A family of any-order diffusion LLMs. In International Conference on Machine Learning, 2026.arxiv.org/abs/2506.01928
- [28]Sanokowski, S., Berghammer, W., Ennemoser, M., Wang, H. P., Hochreiter, S., and Lehner, S. Scalable discrete diffusion samplers: Combinatorial optimization and statistical physics. In International Conference on Learning Representations, 2025.arxiv.org/abs/2502.08696
- [29]Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.04329
- [30]Shih, A., Sadigh, D., and Ermon, S. Training and inference on any-order autoregressive models the right way. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2205.13554
- [31]Turok, G., De Sa, C., and Kuleshov, V. DUEL: Exact likelihood for masked diffusion via deterministic unmasking. arXiv:2603.01367, 2026.
- [32]Wang, Z., Shi, J., Hoess, N., Gretton, A., and Titsias, M. K. Learning-order autoregressive models with application to molecular graph generation. In International Conference on Machine Learning, 2025.arxiv.org/abs/2503.05979
- [35]Xie, T., Xue, S., Feng, Z., Hu, T., Sun, J., Li, Z., and Zhang, C. Variational autoencoding discrete diffusion with enhanced dimensional correlations modeling. In International Conference on Learning Representations, 2026.arxiv.org/abs/2505.17384
- [36]Zhang, T., Zhang, X., Han, L., Shi, H., He, X., Li, Z., Wang, H., Xu, K., Srivastava, A., Mao, C., Wang, H., Pavlovic, V., and Metaxas, D. N. Few-step diffusion language models via trajectory self-distillation. arXiv:2602.12262, 2026.arxiv.org/abs/2602.12262






