betaFeedback

WorksComputer Science

Train the Sampler You Run: Few-Step Masked Diffusion Language Models via Mixture-Likelihood Fine-Tuning

Abstract

When a masked diffusion language model is sampled in a few parallel steps in a random order, the model actually run is a mixture over reveal plans, while cross-entropy training fits token conditionals rather than this mixture. We introduce mixture-likelihood fine-tuning (Mix-FT), which fine-tunes a pretrained model on an importance-weighted bound of its own sampler's likelihood, with no teacher and no change to the architecture or the sampler. On LLaDA-8B, a LoRA fine-tune for 0.002% of the pretraining compute writes better OpenWebText continuations than continued cross-entropy and Alpha diffusion given the same compute, at every tested step count. With half the model evaluations, 16 rather than 32 steps per block, its continuations score better than the original model's in every seed under three judges, with confidence-based decoding as well as random order, and under confidence-based decoding two blinded LLM raters from different model families prefer them in 69 and 71% of pairs. On a public 110M-parameter block diffusion model, Mix-FT takes under four T4-hours and beats continued cross-entropy, Alpha diffusion and SDTT distillation at 4, 8 and 16 steps per block, under two judges in every seed. Exact likelihoods on text8, with a control that weights every plan equally, attribute about three quarters of the likelihood gain over continued cross-entropy to weighting each plan by how well it explains the sequence.

Introduction

Masked diffusion language models [26, 29, 17, 22] learn to fill masked positions given the revealed ones. They are attractive because they can be sampled in a few parallel steps: a block of BB positions is filled in K<BK < B steps, each revealing B/KB/K positions drawn independently given what is already revealed. The factorization error this causes is well known. Current fixes act at sampling time, through temperature [1] or better schedules [18, 3], add heads or latents [15, 35, 14], or distill from a many-step teacher [7, 11, 36]. We change the loss instead, and the objective follows from one observation. Under a random reveal order, the deployed sampler draws from a mixture over reveal plans dd,

qθ(x)=∑dμ(d) qθ,d(x),q_\theta(x) = \sum_d \mu(d)\,q_{\theta,d}(x),

where each component is a product of factorized steps along plan dd. The standard cross-entropy (CE) loss averages log⁡qθ,d\log q_{\theta,d} over plans; it separates over steps and is minimized by the true conditionals, which do not minimize −Eplog⁡qθ-\mathbb{E}_p \log q_\theta. So we fine-tune a CE-trained model on the likelihood of the sampler we run, through an importance-weighted bound [5] over MM random plans (Figure 1). We call the result Mix-FT.

This change of loss alone, for under 0.02% of the pretraining compute, makes the same network with the same sampler write better text. On LLaDA-8B [22], a LoRA fine-tune for 0.002% of the pretraining compute beats continued cross-entropy and Alpha diffusion [10] at every tested step count in every seed. With 16 steps per block, half the model evaluations, it is ahead of LLaDA’s 32 in every seed under three judges, with LLaDA’s confidence planner as well as random order, and under the confidence planner blinded LLM raters from two model families prefer its continuations in 69 and 71% of pairs (Section 5). On the public 110M BD3-LM model of OpenWebText [2], four T4-hours of Mix-FT beat continued cross-entropy, Alpha diffusion and SDTT distillation [7] at 4, 8 and 16 steps per block, all given the same extra compute: two judge models score its samples better than the strongest baseline’s at each step count in every seed, by 0.25 to 0.45 nats/token on average at matched per-sample entropy (Table 1 and Figure 2). Baselines given eight times the budget, more updates or BD3-LM’s own objective stay well behind § 3.

Comparison of cross-entropy and Mix-FT objectives for a 4-step sampler

Figure 1. A KK-step random-order sampler draws a plan, then fills each group of positions independently given the groups before it. Its law is the average of the per-plan products. CE averages the per-plan log-likelihoods and is minimized by the true conditionals. Mix-FT fine-tunes on an importance-weighted bound of the log of the average.

Exact experiments on text8 explain why this works. We compute the sampler’s likelihood exactly with a dynamic program over reveal sets, so we can split the gain. Part of it is the within-step dependence that independent filling loses; the other part appears even at one token per step, where the fine-tuned network is better under the mixture over orders. A control with the same plans, batches and updates but equal plan weights recovers only a quarter of the gain.

Contributions.

  • A teacher-free fine-tune on the random-order sampler’s mixture likelihood that improves sample quality over every baseline we ran at 4, 8 and 16 steps per block, on public 110M, 1.1B and 8B models. On LLaDA-8B its 16 steps per block are ahead of the original’s 32; at 110M, with 256 plans per sequence and 8× the fine-tune budget, it beats the original model by more than 0.5 nats/token at equal steps, and its 8 steps per block close 95% of the gap between the original’s 8 and 16 steps.

  • The finding that training on coarse two-step plans improves 8- and 16-step sampling more than training on four- or eight-step plans at equal compute, and not because coarse plans allow more updates (§ 3).

  • An account of why CE is not the sampler’s optimum, building on Amin et al. [1], tested with exact KK-step likelihoods and a control that isolates the plan weighting (§ 2.1 and § 5).

Setup: the KK-step random-order sampler

We use block diffusion (Arriola et al., [2]): a sequence is split into blocks of BB positions, generated left to right, each by a masked diffusion sampler conditioned on the clean previous blocks. All statements below concern one block given its prefix, which we drop from the notation.

A plan d=(G1,…,GK)d=(G_1,\ldots,G_K) is an ordered partition of the BB positions into KK groups of size g=B/Kg=B/K, drawn uniformly. The denoiser rθ(xi∣xS)r_\theta(x_i\mid x_S) is a distribution over the token at masked position ii given the revealed set SS. At step kk the sampler draws every position of GkG_k independently from rθ(⋅∣xG<k)r_\theta(\cdot\mid x_{G_{<k}}). Under a fixed plan and under the sampler,

qθ,d(x)=∏k=1K∏i∈Gkrθ(xi∣xG<k),q_{\theta,d}(x)=\prod_{k=1}^{K}\prod_{i\in G_k}r_\theta(x_i\mid x_{G_{<k}}),
qθ(x)=∑dμ(d)qθ,d(x).(1)q_\theta(x)=\sum_d\mu(d)q_{\theta,d}(x). \tag*{(1)}

A temperature β\beta replaces rθr_\theta by softmax⁡(β⋅logits⁡)\operatorname{softmax}(\beta\cdot\operatorname{logits}); we sweep β\beta for every model. Masked CE pretraining draws a mask rate and minimizes the CE of masked positions. Our stronger CE baseline draws a plan and a step kk, reveals G<kG_{<k} and minimizes the CE of all masked positions, so it trains exactly at the states the sampler visits. Both have the true conditionals p(xi∣xS)p(x_i\mid x_S) as their optimum.

Fine-tuning on the sampler’s mixture likelihood

Objective. We fine-tune on the random-order sampler’s mixture likelihood, through the importance-weighted (IWAE) bound over MM plans drawn from μ\mu:

LM(θ)=Ex,d1:M[−log⁡1M∑m=1Mqθ,dm(x)].(2)\mathcal{L}_M(\theta)=\mathbb{E}_{x,d_{1:M}}\left[-\log\frac{1}{M}\sum_{m=1}^{M}q_{\theta,d_m}(x)\right]. \tag*{(2)}

an upper bound on −Eplog⁡qθ(x)-\mathbb{E}_p\log q_\theta(x) that tightens monotonically in MM (Burda et al., [5]). Evaluating the bound for a sequence costs 1+M(K−1)1+M(K-1) forward passes: one with the block fully masked, shared by all plans, and K−1K-1 per plan. Blocks share passes, since they are conditionally independent given the clean prefix. With self-normalized weights w~m∝qθ,dm(x)\widetilde{w}_m\propto q_{\theta,d_m}(x) the gradient is

∇θLM=−E[∑mw~m∑k∑i∈Gk(m)∇θlog⁡rθ(xi∣xG<k(m))].\nabla_\theta\mathcal{L}_M=-\mathbb{E}\left[\sum_m\widetilde{w}_m\sum_k\sum_{i\in G_k^{(m)}}\nabla_\theta\log r_\theta\left(x_i\mid x_{G_{<k}^{(m)}}\right)\right].

cross-entropy at the sampler’s own states with each plan weighted by how well it explains xx. At M=1M=1 the weight is 1, and L1\mathcal{L}_1 is the per-path loss below, whose optimum is the true conditionals, so the method needs M>1M>1. As M→∞M\to\infty the weights approach the plan posterior and the gradient approaches that of −log⁡qθ(x)-\log q_\theta(x). In exact toys, training a free table on L1\mathcal{L}_1 leaves it where it started, while M=16M=16 removes 94% of the error of the true conditionals at K=3K=3 (Table 24, Section F). Training targets one random-order sampler at one step count (K=2K=2 or 4); other step counts and confidence decoding are transfers we measure.

samplertrue cond.M=1M = 1M=4M = 4M=16M = 16exact mixturebest temperature
K=3K = 3, random order0.04400.04360.01030.00250.00000.0096
K=2K = 2, random order0.14810.14620.03800.00850.00080.0319

Table 24. Exact toy (L=6L = 6, V=3V = 3, entropy 5.79 nats): KL⁡(p∥q)\operatorname{KL}(p \mathbin{\|} q) after training a free table on LM\mathcal{L}_M with fresh plans every step (1,500 Adam steps). M=1M = 1 is the per-path loss.

Why CE is not the sampler’s optimum.

Proposition 1 (Fixed or state-determined plans). If the plan is fixed, or the next group is a deterministic function of the revealed tokens that does not depend on θ\theta, then

−Eplog⁡qθ,d(x)=∑kEp[−∑i∈Gklog⁡rθ(xi∣xG<k)],-\mathbb{E}_{p}\log q_{\theta,d}(x)=\sum_k\mathbb{E}_{p}\left[-\sum_{i\in G_k}\log r_\theta(x_i\mid x_{G_{<k}})\right],

and every term is minimized by rθ(xi∣xS)=p(xi∣xS)r_\theta(x_i\mid x_S)=p(x_i\mid x_S).

This is the chain rule. For such samplers nothing done to the denoiser helps beyond learning the conditionals; the remaining error is the plan’s factorization error. Planners that read the denoiser’s own outputs, such as confidence-based ones, are not covered. For a random plan, let πθ(d∣x)=μ(d)qθ,d(x)/qθ(x)\pi_\theta(d\mid x)=\mu(d)q_{\theta,d}(x)/q_\theta(x) be the posterior over plans.

Proposition 2 (Per-path loss = mixture loss + plan information). For every xx (a special case of Theorem 1 of Amin et al. [1], whose mixing term is this KL),

Ed∼μ[−log⁡qθ,d(x)]=−log⁡qθ(x)+KL⁡(μ∥πθ(⋅∣x))\mathbb{E}_{d\sim\mu}\left[-\log q_{\theta,d}(x)\right]=-\log q_\theta(x)+\operatorname{KL}\left(\mu\mathbin\Vert\pi_\theta(\cdot\mid x)\right)

Proof. Eμlog⁡(qθ(x)/qθ,d(x))=Eμlog⁡(μ(d)/πθ(d∣x))\mathbb{E}_{\mu}\log\left(q_\theta(x)/q_{\theta,d}(x)\right)=\mathbb{E}_{\mu}\log\left(\mu(d)/\pi_\theta(d\mid x)\right).

The left side, averaged over pp, is the per-path loss: CE at the sampler’s states up to a per-position weighting. It charges the sampler for any information a sample carries about its own plan. No reweighting of masks, steps or orders removes this term, since every such loss keeps the logarithm inside the plan average.

Two effects. Proposition 2 holds at every KK, and it allows two ways in which the mixture optimum differs from what CE learns. (i) Within-step dependence. When a step reveals g>1g>1 positions, the mixture can let plans specialize, for example by committing early to a mode so that later same-step tokens are nearly determined. On small exact problems, re-optimizing a free table of conditionals for the mixture removes nearly all of the error that the true conditionals leave, more than a tuned per-step temperature does (Table 23). (ii) Fit under the order mixture at one token per step. At K=BK=B there is no within-step factorization, and exact conditionals make every order exact, so CE would be optimal with unlimited capacity. A finite network cannot make every order agree. The per-path loss, here the any-order negative ELBO (NELBO), asks it to fit every order equally; the mixture likelihood needs only the average over orders to be good, and the weights move effort toward the orders that explain each sequence. Two observations fit this: on text8 the exact likelihood at one token per step improves, by an amount similar to effect (i), and at 110M the tighter IWAE-64 bound of the one-token-per-step sampler improves while the NELBO, the per-path bound, gets worse (§ 3 and § 5).

samplertrue conditionalsbest global temp.per-step temp.re-optimized table
K=2K = 2, fixed plan1.2049––1.2049 (no change)
K=2K = 2, random order0.38990.21410.11380.0299
K=4K = 4, fixed plan0.5158––0.5158 (no change)
K=4K = 4, random order0.05560.02000.01110.0000

Table 23.

Exact evaluation. Because rθr_\theta depends on the revealed set and not on the order it was revealed in, the sum over plans in Equation (1) collapses to a dynamic program over reveal sets (Section G). At B=8B=8, K=4K=4 it needs 127 forward passes per sequence for 2,520 plans. We use it for all block-8 text8 results, since IWAE-64 is looser for Mix-FT than for CE and so understates the gain.

Main result: a public 110M model

Setup. We fine-tune the public BD3-LM OpenWebText checkpoint with block 16 (Ariola et al. (2025)), a 110M-parameter model trained on sequences of 1,024 GPT-2 tokens, and study its natural sampler. Every arm starts from these weights and gets the same extra compute, counted in forward-equivalents of the network (§ 1). This budget is under 0.02% of the pretraining compute and under 4 T4-hours per arm. Mix-FT trains on two-step plans (K=2K=2) with M=8M=8 plans per sequence. The baselines are continued CE at the four-step sampler’s states; Alpha diffusion (Gushchin et al. [10]), trained exactly like continued CE with only the loss changed, at its best setting k=0.5k=0.5; and SDTT (Deschenaux & Gulcehre [7]) distilled from 16 to 4 steps per block in two rounds (SDTT-4) and from 16 to 8 in one (SDTT-8). Each arm has three seeds (§ 1).

Evaluation. For each step count K∈{2,4,8,16}K\in\{2,4,8,16\} per block we draw 512 samples of 1,024 tokens at each of six temperatures. GPT-2-large and Pythia-1.4B (Radford et al. [25]; Biderman et al. [4]) score them on content tokens, leaving out newline and end-of-text targets, so that a model’s newline rate, which few-step sampling inflates and which differs between methods, does not decide a comparison (Table 8). Scored on all tokens instead, Mix-FT still beats the start and every baseline at K=4K=4, 8 and 16 in every seed under both judges (Section A). Temperature trades quality for diversity, so we compare models at equal per-sample entropy, the entropy of each sample’s own token frequencies averaged over samples. We interpolate judge NLL along each model’s temperature sweep to four entropy levels, from real text’s value to 0.3 nats below it, and average them. This compares whole temperature sweeps, which covers the temperature-tuned baseline of Amin et al. [1]. Unless stated otherwise, every claim below that one model beats another holds in every seed under both judges, and controls run in one seed use seed 0. We wrote down the criterion for each comparison before running it; Section K gives each with its outcome.

modelmatched entropyβ=1\beta= 1
512 samples128 samples512 samples128 samples
K=4K = 4K=8K = 8K=16K = 16K=2K = 2K=4K = 4K=8K = 8K=4K = 4K=8K = 8K=16K = 16K=2K = 2K=4K = 4K=8K = 8
Start (BD3-LM)8.05.44.210.98.75.24.33.53.36.34.43.3
Continued CE9.46.14.712.59.66.14.93.83.56.55.03.6
Alpha (κ=0.5\kappa= 0.5)7.35.54.811.17.55.67.35.34.612.27.55.3
SDTT-42.41.82.23.62.61.81.51.41.91.71.61.3
SDTT-8–3.53.2––––2.52.5–––
Four-step Mix-FT8.55.84.711.28.55.65.23.93.67.05.43.7
Mix-FT (two-step)7.35.54.89.27.15.64.74.24.15.44.74.2
Mix-FT 8×7.55.54.7–––5.84.54.3–––
Mix-FT after SDTT-47.75.64.5–––4.83.83.6–––

Table 8. Share of newline tokens in samples at 110M (percent; real held-out text: 3.35%). Matched entropy: interpolated along each model’s temperature sweep at the matched per-sample entropy, as for the judges. β=1\beta= 1: the untempered sampler. The 512-sample files are the main protocol of § 3; the 128-sample files are the earlier protocol at K=2K = 2, 4 and 8. Means over the three fine-tune seeds. –: not measured.

The first token. BD3-LM never trains position 0 of a window, and its sampler fixes that token to end-of-text; our fine-tunes train it, and our sampler draws it for every model. Drawing it costs the start 0.05 to 0.11 nats/token at 8 and 16 steps, since that output never received a gradient, and saves it 0.04 to 0.07 at 4 steps; it moves the fine-tuned models by at most 0.07. Every table draws it, and every comparison with the start is stated under whichever convention favors the start, with both numbers where they differ (Section B).

Mix-FT gives the best judged samples from 4 to 16 steps per block. At every step count from 4 to 16, Mix-FT’s samples score better than every baseline’s (Table 1 and Figure 2; per seed in Table 6): by 0.25 to 0.45 nats/token over the strongest fine-tuned baseline at each step count, and by 0.52 to 0.72 over continued CE, the usual way to spend extra compute. All methods take the same time per step, so Mix-FT is the best model we measured at every generation budget from 93 seconds up. At two steps per block it still beats the start, by 0.18 to 0.32 nats/token in every seed under both judges with the first token drawn, the start’s better convention there, while SDTT-4, distilled for few steps, leads. It is not merely the model that loses least. Every fine-tuned baseline ends behind the untouched start at K=16K = 16, continued CE by 0.15 to 0.25 nats/token at K=8K = 8 and 16 (0.22 to 0.31 in every seed when both fix the first token), and neither a tenth of the learning rate nor EMA weights bring it level (Table 7e). Even BD3-LM’s own objective, continued for the same budget on its own windows, ends 0.18 to 0.43 behind the start, and Mix-FT on those windows beats it by 0.50 to 0.96 (Table 7g). Mix-FT beats the start at K=16K = 16 by more than 0.3 nats/token, and by 0.22 to 0.26 in every seed when both fix the first token. Per-step temperature schedules, tuned on a development split, improve Mix-FT, Alpha diffusion and SDTT-8 at K=8K = 8 by the same 0.3 nats/token, which leaves their order unchanged.

Figure 2. Sample quality against measured generation time at 110M (block 16, 512 samples per temperature, one T4). (a) GPT-2-large content-token NLL at matched per-sample entropy, seed means; bottom dotted line: held-out real text. With 8× the budget, Mix-FT at 8 steps per block comes within 0.04 nats/token of the start’s 16-step quality in about half the time when both fix the first token, the start’s own convention (hollow star, dashed line), and reaches it when both draw it (filled star, dotted line). (b) Each model’s NLL minus Mix-FT’s at the same step count (positive: Mix-FT better); lines are seed means, dots are the three paired seeds. From 4 steps per block up every baseline lies above zero in every seed; at 2 steps SDTT-4 lies below.

Mix-FT againstKKGPT-2-large seed 0GPT-2-large seed 1GPT-2-large seed 2GPT-2-large meanPythia-1.4B seed 0Pythia-1.4B seed 1Pythia-1.4B seed 2Pythia-1.4B mean
Start (BD3-LM)20.2180.3200.1950.2440.2050.2680.1760.216
Start (BD3-LM)40.5730.5840.5320.5630.5950.5870.5600.581
Start (BD3-LM)80.5210.4970.4690.4960.5440.5180.4920.518
Start (BD3-LM)160.3480.3640.3120.3410.3680.3880.3340.363
Continued CE40.5690.7350.6730.6590.5730.7090.6600.647
Continued CE80.6930.7290.7180.7130.7080.7430.7160.722
Continued CE160.5350.5310.4920.5190.5450.5380.4900.525
SDTT-42−0.431-0.431−0.336-0.336−0.412-0.412−0.393-0.393−0.296-0.296−0.238-0.238−0.261-0.261−0.265-0.265
SDTT-440.2920.2920.2870.2900.3330.3140.3310.326
SDTT-480.6640.6250.6440.6440.6680.6280.6440.647
SDTT-4160.6470.6760.6140.6460.6490.6860.6230.652
SDTT-880.4970.4560.4650.4730.5050.4660.4720.481
SDTT-8160.4470.4500.3990.4320.4520.4600.4060.439
Alpha diffusion (κ=0.5\kappa= 0.5)2−0.001-0.0010.1780.0780.085−0.031-0.0310.0870.0280.028
Alpha diffusion (κ=0.5\kappa= 0.5)40.2260.2980.2350.2530.2390.2710.2360.249
Alpha diffusion (κ=0.5\kappa= 0.5)80.4370.4140.4700.4400.4470.4170.4730.446
Alpha diffusion (κ=0.5\kappa= 0.5)160.4350.4200.3720.4090.4440.4380.3790.421
Four-step Mix-FT (ablation)20.2650.3130.1190.2320.2000.2740.1080.194
Four-step Mix-FT (ablation)40.3130.3990.2300.3140.3170.3770.2280.307
Four-step Mix-FT (ablation)80.2240.2710.1830.2260.2230.2740.1800.225
Four-step Mix-FT (ablation)160.1170.1610.0860.1210.1150.1630.0870.122

Table 6. Per-seed gains of Mix-FT at 110M: each model’s content-token judge NLL minus Mix-FT’s at matched per-sample entropy (nats/token, 512 samples per temperature; positive: Mix-FT better). Mix-FT is the two-step model of Table 1. Means are over the three seeds and are computed before rounding, as are the margins quoted in § 3.

Table 7. Controls at 110M. Each row gives the content-token judge gain of the first arm over the second at matched per-sample entropy (nats/token; positive: first arm better) under GPT-2-large and Pythia-1.4B, with each arm’s compute in multiples of the 1× budget and its number of optimizer updates. Mix-FT runs are named by the plans they train on (two-, four- or eight-step) and by MM, the number of plans per sequence (M=8M = 8 unless stated). Rows with three seeds give seed means whose sign holds in every seed under both judges, except cells marked †; rows with one seed are seed 0. –: not measured.

first armsecond armcomputeupdatesseedsK=4K = 4 GPT-2K=4K = 4 PythiaK=8K = 8 GPT-2K=8K = 8 PythiaK=16K = 16 GPT-2K=16K = 16 Pythia
(a) plan coarseness and update count
two-stepfour-step1 / 11,152 / 4433+0.31+0.31+0.23+0.23+0.12+0.12
two-step, 443 stepsfour-step0.38 / 1443 / 4433+0.23+0.23+0.22+0.22––
two-step, 443 stepstwo-step0.38 / 1443 / 1,1523−0.09−0.08−0.00†−0.01†––
two-stepfour-step, 1,152 steps1 / 2.61,152 / 1,1521+0.25+0.26+0.20+0.20––
two-stepfour-step, M=3M = 31 / 11,152 / 1,0471+0.48+0.48+0.45+0.46––
two-stepeight-step1 / 11,152 / 1981+0.44+0.42+0.26+0.26+0.08+0.08
(b) number of plans and compute (two-step plans)
M=8M = 8M=4M = 41 / 11,152 / 1,9201+0.14+0.15+0.20+0.20––
M=16M = 16M=8M = 81 / 1640 / 1,1521+0.09+0.10+0.11+0.11+0.09+0.09
M=32M = 32M=8M = 81 / 1338 / 1,1523+0.18+0.19+0.24+0.24+0.16+0.16
M=64M = 64M=32M = 321 / 1174 / 3381+0.02+0.02+0.03+0.04+0.01+0.01
M=128M = 128M=64M = 641 / 188 / 1741−0.04−0.02−0.01−0.02−0.01−0.02
M=8M = 8 at 3×M=8M = 8 at 1×3 / 13,456 / 1,1521−0.01−0.01+0.04+0.04−0.00+0.01
M=32M = 32 at 3×M=32M = 32 at 1×3 / 11,016 / 3381+0.06+0.05−0.01+0.00−0.03−0.03
M=64M = 64 at 2×M=32M = 32 at 1×2 / 1349 / 3381+0.11+0.11+0.06+0.07+0.02+0.03
M=128M = 128 at 4×M=64M = 64 at 2×4 / 2354 / 3491+0.08+0.08+0.05+0.05+0.04+0.04
M=256M = 256 at 8×M=128M = 128 at 4×8 / 4357 / 3541+0.07+0.08+0.06+0.06+0.05+0.05
(c) per-path controls: the same runs with equal plan weights
four-step Mix-FTfour-step per-path FT1 / 1443 / 4433+0.53+0.53+0.61+0.62––
four-step per-path FTcontinued CE1 / 1443 / 5,7603−0.18−0.19−0.12−0.12––
two-step Mix-FTtwo-step per-path FT1 / 11,152 / 1,1521+0.80+0.79+0.78+0.79+0.57+0.59
two-step per-path FTcontinued CE1 / 11,152 / 5,7601−0.23−0.21−0.08−0.08−0.04−0.04
(d) per-step temperature schedule against one temperature at K=8K = 8 (same weights)
Mix-FT, scheduleMix-FT, one temp.1 / 11,152 / 1,1521––+0.30+0.31––
Alpha, scheduleAlpha, one temp.1 / 15,760 / 5,7601––+0.29+0.30––
SDTT-8, scheduleSDTT-8, one temp.1 / 13,840 / 3,8401––+0.29+0.30––
(e) learning rate and weight averaging for continued CE
Mix-FTcontinued CE, 0.1× lr1 / 11,152 / 5,7601+0.76+0.76+0.80+0.81+0.51+0.52
Mix-FTcontinued CE, EMA1 / 11,152 / 5,7601+0.63+0.62+0.68+0.69+0.51+0.52
continued CE, 0.1× lruntouched start1 / 05,760 / 01−0.19−0.16−0.28−0.26−0.16−0.15
continued CE, EMAuntouched start1 / 05,760 / 01−0.06−0.03−0.16−0.15−0.17−0.15
(f) baselines at the 8× budget
8× Mix-FTcontinued CE 8×8 / 8357 / 46,0803+1.19+1.21+1.24+1.27+0.87+0.89
8× Mix-FTAlpha 8×8 / 8357 / 46,0803+0.95+0.97+0.98+1.01+0.81+0.84
8× Mix-FTAlpha 8×, two-step states8 / 8357 / ≤23,040‡3+0.78+0.81+0.88+0.90+0.77+0.79
8× Mix-FTSDTT-8 8×8 / 8357 / ≤30,720‡3+0.85+0.90+0.90+0.92+0.72+0.74
(g) BD3-LM’s own objective (null fine-tune) and BD3-LM’s raw windows
null, raw windowsuntouched start1 / 05,760 / 03−0.25−0.24−0.35−0.36−0.23−0.23
Mix-FT, raw windowsnull, raw windows1 / 11,152 / 5,7603+0.72+0.72+0.84+0.86+0.53+0.55
continued CE, raw windowsnull, raw windows1 / 15,760 / 5,7603+0.09+0.08+0.01†+0.01†−0.02†−0.02†
Mix-FT, raw windowscontinued CE, raw windows1 / 11,152 / 5,7603+0.64+0.65+0.83+0.85+0.55+0.56
Mix-FT, raw windowsuntouched start1 / 01,152 / 03+0.47+0.48+0.49+0.50+0.31+0.32
null, raw windowsnull, our windows1 / 15,760 / 5,7603−0.21−0.22−0.25−0.27−0.16−0.17
null, our windowsuntouched start1 / 05,760 / 03−0.04†−0.02†−0.10−0.09−0.07−0.06

caption:

Two-step training plans beat finer ones. Training on two-step plans beats the same recipe on four-step plans at every step count, by 0.12 to 0.31 nats/token on average (Table 4). A two-step plan costs fewer passes, so at equal compute this run takes 1,152 optimizer steps instead of 443, but the extra updates are not the reason (Table 7a): a two-step run of only 443 steps still beats the four-step recipe, while four-step runs given about as many updates as the two-step run stay 0.25 to 0.48 nats/token behind at K=4K = 4. Even training on the eight-step plans that the K=8K = 8 sampler runs is worse at K=8K = 8 than training on two-step plans. Our untested hypothesis is that coarse plans give the objective the most to correct: the start’s plan weights are most concentrated at K=2K = 2 (median effective sample size of eight plans 1.32, against 1.54 at K=4K = 4 and 1.95 at K=8K = 8; Section A), so the plan information that Proposition 2 charges for is largest there. On text8, where the likelihood is exact, two-step plans also beat the four-step recipe at equal compute on the exact K=4K = 4 likelihood and on samples in every seed, though each step reveals half a block (Table 22b). How the two-step model was chosen is in Section K.

training plansopt. stepssampler bound K=4K = 4sampler bound K=2K = 2NELBOPPLjudge NLL at steps per block K=2K = 2judge NLL at steps per block K=4K = 4judge NLL at steps per block K=8K = 8judge NLL at steps per block K=16K = 16
K=2K = 2, M=8M = 8 (main)1,1523.1813.72324.517.075.544.103.37
K=4K = 4, M=8M = 8 (3 seeds)4433.1873.74123.297.305.854.333.49
K=4K = 4, M=3M = 31,0473.1943.74722.73–6.014.53–
K=4K = 4, M=2M = 21,4403.1973.75122.54–6.094.60–
K=2K = 2, M=4M = 41,9203.1813.73123.55–5.674.27–

Table 4. Ablations of Mix-FT at 110M at the compute of Table 1 (training plans KK, number of plans MM), with the same columns; seed 0 except the four-step recipe (3 seeds). The main recipe (K=2K = 2, M=8M = 8) is repeated for reference.

(a) width
widthparameterscontinued CE, K=4K = 4gain at K=4K = 4vs continued CE per stepvs continued CE gapvs CE at K=8K = 8 states per stepvs CE at K=8K = 8 states gap
2564.8M1.7940.0420.0240.0180.0260.016
51219M1.6830.0360.0180.0180.0190.017
76843M1.6660.0320.0130.0190.0150.017
(b) training plans
armexact K=4K = 4: seed 0exact K=4K = 4: seed 1exact K=4K = 4: seed 2exact K=4K = 4: meanK=8K = 8 meanK=2K = 2 meanGG vs four-step, seeds 0, 1, 2
Start (Base 4×)1.80761.80541.80051.80451.70972.1033−
Mix-FT, four-step plans1.75181.75521.74491.75061.67312.0410reference
Two-step plans, equal compute1.74851.75001.74321.74721.69931.9984+0.09, +0.11, +0.06
Two-step plans, equal updates1.75671.76001.74991.75551.70312.0125+0.06, +0.10, +0.02
(c) plans per sequence at fixed updates
arm (compute)exact K=4K = 4: seed 0exact K=4K = 4: seed 1exact K=4K = 4: seed 2exact K=4K = 4: meanIWAE-8 − exactESS of 8GG vs Base 5×
Mix-FT, M=8M = 8 (1)1.75181.75521.74491.75060.0633.39+0.16
M=32M = 32 (3.9)1.74091.74111.73241.73820.0792.95+0.20
M=128M = 128 (15.4)1.73581.73461.72751.73270.0932.65+0.25
M=256M = 256 (30.8)1.73151.73501.72671.73110.0962.56+0.27
Continued CE (30.8)1.65090.0404.51+0.12

Table 22. Text8 ablations (block 8, exact NLL in bpc at the best temperature). (a) Width: the protocol rerun at each width (means of 3 seeds). Gain: continued CE (Base 5×) minus Mix-FT at K=4K = 4. It splits into the part that persists at one token per step (per step: both models sampled at K=8K = 8) and the narrowing of the gap between K=4K = 4 and K=8K = 8 (gap), against continued CE and against CE trained at the K=8K = 8 states. (b) Training plans: Mix-FT on two-step plans at the four-step recipe’s compute (more updates) and at its number of updates (0.36 of its compute). GG: judge gain over the four-step model at K=4K = 4 and matched distinct-4, per seed. (c) Plans per sequence at a fixed number of updates, with compute in units of 691,200 passes; IWAE-8 minus exact: how far the eight-plan bound lies above the exact NLL; ESS: median effective sample size of eight plan weights at β=1\beta= 1; GG against Base 5×, mean of 3 seeds. The last row is continued CE given the compute of M=256M = 256 (seed 0).

Likelihood and its price. Mix-FT has the best sampler bound of all arms at K=4K = 4 and K=2K = 2, below continued CE’s in every seed (Table 1). The NELBO, the per-path bound of the one-token-per-step sampler, gets worse: its perplexity rises from 23.0 to 24.5, while continued CE lowers it to 22.4 under our scoring of the held-out windows (Section B). The tighter IWAE-64 bound of the same sampler does not get worse, and 0.002 to 0.003 nats/token below continued CE’s in every seed. So the NELBO rise widens its gap to the tighter bound, the looser per-path fit that effect (ii) predicts.

Half the steps. At equal compute, more plans help little beyond M=32M = 32 because the update count falls. Holding the update count near 350 and doubling the plans together with the compute narrows the gap between Mix-FT at K=8K = 8 and the start at K=16K = 16 one doubling at a time (Figure 3a). With M=256M = 256 at 8× the budget (0.18% of the pretraining compute as executed, 0.13% as counted; § 17), Mix-FT at K=8K = 8 closes 95% of the start’s own gap between 8 and 16 steps under GPT-2-large (bootstrap interval 92 to 99%) and 97% under Pythia, in about half the generation time (Figure 2a). This is under the convention that favors the start, with the first token fixed for both: there the 8-step model stays 0.04 nats/token behind under GPT-2-large and 0.02 under Pythia, whose interval includes zero, and 0.06 behind under both judges with per-step temperature schedules (Table 27). With the first token drawn, it is level with or ahead of the start in every seed under both judges, with either temperature setting. Its MAUVE at K=8K = 8 is within 0.02 of the start’s at K=16K = 16 in every setting. At equal steps it is far ahead, by more than 0.5 nats/token at K=16K = 16 under either convention. At 16× (M=512M = 512, 0.25% of pretraining) every seed gains another 0.03 to 0.08: its 8 steps beat the start’s 16 in every seed with the first token drawn, and with it fixed the seed means are −0.005 (GPT-2-large) and +0.016 (Pythia; Table 27).

Figure 3. Half the steps. (a) 110M: judge NLL of the untouched start at 16 steps per block minus that of each model at 8 (GPT-2-large, content tokens, matched entropy, one temperature per model; > 0: the 8-step model is better). Doubling plans and nominal budget together at about 350 updates narrows the gap (as executed, 8× is 0.18% of pretraining and 16× 0.25%); at 8× the seeds are 0.03 to 0.06 below the start when both fix the first token (orange), the start’s own convention, and at or above it when both draw it; 16× adds 0.03 to 0.08, while every baseline given 8× stays more than 0.7 below in every seed, near or below the start’s own 8 steps. (b) Blinded pairwise raters, win rate of the first model, both models fixing the first token (both orders, ties half, 95% intervals; Table 33). (c) The same gap per seed (markers) and seed mean (bars) under GPT-2-large (filled circles), Pythia-1.4B (open circles) and Qwen2.5-7B (open diamonds): at 1.1B for the 1× fine-tune and the 8× recipe of (a); at 8B (block 32, 1×) for random order and LLaDA’s confidence planner, at 8 steps against the start’s 16 and at 16 against its 32 (Table 17).

settingjudgeseed 0seed 1seed 2seed mean95% interval
globalGPT-2-large+0.053+0.011+0.002+0.022[−0.011, +0.054]
Pythia-1.4B+0.076+0.040+0.025+0.047[+0.011, +0.083]
Qwen2.5-1.5B+0.079+0.063+0.050+0.064[+0.030, +0.098]
MAUVE at K=8K = 8 (start at K=16K = 16: 0.948)0.9400.9340.9260.933–
scheduleGPT-2-large+0.025+0.024+0.016+0.022[−0.004, +0.050]
Pythia-1.4B+0.041+0.045+0.032+0.039[+0.012, +0.069]
Qwen2.5-1.5B+0.035+0.050+0.037+0.041[+0.012, +0.071]
MAUVE at K=8K = 8 (start at K=16K = 16: 0.952)0.9510.9490.9450.948–
global, first token fixedGPT-2-large−0.032−0.058−0.038−0.043[−0.073, −0.009]
Pythia-1.4B−0.012−0.038−0.018−0.023[−0.059, +0.017]
MAUVE at K=8K = 8 (start at K=16K = 16: 0.942)0.9430.9400.9400.941–
schedule, first token fixedGPT-2-large−0.060−0.066−0.053−0.060[−0.085, −0.035]
Pythia-1.4B−0.057−0.064−0.051−0.057[−0.082, −0.030]
MAUVE at K=8K = 8 (start at K=16K = 16: 0.947)0.9460.9520.9580.952–
16×, global, first token drawnGPT-2-large+0.100+0.058+0.078+0.079[+0.047, +0.113]
Pythia-1.4B+0.126+0.087+0.105+0.106[+0.071, +0.144]
MAUVE at K=8K = 8 (start at K=16K = 16: 0.948)0.9450.9330.9540.944–
16×, global, first token fixedGPT-2-large+0.012−0.024−0.004−0.005[−0.036, +0.028]
Pythia-1.4B+0.034−0.005+0.019+0.016[−0.020, +0.056]
MAUVE at K=8K = 8 (start at K=16K = 16: 0.942)0.9510.9420.9490.947–

Table 27.

We also put the comparison to blinded pairwise raters from two model families, ChatGPT and Claude Sonnet, with the first token fixed (Figure 3b; protocol in Section J). Both pass a built-in control that an open-weights Qwen2.5-7B rater failed. The start’s K=8K = 8 text wins only 6 to 7% of its pairs against its own K=16K = 16 text; the fine-tuned model at K=8K = 8 wins 45% with the raters pooled, so it recovers 88% of the win rate lost by halving the steps (95% interval 79 to 97%; 90% with the first token drawn). Against the start at K=16K = 16 ChatGPT’s reading is inconclusive (0.47) and Claude’s is worse (0.43). At equal steps both raters prefer the fine-tuned model’s text, in 76 and 79% of pairs and in every seed, on coherence, readability and substance (88% with the first token drawn; Table 33).

Table 33. Rater runs, 21 to 31 October 2026 (contrasts in Table 32). ChatGPT names are the product names shown in its interface; the Claude helpers ran claude-sonnet-5-5, at maximum effort in every run we use. The upper block rates samples with the first token fixed on both sides, the start’s convention; the lower blocks rate the earlier samples with it drawn. (a) WW: test side preferred, ties half, with its 95% interval; per seed: WW for fine-tune seeds 0, 1 and 2 (c0 compares the start with itself and has no seeds); WcW_c: control rate; cons.: share of pairs with the same outcome in both display orders; agr.: share of the contrast’s judgments on which the Claude and ChatGPT runs give the same overall verdict; share: the share of the halving gap recovered, (Wc1−Wc0)/(0.5−Wc0)(W_{c1} - W_{c0})/(0.5 - W_{c0}), with its 95% interval. (b) WW in c3 on each criterion of the rubric, with its 95% interval.

(a) all runs
ratercontrastWWper seedWcW_ccons.agr.reading
first token fixed (27 to 31 October)
Claude Sonnet 5.5 helpersc10.428 [0.385, 0.472]0.485, 0.415, 0.3840.831 [0.783, 0.878]0.7876%worse; share 0.84 [0.74, 0.94]
c00.055 [0.032, 0.081]–0.984 [0.953, 1.000]0.9595%
c30.786 [0.710, 0.855]0.848, 0.781, 0.7300.987 [0.971, 1.000]0.8889%preferred
ChatGPT “6.1 Sol Ultra”c10.468 [0.425, 0.513]0.515, 0.455, 0.4350.850 [0.802, 0.896]0.8076%inconclusive; share 0.93 [0.82, 1.03]
c00.072 [0.046, 0.102]–0.984 [0.953, 1.000]0.9395%
c30.758 [0.680, 0.829]0.805, 0.770, 0.6990.964 [0.927, 0.990]0.8789%preferred
both raters pooledc10.448 [0.409, 0.490]0.841share 0.88 [0.79, 0.97]
c00.064 [0.041, 0.089]0.984
first token drawn (21 to 26 October)
Qwen2.5-7B-Instruct (a09a3545)c10.500 [0.498, 0.503]0.499, 0.501, 0.5020.503 [0.495, 0.510]0.003–fails control (second text chosen in 99.2%); rated all 1,536 main pairs
Claude Sonnet 5.5 helpersc10.416 [0.371, 0.462]0.450, 0.401, 0.3960.888 [0.848, 0.926]0.8279%worse; share 0.79 [0.68, 0.91]
c00.091 [0.059, 0.127]–0.902 [0.832, 0.961]0.9695%
c30.882 [0.820, 0.934]0.879, 0.906, 0.8590.898 [0.844, 0.948]0.9490%preferred
ChatGPT “Astra 6 Ultra”c10.502 [0.455, 0.547]0.550, 0.484, 0.4720.893 [0.849, 0.932]0.8179%noninferior; share 1.00 [0.89, 1.12]
c00.089 [0.057, 0.122]–0.930 [0.859, 0.984]0.9795%
ChatGPT “6.1 Sol Ultra”c30.878 [0.823, 0.927]0.898, 0.871, 0.8630.909 [0.854, 0.956]0.9190%preferred
both raters pooled (mean of the four verdicts of a pair)c10.459 [0.416, 0.501]0.891share 0.90 [0.80, 1.00]
c00.090 [0.059, 0.123]0.916
Claude Sonnet 5.5, pilot’s prompt (superseded)c10.453 [0.408, 0.498]0.491, 0.436, 0.4310.876 [0.835, 0.914]0.7778%inconclusive; share 0.88 [0.76, 0.99]
c00.106 [0.073, 0.143]–0.895 [0.820, 0.957]0.9393%
ChatGPT re-rating 192 of its own earlier judgments gave the same verdict on 90%.
(b) c3 by criterion
ratercoherencereadabilitysubstancerepetition
first token fixed
Claude Sonnet 5.5 helpers0.772 [0.699, 0.837]0.824 [0.767, 0.876]0.728 [0.659, 0.792]0.483 [0.422, 0.546]
ChatGPT “6.1 Sol Ultra”0.738 [0.656, 0.815]0.876 [0.816, 0.928]0.738 [0.656, 0.815]0.514 [0.427, 0.602]
first token drawn
Claude Sonnet 5.5 helpers0.867 [0.809, 0.918]0.908 [0.855, 0.951]0.831 [0.772, 0.882]0.474 [0.410, 0.540]
ChatGPT “6.1 Sol Ultra”0.872 [0.816, 0.922]0.928 [0.878, 0.969]0.859 [0.803, 0.909]0.457 [0.375, 0.542]

caption:

Under a confidence planner the ordering is the same. LLaDA’s default low-confidence remasking (Nie et al., [22]) samples every masked token at each step and keeps the B/KB/K most probable. Mix-FT never trained on it, but at 110M it beats every baseline and the start at K=4K = 4, 8 and 16 under this planner too, and the 8× model at K=8K = 8 beats the start’s confidence sampler at K=16K = 16 in every seed, by 0.02 to 0.07 nats/token with the first token fixed and at least 0.15 with it drawn (Table 12).

modelMAUVE, random order K=4K=4MAUVE, random order K=8K=8MAUVE, random order K=16K=16MAUVE, confidence planner K=4K=4MAUVE, confidence planner K=8K=8MAUVE, confidence planner K=16K=16judge NLL gap K=4K=4judge NLL gap K=8K=8judge NLL gap K=16K=16
Start (BD3-LM)0.040.550.950.010.110.720.040.560.50
Continued CE0.080.480.900.040.220.73−0.010.450.42
Alpha (κ=0.5\kappa=0.5)0.170.680.920.060.290.730.140.470.41
SDTT-8–0.520.920.010.060.51–0.600.52
Mix-FT 1×, seed 00.180.890.960.060.540.910.200.510.45
Mix-FT 1×, seed 10.190.880.960.070.520.910.210.480.45
Mix-FT 1×, seed 20.140.860.950.060.540.910.190.460.41
Mix-FT 8×, seed 00.380.940.950.160.910.960.280.380.28
Mix-FT 8×, seed 10.370.930.950.140.870.960.230.330.26
Mix-FT 8×, seed 20.360.930.950.140.870.960.270.350.26
Start, fixed0.040.620.940.020.170.77–––
Continued CE, fixed0.090.540.91––––––
Mix-FT 1×, seed 0, fixed0.180.860.960.070.520.91–––
Mix-FT 8×, fixed–0.940.95–0.96–0.84–0.880.96–––

Table 12. MAUVE at 110M under the random-order sampler and under LLaDA’s low-confidence planner, and the judge gap between the two planners. MAUVE uses GPT-2-large features with the boundary tokens removed (Section A), at matched per-sample entropy, with one global temperature per model and 512 samples per file. The gap is the same model’s content-token GPT-2-large NLL under the confidence planner minus its NLL under random order, at matched entropy (nats/token; positive: the confidence planner is worse). Pythia-1.4B gives the same signs, except for continued CE at K=4K=4, where both judges find the two planners level. Continued CE, Alpha and SDTT-8 are seed 0. Rows marked fixed fix the first token to end-of-text (Section B); the others draw it. –: not measured.

Compute alone does not close the gap. More compute without more plans does not help Mix-FT: at 3× the budget with M=8M = 8 or 32, it gains under 0.05 nats/token at K=8K = 8 (Table 7b). More compute does not help the baselines either. Given the full 8× budget (three seeds each), continued CE, Alpha diffusion on four-step and on two-step states, and SDTT-8, with teacher passes counted and checkpoints chosen on a development split, all stay more than 0.7 nats/token behind the start at K=16K = 16, and their 8-step samples get worse or stay level, except Alpha on two-step states in one seed (0.13 to 0.16 better than 1× Alpha at K=8K = 8, still 0.72 behind; Figure 3a, Table 28). They see 65 to 129 times as many distinct sequences as the 8× Mix-FT, and their bound barely moves: more data and more updates do not reproduce what the objective gives. On text8, continued CE given the compute of 256 plans, as 769 times the updates, reaches a much lower exact NLL than Mix-FT but worse samples (seed 0; Table 22c).

armrecipestepsFEwarmupsequencesH8H_{8}gain
continued CECE at the K=4K = 4 sampler’s states, lr 5×10−55 \times10^{-5} cosine to 10% over all steps46,0802,211,840100368,640−1.24/−1.25-1.24 / -1.25−0.18/−0.20-0.18 / -0.20
seed 1as seed 046,0802,211,840100368,640−1.21/−1.21-1.21 / -1.21−0.09/−0.09-0.09 / -0.09
seed 2as seed 046,0802,211,840100368,640−1.22/−1.22-1.22 / -1.22−0.08/−0.10-0.08 / -0.10
Alphaas CE, Alpha loss κ=0.5\kappa= 0.5, centering constant from the start model (c=−3.608c = -3.608)46,0802,211,840100368,640−1.04/−1.05-1.04 / -1.05−0.24/−0.25-0.24 / -0.25
seed 1as seed 0, with its own constant (c=−3.373c = -3.373)46,0802,211,840100368,640−0.93/−0.94-0.93 / -0.94−0.13/−0.14-0.13 / -0.14
seed 2as seed 0, with its own constant (c=−3.979c = -3.979)46,0802,211,840100368,640−0.90/−0.91-0.90 / -0.91−0.02/−0.04-0.02 / -0.04
SDTT-8one round, teacher the start at 16 steps per block, student 8; reverse KL, lr 6×10−56 \times10^{-5} constant after warmup, EMA 0.99930,7202,211,840500245,760−0.86/−0.86-0.86 / -0.860.00/0.000.00 / 0.00
seed 1as seed 0; evaluated at step 7,68030,7202,211,840500245,760−0.88/−0.87-0.88 / -0.87−0.03/−0.03-0.03 / -0.03
seed 2as seed 0; evaluated at step 15,36030,7202,211,840500245,760−0.90/−0.89-0.90 / -0.89−0.01/−0.02-0.01 / -0.02
Alpha-2as Alpha, at the K=2K = 2 sampler’s states, with the constant measured there (c=−4.231c = -4.231); evaluated at step 23,04046,0802,211,840100368,640−0.93/−0.93-0.93 / -0.93−0.07/−0.09-0.07 / -0.09
seed 1as seed 0, with its own constant (c=−4.093c = -4.093); evaluated at step 11,52046,0802,211,840100368,640−0.91/−0.90-0.91 / -0.90−0.05/−0.06-0.05 / -0.06
seed 2as seed 0, with its own constant (c=−4.454c = -4.454); evaluated at step 11,52046,0802,211,840100368,640−0.72/−0.71-0.72 / -0.71+0.13/+0.13+0.13 / +0.13
Mix-FT (Table 26)K=2K = 2, M=256M = 2563572,210,544442,856+0.05/+0.08+0.05 / +0.08+0.42/+0.43+0.42 / +0.43

Table 28. The 8×8\times baselines. FE as in § 17; SDTT counts its teacher’s passes (9 FE per sequence). H8H_{8}: GPT-2-large / Pythia content, against the start at K=16K = 16. Gain: the same comparison at K=8K = 8 against the arm’s own 1×1\times run in the same seed (Mix-FT: the main 1×1\times model; Alpha-2 in seeds 1 and 2: the seed-0 1×1\times run, the only one). Rows without a seed are seed 0.

MAUVE and diversity agree with the judges. MAUVE (Pillutla et al., [24]) on GPT-2-large features with the boundary tokens removed (Section A) is 0.86 to 0.89 for Mix-FT at K=8K = 8, against 0.68 for Alpha diffusion and 0.55 for the start (0.86 against 0.62 in seed 0 with the first token fixed; Table 12). The 8× model at K=8K = 8 is also more diverse across samples than the start at K=16K = 16 (Table 9), with a 6 to 9% higher Vendi score (Friedman & Dieng, [9]) on GPT-2-large features and a lower self-BLEU-4 in every seed, with the first token drawn or fixed. At equal steps it repeats more 4-grams per sample than the start (6.3% against 3.8%; real text 3.6%), which the raters did not penalize and the judge NLL barely notices (Section A).

modelKKVendiself-BLEU-4distinct-4rep-4 (%)
Real held-out text–1720.3410.8983.6
Start (BD3-LM)81150.2920.9233.1
Start (BD3-LM)161370.3530.8913.8
Continued CE161290.3580.8923.6
Alpha diffusion161330.3630.8695.3
SDTT-8161380.3390.9162.5
Continued CE 8×81000.2800.9283.0
Mix-FT 1×8133–1360.316–0.3230.900–0.9024.1–4.4
Mix-FT 1×16147–1480.365–0.3730.866–0.8675.4–5.6
Mix-FT 8×8146–1480.319–0.3340.8914.7–5.1
Mix-FT 8×16147–1520.363–0.3700.858–0.8596.2–6.4
Start, fixed81150.3020.9183.3
Start, fixed161360.3630.8873.9
Mix-FT 8×, fixed8144–1480.326–0.3360.890–0.8914.7–5.1

Table 9. Diversity across samples and repetition within them, at matched per-sample entropy (110M, block 16, 512 samples per file; each value is the mean over four entropy levels, from real text’s down to 0.3 nats below it). Vendi score on GPT-2-large features (higher: more diverse); self-BLEU-4 (lower: more diverse); distinct-4 over the set of samples; rep-4, the share of 4-grams repeated within a sample. Mix-FT ranges are over three seeds; baselines are seed 0, at 1× unless marked. Rows marked fixed fix the first token to end-of-text for both models (Section B); the others draw it. No sample of any model here has a near duplicate in its file (8-gram Jaccard similarity above 0.5).

At 8× the likelihood bound is level with the start’s. With 256 evaluation plans the 8× model’s bound is within about 0.01 nats/token of the start’s at K=8K = 8 and 16, while its NELBO perplexity rises to 28.8 and its plan weights concentrate on an effective 4 to 6 of the 256 plans (Table 11). Its bound still tightens faster than the start’s as plans are added, so its exact likelihood is not known. On text8, where it is exact, more plans at fixed updates lower the sampler’s NLL in every seed up to M=128M = 128 and then flatten, while the gap between the plan bound and the exact NLL widens by 0.03 bpc (Table 22c): a loose bound can hide a real gain.

modelIWAE-64 K=8K=8IWAE-64 K=16K=16IWAE-256 K=8K=8IWAE-256 K=16K=16ESS of 256 plans K=8K=8ESS of 256 plans K=16K=16tightening K=8K=8tightening K=16K=16NELBO
Start (BD3-LM)3.0272.9663.0082.95510.818.00.0020.0013.135
Continued CE3.0102.9432.9932.93311.719.6reference3.109
Continued CE 8×3.0012.9342.9842.92411.819.80.0000.0003.100
Mix-FT 1×2.9942.9402.9692.9246.69.70.0070.0063.199
Mix-FT 8×, seed 03.0342.9912.9952.9634.45.90.0220.0183.361
Mix-FT 8×, seed 13.0322.9892.9942.9624.56.00.0210.0173.356
Mix-FT 8×, seed 23.0352.9922.9962.9644.55.90.0220.0183.362

Table 11. Likelihood bounds of the 8× model at 110M (nats/token, on 256 held-out sequences). IWAE-MM bounds the KK-step random-order sampler with MM evaluation plans, at each model’s best of β∈{1,1.1,1.25}\beta\in\{1, 1.1, 1.25\}; IWAE-64 is the mean over four disjoint groups of 64 of the 256 plans. The NELBO is the per-path bound of the one-token-per-step sampler. ESS: median effective sample size of the 256 self-normalized plan weights at β=1\beta=1. Tightening: how much more the bound falls than continued CE’s when the evaluation plans go from 64 to 256. Mix-FT 1× is the two-step model; it and both continued CE models are seed 0.

At scale: LLaDA-8B

We fine-tuned LLaDA-8B-Base (Nie et al., [22]) with LoRA on OpenWebText for prefix continuation, with three seeds per arm, and sampled it with its own sampler: after the prompt, six blocks of 32 tokens from left to right, with LLaDA’s default low-confidence remasking or with random order (Section C). Mix-FT uses our 1× recipe (M=8M = 8, four-step plans) and beats continued CE and Alpha diffusion at the same compute at every step count, in every seed under three judges and both planners, by 0.22 to 0.60 and 0.18 to 0.40 nats/token, though their IWAE-8 bounds are within 0.011. At equal steps it beats the start by 0.50 to 0.85 at K=8K = 8 and 16 (Table 2).

Half the model evaluations. At K=16K = 16, two tokens per step as in our 110M comparison, Mix-FT is ahead of LLaDA

Table 2. LLaDA-8B fine-tuned with LoRA for 0.0018% of its pretraining compute (block 32, 192-token continuations of a 64-token prompt). Top: GPT-2-large NLL at matched per-sample entropy (nats/token, 3-seed mean, lower is better); at K=32K = 32 the start reveals one token per step. Middle: the start at K=32K = 32 minus Mix-FT at K=16K = 16, seed mean and 95% interval (positive: Mix-FT better). Bottom: share of 288 blinded pairwise judgments preferring Mix-FT at K=16K = 16 to the start at K=32K = 32, low-confidence decoding only (ties half, 95% interval; Section C). Per seed and K=4K = 4: Table 17 and Table 18.

Table 17. LLaDA-8B-Base fine-tuned with LoRA for OpenWebText prefix continuation (setting in Table 16). (a) Means of 3 seeds (start: one model; M=32M = 32: seed 0) of the IWAE-8 bound of the KK-step random-order sampler (nats/token) and of the GPT-2-large content-token NLL of the continuations at matched per-sample entropy (nats/token, lower is better, 256 continuations per temperature), for each planner. Bold: best entry in the column among the first four rows. (b) Each model at KK steps against the start at 2K2K, judge NLL of the start minus the model’s (positive: the model is better), per seed, with the seed mean, its 95% bootstrap interval over prompts, and the verdict of the rule fixed before the runs (Section C).

random orderGPT-2-largePythia-1.4BQwen2.5-7B
againstKKs0s1s2means0s1s2means0s1s2mean
Continued CE4+0.366+0.411+0.480+0.419+0.372+0.403+0.480+0.418+0.336+0.360+0.428+0.375
8+0.566+0.596+0.617+0.593+0.570+0.602+0.616+0.596+0.565+0.578+0.599+0.581
16+0.472+0.402+0.373+0.416+0.478+0.412+0.392+0.427+0.446+0.406+0.375+0.409
Alpha4+0.268+0.303+0.411+0.327+0.284+0.329+0.414+0.342+0.243+0.281+0.323+0.282
8+0.346+0.374+0.420+0.380+0.366+0.419+0.423+0.403+0.334+0.384+0.370+0.362
16+0.247+0.322+0.221+0.263+0.268+0.348+0.255+0.290+0.238+0.342+0.238+0.273
Start4+0.058+0.085+0.161+0.101+0.306+0.331+0.413+0.350+0.511+0.522+0.587+0.540
8+0.685+0.723+0.747+0.718+0.758+0.811+0.817+0.795+0.820+0.860+0.867+0.849
16+0.478+0.513+0.510+0.500+0.547+0.582+0.589+0.573+0.573+0.629+0.619+0.607
low confidenceGPT-2-largePythia-1.4BQwen2.5-7B
againstKKs0s1s2means0s1s2means0s1s2mean
Continued CE4+0.233+0.302+0.330+0.288+0.231+0.284+0.322+0.279+0.187+0.236+0.245+0.223
8+0.593+0.468+0.507+0.523+0.583+0.463+0.505+0.517+0.493+0.375+0.430+0.433
16+0.460+0.415+0.480+0.452+0.468+0.420+0.489+0.459+0.441+0.380+0.464+0.429
Alpha4+0.208+0.260+0.277+0.248+0.229+0.307+0.308+0.281+0.115+0.227+0.208+0.183
8+0.387+0.348+0.405+0.380+0.393+0.374+0.420+0.396+0.289+0.267+0.309+0.288
16+0.331+0.361+0.445+0.379+0.347+0.374+0.467+0.396+0.322+0.333+0.382+0.345
Start4+0.163+0.195+0.218+0.192+0.247+0.281+0.314+0.281+0.282+0.317+0.321+0.306
8+0.569+0.501+0.587+0.552+0.614+0.552+0.637+0.601+0.554+0.501+0.578+0.544
16+0.648+0.601+0.682+0.644+0.703+0.655+0.738+0.699+0.681+0.639+0.711+0.677

Table 18. Per-seed comparisons at 8B: each arm’s content-token judge NLL minus Mix-FT’s at the same planner and number of steps per block (nats/token, matched per-sample entropy; positive: Mix-FT better). The start is one model, compared with each Mix-FT seed; continued CE and Alpha diffusion are paired by seed.

modelrandom order K=8K = 8random order 16random order 32low confidence K=8K = 8low confidence 16low confidence 32
Start5.013.983.545.725.024.48
Continued CE4.883.89–5.694.83–
Alpha (κ=0.5\kappa= 0.5)4.673.74–5.554.76–
Mix-FT (ours)4.293.48–5.174.38–
start at K=32K = 32 minus Mix-FT at K=16K = 16
GPT-2-large+0.07 [−0.02, +0.15]+0.10 [+0.01, +0.21]
Pythia-1.4B+0.10 [+0.02, +0.19]+0.13 [+0.04, +0.23]
Qwen2.5-7B+0.17 [+0.09, +0.25]+0.13 [+0.04, +0.22]
ChatGPT rater–69% [63, 75]
Claude rater–71% [66, 77]

Table 2.

at K=32K = 32, one token per step, in every seed under the three judges with either planner, so it writes better continuations with 96 instead of 192 forward passes (Table 2, Figure 3c). Under LLaDA’s own confidence decoding every judge’s interval is above zero, MAUVE rises from 0.63 to 0.73–0.77 (0.85 to 0.95 under random order), and blinded pairwise raters from two model families, ChatGPT and Claude Sonnet, prefer Mix-FT’s continuations in 69 and 71% of 288 pairs, in every seed (Table 2); both pass a built-in control (Section C). At K=8K = 8, four tokens per step, Mix-FT closes 66 to 91% of the start’s own gap between 8 and 16 steps in every seed, and a run with 32 plans (seed 0) 78 to 99%, but stays 0.10 to 0.31 behind, where continued CE and Alpha diffusion stay 0.49 to 0.91 behind. At 110M and 1.1B the 8-step result needed 8× the budget, untried at 8B.

1.1B

On the 1.1B SMDM base model (Nie et al., [21]) (LoRA, block 16, under 0.01% of pretraining), Mix-FT also wins at every step count, and with the 110M half-step recipe at 8× its K=8K = 8 is ahead of the start’s K=16K = 16 on average under both judges (Table 30).

modelopt. stepsK=4K=4K=2K=2NELBOPPLK=2K=24816K=2K=24816
Start (SMDM base)–3.2793.86823.477.346.685.424.377.236.585.344.31
Continued CE6002.9973.60517.887.716.635.244.297.366.435.074.13
Alpha (κ=0.5\kappa=0.5)6002.9993.61318.437.596.485.164.377.366.355.044.27
Mix-FT 1×1\times (M=8M=8)6002.9903.59819.377.455.994.463.757.165.804.313.61
Mix-FT 8×8\times (M=256M=256)3393.0243.62923.00–5.654.34––5.484.18–

Table 30. The 1.1B SMDM base model fine-tuned with LoRA on OpenWebText (setting in Table 29). (a) Means of 3 seeds (start: one model) of the IWAE-32 bound of the KK-step sampler (nats/token, best of three temperatures), the NELBO perplexity, and the content-token judge NLL at matched per-sample entropy at KK steps per block (nats/token, lower is better, 256 samples per temperature). Bold: best entry in the column; –: not measured. (b) H8H_8 is the start’s judge NLL minus the model’s (positive: 8 steps better), with the seed mean and its 95% bootstrap interval over samples. MAUVE at matched entropy uses GPT-2-large features of each sample’s first 200 tokens; the start at K=16K=16 scores 0.765, and real text scored against real text gives 0.829 at this sample size, so MAUVE differences near the top are noisy. Per-seed gains at equal steps are in Table 31.

Why it works: exact experiments on text8

Protocol. We use text8 (Mahoney, [20]) in sequences of 256 characters, a 4.82M-parameter block 4 diffusion transformer, block 8 and K=4K = 4 steps (two characters per step) unless stated. Compute is counted in sequence passes; one unit is 691,200 passes. The base model is masked-CE pre-training followed by four units of CE at the sampler’s states (“Base 4×”). Every fine-tune starts from Base 4× and gets one more unit, with three seeds. We report the exact KK-step NLL in bits per character (bpc) on 2,000 held-out sequences at the best of ten temperatures, and GG, the gap in a character-level judge’s NLL to continued CE at matched distinct-4, the share of distinct 4-grams among all 4-grams of the samples, along a sweep of ten temperatures (positive: better than continued CE).

A second stage, not a replacement. Used in place of CE from the start of training, the mixture objective is worse on the sampler’s likelihood, by about 0.06 bpc after four units

(Figure 5), with 25× fewer optimizer steps and a gradient that grows noisier once the weights move (Section E). CE learns the conditionals cheaply, and the mixture objective then corrects them.

Figure 5. Exact K=4K = 4 sampler NLL along the training budget (text8, block 8; thick: mean, thin: seeds). Trained from C0, the mixture objective falls behind CE, and the gap grows with compute. Started from the CE model at 4 units, one more unit of mixture fine-tuning beats one more unit of CE in every seed.

From a CE start it wins. From Base 4×, Mix-FT lowers the exact NLL by 0.039 to 0.043 bpc relative to continued CE in every seed (Table 3 and Figure 6), about three times what continued CE gains from its own fifth unit. Its samples are better at matched diversity in every seed, with equal or slightly higher coverage of real 4-grams (Figure 7). At block 16, where we can only bound the likelihood, the gain on IWAE-64 and IWAE-512 of the same size, and SDTT from the same weights barely moves the bound and gives worse samples than continued CE (Section E).

armNLLΔ\DeltaGGΔ8\Delta_{8}Δ16\Delta_{16}
Base 4× (start)1.804
Continued CE1.7920000
SDTT1.802+0.009−0.038–+0.009
Per-path FT1.782−0.010+0.014––
Hybrid1.764−0.028+0.076–−0.031
Mix-FT1.751−0.042+0.156−0.025−0.039

Table 3. Text8, block 8, K=4K = 4: exact KK-step NLL (bpc, best temperature, 3-seed mean) and differences against continued CE from the same Base 4× weights at equal extra passes (negative Δ\Delta, positive GG: better). Δ8\Delta_{8}: exact NLL at K=8K = 8 (one token per step) against CE trained at the K=8K = 8 states. Δ16\Delta_{16}: block 16, K=4K = 4, IWAE-64. Hybrid: Mix-FT plus the CE loss on the same forward passes. Per seed in Section E.

Figure 6. Text8 sampler likelihood relative to continued CE from the same Base 4× weights and extra compute (bars: mean of 3 seeds; markers: seeds). Block 8 uses the exact NLL; block 16 uses IWAE-64.

Figure 7. Text8 quality against diversity, seed 0. Top: judge NLL against distinct-4, one point per temperature. Bottom: judge NLL minus that of continued CE at matched distinct-4; markers at the four levels that define GG, whose mean is −G-G. Lower is better.

The plan weighting is the main cause. Per-path FT keeps everything else in which Mix-FT differs from continued CE (the same sequences, 8 plans and 25 states per sequence, optimizer and 25× fewer steps) and only replaces − log of the plan average by the average of − log, giving every plan weight 1/8. It is better than continued CE but worse than Mix-FT in every seed, so the weighting accounts for about three quarters of the likelihood gain (73 to 80%; Table 20). On samples, Mix-FT beats per-path FT in every seed, while per-path FT’s gain over continued CE is small and inconsistent; at 110M the same control ends behind continued CE on the bound and 0.53 to 0.62 nats/token behind Mix-FT on samples (Table 7c). The weighting also lets plans specialize, which is the plan information that Proposition 2 charges for: the effective sample size of eight plan weights falls from 4.3 under continued CE to 3.6 (Table 20).

Table 20. Text8 mechanism control (B=8B = 8, K=4K = 4, exact NLL in bpc at the best temperature). Per-path FT uses Mix-FT’s sequences, plans, replicas and step count with equal plan weights. GG: judge gain at matched distinct-4 against per-path FT. ESS: effective sample size of 8 self-normalized plan weights; gap: per-path bound minus IWAE-8, both on 512 held-out sequences. Lower rows: MM ablation at equal passes, seed 0.

armseed 0seed 1seed 2GG vs per-pathESS / gap (s0)
Base 5× (continued CE)1.79441.79461.7878−0.05 to +0.014.33 / 0.130
Per-path FT (M=8M = 8, equal weights)1.78391.78671.776004.37 / 0.129
Mix-FT (M=8M = 8)1.75181.75521.7449+0.14, +0.14, +0.143.57 / 0.241
Mix-FT, M=1M = 11.7853+0.014.34 / 0.130
Mix-FT, M=2M = 21.7645+0.064.01 / 0.168
Mix-FT, M=4M = 41.7553+0.143.75 / 0.210

Table 20.

Splitting the gain

At K=8K = 8 each step reveals one character, so effect (i) is absent. Mix-FT still beats CE trained at the K=8K = 8 states in every seed, by 0.022 to 0.028 bpc on the exact likelihood, and on samples. The model fine-tuned at K=4K = 4 is as good at K=8K = 8 as the one fine-tuned there, so its K=4K = 4 gain of 0.042 bpc splits into two parts of similar size: 0.025 that persists at one token per step, effect (ii), and 0.016 by which the gap between K=4K = 4 and K=8K = 8 narrows, effect (i) (Section E). Widening the network from 4.8M to 43M parameters shrinks the first part by 0.010 to 0.011 bpc in every seed and leaves the second level, as a capacity account of effect (ii) predicts (Table 22a).

Related work

Masked, block and any-order models. Masked diffusion models [26, 29, 17, 22] and block diffusion Arriola et al. [2] train with CE variants optimized by the true conditionals. Any-order autoregressive models [13, 30] are the one-token-per-step case.

Orders as latent variables. Amin et al. [1] analyze the few-step sampler as a mixture over paths; their Theorem 1 contains Proposition 2. They recover much of the gap at sampling time, with temperature 0.9 and float32 instead of float64 categorical sampling, without retraining; the diagnosis is theirs, and we train the denoiser on the mixture. Fox et al. [8] learn a token selector and a variational posterior over unmasking orders that sees the clean sequence, and LO-ARM [32] learns a stochastic order policy with such a posterior for one-token-per-step generation; we learn no order policy and fine-tune the denoiser on a multi-sample bound of a fixed uniform plan distribution. Eso-LMs [27] use a multi-order bound for evaluation and train with one order.

Trajectory-aware training. PUMBA [19] trains on the few-step policy’s own trajectories with a carry state. Its confidence planner and carry state put it outside Proposition 1 and our dynamic program, though not outside Proposition 2 (Section H). T3D [36] distills the model’s own trajectories with a reverse-KL-like loss and MDPO (He et al., 2025) uses policy gradients with intermediate rewards, both under confidence decoding, scoring a sample with its plan. We analyze only the idealized path-space reverse KL with plans independent of θ\theta: like trajectory balance with a fixed backward policy [28] (Whittamer et al. 2023), it pushes a parallel sampler toward plan-invariant behavior, in exact toys toward independent filling (Section F).

Added latents, heads and teachers. Di4C [11] distills into a mixture of products over an auxiliary variable, E-MoE [14] mixes over expert routes with a single-route bound, VADD [35] adds a Gaussian latent and uses IWAE only for evaluation, and tensor-train heads [15] model the joint of masked tokens. Our mixture latent is the plan the sampler already draws. SDTT [7] matches a few-step student to its many-step self. Alpha diffusion [10] fine-tunes teacher-free with a sequence-level α\alpha-divergence at one masking state; it changes the divergence, not the plan marginal.

Theory of parallel samplers. With exact conditionals, Wu (2026) separate schedule-averaged from mixture KL, Li et al. (2026) show that a uniform mixture of fixed schedules can beat any deterministic adaptive policy, and Chen et al. [6] analyze random-partition schedules. DUEL [31] computes exact likelihoods for deterministic unmasking. Sampling-only methods [3, 18] are complementary.

Limitations and conclusion

  • Scale. Our full set of baselines was run at 110M. At 1.1B and 8B we compared with continued CE and Alpha diffusion only, with LoRA and 256-token sequences. At 8B, Mix-FT at K=8K = 8 stays behind LLaDA at K=16K = 16. On a 1.1B GSM8K model fine-tuned further on its own verified solutions, Mix-FT did not differ from continued CE, as the fixed point of the mixture refit predicts for self-generated data (Section D).

  • Two steps per block. At K=2K = 2, SDTT-4’s samples beat Mix-FT’s under both judges in every seed, and SDTT also ranks ahead at matched distinct-4, partly by repairing a newline excess, more than twice real text’s rate, that Mix-FT keeps (Table 6 and Table 8). At 110M the 8× model closes most of the start’s gap between 8 and 16 steps, but its 4-step samples stay about 0.5 nats/token behind the start at 8.

  • Block size. We tested blocks of 8, 16 and 32, LLaDA-8B’s. With one block over 1,024 tokens, as in MDLM, eight plan weights have a median effective sample size of 1.00, against 1.54 for the start at block 16 and K=4K = 4 (Table 13), so a bound with few plans is loose.

  • Price. Mix-FT trades the per-path bound for the sampler’s: the NELBO perplexity rises from 23.0 to 24.5 at 1×1 \times and to 28.8 at 8×8 \times (Table 11). It costs 1+M(K−1)1 + M(K - 1) passes per sequence and needs a CE-trained start: from under-trained weights the mixture refit loses to continued CE (text8 from scratch and a 34M model, Sections A and E).

  • Conventions. BD3-LM trained on raw windows with the first token clean, we on wrapped ones. Claims against the start use the first-token convention that favors it; the window convention moves BD3-LM’s objective by up to 0.38, and Mix-FT leads on either (Table 7g).

  • Scope. We train on uniform random-order samplers, and Proposition 1 does not cover confidence planners. We did not run PUMBA, Di4C or added-head methods. Judges and raters are models.

Table 13. Block 32 (exploratory). Top: content-token judge gain of the first model over the second at matched per-sample entropy (GPT-2-large / Pythia-1.4B, nats/token; positive: first model better). Mix-FT trains on four-step plans, with M=8M = 8 at the 1×1\times budget and M=64M = 64 at 8×8\times. Rows below the line compare a model at KK steps per block with the start at 2K2K. Every row draws the first token, except those where both models fix it to end-of-text (Section B). Bottom: IWAE-32 bound of the four-step sampler, NELBO perplexity and plan ESS (ranges over seeds).

comparisonseedK=4K = 4K=8K = 8K=16K = 16
Mix-FT vs continued CE00.258 / 0.2430.407 / 0.4090.412 / 0.413
Mix-FT vs continued CE10.310 / 0.2900.455 / 0.4750.430 / 0.436
Mix-FT vs continued CE20.125 / 0.1090.304 / 0.3040.370 / 0.371
Mix-FT vs start00.265 / 0.2640.267 / 0.2960.255 / 0.278
both fix the first token00.253 / 0.2390.241 / 0.2380.144 / 0.152
continued CE vs start00.006 / 0.021−0.140 / −0.113−0.157 / −0.135
both fix the first token00.020 / 0.004−0.115 / −0.123−0.252 / −0.263
Mix-FT 8×8\times vs continued CE 8×8\times00.552 / 0.5580.879 / 0.9090.752 / 0.775
Mix-FT at KK vs start at 2K2K0−1.048 / −0.998−0.679 / −0.650−0.193 / −0.177
Mix-FT 8×8\times at KK vs start at 2K2K0−0.847 / −0.786−0.404 / −0.3670.022 / 0.048
both fix the first token0−0.097 / −0.094
continued CE 8×8\times at KK vs start at 2K2K0−1.399 / −1.344−1.283 / −1.275−0.731 / −0.727
modelIWAE-32, K=4K = 4NELBO PPLmedian ESS of 8 plans, K=4K = 4
Start at block 16 (reference)3.22323.001.54
Start (block-32 masks)3.38723.301.13
Continued CE (3 seeds)3.364 to 3.36522.66 to 22.671.14
Continued CE 8×8\times (seed 0)3.35422.441.14
Mix-FT (3 seeds)3.35123.25 to 23.301.10
Mix-FT 8×8\times (seed 0)3.34824.151.07

Table 13.

Few-step masked diffusion samplers are mixtures over reveal plans, and fine-tuning on that mixture’s likelihood is a cheap, teacher-free change of loss. On LLaDA-8B it puts 16 steps per block ahead of the original’s 32. On a public 110M model it gives the best samples from 4 to 16 steps per block, and with more plans its 8 steps close 95% of the gap to the original’s 16. The two-step regime, where distillation still wins, and full fine-tuning at scale remain open.

Impact Statement

This paper presents work whose goal is to advance the field of machine learning, specifically the efficiency of sampling from diffusion language models. Faster and better few-step generation lowers the cost of running such models, with the same potential for misuse as other text generators. We do not see consequences specific to this method that need discussion here.

The 110M experiments in more detail

Per-seed gaps. Table 6 gives every gain of Mix-FT by seed and judge. From K=4K = 4 to 16 the smallest gain over any baseline is 0.226, under GPT-2-large against Alpha diffusion at K=4K = 4 in seed 0. At K=2K = 2 SDTT-4 is better in every seed, and Mix-FT is level with or slightly ahead of Alpha diffusion.

Controls. Table 7 collects the 110M controls that § 3 summarizes. Removing the plan weights costs Mix-FT more than any other change we tried, and two-step training plans beat finer plans whether the compute or the number of updates is held fixed. More plans per sequence matter less: at equal compute the gain levels off beyond M=32M = 32, while doubling the plans together with the compute adds about 0.06 nats/token at K=8K = 8 per doubling (seed 0).

MAUVE without boundary tokens. The standard MAUVE feature is GPT-2-large’s state at the final token. Every real sequence ends with an end-of-text token and no block-sampler sample does, and with that feature every block-sampler model scores about 0.004 regardless of quality. We drop the first and last position of every sequence, real and generated, before featurizing (real against real: 0.97).

Newline rates. Few-step samplers from the BD3-LM start over-produce newlines, which make up 3.35% of real held-out text (Table 8). Sharper sampling emits more of them, and every model except Alpha diffusion needs β>1\beta> 1 to match real text’s per-sample entropy, so for those models the rate at matched entropy, where the judges compare models, is higher than at β=1\beta= 1. Only SDTT stays near or below real text’s rate at both operating points.

All tokens. Scored on every token, newline and end-of-text targets included, every comparison of Mix-FT with the start and the baselines at K=4K = 4, 8 and 16 keeps its direction in every seed under both judges. Its margin over the strongest fine-tuned baseline becomes 0.24, 0.43 and 0.40 nats/token at K=4K = 4, 8 and 16 under GPT-2-large (content tokens: 0.25, 0.44 and 0.41), and over continued CE 0.51 to 0.68 (content tokens: 0.52 to 0.72). A judge predicts a newline at well under 1 nat, against about 4 for a content token, so all-token scores favor whichever model emits more newlines; scoring content tokens removes that advantage for every model alike.

Diversity and repetition. Table 9 gives the measures behind the summary in § 3. At K=8K = 8 the 8× model is more diverse than the start at K=16K = 16 on Vendi and self-BLEU-4 and level on distinct-4. At equal steps Mix-FT has the higher Vendi score but slightly higher self-BLEU-4 and lower distinct-4, and like Alpha diffusion it repeats more 4-grams within a sample than the start or real text. Repetition does not flatter the judges here. Within the 8× model’s files we regressed each sample’s judge NLL on its per-sample entropy (quadratic) and its rep-4: at K=8K = 8, more repetition at the same entropy goes with a worse NLL in every seed under both judges, and at K=16K = 16 it changes the NLL by at most 0.006 nats/token per percentage point. The blinded raters, asked about repetition among other criteria, find it level at equal steps (§ 3).

The four-step Mix-FT. Our first 110M model used four-step plans (443 steps). Its sampler bound was already better than every baseline’s in every seed, including Alpha diffusion at every κ\kappa (Table 4 and Table 10). Its samples beat continued CE on average at every step count, and SDTT-4, SDTT-8 and Alpha diffusion at K=8K = 8 and 16 in every seed. At fewer steps it was weaker: SDTT-4 was better at K=2K = 2 and mixed at K=4K = 4, and Alpha diffusion was ahead in at least two of three seeds at K=2K = 2 and 4 with either sample count. Figure 4 and Table 10 collect these results with the 128-sample protocol; Table 1 supersedes them for the two-step model.

modelK=4K = 4K=2K = 2NELBOPPLGPT-2-L allGPT-2-L contentPythia content
Start (BD3-LM)3.2233.77523.00
Continued CE3.2093.76422.41000
SDTT-43.2453.79624.41+0.091+0.405+0.329
Hybrid3.2003.75222.69+0.113+0.155+0.152
Alpha diffusion3.2023.75823.16+0.255+0.324+0.321
Mix-FT, K=4K = 4 plans3.1873.74123.29+0.295+0.360+0.349
Mix-FT, K=2K = 2 plans3.1813.72324.51+0.442+0.577+0.563

Table 10. Stage 5 to 8 summary at 110M (128 samples per temperature at K=2,4,8K = 2, 4, 8). GG: judge NLL gain over continued CE at matched per-sample entropy, mean over K=2,4,8K = 2, 4, 8 (positive: better than CE); “all” scores every token with GPT-2-large, “content” excludes newline and end-of-text targets. Bounds: IWAE-32, nats/token; seed spreads at most 0.0013.

Figure 4. Four-step Mix-FT at 110M (and 34M FineWeb-Edu). (a) Sampler likelihood relative to continued CE at equal extra compute (bars: mean of 3 seeds; markers: seeds). (b) Content-token judge gain over continued CE at matched per-sample entropy (solid: GPT-2-large, dotted: Pythia-1.4B; K=16K = 16 from 512 samples). (c) Boundary-free MAUVE at matched entropy (solid: 128 samples, dashed: 512). (d) Newline rate at matched entropy; real text has 3.35%.

Hybrid. Adding the CE loss on the same forward passes (λ=1\lambda= 1) gives about 40% of four-step Mix-FT’s sample gain over continued CE (Table 10). At K=8K = 8 and 16 it also beats continued CE and SDTT-4 in every seed under both content judges, by 0.14 to 0.35 nats/token under GPT-2-large.

Mix-FT after SDTT. Fine-tuning the SDTT-4 student with Mix-FT, against giving it the same compute of further SDTT, lowers its K=4K = 4 bound by 0.063 nats/token in every seed and improves its samples at K=8K = 8 and 16 by about 0.5 nats/token. But it lands almost on the model that Mix-FT reaches from the original weights, within 0.003 on the bound and 0.04 on samples on average, and the nonlinear excess that SDTT had removed returns (Table 8). Mix-FT and SDTT are therefore alternatives rather than stages of one recipe. Further SDTT makes the bound and the samples at K≥8K \ge8 slightly worse.

Alpha diffusion. It reweights each block’s CE by exp⁡(κsˉb)\exp(\kappa\bar{s}_b), where sˉb\bar{s}_b is the model’s mean log-likelihood of the block’s masked tokens. It is a strong baseline: it beats continued CE at every KK, most at K=4K = 4 (+0.47 content, 128 samples), but its gain fades with more steps (+0.10 to +0.11 at K=16K = 16), where it falls below the untuned start. Larger κ\kappa degrades the

In (a), “443 steps” and “1,152 steps” are separate runs with their own learning-rate schedule over that many steps, not snapshots of a longer run. In (b), the M=8M = 8 run at 3× continues the 1× run for 2,304 more steps with a fresh warmup and cosine; the M=32M = 32 run at 3× is one run of 1,016 steps. In (c), per-path FT recovers none of Mix-FT’s gain on K=4K = 4 sampler bound: its gain over continued CE is −0.14 of Mix-FT’s for four-step plans and −0.07 for two-step plans. In (d), schedules are chosen at development split and scored on fresh samples. In (e), one run uses a tenth of the peak learning rate and the other is evaluated with an exponential moving average of its weights (decay 0.999). In (f), ‡ evaluated at the checkpoint chosen on a development split: half of Alpha’s 46,080 steps on two-step states in seed 0 and a quarter in seeds 1 and 2, all of SDTT-8’s 30,720 in seed 0, a quarter in seed 1 and half in seed 2; SDTT-8 counts its teacher’s passes. Rows in (f) pair each seed of the 8× Mix-FT with the same seed of the baseline. In (g), the null fine-tune inputs BD3-LM’s raw training objective (its loss and clipped masking schedule, the first position kept clean for both arms, the null is 0.24 to 0.40 behind the start on average and Mix-FT 0.54 to 0.91 ahead of the null. On our windows and with the first token fixed, the null is level with the start at K=4K = 4 and 0.18 to 0.23 behind it at K=8K = 8 and 16. @page 13/41 starts_mid=0 ends_mid=0

model: its K=4K = 4 bound is 3.466 at κ=2\kappa= 2 and 5.706 at κ=8\kappa= 8, against 3.202 at κ=0.5\kappa= 0.5. Smaller values do not help: κ=0.1\kappa= 0.1 is worse than 0.5 on samples at every KK, and κ=0.25\kappa= 0.25 is worse at K=4K = 4 and 8 and about level at K=2K = 2 and 16 (seed 0).

A 34M model trained by us. The recipe needs a CE-trained start. We trained a 34M block diffusion model (8 layers, width 384, 256 GPT-2 tokens, block 16) on 150M tokens of FineWeb-Edu (Penedo et al., [23]) with the text8 protocol. From Base 1×, where a further unit of CE still lowered the bound by 0.12 nats/token, both mixture arms ended 0.022 nats/token behind continued CE (one seed). From Base 2× both beat continued CE in all three seeds, on the bound and on GPT-2-large NLL at matched entropy, and Mix-FT (M=4M = 4) gains 0.030 to 0.040 nats/token on the bound, about twice the hybrid’s gain (Figure 4). At matched distinct-4 the gains are near zero on average at this scale.

Bounds of the 8× model. Table 11 gives the bounds of the 8× model and its comparators with up to 256 evaluation plans. With all 256, the 8× model is level with continued CE at K=8K = 8 and about 0.03 nats/token behind it at K=16K = 16, in every seed. Its plan weights are much more concentrated than continued CE’s, so its bounds tighten faster as plans are added, and the gap at K=16K = 16 is still closing at 256 plans. The NELBO, which weights every plan equally, is the loosest bound and the one Mix-FT raises most. The 1× two-step model is below continued CE and the start at both step counts. Its bound also tightens faster, so its margin over continued CE at K=16K = 16 is larger with 256 plans than with the 64 plans of § 3.

Confidence sampler. Each step draws a token for every masked position from softmax(β\beta logits), scores it by its untempered probability, and keeps the B/KB/K highest. This is LLaDA’s low_confidence rule, and one step matches LLaDA’s reference code in distribution. Temperatures run from β=0.5\beta= 0.5 to 1.15 so that every model’s sweep brackets real text’s per-sample entropy, and each file has 512 samples. Mix-FT has three seeds and the baselines one. Table 12 gives MAUVE under both planners and the judge gap between them. On this checkpoint the confidence planner is worse than random order at matched entropy for every model at K=8K = 8 and 16, by at least 0.25 nats/token, and at K=4K = 4 for every model except continued CE, which is level. So under this planner the 8×8\times model at K=8K = 8 beats the start’s confidence sampler at K=16K = 16 but stays 0.29 to 0.35 nats/token behind the start’s random-order sampler, and at K=4K = 4 it does not reach the start’s confidence sampler at K=8K = 8.

Block 32 (exploratory). No BD3-LM checkpoint at block 32 is public, so we ran the block-16 start with block-32 masks, which raises its NELBO perplexity only from 23.00 to 23.30. Before training, the median effective sample size of eight plan weights at K=4K=4 was 1.13, below the 1.25 our pilot rule asked for, so we ran the fine-tunes as exploratory follow-ups (Table 13). For the start at block 16 the same median is 1.32, 1.54 and 1.95 at K=2K=2, 4 and 8 (1.06, 1.13 and 1.30 at block 32), and for MDLM’s single block of 1,024 tokens it is 1.00 at K=32K=32 and 128. At the 1× budget, with 443 Mix-FT updates at K=4K=4 and M=8M=8, Mix-FT beats continued CE at K=4K=4, 8 and 16 under both judges in every seed and has the better bound. In seed 0 it also beats the start at every step count, by 0.14 to 0.30 nats/token with the first token drawn or fixed, while continued CE falls behind the start at K=8K=8 and 16 either way. With 8× the budget and M=64M=64 (seed 0), Mix-FT at K=16K=16, two tokens per step, matches the start at K=32K=32 under both judges when both draw the first token (+0.02 and +0.05), but falls 0.10 nats/token behind under both when both fix it to end-of-text, the start’s better convention here; fixing it helps the start at K=32K=32 by 0.11 to 0.13 and leaves the 8× model where it was (Section B). It beats continued CE given the same budget by at least 0.55 nats/token at K=4K=4, 8 and 16.

The first token of a sequence

BD3-LM is trained on 1,024-token windows cut from concatenated documents, each document followed by end-of-text. Position 0 of a window is never masked and never scored, and BD3-LM’s sampler sets it to end-of-text before generating. Our fine-tunes follow MDLM’s data pipeline instead: every window begins and ends with end-of-text, and position 0 is masked and predicted like any other position. Our sampler draws position 0 for every model. The start therefore draws its first token from an output that never received a gradient, and the rest of the first block, and through the cache every later block, conditions on that token.

We sampled every model of the main comparison a second time with the first token fixed to end-of-text from the first step, with the same plans and random draws, so that the two files of each pair differ only in that token: the step whose group contains position 0 still draws a token there, which is discarded, and reveals one fewer position. Regenerated with the first token drawn, all 122 files reproduce the files of the main text token for token. Table 14 gives what the fix changes. It helps the start by 0.05 to 0.11 nats/token at K=8K = 8 and 16 and costs it 0.04 to 0.07 at K=4K = 4. The fine-tuned models, trained to predict end-of-text at position 0, move by at most 0.07 and mostly by less than 0.03, whether they draw end-of-text there often (continued CE, in 56 to 64% of samples) or rarely (Mix-FT 1×1\times in 16 to 21%, the 8×8\times model in 1 to 3%); the start draws it in at most 0.2%.

K=4K = 4K=8K = 8K=16K = 16
gain from fixing the first tokenGPGPGP
Start (BD3-LM)−0.070−0.035+0.055+0.091+0.087+0.111
Continued CE, seed mean+0.005+0.000−0.013−0.002+0.015+0.019
Mix-FT 1×, seed mean−0.044−0.021−0.002+0.010−0.016−0.004
Mix-FT 8×, seed mean––+0.022+0.042−0.002+0.016
with per-step schedules––−0.003+0.014––
comparisonfirst token drawnfirst token fixed
GPGP
Mix-FT 8× at K=8K = 8 vs start at K=16K = 16, seed mean+0.022+0.047−0.043−0.023
with per-step schedules+0.022+0.039−0.060−0.057
share of the start’s gap between K=8K = 8 and 16 closed102%105%95%97%
Mix-FT 8× at K=16K = 16 vs start at $K = 16, seed mean | +0.633 | +0.664 | +0.545 | +0.569
Mix-FT 1× vs start, seed mean, K=4K = 4+0.563+0.581+0.588+0.595
K=8K = 8+0.496+0.518+0.439+0.437
K=16K = 16+0.341+0.363+0.238+0.248
Continued CE vs start, seed mean, K=4K = 4−0.096−0.067−0.021−0.031
K=8K = 8−0.218−0.204−0.286−0.297
K=16K = 16−0.178−0.161−0.250−0.254

Table 14. The first token. Top: judge NLL with the first token drawn minus with it fixed to end-of-text, for the same model, KK and random draws, at matched per-sample entropy (content tokens, nats/token; positive: fixing helps). Judging the drawn files with end-of-text put in place of their first token, so that only the samples differ, changes these by at most 0.014. Bottom: comparisons with the start, both models drawing or both fixing the first token (positive: first model better). G: GPT-2-large, P: Pythia-1.4B. Per-seed values and intervals for the 8× model are in Table 27; with the first token fixed, the 1× model beats the start by 0.54 to 0.62, 0.42 to 0.46 and 0.22 to 0.26 nats/token at K=4K = 4, 8 and 16 in every seed under both judges.

Every comparison among fine-tuned models in the paper uses one convention for both sides and stands as reported. Comparisons with the start are stated under the convention that favors the start, with both numbers where they differ. At K=8K = 8 against the start at K=16K = 16, the 8×8\times model closes 95% (GPT-2-large) and 97% (Pythia) of the start’s own gap between 8 and 16 steps with the first token fixed, and all of it with the first token drawn. The 1.1B results are not affected: the SMDM base model was trained to predict position 0, and our 1.1B pipeline follows its convention.

Bounds. Our bounds score all 1,024 positions of the held-out windows, including position 0, which the start never learned to predict, and position 1,023, which holds end-of-text only in our windows. BD3-LM’s validation scores windows without end-of-text tokens and leaves position 0 out. Table 15 gives the bound differences to the start under both scorings. Three comparisons change sign, continued CE’s IWAE-32 and NELBO and the 8×8\times model’s IWAE-32, and all three are within 0.01 nats/token of the start’s under BD3-LM’s scoring. The 1×1\times Mix-FT model’s bound stays below the start’s under both.

modelIWAE-32, K=4K = 4NELBO
oursBD3-LM’soursBD3-LM’s
Continued CE, seed 0−0.014+0.010−0.026+0.002
Mix-FT 1×, seed 0−0.043−0.024+0.064+0.083
Mix-FT 8×, seeds 0, 1, 2−0.010 to −0.009+0.002 to +0.003+0.221 to +0.227+0.229 to +0.234

Table 15. Bound minus the start’s bound (nats/token; negative: better than the start) under our scoring of the held-out windows (Table 1) and under BD3-LM’s (no end-of-text inside the windows, position 0 left out).

LLaDA-8B

Setting. Table 16 gives the setting. We fine-tune LLaDA-8B-Base [22] with LoRA for prefix continuation on OpenWebText and sample it with its own semi-autoregressive sampler. The 256 positions after BOS form eight blocks of 32; the prompt fills the first two, and the model generates the other six from left to right with KK steps per block, one full bidirectional forward per step, the later blocks masked and no KV cache. We sample with both of LLaDA’s planners: random order, the sampler Mix-FT trains on, and LLaDA’s default low-confidence remasking, which at each step draws a token for every masked position of the block and keeps the 32/K32/K most probable. Every arm starts from the same weights and gets the same 480,000 forward-equivalents, 0.0018% of the pretraining compute, on the same windows. Continued CE and Alpha diffusion train at the four-step random-order sampler’s states, as at 1.1B. Mix-FT trains on four-step plans with M=8M = 8, as in our block-32 runs (Table 13), because at block 32 a two-step plan reveals 16 tokens per step. A second Mix-FT run with M=32M = 32 (seed 0) tests whether more plans help at equal compute, since the start’s plan weights are nearly degenerate at four steps (median ESS of 8 plans 1.19 on the evaluation windows). Before any training, our implementation of the network was checked against the Hub’s own code and LLaDA’s likelihood estimator (Table 16).

startLLaDA-8B-Base (Nie et al., [22]), GSAI-ML/LLaDA-8B-Base, revision 0f2787f2, six bf16 shards (sha256 checked); 8.0B parameters, pretrained on 2.3T tokens; vocabulary 126,464
portour implementation, checked against the Hub’s remote code (transformers 4.46.3) before any training: on 4,104 masked positions its fp32 logits agree within 7×10−47 \times10^{-4} and its masked CE to 10−510^{-5} nats; in fp16, the mode used for training and sampling, its masked CE is within 2×10−42 \times10^{-4} nats of the fp32 reference; LLaDA’s own likelihood estimator gives the same mean on 16 prompt–continuation pairs to 0.002 nats (Monte Carlo SE 2.5)
dataOpenWebText, re-tokenized with LLaDA’s tokenizer, documents joined with EOS; 65,536 training windows of 256 tokens from shard 0; evaluation on 256 windows from the last 3,000 documents of shard 79; prompt: BOS and the window’s first 64 tokens; the model generates the other 192
LoRArank 32, alpha 32, fp32 adapters on all attention and MLP linears (83,886,080 parameters); base frozen in fp16 with an fp32 residual stream, layers split at 16 over the two GPUs of a T4×\times2; per-layer gradient checkpointing
budgetFE: a 257-position pass counts 1, or 3 with a backward. Mix-FT: K=4K = 4, M=8M = 8, 8 examples (window, block) per update with 25 rows each, a plan-weight pass without gradient and a weighted gradient pass (4 FE per row), 600 updates, 480,000 FE; Mix-FT M=32M = 32 (seed 0): 4 examples per update, 309 updates, 479,568 FE; continued CE and Alpha (κ=0.5\kappa= 0.5): 267 rows per update at the four-step random-order states, 600 updates, 480,600 FE. Each is 0.0018% of pretraining, counting a forward and a backward for each of the 2.3T pretraining tokens
optimizerAdamW (0.9, 0.999), no weight decay, lr 2×10−42 \times10^{-4}, 5% warmup, cosine to 0, clip 1, fp16 loss scaling from 2122^{12}, micro-batch 8 rows; window order seed 1000+s1000 + s, LoRA init seed 7000+s7000 + s
temperaturesper model, planner and KK: a bracketing pass on 32 prompts, then four main temperatures where the entropy curve reaches real text’s per-sample entropy +0.10+0.10, −0.10-0.10, −0.25-0.25 and −0.40-0.40 nats, 256 continuations each, and further files until real text’s entropy and 0.3 nats below it are both bracketed; the start’s K=32K = 32 sweep is seeded from its K=16K = 16 sweep
judgesGPT-2-large, Pythia-1.4B and Qwen2.5-7B (read) and Qwen2.5-1.5B (reported) on the decoded text, re-tokenized, with the prompt as context; content tokens only; Qwen2.5-7B in fp16 after an fp32 check; MAUVE on GPT-2-large features of prompt and continuation
boundIWAE-8 of the block-32 random-order sampler at K=4K = 4, 8, 16 on the 256 evaluation windows (one generated block per window), with the median effective sample size of the 8 plan weights

Table 16. 8B setting (LLaDA-8B-Base, LoRA, OpenWebText prefix continuation).

Evaluation. Each model, planner and KK gets its own temperature sweep around real text’s per-sample entropy, with 256 continuations per file, and all models share the plan and sampling noise. We compare models at 110M: judge NLL of the continuation’s content tokens given the prompt, interpolated along each sweep to four entropy levels from real text’s value to 0.3 nats below it and averaged. Three judges are read, GPT-2-large, Pythia-1.4B and Qwen2.5-7B, and a fourth, Qwen2.5-1.5B, is reported. Intervals are a paired bootstrap over prompts with 2,000 replicates, shared by every file and by the real continuations, whose entropy and hence the levels are recomputed in each replicate. The start alone is also sampled at K=32K = 32, one token per step, LLaDA’s default full-quality setting and the block-32 analog of our 110M comparison at K=16K = 16. The expectations, written before any run, were: Mix-FT beats continued CE and Alpha diffusion at each planner and K∈{4,8,16}K \in\{4, 8, 16\} under the three judges in every seed; and, separately for each planner, Mix-FT at K=8K = 8 matches the start at K=16K = 16 (H8≥0H_8 \ge0 under the three judges in every seed), is level with it (seed mean at least −0.03 with a 95% upper bound at least 0 under each judge), or does not reach it; the same rule applies to Mix-FT at K=16K = 16 against the start at K=32K = 32.

Results. Mix-FT gives the best samples of the three fine-tunes at every planner and KK, in every seed under the three judges (Table 17a, Table 18): by 0.22 to 0.60 nats/token over continued CE and 0.18 to 0.40 over Alpha diffusion as seed means, and by at least 0.11 in any seed. The bound does not show this. With eight plans at block 32 it stays close to the per-plan bound (median ESS 1.1 to 1.7 for every model), and the three fine-tunes lie within 0.011 nats/token of each other on it, Alpha diffusion lowest, while their samples differ by up to 0.6. Mix-FT’s MAUVE is also higher than both baselines’ at every planner and KK. At equal steps it beats the untouched start by 0.50 to 0.85 at K=8K = 8 and 16 under either planner and by 0.10 to 0.54 at K=4K = 4, where continued CE and Alpha diffusion are behind the start.

With half the steps the result depends on how many tokens each step reveals. At two tokens per step, the setting of our 110M comparison, Mix-FT at K=16K = 16 is ahead of the start at K=32K = 32 in every seed under all four judges with either planner, and the seed mean’s interval is above zero except under GPT-2-large with random order (Table 17b); its MAUVE is 0.95 against the start’s 0.85 under random order and 0.73 to 0.77 against 0.63 under low confidence. At four tokens per step, K=8K = 8 against the start at K=16K = 16, it is behind under both planners, but most of the start’s own loss from halving the steps is gone: Mix-FT closes 72 to 91% of it with low confidence and 66 to 84% with random order, per seed under the three judges, against −3 to 47% for continued CE and Alpha diffusion (seed 0). More plans help at equal compute: the M=32M = 32 run (seed 0) is better than the mean of the M=8M = 8 runs at every planner and KK under GPT-2-large, by 0.04 to 0.12, and closes 78 to 99% of the start’s gap at K=8K = 8, although its bound is 0.01 worse. At 110M and 1.1B the 8-step result needed M=256M = 256 at eight times the budget (Figure 3a), which we did not run at 8B.

As at 110M, LLaDA’s low-confidence planner is worse than random order at matched entropy for every model and KK under GPT-2-large, by 0.2 to 1.0 nats/token, and the ordering of the methods is the same under both. Matched entropy per sample does not by itself show diversity across samples (Table 19). In the comparison of § 4, Mix-FT at K=16K = 16 against the start at K=32K = 32, Mix-FT has the higher Vendi score under both planners (103.2 and 98.8 against 96.9 and 95.3; real text 103.0) and repeats fewer 4-grams within a continuation; its self-BLEU-4 is higher, 0.18 against 0.16, closer to real text’s 0.21. At K=8K = 8 under random order it has a higher Vendi score than the start at K=16K = 16 (94.4 against 92.8) and than either baseline; under low confidence it is slightly below the start at $K = 16 (80.4 against 82.5) and above both baselines. Its repetition at K=8K = 8 (2.5% of 4-grams) is below the start’s at $K = 16 and close to real text’s 2.8%, and no model produces near-duplicate continuations.

plannermodelKKrep4distinct-4self-BLEU-4near-dup.Vendi
random orderStart320.0520.9300.1600.00096.87
Mix-FT160.0400.9370.1820.000103.20
Start160.0420.9430.1480.00092.79
Start80.0240.9650.1150.00076.70
Mix-FT80.0250.9580.1570.00094.39
Continued CE80.0180.9670.1490.00081.06
Alpha80.0240.9580.1570.00087.62
low confidenceStart320.0450.9360.1590.00095.26
Mix-FT160.0390.9380.1780.00098.81
Start160.0350.9500.1440.00082.46
Start80.0180.9720.1020.00062.29
Mix-FT80.0250.9600.1420.00080.43
Continued CE80.0150.9710.1340.00067.11
Alpha80.0250.9590.1420.00070.37
real continuations–0.0280.9350.2070.000103.00

Table 19. Diversity of the 8B continuations at matched per-sample entropy (seed means; each model at K=8K = 8, Mix-FT also at K=16K = 16; the mean over the four entropy levels of each measure interpolated along the temperature sweep). rep4: share of repeated 4-grams within a continuation; distinct-4 and self-BLEU-4 over the 256 continuations; near-dup.: share of continuations whose 8-gram Jaccard similarity with another exceeds 0.5; Vendi on GPT-2-large features of prompt and continuation.

Blinded raters. We put the half-step result at two tokens per step to the same kind of blinded pairwise rater as at 110M (Section J), with the reading rule written down before any of its samples were drawn. Each model was sampled anew with LLaDA’s low-confidence decoding at its matched temperature: Mix-FT at K=16K = 16 (three seeds) and the start at K=32K = 32. A judgment shows the 64-token prompt and the two 192-token continuations. Each seed contributes 96 pairs, 288 over 191 prompts, every pair rated in both orders in separate files of 40 judgments, with ties counted half and intervals from a bootstrap over prompts. Each model also has 8 control pairs on prompts used by no test pair: its matched-temperature continuation against a hotter one of its own, about 1.35 nats/token worse under GPT-2-large by extrapolation along the temperature sweeps above. A rater whose control interval has its lower bound at or below 0.60 is not read. ChatGPT prefers Mix-FT at K=16K = 16 in 68.8% of the pairs, 95% interval 62.9 to 74.5%, and every seed’s interval is above one half (67.4, 70.8 and 68.0%). It prefers each model’s matched-temperature text in 84% of the control pairs (interval 72 to 95%), and it agrees with itself across the two orders in 85% of the test pairs. Mix-FT is preferred on each criterion: coherence 69%, readability 67%, substance 69% and repetition 62%. Claude Sonnet, one fresh instance per file at maximum effort as in our other Claude ratings, prefers Mix-FT at K=16K = 16 in 71.4% of the pairs, 95% interval 65.6 to 77.1%, and every seed’s interval is above one half (72.4, 70.8 and 71.1%). Its control is 84% (interval 73 to 95%), it agrees with itself across the two orders in 85% of the test pairs, and its verdicts match ChatGPT’s in 82% of the pair orders. It prefers Mix-FT on each criterion as well: coherence 69%, readability 69%, substance 67% and repetition 60%.

1.1B GSM8K

We tested the 1.1B masked diffusion model of Nie et al. [21] fine-tuned by its authors for GSM8K, sampled in random order at two steps. LoRA fine-tuning on its own fine-tuning data (one seed) left Mix-FT and continued CE within 0.2 accuracy points, but the model has nearly memorized that data (about 7 nats per answer under the two-step sampler). We therefore fine-tuned on the model’s own verified solutions to 3,368 new questions: every fine-tune gained 1 to 3 points over the start, and Mix-FT and continued CE, at equal compute or at equal examples, did not differ (three seeds; paired difference −0.1-0.1 points, 95% interval [−1.6-1.6, +1.6+1.6]). We found no gain from the method on this task at this scale. This is what the objective predicts for self-generated data: the sampler’s own law is a fixed point of the exact mixture refit (Section F), so on its own samples the mixture likelihood has little to correct beyond the verification filter.

modelsampler bound K=4K = 4816GPT-2-large, random order K=4K = 481632GPT-2-large, low confidence K=4K = 481632
Start (LLaDA-8B-Base)3.0852.7922.6695.955.013.983.546.385.725.024.48
Continued CE3.0502.7572.6356.274.883.89–6.485.694.83–
Alpha (κ=0.5\kappa= 0.5)3.0452.7542.6326.184.673.74–6.445.554.76–
Mix-FT (K=4K = 4, M=8M = 8)3.0482.7612.6435.854.293.48–6.195.174.38–
Mix-FT M=32M = 32 (seed 0)3.0582.7752.6505.734.213.41–6.125.134.34–

Table a. (a) seed means

model, planner, stepsjudgeseed 0seed 1seed 2seed mean95% intervalverdict
Mix-FT, low conf., 8 vs 16GPT-2-large−0.129−0.196−0.111−0.145[−0.245, −0.044]not reached
Pythia-1.4B−0.084−0.145−0.061−0.097[−0.196, −0.001]
Qwen2.5-7B−0.090−0.143−0.065−0.099[−0.182, −0.018]
Qwen2.5-1.5B−0.094−0.150−0.077−0.107[−0.191, −0.025](reported)
M=32M = 32, seed 0, GPT-2-large−0.109[−0.215, −0.008]
Mix-FT, random, 8 vs 16GPT-2-large−0.348−0.309−0.285−0.314[−0.413, −0.208]not reached
Pythia-1.4B−0.275−0.223−0.217−0.238[−0.322, −0.126]
Qwen2.5-7B−0.210−0.170−0.163−0.181[−0.253, −0.062]
Qwen2.5-1.5B−0.219−0.177−0.171−0.189[−0.264, −0.068](reported)
M=32M = 32, seed 0, GPT-2-large−0.230[−0.337, −0.114]
Mix-FT, low conf., 16 vs 32GPT-2-large+0.108+0.061+0.143+0.104[+0.010, +0.208]match
Pythia-1.4B+0.138+0.090+0.173+0.134[+0.040, +0.229]
Qwen2.5-7B+0.137+0.096+0.168+0.134[+0.045, +0.224]
Qwen2.5-1.5B+0.130+0.088+0.154+0.124[+0.034, +0.216](reported)
M=32M = 32, seed 0, GPT-2-large+0.142[+0.034, +0.254]
Mix-FT, random, 16 vs 32GPT-2-large+0.045+0.080+0.077+0.067[−0.018, +0.154]match
Pythia-1.4B+0.077+0.112+0.120+0.103[+0.021, +0.186]
Qwen2.5-7B+0.132+0.188+0.178+0.166[+0.088, +0.250]
Qwen2.5-1.5B+0.136+0.186+0.178+0.167[+0.090, +0.252](reported)
M=32M = 32, seed 0, GPT-2-large+0.137[+0.036, +0.246]
CE, low conf., 8 vs 16GPT-2-large−0.721−0.664−0.618−0.668–
Pythia-1.4B−0.668−0.609−0.566−0.614–
CE, random, 8 vs 16GPT-2-large−0.913−0.905−0.902−0.907–
Pythia-1.4B−0.845−0.825−0.833−0.834–
Alpha, low conf., 8 vs 16GPT-2-large−0.515−0.545−0.516−0.525–
Pythia-1.4B−0.477−0.519−0.481−0.492–
Alpha, random, 8 vs 16GPT-2-large−0.694−0.683−0.705−0.694–
Pythia-1.4B−0.642−0.641−0.639−0.641–

Table b. (b) half the steps: each model at KK against the start at 2K2K

Text8 details

Table 21. Text8 per seed: exact KK-step NLL (bpc, block 8) and IWAE-64 (block 16) at the best temperature. K=8K = 8 rows: exact NLL at one token per step.

armseed 0seed 1seed 2
block 8, K=4K = 4, exact
C0 (masked CE pretraining)1.95951.95401.9339
Base 1× / Mix 1× (from C0)1.8988 / 1.90931.8862 / 1.90311.8768 / 1.8830
Base 4× / Mix 4× (from C0)1.8076 / 1.87351.8054 / 1.86871.8005 / 1.8545
Base 5×1.79441.79461.7878
SDTT from Base 4×1.80411.80331.7974
Hybrid from Base 4×1.76531.76581.7611
Mix-FT from Base 4×1.75181.75521.7449
block 8, K=8K = 8, exact
Base 4×1.71121.71421.7036
CE at K=8K = 8 states / Mix-FT trained at K=8K = 81.7011 / 1.67661.7042 / 1.67651.6896 / 1.6671
Mix-FT trained at K=4K = 41.67421.67771.6673
block 16, K=4K = 4, IWAE-64
Base 5× / SDTT1.9608 / 1.97141.9631 / 1.97041.9636 / 1.9721
Hybrid / Mix-FT1.9300 / 1.92201.9316 / 1.92341.9342 / 1.9245
IWAE-512 (first 256 sequences): Base 5× / Mix-FT2.0080 / 1.96852.0089 / 1.96492.0094 / 1.9646

Table 21.

Decomposition arithmetic. Means over seeds:

K=4 gain:1.7923−1.7506=0.042,part that persists at K=8:1.6983−1.6731=0.025,gap narrowing:(1.7923−1.6983)−(1.7506−1.6731)=0.094−0.078=0.016.\begin{aligned} K = 4\ \text{gain:}\quad&1.7923 - 1.7506 = 0.042,\\ \text{part that persists at } K = 8:\quad&1.6983 - 1.6731 = 0.025,\\ \text{gap narrowing:}\quad&(1.7923 - 1.6983) - (1.7506 - 1.6731) = 0.094 - 0.078 = 0.016. \end{aligned}

The second line compares the K=4K = 4-trained Mix-FT with CE trained at the K=8K = 8 states, both sampled at K=8K = 8. The two parts sum to 0.041 because of rounding.

Width. Table 22a reruns the whole protocol, pretraining included, at widths 256, 512 and 768 (4.8M, 19M and 43M parameters) and splits the K=4K = 4 gain at each width. The wider networks are much better models: continued CE is 0.13 bpc lower at width 768 than at 256. The part of Mix-FT’s gain that persists at one token per step shrinks with width, by 0.010 to 0.011 bpc in every seed against continued CE, while the narrowing of the gap between K=4K = 4 and K=8K = 8 stays level, as a capacity account of effect (ii) predicts. Against CE trained at the K=8K = 8 states the split is the same, and at width 256 this rerun reproduces the gains of Table 21 within 0.001 bpc per seed.

Two-step training plans. Table 22b repeats the 110M comparison of training plans where the likelihood is exact. At B=8B = 8 a two-step plan reveals half a block per step, so we did not know whether the 110M result would carry over. At equal compute it does: two-step plans beat the four-step recipe on the exact K=4K = 4 likelihood, by 0.002 to 0.005 bpc, and on samples, in every seed. Unlike at 110M, the extra updates matter for the likelihood: with the four-step run’s number of updates, the two-step model is 0.005 bpc worse at K=4K = 4 in every seed, though its samples are still better. Each model is best near the step count it trained for: at K=8K = 8 the two-step model is 0.024 to 0.028 bpc worse than the four-step one, at K=2K = 2 about 0.04 better. As at 110M, the start’s plan weights are more concentrated at K=2K = 2 than at K=4K = 4 (median effective sample size of eight plans 3.60 to 3.65 against 4.21 to 4.29).

More plans, and more CE. Table 22c trains Mix-FT with M=8M = 8, 32, 128 and 256 plans per sequence for the same number of updates, so its compute grows with MM. The exact K=4K = 4 NLL improves in every seed up to M=128M = 128 (M=32M = 32 by 0.011 to 0.014 bpc over M=8M = 8, M=128M = 128 by a further 0.005 to 0.007) and is level from 128 to 256 plans (mean −0.002). Over the same ladder the gap between the eight-plan IWAE bound and the exact NLL widens by 0.03 bpc, so a bound with few evaluation plans understates the gain, as it may for the 110M 8× model (Table 11). The samples improve along the whole ladder. Continued CE given the compute of M=256M = 256, as 769 times the updates, reaches a much lower exact NLL, 1.651 against 1.732 bpc (seed 0), but worse samples: its GG against Base 5× is below that of M=8M = 8 at one unit. More plans buy samples that more CE does not, and the exact likelihood alone does not rank these models by their samples.

SDTT on text8. SDTT ([7]) from the same weights, following the paper’s algorithm and optimizer with teacher passes counted, does learn its own objective: its held-out distillation KL falls. But it improves on its start by only 0.002 to 0.004 bpc, less than a fifth unit of CE gives, and its samples are worse than continued CE’s in every seed. At the token level the same implementation removes the newline excess and is the best model at two steps per block (Table 1 and § 7).

Why SDTT does poorly on text8. The data point to two features of this setting, though we have not isolated either. First, at B=8B = 8, K=4K = 4 SDTT runs one round, and its teacher is the same network at one character per step, which is 0.095 bpc better than the student under the exact likelihood (Base 4×: 1.7096 at K=8K = 8, 1.8045 at K=4K = 4). Second, its target is the teacher’s per-position marginals, which by Propositions 1 and 2 carry no within-step dependence, and in character-level text the two characters filled in one step of an 8-character block can belong to the same word.

From scratch. A gradient probe on a small CPU proxy shows that near CE-trained weights the IWAE gradient carries about as much signal per pass as CE’s, but each step costs 25 passes. Once Mix has moved the weights, its plan weights become more unequal (effective sample size 3.8 to 4.1 of 8, against 4.7 to 4.9 at CE weights) and its gradient becomes 5 to 7 times noisier per pass. Raising the learning rate made it worse.

Exact toys

Table 23. Exact toy (L=8L = 8, V=4V = 4, HMM mixture, entropy 8.95 nats). KL(p∥qp \parallel q) in nats for the KK-step sampler with the true conditionals, after tuning a temperature on them, and after re-optimizing all conditionals for the sampler’s exact likelihood. “Fixed” is a left-to-right plan.

Single-trajectory objectives. When fitting a target π⋆\pi^{\star} from rewards or energies, KL(qθ(x,d)∥π⋆(x)μ(d))=KL(qθ∥π⋆)+Ex∼qθKL(πθ(⋅∣x)∥μ)\mathrm{KL}(q_{\theta}(x,d) \parallel\pi^{\star}(x)\mu(d)) = \mathrm{KL}(q_{\theta} \parallel\pi^{\star}) + \mathbb{E}_{x\sim q_{\theta}}\mathrm{KL}(\pi_{\theta}(\cdot\mid x) \parallel\mu). The extra term vanishes only when every plan yields the same law. Plan invariance does not imply independence, but in the Ising toy the factorized sampler reached it by filling positions nearly independently. On a 3×33 \times3 Ising model at inverse temperature 0.9 (K=3K = 3), the path objective collapses the sampler’s total correlation from 1.72 to 0.17 nats, while the marginal objective keeps 1.42 (Table 25). Self-training is a second case: per-token CE on the sampler’s own samples converges to the true conditionals of qθq_\theta, and iterating it re-factorizes the law each round. The per-path refit turns the K=2K = 2 HMM sampler into a near-product law in five rounds; the IWAE refit drifts at a rate that falls with MM, and the exact mixture refit is a fixed point (Figure 8). A test of reward-filtered self-training on Sudoku with a 5M-parameter transformer (two seeds) found no advantage for the IWAE refit.

target, KKobjectiveKL⁡(q∣π⋆)\operatorname{KL}(q \mathbin{|} \pi^\star)TCTC of π⋆\pi^\star
HMM (L=6L = 6), K=2K = 2true conditionals0.13750.2400.806
path reverse KL0.11970.268
marginal reverse KL0.00080.769
Ising 3×33 \times3, inv. temp. 0.9, K=3K = 3true conditionals1.1441.7204.546
path reverse KL0.6800.171
marginal reverse KL0.5731.419

Table 25. Fitting a target from its energy alone: KL⁡(q∥π⋆)\operatorname{KL}(q \mathbin{\|} \pi^\star) and total correlation (TC; 0 is a product law), nats, after 600 (HMM) or 300 (Ising) Adam steps from the true conditionals. The path objective had converged; the marginal one was still improving slowly.

Figure 8. Self-training on exact toys. Each round refits the sampler’s conditionals to its own law. The per-path refit removes dependence round after round; the IWAE refit drifts more slowly as MM grows; the exact mixture refit is a fixed point. Dotted: total correlation of the target.

The HMM targets are mixtures of sticky hidden Markov chains (L=6L = 6, V=3V = 3: 729 sequences; L=8L = 8, V=4V = 4: 65,536 sequences). The model is a free logit table over partial states, the same conditioning a network has; Adam minimizes the exact objective through a differentiable dynamic program from the true-conditional start. In Table 23 the K=2K = 2 random-order value is still falling after 150 Adam steps (0.0299); at L=6L = 6 it reaches 0.0008 after 2,000 steps.

Exact dynamic program

For one block with BB positions, KK steps and g=B/Kg = B/K, let Tk\mathcal{T}_k be the subsets of size kgkg. For T∈TkT \in\mathcal{T}_k, f(T)f(T) is the total probability that the sampler has revealed exactly TT after kk steps with tokens xTx_T, summed over ordered partitions of TT:

log⁡f(T)=logsumexp⁡S⊂T, ∣S∣=(k−1)g[log⁡f(S)+∑i∈T∖Slog⁡rθβ(xi∣xS)],qθ(x)=f({1,…,B})B!/(g!)K.\log f(T) = \operatorname{logsumexp}_{S \subset T,\ |S|=(k-1)g} \left[\log f(S) + \sum_{i \in T \setminus S} \log r^\beta_\theta(x_i \mid x_S)\right], \qquad q_\theta(x) = \frac{f(\{1,\ldots,B\})}{B!/(g!)^K}.

One forward pass per reveal set gives log⁡rθβ\log r^\beta_\theta at all positions and temperatures; blocks share passes because they are conditionally independent given the clean prefix. Reveal sets per sequence: 127 (B=8B = 8, K=4K = 4), 255 (B=8B = 8, K=8K = 8), 16,511 (B=16B = 16, K=4K = 4, not affordable; we report IWAE-64 and IWAE-512 there). A self-test against enumeration of all plans for several small (B,KB, K) gives errors below 1.6×10−61.6 \times10^{-6} nats.

Extended related work: PUMBA

PUMBA ([19]) trains the denoiser on consecutive steps of the few-step policy’s own trajectories, with teacher-forced commits, a carry state and truncated backpropagation through time, scales this to fine-tuning LLaDA-8B, and argues that training should follow the states the sampler visits, finding that a looser alignment helps where exact alignment overfits. Three points reconcile it with our argument. First, when the next group is a deterministic function of the revealed tokens, the per-trajectory likelihood is the sampler’s likelihood, so trajectory training targets the right quantity for such samplers; our argument concerns random-order samplers, and PUMBA’s planner reads the model’s own confidence, which Proposition 1 does not cover. Second, the carry state makes the conditionals depend on the trajectory, not only on the revealed set; Proposition 2 still holds for any mixture over plans, but Equation (1) and our dynamic program do not. Third, PUMBA’s critique targets random-masking CE, whose states do not match the sampler’s; our CE baseline trains at the sampler’s own states and is the closest analogue of trajectory training without a carry, and the mixture fine-tune beats it. A comparison under a policy-driven sampler remains open.

Hyperparameters

110M OpenWebText
startBD3-LM, OpenWebText, block 16 (kuleshov-group/bd3lm-owt-block_size16); 12 layers, width 768, 12 heads, 110M parameters as reported by Arriola et al. [2] (169.6M stored values, 92.4M outside the input embedding and output layer); L=1024L = 1024, GPT-2 tokenizer; NELBO perplexity 23.0 on our held-out split
dataOpenWebText; training sequences from the first of 80 parquet shards; held-out: the last 3,000 documents of the last shard
equal compute276,480 forward-equivalents (FE) per arm; batch 8 × 1024 tokens
forward-equivalentsa 1,024-token pass counts 1, or 3 with a backward; BD3-LM’s pass over the clean context counts too, so a CE sequence costs 6 FE and a Mix-FT sequence 3(2+M(K−1))3(2 + M(K - 1)). The checkpoint was pretrained for 1M steps of 512 sequences (524B tokens; 850K steps at block 1,024 and 150K at block 16; Arriola et al. [2]), at least 1.76×1091.76 \times10^{9} FE by this count, so the 1× budget is under 0.02% of it and the 8× budget under 0.13% (0.18% as the 8× Mix-FT runs executed it; next row)
chunked runsruns with M≥128M \ge128 do not fit a T4’s memory in one pass. The code first computes the plan weights without gradient, then repeats the forward pass with gradient in chunks of 32 plans (Section J), which executes about 1.4 times the counted FE (1,088 instead of 774 per sequence at M=256M = 256) and gives the same loss and gradient as one pass (largest relative gradient difference 7×10−157 \times10^{-15} in fp64 at M=256M = 256). We count the single gradient pass that the method needs. A block-chunked backward, which puts all plans of a block in one chunk instead of chunking by plans, gives each block’s weights inside its chunk and removes the pass without gradient: our implementation of it executes exactly the counted 774 FE per sequence at M=256M = 256, holds 32 rather than 34 sequence-equivalents per forward pass, and gives the single-pass loss and gradient (largest relative gradient difference 1.1×10−141.1 \times10^{-14} in fp64). At 1.1B every Mix-FT run takes that pass, and there we count it (4 FE per row)
Mix-FT (main)K=2K = 2 plans, M=8M = 8, 1,152 steps; ablations: K=4/M=8K = 4/M = 8 443 steps, K=4/M=3K = 4/M = 3 1,047, K=4/M=2K = 4/M = 2 1,440, K=2/M=4K = 2/M = 4 1,920
continued CE, Alpha5,760 steps, CE (Alpha: its reweighted CE) at the K=4K = 4 sampler’s states, same batches; Alpha block weights centered by a constant measured once on the start model; Alpha κ∈{0.1,0.25,0.5,2,8}\kappa\in\{0.1, 0.25, 0.5, 2, 8\}, best κ=0.5\kappa= 0.5
optimizer (CE, Mix-FT, hybrid, Alpha)AdamW (0.9, 0.999), no weight decay, peak lr 5×10−55 \times10^{-5}, warmup min(100, [steps/8]), cosine to 10%, clip 1, fp16 autocast
SDTT-4 / SDTT-82 rounds of 1,920 steps (16 to 8 to 4 steps per block) / 1 round of 3,840 (16 to 8); reverse KL, lr 6×10−56 \times10^{-5} constant after warmup, EMA 0.999, teacher passes counted
likelihoodIWAE-32 of the KK-step sampler, K∈{4,2}K \in\{4, 2\}, 256 held-out sequences, β∈{1,1.1,1.25}\beta\in\{1, 1.1, 1.25\}, best reported; NELBO by the order-averaged estimator
generation512 samples of 1,024 tokens per temperature, β∈{0.95,1,1.05,1.1,1.15,1.25}\beta\in\{0.95, 1, 1.05, 1.1, 1.15, 1.25\}, K∈{2,4,8,16}K \in\{2, 4, 8, 16\}; exact categorical sampling in fp32; one T4, batch 64
judgesGPT-2-large (fp32) and Pythia-1.4B (fp16, checked against fp32); content tokens exclude newline and end-of-text targets; matched per-sample entropy at real −0.3, −0.2, −0.1, 0 nats

Table 35.

text8
model6 layers, d=256d = 256, 4 heads, 4.82M parameters; sequences of 256 characters; block-causal attention over [clean | noisy]
C020,000 steps, lr 10−310^{-3}, warmup 200, cosine to 10−410^{-4}
one unit691,200 sequence passes: 21,600 steps of Base CE (lr 3×10−43 \times10^{-4}) or 864 of Mix-FT (M=8M = 8, 25 passes per sequence)
optimizerAdamW (0.9, 0.95), weight decay 0.01, clip 1.0, batch 32, fresh optimizer state at every restart
SDTTreverse KL, dense Algorithm-1 targets, lr 6×10−56 \times10^{-5}, no weight decay, EMA 0.999, log⁡2(B/K)\log_2(B/K) rounds, teacher passes counted
evaluationexact NLL (B=8B = 8) or IWAE-64 (B=16B = 16) at 10 temperatures, IWAE-512 (B=16B = 16) at 4; 512 samples at β∈{0.8,…,4}\beta\in\{0.8, \ldots, 4\}; judge: causal character transformer (4 layers, d=256d = 256)

Table 36.

Reproducing the half-step, baseline, 1.1B and rater experiments

All training, sampling and judging with open models ran on Kaggle T4 GPUs (16 GB), one job per GPU and usually two jobs per T4×2 kernel; the 8× Mix-FT runs (stage 20) and the schedule jobs (stage 21) had a kernel each. A job longer than one kernel (one Kaggle session, 11 to 11.8 h) checkpointed and resumed in the next. Stage numbers are those of Section K. Each stage’s scripts check the md5 of every imported earlier script and refuse to run otherwise; the md5s are listed in the stage’s handoff file.

The 8×8\times half-step model (110M). Table 26 gives the recipe. It is the 1×1\times two-step recipe of § 17 with M=256M = 256 plans per sequence and 8 times the budget. Each update evaluates the 256 plans in 8 chunks of 32: a pass without gradient over all chunks gives the self-normalized plan weights, then a pass with gradient backpropagates −∑mw~mlog⁡qθ,dm(x)-\sum_m \widetilde{w}_m \log q_{\theta,d_m}(x) chunk by chunk. This is the exact IWAE gradient (relative difference to the unchunked one 7×10−157 \times10^{-15} in fp64) and keeps every forward at 34 rows.

startthe public BD3-LM checkpoint of § 17 (no earlier fine-tune)
plansK=2K = 2 steps per block in training, M=256M = 256 plans per sequence, batch 8 sequences of 1,024 tokens, micro-batch 1
budget3(2+M(K−1))=7743(2 + M(K - 1)) = 774 FE per sequence; 357 updates; 2,210,544 FE = 7.9953×\times the 1×1\times budget of 276,480 (under 0.13% of pretraining); the chunked backward executes 1,088 FE per sequence, 3,107,328 in all, 0.18% of pretraining (§ 17)
sequences seen2,856, a prefix of the 1×1\times run’s data order (no sequence repeats); same plan generator (RNG code 52, seed 1000s + 52)
optimizerAdamW (0.9, 0.999), no weight decay, peak lr 5×10−55 \times10^{-5}, warmup min⁡(100,⌊357/8⌋)=44\min(100, \lfloor357/8\rfloor) = 44 steps, cosine to 10% over the 357 steps, clip 1, fp16 autocast, gradient checkpointing
resultIWAE-32 at K=4K = 4: 3.2141, 3.2128, 3.2145; at K=2K = 2: 3.7293, 3.7289, 3.7300; NELBO PPL 28.82, 28.68, 28.85 (seeds 0, 1, 2)

Table 26. Recipe of the 8×8\times model (mixft_k2m256x8, seeds 0, 1, 2).

Global temperature (stage 20). Every model is sampled at β∈{0.95,1,1.05,1.1,1.15,1.25}\beta\in\{0.95, 1, 1.05, 1.1, 1.15, 1.25\}, 512 samples of 1,024 tokens per β\beta, with the same sampling seeds for every model. For each judge we compute the content-token NLL and the per-sample entropy (the entropy of each sample’s own token frequencies, averaged over samples) at every β\beta. We then interpolate the NLL along the model’s β\beta sweep at four entropy levels, real text’s entropy minus 0.3, 0.2, 0.1 and 0 nats, and average the four values. Real held-out text has 5.397 nats, so the levels are 5.097 to 5.397, and at least 3 of the 4 must be bracketed by the sweep. H8H_8 is this value for the start at K=16K = 16 minus that of the model at K=8K = 8 (positive: K=8K = 8 better).

Per-step schedules (stage 21). Step k∈{0,…,K−1}k \in\{0,\ldots,K - 1\} uses βk=bexp⁡(αuk+γ(uk2−u2‾))\beta_k = b \exp(\alpha u_k + \gamma(u_k^2 - \overline{u^2})) with uk=2k/(K−1)−1u_k = 2k/(K - 1) - 1, so bb is the geometric mean. There are nine shapes, α∈{−0.15,0,0.15}×γ∈{−0.3,0,0.3}\alpha\in\{-0.15, 0, 0.15\} \times\gamma\in\{-0.3, 0, 0.3\}, each swept over b∈{0.95,1,1.05,1.1,1.15,1.25,1.35,1.5}b \in\{0.95, 1, 1.05, 1.1, 1.15, 1.25, 1.35, 1.5\}. The shape is chosen per model on a development split of 256 samples per (shape, bb), drawn with the evaluation seeds plus 15,000,000 and so disjoint from every evaluation seed. We pick the shape with the largest mean over the two judges of its matched-entropy gain over the constant schedule if that mean is positive, and the constant schedule otherwise. The chosen shape is then sampled afresh (512 samples per bb, evaluation seeds) and only these fresh rows enter the reading. All three 8×8\times seeds and the start chose α=0.15\alpha= 0.15, γ=−0.3\gamma= -0.3.

Bootstrap. The bootstrap uses B=2,000B = 2{,}000 replicates. In each, every (model, KK, $\beta) file of 512 samples is resampled with replacement, independently across files, and the same indices are applied to the per-sample entropy and to every judge’s per-sample content NLL sums and counts; the whole interpolation is then recomputed. The start’s files are resampled once per replicate and shared by the three seeds, so the seed mean carries that dependence; the real-text entropy is held fixed; a replicate with fewer than 3 bracketed levels counts as −∞-\infty. Intervals are 95% percentiles.

Third judge and margin. The third judge is Qwen2.5-1.5B base (commit 8faed761, fp16 after an fp32 check), scoring content tokens by the Pythia judge’s rule applied to its own tokenizer. The equivalence (noninferiority) margin is −0.03-0.03 nats/token on the seed-mean H8H_8: the reading is “within the margin” if the lower 95% bound exceeds −0.03-0.03 under both GPT-2-large and Pythia, and “better” if it exceeds 0 under both. The margin was fixed before any bootstrap replicate or third-judge score was computed; the point estimates of H8H_8 were known by then. Table 27 gives H8H_8 per seed and the seed mean with its interval, under each judge and in both temperature settings. The one-sided bootstrap pp of H8≤−0.03H_8 \leq-0.03 is at most 0.002 for every seed mean and judge. These readings are for samples whose first token is drawn. With the first token fixed for both models, the same bootstrap puts the seed-mean lower bound below −0.03-0.03 under both judges, so there H8H_8 is neither within the margin nor better (Table 27).

Share of the gap, and all tokens. The same bootstrap gives an interval for the share of the start’s own gap between K=8K = 8 and 16 that the $8\timesmodelcloses,$1+H8/(start at K=8 minus start at K=16)model closes, \$1 + H_8/(\text{start at }K = 8\text{ minus start at }K = 16), with both terms recomputed in every replicate. With the first token fixed for both models it is 95% [92,99][92, 99] under GPT-2-large and 97% [94,102][94, 102] under Pythia; with it drawn, 102% [99,106][99, 106] and 105% [101,110][101, 110]. Scoring all tokens, newlines and end-of-text targets included, moves H8H_8 toward the 8×8\times model, whose K=8K = 8 samples carry more newlines than the start’s at K=16K = 16 (5.5% against 4.2% at matched entropy, Table 8: with the first token fixed, the all-token seed-mean H8H_8 is −0.023 [−0.051, +0.009] under GPT-2-large and −0.011 [−0.046, +0.026] under Pythia, and with it drawn +0.049 [+0.018, +0.080] and +0.065 [+0.031, +0.100]. Content tokens stay our primary measure because they do not reward the newline excess.

The 16× model (110M). It doubles the 8× recipe once more: M=512M = 512 plans per sequence, 358 updates of eight sequences, 3(2+M(K−1))=1,5423(2 + M(K - 1)) = 1,542 FE per sequence and 4,416,288 in all, 16 times the 1× budget and 0.25% of pretraining. It ran on two T4s with the block-chunked backward of § 17, which executes exactly the counted FE; everything else is the 8× recipe. Table 27 gives its H8H_8 under both first-token conventions, with the bootstrap above. Every seed gains 0.03 to 0.08 over the 8× model under either convention. With the first token drawn its 8 steps beat the start’s 16 in every seed under both judges. With it fixed, the start’s convention, the seed means are −0.005 under GPT-2-large and +0.016 under Pythia, and the GPT-2-large interval misses the −0.03 margin by 0.006. At equal steps it beats the start by 0.58 to 0.62 nats/token with the first token fixed. Its IWAE-32 bound is level with the start’s at K=4K = 4 (0.009 above the 8× model’s) and 0.04 nats/token below it at K=2K = 2, and its NELBO perplexity rises to 29.8 (8×: 28.8; start: 23.0).

Table 27. H8H_8 of the 8× and 16× models at 110M: the content-token judge NLL of the start at K=16K = 16 minus that of the fine-tuned model at K=8K = 8, at matched per-sample entropy (nats/token; positive: 8 steps better), per seed and as the seed mean with its 95% bootstrap interval over samples. Global: one temperature per model. Schedule: each model’s own per-step temperature schedule, chosen on a development split. The first two blocks draw the first token of each sequence, as every table does; the next two fix it to end-of-text for both models (Section B) and were not scored by the third judge. The last two give the 16× model with one temperature under each convention.

The 8× baselines (110M). Table 28 lists them. All use the 8× budget, the same token cache and peak learning rate as their 1× versions, and sequences beyond the 110,721 of the cache from a fixed permutation of OpenWebText shards 1 to 4, so no sequence repeats. Stage 22 evaluates the final weights. Stage 25 instead saves the weights at 1/4, 1/2, 3/4 and all of the steps (EMA weights for SDTT) and samples each checkpoint at K=8K = 8 on a development split (256 samples at each of the six global temperatures, evaluation seeds plus 15,000,000). It then evaluates one checkpoint only: the one with the largest mean matched-entropy gain over the final checkpoint under the two judges if that gain is positive, and otherwise the final one. In seed 0 SDTT-8 kept its final checkpoint and Alpha-2 its half-budget one, in seed 1 both kept the quarter-budget one, and in seed 2 SDTT-8 kept the half-budget one and Alpha-2 the quarter-budget one.

The 1.1B experiments. Table 29 gives the setting. The sampler fills a 256-position canvas in blocks of 16 from left to right; positions after the current block hold the mask token, each step is one full bidirectional forward, and there is no KV cache. The bound is IWAE-32 per block on 512 held-out (window, block) pairs. Stage 14 sampled the start and every 1× model at K∈{2,4,8}K \in\{2, 4, 8\}; stage 23 sampled the same models at K=16K = 16 with the same seeds and no further training; stage 27 trained the 8×8\times model and sampled it at K∈{4,8}K \in\{4, 8\}. In stage 27 each job ran on both GPUs of a T4×2. The 2,827 rows of an update are split in two halves. The per-example plan log-likelihoods are summed across the two processes before the plan weights are formed, so the weights are exact, and the LoRA gradients are summed by one all-reduce before clipping and AdamW. The two replicas’ parameters were compared every 10 steps and stayed identical. Intervals at 1.1B are the bootstrap above with 256 samples per file.

startSDMD 1.1B base ([21]), nieshen/SDMD, file mdm_safetensors/mdm-1028M-330e18-rs1-0.01-bs-1024.safetensors, revision f28f1dfe, sha256 1b7378e0; 1,028M non-embedding parameters, pretrained on SlimPajama with 3.3 × 102110^{21} FLOPs; TinyLlama (Llama-2) tokenizer
dataOpenWebText: 65,536 training windows of 256 tokens from shard 0 (documents packed with EOS); 2,048 held-out windows from the last 3,000 documents of shard 79; real reference: held-out windows 0 to 1,023 (per-sample entropy 4.7235 nats)
LoRArank 32, alpha 32, all attention and MLP linears (24,084,480 parameters); base frozen in fp16
samplerL=256L = 256, block 16; FE: a 256-position pass counts 1, or 3 with a backward
1×1\times (stage 14)Mix-FT: K=2K = 2, M=8M = 8, 22 examples (window, block) per update, 600 updates, 475,200 FE; continued CE and Alpha (κ=0.5\kappa= 0.5); 267 rows per update at the K=4K = 4 states, 600 updates, 480,600 FE; under 0.01% of pretraining
8×8\times (stage 27)Mix-FT: K=2K = 2, M=256M = 256, 11 examples per update, 339 updates, 3,833,412 FE = 7.99 × 480,000 (about 0.06% of pretraining)
optimizerAdamW (0.9, 0.999), no weight decay, lr 2×10−42 \times10^{-4}, 5% warmup (30 and 16 steps), cosine to 0, clip 1, fp16 loss scaling, micro-batch 16 rows; window order seed 1000 + ss, LoRA init seed 7000 + ss
samplingβ∈{0.9,1,1.1,1.2,1.35,1.6}\beta\in\{0.9, 1, 1.1, 1.2, 1.35, 1.6\}, 256 samples per file, batch 64, seeds 4242 / 5353 + 100,000KK + 1,000β\beta (identical across models)
judgesGPT-2-large (fp32, openai-community/gpt2-large@212095d5) and Pythia-1.4B (fp16 after an fp32 check, EleutherAI/pythia-1.4b@fedc38a1) on the decoded text, re-tokenized; content tokens only; matched-entropy levels 4.423 to 4.723 nats

Table 29. 1.1B setting (SMDM base, LoRA, OpenWebText).

modelmeasureseed 0seed 1seed 2seed mean95% interval
Mix-FT 8×8\times (M=256M=256)H8H_8, GPT-2-large+0.090+0.027−0.037+0.027[−0.107, +0.155]
H8H_8, Pythia-1.4B+0.151+0.149+0.098+0.133[+0.013, +0.242]
MAUVE at K=8K=80.9320.9300.8740.912–
Mix-FT 1×1\times (M=8M=8)H8H_8, GPT-2-large−0.013−0.115−0.151−0.093[−0.213, +0.026]
H8H_8, Pythia-1.4B+0.073−0.023−0.043+0.002[−0.111, +0.111]
MAUVE at K=8K=80.8850.8650.8980.883–
Continued CEH8H_8, GPT-2-large−0.760−0.890−0.962−0.871–
H8H_8, Pythia-1.4B−0.666−0.769−0.817−0.751–
MAUVE at K=8K=80.5260.5390.4980.521–
Alpha (κ=0.5\kappa=0.5)H8H_8, GPT-2-large−0.805−0.668−0.896−0.790–
H8H_8, Pythia-1.4B−0.754−0.645−0.765−0.721–
MAUVE at K=8K=80.6060.5810.5140.567–

Table 30b. (a) seed means. (b) half the steps: each model at K=8K=8 against the start at K=16K=16.

Blinded pairwise raters. Every pair shows two complete unconditional samples of 1,024 GPT-2 tokens, decoded exactly, each end-of-text token shown as [document break] on its own line; nothing is excerpted. Table 32 defines the three contrasts, c1 (the 8×8\times model at K=8K=8 against the start at K=16K=16), c0 (the start against itself at K=8K=8 and 16) and c3 (the 8×8\times model against the start, both at K=16K=16), with their temperature settings and controls. In c3 the start’s text is 0.04 to 0.07 nats hotter in per-sample entropy on this coarse grid, which leans toward Mix-FT, but at exactly matched entropy the judges still put Mix-FT 0.62 to 0.67 nats/token ahead. The rerun with the first token fixed on both sides uses the same pairs, orders and files with new opaque ids, each side at the grid point the same rule picks on the fixed-convention samples: the temperatures of Table 32 except the start’s in c3 (β=1.1\beta=1.1). The start’s text is 0.02 to 0.04 nats hotter in c1 (5.471 against 5.427 to 5.451), which leans slightly toward Mix-FT, and 0.065 to 0.095 nats colder in c3 (5.322 against 5.387 to 5.417), which leans toward the start. A fixed derangement pairs sample ii of one side with sample π(i)≠i\pi(i)\ne i of the other, so no pair shares a sampling seed; the three seeds’ pairs that share a start sample form one cluster. Each pair is judged in both display orders by separate rater sessions, and the stored order is a fair coin per pair. Controls pair a model’s matched-temperature text with the same model’s hottest grid point, β=0.95\beta=0.95. A verdict scores 1 if the test side (the fine-tuned model in c1 and c3, the start at K=8K=8 in c0) is preferred, 12\frac{1}{2} for a tie and 0 otherwise; a pair’s score is the mean of its two orders, and WW is the mean over pairs. Intervals are 95% percentile bootstraps with 10,000 draws: over clusters for the pooled WW (ratio estimator), over pairs within a seed, and over control pairs for the control rate WcW_c. A rater counts only if the lower bound of WcW_c exceeds 0.60. On c1 the reading is “noninferior” if the lower bound exceeds 0.45, “inconclusive” if it lies in (0.40, 0.45] with the upper bound at least 0.45, and “worse” otherwise; on c3 it is “preferred” if the lower bound exceeds 0.50. Each rule was fixed before any text of its contrast was rated. The share of the halving gap recovered is

Table 32. The three rater contrasts. Each pair shows one complete sample of 1,024 tokens from each side, and every pair is judged in both display orders. For each model the temperature is the grid point whose per-sample entropy is closest to that of real text, 5.397 nats. A control pairs a model’s text at that temperature with the same model’s text at its hottest grid point, β=0.95\beta= 0.95. In c3 the controls use the start, because at K=16K = 16 the 8× model’s hottest text was too close in quality to its matched text to test a rater.

contrasttest sidereference sidetemperaturesper-sample entropy (nats)main pairscontrol pairsjudgments
c18× model at K=8K = 8start at K=16K = 16each model’s own per-step schedule (the same schedule for all four models), β=1.15\beta= 1.15test 5.431 to 5.444, reference 5.434512, drawn at random from 1,536 (171, 171 and 170 per seed)192 on the 8× model at K=8K = 8 (64 per seed)1,408
c0start at K=8K = 8start at K=16K = 16the same schedule for both, β=1.15\beta= 1.15test 5.403, reference 5.434256, drawn at random from 51264 on the start at K=8K = 8640
c38× model at K=16K = 16start at K=16K = 16one global temperature each: β=1\beta= 1 for the 8× model (every seed), β=1.05\beta= 1.05 for the starttest 5.385 to 5.414, reference 5.457192 (64 per seed)96 on the start at K=16K = 16576

caption:

Wc1−Wc00.5−Wc0,\frac{W_{c1}-W_{c0}}{0.5-W_{c0}},

with an interval from the two bootstrap standard errors (normal approximation, c0 and c1 treated as independent). Table 33 lists the runs. The ChatGPT runs gave each file of 32 to 44 judgments to a fresh agent with no shared memory; the Claude

Table 31. Per-seed comparisons at 1.1B. Upper part: each arm’s content-token judge NLL minus Mix-FT’s at the same number of steps per block (nats/token, matched per-sample entropy, 256 samples per temperature; positive: Mix-FT better). The start is one model, compared with each Mix-FT seed; every other arm is paired by seed. Lower part: the IWAE-32 bound of the four-step sampler, each arm minus Mix-FT 1× (nats/token; positive: Mix-FT 1× better).

armKKGPT-2-large seed 0seed 1seed 2meanPythia-1.4B seed 0seed 1seed 2mean
Mix-FT 1× (M=8M = 8) against
Start (SMDM base)2−0.008−0.158−0.153−0.106+0.181−0.011+0.042+0.071
4+0.883+0.516+0.660+0.686+0.957+0.625+0.766+0.782
8+1.039+0.938+0.902+0.960+1.095+0.999+0.979+1.025
16+0.693+0.540+0.623+0.619+0.780+0.617+0.705+0.701
Continued CE2+0.304+0.246+0.239+0.263+0.249+0.167+0.181+0.199
4+0.684+0.518+0.712+0.638+0.677+0.526+0.715+0.639
8+0.747+0.775+0.811+0.778+0.739+0.746+0.774+0.753
16+0.603+0.405+0.601+0.536+0.565+0.403+0.579+0.516
Alpha (κ=0.5\kappa= 0.5)2+0.340+0.076+0.025+0.147+0.395+0.103+0.120+0.206
4+0.734+0.250+0.460+0.481+0.810+0.353+0.511+0.558
8+0.792+0.553+0.745+0.697+0.827+0.622+0.722+0.724
16+0.862+0.512+0.474+0.616+0.883+0.556+0.540+0.660
Mix-FT 8× (M=256M = 256) against
Start (SMDM base)4+1.131+1.058+0.901+1.030+1.184+1.121+0.979+1.095
8+1.143+1.080+1.016+1.080+1.173+1.172+1.120+1.155
Continued CE4+0.932+1.059+0.953+0.981+0.904+1.022+0.927+0.951
8+0.850+0.917+0.925+0.898+0.817+0.919+0.915+0.883
Mix-FT 1× (M=8M = 8)4+0.248+0.542+0.241+0.344+0.227+0.496+0.213+0.312
8+0.104+0.142+0.114+0.120+0.078+0.173+0.141+0.130

caption:

sampler bound at K=4K = 4, arm minus Mix-FT 1×
armseed 0seed 1seed 2mean
Start (SMDM base)+0.2882+0.2903+0.2868+0.2884
Continued CE+0.0075+0.0089+0.0038+0.0068
Alpha (κ=0.5\kappa= 0.5)+0.0086+0.0104+0.0082+0.0091
Mix-FT 8× (M=256M = 256)+0.0269+0.0379+0.0368+0.0339

Table 31 (PDF p. 33).

Sonnet runs gave each batch of judgments (44 in c1, 40 in c0, 36 per file in c3) to a separate helper; order-1 and order-2 batches were disjoint. The rater instructions below head every ChatGPT file and every file of the Claude runs we use (here for a 44-judgment file); in c1 and c0 the Claude helpers rated the very files ChatGPT rated. An earlier Claude run of c1 and c0 worked from the pilot’s version of this prompt, which we did not archive word for word, and also asked for a short reason. We decided before the re-run that its numbers would replace that run’s in the text whatever they showed, and we list the earlier run in Table 33 for reference.

You are a careful, impartial judge of English writing quality.  This is a blinded rating task for a
research study.

This file holds 44 judgments.  Each judgment shows Text A and Text B.  Both were produced by language
models trained on English web text.  Each excerpt is about the same length (1,024 tokens), may begin or
end in the middle of a sentence, and may contain the marker [document break] where one document ends and
another begins.  Do not reward or penalize an excerpt for its topic, for starting or ending abruptly, or
for containing a document break.  Judge which excerpt is better written, using these four criteria,
weighted equally:
1.  Coherence:  the text stays on topic and makes sense from sentence to sentence and from paragraph to
paragraph.
2.  Readability:  the sentences are grammatical, fluent, and easy to follow.
3.  Repetition:  the text is free of loops, repeated phrases, and degenerate lists (less repetition is
better).
4.  Substance:  the text conveys concrete, specific, and consistent content rather than vague,
contradictory, or empty filler.
The order in which the excerpts are shown is random; do not let it influence you.  If the two excerpts are
of essentially equal quality overall, answer "tie".

Read both texts of every judgment in full and judge each judgment on its own merits.  Do not use code,
scripts or statistics to decide; the verdicts must be your own reading.

Answer with exactly 44 lines of JSON, one per judgment, in order, and nothing else:
{"jid": "...", "coherence": "A|B|tie", "readability": "A|B|tie", "repetition": "A|B|tie", "substance":
"A|B|tie", "overall": "A|B|tie"}

The open-weights Qwen2.5-7B-Instruct run (greedy, fp16) used the same rubric as a single-pair chat prompt that also asked for a reason of one or two sentences.

Pooled over the two raters, the share of the halving gap recovered is 0.88 [0.79, 0.97] with the first token fixed and 0.90 [0.80, 1.00] with it drawn, and the two raters give the same verdict on 76 to 95% of the judgments of each contrast (Table 33). Fixing the first token lowers both raters’ equal-step preference from 88 to 76–79%, as it lowers the judges’ margin: it helps the start more than the fine-tuned model (Section B). For Claude the model, prompt and effort of the two runs are the same, so its change comes from the samples alone. With the archived prompt Claude rates c1 0.04 lower than with the pilot’s prompt; its two runs give the same verdict on 83% of the c1 judgments, little more than either gives with ChatGPT, and on 96% of the c0 judgments, so the c1 pairs are close calls for every rater.

Fine-tuning time. Table 34 gives measured training hours on T4 GPUs, summed over kernels from each kernel’s trainer start line to its budget stop or end of training (model loading and evaluation excluded). The 8×8 \times Mix-FT run took 1.45 times as long as 8×8 \times continued CE, mostly because its chunked backward executes about 1.4 times the counted FE. The block-chunked backward of § 17 executes only the counted FE: on a T4, one 8×8 \times update of eight sequences takes 272 seconds with it instead of 397, which would bring the 8×8 \times run to about 27 hours, the time of 8×8 \times continued CE.

modelrunbudgetT4-hours
110MMix-FT, K=2K = 2, M=8M = 8 (1,152 steps)1×\times3.85, 3.38, 3.76
110Mcontinued CE (5,760 steps)1×\times3.10, 3.58, 3.37
110MSDTT-8 (3,840 steps)1×\times3.35, 3.30, 3.28
110MMix-FT, K=2K = 2, M=256M = 256 (357 steps)8×\times38.9, 37.7, 38.0
110MMix-FT, K=2K = 2, M=512M = 512 (358 steps)16×\times49.7, 48.5, 49.2
110Mcontinued CE (46,080 steps)8×\times26.8, 26.0, 25.4
110MAlpha (46,080 steps)8×\times26.5, 25.5, 26.1
110MSDTT-8 (30,720 steps)8×\times25.2, 25.0, 25.2
1.1BMix-FT, K=2K = 2, M=8M = 8 (600 steps)1×\times6.47, 7.04, 6.93
1.1Bcontinued CE (600 steps)1×\times8.49, 7.94, 7.61
1.1BMix-FT, K=2K = 2, M=256M = 256 (339 steps)8×\times59.6, 56.1, 59.8
8BMix-FT, K=4K = 4, M=8M = 8 (600 steps)1×\times73.3, 74.5, 75.4
8Bcontinued CE (600 steps)1×\times78.6, 80.0, 78.9
8BAlpha (600 steps)1×\times77.7, 83.5, 79.4

Table 34. Measured fine-tuning time (T4 GPU-hours, training only; seeds 0, 1, 2 where run). At 110M every job used one T4 except the 16×\times jobs, which used two. At 1.1B the 1×\times jobs used one T4 and the 8×\times jobs two; at 8B every job used two, with the model split by layers. Jobs on two T4s count wall-clock hours ×\times 2.

Experimental decisions and outcomes

We wrote each expectation in a version-controlled log before the experiment ran, together with its code, arms and compute. The table gives, in order, what we expected, what happened, and where the paper reports the evidence; stage numbers are the log’s. GG is a matched-diversity judge gap (positive: arm better); GcG_c is its content-token version at matched per-sample entropy; Δ\Delta is a likelihood difference, arm minus reference (negative: arm better); H8H_8 is GcG_c of a model at K=8K = 8 against the start at K=16K = 16. Unless noted, three seeds.

In the table, C0 is the model after masked-CE pretraining only, Base n×n\times is C0 followed by nn units of CE at the sampler’s states (stage 1 continued plain masked CE instead), Mix is the mixture objective trained from C0, and the hybrid adds the CE loss on Mix-FT’s forward passes. Q-numbers name the questions within a stage, and a kill condition is the outcome, fixed in advance, that rejects the expectation it names. The coverage guard asks whether, where Mix wins at matched distinct-4, its samples still cover at least as much of real held-out text’s 4-gram occurrences as the baseline’s, within half a percentage point (stage 2b also accepted a win at matched coverage), so that the gain is not bought with lost real 4-grams. The share of a per-path control is the part of Mix-FT’s gain over continued CE that the control recovers, on the K=4K = 4 likelihood in rows 10 and 13A and on the judge gain in row 30.

stagequestion and what we expectedwhat happenedevidence
1Text8 pilot (B=8B = 8, K=2K = 2, 2 seeds, weaker Base): Mix beats Base on IWAE-64 by ≥0.02\ge0.02 bpc without losing >10%> 10\% distinct-4.Pass (0.055, 0.063 bpc). Later stages used a Base trained at the sampler’s states.—
2Mix from C0 against C0, Base, SDTT at B∈{8,16}B \in\{8, 16\}: G>0G > 0 in all seeds and mean G≥0.05G \ge0.05; every baseline must learn.Invalid: SDTT did not learn. Without SDTT the pass condition held.—
2bFixed SDTT; kill “Mix is under-trained” if the Mix minus Base bound gap at 4×4\times is not smaller than at 1×1\times; coverage guard.Pass; explanation rejected: on the K=4K = 4 IWAE-64 bound, averaged over blocks 8 and 16, the gap is +0.020 bpc at 1×1\times and +0.076 at 4×4\times (exact values at block 8 in Table 21); guard holds.Table 21
2cTwo-stage recipe (B=8B = 8, K=4K = 4, exact): some arm with Δ≤+0.005\Delta\le+0.005 and G>0G > 0 against Base 5×5\times in all seeds.Pass for Mix-FT and the hybrid.Table 21
3Q1 Mix-FT vs SDTT (exact, B=8B = 8); Q2a B=16B = 16; Q2b K=8K = 8; pass if Δ<0\Delta< 0 (Q2: ≤+0.005\le+0.005) and G>0G > 0 in all seeds.Pass, pass, pass; coverage guard: no caveat.Table 21
3, Q334M FineWeb-Edu from Base 1×1\times (one seed, reported only).No verdict (Δ=+0.022\Delta= +0.022 for both arms).Figure 4
434M from Base 2×2\times: Mix-FT (M=4M = 4) and hybrid vs Base 3×3\times, Δ≤+0.005\Delta\le+0.005 nats/token and G>0G > 0 in all seeds.Pass for both.Figure 4
5110M BD3-LM: Mix-FT and hybrid vs continued CE and SDTT; ΔK=4≤+0.005\Delta_{K=4} \le+0.005 and all-token G>0G > 0 (mean over K=2,4,8K = 2, 4, 8) in all seeds.Mix-FT passes against both; hybrid passes against CE. The all-token GG against SDTT depends on newline targets, so only the likelihood part of that comparison is claimed.Table 10
6, Q6BMix-FT on two-step plans: GK=2>0G_{K=2} > 0 against SDTT (all tokens) and ΔK=2≤+0.005\Delta_{K=2} \le+0.005 against CE in all seeds; kill if mean GK=2≤0G_{K=2} \le0.Kill (mean GK=2=−0.093G_{K=2} = -0.093), although ΔK=2=−0.041\Delta_{K=2} = -0.041 in every seed.Table 1
7, Q7aFour-step Mix-FT vs SDTT and CE at K=8,16K = 8, 16 (512 samples): Gc>0G_c > 0 under both judgments in every seed.Mix-FT better in all four comparisons.Section A
7, Q7bMix-FT on top of SDTT vs more SDTT: ΔK=4≤−0.005\Delta_{K=4} \le-0.005 and Gc>0G_c > 0 at K=4,8K = 4, 8 in every seed.No verdict (Δ=−0.063\Delta= -0.063 every seed; one seed slightly negative at K=4K = 4).Section A
8Alpha diffusion, κ∈{0.5,2,8}\kappa\in\{0.5, 2, 8\}: likelihood, and samples per KK against the best κ\kappa.Four-step Mix-FT better on likelihood; samples mixed at K=2K = 2, Alpha better at K=4K = 4, Mix-FT better at K=8,16K = 8, 16.Table 10
9c1.1B GSM8K, fine-tuning on the model’s verified solutions: Mix-FT minus CE accuracy >0> 0 in every seed with a bootstrap interval excluding 0.No difference (Section D).Section D
10Text8 mechanism control: weighting is “the main cause” if Mix-FT beats per-path FT in every seed and per-path FT recovers at most half of Mix-FT’s likelihood gain over continued CE.Main cause (per-path FT recovers a share of 0.24 on average); samples: Mix-FT better.Table 20

Table 21 (PDF p. 37).

stagequestion and what we expectedwhat happenedevidence
11110M matched budgets (512 samples): four-step Mix-FT vs SDTT and Alpha at K=4K = 4, vs SDTT-8 at $K = 8, 16. | Mixed at K=4K = 4 against both; Mix-FT better than SDTT-8 at K=8,16K = 8, 16 and on likelihood. | Section A
12, Q12aTwo-step Mix-FT vs Alpha, SDTT-4 (K=4K = 4), SDTT-8 (K=8,16K = 8, 16) and four-step Mix-FT, at K=4,8,16K = 4, 8, 16: Gc>0G_c > 0 under both content judges in every seed.Two-step Mix-FT better in all nine readings.Table 1
12, Q12bSeed 0: d=Gcd = G_c(two-step, M=8M = 8) minus GcG_c(four-step, M=3M = 3, 91% of the updates) at K=4K = 4; “two steps matter” if d>0.05d > 0.05 under both judges.Two steps matter (d=+0.48d = +0.48 under both).tbl:7a
12ESeed 0, equal updates: two-step at 443 steps vs four-step Mix-FT, and two-step vs four-step at 1,152 steps; d>0.05d > 0.05 under both judges at K=4K = 4.Two steps matter in both (+0.25+0.25, +0.25+0.25).tbl:7a
13A110M per-path control: “main cause” if Mix-FT beats per-path FT in every seed and per-path FT recovers at most half of Mix-FT’s gain over CE; samples at K=4,8K = 4, 8.Main cause (share −0.14-0.14); samples: Mix-FT better at both.tbl:7c
13B, CIWAE-64 at K=16K = 16, Mix-FT minus CE per seed; 30 bound claims re-read at the validation temperature.Mix-FT better than CE; no claim changes.Section A
13E, 31Text8 width test, rule on three-seed means: the one-token-per-step part of the gain shrinks with width (by at least 0.008 bpc), the gap part does not.Supported. Against continued CE, the rule’s comparator, the first part falls from 0.024 to 0.013 bpc from width 256 to 768 (by 0.010 to 0.011 in each seed) and the second moves from 0.018 to 0.019; against CE trained at the K=8K = 8 states the split is the same.tbl:22a
141.1B SMDM base on OpenWebText: two-step Mix-FT vs continued CE, ΔK=4≤+0.005\Delta K=4 \leq+0.005 and Gc>0G_c > 0 under both judges in every seed.Pass (Δ\Delta from −0.004-0.004 to −0.009-0.009; GcG_c from +0.48+0.48 to +0.59+0.59).Table 30
15B110M, two-step Mix-FT with M=32M = 32 against M=8M = 8 at equal compute: Gc>0G_c > 0 under both judges at K=4,8,16K = 4, 8, 16 in every seed.M=32M = 32 better in all nine (+0.18+0.18, +0.24+0.24, +0.16+0.16).tbl:7b
16, 16bSeed 0: more plans against M=32M = 32 at K=8K = 8, at equal compute (M=64,128M = 64, 128) and with compute scaled to keep about 350 updates (M=64M = 64 at 2×2\times, M=128M = 128 at 4×4\times).Equal compute saturates (+0.03+0.03, no gain); with compute more plans help (about +0.06+0.06 per doubling).tbl:7b
17, 19Per-step temperature schedules, chosen on a development split for every model and the start: K=8K = 8 matches the start at K=16K = 16 (H8≥0H_8 \geq0 under both judges in every seed).No match (H8=−0.135H_8 = -0.135, −0.119-0.119); the schedule helps every model, the start included, by +0.23+0.23 to +0.28+0.28 on fresh samples (eight-temperature grid).tbl:7d
18Importance proposal over plans instead of uniform plans.Worse (−0.39-0.39); dropped.–
20M=256M = 256 at 8×8\times compute: H8≥0H_8 \geq0 under both judges in every seed (one global temperature).Match in all three (+0.002+0.002 to +0.053+0.053; Pythia +0.025+0.025 to +0.076+0.076), with the first token drawn (row 36).Table 27
21The same with each model’s own schedule.Match in all three (+0.016+0.016 to +0.025+0.025; Pythia +0.032+0.032 to +0.045+0.045), with the first token drawn (row 36).Table 27
231.1B, the 1×1\times two-step fine-tunes of stage 14 sampled at K=16K = 16: Mix-FT at K=8K = 8 against the start at K=16K = 16, H8≥0H_8 \geq0 under both judges in every seed.No match in any seed (−0.15-0.15 to +0.07+0.07). At equal steps (K=16K = 16) Mix-FT beats the start by 0.54 to 0.78 and continued CE by 0.40 to 0.60, in every seed under both judges.Table 30
22Seed 0: continued CE and Alpha diffusion at 8×8\times compute; compute alone closes the gap if their H8≥0H_8 \geq0 under both judges.Does not close it: H8H_8 −1.24-1.24 (CE), −1.04-1.04 (Alpha); Mix-FT beats both at K=4,8,16K = 4, 8, 16. Seeds 1 and 2 (reported): H8H_8 −1.21-1.21, −1.22-1.22 for CE and −0.93-0.93, −0.90-0.90 for Alpha; Mix-FT beats both in the same seed by 0.78 to 1.25 at each KK.Table 28

Table 1 (PDF p. 38).

modelopt. stepsK=4K = 4K=2K = 2NELBO PPLK=2K = 2 (55 s)K=4K = 4 (93 s)K=8K = 8 (162 s)K=16K = 16 (305 s)
Start (BD3-LM)–3.2233.77523.007.316.104.603.71
first token fixed†7.466.174.543.63
Continued CE5,7603.2093.76422.41–6.204.823.89
SDTT-42×1,9202 \times1,9203.2453.79624.416.685.834.754.02
SDTT-83,8403.2283.77523.42––4.583.80
Alpha (κ=0.5\kappa= 0.5)5,7603.2023.75823.167.165.794.543.78
Mix-FT (ours)1,1523.1813.72324.517.075.544.103.37
first token fixed†–5.584.103.39
Real held-out text2.78

Table 1. Public BD3-LM OpenWebText model (110M, block 16), all arms from the same weights at the same extra compute (about 0.02% of pretraining). Left: IWAE-32, the bound of Equation (2) with 32 evaluation plans, for the KK-step sampler (nats/token, best of three temperatures, 256 held-out sequences), and NELBO perplexity, which bounds the one-token-per-step sampler. Right: GPT-2-large NLL of the samples on content tokens at matched per-sample entropy (nats/token, lower is better), with the time per 512 samples of 1,024 tokens on one T4. Means of 3 seeds (start: one model); per seed in Table 6, Pythia-1.4B in Table 5. Bold: best entry. –: not measured. Ablations of Mix-FT in Table 4. †The first token of each sequence fixed to end-of-text, as BD3-LM’s own sampler does; the other rows draw it (Section B). Real text: 256 held-out windows at their own per-sample entropy, the top matched level.

modelK=2K = 2K=4K = 4K=8K = 8K=16K = 16
Start (BD3-LM)7.186.084.593.69
Continued CE–6.144.793.86
SDTT-46.705.824.723.98
SDTT-8––4.553.77
Alpha diffusion (κ=0.5\kappa= 0.5)6.995.754.523.75
Mix-FT (K=2K = 2 plans)6.965.504.073.33
Real held-out text2.65
K=4K = 4 plans, M=8M = 87.165.804.303.45
K=4K = 4 plans, M=3M = 3–5.964.50–
K=4K = 4 plans, M=2M = 2–6.044.57–
K=2K = 2 plans, M=4M = 4–5.634.24–

Table 5. As the right half of Table 1, scored by Pythia-1.4B on content tokens (nats/token, lower is better).

stagequestion and what we expectedwhat happenedevidence
25Seed 0: SDTT-8 (teacher passes counted, development-selected checkpoint) and Alpha diffusion on two-step states, at 8×8\times; same readings as 22.Neither closes it (H8−0.86H_8 -0.86 SDTT-8, −0.93-0.93 Alpha-2); Mix-FT beats both at K=4,8,16K = 4, 8, 16. Seeds 1 and 2 (reported): H8−0.88H_8 -0.88, −0.90-0.90 for SDTT-8 and −0.91-0.91, −0.72-0.72 for Alpha-2; Mix-FT beats both in the same seed by 0.68 to 0.94 at each KK.Table 28
271.1B, M=256M = 256 at 8×8\times: H8≥0H_8 \ge0 under both judges in every seed; seed-mean H8H_8 within the −0.03-0.03 margin under both judges (Section J).Not met in every seed: seeds 0 and 1 match, seed 2 is −0.04-0.04 under GPT-2-large and +0.10+0.10 under Pythia; seed means +0.03+0.03 and +0.13+0.13. Margin shown under Pythia, not under GPT-2-large (interval [−0.11,+0.15][-0.11, +0.15]). MAUVE higher in every seed.Table 30
29Bounds of the 8×8\times model with up to 256 plans at K=8,16K = 8, 16 against continued CE (1×1\times, 8×8\times) and the start: sign per seed and whether the gap shrinks from 64 to 256 plans.Above continued CE (by 0.03 at K=16K = 16; within 0.004 at K=8K = 8), below the start at K=8K = 8; the gap shrinks with more plans in every seed.Table 11
28110M, LLaDA’s low-confidence planner, matched entropy: the $8\times$ model at K=8K = 8 against the start at K=16K = 16 (≥0\ge0 under both judges in every seed), and Mix-FT against every baseline at K=4,8,16.∣Match(K = 4, 8, 16. | Match (+0.15$ to +0.19+0.19; Pythia +0.20+0.20 to +0.22+0.22) with the first token drawn, and +0.02+0.02 to +0.04+0.04 (Pythia +0.04+0.04 to +0.07+0.07) when both fix it (row 38); Mix-FT better in all 24 readings. | Table 12
30Seed 0 controls: (a) Mix-FT on eight-step plans, placed against the two- and four-step recipes (±0.05\pm0.05); (b) two-step per-path control, “main cause” if it recovers under half of Mix-FT’s gain over CE at K=4,8,16K = 4, 8, 16; (c) continued CE at 0.1×0.1\times the learning rate and with EMA weights: does Mix-FT still win widely?(a) Behind two-step plans at every KK; level with four-step plans at K=8,16K = 8, 16, behind at K=2,4K = 2, 4. (b) Main cause: per-path FT ends behind CE (share −0.41-0.41 to −0.07-0.07). (c) Mix-FT still wins by 0.51 to 0.81; both variants stay behind the start.Table 7
33Blocks 32 and 64, before training: is the plan-mixture bound still usable (median ESS of eight plans ≥1.25\ge1.25 and two further gates)? Block 32 then fine-tuned anyway as an exploratory run: Mix-FT at K=4K = 4, 1×1\times, better than continued CE if Gc>0G_c > 0 under both judges in every seed; at 8×8\times (seed 0), H≥0H \ge0 under both judges at K=16K = 16 against the start at K=32K = 32, and Mix-FT better than continued CE at 8×8\times.No-go at both blocks (at block 32 the few-step candidates are K=2,4K = 2, 4 have ESS 1.06, 1.13). Fine-tune: Mix-FT better at K=4,8K = 4, 8 and 16 (+0.11+0.11 to +0.48+0.48), and in seed 0 better than the start at each KK under both conventions (+0.14+0.14 to +0.30+0.30), where continued CE is behind it at K=8K = 8 and 16. At 8×8\times: half-step match at K=16K = 16 with the first token drawn (+0.02+0.02, +0.05+0.05), not when both fix it, the start’s convention (−0.10-0.10, −0.09-0.09), and not at K=4K = 4 or 8; better than continued CE 8×8\times at K=4,8K = 4, 8 (+0.55+0.55 to +0.91+0.91).Table 13
35LLaDA-8B-Base, block 32, LoRA, 0.0018% of pretraining per arm, three seeds, LLaDA’s two planners (Section C). (R1) Mix-FT better than continued CE and Alpha diffusion at K=4,8,16K = 4, 8, 16 under GPT-2-large, Pythia-1.4B and Qwen2.5-7B in every seed. (R2, R2-32) Per planner, Mix-FT at K=8K = 8 against the start at K=16K = 16, and at K=16K = 16 against the start at K=32K = 32: match (≥0\ge0 under the three judges in every seed), level (seed mean ≥−0.03\ge-0.03 and 95% upper bound ≥0\ge0 under each) or not reached.R1: pass at every planner, KK, judge and seed (+0.12+0.12 to +0.62+0.62). R2: not reached under either planner (−0.06-0.06 to −0.35-0.35 per seed, 66 to 91% of the start’s own gap closed); continued CE and Alpha diffusion −0.48-0.48 to −0.91-0.91 under GPT-2-large and Pythia. R2-32: match under both planners (+0.05+0.05 to +0.19+0.19 per seed).Table 17

Table train-the-sampler-you-run. Train the Sampler You Run

stagequestion and what we expectedwhat happenedevidence
36The first token: BD3-LM never masks position 0 in training and its sampler fixes it to end-of-text, while ours draws it. Every model of the main comparison sampled both ways with the same random draws; the rule fires if fixing it helps the start by more than 0.05 under both judges at some KK, and then fixes the consequences: one convention in every table, each claim against the start stated under the convention that favors the start with both numbers, the rater studies rerun. Also a null fine-tune: BD3-LM’s own objective continued for the same budget on its raw windows (three seeds), expected level with or ahead of the start, with Mix-FT ahead of it, continued CE behind it, the window convention immaterial and our comparison unchanged on raw windows.Fires: fixing the first token helps the start by 0.05 to 0.11 at K=8K = 8 and 16 (it costs 0.04 to 0.07 at K=4K = 4) and moves the fine-tuned models by at most 0.07. H8H_8 changes sign in every seed under both judges, with one temperature and with schedules (−0.01-0.01 to −0.07-0.07); at equal steps the 8× model still beats the start by 0.53 to 0.58. In the rater rerun the 8× model at K=8K = 8 recovers 88% of the halving loss (90% before), and at equal steps the raters prefer it in 76 and 79% of pairs (88% before). The null is behind the start by 0.18 to 0.43 at every KK under both conventions (its EMA weights too), so not level or ahead as expected; Mix-FT is ahead of it by 0.50 to 0.96; continued CE against it is mixed. The window convention is not immaterial, though by the rule it is convention-dependent: on our windows the null is better in every seed, by 0.11 to 0.34 with the first token drawn, and with it fixed by 0.06 to 0.38 at K=4K = 4 and 8 but level in one seed at K=16K = 16. On our windows it is level with the start on average at K=4K = 4 (seed 2 behind by 0.06 to 0.10) and behind it at K=8K = 8 and 16 in every seed, by 0.04 to 0.28. On raw windows Mix-FT’s gains move by more than 0.10, so “unchanged” fails: over continued CE they grow by 0.12 to 0.18 at K=8K = 8, and over the start they shrink by 0.10 at K=4K = 4 under Pythia with the first token drawn.Section B, Table 7g, Table 33
38Row 36’s consequence for the claims against the start that row 36 did not cover, with the first token fixed for both models. (L1) Under the confidence planner, the 8× model at K=8K = 8 beats the start at K=16K = 16 if its gain is positive under both judges in every seed; (L2) the 1× model beats the start at each KK under that planner (seed 0); (D1) the 8× model at K=8K = 8 is more diverse than the start at K=16K = 16 if its Vendi score is higher and its self-BLEU-4 lower in every seed; at block 32 (seed 0), (B1) the 8× model at K=16K = 16 matches the start at K=32K = 32 if its gain is ≥0\ge0 under both judges, (B2) the 1× model beats the start and (B3) continued CE trails it at each KK.L1 holds under both conventions (+0.02 to +0.07 fixed, +0.15 to +0.22 drawn). L2 beats at K=4K = 4, 8 and 16 (+0.29 to +0.48). D1 holds. B1 does not hold under the start’s convention (−0.10-0.10, −0.09-0.09, +0.02+0.02, +0.05+0.05 drawn). B2 better at every KK (+0.14 to +0.25); B3 behind at K=8K = 8 and 16. With its seeds 1 and 2 added, the 1× model beats the start at K=4K = 4, 8 and 16 in every seed under both judges (Section B).Section B
39The 16× rung, three seeds: the 8× recipe with M=512M = 512 and 16× the budget (358 updates, 4,416,288 FE, 0.25% of pretraining), read under both first-token conventions. (R1) “Match” if H8H_8 is ≥0\ge0 under both judges in every seed; “level on average” if not, but the 95% interval of the seed mean lies above −0.03-0.03 under both judges.Drawn: match (+0.06 to +0.10; Pythia +0.09 to +0.13). Fixed: no match; the seed means are −0.005-0.005 under GPT-2-large and +0.016 under Pythia, and the GPT-2-large interval, [−0.036,+0.028][-0.036, +0.028], misses the margin by 0.006. Each seed gains 0.03 to 0.08 over 8× under either convention.Table 27
40Text8, exact likelihood, run after the 110M results. (T1) Mix-FT on two-step plans against the four-step recipe at equal compute, both evaluated with the K=4K = 4 sampler: “two-step ahead” if its exact K=4K = 4 NLL is lower and its samples better (judge gap GG) in every seed. We were unsure: at B=8B = 8 a two-step plan reveals a four block per step. (M1) The exact K=4K = 4 NLL along M=8M = 8, 32, 128 and 256 plans at fixed updates: expected to keep improving or to flatten, not to get worse at the top. Reported: continued CE given the compute of M=256M = 256 (seed 0).T1: two-step ahead in every seed (0.002 to 0.005 bpc; GG +0.06 to +0.11). At equal updates its likelihood is 0.005 bpc worse in every seed and its samples still better. M1: improves, then flattens (M=256M = 256 against M=128M = 128: mean −0.002-0.002). Continued CE with the compute of M=256M = 256: much lower exact NLL, worse samples.Table 22b,Table 22c

Table ??. Train the Sampler You Run

stagequestion and what we expectedwhat happenedevidence
41Blinded pairwise raters on the 8B half-step result, before any of its samples were drawn: Mix-FT at K=16K = 16 against LLaDA at K=32K = 32 with low-confidence decoding, new samples at each model’s matched temperature, 288 pairs over three seeds, both orders. “Preferred” if the 95% lower bound of Mix-FT’s win rate exceeds 0.50, “noninferior” if it exceeds 0.45; a rater whose control lower bound is at most 0.60 is not read.ChatGPT: preferred, 0.69 [0.63, 0.75], every seed’s lower bound above 0.50; control 0.84 [0.72, 0.95]. Claude Sonnet: preferred, 0.71 [0.66, 0.77], every seed’s lower bound above 0.50; control 0.84 [0.73, 0.95].Table 2

Table 2 (PDF p. 41).

How the two-step model became the main model. Stage 6 trained it to close SDTT’s lead at K=2K = 2, and that rule was killed. A later look at its 128-sample files showed it ahead of Alpha diffusion and the four-step model at K=2K = 2, 4 and 8. Stage 12 then tested it on the 512-sample protocol with the expectation above written down first. The 512-sample files at K=2K = 2, 4 and 8 and temperatures 1 and 1.1 reuse the sampling seeds of the 128-sample files, so their first 128 samples are the ones already seen. All 36 such files, the two-step model’s at the three step counts and the K=2K = 2 files of the four-step Mix-FT, SDTT-4 and Alpha diffusion, each at both temperatures and in three seeds, reproduce their 128-sample files exactly. The other four temperatures and all of K=16K = 16 are new.

References

  1. [1]Amin, H., Yin, M., and Khanna, R. Simple diffusion language models are more effective few-step generators than reported. arXiv:2609.33947, 2026.arxiv.org/abs/2609.33947
  2. [2]Arriola, M., Gokaslan, A., Chiu, J. T., Yang, Z., Qi, Z., Han, J., Sahoo, S. S., and Kuleshov, V. Block diffusion: Interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations, 2025.
  3. [3]Ben-Hamu, H., Gat, I., Severo, D., Nolte, N., and Karrer, B. Accelerated sampling from masked diffusion models via entropy bounded unmasking. arXiv:2505.24857, 2025.
  4. [4]Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, 2023.arxiv.org/abs/2304.01373
  5. [5]Burda, Y., Grosse, R., and Salakhutdinov, R. Importance weighted autoencoders. In International Conference on Learning Representations, 2016.arxiv.org/abs/1509.00519
  6. [6]Chen, S., Cong, K., and Li, J. Optimal inference schedules for masked diffusion models. arXiv:2511.04647, 2025.arxiv.org/abs/2511.04647
  7. [7]Deschenaux, J. and Gulcehre, C. Beyond autoregression: Fast LLMs via self-distillation through time. arXiv:2410.21035, 2024.arxiv.org/abs/2410.21035
  8. [8]Fox, D., Bowyer, S., Liu, S., Aitchison, L., Santos-Rodriguez, R., and Yang, M. Learning generation orders for masked discrete diffusion models via variational inference. arXiv:2602.23968, 2026.arxiv.org/abs/2602.23968
  9. [9]Friedman, D. and Dieng, A. B. The vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research, 2023.
  10. [10]Gushchin, N., Baranchuk, D., and Korotin, A. Alpha diffusion language models: Factorization alone is not the problem. arXiv:2609.38066, 2026.arxiv.org/abs/2609.38066
  11. [11]Hayakawa, S., Takida, Y., Imaizumi, M., Wakaki, H., and Mitsufuji, Y. Distillation of discrete diffusion through dimensional correlations. In International Conference on Machine Learning, 2025.arxiv.org/abs/2410.08709
  12. [13]Hoogeboom, E., Gritsenko, A. A., Bastings, J., Poole, B., van den Berg, R., and Salimans, T. Autoregressive diffusion models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2110.02037
  13. [14]Ivanov, A., Kolesov, A., Korotin, A., Oseledets, I., and Goncharov, M. E-MoE: Enhanced mixture-of-experts for non-factorized diffusion language models. arXiv:2609.37533, 2026.arxiv.org/abs/2609.37533
  14. [15]Kim, B. and Sung, M. Tensor-train joint modeling for few-step discrete diffusion. arXiv:2607.03788, 2026.arxiv.org/abs/2607.03788
  15. [17]Lou, A., Meng, C., and Ermon, S. Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, 2024.arxiv.org/abs/2310.16834
  16. [18]Luxembourg, O., Permuter, H. H., and Nachmani, E. Plan for speed: Dilated scheduling for masked diffusion language models. In International Conference on Machine Learning, 2026.arxiv.org/abs/2506.19037
  17. [19]Madeira, M., Shidani, A., Bizeul, A., Turrisi, V., Béthune, L., Devnani, B., Busbridge, D., Ablin, P., and Monteiro, J. On trajectory-aware training for masked diffusion language models. arXiv:2609.37974, 2026.arxiv.org/abs/2609.37974
  18. [20]Mahoney, M. About the test data. http://mattmahoney.net/dc/textdata.html, 2011. Large Text Compression Benchmark; text8 described June 2006; page last updated Sept. 1, 2011.
  19. [21]Nie, S., Zhu, F., Du, C., Pang, T., Liu, Q., Zeng, G., Lin, M., and Li, C. Scaling up masked diffusion models on text. In International Conference on Learning Representations, 2025a.arxiv.org/abs/2410.18514
  20. [22]Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. In Advances in Neural Information Processing Systems, 2025b.arxiv.org/abs/2502.09992
  21. [23]Penedo, G., Kydlíček, H., Ben Allal, L., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., and Wolf, T. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024.
  22. [24]Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2102.01454
  23. [25]Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
  24. [26]Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.07524
  25. [27]Sahoo, S. S., Yang, Z., Akhauri, Y., Liu, J., Singh, D., Cheng, Z., Liu, Z., Xing, E. P., Thickstun, J., and Vahdat, A. Esoteric language models: A family of any-order diffusion LLMs. In International Conference on Machine Learning, 2026.arxiv.org/abs/2506.01928
  26. [28]Sanokowski, S., Berghammer, W., Ennemoser, M., Wang, H. P., Hochreiter, S., and Lehner, S. Scalable discrete diffusion samplers: Combinatorial optimization and statistical physics. In International Conference on Learning Representations, 2025.arxiv.org/abs/2502.08696
  27. [29]Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.04329
  28. [30]Shih, A., Sadigh, D., and Ermon, S. Training and inference on any-order autoregressive models the right way. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2205.13554
  29. [31]Turok, G., De Sa, C., and Kuleshov, V. DUEL: Exact likelihood for masked diffusion via deterministic unmasking. arXiv:2603.01367, 2026.
  30. [32]Wang, Z., Shi, J., Hoess, N., Gretton, A., and Titsias, M. K. Learning-order autoregressive models with application to molecular graph generation. In International Conference on Machine Learning, 2025.arxiv.org/abs/2503.05979
  31. [35]Xie, T., Xue, S., Feng, Z., Hu, T., Sun, J., Li, Z., and Zhang, C. Variational autoencoding discrete diffusion with enhanced dimensional correlations modeling. In International Conference on Learning Representations, 2026.arxiv.org/abs/2505.17384
  32. [36]Zhang, T., Zhang, X., Han, L., Shi, H., He, X., Li, Z., Wang, H., Xu, K., Srivastava, A., Mao, C., Wang, H., Pavlovic, V., and Metaxas, D. N. Few-step diffusion language models via trajectory self-distillation. arXiv:2602.12262, 2026.arxiv.org/abs/2602.12262

Paper details

vlnc:W_5vwvf7yw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents