betaFeedback

WorksComputer Science

Asynchronous Flow Map Language Models

Abstract

The order in which tokens are predicted in language modeling has a profound impact on the quality of generated sequences. Autoregressive and masked diffusion models iteratively fill in tokens and can exploit such orderings across generation steps, but how to leverage an ordering for one-step parallel language modeling is unclear. We introduce asynchronous flow map language models (aFMLMs), which incorporate prediction order into one-step modeling by learning the flow maps of asynchronous flows. These flows specify an ordering through a tokenwise time schedule that controls the relative denoising progress of tokens. We formulate a general class of asynchronous flows that admit both prescribed and learned, state-adaptive schedules, show that they define valid flow maps, and develop new methods for their flow map distillation. The resulting models can generate all tokens in a single forward pass by internalizing the ordering encoded in the flow into their hidden computations. On Sudoku solving and code generation for GSM8K math word problems, aFMLM establishes new state-of-the-art one-step performance.

Introduction

How tokens are ordered and scheduled for prediction has a significant impact on language modeling [49, 36, 26, 34], as it determines how sequence prediction is decomposed into a series of simpler modeling subproblems. For instance, autoregressive models (ARMs) [8] encode an ordering as a prescribed left-to-right factorization, whereas masked diffusion models (MDMs) [44, 50] learn over diverse masking patterns and choose orderings for generation based on model outputs. These choices would not matter for perfect models, as any factorization recovers the true joint distribution. In practice, however, models have finite capacity and some subproblems are harder to learn than others, making generation quality sensitive to the chosen order and schedule [20].

Because these orderings and schedules are defined over generation steps, in which tokens decided earlier inform later ones, iterative models such as ARMs and MDMs can naturally use them. In contrast, parallel language models designed for fast generation take only a few steps and must perform more of their computation in hidden space, where it is unclear how to leverage a given ordering or schedule. This problem is most severe in one-step generation [29], which predicts all tokens in a single forward pass and thus leaves no steps over which an order can be followed.

We address this problem with asynchronous flow map language models (aFMLMs) that incorporate prediction order into one-step generation. Our idea builds on flow map language models (FMLMs) [29], which achieve one-step generation by compressing the sampling trajectories of continuous flow language models (FLMs) [29, 22, 13]. While existing FMLMs use synchronous flows in which all tokens progress together, aFMLMs learn the flow maps of asynchronous flows, in which tokenwise times control the relative progress of tokens to specify an ordering or schedule. Building on prior work [40, 5], we formulate a general class of asynchronous flows that admits both prescribed and learned, state-adaptive schedules. By treating tokenwise times as part of the state, we show that these flows define valid flow maps, and develop methods to distill them. The resulting models generate all tokens in a single forward pass, internalizing the ordering encoded in the flow into their hidden computation.

An illustration of our approach is shown in Figure 1. To summarize, our main contributions are:

Figure 1. Overview. (Left) We formulate asynchronous flows that denoise tokens at different rates. To rhyme with “mitten,” “kitten” emerges early and guides the preceding words. (Right) We distill these flows into an asynchronous flow map that internalizes this ordering and generates the full sequence in one forward pass.

  • Asynchronous flows. We formulate the generation of asynchronous flow language models (aaFMLs) as a coupled flow of tokens and their tokenwise times governed by a scheduler, covering prescribed and learned, state-adaptive schedules ported from MDM samplers (§ 4.1).

  • Asynchronous flow maps. We show that asynchronous flows define a valid flow map and derive distillation objectives that learn aaFMLMs as one-step generators, in which a single model jointly predicts how the tokenwise times and the tokens progress (§ 4.2 and § 4.3).

  • One-step generation. We train aaFMLMs that achieve state-of-the-art one-step performance on Sudoku solving and coding for GSM8K math problems, and analyze their hidden computations, showing that they internalize the effect of ordering and scheduling into one forward pass (§ 5).

Related Work

Prediction order in sequence modeling. Although every factorization of a joint distribution yields the same likelihood, the quality of sequence models depends on the order of prediction, a gap attributed to their computational limits [20]. For ARMs, learning English in the left-to-right order leads to a higher likelihood than in reverse order [49, 36], reversing digits makes arithmetic easier to learn [30], and learning Sudoku solving in the order given by a logical solver outperforms the row-by-row order [48]. In MDMs, adaptive samplers that choose the ordering from model outputs often outperform random ordering [34, 7, 19, 28]. MDMs with adaptive samplers have been distilled into fewer steps [14, 27, 31], but along discrete trajectories that make very few-step generation hard [29].

Asynchronous diffusion and flows. Asynchronous (continuous) diffusion models denoise different components at different noise levels. Most prescribe a schedule ahead of generation, such as left-to-right for text [53, 46, 11], sliding window for video and fluid dynamics [43], offsets between image semantics and texture [35, 40, 4], or algorithmic orders for satisfiability problems [9]. Diffusion forcing [12] uses per-token independent noise levels for training so that schedules can change at inference, such as learned ones that depend on model outputs [23, 47] (Wever et al. 2025). Few-step distillation of asynchronous models, however, has been limited to prescribed schedules [54, 24, 11] or their close variants [5], whereas we consider both prescribed and learned schedules.

Ordering in flow language models. Most continuous FLMs evolve all tokens with a single shared time [29, 13, 22], and their flow maps compress the resulting trajectories [29, 42, 38]. Outside prescribed left-to-right ordering [53, 46, 11], order has been introduced into FLMs by finalizing tokens during generation, either stochastically [39, 25] or once the model is confident [3, 1]. These samplers, however, rely on stochastic updates or restarts with fresh noise, which complicates flow map distillation requiring deterministic trajectories. Rekesh et al. [41] adjust tokenwise step sizes using model outputs, which our framework can express as a learned scheduler and distill. Bansal & Sanghavi [5] select among a set of prescribed schedules using model outputs at the first generation step, allowing the use of flow map distillation methods for prescribed schedules. Our approach goes beyond this workaround and handles fully learned schedules that use all intermediate outputs.

Preliminaries

Flow language models. Continuous flow language models [29, 22, 13] model categorical token sequences by learning a continuous flow from noise to continuous token embeddings. Let VV be a token vocabulary and y=(yi)i=1L∈VL\mathbf{y} = (y^{i})_{i=1}^{L} \in V^{L} a sequence. Following Lee et al. [29], we represent each token by its one-hot encoding, x1=(onehot⁡(yi))i=1L∈RL×∣V∣\mathbf{x}_{1} = (\operatorname{onehot}(y^{i}))_{i=1}^{L} \in\mathbb{R}^{L \times|V|}. Let p1p_{1} denote the distribution of these one-hot representations and p0=N(0,I)p_{0} = \mathcal{N}(0, I) the noise distribution. We define a probability path by interpolating between independent draws x0∼p0\mathbf{x}_{0} \sim p_{0} and x1∼p1\mathbf{x}_{1} \sim p_{1}:

It:=(1−t)x0+tx1,pt:=Law⁡(It),t∈[0,1].(1)I_{t} := (1-t)\mathbf{x}_{0} + t\mathbf{x}_{1}, \qquad p_{t} := \operatorname{Law}(I_{t}), \qquad t \in[0,1]. \tag*{(1)}

The probability path admits a deterministic representation,

x˙t=bt(xt),bt(x)=E[ddtIt∣It=x]=E[x1−x0∣It=x],(2)\dot{\mathbf{x}}_{t} = b_{t}(\mathbf{x}_{t}), \qquad b_{t}(\mathbf{x}) = \mathbb{E}\left[\left.\frac{\mathrm{d}}{\mathrm{d}t}I_{t}\right| I_{t}=\mathbf{x}\right] = \mathbb{E}[\mathbf{x}_{1}-\mathbf{x}_{0}\mid I_{t}=\mathbf{x}], \tag*{(2)}

where btb_{t} is the velocity field of the probability flow. Instead of predicting the velocity itself, it is common to learn the denoiser, which predicts the posterior mean of the clean data:

Dt(x):=E[x1∣It=x],bt(x)=Dt(x)−x1−t.(3)D_{t}(\mathbf{x}) := \mathbb{E}[\mathbf{x}_{1}\mid I_{t}=\mathbf{x}], \qquad b_{t}(\mathbf{x}) = \frac{D_{t}(\mathbf{x})-\mathbf{x}}{1-t}. \tag*{(3)}

Since x1\mathbf{x}_{1} is one-hot, Dti(x)D_{t}^{i}(\mathbf{x}) is a probability distribution over VV at each position ii, and the denoiser can be learned by minimizing a tokenwise cross-entropy loss LCE(D^):=Et,x0,x1[∑i[−log⁡D^ti(It)]yi]\mathcal{L}_{\mathrm{CE}}(\hat{D}) := \mathbb{E}_{t,\mathbf{x}_{0},\mathbf{x}_{1}}\left[\sum_{i}[-\log\hat{D}_{t}^{i}(I_{t})]_{y^{i}}\right] [17, 18, 29].

Flow map language models. The deterministic flow given in (2) defines a unique flow map Xs,t:RL×∣V∣→RL×∣V∣X_{s,t}: \mathbb{R}^{L \times|V|} \to\mathbb{R}^{L \times|V|}, the solution operator of the flow xt=Xs,t(xs)\mathbf{x}_{t}=X_{s,t}(\mathbf{x}_{s}), satisfying

Xs,t(x)=x+∫stbr(Xs,r(x))dr=x+(t−s)vs,t(x),(4)X_{s,t}(\mathbf{x}) = \mathbf{x} + \int_{s}^{t} b_{r}(X_{s,r}(\mathbf{x}))\mathrm{d}r = \mathbf{x} + (t-s)v_{s,t}(\mathbf{x}), \tag*{(4)}

where vs,tv_{s,t} is the average velocity between ss and tt, satisfying vs,s=bsv_{s,s}=b_{s}. By learning the flow map, we can jump between arbitrary times, and evaluating it with s=0s=0 and t=1t=1 gives a one-step map from noise to data. Recent work on learning flow maps over categorical data [42, 29, 38] parameterizes it via the two-time denoiser δs,t:RL×∣V∣→(Δ∣V∣−1)L\delta_{s,t} : \mathbb{R}^{L \times|V|} \to(\Delta^{|V|-1})^{L},

δs,t(x):=x+(1−s)vs,t(x),(5)\delta_{s,t}(\mathbf{x}) := \mathbf{x} + (1-s)v_{s,t}(\mathbf{x}), \tag*{(5)}

which extrapolates the flow map displacement s→ts \to t to s→1s \to1 and satisfies δs,s(x)=Ds(x)\delta_{s,s}(\mathbf{x}) = D_{s}(\mathbf{x}). Since it is simplex-valued, δs,t\delta_{s,t} can be learned by self-distillation with KL divergence [29],

LKL(δ^)=Et,x0,x1[KL(Dt(It) ∥ δ^t,t(It))]+Es,u,t,x0,x1[KL(δˉs,t ∥ δ^t,t(Is))],(6)\mathcal{L}_{\mathrm{KL}}(\hat{\delta}) = \mathbb{E}_{t,\mathbf{x}_{0},\mathbf{x}_{1}}\left[\mathrm{KL}(D_{t}(I_{t}) \,\|\, \hat{\delta}_{t,t}(I_{t}))\right] + \mathbb{E}_{s,u,t,\mathbf{x}_{0},\mathbf{x}_{1}}\left[\mathrm{KL}(\bar{\delta}_{s,t} \,\|\, \hat{\delta}_{t,t}(I_{s}))\right], \tag*{(6)}

where, for s<u<ts<u<t and with the stop-gradient operator sg⁡(⋅)\operatorname{sg}(\cdot),

δˉs,t=sg⁡(γδ^s,u(Is)+(1−γ)δ^u,t(X^s,u(Is))),γ=(1−t)(u−s)(1−u)(t−s)∈[0,1],(7)\bar{\delta}_{s,t} = \operatorname{sg}\left(\gamma\hat{\delta}_{s,u}(I_{s}) + (1-\gamma)\hat{\delta}_{u,t}(\hat{X}_{s,u}(I_{s}))\right), \qquad\gamma= \frac{(1-t)(u-s)}{(1-u)(t-s)} \in[0,1], \tag*{(7)}

and we denote by X^s,t(x)=1−t1−sx+t−s1−sδ^s,t(x)\hat{X}_{s,t}(\mathbf{x}) = \frac{1-t}{1-s}\mathbf{x} + \frac{t-s}{1-s}\hat{\delta}_{s,t}(\mathbf{x}) the flow map induced by the learned denoiser δ^\hat{\delta}.

Asynchronous Flow Maps

We first formulate asynchronous flows under a scheduler, then distill them into asynchronous flow maps for one-step generation.

Asynchronous flows with schedulers

To allow token positions to denoise at different rates, we assign each position its own time:

τ=(τ1,…,τL)∈[0,1]L,(8)\tau= (\tau^{1}, \ldots, \tau^{L}) \in[0,1]^{L}, \tag*{(8)}

where τi\tau^{i} measures the denoising progress of position ii [12]. Extending the standard interpolant (1), we associate τ=0\tau= 0 with noise and τ=1\tau= 1 with clean data. For independent draws x0∼p0:=p0\mathbf{x}_{0} \sim p_{0} := p_{0} and x1∼p1:=p1\mathbf{x}_{1} \sim p_{1} := p_{1}, we define the asynchronous interpolant [40]

Iτ:=(1−τ)x0+τx1,pτ:=Law⁡(Iτ),(9)I_{\tau} := (1-\tau)\mathbf{x}_{0} + \tau\mathbf{x}_{1}, \qquad p_{\tau} := \operatorname{Law}(I_{\tau}), \tag*{(9)}

where we write products and divisions of tokenwise vectors as (τx)i=τixi(\tau\mathbf{x})^{i} = \tau^{i}x^{i} and (x/τ)i=xi/τi(\mathbf{x}/\tau)^{i} = x^{i}/\tau^{i}.

Schedules. In standard flows (2), generation moves from noise to data as a shared time advances from zero to one, and we only need to choose step sizes to perform generation. With tokenwise times (8), we can think of generation as the analogous transition of τ∈[0,1]L\tau\in[0,1]^{L} from 0\mathbf{0} to 1\mathbf{1}. Yet, unlike in standard flows, this further leaves a choice of how the tokens progress, as some may advance earlier than others, or all may advance together. These choices give many distinct paths of generation:

Proposition 1. Up to continuous nondecreasing reparameterization, there is a unique continuous nondecreasing path from zero to one. In contrast, there are infinitely many such paths from 0\mathbf{0} to 1\mathbf{1} in [0,1]L[0,1]^{L} for L≥2L \ge2.

A proof is in Appendix § 8.§ 1.

To describe the progress of generation along such a path, we introduce a scalar base time t∈[0,1]t \in[0,1] and call the resulting evolution of tokenwise times τt∈[0,1]L\tau_{t} \in[0,1]^{L} from τ0=0\tau_{0} = 0 to τ1=1\tau_{1} = 1 a schedule. We specify this evolution through a scheduler ωt:RL×∣V∣×[0,1]L→R+L\omega_{t} : \mathbb{R}^{L \times|V|} \times[0,1]^{L} \to\mathbb{R}_{+}^{L} that sets the tokenwise rates:

τ˙t=ωt(xt,τt),τ0=0,(10)\dot{\tau}_{t} = \omega_{t}(\mathbf{x}_{t}, \tau_{t}), \qquad\tau_{0} = 0, \tag*{(10)}

where xt\mathbf{x}_{t} denotes the current token states. We choose the scheduler so that τt→1\tau_{t} \to1 as t↑1t \uparrow1.

Velocity of token states. We now derive the velocity of token states under a chosen scheduler by extending (2). For each pair of noise x0\mathbf{x}_{0} and data x1\mathbf{x}_{1}, we advance the tokenwise times by evaluating the scheduler on the interpolant, τ˙t=ωt(Iτt,τt)\dot{\tau}_{t} = \omega_{t}(I_{\tau_{t}}, \tau_{t}). Since ddtIτt=τ˙t(x1−x0)\frac{d}{dt} I_{\tau_{t}} = \dot{\tau}_{t}(\mathbf{x}_{1} - \mathbf{x}_{0}), the conditional mean

btω(x,τ):=E[x1−x0∣Iτt=x, τt=τ](11)b_{t}^{\omega}(\mathbf{x}, \tau) := \mathbb{E}[\mathbf{x}_{1} - \mathbf{x}_{0} \mid I_{\tau_{t}} = \mathbf{x},\ \tau_{t} = \tau] \tag*{(11)}

gives the asynchronous velocity of token states, τ˙tbtω=ωtbtω\dot{\tau}_{t}b_{t}^{\omega} = \omega_{t}b_{t}^{\omega}. Together with the scheduler, it defines a valid probability flow from noise to data, which we refer to as an asynchronous flow:

Proposition 2. Let (xt,τt)(\mathbf{x}_{t}, \tau_{t}) solve, starting from x0∼p0\mathbf{x}_{0} \sim p_{0} and τ0=0\tau_{0} = 0,

x˙t=ωt(xt,τt)btω(xt,τt),τ˙t=ωt(xt,τt).\dot{\mathbf{x}}_{t} = \omega_{t}(\mathbf{x}_{t}, \tau_{t})b_{t}^{\omega}(\mathbf{x}_{t}, \tau_{t}), \qquad\dot{\tau}_{t} = \omega_{t}(\mathbf{x}_{t}, \tau_{t}).

Then (xt,τt)(\mathbf{x}_{t}, \tau_{t}) has the same joint distribution as the interpolant and its tokenwise times advanced by ω\omega for every t∈[0,1)t \in[0,1), and xt\mathbf{x}_{t} converges in distribution to p1p_{1} as t↑1t \uparrow1.

A proof is in Appendix § 8.§ 2.

Mirroring (3), we can parameterize the velocity through the corresponding asynchronous denoiser:

Dtω(x,τ):=E[x1∣Iτt=x, τt=τ],btω(x,τ)=Dtω(x,τ)−x1−τ.(12)D_{t}^{\omega}(\mathbf{x}, \tau) := \mathbb{E}[\mathbf{x}_{1} \mid I_{\tau_{t}} = \mathbf{x},\ \tau_{t} = \tau], \qquad b_{t}^{\omega}(\mathbf{x}, \tau) = \frac{D_{t}^{\omega}(\mathbf{x}, \tau) - \mathbf{x}}{1-\tau}. \tag*{(12)}

Scheduler-independent training. For a chosen scheduler ω\omega, we can learn DtωD_t^\omega by tokenwise cross-entropy, using (t,Iτt,τt)(t, I_{\tau_t}, \tau_t) as inputs and the clean tokens as targets. Yet, constructing these examples requires simulating ω\omega along each interpolant, tying denoiser training to the scheduler. To allow the scheduler to be chosen after training, we may instead learn the scheduler-independent denoiser

Dτ(x):=E[x1∣Iτ=x],D_\tau(\mathbf{x}) := \mathbb{E}[\mathbf{x}_1 \mid I_\tau= \mathbf{x}],

which takes as input (τ,Iτ)(\tau, I_\tau). For this, we use diffusion forcing Chen et al., [12], which samples the tokenwise times independently of one another and of the noise and data:

Proposition 3. Consider the objective

LCE(D^):=Eτ,x0,x1∑i=1L[−log⁡D^τi(Iτ)]yi,τi∼iidU[0,1].(13)\mathcal{L}_{\mathrm{CE}}(\widehat{D}) := \mathbb{E}_{\tau,\mathbf{x}_0,\mathbf{x}_1} \sum_{i=1}^{L} \left[-\log\widehat{D}_{\tau}^{i}(I_\tau) \right]_{y_i}, \qquad\tau^i \stackrel{\mathrm{iid}}{\sim} U[0,1]. \tag*{(13)}

Any population minimizer D^⋆\widehat{D}^{\star} satisfies D^τ⋆=Dτ\widehat{D}_{\tau}^{\star}=D_\tau almost surely under the training distribution.

A proof is in Appendix § 8.§ 3. We leave further discussion on this training scheme in Appendix § 8.§ 4.

Following (3), the denoiser DτD_\tau gives the scheduler-independent velocity

bτ(x):=E[x1−x0∣Iτ=x]=Dτ(x)−x1−τ,(14)b_\tau(\mathbf{x}) := \mathbb{E}[\mathbf{x}_1-\mathbf{x}_0 \mid I_\tau=\mathbf{x}] = \frac{D_\tau(\mathbf{x})-\mathbf{x}}{1-\tau}, \tag*{(14)}

which we use to approximate btωb_t^\omega at inference under a choice of scheduler ω\omega. Denoting zt:=(xt,τt)\mathbf{z}_t := (\mathbf{x}_t,\tau_t), this gives the asynchronous flow

z˙t=(ωt(xt,τt)bτt(xt),ωt(xt,τt)),z0=(x0,0),x0∼p0.(15)\dot{\mathbf{z}}_t = \left(\omega_t(\mathbf{x}_t,\tau_t)b_{\tau_t}(\mathbf{x}_t),\omega_t(\mathbf{x}_t,\tau_t)\right), \qquad\mathbf{z}_0=(\mathbf{x}_0,0), \qquad\mathbf{x}_0 \sim p_0. \tag*{(15)}

These coupled dynamics define a deterministic ODE on the token states and their tokenwise times. We call the resulting model an asynchronous flow language model (aFLM). We note that setting τti=t\tau_t^i=t for all positions ii recovers the usual FLM, which we refer to as synchronous.

Prescribed and state-adaptive schedules. When ω\omega does not depend on the token states, as in left-to-right generation for instance, τt\tau_t follows a fixed path τ(t)\tau(t) determined ahead of generation, which we call a prescribed schedule. The condition τt=τ(t)\tau_t=\tau(t) then holds for every pair of noise and data, so Dtω(x,τ(t))=Dτ(t)(x)D_t^\omega(\mathbf{x},\tau(t))=D_{\tau(t)}(\mathbf{x}). The velocity approximation is therefore exact:

Corollary 4. Let τ(t)\tau(t) be a prescribed schedule. Then the asynchronous flow in (15) reduces to

x˙t=τ˙(t)bτ(t)(xt),x0∼p0,(16)\dot{\mathbf{x}}_t=\dot{\tau}(t)b_{\tau(t)}(\mathbf{x}_t), \qquad\mathbf{x}_0\sim p_0, \tag*{(16)}

and satisfies xt∼pτ(t)\mathbf{x}_t\sim p_{\tau(t)} for every t∈[0,1)t\in[0,1), with pτ(t)→p1p_{\tau(t)}\to p_1 as t↑1t\uparrow1.

A proof is in Appendix § 8.§ 5. Furthermore, only the denoiser along τ(t)\tau(t) is needed, so we can train it with the cross-entropy loss in (13) using τ=τ(t)\tau=\tau(t) instead of independently sampled tokenwise times.

Schedulers that depend on the current token states yield state-adaptive schedules, and we call those computed from denoiser predictions learned schedules. Then DtωD_t^\omega and DτD_\tau generally differ, and exact recovery of p1p_1 is no longer guaranteed. Nevertheless, this allows training the denoiser without fixing a scheduler in advance. Then during inference, a scheduler can be chosen to improve generation by presenting an imperfect denoiser with easier prediction problems Kim et al., [26]. We construct such schedulers as continuous analogues of unmasking criteria used in MDMs, as given in § 4.3.

Asynchronous flow maps

Asynchronous flows exploit prediction order by letting tokens that denoise earlier inform the prediction of others. This motivates our key idea for leveraging such an order for few-step and one-step parallel generation, by distilling the asynchronous flow into a flow map. In this way, the effect of the schedule is internalized within the model’s hidden computation. Since the flow is defined jointly on the tokens and their tokenwise times (15), we define the asynchronous flow map to evolve both:

Proposition 5. On any interval where the asynchronous flow (15) admits unique solutions, its solution operator Fs,t(zs)=ztF_{s,t}(z_s)=z_t, called the asynchronous flow map, satisfies

Fs,t(zs)=zs+∫stz˙r dr,F_{s,t}(z_s)=z_s+\int_s^t \dot{z}_r\,\mathrm{d}r,
Fs,t=Fu,t∘Fs,u,s≤u≤t<1.(17)F_{s,t}=F_{u,t}\circ F_{s,u},\qquad s\le u\le t<1. \tag*{(17)}

A proof is in Appendix A.6. The composition identity, which we call the semigroup condition, states that a direct jump from ss to tt agrees with two successive jumps through uu. We use this consistency to learn the map by self-distillation.

Parameterization. Following § 3, we express the flow map through its average velocity. For a state z=(x,τ)z=(x,\tau) at time ss, we denote the average rates of change of the tokens and their tokenwise times until time tt by vs,tv_{s,t} and ηs,t\eta_{s,t}, respectively, with s=ts=t reducing to the instantaneous dynamics:

(vs,t(z),ηs,t(z)):=Fs,t(z)−zt−s,s<t,\left(v_{s,t}(z),\eta_{s,t}(z)\right):=\frac{F_{s,t}(z)-z}{t-s},\qquad s<t,
(vt,t(z),ηt,t(z)):=(ωt(x,τ)bτ(x),ωt(x,τ)).(18)\left(v_{t,t}(z),\eta_{t,t}(z)\right):=\left(\omega_t(x,\tau)b_\tau(x),\omega_t(x,\tau)\right). \tag*{(18)}

For the token component vs,tv_{s,t}, we further adapt a two-time denoiser parameterization:

δs,t(z):=x+1−τηs,t(z)vs,t(z).(19)\delta_{s,t}(z):=x+\frac{1-\tau}{\eta_{s,t}(z)}v_{s,t}(z). \tag*{(19)}

This extends (5) by rescaling each token’s displacement according to its tokenwise progress. Along the asynchronous flow, δs,t\delta_{s,t} is a weighted average of denoiser predictions and therefore lies in the vocabulary simplex. We can thus learn it with a categorical output and KL loss, as in FMLMs. The following identities recover the flow map and provide the conditions for learning it:

Proposition 6. Let Fs,tF_{s,t} be the asynchronous flow map. Then δs,t\delta_{s,t} and ηs,t\eta_{s,t} satisfy:

(i) The transition Fs,t(z)=(xt,τt)F_{s,t}(z)=(x_t,\tau_t) is recovered as

xt=x+(t−s)ηs,t(z)1−τ(δs,t(z)−x),x_t=x+\frac{(t-s)\eta_{s,t}(z)}{1-\tau}\left(\delta_{s,t}(z)-x\right),
τt=τ+(t−s)ηs,t(z).(20)\tau_t=\tau+(t-s)\eta_{s,t}(z). \tag*{(20)}

(ii) On the diagonal s=ts=t,

δt,t(z)=Dτ(x),ηt,t(z)=ωt(x,τ).(21)\delta_{t,t}(z)=D_\tau(x),\qquad\eta_{t,t}(z)=\omega_t(x,\tau). \tag*{(21)}

(iii) For s<u<ts<u<t, the semigroup condition (17) is equivalent to

δs,t(z)=γδs,u(z)+(1−γ)δu,t(Fs,u(z)),\delta_{s,t}(z)=\gamma\delta_{s,u}(z)+(1-\gamma)\delta_{u,t}(F_{s,u}(z)),
ηs,t(z)=u−st−sηs,u(z)+t−ut−sηu,t(Fs,u(z)),(22)\eta_{s,t}(z)=\frac{u-s}{t-s}\eta_{s,u}(z)+\frac{t-u}{t-s}\eta_{u,t}(F_{s,u}(z)), \tag*{(22)}

where

γ=(1−τt)(τu−τs)(1−τu)(τt−τs)∈[0,1]L,\gamma=\frac{(1-\tau_t)(\tau_u-\tau_s)}{(1-\tau_u)(\tau_t-\tau_s)}\in[0,1]^L,

with τs=τ\tau_s=\tau and τu,τt\tau_u,\tau_t the tokenwise times along the flow from zz.

A proof is in Appendix A.7.

Training. We learn predictors δ^\hat{\delta} and η^\hat{\eta}, which define a model of the flow map F^\hat{F} through (20). On the diagonal, the denoiser DτD_\tau and scheduler ω\omega provide their respective targets (21). For s<u<ts<u<t, we construct targets for a direct transition from ss to tt by composing two shorter student transitions through uu based on (22):

Ts,u,tδ(z):=γδ^s,u(z)+(1−γ)δ^u,t(F^s,u(z)),\mathcal{T}^{\delta}_{s,u,t}(z) := \gamma\hat{\delta}_{s,u}(z) + (1-\gamma)\hat{\delta}_{u,t}(\hat{F}_{s,u}(z)),
Ts,u,tη(z):=u−st−sη^s,u(z)+t−ut−sη^u,t(F^s,u(z)).(23)\mathcal{T}^{\eta}_{s,u,t}(z) := \frac{u-s}{t-s}\hat{\eta}_{s,u}(z) + \frac{t-u}{t-s}\hat{\eta}_{u,t}(\hat{F}_{s,u}(z)). \tag*{(23)}

Here γ\gamma from (22) is evaluated at the tokenwise times of zz, F^s,u(z)\hat{F}_{s,u}(z), and F^u,t(F^s,u(z))\hat{F}_{u,t}(\hat{F}_{s,u}(z)). Combining these composition targets with the diagonal targets gives the training objective:

Proposition 7. For λ>0\lambda> 0, consider the population objective

L(δ^,η^)=Es,u,t,z[KL⁡(sg⁡(Ts,u,tδ(z))∥δ^s,t(z))]+Et,z[KL⁡(Dτ(x)∥δ^t,t(z))]\mathcal{L}(\hat{\delta},\hat{\eta}) = \mathbb{E}_{s,u,t,z}\left[\operatorname{KL}\left(\operatorname{sg}(\mathcal{T}^{\delta}_{s,u,t}(z)) \mathrel{\|} \hat{\delta}_{s,t}(z)\right)\right] + \mathbb{E}_{t,z}\left[\operatorname{KL}\left(D_{\tau}(x) \mathrel{\|} \hat{\delta}_{t,t}(z)\right)\right]
+λEs,u,t,z∥sg⁡(Ts,u,tη(z))−η^s,t(z)∥22+λEt,z∥ωt(x,τ)−η^t,t(z)∥22.(24)+ \lambda\mathbb{E}_{s,u,t,z}\left\|\operatorname{sg}(\mathcal{T}^{\eta}_{s,u,t}(z))-\hat{\eta}_{s,t}(z)\right\|_{2}^{2} + \lambda\mathbb{E}_{t,z}\left\|\omega_{t}(x,\tau)-\hat{\eta}_{t,t}(z)\right\|_{2}^{2}. \tag*{(24)}

The exact δ\delta and η\eta have zero loss. Conversely, any zero-loss solution satisfies the diagonal and composition identities (21) and (22) almost surely under the training distribution.

A proof is in Appendix § 8.8. We call the model learned with (24) an asynchronous flow map language model (aFMLM).

For state-adaptive schedules, we train on joint states zsz_s visited by the asynchronous flow. These states depend on the preceding trajectory and cannot be sampled directly from the interpolant. We therefore first train the denoiser DτD_{\tau} and simulate its flow under the chosen scheduler to obtain the initial training states. This pretrained teacher also provides the diagonal denoising target in (24). To reduce the cost of simulation, we can later use the learned map to get zsz_s, as described in § 4.3.

On the other hand, for a prescribed schedule τ(t)\tau(t), the time component is fixed:

ηs,t=τ(t)−τ(s)t−s,ηt,t=τ˙(t).\eta_{s,t} = \frac{\tau(t)-\tau(s)}{t-s}, \qquad\eta_{t,t} = \dot{\tau}(t).

Only the two-time denoiser needs to be learned. Moreover, the flow follows pτ(t)p_{\tau(t)} by Corollary 4, so a training state at any base time rr can be sampled directly as zr=(Iτ(r),τ(r))z_r = (I_{\tau(r)},\tau(r)). On the diagonal, the denoiser is the conditional mean of the clean data x1x_1. Replacing its target with data therefore changes the expected loss only by a constant independent of the model:

Corollary 8. Let τ(t)\tau(t) be a prescribed schedule, with η^\hat{\eta} fixed to its exact time component and training states sampled from the corresponding interpolants. Then (24) reduces to

L(δ^)=Es,u,t,z[KL⁡(sg⁡(Ts,u,tδ(z))∥δ^s,t(z))]+Et,z,x1[KL⁡(x1∥δ^t,t(z))],(25)\mathcal{L}(\hat{\delta}) = \mathbb{E}_{s,u,t,z}\left[\operatorname{KL}\left(\operatorname{sg}(\mathcal{T}^{\delta}_{s,u,t}(z)) \mathrel{\|} \hat{\delta}_{s,t}(z)\right)\right] + \mathbb{E}_{t,z,x_1}\left[\operatorname{KL}\left(x_1 \mathrel{\|} \hat{\delta}_{t,t}(z)\right)\right], \tag*{(25)}

up to a constant independent of δ^\hat{\delta}. Its minimizers coincide with the zero-loss solutions of Proposition 7 with η^\hat{\eta} fixed by the schedule.

A proof is in Appendix § 8.9. Thus, a prescribed schedule allows the flow map to be trained from scratch without a pretrained teacher.

Inference. We generate samples by applying the learned flow map on a temporal grid of base times 0=t0<⋯<tN=10=t_0<\cdots<t_N=1:

z^t0=(x0,0),x0∼p0,\hat{z}_{t_0}=(x_0,0), \qquad x_0\sim p_0,
z^tn+1=F^tn,tn+1(z^tn),n=0,…,N−1.(26)\hat{z}_{t_{n+1}}=\hat{F}_{t_n,t_{n+1}}(\hat{z}_{t_n}), \qquad n=0,\ldots,N-1. \tag*{(26)}

The final tokens are obtained by applying tokenwise argmax to δ^tN−1,1(z^tN−1)\hat{\delta}_{t_{N-1},1}(\hat{z}_{t_{N-1}}). For N=1N=1, a single evaluation of δ^0,1\hat{\delta}_{0,1} generates the entire sequence without evaluating the scheduler. For supervised tasks, this allows us to train with a prescribed schedule for each example, even when the corresponding schedule is unavailable for a test input. These schedules guide training, and the one-step model internalizes them to generate its output from the task input and initial noise alone.

Algorithmic details

Scheduler implementation. We remark that a scheduler must send τt→1\tau_t \to1 as t↑1t \uparrow1 (10). We implement the schedulers in a common form that ensures this by construction (Appendix § 9.§ 1):

ωti(x,τ)=1−τi1−tkti(x,τ),(27)\omega_t^i(x,\tau) = \frac{1-\tau^i}{1-t} k_t^i(x,\tau), \tag*{(27)}

where the factor kti>0k_t^i > 0 controls how quickly position ii advances and can be chosen almost freely. Setting kti=1k_t^i = 1, for instance, recovers the synchronous schedule τti=t\tau_t^i = t from τ0=0\tau_0 = 0. For learned schedules, we assign higher factors to positions that the denoiser predicts confidently, using two rules:

  • Top-KK confidence scheduler prioritizes the KK most confident positions, with KK increasing over time [10].

  • Entropy-bounded scheduler uses an entropy budget to determine how many high-confidence positions advance faster, allowing more positions to advance together when their predictions are certain [7].

These schedules are constructed as continuous analogues of unmasking strategies used in MDMs, and their details along with more such examples can be found in Appendix § 9.§ 1.

Denoiser and time heads. For flow map distillation under learned schedules, we use one model with a denoiser head for δ^s,t\hat{\delta}_{s,t} and a time head for η^s,t\hat{\eta}_{s,t}. To keep predicted tokenwise times in [τs,1][\tau_s,1], we parameterize the time head via the effective rate κs,t:=log⁡1−τs1−τt/log⁡1−s1−t\kappa_{s,t} := \log\frac{1-\tau_s}{1-\tau_t} / \log\frac{1-s}{1-t}, which is a weighted average of kk along the trajectory (see Appendix § 9.§ 2 for details). The head outputs κ^s,t>0\hat{\kappa}_{s,t} > 0, predicting

τ^t=τs+(t−s)η^s,t(zs)=1−(1−τs)⊙(1−t1−s)κ^s,t(zs).(28)\hat{\tau}_t = \tau_s + (t-s)\hat{\eta}_{s,t}(z_s) = 1 - (1-\tau_s) \odot\left(\frac{1-t}{1-s}\right)^{\hat{\kappa}_{s,t}(z_s)}. \tag*{(28)}

Each transition advances the tokenwise times toward 1 and reaches 1 at t=1t = 1. On the diagonal, matching κ^t,t\hat{\kappa}_{t,t} to ktk_t is equivalent to matching η^t,t\hat{\eta}_{t,t} to ωt\omega_t (see Appendix § 9.§ 2 for details).

We initialize the backbone and denoiser head from the teacher DτD_\tau and attach the time head to the final hidden features. The time head exponentiates its output to ensure positivity, and we initialize its output before exponentiation to zero, setting κ^s,t=1\hat{\kappa}_{s,t} = 1 at initialization.

Time conditioning. With tokenwise time, both the teacher denoiser and the flow map receive a separate time embedding at each position, obtained by applying a shared embedding layer to τi\tau^i. For the flow map, we additionally embed the base times ss and tt with zero-initialized layers and add their outputs to the embeddings of τs\tau_s, following Lee et al. [29]. The resulting conditioning vectors modulate each block through adaptive layer normalization (adaLN) [37].

Rollout curriculum. To train with (24), we initially obtain training states zsz_s by rolling out the teacher under the chosen scheduler from z0=(x0,0)z_0 = (x_0, 0) to time ss. After the first round, we use the current student flow map F^\hat{F} for rollouts and halve the number of rollout steps at successive rounds. The fixed teacher denoiser and scheduler continue to provide the diagonal targets throughout training.

MDM initialization. Diffusion forcing requires the denoiser to denoise across many combinations of tokenwise noise levels, which can make training demanding [26]. When each tokenwise time is close to either 0 or 1, the task resembles masked token prediction, since each position is nearly pure noise or nearly clean. To exploit this overlap, we optionally initialize the denoiser from a trained MDLM [44], following Agarwal et al. [1].

Noise downscaling. Following Chavan et al. [11], we use initial noise x0∼N(0,σ2I)x_0 \sim\mathcal{N}(0,\sigma^2 I) with σ<1\sigma< 1 during flow map distillation and inference. In supervised tasks where one accurate answer is sufficient, reducing variation in the initial states can ease the burden of fitting the flow map. We apply this on GSM8K, where it improves one-step accuracy (see Figure 4).

Figure 4. GSM8K accuracy (%) of aFMLM (EB). “+ MDLM” initializes the teacher from MDLM, and “+ σ↓\sigma\downarrow” adds noise downscaling.

Experiments

Throughout the experiments, we address these key questions:

  • Q1. Can a proper choice of asynchronous scheduler improve the generation quality of aFLMs?

  • Q2. Does this improvement transfer to one-step generation with aFMLM after distillation?

  • Q3. Do hidden computations of aFMLMs reflect the schedules of aFLMs they are distilled from?

Setup. We evaluate our method on Sudoku solving and code generation for GSM8K math word problems. For Sudoku, we use the dataset with three difficulty levels and report exact solving accuracy (%) on the full test set, following Deschenaux & Gulcehre [16]. For code generation, we train on code completions from TinyGSM (Liu et al. [32]) and measure zero-shot accuracy (%) on the full test set of GSM8K (Cobbe et al. [15]) math problems, following Kim et al. [26].

For the baselines, we consider discrete methods MDLM (Sahoo et al. [44]), Duo (Sahoo et al. [45]) and AlphaDLM (Gushchin et al. [21]), and continuous methods CANDI (Pynadath et al. [39]), FLM (Lee et al. [29]), S-FLM (Deschenaux & Gulcehre [16]), FMLM (Lee et al. [29]), FMLM+ (Agarwal et al. [1]), and DBTM (Tang & Wang [51]). We train and distill for 250k steps on GSM8K and 20k on Sudoku in total, matching the baselines. We leave further details in Appendix § 10.

Asynchronous schedulers for aFLM (Q1)

For each dataset, we train one asynchronous denoiser with diffusion forcing (Chen et al. [12]) and apply various schedulers during inference. We compare the synchronous schedule (Sync.) of FLM (Lee et al. [29]), a schedule fixed by the initial prediction entropy (Init. ent.) of Bansal & Sanghavi [5], and learned schedulers including the top-KK confidence scheduler (Top-KK) and the entropy-bounded scheduler (EB) defined in § 4.3. To test whether generation can also benefit from a prescribed order, we include an algorithmic order (Alg. order) on Sudoku. In this order, the cells that a constraint solver resolves earlier are denoised faster (Shah et al. [48]) (Appendix § 10.§ 3).

The results are in Figure 2, showing that learned schedulers substantially improve generation accuracy on both tasks. With 32-step generation, the EB scheduler improves the accuracy over Sync. from 40.1% to 95.6% on Sudoku-Hard and from 7.4% to 36.4% on GSM8K. Also, schedulers that adapt the order to intermediate predictions (Top-KK and EB) outperform the scheduler that fixes the order from the initial prediction (Init. ent.). Asynchronous flows thus let flow language models benefit from a good generation order, with the largest gains from learned schedulers ported from MDMs.

Figure 2. Accuracy of aFLM under different inference-time schedulers on Sudoku-Hard (left) and GSM8K (right). A well-chosen scheduler improves performance over the synchronous one (Sync.). Alg. order† denotes the aFLM whose denoiser is trained only along the algorithmic order.

While the prescribed algorithmic order improves accuracy on Sudoku-Hard from 40.1% to 59.7%, it still falls short of the learned schedulers. We conjecture that this gap is because diffusion forcing cannot learn all combinations of tokenwise times equally well (Kim et al. [26]) (§ 4.3). Learned schedulers follow the confidence of the denoiser and thus avoid the poorly learned combinations, whereas the algorithmic order may still visit them. Indeed, training the denoiser only along the algorithmic order raises the accuracy from 59.7% to 88.6% (Alg. order† in Figure 2).

One-step generation with aFMLM (Q2)

We distill aFLM from § 5.1 with EB, Top-KK and synchronous schedulers into aFMLM (EB), aFMLM (Top-KK), and aFMLM (Sync.), respectively, and test their one-step generation performance.

The results are shown in Table 1. aFMLM (EB) improves the one-step accuracy over its aFLM teacher, 11.8% vs. 6.1% on Sudoku-Hard and 11.4% vs. 0.3% on GSM8K. In contrast, aFMLM (Sync.), which is distilled from the same teacher, reaches considerably lower performance, showing that the benefit of asynchronous schedulers carries over to one-step generation. aFMLM (Top-KK) shows a similar gain and achieves the highest one-step accuracy on Sudoku-Hard, which shows that the benefit is not specific to a particular choice of the scheduler. Through these advantages, aFMLM surpasses all one-step baselines on both Sudoku and GSM8K and establishes the new state of the art.

ModelEasyMediumHardAccuracy
MDLM ([44])63.616.21.70.0
Duo ([45])69.824.62.40.0
CANDI ([39])69.036.14.00.0
FLM ([29])69.121.52.80.0
S-FLM ([16])77.236.05.20.0
FMLM ([29])75.042.24.90.0
FMLM+ ([1])67.429.94.10.3
DBTM ([51])–––0.8
AlphaDLM ([21])–––6.3
aFLM71.126.56.10.3
aFMLM Sync.77.136.86.41.6
aFMLM EB79.147.811.811.4
aFMLM Top-KK81.446.414.410.5
aFMLM Alg. order82.544.510.0–

Table 1. One-step exact solving accuracy (%) on Sudoku and zero-shot accuracy (%) on GSM8K (see Appendix § 10.§ 2 for baseline details). For aFLM, the schedule does not matter in one-step generation.

In addition, as in Corollary 8, aFMLM can be trained along a prescribed schedule from scratch without a pretrained teacher. We show this with the algorithmic order on Sudoku. As in Table 1, aFMLM (Alg. order) achieves 10.0% one-step accuracy on Sudoku-Hard without a teacher, compared with 11.8% of aFMLM (EB). As shown in Table 2, this approach reduces the total training cost from 5 to 3.2 GPU hours by eliminating teacher pretraining. Our method effectively distills both prescribed and learned schedules, and a useful prescribed schedule offers a more efficient route to one-step generation.

ScheduleTeacherDistillationTotal
Learned1.7h3.3h5.0h
Prescribed0h3.2h3.2h

Table 2. Training cost (GPU hours) on Sudoku-Hard.

In Figure 3 and Figure 4, we analyze the role of our algorithmic choices (§ 4.3) of initializing the aFLM teacher from MDLM and noise downscaling for distillation. Initializing the teacher from

Figure 3. GSM8K accuracy (%) of the teacher aFLM (EB) with 32-step generation across training steps, with and without MDLM initialization.

MDLM accelerates its training, where it reaches 32.9% accuracy on GSM8K with 32-step generation after only 20k training steps, compared with 6.4% from scratch. This suggests MDM initialization eases diffusion forcing training, as the combinations of tokenwise times close to zero or one resemble the problem solved by MDMs. Distilling this stronger teacher improves the one-step accuracy of aFMLM (EB) on GSM8K from 4.2% to 6.3%. Noise downscaling for distillation further improves aFMLM from 6.3% to 11.4% accuracy, confirming that it reduces the burden of fitting the flow map.

Internalization of the schedule into aFMLM (Q3)

We analyze the hidden computation of one-step aFMLM using logit lens [6], which decodes the hidden state of each layer using the output head. For each token, we record the earliest layer from which its decoded prediction matches the final output and remains so, and compare this layer-wise order with the order induced by the schedule. For Sudoku, we compare with the algorithmic order, and for GSM8K, we compare with the order in which the aFLM teacher under the EB scheduler decides the tokens. In Table 3, we measure the alignment between the two orders by the Spearman correlation, computed per sample and averaged over the test set. On both tasks, aFMLM shows a higher correlation with its schedule than aFMLM (Sync.), which indicates that aFMLM internalizes the schedule into its hidden computation. This is consistent with the observation of Asiimwe et al. [2] in image generation that flow maps compress the flow across timesteps into their layers. We further visualize an example in Figure 5. aFMLM (Sync.) does not follow the order of the teacher and produces non-parsable lines, showing that the order matters. In contrast, aFMLM (EB) resolves the tokens that the teacher generates early in its early layers and produces a parsable solution.

Figure 5. A visualization of code solutions to a GSM8K problem generated by one-step aFMLM (EB) (middle) and aFMLM (Sync.) (bottom) in their layer-wise order, compared with the generation order of aFLM (EB) (top). Columns show when 10%, 50%, 85%, and 100% of the tokens are decided.

ScheduleSudoku-HardGSM8K
Synchronous0.5000.091
Asynchronous0.5160.271

Table 3. Spearman correlation with the schedule-induced ordering.

Conclusion

We introduced asynchronous flow map language models (aFMLMs), which distill the ordered generation of flow language models into one-step generators. To this end, we modeled asynchronous generation with an asynchronous flow language model (aFLM), which evolves a coupled flow of tokens and their tokenwise times. We then derived a self-distillation objective that learns the flow map of this flow, which jointly integrates both over any time interval. Empirically, a well-designed scheduler substantially improves aFLMs, and aFMLMs retain these advantage in one-step generation without evaluating the scheduler at test time. We also showed that aFMLMs internalize the order of the scheduler into their hidden computation. Promising future directions include jointly optimizing the scheduler with the denoiser and scaling aFMLMs to larger continuous language models.

Acknowledgments

This work is supported by the National Research Foundation of Korea (RS-2024-00351212) and the Institute of Information & communications Technology Planning & Evaluation (IITP) (RS-2024-00509279, RS-2026-25518317, and RS-2026-25617720), funded by the Korean Government (MSIT). Satoshi Hayakawa was partially supported by JSPS KAKENHI (JP26K25548) and JST PRESTO (JPMJPR26K8).

Proofs and Derivations

Throughout, we assume that the velocity fields in this section are regular enough for their ODEs and continuity equations to have unique solutions on [0,1)[0,1), and that prescribed schedules are absolutely continuous. We also assume ωti>0\omega_t^i > 0 whenever 0<t<10 < t < 1 and τti<1\tau_t^i < 1, and that the tokenwise times reach 11 at t=1t = 1, which holds for the schedulers of (27) with kti>0k_t^i > 0 for 0<t<10 < t < 1 and bounded away from zero near t=1t = 1. Quantities at t=1t = 1 are understood as limits t↑1t \uparrow1, and diagonal quantities at t=st = s as limits t↓st \downarrow s.

Proof of Proposition 1

Proposition 1. Up to continuous nondecreasing reparameterization, there is a unique continuous nondecreasing path from zero to one. In contrast, there are infinitely many such paths from 0 to 1 in [0,1]L[0,1]^L for L≥2L \ge2.

Proof. We parameterize each path by r∈[0,1]r \in[0,1].

Scalar time. Let t(r)t(r) be a continuous nondecreasing path with

t(0)=0,t(1)=1.t(0) = 0,\qquad t(1) = 1.

Then t(r)t(r) passes through every value in [0,1][0,1] in order, so it is a reparameterization of t(r)=rt(r) = r.

Tokenwise times. It suffices to exhibit one infinite family of such paths that remain distinct up to reparameterization.

For L≥2L \ge2 and a>0a > 0, the path τa(r)=(ra,r,…,r)\tau_a(r) = (r^a,r,\ldots,r) goes from 00 to 11 and satisfies τa1=(τa2)a\tau_a^1 = (\tau_a^2)^a along the way.

Since this relation does not change under reparameterization and differs across aa, the paths τa\tau_a are infinitely many and distinct up to reparameterization.

Proof of Proposition 2

Proposition 2. Let (xt,τt)(x_t,\tau_t) solve, starting from x0∼p0x_0 \sim p_0 and τ0=0\tau_0 = 0,

x˙t=ωt(xt,τt)btω(xt,τt),τ˙t=ωt(xt,τt).\dot{x}_t = \omega_t(x_t,\tau_t)b_t^\omega(x_t,\tau_t),\qquad\dot{\tau}_t = \omega_t(x_t,\tau_t).

Then (xt,τt)(x_t,\tau_t) has the same joint distribution as the interpolant and its tokenwise times advanced by ω\omega for every t∈[0,1)t \in[0,1), and xtx_t converges in distribution to p1p_1 as t↑1t \uparrow1.

Proof. For each (x0,x1)(x_0,x_1), let τt\tau_t solve τ˙t=ωt(Iτt,τt)\dot{\tau}_t = \omega_t(I_{\tau_t},\tau_t) with τ0=0\tau_0 = 0, and let Zt:=(Iτt,τt)Z_t := (I_{\tau_t},\tau_t). Since position ii of the interpolant is Iτti=(1−τti)x0i+τtix1iI_{\tau_t}^i = (1-\tau_t^i)x_0^i+\tau_t^i x_1^i, differentiating in tt gives

ddtIτt=τ˙t(x1−x0),soZ˙t=(ωt(Zt)(x1−x0),ωt(Zt)).\frac{\mathrm{d}}{\mathrm{d}t}I_{\tau_t} = \dot{\tau}_t(x_1-x_0),\qquad\text{so}\qquad\dot{Z}_t = \bigl(\omega_t(Z_t)(x_1-x_0),\omega_t(Z_t)\bigr).

Since ωt(Zt)\omega_t(Z_t) is a function of ZtZ_t, taking the conditional mean given Zt=(x,τ)Z_t = (x,\tau) and using (11) gives

E[Z˙t∣Zt=(x,τ)]=(ωt(x,τ)E[x1−x0∣Iτt=x,τt=τ],ωt(x,τ))=(ωt(x,τ)btω(x,τ),ωt(x,τ))=:ut(x,τ).\begin{aligned} \mathbb{E}[\dot{Z}_t\mid Z_t=(x,\tau)] = \bigl(\omega_t(x,\tau)\mathbb{E}[x_1-x_0\mid I_{\tau_t}=x,\tau_t=\tau],\omega_t(x,\tau)\bigr) \\ &= \bigl(\omega_t(x,\tau)b_t^\omega(x,\tau),\omega_t(x,\tau)\bigr)=:u_t(x,\tau). \end{aligned}

Hence, for any smooth test function φ\varphi with compact support, the law qtq_t of ZtZ_t satisfies

ddtE[φ(Zt)]=E[∇φ(Zt)⋅Z˙t]=E[∇φ(Zt)⋅ut(Zt)],\frac{\mathrm{d}}{\mathrm{d}t}\mathbb{E}[\varphi(Z_t)] = \mathbb{E}[\nabla\varphi(Z_t)\cdot\dot{Z}_t] = \mathbb{E}[\nabla\varphi(Z_t)\cdot u_t(Z_t)],

so qtq_t is a weak solution of the continuity equation

∂tqt+∇⋅(qtut)=0,q0=p0⊗δ0.\partial_tq_t+\nabla\cdot(q_tu_t)=0,\qquad q_0=p_0\otimes\delta_0.

The law of (xt,τt)(x_t,\tau_t) in the proposition solves the same equation with the same initial condition, since its velocity is utu_t. Since this solution is unique, the two laws coincide for every t∈[0,1)t \in[0,1). Finally, τt→1\tau_t \to1 as t↑1t \uparrow1, so Iτt→x1I_{\tau_t} \to x_1 for every (x0,x1)(x_0,x_1). Since xtx_t has the law of IτtI_{\tau_t}, it converges in distribution to p1p_1.

Proof of Proposition 3

Proposition 3. Consider the objective

LCE(D^):=Eτ,x0,x1[∑i=1L[−log⁡D^τi(Iτ)]yi],τi∼iidU[0,1].(29)\mathcal{L}_{\mathrm{CE}}(\hat{D}) := \mathbb{E}_{\tau,\mathbf{x}_0,\mathbf{x}_1}\left[\sum_{i=1}^{L}\left[-\log\hat{D}^{i}_{\tau}(I_{\tau})\right]_{y^{i}}\right], \qquad\tau^{i} \mathrel{\overset{\mathrm{iid}}{\sim}} \mathrm{U}[0,1]. \tag*{(29)}

Any population minimizer D^⋆\hat{D}^{\star} satisfies D^τ⋆=Dτ\hat{D}^{\star}_{\tau}=D_{\tau} almost surely under the training distribution.

Proof. Since x1ix^{i}_{1} is the one-hot encoding of yiy^{i}, the loss at position ii can be written as an inner product,

[−log⁡D^τi(Iτ)]yi=−⟨x1i,log⁡D^τi(Iτ)⟩.\left[-\log\hat{D}^{i}_{\tau}(I_{\tau})\right]_{y^{i}}=-\left\langle x^{i}_{1},\log\hat{D}^{i}_{\tau}(I_{\tau})\right\rangle.

Conditioning on (Iτ,τ)(I_{\tau},\tau) and using E[x1∣Iτ,τ]=Dτ(Iτ)\mathbb{E}[x_{1}\mid I_{\tau},\tau]=D_{\tau}(I_{\tau}), which holds because τ\tau is independent of (x0,x1)(x_{0},x_{1}), its expectation is

E[−⟨x1i,log⁡D^τi(Iτ)⟩]=E[−⟨E[x1i∣Iτ,τ],log⁡D^τi(Iτ)⟩]=E[−⟨Dτi(Iτ),log⁡D^τi(Iτ)⟩].\mathbb{E}\left[-\left\langle x^{i}_{1},\log\hat{D}^{i}_{\tau}(I_{\tau})\right\rangle\right]=\mathbb{E}\left[-\left\langle\mathbb{E}[x^{i}_{1}\mid I_{\tau},\tau],\log\hat{D}^{i}_{\tau}(I_{\tau})\right\rangle\right]=\mathbb{E}\left[-\left\langle D^{i}_{\tau}(I_{\tau}),\log\hat{D}^{i}_{\tau}(I_{\tau})\right\rangle\right].

Adding and subtracting ⟨Dτi(Iτ),log⁡Dτi(Iτ)⟩\left\langle D^{i}_{\tau}(I_{\tau}),\log D^{i}_{\tau}(I_{\tau})\right\rangle and summing over positions gives

LCE(D^)=E[∑i=1LH(Dτi(Iτ))]+E[∑i=1LKL⁡(Dτi(Iτ)∥D^τi(Iτ))].(30)\mathcal{L}_{\mathrm{CE}}(\hat{D})=\mathbb{E}\left[\sum_{i=1}^{L}H\left(D^{i}_{\tau}(I_{\tau})\right)\right]+\mathbb{E}\left[\sum_{i=1}^{L}\operatorname{KL}\left(D^{i}_{\tau}(I_{\tau})\middle\|\hat{D}^{i}_{\tau}(I_{\tau})\right)\right]. \tag*{(30)}

where H(p)=−⟨p,log⁡p⟩H(p)=-\langle p,\log p\rangle is the entropy. The first term does not depend on D^\hat{D}, and the second term is nonnegative and vanishes exactly when D^τi(Iτ)=Dτi(Iτ)\hat{D}^{i}_{\tau}(I_{\tau})=D^{i}_{\tau}(I_{\tau}) for every ii. Thus, D^⋆\hat{D}^{\star} minimizes LCE\mathcal{L}_{\mathrm{CE}} if and only if

D^τ⋆(Iτ)=Dτ(Iτ)(31)\hat{D}^{\star}_{\tau}(I_{\tau})=D_{\tau}(I_{\tau}) \tag*{(31)}

almost surely under the training distribution. □

Support of the diffusion forcing training distribution

We show that the training distribution of (Iτ,τ)(I_{\tau},\tau) in (13) has support RL×∣V∣×[0,1]L\mathbb{R}^{L\times|V|}\times[0,1]^{L}, so that every state (xt,τt)(x_{t},\tau_{t}) of an asynchronous flow lies in it regardless of the scheduler.

Proof. Fix τ∈[0,1)L\tau\in[0,1)^{L} and x1x_{1}. Since x0∼N(0,I)x_{0}\sim\mathcal{N}(0,I) is independent of x1x_{1} and τ\tau, each position of the interpolant is distributed as

Iτi=(1−τi)x0i+τix1i∼N(τix1i,(1−(τi)2)I),I^{i}_{\tau}=(1-\tau^{i})x^{i}_{0}+\tau^{i}x^{i}_{1}\sim\mathcal{N}\left(\tau^{i}x^{i}_{1},(1-(\tau^{i})^{2})I\right),

independently across positions. Since 1−τi>01-\tau^{i}>0, IτI_{\tau} has a positive density on RL×∣V∣\mathbb{R}^{L\times|V|}. Moreover, each τi∼U[0,1]\tau^{i}\sim\mathrm{U}[0,1] independently, so τ\tau has a positive density on [0,1)L[0,1)^{L}. The joint distribution of (Iτ,τ)(I_{\tau},\tau) therefore has a positive density on RL×∣V∣×[0,1)L\mathbb{R}^{L\times|V|}\times[0,1)^{L}, and its support is the closure of this set,

RL×∣V∣×[0,1)L‾=RL×∣V∣×[0,1]L,\overline{\mathbb{R}^{L\times|V|}\times[0,1)^{L}}=\mathbb{R}^{L\times|V|}\times[0,1]^{L},

which contains every state (xt,τt)(x_{t},\tau_{t}) of an asynchronous flow. □

Proof of Corollary 4

Corollary 4. Let τ(t)\tau(t) be a prescribed schedule. Then the asynchronous flow in (15) reduces to

x˙t=τ˙(t)bτ(t)(xt),x0∼p0,(32)\dot{x}_{t}=\dot{\tau}(t)b_{\tau(t)}(x_{t}), \qquad x_{0}\sim p_{0}, \tag*{(32)}

and satisfies xt∼pτ(t)x_{t}\sim p_{\tau(t)} for every t∈[0,1)t\in[0,1), with pτ(t)→p1p_{\tau(t)}\to p_{1} as $t\uparrow 1.

Proof. Since ω\omega does not depend on x\mathbf{x}, every interpolant follows the same tokenwise times τt=τ(t)\tau_t = \tau(t) with ωt=τ˙(t)\omega_t = \dot{\tau}(t). The condition τt=τ(t)\tau_t = \tau(t) in (11) therefore holds for every (x0,x1)(\mathbf{x}_0, \mathbf{x}_1), and

btω(x,τ(t))=E[x1−x0∣Iτ(t)=x]=bτ(t)(x).b_t^\omega(\mathbf{x},\tau(t)) = \mathbb{E}[\mathbf{x}_1-\mathbf{x}_0 \mid I_{\tau(t)}=\mathbf{x}] = b_{\tau(t)}(\mathbf{x}).

Substituting into (15) gives (16), whose velocity ωtbτ(t)=ωtbω\omega_t b_{\tau(t)} = \omega_t b^\omega coincides with that of Proposition 2. Hence, xt\mathbf{x}_t has the law of Iτ(t)I_{\tau(t)}, which is pτ(t)p_{\tau(t)} for every t∈[0,1)t \in[0,1), and xt\mathbf{x}_t converges in distribution to p1p_1 as t↑1t \uparrow1.

Proof of Proposition 5

Proposition 5. On any interval where the asynchronous flow (15) admits unique solutions, its solution operator Fs,t(zs)=ztF_{s,t}(z_s)=z_t, called the asynchronous flow map, satisfies

Fs,t(zs)=zs+∫stz˙r dr,F_{s,t}(z_s)=z_s+\int_s^t \dot{z}_r\,\mathrm{d}r,
Fs,t=Fu,t∘Fs,u,s≤u≤t<1.(33)F_{s,t}=F_{u,t}\circ F_{s,u}, \qquad s\leq u\leq t<1. \tag*{(33)}

Proof. Let (xr,τr)=Fs,r(z)(\mathbf{x}_r,\tau_r)=F_{s,r}(z), where z=(x,τ)z=(\mathbf{x},\tau). Integrating the coupled dynamics (15) from ss to tt gives the first identity.

For s≤u≤t<1s\leq u\leq t<1, both r↦Fs,r(z)r\mapsto F_{s,r}(z) and r↦Fu,r(Fs,u(z))r\mapsto F_{u,r}(F_{s,u}(z)) solve the coupled dynamics on [u,t][u,t] with the same state Fs,u(z)F_{s,u}(z) at r=ur=u. Uniqueness of solutions therefore gives Fs,t(z)=Fu,t(Fs,u(z))F_{s,t}(z)=F_{u,t}(F_{s,u}(z)).

Proof of Proposition 6

Proposition 6. Let Fs,tF_{s,t} be the asynchronous flow map. Then δs,t\delta_{s,t} and ηs,t\eta_{s,t} satisfy:

(i) The transition Fs,t(z)=(xt,τt)F_{s,t}(z)=(\mathbf{x}_t,\tau_t) is recovered as

xt=x+(t−s)ηs,t(z)1−τ(δs,t(z)−x),\mathbf{x}_t=\mathbf{x}+\frac{(t-s)\eta_{s,t}(z)}{1-\tau}\bigl(\delta_{s,t}(z)-\mathbf{x}\bigr),
τt=τ+(t−s)ηs,t(z).(34)\tau_t=\tau+(t-s)\eta_{s,t}(z). \tag*{(34)}

(ii) On the diagonal s=ts=t,

δt,t(z)=Dτ(x),ηt,t(z)=ωt(x,τ).(35)\delta_{t,t}(z)=D_\tau(\mathbf{x}), \qquad\eta_{t,t}(z)=\omega_t(\mathbf{x},\tau). \tag*{(35)}

(iii) For s<u<ts<u<t, the semigroup condition (17) is equivalent to

δs,t(z)=γδs,u(z)+(1−γ)δu,t(Fs,u(z)),\delta_{s,t}(z)=\gamma\delta_{s,u}(z)+(1-\gamma)\delta_{u,t}(F_{s,u}(z)),
ηs,t(z)=u−st−sηs,u(z)+t−ut−sηu,t(Fs,u(z)),(36)\eta_{s,t}(z)=\frac{u-s}{t-s}\eta_{s,u}(z)+\frac{t-u}{t-s}\eta_{u,t}(F_{s,u}(z)), \tag*{(36)}

where

γ=(1−τt)(τu−τs)(1−τu)(τt−τs)∈[0,1]L,\gamma=\frac{(1-\tau_t)(\tau_u-\tau_s)}{(1-\tau_u)(\tau_t-\tau_s)}\in[0,1]^L,

with τs=τ\tau_s=\tau and τu,τt\tau_u,\tau_t the tokenwise times along the flow from zz.

Proof. Throughout, let z=(x,τ)z=(\mathbf{x},\tau) and (xr,τr)=Fs,r(z)(\mathbf{x}_r,\tau_r)=F_{s,r}(z) for r≥sr\geq s, so that (xs,τs)=(x,τ)(\mathbf{x}_s,\tau_s)=(\mathbf{x},\tau). By (18), the averages vs,tv_{s,t} and ηs,t\eta_{s,t} satisfy

(t−s)vs,t(z)=∫stωr(xr,τr)bτr(xr) dr=xt−x,(t-s)v_{s,t}(z)=\int_s^t\omega_r(\mathbf{x}_r,\tau_r)b_{\tau_r}(\mathbf{x}_r)\,\mathrm{d}r=\mathbf{x}_t-\mathbf{x},
(t−s)ηs,t(z)=∫stωr(xr,τr) dr=τt−τ.(37)(t-s)\eta_{s,t}(z)=\int_s^t\omega_r(\mathbf{x}_r,\tau_r)\,\mathrm{d}r=\tau_t-\tau. \tag*{(37)}

Substituting (37) into (19),

δs,ti(z)=xi+1−τiηs,ti(z)vs,ti(z)=xi+1−τiτti−τi(xti−xi).(38)\delta^i_{s,t}(z)=\mathbf{x}^i+\frac{1-\tau^i}{\eta^i_{s,t}(z)}\mathbf{v}^i_{s,t}(z)=\mathbf{x}^i+\frac{1-\tau^i}{\tau^i_t-\tau^i}\left(\mathbf{x}^i_t-\mathbf{x}^i\right). \tag*{(38)}

Solving (38) for xt\mathbf{x}_t and (37) for τt\tau_t proves (i).

Next, we write δs,t\delta_{s,t} as a weighted average of the denoiser along the trajectory. Recall the coupled dynamics,

τ˙ri=ωri(xr,τr),x˙ri=ωri(xr,τr)1−τri(Dτri(xr)−xri).\dot{\tau}^i_r=\omega^i_r(\mathbf{x}_r,\tau_r),\qquad \dot{\mathbf{x}}^i_r=\frac{\omega^i_r(\mathbf{x}_r,\tau_r)}{1-\tau^i_r}\left(D^i_{\tau_r}(\mathbf{x}_r)-\mathbf{x}^i_r\right).

The product rule gives

ddrxri1−τri=x˙ri1−τri+ωri(xr,τr)xri(1−τri)2=ωri(xr,τr)(1−τri)2Dτri(xr).(39)\frac{\mathrm{d}}{\mathrm{d}r}\frac{\mathbf{x}^i_r}{1-\tau^i_r} =\frac{\dot{\mathbf{x}}^i_r}{1-\tau^i_r} +\frac{\omega^i_r(\mathbf{x}_r,\tau_r)\mathbf{x}^i_r}{(1-\tau^i_r)^2} =\frac{\omega^i_r(\mathbf{x}_r,\tau_r)}{(1-\tau^i_r)^2}D^i_{\tau_r}(\mathbf{x}_r). \tag*{(39)}

Integrating (39) and using the tokenwise time dynamics,

xti1−τti−xi1−τi=∫stωri(xr,τr)(1−τri)2Dτri(xr) dr,(40)\frac{\mathbf{x}^i_t}{1-\tau^i_t}-\frac{\mathbf{x}^i}{1-\tau^i} =\int_s^t\frac{\omega^i_r(\mathbf{x}_r,\tau_r)}{(1-\tau^i_r)^2}D^i_{\tau_r}(\mathbf{x}_r)\,\mathrm{d}r, \tag*{(40)}
∫stωri(xr,τr)(1−τri)2 dr=11−τti−11−τi=τti−τi(1−τi)(1−τti).(41)\int_s^t\frac{\omega^i_r(\mathbf{x}_r,\tau_r)}{(1-\tau^i_r)^2}\,\mathrm{d}r =\frac{1}{1-\tau^i_t}-\frac{1}{1-\tau^i} =\frac{\tau^i_t-\tau^i}{(1-\tau^i)(1-\tau^i_t)}. \tag*{(41)}

Substituting (40) and (41) into (38),

δs,ti(z)=∫stωri(xr,τr)(1−τri)2Dτri(xr) dr∫stωri(xr,τr)(1−τri)2 dr∈Δ∣V∣−1.(42)\delta^i_{s,t}(z)= \frac{\int_s^t\frac{\omega^i_r(\mathbf{x}_r,\tau_r)}{(1-\tau^i_r)^2}D^i_{\tau_r}(\mathbf{x}_r)\,\mathrm{d}r} {\int_s^t\frac{\omega^i_r(\mathbf{x}_r,\tau_r)}{(1-\tau^i_r)^2}\,\mathrm{d}r} \in\Delta^{|V|-1}. \tag*{(42)}

For (ii), substituting the diagonal of (18) into (19) gives

δt,t(z)=x+1−τωt(x,τ)ωt(x,τ)bτ(x)=Dτ(x),ηt,t(z)=ωt(x,τ).\delta_{t,t}(z)=\mathbf{x}+\frac{1-\tau}{\omega_t(\mathbf{x},\tau)}\omega_t(\mathbf{x},\tau)b_\tau(\mathbf{x})=D_\tau(\mathbf{x}),\qquad \eta_{t,t}(z)=\omega_t(\mathbf{x},\tau).

These diagonal values are also the limits of the off-diagonal ones. Since the weights in (42) are nonnegative,

∥δs,ti(z)−Dτi(x)∥≤sup⁡r∈[s,t]∥Dτri(xr)−Dτi(x)∥→t↓s0,\left\|\delta^i_{s,t}(z)-D^i_\tau(\mathbf{x})\right\| \leq\sup_{r\in[s,t]}\left\|D^i_{\tau_r}(\mathbf{x}_r)-D^i_\tau(\mathbf{x})\right\| \xrightarrow[t\downarrow s]{}0,

and (37) gives

lim⁡t↓sηs,t(z)=lim⁡t↓s1t−s∫stωr(xr,τr) dr=ωs(x,τ).\lim_{t\downarrow s}\eta_{s,t}(z) =\lim_{t\downarrow s}\frac{1}{t-s}\int_s^t\omega_r(\mathbf{x}_r,\tau_r)\,\mathrm{d}r =\omega_s(\mathbf{x},\tau).

For (iii), fix s<u<ts<u<t and write zu=Fs,u(z)z_u=F_{s,u}(z). By (37), the tokenwise time components of the direct map Fs,t(z)F_{s,t}(z) and the composed map Fu,t(zu)F_{u,t}(z_u) are

τ+(t−s)ηs,t(z)andτ+(u−s)ηs,u(z)+(t−u)ηu,t(zu),\tau+(t-s)\eta_{s,t}(z) \qquad\text{and}\qquad \tau+(u-s)\eta_{s,u}(z)+(t-u)\eta_{u,t}(z_u),

which agree if and only if the identity for η\eta in (22) holds. When this identity holds, let τt\tau_t denote the common tokenwise times, and write

ai=τi,bi=τui,ci=τti.a_i=\tau^i,\qquad b_i=\tau^i_u,\qquad c_i=\tau^i_t.

Applying (i) to [s,u][s,u] and then to [u,t][u,t] gives the token state component of the composed map,

1−ci1−bi[1−bi1−aixi+bi−ai1−aiδs,ui(z)]+ci−bi1−biδu,ti(zu)=1−ci1−aixi+ci−ai1−ai[γiδs,ui(z)+(1−γi)δu,ti(zu)].(43)\begin{aligned} \frac{1-c_i}{1-b_i}\left[\frac{1-b_i}{1-a_i}\mathbf{x}^i+\frac{b_i-a_i}{1-a_i}\delta^i_{s,u}(z)\right] +\frac{c_i-b_i}{1-b_i}\delta^i_{u,t}(z_u) \\ &=\frac{1-c_i}{1-a_i}\mathbf{x}^i+ \frac{c_i-a_i}{1-a_i}\left[\gamma_i\delta^i_{s,u}(z)+(1-\gamma_i)\delta^i_{u,t}(z_u)\right]. \tag*{(43)} \end{aligned}

where

γi=(1−ci)(bi−ai)(1−bi)(ci−ai),1−γi=(1−ai)(ci−bi)(1−bi)(ci−ai).\gamma_i = \frac{(1-c_i)(b_i-a_i)}{(1-b_i)(c_i-a_i)}, \qquad1-\gamma_i = \frac{(1-a_i)(c_i-b_i)}{(1-b_i)(c_i-a_i)}.

Both are nonnegative since ai<bi<ci<1a_i < b_i < c_i < 1, so γi∈[0,1]\gamma_i \in[0,1]. By (i), the token state component of the direct map is 1−ci1−aixi+ci−ai1−aiδs,ti(z)\frac{1-c_i}{1-a_i}x^i + \frac{c_i-a_i}{1-a_i}\delta^i_{s,t}(z), so subtracting it from (43) leaves

ci−ai1−ai[γiδs,ui(z)+(1−γi)δu,ti(zu)−δs,ti(z)].\frac{c_i-a_i}{1-a_i}\left[\gamma_i\delta^i_{s,u}(z) + (1-\gamma_i)\delta^i_{u,t}(z_u) - \delta^i_{s,t}(z)\right].

Since ci>aic_i > a_i, the two token state components then agree if and only if the identity for δ\delta in (22) holds. Hence the semigroup condition (17) holds if and only if both identities hold, which proves (iii).

Proof of Proposition 7

Proposition 7. For λ>0\lambda> 0, consider the population objective

L(δ^,η^)=Es,u,t,z[KL⁡sg(Ts,u,tδ(z)∥δ^s,t(z))]+Et,z[KL⁡(Dτ(x)∥δ^t,t(z))]+λEs,u,t,z[∥sg⁡(Ts,u,tη(z))−η^s,t(z)∥22]+λEt,z[∥ωt(x,τ)−η^t,t(z)∥22].(44)\mathcal{L}(\hat{\delta},\hat{\eta}) = \mathbb{E}_{s,u,t,z}\left[\operatorname{KL}_{\mathrm{sg}}\left(T^{\delta}_{s,u,t}(z)\mathrel{\Vert}\hat{\delta}_{s,t}(z)\right)\right] + \mathbb{E}_{t,z}\left[\operatorname{KL}\left(D_\tau(x)\mathrel{\Vert}\hat{\delta}_{t,t}(z)\right)\right] + \lambda\mathbb{E}_{s,u,t,z}\left[\left\lVert\operatorname{sg}\left(T^\eta_{s,u,t}(z)\right)-\hat{\eta}_{s,t}(z)\right\rVert_2^2\right] + \lambda\mathbb{E}_{t,z}\left[\left\lVert\omega_t(x,\tau)-\hat{\eta}_{t,t}(z)\right\rVert_2^2\right]. \tag*{(44)}

The exact δ\delta and η\eta have zero loss. Conversely, any zero-loss solution satisfies the diagonal and composition identities (21) and (22) almost surely under the training distribution.

Proof. Setting δ^=δ\hat{\delta}=\delta and η^=η\hat{\eta}=\eta, the induced map F^\hat{F} is the exact flow map FF by Proposition 6 (i), and Proposition 6 (ii) gives

δt,t(z)=Dτ(x),ηt,t(z)=ωt(x,τ),\delta_{t,t}(z)=D_\tau(x), \qquad\eta_{t,t}(z)=\omega_t(x,\tau),

so both diagonal terms vanish. Moreover, Proposition 6 (iii) gives

δs,t(z)=Ts,u,tδ(z),ηs,t(z)=Ts,u,tη(z),\delta_{s,t}(z)=T^\delta_{s,u,t}(z), \qquad\eta_{s,t}(z)=T^\eta_{s,u,t}(z),

so both off-diagonal terms vanish as well, and L(δ,η)=0\mathcal{L}(\delta,\eta)=0.

Conversely, all four terms are nonnegative and λ>0\lambda>0, so L(δ^,η^)=0\mathcal{L}(\hat{\delta},\hat{\eta})=0 if and only if each term vanishes almost surely under the training distribution. Since KL divergence and the squared norm vanish exactly when their arguments agree, and stop-gradient leaves the target values unchanged, this gives

δ^t,t(z)=Dτ(x),η^t,t(z)=ωt(x,τ),δ^s,t(z)=Ts,u,tδ(z),η^s,t(z)=Ts,u,tη(z).\hat{\delta}_{t,t}(z)=D_\tau(x), \qquad\hat{\eta}_{t,t}(z)=\omega_t(x,\tau), \qquad\hat{\delta}_{s,t}(z)=T^\delta_{s,u,t}(z), \qquad\hat{\eta}_{s,t}(z)=T^\eta_{s,u,t}(z).

The first two are (21), and expanding the targets in the last two gives (22), which hold for δ^\hat{\delta} and η^\hat{\eta}.

Proof of Corollary 8

Corollary 8. Let τ(t)\tau(t) be a prescribed schedule, with η^\hat{\eta} fixed to its exact time component and training states sampled from the corresponding interpolants. Then (24) reduces to

L(δ^)=Es,u,t,z[KL⁡(sg⁡(Ts,u,tδ(z))∥δ^s,t(z))]+Et,z,x1[KL⁡(x1∥δ^t,t(z))],(45)\mathcal{L}(\hat{\delta}) = \mathbb{E}_{s,u,t,z}\left[\operatorname{KL}\left(\operatorname{sg}\left(T^\delta_{s,u,t}(z)\right)\mathrel{\Vert}\hat{\delta}_{s,t}(z)\right)\right] + \mathbb{E}_{t,z,x_1}\left[\operatorname{KL}(x_1\mathrel{\Vert}\hat{\delta}_{t,t}(z))\right], \tag*{(45)}

up to a constant independent of δ^\hat{\delta}. Its minimizers coincide with the zero-loss solutions of Proposition 7 with η^\hat{\eta} fixed by the schedule.

Proof. We first show that the tokenwise time terms of (24) vanish. By assumption, η^\hat{\eta} is fixed to the exact η\eta, and since τt=τ(t)\tau_t=\tau(t) for every state, (18) gives

ηs,t=τ(t)−τ(s)t−s,ηt,t=τ˙(t)=ωt.\eta_{s,t}=\frac{\tau(t)-\tau(s)}{t-s}, \qquad\eta_{t,t}=\dot{\tau}(t)=\omega_t.

The diagonal tokenwise time term therefore vanishes. Substituting η\eta into (23) gives

Ts,u,tη=(u−s)ηs,u+(t−u)ηu,tt−s=(τ(u)−τ(s))+(τ(t)−τ(u))t−s=ηs,t,\mathcal{T}_{s,u,t}^{\eta}=\frac{(u-s)\eta_{s,u}+(t-u)\eta_{u,t}}{t-s}=\frac{(\tau(u)-\tau(s))+(\tau(t)-\tau(u))}{t-s}=\eta_{s,t},

so the off-diagonal tokenwise time term vanishes as well.

We next rewrite the token state terms. Since xt∼pτ(t)x_t\sim p_{\tau(t)} by Corollary 4, the states in (24) are z=(Iτ(t),τ(t))z=(I_{\tau(t)},\tau(t)), and F^s,u\widehat{F}_{s,u} and γ\gamma in Ts,u,tη\mathcal{T}_{s,u,t}^{\eta} are evaluated at τ(s)\tau(s), τ(u)\tau(u), and τ(t)\tau(t). The off-diagonal token state term is thus the first term of (25). For the diagonal term, x1ix_1^i is one-hot and has zero entropy, so

KL⁡(x1∥δ^t,t(z))=∑i=1L[−log⁡δ^t,ti(z)]yi.\operatorname{KL}(x_1\mathbin\Vert\widehat{\delta}_{t,t}(z))=\sum_{i=1}^{L}\left[-\log\widehat{\delta}_{t,t}^{i}(z)\right]_{y_i}.

which is the tokenwise cross-entropy of (13) with D^τ(t)=δ^t,t\widehat{D}_{\tau(t)}=\widehat{\delta}_{t,t}. Applying the decomposition (30) with τ=τ(t)\tau=\tau(t) gives

Et,z,x1[KL⁡(x1∥δ^t,t(z))]=C+Et,z[KL⁡(Dτ(t)(Iτ(t))∥δ^t,t(z))],\mathbb{E}_{t,z,x_1}\left[\operatorname{KL}(x_1\mathbin\Vert\widehat{\delta}_{t,t}(z))\right]=C+\mathbb{E}_{t,z}\left[\operatorname{KL}(D_{\tau(t)}(I_{\tau(t)})\mathbin\Vert\widehat{\delta}_{t,t}(z))\right],
C:=Et,z[∑i=1LH(Dτ(t)i(Iτ(t)))].C:=\mathbb{E}_{t,z}\left[\sum_{i=1}^{L}H\left(D_{\tau(t)}^{i}(I_{\tau(t)})\right)\right].

where KL⁡\operatorname{KL} sums over token positions and CC does not depend on δ^\widehat{\delta}. Hence (25) equals (24) plus CC. Since (24) is nonnegative and vanishes at the exact δ\delta by Proposition 7, the minimum of (25) is CC, and its minimizers are exactly the zero-loss solutions of Proposition 7.

Scheduler Parameterization

Schedule construction

All schedulers in our experiments take the common form of (27),

ωti(x,τ)=1−τi1−tkti(x,τ),t<1.\omega_t^i(\mathbf{x},\tau)=\frac{1-\tau^i}{1-t}k_t^i(\mathbf{x},\tau),\qquad t<1.

This form sends τt→1\tau_t\to1 as t↑1t\uparrow1 whenever kik^i is bounded away from zero near t=1t=1. Writing krik_r^i for ki(xr,τr)k^i(\mathbf{x}_r,\tau_r) along the trajectory, each tokenwise time satisfies

ddrlog⁡(1−τri)=−τ˙ri1−τri=−kri1−r,\frac{\mathrm{d}}{\mathrm{d}r}\log(1-\tau_r^i)=-\frac{\dot{\tau}_r^i}{1-\tau_r^i}=-\frac{k_r^i}{1-r},

and integrating over [s,t][s,t] gives

1−τti=(1−τsi)exp⁡(−∫stkri1−r dr).(46)1-\tau_t^i=(1-\tau_s^i)\exp\left(-\int_s^t\frac{k_r^i}{1-r}\,\mathrm{d}r\right). \tag*{(46)}

Since kri≥0k_r^i\ge0 and τ0i=0\tau_0^i=0, the tokenwise time τti\tau_t^i never decreases and stays in [0,1)[0,1) for every t<1t<1. Moreover, if kri≥ck_r^i\ge c for some c>0c>0 and all r∈[s,1)r\in[s,1), then ∫stdr/(1−r)=log⁡1−s1−t\int_s^t\mathrm{d}r/(1-r)=\log\frac{1-s}{1-t} gives

1−τti≤(1−τsi)(1−t1−s)c→0as t↑1.1-\tau_t^i\le(1-\tau_s^i)\left(\frac{1-t}{1-s}\right)^c\to0\quad\text{as }t\uparrow1.

All schedulers in our experiments satisfy this condition, since the top-KK confidence and entropy-bounded schedulers use ki≥kmin⁡>0k^i\ge k_{\min}>0, and the power schedules have ktj→1k_t^j\to1 as t↑1t\uparrow1.

The schedulers differ in what kik^i depends on. Let

pi=Dτi(x),ci=∥pi∥∞,hi=H(pi)(47)p_i=D_\tau^i(\mathbf{x}),\qquad c_i=\lVert p_i\rVert_\infty,\qquad h_i=H(p_i) \tag*{(47)}

denote the predicted clean-token distribution, its confidence, and its entropy, respectively. All position indices and statistics below refer to generated tokens, and fixed context tokens remain unchanged.

Power schedules. Several of the schedulers below use the power schedule

τj(t)=tαj(48)\tau^j(t)=t^{\alpha_j} \tag*{(48)}

(Figure 6). Its rate ωtj=αjtαj−1\omega_t^j=\alpha_jt^{\alpha_j-1} takes the common form, with kk depending only on tt,

Figure 6. Power schedules τj(t)=tαj\tau^j(t)=t^{\alpha_j} with β=log⁡3\beta=\log3 (left) and their speeds ktjk_t^j (right). Darker curves denote higher-priority positions, with exponents ranging from αj=1/3\alpha_j=1/3 (darkest) to αj=3\alpha_j=3 (lightest), and the dashed line denotes the synchronous schedule.

ktj=ωtj1−t1−τj(t)=αjtαj−11−t1−tαj,(49)k_t^j=\omega_t^j\frac{1-t}{1-\tau^j(t)}=\alpha_jt^{\alpha_j-1}\frac{1-t}{1-t^{\alpha_j}}, \tag*{(49)}

which tends to one as t↑1t\uparrow1. Given an order over the generated positions, let ρj∈[0,1]\rho_j\in[0,1] denote the normalized rank of position jj, with zero denoting the highest priority, and set

αj=exp⁡[β(2ρj−1)],(50)\alpha_j=\exp\left[\beta(2\rho_j-1)\right], \tag*{(50)}

so that the exponents range from e−βe^{-\beta} to eβe^\beta and positions with higher priority are denoised faster. Except for the synchronous schedule, we use β=log⁡3\beta=\log3, so that the exponents range from 1/31/3 to $3. We use the following power schedules.

  • Synchronous. Setting β=0\beta=0 gives kj=1k^j=1 and τj(t)=t\tau^j(t)=t, which recovers the synchronous schedule of Lee et al. [29].

  • Left-to-right. ρj\rho_j is the normalized position of token jj among the generated positions.

  • Algorithmic order for Sudoku. ρj=rc(j)/Rc\rho_j=r_c(j)/R_c, where rc(j)r_c(j) is the round in which cell jj is resolved and RcR_c is the last round (Appendix § 10.§ 3). Cells resolved in the same round share the same exponent.

  • Initial entropy. Following Bansal & Sanghavi [5], ρj\rho_j is the rank of position jj by increasing entropy of the initial prediction D0(x0)D_0(\mathbf{x}_0), kept fixed during generation, so that positions with lower entropy are denoised earlier. kk thus depends on the token states only through the initial state x0\mathbf{x}_0.

The following two schedulers instead set kik^i from the current predictions at every step.

Top-KK confidence scheduler. Following MaskGIT (Chang et al. [10]), we advance the KK most confident positions faster, where KK is a fraction f(t)=1−cos⁡(πt/2)f(t)=1-\cos(\pi t/2) of the generated positions and thus increases over time. Let c~i\widetilde{c}_i denote the confidence cic_i standardized over the generated positions, and let q(t)q(t) be its quantile at level 1−f(t)1-f(t), so that a fraction f(t)f(t) of positions have c~i>q(t)\widetilde{c}_i>q(t). We set

ki=kmin⁡+Asigmoid⁡(c~i−q(t)T),(51)k^i=k_{\min}+A\operatorname{sigmoid}\left(\frac{\widetilde{c}_i-q(t)}{T}\right), \tag*{(51)}

where kmin⁡>0k_{\min}>0 is the speed of the remaining positions, A>0A>0 is the additional speed for the selected ones, and T>0T>0 is a temperature that controls how sharply kk changes around the threshold.

Entropy-bounded scheduler. Following EB-Sampler [7], we order token positions by decreasing confidence or increasing entropy. Let πj\pi_j denote the jj-th position in this order. For the first jj positions, we sum their entropies and subtract the largest:

Bj=∑ℓ=1jhπℓ−max⁡1≤ℓ≤jhπℓ.(52)B_j = \sum_{\ell=1}^{j} h_{\pi_\ell} - \max_{1 \le\ell\le j} h_{\pi_\ell}. \tag*{(52)}

We assign larger kk to positions before BjB_j exceeds a threshold ϵ\epsilon, using a sigmoid to decrease kk gradually:

kπj=kmin⁡+Asigmoid⁡(ϵ−BjT).(53)k^{\pi_j} = k_{\min} + A \operatorname{sigmoid}\left(\frac{\epsilon- B_j}{T}\right). \tag*{(53)}

Here A>0A > 0 controls the additional speed, and T>0T > 0 controls how sharply it decreases around ϵ≥0\epsilon\ge0.

Tokenwise time rate reparameterization

Taking the logarithm of (46) at each position gives

log⁡1−τs1−τt=∫stkr(xr,τr)1−r dr,log⁡1−s1−t=∫st11−r dr.\log\frac{1-\tau_s}{1-\tau_t} = \int_s^t \frac{k_r(x_r,\tau_r)}{1-r}\,\mathrm{d}r,\qquad \log\frac{1-s}{1-t} = \int_s^t \frac{1}{1-r}\,\mathrm{d}r.

Dividing the first by the second shows that κs,t\kappa_{s,t} defined in § 4.3 is the average of krk_r along the trajectory weighted by 1/(1−r)1/(1-r),

κs,t=∫stkr(xr,τr)/(1−r) dr∫st1/(1−r) dr.\kappa_{s,t} = \frac{\int_s^t k_r(x_r,\tau_r)/(1-r)\,\mathrm{d}r}{\int_s^t 1/(1-r)\,\mathrm{d}r}.

We now derive the relation between κ^s,s\hat{\kappa}_{s,s} and the diagonal rate η^s,s\hat{\eta}_{s,s}. By Proposition 6(i), the left side of (28) is the tokenwise time component τs+(t−s)η^s,t(zs)\tau_s + (t-s)\hat{\eta}_{s,t}(z_s) of the flow map. Its derivative in tt at t=st=s is therefore η^s,s(zs)\hat{\eta}_{s,s}(z_s).

Writing the power in (28) as an exponential,

(1−t1−s)κ^s,t=exp⁡[κ^s,tlog⁡1−t1−s],\left(\frac{1-t}{1-s}\right)^{\hat{\kappa}_{s,t}} = \exp\left[\hat{\kappa}_{s,t}\log\frac{1-t}{1-s}\right],

and differentiating with respect to tt by the chain and product rules gives

∂t(1−t1−s)κ^s,t=(1−t1−s)κ^s,t⊙[∂tκ^s,tlog⁡1−t1−s−κ^s,t1−t].\partial_t\left(\frac{1-t}{1-s}\right)^{\hat{\kappa}_{s,t}} = \left(\frac{1-t}{1-s}\right)^{\hat{\kappa}_{s,t}} \odot \left[\partial_t\hat{\kappa}_{s,t}\log\frac{1-t}{1-s}-\frac{\hat{\kappa}_{s,t}}{1-t}\right].

At t=st=s, we have 1−t1−s=1\frac{1-t}{1-s}=1 and log⁡1−t1−s=0\log\frac{1-t}{1-s}=0, so the term involving ∂tκ^s,t\partial_t\hat{\kappa}_{s,t} vanishes and

∂t(1−t1−s)κ^s,t∣t=s=−κ^s,s1−s.\left.\partial_t\left(\frac{1-t}{1-s}\right)^{\hat{\kappa}_{s,t}}\right|_{t=s} = -\frac{\hat{\kappa}_{s,s}}{1-s}.

Differentiating both sides of (28) at t=st=s therefore gives

η^s,s(zs)=−(1−τs)⊙(−κ^s,s(zs)1−s)=1−τs1−s⊙κ^s,s(zs).\hat{\eta}_{s,s}(z_s) = -(1-\tau_s)\odot\left(-\frac{\hat{\kappa}_{s,s}(z_s)}{1-s}\right) = \frac{1-\tau_s}{1-s}\odot\hat{\kappa}_{s,s}(z_s).

On the other hand, the scheduler of (27) at the same state is

ωs(xs,τs)=1−τs1−s⊙ks(xs,τs).\omega_s(x_s,\tau_s)=\frac{1-\tau_s}{1-s}\odot k_s(x_s,\tau_s).

Subtracting the two expressions gives

η^s,s(zs)−ωs(xs,τs)=1−τs1−s⊙(κ^s,s(zs)−ks(xs,τs)).(54)\hat{\eta}_{s,s}(z_s)-\omega_s(x_s,\tau_s) = \frac{1-\tau_s}{1-s}\odot\left(\hat{\kappa}_{s,s}(z_s)-k_s(x_s,\tau_s)\right). \tag*{(54)}

Since every entry of (1−τs)/(1−s)(1-\tau_s)/(1-s) is positive for τs<1\tau_s<1 and s<1s<1, the diagonal identity η^s,s=ωs\hat{\eta}_{s,s}=\omega_s holds if and only if κ^s,s=ks\hat{\kappa}_{s,s}=k_s. Moreover, taking the squared norm of (54) gives

∥η^s,s(zs)−ωs(xs,τs)∥22=∑i=1L(1−τsi1−s)2(κ^s,si(zs)−ksi(xs,τs))2,\left\|\hat{\eta}_{s,s}(z_s)-\omega_s(x_s,\tau_s)\right\|_2^2 = \sum_{i=1}^{L}\left(\frac{1-\tau_s^i}{1-s}\right)^2 \left(\hat{\kappa}_{s,s}^i(z_s)-k_s^i(x_s,\tau_s)\right)^2,

so the implemented term λ∥ks−κ^s,s∥22\lambda\|k_s-\hat{\kappa}_{s,s}\|_2^2 only removes these positive weights, and both terms vanish exactly when κ^s,s=ks\hat{\kappa}_{s,s}=k_s.

Experiment Details

Training details

Architecture and hyperparameters. All models on the same task share the architecture, and the tokenwise time τ∈[0,1]L\tau\in[0, 1]^L is embedded per position for the aFLM teacher and aFMLM. Per-task hyperparameters are listed in Table 4. We train all models with AdamW [33] with β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=10−8\epsilon= 10^{-8}, and no weight decay. The learning rate is linearly warmed up to 3×10−43 \times10^{-4} over 2,500 steps and kept constant afterwards, and gradients are clipped to norm 1.0. We use bfloat16 mixed precision and keep an exponential moving average of the weights with decay 0.9999, which is used for evaluation. All models are trained on NVIDIA A100 and H200 GPUs.

TinyGSM / GSM8KSudoku
Architecture (DiT)
Hidden size768512
adaLN cond. dim128128
Layers128
Attention heads128
FFN multiplier44
Dropout0.10.1
Sequence length LL512180
Vocabulary ∣V∣\lvert V\rvert49152 (SmolLM-135M)12
Parameters (approx.)168M29M
Teacher training
OptimizerAdamW (β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, ϵ=10−8\epsilon=10^{-8})
Weight decay0
Gradient clipping1.0
ScheduleConstant w/ linear warmup
Warmup steps2 500
Peak LR3×10−43 \times10^{-4}3×10−43 \times10^{-4}
Total steps80,00010,000
Global batch size512256
Precisionbfloat16 mixed
EMA decay0.9999
Distillation
Total steps170,00010,000
Peak LR3×10−43 \times10^{-4}3×10−43 \times10^{-4}
Global batch size512256
Training steps per curriculum round2,0002,000
Initial rollout steps3232
Noise scale σ\sigma0.11.0

Table 4. Per-task hyperparameters. All models on the same task share the architecture and optimizer, and differ only in the training and inference schedules.

aFLM training. For each task, we train one aFLM teacher with the diffusion forcing loss of (13) for the number of steps in Table 4. The teacher is initialized from MDLM (§ 4.3) unless stated otherwise. We use the aFLM checkpoint at 80k steps, and Figure 3 continues training up to 240k steps only to compare the two initializations. For Alg. order† in Figure 2, we instead train the denoiser only along the algorithmic order of Appendix § 10.§ 3.

Distillation. aFMLM (EB), aFMLM (Top-KK), and aFMLM (Sync.) are initialized from the aFLM teacher and distilled with (24), while aFMLM (Alg. order) is trained from scratch with (25) without a teacher. Following Lee et al. [29], we sample three times s<u<ts < u < t by first sampling d=t−s∼U[0,1]d = t - s \sim U[0, 1], then sampling s∼U[0,1−d]s \sim U[0, 1 - d], and setting u=(s+t)/2u = (s + t)/2. With learned schedulers, the state at time ss is obtained by rolling out the dynamics from time 0 to ss (§ 4.3). We run this rollout curriculum for five rounds, where the first round rolls out the teacher with 32 steps and each later round rolls out the current flow map with half as many steps, down to 2 steps. The final round with 2 rollout steps continues for the remaining training steps. We set the tokenwise time loss weight in (24) to λ=1.0\lambda= 1.0. For noise downscaling (§ 4.3), we draw the initial noise with σ=0.1\sigma= 0.1 on GSM8K both in distillation and at inference, and keep σ=1.0\sigma= 1.0 on Sudoku.

Scheduler hyperparameters. For the top-KK confidence scheduler, we use (T,A)=(0.2,12)(T, A) = (0.2, 12) on all Sudoku difficulty levels and (0.5,48)(0.5, 48) on GSM8K, with kmin⁡=0.1k_{\min} = 0.1. For the entropy-bounded scheduler, we use (ϵ,T,A)=(0.01,0.001,12)(\epsilon, T, A) = (0.01, 0.001, 12) on all Sudoku difficulty levels and (3,0.3,24)(3, 0.3, 24) on GSM8K, with kmin⁡=0.1k_{\min} = 0.1 throughout. For the prescribed power schedules reported here, we use β=log⁡3\beta= \log3, giving exponents in [1/3,3][1/3, 3], except for the synchronous schedule, which uses β=0\beta= 0. For the initial entropy scheduler, we rank the positions by the entropy of the initial prediction and assign the power schedule with β=log⁡3\beta= \log3 according to this rank.

Evaluation. All models are sampled on a uniform grid of the base time tt with the given number of steps. On this grid, the aFLM teachers integrate (15), aFMLM jumps between adjacent grid points with F^\hat{F}, and the tokenwise times evolve according to the scheduler. We evaluate on the full 1,319 GSM8K test questions and 2,000 Sudoku puzzles for each difficulty. For Sudoku, we report exact solving accuracy, and for GSM8K, we execute the generated code and report the accuracy of its output. We measure the training cost in Table 2 on a single H200 GPU.

Baseline details

For Table 1, we take the one-step results reported in each paper and otherwise evaluate the checkpoints released by Deschenaux & Gulcehre [16] and Agarwal et al. [1]. For FMLM+, we report the GSM8K result with MDLM initialization. For DBTM, we report its only one-step GSM8K result, which uses the refinement-loop objective. For AlphaDLM, we report its best one-step result, which the sequence-level objective with its hyperparameter k=16k = 16 achieves. Since DBTM and AlphaDLM report no one-step results on Sudoku and release no checkpoints, we mark them with –.

Algorithmic ordering for Sudoku

Following Shah et al. [48], we start from the given clues c\mathbf{c} and compute the valid digits of each empty cell under the row, column, and box constraints. At each round, we fill every cell that has only one valid digit or is the only valid location of a digit in its row, column, or box, and we repeat until no cell can be resolved. Figure 7 illustrates the first three rounds. We denote by rc(j)r_{\mathbf{c}}(j) the round in which cell jj is resolved and assign the last round RcR_{\mathbf{c}} to unresolved cells, with Rc=1R_{\mathbf{c}} = 1 if no cell is resolved.

Figure 7. Algorithmic ordering for a hard Sudoku puzzle. Plain digits are the given clues, and labels r1r_1–r3r_3 indicate the round in which each empty cell is resolved. Cells resolved in the current round are shaded in darker blue, while previously assigned rounds remain visible. Cells still blank after round 3 are resolved in later rounds (not shown).

Details of the analysis in § 5.3

We apply the logit lens after the attention and MLP sublayers of each transformer block, decoding each hidden state with the final layer norm and output head of the model. For each token, we take the earliest sublayer from which its decoded prediction matches the final output and remains so. For the aFLM teacher, we run 128-step EB sampling from the same initial noise and regard a token as decided at the first step where its tokenwise time exceeds 0.95. The four columns of Figure 5 show the first 10%, 50%, 85%, and 100% of the tokens in this order, and Table 5 lists the block of each aFMLM column.

Model10%50%85%100%
aFMLM (EB)681112
aFMLM (Sync.)271012

Table 5. Transformer block (out of 12) at which each column of Figure 5 is taken.

References

  1. [1]Manan Agarwal, Sheel Shah, Chanhyuk Lee, Jaehoon Yoo, Jerry Huang, Seunghoon Hong, Aditi Raghunathan, Jinwoo Kim, and Nicholas M Boffi. Posterior refinement: Fast language generation via any-order flow maps. arXiv preprint arXiv:2606.24773, 2026. (pages 3, 8, 9, 10, 25)arxiv.org/abs/2606.24773
  2. [2]Arnold Caleb Asiimwe, William Yang, Sanghyuk Chun, Esin Tureci, and Olga Russakovsky. Depth as time in one-step generative models. arXiv preprint arXiv:2610.03626, 2026. (page 11)arxiv.org/abs/2610.03626
  3. [3]Iskander Azangulov, Kianoosh Ashouritaklimi, Leo Zhang, Simon Vary, and Patrick Rebeschini. Masked language flow models. arXiv preprint arXiv:2606.27617, 2026. (page 3)arxiv.org/abs/2606.27617
  4. [4]Alan Baade, Eric Ryan Chan, Kyle Sargent, Changan Chen, Justin Johnson, Ehsan Adeli, and Li Fei-Fei. Latent forcing: Reordering the diffusion trajectory for pixel-space image generation. arXiv preprint arXiv:2602.11401, 2026. (page 2)arxiv.org/abs/2602.11401
  5. [5]Parikshit Bansal and Sujay Sanghavi. Token time continuous diffusion for language modeling. arXiv preprint arXiv:2607.14106, 2026. (pages 1, 2, 3, 9, 22)arxiv.org/abs/2607.14106
  6. [6]Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023. (page 11)arxiv.org/abs/2303.08112
  7. [7]Heli Ben-Hamu, Itai Gat, Daniel Severo, Niklas Nolte, and Brian Karrer. Accelerated sampling from masked diffusion models via entropy bounded unmasking. arXiv preprint arXiv:2505.24857, 2025. (pages 2, 8, 23)
  8. [8]Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003. (page 1)
  9. [9]Alankrita Bhatt, Mukur Gupta, Germain Kolossov, and Andrea Montanari. Generating from discrete distributions using diffusions: Insights from random constraint satisfaction problems. arXiv preprint arXiv:2603.20589, 2026. (page 2)arxiv.org/abs/2603.20589
  10. [10]Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11305–11315. IEEE, 2022. (pages 8, 22)arxiv.org/abs/2202.04200
  11. [11]Arnav Chavan, Chanhyuk Lee, Udbhav Bamba, Jaehoon Yoo, Seunghoon Hong, Aditi Raghunathan, and Jinwoo Kim. Large flow language models. arXiv preprint arXiv:2610.28559, 2026. (pages 2, 3, 8)arxiv.org/abs/2610.28559
  12. [12]Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081–24125, 2024. (pages 2, 4, 5, 9)arxiv.org/abs/2407.01392
  13. [13]Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, and Ge Liu. Langflow: Continuous diffusion rivals discrete in language modeling. arXiv preprint arXiv:2604.11748, 2026. (pages 1, 3)
  14. [14]Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, and Xinchao Wang. dparallel: Learnable parallel decoding for dllms. arXiv preprint arXiv:2509.26488, 2025. (page 2)arxiv.org/abs/2509.26488
  15. [15]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. (page 9)arxiv.org/abs/2110.14168
  16. [16]Justin Deschenaux and Caglar Gulcehre. Language modeling with hyperspherical flows. arXiv preprint arXiv:2605.11125, 2026. (pages 9, 10, 25)arxiv.org/abs/2605.11125
  17. [17]Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemon d, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089, 2022. (page 3)arxiv.org/abs/2211.15089
  18. [18]Floor Eijkelboom, Grigory Bartosh, Christian A Naesseth, Max Welling, and Jan-Willem van de Meent. Variational flow matching for graph generation. Advances in Neural Information Processing Systems, 37:11735–11764, 2024. (page 3)arxiv.org/abs/2406.04843
  19. [19]Nima Fathi, Torsten Scholak, and Pierre-André Noël. Unifying autoregressive and diffusion-based sequence generation. arXiv preprint arXiv:2504.06416, 2025. (page 2)arxiv.org/abs/2504.06416
  20. [20]Marc Finzi, Shikai Qiu, Yiding Jiang, Pavel Izmailov, J Zico Kolter, and Andrew Gordon Wilson. From entropy to epixepity: Rethinking information for computationally bounded intelligence. arXiv preprint arXiv:2601.03220, 2026. (pages 1, 2)arxiv.org/abs/2601.03220
  21. [21]Nikita Gushchin, Dmitry Baranchuk, and Alexander Korotin. Alpha diffusion language models: Factorization alone is not the problem. arXiv preprint arXiv:2609.38066, 2026. (pages 9, 10)arxiv.org/abs/2609.38066
  22. [22]Keya Hu, Linlu Qiu, Yiyang Lu, Hanhong Zhao, Tianhong Li, Yoon Kim, Jacob Andreas, and Kaiming He. Elf: Embedded language flows. arXiv preprint arXiv:2605.10938, 2026. (pages 1, 3)arxiv.org/abs/2605.10938
  23. [23]Zijing Hu, Yunze Tong, Fengda Zhang, Junkun Yuan, Jun Xiao, and Kun Kuang. Asynchronous denoising diffusion models for aligning text-to-image generation. arXiv preprint arXiv:2510.04504, 2025. (page 2)arxiv.org/abs/2510.04504
  24. [24]Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. arXiv preprint arXiv:2506.08009, 2025. (page 2)arxiv.org/abs/2506.08009
  25. [25]Pascal Jutras-Dubé, Patrick Pynadath, Jeremy Lu, Yuan Gao, and Ruqi Zhang. Sticky jump diffusions: A unifying view of masked, continuous, and hybrid diffusion. arXiv preprint arXiv:2607.10951, 2026. (page 3)
  26. [26]Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06768, 2025a. (pages 1, 5, 8, 9)arxiv.org/abs/2502.06768
  27. [27]Minseo Kim, Chenfeng Xu, Coleman Hooper, Harman Singh, Ben Athiwaratkun, Ce Zhang, Kurt Keutzer, and Amir Gholami. Cdlm: Consistency diffusion language models for faster sampling. arXiv preprint arXiv:2511.19269, 2025b. (page 2)arxiv.org/abs/2511.19269
  28. [28]Seo Hyun Kim, Sunwoo Hong, Hojung Jung, Youngrok Park, and Se-Young Yun. Klass: Kl-guided fast inference in masked diffusion models. Advances in Neural Information Processing Systems, 38:92267–92301, 2026. (page 2)arxiv.org/abs/2511.05664
  29. [29]Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah, Jerry Huang, Aditi Raghunathan, Se-unghoon Hong, Nicholas M Boffi, and Jinwoo Kim. Flow map language models: One-step language modeling via continuous denoising. arXiv preprint arXiv:2602.16813, 2026. (pages 1, 2, 3, 8, 9, 10, 22, 24)arxiv.org/abs/2602.16813
  30. [30]Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381, 2023. (page 2)arxiv.org/abs/2307.03381
  31. [31]Yihao Liang, Ze Wang, Hao Chen, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Emad Barsoum, Zicheng Liu, and Niraj K. Jha. Cd4lm: Consistency distillation and adaptive decoding for diffusion language models. arXiv preprint arXiv:2601.02236, 2026. (page 2)DOI
  32. [32]Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. Tinysmm: achieving >80% on gsm8k with small language models. arXiv preprint arXiv:2312.09241, 2023. (page 9)arxiv.org/abs/2312.09241
  33. [33]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. (page 24)
  34. [34]Shen Nie, Fengqi Zhu, Zəbin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. Advances in Neural Information Processing Systems, 38:50608–50646, 2026. (pages 1, 2)arxiv.org/abs/2502.09992
  35. [35]Yueming Pan, Rouyu Feng, Qi Dai, Yuqi Wang, Wenfeng Lin, Mingyu Guo, Chong Luo, and Nanning Zheng. Semantics lead the way: Harmonizing semantic and texture modeling with asynchronous latent diffusion. arXiv preprint arXiv:2512.04926, 2025. (page 2)arxiv.org/abs/2512.04926
  36. [36]Vassilis Papadopoulos, Jérémie Wenger, and Clément Hongler. Arrows of time for large language models. arXiv preprint arXiv:2401.17505, 2024. (pages 1, 2)arxiv.org/abs/2401.17505
  37. [37]William Peebles and Saining Xie. Scalable diffusion models with transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4172–4182. IEEE, 2023. (page 8)arxiv.org/abs/2212.09748
  38. [38]Peter Potaptchik, Jason Yim, Adhi Saravanan, Peter Holderrieth, Eric Vanden-Eijnden, and Michael S Albergo. Discrete flow maps. arXiv preprint arXiv:2604.09784, 2026. (page 3)arxiv.org/abs/2604.09784
  39. [39]Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang. Candi: Hybrid discrete-continuous diffusion models. arXiv preprint arXiv:2510.22510, 2025. (pages 3, 9, 10)arxiv.org/abs/2510.22510
  40. [40]Bingshuo Qian and Xiang Cheng. Learning when to denoise: Optimizing asynchronous schedules for latent diffusion. arXiv preprint arXiv:2606.19662, 2026. (pages 1, 2, 4)arxiv.org/abs/2606.19662
  41. [41]Andrei Rekesh, Jarrid Rector-Brooks, and Cheng-Hao Liu. Inverse-confidence sampling for continuous diffusion language models. In ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning, 2026. (page 3)
  42. [42]Daan Roos, Oscar Davis, Floor Eijkelboom, Michael Bronstein, Max Welling, Ismail Ilkan Ceylan, Luca Ambrogioni, and Jan-Willem van de Meent. Categorical flow maps. arXiv preprint arXiv:2602.12233, 2026. (page 3)arxiv.org/abs/2602.12233
  43. [43]David Ruhe, Jonathan Heek, Tim Salimans, and Emiel Hoogeboom. Rolling diffusion models. arXiv preprint arXiv:2402.09470, 2024. (page 2)arxiv.org/abs/2402.09470
  44. [44]Subham S Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136–130184, 2024. (pages 1, 8, 9, 10)arxiv.org/abs/2406.07524
  45. [45]Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin Chiu, and Volodymyr Kuleshov. The diffusion duality. Proceedings of machine learning research, 267:52584, 2025. (pages 9, 10)arxiv.org/abs/2506.10892
  46. [46]Yair Schiff, Omer Belhasin, Roy Uziel, Matan Rusanovsky, Ran Zilberstein, Marianne Arriola, Gilad Turok, Guanghan Wang, Volodymyr Kuleshov, and Michael Elad. Clock diffusion: Efficient semi-autoregressive continuous diffusion language models. arXiv preprint arXiv:2610.00894, 2026. (pages 2, 3)arxiv.org/abs/2610.00894
  47. [47]Johannes Schusterbauer, Ming Gui, Yusong Li, Pingchuan Ma, Felix Krause, and Björn Ommer. Denoising, fast and slow: Difficulty-aware adaptive sampling for image generation. arXiv preprint arXiv:2604.19141, 2026. (page 2)arxiv.org/abs/2604.19141
  48. [48]Kulin Shah, Nishanth Dikkala, Xin Wang, and Rina Panigrahy. Causal language modeling can elicit search and reasoning capabilities on logic puzzles. arXiv preprint arXiv:2409.10502, 2024. (pages 2, 9, 25)arxiv.org/abs/2409.10502
  49. [49]Claude E Shannon. Prediction and entropy of printed english. Bell system technical journal, 30(1):50–64, 1951. (pages 1, 2)
  50. [50]Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37:103131–103167, 2024. (page 1)arxiv.org/abs/2406.04329
  51. [51]Sophia Tang and Shivi Wang. Discrete beckmann transport models for one-step language modeling and reasoning. arXiv preprint arXiv:2609.15903, 2026. (pages 9, 10)arxiv.org/abs/2609.15903
  52. [53]Tong Wu, Zihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, et al. Ar-diffusion: Auto-regressive diffusion model for text generation. Advances in Neural Information Processing Systems, 36:39957–39974, 2023. (pages 2, 3)arxiv.org/abs/2305.09515
  53. [54]Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. arXiv preprint arXiv:2412.07772, 2024. (page 2)arxiv.org/abs/2412.07772

Paper details

vlnc:W_s2t77uwp
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents