Introduction
★ Seedance Team Research Project ★
Introduction
Large language models (LLMs) have made remarkable progress under the autoregressive (AR) paradigm [6 , 43 ] , but the fixed left-to-right generation order imposes an inherent serial dependency on inference. Diffusion language models (DLMs) [1 , 2 , 34 , 41 ] offer a new way to break this constraint. Cola DLM [13 ] further decomposes text generation into semantic organization in a continuous latent space and text realization in a discrete space; through latent prior modeling and conditional decoding, it provides a viable path toward hierarchical language generation beyond direct token modeling.
However, a structural division between continuous and discrete spaces does not automatically yield an effective division of labor. The key questions are what information the latent space should compress and how the discrete generator should use it. Shortening the latent sequence is not equivalent to obtaining a compact semantic representation, and a reconstruction-only objective does not guarantee that the latents can predict missing or future text [19 , 24 , 29 ] . Meanwhile, the decoder must not only faithfully realize the latent semantics, but also generate the remaining content from the text context when the latent condition is incomplete or imprecise [44 ] . Semantic compression and discrete generation therefore cannot be designed independently: the information the latents should retain must be determined jointly with the information the decoder can complete.
This interdependence further raises the question of how continuous and discrete representations should be learned jointly. Fixing the encoder in advance determines the information carried by the latents before the discrete generator is fully learned, limiting the continual adaptation of the representation to decoding needs. At the same time, letting the latent-prior fitting objective directly shape the encoder may push the latents toward directions that are easier to fit but not necessarily more beneficial for discrete generation. Hierarchical language modeling thus requires not only learning continuous representations and a discrete generator, but also specifying which objectives should jointly shape the representation and which learning processes should remain decoupled.
To address these issues, we propose Cola2 , which jointly studies continuous semantic compression, the discrete reconstruction–generation balance, and their co-adaptation within a unified hierarchical language modeling framework. Its core idea is that, under limited latent capacity, the actual demands of text reconstruction and generation jointly determine what the continuous representation should retain, while the discrete generator handles the residual uncertainty that the latents do not explain. Specifically, Cola2 dynamically segments text into semantic spans, controls the compression rate with an auxiliary-loss-free negative-feedback mechanism, and combines reconstruction and generation supervision to learn compact and predictive latent representations. On the discrete side, a block-causal masked diffusion model (MDM) decoder [1 , 41 ] is jointly trained under clean, noisy, and dropped latent conditions, balancing semantic reconstruction with context-based autonomous generation, and classifier-free guidance (CFG) [15 ] adjusts this balance at inference.
During training, the token-prediction loss is backpropagated through the latents to the encoder, so that the continuous representation and discrete generation co-adapt; the Diffusion Transformer (DiT) [38 ] learns the current latent distribution via Flow Matching [26 ] without backpropagating gradients to the encoder, thereby decoupling prior fitting from representation learning. At inference, the model re-encodes the generated text block by block under the fixed span layout, keeping the latent history consistent with the actual text context and aligning the two spaces throughout generation.
Building on these ideas, we systematically study co-adaptive hierarchical language generation across continuous and discrete spaces from three perspectives: probabilistic modeling, representation learning, and generation performance. Our main contributions are as follows:
We propose Cola2 , a co-adaptive hierarchical language modeling framework. From the perspective of conditional information decomposition, we analyze semantic compression, the reconstruction–generation balance, and continuous–discrete co-adaptation in a unified manner, clarifying the relation between the information retained by the latents and the residual left to discrete generation.
We develop a generation-oriented joint learning mechanism for continuous and discrete representations. It combines dynamic semantic compression, generation-oriented supervision, and training under latent
conditions of varying reliability, and coordinates representation learning with prior fitting through gradient decoupling, so that compact semantic representations and discrete generation capability co-evolve.
We validate the core designs and their scaling potential through systematic experiments. Controlled five-task scaling to about 210B tokens and a separate 346.03B-source-token pretraining run show competitive, unsaturated scaling. The latter reaches a 60.10 average over eight tasks and surpasses Llama-3-8B on HellaSwag and StoryCloze despite over 43× fewer reported tokens, demonstrating decoder capability transfer and compatibility with autoregressive and masked-diffusion token-space generators.
Related Work
Semantic Compression and Hierarchical Language Models
Fixed-granularity methods shorten the modeled sequence through chunking or downsampling [33 , 48 ] . Adaptive compression methods dynamically aggregate bytes or tokens according to content [18 , 36 , 39 ] . Concept-level methods organize generation through sentence representations or concept prediction [3 , 7 ] . In contrast, Cola2 maintains the target compression rate through negative-feedback control without an auxiliary compression-rate loss, and combines reconstruction and generation supervision to learn compact and predictive continuous semantic representations.
Token-Space Language Models
Autoregressive methods generate text serially, token by token [6 , 11 , 45 ] . Discrete diffusion methods predict multiple positions in parallel through iterative denoising [2 , 27 , 34 , 41 , 42 , 47 ] , and have been further extended to block-level generation and token editing [1 , 4 , 5 ] . Continuous-augmented methods use paired continuous states to assist discrete denoising [44 , 51 , 52 ] . In contrast, Cola2 learns a semantic prior in a compressed latent space and then performs conditional generation with a discrete decoder, handling semantic modeling and text realization hierarchically.
Continuous Representations and Latent-Space Language Models
Token-aligned methods perform diffusion or flow modeling on continuous representations [8 , 10 , 12 , 16 , 22 , 23 , 46 ] . Autoencoder-based latent-space methods separate latent generation from text decoding [28 , 29 , 50 ] . Joint-learning methods further co-optimize the latent space and the generative model [13 , 30 ] . In contrast, Cola2 drives representation learning with discrete reconstruction and generation supervision while isolating the prior-fitting gradient, so that semantic compression and the decoder’s reconstruction–generation balance are optimized jointly.
Cola2: Continuous Latent Diffusion for Hierarchical Text Modeling
We formulate hierarchical text generation as a continuous latent-variable model and, from an information-decomposition perspective, discuss semantic compression, the reconstruction–generation balance, and continuous–discrete co-adaptation (§ 4.1 ). We then present the unified model, training procedure, latent-feedback inference procedure, and computational cost (§ 4.2 –§ 4.3 ).
Probabilistic and Information-Theoretic Perspective
Hierarchical Probabilistic Modeling
Let C C C denote the observed context, Y Y Y the text to be generated, and Z = ( z 1 , … , z K ) Z = (z_1,\ldots,z_K) Z = ( z 1 , … , z K ) a continuous latent sequence with z k ∈ R d z z_k \in\mathbb{R}^{d_z} z k ∈ R d z . If Y Y Y contains T T T tokens, the compression rate is ρ = K / T \rho= K/T ρ = K / T . Cola2 factorizes the text probability into a continuous latent prior and discrete conditional generation:
p ( Y ∣ C ) = ∫ p θ ( Z ∣ C ) p ψ ( Y ∣ C , Z ) d Z . (1) p(Y \mid C) = \int p_\theta(Z \mid C) p_\psi(Y \mid C, Z)\,\mathrm{d}Z. \tag*{(1)} p ( Y ∣ C ) = ∫ p θ ( Z ∣ C ) p ψ ( Y ∣ C , Z ) d Z . (1)
During training, introducing the encoder q ϕ ( Z ∣ C , Y ) q_{\phi}(Z \mid C,Y) q ϕ ( Z ∣ C , Y ) as an approximate posterior yields the conditional evidence lower bound (ELBO)
L c E L B O = E q ϕ [ log p ψ ( Y ∣ C , Z ) ] − KL ( q ϕ ( Z ∣ C , Y ) ∥ p θ ( Z ∣ C ) ) ≤ log p ( Y ∣ C ) . (2) \mathcal{L}_{\mathrm{cELBO}} = \mathbb{E}_{q_{\phi}}[\log p_{\psi}(Y \mid C,Z)] - \operatorname{KL}(q_{\phi}(Z \mid C,Y) \mathbin{\|} p_{\theta}(Z \mid C)) \le\log p(Y \mid C). \tag*{(2)} L cELBO = E q ϕ [ log p ψ ( Y ∣ C , Z )] − KL ( q ϕ ( Z ∣ C , Y ) ∥ p θ ( Z ∣ C )) ≤ log p ( Y ∣ C ) . (2)
Defining the conditional aggregated posterior
q ˉ ϕ ( Z ∣ C ) = ∫ q ϕ ( Z ∣ C , Y ) p d a t a ( Y ∣ C ) d Y , (3) \bar{q}_{\phi}(Z \mid C) = \int q_{\phi}(Z \mid C,Y) p_{\mathrm{data}}(Y \mid C)\,\mathrm{d}Y, \tag*{(3)} q ˉ ϕ ( Z ∣ C ) = ∫ q ϕ ( Z ∣ C , Y ) p data ( Y ∣ C ) d Y , (3)
the average negative ELBO decomposes as
− L c E L B O = E [ − log p ψ ( Y ∣ C , Z ) ] + I ( Y ; Z ∣ C ) + E C KL ( q ˉ ϕ ( Z ∣ C ) ∥ p θ ( Z ∣ C ) ) . (4) -\mathcal{L}_{\mathrm{cELBO}} = \mathbb{E}[-\log p_{\psi}(Y \mid C,Z)] + I(Y;Z \mid C) + \mathbb{E}_{C}\operatorname{KL}(\bar{q}_{\phi}(Z \mid C) \mathbin{\|} p_{\theta}(Z \mid C)). \tag*{(4)} − L cELBO = E [ − log p ψ ( Y ∣ C , Z )] + I ( Y ; Z ∣ C ) + E C KL ( q ˉ ϕ ( Z ∣ C ) ∥ p θ ( Z ∣ C )) . (4)
Thus, the encoder determines which information is retained within the limited latent capacity, the discrete generator decoder implements p ψ ( Y ∣ C , Z ) p_{\psi}(Y \mid C,Z) p ψ ( Y ∣ C , Z ) , and the latent DiT only needs to fit the p θ ( Z ∣ C ) p_{\theta}(Z \mid C) p θ ( Z ∣ C ) induced by the online encoder.
Semantic Compression and Information Density
Given C C C , the uncertainty of the target text satisfies
H ( Y ∣ C ) = I ( Y ; Z ∣ C ) + H ( Y ∣ C , Z ) . (5) H(Y \mid C) = I(Y;Z \mid C) + H(Y \mid C,Z). \tag*{(5)} H ( Y ∣ C ) = I ( Y ; Z ∣ C ) + H ( Y ∣ C , Z ) . (5)
If S S S denotes a high-level semantic variable determined by Y Y Y , the chain rule of mutual information gives
H ( Y ∣ C ) = I ( S ; Z ∣ C ) + I ( Y ; Z ∣ S , C ) + H ( Y ∣ C , Z ) . (6) \begin{aligned}
H(Y \mid C) ={}& I(S;Z \mid C) + I(Y;Z \mid S,C) \\
&+ H(Y \mid C,Z). \tag*{(6)}
\end{aligned} H ( Y ∣ C ) = I ( S ; Z ∣ C ) + I ( Y ; Z ∣ S , C ) + H ( Y ∣ C , Z ) . (6)
Under the compression constraint K = ρ T K = \rho T K = ρT , whether the latents are worth further modeling in continuous space depends not only on whether K K K is sufficiently short, but more importantly on whether the information retained in the limited latent positions is semantic and predictive. To characterize this, we define the analytical quantity
D s e m : = I ( S ; Z ∣ C ) K = I ( S ; Z ∣ C ) ρ T , (7) \mathcal{D}_{\mathrm{sem}} := \frac{I(S;Z \mid C)}{K} = \frac{I(S;Z \mid C)}{\rho T}, \tag*{(7)} D sem := K I ( S ; Z ∣ C ) = ρT I ( S ; Z ∣ C ) , (7)
which measures the density of high-level semantic information carried per latent position. This quantity characterizes the design objective rather than serving as a loss estimated directly during training.
Furthermore, partitioning Y Y Y into a visible part Y v i s Y^{\mathrm{vis}} Y vis and a target part Y t a r Y^{\mathrm{tar}} Y tar gives
I ( Y t a r ; Z ∣ C , Y v i s ) = H ( Y t a r ∣ C , Y v i s ) − H ( Y t a r ∣ C , Y v i s , Z ) . (8) I(Y^{\mathrm{tar}};Z \mid C,Y^{\mathrm{vis}}) = H(Y^{\mathrm{tar}} \mid C,Y^{\mathrm{vis}}) - H(Y^{\mathrm{tar}} \mid C,Y^{\mathrm{vis}},Z). \tag*{(8)} I ( Y tar ; Z ∣ C , Y vis ) = H ( Y tar ∣ C , Y vis ) − H ( Y tar ∣ C , Y vis , Z ) . (8)
Therefore, beyond local reconstructability, a high-quality latent space should also reduce the conditional uncertainty of future or missing tokens, i.e., concentrate more generative information within its limited capacity. Section 3.2 instantiates this principle through dynamic semantic compression and generation-oriented latent supervision.
Reconstruction–Generation Balance and Continuous–Discrete Co-adaptation
Let Z ~ \widetilde{Z} Z be the latent condition obtained by passing Z Z Z through a stochastic degradation channel, satisfying the conditional Markov chain Y → Z → Z ~ Y \to Z \to\widetilde{Z} Y → Z → Z given C C C . By the data processing inequality,
I ( Y ; Z ∣ C ) ≥ I ( Y ; Z ~ ∣ C ) , H ( Y ∣ C , Z ) ≤ H ( Y ∣ C , Z ~ ) . (9) I(Y;Z \mid C) \ge I(Y;\widetilde{Z} \mid C), \qquad H(Y \mid C,Z) \le H(Y \mid C,\widetilde{Z}). \tag*{(9)} I ( Y ; Z ∣ C ) ≥ I ( Y ; Z ∣ C ) , H ( Y ∣ C , Z ) ≤ H ( Y ∣ C , Z ) . (9)
When the latent condition is intact, the decoder’s main task is to realize the continuous semantics as the corresponding discrete tokens; as the condition is perturbed or removed, H ( Y ∣ C , Z ϕ ) H(Y \mid C, Z_\phi) H ( Y ∣ C , Z ϕ ) increases, and more uncertainty must be handled by the decoder’s own discrete prediction capability. This gives an information-theoretic form of the reconstruction–generation trade-off.
On the other hand, since H ( Y ∣ C ) H(Y \mid C) H ( Y ∣ C ) is determined by the data distribution, changing the encoder effectively reallocates
H ( Y ∣ C ) = I ( Y ; Z ϕ ∣ C ) + H ( Y ∣ C , Z ϕ ) . (10) H(Y \mid C) = I(Y; Z_\phi\mid C) + H(Y \mid C, Z_\phi). \tag*{(10)} H ( Y ∣ C ) = I ( Y ; Z ϕ ∣ C ) + H ( Y ∣ C , Z ϕ ) . (10)
If E ϕ E_\phi E ϕ is fixed before the decoder is trained, this allocation is determined before the discrete generator is learned. Cola2 instead jointly optimizes
( ϕ ⋆ , ψ ⋆ ) = arg min ϕ , ψ E p d a t a ( C , Y ) E q ϕ ( Z ∣ C , Y ) [ − log p ψ ( Y ∣ C , Z ) ] s.t. K T ≈ ρ ⋆ , (11) (\phi^\star, \psi^\star) = \underset{\phi,\psi}{\arg\min}\ \mathbb{E}_{p_{\mathrm{data}}(C,Y)}\mathbb{E}_{q_\phi(Z\mid C,Y)}[-\log p_\psi(Y\mid C,Z)] \quad\text{s.t.} \quad\frac{K}{T} \approx\rho^\star, \tag*{(11)} ( ϕ ⋆ , ψ ⋆ ) = ϕ , ψ arg min E p data ( C , Y ) E q ϕ ( Z ∣ C , Y ) [ − log p ψ ( Y ∣ C , Z )] s.t. T K ≈ ρ ⋆ , (11)
so that token prediction retains a direct gradient to the encoder:
∇ ϕ L t o k = ∂ L t o k ∂ Z ∂ Z ∂ ϕ ≠ 0. (12) \nabla_\phi\mathcal{L}_{\mathrm{tok}} = \frac{\partial\mathcal{L}_{\mathrm{tok}}}{\partial Z}\frac{\partial Z}{\partial\phi} \ne0. \tag*{(12)} ∇ ϕ L tok = ∂ Z ∂ L tok ∂ ϕ ∂ Z = 0. (12)
This lets the continuous latent space and the discrete realization co-adapt: the encoder learns semantic representations better suited to reconstruction and generation by the actual decoder, while latent-prior fitting is decoupled from this representation learning. The concrete mechanisms for compression, reliability training, gradient routing, and inference alignment are given in the next section.
Workflow
Figure 1 instantiates the above information decomposition with three coordinated modules: the compression variational autoencoder (VAE) encoder E ϕ E_\phi E ϕ learns continuous latents, the discrete generator decoder D ψ D_\psi D ψ realizes the latent condition as discrete tokens, and the block-causal DiT G θ G_\theta G θ learns the generative distribution of the detached online latents. Their relationship can be summarized as
Figure 1. Method overview. Stage 1 jointly trains the compression encoder and the reconstruction–generation balanced decoder; Stage 2 additionally trains a DiT on detached latents; at inference, the DiT generates latents block by block and the decoder decodes them into tokens.
( C , Y ) → E ϕ Z , ( C , Z ) → D ψ Y , sg ( Z ) → L F M G θ . (13) (C,Y) \xrightarrow{E_\phi} Z,\qquad(C,Z) \xrightarrow{D_\psi} Y,\qquad\operatorname{sg}(Z) \xrightarrow{\mathcal{L}_{\mathrm{FM}}} G_\theta. \tag*{(13)} ( C , Y ) E ϕ Z , ( C , Z ) D ψ Y , sg ( Z ) L FM G θ . (13)
Here D ψ D_\psi D ψ is a block-causal masked-diffusion decoder (MDM), which is causal across blocks and performs bidirectional token denoising within the current block. Stage 1 jointly learns E ϕ E_\phi E ϕ and D ψ D_\psi D ψ ; Stage 2 retains this pathway and additionally trains a detached latent DiT. Full implementation details are given in Appendix § 15 .
Model Architecture
Compression VAE encoder. The target sequence Y = ( y 1 , … , y T ) Y = (y_1,\ldots,y_T) Y = ( y 1 , … , y T ) is dynamically partitioned into K K K contiguous spans. Let S k S_k S k denote the token index set of the k k k -th span and L k = ∣ S k ∣ L_k = |S_k| L k = ∣ S k ∣ its length. Then
⋃ k = 1 K S k = { 1 , … , T } , 1 ≤ L k ≤ L max , ρ = K T . (14) \bigcup_{k=1}^{K} S_k = \{1,\ldots,T\},\qquad1 \le L_k \le L_{\max},\qquad\rho= \frac{K}{T}. \tag*{(14)} k = 1 ⋃ K S k = { 1 , … , T } , 1 ≤ L k ≤ L m a x , ρ = T K . (14)
The encoder E ϕ E_\phi E ϕ first performs dynamic segmentation over token representations, then aggregates bidirectionally within each span and models causally along the latent axis, yielding a diagonal Gaussian posterior
q ϕ ( z k ∣ C , Y ) = N ( μ k , diag ( σ k 2 ) ) , z k = μ k + σ k ⊙ ϵ k , ϵ k ∼ N ( 0 , I ) . (15) q_\phi(z_k \mid C,Y) = \mathcal{N}\left(\mu_k,\operatorname{diag}(\sigma_k^2)\right),\qquad z_k = \mu_k + \sigma_k \odot\epsilon_k,\qquad\epsilon_k \sim\mathcal{N}(0,I). \tag*{(15)} q ϕ ( z k ∣ C , Y ) = N ( μ k , diag ( σ k 2 ) ) , z k = μ k + σ k ⊙ ϵ k , ϵ k ∼ N ( 0 , I ) . (15)
The compression rate is maintained near the target ρ ⋆ \rho^\star ρ ⋆ by an independent, gradient-free rate-bias controller rather than learned through an additional differentiable rate loss. The dynamic boundaries, span-local pooling, and convergence properties of the controller are given in Appendix § 15 .§ 2 .
Latent-to-token interface and block-causal MDM decoder. Each latent is first projected into the decoder hidden space, and a duration head predicts the number of tokens it covers:
p ψ ( L k ∣ z k ) = softmax ( MLP d u r ( W i n z k ) ) . (16) p_{\psi}(L_k \mid z_k) = \operatorname{softmax}\left(\operatorname{MLP}_{\mathrm{dur}}(W_{\mathrm{in}}z_k)\right). \tag*{(16)} p ψ ( L k ∣ z k ) = softmax ( MLP dur ( W in z k ) ) . (16)
Given the span lengths, the latents are expanded into a token-aligned condition H Z = ( h 1 Z , … , h T Z ) H^Z = (h_1^Z, \ldots, h_T^Z) H Z = ( h 1 Z , … , h T Z ) . If g ( t ) = k g(t) = k g ( t ) = k indicates that token t t t belongs to span k k k , then
h t Z = Expand ( z g ( t ) , j t , L g ( t ) ) , (17) h_t^Z = \operatorname{Expand}\left(z_{g(t)}, j_t, L_{g(t)}\right), \tag*{(17)} h t Z = Expand ( z g ( t ) , j t , L g ( t ) ) , (17)
where j t j_t j t is the relative position of token t t t within its span. The block-causal MDM decoder D ψ D_{\psi} D ψ then predicts the target tokens in token space:
p ψ ( y t ∣ K t , H t Z ) = softmax ( Head ( D ψ ( K t ) , H t Z ) ) , (18) p_{\psi}(y_t \mid K_t, H_t^Z) = \operatorname{softmax}\left(\operatorname{Head}\left(D_{\psi}(K_t), H_t^Z\right)\right), \tag*{(18)} p ψ ( y t ∣ K t , H t Z ) = softmax ( Head ( D ψ ( K t ) , H t Z ) ) , (18)
where K t K_t K t contains the historical clean blocks and the current masked block; attention is causal across blocks and bidirectional denoising is allowed within the current block, which cannot read its own clean copy. The attention mask and training objective are detailed in Appendix § 15 .§ 4 .
Latent generator. The latent sequence is partitioned into contiguous blocks of fixed size B B B ,
B j = { ( j − 1 ) B + 1 , … , min ( j B , K ) } , J = ⌈ K B ⌉ . (19) \mathcal{B}_j = \{(j-1)B+1,\ldots,\min(jB,K)\}, \qquad J = \left\lceil\frac{K}{B} \right\rceil. \tag*{(19)} B j = {( j − 1 ) B + 1 , … , min ( j B , K )} , J = ⌈ B K ⌉ . (19)
The generator G θ G_{\theta} G θ is a block-causal DiT: causal across blocks and bidirectional within the current block. It learns p θ ( Z ∣ C ) p_{\theta}(Z \mid C) p θ ( Z ∣ C ) in continuous latent space via Flow Matching. High-level generation therefore iterates along a latent axis of length K ≈ ρ T K \approx\rho T K ≈ ρT , while discrete details are recovered by D ψ D_{\psi} D ψ in token space.
Training Procedure and Objectives
Stage 1: semantic compression and reconstruction–generation co-adaptation. Stage 1 jointly optimizes E ϕ E_{\phi} E ϕ , the latent-to-token interface, and D ψ D_{\psi} D ψ . To cover the spectrum in Eq. (9) from strong-condition reconstruction to weak-condition generation, we sample one of three mutually exclusive condition-reliability states for each latent block. Let A ∈ { c o n d , n o i s e , d r o p } A \in\{\mathbf{cond},\mathbf{noise},\mathbf{drop}\} A ∈ { cond , noise , drop } and define
Z ~ A = { Z , A = c o n d , ( 1 − u c ) Z + u c ε , A = n o i s e , ε ∼ N ( 0 , I ) , ∅ , A = d r o p , (20) \widetilde{Z}_{A} =
\begin{cases}
Z, & A = \mathbf{cond},\\
(1-u^{c})Z + u^{c}\varepsilon, & A = \mathbf{noise},\quad\varepsilon\sim\mathcal{N}(0,I),\\
\varnothing, & A = \mathbf{drop},
\end{cases} \tag*{(20)} Z A = ⎩ ⎨ ⎧ Z , ( 1 − u c ) Z + u c ε , ∅ , A = cond , A = noise , ε ∼ N ( 0 , I ) , A = drop , (20)
where ∅ \varnothing ∅ removes only the latent condition of the current block while retaining the same discrete token context. Let L t o k A \mathcal{L}_{\mathrm{tok}}^{A} L tok A denote the MDM prediction objective under state A A A . The three states form
L b a l = λ c L t o k c o n d + λ n L t o k n o i s e + λ d L t o k d r o p , (21) \mathcal{L}_{\mathrm{bal}} = \lambda_{c}\mathcal{L}_{\mathrm{tok}}^{\mathbf{cond}} + \lambda_{n}\mathcal{L}_{\mathrm{tok}}^{\mathbf{noise}} + \lambda_{d}\mathcal{L}_{\mathrm{tok}}^{\mathbf{drop}}, \tag*{(21)} L bal = λ c L tok cond + λ n L tok noise + λ d L tok drop , (21)
where L t o k A \mathcal{L}_{\mathrm{tok}}^{A} L tok A is the masked denoising loss under the corresponding state, defined in Appendix G.3.
To increase the generative information in Eq. (8) , we additionally introduce
L g e n = L e n c m a s k , (22) \mathcal{L}_{\mathrm{gen}} = \mathcal{L}_{\mathrm{encmask}}, \tag*{(22)} L gen = L encmask , (22)
which masks part of the encoder input under a fixed span layout and directly supervises the encoder through the corresponding token prediction. Thus,
L D G D = L b a l + λ g e n L g e n . (23) \mathcal{L}_{\mathrm{DGD}} = \mathcal{L}_{\mathrm{bal}} + \lambda_{\mathrm{gen}}\mathcal{L}_{\mathrm{gen}}. \tag*{(23)} L DGD = L bal + λ gen L gen . (23)
The KL regularization of the diagonal Gaussian posterior is
L K L = K r e f d r e f T r e f 1 K d z ∑ k = 1 K KL ( q ϕ ( z k ∣ C , Y ) ∥ N ( 0 , I ) ) , (24) \mathcal{L}_{\mathrm{KL}} = \frac{K_{\mathrm{ref}}d_{\mathrm{ref}}}{T_{\mathrm{ref}}}\frac{1}{Kd_z}\sum_{k=1}^{K}\operatorname{KL}\left(q_{\phi}(z_k \mid C,Y)\mathbin{\|}\mathcal{N}(0,I)\right), \tag*{(24)} L KL = T ref K ref d ref K d z 1 k = 1 ∑ K KL ( q ϕ ( z k ∣ C , Y ) ∥ N ( 0 , I ) ) , (24)
and the duration predictor is supervised with ground-truth span lengths, giving L d u r \mathcal{L}_{\mathrm{dur}} L dur . The unified Stage 1 objective is
L ( 1 ) = L D G D + β K L L K L + λ d u r L d u r . (25) \mathcal{L}^{(1)} = \mathcal{L}_{\mathrm{DGD}} + \beta_{\mathrm{KL}}\mathcal{L}_{\mathrm{KL}} + \lambda_{\mathrm{dur}}\mathcal{L}_{\mathrm{dur}}. \tag*{(25)} L ( 1 ) = L DGD + β KL L KL + λ dur L dur . (25)
Its key gradient relation is
∇ ϕ L D G D = ∂ L D G D ∂ Z ∂ Z ∂ ϕ ≠ 0 , (26) \nabla_{\phi}\mathcal{L}_{\mathrm{DGD}} = \frac{\partial\mathcal{L}_{\mathrm{DGD}}}{\partial Z}\frac{\partial Z}{\partial\phi} \ne0, \tag*{(26)} ∇ ϕ L DGD = ∂ Z ∂ L DGD ∂ ϕ ∂ Z = 0 , (26)
i.e., discrete token prediction directly shapes the encoder latents, so that E ϕ E_{\phi} E ϕ and D ψ D_{\psi} D ψ jointly determine the information allocation in Eq. (10) . This pathway remains active in Stage 2.
Stage 2: detached latent diffusion co-training. Stage 2 retains Eq. (25) and adds only detached latent-prior fitting. Let Z ˉ = sg ( Z ) \bar{Z} = \operatorname{sg}(Z) Z ˉ = sg ( Z ) ; for the j j j -th latent block, we sample
u j = σ ( ξ j ) , ξ j ∼ N ( μ u , σ u 2 ) , (27) u_j = \sigma(\xi_j), \qquad\xi_j \sim\mathcal{N}(\mu_u,\sigma_u^2), \tag*{(27)} u j = σ ( ξ j ) , ξ j ∼ N ( μ u , σ u 2 ) , (27)
and all latents within the block share the same u j u_j u j . Under the linear Flow Matching path,
z u = ( 1 − u ) z ˉ + u ε , v ⋆ = ε − z ˉ , ε ∼ N ( 0 , I ) , (28) z_u = (1-u)\bar{z} + u\varepsilon, \qquad v^{\star} = \varepsilon- \bar{z}, \qquad\varepsilon\sim\mathcal{N}(0,I), \tag*{(28)} z u = ( 1 − u ) z ˉ + u ε , v ⋆ = ε − z ˉ , ε ∼ N ( 0 , I ) , (28)
which gives
L F M = E [ ∥ G θ ( z u , u ; Z ˉ < j ) − v ⋆ ∥ 2 2 ] . (29) \mathcal{L}_{\mathrm{FM}} = \mathbb{E}\left[\left\|G_{\theta}(z_u,u;\bar{Z}_{<j}) - v^{\star}\right\|_2^2\right]. \tag*{(29)} L FM = E [ G θ ( z u , u ; Z ˉ < j ) − v ⋆ 2 2 ] . (29)
The total Stage 2 objective is therefore
L ( 2 ) = L ( 1 ) + λ F M L F M , (30) \mathcal{L}^{(2)} = \mathcal{L}^{(1)} + \lambda_{\mathrm{FM}}\mathcal{L}_{\mathrm{FM}}, \tag*{(30)} L ( 2 ) = L ( 1 ) + λ FM L FM , (30)
and hence
∇ ϕ L F M = 0 , ∇ ϕ L D G D ≠ 0 , (31) \nabla_{\phi}\mathcal{L}_{\mathrm{FM}} = 0,\qquad\nabla_{\phi}\mathcal{L}_{\mathrm{DGD}} \ne0, \tag*{(31)} ∇ ϕ L FM = 0 , ∇ ϕ L DGD = 0 , (31)
i.e., prior fitting is decoupled from representation learning, while the continuous representation and the discrete realization remain co-adapted end to end. The complete gradient routing is given in Appendix § 15 .§ 6 .
Inference Procedure
Propose–decode–re-encode–commit. Given a prompt C C C , the encoder takes the posterior mode to obtain K p K_{\mathrm{p}} K p prompt latents. Writing
K p = B ⌊ K p B ⌋ + r , 0 ≤ r < B , (32) K_{\mathrm{p}} = B\left\lfloor\frac{K_{\mathrm{p}}}{B} \right\rfloor+ r,\qquad0 \le r < B, \tag*{(32)} K p = B ⌊ B K p ⌋ + r , 0 ≤ r < B , (32)
only the complete block-aligned prefix is persistently committed; the trailing r r r latents and their tokens serve as the clean prefix of the first generated block, and the remaining B − r B-r B − r latent slots are initialized from Gaussian noise.
The j j j -th generated block follows a unified transaction
Z j p r o p → D ψ Y j → E ϕ ( ⋅ ; S ^ j ) Z j c o r r → c o m m i t K j + 1 , (33) Z^{\mathrm{prop}}_j \xrightarrow{D_{\psi}} Y_j \xrightarrow{E_{\phi}(\,\cdot\,;\widehat{S}_j)} Z^{\mathrm{corr}}_j \xrightarrow{\mathrm{commit}} K_{j+1}, \tag*{(33)} Z j prop D ψ Y j E ϕ ( ⋅ ; S j ) Z j corr commit K j + 1 , (33)
where S ^ j \widehat{S}_j S j is the span layout already determined by the duration predictor; feedback does not rerun the dynamic segmenter. The proposal serves only the current token block, and the persistent latent history always stores Z j c o r r Z^{\mathrm{corr}}_j Z j corr .
This design first preserves train–inference consistency. During training, the clean history is obtained by encoding ground-truth tokens; at inference,
Z ≤ j h i s t = E ϕ ( Y ≤ j ; S ^ ≤ j ) , (34) Z^{\mathrm{hist}}_{\leq j} = E_{\phi}\left(Y_{\leq j};\widehat{S}_{\leq j}\right), \tag*{(34)} Z ≤ j hist = E ϕ ( Y ≤ j ; S ≤ j ) , (34)
so the latent history read by the next block is still produced by the same encoder path as in training.
It also keeps the continuous latent context aligned with the discrete token context:
K ≤ j Z = E ϕ ( K ≤ j Y ; S ^ ≤ j ) . (35) K^{Z}_{\leq j} = E_{\phi}\left(K^{Y}_{\leq j};\widehat{S}_{\leq j}\right). \tag*{(35)} K ≤ j Z = E ϕ ( K ≤ j Y ; S ≤ j ) . (35)
If Z j p r o p Z^{\mathrm{prop}}_j Z j prop were committed directly, the subsequent DiT history and the token history actually generated by the decoder could stem from different semantic states, causing cumulative context bias; fixed-span feedback projects both back onto the same committed history. The cache protocol is detailed in Appendix § 15 .§ 7 .
Two-level CFG. The latent DiT modulates the historical latent condition in velocity space:
v ^ γ = v u + γ g e n ( v c − v u ) , (36) \hat{v}_{\gamma} = v_{\mathrm{u}} + \gamma_{\mathrm{gen}}(v_{\mathrm{c}} - v_{\mathrm{u}}), \tag*{(36)} v ^ γ = v u + γ gen ( v c − v u ) , (36)
where v c v_{\mathrm{c}} v c and v u v_{\mathrm{u}} v u are the velocity outputs with and without the historical latent condition, respectively, and γ g e n \gamma_{\mathrm{gen}} γ gen is the guidance scale. Its local probabilistic interpretation is given in Appendix § 14 .
The MDM decoder uses masked_only conditioning, so the historical token KV contains no latent condition, and applies prefix–latent two-axis CFG in logit space. Fixing the current masked block and the span layout, let ℓ a b \ell_{ab} ℓ ab denote the logits obtained by keeping (1) or masking (0) the historical token prefix and the current latent condition, where a , b ∈ { 0 , 1 } a,b \in\{0,1\} a , b ∈ { 0 , 1 } correspond to the two conditions, respectively. With two-axis scales γ p , γ z \gamma_{\mathrm{p}},\gamma_{\mathrm{z}} γ p , γ z , we define
ℓ γ p , γ z = γ p γ z ℓ 11 + γ p ( 1 − γ z ) ℓ 10 + ( 1 − γ p ) γ z ℓ 01 + ( 1 − γ p ) ( 1 − γ z ) ℓ 00 . (37) \ell_{\gamma_{\mathrm{p}},\gamma_{\mathrm{z}}}
= \gamma_{\mathrm{p}}\gamma_{\mathrm{z}}\ell_{11}
+ \gamma_{\mathrm{p}}(1-\gamma_{\mathrm{z}})\ell_{10}
+ (1-\gamma_{\mathrm{p}})\gamma_{\mathrm{z}}\ell_{01}
+ (1-\gamma_{\mathrm{p}})(1-\gamma_{\mathrm{z}})\ell_{00}. \tag*{(37)} ℓ γ p , γ z = γ p γ z ℓ 11 + γ p ( 1 − γ z ) ℓ 10 + ( 1 − γ p ) γ z ℓ 01 + ( 1 − γ p ) ( 1 − γ z ) ℓ 00 . (37)
Setting γ p = 1 \gamma_p = 1 γ p = 1 modulates only the latent, setting γ z = 1 \gamma_z = 1 γ z = 1 modulates only the prefix, and setting both to 1 recovers the fully conditioned branch. This rule changes neither the training objective nor the feedback protocol. The four-branch definition and cache implementation are given in Appendices F.3 and G.6.1.
Let S g e n S_{\mathrm{gen}} S gen be the number of DiT denoising steps. On the uniform time grid u i = 1 − i / S g e n u_i = 1 - i/S_{\mathrm{gen}} u i = 1 − i / S gen , the explicit Euler update is
Z j ( i + 1 ) = Z j ( i ) − 1 S g e n v ^ γ ( Z j ( i ) , u i ) , i = 0 , … , S g e n − 1. (38) Z_j^{(i+1)} = Z_j^{(i)} - \frac{1}{S_{\mathrm{gen}}}\hat{v}_{\gamma}\left(Z_j^{(i)}, u_i\right), \qquad i = 0,\ldots,S_{\mathrm{gen}} - 1. \tag*{(38)} Z j ( i + 1 ) = Z j ( i ) − S gen 1 v ^ γ ( Z j ( i ) , u i ) , i = 0 , … , S gen − 1. (38)
Given the proposal, D ψ D_{\psi} D ψ completes the current token block with a few iterative demasking steps, followed by the feedback correction in Eq. (33) .
Computational Cost
Let D s D_s D s be the training token budget of Stage s s s , c r e p ( s ) c_{\mathrm{rep}}^{(s)} c rep ( s ) the average per-token cost of the encoder–decoder path in that stage, and c G ( 2 ) c_G^{(2)} c G ( 2 ) the average per-token cost of the added DiT, both including the corresponding attention and output projections. Then
C t r a i n ≈ D 1 c r e p ( 1 ) + D 2 ( c r e p ( 2 ) + c G ( 2 ) ) . (39) C_{\mathrm{train}} \approx D_1 c_{\mathrm{rep}}^{(1)} + D_2\left(c_{\mathrm{rep}}^{(2)} + c_G^{(2)}\right). \tag*{(39)} C train ≈ D 1 c rep ( 1 ) + D 2 ( c rep ( 2 ) + c G ( 2 ) ) . (39)
Under the same prior architecture and a comparable layout, K = ρ T K = \rho T K = ρT reduces the dominant linear-layer and attention-pair terms to O ( ρ ) O(\rho) O ( ρ ) and O ( ρ 2 ) O(\rho^2) O ( ρ 2 ) of their token-axis counterparts, respectively, but the representation path persists in both stages.
Inference cost consists of DiT proposal/commit, discrete decoding (including condition preparation), and fixed-span feedback, denoted by C G i n f e r C_G^{\mathrm{infer}} C G infer , C D i n f e r C_D^{\mathrm{infer}} C D infer , and C f b C_{\mathrm{fb}} C fb , respectively. Taking as reference the AR cost C A R i n f e r C_{\mathrm{AR}}^{\mathrm{infer}} C AR infer under the same prompt/output budget and accounting convention,
R i n f e r = C G i n f e r + C D i n f e r + C f b C A R i n f e r . (40) R_{\mathrm{infer}} = \frac{C_G^{\mathrm{infer}} + C_D^{\mathrm{infer}} + C_{\mathrm{fb}}}{C_{\mathrm{AR}}^{\mathrm{infer}}}. \tag*{(40)} R infer = C AR infer C G infer + C D infer + C fb . (40)
Both sides exclude the one-time prefill but retain the cost of subsequently reading the prompt KV.
Let ρ g e n \rho_{\mathrm{gen}} ρ gen be the realized latent/token ratio of the generated text, and let H D H_D H D and H F H_F H F be the numbers of per-block call rounds for condition preparation/decoding/clean-token commit and for feedback/corrected-cache commit, respectively. Ignoring boundary effects and assuming ideally parallel CFG branches, the number of serial call rounds H \mathcal{H} H satisfies
H C o l a 2 H A R ≈ ρ g e n B ( S g e n + H D + H F ) . (41) \frac{\mathcal{H}_{\mathrm{Cola2}}}{\mathcal{H}_{\mathrm{AR}}} \approx\frac{\rho_{\mathrm{gen}}}{B}\left(S_{\mathrm{gen}} + H_D + H_F\right). \tag*{(41)} H AR H Cola2 ≈ B ρ gen ( S gen + H D + H F ) . (41)
This is a forward-round proxy rather than a latency guarantee. The end-to-end cost also depends on the actual number of CFG branches, the token decoder, and feedback; per-term counts, the AR comparison, and the KV analysis are given in Appendix H.
Experiments
Following the training pipeline in § 4.2 , our experiments address three questions: (1) how to build a compressed latent space that is semantically rich and easy to model; (2) how to train the latent generator; and (3) how the decoder should allocate its capacity between conditional reconstruction and autonomous generation.
Experimental Setup
Datasets. For the component analyses and controlled scaling experiments, we train on the open-source OLMo 2 pretraining corpus [35 ] . The component analyses are conducted mainly on LAMBADA and examine the construction of the semantic latent space, the learning of the latent generator, and the reconstruction–generation balance of the decoder. The controlled scaling experiments are conducted on five downstream tasks, RACE [21 ] , OBQA [31 ] , Story Cloze [32 ] , LAMBADA [37 ] , and SQuAD [40 ] , and compare how different generation paradigms improve as training scales. The post-pretraining comparison additionally evaluates HellaSwag [49 ] , NarrativeQA [20 ] , ARC-Easy [9 ] , TruthfulQA [25 ] , and MMLU [14 ] .
Baselines. Component studies compare Cola2 variants and DLCM [39 ] for rate control. Controlled scaling compares AR [6 ] , LLaDA [34 ] , Cola DLM [13 ] , and the uncompressed Cola-MDM (ρ = 1 \rho= 1 ρ = 1 ) under matched model size, data, and evaluation; Cola-MDM shares the Cola2 -MDM decoder to isolate the effect of compression. Post-pretraining comparisons use Llama-3-8B [11 ] , Qwen2.5-7B [17 ] , Qwen3-8B-Base [45 ] , and LLaDA-8B-Base [34 ] .
Metrics. Answers are extracted from model outputs according to each task format, and accuracy (%) is computed by exact match. All tasks except LAMBADA use few-shot prompting. The internal ablations report LAMBADA accuracy by default, and the controlled scaling experiments report the accuracy on all five tasks and their average.
Settings. In the component analyses and controlled scaling experiments, we use the OLMo 2 (dolma2) tokenizer and the AdamW optimizer with β = ( 0.9 , 0.95 ) \beta= (0.9, 0.95) β = ( 0.9 , 0.95 ) , weight decay 0.01, and a maximum sequence length of 512. Stage 1 jointly trains the encoder and decoder; Stage 2 trains the DiT while continuing to update the VAE, and blocks the Flow Matching gradient from reaching the encoder. The learning rates of the Stage 1 VAE and the Stage 2 DiT both start at 10 − 6 10^{-6} 1 0 − 6 , increase linearly over 5,000 warmup steps to 10 − 4 10^{-4} 1 0 − 4 and 1.5 × 10 − 4 1.5 \times10^{-4} 1.5 × 1 0 − 4 , respectively, and then follow cosine decay to 10 − 6 10^{-6} 1 0 − 6 at step 50,000 and 10 − 5 10^{-5} 1 0 − 5 at step 1,000,000, respectively. In Stage 2, the VAE learning rate continues from Stage 1 with a 5,000-step transition, and the encoder learning rate is 0.1× that of the decoder. Unless otherwise stated, each latent covers 2 tokens on average, the latent dimension is 64, and the maximum span length is 8. DiT noise times follow a logit-normal distribution (loc = 2.0, scale = 1.0); the decoder samples clean, noisy, and dropped latent conditions per block with probabilities 0.2, 0.4, and 0.4, respectively. Inference uses 8 DiT denoising steps and 4 MDM demasking steps by default, and the controlled scaling experiments train on about 210B tokens.
Building the Semantic Latent Space
Dynamic Segmentation and Rate Control
Obs. 1 Dynamic segmentation is essential. With ρ = 1 / 2 \rho= 1/2 ρ = 1/2 and L max = 4 L_{\max} = 4 L m a x = 4 , dynamic segmentation (25.0%) outperforms random spans and fixed-length segmentation by 4.4 and 8.2 points (Figure 2 ); relaxing L max L_{\max} L m a x to 8 reaches 29.4%, exceeding the uncompressed model (28.7%), which shows that content-based compression is not only lossless but also beneficial.
Figure 2. Figure 2 Segmentation and rate control. Dynamic segmentation outperforms fixed-length and random segmentation; auxiliary-free negative-feedback control outperforms DLCM [39 ] .
Obs. 2 Feedback rate control beats an auxiliary rate loss. Under the same setting, the former reaches 25.0% whereas DLCM [39 ] reaches only 18.4%: directly regulating the hard compression rate avoids the competition between the rate loss and representation learning.
VAE Architecture
Obs. 3 Encoder layout matters more than size. The four-layer 0+2+2 encoder achieves the best result of 29.6%, outperforming all six-layer variants (20.9–26.3%); removing the pooling layers (2+0+2) yields only 15.7%, whereas reallocating decoder layers has almost no effect (25.0% vs. 24.9%; Figure 3 , left). An oversized encoder makes the latent distribution more complex and harder for the DiT to model. Intra-span bidirectional attention performs best (26.7%), followed by block-causal attention (25.0%), while fully causal attention, endpoint pooling, and mean pooling all stay at or below 19.0% (Figure 3 , right).
Figure 3. VAE architecture. Left: the allocation of encoder layers matters more than depth, with 0+2+2 performing best. Right: intra-span bidirectional attention is the best pooling operator.
Compression and Latent Capacity
Obs. 4 Wider latents are not always better. With mean span length 2, accuracy peaks at 32.3% at d z = 64 d_z = 64 d z = 64 and drops to 14.5% at d z = 512 d_z = 512 d z = 512 (Figure 4 , left). A larger d z d_z d z adds capacity, but also increases the complexity of the distribution that the DiT must fit.
Figure 4. Latent width and maximum span length. Left: d z d_z d z has an interior optimum. Right: the effect of L max L_{\max} L m a x is non-monotonic.
Obs. 5 Compression rate, d z d_z d z , and L max L_{\max} L m a x must be tuned jointly. Even with the best configuration at each compression level, accuracy still falls from 32.3% (span length 2) to 25.2% (span length 4), as shown in Table 1 . The effect of L max L_{\max} L m a x (Figure 4 , right) is governed by the truncation ratio (Figure 5 ): when truncation is frequent, a smaller L max L_{\max} L m a x keeps each latent simpler; once the cap mass drops to about 1% (span length 4, L max = 18 L_{\max}=18 L m a x = 18 ), preserving complete semantic spans dominates and accuracy rises to 21.7%.
Figure 5. Distribution of the number of tokens covered by each latent. Increasing L max L_{\max} L m a x reduces the number of forcibly truncated spans.
Table 1. Best configuration at each compression level. Accuracy decreases as compression becomes stronger.
Generative Supervision of the Encoder
Lemma 4.1 (Generative Supervision Promotes Predictive Representations). When the supervised tokens are hidden from the encoder, the generative objective cannot be satisfied by copying the current targets. When the encoder is trainable, reducing this objective encourages the latent to retain information predictive of missing or future tokens. If the encoder is frozen, the objective only improves the decoder’s use of information already present in the latent. See Appendix § 9 for the proof.
Obs. 6 Generative supervision is key; its form is secondary. Pure reconstruction reaches only 15.7%, and adding the Span-AR or BERT-style objective raises accuracy to 25.2% and 26.5%, respectively (Figure 6 ). With a frozen encoder, the BERT-style objective yields a 2.2-point gain (28.9% → 31.1%), and unfreezing the encoder reaches 35.2%. Consistent with Lemma 4.1, the generative objective improves both the completion ability of the decoder and the latent itself.
Figure 6. Generative supervision. Left: generative objectives substantially outperform pure reconstruction. Middle: the gain is larger when the encoder is trainable. Right: co-evolving the encoder and decoder with the generative loss performs best.
Obs. 7 Encoder and decoder should co-evolve. With the MDM decoder, a frozen encoder yields 47.6%, end-to-end co-evolution raises accuracy to 53.5%, and further adding the generative loss reaches 58.2% (Figure 6 , right). What the latent should retain must be determined jointly with what the decoder can complete.
Learning the Latent Generator
Noise-Schedule Calibration
VAE Update Policy and Gradient Detachment
Lemma 4.2 (Prior Fitting and Representation Learning Should Be Decoupled). The DiT Flow Matching objective only requires the current latent to be easier for the prior to fit; its gradient is not guaranteed to align with the token-side representation objective. Applying stop-gradient to the DiT path ensures that prior fitting does not directly rewrite the encoder, while preserving end-to-end encoder updates from the decoder token loss. See Appendix § 10 for the proof.
Obs. 8 Keep training the VAE, but detach the DiT gradient. A trainable VAE reaches 32.8%, whereas a frozen VAE reaches only 28.7% (Figure 7 , left). Allowing the DiT gradient to flow into the encoder yields a 2.2-point lead early on, but this is overtaken after 80k steps and becomes a 1.0-point deficit at 140k steps (Table 2 ). This supports Lemma 4.2: the DiT gradient rewrites the latent to be easier to fit rather than more useful for text modeling, and detaching the gradient blocks this latent hacking.
Figure 7. VAE update policy and noise schedule. Left: continuing to train the VAE while detaching the DiT gradient performs best. Right: differences relative to loc = 2.0 \operatorname{loc}=2.0 loc = 2.0 ; all three configurations are optimal at 2.0 2.0 2.0 .
Table 2. Whether the DiT gradient flows into the encoder. Gradient injection leads early, but is overtaken by full detachment in the middle and late stages of training.
Noise-Schedule Calibration
Lemma 4.3 (Effective Noise Is Jointly Determined by the Latent and Diffusion). At the same DiT timestep, the available latent signal is affected by both VAE posterior uncertainty and diffusion corruption. Consequently, changing the noise scale of the Stage 1 latent changes the effective signal-to-noise ratio induced by the same DiT schedule, so the noise schedule must be calibrated jointly with the latent distribution. See Appendix § 11 for the proof.
Obs. 9 The noise schedule must match the representation. All three configurations achieve their best results at loc = 2.0 \operatorname{loc}=2.0 loc = 2.0 , and deviating from it costs 0.3–3.0 points, with the magnitude varying with d z d_z d z and the compression level (Figure 7 , right). By Lemma 4.3 , the schedule should be recalibrated when the representation changes.
Block Size and Generator Capacity
Obs. 10 Moderate blocks are best, and larger DiTs tolerate larger blocks. For the 2B DiT, B = 4 B=4 B = 4 is optimal (36.4%), whereas B = 1 B=1 B = 1 and B = 24 B=24 B = 24 reach only 31.8% and 27.3%, respectively (Figure 8 , left). In the right panel of Figure 8 , the slope of the decline for B > 8 B>8 B > 8 becomes monotonically flatter with DiT size: the loss from B = 8 → 16 B=8\rightarrow16 B = 8 → 16 decreases from 4.5 points for 2B to 3.4 points for 7B and 2.7 points for 32B, and the same trend holds at mean span length 4. A larger DiT can jointly model more latents within a single block, which enlarges the range of block sizes it can use effectively.
Figure 8. DiT block size and capacity. Left: moderate block sizes are optimal. Right: larger DiTs yield flatter curves.
Generator Guidance and Inference Steps
Obs. 11 The generator needs strong CFG and about 8 steps. At 8 steps, accuracy rises from 12.2% at γ g e n = 1 \gamma_{\mathrm{gen}} = 1 γ gen = 1 to 37.1% at γ g e n = 14 \gamma_{\mathrm{gen}} = 14 γ gen = 14 ; at 16 steps, the peak moves to γ g e n = 20 \gamma_{\mathrm{gen}} = 20 γ gen = 20 (38.5%) (Figure 9 , left). Using 1 and 4 steps loses about 14 and 5 points, respectively, whereas results between 8 and 128 steps differ by at most 0.3 points (Figure 9 , right).
Figure 9. DiT inference. Left: generator CFG sweep. Right: the DiT saturates at 8 steps.
Balancing the Reconstruction–Generation Decoder
Lemma 4.4 (Latent-Condition Strength Determines Residual Generation). Given the same discrete context, a more reliable latent condition provides more information and leaves less residual uncertainty for the decoder to generate on its own; when the latent is corrupted or removed, the residual uncertainty increases accordingly. Thus, mixed cond/noise/drop training covers the continuum from conditional reconstruction to autonomous generation. See Appendix § 12 for the proof.
Decoder Type
Obs. 12 Decoder factorization matters little. At every checkpoint, the gap between them is at most 1.4 points, and the leader alternates during training (Table 3 ): AR-MTP leads by 0.9 points at 50k steps, whereas MDM leads by 0.7 and 1.4 points at 100k and 150k steps. This indicates that, given a shared latent–token interface, the specific decoder factorization is not the decisive factor for performance, and the hierarchical framework applies to both; subsequent experiments use the MDM decoder.
Table 3. Decoder factorization. The two decoders perform similarly.
Decoder Data Partition
Obs. 13 Train mostly on noisy and dropped conditions, but keep clean ones. Reconstruction accuracy stays above 99.5% in all configurations, even when only 10% of blocks are clean (Figure 10 ), so most of the data can be devoted to generation under degraded conditions. With the drop ratio fixed at 0.2, shifting noisy/clean from 0.0/0.8 to 0.6/0.2 raises LAMBADA accuracy from 39.5% to 43.2%; raising the drop ratio from 0.2 to 0.4 adds another 2.2–2.4 points, while pushing further brings no gain (noisy 0.7: 42.7%; drop 0.7: 43.1%). The selected 0.4/0.4/0.2 (drop/noisy/clean) ties for the best result of 44.7% while retaining more clean data, confirming the reconstruction–generation continuum in Lemma 4.4 .
Figure 10. Decoder data partition. Left: varying the noisy-to-clean ratio. Right: varying the drop ratio. Reconstruction accuracy (right axis) stays above 99.5% throughout; 0.4/0.4/0.2 gives the best balance.
Decoder Guidance
Obs. 14 Decoder CFG must match the training mixture. The balanced decoder reaches 44.7% at w d e c = − 0.9 w_{\mathrm{dec}}=-0.9 w dec = − 0.9 and collapses rapidly at more negative weights, whereas the clean-heavy decoder reaches at best 40.3% (Figure 11 , left), indicating that inference-time guidance cannot compensate for an imbalanced training distribution.
Figure 11. Decoder guidance and parameter allocation. Left: the balanced decoder is optimal at w d e c = − 0.9 w_{\mathrm{dec}}=-0.9 w dec = − 0.9 . Right: with 3B total parameters, a 2B DiT with a 1B decoder is optimal.
Decoder Capacity
Obs. 15 Under a fixed budget, the DiT matters more. With the DiT fixed, increasing the decoder from 500M to 2B yields monotonic gains of 2.1–2.4 points (Figure 12 ); under a total budget of 3B, a 2B DiT with a 1B decoder (44.4%) outperforms the other allocations by 3.3–4.4 points, and imbalance in either direction degrades performance (Figure 11 , right).
Figure 12. Scaling the decoder with a fixed DiT. Accuracy improves monotonically with decoder size.
Demasking Steps and Latent Use
Obs. 16 Demasking saturates at about 8 steps. 8 DiT/8 demasking steps achieve the best result of 49.0% (Figure 13 , left); fewer than 8 DiT steps cause a clear drop, while increasing demasking can partially compensate when the DiT uses too few steps. The 8/4 setting (48.2%) is only 0.8 points below the optimum and serves as the default low-cost setting.
Figure 13. Decoder inference and latent ablation. Left: 8/8 steps are optimal. Right: corrupting the latent substantially degrades performance, with shuffling being the most harmful.
Obs. 17 The decoder relies on aligned latents. Zeroing, resampling, and shuffling the latent reduce accuracy
from 46.5% to 41.2%, 39.3%, and 34.3%, respectively (Figure 13 , right). Zeroing corresponds to the drop state seen in training and therefore degrades more gracefully; shuffled latents are statistically realistic but semantically misaligned and cause the largest damage, showing that the decoder follows the latent content.
Scaling Experiments
Controlled Scaling Trends
Figure 14 extends training to about 210B tokens and compares the models on RACE, OBQA, Story Cloze, LAMBADA, and SQuAD. Cola-MDM (ρ = 1 \rho= 1 ρ = 1 ) and Cola2 -MDM (with the compression VAE from § 5.2 ) share the MDM decoder selected in § 5.7 , which separates the effect of the decoder design from that of dynamic compression.
Figure 14. Scaling trends with training tokens. Cola-MDM achieves the strongest overall scaling; Cola2-MDM also exhibits strong and unsaturated scaling potential under compression.
Obs. 18 The MDM decoder yields the strongest scaling. Cola-MDM finally reaches a task average of about 34.5%, higher than AR (about 33%), and about 64% on LAMBADA, about 15 points above Cola DLM. The weakness of Cola DLM on generative tasks stems from its decoder rather than from the hierarchical framework.
Obs. 19 Compression preserves semantic tasks, with costs on exact-token tasks. Cola2 -MDM matches AR on
Figure 15. Optimal decoder CFG weight during training. For both models, the optimal weight remains within [ − 1.0 , − 0.8 ] [-1.0,-0.8] [ − 1.0 , − 0.8 ] , consistent with the − 0.9 -0.9 − 0.9 selected in § 5.7 , and drifts slowly toward 0 during training; the drift of Cola2-MDM is more gradual, consistent with its slower convergence.
LAMBADA (about 61%) and leads substantially on RACE (about 10.5% vs. 4.5%), with the gap concentrated on SQuAD and Story Cloze, which rely on verbatim reproduction.
Obs. 20 The compressed variant converges more slowly but keeps closing the gap. Since each latent summarizes more tokens, the latent DiT converges more slowly than the discrete generator. As training continues, the latent condition becomes increasingly useful: after about 150B tokens, the average gap to LLaDA shrinks from about 4 to 1.5 points without saturating. The mixed decoder setting w d e c = − 0.9 w_{\mathrm{dec}} = -0.9 w dec = − 0.9 (44.7%) also outperforms both the latent-free (43.5%) and fully conditioned (37.0%) settings, confirming that latent semantics and discrete generation are complementary.
Post-Pretraining Benchmark Comparison
The final checkpoint evaluated here uses a separate large-scale recipe from the controlled scaling study. It is trained on OLMo 2 pretraining data retokenized with the Qwen3-4B tokenizer, using length-4096 sequences. The decoder and the VAE encoder embedding are initialized from Qwen3-4B-Base. During decoder training, masked positions are represented by [MASK] inputs, while decoder self-attention remains causal. Stage I trains the dynamic-compression VAE on 52.43B source tokens. Stage II jointly updates the VAE and an approximately 4B-parameter latent DiT, with detached VAE latents, on an additional 125.83B source tokens. Stage III freezes the Stage-II EMA VAE and continues training only the DiT on 167.77B source tokens. The total is 346.03B source tokens. Architecture, optimization, objective, EMA, and state-transition details are provided in Appendix § 17 .
Inference uses a low-noise-biased eight-step Euler DiT schedule, t i = T ( 1 − i / 8 ) 2 t_i = T(1 - i/8)^2 t i = T ( 1 − i /8 ) 2 for i = 0 , … , 8 i = 0, \ldots, 8 i = 0 , … , 8 , with DiT CFG set to 10. Decoder CFG is − 0.9 -0.9 − 0.9 , each decoding call generates one latent span, dynamic duration and latent feedback are enabled, and decoding is greedy.
We compare this checkpoint with representative 7B–8B pretrained models across eight downstream tasks. Each Cola2 task score reported below is the average of three sampled evaluations. Because the models’ training recipes, tokenizers, and token-counting conventions differ, this comparison contextualizes post-pretraining performance rather than isolating the effect of the generation paradigm.
Obs. 21 Cola2 shows strong scaling potential under a limited training budget. With only 346.03B source tokens and before convergence, Cola2 reaches an eight-task average of 60.10, only 6.15 points behind Llama-3-8B (66.25), which uses at least 15T tokens—over 43× more data. Cola2 also surpasses Llama-3-8B on HellaSwag (41.75 vs. 39.21) and StoryCloze (80.42 vs. 79.49), while reaching 87.62 on ARC-Easy. These results indicate substantial scaling headroom and successful transfer of the Qwen-initialized causal decoder’s text generation
capability. More broadly, continuous-latent hierarchical modeling can integrate effectively with established token-space generation paradigms, including autoregressive and masked-diffusion decoders.
Summary
The results support the design of Cola2 in three respects. Semantic compression: dynamic segmentation and generative supervision keep the latents strongly semantic at high compression rates; the former slightly surpasses the uncompressed model with an appropriate L max L_{\max} L m a x , and the latter raises accuracy from 15.7% to 26.7%. Reconstruction--generation balance: mixed training dominated by noisy and dropped conditions enables the decoder both to faithfully use the latent (shuffling the latent costs 12.2 points) and to autonomously generate the residual information. Continuous--discrete co-adaptation: the information placed in the latent should be determined by the decoder; co-training the encoder and decoder yields a 5.9-point gain, and both continue to benefit from more training data, giving the hierarchical model scaling trends comparable to or better than AR and LLaDA.
Limitations & Future Prospects
Although our results provide initial evidence for the feasibility, competitiveness, and scaling potential of Cola2 , we view the current system as a starting point rather than a finished endpoint. First, the model remains far from convergence, especially the latent DiT, while the present training scale is limited by available resources. Larger models, longer training, and broader data and compute budgets are therefore needed to establish its performance ceiling. Second, compression reduces the dominant linear and attention FLOPs of the latent prior in theory, but end-to-end generation speed also depends on the decoder, CFG branches, latent feedback, and hardware utilization. Infrastructure co-design—including specialized kernels, cache management, and parallel scheduling—is needed to translate theoretical savings into practical speedups. Third, Cola2 already shows compatibility with autoregressive and masked-diffusion decoders, but their co-training objectives, inference trade-offs, and optimal division of labor remain underexplored. Finally, the continuous latent space may provide a shared interface between text and other continuous modalities such as vision, opening a path toward unified latent representations, cross-modal priors, and unified understanding and generation.
Conclusion
We presented Cola2 , a hierarchical language modeling framework that unifies dynamic semantic compression with balanced discrete reconstruction and generation through continuous--discrete co-adaptation. Dynamic compression and generation-oriented supervision encourage compact, predictive latent representations, while the decoder realizes latent semantics and handles residual uncertainty from context. Token-prediction gradients drive encoder--decoder co-adaptation, whereas latent-prior fitting remains decoupled from representation learning. Systematic component analyses and controlled evaluations on five downstream tasks, scaled to about 210B training tokens, support these design principles and show competitive, unsaturated scaling. In a separate resource-constrained run, the final checkpoint trained on 346.03B source tokens achieves a 60.10 average over eight tasks and surpasses Llama-3-8B on HellaSwag and StoryCloze, although that baseline reports over 43 × 43 \times 43 × more pretraining tokens. These results indicate successful transfer of the Qwen-initialized causal decoder’s generation capability and substantial scaling headroom, especially because the latent DiT remains far from convergence. More broadly, Cola2 shows that continuous-latent hierarchical modeling can integrate effectively with established token-space generators, including autoregressive and masked-diffusion decoders, rather than depending on a single decoder paradigm.
Detailed Derivations of the Probabilistic Model and Information Decomposition
Conditional ELBO
During training, the encoder q ϕ ( Z ∣ C , Y ) q_{\phi}(Z \mid C,Y) q ϕ ( Z ∣ C , Y ) serves as the approximate posterior, where ϕ \phi ϕ denotes the encoder parameters. By Jensen’s inequality, the conditional log-likelihood in Eq. (1) satisfies
log p ( Y ∣ C ) ≥ E q ϕ ( Z ∣ C , Y ) [ log p ψ ( Y ∣ C , Z ) − log q ϕ ( Z ∣ C , Y ) p θ ( Z ∣ C ) ] . (42) \log p(Y \mid C) \ge\mathbb{E}_{q_{\phi}(Z\mid C,Y)}\left[\log p_{\psi}(Y \mid C,Z)-\log\frac{q_{\phi}(Z \mid C,Y)}{p_{\theta}(Z \mid C)}\right]. \tag*{(42)} log p ( Y ∣ C ) ≥ E q ϕ ( Z ∣ C , Y ) [ log p ψ ( Y ∣ C , Z ) − log p θ ( Z ∣ C ) q ϕ ( Z ∣ C , Y ) ] . (42)
Define the conditional aggregated posterior as
q ˉ ϕ ( Z ∣ C ) = ∫ q ϕ ( Z ∣ C , Y ) p d a t a ( Y ∣ C ) d Y . (43) \bar{q}_{\phi}(Z \mid C)=\int q_{\phi}(Z \mid C,Y)p_{\mathrm{data}}(Y \mid C)\,\mathrm{d}Y. \tag*{(43)} q ˉ ϕ ( Z ∣ C ) = ∫ q ϕ ( Z ∣ C , Y ) p data ( Y ∣ C ) d Y . (43)
Adding and subtracting log q ˉ ϕ ( Z ∣ C ) \log\bar{q}_{\phi}(Z \mid C) log q ˉ ϕ ( Z ∣ C ) in the average KL term gives
E K L ( q ϕ ( Z ∣ C , Y ) ∥ p θ ( Z ∣ C ) ) = I ( Y ; Z ∣ C ) + E C K L ( q ˉ ϕ ( Z ∣ C ) ∥ p θ ( Z ∣ C ) ) . (44) \begin{aligned}
\mathbb{E}\,\mathrm{KL}(q_{\phi}(Z \mid C,Y)\|p_{\theta}(Z \mid C))
&=I(Y;Z \mid C)+\mathbb{E}_{C}\,\mathrm{KL}(\bar{q}_{\phi}(Z \mid C)\|p_{\theta}(Z \mid C)). \tag*{(44)}
\end{aligned} E KL ( q ϕ ( Z ∣ C , Y ) ∥ p θ ( Z ∣ C )) = I ( Y ; Z ∣ C ) + E C KL ( q ˉ ϕ ( Z ∣ C ) ∥ p θ ( Z ∣ C )) . (44)
Eq. (A.3) clarifies the probabilistic role of each of the three modules: the encoder determines what information the latent carries from the target text, the latent generator fits the conditional latent distribution induced by the encoder, and the decoder models tokens conditioned on the latent. Flow Matching is merely the particular solver used in this work to learn p θ ( Z ∣ C ) p_{\theta}(Z \mid C) p θ ( Z ∣ C ) , not part of the definition of the generative model itself.
Conditional mutual information is defined as
I ( Y ; Z ∣ C ) = H ( Y ∣ C ) − H ( Y ∣ C , Z ) , (45) I(Y;Z \mid C)=H(Y \mid C)-H(Y \mid C,Z), \tag*{(45)} I ( Y ; Z ∣ C ) = H ( Y ∣ C ) − H ( Y ∣ C , Z ) , (45)
and rearranging directly yields Eq. (5) in the main text.
If S S S is a high-level semantic variable determined by Y Y Y , then I ( Y ; Z ∣ C ) = I ( S , Y ; Z ∣ C ) I(Y;Z \mid C)=I(S,Y;Z \mid C) I ( Y ; Z ∣ C ) = I ( S , Y ; Z ∣ C ) . Applying the chain rule of mutual information,
I ( Y ; Z ∣ C ) = I ( S , Y ; Z ∣ C ) = I ( S ; Z ∣ C ) + I ( Y ; Z ∣ S , C ) , (46) \begin{aligned}
I(Y;Z \mid C)&=I(S,Y;Z \mid C)\\
&=I(S;Z \mid C)+I(Y;Z \mid S,C), \tag*{(46)}
\end{aligned} I ( Y ; Z ∣ C ) = I ( S , Y ; Z ∣ C ) = I ( S ; Z ∣ C ) + I ( Y ; Z ∣ S , C ) , (46)
which yields the semantic decomposition in Eq. (6) . Under the compression constraint K = ρ T K=\rho T K = ρT , we further define the semantic information density per latent position as
D s e m = I ( S ; Z ∣ C ) K . (47) D_{\mathrm{sem}}=\frac{I(S;Z \mid C)}{K}. \tag*{(47)} D sem = K I ( S ; Z ∣ C ) . (47)
This quantity need not be estimated explicitly during training; it distinguishes two different objectives: merely shortening the latent sequence versus concentrating semantic information in a limited number of latent positions.
If the target tokens are partitioned into a visible part Y v i s Y^{\mathrm{vis}} Y vis and a target part Y t a r Y^{\mathrm{tar}} Y tar , then
I ( Y t a r ; Z ∣ C , Y v i s ) = H ( Y t a r ∣ C , Y v i s ) − H ( Y t a r ∣ C , Y v i s , Z ) . (48) I(Y^{\mathrm{tar}};Z \mid C,Y^{\mathrm{vis}})=H(Y^{\mathrm{tar}} \mid C,Y^{\mathrm{vis}})-H(Y^{\mathrm{tar}} \mid C,Y^{\mathrm{vis}},Z). \tag*{(48)} I ( Y tar ; Z ∣ C , Y vis ) = H ( Y tar ∣ C , Y vis ) − H ( Y tar ∣ C , Y vis , Z ) . (48)
Generation-oriented latent supervision can therefore be interpreted as biasing the limited latent capacity toward predictive semantic information by reducing the conditional uncertainty of missing or future tokens given the latent. Section B gives a concrete interpretation of the fixed-span encoder-mask objective.
Let p 0 ( Z ∣ C ) p_0(Z \mid C) p 0 ( Z ∣ C ) be the reference distribution used for regularization in Stage 1. For any such reference distribution,
E C , Y KL ( q ϕ ( Z ∣ C , Y ) ∥ p 0 ( Z ∣ C ) ) = I ( Y ; Z ∣ C ) + E C KL ( q ˉ ϕ ( Z ∣ C ) ∥ p 0 ( Z ∣ C ) ) ≥ I ( Y ; Z ∣ C ) . (49) \mathbb{E}_{C,Y}\operatorname{KL}\left(q_{\phi}(Z \mid C,Y)\mathbin{\|}p_0(Z \mid C)\right)
= I(Y;Z \mid C) + \mathbb{E}_C\operatorname{KL}\left(\bar{q}_{\phi}(Z \mid C)\mathbin{\|}p_0(Z \mid C)\right)
\ge I(Y;Z \mid C). \tag*{(49)} E C , Y KL ( q ϕ ( Z ∣ C , Y ) ∥ p 0 ( Z ∣ C ) ) = I ( Y ; Z ∣ C ) + E C KL ( q ˉ ϕ ( Z ∣ C ) ∥ p 0 ( Z ∣ C ) ) ≥ I ( Y ; Z ∣ C ) . (49)
Thus, the VAE KL term not only constrains the latent distribution but also upper-bounds the amount of information the latent can carry. If the posterior approximately factorizes across latents and dimensions, and the average KL per latent dimension stays roughly at r ˉ \bar{r} r ˉ , then the total KL scales approximately with K d z K d_z K d z . Since K = ρ T K = \rho T K = ρT , the information capacity available per token follows the approximate scaling
1 T I ( Y ; Z ∣ C ) ≲ ρ d z r ˉ . (50) \frac{1}{T} I(Y;Z \mid C) \lesssim\rho d_z \bar{r}. \tag*{(50)} T 1 I ( Y ; Z ∣ C ) ≲ ρ d z r ˉ . (50)
Cola2 normalizes the KL per latent element so that the regularization pressure on each latent dimension remains as comparable as possible across different K K K and d z d_z d z . Changing ρ \rho ρ can therefore be understood more directly as changing the number of latent channels available per token. Dynamic segmentation further determines where these limited latent positions are allocated under a fixed target compression rate: the compression rate controls how many representation positions exist, and semantic boundaries determine where they are placed.
Information-Theoretic Interpretation of Generation-Oriented Latent Supervision
The L g e n \mathcal{L}_{\mathrm{gen}} L gen term in the main text is not an additional inference-time sampler, but a fixed-span encoder-mask objective applied directly to the encoder. Its constraint is that a supervised token cannot simply rely on a complete current latent observation in one-to-one correspondence with itself, but must be predicted from a context in which its input is missing. The objective therefore specifically increases the predictive information of the latent rather than merely its local reconstructability.
Fixed-Span Encoder-Mask / BERT-Style Objective
Let M M M denote a randomly selected set of spans, Y M Y_M Y M the text in these spans that is masked on the encoder side, Y M ‾ Y_{\overline{M}} Y M the remaining visible text, and Z M Z^M Z M the latent obtained by encoding the masked input under the fixed original span layout. Since the encoder cannot observe the ground-truth Y M Y_M Y M in this forward pass, we have, given ( C , M ) (C,M) ( C , M ) , the Markov chain
Y M ⟶ ( C , Y M ‾ ) ⟶ Z M . (51) Y_M \longrightarrow(C,Y_{\overline{M}}) \longrightarrow Z^M. \tag*{(51)} Y M ⟶ ( C , Y M ) ⟶ Z M . (51)
By the data processing inequality,
I ( Y M ; Z M ∣ C , M ) ≤ I ( Y M ; Y M ‾ ∣ C , M ) . (52) I(Y_M;Z^M \mid C,M) \le I(Y_M;Y_{\overline{M}} \mid C,M). \tag*{(52)} I ( Y M ; Z M ∣ C , M ) ≤ I ( Y M ; Y M ∣ C , M ) . (52)
Hence the objective cannot be satisfied by directly copying the masked tokens; it can only exploit information in the visible context that is statistically related to the target spans. For any predictor p η ( Y M ∣ C , Z M , M ) p_{\eta}(Y_M \mid C,Z^M,M) p η ( Y M ∣ C , Z M , M ) , the cross-entropy can be written as
L g e n m a s k = H ( Y M ∣ C , Z M , M ) + E KL ( q ( Y M ∣ C , Z M , M ) ∥ p η ( Y M ∣ C , Z M , M ) ) . (53) \mathcal{L}_{\mathrm{gen}}^{\mathrm{mask}} = H(Y_M \mid C,Z^M,M)
+ \mathbb{E}\operatorname{KL}\left(q(Y_M \mid C,Z^M,M)\mathbin{\|}p_{\eta}(Y_M \mid C,Z^M,M)\right). \tag*{(53)} L gen mask = H ( Y M ∣ C , Z M , M ) + E KL ( q ( Y M ∣ C , Z M , M ) ∥ p η ( Y M ∣ C , Z M , M ) ) . (53)
If the predictor is conditionally optimal, then
( L g e n m a s k ) ∗ = H ( Y M ∣ C , Z M , M ) = H ( Y M ∣ C , M ) − I ( Y M ; Z M ∣ C , M ) . (54) \left(\mathcal{L}_{\mathrm{gen}}^{\mathrm{mask}}\right)^* = H(Y_M \mid C,Z^M,M)
= H(Y_M \mid C,M) - I(Y_M;Z^M \mid C,M). \tag*{(54)} ( L gen mask ) ∗ = H ( Y M ∣ C , Z M , M ) = H ( Y M ∣ C , M ) − I ( Y M ; Z M ∣ C , M ) . (54)
Therefore, when the encoder is trainable, reducing this objective drives the latent to contain more predictive information about the missing spans. The current MDM implementation uses fixed-span encoder-mask MDM: the first clean forward pass determines the hard spans, and the second pass replaces only the encoder observations without re-segmenting. Functionally, it thus imposes BERT-style generation pressure, but it is not equivalent to arbitrary forms of the span-BERT loss used in earlier implementations.
Latent Hacking and Gradient Decoupling in Compressed Representations
This appendix proves Lemma 4.2, and first distinguishes two kinds of “joint training.” We deliberately retain the gradient from the discrete generator decoder to the encoder, because token reconstruction and generation are precisely the discrete realization that the latent must ultimately serve. The risk discussed here refers only to allowing the DiT Flow Matching prior loss to backpropagate into the encoder. That loss optimizes how easily the current latent can be fit by the prior solver, which is not equivalent to the reconstructability or generability of the discrete semantics. Below, we first analyze the potential conflict between the two gradients, and then discuss representation drift under fixed capacity and local decoder geometry. These analyses do not require compression to necessarily shrink the valid region, nor do they constitute a general guarantee of training collapse.
Joint Objective and Gradient Conflict
Collect the Stage 1 objectives that shape the representation into
L r e p ( ϕ , ψ ) = L b a l + λ g e n L g e n + λ d u r L d u r + λ K L L K L , (55) \mathcal{L}_{\mathrm{rep}}(\phi,\psi)=\mathcal{L}_{\mathrm{bal}}+\lambda_{\mathrm{gen}}\mathcal{L}_{\mathrm{gen}}+\lambda_{\mathrm{dur}}\mathcal{L}_{\mathrm{dur}}+\lambda_{\mathrm{KL}}\mathcal{L}_{\mathrm{KL}}, \tag*{(55)} L rep ( ϕ , ψ ) = L bal + λ gen L gen + λ dur L dur + λ KL L KL , (55)
where ϕ \phi ϕ and ψ \psi ψ denote the encoder and decoder parameters, respectively; both L b a l \mathcal{L}_{\mathrm{bal}} L bal and L g e n \mathcal{L}_{\mathrm{gen}} L gen update the encoder through the token-prediction path of the discrete generator decoder. If the gradient of the DiT Flow Matching loss is fully allowed to reach the encoder, the encoder effectively optimizes
L j o i n t = L r e p + λ F M L F M . (56) \mathcal{L}_{\mathrm{joint}}=\mathcal{L}_{\mathrm{rep}}+\lambda_{\mathrm{FM}}\mathcal{L}_{\mathrm{FM}}. \tag*{(56)} L joint = L rep + λ FM L FM . (56)
Define
g r e p = ∇ ϕ L r e p , g F M = ∇ ϕ L F M . (57) g_{\mathrm{rep}}=\nabla_{\phi}\mathcal{L}_{\mathrm{rep}},\qquad g_{\mathrm{FM}}=\nabla_{\phi}\mathcal{L}_{\mathrm{FM}}. \tag*{(57)} g rep = ∇ ϕ L rep , g FM = ∇ ϕ L FM . (57)
A single SGD update is
ϕ + = ϕ − η ( g r e p + λ F M g F M ) . (58) \phi^{+}=\phi-\eta\left(g_{\mathrm{rep}}+\lambda_{\mathrm{FM}}g_{\mathrm{FM}}\right). \tag*{(58)} ϕ + = ϕ − η ( g rep + λ FM g FM ) . (58)
A first-order expansion of L r e p \mathcal{L}_{\mathrm{rep}} L rep gives
Δ L r e p ≈ − η ∥ g r e p ∥ 2 2 − η λ F M ⟨ g r e p , g F M ⟩ . (59) \Delta\mathcal{L}_{\mathrm{rep}}\approx-\eta\lVert g_{\mathrm{rep}}\rVert_{2}^{2}-\eta\lambda_{\mathrm{FM}}\langle g_{\mathrm{rep}},g_{\mathrm{FM}}\rangle. \tag*{(59)} Δ L rep ≈ − η ∥ g rep ∥ 2 2 − η λ FM ⟨ g rep , g FM ⟩ . (59)
Therefore, whenever
⟨ g r e p , g F M ⟩ < − ∥ g r e p ∥ 2 2 λ F M , (60) \langle g_{\mathrm{rep}},g_{\mathrm{FM}}\rangle<-\frac{\lVert g_{\mathrm{rep}}\rVert_{2}^{2}}{\lambda_{\mathrm{FM}}}, \tag*{(60)} ⟨ g rep , g FM ⟩ < − λ FM ∥ g rep ∥ 2 2 , (60)
the DiT loss can decrease while the representation objective locally increases. This is the most basic condition for unstable joint training: the two gradients are not inherently aligned.
The DiT-to-Encoder Gradient Is Not a Semantic Gradient
Let the encoder produce the clean latent via reparameterization,
z 0 = g ϕ ( x , ϵ 0 ) , ϵ 0 ∼ N ( 0 , I ) . (61) z_{0}=g_{\phi}(x,\epsilon_{0}),\qquad\epsilon_{0}\sim\mathcal{N}(0,I). \tag*{(61)} z 0 = g ϕ ( x , ϵ 0 ) , ϵ 0 ∼ N ( 0 , I ) . (61)
A general v-prediction / Flow Matching process can be written as
z t = α t z 0 + σ t ϵ , v t ∗ = a t z 0 + b t ϵ , ϵ ∼ N ( 0 , I ) , (62) z_{t}=\alpha_{t}z_{0}+\sigma_{t}\epsilon,\qquad v_{t}^{*}=a_{t}z_{0}+b_{t}\epsilon,\qquad\epsilon\sim\mathcal{N}(0,I), \tag*{(62)} z t = α t z 0 + σ t ϵ , v t ∗ = a t z 0 + b t ϵ , ϵ ∼ N ( 0 , I ) , (62)
and we define
r = v θ ( z t , t , c ) − v t ∗ , ℓ v = ∥ r ∥ 2 2 , (63) r=v_{\theta}(z_{t},t,c)-v_{t}^{*},\qquad\ell_{v}=\lVert r\rVert_{2}^{2}, \tag*{(63)} r = v θ ( z t , t , c ) − v t ∗ , ℓ v = ∥ r ∥ 2 2 , (63)
where c c c denotes conditions such as previous latent blocks or the prompt representation. If c c c is detached from z 0 z_0 z 0 , the chain rule gives
∇ z 0 ℓ v = 2 ( α t J v , z t T − a t I ) r , (64) \nabla_{z_0}\ell_v = 2(\alpha_t J_{v,z_t}^{\mathsf T} - a_t I)r, \tag*{(64)} ∇ z 0 ℓ v = 2 ( α t J v , z t T − a t I ) r , (64)
where J v , z t = ∂ v θ / ∂ z t J_{v,z_t} = \partial v_\theta/\partial z_t J v , z t = ∂ v θ / ∂ z t . If the condition path also depends on z 0 z_0 z 0 and is not detached, an additional term appears:
2 J c , z 0 T J v , c T r . (65) 2J_{c,z_0}^{\mathsf T}J_{v,c}^{\mathsf T}r. \tag*{(65)} 2 J c , z 0 T J v , c T r . (65)
Finally,
∇ ϕ ℓ v = ( ∇ z 0 ℓ v ) T ∂ z 0 ∂ ϕ . (66) \nabla_{\phi}\ell_v = (\nabla_{z_0}\ell_v)^{\mathsf T}\frac{\partial z_0}{\partial\phi}. \tag*{(66)} ∇ ϕ ℓ v = ( ∇ z 0 ℓ v ) T ∂ ϕ ∂ z 0 . (66)
These gradients are determined jointly by the denoising residual, the DiT Jacobian, the condition path, and the noise schedule; they contain neither a reconstruction likelihood nor a semantic objective. Therefore, g F M g_{\mathrm{FM}} g FM serves to make the current latent representation easier for the prior solver to fit, rather than to guarantee that the decodable semantics of the latent remain intact.
Lowering the FM Loss Admits a ``Simpler Latent'' Shortcut
Let W = ( z t , t , c ) W=(z_t,t,c) W = ( z t , t , c ) and let the target velocity be u t u_t u t . For any vector field v ( W ) v(W) v ( W ) , the squared loss admits the conditional-expectation decomposition
E ∥ v ( W ) − u t ∥ 2 2 = E ∥ v ( W ) − E [ u t ∣ W ] ∥ 2 2 + E ∥ u t − E [ u t ∣ W ] ∥ 2 2 . (67) \mathbb{E}\lVert v(W)-u_t\rVert_2^2
=
\mathbb{E}\lVert v(W)-\mathbb{E}[u_t\mid W]\rVert_2^2
+
\mathbb{E}\lVert u_t-\mathbb{E}[u_t\mid W]\rVert_2^2. \tag*{(67)} E ∥ v ( W ) − u t ∥ 2 2 = E ∥ v ( W ) − E [ u t ∣ W ] ∥ 2 2 + E ∥ u t − E [ u t ∣ W ] ∥ 2 2 . (67)
Hence
v ⋆ ( W ) = E [ u t ∣ W ] , L F M = E [ Var ( u t ∣ z t , t , c ) ] . (68) v^\star(W)=\mathbb{E}[u_t\mid W],\qquad
\mathcal{L}_{\mathrm{FM}}=\mathbb{E}[\operatorname{Var}(u_t\mid z_t,t,c)]. \tag*{(68)} v ⋆ ( W ) = E [ u t ∣ W ] , L FM = E [ Var ( u t ∣ z t , t , c )] . (68)
Eq. (C.14) shows that, when the encoder is also trainable, the FM loss can decrease in two ways: the DiT can better approximate the conditional mean, or the encoder can actively alter the latent geometry so that the conditional variance of the target velocity itself decreases. The latter path directly flattens semantically meaningful modes.
As a simple counterexample, let the semantic variable S ∈ { − 1 , + 1 } S\in\{-1,+1\} S ∈ { − 1 , + 1 } be equiprobable and construct
q s e m ( z ∣ S ) = N ( m S e 1 , I ) , m > 0 , q c o l ( z ∣ S ) = N ( 0 , I ) . (69) q_{\mathrm{sem}}(z\mid S)=\mathcal{N}(mS e_1,I),\qquad m>0,\qquad
q_{\mathrm{col}}(z\mid S)=\mathcal{N}(0,I). \tag*{(69)} q sem ( z ∣ S ) = N ( m S e 1 , I ) , m > 0 , q col ( z ∣ S ) = N ( 0 , I ) . (69)
The former satisfies I ( S ; Z s e m ) > 0 I(S;Z_{\mathrm{sem}})>0 I ( S ; Z sem ) > 0 , whereas the latter satisfies I ( S ; Z c o l ) = 0 I(S;Z_{\mathrm{col}})=0 I ( S ; Z col ) = 0 . However, in high-noise or weak-conditioning regions, transporting the bimodal semantic distribution is more complex, while the collapsed distribution is closer to the base Gaussian. It is therefore possible that
L F M ( q c o l ) < L F M ( q s e m ) , I ( S ; Z c o l ) < I ( S ; Z s e m ) . (70) \mathcal{L}_{\mathrm{FM}}(q_{\mathrm{col}})<\mathcal{L}_{\mathrm{FM}}(q_{\mathrm{sem}}),\qquad
I(S;Z_{\mathrm{col}})<I(S;Z_{\mathrm{sem}}). \tag*{(70)} L FM ( q col ) < L FM ( q sem ) , I ( S ; Z col ) < I ( S ; Z sem ) . (70)
Consequently, a lower FM loss alone cannot rule out semantic collapse; under unrestricted joint training, collapse may even become an easier optimization direction.
Information Constraints and Local Decoder Geometry under Compression
In this subsection, n n n denotes the number of tokens covered by a single latent Z Z Z , to distinguish it from the latent sequence length K K K used elsewhere. Given the data distribution and context C C C , define the minimum conditional information required to reach a total reconstruction-loss threshold ε \varepsilon ε as
R n ⋆ ( ε ) = inf q ( z ∣ X 1 : n , C ) { I ( X 1 : n ; Z ∣ C ) : E p ψ [ − log p ψ ( X 1 : n ∣ Z , C ) ] ≤ ε } . (71) R_n^\star(\varepsilon)
=
\inf_{q(z\mid X_{1:n},C)}
\left\{
I(X_{1:n};Z\mid C):
\mathbb{E}_{p_\psi}\left[-\log p_\psi(X_{1:n}\mid Z,C)\right]\le\varepsilon
\right\}. \tag*{(71)} R n ⋆ ( ε ) = q ( z ∣ X 1 : n , C ) inf { I ( X 1 : n ; Z ∣ C ) : E p ψ [ − log p ψ ( X 1 : n ∣ Z , C ) ] ≤ ε } . (71)
If the effective conditional information capacity of this latent is C n C_n C n , the information slack is
Δ n = C n − R n ⋆ ( ε ) . (72) \Delta_n=C_n-R_n^\star(\varepsilon). \tag*{(72)} Δ n = C n − R n ⋆ ( ε ) . (72)
If additional tokens increase the required predictive information while C n C_n C n does not increase accordingly, the slack may decrease. This depends on the data, the context, and the distortion criterion, and does not imply an unconditional monotonic relation among separately retrained models with different compression rates.
To illustrate the geometric constraint, fix the decoder, the target span x 1 : n x_{1:n} x 1 : n , and the context c c c . Under the causal factorization, let p i ( z ) = p ψ ( x i ∣ x < i , z , c ) p_i(z) = p_\psi(x_i \mid x_{<i}, z, c) p i ( z ) = p ψ ( x i ∣ x < i , z , c ) , and let τ , τ i ∈ ( 0 , 1 ] \tau, \tau_i \in(0,1] τ , τ i ∈ ( 0 , 1 ] denote probability thresholds. Define
A 1 : n ( τ ) = { z : ∏ i = 1 n p i ( z ) ≥ τ } , A i ( τ i ) = { z : p i ( z ) ≥ τ i } . (73) A_{1:n}(\tau) = \left\{z : \prod_{i=1}^{n} p_i(z) \ge\tau\right\}, \qquad A_i(\tau_i) = \{z : p_i(z) \ge\tau_i\}. \tag*{(73)} A 1 : n ( τ ) = { z : i = 1 ∏ n p i ( z ) ≥ τ } , A i ( τ i ) = { z : p i ( z ) ≥ τ i } . (73)
Satisfying every per-token threshold is sufficient for satisfying the joint threshold, so
B n : = ⋂ i = 1 n A i ( τ i ) ⊆ A 1 : n ( ∏ i = 1 n τ i ) . (74) B_n := \bigcap_{i=1}^{n} A_i(\tau_i) \subseteq A_{1:n}\left(\prod_{i=1}^{n} \tau_i\right). \tag*{(74)} B n := i = 1 ⋂ n A i ( τ i ) ⊆ A 1 : n ( i = 1 ∏ n τ i ) . (74)
With the existing p i p_i p i and thresholds unchanged, adding one token constraint gives
B n + 1 = B n ∩ A n + 1 ( τ n + 1 ) ⊆ B n . (75) B_{n+1} = B_n \cap A_{n+1}(\tau_{n+1}) \subseteq B_n. \tag*{(75)} B n + 1 = B n ∩ A n + 1 ( τ n + 1 ) ⊆ B n . (75)
This only states that the intersection of fixed constraints does not grow; it does not guarantee strict shrinkage, nor does it directly compare relearned latent geometries.
To exclude the trivial scale growth caused by summing losses, we use the per-token average loss below. With ℓ i ( z ) = − log p i ( z ) \ell_i(z) = -\log p_i(z) ℓ i ( z ) = − log p i ( z ) ,
L d e c ( n ) ( z ) = 1 n ∑ i = 1 n ℓ i ( z ) . (76) \mathcal{L}_{\mathrm{dec}}^{(n)}(z) = \frac{1}{n}\sum_{i=1}^{n}\ell_i(z). \tag*{(76)} L dec ( n ) ( z ) = n 1 i = 1 ∑ n ℓ i ( z ) . (76)
Its local Hessian is
H n ( z ) = ∇ z 2 L d e c ( n ) ( z ) = 1 n ∑ i = 1 n ∇ z 2 ℓ i ( z ) . (77) H_n(z) = \nabla_z^2\mathcal{L}_{\mathrm{dec}}^{(n)}(z) = \frac{1}{n}\sum_{i=1}^{n}\nabla_z^2\ell_i(z). \tag*{(77)} H n ( z ) = ∇ z 2 L dec ( n ) ( z ) = n 1 i = 1 ∑ n ∇ z 2 ℓ i ( z ) . (77)
For MDM, the same local analysis can take ℓ i \ell_i ℓ i as the token denoising loss under a fixed masked state, without treating the product as the full MDM sequence likelihood. If the loss is twice continuously differentiable near z z z , then for a latent drift δ \delta δ ,
L d e c ( n ) ( z + δ ) − L d e c ( n ) ( z ) = ∇ L d e c ( n ) ( z ) T δ + 1 2 δ T H n ( z ) δ + o ( ∥ δ ∥ 2 2 ) . (78) \mathcal{L}_{\mathrm{dec}}^{(n)}(z+\delta)-\mathcal{L}_{\mathrm{dec}}^{(n)}(z)
= \nabla\mathcal{L}_{\mathrm{dec}}^{(n)}(z)^{\mathsf{T}}\delta
+ \frac{1}{2}\delta^{\mathsf{T}}H_n(z)\delta
+ o(\lVert\delta\rVert_2^2). \tag*{(78)} L dec ( n ) ( z + δ ) − L dec ( n ) ( z ) = ∇ L dec ( n ) ( z ) T δ + 2 1 δ T H n ( z ) δ + o (∥ δ ∥ 2 2 ) . (78)
Only when z z z is close to a stationary point, the first-order and higher-order terms are negligible, and ∥ H n ( z ) ∥ 2 > 0 \lVert H_n(z)\rVert_2 > 0 ∥ H n ( z ) ∥ 2 > 0 , the local quadratic perturbation scale corresponding to an allowed average-loss increase ϵ d > 0 \epsilon_d > 0 ϵ d > 0 is approximately
r n q u a d ( ϵ d ) ≃ 2 ϵ d ∥ H n ( z ) ∥ 2 . (79) r_n^{\mathrm{quad}}(\epsilon_d) \simeq\sqrt{\frac{2\epsilon_d}{\lVert H_n(z)\rVert_2}}. \tag*{(79)} r n quad ( ϵ d ) ≃ ∥ H n ( z ) ∥ 2 2 ϵ d . (79)
This scale is neither the boundary of the actual valid region nor a guaranteed safe radius. More tokens need not increase the spectral norm of the average Hessian, and the actual loss change also depends on the drift direction. Only when the additional constraints increase the local curvature along the relevant directions can the decoder become more sensitive to drift along those directions.
Prior-Induced Local Drift and Gradient Decoupling
Considering only the DiT gradient component acting on the encoder, one parameter update is
δ ϕ F M = − η λ F M g F M . (80) \delta\phi_{\mathrm{FM}} = -\eta\lambda_{\mathrm{FM}}g_{\mathrm{FM}}. \tag*{(80)} δ ϕ FM = − η λ FM g FM . (80)
Let J z , ϕ = ∂ z 0 / ∂ ϕ J_{z,\phi} = \partial z_0/\partial\phi J z , ϕ = ∂ z 0 / ∂ ϕ be the encoder Jacobian. Taking g F M = ∇ ϕ ℓ v g_{\mathrm{FM}} = \nabla_\phi\ell_v g FM = ∇ ϕ ℓ v for the single-sample FM term, the induced first-order latent drift is
δ z F M ≈ J z , ϕ δ ϕ F M = − η λ F M J z , ϕ J z , ϕ T ∇ z 0 ℓ v . (81) \begin{aligned}
\delta z_{\mathrm{FM}} &\approx J_{z,\phi}\delta\phi_{\mathrm{FM}} \\
&= -\eta\lambda_{\mathrm{FM}}J_{z,\phi}J_{z,\phi}^{\mathsf{T}}\nabla_{z_0}\ell_v. \tag*{(81)}
\end{aligned} δ z FM ≈ J z , ϕ δ ϕ FM = − η λ FM J z , ϕ J z , ϕ T ∇ z 0 ℓ v . (81)
For batch updates, g F M g_{\mathrm{FM}} g FM should be replaced by the corresponding aggregated parameter gradient. Fixing the current decoder and setting z = z 0 z = z_0 z = z 0 , Eq. (C.24) gives
Δ L d e c F M : = L d e c ( n ) ( z + δ z F M ) − L d e c ( n ) ( z ) ≈ ∇ L d e c ( n ) ( z ) T δ z F M + 1 2 δ z F M T H n ( z ) δ z F M . (82) \begin{aligned}
\Delta\mathcal{L}_{\mathrm{dec}}^{\mathrm{FM}} &:= \mathcal{L}_{\mathrm{dec}}^{(n)}(z+\delta z_{\mathrm{FM}})-\mathcal{L}_{\mathrm{dec}}^{(n)}(z) \\
&\approx\nabla\mathcal{L}_{\mathrm{dec}}^{(n)}(z)^{\mathsf{T}}\delta z_{\mathrm{FM}}+\frac{1}{2}\delta z_{\mathrm{FM}}^{\mathsf{T}}H_n(z)\delta z_{\mathrm{FM}}. \tag*{(82)}
\end{aligned} Δ L dec FM := L dec ( n ) ( z + δ z FM ) − L dec ( n ) ( z ) ≈ ∇ L dec ( n ) ( z ) T δ z FM + 2 1 δ z FM T H n ( z ) δ z FM . (82)
Hence, the FM-induced drift may increase the token loss along sensitive directions of the decoder, with an effect that depends on the encoder Jacobian, the direction and magnitude of the drift, and the simultaneous adaptation of the decoder. This analysis characterizes the local risk of prior-induced representation drift; it does not establish a general relation between the compression rate and training collapse.
Full detachment directly removes the prior-solver gradient acting on the encoder:
∇ ϕ L j o i n t → full detach ∇ ϕ L r e p , ∇ θ L j o i n t = λ F M ∇ θ L F M . (83) \nabla_\phi\mathcal{L}_{\mathrm{joint}} \xrightarrow{\text{full detach}} \nabla_\phi\mathcal{L}_{\mathrm{rep}}, \qquad\nabla_\theta\mathcal{L}_{\mathrm{joint}} = \lambda_{\mathrm{FM}}\nabla_\theta\mathcal{L}_{\mathrm{FM}}. \tag*{(83)} ∇ ϕ L joint full detach ∇ ϕ L rep , ∇ θ L joint = λ FM ∇ θ L FM . (83)
The VAE still learns under its own objectives, and the DiT still fits the aggregated posterior using samples from the current encoder. This operation guarantees that the path by which the prior loss directly rewrites the encoder is blocked, but does not guarantee that the full training process is necessarily stable.
Joint Information-Theoretic Interpretation of VAE and DiT Noise
This appendix proves Lemma 4.3 . To analyze the relation between the Stage 1 latent noise and the Stage 2 diffusion noise, we write the clean latent in the simplified form
Z = G + N 0 , (84) Z = G + N_0, \tag*{(84)} Z = G + N 0 , (84)
where G G G denotes the signal associated with high-level semantics and N 0 N_0 N 0 denotes the VAE posterior noise. The DiT further constructs
Z t = α t Z + σ t ϵ , ϵ ∼ N ( 0 , I ) . (85) Z_t = \alpha_t Z + \sigma_t\epsilon, \qquad\epsilon\sim\mathcal{N}(0,I). \tag*{(85)} Z t = α t Z + σ t ϵ , ϵ ∼ N ( 0 , I ) . (85)
Under a Gaussian approximation, assume that G ∼ N ( 0 , Σ G ) G \sim\mathcal{N}(0,\Sigma_G) G ∼ N ( 0 , Σ G ) and N 0 ∼ N ( 0 , Σ 0 ) N_0 \sim\mathcal{N}(0,\Sigma_0) N 0 ∼ N ( 0 , Σ 0 ) are independent. Conditioned on G G G , the effective noise covariance of Z t Z_t Z t is α t 2 Σ 0 + σ t 2 I \alpha_t^2\Sigma_0+\sigma_t^2I α t 2 Σ 0 + σ t 2 I , and therefore
I ( G ; Z t ) = 1 2 log det [ I + α t 2 Σ G ( α t 2 Σ 0 + σ t 2 I ) − 1 ] . (86) I(G;Z_t) = \frac{1}{2}\log\det\left[I+\alpha_t^2\Sigma_G\left(\alpha_t^2\Sigma_0+\sigma_t^2I\right)^{-1}\right]. \tag*{(86)} I ( G ; Z t ) = 2 1 log det [ I + α t 2 Σ G ( α t 2 Σ 0 + σ t 2 I ) − 1 ] . (86)
Defining λ t = log ( α t 2 / σ t 2 ) \lambda_t = \log(\alpha_t^2/\sigma_t^2) λ t = log ( α t 2 / σ t 2 ) , the same expression can equivalently be written as
I ( G ; Z t ) = 1 2 log det [ I + Σ G ( Σ 0 + e − λ t I ) − 1 ] . (87) I(G;Z_t) = \frac{1}{2}\log\det\left[I+\Sigma_G\left(\Sigma_0+e^{-\lambda_t}I\right)^{-1}\right]. \tag*{(87)} I ( G ; Z t ) = 2 1 log det [ I + Σ G ( Σ 0 + e − λ t I ) − 1 ] . (87)
Eq. (D.4) expresses a central fact: the VAE posterior noise and the DiT corruption jointly determine how discriminable the latent semantic signal is. Applying the same DiT noise schedule to VAE latents of different sharpness therefore corresponds to different semantic-information regimes. We accordingly treat the two noise sources as quantities that must be calibrated jointly, rather than claiming that any fixed logSNR or timestep is a model-independent theoretical optimum.
Reconstruction–Generation Balance in the Discrete Generator Decoder
This appendix proves Lemma 4.4 .
Let Z A Z_A Z A denote the current-block latent condition actually received by the discrete generator decoder under condition mode A A A , where A ∈ { cond , noise , drop } A \in\{\text{cond}, \text{noise}, \text{drop}\} A ∈ { cond , noise , drop } . For each condition, Eq. (5) in the main text holds:
H ( Y ∣ C ) = I ( Y ; Z A ∣ C ) + H ( Y ∣ C , Z A ) . (88) H(Y \mid C) = I(Y; Z_A \mid C) + H(Y \mid C, Z_A). \tag*{(88)} H ( Y ∣ C ) = I ( Y ; Z A ∣ C ) + H ( Y ∣ C , Z A ) . (88)
The cond state uses the complete clean latent and therefore typically provides the most informative condition for the current block. The drop state zeros the latent condition of the current block, which corresponds to masked token prediction that does not depend on that latent. The noise state applies a small random perturbation to the clean latent to train the decoder’s robustness to imprecise conditions. Together, the three states cover three training regimes: strongly conditioned reconstruction, local robustness, and weakly conditioned generation.
For any discrete generator decoder p ψ ( Y ∣ C , Z A ) p_\psi(Y \mid C, Z_A) p ψ ( Y ∣ C , Z A ) , the cross-entropy decomposes exactly as
L A ( ψ ) = H ( Y ∣ C , Z A ) + E KL ( q ( Y ∣ C , Z A ) ∥ p ψ ( Y ∣ C , Z A ) ) . (89) \mathcal{L}_A(\psi) = H(Y \mid C, Z_A)
+ \mathbb{E}\operatorname{KL}(q(Y \mid C, Z_A)\Vert p_\psi(Y \mid C, Z_A)). \tag*{(89)} L A ( ψ ) = H ( Y ∣ C , Z A ) + E KL ( q ( Y ∣ C , Z A ) ∥ p ψ ( Y ∣ C , Z A )) . (89)
The first term is determined by how much uncertainty remains under the given condition; the second is the modeling error of the finite-capacity discrete generator decoder relative to the optimal conditional distribution. Training only under cond mainly optimizes the strongly conditioned region and may make the model overly dependent on precise latents, whereas training only under drop may cause the discrete generator decoder to ignore the latent. Mixing cond , noise , and drop trains the same decoder parameters to reduce the modeling error across different levels of condition reliability. The optimal mixture depends on model capacity, training budget, and the distribution of latent errors at inference, and should therefore be determined empirically.
Local Perturbation Path of the Noise Condition
The noise condition uses the same linear interpolation as the latent generator, but restricts the range of the perturbation coefficient. Let u ∈ [ 0 , u max ] u \in[0, u_{\max}] u ∈ [ 0 , u m a x ] , where u max ≪ 1 u_{\max} \ll1 u m a x ≪ 1 controls the perturbation scale; then
Z u = ( 1 − u ) Z + u ε , ε ∼ N ( 0 , I ) . (90) Z_u = (1-u)Z + u\varepsilon,\qquad\varepsilon\sim\mathcal{N}(0,I). \tag*{(90)} Z u = ( 1 − u ) Z + u ε , ε ∼ N ( 0 , I ) . (90)
For a single d z d_z d z -dimensional latent, let Δ Z u = Z u − Z \Delta Z_u = Z_u - Z Δ Z u = Z u − Z . Independent standard Gaussian noise gives
E [ ∥ Δ Z u ∥ 2 2 ∣ Z , u ] = u 2 ( d z + ∥ Z ∥ 2 2 ) . (91) \mathbb{E}[\lVert\Delta Z_u\rVert_2^2 \mid Z,u] = u^2(d_z + \lVert Z\rVert_2^2). \tag*{(91)} E [∥ Δ Z u ∥ 2 2 ∣ Z , u ] = u 2 ( d z + ∥ Z ∥ 2 2 ) . (91)
A small u u u controls the mean-squared perturbation scale, but Gaussian noise does not have finite support. For each u u u , the decoder receives information I ( Y ; Z u ∣ C ) I(Y; Z_u \mid C) I ( Y ; Z u ∣ C ) ; if a stronger perturbation constitutes a more degraded stochastic channel, this mutual information does not increase, and the corresponding H ( Y ∣ C , Z u ) H(Y \mid C, Z_u) H ( Y ∣ C , Z u ) does not decrease. The noise branch optimizes the expected token loss under this perturbation distribution to improve adaptation to imprecise latents; it neither assumes that the artificial perturbation shares the distribution of DiT output errors, nor guarantees uniform robustness to all inference-time deviations.
Probabilistic Derivation of Classifier-Free Guidance
The single-axis guidance of the latent DiT and each axis of the two-axis MDM guidance are based on the linear operator
f γ = f u + γ ( f c − f u ) , (92) f_\gamma= f_u + \gamma(f_c - f_u), \tag*{(92)} f γ = f u + γ ( f c − f u ) , (92)
where f c f_c f c and f u f_u f u are the branch outputs with and without the selected condition, and γ \gamma γ is the guidance scale; the latent generator applies this operator to velocities, whereas the MDM decoder combines logits along the prefix and latent axes. Below, we give a per-noise-time interpretation under exact scores and optimal FM regression fields, and derive the token-probability reweighting under a fixed decoding state; neither automatically yields a power-mixture guarantee for the complete generated distribution.
From Bayes' Rule to the Guided Score
From Bayes’ Rule to the Guided Score
First consider a continuous variable X u X_u X u with a differentiable positive density, where u u u is the noise time and x x x its value. Let K \mathcal{K} K denote the context shared by the conditional and unconditional branches, let A A A denote the additional condition dropped in CFG, and define
s c ( x , u ) = ∇ x log p u ( x ∣ K , A ) , s u ( x , u ) = ∇ x log p u ( x ∣ K ) . (93) \begin{aligned}
s_c(x,u) &= \nabla_x \log p_u(x \mid\mathcal{K}, A),\\
s_u(x,u) &= \nabla_x \log p_u(x \mid\mathcal{K}). \tag*{(93)}
\end{aligned} s c ( x , u ) s u ( x , u ) = ∇ x log p u ( x ∣ K , A ) , = ∇ x log p u ( x ∣ K ) . (93)
By Bayes’ rule,
p u ( x ∣ K , A ) = p u ( A ∣ x , K ) p u ( x ∣ K ) p ( A ∣ K ) , (94) p_u(x \mid\mathcal{K}, A) = \frac{p_u(A \mid x,\mathcal{K})p_u(x \mid\mathcal{K})}{p(A \mid\mathcal{K})}, \tag*{(94)} p u ( x ∣ K , A ) = p ( A ∣ K ) p u ( A ∣ x , K ) p u ( x ∣ K ) , (94)
and taking the gradient with respect to x x x yields
s c ( x , u ) − s u ( x , u ) = ∇ x log p u ( A ∣ x , K ) . (95) s_c(x,u) - s_u(x,u) = \nabla_x \log p_u(A \mid x,\mathcal{K}). \tag*{(95)} s c ( x , u ) − s u ( x , u ) = ∇ x log p u ( A ∣ x , K ) . (95)
The standard CFG score therefore satisfies
s γ = s u + γ ( s c − s u ) = ∇ x log p u ( x ∣ K ) + γ ∇ x log p u ( A ∣ x , K ) = ∇ x log p ~ γ , u ( x ∣ K , A ) , (96) \begin{aligned}
s_\gamma&= s_u + \gamma(s_c - s_u)\\
&= \nabla_x \log p_u(x \mid\mathcal{K}) + \gamma\nabla_x \log p_u(A \mid x,\mathcal{K})\\
&= \nabla_x \log\widetilde{p}_{\gamma,u}(x \mid\mathcal{K}, A), \tag*{(96)}
\end{aligned} s γ = s u + γ ( s c − s u ) = ∇ x log p u ( x ∣ K ) + γ ∇ x log p u ( A ∣ x , K ) = ∇ x log p γ , u ( x ∣ K , A ) , (96)
where the unnormalized guided distribution is
p ~ γ , u ( x ∣ K , A ) ∝ p u ( x ∣ K ) p u ( A ∣ x , K ) γ ∝ p u ( x ∣ K , A ) γ p u ( x ∣ K ) 1 − γ . (97) \begin{aligned}
\widetilde{p}_{\gamma,u}(x \mid\mathcal{K}, A) &\propto p_u(x \mid\mathcal{K})p_u(A \mid x,\mathcal{K})^\gamma\\
&\propto p_u(x \mid\mathcal{K}, A)^\gamma p_u(x \mid\mathcal{K})^{1-\gamma}. \tag*{(97)}
\end{aligned} p γ , u ( x ∣ K , A ) ∝ p u ( x ∣ K ) p u ( A ∣ x , K ) γ ∝ p u ( x ∣ K , A ) γ p u ( x ∣ K ) 1 − γ . (97)
When the normalizing constant is finite, it is independent of x x x and does not affect the score. For a fixed u u u , in Eq. (F.6), γ = 0 \gamma= 0 γ = 0 recovers the base density at that time, γ = 1 \gamma= 1 γ = 1 recovers the conditional density, 0 < γ < 1 0 < \gamma< 1 0 < γ < 1 attenuates the conditional likelihood ratio, and γ > 1 \gamma> 1 γ > 1 amplifies it. This equivalence describes the instantaneous score field rather than the final sampling distribution.
Equivalence of Flow Matching Velocity CFG and Score CFG
The latent generator of Cola2 predicts velocities rather than scores, so it remains to explain why applying the same linear combination to velocities, as in Eq. (36) , is still valid. Consider a general Gaussian corruption path
X u = α u X 0 + σ u ϵ , ϵ ∼ N ( 0 , I ) , (98) X_u = \alpha_u X_0 + \sigma_u \epsilon,\qquad\epsilon\sim\mathcal{N}(0,I), \tag*{(98)} X u = α u X 0 + σ u ϵ , ϵ ∼ N ( 0 , I ) , (98)
where α u \alpha_u α u and σ u \sigma_u σ u are differentiable path coefficients and dots denote derivatives with respect to the noise time u u u . The following identities apply at interior times with α u ≠ 0 \alpha_u \ne0 α u = 0 and σ u > 0 \sigma_u > 0 σ u > 0 . For an arbitrary condition Q Q Q , write
s Q ( x , u ) = ∇ x log p u ( x ∣ Q ) . (99) s_Q(x,u) = \nabla_x \log p_u(x \mid Q). \tag*{(99)} s Q ( x , u ) = ∇ x log p u ( x ∣ Q ) . (99)
The score identity for Gaussian convolution gives
E [ ϵ ∣ X u = x , Q ] = − σ u s Q ( x , u ) . (100) \mathbb{E}[\epsilon\mid X_u = x,Q] = -\sigma_u s_Q(x,u). \tag*{(100)} E [ ϵ ∣ X u = x , Q ] = − σ u s Q ( x , u ) . (100)
On the other hand, since x = α u X 0 + σ u ϵ x = \alpha_u X_0 + \sigma_u\epsilon x = α u X 0 + σ u ϵ ,
E [ X 0 ∣ X u = x , Q ] = x + σ u 2 s Q ( x , u ) α u . (101) \mathbb{E}[X_0 \mid X_u = x,Q] = \frac{x + \sigma_u^2 s_Q(x,u)}{\alpha_u}. \tag*{(101)} E [ X 0 ∣ X u = x , Q ] = α u x + σ u 2 s Q ( x , u ) . (101)
The optimal Flow Matching regression target is the conditional expected path velocity, so
v Q ( x , u ) = E [ α ˙ u X 0 + σ ˙ u ϵ ∣ X u = x , Q ] = α ˙ u α u x + σ u ( α ˙ u σ u α u − σ ˙ u ) s Q ( x , u ) . (102) \begin{aligned}
v_Q(x,u) &= \mathbb{E}[\dot{\alpha}_u X_0 + \dot{\sigma}_u \epsilon\mid X_u = x,Q]\\
&= \frac{\dot{\alpha}_u}{\alpha_u}x + \sigma_u\left(\frac{\dot{\alpha}_u\sigma_u}{\alpha_u} - \dot{\sigma}_u\right)s_Q(x,u). \tag*{(102)}
\end{aligned} v Q ( x , u ) = E [ α ˙ u X 0 + σ ˙ u ϵ ∣ X u = x , Q ] = α u α ˙ u x + σ u ( α u α ˙ u σ u − σ ˙ u ) s Q ( x , u ) . (102)
In other words, at a fixed u u u , the velocity and the score are related by an affine transformation shared by all conditions,
v Q ( x , u ) = a u x + b u s Q ( x , u ) (103) v_Q(x,u)=a_u x+b_u s_Q(x,u) \tag*{(103)} v Q ( x , u ) = a u x + b u s Q ( x , u ) (103)
where a u = α ˙ u / α u a_u=\dot{\alpha}_u/\alpha_u a u = α ˙ u / α u and b u = σ u ( α ˙ u σ u / α u − σ ˙ u ) b_u=\sigma_u(\dot{\alpha}_u\sigma_u/\alpha_u-\dot{\sigma}_u) b u = σ u ( α ˙ u σ u / α u − σ ˙ u ) . When the conditional and unconditional branches are subtracted, the shared term a u x a_u x a u x cancels exactly, so
v u + γ g e n ( v c − v u ) = a u x + b u [ s u + γ g e n ( s c − s u ) ] = a u x + b u s γ . (104) \begin{aligned}
v_u+\gamma_{\mathrm{gen}}(v_c-v_u)&=a_u x+b_u\left[s_u+\gamma_{\mathrm{gen}}(s_c-s_u)\right]\\
&=a_u x+b_u s_\gamma. \tag*{(104)}
\end{aligned} v u + γ gen ( v c − v u ) = a u x + b u [ s u + γ gen ( s c − s u ) ] = a u x + b u s γ . (104)
Therefore, under a shared Gaussian path and optimal branch regression fields, velocity CFG and score CFG correspond pointwise, with γ = γ g e n \gamma=\gamma_{\mathrm{gen}} γ = γ gen . For the linear path used in this work,
α u = 1 − u , σ u = u , (105) \alpha_u=1-u,\qquad\sigma_u=u, \tag*{(105)} α u = 1 − u , σ u = u , (105)
Eq. (F.11) further reduces to
v Q ( x , u ) = − x 1 − u − u 1 − u s Q ( x , u ) , 0 < u < 1 , (106) v_Q(x,u)=-\frac{x}{1-u}-\frac{u}{1-u}s_Q(x,u),\qquad0<u<1, \tag*{(106)} v Q ( x , u ) = − 1 − u x − 1 − u u s Q ( x , u ) , 0 < u < 1 , (106)
with the endpoints understood as continuous limits when these exist. In Cola2 , v c v_c v c uses the historical latent cache while v u v_u v u clears it, so γ g e n \gamma_{\mathrm{gen}} γ gen modulates the influence of the historical condition on the instantaneous field.
Let v γ = v u + γ ( v c − v u ) v_\gamma=v_u+\gamma(v_c-v_u) v γ = v u + γ ( v c − v u ) . If the normalized density in Eq. (F.6) were also the actual marginal of this flow, it would have to satisfy the continuity equation
∂ u p ~ γ , u + ∇ x ⋅ ( p ~ γ , u v γ ) = 0. (107) \partial_u\widetilde{p}_{\gamma,u}+\nabla_x\cdot\left(\widetilde{p}_{\gamma,u}v_\gamma\right)=0. \tag*{(107)} ∂ u p γ , u + ∇ x ⋅ ( p γ , u v γ ) = 0. (107)
The per-time score identity alone does not guarantee this path-consistency condition. Hence, even with exact branch fields, one cannot conclude that the endpoint of guided integration follows the data distribution reweighted by the same power.
Prefix–Latent Two-Axis CFG for MDM
For the masked_only MDM, let P \mathcal{P} P be the committed pure-token prefix, U \mathcal{U} U the token/mask state of the current block together with the fixed span layout, and H Z H^Z H Z the current latent condition refined by the condition stack. Denoting the decoder logits by ℓ ( y ∣ ⋅ ) \ell(y\mid\cdot) ℓ ( y ∣ ⋅ ) , the four branches are
ℓ 11 ( y ) = ℓ ( y ∣ U , P , H Z ) , ℓ 10 ( y ) = ℓ ( y ∣ U , P , 0 ) , ℓ 01 ( y ) = ℓ ( y ∣ U , ∅ , H Z ) , ℓ 00 ( y ) = ℓ ( y ∣ U , ∅ , 0 ) . (108) \begin{aligned}
\ell_{11}(y)&=\ell(y\mid\mathcal{U},\mathcal{P},H^Z),\qquad\ell_{10}(y)=\ell(y\mid\mathcal{U},\mathcal{P},0),\\
\ell_{01}(y)&=\ell(y\mid\mathcal{U},\varnothing,H^Z),\qquad\ell_{00}(y)=\ell(y\mid\mathcal{U},\varnothing,0). \tag*{(108)}
\end{aligned} ℓ 11 ( y ) ℓ 01 ( y ) = ℓ ( y ∣ U , P , H Z ) , ℓ 10 ( y ) = ℓ ( y ∣ U , P , 0 ) , = ℓ ( y ∣ U , ∅ , H Z ) , ℓ 00 ( y ) = ℓ ( y ∣ U , ∅ , 0 ) . (108)
The first subscript controls the visibility of the prefix token KV, and the second controls the additive injection of the current latent; 0 0 0 is a zero condition tensor rather than a learnable null embedding. All four branches share U \mathcal{U} U and the same H Z H^Z H Z , so ℓ 00 \ell_{00} ℓ 00 can still read the known tokens of the current block and is not an uninformed unconditional text model.
Using guidance scales γ p \gamma_p γ p and γ z \gamma_z γ z for the prefix and latent axes, respectively, the expansion is
ℓ γ p , γ z = γ p γ z ℓ 11 + γ p ( 1 − γ z ) ℓ 10 + ( 1 − γ p ) γ z ℓ 01 + ( 1 − γ p ) ( 1 − γ z ) ℓ 00 . (109) \ell_{\gamma_p,\gamma_z}=\gamma_p\gamma_z\ell_{11}+\gamma_p(1-\gamma_z)\ell_{10}+(1-\gamma_p)\gamma_z\ell_{01}+(1-\gamma_p)(1-\gamma_z)\ell_{00}. \tag*{(109)} ℓ γ p , γ z = γ p γ z ℓ 11 + γ p ( 1 − γ z ) ℓ 10 + ( 1 − γ p ) γ z ℓ 01 + ( 1 − γ p ) ( 1 − γ z ) ℓ 00 . (109)
This bilinear combination is independent of the nesting order of the two axes. Setting γ p = 1 \gamma_p=1 γ p = 1 modulates only the latent, γ z = 1 \gamma_z=1 γ z = 1 modulates only the prefix, and γ z = 0 \gamma_z=0 γ z = 0 does not use the current latent condition. In general, dropping nonzero terms no longer yields this two-axis rule.
Let c a b c_{ab} c ab be the coefficient of ℓ a b \ell_{ab} ℓ ab in Eq. (F.18), and p a b = softmax ( ℓ a b ) p_{ab}=\operatorname{softmax}(\ell_{ab}) p ab = softmax ( ℓ ab ) the corresponding token probability. For any candidate token y y y ,
ℓ a b ( y ) = log p a b ( y ) + κ a b , (110) \ell_{ab}(y)=\log p_{ab}(y)+\kappa_{ab}, \tag*{(110)} ℓ ab ( y ) = log p ab ( y ) + κ ab , (110)
where κ a b \kappa_{ab} κ ab is the log-partition constant of each branch and is independent of y y y . For a fixed current state,
∑ a , b ∈ { 0 , 1 } c a b = 1 , p γ p , γ z ( y ) ∝ ∏ a , b ∈ { 0 , 1 } p a b ( y ) c a b . (111) \sum_{a,b\in\{0,1\}} c_{ab}=1,\qquad p_{\gamma_p,\gamma_z}(y)\propto\prod_{a,b\in\{0,1\}}p_{ab}(y)^{c_{ab}}. \tag*{(111)} a , b ∈ { 0 , 1 } ∑ c ab = 1 , p γ p , γ z ( y ) ∝ a , b ∈ { 0 , 1 } ∏ p ab ( y ) c ab . (111)
This algebraic identity does not require the network to reach the true conditional distribution. The coefficients can be negative, so this is an affine combination of logits rather than a convex mixture of probabilities; coefficients summing to 1 also do not guarantee that the sharpness of the distribution or the confidence ranking used for demasking is preserved. Prefix-off only masks the token-cache read path and does not remove the historical information retained in H Z H^Z H Z , so these operational branches are not directly equated with four marginal conditions of the true joint distribution.
This rule acts on the token distribution of each prediction; since the states and normalizers of multi-step demasking change with the generated results, a global power mixture of the complete text distribution does not follow automatically. The key mask, position indexing, and read-only cache protocol are given in Appendix § 15 .6.1.
Implementation Details of the Overall Workflow
This appendix supplements the dynamic segmentation, the block-causal MDM decoder, the Stage 1/Stage 2 gradient routing, and the complete inference procedure with latent feedback omitted from § 4.2 . The main text retains only the core definitions needed for unified modeling; implementation details are expanded here.
Dynamic Semantic Segmentation and Compression-Rate Control
Let h t = Emb ( y t ) h_t=\operatorname{Emb}(y_t) h t = Emb ( y t ) denote the token representation before segmentation. The semantic discontinuity between adjacent positions is defined as
δ t = 1 − cos ( LN ( h t ) , LN ( h t + 1 ) ) , t = 1 , … , T − 1 , (112) \delta_t=1-\cos\left(\operatorname{LN}(h_t),\operatorname{LN}(h_{t+1})\right),\qquad t=1,\ldots,T-1, \tag*{(112)} δ t = 1 − cos ( LN ( h t ) , LN ( h t + 1 ) ) , t = 1 , … , T − 1 , (112)
and is mapped to a boundary probability by
p t = σ ( α ( δ t − τ 0 ) + b ) , boundary t = 1 [ p t > 0.5 ] , (113) p_t=\sigma\left(\alpha(\delta_t-\tau_0)+b\right),\qquad\operatorname{boundary}_t=\mathbb{1}[p_t>0.5], \tag*{(113)} p t = σ ( α ( δ t − τ 0 ) + b ) , boundary t = 1 [ p t > 0.5 ] , (113)
where α \alpha α is the slope, τ 0 \tau_0 τ 0 the threshold, and b b b a scalar bias controlling the overall boundary density. If the target compression rate is ρ ⋆ \rho^\star ρ ⋆ and the measured rate on the current global batch is ρ ^ \hat{\rho} ρ ^ , then b b b does not participate in backpropagation and is updated as
b ← clip ( b − η b ( ρ ^ − ρ ⋆ ) , − c b , c b ) , (114) b\leftarrow\operatorname{clip}\left(b-\eta_b(\hat{\rho}-\rho^\star),-c_b,c_b\right), \tag*{(114)} b ← clip ( b − η b ( ρ ^ − ρ ⋆ ) , − c b , c b ) , (114)
where η b \eta_b η b is the control step size and c b c_b c b the clipping range. The upper bound L max L_{\max} L m a x is enforced by forced cuts during a greedy scan.
Convergence of Compression-Rate Control
By Eq. (G.2), a hard boundary satisfies
p t > 1 2 ⟺ δ t > τ e f f ( b ) : = τ 0 − b α . (115) p_t>\frac{1}{2}\Longleftrightarrow\delta_t>\tau_{\mathrm{eff}}(b):=\tau_0-\frac{b}{\alpha}. \tag*{(115)} p t > 2 1 ⟺ δ t > τ eff ( b ) := τ 0 − α b . (115)
Thus, increasing b b b lowers the effective threshold and increases the boundary density; together with Eq. (G.3), the sign of b b b forms a negative-feedback loop. With the encoder and data distribution fixed, define
r ( b ) : = E [ ρ ^ ∣ b ] (116) r(b):=\mathbb{E}[\hat{\rho}\mid b] \tag*{(116)} r ( b ) := E [ ρ ^ ∣ b ] (116)
as the expected compression rate after hard thresholding and length constraints. Length-based forced cuts introduce plateaus and an attainable lower bound into r ( b ) r(b) r ( b ) , but as long as the target ρ ⋆ \rho^\star ρ ⋆ lies within the attainable range and the local rate response near the equilibrium remains monotone, the following stability result holds.
Lemma G.1 (Local Convergence of Dynamic Compression-Rate Control). Assume that there exists b ⋆ ∈ ( − c b , c b ) b^\star\in(-c_b,c_b) b ⋆ ∈ ( − c b , c b ) such that r ( b ⋆ ) = ρ ⋆ r(b^\star)=\rho^\star r ( b ⋆ ) = ρ ⋆ . If r r r is differentiable in a neighborhood I \mathcal{I} I of b ⋆ b^\star b ⋆ and satisfies
0 < m ≤ r ′ ( b ) ≤ L < ∞ , b ∈ I , (117) 0 < m \le r'(b) \le L < \infty,\qquad b \in\mathcal{I}, \tag*{(117)} 0 < m ≤ r ′ ( b ) ≤ L < ∞ , b ∈ I , (117) then for any 0 < η b < 2 / L 0 < \eta_b < 2/L 0 < η b < 2/ L , the mean-response update map
F ( b ) = Π [ − c b , c b ] [ b − η b ( r ( b ) − ρ ⋆ ) ] (118) F(b)=\Pi_{[-c_b,c_b]}\left[b-\eta_b\left(r(b)-\rho^\star\right)\right] \tag*{(118)} F ( b ) = Π [ − c b , c b ] [ b − η b ( r ( b ) − ρ ⋆ ) ] (118) is a contraction in a sufficiently small neighborhood of b ⋆ b^\star b ⋆ , with
∣ b n + 1 − b ⋆ ∣ ≤ q ∣ b n − b ⋆ ∣ , q = max { ∣ 1 − η b m ∣ , ∣ 1 − η b L ∣ } < 1. (119) |b_{n+1}-b^\star| \le q|b_n-b^\star|,\qquad q=\max\{|1-\eta_bm|,|1-\eta_bL|\}<1. \tag*{(119)} ∣ b n + 1 − b ⋆ ∣ ≤ q ∣ b n − b ⋆ ∣ , q = max { ∣1 − η b m ∣ , ∣1 − η b L ∣ } < 1. (119) Consequently, b n → b ⋆ b_n\to b^\star b n → b ⋆ and r ( b n ) → ρ ⋆ r(b_n)\to\rho^\star r ( b n ) → ρ ⋆ , with at least a geometric convergence rate O ( q n ) O(q^n) O ( q n ) .
Proof. Let e n = b n − b ⋆ e_n=b_n-b^\star e n = b n − b ⋆ . By the mean value theorem, there exists ξ n \xi_n ξ n between b n b_n b n and b ⋆ b^\star b ⋆ such that
r ( b n ) − r ( b ⋆ ) = r ′ ( ξ n ) e n . (120) r(b_n)-r(b^\star)=r'(\xi_n)e_n. \tag*{(120)} r ( b n ) − r ( b ⋆ ) = r ′ ( ξ n ) e n . (120)
Ignoring the projection, the mean-response update in Eq. (G.7) gives
e n + 1 = ( 1 − η b r ′ ( ξ n ) ) e n . (121) e_{n+1}=\left(1-\eta_b r'(\xi_n)\right)e_n. \tag*{(121)} e n + 1 = ( 1 − η b r ′ ( ξ n ) ) e n . (121)
From Eq. (G.6) and 0 < η b < 2 / L 0<\eta_b<2/L 0 < η b < 2/ L , we have ∣ 1 − η b r ′ ( ξ n ) ∣ ≤ q < 1 |1-\eta_b r'(\xi_n)|\le q<1 ∣1 − η b r ′ ( ξ n ) ∣ ≤ q < 1 . The interval projection Π [ − c b , c b ] \Pi_{[-c_b,c_b]} Π [ − c b , c b ] is non-expansive, and since b ⋆ b^\star b ⋆ lies in the interior of the interval, the projection does not enlarge the error. This yields Eq. (G.8) and completes the proof. □
In practice, with the encoder fixed, the hard rate on a finite global batch can be written as ρ ^ n = r ( b n ) + ϵ n \hat{\rho}_n=r(b_n)+\epsilon_n ρ ^ n = r ( b n ) + ϵ n , where ϵ n \epsilon_n ϵ n denotes measurement fluctuation. If the i i i -th sample contains T i T_i T i valid tokens, the quantization step of the rate is approximately 1 / ∑ i T i 1/\sum_i T_i 1/ ∑ i T i . Under local stability and controlled noise, a fixed η b \eta_b η b typically results in fluctuation around the target; using a decreasing step size with ∑ n η n = ∞ \sum_n\eta_n=\infty ∑ n η n = ∞ and ∑ n η n 2 < ∞ \sum_n\eta_n^2<\infty ∑ n η n 2 < ∞ additionally requires stochastic-approximation conditions such as zero-mean bounded-variance noise and iterate stability before convergence can be discussed.
During joint training, let ϕ n \phi_n ϕ n be the encoder parameters at the n n n -th update, r n ( b ) = E [ ρ ^ n ∣ b , ϕ n ] r_n(b)=\mathbb{E}[\hat{\rho}_n\mid b,\phi_n] r n ( b ) = E [ ρ ^ n ∣ b , ϕ n ] the actual response, and b n ⋆ b_n^\star b n ⋆ its equilibrium satisfying r n ( b n ⋆ ) = ρ ⋆ r_n(b_n^\star)=\rho^\star r n ( b n ⋆ ) = ρ ⋆ . If the response at every step has a uniform contraction factor q < 1 q<1 q < 1 over the visited region, then
∣ b n + 1 − b n + 1 ⋆ ∣ ≤ q ∣ b n − b n ⋆ ∣ + η b ∣ ϵ n ∣ + ∣ b n + 1 ⋆ − b n ⋆ ∣ , (122) |b_{n+1}-b_{n+1}^\star|\le q|b_n-b_n^\star|+\eta_b|\epsilon_n|+|b_{n+1}^\star-b_n^\star|, \tag*{(122)} ∣ b n + 1 − b n + 1 ⋆ ∣ ≤ q ∣ b n − b n ⋆ ∣ + η b ∣ ϵ n ∣ + ∣ b n + 1 ⋆ − b n ⋆ ∣ , (122)
where ϵ n = ρ ^ n − r n ( b n ) \epsilon_n=\hat{\rho}_n-r_n(b_n) ϵ n = ρ ^ n − r n ( b n ) . The last term captures the target drift caused by encoder updates. The lemma therefore analyzes local convergence under a fixed response and provides a tracking interpretation for slowly varying responses, rather than a convergence guarantee for fully non-stationary joint training.
Given the assignment map g ( t ) g(t) g ( t ) , span pooling uses block-diagonal bidirectional attention:
M i j p o o l = 1 [ g ( i ) = g ( j ) ] , r k = 1 L k ∑ t ∈ S k h ~ t , (123) M_{ij}^{\mathrm{pool}}=\mathbb{1}[g(i)=g(j)],\qquad r_k=\frac{1}{L_k}\sum_{t\in\mathcal{S}_k}\tilde{h}_t, \tag*{(123)} M ij pool = 1 [ g ( i ) = g ( j )] , r k = L k 1 t ∈ S k ∑ h ~ t , (123)
where h ~ = Attn M p o o l ( h ) \tilde{h}=\operatorname{Attn}_{M^{\mathrm{pool}}}(h) h ~ = Attn M pool ( h ) . A causal Transformer along the latent axis then produces r ˉ k \bar{r}_k r ˉ k , and the posterior parameters are
[ μ k , log σ k 2 ] = W p o s t r ˉ k . (124) [\mu_k,\log\sigma_k^2]=W_{\mathrm{post}}\bar{r}_k. \tag*{(124)} [ μ k , log σ k 2 ] = W post r ˉ k . (124)
Reparameterized sampling is used during training, and the posterior mean is used at inference.
Decoder Duration and Token-Level Conditioning
Decoder Duration and Token-Level Conditioning
Let g k = W i n z k g_k = W_{\mathrm{in}}z_k g k = W in z k . The duration head outputs a distribution over L max L_{\max} L m a x classes,
p ( L k ∣ z k ) = softmax ( MLP d u r ( g k ) ) . (125) p(L_k \mid z_k) = \operatorname{softmax}(\operatorname{MLP}_{\mathrm{dur}}(g_k)). \tag*{(125)} p ( L k ∣ z k ) = softmax ( MLP dur ( g k )) . (125)
Given L k L_k L k , g k g_k g k is repeated L k L_k L k times and concatenated with the relative position within the span. If token t t t has local index j j j in its span of length L L L , then
h t Z = W c [ g g ( t ) ∥ MLP l o c ( j L , L − j L , log L ) ] . (126) h_t^Z = W_c\left[g_{g(t)} \mathbin{\|} \operatorname{MLP}_{\mathrm{loc}}\left(\frac{j}{L},\frac{L-j}{L},\log L\right)\right]. \tag*{(126)} h t Z = W c [ g g ( t ) ∥ MLP loc ( L j , L L − j , log L ) ] . (126)
Before being injected into the token decoder, this condition is further refined by a few block-causal conditioning layers. Consequently, multiple token positions expanded from the same latent remain distinguishable.
Block-Causal Masked-Diffusion Decoder
The main text denotes the token-side module by the discrete generator decoder D ψ D_\psi D ψ , instantiated as a block-causal MDM. It receives the committed token context and the token-aligned latent condition H Z H^Z H Z , and realizes continuous semantics as discrete tokens. The decoder is connected to the compression encoder through the duration/expansion interface, uses latent reliability augmentation, and is trained jointly with the detached DiT in Stage 2.
During training, the decoder uses a dual layout [clean | masked] of length 2 T 2T 2 T . The clean half stores the ground-truth tokens, while selected positions in the masked half are replaced with a learnable [MASK] embedding. Let β ( i ) \beta(i) β ( i ) denote the latent block containing token i i i , and let cl ( i ) ∈ { 0 , 1 } \operatorname{cl}(i) \in\{0,1\} cl ( i ) ∈ { 0 , 1 } indicate whether the position belongs to the clean half. The attention mask is
Allow ( i , j ) = { cl ( j ) ∧ [ β ( j ) ≤ β ( i ) ] , cl ( i ) = 1 , cl ( j ) ∧ [ β ( j ) < β ( i ) ] ∨ ¬ cl ( j ) ∧ [ β ( j ) = β ( i ) ] , cl ( i ) = 0. (127) \operatorname{Allow}(i,j)=
\begin{cases}
\operatorname{cl}(j) \land[\beta(j) \le\beta(i)], & \operatorname{cl}(i)=1,\\
\operatorname{cl}(j) \land[\beta(j) < \beta(i)] \lor\neg\operatorname{cl}(j) \land[\beta(j)=\beta(i)], & \operatorname{cl}(i)=0.
\end{cases} \tag*{(127)} Allow ( i , j ) = { cl ( j ) ∧ [ β ( j ) ≤ β ( i )] , cl ( j ) ∧ [ β ( j ) < β ( i )] ∨ ¬ cl ( j ) ∧ [ β ( j ) = β ( i )] , cl ( i ) = 1 , cl ( i ) = 0. (127)
Hence token blocks are causal with respect to one another, while bidirectional denoising is allowed within the current masked block; a masked query cannot read the clean copy of its current block, which prevents target leakage. Under the masked_only configuration, the clean half receives no latent condition, while the masked half receives a clean, noisy, or zero latent condition according to the condition state. This design naturally decouples the historical clean-token cache from the latent condition and allows the conditioned and unconditioned logits to share the same prefix cache.
During training, each token block independently samples a masking ratio π j \pi_j π j , and the cross-entropy of each masked token is weighted by π j − 1 \pi_j^{-1} π j − 1 . With M j \mathcal{M}_j M j denoting the masked positions in block j j j ,
L M D M A = 1 ∣ M A ∣ ∑ t ∈ M A − log p ψ ( y t ∣ Y ~ , H A Z ) π β ( t ) , A ∈ { c o n d , n o i s e , d r o p } , (128) \mathcal{L}_{\mathrm{MDM}}^A = \frac{1}{|\mathcal{M}_A|}\sum_{t\in\mathcal{M}_A}\frac{-\log p_\psi(y_t \mid\widetilde{Y}, H^Z_A)}{\pi_{\beta(t)}}, \qquad A \in\{\mathbf{cond},\mathbf{noise},\mathbf{drop}\}, \tag*{(128)} L MDM A = ∣ M A ∣ 1 t ∈ M A ∑ π β ( t ) − log p ψ ( y t ∣ Y , H A Z ) , A ∈ { cond , noise , drop } , (128)
where the supervised sets of the three states are normalized independently. The implementation also uses fully masked blocks with some probability to directly train the recovery of a complete new block from the latent and the historical blocks.
The decoder uses the encoder-mask generation objective as L g e n \mathcal{L}_{\mathrm{gen}} L gen . A clean encoder forward pass first fixes the hard spans; a random subset of spans is then selected, their encoder input tokens are replaced with encoder-side masks, and a second encoder forward pass is performed without rerunning the dynamic segmenter. The decoder uses fully masked queries for blocks touched by these spans, but supervises only the tokens actually masked on the encoder side:
L e n c m a s k = − 1 ∣ M e ∣ ∑ t ∈ M e log p ψ ( y t ∣ Y ~ , Z e n c m a s k ) . (129) \mathcal{L}_{\mathrm{encmask}} = -\frac{1}{|\mathcal{M}_e|}\sum_{t\in\mathcal{M}_e}\log p_\psi(y_t \mid\widetilde{Y}, Z^{\mathrm{encmask}}). \tag*{(129)} L encmask = − ∣ M e ∣ 1 t ∈ M e ∑ log p ψ ( y t ∣ Y , Z encmask ) . (129)
Since the span layout is fixed by stop-gradient, this branch forces the encoder to use context to form decodable latents when local observations are missing, rather than lowering the task difficulty by moving boundaries.
Stage 1: Unified Compression VAE / Discrete Generator Decoder Objective
The block-causal MDM decoder is trained end to end with the encoder in Stage 1; this is the core of the co-adaptation between continuous semantics and discrete realization in this work, as opposed to first fixing the encoder and then fitting the decoder. For each block, a clean latent Z Z Z is first constructed, and then a mutually exclusive condition state is sampled. The clean state uses Z Z Z directly; the noise state locally perturbs Z Z Z ; the drop state removes only the latent condition of the current block while keeping the same token history. A generic formulation is
z ~ k = { z k , A k = c o n d , ( 1 − u k c ) z k + u k c ϵ k , A k = n o i s e , ∅ , A k = d r o p , u k c ∈ [ 0 , u max ] , ϵ k ∼ N ( 0 , I ) . (130) \tilde{z}_k =
\begin{cases}
z_k, & A_k = \mathbf{cond},\\
(1-u_k^c)z_k + u_k^c\epsilon_k, & A_k = \mathbf{noise},\\
\varnothing, & A_k = \mathbf{drop},
\end{cases}
\qquad u_k^c \in[0,u_{\max}],\quad\epsilon_k \sim\mathcal{N}(0,I). \tag*{(130)} z ~ k = ⎩ ⎨ ⎧ z k , ( 1 − u k c ) z k + u k c ϵ k , ∅ , A k = cond , A k = noise , A k = drop , u k c ∈ [ 0 , u m a x ] , ϵ k ∼ N ( 0 , I ) . (130)
The implementation may additionally apply a weaker sample-level latent perturbation before state partitioning to improve adaptation to small latent perturbations; this is input augmentation rather than a separate reconstruction loss, and it does not assume the same distribution as the terminal error of the DiT.
Let L t o k A \mathcal{L}_{\mathrm{tok}}^A L tok A denote the MDM masked denoising objective under state A A A . The reconstruction–generation balance is
L b a l = λ c L t o k c o n d + λ n L t o k n o i s e + λ d L t o k d r o p , (131) \mathcal{L}_{\mathrm{bal}} = \lambda_c\mathcal{L}_{\mathrm{tok}}^{\mathrm{cond}} + \lambda_n\mathcal{L}_{\mathrm{tok}}^{\mathrm{noise}} + \lambda_d\mathcal{L}_{\mathrm{tok}}^{\mathrm{drop}}, \tag*{(131)} L bal = λ c L tok cond + λ n L tok noise + λ d L tok drop , (131)
where L t o k A = L M D M A \mathcal{L}_{\mathrm{tok}}^A = \mathcal{L}_{\mathrm{MDM}}^A L tok A = L MDM A as defined in Eq. (G.17).
Separately from the three-state decoder balance, we denote the objective dedicated to injecting generative predictive information into the encoder by
L g e n = L e n c m a s k . (132) \mathcal{L}_{\mathrm{gen}} = \mathcal{L}_{\mathrm{encmask}}. \tag*{(132)} L gen = L encmask . (132)
The unified objective of the discrete generator decoder is therefore
L D G D = L b a l + λ g e n L g e n . (133) \mathcal{L}_{\mathrm{DGD}} = \mathcal{L}_{\mathrm{bal}} + \lambda_{\mathrm{gen}}\mathcal{L}_{\mathrm{gen}}. \tag*{(133)} L DGD = L bal + λ gen L gen . (133)
The key target of L g e n \mathcal{L}_{\mathrm{gen}} L gen is the latent representation: the masked-input encoder must recover the missing semantics from context and receives gradients directly from ground-truth discrete token prediction.
The Gaussian KL of each latent is
K L k = 1 2 ∑ d = 1 d z ( μ k , d 2 + σ k , d 2 − 1 − log σ k , d 2 ) , (134) \mathrm{KL}_k = \frac{1}{2}\sum_{d=1}^{d_z}\left(\mu_{k,d}^2 + \sigma_{k,d}^2 - 1 - \log\sigma_{k,d}^2\right), \tag*{(134)} KL k = 2 1 d = 1 ∑ d z ( μ k , d 2 + σ k , d 2 − 1 − log σ k , d 2 ) , (134)
and the duration loss is
L d u r = − 1 K ∑ k = 1 K log p ψ ( L k ∣ z k ) . (135) \mathcal{L}_{\mathrm{dur}} = -\frac{1}{K}\sum_{k=1}^{K}\log p_{\psi}(L_k \mid z_k). \tag*{(135)} L dur = − K 1 k = 1 ∑ K log p ψ ( L k ∣ z k ) . (135)
Combined with Eq. (24) , the total Stage 1 objective is
L ( 1 ) = L D G D + λ d u r L d u r + β K L L K L . (136) \mathcal{L}^{(1)} = \mathcal{L}_{\mathrm{DGD}} + \lambda_{\mathrm{dur}}\mathcal{L}_{\mathrm{dur}} + \beta_{\mathrm{KL}}\mathcal{L}_{\mathrm{KL}}. \tag*{(136)} L ( 1 ) = L DGD + λ dur L dur + β KL L KL . (136)
This objective remains active in Stage 2; the transition from Stage 1 to Stage 2 is therefore a full-state handoff rather than “train the VAE, then freeze the VAE.” The discrete token losses can always shape the encoder latents through the condition path, and only the DiT prior loss is isolated by stop-gradient.
Stage 2: Detached Block-Causal Flow Matching
Stage 2 additionally trains the latent DiT on top of Eq. (G.25). Let Z Z Z be the sampled posterior latent of the current online encoder. The DiT reads only
Z ˉ = sg ( Z ) , (137) \bar{Z} = \operatorname{sg}(Z), \tag*{(137)} Z ˉ = sg ( Z ) , (137)
so the DiT objective cannot backpropagate into the encoder; meanwhile, E ϕ E_{\phi} E ϕ , the latent condition path, and D ψ D_{\psi} D ψ are still updated normally by L ( 1 ) \mathcal{L}^{(1)} L ( 1 ) .
The latent sequence is grouped into blocks of B B B latents. Each block independently samples a noise level u j u_j u j , all latents in a block share the same timestep, and the linear path in Eq. (28) is used. The training layout consists of clean history and noisy current blocks: every noisy query can read only strictly earlier clean latent blocks and interacts bidirectionally within its current noisy block. The attention mask can be written as
AllowDiT ( i , j ) = 1 [ β ( j ) < β ( i ) ] cl ( j ) ∨ 1 [ β ( j ) = β ( i ) ] ¬ cl ( j ) (138) \operatorname{AllowDiT}(i,j)=\mathbb{1}[\beta(j)<\beta(i)]\operatorname{cl}(j)\lor\mathbb{1}[\beta(j)=\beta(i)]\neg\operatorname{cl}(j) \tag*{(138)} AllowDiT ( i , j ) = 1 [ β ( j ) < β ( i )] cl ( j ) ∨ 1 [ β ( j ) = β ( i )] ¬ cl ( j ) (138)
where clean-history positions act as read-only conditions, and the current noisy block cannot read its corresponding clean target copy.
To cover inference cases in which the prompt is not aligned with latent block boundaries, training may randomly restore the first c ∈ { 1 , … , B − 1 } c \in\{1,\ldots,B-1\} c ∈ { 1 , … , B − 1 } latents of a noisy block to their clean values and remove them from the corresponding Flow Matching supervision. This partial-clean-prefix augmentation matches the handling of the first mixed block at inference. Finally,
∇ ϕ L F M = 0 , ∇ θ L F M ≠ 0 (139) \nabla_{\phi}\mathcal{L}_{\mathrm{FM}}=0,\qquad\nabla_{\theta}\mathcal{L}_{\mathrm{FM}}\ne0 \tag*{(139)} ∇ ϕ L FM = 0 , ∇ θ L FM = 0 (139)
and
L ( 2 ) = L ( 1 ) + λ F M L F M . (140) \mathcal{L}^{(2)}=\mathcal{L}^{(1)}+\lambda_{\mathrm{FM}}\mathcal{L}_{\mathrm{FM}}. \tag*{(140)} L ( 2 ) = L ( 1 ) + λ FM L FM . (140)
Stage 2 can thus be understood as continuously updating the online compression encoder and discrete generator decoder while letting the DiT track their evolving latent distribution, but forbidding the DiT from shaping the encoder through the diffusion loss.
Complete Inference Procedure and Latent Feedback
Inference follows propose–decode–re-encode–commit . The dynamic encoder runs on the prompt only once at the beginning and uses the posterior mode. Suppose the prompt yields K p K_{\mathrm{p}} K p latents with
K p = B q + r , 0 ≤ r < B . (141) K_{\mathrm{p}}=Bq+r,\qquad0\le r<B. \tag*{(141)} K p = B q + r , 0 ≤ r < B . (141)
The first B q Bq B q block-aligned latents are prefilled into the persistent caches. If r > 0 r>0 r > 0 , the trailing r r r prompt latents and their tokens are not written into the persistent cache as a completed block; instead, they are kept as the clean prefix of the first generated block, and the remaining B − r B-r B − r latent slots are initialized from Gaussian noise:
Z j ( 0 ) = [ z K p − r + 1 : K p P ∥ ϵ 1 , … , ϵ B − r ] , ϵ i ∼ N ( 0 , I ) . (142) Z_j^{(0)}=\left[z^P_{K_{\mathrm{p}}-r+1:K_{\mathrm{p}}}\,\Vert\,\epsilon_1,\ldots,\epsilon_{B-r}\right],\qquad\epsilon_i\sim\mathcal{N}(0,I). \tag*{(142)} Z j ( 0 ) = [ z K p − r + 1 : K p P ∥ ϵ 1 , … , ϵ B − r ] , ϵ i ∼ N ( 0 , I ) . (142)
This explains the meaning of “discard trailing remainder” in Figure 1 : the trailing remainder is excluded from the committed cache , not permanently removed from the prompt semantics. After each DiT Euler update, the clean prompt prefix in the mixed block is restored to its original values.
Latent proposal. The DiT performs S g e n S_{\mathrm{gen}} S gen denoising steps on top of the committed latent prefix. With CFG,
v ^ = v u + γ g e n ( v c − v u ) , (143) \hat{v}=v_u+\gamma_{\mathrm{gen}}(v_c-v_u), \tag*{(143)} v ^ = v u + γ gen ( v c − v u ) , (143)
where the conditional branch reads the committed corrected latent cache and the unconditional branch removes this prefix. Importantly, the current proposal remains a read-only temporary state until token generation and feedback correction are complete, and is not written into the persistent DiT/condition caches.
Duration and token decoding. The proposal latents first pass through the duration/expansion path. For newly generated latent slots,
L ^ k = arg max L p ψ ( L ∣ z k ) , (144) \hat{L}_k=\underset{L}{\arg\max}\;p_{\psi}(L\mid z_k), \tag*{(144)} L ^ k = L arg max p ψ ( L ∣ z k ) , (144)
whereas the slots belonging to the prompt tail in the mixed first block keep the ground-truth span lengths known to the encoder. This yields the effective width and the token-aligned condition of the current token block.
The MDM decoder initializes unknown positions to [MASK], repeatedly predicts them over several refinement steps, and progressively commits them by confidence; intermediate masked states do not enter the KV cache, and only after the whole block is determined is a single clean forward pass used to commit the final tokens, yielding the finalized clean token block Y j Y_j Y j .
The decoder logits use the prefix–latent two-axis CFG in Eq. (37) . The four branches share the current block state and the latent-free clean-token prefix cache; the prefix-off branches mask the historical token keys, and the latent-off branches zero the injection of the current proposal latent condition. Execution details are given below.
Fixed-span re-encoding. After the current token block is finalized, the feedback encoder does not rerun the dynamic segmenter. Instead, it directly constructs fixed span boundaries from the span lengths just determined by the duration predictor. It applies to these tokens the same span-local bidirectional pooling as the initial encoder, reads the committed encoder latent-prefix KV, obtains the posterior of the current block through the causal latent encoder, and takes its mode:
Z j c o r r = μ ϕ ( Y j ; L ^ B j , K < j E ) . (145) Z^{\mathrm{corr}}_j = \mu_\phi\left(Y_j; \hat{L}_{B_j}, \mathcal{K}^{E}_{<j}\right). \tag*{(145)} Z j corr = μ ϕ ( Y j ; L ^ B j , K < j E ) . (145)
For the first mixed block, the previously saved pooled representations of the prompt tail are concatenated with those of the newly generated spans to form a complete B B B -latent block before running the latent encoder. Since the span boundaries are fixed, this correction changes “the latent obtained by re-encoding the generated tokens under this span layout,” rather than re-optimizing the segmentation.
Atomic commit. After feedback, the proposal latent is replaced by Z j c o r r Z^{\mathrm{corr}}_j Z j corr , and the caches are updated according to the invariants
K ≤ j D i T = Z ≤ j c o r r , K j c o n d = Cond ( Z ≤ j c o r r ) , K j E = EncState ( Y ≤ j ) , K j t o k = Y ≤ j c l e a n . (146) \begin{aligned}
\mathcal{K}^{\mathrm{DiT}}_{\leq j} &= Z^{\mathrm{corr}}_{\leq j}, \\
\mathcal{K}^{\mathrm{cond}}_j &= \operatorname{Cond}\left(Z^{\mathrm{corr}}_{\leq j}\right), \\
\mathcal{K}^{E}_j &= \operatorname{EncState}\left(Y_{\leq j}\right), \\
\mathcal{K}^{\mathrm{tok}}_j &= Y^{\mathrm{clean}}_{\leq j}. \tag*{(146)}
\end{aligned} K ≤ j DiT K j cond K j E K j tok = Z ≤ j corr , = Cond ( Z ≤ j corr ) , = EncState ( Y ≤ j ) , = Y ≤ j clean . (146)
That is, the DiT history, the condition history, and the encoder latent history all correspond to corrected encoder latents, while the token cache contains only the final clean tokens. The proposal latent never enters the persistent history. The current block is not regenerated after correction, so the effect of feedback starts from the next latent block.
This transaction also establishes an important train–inference consistency invariant. During training, the clean history fitted by the DiT consists of online-encoder latents of ground-truth tokens; at inference, after each block is completed, the next history is obtained from the final committed tokens through the same fixed-span encoder path. Hence, for the committed prefix,
K ≤ j Z = E ϕ ( K ≤ j Y ; S ^ ≤ j ) , (147) \mathcal{K}^{Z}_{\leq j} = E_\phi\left(\mathcal{K}^{Y}_{\leq j}; \hat{S}_{\leq j}\right), \tag*{(147)} K ≤ j Z = E ϕ ( K ≤ j Y ; S ^ ≤ j ) , (147)
rather than K ≤ j Z = Z ≤ j p r o p \mathcal{K}^{Z}_{\leq j} = Z^{\mathrm{prop}}_{\leq j} K ≤ j Z = Z ≤ j prop . The former keeps the continuous context of the DiT, the latent history of the condition stack, and the clean-token context of the discrete generator decoder pointing to the same finalized content. If the proposal were committed directly, any proposal–decode discrepancy would be permanently written into the latent cache and would cause cross-space context bias in subsequent blocks; fixed-span correction is designed precisely to eliminate this cumulative mismatch. Since the correction reuses the spans just predicted, it only aligns the representation content without reintroducing segmentation drift at inference.
Algorithm 1 summarizes the unified procedure.
Execution Details of MDM Prefix–Latent CFG
Condition preparation and scope. This path requires decoder_cond_drop_scope=masked_only, which guarantees that the prefix token KV contains no latent condition. We follow the branch subscripts a , b a,b a , b of
Algorithm 1 Unified block-wise inference of Cola2
Require: Prompt C C C , latent block size B B B , DiT steps S g e n S_{\mathrm{gen}} S gen
z 1 : K p ← μ ( E ϕ ( C ) ) z_{1:K_p} \leftarrow\mu(E_{\phi}(C)) z 1 : K p ← μ ( E ϕ ( C ))
Prefill the complete block-aligned prompt prefix into the persistent caches, and keep the remainder as the clean prefix of the first block
while stopping criterion not met do
Initialize the current latent block from Gaussian noise (and the optional clean prompt tail)
for i = 0 , … , S g e n − 1 i = 0,\ldots,S_{\mathrm{gen}}-1 i = 0 , … , S gen − 1 do
Update the DiT proposal on the committed corrected-latent prefix; restore the clean tail if present
end for
Predict span lengths from the proposal latents and construct the token-aligned latent condition
Complete the current token block with the block-causal MDM decoder using the two-axis CFG in Eq. (37)
Re-encode the final tokens with the fixed predicted spans to obtain Z j c o r r Z_j^{\mathrm{corr}} Z j corr
Atomically commit the finalized tokens, Z j c o r r Z_j^{\mathrm{corr}} Z j corr , and the corresponding condition/encoder states
end while
return the generated token sequence
Appendix § 14 .§ 4 . The refined condition is prepared only once per block:
H j Z = Cond ( Expand ( Z j p r o p , L ^ b B j ) ; K < j c o n d ) , H j Z , ( b ) = b H j Z , (148) H_j^Z = \operatorname{Cond}\left(\operatorname{Expand}\left(Z_j^{\mathrm{prop}},\widehat{L}_{bB_j}\right);\mathcal{K}_{<j}^{\mathrm{cond}}\right), \qquad H_j^{Z,(b)} = bH_j^Z, \tag*{(148)} H j Z = Cond ( Expand ( Z j prop , L b B j ) ; K < j cond ) , H j Z , ( b ) = b H j Z , (148)
where K < j c o n d \mathcal{K}_{<j}^{\mathrm{cond}} K < j cond is the committed condition-stack history and L ^ b B j \widehat{L}_{bB_j} L b B j follows the existing duration rule. All four branches share the same H j Z H_j^Z H j Z ; b = 0 b=0 b = 0 zeros the injected condition after refinement, without recomputing the duration or the condition stack.
Prefix-key masking and position indexing. Omitting the batch index, let m r ∈ { 0 , 1 } m_r \in\{0,1\} m r ∈ { 0 , 1 } indicate the validity of the r r r -th key slot, N k N_k N k the total number of key slots, and N q N_q N q the number of current query slots. The current block occupies the trailing index set Q j = { N k − N q + 1 , … , N k } Q_j = \{N_k-N_q+1,\ldots,N_k\} Q j = { N k − N q + 1 , … , N k } . The two prefix visibilities correspond to
m r ( a ) = m r ( a + ( 1 − a ) 1 [ r ∈ Q j ] ) , pos r ( a ) = max ( ∑ s = 1 r m s ( a ) − 1 , 0 ) . (149) m_r^{(a)} = m_r\left(a+(1-a)\mathbb{1}[r\in Q_j]\right), \qquad\operatorname{pos}_r^{(a)} = \max\left(\sum_{s=1}^{r}m_s^{(a)}-1,0\right). \tag*{(149)} m r ( a ) = m r ( a + ( 1 − a ) 1 [ r ∈ Q j ] ) , pos r ( a ) = max ( s = 1 ∑ r m s ( a ) − 1 , 0 ) . (149)
For valid current-block queries, m ( a ) m^{(a)} m ( a ) controls both key visibility and position indexing. With a = 0 a=0 a = 0 , only the valid keys of the current block are kept, and their positions are indexed contiguously from 0. If the first mixed block contains a clean prompt tail, the tail still belongs to Q j Q_j Q j and is not masked together with the persistent prefix.
Our implementation caches keys before RoPE and applies RoPE according to pos ( a ) \operatorname{pos}^{(a)} pos ( a ) after concatenating the cached and current keys. The two prefix branches thus reuse the same persistent KV without an additional cache or a modified RoPE operator, although the effective position indices differ between branches. Masking does not modify the tokens, the DiT proposal, or the condition-stack history. This rule adds no training objective; the first training block provides an input form without historical token keys, but we do not assume from this that all prefix-off inference states are in-distribution with respect to training.
Read-only branches, demasking, and commit. Let M j M_j M j be the number of refinement steps of the current block, s ∈ { 0 , … , M j − 1 } s\in\{0,\ldots,M_j-1\} s ∈ { 0 , … , M j − 1 } the iteration index, and M s M_s M s the valid positions that are still masked. At each step, the branches with nonzero coefficients are computed on the same current token state, all with update_kv=False, and the logits are then combined according to Eq. (F.18). Denoting the combined logits at position t t t by ℓ t ( s ) \ell_t^{(s)} ℓ t ( s ) , the candidate token and confidence under greedy decoding are
y ^ t ( s ) = arg max y ℓ t ( s ) ( y ) , c t ( s ) = log softmax ( ℓ t ( s ) ) ( y ^ t ( s ) ) , t ∈ M s . (150) \widehat{y}_t^{(s)} = \underset{y}{\arg\max}\ \ell_t^{(s)}(y), \qquad c_t^{(s)} = \log\operatorname{softmax}\left(\ell_t^{(s)}\right)\left(\widehat{y}_t^{(s)}\right), \qquad t\in M_s. \tag*{(150)} y t ( s ) = y arg max ℓ t ( s ) ( y ) , c t ( s ) = log softmax ( ℓ t ( s ) ) ( y t ( s ) ) , t ∈ M s . (150)
Under non-greedy decoding, candidates are sampled from the guided logits as configured. At each step, the highest-confidence candidates are kept according to the existing quota and the remaining positions stay masked; the known prompt tail does not participate in generation. Since c t ( s ) c_t^{(s)} c t ( s ) is also derived from the guided logits, the two-axis weights affect both token selection and potentially the unmasking order, and confidence effects cannot be ruled out merely by the scale invariance of the greedy argmax.
After the block is finalized, the final tokens are written into the cache by a single latent-free clean forward pass (update_kv=True) without CFG, restoring full prefix visibility and normal position indexing. The original fixed-span feedback and corrected-history commit remain unchanged. The persistent token KV is determined only by finalized tokens; branch hidden states and masked states are temporary.
The two configuration parameters are mdm_decoder_cfg_weight (γ z − 1 \gamma_z - 1 γ z − 1 ) and mdm_decoder_prefix_cfg_weight (γ p − 1 \gamma_p - 1 γ p − 1 ). The prefix parameter defaults to 0, corresponding to γ p = 1 \gamma_p = 1 γ p = 1 . For a fixed single-step state, the four branch logits can be cached and recombined with different weights; however, different weights change subsequent tokens, the unmasking order, and the committed history, so a complete rollout cannot share a single offline logits trajectory.
Detailed Derivation of the Computational Cost
This section re-derives the computational cost according to the current training and inference execution graphs. There are two key differences: Stage 2 continues training the same compression encoder and discrete generator decoder and only additionally adds a detached DiT; and inference performs fixed-span feedback after the discrete tokens of each block are finalized, before committing the corrected latent history. Hence, the previous accounting of “Stage 1 VAE + Stage 2 DiT/decoder” or a one-way “DiT → decoder” no longer applies.
Notation
Below, one multiply–add counts as two FLOPs, and we count backbone linear layers, attention matrix multiplications, output heads, and auxiliary projections, while omitting element-wise operations such as embedding lookup, normalization, and softmax. Training includes the forward and backward passes, but excludes activation recomputation, optimizer updates, and communication. If these additional operations are counted, Cola2 and AR should use the same convention.
For a module R R R , let N R N_R N R be the number of its backbone linear-layer parameters (excluding separate heads/interfaces), n R n_R n R its number of layers, d R q d_R^q d R q the total width of all query heads per layer, Q R Q_R Q R the cumulative number of processed positions, and A R e x e c A_R^{\mathrm{exec}} A R exec the cumulative number of executed query–key pairs per layer. When all layers share the same shape,
C R , b a c k b o n e f w d ≈ 2 N R Q R + 4 n R d R q A R e x e c , C_{R,\mathrm{backbone}}^{\mathrm{fwd}} \approx2N_RQ_R + 4n_Rd_R^qA_R^{\mathrm{exec}}, C R , backbone fwd ≈ 2 N R Q R + 4 n R d R q A R exec ,
C R , b a c k b o n e t r a i n ≈ 6 N R Q R + 12 n R d R q A R e x e c , (151) C_{R,\mathrm{backbone}}^{\mathrm{train}} \approx6N_RQ_R + 12n_Rd_R^qA_R^{\mathrm{exec}}, \tag*{(151)} C R , backbone train ≈ 6 N R Q R + 12 n R d R q A R exec , (151)
where the attention term includes both Q K T QK^{\mathsf{T}} Q K T and the attention–value product; heterogeneous layers are summed layer by layer. Let A R v a l i d A_R^{\mathrm{valid}} A R valid be the number of pairs allowed by the mask; it equals the executed amount only when the kernel actually skips invalid regions, and dense masks, tile boundaries, and padding must be charged according to the actual shapes.
If the vocabulary size is V V V , the output hidden width is d o u t d_{\mathrm{out}} d out , and logits are actually computed at Q l o g i t Q_{\mathrm{logit}} Q logit positions, then
C h e a d f w d ≈ 2 V d o u t Q l o g i t , C h e a d t r a i n ≈ 6 V d o u t Q l o g i t . (152) C_{\mathrm{head}}^{\mathrm{fwd}} \approx2Vd_{\mathrm{out}}Q_{\mathrm{logit}}, \qquad C_{\mathrm{head}}^{\mathrm{train}} \approx6Vd_{\mathrm{out}}Q_{\mathrm{logit}}. \tag*{(152)} C head fwd ≈ 2 V d out Q logit , C head train ≈ 6 V d out Q logit . (152)
Weight sharing with the embedding does not eliminate this output projection, and supervising only a subset of positions does not automatically reduce the backbone Q R Q_R Q R . All of the above are approximations of the dominant matrix-multiplication terms; the total cost accumulates over actual calls rather than being counted by the number of distinct parameters.
A training sequence contains T T T valid tokens and, after compression,
K = ρ T . (153) K = \rho T. \tag*{(153)} K = ρT . (153)
At inference, let T n e w T_{\mathrm{new}} T new be the number of newly generated tokens, K n e w K_{\mathrm{new}} K new the number of corresponding latents, and ρ g e n = K n e w / T n e w \rho_{\mathrm{gen}} = K_{\mathrm{new}}/T_{\mathrm{new}} ρ gen = K new / T new the realized ratio. Ignoring truncation at the beginning and end, the average number of tokens P P P covered by a latent block of size B B B and the number of blocks J J J satisfy
P ≈ B ρ g e n , J ≈ ρ g e n T n e w B . (154) P \approx\frac{B}{\rho_{\mathrm{gen}}}, \qquad J \approx\frac{\rho_{\mathrm{gen}}T_{\mathrm{new}}}{B}. \tag*{(154)} P ≈ ρ gen B , J ≈ B ρ gen T new . (154)
Duration prediction does not guarantee ρ g e n = ρ ⋆ \rho_{\mathrm{gen}} = \rho^\star ρ gen = ρ ⋆ . The prompt tail in the first block, truncation in the last block, and batch padding may make the actual processed amount larger than the effective output, so we first give per-call counts and then approximations that ignore boundary effects.
Training: Persistent Representation/Decoder Path and an Additional Detached DiT
Stage 1 representation/decoder path. Denote the training cost of this path by C r e p ( T , K ) C_{\mathrm{rep}}(T,K) C rep ( T , K ) . Clean, noisy, and drop are mutually exclusive condition states within the same main request, so the backbone forward pass is not multiplied by three; the additional requests of the generation-oriented L g e n \mathcal{L}_{\mathrm{gen}} L gen , however, must be counted separately. Below, C E C_E C E includes the encoder and the posterior projection, C D M D M C_D^{\mathrm{MDM}} C D MDM includes the MDM token backbone, attention, and the corresponding output head, and the remaining operations such as conditioning and duration are listed as separate terms.
The MDM main request uses one clean encoder pass and one dual-layout decoder pass of length 2 T 2T 2 T ; L e n c m a s k \mathcal{L}_{\mathrm{encmask}} L encmask fixes the original spans and runs an additional masked-input encoder pass and a separate MDM request. Hence, when both requests are fully executed,
C r e p M D M ( T , K ) ≈ 2 C E ( T , K ) + 2 C D M D M ( 2 T ) + C c o n d / d u r / K L M D M , (155) C_{\mathrm{rep}}^{\mathrm{MDM}}(T,K) \approx2C_E(T,K) + 2C_D^{\mathrm{MDM}}(2T)
+ C_{\mathrm{cond/dur/KL}}^{\mathrm{MDM}}, \tag*{(155)} C rep MDM ( T , K ) ≈ 2 C E ( T , K ) + 2 C D MDM ( 2 T ) + C cond/dur/KL MDM , (155)
where each single-pass cost is the average over the two requests. The two decoder requests together process about 4 T 4T 4 T backbone positions, with a dominant linear training term of 24 N D T 24N_DT 24 N D T , where N D N_D N D is the number of MDM backbone parameters. Concatenating the two requests along the batch dimension does not change this count; the output head is charged by the actual logit positions, and repeated calls of the condition path are included in the last term. The two-axis inference CFG does not increase the number of training requests in either stage.
Let E s \mathbb{E}_s E s denote the average over the actual training samples or batch layouts of Stage s s s , and T T T the corresponding number of valid tokens. Define
c r e p ( s ) : = E s [ C r e p ( T , K ) ] E s [ T ] , s ∈ { 1 , 2 } . (156) c_{\mathrm{rep}}^{(s)} := \frac{\mathbb{E}_s[C_{\mathrm{rep}}(T,K)]}{\mathbb{E}_s[T]}, \qquad s \in\{1,2\}. \tag*{(156)} c rep ( s ) := E s [ T ] E s [ C rep ( T , K )] , s ∈ { 1 , 2 } . (156)
This definition normalizes the actual executed amount by valid tokens. The representation path remains fully present in Stage 2, but when the sequence length, compression layout, or request configuration changes, c r e p ( 1 ) = c r e p ( 2 ) c_{\mathrm{rep}}^{(1)} = c_{\mathrm{rep}}^{(2)} c rep ( 1 ) = c rep ( 2 ) need not hold.
Stage 2 detached DiT. The same clean encoder latent participates in the token objectives and also enters the DiT as Z ˉ = sg ( Z ) \bar{Z} = \operatorname{sg}(Z) Z ˉ = sg ( Z ) . Suppose the training sequence has J t r = ⌈ K / B ⌉ J_{\mathrm{tr}} = \lceil K/B \rceil J tr = ⌈ K / B ⌉ latent blocks, and the last block has length r l a s t = K − B ( J t r − 1 ) ∈ { 1 , … , B } r_{\mathrm{last}} = K - B(J_{\mathrm{tr}} - 1) \in\{1,\ldots,B\} r last = K − B ( J tr − 1 ) ∈ { 1 , … , B } . Omitting the clean copy of the last block, K c = K − r l a s t K_c = K-r_{\mathrm{last}} K c = K − r last , and we define
χ G = K + K c K , 1 ≤ χ G ≤ 2. (157) \chi_G = \frac{K+K_c}{K}, \qquad1 \leq\chi_G \leq2. \tag*{(157)} χ G = K K + K c , 1 ≤ χ G ≤ 2. (157)
When K K K is divisible by B B B , r l a s t = B r_{\mathrm{last}} = B r last = B rather than 0. The dominant linear training term of the DiT backbone is
C G , l i n e a r t r a i n ( T , K ) ≈ 6 N G ( K + K c ) = 6 χ G ρ N G T . (158) C_{G,\mathrm{linear}}^{\mathrm{train}}(T,K) \approx6N_G(K+K_c) = 6\chi_G\rho N_GT. \tag*{(158)} C G , linear train ( T , K ) ≈ 6 N G ( K + K c ) = 6 χ G ρ N G T . (158)
Detachment only cuts the gradient to the encoder; it does not skip the backward pass of the DiT itself.
When K = J t r B K = J_{\mathrm{tr}}B K = J tr B and the clean stream is also computed block-causally, the numbers of valid pairs of clean and noisy queries correspond to the first J t r − 1 J_{\mathrm{tr}} - 1 J tr − 1 blocks and all J t r J_{\mathrm{tr}} J tr blocks, respectively:
A G v a l i d = B 2 ( ∑ j = 1 J t r − 1 j + ∑ j = 1 J t r j ) = K 2 . (159) \mathcal{A}^{\mathrm{valid}}_{G} = B^2\left(\sum_{j=1}^{J_{\mathrm{tr}}-1} j + \sum_{j=1}^{J_{\mathrm{tr}}} j\right) = K^2. \tag*{(159)} A G valid = B 2 ( j = 1 ∑ J tr − 1 j + j = 1 ∑ J tr j ) = K 2 . (159)
Layouts with incomplete blocks remain O ( K 2 ) O(K^2) O ( K 2 ) . The dominant attention training term is
C G , a t t n t r a i n ≈ 12 n G d G q A G e x e c , A G v a l i d = O ( ρ 2 T 2 ) , (160) C^{\mathrm{train}}_{G,\mathrm{attn}} \approx12n_Gd^{q}_{G}A^{\mathrm{exec}}_{G}, \qquad\mathcal{A}^{\mathrm{valid}}_{G} = O(\rho^2T^2), \tag*{(160)} C G , attn train ≈ 12 n G d G q A G exec , A G valid = O ( ρ 2 T 2 ) , (160)
where n G n_G n G and d G q d^{q}_{G} d G q follow the module-level notation. Let C G , a u x t r a i n C^{\mathrm{train}}_{G,\mathrm{aux}} C G , aux train include the latent input/output and time/condition projections not counted in N G N_G N G . The average per-token cost of the added DiT is then
c G ( 2 ) = E 2 [ C G , l i n e a r t r a i n + C G , a t t n t r a i n + C G , a u x t r a i n ] E 2 [ T ] . (161) c^{(2)}_{G} = \frac{\mathbb{E}_2[C^{\mathrm{train}}_{G,\mathrm{linear}} + C^{\mathrm{train}}_{G,\mathrm{attn}} + C^{\mathrm{train}}_{G,\mathrm{aux}}]}{\mathbb{E}_2[T]}. \tag*{(161)} c G ( 2 ) = E 2 [ T ] E 2 [ C G , linear train + C G , attn train + C G , aux train ] . (161)
Hence ρ \rho ρ and ρ 2 \rho^2 ρ 2 describe the scaling of the linear length and the attention pairs under the same architecture and a comparable layout, rather than a fixed speedup factor for the complete system or an arbitrary kernel.
Total training cost and AR reference. If the two stages process D 1 D_1 D 1 and D 2 D_2 D 2 valid tokens, respectively, then
C C o l a 2 t r a i n ≈ D 1 c r e p ( 1 ) + D 2 ( c r e p ( 2 ) + c G ( 2 ) ) . (162) C^{\mathrm{train}}_{\mathrm{Cola2}} \approx D_1c^{(1)}_{\mathrm{rep}} + D_2(c^{(2)}_{\mathrm{rep}} + c^{(2)}_{G}). \tag*{(162)} C Cola2 train ≈ D 1 c rep ( 1 ) + D 2 ( c rep ( 2 ) + c G ( 2 ) ) . (162)
With the same representation configuration, the first part can be merged into ( D 1 + D 2 ) c r e p (D_1 + D_2)c_{\mathrm{rep}} ( D 1 + D 2 ) c rep , but the DiT attention and auxiliary projections must still be retained.
Taking as reference an AR model with training budget D A R D_{\mathrm{AR}} D AR and average per-token cost c A R t r a i n c^{\mathrm{train}}_{\mathrm{AR}} c AR train under the same convention, define
r r e p ( s ) = c r e p ( s ) c A R t r a i n , r G t r a i n = c G ( 2 ) c A R t r a i n . (163) r^{(s)}_{\mathrm{rep}} = \frac{c^{(s)}_{\mathrm{rep}}}{c^{\mathrm{train}}_{\mathrm{AR}}}, \qquad r^{\mathrm{train}}_{G} = \frac{c^{(2)}_{G}}{c^{\mathrm{train}}_{\mathrm{AR}}}. \tag*{(163)} r rep ( s ) = c AR train c rep ( s ) , r G train = c AR train c G ( 2 ) . (163)
Then
R t r a i n : = C C o l a 2 t r a i n D A R c A R t r a i n ≈ D 1 D A R r r e p ( 1 ) + D 2 D A R ( r r e p ( 2 ) + r G t r a i n ) . (164) \begin{aligned}
R_{\mathrm{train}} := \frac{C^{\mathrm{train}}_{\mathrm{Cola2}}}{D_{\mathrm{AR}}c^{\mathrm{train}}_{\mathrm{AR}}} \\
&\approx\frac{D_1}{D_{\mathrm{AR}}}r^{(1)}_{\mathrm{rep}} + \frac{D_2}{D_{\mathrm{AR}}}\left(r^{(2)}_{\mathrm{rep}} + r^{\mathrm{train}}_{G}\right). \tag*{(164)}
\end{aligned} R train := D AR c AR train C Cola2 train ≈ D AR D 1 r rep ( 1 ) + D AR D 2 ( r rep ( 2 ) + r G train ) . (164)
The AR cost c A R t r a i n c^{\mathrm{train}}_{\mathrm{AR}} c AR train likewise includes attention and the output head. 6 N A R D A R 6N_{\mathrm{AR}}D_{\mathrm{AR}} 6 N AR D AR is only its backbone linear reference and cannot serve as the denominator for a complete cost that includes other operations. Compression mainly reduces the cost of the added prior and does not automatically make the entire training scale with ρ \rho ρ .
Inference: Proposal, Discrete Decoding, Feedback, and Commit
Excluding the one-time prompt encoding/prefill, the transaction of each generated block is
DiT proposal → DGD token decoding → fixed-span re-encoding → corrected-cache commit . (165) \text{DiT proposal} \mathbin{\to} \text{DGD token decoding} \mathbin{\to} \text{fixed-span re-encoding} \mathbin{\to} \text{corrected-cache commit}. \tag*{(165)} DiT proposal → DGD token decoding → fixed-span re-encoding → corrected-cache commit . (165)
Subsequent attention still reads the prompt KV, so its history length cannot be dropped merely because prefill is excluded. Below, C G i n f e r C^{\mathrm{infer}}_{G} C G infer includes the DiT proposal/commit, C D i n f e r C^{\mathrm{infer}}_{D} C D infer includes condition preparation, token decoding, and the token-cache commit, and C f b C_{\mathrm{fb}} C fb includes the feedback encoder and the corrected condition commit; the three parts do not overlap.
DiT proposal and corrected DiT commit. Let B j e x e c B_j^{\mathrm{exec}} B j exec be the number of latent query slots actually processed in the j j j -th block, and ν G , j , i \nu_{G,j,i} ν G , j , i the number of velocity branches actually executed at its i i i -th denoising step. Then
Q G p r o p = ∑ j = 1 J B j e x e c ∑ i = 1 S g e n ν G , j , i ≈ ρ g e n ν g e n S g e n T n e w . (166) \begin{aligned}
Q_G^{\mathrm{prop}}=\sum_{j=1}^{J}B_j^{\mathrm{exec}}\sum_{i=1}^{S_{\mathrm{gen}}}\nu_{G,j,i} \\
&\approx\rho_{\mathrm{gen}}\nu_{\mathrm{gen}}S_{\mathrm{gen}}T_{\mathrm{new}}. \tag*{(166)}
\end{aligned} Q G prop = j = 1 ∑ J B j exec i = 1 ∑ S gen ν G , j , i ≈ ρ gen ν gen S gen T new . (166)
The second line assumes a uniform branch count ν g e n \nu_{\mathrm{gen}} ν gen and ignores boundaries and padding. The proposal does not write to the persistent cache; corrected latents are committed only after feedback. To avoid folding attention into the linear coefficient, define
κ G c o m m i t : = C G , l i n e a r c o m m i t 2 N G K n e w , (167) \kappa_G^{\mathrm{commit}} := \frac{C_{G,\mathrm{linear}}^{\mathrm{commit}}}{2N_GK_{\mathrm{new}}}, \tag*{(167)} κ G commit := 2 N G K new C G , linear commit , (167)
where the numerator is only the backbone linear cost of the corrected DiT commit. If each new latent requires one additional full clean forward pass, κ G c o m m i t ≈ 1 \kappa_G^{\mathrm{commit}}\approx1 κ G commit ≈ 1 ignoring boundaries. The total cost is
C G i n f e r ≈ 2 N G Q G p r o p + C G , l i n e a r c o m m i t + C G , a t t n i n f e r + C G , a u x i n f e r , (168) \begin{aligned}
C_G^{\mathrm{infer}}\approx2N_GQ_G^{\mathrm{prop}}+C_{G,\mathrm{linear}}^{\mathrm{commit}} \\
&+C_{G,\mathrm{attn}}^{\mathrm{infer}}+C_{G,\mathrm{aux}}^{\mathrm{infer}}, \tag*{(168)}
\end{aligned} C G infer ≈ 2 N G Q G prop + C G , linear commit + C G , attn infer + C G , aux infer , (168)
where the last two terms accumulate the attention and remaining input/output projections of the proposal and commit. If the current effective block width is b j b_j b j and the committed latent history length is P j Z P_j^Z P j Z , the numbers of valid pairs per branch with prefix on and off are b j ( P j Z + b j ) b_j(P_j^Z+b_j) b j ( P j Z + b j ) and b j 2 b_j^2 b j 2 , respectively; these are accumulated over the actual denoising steps and commits, and the executed pair counts are substituted into Eq. (H.1).
MDM discrete generator decoder. Let W j e x e c W_j^{\mathrm{exec}} W j exec be the query width actually processed in each refinement of the j j j -th block, W j c o m W_j^{\mathrm{com}} W j com the number of positions processed in the final clean commit, M j M_j M j the number of refinement steps, and ν D , j , i \nu_{D,j,i} ν D , j , i the number of decoder branches actually executed at step i i i . Then
C D M D M ≈ 2 N D ∑ j = 1 J ( W j e x e c ∑ i = 1 M j ν D , j , i + W j c o m ) + C D , a t t n M D M + C D , h e a d / i n t e r f a c e M D M . (169) \begin{aligned}
C_D^{\mathrm{MDM}}\approx2N_D\sum_{j=1}^{J}\left(W_j^{\mathrm{exec}}\sum_{i=1}^{M_j}\nu_{D,j,i}+W_j^{\mathrm{com}}\right) \\
&+C_{D,\mathrm{attn}}^{\mathrm{MDM}}+C_{D,\mathrm{head/interface}}^{\mathrm{MDM}}. \tag*{(169)}
\end{aligned} C D MDM ≈ 2 N D j = 1 ∑ J W j exec i = 1 ∑ M j ν D , j , i + W j com + C D , attn MDM + C D , head/interface MDM . (169)
Ignoring the beginning, end, and padding, if ν d e c \nu_{\mathrm{dec}} ν dec branches are computed at every step, the dominant linear term reduces to 2 N D ∑ j ( ν d e c M j + 1 ) W j 2N_D\sum_j(\nu_{\mathrm{dec}}M_j+1)W_j 2 N D ∑ j ( ν dec M j + 1 ) W j , where W j W_j W j is the effective block width; with a uniform M j = M M_j=M M j = M , it is approximately 2 N D ( ν d e c M + 1 ) T n e w 2N_D(\nu_{\mathrm{dec}}M+1)T_{\mathrm{new}} 2 N D ( ν dec M + 1 ) T new .
The full two-axis CFG executes four branches, but ν D , j , i \nu_{D,j,i} ν D , j , i counts the actual forward passes rather than only the nonzero coefficients. The cost is reduced only if the code skips zero-coefficient branches or merges identical branches before the call; multiplying logits by zero after the call saves no backbone FLOPs. The four branches share one proposal condition preparation per block, and only one clean-token commit without CFG is performed at the end.
To expand the effect of prefix visibility, let P j Y P_j^Y P j Y be the number of tokens committed before the j j j -th block (including the prompt), and ν j , i o n \nu_{j,i}^{\mathrm{on}} ν j , i on the number of branches with the prefix kept at that step. With effective width W j W_j W j , the number of valid pairs over all refinements and the clean commit is
A D v a l i d = ∑ j = 1 J [ ∑ i = 1 M j ( ν j , i o n W j P j Y + ν D , j , i W j 2 ) A_D^{\mathrm{valid}}=\sum_{j=1}^{J}\left[\sum_{i=1}^{M_j}\left(\nu_{j,i}^{\mathrm{on}}W_jP_j^Y+\nu_{D,j,i}W_j^2\right)
\right. A D valid = j = 1 ∑ J i = 1 ∑ M j ( ν j , i on W j P j Y + ν D , j , i W j 2 )
+ W j ( P j Y + W j ) ] . (170) \left.
+W_j(P_j^Y+W_j)\right]. \tag*{(170)} + W j ( P j Y + W j ) ] . (170)
The full two-axis CFG has ( ν o n , ν d e c ) = ( 2 , 4 ) (\nu_{\mathrm{on}},\nu_{\mathrm{dec}})=(2,4) ( ν on , ν dec ) = ( 2 , 4 ) . Prefix-off branches only allow keys from the current block, but if the kernel does not skip the masked historical blocks, the actual FLOPs must use A D e x e c A_D^{\mathrm{exec}} A D exec :
C D , a t t n M D M ≈ 4 n D d D q A D e x e c . (171) C_{D,\mathrm{attn}}^{\mathrm{MDM}} \approx4n_Dd_D^q A_D^{\mathrm{exec}}. \tag*{(171)} C D , attn MDM ≈ 4 n D d D q A D exec . (171)
C h e a d / i n t e r f a c e M D M C_{\mathrm{head/interface}}^{\mathrm{MDM}} C head/interface MDM includes all actual logit projections and one proposal duration/expansion/condition refinement. The recomputation of the corrected condition is attributed to feedback and is not double-counted here.
Fixed-span feedback and corrected condition commit. Each finalized block is re-encoded with fixed spans, without rerunning the dynamic segmenter. Let N F , t N_{F,t} N F , t and N F , z N_{F,z} N F , z be the numbers of linear-layer parameters of the span-local token pooling and the causal latent encoder, and Q F , t Q_{F,t} Q F , t and Q F , z Q_{F,z} Q F , z the numbers of positions they actually process. Then
C f b , e n c ≈ 2 N F , t Q F , t + 2 N F , z Q F , z + C f b , a t t n + C f b , p r o j , (172) \begin{aligned}
C_{\mathrm{fb,enc}} &\approx2N_{F,t}Q_{F,t}+2N_{F,z}Q_{F,z}\\
&\quad+C_{\mathrm{fb,attn}}+C_{\mathrm{fb,proj}}, \tag*{(172)}
\end{aligned} C fb , enc ≈ 2 N F , t Q F , t + 2 N F , z Q F , z + C fb , attn + C fb , proj , (172)
where C f b , p r o j C_{\mathrm{fb,proj}} C fb , proj includes projections such as the posterior head that are not counted in the above parameters; ignoring boundaries and padding, Q F , t ≈ T n e w Q_{F,t}\approx T_{\mathrm{new}} Q F , t ≈ T new and Q F , z ≈ K n e w Q_{F,z}\approx K_{\mathrm{new}} Q F , z ≈ K new . A token-encoder depth of zero does not eliminate the computation of span-local pooling. Its valid attention pairs accumulate as ∑ k L k 2 \sum_k L_k^2 ∑ k L k 2 over spans, and the latent encoder is charged according to the causal prefix/current layout.
After obtaining Z c o r r Z^{\mathrm{corr}} Z corr , the corrected condition-stack KV must also be rebuilt and committed; denoting its cumulative cost by C c o n d , c o m m i t C_{\mathrm{cond,commit}} C cond , commit , we have C f b = C f b , e n c + C c o n d , c o m m i t C_{\mathrm{fb}}=C_{\mathrm{fb,enc}}+C_{\mathrm{cond,commit}} C fb = C fb , enc + C cond , commit . The proposal condition preparation and the corrected condition commit are two distinct computations, but neither is multiplied by four for the four CFG branches; the corrected DiT commit is already included in C G i n f e r C_G^{\mathrm{infer}} C G infer .
Unified AR comparison convention. AR likewise excludes the one-time prefill. Let Q A R Q_{\mathrm{AR}} Q AR be the number of positions actually processed by its incremental backbone, A A R e x e c A_{\mathrm{AR}}^{\mathrm{exec}} A AR exec the executed attention pairs, and C h e a d , A R i n f e r C_{\mathrm{head,AR}}^{\mathrm{infer}} C head , AR infer the output-projection cost. Then
C A R i n f e r ≈ 2 N A R Q A R + 4 n A R d A R q A A R e x e c + C h e a d , A R i n f e r . (173) \begin{aligned}
C_{\mathrm{AR}}^{\mathrm{infer}} &\approx2N_{\mathrm{AR}}Q_{\mathrm{AR}}+4n_{\mathrm{AR}}d_{\mathrm{AR}}^q A_{\mathrm{AR}}^{\mathrm{exec}}\\
&\quad+C_{\mathrm{head,AR}}^{\mathrm{infer}}. \tag*{(173)}
\end{aligned} C AR infer ≈ 2 N AR Q AR + 4 n AR d AR q A AR exec + C head , AR infer . (173)
If the initial KV length is T p T_p T p , processing Q A R Q_{\mathrm{AR}} Q AR new queries consecutively yields Q A R T p + Q A R ( Q A R + 1 ) / 2 Q_{\mathrm{AR}}T_p+Q_{\mathrm{AR}}(Q_{\mathrm{AR}}+1)/2 Q AR T p + Q AR ( Q AR + 1 ) /2 valid pairs; Q A R ≈ T n e w Q_{\mathrm{AR}}\approx T_{\mathrm{new}} Q AR ≈ T new only ignores boundary differences in where logits are taken. Using denominators under the same convention, define
r G i n f e r = C G i n f e r C A R i n f e r , r D G D = C D i n f e r C A R i n f e r , r f b = C f b C A R i n f e r , (174) r_G^{\mathrm{infer}}=\frac{C_G^{\mathrm{infer}}}{C_{\mathrm{AR}}^{\mathrm{infer}}},\qquad
r_{\mathrm{DGD}}=\frac{C_D^{\mathrm{infer}}}{C_{\mathrm{AR}}^{\mathrm{infer}}},\qquad
r_{\mathrm{fb}}=\frac{C_{\mathrm{fb}}}{C_{\mathrm{AR}}^{\mathrm{infer}}}, \tag*{(174)} r G infer = C AR infer C G infer , r DGD = C AR infer C D infer , r fb = C AR infer C fb , (174)
so that
R i n f e r = C G i n f e r + C D i n f e r + C f b C A R i n f e r = r G i n f e r + r D G D + r f b . (175) R_{\mathrm{infer}}=\frac{C_G^{\mathrm{infer}}+C_D^{\mathrm{infer}}+C_{\mathrm{fb}}}{C_{\mathrm{AR}}^{\mathrm{infer}}}=r_G^{\mathrm{infer}}+r_{\mathrm{DGD}}+r_{\mathrm{fb}}. \tag*{(175)} R infer = C AR infer C G infer + C D infer + C fb = r G infer + r DGD + r fb . (175)
Using only the AR backbone linear term as reference and ignoring boundaries, the corresponding DiT contribution is
2 N G Q G p r o p + C G , l i n e a r c o m m i t 2 N A R T n e w ≈ ρ g e n ( ν g e n S g e n + r G c o m m i t ) N G N A R . (176) \frac{2N_GQ_G^{\mathrm{prop}}+C_{G,\mathrm{linear}}^{\mathrm{commit}}}{2N_{\mathrm{AR}}T_{\mathrm{new}}}
\approx\rho_{\mathrm{gen}}\left(\nu_{\mathrm{gen}}S_{\mathrm{gen}}+r_G^{\mathrm{commit}}\right)\frac{N_G}{N_{\mathrm{AR}}}. \tag*{(176)} 2 N AR T new 2 N G Q G prop + C G , linear commit ≈ ρ gen ( ν gen S gen + r G commit ) N AR N G . (176)
This is not the total inference ratio. When end-to-end initialization is included, the numerator and denominator of Eq. (H.25) should add the prefill costs of Cola2 and AR, respectively.
Serial Call Rounds, Latency, and Block Parallelism
We use H \mathcal{H} H as a forward-round proxy, not as Transformer layer depth or actual latency. By convention, one module forward pass counts as one round, and independent CFG branches at the same step count as one group under ideal parallelism. Let H D , j H_{D,j} H D , j be the number of call rounds for condition preparation, token decoding, and clean-token commit of the j j j -th block, and H F , j H_{F,j} H F , j the number of rounds for the feedback encoder and the corrected condition/DiT commit. Then
H C o l a 2 ≈ ∑ j = 1 J ( S g e n + H D , j + H F , j ) . (177) \mathcal{H}_{Cola2} \approx\sum_{j=1}^{J} \left(S_{\mathrm{gen}} + H_{D,j} + H_{F,j}\right). \tag*{(177)} H C o l a 2 ≈ j = 1 ∑ J ( S gen + H D , j + H F , j ) . (177)
Ignoring boundaries, AR corresponds to H A R ≈ T n e w \mathcal{H}_{AR} \approx T_{\mathrm{new}} H A R ≈ T new . With H D H_D H D and H F H_F H F denoting the average per-block numbers of rounds,
H C o l a 2 H A R ≈ ρ g e n B ( S g e n + H D + H F ) . (178) \frac{\mathcal{H}_{Cola2}}{\mathcal{H}_{AR}} \approx\frac{\rho_{\mathrm{gen}}}{B}\left(S_{\mathrm{gen}} + H_D + H_F\right). \tag*{(178)} H A R H C o l a 2 ≈ B ρ gen ( S gen + H D + H F ) . (178)
Thus, the condition under which this idealized call-round proxy is smaller is
B > ρ g e n ( S g e n + H D + H F ) , (179) B > \rho_{\mathrm{gen}}\left(S_{\mathrm{gen}} + H_D + H_F\right), \tag*{(179)} B > ρ gen ( S gen + H D + H F ) , (179)
which is not a sufficient condition for end-to-end speedup. The decoding part of MDM contains about M j M_j M j refinement groups and one clean commit. If the CFG branches are invoked sequentially, every forward pass must be counted, and they cannot be treated as one round merely because the formula is parallelizable.
Let τ i n i t \tau_{\mathrm{init}} τ init be the one-time initialization latency, τ G , j , i \tau_{G,j,i} τ G , j , i the latency of the i i i -th DiT update of the j j j -th block (including the actual CFG scheduling), and τ p r e p , j \tau_{\mathrm{prep},j} τ prep , j , τ D , j \tau_{D,j} τ D , j , and τ F , j \tau_{F,j} τ F , j the latencies of condition preparation, token decoding/commit, and feedback/corrected-cache commit, respectively. Under the current serial transaction,
τ C o l a 2 ≈ τ i n i t + ∑ j = 1 J [ ∑ i = 1 S g e n τ G , j , i + τ p r e p , j + τ D , j + τ F , j ] . (180) \tau_{Cola2} \approx\tau_{\mathrm{init}} + \sum_{j=1}^{J} \left[\sum_{i=1}^{S_{\mathrm{gen}}} \tau_{G,j,i} + \tau_{\mathrm{prep},j} + \tau_{D,j} + \tau_{F,j}\right]. \tag*{(180)} τ C o l a 2 ≈ τ init + j = 1 ∑ J i = 1 ∑ S gen τ G , j , i + τ prep , j + τ D , j + τ F , j . (180)
These latencies depend on module size, query/prefix length, branch batching, and hardware utilization, and should be measured on the corresponding execution paths; the same time cannot be assigned to every module forward pass.
KV Cache and Temporary States
Let K h i s t K_{\mathrm{hist}} K hist and T h i s t T_{\mathrm{hist}} T hist be the lengths of the committed latent/token histories, including the prompt. The DiT and feedback latent-encoder KV lie on the latent axis with length O ( K h i s t ) O(K_{\mathrm{hist}}) O ( K hist ) ; the condition-stack and token-decoder KV lie on the token axis with length O ( T h i s t ) O(T_{\mathrm{hist}}) O ( T hist ) . The MDM token cache is bidirectional within a block and causal across blocks.
For R ∈ { G , E , C , D } R \in\{G, E, C, D\} R ∈ { G , E , C , D } , denoting the DiT, the feedback latent encoder, the condition stack, and the token decoder, respectively, let n R n_R n R be the number of KV layers and d R K V = n k v , R d h e a d , R d_R^{\mathrm{KV}} = n_{\mathrm{kv},R}d_{\mathrm{head},R} d R KV = n kv , R d head , R the total KV-head width per layer, where n k v , R n_{\mathrm{kv},R} n kv , R and d h e a d , R d_{\mathrm{head},R} d head , R are the number of KV heads and the per-head width. For an equal-length batch of N s e q N_{\mathrm{seq}} N seq sequences with s K V s_{\mathrm{KV}} s KV bytes per cached element,
Mem K V ≈ 2 s K V N s e q [ ( n G d G K V + n E d E K V ) K h i s t + ( n C d C K V + n D d D K V ) T h i s t ] . (181) \operatorname{Mem}_{\mathrm{KV}} \approx2s_{\mathrm{KV}}N_{\mathrm{seq}}\left[(n_Gd_G^{\mathrm{KV}} + n_Ed_E^{\mathrm{KV}})K_{\mathrm{hist}} + (n_Cd_C^{\mathrm{KV}} + n_Dd_D^{\mathrm{KV}})T_{\mathrm{hist}}\right]. \tag*{(181)} Mem KV ≈ 2 s KV N seq [ ( n G d G KV + n E d E KV ) K hist + ( n C d C KV + n D d D KV ) T hist ] . (181)
The factor 2 accounts for keys and values; different lengths, KV layer shapes, or precisions are summed according to the actual allocation, and padding must also be counted. Under GQA, d R K V d_R^{\mathrm{KV}} d R KV should be used instead of the hidden width.
The four MDM CFG branches share one persistent token KV, so Eq. (H.31) is not multiplied by four; temporary logits, branch hidden states, and attention workspaces still affect peak memory. Proposals and masked states do not accumulate as history, but the sizes of cache concatenation, RoPE, and attention temporaries may still depend on the prefix length. This formula therefore counts only persistent KV and does not equal the total peak memory.
In summary, compression shortens the sequence and the latent-axis history of the latent prior, while continuous representation learning, token decoding, CFG, and feedback all retain their own costs. End-to-end comparisons should use a unified FLOP accounting scope and separately report actually processed positions, serial call rounds, latency, and KV capacity, rather than equating any single metric with overall speedup.
Final-Checkpoint Pretraining Recipe
This section describes the large-scale recipe used for the final checkpoint in Table 4 . It is separate from the matched component and controlled scaling setup in § 5.1 .
Table 4. Post-pretraining evaluation. Accuracy (%) on eight downstream tasks, with tasks grouped by evaluation focus and models by generation paradigm. Cola2 uses 346.03B source tokens; the models are not compute-matched.
Shared Training Configuration
The pretraining corpus uses OLMo 2 pretraining data retokenized with the Qwen3-4B tokenizer. The sequence length is 4096.
All stages use AdamW with β = ( 0.9 , 0.95 ) \beta= (0.9, 0.95) β = ( 0.9 , 0.95 ) and weight decay 0.01. The EMA decay in Stages II and III is 0.9998. At stage boundaries, model parameters, optimizer state, the learning-rate scheduler, and the data cursor are restored.
Stage I: Dynamic-Compression VAE
Stage I processes approximately 52.43B source tokens. Dynamic segmentation compresses about two tokens into one latent on average, with maximum span length 8 and latent dimension 64. The decoder is initialized from Qwen3-4B-Base, and the VAE encoder embedding is copied from the same model; the remaining compression modules are initialized from scratch.
The Qwen-initialized parameters remain frozen for the first 5.24B source tokens and are subsequently updated at 0.1 × 0.1\times 0.1 × the main learning rate; the compression encoder also uses a 0.1 × 0.1\times 0.1 × multiplier. The learning rate is linearly warmed up from 10 − 6 10^{-6} 1 0 − 6 to 10 − 4 10^{-4} 1 0 − 4 over the first 5.24B source tokens and then cosine-decayed to 10 − 6 10^{-6} 1 0 − 6 .
The decoder uses a causal dual-stream objective. Masked positions are supplied as [MASK] inputs, but the decoder self-attention remains causal rather than becoming bidirectional. Clean, noisy, and dropped latent conditions are sampled with proportions 0.4/0.3/0.3, and their three token-loss weights are all 1. The KL, duration, and encoder-mask auxiliary loss weights are 10 − 4 10^{-4} 1 0 − 4 , 0.1, and 1, respectively, and the encoder-mask span ratio is 0.15.
Stage II: Joint VAE and Latent-DiT Training
Stage II restores the complete VAE checkpoint, optimizer, scheduler, and data cursor from Stage I. It introduces an approximately 4B-parameter latent DiT with 34 Transformer layers, hidden dimension 2560, 20 attention heads, and latent block size 8. This stage processes approximately 125.83B source tokens.
The VAE continues to use the Stage-I causal dual-stream objective, with the encoder-mask span ratio increased to 0.30. The DiT reads detached VAE latents, so its diffusion loss does not backpropagate into the VAE; the VAE is updated only by its compression and decoder objectives. The DiT uses velocity-prediction MSE, with timesteps sampled from LogitNormal(loc = 2, scale = 1).
The DiT learning rate is warmed up from 10 − 6 10^{-6} 1 0 − 6 to 1.5 × 10 − 4 1.5 \times10^{-4} 1.5 × 1 0 − 4 and then cosine-decayed. The schedule is defined over a nominal 2.1T-token horizon, although training terminates before reaching that horizon.
Stage III: Frozen VAE and Continued DiT Training
Stage III restores the complete DiT parameters, EMA, optimizer, learning-rate scheduler, and data cursor from the Stage-II checkpoint. The Stage-II EMA VAE encoder is frozen, and only the DiT continues to be optimized. The released model uses the matching EMA VAE decoder associated with this encoder.
The timestep distribution remains LogitNormal(loc = 2, scale = 1). This stage processes approximately 167.77B source tokens from the retokenized OLMo 2 corpus.
Source-Token Accounting
Table 5. Final-checkpoint pretraining budget. Token counts report source-token positions processed in each stage.