betaFeedback

WorksComputer Science

Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation

Abstract

Hierarchical language generation separates continuous semantic modeling from discrete text realization, but its effectiveness depends on what the latent space compresses and how the discrete generator uses it. We propose Cola2, a framework that jointly studies continuous semantic compression, the balance between reconstruction and generation, and continuous-discrete co-adaptation. Cola2 dynamically segments text into semantic spans, controls the compression rate through auxiliary-loss-free negative feedback, and combines reconstruction and generation supervision to learn compact, predictive latent representations. A block-causal masked diffusion decoder is trained with clean, noisy, and dropped latent conditions, while classifier-free guidance adjusts the balance between semantic reconstruction and autonomous generation. Token-prediction gradients update the encoder through the latents, whereas Flow Matching prior learning is decoupled from representation learning by stop-gradient. During inference, generated text is re-encoded under a fixed span layout to keep the latent history consistent with the actual text context. Experiments on five downstream tasks with about 210 billion training tokens validate the core designs and show competitive, unsaturated scaling. With 346.03 billion source tokens, an unconverged checkpoint reaches a 60.10 average across eight tasks and surpasses Llama-3-8B on HellaSwag and StoryCloze despite its reported pretraining budget being over 43 times larger. These results demonstrate the potential of continuous-latent hierarchical language generation.

Introduction

★ Seedance Team Research Project ★

Introduction

Large language models (LLMs) have made remarkable progress under the autoregressive (AR) paradigm [6, 43], but the fixed left-to-right generation order imposes an inherent serial dependency on inference. Diffusion language models (DLMs) [1, 2, 34, 41] offer a new way to break this constraint. Cola DLM [13] further decomposes text generation into semantic organization in a continuous latent space and text realization in a discrete space; through latent prior modeling and conditional decoding, it provides a viable path toward hierarchical language generation beyond direct token modeling.

However, a structural division between continuous and discrete spaces does not automatically yield an effective division of labor. The key questions are what information the latent space should compress and how the discrete generator should use it. Shortening the latent sequence is not equivalent to obtaining a compact semantic representation, and a reconstruction-only objective does not guarantee that the latents can predict missing or future text [19, 24, 29]. Meanwhile, the decoder must not only faithfully realize the latent semantics, but also generate the remaining content from the text context when the latent condition is incomplete or imprecise [44]. Semantic compression and discrete generation therefore cannot be designed independently: the information the latents should retain must be determined jointly with the information the decoder can complete.

This interdependence further raises the question of how continuous and discrete representations should be learned jointly. Fixing the encoder in advance determines the information carried by the latents before the discrete generator is fully learned, limiting the continual adaptation of the representation to decoding needs. At the same time, letting the latent-prior fitting objective directly shape the encoder may push the latents toward directions that are easier to fit but not necessarily more beneficial for discrete generation. Hierarchical language modeling thus requires not only learning continuous representations and a discrete generator, but also specifying which objectives should jointly shape the representation and which learning processes should remain decoupled.

To address these issues, we propose Cola2, which jointly studies continuous semantic compression, the discrete reconstruction–generation balance, and their co-adaptation within a unified hierarchical language modeling framework. Its core idea is that, under limited latent capacity, the actual demands of text reconstruction and generation jointly determine what the continuous representation should retain, while the discrete generator handles the residual uncertainty that the latents do not explain. Specifically, Cola2 dynamically segments text into semantic spans, controls the compression rate with an auxiliary-loss-free negative-feedback mechanism, and combines reconstruction and generation supervision to learn compact and predictive latent representations. On the discrete side, a block-causal masked diffusion model (MDM) decoder [1, 41] is jointly trained under clean, noisy, and dropped latent conditions, balancing semantic reconstruction with context-based autonomous generation, and classifier-free guidance (CFG) [15] adjusts this balance at inference.

During training, the token-prediction loss is backpropagated through the latents to the encoder, so that the continuous representation and discrete generation co-adapt; the Diffusion Transformer (DiT) [38] learns the current latent distribution via Flow Matching [26] without backpropagating gradients to the encoder, thereby decoupling prior fitting from representation learning. At inference, the model re-encodes the generated text block by block under the fixed span layout, keeping the latent history consistent with the actual text context and aligning the two spaces throughout generation.

Building on these ideas, we systematically study co-adaptive hierarchical language generation across continuous and discrete spaces from three perspectives: probabilistic modeling, representation learning, and generation performance. Our main contributions are as follows:

  • We propose Cola2, a co-adaptive hierarchical language modeling framework. From the perspective of conditional information decomposition, we analyze semantic compression, the reconstruction–generation balance, and continuous–discrete co-adaptation in a unified manner, clarifying the relation between the information retained by the latents and the residual left to discrete generation.

  • We develop a generation-oriented joint learning mechanism for continuous and discrete representations. It combines dynamic semantic compression, generation-oriented supervision, and training under latent

conditions of varying reliability, and coordinates representation learning with prior fitting through gradient decoupling, so that compact semantic representations and discrete generation capability co-evolve.

  • We validate the core designs and their scaling potential through systematic experiments. Controlled five-task scaling to about 210B tokens and a separate 346.03B-source-token pretraining run show competitive, unsaturated scaling. The latter reaches a 60.10 average over eight tasks and surpasses Llama-3-8B on HellaSwag and StoryCloze despite over 43× fewer reported tokens, demonstrating decoder capability transfer and compatibility with autoregressive and masked-diffusion token-space generators.

Related Work

Semantic Compression and Hierarchical Language Models

Fixed-granularity methods shorten the modeled sequence through chunking or downsampling [33, 48]. Adaptive compression methods dynamically aggregate bytes or tokens according to content [18, 36, 39]. Concept-level methods organize generation through sentence representations or concept prediction [3, 7]. In contrast, Cola2 maintains the target compression rate through negative-feedback control without an auxiliary compression-rate loss, and combines reconstruction and generation supervision to learn compact and predictive continuous semantic representations.

Token-Space Language Models

Autoregressive methods generate text serially, token by token [6, 11, 45]. Discrete diffusion methods predict multiple positions in parallel through iterative denoising [2, 27, 34, 41, 42, 47], and have been further extended to block-level generation and token editing [1, 4, 5]. Continuous-augmented methods use paired continuous states to assist discrete denoising [44, 51, 52]. In contrast, Cola2 learns a semantic prior in a compressed latent space and then performs conditional generation with a discrete decoder, handling semantic modeling and text realization hierarchically.

Continuous Representations and Latent-Space Language Models

Token-aligned methods perform diffusion or flow modeling on continuous representations [8, 10, 12, 16, 22, 23, 46]. Autoencoder-based latent-space methods separate latent generation from text decoding [28, 29, 50]. Joint-learning methods further co-optimize the latent space and the generative model [13, 30]. In contrast, Cola2 drives representation learning with discrete reconstruction and generation supervision while isolating the prior-fitting gradient, so that semantic compression and the decoder’s reconstruction–generation balance are optimized jointly.

Cola2: Continuous Latent Diffusion for Hierarchical Text Modeling

We formulate hierarchical text generation as a continuous latent-variable model and, from an information-decomposition perspective, discuss semantic compression, the reconstruction–generation balance, and continuous–discrete co-adaptation (§ 4.1). We then present the unified model, training procedure, latent-feedback inference procedure, and computational cost (§ 4.2–§ 4.3).

Probabilistic and Information-Theoretic Perspective

Hierarchical Probabilistic Modeling

Let CC denote the observed context, YY the text to be generated, and Z=(z1,…,zK)Z = (z_1,\ldots,z_K) a continuous latent sequence with zk∈Rdzz_k \in\mathbb{R}^{d_z}. If YY contains TT tokens, the compression rate is ρ=K/T\rho= K/T. Cola2 factorizes the text probability into a continuous latent prior and discrete conditional generation:

p(Y∣C)=∫pθ(Z∣C)pψ(Y∣C,Z) dZ.(1)p(Y \mid C) = \int p_\theta(Z \mid C) p_\psi(Y \mid C, Z)\,\mathrm{d}Z. \tag*{(1)}

During training, introducing the encoder qϕ(Z∣C,Y)q_{\phi}(Z \mid C,Y) as an approximate posterior yields the conditional evidence lower bound (ELBO)

LcELBO=Eqϕ[log⁡pψ(Y∣C,Z)]−KL⁡(qϕ(Z∣C,Y)∥pθ(Z∣C))≤log⁡p(Y∣C).(2)\mathcal{L}_{\mathrm{cELBO}} = \mathbb{E}_{q_{\phi}}[\log p_{\psi}(Y \mid C,Z)] - \operatorname{KL}(q_{\phi}(Z \mid C,Y) \mathbin{\|} p_{\theta}(Z \mid C)) \le\log p(Y \mid C). \tag*{(2)}

Defining the conditional aggregated posterior

qˉϕ(Z∣C)=∫qϕ(Z∣C,Y)pdata(Y∣C) dY,(3)\bar{q}_{\phi}(Z \mid C) = \int q_{\phi}(Z \mid C,Y) p_{\mathrm{data}}(Y \mid C)\,\mathrm{d}Y, \tag*{(3)}

the average negative ELBO decomposes as

−LcELBO=E[−log⁡pψ(Y∣C,Z)]+I(Y;Z∣C)+ECKL⁡(qˉϕ(Z∣C)∥pθ(Z∣C)).(4)-\mathcal{L}_{\mathrm{cELBO}} = \mathbb{E}[-\log p_{\psi}(Y \mid C,Z)] + I(Y;Z \mid C) + \mathbb{E}_{C}\operatorname{KL}(\bar{q}_{\phi}(Z \mid C) \mathbin{\|} p_{\theta}(Z \mid C)). \tag*{(4)}

Thus, the encoder determines which information is retained within the limited latent capacity, the discrete generator decoder implements pψ(Y∣C,Z)p_{\psi}(Y \mid C,Z), and the latent DiT only needs to fit the pθ(Z∣C)p_{\theta}(Z \mid C) induced by the online encoder.

Semantic Compression and Information Density

Given CC, the uncertainty of the target text satisfies

H(Y∣C)=I(Y;Z∣C)+H(Y∣C,Z).(5)H(Y \mid C) = I(Y;Z \mid C) + H(Y \mid C,Z). \tag*{(5)}

If SS denotes a high-level semantic variable determined by YY, the chain rule of mutual information gives

H(Y∣C)=I(S;Z∣C)+I(Y;Z∣S,C)+H(Y∣C,Z).(6)\begin{aligned} H(Y \mid C) ={}& I(S;Z \mid C) + I(Y;Z \mid S,C) \\ &+ H(Y \mid C,Z). \tag*{(6)} \end{aligned}

Under the compression constraint K=ρTK = \rho T, whether the latents are worth further modeling in continuous space depends not only on whether KK is sufficiently short, but more importantly on whether the information retained in the limited latent positions is semantic and predictive. To characterize this, we define the analytical quantity

Dsem:=I(S;Z∣C)K=I(S;Z∣C)ρT,(7)\mathcal{D}_{\mathrm{sem}} := \frac{I(S;Z \mid C)}{K} = \frac{I(S;Z \mid C)}{\rho T}, \tag*{(7)}

which measures the density of high-level semantic information carried per latent position. This quantity characterizes the design objective rather than serving as a loss estimated directly during training.

Furthermore, partitioning YY into a visible part YvisY^{\mathrm{vis}} and a target part YtarY^{\mathrm{tar}} gives

I(Ytar;Z∣C,Yvis)=H(Ytar∣C,Yvis)−H(Ytar∣C,Yvis,Z).(8)I(Y^{\mathrm{tar}};Z \mid C,Y^{\mathrm{vis}}) = H(Y^{\mathrm{tar}} \mid C,Y^{\mathrm{vis}}) - H(Y^{\mathrm{tar}} \mid C,Y^{\mathrm{vis}},Z). \tag*{(8)}

Therefore, beyond local reconstructability, a high-quality latent space should also reduce the conditional uncertainty of future or missing tokens, i.e., concentrate more generative information within its limited capacity. Section 3.2 instantiates this principle through dynamic semantic compression and generation-oriented latent supervision.

Reconstruction–Generation Balance and Continuous–Discrete Co-adaptation

Let Z~\widetilde{Z} be the latent condition obtained by passing ZZ through a stochastic degradation channel, satisfying the conditional Markov chain Y→Z→Z~Y \to Z \to\widetilde{Z} given CC. By the data processing inequality,

I(Y;Z∣C)≥I(Y;Z~∣C),H(Y∣C,Z)≤H(Y∣C,Z~).(9)I(Y;Z \mid C) \ge I(Y;\widetilde{Z} \mid C), \qquad H(Y \mid C,Z) \le H(Y \mid C,\widetilde{Z}). \tag*{(9)}

When the latent condition is intact, the decoder’s main task is to realize the continuous semantics as the corresponding discrete tokens; as the condition is perturbed or removed, H(Y∣C,Zϕ)H(Y \mid C, Z_\phi) increases, and more uncertainty must be handled by the decoder’s own discrete prediction capability. This gives an information-theoretic form of the reconstruction–generation trade-off.

On the other hand, since H(Y∣C)H(Y \mid C) is determined by the data distribution, changing the encoder effectively reallocates

H(Y∣C)=I(Y;Zϕ∣C)+H(Y∣C,Zϕ).(10)H(Y \mid C) = I(Y; Z_\phi\mid C) + H(Y \mid C, Z_\phi). \tag*{(10)}

If EϕE_\phi is fixed before the decoder is trained, this allocation is determined before the discrete generator is learned. Cola2 instead jointly optimizes

(ϕ⋆,ψ⋆)=arg⁡min⁡ϕ,ψ Epdata(C,Y)Eqϕ(Z∣C,Y)[−log⁡pψ(Y∣C,Z)]s.t.KT≈ρ⋆,(11)(\phi^\star, \psi^\star) = \underset{\phi,\psi}{\arg\min}\ \mathbb{E}_{p_{\mathrm{data}}(C,Y)}\mathbb{E}_{q_\phi(Z\mid C,Y)}[-\log p_\psi(Y\mid C,Z)] \quad\text{s.t.} \quad\frac{K}{T} \approx\rho^\star, \tag*{(11)}

so that token prediction retains a direct gradient to the encoder:

∇ϕLtok=∂Ltok∂Z∂Z∂ϕ≠0.(12)\nabla_\phi\mathcal{L}_{\mathrm{tok}} = \frac{\partial\mathcal{L}_{\mathrm{tok}}}{\partial Z}\frac{\partial Z}{\partial\phi} \ne0. \tag*{(12)}

This lets the continuous latent space and the discrete realization co-adapt: the encoder learns semantic representations better suited to reconstruction and generation by the actual decoder, while latent-prior fitting is decoupled from this representation learning. The concrete mechanisms for compression, reliability training, gradient routing, and inference alignment are given in the next section.

Workflow

Figure 1 instantiates the above information decomposition with three coordinated modules: the compression variational autoencoder (VAE) encoder EϕE_\phi learns continuous latents, the discrete generator decoder DψD_\psi realizes the latent condition as discrete tokens, and the block-causal DiT GθG_\theta learns the generative distribution of the detached online latents. Their relationship can be summarized as

Figure 1. Method overview. Stage 1 jointly trains the compression encoder and the reconstruction–generation balanced decoder; Stage 2 additionally trains a DiT on detached latents; at inference, the DiT generates latents block by block and the decoder decodes them into tokens.

(C,Y)→EϕZ,(C,Z)→DψY,sg⁡(Z)→LFMGθ.(13)(C,Y) \xrightarrow{E_\phi} Z,\qquad(C,Z) \xrightarrow{D_\psi} Y,\qquad\operatorname{sg}(Z) \xrightarrow{\mathcal{L}_{\mathrm{FM}}} G_\theta. \tag*{(13)}

Here DψD_\psi is a block-causal masked-diffusion decoder (MDM), which is causal across blocks and performs bidirectional token denoising within the current block. Stage 1 jointly learns EϕE_\phi and DψD_\psi; Stage 2 retains this pathway and additionally trains a detached latent DiT. Full implementation details are given in Appendix § 15.

Model Architecture

Compression VAE encoder. The target sequence Y=(y1,…,yT)Y = (y_1,\ldots,y_T) is dynamically partitioned into KK contiguous spans. Let SkS_k denote the token index set of the kk-th span and Lk=∣Sk∣L_k = |S_k| its length. Then

⋃k=1KSk={1,…,T},1≤Lk≤Lmax⁡,ρ=KT.(14)\bigcup_{k=1}^{K} S_k = \{1,\ldots,T\},\qquad1 \le L_k \le L_{\max},\qquad\rho= \frac{K}{T}. \tag*{(14)}

The encoder EϕE_\phi first performs dynamic segmentation over token representations, then aggregates bidirectionally within each span and models causally along the latent axis, yielding a diagonal Gaussian posterior

qϕ(zk∣C,Y)=N(μk,diag⁡(σk2)),zk=μk+σk⊙ϵk,ϵk∼N(0,I).(15)q_\phi(z_k \mid C,Y) = \mathcal{N}\left(\mu_k,\operatorname{diag}(\sigma_k^2)\right),\qquad z_k = \mu_k + \sigma_k \odot\epsilon_k,\qquad\epsilon_k \sim\mathcal{N}(0,I). \tag*{(15)}

The compression rate is maintained near the target ρ⋆\rho^\star by an independent, gradient-free rate-bias controller rather than learned through an additional differentiable rate loss. The dynamic boundaries, span-local pooling, and convergence properties of the controller are given in Appendix § 15.§ 2.

Latent-to-token interface and block-causal MDM decoder. Each latent is first projected into the decoder hidden space, and a duration head predicts the number of tokens it covers:

pψ(Lk∣zk)=softmax⁡(MLP⁡dur(Winzk)).(16)p_{\psi}(L_k \mid z_k) = \operatorname{softmax}\left(\operatorname{MLP}_{\mathrm{dur}}(W_{\mathrm{in}}z_k)\right). \tag*{(16)}

Given the span lengths, the latents are expanded into a token-aligned condition HZ=(h1Z,…,hTZ)H^Z = (h_1^Z, \ldots, h_T^Z). If g(t)=kg(t) = k indicates that token tt belongs to span kk, then

References

  1. [1]Marianne Arriola, Aaron Gokaslan, Justin Chiu, Zhihan Yang, Zhiyuan Qi, Jiaqi Han, Subham Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations, volume 2025, pages 50726–50753, 2025.arxiv.org/abs/2503.09573
  2. [2]Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981–17993, 2021.arxiv.org/abs/2107.03006
  3. [3]Loïc Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alstruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R Costa-jussà, David Dale, et al. Large concept models: Language modeling in a sentence representation space. arXiv preprint arXiv:2412.08821, 2024.arxiv.org/abs/2412.08821
  4. [4]Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, et al. Llada2. 0: Scaling up diffusion language models to 100b. arXiv preprint arXiv:2512.15745, 2025.arxiv.org/abs/2512.15745
  5. [5]Tiwei Bie, Maosong Cao, Xiang Cao, Bingsen Chen, Fuyuan Chen, Kun Chen, Lun Du, Daozhuo Feng, Haibo Feng, Mingliang Gong, et al. Llada2. 1: Speeding up text diffusion via token editing. arXiv preprint arXiv:2602.08676, 2026.arxiv.org/abs/2602.08676
  6. [6]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.arxiv.org/abs/2005.14165
  7. [7]Jiaqi Cao, Chiyu Chen, Shuang Cheng, Xu Cheng, Beiya Dai, Yufan Feng, Kewen Ge, Ruijun Ge, Jiayi Huang, Yang Jiao, et al. Ncp-archpreview technical report: Moving towards latent space language models through next concept prediction. arXiv preprint arXiv:2609.10715, 2026.arxiv.org/abs/2609.10715
  8. [8]Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, and Ge Liu. Langflow: Continuous diffusion rivals discrete in language modeling. arXiv preprint arXiv:2604.11748, 2026.arxiv.org/abs/2604.11748
  9. [9]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
  10. [10]Shanshan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933, 2022.arxiv.org/abs/2210.08933
  11. [11]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  12. [12]Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36:16693–16715, 2023.arxiv.org/abs/2305.18619
  13. [13]Hongcan Guo, Qinyu Zhao, Yian Zhao, Shen Nie, Rui Zhu, Qiushan Guo, Feng Wang, Tao Yang, Hengshuang Zhao, Guoqiang Wei, et al. Continuous latent diffusion language model. arXiv preprint arXiv:2605.06548, 2026.arxiv.org/abs/2605.06548
  14. [14]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  15. [15]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.arxiv.org/abs/2207.12598
  16. [16]Keya Hu, Linlu Qiu, Yiyang Lu, Hanhong Zhao, Tianhong Li, Yoon Kim, Jacob Andreas, and Kaiming He. Elf: Embedded language flows. arXiv preprint arXiv:2605.10938, 2026.arxiv.org/abs/2605.10938
  17. [17]Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Daiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2.5-Coder technical report. arXiv preprint arXiv:2409.12186, 2024.arxiv.org/abs/2409.12186
  18. [18]Sukiun Hwang, Brandon Wang, and Albert Gu. Dynamic chunking for end-to-end hierarchical sequence modeling. In International Conference on Learning Representations, volume 2026, pages 149273–149313, 2026.arxiv.org/abs/2507.07955
  19. [19]Jiaxiu Jiang, Jingjing Ren, Wenbo Li, Bo Wang, Haoze Sun, Yijun Yang, Jianhui Liu, Yanbing Zhang, Shenghe Zheng, Yuan Zhang, Haoyang Huang, Nan Duan, and Wangmeng Zuo. TextLDM: Language modeling with continuous latent diffusion. arXiv preprint arXiv:2605.07748, 2026.arxiv.org/abs/2605.07748
  20. [20]Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018.arxiv.org/abs/1712.07040
  21. [21]Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 785–794, 2017.arxiv.org/abs/1704.04683
  22. [22]Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah, Jerry Huang, Aditi Raghunathan, Seunghoon Hong, Nicholas M Boffi, and Jinwoo Kim. Flow map language models: One-step language modeling via continuous denoising. arXiv preprint arXiv:2602.16813, 2026.arxiv.org/abs/2602.16813
  23. [23]Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. Advances in neural information processing systems, 35:4328–4343, 2022.arxiv.org/abs/2205.14217
  24. [24]Jiajun Liang, Yucheng Liao, Yukang Cao, Jiazhe Wei, Ken Li, Wende Tan, Jiankun Zhang, ZY Cui, Jingkang Yang, Liucheng Guo, Shiqi Yang, B. Yang, Caifeng Shan, Zewi Liu, and Chenyang Si. AURORA-LM: Autoencoding unified representation for continuous-latent diffusion language modeling. arXiv preprint arXiv:2608.02602, 2026.arxiv.org/abs/2608.02602
  25. [25]Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, 2022.arxiv.org/abs/2109.07958
  26. [26]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02747
  27. [27]Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834, 2023.arxiv.org/abs/2310.16834
  28. [28]Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. Latent diffusion for language generation. Advances in Neural Information Processing Systems, 36:56998–57025, 2023.arxiv.org/abs/2212.09462
  29. [29]Viacheslav Meshchaninov, Egor Chimbulatov, Alexander Shabalin, Aleksandr Abramov, and Dmitry Vetrov. Cosmos: Compressed and smooth latent space for text diffusion modeling. Advances in Neural Information Processing Systems, 38:14271–14299, 2026.arxiv.org/abs/2506.21170
  30. [30]Viacheslav Meshchaninov, Alexander Shabalin, Egor Chimbulatov, Nikita Gushchin, Ilya Koziev, Alexander Korotin, and Dmitry Vetrov. How to train your latent diffusion language model jointly with the latent space. arXiv preprint arXiv:2605.07933, 2026.arxiv.org/abs/2605.07933
  31. [31]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391, 2018.arxiv.org/abs/1809.02789
  32. [32]Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, 2016.DOI
  33. [33]Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski, Łukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1559–1571, 2022.arxiv.org/abs/2110.13711
  34. [34]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongyuan Li. Large language diffusion models. Advances in Neural Information Processing Systems, 38:50608–50646, 2026.arxiv.org/abs/2502.09992
  35. [35]Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024.arxiv.org/abs/2501.00656
  36. [36]Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E Weston, Luke Zettlemoyer, et al. Byte latent transformer: Patches scale better than tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9238–9258, 2025.
  37. [37]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 1525–1534, 2016.arxiv.org/abs/1606.06031
  38. [38]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.arxiv.org/abs/2212.09748
  39. [39]Xingwei Qu, Shaowen Wang, Zihao Huang, Kai Hua, Fan Yin, Rui-Jie Zhu, Jundong Zhou, Qiyang Min, Zihao Wang, Yizhi Li, et al. Dynamic large concept models: Latent reasoning in an adaptive semantic space. arXiv preprint arXiv:2512.24617, 2025.arxiv.org/abs/2512.24617
  40. [40]Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 2383–2392, 2016.arxiv.org/abs/1606.05250
  41. [41]Subham S Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marrouquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136–130184, 2024.arxiv.org/abs/2406.07524
  42. [42]Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin Chiu, and Volodymyr Kuleshov. The diffusion duality. Proceedings of machine learning research, 267:52584, 2025.arxiv.org/abs/2506.10892
  43. [43]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.arxiv.org/abs/2302.13971
  44. [44]Roy Uziel, Omer Belhasin, Itay Levy, Akhiad Bercovich, Ran El-Yaniv, Ran Zilberstein, and Michael Elad. Crocodili: Continuous and robust conditioned diffusion for language. arXiv preprint arXiv:2603.02010, 2026.
  45. [45]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Cheng en Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  46. [46]Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, and John Thickstun. Continuous diffusion scales competitively with discrete diffusion for language. arXiv preprint arXiv:2605.18530, 2026.arxiv.org/abs/2605.18530
  47. [47]Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models, 2025. URL https://arxiv.org/abs/2508.15487.
  48. [48]Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems, 36:78808–78823, 2023.arxiv.org/abs/2305.07185
  49. [49]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Anli Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019.arxiv.org/abs/1905.07830
  50. [50]Yizhe Zhang, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Joshua Susskind, and Navdeep Jaitly. Planner: Generating diversified paragraph via latent language diffusion model. Advances in Neural Information Processing Systems, 36:80178–80190, 2023.arxiv.org/abs/2306.02531
  51. [51]Huangjie Zheng, Shanshan Gong, Ruixiang Zhang, Tianrong Chen, Jiatao Gu, Mingyuan Zhou, Navdeep Jaitly, and Yizhe Zhang. Continuously augmented discrete diffusion model for categorical generative modeling. In International Conference on Learning Representations, volume 2026, pages 34212–34242, 2026.arxiv.org/abs/2510.01329
  52. [52]Cai Zhou, Chenxiao Yang, Yi Hu, Chenyu Wang, Chubin Zhang, Muhan Zhang, Lester Mackey, Tommi Jaakkola, Stephen Bates, and Dinghuai Zhang. Coevolutionary continuous discrete diffusion: Make your diffusion language model a latent reasoner. arXiv preprint arXiv:2510.03206, 2025.arxiv.org/abs/2510.03206

Paper details

vlnc:W_39nd797a
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents