betaFeedback

WorksComputer Science

Dual-Manifold Variational Autoencoders

Abstract

In latent diffusion models, expanding the latent dimension of variational autoencoders (VAEs) is essential to push the upper bound of reconstruction quality. However, without explicit structural constraints, such unregularized expansion easily yields a rugged and ill-conditioned latent manifold, causing generated latents to drift off the manifold during generation (i.e., off-manifold problem). Recent works smooth this manifold by using external foundation models (e.g., DINOv2) as a topological regularizer. However, we argue that tying the latent space to frozen external priors inherently restricts generalization: bound by their pre-training distributions and semantic biases, they may fail to capture faithful visual details on out-of-distribution domains. Observing that low-dimensional latents naturally form a smoother, generation-friendly manifold, we propose dual-manifold VAE (DM-VAE) to leverage internal representations as a topological regularizer. Specifically, DM-VAE pairs a high-dimensional reconstructive branch with a low-dimensional branch that extracts a clean topological skeleton, anchoring the high-dimensional manifold to this smooth subspace. Building upon this framework, we further introduce topological representation alignment (TRA) to supervise intermediate generator features during training, and internal guidance with topological skeleton (IG_TS) to guide structural components while preserving high-frequency details at inference. On ImageNet-1K (256x256), our foundation-model-free DM-VAE achieves a state-of-the-art rFID of 0.12 and a generation FID50K of 1.28 (w/o CFG) and 1.20 (w/CFG), and further reaches 1.11 when orthogonally combined with DINOv2.

Introduction

Latent diffusion models (LDMs) [26, 32, 35, 38] employ a continuous-valued variational autoencoder (VAE) [18, 22] to project raw pixel images into a low-dimensional latent space. Operating within this compressed space, generative models (e.g., DiT [35]) learn to iteratively generate valid in-distribution latents from random noise, which are subsequently decoded back into the image domain by the VAE.

Specifically, the LDM generative modeling pipeline consists of two stages: 1) Reconstruction: In the reconstruction stage (cf., Figure 2(a)), the VAE first encodes images into latents (i.e., mapping image distribution to latent distribution) and then decodes these latents to reconstructed images. It usually jointly utilizes multiple training objectives to improve the visual quality of the reconstructed images, such as reconstruction [51], Kullback-Leibler (KL), and adversarial [12, 15] losses. 2) Generation: In the generation stage, generative models, e.g., flow-based models [30, 31, 32], learn the velocity field to transport the prior distribution (e.g., a Gaussian distribution) towards the latent distribution. The training objective typically minimizes the regression errors between the predicted velocities and the ground truth velocities. Crucially, since the generation stage operates within the latent space, the overall generative performance is upper-bounded by the reconstruction quality.

Figure 2. Comparison of reconstruction paradigms in LDMs. Unlike (a) vanilla VAEs [38] that suffer from unconstrained, rugged high-dimensional manifolds, or (b) foundation-model-aligned [49] and (c) foundation-model-based [52] VAEs that rely on external priors (e.g., Dino [34]) with limited generalization ability, (d) our foundation-model-free DM-VAE constructs an internal topological regularizer. Specifically, given an encoder E\mathcal{E} mapping the input to a high-dimensional latent zz and a main decoder D\mathcal{D} for reconstruction, we compress zz via Pdown\mathcal{P}_{\mathrm{down}} into a low-dimensional branch (comprising zlowz_{\mathrm{low}} and a lightweight decoder Dlow\mathcal{D}_{\mathrm{low}}). By enforcing the alignment between zz and the lifted z^\hat{z}, our method compels the high-dimensional manifold to inherit a smooth, generation-friendly geometry from zlowz_{\mathrm{low}}. (e) Smooth vs. rugged manifolds, where “smooth” denotes a geometry dominated by low-frequency principal structures with bounded high-frequency oscillations along normal directions (cf., Appendix § 9).

Therefore, to push the upper bound of generative performance, enhancing reconstruction quality is essential. Due to the VAE’s inherent information bottleneck [2], a straightforward approach is expanding the dimension of the latent space (e.g., channels) to further minimize the reconstruction error. However, it inevitably introduces excessive degrees of freedom to the manifold corresponding to the latent distribution (i.e., latent manifold). Since there is no explicit structural constraint on the underlying latent manifold, this unregularized channel expansion often yields a rugged and ill-conditioned manifold [49]. Consequently, during generation, unconstrained high-frequency components disrupt velocity learning and easily cause generated latents to drift off the manifold, known as the off-manifold problem [40].

To address this off-manifold issue, recent approaches have increasingly relied on external foundation models (e.g., DINOv2 [34], MAE [16]). They either align latents with their features [49] (cf., Figure 2(b)) or adopt these features directly as latents [39, 52] (cf., Figure 2(c)). Specifically, their pre-trained features encode semantics that form a “smooth” reference manifold, acting as a topological regularizer. These regularizers can effectively smooth the unconstrained latent manifold, easing subsequent generative training. However, tying the latent space to external priors inherently restricts the VAE’s generalization capability [3, 7]. Because frozen foundation models are bound by their pre-training distributions and semantic biases, they fail to capture faithful visual details on out-of-distribution (OOD) domains, resulting in severe reconstruction degradation (cf., Figure 1).

Qualitative comparison of out-of-distribution image reconstruction

Figure 1. Qualitative comparison of out-of-distribution image reconstruction. All methods are trained on ImageNet-1K. Without relying on external foundation models, our DM-VAE (1.66d64) exhibits superior generalization. Average PSNR [20] on above three samples: RAE [52] (16.32 dB), Ours (low branch) (27.19 dB), and Ours (high branch) (30.10 dB).

To this end, we propose a dual-manifold VAE (DM-VAE) framework to mitigate the off-manifold problem by leveraging internal representations as a topological regularizer.

Our design is motivated by a key observation: although low-dimensional latents sacrifice reconstruction fidelity, they consistently exhibit superior generative learnability [49] (e.g., SD-VAE [38] in Figure 3), reflecting an inherently smoother underlying manifold [48]. Building on this insight, DM-VAE branches a shared encoder into two coupled latent spaces (cf., Figure 2(d)): a high-dimensional space (zz) decoded by the main decoder for high-fidelity reconstruction, and a low-dimensional space (zlowz_{\mathrm{low}}) (projected via Pdown\mathcal{P}_{\mathrm{down}} and supervised by a lightweight decoder Dlow\mathcal{D}_{\mathrm{low}}) that captures a smooth topological “skeleton”. To prevent unconstrained optimization on the high-dimensional latent manifold, we lift zlowz_{\mathrm{low}} back to the high-dimensional space via Pup\mathcal{P}_{\mathrm{up}} to obtain z^\hat{z}, and minimize its discrepancy with zz. This dual-manifold alignment compels the high-dimensional latent space to inherit the smooth geometry of its low-dimensional counterpart, smoothing the rugged manifold (cf., Figure 2(e)).

rFID vs. gFID$_{10K}$ comparison for VA-VAE, SD-VAE, DM-VAE, and DM-VAE+IGT$_S$ on DiT-XL

Figure 3. rFID vs. gFID10K_{10K} comparison on DiT-XL (80 epochs). When scaling from d32 to d64, DM-VAE with IGTS_S effectively overcomes the reconstruction–generation trade-off without external foundation models.

Under our DM-VAE framework, the extracted topological skeleton naturally provides a structural prior for the generation stage. To further facilitate generation within this regularized latent space, we introduce topological representation alignment (TRA) as an additional supervision signal. Specifically, we align the intermediate hidden features of the generative model with the low-dimensional latents during training. Given that the low-dimensional manifold captures the principal structural components of its high-dimensional counterpart, this alignment serves as an explicit structural prior, guiding the model to learn a more accurate velocity field within the high-dimensional space.

Beyond benefiting the training, these aligned intermediate features also enable an effective guidance strategy at the inference stage, internal guidance with topological skeleton (IGTS_S). Although recent internal guidance (IG) [54] provides an effective paradigm by using intermediate features, its potential is bottlenecked when directly applied to the entire latent space, as it tends to damage high-frequency details and exacerbate the off-manifold problem. In contrast, since our intermediate features are explicitly aligned with the low-dimensional latents, they naturally capture the clean topological skeleton. Building on this, IGTS_S separates the predicted high-dimensional latents into the low-dimensional topological skeleton and high-frequency details. We then use the aligned intermediate features to guide only the skeleton part, and add the unchanged high-frequency details back. In this way, IGTS_S improves structural quality while keeping the generated latents on the manifold, all within a single forward pass.

To validate the effectiveness of our method, we have conducted extensive experiments on ImageNet-1K [10] at 256×256256 \times256 resolution using DiT [35], following the LightningDiT [49] training settings. As shown in Figure 3, our approach combined with IGTS_S breaks the reconstruction–generation dilemma when scaling to higher-dimensional latent spaces (d32 to d64), consistently outperforming other baselines. Without relying on external foundation models, our DM-VAE (f16d64) achieves a reconstruction FID (rFID) of 0.12 within 50 reconstructive training epochs and a 400-epoch generation FID50K_{50K} of 1.28 with IGTS_S alone (1.20 under CFG). Our f16d32 variant reaches 1.67 under CFG at only 64 epochs and further advances to 1.25 with IGTS_S alone (1.17 under CFG) at 400 epochs.

In summary, our main contributions are threefold: 1) We propose dual-manifold VAE (DM-VAE), a foundation-model-free VAE framework that regularizes rugged high-dimensional latent manifolds via a smooth low-dimensional topological skeleton. 2) We introduce topological representation alignment (TRA) for training and internal guidance with topological skeleton (IGTS_S) for inference, leveraging the skeleton for structural supervision and single-pass guidance while keeping high-frequency details intact. 3) Extensive experiments demonstrate superior reconstruction and generation performance in our DM-VAE framework.

Related Work

Variational Autoencoder (VAE). Continuous-valued VAEs [5, 11, 22, 37] serve as the cornerstone of latent diffusion models (LDMs) [8, 12, 13, 26, 28, 32, 35, 38]. While vanilla VAEs were originally designed as end-to-end generative models that decode images directly from a latent prior distribution, they suffer from a structural mismatch between this predefined prior and the aggregated posterior distribution [19, 42]. Consequently, direct sampling from the vanilla VAE prior often yields blurry and low-fidelity images. To address this, LDMs repurpose the VAE as a reconstructive compressor, mapping high-dimensional pixel data into a compact latent space to facilitate subsequent generative training. Typically, training reconstructive VAE relies on reconstruction losses (e.g., L1L_1 and LPIPS [51]) and adversarial [12, 15] loss, paired with a Kullback-Leibler (KL) divergence penalty to regularize the latent space. To preserve information for high reconstruction fidelity, the KL weight is usually set to a small value (e.g., β=10−6\beta= 10^{-6}), exerting minimal influence on the latent distribution. However, without further structural regularization beyond this weak KL constraint, the learned latent space is prone to off-manifold issues during generation.

Representation Alignment in LDMs. Incorporating external or internal representation alignment objectives in latent generative modeling has proven effective in accelerating the training process. External alignment approaches, such as VA-VAE [49], REPA [50], and REPA-E [27], leverage external foundation models (i.e., DINOv2 [34], MAE [16], CLIP [36], SigLIP [44]) to guide the alignment of either the latent space or the intermediate features of DiTs. However, relying on external foundation models can lead to generalization issues. To address this, internal alignment methods (e.g., SRA [21] and Self-Flow [7]) instead align representations derived from different views of the same training sample in generative training. Unlike these existing approaches, our proposed dual-manifold VAE can optimize the latent space itself through internal alignment, thereby further facilitating generative training.

Methodology

In this section, we first review the preliminary formulation of latent diffusion models (§ 3.1) and analyze the inherent off-manifold problem in unconstrained high-dimensional latent spaces. To mitigate this, we introduce the dual-manifold VAE framework that internalizes topological regularization without relying on external foundation models. We detail the dual-manifold alignment for the reconstruction stage (§ 3.2). Additionally, we introduce topological representation alignment (TRA) for the subsequent generation stage, along with an inference strategy, internal guidance with topological skeleton (IGTS_S) (§ 3.3).

Preliminaries: Latent Diffusion Models

Latent diffusion models (LDMs) decouple the image generation pipeline into two distinct stages: 1) In the reconstruction stage, given an image x∈Xx \in\mathcal{X}, a continuous-valued VAE utilizes its encoder E\mathcal{E} to map the image into a low-dimensional latent representation z=E(x)z = \mathcal{E}(x), and the decoder D\mathcal{D} to reconstruct the image x^=D(z)\hat{x} = \mathcal{D}(z). The reconstructive VAE is optimized using a combination of reconstruction loss Lrec\mathcal{L}_{\mathrm{rec}} (e.g., L1L_1 and LPIPS [51]), a Kullback–Leibler (KL) divergence penalty LKL\mathcal{L}_{\mathrm{KL}} towards a standard Gaussian prior, and an adversarial loss Ladv\mathcal{L}_{\mathrm{adv}} [12, 15]:

LVAE(x,x^)=Lrec(x,x^)+βLKL+λadvLadv(x^),(1)\mathcal{L}_{\mathrm{VAE}}(x, \hat{x}) = \mathcal{L}_{\mathrm{rec}}(x, \hat{x}) + \beta\mathcal{L}_{\mathrm{KL}} + \lambda_{\mathrm{adv}}\mathcal{L}_{\mathrm{adv}}(\hat{x}), \tag*{(1)}

where β\beta and λadv\lambda_{\mathrm{adv}} denote the loss weights. 2) In the generation stage, flow matching models operate within the constructed latent space Z\mathcal{Z}. Specifically, let z1=z∈Zz_1 = z \in\mathcal{Z} denote the encoded target latent and z0∼N(0,I)z_0 \sim\mathcal{N}(0, I) denote standard Gaussian noise. Flow matching constructs a probability path via a linear interpolation: zt=tz1+(1−t)z0z_t = tz_1 + (1-t)z_0 for t∈[0,1]t \in[0, 1]. To transport the prior distribution towards the latent distribution, the generative model learns a neural velocity field vθ(zt,t)v_\theta(z_t, t) by regressing the constant ground-truth vector field ut=z1−z0u_t = z_1 - z_0. The training objective minimizes:

LFM=Et,z0,z1[∥vθ(zt,t)−(z1−z0)∥2],(2)\mathcal{L}_{\mathrm{FM}} = \mathbb{E}_{t,z_0,z_1}\left[\left\|v_\theta(z_t,t) - (z_1-z_0)\right\|^2\right], \tag*{(2)}

where t∼U(0,1)t \sim\mathcal{U}(0, 1). If the underlying latent manifold is ill-conditioned, unconstrained high-frequency components disrupt velocity regression and cause generated latents to drift off the manifold (i.e., the off-manifold problem), degrading generation in high-dimensional latent spaces.

Dual-Manifold Variational Autoencoders

Off-Manifold Analysis. As discussed in § 1, the overall generative performance is inherently upper-bounded by reconstruction quality. To push this bound, increasing the latent channel dimension CC is a standard practice. From an information-theoretic perspective [1, 9], the minimum reconstruction distortion εrecon\varepsilon_{\mathrm{recon}} is limited by the latent representation capacity. Since the latent spatial resolution (H×W)(H \times W) is typically fixed, this capacity is dominated by CC, yielding an exponential theoretical bound1:

εrecon≥αexp⁡(−γ⋅C),\varepsilon_{\mathrm{recon}} \geq\alpha\exp(-\gamma\cdot C),

where α\alpha and γ\gamma are positive constants determined by the intrinsic dimension and signal variance of the raw image manifold (manifold assumption [6, 29]). This inequality explicitly demonstrates that expanding the channel dimension is the most direct approach to minimize reconstruction error. However, this unregularized expansion inevitably introduces excessive degrees of freedom to the latent manifold. Mapping the intrinsically dd-dimensional image manifold MX\mathcal{M}_{\mathcal{X}} into a high-dimensional space RC×H×W\mathbb{R}^{C \times H \times W} forms a latent manifold MZ\mathcal{M}_{\mathcal{Z}} with an exploding codimension c=C×H×W−dc = C \times H \times W - d. Without any constraint, the encoder tends to exploit these excessive degrees of freedom. It may scatter substantial high-frequency energy across the cc normal directions to minimize pixel and adversarial errors, yielding an ill-conditioned latent manifold. During flow matching, the high-variance high-frequency components severely interfere with low-frequency structural velocity learning (cf., Appendix § 9), causing the off-manifold problem.

Dual-Manifold Alignment. While VAEs with low-dimensional latents exhibit degraded reconstruction quality compared to their high-dimensional counterparts, they typically achieve better generation [49]. This indicates that the low-dimensional latent manifold is inherently smoother and more generation-friendly [48]. Motivated by this, a straightforward idea is to leverage the low-dimensional latent manifold as a topological regularizer, compelling the high-dimensional manifold to inherit this property.

To achieve this, we design a dual-manifold VAE framework that simultaneously constructs two latent spaces sharing the encoder E\mathcal{E}. Given an input image x∈Xx \in\mathcal{X}, the encoder first maps it into a high-dimensional latent z=E(x)∈RC×H×Wz = \mathcal{E}(x) \in\mathbb{R}^{C \times H \times W}, which is decoded by the main decoder D\mathcal{D} into a high-fidelity reconstruction x^=D(z)\hat{x} = \mathcal{D}(z). In parallel, we compress zz into a low-dimensional latent zlow∈RClow×H×Wz_{\mathrm{low}} \in\mathbb{R}^{C_{\mathrm{low}} \times H \times W} (Clow<C)(C_{\mathrm{low}} < C) via a linear down-projection layer Pdown\mathcal{P}_{\mathrm{down}} (i.e., a 1×11 \times1 convolution), followed by a lightweight decoder Dlow\mathcal{D}_{\mathrm{low}} to output a structural reconstruction:

zlow=Pdown(z),x^low=Dlow(zlow).(3)z_{\mathrm{low}} = \mathcal{P}_{\mathrm{down}}(z), \qquad\hat{x}_{\mathrm{low}} = \mathcal{D}_{\mathrm{low}}(z_{\mathrm{low}}). \tag*{(3)}

While the main branch (x^)(\hat{x}) is optimized with the full reconstructive objective LVAE(x,x^)\mathcal{L}_{\mathrm{VAE}}(x,\hat{x}), the low-dimensional branch (x^low)(\hat{x}_{\mathrm{low}}) is supervised solely by reconstruction loss Lrec(x,x^low)\mathcal{L}_{\mathrm{rec}}(x,\hat{x}_{\mathrm{low}}). Constrained by the bottleneck ClowC_{\mathrm{low}} and free from high-frequency adversarial training, zlowz_{\mathrm{low}} forms a smooth manifold that captures the clean topological “skeleton” from zz (i.e., principal structural components). Rather than allowing the high-dimensional manifold to be optimized without any restriction, we treat zlowz_{\mathrm{low}} as an internal topological regularizer. Concretely, we project zlowz_{\mathrm{low}} back into the high-dimensional space via a corresponding up-projection layer Pup\mathcal{P}_{\mathrm{up}} to obtain the projected topological skeleton z^=Pup(zlow)∈RC×H×W\hat{z} = \mathcal{P}_{\mathrm{up}}(z_{\mathrm{low}}) \in\mathbb{R}^{C \times H \times W}, and minimize the L2L_2 distance between zz and z^\hat{z} (i.e., dual-manifold alignment:

LDM=Ex[∥z−z^∥22]\mathcal{L}_{\mathrm{DM}}=\mathbb{E}_{x}\left[\lVert z-\hat{z}\rVert_{2}^{2}\right]. The total training objective is:

Ldual=LVAE(x,x^)+λlowLrec(x,x^low)+λDMLDM,(4)\mathcal{L}_{\mathrm{dual}}=\mathcal{L}_{\mathrm{VAE}}(x,\hat{x})+\lambda_{\mathrm{low}}\mathcal{L}_{\mathrm{rec}}(x,\hat{x}_{\mathrm{low}})+\lambda_{\mathrm{DM}}\mathcal{L}_{\mathrm{DM}}, \tag*{(4)}

where λlow\lambda_{\mathrm{low}} and λDM\lambda_{\mathrm{DM}} are coefficients. Through LDM\mathcal{L}_{\mathrm{DM}}, DM-VAE suppresses transverse high-frequency residual energy and confines zz within a narrow tubular neighborhood around lifted zlowz_{\mathrm{low}}, decoupling structural velocity learning from high-frequency interference and mitigating the off-manifold problem (cf., Appendix B).

Topology-Guided Generation and Inference

Topological Representation Alignment. Building on the dual-manifold VAE (DM-VAE) framework, we naturally utilize the low-dimensional latent zlowz_{\mathrm{low}} to guide generative training in the main high-dimensional latent space (cf., Figure 4). Specifically, given an interpolated latent ztz_t at timestep tt, we extract the intermediate hidden features hth_t from the generative model and project them into the low-dimensional space via a lightweight projection head Palign\mathcal{P}_{\mathrm{align}} (e.g., an MLP) to obtain z~low=Palign(ht)∈RClow×H×W\tilde{z}_{\mathrm{low}}=\mathcal{P}_{\mathrm{align}}(h_t)\in\mathbb{R}^{C_{\mathrm{low}}\times H\times W}. We then align z~low\tilde{z}_{\mathrm{low}} with the target topological skeleton zlow=Pdown(z1)z_{\mathrm{low}}=\mathcal{P}_{\mathrm{down}}(z_1) by maximizing their cosine similarity:

Figure 4. Overview of topological representation alignment (TRA) and internal guidance with topological skeleton (IGTS_S).

LTRA=−Et,z0,z1[zlow⋅z~low∥zlow∥2∥z~low∥2].(5)\mathcal{L}_{\mathrm{TRA}}=-\mathbb{E}_{t,z_0,z_1}\left[\frac{z_{\mathrm{low}}\cdot\tilde{z}_{\mathrm{low}}}{\lVert z_{\mathrm{low}}\rVert_2\lVert\tilde{z}_{\mathrm{low}}\rVert_2}\right]. \tag*{(5)}

The overall generative training objective is formulated as: Lgen=LFM+λTRALTRA\mathcal{L}_{\mathrm{gen}}=\mathcal{L}_{\mathrm{FM}}+\lambda_{\mathrm{TRA}}\mathcal{L}_{\mathrm{TRA}}, where λTRA\lambda_{\mathrm{TRA}} is a coefficient. Since zlowz_{\mathrm{low}} captures the principal structural components of z1z_1 on a smooth manifold, LTRA\mathcal{L}_{\mathrm{TRA}} compels the intermediate layers to establish a reliable structural representation early in the network. This explicit structural prior eases the subsequent regression of high-dimensional details, guiding the model to learn a more accurate velocity field vθ(zt,t)v_\theta(z_t,t).

Algorithm 1 Internal Guidance with Topological Skeleton

Require: Model vθv_\theta with Palign\mathcal{P}_{\mathrm{align}}, DM-VAE {Pdown,Pup,D}\{\mathcal{P}_{\mathrm{down}},\mathcal{P}_{\mathrm{up}},\mathcal{D}\}, scale ω≥1\omega\geq1, interval [tmin⁡,tmax⁡][t_{\min},t_{\max}], steps NN. Ensure: Image x^\hat{x}.

  1. Initialize z0∼N(0,I)z_0\sim\mathcal{N}(0,I), Δt←1/N\Delta t\leftarrow1/N

  2. for i=0,…,N−1i=0,\ldots,N-1 do

  3. t←i/Nt\leftarrow i/N; ωt←ω\omega_t\leftarrow\omega if t∈[tmin⁡,tmax⁡]t\in[t_{\min},t_{\max}] else 11

  4. % 1. Single forward pass for velocity and skeleton

  5. vθ(zt,t),ht←Forward⁡(zt,t)v_\theta(z_t,t),h_t\leftarrow\operatorname{Forward}(z_t,t); z~low←Palign(ht)\tilde{z}_{\mathrm{low}}\leftarrow\mathcal{P}_{\mathrm{align}}(h_t)

  6. z^1←zt+(1−t)vθ(zt,t)\hat{z}_1\leftarrow z_t+(1-t)v_\theta(z_t,t)

  7. % 2. Separate skeleton and high-frequency details

  8. z~low←Pdown(z^1)\tilde{z}_{\mathrm{low}}\leftarrow\mathcal{P}_{\mathrm{down}}(\hat{z}_1); z^1base←Pup(z~low)\hat{z}_1^{\mathrm{base}}\leftarrow\mathcal{P}_{\mathrm{up}}(\tilde{z}_{\mathrm{low}}); z^1detail←z^1−z^1base\hat{z}_1^{\mathrm{detail}}\leftarrow\hat{z}_1-\hat{z}_1^{\mathrm{base}}

  9. % 3. Calibrate intermediate norm and guide the skeleton

  10. z^1base←Pup(∥z~low∥2⋅z~low/∥z~low∥2)\hat{z}_1^{\mathrm{base}}\leftarrow\mathcal{P}_{\mathrm{up}}(\lVert\tilde{z}_{\mathrm{low}}\rVert_2\cdot\tilde{z}_{\mathrm{low}}/\lVert\tilde{z}_{\mathrm{low}}\rVert_2)

  11. z^1guided←z^1base+(ωt−1)(z^1base−z^1base)+z^1detail\hat{z}_1^{\mathrm{guided}}\leftarrow\hat{z}_1^{\mathrm{base}}+(\omega_t-1)(\hat{z}_1^{\mathrm{base}}-\hat{z}_1^{\mathrm{base}})+\hat{z}_1^{\mathrm{detail}}, Eq. (6)

  12. % 4. Update guided velocity and latent state

  13. vguided←(z^1guided−zt)/(1−t)v_{\mathrm{guided}}\leftarrow(\hat{z}_1^{\mathrm{guided}}-z_t)/(1-t); zt+Δt←zt+vguided⋅Δtz_{t+\Delta t}\leftarrow z_t+v_{\mathrm{guided}}\cdot\Delta t

  14. end for

  15. return x^=D(z1)\hat{x}=\mathcal{D}(z_1)

Internal Guidance with Topological Skeleton. Beyond benefiting generative training, the aligned intermediate prediction z~low\tilde{z}_{\mathrm{low}} also enables an effective inference strategy, internal guidance with topological skeleton (IGTS_S). As shown in Figure 4, at timestep tt, the model outputs both the predicted velocity vθ(zt,t)v_\theta(z_t,t) and z~low\tilde{z}_{\mathrm{low}} in a single forward pass, yielding the one-step target estimate z^1=zt+(1−t)vθ(zt,t)\hat{z}_1=z_t+(1-t)v_\theta(z_t,t). While standard internal guidance (IG) [54] extrapolates between final and intermediate predictions across the entire latent space and may corrupt high-frequency details, IGTS_S uses our dual-manifold VAE projections Pdown\mathcal{P}_{\mathrm{down}} and Pup\mathcal{P}_{\mathrm{up}} to separate z^1\hat{z}_1 into the topological skeleton z^1base\hat{z}_1^{\mathrm{base}} and high-frequency details z^1detail\hat{z}_1^{\mathrm{detail}}: z^1base=Pdown(z^1)\hat{z}_1^{\mathrm{base}}=\mathcal{P}_{\mathrm{down}}(\hat{z}_1), z^1base=Pup(z^low)\hat{z}_1^{\mathrm{base}}=\mathcal{P}_{\mathrm{up}}(\hat{z}_{\mathrm{low}}), z^1detail=z^1−z^1base\hat{z}_1^{\mathrm{detail}}=\hat{z}_1-\hat{z}_1^{\mathrm{base}}. Since z~low\tilde{z}_{\mathrm{low}} is trained only with cosine similarity, we calibrate its norm using z^low\hat{z}_{\mathrm{low}} and project it to obtain the intermediate skeleton z^1base\hat{z}_1^{\mathrm{base}}: z^1base=Pup(∥z~low∥2⋅z~low/∥z~low∥2)\hat{z}_1^{\mathrm{base}}=\mathcal{P}_{\mathrm{up}}(\lVert\tilde{z}_{\mathrm{low}}\rVert_2\cdot\tilde{z}_{\mathrm{low}}/\lVert\tilde{z}_{\mathrm{low}}\rVert_2). We then apply skeleton guidance (i.e., z^1guided=z^1base+(ω−1)(z^1base−z^1base)\hat{z}_1^{\mathrm{guided}}=\hat{z}_1^{\mathrm{base}}+(\omega-1)(\hat{z}_1^{\mathrm{base}}-\hat{z}_1^{\mathrm{base}})) and add the high-frequency details back:

z^1guided=z^1base+(ω−1)(z^1base−z^1base)+z^1detail,(6)\hat{z}_1^{\mathrm{guided}}=\hat{z}_1^{\mathrm{base}}+(\omega-1)(\hat{z}_1^{\mathrm{base}}-\hat{z}_1^{\mathrm{base}})+\hat{z}_1^{\mathrm{detail}}, \tag*{(6)}

where ω≥1\omega\geq1 is the internal guidance scale. The guided velocity is then updated as vguided(zt,t)=(z^1guided−zt)/(1−t)v_{\mathrm{guided}}(z_t,t)=(\hat{z}_1^{\mathrm{guided}}-z_t)/(1-t). In this way, IGTS_S improves structural quality while mitigating off-manifold extrapolation drift by leaving the high-frequency details unperturbed. The detailed IGTS_S process with time interval [25] is shown in Algorithm 1.

Experiments

Experimental Settings

Datasets, Baselines, and Evaluation Metrics. We evaluate both image reconstruction and class-conditional generation on ImageNet-1K [10] at a resolution of 256×256256\times256. Following the categorization in our main comparisons, we benchmark against two categories of baselines: 1) Vanilla latent diffusion models (without external foundation models), including DiT [35], MaskDiT [53], SiT [32], TREAD [23], and MDTv2 [14]; 2) Foundation-model-aligned and guided LDMs, which regularize either the VAE latent space (e.g., MAE-VAE and VA-VAE with LightningDiT [49], RAE [52], IG (VA-VAE) [54]) or the intermediate generative features (e.g., REPA [50], REPA-E [27], and REG [47]). To verify versatility across generative architectures, we integrate our DM-VAE and IGTSIGT_S into DiT [35], following the LightningDiT [49] training settings across both Base (B) and Extra-Large (XL) scales. For evaluation metrics, we adopt reconstruction FID (rFID), PSNR, LPIPS [51], and SSIM [46] on both the ImageNet-1K 50K validation set (in-domain) and the Anime-Faces-256 benchmark [4] (5K, OOD) for the first stage, and reported Fréchet Inception Distance (FID) [17] (FID10K_{10K} and FID50K_{50K}), spatial FID (sFID) [33], Inception Score (IS), and Precision-Recall [24] for generation quality.

Implementation Details. 1) Reconstruction stage: During reconstructive training, we employ adaptive weighting coefficients to dynamically balance the loss terms. For our foundation-model-free variant (DM-VAE), we train two configurations within 50 epochs with a downsampling factor of 16: f16d32 (C=32C = 32, low-dimensional branch Clow=16C_{\mathrm{low}} = 16) and f16d64 (C=64C = 64, low-dimensional branch Clow=32C_{\mathrm{low}} = 32). To demonstrate orthogonality with external representations, we also train a DINOv2-integrated [34] variant (DM-VAEDino_{\mathrm{Dino}}) under the f16d32 setting3. All training hyperparameters follow VA-VAE [49] (e.g., a global batch size of 256 and a learning rate of 1×10−41 \times10^{-4}). Detailed training settings are provided in Appendix § 10. 2) Generation stage: We conduct generative training on ImageNet-1K at a resolution of 256×256256 \times256 using DiT4, adhering to the official default training protocols [49, 54] (e.g., a global batch size of 1024 and a learning rate of 2×10−42 \times10^{-4}). We then employ a 250-step Euler solver for sampling.

Main Results

In-Domain and OOD Reconstruction. Table 1 compares reconstruction quality on ImageNet-1K (in-domain) and Anime-Faces-256 [4] (zero-shot OOD). 1) In-domain fidelity: Decoding from frozen semantic encoders in RAE [52] and RAEv2 [39] severely degrades pixel fidelity (19.20 and 23.29 dB PSNR), while external alignment in MAE-VAE and VA-VAE [49] (0.28 rFID) slightly trails vanilla SD-VAE (0.26 rFID). Conversely, our DM-VAE preserves local details and improves rFID to 0.21 (f16d32) and 0.12 (f16d64, 30.76 dB PSNR). 2) OOD generalization: On OOD anime illustrations, foundation-model-based tokenizers suffer from domain shift, with RAE/RAEv2 collapsing to 9.85/5.09 rFID and VA-VAE/MAE-VAE lagging behind SD-VAE in pixel and perceptual fidelity (29.78/29.33 dB vs. 30.19 dB PSNR). Free from external pretraining biases, DM-VAE generalizes robustly, achieving 1.93 rFID (30.30 dB PSNR) at f16d32 and 1.12 rFID (32.87 dB PSNR, 0.95 SSIM) at f16d64.

MethodsrFID↓PSNR↑LPIPS↓SSIM↑
In-domain reconstruction performance
MAE-VAE (f16d32) [49]0.2828.330.0910.80
VA-VAE (f16d32) [49]0.2827.960.0960.79
RAE [52]0.5719.200.2540.44
RAEv2 [39]0.3123.290.1480.61
SD-VAE (f16d32) [38]0.2628.590.0890.80
DM-VAE (f16d32)0.2128.500.0880.80
DM-VAE (f16d64)0.1230.760.0570.87
Out-of-distribution reconstruction performance
MAE-VAE (f16d32) [49]2.3829.780.0480.91
VA-VAE (f16d32) [49]2.0529.330.0520.90
RAE [52]9.8517.530.2390.50
RAEv2 [39]5.0922.200.1200.70
SD-VAE (f16d32) [38]2.0830.190.0460.92
DM-VAE (f16d32)1.9330.300.0460.92
DM-VAE (f16d64)1.1232.870.0270.95

Table 1. In-domain (ImageNet-1K) and out-of-distribution (OOD) reconstruction performance comparison. Without relying on external foundation models, our DM-VAE achieves state-of-the-art reconstruction fidelity across both benchmarks.

Integration with DiT. Table 2 evaluates reconstruction and generation performance on DiT-B and DiT-XL (80/160 epochs, without CFG). 1) Reconciling the reconstruction–generation dilemma without external priors: At f16d32, DM-VAE improves rFID over SD-VAE (0.26 → 0.21) while lowering FID10K_{10K} to 21.41/17.66 (B) and 8.89/7.97 (XL); pairing with IGTS_S further reaches 12.60/9.22 (B) and 5.16/4.20 (XL), surpassing DINOv2-aligned VA-VAE [49] (20.34/15.73 and 8.22/6.75). More importantly, when expanding to f16d64, vanilla SD-VAE suffers severe off-manifold degradation (FID10K_{10K} worsens to 17.24/12.38 on XL despite 0.17 rFID). Conversely, DM-VAE regularizes the 64-channel space to achieve 0.12 rFID alongside 10.40/8.59 FID10K_{10K} on DiT-XL, and 5.44/4.26 with IGTS_S. Figure 5 confirms sharp textures and coherent structures at only 64 epochs without CFG. 2) Synergy with external foundation models: Orthogonally combining DINOv2 in DM-VAEDino_{Dino} (f16d32) yields 0.27 rFID and further accelerates convergence, reaching 6.99/5.97 FID10K_{10K} on DiT-XL and 4.86/4.33 with IGTS_S (6.72/6.51 on DiT-B).

Qualitative generation results on ImageNet using DiT-XL with our foundation-model-free DM-VAE (f16d32) and $IGT_S$

Figure 5. Qualitative generation results on ImageNet using DiT-XL with our foundation-model-free DM-VAE (f16d32) and IGTSIGT_S at only 64 training epochs (without CFG).

Comparison with State-of-the-Art Methods. In Table 3, we benchmark our full framework against leading vanilla LDMs and foundation-model-guided LDMs on ImageNet-1K (256 × 256). 1) Foundation-model-free regime: Compared with vanilla LDMs such as vanilla DiT [35], SiT [32], and MDTv2 [14] that require 1080 ∼ 1600 epochs to converge, DiT paired with our DM-VAE (f16d32) reaches an FID50K_{50K} of 2.62 (IGTS_S alone, w/o CFG) and 1.67 (with CFG) at only 64 epochs, and achieves 1.25 without CFG and 1.17 with CFG at 400 epochs. Notably, when scaling to a 64-channel latent space (f16d64, 0.12 rFID), our dual-manifold regularization and IGTS_S effectively prevent high-dimensional off-manifold degradation, achieving an FID50K_{50K} of 3.01/2.10 at 64 epochs, 1.71/1.43 at 160 epochs, and 1.28 (w/o CFG) / 1.20 (with CFG) at 400 epochs. Consequently, both our f16d32 and f16d64 variants not only outperform all vanilla LDMs, but also surpass 800-epoch foundation-model-aligned methods including REPA [50] (1.42), REPA-E [27] (1.26), VA-VAE [49] (1.35), and RAE [52] (1.28). 2) Foundation-model-assisted regime: Furthermore, our framework is orthogonal to external priors; when combined with DINOv2 (DM-VAEDino_{Dino}), it reaches an FID50K_{50K} of 2.26 (w/o CFG) within 60 epochs and further pushes the state-of-the-art FID50K_{50K} (400 epochs) to 1.29 without CFG and 1.11 with CFG, outperforming IG [54] (VA-VAE) (1.19 at 680 epochs).

MethodEpochs#ParamsGeneration@256 w/o CFG FID↓sFID↓IS↑Prec.↑Rec.↑Generation@256 w/ CFG or Autoguidance FID↓sFID↓IS↑Prec.↑Rec.↑
Latent Diffusion with Self-supervised Representation Models
REPA [50]80675M7.905.06122.60.700.65-----
8005.905.69157.80.700.691.424.70305.70.800.65
REPA-E [27]80675M3.464.17159.80.770.631.674.12266.30.800.63
8001.83-217.3--1.264.11314.90.790.66
REG [47]80677M3.40-184.1--1.864.49321.40.760.63
4802.204.67219.10.770.661.404.24296.90.770.66
VA-VAE [49]64675M5.144.22130.20.760.622.114.16252.30.810.58
8002.174.36205.60.770.651.354.15295.30.790.65
DDT [45]80675M6.62-135.20.690.671.52-263.70.780.63
2566.30-146.70.680.681.31-308.10.780.62
4006.27-154.70.680.691.26-310.60.790.65
IG [54] (VA-VAE)60675M2.423.81173.70.790.62-----
6801.343.94229.30.780.651.194.11269.00.790.66
RAE (DiTᴰᴴ) [52]800839M1.605.40242.70.790.651.284.72262.90.780.67
DiT-XL (DM-VAEᴰᵢₙₒ)60675M2.264.81200.30.790.601.364.12276.30.790.63
4001.294.28253.30.790.641.114.03297.40.790.65
6401.294.22260.40.790.651.154.22303.80.790.65
Vanilla Latent Diffusion (without foundation-model alignment)
DiT [35]1400675M9.626.85121.50.670.672.274.60278.20.830.57
MaskDiT [53]1600675M5.6910.34177.90.740.602.285.67276.60.800.61
SiT [32]1400675M8.616.32131.70.680.672.064.50270.30.820.59
TREAD [23]740675M-----1.694.73292.70.810.63
MDTv2 [14]1080675M-----1.584.52314.70.790.65
DiT-XL (DM-VAE, f16d32)64675M2.624.62190.10.780.601.674.12253.40.800.62
1601.614.23222.80.790.631.294.01264.30.800.64
4001.254.16250.80.790.651.174.02283.00.800.64
DiT-XL (DM-VAE, f16d64)64675M3.014.61178.40.790.582.104.24224.60.810.59
1601.714.32218.10.800.621.434.14252.10.810.62
4001.284.16245.70.790.641.204.10274.80.800.65

Table 3. Quantitative comparison of state-of-the-art generative models on ImageNet-1K (256 × 256). We benchmark our method against vanilla latent diffusion models and foundation-model-aligned LDMs across different training epochs and model capacities. Our method establishes new state-of-the-art results with faster convergence in both foundation-model-free and foundation-model-assisted regimes. Note that our results in the “w/o CFG” column are evaluated with single-pass IGTS_S.

Ablation Study

We conduct ablation studies on DiT-B at 80 training epochs (without CFG) in Table 4 to isolate the effectiveness of DM-VAE, topological representation alignment (TRA), and internal guidance with topological skeleton (IGTS_S).

VAEAlign TargetGuidanceFID50K_{50K}↓sFID↓IS↑Prec.↑Rec.↑
SD-VAE [38]NoneNone25.375.4752.220.570.64
DM-VAENoneNone19.035.4769.060.610.64
HB (zz)None19.525.2867.030.610.63
HB (zz)IG [54]10.984.0891.400.700.61
LB (zlowz_{\mathrm{low}})None18.595.3269.770.620.63
LB (zlowz_{\mathrm{low}})IGTS_S9.944.3397.550.710.60

Table 4. Ablation study on DM-VAE, representation alignment, and internal guidance strategy at 80 epochs. Results are obtained under the f16d32 setting using DiT-B [35] without CFG. HB/LB denote the high/low-dimensional branch; IG [54] requires full-channel (CC-dim) HB predictions, whereas LB alignment predicts only the ClowC_{\mathrm{low}}-dim skeleton and thus pairs with IGTS_S.

DM-VAE and Low-Dimensional Alignment in TRA. Without any alignment or guidance, replacing SD-VAE with DM-VAE slashes FID50K_{50K} from 25.37 to 19.03 and boosts IS from 52.22 to 69.06, confirming that our dual-manifold regularization yields a much more generation-friendly latent space. On top of DM-VAE, aligning intermediate features hth_t with HB (zz) degrades FID50K_{50K} to 19.52, since regressing high-frequency details disrupts global structural learning. In contrast, supervising hth_t with the skeleton LB (zlowz_{\mathrm{low}}) further reduces FID50K_{50K} to 18.59, providing a clean structural prior for velocity learning.

IGTS_S and Standard IG. We further compare standard internal guidance (IG) [54] against our IGTS_S. While standard IG on HB (zz) lowers FID50K_{50K} to 10.98, extrapolating unaligned residuals across all channels amplifies velocity errors and limits semantic quality (91.40 IS). Selectively guiding only the smooth skeleton LB (zlowz_{\mathrm{low}}) while leaving residual details unperturbed achieves the best FID50K_{50K} of 9.94, IS of 97.55, and Precision of 0.71.

Conclusion

In this paper, we identify that the off-manifold problem in high-dimensional VAEs stems from unconstrained channel expansion, whereas relying on external foundation models to regularize the manifold compromises out-of-distribution generalization. To resolve this dilemma internally, we propose the DM-VAE framework, which regularizes the high-dimensional latent space by anchoring it to a smooth low-dimensional topological skeleton. Building upon this, we introduce topological representation alignment (TRA) to provide explicit structural supervision for intermediate features during generative training, and internal guidance with topological skeleton (IGTS_S) to guide structural components while keeping high-frequency details intact within a single forward pass at inference. Extensive experiments show that our framework reconciles in-domain and OOD reconstruction with fast generative convergence, achieving a 0.12 rFID and 1.28/1.20 FID50K_{50K} (w/o and w/ CFG) at f16d64, 1.25/1.17 at f16d32 without external priors, and 1.11 when orthogonally combined with DINOv2.

LLM Usage Statement

LLMs (Gemini) were used solely at the Assist level to polish author-written text in the Abstract and Introduction. No LLMs or AI tools were used to generate research ideas or conclusions, compile related work or citations, create figures, or produce data, nor was the manuscript submitted to any LLM for reviewer-style feedback. The authors have verified all content and take full responsibility for the entire submission.

Detailed Off-Manifold Analysis

We derive the exponential reconstruction distortion bound in § 3.2. Under the manifold assumption [6, 29], natural images x∈Xx \in\mathcal{X} lie on an intrinsically dd-dimensional manifold MX\mathcal{M}_X. Let u∈Rd\mathbf{u} \in\mathbb{R}^{d} denote the local intrinsic coordinates on MX\mathcal{M}_X with finite differential entropy h(u)h(\mathbf{u}), and let u^∈Rd\hat{\mathbf{u}} \in\mathbb{R}^{d} denote the intrinsic coordinates of the nearest-point projection of x^\hat{x} onto MX\mathcal{M}_X. Under a locally bi-Lipschitz embedding, the pixel distortion ϵrecon=E[∥x−x^∥22]\epsilon_{\mathrm{recon}} = \mathbb{E}[\lVert x-\hat{x}\rVert_{2}^{2}] satisfies ϵrecon≥μE[∥u−u^∥22]\epsilon_{\mathrm{recon}} \ge\mu\mathbb{E}[\lVert\mathbf{u}-\hat{\mathbf{u}}\rVert_{2}^{2}] for a constant μ>0\mu> 0. In the VAE Markov chain u↔x→z→x^→u^\mathbf{u} \leftrightarrow x \to z \to\hat{x} \to\hat{\mathbf{u}}, the data processing inequality and the Shannon lower bound [9] yield:

I(x;z)≥I(u;u^)=h(u)−h(u−u^∣u^)≥h(u)−h(u−u^)≥h(u)−d2log⁡(2πeϵrecondμ)=d2log⁡(αϵrecon),(7)\begin{aligned} I(x;z) \ge I(\mathbf{u};\hat{\mathbf{u}}) = h(\mathbf{u}) - h(\mathbf{u}-\hat{\mathbf{u}}\mid\hat{\mathbf{u}}) \\ &\ge h(\mathbf{u}) - h(\mathbf{u}-\hat{\mathbf{u}}) \ge h(\mathbf{u})-\frac{d}{2}\log\left(\frac{2\pi e\epsilon_{\mathrm{recon}}}{d\mu}\right) \\ &= \frac{d}{2}\log\left(\frac{\alpha}{\epsilon_{\mathrm{recon}}}\right), \tag*{(7)} \end{aligned}

where the third inequality follows from the maximum entropy of a dd-dimensional Gaussian with average coordinate variance ϵrecon/(dμ)\epsilon_{\mathrm{recon}}/(d\mu), and α=dμ2πeexp⁡(2dh(u))>0\alpha= \frac{d\mu}{2\pi e}\exp\left(\frac{2}{d}h(\mathbf{u})\right) > 0 is a constant determined by the intrinsic dimension dd and intrinsic entropy power of MX\mathcal{M}_X. Conversely, in a continuous VAE with encoder posterior qE(z∣x)q_E(z\mid x) and Gaussian prior p(z)=N(0,I)p(z)=\mathcal{N}(0,I) on RC×H×W\mathbb{R}^{C\times H\times W}, the variational information bottleneck bound [1, 2] upper-bounds the mutual information by the KL divergence LKL\mathcal{L}_{\mathrm{KL}}:

I(x;z)≤Ep(x)[DKL(qE(z∣x)∥p(z))]≤κ⋅CHW,(8)I(x;z) \le\mathbb{E}_{p(x)}\left[D_{\mathrm{KL}}\left(q_E(z\mid x)\mathrel{\|}p(z)\right)\right] \le\kappa\cdot CHW, \tag*{(8)}

where κ>0\kappa> 0 denotes the average KL channel capacity per scalar latent entry. Combining Equations (7) and (8) gives d2log⁡(α/ϵrecon)≤κCHW\frac{d}{2}\log(\alpha/\epsilon_{\mathrm{recon}}) \le\kappa CHW, which establishes the exponential bound in § 3.2:

ϵrecon≥αexp⁡(−γ⋅C),where γ=2κHWd>0.(9)\epsilon_{\mathrm{recon}} \ge\alpha\exp(-\gamma\cdot C), \qquad\text{where }\gamma=\frac{2\kappa HW}{d}>0. \tag*{(9)}

While increasing CC exponentially lowers this distortion floor, embedding the dd-dimensional MX\mathcal{M}_X into RC×H×W\mathbb{R}^{C\times H\times W} introduces a codimension of c=CHW−dc=CHW-d normal directions. Because LVAE\mathcal{L}_{\mathrm{VAE}} (Eq. (1)) optimizes pixel and adversarial fidelity without constraining this cc-dimensional normal space, the encoder scatters substantial high-frequency energy along the normal directions, whose large variance disrupts low-frequency structural velocity learning in flow matching (cf., Appendix § 9).

Latent Manifold Geometry Analysis

In this section, we provide a geometric and information-theoretic analysis of how dual-manifold alignment (LDM\mathcal{L}_{\mathrm{DM}}) shapes the high-dimensional latent manifold MZ\mathcal{M}_Z into a generation-friendly tubular structure and decouples multi-frequency velocity learning in flow matching (FM).

Tubular Geometry of the DM-VAE Latent Space. Recall from § 3.2 that for any encoded latent z=E(x)∈RC×H×Wz=\mathcal{E}(x)\in\mathbb{R}^{C\times H\times W}, DM-VAE extracts zlow=Pdown(z)∈RClow×H×Wz_{\mathrm{low}}=\mathcal{P}_{\mathrm{down}}(z)\in\mathbb{R}^{C_{\mathrm{low}}\times H\times W} (Clow<CC_{\mathrm{low}}<C) and lifts it via Pup\mathcal{P}_{\mathrm{up}} to obtain the topological skeleton z^≜P^up(zlow)∈RC×H×W\hat{z}\triangleq\hat{\mathcal{P}}_{\mathrm{up}}(z_{\mathrm{low}})\in\mathbb{R}^{C\times H\times W}. Consistent with § 3.3, any latent z∈MZz\in\mathcal{M}_Z can be decomposed into:

z=zbase⏟low-dimensional skeleton+zdetail⏟high-frequency residual (details),(10)z=\underbrace{z^{\mathrm{base}}}_{\text{low-dimensional skeleton}}+\underbrace{z^{\mathrm{detail}}}_{\text{high-frequency residual (details)}}, \qquad \tag*{(10)}

where zbasez^{\mathrm{base}} lies on the smooth base manifold Mbase⊂Im⁡(Pup)\mathcal{M}_{\mathrm{base}}\subset\operatorname{Im}(\mathcal{P}_{\mathrm{up}}) (of subspace dimension Dbase=ClowHWD_{\mathrm{base}}=C_{\mathrm{low}}HW) supervised solely by Lrec(x,x^low)\mathcal{L}_{\mathrm{rec}}(x,\hat{x}_{\mathrm{low}}), and zdetailz^{\mathrm{detail}} captures fine-grained high-frequency residuals. By the Eckart–Young–Mirsky theorem, minimizing LDM=Ex[∥z−PupPdownz∥22]\mathcal{L}_{\mathrm{DM}}=\mathbb{E}_{x}[\lVert z-\mathcal{P}_{\mathrm{up}}\mathcal{P}_{\mathrm{down}}z\rVert_{2}^{2}] over (Pup,Pdown)(\mathcal{P}_{\mathrm{up}},\mathcal{P}_{\mathrm{down}}) naturally favors the rank-ClowC_{\mathrm{low}} orthogonal projector P=PT=P2\mathcal{P}=\mathcal{P}^{T}=\mathcal{P}^{2} onto the principal subspace Im⁡(Pup)\operatorname{Im}(\mathcal{P}_{\mathrm{up}}). Since Pdown\mathcal{P}_{\mathrm{down}} is simultaneously supervised by Lrec(x,x^low)\mathcal{L}_{\mathrm{rec}}(x,\hat{x}_{\mathrm{low}}), in practice we impose a lightweight orthogonality penalty on the 1×11\times1 projection weights (PdownPup≈I\mathcal{P}_{\mathrm{down}}\mathcal{P}_{\mathrm{up}}\approx I, PupPdown≈PdownPup\mathcal{P}_{\mathrm{up}}\mathcal{P}_{\mathrm{down}}\approx\mathcal{P}_{\mathrm{down}}\mathcal{P}_{\mathrm{up}}) to preserve Euclidean orthogonality between the two subspaces:

⟨zbase,zdetail⟩≈0,∥z∥22≈∥zbase∥22+∥zdetail∥22.(11)\langle z^{\mathrm{base}},z^{\mathrm{detail}}\rangle\approx0,\qquad\lVert z\rVert_{2}^{2}\approx\lVert z^{\mathrm{base}}\rVert_{2}^{2}+\lVert z^{\mathrm{detail}}\rVert_{2}^{2}. \tag*{(11)}

Furthermore, LDM\mathcal{L}_{\mathrm{DM}} directly bounds the expected energy of the transverse residual:

LDM=Ex[∥z−z^∥22]=Ex[∥zdetail∥22]≤ϵ2≪Ex[∥zbase∥22].(12)\mathcal{L}_{\mathrm{DM}}=\mathbb{E}_{x}\left[\lVert z-\hat{z}\rVert_{2}^{2}\right]=\mathbb{E}_{x}\left[\lVert z^{\mathrm{detail}}\rVert_{2}^{2}\right]\le\epsilon^{2}\ll\mathbb{E}_{x}\left[\lVert z^{\mathrm{base}}\rVert_{2}^{2}\right]. \tag*{(12)}

By Markov’s inequality, P(∥zdetail∥2>δ)≤ϵ2/δ2\mathbb{P}(\lVert z^{\mathrm{detail}}\rVert_{2}>\delta)\le\epsilon^{2}/\delta^{2} for any radius δ>ϵ\delta>\epsilon. Consequently, up to a negligible tail probability η=ϵ2/δ2\eta=\epsilon^{2}/\delta^{2}, the probability mass of MZ\mathcal{M}_Z is concentrated within a tubular neighborhood of radius δ\delta around Mbase\mathcal{M}_{\mathrm{base}}:

Tδ(Mbase)≜{zbase+zdetail∣zbase∈Mbase, ∥zdetail∥2≤δ}.(13)\mathcal{T}_{\delta}(\mathcal{M}_{\mathrm{base}})\triangleq\left\{z^{\mathrm{base}}+z^{\mathrm{detail}}\mid z^{\mathrm{base}}\in\mathcal{M}_{\mathrm{base}},\ \lVert z^{\mathrm{detail}}\rVert_{2}\le\delta\right\}. \tag*{(13)}

This tubular concentration provides two key benefits for generation: 1) Primary-to-residual velocity learning: Since E[∥z1base∥22]≫E[∥z1detail∥22]\mathbb{E}[\lVert z_{1}^{\mathrm{base}}\rVert_{2}^{2}]\gg\mathbb{E}[\lVert z_{1}^{\mathrm{detail}}\rVert_{2}^{2}], LFM\mathcal{L}_{\mathrm{FM}} naturally prioritizes transporting noise toward the smooth skeleton Mbase\mathcal{M}_{\mathrm{base}} before refining local residuals. **2) Bounded off-manifold drift: ** Even if the learned velocity vθ(zt,t)v_{\theta}(z_t,t) incurs errors along the high-frequency residual directions, the small tubular radius ensures that generated latents remain tightly anchored around the valid skeleton Mbase\mathcal{M}_{\mathrm{base}}.

Temporal SNR Decoupling in FM Interpolation. Consider the interpolation zt=tz1+(1−t)z0z_t = tz_1 + (1-t)z_0 (t∈[0,1]t \in[0,1], z0∼N(0,I)z_0 \sim\mathcal{N}(0,I)). Projecting ztz_t via the orthogonal projectors P\mathcal{P} and I−P\mathbf{I} - \mathcal{P} onto the skeleton subspace (of dimension Dbase=ClowHWD_{\mathrm{base}} = C_{\mathrm{low}}HW) and residual subspace (of dimension Ddetail=(C−Clow)HWD_{\mathrm{detail}} = (C - C_{\mathrm{low}})HW) yields two decoupled processes:

ztbase=tz1base+(1−t)z0base,ztdetail=tz1detail+(1−t)z0detail.(14)z_t^{\mathrm{base}} = tz_1^{\mathrm{base}}+(1-t)z_0^{\mathrm{base}}, \qquad z_t^{\mathrm{detail}} = tz_1^{\mathrm{detail}}+(1-t)z_0^{\mathrm{detail}} . \tag*{(14)}

Within each subspace’s orthonormal basis, the projected standard Gaussian noises z0basez_0^{\mathrm{base}} and z0detailz_0^{\mathrm{detail}} are independent and retain unit variance per active subspace dimension (1DkE[∥z0k∥22]=1(\frac{1}{D_k}\mathbb{E}[\lVert z_0^k\rVert_2^2] = 1 for k∈{base,detail}k \in\{\mathrm{base},\mathrm{detail}\}). Under a zero-mean isotropic Gaussian approximation z1k∼N(0,σk2IDk)z_1^k \sim\mathcal{N}(0,\sigma_k^2\mathbf{I}_{D_k}) (encouraged by the standard normal KL prior) with per-subspace-dimension signal variance σk2=1DkE[∥z1k∥22]\sigma_k^2 = \frac{1}{D_k}\mathbb{E}[\lVert z_1^k\rVert_2^2], the instantaneous subspace signal-to-noise ratio (SNR) at timestep tt is:

SNRk(t)=t2σk2(1−t)2,k∈{base,detail}.(15)\mathrm{SNR}_k(t) = \frac{t^2\sigma_k^2}{(1-t)^2}, \qquad k \in\{\mathrm{base},\mathrm{detail}\}. \tag*{(15)}

Minimizing the L2L_2 loss LFM\mathcal{L}_{\mathrm{FM}} yields the optimal velocity v∗(zt,t)=E[z1−z0∣zt]=E[z1∣zt]1−t−zt1−tv^*(z_t,t) = \mathbb{E}[z_1-z_0 \mid z_t] = \frac{\mathbb{E}[z_1 \mid z_t]}{1-t} - \frac{z_t}{1-t}, where the minimum mean-squared error (MMSE) denoiser E[z1k∣ztk]\mathbb{E}[z_1^k \mid z_t^k] in each subspace takes the closed-form Wiener filter expression:

E[z1k∣ztk]=1tSNRk(t)1+SNRk(t)ztk,k∈{base,detail}.(16)\mathbb{E}[z_1^k \mid z_t^k] = \frac{1}{t}\frac{\mathrm{SNR}_k(t)}{1+\mathrm{SNR}_k(t)}z_t^k, \qquad k \in\{\mathrm{base},\mathrm{detail}\}. \tag*{(16)}

When SNRk(t)≪1\mathrm{SNR}_k(t) \ll1, the Wiener gain Wk(t)≈0W_k(t) \approx0 (the signal z1kz_1^k is submerged in noise and unrecoverable); when SNRk(t)≫1\mathrm{SNR}_k(t) \gg1, Wk(t)≈1W_k(t) \approx1 (trivial linear recovery). The rate of information transition with respect to log-SNR, dWkdlog⁡SNRk=SNRk(1+SNRk)2\frac{\mathrm{d}W_k}{\mathrm{d}\log\mathrm{SNR}_k} = \frac{\mathrm{SNR}_k}{(1+\mathrm{SNR}_k)^2}, attains its unique global maximum at SNRk(tk∗)=1\mathrm{SNR}_k(t_k^*) = 1 [13], yielding the critical transition timestep:

tk∗1−tk∗=1σk⟹tk∗=11+σk.(17)\frac{t_k^*}{1-t_k^*} = \frac{1}{\sigma_k} \quad\Longrightarrow\quad t_k^* = \frac{1}{1+\sigma_k}. \tag*{(17)}

Equation (17) reveals a key distinction between unconstrained VAEs and DM-VAE: 1) Spectral interference in unconstrained VAEs: In standard high-reconstruction VAEs (e.g., SD-VAE [38]), unregularized high-frequency details carry per-mode variance comparable to low-frequency structures (σdetail≈σbase\sigma_{\mathrm{detail}} \approx\sigma_{\mathrm{base}}), causing their transition windows to overlap (tdetail∗≈tbase∗t_{\mathrm{detail}}^* \approx t_{\mathrm{base}}^*). Because high-frequency targets exhibit large sample-to-sample variance, regressing them simultaneously at intermediate timesteps injects severe gradient variance that disrupts global structural learning. 2) Natural temporal decoupling in DM-VAE: By suppressing transverse residual energy via LDM\mathcal{L}_{\mathrm{DM}} (σdetail≪σbase\sigma_{\mathrm{detail}} \ll\sigma_{\mathrm{base}}), Equation (17) strictly separates the two transition windows:

tbase∗=11+σbase<tdetail∗=11+σdetail⟶1.(18)t_{\mathrm{base}}^* = \frac{1}{1+\sigma_{\mathrm{base}}} < t_{\mathrm{detail}}^* = \frac{1}{1+\sigma_{\mathrm{detail}}} \longrightarrow1. \tag*{(18)}

At intermediate timesteps t≈tbase∗t \approx t_{\mathrm{base}}^*, SNRbase(t)≈1\mathrm{SNR}_{\mathrm{base}}(t) \approx1 while SNRdetail(t)≪1\mathrm{SNR}_{\mathrm{detail}}(t) \ll1 (Wdetail(t)≈0W_{\mathrm{detail}}(t) \approx0), allowing the network to learn the low-variance velocity toward the global skeleton z1basez_1^{\mathrm{base}} without high-frequency interference, before resolving fine textures near t→1t \to1.

Empirical Verification. Figure 6 and Figure 7 empirically validate our geometric and SNR analysis on 5K ImageNet validation images under f16d64 (Clow=32C_{\mathrm{low}} = 32). First, encoded latents exhibit near-exact subspace orthogonality with ∣cos⁡(zbase,zdetail)∣=0.0047|\cos(z^{\mathrm{base}}, z^{\mathrm{detail}})| = \mathbf{0.0047}, corroborating Eq. (11). In pixel space (Figure 6(a)), D(z^)\mathcal{D}(\hat{z}) closely matches D(z)\mathcal{D}(z) across global low frequencies (k≤8k \le8, with PSDres/PSDfull<−27\mathrm{PSD}_{\mathrm{res}}/\mathrm{PSD}_{\mathrm{full}} < -27 dB), while the detail increment D(z)−D(z^)\mathcal{D}(z) - \mathcal{D}(\hat{z}) concentrates on intra-patch high-frequency textures (k≥16k \ge16). Second, in latent space (cf., Figure 6(b)), even when granting SD-VAE its optimal post-hoc rank-32 PCA projector (the theoretical lower bound on σdetail\sigma_{\mathrm{detail}}), its detail transition window (tdetail∗=0.68t^{*}_{\mathrm{detail}} = 0.68) heavily overlaps with the base window (tbase∗=0.43t^{*}_{\mathrm{base}} = 0.43) around intermediate timesteps. Conversely, DM-VAE suppresses σdetail\sigma_{\mathrm{detail}} to push tdetail∗t^{*}_{\mathrm{detail}} to 0.91 (Δt∗=0.48\Delta t^{*} = 0.48 vs. 0.25), keeping detail transition rates near zero throughout [0,0.6][0, 0.6].

Figure 6. Empirical verification of spectral decomposition and temporal SNR decoupling on 5K ImageNet validation images (f16d64f16d64, Clow=32C_{\mathrm{low}} = 32). (a) Radially averaged 2D Fourier PSD in pixel space shows that the skeleton reconstruction D(z^)D(\hat{z}) preserves global low-frequency structures, whereas the detail increment D(z)−D(z^)D(z) - D(\hat{z}) rises by over 44 dB in relative power ratio (PSDres/PSDfull\mathrm{PSD}_{\mathrm{res}}/\mathrm{PSD}_{\mathrm{full}}) from k=1k = 1 to k=128k = 128. (b) Transition rate curves dWk/dlog⁡SNRk\mathrm{d}W_k/\mathrm{d}\log\mathrm{SNR}_k (Eqs. (16)–(17)) parameterized by empirical σbase\sigma_{\mathrm{base}}, σdetail\sigma_{\mathrm{detail}} on globally unit-variance-normalized latents confirm that while SD-VAE exhibits substantial base–detail overlap (tdetail∗=0.68t^{*}_{\mathrm{detail}} = 0.68), DM-VAE pushes the detail transition to tdetail∗=0.91t^{*}_{\mathrm{detail}} = 0.91, isolating structural velocity learning within [0,0.6][0, 0.6].

Qualitative visualization of latent subspace decomposition on ImageNet validation samples using DM-VAE

Figure 7. Qualitative visualization of latent subspace decomposition on ImageNet validation samples using DM-VAE (f16d64, Clow=32C_{\mathrm{low}} = 32). From top to bottom: input image xx, full reconstruction D(z)\mathcal{D}(z), decoded skeleton D(z^)\mathcal{D}(\hat{z}), and residual ∣D(z)−D(z^)∣|\mathcal{D}(z)-\mathcal{D}(\hat{z})|. While D(z^)\mathcal{D}(\hat{z}) preserves global structure and color layout, the residual isolates fine-grained edges and high-frequency textures.

Detailed Training Settings

DM-VAE Training. We train DM-VAE on the ImageNet-1K [10] training set at 256×256256 \times256 resolution (with random cropping and horizontal flipping), following the architectural and optimization conventions of SD-VAE [38] and VA-VAE [49]. 1) Architecture: The shared encoder E\mathcal{E} and main decoder D\mathcal{D} adopt a 5-stage convolutional backbone with channel multipliers (1,1,2,2,4)(1, 1, 2, 2, 4) (spatial downsampling factor f=16f = 16), base channel width 128, 2 residual blocks per stage, and self-attention at the 16×1616 \times16 latent resolution. To ensure efficiency and structural inductive bias, the auxiliary lightweight decoder Dlow\mathcal{D}_{\mathrm{low}} halves the base channel width to 64, uses 1 residual block per stage, and removes all self-attention layers; Dlow\mathcal{D}_{\mathrm{low}} is used only during training and discarded at inference. The linear projections Pdown\mathcal{P}_{\mathrm{down}} and Pup\mathcal{P}_{\mathrm{up}} are implemented as bias-free 1×11 \times1 convolutions with orthogonal initialization (Wdown=WupT\mathbf{W}_{\mathrm{down}} = \mathbf{W}_{\mathrm{up}}^{\mathsf{T}}). 2) Objectives and adaptive weighting: Both Lrec(x,x^)\mathcal{L}_{\mathrm{rec}}(x, \hat{x}) and Lrec(x,x^low)\mathcal{L}_{\mathrm{rec}}(x, \hat{x}_{\mathrm{low}}) sum pixel-level L1L_1 and perceptual LPIPS [51] losses with unit weights. We set the KL penalty weight β=10−6\beta= 10^{-6} and the low-dimensional branch weight λlow=0.2\lambda_{\mathrm{low}} = 0.2. Adversarial training (Ladv\mathcal{L}_{\mathrm{adv}}) is applied exclusively to the main reconstruction x^\hat{x} via a 3-layer PatchGAN discriminator [12] with hinge loss after 1,000 warmup steps, balanced adaptively by the gradient-norm ratio at the last convolutional layer of D\mathcal{D} (with base weight 0.5). For dual-manifold alignment, LDM\mathcal{L}_{\mathrm{DM}} is computed on the posterior means μ\mu and μ^=Pup(Pdown(μ))\hat{\mu} = \mathcal{P}_{\mathrm{up}}(\mathcal{P}_{\mathrm{down}}(\mu)) using the symmetric stop-gradient (sg[·]) formulation in § 3.2. Its effective coefficient λDM=0.1 wDM\lambda_{\mathrm{DM}} = 0.1\,w_{\mathrm{DM}} is dynamically balanced every 20 steps via an exponential moving average (EMA, decay 0.8) of the gradient-norm ratio at the shared pre-latent layer θElast\theta_{\mathcal{E}}^{\mathrm{last}}:

wDM←0.8wDM+0.2∥∇θElastLrec(x,x^)∥2∥∇θElast∥μ−sg⁡[μ^]∥22∥2+10−2.(19)w_{\mathrm{DM}} \leftarrow0.8w_{\mathrm{DM}} + 0.2 \frac{\left\lVert\nabla_{\theta_{\mathcal{E}}^{\mathrm{last}}}\mathcal{L}_{\mathrm{rec}}(x,\hat{x}) \right\rVert_2}{\left\lVert\nabla_{\theta_{\mathcal{E}}^{\mathrm{last}}}\left\lVert\mu-\operatorname{sg}[\hat{\mu}]\right\rVert_2^2 \right\rVert_2 + 10^{-2}}. \tag*{(19)}

In addition, let Wdown∈RClow×C\mathbf{W}_{\mathrm{down}} \in\mathbb{R}^{C_{\mathrm{low}}\times C} and Wup∈RC×Clow\mathbf{W}_{\mathrm{up}} \in\mathbb{R}^{C\times C_{\mathrm{low}}} denote the reshaped 1×11 \times1 kernel matrices of Pdown\mathcal{P}_{\mathrm{down}} and Pup\mathcal{P}_{\mathrm{up}}. Consistent with Appendix § 9, we include an orthogonality regularizer λorthoLortho\lambda_{\mathrm{ortho}}\mathcal{L}_{\mathrm{ortho}} (λortho=10.0\lambda_{\mathrm{ortho}} = 10.0) purely as a safeguard against degenerate rescaling or basis drift; given our orthogonal initialization, Lortho\mathcal{L}_{\mathrm{ortho}} remains near zero throughout training and exerts negligible influence on optimization:

Lortho=MSE⁡(WdownWdownT,I)+MSE⁡(WupTWup,I)+0.5MSE⁡(Wdown,WupT).(20)\mathcal{L}_{\mathrm{ortho}} = \operatorname{MSE}(\mathbf{W}_{\mathrm{down}}\mathbf{W}_{\mathrm{down}}^{\mathsf{T}}, \mathbf{I}) + \operatorname{MSE}(\mathbf{W}_{\mathrm{up}}^{\mathsf{T}}\mathbf{W}_{\mathrm{up}}, \mathbf{I}) + 0.5\operatorname{MSE}(\mathbf{W}_{\mathrm{down}}, \mathbf{W}_{\mathrm{up}}^{\mathsf{T}}). \tag*{(20)}

3) Optimization: Both DM-VAE and the discriminator are trained within 50 epochs using Adam (lr=1×10−4\mathrm{lr} = 1 \times10^{-4}, β1=0.5\beta_1 = 0.5, β2=0.9\beta_2 = 0.9) in bf16 on a 16 machine setup with a global batch size of 256, gradient norm clipping at 1.0, and an EMA decay of 0.9999.

We omit adversarial loss on the low-dimensional branch (x^low\hat{x}_{\mathrm{low}}) for both training efficiency and optimization stability. First, incorporating an auxiliary adversarial objective noticeably slows down DM-VAE training due to additional discriminator passes and adaptive gradient-norm evaluations. Second, adversarial training is inherently unstable under a strict capacity bottleneck: in our preliminary experiments, applying a GAN loss to x^low\hat{x}_{\mathrm{low}} consistently triggered late-stage adversarial collapse, as the discriminator easily overpowered the capacity-limited lightweight decoder Dlow\mathcal{D}_{\mathrm{low}} and broke the minimax equilibrium.

DiT Training. We conduct class-conditional generative training on ImageNet-1K (256×256256 \times256) [10] strictly adhering to the official settings of LightningDiT [49] and IG [54]. For topological representation alignment (LTRA), the intermediate features hth_t are extracted from the same default transformer layer as in IG [54] and mapped to RClow×H×W\mathbb{R}^{C_{\mathrm{low}}\times H\times W} via a lightweight MLP projection head Palign\mathcal{P}_{\mathrm{align}}. For fair evaluation at inference, both our methods and all reproduced baselines adopt an identical 250-step Euler solver with uniform class-balanced sampling across all classes.

More Results

Quantitative Results on DDT. To verify the generality of our framework beyond standard DiT, we further integrate DM-VAE (f16d32) and IGTS_S into the Decoupled Diffusion Transformer (DDT) [45] across Base (B) and Extra-Large (XL) scales at 80 training epochs without CFG (cf., Table 5). Using our foundation-model-free DM-VAE without guidance, DDT (DM-VAE) substantially outperforms both Improved-SiT [32] (25.1 →\rightarrow 15.2 FID50K_{50K} on B; 10.9 →\rightarrow 5.9 on XL) and Improved-REPA [50] (19.1 →\rightarrow 15.2 on B; 8.1 →\rightarrow 5.9 on XL), while simultaneously improving spatial fidelity (6.08/4.75 sFID) and diversity (0.66/0.68 Recall). Incorporating DINOv2 alignment in DM-VAEDino further reduces FID50K_{50K} to 11.0 (B) and 4.1 (XL). When combined with single-pass IGTS_S, generation quality improves, reaching 5.1 FID50K_{50K} (136.9 IS) on DDT-B and 2.0 FID50K_{50K} (4.35 sFID, 213.3 IS) on DDT-XL without CFG, confirming that our dual-manifold alignment and skeleton guidance generalize across distinct architectures.

MethodsFID50K↓_{50K}\downarrowsFID↓\downarrowIS↑\uparrowPrec.↑\uparrowRec.↑\uparrow
Improved-SiT-B†^\dagger [32]25.16.5458.80.570.64
Improved-REPA-B†^\dagger [50]19.16.8876.50.600.66
DDT-B (DM-VAE)15.26.0884.00.630.66
DDT-B (DM-VAEDino)11.05.4794.80.700.62
+ IGTS_S5.15.23136.90.770.57
Improved-SiT-XL†^\dagger [32]10.95.30103.40.660.65
Improved-REPA-XL†^\dagger [50]8.15.34124.90.680.67
DDT-XL (DM-VAE)5.94.75139.80.700.68
DDT-XL (DM-VAEDino)4.14.43157.00.750.64
+ IGTS_S2.04.35213.30.800.61

Table 5. Generation performance comparison on ImageNet at 80 generative training epochs using DDT [45] without CFG. †^\dagger denotes baselines improved by SwiGLU [43], RoPE [41], and RMSNorm [43]. Our DM-VAEs adopt the f16d32 configuration. For IGTS_S, we consistently apply a guidance scale of 1.7 across the timestep interval of [0,0.6][0, 0.6].

Qualitative Generation Results. We provide more visual samples in Figure 8 and Figure 9 without using CFG.

Qualitative generation results using DiT-XL with our foundation-model-free DM-VAE (f16d64) at 400 epochs

Figure 8. Qualitative generation results using DiT-XL with our foundation-model-free DM-VAE (f16d64) at 400 epochs.

Qualitative generation results using DiT-XL with our foundation-model-free DM-VAE (f16d64) at 400 epochs.

Figure 9. Qualitative generation results using DiT-XL with our foundation-model-free DM-VAE (f16d64) at 400 epochs.

Impact of Low Dimension ClowC_{\mathrm{low}}. We further investigate the choice of the low-dimensional bottleneck ClowC_{\mathrm{low}} under the f16d64 (C=64C = 64) configuration. On DiT-XL at 80 training epochs (without CFG or IGTS_S), setting Clow=32C_{\mathrm{low}} = 32 achieves an FID10K_{10K} of 10.40, whereas further compressing it to Clow=16C_{\mathrm{low}} = 16 degrades FID10K_{10K} to 12.00. This reflects an inherent trade-off in selecting ClowC_{\mathrm{low}}, which should be neither overly large nor overly small. An excessively large ClowC_{\mathrm{low}} weakens the information bottleneck, leaving zlowz_{\mathrm{low}} itself unconstrained with excessive degrees of freedom and prone to manifold degradation, which in turn compromises its topological regularization on the high-dimensional latent zz. Conversely, an overly small ClowC_{\mathrm{low}} severely restricts skeleton capacity and fails to adequately suppress transverse high-frequency residual energy, widening the tubular neighborhood around lifted zlowz_{\mathrm{low}} and limiting its ability to mitigate the off-manifold problem. Balancing skeleton smoothness and residual energy suppression, we set Clow=C/2C_{\mathrm{low}} = C/2 by default across our main experiments.

References

  1. [1]Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rifat A Saurous, and Kevin Murphy. Fixing a broken elbo. In ICML, pages 159–168. PMLR, 2018.arxiv.org/abs/1711.00464
  2. [2]Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint, 2016.arxiv.org/abs/1612.00410
  3. [3]Black Forest Labs. Flux.2 latent-space representation comparison. https://bfl.ai/research/representation-comparison, 2025. Accessed: 2026.
  4. [4]Gwern Branwen, Anonymous, and The Danbooru Community. Danbooru2021: A large-scale crowdsourced and tagged anime illustration dataset. https://www.gwern.net/Danbooru2021, 2022.
  5. [5]Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint, 2015.arxiv.org/abs/1509.00519
  6. [6]Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542–542, 2009.
  7. [7]Hila Chefer, Patrick Esser, Dominik Lorenz, Dustin Podell, Vikash Raja, Vinh Tong, Antonio Torralba, and Robin Rombach. Self-supervised flow matching for scalable multi-modal synthesis. arXiv preprint, 2026.arxiv.org/abs/2603.06507
  8. [8]Hongxu Chen, Hongxiang Li, Zhen Wang, and Long Chen. Bi-anchor interpolation solver for accelerating generative modeling. arXiv preprint, 2026.arxiv.org/abs/2601.21542
  9. [9]Thomas M Cover, Joy A Thomas, and John Kieffer. Elements of information theory. wiley New York, 1991.
  10. [10]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. IEEE, 2009.
  11. [11]Carl Doersch. Tutorial on variational autoencoders. arXiv preprint, 2016.arxiv.org/abs/1606.05908
  12. [12]Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873–12883, 2021.arxiv.org/abs/2012.09841
  13. [13]Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.arxiv.org/abs/2403.03206
  14. [14]Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is a strong image synthesizer. arXiv preprint, 2023.DOI
  15. [15]Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. ACM, 63(11):139–144, 2020.
  16. [16]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 15979–15988. IEEE, 2022.arxiv.org/abs/2111.06377
  17. [17]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local Nash equilibrium. NeurIPS, 30, 2017.
  18. [18]Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
  19. [19]Matthew D Hoffman, Matthew J Johnson, et al. Elbo surgery: yet another way to carve up the variational evidence lower bound. In NeurIPS Workshop, 2016.
  20. [20]Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In ICIP, pages 2366–2369. IEEE, 2010.DOI
  21. [21]Dengyang Jiang, Mengmeng Wang, Liuzhuozheng Li, Lei Zhang, Haoyu Wang, Wei Wei, Guang Dai, Yanning Zhang, and Jingdong Wang. No other representation component is needed: Diffusion transformers can provide representation guidance by themselves. arXiv preprint, 2025.arxiv.org/abs/2505.02831
  22. [22]Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint, 2013.arxiv.org/abs/1312.6114
  23. [23]Felix Krause, Timy Phan, Ming Gui, Stefan Andreas Baumann, Vincent Tao Hu, and Björn Ommer. Tread: Token routing for efficient architecture-agnostic diffusion training. In ICCV, pages 15703–15713. IEEE, 2025.arxiv.org/abs/2501.04765
  24. [24]Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. NeurIPS, 32, 2019.
  25. [25]Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. NeurIPS, 37:122458–122483, 2024.
  26. [26]Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, et al. Flux. 1 kon-text: Flow matching for in-context image generation and editing in latent space. arXiv preprint, 2025.arxiv.org/abs/2506.15742
  27. [27]Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-1: Unlocking vae for end-to-end tuning with latent diffusion transformers. In ICCV, pages 18262–18272. IEEE, 2025.
  28. [28]Hongxiang Li, Hongxu Chen, Chenyang Zhu, Xiaoshuang Huang, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Long Chen. Spar: Semantic-pixel self-alignment and adaptive routing for unified multimodal models. In ECCV, pages 324–343. Springer, 2026.arxiv.org/abs/2606.23041
  29. [29]Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. In CVPR, pages 36115–36125, 2026.arxiv.org/abs/2511.13720
  30. [30]Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint, 2022.arxiv.org/abs/2210.02747
  31. [31]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint, 2022.arxiv.org/abs/2209.03003
  32. [32]Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In ECCV, pages 23–40. Springer, 2024.arxiv.org/abs/2401.08740
  33. [33]Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint, 2021.arxiv.org/abs/2103.03841
  34. [34]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint, 2023.arxiv.org/abs/2304.07193
  35. [35]William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4172–4182. IEEE, 2023.arxiv.org/abs/2212.09748
  36. [36]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
  37. [37]Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278–1286. PMLR, 2014.
  38. [38]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10674–10685. IEEE, 2022.arxiv.org/abs/2112.10752
  39. [39]Jaskirat Singh, Boyang Zheng, Zongze Wu, Richard Zhang, Eli Shechtman, and Saining Xie. Improved baselines with representation autoencoders. arXiv preprint, 2026.arxiv.org/abs/2605.18324
  40. [40]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS, 32, 2019.arxiv.org/abs/1907.05600
  41. [41]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
  42. [42]Jakub Tomczak and Max Welling. Vae with a vampprior. In AISTATS, pages 1214–1223. PMLR, 2018.arxiv.org/abs/1705.07120
  43. [43]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Ahmad Almahairi, Yasmin Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint, 2023.arxiv.org/abs/2307.09288
  44. [44]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint, 2025.arxiv.org/abs/2502.14786
  45. [45]Shuai Wang, Zhi Tian, Weilin Huang, and Limin Wang. Ddt: Decoupled diffusion transformer. In CVPR, pages 40633–40642, 2026.arxiv.org/abs/2504.05741
  46. [46]Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 13(4):600–612, 2004.pubmed.ncbi.nlm.nih.gov/15376593
  47. [47]Ge Wu, Shen Zhang, Ruijing Shi, Shanghua Gao, Zhenyuan Chen, Lei Wang, Zhaowei Chen, Hongcheng Gao, Yao Tang, Ming-Ming Cheng, et al. Representation entanglement for generation: Training diffusion transformers is much easier than you think. NeurIPS, 38:7714–7743, 2026.arxiv.org/abs/2507.01467
  48. [48]Tongda Xu, Mingwei He, Shady Abu-Hussein, Jose Miguel Hernandez-Lobato, Chunhang Zheng, Kai Zhao, Chao Zhou, Ya-Qin Zhang, and Yan Wang. Making reconstruction fid predictive of diffusion generation fid, 2026.arxiv.org/abs/2603.05630
  49. [49]Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. In CVPR, pages 15703–15712. IEEE, 2025.arxiv.org/abs/2501.01423
  50. [50]Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint, 2024.arxiv.org/abs/2410.06940
  51. [51]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586–595. IEEE, 2018.arxiv.org/abs/1801.03924
  52. [52]Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders. In ICLR, pages 35791–35820, 2026.arxiv.org/abs/2510.11690
  53. [53]Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers, 2024.arxiv.org/abs/2306.09305
  54. [54]Xingyu Zhou, Qifan Li, Xiaobin Hu, Hai Chen, and Shuhang Gu. Guiding a diffusion transformer with the internal dynamics of itself. In CVPR, pages 11536–11545, 2026.arxiv.org/abs/2512.24176

Paper details

vlnc:W_vebsz3u9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Gemini (multiple versions)

View statement
Looking for related papers…
Contents