WorksComputer Science
Dual-Manifold Variational Autoencoders
Abstract
In latent diffusion models, expanding the latent dimension of variational autoencoders (VAEs) is essential to push the upper bound of reconstruction quality. However, without explicit structural constraints, such unregularized expansion easily yields a rugged and ill-conditioned latent manifold, causing generated latents to drift off the manifold during generation (i.e., off-manifold problem). Recent works smooth this manifold by using external foundation models (e.g., DINOv2) as a topological regularizer. However, we argue that tying the latent space to frozen external priors inherently restricts generalization: bound by their pre-training distributions and semantic biases, they may fail to capture faithful visual details on out-of-distribution domains. Observing that low-dimensional latents naturally form a smoother, generation-friendly manifold, we propose dual-manifold VAE (DM-VAE) to leverage internal representations as a topological regularizer. Specifically, DM-VAE pairs a high-dimensional reconstructive branch with a low-dimensional branch that extracts a clean topological skeleton, anchoring the high-dimensional manifold to this smooth subspace. Building upon this framework, we further introduce topological representation alignment (TRA) to supervise intermediate generator features during training, and internal guidance with topological skeleton (IG_TS) to guide structural components while preserving high-frequency details at inference. On ImageNet-1K (256x256), our foundation-model-free DM-VAE achieves a state-of-the-art rFID of 0.12 and a generation FID50K of 1.28 (w/o CFG) and 1.20 (w/CFG), and further reaches 1.11 when orthogonally combined with DINOv2.
Introduction
Latent diffusion models (LDMs) [26, 32, 35, 38] employ a continuous-valued variational autoencoder (VAE) [18, 22] to project raw pixel images into a low-dimensional latent space. Operating within this compressed space, generative models (e.g., DiT [35]) learn to iteratively generate valid in-distribution latents from random noise, which are subsequently decoded back into the image domain by the VAE.
Specifically, the LDM generative modeling pipeline consists of two stages: 1) Reconstruction: In the reconstruction stage (cf., Figure 2(a)), the VAE first encodes images into latents (i.e., mapping image distribution to latent distribution) and then decodes these latents to reconstructed images. It usually jointly utilizes multiple training objectives to improve the visual quality of the reconstructed images, such as reconstruction [51], Kullback-Leibler (KL), and adversarial [12, 15] losses. 2) Generation: In the generation stage, generative models, e.g., flow-based models [30, 31, 32], learn the velocity field to transport the prior distribution (e.g., a Gaussian distribution) towards the latent distribution. The training objective typically minimizes the regression errors between the predicted velocities and the ground truth velocities. Crucially, since the generation stage operates within the latent space, the overall generative performance is upper-bounded by the reconstruction quality.
Figure 2. Comparison of reconstruction paradigms in LDMs. Unlike (a) vanilla VAEs [38] that suffer from unconstrained, rugged high-dimensional manifolds, or (b) foundation-model-aligned [49] and (c) foundation-model-based [52] VAEs that rely on external priors (e.g., Dino [34]) with limited generalization ability, (d) our foundation-model-free DM-VAE constructs an internal topological regularizer. Specifically, given an encoder mapping the input to a high-dimensional latent and a main decoder for reconstruction, we compress via into a low-dimensional branch (comprising and a lightweight decoder ). By enforcing the alignment between and the lifted , our method compels the high-dimensional manifold to inherit a smooth, generation-friendly geometry from . (e) Smooth vs. rugged manifolds, where “smooth” denotes a geometry dominated by low-frequency principal structures with bounded high-frequency oscillations along normal directions (cf., Appendix § 9).
Therefore, to push the upper bound of generative performance, enhancing reconstruction quality is essential. Due to the VAE’s inherent information bottleneck [2], a straightforward approach is expanding the dimension of the latent space (e.g., channels) to further minimize the reconstruction error. However, it inevitably introduces excessive degrees of freedom to the manifold corresponding to the latent distribution (i.e., latent manifold). Since there is no explicit structural constraint on the underlying latent manifold, this unregularized channel expansion often yields a rugged and ill-conditioned manifold [49]. Consequently, during generation, unconstrained high-frequency components disrupt velocity learning and easily cause generated latents to drift off the manifold, known as the off-manifold problem [40].
To address this off-manifold issue, recent approaches have increasingly relied on external foundation models (e.g., DINOv2 [34], MAE [16]). They either align latents with their features [49] (cf., Figure 2(b)) or adopt these features directly as latents [39, 52] (cf., Figure 2(c)). Specifically, their pre-trained features encode semantics that form a “smooth” reference manifold, acting as a topological regularizer. These regularizers can effectively smooth the unconstrained latent manifold, easing subsequent generative training. However, tying the latent space to external priors inherently restricts the VAE’s generalization capability [3, 7]. Because frozen foundation models are bound by their pre-training distributions and semantic biases, they fail to capture faithful visual details on out-of-distribution (OOD) domains, resulting in severe reconstruction degradation (cf., Figure 1).

Figure 1. Qualitative comparison of out-of-distribution image reconstruction. All methods are trained on ImageNet-1K. Without relying on external foundation models, our DM-VAE (1.66d64) exhibits superior generalization. Average PSNR [20] on above three samples: RAE [52] (16.32 dB), Ours (low branch) (27.19 dB), and Ours (high branch) (30.10 dB).
To this end, we propose a dual-manifold VAE (DM-VAE) framework to mitigate the off-manifold problem by leveraging internal representations as a topological regularizer.
Our design is motivated by a key observation: although low-dimensional latents sacrifice reconstruction fidelity, they consistently exhibit superior generative learnability [49] (e.g., SD-VAE [38] in Figure 3), reflecting an inherently smoother underlying manifold [48]. Building on this insight, DM-VAE branches a shared encoder into two coupled latent spaces (cf., Figure 2(d)): a high-dimensional space () decoded by the main decoder for high-fidelity reconstruction, and a low-dimensional space () (projected via and supervised by a lightweight decoder ) that captures a smooth topological “skeleton”. To prevent unconstrained optimization on the high-dimensional latent manifold, we lift back to the high-dimensional space via to obtain , and minimize its discrepancy with . This dual-manifold alignment compels the high-dimensional latent space to inherit the smooth geometry of its low-dimensional counterpart, smoothing the rugged manifold (cf., Figure 2(e)).

Figure 3. rFID vs. gFID comparison on DiT-XL (80 epochs). When scaling from d32 to d64, DM-VAE with IGT effectively overcomes the reconstruction–generation trade-off without external foundation models.
Under our DM-VAE framework, the extracted topological skeleton naturally provides a structural prior for the generation stage. To further facilitate generation within this regularized latent space, we introduce topological representation alignment (TRA) as an additional supervision signal. Specifically, we align the intermediate hidden features of the generative model with the low-dimensional latents during training. Given that the low-dimensional manifold captures the principal structural components of its high-dimensional counterpart, this alignment serves as an explicit structural prior, guiding the model to learn a more accurate velocity field within the high-dimensional space.
Beyond benefiting the training, these aligned intermediate features also enable an effective guidance strategy at the inference stage, internal guidance with topological skeleton (IGT). Although recent internal guidance (IG) [54] provides an effective paradigm by using intermediate features, its potential is bottlenecked when directly applied to the entire latent space, as it tends to damage high-frequency details and exacerbate the off-manifold problem. In contrast, since our intermediate features are explicitly aligned with the low-dimensional latents, they naturally capture the clean topological skeleton. Building on this, IGT separates the predicted high-dimensional latents into the low-dimensional topological skeleton and high-frequency details. We then use the aligned intermediate features to guide only the skeleton part, and add the unchanged high-frequency details back. In this way, IGT improves structural quality while keeping the generated latents on the manifold, all within a single forward pass.
To validate the effectiveness of our method, we have conducted extensive experiments on ImageNet-1K [10] at resolution using DiT [35], following the LightningDiT [49] training settings. As shown in Figure 3, our approach combined with IGT breaks the reconstruction–generation dilemma when scaling to higher-dimensional latent spaces (d32 to d64), consistently outperforming other baselines. Without relying on external foundation models, our DM-VAE (f16d64) achieves a reconstruction FID (rFID) of 0.12 within 50 reconstructive training epochs and a 400-epoch generation FID of 1.28 with IGT alone (1.20 under CFG). Our f16d32 variant reaches 1.67 under CFG at only 64 epochs and further advances to 1.25 with IGT alone (1.17 under CFG) at 400 epochs.
In summary, our main contributions are threefold: 1) We propose dual-manifold VAE (DM-VAE), a foundation-model-free VAE framework that regularizes rugged high-dimensional latent manifolds via a smooth low-dimensional topological skeleton. 2) We introduce topological representation alignment (TRA) for training and internal guidance with topological skeleton (IGT) for inference, leveraging the skeleton for structural supervision and single-pass guidance while keeping high-frequency details intact. 3) Extensive experiments demonstrate superior reconstruction and generation performance in our DM-VAE framework.
Related Work
Variational Autoencoder (VAE). Continuous-valued VAEs [5, 11, 22, 37] serve as the cornerstone of latent diffusion models (LDMs) [8, 12, 13, 26, 28, 32, 35, 38]. While vanilla VAEs were originally designed as end-to-end generative models that decode images directly from a latent prior distribution, they suffer from a structural mismatch between this predefined prior and the aggregated posterior distribution [19, 42]. Consequently, direct sampling from the vanilla VAE prior often yields blurry and low-fidelity images. To address this, LDMs repurpose the VAE as a reconstructive compressor, mapping high-dimensional pixel data into a compact latent space to facilitate subsequent generative training. Typically, training reconstructive VAE relies on reconstruction losses (e.g., and LPIPS [51]) and adversarial [12, 15] loss, paired with a Kullback-Leibler (KL) divergence penalty to regularize the latent space. To preserve information for high reconstruction fidelity, the KL weight is usually set to a small value (e.g., ), exerting minimal influence on the latent distribution. However, without further structural regularization beyond this weak KL constraint, the learned latent space is prone to off-manifold issues during generation.
Representation Alignment in LDMs. Incorporating external or internal representation alignment objectives in latent generative modeling has proven effective in accelerating the training process. External alignment approaches, such as VA-VAE [49], REPA [50], and REPA-E [27], leverage external foundation models (i.e., DINOv2 [34], MAE [16], CLIP [36], SigLIP [44]) to guide the alignment of either the latent space or the intermediate features of DiTs. However, relying on external foundation models can lead to generalization issues. To address this, internal alignment methods (e.g., SRA [21] and Self-Flow [7]) instead align representations derived from different views of the same training sample in generative training. Unlike these existing approaches, our proposed dual-manifold VAE can optimize the latent space itself through internal alignment, thereby further facilitating generative training.
Methodology
In this section, we first review the preliminary formulation of latent diffusion models (§ 3.1) and analyze the inherent off-manifold problem in unconstrained high-dimensional latent spaces. To mitigate this, we introduce the dual-manifold VAE framework that internalizes topological regularization without relying on external foundation models. We detail the dual-manifold alignment for the reconstruction stage (§ 3.2). Additionally, we introduce topological representation alignment (TRA) for the subsequent generation stage, along with an inference strategy, internal guidance with topological skeleton (IGT) (§ 3.3).
Preliminaries: Latent Diffusion Models
Latent diffusion models (LDMs) decouple the image generation pipeline into two distinct stages: 1) In the reconstruction stage, given an image , a continuous-valued VAE utilizes its encoder to map the image into a low-dimensional latent representation , and the decoder to reconstruct the image . The reconstructive VAE is optimized using a combination of reconstruction loss (e.g., and LPIPS [51]), a Kullback–Leibler (KL) divergence penalty towards a standard Gaussian prior, and an adversarial loss [12, 15]:
where and denote the loss weights. 2) In the generation stage, flow matching models operate within the constructed latent space . Specifically, let denote the encoded target latent and denote standard Gaussian noise. Flow matching constructs a probability path via a linear interpolation: for . To transport the prior distribution towards the latent distribution, the generative model learns a neural velocity field by regressing the constant ground-truth vector field . The training objective minimizes:
where . If the underlying latent manifold is ill-conditioned, unconstrained high-frequency components disrupt velocity regression and cause generated latents to drift off the manifold (i.e., the off-manifold problem), degrading generation in high-dimensional latent spaces.
Dual-Manifold Variational Autoencoders
Off-Manifold Analysis. As discussed in § 1, the overall generative performance is inherently upper-bounded by reconstruction quality. To push this bound, increasing the latent channel dimension is a standard practice. From an information-theoretic perspective [1, 9], the minimum reconstruction distortion is limited by the latent representation capacity. Since the latent spatial resolution is typically fixed, this capacity is dominated by , yielding an exponential theoretical bound1:
where and are positive constants determined by the intrinsic dimension and signal variance of the raw image manifold (manifold assumption [6, 29]). This inequality explicitly demonstrates that expanding the channel dimension is the most direct approach to minimize reconstruction error. However, this unregularized expansion inevitably introduces excessive degrees of freedom to the latent manifold. Mapping the intrinsically -dimensional image manifold into a high-dimensional space forms a latent manifold with an exploding codimension . Without any constraint, the encoder tends to exploit these excessive degrees of freedom. It may scatter substantial high-frequency energy across the normal directions to minimize pixel and adversarial errors, yielding an ill-conditioned latent manifold. During flow matching, the high-variance high-frequency components severely interfere with low-frequency structural velocity learning (cf., Appendix § 9), causing the off-manifold problem.
Dual-Manifold Alignment. While VAEs with low-dimensional latents exhibit degraded reconstruction quality compared to their high-dimensional counterparts, they typically achieve better generation [49]. This indicates that the low-dimensional latent manifold is inherently smoother and more generation-friendly [48]. Motivated by this, a straightforward idea is to leverage the low-dimensional latent manifold as a topological regularizer, compelling the high-dimensional manifold to inherit this property.
To achieve this, we design a dual-manifold VAE framework that simultaneously constructs two latent spaces sharing the encoder . Given an input image , the encoder first maps it into a high-dimensional latent , which is decoded by the main decoder into a high-fidelity reconstruction . In parallel, we compress into a low-dimensional latent via a linear down-projection layer (i.e., a convolution), followed by a lightweight decoder to output a structural reconstruction:
While the main branch is optimized with the full reconstructive objective , the low-dimensional branch is supervised solely by reconstruction loss . Constrained by the bottleneck and free from high-frequency adversarial training, forms a smooth manifold that captures the clean topological “skeleton” from (i.e., principal structural components). Rather than allowing the high-dimensional manifold to be optimized without any restriction, we treat as an internal topological regularizer. Concretely, we project back into the high-dimensional space via a corresponding up-projection layer to obtain the projected topological skeleton , and minimize the distance between and (i.e., dual-manifold alignment:
. The total training objective is:
where and are coefficients. Through , DM-VAE suppresses transverse high-frequency residual energy and confines within a narrow tubular neighborhood around lifted , decoupling structural velocity learning from high-frequency interference and mitigating the off-manifold problem (cf., Appendix B).
Topology-Guided Generation and Inference
Topological Representation Alignment. Building on the dual-manifold VAE (DM-VAE) framework, we naturally utilize the low-dimensional latent to guide generative training in the main high-dimensional latent space (cf., Figure 4). Specifically, given an interpolated latent at timestep , we extract the intermediate hidden features from the generative model and project them into the low-dimensional space via a lightweight projection head (e.g., an MLP) to obtain . We then align with the target topological skeleton by maximizing their cosine similarity:
Figure 4. Overview of topological representation alignment (TRA) and internal guidance with topological skeleton (IGT).
The overall generative training objective is formulated as: , where is a coefficient. Since captures the principal structural components of on a smooth manifold, compels the intermediate layers to establish a reliable structural representation early in the network. This explicit structural prior eases the subsequent regression of high-dimensional details, guiding the model to learn a more accurate velocity field .
Algorithm 1 Internal Guidance with Topological Skeleton
Require: Model with , DM-VAE , scale , interval , steps . Ensure: Image .
Initialize ,
for do
; if else
% 1. Single forward pass for velocity and skeleton
;
% 2. Separate skeleton and high-frequency details
; ;
% 3. Calibrate intermediate norm and guide the skeleton
, Eq. (6)
% 4. Update guided velocity and latent state
;
end for
return
Internal Guidance with Topological Skeleton. Beyond benefiting generative training, the aligned intermediate prediction also enables an effective inference strategy, internal guidance with topological skeleton (IGT). As shown in Figure 4, at timestep , the model outputs both the predicted velocity and in a single forward pass, yielding the one-step target estimate . While standard internal guidance (IG) [54] extrapolates between final and intermediate predictions across the entire latent space and may corrupt high-frequency details, IGT uses our dual-manifold VAE projections and to separate into the topological skeleton and high-frequency details : , , . Since is trained only with cosine similarity, we calibrate its norm using and project it to obtain the intermediate skeleton : . We then apply skeleton guidance (i.e., ) and add the high-frequency details back:
where is the internal guidance scale. The guided velocity is then updated as . In this way, IGT improves structural quality while mitigating off-manifold extrapolation drift by leaving the high-frequency details unperturbed. The detailed IGT process with time interval [25] is shown in Algorithm 1.
Experiments
Experimental Settings
Datasets, Baselines, and Evaluation Metrics. We evaluate both image reconstruction and class-conditional generation on ImageNet-1K [10] at a resolution of . Following the categorization in our main comparisons, we benchmark against two categories of baselines: 1) Vanilla latent diffusion models (without external foundation models), including DiT [35], MaskDiT [53], SiT [32], TREAD [23], and MDTv2 [14]; 2) Foundation-model-aligned and guided LDMs, which regularize either the VAE latent space (e.g., MAE-VAE and VA-VAE with LightningDiT [49], RAE [52], IG (VA-VAE) [54]) or the intermediate generative features (e.g., REPA [50], REPA-E [27], and REG [47]). To verify versatility across generative architectures, we integrate our DM-VAE and into DiT [35], following the LightningDiT [49] training settings across both Base (B) and Extra-Large (XL) scales. For evaluation metrics, we adopt reconstruction FID (rFID), PSNR, LPIPS [51], and SSIM [46] on both the ImageNet-1K 50K validation set (in-domain) and the Anime-Faces-256 benchmark [4] (5K, OOD) for the first stage, and reported Fréchet Inception Distance (FID) [17] (FID and FID), spatial FID (sFID) [33], Inception Score (IS), and Precision-Recall [24] for generation quality.
Implementation Details. 1) Reconstruction stage: During reconstructive training, we employ adaptive weighting coefficients to dynamically balance the loss terms. For our foundation-model-free variant (DM-VAE), we train two configurations within 50 epochs with a downsampling factor of 16: f16d32 (, low-dimensional branch ) and f16d64 (, low-dimensional branch ). To demonstrate orthogonality with external representations, we also train a DINOv2-integrated [34] variant (DM-VAE) under the f16d32 setting3. All training hyperparameters follow VA-VAE [49] (e.g., a global batch size of 256 and a learning rate of ). Detailed training settings are provided in Appendix § 10. 2) Generation stage: We conduct generative training on ImageNet-1K at a resolution of using DiT4, adhering to the official default training protocols [49, 54] (e.g., a global batch size of 1024 and a learning rate of ). We then employ a 250-step Euler solver for sampling.
Main Results
In-Domain and OOD Reconstruction. Table 1 compares reconstruction quality on ImageNet-1K (in-domain) and Anime-Faces-256 [4] (zero-shot OOD). 1) In-domain fidelity: Decoding from frozen semantic encoders in RAE [52] and RAEv2 [39] severely degrades pixel fidelity (19.20 and 23.29 dB PSNR), while external alignment in MAE-VAE and VA-VAE [49] (0.28 rFID) slightly trails vanilla SD-VAE (0.26 rFID). Conversely, our DM-VAE preserves local details and improves rFID to 0.21 (f16d32) and 0.12 (f16d64, 30.76 dB PSNR). 2) OOD generalization: On OOD anime illustrations, foundation-model-based tokenizers suffer from domain shift, with RAE/RAEv2 collapsing to 9.85/5.09 rFID and VA-VAE/MAE-VAE lagging behind SD-VAE in pixel and perceptual fidelity (29.78/29.33 dB vs. 30.19 dB PSNR). Free from external pretraining biases, DM-VAE generalizes robustly, achieving 1.93 rFID (30.30 dB PSNR) at f16d32 and 1.12 rFID (32.87 dB PSNR, 0.95 SSIM) at f16d64.
| Methods | rFID↓ | PSNR↑ | LPIPS↓ | SSIM↑ |
| In-domain reconstruction performance | ||||
| MAE-VAE (f16d32) [49] | 0.28 | 28.33 | 0.091 | 0.80 |
| VA-VAE (f16d32) [49] | 0.28 | 27.96 | 0.096 | 0.79 |
| RAE [52] | 0.57 | 19.20 | 0.254 | 0.44 |
| RAEv2 [39] | 0.31 | 23.29 | 0.148 | 0.61 |
| SD-VAE (f16d32) [38] | 0.26 | 28.59 | 0.089 | 0.80 |
| DM-VAE (f16d32) | 0.21 | 28.50 | 0.088 | 0.80 |
| DM-VAE (f16d64) | 0.12 | 30.76 | 0.057 | 0.87 |
| Out-of-distribution reconstruction performance | ||||
| MAE-VAE (f16d32) [49] | 2.38 | 29.78 | 0.048 | 0.91 |
| VA-VAE (f16d32) [49] | 2.05 | 29.33 | 0.052 | 0.90 |
| RAE [52] | 9.85 | 17.53 | 0.239 | 0.50 |
| RAEv2 [39] | 5.09 | 22.20 | 0.120 | 0.70 |
| SD-VAE (f16d32) [38] | 2.08 | 30.19 | 0.046 | 0.92 |
| DM-VAE (f16d32) | 1.93 | 30.30 | 0.046 | 0.92 |
| DM-VAE (f16d64) | 1.12 | 32.87 | 0.027 | 0.95 |
Table 1. In-domain (ImageNet-1K) and out-of-distribution (OOD) reconstruction performance comparison. Without relying on external foundation models, our DM-VAE achieves state-of-the-art reconstruction fidelity across both benchmarks.
Integration with DiT. Table 2 evaluates reconstruction and generation performance on DiT-B and DiT-XL (80/160 epochs, without CFG). 1) Reconciling the reconstruction–generation dilemma without external priors: At f16d32, DM-VAE improves rFID over SD-VAE (0.26 → 0.21) while lowering FID to 21.41/17.66 (B) and 8.89/7.97 (XL); pairing with IGT further reaches 12.60/9.22 (B) and 5.16/4.20 (XL), surpassing DINOv2-aligned VA-VAE [49] (20.34/15.73 and 8.22/6.75). More importantly, when expanding to f16d64, vanilla SD-VAE suffers severe off-manifold degradation (FID worsens to 17.24/12.38 on XL despite 0.17 rFID). Conversely, DM-VAE regularizes the 64-channel space to achieve 0.12 rFID alongside 10.40/8.59 FID on DiT-XL, and 5.44/4.26 with IGT. Figure 5 confirms sharp textures and coherent structures at only 64 epochs without CFG. 2) Synergy with external foundation models: Orthogonally combining DINOv2 in DM-VAE (f16d32) yields 0.27 rFID and further accelerates convergence, reaching 6.99/5.97 FID on DiT-XL and 4.86/4.33 with IGT (6.72/6.51 on DiT-B).

Figure 5. Qualitative generation results on ImageNet using DiT-XL with our foundation-model-free DM-VAE (f16d32) and at only 64 training epochs (without CFG).
Comparison with State-of-the-Art Methods. In Table 3, we benchmark our full framework against leading vanilla LDMs and foundation-model-guided LDMs on ImageNet-1K (256 × 256). 1) Foundation-model-free regime: Compared with vanilla LDMs such as vanilla DiT [35], SiT [32], and MDTv2 [14] that require 1080 ∼ 1600 epochs to converge, DiT paired with our DM-VAE (f16d32) reaches an FID of 2.62 (IGT alone, w/o CFG) and 1.67 (with CFG) at only 64 epochs, and achieves 1.25 without CFG and 1.17 with CFG at 400 epochs. Notably, when scaling to a 64-channel latent space (f16d64, 0.12 rFID), our dual-manifold regularization and IGT effectively prevent high-dimensional off-manifold degradation, achieving an FID of 3.01/2.10 at 64 epochs, 1.71/1.43 at 160 epochs, and 1.28 (w/o CFG) / 1.20 (with CFG) at 400 epochs. Consequently, both our f16d32 and f16d64 variants not only outperform all vanilla LDMs, but also surpass 800-epoch foundation-model-aligned methods including REPA [50] (1.42), REPA-E [27] (1.26), VA-VAE [49] (1.35), and RAE [52] (1.28). 2) Foundation-model-assisted regime: Furthermore, our framework is orthogonal to external priors; when combined with DINOv2 (DM-VAE), it reaches an FID of 2.26 (w/o CFG) within 60 epochs and further pushes the state-of-the-art FID (400 epochs) to 1.29 without CFG and 1.11 with CFG, outperforming IG [54] (VA-VAE) (1.19 at 680 epochs).
| Method | Epochs | #Params | Generation@256 w/o CFG FID↓ | sFID↓ | IS↑ | Prec.↑ | Rec.↑ | Generation@256 w/ CFG or Autoguidance FID↓ | sFID↓ | IS↑ | Prec.↑ | Rec.↑ |
| Latent Diffusion with Self-supervised Representation Models | ||||||||||||
| REPA [50] | 80 | 675M | 7.90 | 5.06 | 122.6 | 0.70 | 0.65 | - | - | - | - | - |
| 800 | 5.90 | 5.69 | 157.8 | 0.70 | 0.69 | 1.42 | 4.70 | 305.7 | 0.80 | 0.65 | ||
| REPA-E [27] | 80 | 675M | 3.46 | 4.17 | 159.8 | 0.77 | 0.63 | 1.67 | 4.12 | 266.3 | 0.80 | 0.63 |
| 800 | 1.83 | - | 217.3 | - | - | 1.26 | 4.11 | 314.9 | 0.79 | 0.66 | ||
| REG [47] | 80 | 677M | 3.40 | - | 184.1 | - | - | 1.86 | 4.49 | 321.4 | 0.76 | 0.63 |
| 480 | 2.20 | 4.67 | 219.1 | 0.77 | 0.66 | 1.40 | 4.24 | 296.9 | 0.77 | 0.66 | ||
| VA-VAE [49] | 64 | 675M | 5.14 | 4.22 | 130.2 | 0.76 | 0.62 | 2.11 | 4.16 | 252.3 | 0.81 | 0.58 |
| 800 | 2.17 | 4.36 | 205.6 | 0.77 | 0.65 | 1.35 | 4.15 | 295.3 | 0.79 | 0.65 | ||
| DDT [45] | 80 | 675M | 6.62 | - | 135.2 | 0.69 | 0.67 | 1.52 | - | 263.7 | 0.78 | 0.63 |
| 256 | 6.30 | - | 146.7 | 0.68 | 0.68 | 1.31 | - | 308.1 | 0.78 | 0.62 | ||
| 400 | 6.27 | - | 154.7 | 0.68 | 0.69 | 1.26 | - | 310.6 | 0.79 | 0.65 | ||
| IG [54] (VA-VAE) | 60 | 675M | 2.42 | 3.81 | 173.7 | 0.79 | 0.62 | - | - | - | - | - |
| 680 | 1.34 | 3.94 | 229.3 | 0.78 | 0.65 | 1.19 | 4.11 | 269.0 | 0.79 | 0.66 | ||
| RAE (DiTᴰᴴ) [52] | 800 | 839M | 1.60 | 5.40 | 242.7 | 0.79 | 0.65 | 1.28 | 4.72 | 262.9 | 0.78 | 0.67 |
| DiT-XL (DM-VAEᴰᵢₙₒ) | 60 | 675M | 2.26 | 4.81 | 200.3 | 0.79 | 0.60 | 1.36 | 4.12 | 276.3 | 0.79 | 0.63 |
| 400 | 1.29 | 4.28 | 253.3 | 0.79 | 0.64 | 1.11 | 4.03 | 297.4 | 0.79 | 0.65 | ||
| 640 | 1.29 | 4.22 | 260.4 | 0.79 | 0.65 | 1.15 | 4.22 | 303.8 | 0.79 | 0.65 | ||
| Vanilla Latent Diffusion (without foundation-model alignment) | ||||||||||||
| DiT [35] | 1400 | 675M | 9.62 | 6.85 | 121.5 | 0.67 | 0.67 | 2.27 | 4.60 | 278.2 | 0.83 | 0.57 |
| MaskDiT [53] | 1600 | 675M | 5.69 | 10.34 | 177.9 | 0.74 | 0.60 | 2.28 | 5.67 | 276.6 | 0.80 | 0.61 |
| SiT [32] | 1400 | 675M | 8.61 | 6.32 | 131.7 | 0.68 | 0.67 | 2.06 | 4.50 | 270.3 | 0.82 | 0.59 |
| TREAD [23] | 740 | 675M | - | - | - | - | - | 1.69 | 4.73 | 292.7 | 0.81 | 0.63 |
| MDTv2 [14] | 1080 | 675M | - | - | - | - | - | 1.58 | 4.52 | 314.7 | 0.79 | 0.65 |
| DiT-XL (DM-VAE, f16d32) | 64 | 675M | 2.62 | 4.62 | 190.1 | 0.78 | 0.60 | 1.67 | 4.12 | 253.4 | 0.80 | 0.62 |
| 160 | 1.61 | 4.23 | 222.8 | 0.79 | 0.63 | 1.29 | 4.01 | 264.3 | 0.80 | 0.64 | ||
| 400 | 1.25 | 4.16 | 250.8 | 0.79 | 0.65 | 1.17 | 4.02 | 283.0 | 0.80 | 0.64 | ||
| DiT-XL (DM-VAE, f16d64) | 64 | 675M | 3.01 | 4.61 | 178.4 | 0.79 | 0.58 | 2.10 | 4.24 | 224.6 | 0.81 | 0.59 |
| 160 | 1.71 | 4.32 | 218.1 | 0.80 | 0.62 | 1.43 | 4.14 | 252.1 | 0.81 | 0.62 | ||
| 400 | 1.28 | 4.16 | 245.7 | 0.79 | 0.64 | 1.20 | 4.10 | 274.8 | 0.80 | 0.65 |
Table 3. Quantitative comparison of state-of-the-art generative models on ImageNet-1K (256 × 256). We benchmark our method against vanilla latent diffusion models and foundation-model-aligned LDMs across different training epochs and model capacities. Our method establishes new state-of-the-art results with faster convergence in both foundation-model-free and foundation-model-assisted regimes. Note that our results in the “w/o CFG” column are evaluated with single-pass IGT.
Ablation Study
We conduct ablation studies on DiT-B at 80 training epochs (without CFG) in Table 4 to isolate the effectiveness of DM-VAE, topological representation alignment (TRA), and internal guidance with topological skeleton (IGT).
Table 4. Ablation study on DM-VAE, representation alignment, and internal guidance strategy at 80 epochs. Results are obtained under the f16d32 setting using DiT-B [35] without CFG. HB/LB denote the high/low-dimensional branch; IG [54] requires full-channel (-dim) HB predictions, whereas LB alignment predicts only the -dim skeleton and thus pairs with IGT.
DM-VAE and Low-Dimensional Alignment in TRA. Without any alignment or guidance, replacing SD-VAE with DM-VAE slashes FID from 25.37 to 19.03 and boosts IS from 52.22 to 69.06, confirming that our dual-manifold regularization yields a much more generation-friendly latent space. On top of DM-VAE, aligning intermediate features with HB () degrades FID to 19.52, since regressing high-frequency details disrupts global structural learning. In contrast, supervising with the skeleton LB () further reduces FID to 18.59, providing a clean structural prior for velocity learning.
IGT and Standard IG. We further compare standard internal guidance (IG) [54] against our IGT. While standard IG on HB () lowers FID to 10.98, extrapolating unaligned residuals across all channels amplifies velocity errors and limits semantic quality (91.40 IS). Selectively guiding only the smooth skeleton LB () while leaving residual details unperturbed achieves the best FID of 9.94, IS of 97.55, and Precision of 0.71.
Conclusion
In this paper, we identify that the off-manifold problem in high-dimensional VAEs stems from unconstrained channel expansion, whereas relying on external foundation models to regularize the manifold compromises out-of-distribution generalization. To resolve this dilemma internally, we propose the DM-VAE framework, which regularizes the high-dimensional latent space by anchoring it to a smooth low-dimensional topological skeleton. Building upon this, we introduce topological representation alignment (TRA) to provide explicit structural supervision for intermediate features during generative training, and internal guidance with topological skeleton (IGT) to guide structural components while keeping high-frequency details intact within a single forward pass at inference. Extensive experiments show that our framework reconciles in-domain and OOD reconstruction with fast generative convergence, achieving a 0.12 rFID and 1.28/1.20 FID (w/o and w/ CFG) at f16d64, 1.25/1.17 at f16d32 without external priors, and 1.11 when orthogonally combined with DINOv2.
LLM Usage Statement
LLMs (Gemini) were used solely at the Assist level to polish author-written text in the Abstract and Introduction. No LLMs or AI tools were used to generate research ideas or conclusions, compile related work or citations, create figures, or produce data, nor was the manuscript submitted to any LLM for reviewer-style feedback. The authors have verified all content and take full responsibility for the entire submission.
Detailed Off-Manifold Analysis
We derive the exponential reconstruction distortion bound in § 3.2. Under the manifold assumption [6, 29], natural images lie on an intrinsically -dimensional manifold . Let denote the local intrinsic coordinates on with finite differential entropy , and let denote the intrinsic coordinates of the nearest-point projection of onto . Under a locally bi-Lipschitz embedding, the pixel distortion satisfies for a constant . In the VAE Markov chain , the data processing inequality and the Shannon lower bound [9] yield:
where the third inequality follows from the maximum entropy of a -dimensional Gaussian with average coordinate variance , and is a constant determined by the intrinsic dimension and intrinsic entropy power of . Conversely, in a continuous VAE with encoder posterior and Gaussian prior on , the variational information bottleneck bound [1, 2] upper-bounds the mutual information by the KL divergence :
where denotes the average KL channel capacity per scalar latent entry. Combining Equations (7) and (8) gives , which establishes the exponential bound in § 3.2:
While increasing exponentially lowers this distortion floor, embedding the -dimensional into introduces a codimension of normal directions. Because (Eq. (1)) optimizes pixel and adversarial fidelity without constraining this -dimensional normal space, the encoder scatters substantial high-frequency energy along the normal directions, whose large variance disrupts low-frequency structural velocity learning in flow matching (cf., Appendix § 9).
Latent Manifold Geometry Analysis
In this section, we provide a geometric and information-theoretic analysis of how dual-manifold alignment () shapes the high-dimensional latent manifold into a generation-friendly tubular structure and decouples multi-frequency velocity learning in flow matching (FM).
Tubular Geometry of the DM-VAE Latent Space. Recall from § 3.2 that for any encoded latent , DM-VAE extracts () and lifts it via to obtain the topological skeleton . Consistent with § 3.3, any latent can be decomposed into:
where lies on the smooth base manifold (of subspace dimension ) supervised solely by , and captures fine-grained high-frequency residuals. By the Eckart–Young–Mirsky theorem, minimizing over naturally favors the rank- orthogonal projector onto the principal subspace . Since is simultaneously supervised by , in practice we impose a lightweight orthogonality penalty on the projection weights (, ) to preserve Euclidean orthogonality between the two subspaces:
Furthermore, directly bounds the expected energy of the transverse residual:
By Markov’s inequality, for any radius . Consequently, up to a negligible tail probability , the probability mass of is concentrated within a tubular neighborhood of radius around :
This tubular concentration provides two key benefits for generation: 1) Primary-to-residual velocity learning: Since , naturally prioritizes transporting noise toward the smooth skeleton before refining local residuals. **2) Bounded off-manifold drift: ** Even if the learned velocity incurs errors along the high-frequency residual directions, the small tubular radius ensures that generated latents remain tightly anchored around the valid skeleton .
Temporal SNR Decoupling in FM Interpolation. Consider the interpolation (, ). Projecting via the orthogonal projectors and onto the skeleton subspace (of dimension ) and residual subspace (of dimension ) yields two decoupled processes:
Within each subspace’s orthonormal basis, the projected standard Gaussian noises and are independent and retain unit variance per active subspace dimension for ). Under a zero-mean isotropic Gaussian approximation (encouraged by the standard normal KL prior) with per-subspace-dimension signal variance , the instantaneous subspace signal-to-noise ratio (SNR) at timestep is:
Minimizing the loss yields the optimal velocity , where the minimum mean-squared error (MMSE) denoiser in each subspace takes the closed-form Wiener filter expression:
When , the Wiener gain (the signal is submerged in noise and unrecoverable); when , (trivial linear recovery). The rate of information transition with respect to log-SNR, , attains its unique global maximum at [13], yielding the critical transition timestep:
Equation (17) reveals a key distinction between unconstrained VAEs and DM-VAE: 1) Spectral interference in unconstrained VAEs: In standard high-reconstruction VAEs (e.g., SD-VAE [38]), unregularized high-frequency details carry per-mode variance comparable to low-frequency structures (), causing their transition windows to overlap (). Because high-frequency targets exhibit large sample-to-sample variance, regressing them simultaneously at intermediate timesteps injects severe gradient variance that disrupts global structural learning. 2) Natural temporal decoupling in DM-VAE: By suppressing transverse residual energy via (), Equation (17) strictly separates the two transition windows:
At intermediate timesteps , while (), allowing the network to learn the low-variance velocity toward the global skeleton without high-frequency interference, before resolving fine textures near .
Empirical Verification. Figure 6 and Figure 7 empirically validate our geometric and SNR analysis on 5K ImageNet validation images under f16d64 (). First, encoded latents exhibit near-exact subspace orthogonality with , corroborating Eq. (11). In pixel space (Figure 6(a)), closely matches across global low frequencies (, with dB), while the detail increment concentrates on intra-patch high-frequency textures (). Second, in latent space (cf., Figure 6(b)), even when granting SD-VAE its optimal post-hoc rank-32 PCA projector (the theoretical lower bound on ), its detail transition window () heavily overlaps with the base window () around intermediate timesteps. Conversely, DM-VAE suppresses to push to 0.91 ( vs. 0.25), keeping detail transition rates near zero throughout .
Figure 6. Empirical verification of spectral decomposition and temporal SNR decoupling on 5K ImageNet validation images (, ). (a) Radially averaged 2D Fourier PSD in pixel space shows that the skeleton reconstruction preserves global low-frequency structures, whereas the detail increment rises by over 44 dB in relative power ratio () from to . (b) Transition rate curves (Eqs. (16)–(17)) parameterized by empirical , on globally unit-variance-normalized latents confirm that while SD-VAE exhibits substantial base–detail overlap (), DM-VAE pushes the detail transition to , isolating structural velocity learning within .

Figure 7. Qualitative visualization of latent subspace decomposition on ImageNet validation samples using DM-VAE (f16d64, ). From top to bottom: input image , full reconstruction , decoded skeleton , and residual . While preserves global structure and color layout, the residual isolates fine-grained edges and high-frequency textures.
Detailed Training Settings
DM-VAE Training. We train DM-VAE on the ImageNet-1K [10] training set at resolution (with random cropping and horizontal flipping), following the architectural and optimization conventions of SD-VAE [38] and VA-VAE [49]. 1) Architecture: The shared encoder and main decoder adopt a 5-stage convolutional backbone with channel multipliers (spatial downsampling factor ), base channel width 128, 2 residual blocks per stage, and self-attention at the latent resolution. To ensure efficiency and structural inductive bias, the auxiliary lightweight decoder halves the base channel width to 64, uses 1 residual block per stage, and removes all self-attention layers; is used only during training and discarded at inference. The linear projections and are implemented as bias-free convolutions with orthogonal initialization (). 2) Objectives and adaptive weighting: Both and sum pixel-level and perceptual LPIPS [51] losses with unit weights. We set the KL penalty weight and the low-dimensional branch weight . Adversarial training () is applied exclusively to the main reconstruction via a 3-layer PatchGAN discriminator [12] with hinge loss after 1,000 warmup steps, balanced adaptively by the gradient-norm ratio at the last convolutional layer of (with base weight 0.5). For dual-manifold alignment, is computed on the posterior means and using the symmetric stop-gradient (sg[·]) formulation in § 3.2. Its effective coefficient is dynamically balanced every 20 steps via an exponential moving average (EMA, decay 0.8) of the gradient-norm ratio at the shared pre-latent layer :
In addition, let and denote the reshaped kernel matrices of and . Consistent with Appendix § 9, we include an orthogonality regularizer () purely as a safeguard against degenerate rescaling or basis drift; given our orthogonal initialization, remains near zero throughout training and exerts negligible influence on optimization:
3) Optimization: Both DM-VAE and the discriminator are trained within 50 epochs using Adam (, , ) in bf16 on a 16 machine setup with a global batch size of 256, gradient norm clipping at 1.0, and an EMA decay of 0.9999.
We omit adversarial loss on the low-dimensional branch () for both training efficiency and optimization stability. First, incorporating an auxiliary adversarial objective noticeably slows down DM-VAE training due to additional discriminator passes and adaptive gradient-norm evaluations. Second, adversarial training is inherently unstable under a strict capacity bottleneck: in our preliminary experiments, applying a GAN loss to consistently triggered late-stage adversarial collapse, as the discriminator easily overpowered the capacity-limited lightweight decoder and broke the minimax equilibrium.
DiT Training. We conduct class-conditional generative training on ImageNet-1K () [10] strictly adhering to the official settings of LightningDiT [49] and IG [54]. For topological representation alignment (LTRA), the intermediate features are extracted from the same default transformer layer as in IG [54] and mapped to via a lightweight MLP projection head . For fair evaluation at inference, both our methods and all reproduced baselines adopt an identical 250-step Euler solver with uniform class-balanced sampling across all classes.
More Results
Quantitative Results on DDT. To verify the generality of our framework beyond standard DiT, we further integrate DM-VAE (f16d32) and IGT into the Decoupled Diffusion Transformer (DDT) [45] across Base (B) and Extra-Large (XL) scales at 80 training epochs without CFG (cf., Table 5). Using our foundation-model-free DM-VAE without guidance, DDT (DM-VAE) substantially outperforms both Improved-SiT [32] (25.1 15.2 FID on B; 10.9 5.9 on XL) and Improved-REPA [50] (19.1 15.2 on B; 8.1 5.9 on XL), while simultaneously improving spatial fidelity (6.08/4.75 sFID) and diversity (0.66/0.68 Recall). Incorporating DINOv2 alignment in DM-VAEDino further reduces FID to 11.0 (B) and 4.1 (XL). When combined with single-pass IGT, generation quality improves, reaching 5.1 FID (136.9 IS) on DDT-B and 2.0 FID (4.35 sFID, 213.3 IS) on DDT-XL without CFG, confirming that our dual-manifold alignment and skeleton guidance generalize across distinct architectures.
| Methods | FID | sFID | IS | Prec. | Rec. |
| Improved-SiT-B [32] | 25.1 | 6.54 | 58.8 | 0.57 | 0.64 |
| Improved-REPA-B [50] | 19.1 | 6.88 | 76.5 | 0.60 | 0.66 |
| DDT-B (DM-VAE) | 15.2 | 6.08 | 84.0 | 0.63 | 0.66 |
| DDT-B (DM-VAEDino) | 11.0 | 5.47 | 94.8 | 0.70 | 0.62 |
| + IGT | 5.1 | 5.23 | 136.9 | 0.77 | 0.57 |
| Improved-SiT-XL [32] | 10.9 | 5.30 | 103.4 | 0.66 | 0.65 |
| Improved-REPA-XL [50] | 8.1 | 5.34 | 124.9 | 0.68 | 0.67 |
| DDT-XL (DM-VAE) | 5.9 | 4.75 | 139.8 | 0.70 | 0.68 |
| DDT-XL (DM-VAEDino) | 4.1 | 4.43 | 157.0 | 0.75 | 0.64 |
| + IGT | 2.0 | 4.35 | 213.3 | 0.80 | 0.61 |
Table 5. Generation performance comparison on ImageNet at 80 generative training epochs using DDT [45] without CFG. denotes baselines improved by SwiGLU [43], RoPE [41], and RMSNorm [43]. Our DM-VAEs adopt the f16d32 configuration. For IGT, we consistently apply a guidance scale of 1.7 across the timestep interval of .
Qualitative Generation Results. We provide more visual samples in Figure 8 and Figure 9 without using CFG.

Figure 8. Qualitative generation results using DiT-XL with our foundation-model-free DM-VAE (f16d64) at 400 epochs.

Figure 9. Qualitative generation results using DiT-XL with our foundation-model-free DM-VAE (f16d64) at 400 epochs.
Impact of Low Dimension . We further investigate the choice of the low-dimensional bottleneck under the f16d64 () configuration. On DiT-XL at 80 training epochs (without CFG or IGT), setting achieves an FID of 10.40, whereas further compressing it to degrades FID to 12.00. This reflects an inherent trade-off in selecting , which should be neither overly large nor overly small. An excessively large weakens the information bottleneck, leaving itself unconstrained with excessive degrees of freedom and prone to manifold degradation, which in turn compromises its topological regularization on the high-dimensional latent . Conversely, an overly small severely restricts skeleton capacity and fails to adequately suppress transverse high-frequency residual energy, widening the tubular neighborhood around lifted and limiting its ability to mitigate the off-manifold problem. Balancing skeleton smoothness and residual energy suppression, we set by default across our main experiments.
References
- [1]Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rifat A Saurous, and Kevin Murphy. Fixing a broken elbo. In ICML, pages 159–168. PMLR, 2018.arxiv.org/abs/1711.00464
- [2]Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint, 2016.arxiv.org/abs/1612.00410
- [3]Black Forest Labs. Flux.2 latent-space representation comparison. https://bfl.ai/research/representation-comparison, 2025. Accessed: 2026.
- [4]Gwern Branwen, Anonymous, and The Danbooru Community. Danbooru2021: A large-scale crowdsourced and tagged anime illustration dataset. https://www.gwern.net/Danbooru2021, 2022.
- [5]Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint, 2015.arxiv.org/abs/1509.00519
- [6]Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542–542, 2009.
- [7]Hila Chefer, Patrick Esser, Dominik Lorenz, Dustin Podell, Vikash Raja, Vinh Tong, Antonio Torralba, and Robin Rombach. Self-supervised flow matching for scalable multi-modal synthesis. arXiv preprint, 2026.arxiv.org/abs/2603.06507
- [8]Hongxu Chen, Hongxiang Li, Zhen Wang, and Long Chen. Bi-anchor interpolation solver for accelerating generative modeling. arXiv preprint, 2026.arxiv.org/abs/2601.21542
- [9]Thomas M Cover, Joy A Thomas, and John Kieffer. Elements of information theory. wiley New York, 1991.
- [10]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. IEEE, 2009.
- [11]Carl Doersch. Tutorial on variational autoencoders. arXiv preprint, 2016.arxiv.org/abs/1606.05908
- [12]Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873–12883, 2021.arxiv.org/abs/2012.09841
- [13]Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.arxiv.org/abs/2403.03206
- [14]Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is a strong image synthesizer. arXiv preprint, 2023.DOI
- [15]Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. ACM, 63(11):139–144, 2020.
- [16]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 15979–15988. IEEE, 2022.arxiv.org/abs/2111.06377
- [17]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local Nash equilibrium. NeurIPS, 30, 2017.
- [18]Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
- [19]Matthew D Hoffman, Matthew J Johnson, et al. Elbo surgery: yet another way to carve up the variational evidence lower bound. In NeurIPS Workshop, 2016.
- [20]Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In ICIP, pages 2366–2369. IEEE, 2010.DOI
- [21]Dengyang Jiang, Mengmeng Wang, Liuzhuozheng Li, Lei Zhang, Haoyu Wang, Wei Wei, Guang Dai, Yanning Zhang, and Jingdong Wang. No other representation component is needed: Diffusion transformers can provide representation guidance by themselves. arXiv preprint, 2025.arxiv.org/abs/2505.02831
- [22]Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint, 2013.arxiv.org/abs/1312.6114
- [23]Felix Krause, Timy Phan, Ming Gui, Stefan Andreas Baumann, Vincent Tao Hu, and Björn Ommer. Tread: Token routing for efficient architecture-agnostic diffusion training. In ICCV, pages 15703–15713. IEEE, 2025.arxiv.org/abs/2501.04765
- [24]Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. NeurIPS, 32, 2019.
- [25]Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. NeurIPS, 37:122458–122483, 2024.
- [26]Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, et al. Flux. 1 kon-text: Flow matching for in-context image generation and editing in latent space. arXiv preprint, 2025.arxiv.org/abs/2506.15742
- [27]Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-1: Unlocking vae for end-to-end tuning with latent diffusion transformers. In ICCV, pages 18262–18272. IEEE, 2025.
- [28]Hongxiang Li, Hongxu Chen, Chenyang Zhu, Xiaoshuang Huang, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Long Chen. Spar: Semantic-pixel self-alignment and adaptive routing for unified multimodal models. In ECCV, pages 324–343. Springer, 2026.arxiv.org/abs/2606.23041
- [29]Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. In CVPR, pages 36115–36125, 2026.arxiv.org/abs/2511.13720
- [30]Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint, 2022.arxiv.org/abs/2210.02747
- [31]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint, 2022.arxiv.org/abs/2209.03003
- [32]Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In ECCV, pages 23–40. Springer, 2024.arxiv.org/abs/2401.08740
- [33]Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint, 2021.arxiv.org/abs/2103.03841
- [34]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint, 2023.arxiv.org/abs/2304.07193
- [35]William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4172–4182. IEEE, 2023.arxiv.org/abs/2212.09748
- [36]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
- [37]Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278–1286. PMLR, 2014.
- [38]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10674–10685. IEEE, 2022.arxiv.org/abs/2112.10752
- [39]Jaskirat Singh, Boyang Zheng, Zongze Wu, Richard Zhang, Eli Shechtman, and Saining Xie. Improved baselines with representation autoencoders. arXiv preprint, 2026.arxiv.org/abs/2605.18324
- [40]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS, 32, 2019.arxiv.org/abs/1907.05600
- [41]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
- [42]Jakub Tomczak and Max Welling. Vae with a vampprior. In AISTATS, pages 1214–1223. PMLR, 2018.arxiv.org/abs/1705.07120
- [43]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Ahmad Almahairi, Yasmin Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint, 2023.arxiv.org/abs/2307.09288
- [44]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint, 2025.arxiv.org/abs/2502.14786
- [45]Shuai Wang, Zhi Tian, Weilin Huang, and Limin Wang. Ddt: Decoupled diffusion transformer. In CVPR, pages 40633–40642, 2026.arxiv.org/abs/2504.05741
- [46]Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 13(4):600–612, 2004.pubmed.ncbi.nlm.nih.gov/15376593
- [47]Ge Wu, Shen Zhang, Ruijing Shi, Shanghua Gao, Zhenyuan Chen, Lei Wang, Zhaowei Chen, Hongcheng Gao, Yao Tang, Ming-Ming Cheng, et al. Representation entanglement for generation: Training diffusion transformers is much easier than you think. NeurIPS, 38:7714–7743, 2026.arxiv.org/abs/2507.01467
- [48]Tongda Xu, Mingwei He, Shady Abu-Hussein, Jose Miguel Hernandez-Lobato, Chunhang Zheng, Kai Zhao, Chao Zhou, Ya-Qin Zhang, and Yan Wang. Making reconstruction fid predictive of diffusion generation fid, 2026.arxiv.org/abs/2603.05630
- [49]Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. In CVPR, pages 15703–15712. IEEE, 2025.arxiv.org/abs/2501.01423
- [50]Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint, 2024.arxiv.org/abs/2410.06940
- [51]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586–595. IEEE, 2018.arxiv.org/abs/1801.03924
- [52]Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders. In ICLR, pages 35791–35820, 2026.arxiv.org/abs/2510.11690
- [53]Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers, 2024.arxiv.org/abs/2306.09305
- [54]Xingyu Zhou, Qifan Li, Xiaobin Hu, Hai Chen, and Shuhang Gu. Guiding a diffusion transformer with the internal dynamics of itself. In CVPR, pages 11536–11545, 2026.arxiv.org/abs/2512.24176


