WorksComputer Science
Analyzing Learning-Dynamics Across Loss and Prediction Choices for Diffusion Models
Abstract
Denoising generative models predict clean data (), noise (), or velocity (), while training losses can be measured in any of these spaces. Although the resulting nine combinations are algebraically related, they can yield substantially different denoising accuracy and sample quality. We disentangle the effects of prediction and loss spaces. For a fixed prediction target, we show that changing the loss space is exactly equivalent to timestep reweighting and provide a heuristic analysis of how this reweighting affects generation quality. We then show that a rank- linear bottleneck compressing a -dimensional noisy input imposes an irreducible squared-error floor of on noise and velocity prediction, whereas clean-data prediction can exploit low-dimensional data structure. Combining this bound with prediction-dependent error conversion yields distinct necessary conditions on bottleneck capacity. Experiments with U-Net and JiT across CIFAR-10, Imagenette, and ImageNet-256 support these theoretical predictions. Under compression, prediction choice strongly affects both denoising accuracy and sample quality, with noise-prediction errors closely matching the theoretical lower bound. Our results provide a quantitative understanding of how architectural bottlenecks interact with prediction and loss choices in denoising generative models.
Introduction
Diffusion and flow-based generative models train a neural network to reverse a corruption process: given a noisy observation at noise level , the network predicts a target quantity (clean data , noise , or velocity ) from which the original signal can be recovered [18, 7, 19, 14, 15]. Since are related by affine transforms conditioned on , these parameterizations are algebraically equivalent in the oracle regime, and prior work often treats them as interchangeable up to loss reweighting [17, 10, 11].
Yet algebraic equivalence does not guarantee comparable performance in finite-capacity networks. Li and He [ [13] introduce JiT, a pixel-space Transformer with low-rank patch embeddings, and report substantially better FID with -prediction than with - or -prediction. They motivate -prediction through the low-dimensional structure of natural images, suggesting that architectural compression may affect prediction targets differently. However, how such compression affects the relative difficulty of different prediction targets remains insufficiently understood.
We revisit this question by separating two design choices: the prediction space (what the network predicts) and the loss space (where errors are measured), considering all nine combinations over . Although prior work has examined prediction parameterizations and loss weightings jointly [13, 4], the effects of architectural information loss and prediction-dependent error conversion remain less understood. We analyze loss-space choices through timestep reweighting and prediction-space choices through bottleneck-induced error bounds and conversion amplification. Together, these analyses yield testable predictions about how architectural compression affects denoising accuracy across prediction targets.
Contributions. We summarize our contributions as follows.
A joint analysis of prediction and loss spaces. We show that changing the loss space for a fixed prediction target is exactly equivalent to deterministic timestep reweighting (Proposition 1), collecting known affine identities into a unified formulation. We further provide a heuristic analysis of how this reweighting affects generation quality, suggesting that - and -loss can outperform -loss under high- sensitivity assumptions (Proposition 2).
Irreducible prediction errors under architectural bottlenecks. For a rank- linear bottleneck compressing a -dimensional noisy input, we establish an error floor of for - and -prediction, whereas -prediction can exploit the low-dimensional structure of the data (Theorem 4 and Corollary 5). This result quantifies the information-loss mechanism underlying the prediction-space asymmetry observed in JiT [13].
Prediction-dependent error amplification. We characterize how converting different prediction targets to clean-data estimates scales their errors, with factors for -prediction and for -prediction (Proposition 6). Combining these identities with the bottleneck error floor yields prediction-dependent necessary conditions on bottleneck capacity (Corollary 7).
Empirical validation across architectures and scales. We evaluate prediction and loss choices using synthetic data, U-Net, and JiT across CIFAR-10, Imagenette, and ImageNet-256. Our experiments support the predicted effects of loss-space reweighting and quantitatively validate the bottleneck error bounds. In particular, the -prediction error closely matches the theoretical floor under strong compression, while bottleneck-width sweeps reveal distinct capacity requirements across prediction targets.
Related work
Prediction parameterizations and loss weighting. Common diffusion parameterizations predict noise [7], the score [20], or a velocity-related quantity. Salimans and Ho [ [17] study prediction parameterizations and loss weightings jointly. EDM [10] introduces preconditioning that normalizes the scales of network inputs and regression targets across noise levels. Kingma and Gao [ [11] express common diffusion objectives as weighted losses and connect monotonic weightings to ELBO maximization with noisy data augmentation. Explicit weighting strategies include Min-SNR [5] and P2 [2], while Chen [ [1] study noise-schedule design. For our linear interpolation, Proposition 1 collects the established loss-conversion identities into a common statement for a fixed prediction space.
Joint analyses of prediction and loss. JiT [13], a pixel-space Transformer using -prediction with -loss, reports substantial performance differences across prediction targets, motivating a systematic analysis of prediction and loss choices. Their ablation evaluates all nine combinations of prediction and loss spaces. Kumar et al. [ [12] compare native-target squared-error objectives and their ELBO-derived counterparts across data, noise, velocity, and score parameterizations, and analyze the rescalings relating these objectives. Gagneux et al. [ [4] systematically study the interplay between output parameterization and loss weighting, comparing denoising PSNR across noise levels and FID under different architectures and data regimes. Their inverse-variance analysis further provides a statistical motivation for emphasizing low-noise timesteps.
Dimensionality, bottlenecks, and error conversion. Jin and Wang [ [9] analyze the learning dynamics of linear diffusion predictors by decomposing their behavior into components parallel and orthogonal to the data subspace. They characterize how data dimensionality affects the optimal prediction target and propose -Diff, which learns a data-dependent prediction parameter. We study predictions formed from a rank- linear compression of a -dimensional noisy input. Under the stated assumptions, we derive an expected squared error floor of for noise and velocity prediction (Theorem 4 and Corollary 5). Prior work has discussed numerical instability and error amplification associated with converting noise predictions to clean-data estimates [17, 10, 8]. Under linear interpolation, we combine the corresponding error-scaling identities with our bottleneck bounds to obtain prediction-dependent necessary conditions on bottleneck rank (Proposition 6 and Corollary 7).
Setting and notation
We consider training data in and noise . For , define the noisy observation
and the velocity
A model is parameterized by a neural network whose direct output (prediction space) is one of
where is modeled as a trainable neural network parameterized by . Given and one predicted quantity, the other two are obtained by solving the linear system induced by (1) and (2). This yields deterministic affine transforms among .
Training uses a squared loss in a chosen loss space :
where is computed from by the appropriate transform. Note that -prediction determines the score: by Tweedie’s formula, [20], an affine map whose time-dependent scaling corresponds to a specific loss weighting within our framework.
Goal. In the oracle regime, all are equivalent up to deterministic transformations. The scientific question is to formalize how learning dynamics differ for finite capacity and gradient-based optimization, and to design experiments that disentangle (i) timestep reweighting effects from (ii) geometric and capacity effects.
Learning dynamics of diffusion
We analyze how prediction and loss choices affect learning through two distinct mechanisms. We first establish that changing the loss space induces deterministic timestep reweighting and examine how this weighting affects generation quality. We then turn to the prediction-space axis, studying how architectural bottlenecks introduce irreducible errors and how prediction-dependent conversion amplifies these errors. Proofs for the theoretical results are collected in Appendix § 8.
Proposition 1 (Equivalence of loss spaces as timestep reweighting). Assume the linear interpolation and velocity definition
with sampled from the data and noise distributions. Fix a prediction space , meaning the network directly outputs
For any loss space , define as the quantity obtained by solving the linear system induced by together with the constraints defining the pair , with replaced by . Then for every pair there exist:
a scalar slope that depends only on , and
*an intercept term that depends on but is independent of ,
such that the true target and the model prediction satisfy the aligned affine relations
and
Consequently, the residuals obey the pointwise scaling identity
and hence the squared loss in -space is exactly a weighted squared loss in the direct output -space:
where . Since does not depend on , the Bayes-optimal predictor is the same for all loss spaces . For the linear schedule, the weights are listed in Table 1; the individual identities appear in prior work [17, 11, 10], and we collect them here for reference.
| Weight | Loss | Loss | Loss |
| Pred | |||
| Pred | |||
| Pred |
Table 1. The implicit timestep weights induced by different loss-space choices, derived from the linear-schedule relations between collected from prior work [17, 11, 10]. Diagonal entries () are unity. Off-diagonal entries involving or diverge at endpoints and require clamping in practice.
Proposition 1 states that changing loss space only reweights timesteps; the Bayes-optimal predictor is unchanged. Empirical differences must arise from finite capacity and optimization dynamics, not from a different oracle. In expectation, applying a weight in the loss is equivalent to sampling from a tilted distribution . In practice, off-diagonal weights can diverge (e.g., ); following [13], we clamp . By contrast, changing prediction space changes itself, so is not covered by this equivalence.
The weights connect to the SNR framework [11]: for -prediction, , so -loss concentrates optimization on high-SNR timesteps while -loss distributes it uniformly. This subsumes EDM’s weighting [10] and explains why -loss emphasize the generation-critical high- regime.
Loss-space ordering: Why some weightings are better
Proposition 1 establishes that changing the loss space (with fixed prediction space ) is equivalent to changing the timestep weighting . At the population level, all weightings yield the same Bayes-optimal predictor . However, with finite model capacity, the weighting determines which timesteps receive more optimization pressure. We now provide a heuristic analysis of why this matters for generation quality, then validate the key prediction empirically.
Setup. Consider an ODE-based sampler that integrates from (noise) to (data) using the learned velocity field . Let denote the velocity error at time , and . By a first-order perturbation of the ODE , the final-sample error satisfies
This linearization is accurate when velocity errors are small relative to the signal. For the heuristic surrogate below, we restrict the integration to an interior timestep interval , neglecting contributions outside this interval. By Proposition 1 with , a loss in space weights the squared velocity error by the effective weight , which is , , and for -, -, and -loss (see Table 1). These effective weights do not depend on the prediction space .
Proposition 2 (Loss-Space Ordering (heuristic)). Consider the following modeling assumptions:
(i) First-order approximation: The sample error is dominated by (6).
(ii) Cross-time correlations: By (i), . We summarize the cross-time correlations by a sensitivity , so that , and assume that does not depend on the training weighting.
(iii) Capacity allocation: With finite model capacity trained under effective weight and time distribution , the per-timestep velocity MSE scales as
Under these assumptions, the expected sample error satisfies
If in addition
(iv) High- sensitivity: for natural images, is concentrated at high ,
then
Derivation. By (i) and (ii), , and substituting (iii) gives (8). Under (iv), is dominated by high . At high , the -loss weight approaches the -loss weight 1, whereas the -loss weight approaches zero, thereby placing diminishing emphasis on velocity-space errors. The - and -loss integrals are therefore close, and the -loss integral is much larger, which is (9).
Discussion. Assumption (i) approximates the accumulated sampling error by integrating local velocity errors, neglecting their propagation through the evolving trajectory. Assumptions (ii) and (iii) are heuristics: real networks share parameters across timesteps and the actual allocation depends on architecture, optimizer, and data. Assumption (iv) is a property of the data rather than of the model. We state the result as a proposition to make each premise falsifiable; the ordering is heuristic, not universal. Section B.6 evaluates under the time distributions used in training. With constant , the -loss integral already exceeds the other two by one to two orders of magnitude, so (iv) is needed only to place -loss next to -loss. Empirically, the ordering holds for - and -prediction on the U-Net and, more weakly, for -prediction on JiT-B/8 (§ 5). It fails on simple 2D distributions, where (iv) does not hold (Appendix § 9), and under strong compression on ImageNet-256, where -loss gives the worst FID for -prediction (Table 4).
| Config | Pred\Loss | -MSE: | -MSE: | -MSE: | FID: | FID: | FID: |
| () | 0.023 | 0.023 | 0.023 | 23.9 | 36.8 | 20.1 | |
| () | 14.61 | 14.61 | 14.61 | 421.2 | 419.0 | 420.1 | |
| () | 0.294 | 0.294 | 0.294 | 201.5 | 202.3 | 199.0 | |
| () | — | — | 0.022 | — | — | 5.7 | |
| () | — | — | 0.043 | — | — | 330.6 | |
| () | — | — | 0.026 | — | — | 140.6 |
Table 4. JiT-B/16 on class-conditional ImageNet . Validation -MSE and FID (, 50K samples). Top: (). Bottom: (). Under compression, prediction type dominates both metrics (-space MSE 0.834 ; see Section B.8). Without compression and at full budget, MSE converges across prediction types (0.022–0.043) but FID does not, so aggregate denoising MSE does not determine FID.
Remark 1 (Connection to existing strategies). Our framework subsumes existing weighting strategies: EDM [10] weights the -space error by our -loss weight plus a constant (Appendix § 10), min-SNR [5] interpolates between uniform and -loss, and P2 [2] modulates by perceptual priority. All of these upweight high- relative to -loss, consistent with (9). DDPM [7] uses , SD3 [3] uses , JiT [13] uses .
Bottleneck lower bounds from information loss: why is special under manifold structure
We now turn from loss-space reweighting to the effect of prediction-space choices under architectural bottlenecks. Clean data lies on or near a low-dimensional manifold of natural images, whereas the Gaussian noise and the velocity fill the ambient . We will show that combining this asymmetry with an architectural information bottleneck yields a sharp consequence: - and -prediction incur an irreducible error floor, while -prediction remains feasible.
Definition 3 (Bottleneck model). We use the following assumptions.
(A1) Subspace data: , where satisfies with , and on .
(A2) Isotropic noise: , independent of .
(A3) Linear bottleneck: the model observes only , where and .
(A1) is the linear idealization of the manifold hypothesis. (A3) matches JiT’s patch embedding, which applies two linear convolutions with no nonlinearity in between.
For a Vision Transformer, corresponds to the flattened patch dimension for patches with channels, and corresponds to the patch-embedding dimension to which the architecture compresses each patch before processing. We use and throughout when referring to these quantities for concrete architectures, and write for the bottleneck ratio, where indicates patch-level compression and no compression.
Theorem 4 (Irreducible error from null-space components). Assume (A3) and let denote the orthogonal projector onto . For any measurable estimator of a target ,
If in addition (A2) holds, then is independent of , and for
Intuitively, the observation carries no information about the component of in , and that component has expected squared norm exactly . No estimator that sees only can recover it, so the null-space residual is an irreducible error floor from information loss through . Data on a low-dimensional subspace can pass through the projection intact, while full-dimensional noise cannot. The next corollary makes this precise.
Corollary 5 (Bottleneck separation between and ). Assume (A1) to (A3) and
(A4) Injectivity on the data subspace: , which requires .
Then the bottleneck preserves all directions in the -dimensional data subspace, since . In contrast, every estimator of based on satisfies
Thus, the bottleneck can preserve the low-dimensional data subspace while discarding noise components in its -dimensional null space. This creates an irreducible prediction-error floor for - and -prediction that does not arise from the same null-space mechanism for -prediction.
When does the bottleneck apply? U-Nets bypass this via skip connections (effective rank ), and EDM’s [10] provides a direct input-to-output path; Appendix § 10 places EDM in our framework. Architectures with patch-level compression and no skip path (JiT [13], any ViT with ) satisfy the assumption directly. Since is learned end-to-end, the gradient incentivizes it to preserve the data subspace for -prediction, confirmed by the near-identical MSE across loss types in Table 3.
| Config | Pred\Loss | MSE (-space): | MSE (-space): | MSE (-space): | FID: | FID: | FID: |
| (ratio 0.67) | 0.050 | 0.048 | 0.048 | 41.0 | 37.4 | 36.0 | |
| (ratio 0.67) | 5.855 | 5.848 | 5.852 | 146.6 | 147.6 | 151.0 | |
| (ratio 0.67) | 0.152 | 0.152 | 0.152 | 113.2 | 108.3 | 113.0 | |
| (ratio 1.33) | 0.049 | 0.048 | 0.048 | 42.1 | 38.9 | 37.1 | |
| (ratio 1.33) | 0.051 | 0.051 | 0.051 | 114.1 | 100.1 | 123.6 | |
| (ratio 1.33) | 0.049 | 0.050 | 0.049 | 58.9 | 60.3 | 58.1 |
Table 3. JiT-B (M params) on Imagenette with class conditioning and CFG. Top: (). Bottom: (). Under compression, -prediction achieves threefold better FID than -prediction and 120-fold better MSE than -prediction.
Remark 2 (Nonlinearity after linear compression). The lower bound of Theorem 4 applies to any predictor that observes the noisy input only through a rank- linear projection , regardless of nonlinear processing applied after this projection. In particular, inserting a GELU between JiT’s two bottleneck convolutions preserves the initial linear compression and therefore does not violate the assumptions of the theorem. Our ablation in Table 10 shows similar prediction-space asymmetry with and without this additional nonlinearity. This demonstrates robustness to nonlinear processing after compression, rather than extending the theoretical guarantee to arbitrary nonlinear compression maps .
| (compression) | (no compr.) | |||
| -pred | -pred | -pred | -pred | |
| Linear | 0.048 | 5.85 (120×) | 0.048 | 0.051 (6%) |
| GeLU | 0.049 | 5.9 (120×) | 0.049 | 0.059 (21%) |
Table 10. Effect of post-bottleneck nonlinearity on Imagenette. A 2×2 factorial design compares JiT’s original linear patch embedding with a variant inserting a GeLU between its two convolutions, under compressed and uncompressed configurations. Cells show test MSE; numbers in parentheses give the -prediction gap relative to -prediction.
Conversion amplification: Why -prediction requires more capacity
Corollary 5 gives the same lower bound, , on the expected squared prediction errors for and under a bottleneck. We show that conversion to -space yields different lower bounds and, consequently, different necessary conditions on bottleneck rank.
Proposition 6 (Conversion amplification). Let be a model’s prediction in space , and let denote the clean-data estimate obtained by algebraic conversion. If the model incurs pointwise prediction error against the true target, then the induced -space error satisfies:
The proof follows by direct algebraic conversion: and .
Under the bottleneck assumptions, both - and -prediction incur an expected squared error of at least (Corollary 5). Combining this error floor with the conversion factors in Proposition 6 gives prediction-dependent lower bounds on the -space error. The following corollary states the resulting necessary conditions on bottleneck rank.
Corollary 7 (Prediction-type-dependent threshold). Under (A1)–(A4), for any - or -predictor based on , the converted -space MSE satisfies
Under logit-normal sampling with and , the expected conversion weights are approximately 0.40 for -prediction and 20 for -prediction. For the same target -space MSE, the necessary upper bound on is therefore about 50 times smaller for -prediction than for -prediction. These lower bounds become trivial at full rank .
We empirically examine the predicted difference in bottleneck sensitivity in § 5. The bounds in Corollary 7 become trivial without compression and therefore do not explain the FID differences in that regime. Nevertheless, the per-timestep analysis in Section B.8 reveals error patterns consistent with conversion amplification, although a complete explanation of the FID gap remains open.
Experiments
We systematically evaluate all nine combinations of prediction target and loss space across two architectures: a standard U-Net with skip connections (97M params, no information bottleneck) and the JiT-B architecture [13] (130M params, with a configurable bottleneck in the patch embedding). Experiments span CIFAR-10 (), Imagenette () and ImageNet-256, with synthetic data validating dimensional scaling.
Setup. The U-Net is trained unconditionally with Adam (, , lr ), logit-normal time sampling (, ), gradient clipping at norm 1.0, and EMA with decay 0.9999. JiT-B is trained class-conditionally with AdamW (, ), logit-normal time sampling (, ), label dropout 0.1 for classifier-free guidance (CFG) [6], and EMA with decay 0.9999. The CIFAR-10 and Imagenette models are trained for 500 epochs. For ImageNet-256, we train the compressed configurations for 100 epochs and the uncompressed controls for 600 epochs, as detailed in Section B.8. JiT uses the full architecture from [13] including RMSNorm, RoPE, QK-norm, SwiGLU, and in-context class tokens. We report validation MSE in -space (averaged over 3 seeds for CIFAR-10 U-Net, 2 seeds for JiT CIFAR-10) and FID (computed with clean-fid [16]; U-Net: Heun, 100 steps, unconditional; JiT: Heun, 50 steps, CFG). JiT’s BottleneckPatchEmbed applies two linear convolutions without an intermediate nonlinearity, so the bottleneck is a rank- linear projection: precisely the setting of Theorem 4. We characterize each configuration by the bottleneck ratio , where indicates patch-level compression and no compression.
Loss-space effects without bottleneck: generation quality. Proposition 2 predicts that -loss and -loss should outperform -loss in generation quality by placing greater optimization weight on high- timesteps. We focus on - and -prediction, which achieve substantially better FID than
-prediction under the U-Net setting. The U-Net results in Table 2 support the predicted loss ordering. On CIFAR-10, switching from -loss to -loss reduces FID from 11.79 to 2.20 for -prediction and from 12.51 to 1.12 for -prediction, with similar improvements on Imagenette. -loss also consistently outperforms -loss in these settings.
| Dataset | Pred\Loss | \multicolumn{3}{c}{MSE ()} | \multicolumn{3}{c}{FID} | ||||
| CIFAR-10 | 5.395 | 5.360 | 5.361 | 11.79 | 2.20 | 2.73 | |
| CIFAR-10 | 5.401 | 5.372 | 5.367 | 16.45 | 25.49 | 5.88 | |
| CIFAR-10 | 5.392 | 5.356 | 5.359 | 12.51 | 1.12 | 1.68 | |
| Imagenette | 4.810 | 4.664 | 4.632 | 53.40 | 12.76 | 21.05 | |
| Imagenette | n/a1 | 4.709 | 4.670 | 85.77 | 95.41 | 100.98 | |
| Imagenette | 4.763 | 4.649 | 4.630 | 43.11 | 7.06 | 14.32 |
Table 2. U-Net results across datasets. Validation MSE in -space and FID ().
Prediction-space effects under bottleneck. With an architectural information bottleneck, the prediction target becomes the dominant factor in denoising accuracy. Theorem 4 and Corollary 5 establish an irreducible error floor of for - and -prediction under linear compression. Our synthetic experiments in Figure 1 directly validate this bound: the null-space MSE of -prediction closely matches , while that of -prediction remains above the bound. On Imagenette, Table 3 shows that JiT-B with bottleneck ratio achieves an -space MSE of approximately 0.048 for -prediction, compared with 0.152 for -prediction and 5.85 for -prediction. The FID results exhibit the same ordering, while changing the loss space within each prediction type has little effect on MSE. When the bottleneck is removed (), the MSE gap narrows substantially: - and -prediction achieve MSE around 0.049, while -prediction reaches 0.051. Together, these results show that loss-space reweighting has a relatively modest effect on denoising error, whereas representational information loss can make the prediction target decisive. The same separation is observed on ImageNet-256 under strong compression (), where -prediction achieves an -space MSE of 0.023, compared with 0.294 for -prediction and 14.61 for -prediction (Table 4). An additional ablation inserting a GELU after the initial linear compression exhibits a similar prediction-space asymmetry, indicating robustness to post-bottleneck nonlinear processing (Table 10).
Figure 1. Synthetic validation of the bottleneck lower bound (Theorem 4) with and . Left: Prediction MSE in each target’s native space as a function of bottleneck dimension . The - and -prediction errors are bounded below by (dashed), while -prediction maintains low error for . Middle, Right: Decomposition of the prediction MSE into null-space and range-space components for - and -prediction, respectively.
Bottleneck capacity threshold. Corollary 7 shows that, under compression (), the irreducible error floor imposes a stronger constraint on -prediction than on -prediction, suggesting that -prediction requires greater bottleneck capacity to achieve comparable denoising accuracy. Since the bound becomes trivial at , we empirically investigate recovery by varying the bottleneck width of BottleneckViT, a lightweight ViT that follows JiT’s patch-embedding design but adds a GELU after the initial linear projection (see Appendix § 9). On CIFAR-10, the gap between - and -prediction nearly disappears once compression is removed (Table 7). In contrast, on Imagenette, -prediction remains less accurate even without compression and approaches -prediction only with a substantially wider embedding (; Table 8). This recovery threshold is empirical and is not predicted by Corollary 7. Additional synthetic sweeps over , , and (Section B.1) and a patch-size sweep at fixed embedding width (Section B.2) further support the predicted dependence of prediction errors on .
| Ratio | -pred MSE | -pred MSE | Gap | |
| 32 | 0.67 | 0.054 | 0.158 | +191% |
| 64 | 1.33 | 0.054 | 0.054 | +0.2% |
| 128 | 2.67 | 0.054 | 0.054 | +0.0% |
| 256 | 5.33 | 0.054 | 0.054 | +0.0% |
Table 7. Bottleneck dimension sweep on CIFAR-10 (BottleneckViT, ).
| Ratio | -pred MSE | -pred MSE | Gap | Floor | |
| 32 | 0.17 | 0.049 | 14.61 | 14.61 | |
| 256 | 1.33 | 0.048 | 0.095 | +98% | — |
| 384 | 2.00 | 0.048 | 0.056 | +17% | — |
| 512 | 2.67 | 0.048 | 0.053 | +9% | — |
| 768 | 4.00 | 0.048 | 0.050 | +2.6% | — |
Table 8. -prediction recovery with increasing bottleneck width on Imagenette (). Results use BottleneckViT with -loss, 500 training epochs, and seed 42. Validation -space MSE is averaged over . The floor is , obtained from Corollary 7 on this evaluation grid.
Convergence behavior without compression. The bottleneck lower bound in Theorem 4 applies to rank-deficient embeddings () and becomes trivial when . On JiT-B/4 CIFAR-10 (), the initially large denoising MSE gap between - and -prediction closes with longer training (Figure 4), suggesting that prediction-type differences without compression can partly reflect optimization difficulty rather than irreversible information loss. However, comparable denoising accuracy does not necessarily imply comparable generation quality. On ImageNet-256 without compression (), the models trained for 600 epochs achieve relatively similar -space MSE, yet their FID scores differ substantially (Table 4).
Figure 4. Loss curves on JiT-B/4 CIFAR-10 (, -loss). (a) -prediction starts roughly 80-fold worse than -prediction but reaches parity by epoch 200. (b) The MSE ratio decays from 80-fold to roughly unity, suggesting that the bottleneck effect at is a convergence penalty rather than an irreducible error.
To examine the role of error conversion, Figure 2 compares per-timestep errors in velocity space, which is used by ODE sampling, and in -space after conversion. The -prediction model exhibits sharply increasing velocity-space MSE at low , consistent with the amplification of squared errors during -to-velocity conversion. In contrast, - and -prediction maintain substantially lower errors in this region. These observations suggest that conversion amplification contributes to the observed FID gap, although it does not fully explain the differences, particularly the poor FID of -prediction. Section B.8 provides further analyses in a common velocity space and compares ODE and SDE sampling.

Figure 2. Per-timestep prediction errors on ImageNet-256 using JiT-B/16 without compression (, 600 epochs). Left: Velocity-space MSE against sampled targets, used to assess errors relevant to ODE sampling. Right: -space MSE after conversion. The velocity-space error of -prediction increases sharply at low , consistent with the amplification factor, while - and -prediction remain bounded over the evaluated timesteps. Results below also reflect denominator clamping.
Conclusion
We have presented a joint analysis of prediction and loss spaces in denoising generative models. For a fixed prediction target, changing the loss space is equivalent to timestep reweighting, which can affect generation quality under finite model capacity (Propositions 1 and 2). For prediction-space choices, we show that a rank-deficient linear bottleneck introduces an irreducible error floor of for - and -prediction, whereas -prediction can exploit low-dimensional data structure (Theorem 4 and Corollary 5). Combining this bound with prediction-dependent error conversion yields necessary conditions on bottleneck capacity that are more restrictive for -prediction than for -prediction (Proposition 6 and Corollary 7).
Experiments with U-Net and JiT across CIFAR-10, Imagenette, and ImageNet-256 support these analyses. Under compression, -prediction maintains low denoising error, while - and -prediction are more sensitive to bottleneck capacity. In particular, the observed -prediction error closely matches the theoretical lower bound under strong compression, and bottleneck-width sweeps reveal distinct recovery patterns across prediction targets. Together, our findings clarify how prediction and loss choices interact with architectural information constraints and provide guidance for selecting prediction targets in compressed generative models.
Limitations. Our theoretical bounds rely on a linear bottleneck and a low-dimensional subspace model of the data. Our GeLU ablation retains the initial linear bottleneck and therefore does not establish corresponding guarantees for architectures with general nonlinear compression maps. The loss-ordering analysis is heuristic and depends on assumptions about sampling sensitivity and the allocation of model capacity across timesteps.
Our experiments span different architectures, conditioning schemes, and datasets, so absolute FID values are not directly comparable across settings. Moreover, the compressed ImageNet-256 configurations were trained for a shorter budget than the uncompressed controls, and we did not evaluate their convergence under extended training. Without compression, our error bounds become trivial. Although prediction-dependent conversion can amplify errors during sampling, particularly for -prediction, it does not fully explain the observed FID differences. Our stochastic sampling ablation is also limited to samplers that share the same velocity conversion. Extending the analysis to latent diffusion, text conditioning, and architectures with explicit skip connections or EDM-style preconditioning remains an important direction for future work.
Acknowledgments and Disclosure of Funding
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2024-00337955); by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (RS-2024-00457882, Artificial Intelligence Research Hub Project; RS-2019-II191906, Artificial Intelligence Graduate School Program (POSTECH)); and by IITP grants funded by the Korea government (MSIT) under the Leading Generative AI Human Resources Development Program (IITP-2026-RS-2026-25546560) and the AI Star Fellowship Support Program (IITP-2026-RS-2026-25616313).
Proofs
Proof of Proposition 1
Proposition 1 (Equivalence of loss spaces as timestep reweighting). Assume the linear interpolation and velocity definition
with sampled from the data and noise distributions. Fix a prediction space , meaning the network directly outputs
For any loss space , define as the quantity obtained by solving the linear system induced by together with the constraints defining the pair , with replaced by .
Then for every pair there exist:
a scalar slope that depends only on , and
an intercept term that depends on but is independent of ,
such that the true target and the model prediction satisfy the aligned affine relations
and
Consequently, the residuals obey the pointwise scaling identity
and hence the squared loss in -space is exactly a weighted squared loss in the direct output -space:
*where . Since does not depend on , the Bayes-optimal predictor is the same for all loss spaces . For the linear schedule, the weights are listed in Table 1; the individual identities appear in prior work [17, 11, 10], and we collect them here for reference.
Proof. The result follows from the affine conversion formulas among given . From and , any two of determine the third:
For any prediction space and loss space , substituting for in these formulas yields , and the same substitution of the true yields the true . Explicitly, the slopes and intercepts are:
Pred : From the linear interpolation, we have
Therefore,
Pred : , so . , so .
Pred : , so . , so .
In all cases, depends only on , and the intercept is independent of (it involves only and ). Since by linearity, squaring and taking expectations gives , where . The diagonal entries are , and the off-diagonal entries are:
which are exactly the entries in Table 1. Since depends only on , the population minimizer over all measurable predictors is for every loss space .
Proof of thm:nullspacelb
Theorem 4 (Irreducible error from null-space components). Assume (A3) and let denote the orthogonal projector onto . For any measurable estimator of a target ,
If in addition (A2) holds, then is independent of , and for
Proof. Let be any estimator that depends only on the inputs . We define the ideal Bayes estimator . We can decompose the error of as:
Squaring the norm and taking expectations:
The cross-term vanishes by the tower property of conditional expectation: since is measurable with respect to , it can be pulled out of the inner expectation:
Thus, we obtain the standard decomposition:
This proves the first inequality.
The term represents the total variance of the unrecoverable residual. For any vector , and any orthogonal projector , we have . Applying this to the residual vector with the null-space projector :
Since is a deterministic linear operator, it commutes with the conditional expectation by linearity of expectation:
Taking expectations proves the second inequality:
Consider the case with isotropic noise . Write , where projects onto the column space of . Since is isotropic Gaussian and are orthogonal projectors onto complementary subspaces, the components and are independent Gaussian vectors (a standard property of orthogonal projections of Gaussian vectors). Since depends on only through (as by definition), we have . Therefore:
Substituting this into our bound:
For isotropic Gaussian noise, the expected squared norm of its projection is simply the dimension of the subspace:
Since has rank , the dimension of its null space is . Thus, the error is at least .
Proof of cor:separation
Proof of prop:conversionamp
Proof of cor:predthreshold
Proof of Corollary 5
Proof. Under (A1), with . By (A4), has full column rank, so the bottleneck mapping is injective on the data subspace. Thus, unlike the noise components in the null space of , the data-subspace components are not irreversibly lost through the linear projection.
For -prediction, Theorem 4 directly gives
For , we have . By (A2), is independent of and has zero mean. Therefore,
Applying Theorem 4 and using this independence, we obtain
Proof of Proposition 6
The conversions from prediction space to -space are: , so ; , so ; , so . Taking squared norms yields (11) through (13).
Proof of Corollary 7
Fix . Under (A2) and (A3), is independent of . Hence, the arguments of Theorem 4 and Corollary 5 apply conditionally on , giving
| Pred\Loss | U-Net (no bottleneck) | U-Net (no bottleneck) | U-Net (no bottleneck) | JiT-B (, bottleneck) | JiT-B (, bottleneck) | JiT-B (, bottleneck) |
| 0.0481 | 0.0466 | 0.0463 | 0.050 | 0.048 | 0.048 | |
| 0.0486 | 0.0471 | 0.0467 | 5.9 | 5.8 | 5.9 | |
| 0.0476 | 0.0465 | 0.0463 | 0.152 | 0.152 | 0.152 |
Table 5. Comparison of validation -space MSE across prediction and loss choices on Imagenette. For the U-Net without an information bottleneck, MSE varies primarily with the loss space. For JiT-B with a bottleneck (), MSE varies primarily with the prediction space. The full results, including FID, are reported in Table 2 and Table 3.
By Proposition 6, converting these prediction errors to -space multiplies them by and , respectively. Taking expectations over yields the stated lower bounds.
For the numerical values, let with and . Since ,
while numerical quadrature gives
The ratio of these conversion weights is approximately 50.
On the discrete evaluation grid used in Table 8 and Table 9, the corresponding averages are 17.53 and 0.33, respectively. The tabulated lower bounds additionally divide by to match the reported per-dimension MSE.
| x-space error | ϵ-space error | ||||||
| -pred | -pred | -pred | -pred | ||||
| 2 | 12 | 2.67 | — | 0.054 | 0.055 | 0.054 | 0.060 |
| 4 | 48 | 0.67 | 0.333 | 0.054 | 5.85 | 0.158 | 0.353 |
| 8 | 192 | 0.17 | 0.833 | 0.056 | 14.62 | 0.321 | 0.842 |
Table 9. Patch-size sweep on CIFAR-10 (BottleneckViT, , -loss, 200 epochs, seed 42). Validation MSE per dimension, averaged over . The first three error columns are in -space; the last is the -space error of -prediction, to be compared with the floor .
Additional experiments and details
Bottleneck lower bound: synthetic validation
We validate the irreducible error lower bound (Theorem 4) using the synthetic manifold setup with a controlled linear bottleneck. Fix ambient dimension and intrinsic dimension . We sample a column-orthonormal matrix , draw from a mixture of Gaussians on , and embed as . With and , the model accesses only through a trainable embedding . We train all nine combinations and sweep from 1 to , decomposing the MSE into null-space and range-space components. We report prediction MSE in each target’s native space, together with its null-space and range-space components and convergence diagnostics. As predicted in Figure 1, - and -prediction exhibit an irreducible error floor, while -prediction achieves low error for all . For , the null-space MSE matches precisely, confirming the theorem; for , it lies on or above this floor, as Corollary 5 requires.
Sweep over , , , and the embedding . We extend this setup along all three dimensions of Theorem 4 and Corollary 5: , , and from below up to , for all three prediction types. Each configuration is trained with -loss for 2,000 epochs, once with a learned and once with a random frozen at initialization, over two seeds, for 444 runs in total. We report squared error per vector with , averaged over seeds; Table 6 lists every configuration. Three findings stand out. First, the floor of Theorem 4 is exact in all three variables. Across the 62 configurations with , the null-space error of -prediction lies between 0.997 and 1.012 times . -prediction lies above the same floor in every configuration, as Corollary 5 predicts. Its proof adds the posterior error of to the floor, and with a learned and this excess stays between 12.8 and 18.7 for every and . Second, -prediction depends on whether rather than on or . With a learned , its error stays between 11.4 and 12.5 whenever and rises to between 15.9 and 17.1 when , so the transition sits at the injectivity condition (A4). Third, a random separates the two mechanisms. The floor is the same for random and learned to within 1.5%, so it depends only on and is information-theoretic rather than an artifact of training the embedding. For -prediction, a random is much worse near , for example 24.9 against 11.4 at and , and approaches the learned value as . A learned thus finds the subspace-aligned solution already at the minimal rank , consistent with the discussion after Corollary 5 that end-to-end training pushes to preserve the data subspace.
| null / learned | null / random | null / learned | error learned | error random | |||
| 16 | 2 | 1 | 1.001 | 1.001 | 1.96 | 17.0 | 23.5 |
| 16 | 2 | 2 | 1.002 | 1.003 | 2.04 | 11.5 | 22.7 |
| 16 | 2 | 4 | 1.003 | 1.003 | 2.19 | 11.4 | 16.0 |
| 16 | 2 | 8 | 1.006 | 1.005 | 2.80 | 11.6 | 13.0 |
| 16 | 2 | 12 | 0.997 | 1.012 | 4.64 | 11.6 | 12.2 |
| 16 | 2 | 16 | — | — | — | 11.7 | 11.6 |
| 16 | 8 | 4 | 1.000 | 1.003 | 2.22 | 16.0 | 18.2 |
| 16 | 8 | 8 | 1.007 | 1.001 | 2.80 | 12.2 | 16.1 |
| 16 | 8 | 12 | 1.006 | 1.000 | 4.21 | 11.6 | 12.8 |
| 16 | 8 | 16 | — | — | — | 11.5 | 11.5 |
| 64 | 2 | 1 | 1.002 | 1.003 | 1.23 | 17.1 | 21.9 |
| 64 | 2 | 2 | 1.003 | 1.003 | 1.24 | 11.7 | 18.8 |
| 64 | 2 | 4 | 1.002 | 1.003 | 1.25 | 11.7 | 16.7 |
| 64 | 2 | 16 | 1.004 | 1.004 | 1.30 | 11.6 | 14.8 |
| 64 | 2 | 32 | 1.006 | 1.005 | 1.46 | 11.8 | 13.0 |
| 64 | 2 | 48 | 1.007 | 1.006 | 1.90 | 11.8 | 12.2 |
| 64 | 2 | 64 | — | — | — | 11.9 | 11.3 |
| 64 | 8 | 4 | 1.003 | 1.003 | 1.25 | 15.9 | 18.8 |
| 64 | 8 | 8 | 1.004 | 1.003 | 1.27 | 12.2 | 17.5 |
| 64 | 8 | 16 | 1.004 | 1.005 | 1.37 | 11.7 | 15.6 |
| 64 | 8 | 32 | 1.007 | 1.007 | 1.58 | 11.6 | 13.5 |
| 64 | 8 | 48 | 1.007 | 1.006 | 2.13 | 11.6 | 12.3 |
| 64 | 8 | 64 | — | — | — | 11.8 | 11.4 |
| 128 | 2 | 1 | 1.004 | 1.004 | 1.12 | 16.8 | 24.6 |
| 128 | 2 | 2 | 1.004 | 1.004 | 1.12 | 11.4 | 24.9 |
| 128 | 2 | 4 | 1.004 | 1.004 | 1.12 | 11.6 | 22.9 |
| 128 | 2 | 32 | 1.006 | 1.006 | 1.15 | 11.6 | 14.0 |
| 128 | 2 | 64 | 1.007 | 1.008 | 1.23 | 11.7 | 12.7 |
| 128 | 2 | 96 | 1.012 | 1.010 | 1.43 | 11.6 | 12.0 |
| 128 | 2 | 128 | — | — | — | 11.5 | 11.6 |
| 128 | 8 | 4 | 1.004 | 1.004 | 1.13 | 15.9 | 19.6 |
| 128 | 8 | 8 | 1.005 | 1.004 | 1.13 | 12.5 | 18.1 |
| 128 | 8 | 16 | 1.005 | 1.005 | 1.17 | 11.7 | 17.2 |
| 128 | 8 | 32 | 1.005 | 1.006 | 1.19 | 11.7 | 15.0 |
| 128 | 8 | 64 | 1.006 | 1.008 | 1.28 | 11.6 | 13.2 |
| 128 | 8 | 96 | 1.009 | 1.011 | 1.56 | 11.7 | 12.4 |
| 128 | 8 | 128 | — | — | — | 11.6 | 11.7 |
Table 6. Synthetic sweep of Theorem 4 and Corollary 5 over ambient dimension , intrinsic dimension , and bottleneck rank , with a learned or a frozen random embedding (BottleneckMLP, -loss, 2 seeds). The and columns give the null-space error of - and -prediction divided by the floor ; the columns give the total error of -prediction. Errors are squared norms per vector with . Rows with have no floor.
Bottleneck threshold sweep
BottleneckViT and BottleneckMLP. BottleneckViT is a simplified Vision Transformer used as a computationally cheaper proxy for JiT-B in bottleneck-width sweeps.
Its patch embedding follows JiT’s BottleneckPatchEmbed, but inserts a GELU between the two convolutions. Importantly, the first convolution is a linear mapping from the noisy patch to a -dimensional representation. All subsequent computations, including the GELU, depend on the input only through this representation. Thus, under compression (), the architecture satisfies assumption (A3) of Theorem 4, provided the first mapping has full row rank.
BottleneckViT uses fewer transformer blocks and a smaller hidden width than JiT-B, giving approximately 22M parameters versus JiT-B’s 130M. This enables systematic bottleneck-width sweeps at substantially lower computational cost.
BottleneckMLP is the analogous synthetic-data variant, using an MLP backbone with a linear input bottleneck.
Tables 7 and 8 show the full bottleneck-ratio sweep on BottleneckViT.
Patch size at fixed embedding width. Theorem 4 predicts that the floor depends on the patch through . Increasing the patch size at fixed should therefore degrade - and -prediction monotonically, while leaving -prediction unaffected as long as exceeds the intrinsic dimension of a patch. We test this with BottleneckViT on CIFAR-10 at and fixed , so that ranges from 12 to 192 under the same dataset and training recipe: -loss, 200 epochs, seed 42. Table 9 reports validation MSE averaged over . At the embedding is wider than the patch, so there is no bottleneck and the three prediction types match. As grows, -prediction stays flat at 0.054 to 0.056, while - and -prediction degrade monotonically. The -space error of -prediction lands on the floor at both bottlenecked sizes. Averaged over it is 0.353 against 0.333 and 0.842 against 0.833, and for it matches the floor to within 1%. Through Proposition 6, the floor then predicts the -space error of -prediction, as in Corollary 7. The mean amplification over the evaluation grid is 17.53, so the prediction is 5.84 at and 14.61 at , against measured values of 5.85 and 14.62. -prediction lies above its corresponding bound , which is 0.11 and 0.28, because its floor contains the component. The same values recur across setups. At the -prediction error of 5.85 equals that of JiT-B/8 on Imagenette (Table 3), and at the value 14.62 matches JiT-B/16 on ImageNet-256 (Table 4). Since the error sits on the floor and the amplification depends only on the evaluation grid, the bottleneck ratio governs this error, not the resolution or the architecture.
Compression vs. post-bottleneck nonlinearity
We examine whether introducing nonlinear processing after the initial linear bottleneck affects the prediction-space asymmetry. Specifically, we insert a GeLU between JiT’s two patch-embedding convolutions. In both variants, the first convolution compresses each noisy patch into a -dimensional representation, and all subsequent computations depend only on this representation. Thus, the linear bottleneck assumption of Theorem 4 remains satisfied under compression.
As shown in Table 10, -prediction exhibits similarly large errors under compression in both variants, while its error decreases substantially when compression is removed. These results demonstrate that the prediction-space asymmetry persists despite the additional nonlinearity after the initial linear compression.
Generated sample comparison
To complement the quantitative MSE and FID gaps reported in Table 3, we visualize the qualitative effect of prediction-space choice under a fixed bottleneck. Figure 3 shows samples generated by JiT-B/8 with at ratio 0.67 on Imagenette using classifier-free guidance, with 100 Heun steps. All three rows use the same architecture and training recipe. - and -prediction are trained with -loss and -prediction with -loss; under this bottleneck the loss choice changes MSE by at most 3.3% within a prediction type (Table 3), so the rows differ mainly in the prediction target. The visual ordering tracks the MSE ordering precisely: -prediction yields recognizable class-conditional images, -prediction yields heavily degraded but still structured samples, and -prediction collapses to pure noise. This corroborates Theorem 4: the irreducible null-space error of for propagates through the ODE sampler to produce samples with no learnable structure, while -prediction, free of the null-space penalty under the manifold assumption, recovers usable samples even at the same bottleneck width.
Figure 3. Generated samples from JiT-B/8 (, ratio 0.67) on Imagenette with classifier-free guidance. -prediction produces recognizable images; -prediction is heavily degraded; -prediction produces pure noise. All three use the same 130M-parameter architecture and 100 Heun steps; - and -prediction use -loss and -prediction uses -loss.
JiT-B/4 convergence behavior
Figure 4 reports per-epoch validation MSE for JiT-B/4 on CIFAR-10 at bottleneck ratio , trained for 500 epochs with -loss, alongside an -prediction baseline trained under the same setup. Panel (a) plots the absolute loss curves; panel (b) shows their ratio. This single training run is the evidence for the convergence-penalty framing in § 5: it shows that, on this configuration, the prediction-type gap closes with continued training rather than persisting as an information-theoretic floor. We have not measured this systematically across , datasets, or architectures. The ImageNet-256 study (Section B.8) adds one more point at the regime boundary: at and 600 epochs, the -prediction denoising gap narrows to a factor of two without fully closing, and the FID gap from conversion amplification persists regardless.
Numerical evaluation of the loss-ordering integral
We evaluate the integral of Proposition 2 for the three effective weights under the logit-normal time distributions used in training. A logit-normal density vanishes at both endpoints, so diverges on . We therefore integrate over . This cutoff is a modeling choice: it equals the conversion clamp of the JiT recipe, but the training time distribution itself covers . Table 11 normalizes each integral by its -loss value. With constant , which ignores assumption (iv), the -loss integral is 126 to 256 times the -loss integral, while the -loss integral is 1.5 to 11.4 times. The excess of -loss comes from low , where its weight vanishes. Restricting the sensitivity to high , , brings the -loss integral within a factor of 1.43 of the -loss integral and leaves the -loss integral more than two orders of magnitude larger, which is the ordering (9). For integration cutoffs between 0.02 and 0.1, the -loss integral remains the largest, by a factor of at least 28.
| Time distribution | Sensitivity | -loss | -loss | -loss |
| logit-normal(, 1.0), U-Net | 1 | 1 | 11.4 | 125.9 |
| 1 | 1.43 | 148.5 | ||
| logit-normal(, 0.8), JiT | 1 | 1 | 1.49 | 255.9 |
| 1 | 1.17 | 257.0 |
Table 11. The loss-ordering integral of Proposition 2 over , normalized by its -loss value. Effective weights are 1, , and for -, -, and -loss.
Loss ordering on simple distributions
The ordering (9) relies on assumption (iv) of Proposition 2, that generation quality depends disproportionately on high- accuracy. This holds for natural images, where fine-grained details such as textures and edges are resolved near and dominate FID. However, for simple distributions such as 2D synthetic datasets, the denoising task is approximately equally difficult at all timesteps. In this regime, our synthetic experiments show that the loss ordering partially reverses: diagonal losses (, uniform weight) become near-optimal, and -loss can underperform -loss. This is consistent with the analysis: when denoising difficulty is uniform across , concentrating capacity on any particular timestep range cannot help.
ImageNet-256 scale validation
To confirm that both regimes persist at the resolution and dataset scale of modern generative models, we train the full JiT-B/16 architecture on class-conditional ImageNet-1k at (1.28M images, 1000 classes), where each patch has . We run the complete design at (ratio , strong compression) for 100 epochs each, and a three-configuration control at (ratio , no compression) trained to 600 epochs. All runs use batch size 1024, AdamW (, , lr , constant after a 5-epoch warmup), logit-normal time sampling (, ), label dropout 0.1, velocity clamp , and EMA decay 0.9999, matching the JiT recipe of the Imagenette experiments. FID is computed with clean-fid [16] over 50K samples (Heun, 50 steps, CFG 2.9) against the full training set. Table 4 reports the results.
Both phenomena from the smaller-scale experiments reproduce. Under compression (), the prediction-space dichotomy is stark: -prediction attains val -MSE 0.023 and FID 20–37 across all loss types, -prediction degrades to MSE 0.29 and FID , and -prediction collapses to MSE 14.6 and FID . The -prediction error in -space sits at 0.834–0.837 across the three loss types, within 0.5% of the closed-form floor of Theorem 4, a direct quantitative confirmation of the bottleneck lower bound at scale.
Removing the bottleneck () sharpens the central message of the paper. We train all three control configurations to the full 600-epoch budget so that the comparison is at convergence rather than confounded by training time. The val -MSE spread collapses from 600-fold under compression to within a factor of two (0.022–0.043), confirming that information is no longer lost; yet the FID gap remains enormous (5.7 for -pred, 140.6 for -pred, 330.6 for -pred). This is the dual dichotomy: once the architecture can represent the target, denoising quality (MSE) nearly equalizes, while prediction type still dominates generation quality (FID). For -prediction, the per-timestep analysis below locates the gap at low , where the conversion of Proposition 6 is ill-conditioned. -prediction needs no conversion for ODE sampling, so Proposition 6 does not explain its gap. Its velocity error stays within 30% of the -model’s at every we measure and is lower at (Table 12), and we leave the cause of its FID gap open. Crucially, the gap is not an undertraining artifact: from 100 to 600 epochs the -prediction FID falls sharply (24.8 5.7, reaching the range of published ImageNet-256 models [13]), while -prediction barely moves (340.2 330.6) and -prediction improves only modestly (173.5 140.6). At convergence, -prediction remains worse than -prediction in FID despite a near-identical -MSE (0.026 vs. 0.022), so the FID-vs-MSE decoupling is not a transient of finite training or an artifact of the bottleneck.
| Velocity-space error | Native error | |||
| -model | -model | -model | -model | |
| 0.06 | 0.106 | 0.443 | 0.116 | 0.0016 |
| 0.10 | 0.081 | 0.198 | 0.093 | 0.0020 |
| 0.50 | 0.051 | 0.067 | 0.066 | 0.017 |
| 0.90 | 0.189 | 0.197 | 0.197 | 0.160 |
| 0.95 | 0.462 | 0.337 | 0.333 | 0.304 |
Table 12. Per-timestep error of the three separately trained JiT-B/16 models on ImageNet-256 (, 600 epochs, -loss) in the common velocity space, together with the -model’s native error . Mean squared error per dimension against sampled targets, which includes the Bayes risk, over 2,560 validation images.
Figure 2 makes the mechanism explicit. Measuring per-timestep error on the converged checkpoints, -prediction’s velocity-space error, the quantity the ODE integrates, grows sharply as , reaching 0.44 at and 1.07 at versus at mid-range. Above this is the low- blow-up predicted by Proposition 6, and below the conversion clamp described next also contributes: - and -prediction stay bounded. Following the JiT recipe, every conversion clamps its denominator or at . The sampler starts at , so the first steps of -prediction, at , 0.02, and 0.04, fall in the clamped region, as do the last steps of -prediction at and 0.98. The clamp caps the amplification at , but even an exact network output then yields a velocity shrunk by the factor for -prediction, or for -prediction. The -prediction FID gap therefore reflects the conversion together with the clamp that regularizes it, and we do not separate the two contributions.
Comparing separately trained models. Different models may have different prediction errors before conversion. We therefore compare their predictions in velocity space, using the same target for all three models. This is also the quantity used by the ODE sampler.
Remark 3 (Comparing prediction errors in velocity space). For a model predicting , let be the Bayes predictor and the error relative to it. The class label is implicit in the conditioning of the Bayes predictors. These predictors map exactly onto one another under the affine conversions in Proposition 1.
Let denote the velocity estimate obtained from . At each fixed where denominator clamping is inactive,
where is given in Table 1. The cross term vanishes because and is determined by the model’s inputs.
All three prediction types share the same Bayes risk in velocity space. Differences in their expected velocity MSEs therefore equal differences in their excess MSEs. The converted excess MSE depends on both the conversion weight and the excess MSE before conversion. The same decomposition holds for MSE per dimension after normalizing all terms by the number of output coordinates.
Table 12 reports empirical estimates of these velocity-space MSEs. At , the -model’s MSE is 0.0016 before conversion and 0.443 in velocity space, where the conversion weight is . The -model’s velocity-space MSE is 0.106. The observed difference of 0.337 estimates the corresponding difference in excess MSE.
The ordering between these two models reverses at : the -model has a lower velocity-space MSE than the -model (0.337 versus 0.462). Thus, the comparison depends on the learned prediction errors as well as the conversion weights. The table excludes timesteps where denominator clamping changes the conversion formulas.
Stochastic sampling. Proposition 6 concerns the drift field, not the integrator. Consider the reverse samplers that share the marginals of the probability-flow ODE: for any schedule ,
where estimates the score and recovers the ODE. Both drift terms are affine conversions of the same network output. For an -prediction model, and , so a native error gives
The low- amplification therefore stays in the drift for every choice of . Injected noise can change how a drift error affects the sample distribution, but it does not remove a systematic error in the field. We test this on the 600-epoch checkpoints with Euler–Maruyama sampling of (19), , and 50 steps, against the Heun ODE with the same step count. Heun evaluates the network twice per step, so the ODE uses 99 network evaluations against 50 for the SDE. Both samplers use a matched 4K-sample FID protocol, which is upward-biased relative to the 50K numbers in Table 4 but comparable across samplers. Table 13 shows that -prediction does not recover under stochastic sampling, while -prediction changes only modestly. Scaling the schedule to and keeps the -prediction FID between 328 and 372. The FID gap thus originates in a drift-field error that the ODE and this SDE share. This conclusion is limited to samplers that use the same conversion. Samplers that avoid it, such as DDPM ancestral sampling in native -space, also avoid the amplification (§ 5).
| FID (4K samples) | ODE (Heun) | SDE |
| -prediction | 14.5 | 17.1 |
| -prediction | 333.4 | 330.2 |
Table 13. ODE versus SDE sampling on the 600-epoch ImageNet-256 checkpoints (, -loss). FID over 4K samples, 50 steps, CFG 2.9. The SDE is Euler–Maruyama on ((19)) with .
Relation to EDM preconditioning
EDM [10] writes the denoiser as
where with . Our interpolant takes this form with and .
EDM lies outside our static grid for two reasons. First, its network target changes continuously with the noise level. As , and targets the rescaled data . As , and targets a residual close to , an -like quantity. Second, the denoiser includes the full-rank skip path .
Our framework still applies to each ingredient. An error in the network output enters the -space estimate as , so plays the role of the amplification factor in Proposition 6. In the same coordinates, that proposition gives the factor for -prediction, which is unbounded at high noise, and for -prediction, which does not vanish at low noise. EDM’s factor satisfies and is bounded at both ends. EDM’s loss weight makes the effective weight on uniform over noise levels, which is a loss-space reweighting in the sense of Proposition 1. On the -space error, , our -loss weight plus a constant. The skip term bypasses any compressed representation, so the final denoiser is not a function of alone. Assumption (A3) then fails, and the floor of Theorem 4 does not apply.
We therefore expect EDM preconditioning to mitigate the bottleneck failure of - and -prediction, but we have neither a formal result nor a controlled experiment that establishes this. A direct ablation on a bottlenecked ViT is an important next step.
References
- [1]Ting Chen. On the importance of noise scheduling for diffusion models. arXiv preprint arXiv:2301.10972, 2023.arxiv.org/abs/2301.10972
- [2]Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472–11481, 2022.arxiv.org/abs/2204.00227
- [3]Patrick Esser, Sumith Kullal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 12606–12633, 2024.arxiv.org/abs/2403.03206
- [4]Anne Gagneux, Ségolène Martin, Rémi Gribonval, and Mathurin Massias. Training flow matching: The role of weighting and parameterization. In ICLR 2026 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2026. arXiv:2603.06454.arxiv.org/abs/2603.06454
- [5]Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-SNR weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7441–7451, 2023.arxiv.org/abs/2303.09556
- [6]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.arxiv.org/abs/2207.12598
- [7]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.arxiv.org/abs/2006.11239
- [8]Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pages 13213–13232, 2023.arxiv.org/abs/2301.11093
- [9]Qing Jin and Chaoyang Wang. Revisiting diffusion model predictions through dimensionality. In Advances in Neural Information Processing Systems, 2026.DOI
- [10]Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022.arxiv.org/abs/2206.00364
- [11]Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data augmentation. Advances in Neural Information Processing Systems, 36:65484–65516, 2023.arxiv.org/abs/2303.00848
- [12]Dibyanshu Kumar, Philipp Váth, and Magda Gregorová. Loss functions in diffusion models: A comparative study. In Machine Learning and Knowledge Discovery in Databases. Research Track (ECML PKDD 2025), Lecture Notes in Computer Science, pages 190–205. Springer, 2025. doi: 10.1007/978-3-032-05981-9_12.
- [13]Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 36115–36125, 2026.arxiv.org/abs/2511.13720
- [14]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02747
- [15]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, 2023.arxiv.org/abs/2209.03003
- [16]Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On biased resizing and surprising subtleties in GAN evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11410–11420, 2022.arxiv.org/abs/2104.11222
- [17]Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2202.00512
- [18]Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. pmlr, 2015.
- [19]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1907.05600
- [20]Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.


