betaFeedback

WorksComputer Science

Analyzing Learning-Dynamics Across Loss and Prediction Choices for Diffusion Models

Abstract

Denoising generative models predict clean data (xx), noise (εε), or velocity (vv), while training losses can be measured in any of these spaces. Although the resulting nine combinations are algebraically related, they can yield substantially different denoising accuracy and sample quality. We disentangle the effects of prediction and loss spaces. For a fixed prediction target, we show that changing the loss space is exactly equivalent to timestep reweighting and provide a heuristic analysis of how this reweighting affects generation quality. We then show that a rank-mm linear bottleneck compressing a DD-dimensional noisy input imposes an irreducible squared-error floor of D−mD-m on noise and velocity prediction, whereas clean-data prediction can exploit low-dimensional data structure. Combining this bound with prediction-dependent error conversion yields distinct necessary conditions on bottleneck capacity. Experiments with U-Net and JiT across CIFAR-10, Imagenette, and ImageNet-256 support these theoretical predictions. Under compression, prediction choice strongly affects both denoising accuracy and sample quality, with noise-prediction errors closely matching the theoretical lower bound. Our results provide a quantitative understanding of how architectural bottlenecks interact with prediction and loss choices in denoising generative models.

Introduction

Diffusion and flow-based generative models train a neural network to reverse a corruption process: given a noisy observation ztz_t at noise level tt, the network predicts a target quantity (clean data xx, noise ϵ\epsilon, or velocity vv) from which the original signal can be recovered [18, 7, 19, 14, 15]. Since (x,ϵ,v)(x,\epsilon,v) are related by affine transforms conditioned on (zt,t)(z_t,t), these parameterizations are algebraically equivalent in the oracle regime, and prior work often treats them as interchangeable up to loss reweighting [17, 10, 11].

Yet algebraic equivalence does not guarantee comparable performance in finite-capacity networks. Li and He [ [13] introduce JiT, a pixel-space Transformer with low-rank patch embeddings, and report substantially better FID with xx-prediction than with ϵ\epsilon- or vv-prediction. They motivate xx-prediction through the low-dimensional structure of natural images, suggesting that architectural compression may affect prediction targets differently. However, how such compression affects the relative difficulty of different prediction targets remains insufficiently understood.

We revisit this question by separating two design choices: the prediction space (what the network predicts) and the loss space (where errors are measured), considering all nine combinations over {x,ϵ,v}\{x,\epsilon,v\}. Although prior work has examined prediction parameterizations and loss weightings jointly [13, 4], the effects of architectural information loss and prediction-dependent error conversion remain less understood. We analyze loss-space choices through timestep reweighting and prediction-space choices through bottleneck-induced error bounds and conversion amplification. Together, these analyses yield testable predictions about how architectural compression affects denoising accuracy across prediction targets.

Contributions. We summarize our contributions as follows.

  1. A joint 3×33 \times3 analysis of prediction and loss spaces. We show that changing the loss space for a fixed prediction target is exactly equivalent to deterministic timestep reweighting (Proposition 1), collecting known affine identities into a unified formulation. We further provide a heuristic analysis of how this reweighting affects generation quality, suggesting that ϵ\epsilon- and vv-loss can outperform xx-loss under high-tt sensitivity assumptions (Proposition 2).

  1. Irreducible prediction errors under architectural bottlenecks. For a rank-mm linear bottleneck compressing a DD-dimensional noisy input, we establish an error floor of D−mD-m for ϵ\epsilon- and vv-prediction, whereas xx-prediction can exploit the low-dimensional structure of the data (Theorem 4 and Corollary 5). This result quantifies the information-loss mechanism underlying the prediction-space asymmetry observed in JiT [13].

  1. Prediction-dependent error amplification. We characterize how converting different prediction targets to clean-data estimates scales their errors, with factors (1−t)2/t2(1-t)^2/t^2 for ϵ\epsilon-prediction and (1−t)(1-t) for vv-prediction (Proposition 6). Combining these identities with the bottleneck error floor yields prediction-dependent necessary conditions on bottleneck capacity (Corollary 7).

  1. Empirical validation across architectures and scales. We evaluate prediction and loss choices using synthetic data, U-Net, and JiT across CIFAR-10, Imagenette, and ImageNet-256. Our experiments support the predicted effects of loss-space reweighting and quantitatively validate the bottleneck error bounds. In particular, the ϵ\epsilon-prediction error closely matches the theoretical floor under strong compression, while bottleneck-width sweeps reveal distinct capacity requirements across prediction targets.

Related work

Prediction parameterizations and loss weighting. Common diffusion parameterizations predict noise [7], the score [20], or a velocity-related quantity. Salimans and Ho [ [17] study prediction parameterizations and loss weightings jointly. EDM [10] introduces preconditioning that normalizes the scales of network inputs and regression targets across noise levels. Kingma and Gao [ [11] express common diffusion objectives as weighted losses and connect monotonic weightings to ELBO maximization with noisy data augmentation. Explicit weighting strategies include Min-SNR [5] and P2 [2], while Chen [ [1] study noise-schedule design. For our linear interpolation, Proposition 1 collects the established loss-conversion identities into a common statement for a fixed prediction space.

Joint analyses of prediction and loss. JiT [13], a pixel-space Transformer using xx-prediction with vv-loss, reports substantial performance differences across prediction targets, motivating a systematic analysis of prediction and loss choices. Their ablation evaluates all nine combinations of prediction and loss spaces. Kumar et al. [ [12] compare native-target squared-error objectives and their ELBO-derived counterparts across data, noise, velocity, and score parameterizations, and analyze the rescalings relating these objectives. Gagneux et al. [ [4] systematically study the interplay between output parameterization and loss weighting, comparing denoising PSNR across noise levels and FID under different architectures and data regimes. Their inverse-variance analysis further provides a statistical motivation for emphasizing low-noise timesteps.

Dimensionality, bottlenecks, and error conversion. Jin and Wang [ [9] analyze the learning dynamics of linear diffusion predictors by decomposing their behavior into components parallel and orthogonal to the data subspace. They characterize how data dimensionality affects the optimal prediction target and propose kk-Diff, which learns a data-dependent prediction parameter. We study predictions formed from a rank-mm linear compression of a DD-dimensional noisy input. Under the stated assumptions, we derive an expected squared ℓ2\ell_2 error floor of D−mD-m for noise and velocity prediction (Theorem 4 and Corollary 5). Prior work has discussed numerical instability and error amplification associated with converting noise predictions to clean-data estimates [17, 10, 8]. Under linear interpolation, we combine the corresponding error-scaling identities with our bottleneck bounds to obtain prediction-dependent necessary conditions on bottleneck rank (Proposition 6 and Corollary 7).

Setting and notation

We consider training data x∼pdatax \sim p_{\mathrm{data}} in RD\mathbb{R}^{D} and noise ϵ∼N(0,ID)\epsilon\sim\mathcal{N}(0,I_{D}). For t∈(0,1)t \in(0,1), define the noisy observation

zt=tx+(1−t)ϵ,(1)z_{t}=tx+(1-t)\epsilon, \tag*{(1)}

and the velocity

v=x−ϵ.(2)v=x-\epsilon. \tag*{(2)}

A model is parameterized by a neural network whose direct output (prediction space) is one of

u∈{x,ϵ,v},uθ(zt,t)=net⁡θ(zt,t),u \in\{x,\epsilon,v\}, \qquad u_{\theta}(z_{t},t)=\operatorname{net}_{\theta}(z_{t},t),

where net⁡θ(zt,t)\operatorname{net}_{\theta}(z_{t},t) is modeled as a trainable neural network parameterized by θ\theta. Given (zt,t)(z_{t},t) and one predicted quantity, the other two are obtained by solving the linear system induced by (1) and (2). This yields deterministic affine transforms among {xθ,ϵθ,vθ}\{x_{\theta},\epsilon_{\theta},v_{\theta}\}.

Training uses a squared loss in a chosen loss space s∈{x,ϵ,v}s \in\{x,\epsilon,v\}:

Ls∣u(θ)=E[∥sθ(zt,t)−s∥2],(3)\mathcal{L}_{s\mid u}(\theta)=\mathbb{E}\left[\left\|s_{\theta}(z_{t},t)-s\right\|^{2}\right], \tag*{(3)}

where sθs_{\theta} is computed from uθu_{\theta} by the appropriate transform. Note that vv-prediction determines the score: by Tweedie’s formula, ∇ztlog⁡p(zt)=(t v⋆(zt,t)−zt)/(1−t)\nabla_{z_{t}}\log p(z_{t})=(t\,v^{\star}(z_{t},t)-z_{t})/(1-t) [20], an affine map whose time-dependent scaling corresponds to a specific loss weighting within our framework.

Goal. In the oracle regime, all (s,u)(s,u) are equivalent up to deterministic transformations. The scientific question is to formalize how learning dynamics differ for finite capacity and gradient-based optimization, and to design experiments that disentangle (i) timestep reweighting effects from (ii) geometric and capacity effects.

Learning dynamics of diffusion

We analyze how prediction and loss choices affect learning through two distinct mechanisms. We first establish that changing the loss space induces deterministic timestep reweighting and examine how this weighting affects generation quality. We then turn to the prediction-space axis, studying how architectural bottlenecks introduce irreducible errors and how prediction-dependent conversion amplifies these errors. Proofs for the theoretical results are collected in Appendix § 8.

Proposition 1 (Equivalence of loss spaces as timestep reweighting). Assume the linear interpolation and velocity definition

zt=tx+(1−t)ϵ,v=x−ϵ,t∈(0,1),z_{t}=tx+(1-t)\epsilon,\qquad v=x-\epsilon,\qquad t\in(0,1),

with (x,ϵ)(x,\epsilon) sampled from the data and noise distributions. Fix a prediction space u∈{x,ϵ,v}u\in\{x,\epsilon,v\}, meaning the network directly outputs

uθ(zt,t)=net⁡θ(zt,t).u_{\theta}(z_{t},t)=\operatorname{net}_{\theta}(z_{t},t).

For any loss space s∈{x,ϵ,v}s\in\{x,\epsilon,v\}, define sθ(zt,t)s_{\theta}(z_{t},t) as the quantity obtained by solving the linear system induced by (zt,t)(z_{t},t) together with the constraints defining the pair (x,ϵ,v)(x,\epsilon,v), with uu replaced by uθu_{\theta}. Then for every pair (s,u)∈{x,ϵ,v}2(s,u)\in\{x,\epsilon,v\}^{2} there exist:

  • a scalar slope as∣u(t)≠0a_{s\mid u}(t)\ne0 that depends only on tt, and

  • *an intercept term bs∣u(zt,t)b_{s\mid u}(z_{t},t) that depends on (zt,t)(z_{t},t) but is independent of uu,

such that the true target and the model prediction satisfy the aligned affine relations

s=as∣u(t)u+bs∣u(zt,t),s=a_{s\mid u}(t)u+b_{s\mid u}(z_{t},t),

and

sθ(zt,t)=as∣u(t)uθ(zt,t)+bs∣u(zt,t).s_{\theta}(z_{t},t)=a_{s\mid u}(t)u_{\theta}(z_{t},t)+b_{s\mid u}(z_{t},t).

Consequently, the residuals obey the pointwise scaling identity

sθ(zt,t)−s=as∣u(t)(uθ(zt,t)−u).(4)s_{\theta}(z_{t},t)-s=a_{s\mid u}(t)\left(u_{\theta}(z_{t},t)-u\right). \tag*{(4)}

and hence the squared loss in ss-space is exactly a weighted squared loss in the direct output uu-space:

Ls∣u(θ):=E[∥sθ(zt,t)−s∥2]=E[ws∣u(t)∥uθ(zt,t)−u∥2],(5)\mathcal{L}_{s\mid u}(\theta) := \mathbb{E}\left[\lVert s_\theta(z_t,t)-s\rVert^2\right] = \mathbb{E}\left[w_{s\mid u}(t)\lVert u_\theta(z_t,t)-u\rVert^2\right], \tag*{(5)}

where ws∣u(t):=as∣u(t)2>0w_{s\mid u}(t) := a_{s\mid u}(t)^2 > 0. Since ws∣uw_{s\mid u} does not depend on θ\theta, the Bayes-optimal predictor u⋆(zt,t)=E[u∣zt,t]u^\star(z_t,t)=\mathbb{E}[u\mid z_t,t] is the same for all loss spaces ss. For the linear schedule, the weights are listed in Table 1; the individual identities appear in prior work [17, 11, 10], and we collect them here for reference.

Weight ws∣u(t)w_{s\mid u}(t)Loss s=xs=xLoss s=ϵs=\epsilonLoss s=vs=v
Pred u=xu=x11(t1−t)2\left(\frac{t}{1-t}\right)^2(11−t)2\left(\frac{1}{1-t}\right)^2
Pred u=ϵu=\epsilon(1−tt)2\left(\frac{1-t}{t}\right)^211(1t)2\left(\frac{1}{t}\right)^2
Pred u=vu=v(1−t)2(1-t)^2t2t^211

Table 1. The implicit timestep weights ws∣u(t)w_{s\mid u}(t) induced by different loss-space choices, derived from the linear-schedule relations between (x,ϵ,v)(x,\epsilon,v) collected from prior work [17, 11, 10]. Diagonal entries (u=su=s) are unity. Off-diagonal entries involving (1−t)−1(1-t)^{-1} or t−1t^{-1} diverge at endpoints and require clamping in practice.

Proposition 1 states that changing loss space ss only reweights timesteps; the Bayes-optimal predictor u⋆u^\star is unchanged. Empirical differences must arise from finite capacity and optimization dynamics, not from a different oracle. In expectation, applying a weight w(t)w(t) in the loss is equivalent to sampling tt from a tilted distribution ∝w(t)p(t)\propto w(t)p(t). In practice, off-diagonal weights can diverge (e.g., wv∣x(t)=(1−t)−2w_{v\mid x}(t)=(1-t)^{-2}); following [13], we clamp (1−t)≥0.05(1-t)\ge0.05. By contrast, changing prediction space uu changes u⋆u^\star itself, so is not covered by this equivalence.

The weights connect to the SNR framework [11]: for xx-prediction, wϵ∣x(t)=SNR⁡(t)=t2/(1−t)2w_{\epsilon\mid x}(t)=\operatorname{SNR}(t)=t^2/(1-t)^2, so ϵ\epsilon-loss concentrates optimization on high-SNR timesteps while xx-loss distributes it uniformly. This subsumes EDM’s weighting [10] and explains why ϵ/v\epsilon/v-loss emphasize the generation-critical high-tt regime.

Loss-space ordering: Why some weightings are better

Proposition 1 establishes that changing the loss space ss (with fixed prediction space uu) is equivalent to changing the timestep weighting ws∣u(t)w_{s\mid u}(t). At the population level, all weightings yield the same Bayes-optimal predictor u⋆(zt,t)u^\star(z_t,t). However, with finite model capacity, the weighting determines which timesteps receive more optimization pressure. We now provide a heuristic analysis of why this matters for generation quality, then validate the key prediction empirically.

Setup. Consider an ODE-based sampler that integrates from t=0t=0 (noise) to t=1t=1 (data) using the learned velocity field v^(zt,t)\hat{v}(z_t,t). Let δv(t):=v^(zt,t)−v⋆(zt,t)\delta v(t):=\hat{v}(z_t,t)-v^\star(z_t,t) denote the velocity error at time tt, and MSE⁡v(t):=E∥δv(t)∥2\operatorname{MSE}_v(t):=\mathbb{E}\lVert\delta v(t)\rVert^2. By a first-order perturbation of the ODE dz/dt=v(zt,t)dz/dt=v(z_t,t), the final-sample error satisfies

δz1≈∫01δv(t) dt.(6)\delta z_1 \approx\int_0^1 \delta v(t)\,\mathrm{d}t. \tag*{(6)}

This linearization is accurate when velocity errors are small relative to the signal. For the heuristic surrogate below, we restrict the integration to an interior timestep interval T=[0.05,0.95]\mathcal{T}=[0.05,0.95], neglecting contributions outside this interval. By Proposition 1 with u=vu=v, a loss in space ss weights the squared velocity error by the effective weight weff(t):=ws∣v(t)w_{\mathrm{eff}}(t):=w_{s\mid v}(t), which is (1−t)2(1-t)^2, t2t^2, and 11 for xx-, ϵ\epsilon-, and vv-loss (see Table 1). These effective weights do not depend on the prediction space uu.

Proposition 2 (Loss-Space Ordering (heuristic)). Consider the following modeling assumptions:

(i) First-order approximation: The sample error is dominated by (6).

(ii) Cross-time correlations: By (i), E∥δz1∥2≈∫01∫01E[δv(t)⊤δv(t′)] dt dt′\mathbb{E}\lVert\delta z_1\rVert^2 \approx\int_0^1\int_0^1 \mathbb{E}[\delta v(t)^\top\delta v(t')]\,dt\,dt'. We summarize the cross-time correlations by a sensitivity κ(t)≥0\kappa(t) \ge0, so that E∥δz1∥2≈∫Tκ(t)MSE⁡v(t) dt\mathbb{E}\lVert\delta z_1\rVert^2 \approx\int_{\mathcal{T}}\kappa(t)\operatorname{MSE}_v(t)\,dt, and assume that κ\kappa does not depend on the training weighting.

(iii) Capacity allocation: With finite model capacity trained under effective weight weff(t)w_{\mathrm{eff}}(t) and time distribution p(t)p(t), the per-timestep velocity MSE scales as

MSE⁡v(t)∝1p(t)weff(t).(7)\operatorname{MSE}_v(t) \propto\frac{1}{p(t)w_{\mathrm{eff}}(t)}. \tag*{(7)}

Under these assumptions, the expected sample error satisfies

E[∥δz1∥2]∝Jκ(weff):=∫Tκ(t)p(t)weff(t) dt.(8)\mathbb{E}\left[\lVert\delta z_1\rVert^2\right] \propto J_\kappa(w_{\mathrm{eff}}) := \int_{\mathcal{T}}\frac{\kappa(t)}{p(t)w_{\mathrm{eff}}(t)}\,dt. \tag*{(8)}

If in addition

(iv) High-tt sensitivity: for natural images, κ(t)\kappa(t) is concentrated at high tt,

then

generation quality⁡(ϵ-loss)≈generation quality⁡(v-loss)≫generation quality⁡(x-loss).(9)\operatorname{generation\ quality}(\epsilon\text{-loss}) \approx\operatorname{generation\ quality}(v\text{-loss}) \gg\operatorname{generation\ quality}(x\text{-loss}). \tag*{(9)}

Derivation. By (i) and (ii), E∥δz1∥2≈∫01κ(t)MSE⁡v(t) dt\mathbb{E}\lVert\delta z_1\rVert^2 \approx\int_0^1 \kappa(t)\operatorname{MSE}_v(t)\,dt, and substituting (iii) gives (8). Under (iv), JκJ_\kappa is dominated by high tt. At high tt, the ϵ\epsilon-loss weight t2t^2 approaches the vv-loss weight 1, whereas the xx-loss weight (1−t)2(1-t)^2 approaches zero, thereby placing diminishing emphasis on velocity-space errors. The ϵ\epsilon- and vv-loss integrals are therefore close, and the xx-loss integral is much larger, which is (9).

Discussion. Assumption (i) approximates the accumulated sampling error by integrating local velocity errors, neglecting their propagation through the evolving trajectory. Assumptions (ii) and (iii) are heuristics: real networks share parameters across timesteps and the actual allocation depends on architecture, optimizer, and data. Assumption (iv) is a property of the data rather than of the model. We state the result as a proposition to make each premise falsifiable; the ordering is heuristic, not universal. Section B.6 evaluates JκJ_\kappa under the time distributions used in training. With constant κ\kappa, the xx-loss integral already exceeds the other two by one to two orders of magnitude, so (iv) is needed only to place ϵ\epsilon-loss next to vv-loss. Empirically, the ordering holds for xx- and vv-prediction on the U-Net and, more weakly, for xx-prediction on JiT-B/8 (§ 5). It fails on simple 2D distributions, where (iv) does not hold (Appendix § 9), and under strong compression on ImageNet-256, where ϵ\epsilon-loss gives the worst FID for xx-prediction (Table 4).

ConfigPred\Lossxx-MSE: xxxx-MSE: ϵ\epsilonxx-MSE: vvFID: xxFID: ϵ\epsilonFID: vv
dbn=128d_{\mathrm{bn}}=128 (r=0.17r=0.17)xx0.0230.0230.02323.936.820.1
dbn=128d_{\mathrm{bn}}=128 (r=0.17r=0.17)ϵ\epsilon14.6114.6114.61421.2419.0420.1
dbn=128d_{\mathrm{bn}}=128 (r=0.17r=0.17)vv0.2940.2940.294201.5202.3199.0
dbn=768d_{\mathrm{bn}}=768 (r=1.0r=1.0)xx——0.022——5.7
dbn=768d_{\mathrm{bn}}=768 (r=1.0r=1.0)ϵ\epsilon——0.043——330.6
dbn=768d_{\mathrm{bn}}=768 (r=1.0r=1.0)vv——0.026——140.6

Table 4. JiT-B/16 on class-conditional ImageNet 256×256256 \times256. Validation xx-MSE and FID (↓\downarrow, 50K samples). Top: dbn=128d_{\mathrm{bn}}=128 (r=0.17r=0.17). Bottom: dbn=768d_{\mathrm{bn}}=768 (r=1.0r=1.0). Under compression, prediction type dominates both metrics (ϵ\epsilon-space MSE 0.834 ≈(D−m)/D=0.833\approx(D-m)/D=0.833; see Section B.8). Without compression and at full budget, MSE converges across prediction types (0.022–0.043) but FID does not, so aggregate denoising MSE does not determine FID.

Remark 1 (Connection to existing strategies). Our framework subsumes existing weighting strategies: EDM [10] weights the xx-space error by our ϵ\epsilon-loss weight plus a constant (Appendix § 10), min-SNR [5] interpolates between uniform and ϵ\epsilon-loss, and P2 [2] modulates by perceptual priority. All of these upweight high-tt relative to xx-loss, consistent with (9). DDPM [7] uses (ϵ,ϵ)(\epsilon,\epsilon), SD3 [3] uses (v,v)(v,v), JiT [13] uses (x,v)(x,v).

Bottleneck lower bounds from information loss: why xx is special under manifold structure

We now turn from loss-space reweighting to the effect of prediction-space choices under architectural bottlenecks. Clean data xx lies on or near a low-dimensional manifold of natural images, whereas the Gaussian noise ϵ\epsilon and the velocity v=x−ϵv=x-\epsilon fill the ambient RD\mathbb{R}^D. We will show that combining this asymmetry with an architectural information bottleneck yields a sharp consequence: ϵ\epsilon- and vv-prediction incur an irreducible error floor, while xx-prediction remains feasible.

Definition 3 (Bottleneck model). We use the following assumptions.

(A1) Subspace data: x=Px^x=P\hat{x}, where P∈RD×dP\in\mathbb{R}^{D\times d} satisfies P⊤P=IdP^\top P=I_d with d≪Dd\ll D, and x^∼px^\hat{x}\sim p_{\hat{x}} on Rd\mathbb{R}^d.

(A2) Isotropic noise: ϵ∼N(0,ID)\epsilon\sim\mathcal{N}(0,I_D), independent of xx.

(A3) Linear bottleneck: the model observes only ρ=Ezt∈Rm\rho= Ez_t \in\mathbb{R}^m, where E∈Rm×DE \in\mathbb{R}^{m \times D} and rank⁡(E)=m<D\operatorname{rank}(E) = m < D.

(A1) is the linear idealization of the manifold hypothesis. (A3) matches JiT’s patch embedding, which applies two linear convolutions with no nonlinearity in between.

For a Vision Transformer, DD corresponds to the flattened patch dimension dpatch=p2Cd_{\mathrm{patch}} = p^2C for p×pp \times p patches with CC channels, and mm corresponds to the patch-embedding dimension dbnd_{\mathrm{bn}} to which the architecture compresses each patch before processing. We use dpatchd_{\mathrm{patch}} and dbnd_{\mathrm{bn}} throughout when referring to these quantities for concrete architectures, and write r:=dbn/dpatchr := d_{\mathrm{bn}}/d_{\mathrm{patch}} for the bottleneck ratio, where r<1r < 1 indicates patch-level compression and r≥1r \ge1 no compression.

Theorem 4 (Irreducible error from null-space components). Assume (A3) and let Πnull\Pi_{\mathrm{null}} denote the orthogonal projector onto null⁡(E)\operatorname{null}(E). For any measurable estimator u^(ρ,t)\hat{u}(\rho,t) of a target u∈RDu \in\mathbb{R}^D,

E[∥u^(ρ,t)−u∥2]≥E[∥E[u∣ρ,t]−u∥2]≥E[∥Πnullu−E[Πnullu∣ρ,t]∥2].(10)\begin{aligned} \mathbb{E}\left[\left\|\hat{u}(\rho,t)-u\right\|^2\right] \ge\mathbb{E}\left[\left\|\mathbb{E}[u\mid\rho,t]-u\right\|^2\right] \\ &\ge\mathbb{E}\left[\left\|\Pi_{\mathrm{null}}u-\mathbb{E}[\Pi_{\mathrm{null}}u\mid\rho,t]\right\|^2\right]. \tag*{(10)} \end{aligned}

If in addition (A2) holds, then Πnullϵ\Pi_{\mathrm{null}}\epsilon is independent of (ρ,t)(\rho,t), and for u=ϵu=\epsilon

E[∥ϵ^(ρ,t)−ϵ∥2]≥E[∥Πnullϵ∥2]=(D−m).\mathbb{E}\left[\left\|\hat{\epsilon}(\rho,t)-\epsilon\right\|^2\right] \ge\mathbb{E}\left[\left\|\Pi_{\mathrm{null}}\epsilon\right\|^2\right] = (D-m).

Intuitively, the observation ρ\rho carries no information about the component of ϵ\epsilon in null⁡(E)\operatorname{null}(E), and that component has expected squared norm exactly D−mD-m. No estimator that sees only ρ\rho can recover it, so the null-space residual E[∥Πnullu−E[Πnullu∣ρ,t]∥2]\mathbb{E}\left[\|\Pi_{\mathrm{null}}u-\mathbb{E}[\Pi_{\mathrm{null}}u\mid\rho,t]\|^2\right] is an irreducible error floor from information loss through EE. Data on a low-dimensional subspace can pass through the projection intact, while full-dimensional noise cannot. The next corollary makes this precise.

Corollary 5 (Bottleneck separation between xx and ϵ/v\epsilon/v). Assume (A1) to (A3) and

(A4) Injectivity on the data subspace: rank⁡(EP)=d\operatorname{rank}(EP)=d, which requires m≥dm \ge d.

Then the bottleneck preserves all directions in the dd-dimensional data subspace, since rank⁡(EP)=d\operatorname{rank}(EP)=d. In contrast, every estimator of u∈{ϵ,v}u \in\{\epsilon,v\} based on (ρ,t)(\rho,t) satisfies

E[∥u^−u∥2]≥D−m.\mathbb{E}\left[\left\|\hat{u}-u\right\|^2\right] \ge D-m.

Thus, the bottleneck can preserve the low-dimensional data subspace while discarding noise components in its (D−m)(D-m)-dimensional null space. This creates an irreducible prediction-error floor for ϵ\epsilon- and vv-prediction that does not arise from the same null-space mechanism for xx-prediction.

When does the bottleneck apply? U-Nets bypass this via skip connections (effective rank DD), and EDM’s cskipc_{\mathrm{skip}} [10] provides a direct input-to-output path; Appendix § 10 places EDM in our framework. Architectures with patch-level compression and no skip path (JiT [13], any ViT with dbn<dpatchd_{\mathrm{bn}} < d_{\mathrm{patch}}) satisfy the assumption directly. Since EE is learned end-to-end, the gradient incentivizes it to preserve the data subspace for xx-prediction, confirmed by the near-identical MSE across loss types in Table 3.

ConfigPred\LossMSE (xx-space): xxMSE (xx-space): ϵ\epsilonMSE (xx-space): vvFID: xxFID: ϵ\epsilonFID: vv
dbn=128d_{\mathrm{bn}}=128 (ratio 0.67)xx0.0500.0480.04841.037.436.0
dbn=128d_{\mathrm{bn}}=128 (ratio 0.67)ϵ\epsilon5.8555.8485.852146.6147.6151.0
dbn=128d_{\mathrm{bn}}=128 (ratio 0.67)vv0.1520.1520.152113.2108.3113.0
dbn=256d_{\mathrm{bn}}=256 (ratio 1.33)xx0.0490.0480.04842.138.937.1
dbn=256d_{\mathrm{bn}}=256 (ratio 1.33)ϵ\epsilon0.0510.0510.051114.1100.1123.6
dbn=256d_{\mathrm{bn}}=256 (ratio 1.33)vv0.0490.0500.04958.960.358.1

Table 3. JiT-B (∼130\sim130M params) on Imagenette 64×6464 \times64 with class conditioning and CFG. Top: dbn=128d_{\mathrm{bn}}=128 (r=0.67r=0.67). Bottom: dbn=256d_{\mathrm{bn}}=256 (r=1.33r=1.33). Under compression, xx-prediction achieves threefold better FID than vv-prediction and 120-fold better MSE than ϵ\epsilon-prediction.

Remark 2 (Nonlinearity after linear compression). The lower bound of Theorem 4 applies to any predictor that observes the noisy input only through a rank-mm linear projection ρ=Ezt\rho= Ez_t, regardless of nonlinear processing applied after this projection. In particular, inserting a GELU between JiT’s two bottleneck convolutions preserves the initial linear compression and therefore does not violate the assumptions of the theorem. Our ablation in Table 10 shows similar prediction-space asymmetry with and without this additional nonlinearity. This demonstrates robustness to nonlinear processing after compression, rather than extending the theoretical guarantee to arbitrary nonlinear compression maps f:RD→Rmf:\mathbb{R}^D \to\mathbb{R}^m.

dbn=128d_{\mathrm{bn}}=128 (compression)dbn=256d_{\mathrm{bn}}=256 (no compr.)
xx-predϵ\epsilon-predxx-predϵ\epsilon-pred
Linear0.0485.85 (120×)0.0480.051 (6%)
GeLU0.0495.9 (120×)0.0490.059 (21%)

Table 10. Effect of post-bottleneck nonlinearity on Imagenette. A 2×2 factorial design compares JiT’s original linear patch embedding with a variant inserting a GeLU between its two convolutions, under compressed and uncompressed configurations. Cells show test MSE; numbers in parentheses give the ϵ\epsilon-prediction gap relative to xx-prediction.

Conversion amplification: Why ϵ\epsilon-prediction requires more capacity

Corollary 5 gives the same lower bound, D−mD-m, on the expected squared prediction errors for ϵ\epsilon and vv under a bottleneck. We show that conversion to xx-space yields different lower bounds and, consequently, different necessary conditions on bottleneck rank.

Proposition 6 (Conversion amplification). Let u^(zt,t)\hat{u}(z_t,t) be a model’s prediction in space u∈{x,ϵ,v}u \in\{x,\epsilon,v\}, and let x^u(zt,t)\hat{x}_u(z_t,t) denote the clean-data estimate obtained by algebraic conversion. If the model incurs pointwise prediction error δu:=u^−u\delta_u := \hat{u}-u against the true target, then the induced xx-space error satisfies:

∥x^x−x∥2=∥δx∥2,(11)\left\|\hat{x}_x-x\right\|^2=\left\|\delta_x\right\|^2, \tag*{(11)}
∥x^v−x∥2=(1−t)2∥δv∥2,(12)\left\|\hat{x}_v-x\right\|^2=(1-t)^2\left\|\delta_v\right\|^2, \tag*{(12)}
∥x^ϵ−x∥2=(1−t)2t2∥δϵ∥2.(13)\left\|\hat{x}_\epsilon-x\right\|^2=\frac{(1-t)^2}{t^2}\left\|\delta_\epsilon\right\|^2. \tag*{(13)}

The proof follows by direct algebraic conversion: x^v−x=(1−t)δv\hat{x}_v-x=(1-t)\delta_v and x^ϵ−x=−1−ttδϵ\hat{x}_\epsilon-x=-\frac{1-t}{t}\delta_\epsilon.

Under the bottleneck assumptions, both ϵ\epsilon- and vv-prediction incur an expected squared error of at least D−mD-m (Corollary 5). Combining this error floor with the conversion factors in Proposition 6 gives prediction-dependent lower bounds on the xx-space error. The following corollary states the resulting necessary conditions on bottleneck rank.

Corollary 7 (Prediction-type-dependent threshold). Under (A1)–(A4), for any ϵ\epsilon- or vv-predictor based on (ρ,t)(\rho,t), the converted xx-space MSE satisfies

MSEv-pred≥Et[(1−t)2]⋅(D−m)+,(14)\mathrm{MSE}_{v\text{-pred}}\geq\mathbb{E}_t\left[(1-t)^2\right]\cdot(D-m)_+, \tag*{(14)}
MSEϵ-pred≥Et[(1−t)2t2]⋅(D−m)+.(15)\mathrm{MSE}_{\epsilon\text{-pred}}\geq\mathbb{E}_t\left[\frac{(1-t)^2}{t^2}\right]\cdot(D-m)_+. \tag*{(15)}

Under logit-normal sampling with μ=−0.5\mu=-0.5 and σ=1\sigma=1, the expected conversion weights are approximately 0.40 for vv-prediction and 20 for ϵ\epsilon-prediction. For the same target xx-space MSE, the necessary upper bound on D−mD-m is therefore about 50 times smaller for ϵ\epsilon-prediction than for vv-prediction. These lower bounds become trivial at full rank (m=D)(m=D).

We empirically examine the predicted difference in bottleneck sensitivity in § 5. The bounds in Corollary 7 become trivial without compression and therefore do not explain the FID differences in that regime. Nevertheless, the per-timestep analysis in Section B.8 reveals error patterns consistent with conversion amplification, although a complete explanation of the FID gap remains open.

Experiments

We systematically evaluate all nine combinations of prediction target u∈{x,ϵ,v}u \in\{x,\epsilon,v\} and loss space s∈{x,ϵ,v}s \in\{x,\epsilon,v\} across two architectures: a standard U-Net with skip connections (∼\sim97M params, no information bottleneck) and the JiT-B architecture [13] (∼\sim130M params, with a configurable bottleneck in the patch embedding). Experiments span CIFAR-10 (32×3232\times32), Imagenette (64×6464\times64) and ImageNet-256, with synthetic data validating dimensional scaling.

Setup. The U-Net is trained unconditionally with Adam (β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, lr =2×10−4=2\times10^{-4}), logit-normal time sampling (μ=−0.5\mu=-0.5, σ=1.0\sigma=1.0), gradient clipping at norm 1.0, and EMA with decay 0.9999. JiT-B is trained class-conditionally with AdamW (β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95), logit-normal time sampling (μ=−0.8\mu=-0.8, σ=0.8\sigma=0.8), label dropout 0.1 for classifier-free guidance (CFG) [6], and EMA with decay 0.9999. The CIFAR-10 and Imagenette models are trained for 500 epochs. For ImageNet-256, we train the compressed 3×33\times3 configurations for 100 epochs and the uncompressed controls for 600 epochs, as detailed in Section B.8. JiT uses the full architecture from [13] including RMSNorm, RoPE, QK-norm, SwiGLU, and in-context class tokens. We report validation MSE in xx-space (averaged over 3 seeds for CIFAR-10 U-Net, 2 seeds for JiT CIFAR-10) and FID (computed with clean-fid [16]; U-Net: Heun, 100 steps, unconditional; JiT: Heun, 50 steps, CFG). JiT’s BottleneckPatchEmbed applies two linear convolutions without an intermediate nonlinearity, so the bottleneck is a rank-dbnd_{\mathrm{bn}} linear projection: precisely the setting of Theorem 4. We characterize each configuration by the bottleneck ratio r:=dbn/dpatchr:=d_{\mathrm{bn}}/d_{\mathrm{patch}}, where r<1r<1 indicates patch-level compression and r≥1r\geq1 no compression.

Loss-space effects without bottleneck: generation quality. Proposition 2 predicts that ϵ\epsilon-loss and vv-loss should outperform xx-loss in generation quality by placing greater optimization weight on high-tt timesteps. We focus on xx- and vv-prediction, which achieve substantially better FID than

ϵ\epsilon-prediction under the U-Net setting. The U-Net results in Table 2 support the predicted loss ordering. On CIFAR-10, switching from xx-loss to ϵ\epsilon-loss reduces FID from 11.79 to 2.20 for xx-prediction and from 12.51 to 1.12 for vv-prediction, with similar improvements on Imagenette. vv-loss also consistently outperforms xx-loss in these settings.

DatasetPred\Loss\multicolumn{3}{c}{MSE (×10−2\times10^{-2})}\multicolumn{3}{c}{FID}
xxϵ\epsilonvvxxϵ\epsilonvv
CIFAR-10xx5.3955.3605.36111.792.202.73
CIFAR-10ϵ\epsilon5.4015.3725.36716.4525.495.88
CIFAR-10vv5.3925.3565.35912.511.121.68
Imagenettexx4.8104.6644.63253.4012.7621.05
Imagenetteϵ\epsilonn/a14.7094.67085.7795.41100.98
Imagenettevv4.7634.6494.63043.117.0614.32

Table 2. U-Net results across datasets. Validation MSE in xx-space and FID (↓\downarrow).

Prediction-space effects under bottleneck. With an architectural information bottleneck, the prediction target becomes the dominant factor in denoising accuracy. Theorem 4 and Corollary 5 establish an irreducible error floor of D−mD - m for ϵ\epsilon- and vv-prediction under linear compression. Our synthetic experiments in Figure 1 directly validate this bound: the null-space MSE of ϵ\epsilon-prediction closely matches D−mD - m, while that of vv-prediction remains above the bound. On Imagenette, Table 3 shows that JiT-B with bottleneck ratio r=0.67r = 0.67 achieves an xx-space MSE of approximately 0.048 for xx-prediction, compared with 0.152 for vv-prediction and 5.85 for ϵ\epsilon-prediction. The FID results exhibit the same ordering, while changing the loss space within each prediction type has little effect on MSE. When the bottleneck is removed (r=1.33r = 1.33), the MSE gap narrows substantially: xx- and vv-prediction achieve MSE around 0.049, while ϵ\epsilon-prediction reaches 0.051. Together, these results show that loss-space reweighting has a relatively modest effect on denoising error, whereas representational information loss can make the prediction target decisive. The same separation is observed on ImageNet-256 under strong compression (r=0.17r = 0.17), where xx-prediction achieves an xx-space MSE of 0.023, compared with 0.294 for vv-prediction and 14.61 for ϵ\epsilon-prediction (Table 4). An additional ablation inserting a GELU after the initial linear compression exhibits a similar prediction-space asymmetry, indicating robustness to post-bottleneck nonlinear processing (Table 10).

Figure 1. Synthetic validation of the bottleneck lower bound (Theorem 4) with D=64D = 64 and d=2d = 2. Left: Prediction MSE in each target’s native space as a function of bottleneck dimension mm. The ϵ\epsilon- and vv-prediction errors are bounded below by D−mD - m (dashed), while xx-prediction maintains low error for m≥dm \ge d. Middle, Right: Decomposition of the prediction MSE into null-space and range-space components for ϵ\epsilon- and vv-prediction, respectively.

Bottleneck capacity threshold. Corollary 7 shows that, under compression (r<1r < 1), the irreducible error floor imposes a stronger constraint on ϵ\epsilon-prediction than on vv-prediction, suggesting that ϵ\epsilon-prediction requires greater bottleneck capacity to achieve comparable denoising accuracy. Since the bound becomes trivial at r≥1r \ge1, we empirically investigate recovery by varying the bottleneck width of BottleneckViT, a lightweight ViT that follows JiT’s patch-embedding design but adds a GELU after the initial linear projection (see Appendix § 9). On CIFAR-10, the gap between vv- and xx-prediction nearly disappears once compression is removed (Table 7). In contrast, on Imagenette, ϵ\epsilon-prediction remains less accurate even without compression and approaches xx-prediction only with a substantially wider embedding (r≈4r \approx4; Table 8). This recovery threshold is empirical and is not predicted by Corollary 7. Additional synthetic sweeps over DD, dd, and mm (Section B.1) and a patch-size sweep at fixed embedding width (Section B.2) further support the predicted dependence of prediction errors on D−mD-m.

dbnd_{\mathrm{bn}}Ratio rrxx-pred MSEvv-pred MSEGap
320.670.0540.158+191%
641.330.0540.054+0.2%
1282.670.0540.054+0.0%
2565.330.0540.054+0.0%

Table 7. Bottleneck dimension sweep on CIFAR-10 (BottleneckViT, dpatch=48d_{\mathrm{patch}} = 48).

dbnd_{\mathrm{bn}}Ratio rrxx-pred MSEϵ\epsilon-pred MSEGapFloor
320.170.04914.61297×297\times14.61
2561.330.0480.095+98%—
3842.000.0480.056+17%—
5122.670.0480.053+9%—
7684.000.0480.050+2.6%—

Table 8. ϵ\epsilon-prediction recovery with increasing bottleneck width on Imagenette (dpatch=192d_{\mathrm{patch}} = 192). Results use BottleneckViT with ϵ\epsilon-loss, 500 training epochs, and seed 42. Validation xx-space MSE is averaged over t∈{0.1,0.3,0.5,0.7,0.9}t \in\{0.1, 0.3, 0.5, 0.7, 0.9\}. The floor is 17.53(D−m)+/D17.53(D-m)^{+}/D, obtained from Corollary 7 on this evaluation grid.

Convergence behavior without compression. The bottleneck lower bound in Theorem 4 applies to rank-deficient embeddings (r<1r<1) and becomes trivial when r≥1r \ge1. On JiT-B/4 CIFAR-10 (r=2.67r=2.67), the initially large denoising MSE gap between ϵ\epsilon- and xx-prediction closes with longer training (Figure 4), suggesting that prediction-type differences without compression can partly reflect optimization difficulty rather than irreversible information loss. However, comparable denoising accuracy does not necessarily imply comparable generation quality. On ImageNet-256 without compression (r=1.0r=1.0), the models trained for 600 epochs achieve relatively similar xx-space MSE, yet their FID scores differ substantially (Table 4).

Figure 4. Loss curves on JiT-B/4 CIFAR-10 (r=2.67r=2.67, ϵ\epsilon-loss). (a) ϵ\epsilon-prediction starts roughly 80-fold worse than xx-prediction but reaches parity by epoch 200. (b) The MSE ratio decays from 80-fold to roughly unity, suggesting that the bottleneck effect at r≥1r \ge1 is a convergence penalty rather than an irreducible error.

To examine the role of error conversion, Figure 2 compares per-timestep errors in velocity space, which is used by ODE sampling, and in xx-space after conversion. The ϵ\epsilon-prediction model exhibits sharply increasing velocity-space MSE at low tt, consistent with the 1/t21/t^2 amplification of squared errors during ϵ\epsilon-to-velocity conversion. In contrast, xx- and vv-prediction maintain substantially lower errors in this region. These observations suggest that conversion amplification contributes to the observed FID gap, although it does not fully explain the differences, particularly the poor FID of vv-prediction. Section B.8 provides further analyses in a common velocity space and compares ODE and SDE sampling.

Per-timestep prediction errors on ImageNet-256 using JiT-B/16 without compression

Figure 2. Per-timestep prediction errors on ImageNet-256 using JiT-B/16 without compression (r=1.0r = 1.0, 600 epochs). Left: Velocity-space MSE against sampled targets, used to assess errors relevant to ODE sampling. Right: xx-space MSE after conversion. The velocity-space error of ϵ\epsilon-prediction increases sharply at low tt, consistent with the 1/t21/t^2 amplification factor, while xx- and vv-prediction remain bounded over the evaluated timesteps. Results below t=0.05t = 0.05 also reflect denominator clamping.

Conclusion

We have presented a joint 3×33 \times3 analysis of prediction and loss spaces in denoising generative models. For a fixed prediction target, changing the loss space is equivalent to timestep reweighting, which can affect generation quality under finite model capacity (Propositions 1 and 2). For prediction-space choices, we show that a rank-deficient linear bottleneck introduces an irreducible error floor of D−mD - m for ϵ\epsilon- and vv-prediction, whereas xx-prediction can exploit low-dimensional data structure (Theorem 4 and Corollary 5). Combining this bound with prediction-dependent error conversion yields necessary conditions on bottleneck capacity that are more restrictive for ϵ\epsilon-prediction than for vv-prediction (Proposition 6 and Corollary 7).

Experiments with U-Net and JiT across CIFAR-10, Imagenette, and ImageNet-256 support these analyses. Under compression, xx-prediction maintains low denoising error, while ϵ\epsilon- and vv-prediction are more sensitive to bottleneck capacity. In particular, the observed ϵ\epsilon-prediction error closely matches the theoretical lower bound under strong compression, and bottleneck-width sweeps reveal distinct recovery patterns across prediction targets. Together, our findings clarify how prediction and loss choices interact with architectural information constraints and provide guidance for selecting prediction targets in compressed generative models.

Limitations. Our theoretical bounds rely on a linear bottleneck and a low-dimensional subspace model of the data. Our GeLU ablation retains the initial linear bottleneck and therefore does not establish corresponding guarantees for architectures with general nonlinear compression maps. The loss-ordering analysis is heuristic and depends on assumptions about sampling sensitivity and the allocation of model capacity across timesteps.

Our experiments span different architectures, conditioning schemes, and datasets, so absolute FID values are not directly comparable across settings. Moreover, the compressed ImageNet-256 configurations were trained for a shorter budget than the uncompressed controls, and we did not evaluate their convergence under extended training. Without compression, our error bounds become trivial. Although prediction-dependent conversion can amplify errors during sampling, particularly for ϵ\epsilon-prediction, it does not fully explain the observed FID differences. Our stochastic sampling ablation is also limited to samplers that share the same velocity conversion. Extending the analysis to latent diffusion, text conditioning, and architectures with explicit skip connections or EDM-style preconditioning remains an important direction for future work.

Acknowledgments and Disclosure of Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2024-00337955); by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (RS-2024-00457882, Artificial Intelligence Research Hub Project; RS-2019-II191906, Artificial Intelligence Graduate School Program (POSTECH)); and by IITP grants funded by the Korea government (MSIT) under the Leading Generative AI Human Resources Development Program (IITP-2026-RS-2026-25546560) and the AI Star Fellowship Support Program (IITP-2026-RS-2026-25616313).

Proofs

Proof of Proposition 1

Proposition 1 (Equivalence of loss spaces as timestep reweighting). Assume the linear interpolation and velocity definition

zt=tx+(1−t)ϵ,v=x−ϵ,t∈(0,1),z_t = tx + (1-t)\epsilon,\qquad v = x-\epsilon,\qquad t \in(0,1),

with (x,ϵ)(x,\epsilon) sampled from the data and noise distributions. Fix a prediction space u∈{x,ϵ,v}u \in\{x,\epsilon,v\}, meaning the network directly outputs

uθ(zt,t)=net⁡θ(zt,t).u_\theta(z_t,t)=\operatorname{net}_\theta(z_t,t).

For any loss space s∈{x,ϵ,v}s \in\{x,\epsilon,v\}, define sθ(zt,t)s_\theta(z_t,t) as the quantity obtained by solving the linear system induced by (zt,t)(z_t,t) together with the constraints defining the pair (x,ϵ,v)(x,\epsilon,v), with uu replaced by uθu_\theta.

Then for every pair (s,u)∈{x,ϵ,v}2(s,u) \in\{x,\epsilon,v\}^2 there exist:

  • a scalar slope as∣u(t)≠0a_{s|u}(t) \ne0 that depends only on tt, and

  • an intercept term bs∣u(zt,t)b_{s|u}(z_t,t) that depends on (zt,t)(z_t,t) but is independent of uu,

such that the true target and the model prediction satisfy the aligned affine relations

s=as∣u(t)u+bs∣u(zt,t),s = a_{s|u}(t)u+b_{s|u}(z_t,t),

and

sθ(zt,t)=as∣u(t)uθ(zt,t)+bs∣u(zt,t).s_\theta(z_t,t)=a_{s|u}(t)u_\theta(z_t,t)+b_{s|u}(z_t,t).

Consequently, the residuals obey the pointwise scaling identity

sθ(zt,t)−s=as∣u(t)(uθ(zt,t)−u).(16)s_\theta(z_t,t)-s=a_{s|u}(t)\left(u_\theta(z_t,t)-u\right). \tag*{(16)}

and hence the squared loss in ss-space is exactly a weighted squared loss in the direct output uu-space:

Ls∣u(θ):=E[∥sθ(zt,t)−s∥2]=E[ws∣u(t)∥uθ(zt,t)−u∥2].(17)\mathcal{L}_{s|u}(\theta):=\mathbb{E}\left[\left\|s_\theta(z_t,t)-s\right\|^2\right] =\mathbb{E}\left[w_{s|u}(t)\left\|u_\theta(z_t,t)-u\right\|^2\right]. \tag*{(17)}

*where ws∣u(t):=as∣u(t)2>0w_{s|u}(t):=a_{s|u}(t)^2>0. Since ws∣uw_{s|u} does not depend on θ\theta, the Bayes-optimal predictor u∗(zt,t)=E[u∣zt,t]u^*(z_t,t)=\mathbb{E}[u\mid z_t,t] is the same for all loss spaces ss. For the linear schedule, the weights are listed in Table 1; the individual identities appear in prior work [17, 11, 10], and we collect them here for reference.

Proof. The result follows from the affine conversion formulas among (x,ϵ,v)(x,\epsilon,v) given (zt,t)(z_t,t). From zt=tx+(1−t)ϵz_t=tx+(1-t)\epsilon and v=x−ϵv=x-\epsilon, any two of (x,ϵ,v)(x,\epsilon,v) determine the third:

x=zt−(1−t)ϵt=zt+(1−t)v,x=\frac{z_t-(1-t)\epsilon}{t}=z_t+(1-t)v,
ϵ=zt−tx1−t=zt−tv,\epsilon=\frac{z_t-tx}{1-t}=z_t-tv,
v=x−ϵ=x−zt1−t=zt−ϵt.v=x-\epsilon=\frac{x-z_t}{1-t}=\frac{z_t-\epsilon}{t}.

For any prediction space uu and loss space ss, substituting uθu_\theta for uu in these formulas yields sθs_\theta, and the same substitution of the true uu yields the true ss. Explicitly, the slopes as∣u(t)a_{s|u}(t) and intercepts bs∣u(zt,t)b_{s|u}(z_t,t) are:

Pred u=xu=x: From the linear interpolation, we have

ϵ=−t1−tx+zt1−t,v=x−zt1−t.\epsilon=-\frac{t}{1-t}x+\frac{z_t}{1-t},\qquad v=\frac{x-z_t}{1-t}.

Therefore,

aϵ∣x(t)=−t1−t,bϵ∣x(zt,t)=zt1−t,a_{\epsilon|x}(t)=-\frac{t}{1-t},\qquad b_{\epsilon|x}(z_t,t)=\frac{z_t}{1-t},
av∣x(t)=11−t,bv∣x(zt,t)=−zt1−t.a_{v|x}(t)=\frac{1}{1-t},\qquad b_{v|x}(z_t,t)=-\frac{z_t}{1-t}.

Pred u=ϵu=\epsilon: x=(zt−(1−t)ϵ)/tx=(z_t-(1-t)\epsilon)/t, so ax∣ϵ=−(1−t)/ta_{x|\epsilon}=-(1-t)/t. v=x−ϵ=(zt−ϵ)/tv=x-\epsilon=(z_t-\epsilon)/t, so av∣ϵ=−1/ta_{v|\epsilon}=-1/t.

Pred u=vu=v: x=zt+(1−t)vx=z_t+(1-t)v, so ax∣v=1−ta_{x|v}=1-t. ϵ=zt−tv\epsilon=z_t-tv, so aϵ∣v=−ta_{\epsilon|v}=-t.

In all cases, as∣u(t)a_{s|u}(t) depends only on tt, and the intercept bs∣u(zt,t)b_{s|u}(z_t,t) is independent of uθu_\theta (it involves only ztz_t and tt). Since sθ−s=as∣u(t)(uθ−u)s_\theta-s=a_{s|u}(t)(u_\theta-u) by linearity, squaring and taking expectations gives Ls∣u=E[as∣u(t)2∥uθ−u∥2]=E[ws∣u(t)∥uθ−u∥2]L_{s|u}=\mathbb{E}[a_{s|u}(t)^2\lVert u_\theta-u\rVert^2]=\mathbb{E}[w_{s|u}(t)\lVert u_\theta-u\rVert^2], where ws∣u(t)=as∣u(t)2w_{s|u}(t)=a_{s|u}(t)^2. The diagonal entries are wu∣u=1w_{u|u}=1, and the off-diagonal entries are:

wϵ∣x=(t1−t)2,wv∣x=(11−t)2,wx∣ϵ=(1−tt)2,w_{\epsilon|x}=\left(\frac{t}{1-t}\right)^2,\qquad w_{v|x}=\left(\frac{1}{1-t}\right)^2,\qquad w_{x|\epsilon}=\left(\frac{1-t}{t}\right)^2,
wv∣ϵ=(1t)2,wx∣v=(1−t)2,wϵ∣v=t2,w_{v|\epsilon}=\left(\frac{1}{t}\right)^2,\qquad w_{x|v}=(1-t)^2,\qquad w_{\epsilon|v}=t^2,

which are exactly the entries in Table 1. Since ws∣u(t)>0w_{s|u}(t)>0 depends only on tt, the population minimizer over all measurable predictors is u∗(zt,t)=E[u∣zt,t]u^*(z_t,t)=\mathbb{E}[u\mid z_t,t] for every loss space ss.

Proof of thm:nullspacelb

Theorem 4 (Irreducible error from null-space components). Assume (A3) and let Πnull\Pi_{\mathrm{null}} denote the orthogonal projector onto null⁡(E)\operatorname{null}(E). For any measurable estimator u^(ρ,t)\hat{u}(\rho,t) of a target u∈RDu\in\mathbb{R}^D,

E[∥u^(ρ,t)−u∥2]≥E[∥E[u∣ρ,t]−u∥2]≥E[∥Πnullu−E[Πnullu∣ρ,t]∥2].\mathbb{E}\left[\lVert\hat{u}(\rho,t)-u\rVert^2\right]\geq\mathbb{E}\left[\lVert\mathbb{E}[u\mid\rho,t]-u\rVert^2\right] \geq\mathbb{E}\left[\lVert\Pi_{\mathrm{null}}u-\mathbb{E}[\Pi_{\mathrm{null}}u\mid\rho,t]\rVert^2\right].

If in addition (A2) holds, then Πnullϵ\Pi_{\mathrm{null}}\epsilon is independent of (ρ,t)(\rho,t), and for u=ϵu=\epsilon

E[∥ϵ^(ρ,t)−ϵ∥2]≥E[∥Πnullϵ∥2]=(D−m).\mathbb{E}\left[\lVert\hat{\epsilon}(\rho,t)-\epsilon\rVert^2\right]\geq\mathbb{E}\left[\lVert\Pi_{\mathrm{null}}\epsilon\rVert^2\right]=(D-m).

Proof. Let u^(ρ,t)\hat{u}(\rho,t) be any estimator that depends only on the inputs (ρ,t)(\rho,t). We define the ideal Bayes estimator u∗=E[u∣ρ,t]u^*=\mathbb{E}[u\mid\rho,t]. We can decompose the error of u^\hat{u} as:

u^−u=(u^−u∗)+(u∗−u).\hat{u}-u=(\hat{u}-u^*)+(u^*-u).

Squaring the norm and taking expectations:

E∥u^−u∥2=E∥u^−u∗∥2+E∥u∗−u∥2+2E[(u^−u∗)T(u∗−u)].\mathbb{E}\lVert\hat{u}-u\rVert^2=\mathbb{E}\lVert\hat{u}-u^*\rVert^2+\mathbb{E}\lVert u^*-u\rVert^2+2\mathbb{E}\left[(\hat{u}-u^*)^{\mathsf{T}}(u^*-u)\right].

The cross-term vanishes by the tower property of conditional expectation: since (u^−u∗)(\hat{u}-u^*) is measurable with respect to (ρ,t)(\rho,t), it can be pulled out of the inner expectation:

E[(u^−u∗)T(u∗−u)]=Eρ,t[(u^−u∗)TE[u∗−u∣ρ,t]]=0.\mathbb{E}\left[(\hat{u}-u^*)^{\mathsf{T}}(u^*-u)\right]=\mathbb{E}_{\rho,t}\left[(\hat{u}-u^*)^{\mathsf{T}}\mathbb{E}[u^*-u\mid\rho,t]\right]=0.

Thus, we obtain the standard decomposition:

E∥u^−u∥2=E∥u^−u∗∥2+E∥u∗−u∥2≥E∥u∗−u∥2.\mathbb{E}\lVert\hat{u}-u\rVert^2=\mathbb{E}\lVert\hat{u}-u^*\rVert^2+\mathbb{E}\lVert u^*-u\rVert^2\geq\mathbb{E}\lVert u^*-u\rVert^2.

This proves the first inequality.

The term E∥u∗−u∥2\mathbb{E}\lVert u^*-u\rVert^2 represents the total variance of the unrecoverable residual. For any vector zz, and any orthogonal projector Π\Pi, we have ∥z∥2≥∥Πz∥2\lVert z\rVert^2\geq\lVert\Pi z\rVert^2. Applying this to the residual vector z=u−u∗z=u-u^* with the null-space projector Πnull\Pi_{\mathrm{null}}:

∥u−u∗∥2≥∥Πnull(u−u∗)∥2.(18)\lVert u-u^*\rVert^2\geq\lVert\Pi_{\mathrm{null}}(u-u^*)\rVert^2. \tag*{(18)}

Since Πnull\Pi_{\mathrm{null}} is a deterministic linear operator, it commutes with the conditional expectation by linearity of expectation:

Πnull(u−u⋆)=Πnullu−ΠnullE[u∣ρ,t]=Πnullu−E[Πnullu∣ρ,t].\Pi_{\mathrm{null}}(u-u^{\star})=\Pi_{\mathrm{null}}u-\Pi_{\mathrm{null}}\mathbb{E}[u\mid\rho,t]=\Pi_{\mathrm{null}}u-\mathbb{E}[\Pi_{\mathrm{null}}u\mid\rho,t].

Taking expectations proves the second inequality:

E∥u⋆−u∥2≥E∥Πnullu−E[Πnullu∣ρ,t]∥2.\mathbb{E}\lVert u^{\star}-u\rVert^{2}\geq\mathbb{E}\lVert\Pi_{\mathrm{null}}u-\mathbb{E}[\Pi_{\mathrm{null}}u\mid\rho,t]\rVert^{2}.

Consider the case u=ϵu=\epsilon with isotropic noise ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I). Write ϵ=Πcolϵ+Πnullϵ\epsilon=\Pi_{\mathrm{col}}\epsilon+\Pi_{\mathrm{null}}\epsilon, where Πcol=E†E\Pi_{\mathrm{col}}=E^{\dagger}E projects onto the column space of EE. Since ϵ\epsilon is isotropic Gaussian and Πcol,Πnull\Pi_{\mathrm{col}},\Pi_{\mathrm{null}} are orthogonal projectors onto complementary subspaces, the components Πcolϵ\Pi_{\mathrm{col}}\epsilon and Πnullϵ\Pi_{\mathrm{null}}\epsilon are independent Gaussian vectors (a standard property of orthogonal projections of Gaussian vectors). Since ρ=Ezt=E(tx+(1−t)ϵ)\rho=E z_t=E(tx+(1-t)\epsilon) depends on ϵ\epsilon only through Eϵ=EΠcolϵE\epsilon=E\Pi_{\mathrm{col}}\epsilon (as EΠnull=0E\Pi_{\mathrm{null}}=0 by definition), we have Πnullϵ⊥ ⁣ ⁣ ⁣⊥(ρ,t)\Pi_{\mathrm{null}}\epsilon\perp\!\!\!\perp(\rho,t). Therefore:

E[Πnullϵ∣ρ,t]=E[Πnullϵ]=0.\mathbb{E}[\Pi_{\mathrm{null}}\epsilon\mid\rho,t]=\mathbb{E}[\Pi_{\mathrm{null}}\epsilon]=0.

Substituting this into our bound:

E∥ϵ^−ϵ∥2≥E∥Πnullϵ−0∥2=E∥Πnullϵ∥2.\mathbb{E}\lVert\hat{\epsilon}-\epsilon\rVert^{2}\geq\mathbb{E}\lVert\Pi_{\mathrm{null}}\epsilon-0\rVert^{2}=\mathbb{E}\lVert\Pi_{\mathrm{null}}\epsilon\rVert^{2}.

For isotropic Gaussian noise, the expected squared norm of its projection is simply the dimension of the subspace:

E∥Πnullϵ∥2=tr⁡(Cov⁡(Πnullϵ))=tr⁡(ΠnullIΠnullT)=rank⁡(Πnull).\mathbb{E}\lVert\Pi_{\mathrm{null}}\epsilon\rVert^{2}=\operatorname{tr}(\operatorname{Cov}(\Pi_{\mathrm{null}}\epsilon))=\operatorname{tr}(\Pi_{\mathrm{null}}I\Pi_{\mathrm{null}}^{\mathsf{T}})=\operatorname{rank}(\Pi_{\mathrm{null}}).

Since E∈Rm×DE\in\mathbb{R}^{m\times D} has rank mm, the dimension of its null space is D−mD-m. Thus, the error is at least D−mD-m.

Proof of cor:separation

Proof of prop:conversionamp

Proof of cor:predthreshold

Proof of Corollary 5

Proof. Under (A1), x=Px^x=P\hat{x} with x^∈Rd\hat{x}\in\mathbb{R}^{d}. By (A4), EP∈Rm×dEP\in\mathbb{R}^{m\times d} has full column rank, so the bottleneck mapping is injective on the data subspace. Thus, unlike the noise components in the null space of EE, the data-subspace components are not irreversibly lost through the linear projection.

For ϵ\epsilon-prediction, Theorem 4 directly gives

E∥ϵ^−ϵ∥2≥D−m.\mathbb{E}\lVert\hat{\epsilon}-\epsilon\rVert^{2}\geq D-m.

For v=x−ϵv=x-\epsilon, we have Πnullv=Πnullx−Πnullϵ\Pi_{\mathrm{null}}v=\Pi_{\mathrm{null}}x-\Pi_{\mathrm{null}}\epsilon. By (A2), Πnullϵ\Pi_{\mathrm{null}}\epsilon is independent of (x,ρ,t)(x,\rho,t) and has zero mean. Therefore,

E[Πnullv∣ρ,t]=E[Πnullx∣ρ,t].\mathbb{E}[\Pi_{\mathrm{null}}v\mid\rho,t]=\mathbb{E}[\Pi_{\mathrm{null}}x\mid\rho,t].

Applying Theorem 4 and using this independence, we obtain

E∥v^−v∥2≥E∥Πnullv−E[Πnullv∣ρ,t]∥2=E∥Πnullx−E[Πnullx∣ρ,t]∥2+E∥Πnullϵ∥2≥D−m.\begin{aligned} \mathbb{E}\lVert\hat{v}-v\rVert^{2} &\geq\mathbb{E}\lVert\Pi_{\mathrm{null}}v-\mathbb{E}[\Pi_{\mathrm{null}}v\mid\rho,t]\rVert^{2}\\ &=\mathbb{E}\lVert\Pi_{\mathrm{null}}x-\mathbb{E}[\Pi_{\mathrm{null}}x\mid\rho,t]\rVert^{2}+\mathbb{E}\lVert\Pi_{\mathrm{null}}\epsilon\rVert^{2}\\ &\geq D-m. \end{aligned}

Proof of Proposition 6

The conversions from prediction space to xx-space are: x^x=x^\hat{x}_x=\hat{x}, so x^x−x=δx\hat{x}_x-x=\delta_x; x^v=zt+(1−t)v^\hat{x}_v=z_t+(1-t)\hat{v}, so x^v−x=(1−t)δv\hat{x}_v-x=(1-t)\delta_v; x^ϵ=(zt−(1−t)ϵ^)/t\hat{x}_{\epsilon}=(z_t-(1-t)\hat{\epsilon})/t, so x^ϵ−x=−(1−t)δϵ/t\hat{x}_{\epsilon}-x=-(1-t)\delta_{\epsilon}/t. Taking squared norms yields (11) through (13).

Proof of Corollary 7

Fix t∈(0,1)t\in(0,1). Under (A2) and (A3), Πnullϵ\Pi_{\mathrm{null}}\epsilon is independent of (ρ,t)(\rho,t). Hence, the arguments of Theorem 4 and Corollary 5 apply conditionally on tt, giving

E[∥ϵ^−ϵ∥2∣t]≥D−m,E[∥v^−v∥2∣t]≥D−m.\mathbb{E}[\lVert\hat{\epsilon}-\epsilon\rVert^{2}\mid t]\geq D-m,\qquad\mathbb{E}[\lVert\hat{v}-v\rVert^{2}\mid t]\geq D-m.
Pred\LossU-Net (no bottleneck) xxU-Net (no bottleneck) ϵ\epsilonU-Net (no bottleneck) vvJiT-B (r=0.67r=0.67, bottleneck) xxJiT-B (r=0.67r=0.67, bottleneck) ϵ\epsilonJiT-B (r=0.67r=0.67, bottleneck) vv
xx0.04810.04660.04630.0500.0480.048
ϵ\epsilon0.04860.04710.04675.95.85.9
vv0.04760.04650.04630.1520.1520.152

Table 5. Comparison of validation xx-space MSE across prediction and loss choices on Imagenette. For the U-Net without an information bottleneck, MSE varies primarily with the loss space. For JiT-B with a bottleneck (r=0.67r=0.67), MSE varies primarily with the prediction space. The full results, including FID, are reported in Table 2 and Table 3.

By Proposition 6, converting these prediction errors to xx-space multiplies them by (1−t)2/t2(1-t)^2/t^2 and (1−t)2(1-t)^2, respectively. Taking expectations over tt yields the stated lower bounds.

For the numerical values, let Z=logit⁡(t)∼N(μ,σ2)Z=\operatorname{logit}(t)\sim\mathcal{N}(\mu,\sigma^2) with μ=−0.5\mu=-0.5 and σ=1\sigma=1. Since (1−t)/t=e−Z(1-t)/t=e^{-Z},

Et[(1−t)2t2]=E[e−2Z]=e−2μ+2σ2=e3≈20.1,\mathbb{E}_t\left[\frac{(1-t)^2}{t^2}\right]=\mathbb{E}[e^{-2Z}]=e^{-2\mu+2\sigma^2}=e^3\approx20.1,

while numerical quadrature gives

Et[(1−t)2]≈0.403.\mathbb{E}_t[(1-t)^2]\approx0.403.

The ratio of these conversion weights is approximately 50.

On the discrete evaluation grid t∈{0.1,0.3,0.5,0.7,0.9}t\in\{0.1,0.3,0.5,0.7,0.9\} used in Table 8 and Table 9, the corresponding averages are 17.53 and 0.33, respectively. The tabulated lower bounds additionally divide by DD to match the reported per-dimension MSE.

x-space errorϵ-space error
ppD=3p2D=3p^2rr(D−m)/D(D-m)/Dxx-predϵ\epsilon-predvv-predϵ\epsilon-pred
2122.67—0.0540.0550.0540.060
4480.670.3330.0545.850.1580.353
81920.170.8330.05614.620.3210.842

Table 9. Patch-size sweep on CIFAR-10 (BottleneckViT, dbn=32d_{\mathrm{bn}}=32, vv-loss, 200 epochs, seed 42). Validation MSE per dimension, averaged over t∈{0.1,0.3,0.5,0.7,0.9}t \in\{0.1, 0.3, 0.5, 0.7, 0.9\}. The first three error columns are in xx-space; the last is the ϵ\epsilon-space error of ϵ\epsilon-prediction, to be compared with the floor (D−m)/D(D-m)/D.

Additional experiments and details

Bottleneck lower bound: synthetic validation

We validate the irreducible error lower bound (Theorem 4) using the synthetic manifold setup with a controlled linear bottleneck. Fix ambient dimension DD and intrinsic dimension d≪Dd\ll D. We sample a column-orthonormal matrix P∈RD×dP\in\mathbb{R}^{D\times d}, draw x^∼px^\hat{x}\sim p_{\hat{x}} from a mixture of Gaussians on Rd\mathbb{R}^d, and embed as x=Px^x=P\hat{x}. With ϵ∼N(0,ID)\epsilon\sim\mathcal{N}(0,I_D) and zt=tx+(1−t)ϵz_t=tx+(1-t)\epsilon, the model accesses only ρ=Ezt∈Rm\rho=Ez_t\in\mathbb{R}^m through a trainable embedding E∈Rm×DE\in\mathbb{R}^{m\times D}. We train all nine (s,u)(s,u) combinations and sweep mm from 1 to DD, decomposing the MSE into null-space (Πnull)(\Pi_{\mathrm{null}}) and range-space (Πrange)(\Pi_{\mathrm{range}}) components. We report prediction MSE in each target’s native space, together with its null-space and range-space components and convergence diagnostics. As predicted in Figure 1, ϵ\epsilon- and vv-prediction exhibit an irreducible error floor, while xx-prediction achieves low error for all m≥dm\ge d. For ϵ\epsilon, the null-space MSE matches D−mD-m precisely, confirming the theorem; for vv, it lies on or above this floor, as Corollary 5 requires.

Sweep over DD, dd, mm, and the embedding EE. We extend this setup along all three dimensions of Theorem 4 and Corollary 5: D∈{16,64,128}D\in\{16,64,128\}, d∈{2,8}d\in\{2,8\}, and mm from below dd up to DD, for all three prediction types. Each configuration is trained with vv-loss for 2,000 epochs, once with a learned EE and once with a random EE frozen at initialization, over two seeds, for 444 runs in total. We report squared error per vector with t∼U(0,1)t\sim\mathcal{U}(0,1), averaged over seeds; Table 6 lists every configuration. Three findings stand out. First, the floor of Theorem 4 is exact in all three variables. Across the 62 configurations with m<Dm<D, the null-space error of ϵ\epsilon-prediction lies between 0.997 and 1.012 times D−mD-m. vv-prediction lies above the same floor in every configuration, as Corollary 5 predicts. Its proof adds the posterior error of Πnullx\Pi_{\mathrm{null}}x to the ϵ\epsilon floor, and with a learned EE and m≥dm\ge d this excess stays between 12.8 and 18.7 for every DD and mm. Second, xx-prediction depends on whether m≥dm\ge d rather than on DD or mm. With a learned EE, its error stays between 11.4 and 12.5 whenever m≥dm\ge d and rises to between 15.9 and 17.1 when m<dm<d, so the transition sits at the injectivity condition (A4). Third, a random EE separates the two mechanisms. The ϵ\epsilon floor is the same for random and learned EE to within 1.5%, so it depends only on rank⁡(E)\operatorname{rank}(E) and is information-theoretic rather than an artifact of training the embedding. For xx-prediction, a random EE is much worse near m=dm=d, for example 24.9 against 11.4 at D=128D=128 and m=d=2m=d=2, and approaches the learned value as m→Dm \to D. A learned EE thus finds the subspace-aligned solution already at the minimal rank m=dm=d, consistent with the discussion after Corollary 5 that end-to-end training pushes EE to preserve the data subspace.

DDddmmϵ\epsilon null / (D−m)(D-m) learnedϵ\epsilon null / (D−m)(D-m) randomvv null / (D−m)(D-m) learnedxx error learnedxx error random
16211.0011.0011.9617.023.5
16221.0021.0032.0411.522.7
16241.0031.0032.1911.416.0
16281.0061.0052.8011.613.0
162120.9971.0124.6411.612.2
16216———11.711.6
16841.0001.0032.2216.018.2
16881.0071.0012.8012.216.1
168121.0061.0004.2111.612.8
16816———11.511.5
64211.0021.0031.2317.121.9
64221.0031.0031.2411.718.8
64241.0021.0031.2511.716.7
642161.0041.0041.3011.614.8
642321.0061.0051.4611.813.0
642481.0071.0061.9011.812.2
64264———11.911.3
64841.0031.0031.2515.918.8
64881.0041.0031.2712.217.5
648161.0041.0051.3711.715.6
648321.0071.0071.5811.613.5
648481.0071.0062.1311.612.3
64864———11.811.4
128211.0041.0041.1216.824.6
128221.0041.0041.1211.424.9
128241.0041.0041.1211.622.9
1282321.0061.0061.1511.614.0
1282641.0071.0081.2311.712.7
1282961.0121.0101.4311.612.0
1282128———11.511.6
128841.0041.0041.1315.919.6
128881.0051.0041.1312.518.1
1288161.0051.0051.1711.717.2
1288321.0051.0061.1911.715.0
1288641.0061.0081.2811.613.2
1288961.0091.0111.5611.712.4
1288128———11.611.7

Table 6. Synthetic sweep of Theorem 4 and Corollary 5 over ambient dimension DD, intrinsic dimension dd, and bottleneck rank mm, with a learned or a frozen random embedding EE (BottleneckMLP, vv-loss, 2 seeds). The ϵ\epsilon and vv columns give the null-space error of ϵ\epsilon- and vv-prediction divided by the floor D−mD-m; the xx columns give the total error of xx-prediction. Errors are squared norms per vector with t∼U(0,1)t \sim U(0,1). Rows with m=Dm=D have no floor.

Bottleneck threshold sweep

BottleneckViT and BottleneckMLP. BottleneckViT is a simplified Vision Transformer used as a computationally cheaper proxy for JiT-B in bottleneck-width sweeps.

Its patch embedding follows JiT’s BottleneckPatchEmbed, but inserts a GELU between the two convolutions. Importantly, the first convolution is a linear mapping from the noisy patch to a dbnd_{\mathrm{bn}}-dimensional representation. All subsequent computations, including the GELU, depend on the input only through this representation. Thus, under compression (dbn<dpatchd_{\mathrm{bn}} < d_{\mathrm{patch}}), the architecture satisfies assumption (A3) of Theorem 4, provided the first mapping has full row rank.

BottleneckViT uses fewer transformer blocks and a smaller hidden width than JiT-B, giving approximately 22M parameters versus JiT-B’s 130M. This enables systematic bottleneck-width sweeps at substantially lower computational cost.

BottleneckMLP is the analogous synthetic-data variant, using an MLP backbone with a linear input bottleneck.

Tables 7 and 8 show the full bottleneck-ratio sweep on BottleneckViT.

Patch size at fixed embedding width. Theorem 4 predicts that the floor depends on the patch through D−m=p2C−dbnD-m = p^{2}C-d_{\mathrm{bn}}. Increasing the patch size pp at fixed dbnd_{\mathrm{bn}} should therefore degrade ϵ\epsilon- and vv-prediction monotonically, while leaving xx-prediction unaffected as long as dbnd_{\mathrm{bn}} exceeds the intrinsic dimension of a patch. We test this with BottleneckViT on CIFAR-10 at p∈{2,4,8}p \in\{2, 4, 8\} and fixed dbn=32d_{\mathrm{bn}} = 32, so that D=3p2D = 3p^{2} ranges from 12 to 192 under the same dataset and training recipe: vv-loss, 200 epochs, seed 42. Table 9 reports validation MSE averaged over t∈{0.1,0.3,0.5,0.7,0.9}t \in\{0.1, 0.3, 0.5, 0.7, 0.9\}. At p=2p=2 the embedding is wider than the patch, so there is no bottleneck and the three prediction types match. As pp grows, xx-prediction stays flat at 0.054 to 0.056, while ϵ\epsilon- and vv-prediction degrade monotonically. The ϵ\epsilon-space error of ϵ\epsilon-prediction lands on the floor at both bottlenecked sizes. Averaged over tt it is 0.353 against 0.333 and 0.842 against 0.833, and for t≤0.5t \le0.5 it matches the floor to within 1%. Through Proposition 6, the floor then predicts the xx-space error of ϵ\epsilon-prediction, as in Corollary 7. The mean amplification (1−t)2/t2(1-t)^{2}/t^{2} over the evaluation grid is 17.53, so the prediction 17.53(D−m)/D17.53(D-m)/D is 5.84 at p=4p=4 and 14.61 at p=8p=8, against measured values of 5.85 and 14.62. vv-prediction lies above its corresponding bound 0.33(D−m)/D0.33(D-m)/D, which is 0.11 and 0.28, because its floor contains the xx component. The same values recur across setups. At r=0.67r=0.67 the ϵ\epsilon-prediction error of 5.85 equals that of JiT-B/8 on Imagenette (Table 3), and at r=0.17r=0.17 the value 14.62 matches JiT-B/16 on ImageNet-256 (Table 4). Since the error sits on the floor and the amplification depends only on the evaluation grid, the bottleneck ratio governs this error, not the resolution or the architecture.

Compression vs. post-bottleneck nonlinearity

We examine whether introducing nonlinear processing after the initial linear bottleneck affects the prediction-space asymmetry. Specifically, we insert a GeLU between JiT’s two patch-embedding convolutions. In both variants, the first convolution compresses each noisy patch into a dbnd_{\mathrm{bn}}-dimensional representation, and all subsequent computations depend only on this representation. Thus, the linear bottleneck assumption of Theorem 4 remains satisfied under compression.

As shown in Table 10, ϵ\epsilon-prediction exhibits similarly large errors under compression in both variants, while its error decreases substantially when compression is removed. These results demonstrate that the prediction-space asymmetry persists despite the additional nonlinearity after the initial linear compression.

Generated sample comparison

To complement the quantitative MSE and FID gaps reported in Table 3, we visualize the qualitative effect of prediction-space choice under a fixed bottleneck. Figure 3 shows samples generated by JiT-B/8 with dbn=128d_{\mathrm{bn}}=128 at ratio 0.67 on Imagenette using classifier-free guidance, with 100 Heun steps. All three rows use the same architecture and training recipe. xx- and ϵ\epsilon-prediction are trained with ϵ\epsilon-loss and vv-prediction with vv-loss; under this bottleneck the loss choice changes MSE by at most 3.3% within a prediction type (Table 3), so the rows differ mainly in the prediction target. The visual ordering tracks the MSE ordering precisely: xx-prediction yields recognizable class-conditional images, vv-prediction yields heavily degraded but still structured samples, and ϵ\epsilon-prediction collapses to pure noise. This corroborates Theorem 4: the irreducible null-space error of D−mD-m for ϵ\epsilon propagates through the ODE sampler to produce samples with no learnable structure, while xx-prediction, free of the null-space penalty under the manifold assumption, recovers usable samples even at the same bottleneck width.

Figure 3. Generated samples from JiT-B/8 (dbn=128d_{\mathrm{bn}}=128, ratio 0.67) on Imagenette with classifier-free guidance. xx-prediction produces recognizable images; vv-prediction is heavily degraded; ϵ\epsilon-prediction produces pure noise. All three use the same 130M-parameter architecture and 100 Heun steps; xx- and ϵ\epsilon-prediction use ϵ\epsilon-loss and vv-prediction uses vv-loss.

JiT-B/4 convergence behavior

Figure 4 reports per-epoch validation MSE for JiT-B/4 on CIFAR-10 at bottleneck ratio r=2.67r=2.67, trained for 500 epochs with ϵ\epsilon-loss, alongside an xx-prediction baseline trained under the same setup. Panel (a) plots the absolute loss curves; panel (b) shows their ratio. This single training run is the evidence for the convergence-penalty framing in § 5: it shows that, on this configuration, the prediction-type gap closes with continued training rather than persisting as an information-theoretic floor. We have not measured this systematically across rr, datasets, or architectures. The ImageNet-256 study (Section B.8) adds one more point at the regime boundary: at r=1.0r=1.0 and 600 epochs, the ϵ\epsilon-prediction denoising gap narrows to a factor of two without fully closing, and the FID gap from conversion amplification persists regardless.

Numerical evaluation of the loss-ordering integral

We evaluate the integral Jκ(weff)J_{\kappa}(w_{\mathrm{eff}}) of Proposition 2 for the three effective weights under the logit-normal time distributions used in training. A logit-normal density vanishes at both endpoints, so JκJ_{\kappa} diverges on (0,1)(0,1). We therefore integrate over t∈[0.05,0.95]t \in[0.05, 0.95]. This cutoff is a modeling choice: it equals the conversion clamp tϵ=0.05t_{\epsilon}=0.05 of the JiT recipe, but the training time distribution itself covers (0,1)(0,1). Table 11 normalizes each integral by its vv-loss value. With constant κ\kappa, which ignores assumption (iv), the xx-loss integral is 126 to 256 times the vv-loss integral, while the ϵ\epsilon-loss integral is 1.5 to 11.4 times. The excess of ϵ\epsilon-loss comes from low tt, where its weight t2t^2 vanishes. Restricting the sensitivity to high tt, κ(t)=1[t>1/2]\kappa(t)=1[t>1/2], brings the ϵ\epsilon-loss integral within a factor of 1.43 of the vv-loss integral and leaves the xx-loss integral more than two orders of magnitude larger, which is the ordering (9). For integration cutoffs between 0.02 and 0.1, the xx-loss integral remains the largest, by a factor of at least 28.

Time distributionSensitivity κ(t)\kappa(t)vv-lossϵ\epsilon-lossxx-loss
logit-normal(−0.5-0.5, 1.0), U-Net1111.4125.9
1[t>1/2]1[t>1/2]11.43148.5
logit-normal(−0.8-0.8, 0.8), JiT111.49255.9
1[t>1/2]1[t>1/2]11.17257.0

Table 11. The loss-ordering integral Jκ(weff)J_{\kappa}(w_{\mathrm{eff}}) of Proposition 2 over t∈[0.05,0.95]t \in[0.05,0.95], normalized by its vv-loss value. Effective weights are 1, t2t^2, and (1−t)2(1-t)^2 for vv-, ϵ\epsilon-, and xx-loss.

Loss ordering on simple distributions

The ordering (9) relies on assumption (iv) of Proposition 2, that generation quality depends disproportionately on high-tt accuracy. This holds for natural images, where fine-grained details such as textures and edges are resolved near t=1t=1 and dominate FID. However, for simple distributions such as 2D synthetic datasets, the denoising task is approximately equally difficult at all timesteps. In this regime, our synthetic experiments show that the loss ordering partially reverses: diagonal losses (u=su=s, uniform weight) become near-optimal, and ϵ\epsilon-loss can underperform xx-loss. This is consistent with the analysis: when denoising difficulty is uniform across tt, concentrating capacity on any particular timestep range cannot help.

ImageNet-256 scale validation

To confirm that both regimes persist at the resolution and dataset scale of modern generative models, we train the full JiT-B/16 architecture on class-conditional ImageNet-1k at 256×256256 \times256 (1.28M images, 1000 classes), where each patch has dpatch=162⋅3=768d_{\mathrm{patch}} = 16^2 \cdot3 = 768. We run the complete 3×33 \times3 design at dbn=128d_{\mathrm{bn}}=128 (ratio r=0.17r=0.17, strong compression) for 100 epochs each, and a three-configuration control at dbn=768d_{\mathrm{bn}}=768 (ratio r=1.0r=1.0, no compression) trained to 600 epochs. All runs use batch size 1024, AdamW (β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95, lr 10−410^{-4}, constant after a 5-epoch warmup), logit-normal time sampling (μ=−0.8\mu=-0.8, σ=0.8\sigma=0.8), label dropout 0.1, velocity clamp tϵ=0.05t_{\epsilon}=0.05, and EMA decay 0.9999, matching the JiT recipe of the Imagenette experiments. FID is computed with clean-fid [16] over 50K samples (Heun, 50 steps, CFG 2.9) against the full training set. Table 4 reports the results.

Both phenomena from the smaller-scale experiments reproduce. Under compression (r=0.17r=0.17), the prediction-space dichotomy is stark: xx-prediction attains val xx-MSE 0.023 and FID 20–37 across all loss types, vv-prediction degrades to MSE 0.29 and FID ≈200\approx200, and ϵ\epsilon-prediction collapses to MSE 14.6 and FID ≈420\approx420. The ϵ\epsilon-prediction error in ϵ\epsilon-space sits at 0.834–0.837 across the three loss types, within 0.5% of the closed-form floor (D−m)/D=640/768=0.833(D-m)/D = 640/768 = 0.833 of Theorem 4, a direct quantitative confirmation of the bottleneck lower bound at scale.

Removing the bottleneck (r=1.0r=1.0) sharpens the central message of the paper. We train all three control configurations to the full 600-epoch budget so that the comparison is at convergence rather than confounded by training time. The val xx-MSE spread collapses from 600-fold under compression to within a factor of two (0.022–0.043), confirming that information is no longer lost; yet the FID gap remains enormous (5.7 for xx-pred, 140.6 for vv-pred, 330.6 for ϵ\epsilon-pred). This is the dual dichotomy: once the architecture can represent the target, denoising quality (MSE) nearly equalizes, while prediction type still dominates generation quality (FID). For ϵ\epsilon-prediction, the per-timestep analysis below locates the gap at low tt, where the ϵ→v\epsilon\to v conversion of Proposition 6 is ill-conditioned. vv-prediction needs no conversion for ODE sampling, so Proposition 6 does not explain its gap. Its velocity error stays within 30% of the xx-model’s at every tt we measure and is lower at t=0.95t=0.95 (Table 12), and we leave the cause of its FID gap open. Crucially, the gap is not an undertraining artifact: from 100 to 600 epochs the xx-prediction FID falls sharply (24.8 →\to 5.7, reaching the range of published ImageNet-256 models [13]), while ϵ\epsilon-prediction barely moves (340.2 →\to 330.6) and vv-prediction improves only modestly (173.5 →\to 140.6). At convergence, vv-prediction remains ≈25×\approx25\times worse than xx-prediction in FID despite a near-identical xx-MSE (0.026 vs. 0.022), so the FID-vs-MSE decoupling is not a transient of finite training or an artifact of the bottleneck.

ttVelocity-space errorNative error
xx-modelϵ\epsilon-modelvv-modelϵ\epsilon-model
0.060.1060.4430.1160.0016
0.100.0810.1980.0930.0020
0.500.0510.0670.0660.017
0.900.1890.1970.1970.160
0.950.4620.3370.3330.304

Table 12. Per-timestep error of the three separately trained JiT-B/16 models on ImageNet-256 (r=1.0r=1.0, 600 epochs, vv-loss) in the common velocity space, together with the ϵ\epsilon-model’s native error ∥δϵ∥2=t2∥δv∥2\lVert\delta_\epsilon\rVert^2 = t^2\lVert\delta_v\rVert^2. Mean squared error per dimension against sampled targets, which includes the Bayes risk, over 2,560 validation images.

Figure 2 makes the mechanism explicit. Measuring per-timestep error on the converged r=1.0r=1.0 checkpoints, ϵ\epsilon-prediction’s velocity-space error, the quantity the ODE integrates, grows sharply as t→0t \to0, reaching 0.44 at t=0.06t=0.06 and 1.07 at t=0.02t=0.02 versus ≈0.065\approx0.065 at mid-range. Above tϵt_{\epsilon} this is the low-tt blow-up predicted by Proposition 6, and below tϵt_{\epsilon} the conversion clamp described next also contributes: xx- and vv-prediction stay bounded. Following the JiT recipe, every conversion clamps its denominator tt or 1−t1-t at tϵ=0.05t_{\epsilon}=0.05. The sampler starts at t=0t=0, so the first steps of ϵ\epsilon-prediction, at t=0t=0, 0.02, and 0.04, fall in the clamped region, as do the last steps of xx-prediction at t=0.96t=0.96 and 0.98. The clamp caps the amplification at 1/tϵ1/t_{\epsilon}, but even an exact network output then yields a velocity shrunk by the factor t/tϵt/t_{\epsilon} for ϵ\epsilon-prediction, or (1−t)/tϵ(1-t)/t_{\epsilon} for xx-prediction. The ϵ\epsilon-prediction FID gap therefore reflects the ϵ→v\epsilon\to v conversion together with the clamp that regularizes it, and we do not separate the two contributions.

Comparing separately trained models. Different models may have different prediction errors before conversion. We therefore compare their predictions in velocity space, using the same target v=x−ϵv = x - \epsilon for all three models. This is also the quantity used by the ODE sampler.

Remark 3 (Comparing prediction errors in velocity space). For a model predicting u∈{x,ϵ,v}u \in\{x,\epsilon,v\}, let u⋆=E[u∣zt,t]u^\star= \mathbb{E}[u \mid z_t,t] be the Bayes predictor and eu=u^−u⋆e_u = \hat{u} - u^\star the error relative to it. The class label is implicit in the conditioning of the Bayes predictors. These predictors map exactly onto one another under the affine conversions in Proposition 1.

Let v^u\hat{v}_u denote the velocity estimate obtained from u^\hat{u}. At each fixed tt where denominator clamping is inactive,

E[∥v^u−v∥2∣t]=E[∥v⋆−v∥2∣t]+wv∣u(t)E[∥eu∥2∣t],\mathbb{E}[\lVert\hat{v}_u-v\rVert^2 \mid t] = \mathbb{E}[\lVert v^\star-v\rVert^2 \mid t] + w_{v\mid u}(t)\mathbb{E}[\lVert e_u\rVert^2 \mid t],

where wv∣u(t)w_{v\mid u}(t) is given in Table 1. The cross term vanishes because E[v−v⋆∣zt,t]=0\mathbb{E}[v-v^\star\mid z_t,t] = 0 and v^u−v⋆\hat{v}_u-v^\star is determined by the model’s inputs.

All three prediction types share the same Bayes risk in velocity space. Differences in their expected velocity MSEs therefore equal differences in their excess MSEs. The converted excess MSE depends on both the conversion weight and the excess MSE before conversion. The same decomposition holds for MSE per dimension after normalizing all terms by the number of output coordinates.

Table 12 reports empirical estimates of these velocity-space MSEs. At t=0.06t = 0.06, the ϵ\epsilon-model’s MSE is 0.0016 before conversion and 0.443 in velocity space, where the conversion weight is wv∣ϵ(t)=1/t2≈278w_{v\mid\epsilon}(t) = 1/t^2 \approx278. The xx-model’s velocity-space MSE is 0.106. The observed difference of 0.337 estimates the corresponding difference in excess MSE.

The ordering between these two models reverses at t=0.95t = 0.95: the ϵ\epsilon-model has a lower velocity-space MSE than the xx-model (0.337 versus 0.462). Thus, the comparison depends on the learned prediction errors as well as the conversion weights. The table excludes timesteps where denominator clamping changes the conversion formulas.

Stochastic sampling. Proposition 6 concerns the drift field, not the integrator. Consider the reverse samplers that share the marginals of the probability-flow ODE: for any schedule w(t)≥0w(t) \ge0,

dz=[v^(z,t)+w(t)s^(z,t)] dt+2w(t) dWt,(19)\mathrm{d}z = [\hat{v}(z,t) + w(t)\hat{s}(z,t)]\,\mathrm{d}t + \sqrt{2w(t)}\,\mathrm{d}W_t, \tag*{(19)}

where s^\hat{s} estimates the score ∇log⁡pt\nabla\log p_t and w=0w = 0 recovers the ODE. Both drift terms are affine conversions of the same network output. For an ϵ\epsilon-prediction model, v^=(zt−ϵ^)/t\hat{v} = (z_t-\hat{\epsilon})/t and s^=−ϵ^/(1−t)\hat{s} = -\hat{\epsilon}/(1-t), so a native error δϵ\delta_\epsilon gives

δv=−δϵt,δs=−δϵ1−t.\delta_v = -\frac{\delta_\epsilon}{t}, \qquad\delta_s = -\frac{\delta_\epsilon}{1-t}.

The low-tt amplification therefore stays in the drift for every choice of ww. Injected noise can change how a drift error affects the sample distribution, but it does not remove a systematic error in the field. We test this on the 600-epoch r=1.0r=1.0 checkpoints with Euler–Maruyama sampling of (19), w(t)=1−tw(t)=1-t, and 50 steps, against the Heun ODE with the same step count. Heun evaluates the network twice per step, so the ODE uses 99 network evaluations against 50 for the SDE. Both samplers use a matched 4K-sample FID protocol, which is upward-biased relative to the 50K numbers in Table 4 but comparable across samplers. Table 13 shows that ϵ\epsilon-prediction does not recover under stochastic sampling, while xx-prediction changes only modestly. Scaling the schedule to w(t)=4(1−t)w(t) = 4(1-t) and 16(1−t)16(1-t) keeps the ϵ\epsilon-prediction FID between 328 and 372. The FID gap thus originates in a drift-field error that the ODE and this SDE share. This conclusion is limited to samplers that use the same conversion. Samplers that avoid it, such as DDPM ancestral sampling in native ϵ\epsilon-space, also avoid the amplification (§ 5).

FID (4K samples)ODE (Heun)SDE
xx-prediction14.517.1
ϵ\epsilon-prediction333.4330.2

Table 13. ODE versus SDE sampling on the 600-epoch ImageNet-256 checkpoints (r=1.0r=1.0, vv-loss). FID over 4K samples, 50 steps, CFG 2.9. The SDE is Euler–Maruyama on ((19)) with w(t)=1−tw(t) = 1-t.

Relation to EDM preconditioning

EDM [10] writes the denoiser as

Dθ(z~;σ)=cskip(σ)z~+cout(σ)Fθ(cin(σ)z~;σ),D_{\theta}(\tilde{z}; \sigma) = c_{\mathrm{skip}}(\sigma)\tilde{z} + c_{\mathrm{out}}(\sigma)F_{\theta}\left(c_{\mathrm{in}}(\sigma)\tilde{z}; \sigma\right),
cskip=σdata2σ2+σdata2,cout=σσdataσ2+σdata2c_{\mathrm{skip}} = \frac{\sigma_{\mathrm{data}}^{2}}{\sigma^{2}+\sigma_{\mathrm{data}}^{2}}, \qquad c_{\mathrm{out}} = \frac{\sigma\sigma_{\mathrm{data}}}{\sqrt{\sigma^{2}+\sigma_{\mathrm{data}}^{2}}}

where z~=x+σn\tilde{z} = x + \sigma n with n∼N(0,ID)n \sim\mathcal{N}(0,I_{\mathrm{D}}). Our interpolant takes this form with z~=zt/t\tilde{z} = z_t/t and σ=(1−t)/t\sigma= (1-t)/t.

EDM lies outside our static 3×33 \times3 grid for two reasons. First, its network target changes continuously with the noise level. As σ→∞\sigma\to\infty, cskip→0c_{\mathrm{skip}} \to0 and FθF_{\theta} targets the rescaled data x/σdatax/\sigma_{\mathrm{data}}. As σ→0\sigma\to0, cskip→1c_{\mathrm{skip}} \to1 and FθF_{\theta} targets a residual close to −n-n, an ϵ\epsilon-like quantity. Second, the denoiser includes the full-rank skip path cskip(σ)z~c_{\mathrm{skip}}(\sigma)\tilde{z}.

Our framework still applies to each ingredient. An error δF\delta_F in the network output enters the xx-space estimate as cout(σ)δFc_{\mathrm{out}}(\sigma)\delta_F, so cout2c_{\mathrm{out}}^2 plays the role of the amplification factor in Proposition 6. In the same coordinates, that proposition gives the factor σ2\sigma^2 for ϵ\epsilon-prediction, which is unbounded at high noise, and 11 for xx-prediction, which does not vanish at low noise. EDM’s factor satisfies cout2≤min⁡(σ2,σdata2)c_{\mathrm{out}}^2 \le\min(\sigma^2,\sigma_{\mathrm{data}}^2) and is bounded at both ends. EDM’s loss weight λ(σ)=1/cout(σ)2\lambda(\sigma) = 1/c_{\mathrm{out}}(\sigma)^2 makes the effective weight on δF\delta_F uniform over noise levels, which is a loss-space reweighting in the sense of Proposition 1. On the xx-space error, λ(σ)=σ−2+σdata−2\lambda(\sigma) = \sigma^{-2}+\sigma_{\mathrm{data}}^{-2}, our ϵ\epsilon-loss weight wϵ∣x=t2/(1−t)2=σ−2w_{\epsilon|x} = t^2/(1-t)^2 = \sigma^{-2} plus a constant. The skip term bypasses any compressed representation, so the final denoiser is not a function of ρ=Ezt\rho= \mathbb{E}z_t alone. Assumption (A3) then fails, and the D−mD-m floor of Theorem 4 does not apply.

We therefore expect EDM preconditioning to mitigate the bottleneck failure of ϵ\epsilon- and vv-prediction, but we have neither a formal result nor a controlled experiment that establishes this. A direct cskipc_{\mathrm{skip}} ablation on a bottlenecked ViT is an important next step.

References

  1. [1]Ting Chen. On the importance of noise scheduling for diffusion models. arXiv preprint arXiv:2301.10972, 2023.arxiv.org/abs/2301.10972
  2. [2]Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472–11481, 2022.arxiv.org/abs/2204.00227
  3. [3]Patrick Esser, Sumith Kullal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 12606–12633, 2024.arxiv.org/abs/2403.03206
  4. [4]Anne Gagneux, Ségolène Martin, Rémi Gribonval, and Mathurin Massias. Training flow matching: The role of weighting and parameterization. In ICLR 2026 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2026. arXiv:2603.06454.arxiv.org/abs/2603.06454
  5. [5]Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-SNR weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7441–7451, 2023.arxiv.org/abs/2303.09556
  6. [6]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.arxiv.org/abs/2207.12598
  7. [7]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.arxiv.org/abs/2006.11239
  8. [8]Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pages 13213–13232, 2023.arxiv.org/abs/2301.11093
  9. [9]Qing Jin and Chaoyang Wang. Revisiting diffusion model predictions through dimensionality. In Advances in Neural Information Processing Systems, 2026.DOI
  10. [10]Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022.arxiv.org/abs/2206.00364
  11. [11]Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data augmentation. Advances in Neural Information Processing Systems, 36:65484–65516, 2023.arxiv.org/abs/2303.00848
  12. [12]Dibyanshu Kumar, Philipp Váth, and Magda Gregorová. Loss functions in diffusion models: A comparative study. In Machine Learning and Knowledge Discovery in Databases. Research Track (ECML PKDD 2025), Lecture Notes in Computer Science, pages 190–205. Springer, 2025. doi: 10.1007/978-3-032-05981-9_12.
  13. [13]Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 36115–36125, 2026.arxiv.org/abs/2511.13720
  14. [14]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02747
  15. [15]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, 2023.arxiv.org/abs/2209.03003
  16. [16]Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On biased resizing and surprising subtleties in GAN evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11410–11420, 2022.arxiv.org/abs/2104.11222
  17. [17]Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2202.00512
  18. [18]Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. pmlr, 2015.
  19. [19]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1907.05600
  20. [20]Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.

Paper details

vlnc:W_r72nxetx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents