betaFeedback

WorksComputer Science

Density-Level Backward Error for Low-NFE Generative Flows

Abstract

Low-NFE generative ODE sampling is constrained not only by solver order, but also by how a fixed budget of network evaluations is distributed along the flow. We develop a density-level backward-error framework that identifies the component of a solver's modified-equation defect that changes the evolving marginal, while excluding solenoidal trajectory distortion that is invisible in Wasserstein tangent geometry. The resulting residual induces a solver-order-aware compute-allocation rule and, through gauge freedom, an objective for selecting numerically favorable velocity representatives of the same density path. This separates our approach from schedule search driven by trajectory or endpoint surrogates and from higher-order solver design. Exact Gaussian and mixture models establish the mechanism, including cases where curvature-based allocation fails and continuum-optimal transport is not discretization-optimal. On a pretrained CIFAR-10 EDM probability-flow ODE, density-aware allocation consistently improves fixed-update Euler sampling in the low-NFE regime. Matched-seed convergence to a dense Euler reference confirms that the method accelerates approximation of the same high-NFE flow rather than changing the asymptotic sampler.

Introduction

Diffusion and flow-based generators sample by numerically integrating a learned time-dependent velocity field. Let vtv_t be a reference velocity transporting a prescribed density path ptp_t through

∂tpt+∇⋅(ptvt)=0.(1)\partial_t p_t + \nabla\cdot(p_t v_t) = 0. \tag*{(1)}

When only a few network evaluations are available, generation quality depends not only on the learned field and solver formula, but also on where those evaluations are placed along the path.

Timestep design has therefore become an optimization problem in its own right. Align Your Steps optimizes schedules for a given solver, model, and dataset; optimized-time-step methods fit nodes against a reference ODE solution; LD3 learns a discretization for a pretrained sampler; Optimal Stepsize Distillation uses dynamic programming over reference trajectories; and differentiable solver search jointly tunes timesteps and solver coefficients ([28, 33, 25, 37] (Xue et al. 2024)). These methods show that evaluation placement matters. They do not, however, isolate which part of a local numerical defect changes the evolving marginal, nor do they use the fact that several velocity fields can generate the same density path.

Three design variables should therefore be separated. Solver design chooses the update formula. Schedule design chooses where that formula evaluates the field. Representative design chooses which velocity field to discretize when several fields generate the same continuum marginals. Our paper connects the last two through density-level backward error.

To make this non-uniqueness explicit, define Up={u:∇⋅(ptut)=0 for almost every t}\mathcal{U}_p = \{u : \nabla\cdot(p_t u_t) = 0\text{ for almost every }t\}. If u∈Upu \in\mathcal{U}_p, then w=v+uw = v + u generates the same continuity curve as vv, although a finite numerical map need not preserve this equivalence. A trajectory-level defect can therefore contain substantial particle motion that is invisible to the marginal. The relevant question is not simply where is the ODE difficult?, but which part of the solver defect moves probability mass?

This also separates continuum and finite-step optimality. Probability-flow endpoint maps coincide with quadratic Monge maps in important Gaussian and symmetric settings ([17]), but not in general Lavenant & Santambrogio, [18]. Here the density curve ptp_t is fixed. We ask which continuum-equivalent velocity representative and which timestep grid make the discrete transport maps most accurate in W2W_2. Even for isotropic Gaussians, the minimum-action gradient representative can be less accurate for Euler than a non-gradient marginal-preserving representative.

Let Ψhw\Psi_h^w be one numerical step for a representative ww, and suppose an order-rr method admits the local modified field

wh=w+hrEr[w]+O(hr+1).(2)w_h = w + h^r \mathcal{E}_r[w] + O(h^{r+1}). \tag*{(2)}

Write P∇,ptP_{\nabla,p_t} for the L2(pt)L^2(p_t)-orthogonal projection onto gradient fields. This projection keeps the Wasserstein tangent component that changes the marginal and removes marginal-invisible solenoidal motion. Our central local result is

(Ψhw)#pt−pt+h=−hr+1∇⋅(ptEr[w])+o(hr+1),(3)(\Psi_h^w)_\#p_t - p_{t+h} = -h^{r+1}\nabla\cdot(p_t\mathcal{E}_r[w]) + o(h^{r+1}), \tag*{(3)}
W2((Ψhw)#pt,pt+h)=hr+1∥P∇,ptEr[w]∥L2(pt)+o(hr+1).(4)W_2((\Psi_h^w)_\#p_t,p_{t+h}) = h^{r+1}\lVert P_{\nabla,p_t}\mathcal{E}_r[w]\rVert_{L^2(p_t)} + o(h^{r+1}). \tag*{(4)}

For Euler, E1[w]=−12aw\mathcal{E}_1[w] = -\frac{1}{2}a_w with aw=∂tw+∇wwa_w = \partial_t w + \nabla_w w. Thus the projected modified-equation defect, rather than the full trajectory defect, determines the leading marginal error.

This separates continuum transport optimality from two finite-solver questions: where to place evaluations and which continuum-equivalent field to discretize.

Contributions.

  1. Density-level backward error. We project the first nonzero modified-equation differential onto the weighted Wasserstein tangent space. Equations (3)–(4) identify its norm with the leading local marginal W2W_2 defect and discard solenoidal motion that is invisible to the density.

  1. Finite-budget compute allocation. Let Rr(t)=∥P∇,ptEr[w]∥L2(pt)R_r(t) = \lVert P_{\nabla,p_t}\mathcal{E}_r[w]\rVert_{L^2(p_t)}, and let K(1,t)K(1,t) denote the amplification of a density perturbation introduced at time tt and propagated to terminal time. Then

n∗(t)∝[K(1,t)Rr(t)]1/(r+1).n^*(t) \propto[K(1,t)R_r(t)]^{1/(r+1)}.

The solver order fixes the exponent; the associated quantile grid exactly minimizes the finite-NN local surrogate defined in § 6, and a Rényi identity quantifies residual misspecification.

  1. Solver-aware representative selection. Among fields w=v+uw = v + u with u∈Upu \in\mathcal{U}_p, we define a continuum solver objective. For Euler, renormalized discrete local-W2W_2 objectives converge uniformly—and hence Γ\Gamma-converge—to this functional. Gaussian paths give explicit controls and separate continuum minimum action from finite-step solver accuracy.

  1. Mechanism-aligned validation. Analytic Gaussian and mixture experiments test the projected defect, propagation, and representative-selection predictions. On pretrained CIFAR-10 EDM, we keep the Euler update fixed and change only the timestep grid. A dense-reference study verifies faster convergence to the same high-NFE Euler limit rather than a change of the target sampler.

Two optimization problems. Representative selection minimizes

min⁡u∈UpB(v+u),B(w)=12∫01∥P∇,pta[w]∥L2(pt)2 dt.\min_{u\in\mathcal{U}_p}\mathcal{B}(v+u), \qquad\mathcal{B}(w)=\frac{1}{2}\int_0^1\lVert P_{\nabla,p_t}a[w]\rVert_{L^2(p_t)}^2\,dt.

For an order-rr solver, let Wr(t)=K(1,t)Rr(t)W_r(t)=K(1,t)R_r(t). Timestep allocation solves

min⁡n>0∫01Wr(t)n(t)−r dts.t.∫01n(t) dt=N.\min_{n>0}\int_0^1 W_r(t)n(t)^{-r}\,dt \quad\text{s.t.}\quad \int_0^1 n(t)\,dt=N.

The first problem chooses a continuum-equivalent velocity field for a fixed solver; the second distributes that solver’s evaluation budget along the path.

The empirical comparisons keep this separation explicit. Euler schedules use the same network, update rule, NFE, and sampling cost. DPM++ and the original EDM Heun sampler are shown only as stronger-solver context: they change the integration formula, whereas our method changes where a fixed first-order solver spends its evaluations. The offline allocation profile is reusable across generated samples and NFE choices.

Related Work and Positioning

Generative flows and optimal transport. Continuous normalizing flows and neural ODEs formulate generation as numerical integration of a learned velocity field [5]. Diffusion and score-based models admit probability-flow ODE descriptions [11, 30], while flow matching directly regresses vector fields for prescribed probability paths [20, 1]. Rectified and multisample flow matching simplify transport through straighter trajectories or improved couplings [21, 27]. Probability-flow endpoint maps agree with quadratic optimal transport in important structured cases [17], but not universally [18]. These works concern the continuum path or endpoint map. We fix the marginal curve and optimize the numerical realization of its gauge-equivalent lifts. This also differs from Wasserstein spline and second-order transport models, which optimize the curve itself [4].

Solver and schedule optimization. Backward error analysis and B-series modified equations are classical in geometric numerical integration [9] and have been applied to diffusion sampling [8]; the probability-flow ODE also admits nonasymptotic convergence analyses [6]. Solver design ranges from the EDM Heun construction to DPM-Solver and empirical-statistics refinements [15, 22] (Zheng et al. 2023).

A complementary line optimizes the discretization itself. Align Your Steps uses stochastic-calculus objectives specialized to a solver, model, and data distribution [28]. Xue et al. (2024) optimize time nodes by comparing the numerical and reference ODE solutions. LD3 learns a lightweight discretization for pretrained samplers [33], while Optimal Stepsize Distillation applies dynamic programming to reference trajectories [25]. Differentiable Solver Search jointly searches timesteps and solver coefficients [37]. Adaptive step control, Wasserstein-bounded schedules, continuous-time clock control, and instance-aware grids further enlarge the schedule class [7, 14, 13] (Yuan et al. 2026). Our contribution is orthogonal to how that class is searched: we derive the local quantity to allocate by projecting the first modified-equation differential through the continuity equation. This yields a marginal residual, a solver-order exponent, and an additional optimization variable—the marginal-preserving velocity representative.

Gauge freedom and marginal observability. The non-uniqueness of a velocity field realizing a fixed density path is increasingly visible in diffusion and flow modeling. Horvat & Pfister (2024) analyze gauge freedom and conservativity; recent work separates gradient and solenoidal components by marginal observability [16]. EDDY uses divergence-free freedom for marginal-preserving particle diversity [35], while Gauge Flow Models, Flux Matching, and Wasserstein-flow reinterpretations treat the field itself as a design variable [31, 24, 36]. Curvature minimization and strain/vorticity decompositions instead diagnose trajectory-level integration difficulty [19, 32]. We apply the weighted Hodge projection to the solver’s modified-equation defect: only its gradient component contributes to the leading marginal W2W_2 error, and the remaining gauge freedom can be optimized for numerical integrability.

Optimization viewpoint. The schedule layer is a continuous resource-allocation problem, whereas representative selection is a PDE-constrained optimization problem with weighted Poisson projections. This connects the construction to computational optimal transport and Wasserstein geometry [3, 23, 2, 34, 29] and to PDE-constrained optimization [10]. The local theorem provides an exact solver-level bridge, while the variational limit connects the discrete and continuum optimization problems. Exact Gaussian analyses of diffusion discretization provide a complementary controlled perspective [26].

Marginal-Preserving Gauges

Analytic setting. Throughout the main text we work on either the flat torus Td\mathbb{T}^{d}, a smooth compact boundaryless domain, or Rd\mathbb{R}^{d} with sufficient decay so that all integrations by parts below have no boundary contribution. We assume pt>0p_t>0, ∫pt=1\int p_t=1, and enough smoothness of ptp_t and the vector fields to justify the displayed identities; a convenient sufficient condition is p∈Ct,x2p\in C^{2}_{t,x} and v,u\in C^{1}_{t,x}^{2} with bounded derivatives on compact time intervals. On Rd\mathbb{R}^{d} test functions are compactly supported unless stronger decay is imposed. All divergence constraints may equivalently be read in the weak sense. Let T=C∞(M)/R\mathcal{T}=C^{\infty}(M)/\mathbb{R} on a compact boundaryless domain and T=Cc∞(Rd)\mathcal{T}=C^{\infty}_{c}(\mathbb{R}^{d}) in the Euclidean case.

Definition 1 (Solenoidal gauge class). For a fixed density ptp_t, define

Upt={u∈L2(pt):∫pt⟨u,∇ϕ⟩ dx=0∀ϕ∈T}.(5)\mathcal{U}_{p_t}=\left\{u\in L^{2}(p_t):\int p_t\langle u,\nabla\phi\rangle\,dx=0\quad\forall\phi\in\mathcal{T}\right\}. \tag*{(5)}

For smooth fields this is equivalent to ∇⋅(ptut)=0\nabla\cdot(p_tu_t)=0 in distributions. Over time we write Up={ut∈Upt for a.e. t}\mathcal{U}_{p}=\{u_t\in\mathcal{U}_{p_t}\text{ for a.e. }t\} and the representative fiber as G(v,p)=v+Up\mathcal{G}(v,p)=v+\mathcal{U}_{p}.

Proposition 1 (Gauge invariance). If vtv_t satisfies equation 1 and ut∈Upu_t\in\mathcal{U}_{p}, then wt=vt+utw_t=v_t+u_t satisfies

∂tpt+∇⋅(ptwt)=0.(6)\partial_t p_t+\nabla\cdot(p_tw_t)=0. \tag*{(6)}

Proof. By linearity,

∂tpt+∇⋅(pt(vt+ut))=∂tpt+∇⋅(ptvt)+∇⋅(ptut)=0.\partial_t p_t+\nabla\cdot\left(p_t(v_t+u_t)\right)=\partial_t p_t+\nabla\cdot(p_tv_t)+\nabla\cdot(p_tu_t)=0.

The proposition is elementary, but it defines the feasible set of the optimization problem. In one dimension, this feasible set is trivial under standard decay assumptions: ∂x(ptut)=0\partial_x(p_tu_t)=0 implies ptut=c(t)p_tu_t=c(t), and decay gives c(t)=0c(t)=0. Hence nontrivial gauge optimization is intrinsically multidimensional.

Solver-Aware Density Defect

Let Ψhw\Psi_h^w denote one step of a numerical method applied to the non-autonomous ODE x˙t=wt(xt)\dot{x}_t=w_t(x_t). We use the extended system (x,τ)′=(wτ(x),1)(x,\tau)'=(w_\tau(x),1) when invoking modified-equation expansions.

References

  1. [1]Michael S. Albergo, Nicholas M. Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions, 2023. arXiv preprint arXiv:2303.08797.arxiv.org/abs/2303.08797
  2. [2]Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient Flows in Metric Spaces and in the Space of Probability Measures. Birkhäuser, 2008.
  3. [3]Jean-David Benamou and Yann Brenier. A computational fluid mechanics solution to the Monge–Kantorovich mass transfer problem. Numerische Mathematik, 84(3):375–393, 2000.DOI
  4. [4]Jean-David Benamou, Thomas O. Gallouët, and François-Xavier Vialard. Second-order models for optimal transport and cubic splines on the Wasserstein space. Foundations of Computational Mathematics, 19(5):1113–1143, 2019. doi: 10.1007/s10208-018-9389-x.
  5. [5]Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, 2018.arxiv.org/abs/1806.07366
  6. [6]Sitan Chen, Sinho Chewi, Holden Lee, Yuanzhi Li, Jianfeng Lu, and Adil Salim. The probability flow ODE is provably fast. In Advances in Neural Information Processing Systems, volume 36, pp. 68552–68575, 2023.arxiv.org/abs/2305.11798
  7. [7]Yuzhu Chen, Fengxiang He, Shi Fu, Xinmei Tian, and Dacheng Tao. Adaptive time-stepping schedules for diffusion models. In Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, volume 244 of Proceedings of Machine Learning Research, pp. 685–697. PMLR, 2024.
  8. [8]Yansong Gao, Zhihong Pan, Xin Zhou, Le Kang, and Pratik Chaudhari. Fast diffusion probabilistic model sampling through the lens of backward error analysis, 2023. arXiv preprint arXiv:2304.11446.
  9. [9]Ernst Hairer, Christian Lubich, and Gerhard Wanner. Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations. Springer, 2006.
  10. [10]Michael Hinze, Rene Pinnau, Michael Ulbrich, and Stefan Ulbrich. Optimization with PDE Constraints. Springer, 2009.DOI
  11. [11]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2006.11239
  12. [13]Yilie Huang, Wenpin Tang, and Xun Yu Zhou. ART for diffusion sampling: Continuous-time control and actor–critic learning, 2026. arXiv preprint arXiv:2607.02137.
  13. [14]Sangwoo Jo and Sungjoon Choi. Formalizing the sampling design space of diffusion-based generative models via adaptive solvers and wasserstein-bounded timesteps, 2026. arXiv preprint arXiv:2602.12624.
  14. [15]Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, volume 35, pp. 26565–26577, 2022.arxiv.org/abs/2206.00364
  15. [16]Naïl B. Khelifa, Richard E. Turner, and Ramji Venkataramanan. Diffusion models observe only gradients: A geometric perspective on score matching errors, 2026. arXiv preprint arXiv:2606.06179.arxiv.org/abs/2606.06179
  16. [17]Valentin Khrulkov, Gleb Ryzhakov, Andrei Chertkov, and Ivan Oseledets. Understanding DDPM latent codes through optimal transport. In International Conference on Learning Representations, 2023.
  17. [18]Hugo Lavenant and Filippo Santambrogio. The flow map of the Fokker–Planck equation does not provide optimal transport. Applied Mathematics Letters, 133:108225, 2022. doi: 10.1016/j.aml.2022.108225.DOI
  18. [19]Sangyun Lee, Beomsu Kim, and Jong Chul Ye. Minimizing trajectory curvature of ODE-based generative models. In International Conference on Machine Learning, 2023.arxiv.org/abs/2301.12003
  19. [20]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02747
  20. [21]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, 2023.arxiv.org/abs/2209.03003
  21. [22]Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2206.00927
  22. [23]Felix Otto. The geometry of dissipative evolution equations: The porous medium equation. Communications in Partial Differential Equations, 26(1–2):101–174, 2001.DOI
  23. [24]Peter Pao-Huang, Xiaojie Qiu, and Stefano Ermon. Generative modeling with flux matching, 2026. arXiv preprint arXiv:2605.07319.
  24. [25]Jianning Pei, Han Hu, and Shuyang Gu. Optimal stepsize for diffusion sampling, 2025. arXiv preprint arXiv:2503.21774.arxiv.org/abs/2503.21774
  25. [26]Emile Pierret and Bruno Galerne. Diffusion models for gaussian distributions: Exact solutions and wasserstein errors, 2025.arxiv.org/abs/2405.14250
  26. [27]Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, and Yaron Lipman. Multisample flow matching: Straightening flows with minibatch couplings. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28100–28127. PMLR, 2023.arxiv.org/abs/2304.14772
  27. [28]Amirmojtaba Sabour, Sanja Fidler, and Karsten Kreis. Align your steps: Optimizing sampling schedules in diffusion models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 42947–42975. PMLR, 2024.arxiv.org/abs/2404.14507
  28. [29]Filippo Santambrogio. Optimal Transport for Applied Mathematicians. Birkhäuser, 2015.DOI
  29. [30]Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  30. [31]Alexander Strunk and Roland Assam. Gauge flow models, 2025. arXiv preprint arXiv:2507.13414.arxiv.org/abs/2507.13414
  31. [32]Chenxi Tao and Seung-Kyum Choi. On the role of strain and vorticity in numerical integration error for flow matching, 2026. arXiv preprint arXiv:2605.06680.arxiv.org/abs/2605.06680
  32. [33]Vinh Tong, Dung Trung Hoang, Anji Liu, Guy Van den Broeck, and Mathias Niepert. Learning to discretize denoising diffusion ODEs. In The Thirteenth International Conference on Learning Representations, 2025.
  33. [34]Cédric Villani. Optimal transport: Old and new, 2009.
  34. [35]Gal Vinograd, Idan Achituve, and Ethan Fetaya. Diverse sampling in diffusion models with marginal preserving particle guidance, 2026. arXiv preprint arXiv:2605.06553.arxiv.org/abs/2605.06553
  35. [36]An B. Vuong, Michael T. McCann, Javier E. Santos, and Yen Ting Lin. Are we really learning the score function? reinterpreting diffusion models through wasserstein gradient flow matching, 2025. arXiv preprint arXiv:2509.00336.arxiv.org/abs/2509.00336
  36. [37]Shuai Wang, Zexian Li, Qipeng Zhang, Tianhui Song, Xubin Li, Tiezeng Ge, Bo Zheng, and Limin Wang. Differentiable solver search for fast diffusion sampling. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 63549–63568. PMLR, 2025.arxiv.org/abs/2505.21114

Paper details

vlnc:W_9mnjsajb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents