betaFeedback

WorksComputer Science

VA-TDM: A Unified View for Improving Few-Step Diffusion Models by Velocity Adversarial Distillation

Abstract

Few-step distillation offers efficient generation with diffusion and flow models, but preserving quality with only a few sampling steps remains challenging. Distribution-matching methods via score distillation provide a strong foundation, yet they rely on accurately tracking the student's changing distribution. At intermediate generation steps in few-step sampling, student-score estimation can be difficult, and commonly used estimators may describe a different distribution from the one used to update the student. Flow matching allows a more flexible approach to supervision through velocity fields constructed from the student's sampling process. Building on this perspective, we introduce Velocity Adversarial Distillation (VAD), which uses a learned critic to compare student and teacher fields and turn their disagreement into feedback for learning. VAD unifies score distillation, Fisher-based matching, and moment matching, also explaining existing heuristic MMD updates and delivering new training objectives. We further develop VA-TDM, which follows trajectory distribution matching: providing supervision for partial denoising along the student trajectory. Beyond this, VA-TDM combines transition-aware online velocity learning and dual-form VAD supervision across intermediate student transitions. Extensive results show that VA-TDM is powerful for learning few-step generators, achieving state-of-the-art four-step performance in both text-to-image and text-to-video generation across various backbones and scaling effectively to 33B MiniMax-H3. In particular, our VA-TDM-H3 notably outperforms the few-step Fast-H3 and is closer to the original many-step H3 according to human preference.

Few-step distillation offers efficient generation with diffusion and flow models, but preserving quality with only a few sampling steps remains challenging. Distribution-matching methods via score distillation provide a strong foundation, yet they rely on accurately tracking the student’s changing distribution. At intermediate generation steps in few-step sampling, student-score estimation can be difficult, and commonly used estimators may describe a different distribution from the one used to update the student. Flow matching allows a more flexible approach to supervision through velocity fields constructed from the student’s sampling process. Building on this perspective, we introduce Velocity Adversarial Distillation (VAD), which uses a learned critic to compare student and teacher fields and turn their disagreement into feedback for learning. VAD unifies score distillation, Fisher-based matching, and moment matching, also explaining existing heuristic MMD updates and delivering new training objectives. We further develop VA-TDM, which follows trajectory distribution matching: providing supervision for partial denoising along the student trajectory. Beyond this, VA-TDM combines transition-aware online velocity learning and dual-form VAD supervision across intermediate student transitions. Extensive results show that VA-TDM is powerful for learning few-step generators, achieving state-of-the-art four-step performance in both text-to-image and text-to-video generation across various backbones and scaling effectively to 33B MiniMax-H3. In particular, our VA-TDM-H3 notably outperforms the few-step Fast-H3 and is closer to the original many-step H3 according to human preference. Code is available at https://github.com/Luo-Yihong/VA-TDM.

Introduction

Few-step distillation seeks to make diffusion and flow models efficient without sacrificing generation quality [3, 6]. A central challenge is to provide effective supervision when a small number of student transitions must replace a long teacher trajectory. Trajectory distillation learns from individual teacher trajectories [37], whereas distribution matching allows the student to learn its own mapping by aligning student and teacher distributions [20, 44]. Trajectory Distribution Matching (TDM) [22] extends this principle to intermediate transitions. Each step then learns a partial denoising task, distributing supervision across the student trajectory and providing a natural foundation for few-step distillation.

For intermediate supervision to be effective, however, the training signal must faithfully reflect the distribution produced at each student step. In score-based methods, generator updates depend on the difference between teacher and student scores, requiring an auxiliary model to track the evolving student distribution. Intermediate transitions complicate this estimation: the student’s clean prediction and retained residual can be correlated, so perturbing its intermediate output generally differs from adding fresh Gaussian noise to the clean prediction. Our analysis of TDM shows why this distinction matters. Direct score estimation for the perturbed intermediate output can involve high-variance targets, while clean-target denoising can estimate a different distribution from the one used for generator supervision.

These observations motivate supervision that explicitly accounts for the student’s sampling process while allowing alternatives to its noisy-marginal score. Velocity targets offer this flexibility because they can be defined through paths with coupled endpoints [17], including dependencies inherited from a student transition. We build on this flexibility to introduce Velocity Adversarial Distillation (VAD), a framework for constructing student velocity fields and using their discrepancies from the teacher to guide distillation. Two prototypical velocity constructions are derived within VAD. A-form is a rectified velocity that captures the expected clean prediction conditioned on a perturbed sample; B-form is a score-equivalent Gaussian velocity for the student’s perturbed distribution. The two forms thus express conditional-moment and distributional supervision within the same sampling process. A vector-valued critic estimates the discrepancy between the chosen student field and the teacher field, and the generator learns from this discrepancy.

The resulting framework provides a unified perspective along two axes: which field to match and how to use its discrepancy to update the generator. With different generator updates, A-form connects to conditional moment matching, while B-form connects reverse-KL score distillation and Fisher-based field matching. These connections also clarify the relationship between an intended objective and its practical update. In particular, VAD explains the heuristic update of MMD [30] and derives an alternative surrogate that recovers the exact gradient of MMD’s desired objective.

To use both types of supervision along a few-step trajectory, we develop VA-TDM, retaining TDM’s supervision of intermediate transitions. VA-TDM combines A-form and B-form through a shared auxiliary backbone and blends complementary generator updates. Because samples from different student steps can follow different distributions even at the same noise level, we also condition field estimation on the transition being supervised. This provides different and accurate supervision signals among different steps.

Overall, our unified VAD framework connects existing distillation updates and derives new objectives, which VA-TDM combines with complementary velocity fields and transition-aware estimation for four-step generation, achieving state-of-the-art performance across backbones and tasks. On SD3.5-Medium, VA-TDM outperforms the compared few-step baselines across prompt following, image quality, and preference metrics, improving GenEval from TDM’s 0.61 to 0.67 while increasing sample diversity. On FLUX.2-Klein-Base 4B, it surpasses the released four-step model, raising GenEval from 0.78 to 0.80 and OCR similarity from 0.64 to 0.69, with diversity measured by LPIPS improving from 0.47 to 0.56. Most notably, VA-TDM scales to MiniMax-H3 (MiniMax, [25]), a state-of-the-art open-source audio-video model with 33B parameters. In our user study (Figure 2), VA-TDM wins 61.33% of comparisons against Fast-H3 at the same 4-NFE budget and achieves a 45.53% preference rate against the 100-NFE teacher, compared with 24.67% for Fast-H3. These results demonstrate that VA-TDM substantially narrows the perceptual quality gap to the teacher while requiring only 4-NFE.

Figure 2. Human preference on video synthesis.

Preliminaries

Diffusion and Flow Models. Diffusion models ([6, 34]) use the Gaussian forward process xτ=ατx0+στϵ\mathbf{x}_{\tau} = \alpha_{\tau}\mathbf{x}_{0} + \sigma_{\tau}\boldsymbol{\epsilon}, where x0∼q0\mathbf{x}_{0} \sim q_{0}, ϵ∼N(0,I)\boldsymbol{\epsilon} \sim\mathcal{N}(0,\mathbf{I}) independently, and qτq_{\tau} denotes the perturbed marginal. Time τ∈[0,1]\tau\in[0,1] runs from clean data to the noise prior; conditioning on c\mathbf{c} is omitted. For στ>0\sigma_{\tau} > 0, denoising score matching (DSM) learns the marginal score by regressing onto the conditional target:

sτq(xτ):=∇xτlog⁡qτ(xτ)=Eq0(x0∣xτ=x)[∇xτlog⁡qτ(x∣x0)].(1)\mathbf{s}^{q}_{\tau}(\mathbf{x}_{\tau}) := \nabla_{\mathbf{x}_{\tau}} \log q_{\tau}(\mathbf{x}_{\tau}) = \mathbb{E}_{q_{0}(\mathbf{x}_{0}\mid\mathbf{x}_{\tau}=\mathbf{x})}\left[\nabla_{\mathbf{x}_{\tau}}\log q_{\tau}(\mathbf{x}\mid\mathbf{x}_{0})\right]. \tag*{(1)}

Rectified flow (Liu et al., [17]) uses the linear schedule ατ=1−τ\alpha_{\tau} = 1 - \tau, στ=τ\sigma_{\tau} = \tau and velocity uτ(x)=E[ϵ−x0∣xτ=x]\mathbf{u}_{\tau}(\mathbf{x}) = \mathbb{E}[\boldsymbol{\epsilon} - \mathbf{x}_{0} \mid\mathbf{x}_{\tau} = \mathbf{x}]. For Gaussian forward processes, it is equivalent to diffusion models.

Trajectory Distribution Matching (TDM). TDM (Luo et al., [22]) matches intermediate student and teacher distributions along a KK-step student trajectory 0=t0<⋯<tK=10 = t_{0} < \cdots< t_{K} = 1. For a transition t=tk→s=tk−1t = t_{k} \to s = t_{k-1}, perturbing xs∼pθ,k(xs)\mathbf{x}_{s} \sim p_{\theta,k}(\mathbf{x}_{s}) with q(xτ∣xs)q(\mathbf{x}_{\tau} \mid\mathbf{x}_{s}), gives student marginal pθ,τpathp^{\mathrm{path}}_{\theta,\tau}. Learning is performed by reverse-KL divergence (Luo et al., [20]) along the student trajectory:

∇θLTDM=Ek,t>τ>sEq(xτ∣xs)pθ,k(xs)[wk,τ(∂xτ∂θ)⊤(∇xτlog⁡pθ,τpath(xτ)−sτq(xτ))].(2)\nabla_{\theta}\mathcal{L}_{\mathrm{TDM}} = \mathbb{E}_{k,t>\tau>s}\mathbb{E}_{q(\mathbf{x}_{\tau}\mid\mathbf{x}_{s})p_{\theta,k}(\mathbf{x}_{s})}\left[w_{k,\tau}\left(\frac{\partial\mathbf{x}_{\tau}}{\partial\theta}\right)^{\top}\left(\nabla_{\mathbf{x}_{\tau}}\log p^{\mathrm{path}}_{\theta,\tau}(\mathbf{x}_{\tau})-\mathbf{s}^{q}_{\tau}(\mathbf{x}_{\tau})\right)\right]. \tag*{(2)}

Teacher scores are supplied by the pretrained model; student scores require online estimation.

Method

Problem Setup. We consider distilling a pretrained conditional diffusion or flow model into an efficient few-step generator. Let qt(xt∣c)q_{t}(\mathbf{x}_{t}\mid\mathbf{c}) and ut(xt,c)\mathbf{u}_{t}(\mathbf{x}_{t},\mathbf{c}) denote the teacher’s marginal distribution and velocity field at time t∈[0,1]t \in[0,1], respectively, where t=0t = 0 corresponds to clean samples and t=1t = 1 to the noise prior. Given a condition c\mathbf{c} and a desired number of sampling steps KK, our goal is to learn a student that transforms noise into a sample through KK transitions while matching the teacher’s marginal distributions along the generation trajectory.

Overview. We introduce Velocity Adversarial Distillation (VAD) as a unified framework for distilling diffusion models through aligning student and teacher velocity fields by an adversarial game. § 3.1 motivates this formulation by examining the challenges of student score estimation and introduces two constructions of student velocity. § 3.2 develops the adversarial velocity game, characterizes its matching equilibrium, and establishes connections to existing distillation methods. § 3.3 presents practical generator and critic updates and applies the framework along the student trajectory, with transition-aware field estimation.

From Score Distillation to Velocity Distillation

We focus on learning a one-step marginal transition from xt\mathbf{x}_{t} to a less noisy xs\mathbf{x}_{s}, where 0≤s<t≤10 \le s < t \le1, whose output distribution pθ,sp_{\theta,s} should match the teacher marginal qsq_{s}. Matching an intermediate distribution requires only partial denoising at each step, easing the learning task compared with requiring a clean output at every step, and is more efficient and effective (Luo et al., [22]).

Under the rectified-flow schedule, the student predicts a clean anchor and its residual given xtx_t:

x^0=gθ(xt,t)=xt−tvθ(xt,t),ε^=xt−(1−t)x^0t.(3)\hat{x}_0 = g_\theta(x_t,t) = x_t - t v_\theta(x_t,t), \qquad\hat{\varepsilon} = \frac{x_t - (1-t)\hat{x}_0}{t}. \tag*{(3)}

Given x^0\hat{x}_0 and ε^\hat{\varepsilon}, we can parameterize a DDIM-style intermediate transition (Song et al., [35]):

xs=(1−s)x^0+ηsε^+s1−η2zs∼pθ(xs∣xt)≜N((1−s)x^0+ηsε^,s2(1−η2)I),(4)x_s = (1-s)\hat{x}_0 + \eta s\hat{\varepsilon} + s\sqrt{1-\eta^2}z_s \sim p_\theta(x_s \mid x_t) \triangleq\mathcal{N}\left((1-s)\hat{x}_0 + \eta s\hat{\varepsilon}, s^2(1-\eta^2)I\right), \tag*{(4)}

where 0≤η≤10 \le\eta\le1 and zs∼N(0,I)z_s \sim\mathcal{N}(0,I). Note that the clean anchor is a parameterization of the transition; the desired matching objective is imposed on xsx_s, not separately on x^0\hat{x}_0. By default, we let η=1\eta= 1, which forms a deterministic transition, while its outputs remain random across inputs.

Distributional Supervision at a Perturbed Time. To supervise the intermediate distribution, existing works in the line of score distillation (Luo et al., [20, 22]) further perturb xsx_s to xτx_\tau, where τ>s\tau> s. This smooths the distributions and makes the teacher’s knowledge at additional noise levels available for supervision. For s<τ<1s < \tau< 1, we use the teacher-compatible kernel

q(xτ∣xs)=N(xτ;ατ∣sxs,βτ∣s2I),ατ∣s=1−τ1−s,βτ∣s2=τ2−ατ∣s2s2.(5)q(x_\tau\mid x_s) = \mathcal{N}(x_\tau; \alpha_{\tau|s}x_s, \beta_{\tau|s}^{2}I), \qquad\alpha_{\tau|s} = \frac{1-\tau}{1-s}, \qquad\beta_{\tau|s}^{2} = \tau^2 - \alpha_{\tau|s}^{2}s^2. \tag*{(5)}

Thus xτ=ατ∣sxs+βτ∣szτx_\tau= \alpha_{\tau|s}x_s + \beta_{\tau|s}z_\tau, where zτ∼N(0,I)z_\tau\sim\mathcal{N}(0,I). The full sampling chain factors into q(xs∣x^0,ε^,η)q(x_s \mid\hat{x}_0, \hat{\varepsilon}, \eta) and q(xτ∣xs)q(x_\tau\mid x_s). Unless otherwise specified, expectations are taken over sampled τ\tau, this student transition and subsequent perturbation, with conditioning fixed. Let pθ,sp_{\theta,s} denote the distribution of the student’s intermediate output xsx_s. After applying the perturbation kernel, the resulting perturbed marginal distribution is

pθ,τpath(xτ)=∫q(xτ∣xs) pθ,s(xs) dxs.(6)p^{\mathrm{path}}_{\theta,\tau}(x_\tau) = \int q(x_\tau\mid x_s)\,p_{\theta,s}(x_s)\,\mathrm{d}x_s. \tag*{(6)}

This is the student distribution whose score is required for reverse-KL supervision at time τ\tau. Because xsx_s retains the residual inherited from the student transition, this distribution generally differs from one obtained by adding independent Gaussian noise directly to x^0\hat{x}_0.

If the optimization is formed by reverse-KL following previous works (Luo et al., [22]), we would require the score difference ∇log⁡pθ,τpath−∇log⁡qτ\nabla\log p^{\mathrm{path}}_{\theta,\tau} - \nabla\log q_\tau. The teacher supplies the latter score, whereas the former must be learned online for the current student’s actual perturbation distribution.

The Difficulty of Learning the Student Score. DSM can estimate this score ∇log⁡pθ,τpath\nabla\log p^{\mathrm{path}}_{\theta,\tau} by regressing onto ∇xτlog⁡q(xτ∣xs)\nabla_{x_\tau}\log q(x_\tau\mid x_s). Although population-correct, under the reference Gaussian forward process this target has larger conditional variance than standard DSM, which regresses ∇xτlog⁡q(xτ∣x0)\nabla_{x_\tau}\log q(x_\tau\mid x_0) (Xue et al., [42]), thereby making the online score harder to learn.

Remark 1 (Mismatch in TDM). To avoid this difficulty, TDM learns its online score through clean-target denoising with importance sampling (Luo et al., [22]). Although importance sampling allows training samples to be drawn from pθ,τpathp^{\mathrm{path}}_{\theta,\tau}, the objective still estimates the score of the distribution obtained by independently perturbing x^0\hat{x}_0 to τ\tau. This generally differs from the score of pθ,τpathp^{\mathrm{path}}_{\theta,\tau} required by the generator update, introducing a distribution mismatch (See detailed derivation in Appendix § 8.§ 2).

Velocity as a Choice of Path and Coupling. This distinction motivates a shift from score distillation to velocity distillation, which can define student fields directly from the actual sampling chain. In particular, Velocity targets can be induced by differentiable paths with correlated or non-Gaussian endpoint couplings, therefore giving us greater flexibility. In the student transition t→st \to s, x^0\hat{x}_0 and ε^\hat{\varepsilon} share the same input and can be dependent, alongside optional fresh noise; the forward perturbation s→τs \to\tau adds independent Gaussian noise. This gives two choices: use the clean-to-perturbed coupling induced by the composite process t→s→τt \to s \to\tau, or follow the s→τs \to\tau path alone. Throughout the remaining sections, pθ,τp_{\theta,\tau} abbreviates pθ,τpathp^{\mathrm{path}}_{\theta,\tau}, the perturbed marginal.

A-form: Rectified Velocity. The composite process t→s→τt \to s \to\tau couples the clean prediction x^0\hat{x}_0 to xτx_\tau. Expanding Eq. (4) yields

xτ=(1−τ)x^0+ξθ,s,τ,ξθ,s,τ=ατ∣sηsϵ^+ατ∣ss1−η2zs+βτ∣szτ.(7)x_\tau= (1-\tau)\hat{x}_0 + \xi_{\theta,s,\tau}, \qquad \xi_{\theta,s,\tau} = \alpha_{\tau\mid s}\eta_s\hat{\epsilon} + \alpha_{\tau\mid s}s\sqrt{1-\eta^2}z_s + \beta_{\tau\mid s}z_\tau. \tag*{(7)}

The effective noise ϵeff=ξθ,s,τ/τ\epsilon_{\mathrm{eff}} = \xi_{\theta,s,\tau}/\tau contains the residual inherited from the student and standard Gaussian noise. This can deliver a corresponding rectified velocity coupled with the endpoint:

vτA(xτ∣xs,x^0)=ϵeff−x^0=xτ−x^0τ.(8)v^A_\tau(x_\tau\mid x_s,\hat{x}_0) = \epsilon_{\mathrm{eff}}-\hat{x}_0 = \frac{x_\tau-\hat{x}_0}{\tau}. \tag*{(8)}

The A-form rectified velocity uses the clean prediction as its anchor through the composite coupling. It produces reduced training variance when s>0s>0, which benefits learning of the online field. This marginal velocity is an affine reparameterization of a moment E[x^0∣xτ]\mathbb{E}[\hat{x}_0 \mid x_\tau].

B-form: Gaussian Velocity. The perturbation s→τs \to\tau defines a path starting at xsx_s. Differentiating this path with respect to τ\tau, and using zτ=(xτ−ατ∣sxs)/βτ∣sz_\tau=(x_\tau-\alpha_{\tau\mid s}x_s)/\beta_{\tau\mid s}, gives the conditional velocity

vτB(xτ∣xs,x^0)=α˙τ∣sxs+β˙τ∣sβτ∣s(xτ−ατ∣sxs).(9)v^B_\tau(x_\tau\mid x_s,\hat{x}_0) = \dot{\alpha}_{\tau\mid s}x_s + \frac{\dot{\beta}_{\tau\mid s}}{\beta_{\tau\mid s}}(x_\tau-\alpha_{\tau\mid s}x_s). \tag*{(9)}

Dots denote τ\tau derivatives with ss fixed. Its conditional average is the marginal velocity of the s→τs \to\tau perturbation path. This marginal velocity is an affine reparameterization of the score of the actual perturbed distribution (see proof in Appendix § 7.§ 4).

The Induced Student Field. We use the common notation vτF(xτ∣xs,x^0)v^F_\tau(x_\tau\mid x_s,\hat{x}_0) for these two conditional targets, with F∈{A,B}F \in\{A,B\}. The field to be compared with the teacher is

vˉθ,τF(x)=Epθ(xs,x^0∣xτ=x)[vτF(x∣xs,x^0)].(10)\bar{v}^F_{\theta,\tau}(x) = \mathbb{E}_{p_\theta(x_s,\hat{x}_0\mid x_\tau=x)}\left[v^F_\tau(x\mid x_s,\hat{x}_0)\right]. \tag*{(10)}

Both formulations can be computed via student samples, but their marginal fields require averaging over the implicit student distribution.

Velocity Adversarial Distillation

Diff-Instruct has discussed that reverse-KL-based score distillation is related to GANs ([5, 20]). We further find that, after shifting from score distillation to velocity distillation, explicitly defining an adversarial game that aims to align the teacher velocity with the student velocity forms a unified framework and view, which deepens the understanding of existing methods and might inspire the design of new distillation losses.

Matching Implicit Velocity. We propose to align the student and teacher velocity fields through an adversarial game defined by their inner products with a vector-valued critic f\mathbf{f}, i.e., fTv\mathbf{f}^{\mathsf{T}}\mathbf{v}. In particular, the critic maximizes its alignment with the teacher velocity and minimizes its alignment with the student velocity, subject to a quadratic penalty. The student opposes the critic by maximizing its own velocity’s alignment with the critic, with the critic held fixed.

For brevity, we write vτF≡vτF(xτ∣xs,x^0)v^F_\tau\equiv v^F_\tau(x_\tau\mid x_s,\hat{x}_0) for the sampled conditional velocity, and let E\mathbb{E} denote expectation over τ\tau and the full sampling chain pθ(xt,xs,x^0)q(xτ∣xs)p_\theta(x_t,x_s,\hat{x}_0)q(x_\tau\mid x_s). Since f\mathbf{f} depends only on xτx_\tau, replacing the sampled velocity vτF\mathbf{v}^{F}_{\tau} by its conditional-mean velocity vˉθ,τF\bar{\mathbf{v}}^{F}_{\theta,\tau} leaves the expected inner product unchanged:

E[f(xτ)TvτF]=Exτ∼pθ,τ[f(xτ)Tvˉθ,τF(xτ)].(11)\mathbb{E}\left[\mathbf{f}(\mathbf{x}_{\tau})^{\mathsf{T}}\mathbf{v}^{F}_{\tau}\right]=\mathbb{E}_{\mathbf{x}_{\tau}\sim p_{\theta,\tau}}\left[\mathbf{f}(\mathbf{x}_{\tau})^{\mathsf{T}}\bar{\mathbf{v}}^{F}_{\theta,\tau}(\mathbf{x}_{\tau})\right]. \tag*{(11)}

This identity motivates two generator losses with the same expected value but different gradient conventions. The pathwise update differentiates through the full sampling process, changing both the outer sampling distribution and the student velocity at those locations. The conditional-mean update stops the gradient of the sampling distribution and adjusts only the student velocity field.

In particular, the adversarial game in the velocity space can be formulated as:

max⁡fVF(f,θ)=E[f(xτ)T(uτ(xτ)−vˉθ,τF)−12∥f(xτ)∥2],(12)\max_{\mathbf{f}}\quad V_{F}(\mathbf{f},\theta)=\mathbb{E}\left[\mathbf{f}(\mathbf{x}_{\tau})^{\mathsf{T}}\left(\mathbf{u}_{\tau}(\mathbf{x}_{\tau})-\bar{\mathbf{v}}^{F}_{\theta,\tau}\right)-\frac{1}{2}\lVert\mathbf{f}(\mathbf{x}_{\tau})\rVert^{2}\right], \tag*{(12)}
min⁡θLpathF(θ;f)=−E[f(xτ)TvτF],\min_{\theta}\quad\mathcal{L}^{F}_{\mathrm{path}}(\theta;\mathbf{f})=-\mathbb{E}\left[\mathbf{f}(\mathbf{x}_{\tau})^{\mathsf{T}}\mathbf{v}^{F}_{\tau}\right],

or

min⁡θGmeanF(θ;f)=−Exτ∼psg(θ),τ[f(xτ)Tvˉθ,τF(xτ)].(13)\min_{\theta}\quad\mathcal{G}^{F}_{\mathrm{mean}}(\theta;\mathbf{f})=-\mathbb{E}_{\mathbf{x}_{\tau}\sim p_{\mathrm{sg}(\theta),\tau}}\left[\mathbf{f}(\mathbf{x}_{\tau})^{\mathsf{T}}\bar{\mathbf{v}}^{F}_{\theta,\tau}(\mathbf{x}_{\tau})\right]. \tag*{(13)}

Here sg denotes stop-gradient. Only the outer distribution is stopped in GmeanF\mathcal{G}^{F}_{\mathrm{mean}}; vˉθ,τF\bar{\mathbf{v}}^{F}_{\theta,\tau} retains its parameter dependence. The following lemma reveals the formulation of the optimal critic:

Lemma 2 (Optimal critic). For a fixed student θ\theta, under the regularity and critic assumptions detailed in Appendix § 7, the maximizing critic is unique up to pθ,τp_{\theta,\tau}-null sets and satisfies

fF⋆(x)=uτ(x)−vˉθ,τF(x).(14)\mathbf{f}^{\star}_{F}(\mathbf{x})=\mathbf{u}_{\tau}(\mathbf{x})-\bar{\mathbf{v}}^{F}_{\theta,\tau}(\mathbf{x}). \tag*{(14)}

The proof is given in Appendix § 7.§ 1.

Matching Equilibrium. The optimal critic in Theorem 2 is the velocity residual. Its zero identifies the following equilibrium of the two-player game.

Proposition 3 (Matching equilibrium). Under the regularity, critic, and realizability assumptions in Appendix § 7, the game in Eq. (12) admits a Nash equilibrium satisfying

vˉθ⋆,τF=uτpθ⋆,τ-almost everywhere,f⋆≡0.(15)\bar{\mathbf{v}}^{F}_{\theta^{\star},\tau}=\mathbf{u}_{\tau}\quad p_{\theta^{\star},\tau}\text{-almost everywhere},\qquad\mathbf{f}^{\star}\equiv\mathbf{0}. \tag*{(15)}

It attains zero velocity discrepancy, the minimum of the maximized critic payoff.

The proof is in Appendix § 7.§ 2.

Directly computing the conditional-mean update is intractable because it requires differentiating the implicit field vˉθ,τF\bar{\mathbf{v}}^{F}_{\theta,\tau} with respect to θ\theta. The following lemma gives a general flow-product update through a tractable surrogate whose gradient agrees with the conditional-mean update.

Lemma 4 (General Flow-Product Update). Fix a reference parameter θ0=θˉ\theta_{0}=\bar{\theta} and let h0F(x)=vˉθ0,τF(x)\mathbf{h}^{F}_{0}(\mathbf{x})=\bar{\mathbf{v}}^{F}_{\theta_{0},\tau}(\mathbf{x}). Hold the parameters of f\mathbf{f} and h0F\mathbf{h}^{F}_{0} fixed, while retaining their input derivatives. Define

GfpF(θ;f,h0):=−Epθ(xt,xs,x^0)q(xτ∣xs)[f(xτ)T(vτF(xτ∣xs,x^0)−h0F(xτ))].(16)\mathcal{G}^{F}_{\mathrm{fp}}(\theta;\mathbf{f},\mathbf{h}_{0}) :=-\mathbb{E}_{p_{\theta}(\mathbf{x}_{t},\mathbf{x}_{s},\hat{\mathbf{x}}_{0})q(\mathbf{x}_{\tau}\mid\mathbf{x}_{s})}\left[\mathbf{f}(\mathbf{x}_{\tau})^{\mathsf{T}}\left(\mathbf{v}^{F}_{\tau}(\mathbf{x}_{\tau}\mid\mathbf{x}_{s},\hat{\mathbf{x}}_{0})-\mathbf{h}^{F}_{0}(\mathbf{x}_{\tau})\right)\right]. \tag*{(16)}

Under the regularity conditions detailed in Appendix § 7.§ 3,

∇θGfpF∣θ0=∇θGmeanF(θ;θ0,f)∣θ0=−Exτ∼pθ0,τ[f(xτ)T∂θvˉθ,τF(xτ)∣θ0].(17)\left.\nabla_{\theta}\mathcal{G}^{F}_{\mathrm{fp}}\right|_{\theta_{0}}=\left.\nabla_{\theta}\mathcal{G}^{F}_{\mathrm{mean}}(\theta;\theta_{0},\mathbf{f})\right|_{\theta_{0}}=-\mathbb{E}_{\mathbf{x}_{\tau}\sim p_{\theta_{0},\tau}}\left[\mathbf{f}(\mathbf{x}_{\tau})^{\mathsf{T}}\left.\partial_{\theta}\bar{\mathbf{v}}^{F}_{\theta,\tau}(\mathbf{x}_{\tau})\right|_{\theta_{0}}\right]. \tag*{(17)}

The proof is given in Appendix § 7.§ 3.

Theorem 5. (Matching meanings and algorithmic unification). Use the compatible Gaussian kernel in Eq. (5) and an exact teacher. For the gradient identities, fix the input law and times 0≤s<t≤10 \le s < t \le1, s<τ<1s < \tau< 1, take η=1\eta= 1, and assume exact critics and reference fields with the differentiability conditions of Appendix § 7.§ 3.

(i) Matching. A-form velocity matching is equivalent to the conditional-moment matching

mθ(x):=E[x^0∣xτ=x]=mq(x):=x−τuτ(x).(18)m_\theta(\mathbf{x}) := \mathbb{E}[\hat{\mathbf{x}}_0 \mid\mathbf{x}_\tau= \mathbf{x}] = m_q(\mathbf{x}) := \mathbf{x} - \tau\mathbf{u}_\tau(\mathbf{x}). \tag*{(18)}

B-form velocity matching is equivalent to distribution matching pθ,τ=qτp_{\theta,\tau} = q_\tau, and hence pθ,s=qsp_{\theta,s} = q_s.

(ii) Equivalence at s=0s = 0. At s=0s = 0, AA and BB coincide, with identical matching conditions and population gradients under the same generator update.

(iii) Unifying Algorithms. The two forms and two gradient choices yield the correspondences below, up to positive time weights.

FormPathwise w/ detached critic sg⁡(f(xτ))\operatorname{sg}(f(\mathbf{x}_\tau))Flow-product w/ non-detach critic f(xτ)f(\mathbf{x}_\tau)
AMMD practical update (Salimans et al., [30])Exact Moment Matching
BReverse-KL (Luo et al., [20, 22]; Yin et al. [44])Fisher Divergence (Luo et al., [21]; Zhou et al. [50])

The proof and time coefficients are in Appendix § 7.§ 4.

:::

MMD: Objective and Approximate Update. MMD (Salimans et al., [30]) aims to minimize a squared conditional-moment discrepancy to align the teacher moment and student moment. Because differentiating the student’s implicit moment is intractable, it adopts a heuristic approximate gradient to update the student generator. Our unified framework provides both an exact gradient and a mechanistic explanation of this heuristic: A-form flow-product updates recover the intended objective’s exact gradient under MMD’s fixed-outer-measure convention, while A-form pathwise updates with a detached critic recover the surrogate update as a generator update in the adversarial velocity game, up to positive time weighting on the same sampling chain. Appendix § 7.§ 4 explicitly gives the target objective, the approximation leading to the surrogate, and both gradient identities.

VA-TDM: Distillation Along the Student Trajectory

Generator Learning. We observe unstable training when gradients are propagated through the critic output (Figure 3). We therefore stop the entire critic output in the practical update. Let sg⁡\operatorname{sg} denote this stop-gradient operation and define

Figure 3. Comparison between detached critic and non-detached critic. We found that the non-detached critic leads to unstable training and collapse.

Lpath,sgF=−Epθ(xt,xs,x^0)q(xτ∣xs)[sg⁡(fF(xτ))TvτF(xτ∣xs,x^0)],\mathcal{L}^{F}_{\mathrm{path,sg}} = -\mathbb{E}_{p_\theta(\mathbf{x}_t,\mathbf{x}_s,\hat{\mathbf{x}}_0)q(\mathbf{x}_\tau\mid\mathbf{x}_s)}\left[\operatorname{sg}(f^F(\mathbf{x}_\tau))^\mathsf{T}v^F_\tau(\mathbf{x}_\tau\mid\mathbf{x}_s,\hat{\mathbf{x}}_0)\right],
Lfp,sgF=−Epθ(xt,xs,x^0)q(xτ∣xs)[sg⁡(fF(xτ))T(vτF(xτ∣xs,x^0)−vˉθ,τF)].(19)\mathcal{L}^{F}_{\mathrm{fp,sg}} = -\mathbb{E}_{p_\theta(\mathbf{x}_t,\mathbf{x}_s,\hat{\mathbf{x}}_0)q(\mathbf{x}_\tau\mid\mathbf{x}_s)}\left[\operatorname{sg}(f^F(\mathbf{x}_\tau))^\mathsf{T}\left(v^F_\tau(\mathbf{x}_\tau\mid\mathbf{x}_s,\hat{\mathbf{x}}_0)-\bar{v}^{F}_{\theta,\tau}\right)\right]. \tag*{(19)}

Note that Lfp,sgFL^{F}_{\mathrm{fp,sg}} and Lpath,sgFL^{F}_{\mathrm{path,sg}} correspond to different adversarial flexibility for the generator: the latter allows the generator to cheat the critic by altering both the outer sampling distribution and the velocity. Since the balance of the adversarial game is crucial for performance, we propose optimizing the hybrid loss to provide an explicit knob for controlling the balance.

LmixF=(1−λ)Lpath,sgF+λLfp,sgF.(20)L^{F}_{\mathrm{mix}} = (1-\lambda)L^{F}_{\mathrm{path,sg}}+\lambda L^{F}_{\mathrm{fp,sg}}. \tag*{(20)}

Critic Learning. With the help of Theorem 2, we parameterize the critic as fψF=uτ−hψFf^{F}_{\psi}=u_{\tau}-h^{F}_{\psi} and fit the student field with

LfieldF(ψ)=Epθ(xs,x^0)q(xτ∣xs)[∥hψF(xτ)−vτF(xτ∣xs,x^0)∥22].(21)L^{F}_{\mathrm{field}}(\psi)=\mathbb{E}_{p_{\theta}(x_{s},\hat{x}_{0})q(x_{\tau}\mid x_{s})}\left[\left\lVert h^{F}_{\psi}(x_{\tau})-v^{F}_{\tau}(x_{\tau}\mid x_{s},\hat{x}_{0})\right\rVert_{2}^{2}\right]. \tag*{(21)}

To retain the pre-trained knowledge, we add the regularization term λrE[∥hψF(xτ)−uτ(xτ)∥22]\lambda_{r}\mathbb{E}[\lVert h^{F}_{\psi}(x_{\tau})-u_{\tau}(x_{\tau})\rVert_{2}^{2}], with λr>0\lambda_{r}>0, under the same sampling distribution. The optimal minimizer then becomes

hψ⋆F(x)=vˉθ,τF(x)+λruτ(x)1+λr.h^{F}_{\psi^{\star}}(x)=\frac{\bar{v}^{F}_{\theta,\tau}(x)+\lambda_{r}u_{\tau}(x)}{1+\lambda_{r}}.

We therefore should correctly parameterize the critic as fψF=(1+λr)(uτ−hψF)f^{F}_{\psi}=(1+\lambda_{r})(u_{\tau}-h^{F}_{\psi}) to perform best.

Dual-Form Velocity Game with a Shared Backbone. The A-form produces low-variance online field learning, but it only captures the conditional-mean moment of the distribution; the B-form induces high-variance online field learning, yet it encodes the full distributional information. Since no single velocity construction enjoys both benefits, we propose to optimize the adversarial games induced by the two velocity constructions simultaneously, the generator loss becomes:

Lmix=λmLmixA+(1−λm)LmixB,(22)L_{\mathrm{mix}}=\lambda_{m}L^{A}_{\mathrm{mix}}+(1-\lambda_{m})L^{B}_{\mathrm{mix}}, \tag*{(22)}

where each term follows Eq. (20) with its corresponding velocity target. To avoid the cost of maintaining two online fields, we propose a shared-backbone strategy: the two field networks share the backbone with separate output heads. Table 2 shows that this sharing strategy not only reduces memory and training overhead, but also improves performance.

ConfigurationVerifiableImgQPreferenceDiv.
GenEvalOCRAest.ImgRwdPickScoreHPSv2HPSv3LPIPS
λm=1\lambda_m=1 (A-form only)0.640.555.531.000.86710.294911.20<u>0.58</u>
λm=0\lambda_m=0 (B-form only)0.610.575.570.990.8665<u>0.2987</u><u>11.47</u>0.59
λm=0\lambda_m=0 w/o transition-aware field0.580.535.560.990.8655<u>0.2964</u>11.130.56
λm=0.8\lambda_m=0.8 w/o shared backbone0.620.565.550.990.86690.297111.330.57
λm=0.8\lambda_m=0.80.63<u>0.58</u>5.56<u>1.01</u>0.86740.2983<u>11.47</u><u>0.58</u>
+ path/flow-product mixing (λ=0.7\lambda=0.7)0.65<u>0.57</u>5.560.96<u>0.8687</u>0.2950<u>11.39</u>0.59
+ field regularization (λr=1\lambda_r=1)<u>0.66</u><u>0.58</u><u>5.58</u>0.960.86820.295211.440.57
+ regularization-corrected critic0.670.595.651.020.87020.301011.89<u>0.58</u>

Table 2. Ablation study. Best / second-best results are bold / <u>underlined</u>.

Distillation Along the Student Trajectory. Matching only the final output places the entire distributional correction at the end of generation. We have discussed the algorithm for learning one-step marginal transition from xtx_{t} to xsx_{s}. Following TDM [22], we instead apply VAD to intermediate student transitions along the student trajectory. Let 0=t0<t1<⋯<tK=10=t_{0}<t_{1}<\cdots<t_{K}=1 be a sampling grid. Starting from xtK∼N(0,I)x_{t_{K}}\sim\mathcal{N}(0,I), the student repeatedly applies the student transition in Eq. (4) with (t,s)=(tk,tk−1)(t,s)=(t_{k},t_{k-1}) and η=1\eta=1. Each transition learns a distributional mapping from its current noisy state to the next state; no pointwise teacher ODE solution is needed.

On-Trajectory ODE Regression Warm-up. We initialize VA-TDM with ODE regression along the student’s own sampling trajectory. For a sampled transition (tk,tk−1)(t_{k},t_{k-1}), we roll out the student from noise to obtain xtkx_{t_{k}} and integrate the teacher ODE over the same interval using multiple solver steps:

xtk−1tea=ODESolver⁡(xtk,vtea,tk,tk−1).(23)x^{\mathrm{tea}}_{t_{k-1}}=\operatorname{ODESolver}(x_{t_{k}},v_{\mathrm{tea}},t_{k},t_{k-1}). \tag*{(23)}

We train the student transition in Eq. (4), whose output is denoted by x^tk−1\hat{x}_{t_{k-1}}, with

Lwarm=Ek,xtk∼pθ,tk[d(x^tk−1,sg⁡[xtk−1tea])],(24)L_{\mathrm{warm}}=\mathbb{E}_{k,x_{t_{k}}\sim p_{\theta,t_{k}}}\left[d\left(\hat{x}_{t_{k-1}},\operatorname{sg}\left[x^{\mathrm{tea}}_{t_{k-1}}\right]\right)\right], \tag*{(24)}

where d(⋅,⋅)d(\cdot,\cdot) is the distance metric and pθ,tkp_{\theta,t_{k}} is the student marginal at tkt_{k} and sg denotes stop-gradient. Following the connection between ODE regression and distribution matching in TDM [22], perfect pointwise matching also implies trajectory distribution alignment. Specifically, assuming exact teacher integration and that every transition is sampled with positive probability,

Lwarm=0,pθ,tK=ptea,tK⟹pθ,tk=ptea,tk,k=0,…,K,(25)\mathcal{L}_{\mathrm{warm}} = 0,\qquad p_{\theta,t_K} = p_{\mathrm{tea},t_K}\qquad\Longrightarrow\qquad p_{\theta,t_k} = p_{\mathrm{tea},t_k},\quad k = 0,\ldots,K, \tag*{(25)}

where ptea,tkp_{\mathrm{tea},t_k} denotes the teacher ODE marginal. This follows by induction: each perfectly matched transition pushes the shared current marginal to the same next marginal. Thus, ODE regression imposes a stronger pointwise constraint whose exact satisfaction also fulfills trajectory distribution matching. Therefore, this initialization directly targets the intermediate transitions and marginals optimized by VA-TDM. We subsequently switch to VAD, relaxing pointwise regression to distributional alignment.

Transition-Aware Field Estimation. In both our framework and existing score distillation work, it is crucial for the auxiliary score to accurately track the student distribution in order to achieve strong performance. However, we identify an aspect that has been consistently overlooked in prior work [16, 45]: in the few-step generation setting, student transitions with different step indices generally induce distinct noisy distributions, yet existing methods typically model them with a shared auxiliary score, which theoretically precludes the possibility of precisely tracking the student distribution. TDM tries to avoid this by assigning student transitions with different step indices to disjoint perturbation intervals, which is partially effective but sacrifices part of supervision signals. To resolve this issue, we propose that the field network should additionally receive the identity of the transition—for instance, in our setting, the target timestep of the student transition can be injected as extra input, i.e., learn hψ(xτ,τ,s)h_\psi(x_\tau,\tau,s) as online velocity. Results in Table 2 show that this modification is essential for providing accurate supervision signals, and substantially improves diversity and fine-grained rendering of images, such as text rendering.

Experiments

Models and Baselines. We distill SD3.5-Medium [3], FLUX.2-Klein-Base 4B [1] and MiniMax-H3 [25] into four-step generators. For SD3.5-Medium, the baselines include DMD2 [45], D-DMD [16], DPDMD [39], and TDM [22]. For FLUX.2 4B, we compare with the officially released four-step model.

Evaluation Protocol. We evaluate images along four complementary dimensions. Prompt following is assessed using GenEval [4] for compositional generation and OCR for visual text rendering. Image quality is measured by Aesthetic Score [32], while preference is assessed with ImageReward [40], PickScore [12], HPSv2, and HPSv3; these quality and preference metrics are evaluated on DrawBench [28]. Finally, diversity is measured by average pairwise LPIPS [47] between independently generated images by the same prompt. For audio-video generation, we conduct anonymized pairwise human comparisons of audio-visual quality and prompt alignment (Appendix § 9.§ 2).

Main Results

Quantitative Results. Table 1 compares VA-TDM with few-step baselines and their teachers on text-to-image generation. At four NFE, VA-TDM improves compositional prompt following and text rendering on both backbones. On SD3.5-Medium, it leads the compared few-step models on all verifiable, quality, and preference metrics, raising GenEval from the strongest baseline TDM’s 0.61 to 0.67 while increasing TDM’s diversity. On FLUX2, gains in prompt following and most quality and preference metrics are accompanied by an LPIPS increase from 0.47 to 0.56. Together, the results demonstrate the state-of-the-art performance of our proposed VA-TDM in few-step generation.

ModelNFEGenEvalOCRAest.ImgRwdPickScoreHPSv2HPSv3LPIPS
SD3.5-Medium (teacher, CFG=4.5)50×250 \times20.650.585.370.860.86280.28059.780.61
SD3.5-Medium (teacher, CFG=7)50×250 \times20.670.625.370.960.86400.28549.870.60
DMD240.600.535.551.000.8648<u>0.2977</u>11.180.53
D-DMD40.550.565.510.990.8567<u>0.2897</u>10.930.57
DPDMD4<u>0.62</u>0.335.280.800.85300.27078.130.59
TDM40.61<u>0.57</u><u>5.58</u><u>1.01</u><u>0.8679</u>0.2969<u>11.54</u>0.57
VA-TDM (Ours)40.670.595.651.020.87020.301011.89<u>0.58</u>
FLUX.2-Klein-Base 4B (teacher, CFG=4.0)50×250 \times20.770.605.310.850.86050.276810.280.56
FLUX.2-Klein-Base 4B (teacher, CFG=7)50×250 \times20.800.645.380.930.86200.284010.580.54
FLUX.2-Klein 4B4<u>0.78</u><u>0.64</u><u>5.28</u><u>1.04</u>0.8746<u>0.2886</u><u>11.27</u><u>0.47</u>
VA-TDM (Ours)40.800.695.431.07<u>0.8694</u>0.293011.580.56

Table 1. Table 1 | Systematic evaluation. Best / second-best few-step results per backbone are bold / underlined.

Table 1 | Systematic evaluation. Best / second-best few-step results per backbone are bold / <u>underlined</u>.

Qualitative Results. Across visual comparisons in image and video generation (Figure 1 and Figure 4), VA-TDM consistently delivers better visual quality compared with few-step baselines, significantly approaching the many-step base model. Moreover, we find that previous methods either suffer from oversaturation or excessively dark outputs, whereas VA-TDM effectively avoids these undesirable artifacts.

Figure 1. Comparison of VA-TDM-H3 and competing baselines. Our VA-TDM delivers better visual quality and suffers less oversaturation than the few-step competitor. More samples in Appendix § 10.

Figure 4. Qualitative comparison among VA-TDM and competing baselines. Printed labels: Base Model, 50 NFE; DMD2, 4 NFE; DP-DMD, 4 NFE; D-DMD, 4 NFE; TDM, 4 NFE; VA-TDM (Ours), 4 NFE. Prompts: A majestic lion with a thick mane in warm evening light, close-up wildlife portrait. A small wooden toy airplane on top of a stack of colorful books, soft studio light. Fresh strawberries and blueberries falling into a bowl of yogurt, high-speed food photograph.

Ablation Study

Ablation Study

We examine the components introduced in § 3.3 using four-step SD3.5-Medium (Table 2). Following Eq. (22), λm\lambda_m mixes the A-form and B-form games; the separate coefficient λ\lambda in Eq. (20) controls the path/flow-product mixture within each form, and λr\lambda_r weights the online field’s regularization toward the teacher velocity. Starting from λm=0.8\lambda_m=0.8, the final three rows report cumulative configurations with generator-update mixing, field regularization, and critic correction.

Dual-Form Velocity Mixing. The A-form rectified velocity and B-form Gaussian velocity in § 3.1 yield complementary results. A-form alone improves GenEval over B-form alone, but has lower OCR and HPS scores. Combining the two games produces higher OCR and ImageReward than either endpoint while retaining the A-form diversity. This supports combining their supervision.

Shared Backbone. Sharing the online-field backbone between the two forms not only reduces training cost but also improves performance. We believe that this is because the high-variance B-form is difficult to learn on its own, whereas the shared backbone allows the network to learn from both online fields jointly, which potentially leads to better internal representations and thus provides better supervision signals.

Path/Flow-Product Mixing. The mixed update notably improves GenEval and diversity, while the other metrics remain roughly unchanged. This indicates that the mixed update allows us to find a better-performing sweet spot.

Online Field Regularization and Regularization-Corrected Critic. We augment the field regression in Eq. (21) with the teacher-velocity regularizer, using λr=1\lambda_r=1. Even without critic correction, this also slightly improves GenEval, OCR, supporting the benefit of retaining teacher information during online field learning. By applying the regularization-corrected critic fψF=(1+λr)(uτ−hψF)f_\psi^{F}=(1+\lambda_r)(u_\tau-h_\psi^{F}) and adjusted loss coefficients, the resulting configuration notably improves the performance and achieves the strongest verifiable, quality, and preference scores among variants.

Transition-Aware Field Estimation. This enables accurate tracking of the student field and notably improves instruction following (GenEval), text rendering (OCR), and diversity (LPIPS).

Related Works

Training-Free Diffusion Acceleration. Training-free methods for accelerating diffusion model sampling have progressed substantially ([14, 18, 23, 33, 41, 48]). However, diffusion distillation remains essential for high-quality few-step generation.

Diffusion Distillation. Distilled models generate images from noise in one or a few steps. Two main approaches transfer knowledge from pre-trained DMs: 1) Trajectory distillation [19, 24, 29, 36, 37, 43] matches teacher ODE trajectories at the instance level, facing challenging instance-wise alignment and numerical errors in teacher PF-ODE solutions; 2) Distribution matching via score distillation [16, 20, 39, 44, 49 b] aligns student and teacher distributions using objectives such as reverse KL, Fisher divergence, and moment matching. FGM aligns velocity fields but considers only Gaussian forward processes [7], where its objective reduces to Fisher divergence [50]. TDM [22] has shown strong few-step performance by learning partial denoising at each step. Our VA-TDM unifies score distillation, Fisher divergence, and moment matching while yielding new distillation objectives. Combined with transition-aware online field learning, it achieves state-of-the-art few-step performance.

Generative adversarial networks. Early GAN research achieved strong results on domain-specific datasets [10, 11, 26, 46], while it has been effectively extended these successes to ImageNet [2, 27, 31]. However, training instability and limited scalability remain key challenges. Large-scale text-to-image GANs still rely on convolutional architectures with complex designs [9, 51], and scaling transformer-based GANs remains difficult [8, 13]. Adversarial Flow Models (AFM) [15] train many-step flow-matching models through an adversarial game in velocity space, requiring discriminator Jacobian-vector products (JVPs). Our work also formulates an adversarial game in velocity space, but targets few-step distillation and avoids expensive JVP computation.

Conclusions

We introduced Velocity Adversarial Distillation (VAD), a unified framework that guides distillation through velocity fields constructed from the student’s sampling process. By separating the choice of field from the generator update, VAD connects score distillation, Fisher-based matching, and moment matching, clarifies MMD’s heuristic updates, and derives new training objectives. Building on this framework, VA-TDM combines dual-form supervision with transition-aware online velocity learning to supervise partial denoising along few-step student trajectories, without adding inference cost. Experiments across text-to-image and text-to-video generation demonstrate VA-TDM consistently achieves state-of-the-art four-step performance. VA-TDM also scales effectively to 33B MiniMax-H3, outperforming Fast-H3 in human evaluation and substantially narrowing the perceptual quality gap to the many-step teacher. These results highlight the value of our work, establishing VAD as a principled foundation for designing effective few-step distillation methods.

Proofs of Main Results

Throughout, time variables and conditioning information are fixed. For the population game in Eq. (12), assume that the teacher field and sampled target have finite second moments, and that critic optimization ranges over all square-integrable fields under the student’s marginal law. Theorem 3 additionally assumes realizability: there exists a student θ⋆\theta^\star with vˉθ⋆,τF=uτ\bar{v}^{F}_{\theta^\star,\tau}=u_\tau almost everywhere under pθ⋆,τp_{\theta^\star,\tau}. The critic maximizes VFV_F, while the student minimizes LpathF\mathcal{L}^{F}_{\mathrm{path}} with the critic fixed.

Proof of Lemma 2

Proof. Conditioning on xτx_\tau gives the marginal payoff

VF(f,θ)=Epθ,τ[fT(uτ−vˉθ,τF)−12∥f∥2].(26)V_F(f,\theta)=\mathbb{E}_{p_{\theta,\tau}}\left[f^{\mathsf{T}}\left(u_\tau-\bar{v}^{F}_{\theta,\tau}\right)-\frac{1}{2}\lVert f\rVert^2\right]. \tag*{(26)}

Writing rθ=uτ−vˉθ,τFr_\theta=u_\tau-\bar{v}^{F}_{\theta,\tau} and completing the square yields

VF(f,θ)=12Epθ,τ∥rθ∥2−12Epθ,τ∥f−rθ∥2.(27)V_F(f,\theta)=\frac{1}{2}\mathbb{E}_{p_{\theta,\tau}}\lVert r_\theta\rVert^2-\frac{1}{2}\mathbb{E}_{p_{\theta,\tau}}\lVert f-r_\theta\rVert^2. \tag*{(27)}

The upper bound is attained exactly when f=rθf=r_\theta almost everywhere under pθ,τp_{\theta,\tau}, proving the optimal critic and value. □

Parameterizing fψF=uτ−hψFf^F_\psi=u_\tau-h^F_\psi makes critic optimization equivalent to the field regression in Eq. (21), by the conditional variance decomposition.

Proof of Proposition 3

Proof. Choose a realizable matching student θ⋆\theta^\star and the globally zero critic f⋆≡0f^\star\equiv0. For any admissible critic,

VF(f,θ⋆)=−12Epθ⋆,τ∥f∥2≤0=VF(f⋆,θ⋆),(28)V_F(f,\theta^\star)=-\frac{1}{2}\mathbb{E}_{p_{\theta^\star,\tau}}\lVert f\rVert^2\leq0=V_F(f^\star,\theta^\star), \tag*{(28)}

so f⋆f^\star is a best response. With this critic, LpathF(θ;f⋆)=0\mathcal{L}^{F}_{\mathrm{path}}(\theta;f^\star)=0 for every student, so θ⋆\theta^\star is also a best response. Thus (θ⋆,f⋆)(\theta^\star,f^\star) is a Nash equilibrium. By Theorem 2, the maximized critic payoff is nonnegative and vanishes at θ⋆\theta^\star, establishing the claimed optimality. □

Proof of Lemma 4

Our proof uses the conditional-projection principle underlying denoising score matching [38] and the score-projection identity used in SiD [50] and SIM [21]. FGM [7] develops the corresponding flow-product identity. Here the argument applies to conditional target fields under the student’s composite sampling law, without requiring a Gaussian forward kernel.

Assume differentiable marginal densities and conditional fields near θ0\theta_0, integrability, and domination sufficient to interchange differentiation and integration. Reparameterized derivatives are assumed to exist and commute with expectation. Input laws and time variables are held fixed during differentiation; auxiliary parameters are frozen while their input derivatives are retained. Proof. Let PθP_\theta be the joint sampling law, write X=xτX=x_\tau and VθF=VτF(xτ∣xs,x0)V^F_\theta=V^F_\tau(x_\tau\mid x_s,x_0), and define Vˉθ,τF(x)=EPθ[VθF∣X=x]\bar{V}^F_{\theta,\tau}(x)=\mathbb{E}_{P_\theta}[V^F_\theta\mid X=x]. For any admissible test field aθa_\theta, conditional expectation gives

EPθ[aθ(X)T(VθF−Vˉθ,τF(X))]=0.(29)\mathbb{E}_{P_\theta}\left[a_\theta(X)^\mathsf{T}\left(V^F_\theta-\bar{V}^F_{\theta,\tau}(X)\right)\right]=0. \tag*{(29)}

Fix θ0\theta_0, set h0F=Vˉθ0,τFh^F_0=\bar{V}^F_{\theta_0,\tau}, and hold f0f_0 and h0Fh^F_0 fixed as functions. The projection identity gives

H(θ;θ0)=EPθ[f0(X)T(VθF−h0F(X))](30)H(\theta;\theta_0)=\mathbb{E}_{P_\theta}\left[f_0(X)^\mathsf{T}\left(V^F_\theta-h^F_0(X)\right)\right] \tag*{(30)}
=∫pθ,τ(x)f0(x)T(Vˉθ,τF(x)−h0F(x)) dx.(31)=\int p_{\theta,\tau}(x)f_0(x)^\mathsf{T}\left(\bar{V}^F_{\theta,\tau}(x)-h^F_0(x)\right)\,\mathrm{d}x. \tag*{(31)}

At θ=θ0\theta=\theta_0, the centered difference vanishes pointwise, so its product with the sampling-density derivative is zero. Hence

∇θH(θ;θ0)∣θ0=Epθ0,τ[f0(x)T∂θVˉθ,τF(x)∣θ0],(32)\left.\nabla_\theta H(\theta;\theta_0)\right|_{\theta_0} =\mathbb{E}_{p_{\theta_0,\tau}}\left[\left.f_0(x)^\mathsf{T}\partial_\theta\bar{V}^F_{\theta,\tau}(x)\right|_{\theta_0}\right], \tag*{(32)}

where ∂θ\partial_\theta differentiates at a fixed spatial argument. Since GfpF=−HG^F_{\mathrm{fp}}=-H, this is precisely

∇θGfpF∣θ0=∇θGmeanF(θ;θ0,f0)∣θ0,(33)\left.\nabla_\theta G^F_{\mathrm{fp}}\right|_{\theta_0} =\left.\nabla_\theta G^F_{\mathrm{mean}}(\theta;\theta_0,f_0)\right|_{\theta_0}, \tag*{(33)}

whose right-hand side holds the outer measure fixed at pθ0,τp_{\theta_0,\tau}. The argument allows both the target and the joint law to depend on θ\theta, and applies to both A-form and B-form. □

The practical mixed update in Eq. (20) uses learned auxiliary fields and detaches the critic output while retaining input gradients through the centering field.

Proof of Theorem 5

Fix 0≤s<t≤10\le s<t\le1, s<τ<1s<\tau<1, and η=1\eta=1. Let PθP_\theta denote the law of the local reparameterized chain

x^0=gθ(xt,t),xs=stxt+(1−st)x^0,xτ=ατ∣sxs+βτ∣sε,(34)\hat{x}_0=g_\theta(x_t,t),\qquad x_s=\frac{s}{t}x_t+\left(1-\frac{s}{t}\right)\hat{x}_0,\qquad x_\tau=\alpha_{\tau\mid s}x_s+\beta_{\tau\mid s}\varepsilon, \tag*{(34)}

with fixed input law and independent standard Gaussian ε\varepsilon. Use the regularity conditions of Appendix § 7.§ 3, an exact teacher, and finite reverse KL with the boundary conditions needed for its pathwise gradient. Write DθD_\theta for total sample derivatives with base noises fixed, and ∂θ\partial_\theta for field derivatives at fixed spatial arguments. All gradient identities are evaluated at a reference student θˉ\bar{\theta}, using the exact reference field vˉθˉ,τF\bar{v}^F_{\bar{\theta},\tau} and critic uτ−vˉθˉ,τFu_\tau-\bar{v}^F_{\bar{\theta},\tau}. For a teacher-regularized regressor, the unregularized reference field is (1+λr)hψ⋆F−λruτ(1+\lambda_r)h^F_{\psi^\star}-\lambda_r u_\tau.

Proof. (i) Matching. For A-form, conditioning the target on xτx_\tau gives

vˉθ,τA(x)=x−mθ(x)τ,fA⋆(x)=mθ(x)−mq(x)τ,(35)\bar{v}^A_{\theta,\tau}(x)=\frac{x-m_\theta(x)}{\tau},\qquad f^\star_A(x)=\frac{m_\theta(x)-m_q(x)}{\tau}, \tag*{(35)}

with the moments defined in Eq. (18). Thus zero residual is equivalent to conditional-moment matching. For B-form, Gaussian differentiation gives E[ϵ∣xτ=x]=−βτ∣s∇log⁡pθ,τ(x)\mathbb{E}[\boldsymbol{\epsilon}\mid\mathbf{x}_{\tau}=\mathbf{x}]=-\beta_{\tau\mid s}\nabla\log p_{\theta,\tau}(\mathbf{x}). Using the coefficients of Eq. (5), with α=ατ∣s\alpha=\alpha_{\tau\mid s} and β=βτ∣s\beta=\beta_{\tau\mid s}, yields

vˉθ,τB(x)=α˙αx−(ββ˙−α˙αβ2)∇log⁡pθ,τ(x)=−x1−τ−τ1−τ∇log⁡pθ,τ(x),fB⋆=τ1−τ(∇log⁡pθ,τ−∇log⁡qτ).(36)\begin{aligned} \bar{\mathbf{v}}_{\theta,\tau}^{B}(\mathbf{x})&=\frac{\dot{\alpha}}{\alpha}\mathbf{x}-\left(\beta\dot{\beta}-\frac{\dot{\alpha}}{\alpha}\beta^{2}\right)\nabla\log p_{\theta,\tau}(\mathbf{x})\\ &=-\frac{\mathbf{x}}{1-\tau}-\frac{\tau}{1-\tau}\nabla\log p_{\theta,\tau}(\mathbf{x}),\\ \mathbf{f}_{B}^{\star}&=\frac{\tau}{1-\tau}\left(\nabla\log p_{\theta,\tau}-\nabla\log q_{\tau}\right). \tag*{(36)} \end{aligned}

The kernel maps qsq_s to qτq_\tau, since ατ∣s(1−s)=1−τ\alpha_{\tau\mid s}(1-s)=1-\tau and ατ∣s2s2+βτ∣s2=τ2\alpha_{\tau\mid s}^{2}s^{2}+\beta_{\tau\mid s}^{2}=\tau^{2}. Gaussian smoothing gives positive densities on Rd\mathbb{R}^{d}, so equal scores imply equal normalized densities. Since ατ∣s>0\alpha_{\tau\mid s}>0 and the Gaussian characteristic function never vanishes, deconvolution makes pθ,τ=qτp_{\theta,\tau}=q_\tau equivalent to pθ,s=qsp_{\theta,s}=q_s.

(ii) The clean endpoint. At s=0s=0, xs=x^0\mathbf{x}_s=\hat{\mathbf{x}}_0 and xτ=(1−τ)x^0+τzτ\mathbf{x}_\tau=(1-\tau)\hat{\mathbf{x}}_0+\tau\mathbf{z}_\tau, giving

vτA=xτ−x^0τ=zτ−x^0=vτB.(37)\mathbf{v}_{\tau}^{A}=\frac{\mathbf{x}_\tau-\hat{\mathbf{x}}_0}{\tau}=\mathbf{z}_\tau-\hat{\mathbf{x}}_0=\mathbf{v}_{\tau}^{B}. \tag*{(37)}

The sampled targets coincide, so their conditional means, optimal critics, and gradients under identical differentiation conventions coincide.

(iii) Pathwise updates with the critic output detached. Set a=1−s/t>0a=1-s/t>0 and c=ατ∣sa∈(0,1)c=\alpha_{\tau\mid s}a\in(0,1). From Eq. (34),

Dθxs=aDθx^0,Dθxτ=cDθx^0,(38)D_{\theta}\mathbf{x}_s=aD_{\theta}\hat{\mathbf{x}}_0,\qquad D_{\theta}\mathbf{x}_\tau=cD_{\theta}\hat{\mathbf{x}}_0, \tag*{(38)}

and the sampled targets satisfy

DθvτA=−1−cτDθx^0,DθvτB=−a1−sDθx^0.(39)D_{\theta}\mathbf{v}_{\tau}^{A}=-\frac{1-c}{\tau}D_{\theta}\hat{\mathbf{x}}_0,\qquad D_{\theta}\mathbf{v}_{\tau}^{B}=-\frac{a}{1-s}D_{\theta}\hat{\mathbf{x}}_0. \tag*{(39)}

The reverse-KL pathwise identity is

∇θKL(pθ,τ∥qτ)=E[(Dθxτ)T(∇log⁡pθ,τ−∇log⁡qτ)(xτ)].(40)\nabla_{\theta}\mathrm{KL}(p_{\theta,\tau}\mathbin{\|}q_\tau)=\mathbb{E}\left[(D_{\theta}\mathbf{x}_\tau)^{\mathsf{T}}(\nabla\log p_{\theta,\tau}-\nabla\log q_\tau)(\mathbf{x}_\tau)\right]. \tag*{(40)}

Substituting the optimal critics into Eq. (19) gives

∇θLpath,sgA=1−cτ2E[(Dθx^0)T(mθ−mq)(xτ)],(41)\nabla_{\theta}\mathcal{L}_{\mathrm{path,sg}}^{A}=\frac{1-c}{\tau^{2}}\mathbb{E}\left[(D_{\theta}\hat{\mathbf{x}}_0)^{\mathsf{T}}(\mathbf{m}_{\theta}-\mathbf{m}_{q})(\mathbf{x}_\tau)\right], \tag*{(41)}
∇θLpath,sgB=a1−sE[(Dθx^0)TfB⋆]=τ(1−τ)2∇θKL(pθ,τ∥qτ).(42)\nabla_{\theta}\mathcal{L}_{\mathrm{path,sg}}^{B}=\frac{a}{1-s}\mathbb{E}\left[(D_{\theta}\hat{\mathbf{x}}_0)^{\mathsf{T}}\mathbf{f}_{B}^{\star}\right]=\frac{\tau}{(1-\tau)^{2}}\nabla_{\theta}\mathrm{KL}(p_{\theta,\tau}\mathbin{\|}q_\tau). \tag*{(42)}

The A-form expectation is the MMD surrogate update made explicit below; the B-form gives reverse KL. Both coefficients are positive time weights.

(iv) Flow-product updates without output detachment. Define the discrepancy with its outer measure held fixed:

DF(θ;θˉ)=12Epθˉ,τ∥uτ−vˉθ,τF∥2.(43)\mathcal{D}_{F}(\theta;\bar{\theta})=\frac{1}{2}\mathbb{E}_{p_{\bar{\theta},\tau}}\left\|\mathbf{u}_{\tau}-\bar{\mathbf{v}}_{\theta,\tau}^{F}\right\|^{2}. \tag*{(43)}

Theorem 4, applied to the optimal reference critic, gives

∇θGfpF∣θˉ=−Epθˉ,τ[(∂θvˉθ,τF∣θˉ)TfF⋆]=∇θDF(θ;θˉ)∣θˉ.(44)\nabla_{\theta}\mathcal{G}_{\mathrm{fp}}^{F}\big|_{\bar{\theta}}=-\mathbb{E}_{p_{\bar{\theta},\tau}}\left[\left(\partial_{\theta}\bar{\mathbf{v}}_{\theta,\tau}^{F}\big|_{\bar{\theta}}\right)^{\mathsf{T}}\mathbf{f}_{F}^{\star}\right]=\left.\nabla_{\theta}\mathcal{D}_{F}(\theta;\bar{\theta})\right|_{\bar{\theta}}. \tag*{(44)}

The residual identities in part (i) identify the objectives as

DA(θ;θˉ)=12τ2Epθˉ,τ∥mθ−mq∥2,(45)\mathcal{D}_{A}(\theta;\bar{\theta})=\frac{1}{2\tau^{2}}\mathbb{E}_{p_{\bar{\theta},\tau}}\left\|\mathbf{m}_{\theta}-\mathbf{m}_{q}\right\|^{2}, \tag*{(45)}
DB(θ;θˉ)=τ22(1−τ)2Epθˉ,τ∥∇log⁡pθ,τ−∇log⁡qτ∥2.(46)\mathcal{D}_{B}(\theta;\bar{\theta})=\frac{\tau^{2}}{2(1-\tau)^{2}}\mathbb{E}_{p_{\bar{\theta},\tau}}\left\|\nabla\log p_{\theta,\tau}-\nabla\log q_\tau\right\|^{2}. \tag*{(46)}

These are the conditional-moment and scaled Fisher discrepancies, respectively, with the fixed-outer-measure gradient convention.

MMD objective and surrogate. To make the MMD correspondence explicit [30], write δθ=mθ−mq\delta_\theta= m_\theta- m_q. Its moment objective and local fixed-measure version are

JMM(θ)=12Epθ,τ∥δθ∥2,LMM(θ;θˉ)=12Epθˉ,τ∥δθ∥2=τ2DA(θ;θˉ).(47)J_{\mathrm{MM}}(\theta) = \frac{1}{2}\mathbb{E}_{p_{\theta,\tau}}\|\delta_\theta\|^2,\qquad L_{\mathrm{MM}}(\theta;\bar{\theta}) = \frac{1}{2}\mathbb{E}_{p_{\bar{\theta},\tau}}\|\delta_\theta\|^2 = \tau^2\mathcal{D}_A(\theta;\bar{\theta}). \tag*{(47)}

The exact local gradient is

gMM=∇θLMM∣θˉ=Epθˉ,τ[(∂θmθ∣θˉ)Tδθˉ].(48)\mathbf{g}_{\mathrm{MM}} = \left.\nabla_\theta L_{\mathrm{MM}}\right|_{\bar{\theta}} = \mathbb{E}_{p_{\bar{\theta},\tau}}\left[\left(\left.\partial_\theta m_\theta\right|_{\bar{\theta}}\right)^{\mathsf{T}}\delta_{\bar{\theta}}\right]. \tag*{(48)}

MMD approximates the implicit conditional-mean derivative by the sampled clean-prediction derivative, giving

gMM≈gsur≔EPθˉ[(Dθx^0∣θˉ)Tδθˉ(xτ)].(49)\mathbf{g}_{\mathrm{MM}} \approx\mathbf{g}_{\mathrm{sur}} \coloneqq\mathbb{E}_{P_{\bar{\theta}}}\left[\left(\left.D_\theta\hat{x}_0\right|_{\bar{\theta}}\right)^{\mathsf{T}}\delta_{\bar{\theta}}(x_\tau)\right]. \tag*{(49)}

This is the gradient of the detached surrogate

Lsur(θ)=EPθ[x^0Tsg⁡(δθ(xτ))].(50)L_{\mathrm{sur}}(\theta) = \mathbb{E}_{P_\theta}\left[\hat{x}_0^{\mathsf{T}}\operatorname{sg}\left(\delta_\theta(x_\tau)\right)\right]. \tag*{(50)}

Parts (iii)–(iv) therefore give both correspondences directly:

τ2∇θGfpA∣θˉ=gMM,(51)\left.\tau^2\nabla_\theta\mathcal{G}_{\mathrm{fp}}^A\right|_{\bar{\theta}} = \mathbf{g}_{\mathrm{MM}}, \tag*{(51)}
∇θLpath,sgA∣θˉ=1−cτ2gsur.(52)\left.\nabla_\theta\mathcal{L}_{\mathrm{path,sg}}^A\right|_{\bar{\theta}} = \frac{1-c}{\tau^2}\mathbf{g}_{\mathrm{sur}}. \tag*{(52)}

Thus flow-product recovers the exact local moment gradient, while the detached pathwise update recovers MMD’s surrogate direction up to time weighting. □

Supporting Derivations

Reference DSM Variance

Consider the reference Gaussian forward process with 0<s<τ<10 < s < \tau< 1, xs=(1−s)x0+sesx_s = (1-s)x_0 + s e_s, and xτ=ατ∣sxs+βτ∣seτx_\tau= \alpha_{\tau\mid s}x_s + \beta_{\tau\mid s}e_\tau, where the standard Gaussian noises are independent of each other and x0x_0. The two DSM targets

Ss→τ=−xτ−ατ∣sxsβτ∣s2,S0→τ=−xτ−(1−τ)x0τ2(53)S_{s\to\tau} = -\frac{x_\tau-\alpha_{\tau\mid s}x_s}{\beta_{\tau\mid s}^{2}},\qquad S_{0\to\tau} = -\frac{x_\tau-(1-\tau)x_0}{\tau^2} \tag*{(53)}

have the same conditional mean ∇log⁡qτ(xτ)\nabla\log q_\tau(x_\tau). The Gaussian bridge and the law of total covariance give the comparison of Xue et al. [42] in our notation:

Cov⁡(Ss→τ∣xτ)=Cov⁡(S0→τ∣xτ)+(1βτ∣s2−1τ2)I.(54)\operatorname{Cov}(S_{s\to\tau}\mid x_\tau) = \operatorname{Cov}(S_{0\to\tau}\mid x_\tau) + \left(\frac{1}{\beta_{\tau\mid s}^{2}}-\frac{1}{\tau^2}\right)I. \tag*{(54)}

The excess is positive for 0<s<τ<10 < s < \tau< 1 and vanishes at s=0s = 0. This reference-process comparison motivates the noisy-target DSM discussion in § 3.1.

Clean-Target and Path-Marginal Score Mismatch

Fix tt, ss, τ\tau with s<τ<1s < \tau< 1 and the input law. Let Rθ(dxs,dx^0)R_\theta(\mathrm{d}x_s,\mathrm{d}\hat{x}_0) be the joint law of the intermediate output and clean anchor. The two perturbed marginals are

q(xτ∣x^0)=N(xτ;(1−τ)x^0,τ2I),pθ,τclean(xτ)=∫q(xτ∣x^0)Rθ(dxs,dx^0),pθ,τpath(xτ)=∫q(xτ∣xs)Rθ(dxs,dx^0).(55)\begin{aligned} q(x_\tau\mid\hat{x}_0) &= \mathcal{N}(x_\tau; (1-\tau)\hat{x}_0,\tau^2 I),\\ p^{\mathrm{clean}}_{\theta,\tau}(x_\tau) &= \int q(x_\tau\mid\hat{x}_0)R_\theta(\mathrm{d}x_s,\mathrm{d}\hat{x}_0),\\ p^{\mathrm{path}}_{\theta,\tau}(x_\tau) &= \int q(x_\tau\mid x_s)R_\theta(\mathrm{d}x_s,\mathrm{d}\hat{x}_0). \tag*{(55)} \end{aligned}

TDM’s ideal importance-weighted clean-target objective [22] uses the ratio of these conditional kernels. Since both kernels have positive densities, for any integrable loss ℓψ\ell_\psi,

w(xτ,xs,x^0)=q(xτ∣x^0)q(xτ∣xs),(56)w(x_\tau,x_s,\hat{x}_0)=\frac{q(x_\tau\mid\hat{x}_0)}{q(x_\tau\mid x_s)}, \tag*{(56)}
ERθq(xτ∣xs)[wℓψ]=ERθq(xτ∣x^0)[ℓψ].(57)\mathbb{E}_{R_\theta q(x_\tau\mid x_s)}[w\ell_\psi]=\mathbb{E}_{R_\theta q(x_\tau\mid\hat{x}_0)}[\ell_\psi]. \tag*{(57)}

For ℓψ=ωτ∥hψ(xτ,τ)−x^0∥2\ell_\psi=\omega_\tau\lVert h_\psi(x_\tau,\tau)-\hat{x}_0\rVert^2 with ωτ>0\omega_\tau>0, an unrestricted regressor with finite second moments has optimum h∗(xτ,τ)=ERθq(xτ∣x^0)[x^0∣xτ]h^*(x_\tau,\tau)=\mathbb{E}_{R_\theta q(x_\tau\mid\hat{x}_0)}[\hat{x}_0\mid x_\tau]. Gaussian differentiation identifies its score and the required path score as

∇log⁡pθ,τclean(xτ)=−xτ−(1−τ)h∗(xτ,τ)τ2,\nabla\log p^{\mathrm{clean}}_{\theta,\tau}(x_\tau)=-\frac{x_\tau-(1-\tau)h^*(x_\tau,\tau)}{\tau^2},
∇log⁡pθ,τpath(xτ)=ERθq(xτ∣xs)[xτ−ατ∣sxsβτ∣s2∣xτ].(58)\nabla\log p^{\mathrm{path}}_{\theta,\tau}(x_\tau)=\mathbb{E}_{R_\theta q(x_\tau\mid x_s)}\left[\left.\frac{x_\tau-\alpha_{\tau\mid s}x_s}{\beta^2_{\tau\mid s}}\right|x_\tau\right]. \tag*{(58)}

Thus importance weighting recovers the clean-denoising objective, whose population score can differ from the path score needed by the generator. For example, in one dimension with x^0=ϵ^=0\hat{x}_0=\hat{\epsilon}=0 and η=1\eta=1, the transition gives xs=0x_s=0: the path marginal is N(0,βτ∣s2)\mathcal{N}(0,\beta^2_{\tau\mid s}) and the clean marginal is N(0,τ2)\mathcal{N}(0,\tau^2), with different variances for s>0s>0. At s=0s=0, xs=x^0x_s=\hat{x}_0 and the two marginals coincide.

Experimental Details

This section provides the training details for the experiments in § 4.

Training Configurations

VA-TDM trains a generator and an online auxiliary field while keeping the teacher fixed. The generator combines the two velocity forms through Eq. (22), and the auxiliary field follows the regression and regularization objectives in § 3.3. By default, we apply LoRA fine-tuning with rank 64. We update the online score and generator once each iteration. We use the AdamW optimizer and set (β1,β2)=(0.,0.95)(\beta_1,\beta_2)=(0.,0.95).

User Study

The study in Figure 2 compares MiniMax-H3 with Fast-H3, VA-TDM with MiniMax-H3, and VA-TDM with Fast-H3. We carried out a user study in which participants were shown two anonymized audio-videos, each produced by a different model, and were asked to pick the one with superior audio-visual quality and closer alignment with the prompt. For audio-video generation, we randomly chose 10 prompts, and every generated audio-video was manually checked to confirm it contained no inappropriate or harmful content.

Additional Visual Samples

Figs. 5 to 9 provide additional text-to-video comparisons complementing Figure 1. Each figure presents six selected frames per method, with the summarized prompt shown above the panels. From top to bottom, the rows correspond to MiniMax-H3 (100 NFE), Fast-H3 (4 NFE), and VA-TDM-H3 (4 NFE). The examples cover human–animal interactions, clothing changes, fantasy combat, stylized game scenes, and multi-scene cinematic sequences.

Figure 5. Summarized Prompt: A figure wearing a rabbit mask and pastel clothes walks and sprints beside a white rhino in spiked blue armor, charging the camera before stopping together as desert dust settles. Figure 5 | Additional visual comparison of MiniMax-H3 (100 NFE), Fast-H3 (4 NFE), and VA-TDM-H3 (4 NFE).

Figure 6. Summarized Prompt: In one continuous backward tracking shot, a walking model’s clothes morph through suits, silk dresses, denim, and coats amid passing buses, preserving her identity before she stops. Figure 6 | Additional visual comparison of MiniMax-H3 (100 NFE), Fast-H3 (4 NFE), and VA-TDM-H3 (4 NFE).

Figure 7. Additional visual comparison of MiniMax-H3 (100 NFE), Fast-H3 (4 NFE), and VA-TDM-H3 (4 NFE).

Figure 8. Summarized Prompt: Six chibi characters battle on a floating arena using cards, a golden beast, and shields, displaying “-120”, “-128”, “-150” damage as “TURN 1” becomes “TURN 2”, with “END TURN” visible. Additional visual comparison of MiniMax-H3 (100 NFE), Fast-H3 (4 NFE), and VA-TDM-H3 (4 NFE).

Figure 9. Summarized Prompt: Robots assemble a sports car around a man before neon drifting, wet smartphone camera closeups, and a showroom portrait distorted by a glass lens, ending with “Created with MiniMax H3”. Additional visual comparison of MiniMax-H3 (100 NFE), Fast-H3 (4 NFE), and VA-TDM-H3 (4 NFE).

References

  1. [1]Black Forest Labs. FLUX.2 [klein]. https://bfl.ai/models/flux-2-klein, 2026.
  2. [2]Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.arxiv.org/abs/1809.11096
  3. [3]Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yash Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024. URL https://arxiv.org/abs/2403.03206.
  4. [4]Dhruba Ghosh, Hanneah Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132–52152, 2023.arxiv.org/abs/2310.11513
  5. [5]Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  6. [6]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.arxiv.org/abs/2006.11239
  7. [7]Zemin Huang, Zhengyang Geng, Weijian Luo, and Guo jun Qi. Flow generator matching, 2024. URL https://arxiv.org/abs/2410.19310.
  8. [8]Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34: 14745–14758, 2021.
  9. [9]Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10124–10134, 2023.arxiv.org/abs/2303.05511
  10. [10]Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.arxiv.org/abs/1912.04958
  11. [11]Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852–863, 2021.arxiv.org/abs/2106.12423
  12. [12]Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems, 36:36652–36663, 2023.arxiv.org/abs/2305.01569
  13. [13]Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. arXiv preprint arXiv:2107.04589, 2021.arxiv.org/abs/2107.04589
  14. [14]Tiancheng Li, Weijian Luo, Zhiyang Chen, Liyuan Ma, and Guo-Jun Qi. Self-guidance: Boosting flow and diffusion generation on their own. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.arxiv.org/abs/2412.05827
  15. [15]Shanchuan Lin, Ceyuan Yang, Zhijie Lin, Hao Chen, and Haoqi Fan. Adversarial flow models, 2026. URL https://arxiv.org/abs/2511.22475.
  16. [16]Dongyang Liu, Peng Gao, David Liu, Ruoyi Du, Zhen Li, Qilong Wu, Xin Jin, Sihan Cao, Shifeng Zhang, Steven Hoi, and Hongsheng Li. Decoupled DMD: CFG augmentation as the spear, distribution matching as the shield. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=jBztv0iCKE.arxiv.org/abs/2511.22677
  17. [17]Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2022.arxiv.org/abs/2209.03003
  18. [18]Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787, 2022.arxiv.org/abs/2206.00927
  19. [19]Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021.arxiv.org/abs/2101.02388
  20. [20]Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2023.arxiv.org/abs/2305.18455
  21. [21]Weijian Luo, Zemin Huang, Zhengyang Geng, J Zico Kolter, and Guo-Jun Qi. One-step diffusion distillation through score implicit matching. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=ogk236hsJM.arxiv.org/abs/2410.16794
  22. [22]Yihong Luo, Tianyang Hu, Jiacheng Sun, Yujun Cai, and Jing Tang. Learning few-step diffusion models by trajectory distribution matching. arXiv preprint arXiv:2503.06674, 2025.arxiv.org/abs/2503.06674
  23. [23]Jiajun Ma, Shuchen Xue, Tianyang Hu, Wenjia Wang, Zhaoqiang Liu, Zhenguo Li, Zhi-Ming Ma, and Kenji Kawaguchi. The surprising effectiveness of skip-tuning in diffusion sampling. arXiv preprint arXiv:2402.15170, 2024.arxiv.org/abs/2402.15170
  24. [24]Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14297–14306, June 2023.arxiv.org/abs/2210.03142
  25. [25]MiniMax. MiniMax-H3. Hugging Face model card, 2026. URL https://huggingface.co/MiniMaxAI/MiniMax-H3. Accessed: 2026-09-26.
  26. [26]Scott Reed, Zeynep Akata, Xinchen Yan, Lajianlugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pages 1060–1069. PMLR, 2016.arxiv.org/abs/1605.05396
  27. [27]Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.
  28. [28]Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479–36494, 2022.arxiv.org/abs/2205.11487
  29. [29]Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2202.00512
  30. [30]Tim Salimans, Thomas Mensink, Jonathan Heek, and Emiel Hoogeboom. Multistep distillation of diffusion models via moment matching, 2024. URL https://arxiv.org/abs/2406.04103.
  31. [31]Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1–10, 2022.arxiv.org/abs/2202.00273
  32. [32]Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  33. [33]Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4733–4743, 2024.arxiv.org/abs/2309.11497
  34. [34]Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  35. [35]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=St1giarCHLP.arxiv.org/abs/2010.02502
  36. [36]Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=WNzy9bRDvG.arxiv.org/abs/2310.14189
  37. [37]Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.arxiv.org/abs/2303.01469
  38. [38]Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7):1661–1674, 2011.DOI
  39. [39]Tianhe Wu, Ruibin Li, Lei Zhang, and Kede Ma. Diversity-preserved distribution matching distillation for fast visual synthesis. arXiv preprint arXiv:2602.03139, 2026.
  40. [40]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to-image generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 15903–15935, 2023.arxiv.org/abs/2304.05977
  41. [41]Shuchen Xue, Zhaoqiang Liu, Fei Chen, Shifeng Zhang, Tianyang Hu, Enze Xie, and Zhenguo Li. Accelerating diffusion sampling with optimized time steps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8292–8301, 2024.arxiv.org/abs/2402.17376
  42. [42]Shuchen Xue, Chongjian Ge, Shilong Zhang, Yichen Li, and Zhi-Ming Ma. Advantage weighted matching: Aligning rl with pretraining in diffusion models, 2025. URL https://arxiv.org/abs/2509.25050.
  43. [43]Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and-play accelerator. arXiv preprint arXiv:2405.07510, 2024.arxiv.org/abs/2405.07510
  44. [44]Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828, 2023.arxiv.org/abs/2311.18828
  45. [45]Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T. Freeman. Improved distribution matching distillation for fast image synthesis, 2024. URL https://arxiv.org/abs/2405.14867.
  46. [46]Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 5907–5915, 2017.arxiv.org/abs/1612.03242
  47. [47]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.arxiv.org/abs/1801.03924
  48. [48]Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36:49842–49869, 2023.arxiv.org/abs/2302.04867
  49. [49]Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, and Hai Huang. Adversarial score identity distillation: Rapidly surpassing the teacher in one step, 2024a. URL https://arxiv.org/abs/2410.14919.
  50. [50]Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In International Conference on Machine Learning, 2024b.arxiv.org/abs/2404.04057
  51. [51]Jiapeng Zhu, Ceyuan Yang, Kecheng Zheng, Yinghao Xu, Zifan Shi, Yifei Zhang, Qifeng Chen, and Yujun Shen. Exploring sparse moe in gans for text-conditioned image synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18411–18423, 2025.arxiv.org/abs/2309.03904

Paper details

vlnc:W_afepzp7d
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 12

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents