betaFeedback

WorksComputer Science

FisherDistill: Diffusion Distribution Distillation via Closed-Form Fisher Critics

Abstract

Recent advances in diffusion distillation have enabled high-quality image generation in only one or a few network function evaluations (NFEs). Despite their efficiency, aggressively distilled generators often retain a residual distribution gap from their multi-step teachers, motivating recent methods to introduce adversarial refinement with a learned discriminator. In this work, we ask whether such refinement necessarily requires a learned adversarial discriminator. We introduce FisherDistill, a simple alternative based on Fisher linear discriminant analysis. Using intermediate representations from the frozen teacher, we estimate the first- and second-order statistics of real and generated samples and derive a closed-form Fisher direction that maximally separates their feature distributions relative to within-class variation. This direction provides a lightweight refinement signal that complements the original distillation objective without requiring an additional adversarial discriminator or its optimization loop. Across multiple distillation frameworks, our approach consistently improves generation quality over their respective distilled baselines and substantially closes the gap to adversarially refined counterparts. Code: https://bhosalems.github.io/FisherDistill/

Introduction

Diffusion-based generative models [10, 17, 28, 37, 38, 39, 48] have emerged as a powerful paradigm for high-fidelity content generation, achieving remarkable performance across images [10, 37, 39], videos [18, 19], audio [29], and scientific domains [2, 52]. Recent architectural, scaling, and formulation advances have further improved sample quality and realism [10, 37]. However, high-quality generation still requires iterative sampling with repeated network evaluations, incurring a high inference cost [17, 48]. Training-free methods reduced this cost through shorter deterministic

trajectories [5, 30], while consistency models enabled direct few-step generation by learning mappings across the diffusion trajectory [31, 50]. Building on these advances, diffusion distillation compresses the sampling trajectory of a pretrained diffusion model into a one- or few-step generator [25, 60, 61, 65]. In particular, recent distribution-matching [60, 61] and score-identity distillation [65, 67] approaches have demonstrated that such aggressively distilled generators can retain much of the quality and diversity of their multi-step teachers, making few-step generation an increasingly practical alternative to iterative sampling.

Despite this progress, aggressive one- or few-step distillation can leave a residual distribution gap from the multi-step teacher (Figure 1). Recent work shows that adding adversarial supervision to an already distilled generator improves sample quality [25, 60, 67]. This motivates a coarse-to-fine view of distillation: a base distillation objective first learns most of the teacher distribution, while adversarial supervision refines the remaining discrepancies. When used for such refinement, however, adversarial supervision requires a jointly learned discriminator and an additional min-max objective, inheriting the complexity and limited scalability of adversarial training [6, 53].

Figure 1. Distribution gap in frozen teacher features on ImageNet-64. One-step DMD2 w/o GAN drifts away from the multi-step teacher, while our closed-form Fisher refinement brings it back to the teacher. Features are held-out Fisher projections of EDM-teacher. Dashed lines mark means.

This limitation raises a natural question of whether closing the residual distribution gap requires a learned adversarial discriminator. We show that it does not (Figure 1 and Figure 2). Instead, we exploit the frozen teacher’s representation space and estimate the first- and second-order statistics of real and generated features. From these statistics, we derive a closed-form Fisher critic that identifies the direction of maximal separation between the two feature distributions relative to their within-class variation. This direction provides a lightweight refinement signal that complements the original distillation objective, requiring neither a learned discriminator nor adversarial optimization. Our approach thus retains the coarse-to-fine principle while replacing learned adversarial refinement with a closed-form Fisher critic.

1024×1024 samples generated by our 2-step FisherDistill

Figure 2. 1024×10241024 \times1024 samples generated by our 2-step FisherDistill built on 1.x-Distill Stage-I [25] and distilled from SD3-Medium [10].

Our main contributions are summarized as follows:

  1. We introduce FisherDistill, a non-adversarial refinement method that replaces a learned discriminator with a closed-form Fisher critic computed from frozen-teacher feature statistics.

  1. Across multiple distillation frameworks [25, 60, 66], FisherDistill consistently improves the distilled generators and outperforms adversarial refinement in most settings, while requiring substantially fewer refinement samples.

Related Work

Few-step diffusion and flow distillation. Reducing the inference cost of diffusion and flow models has motivated many few-step distillation methods. Consistency-based approaches learn trajectory-consistent mappings [21, 50], while rCM [64] adds distribution-matching regularization to improve fidelity. Distribution-matching methods align the student and target distributions using score-estimated KL objectives [32, 61] or the Fisher divergence [65], or match intermediate trajectory distributions [33]. These ideas also extend to flow-based generators [4, 8, 11, 40, 66]. Despite these advances, aggressive distillation can leave a quality gap relative to multi-step generation [1].

Adversarial refinement. Adversarial objectives further improve few-step generators. ADD [45] combines score distillation with an adversarial objective built on a frozen pretrained feature extractor and trainable discriminator heads, while LADD [44] uses generative features of a pretrained latent diffusion model. SDXL-Lightning [41] combines adversarial training with progressive distillation, and related approaches adversarially train or post-train diffusion models for one-step image [58] and video [26] generation. Such objectives have also been incorporated into distribution-matching [60] and score-identity [67] distillation, while 1.x-Distill [25] follows distribution matching with an adversarial stage. Although effective, these approaches rely on a learned discriminator and min-max optimization, which adds optimization and loss-balancing complexity [6, 11, 25, 60, 67].

Alternatives to adversarial refinement. RealUID [23] incorporates real-data supervision without a GAN discriminator, but retains a learned auxiliary matching function within min-max optimization. TwinFlow [6] avoids an external discriminator by modifying the distillation framework to learn self-adversarial trajectories within a single model. In contrast, FisherDistill is an auxiliary refinement objective that derives its direction in closed form from real and generated statistics of frozen teacher features Figure 3.

Figure 3. Overview of FisherDistill. We augment an existing distillation framework with Fisher refinement without modifying its base distillation objective. Real-image teacher features are precomputed offline, while generated-image features and their statistics are estimated online. Here, μℓr\mu_{\ell}^{r}, μℓg\mu_{\ell}^{g} and Σℓr\Sigma_{\ell}^{r}, Σℓg\Sigma_{\ell}^{g} denote the means and covariances of the real and generated pooled features, respectively. At each selected layer, the closed-form Fisher direction uℓ∗u_{\ell}^{*} points from the generated toward the real feature mean, where λℓ\lambda_{\ell} regularizes the covariance for numerical stability. The Fisher loss converts this direction into a corrective signal toward the real distribution, while gradients propagate through the frozen teacher to update only the generator. No learned discriminator or adversarial optimization is required.

Feature-space and statistical discrimination. Feature matching [42] trains generators by matching intermediate discriminator features, while Fisher GAN [35] formulates a Fisher IPM with a second-moment constraint on a learned critic. Projected GAN [43] and Vision-aided GAN [24] train discriminators on fixed pretrained representations. Scale-wise Distillation [51] introduces a MMD-based objective on frozen diffusion-model features, which reduces to feature-mean matching under a linear kernel. Our Mean variant (Table 4) evaluates this mean-difference direction under our logistic loss and performs worse than FisherDistill, which additionally exploits the full within-class covariance. More closely related, Fréchet-based objectives [34, 59] match the means and covariances of real and generated features in fixed pretrained representations. In contrast, we reuse intermediate representations of the frozen teacher being distilled and analytically construct a covariance-aware Fisher direction that maximizes real-vs.-generated separation relative to within-class variation, rather than minimizing a symmetric distance between distributions.

Table 4. Fisher covariance structure ablation on SD v1.5 [39] DMD2 [60]/LAION setup, NFE=1. Mean uses only the class-mean difference, uℓ=Δℓu_\ell= \Delta_\ell; Diag additionally accounts for per-feature variance, uℓ=(diag⁡Σℓ+λℓI)−1Δℓu_\ell= \left(\operatorname{diag} \Sigma_\ell+ \lambda_\ell I\right)^{-1}\Delta_\ell; Full uses the complete feature covariance, uℓ=(Σℓ+λℓI)−1Δℓu_\ell= \left(\Sigma_\ell+ \lambda_\ell I\right)^{-1}\Delta_\ell, as in Eq. (8). All arms resume the same released checkpoint (Init.).

Direction uℓu_\ellFID↓CLIP↑HPSv2↑ImgRwd↑PickScr↑
Init. (no refinement)9.280.3020.255−0.41020.01
Mean9.850.3050.258−0.39420.03
Diag9.180.3050.258−0.35820.07
Full8.460.3040.257−0.35020.07

Table 4.

Method

We augment the original distribution-matching distillation objective with a critic constructed from statistics of intermediate representations of the frozen generative teacher (Figure 3). We first describe the underlying distillation objective in a general form. Let xg=Gθ(z)x^{g}=G_{\theta}(z), z∼pnoisez \sim p_{\mathrm{noise}}, denote a sample from the student distribution pθp_{\theta}. Rather than directly matching the clean, high-dimensional student and teacher distributions, distribution-matching distillation compares their diffused marginals. We perturb the generated sample at timestep tt as

xtg=αtxg+σtϵ,t∼π(t),ϵ∼N(0,I),(1)x_{t}^{g}=\alpha_{t}x^{g}+\sigma_{t}\epsilon,\qquad t\sim\pi(t),\qquad\epsilon\sim\mathcal{N}(0,\mathbf{I}), \tag*{(1)}

and denote the resulting student marginal by pθtp_{\theta}^{t} and the corresponding teacher marginal by pteachertp_{\mathrm{teacher}}^{t}. A broad class of distribution-matching distillation objectives can then be written as

Ldistill(θ)=Et[D(pθt ∥ pteachert)],(2)\mathcal{L}_{\mathrm{distill}}(\theta)=\mathbb{E}_{t}\left[\mathcal{D}\left(p_{\theta}^{t}\,\|\,p_{\mathrm{teacher}}^{t}\right)\right], \tag*{(2)}

where D\mathcal{D} is a divergence between the diffused student and teacher marginals. Different distillation methods correspond to different choices of D\mathcal{D}. For example, DMD [60, 61] minimizes the reverse KL divergence Epθt[log⁡(pθt/pteachert)]\mathbb{E}_{p_{\theta}^{t}}[\log(p_{\theta}^{t}/p_{\mathrm{teacher}}^{t})], whereas SiD [65, 67] minimizes the Fisher divergence Epθt[∥∇log⁡pθt−∇log⁡pteachert∥22]\mathbb{E}_{p_{\theta}^{t}}[\|\nabla\log p_{\theta}^{t}-\nabla\log p_{\mathrm{teacher}}^{t}\|_{2}^{2}]. In practice, these objectives are optimized through method-specific estimators of the teacher and student scores.

Role of Adversarial Refinement

Adversarial refinement supplements the base distribution-matching loss in Eq. (2) with an additional discriminator-derived correction LadvG\mathcal{L}_{\mathrm{adv}}^{G}. We first examine this correction before introducing its closed-form Fisher counterpart. Let hψ(xtg,t)h_{\psi}(x_{t}^{g},t) denote the discriminator logit, and let

Dψ(xtg,t)=σ(hψ(xtg,t))D_{\psi}(x_{t}^{g},t)=\sigma(h_{\psi}(x_{t}^{g},t)) denote the predicted probability that xtgx_{t}^{g} is real. The non-saturating adversarial generator loss is

LadvG=−Ez,t,ϵ[log⁡Dψ(xtg,t)].(3)\mathcal{L}_{\mathrm{adv}}^{G}=-\mathbb{E}_{z,t,\epsilon}\left[\log D_{\psi}(x_{t}^{g},t)\right]. \tag*{(3)}

The adversarially refined generator is then optimized with LG=Ldistill+λadvLadvG\mathcal{L}_{G}=\mathcal{L}_{\mathrm{distill}}+\lambda_{\mathrm{adv}}\mathcal{L}_{\mathrm{adv}}^{G}. To understand the refinement signal contributed by LadvG\mathcal{L}_{\mathrm{adv}}^{G}, we examine its per-sample image-space descent direction, which decomposes as

−∇xg[−log⁡Dψ(xtg,t)]=(1−Dψ(xtg,t))⏟1αt⏟2∇xtghψ(xtg,t)⏟3.(4)-\nabla_{x^{g}}\left[-\log D_{\psi}(x_{t}^{g},t)\right] = \underbrace{(1-D_{\psi}(x_{t}^{g},t))}_{1}\underbrace{\alpha_{t}}_{2}\underbrace{\nabla_{x_{t}^{g}}h_{\psi}(x_{t}^{g},t)}_{3}. \tag*{(4)}

Eq. (4) exposes three components of the adversarial refinement signal (by the chain rule, since ∂xtg/∂xg=αtI\partial x_{t}^{g}/\partial x^{g}=\alpha_{t}I). Term (1) is a sample-dependent weight, assigning larger corrections to samples judged less real. Term (2), αt\alpha_{t}, is the chain-rule scaling induced by the forward process. Term (3) is the learned discriminative direction that increases the realness logit. Thus, the discriminator determines both the sample-wise magnitude and direction of the refinement, while the forward process contributes the known diffusion-dependent scaling. FisherDistill retains term (2) but replaces the learned discriminator-dependent terms (1) and (3) with their closed-form counterparts derived from real and generated statistics in the frozen teacher feature space.

Residual Distribution Gaps in Teacher Features

The decomposition in Eq. (4) shows that the discriminator determines which generated samples require stronger correction and provides the direction of that correction. We ask whether this information is already present in the representations of the frozen teacher.

For the Fisher critic, we evaluate teacher features at a fixed noise level tFt_{F}, selected based on real-vs.-generated feature separability (more details on the selection in the supplementary). This noise level is independent of the timestep sampled by the base distillation objective. Given a clean sample x\mathbf{x}, we construct

xtF=αtFx+σtFϵ,ϵ∼N(0,I),(5)\mathbf{x}_{t_{F}}=\alpha_{t_{F}}\mathbf{x}+\sigma_{t_{F}}\epsilon,\qquad\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I}), \tag*{(5)}

using a fresh noise draw. For each selected teacher layer ℓ\ell, we denote by fℓrf_{\ell}^{r} the feature vector obtained from the activations of xtF\mathbf{x}_{t_{F}}, where ϵ∈{r,g}\epsilon\in\{r,g\} denotes real and generated samples, respectively. The feature vector concatenates the spatial mean and standard deviation of each channel for U-Nets, or the corresponding statistics over image tokens for DiTs. The mean captures the average feature response, while the standard deviation preserves spatial variation such as contrast and texture. Each feature is obtained from a separate conditional teacher forward pass using the sample’s own class label or prompt, without classifier-free guidance.

We measure real-vs.-generated separability at each teacher layer using the held-out AUROC of a Fisher critic. As shown in Figure 5, separability varies substantially across layers, indicating that residual distribution gaps are reflected in the frozen teacher features. We therefore construct the refinement signal directly in this space from the most separable layers and next derive the closed-form Fisher direction.

Layer-separability ablation plots

Figure 5. Layer-separability ablation on EDM [20]/ImageNet 64 × 64. (a) Held-out Fisher AUROC across frozen teacher layers for real-vs.-generated features, together with a real-vs.-real control that estimates the null floor. Blue stars mark the three most separable layers and orange squares mark three near-null layers. (b) Training with the separable layers yields substantially better FID than near-null layers, which closely track the no-critic continuation.

Binary Closed-Form Fisher Critic

Let μℓr,μℓg\mu_\ell^r,\mu_\ell^g be the pooled real and generated feature means across samples at the fixed Fisher noise level tFt_F, and let Δℓ\Delta_\ell denote their difference, i.e.,

μℓr=E[fℓr],μℓg=E[fℓg],Δℓ=μℓr−μℓg.(6)\mu_\ell^r = \mathbb{E}[f_\ell^r], \qquad\mu_\ell^g = \mathbb{E}[f_\ell^g], \qquad\Delta_\ell= \mu_\ell^r-\mu_\ell^g. \tag*{(6)}

For this binary problem, the between-class (real-vs.-generated) scatter is SℓB=ΔℓΔℓ⊤S_\ell^B=\Delta_\ell\Delta_\ell^\top. Let Σℓr\Sigma_\ell^r and Σℓg\Sigma_\ell^g denote the real and generated feature covariances, and define the shared covariance as Σℓ=12(Σℓr+Σℓg)\Sigma_\ell=\frac{1}{2}(\Sigma_\ell^r+\Sigma_\ell^g). Fisher discriminant analysis seeks a direction that maximizes the separation between the projected class means relative to their within-class variation:

uℓ∗=arg⁡max⁡uu⊤SℓBuu⊤Σℓu.(7)\mathbf{u}_\ell^*=\arg\max_{\mathbf{u}}\frac{\mathbf{u}^{\top}S_\ell^B\mathbf{u}}{\mathbf{u}^{\top}\Sigma_\ell\mathbf{u}}. \tag*{(7)}

Since SℓB=ΔℓΔℓ⊤S_\ell^B=\Delta_\ell\Delta_\ell^\top has rank one, the binary problem has a single non-zero discriminative direction. The solution satisfies the generalized eigenvalue problem SℓBuℓ=γΣℓuℓS_\ell^B\mathbf{u}_\ell=\gamma\Sigma_\ell\mathbf{u}_\ell, where γ\gamma is the generalized eigenvalue; the derivation is provided in the supplementary material. For the binary case, this yields uℓ∗∝Σℓ−1Δℓ\mathbf{u}_\ell^*\propto\Sigma_\ell^{-1}\Delta_\ell. In practice, we regularize the covariance and use

uℓ∗=Σˉℓ−1Δℓ,(8)\mathbf{u}_\ell^*=\bar{\Sigma}_\ell^{-1}\Delta_\ell, \tag*{(8)}

where Σˉℓ=Σℓ+λℓI\bar{\Sigma}_\ell=\Sigma_\ell+\lambda_\ell I, with λℓ>0\lambda_\ell>0, is the regularized shared covariance that improves conditioning. Rather than explicitly computing Σˉℓ−1\bar{\Sigma}_\ell^{-1}, we solve the linear system Σˉℓuℓ∗=Δℓ\bar{\Sigma}_\ell\mathbf{u}_\ell^*=\Delta_\ell using a Cholesky factorization [13].

The Fisher direction in Eq. (8) provides a one-dimensional projection along which the real and generated feature distributions are maximally separated relative to their within-class variation. We define the signed Fisher margin of a feature f\mathbf{f} relative to the midpoint of the projected class means as

rℓ(f)=f⊤uℓ∗−12(uℓ∗⊤μℓr+uℓ∗⊤μℓg).(9)r_\ell(\mathbf{f})=\mathbf{f}^{\top}\mathbf{u}_\ell^*-\frac{1}{2}\left(\mathbf{u}_\ell^{*\top}\mu_\ell^r+\mathbf{u}_\ell^{*\top}\mu_\ell^g\right). \tag*{(9)}

Positive margins correspond to the real side and negative margins to the generated side. Under the shared-covariance Gaussian model p(f∣s)=N(f;μℓs,Σˉℓ)p(\mathbf{f}\mid s)=\mathcal{N}(\mathbf{f};\mu_\ell^s,\bar{\Sigma}_\ell), s∈{r,g}s\in\{r,g\}, the real-to-generated log-density ratio is

log⁡p(f∣r)p(f∣g)=f⊤Σˉℓ−1Δℓ−12(μℓr+μℓg)⊤Σˉℓ−1Δℓ=rℓ(f).(10)\log\frac{p(\mathbf{f}\mid r)}{p(\mathbf{f}\mid g)} =\mathbf{f}^{\top}\bar{\Sigma}_\ell^{-1}\Delta_\ell-\frac{1}{2}(\mu_\ell^r+\mu_\ell^g)^{\top}\bar{\Sigma}_\ell^{-1}\Delta_\ell =r_\ell(\mathbf{f}). \tag*{(10)}

Relation to Adversarial Refinement

Thus, under this model, the Fisher margin is exactly the real-to-generated log-density ratio [14]. With equal class priors, the posterior probability that f\mathbf{f} is real is therefore DℓF(f)=σ(rℓ(f))D_\ell^F(\mathbf{f})=\sigma(r_\ell(\mathbf{f})), a closed-form counterpart of the discriminator DψD_\psi in Eq. (3). Our method, however, constructs rℓ(f)r_\ell(\mathbf{f}) directly from the Fisher direction and projected class means, and therefore does not require the teacher feature distributions themselves to be Gaussian.

Treating DℓFD_\ell^F as a closed-form discriminator, we train the generator with the logistic loss, i.e., the non-saturating loss in Eq. (3) with DℓFD_\ell^F in place of DψD_\psi,

LℓF=−E[log⁡DℓF(fℓg)].(11)\mathcal{L}_\ell^F=-\mathbb{E}\left[\log D_\ell^F(f_\ell^g)\right]. \tag*{(11)}

Minimizing this objective pushes generated features toward the real side of the boundary.

The Fisher critic induces a refinement direction with the same structure as the adversarial refinement in Eq. (4). For a generated sample, the per-layer descent direction is

−∇xg[−log⁡DℓF(fℓg)]=(1−DℓF(fℓg))∂fℓg∂xtFg⊤uℓ∗.(12)-\nabla_{x_g}\left[-\log D_\ell^F(f_\ell^g)\right] =(1-D_\ell^F(f_\ell^g))\frac{\partial f_\ell^g}{\partial x_{t_F}^g}^{\top}\mathbf{u}_\ell^*. \tag*{(12)}

Here, term (1) is the sample-dependent weight of Eq. (4) with the closed-form DℓFD_\ell^F in place of DψD_\psi, term (2) retains the diffusion scaling at the fixed Fisher noise level, and term (3) provides the corrective direction by propagating the closed-form Fisher direction through the frozen teacher. Thus, FisherDistill replaces the learned discriminator-dependent terms with quantities obtained directly from teacher feature statistics, without adversarial critic optimization. We derive Eq. (12) in the supplementary material.

Online Statistics and Lagged Fisher Updates

The real-data statistics are estimated once using the frozen teacher and remain fixed throughout training. Given real features {fℓ,ir}i=1Nr\{f_{\ell,i}^r\}_{i=1}^{N_r}, we compute

μℓr=1Nr∑i=1Nrfℓ,ir,Σℓr=1Nr∑i=1Nrfℓ,irfℓ,ir⊤−μℓrμℓr⊤.(13)\mu_\ell^r=\frac{1}{N_r}\sum_{i=1}^{N_r}f_{\ell,i}^r,\qquad \Sigma_\ell^r=\frac{1}{N_r}\sum_{i=1}^{N_r}f_{\ell,i}^r f_{\ell,i}^{r\top}-\mu_\ell^r\mu_\ell^{r\top}. \tag*{(13)}

Since the generated distribution evolves with GθG_\theta, its statistics are tracked online. At iteration kk, we compute the batch first and second moments

μ^ℓ,kg=1Nk∑i=1Nkfℓ,ig,M^ℓ,kg=1Nk∑i=1Nkfℓ,igfℓ,ig⊤.(14)\hat{\mu}_{\ell,k}^g=\frac{1}{N_k}\sum_{i=1}^{N_k}f_{\ell,i}^g,\qquad \hat{M}_{\ell,k}^g=\frac{1}{N_k}\sum_{i=1}^{N_k}f_{\ell,i}^g f_{\ell,i}^{g\top}. \tag*{(14)}

We maintain exponential moving averages (EMAs)

μℓ,kg=ρkμℓ,k−1g+(1−ρk)μ^ℓ,kg,(15)\mu_{\ell,k}^{g} = \rho_k \mu_{\ell,k-1}^{g} + (1-\rho_k)\hat{\mu}_{\ell,k}^{g}, \tag*{(15)}
Mℓ,kg=ρkMℓ,k−1g+(1−ρk)M^ℓ,kg,(16)M_{\ell,k}^{g} = \rho_k M_{\ell,k-1}^{g} + (1-\rho_k)\hat{M}_{\ell,k}^{g}, \tag*{(16)}

where ρk∈[0,1)\rho_k \in[0,1) is the EMA decay rate, and recover the generated covariance as

Σℓ,kg=Mℓ,kg−μℓ,kgμℓ,kg⊤.(17)\Sigma_{\ell,k}^{g} = M_{\ell,k}^{g} - \mu_{\ell,k}^{g}\mu_{\ell,k}^{g\top}. \tag*{(17)}

We use a one-step-lagged critic: at iteration kk, the Fisher critic is constructed from fixed real statistics and generated statistics accumulated through iteration k−1k-1, then held fixed while scoring the current batch. The current generated statistics are incorporated into the EMA afterward and used at iteration k+1k+1. Statistics and the Fisher solve are treated as stop-gradient quantities, while gradients propagate through the current teacher features to the generator. The EMA and covariance regularization stabilize the solve, while the lag prevents the critic from being estimated from the same features it scores, which improves optimization stability, as shown in Table 5.

Table 5. Fisher critic life-cycle ablation on EDM [20]/ImageNet 64×64, NFE=1. All variants start from the same SiD [65] initialization and use the same Fisher formulation under a matched 2.5M-image budget. Updating the critic once per optimizer step with a one-step lag substantially improves FID, while delayed Fisher activation as in SiDA [67] provides a further gain. Results are mean±std over three evaluation trials.

ConfigurationFID↓Prec.↑Rec.↑
Init1.52 ± 0.020.740 ± 0.0010.627 ± 0.001
+ Legacy life-cycle1.63 ± 0.050.771 ± 0.0000.594 ± 0.001
+ Per-step lagged1.38 ± 0.010.754 ± 0.0020.614 ± 0.005
+ Delayed activation (Ours)1.34 ± 0.020.756 ± 0.0020.626 ± 0.004

Table 5.

Final Training Objective

For the selected teacher layers S\mathcal{S}, we average the layer-wise Fisher losses:

LF=1∣S∣∑ℓ∈SLℓF.(18)\mathcal{L}^{F} = \frac{1}{|\mathcal{S}|} \sum_{\ell\in\mathcal{S}} \mathcal{L}_{\ell}^{F}. \tag*{(18)}

The final generator objective augments the original distillation loss with the Fisher refinement term,

LG=Ldistill+λF(k)LF,(19)\mathcal{L}_{G} = \mathcal{L}_{\mathrm{distill}} + \lambda_{F}(k)\mathcal{L}^{F}, \tag*{(19)}

where λF(k)\lambda_{F}(k) controls the refinement strength. We initially disable the Fisher loss, allowing the online generated statistics to stabilize, and then activate it at full weight. The generator’s training schedule follows that of the base distillation framework’s adversarial variant: for SiD [65, 67], SiD-DiT [66] and 1.x-Distill [25], the generator is frozen for the first few steps, whereas for DMD2 [60] it is trained from the first step with the Fisher term disabled.

The teacher remains frozen throughout training, and the Fisher critics introduce no learnable parameters or separate optimizer. The generator is therefore optimized jointly by the distillation objective and the closed-form Fisher refinement.

Experiments

We evaluate whether closed-form Fisher refinement can improve few-step diffusion distillation without a learned discriminator. We apply it to different distillation objectives on class-conditioned and text-to-image generation, and compare against the distilled baselines and their adversarially refined variants.

Class-Conditioned Generation

Evaluation metrics. For class-conditioned generation, we report Fréchet Inception Distance (FID) [16] with precision and recall to measure sample fidelity and coverage. For text-to-image generation, we use FID for distributional quality, CLIP score [15] for text-image alignment, GenEval [12] for compositional prompt adherence, Aesthetic Score [46] for aesthetic quality, HPSv2 [54], ImageReward [57], and PickScore [22] for human-preference alignment.

Setup

We evaluate class-conditioned generation on ImageNet 64×6464 \times64 with the pretrained EDM [20] teacher. We study Fisher refinement on two one-step distillation families: distribution matching (DMD [61]) and score-identity distillation (SiD [65]). We compare with their GAN refinements DMD2 [60] and SiD2^{2}A [67], and with RealUID [23], a GAN-free baseline that brings real data into the fake-score objective. For DMD, we resume from the DMD-only checkpoint (405.5K steps, FID 2.64) and fine-tune for 40K steps. For SiD, we initialize from the released checkpoint (α\alpha=1.2, 939K) and follow the SiDA training budget of 2.5M generated images. RealUID uses the same initialization and budget as each Fisher model. Fisher is applied to five teacher layers for DMD2 and three for SiD. In both settings, real feature statistics are computed once from 50K images and kept fixed, while generated statistics are tracked with an EMA decay rate of ρk=0.99\rho_k = 0.99.

Results. Table 1 shows that Fisher refinement consistently improves both distillation objectives. On DMD2, Fisher reduces FID from 1.42 to 1.40 and improves recall from 0.60 to 0.62, while preserving precision at 0.74. It also outperforms RealUID, with substantially lower FID (1.40 vs. 3.07) and higher recall (0.62 vs. 0.58) at the same precision. On SiD, Fisher reduces FID from 1.54 to 1.34 and increases precision from 0.74 to 0.76 without reducing recall. Compared with RealUID, Fisher achieves lower FID and higher precision and recall. Fisher is also competitive with adversarially refined SiD2^{2}A. Overall, Fisher refinement improves sample fidelity without compromising distributional coverage.

Class-conditioned ImageNet 64×6464 \times64 results using a pretrained EDM teacher with diffusion solvers and prior distillation methods. FisherDistill is evaluated on top of DMD2 and SiD, with precision and recall reported where available. Bold: best among the compared methods below the faint rule within each distribution-matching block. Results outside the “Critic Based” family are from [65] or original papers. †: reproduced with the official code under a matched training budget.

FamilyModelNFEFID ↓Prec. ↑Rec. ↑
TeacherVP EDM ODE5112.32
VP EDM SDE5111.32
SolverDDIM [48]822.59
DPM-Solver [30]812.42
AMED [68]87.01
GITS [5]85.64
INDIS [62]84.49
DistillationPD [41]28.950.630.65
CD-LPIPS [50]24.700.690.64
DFNO [63]17.830.61
Diff-Instruct [32]15.57
iCT [49]14.020.700.63
iCT-deep [49]13.250.720.63
EMD-16 [56]12.20
CTM [21]11.920.57
Critic BasedDMD [61]12.640.760.59
DMD2† [60]11.420.740.60
RealUID† [23]13.070.740.58
FisherDistill11.400.740.62
SiD [65], α=1.2\alpha= 1.211.540.740.63
SiD2^{2}A† [67], α=1.0\alpha= 1.011.310.740.62
RealUID† [23]11.450.740.62
FisherDistill11.340.760.63

Table 1.

Text-to-Image Generation

Setup

We evaluate Fisher refinement in three text-to-image settings: DMD2 [60] with SD v1.5 [39], 1.x-Distill [25] with SD3-Medium [10], and SiD-DiT [66] with SANA-0.6B and SANA-1.6B [55]. We follow the baseline protocols and compare with the matched GAN refinement and GAN-free RealUID [23], starting from the same stage-1 checkpoints, i.e., models before GAN fine-tuning.

For DMD2, we fine-tune the released DMD-only checkpoint on LAION-Aesthetics prompts [47]. Fisher uses three frozen teacher layers, with real statistics precomputed from 8K random images from DMD2’s GAN training set.

For 1.x-Distill, whose official weights are unavailable, we reproduce the Stage-I student and 8-step reference model using 20K Midjourney prompts [3], followed by Stage-II refinement for 2K iterations. Fisher uses three frozen teacher blocks, with real statistics precomputed from 10K latents generated by SD3-Medium (28 steps, CFG 7.0). Both the GAN baseline and RealUID use the 8-step reference model as real data. RealUID replaces the Stage-II GAN loss, and all methods use a matched generator-update budget.

For SiD-DiT, we refine the released SANA checkpoints at both scales using LAION-Aesthetics prompts [47]. Fisher uses ten (0.6B) and eight (1.6B) frozen teacher layers, with real statistics precomputed from 10K Midjourney images [3] also used by the SiD2^{2}A discriminator and RealUID. All models are evaluated zero-shot on COCO-2014 [27].

Evaluation protocols. Since each baseline has its own protocol, we evaluate every method with the protocol of its base framework and compare only within each group of Table 2. Details are in the supplementary material.

Table 2. Text-to-image generation across distillation frameworks and teacher models. We compare Fisher refinement with adversarial refinement and RealUID. Bold and _underline_ denote the best and second-best results, respectively, among the three refinement methods within each setting; ties receive the same formatting. †: trained by us with the official code under a matched training budget. ×2 indicates classifier-free guidance.

ModelNFEFID ↓\downarrowCLIP ↑\uparrowGenEval ↑\uparrowAesth. ↑\uparrowHPSv2 ↑\uparrowImgRwd ↑\uparrowPickScore ↑\uparrow
SD v1.5 [39]50×250 \times28.600.3100.4425.4660.2670.09620.61
DMD [61]19.280.3020.3475.3330.255-0.41020.01
DMD2† [60]1_8.58_0.3000.3435.414_0.256_-0.37820.00
RealUID† [23]19.37_0.303_0.3525.346_0.256_-0.332_20.06_
FisherDistill18.460.304_0.348__5.393_0.257_-0.350_20.07
SD3-Medium [10]28×228 \times223.390.3300.7085.3700.2941.02722.50
1.x-Distill Stage I† [25]232.350.3290.6515.2830.2840.98522.15
1.x-Distill Stage II† [25]2_30.20_0.330_0.655__5.430__0.288__0.998_22.59
RealUID† [23]234.030.3260.6415.1010.2790.86221.99
FisherDistill229.540.3300.6825.4830.2941.059_22.37_
SANA 0.6B [55]20×220 \times226.380.3380.6515.7300.3110.99922.50
SiD-DiT [66]128.940.3230.5945.7510.2780.70921.51
SiD2a_2^a-DiT† [66]134.130.2980.4865.3830.2440.28721.00
RealUID† [23]1_31.87__0.320__0.590__5.740__0.274__0.688__21.46_
FisherDistill125.240.3260.5975.7440.2860.82021.80
SANA 1.6B [55]20×220 \times226.830.3340.6525.8400.3121.05422.56
SiD-DiT [66]128.860.3250.5815.8810.2850.88021.71
SiD2a_2^a-DiT† [66]1_28.58__0.323_0.615_5.889_0.289_0.883_21.81
RealUID† [23]134.280.3220.5785.9160.2790.86721.60
FisherDistill126.730.326_0.603_5.8160.2890.884_21.73_

Table 2.

Results. As shown in Table 2, Fisher consistently improves distributional quality across all settings. On DMD2, FID improves from 8.58 to 8.46, while on 1.x-Distill it improves from 32.35/30.20 for Stages I/II to 29.54, with competitive alignment and preference metrics. The same trend holds for SANA at both scales; for example, on SANA-0.6B Fisher reduces FID from 28.94 to 25.24 and improves most alignment and preference metrics. In contrast, RealUID does not improve the first-stage models on FID and mostly fails to improve the other metrics. Notably, the one-step Fisher models also match or surpass the multi-step SANA teachers in FID at both scales (25.24 vs. 26.38 and 26.73 vs. 26.83).

Fisher also reaches its best performance substantially earlier than GAN refinement. On DMD2, it reaches its best FID after 1.34M samples, compared with 4.93M for the matched GAN arm; a similar trend holds for SiD-DiT (~1.0M vs. ~2.0M). We attribute this efficiency to computing the direction directly from frozen teacher statistics rather than learning it through a separately optimized discriminator. Overall, Fisher improves FID across architectures, scales, and distillation objectives while preserving competitive semantic and perceptual quality (Figure 4).

Figure 4. 512×512512 \times512 samples produced by our 1-step generators distilled from SD v1.5 (left) and SANA-1.6B (right), compared with the teachers and the GAN-refined DMD2 and SiD-DiT.

Ablations

Layer selection for Fisher discriminants. The closed-form Fisher critic reads generated samples through a small set of teacher layers, so its usefulness depends on whether those layers can distinguish real from generated features. Figure 5(a) quantifies this: for each of the 38 pooled teacher layers, we fit a Fisher discriminant between real images and samples from the SiD initialization [65] on ImageNet [9] and report its held-out AUROC (5-fold cross-validation). A real-vs.-real split of the dataset gives the null floor. Separability varies substantially across layers: mid-resolution decoder blocks reach AUROC 0.80–0.84, while several layers remain close to the floor. We select the three most separable layers (best arm, ★) and three near-null layers (null arm, ■, AUROC 0.51–0.53 versus a floor of ≈0.54), and train both under identical settings.

As shown in Figure 5(b), the separable layers yield a stable improvement on a FID of 1.34, while the near-null layers reach only 1.69 and behave similarly to continuing training without the critic. This result shows that Fisher refinement is effective only when the selected teacher representation contains reliable real-vs.-generated information. The held-out AUROC sweep therefore provides a simple criterion for selecting layers before training.

Effect of Fisher refinement. Since the Fisher critic is introduced while fine-tuning an already distilled generator, improvements could arise simply from additional score-distillation updates. We therefore compare Fisher against continuing pure SiD from the same initialization under a matched training budget. As shown in Table 3, Fisher improves single-step FID on ImageNet-64 from 1.75 to 1.38. On SANA-0.6B [55], FID improves from 28.45 to 27.33, with CLIP unchanged at 0.33. These results indicate that the gains arise from the additional Fisher correction rather than continued score distillation alone.

Table 3. Continuation of distillation objective, NFE=1. Starting from the same initialization, we compare continued SiD [65] training with and without the proposed Fisher refinement. Fisher improves FID in both EDM [20]/ImageNet-64 and SANA-0.6B [55]/COCO-10K at a comparable budget (1.7M images).

MethodEDM/ImageNet-64SANA-0.6B/COCO-30K
FID↓FID↓CLIP↑
SiD only1.7528.450.33
FisherDistill1.3827.330.33

Table 3.

Effect of the full covariance. Simpler discriminant directions could also pull generated teacher features toward the real ones without modeling the full within-class covariance. We therefore compare the raw mean difference (Mean), per-dimension variance normalization (Diag), and the full Fisher direction (Full), while matching their initial gradient magnitudes. As shown in Table 4, incorporating the full covariance is important for distributional quality. Full improves FID from 9.28 at initialization to 8.46, whereas Diag reaches 9.18 and Mean degrades FID to 9.85. In contrast, CLIP and preference metrics vary only modestly. These results indicate that modeling cross-feature correlations provides information beyond the mean difference or independent per-feature variances, particularly for FID.

Fisher critic life-cycle. All variants in Table 5 start from the same SiD-initialized generator [65] (FID 1.52) and differ only in the critic update life-cycle and the staging of training. In the legacy life-cycle, generated statistics are updated at every gradient-accumulation round. With 64 accumulation rounds per optimizer step, this changes the Fisher critic 64 times per generator update, shortens the effective memory of the statistics EMA to roughly one step (0.9964≈0.530.99^{64} \approx0.53), and allows current samples to enter the statistics used to score subsequent samples. This configuration degrades FID to 1.63 and exhibits the strongest precision-recall trade-off in the table: precision increases by 0.031, while recall decreases by 0.036. This trade-off suggests that the unstable critic favors precision at the expense of coverage.

Updating the statistics once per optimizer step and scoring the current batch with the critic from the previous step removes both within-step drift and self-scoring. This lagged life-cycle improves FID to 1.38 and raises recall from 0.594 to 0.614, while retaining high precision, which indicates a more balanced fidelity–coverage trade-off than the legacy scheme. Finally, we follow SiDA’s [67] staged training strategy: the generator is initially held fixed while the Fisher statistics stabilize, and Fisher refinement is activated only afterward. We match SiDA’s loss weighting so that the two settings differ only in the refinement term. This staged activation reduces the early optimization transient and yields our full method with FID 1.34. Together, these results show that per-step lagged updates and delayed Fisher activation are important for stable training under gradient accumulation.

Conclusion

We introduced FisherDistill, a simple non-adversarial refinement method for few-step generative distillation. Rather than learning an adversarial discriminator, FisherDistill exploits intermediate representations of the frozen generative teacher and derives a closed-form Fisher critic from the first- and second-order statistics of real and generated features. The resulting covariance-aware refinement signal is incorporated alongside the original distillation objective without requiring a learned critic or alternating adversarial optimization. Across multiple distillation frameworks and generation settings, FisherDistill consistently improves the underlying distilled generators and surpasses their adversarially refined counterparts. These results suggest that the residual distribution gap left by aggressive distillation can be effectively corrected using simple statistics already exposed by the frozen teacher, providing a lightweight alternative to learned adversarial refinement.

Derivation of the Adversarial Refinement Direction

We derive the per-sample descent direction used in Eq. (4). For a generated sample xtgx_t^g, the non-saturating adversarial loss is given by,

ℓadvG=−log⁡Dψ(xtg,t),(20)\ell_{\mathrm{adv}}^G=-\log D_\psi(x_t^g,t), \tag*{(20)}

where

Dψ(xtg,t)=σ(hψ(xtg,t)).(21)D_\psi(x_t^g,t)=\sigma(h_\psi(x_t^g,t)). \tag*{(21)}

For brevity, let Dψ=Dψ(xtg,t)D_\psi=D_\psi(x_t^g,t) and hψ=hψ(xtg,t)h_\psi=h_\psi(x_t^g,t). Since

ddh[−log⁡σ(h)]=−(1−σ(h)),(22)\frac{d}{dh}\left[-\log\sigma(h)\right]=-(1-\sigma(h)), \tag*{(22)}

the gradient with respect to the noised generated sample is

∇xtgℓadvG=−(1−Dψ)∇xtghψ.(23)\nabla_{x_t^g}\ell_{\mathrm{adv}}^G=-(1-D_\psi)\nabla_{x_t^g}h_\psi. \tag*{(23)}

From the forward process, xtg=αtxg+σtϵx_t^g=\alpha_t x^g+\sigma_t\epsilon, we have,

∂xtg∂xg=αtI.(24)\frac{\partial x_t^g}{\partial x^g}=\alpha_t I. \tag*{(24)}

Applying the chain rule gives,

∇xgℓadvG=αt∇xtgℓadvG=−(1−Dψ)αt∇xtghψ.(25)\begin{aligned} \nabla_{x^g}\ell_{\mathrm{adv}}^G=\alpha_t\nabla_{x_t^g}\ell_{\mathrm{adv}}^G \\ &=-(1-D_\psi)\alpha_t\nabla_{x_t^g}h_\psi. \tag*{(25)} \end{aligned}

Therefore, the corresponding descent direction is:

−∇xgℓadvG=(1−Dψ(xtg,t))⏟sample weightαt⏟diffusion scaling∇xtghψ(xtg,t)⏟discriminative direction,(26)-\nabla_{x^g}\ell_{\mathrm{adv}}^G = \underbrace{(1-D_\psi(x_t^g,t))}_{\text{sample weight}} \underbrace{\alpha_t}_{\text{diffusion scaling}} \underbrace{\nabla_{x_t^g}h_\psi(x_t^g,t)}_{\text{discriminative direction}}, \tag*{(26)}

which gives Eq. (4).

Generalized Eigenvalue Derivation for the Binary Fisher Critic

We provide the derivation of the generalized eigenvalue problem used in § 3.3. The goal of Fisher discriminant analysis is to find a projection direction along which the separation between the real and generated feature means is large relative to the variation within the two classes. For a projection direction uℓ\mathbf{u}_\ell, this trade-off is measured by the Fisher criterion

J(uℓ)=uℓ⊤SℓBuℓuℓ⊤Σℓuℓ.(27)J(\mathbf{u}_\ell)=\frac{\mathbf{u}_\ell^{\top}S_\ell^B\mathbf{u}_\ell}{\mathbf{u}_\ell^{\top}\Sigma_\ell\mathbf{u}_\ell}. \tag*{(27)}

The numerator measures the separation between the projected class means, whereas the denominator measures the corresponding within-class variation.

The value of J(uℓ)J(\mathbf{u}_\ell) depends only on the direction of uℓ\mathbf{u}_\ell. Indeed, for any non-zero scalar aa,

J(auℓ)=a2uℓ⊤SℓBuℓa2uℓ⊤Σℓuℓ=J(uℓ).(28)J(a\mathbf{u}_\ell)=\frac{a^2\mathbf{u}_\ell^{\top}S_\ell^B\mathbf{u}_\ell}{a^2\mathbf{u}_\ell^{\top}\Sigma_\ell\mathbf{u}_\ell}=J(\mathbf{u}_\ell). \tag*{(28)}

We can therefore fix the scale of uℓ\mathbf{u}_\ell without changing the optimization. Choosing uℓ⊤Σℓuℓ=1\mathbf{u}_\ell^{\top}\Sigma_\ell\mathbf{u}_\ell=1 turns the Fisher criterion into the equivalent constrained problem

max⁡uℓuℓ⊤SℓBuℓsubject touℓ⊤Σℓuℓ=1.(29)\max_{\mathbf{u}_\ell}\quad\mathbf{u}_\ell^{\top}S_\ell^B\mathbf{u}_\ell \qquad\text{subject to}\qquad \mathbf{u}_\ell^{\top}\Sigma_\ell\mathbf{u}_\ell=1. \tag*{(29)}

Introducing a Lagrange multiplier γ\gamma for the constraint gives

J(uℓ,γ)=uℓ⊤SℓBuℓ−γ(uℓ⊤Σℓuℓ−1).(30)\mathcal{J}(\mathbf{u}_\ell,\gamma)=\mathbf{u}_\ell^{\top}S_\ell^B\mathbf{u}_\ell-\gamma\left(\mathbf{u}_\ell^{\top}\Sigma_\ell\mathbf{u}_\ell-1\right). \tag*{(30)}

At a stationary point, the derivative with respect to uℓ\mathbf{u}_\ell must vanish. Since both SℓBS_\ell^B and Σℓ\Sigma_\ell are symmetric,

∂J∂uℓ=2SℓBuℓ−2γΣℓuℓ=0.(31)\frac{\partial\mathcal{J}}{\partial\mathbf{u}_\ell}=2S_\ell^B\mathbf{u}_\ell-2\gamma\Sigma_\ell\mathbf{u}_\ell=0. \tag*{(31)}

Rearranging gives

SℓBuℓ=γΣℓuℓ,(32)S_\ell^B\mathbf{u}_\ell=\gamma\Sigma_\ell\mathbf{u}_\ell, \tag*{(32)}

which is the generalized eigenvalue problem associated with the matrix pair (SℓB,Σℓ)(S_\ell^B,\Sigma_\ell). Multiplying both sides by uℓ⊤\mathbf{u}_\ell^{\top} shows that

γ=uℓ⊤SℓBuℓuℓ⊤Σℓuℓ=J(uℓ),(33)\gamma=\frac{\mathbf{u}_\ell^{\top}S_\ell^B\mathbf{u}_\ell}{\mathbf{u}_\ell^{\top}\Sigma_\ell\mathbf{u}_\ell}=J(\mathbf{u}_\ell), \tag*{(33)}

so the generalized eigenvalue is exactly the Fisher criterion attained by that direction. The maximizing solution therefore corresponds to the largest generalized eigenvalue.

For our binary real-vs.-generated problem,

SℓB=ΔℓΔℓ⊤,(34)S_\ell^B=\Delta_\ell\Delta_\ell^{\top}, \tag*{(34)}

which has rank one. Substituting this form into Eq. (32) gives

Δℓ(Δℓ⊤uℓ)=γΣℓuℓ.(35)\Delta_\ell\left(\Delta_\ell^{\top}\mathbf{u}_\ell\right)=\gamma\Sigma_\ell\mathbf{u}_\ell. \tag*{(35)}

The quantity Δℓ⊤uℓ\Delta_\ell^{\top}\mathbf{u}_\ell is a scalar, so for the non-zero generalized eigenvalue, Σℓuℓ\Sigma_\ell\mathbf{u}_\ell must lie in the direction of Δℓ\Delta_\ell. When Σℓ\Sigma_\ell is invertible, this gives

uℓ∗∝Σℓ−1Δℓ.(36)\mathbf{u}_\ell^*\propto\Sigma_\ell^{-1}\Delta_\ell. \tag*{(36)}

Thus, in the binary case, the Fisher criterion yields a single non-zero discriminative direction given by the covariance-adjusted difference between the real and generated feature means.

Derivation of the Fisher Refinement Direction

We derive the per-sample refinement direction induced by the Fisher objective and show its correspondence to the adversarial refinement gradient in Eq. (4). During each generator update, the feature statistics used to construct the Fisher critic, including the class means and the Fisher direction uℓ∗\mathbf{u}_{\ell}^{*}, are treated as fixed. Thus, gradients propagate through the current generated feature fℓg\mathbf{f}_{\ell}^{g} and the frozen teacher network, but not through the statistics defining the critic. For a single generated sample, the Fisher loss at layer ℓ\ell is the logistic loss of the closed-form discriminator DℓF(f)=σ(rℓ(f))D_{\ell}^{F}(\mathbf{f}) = \sigma(r_{\ell}(\mathbf{f})),

ℓℓF=−log⁡DℓF(fℓg)=log⁡(1+e−rℓ(fℓg)).(37)\ell_{\ell}^{F} = -\log D_{\ell}^{F}(\mathbf{f}_{\ell}^{g}) = \log\left(1+e^{-r_{\ell}(\mathbf{f}_{\ell}^{g})}\right). \tag*{(37)}

Since

ddrlog⁡(1+e−r)=−σ(−r)=−(1−σ(r)),(38)\frac{d}{dr}\log\left(1+e^{-r}\right)=-\sigma(-r)=-(1-\sigma(r)), \tag*{(38)}

the gradient of the loss with respect to the generated sample is

∇xgℓℓF=−(1−DℓF(fℓg))∇xgrℓ(fℓg).(39)\nabla_{\mathbf{x}^{g}}\ell_{\ell}^{F}=-(1-D_{\ell}^{F}(\mathbf{f}_{\ell}^{g}))\nabla_{\mathbf{x}^{g}}r_{\ell}(\mathbf{f}_{\ell}^{g}). \tag*{(39)}

Recall that the Fisher margin is

rℓ(f)=fTuℓ∗−12(uℓ∗Tμℓr+uℓ∗Tμℓg).(40)r_{\ell}(\mathbf{f})=\mathbf{f}^{\mathsf{T}}\mathbf{u}_{\ell}^{*}-\frac{1}{2}\left(\mathbf{u}_{\ell}^{*\mathsf{T}}\boldsymbol{\mu}_{\ell}^{r}+\mathbf{u}_{\ell}^{*\mathsf{T}}\boldsymbol{\mu}_{\ell}^{g}\right). \tag*{(40)}

Because uℓ∗\mathbf{u}_{\ell}^{*} and the class means are fixed during the current generator update, the margin depends on xg\mathbf{x}^{g} only through fℓg\mathbf{f}_{\ell}^{g}. Therefore,

∇fℓgrℓ(fℓg)=uℓ∗.(41)\nabla_{\mathbf{f}_{\ell}^{g}}r_{\ell}(\mathbf{f}_{\ell}^{g})=\mathbf{u}_{\ell}^{*}. \tag*{(41)}

The generated feature fℓg\mathbf{f}_{\ell}^{g} is obtained from the frozen teacher using the noised generated sample

xtFg=αtFxg+σtFϵ.(42)\mathbf{x}_{t_{F}}^{g}=\alpha_{t_{F}}\mathbf{x}^{g}+\sigma_{t_{F}}\boldsymbol{\epsilon}. \tag*{(42)}

Applying the chain rule through the teacher feature gives

∇xtFgrℓ(fℓg)=(∂fℓg∂xtFg)Tuℓ∗.(43)\nabla_{\mathbf{x}_{t_{F}}^{g}}r_{\ell}(\mathbf{f}_{\ell}^{g})=\left(\frac{\partial\mathbf{f}_{\ell}^{g}}{\partial\mathbf{x}_{t_{F}}^{g}}\right)^{\mathsf{T}}\mathbf{u}_{\ell}^{*}. \tag*{(43)}

Since

∂xtFg∂xg=αtFI,(44)\frac{\partial\mathbf{x}_{t_{F}}^{g}}{\partial\mathbf{x}^{g}}=\alpha_{t_{F}}\mathbf{I}, \tag*{(44)}

we obtain

∇xgrℓ(fℓg)=αtF(∂fℓg∂xtFg)Tuℓ∗.(45)\nabla_{\mathbf{x}^{g}}r_{\ell}(\mathbf{f}_{\ell}^{g})=\alpha_{t_{F}}\left(\frac{\partial\mathbf{f}_{\ell}^{g}}{\partial\mathbf{x}_{t_{F}}^{g}}\right)^{\mathsf{T}}\mathbf{u}_{\ell}^{*}. \tag*{(45)}

Substituting the margin gradient into Eq. (39) gives the per-sample descent direction

−∇xgℓℓF=(1−DℓF(fℓg))⏟sample-dependent weightαtF⏟diffusion scaling(∂fℓg∂xtFg)Tuℓ∗⏟corrective direction.(46)-\nabla_{\mathbf{x}^{g}}\ell_{\ell}^{F}=\underbrace{\left(1-D_{\ell}^{F}(\mathbf{f}_{\ell}^{g})\right)}_{\text{sample-dependent weight}}\underbrace{\alpha_{t_{F}}}_{\text{diffusion scaling}}\underbrace{\left(\frac{\partial\mathbf{f}_{\ell}^{g}}{\partial\mathbf{x}_{t_{F}}^{g}}\right)^{\mathsf{T}}\mathbf{u}_{\ell}^{*}}_{\text{corrective direction}}. \tag*{(46)}

This directly mirrors the adversarial refinement direction in Eq. (4). The factor 1−DℓF(fℓg)=σ(−rℓ(fℓg))1-D_{\ell}^{F}(\mathbf{f}_{\ell}^{g})=\sigma(-r_{\ell}(\mathbf{f}_{\ell}^{g})) is the closed-form counterpart of the discriminator weight 1−Dψ1-D_{\psi}. The Fisher direction uℓ∗\mathbf{u}_{\ell}^{*}, propagated through the frozen teacher, replaces the learned discriminator gradient, while αtF\alpha_{t_{F}} provides the same diffusion scaling evaluated at the fixed Fisher noise level. The gradient of the full Fisher objective is obtained by taking the expectation of these per-sample gradients.

Text-to-Image Evaluation Protocol

For the DMD2 [60] setting, we follow its zero-shot COCO-2014 [27] protocol: 512×512512 \times512 images generated with 30K validation captions, scored with clean-FID [36] against 40,504 validation images, with CLIP score [15] computed using ViT-g-14 [7]. For 1.x-Distill [25], we follow its SD3-Medium [10] protocol: 10K COCO-2014 [27] prompts at 1024×10241024 \times1024. 1.x-Distill [25] computes FID [16, 36] against teacher-generated images. In contrast, we compute FID against real images, which provides a direct measure of distributional fidelity to the target data distribution and follows the standard evaluation protocol used in prior generative modeling literature [39, 60, 61]. For CLIP score, we use ViT-B/32 [7]. For SiD-DiT [66], we follow its 10K-COCO [27] protocol at the SANA [55] native resolution. Human-preference metrics (Aesthetics [46], HPSv2 [54], ImageReward [57], PickScore [22]) are computed on 2,048 Pick-a-Pic [22] validation prompts for DMD2 and COCO-10K [27] for the rest.

Additional Ablations

Selecting the teacher-feature noise level. We choose tFt_F by measuring real-generated separability in frozen teacher features across candidate noise levels and selecting a low-noise regime that preserves strong separation across the layers used by FisherDistill. Shown here for SD3 [10] (Figure 6), we measure the Fisher separation margin between real images and Stage-I generations using 5K samples per set, Mℓ(tF)=∣uℓ∗⊤(μℓg−μℓr)∣uℓ∗⊤Σℓuℓ∗M_\ell(t_F) = \frac{\left\lvert u_\ell^{*\top}(\mu_\ell^g-\mu_\ell^r)\right\rvert}{\sqrt{u_\ell^{*\top}\Sigma_\ell u_\ell^*}}. Gray denotes individual teacher layers, black the mean across all layers, and blue the mean across the layers selected for FisherDistill. Separation decreases with increasing tFt_F for nearly all layers as noise suppresses residual real-generated differences. We therefore use tF=0.05t_F = 0.05 for SD3 [10]. We similarly decide the noise level of 0.085 for EDM [20], 0.2 for the SANA [55] models and t=21t = 21 for SD1.5 [39].

Teacher Feature Noise Selection $t_F$ on SD3-Medium

Figure 6. Teacher Feature Noise Selection tFt_F on SD3-Medium [10]. Fisher separation margin between real images and Stage-I generations (5K each), computed on frozen SD3 teacher features at four noise levels. Grey: individual teacher layers. Black: mean over all layers. Blue: mean over the layers selected for FisherDistill. Separation decreases monotonically with tFt_F for nearly all layers; we use tF=0.05t_F = 0.05.

ImageNet Results

Results on 100 ImageNet classes in their original 64×6464 \times64 dimensions distilled from EDM are in Figure 7 to Figure 9 for GAN refined [60], FisherDistill refined and RealUID [23] refined generators.

Uncurated one-step samples from GAN refined generator on ImageNet

Figure 7. Uncurated one-step samples from GAN refined [60] generator on ImageNet 64×6464 \times64 (FID 1.42) distilled from EDM [20] in DMD [61], one sample per class for classes 0-99.

Uncurated one-step samples from RealUID generator on ImageNet

Figure 9. Uncurated one-step samples from RealUID [23] generator on ImageNet 64×6464 \times64 (FID 3.07) distilled from EDM [20] in DMD [61], one sample per class for classes 0-99.

Uncurated one-step samples from FisherDistill generator on ImageNet

Figure 8. Uncurated one-step samples from FisherDistill generator on ImageNet 64×6464 \times64 (FID 1.40) distilled from EDM [20] in DMD [61], one sample per class for classes 0-99.

References

  1. [1]Lichen Bai, Zikai Zhou, Shitong Shao, Wenliang Zhong, Shuo Yang, Shuo Chen, Bojun Chen, and Zeke Xie. Optimizing few-step generation with adaptive matching distillation. In Proceedings of the 43rd International Conference on Machine Learning, pages 5540–5565, 2026. 3arxiv.org/abs/2602.07345
  2. [2]Mahesh Bhosale, Abdul Wasi, Yuanhao Zhai, Yunjie Tian, Samuel Border, Nan Xi, Pinaki Sarder, Junsong Yuan, David Doermann, and Xuan Gong. Pathdiff: Histopathology image synthesis with unpaired text and mask conditions. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 22415–22424. IEEE, 2025. 1
  3. [3]brivangl. Midjourney-v6-llava. https://huggingface.co/datasets/brivangl/midjourney-v6-llava, 2024. 7DOI
  4. [4]Xu Cai, Yang Wu, Qianli Chen, Haoran Wu, Lichuan Xiang, and Hongkai Wen. Shortcutting pre-trained flow matching diffusion models is almost free lunch. Advances in Neural Information Processing Systems, 38:153066–153094, 2025. 3arxiv.org/abs/2510.17858
  5. [5]Defang Chen, Zhenyu Zhou, Can Wang, Chunhua Shen, and Siwei Lyu. On the trajectory regularity of ode-based diffusion sampling. In International Conference on Machine Learning, pages 7905–7934, 2024. 1, 7arxiv.org/abs/2405.11326
  6. [6]Zhenglin Cheng, Peng Sun, Jianguo Li, and Tao Lin. Twin-flow: Realizing one-step generation on large models with self-adversarial flows. In International Conference on Learning Representations, pages 151237–151264, 2026. 2, 3arxiv.org/abs/2512.05150
  7. [7]Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2829. IEEE, 2023. 15DOI
  8. [8]Quan Dao, Hao Phung, Trung Tuan Dao, Dimitris N Metaxas, and Anh Tran. Self-corrected flow distillation for consistent one-step and few-step image generation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2654–2662, 2025. 3
  9. [9]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. 7
  10. [10]Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the 41st International Conference on Machine Learning, pages 12606–12633. PMLR, 2024. 1, 2, 6, 9, 15, 16arxiv.org/abs/2403.03206
  11. [11]Xingtong Ge, Xin Zhang, Tongda Xu, Yi Zhang, Xinjie Zhang, Yan Wang, and Jun Zhang. Senseflow: Scaling distribution matching for flow-based text-to-image distillation. arXiv preprint arXiv:2506.00523, 2025. 3arxiv.org/abs/2506.00523
  12. [12]Dhruba Ghosh, Hannah Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132–52152, 2023. 6arxiv.org/abs/2310.11513
  13. [13]Gene H. Golub and Charles F. Van Loan. Matrix Computations - 4th Edition. Johns Hopkins University Press, Philadelphia, PA, 2013. 5
  14. [14]T. Hastie, R. Tibshirani, and J.H. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2009. 5
  15. [15]Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 7514–7528, 2021. 6, 15
  16. [16]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 6, 15
  17. [17]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1arxiv.org/abs/2006.11239
  18. [18]Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022. 1arxiv.org/abs/2210.02303
  19. [19]Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in neural information processing systems, 35:8633–8646, 2022. 1
  20. [20]Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022. 6, 9, 10, 16, 17, 18, 19arxiv.org/abs/2206.00364
  21. [21]Dongjun Kim, Chieh-Hsin Lai, WeiHsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. In International Conference on Learning Representations, pages 44493–44525, 2024. 3, 7arxiv.org/abs/2310.02279
  22. [22]Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems, 36:36652–36663, 2023. 6, 15arxiv.org/abs/2305.01569
  23. [23]Nikita Kornilov, David Li, Tikhon Mavrin, Aleksei Leonov, Nikita Gushchin, Evgeny Burnaev, Iaroslav Koshelev, and Aleksandr Korotin. Universal inverse distillation for matching models with real-data supervision (no gans). In International Conference on Learning Representations, pages 67906–67948, 2026. 3, 6, 7, 9, 16, 19arxiv.org/abs/2509.22459
  24. [24]Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Ensembling off-the-shelf models for gan training. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10641–10652. IEEE, 2022. 3arxiv.org/abs/2112.09130
  25. [25]Haoyu Li, Tingyan Wen, Lin Qi, Zhe Wu, Yihuan Chen, Xing Zhou, Lifei Zhu, Xueqian Wang, and Kai Zhang. 1. x-distill: Breaking the diversity, quality, and efficiency barrier in distribution matching distillation. arXiv preprint arXiv:2604.04018, 2026. 1, 2, 3, 6, 9, 15arxiv.org/abs/2604.04018
  26. [26]Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, and Lu Jiang. Diffusion adversarial post-training for one-step video generation. arXiv preprint arXiv:2501.08316, 2025. 3arxiv.org/abs/2501.08316
  27. [27]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 7, 15arxiv.org/abs/1405.0312
  28. [28]Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. 1arxiv.org/abs/2210.02747
  29. [29]Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503, 2023. 1arxiv.org/abs/2301.12503
  30. [30]Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, pages 5775–5787, 2022. 1, 7arxiv.org/abs/2206.00927
  31. [31]Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023. 1arxiv.org/abs/2310.04378
  32. [32]Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36:76525–76546, 2023. 3, 7arxiv.org/abs/2305.18455
  33. [33]Yihong Luo, Tianyang Hu, Jiacheng Sun, Yujun Cai, and Jing Tang. Learning few-step diffusion models by trajectory distribution matching. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 17719–17728. IEEE, 2025. 3arxiv.org/abs/2503.06674
  34. [34]Alexander Mathiasen and Frederik Hvilshøj. Backpropagating through fréchet inception distance. arXiv preprint arXiv:2009.14075, 2020. 3
  35. [35]Youssef Mroueh and Tom Sercu. Fisher gan. Advances in neural information processing systems, 30, 2017. 3
  36. [36]Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 11400–11410. IEEE, 2022. 15arxiv.org/abs/2104.11222
  37. [37]William Peebles and Saining Xie. Scalable diffusion models with transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4172–4182. IEEE, 2023. 1arxiv.org/abs/2212.09748
  38. [38]Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2): 3, 2022.arxiv.org/abs/2204.06125
  39. [39]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 10674–10685. ieee, 2022. 1, 6, 9, 10, 15, 16arxiv.org/abs/2112.10752
  40. [40]Amirmojtaba Sabour, Sanja Fidler, and Karsten Kreis. Align your flow: Scaling continuous-time flow map distillation. Advances in Neural Information Processing Systems, 38:146459–146512, 2025. 3arxiv.org/abs/2506.14603
  41. [41]Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 3, 7arxiv.org/abs/2202.00512
  42. [42]Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016. 3arxiv.org/abs/1606.03498
  43. [43]Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Neural Information Processing Systems, 34:17480–17492, 2021. 3arxiv.org/abs/2111.01007
  44. [44]Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. In SIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024. 2arxiv.org/abs/2403.12015
  45. [45]Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision, pages 87–103. Springer, 2024. 3arxiv.org/abs/2311.17042
  46. [46]Christoph Schuhmann. Clip+mlp aesthetic score predictor. https://github.com/christophschuhmann/improved-aesthetic-predictor, 2022. 6, 15
  47. [47]Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278–25294, 2022. 6, 7
  48. [48]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 1, 7arxiv.org/abs/2010.02502
  49. [49]Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In International Conference on Learning Representations, pages 15078–15097, 2024. 7arxiv.org/abs/2310.14189
  50. [50]Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023. 1, 3, 7arxiv.org/abs/2303.01469
  51. [51]Nikita Starodubcev, Ilya Drobyshevskiy, Denis Kuznedelev, Artem Babenko, and Dmitry Baranchuk. Scale-wise distillation of diffusion models. In International Conference on Learning Representations, pages 146774–146804, 2026. 3arxiv.org/abs/2503.16397
  52. [52]Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel, Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bartoli, Emile Mathieu, Sergey Ovchinnikov, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyoung Baek, and David Baker. De novo design of protein structure and function with rfdiffusion. Nature, 620: 1089–1100, 2023. 1DOI
  53. [53]Tianhe Wu, Ruini Li, Lei Zhang, and Kede Ma. Diversity-preserved distribution matching distillation for fast visual synthesis. arXiv preprint arXiv:2602.03139, 2026. 2arxiv.org/abs/2602.03139
  54. [54]Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023. 6, 15arxiv.org/abs/2306.09341
  55. [55]Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629, 2024. 6, 8, 9, 10, 15, 16arxiv.org/abs/2410.10629
  56. [56]Sirui Xie, Zhisheng Xiao, Diederik P Kingma, Tingbo Hou, Ying N Wu, Kevin Murphy, Tim Salimans, Ben Poole, and Ruiqi Gao. Em distillation for one-step diffusion models. Advances in Neural Information Processing Systems, 37:45073–45104, 2024. 7arxiv.org/abs/2405.16852
  57. [57]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imageareward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023. 6, 15arxiv.org/abs/2304.05977
  58. [58]Yuan Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8196–8206. IEEE, 2024. 3arxiv.org/abs/2311.09257
  59. [59]Jiawei Yang, Zhengyang Geng, Xuan Ju, Yonglong Tian, and Yue Wang. Representation fréchet loss for visual generation. arXiv preprint arXiv:2604.28190, 2026. 3
  60. [60]Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 37:47455–47487, 2024. 1, 3, 6, 7, 9, 10, 15, 16, 17arxiv.org/abs/2405.14867
  61. [61]Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Schechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6613–6623. IEEE, 2024. 1, 3, 6, 7, 9, 15, 17, 18, 19arxiv.org/abs/2311.18828
  62. [62]Liangyu Yuan, Ruoyu Wang, Tong Zhao, Dingwen Fu, Mingkun Lei, Beier Zhu, and Chi Zhang. Few-step diffusion sampling through instance-aware discretizations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 35882–35892, 2026. 7arxiv.org/abs/2603.17671
  63. [63]Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizadehnesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In Proceedings of the 40th International Conference on Machine Learning, pages 42390–42402. PMLR, 2023. 7arxiv.org/abs/2211.13449
  64. [64]Kaiwen Zheng, Yuji Wang, Qianli Ma, Huayu Chen, Jintao Zhang, Yogesh Balaji, Jianfei Chen, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Large scale diffusion distillation via score-regularized continuous-time consistency. In International Conference on Learning Representations, pages 2582–2603, 2026. 3arxiv.org/abs/2510.08431
  65. [65]Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Proceedings of the 41st International Conference on Machine Learning, pages 62307–62331. PMLR, 2024. 1, 3, 6, 7, 8, 10arxiv.org/abs/2404.04057
  66. [66]Mingyuan Zhou, Yi Gu, Huangjie Zheng, Liangchen Song, Guande He, Yizhe Zhang, Wenze Hu, and Yinfei Yang. Score distillation of flow matching models. arXiv preprint arXiv:2509.25127, 2025. 3, 6, 9, 15arxiv.org/abs/2509.25127
  67. [67]Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, and Hai Huang. Adversarial score identity distillation: Rapidly surpassing the teacher in one step. In International Conference on Learning Representations, pages 83214–83247, 2025. 1, 3, 6, 7, 10arxiv.org/abs/2410.14919
  68. [68]Zhenyu Zhou, Defang Chen, Can Wang, and Chun Chen. Fast ode-based sampling for diffusion models in around 5 steps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7777–7786, 2024. 7arxiv.org/abs/2312.00094

Paper details

vlnc:W_rxwkrcj2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 12

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents