WorksComputer Science
Correcting Flows with Marginal Matching
Abstract
Flow matching models, ODE-based generative models, generate samples by gradually morphing a simple source distribution into a target distribution. However, these models still face challenges in optimally balancing computational efficiency with generation fidelity. While predictor-corrector schemes have been proven to help address this issue in diffusion models effectively, their application to flow matching models remains under-explored. In this work we present a novel training-based predictor-corrector framework for flow matching, which exploits the fact that the ODE can be easily reversed. Our key idea, termed Marginal Matching, is to correct samples at a particular time step in the forward ODE towards a matching marginal distribution of the backward ODE. This scheme proves to reduce a bound on the distance between the data distribution and the learned distribution, motivating two different implementations for the correction function. We show that our proposed method improves sample quality on CIFAR-10 and ImageNet-64, with minimal overhead in computation time.
Introduction
Flow-based generative models have emerged as a promising alternative to diffusion models, offering faster sampling while maintaining high generation quality [27, 28, 47]. While these models require fewer sampling steps than diffusion probabilistic models (DPMs), they still face challenges in balancing computational efficiency with generation quality. This trade-off is crucial, as applications like image synthesis and audio generation require real-time performance, while protein design and robotics demand efficient resource utilization for complex tasks [55, 29, 16] (Le et al. 2024; Hu et al. 2023).
In DPMs, predictor-corrector schemes have proven effective at improving sample quality. These schemes combine prediction steps, which estimate the next time step’s sample, with correction steps that refine samples to better match the current time-step distribution. Previous work include both training-free methods [44, 57, 58], which do not require additional model training, and training-based techniques [26], which involve training a model specifically tailored to the task. However, applying these schemes to flow matching models remained relatively unexplored until recently. Song & Lai [42] demonstrated the viability of implementing a predictor-corrector scheme for flow matching models without requiring additional training.
Traditional predictor-corrector frameworks aim to refine a sample at time step of the generation towards the marginal distribution of the learned diffusion process at time . For flow matching, the marginal distribution at time is the push-forward of the initial Gaussian distribution by the ODE up to time . However, due to limitations of the neural network training, a learned flow does not exactly map Gaussian noise to perfect images, even with perfect integration. Here, we introduce a novel training-based predictor-corrector approach specifically tailored for flow matching models that leverages known marginal distributions to provably yield the exact data distribution. Our method exploits time reversibility, which has been explored in diffusion models [51, 33], but is arguably less involved in flow matching models as it only requires solving the ODE in reverse time [28].
Consider a reverse ODE, starting from the data samples and flowing back in time according to the learned flow model. Our main insight is that starting the generation from samples on this trajectory will generate perfect images. Building on this, we enhance generation quality through a predictor-corrector scheme that alternates between ODE solver steps (predictor) and learned correction steps, reducing the discrepancy between generated samples and the time-reversal ODE solution. We provide theoretical guarantees that such a correction function minimizes a bound on the Wasserstein distance [17] between the target approximation and the true target distribution, as a function of the error reduction of the corrections at different time steps. We hypothesize that for practical flow models, the forward and backward ODEs will be relatively aligned, making the correction easy to learn and apply with minimal overhead.
We examine two practical implementations for the correction function. The first implementation minimizes a theoretical bound, but has access to only subset of the data during training. The second has access to the full dataset, but makes no theoretical guarantee. We examine these correction models empirically and perform an extensive ablation study for our design choices. Our results show significant improvement with minimal additional steps.
In summary, our main contributions are as follows:
A novel corrector model for predictor-corrector sampling of flow matching models based on time reversal.
A theoretical analysis of this general framework, and two practical implementations.
We empirically show that our proposed correction models improve results on CIFAR-10 and ImageNet-64 datasets, achieving lower FID scores with fewer sampling steps.
Background
Flow Models
Given a source distribution and a target distribution , both over , flow matching (Lipman et al., [27]; Liu et al., [28]; Tong et al., [47]) models their mapping. This is done with a smooth time-dependent vector field which defines an ODE , where , and is the flow map with the initial condition . A time-varying density function that satisfies the continuity equation:
with the initial density over is called the marginal probability path generated by the vector field , and is said to be the probability flow ODE for the density function .
The flow matching (FM) objective regresses , a time-dependent vector field parametrized as a neural network with weights , to a target vector field via the following mean squared error (MSE) loss function:
Numerous mappings exist between probability distributions and . The distributions and are not unique and typically impossible to compute directly. To address this, Lipman et al. [27] introduced the conditional flow matching (CFM) loss, which provides equivalent gradients to Eq. (2) with respect to .
where is some conditional variable sampled from a distribution . The full derivation is available in Lipman et al. [27]; Tong et al. [47]. Many efficient parameterizations exist for and , in this work we focus on the parametrization described in Tong et al. [47]:
where and is the interpolation coefficient. The conditional density specifies one of the possible distributions and is easy to sample from and learn. The vector field and its corresponding marginal distribution can be obtained in terms of the conditional ones:
We examine two recently studied flow models by Tong et al. [47], one with independent coupling (I-CFM) and another with optimal transport coupling (OT-CFM) . The optimal transport (OT) problem maps to such that a displacement cost, (typically the squared Euclidean distance), is minimized, for more details see Appendix § 11. The OT coupling is calculated on batches during training and results in straighter trajectories, (Pooladian et al., [37]; Tong et al., [47]).
Score Models
In score-based generative models (Song et al., [44]; Song & Ermon, [43]) the forward and reverse processes are modeled using standard diffusion and reverse-time stochastic differential equations (SDEs). The forward process is described by , while the reverse-time SDE is given by . In these equations, represent standard Wiener processes, is the drift coefficient, is the diffusion coefficient, and ranges from 0 to . The reverse SDE is used for generating samples, evolving from to , for example by using a numerical SDE solver.
The score model estimates the score function , where is the target distribution defined above. The training process typically involves denoising score matching (DSM):
Here, denotes a neural network parameterized by , and represents the probability of a noisy sample from the target distribution. Vincent [50] demonstrated that the optimal score network , which minimizes Eq. (4), satisfies almost surely. This result shows that the network learns to estimate the score function of , which is crucial for effectively denoising it. However, the approximation holds true only when the noise level is sufficiently small, such that .
Predictor-Corrector Methods
Algorithm 1 General Predictor-Corrector Inference
Require: number of prediction discretization steps , number of correction steps
for to do
for to do
end for
end for
return
Predictor-corrector approaches, as shown in {(citep)}(Zhao et al., 2023; Lezama et al., 2022; Song et al., 2021b), are common in DPMs, where a corrector is intermittently applied during inference to improve performance (Algorithm 1). Previous work often employs the same pre-trained model for both prediction and correction, differing only in whether the model predicts the next step or refines the current one. For instance, Song et al. [44] use a score model with an ODE solver for prediction and Langevin Dynamics for correction. Meanwhile, Song & Lai [42] store flow model outputs in a buffer for use during correction. Unlike previous approaches, we propose using a dedicated correction model and investigate its design.
Method
Motivation and Problem Formulation
Given a frozen pre-trained flow model , our goal is to improve the sampling of . In this paper, we focus on the correction phase of the predictor-corrector framework and propose training an auxiliary correction model to enhance the approximation of the marginal distributions along the inference trajectory. We term our predictor-corrector method Marginal Matching (MM).
Langevin Dynamics (Song et al. [44]) is a common correction method, but its implementation becomes challenging with flow models that do not inherently represent a score function. A potential solution is to train an auxiliary score model to capture every intermediate marginal distribution. During prediction, the flow model advances to the next time-step, while the correction phase leverages the score model to more accurately approximate the marginal distributions. The key challenge lies in defining these marginals. An initial approach would be to use the marginal distributions the flow model was originally trained on— an interpolation between the target distribution and Gaussian noise. In the subsequent discussion, we will explore an alternative choice for these marginal distributions.
The flow model is trained to map a source distribution to a target distribution 1, but in practice it maps to an approximation of . Let denote the distribution of samples generated by , which does not exactly match .
If the vector field represented by is invertible, a well defined “reverse flow” exists from the target distribution according to the reverse of the vector field, . The key observation is that starting a forward flow using from any point on the reverse flow will converge to the target distribution . That is, we have found a trajectory of densities that works perfectly with our imperfect model . We assume that the flow is invertible, implying cycle consistency between forward and backward vector field mappings ([59]). A similar assumption is used in Liu et al. [28] in experiments on image translation. While we cannot verify that this assumption holds in practice, we empirically validate our method on popular datasets.
We next formally define the forward and backward marginals, representing the time-evolving probability distributions generated by the vector field.
Definition 3.1. Forward and Backward Marginals: Let be a continuously globally Lipschitz time-dependent function . Let be the reverse vector field defined as: . The discretization of the time-interval into steps is .
Forward Marginals are the probability path generated by from time 0 to 1, with initial distribution . The final distribution is . Their steps discretization is defined as .
Backward Marginals are the probability path generated by from time 1 to 0, with initial distribution . The final distribution is . Their steps discretization is defined as .2
We refer to as the approximate source distribution to distinguish it from , the source distribution, and as the approximate target distribution to distinguish it from , the target distribution. Figure 1 illustrates the forward and backward marginals of a flow model sampled with 10-steps Euler integration. It demonstrates the probability marginals trajectory and that for every . Additionally, there is a substantial intersection between the forward and backward marginals, and their symmetric difference represents the outliers that do not reach the target distribution, for more details see Appendix. F. Therefore, we predict that refining samples towards the marginals was trained on will be beneficial, though less so than backward marginals, as supported by the experiments.

Figure 1. Illustration of Marginals (a) Gaussian source distribution (red), and target distribution (blue). (b) Discrete forward marginals . (c) Discrete backward marginals . Note that for every , specifically , black) differs from , red).
Our next observation is that if the forward marginals trajectory reaches the target distribution, then it aligns with the backward trajectory for every time-step. This is formally stated in Lemma. 3.2
Lemma 3.2. Let be a continuous time-dependent vector field, satisfying: , , . Then, if and only if .
Based on this observation, we suggest refining the generated samples to better align with the backward marginals, ensuring that the final sample distribution more effectively matches the target distribution. Figure 2 illustrates this idea.

Figure 2. Marginals in Vector Field The black lines show two key trajectories in a vector field. The lower is the forward trajectory, starts at and ends at . The upper is the backward trajectory, begins at and ends at the data distribution . Both are discretized at time-steps and . Our objective is to transition from forward to backward marginals trajectory.
Practically, the backward marginals can be obtained by running an ODE-solver in reverse (from time 1 to time 0) on the frozen pre-trained model starting at the data samples, (see Appendix. D.1). Then, a correction model can be trained to bridge between the forward and backward marginals.
Correction Steps Prioritization
In the prediction-correction scheme, correction steps are typically performed after each prediction step. While these steps can enhance performance, they also increase computational cost. The following theorem indicates that not all correction steps are equally significant, suggesting that to balance performance and computation time, we should prioritize the correction steps that offer the greatest benefit.
Theorem 3.3. (Informal) Let be a well-behaved vector field with Lipshitz constant , and be two time-dependent probability density functions satisfying the continuity equation Eq. (1) with respect to . Suppose that
the initial Wasserstein distance is . At each time step , a correction function is applied to and the flow function continues from . Assuming reduces the distance to by , the reduction on the bound of the final distance is: .
A formal theorem statement and proofs are available in Appendix. B, which also includes calculations for two specific error reduction functions—additive and multiplicative. The following examples demonstrate that in each case the total error reduction is spread differently across the time steps, illustrating an important idea – we should prioritize the corrections applied in particular steps based on the error reduction model. We will revisit this idea in our experiments.
Example 3.4. Linear Reduction This corresponds to the case of linear error reduction in Thm. 3.3. For , the bound is: . Assuming , the reduction term at step 0 has a greater impact on the final bound due to the larger exponential factor. This highlights the importance of prioritizing corrections at earlier steps in the sequence.
Example 3.5. Multiplicative Reduction This corresponds to the case of multiplicative error reduction in Thm. 3.3. For , the bound is: . Here, the reduction terms and contribute equally to the final bound, irrespective of their position in the sequence. Thus, the optimal strategy is to prioritize improvements at steps with the largest .
Thm.3.3 states that the transport map can be utilized to reduce the distance between the forward and backward marginals . In the following, slightly abusing notation, we assume that is the push-forward of with a mapping . In practice, the specific error reduction is unknown and we can only require that the bound on the Wasserstein distance will decrease after is applied.
Let be a neural network with weights that approximates the correction function . We consider two models for implementing the correction model .
Correction Models Implementation
Score Model
Algorithm 2 Score Model Training
Require: flow model , score model , , backward marginals
repeat
Take gradient descent step on
until converged
Our first proposed approach is a time-dependent score model , presented in Algorithm 2, to approximate the backward marginals . We assume this score function is well-defined when evaluated on samples from the forward marginal distributions (i.e. ), which allows us to use it to move between these distributions (our experiments provide empirical support for this assumption). Motivated by the bound of Kwon et al. [24] (Thm. 2) for score models generation, the following lemma bounds the final Wasserstein distance after applying the score correction on a single time-step marginal:
Lemma 3.6. (Informal) Applying from to approximate as described in § 2.2 bounds the final distance as follows:
where depends on the score model’s loss as defined in § 2.2. For small enough , this bound improves upon using no correction. For more details and a bound on the general case—applying score correction on multiple marginals, see Appendix. § 12.
In practice, we noticed that a single correction step is enough, and sampling with Langevin Dynamics is unnecessary. This correction step can be done with any gradient that points in the direction of the backward marginals.
Robust Classifier
In the score model, we did not use data from the forward marginals during training. We next propose a method that explicitly transitions between samples from the forward and backward marginals. This approach utilizes a time-dependent classifier that distinguishes between forward and backward marginals ( and ), evaluating each sample independently. By learning to categorize samples as belonging to either the forward or backward marginal at each time step, the classifier’s gradients provide corrective guidance, effectively steering samples toward the backward marginals. The gradients will be zero upon reaching the correct class.
To enhance the classifier’s gradient performance, we implement two key improvements: Adversarial Training (AT) and Gradient Alignment (GA). These keep the classifier’s gradients stable and meaningful, for more details and the complete algorithm for training see Appendix. § 14.§ 6.
Usage for Class Conditioning Classifier guidance is used in diffusion models and more recently in flow models (Sun et al., [46]) in order to generate conditional images. While MM generates data samples, it can also extend to conditional data generation. For this purpose, our classifier can be conditioned where is a class label, and trained to categorize different classes in the backward trajectory. For an illustration see Appendix. § 17.§ 2.
Related Work
Diffusion predictor-corrector methods – Predictor-corrector methods have gained significant attention for balancing computational efficiency and sampling accuracy in diffusion processes. First introduced by Song et al. [44], this approach uses a score model for both prediction and correction, employing an SDE/ODE solver for prediction and Langevin dynamics to refine marginal distributions for correction. UniPC (Zhao et al., [57]) improved upon Lu et al. (2022) by leveraging a unified predictor-corrector framework that incorporates previous time-step predictions. Subsequently, Zhao et al. [58] introduced further refinements with a dynamic compensation strategy. These methods are all training-free, as they utilize the same model for both prediction and correction. For discrete data, Lezama et al. [26] proposed a method that uses pre-trained base models in conjunction with a learned corrector to enhance synthesis performance. Similarly, we adopt an approach that trains a correction model based on a pre-trained base model. However, it is important to note that while these methods are designed for DPMs, our work focuses on flow models, which operate under a different framework.
Acceleration methods for flow models – Acceleration methods for flow-based models have focused on improving ODE-solving efficiency. Concurrent with our work, Song & Lai [42] introduced a predictor-corrector method based on a training-free flow solver that minimized function evaluations by caching prior results and leveraging Taylor expansions to approximate ODE solutions efficiently. Rectified flow Liu et al. [28], demonstrated an acceleration approach through reflow operations to straighten probabilistic paths and further refined this process by training a distillation model over the final result. Moreover, Shaul et al. [39] proposed a training-based acceleration technique that adapts flow-based models into DPMs for faster computation. Despite these advancements, the exploration of ODE-based acceleration techniques specifically tailored for flow matching models remains underdeveloped compared to their DPMs counterparts due to their unique probabilistic path representation. For extended related work see Appendix. E.
Experiments
In this section, we empirically analyze the performance of MM’s image generation under varying marginals, comparing score correction model and classifier correction model. We then investigate the performance under interpolated and backward distributions, followed by an examination of the effects of sampling a score correction model using Langevin Dynamics versus a single-step approach. The score model is trained with a constant noise . We conduct experiments on two datasets: unconditional CIFAR-10 (Krizhevsky et al. [23]) and ImageNet-64 (Chrabaszcz et al. [3]; Deng et al. [6]). To assess the quality of generated images, we employ Fréchet Inception Distance (FID) (Heusel et al. [13]) score as our evaluation metric.
A key factor in diffusion and flow models is inference compute time, measured by number of function evaluations (NFE). In all tables, NFE represents the total number of sampling steps, and C-NFE denotes the steps taken with our correction model (out of the NFE). We note that a classifier correction step require two NFEs, due to input derivative calculations, while score correction requires only one NFE.
For the flow model we use the UNet architecture with the same hyper-parameters as Tong et al. [47]. For the score and classifier models, we use UNet with half the number of parameters used for the flow model. Additionally, these correction models undergo significantly fewer training iterations compared to the flow model. Unless otherwise specified, the correction models were trained on trajectories consisting of marginals with OT-CFM (Tong et al., [47]) as the flow model, and the flow model was sampled with 10-step RK4, which is 40 NFEs. For more implementation details, please refer to the Appendix § 18. The code will be available upon publication.
CIFAR-10 Sample Quality
Score Correction vs. Classifier Correction Model To showcase the improvement of MM we implement our score and classifier correction models with OT-CFM and I-CFM base flow models on CIFAR-10. Table 1 shows comparison of our results to (Ho et al., 2020; Song et al., [44]; Lipman et al., [27]; Tong et al., [47]).
| Model | Sampler | NFE ↓ | FID ↓ |
| DDPM | Adaptive | 274 | 7.48 |
| Score Matching | Adaptive | 242 | 19.94 |
| OT-FM | Adaptive | 142 | 6.35 |
| OT-CFM | RK4 | 40 | 4.34 |
| Adaptive | 133.94 | 3.58 | |
| I-CFM | RK4 | 40 | 4.29 |
| Adaptive | 146.42 | 3.66 | |
| Ours | NFE ↓ | C-NFE ↓ | |
| Score | 41 | 1 () | OT-CFM FID ↓ 3.45; I-CFM FID ↓ 3.47 |
| 43 | 3 () | OT-CFM FID ↓ 3.38; I-CFM FID ↓ 3.39 | |
| 51 | 11 () | OT-CFM FID ↓ 3.37; I-CFM FID ↓ 3.38 | |
| Classifier | 42 | 1 () | OT-CFM FID ↓ 3.57; I-CFM FID ↓ 3.77 |
| 46 | 3 () | OT-CFM FID ↓ 3.48; I-CFM FID ↓ 3.67 | |
| 50 | 5 () | OT-CFM FID ↓ 3.47; I-CFM FID ↓ 3.62 | |
| 62 | 11 () | OT-CFM FID ↓ 3.48; I-CFM FID ↓ 3.63 |
Table 1. CIFAR-10 performance. (Top) Leading baseline models. (Bottom) Our correction models (score and classifier) with OT-CFM and I-CFM base flow models. The base flow models are sampled with 10-step RK4 (40 NFE). Correction steps improve performance with minimal additional NFEs, and generally, increasing the number of applied corrections (C-NFE) enhances overall results. Additional comparisons are available in Appendix. G.6.
Previous works employ an adaptive sampler (DOPRI5, dor (1980)), while we use RK4 as the ODE-sampler to ease the integration of the correction steps. We compare with results reported in Lipman et al. [27], implemented using the same UNet (Dhariwal & Nichol, [7]), and those in Tong et al. [47]. As Tong et al. [47] did not report RK4 sampling results, we conduct these experiments and report baseline results using the models’ checkpoints as our foundational flow models. Following the results of § 4, we investigate applying the correction on various time steps (C-NFE > 1)3. We examine several options: for the score model the correction steps were taken on and for the classifier on .
We significantly outperform the baselines with minimal additional NFEs. The score model slightly outperforms the classifier in FID score, despite not being exposed to forward marginals during training. It also demonstrates consistent performance regardless of whether it is applied to OT-CFM or I-CFM, indicating a robust denoising ability from forward to backward marginals. Whereas, the robust classifier is sensitive to the model choice, showing superior performance with OT-CFM. We did not observe any benefit from applying both the score and classifier correction models together.
Figure 4 (Left) presents a qualitative comparison of the correction performed by each algorithm, (additional images are available in Appendix § 19). The correction models improve clarity and sometimes make significant changes to the image. The score and classifier corrections show varying levels of effectiveness depending on the specific image. In some cases, one method outperforms the other, while in other instances, they surprisingly produce similar results.

Figure 4. Correction models with OT-CFM as the base flow model on CIFAR-10 (Left) and ImageNet-64 (Right). The ”-Diff” shows the difference between corrected and uncorrected images. Corrected images show improved sharpness and better alignment with the dataset.
Interpolated Marginals vs. Backward Marginals Figure 3 compares two correction approaches: refining toward interpolated versus backward marginals using the score model. Both corrections improve results, but correcting toward backward marginals performs better as they guide the process toward the true target distribution . Using all correction steps for marginal interpolation steered the flow off course, resulting in degraded performance.

Figure 3. CIFAR-10 sample quality comparison of score correction trained on the interpolated ( and ) vs. backward marginals . The base flow is sampled with 10-step RK4.
Langevin Dynamics vs. One-Step Sampling
Figure 5 compares two approaches for the correction score model trained on backward marginals: using multi-step Langevin Dynamics (with 50 and 100 steps) versus applying a single correction step at the 10th time-step marginal (as used in Table. Table 1). The multi-step Langevin Dynamics approach exhibited limitations - small step sizes resulted in minimal image changes, while larger steps introduced noise. The single correction step proved both computationally more efficient and produced better results.

Figure 5. CIFAR-10 sample quality of score correction sampled with 50 and 100 step Langevin Dynamics versus a single correction step at the 10th time-step marginal. The base flow is sampled with 10-step RK4.
ImageNet-64 Sample Quality
We perform an experiment on ImageNet-64 to examine MM in a higher-dimensional settings. Given a pre-trained OT-CFM flow model trained on ImageNet-64, we train a score correction model to improve its sample quality, as the score model outperform the classifier on CIFAR, we chose to perform these resource-intensive experiments with it. The score model was trained on trajectories consisting of marginals. Table 2 presents the improvement in FID of a correction model trained on the backward marginals trajectory compared to Ho et al., 2020; Song et al., [44]; Lipman et al., [27]; Tong et al., [47]. Similarly to the CIFAR-10 results, we observe that MM improves FID with minimal addition of NFEs. The correction is qualitatively demonstrated in Figure 4 (Right), (for more illustrations see Appendix § 19). The correction model removes the residual noise in the model’s approximation and changes the images structure to better align with the ImageNet dataset photos.
| Model | NFE ↓ | Sample | FID ↓ |
| DDPM | 264 | Adaptive | 17.36 |
| Score Matching | 441 | Adaptive | 19.74 |
| OT-FM | 138 | Adaptive | 14.45 |
| OT-CFM | 20 | RK4 | 17.99 |
| 24 | RK4 | 16.60 | |
| 28 | RK4 | 15.71 | |
| Ours | NFE ↓ | C-NFE ↓ | FID |
| Score | 21 | 1 () | 15.91 |
| 22 | 2 () | 15.53 | |
| 26 | 6 () | 15.57 |
Table 2. ImageNet-64 results for top generative models (Top) vs. score correction with OT-CFM base flow (Bottom), which is sampled with 5-step RK4 (20 NFE).
Ablation Study
Marginals Ablation
In this ablation we study the effect of applying correction on different time-steps. Figure 6 (Left) demonstrates the cumulative effect of correcting all time-step marginals (from 0 to 10). The score model demonstrates significant improvements primarily during the middle and final steps, with minimal contributions from early stages. In contrast, the classifier exhibits a more uniform pattern of enhancement. Figure 6 (Middle) presents the improvement in FID when applying a single correction step on different time-step marginals. The classifier model’s correction consistently improves more than the score model at every time-step, except the final one. As predicted by the theoretical analysis in § 4, we observe that different correction algorithms yield varying levels of improvement when applied at different time steps.

Figure 6. CIFAR-10 sample quality. The base flow model is OT-CFM sampled with 10-step RK4. Left: FID vs accumulative correction time-steps. Middle: FID vs single correction time-step. Right: Ablation study on classifier loss components sampled across different marginals.
Notably, early-stage corrections mainly affect low-frequency components, shaping overall image structure. In contrast, later-stage corrections primarily impact high-frequency details, addressing noise-like elements (Kim et al., [19]). The striking impact of executing the last time step with the score model suggests residual noise in the model’s predictions (see Appendix § 19 for illustrations).
Classifier Loss Ablation
To assess the significance of each improvement to the classifier, namely Adversarial Training (AT) and Gradient Alignment (GA), we conduct an ablation study by removing these components. The results are presented in Figure 6 (Right). Without AT, the gradients become unstable, as seen from the inconsistent improvement with additional steps. Without GA, the gradients are small, and adding steps does not improve performance.
For additional results, see Appendix. § 17 and Appendix. § 14.§ 3.
Conclusion and Limitations
This paper introduces a novel predictor-corrector framework for flow matching models, featuring a dedicated correction model. Our main insight is that given an imperfect flow model, a trajectory that exactly reaches the data distribution can be computed by reversing its vector field. Based on this insight, we propose Marginal Matching, a simple algorithm that steers the inference-time trajectory to better align with the path that accurately reaches the data. Unlike conventional predictor-corrector, which typically refine samples by aligning with the interpolated marginal distribution at each step, our approach directs samples toward a marginal distribution that steers toward the true data distribution. We demonstrate superior performance on CIFAR-10 and ImageNet-64, along with an extensive ablation study.
Ideally, all generative models could potentially improve with a correction phase where refinement targets a marginal distribution that leads to the data distribution. A major limitation of our work is the assumption on the reversibility of the vector field, constraining our method from generalizing to non-reversible models. This limitation may be relaxed for approximately reversible models such as diffusion models [51]. In general, extending our approach to non-reversible models is an interesting future direction.
Another aspect is the practical implementation of our theory. In Thm. 3.3 we assumed knowledge of error reductions. In practice, we do not have access to the model’s error reduction (see § 4). However, we can bound that reduction in the case of score correction (Lemma. 3.6) and propose a bound on the final Wasserstein distance. Future work could explore theoretical bounds also for our classifier-based correction model.
Finally, our method requires training an additional model after a flow model has been pre-trained, adding extra compute and parameters. It is worth noting that in all our experiments, this auxiliary model is smaller than the original flow model, and training it takes significantly less time. An alternative approach could involve training the flow and correction models together, either as a single model with both flow and correction outputs or as two models trained simultaneously.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Optimal Transport
Optimal Transport (OT) is a mathematical framework that addresses the problem of efficiently moving probability mass from one distribution to another while minimizing a certain cost. The central idea is to find the optimal way to transform one probability distribution into another, considering the ”cost” of moving each unit of mass.
In the context of flow matching, before learning the flow, an optimal transport map is computed between batches of samples from and . This OT step uses the 2-Wasserstein distance as its metric, with a cost function defined as the Euclidean distance between points. The flow is then learned based on these optimally transported samples, resulting in straighter trajectories [37, 47].
Theoretical Proofs
Proof of Lemma 3.2. If for every then is trivial.
Assume . We need to show that for every .
The flow is a diffeomorphism for each due to the properties of the vector field (continuous and globally Lipschitz).
For any , we can write:
where is the inverse map of flowing from to . Since the following equality holds: , we can replace and with in the above equations: .
Lemma B.1. Let be the flow map of the vector field that induces the push-forward , for any two probability density distributions and over , where . The 2-Wasserstein distance between and satisfies:
Corollary B.2. Assume : .
Proof. Let be the optimal coupling between and . Denote the push-forward of as . Then:
where in (5) we changed variables, in (6) we used Leibniz’s rule and in (8) the Lipschitz constraint.
Using Grönwall’s inequality, we have:
By the definition of Wasserstein distance:
Substituting the bound:
Taking the square root of both sides yields the result:
Theorem B.3. [Thm.3.3 in the main paper] Let be the space of probability densities on with finite second moments. Let be defined as in Lemma. 3.2. Let and be two time-dependent probability density functions in satisfying the continuity equation Eq. (1) with initial densities , and terminal densities , . Suppose that the initial 2-Wasserstein distance between them is . The time interval is discretized into equal steps: , where . At each time step , a time-dependent transport map is applied to . After applying , the flow function at time continues from the probability density of . We consider two variations for :
A If reduces the 2-Wasserstein distance by a constant amount at each step: . Then, the following bound hold for : ,
B If reduces the 2-Wasserstein distance to a proportion of the current distance: where for all . Then, the following bound hold for : .
Proof of Theorem B.3. We will prove parts A and B of the theorem separately.
Part A:
Let . We will prove by induction that:
Base case (): Trivially true as .
Inductive step: Assume the inequality holds for . We’ll prove it for .
After applying at , we have:
By Lemma. B.1 over the interval , we have:
Substituting the inductive hypothesis:
This completes the induction. The final bound at follows by setting .
Part B:
Again, let . We will prove by induction that:
Base case (): Trivially true.
Inductive step: Assume the inequality holds for . We’ll prove it for .
After applying at , we have:
By Lemma. B.1, over the interval we have:
Substituting the inductive hypothesis:
This completes the induction. The final bound at follows by setting .
Score Wasserstein Bound:
The score-based models losses are:
where is a positive weighting function. The score model is typically trained using the widely adopted loss function known as .
Definition B.4. Noisy Backward Marginal: Let be the noisy backward marginal. Then, , .
Theorem B.5 (Kwon et al., [24]). If satisfies:
then we have:
were . Additionally, and are defined as follows:
(A1) The drift coefficient is Lipschitz continuous in the space variable : there exists a positive constant , depending on , such that for all
(A2) satisfies the one-sided Lipschitz condition [14, Definition 2.1]: there exists a constant , depending on , satisfying for all
For more details, see Kwon et al., [24]. In our case and , the backward and forward marginals respectively. We assume the forward marginals are a ”noisy” version of the backward marginals (Def. B.4) with a small . Thus, and the first time the score model is applied, (the starting point is on the forward marginal). Consequently, as the loss converges to zero, so is the bound on the Wasserstein distance.
Conditions: under the following sufficient conditions: (1) Lipschitz continuity of the drift function for the forward diffusion process, and (2) boundedness of the noise schedule. Our scenario satisfies these conditions as follows:
The drift function is zero, which is Lipschitz continuous.
The noise schedule is between and , thus it is bounded above and below.
Therefore, we meet these sufficient conditions and adopt the additional assumptions from Kwon et al. [24], enabling us to apply Thm. B.5.
Lemma B.6. Let be a continuous time-dependent vector field, satisfying the Lipschitz condition: for any and , . Let and represent the backward and forward marginals respectively, where the corresponding marginals continuous time is . Let denote the trained score model. Then, applying from to as described in § 2.2 lowers the final bound on :
Proof of Lemma 3.6. Denote . After applying the score models :
where is the score model’s approximation and , are the integration times of the score model and . According to Lemma. B.1:
where the second inequality is the bound without applying the correction step.
The final bound can be obtained by substituting the bound on :
Assuming the score model is trained, is negligible: . Since the forward marginals are a noisy version of the backward marginals . In total, the final bound after applying a correction step is lower than the bound when applying no correction steps:
Theorem B.7. Let be a continuous time-dependent vector field, satisfying the Lipschitz condition: for any and , . Denote , where refers to the loss of on the marginal. Following Lemma. 3.6 and assuming that for every where is the initial distribution of the correction of the score model, then the application of to multiple marginals upper bounds the final Wasserstein distance. This can be expressed as:
Under the assumption that the forward marginals are a noisy version of the backward marginals when , the inequality: is trivial. After applying a correction step and continuing the flow, the trajectory at the next time-step lies between the forward and backward marginal paths, rather than strictly on the forward marginals path (). This inequality holds true provided that decreases at a rate corresponding to this intermediate positioning.
Corollary B.8. Assuming the score model is well-trained, becomes negligible. In our case, the drift is , resulting in a negligible . Furthermore, [24] showed that with a DDPM schedule and , the one-sided Lipschitz constant is . This can be generalized to a VE schedule. We employ such a schedule, but lack a closed form for . Assuming holds in our case as well, such that , specifically . Applying more correction score steps decreases the upper bound of the final Wasserstein distance by , similarly to the multiplicative case (B) in Thm. B.3.
Proof of Theorem B.7. Denote as . We will prove by induction that:
Base case (): Trivially true.
Inductive step: Assume the inequality holds for . We’ll prove it for .
By Lemma. 3.6, after applying the score function:
in the second inequality we used the assumption that . By Lemma. B.1 over the interval , we have:
Substituting the inductive hypothesis:
This completes the induction. The final bound at follows by setting .
Marginal Matching Inference
Algorithm 3 presents our proposed approach. The correction steps using are taken before each step of the flow model and at the end. For clarity, the algorithm assumes Euler integration for sampling and single correction steps, though multiple corrections per time-step are possible. Adding Gaussian perturbation before each correction step, except the last one, improves performance. Algorithm 3 Marginal Matching
Require: flow model , correction model , number of iterations , step size of corrector steps , scale added noise
for do
{Correction}
{Flow}
end for
{Correction}
return
Practical Considerations
Backward Marginals Generation
Once a flow model is defined, we can generate samples from the backward marginal distributions using any ODE solver to integrate from time to . We present two example implementations in Algorithm 4 and Algorithm 5, using RK4 and Euler integration methods respectively. The intermediate states at each time-step correspond to samples from the backward marginal distributions at those times (e.g., at we sample from , and at we sample from ).
Algorithm 4 Backward Marginal Generation with RK4 Integration
Require: Initial condition , number of steps , flow model
{Negative step size for backward integration}
{Start time}
{Initial value}
for to do
end for
return {Solution at }
Algorithm 5 Backward Marginal Generation with Euler Integration
Require: Initial condition , number of steps , flow model
{Negative step size for backward integration}
{Start time}
{Initial value}
for to do
{Euler step}
end for
return {Solution at }
Practical Considerations of
When designing the learning objective for an important issue should be taken into consideration. The pre-trained flow model has established a matching between each distribution on the forward trajectory and between each distribution on the backward trajectory across different time-steps. Additionally, there exists a matching between and . Through transitivity, this implies a pairing between and for every time step . During the training process, samples from and are accessible. Models that try to establish pairing between these distributions will yield sub-optimal results, since the correct pairing is unknown. Thus, we opt for different generative models such as score and classifier.
Alternative Correction Models: Alternative correction models include the more sophisticated score models for traversing between forward and backward marginals, provided the noise level for the forward marginal is known. A supplementary model can be trained to predict the noise level, thereby assisting the score model. Additionally, Energy-Based Models (EBMs) (Du et al., 2021), could also be a good fit to explore the alignment of the sampling trajectory with the backward trajectory.
Approximations
Interpolating and
The correction model is trained on different time-step marginals. The marginals are generated and stored in advance to avoid running the ODE-solver multiple times while training . Assuming the flow model’s trajectories are sufficiently straight, we can save this extra storage by approximating the backward trajectory through interpolation between , the approximate source distribution, and , the target distribution: . This reduces storage requirements to just the data and approximate source distribution, making the dataset size twice rather than times the original size. Table 3 presents the results of the interpolated backward trajectory on CIFAR-10. The score model’s performance remains steady, while the classifier exhibits degradation in its performance. Table 4 compares the results of a score correction model on ImageNet-64 trained on the backward marginals versus the approximated marginals. The model trained on approximated marginals shows improved results, albeit not as significantly as the non-approximated version.
| Correction Model | NFE | C-NFE | OT-CFM FID | I-CFM FID |
| No Correction | 40 | 0 | 4.34 | 4.29 |
| Classifier | 42 | 1 () | 3.57 | 3.77 |
| Classifier | 46 | 3 () | 3.48 | 3.67 |
| Classifier | 50 | 5 () | 3.47 | 3.62 |
| Classifier | 62 | 11 () | 3.48 | 3.63 |
| P-Classifier | 40 | 1 () | 3.65 | 3.75 |
| P-Classifier | 40 | 3 () | 3.60 | 3.73 |
| P-Classifier | 40 | 5 () | 3.59 | 3.67 |
| P-Classifier | 40 | 10 () | 3.74 | 3.80 |
| Classifier (Backward Marginals Interpolation) | 42 | 1 () | 3.77 | 3.74 |
| Classifier (Backward Marginals Interpolation) | 46 | 3 () | 3.72 | 3.62 |
| Classifier (Backward Marginals Interpolation) | 50 | 5 () | 3.67 | 3.69 |
| Classifier (Backward Marginals Interpolation) | 62 | 11 () | 3.68 | 3.67 |
| Score | 41 | 1 () | 3.45 | 3.47 |
| Score | 43 | 3 () | 3.38 | 3.39 |
| Score | 51 | 11 () | 3.37 | 3.38 |
| P-Score | 40 | 1 () | 4.29 | 4.27 |
| P-Score | 40 | 3 () | 4.08 | 3.99 |
| P-Score | 40 | 10 () | 4.08 | 3.99 |
| Score (Backward Marginals Interpolation) | 41 | 1 () | 3.46 | 3.47 |
| Score (Backward Marginals Interpolation) | 43 | 3 () | 3.38 | 3.40 |
| Score (Backward Marginals Interpolation) | 51 | 11 () | 3.41 | 3.41 |
Table 3. Comparison of two variants of classifier and score correction models on CIFAR-10. First, a correction model that runs in parallel with (“P-”) the flow model. Second, a correction model that learns an interpolation of the data and approximate source distribution. The base flow models are sampled with 10-step RK4 (40 NFE). The classifier succeeds in improving the FID even with the parallel approximation, while the score shows less improvement. Training with interpolated backward marginals improves the FID score, though less than using their exact version.
| Correction Model | NFE | C-NFE | FID |
| Score | 21 | 1 () | 15.91 |
| Score | 22 | 2 () | 15.53 |
| Score | 26 | 6 () | 15.57 |
| Score (Backward Marginals Interpolation) | 21 | 1 () | 15.89 |
| Score (Backward Marginals Interpolation) | 22 | 2 () | 15.62 |
| Score (Backward Marginals Interpolation) | 26 | 6 () | 15.69 |
Table 4. Comparison of backward marginals and approximated marginals results on ImageNet-64 using OT-CFM as the base flow model, sampled with 5-step RK4 (20 NFE). Both versions improve the original score, though the approximation’s benefit is smaller.
Parallel Sampling
The corrected inference process, as described in Algorithm 3, introduces additional sampling time for each correction step. To reduce the overhead, we propose a correction algorithm that is parallelizable in principle, by running the correction and the flow model in parallel. The correction and the flow model both operate on the same input at the same time, then the correction and the flow model’s direction are summed, approximating the full correction process. While this is an approximation, the effectiveness of this approximation relies on the assumption that and do not change too rapidly.
Figure 7 presents the approximation of parallel sampling the correction and flow model. The algorithm for parallel sampling is presented in Alg. 6. For clarity, the algorithm assumes Euler integration as the ODE-solver. To maintain computational efficiency, a single correction step is implemented, ensuring that the total computation time remains equivalent to that of the uncorrected method.

Figure 7. Parallel Sampling The black arrows denote the path of first flowing with and only then taking a correction step with . In contrast, the dashed green arrow shows the approximation obtained by summing the direction of the flow model (black arrow), and the correction model (green arrow) on the current step.
This algorithm is parallel in principle. Cuda has a queue, where the CPU sends tasks to be run on the GPU. The GPU may execute tasks in parallel that are independent of one another, such as our parallel sampling. The CPU may wait when the queue is full or during synchronization events, (i.e. item(), synchronize(), etc.). Given a powerful enough Cuda machine, the models could be run in parallel on different GPUs or using advanced parallelism techniques. Parallel calculation on separate devices could be useful when the correction calculation takes more time than the overhead of moving between devices, as is the case for large models. The extent of parallel processing viability will depend on the specific hardware and infrastructure available.
Table 3 presents the complete results on CIFAR-10 of applying the correction models (score and classifier) in parallel with the flow model, (to save computation time), and with an approximation of the backward marginals (to save storage during training).
Score Model Training
Score models are generative AI systems that transform random noise into meaningful data through iterative denoising. At their core is the score function - the gradient of the log probability density - which guides samples toward higher likelihood regions of a target distribution.
These models utilize varying noise levels () during generation. At high noise levels, samples differ significantly from the target distribution, but the score function provides stable gradients far from the data manifold. At low noise levels, as samples approach the target distribution, the score function enables detailed refinement. During generation, the model progressively reduces noise levels while following each corresponding score function, establishing a path between random noise and complex data distributions.
By assuming the forward marginals are a noisy version of the backward marginals (with a small ), a score model can be used to transverse between them.
While sophisticated approaches like annealing score models with time-varying are widely used for generation, they proved unsuitable for our needs. This is due to our inference process that begins at a forward marginal (or an intermediate point), where we lack information about the current noise level, precluding the effective use of such models.
Robust Classifier Training
Adversarial Training (AT): Srinivas & Fleuret [45] demonstrated that the gradients of standard classifiers can be arbitrarily altered without impacting their cross-entropy loss or accuracy. Building on this insight, Kawar et al. [18] proposed using gradients from a robust classifier for guidance, rather than those from a conventional classifier. The gradients of robust classifiers are resistant to arbitrary manipulation as a results of the adversarial attack used in their loss computation is directly dependent on the model’s gradients. An intriguing characteristic of robust classifiers is that their gradients have been shown to align well with human perception, as noted by Tsipras et al. [48]. When robust classifiers are employed to guide towards a specific class , they are anticipated to produce significant features that correspond well with the target class. As a result, the modifications applied to are likely to be visually convincing and aligned with human perception of the class characteristics. Inspired by these works we enhance the classifier with Projected Gradient Descent (PGD) attack and Adversarial Training (AT) robustification method (Madry et al. [32]), with the PGD pseudo-algorithm detailed in
Algorithm 6 Parallel Marginal Matching
Require: flow model , correction model , number of iterations , step size of corrector steps , scale added noise
for do
{Correction}
{Flow}
end for
return
Alg. 7. Our findings indicate that adversarial training stabilizes gradients and enhances their meaningfulness.
Algorithm 7 Targeted Projected Gradient Descent
Require: classifier , input , class , number of iterations , step size , radius , loss function ,
for do
end for
Return
Where is a projection operator that keeps ’s norm below so the adversarial example will not stray too far from the input; and the loss function in our case is cross entropy.
Gradient Alignment (GA): Inspired by Yadin et al. [54] and Song et al. [44], we incorporate a cosine-similarity loss term for the gradient. This aims to align the classifier’s gradient direction with the backward marginals when in close proximity. We artificially move away from the backward marginals by introducing small Gaussian perturbations to sampled points, and training the classifier gradient to point towards the original samples by aligning them with the negative direction of the noise.
The class for the forward marginals is represented by 0 and for the backward marginals by 1.
Algorithm 8 Robust Classifier Training
Require: classifier , forwards marginals , backward marginals , loss weights , scale of noise, adversarial step size , adversarial number of steps , adversarial radius
repeat
\ell_{CS} = cosine\mathchar`-similarity(c_n,-\epsilon)
Take gradient descent step on
until converged
Extended Related Work
Different sampling trajectories – Previous work explored different sampling trajectories primarily by studying various source distributions. In Denoising Diffusion Probabilistic Models [40] (Ho et al. 2020), various source distributions proved to decrease inference time while maintaining quality. gil Lee et al. [11] proposed using a Gaussian prior distribution with parameters derived from data statistics. Lyu et al. [31]; ); Popov et al. [38] trained a model for prior distribution using an encoder or VAE [22] and incorporated it into the standard diffusion process by adding noise. Other studies have shown that the prior distribution need not be Gaussian [2, 12]. Flow based models [27, 47, 28], a new family of generative models, generalized the mapping to include general source distributions.
A work aligned with ours is that of Xu et al. (2024) as they consider the learned errors throughout the whole sampling trajectory. They use a sequence of NF blocks to learn a sequential transformation from Gaussian noise to data, where each block represents a different time-step. This requires training the blocks sequentially, as the learned error in each block is accounted for in the next block, slowing the training process. The sampling process initiates from Gaussian noise, where the model’s error is not accounted for. In contrast, our work differs in that we focus on correcting the inference-time trajectory rather than the training trajectory, and we enhance flow matching models instead of NF.
Training and inference mismatch – Previous work recognized training and inference mismatch in different settings. In score models [44, 43], the source distribution during inference is , which is intended to approximate the ”noisiest” training distribution (). However, as these distributions are not identical, starting from will yield sub-optimal results. Two different solutions were proposed to minimize this gap as a pre-sampling procedure: Franzese et al. [10] suggested an auxiliary model that learns the mapping from to . Alternatively, Pedrotti et al. [36] proposed to use Langevin dynamics [52], leveraging the score function .
Even if the source distribution during training and inference is the same, practically, the actual distribution that reaches the data is different. Coeurdoux et al. [4] leveraged the invertibility property of normalizing flow (NF) models (Kingma & Dhariwal [21]; Dinh et al. [8]) to reach higher probability samples in , which are more likely to belong to . They achieved this by employing MCMC algorithm, where the score is computed using the Jacobian of the inverse mapping. This discrepancy between the learned and actual source distribution is common in generative models like VAEs (Kingma et al., [20]), which assume a Gaussian prior during inference. (Dai & Wipf, [5]) addressed this by learning an additional VAE model to predict the actual prior distribution, thereby removing the Gaussian assumption, which lead to improved results.
Inverse diffusion – Previous work researched inversion of diffusion models, which are ”approximately invertible” models, particularly in the context of image editing. DDIM inversion (Song et al., [41]) method laid the groundwork for this approach, enabling the reversal of the diffusion process to obtain latent representations of images by utilizing a deterministic forward process. Building upon this foundation, Wallace et al. [51] improved the efficiency of the inversion by training an encoder network to directly map images to their corresponding noise representations in the diffusion process. Furthermore, Mokady et al. [33] enabled precise edits on real images through a text-guided approach, using a null-text optimization to find an optimal noise that, when denoised, produces the target image. Zhang & Kleijn [56] improved the accuracy of the inversion process by combining forward and backward trajectories to minimize approximation errors, while Pan et al. [35] focused on enhancing both the speed and quality of image editing operations by iteratively refining the inverted latent code.
Forward and Backward Trajectory Comparison
We illustrate the distinction between the forward and backward marginals of OT-CFM flow model (Tong et al., [47]) trained on the CIFAR-10 dataset and sampled with 10-step RK4. Figure 10 presents a visual representation of these marginals at each time-step , created with t-SNE (Van der Maaten & Hinton [49]) in order to reduce the marginals’ dimensionality to two. Figure 9 offers an alternative visualization, where the t-SNE is applied to the classifier correction model’s features of the marginals at each time-step. The visualizations demonstrate that as increases there is a trend of growing similarity between the forward and backward marginals. However, this pattern of convergence does not extend to the final time step (), where the comparison is between the actual data and the flow model’s approximation of it.

Figure 9. TSNE of Classifier Features Red represents the forward marginals, while blue denotes the backward marginals. The classifier’s features can differentiate between the forward and backward marginals in most cases. However, in some instances, these marginals are indistinguishable from one another.

Figure 10. TSNE of Forward and Backward Marginals Red represents the forward marginals, while blue denotes the backward marginals. As time progresses, we observe a convergence between these two sets of marginals. However, it’s important to note that despite this increasing proximity, they never achieve perfect alignment or identity. compares the approximate source distribution distribution () and Gaussian noise while compares the model’s data approximation () and the real data.
Reinforcing this observation, Figure 8 examines the intersection of forward and backward marginals by presenting a histogram of the differences in VGG5 features between uncorrected and corrected samples. The distribution reveals that for both the classifier and score models, the majority of images exhibit minimal changes. This pattern indicates that corrections are primarily applied to images requiring adjustment, suggesting a close alignment between the forward and backward marginals. The selective nature of these corrections implies that the models effectively identify and address outliers, refining the overall distribution while leaving well-formed samples largely unchanged.

Figure 8. Percentage vs difference. Histogram showing differences of VGG features between samples generated using a flow model alone versus with a correction model. The samples are generated from identical Gaussian noise. The percentage indicates the proportion of images with the corresponding difference. The score and classifier correction models leave most images largely unchanged, with differences concentrated in a small range, while significantly altering only outliers. This suggests that there is an intersection between the forward and backward marginals.
Additional Experiments
Score Ablation Study
An ablation study on different values for the score model trained on CIFAR-10 is presented in Table 5. The correction step-sizes for different sigmas were multiplied by a constant value, as the score function is multiplied by sigma during evaluation. For all other evaluations, we employed the score model trained using , as it yielded superior performance compared to other values.
| Correction Model | NFE | C-NFE | FID |
| 0.001-Score | 41 | 1 | 3.83 |
| 43 | 3 | 3.75 | |
| 51 | 11 | 3.77 | |
| 0.003-Score | 41 | 1 | 3.51 |
| 43 | 3 | 3.44 | |
| 51 | 11 | 3.45 | |
| 0.005-Score | 41 | 1 | 3.45 |
| 43 | 3 | 3.38 | |
| 51 | 11 | 3.37 | |
| 0.01-Score | 41 | 1 | 3.49 |
| 43 | 3 | 3.42 | |
| 51 | 11 | 3.38 |
Table 5. Ablation study on the sigma value of score correction model sampled on marginals on CIFAR-10. The -Score represents the value of the score model was trained with. The base flow model is OT-CFM sampled with 10-step RK4 (40 NFE). The results demonstrate that the optimal performance is achieved when (the value used in the main paper).
CIFAR-10 Classifier Guidance
We implement classifier guidance as described in § 4.1.2 on CIFAR-10 dataset. Figure 11 presents images produced with our corrected inference algorithm that use the class-condition classifier, where the trajectory is steered toward the correct class in the backward marginals. On the left, the original images of the flow model are presented (with no correction), and on the right the images from the same source noise, but with correction steps toward the matching classes. Even when the classifier’s class matches the original class of the source noise it produces a different image of the same class, more closely aligned with the backward trajectory.

Figure 11. CIFAR-10 images generated by OT-CFM sampled with 10-step RK4, shown alongside their counterparts produced with class conditioned classifier correction. The classifier successfully directs the noise to the correct class.
Flow as Correction Model
In this experiment we perform MM with a flow model (OT-CFM) as the correction model. The correction is performed only on the first and last marginals with a different number of correction steps, see Table. Table 6. The correction of the first marginal is done between Gaussian noise () to the approximate source distribution (), and on the last marginal from the approximate target distribution () to the target distribution (). The first row in the table presents the FID result with no correction model, but only 10-step RK4 sampling of the OT-CFM base flow model.
| Correction Model | NFE | C-NFE | FID |
| No Correction | 40 | 0 | 4.34 |
| OT-CFM – Marginal | 48 | 2 | 9.74 |
| OT-CFM – Marginal | 64 | 6 | 6.31 |
| OT-CFM – Marginal | 120 | 20 | 6.33 |
| OT-CFM – Marginal | 48 | 2 | 7.43 |
| OT-CFM – Marginal | 64 | 6 | 4.39 |
| OT-CFM – Marginal | 120 | 20 | 4.41 |
Table 6. CIFAR-10 FID comparison of OT-CFM correction and flow model trained on first and last marginals. The correction score model was sampled with C-NFE RK4 steps, while the base flow model was sampled with 10-step RK4 (40 NFE). The correction flow model degrades the results.
The flow model only degrades the results. We hypothesize that this is due to the pairing problem, for more details refer to Appendix. § 14. That led us to explore other correction models - score and robust classifier.
CIFAR-10 Test Sample Quality
The performance of OT-CFM flow model with and without correction on CIFAR-10 test set is presented in Table 7. The correction model helps the generation quality, even though it was trained to match the training data marginals. Where the C-NFE is greater than one, the correction steps were taken on different time-step marginals; we examine several options: for the score model the correction steps were taken on and for the classifier on , same as Table 1.
| Flow Model | NFE ↓ | FID ↓ | |
| OT-CFM | 40 | 6.43 | |
| 44 | 6.04 | ||
| 48 | 5.80 | ||
| 52 | 5.70 | ||
| Correction Model | NFE ↓ | C-NFE ↓ | FID ↓ |
| Score | 41 | 1 () | 5.52 |
| 43 | 3 () | 5.46 | |
| Classifier | 42 | 1 () | 5.66 |
| 46 | 3 () | 5.57 | |
| 50 | 5 () | 5.56 |
Table 7. CIFAR-10 test set FID performance of OT-CFM flow model with and without our correction models (score and classifier). OT-CFM is sampled without correction using 10,20, and 50-step RK4. Our correction models use OT-CFM with 10-step RK4 (40 NFE). In general, adding correction steps (C-NFE) improves the results.
Residual Model
Another baseline we measure against is a straightforward approach to enhance sampling, assuming we have access to the data. This involves training a residual flow matching network using the standard flow matching objective. This leverages an increase in model size effectively. The training process involves using the frozen network , adding the output of the residual network (), and aiming to predict the vector field (refer to § 2.1). Table 8 shows the results of this approach. While it degrades the original performance, our method demonstrates improvements. We hypothesize that the residual flow matching objective is more challenging to optimize than our approach, leading to worse performance under equivalent training conditions and model sizes.
| NFE | FID ↓ |
| 40 | 4.44 |
| 44 | 4.09 |
| 48 | 3.79 |
| 52 | 3.56 |
Table 8. Results of residual networks combined with both the base flow model and the residual network OT-CFM flow models.
Flow Models RK4 Results
Table 9. CIFAR-10 FID scores for OT-CFM and I-CFM flow models sampled with RK4. The number of RK4 steps is 1/4 of the number of the NFEs.
Implementation Details
Flow Models Architecture
Our flow models architecture is based on the UNet design from [34] that was employed in [47] and Lipman et al. [27].
| CIFAR-10 | ImageNet-64 | |
| Channels | 128 | 192 |
| Depth | 2 | 3 |
| Channels multiple | 1,2,2,2 | 1,2,3,4 |
| Heads | 4 | 4 |
| Heads Channels | 64 | 64 |
| Attention resolution | 16 | 32,16,8 |
| Dropout | 0.1 | 0.0 |
| Batch size | 128 | 512 |
| Iterations | 400K | 600k |
| Learning Rate | 5e-4 | 1e-4 |
| Learning Rate Scheduler | Polynomial Decay | Constant |
| Warmup Steps | 5000 | 0 |
Table 10. Hyper-parameters used for CIFAR-10 and ImageNet-64 flow models.
Corrections Models Architecture
The correction models utilized the same UNet architecture with different hyper-parameters than the flow models. The classifier model required an additional convolution layer and two linear layers to reduce the output dimension to a single scalar. The correction models were trained for 100, 000 optimization steps, however we observed convergence within 30, 000 to 40, 000 steps. The models sizes are half the number of trainable parameters of the original flow models, requiring significantly less time to converge.
| Classifier | Score | ||
| CIFAR-10 | CIFAR-10 | ImageNet-64 | |
| Channels | 128 | 128 | 192 |
| Depth | 1 | 1 | 2 |
| Channels multiple | 1,2,1,2 | 1, 2, 1, 2 | 1, 2, 3, 2 |
| Heads | 4 | 4 | 4 |
| Heads Channels | 32 | 32 | 64 |
| Attention resolution | 16 | 16 | 32, 16, 8 |
| Dropout | 0.1 | 0.1 | 0.0 |
| Batch size | 128 | 128 | 512 |
| Iterations | 100K | 100K | 100K |
| Learning Rate | 0.0001 | 0.0002 | 0.0001 |
| 0.01 | 0.005 | 0.05 | |
| Linear Layers Output | 64, 1 | - | - |
Table 11. Correction models hyper-parameters.
Losses
Score Model: Our score model was trained using the denoising score matching loss (DSM) over the backward marginals with a constant noise scale :
where is the score model parameterized by . For more details see § 2.2.
Classifier Model: Our robust classifier was trained using which is comprised of 3 terms:
Binary Cross-Entropy: The binary cross-entropy (BCE) is used to distinguish between the backward marginals and the forward marginals :
where is the classifier output for input at time , parameterized by .
Adversarial Training (AT): To enhance its robustness, the classifier was also trained on adversarial examples. This process involves:
Sampling and .
Optimizing norm-clipped additive perturbations and to:
Decrease the classifier value for :
Increase the classifier value for :
For a more detailed explanation see Sec. D.5.
Gradient Alignment (GA): To align the classifier’s gradient near the backward marginals, we introduce a small amount of Gaussian noise and use cosine similarity to adjust the classifier’s gradient in the direction opposite to the noise (towards the backward marginal).
Evaluation Parameters
Hyper-parameters:
Step size : The configuration with the best FID was selected from a grid search over the interval with a step size of . For the classifier’s final step, the grid search was conducted over the interval with a step size of .
Noise : The configuration with the best FID was selected from a grid search over the interval with a step size of .
Sampling:
The hyper-parameters for all evaluations are described below, except for parallel sampling, where the time-steps are shifted by 1 (10 → 9, 9 → 8, etc). Additionally, for parallel classifier the final step-size is 0.02 instead of 0.06.
The score model’s Langevin Dynamics sampling used 50 and 100 steps with step sizes of 0.1 and 0.05, respectively, yielding the best results among step sizes .
| Corr. Step | Classifier | |||||||
| 10 Steps | 5 Steps | 3 Steps | 1 Step | |||||
| Step Size | Noise | Step Size | Noise | Step Size | Noise | Step Size | Noise | |
| 0 | 1.5 | 0.05 | - | - | - | - | - | - |
| 1 | 1.5 | 0.0 | - | - | - | - | - | - |
| 2 | 1.5 | 0.0 | - | - | - | - | - | - |
| 3 | 1.5 | 0.0 | - | - | - | - | - | - |
| 4 | 1.5 | 0.0 | - | - | - | - | - | - |
| 5 | 1.5 | 0.0 | - | - | - | - | - | - |
| 6 | 1.5 | 0.08 | 1.5 | 0.08 | - | - | - | - |
| 7 | 1.5 | 0.0 | 1.5 | 0.0 | - | - | - | - |
| 8 | 1.5 | 0.0 | 1.5 | 0.0 | 1.0 | 0.05 | 1.0 | 0.05 |
| 9 | 1.5 | 0.0 | 1.5 | 0.0 | 0.4 | 0.0 | - | - |
| 10 | 0.06 | 0.0 | 0.06 | 0.0 | 0.06 | 0.0 | - | - |
Table 12. CIFAR-10 evaluation hyper-parameters for the classifier correction model.
| Corr. Step | Score | |||||
| 10 Steps | 3 Steps | 1 Step | ||||
| Step Size | Noise | Step Size | Noise | Step Size | Noise | |
| 0 | 0.4 | 0.03 | 0.35 | 0.03 | - | - |
| 1 | 0.3 | 0.0 | - | - | - | - |
| 2 | 0.3 | 0.0 | - | - | - | - |
| 3 | 0.3 | 0.0 | - | - | - | - |
| 4 | 0.3 | 0.01 | - | - | - | - |
| 5 | 0.35 | 0.05 | 0.45 | 0.05 | - | - |
| 6 | 0.3 | 0.0 | - | - | - | - |
| 7 | 0.3 | 0.0 | - | - | - | - |
| 8 | 0.3 | 0.01 | - | - | - | - |
| 9 | 0.3 | 0.00 | - | - | - | - |
| 10 | 2.0 | 0.0 | 2.0 | 0.0 | 2.0 | 0.0 |
Table 13. CIFAR-10 evaluation hyper-parameters for score correction model.
| Corr. Step | Score | |||||
| 5 Steps | 2 Steps | 1 Step | ||||
| Step Size | Noise | Step Size | Noise | Step Size | Noise | |
| 0 | 0.4 | 0.1 | 0.4 | 0.01 | - | - |
| 1 | 0.2 | 0.05 | - | - | - | - |
| 2 | 0.2 | 0.0 | - | - | - | - |
| 3 | 0.2 | 0.0 | - | - | - | - |
| 4 | 0.2 | 0.05 | - | - | - | - |
| 5 | 0.4 | 0.0 | 0.4 | 0.0 | 0.4 | 0.0 |
Table 14. ImageNet-64 evaluation hyper-parameters for score correction model.
Qualitative Examples

Figure 12. ImageNet-64 image generation using OT-CFM alone and with score correction model. The corrected images demonstrate enhanced sharpness and definition compared to their uncorrected counterparts.

Figure 13. (a) OT-CFM; (b) Score Correction Model; (c) Classifier Correction Model. CIFAR-10 image generation using OT-CFM alone and with score correction and classifier correction models. Despite the distinct nature of these correction models, their suggested improvements often exhibit remarkable similarity. The corrected images demonstrate enhanced sharpness and definition compared to their uncorrected counterparts.

(a) OT-CFM

(b) Last Marginal Score Correction Model

(c) Correction Difference
Figure 14. ImageNet-64 images generated by OT-CFM alone and with score correction model applied solely to the final step, accompanied by their difference (amplified for visibility). The noise-like appearance of the difference suggests the presence of residual noise in the model’s predictions.

Figure 15. CIFAR-10 images generated by OT-CFM alone and with score correction model applied solely to the final step, accompanied by their difference (amplified for visibility). The noise-like appearance of the difference suggests the presence of residual noise in the model’s predictions.
References
- [2]Bansal, A., Borgnia, E., Chu, H.-M., Li, J., Kazemi, H., Huang, F., Goldblum, M., Geiping, J., and Goldstein, T. Cold diffusion: Inverting arbitrary image transforms without noise. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2208.09392
- [3]Chrabaszcz, P., Loshchilov, I., and Hutter, F. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819, 2017.arxiv.org/abs/1707.08819
- [4]Coeurdoux, F., Dobigeon, N., and Chainais, P. Normalizing flow sampling with langevin dynamics in the latent space, 2023. URL https://arxiv.org/abs/2305.12149.
- [5]Dai, B. and Wipf, D. Diagnosing and enhancing VAE models. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=B1e0X3C9tQ.arxiv.org/abs/1903.05789
- [6]Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
- [7]Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.
- [8]Dinh, L., Krueger, D., and Bengio, Y. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.DOI
- [10]Franzese, G., Rossi, S., Yang, L., Finamore, A., Rossi, D., Filippone, M., and Michiardi, P. How much is enough? a study on diffusion times in score-based generative models. Entropy, 25(4):633, 2023.DOI
- [11]gil Lee, S., Kim, H., Shin, C., Tan, X., Liu, C., Meng, Q., Qin, T., Chen, W., Yoon, S., and Liu, T.-Y. Priorgrad: Improving conditional denoising diffusion models with data-dependent adaptive prior. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=_BNiN4IjC5.arxiv.org/abs/2106.06406
- [12]Heitz, E., Belcour, L., and Chambon, T. Iterative α-(de)blending: A minimalist deterministic diffusion model. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1–8, 2023.DOI
- [13]Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
- [16]Irwin, R., Tibo, A., Janet, J.-P., and Olsson, S. Efficient 3d molecular generation with flow matching and scale optimal transport. arXiv preprint arXiv:2406.07266, 2024.
- [17]Kantorovich, L. V. On the translocation of masses. In Dokl. Akad. Nauk. USSR (NS), volume 37, pp. 199–201, 1942.
- [18]Kawar, B., Ganz, R., and Elad, M. Enhancing diffusion-based image synthesis with robust classifier guidance. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=tEVpz2xJWX.arxiv.org/abs/2208.08664
- [19]Kim, Y., Jo, D., Jeon, H., Kim, T., Ahn, D., Kim, H., et al. Leveraging early-stage robustness in diffusion models for efficient and high-quality image synthesis. Advances in Neural Information Processing Systems, 36, 2024.DOI
- [20]Kingma, D., Salimans, T., Poole, B., and Ho, J. Variational diffusion models. Advances in neural information processing systems, 34:21696–21707, 2021.arxiv.org/abs/2107.00630
- [21]Kingma, D. P. and Dhariwal, P. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1807.03039
- [22]Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.arxiv.org/abs/1312.6114
- [23]Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
- [24]Kwon, D., Fan, Y., and Lee, K. Score-based generative modeling secretly minimizes the wasserstein distance. Advances in Neural Information Processing Systems, 35: 20205–20217, 2022.arxiv.org/abs/2212.06359
- [26]Lezama, J., Salimans, T., Jiang, L., Chang, H., Ho, J., and Essa, I. Discrete predictor-corrector diffusion models for image synthesis. In The Eleventh International Conference on Learning Representations, 2022.
- [27]Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=PqvMRDCJT9t.arxiv.org/abs/2210.02747
- [28]Liu, X., Gong, C., and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=XVjTT1nw5z.arxiv.org/abs/2209.03003
- [29]Liu, X., Zhang, X., Ma, J., Peng, J., et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023b.arxiv.org/abs/2309.06380
- [31]Lyu, Z., XU, X., Yang, C., Lin, D., and Dai, B. Accelerating diffusion models via early stop of the diffusion process, 2022.arxiv.org/abs/2205.12524
- [32]Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJzIBfZAb.arxiv.org/abs/1706.06083
- [33]Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen-Or, D. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038–6047, 2023.arxiv.org/abs/2211.09794
- [34]Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162–8171. PMLR, 2021.arxiv.org/abs/2102.09672
- [35]Pan, Z., Gherardi, R., Xie, X., and Huang, S. Effective real image editing with accelerated iterative diffusion inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15912–15921, 2023.arxiv.org/abs/2309.04907
- [36]Pedrotti, F., Maas, J., and Mondelli, M. Improved convergence of score-based diffusion models via prediction-correction. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=0zKvH7YiAq.arxiv.org/abs/2305.14164
- [37]Pooladian, A. A., Ben-Hamu, H., Domingo-Enrich, C., Amos, B., Lipman, Y., and Chen, R. T. Multisample flow matching: Straightening flows with minibatch couplings. Proceedings of Machine Learning Research, 202: 28100–28127, 2023.arxiv.org/abs/2304.14772
- [38]Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudinov, M. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp. 8599–8608. PMLR, 2021.arxiv.org/abs/2105.06337
- [39]Shaul, N., Perez, J., Chen, R. T., Thabet, A., Pumarola, A., and Lipman, Y. Bespoke solvers for generative flow models. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2310.19075
- [40]Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256–2265. PMLR, 2015.
- [41]Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=St1giarCHLP.arxiv.org/abs/2010.02502
- [42]Song, K. and Lai, H. Leveraging previous steps: A training-free fast solver for flow diffusion. arXiv preprint arXiv:2411.07627, 2024.arxiv.org/abs/2411.07627
- [43]Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1907.05600
- [44]Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b. URL https://openreview.net/forum?id=PxTIG12RRHS.
- [45]Srinivas, S. and Fleuret, F. Rethinking the role of gradient-based attribution methods for model interpretability. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=dYeAHXnpwJ4.arxiv.org/abs/2006.09128
- [46]Sun, Z., Yang, Z., Jin, Y., Chi, H., Xu, K., Xu, K., Chen, L., Jiang, H., Zhang, D., Song, Y., Gai, K., and Mu, Y. Rectif: Personalizing rectified flow with anchored classifier guidance. arXiv preprint arXiv:2405.14677, 2024.arxiv.org/abs/2405.14677
- [47]Tong, A., FATRAS, K., Malkin, N., Huguet, G., Zhang, Y., Rector-Brooks, J., Wolf, G., and Bengio, Y. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=CD9Snc73AW. Expert Certification.arxiv.org/abs/2302.00482
- [48]Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A. Robustness may be at odds with accuracy. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=SyxAb30cY7.arxiv.org/abs/1805.12152
- [49]Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
- [50]Vincent, P. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011.DOI
- [51]Wallace, B., Gokul, A., and Naik, N. Edict: Exact diffusion inversion via coupled transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22532–22541, 2023.arxiv.org/abs/2211.12446
- [52]Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681–688. Citeseer, 2011.
- [54]Yadin, S., Elata, N., and Michaeli, T. Classification diffusion models. arXiv preprint arXiv:2402.10095, 2024.arxiv.org/abs/2402.10095
- [55]Yan, H., Liu, X., Pan, J., Liew, J. H., Liu, Q., and Feng, J. Perflow: Accelerating diffusion models via piecewise rectified flow. 2024.
- [56]Zhang, G. and Kleijn, W. B. Exact diffusion inversion via bi-directional integration approximation. arXiv preprint arXiv:2307.10829, 2023.arxiv.org/abs/2307.10829
- [57]Zhao, W., Bai, L., Rao, Y., Zhou, J., and Lu, J. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.04867, 2023.arxiv.org/abs/2302.04867
- [58]Zhao, W., Wang, H., Zhou, J., and Lu, J. Dc-solver: Improving predictor-corrector diffusion sampler via dynamic compensation. arXiv preprint arXiv:2409.03755, 2024.arxiv.org/abs/2409.03755
- [59]Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223–2232, 2017.arxiv.org/abs/1703.10593