Affiliation: 1 ^{1} 1 Tencent Hunyuan, 2 ^{2} 2 Westlake University, 3 ^{3} 3 University of Chinese Academy of Sciences
*Equal contribution; †Project Leader; ‡Corresponding author: wutailin@westlake.edu.cn
Project Page: fewsteprl.page
Code: fewsteprl-sd35 (images), fewsteprl-wan21 (videos)
Checkpoints: Hugging Face (coming soon)
Introduction
Diffusion models and their flow-matching counterparts generate data by integrating a learned transport from noise to samples, and have become a dominant approach to image and video synthesis [31 , 15 ] (Song et al. 2021b; Lipman et al. 2023; Liu et al. 2023; Rombach et al. 2022). Because the learned field is accurate only locally, sampling typically takes tens of network evaluations (Song et al., 2021a).
Few-step models remove most of this cost by learning the transport over long intervals directly. Consistency models and MeanFlow, which models the average velocity over an interval, and on-policy distilled flow maps such as DMD and AnyFlow generate in one to four evaluations [23 , 24 , 26 ] (Song et al. 2023; Yin et al. 2024b).
Reinforcement learning (RL) has become a standard post-training method for diffusion models. Fine-tuning against reward models of human preference, text rendering or composition measurably improves the outputs [3 , 19 , 11 ] (Xu et al. 2023; Liu et al. 2025a; Xue et al. 2026; Zheng et al. 2026a). This gain, however, typically requires samples from the current policy at every iteration and must be protected against reward hacking, in which the policy exploits the reward model instead of improving.
With a many-step sampler, these rollouts typically account for most of the time and memory of training, and more so for long video sequences. A few-step model shrinks this cost, but it models an average velocity over an interval, whereas the RL objectives above are built for an instantaneous velocity or for the transitions of a many-step sampler. Applied directly, they leave open which network evaluation should be differentiated, how the reference policy should be evaluated, and which parts of the interval are trained. Prior work typically resolves these choices for one objective at a time (Huang et al., 2026b,a; Li et al., 2026b); what is, to our knowledge, missing is a systematic way to adapt RL objectives to few-step models. Figure 1 illustrates the effect of leaving them unresolved. Substituting the average velocity of a distilled few-step model for the instantaneous velocity required by FlowGRPO, AWM and DiffusionNFT, i.e., applying the original objective to the distilled model without modification, gives a lower best GenEval2 than the adapted objective derived in this paper in each of the three cases.
Figure 1. Held-out GenEval2 against the RL iteration in the mixed-reward setting of § 6.1 , from the shared pre-RL few-step checkpoint. Dashed: the original objective applied to the distilled model with the average velocity substituted for the instantaneous velocity (FlowGRPO, AWM, DiffusionNFT). Solid: the corresponding few-step adaptation of this paper (FewStepGRPO, MeanFlowAWM, MeanFlowNFT; § 5 ). Each annotated gap is the adaptation minus the baseline, at the best checkpoint of the baseline and at iteration 400; the two coincide when the baseline is best at iteration 400.
This paper builds one in three phases (Figure 2 ), namely a transformation, a decomposition and an adaptation. The transformation maps each objective to x x x -space. Under a linear noise schedule, a velocity defines an affine map between noise levels, so each objective we consider can be expressed through a common policy prediction, at the clean end by default and at an explicitly marked level for transition or flow-map objectives. In x x x -space, objectives stated over densities, velocities or rewards become directly comparable. The decomposition splits the loss gradient. The gradient of every objective we consider is the sum of a self-regression term, which keeps the policy close to a reference, and a reward-augmentation term, which contains the reward, so that, at the gradient level, a method reduces to a choice of anchors and weights. The adaptation extends the anchors to few-step models. We show how each anchor is computed from a few-step model, either at the end of its transport or along the tangent of the trajectory, with the instantaneous velocity recovered from the network itself, as in MeanFlow, or from an auxiliary network, as in DMD.
Figure 2. Overview of the framework. (a) RL increases the relative mass of the high-reward mode under the rollout distribution p o l d p_{\mathrm{old}} p old of the old policy. The default objective is evaluated at the clean end, on the policy prediction (renoise-and-denoise sample) x ~ θ = X [ x ~ t , v θ , 0 ] = x ~ t − t v θ \tilde{x}_{\theta}=X[\tilde{x}_{t},v_{\theta},0]=\tilde{x}_{t}-tv_{\theta} x ~ θ = X [ x ~ t , v θ , 0 ] = x ~ t − t v θ , whose reward augmentation attracts it toward high-reward rollout samples and repels it from low-reward ones; in the figure, x ^ + \hat{x}^{+} x ^ + and x ^ − \hat{x}^{-} x ^ − denote a high-reward and a low-reward rollout sample x ^ ( i ) \hat{x}^{(i)} x ^ ( i ) , respectively. (b) The RL-loss gradient can be decoupled into a self-regression term, visualised as a regression force toward the regression anchor x ~ r e g \tilde{x}_{\mathrm{reg}} x ~ reg , and a reward-augmentation term, visualised as a reward force along the displacement x ~ r w d − x ~ b a s e \tilde{x}_{\mathrm{rwd}}-\tilde{x}_{\mathrm{base}} x ~ rwd − x ~ base from the baseline anchor x ~ b a s e \tilde{x}_{\mathrm{base}} x ~ base , which lies between the two modes, to the rewarded anchor x ~ r w d \tilde{x}_{\mathrm{rwd}} x ~ rwd . A high-reward sample produces attraction along that displacement (blue); a low-reward sample produces repulsion against it (orange), so both updates favour higher reward (§ 4.2 ). (c) For a few-step model, the prediction and the anchors { x ~ r w d , x ~ b a s e , x ~ r e g } \{\tilde{x}_{\mathrm{rwd}},\tilde{x}_{\mathrm{base}},\tilde{x}_{\mathrm{reg}}\} { x ~ rwd , x ~ base , x ~ reg } are computed either at the end of the transport curve, x ~ t − t u \tilde{x}_{t}-tu x ~ t − t u (u-form), or along the tangent at x ~ t \tilde{x}_{t} x ~ t , x ~ t − t v \tilde{x}_{t}-tv x ~ t − t v (v-form). The instantaneous velocity is recovered from the network itself for MeanFlow, v = u + ( t − r ) sg ( ∂ t u ) v=u+(t-r)\operatorname{sg}(\partial_tu) v = u + ( t − r ) sg ( ∂ t u ) with the detached correction evaluated on the old policy in the implemented methods (Equation (56) ), and from an auxiliary network for DMD, v = v ϕ v=v_{\phi} v = v ϕ , fitted so that v ϕ ( x ~ t , t ) ≈ E [ ϵ − x ^ ∣ x ~ t ] v_{\phi}(\tilde{x}_{t},t)\approx\mathbb{E}[\epsilon-\hat{x}\mid\tilde{x}_{t}] v ϕ ( x ~ t , t ) ≈ E [ ϵ − x ^ ∣ x ~ t ] (§ 4.3 ).
All methods share one codebase, which is represented by a single training step with several method-specific stages, namely rollout, reward and advantage, anchors and weights, and loss (plus auxiliary fitting and the refresh of the old policy), where each method overrides only the hooks in which it differs (§ 5 ). The methods are therefore compared under shared rollout, reward and evaluation code, and a new variant is specified mainly by its anchors and weights. Experimentally, we compare seven implementations on image generation with an AnyFlow-distilled SD3.5-Medium ([26 ] (Stability AI 2024)), under a non-differentiable OCR reward and a differentiable mixed reward, and on video generation with an AnyFlow-distilled Wan2.1-T2V-1.3B ([26 ] (Team Wan et al. 2025)); we further test whether the fine-tuned few-step models improve when sampled with more steps.
The contributions of this paper are as follows.
A general view. After the transformation to x x x -space, the gradients of the RL objectives considered here share one decoupled form, the sum of a self-regression term and a reward-augmentation term, so that their gradients differ only in their anchors and weights.
A systematic adaptation to few-step models. Each of these anchors can be computed from an average-velocity model at the end of the transport or along its tangent, which yields few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD and ReFL within one modular codebase.
Empirical guidance with experimental analysis. Where the reward is differentiable and its gradient affordable, reward-gradient-based methods are the most effective in our experiments, and ReFL alone is a strong baseline. Otherwise, group-advantage methods apply. Among them, DiffusionNFT-style methods generally reach high rewards in relatively few iterations, even when one iteration is expensive, and DMD-based variants are faster per iteration on image but slower on video, depending on the relative cost of forward and backward passes, and often coincide with larger held-out degradation. Additional inference steps bring little aggregate improvement in generation quality, while visual gains appear in selected outputs with more sampling steps than training.
Background
Diffusion and flow-based generators. Diffusion models learn to reverse a fixed noising process [31 ] ; score-based models cast this reversal as a reverse-time SDE whose probability-flow ODE gives a deterministic sampler, of which DDIM is a discrete instance (Song et al., 2021b,a). Flow matching and rectified flow instead regress the velocity of a prescribed interpolation between noise and data, the latter straightening the paths so that fewer integration steps suffice (Lipman et al., 2023; Liu et al., 2023). These models were scaled by moving to a latent space (Rombach et al., 2022), by transformer backbones (Peebles and Xie, 2023; Ma et al., 2024) and by larger, more efficient architectures [15 ] (Podell et al. 2024; Xie et al. 2025a), with classifier-free guidance as a standard way to trade diversity for prompt fidelity [30 ] . Related generators include current text-to-image systems [2 , 4 , 5 ] (OpenAI 2022; Saharia et al. 2022; Team Seedream et al. 2025; Z-Image Team et al. 2025; Qin et al. 2025), multimodal models that both generate and understand images in one network [10 , 9 ] (Ma et al. 2025a; Xie et al. 2025b; Wang et al. 2024b; OpenAI et al. 2024), trained on web-scale image–text data (Schuhmann et al., 2022; Zou, 2024), and video generators such as Wan and LongCat-Video (Team Wan et al., 2025; Meituan LongCat Team et al., 2025). These models typically learn the instantaneous velocity of the sampling flow, or an affine transformation of it such as the score or the predicted noise. Since this velocity is instantaneous, the sampler integrates it with many small steps, typically 32 or 64, which makes inference costly.
Few-step generation. Sampling cost has been cut along several lines. Progressive distillation halves the number of steps repeatedly by training a student to match two teacher steps with one (Salimans and Ho, 2022). Consistency models map every point of a trajectory to its endpoint, and were extended to latent space, arbitrary trajectory jumps, multiple sub-intervals and segmented trajectories (Song et al., 2023; Luo et al., 2023a; Kim et al., 2024; Wang et al., 2024a; Ren et al., 2024); consistency has also been used as a training regulariser against sampling drift and to learn from noisy data [12 , 13 ] . Shortcut models condition on the step size and learn to take large steps directly [20 ] . Adversarial distillation adds a discriminator so that one to four steps produce sharp samples [6 ] (Wang et al. 2023a; Sauer et al. 2025; Sauer et al. 2024; Lin et al. 2024; Lu and Song 2025). Distribution matching distillation (DMD) trains the generator so that its output distribution matches the teacher’s, using the difference between a teacher score and the score of an auxiliary network fitted to the generator’s samples (Yin et al., 2024b,a), an idea shared with Diff-Instruct and variational score distillation (Luo et al., 2023b; Wang et al., 2023b); later work scales it to flow models, splits it into sub-intervals, decouples guidance from matching, moves it to continuous time, regularises it with consistency, and adapts it to video (Ge et al., 2026a; Fan et al., 2026b; Liu et al., 2026a,b; Zheng et al., 2026b; Ge et al., 2026b; Nie et al., 2026; Luo et al., 2025). Flow maps learn the transport between any two noise levels. MeanFlow parameterises the average velocity over an interval and trains it through its identity with the instantaneous velocity [23 , 24 ] (Lin et al. 2026), and AnyFlow distils such a map on policy for any step count [26 ] . Unlike the many-step generators above, these flow-map models learn an average velocity over an interval, which can be read as the direction of a secant of the sampling trajectory, differing substantially from the instantaneous velocity where the trajectory is curved.
Reinforcement learning for diffusion models. RL fine-tuning of generators builds on policy optimisation (Levine, 2018; Schulman et al., 2017), preference optimisation (Rafailov et al., 2023) and group-relative advantages, which normalise rewards within a group of samples for the same prompt [27 ] (Shao et al. 2024). For diffusion models, early work fine-tunes on human feedback (Lee et al., 2023), treats denoising as a Markov decision process optimised by policy gradients [3 , 19 ] , or optimises pairwise preferences directly (Wallace et al., 2024). FlowGRPO brings group-relative policy optimisation to flow models by turning the ODE sampler into an SDE with tractable transition likelihoods (Liu et al., 2025a); follow-ups extend it to video, mix ODE and SDE steps, weight timesteps, and preserve the noise coefficients of the sampler [28 ] (Xue et al. 2025; Li et al. 2026a; Wang and Yu 2025). A second family avoids transition likelihoods by acting on the forward process: AWM weights the pretraining loss by the advantage (Xue et al., 2026), reward-weighted and policy-gradient flow matching reweight or rescale the velocity regression [16 ] (McAllister et al. 2026), adjoint matching is scaled to RL post-training [1 ] , and DiffusionNFT contrasts positive and negative implicit policies built from the model and the frozen policy that generated its rollouts (Zheng et al., 2026a), following NFT in language models [8 ] . A third family backpropagates differentiable rewards through the sampler, as in DRaFT and ReFL [11 ] (Xu et al. 2023; Wang et al. 2026); the recently proposed DiffusionOPSD (Zhou et al., 2026) belongs to this family but constructs detached points with normalised reward-gradient steps and fits the policy to them, thereby forming a surrogate gradient. These methods are built for an instantaneous velocity, and most of them need on-policy rollouts at every iteration, which are expensive with the many-step samplers discussed above; how to apply RL to few-step models is therefore an important question.
RL for few-step generators. Closest to this paper are methods that fine-tune few-step models with rewards. Several combine DMD with RL, by adding a reward term to distillation, guiding distillation with reward gradients, or recasting distribution matching itself as a reward [7 , 14 , 17 ] (Jiang et al. 2026); others tune distilled models with non-differentiable rewards, Kullback–Leibler (KL) regularised alignment or pairwise sample optimisation (Luo et al., 2026; Luo, 2025; Miao et al., 2025). For flow maps, Flow-Map GRPO composes stochastic steps of a flow-map generator (Li et al., 2026b), MeanFlowNFT applies DiffusionNFT to an average-velocity model through its self-induced instantaneous field [34 ] , and RTDMD optimises a reward-tilted distribution matching objective [33 ] . These works mostly adapt one objective to one kind of few-step model; we instead give a common construction that extends FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD and ReFL to average-velocity models, and compare them under one implementation.
Preliminaries
Few-step Diffusion Models with On-Policy Distillation
We write t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] for the noise level, with data at t = 0 t=0 t = 0 and Gaussian noise at t = 1 t=1 t = 1 . A plain x t x_t x t denotes a point at level t t t on a sampling trajectory, and x ~ t \tilde{x}_t x ~ t denotes a renoised state at level t t t obtained by adding fresh noise to a clean sample Equation (2) ; every map below applies to either. The setting we focus on throughout is distillation , where a pretrained instantaneous-velocity field v r e a l ( x ~ t , t ) v_{\mathrm{real}}(\tilde{x}_t,t) v real ( x ~ t , t ) is available and frozen, which we call the teacher, and the student u θ ( x ~ t , t , r ) u_\theta(\tilde{x}_t,t,r) u θ ( x ~ t , t , r ) must reproduce its transport in one or a few network calls. The teacher is an ordinary flow-matching model, trained by regressing onto the conditional velocity v c o n d = ϵ − x v_{\mathrm{cond}}=\epsilon-x v cond = ϵ − x of a clean sample x x x and Gaussian noise ϵ ∼ N ( 0 , I ) \epsilon\sim\mathcal{N}(0,I) ϵ ∼ N ( 0 , I ) , and it provides the student’s regression target in Equation (8) (Lipman et al., 2023; Liu et al., 2023). The student is the average-velocity generator [23 , 24 , 26 ] ,
u θ ( x t , t , r ) = 1 t − r ∫ r t v θ ( x v , v ) d v , (1) u_\theta(x_t,t,r) = \frac{1}{t-r}\int_r^t v_\theta(x_v,v)\,\mathrm{d}v, \tag*{(1)} u θ ( x t , t , r ) = t − r 1 ∫ r t v θ ( x v , v ) d v , (1)
where v θ v_\theta v θ is the instantaneous velocity induced by the student, x ν x_\nu x ν is the state at level ν \nu ν on the probability-flow trajectory of v θ v_\theta v θ through x t x_t x t , and u θ ( x t , t , t ) : = v θ ( x t , t ) u_\theta(x_t,t,t) := v_\theta(x_t,t) u θ ( x t , t , t ) := v θ ( x t , t ) closes the definition at r = t r=t r = t . Throughout, we identify this induced field with the flow-matching marginal velocity of the student’s sample distribution p θ p_\theta p θ , v θ ( x t , t ) ≡ E [ ϵ − x ^ ∣ x t ] v_\theta(x_t,t) \equiv\mathbb{E}[\epsilon-\hat{x}\mid x_t] v θ ( x t , t ) ≡ E [ ϵ − x ^ ∣ x t ] with x ^ ∼ p θ \hat{x}\sim p_\theta x ^ ∼ p θ (Appendix A); the two coincide when the flow map is exact.
Unlike off-policy methods, which query the model at x ~ t = AddNoise ( x 0 , t ) \tilde{x}_t=\operatorname{AddNoise}(x_0,t) x ~ t = AddNoise ( x 0 , t ) with x 0 ∼ p d a t a x_0\sim p_{\mathrm{data}} x 0 ∼ p data , on-policy training draws the noised state from the student’s own distribution:
x ~ t = AddNoise ( x ^ θ , t ) = ( 1 − t ) x ^ θ + t ϵ , x ^ θ ∼ p θ ( x ) , ϵ ∼ N ( 0 , I ) . (2) \tilde{x}_t=\operatorname{AddNoise}(\hat{x}_\theta,t)=(1-t)\hat{x}_\theta+t\epsilon,\qquad\hat{x}_\theta\sim p_\theta(x),\qquad\epsilon\sim\mathcal{N}(0,I). \tag*{(2)} x ~ t = AddNoise ( x ^ θ , t ) = ( 1 − t ) x ^ θ + t ϵ , x ^ θ ∼ p θ ( x ) , ϵ ∼ N ( 0 , I ) . (2)
The subscript θ \theta θ in Equation (2) is kept only where the sampler is differentiated. The samples x ^ θ \hat{x}_\theta x ^ θ are produced by a rollout operator Rollout \operatorname{Rollout} Rollout , an ODE or SDE solver that maps Gaussian noise at level 1 to a clean sample at level 0 for a text prompt c c c in M s t e p M_{\mathrm{step}} M step steps, on a grid 1 = t 0 > t 1 > ⋯ > t M s t e p = 0 1=t_0>t_1>\cdots>t_{M_{\mathrm{step}}}=0 1 = t 0 > t 1 > ⋯ > t M step = 0 :
x ^ θ = Rollout ( ϵ , u θ , c ) , x ^ θ , t 0 = ϵ ∼ N ( 0 , I ) . (3) \hat{x}_\theta=\operatorname{Rollout}(\epsilon,u_\theta,c),\qquad\hat{x}_{\theta,t_0}=\epsilon\sim\mathcal{N}(0,I). \tag*{(3)} x ^ θ = Rollout ( ϵ , u θ , c ) , x ^ θ , t 0 = ϵ ∼ N ( 0 , I ) . (3)
For an instantaneous field, one Euler step of the solver between t j t_j t j and t j + 1 t_{j+1} t j + 1 is
x ^ θ , t j + 1 = x ^ θ , t j − ( t j − t j + 1 ) v θ ( x ^ θ , t j , t j ) , (4) \hat{x}_{\theta,t_{j+1}}=\hat{x}_{\theta,t_j}-(t_j-t_{j+1})v_\theta(\hat{x}_{\theta,t_j},t_j), \tag*{(4)} x ^ θ , t j + 1 = x ^ θ , t j − ( t j − t j + 1 ) v θ ( x ^ θ , t j , t j ) , (4)
which is only first-order accurate, so M s t e p M_{\mathrm{step}} M step must be large. A displacement is the time integral of the velocity, and by Equation (1) ( t − r ) u θ (t-r)u_\theta ( t − r ) u θ equals this integral over [ r , t ] [r,t] [ r , t ] . With u θ u_\theta u θ as the generator, the Euler step is replaced by an exact jump,
x ^ θ , r = x ^ θ , t − ( t − r ) u θ ( x ^ θ , t , t , r ) , (5) \hat{x}_{\theta,r}=\hat{x}_{\theta,t}-(t-r)u_\theta(\hat{x}_{\theta,t},t,r), \tag*{(5)} x ^ θ , r = x ^ θ , t − ( t − r ) u θ ( x ^ θ , t , t , r ) , (5)
so a few-step sampler is a chain of M s t e p M_{\mathrm{step}} M step such jumps, each a single subtraction rather than an integration. Taking the whole interval at once, M s t e p = 1 M_{\mathrm{step}}=1 M step = 1 , t = 1 t=1 t = 1 and r = 0 r=0 r = 0 , recovers the one-call generator.
MeanFlow. According to the definition of u θ u_\theta u θ in Equation (1) (Geng et al. [23 ] ), multiplying by ( t − r ) (t-r) ( t − r ) and differentiating in t t t eliminates the integral and leaves the MeanFlow identity
u θ ( x ~ t , t , r ) = v θ ( x ~ t , t ) − ( t − r ) ∂ t u θ ( x ~ t , t , r ) , (6) u_\theta(\tilde{x}_t,t,r)=v_\theta(\tilde{x}_t,t)-(t-r)\partial_tu_\theta(\tilde{x}_t,t,r), \tag*{(6)} u θ ( x ~ t , t , r ) = v θ ( x ~ t , t ) − ( t − r ) ∂ t u θ ( x ~ t , t , r ) , (6)
where ∂ t u θ = d u θ / d t \partial_tu_\theta=\mathrm{d}u_\theta/\mathrm{d}t ∂ t u θ = d u θ / d t is the total derivative along the trajectory x ν x_\nu x ν of Equation (1) ; the implementation estimates it by a directional-difference estimate (Equation (87) ). The identity follows from the definition Equation (1) and imposes no additional constraint; it expresses the average velocity as the instantaneous velocity minus ( t − r ) (t-r) ( t − r ) times the total derivative of the average velocity with respect to t t t .
Every use of Equation (6) treats this correction as detached, so we define it once with the stop-gradient sg ( ⋅ ) \operatorname{sg}(\cdot) sg ( ⋅ ) , which is the identity in the forward pass and zero in the backward pass,
J [ u , t , r ] ≔ ( t − r ) sg ( ∂ t u ) , (7) J[u,t,r]\coloneqq(t-r)\operatorname{sg}(\partial_tu), \tag*{(7)} J [ u , t , r ] : = ( t − r ) sg ( ∂ t u ) , (7)
the detached Jacobian-vector term . It gives the gap between an average velocity over [ r , t ] [r,t] [ r , t ] and the instantaneous one at t t t , measured on the network itself and frozen before it is used. Substituting the teacher for v θ v_\theta v θ in Equation (6) then turns the identity into a regression whose target is assembled entirely from quantities available at the current state,
L M F = E x ~ t , t , r [ ∥ u θ ( x ~ t , t , r ) − ( v r e a l ( x ~ t , t ) − J [ u θ , t , r ] ) ∥ 2 ] , (8) \mathcal{L}_{\mathrm{MF}}=\mathbb{E}_{\tilde{x}_t,t,r}\left[\left\|u_\theta(\tilde{x}_t,t,r)-\left(v_{\mathrm{real}}(\tilde{x}_t,t)-J[u_\theta,t,r]\right)\right\|^2\right], \tag*{(8)} L MF = E x ~ t , t , r [ ∥ u θ ( x ~ t , t , r ) − ( v real ( x ~ t , t ) − J [ u θ , t , r ] ) ∥ 2 ] , (8)
in which v r e a l v_{\mathrm{real}} v real is frozen and J J J contains a stop-gradient, so u θ u_\theta u θ is the only differentiated quantity. Setting r = t r=t r = t recovers instantaneous-velocity distillation onto v r e a l v_{\mathrm{real}} v real ; the case r < t r<t r < t trains the flow map and hence the one-step transport x ~ θ , r = x ~ t − ( t − r ) u θ ( x ~ t , t , r ) \tilde{x}_{\theta,r}=\tilde{x}_t-(t-r)u_\theta(\tilde{x}_t,t,r) x ~ θ , r = x ~ t − ( t − r ) u θ ( x ~ t , t , r ) to level r r r .
DMD. Distribution matching distillation (Yin et al., 2024b,a) does not regress u θ u_\theta u θ onto a target. It maintains a second, auxiliary velocity v ϕ ( x ~ t , t ) v_\phi(\tilde{x}_t,t) v ϕ ( x ~ t , t ) with its own parameters ϕ \phi ϕ , fitted online by flow matching to the samples generated by the student. Its target is the conditional velocity of each sample instead of the teacher, because v ϕ v_\phi v ϕ is meant to approximate the velocity of p θ p_\theta p θ instead of p d a t a p_{\mathrm{data}} p data :
L a u x = E x ^ ∼ p θ , ϵ , t [ ∥ v ϕ ( x ~ t , t ) − v c o n d ∥ 2 ] , x ~ t = AddNoise ( x ^ , t ) , v c o n d = ϵ − x ^ . (9) \mathcal{L}_{\mathrm{aux}}=\mathbb{E}_{\hat{x}\sim p_\theta,\epsilon,t}\left[\left\lVert v_\phi(\tilde{x}_t,t)-v_{\mathrm{cond}}\right\rVert^2\right],\qquad\tilde{x}_t=\operatorname{AddNoise}(\hat{x},t),\qquad v_{\mathrm{cond}}=\epsilon-\hat{x}. \tag*{(9)} L aux = E x ^ ∼ p θ , ϵ , t [ ∥ v ϕ ( x ~ t , t ) − v cond ∥ 2 ] , x ~ t = AddNoise ( x ^ , t ) , v cond = ϵ − x ^ . (9)
The difference between the two fields then gives the score difference in the gradient of the reverse KL divergence, which is used to train u θ ( x ~ t , t , r ) u_\theta(\tilde{x}_t,t,r) u θ ( x ~ t , t , r ) :
∇ θ L D M D = E ϵ , t ∈ ( 0 , 1 ) [ 1 − t t sg [ v r e a l ( x ~ t , t ) − v ϕ ( x ~ t , t ) ] ⊤ ∂ x ^ θ ∂ θ ] . (10) \nabla_\theta\mathcal{L}_{\mathrm{DMD}}=\mathbb{E}_{\epsilon,t\in(0,1)}\left[\frac{1-t}{t}\operatorname{sg}\left[v_{\mathrm{real}}(\tilde{x}_t,t)-v_\phi(\tilde{x}_t,t)\right]^\top\frac{\partial\hat{x}_\theta}{\partial\theta}\right]. \tag*{(10)} ∇ θ L DMD = E ϵ , t ∈ ( 0 , 1 ) [ t 1 − t sg [ v real ( x ~ t , t ) − v ϕ ( x ~ t , t ) ] ⊤ ∂ θ ∂ x ^ θ ] . (10)
The order of the two fields is fixed by the conversion between them. The score of the linear path is ∇ log p t ( x ~ t ) = − E [ ϵ ∣ x ~ t ] / t \nabla\log p_t(\tilde{x}_t)=-\mathbb{E}[\epsilon\mid\tilde{x}_t]/t ∇ log p t ( x ~ t ) = − E [ ϵ ∣ x ~ t ] / t with E [ ϵ ∣ x ~ t ] = x ~ t + ( 1 − t ) v \mathbb{E}[\epsilon\mid\tilde{x}_t]=\tilde{x}_t+(1-t)v E [ ϵ ∣ x ~ t ] = x ~ t + ( 1 − t ) v , so the difference ∇ log p t ϕ − ∇ log p t r e a l \nabla\log p_t^\phi-\nabla\log p_t^{\mathrm{real}} ∇ log p t ϕ − ∇ log p t real of the scores of the two fields, which the reverse KL produces, equals 1 − t t ( v r e a l − v ϕ ) \frac{1-t}{t}(v_{\mathrm{real}}-v_\phi) t 1 − t ( v real − v ϕ ) . Converting scores to velocities reverses the order of the two fields and introduces the score-to-velocity factor 1 − t t \frac{1-t}{t} t 1 − t , which is positive on ( 0 , 1 ) (0,1) ( 0 , 1 ) and is absorbed into a per-t t t weight in practice. Descending Equation (10) therefore applies the direction + ( v ϕ − v r e a l ) +(v_\phi-v_{\mathrm{real}}) + ( v ϕ − v real ) , reducing the mismatch to the data field relative to the auxiliary field that tracks p θ p_\theta p θ .
The term ∂ x ^ θ / ∂ θ \partial\hat{x}_\theta/\partial\theta ∂ x ^ θ / ∂ θ requires a differentiable sampler, and the sampler used in practice is stochastic. A step at level t t t does not move directly to the next level; it first maps the state to r = 0 r=0 r = 0 with one network evaluation and then renoises the result to an intermediate level t ′ t' t ′ with 0 < t ′ < t 0<t'<t 0 < t ′ < t :
x ^ θ , 0 = x ^ θ , t − t u θ ( x ^ θ , t , t , 0 ) , x ^ θ , t ′ = ( 1 − t ′ ) x ^ θ , 0 + t ′ ϵ ′ , ϵ ′ ∼ N ( 0 , I ) . (11) \hat{x}_{\theta,0}=\hat{x}_{\theta,t}-t\,u_\theta(\hat{x}_{\theta,t},t,0),\qquad\hat{x}_{\theta,t'}=(1-t')\hat{x}_{\theta,0}+t'\epsilon',\qquad\epsilon'\sim\mathcal{N}(0,I). \tag*{(11)} x ^ θ , 0 = x ^ θ , t − t u θ ( x ^ θ , t , t , 0 ) , x ^ θ , t ′ = ( 1 − t ′ ) x ^ θ , 0 + t ′ ϵ ′ , ϵ ′ ∼ N ( 0 , I ) . (11)
Unlike the deterministic chain Equation (5) , this sampler draws fresh noise ϵ ′ \epsilon' ϵ ′ at every step, so the state at level t ′ t' t ′ is not a deterministic function of the previous state.
The levels follow a fixed grid of M s t e p + 1 M_{\mathrm{step}}+1 M step + 1 values spanning M s t e p M_{\mathrm{step}} M step intervals; a step count ℓ ∈ { 1 , … , M s t e p } \ell\in\{1,\ldots,M_{\mathrm{step}}\} ℓ ∈ { 1 , … , M step } is drawn per update, and only the last step remains in the computation graph, with all earlier steps detached. The backward pass therefore requires one network evaluation for any ℓ \ell ℓ . Moreover, the differentiated evaluation is u θ ( ⋅ , t , 0 ) u_\theta(\cdot,t,0) u θ ( ⋅ , t , 0 ) at the level that the grid assigns to step ℓ \ell ℓ , so randomising ℓ \ell ℓ changes this level across updates. Over training, the gradient therefore reaches u θ ( ⋅ , t , 0 ) u_\theta(\cdot,t,0) u θ ( ⋅ , t , 0 ) at every grid level t t t instead of a single one.
Reinforcement Learning for Diffusion Models
FlowGRPO. FlowGRPO combines a GRPO-style group baseline with a PPO transition objective (Schulman et al., 2017; Shao et al., 2024; Liu et al., 2025a). It scores a group of K K K rollouts that share a prompt c c c , and turns the raw scores into a group-normalised advantage. Writing R ( i ) = R ( D ( x ^ ( i ) ) , c ) R^{(i)}=R(\mathcal{D}(\hat{x}^{(i)}),c) R ( i ) = R ( D ( x ^ ( i ) ) , c ) for the endpoint reward of member i i i (D \mathcal{D} D the decoder), the advantage standardises the reward within the group,
A ( i ) = R ( i ) − mean ( { R } ) std ( { R } ) + ε s t d , i = 1 , … , K . (12) A^{(i)}=\frac{R^{(i)}-\operatorname{mean}(\{R\})}{\operatorname{std}(\{R\})+\varepsilon_{\mathrm{std}}},\qquad i=1,\ldots,K. \tag*{(12)} A ( i ) = std ({ R }) + ε std R ( i ) − mean ({ R }) , i = 1 , … , K . (12)
Here { R } \{R\} { R } denotes the K K K rewards of the group and ε s t d > 0 \varepsilon_{\mathrm{std}}>0 ε std > 0 is a small floor. The advantage is a detached per-sample scalar with zero mean across the group; no value network is fitted, and the group mean serves as the baseline. AWM below uses the same A ( i ) A^{(i)} A ( i ) ; DiffusionNFT uses a bounded transform of it.
FlowGRPO formulates the stochastic differential equation (SDE) sampler as a Markov decision process, whose state is the current noisy sample x t x_t x t (a point on the sampling trajectory, in the convention of § 3 ) together with the prompt and the level and whose action is the next state produced by the sampler; the Gaussian transition p θ t r ( x r ∣ x t ) p_\theta^{\mathrm{tr}}(x_r \mid x_t) p θ tr ( x r ∣ x t ) then provides a likelihood, which the deterministic sampler lacks. Optimisation is proximal policy optimisation on the realised action x r x_r x r ,
L G R P O = − E i , t [ min { A t , r ( i ) ρ t , r ( i ) , A t , r ( i ) clip ( ρ t , r ( i ) , 1 ± ε c l i p ) } ] , ρ t , r = p θ t r ( x r ∣ x t ) p o l d t r ( x r ∣ x t ) . (13) \mathcal{L}_{\mathrm{GRPO}}=-\mathbb{E}_{i,t}\left[\min\left\{A_{t,r}^{(i)}\rho_{t,r}^{(i)},A_{t,r}^{(i)}\operatorname{clip}\left(\rho_{t,r}^{(i)},1\pm\varepsilon_{\mathrm{clip}}\right)\right\}\right],\qquad\rho_{t,r}=\frac{p_\theta^{\mathrm{tr}}(x_r\mid x_t)}{p_{\mathrm{old}}^{\mathrm{tr}}(x_r\mid x_t)}. \tag*{(13)} L GRPO = − E i , t [ min { A t , r ( i ) ρ t , r ( i ) , A t , r ( i ) clip ( ρ t , r ( i ) , 1 ± ε clip ) } ] , ρ t , r = p old tr ( x r ∣ x t ) p θ tr ( x r ∣ x t ) . (13)
Here the old policy is the frozen policy, with parameters θ o l d \theta_{\mathrm{old}} θ old , that generated the rollouts on which the current policy is trained; it receives no gradient. In our implementation, θ o l d \theta_{\mathrm{old}} θ old is an exponential moving average (EMA) of θ \theta θ , refreshed after every outer iteration (§ 5.1 ) with the EMA-decay ratio as the weight kept on the previous θ o l d \theta_{\mathrm{old}} θ old ; an EMA-decay ratio of 0.0 therefore copies θ \theta θ , so that θ o l d = θ \theta_{\mathrm{old}}=\theta θ old = θ at the start of every outer iteration. The subscript old marks its quantities, so p o l d t r p_{\mathrm{old}}^{\mathrm{tr}} p old tr is its transition density, v o l d v_{\mathrm{old}} v old its instantaneous velocity and p o l d p_{\mathrm{old}} p old the distribution of its rollouts. The clipping radius is ε c l i p \varepsilon_{\mathrm{clip}} ε clip . For later clipped surrogates, define the detached unclipped-gradient gate
χ ( z , A ) : = 1 [ ( A > 0 ∧ z ≤ 1 + ε c l i p ) ∨ ( A < 0 ∧ z ≥ 1 − ε c l i p ) ] , χ ( z , 0 ) : = 0. (14) \chi(z,A):=\mathbf{1}\left[(A>0\wedge z\le1+\varepsilon_{\mathrm{clip}})\vee(A<0\wedge z\ge1-\varepsilon_{\mathrm{clip}})\right],\qquad\chi(z,0):=0. \tag*{(14)} χ ( z , A ) := 1 [ ( A > 0 ∧ z ≤ 1 + ε clip ) ∨ ( A < 0 ∧ z ≥ 1 − ε clip ) ] , χ ( z , 0 ) := 0. (14)
At the two clipping kinks this convention selects the derivative of the unclipped branch. The sign of the reward advantage determines the direction of the update of the mean μ θ \mu_\theta μ θ of the Gaussian transition (the scoring kernel defined in Equation (95) ) relative to the visited noisy state x r x_r x r .
AWM. Advantage Weighted Matching ((Xue et al., 2026)) keeps the signed advantage but does not evaluate a likelihood on the reverse process. Each update draws a fresh ( t , ϵ ) (t,\epsilon) ( t , ϵ ) and constructs both the noised state and the regression target from the rollout sample,
x ~ t ( i ) = AddNoise ( x ^ ( i ) , t ) = ( 1 − t ) x ^ ( i ) + t ϵ , v c o n d ( i ) = ϵ − x ^ ( i ) , (15) \widetilde{x}_t^{(i)}=\operatorname{AddNoise}(\widehat{x}^{(i)},t)=(1-t)\widehat{x}^{(i)}+t\epsilon,\qquad v_{\mathrm{cond}}^{(i)}=\epsilon-\widehat{x}^{(i)}, \tag*{(15)} x t ( i ) = AddNoise ( x ( i ) , t ) = ( 1 − t ) x ( i ) + t ϵ , v cond ( i ) = ϵ − x ( i ) , (15)
where v c o n d ( i ) v_{\mathrm{cond}}^{(i)} v cond ( i ) is the conditional velocity , i.e., the velocity of the straight path from x ^ ( i ) \widehat{x}^{(i)} x ( i ) to ϵ \epsilon ϵ , which is computed from the two endpoints without a network evaluation. AWM then observes that the flow-matching loss against that target is, for a suitable time weight w ( t ) w(t) w ( t ) , an evidence-lower-bound surrogate for the sequence-level log-likelihood of the clean sample,
log p θ ( x ^ ( i ) ∣ c ) ≥ log p ^ θ ( x ^ ( i ) ∣ c ) + c o n s t . , log p ^ θ ( x ^ ( i ) ∣ c ) : = − E t , ϵ [ w ( t ) ∥ v θ ( x ~ t ( i ) , t ) − v c o n d ( i ) ∥ 2 ] , (16) \log p_\theta(\widehat{x}^{(i)}\mid c)\ge\log\widehat{p}_\theta(\widehat{x}^{(i)}\mid c)+\mathrm{const.},\qquad\log\widehat{p}_\theta(\widehat{x}^{(i)}\mid c):=-\mathbb{E}_{t,\epsilon}\left[w(t)\left\|v_\theta(\widetilde{x}_t^{(i)},t)-v_{\mathrm{cond}}^{(i)}\right\|^2\right], \tag*{(16)} log p θ ( x ( i ) ∣ c ) ≥ log p θ ( x ( i ) ∣ c ) + const. , log p θ ( x ( i ) ∣ c ) := − E t , ϵ [ w ( t ) v θ ( x t ( i ) , t ) − v cond ( i ) 2 ] , (16)
so that the policy-gradient term A ( i ) ∇ θ log p ^ θ A^{(i)}\nabla_\theta\log\widehat{p}_\theta A ( i ) ∇ θ log p θ reduces to the pretraining objective reweighted per sample:
L A W M = E i , t , ϵ [ A ( i ) w ( t ) ∥ v θ ( x ~ t ( i ) , t ) − v c o n d ( i ) ∥ 2 ] , (17) \mathcal{L}_{\mathrm{AWM}}=\mathbb{E}_{i,t,\epsilon}\left[A^{(i)}w(t)\left\|v_\theta(\widetilde{x}_t^{(i)},t)-v_{\mathrm{cond}}^{(i)}\right\|^2\right], \tag*{(17)} L AWM = E i , t , ϵ [ A ( i ) w ( t ) v θ ( x t ( i ) , t ) − v cond ( i ) 2 ] , (17)
with the time weight w ( t ) = 1 w(t)=1 w ( t ) = 1 in the original AWM (our implementation effectively uses w ( t ) = τ A W M 2 w(t)=\tau_{\mathrm{AWM}}^2 w ( t ) = τ AWM 2 , § 5.4 ) and a velocity-space proximal penalty ∥ v θ − v o l d ∥ 2 \|v_\theta-v_{\mathrm{old}}\|^2 ∥ v θ − v old ∥ 2 against the old policy v o l d v_{\mathrm{old}} v old , called a KL term in AWM, added for stability. Minimising Equation (17) decreases the matching residual at x ^ ( i ) \widehat{x}^{(i)} x ( i ) when A ( i ) > 0 A^{(i)}>0 A ( i ) > 0 and increases it when A ( i ) < 0 A^{(i)}<0 A ( i ) < 0 . Since no reverse-time discretisation appears, the sampler can be chosen freely.
DiffusionNFT. Negative-aware fine-tuning ((Zheng et al., 2026a)) builds on an optimality-variable construction ((Levine, 2018)) and requires neither a likelihood nor a signed weight. It first maps the advantage to a bounded optimality probability with a normaliser Z c = std ( { R } ) + ϵ std Z_c = \operatorname{std}(\{R\}) + \epsilon_{\operatorname{std}} Z c = std ({ R }) + ϵ std ,
r ^ ( i ) = 1 2 + 1 2 clip [ R ( i ) − mean ( { R } ) Z c , − 1 , 1 ] = 1 2 + 1 2 clip ( A ( i ) , − 1 , 1 ) ∈ [ 0 , 1 ] , (18) \hat{r}^{(i)} = \frac{1}{2} + \frac{1}{2}\operatorname{clip}\left[\frac{R^{(i)}-\operatorname{mean}(\{R\})}{Z_c},-1,1\right] = \frac{1}{2} + \frac{1}{2}\operatorname{clip}\left(A^{(i)},-1,1\right) \in[0,1], \tag*{(18)} r ^ ( i ) = 2 1 + 2 1 clip [ Z c R ( i ) − mean ({ R }) , − 1 , 1 ] = 2 1 + 2 1 clip ( A ( i ) , − 1 , 1 ) ∈ [ 0 , 1 ] , (18)
since the bracketed quotient then equals Equation (12) . Thus A ( i ) A^{(i)} A ( i ) is signed, unclipped and zero-mean under Equation (12) and enters as a multiplier , whereas r ^ ( i ) \hat{r}^{(i)} r ^ ( i ) is bounded, monotone in A ( i ) A^{(i)} A ( i ) and equal to 1 2 \frac{1}{2} 2 1 when A ( i ) = 0 A^{(i)} = 0 A ( i ) = 0 , and enters as a mixing weight . A positive advantage gives r ^ ( i ) > 1 2 \hat{r}^{(i)} > \frac{1}{2} r ^ ( i ) > 2 1 and hence a larger weight on the positive branch.
Each update then draws a fresh ( t , ϵ ) (t,\epsilon) ( t , ϵ ) and forms the noised state x ~ t ( i ) \tilde{x}_t^{(i)} x ~ t ( i ) and the conditional velocity v c o n d ( i ) v_{\mathrm{cond}}^{(i)} v cond ( i ) as in Equation (15) ; the target is the same as in AWM, and, besides the weight, only the regressed quantity differs. Two implicit policies are built from one trainable network v θ v_\theta v θ and the frozen old policy v o l d v_{\mathrm{old}} v old , both evaluated at that same x ~ t ( i ) \tilde{x}_t^{(i)} x ~ t ( i ) and mixed by a branch coefficient β \beta β :
v θ ± ( x ~ t ( i ) , t ) : = ( 1 ∓ β ) v o l d ( x ~ t ( i ) , t ) ± β v θ ( x ~ t ( i ) , t ) . (19) v_\theta^\pm(\tilde{x}_t^{(i)},t) := (1 \mp\beta)v_{\mathrm{old}}(\tilde{x}_t^{(i)},t) \pm\beta v_\theta(\tilde{x}_t^{(i)},t). \tag*{(19)} v θ ± ( x ~ t ( i ) , t ) := ( 1 ∓ β ) v old ( x ~ t ( i ) , t ) ± β v θ ( x ~ t ( i ) , t ) . (19)
Both are regressed onto the same target v c o n d ( i ) v_{\mathrm{cond}}^{(i)} v cond ( i ) , mixed by r ^ ( i ) \hat{r}^{(i)} r ^ ( i ) :
L N F T = E i , t , ϵ [ r ^ ( i ) ∥ v θ + − v c o n d ( i ) ∥ 2 + ( 1 − r ^ ( i ) ) ∥ v θ − − v c o n d ( i ) ∥ 2 ] . (20) \mathcal{L}_{\mathrm{NFT}} = \mathbb{E}_{i,t,\epsilon}\left[\hat{r}^{(i)}\left\|v_\theta^+ - v_{\mathrm{cond}}^{(i)}\right\|^2 + \left(1-\hat{r}^{(i)}\right)\left\|v_\theta^- - v_{\mathrm{cond}}^{(i)}\right\|^2\right]. \tag*{(20)} L NFT = E i , t , ϵ [ r ^ ( i ) v θ + − v cond ( i ) 2 + ( 1 − r ^ ( i ) ) v θ − − v cond ( i ) 2 ] . (20)
Both branches are affine in the same trainable network v θ v_\theta v θ ; since v o l d v_{\mathrm{old}} v old , v c o n d ( i ) v_{\mathrm{cond}}^{(i)} v cond ( i ) and r ^ ( i ) \hat{r}^{(i)} r ^ ( i ) are detached, v θ v_\theta v θ is the only differentiated quantity in Equation (20) .
Framework
Each RL method of § 3.2 increases the relative probability of high-reward samples under the rollout distribution p o l d p_{\mathrm{old}} p old of the old policy. The methods differ in the space in which the update is formulated (transition densities, velocities, or rewards differentiated through the sample) and in how a few-step network computes the quantities that the update requires. This section resolves both differences in three steps, summarised in Figure 2 .
Transformation to x-space (§ 4.1 , Figure 2 a). Under the linear noise schedule a velocity transports a state affinely, so each objective of § 3 can be expressed through a common prediction in x x x -space. By default this prediction is the policy prediction x ~ θ \tilde{x}_\theta x ~ θ ; MeanFlow and FewStepGRPO keep an explicitly marked level-r r r quantity. In x x x -space the methods become directly comparable.
Decomposition of the gradient (§ 4.2 , Figure 2 b). In x x x -space, the gradient of each of these losses is the transposed Jacobian ( ∂ x ~ θ ∂ θ ) ⊤ \left(\frac{\partial\tilde{x}_\theta}{\partial\theta}\right)^\top ( ∂ θ ∂ x ~ θ ) ⊤ applied to the sum of a reward-free self-regression toward an anchor x ~ r e g \tilde{x}_{\mathrm{reg}} x ~ reg , which limits how far the prediction moves, and a reward augmentation along x ~ r w d − x ~ b a s e \tilde{x}_{\mathrm{rwd}}-\tilde{x}_{\mathrm{base}} x ~ rwd − x ~ base , which contains the reward. A method is then specified by the prediction, three anchors ( x ~ r e g , x ~ r w d , x ~ b a s e ) (\tilde{x}_{\mathrm{reg}},\tilde{x}_{\mathrm{rwd}},\tilde{x}_{\mathrm{base}}) ( x ~ reg , x ~ rwd , x ~ base ) and three weights ( W r e g , W r w d , ω ) (W_{\mathrm{reg}},W_{\mathrm{rwd}},\omega) ( W reg , W rwd , ω ) instead of a new loss.
Adaptation to few-step models (§ 4.3 , Figure 2 c). The framework requires instantaneous velocities, whereas a few-step model outputs an average velocity. Its anchors are therefore computed either at the end of the transport curve (u-form) or along the tangent at the current state (v-form). The latter recovers the instantaneous velocity from the network itself, as in MeanFlow, or from an auxiliary network, as in DMD.
§ 5 then specifies the predictions, anchors and weights of each family.
Transformation to x-space
The objectives of § 3 are written in transition-density or velocity space. Under the linear noise schedule, a velocity instead identifies a point in x x x -space at any chosen level r r r . This x x x -space includes noisy level-r r r points; clean samples correspond to r = 0 r=0 r = 0 .
For any point x t x_t x t at level t t t (in particular a renoised state x ~ t \tilde{x}_t x ~ t ), define the affine transport
X [ x t , v , r ] ≔ x t − ( t − r ) v . (21) X[x_t,v,r]\coloneqq x_t-(t-r)v. \tag*{(21)} X [ x t , v , r ] : = x t − ( t − r ) v . (21)
For two velocities evaluated at the same x t x_t x t and transported to the same level r < t r<t r < t ,
∥ v a − v b ∥ 2 = 1 ( t − r ) 2 ∥ X [ x t , v a , r ] − X [ x t , v b , r ] ∥ 2 . (22) \lVert v_a-v_b\rVert^2=\frac{1}{(t-r)^2}\left\lVert X[x_t,v_a,r]-X[x_t,v_b,r]\right\rVert^2. \tag*{(22)} ∥ v a − v b ∥ 2 = ( t − r ) 2 1 ∥ X [ x t , v a , r ] − X [ x t , v b , r ] ∥ 2 . (22)
Thus a velocity MSE equals a scaled MSE between transported points in x x x -space. At the clean level r = 0 r=0 r = 0 the factor is t − 2 t^{-2} t − 2 , and we omit the level subscript. In particular, for x ~ t = ( 1 − t ) x + t ϵ \tilde{x}_t=(1-t)x+t\epsilon x ~ t = ( 1 − t ) x + t ϵ and v c o n d = ϵ − x v_{\mathrm{cond}}=\epsilon-x v cond = ϵ − x , Equation (21) gives X [ x ~ t , v c o n d , 0 ] = x X[\tilde{x}_t,v_{\mathrm{cond}},0]=x X [ x ~ t , v cond , 0 ] = x .
For the reward objectives below, a detached rollout endpoint x ^ ( i ) \hat{x}^{(i)} x ^ ( i ) is re-noised to x ~ t ( i ) \tilde{x}_{t}^{(i)} x ~ t ( i ) , and the policy provides the current policy prediction
x ~ θ ( i ) = X [ x ~ t ( i ) , v θ , 0 ] = x ~ t ( i ) − t v θ , (23) \tilde{x}_{\theta}^{(i)} = X[\tilde{x}_{t}^{(i)}, v_{\theta}, 0] = \tilde{x}_{t}^{(i)} - t v_{\theta}, \tag*{(23)} x ~ θ ( i ) = X [ x ~ t ( i ) , v θ , 0 ] = x ~ t ( i ) − t v θ , (23)
while the conditional target maps back to x ^ ( i ) \hat{x}^{(i)} x ^ ( i ) . Hence the loss compares predictions at the same state x ~ t ( i ) \tilde{x}_{t}^{(i)} x ~ t ( i ) , and only x ~ θ ( i ) \tilde{x}_{\theta}^{(i)} x ~ θ ( i ) is differentiated. § 4.3 constructs this prediction when the network outputs an average velocity instead of an instantaneous one. Analysing a loss in x x x -space therefore shows how the prediction x ~ θ \tilde{x}_{\theta} x ~ θ must change to reduce the loss, i.e., which points x ~ θ \tilde{x}_{\theta} x ~ θ moves toward and which it moves away from. The same map applied to the old policy gives the old policy’s prediction
x ~ o l d ( i ) = X [ x ~ t ( i ) , v o l d , 0 ] , (24) \tilde{x}_{\mathrm{old}}^{(i)} = X[\tilde{x}_{t}^{(i)}, v_{\mathrm{old}}, 0], \tag*{(24)} x ~ old ( i ) = X [ x ~ t ( i ) , v old , 0 ] , (24)
which serves as an anchor in several methods below; its few-step variants, the self-induced prediction of Example 5 and the u-prediction and J J J -free query of § 4.3 , are collected in Appendix A.
Example 1 (AWM in x-space). The target of Equation (17) is v c o n d ( i ) v_{\mathrm{cond}}^{(i)} v cond ( i ) , and Equation (21) sends it to the rollout endpoint itself, X [ x ~ t ( i ) , v c o n d ( i ) , 0 ] = x ^ ( i ) X[\tilde{x}_{t}^{(i)}, v_{\mathrm{cond}}^{(i)}, 0] = \hat{x}^{(i)} X [ x ~ t ( i ) , v cond ( i ) , 0 ] = x ^ ( i ) . Applying Equation (22) ,
L A W M = E i , t , ϵ [ A ( i ) w ( t ) t 2 ∥ x ~ θ ( i ) − x ^ ( i ) ∥ 2 ] . (25) \mathcal{L}_{\mathrm{AWM}} = \mathbb{E}_{i,t,\epsilon}\left[A^{(i)}\frac{w(t)}{t^{2}}\left\|\tilde{x}_{\theta}^{(i)}-\hat{x}^{(i)}\right\|^{2}\right]. \tag*{(25)} L AWM = E i , t , ϵ [ A ( i ) t 2 w ( t ) x ~ θ ( i ) − x ^ ( i ) 2 ] . (25) The method is a squared residual between the current prediction and the model’s sample, weighted by the signed advantage. Taking w ( t ) = t 2 w(t)=t^{2} w ( t ) = t 2 cancels the prefactor and gives an x x x -space weight independent of t t t ; this is the choice our few-step implementation makes up to a floor (§ 5.4 ).
Example 2 (DiffusionNFT in x-space). The target of Equation (20) is again v c o n d ( i ) v_{\mathrm{cond}}^{(i)} v cond ( i ) , so Equation (21) sends it to the same rollout endpoint x ^ ( i ) \hat{x}^{(i)} x ^ ( i ) . Only the prediction differs; the two implicit policies Equation (19) are mapped to
x ~ θ ± = X [ x ~ t ( i ) , v θ ± , 0 ] = x ~ o l d ( i ) ± β ( x ~ θ ( i ) − x ~ o l d ( i ) ) , (26) \tilde{x}_{\theta}^{\pm} = X[\tilde{x}_{t}^{(i)},v_{\theta}^{\pm},0] = \tilde{x}_{\mathrm{old}}^{(i)} \pm\beta(\tilde{x}_{\theta}^{(i)}-\tilde{x}_{\mathrm{old}}^{(i)}), \tag*{(26)} x ~ θ ± = X [ x ~ t ( i ) , v θ ± , 0 ] = x ~ old ( i ) ± β ( x ~ θ ( i ) − x ~ old ( i ) ) , (26) with x ~ o l d ( i ) \tilde{x}_{\mathrm{old}}^{(i)} x ~ old ( i ) as in Equation (24) . Applying Equation (22) to each branch,
L N F T = E i , t , ϵ [ 1 t 2 ( r ^ ( i ) ∥ x ~ θ + − x ^ ( i ) ∥ 2 + ( 1 − r ^ ( i ) ) ∥ x ~ θ − − x ^ ( i ) ∥ 2 ) ] . (27) \mathcal{L}_{\mathrm{NFT}} = \mathbb{E}_{i,t,\epsilon}\left[\frac{1}{t^{2}}\left(\hat{r}^{(i)}\left\|\tilde{x}_{\theta}^{+}-\hat{x}^{(i)}\right\|^{2}+\left(1-\hat{r}^{(i)}\right)\left\|\tilde{x}_{\theta}^{-}-\hat{x}^{(i)}\right\|^{2}\right)\right]. \tag*{(27)} L NFT = E i , t , ϵ [ t 2 1 ( r ^ ( i ) x ~ θ + − x ^ ( i ) 2 + ( 1 − r ^ ( i ) ) x ~ θ − − x ^ ( i ) 2 ) ] . (27) The method mixes two squared residuals to the model’s sample with weights r ^ ( i ) \hat{r}^{(i)} r ^ ( i ) and 1 − r ^ ( i ) 1-\hat{r}^{(i)} 1 − r ^ ( i ) . The positive branch receives more weight when the reward of the sample is above the group mean, and the negative branch when it is below. The common prefactor t − 2 t^{-2} t − 2 can be absorbed into the weights, so the mixing itself is independent of t t t .
The Decoupled Loss Decomposition
Under the transformation Equation (21) , the gradient of each objective considered here is the transposed Jacobian ( ∂ x ~ θ ( i ) / ∂ θ ) ⊤ (\partial\tilde{x}_{\theta}^{(i)}/\partial\theta)^{\top} ( ∂ x ~ θ ( i ) / ∂ θ ) ⊤ applied to the sum of two vectors in x x x -space (at the clean level by default and at level r r r in the explicit exceptions). We adopt this form as the general framework,
∇ θ L ( i ) = 2 ( ∂ x ~ θ ( i ) ∂ θ ) ⊤ [ W r e g ( i ) ( x ~ θ ( i ) − x ~ r e g ( i ) ) ⏟ self-regression − W r w d ( i ) ω ( i ) ( x ~ r w d ( i ) − x ~ b a s e ( i ) ) ⏟ reward augmentation ] . (28) \nabla_{\theta}\mathcal{L}^{(i)} = 2\left(\frac{\partial\tilde{x}_{\theta}^{(i)}}{\partial\theta}\right)^{\top}\left[\underbrace{W_{\mathrm{reg}}^{(i)}(\tilde{x}_{\theta}^{(i)}-\tilde{x}_{\mathrm{reg}}^{(i)})}_{\text{self-regression}}-\underbrace{W_{\mathrm{rwd}}^{(i)}\omega^{(i)}(\tilde{x}_{\mathrm{rwd}}^{(i)}-\tilde{x}_{\mathrm{base}}^{(i)})}_{\text{reward augmentation}}\right]. \tag*{(28)} ∇ θ L ( i ) = 2 ( ∂ θ ∂ x ~ θ ( i ) ) ⊤ self-regression W reg ( i ) ( x ~ θ ( i ) − x ~ reg ( i ) ) − reward augmentation W rwd ( i ) ω ( i ) ( x ~ rwd ( i ) − x ~ base ( i ) ) . (28)
with W r e g ( i ) , W r w d ( i ) ≥ 0 W_{\mathrm{reg}}^{(i)}, W_{\mathrm{rwd}}^{(i)} \ge0 W reg ( i ) , W rwd ( i ) ≥ 0 and ω ( i ) ∈ R \omega^{(i)} \in\mathbb{R} ω ( i ) ∈ R detached scalars. The factor 2 from differentiating a square is kept outside the bracket, so all weights below are defined relative to Equation (28) ; for example, the loss W ∥ x ~ θ − x ~ r w d ∥ 2 W\left\|\tilde{x}_{\theta}-\tilde{x}_{\mathrm{rwd}}\right\|^{2} W ∥ x ~ θ − x ~ rwd ∥ 2 corresponds to W r w d = W W_{\mathrm{rwd}}=W W rwd = W . The prediction x ~ θ ( i ) \tilde{x}_{\theta}^{(i)} x ~ θ ( i ) is constructed by first noising the rollout sample x ^ ( i ) \hat{x}^{(i)} x ^ ( i ) to level t t t and then transporting the resulting state x ~ t ( i ) \tilde{x}_{t}^{(i)} x ~ t ( i ) back to r = 0 r=0 r = 0 by Equation (21) . Here ω ( i ) \omega^{(i)} ω ( i ) is the signed reward coefficient used in the general loss Equation (30) , and ∣ ω ( i ) ∣ \lvert\omega^{(i)}\rvert ∣ ω ( i ) ∣ is its magnitude. For methods whose coefficient follows the advantage, a non-zero ω ( i ) \omega^{(i)} ω ( i ) has the sign of A ( i ) A^{(i)} A ( i ) in the theoretical weighting (the implemented DiffusionNFT weighting of §5.5 can deviate), and a clipping gate may set it to zero.
Self-regression. The first term does not depend on the reward. It vanishes at x ~ θ ( i ) = x ~ r e g ( i ) \tilde{x}_{\theta}^{(i)}=\tilde{x}_{\mathrm{reg}}^{(i)} x ~ θ ( i ) = x ~ reg ( i ) and grows with the distance from this anchor, so its gradient moves the prediction back toward the anchor. This self-regression term limits how far the reward-augmentation term can move the prediction and therefore helps protect against reward hacking.
The regression anchor is therefore chosen as a prediction that the policy should stay close to. A common choice is the old policy’s prediction x ~ o l d ( i ) = X [ x ~ t ( i ) , v o l d , 0 ] \tilde{x}_{\mathrm{old}}^{(i)}=X\left[\tilde{x}_{t}^{(i)},v_{\mathrm{old}},0\right] x ~ old ( i ) = X [ x ~ t ( i ) , v old , 0 ] , so that the term penalises the deviation of the current prediction from that of the policy used for sampling. Since x ~ θ ( i ) \tilde{x}_{\theta}^{(i)} x ~ θ ( i ) is computed from x ~ t ( i ) \tilde{x}_{t}^{(i)} x ~ t ( i ) , the anchor x ~ r e g ( i ) \tilde{x}_{\mathrm{reg}}^{(i)} x ~ reg ( i ) must be computed from the same x ~ t ( i ) \tilde{x}_{t}^{(i)} x ~ t ( i ) ; otherwise, the difference x ~ θ ( i ) − x ~ r e g ( i ) \tilde{x}_{\theta}^{(i)}-\tilde{x}_{\mathrm{reg}}^{(i)} x ~ θ ( i ) − x ~ reg ( i ) would mix a change of the velocity field with a change of the state and would no longer measure the deviation that the term is meant to bound.
Reward augmentation. For fixed anchors, the second term contributes a vector that does not depend on the current prediction and contains the reward. Under gradient descent it moves x ~ θ ( i ) \tilde{x}_{\theta}^{(i)} x ~ θ ( i ) along + ω ( i ) ( x ~ r w d ( i ) − x ~ b a s e ( i ) ) +\omega^{(i)}\left(\tilde{x}_{\mathrm{rwd}}^{(i)}-\tilde{x}_{\mathrm{base}}^{(i)}\right) + ω ( i ) ( x ~ rwd ( i ) − x ~ base ( i ) ) , a displacement whose direction is set by the sign of ω ( i ) \omega^{(i)} ω ( i ) and whose coefficient has magnitude ∣ ω ( i ) ∣ \lvert\omega^{(i)}\rvert ∣ ω ( i ) ∣ . The magnitude of the full term also depends on W r w d ( i ) W_{\mathrm{rwd}}^{(i)} W rwd ( i ) and on the anchor displacement x ~ r w d ( i ) − x ~ b a s e ( i ) \tilde{x}_{\mathrm{rwd}}^{(i)}-\tilde{x}_{\mathrm{base}}^{(i)} x ~ rwd ( i ) − x ~ base ( i ) .
The rewarded anchor x ~ r w d ( i ) \tilde{x}_{\mathrm{rwd}}^{(i)} x ~ rwd ( i ) is usually the rollout sample, since transporting x ~ t ( i ) \tilde{x}_{t}^{(i)} x ~ t ( i ) with the conditional velocity gives X [ x ~ t ( i ) , v c o n d ( i ) , 0 ] = x ^ ( i ) X\left[\tilde{x}_{t}^{(i)},v_{\mathrm{cond}}^{(i)},0\right]=\hat{x}^{(i)} X [ x ~ t ( i ) , v cond ( i ) , 0 ] = x ^ ( i ) . This choice is natural because A ( i ) A^{(i)} A ( i ) is obtained by scoring x ^ ( i ) \hat{x}^{(i)} x ^ ( i ) and provides no direct information about other points. For methods whose coefficient follows the advantage, the sign of ω ( i ) \omega^{(i)} ω ( i ) determines whether the prediction moves toward x ~ r w d ( i ) \tilde{x}_{\mathrm{rwd}}^{(i)} x ~ rwd ( i ) or away from it.
The other end, x ~ b a s e ( i ) \tilde{x}_{\mathrm{base}}^{(i)} x ~ base ( i ) , is the baseline anchor. The displacement x ~ r w d ( i ) − x ~ b a s e ( i ) \tilde{x}_{\mathrm{rwd}}^{(i)}-\tilde{x}_{\mathrm{base}}^{(i)} x ~ rwd ( i ) − x ~ base ( i ) points from the baseline anchor to the rewarded anchor, so x ~ b a s e ( i ) \tilde{x}_{\mathrm{base}}^{(i)} x ~ base ( i ) sets the reference point and the line along which the reward acts. Admissible choices for x ~ b a s e ( i ) \tilde{x}_{\mathrm{base}}^{(i)} x ~ base ( i ) include the old policy’s prediction x ~ o l d ( i ) \tilde{x}_{\mathrm{old}}^{(i)} x ~ old ( i ) and the detached current prediction sg ( x ~ θ ( i ) ) \operatorname{sg}(\tilde{x}_{\theta}^{(i)}) sg ( x ~ θ ( i ) ) . In the RL methods, x ~ b a s e ( i ) \tilde{x}_{\mathrm{base}}^{(i)} x ~ base ( i ) is also evaluated at the state x ~ t ( i ) \tilde{x}_{t}^{(i)} x ~ t ( i ) from which x ~ θ ( i ) \tilde{x}_{\theta}^{(i)} x ~ θ ( i ) is computed, so that the two anchors define a displacement in the space on which the update acts.
An intuitive example. Consider a group-advantage-based method with the group-mean advantage of Equation (12) and a group in which A ( 1 ) > A ( 2 ) > 0 > A ( 3 ) A^{(1)}>A^{(2)}>0>A^{(3)} A ( 1 ) > A ( 2 ) > 0 > A ( 3 ) . Members 1 and 2 both score above the group mean, so ω ( 1 ) , ω ( 2 ) > 0 \omega^{(1)},\omega^{(2)}>0 ω ( 1 ) , ω ( 2 ) > 0 and both predictions are attracted toward their rewarded samples. Member 2 is not penalised for scoring lower than member 1; it only receives a coefficient of smaller or equal magnitude, since ∣ ω ( 1 ) ∣ ≥ ∣ ω ( 2 ) ∣ \lvert\omega^{(1)}\rvert\ge\lvert\omega^{(2)}\rvert ∣ ω ( 1 ) ∣ ≥ ∣ ω ( 2 ) ∣ in AWM and in the theoretical DiffusionNFT weighting below. Member 3 scores below the mean, so ω ( 3 ) < 0 \omega^{(3)}<0 ω ( 3 ) < 0 and its prediction is repelled away from x ~ r w d ( 3 ) \tilde{x}_{\mathrm{rwd}}^{(3)} x ~ rwd ( 3 ) , in the direction x ~ b a s e ( 3 ) − x ~ r w d ( 3 ) \tilde{x}_{\mathrm{base}}^{(3)}-\tilde{x}_{\mathrm{rwd}}^{(3)} x ~ base ( 3 ) − x ~ rwd ( 3 ) . Whether a prediction is attracted or repelled is therefore determined by the baseline-subtracted advantage A ( i ) A^{(i)} A ( i ) and not by the raw reward; a sample with a high reward is still repelled if it scores below its group mean.
Recovering the loss. For any V θ V_\theta V θ and any detached U U U ,
− ( ∂ V θ ∂ θ ) ⊤ U = ∇ θ 1 2 ∥ V θ − sg ( V θ + U ) ∥ 2 . (29) -\left(\frac{\partial V_\theta}{\partial\theta}\right)^\top U=\nabla_\theta\frac{1}{2}\left\|V_\theta-\operatorname{sg}(V_\theta+U)\right\|^2. \tag*{(29)} − ( ∂ θ ∂ V θ ) ⊤ U = ∇ θ 2 1 ∥ V θ − sg ( V θ + U ) ∥ 2 . (29)
Therefore, applying Equation (29) with V θ = x ~ θ ( i ) V_\theta=\tilde{x}^{(i)}_\theta V θ = x ~ θ ( i ) and U = g ( i ) : = ω ( i ) ( x ~ r w d ( i ) − x ~ b a s e ( i ) ) U=g^{(i)}:=\omega^{(i)}(\tilde{x}^{(i)}_{\mathrm{rwd}}-\tilde{x}^{(i)}_{\mathrm{base}}) U = g ( i ) := ω ( i ) ( x ~ rwd ( i ) − x ~ base ( i ) ) writes Equation (28) as a sum of two squares,
L ( i ) = W r e g ( i ) ∥ x ~ θ ( i ) − x ~ r e g ( i ) ∥ 2 + W r w d ( i ) ∥ x ~ θ ( i ) − sg ( x ~ θ ( i ) + g ( i ) ) ∥ 2 , (30) \mathcal{L}^{(i)}=W^{(i)}_{\mathrm{reg}}\left\|\tilde{x}^{(i)}_\theta-\tilde{x}^{(i)}_{\mathrm{reg}}\right\|^2+W^{(i)}_{\mathrm{rwd}}\left\|\tilde{x}^{(i)}_\theta-\operatorname{sg}\left(\tilde{x}^{(i)}_\theta+g^{(i)}\right)\right\|^2, \tag*{(30)} L ( i ) = W reg ( i ) x ~ θ ( i ) − x ~ reg ( i ) 2 + W rwd ( i ) x ~ θ ( i ) − sg ( x ~ θ ( i ) + g ( i ) ) 2 , (30)
which is the form § 5 implements.
Example 3 (Decoupling AWM loss). Applying the same decomposition to Equation (25) gives a limiting case of Equation (28) . Differentiating it and rewriting the residual as a difference of anchors,
∇ θ L A W M ( i ) = 2 w ( t ) t 2 A ( i ) ( ∂ x ~ θ ( i ) ∂ θ ) ⊤ ( x ~ θ ( i ) − x ^ ( i ) ) \nabla_\theta\mathcal{L}^{(i)}_{\mathrm{AWM}}=2\frac{w(t)}{t^2}A^{(i)}\left(\frac{\partial\tilde{x}^{(i)}_\theta}{\partial\theta}\right)^\top\left(\tilde{x}^{(i)}_\theta-\hat{x}^{(i)}\right) ∇ θ L AWM ( i ) = 2 t 2 w ( t ) A ( i ) ( ∂ θ ∂ x ~ θ ( i ) ) ⊤ ( x ~ θ ( i ) − x ^ ( i ) ) = − 2 ( ∂ x ~ θ ( i ) ∂ θ ) ⊤ w ( t ) t 2 A ( i ) ( x ^ ( i ) − sg ( x ~ θ ( i ) ) ) , (31) =-2\left(\frac{\partial\tilde{x}^{(i)}_\theta}{\partial\theta}\right)^\top\frac{w(t)}{t^2}A^{(i)}\left(\hat{x}^{(i)}-\operatorname{sg}\left(\tilde{x}^{(i)}_\theta\right)\right), \tag*{(31)} = − 2 ( ∂ θ ∂ x ~ θ ( i ) ) ⊤ t 2 w ( t ) A ( i ) ( x ^ ( i ) − sg ( x ~ θ ( i ) ) ) , (31) which is Equation (28) with W r e g = 0 W_{\mathrm{reg}}=0 W reg = 0 :
W r e g ( i ) = 0 , x ~ r w d ( i ) = x ^ ( i ) , x ~ b a s e ( i ) = sg ( x ~ θ ( i ) ) , W r w d ( i ) = w ( t ) t 2 , ω ( i ) = A ( i ) . (32) W^{(i)}_{\mathrm{reg}}=0,\qquad\tilde{x}^{(i)}_{\mathrm{rwd}}=\hat{x}^{(i)},\qquad\tilde{x}^{(i)}_{\mathrm{base}}=\operatorname{sg}\left(\tilde{x}^{(i)}_\theta\right),\qquad W^{(i)}_{\mathrm{rwd}}=\frac{w(t)}{t^2},\qquad\omega^{(i)}=A^{(i)}. \tag*{(32)} W reg ( i ) = 0 , x ~ rwd ( i ) = x ^ ( i ) , x ~ base ( i ) = sg ( x ~ θ ( i ) ) , W rwd ( i ) = t 2 w ( t ) , ω ( i ) = A ( i ) . (32) The anchor x ~ r e g ( i ) \tilde{x}^{(i)}_{\mathrm{reg}} x ~ reg ( i ) is arbitrary, since it is multiplied by zero. Compared with the DiffusionNFT decomposition (Example 4 below, Equation (35) ), there are three main differences. First, the baseline anchor is the detached current prediction, so one end of the reward displacement is fixed and the other moves with the policy. The term is therefore recomputed at every step and is proportional to the residual x ^ ( i ) − sg ( x ~ θ ( i ) ) \hat{x}^{(i)}-\operatorname{sg}(\tilde{x}^{(i)}_\theta) x ^ ( i ) − sg ( x ~ θ ( i ) ) instead of being a fixed displacement; this is why AWM can be written as the single weighted square Equation (25) . Second, in Equation (32) the magnitude ∣ ω ( i ) ∣ = ∣ A ( i ) ∣ |\omega^{(i)}|=|A^{(i)}| ∣ ω ( i ) ∣ = ∣ A ( i ) ∣ is not clipped, whereas in DiffusionNFT it saturates at 1 / β 1/\beta 1/ β . Third, with W r e g = 0 W_{\mathrm{reg}}=0 W reg = 0 the matching loss contains no self-regression term; this role is played by the additional velocity-space proximal penalty ∥ v θ − v o l d ∥ 2 \|v_\theta-v_{\mathrm{old}}\|^2 ∥ v θ − v old ∥ 2 of § 3.2 , which under Equation (22) is a self-regression toward the old policy’s prediction x ~ o l d ( i ) \tilde{x}^{(i)}_{\mathrm{old}} x ~ old ( i ) .
Example 4 (Decoupling DiffusionNFT loss). Applying the same decomposition to Equation (27) yields both terms of Equation (28) . The two branches Equation (26) are affine in the same current prediction, so
∂ x ~ θ + ∂ θ = β ∂ x ~ θ ( i ) ∂ θ , ∂ x ~ θ − ∂ θ = − β ∂ x ~ θ ( i ) ∂ θ . (33) \frac{\partial\tilde{x}^{+}_\theta}{\partial\theta}=\beta\frac{\partial\tilde{x}^{(i)}_\theta}{\partial\theta},\qquad\frac{\partial\tilde{x}^{-}_\theta}{\partial\theta}=-\beta\frac{\partial\tilde{x}^{(i)}_\theta}{\partial\theta}. \tag*{(33)} ∂ θ ∂ x ~ θ + = β ∂ θ ∂ x ~ θ ( i ) , ∂ θ ∂ x ~ θ − = − β ∂ θ ∂ x ~ θ ( i ) . (33)
Differentiating Equation (27) and writing x ~ b a s e ( i ) : = x ~ o l d ( i ) \tilde{x}_{\mathrm{base}}^{(i)} := \tilde{x}_{\mathrm{old}}^{(i)} x ~ base ( i ) := x ~ old ( i ) , the two branches collapse onto a single Jacobian. Since r ^ ( i ) = 1 2 + 1 2 clip ( A ( i ) , − 1 , 1 ) \hat{r}^{(i)} = \frac{1}{2} + \frac{1}{2}\operatorname{clip}(A^{(i)},-1,1) r ^ ( i ) = 2 1 + 2 1 clip ( A ( i ) , − 1 , 1 ) ,
∇ θ L N F T ( i ) = 2 β 2 t 2 ( ∂ x ~ θ ( i ) ∂ θ ) ⊤ [ ( x ~ θ ( i ) − x ~ b a s e ( i ) ) − clip ( A ( i ) , − 1 , 1 ) β ( x ^ ( i ) − x ~ b a s e ( i ) ) ] , (34) \nabla_{\theta}\mathcal{L}_{\mathrm{NFT}}^{(i)} = \frac{2\beta^{2}}{t^{2}}\left(\frac{\partial\tilde{x}_{\theta}^{(i)}}{\partial\theta}\right)^{\top}\left[\left(\tilde{x}_{\theta}^{(i)}-\tilde{x}_{\mathrm{base}}^{(i)}\right)-\frac{\operatorname{clip}(A^{(i)},-1,1)}{\beta}\left(\hat{x}^{(i)}-\tilde{x}_{\mathrm{base}}^{(i)}\right)\right], \tag*{(34)} ∇ θ L NFT ( i ) = t 2 2 β 2 ( ∂ θ ∂ x ~ θ ( i ) ) ⊤ [ ( x ~ θ ( i ) − x ~ base ( i ) ) − β clip ( A ( i ) , − 1 , 1 ) ( x ^ ( i ) − x ~ base ( i ) ) ] , (34)
which matches Equation (28) term by term, with a self-regression toward x ~ b a s e ( i ) \tilde{x}_{\mathrm{base}}^{(i)} x ~ base ( i ) and a reward augmentation along the fixed direction x ^ ( i ) − x ~ b a s e ( i ) \hat{x}^{(i)}-\tilde{x}_{\mathrm{base}}^{(i)} x ^ ( i ) − x ~ base ( i ) . The corresponding anchors and weights are
x ~ r e g ( i ) = x ~ b a s e ( i ) = X [ x ~ t ( i ) , v o l d , 0 ] , x ~ r w d ( i ) = x ^ ( i ) , W r e g ( i ) = W r w d ( i ) = β 2 t 2 , ω ( i ) = clip ( A ( i ) , − 1 , 1 ) β . (35) \tilde{x}_{\mathrm{reg}}^{(i)}=\tilde{x}_{\mathrm{base}}^{(i)}=X\left[\tilde{x}_{t}^{(i)},v_{\mathrm{old}},0\right],\qquad\tilde{x}_{\mathrm{rwd}}^{(i)}=\hat{x}^{(i)},\qquad W_{\mathrm{reg}}^{(i)}=W_{\mathrm{rwd}}^{(i)}=\frac{\beta^{2}}{t^{2}},\qquad\omega^{(i)}=\frac{\operatorname{clip}(A^{(i)},-1,1)}{\beta}. \tag*{(35)} x ~ reg ( i ) = x ~ base ( i ) = X [ x ~ t ( i ) , v old , 0 ] , x ~ rwd ( i ) = x ^ ( i ) , W reg ( i ) = W rwd ( i ) = t 2 β 2 , ω ( i ) = β clip ( A ( i ) , − 1 , 1 ) . (35)
Both anchors coincide at the old policy’s clean prediction, so DiffusionNFT regularises toward the point from which it measures the reward. The rewarded sample is the rollout produced by the model. The two weights are equal, which allows the published loss to be written as one fused square up to a detached term. The signed reward coefficient ω ( i ) = clip ( A ( i ) , − 1 , 1 ) / β \omega^{(i)}=\operatorname{clip}(A^{(i)},-1,1)/\beta ω ( i ) = clip ( A ( i ) , − 1 , 1 ) / β has a magnitude that increases with ∣ A ( i ) ∣ \lvert A^{(i)}\rvert ∣ A ( i ) ∣ and saturates at 1 / β 1/\beta 1/ β because of the clip. In comparison, methods with W r e g ( i ) = 0 W_{\mathrm{reg}}^{(i)}=0 W reg ( i ) = 0 , such as AWM above, drop the self-regression from the matching loss.
Example 5 (Reward Augmentation for DMD and MeanFlow). The decomposition also applies to reward-free distillation losses and yields their decoupled form. Neither objective scores a group, so there is no advantage and no index ( i ) (i) ( i ) ; both move the policy toward the teacher. In the decomposition this is the limiting case
i.e., a unit positive coefficient for every sample, with the rewarded anchor set to the teacher’s prediction
x ~ r e a l = X [ x ~ t , v r e a l , 0 ] = x ~ t − t v r e a l . (36) \tilde{x}_{\mathrm{real}}=X\left[\tilde{x}_{t},v_{\mathrm{real}},0\right]=\tilde{x}_{t}-tv_{\mathrm{real}}. \tag*{(36)} x ~ real = X [ x ~ t , v real , 0 ] = x ~ t − t v real . (36)
MeanFlow. The network output in Equation (8) is an average velocity, so the corresponding instantaneous velocity is recovered before Equation (21) is applied. Rearranging the MeanFlow identity Equation (6) with Equation (7) recovers it as u θ + J [ u θ , t , r ] u_{\theta}+J[u_{\theta},t,r] u θ + J [ u θ , t , r ] . The correction J J J is detached, so we evaluate it, together with the average velocity it corrects, on the frozen copy u o l d u_{\mathrm{old}} u old of the network, which is sg ( u θ ) \operatorname{sg}(u_{\theta}) sg ( u θ ) in distillation and the old policy in RL. In particular, in MeanFlow u o l d u_{\mathrm{old}} u old equals u θ u_{\theta} u θ in value; we write u o l d u_{\mathrm{old}} u old only to unify the notation with the RL methods below:
v i n d ( x ~ t , t , r ) : = u o l d ( x ~ t , t , r ) + J [ u o l d , t , r ] , (37) v_{\mathrm{ind}}(\tilde{x}_{t},t,r):=u_{\mathrm{old}}(\tilde{x}_{t},t,r)+J[u_{\mathrm{old}},t,r], \tag*{(37)} v ind ( x ~ t , t , r ) := u old ( x ~ t , t , r ) + J [ u old , t , r ] , (37)
and we call v i n d v_{\mathrm{ind}} v ind the self-induced instantaneous velocity . It is computed from the average velocity and its total derivative with respect to t t t , without a second network, and coincides with the instantaneous velocity of u o l d u_{\mathrm{old}} u old wherever the flow map is exact. Its clean prediction is
x ~ i n d = X [ x ~ t , v i n d , 0 ] = x ~ t − t v i n d , (38) \tilde{x}_{\mathrm{ind}}=X\left[\tilde{x}_{t},v_{\mathrm{ind}},0\right]=\tilde{x}_{t}-tv_{\mathrm{ind}}, \tag*{(38)} x ~ ind = X [ x ~ t , v ind , 0 ] = x ~ t − t v ind , (38)
where the subscript indicates the velocity used. Both v i n d v_{\mathrm{ind}} v ind and x ~ i n d \tilde{x}_{\mathrm{ind}} x ~ ind are detached. Both fields are now instantaneous, so Equation (21) transports them over the same interval. According to Equation (8) , the prediction in x x x -space is extended from the clean level to any level r < t r<t r < t . Transporting with the teacher and with the self-induced velocity to level r r r ,
x ~ r e a l , r = X [ x ~ t , v r e a l , r ] = x ~ t + ( r − t ) v r e a l , x ~ i n d , r = X [ x ~ t , v i n d , r ] = x ~ t + ( r − t ) v i n d , (39) \tilde{x}_{\mathrm{real},r}=X\left[\tilde{x}_{t},v_{\mathrm{real}},r\right]=\tilde{x}_{t}+(r-t)v_{\mathrm{real}},\qquad\tilde{x}_{\mathrm{ind},r}=X\left[\tilde{x}_{t},v_{\mathrm{ind}},r\right]=\tilde{x}_{t}+(r-t)v_{\mathrm{ind}}, \tag*{(39)} x ~ real , r = X [ x ~ t , v real , r ] = x ~ t + ( r − t ) v real , x ~ ind , r = X [ x ~ t , v ind , r ] = x ~ t + ( r − t ) v ind , (39)
whose clean ends at r = 0 r = 0 r = 0 are Equation (36) and Equation (38) . The policy’s own transport with the same detached J = J [ u o l d , t , r ] J = J[u_{\mathrm{old}}, t, r] J = J [ u old , t , r ] is X [ x ~ t , u θ + J , r ] = x ~ θ , r + ( r − t ) J X[\tilde{x}_t, u_\theta+ J, r] = \tilde{x}_{\theta,r} + (r - t)J X [ x ~ t , u θ + J , r ] = x ~ θ , r + ( r − t ) J , with x ~ θ , r = X [ x ~ t , u θ , r ] \tilde{x}_{\theta,r} = X[\tilde{x}_t, u_\theta, r] x ~ θ , r = X [ x ~ t , u θ , r ] , and equals x ~ i n d , r \tilde{x}_{\mathrm{ind},r} x ~ ind , r in value. For r < t r < t r < t , Equation (8) is
L M F = E x ~ t , t , r [ ∥ X [ x ~ t , u θ + J , r ] − x ~ r e a l , r ∥ 2 ( t − r ) 2 ] , r < t . (40) \mathcal{L}_{\mathrm{MF}} = \mathbb{E}_{\tilde{x}_t,t,r}\left[\frac{\left\|X[\tilde{x}_t,u_\theta+J,r]-\tilde{x}_{\mathrm{real},r}\right\|^2}{(t-r)^2}\right], \qquad r<t. \tag*{(40)} L MF = E x ~ t , t , r [ ( t − r ) 2 ∥ X [ x ~ t , u θ + J , r ] − x ~ real , r ∥ 2 ] , r < t . (40)
At r = t r = t r = t the quotient is defined by continuity:
lim r ↑ t ∥ X [ x ~ t , u θ + J , r ] − x ~ r e a l , r ∥ 2 ( t − r ) 2 = ∥ u θ ( x ~ t , t , t ) − v r e a l ( x ~ t , t ) ∥ 2 . \lim_{r \uparrow t}\frac{\left\|X[\tilde{x}_t,u_\theta+J,r]-\tilde{x}_{\mathrm{real},r}\right\|^2}{(t-r)^2}=\left\|u_\theta(\tilde{x}_t,t,t)-v_{\mathrm{real}}(\tilde{x}_t,t)\right\|^2. r ↑ t lim ( t − r ) 2 ∥ X [ x ~ t , u θ + J , r ] − x ~ real , r ∥ 2 = ∥ u θ ( x ~ t , t , t ) − v real ( x ~ t , t ) ∥ 2 .
Thus MeanFlow is the squared velocity residual written as a scaled distance between level-r r r predictions. To obtain the loss form from Equation (28) , we differentiate the average velocity, since J J J is detached:
∂ ∂ θ X [ x ~ t , u θ + J , r ] = ∂ ∂ θ [ x ~ t + ( r − t ) ( u θ + J [ u o l d , t , r ] ) ] = ∂ x ~ θ , r ∂ θ . (41) \frac{\partial}{\partial\theta}X[\tilde{x}_t,u_\theta+J,r]=\frac{\partial}{\partial\theta}\left[\tilde{x}_t+(r-t)\left(u_\theta+J[u_{\mathrm{old}},t,r]\right)\right]=\frac{\partial\tilde{x}_{\theta,r}}{\partial\theta}. \tag*{(41)} ∂ θ ∂ X [ x ~ t , u θ + J , r ] = ∂ θ ∂ [ x ~ t + ( r − t ) ( u θ + J [ u old , t , r ] ) ] = ∂ θ ∂ x ~ θ , r . (41)
where x ~ θ , r \tilde{x}_{\theta,r} x ~ θ , r is the one-step transport Equation (5) used by the sampler. Since J J J is detached, it shifts the prediction without changing its Jacobian. For r < t r < t r < t , the gradient therefore corresponds to the reward augmentation,
∇ θ L M F = − 2 ( ∂ x ~ θ , r ∂ θ ) T 1 ( t − r ) 2 ( x ~ r e a l , r − x ~ i n d , r ) , (42) \nabla_\theta\mathcal{L}_{\mathrm{MF}}=-2\left(\frac{\partial\tilde{x}_{\theta,r}}{\partial\theta}\right)^{\mathsf{T}}\frac{1}{(t-r)^2}\left(\tilde{x}_{\mathrm{real},r}-\tilde{x}_{\mathrm{ind},r}\right), \tag*{(42)} ∇ θ L MF = − 2 ( ∂ θ ∂ x ~ θ , r ) T ( t − r ) 2 1 ( x ~ real , r − x ~ ind , r ) , (42)
which is the second term of Equation (28) , as
W r e g = 0 , x ~ r w d = x ~ r e a l , r , x ~ b a s e = x ~ i n d , r , W r w d = 1 ( t − r ) 2 , ω = 1. (43) W_{\mathrm{reg}}=0,\qquad\tilde{x}_{\mathrm{rwd}}=\tilde{x}_{\mathrm{real},r},\qquad\tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{ind},r},\qquad W_{\mathrm{rwd}}=\frac{1}{(t-r)^2},\qquad\omega=1. \tag*{(43)} W reg = 0 , x ~ rwd = x ~ real , r , x ~ base = x ~ ind , r , W rwd = ( t − r ) 2 1 , ω = 1. (43)
These x x x -space definitions apply for r < t r < t r < t . At r = t r = t r = t we use the velocity-space limit above; no finite value of W r w d = ( t − r ) − 2 W_{\mathrm{rwd}}=(t-r)^{-2} W rwd = ( t − r ) − 2 is assigned to this case. Here the Jacobian is ∂ x ~ θ , r / ∂ θ \partial\tilde{x}_{\theta,r}/\partial\theta ∂ x ~ θ , r / ∂ θ instead of ∂ x ~ θ / ∂ θ \partial\tilde{x}_{\theta}/\partial\theta ∂ x ~ θ / ∂ θ , because the reward augmentation acts at level r r r , which is resampled at every update.
DMD. The DMD gradient Equation (10) contains a velocity difference, and Equation (21) converts it into a difference of clean predictions. With x ~ θ \tilde{x}_\theta x ~ θ in place of the sampler output x ^ θ \hat{x}_\theta x ^ θ , the reverse KL of DMD evaluates the two fields at the noised prediction x ~ t ′ = AddNoise ( sg ( x ~ θ ) , t ) \tilde{x}'_t=\operatorname{AddNoise}(\operatorname{sg}(\tilde{x}_\theta),t) x ~ t ′ = AddNoise ( sg ( x ~ θ ) , t ) rather than at x ~ t \tilde{x}_t x ~ t ; we write x ~ r e a l = X [ x ~ t ′ , v r e a l , 0 ] \tilde{x}_{\mathrm{real}}=X[\tilde{x}'_t,v_{\mathrm{real}},0] x ~ real = X [ x ~ t ′ , v real , 0 ] and x ~ a u x = X [ x ~ t ′ , v ϕ , 0 ] \tilde{x}_{\mathrm{aux}}=X[\tilde{x}'_t,v_\phi,0] x ~ aux = X [ x ~ t ′ , v ϕ , 0 ] for the teacher’s and the auxiliary network’s predictions at that state, so that
v r e a l ( x ~ t ′ , t ) − v ϕ ( x ~ t ′ , t ) = − 1 t ( x ~ r e a l − x ~ a u x ) . (44) v_{\mathrm{real}}(\tilde{x}'_t,t)-v_\phi(\tilde{x}'_t,t)=-\frac{1}{t}\left(\tilde{x}_{\mathrm{real}}-\tilde{x}_{\mathrm{aux}}\right). \tag*{(44)} v real ( x ~ t ′ , t ) − v ϕ ( x ~ t ′ , t ) = − t 1 ( x ~ real − x ~ aux ) . (44)
The noised prediction is affine in x ~ θ \tilde{x}_\theta x ~ θ , so its Jacobian passes through to ∂ x ~ θ / ∂ θ \partial\tilde{x}_\theta/\partial\theta ∂ x ~ θ / ∂ θ ,
∇ θ L D M D = − 1 − t t 2 ( ∂ x ~ θ ∂ θ ) T ( x ~ r e a l − x ~ a u x ) . (45) \nabla_\theta\mathcal{L}_{\mathrm{DMD}}=-\frac{1-t}{t^2}\left(\frac{\partial\tilde{x}_\theta}{\partial\theta}\right)^{\mathsf{T}}\left(\tilde{x}_{\mathrm{real}}-\tilde{x}_{\mathrm{aux}}\right). \tag*{(45)} ∇ θ L DMD = − t 2 1 − t ( ∂ θ ∂ x ~ θ ) T ( x ~ real − x ~ aux ) . (45)
With this convention, Equation (45) is Equation (28) with
x ~ r w d = x ~ r e a l , x ~ b a s e = x ~ a u x , W r e g = 0 , W r w d = 1 − t 2 t 2 , ω = 1. (46) \tilde{x}_{\mathrm{rwd}}=\tilde{x}_{\mathrm{real}},\qquad\tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{aux}},\qquad W_{\mathrm{reg}}=0,\qquad W_{\mathrm{rwd}}=\frac{1-t}{2t^2},\qquad\omega=1. \tag*{(46)} x ~ rwd = x ~ real , x ~ base = x ~ aux , W reg = 0 , W rwd = 2 t 2 1 − t , ω = 1. (46)
The two distillation implementations therefore differ mainly in (i) how the instantaneous velocity of the current model distribution p θ p_\theta p θ is estimated and (ii) the state at which x ~ r e a l \tilde{x}_{\mathrm{real}} x ~ real and x ~ b a s e \tilde{x}_{\mathrm{base}} x ~ base are evaluated.
DMD parameterises the instantaneous velocity with a second network v ϕ v_{\phi} v ϕ with parameters ϕ \phi ϕ , fitted online to the student’s samples by Equation (9) , and sets x ~ b a s e = x ~ a u x \tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{aux}} x ~ base = x ~ aux at x ~ t ′ \tilde{x}'_{t} x ~ t ′ . It uses the clean-end prediction because Equation (45) uses ∂ x ~ θ / ∂ θ \partial\tilde{x}_{\theta}/\partial\theta ∂ x ~ θ / ∂ θ ; varying t t t distributes the update over noise levels but always trains the same map u θ ( ⋅ , t , 0 ) u_{\theta}(\cdot,t,0) u θ ( ⋅ , t , 0 ) to the endpoint of the trajectory.
MeanFlow induces the instantaneous velocity, computing v i n d = v o l d + J [ v o l d , t , r ] v_{\mathrm{ind}}=v_{\mathrm{old}}+J[v_{\mathrm{old}},t,r] v ind = v old + J [ v old , t , r ] from the network itself, and sets x ~ b a s e = x ~ i n d , r \tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{ind},r} x ~ base = x ~ ind , r , whose r = 0 r=0 r = 0 case is the clean end Equation (38) . Since v i n d v_{\mathrm{ind}} v ind is defined at every queried ( t , r ) (t,r) ( t , r ) , this restriction disappears; Equation (42) uses ∂ x ~ θ , r / ∂ θ \partial\tilde{x}_{\theta,r}/\partial\theta ∂ x ~ θ , r / ∂ θ , and resampling r r r at every update trains u θ ( ⋅ , t , r ) u_{\theta}(\cdot,t,r) u θ ( ⋅ , t , r ) for all r ∈ [ 0 , t ] r\in[0,t] r ∈ [ 0 , t ] .
Adaptation to Few-Step Models
The methods of § 3.2 are built on an instantaneous velocity, where the clean predictions in § 4 are x ~ θ ( i ) = X [ x ~ t ( i ) , v θ , 0 ] \tilde{x}_{\theta}^{(i)}=X[\tilde{x}_{t}^{(i)},v_{\theta},0] x ~ θ ( i ) = X [ x ~ t ( i ) , v θ , 0 ] , Equation (30) uses ∂ x ~ θ ( i ) / ∂ θ \partial\tilde{x}_{\theta}^{(i)}/\partial\theta ∂ x ~ θ ( i ) / ∂ θ , and the anchors apply the same map to v o l d v_{\mathrm{old}} v old or v r e a l v_{\mathrm{real}} v real . A few-step policy instead outputs an average velocity u θ ( x ~ t , t , r ) u_{\theta}(\tilde{x}_{t},t,r) u θ ( x ~ t , t , r ) . To evaluate Equation (30) , the velocity in Equation (21) can be replaced by a velocity recovered from u θ u_{\theta} u θ , and the anchors by the matching predictions of u o l d u_{\mathrm{old}} u old or the teacher. Therefore, methods differ in which predictions and anchors they use. A rollout sample x ^ ( i ) \hat{x}^{(i)} x ^ ( i ) is already a sample and needs no reconstruction, and the teacher prediction Equation (36) remains instantaneous, so only x ~ θ \tilde{x}_{\theta} x ~ θ and the old-policy anchors need to be rebuilt.
Building anchors with u-prediction. The direct substitution queries only the r = 0 r=0 r = 0 edge of the domain { 0 ≤ r ≤ t ≤ 1 } \{0\leq r\leq t\leq1\} { 0 ≤ r ≤ t ≤ 1 } of the flow map,
x ~ θ , u = X [ x ~ t , u θ ( x ~ t , t , 0 ) , 0 ] = x ~ t − t u θ ( x ~ t , t , 0 ) . (47) \tilde{x}_{\theta,\mathrm{u}}=X[\tilde{x}_{t},u_{\theta}(\tilde{x}_{t},t,0),0]=\tilde{x}_{t}-t\,u_{\theta}(\tilde{x}_{t},t,0). \tag*{(47)} x ~ θ , u = X [ x ~ t , u θ ( x ~ t , t , 0 ) , 0 ] = x ~ t − t u θ ( x ~ t , t , 0 ) . (47)
The policy velocity v θ ( x ~ t , t ) v_{\theta}(\tilde{x}_{t},t) v θ ( x ~ t , t ) in § 4 is replaced by u θ ( x ~ t , t , 0 ) u_{\theta}(\tilde{x}_{t},t,0) u θ ( x ~ t , t , 0 ) , and the old policy is treated in the same way,
x ~ o l d , u = X [ x ~ t , u o l d ( x ~ t , t , 0 ) , 0 ] = x ~ t − t u o l d ( x ~ t , t , 0 ) . (48) \tilde{x}_{\mathrm{old,u}}=X[\tilde{x}_{t},u_{\mathrm{old}}(\tilde{x}_{t},t,0),0]=\tilde{x}_{t}-t\,u_{\mathrm{old}}(\tilde{x}_{t},t,0). \tag*{(48)} x ~ old , u = X [ x ~ t , u old ( x ~ t , t , 0 ) , 0 ] = x ~ t − t u old ( x ~ t , t , 0 ) . (48)
The current prediction of Equation (30) is then x ~ θ ← x ~ θ , u \tilde{x}_{\theta}\leftarrow\tilde{x}_{\theta,\mathrm{u}} x ~ θ ← x ~ θ , u , and both the regression and the baseline anchor are x ~ o l d , u \tilde{x}_{\mathrm{old,u}} x ~ old , u , or x ~ b a s e = sg ( x ~ θ , u ) \tilde{x}_{\mathrm{base}}=\operatorname{sg}(\tilde{x}_{\theta,\mathrm{u}}) x ~ base = sg ( x ~ θ , u ) in the AWM case where the baseline follows the policy.
The Jacobian in Equation (28) is then ∂ x ~ θ , u / ∂ θ \partial\tilde{x}_{\theta,\mathrm{u}}/\partial\theta ∂ x ~ θ , u / ∂ θ , so each update trains only u θ ( ⋅ , t , 0 ) u_{\theta}(\cdot,t,0) u θ ( ⋅ , t , 0 ) . The sampler Equation (5) uses x ^ θ , r = x ^ θ , t − ( t − r ) u θ ( ⋅ , t , r ) \hat{x}_{\theta,r}=\hat{x}_{\theta,t}-(t-r)u_{\theta}(\cdot,t,r) x ^ θ , r = x ^ θ , t − ( t − r ) u θ ( ⋅ , t , r ) with r r r not necessarily zero. Therefore, in u-prediction, the flow map is trained only on its r = 0 r=0 r = 0 edge and leaves the entries with r ≠ 0 r\ne0 r = 0 , which the sampler uses, without a gradient.
Building anchors with v-prediction. In u-prediction, the instantaneous velocity is not used. Alternatively, Equation (30) keeps x ~ θ = X [ x ~ t , v , 0 ] \tilde{x}_{\theta}=X[\tilde{x}_{t},v,0] x ~ θ = X [ x ~ t , v , 0 ] , and the remaining question is how to construct v v v from a network that outputs an average velocity. Following Example 5, there are two constructions.
Self-induced velocity. Equation (37) recovers the instantaneous velocity from the average velocity itself, as in MeanFlowNFT, which applies DiffusionNFT, regressed on the forward (noising) process, to an average-velocity policy ([34 ] ). The current prediction adds the same detached J J J to u θ u_{\theta} u θ , which gives the v-prediction
x ~ θ , v = X [ x ~ t , u θ + J [ u o l d , t , r ] , 0 ] = x ~ t − t ( u θ + J [ u o l d , t , r ] ) , (49) \tilde{x}_{\theta,\mathrm{v}}=X[\tilde{x}_{t},u_{\theta}+J[u_{\mathrm{old}},t,r],0]=\tilde{x}_{t}-t\left(u_{\theta}+J[u_{\mathrm{old}},t,r]\right), \tag*{(49)} x ~ θ , v = X [ x ~ t , u θ + J [ u old , t , r ] , 0 ] = x ~ t − t ( u θ + J [ u old , t , r ] ) , (49)
and the old policy gives the self-induced anchor Equation (38) ,
x ~ i n d = X [ x ~ t , v i n d , 0 ] = x ~ t − t ( u o l d + J [ u o l d , t , r ] ) , (50) \tilde{x}_{\mathrm{ind}}=X[\tilde{x}_{t},v_{\mathrm{ind}},0]=\tilde{x}_{t}-t\left(u_{\mathrm{old}}+J[u_{\mathrm{old}},t,r]\right), \tag*{(50)} x ~ ind = X [ x ~ t , v ind , 0 ] = x ~ t − t ( u old + J [ u old , t , r ] ) , (50)
so that x ~ θ , v − x ~ i n d = − t ( u θ − u o l d ) \tilde{x}_{\theta,\mathrm{v}}-\tilde{x}_{\mathrm{ind}}=-t(u_{\theta}-u_{\mathrm{old}}) x ~ θ , v − x ~ ind = − t ( u θ − u old ) . Since J J J is detached, Equation (41) still holds; J J J shifts the prediction but does not change its Jacobian. The pair ( t , r ) (t,r) ( t , r ) is the input of the network and r r r is resampled at every update, so every u θ ( ⋅ , t , r ) u_{\theta}(\cdot,t,r) u θ ( ⋅ , t , r ) on the trajectory enters ∂ x ~ θ / ∂ θ \partial\tilde{x}_{\theta}/\partial\theta ∂ x ~ θ / ∂ θ . The loss itself is still evaluated at r = 0 r=0 r = 0 ; r r r enters only as a network input and through J J J , and does not change the level at which Equation (30) is evaluated.
Auxiliary velocity. The instantaneous velocity can instead be parameterised by a second network v ϕ v_{\phi} v ϕ , fitted by Equation (9) to the samples of the model, so that its population target is the marginal velocity of p θ p_{\theta} p θ ,
v ϕ ( x ~ t , t ) ≈ E x ^ ∼ p θ , ϵ [ ϵ − x ^ ∣ x ~ t ] , x ~ t = ( 1 − t ) x ^ + t ϵ . (51) v_{\phi}(\tilde{x}_t,t) \approx\mathbb{E}_{\hat{x}\sim p_{\theta},\epsilon}\left[\epsilon-\hat{x}\mid\tilde{x}_t\right], \qquad\tilde{x}_t=(1-t)\hat{x}+t\epsilon. \tag*{(51)} v ϕ ( x ~ t , t ) ≈ E x ^ ∼ p θ , ϵ [ ϵ − x ^ ∣ x ~ t ] , x ~ t = ( 1 − t ) x ^ + t ϵ . (51)
In the RL setting of § 3.2 , however, the rollouts are samples from p o l d p_{\mathrm{old}} p old and not from p θ p_{\theta} p θ , so the target is the marginal velocity of p o l d p_{\mathrm{old}} p old ,
v ϕ ( x ~ t , t ) ≈ E x ^ ∼ p o l d , ϵ [ ϵ − x ^ ∣ x ~ t ] , x ~ t = ( 1 − t ) x ^ + t ϵ . (52) v_{\phi}(\tilde{x}_t,t) \approx\mathbb{E}_{\hat{x}\sim p_{\mathrm{old}},\epsilon}\left[\epsilon-\hat{x}\mid\tilde{x}_t\right], \qquad\tilde{x}_t=(1-t)\hat{x}+t\epsilon. \tag*{(52)} v ϕ ( x ~ t , t ) ≈ E x ^ ∼ p old , ϵ [ ϵ − x ^ ∣ x ~ t ] , x ~ t = ( 1 − t ) x ^ + t ϵ . (52)
Its clean prediction x ~ a u x = X [ x ~ t , v ϕ , 0 ] \tilde{x}_{\mathrm{aux}}=X[\tilde{x}_t,v_{\phi},0] x ~ aux = X [ x ~ t , v ϕ , 0 ] is detached, like x ~ i n d \tilde{x}_{\mathrm{ind}} x ~ ind , and serves only as an anchor. The current prediction cannot be computed from v ϕ v_{\phi} v ϕ , since ∂ x ~ θ / ∂ θ \partial\tilde{x}_{\theta}/\partial\theta ∂ x ~ θ / ∂ θ would then vanish and the reward would not affect the policy. It is therefore computed from u θ u_{\theta} u θ , e.g., as x ~ θ , u \tilde{x}_{\theta,u} x ~ θ , u , x ~ θ , v \tilde{x}_{\theta,v} x ~ θ , v , or the J J J -free ( t , r ) (t,r) ( t , r ) query X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] X[\tilde{x}_t,u_{\theta}(\tilde{x}_t,t,r),0] X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] , i.e., the transport of x ~ t \tilde{x}_t x ~ t to the clean end with the average velocity queried at ( t , r ) (t,r) ( t , r ) , which has the same Jacobian as x ~ θ , v \tilde{x}_{\theta,v} x ~ θ , v . Its old-policy counterpart is x ~ i n d \tilde{x}_{\mathrm{ind}} x ~ ind with J J J dropped,
x ~ i n d ∣ J = 0 = X [ x ~ t , u o l d ( x ~ t , t , r ) , 0 ] . \tilde{x}_{\mathrm{ind}}\big|_{J=0}=X[\tilde{x}_t,u_{\mathrm{old}}(\tilde{x}_t,t,r),0]. x ~ ind J = 0 = X [ x ~ t , u old ( x ~ t , t , r ) , 0 ] .
The self-induced construction requires a Jacobian-vector product with a stop-gradient; the auxiliary construction requires a second network and optimiser that must follow a moving target.
Within the framework, a few-step method therefore does not introduce a new loss; it specifies how the four points of Equation (30) are computed and which entries u θ ( ⋅ , t , r ) u_{\theta}(\cdot,t,r) u θ ( ⋅ , t , r ) of the flow map receive a gradient through ∂ x ~ θ / ∂ θ \partial\tilde{x}_{\theta}/\partial\theta ∂ x ~ θ / ∂ θ . Table 1 summarises the three forms.
Table 1. The three few-step forms of the prediction and the anchors, with J = J [ u o l d , t , r ] J=J[u_{\mathrm{old}},t,r] J = J [ u old , t , r ] , v i n d = u o l d + J v_{\mathrm{ind}}=u_{\mathrm{old}}+J v ind = u old + J and X [ x ~ t , v c o n d , 0 ] = x ^ X[\tilde{x}_t,v_{\mathrm{cond}},0]=\hat{x} X [ x ~ t , v cond , 0 ] = x ^ . The labels v-ind and v-aux are used in § 5 .
The first column trains only the r = 0 r=0 r = 0 edge of the map; the second and the third train all of it. The third column uses the auxiliary velocity only for the baseline and computes x ~ θ \tilde{x}_{\theta} x ~ θ and x ~ r e g \tilde{x}_{\mathrm{reg}} x ~ reg at the same ( t , r ) (t,r) ( t , r ) without J J J ; x ~ θ \tilde{x}_{\theta} x ~ θ has the same Jacobian as x ~ θ , v \tilde{x}_{\theta,v} x ~ θ , v .
Example 6 (DiffusionNFT with u-prediction). With u-prediction, the weights and the rewarded sample of Equation (35) are unchanged, W r e g = W r w d = β 2 / t 2 W_{\mathrm{reg}}=W_{\mathrm{rwd}}=\beta^2/t^2 W reg = W rwd = β 2 / t 2 and x ~ r w d = x ^ \tilde{x}_{\mathrm{rwd}}=\hat{x} x ~ rwd = x ^ , and both anchors become the r = 0 r=0 r = 0 prediction of the old policy,
x ~ θ = x ~ θ , u = X [ x ~ t , u θ , 0 ] , x ~ r e g = x ~ b a s e = x ~ o l d , u = X [ x ~ t , u o l d , 0 ] . (53) \tilde{x}_{\theta}=\tilde{x}_{\theta,u}=X[\tilde{x}_t,u_{\theta},0], \qquad\tilde{x}_{\mathrm{reg}}=\tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{old},u}=X[\tilde{x}_t,u_{\mathrm{old}},0]. \tag*{(53)} x ~ θ = x ~ θ , u = X [ x ~ t , u θ , 0 ] , x ~ reg = x ~ base = x ~ old , u = X [ x ~ t , u old , 0 ] . (53)
Starting from the two squares of Equation (30) ,
L r e g = W r e g ∥ X [ x ~ t , u θ , 0 ] − X [ x ~ t , u o l d , 0 ] ∥ 2 , (54) \mathcal{L}_{\mathrm{reg}} = W_{\mathrm{reg}} \left\lVert X[\tilde{x}_t, u_\theta, 0] - X[\tilde{x}_t, u_{\mathrm{old}}, 0] \right\rVert^2, \tag*{(54)} L reg = W reg ∥ X [ x ~ t , u θ , 0 ] − X [ x ~ t , u old , 0 ] ∥ 2 , (54)
L r w d = W r w d ∥ X [ x ~ t , u θ , 0 ] − sg ( X [ x ~ t , u θ , 0 ] + g u ) ∥ 2 , (55) \mathcal{L}_{\mathrm{rwd}} = W_{\mathrm{rwd}} \left\lVert X[\tilde{x}_t, u_\theta, 0] - \operatorname{sg}\left(X[\tilde{x}_t, u_\theta, 0] + g_u\right) \right\rVert^2, \tag*{(55)} L rwd = W rwd ∥ X [ x ~ t , u θ , 0 ] − sg ( X [ x ~ t , u θ , 0 ] + g u ) ∥ 2 , (55)
where x ~ θ = X [ x ~ t , u θ , 0 ] \tilde{x}_\theta= X[\tilde{x}_t, u_\theta, 0] x ~ θ = X [ x ~ t , u θ , 0 ] and g u = ω ( x ^ − X [ x ~ t , u o l d , 0 ] ) g_u = \omega\left(\hat{x} - X[\tilde{x}_t, u_{\mathrm{old}}, 0]\right) g u = ω ( x ^ − X [ x ~ t , u old , 0 ] ) . The copy of x ~ θ \tilde{x}_\theta x ~ θ inside sg \operatorname{sg} sg does not cancel with the copy outside. No J J J appears, the two anchors coincide, and ∂ x ~ θ / ∂ θ \partial\tilde{x}_\theta/\partial\theta ∂ x ~ θ / ∂ θ involves only r = 0 r=0 r = 0 . This is DiffusionNFT evaluated on the r = 0 r=0 r = 0 edge ( t , 0 ) (t,0) ( t , 0 ) .
Example 7 (DiffusionNFT with v-prediction). The decomposition Equation (35) carries over, and the self-induced construction below keeps x ~ r e g = x ~ b a s e \tilde{x}_{\mathrm{reg}}=\tilde{x}_{\mathrm{base}} x ~ reg = x ~ base . The remaining choice is how to recover the instantaneous velocity used in Equation (21) from an average-velocity network, and Equation (49) –(52) give two ways to do so. The first is to use the self-induced velocity v i n d = u o l d + J [ u o l d , t , r ] v_{\mathrm{ind}}=u_{\mathrm{old}}+J[u_{\mathrm{old}},t,r] v ind = u old + J [ u old , t , r ] , which adds the Jacobian-vector term J [ u o l d , t , r ] J[u_{\mathrm{old}},t,r] J [ u old , t , r ] to every prediction of the policy and of the old-policy anchor, as in Equation (49) –(50) ; this construction gives the MeanFlowNFT -style variant ([34 ] ). The second is to fit an auxiliary network v ϕ v_\phi v ϕ to the marginal velocity of the rollouts, Equation (51) –(52) , and compute the baseline as x ~ a u x \tilde{x}_{\mathrm{aux}} x ~ aux , which leads to DMDNFT . In both cases the current prediction x ~ θ \tilde{x}_\theta x ~ θ is computed from u θ u_\theta u θ , since otherwise ∂ x ~ θ / ∂ θ \partial\tilde{x}_\theta/\partial\theta ∂ x ~ θ / ∂ θ would vanish.
MeanFlowNFT queries the network at ( t , r ) (t,r) ( t , r ) with 0 ≤ r ≤ t 0 \le r \le t 0 ≤ r ≤ t , so every prediction of the policy is a v-prediction. Following prior directional-difference flow-map work ([26 ] (Nie et al. 2026)), the MeanFlowNFT construction evaluates one derivative on the frozen old policy and shares it between the current and the old policy ([34 ] ), which gives the four points
x ~ θ = x ~ θ , v = X [ x ~ t , u θ + J [ u o l d , t , r ] , 0 ] , x ~ r e g = x ~ b a s e = x ~ i n d = X [ x ~ t , u o l d + J [ u o l d , t , r ] , 0 ] x ~ r w d = x ^ . (56) \begin{aligned}
\tilde{x}_\theta= \tilde{x}_{\theta,v} &= X[\tilde{x}_t, u_\theta+ J[u_{\mathrm{old}},t,r], 0],\\
\tilde{x}_{\mathrm{reg}} = \tilde{x}_{\mathrm{base}} = \tilde{x}_{\mathrm{ind}} &= X[\tilde{x}_t, u_{\mathrm{old}} + J[u_{\mathrm{old}},t,r], 0]\\
\tilde{x}_{\mathrm{rwd}} &= \hat{x}. \tag*{(56)}
\end{aligned} x ~ θ = x ~ θ , v x ~ reg = x ~ base = x ~ ind x ~ rwd = X [ x ~ t , u θ + J [ u old , t , r ] , 0 ] , = X [ x ~ t , u old + J [ u old , t , r ] , 0 ] = x ^ . (56) Starting from the two squares of Equation (30) ,
L r e g = W r e g ∥ X [ x ~ t , u θ + J [ u o l d , t , r ] , 0 ] − X [ x ~ t , u o l d + J [ u o l d , t , r ] , 0 ] ∥ 2 = W r e g ∥ X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] − X [ x ~ t , u o l d ( x ~ t , t , r ) , 0 ] ∥ 2 , (57) \begin{aligned}
\mathcal{L}_{\mathrm{reg}} &= W_{\mathrm{reg}} \left\lVert X[\tilde{x}_t,u_\theta+J[u_{\mathrm{old}},t,r],0]-X[\tilde{x}_t,u_{\mathrm{old}}+J[u_{\mathrm{old}},t,r],0]\right\rVert^2\\
&= W_{\mathrm{reg}} \left\lVert X[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r),0]-X[\tilde{x}_t,u_{\mathrm{old}}(\tilde{x}_t,t,r),0]\right\rVert^2, \tag*{(57)}
\end{aligned} L reg = W reg ∥ X [ x ~ t , u θ + J [ u old , t , r ] , 0 ] − X [ x ~ t , u old + J [ u old , t , r ] , 0 ] ∥ 2 = W reg ∥ X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] − X [ x ~ t , u old ( x ~ t , t , r ) , 0 ] ∥ 2 , (57) L r w d = W r w d ∥ X [ x ~ t , u θ + J [ u o l d , t , r ] , 0 ] − sg ( X [ x ~ t , u θ + J [ u o l d , t , r ] , 0 ] + g v ) ∥ 2 = W r w d ∥ X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] − sg ( X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] + g v ) ∥ 2 . (58) \begin{aligned}
\mathcal{L}_{\mathrm{rwd}} &= W_{\mathrm{rwd}} \left\lVert X[\tilde{x}_t,u_\theta+J[u_{\mathrm{old}},t,r],0]-\operatorname{sg}\left(X[\tilde{x}_t,u_\theta+J[u_{\mathrm{old}},t,r],0]+g_v\right)\right\rVert^2\\
&= W_{\mathrm{rwd}} \left\lVert X[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r),0]-\operatorname{sg}\left(X[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r),0]+g_v\right)\right\rVert^2. \tag*{(58)}
\end{aligned} L rwd = W rwd ∥ X [ x ~ t , u θ + J [ u old , t , r ] , 0 ] − sg ( X [ x ~ t , u θ + J [ u old , t , r ] , 0 ] + g v ) ∥ 2 = W rwd ∥ X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] − sg ( X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] + g v ) ∥ 2 . (58) The prediction x ~ θ \tilde{x}_\theta x ~ θ is still the v-prediction of Equation (56) . In L r e g \mathcal{L}_{\mathrm{reg}} L reg , the shared detached J J J cancels between the current and old-policy predictions; in L r w d \mathcal{L}_{\mathrm{rwd}} L rwd , it cancels between the two copies of the current prediction. The reward displacement still contains the old-policy prediction,
g v = ω ( x ^ − X [ x ~ t , u o l d + J [ u o l d , t , r ] , 0 ] ) , g_v = \omega\left(\hat{x}-X[\tilde{x}_t,u_{\mathrm{old}}+J[u_{\mathrm{old}},t,r],0]\right), g v = ω ( x ^ − X [ x ~ t , u old + J [ u old , t , r ] , 0 ] ) , and therefore retains J J J . Since J J J is detached, the Jacobian is that of X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] X[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r),0] X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] . Resampling r r r then updates the family u θ ( ⋅ , t , r ) u_\theta(\cdot,t,r) u θ ( ⋅ , t , r ) instead of only its r = 0 r=0 r = 0 edge.
DMDNFT is the alternative construction. In the losses, only the baseline changes, with v ϕ v_\phi v ϕ fitted online to the old-policy rollouts by Equation (52) . The current prediction is still computed from u θ u_\theta u θ , since substituting v ϕ v_\phi v ϕ would give ∂ x ~ θ , v / ∂ θ = 0 \partial\tilde{x}_{\theta,v}/\partial\theta=0 ∂ x ~ θ , v / ∂ θ = 0 . As in Equation (41) , J J J is detached, so
∂ x ~ θ , v ∂ θ = ∂ ∂ θ X [ x ~ t , u θ + J [ u o l d , t , r ] , 0 ] = ∂ ∂ θ X [ x ~ t , u θ , 0 ] . (59) \frac{\partial\tilde{x}_{\theta,v}}{\partial\theta}
= \frac{\partial}{\partial\theta}X[\tilde{x}_t,u_\theta+J[u_{\mathrm{old}},t,r],0]
= \frac{\partial}{\partial\theta}X[\tilde{x}_t,u_\theta,0]. \tag*{(59)} ∂ θ ∂ x ~ θ , v = ∂ θ ∂ X [ x ~ t , u θ + J [ u old , t , r ] , 0 ] = ∂ θ ∂ X [ x ~ t , u θ , 0 ] . (59)
Therefore, the four points are
x ~ θ = X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] , x ~ r e g = x ~ i n d ∣ J = 0 = X [ x ~ t , u o l d ( x ~ t , t , r ) , 0 ] , x ~ b a s e = x ~ a u x , x ~ r w d = x ^ . (60) \begin{aligned}
\tilde{x}_{\theta} &= X\left[\tilde{x}_{t}, u_{\theta}(\tilde{x}_{t},t,r),0\right],\\
\tilde{x}_{\mathrm{reg}} &= \left.\tilde{x}_{\mathrm{ind}}\right|_{J=0}=X\left[\tilde{x}_{t},u_{\mathrm{old}}(\tilde{x}_{t},t,r),0\right],\\
\tilde{x}_{\mathrm{base}} &= \tilde{x}_{\mathrm{aux}},\\
\tilde{x}_{\mathrm{rwd}} &= \hat{x}. \tag*{(60)}
\end{aligned} x ~ θ x ~ reg x ~ base x ~ rwd = X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] , = x ~ ind ∣ J = 0 = X [ x ~ t , u old ( x ~ t , t , r ) , 0 ] , = x ~ aux , = x ^ . (60)
Example 8 (Few-step DiffusionOPSD). DiffusionOPSD keeps the training scheme of DiffusionNFT but constructs the point toward which the policy should move, which requires a reward R ( D ( ⋅ ) , c ) R(D(\cdot),c) R ( D ( ⋅ ) , c ) that is differentiable with respect to the clean sample. Three clean points are constructed at one query state x ~ t ( i ) = AddNoise ( x ^ ( i ) , t ) \tilde{x}_{t}^{(i)}=\operatorname{AddNoise}(\hat{x}^{(i)},t) x ~ t ( i ) = AddNoise ( x ^ ( i ) , t ) at a small noise level t t t (drawn from a law concentrated below 0.5 0.5 0.5 , § 6.1 ). The first is the old policy’s prediction at this state, given by Equation (24) ,
x ~ o l d ( i ) = X [ x ~ t ( i ) , v o l d , 0 ] . (61) \tilde{x}_{\mathrm{old}}^{(i)}=X\left[\tilde{x}_{t}^{(i)},v_{\mathrm{old}},0\right]. \tag*{(61)} x ~ old ( i ) = X [ x ~ t ( i ) , v old , 0 ] . (61) The other two points are obtained by M t g t M_{\mathrm{tgt}} M tgt normalised reward-gradient steps in x x x -space, in the ascent and in the descent direction. Suppressing ( i ) (i) ( i ) on the iterates and indexing them by [ m ] [m] [ m ] , x ~ ± [ 0 ] = x ~ o l d ( i ) \tilde{x}_{\pm}^{[0]}=\tilde{x}_{\mathrm{old}}^{(i)} x ~ ± [ 0 ] = x ~ old ( i ) and
x ~ ± [ m + 1 ] = x ~ ± [ m ] ± h ∇ R ( D ( x ~ ± [ m ] ) , c ) ∥ ∇ R ( D ( x ~ ± [ m ] ) , c ) ∥ + ε g , h = η κ ∥ x ~ o l d ( i ) ∥ M t g t , (62) \tilde{x}_{\pm}^{[m+1]}=\tilde{x}_{\pm}^{[m]}\pm h\frac{\nabla R(D(\tilde{x}_{\pm}^{[m]}),c)}{\left\|\nabla R(D(\tilde{x}_{\pm}^{[m]}),c)\right\|+\varepsilon_g},\qquad h=\frac{\eta\kappa\left\|\tilde{x}_{\mathrm{old}}^{(i)}\right\|}{M_{\mathrm{tgt}}}, \tag*{(62)} x ~ ± [ m + 1 ] = x ~ ± [ m ] ± h ∇ R ( D ( x ~ ± [ m ] ) , c ) + ε g ∇ R ( D ( x ~ ± [ m ] ) , c ) , h = M tgt η κ x ~ old ( i ) , (62) for m = 0 , … , M t g t − 1 m=0,\ldots,M_{\mathrm{tgt}}-1 m = 0 , … , M tgt − 1 , with a small floor ε g > 0 \varepsilon_g>0 ε g > 0 , the relative trust radius κ > 0 \kappa>0 κ > 0 and the total step length η > 0 \eta>0 η > 0 in units of that radius. Each iterate is projected onto the ball of radius κ ∥ x ~ o l d ( i ) ∥ \kappa\left\|\tilde{x}_{\mathrm{old}}^{(i)}\right\| κ x ~ old ( i ) around the anchor; before projection, M t g t h M_{\mathrm{tgt}}h M tgt h is η \eta η times that radius. The final points x ~ ± ( i ) = sg ( x ~ ± [ M t g t ] ) \tilde{x}_{\pm}^{(i)}=\operatorname{sg}(\tilde{x}_{\pm}^{[M_{\mathrm{tgt}]})} x ~ ± ( i ) = sg ( x ~ ± [ M tgt ] ) are detached. For η ≤ 1 \eta\leq1 η ≤ 1 the projection is inactive, and their displacements are
Δ r w d x ~ ± ( i ) : = x ~ ± ( i ) − x ~ o l d ( i ) = ± h ∑ m = 0 M t g t − 1 ∇ R ( D ( x ~ ± [ m ] ) , c ) ∥ ∇ R ( D ( x ~ ± [ m ] ) , c ) ∥ + ε g . (63) \Delta_{\mathrm{rwd}}\tilde{x}_{\pm}^{(i)}:=\tilde{x}_{\pm}^{(i)}-\tilde{x}_{\mathrm{old}}^{(i)}=\pm h\sum_{m=0}^{M_{\mathrm{tgt}}-1}\frac{\nabla R(D(\tilde{x}_{\pm}^{[m]}),c)}{\left\|\nabla R(D(\tilde{x}_{\pm}^{[m]}),c)\right\|+\varepsilon_g}. \tag*{(63)} Δ rwd x ~ ± ( i ) := x ~ ± ( i ) − x ~ old ( i ) = ± h m = 0 ∑ M tgt − 1 ∇ R ( D ( x ~ ± [ m ] ) , c ) + ε g ∇ R ( D ( x ~ ± [ m ] ) , c ) . (63) The labels ± \pm ± indicate the directions of construction. We construct the loss function in three steps:
(i) Fitting reuses the two x x x -space branches of Equation (26) , now regressed onto the constructed points instead of the rollout sample:
x ~ θ ± = x ~ o l d ( i ) ± β ( x ~ θ ( i ) − x ~ o l d ( i ) ) , \tilde{x}_{\theta}^{\pm}=\tilde{x}_{\mathrm{old}}^{(i)}\pm\beta\left(\tilde{x}_{\theta}^{(i)}-\tilde{x}_{\mathrm{old}}^{(i)}\right), x ~ θ ± = x ~ old ( i ) ± β ( x ~ θ ( i ) − x ~ old ( i ) ) , L O P S D = E i [ L O P S D ( i ) ] , L O P S D ( i ) = a + ∥ x ~ θ ( i ) − x ~ + ( i ) ∥ 2 + a − ∥ x ~ θ ( i ) − x ~ − ( i ) ∥ 2 . (64) \mathcal{L}_{\mathrm{OPSD}}=\mathbb{E}_{i}\left[\mathcal{L}_{\mathrm{OPSD}}^{(i)}\right],\qquad\mathcal{L}_{\mathrm{OPSD}}^{(i)}=a_{+}\left\|\tilde{x}_{\theta}^{(i)}-\tilde{x}_{+}^{(i)}\right\|^{2}+a_{-}\left\|\tilde{x}_{\theta}^{(i)}-\tilde{x}_{-}^{(i)}\right\|^{2}. \tag*{(64)} L OPSD = E i [ L OPSD ( i ) ] , L OPSD ( i ) = a + x ~ θ ( i ) − x ~ + ( i ) 2 + a − x ~ θ ( i ) − x ~ − ( i ) 2 . (64) with a + = r ^ ( i ) / γ + a_{+}=\hat{r}^{(i)}/\gamma_{+} a + = r ^ ( i ) / γ + and a − = ( 1 − r ^ ( i ) ) / γ − a_{-}=(1-\hat{r}^{(i)})/\gamma_{-} a − = ( 1 − r ^ ( i ) ) / γ − non-negative and detached, r ^ ( i ) \hat{r}^{(i)} r ^ ( i ) the weight of Equation (18) and γ ± = max { sg mean s p [ ∥ x ~ θ ± − x ~ ± ( i ) ∥ ] , ε γ } \gamma_{\pm}=\max\{\operatorname{sg}\operatorname{mean}_{\mathrm{sp}}[\left\|\tilde{x}_{\theta}^{\pm}-\tilde{x}_{\pm}^{(i)}\right\|],\varepsilon_{\gamma}\} γ ± = max { sg mean sp [ x ~ θ ± − x ~ ± ( i ) ] , ε γ } the adaptive normalisers with floor ε γ > 0 \varepsilon_{\gamma}>0 ε γ > 0 , and where mean s p [ ⋅ ] \operatorname{mean}_{\mathrm{sp}}[\cdot] mean sp [ ⋅ ] denotes the per-sample mean, i.e., the average over the N N N elements of one sample.
(ii) To decouple the loss into Equation (30) with gradient form Equation (28) , note that the two branches are reflections of each other about the anchor, so both residuals are affine in the same displacement x ~ θ ( i ) − x ~ o l d ( i ) \tilde{x}_{\theta}^{(i)}-\tilde{x}_{\mathrm{old}}^{(i)} x ~ θ ( i ) − x ~ old ( i ) . Expanding the squares of L O P S D ( i ) \mathcal{L}_{\mathrm{OPSD}}^{(i)} L OPSD ( i ) and discarding the θ \theta θ -free remainder therefore
splits it into one square and two inner products, each written directly on the three constructed points:
L O P S D ( i ) = β 2 ( a + + a − ) ∥ x ~ θ ( i ) − x ~ o l d ( i ) ∥ 2 ⏟ self-regression − 2 β a + ( x ~ θ ( i ) − x ~ o l d ( i ) ) T Δ r w d x ~ + ( i ) ⏟ reward augmentation, positive + 2 β a − ( x ~ θ ( i ) − x ~ o l d ( i ) ) T Δ r w d x ~ − ( i ) ⏟ reward augmentation, negative + c o n s t , (65) \mathcal{L}^{(i)}_{\mathrm{OPSD}} = \beta^{2}(a_{+}+a_{-})\underbrace{\left\lVert\tilde{x}^{(i)}_{\theta}-\tilde{x}^{(i)}_{\mathrm{old}}\right\rVert^{2}}_{\text{self-regression}} - 2\beta a_{+}\underbrace{\left(\tilde{x}^{(i)}_{\theta}-\tilde{x}^{(i)}_{\mathrm{old}}\right)^{\mathsf{T}}\Delta_{\mathrm{rwd}}\tilde{x}^{(i)}_{+}}_{\text{reward augmentation, positive}} + 2\beta a_{-}\underbrace{\left(\tilde{x}^{(i)}_{\theta}-\tilde{x}^{(i)}_{\mathrm{old}}\right)^{\mathsf{T}}\Delta_{\mathrm{rwd}}\tilde{x}^{(i)}_{-}}_{\text{reward augmentation, negative}} + \mathrm{const}, \tag*{(65)} L OPSD ( i ) = β 2 ( a + + a − ) self-regression x ~ θ ( i ) − x ~ old ( i ) 2 − 2 β a + reward augmentation, positive ( x ~ θ ( i ) − x ~ old ( i ) ) T Δ rwd x ~ + ( i ) + 2 β a − reward augmentation, negative ( x ~ θ ( i ) − x ~ old ( i ) ) T Δ rwd x ~ − ( i ) + const , (65)
the constant being a + ∥ Δ r w d x ~ + ( i ) ∥ 2 + a − ∥ Δ r w d x ~ − ( i ) ∥ 2 a_{+}\left\lVert\Delta_{\mathrm{rwd}}\tilde{x}^{(i)}_{+}\right\rVert^{2}+a_{-}\left\lVert\Delta_{\mathrm{rwd}}\tilde{x}^{(i)}_{-}\right\rVert^{2} a + Δ rwd x ~ + ( i ) 2 + a − Δ rwd x ~ − ( i ) 2 , which has no gradient. Differentiating gives three terms with the same Jacobian,
∇ θ L O P S D ( i ) = 2 ( ∂ x ~ θ ( i ) ∂ θ ) T [ β 2 ( a + + a − ) ( x ~ θ ( i ) − x ~ o l d ( i ) ) − β a + Δ r w d x ~ + ( i ) + β a − Δ r w d x ~ − ( i ) ] . (66) \nabla_{\theta}\mathcal{L}^{(i)}_{\mathrm{OPSD}} = 2\left(\frac{\partial\tilde{x}^{(i)}_{\theta}}{\partial\theta}\right)^{\mathsf{T}}\left[\beta^{2}(a_{+}+a_{-})\left(\tilde{x}^{(i)}_{\theta}-\tilde{x}^{(i)}_{\mathrm{old}}\right)-\beta a_{+}\Delta_{\mathrm{rwd}}\tilde{x}^{(i)}_{+}+\beta a_{-}\Delta_{\mathrm{rwd}}\tilde{x}^{(i)}_{-}\right]. \tag*{(66)} ∇ θ L OPSD ( i ) = 2 ( ∂ θ ∂ x ~ θ ( i ) ) T [ β 2 ( a + + a − ) ( x ~ θ ( i ) − x ~ old ( i ) ) − β a + Δ rwd x ~ + ( i ) + β a − Δ rwd x ~ − ( i ) ] . (66)
Equations (65) and (66) contain one self-regression term and two reward-augmentation terms, indexed by k = 1 , 2 k=1,2 k = 1 , 2 :
x ~ r e g ( i ) = x ~ o l d ( i ) , W r e g = W r w d = β 2 ( a + + a − ) , \tilde{x}^{(i)}_{\mathrm{reg}}=\tilde{x}^{(i)}_{\mathrm{old}},\qquad W_{\mathrm{reg}}=W_{\mathrm{rwd}}=\beta^{2}(a_{+}+a_{-}), x ~ reg ( i ) = x ~ old ( i ) , W reg = W rwd = β 2 ( a + + a − ) ,
x ~ r w d , 1 ( i ) = x ~ + ( i ) , x ~ b a s e , 1 ( i ) = x ~ o l d ( i ) , ω 1 = a + β ( a + + a − ) , \tilde{x}^{(i)}_{\mathrm{rwd},1}=\tilde{x}^{(i)}_{+},\qquad\tilde{x}^{(i)}_{\mathrm{base},1}=\tilde{x}^{(i)}_{\mathrm{old}},\qquad\omega_{1}=\frac{a_{+}}{\beta(a_{+}+a_{-})}, x ~ rwd , 1 ( i ) = x ~ + ( i ) , x ~ base , 1 ( i ) = x ~ old ( i ) , ω 1 = β ( a + + a − ) a + ,
x ~ r w d , 2 ( i ) = x ~ − ( i ) , x ~ b a s e , 2 ( i ) = x ~ o l d ( i ) , ω 2 = − a − β ( a + + a − ) . (67) \tilde{x}^{(i)}_{\mathrm{rwd},2}=\tilde{x}^{(i)}_{-},\qquad\tilde{x}^{(i)}_{\mathrm{base},2}=\tilde{x}^{(i)}_{\mathrm{old}},\qquad\omega_{2}=-\frac{a_{-}}{\beta(a_{+}+a_{-})}. \tag*{(67)} x ~ rwd , 2 ( i ) = x ~ − ( i ) , x ~ base , 2 ( i ) = x ~ old ( i ) , ω 2 = − β ( a + + a − ) a − . (67)
The self-regression is toward the old policy’s prediction, as in Equation (35) . The two reward terms share this baseline, x ~ b a s e , 1 ( i ) = x ~ b a s e , 2 ( i ) = x ~ r e g ( i ) \tilde{x}^{(i)}_{\mathrm{base},1}=\tilde{x}^{(i)}_{\mathrm{base},2}=\tilde{x}^{(i)}_{\mathrm{reg}} x ~ base , 1 ( i ) = x ~ base , 2 ( i ) = x ~ reg ( i ) , and differ only in the constructed rewarded point x ~ r w d , k ( i ) \tilde{x}^{(i)}_{\mathrm{rwd},k} x ~ rwd , k ( i ) and its signed reward coefficient. The signs of ω 1 \omega_{1} ω 1 and ω 2 \omega_{2} ω 2 are not determined by the endpoint advantage A ( i ) A^{(i)} A ( i ) ; no advantage is defined for the constructed points. They are fixed to be non-negative for the ascent branch and non-positive for the descent branch.
Applying Equation (29) to each reward term separately gives a loss with gradient Equation (66) , in the form Equation (30) with two reward squares,
L O P S D ( i ) = L r e g ( i ) + L r w d , 1 ( i ) + L r w d , 2 ( i ) , \mathcal{L}^{(i)}_{\mathrm{OPSD}}=\mathcal{L}^{(i)}_{\mathrm{reg}}+\mathcal{L}^{(i)}_{\mathrm{rwd},1}+\mathcal{L}^{(i)}_{\mathrm{rwd},2}, L OPSD ( i ) = L reg ( i ) + L rwd , 1 ( i ) + L rwd , 2 ( i ) ,
L r e g ( i ) = W r e g ∥ x ~ θ ( i ) − x ~ r e g ( i ) ∥ 2 , \mathcal{L}^{(i)}_{\mathrm{reg}}=W_{\mathrm{reg}}\left\lVert\tilde{x}^{(i)}_{\theta}-\tilde{x}^{(i)}_{\mathrm{reg}}\right\rVert^{2}, L reg ( i ) = W reg x ~ θ ( i ) − x ~ reg ( i ) 2 ,
L r w d , k ( i ) = W r w d ∥ x ~ θ ( i ) − sg ( x ~ θ ( i ) + g k ( i ) ) ∥ 2 , k = 1 , 2 , (68) \mathcal{L}^{(i)}_{\mathrm{rwd},k}=W_{\mathrm{rwd}}\left\lVert\tilde{x}^{(i)}_{\theta}-\operatorname{sg}\left(\tilde{x}^{(i)}_{\theta}+g^{(i)}_{k}\right)\right\rVert^{2},\qquad k=1,2, \tag*{(68)} L rwd , k ( i ) = W rwd x ~ θ ( i ) − sg ( x ~ θ ( i ) + g k ( i ) ) 2 , k = 1 , 2 , (68)
where g k ( i ) = ω k ( x ~ r w d , k ( i ) − x ~ b a s e , k ( i ) ) g^{(i)}_{k}=\omega_{k}\left(\tilde{x}^{(i)}_{\mathrm{rwd},k}-\tilde{x}^{(i)}_{\mathrm{base},k}\right) g k ( i ) = ω k ( x ~ rwd , k ( i ) − x ~ base , k ( i ) ) , i.e. g 1 ( i ) = ω 1 Δ r w d x ~ + ( i ) g^{(i)}_{1}=\omega_{1}\Delta_{\mathrm{rwd}}\tilde{x}^{(i)}_{+} g 1 ( i ) = ω 1 Δ rwd x ~ + ( i ) and g 2 ( i ) = ω 2 Δ r w d x ~ − ( i ) g^{(i)}_{2}=\omega_{2}\Delta_{\mathrm{rwd}}\tilde{x}^{(i)}_{-} g 2 ( i ) = ω 2 Δ rwd x ~ − ( i ) . The branch magnitudes satisfy ∣ ω 1 ∣ + ∣ ω 2 ∣ = 1 / β \lvert\omega_{1}\rvert+\lvert\omega_{2}\rvert=1/\beta ∣ ω 1 ∣ + ∣ ω 2 ∣ = 1/ β , so r ^ ( i ) \hat{r}^{(i)} r ^ ( i ) redistributes that total between the two branches. It can also change the common weight W = β 2 ( a + + a − ) W=\beta^{2}(a_{+}+a_{-}) W = β 2 ( a + + a − ) whenever γ + ≠ γ − \gamma_{+}\ne\gamma_{-} γ + = γ − . At r ^ ( i ) → 1 \hat{r}^{(i)}\to1 r ^ ( i ) → 1 only the ascent displacement remains, at r ^ ( i ) → 0 \hat{r}^{(i)}\to0 r ^ ( i ) → 0 only the descent displacement remains, and the self-regression and two reward-augmentation contributions sum to zero at x ~ θ ( i ) = x ~ r e g ( i ) + g 1 ( i ) + g 2 ( i ) \tilde{x}^{(i)}_{\theta}=\tilde{x}^{(i)}_{\mathrm{reg}}+g^{(i)}_{1}+g^{(i)}_{2} x ~ θ ( i ) = x ~ reg ( i ) + g 1 ( i ) + g 2 ( i ) .
( i i i ) (iii) ( iii ) For an average-velocity policy we take the v-prediction form of Equations (49) –(50) , as in Example 7. Suppressing ( i ) (i) ( i ) and writing J = J [ u o l d , t , r ] J=J[u_{\mathrm{old}},t,r] J = J [ u old , t , r ] , the corresponding quantities are
x ~ θ = X [ x ~ t , u θ + J , 0 ] , \tilde{x}_{\theta}=X\left[\tilde{x}_{t},u_{\theta}+J,0\right], x ~ θ = X [ x ~ t , u θ + J , 0 ] ,
x ~ r e g = x ~ b a s e , 1 = x ~ b a s e , 2 = x ~ i n d = X [ x ~ t , u o l d + J , 0 ] , \tilde{x}_{\mathrm{reg}}=\tilde{x}_{\mathrm{base},1}=\tilde{x}_{\mathrm{base},2}=\tilde{x}_{\mathrm{ind}}=X\left[\tilde{x}_{t},u_{\mathrm{old}}+J,0\right], x ~ reg = x ~ base , 1 = x ~ base , 2 = x ~ ind = X [ x ~ t , u old + J , 0 ] ,
x ~ r w d , 1 = x ~ + , x ~ r w d , 2 = x ~ − , (69) \tilde{x}_{\mathrm{rwd},1}=\tilde{x}_{+},\qquad\tilde{x}_{\mathrm{rwd},2}=\tilde{x}_{-}, \tag*{(69)} x ~ rwd , 1 = x ~ + , x ~ rwd , 2 = x ~ − , (69)
where the gradient steps Equation (62) are not started at x ~ i n d \tilde{x}_{\mathrm{ind}} x ~ ind , which contains J J J , but at the r = 0 r=0 r = 0 prediction x ~ o l d , u \tilde{x}_{\mathrm{old},u} x ~ old , u of Equation (48) , and only the resulting displacement is transferred to the anchor:
Δ = x ~ o l d , u − x ~ i n d , x ~ ± = [ apply xref‘Equation (62)‘ from x ~ ± [ 0 ] = x ~ o l d , u ] − Δ . (70) \Delta=\tilde{x}_{\mathrm{old},u}-\tilde{x}_{\mathrm{ind}},\qquad\tilde{x}_{\pm}=\left[\text{apply {xref}`Equation (62)` from }\tilde{x}_{\pm}^{[0]}=\tilde{x}_{\mathrm{old},u}\right]-\Delta. \tag*{(70)} Δ = x ~ old , u − x ~ ind , x ~ ± = [ apply xref ‘Equation (62)‘ from x ~ ± [ 0 ] = x ~ old , u ] − Δ. (70)
The weights are unchanged. We expand the three squares of Equation (68) in turn.
The self-regression is the self-regression square of Equation (57) :
L r e g = W r e g ∥ X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] − X [ x ~ t , u o l d ( x ~ t , t , r ) , 0 ] ∥ 2 . (71) \mathcal{L}_{\mathrm{reg}}=W_{\mathrm{reg}}\left\|X[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r),0]-X[\tilde{x}_t,u_{\mathrm{old}}(\tilde{x}_t,t,r),0]\right\|^2. \tag*{(71)} L reg = W reg ∥ X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] − X [ x ~ t , u old ( x ~ t , t , r ) , 0 ] ∥ 2 . (71)
Both copies of J J J are the same detached estimate computed from u o l d u_{\mathrm{old}} u old , so they cancel.
The two reward squares are
L r w d , k = W r w d ∥ X [ x ~ t , u θ + J , 0 ] − sg ( X [ x ~ t , u θ + J , 0 ] + g k ) ∥ 2 = W r w d ∥ X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] − sg ( X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] + g k ) ∥ 2 , k = 1 , 2 , (72) \begin{aligned}
\mathcal{L}_{\mathrm{rwd},k}&=W_{\mathrm{rwd}}\left\|X[\tilde{x}_t,u_\theta+J,0]-\operatorname{sg}\left(X[\tilde{x}_t,u_\theta+J,0]+g_k\right)\right\|^2\\
&=W_{\mathrm{rwd}}\left\|X[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r),0]-\operatorname{sg}\left(X[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r),0]+g_k\right)\right\|^2,\qquad k=1,2, \tag*{(72)}
\end{aligned} L rwd , k = W rwd ∥ X [ x ~ t , u θ + J , 0 ] − sg ( X [ x ~ t , u θ + J , 0 ] + g k ) ∥ 2 = W rwd ∥ X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] − sg ( X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] + g k ) ∥ 2 , k = 1 , 2 , (72)
where, by Equation (69) ,
g k = ω k ( x ~ r w d , k − x ~ b a s e , k ) , i.e. g 1 = ω 1 ( x ~ + − x ~ i n d ) , g 2 = ω 2 ( x ~ − − x ~ i n d ) . (73) g_k=\omega_k(\tilde{x}_{\mathrm{rwd},k}-\tilde{x}_{\mathrm{base},k}),\qquad\text{i.e.}\qquad g_1=\omega_1(\tilde{x}_{+}-\tilde{x}_{\mathrm{ind}}),\qquad g_2=\omega_2(\tilde{x}_{-}-\tilde{x}_{\mathrm{ind}}). \tag*{(73)} g k = ω k ( x ~ rwd , k − x ~ base , k ) , i.e. g 1 = ω 1 ( x ~ + − x ~ ind ) , g 2 = ω 2 ( x ~ − − x ~ ind ) . (73)
Because x ~ ± − x ~ i n d \tilde{x}_{\pm}-\tilde{x}_{\mathrm{ind}} x ~ ± − x ~ ind is constructed from x ~ o l d , u \tilde{x}_{\mathrm{old},u} x ~ old , u by Equation (70) , J J J also cancels from both reward squares (Appendix § 11 ). Hence no square of the loss contains J J J , Equation (68) depends only on x ~ θ = X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] \tilde{x}_{\theta}=X[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r),0] x ~ θ = X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] , x ~ r e g = x ~ i n d ∣ J = 0 \tilde{x}_{\mathrm{reg}}=\tilde{x}_{\mathrm{ind}}|_{J=0} x ~ reg = x ~ ind ∣ J = 0 and the J J J -free g k g_k g k , and by Equation (41) the Jacobian is that of X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] X[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r),0] X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] . FewStepOPSD in v-prediction form thus reduces to the J J J -free ( t , r ) (t,r) ( t , r ) query of §4.3 for every ( t , r ) (t,r) ( t , r ) , not only at r = 0 r=0 r = 0 .
Example 9 (ReFL as Few-step DiffusionOPSD). Reward feedback learning (ReFL) backpropagates the gradient of a differentiable reward through a single late denoising step. Classic ReFL stops a rollout of the old-policy sampler at a small noise level t t t without gradient and lets the policy predict the clean sample from that state with one network evaluation; as in Example 8, we instead evaluate the network at the re-noised state x ~ t ( i ) = AddNoise ( x ^ ( i ) , t ) \tilde{x}_t^{(i)}=\operatorname{AddNoise}(\hat{x}^{(i)},t) x ~ t ( i ) = AddNoise ( x ^ ( i ) , t ) and maximise the reward of the resulting prediction,
L R e F L = − λ E i , t [ R ( D ( x ~ θ ( i ) ) , c ) ] , x ~ θ ( i ) = X [ x ~ t ( i ) , v θ , 0 ] , (74) \mathcal{L}_{\mathrm{ReFL}}=-\lambda\mathbb{E}_{i,t}\left[R(D(\tilde{x}_{\theta}^{(i)}),c)\right],\qquad\tilde{x}_{\theta}^{(i)}=X[\tilde{x}_t^{(i)},v_\theta,0], \tag*{(74)} L ReFL = − λ E i , t [ R ( D ( x ~ θ ( i ) ) , c ) ] , x ~ θ ( i ) = X [ x ~ t ( i ) , v θ , 0 ] , (74) with a loss weight λ > 0 \lambda>0 λ > 0 . Only this final evaluation is differentiated,
∇ θ L R e F L ( i ) = − λ ( ∂ x ~ θ ( i ) ∂ θ ) ⊤ ∇ R ( D ( x ~ θ ( i ) ) , c ) , (75) \nabla_\theta\mathcal{L}_{\mathrm{ReFL}}^{(i)}=-\lambda\left(\frac{\partial\tilde{x}_{\theta}^{(i)}}{\partial\theta}\right)^\top\nabla R(D(\tilde{x}_{\theta}^{(i)}),c), \tag*{(75)} ∇ θ L ReFL ( i ) = − λ ( ∂ θ ∂ x ~ θ ( i ) ) ⊤ ∇ R ( D ( x ~ θ ( i ) ) , c ) , (75) where ∇ R ( D ( ⋅ ) , c ) \nabla R(D(\cdot),c) ∇ R ( D ( ⋅ ) , c ) is the gradient with respect to the clean sample, as in Equation (62) . The backward pass therefore propagates through the decoder and the reward model.
Consider Equation (68) with a single reward-gradient step, M t g t = 1 M_{\mathrm{tgt}}=1 M tgt = 1 , no self-regression, W r e g = 0 W_{\mathrm{reg}}=0 W reg = 0 , and only the ascent term L r w d , 1 \mathcal{L}_{\mathrm{rwd},1} L rwd , 1 , with any detached W r w d , ω 1 > 0 W_{\mathrm{rwd}},\omega_1>0 W rwd , ω 1 > 0 . For η ≤ 1 \eta\leq1 η ≤ 1 the single step is shorter than κ ∥ x ~ o l d ∥ \kappa\left\|\tilde{x}_{\mathrm{old}}\right\| κ ∥ x ~ old ∥ , the projection is inactive, and Equation (63) reads
x ~ + − x ~ o l d = h ∇ R ( D ( x ~ o l d ) , c ) ∥ ∇ R ( D ( x ~ o l d ) , c ) ∥ + ϵ g . (76) \tilde{x}_{+}-\tilde{x}_{\mathrm{old}}=h\frac{\nabla R(D(\tilde{x}_{\mathrm{old}}),c)}{\left\|\nabla R(D(\tilde{x}_{\mathrm{old}}),c)\right\|+\epsilon_g}. \tag*{(76)} x ~ + − x ~ old = h ∥ ∇ R ( D ( x ~ old ) , c ) ∥ + ϵ g ∇ R ( D ( x ~ old ) , c ) . (76)
The loss is the single square L r w d , 1 = W r w d ∥ x ~ θ − sg ( x ~ θ + g 1 ) ∥ 2 \mathcal{L}_{\mathrm{rwd},1}=W_{\mathrm{rwd}}\left\lVert\tilde{x}_{\theta}-\operatorname{sg}(\tilde{x}_{\theta}+g_1)\right\rVert^2 L rwd , 1 = W rwd ∥ x ~ θ − sg ( x ~ θ + g 1 ) ∥ 2 of Equation (68) , using Equation (67) , x ~ r w d , 1 = x ~ + \tilde{x}_{\mathrm{rwd},1}=\tilde{x}_{+} x ~ rwd , 1 = x ~ + , x ~ b a s e , 1 = x ~ o l d \tilde{x}_{\mathrm{base},1}=\tilde{x}_{\mathrm{old}} x ~ base , 1 = x ~ old and ω 1 > 0 \omega_1>0 ω 1 > 0 . Its gradient Equation (28) is
∇ θ L r w d , 1 = − 2 ( ∂ x ~ θ ∂ θ ) ⊤ W r w d ω 1 ( x ~ r w d , 1 − x ~ b a s e , 1 ) = − 2 W r w d ω 1 ( ∂ x ~ θ ∂ θ ) ⊤ ( x ~ + − x ~ o l d ) = − 2 W r w d ω 1 h ∥ ∇ R ( D ( x ~ o l d ) , c ) ∥ + ϵ g ( ∂ x ~ θ ∂ θ ) ⊤ ∇ R ( D ( x ~ o l d ) , c ) . (77) \begin{aligned}
\nabla_{\theta}\mathcal{L}_{\mathrm{rwd},1}=-2\left(\frac{\partial\tilde{x}_{\theta}}{\partial\theta}\right)^{\top}W_{\mathrm{rwd}}\omega_1\left(\tilde{x}_{\mathrm{rwd},1}-\tilde{x}_{\mathrm{base},1}\right)=-2W_{\mathrm{rwd}}\omega_1\left(\frac{\partial\tilde{x}_{\theta}}{\partial\theta}\right)^{\top}\left(\tilde{x}_{+}-\tilde{x}_{\mathrm{old}}\right) \\
&=-\frac{2W_{\mathrm{rwd}}\omega_1 h}{\left\lVert\nabla R\left(D(\tilde{x}_{\mathrm{old}}),c\right)\right\rVert+\epsilon_g}\left(\frac{\partial\tilde{x}_{\theta}}{\partial\theta}\right)^{\top}\nabla R\left(D(\tilde{x}_{\mathrm{old}}),c\right). \tag*{(77)}
\end{aligned} ∇ θ L rwd , 1 = − 2 ( ∂ θ ∂ x ~ θ ) ⊤ W rwd ω 1 ( x ~ rwd , 1 − x ~ base , 1 ) = − 2 W rwd ω 1 ( ∂ θ ∂ x ~ θ ) ⊤ ( x ~ + − x ~ old ) = − ∥ ∇ R ( D ( x ~ old ) , c ) ∥ + ϵ g 2 W rwd ω 1 h ( ∂ θ ∂ x ~ θ ) ⊤ ∇ R ( D ( x ~ old ) , c ) . (77)
Compared with Equation (75) , Equation (77) replaces λ \lambda λ by a detached, positive, sample-dependent factor and evaluates the reward gradient at the old policy’s prediction instead of at x ~ θ \tilde{x}_{\theta} x ~ θ (at x ~ o l d \tilde{x}_{\mathrm{old}} x ~ old for an instantaneous-velocity policy, and at x ~ o l d , u \tilde{x}_{\mathrm{old,u}} x ~ old , u in the few-step form below). It is therefore only a proxy for ReFL. If θ o l d = θ \theta_{\mathrm{old}}=\theta θ old = θ at the evaluation point, which holds at the first inner iteration after each refresh when the EMA-decay ratio is 0.0, then x ~ θ = x ~ o l d \tilde{x}_{\theta}=\tilde{x}_{\mathrm{old}} x ~ θ = x ~ old and the per-sample directions are collinear; exact equality additionally requires the positive factor to equal λ \lambda λ . Thus the DiffusionOPSD configuration with a single ascent step and no self-regression is a detached proxy of ReFL, by setting M t g t = 1 M_{\mathrm{tgt}}=1 M tgt = 1 , W r e g = 0 W_{\mathrm{reg}}=0 W reg = 0 , L r w d , 2 = 0 \mathcal{L}_{\mathrm{rwd},2}=0 L rwd , 2 = 0 and the EMA-decay ratio as 0.0.
For the implemented few-step form, the current prediction and the old-policy anchor follow Equation (69) , while the reward displacement is computed at the r = 0 r=0 r = 0 prediction x ~ o l d , u \tilde{x}_{\mathrm{old,u}} x ~ old , u and transferred to x ~ i n d \tilde{x}_{\mathrm{ind}} x ~ ind by Equation (70) :
x ~ b a s e , 1 = x ~ i n d = X [ x ~ t , u o l d + J [ u o l d , t , r , 0 ] ] , \tilde{x}_{\mathrm{base},1}=\tilde{x}_{\mathrm{ind}}=X\left[\tilde{x}_{t,u_{\mathrm{old}}}+J\left[u_{\mathrm{old}},t,r,0\right]\right], x ~ base , 1 = x ~ ind = X [ x ~ t , u old + J [ u old , t , r , 0 ] ] ,
Δ r w d x ~ + = h ∇ R ( D ( x ~ o l d , u ) , c ) ∥ ∇ R ( D ( x ~ o l d , u ) , c ) ∥ + ϵ g , x ~ r w d , 1 = x ~ + = x ~ i n d + Δ r w d x ~ + , \Delta_{\mathrm{rwd}}\tilde{x}_{+}=h\frac{\nabla R\left(D(\tilde{x}_{\mathrm{old,u}}),c\right)}{\left\lVert\nabla R\left(D(\tilde{x}_{\mathrm{old,u}}),c\right)\right\rVert+\epsilon_g},\qquad\tilde{x}_{\mathrm{rwd},1}=\tilde{x}_{+}=\tilde{x}_{\mathrm{ind}}+\Delta_{\mathrm{rwd}}\tilde{x}_{+}, Δ rwd x ~ + = h ∥ ∇ R ( D ( x ~ old , u ) , c ) ∥ + ϵ g ∇ R ( D ( x ~ old , u ) , c ) , x ~ rwd , 1 = x ~ + = x ~ ind + Δ rwd x ~ + ,
g 1 = ω 1 ( x ~ r w d , 1 − x ~ b a s e , 1 ) = ω 1 Δ r w d x ~ + . (78) g_1=\omega_1\left(\tilde{x}_{\mathrm{rwd},1}-\tilde{x}_{\mathrm{base},1}\right)=\omega_1\Delta_{\mathrm{rwd}}\tilde{x}_{+}. \tag*{(78)} g 1 = ω 1 ( x ~ rwd , 1 − x ~ base , 1 ) = ω 1 Δ rwd x ~ + . (78)
The reward augmentation loss is Equation (72) with this g 1 g_1 g 1 ,
L r w d , 1 = W r w d ∥ X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] − sg ( X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] + g 1 ) ∥ 2 , (79) \mathcal{L}_{\mathrm{rwd},1}=W_{\mathrm{rwd}}\left\lVert X\left[\tilde{x}_{t},u_{\theta}(\tilde{x}_{t},t,r),0\right]-\operatorname{sg}\left(X\left[\tilde{x}_{t},u_{\theta}(\tilde{x}_{t},t,r),0\right]+g_1\right)\right\rVert^2, \tag*{(79)} L rwd , 1 = W rwd ∥ X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] − sg ( X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] + g 1 ) ∥ 2 , (79)
whose reward augmentation moves the prediction X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] X\left[\tilde{x}_{t},u_{\theta}(\tilde{x}_{t},t,r),0\right] X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] at each sampled ( t , r ) (t,r) ( t , r ) along ∇ R ( D ( x ~ o l d , u ) , c ) \nabla R(D(\tilde{x}_{\mathrm{old,u}}),c) ∇ R ( D ( x ~ old , u ) , c ) . Apart from W r w d W_{\mathrm{rwd}} W rwd and the velocity penalty, a step requires one detached rollout, two forward passes of u o l d u_{\mathrm{old}} u old (at r r r and at r = 0 r=0 r = 0 ), one reward gradient, and one forward and backward pass of u θ u_{\theta} u θ ; J J J cancels, because it shifts both ends of the displacement equally, and is therefore dropped. With J J J dropped and r = 0 r=0 r = 0 we have x ~ i n d = x ~ o l d , u \tilde{x}_{\mathrm{ind}}=\tilde{x}_{\mathrm{old,u}} x ~ ind = x ~ old , u , which is the u u u -form. At a generic r r r , only the current prediction and its anchor use ( t , r ) (t,r) ( t , r ) ; the decoder and the reward are still evaluated at the clean prediction x ~ o l d , u \tilde{x}_{\mathrm{old,u}} x ~ old , u , while the resulting displacement trains u θ ( ⋅ , t , r ) u_{\theta}(\cdot,t,r) u θ ( ⋅ , t , r ) .
Implementation
The framework of §4 is implemented as one base trainer whose method step runs one outer iteration. Each trainer of §§5.3–5.6 is a subclass that overrides a few stage methods, and the instantaneous-velocity estimator of §4.3, v-ind or v-aux, is a separate object passed to the trainer. Here v-ind uses the MeanFlow identity with the shared old-policy correction J J J to construct an instantaneous velocity from an average velocity, whereas v-aux uses an auxiliary velocity to approximate the instantaneous velocity of the old-policy rollout distribution ρ o l d \rho_{\mathrm{old}} ρ old (which DMD takes to be ρ θ \rho_{\theta} ρ θ ), as in Equation (52) . Stages 1 and 2 below operate on groups, since they produce and score the K K K members of a prompt, and keep the index ( i ) (i) ( i ) as in §§3–4.3. From Stage 3 onward the framework acts on one member at a time. The prediction, the three anchors and the three weights are per-sample quantities; the index ( i ) (i) ( i ) is omitted in the equations and in the loss terms L r e g \mathcal{L}_{\mathrm{reg}} L reg , L r w d \mathcal{L}_{\mathrm{rwd}} L rwd , and becomes the leading batch axis in code.
A General Training Step
One outer iteration fixes an old policy u o l d u_{\mathrm{old}} u old , rolls it out, scores the rollouts, and then takes one gradient step on the mean over n i t n_{\mathrm{it}} n it queries of
L ( θ ) = E [ W r e g mean s p [ ( x ~ θ − x ~ r e g ) 2 ] + W r w d mean s p [ ( x ~ θ − sg ( x ~ θ + g ) ) 2 ] + λ K L L K L ] (80) \mathcal{L}(\theta) = \mathbb{E}\left[W_{\mathrm{reg}}\,\operatorname{mean}_{\mathrm{sp}}\left[(\tilde{x}_{\theta}-\tilde{x}_{\mathrm{reg}})^2\right] + W_{\mathrm{rwd}}\,\operatorname{mean}_{\mathrm{sp}}\left[(\tilde{x}_{\theta}-\operatorname{sg}(\tilde{x}_{\theta}+g))^2\right] + \lambda_{\mathrm{KL}}\mathcal{L}_{\mathrm{KL}}\right] \tag*{(80)} L ( θ ) = E [ W reg mean sp [ ( x ~ θ − x ~ reg ) 2 ] + W rwd mean sp [ ( x ~ θ − sg ( x ~ θ + g ) ) 2 ] + λ KL L KL ] (80)
where x ^ ( i ) = Rollout ( ϵ ( i ) , u o l d , c ) \hat{x}^{(i)}=\operatorname{Rollout}(\epsilon^{(i)},u_{\mathrm{old}},c) x ^ ( i ) = Rollout ( ϵ ( i ) , u old , c ) is a detached rollout of prompt c c c . The per-sample mean mean s p [ ⋅ ] \operatorname{mean}_{\mathrm{sp}}[\cdot] mean sp [ ⋅ ] rescales every weight W W W by 1 / N 1/N 1/ N relative to the squared norms of § 4.2 ; this constant is absorbed into the learning rate. L K L \mathcal{L}_{\mathrm{KL}} L KL is the velocity penalty of Stage 4 with weight λ K L \lambda_{\mathrm{KL}} λ KL , evaluated on the same method-specific query.
The training workflow. One outer iteration consists of five operations, executed in order; the four method-specific stages of § 1 are operations 1, 2, 4 and the loss inside 4, while operations 3 and 5 are shared bookkeeping.
Rollout: the old policy u o l d u_{\mathrm{old}} u old samples K K K members per prompt with the sampler of Stage 1, and the endpoints x ^ ( i ) \hat{x}^{(i)} x ^ ( i ) are detached.
Reward and advantage: each endpoint is decoded and scored, R ( i ) = R ( D ( x ^ ( i ) ) , c ) R^{(i)}=R(\mathcal{D}(\hat{x}^{(i)}),c) R ( i ) = R ( D ( x ^ ( i ) ) , c ) , and Stage 2 turns the rewards into the advantage A ( i ) A^{(i)} A ( i ) and the mixing weight r ^ ( i ) \hat{r}^{(i)} r ^ ( i ) (Equation (83) , Equation (84) ).
Auxiliary fit: a trainer that uses the auxiliary velocity v ϕ v_{\phi} v ϕ first fits it to these rollouts with the flow-matching loss Equation (88) for n f m n_{\mathrm{fm}} n fm steps, before the policy is updated; the other trainers skip this step.
Training loss: n i t n_{\mathrm{it}} n it inner iterations each draw a query (Stage 3 (i)), compute the prediction, anchors and coefficients (Stage 3 (ii)–(iv)), and accumulate the gradient of Equation (80) (Stage 4); one optimiser step is taken on their mean.
Refresh: u o l d u_{\mathrm{old}} u old becomes the EMA of u θ u_{\theta} u θ , and the next iteration rolls out with it.
In code, this is the method step of the base trainer in Code 1, one block per operation: rollout, advantage, estimator.fit, the loop over queries with anchors, weights and loss, and refresh_old. Each Query object is a view of the batch for one method-specific training query; fields it does not set, such as the advantage, are read from the batch.
A trainer of §§§ 5.3 –§ 5.6 is a subclass that implements anchors and weights, and overrides another hook only where it changes that stage (Table 2 ). anchors returns the prediction and three anchors of Equation (90) , with a tuple of rewarded anchors for FewStepOPSD; weights returns the detached coefficients, with ω \omega ω signed and a tuple for FewStepOPSD, and loss is shared by every trainer except FewStepOPSD and ReFL, which override it. The named methods of § 6 other than ReFL differ from their family only in the estimator they use.
Table 2. Specific implementation of stages of Code 1 for each trainer of §6. The method-specific quantities μ θ \mu_\theta μ θ , ρ ˉ G R P O \bar{\rho}_{\mathrm{GRPO}} ρ ˉ GRPO , ρ A W M \rho_{\mathrm{AWM}} ρ AWM , χ G R P O \chi_{\mathrm{GRPO}} χ GRPO , χ A W M \chi_{\mathrm{AWM}} χ AWM , σ k \sigma_k σ k , τ A W M \tau_{\mathrm{AWM}} τ AWM , a ± a_{\pm} a ± , γ ± \gamma_{\pm} γ ± and c a d v c_{\mathrm{adv}} c adv are defined in §§5.2–5.5.
Code 1: The base trainer: one outer iteration
1 @dataclass
2 class Points: # prediction and anchors of Eq. (90)
3 x_z: Tensor # the only tensor with a gradient
4 x_reg: Tensor | None # None disables self-regression
5 x_base: Tensor | None # None disables reward augmentation
6 x_rwd: Tensor | tuple[Tensor, ...] # a tuple for multi-branch FewStepOPSD
7
8 class FewStepRLTrainer:
9 estimator = VInd() # Stage 3(i)-(iii): \texttt{v-ind} or
↪ \texttt{v-aux}
10 n_it = n_ts # default; FewStepGRPO uses $M_{\mathrm{step}}$, FewStepOPSD
↪ uses 1
11
12 def step(self, prompts): # ONE outer iteration
13 batch = self.rollout(prompts) # Stage 1, detached
14 R = reward(decode(batch.x_hat), batch.c) # Stage 2
15 batch.A, batch.r_hat = self.advantage(R, batch.c)
16 self.estimator.fit(batch) # \texttt{v-aux} only, Eq. (88)
17 for q in self.queries(batch): # Stage 3(i): $n_{\mathrm{it}}$ queries
18 p = self.anchors(q) # Stage 3(ii)-(iii)
19 W_reg, W_rwd, omega = self.weights(q, p) # Stage 3(iv)
20 loss = self.loss(p, W_reg, W_rwd, omega) # Stage 4
21 accumulate(mean_batch(loss + lam_kl * self.kl(q)) / self.n_it)
22 self.update() # one step on $\theta$
23 self.refresh_old() # $u_{\mathrm{old}} \leftarrow \mathrm{EMA}$ of $u_\theta$
24
25 def rollout(self, prompts): # Eq. (81)
26 return ode_rollout(prompts, u_old)
27 def advantage(self, R, c): # Eqs. (83), (84)
28 return adv_and_rhat(R, c)
29 def queries(self, batch): # Stage 3(i)
30 for _ in range(self.n_it):
31 t, r = draw_tr(batch) # $r \mid t \sim U(0,t)$; pinned $r=t$, $r=0$
32 eps = randn_like(batch.x_hat) # fresh noise at every query
33 yield Query(batch, t=t, r=r, eps=eps, x_t=add_noise(batch.x_hat, t, eps))
34 def anchors(self, q): raise NotImplementedError # -> Points
35 def weights(self, q, p): raise NotImplementedError # -> W_reg, W_rwd, omega
36 def loss(self, p, W_reg, W_rwd, omega): # Code 3; OPSD overrides
37 return loss(p, W_reg, W_rwd, omega)
Introduction to the Key Stages
This subsection describes, for each stage of Code 1, the default computation, its cost, and the choices that the framework determines.
Stage 1: the rollout operator Let Rollout denote the sampler. We distinguish two cases.
(a) Sample-independent rollout. For every trainer except the FewStepOPSD family, the group members of a prompt are i.i.d. draws:
x ^ ( i ) = Rollout ( ϵ ( i ) , u o l d , c ) , ϵ ( i ) ∼ i i d N ( 0 , I ) , i = 1 , … , K , (81) \hat{x}^{(i)} = \operatorname{Rollout}(\epsilon^{(i)}, u_{\mathrm{old}}, c), \qquad\epsilon^{(i)} \overset{\mathrm{iid}}{\sim} \mathcal{N}(0,I), \qquad i=1,\ldots,K, \tag*{(81)} x ^ ( i ) = Rollout ( ϵ ( i ) , u old , c ) , ϵ ( i ) ∼ iid N ( 0 , I ) , i = 1 , … , K , (81)
i.e. the only stochasticity is the initial noise (and, for the stochastic sampler below, the per-step renoise). Members do not interact.
†-general is FewStepOPSD in its general form, with both the ascent and the descent points x ~ ± \tilde{x}_{\pm} x ~ ± . The ReFL proxy configuration sets r ^ ≡ 1 \hat{r}\equiv1 r ^ ≡ 1 , keeps only the ascent point x ~ + \tilde{x}_{+} x ~ + , and sets W r e g = 0 W_{\mathrm{reg}}=0 W reg = 0 .
‡ x ~ i n d ∣ J = 0 = X [ x ~ t , u o l d ( x ~ t , t , r ) , 0 ] \tilde{x}_{\mathrm{ind}\mid J=0}=X[\tilde{x}_t,u_{\mathrm{old}}(\tilde{x}_t,t,r),0] x ~ ind ∣ J = 0 = X [ x ~ t , u old ( x ~ t , t , r ) , 0 ] , used together with the J J J -free ( t , r ) (t,r) ( t , r ) query X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] X[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r),0] X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] . Dropping J J J is exact, because J J J cancels in every difference whose two terms contain the same detached J J J . In DMDNFT it cancels in the self-regression square, Equation (60) , while the baseline x ~ a u x \tilde{x}_{\mathrm{aux}} x ~ aux contains no J J J ; in FewStepOPSD and ReFL it cancels in every square of the loss, Equation (71) –Equation (72) , so the loss and its gradient are the same for every detached J J J (Section C.2).
⋆ a + = r ^ / γ + a_{+}=\hat{r}/\gamma_{+} a + = r ^ / γ + and a − = ( 1 − r ^ ) / γ − a_{-}=(1-\hat{r})/\gamma_{-} a − = ( 1 − r ^ ) / γ − , with the adaptive normalisers γ ± \gamma_{\pm} γ ± of Equation (105) : measured against x ^ \hat{x} x ^ in MeanFlowNFT, centred at x ~ b a s e \tilde{x}_{\mathrm{base}} x ~ base and displaced along sg ( x ~ θ ) − x ~ r e g \operatorname{sg}(\tilde{x}_\theta)-\tilde{x}_{\mathrm{reg}} sg ( x ~ θ ) − x ~ reg in DMDNFT, where x ~ r e g ≠ x ~ b a s e \tilde{x}_{\mathrm{reg}}\ne\tilde{x}_{\mathrm{base}} x ~ reg = x ~ base , and measured against the constructed points x ~ ± \tilde{x}_{\pm} x ~ ± in FewStepOPSD and ReFL (Equation (107) ). The factor c a d v c_{\mathrm{adv}} c adv scales W W W in NFT and general OPSD to undo the compression of the advantage in Equation (84) (§ 5.5 ), and is kept by the ReFL override. In FewStepOPSD and ReFL the sign of ω \omega ω is fixed by construction, + + + for the ascent term and − - − for the descent term (Equation (67) ), and is not determined by A A A .
The sampler uses the old policy u o l d u_{\mathrm{old}} u old and not u θ u_\theta u θ , as indicated by the argument of Equation (81) . One old policy is fixed per outer iteration, so that the rewarded endpoint x ^ ( i ) \hat{x}^{(i)} x ^ ( i ) , the regression anchor x ~ r e g ( i ) \tilde{x}_{\mathrm{reg}}^{(i)} x ~ reg ( i ) and the baseline anchor x ~ b a s e ( i ) \tilde{x}_{\mathrm{base}}^{(i)} x ~ base ( i ) all refer to the same distribution, as required in § 4.2 , where x ~ ( i ) \tilde{x}^{(i)} x ~ ( i ) and x ~ r e g ( i ) \tilde{x}_{\mathrm{reg}}^{(i)} x ~ reg ( i ) must be evaluated at the state from which x ~ θ ( i ) \tilde{x}_{\theta}^{(i)} x ~ θ ( i ) is computed. If the samples were drawn from u θ u_\theta u θ , the reward would be evaluated on a distribution different from the one described by the anchors, and A ( i ) A^{(i)} A ( i ) would not be the advantage of the policy that defines the anchors.
(b) Reward-coupled rollout. For reward-coupled rollout, FewStepOPSD extends the rollout with the reward,
{ x ^ ( i ) , x ~ o l d u , ( i ) , x ~ ± ( i ) } i = 1 K = Rollout ( ϵ ( i ) , u o l d , R , c ) . (82) \left\{\hat{x}^{(i)},\tilde{x}_{\mathrm{old}}^{u,(i)},\tilde{x}_{\pm}^{(i)}\right\}_{i=1}^{K}
= \operatorname{Rollout}(\epsilon^{(i)},u_{\mathrm{old}},R,c). \tag*{(82)} { x ^ ( i ) , x ~ old u , ( i ) , x ~ ± ( i ) } i = 1 K = Rollout ( ϵ ( i ) , u old , R , c ) . (82)
The members remain i.i.d.; the reward does not couple different members. For each member, the sampler of Equation (81) runs unchanged. The reward-coupled construction then applies Equation (62) from the r = 0 r=0 r = 0 prediction x ~ o l d u , ( i ) \tilde{x}_{\mathrm{old}}^{u,(i)} x ~ old u , ( i ) and transfers the resulting displacements to the anchor by Equation (70) , giving x ~ ± ( i ) \tilde{x}_{\pm}^{(i)} x ~ ± ( i ) together with the endpoint.
In code, rollout defaults to (a) with the deterministic sampler; FewStepGRPO overrides it with the stochastic sampler of § 5.3 , keeping the transitions, and FewStepOPSD appends the reward-coupled construction of (b).
Stage 2: reward and group advantage The generic advantage Equation (12) takes both the mean and the standard deviation over the group; the implementation changes the population of the standard deviation. With R ( i ) = R ( D ( x ^ ( i ) ) , c ) R^{(i)}=R(\mathcal{D}(\hat{x}^{(i)}),c) R ( i ) = R ( D ( x ^ ( i ) ) , c ) , the endpoint reward (D \mathcal{D} D the VAE decoder), the advantage is
A ( i ) = R ( i ) − R ˉ c std ( { R } B ) + 10 − 4 , R ˉ c = 1 ∣ H c ∣ ∑ R ∈ H c R , (83) A^{(i)} = \frac{R^{(i)}-\bar{R}_c}{\operatorname{std}(\{R\}_B)+10^{-4}}, \qquad\bar{R}_c = \frac{1}{|H_c|}\sum_{R\in H_c} R, \tag*{(83)} A ( i ) = std ({ R } B ) + 1 0 − 4 R ( i ) − R ˉ c , R ˉ c = ∣ H c ∣ 1 R ∈ H c ∑ R , (83)
where H c H_c H c is the set of rewards of prompt c c c in the current outer iteration and { R } B \{R\}_B { R } B is the current global batch gathered across all ranks. The numerator and denominator use different populations. The per-prompt mean reduces the effect of prompt difficulty, and the global standard deviation avoids division by a near-zero value when all rewards in a group are equal.
The advantage is then compressed into a bounded mixing weight, generalising Equation (18) by replacing the unit clip range with a tunable c a d v c_{\mathrm{adv}} c adv :
r ^ ( i ) = clip ( A ( i ) , − c a d v , c a d v ) 2 c a d v + 1 2 ∈ [ 0 , 1 ] . (84) \hat{r}^{(i)} = \frac{\operatorname{clip}\left(A^{(i)},-c_{\mathrm{adv}},c_{\mathrm{adv}}\right)}{2c_{\mathrm{adv}}}+\frac{1}{2}\in[0,1]. \tag*{(84)} r ^ ( i ) = 2 c adv clip ( A ( i ) , − c adv , c adv ) + 2 1 ∈ [ 0 , 1 ] . (84)
Here r ^ → 1 \hat{r}\to1 r ^ → 1 means “far above the prompt’s mean”, r ^ → 0 \hat{r}\to0 r ^ → 0 “far below”, r ^ = 1 2 \hat{r}=\frac{1}{2} r ^ = 2 1 “average”. This [ 0 , 1 ] [0,1] [ 0 , 1 ] form distinguishes the NFT/OPSD families from FewStepGRPO and FewStepAWM, which use clip ( A , − c a d v , c a d v ) \operatorname{clip}(A,-c_{\mathrm{adv}},c_{\mathrm{adv}}) clip ( A , − c adv , c adv ) as a signed multiplier (§ 3.2 ); A A A in Equation (97) and Equation (103) denotes this clipped value.
In code, advantage returns ( A , r ^ ) (A,\hat{r}) ( A , r ^ ) ; ReFL overrides it with r ^ ≡ 1 \hat{r}\equiv1 r ^ ≡ 1 .
Stage 3: anchors and weights The trainers differ mainly in this stage. It receives the detached rollout and returns the current prediction, the anchors and the detached coefficients; FewStepOPSD returns branch-indexed tuples for its rewarded anchors and ω \omega ω . We describe it in four parts, namely ( i ) (i) ( i ) the query levels, ( i i ) (ii) ( ii ) the two ways to compute the clean prediction, ( i i i ) (iii) ( iii ) the two estimators of the instantaneous velocity that these predictions require, and ( i v ) (iv) ( i v ) the weights.
( i ) (i) ( i ) The training query ( t , r ) (t,r) ( t , r ) . By default, a fresh pair ( t , r ) (t,r) ( t , r ) is drawn at every inner iteration; “r r r is resampled at every update” in § 4.3 refers to this draw: t t t is drawn from a selectable law (the maximum of two uniforms or a logit-normal), r ∣ t ∼ U ( 0 , t ) r\mid t\sim U(0,t) r ∣ t ∼ U ( 0 , t ) , and a fixed fraction of the batch is set to r = t r=t r = t and to r = 0 r=0 r = 0 . The three cases play different roles. For r = t r=t r = t the interval has zero length and J ≡ 0 J\equiv0 J ≡ 0 , so the network query is instantaneous and the loss reduces to its instantaneous-velocity form; r = 0 r=0 r = 0 trains the one-step generator and gives J J J its largest prefactor t − r t-r t − r ; the generic case lets ∂ x ~ θ / ∂ θ \partial\tilde{x}_\theta/\partial\theta ∂ x ~ θ / ∂ θ reach u θ ( ⋅ , t , r ) u_\theta(\cdot,t,r) u θ ( ⋅ , t , r ) for every r r r in between, as required by the second column of Table 1 .
( i i ) (ii) ( ii ) The u u u - and v v v -predictions. Except for the replayed transition score of FewStepGRPO, the implemented reward losses evaluate Equation (21) at the clean level r = 0 r=0 r = 0 , but the network outputs the average velocity u θ ( x ~ t , t , r ) u_\theta(\tilde{x}_t,t,r) u θ ( x ~ t , t , r ) instead of the instantaneous velocity required by the map. Using u θ u_\theta u θ directly gives the u-form:
x ~ θ = x ~ θ , u = X [ x ~ t , u θ ( x ~ t , t , 0 ) , 0 ] (u-form) . (85) \tilde{x}_\theta=\tilde{x}_{\theta,u}=X\left[\tilde{x}_t,u_\theta(\tilde{x}_t,t,0),0\right]\qquad\text{(u-form)}. \tag*{(85)} x ~ θ = x ~ θ , u = X [ x ~ t , u θ ( x ~ t , t , 0 ) , 0 ] (u-form) . (85)
Adding the detached correction of Equation (37) gives the v-form; since the prediction carries the gradient, v-ind builds it with the self-induced velocity, i.e.,
x ~ θ = x ~ θ , v = X [ x ~ t , u θ ( x ~ t , t , r ) + J [ u o l d , t , r ] , 0 ] (v-form, v-ind) . (86) \tilde{x}_\theta=\tilde{x}_{\theta,v}=X\left[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r)+J\left[u_{\mathrm{old}},t,r\right],0\right]\qquad\text{(v-form, v-ind)}. \tag*{(86)} x ~ θ = x ~ θ , v = X [ x ~ t , u θ ( x ~ t , t , r ) + J [ u old , t , r ] , 0 ] (v-form, v-ind) . (86)
As in Equation (37) , J J J is computed from u o l d u_{\mathrm{old}} u old , so that a single estimate can be shared by all predictions, as required in ( i i i ) (iii) ( iii ) . Since J J J is detached, by Equation (41) the Jacobian of the v-form x ~ θ \tilde{x}_\theta x ~ θ is that of X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] X\left[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r),0\right] X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] .
For the anchors x ~ b a s e \tilde{x}_{\mathrm{base}} x ~ base , x ~ r e g \tilde{x}_{\mathrm{reg}} x ~ reg and x ~ r w d \tilde{x}_{\mathrm{rwd}} x ~ rwd , there are two alternatives in v-form, namely v-aux and v-ind.
(iii) Estimating the instantaneous velocity. There are two estimators of the instantaneous velocity, following MeanFlow (a.k.a. v-ind) and DMD (a.k.a. v-aux), respectively.
Self-induced v i n d v_{\mathrm{ind}} v ind , with shared J J J (v-ind). The v-form needs J J J only as a detached vector. The directional-difference estimate, for a step size ε c d > 0 \varepsilon_{\mathrm{cd}} > 0 ε cd > 0 , is
J [ u o l d , t , r ] = ( t − r ) u o l d ( x ~ t + ε c d v c o n d , t + ε c d , r ) − u o l d ( x ~ t − ε c d v c o n d , t − ε c d , r ) 2 ε c d . (87) J[u_{\mathrm{old}}, t, r] = (t-r)\frac{u_{\mathrm{old}}(\tilde{x}_t+\varepsilon_{\mathrm{cd}}v_{\mathrm{cond},t+\varepsilon_{\mathrm{cd}},r})-u_{\mathrm{old}}(\tilde{x}_t-\varepsilon_{\mathrm{cd}}v_{\mathrm{cond},t-\varepsilon_{\mathrm{cd}},r})}{2\varepsilon_{\mathrm{cd}}}. \tag*{(87)} J [ u old , t , r ] = ( t − r ) 2 ε cd u old ( x ~ t + ε cd v cond , t + ε cd , r ) − u old ( x ~ t − ε cd v cond , t − ε cd , r ) . (87)
For r = t r=t r = t the prefactor t − r t-r t − r makes it zero. It requires two additional forward passes of u o l d u_{\mathrm{old}} u old per query, and one estimate is shared by the current prediction and the anchors, so the correction cancels in the self-regression difference x ~ θ − x ~ r e g \tilde{x}_{\theta}-\tilde{x}_{\mathrm{reg}} x ~ θ − x ~ reg and across sg ( ⋅ ) \operatorname{sg}(\cdot) sg ( ⋅ ) , as discussed in Example 7 .
Auxiliary v ϕ v_{\phi} v ϕ , trained online (v-aux). v ϕ v_{\phi} v ϕ is instead trained at every update by plain flow matching on the current rollout pool,
L a u x = mean s p [ ( v ϕ ( x ~ t , t ) − v c o n d ) 2 ] , x ~ t = AddNoise ( x ^ , t ) , (88) \mathcal{L}_{\mathrm{aux}}=\operatorname{mean}_{\mathrm{sp}}\left[(v_{\phi}(\tilde{x}_t,t)-v_{\mathrm{cond}})^2\right], \qquad\tilde{x}_t=\operatorname{AddNoise}(\hat{x},t), \tag*{(88)} L aux = mean sp [ ( v ϕ ( x ~ t , t ) − v cond ) 2 ] , x ~ t = AddNoise ( x ^ , t ) , (88)
running n f m n_{\mathrm{fm}} n fm steps of Equation (88) per outer iteration, before the policy update. The auxiliary-to-policy optimiser-step ratio is n f m : 1 n_{\mathrm{fm}}:1 n fm : 1 (n f m = 5 n_{\mathrm{fm}}=5 n fm = 5 in § 6 ). This requires no extra rollout and no J J J , but it requires a second network and optimiser that must follow a target that changes with the policy, as stated in Example 7 .
Code 2 implements the two instantaneous-velocity estimators as estimator objects. Their method fields returns the current and old-policy velocities of one query with one J J J , so the estimate of (iii) is shared by construction; VInd adds the correction Equation (87) , ReplayVInd uses Equation (109) , VInd(cd=False) drops J J J where it cancels (Appendix § 11 ), and VAux adds v ϕ v_{\phi} v ϕ and fits it once per outer iteration by Equation (88) .
class VelocityEstimator: # Stage 3(ii)-(iii)
def r(self, q): return q.r # lower endpoint of the flow map
def J(self, q, r): return 0. # detached correction $J[u_{\rm old},t,r]$
def v_base(self, q): return None # field of $\tilde{x}_{\rm base}$, \texttt{v-aux} only
def fit(self, batch): pass # once per outer iteration
def fields(self, q, r=None): # ONE $J$ for current and old fields
r = self.r(q) if r is None else r
J = self.J(q, r)
v_th = u_theta(q.x_t, q.t, r) + J # current field; differentiated
v_old = sg(u_old(q.x_t, q.t, r)) + J # old policy
return v_th, v_old, self.v_base(q)
class VInd(VelocityEstimator): # Eq. (86)
def __init__(self, cd=True): self.cd = cd # cd=False: $J$ cancels in the loss
def directional_J(self, q, lower, tangent): # Eq. (87), elementwise
if not self.cd: return 0.
up = u_old(q.x_t + eps_cd * tangent, q.t + eps_cd, lower)
dn = u_old(q.x_t - eps_cd * tangent, q.t - eps_cd, lower)
return (q.t - lower) * sg(up - dn) / (2 * eps_cd)
def J(self, q, r):
return self.directional_J(q, r, q.v_cond) # tangent $v_{\rm cond} = \epsilon - \hat{x}$
class ReplayVInd(VInd): # Eq. (109), GRPO only
def J(self, q, s):
return self.directional_J(q, s, (q.x_t - q.x_hat) / q.t) # two forwards
class VAux(VelocityEstimator): # Eq. (60): no $J$
def v_base(self, q): return sg(v_phi(q.x_t, q.t))
def fit(self, batch): # Eq. (88), before the update
for _ in range(n_fm):
t, eps = draw_t(batch), randn_like(batch.x_hat)
x_t = add_noise(batch.x_hat, t, eps)
v_cond = eps - batch.x_hat
aux_step(mean_sp((v_phi(x_t, t) - v_cond) ** 2))
(iv) The loss weights. Equation (28) leaves three per-sample, detached weights free,
( W r e g , W r w d , ω ) , (89) (W_{\rm reg}, W_{\rm rwd}, \omega), \tag*{(89)} ( W reg , W rwd , ω ) , (89)
namely the weight of the self-regression term, the weight of the reward-augmentation term, and the signed reward coefficient of the reward step. The four families differ mainly in these weights, which are given in Equation (97) , Equation (103) , Equation (106) and Equation (107) and returned, with the sign of ω \omega ω , by the hook weights.
Stage 4: the decoupled two-term loss This stage implements Equation (30) without the group index and with the norms replaced by the per-sample mean meansp[·] of Appendix A, which is the reduction used in the code. Given ( x ~ θ ; x ~ r e g , x ~ b a s e , x ~ r w d ; W r e g , W r w d , ω ) (\tilde{x}_{\theta};\tilde{x}_{\rm reg},\tilde{x}_{\rm base},\tilde{x}_{\rm rwd};W_{\rm reg},W_{\rm rwd},\omega) ( x ~ θ ; x ~ reg , x ~ base , x ~ rwd ; W reg , W rwd , ω ) , every trainer except FewStepOPSD, which adds a second reward square (§ 5.6 ), minimises
L = W r e g mean s p [ ( x ~ θ − x ~ r e g ) 2 ] + W r w d mean s p [ ( x ~ θ − sg ( x ~ θ + g ) ) 2 ] , (90) \mathcal{L}
= W_{\rm reg}\operatorname{mean}_{\rm sp}\left[(\tilde{x}_{\theta}-\tilde{x}_{\rm reg})^2\right]
+ W_{\rm rwd}\operatorname{mean}_{\rm sp}\left[(\tilde{x}_{\theta}-\operatorname{sg}(\tilde{x}_{\theta}+g))^2\right], \tag*{(90)} L = W reg mean sp [ ( x ~ θ − x ~ reg ) 2 ] + W rwd mean sp [ ( x ~ θ − sg ( x ~ θ + g ) ) 2 ] , (90)
with W r e g mean s p [ ( x ~ θ − x ~ r e g ) 2 ] ⏟ L r e g ( s e l f - r e g r e s s i o n ) \underbrace{W_{\rm reg}\operatorname{mean}_{\rm sp}\left[(\tilde{x}_{\theta}-\tilde{x}_{\rm reg})^2\right]}_{\mathcal{L}_{\rm reg}\ {\rm(self\text{-}regression)}} L reg ( self - regression ) W reg mean sp [ ( x ~ θ − x ~ reg ) 2 ] and W r w d mean s p [ ( x ~ θ − sg ( x ~ θ + g ) ) 2 ] ⏟ L r w d ( r e w a r d a u g m e n t a t i o n ) \underbrace{W_{\rm rwd}\operatorname{mean}_{\rm sp}\left[(\tilde{x}_{\theta}-\operatorname{sg}(\tilde{x}_{\theta}+g))^2\right]}_{\mathcal{L}_{\rm rwd}\ {\rm(reward\ augmentation)}} L rwd ( reward augmentation ) W rwd mean sp [ ( x ~ θ − sg ( x ~ θ + g ) ) 2 ] .
with the reward displacement
g = ω ( x ~ r w d − x ~ b a s e ) (detached) . (91) g = \omega\left(\tilde{x}_{\mathrm{rwd}} - \tilde{x}_{\mathrm{base}}\right) \qquad\text{(detached)}. \tag*{(91)} g = ω ( x ~ rwd − x ~ base ) (detached) . (91)
With a single reward term, the base interface is specified by six entries:
( x ~ r e g , x ~ b a s e , x ~ r w d , W r e g , W r w d , ω ) . \left(\tilde{x}_{\mathrm{reg}},\ \tilde{x}_{\mathrm{base}},\ \tilde{x}_{\mathrm{rwd}},\ W_{\mathrm{reg}},\ W_{\mathrm{rwd}},\ \omega\right). ( x ~ reg , x ~ base , x ~ rwd , W reg , W rwd , ω ) .
§§5.3–5.6 specify these quantities, one family at a time, from the derivations of Examples 6–9; FewStepOPSD provides branch-indexed ( x ~ r w d , k , ω k ) (\tilde{x}_{\mathrm{rwd},k}, \omega_k) ( x ~ rwd , k , ω k ) entries. Setting x ~ r e g = ∅ \tilde{x}_{\mathrm{reg}} = \varnothing x ~ reg = ∅ switches off L r e g \mathcal{L}_{\mathrm{reg}} L reg ; setting x ~ b a s e = ∅ \tilde{x}_{\mathrm{base}} = \varnothing x ~ base = ∅ switches off L r w d \mathcal{L}_{\mathrm{rwd}} L rwd .
def self_regression(x_z, x_reg, W_reg): # L_reg
return W_reg * mean_sp((x_z - x_reg) ** 2)
def reward_augmentation(x_z, x_rwd, x_base, omega, W_rwd): # L_rwd
g = omega * (x_rwd - x_base) # Eq. (91); omega is signed
# value −g, Jacobian the identity: Eq. (29) holds exactly
return W_rwd * mean_sp((x_z - sg(x_z + g)) ** 2)
def loss(p, W_reg, W_rwd, omega): # Stage 4, Eq. (90)
l = 0.
if p.x_reg is not None: # x̃_reg = ∅: no self-regression
l = l + self_regression(p.x_z, p.x_reg, W_reg)
if p.x_base is not None: # x̃_base = ∅: no reward augmentation
l = l + reward_augmentation(p.x_z, p.x_rwd, p.x_base, omega, W_rwd)
return l
Code 3 implements Stage 4 except the velocity penalty. The ∅ \varnothing ∅ cases correspond to the two None checks, so every trainer except FewStepOPSD and ReFL runs the same loss regardless of the anchors it defines.
In addition to Equation (90) , an implemented loss may include a velocity penalty against a frozen reference with its own weight, which encourages the policy to stay close to the reference. This penalty is a separate term and not an additional anchor. The average-velocity reference u r e f u_{\mathrm{ref}} u ref is the initial pre-RL policy and remains fixed throughout training. For an endpoint-renoise query the KL-u penalty is
L K L u = mean s p [ ( u θ ( x ~ t , t , r ) − u r e f ( x ~ t , t , r ) ) 2 ] . (92) \mathcal{L}_{\mathrm{KL}}^{\mathrm{u}} = \operatorname{mean}_{\mathrm{sp}}\left[\left(u_{\theta}(\tilde{x}_t,t,r) - u_{\mathrm{ref}}(\tilde{x}_t,t,r)\right)^2\right]. \tag*{(92)} L KL u = mean sp [ ( u θ ( x ~ t , t , r ) − u ref ( x ~ t , t , r ) ) 2 ] . (92)
When the reference is an instantaneous velocity, here the teacher v r e a l v_{\mathrm{real}} v real , it cannot be compared with u θ u_{\theta} u θ directly for r ≠ t r \ne t r = t . We therefore convert it to an average-velocity target as in MeanFlow and regress u θ u_{\theta} u θ onto it (the KL-v penalty),
L K L v = mean s p [ ( u θ ( x ~ t , t , r ) − ( v r e a l ( x ~ t , t ) − J [ u o l d , t , r ] ) ) 2 ] . (93) \mathcal{L}_{\mathrm{KL}}^{\mathrm{v}} = \operatorname{mean}_{\mathrm{sp}}\left[\left(u_{\theta}(\tilde{x}_t,t,r) - \left(v_{\mathrm{real}}(\tilde{x}_t,t) - J[u_{\mathrm{old}},t,r]\right)\right)^2\right]. \tag*{(93)} L KL v = mean sp [ ( u θ ( x ~ t , t , r ) − ( v real ( x ~ t , t ) − J [ u old , t , r ] ) ) 2 ] . (93)
FewStepGRPO: Replaying Stored Trajectories
FewStepGRPO is our replay-based surrogate adaptation of FlowGRPO (Liu et al., 2025a).
Stochastic sampler. The stochastic sampler generalises Equation (11) . Each step first moves from t j t_j t j to a level s j ≤ t j + 1 s_j \le t_{j+1} s j ≤ t j + 1 , drawn uniformly from the remaining grid levels, and then adds noise to return to level t j + 1 t_{j+1} t j + 1 on the linear path,
x ^ s j = X [ x ^ t j , u o l d ( x ^ t j , t j , s j ) , s j ] , x ^ t j + 1 = 1 − t j + 1 1 − s j x ^ s j + τ j ϵ ′ , ϵ ′ ∼ N ( 0 , I ) , (94) \hat{x}_{s_j} = X\left[\hat{x}_{t_j}, u_{\mathrm{old}}(\hat{x}_{t_j}, t_j, s_j), s_j\right], \qquad\hat{x}_{t_{j+1}} = \frac{1-t_{j+1}}{1-s_j}\hat{x}_{s_j} + \tau_j\epsilon', \qquad\epsilon' \sim\mathcal{N}(0,I), \tag*{(94)} x ^ s j = X [ x ^ t j , u old ( x ^ t j , t j , s j ) , s j ] , x ^ t j + 1 = 1 − s j 1 − t j + 1 x ^ s j + τ j ϵ ′ , ϵ ′ ∼ N ( 0 , I ) , (94)
where the leading coefficient matches the clean component at level t j + 1 t_{j+1} t j + 1 and τ j \tau_j τ j matches the noise variance, τ j 2 = t j + 1 2 − ( 1 − t j + 1 1 − s j ) 2 s j 2 \tau_j^2 = t_{j+1}^2 - \left(\frac{1-t_{j+1}}{1-s_j}\right)^2s_j^2 τ j 2 = t j + 1 2 − ( 1 − s j 1 − t j + 1 ) 2 s j 2 , which is non-negative if and only if s j ≤ t j + 1 s_j \le t_{j+1} s j ≤ t j + 1 . In Equation (94) , s j s_j s j is the level at which the flow map is queried and t j + 1 t_{j+1} t j + 1 is the target level of the step; in the notation of Equation (5) , the flow map is thus queried with lower endpoint s j s_j s j rather than r = t j + 1 r=t_{j+1} r = t j + 1 , and r = t j + 1 r=t_{j+1} r = t j + 1 denotes the target level of the transition. Equation (11) is the case s j = 0 s_j=0 s j = 0 , with t ′ t' t ′ in place of t j + 1 t_{j+1} t j + 1 .
Replay-trajectory query. Unlike endpoint-renoise trainers, it retains the M s t e p M_{\mathrm{step}} M step transitions of the stochastic rollout, replays one stored ( x t , x r , t , r ) (x_t,x_r,t,r) ( x t , x r , t , r ) per inner iteration, and redraws only the flow-map query level s s s , with the law of r r r in Stage 3 ( i ) (i) ( i ) . The redrawn s s s need not equal the grid level s j s_j s j used at rollout time; Section B.2 shows that the scored kernel is unaffected by this choice at zero drift. Thus n i t = M s t e p n_{\mathrm{it}}=M_{\mathrm{step}} n it = M step , one endpoint advantage supervises all transitions of a member, and the differentiated quantity is the mean μ θ \mu_\theta μ θ of the scoring kernel at level r r r instead of a clean prediction.
Scoring kernel. FewStepGRPO scores x r x_r x r with an Euler–Maruyama Gaussian, which reads
μ θ : = α t x t − b t ( u θ ( x t , t , s ) + J ) , μ o l d : = α t x t − b t ( u o l d ( x t , t , s ) + J ) , α t : = 1 − ς t 2 2 t Δ t , b t : = ( 1 + ς t 2 ( 1 − t ) 2 t ) Δ t , σ k : = ς t Δ t . (95) \begin{aligned}
\mu_\theta&:= \alpha_t x_t - b_t\left(u_\theta(x_t,t,s)+J\right),\\
\mu_{\mathrm{old}} &:= \alpha_t x_t - b_t\left(u_{\mathrm{old}}(x_t,t,s)+J\right),\\
\alpha_t &:= 1-\frac{\varsigma_t^2}{2t}\Delta t,\qquad b_t := \left(1+\frac{\varsigma_t^2(1-t)}{2t}\right)\Delta t,\qquad\sigma_k := \varsigma_t\sqrt{\Delta t}. \tag*{(95)}
\end{aligned} μ θ μ old α t := α t x t − b t ( u θ ( x t , t , s ) + J ) , := α t x t − b t ( u old ( x t , t , s ) + J ) , := 1 − 2 t ς t 2 Δ t , b t := ( 1 + 2 t ς t 2 ( 1 − t ) ) Δ t , σ k := ς t Δ t . (95)
Here Δ t = t − r \Delta t=t-r Δ t = t − r , α t \alpha_t α t and b t b_t b t are the Euler–Maruyama coefficients, σ k \sigma_k σ k is the kernel width, ς t \varsigma_t ς t is the noise scale of the scoring kernel, and both means use the same detached correction J = J [ u o l d , t , s ] J=J[u_{\mathrm{old}},t,s] J = J [ u old , t , s ] evaluated on the replay trajectory with lower endpoint s s s . The definition of ς t \varsigma_t ς t that avoids the singularity at t 0 = 1 t_0=1 t 0 = 1 , the trajectory correction and the SDE derivation are deferred to Section B.2.
Anchors and weights. The tempered (per-element-averaged) scoring ratio and its detached clipping gate are, explicitly,
ρ ˉ G R P O : = exp [ mean s p [ ( μ o l d − x r ) 2 ] − mean s p [ ( μ θ − x r ) 2 ] 2 σ k 2 ] , \bar{\rho}_{\mathrm{GRPO}} := \exp\left[\frac{\operatorname{mean}_{\mathrm{sp}}\left[(\mu_{\mathrm{old}}-x_r)^2\right]-\operatorname{mean}_{\mathrm{sp}}\left[(\mu_\theta-x_r)^2\right]}{2\sigma_k^2}\right], ρ ˉ GRPO := exp [ 2 σ k 2 mean sp [ ( μ old − x r ) 2 ] − mean sp [ ( μ θ − x r ) 2 ] ] ,
χ G R P O : = 1 [ ( A > 0 ∧ ρ ˉ G R P O ≤ 1 + ϵ c l i p ) ∨ ( A < 0 ∧ ρ ˉ G R P O ≥ 1 − ϵ c l i p ) ] . (96) \chi_{\mathrm{GRPO}} := \mathbf{1}\left[(A>0\wedge\bar{\rho}_{\mathrm{GRPO}}\le1+\epsilon_{\mathrm{clip}})\vee(A<0\wedge\bar{\rho}_{\mathrm{GRPO}}\ge1-\epsilon_{\mathrm{clip}})\right]. \tag*{(96)} χ GRPO := 1 [ ( A > 0 ∧ ρ ˉ GRPO ≤ 1 + ϵ clip ) ∨ ( A < 0 ∧ ρ ˉ GRPO ≥ 1 − ϵ clip ) ] . (96)
Thus χ G R P O = 0 \chi_{\mathrm{GRPO}}=0 χ GRPO = 0 when A = 0 A=0 A = 0 . The quantity ρ ˉ G R P O \bar{\rho}_{\mathrm{GRPO}} ρ ˉ GRPO is a surrogate scoring ratio rather than a rollout importance weight. The anchors and weights are
x ~ θ = μ θ , x ~ r e g = ∅ , x ~ b a s e = sg ( μ θ ) , x ~ r w d = x r , \tilde{x}_\theta=\mu_\theta,\qquad\tilde{x}_{\mathrm{reg}}=\varnothing,\qquad\tilde{x}_{\mathrm{base}}=\operatorname{sg}(\mu_\theta),\qquad\tilde{x}_{\mathrm{rwd}}=x_r, x ~ θ = μ θ , x ~ reg = ∅ , x ~ base = sg ( μ θ ) , x ~ rwd = x r ,
W r e g = 0 , W r w d = W = ρ ˉ G R P O , ω = χ G R P O A 2 σ k 2 , (97) W_{\mathrm{reg}}=0,\qquad W_{\mathrm{rwd}}=W=\bar{\rho}_{\mathrm{GRPO}},\qquad\omega=\frac{\chi_{\mathrm{GRPO}}A}{2\sigma_k^2}, \tag*{(97)} W reg = 0 , W rwd = W = ρ ˉ GRPO , ω = 2 σ k 2 χ GRPO A , (97)
so only μ θ \mu_\theta μ θ is differentiated. Sections B.2 and B.3 give the tempered interpretation and show that Equation (90) reproduces its clipped gradient. An auxiliary (v-aux) variant is not used because v ϕ v_\phi v ϕ is fitted on endpoint-renoised states rather than replayed sampler states (Section B.1).
Code 4: FewStepGRPO: the replayed transition
1 class FewStepGRPO(FewStepRLTrainer): # replaces Stage 1 and Stage 3(i)
2 estimator = ReplayVInd() # Eq. (109)
3 n_it = M_step
4
5 def rollout(self, prompts): # Eq. (94)
6 return sde_rollout(prompts, u_old, keep_transitions=True)
7 def queries(self, batch): # replay; one A per member
8 for x_t, x_r, t, r in batch.transitions: # r: target level of the step
9 s = draw_s(t) # s drawn as r in Stage 3(i)
10 yield Query(batch, t=t, r=r, s=s, x_t=x_t, x_r=x_r)
11 def anchors(self, q): # Eqs. (95), (97)
12 v_th, v_old, _ = self.estimator.fields(q, r=q.s) # the velocity at (t, s)
13 a, b, q.tau = em_coeffs(q.t, q.r) # α_t, b_t, σ_k of Eq. (95)
14 mu = a * q.x_t - b * v_th # x̃_θ = μ_θ, differentiated
15 q.mu_old = a * q.x_t - b * v_old # enters the ratio only
16 return Points(x_z=mu, x_reg=None, x_base=sg(mu), x_rwd=q.x_r)
17 def weights(self, q, p):
18 l_new = mean_sp((p.x_base - p.x_rwd) ** 2)
19 l_old = mean_sp((q.mu_old - p.x_rwd) ** 2)
20 rho_grpo = exp((l_old - l_new) / (2 * q.tau ** 2)) # ρ̄_GRPO
21 chi_grpo = clip_gate(rho_grpo, q.A, clip_range) # χ_GRPO
22 return 0., rho_grpo, chi_grpo * q.A / (2 * q.tau ** 2)
23 def kl(self, q): # same (x_t, t, s)
24 return mean_sp((u_theta(q.x_t, q.t, q.s)
25 - u_ref(q.x_t, q.t, q.s)) ** 2)
FewStepAWM: MeanFlowAWM and DMDAWM
FewStepAWM adapts Equation (17) as one clipped reward-augmentation term on the matching residual of the model. It has no explicit self-regression term, and in MeanFlowAWM the reward displacement is measured from the detached current prediction.
Rollout and prediction. Stage 1 uses the deterministic rollout, n i t = n t s n_{\mathrm{it}} = n_{\mathrm{ts}} n it = n ts , and every inner iteration draws a fresh ( t , r , ϵ ) (t,r,\epsilon) ( t , r , ϵ ) , re-noises the rewarded endpoint and computes the v-prediction Equation (86) ,
x ~ t = AddNoise ( x ^ , t ) , x ~ θ = x ~ θ , v = X [ x ~ t , u θ ( x ~ t , t , r ) + J [ u o l d , t , r ] , 0 ] , (98) \tilde{x}_t = \operatorname{AddNoise}(\hat{x},t), \qquad\tilde{x}_\theta= \tilde{x}_{\theta,v} = X\left[\tilde{x}_t,u_\theta(\tilde{x}_t,t,r)+J[u_{\mathrm{old}},t,r],0\right], \tag*{(98)} x ~ t = AddNoise ( x ^ , t ) , x ~ θ = x ~ θ , v = X [ x ~ t , u θ ( x ~ t , t , r ) + J [ u old , t , r ] , 0 ] , (98)
so that, by Equation (22) ,
x ~ θ − x ^ = − t ( u θ − ( v c o n d − J [ u o l d , t , r ] ) ) , v c o n d = ϵ − x ^ . (99) \tilde{x}_\theta-\hat{x}=-t\left(u_\theta-\left(v_{\mathrm{cond}}-J[u_{\mathrm{old}},t,r]\right)\right), \qquad v_{\mathrm{cond}}=\epsilon-\hat{x}. \tag*{(99)} x ~ θ − x ^ = − t ( u θ − ( v cond − J [ u old , t , r ] ) ) , v cond = ϵ − x ^ . (99)
Thus u θ u_\theta u θ regresses onto v c o n d − J v_{\mathrm{cond}}-J v cond − J , the MeanFlow target Equation (8) with the conditional velocity in place of the teacher.
MeanFlowAWM anchors and weights. The method uses the rollout endpoint as the rewarded anchor, no explicit self-regression, and the current prediction as baseline: x ~ r w d = x ^ \tilde{x}_{\mathrm{rwd}}=\hat{x} x ~ rwd = x ^ , x ~ r e g = ∅ \tilde{x}_{\mathrm{reg}}=\varnothing x ~ reg = ∅ , x ~ b a s e = sg ( x ~ θ ) \tilde{x}_{\mathrm{base}}=\operatorname{sg}(\tilde{x}_\theta) x ~ base = sg ( x ~ θ ) . Besides the shared J J J in x ~ θ \tilde{x}_\theta x ~ θ , the old policy enters only through the weights, via the shared-J J J prediction of Stage 3 (iii),
x ~ i n d = X [ x ~ t , u o l d ( x ~ t , t , r ) + J [ u o l d , t , r ] , 0 ] , x ~ θ − x ~ i n d = − t ( u θ − u o l d ) . (100) \tilde{x}_{\mathrm{ind}}=X\left[\tilde{x}_t,u_{\mathrm{old}}(\tilde{x}_t,t,r)+J[u_{\mathrm{old}},t,r],0\right], \qquad\tilde{x}_\theta-\tilde{x}_{\mathrm{ind}}=-t(u_\theta-u_{\mathrm{old}}). \tag*{(100)} x ~ ind = X [ x ~ t , u old ( x ~ t , t , r ) + J [ u old , t , r ] , 0 ] , x ~ θ − x ~ ind = − t ( u θ − u old ) . (100)
The weights reuse the quadratic-exponential form of § 5.3 , but not its probabilistic interpretation:
ρ A W M : = exp [ mean s p [ ( x ~ i n d − x ^ ) 2 ] − mean s p [ ( x ~ θ − x ^ ) 2 ] 2 τ A W M 2 ] , χ A W M : = 1 [ ( A > 0 ∧ ρ A W M ≤ 1 + ϵ c l i p ) ∨ ( A < 0 ∧ ρ A W M ≥ 1 − ϵ c l i p ) ] . (101) \begin{aligned}
\rho_{\mathrm{AWM}} &:= \exp\left[\frac{\operatorname{mean}_{\mathrm{sp}}\left[(\tilde{x}_{\mathrm{ind}}-\hat{x})^2\right]-\operatorname{mean}_{\mathrm{sp}}\left[(\tilde{x}_{\theta}-\hat{x})^2\right]}{2\tau_{\mathrm{AWM}}^2}\right],\\
\chi_{\mathrm{AWM}} &:= 1\left[\left(A>0\ \land\ \rho_{\mathrm{AWM}}\le1+\epsilon_{\mathrm{clip}}\right)\lor\left(A<0\ \land\ \rho_{\mathrm{AWM}}\ge1-\epsilon_{\mathrm{clip}}\right)\right]. \tag*{(101)}
\end{aligned} ρ AWM χ AWM := exp [ 2 τ AWM 2 mean sp [ ( x ~ ind − x ^ ) 2 ] − mean sp [ ( x ~ θ − x ^ ) 2 ] ] , := 1 [ ( A > 0 ∧ ρ AWM ≤ 1 + ϵ clip ) ∨ ( A < 0 ∧ ρ AWM ≥ 1 − ϵ clip ) ] . (101)
Thus χ A W M = 0 \chi_{\mathrm{AWM}}=0 χ AWM = 0 when A = 0 A=0 A = 0 . The implemented residual scale is τ A W M = max { t , τ min } \tau_{\mathrm{AWM}}=\max\{t,\tau_{\min}\} τ AWM = max { t , τ m i n } ; unlike in FewStepGRPO, no sampler kernel determines this scale. The choice τ A W M = t \tau_{\mathrm{AWM}}=t τ AWM = t would cancel the factor t 2 t^2 t 2 in both residuals; when the floor is active, the factor t 2 / τ A W M 2 t^2/\tau_{\mathrm{AWM}}^2 t 2 / τ AWM 2 remains:
∂ L ∂ u θ = ρ A W M χ A W M A N t 2 τ A W M 2 ( u θ − ( v c o n d − J [ u o l d , t , r ] ) ) (102) \frac{\partial\mathcal{L}}{\partial u_{\theta}}=\frac{\rho_{\mathrm{AWM}}\chi_{\mathrm{AWM}}A}{N}\frac{t^2}{\tau_{\mathrm{AWM}}^2}\left(u_{\theta}-\left(v_{\mathrm{cond}}-J\left[u_{\mathrm{old}},t,r\right]\right)\right) \tag*{(102)} ∂ u θ ∂ L = N ρ AWM χ AWM A τ AWM 2 t 2 ( u θ − ( v cond − J [ u old , t , r ] ) ) (102)
and its factor t 2 / τ A W M 2 t^2/\tau_{\mathrm{AWM}}^2 t 2 / τ AWM 2 equals one only above the floor. The resulting quantities are
x ~ r e g = ∅ , x ~ b a s e = sg ( x ~ θ ) , x ~ r w d = x ^ , W r e g = 0 , W r w d = W = ρ A W M , ω = χ A W M A 2 τ A W M 2 . (103) \tilde{x}_{\mathrm{reg}}=\varnothing,\quad\tilde{x}_{\mathrm{base}}=\operatorname{sg}(\tilde{x}_{\theta}),\quad\tilde{x}_{\mathrm{rwd}}=\hat{x},\quad W_{\mathrm{reg}}=0,\quad W_{\mathrm{rwd}}=W=\rho_{\mathrm{AWM}},\quad\omega=\frac{\chi_{\mathrm{AWM}}A}{2\tau_{\mathrm{AWM}}^2}. \tag*{(103)} x ~ reg = ∅ , x ~ base = sg ( x ~ θ ) , x ~ rwd = x ^ , W reg = 0 , W rwd = W = ρ AWM , ω = 2 τ AWM 2 χ AWM A . (103)
Relative to Equation (32) with w ( t ) = 1 w(t)=1 w ( t ) = 1 , the step has an extra factor 1 2 \frac{1}{2} 2 1 , which comes from the Gaussian form of the statistic and is absorbed by the learning rate. ρ A W M \rho_{\mathrm{AWM}} ρ AWM is not an importance weight, since ( x ~ t , x ^ ) (\tilde{x}_t,\hat{x}) ( x ~ t , x ^ ) is not a transition of the rollout; through the gate χ A W M \chi_{\mathrm{AWM}} χ AWM , it limits one-sided changes of the matching residual relative to u o l d u_{\mathrm{old}} u old . We call this variant MeanFlowAWM .
DMDAWM variant. The reported v-aux variant, DMDAWM , drops J J J in the current and old-policy predictions (the J J J -free ( t , r ) (t,r) ( t , r ) query and x ~ i n d ∣ J = 0 \tilde{x}_{\mathrm{ind}\mid J=0} x ~ ind ∣ J = 0 of § 4.3 ) and replaces the reward baseline with the prediction of the auxiliary network:
x ~ θ = X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] , x ~ i n d ∣ J = 0 = X [ x ~ t , u o l d ( x ~ t , t , r ) , 0 ] , x ~ b a s e = x ~ a u x , ρ A W M = exp [ mean s p [ ( x ~ i n d ∣ J = 0 − x ^ ) 2 ] − mean s p [ ( x ~ θ − x ^ ) 2 ] 2 τ A W M 2 ] , ∂ L ∂ u θ = ρ A W M χ A W M A N t 2 τ A W M 2 ( v ϕ − v c o n d ) . (104) \begin{aligned}
\tilde{x}_{\theta} &= X\left[\tilde{x}_t,u_{\theta}(\tilde{x}_t,t,r),0\right],\qquad\tilde{x}_{\mathrm{ind}\mid J=0}=X\left[\tilde{x}_t,u_{\mathrm{old}}(\tilde{x}_t,t,r),0\right],\qquad\tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{aux}},\\
\rho_{\mathrm{AWM}} &= \exp\left[\frac{\operatorname{mean}_{\mathrm{sp}}\left[(\tilde{x}_{\mathrm{ind}\mid J=0}-\hat{x})^2\right]-\operatorname{mean}_{\mathrm{sp}}\left[(\tilde{x}_{\theta}-\hat{x})^2\right]}{2\tau_{\mathrm{AWM}}^2}\right],\\
\frac{\partial\mathcal{L}}{\partial u_{\theta}} &= \frac{\rho_{\mathrm{AWM}}\chi_{\mathrm{AWM}}A}{N}\frac{t^2}{\tau_{\mathrm{AWM}}^2}(v_{\phi}-v_{\mathrm{cond}}). \tag*{(104)}
\end{aligned} x ~ θ ρ AWM ∂ u θ ∂ L = X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] , x ~ ind ∣ J = 0 = X [ x ~ t , u old ( x ~ t , t , r ) , 0 ] , x ~ base = x ~ aux , = exp [ 2 τ AWM 2 mean sp [ ( x ~ ind ∣ J = 0 − x ^ ) 2 ] − mean sp [ ( x ~ θ − x ^ ) 2 ] ] , = N ρ AWM χ AWM A τ AWM 2 t 2 ( v ϕ − v cond ) . (104)
Above the floor the factor t 2 / τ A W M 2 t^2/\tau_{\mathrm{AWM}}^2 t 2 / τ AWM 2 equals one; descent moves u θ u_{\theta} u θ along + A ( v c o n d − v ϕ ) +A(v_{\mathrm{cond}}-v_{\phi}) + A ( v cond − v ϕ ) , with v ϕ v_{\phi} v ϕ as a learned per-sample baseline. If a detached J J J were kept in the first two predictions, it would change only the scalar ratio ρ A W M \rho_{\mathrm{AWM}} ρ AWM and the clipping gate, but not the gradient direction. The reported method omits it, so it requires no directional-difference evaluation of J J J .
Decoupled loss. Both variants use Equation (90) with W r e g = 0 W_{\mathrm{reg}}=0 W reg = 0 , so only L r w d \mathcal{L}_{\mathrm{rwd}} L rwd remains. MeanFlowAWM measures the reward displacement from sg ( x ~ θ ) \operatorname{sg}(\tilde{x}_{\theta}) sg ( x ~ θ ) ; DMDAWM measures it from the auxiliary prediction x ~ a u x \tilde{x}_{\mathrm{aux}} x ~ aux . In code, FewStepAWM implements anchors and weights, and MeanFlowAWM and DMDAWM differ only in their estimator (Code 5).
class FewStepAWM(FewStepRLTrainer): # no self-regression; x_rwd = x̂
def anchors(self, q):
v_th, v_old, v_b = self.estimator.fields(q)
x_z = X(q.x_t, v_th, 0) # Eq. (98)
q.x_old = X(q.x_t, v_old, 0) # Eq. (100): statistic only
# baseline: the current prediction, or the prediction of vϕ, Eq. (104)
x_base = sg(x_z) if v_b is None else X(q.x_t, v_b, 0)
return Points(x_z=x_z, x_reg=None, x_base=x_base, x_rwd=q.x_hat)
def weights(self, q, p): # Eq. (103)
tau = clamp(q.t, min=TAU_MIN) # τAWM = max(t, τmin)
l_new = mean_sp((sg(p.x_z) - p.x_rwd) ** 2)
l_old = mean_sp((q.x_old - p.x_rwd) ** 2)
rho_awm = exp((l_old - l_new) / (2 * tau ** 2)) # ρAWM
chi_awm = clip_gate(rho_awm, q.A, clip_range) # χAWM
return 0., rho_awm, chi_awm * q.A / (2 * tau ** 2)
class MeanFlowAWM(FewStepAWM): estimator = VInd() # shared J
class DMDAWM(FewStepAWM): estimator = VAux() # vϕ baseline, no J
FewStepNFT: MeanFlowNFT and DMDNFT
FewStepNFT maps the two implicit branches of DiffusionNFT to the two terms of Equation (90) . It contains both a self-regression term and a signed reward-augmentation term, with x ~ r w d = x ^ \tilde{x}_{\mathrm{rwd}}=\hat{x} x ~ rwd = x ^ in both variants.
Anchors, predictions and weights. For the two variants, the quantities of Equation (35) are listed in Table 3 .
Table 3. Prediction and anchors of the two FewStepNFT variants; J = J [ u o l d , t , r ] J=J[u_{\mathrm{old}},t,r] J = J [ u old , t , r ] .
The theoretical weights of Equation (35) are modified in the implementation, following its official implementation in Zheng et al. (2026a). The derivation of §4.2 gives W r e g = W r w d = β 2 / t 2 W_{\mathrm{reg}}=W_{\mathrm{rwd}}=\beta^2/t^2 W reg = W rwd = β 2 / t 2 and ω = clip ( A , − 1 , 1 ) / β \omega=\operatorname{clip}(A,-1,1)/\beta ω = clip ( A , − 1 , 1 ) / β , i.e. one fixed per-t t t scale for both branches, with r ^ \hat{r} r ^ from Equation (18) and A A A normalised within the group. The implementation keeps the two branches but changes the following: the per-t t t scale is replaced by one adaptive normaliser per branch, γ ± \gamma_{\pm} γ ± , the mean absolute residual of that branch; r ^ \hat{r} r ^ is the compressed weight Equation (84) with clip range c a d v c_{\mathrm{adv}} c adv ; and A A A is the advantage Equation (83) . The weights are then
γ ± = max { mean s p [ ∣ x ~ b a s e ± β ( sg ( x ~ θ ) − x ~ r e g ) − x ^ ∣ ] , ε γ } , a + = r ^ γ + , a − = 1 − r ^ γ − . (105) \gamma_{\pm}=\max\left\{\operatorname{mean}_{\mathrm{sp}}\left[\left|\tilde{x}_{\mathrm{base}}\pm\beta\left(\operatorname{sg}(\tilde{x}_{\theta})-\tilde{x}_{\mathrm{reg}}\right)-\hat{x}\right|\right],\varepsilon_{\gamma}\right\},\qquad a_{+}=\frac{\hat{r}}{\gamma_{+}},\qquad a_{-}=\frac{1-\hat{r}}{\gamma_{-}}. \tag*{(105)} γ ± = max { mean sp [ ∣ x ~ base ± β ( sg ( x ~ θ ) − x ~ reg ) − x ^ ∣ ] , ε γ } , a + = γ + r ^ , a − = γ − 1 − r ^ . (105)
W r e g = W r w d = W = c a d v β 2 ( a + + a − ) , ω = a + − a − β ( a + + a − ) . (106) W_{\mathrm{reg}}=W_{\mathrm{rwd}}=W=c_{\mathrm{adv}}\beta^2(a_{+}+a_{-}),\qquad\omega=\frac{a_{+}-a_{-}}{\beta(a_{+}+a_{-})}. \tag*{(106)} W reg = W rwd = W = c adv β 2 ( a + + a − ) , ω = β ( a + + a − ) a + − a − . (106)
If γ + = γ − = t 2 \gamma_{+} = \gamma_{-} = t^{2} γ + = γ − = t 2 , c a d v = 1 c_{\mathrm{adv}} = 1 c adv = 1 and A A A is the group-normalised advantage Equation (12) , they reduce to Equation (35) , with W = β 2 / t 2 W = \beta^{2}/t^{2} W = β 2 / t 2 and ω = clip ( A , − 1 , 1 ) / β \omega= \operatorname{clip}(A,-1,1)/\beta ω = clip ( A , − 1 , 1 ) / β . In the implemented weighting, r ^ \hat{r} r ^ enters through a ± a_{\pm} a ± as a mixing weight, c a d v c_{\mathrm{adv}} c adv scales both loss terms to undo the compression of the advantage by 1 / ( 2 c a d v ) 1/(2c_{\mathrm{adv}}) 1/ ( 2 c adv ) in Equation (84) , so that the effective step does not shrink when the clip range is widened (Section B.3), and ω \omega ω is defined directly. The v-ind estimator gives MeanFlowNFT , and the v-aux estimator gives DMDNFT .
Decoupled loss. Both variants minimise L r e g + L r w d \mathcal{L}_{\mathrm{reg}} + \mathcal{L}_{\mathrm{rwd}} L reg + L rwd with the common weight W W W , the rollout endpoint as x ~ r w d \tilde{x}_{\mathrm{rwd}} x ~ rwd , and Table 3 defining x ~ r e g \tilde{x}_{\mathrm{reg}} x ~ reg and x ~ b a s e \tilde{x}_{\mathrm{base}} x ~ base .
Code 6: FewStepNFT: anchors and weights
1 class FewStepNFT(FewStepRLTrainer): # both squares on; x̃_rwd = x̂
2 def anchors(self, q):
3 v_th, v_old, v_b = self.estimator.fields(q)
4 x_z, x_reg = X(q.x_t, v_th, 0), X(q.x_t, v_old, 0) # same (t, r), same J
5 # baseline: aliased to x̃_reg, or the prediction of v_ϕ, Eq. (60)
6 x_base = x_reg if v_b is None else X(q.x_t, v_b, 0)
7 return Points(x_z=x_z, x_reg=x_reg, x_base=x_base, x_rwd=q.x_hat)
8 def branch_weights(self, q, p, x_pos, x_neg): # Eq. (105)
9 dev = sg(p.x_z) - p.x_reg # direction from x̃_reg, centre x̃_base
10 g_pos = clamp(mean_sp(abs(p.x_base + beta * dev - x_pos)), min=eps_gamma)
11 g_neg = clamp(mean_sp(abs(p.x_base - beta * dev - x_neg)), min=eps_gamma)
12 return q.r_hat / g_pos, (1 - q.r_hat) / g_neg # r̂ enters ONLY here
13 def weights(self, q, p): # Eq. (106), not (35)
14 a_pos, a_neg = self.branch_weights(q, p, p.x_rwd, p.x_rwd)
15 W = c_adv * beta ** 2 * (a_pos + a_neg)
16 return W, W, (a_pos - a_neg) / (beta * (a_pos + a_neg))
18 class MeanFlowNFT(FewStepNFT): estimator = VInd() # x̃_θ = x̃_{θ,v}, x̃_reg = x̃_base = x̃_ind
19 class DMDNFT(FewStepNFT): estimator = VAux() # x̃_base = x̃_aux ≠ x̃_reg
FewStepOPSD: Extended Reward Augmentation and ReFL
FewStepOPSD belongs to the family of reward-gradient-based methods ([11 ] (Xu et al. 2023; Wang et al. 2026)) and is related to gradient-guided distribution matching ([14 ] ). It implements Example 8 (iii) as a FewStepNFT trainer whose rewarded points are constructed from the reward gradient instead of sampled; it therefore uses the normalisers of Equation (105) , and its configuration with a single ascent step is the ReFL proxy of Example 9. It modifies more stages than FewStepNFT. Stage 1 depends on the reward, the rewarded anchors are constructed points instead of rollout samples, and Stage 4 overrides loss. Stage 2 is unchanged, and r ^ \hat{r} r ^ enters only through the weights.
Rollout. FewStepOPSD uses the reward-coupled rollout Equation (82) . It first draws the i.i.d. base samples with Equation (81) . The reward-coupled construction then applies Equation (62) , taking M t g t M_{\mathrm{tgt}} M tgt normalised reward-gradient steps in each direction from the r = 0 r = 0 r = 0 prediction x ~ o l d , u \tilde{x}_{\mathrm{old},u} x ~ old , u , with the displacement transferred to the anchor by Equation (70) , giving x ~ ± \tilde{x}_{\pm} x ~ ± . The points x ~ ± \tilde{x}_{\pm} x ~ ± are detached, so the computation graph of the reward ends here and the policy update does not back-propagate through the decoder or the reward model; the reward must be differentiable.
Anchors and weights. (i) Query. Each member is queried at a single level t t t , drawn from the law given in § 6.1 ; hence n i t = 1 n_{it}=1 n i t = 1 . As in Example 8 and Example 9, the query state is the re-noised endpoint x ~ t = AddNoise ( x ^ , t ) \tilde{x}_t = \operatorname{AddNoise}(\hat{x},t) x ~ t = AddNoise ( x ^ , t ) , so the anchor is evaluated at the same state as the current prediction. The drawn r r r is stored and reused at fitting time, so the current prediction and the anchor are queried at the same ( t , r ) (t,r) ( t , r ) .
(ii) Anchors. The u-form uses x ~ r e g = x ~ b a s e = x ~ o l d , u \tilde{x}_{\mathrm{reg}} = \tilde{x}_{\mathrm{base}} = \tilde{x}_{\mathrm{old,u}} x ~ reg = x ~ base = x ~ old , u and the v-form uses x ~ r e g = x ~ b a s e = x ~ i n d \tilde{x}_{\mathrm{reg}} = \tilde{x}_{\mathrm{base}} = \tilde{x}_{\mathrm{ind}} x ~ reg = x ~ base = x ~ ind , Equation (69) ; in both, the rewarded anchors are x ~ ± \tilde{x}_{\pm} x ~ ± . For the v-form, the gradient steps start from the r = 0 r=0 r = 0 prediction x ~ o l d , u \tilde{x}_{\mathrm{old,u}} x ~ old , u , which does not depend on J J J , and only the resulting displacement is added to the anchor x ~ i n d \tilde{x}_{\mathrm{ind}} x ~ ind by Equation (70) .
(iii) Instantaneous velocity. The v-form uses the shared J J J of Stage 3 (iii) . Because both rewarded points are constructed from the anchor, J J J cancels in every difference in the loss, as in Equation (71) and Equation (72) . The v-form reported in §6 drops J J J and evaluates no directional difference (Equation (87) ); in code its estimator is VInd(cd=False).
(iv) Weights. With Equation (105) evaluated at x ~ ± \tilde{x}_{\pm} x ~ ± ,
W r e g = W r w d = W = c a d v β 2 ( a + + a − ) , W_{\mathrm{reg}} = W_{\mathrm{rwd}} = W = c_{\mathrm{adv}}\beta^2(a_{+}+a_{-}), W reg = W rwd = W = c adv β 2 ( a + + a − ) ,
ω 1 = a + β ( a + + a − ) , ω 2 = − a − β ( a + + a − ) , ∣ ω 1 ∣ + ∣ ω 2 ∣ = 1 β , (107) \omega_1 = \frac{a_{+}}{\beta(a_{+}+a_{-})}, \qquad\omega_2 = -\frac{a_{-}}{\beta(a_{+}+a_{-})}, \qquad|\omega_1|+|\omega_2| = \frac{1}{\beta}, \tag*{(107)} ω 1 = β ( a + + a − ) a + , ω 2 = − β ( a + + a − ) a − , ∣ ω 1 ∣ + ∣ ω 2 ∣ = β 1 , (107)
so the sum of coefficient magnitudes is fixed at 1 / β 1/\beta 1/ β and r ^ \hat{r} r ^ redistributes it between the two terms. It can still change the common weight W W W when γ + ≠ γ − \gamma_{+} \ne\gamma_{-} γ + = γ − . The factor c a d v c_{\mathrm{adv}} c adv scales the common weight but not the split between the branches.
Decoupled loss. The loss Equation (68) consists of one self-regression term and two reward-augmentation terms, which share the baseline and differ in the rewarded point x ~ r w d , k \tilde{x}_{\mathrm{rwd},k} x ~ rwd , k and the coefficient ω k \omega_k ω k ; the signs of ω k \omega_k ω k are fixed by construction and do not depend on A A A .
Code 7: FewStepOPSD: reward-coupled construction and loss
1 def tr_steps(x0, c, direction, M_tgt): # Eq. (62)
2 radius = kappa * norm(x0); h = eta * radius / M_tgt
3 x = x0
4 for _ in range(M_tgt):
5 grad_R = grad(reward(decode(x), c), x) # through reward model AND decoder
6 x = x + direction * h * grad_R / (norm(grad_R) + eps_g)
7 d = x - x0 # trust region on the displacement
8 x = x0 + d * clamp(radius / (norm(d) + eps_g), max=1.)
9 return sg(x)
10
11 class FewStepOPSD(FewStepNFT): # NFT onto constructed points
12 estimator = VInd(cd=False) # J cancels in the loss
13 M_tgt, directions = 2, (+1, -1)
14 def rollout(self, prompts): # Eq. (82)
15 b = super().rollout(prompts)
16 b.t, b.r = draw_t_logitnormal(b), draw_r(b) # stored, one per member
17 b.x_t = add_noise(b.x_hat, b.t)
18 x_old = X(b.x_t, u_old(b.x_t, b.t, b.r), 0) # the anchor
19 shift = X(b.x_t, u_old(b.x_t, b.t, 0.), 0) - x_old # Eq. (70)
20 b.x_pm = [tr_steps(x_old + shift, b.c, d, self.M_tgt) - shift
21 for d in self.directions] # detached
22 return b
23 def queries(self, batch): # n_it = 1, stored (t, r)
24 yield Query(batch, t=batch.t, r=batch.r, x_t=batch.x_t)
25 def anchors(self, q): # Eq. (69)
26 p = super().anchors(q) # x̃_reg = x̃_base: old-policy prediction
27 return Points(x_z=p.x_z, x_reg=p.x_reg, x_base=p.x_base, x_rwd=q.x_pm)
28 def weights(self, q, p): # Eq. (107)
29 a_pos, a_neg = self.branch_weights(q, p, *p.x_rwd) # γ± at x̃±
30 W = c_adv * beta ** 2 * (a_pos + a_neg)
31 s = beta * (a_pos + a_neg)
32 return W, W, (a_pos / s, -a_neg / s) # |ω₁| + |ω₂| = 1/β
33 def loss(self, p, W_reg, W_rwd, omega): # override, Eq. (68)
34 (x_pos, x_neg), (om_1, om_2) = p.x_rwd, omega
35 l_reg = self_regression(p.x_z, p.x_reg, W_reg)
36 l_pos = reward_augmentation(p.x_z, x_pos, p.x_base, om_1, W_rwd) # positive
↪ branch
37 l_neg = reward_augmentation(p.x_z, x_neg, p.x_base, om_2, W_rwd) # negative
↪ branch
38 return l_reg + l_pos + l_neg
ReFL. The ReFL configuration of Equation (68) , identified in Example 9 , reuses the stages above, with the changes below. Stage 1 takes a single ascent step, M t g t = 1 M_{\mathrm{tgt}} = 1 M tgt = 1 , without a descent point. As for FewStepOPSD, the step starts at the r = 0 r = 0 r = 0 prediction and its displacement is added back by Equation (70) , so ∇ R \nabla R ∇ R is evaluated at x ~ o l d , u \tilde{x}_{\mathrm{old,u}} x ~ old , u as specified by Equation (78) . Stage 2 is not used, because ReFL has no advantage; ω 1 > 0 \omega_1 > 0 ω 1 > 0 is fixed by construction and the implementation sets r ^ ≡ 1 \hat{r} \equiv1 r ^ ≡ 1 in Equation (107) , giving ω 1 = 1 / β \omega_1 = 1/\beta ω 1 = 1/ β and W r w d = c a d v β 2 / γ + W_{\mathrm{rwd}} = c_{\mathrm{adv}}\beta^2/\gamma_+ W rwd = c adv β 2 / γ + , detached and positive as Example 9 allows. The ReFL override keeps the factor c a d v c_{\mathrm{adv}} c adv of Equation (107) . Stage 3 keeps the same anchors. Stage 4 keeps only L r w d , 1 \mathcal{L}_{\mathrm{rwd},1} L rwd , 1 with W r e g = 0 W_{\mathrm{reg}} = 0 W reg = 0 , i.e. the loss of Code 7 reduces to its ascent term (Code 8). The resulting gradient is that of Equation (79) , the few-step form of Equation (77) .
class ReFL(FewStepOPSD):
M_tgt, directions = 1, (+1,)
def advantage(self, R, c):
return None, ones_like(R)
def weights(self, q, p):
a_pos, _ = self.branch_weights(q, p, p.x_rwd[0], p.x_rwd[0])
return 0., c_adv * beta ** 2 * a_pos, (1. / beta,)
def loss(self, p, W_reg, W_rwd, omega):
return reward_augmentation(p.x_z, p.x_rwd[0], p.x_base, omega[0], W_rwd)
Cost of an Outer Iteration
Table 4 lists the network evaluations per inner iteration for each network. The rollout of Stage 1 is amortised over one outer iteration and is not counted here. Each entry is determined by the family and the estimator of § 5.1 : n i t n_{\mathrm{it}} n it by queries, c J c_J c J by the estimator (2 for the directional-difference estimate of v-ind, and 0 otherwise), and c ϕ c_\phi c ϕ and the auxiliary part of c e x t r a c_{\mathrm{extra}} c extra by v-aux; the reward-coupled construction of FewStepOPSD and ReFL is counted in c e x t r a c_{\mathrm{extra}} c extra . The counters c θ c_\theta c θ , c o l d c_{\mathrm{old}} c old , c J c_J c J , c ϕ c_\phi c ϕ , c r e f c_{\mathrm{ref}} c ref , c e x t r a c_{\mathrm{extra}} c extra , C C C and C r o l l o u t C_{\mathrm{rollout}} C rollout are defined in the caption and listed in Appendix A.
Table 4. Network evaluations per inner iteration. n i t n_{\mathrm{it}} n it : number of inner iterations per outer iteration, each querying every member once; c θ c_\theta c θ : forward–backward passes of u θ u_\theta u θ ; c o l d c_{\mathrm{old}} c old : no-grad forwards of u o l d u_{\mathrm{old}} u old for the old-policy anchor; c J c_J c J : maximum extra no-grad forwards of u o l d u_{\mathrm{old}} u old for the directional-difference estimate of J J J (Equation (87) or Equation (109) ; 0 where the estimator uses no J J J ; †with J J J dropped because it cancels, which leaves the loss and its gradient unchanged by Appendix § 11 ); c ϕ c_\phi c ϕ : forwards with no gradient of the auxiliary network v ϕ v_\phi v ϕ ; c r e f c_{\mathrm{ref}} c ref : forwards of the frozen reference for the velocity penalty; c e x t r a c_{\mathrm{extra}} c extra : extra cost per outer iteration, where n f m n_{\mathrm{fm}} n fm is the number of flow-matching steps that fit v ϕ v_\phi v ϕ . The cost of one outer iteration is thus C = C r o l l o u t + c e x t r a + n i t ( c θ + c o l d + c J + c ϕ + c r e f ) C = C_{\mathrm{rollout}} + c_{\mathrm{extra}} + n_{\mathrm{it}}(c_\theta+ c_{\mathrm{old}} + c_J + c_\phi+ c_{\mathrm{ref}}) C = C rollout + c extra + n it ( c θ + c old + c J + c ϕ + c ref ) , where C r o l l o u t C_{\mathrm{rollout}} C rollout is the plain rollout of Stage 1.
Experiment
Having established a general view of these methods, we use experiments to answer three questions. Every trainer of §§5.3–5.6 implements one of these methods, minimises the same loss form, and differs mainly in its anchors and weights. (i) Which of these choices, in particular the reward signal and the estimator of the instantaneous velocity, determine how effectively and how efficiently a few-step generator is fine-tuned, and how strongly it exploits the reward? (ii) Do these conclusions carry
caption: The methods compared; ReFL is Example 9 . Class: the first five methods compute scalar rewards for the members of each prompt group and form per-sample advantages using the per-prompt mean and current global-batch standard deviation of Equation (83) , so they are group-advantage-based. The final two obtain their clean-sample update direction from ∇ R \nabla R ∇ R and are reward-gradient-based; FewStepOPSD still uses r ^ \hat{r} r ^ only to split the ascent/descent weights. Reward signal: the advantage as a signed multiplier (A A A ), as a mixing weight (r ^ \hat{r} r ^ , Stage 2), or the reward gradient (∇ R \nabla R ∇ R , Equation (62) ). Every method except FewStepGRPO uses the deterministic sampler of Equation (5) in Stage 1. Appendix § 11 establishes when dropping J J J (last column) is admissible.
:::
over to a different modality, or do they show limitations in other scenarios? (iii) Does a generator fine-tuned at a few steps improve when it is sampled with more steps, without retraining?
Setup on Image Generation
Generator and sampler. The policy is the average-velocity network u θ ( x ~ t , t , r ) u_\theta(\tilde{x}_t,t,r) u θ ( x ~ t , t , r ) of §3, initialised from the pretrained SD3.5-Medium backbone (Stability AI, (2024); Esser et al. [15 ] ) and distilled with AnyFlow (Gu et al., [26 ] ), which first trains with the MeanFlow identity Equation (6) and then applies on-policy distribution matching Equation (10) against the frozen teacher. Reinforcement learning trains a LoRA (Hu et al., [32 ] ) of rank 32 (scaling 64) on the attention projections, at resolution 512 × 512 512 \times512 512 × 512 . We set M s t e p = 4 M_{\mathrm{step}}=4 M step = 4 steps on the grid of levels { 1 , 0.9 , 0.75 , 0.5 , 0 } \{1,0.9,0.75,0.5,0\} { 1 , 0.9 , 0.75 , 0.5 , 0 } , without classifier-free guidance (Ho and Salimans, [30 ] ), with the deterministic sampler of Equation (5) for every trainer except FewStepGRPO, which requires the stochastic sampler of Equation (94) . Evaluation always uses the deterministic sampler on the same grid.
Optimisation. In Stage 1 and Stage 2, each outer iteration draws 48 prompts with K = 24 K=24 K = 24 members each, i.e. 1152 rollouts, on 8 NVIDIA H20 GPUs. All methods use AdamW (Loshchilov and Hutter, 2019) with learning rate 3 × 10 − 6 3 \times10^{-6} 3 × 1 0 − 6 , weight decay 10 − 4 10^{-4} 1 0 − 4 and gradient-norm clipping at 1; an old policy u o l d u_{\mathrm{old}} u old given by an EMA of u θ u_\theta u θ , refreshed once per outer iteration after the policy update with the decay schedule of the official DiffusionNFT implementation (Section D.1); for the NFT/OPSD families, β = 0.1 \beta=0.1 β = 0.1 and c a d v = 5 c_{\mathrm{adv}}=5 c adv = 5 in Equation (84) ; and the KL-u u u penalty against the fixed AnyFlow initial policy, using Equation (92) with weight 10 − 4 10^{-4} 1 0 − 4 . An ablation with the penalty Equation (93) , labelled KL-v v v , on MeanFlowNFT is given in Section D.4. Stage 3 (i) draws t t t and r r r as the maximum and the minimum of two uniforms before the time shift of the SD3 scheduler, t ↦ 3 t / ( 1 + 2 t ) t \mapsto3t/(1+2t) t ↦ 3 t / ( 1 + 2 t ) (shift 3.0, the same shifted grid as in sampling), sets half of each batch to r = t r=t r = t and a quarter to r = 0 r=0 r = 0 , and uses n t s = 4 n_{\mathrm{ts}}=4 n ts = 4 queries per member and outer iteration. The floors are τ min = 10 − 2 \tau_{\min}=10^{-2} τ m i n = 1 0 − 2 , ε γ = 10 − 5 \varepsilon_\gamma=10^{-5} ε γ = 1 0 − 5 , ε g = 10 − 12 \varepsilon_g=10^{-12} ε g = 1 0 − 12 and ε s t d = 10 − 4 \varepsilon_{\mathrm{std}}=10^{-4} ε std = 1 0 − 4 , and ε c d = 5 × 10 − 3 \varepsilon_{\mathrm{cd}}=5 \times10^{-3} ε cd = 5 × 1 0 − 3 (Section D.1).
Rewards. We use two reward settings to test the methods. The first is the OCR setting, where the reward is the OCR accuracy of the text rendered in the image with the OCR prompts. It has no gradient with respect to the clean sample, so FewStepOPSD and ReFL, which require ∇ R \nabla R ∇ R , do not apply. In the mixed-reward setting the reward is the unweighted sum of CLIPScore, HPSv2 and PickScore on the PickScore prompt set [29 ] (Wu et al. 2023; Kirstain et al. 2023).
Specific settings. For the group-advantage-based methods, n i t = 4 n_{\mathrm{it}}=4 n it = 4 , with n i t = M s t e p n_{\mathrm{it}}=M_{\mathrm{step}} n it = M step for FewStepGRPO and n i t = n t s n_{\mathrm{it}}=n_{\mathrm{ts}} n it = n ts for the others. FewStepGRPO retains the states of Equation (94) and uses the base noise scale ζ = 0.25 \zeta=0.25 ζ = 0.25 of ζ t \zeta_t ζ t in Equation (95) (Equation (110) ), with clipping radius ε c l i p = 4 × 10 − 3 \varepsilon_{\mathrm{clip}}=4\times10^{-3} ε clip = 4 × 1 0 − 3 (OCR) or 3 × 10 − 3 3\times10^{-3} 3 × 1 0 − 3 (mixed reward) in χ G R P O \chi_{\mathrm{GRPO}} χ GRPO ; MeanFlowAWM and DMDAWM use radius 10 − 2 10^{-2} 1 0 − 2 (OCR) or 2 × 10 − 3 2\times10^{-3} 2 × 1 0 − 3 (mixed reward) in χ A W M \chi_{\mathrm{AWM}} χ AWM . For DMDAWM and DMDNFT, the auxiliary velocity v ϕ v_\phi v ϕ is fitted by Equation (88) with learning rate 10 − 5 10^{-5} 1 0 − 5 and n f m = 5 n_{\mathrm{fm}}=5 n fm = 5 steps per outer iteration. FewStepOPSD uses κ = 0.1 \kappa=0.1 κ = 0.1 , M t g t = 2 M_{\mathrm{tgt}}=2 M tgt = 2 and η = 1 \eta=1 η = 1 in Equation (62) , with ∇ R \nabla R ∇ R of the full training reward, drawing t t t from a logit-normal law with median 0.278 (5–95% range [0.125, 0.508]), i.e., the small noise levels of Example 8 , and r r r as in Stage 3 (i). ReFL is Example 9 with M t g t = 1 M_{\mathrm{tgt}}=1 M tgt = 1 , r ^ ≡ 1 \hat{r}\equiv1 r ^ ≡ 1 and ∇ R \nabla R ∇ R of the full training reward; for a fair comparison, u o l d u_{\mathrm{old}} u old is also the EMA of u θ u_\theta u θ . A summary of the methods is given in Table 5.
Baselines. In the mixed-reward setting we additionally run the original objectives on the pre-RL few-step model without the adaptation of § 4.3 , i.e., with the average velocity u θ ( x ~ t , t , r ) u_\theta(\tilde{x}_t,t,r) u θ ( x ~ t , t , r ) , r r r drawn as in Stage 3 (i) (s s s for FlowGRPO), substituted for the instantaneous velocity wherever the original objective requires one. For DiffusionNFT and AWM, it is Equation (47) with u θ ( x ~ t , t , r ) u_\theta(\tilde{x}_t,t,r) u θ ( x ~ t , t , r ) in place of u θ ( x ~ t , t , 0 ) u_\theta(\tilde{x}_t,t,0) u θ ( x ~ t , t , 0 ) . For FlowGRPO, this substitution is not theoretically solid. We denote these baselines by the names of the original methods, FlowGRPO, AWM and DiffusionNFT, and compare them with FewStepGRPO, MeanFlowAWM and MeanFlowNFT, respectively, while keeping the other setting the same. In addition, by Appendix § 11 , the detached correction J J J cancels from its v-form loss, so the u-form in the FewStepOPSD family is the special case of the v-form in which r r r is fixed to 0 instead of being drawn at every query, so we did not compare them.
Evaluation. Three sources are used, with the conventions of the figures. ( i ) (i) ( i ) Online, every 50 iterations, within training: the aesthetic score, CLIPScore and OCR accuracy, and in the mixed-reward setting HPSv2 and PickScore; on DrawBench prompts (Saharia et al., 2022) in the mixed-reward setting, and on the OCR prompt set for OCR accuracy and for every dimension of the OCR setting. ( i i ) (ii) ( ii ) Offline, at the checkpoints saved every 100 iterations, on 256 images with seed 12345 and the same prompts: HPSv3 and ImageReward (Ma et al., 2025b; Xu et al., 2023). ( i i i ) (iii) ( iii ) Offline, GenEval2 (Soft-TIFA) (Kamath et al., 2025), the successor to GenEval [25 ] , in its official evaluation setting, following MeanFlowNFT [34 ] (Section D.1). The LAION aesthetic score (Schuhmann, 2022), HPSv3, ImageReward and GenEval2 serve as held-out evaluators, as do OCR accuracy in the mixed-reward setting and CLIPScore in the OCR setting. Iteration 0 is the shared AnyFlow Init. checkpoint.
Setup on Video Generation
Generator and sampler. The policy is the few-step generator of § 3 on the Wan2.1-T2V-1.3B backbone (Team Wan et al., 2025), distilled on-policy with AnyFlow [26 ] against its frozen teacher as in the image setting. Reinforcement learning trains a LoRA [32 ] of rank 32 (scaling 64) on the attention projections, at 480 × \times × 832 and 81 frames, i.e. latents of shape 21 × 16 × 60 × 104 21 \times16 \times60 \times104 21 × 16 × 60 × 104 . We set M s t e p = 4 M_{\mathrm{step}} = 4 M step = 4 steps without classifier-free guidance [30 ] , with the deterministic sampler of Equation (5) for every method except FewStepGRPO, which requires Equation (94) . Evaluation always uses the deterministic sampler.
Optimisation. Each outer iteration draws two batches, each with 2 prompts per rank and K = 16 K = 16 K = 16 members, on 32 NVIDIA H20 GPUs, i.e. 2048 rollouts per outer iteration. We use AdamW (Loshchilov and Hutter, 2019) with learning rate 3 × 10 − 6 3 \times10^{-6} 3 × 1 0 − 6 and gradient-norm clipping at 1; u o l d u_{\mathrm{old}} u old is an EMA of u θ u_{\theta} u θ refreshed once per outer iteration; for the NFT/OPSD families, β = 0.1 \beta= 0.1 β = 0.1 and c a d v = 5 c_{\mathrm{adv}} = 5 c adv = 5 in Equation (84) ; the KL-u penalty against the fixed AnyFlow initial policy uses Equation (92) with weight 10 − 4 10^{-4} 1 0 − 4 ; and each member is queried n t s = 4 n_{\mathrm{ts}} = 4 n ts = 4 times, with half of each batch set to r = t r = t r = t and a quarter to r = 0 r = 0 r = 0 . For FewStepOPSD and ReFL, the world size is 48 instead, because computing the gradient with respect to the video latents requires much memory, while the per-rank batch is unchanged. FewStepOPSD and ReFL therefore use 3072 rollouts per outer iteration, 1.5 times as many as the 32-GPU runs. The runs are thus matched in the number of outer iterations but not in the number of samples, and the quality comparison should be interpreted with this caveat. GPU-hours remain comparable across runs, whereas s/iter is reported but is not comparable across runs with different world sizes.
Rewards. Using the same four reward components as the LongCat/MeanFlowNFT video setup [34 ] (Meituan LongCat Team et al. 2025), we define our training reward as an unweighted sum. The HPSv3 scorer (Ma et al., 2025b) enters through HPSv3-general and HPSv3-percentile (per-frame, the latter the mean of the top 30% of frames under the caption); VideoAlign’s motion quality (MQ) and text alignment (TA) are drawn from video-feedback scoring (Liu et al., 2025b). MQ and TA are z z z -normalised before entering the sum; tables report their raw scorer outputs. Only HPSv3 is differentiable here, and its two reductions account for 96% of the aggregate reward under the z z z -normalised weights (Section D.5). The VideoAlign reward takes file paths as input and decodes the videos in its own pipeline, so there is no differentiable path from the latents to this reward. FewStepOPSD and ReFL therefore use only the gradients of the HPSv3 terms, while they are evaluated on all four terms; Section D.5 quantifies this asymmetry, which does not occur in the image setting.
Specific settings. As in the image setting, n i t = 4 n_{\mathrm{it}} = 4 n it = 4 for the group-advantage-based methods (n i t = M s t e p n_{\mathrm{it}} = M_{\mathrm{step}} n it = M step for FewStepGRPO, n i t = n t s n_{\mathrm{it}} = n_{\mathrm{ts}} n it = n ts for the others), and the auxiliary velocity v ϕ v_{\phi} v ϕ is fitted by Equation (88) with learning rate 10 − 5 10^{-5} 1 0 − 5 and n f m = 5 n_{\mathrm{fm}} = 5 n fm = 5 steps per outer iteration for DMDAWM and DMDNFT. Because there is one reward setting rather than two, the gate radius ϵ c l i p = 10 − 3 \epsilon_{\mathrm{clip}} = 10^{-3} ϵ clip = 1 0 − 3 is used for both χ G R P O \chi_{\mathrm{GRPO}} χ GRPO in FewStepGRPO and χ A W M \chi_{\mathrm{AWM}} χ AWM in MeanFlowAWM/ DMDAWM, and FewStepGRPO uses ζ = 0.7 \zeta= 0.7 ζ = 0.7 in Equation (95) . FewStepOPSD and ReFL require further settings, because the video backbone imposes three constraints on ∇ R \nabla R ∇ R that do not occur in the image setting. First, as stated above, ∇ R \nabla R ∇ R is computed from the two HPSv3 reductions instead of the full training reward; the VideoAlign terms still enter the reward and the advantage but do not contribute to the gradient. Second, the reward model is cast to fp32 with the sdpa attention backend; with the released bfloat16 weights, the reward gradient changes direction substantially while the reward value changes by only ∼ \sim ∼ 0.003, and since Equation (62) normalises ∇ R \nabla R ∇ R to unit length and thus uses only its direction, this error directly affects the update. Third, the video is decoded one latent frame at a time, so ∇ R \nabla R ∇ R is computed from 21 of the 81 pixel frames of a clip, each decoded without its causal prefix; Section D.5 reports the memory measurements that make this necessary.
Table 6. (a) OCR setting
Figure 3. OCR setting: the evaluated dimensions against the iteration, over the first 200 iterations. Top row: online evaluation every 50 iterations. Bottom row: offline evaluation at saved checkpoints, every 100 iterations; HPSv3 and ImageReward on the OCR prompts, GenEval2 on its own prompts.
Table 6 (PDF p. 42). (b) Mixed-reward setting, iteration 400
Table 6: Main results. (a) OCR setting, restricted to the 200-iteration horizon of Figure 3 . (b) Mixed-reward setting at iteration 400: s/iter and wall-clock hours to iteration 400; the training reward and its three summands; the held-out evaluators; GenEval2 at iteration 400, averaged over iterations 100–400, and at the best checkpoint. Bold: best among the trained methods per comparable column.
Evaluation. We use two evaluations that measure different properties. (i) Every 10 iterations, we compute the four training rewards on 256 held-out prompts with seed 12345. Since this is the training objective, it cannot distinguish a quality gain from reward hacking. (ii) At iteration 250, we evaluate VBench (Huang et al., 2024) on 946 GPT-augmented prompts × 5 samples = 4730 videos per method, with all 16 dimensions aggregated by the quality/semantic/overall weights of AnyFlow (Gu et al., [26 ] ). No method optimises any part of it. Iteration 0 is the shared pre-RL checkpoint trained with AnyFlow.
Main Results on Image Generation
Effectiveness. Table 6 and Figure 3 and Figure 4 support the following observations.
Figure 4. Mixed-reward setting: the evaluated dimensions against the iteration, over the first 400 iterations. The first panel is the training reward; OCR accuracy, HPSv3, ImageReward and GenEval2 are held out. Online evaluation every 50 iterations, offline evaluation at saved checkpoints every 100.
(i) The adaptation of §4.3 is beneficial in these runs, and the v-prediction form gives better performance in most of these comparisons (Figure 1 and Figure 6 ). FewStepGRPO raises the training reward from 1.450 to 1.461 and the held-out HPSv3 from 10.30 to 11.58, whereas FlowGRPO, its counterpart with direct employment of average-velocity, leaves the training reward at 1.438 and lowers HPSv3 to 9.62. Meanwhile, MeanFlowNFT as the v-prediction counterpart of DiffusionNFT, which is the direct employment of average-velocity, reaches a training reward of
Figure 6. Mixed-reward setting: the baselines of § 6.1 against the few-step adaptations of this paper (solid), from the shared pre-RL checkpoint. Panels as in Figure 4 . Each annotated gap is the adaptation minus the baseline, at the best checkpoint of the baseline and at iteration 400; the two coincide when the baseline is best at iteration 400. GenEval2 for the same runs is shown in Figure 1 .
1.509 against 1.440, and is higher on the held-out HPSv3 (13.23 against 11.79), ImageReward (1.385 against 1.317) and best GenEval2 (0.411 against 0.399), the last within the ∼0.03 resolution.
(ii) In the OCR setting, DMDAWM, MeanFlowNFT and DMDNFT raise OCR accuracy from 0.498 to at least 0.85 within 100–150 iterations, whereas MeanFlowAWM and FewStepGRPO reach only 0.802 and 0.776 at iteration 200. Furthermore, it can be observed that within the 200-iteration horizon of Figure 3 , the highest logged OCR accuracy is reached by MeanFlowNFT, about 0.945 at iteration 150, and it is also the first to reach 0.85, at iteration 100. DMDAWM starts lower at iteration 50 but rises steeply thereafter and reaches 0.936 at iteration 200, where the two curves coincide within the resolution of the plot (Table 6a). Its rapid rise is consistent with a combination of two properties. (i) It has no self-regression (x ~ r e g = ∅ \tilde{x}_{\mathrm{reg}} = \varnothing x ~ reg = ∅ ), so only reward augmentation contributes to its decoupled update. (ii) It is DMD-based and models the instantaneous velocity with the auxiliary network v ϕ v_{\phi} v ϕ , which is refitted to the current rollouts at every iteration, so the baseline x ~ b a s e = x ~ a u x \tilde{x}_{\mathrm{base}} = \tilde{x}_{\mathrm{aux}} x ~ base = x ~ aux follows the policy; since this baseline is fixed within an update, reward augmentation uses the direction A ( v c o n d − v ϕ ) A\left(v_{\mathrm{cond}} - v_{\phi}\right) A ( v cond − v ϕ ) instead of one based on the directional-difference estimate Equation (87) . This fast increase coincides with the large held-out degradation; at its first checkpoint with OCR accuracy above 0.82, HPSv3 drops from 12.86 to 5.05 and ImageReward from 1.233 to 0.092.
(iii) In the mixed-reward setting, FewStepOPSD and ReFL achieve the two highest training rewards, with FewStepOPSD highest at the final iteration. Using the gradient of the full reward, the two raise the training reward from 1.450 to 1.64 and 1.62, against 1.46–1.52 for the five group-advantage methods, and they also lead on ImageReward and GenEval2. Their curves nearly coincide (Figure 4 ); the logged coefficient magnitudes of FewStepOPSD differ by 0.24 between the two branches, i.e. 2.4% of their fixed sum 1 / β = 10 1/\beta= 10 1/ β = 10 , although the two reward-augmentation vectors also depend on their different increments (Section D.3). Among the five group-advantage
methods, DMDNFT and MeanFlowNFT come next and perform similarly (training reward 1.51–1.52, GenEval2 0.41 at iteration 400).
(iv) In the default mixed-reward configuration, none of the five group-advantage runs increases CLIPScore, which decreases slightly from 0.289 to 0.279–0.285, whereas FewStepOPSD and ReFL raise it to 0.317–0.322. Dedicated CLIP-focused group-advantage ablations do increase it (Section D.2).
Efficiency. We measure efficiency by the time per iteration (Table 7 ) and the time to reach the best checkpoint (Figure 5 a). It can be observed that
Figure 5. Mixed-reward setting: GenEval2 of each trainer’s best checkpoint within 400 iterations. (a) Left: against the wall-clock time to reach it, s/iter times the iteration, on 8 GPUs. (b) Right: against s/iter. Both panels show the same checkpoints, so they share the vertical axis and the legend, and a trainer moves only horizontally from (a) to (b). The number beside each marker is the iteration of that checkpoint.
Table 7. Cost of one outer iteration in the mixed-reward setting, split into the sampling time (rollout, reward and, for FewStepOPSD and ReFL, the reward-coupled construction of Equation (82) ) and the update time; medians over the timers in the training logs, on 8 H20 GPUs. The small residual between the sum of the two times and s/iter covers the refresh of u o l d u_{\mathrm{old}} u old and logging.
(i) The two reward-gradient-based methods are the cheapest per iteration, 219 s for ReFL and 268 s for FewStepOPSD, because they compute one update direction per member in the clean-sample space with one query of u θ u_{\theta} u θ instead of four (n i t = 1 n_{\mathrm{it}} = 1 n it = 1 ), and because J J J cancels in their loss. However, their sampling time, which includes the reward-coupled rollout of Equation (82) , is 175–225 s per iteration, against about 78 s for the plain rollout of the other five methods (Table 7 ). Since the reward is defined in pixel space, its gradient with respect to the clean latent requires back-propagation through both the reward model and the VAE decoder.
(ii) The directional-difference evaluations Equation (87) and Equation (109) for the shared-J J J v-form plausibly account for most of the cost difference. FewStepGRPO, MeanFlowAWM and MeanFlowNFT require two additional forward passes of u o l d u_{\mathrm{old}} u old per query. These methods take 423–435 s, against 304–326 s for DMDAWM and DMDNFT, which compute the baseline from v ϕ v_{\phi} v ϕ and compute no J J J . The difference arises in the update time (344–355 against 223–245 s), and the OCR setting gives the same ordering (451–496 against 342–346 s).
(iii) ReFL has a marginally higher best GenEval2 point estimate than FewStepOPSD (0.465 against 0.463), but the 0.002 gap is below the stated 0.03 resolution and does not separate the two methods. ReFL reaches its selected checkpoint after 12.2 h (iteration 200), whereas FewStepOPSD continues to iteration 400 (29.8 h); both peak on the held-out aesthetic score and HPSv3 by iteration 100 (the aesthetic trajectory of FewStepOPSD is tabulated in Table S1 ; HPSv3 is read from Figure 4 ), after which these scores decline while the training reward still rises, which is consistent with the onset of reward hacking. Among the group-advantage-based methods, MeanFlowAWM and DMDAWM reach their best GenEval2 early, at iteration 100 (11.8 and 8.4 h), but at lower values (0.397 and 0.377).
Table S1. Comparison of retaining J J J and setting J = 0 J = 0 J = 0 in FewStepOPSD and in ReFL with ∇ R \nabla R ∇ R of the full reward. For each trainer, the two runs differ only in whether Equation (87) is evaluated. Each cell reads “J J J retained / J = 0 J = 0 J = 0 ”; the runs with J = 0 J = 0 J = 0 are those of § 6.3 , and their entries are those of Table 6, Table S4 and Figure 4 . Top: the runs with J J J retained are evaluated offline (256 images, seed 12345, DrawBench prompts, OCR prompts for OCR accuracy), with the same protocol as the runs with J = 0 J = 0 J = 0 at iteration 400. Bottom: cost per iteration, medians over the training logs on 8 GPUs; the runs with J J J retained were not timed on the same machine as those with J = 0 J = 0 J = 0 .
The four component columns show raw scorer outputs; the sum uses the logged z z z -scores for MQ and TA. GPU-h = GPUs × s/iter ×250/3600, truncated to integer hours, is the comparable cost because the methods ran on different world sizes with a fixed per-rank batch. Consequently, the 48-GPU rows use 1.5× as many rollouts per iteration and are not sample-budget-matched to the 32-GPU rows. The dimension-level VBench results underlying the three aggregates are reported in Table S2 of Appendix § 12 . For FewStepOPSD and ReFL, only part of the video reward is differentiable, unlike in the image setting (Section D.5).
Main Results on Video Generation
Effectiveness. Table 8 and Table S2 support the following observations.
Table 8. Main results on video generation at iteration 250. Left: cost. Middle: the aggregate training reward and its four components on 256 held-out prompts. Right: VBench, which no method optimises. Bold: best per comparable column; s/iter is compared only among the 32-GPU rows.
(i) FewStepGRPO is the least effective at increasing the training reward. It is the only group-advantage-based method that ends below the shared initialisation (15.89 against 16.54), and its TA drops markedly, from 1.85 to 1.23. Among the group-advantage-based methods DMDNFT is the most effective, raising the training reward to 17.93 (+1.39) and leading on three of its four terms. In this run the larger training-reward gain coincides with larger held-out degradation. Against MeanFlowAWM, the highest on VBench overall, DMDNFT is 0.32 lower in VBench overall (83.38 against 83.70, last of the seven), 5.56 lower in dynamic degree (53.33 against 58.89, the lowest of any method) and 0.16 lower in TA (1.68 against 1.84).
(ii) Among the remaining methods, FewStepOPSD and ReFL have the lowest training reward after FewStepGRPO, ending at 16.26 and 16.46, slightly below the initialisation. For video, the reward gradient must be computed for every latent token of a long sequence, through the decoder and the reward model, which requires several numerical simplifications in the reward-gradient-based runs. Their VBench point estimates degrade less than that of DMDNFT in these runs, which does not establish that they protect against reward exploitation. TA rises above the initialisation (1.91 and 1.89 against 1.85), ReFL is second on VBench overall (83.69) and first on quality (85.40), and FewStepOPSD has the highest aesthetic quality, background consistency and appearance style of any method (70.87, 95.68 and 74.53); FewStepOPSD and ReFL have the second- and third-highest dynamic degree (59.44 and 59.17).
Efficiency. The GPU-hours in Table 8 lead to the following observations. (i) Unlike in the image setting, the DMD-based methods are slower than their MeanFlow counterparts, in both s/iter (821 against 734–737 s) and GPU-hours. With n f m = 5 n_{\mathrm{fm}} = 5 n fm = 5 , omitting the directional difference saves up to 8 forward passes of u o l d u_{\mathrm{old}} u old without gradient per member (n t s = 4 n_{\mathrm{ts}} = 4 n ts = 4 queries, two each) but adds 5 forward and backward passes of v ϕ v_{\phi} v ϕ per outer iteration. The fit costs about 144 s per iteration and the net s/iter difference is +84–87 s, so the saved J J J passes account for about 60 s by difference; the fit dominates, plausibly because backward passes through the larger video model are expensive.
(ii) The reward-gradient-based methods lose their cost advantage from the image setting and are the two most expensive in GPU-hours. FewStepOPSD and ReFL have a sampling time of up to 674 s per iteration (FewStepOPSD), which includes the reward-gradient computation, against ∼ 291 \sim291 ∼ 291 s for the other methods, and an update time of only 112 s. The gradient of a video reward with respect to the latent tokens is expensive, plausibly because the latent sequence is much longer than for images and because both the reward model and the decoder are larger than their image counterparts. In addition, MQ and TA are not differentiable (§ 6.2 ), which further limits these two methods.
Test-Time Scaling Analysis
In deployment, a few-step model should ideally support several operating points on the quality–latency trade-off, using its training step count for most requests and more sampling steps when higher quality is required. This motivates training at a small M s t e p M_{\mathrm{step}} M step while allowing more solver steps at inference. If quality improves without retraining, the model supports test-time scaling and can trade compute for output quality. Flexible-step generation and inference-time scaling have also been studied for consistency-type models, improved MeanFlow and scalable generators ([24 ] (Kim et al. 2024; Wang et al. 2024a; Xie et al. 2025a)); our experiment is restricted to the checkpoints and solver grid below.
Setup. We therefore consider the image generation task, and take the checkpoints trained and evaluated at M s t e p = 4 M_{\mathrm{step}} = 4 M step = 4 in § 6.1 and sample each at M s t e p ∈ { 4 , 8 , 16 , 32 } M_{\mathrm{step}} \in\{4, 8, 16, 32\} M step ∈ { 4 , 8 , 16 , 32 } . We use the checkpoints listed in Table S3 . We keep the 256 DrawBench prompts, seed 12345, resolution, deterministic sampler Equation (5) , absence of classifier-free guidance, and the time shift 3.0 of § 6.1 fixed. Increasing M s t e p M_{\mathrm{step}} M step only refines the same shifted solver grid; no model is retrained. Thus the experiment isolates a 1 × 1 \times 1 × , 2 × 2 \times 2 × , 4 × 4 \times 4 × , or 8 × 8 \times 8 × inference-step budget. As references, we also evaluate the AnyFlow initialisation, from which all seven trainers start, and the pretrained SD3.5-Medium, with the same prompts, seed, resolution and shifted grid. SD3.5-Medium models the instantaneous velocity and is not distilled for few-step sampling, so it uses the Euler step Equation (4) with classifier-free guidance of scale 4.5, i.e., two network evaluations per step.
Table S3. Test-time scaling in the mixed-reward setting. Left: HPSv3, a held-out evaluator with a clear signal, at each step count. Right: the percentage change of every evaluated dimension between M s t e p = 4 M_{\mathrm{step}} = 4 M step = 4 and 32. CLIPScore, HPSv2 and PickScore are the three summands of the training reward; HPSv3, ImageReward, GenEval2 and the aesthetic score are held out. The GenEval2 column is reported for completeness only, since its largest change from 4 to 32 steps (0.028) and its largest intermediate change (0.032) are within or at the ∼0.03 resolution stated in § 6.1 for 256 images. Trainer labels are as in Table 5; “it.” is the checkpoint used.
Main results. Figure 7 shows the per-step trends, and Table S3 in Appendix § 12 reports the percentage changes. We draw four conclusions.
Figure 7. Absolute value of each evaluated dimension against M s t e p M_{\mathrm{step}} M step , one line per fine-tuned model, with the AnyFlow initialisation (dashed) and the pretrained SD3.5-Medium sampled with classifier-free guidance (dotted) as references. For the fine-tuned models, the three training-reward summands (CLIPScore, HPSv2 and PickScore) show little systematic gain from additional steps, while the held-out HPSv3 and aesthetic score more often decline. Trainer labels are as in Table 5; the checkpoint is given in Table S3 . Figure S1 and Figure S2 in Appendix § 12 show the same results as percentage changes from the four-step value, grouped by evaluator and by model, respectively.
Figure S1. Percentage change of each evaluated dimension from its four-step value against M s t e p M_{\mathrm{step}} M step , one panel per evaluator and one line per model, including the AnyFlow initialisation and the pretrained SD3.5-Medium. Values of SD3.5-Medium outside the plotted range are printed at the top of each panel. The shaded band in the GenEval2 panel marks the ∼0.03 resolution of § 6.1 ; the changes of the fine-tuned models stay within or at this resolution, whereas both references move beyond it.
Figure S2. The percentage changes of Figure S1 , one panel per model and one line per evaluator. Solid lines are held-out evaluators, and dashed lines are the summands of the training reward.
(i) Models at different training stages show different test-time scaling behaviour. The pretrained SD3.5-Medium models the instantaneous velocity, so four Euler steps are far too coarse; its scores rise sharply from four to eight steps and keep improving more slowly up to 32 steps (HPSv3 from −4.49 to 9.96, Figure 7 ). Even at 32 steps, i.e., 64 network evaluations, it remains below every fine-tuned model at four steps in HPSv3, ImageReward, the aesthetic score, HPSv2 and PickScore. In contrast, the AnyFlow initialisation, distilled from SD3.5-Medium for few-step sampling, appears to have already fixed its output patterns at the step count used in training, and its HPSv3, ImageReward and GenEval2 drop markedly with more steps; for example, from 4 to 32 steps, ImageReward falls from 1.11 to 1.02 and GenEval2 from 0.363 to 0.320 (Figure 7 ), beyond the ∼ 0.03 \sim0.03 ∼ 0.03 resolution. By comparison, the RL fine-tuned models, which all start from this initialisation, fluctuate less in these three evaluators.
(ii) There is no systematic gain on the training reward. Across the fine-tuned models, the three reward summands remain nearly flat (Figure 7 and Table S3 ). From 4 to 32 steps, the held-out HPSv3 changes by between +1.0% and −6.8% for six of the seven trainers, compared with −17% for the AnyFlow initialisation (Figure 7 ). The stronger degradation of the initialisation itself suggests that the decline in rewards of the fine-tuned models is inherited from the initialisation rather than introduced by RL fine-tuning. Further, these small changes are consistent with the 4-step outputs already having settled into patterns that score well under the training reward, including sampler-specific reward exploitation. Doubling to M s t e p = 8 M_{\mathrm{step}} = 8 M step = 8 often preserves or slightly improves a score, but the gain generally saturates or reverses as the grid is refined further. Figure 8 shows both behaviours; MeanFlowAWM improves at eight steps and then declines, whereas ReFL declines gradually.
Figure 8. Two selected mild-degradation cases; each panel shows the six evaluated scores of that image (CLIPScore, HPSv2, PickScore, HPSv3, aesthetic score and ImageReward). Top, ReFL on the three-cube prompt, where HPSv3 gradually falls from 15.63 to 14.78. Bottom, MeanFlowAWM on “A panda making latte art”, where HPSv3 first rises from 11.34 to 11.59 at M step = 8 M_{\text{step}} = 8 M step = 8 , then falls to 10.77 at 32 as an extra cup appears.
(iii) OCR-sensitive prompts can expose a much sharper failure mode. The selected failures occur on malformed or OCR-heavy prompts. A clear example is shown in Figure 9 , where the DMDNFT and MeanFlowNFT samples collapse at M s t e p = 32 M_{\mathrm{step}} = 32 M step = 32 to little more than foreground text. HPSv3 falls from 12.18 to 0.32 and from 11.36 to 0.32, respectively, even as CLIPScore rises from 0.275 to 0.324 and from 0.292 to 0.309. In these selected cases, additional solver steps produce increasingly text-dominated outputs rather than a visual-quality improvement.
Figure 9. Two selected OCR-sensitive text-dominance cases under test-time scaling: DMDNFT (top) and MeanFlowNFT (bottom). Within each row, the prompt and seed are fixed and only M s t e p M_{\mathrm{step}} M step changes; the values below each image are that image’s rewards. These are selected qualitative failure cases rather than population averages.
(iv) Visible sample-level scaling exists. On inspection, many samples improve with 2–4 times as many sampling steps, even though the aggregate rewards barely change; on the selected samples, HPSv3 does rise (Figure 10 ). At four steps, they have weak clarity or unstable details, while eight or sixteen steps yield cleaner and more stable generations; Figure 10 shows two such cases. The aggregate rewards thus appear less sensitive to these improvements than visual inspection, and test-time scaling may be useful beyond what the aggregate scores suggest, although more steps do not improve quality monotonically.
Figure 10. Selected cases where a modest inference-step increase helps. Top, FewStepOPSD on the nail-products prompt: the 4-step image is cluttered and soft, whereas the 8- and 16-step outputs have clearer objects and a more stable arrangement.
Practical Guidance
The empirical results, each from a single run, suggest the following guidance:
(i) Where it is affordable, the reward gradient is our recommended approach. Driving the update directly by ∇ R \nabla R ∇ R was the most effective choice in our mixed-reward image experiments; FewStepOPSD and ReFL achieve the two highest training rewards in the mixed-reward setting and improve all three sub-terms (§ 6.3 ). However, the applicability of these methods is restricted in practice. (i) The gradient can become too expensive when the reward model and the decoder are large or the latent token sequence is long, as on video (§ 6.4 ). (ii) The reward must be differentiable with respect to the clean sample, which excludes OCR accuracy and VideoAlign’s MQ and TA (§ 6.1 , Section D.5). Where both conditions hold, ReFL alone is already a strong baseline in our runs.
(ii) Otherwise, group-advantage-based methods apply. When the reward gradient is too expensive or unavailable, the reward enters only through the group advantage, and these methods must convert the average velocity into the instantaneous one (§ 4.3 ). Among them, the DMD-based methods, which estimate the instantaneous velocity with the auxiliary network v ϕ v_{\phi} v ϕ , have a lower per-iteration cost in the image setting but a higher one on video, depending on the relative cost of the saved forward passes for J J J and the added forward and backward passes of v ϕ v_{\phi} v ϕ (§ 6.3 , § 6.4 ). They need extra optimisation steps and memory for v ϕ v_{\phi} v ϕ , and in some of the reported runs their larger reward gains coincide with larger held-out degradation.
(iii) Among the group-advantage-based methods, the DiffusionNFT family reached high reward values in relatively few iterations. The FlowGRPO-style replay of FewStepGRPO optimises slowly and has the weakest observed reward endpoints within the common iteration budget: it barely improves the mixed reward on images and loses the most reward on video. The DiffusionNFT family reached high reward values within the common iteration budget in both settings, although the cost per iteration varies; MeanFlowNFT is expensive in the image setting and DMDNFT on video. AWM mostly lies between the two in the reported reward outcomes and uses a PPO-style clip to limit each update.
(iv) Test-time scaling of the fine-tuned few-step models saturates quickly. Aggregate scores show no systematic gain; three of the seven HPSv3 curves peak at the training step count of four and one is tied between four and eight steps (Table S3 ). By visual inspection, however, many individual samples look cleaner with 2–4 times as many steps before the effect saturates or reverses. Additional steps therefore improve the visual quality of some samples but do not improve aggregate scores monotonically (§ 6.5 ), and can degrade sample quality on OCR-sensitive prompts.
Conclusion
Few-step generators reduce the rollout cost that typically dominates RL post-training of diffusion models, but they model an average velocity over an interval, whereas existing RL objectives assume an instantaneous velocity and a many-step sampler. This paper adapts these objectives to few-step models systematically in three phases and compares the resulting methods within one codebase. The transformation to x x x -space expresses each objective we consider through a common policy prediction. The decomposition writes the gradient of each objective as the sum of a reward-free self-regression and a reward augmentation, so that a method reduces to a choice of three anchors and three weights. The adaptation computes these anchors from a few-step model, at the end of its transport or along its tangent, with the instantaneous velocity recovered from the network itself through the MeanFlow identity (the self-induced velocity) or from an auxiliary network. The seven resulting trainers share one training step and are compared under shared rollout, reward and evaluation code, each in a single run. Where the reward is differentiable and its gradient affordable, FewStepOPSD and ReFL, which use the reward gradient, were the most effective. Otherwise, the DiffusionNFT family reached high rewards in relatively few iterations, and the DMD-based variants were faster per iteration on image but slower on video, depending on the relative cost of forward and backward passes, and in some runs coincided with larger held-out degradation. For the fine-tuned models, additional sampling steps gave no systematic aggregate gain, although many individual samples improve with two to four times as many steps as in training.
Open problems. Several questions remain open.
(ii) Reward-gradient-based methods require a reward that is differentiable with respect to the clean sample and a gradient that fits in memory. On video, the motion-quality and text-alignment terms of VideoAlign provide no gradient, and the per-frame decoding required by GPU memory makes the gradient deviate strongly from that of a joint decode. Cheaper and more faithful reward gradients for long latent sequences are needed.
(iii) With a scalar group advantage in the mixed-reward setting, whether every reward summand takes effect depends on the weighting of the summands, on the rescaling by the global-batch standard deviation in Equation (83) , and on the small within-group spread of some summands (Section D.2). A general and more reliable construction of the group advantage remains open.
(iv) In our experiments, a few-step model fine-tuned at one step count does not improve systematically with more steps. Fine-tuning at more step counts may give test-time scaling but makes the rollouts more expensive. Whether it can be introduced cheaply during training, or without training, so that a model fine-tuned at a small step count still improves with more steps at test time, remains open.
Notation
This paper connects several families of methods and therefore uses a large number of symbols. We collect them in the following table; each symbol is also defined where it first appears.
Table 5.
FewStepGRPO
Table 9.
FewStepGRPO
Throughout this appendix, ( x t , x r ) (x_t, x_r) ( x t , x r ) denotes a transition of the stochastic sampler Equation (94) replayed as in § 5.3 , where t > r t>r t > r are consecutive levels of the rollout grid, Δ t = t − r \Delta t=t-r Δ t = t − r , and s ∈ [ 0 , t ] s\in[0,t] s ∈ [ 0 , t ] is the lower endpoint of the flow map, redrawn at every inner iteration. The same estimate J = J [ u o l d , t , s ] J=J[u_{\mathrm{old}},t,s] J = J [ u old , t , s ] is added to both means of Equation (95) ,
μ θ = α t x t − b t ( u θ ( x t , t , s ) + J ) , μ o l d = α t x t − b t ( u o l d ( x t , t , s ) + J ) , (108) \mu_\theta=\alpha_t x_t-b_t\left(u_\theta(x_t,t,s)+J\right),\qquad\mu_{\mathrm{old}}=\alpha_t x_t-b_t\left(u_{\mathrm{old}}(x_t,t,s)+J\right), \tag*{(108)} μ θ = α t x t − b t ( u θ ( x t , t , s ) + J ) , μ old = α t x t − b t ( u old ( x t , t , s ) + J ) , (108)
where J J J is evaluated by Equation (109) , with tangent ( x t − x ^ ) / t (x_t-\hat{x})/t ( x t − x ^ ) / t . Squared norms are per-sample means means p [ ⋅ ] \operatorname{means}_p[\cdot] means p [ ⋅ ] , as in Stage 4.
Why there is no auxiliary-velocity variant
The auxiliary velocity cannot be used for the current prediction, because substituting v ϕ v_\phi v ϕ would give ∂ x ~ θ / ∂ θ = 0 \partial\tilde{x}_\theta/\partial\theta=0 ∂ x ~ θ / ∂ θ = 0 (§ 4.3 ); the trainers that use v ϕ v_\phi v ϕ therefore use it only for the baseline, x ~ b a s e = x ~ a u x \tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{aux}} x ~ base = x ~ aux (Equation (60) , Equation (104) ). In FewStepGRPO, both anchors that an auxiliary field could replace are already fixed, x ~ r e g = ∅ \tilde{x}_{\mathrm{reg}}=\varnothing x ~ reg = ∅ and x ~ b a s e = sg ( x ~ θ ) = sg ( μ θ ) \tilde{x}_{\mathrm{base}}=\operatorname{sg}(\tilde{x}_\theta)=\operatorname{sg}(\mu_\theta) x ~ base = sg ( x ~ θ ) = sg ( μ θ ) (Equation (97) ), and §B.3 shows that this choice reproduces the gradient of the tempered Gaussian log-score of the action. Apart from the detached J J J shared by both means, the old policy enters through μ o l d \mu_{\mathrm{old}} μ old in ρ ˉ G R P O \bar{\rho}_{\mathrm{GRPO}} ρ ˉ GRPO . An auxiliary variant would therefore have to place v ϕ v_\phi v ϕ inside the scoring kernel.
(a) v ϕ v_\phi v ϕ as the old-policy field. If μ o l d \mu_{\mathrm{old}} μ old were replaced by μ ϕ = α t x t − b t v ϕ ( x t , t ) \mu_\phi=\alpha_t x_t-b_t v_\phi(x_t,t) μ ϕ = α t x t − b t v ϕ ( x t , t ) , then at θ = θ o l d \theta=\theta_{\mathrm{old}} θ = θ old the scoring ratio would compare the distances of μ ϕ \mu_\phi μ ϕ and μ o l d \mu_{\mathrm{old}} μ old to x r x_r x r . The ratio would then generally differ from one at zero policy drift, so the PPO clip would respond to the approximation error of v ϕ v_\phi v ϕ as well as to the change of the policy.
(b) v ϕ v_\phi v ϕ as the source of the shared correction. Replacing J J J by v ϕ ( x t , t ) − u o l d ( x t , t , s ) v_\phi(x_t,t)-u_{\mathrm{old}}(x_t,t,s) v ϕ ( x t , t ) − u old ( x t , t , s ) in both means of Equation (108) keeps μ θ − μ o l d = − b t ( u θ − u o l d ) \mu_\theta-\mu_{\mathrm{old}}=-b_t(u_\theta-u_{\mathrm{old}}) μ θ − μ old = − b t ( u θ − u old ) , hence ρ ˉ G R P O = 1 \bar{\rho}_{\mathrm{GRPO}}=1 ρ ˉ GRPO = 1 at θ = θ o l d \theta=\theta_{\mathrm{old}} θ = θ old , but it requires v ϕ v_\phi v ϕ at the replayed state x t x_t x t . By Equation (88) , v ϕ v_\phi v ϕ is fitted on re-noised states AddNoise ( x ^ , t ) = ( 1 − t ) x ^ + t ϵ \operatorname{AddNoise}(\hat{x},t)=(1-t)\hat{x}+t\epsilon AddNoise ( x ^ , t ) = ( 1 − t ) x ^ + t ϵ , with ϵ \epsilon ϵ independent of x ^ \hat{x} x ^ , against the conditional velocity v c o n d = ϵ − x ^ v_{\mathrm{cond}}=\epsilon-\hat{x} v cond = ϵ − x ^ . A replayed state is not of this form, because Equation (94) adds fresh noise at every step; hence x t x_t x t and the endpoint x ^ \hat{x} x ^ of the same trajectory are not connected by a straight path with independent noise ϵ \epsilon ϵ , and v c o n d v_{\mathrm{cond}} v cond is not defined at x t x_t x t . Exact transport at every step would be sufficient for the two state marginals to agree, but the finite-step map used here is not exact; the replay query can therefore differ from the distribution on which v ϕ v_\phi v ϕ is fitted. In an earlier run of construction (b), not among those of § 6 , which queried v ϕ v_\phi v ϕ at the replayed states, the evaluated OCR accuracy fell from 0.498 to 0.016 within 40 iterations while the loss Equation (88) decreased from 0.354 to 0.152. This is consistent with v ϕ v_\phi v ϕ improving on its training distribution while being evaluated on a different one. Querying v ϕ v_\phi v ϕ at a re-noised state instead removes the mismatch, but the correction is then evaluated at a state other than the one at which the kernel is centred, and the scored density is no longer that of the transition that was taken.
The trainers of §§§ 5.4 –§ 5.5 avoid both problems. Their query state AddNoise ( x ^ , t ) \operatorname{AddNoise}(\hat{x},t) AddNoise ( x ^ , t ) follows the training distribution of v ϕ v_\phi v ϕ (Stage 3 (i)), and v ϕ v_\phi v ϕ enters only through their baseline. FewStepGRPO, whose kernel has to be evaluated at the states the sampler visited, therefore uses only the shared-J J J construction.
The transition kernel
Replay trajectory. A state stored on the replay trajectory is not an endpoint-renoise state AddNoise ( x ^ , t ) \operatorname{AddNoise}(\hat{x},t) AddNoise ( x ^ , t ) , so v c o n d v_{\mathrm{cond}} v cond is not its tangent. We instead use ( x t − x ^ ) / t (x_t-\hat{x})/t ( x t − x ^ ) / t :
J [ u o l d , t , s ] = ( t − s ) u o l d ( x t + ϵ c d x t − x ^ t , t + ϵ c d , s ) − u o l d ( x t − ϵ c d x t − x ^ t , t − ϵ c d , s ) 2 ϵ c d . (109) J[u_{\mathrm{old}},t,s]=(t-s)\frac{u_{\mathrm{old}}\left(x_t+\epsilon_{\mathrm{cd}}\frac{x_t-\hat{x}}{t},t+\epsilon_{\mathrm{cd}},s\right)-u_{\mathrm{old}}\left(x_t-\epsilon_{\mathrm{cd}}\frac{x_t-\hat{x}}{t},t-\epsilon_{\mathrm{cd}},s\right)}{2\epsilon_{\mathrm{cd}}}. \tag*{(109)} J [ u old , t , s ] = ( t − s ) 2 ϵ cd u old ( x t + ϵ cd t x t − x ^ , t + ϵ cd , s ) − u old ( x t − ϵ cd t x t − x ^ , t − ϵ cd , s ) . (109)
The result is detached and shared by both means. It costs two no-grad forwards of u o l d u_{\mathrm{old}} u old .
Boundary condition. The coefficients of Equation (95) use the following implemented noise scale on the rollout grid:
ς t j = { ς t 0 1 − t 1 , j = 0 , ς t j 1 − t j , j = 1 , … , M s t e p − 1. (110) \varsigma_{t_j}=
\begin{cases}
\varsigma\sqrt{\frac{t_0}{1-t_1}}, & j=0,\\
\varsigma\sqrt{\frac{t_j}{1-t_j}}, & j=1,\ldots,M_{\mathrm{step}}-1.
\end{cases} \tag*{(110)} ς t j = ⎩ ⎨ ⎧ ς 1 − t 1 t 0 , ς 1 − t j t j , j = 0 , j = 1 , … , M step − 1. (110)
The coefficients α t \alpha_t α t and b t b_t b t retain the source level t = t j t=t_j t = t j ; only the singular denominator of ς t 0 \varsigma_{t_0} ς t 0 is evaluated at t 1 t_1 t 1 .
Derivation of Equation (95) . Write σ \sigma σ for the level along the linear path, on which the probability-flow ODE reads d x σ = v ( x σ , σ ) d σ \mathrm{d}x_\sigma=v(x_\sigma,\sigma)\,\mathrm{d}\sigma d x σ = v ( x σ , σ ) d σ , with v v v the marginal velocity, and the score is ∇ x σ log p σ ( x σ ) = − ( x σ + ( 1 − σ ) v ( x σ , σ ) ) / σ \nabla_{x_\sigma}\log p_\sigma(x_\sigma)=-(x_\sigma+(1-\sigma)v(x_\sigma,\sigma))/\sigma ∇ x σ log p σ ( x σ ) = − ( x σ + ( 1 − σ ) v ( x σ , σ )) / σ (§ 3 , the paragraph on DMD). For any diffusion coefficient ς σ ≥ 0 \varsigma_\sigma\ge0 ς σ ≥ 0 , the forward-time SDE
d x σ = [ v ( x σ , σ ) + 1 2 ς σ 2 ∇ x σ log p σ ( x σ ) ] d σ + ς σ d B (111) \mathrm{d}x_\sigma=\left[v(x_\sigma,\sigma)+\frac{1}{2}\varsigma_\sigma^2\nabla_{x_\sigma}\log p_\sigma(x_\sigma)\right]\mathrm{d}\sigma+\varsigma_\sigma\,\mathrm{d}B \tag*{(111)} d x σ = [ v ( x σ , σ ) + 2 1 ς σ 2 ∇ x σ log p σ ( x σ ) ] d σ + ς σ d B (111)
has the same marginals as the probability-flow ODE: in its Fokker–Planck equation, the term − 1 2 ς σ 2 ∇ ⋅ ( p σ ∇ log p σ ) -\frac{1}{2}\varsigma_\sigma^2\nabla\cdot(p_\sigma\nabla\log p_\sigma) − 2 1 ς σ 2 ∇ ⋅ ( p σ ∇ log p σ ) contributed by the added drift and the diffusion term 1 2 ς σ 2 ∇ 2 p σ \frac{1}{2}\varsigma_\sigma^2\nabla^2p_\sigma 2 1 ς σ 2 ∇ 2 p σ cancel, since p σ ∇ log p σ = ∇ p σ p_\sigma\nabla\log p_\sigma=\nabla p_\sigma p σ ∇ log p σ = ∇ p σ . The time reversal of Equation (111) , run from σ = 1 \sigma=1 σ = 1 to σ = 0 \sigma=0 σ = 0 , is
d x σ = [ v ( x σ , σ ) − 1 2 ς σ 2 ∇ x σ log p σ ( x σ ) ] d σ + ς σ d B ˉ , d σ < 0. (112) \mathrm{d}x_\sigma=\left[v(x_\sigma,\sigma)-\frac{1}{2}\varsigma_\sigma^2\nabla_{x_\sigma}\log p_\sigma(x_\sigma)\right]\mathrm{d}\sigma+\varsigma_\sigma\,\mathrm{d}\bar{B},\qquad\mathrm{d}\sigma<0. \tag*{(112)} d x σ = [ v ( x σ , σ ) − 2 1 ς σ 2 ∇ x σ log p σ ( x σ ) ] d σ + ς σ d B ˉ , d σ < 0. (112)
One Euler–Maruyama step from t t t to r r r , i.e. d σ = − Δ t \mathrm{d}\sigma=-\Delta t d σ = − Δ t , followed by the substitution of the score, gives
x r = x t − Δ t v − ς t 2 Δ t 2 t ( x t + ( 1 − t ) v ) + ς t Δ t ξ = ( 1 − ς t 2 Δ t 2 t ) x t − ( 1 + ς t 2 ( 1 − t ) 2 t ) Δ t v + ς t Δ t ⏟ σ k ξ , (113) \begin{aligned}
x_r=x_t-\Delta t\,v-\frac{\varsigma_t^2\Delta t}{2t}\left(x_t+(1-t)v\right)+\varsigma_t\sqrt{\Delta t}\,\xi\\
&=\left(1-\frac{\varsigma_t^2\Delta t}{2t}\right)x_t-\left(1+\frac{\varsigma_t^2(1-t)}{2t}\right)\Delta t\,v+\underbrace{\varsigma_t\sqrt{\Delta t}}_{\sigma_k}\,\xi, \tag*{(113)}
\end{aligned} x r = x t − Δ t v − 2 t ς t 2 Δ t ( x t + ( 1 − t ) v ) + ς t Δ t ξ = ( 1 − 2 t ς t 2 Δ t ) x t − ( 1 + 2 t ς t 2 ( 1 − t ) ) Δ t v + σ k ς t Δ t ξ , (113)
with ξ ∼ N ( 0 , I ) \xi\sim\mathcal{N}(0,I) ξ ∼ N ( 0 , I ) , which is Equation (95) . The choice Equation (110) of ς t \varsigma_t ς t , which avoids the singularity at the source endpoint, fixes the coefficients; at t 0 = 1 t_0=1 t 0 = 1 only the denominator of ς t 0 \varsigma_{t_0} ς t 0 is evaluated at t 1 t_1 t 1 , while α t \alpha_t α t and b t b_t b t retain the source level t 0 t_0 t 0 . The transition density is N ( α t x t − b t v , σ k 2 I ) \mathcal{N}(\alpha_t x_t-b_t v,\sigma_k^2I) N ( α t x t − b t v , σ k 2 I ) , whose mean is affine in v v v ; moreover, σ k > 0 \sigma_k>0 σ k > 0 on every transition of the grid.
Equivalence with the kernel of an instantaneous-velocity policy. For an average-velocity policy the MeanFlow identity, Equation (6) , gives u θ ( x t , t , s ) + ( t − s ) ∂ t u θ ( x t , t , s ) = v θ ( x t , t ) u_\theta(x_t,t,s)+(t-s)\partial_tu_\theta(x_t,t,s)=v_\theta(x_t,t) u θ ( x t , t , s ) + ( t − s ) ∂ t u θ ( x t , t , s ) = v θ ( x t , t ) for every s ∈ [ 0 , t ] s\in[0,t] s ∈ [ 0 , t ] . If J J J were computed from u θ u_\theta u θ , the velocity used in Equation (113) would therefore be the instantaneous velocity of the policy, the same for every s s s , and the transition density of FewStepGRPO would be that of FlowGRPO for the policy v θ v_\theta v θ . Computing J J J from u o l d u_{\mathrm{old}} u old , as every v-form trainer does (Stage 3 (ii)), makes this exact at θ = θ o l d \theta= \theta_{\mathrm{old}} θ = θ old , up to the error of Equation (109) ; for θ ≠ θ o l d \theta\ne\theta_{\mathrm{old}} θ = θ old the field is v θ − ( t − s ) ∂ t ( u θ − u o l d ) v_\theta- (t-s)\,\partial_t(u_\theta-u_{\mathrm{old}}) v θ − ( t − s ) ∂ t ( u θ − u old ) , which depends on s s s only through a term of the order of the drift of the policy from the old policy. Two consequences follow.
s s s enters the gradient through the entry of the flow map that is differentiated, ∂ μ θ / ∂ θ = − b t ∂ u θ ( x t , t , s ) / ∂ θ \partial\mu_\theta/\partial\theta= -b_t\,\partial u_\theta(x_t,t,s)/\partial\theta ∂ μ θ / ∂ θ = − b t ∂ u θ ( x t , t , s ) / ∂ θ , and the scored density only through that drift term. Redrawing s ∣ t s \mid t s ∣ t at every inner iteration therefore distributes the gradient of one transition over the entries u θ ( ⋅ , t , s ) u_\theta(\cdot,t,s) u θ ( ⋅ , t , s ) , s ∈ [ 0 , t ] s \in[0,t] s ∈ [ 0 , t ] , as Stage 3 (i) does for the trainers that re-noise, without changing what is scored at θ = θ o l d \theta= \theta_{\mathrm{old}} θ = θ old .
For any objective L ( μ θ ) \mathcal{L}(\mu_\theta) L ( μ θ ) , ∇ θ L = − b t ( ∂ u θ / ∂ θ ) ⊤ ∂ L / ∂ μ θ \nabla_\theta\mathcal{L} = -b_t\,(\partial u_\theta/\partial\theta)^\top\partial\mathcal{L}/\partial\mu_\theta ∇ θ L = − b t ( ∂ u θ / ∂ θ ) ⊤ ∂ L / ∂ μ θ , since J J J is detached (Equation (41) ). This equals the FlowGRPO gradient with ∂ v θ / ∂ θ \partial v_\theta/\partial\theta ∂ v θ / ∂ θ replaced by ∂ u θ ( ⋅ , t , s ) / ∂ θ \partial u_\theta(\cdot,t,s)/\partial\theta ∂ u θ ( ⋅ , t , s ) / ∂ θ .
The tempered scoring ratio. The ratio Equation (96) is the 1 / N 1/N 1/ N power of the Gaussian density ratio, equivalently the ratio at temperature N σ k 2 N\sigma_k^2 N σ k 2 .
The tempered ratio at zero drift. Since the same J J J is added to both means, μ θ − μ o l d = − b t ( u θ − u o l d ) ( x t , t , s ) \mu_\theta-\mu_{\mathrm{old}}=-b_t(u_\theta-u_{\mathrm{old}})(x_t,t,s) μ θ − μ old = − b t ( u θ − u old ) ( x t , t , s ) , and expanding the two squares of log ρ ˉ G R P O \log\bar{\rho}_{\mathrm{GRPO}} log ρ ˉ GRPO in Equation (96) gives
log ρ ˉ G R P O = − b t σ k 2 mean S P [ ( u θ − u o l d ) ⊙ ( x r − μ o l d ) ] − b t 2 2 σ k 2 mean S P [ ( u θ − u o l d ) 2 ] . (114) \log\bar{\rho}_{\mathrm{GRPO}} = -\frac{b_t}{\sigma_k^2}\operatorname{mean}_{\mathrm{SP}}\left[(u_\theta-u_{\mathrm{old}})\odot(x_r-\mu_{\mathrm{old}})\right]-\frac{b_t^2}{2\sigma_k^2}\operatorname{mean}_{\mathrm{SP}}\left[(u_\theta-u_{\mathrm{old}})^2\right]. \tag*{(114)} log ρ ˉ GRPO = − σ k 2 b t mean SP [ ( u θ − u old ) ⊙ ( x r − μ old ) ] − 2 σ k 2 b t 2 mean SP [ ( u θ − u old ) 2 ] . (114)
Hence ρ ˉ G R P O = 1 \bar{\rho}_{\mathrm{GRPO}}=1 ρ ˉ GRPO = 1 at θ = θ o l d \theta=\theta_{\mathrm{old}} θ = θ old , regardless of the error of J J J and of the kernel that generated x r x_r x r , because the same J J J enters both means and the same x r x_r x r enters both squares. The ratio differs from one only through u θ − u o l d u_\theta-u_{\mathrm{old}} u θ − u old .
Relation to the kernel that produced the action. In the notation of Equation (94) , x t = x ^ t j x_t=\hat{x}_{t_j} x t = x ^ t j and x r = x ^ t j + 1 x_r=\hat{x}_{t_{j+1}} x r = x ^ t j + 1 , and the action was drawn by the flow-map step to the level s j s_j s j drawn by the sampler, x ^ s j = x ^ t j − ( t j − s j ) u o l d ( x ^ t j , t j , s j ) \hat{x}_{s_j}=\hat{x}_{t_j}-(t_j-s_j)u_{\mathrm{old}}(\hat{x}_{t_j},t_j,s_j) x ^ s j = x ^ t j − ( t j − s j ) u old ( x ^ t j , t j , s j ) , followed by x ^ t j + 1 = 1 − t j + 1 1 − s j x ^ s j + τ j ϵ ′ \hat{x}_{t_{j+1}}=\frac{1-t_{j+1}}{1-s_j}\hat{x}_{s_j}+\tau_j\epsilon' x ^ t j + 1 = 1 − s j 1 − t j + 1 x ^ s j + τ j ϵ ′ . Given an exact flow-map jump to s j s_j s j , this rollout kernel preserves the linear-path marginal exactly: if x ^ s j = ( 1 − s j ) x + s j ϵ \hat{x}_{s_j}=(1-s_j)x+s_j\epsilon x ^ s j = ( 1 − s j ) x + s j ϵ , then x ^ t j + 1 = ( 1 − t j + 1 ) x + 1 − t j + 1 1 − s j s j ϵ + τ j ϵ ′ \hat{x}_{t_{j+1}}=(1-t_{j+1})x+\frac{1-t_{j+1}}{1-s_j}s_j\epsilon+\tau_j\epsilon' x ^ t j + 1 = ( 1 − t j + 1 ) x + 1 − s j 1 − t j + 1 s j ϵ + τ j ϵ ′ , a noise of variance t j + 1 2 t_{j+1}^2 t j + 1 2 by the definition of τ j \tau_j τ j . This kernel, however, differs from Equation (113) . Its mean is 1 − t j + 1 1 − s j ( x ^ t j − ( t j − s j ) u o l d ) \frac{1-t_{j+1}}{1-s_j}(\hat{x}_{t_j}-(t_j-s_j)u_{\mathrm{old}}) 1 − s j 1 − t j + 1 ( x ^ t j − ( t j − s j ) u old ) and its width is τ j \tau_j τ j , with τ j = 0 \tau_j=0 τ j = 0 whenever s j = t j + 1 s_j=t_{j+1} s j = t j + 1 , in which case the kernel is deterministic and has no likelihood ratio. On the four-step grid, with s j s_j s j drawn uniformly from the remaining grid levels, this happens for 52% of the transitions, including every final transition. FewStepGRPO therefore scores the action under Equation (113) , for three reasons: σ k > 0 \sigma_k>0 σ k > 0 on every transition; the mean is affine in the instantaneous velocity, so that the equivalence above holds; and the tempered ratio is exact at zero drift by Equation (114) . The approximation lies in the scored transition. Besides the noise τ j ϵ ′ \tau_j\epsilon' τ j ϵ ′ , both x r − μ o l d x_r-\mu_{\mathrm{old}} x r − μ old in Equation (114) and x r − μ θ x_r-\mu_\theta x r − μ θ in the gradient of §B.3 contain the deterministic difference between the means of the two kernels. Consequently, ρ ˉ G R P O \bar{\rho}_{\mathrm{GRPO}} ρ ˉ GRPO is a tempered ratio of the surrogate scoring kernels and not an exact importance ratio for the rollout kernel.
The decoupled loss
The decoupled loss reproduces the gradient of the tempered clipped surrogate. Per transition, define
L G R P O t e m p : = − min { A ˉ ρ ˉ G R P O , A ˉ clip ( ρ ˉ G R P O , 1 ± ϵ c l i p ) } . \mathcal{L}_{\mathrm{GRPO}}^{\mathrm{temp}} := -\min\{\bar{A}\bar{\rho}_{\mathrm{GRPO}},\ \bar{A}\operatorname{clip}(\bar{\rho}_{\mathrm{GRPO}},1\pm\epsilon_{\mathrm{clip}})\}. L GRPO temp := − min { A ˉ ρ ˉ GRPO , A ˉ clip ( ρ ˉ GRPO , 1 ± ϵ clip )} .
Where the first branch attains the minimum, its gradient is − A ρ ˉ G R P O ∇ θ log ρ ˉ G R P O -A\bar{\rho}_{\mathrm{GRPO}}\nabla_{\theta}\log\bar{\rho}_{\mathrm{GRPO}} − A ρ ˉ GRPO ∇ θ log ρ ˉ GRPO ; where the second does, it is constant in θ \theta θ and contributes no gradient. With the explicit gate χ G R P O \chi_{\mathrm{GRPO}} χ GRPO of Equation (96) and the per-element Gaussian log-score s θ G R P O : = − mean s p [ ( μ θ − x r ) 2 ] / ( 2 σ k 2 ) s_{\theta}^{\mathrm{GRPO}} := -\operatorname{mean}_{\mathrm{sp}}\left[(\mu_{\theta}-x_r)^2\right]/(2\sigma_k^2) s θ GRPO := − mean sp [ ( μ θ − x r ) 2 ] / ( 2 σ k 2 ) ,
∇ θ L G R P O t e m p = − χ G R P O A ρ ˉ G R P O N σ k 2 ( ∂ μ θ ∂ θ ) T ( x r − μ θ ) . (115) \nabla_{\theta}\mathcal{L}_{\mathrm{GRPO}}^{\mathrm{temp}}=-\frac{\chi_{\mathrm{GRPO}}A\bar{\rho}_{\mathrm{GRPO}}}{N\sigma_k^2}\left(\frac{\partial\mu_{\theta}}{\partial\theta}\right)^{\mathsf{T}}(x_r-\mu_{\theta}). \tag*{(115)} ∇ θ L GRPO temp = − N σ k 2 χ GRPO A ρ ˉ GRPO ( ∂ θ ∂ μ θ ) T ( x r − μ θ ) . (115)
Substituting Equation (97) into Equation (90) gives L r w d = W mean s p [ ( x ~ θ − sg ( x ~ θ ) − g ) 2 ] \mathcal{L}_{\mathrm{rwd}}=W\operatorname{mean}_{\mathrm{sp}}\left[(\tilde{x}_{\theta}-\operatorname{sg}(\tilde{x}_{\theta})-g)^2\right] L rwd = W mean sp [ ( x ~ θ − sg ( x ~ θ ) − g ) 2 ] with x ~ θ = μ θ \tilde{x}_{\theta}=\mu_{\theta} x ~ θ = μ θ and g = ω ( x r − sg ( μ θ ) ) g=\omega(x_r-\operatorname{sg}(\mu_{\theta})) g = ω ( x r − sg ( μ θ )) ; since x ~ θ − sg ( x ~ θ ) \tilde{x}_{\theta}-\operatorname{sg}(\tilde{x}_{\theta}) x ~ θ − sg ( x ~ θ ) vanishes in value and has the identity as Jacobian,
∇ θ L r w d = − 2 W ω N ( ∂ μ θ ∂ θ ) T ( x r − μ θ ) , \nabla_{\theta}\mathcal{L}_{\mathrm{rwd}}=-\frac{2W\omega}{N}\left(\frac{\partial\mu_{\theta}}{\partial\theta}\right)^{\mathsf{T}}(x_r-\mu_{\theta}), ∇ θ L rwd = − N 2 W ω ( ∂ θ ∂ μ θ ) T ( x r − μ θ ) ,
which, with W = ρ ˉ G R P O W=\bar{\rho}_{\mathrm{GRPO}} W = ρ ˉ GRPO and ω = χ G R P O A / ( 2 σ k 2 ) \omega=\chi_{\mathrm{GRPO}}A/(2\sigma_k^2) ω = χ GRPO A / ( 2 σ k 2 ) , is Equation (115) term by term. The equality holds for gradients; the value of L r w d \mathcal{L}_{\mathrm{rwd}} L rwd is W mean s p [ g 2 ] W\operatorname{mean}_{\mathrm{sp}}[g^2] W mean sp [ g 2 ] , as for every trainer (Stage 4).
Justification of each factor.
The per-element mean. A Gaussian log-density is a sum over the N N N non-batch elements of one latent; taking it as the mean mean s p [ ⋅ ] \operatorname{mean}_{\mathrm{sp}}[\cdot] mean sp [ ⋅ ] divides the exponent by N N N , which is why ρ ˉ G R P O \bar{\rho}_{\mathrm{GRPO}} ρ ˉ GRPO is the 1 / N 1/N 1/ N power of the density ratio, or equivalently a score at temperature N σ k 2 N\sigma_k^2 N σ k 2 . Here N = 65,536 N=65{,}536 N = 65 , 536 for the image latent and N = 2,096,640 N=2{,}096{,}640 N = 2 , 096 , 640 for the video latent. Using the mean keeps the ratio within floating-point range (with the sum, the exponent can overflow for differences allowed by the clip), follows FlowGRPO, and is consistent with a clipping radius of order 10 − 3 10^{-3} 1 0 − 3 .
The factor 1 / ( 2 σ k 2 ) 1/(2\sigma_k^2) 1/ ( 2 σ k 2 ) in ω \omega ω . The reward term of Equation (90) contributes 2 W ω / N 2W\omega/N 2 W ω / N times the displacement, so ω \omega ω must contain the factor 1 / ( 2 σ k 2 ) 1/(2\sigma_k^2) 1/ ( 2 σ k 2 ) of the log-score for Equation (115) to hold. Without it, each transition would be weighted by 2 σ k 2 2\sigma_k^2 2 σ k 2 , which varies by a factor of 2.7 across the four transitions of the grid; a single learning rate cannot compensate for such a transition-dependent scale.
x ~ b a s e = sg ( x ~ θ ) \tilde{x}_{\mathrm{base}}=\operatorname{sg}(\tilde{x}_{\theta}) x ~ base = sg ( x ~ θ ) . The score of the Gaussian at the action, ( x r − μ θ ) / σ k 2 (x_r-\mu_{\theta})/\sigma_k^2 ( x r − μ θ ) / σ k 2 , is measured from the current mean, so the origin of the reward direction has to follow the policy. A fixed origin μ o l d \mu_{\mathrm{old}} μ old , i.e. a translation as in DiffusionNFT, coincides with it only at θ = θ o l d \theta=\theta_{\mathrm{old}} θ = θ old .
Method-specific c a d v c_{\mathrm{adv}} c adv scaling. The implemented NFT and general OPSD trainers multiply both loss weights W W W by c a d v c_{\mathrm{adv}} c adv to undo the compression of Equation (84) ; the factor is written explicitly in Equation (106) and Equation (107) . ReFL keeps it although it fixes r ^ ≡ 1 \hat{r}\equiv1 r ^ ≡ 1 . FewStepGRPO does not use it, since ω \omega ω already contains the gated signed advantage χ G R P O A \chi_{\mathrm{GRPO}}A χ GRPO A .
FewStepOPSD
This appendix proves that, in the v-form of § 5.6 , the loss and its gradient do not depend on the detached term J J J . After dropping J J J , the current prediction and its anchors reduce to the J J J -free ( t , r ) (t,r) ( t , r ) query of § 4.3 , i.e., the average velocity queried at ( t , r ) (t,r) ( t , r ) and transported to the clean end without correction, while the reward-gradient steps still start from the r = 0 r = 0 r = 0 prediction. We follow one member and suppress the index ( i ) (i) ( i ) . Every prediction is evaluated at the query state x ~ t \tilde{x}_t x ~ t of level t t t and at the r r r drawn during the reward-coupled construction and reused at fitting (§ 5.6 , ( i ) (i) ( i ) ). Four properties of the implementation are used:
(a) the same detached estimate J = J [ u o l d , t , r ] J = J[u_{\mathrm{old}}, t, r] J = J [ u old , t , r ] is added to every prediction (Stage 3 (iii));
(b) the old policy u o l d u_{\mathrm{old}} u old is the same in the reward-coupled construction and in fitting, being refreshed only after the policy update of the outer iteration (Stage 1);
(c) the points x ~ ± \tilde{x}_{\pm} x ~ ± are constructed by Equation (70) ; the gradient steps start from the r = 0 r = 0 r = 0 prediction, and only the resulting displacement is added to the anchor;
(d) the velocity penalty, where present, is that of Equation (92) .
The quantities
By Equation (86) and Equation (69) , the current prediction and the anchors are
x ~ θ = x ~ t − t ( u θ ( x ~ t , t , r ) + J ) , x ~ r e g = x ~ b a s e , 1 = x ~ b a s e , 2 = x ~ i n d = x ~ t − t ( u o l d ( x ~ t , t , r ) + J ) . (116) \tilde{x}_{\theta} = \tilde{x}_t - t\left(u_{\theta}(\tilde{x}_t,t,r) + J\right),
\qquad
\tilde{x}_{\mathrm{reg}} = \tilde{x}_{\mathrm{base},1} = \tilde{x}_{\mathrm{base},2} = \tilde{x}_{\mathrm{ind}} = \tilde{x}_t - t\left(u_{\mathrm{old}}(\tilde{x}_t,t,r) + J\right). \tag*{(116)} x ~ θ = x ~ t − t ( u θ ( x ~ t , t , r ) + J ) , x ~ reg = x ~ base , 1 = x ~ base , 2 = x ~ ind = x ~ t − t ( u old ( x ~ t , t , r ) + J ) . (116)
By (c), the iteration Equation (62) starts at x ~ i n d + Δ = x ~ o l d , u \tilde{x}_{\mathrm{ind}} + \Delta= \tilde{x}_{\mathrm{old},u} x ~ ind + Δ = x ~ old , u , the r = 0 r = 0 r = 0 prediction of Equation (48) ; that is, x ~ o l d \tilde{x}_{\mathrm{old}} x ~ old in Equation (62) is replaced by x ~ o l d , u \tilde{x}_{\mathrm{old},u} x ~ old , u in the starting point x ~ ± [ 0 ] = x ~ o l d , u \tilde{x}_{\pm}^{[0]} = \tilde{x}_{\mathrm{old},u} x ~ ± [ 0 ] = x ~ old , u , in the step length h = η κ ∥ x ~ o l d , u ∥ / M t g t h = \eta\kappa\lVert\tilde{x}_{\mathrm{old},u}\rVert/M_{\mathrm{tgt}} h = η κ ∥ x ~ old , u ∥ / M tgt and in the trust region, the ball of radius κ ∥ x ~ o l d , u ∥ \kappa\lVert\tilde{x}_{\mathrm{old},u}\rVert κ ∥ x ~ old , u ∥ around x ~ o l d , u \tilde{x}_{\mathrm{old},u} x ~ old , u . The resulting points are x ~ ± = x ~ ± [ M t g t ] − Δ \tilde{x}_{\pm} = \tilde{x}_{\pm}^{[M_{\mathrm{tgt}}]} - \Delta x ~ ± = x ~ ± [ M tgt ] − Δ , so that their increments over the anchor, in the notation of Equation (63) , are
Δ r w d x ~ ± : = x ~ ± − x ~ i n d = x ~ ± [ M t g t ] − x ~ o l d , u . (117) \Delta_{\mathrm{rwd}}\tilde{x}_{\pm} := \tilde{x}_{\pm} - \tilde{x}_{\mathrm{ind}} = \tilde{x}_{\pm}^{[M_{\mathrm{tgt}}]} - \tilde{x}_{\mathrm{old},u}. \tag*{(117)} Δ rwd x ~ ± := x ~ ± − x ~ ind = x ~ ± [ M tgt ] − x ~ old , u . (117)
Cancellation of J J J
(1) the increments do not contain J J J .
Lemma C.1. Δ r w d x ~ ± \Delta_{\mathrm{rwd}}\tilde{x}_{\pm} Δ rwd x ~ ± is a function of x ~ o l d , u \tilde{x}_{\mathrm{old},u} x ~ old , u , of the reward R ( D ( ⋅ ) , c ) R(\mathcal{D}(\cdot),c) R ( D ( ⋅ ) , c ) and of the constants η , κ , M t g t , ε g \eta,\kappa,M_{\mathrm{tgt}},\varepsilon_g η , κ , M tgt , ε g alone; in particular it does not depend on J J J .
Proof. x ~ o l d , u = x ~ t − t u o l d ( x ~ t , t , 0 ) \tilde{x}_{\mathrm{old},u} = \tilde{x}_t - t u_{\mathrm{old}}(\tilde{x}_t,t,0) x ~ old , u = x ~ t − t u old ( x ~ t , t , 0 ) contains no J J J . If x ~ ± [ m ] \tilde{x}_{\pm}^{[m]} x ~ ± [ m ] does not, neither does the reward gradient evaluated at it, nor the step length h h h , nor the centre and radius of the trust region, all of which are evaluated at x ~ o l d , u \tilde{x}_{\mathrm{old},u} x ~ old , u ; hence x ~ ± [ m + 1 ] \tilde{x}_{\pm}^{[m+1]} x ~ ± [ m + 1 ] does not. By induction x ~ ± [ M t g t ] \tilde{x}_{\pm}^{[M_{\mathrm{tgt}}]} x ~ ± [ M tgt ] does not, and neither does Equation (117) .
The shift Δ = x ~ o l d , u − x ~ i n d = t ( u o l d ( x ~ t , t , r ) + J − u o l d ( x ~ t , t , 0 ) ) \Delta= \tilde{x}_{\mathrm{old},u} - \tilde{x}_{\mathrm{ind}} = t\left(u_{\mathrm{old}}(\tilde{x}_t,t,r) + J - u_{\mathrm{old}}(\tilde{x}_t,t,0)\right) Δ = x ~ old , u − x ~ ind = t ( u old ( x ~ t , t , r ) + J − u old ( x ~ t , t , 0 ) ) does contain J J J ; it enters x ~ ± \tilde{x}_{\pm} x ~ ± only as the offset subtracted from the end point of the iteration, and cancels in Equation (117) .
(2) the self-regression residual. By (a) the two copies of J J J in Equation (116) are one detached vector, and
x ~ θ − x ~ r e g = − t ( u θ − u o l d ) ( x ~ t , t , r ) , (118) \tilde{x}_{\theta} - \tilde{x}_{\mathrm{reg}} = -t\left(u_{\theta} - u_{\mathrm{old}}\right)(\tilde{x}_t,t,r), \tag*{(118)} x ~ θ − x ~ reg = − t ( u θ − u old ) ( x ~ t , t , r ) , (118)
which is Equation (71) .
(3) the weights. The normalisers of Equation (105) , evaluated at x ~ ± \tilde{x}_{\pm} x ~ ± as in § 5.6 (iv), take the branch residuals
x ~ b a s e ± β ( sg ( x ~ θ ) − x ~ r e g ) − x ~ ± = ± β ( sg ( x ~ θ ) − x ~ r e g ) − Δ r w d x ~ ± = ∓ β t ( u θ − u o l d ) − Δ r w d x ~ ± , \tilde{x}_{\mathrm{base}} \pm\beta\left(\operatorname{sg}(\tilde{x}_{\theta})-\tilde{x}_{\mathrm{reg}}\right)-\tilde{x}_{\pm}=\pm\beta\left(\operatorname{sg}(\tilde{x}_{\theta})-\tilde{x}_{\mathrm{reg}}\right)-\Delta_{\mathrm{rwd}}\tilde{x}_{\pm}=\mp\beta t(u_{\theta}-u_{\mathrm{old}})-\Delta_{\mathrm{rwd}}\tilde{x}_{\pm}, x ~ base ± β ( sg ( x ~ θ ) − x ~ reg ) − x ~ ± = ± β ( sg ( x ~ θ ) − x ~ reg ) − Δ rwd x ~ ± = ∓ β t ( u θ − u old ) − Δ rwd x ~ ± ,
by x ~ b a s e = x ~ i n d \tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{ind}} x ~ base = x ~ ind , Equation (117) and Equation (118) . By Lemma C.1 neither term contains J J J ; hence neither do γ ± \gamma_{\pm} γ ± , a + = r ^ / γ + a_{+}=\hat{r}/\gamma_{+} a + = r ^ / γ + , a − = ( 1 − r ^ ) / γ − a_{-}=(1-\hat{r})/\gamma_{-} a − = ( 1 − r ^ ) / γ − , nor the weights W W W , ω 1 \omega_{1} ω 1 , ω 2 \omega_{2} ω 2 of Equation (107) . r ^ \hat{r} r ^ depends only on the rewards of the rollouts (Stage 2).
(4) the reward displacements. From Equation (67) ,
g 1 = ω 1 ( x ~ + − x ~ i n d ) = ω 1 Δ r w d x ~ + , g 2 = ω 2 ( x ~ − − x ~ i n d ) = ω 2 Δ r w d x ~ − , (119) g_{1}=\omega_{1}\left(\tilde{x}_{+}-\tilde{x}_{\mathrm{ind}}\right)=\omega_{1}\Delta_{\mathrm{rwd}}\tilde{x}_{+},\qquad
g_{2}=\omega_{2}\left(\tilde{x}_{-}-\tilde{x}_{\mathrm{ind}}\right)=\omega_{2}\Delta_{\mathrm{rwd}}\tilde{x}_{-}, \tag*{(119)} g 1 = ω 1 ( x ~ + − x ~ ind ) = ω 1 Δ rwd x ~ + , g 2 = ω 2 ( x ~ − − x ~ ind ) = ω 2 Δ rwd x ~ − , (119)
neither of which contains J J J .
(5) the Jacobian. J J J is detached, so by Equation (41) ∂ x ~ θ / ∂ θ = − t ∂ u θ ( x ~ t , t , r ) / ∂ θ \partial\tilde{x}_{\theta}/\partial\theta=-t\,\partial u_{\theta}(\tilde{x}_{t},t,r)/\partial\theta ∂ x ~ θ / ∂ θ = − t ∂ u θ ( x ~ t , t , r ) / ∂ θ .
(6) the loss and its gradient. Each reward square of Equation (68) equals W mean s p [ g k 2 ] W\operatorname{mean}_{\mathrm{sp}}[g_{k}^{2}] W mean sp [ g k 2 ] in value (Stage 4), so the value of the loss is
L = W mean s p [ ( x ~ θ − x ~ r e g ) 2 ] + W mean s p [ g 1 2 ] + W mean s p [ g 2 2 ] , (120) \mathcal{L}=W\operatorname{mean}_{\mathrm{sp}}\left[(\tilde{x}_{\theta}-\tilde{x}_{\mathrm{reg}})^{2}\right]+W\operatorname{mean}_{\mathrm{sp}}\left[g_{1}^{2}\right]+W\operatorname{mean}_{\mathrm{sp}}\left[g_{2}^{2}\right], \tag*{(120)} L = W mean sp [ ( x ~ θ − x ~ reg ) 2 ] + W mean sp [ g 1 2 ] + W mean sp [ g 2 2 ] , (120)
and its gradient, which is Equation (66) written with the per-sample mean, is by Steps 2–5
∇ θ L = − 2 t W N ( ∂ u θ ( x ~ t , t , r ) ∂ θ ) T [ − t ( u θ − u o l d ) − ω 1 Δ r w d x ~ + − ω 2 Δ r w d x ~ − ] . (121) \nabla_{\theta}\mathcal{L}=-\frac{2tW}{N}\left(\frac{\partial u_{\theta}(\tilde{x}_{t},t,r)}{\partial\theta}\right)^{\mathsf{T}}\left[-t(u_{\theta}-u_{\mathrm{old}})-\omega_{1}\Delta_{\mathrm{rwd}}\tilde{x}_{+}-\omega_{2}\Delta_{\mathrm{rwd}}\tilde{x}_{-}\right]. \tag*{(121)} ∇ θ L = − N 2 t W ( ∂ θ ∂ u θ ( x ~ t , t , r ) ) T [ − t ( u θ − u old ) − ω 1 Δ rwd x ~ + − ω 2 Δ rwd x ~ − ] . (121)
Proposition C.2. The value and the gradient of the v v v -form FewStepOPSD loss, Equation (120) and Equation (121) , do not depend on J J J . In particular, they are those obtained with J = 0 J=0 J = 0 .
Proof. Every factor of Equation (120) and Equation (121) is one of x ~ θ − x ~ r e g \tilde{x}_{\theta}-\tilde{x}_{\mathrm{reg}} x ~ θ − x ~ reg , g 1 g_{1} g 1 , g 2 g_{2} g 2 , W W W , ω 1 \omega_{1} ω 1 , ω 2 \omega_{2} ω 2 , Δ r w d x ~ ± \Delta_{\mathrm{rwd}}\tilde{x}_{\pm} Δ rwd x ~ ± and ∂ x ~ θ / ∂ θ \partial\tilde{x}_{\theta}/\partial\theta ∂ x ~ θ / ∂ θ , none of which contains J J J by Steps 1–5. The penalty of Equation (92) compares u θ u_{\theta} u θ and u r e f u_{\mathrm{ref}} u ref at the same ( x ~ t , t , r ) (\tilde{x}_{t},t,r) ( x ~ t , t , r ) and contains no J J J either.
Corollary C.3. Dropping J J J in Equation (116) gives
x ~ θ = X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] , x ~ r e g = x ~ b a s e = x ~ i n d ∣ J = 0 = X [ x ~ t , u o l d ( x ~ t , t , r ) , 0 ] , \tilde{x}_{\theta}=X\left[\tilde{x}_{t},u_{\theta}(\tilde{x}_{t},t,r),0\right],\qquad
\tilde{x}_{\mathrm{reg}}=\tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{ind}}\big|_{J=0}=X\left[\tilde{x}_{t},u_{\mathrm{old}}(\tilde{x}_{t},t,r),0\right], x ~ θ = X [ x ~ t , u θ ( x ~ t , t , r ) , 0 ] , x ~ reg = x ~ base = x ~ ind J = 0 = X [ x ~ t , u old ( x ~ t , t , r ) , 0 ] , x ~ ± = x ~ b a s e + Δ r w d x ~ ± , \tilde{x}_{\pm}=\tilde{x}_{\mathrm{base}}+\Delta_{\mathrm{rwd}}\tilde{x}_{\pm}, x ~ ± = x ~ base + Δ rwd x ~ ± , with Δ r w d x ~ ± \Delta_{\mathrm{rwd}}\tilde{x}_{\pm} Δ rwd x ~ ± computed from x ~ o l d , u \tilde{x}_{\mathrm{old},u} x ~ old , u . These are the J J J -free ( t , r ) (t,r) ( t , r ) queries of § 4.3 ; they coincide with the u u u -form quantities of § 5.6 (ii) except that the current prediction and its anchors are queried at ( t , r ) (t,r) ( t , r ) instead of ( t , 0 ) (t,0) ( t , 0 ) ; the reward displacement is still computed from x ~ o l d , u \tilde{x}_{\mathrm{old},u} x ~ old , u . At r = 0 r=0 r = 0 the two constructions coincide, since then x ~ b a s e = x ~ o l d , u \tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{old},u} x ~ base = x ~ old , u and Δ = 0 \Delta=0 Δ = 0 . Freeing r r r therefore trains every u θ ( ⋅ , t , r ) u_{\theta}(\cdot,t,r) u θ ( ⋅ , t , r ) while the decoder and the reward gradient are still evaluated at the clean prediction x ~ o l d , u \tilde{x}_{\mathrm{old},u} x ~ old , u ; the u u u -form trains only u θ ( ⋅ , t , 0 ) u_{\theta}(\cdot,t,0) u θ ( ⋅ , t , 0 ) .
Remark C.4 (The cancellation as a translation). Steps 1–4 state that J J J shifts the five points x ~ θ \tilde{x}_{\theta} x ~ θ , x ~ r e g \tilde{x}_{\mathrm{reg}} x ~ reg , x ~ b a s e \tilde{x}_{\mathrm{base}} x ~ base , x ~ + \tilde{x}_{+} x ~ + , x ~ − \tilde{x}_{-} x ~ − by the same detached vector − t J -tJ − t J (with x ~ ± \tilde{x}_{\pm} x ~ ± shifted through the offset Δ \Delta Δ of Equation (70) ), whereas the loss depends on them only through x ~ θ − x ~ r e g \tilde{x}_{\theta}-\tilde{x}_{\mathrm{reg}} x ~ θ − x ~ reg and x ~ ± − x ~ b a s e \tilde{x}_{\pm}-\tilde{x}_{\mathrm{base}} x ~ ± − x ~ base . The two quantities of the construction that depend on an absolute position, the points at which ∇ R \nabla R ∇ R is evaluated and the radius of the trust region, are evaluated at x ~ o l d , u \tilde{x}_{\mathrm{old},u} x ~ old , u , which does not depend on J J J .
Cost, and an empirical check
J J J requires one directional-difference estimate Equation (87) , i.e. two no-grad forwards of u o l d u_{\mathrm{old}} u old . With J J J retained, the v-form evaluates it twice per member and outer iteration: in the reward-coupled construction, to form x ~ i n d \tilde{x}_{\mathrm{ind}} x ~ ind , where the members are processed in micro-batches of six members to bound the memory of the reward backward pass; and in fitting, to form x ~ θ \tilde{x}_{\theta} x ~ θ and x ~ r e g \tilde{x}_{\mathrm{reg}} x ~ reg . By Corollary C.3 neither evaluation is needed. Table S1 compares two runs of the v-form that differ only in this respect, and the same pair for ReFL with ∇ R \nabla R ∇ R of the full reward. The training reward agrees to within 0.002 for the v-form at every evaluated iteration, and to within 0.006 for ReFL at iteration 400, below the 1% (about 0.016) by which two evaluation draws of one checkpoint may differ; held-out OCR accuracy differs more.
Supplementary Experiments
Shared hyperparameters
The following values are shared by every trainer of § 6.1 and § 6.2 and complement the settings listed there.
Old-policy EMA. The old policy u o l d u_{\mathrm{old}} u old is refreshed once per outer iteration i i i by θ o l d ← η i θ o l d + ( 1 − η i ) θ \theta_{\mathrm{old}} \leftarrow\eta_i \theta_{\mathrm{old}} + (1-\eta_i)\theta θ old ← η i θ old + ( 1 − η i ) θ , with the decay schedule of the official DiffusionNFT implementation (Zheng et al., 2026a): the decay ratio η i = min { 10 − 3 i , 0.5 } \eta_i = \min\{10^{-3i}, 0.5\} η i = min { 1 0 − 3 i , 0.5 } increases linearly with the outer iteration i i i to a cap of 0.5. Since η i > 0 \eta_i > 0 η i > 0 for i ≥ 1 i \ge1 i ≥ 1 , the condition θ o l d = θ \theta_{\mathrm{old}} = \theta θ old = θ of Example 9 holds only at the first outer iteration.
Floors. The floors are: the residual-scale floor τ min = 10 − 2 \tau_{\min} = 10^{-2} τ m i n = 1 0 − 2 of FewStepAWM (Equation (101) ), the directional-difference step ϵ c d = 5 × 10 − 3 \epsilon_{\mathrm{cd}} = 5 \times10^{-3} ϵ cd = 5 × 1 0 − 3 (Equation (87) ), the adaptive-normaliser floor ϵ γ = 10 − 5 \epsilon_\gamma= 10^{-5} ϵ γ = 1 0 − 5 (Equation (105) ), the reward-gradient-norm floor ϵ g = 10 − 12 \epsilon_g = 10^{-12} ϵ g = 1 0 − 12 (Equation (62) ) and the advantage standard-deviation floor ϵ s t d = 10 − 4 \epsilon_{\mathrm{std}} = 10^{-4} ϵ std = 1 0 − 4 (Equation (83) ).
GenEval2. Following MeanFlowNFT [34 ] , we adopt the official GenEval2 evaluation setting (Kamath et al., 2025).
Table S2. (a) Quality dimensions (Q).
Table S2 (PDF p. 82). (b) Semantic dimensions (S).
Table S2 (PDF p. 82, second). (c) Overall.
Quality dimensions: DD dynamic degree, SC subject consistency, MS motion smoothness, TF temporal flickering, BC background consistency, AQ aesthetic quality, IQ imaging quality. Semantic dimensions: OC object class, MO multiple objects, HA human action, Co color, SR spatial relationship, Sc scene, AS appearance style, TS temporal style, OCs overall consistency. * marks the eight dimensions rescaled by fixed min–max ranges beforehand; their raw values differ substantially and are not comparable with these. quality: the seven Q dimensions aggregated with DD at weight 0.5 and the other six at weight 1, divided by 6.5 (the VBench convention); semantic: the nine S dimensions; overall = 0.8 quality + 0.2 semantic.
Table S2: All 16 VBench dimensions at iteration 250, normalised (%), i.e. the values entering the aggregates of Table 8 . (a) Quality and (b) semantic dimensions, each with its aggregate and, below, the mean and spread over the seven methods. (c) Training reward against the VBench aggregates. Only DD (6.29) and MO (2.95) have a spread above 2.6; the remaining dimension spreads are at most 2.52. In the method rows, bold marks the highest per column; in the spread rows, it marks values above 2.6.
Analysis of no gain in CLIPScore for group-advantage runs
In the mixed-reward setting, all five default group-advantage runs end below the pre-RL CLIPScore. The two runs with the full reward gradient instead increase CLIPScore at every listed 100-iteration checkpoint, to 0.3217 and 0.3172. In contrast, all three CLIP-focused MeanFlowNFT ablations in the lower block of Table S4 end above the pre-RL value.
Table S4. CLIPScore against the iteration in the mixed-reward setting, the summand of the training reward on which the two groups of trainers disagree. Upper block: the update direction contains ∇ R \nabla R ∇ R . Lower block: the reward enters as the scalar A A A or r ^ \hat r r ^ . Evaluation protocol as in § 6.1 ; the HPSv2-only ReFL row uses a different training reward. Iteration 0 is the shared pre-RL checkpoint, CLIPScore 0.2886.
The training reward weights its three summands equally, but CLIPScore varies little within a prompt group, where the advantage is computed; at the pre-RL policy, its within-group standard deviation is only about 0.01 (0.010 and 0.014 on the first iteration of the two CLIPScore-only runs below), roughly a quarter of its global-batch standard deviation and 3.5–5% of its value. It accounts for 30.4% of the variance of the three summands, against 53.6% for PickScore and 16.0% for HPSv2, so an equal weight does not give it an equal share of the signal.
We therefore tested three ways of changing that share, the first two on MeanFlowNFT (Table S4 , lower block). (i) Raising the weight of CLIPScore: the weights ( w C L I P , w H P S , w P i c k ) = ( 2.4 , 0.3 , 0.3 ) (w_{\mathrm{CLIP}}, w_{\mathrm{HPS}}, w_{\mathrm{Pick}}) = (2.4, 0.3, 0.3) ( w CLIP , w HPS , w Pick ) = ( 2.4 , 0.3 , 0.3 ) , i.e. 8:1:1, and (3.0, 0, 0), CLIPScore alone, the limit of an unbounded relative weight; both keep K = 24 K = 24 K = 24 . (ii) Enlarging the group: CLIPScore alone with K = 72 K = 72 K = 72 images per prompt instead of 24. (iii) Removing it from the gradient: ReFL differentiating HPSv2 alone. The first two increase CLIPScore, and the increase grows with its share; by iteration 400, it gains 0.0133 at 8:1:1, 0.0191 with CLIPScore alone, and 0.0363 with K = 72 K = 72 K = 72 , more than either gradient-based trainer (0.0286–0.0331), although this run draws three times as many images per prompt. In all three runs, however, the other two summands stop improving, and HPSv2 and PickScore oscillate within [0.2473, 0.2974] and [0.8459, 0.8691], the range they occupy at the start of training; under CLIPScore alone the held-out aesthetic score and OCR accuracy also decrease, by 0.64–0.98 and 0.18–0.19. Conversely, ReFL with the gradient of HPSv2 alone leaves CLIPScore near its pre-RL value (0.2844–0.2883 over iterations 100–400) while raising HPSv2 to 0.3946, the highest of any reported variant.
In these ablations, increasing CLIPScore under a scalar advantage coincides with the gains of the other summands weakening or vanishing. The scalar is shared according to the covariance of each summand with the group reward, so reweighting redistributes the observed signal. The reward gradient instead contains a direction from every summand; in the reported FewStepOPSD and ReFL runs, CLIPScore, HPSv2 and PickScore rise together by 0.0286–0.0331, approximately 0.0705–0.0764, and approximately 0.0738–0.0818, respectively. As noted in the main text, this approach requires a reward that is differentiable with respect to the clean sample.
Conclusion. In the reported group-advantage ablations, raising the share of CLIPScore or enlarging the group raises CLIPScore while the gains of the other summands are reduced or vanish. In these runs, the weighting of the summands therefore affects which rewards improve, not only how fast they improve. Reward-gradient-based methods provide a direction from every summand, and all three summands rise together in the reported runs; however, these methods apply only when the reward is differentiable with respect to the clean sample, which excludes the OCR setting (§6).
Why FewStepOPSD resembles ReFL
In FewStepOPSD, the signed reward coefficients satisfy ∣ ω 1 ∣ + ∣ ω 2 ∣ = 1 / β |\omega_1| + |\omega_2| = 1/\beta ∣ ω 1 ∣ + ∣ ω 2 ∣ = 1/ β ; r ^ \hat r r ^ redistributes this magnitude between the ascent and descent terms and can also change their common weight W W W (Equation (107) ). Both terms move the prediction in the direction of increasing reward, because g 1 = ω 1 Δ r w d x ~ + g_1 = \omega_1\Delta_{\mathrm{rwd}}\tilde{x}_{+} g 1 = ω 1 Δ rwd x ~ + and g 2 = ω 2 Δ r w d x ~ − g_2 = \omega_2\Delta_{\mathrm{rwd}}\tilde{x}_{-} g 2 = ω 2 Δ rwd x ~ − with ω 2 ≤ 0 \omega_2 \le0 ω 2 ≤ 0 , while the two increments follow + ∇ R +\nabla R + ∇ R and − ∇ R -\nabla R − ∇ R , respectively. The logged mean magnitudes ∣ ω 1 ∣ = 5.12 |\omega_1| = 5.12 ∣ ω 1 ∣ = 5.12 and ∣ ω 2 ∣ = 4.88 |\omega_2| = 4.88 ∣ ω 2 ∣ = 4.88 differ by 0.24, i.e. 2.4% of their fixed sum 1 / β = 10 1/\beta= 10 1/ β = 10 ; they do not by themselves determine the imbalance of the two reward-augmentation vectors. FewStepOPSD is therefore close to the ReFL reward augmentation of Example 9, with an added self-regression, a descent term and several gradient steps; this is consistent with the similar curves in Figure 4 .
The velocity penalty: KL-u against KL-v
The velocity penalty, labelled KL in the run names although it is a squared velocity residual (§5.2), admits two constructions: the fixed initial average-velocity reference of Equation (92) , which compares u θ u_\theta u θ and u r e f u_{\mathrm{ref}} u ref at the same ( x ~ t , t , r ) (\tilde{x}_t,t,r) ( x ~ t , t , r ) (KL-u), and the frozen instantaneous teacher of Equation (93) , converted through J [ u o l d , t , r ] \mathcal{J}[u_{\mathrm{old}},t,r] J [ u old , t , r ] , which depends on the old policy (KL-v). The variants therefore differ in both reference construction and parameterisation space, and the weight is 10 − 4 10^{-4} 1 0 − 4 in both; the main text reports KL-u under the name MeanFlowNFT.
In the mixed-reward setting, the two variants are close at iteration 400; both reach a training reward of 1.509, and the held-out evaluators differ by about 1–1.5% in either direction (HPSv3 13.23 against 13.36, ImageReward 1.385 against 1.365), with neither variant ahead on both. GenEval2 is 0.407 against 0.400 at iteration 400. Its average over iterations 100–400 is 0.393 against 0.365; the 0.028 gap is near the stated ∼ \sim ∼ 0.03 resolution, so we do not interpret it as a clear difference.
In the OCR setting, KL-u reaches OCR accuracy 0.85 earlier, at iteration 100 instead of 150, with 0.785, 0.850, 0.945 at iterations 50, 100, 150 (the values of Table 6a) against 0.693, 0.834, 0.897. At matched OCR accuracy the difference is mixed; at the first saved checkpoint with OCR accuracy of at least 0.82, HPSv3 is 10.64 against 10.24 and ImageReward 0.887 against 0.896, each variant ahead on one of the two. At this weight, the choice of reference therefore has a limited effect in these runs; it changes the iteration at which the OCR threshold is reached but has no clear effect on the quality at that point, and both variants have nearly the same cost per iteration (435 against 436 s in the mixed-reward setting). The main text therefore reports only KL-u.
Partial differentiability of the video reward
This restriction does not occur in the image setting and affects only FewStepOPSD and ReFL.
Every method is evaluated on the same four-term reward. A method that requires ∇ R \nabla R ∇ R can only use the differentiable terms, which on this backbone are HPSv3-general (39% of the sum) and HPSv3-percentile (57%); VideoAlign MQ (6%) and TA (−2%) are not differentiable. The FewStepOPSD and ReFL runs in Table 8 use the gradients of both HPSv3 reductions, together 96% of the aggregate under the logged normalisation, but no gradient of motion quality or text alignment. On SD3.5, the gradient covers 100% of the training reward, since all three summands are differentiable image scorers.
This restriction is not visible in the reported scalar rewards, since the reward, the online evaluation and the advantage of FewStepOPSD all use the full four-term scorer (ReFL has no advantage). The restriction affects the direction of the gradient and not necessarily the norm of the update.
The released bfloat16 HPSv3 scorer changes the reward value by only ∼ \sim ∼ 0.003 relative to fp32 but rotates its gradient by 76.2 degrees on average. The runs in Table 8 therefore use fp32 with the sdpa attention backend, so this bf16 error does not affect the reported results. One remaining constraint is that the clip is decoded one latent frame at a time to limit memory; since the VAE is causal, a frame decoded alone is treated as the first frame of a clip. Compared with a joint decode of the same two frames, the gradient deviates by 79.2 degrees, and the gradient of the second frame is nearly orthogonal to the joint-decode reference.
The FewStepOPSD and ReFL rows of Table 8 therefore use the gradient of the two HPSv3 terms with per-frame decoding, and not the gradient of the full reward. Removing the causal-decode constraint requires a joint window of at least two frames, which needs 126.1 GB of GPU memory without activation offloading on a 144 GB card with 25.6 GB already in use, or 189 s per iteration with offloading. Across 48 GPUs and 250 iterations this amounts to 26.25 GPU-days (about 0.547 days of wall-clock time if fully parallel). This limitation comes from the memory of a single GPU and not from the method.