WorksComputer Science
Variational Optimal Control for Diffusion Guidance
Abstract
We present a variational control framework that connects sampling-time guidance and reward-based model adaptation through a shared Jacobian pullback. Starting from terminal reward maximization with a path-space KL penalty, the Gibbs variational principle and Doob transform yield a correction to the generative dynamics. A local denoising approximation exposes the pullback that transports output preferences to the controlled variable. This construction gives Variational Optimal Guidance (VOG) in state space and extends to Distilled Variational Optimal Control (DVOC) by amortizing locally improved endpoints into model parameters. The resulting controls share a local variational rule: VOG acts through the denoising Jacobian, while DVOC realizes the parameter pullback by endpoint distillation. Under a linear MMSE model, VOG's exact correction lies in the data subspace. On ImageNet, VOG reduces DiT-XL/2 FID from 2.27 to 2.04 at 250 sampling steps and from 7.58 to 4.44 at 25 steps. After 20 updates on Z-Image-Turbo, DVOC improves HPSv3++ and four held-out evaluator scores over the untrained model, with the highest corresponding means among the trained methods compared. These results connect inference-time control and reward adaptation through a common geometric principle.
Introduction
Diffusion models have become powerful image generators by learning to transform noise into samples from a data distribution [17, 19, 25, 34]. In many applications, however, faithfully reproducing that distribution is only a starting point: users seek images with particular content, stronger prompt alignment, or preferred visual attributes. These goals call for adapting a pretrained generator to an objective that may differ from its training criterion. The challenge is to improve the desired objective while retaining the quality and diversity supported by the learned prior.
Sampling-time guidance and reward-based fine-tuning offer two ways to pursue this goal. Classifier-free guidance (CFG) [16] strengthens conditioning by extrapolating between conditional and unconditional predictions, while gradient-based methods steer sampling using an objective evaluated on a predicted clean image [1, 4, 45]. Fine-tuning instead updates model parameters through reward backpropagation or reinforcement learning, incorporating preferences into future generations [2, 6, 27, 42]. Stronger intervention is not always beneficial: increasing the CFG scale can reduce distributional coverage [3, 30], and an external objective can favor directions that depart from the data structure [13].
A central difficulty is that preferences are defined on a completed image, whereas interventions act on noisy intermediate states or model parameters. Translating a desired output change into an effective intervention requires accounting for how the generator responds. We therefore ask how a terminal objective and a cost of intervention should determine the control, and whether a common local principle can govern both sampling-time guidance and parameter adaptation.
We address these questions through variational optimal control (VOC). For sampling, the pretrained conditional process provides a reference, and control maximizes terminal reward while penalizing deviation from that reference through a path-space KL divergence. This objective makes the trade-off between preference and prior explicit. Its optimal solution reweights future denoising trajectories by terminal reward and determines a corresponding correction to the dynamics. A local Gaussian approximation and a denoised reward proxy then yield Variational Optimal Guidance (VOG): the output gradient is transported to the current state through the denoising Jacobian. This Jacobian accounts for which changes in the noisy state affect the predicted clean sample. Under a linear subspace model with an optimal denoiser, the exact correction lies in the data subspace; the practical sampler approximates this pullback for neural denoisers.
The local control principle also suggests how to incorporate terminal improvements into the generator. We introduce Distilled Variational Optimal Control (DVOC), which first refines a sampled terminal latent using reward gradients within a bounded neighborhood, then trains the model to reproduce the resulting target. Differentiating the distillation loss transports the target displacement through a parameter Jacobian, realizing the parameter-space counterpart of the local VOG control. Selected denoising steps carry this parameter gradient, and the adapted model uses its ordinary sampler at inference. VOG and DVOC thus share a concrete mechanism: a preferred output change is mapped through the sensitivity of the controlled variable, acting on the current state in VOG and on model parameters in DVOC (Figure 1).
Figure 1. Variational control for sampling-time guidance and model adaptation. The gray curve illustrates generation from noise to the sampled endpoint. VOG (blue) evaluates the objective at the denoised prediction and maps its gradient through to correct the current sampling state. DVOC (pink) caches the trajectory, refines the actual endpoint within a local neighborhood, and distills the fixed target into LoRA parameters. The three ringed states supply the selected velocity evaluations () used for endpoint distillation. Target refinement uses two projected reward-ascent steps ().
Our contributions are:
A variational account of diffusion guidance. We derive VOG from terminal reward maximization with a path-space KL penalty and characterize the subspace support of its exact correction under a linear MMSE model.
A connection between guidance and model adaptation. We identify a shared local control principle and derive DVOC’s endpoint distillation objective as its parameter-space realization.
Evaluation across guidance and reward adaptation. VOG improves FID in all twelve ImageNet model–guidance pairings and reduces 25-step DiT-XL/2 FID from 7.58 to 4.44. After 20 updates, DVOC improves HPSv3++ and four held-out evaluator scores over pretrained Z-Image-Turbo, with the highest corresponding means among the trained methods compared.
Related Work
Sampling-time guidance. Classifier guidance [9] steers sampling with a learned classifier gradient, while CFG [16] combines conditional and unconditional predictions. Subsequent methods adjust guidance direction, strength, or schedule [5, 18, 21, 29, 30, 41]. Manifold-aware guidance uses denoiser geometry [13, 14]. DPS, FreeDoM, and Universal Guidance propagate objectives through a clean prediction [1, 4, 45]. We derive state corrections from a terminal control objective, with their geometry determined by the denoising Jacobian.
Reward-based model adaptation. DDPO and DPOK use policy gradients [2, 11], while Diffusion-DPO learns from preference pairs [36]. Flow-GRPO [22] adapts group-relative policy optimization to flow matching via stochastic rollouts. DiffusionNFT [46] contrasts positive and negative generations through forward-process flow matching. Advantage Weighted Matching (AWM) [43] reweights score/flow-matching pretraining losses by sample advantages. ReFL, DRaFT, AlignProp, and DRTune backpropagate differentiable rewards [6, 27, 40, 42]. FlowBP [37] uses a lightweight backward trajectory, and DiffusionPSD [47] distills reward-improved clean outputs. We formulate reward adaptation as amortized control through the same local variational principle and Jacobian pullback as sampling-time guidance.
Variational control of generative models. Stochastic optimal control balances terminal reward against deviation from a reference process. Adjoint Matching [10] expresses reward adaptation as control regression. DEFT and h-Edit [8, 24] use Doob transforms for conditioning and editing. Within our variational framework, terminal preferences yield state and parameter corrections through Jacobian pullbacks. VOG and DVOC realize these two forms of control as trajectory guidance and its amortization in the generator.
Variational Optimal Control and Model Geometry
From terminal preferences to control. VOG and DVOC translate terminal preferences into corrections through the sensitivity of the controlled variable. VOG differentiates the denoised prediction with respect to the current state. DVOC differentiates the generated endpoint with respect to model parameters. We first derive the state-space control from the reference process, then develop parameter adaptation from the same local variational principle in § 3.5.
To analyze VOG’s geometry, we adopt a low-dimensional linear-subspace model [13, 26]. The model gives an explicit characterization of the denoising Jacobian.
Assumption 3.1 (Linear Latent Subspace). We assume the clean data resides strictly within a low-dimensional linear subspace of intrinsic dimension . Specifically, the data is generated as , where is an orthogonal basis matrix satisfying , and the latent variable follows a standard Gaussian prior . Consequently, the data support is defined as .
Let and . Every vector decomposes into components parallel and normal to this subspace.
How CFG Shifts the Mean and Reduces Coverage
CFG extrapolates between conditional and unconditional predictions. On the branching-tree example in Figure 2, stronger guidance suppresses outliers but reduces branch coverage. A local probability-flow analysis explains how the prediction contrast can shift the generated distribution. Its assumptions and derivation appear in Appendix § 10.

Figure 2. Toy class-conditional generation on fractal-like 2D branching-tree distributions [18] under two increasingly challenging settings: a two-class benchmark (CLS = 2) and a three-class variant with shared structure (CLS = 3), which induces stronger class overlap. We quantify generation difficulty using of the class-conditional ground-truth mixture, where larger values indicate greater difficulty. (a) Ground truth samples from the target class distribution. (b) No guidance: conditional sampling with a small diffusion model produces outliers and off-class leakage. (c) CFG () suppresses outliers but shows mode collapse, reducing branch coverage; the collapse is more severe as difficulty increases from (CLS = 2) to (CLS = 3). (d) VOG (ours) preserves diversity and branch coverage while achieving strong conditional alignment. Detailed quantitative results are provided in Table S11.
Guidance-induced mean shift. A first-order perturbation analysis of the VP probability flow ODE gives
where the expectation is over samples generated by the -guided sampler, is the unconditional data mean, the target class mean, the posterior-weighted dynamic mean, and the probability flow kernel of Eq. (26). The integrand is thus the class-uncertainty gap, transported to by the flow. The trajectory-dependent term also varies with .
Writing separates ordinary conditional prediction from further extrapolation. This motivates anchoring the control in § 3.2 at the conditional policy.
The analysis assumes Bayes-optimal branch mixing and locally affine class denoisers with shared contraction (Assumption E.1 and Assumption E.2).
Geometric interpretation. The transported class contrast links guidance to changes in distributional coverage. An exact subspace denoiser keeps the guided mean in the subspace. An imperfect denoiser can introduce cross-subspace coupling. Whether it yields a normal mean component also depends on the class contrast and cancellation along the trajectory (Appendix § 10.§ 2).
Variational Optimal Control (VOC)
We formulate sampling-time guidance as variational optimal control: maximize a terminal objective while penalizing deviation from a conditional reference process. Figure 1 relates the resulting state correction to DVOC’s parameter-space pullback.
Current-state conditional formulation. We parameterize denoising time as (: noise, : clean sample). At time , the state is already realized, so guidance can only modify the future continuation from to . Let be the future path measure of a fixed baseline policy , and let the terminal objective say which clean samples are preferred. We seek a controlled path measure solving
where and abbreviate the future path measures conditioned on .
Conditional reference. For the stochastic VOG formulation, is the pretrained conditional policy predicting . Zero control means ordinary conditional sampling. We use the terminal log-ratio , with control strength . The reference already carries the conditioning, so extrapolation enters through the terminal objective. For reward adaptation, we instantiate the endpoint objective with a learned evaluator and constrain local improvement within a trust region (§ 3.5).
Optimal solution via exponential tilting. Define the desirability and its log . By the Gibbs variational identity, the optimizer of Eq. (2) is the exponentially tilted path measure (proof in Appendix § 7):
Using an increasing clock along the denoising direction, the Doob -transform shifts the reference drift by , where is the diffusion covariance. This covariance is preserved by the transform.
Practical discretization. For a single denoising step of size (advancing from to ), we use the local Gaussian approximation with and . Applying the short-time Doob tilt and linearizing the log-desirability yields a Gaussian with the same covariance and a shifted mean,
Since the exact requires integrating over all future continuations, we approximate it by the denoised reward proxy and pull the gradient back through the denoising Jacobian , giving . For isotropic step covariance and , mapping this mean-space correction to the noise-prediction parameterization (, ) and absorbing all sampler-dependent constants into a positive schedule gives our main result:
Proposition 3.2 (Variational Optimal Guidance). Under the current-state conditional VOC of Eq. (2), the Doob -transform preserves the diffusion covariance and shifts the drift by . With isotropic step covariance, the short-time Gaussian approximation, and the denoised reward proxy, the first-order local approximation to the optimal noise-space correction is
where is the conditional reference prediction, , and is a sampler-dependent schedule. The derivation is in Appendix § 7.
Role of the denoising Jacobian. The key structural difference from CFG is the Jacobian : CFG adds the guidance signal to the noise prediction directly, whereas VOG first pulls it back through the denoiser via a vector-Jacobian product (VJP). In the linear subspace setting below the MMSE denoising Jacobian is a scaled orthogonal projector, so this VJP is exactly a projection onto the data subspace. For nonlinear denoisers it weights the correction by learned sensitivity. This provides a geometric mechanism for mitigating the normal drift () of § 3.1.
Geometric Consistency Analysis
We characterize the exact pullback relative to the data subspace of Assumption 3.1.
Theorem 3.3 (Subspace support of the exact correction). Let project onto the orthogonal complement of the data subspace, and let denote the correction that VOG adds to the baseline. Under Assumption 3.1 and the optimal linear MMSE denoiser , this correction lies in the data subspace, .
Proof. For the linear subspace denoiser, is symmetric with both column and row space contained in . Hence , so .
The exact linear denoiser therefore filters the reward gradient into . The implementation below approximates this pullback for neural denoisers.
Practical Implementation Strategies
We use three strategies to balance efficiency and stability in practice.
Strided gradient computation. We recompute the pullback every steps and reuse the cached correction in between. This inference stride is distinct from DVOC’s active-step count .
Factorized VJP surrogate. We approximate the conditional-noise VJP by , where and . The sensitivity factor is computed with one backward pass and can be cached separately from the contrast. Appendix § 8 derives the controller and gives the residual bound and differentiation-branch ablation.
Momentum-enhanced guidance. High-frequency fluctuations in the raw guidance signal can trigger over-suppression under Jacobian-based regularization. We smooth it with exponential momentum, , and replace by in the update rule. Here is the momentum coefficient, distinct from the correction schedule of Eq. (5). The uncached, zero-momentum core is given in Algorithm S1. The schedules and caching described here are practical extensions.
From State Control to Model Adaptation
The variational construction couples a preferred terminal change with the sensitivity of the controlled variable. VOG acts on the current state. DVOC acts on model parameters, amortizing terminal preferences across future trajectories. Their common local control problem yields the endpoint distillation objective below.
A shared local variational principle. Let map a control variable to an output representation, and let denote a preferred change in that representation. Write . Linearizing the output response and assigning a quadratic cost to the control gives
where sets the control scale. This is the pullback structure exposed by the local VOG derivation. For state control, is the denoised prediction and . For parameter control, the output is a terminal latent and . Changing the controlled variable changes the Jacobian through which the preferred output displacement acts.
Constructing the terminal control target. For a prompt , sample a trajectory with the current model and denote its terminal latent by . Let be the frozen decoder and define . We seek a terminal improvement within a local displacement budget:
Starting at , we approximate this problem with normalized projected ascent steps. The reward gradient is itself a pullback, . We denote the resulting detached target by and its displacement by . The relative radius makes the control budget proportional to the sampled latent’s scale. Updating the reward direction at each ascent step accounts for the local reward geometry along the target path. The projection and iteration are specified in Appendix § 9.
Distilling the control into parameters. We transfer to the generator by fitting through an endpoint connector . Its forward value is the sampled endpoint , and its parameter Jacobian is . For a latent with coordinates, the per-sample objective is
The target and normalization are held fixed during fitting. Since the residual at the sampled endpoint is , differentiation yields
Thus endpoint distillation realizes the parameter-space form of Eq. (6), with and scale . The terminal target supplies the control signal, while the parameter Jacobian determines how that signal is expressed by the generator. Averaging this loss over sampled prompts and trajectories amortizes the terminal controls into a shared model.
Algorithm 1 DVOC training update
Require: LoRA parameters , decoder , reward , sampling steps, active steps, target steps.
Sample prompts and generate -step trajectories.
Cache states, velocities, and terminal latents without gradients.
Initialize and radius .
for do
Compute .
Take a normalized ascent step of length and project onto .
end for
Detach the target .
for each optimization microbatch do
Sample active steps uniformly without replacement.
Re-evaluate active velocities at the cached states.
Form the endpoint connector in Eq. (23).
Accumulate scaled gradients of Eq. (8).
end for
Average gradients across GPUs and clip their norm.
Update with AdamW and update the EMA adapter.
Efficient parameter pullback. We sample an Euler trajectory without retaining its computation graph, then choose steps uniformly without replacement. At these steps, we re-evaluate velocities at the cached states with parameter gradients enabled. Their Euler-weighted derivatives define . The endpoint connector preserves the sampled forward value while passing gradients through these selected evaluations, as specified in Eq. (23). The complete endpoint supplies the target, and sets the backward computation. Our experiments use nine sampling steps, , two target-ascent steps, relative radius , and step multiplier . We optimize LoRA parameters and use the EMA adapter for evaluation.
Control geometry and amortization. In Eqs. (5) and (9), both controls weight terminal preferences by the sensitivity of the controlled variable. VOG realizes the state-space geometry characterized in Theorem 3.3. DVOC shares parameter updates across sampled conditions and stores the control in the generator. The adapted model then uses the ordinary sampler at inference. Figure 1 summarizes the two realizations, and Algorithm 1 gives the training procedure.
Experiments
We evaluate VOG on synthetic and class-conditional ImageNet generation, and DVOC on reward-based Z-Image-Turbo adaptation.
Evaluation Protocols
Sampling-time guidance. We use EDM2 [19] and DiT [25] on ImageNet [7] at , , and resolutions. Baselines include CFG, Cosine and Interval schedules, AutoGuidance, APG, and ReCFG [16, 18, 21, 30, 41]. We report FID [15] from 50,000 samples together with Precision and Recall [31]. Architecture-specific settings are provided in the supplement.
Reward-based post-training. DVOC, ReFL, DRTune, and DiffusionOPSD start from Z-Image-Turbo [35]. We also evaluate its untrained original weights. The trained methods share the training prompt source, the weighted reward , and a nominal budget of 576 trajectories per optimizer update. We evaluate their EMA adapters at update 20, with nine denoising steps per evaluation sample. Each trained experiment uses eight GPUs, LoRA rank 32 and scale 64, AdamW with learning rate , and weight decay (DVOC, DiffusionOPSD) or (ReFL, DRTune). The supplement reports method settings and observed per-update times (Table S7).
| Method | GPUs | Seconds / update | Minutes / update | GPU-min / update |
| ReFL | 8 | 272.8 ± 1.0 | 4.55 | 36.4 |
| DRTune | 8 | 336.8 ± 1.3 | 5.61 | 44.9 |
| DiffusionOPSD | 8 | 397.6 ± 0.8 | 6.63 | 53.0 |
| DVOC ()† | 8 | 325.8 ± 0.6 | 5.43 | 43.4 |
| DVOC ()† | 8 | 391.2 ± 0.4 | 6.52 | 52.2 |
| DVOC () | 8 | 458.0 ± 0.5 | 7.63 | 61.1 |
Table S7. Training time per optimizer update on one eight-GPU node. Means and standard deviations use 18 intervals from updates 2–20, excluding update 11 because it follows evaluation and checkpointing. Each update uses 576 training trajectories. Standard deviations describe timing variation within a single run. † denotes estimates from rollout-completion intervals; other rows use optimizer-completion intervals.
The test set is the complete 1,000-prompt Pick-a-Pic split [20]. All methods use the same prompts and initial noise, with outputs, nine deterministic Euler steps, and guidance scale 0. All seven metrics are computed on the same generated images. We use HPSv3++ [23] and CLIP [28] as in-domain evaluators because they enter the training reward. Q-Align quality and aesthetics [39], AES [32], ImageReward [42], and DeQA [44] are out-of-domain (OOD) evaluators because they are held out of optimization. Here OOD denotes evaluators held out of the training objective. All post-training results use one training seed.
Sampling-Time Guidance Results
Synthetic geometry. On the 2D branching-tree benchmark, increasing class overlap makes coverage more difficult. The two-class and three-class settings have covariance traces 0.228 and 0.403. CFG removes outliers but reduces Recall to 0.876 and 0.779, respectively. VOG attains Recall 0.957 and 0.933 while achieving the highest Precision among the guided samplers. The visualization and numerical results are in Figure 2 and the supplement.
ImageNet. Table 1 evaluates VOG on top of three guidance baselines in four model–resolution settings. FID improves in all twelve pairings, including from 1.99 to 1.42 for EDM2-L at . For DiT-XL/2 with Cosine guidance, Recall rises from 0.520 to 0.612 while Precision decreases, producing a better FID operating point.
| Guidance | EDM2-S (512 × 512) | DiT-XL/2 (256 × 256) | EDM2-L (64 × 64) | DiT-XL/2 (512 × 512) | ||||||||
| FID | Precision | Recall | FID | Precision | Recall | FID | Precision | Recall | FID | Precision | Recall | |
| CFG [16] | 2.29 | 0.751 | 0.582 | 2.27 | 0.829 | 0.584 | 1.99 | 0.710 | 0.601 | 3.04 | 0.838 | 0.541 |
| + VOG (Ours) | 2.03 | 0.755 | 0.591 | 2.04 | 0.756 | 0.607 | 1.42 | 0.714 | <u>0.597</u> | 2.89 | 0.745 | 0.560 |
| Interval [21] | <u>1.87</u> | 0.760 | 0.598 | <u>1.97</u> | 0.801 | 0.632 | 1.64 | 0.730 | 0.597 | <u>2.41</u> | <u>0.792</u> | 0.611 |
| + VOG (Ours) | 1.67 | 0.742 | 0.614 | 1.94 | 0.765 | 0.611 | 1.52 | 0.720 | 0.593 | 2.39 | 0.760 | 0.567 |
| Cosine [12] | 2.15 | 0.770 | <u>0.619</u> | 2.30 | 0.861 | 0.520 | 1.67 | 0.726 | 0.587 | 3.48 | 0.713 | <u>0.586</u> |
| + VOG (Ours) | 2.06 | 0.758 | 0.589 | 2.03 | 0.757 | 0.612 | 1.65 | <u>0.727</u> | <u>0.588</u> | 2.83 | 0.747 | <u>0.563</u> |
| Scheduler [38] | 2.03 | <u>0.762</u> | 0.591 | 2.14 | <u>0.840</u> | 0.614 | - | - | - | - | - | - |
| APG [30] | 2.13 | 0.756 | 0.640 | 2.11 | 0.815 | <u>0.628</u> | - | - | - | - | - | - |
| ReCFG [41] | 2.23 | 0.761 | 0.608 | 2.13 | 0.799 | 0.601 | - | - | - | - | - | - |
Table 1. VOG as a plug-in correction on top of CFG-style guidance baselines, on class-conditional ImageNet. Shaded rows apply VOG to the baseline immediately above them. We report FID (↓), Precision (↑), and Recall (↑); within each column the best result is bolded and the second best <u>underlined</u>. A dash indicates a configuration the baseline does not support.
Guidance scale and sampling budget. VOG remains effective over a wider range of guidance strengths than CFG in Figure S2. On DiT-XL/2, its FID is below 2.60 at guidance scale 1.7, where CFG exceeds 3.40. The benefit also persists with fewer sampling steps: FID decreases from 7.58 to 4.44 at 25 steps and from 3.69 to 2.99 at 50 steps. The VJP stride and momentum sweep in Figure S2 concerns VOG inference, independently of the active-step budget used by DVOC.
Figure S2. FID as a function of the guidance scale for DiT-XL/2 (256 × 256) and EDM2-S (512 × 512).
Reward Alignment and Generalization
In-domain alignment and held-out scores. Table 2 uses the untrained original model as the reference, with DVOC at . DVOC raises HPSv3++ from 6.72 to 13.79, the highest trained mean. CLIP changes from 0.2757 to 0.2765. DiffusionOPSD achieves the highest CLIP score at 0.2782.
| Method | In-Domain HPSv3++ Score↑ | Δ | In-Domain CLIP Score↑ | Δ | Out-of-Domain Q-Align Q. Score↑ | Δ | Out-of-Domain Q-Align A. Score↑ | Δ | Out-of-Domain AES Score↑ | Δ | Out-of-Domain ImageReward Score↑ | Δ | Out-of-Domain DeQA Score↑ | Δ |
| Z-Image-Turbo, , nine-step Euler sampler | ||||||||||||||
| Z-Image-Turbo (untrained) | 6.72 | — | 0.2757 | — | 4.825 | — | 3.533 | — | 5.412 | — | 0.959 | — | 4.439 | — |
| ReFL [42] | 12.22 | +5.50 | 0.2744 | −0.0013 | 4.844 | +0.020 | 4.223 | +0.691 | 5.955 | +0.543 | 1.159 | +0.200 | 4.133 | −0.306 |
| DRTune [40] | 12.70 | +5.97 | 0.2688 | −0.0070 | 4.932 | +0.107 | 4.489 | +0.956 | 6.095 | +0.683 | 1.225 | +0.266 | 4.262 | −0.177 |
| DiffusionOPSD [47] | 13.56 | +6.83 | 0.2782 | +0.0025 | 4.886 | +0.061 | 4.452 | +0.920 | 6.101 | +0.689 | 1.196 | +0.237 | 4.014 | −0.425 |
| DVOC () | 13.79 | +7.06 | 0.2765 | +0.0008 | 4.943 | +0.118 | 4.618 | +1.085 | 6.288 | +0.876 | 1.279 | +0.320 | 4.237 | −0.202 |
Table 2. Reward alignment after 20 optimization updates on Z-Image-Turbo. The untrained base and four trained methods use the same 1,000 prompts and initial noise. HPSv3++/CLIP are in-domain; the others are OOD. Δ: score minus untrained base. Higher is better. Best/second-best: bold/underlined. Blue: DVOC.
DVOC leads trained methods on Q-Align quality (4.943), Q-Align aesthetics (4.618), AES (6.288), and ImageReward (1.279), improving over the base on all four. DeQA is 4.237, compared with 4.439 for the base. Table S8 reports paired uncertainty for all seven main-result differences.
| Metric | Mean difference | 95% interval |
| HPSv3++ | +7.06357 | [+6.90034, +7.22679] |
| CLIP | +0.00077 | [-0.00142, +0.00297] |
| Q-Align Q. | +0.11801 | [+0.10556, +0.13046] |
| Q-Align A. | +1.08532 | [+1.05159, +1.11906] |
| AES | +0.87600 | [+0.84961, +0.90238] |
| ImageReward | +0.31966 | [+0.28264, +0.35668] |
| DeQA | -0.20233 | [-0.21682, -0.18784] |
Table S8. Paired differences: default DVOC () minus untrained Z-Image-Turbo. Normal-approximation 95% intervals use 1,000 paired prompts and are reported per metric.
Qualitative comparisons with the base model and the three trained baselines are provided in Figure S1 in the appendix.

Figure S1. Qualitative comparison of reward-based adaptation. Columns show Z-Image-Turbo, ReFL, DRTune, DiffusionOPSD, and DVOC. Rows show (a) a museum interior, (b) a linen jacket, (c) a bicycle, and (d) a reading room. The examples illustrate differences in material texture, lighting, and local detail across methods.
Optimization dynamics. Figure 3(a,b) tracks the two in-domain evaluators through 100 updates. From update 20 to 100, DVOC’s online CLIP increases from 0.2768 to 0.3111 and HPSv3++ from 12.71 to 15.59. Panels (c,d) show logged training reward across methods and across , respectively. These online evaluations use a different protocol from the paired main table. Extended histories are in Appendix § 13.
Figure 3. Evaluation and training dynamics. (a,b) Online EMA evaluation every ten updates, using HPSv3++ condition 0.0 and trainer-specific noise sampling. (c) Training reward, 0.03846 : ReFL/DRTune score late-state proxies; DVOC/OPSD score sampled endpoints. (d) DVOC training reward for . Training panels show raw means (faint) and trailing five-update averages; evaluation is unsmoothed. The vertical line marks update 20. DVOC uses in (a–c); (d) is the ablation.
DVOC Ablations
Active-step budget. Table 4 varies with shared forward sampling and target construction. leads on HPSv3++, CLIP, AES, and ImageReward. leads on both Q-Align scores, and on DeQA, showing that the preferred budget depends on the evaluator.
| DVOC variant | HPSv3++↑ | CLIP↑ | Q-Align Q.↑ | Q-Align A.↑ | AES↑ | ImageReward↑ | DeQA↑ |
| 13.36 | 0.2729 | 4.948 | 4.628 | 6.272 | 1.224 | <u>4.249</u> | |
| <u>13.69</u> | <u>0.2751</u> | <u>4.947</u> | 4.610 | <u>6.274</u> | <u>1.257</u> | 4.263 | |
| (default) | 13.79 | 0.2765 | 4.943 | <u>4.618</u> | 6.288 | 1.279 | 4.237 |
Table 4. Active-step ablation. Only changes; and the endpoint radius are fixed. Means use the same 1,000 paired prompts at update 20. Bold/underlining mark best/second-best within this table.
Target construction. Three paired runs fix and share the initial adapter and training streams (Appendix § 11.§ 4). Relative to one ascent step, endpoint raises Q-Align quality by 0.0202, AES by 0.0692, and DeQA by 0.1203, with a CLIP decrease of 0.00210. At the same nominal ascent length, two steps refresh the intermediate reward gradient. One step takes 414.1 seconds per update versus 461.7 for two steps, a 10.3% reduction.
The proxy ablation optimizes the current policy’s late-state clean prediction and transports its displacement to the sampled endpoint. Endpoint improves Q-Align quality, Q-Align aesthetics, and DeQA by 0.0125, 0.0215, and 0.0100, respectively. The full seven-metric comparison is in Table 3.
| Variant | HPSv3++ | CLIP | Q-Quality | Q-Aesth. | AES | ImageReward | DeQA |
| Endpoint, | 13.727 | 0.2760 | 4.947 | 4.640 | 6.281 | 1.2678 | 4.252 |
| Endpoint, | 13.653 | 0.2781 | 4.927 | 4.629 | 6.211 | 1.2673 | 4.131 |
| Proxy, | 13.705 | 0.2763 | 4.935 | 4.618 | 6.277 | 1.2684 | 4.242 |
Table 3. Target construction at fixed . Three paired runs use 20 updates and 1,000 matched test images. All scores are higher-is-better. Bold marks the highest mean within this ablation.
Computational Cost
The nine-step rollout and two target-gradient evaluations are shared across the active-step variants. Increasing expands the parameter pullback while retaining the same sampled trajectories and target construction. On one eight-GPU node, the observed mean times are 325.8, 391.2, and 458.0 seconds per update for , respectively (Table S7). Relative to , reduces update time by 28.9% and by 14.6%. The main configuration uses , which leads the active-step comparison on HPSv3++, CLIP, AES, and ImageReward.
At fixed , one and two target steps take 414.1 and 461.7 seconds per update. The target ablation in Table 3 reports their quality–cost trade-off. At inference, all DVOC variants use the same nine-step sampler.
Conclusion
Variational control connects sampling-time guidance and reward adaptation through a shared Jacobian pullback. VOG realizes control on individual trajectories, and DVOC amortizes endpoint improvements into the generator. VOG improves FID across the evaluated ImageNet settings. After 20 updates on Z-Image-Turbo, DVOC improves HPSv3++ and four held-out scores over the base and leads the trained comparators on these metrics. Our analyses characterize the common control geometry through subspace structure and the variational derivation of parameter control.
Preliminaries and Notation
We collect the diffusion and guidance notation used in the derivations.
Diffusion and Guidance
The VP perturbation is , with and cumulative signal power . A noise predictor trained by denoising regression [17] gives the model score [33, 34].
Classifier guidance adds a scaled gradient of a noisy-image classifier’s log probability to the model score [9]. CFG uses the contrast between conditional and unconditional predictions [16].
Classifier-Free Guidance (CFG). To avoid the complexity of external classifiers, Ho and Salimans [16] proposed effectively learning an implicit classifier by jointly training a single model for both conditional and unconditional generation. During inference, the guided noise prediction is computed as a linear extrapolation:
Here, serves as the guidance scale. Eq. (10) can be interpreted as implicitly performing gradient ascent on the log-likelihood ratio , treating the difference term as a proxy for the gradient of the condition. Large can increase saturation and reduce sample coverage [5, 30].
Summary of Notation
Table S1 collects the notation. We distinguish the CFG scale , VOG correction schedule , momentum coefficient , and VP noise rate .
| Symbol | Meaning |
| , | noisy state at time ; denoised prediction of the clean sample |
| , , | conditional and unconditional predictions; VOG-corrected prediction |
| , | cumulative signal power and VP noise standard deviation |
| , | conditioning signal; terminal reward on the clean sample |
| , | CFG guidance scale; its conditional-centered form, the control strength |
| reward temperature in the VOC objective | |
| schedule of the VOG noise-space correction | |
| momentum coefficient for smoothing the guidance signal | |
| inference-time interval between VJP recomputations | |
| , , | DVOC active-step count, ascent iterations, and relative endpoint radius |
| denoising Jacobian; the correction uses the VJP | |
| , | desirability function and its logarithm |
| , | drift and diffusion covariance of the pretrained sampler |
| , , | data subspace and the projectors onto it and its complement |
| , , | LoRA parameters, endpoint target displacement, and detached loss normalization |
| , | selected parameter Jacobian and its endpoint connector |
Table S1. Notation used in the main text.
Derivation of Variational Optimal Guidance
We derive the tilted path measure, its drift correction, and the local Jacobian form used by VOG. Throughout, is the pretrained conditional continuation from the current state to the clean endpoint.
Optimal path measure
Let and . Assume and finite reward and KL terms. For the tilted measure in Eq. (3), substitution of its density gives
The nonnegativity of KL establishes that uniquely maximizes Eq. (2), with value .
Drift correction and control cost
Use the increasing denoising clock , with . The reference process has drift , diffusion coefficient , and covariance . Its generator is . The Markov property of the tilted measure gives
The desirability satisfies with . Applying the product rule to the transformed generator yields
Thus the optimal drift is , with the reference diffusion covariance unchanged.
For a drift perturbation and invertible , Girsanov’s identity expresses the path KL as
This identity assumes an integrable control energy and a true Girsanov martingale. It identifies the covariance-weighted quadratic cost underlying the local control objective.
Local transition and Jacobian pullback
For a step of size , approximate the reference transition by , where and . Linearizing in and writing gives
Completing the square gives , proving the mean shift in Eq. (4).
We evaluate the terminal reward on the conditional clean prediction . The proxy gives the chain-rule correction
The reverse mean is affine in noise prediction, with , so . For isotropic covariance , setting yields Eq. (5). This completes the derivation of Proposition 3.2.
The CFG-Specific VOG Controller
We instantiate the terminal objective with the conditional log-ratio defined in § 3.2. Write , , , and . Standard CFG is .
Conditional clean-prediction pullback
At noise level , the model score contrast is . We use this contrast as a proxy for the clean-space log-ratio gradient. Its negative sign cancels the mean-to-noise sign in the preceding derivation. Let and . Differentiating the conditional clean prediction gives
The positive scalar factor is included in the guidance schedule.
Factorized VJP
We compute the sensitivity with one backward pass and combine it element-wise with the guidance contrast. The difference from the full VJP is
This identity follows by adding and subtracting in each coordinate sum. For the influence set , the triangle inequality bounds the residual by
For an empty influence set, the maximum is defined as zero. The residual also vanishes when the contrast is constant on the influence set, including the case of a diagonal Jacobian. Dropping it gives the practical controller
The sensitivity factor can be cached independently of the contrast. The element-wise update also preserves zero coordinates of the CFG direction. Stride and momentum are specified in § 3.4.
Choice of differentiation branch
We differentiate the conditional prediction that defines the reference process. To examine this choice, consider differentiating instead. With and the resulting clean-prediction Jacobian, its pullback is
The additional term is quadratic in guidance strength. Using the conditional branch keeps the sensitivity map independent of at a fixed state. Table S2 evaluates the two choices.
| Controller | FID () | Relative runtime |
| Pullback through | 5.48 | – |
| Pullback through (VOG) | 2.04 | 1.55× |
Table S2. Differentiation-branch ablation on DiT-XL/2 (256 × 256), with VJP stride . Runtime is relative to CFG.
Sampling procedure.
Algorithm S1 VOG factorized core controller
Require: Conditional/unconditional predictor, condition , guidance strength , and baseline numerical sampler.
Sample ; set .
for each reverse sampling step do
Compute , , and .
Compute by one VJP.
Set for the VP parameterization.
Form as in Eq. (20).
Advance the numerical sampler using .
end for
return Final sample.
Architecture-specific schedules, momentum, and VJP stride extend this core as described in § 3.4.
Parameter Control: Target and Implementation
The local objective and its distillation gradient are given in Eqs. (6) and (9). Here we specify how DVOC constructs the target and computes the selected parameter derivatives.
Endpoint optimization
Let , , and . We use normalized projected ascent:
The Euclidean projection clips the displacement from to length . We use , , , , and zero ascent momentum. The frozen FP32 decoder and reward evaluator supply the latent gradients. The final target is .
Cached endpoint connector
Sample the Euler trajectory without retaining its computation graph and cache states , velocities , and increments . The endpoint is . Choose a size- subset uniformly without replacement and re-evaluate its velocities with gradients enabled. Define
All cached quantities are detached. The connector equals in the forward pass and passes parameter gradients through the selected velocity evaluations. We insert it into Eq. (8) and optimize the LoRA parameters with AdamW. Checkpoint evaluation uses the EMA adapter.
Mean Shift of CFG under the VP Probability Flow ODE
This section derives the linearized mean shift in Eq. (1). Let denote the VP noise rate, the cumulative signal power, and the mean of class . The posterior-weighted class mean is .
Class contrast and guided dynamics
Assumption E.1 (Bayes-optimal branch mixing). The denoisers estimate conditional means, so the unconditional clean prediction satisfies .
Assumption E.2 (Locally affine class denoisers). The class denoisers have the local form , with a symmetric contraction shared across classes.
Subtracting the unconditional mixture from the target-class prediction gives . Since the clean-prediction map is affine in noise prediction, CFG induces
Write and . Substituting the denoiser-score relation into the VP probability flow ODE gives
The ODE is integrated from noise time to clean time .
Transport of the guidance perturbation
Let be the unguided trajectory with the same initial noise and the state transition matrix for along this trajectory. Define
Linearizing around and applying variation of constants gives
Assuming the transported contrast is integrable in expectation, taking expectations and using gives Eq. (1). The expectation includes the path-dependent kernel. The guided class-posterior gap also depends on .
Subspace interpretation. If every class mean lies in , the contrast lies in on each trajectory. Its normal response is therefore determined by . When preserves , both the local filter and the unguided linearized flow preserve it, so this normal response vanishes. A learned denoiser can introduce cross-subspace coupling, whose effect depends on the transported contrast. The projection property of VOG’s exact linear-MMSE correction is proved directly in Theorem 3.3.
Detailed Experimental Settings
General Configuration and Hyperparameters
The VOG class-conditional sampling experiments use JAX. The DVOC and baseline post-training comparison uses PyTorch. All FID scores use 50k generated samples per configuration. Sampling parameters follow standard practice for each architecture: for EDM2-S we use an evaluation batch size of 8 per device. For DiT models we use 250 sampling steps with a per-process batch size of 8 and the MSE-tuned VAE from the official Stable Diffusion repository.
Table S4: Hyperparameters and settings for ImageNet-256 and ImageNet-512. The experiments are conducted using the DiT-XL/2 architecture. The column denotes the guidance scale. The column Guidance interval specifies the range of noise levels where classifier-free guidance is applied. For our proposed method (+Ours), represents the frequency of regularization steps (computing gradients every steps), and indicates the momentum coefficient. The parameter refers to the exponent used in Guidance cosine.
We tune the guidance weight separately for each architecture and task. The specific hyperparameter configurations and the chosen ranges are detailed in Table S3 and Table S4. All class-conditional generation results were obtained using eight NVIDIA RTX 4090 GPUs. The DVOC comparison uses eight NVIDIA L20X GPUs per run and evaluates the checkpoint after 20 updates.
| Resolution | Method | Guidance interval | ||||||
| 64 × 64 | EDM2-L | |||||||
| 64 × 64 | Classifier-free guidance | 1.15 | - | - | - | Full | 0.040 | 0.040 |
| 64 × 64 | + VOG (Ours) | 1.05 | 2 | 0.15 | - | Full | 0.040 | 0.040 |
| 64 × 64 | Guidance interval | 1.15 | - | - | - | (0.3,1.5] | 0.040 | 0.040 |
| 64 × 64 | + VOG (Ours) | 1.10 | 2 | 0.15 | - | (0.3,1.5] | 0.040 | 0.040 |
| 64 × 64 | Guidance cosine | 1.05 | - | - | 0.3 | Full | 0.040 | 0.040 |
| 64 × 64 | + VOG (Ours) | 1.05 | 2 | 0.15 | 0.3 | Full | 0.040 | 0.040 |
| 64 × 64 | EDM2-S | |||||||
| 64 × 64 | Autoguidance | 1.70 | - | - | - | Full | 0.045 | 0.110 |
| 64 × 64 | + VOG (Ours) | 1.70 | 2 | 0.15 | - | Full | 0.045 | 0.110 |
| 512 × 512 | EDM2-S | |||||||
| 512 × 512 | Classifier-free guidance | 1.40 | - | - | - | Full | 0.025 | 0.025 |
| 512 × 512 | + VOG (Ours) | 1.60 | 2 | 0.15 | - | Full | 0.025 | 0.025 |
| 512 × 512 | Guidance interval | 2.10 | - | - | - | (0.28,2.90] | 0.025 | 0.025 |
| 512 × 512 | + VOG (Ours) | 2.30 | 2 | 0.15 | - | (0.28,2.90] | 0.025 | 0.025 |
| 512 × 512 | Guidance cosine | 1.60 | - | - | 0.3 | Full | 0.025 | 0.025 |
| 512 × 512 | + VOG (Ours) | 1.65 | 2 | 0.15 | 0.3 | Full | 0.025 | 0.025 |
Table S3. Hyperparameters and settings for ImageNet-64 and ImageNet-512. The experiments are conducted using EDM2-L and EDM2-S architectures. The column denotes the guidance scale. The column Guidance interval specifies the range of noise levels where classifier-free guidance is applied. For our proposed method (+Ours), represents the frequency of regularization steps (computing gradients every steps), and indicates the momentum coefficient. The parameter refers to the exponent used in Guidance cosine. The columns and indicate the EMA decay lengths for the main generative model and the guiding model, respectively.
| Resolution | Method | Guidance interval | ||||
| 256 × 256 | Classifier-free guidance | 1.50 | - | - | - | Full |
| 256 × 256 | + VOG (Ours) | 1.50 | 4 | 0.15 | - | Full |
| 256 × 256 | Guidance interval | 1.50 | - | - | - | (0.34, 1.02] |
| 256 × 256 | + VOG (Ours) | 1.90 | 2 | 0.15 | - | (0.34, 1.02] |
| 256 × 256 | Guidance cosine | 1.50 | - | - | 0.3 | Full |
| 256 × 256 | + VOG (Ours) | 1.50 | 2 | 0.15 | 0.3 | Full |
| 512 × 512 | Classifier-free guidance | 1.50 | - | - | - | Full |
| 512 × 512 | + VOG (Ours) | 1.50 | 2 | 0.15 | - | Full |
| 512 × 512 | Guidance interval | 2.50 | - | - | - | (0.34, 1.02] |
| 512 × 512 | + VOG (Ours) | 2.50 | 2 | 0.15 | - | (0.34, 1.02] |
| 512 × 512 | Guidance cosine | 1.50 | - | - | 0.3 | Full |
| 512 × 512 | + VOG (Ours) | 1.50 | 2 | 0.20 | 0.3 | Full |
Table S4.
Structure of the 2D Synthetic Distributions
The branching-tree datasets are Gaussian mixtures. Table S5 summarizes their component weights and spatial spread. The entropy-based effective count is the exponential of the weight entropy, and the HHI-based count is the inverse sum of squared weights. Top 5% Mass is the total weight of the largest 5% of components. The covariance trace measures total spatial variance, and MST is the total Euclidean length of the minimum spanning tree connecting component means.
| Dataset | Top 5% Mass | tr(Cov) | MST | |||
| CLS-2 (Class A) | 1016 | 249.81 | 90.29 | 0.5429 | 0.2275 | 15.35 |
| CLS-3 (Class A) | 1036 | 214.05 | 102.91 | 0.6361 | 0.4029 | 18.57 |
Table S5. Quantitative difficulty metrics for the 2D synthetic datasets (Class A).
As indicated in Table S5, the CLS-3 dataset is the harder of the two. Geometrically, CLS-3 exhibits a much larger spatial dispersion, evidenced by the higher total variance (tr(Cov) 0.40 vs. 0.23) and a more extensive Minimum Spanning Tree (18.57 vs. 15.35). Statistically, the distribution of CLS-3 is more skewed. Despite having a similar number of components, its probability mass is more concentrated in the top few modes (Top 5% Mass 63.6%), resulting in a lower entropy-based effective number of components (). This combination of high spatial variance and weight imbalance requires the generative model to capture a broader manifold structure while handling sharper density peaks.
DVOC and Baseline Reproduction Details
The main DVOC configuration uses , with as active-step ablations. We compare against the untrained Z-Image-Turbo reference and three trained methods: ReFL, DRTune, and DiffusionOPSD. All methods use shared prompts and initial noise for paired evaluation.
Hardware and update accounting. Each trained main method and both DVOC ablations use one eight-GPU node, with reward models colocated on the same devices. The untrained reference uses the same hardware for generation and scoring.
One training step denotes one optimizer update, including its generated trajectories, reward or target computation, accumulated backward passes, and parameter update. It is distinct from an Euler denoising step. The effective trajectory batch is , comprising 48 prompt groups with 12 trajectories each. Thus update 20 corresponds to 11,520 training trajectories. ReFL and DRTune stop their training rollouts at the selected late states. Their separate 512-prompt reward calibration is initialization overhead, outside this per-update trajectory count. The configured runs continue to 100 updates; the main table evaluates their update-20 EMA checkpoints.
| Method | Backward path | Reward target or loss | Weight decay |
| ReFL | One selected late-state evaluation | Negative standardized reward; loss scale ; no hinge | |
| DRTune | Three connected evaluations in a truncated rollout | Same reward loss and scale as ReFL; early-stop ratio 0.4 | |
| DiffusionOPSD | One queried state | Positive and negative PGD targets; old-policy mixing; , , | |
| DVOC () | Three cached states; detached state dependencies | Actual-endpoint PGD target; , , ; normalized target MSE |
Table S6. Method-specific training settings. A differentiated evaluation carries a parameter gradient.
DVOC settings. The active indices are sampled uniformly without replacement from all nine Euler steps; no correction rescales their sum. PGD uses two normalized ascent steps, radius , step multiplier 1, zero momentum, and target-construction microbatch size one. The actual endpoint, optimized target, and cached rollout states are detached. The squared fitting residual is divided by its detached mean absolute displacement, floored at . Rollout generation uses three trajectories per GPU call and 24 calls per update; fitting uses 72 one-trajectory microbatches. The ablations change only the active-step count.
ReFL and DRTune settings. Both use a fixed 512-prompt training-set calibration to standardize the mixed reward, followed by a negative-reward loss with scale and no hinge, auxiliary pretraining loss, or reference KL/MSE. ReFL samples one evaluation state in the final 25% of the nine-step schedule. DRTune uses three connected trainable steps with an early-stop ratio of 0.4. The rollout prefix is computed without gradients and the unused suffix is omitted. The two methods share random draws for prompts, noise, state selection, and calibration.
DiffusionOPSD settings. DiffusionOPSD generates 24 batches of three trajectories per GPU and uses prompt-group advantages clipped at 5, with both advantage signs enabled and policy-mixing coefficient 1.
OPSD forms ascent and descent targets at the old policy’s one-step clean prediction from a queried rollout state. Its requested query noise level is 0.273, mapped to the available schedule. Each target uses two normalized steps, relative radius 0.1, multiplier 1, zero momentum, and microbatch one; both targets are accepted without reward certification. It fits one queried state and applies no auxiliary reference-prediction penalty.
Training time measurement. For the four main methods, we measure elapsed time between consecutive optimizer updates. Table S7 reports means and standard deviations over 18 intervals from updates 2–20, excluding update 11 because it follows evaluation and checkpointing. The intervals include sampling, reward or target construction, fitting, and routine communication overhead. Initialization, periodic evaluation and checkpointing, and the separate offline evaluation are excluded.
For , the times are estimated from consecutive rollout-completion records and marked †. Each interval spans one fitting phase and the following rollout. Applying the same estimator to reproduces its measured mean of 458.0 s over the same 18 intervals, with a maximum per-interval difference of 1.4 s.
The same exclusion rule over the 100-update runs gives mean times of 272.9 s (ReFL), 337.2 s (DRTune), 396.8 s (DiffusionOPSD), and 456.5 s (DVOC), using 90 intervals per method.
Scorer definitions. All methods use the same evaluator weights and preprocessing. HPSv3++ uses iteration condition 0.3. Q-Align quality and aesthetics are evaluated with OneAlign, and DeQA uses DeQA-Score-Mix3. These two models use half-precision inference with batch size one. AES and Q-Align aesthetics are distinct evaluators and appear in separate columns.
Evaluation setup. Evaluation uses the EMA adapter after 20 optimizer updates and the original Z-Image-Turbo weights as the untrained reference. Each method generates images for the complete 1,000-prompt test set, and all seven evaluators score the same images. Online evaluations use a different noise sampling scheme and HPSv3++ condition 0.0; their results are reported separately in Appendix § 13.§ 1.
Scope of the comparison. The trained rows compare fixed update-20 checkpoints from runs configured for a longer training schedule. They share a nominal generated-sample and optimizer-update budget but differ in gradient paths, reward calls, and targets, giving different per-update computational costs. The main table’s gray differences are relative to the paired untrained original model. The in-domain/OOD split specifies evaluator use during optimization.
Paired uncertainty. For paired metric differences , Table S8 reports . These per-metric intervals summarize variation across the fixed 1,000 prompt/image pairs.
Endpoint-Target Ablations
We compare three target constructions at : the actual endpoint with two ascent steps (), the actual endpoint with one step (), and a late-state clean-prediction proxy with two steps. These paired runs share the initial LoRA adapter and random streams for prompts, initial noise, active states, and reward evaluations. Their common initialization is separate from the main experiment.
Target construction. The endpoint target starts from the terminal latent . The proxy starts from at the schedule point nearest the requested noise level 0.273, giving and . Its optimized displacement is transferred to the endpoint as . All variants use radius , step multiplier , zero momentum, and ascent-step length .
Evaluation and results. Each run uses 20 updates, 576 trajectories per update, and the shared optimizer configuration. The update-20 EMA adapters generate the same 1,000 test prompts with matched seeds and initial latents. All seven evaluators score these saved images.
Two ascent steps improve HPSv3++, Q-Align quality, AES, and DeQA relative to one step. The gains in Q-Align quality, AES, and DeQA are 0.0202, 0.0692, and 0.1203, respectively. CLIP is higher for . Relative to the proxy, the two-step endpoint target improves Q-Align quality, Q-Align aesthetics, and DeQA by 0.0125, 0.0215, and 0.0100. The complete score profile is shown in Table 3.
Computation. On the eight-GPU node, mean update times are 461.7 seconds for endpoint , 414.1 seconds for endpoint , and 463.0 seconds for proxy . These measurements use updates 2–20 and exclude evaluation and checkpoint I/O. One target step reduces update time by 10.3%.
Additional Experimental Results
We provide supplementary quantitative and qualitative evaluations to further validate the behavior of the evaluated guidance and post-training settings.
Qualitative comparison of reward-based adaptation. Figure S1 compares DVOC with the base model and the three trained baselines across four scenes. These examples complement the evaluator scores in Table 2 by showing how adaptation affects the visual appearance of generated images.
Computational efficiency. Table S9 details the computational overhead of VOG. Due to the inclusion of vector-Jacobian products, VOG increases runtime by to and peak GPU memory by to across varying resolutions. Despite this overhead, VOG remains practically feasible, with the per-device sampling benchmark fitting on one NVIDIA RTX 4090 GPU. DVOC training uses the separate allocation reported above.
| Model | Res/BS | CFG / VOG Runtime (sec) |
| EDM2-S | 64/8 | 3.66 / 5.33 (1.46x) |
| DiT-XL/2 | 256/8 | 10.97 / 17.04 (1.55x) |
| EDM2-S | 512/8 | 4.66 / 9.22 (1.98x) |
| DiT-XL/2 | 512/8 | 39.74 / 65.03 (1.64x) |
| Model | Res/BS | CFG / VOG Memory (GB) |
| EDM2-S | 64/1 | 1.18 / 1.79 (1.53x) |
| DiT-XL/2 | 256/1 | 4.14 / 4.99 (1.21x) |
| EDM2-S | 512/1 | 1.19 / 1.81 (1.53x) |
| DiT-XL/2 | 512/1 | 4.18 / 7.18 (1.72x) |
Table S9. Runtime and peak memory of VOG relative to CFG, reported as CFG / VOG with the ratio in parentheses. Res/BS denotes resolution and batch size; all measurements are taken on a single NVIDIA RTX 4090.
Performance in few-step regimes (Table S10). Table S10 compares 25-step sampling performance on DiT and SiT architectures. For SiT, although the model is trained to predict the interpolant velocity field , we follow the diffusion-SDE sampling setting by converting the learned velocity into the corresponding score:
| Model | Method | FID | Recall | Precision | |||
| DiT-XL/2 | CFG | 7.58 | 0.429 | 0.784 | 1.50 | - | - |
| VOG | 4.44 | 0.545 | 0.699 | 1.50 | 0.15 | 2 | |
| SiT-XL/2 | CFG | 6.20 | 0.555 | 0.685 | 1.50 | - | - |
| VOG | 3.89 | 0.516 | 0.735 | 1.10 | 0.15 | 2 |
Table S10. Quantitative comparison of 25-step sampling on DiT-XL/2 and SiT-XL/2 (256 × 256, ImageNet). We report FID (↓), Recall (↑), and Precision (↑), together with the guidance scale , the momentum coefficient , and the VJP recomputation interval .
where and denote the SiT interpolant coefficients. We then apply CFG and VOG on this converted score and use the same diffusion SDE sampler for evaluation. VOG improves FID on both architectures, from 7.58 to 4.44 on DiT-XL/2 and from 6.20 to 3.89 on SiT-XL/2. The two architectures trade Precision against Recall in opposite directions: on DiT-XL/2 the gain comes from coverage (Recall , Precision , whereas on SiT-XL/2 it comes from sample quality (Precision , Recall ). These configurations also use different tuned guidance scales ( for DiT-XL/2, for SiT-XL/2). In both cases VOG moves the operating point toward a better FID with architecture-dependent Precision–Recall trade-offs. At 50 steps, DiT-XL/2 FID decreases from 3.69 to 2.99. The relative reduction therefore grows from 19% at 50 steps to 41% at 25 steps.
Quantitative analysis on 2D synthetic benchmarks. Table S11 provides the numerical evaluation corresponding to the 2D branching-tree visualizations in Figure 2. While standard CFG effectively filters outliers (high Precision), it loses coverage as the distribution complexity increases, evidenced by the drop in Recall to 0.7785 on the three-class dataset. VOG achieves higher F1-scores and lower Wasserstein distances in both settings, indicating that it aligns samples with the conditional target while better preserving the support of the data manifold.
| Method | Two-class (CLS=2) Prec. | Rec. | F1 | Wass. | Three-class (CLS=3) Prec. | Rec. | F1 | Wass. |
| Ground Truth | 0.9916 | 1.0000 | 0.9958 | 0.0021 | 0.9862 | 1.0000 | 0.9931 | 0.0026 |
| No Guidance | 0.8948 | 0.9989 | 0.9440 | 0.0031 | 0.9194 | 0.9970 | 0.9567 | 0.0035 |
| Standard CFG | 0.9110 | 0.8755 | 0.8929 | 0.0064 | 0.9392 | 0.7785 | 0.8514 | 0.0071 |
| VOG (ours) | 0.9230 | 0.9568 | 0.9396 | 0.0043 | 0.9536 | 0.9328 | 0.9431 | 0.0042 |
Table S11. Quantitative results on 2D branching-tree generation. We report Precision, Recall, F1-score (higher is better), and Wasserstein distance (lower is better) for both Two-class and Three-class settings. Best results (excluding Ground Truth) are highlighted in bold.
Extended sensitivity analysis. Figure S3 and Figure S4 supplement the main text with guidance-scale sweeps on EDM2-S and DiT-XL/2. Across the sweep, VOG mitigates the coverage loss that standard CFG exhibits at high guidance scales (e.g., maintaining Recall where CFG drops below 0.50) and avoids the accompanying FID spike, which widens the range of over which guidance remains usable.
Figure S3. Quantitative comparison on EDM2-S (512 × 512) across varying guidance scales. We report FID (↓), Precision (↑), and Recall (↑) as a function of the guidance weight . VOG (red) demonstrates superior robustness at high scales compared to standard CFG (blue), maintaining an FID of ≈ 2.4 at while the baseline degrades beyond 4.0.
Figure S4. Quantitative comparison on DiT-XL/2 (256 × 256) across varying guidance scales. While standard CFG suffers from sharp performance degradation at high guidance strengths (e.g., FID > 3.40 at ), VOG maintains structural fidelity (high Recall) and generation quality (FID < 2.60), effectively preventing manifold drift.
Composition with other guidance heuristics. VOG is complementary to existing heuristic schedules rather than a replacement for them. Applied on top of Cosine, Interval, and AutoGuidance (Table S12), it improves FID in every case, for instance reducing Interval CFG from 1.637 to 1.523 on EDM2-L, which supports its use as a plug-in guidance correction.
| Benchmark | FID | Precision | Recall |
| EDM2-S | |||
| +AutoGuidance | 1.044 | 0.701 | 0.626 |
| +VOG (Ours) | 1.031 | 0.696 | 0.620 |
| EDM2-L | |||
| +Standard CFG | 1.993 | 0.710 | 0.601 |
| +VOG (Ours) | 1.418 | 0.714 | 0.597 |
| +Cosine CFG | 1.667 | 0.726 | 0.587 |
| +VOG (Ours) | 1.649 | 0.727 | 0.588 |
| +Interval CFG | 1.637 | 0.730 | 0.597 |
| +VOG (Ours) | 1.523 | 0.720 | 0.593 |
Table S12. VOG composed with different guidance heuristics on ImageNet 64 × 64. Each shaded row applies VOG on top of the heuristic immediately above it.
Figure S2b. FID as a function of the momentum coefficient and the VJP recomputation interval , on DiT-XL/2 (256 × 256).
Figure S2: Ablation studies on guidance-scale sensitivity (left) and hyperparameter robustness (right).
Reward and Evaluation Histories
We report 100-update training runs for ReFL, DRTune, DiffusionOPSD, and DVOC (), with confined to the ablation. The primary seven-metric comparison remains the fixed, paired update-20 evaluation in the main paper.
Training reward and evaluation reward. The training reward is . ReFL and DRTune score one-step clean predictions from sampled late states, whereas DVOC and DiffusionOPSD score complete sampled endpoints. These quantities are plotted in separate panels in Figure S5; the two upper panels share their vertical scale. The lower panels show the HPSv3++ and CLIP components for DVOC and DiffusionOPSD.
Figure S5. Training reward histories. Separate panels distinguish sampled endpoints from late-state proxies. Thin lines show every raw update mean; strong lines show the trailing five-update mean. The vertical reference marks update 20. The HPSv3++ and CLIP panels show DVOC and DiffusionOPSD.
All raw update means are shown as thin, translucent lines. The stronger line is a trailing mean over at most five updates, using the same window for every method. The horizontal axis indicates the optimizer update associated with each training batch, from 1 to 100. Each point is the mean reward on that batch.
Online Evaluation
We evaluate each method at initialization and every ten updates through update 100, giving 11 checkpoints. Every evaluation uses the EMA adapter, the same 1,000 prompts, outputs, nine sampling steps, guidance scale zero, and HPSv3++ iteration condition 0.0. Online evaluations use each trainer’s noise sampling scheme; the main comparison pairs initial noise across methods and uses HPSv3++ condition 0.3.
Figure S6 reports HPSv3++ and CLIP as in-domain metrics, alongside the held-out ImageReward evaluator. Markers show measured checkpoint means, with straight lines connecting successive evaluations. The first two panels of Figure 3 show the same CLIP and HPSv3++ histories. The combined evaluation reward is 0.03846 HPSv3 + + + CLIP.
Figure S6. Online evaluation histories. HPSv3++ and CLIP are in-domain evaluators; ImageReward is held out of training. Each marker is a checkpoint mean, with no smoothing. The vertical line marks update 20. The HPSv3++ condition and noise sampling differ from the paired evaluation in the main table.
For DVOC, the online weighted evaluation reward rises from 0.766 at update 20 to 0.911 at update 100. ImageReward rises from 1.267 to 1.366 over the same interval.
Active-Step Histories
Figure S7 complements the fixed update-20 ablation with all recorded histories for . Each run uses the same nine-step sampler and two-step PGD target construction. Panel (a) shows training reward with the same raw-line and trailing-mean convention as Figure S5; panels (b)–(f) show unsmoothed online EMA evaluation. The three variants have exactly matching initialization scores on all four displayed evaluators.
Figure S7. DVOC active-step ablation over optimization. appear only in the ablation; the main method remains . Training reward includes all raw points and a trailing five-update mean. Evaluation panels show only recorded checkpoint means. The vertical line marks the main paper’s update-20 checkpoint.
Larger improves the later weighted evaluation reward in these runs, while retains a higher
late-training AES score. ImageReward varies without a consistent ordering across all checkpoints. These histories complement the fixed update-20 comparison; per-update costs are reported in Table S7.
References
- [1]Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proc. ICLR, 2024.arxiv.org/abs/2302.07121
- [2]Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In Proc. ICLR, 2024.arxiv.org/abs/2305.13301
- [3]Muthu Chidambaram, Khashayar Gatmiry, Sitan Chen, Holden Lee, and Jianfeng Lu. What does guidance do? a fine-grained analysis in a simple setting. In Proc. NeurIPS, 2024.
- [4]Hyungjin Chung, Jeongsol Kim, Michael Thompson McCann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In Proc. ICLR, 2023.arxiv.org/abs/2209.14687
- [5]Hyungjin Chung, Jeongsol Kim, Geon Yeong Park, Hyelin Nam, and Jong Chul Ye. CFG++: Manifold-constrained classifier free guidance for diffusion models. In Proc. ICLR, 2025.arxiv.org/abs/2406.08070
- [6]Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly fine-tuning diffusion models on differentiable rewards. In Proc. ICLR, 2024.arxiv.org/abs/2309.17400
- [7]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proc. CVPR, 2009.
- [8]Alexander Denker, Francisco Vargas, Shreyas Padhy, Kieran Didi, Simon Mathis, Vincent Dutordoir, Riccardo Barbano, Emile Mathieu, Urszula Julia Komorowska, and Pietro Lio. DEFT: Efficient fine-tuning of diffusion models by learning the generalised h-transform. In Proc. NeurIPS, 2024.
- [9]Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Proc. NeurIPS, 2021.
- [10]Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky T. Q. Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. In Proc. ICLR, 2025.
- [11]Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangkook Lee, and Kimin Lee. DPOK: Reinforcement learning for fine-tuning text-to-image diffusion models. In Proc. NeurIPS, 2023.arxiv.org/abs/2305.16381
- [12]Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In Proc. ICCV, 2023.DOI
- [13]Yingqing Guo, Hui Yuan, Yukang Yang, Minshuo Chen, and Mengdi Wang. Gradient guidance for diffusion models: An optimization perspective. In Proc. NeurIPS, 2024.arxiv.org/abs/2404.14743
- [14]Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J. Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. In Proc. ICLR, 2024.arxiv.org/abs/2311.16424
- [15]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Proc. NeurIPS, 2017.
- [16]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.arxiv.org/abs/2207.12598
- [17]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. NeurIPS, 2020.arxiv.org/abs/2006.11239
- [18]Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. In Proc. NeurIPS, 2024.arxiv.org/abs/2406.02507
- [19]Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proc. CVPR, 2024.arxiv.org/abs/2312.02696
- [20]Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-Pic: An open dataset of user preferences for text-to-image generation. In Proc. NeurIPS, 2023.arxiv.org/abs/2305.01569
- [21]Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. In Proc. NeurIPS, 2024.
- [22]Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-GRPO: Training flow matching models via online RL. In Proc. NeurIPS, 2025.arxiv.org/abs/2505.05470
- [23]Yijun Liu, Jie Huang, Zeyue Xue, Yuming Li, Ruizhe He, Haoran Li, Shijia Ge, and Siming Fu. HPSv3++: Scaling reward models across the full spectrum of diffusion model capabilities. arXiv preprint arXiv:2606.14657, 2026.arxiv.org/abs/2606.14657
- [24]Toan Nguyen, Kien Do, Duc Kieu, and Thin Nguyen. h-Edit: Effective and flexible diffusion-based editing via Doob’s h-transform. In Proc. CVPR, 2025.arxiv.org/abs/2503.02187
- [25]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proc. ICCV, 2023.arxiv.org/abs/2212.09748
- [26]Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In Proc. ICLR, 2021.arxiv.org/abs/2104.08894
- [27]Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. CoRR, abs/2310.03739, 2023.arxiv.org/abs/2310.03739
- [28]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748–8763, 2021.arxiv.org/abs/2103.00020
- [29]Seyedmortreza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS: Unleashing the diversity of diffusion models through condition-annealed sampling. In Proc. ICLR, 2024.arxiv.org/abs/2310.17347
- [30]Seyedmortreza Sadat, Otmar Hilliges, and Romann M. Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In Proc. ICLR, 2025.arxiv.org/abs/2410.02416
- [31]Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Proc. NeurIPS, 2018.arxiv.org/abs/1806.00035
- [32]Christoph Schuhmann. LAION-Aesthetics. LAION blog, 2022.
- [33]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proc. NeurIPS, 2019.arxiv.org/abs/1907.05600
- [34]Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proc. ICLR, 2021.
- [35]Tongyi-MAI. Z-Image-Turbo. Model card and released checkpoint, 2025.
- [36]Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proc. CVPR, 2024.arxiv.org/abs/2311.12908
- [37]Ruoyu Wang, Boye Niu, Xiangxin Zhou, Yushi Huang, Tongliang Liu, and Chi Zhang. Exploring the design space of reward backpropagation for flow matching. arXiv preprint arXiv:2606.11075, 2026.arxiv.org/abs/2606.11075
- [38]Xi Wang, Nicolas Dufour, Nefeli Andreou, Marie-Paule Cani, Victoria Fernández Abrevaya, David Picard, and Vicky Kalogeiton. Analysis of classifier-free guidance weight schedulers. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2404.13040
- [39]Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, Qiong Yan, Xiongkuo Min, Guangtao Zhai, and Weisi Lin. Q-Align: Teaching LMMs for visual scoring via discrete text-defined levels. In Proc. ICML, pages 54015–54029, 2024.arxiv.org/abs/2312.17090
- [40]Xiaoshi Wu, Yiming Hao, Manyuan Zhang, Keqiang Song, Zhaoyang Huang, Guanglu Song, Yu Liu, and Hongsheng Li. Deep reward supervisions for tuning text-to-image diffusion models. In European Conference on Computer Vision, pages 108–124. Springer, 2024.arxiv.org/abs/2405.00760
- [41]Mengfei Xia, Nan Xue, Yujun Shen, Ran Yi, Tieliang Gong, and Yong-Jin Liu. Rectified diffusion guidance for conditional generation. In Proc. CVPR, 2025.arxiv.org/abs/2410.18737
- [42]Jiazhen Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and evaluating human preferences for text-to-image generation. In Proc. NeurIPS, 2023.arxiv.org/abs/2304.05977
- [43]Shuchen Xue, Chongjian Ge, Shilong Zhang, Yichen Li, and Zhi-Ming Ma. Advantage weighted matching: Aligning RL with pretraining in diffusion models. In Proc. ICML, pages 143146–143169, 2026.arxiv.org/abs/2509.25050
- [44]Zhiyuan You, Xin Cai, Jinjing Gu, Tianfan Xue, and Chao Dong. Teaching large language models to regress accurate image quality scores using score distribution. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14483–14494, 2025.arxiv.org/abs/2501.11561
- [45]Jiwen Yu, Yinhua Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. FreeDoM: Training-free energy-guided conditional diffusion model. In Proc. ICCV, 2023.arxiv.org/abs/2303.09833
- [46]Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, and Ming-Yu Liu. DiffusionNFT: Online diffusion reinforcement with forward process. In Proc. ICLR, 2026.arxiv.org/abs/2509.16117
- [47]Wei Zhou, Xiongwei Zhu, Lingdong Kong, Bo Chen, Lei Zhang, Yongyuan Liang, Xiaoxia Hou, Ye Tian, Xian Sun, Yingshuo Wang, Linfeng Li, Shengqiong Wu, Leigang Qu, Feng Li, Wei Liu, Julian McAuley, and Tat-Seng Chua. On-policy self-distillation in diffusion models. arXiv preprint arXiv:2608.24646, 2026.arxiv.org/abs/2608.24646








