WorksComputer Science
Spectral Steering: Local Inverse Geometry for Frozen Diffusion Generators
Abstract
A frozen diffusion generator may need to change several output properties while preserving others. We study when guidance faithfully realizes such a structured request and whether local inverse correction improves its direction. Our account links this benefit to the interaction between the request and the objective-specific remaining-sampler Jacobian: conditioning alone is insufficient. Spectral steering combines a request-specific cosine-gain predictor with damped inverse correction and fixed-time relinearization for finite edits. Scalar nulls and matched-movement controls test the predicted mechanism. Across 8,923 instances in a retrospectively identified local regime, predicted and realized gains correlate at Spearman 0.79, versus -0.11 for condition number. Experiments span images, audio, robotics, and control and planning, including complex character motion generation. On 50 paired inpainting cases, four corrections improve unobserved-pixel PSNR over Adam by 3.25 dB at equal total derivative traversals. Together, the analysis and controls connect request-dependent geometry to directional gains and establish where these gains translate into downstream performance.
Introduction
Frozen diffusion generators must often satisfy new tasks without retraining (Figure 1). Differentiable guidance propagates output objectives to sampling states [1, 5, 36], while differentiation through generation enables direct latent optimization [30, 2, 14]. We ask when gradients faithfully realize a specified pattern of downstream changes, and whether directional correction gains can be predicted.
Figure 1. Spectral steering of a frozen generator. Spectral steering uses a damped local inverse of the requested downstream properties.
Requests combine change and preservation: brighten one region while holding another fixed, alter frequency bands while preserving context, or move a character’s wrist while keeping its feet planted. Loss decrease alone does not describe which components move. We evaluate request fidelity alongside task success.
The relevant local input–output map is the Jacobian of the selected properties through the entire remaining sampler. Pulling a request back as predicts , so unequal singular gains can alter its relative components. Spectral steering uses classical damped least squares [16, 19] to approximately compensate these gains. For finite edits, it refreshes the Jacobian and residual toward the original target at fixed diffusion time (Figure 2).
Figure 2. From gradient pullback to local inversion. The frozen remaining sampler maps an intervention to downstream properties. FO reweights a request by ; the damped inverse compensates unequal gains. K2/K4 refresh the Jacobian and residual at the same diffusion time. Right: unsteered, matched FO, and K4 character motions at two common instants; red spheres mark skeleton–obstacle intersection.
Correction gains depend on the request. Even an ill-conditioned Jacobian preserves a supported single-mode request’s direction. Distortion depends jointly on the spectrum and the request’s modal composition. We express this dependence as a prediction of cosine improvement. An exact scalar null and matched-movement controls make the account experimentally testable beyond aggregate method rankings.
Downstream systems require selected quantities to change by specified amounts while others remain fixed. Character motion requires coordinated joint changes and preservation; robotics adds execution dynamics, and content editing requires preserved context. Our experiments connect predicted directional gains to realized edits and task outcomes, including failures of local approximation or downstream transfer.
Our contributions are:
(1) Request-specific geometry of faithful steering. We characterize how the objective-specific remaining-sampler geometry distorts structured requests, identifying the role of modal composition and an exact scalar null after magnitude matching (§ 3 and § 5.1).
(2) Predicting directional gains and refining finite edits. Our score predicts local cosine improvement over first-order guidance. Spectral correction implements the damped inverse, and fixed-time relinearization refreshes the geometry for finite requests (§ 3 and § 5.2).
(3) Controlled evidence across generative domains. Movement controls and optimizer comparisons test directional gains, task success, and executed outcomes across images, audio, robotics, and control and planning—beginning with complex character motion generation (§ 5).
Related Work
Gradient guidance. Classifier guidance, Universal Guidance, Loss-Guided Diffusion and DPS condition sampling through gradients ([6, 1, 26, 5]); FreeDoM uses energy-based guidance ([36]). These methods establish general differentiable conditioning; our question is how faithfully a gradient transmits a specified vector displacement.
End-to-end optimization. Earlier text-guided face editing uses GAN inversion ([33]). Full-process differentiation is established: FlowGrad, DOODL and D-Flow optimize generative states through the resulting output ([17, 30, 2]); DNO applies this principle to motion and DITTO to music ([14, 20]). D-Flow also analyzes the generator Jacobian’s action on output gradients. OC-Flow formulates optimal control ([31]), and posterior and end-to-end guidance have been connected theoretically ([3]). We examine how a prescribed vector request interacts with this geometry, predicting directional correction gains and testing them under movement controls.
Geometry and inverse guidance. DDRM and DDNM exploit measurement structure ([15, 32]); ΠGDM uses measurement-model pseudoinverses for conditional-score estimation [25]. Spherical Gaussian and manifold-preserving guidance constrain intermediate or data-manifold geometry [35, 9]; Park et al. analyze latent singular directions [21]. JAPS studies denoiser Jacobians, while CLAMP applies denoiser-pulled curvature and manifold-aligned damping to posterior correction [10, 23]. We study the selected downstream representation composed with the complete remaining sampler, asking which requests benefit from inverse correction. The contribution combines a request-dependent prediction with scalar nulls, movement controls, and finite-edit evaluation. The damped least-squares update supplies the correction; it does not define a posterior sampler.
Planning and policy steering. Offline demonstration learning highlights the role of data and evaluation in manipulation [18]. Diffuser and Diffusion Policy generate trajectories and actions [12, 4]; SafeDiffuser and JM2D address constrained planning [34, 13]. DynaGuide uses a dynamics model to steer policies, whereas DSRL learns a latent-noise controller from environment interaction [7, 29]. Spectral steering requires a differentiable specification without training another controller. Execution and safety remain separate evaluation requirements. Appendix A summarizes the control interpretation.
Objective-Specific Local Inverse Geometry
Structured requests and local geometry
At fixed diffusion time , let be the intervention state and the downstream properties. The map includes the complete remaining sampler, decoding, and task-specific coordinate scaling; conditioning and remaining randomness are fixed. Its Jacobian is
For target , the request specifies changes and held properties. We seek a finite displacement , using the local model
Preservation is a soft least-squares target; omitted properties are uncontrolled. Gradient descent on gives
This is a descent direction when nonzero, but its predicted response need not align with . A scalar step size changes the response magnitude without correcting the relative components. The relevant geometry depends on both the generator and the downstream representation: changing the measurements or their scaling changes the operator being inverted.
Damped inversion and request-specific prediction
For , spectral steering solves the regularized local problem
One correction is K1. The equivalent primal and dual forms solve in state and property space, respectively. For compact SVD and request , with , the predicted responses are
For , inverse gains approach one; weak modes remain attenuated and is locally unreachable. A supported single-mode request has collinear response regardless of the condition number.
Directional benefit therefore depends on which modes the request occupies, not just the spectrum. For , magnitude matching makes the latent updates identical (Appendix § 11).
Writing , we predict the directional gain by
For nonzero requests and responses, the score predicts local cosine gain before editing. Different requests can have different predicted gains at the same Jacobian. Agreement with nonlinear edits is tested empirically; magnitude and task utility are evaluated separately.
Fixed-time relinearization and computation
K2/K4 repeat the correction toward the original target. At , recompute , set , and update
The index counts corrections, not denoising steps; the final output is . Each candidate completes the same remaining sampler, measures its residual relative to the original target, and updates the intervention state. Keeping that target fixed avoids requesting the full edit again at every correction. Refreshing the local model addresses nonlinear error without guaranteeing convergence (Figure 3). The coefficient 0.01 is fixed; “adaptive” damping means refreshing at each correction, rather than a line search or a latent-norm bound.
Figure 3. Structured steering and fixed-time relinearization. (a) Changes and preservation must be realized jointly through the frozen generator. (b) Successive corrections refresh the Jacobian and residual toward the original target at the same diffusion time. denotes the target action set that becomes reachable after steering.
Explicit solvers use reduced-SVD filtering without hard rank truncation and obtain from the Jacobian. Matrix-free solvers use up to 16 request-started Golub–Kahan iterations, estimate from the projected matrix, and solve the damped problem there. Each iteration uses one VJP and one JVP through the remaining sampler: one correction costs up to 16 of each, and corrections up to of each. “Direct” inversion refers to the local inverse relation, not iteration-free computation. Numerical-support thresholds are diagnostic, separate from damping; setting-specific thresholds, stopping rules, and solver variants are in Appendix § 19.§ 1.
Experimental Protocol
Vision uses a CelebA-HQ DDPM [11] for restoration and Stable Diffusion 1.5 [22] for structured edits. Audio covers MAESTRO waveform excerpts [8] and generated mel spectrograms; robotics measures MuJoCo execution [28]. Control and planning begins with 60-frame MDM character motions [27], followed by trajectory and dynamical-system tasks. Table 1 identifies models; Appendix § 22 defines objectives and endpoints.
| Data / simulator | Diffusion model | Task / endpoint | Unst. | FO | SGD | Adam | L-BFGS | Spectral (K1) | Spectral (best K) |
| Vision | |||||||||
| CelebA-HQ | DDPM EMA-256 | Inpainting PSNR ↑ | 16.55 | 19.61 | 25.57 | 25.98 | 25.76 | 23.69 | 27.75 (K4) |
| 4× SR PSNR ↑ | 25.05 | 26.32 | 27.69 | 27.79 | 27.58 | 27.39 | 27.90 (K2) | ||
| Deblur PSNR ↑ | 24.03 | 25.26 | 27.26 | 27.92ns | 27.90 | 26.61 | 27.94 (K16)ns | ||
| Multi-constraint PSNR ↑ | 25.81 | 26.12 | 27.69 | 28.61 | 28.54 | 26.89 | 27.75 (K4) | ||
| Generated images | Stable Diff. 1.5 | Structured edit [0-1] ↑ | 0.000 | 0.574 | 0.734 | 0.774 | 0.765 | 0.954 | 1.000 (K8) |
| Audio | |||||||||
| MAESTRO 150k | Localized EQ err. ↓ | 7.000 | 5.878 | 6.623 | 4.692 | 3.895 | 5.538 | 2.691 (K16) | |
| Bandwidth LSD ↓ | 12.757 | 13.408 | 13.151 | 26.604 | 13.487 | 12.733 | 12.730 (K4) | ||
| Generated mel | Audio Diffusion | EQ-16 success ↑ | 0.000 | 0.32 | 1.00 | 1.00 | 1.00 | 0.98 | 1.00 (K2) |
| Selective EQ dB ↓ | 2.5981 | 1.0883 | 0.0005 | 0.0300 | 0.0005 | 0.3935 | 0.0004 (K4) | ||
| Robotics | |||||||||
| MuJoCo | Temporal DDPM | Panda reach err. cm ↓ | 3.89 | 2.39 | 6.54 | 2.03 | 0.63 | 1.06 | 0.74 (K16) |
| Panda pose err. cm ↓ | 5.10 | 2.28 | 0.82 | 0.97 | 0.79 | 1.09 | 0.70 (K4) | ||
| Cheetah target cm ↓ | 16.16 | 11.57 | 6.79 | 4.18 | 5.74 | 6.67 | 4.31 (K16) | ||
| Cheetah traj. cm ↓ | 13.29 | 9.00 | 5.57 | 6.19 | 5.19 | 6.44 | 4.72 (K16) | ||
| Panda contact [0-1] ↑ | 0.000 | 0.769 | 0.766 | 0.851 | 0.926 | 0.718 | 0.894 (K16) | ||
| Control & planning | |||||||||
| Generated motion | MDM | Char. obstacle [0-1] ↑ | 0.000 | 0.000 | 0.960ns | 0.880 | 0.980 | 0.180 | 0.980 (K16)ns |
| Char. reaching [0-1] ↑ | 0.000 | 0.000 | 0.480 | 0.000 | 0.960ns | 0.140 | 0.960 (K4)ns | ||
| 2D integrator | Temporal DDPM | 1-waypoint err. cm ↓ | 110.86 | 30.35 | 37.69 | 7.48 | 26.34 | 7.41 | 3.05 (K4) |
| Multi-waypoint [0-1] ↑ | 0.000 | 0.223 | 0.873 | 0.534 | 0.945 | 0.854 | 1.000 (K4) | ||
| Point-mass [0-1] ↑ | 0.000 | 0.983 | 0.999 | 0.997 | 1.000ns | 0.997 | 1.000 (K4)ns |
Table 1. General optimizers across domains. Bold: best displayed value, including rounded ties. Settings and selected budgets vary by task. ns: no statistically significant difference between the two marked methods in the same row. GD/Adam learning rates selected by a hyperparameter search, L-BFGS learning rate selected on the evaluation set. Character optimizers match K4’s nominal backward-pass count. Protocols and checkpoints: Appendices G, E, and L.
Matching and baselines. FO and K1 share the initial state, time, sampler, objective, and request. FO matches K1’s predicted objective-space movement with for nonzero denominator, using the task’s scaled coordinates. It reuses K1’s Jacobian or corresponding Krylov quantities; this isolates direction, not standalone cost. A separate control matches latent displacement. SGD, Adam, L-BFGS and domain-specific baselines use fixed, pilot-selected, or evaluation-selected settings as documented in Appendix § 17; the main table is not a common-budget ranking.
Evaluation. We distinguish request fidelity, application outcomes, and executed behavior. Paired comparisons are used where available. Compute comparisons separately count vector–Jacobian products (VJPs), total VJP/Jacobian–vector-product (JVP) traversals, or network evaluations; these do not establish equal wall-clock cost. Shared-task adaptations are distinguished from native baseline protocols. A matrix-free spectral solve uses up to 16 VJPs and 16 JVPs, whereas each SGD/Adam step uses one VJP; explicit implementations have different derivative representations. Sample sets, uncertainty, solver mappings, and qualitative selection rules are specified with the results and in the appendix.
Results and Evaluation
Correcting modal gains improves structured responses
Figure 4(a,b) connects a four-region audio request to the modal mechanism: the pullback overweights sensitive modes, while inverse responses lie within 5% of the request in the displayed example. The scalar null supplies a falsifiable control: all 935 tested pairs tie to machine precision at . Benefits emerge with multiple coordinates. At 32 audio cells, joint success is 4% for FO, 82% for K1, and 100% for K2 (), showing improved simultaneous realization and the value of relinearization.
Local geometry predicts directional gains
On 50 character-motion avoidance seeds, predicted and observed cosine gains correlate at , (Figure 4(d)). Across 8,923 instances in 27 vision, audio, and robotics populations within the measured local regime, and , versus and for log condition number. Centering within populations retains , and Pearson association is positive in 26 of 27 populations, supporting an association beyond pooled between-task differences (Appendix § 14.§ 2). The score is computed before editing, but regime selection is retrospective; this supports a local diagnostic, not a validated acceptance rule.

Figure 4. Testing the directional mechanism. (a) Four-region audio request; spectrograms share a color scale (time right, frequency up). (b) FO overweights sensitive modes; inverse responses lie within 5% of each displayed request. (c) At 32 audio cells, FO/K1/K2 attain 4/82/100% success (; 95% intervals). (d) Predicted versus realized cosine gain on 50 character-avoidance seeds (, ); red line: linear fit. Appendix § 14.§ 2 reports the cross-domain analysis.
Matched movement. Across 576 instances with latent norms agreeing within , K1 improves request realization on 100% of inpainting images, 85% of audio excerpts, and 93% of Panda instances. Interpolating toward the inverse at fixed predicted response norm improves reconstruction by 5.34 dB, with improvements on all 50 images at every tested step (Appendix § 12). These controls isolate a directional benefit. They also separate the proposed mechanism from simply taking a larger step: the latent-norm control fixes intervention size, while the interpolation fixes predicted downstream movement (Appendix § 12).
Vision: reconstruction and structured edits
Generic optimizers. In Table 1, K4 reaches 27.75 dB paired inpainting PSNR versus Adam’s 25.98. Structured editing gives cosine 0.954 for K1 and 1.000 for K8, versus 0.774 for Adam and 0.765 for L-BFGS. Spectral also leads reported super-resolution PSNR. Adam leads composite restoration; the K16 deblurring comparison is statistically unresolved and costs eight times Adam’s network evaluations in that audit.
Residual versus reconstruction. On the separate paired trajectory in Figure 6, Adam peaks at 25.60 dB near step 54, then loses 1.59 dB by step 256 despite decreasing measurement loss. K4 reaches 28.32 dB: it exceeds Adam@64 by 2.74 dB at equal VJP count and Adam@128 by 3.25 dB at equal total derivative traversals (95% interval [2.32, 4.35]; 86% paired wins). K1 trails Adam under both conventions (Appendix § 16); the benefit here comes from repeated geometry correction.
Figure 6. Residual minimization can degrade reconstruction. (a–c) Adam and spectral use 50 paired reference images; SGD/L-BFGS use the separate 50-image optimizer set also in (d). Adam loses 1.59 dB after its peak while residual falls. Dashed guides mark 64 VJPs; dotted guides mark clean-image residual. L-BFGS uses recorded backward counts. (d) Log-scale UNet evaluations charge VJP and JVP traversals. Paired budget comparisons: Appendix § 16.
Guidance and structured edits. In Figure 5(a–Figure 5d), spectral exceeds selected Universal Guidance by 0.133 dB on super-resolution and 0.140 dB on deblurring, with no significant paired difference on inpainting or composite restoration; guidance obtains early gains at lower cost on three operators. At a 4× structured-edit request, K1/K2 reach cosine 0.929/0.996 versus 0.845 for CLAMP. The stress test probes realization of an analytic property beyond reconstruction. These are descriptive objective adaptations; marginal intervals alone do not establish paired significance. Across the evaluated settings, spectral has no below-prior outcomes, while selected unstable guidance settings do. Implementation and sensitivity details are in Appendices § 17 and § 20.

Figure 5. Vision accuracy and reconstruction. (a–d) CelebA-HQ/DDPM PSNR, 50 reference images per task, 95% intervals; hollow DPS uses deblur and composite . UG uses selected settings; native DPS uses 1000 steps. (e) SD1.5 request cosine at 4× scale (window 10, ). Deeper spectral rungs are K2 in (b,e), K4 elsewhere. DPS, ΠGDM, and CLAMP in (e) are objective adaptations. (f) One paired reconstruction. Protocols and differing image sets: Appendices § 17 and § 20.
Audio: precise change and preservation
Across four practical mel-domain tasks, K2 attains 100% joint success versus 8–82% for FO. At 64 backward passes on 30 selective-EQ clips, K4 achieves 0.00038 dB all-coordinate RMSE versus Adam’s 0.02997, winning all pairs. SGD reaches 0.00047 dB with no significant difference from K4 (); all three satisfy the success threshold. The precision comparison remains informative after the binary success threshold saturates: K4 and Adam both pass, but differ in realized coordinate errors. Transition edits further show that meeting the measured request can increase an unoptimized abruptness diagnostic. These are acoustic-control results, without a listening-study claim (Appendix § 14.§ 1).
Waveform localized EQ reaches 2.691 dB error at K16 versus 3.895 for L-BFGS at the reported settings. Bandwidth extension is budget sensitive: Adam’s error is 26.604 dB at 16 steps versus K4’s 12.730, but early-stopped Adam reaches 12.525 with no significant paired difference. Table 1 therefore establishes neither uniform superiority nor equal-cost advantages across audio tasks.
Robotics: transferring corrections to execution
On 276 Panda instances, K1 reduces median executed reaching error from FO’s 2.39 cm to 1.06 cm; learned DSRL reaches 6.01 cm after 8,000 environment episodes per seed. DSRL learns from black-box rewards; spectral uses differentiability at inference time, so their access and deployment conditions differ. Figure 7(a–c,e) covers reaching, preservation, clearance, and paired manipulation examples.

Figure 7. Robotics and planning outcomes. (a,b) Executed error medians and 95% intervals for Panda reaching, orientation/height preservation, and Cheetah goal/path tracking. (c) Contact-free execution rates. Unsteered fails the reach, so its contact-free score is omitted. (d) Endpoints normalized by unsteered (lower is better): Maze2D distance, dense/novel-obstacle collision, arm error, shifted-goal failure. Counts: Appendix G. (e) Paired obstacle-clearance (left) and constant-height (right) examples; symbols identify methods via the shared legend.
Table 1 shows benefits on Panda pose and Gymnasium HalfCheetah trajectory tracking ([18]). On shared reaching instances, K8 ties L-BFGS at 0.63 cm; K16’s 0.59 cm is not a significant improvement. Greater clearance need not increase joint reach-and-clear success, and depth can be non-monotone. The unsteered contact-free score is omitted from the plot because it fails the reach. The qualitative rows pair obstacle clearance with a constant-height constraint. Generated fidelity predicts executed improvement more strongly once requests exceed servo resolution, reaching rank correlation 0.77; smaller corrections can be attenuated during tracking. Execution and refinement analyses are in Appendices § 13 and § 17.
Control and planning
Character motion generation. A frozen MDM produces three-second clips with 25 deterministic DDIM transitions remaining at intervention. On 50 paired seeds per task, avoidance () and multi-objective reaching () combine pelvis or wrist changes with preservation (Figure 2; Appendix § 15). K1 improves cosine over movement-matched FO from 0.458 to 0.976 and from 0.701 to 0.931, winning every pair. Relinearization reduces remaining positional error. At K4-matched nominal backward-pass counts, K4 achieves 96% joint success on both tasks versus L-BFGS’s 98% and 96%. On reaching, K4 limits bone drift to 1.05 cm versus 5.66 for SGD and 7.20 for Adam. Joint success checks positions and jerk; collision, velocity, and bone drift are separate diagnostics. For example, Adam satisfies the avoidance success gate in 88% of cases but is collision-free in 78%. This distinguishes coordination of measured motion properties from additional physical constraints. The comparison matches nominal backward passes, not standalone runtime. These results support structured motion control while leaving generic optimizers competitive.
Trajectory control and planning. Across 45 executed waypoint pairs, K4 reduces Adam’s mean error from 7.48 to 3.05 cm (59%; 41/45 wins; paired difference interval [2.96, 5.86] cm), also improving on fixed GD and L-BFGS-B settings (Appendix § 17). Figure 7(d) shows shifted-goal success rising from FO’s 26.5% to K4’s 77.4%, although K1 reaches only 2.6%; unseen-obstacle collision falls from 79.5% to 11.8%. Specialized guidance can achieve lower collision rates. The relative ranking depends on goal accuracy, clearance, and budget (Appendix § 18).
Discussion and Limitations
The evidence supports correction of structured requests whose local model remains informative. Inverse corrections and nonlinear optimizers both refine residuals; their relative outcomes depend on task and budget, and relinearization need not improve monotonically. Coordinate scaling determines the geometry; omitted properties receive no protection. Generated fidelity can be attenuated or overwritten during execution and replanning. The predictor’s pooled validation uses retrospective regime selection and concerns cosine alignment rather than task success. Derivative counts do not imply runtime gains. Robotics is simulated and audio has no listening study; the appendix analyzes these boundaries alongside optimizer counterexamples.
Toward precise downstream control. A natural direction is to use the local inverse as a precision layer within closed-loop diffusion policies, which already support receding-horizon execution (Chi et al., [4]). New observations would trigger a new geometry estimate and requests for tracking, clearance, or terminal-state changes. Differentiable dynamics and conservation residuals could encode physical-law targets. Such a layer would require execution-aware validation and independent safety enforcement: local request fidelity alone does not guarantee constraint satisfaction. A complementary direction replaces hand-designed properties with learned semantic features, requesting targeted edits while preserving selected attributes. Its value depends on feature calibration, local controllability, and whether the representation measures the intended semantics. Appendix § 11 formulates these directions and their evaluation requirements.
Conclusion
Spectral steering connects a structured request’s modal composition to its distortion through a frozen generator, the predicted benefit of local inverse correction, and the realized outcome. Scalar nulls and matched-movement controls test this connection; cross-domain experiments establish its practical relevance, including complex character motion generation. Classical damped least squares supplies the correction, while fixed-time relinearization extends its use to finite edits. The evidence supports request-dependent geometry as a useful control interface, with benefits bounded by local-model accuracy, objective coverage, and downstream execution. Future work should further characterize how precise generator-space steering transfers to downstream tasks, including closed-loop control under physical and safety constraints and targeted semantic editing with learned representations, while deriving theoretical locality bounds on the maximum reliable steering magnitude.
AI Use Statement
Generative AI tools were used for code assistance, result interpretation, and manuscript drafting/editing. The authors remain responsible for citation accuracy, originality, policy compliance, and the final manuscript.
Ethics Statement
This work studies inference-time steering of pretrained generators and inherits their misuse and deployment risks. Robotics experiments are simulator based and do not establish safety guarantees. Improved request fidelity is not guaranteed task success.
Reproducibility Statement
The supplement records objectives, checkpoints, matching protocols, optimizer budgets, paired statistics, figure source data, and available qualitative selection rules. The accompanying additional material supports the paper’s claims with code and media files. Media files include image, video, and audio samples of the steered models. The appendix specifies implementation and evaluation protocols.
Supporting Analysis and Comparisons
We give the scalar control and support diagnostic, then summarize the control interpretation and complementary domain findings. Detailed protocols and qualitative examples follow.
Local model and exact scalar null. Section 3 gives the local model and primal/dual solves; the dual system has dimension rather than . For , write with and . The inverse is . Matching predicted objective-space movement sets FO’s step size to , making the latent updates, and hence the outputs under the same sampler randomness, identical. The 935-pair scalar experiment tests this exact null; Appendix B tests the multi-coordinate case.
Damping and numerical support. The coefficient is fixed; is refreshed at each correction. For nonzero , numerical reachability is
The SD1.5 diagnostic retains without truncating the damped correction. Measured real-task requests were fully reachable, so natural rank deficiency is not established here. If , no first-order response is available. Appendix I.1 specifies solver variants, thresholds, and stopping rules.
Local inversion as a control interface. The Jacobian maps perturbations of the full intermediate state to selected properties; its damped inverse maps a request back to an intervention. “Direct” describes this inverse relation: matrix-free solves and finite-edit relinearization still iterate. A future precision layer could re-estimate this geometry from each new observation within a receding-horizon diffusion policy (Chi et al., [4]), using tracking, clearance, energy, or dynamics residuals as properties. For example,
A zero dynamics target requires ; zero displacement instead preserves the current residual. Execution-aware validation and independent safety enforcement remain necessary. Learned semantic properties offer a complementary direction, with edit and preservation quality evaluated independently of the optimized features. Neither extension is evaluated here.
Table conventions. Bold marks numerical winners, including rounded ties. The two ns-marked methods in a row have no reported statistically significant difference; this does not establish equivalence. The structured-edit unsteered cosine of 0.000 is a display convention for an undefined zero-edit cosine. Qualitative selection rules appear in Appendix K.
Vision. The selected-guidance comparisons give spectral advantages of 0.133 dB on super-resolution () and 0.140 dB on deblurring (). Appendix J reports the setting-specific instability counts, and Appendix F gives the paired optimizer comparisons.
Audio. K1 raises localized mel-EQ success from 8% to 94% and paired-clip consistency from 25% to 100%. On 30 selective-EQ clips at 64 backward passes, K4/Adam/SGD latent displacements are 7.85/19.03/7.02. Appendix D.1 reports coordinate errors and the unoptimized abruptness diagnostic; these are acoustic-control measurements, not listening-study outcomes.
Robotics. DSRL’s 6.01 cm reaching error improves its separate 23.6 cm training reference after 8,000 episodes per seed over five training seeds; spectral’s unsteered reference is 3.89 cm. DSRL amortizes learned adaptation, while spectral differentiates at inference time. Figure 10 shows how request scale affects transfer to execution. A via-point example improves generated cosine by 0.0674 without detectable executed waypoint improvement ().
Figure 10. When generated fidelity transfers to robot execution. For Panda reaching, Spearman correlation between the K1-minus-FO gain in generated request cosine and the FO-minus-K1 reduction in executed target error, within each request size. Error bars show 95% bootstrap intervals (–59 per size). The correlation rises from 0.11 at 1 cm to 0.77 at 16 cm; shading marks requests below the servo’s resolution. This scale dependence complements the transmission failure in Figure 9b.
Control and planning. After K1, median character errors are 2.92 cm for avoidance and 2.21 cm for reaching, above their 1.66 cm and 1.46 cm tolerances. Directional fidelity therefore need not complete the request (Appendix § 15). For trajectory control, Appendix § 17 gives paired executed errors; Appendix § 18 compares clearance and goal accuracy. Dropping Panda clearance coordinates reduces target error from 1.30 to 0.70 cm while lowering contact-free execution from 83.2% to 61.9%, illustrating the importance of objective coverage (Appendix § 13).
Connection to prior control and guidance. End-to-end optimization, including D-Flow [2], establishes full-process differentiation and output-gradient geometry; damped least squares and relinearization are classical (§ 2). Here the predictor, scalar null, and movement controls test how request–geometry interactions explain realized directional gains. Finite-edit and execution results test their practical consequences. The predictor requires derivative information and is not a certified step-acceptance rule.
The Correction Is a Direction
The scalar null in Appendix § 11 predicts identical updates for a single property. The multi-coordinate controls here test whether spectral steering benefits from its update direction after accounting for movement magnitude. Figure 8 first fixes latent displacement, then varies the direction at fixed predicted downstream movement. The two controls separate geometry correction from simply taking a larger step.
Figure 8. (a) With latent-displacement norms agreeing within , the spectral direction realizes the request better on almost every instance in three modalities. (b) The fractional family at exactly matched predicted objective-space magnitude interpolates from the first-order pullback () to the damped inverse () and moves reconstruction monotonically, improving on 50 of 50 images at every step. Neither panel is a compute comparison; both vary only the direction.
Failure Modes and Scope
Local request fidelity is only one requirement for useful steering. Figure 9 separates incomplete specifications, loss of a correction during execution, and an excessive requested change. The transfer analysis and optimizer counterexample below identify further limits on the main-paper gains.

Figure 9. (a) Dropping the clearance coordinate improves the executed target error and lowers contact-free execution, and the strongest rung loses the most: a more faithful inverse of an incomplete request is a worse outcome. (b) A signed speed request is realized on the generated reference at cosine 0.998 and the robot executes 41% of what was asked; generated and executed quantities are separate bars because they are separate measurements. (c) At a doubled request the deeper rungs pursue the overshoot faithfully and finish below one solve.
A complete optimizer counterexample. On the separate frozen dense-obstacle B2 task (), tuned optimizers outperform the spectral ladder on the preregistered request residual. At about 128 backward passes, SGD and L-BFGS achieve mean residuals 0.0446 and 0.0422 versus K4’s 0.6364; L-BFGS also has higher mean minimum clearance (0.09988 versus 0.09294). Thus the executed waypoint advantage does not establish a general planning advantage. An exploratory path-length difference in this campaign is not used to reverse its primary-endpoint conclusion.
Across these cases, useful steering requires objective coverage, a valid local model, and reliable downstream execution.
Additional Domain Analyses
This section supplements the main results with acoustic precision and locality analyses, followed by the pooled directional-prediction audit. It distinguishes an accurately realized request from independent quality criteria and documents how the local-regime population is selected.
Practical audio protocol and remaining boundaries
The practical audio campaign uses one frozen mel-domain prior with differentiable time–frequency and loudness coordinates. At the headline request, localized EQ, two-clip transitions, reference tonal matching, and paired-clip consistency contain 50, 50, 50, and 40 paired instances. All coordinates must meet the campaign’s predefined tolerances. The localized-EQ optimizer comparison uses the first 30 instances of request A1c, with three change and thirteen preservation coordinates. The selective-EQ row in Table 1 reports the mean of the per-instance RMSE over all sixteen coordinates, not a mean of the two block RMSEs. Adam uses learning rate 0.01 and SGD 0.1, selected using 32-step runs on five disjoint pilot seeds (900–904), then frozen for all evaluation budgets. Budgets of 16, 32, and 64 backward passes correspond to K1, K2, and K4 in this explicit-Jacobian audio implementation; these counts are not wall-clock seconds.
At 64 passes, change / preservation RMSE is 0.000308 / 0.000392 dB for spectral, 0.034511 / 0.028056 for Adam, and 0.000413 / 0.000477 for SGD. The mean all-coordinate error reduction relative to Adam@64 is 0.02958 dB (95% paired bootstrap interval [0.02760, 0.03144], 30/30 wins; two-sided Wilcoxon ). Against SGD@64 the interval includes zero (). SGD wins at smaller budgets and nearly matches spectral at 64 passes; no significant spectral advantage over SGD is claimed. On transitions, every steered variant increases unoptimized energy abruptness from 9.47 to 11.3–11.6 dB. Separate waveform-prior tests show that one solve does not improve bandwidth extension and that preservation requires coordinates that actually isolate the held region: waveform localized EQ worsens pinned cells by 0.0834 dB at . These outcomes do not imply a listening-study or naturalness benefit. Figure 11 examines how request scale changes success and local-model accuracy.
Figure 11. Audio request scale separates direction from locality. At the calibrated edit, one solve improves success; at twice the request, relinearization is needed. FO and spectral incur similar linearization error at matched scales (16-cell diagnostic), so their intermediate-scale difference is directional. These scale sweeps complement the practical tasks in the main text.
Predictor scope
We compare the predicted and realized gain in request cosine, then examine where the local model ceases to be informative. Figure 12 summarizes the retained populations; Figure 13 shows the excluded nonlinear regimes. Selection is retrospective, as detailed below.
Figure 12. Cross-domain prediction of directional benefit. Each point is one of 8,923 instances from 27 vision, audio, and robotics populations in the measured local regime. The horizontal axis is the pre-edit predicted cosine gain; the vertical axis is the observed cosine gain of K1 over magnitude-matched FO (, ). Positive vertical values favor K1. Population-level local-regime selection uses nonlinear responses and is retrospective. The motion illustration in Fig. 4d is a separate diagnostic population and is not included here.
Figure 13. The local-model boundary. The 1,439 instances from the three populations excluded by the retrospective linearization-error criterion. Their broad realized responses show why the headline local-regime correlation should not be extrapolated to large nonlinear edits.
The complete diagnostic collection contains 30 populations and 10,362 instances. Three populations have median one-shot linearization error above one: Panda rotation with position preservation, Maze2D goal reaching, and the test-time obstacle. Removing those leaves the 27 populations and 8,923 instances in Figure 12, with positive Pearson correlation in 26 populations. After centering within each population, the pooled correlations remain Pearson 0.74 and Spearman 0.77. Regime assignment uses measured nonlinear responses and is retrospective, so these results do not establish an automatic pre-edit acceptance rule.
On a separate composite-restoration evaluation (), the predictor correlates positively with realized cosine gain (, ), but negatively with change-coordinate error reduction (, ). Directional fidelity and error reduction are distinct endpoints; neither establishes independent reconstruction quality.
Control and Planning: Character Motion Generation
Population and generator. We report results on 50 seeds per task, using the same seeds across methods. Comparator hyperparameters were selected on six separate calibration instances, and the jerk threshold was calibrated on 24 additional generator samples. The frozen text-conditioned MDM (Tevet et al., [27]) generator produces 60 frames at 20 fps using a native 50-step sampling schedule and classifier-free guidance 2.5. Steering acts on an intermediate HumanML3D feature state of dimension , with 25 deterministic DDIM transitions remaining (Song et al., [24]). All local corrections act at this same diffusion time.
The main table reports joint success under the positional-target, preserved-position, and jerk criteria defined below, whereas Table 2 reports target-only success. These criteria must not be interchanged. The main-table entries for SGD, Adam, and L-BFGS use the K4-matched nominal backward-pass count; this does not match forward evaluations or standalone runtime.
| Task | Endpoint | Unst. | FO | K1 | K2 | K4 | K16 |
| Obstacle | error | 16.64 | 10.32 | 3.54 | 0.95 | 0.23 | 0.01 |
| Obstacle | success % | – | – | 18 | 88 | 98 | 100 |
| Obstacle | – | – | 0.164 | 0.140 | 0.135 | 0.132 | |
| Multi-objective | error | 10.62 | 4.85 | 2.52 | 1.40 | 0.79 | 0.19 |
| Multi-objective | success % | – | – | 14 | 58 | 96 | 98 |
| Multi-objective | – | – | 0.188 | 0.204 | 0.210 | 0.185 |
Table 2. Character-animation relinearization ladder (). Target error is averaged over the 50 seeds and reported in cm; success is the fraction within 10% of the requested amplitude. measures deviation of the realized change from the tangent prediction.
Solver and first-order control. We differentiate the scaled objectives through the complete remaining sampler and differentiable joint reconstruction. Each correction uses the explicit Jacobian with reduced-SVD filtering and , recomputed at each relinearization toward the fixed target . FO is analytically scaled to match K1’s predicted objective-space displacement norm.
Requests and preservation coordinates. Directions are expressed in a subject-centred frame measured from each baseline motion. The requested magnitude is one natural excursion of that motion: 16.6 cm of lateral pelvis travel for avoidance and 14.6 cm of wrist-relative-to-pelvis travel for reaching. Obstacle avoidance uses : a 20 cm sphere is placed on the body path, the mid-clip pelvis is moved far enough to include a 25% clearance margin, and the final pelvis plus seven arm-bone lengths are preserved. Multi-objective reaching uses : the wrist follows a five-frame profile, both feet are pinned at two additional frames, a zero wrist-velocity target is imposed on arrival, and bone lengths are preserved at all waypoint frames. Joint success requires target error below , preserved-position error below , and mean third-order joint-position differences below m, computed without time-step normalization. Here, m for avoidance and m for reaching. Target error is the pelvis-position error at the avoidance waypoint or the mean wrist-target distance over the five reaching waypoints. Preservation error is the maximum displacement over preserved joint–frame positions. Position and bone-length coordinates are divided by m, and velocity coordinates by m/s before constructing the objective and its Jacobian.
Table 2 shows how relinearization reduces positional target error across both tasks.
A separate computational benchmark on the task reports 8.61 s for 77 optimizer steps, compared with 3.80 s for sequential Jacobian-row construction and 0.95 s for vectorized construction. The Jacobian rows share one forward graph, whereas optimizer steps evaluate successive states. These component timings characterize Jacobian-construction efficiency; they do not measure the complete standalone runtime of K4.
Figure 14 reports character-motion correlations separately from the 27-population aggregate.

Figure 14. The request-specific predictor transfers to character animation. , computed before editing, predicts the realized cosine gain of K1 over magnitude-matched FO on obstacle avoidance and multi-objective reaching (Pearson and ; Spearman for both; seeds per task). Raw conditioning does not predict that gain within either task (right; and ). Correlations are computed within each task to avoid between-task offsets. The endpoint is cosine gain.
Table 4 contrasts deep refinement with adapted guidance. The normalized residuals of matched FO and K1 are respectively 0.972 and 0.260 on avoidance, and 0.738 and 0.379 on multi-objective reaching. Their paired cosine and leakage comparisons appear once, in Table 3.
| Task | Quantity | FO | K1 | 95% CI | W/L | ||
| Obstacle | fidelity cosine | 0.458 | 0.976 | +0.519 | [+0.492, +0.546] | 50/0 | |
| Obstacle | leakage | 1.215 | 0.219 | −0.997 | [−1.073, −0.918] | 50/0 | |
| Obstacle | bone drift (m) | 0.011 | 0.011 | +0.000 | [−0.000, +0.000] | 27/23 | 0.67 (ns) |
| Multi-objective | fidelity cosine | 0.701 | 0.931 | +0.229 | [+0.216, +0.242] | 50/0 | |
| Multi-objective | leakage | 1.477 | 0.507 | −0.970 | [−1.040, −0.894] | 50/0 | |
| Multi-objective | bone drift (m) | 0.011 | 0.010 | −0.000 | [−0.000, −0.000] | 27/23 | 0.67 (ns) |
Table 3. One spectral solve versus magnitude-matched FO (, paired). Both methods start from the same and request and are matched in predicted objective-space displacement norm. Leakage is unintended movement relative to requested movement.
| Method | Obstacle cosine | Obstacle residual | Multi-objective cosine | Multi-objective residual |
| Spectral (K16) | 1.000 | 0.001 | 0.996 | 0.044 |
| DPS-adapted | 0.290 | 1.605 | 0.575 | 1.777 |
| UQG-adapted | 0.435 | 1.466 | 0.325 | 1.619 |
Table 4. Deep refinement and adapted guidance (). A normalized residual above one means the result is farther from than the unsteered baseline.
Intervention time and guidance adaptation. Moving the obstacle intervention from 25 remaining denoising steps to only two improves the normalized residual of DPS-adapted from 1.605 to 0.930 and Universal Guidance from 1.466 to 0.437, but conjunctive success remains 0% for DPS and rises only to 8% for Universal Guidance. K4 remains at 96–98% over the same sweep, while K1 degrades from 18% to 0%, exposing its sensitivity to a single local model. Universal Guidance operates in the generator’s clean motion-feature space. The reported configuration uses five backward-guidance steps and two self-recurrences, for 300 backward passes.
Paired Adam Budget Audit
The same 50 inpainting cases support two complementary compute comparisons: one matches VJP counts and the other counts both VJPs and JVPs. Table 5 reports reconstruction differences with paired uncertainty. Adam leads one spectral solve at these budgets, whereas K4 leads under both conventions; neither convention establishes equal wall-clock cost.
| Accounting | Paired methods | PSNR dB | 95% interval | Spectral win % |
| VJP count | Spectral (K1) vs Adam@16 | −0.96 | [−1.45, −0.51] | 26 |
| VJP count | Spectral (K4) vs Adam@64 | +2.74 | [+1.78, +3.89] | 78 |
| Total traversals | Spectral (K1) vs Adam@32 | −2.07 | [−2.73, −1.44] | 18 |
| Total traversals | Spectral (K4) vs Adam@128 | +3.25 | [+2.32, +4.35] | 86 |
Table 5. Paired inpainting reconstruction versus Adam at two resource-accounting conventions (). The difference is Spectral minus Adam on missing-pixel PSNR; positive values favor Spectral. Each spectral solve costs 16 VJPs and 16 JVPs, while each Adam iteration costs one VJP. Intervals are paired BCa 95% intervals. The K4 leads survive Holm correction; bold marks the winner in each row.
Comparison Protocols
This section specifies sample sets, optimizer selection, and aggregation for the main comparisons, in table order: vision, audio, robotics, and control and planning, beginning with character motion generation.
Vision restoration. The inpainting table uses 50 paired images (760–809); the guidance and optimizer-trajectory figures use images 200–249, with a spectral ladder on each set. On the table’s pairs, SGD/Adam/L-BFGS minus K4 PSNR is dB, with respective 95% intervals , , and . For SR, deblurring, and composite restoration, SGD/Adam use 50 images and L-BFGS uses 20. L-BFGS selects the best evaluation PSNR over 1, 2, 4, 8, 16, 32, 64 iterations, choosing 16/64/64 (32/128/128 backward calls).
Structured-edit optimizer row. SD1.5 uses the standard-scale box-average color request (Objective B, window 10), seeds 6000–6019, and realized-request cosine. GD/Adam use pilot-frozen ten-step runs at rates 100/0.01; spectral uses refreshed relative damping and peaks at K8. L-BFGS reaches 0.765 at ten backward passes (five strong-Wolfe iterations), with the same regularized loss as GD/Adam, . Its rate 1.0 is evaluation-selected; rates 0.1/0.3 are ineligible because each stalls on one case. Paired tests use in-process GD/Adam reruns; the table retains their frozen summaries. Figure 5(e) uses 4× requests.
Structured-edit guidance. The 4× stress test uses objective adaptations. CLAMP implements the published equations with stand-in constants, without validation against an official code example. The displayed marginal intervals do not establish paired significance; spectral collateral change was not measured on the same pass.
Waveform audio and budget sensitivity. SGD/Adam use 25 shared excerpts (100–124), with rates selected on the tuning objective using two disjoint excerpts. Table 1 reports 16-step results. Bandwidth extension evaluates high-band reconstruction LSD, not the optimized measurement loss. Selecting the best evaluation LSD over 1, 2, 4, 8, 16 steps yields SGD@1 (rate 0.3) and Adam@2 (rate 0.01). Adam@2 reaches 12.525 dB versus K4’s 12.730, with paired K4-minus-Adam interval [−0.44, 0.83] dB and . This stopping-budget sensitivity test does not compare the table’s Adam@16 cell.
Localized EQ reports requested-region error (on err_db). SGD/Adam use rates ; SGD’s near-zero step reflects its tuning rule. The completed row includes L-BFGS at 3.895 dB and K16 at 2.691 dB. Appendix D.1 describes the separate mel-audio protocol.
Robotics refinement depth. On shared reaching instances, K8 and L-BFGS both have median error 0.63 cm (); K16 reaches 0.59 cm without a significant paired difference. HalfCheetah target error worsens at K8 before recovering at K16. On obstacle reaching, K4/K8/K16 joint success is 75.8/75.5/76.2% ( for K4 versus K16), while contact-free execution increases from 83.2% at K4 to 89.4% at K16. Panda depth tests are paired within one re-execution.
Character motion generation. The character comparison uses 50 seeds per task, with separate calibration instances and nominal backward-pass accounting. Appendix E defines its requests, matching rule, success criteria, and solver. Target-only ladder success and the main table’s joint success are distinct reported endpoints.
Executed one-waypoint control. The additional table row reanalyzes the recorded executed waypoint endpoint from the optimization extension: three frozen missions, fifteen seeds each, with all methods on exactly the same 45 pairs. A six-coordinate request specifies the timed waypoint, terminal position, and terminal velocity at a frozen diffusion intervention. The SGD column is full-batch GD (30 steps, rate 0.5); Adam uses 40 steps at rate 0.3. The L-BFGS column is SciPy L-BFGS-B with no variable bounds, 100 maximum iterations and 300 maximum objective/gradient evaluations. All three minimize the squared nonlinear request error plus ; their settings are fixed by the source protocol, without a reported learning-rate sweep. Spectral uses the same frozen target with its damped local solves. The record count for K4 is 28 Jacobian-building forward/backward calls; the L-BFGS-B objective/gradient count averages 85.3. These counters combine different operations and do not establish a runtime ratio.
Against GD, Adam, and L-BFGS-B, the paired mean executed-error reductions are 34.65, 4.43, and 23.30 cm; the respective percentile bootstrap 95% intervals are [29.46, 40.09], [2.96, 5.86], and [13.38, 34.49] cm (20,000 paired resamples). All three two-sided Wilcoxon tests remain below 10−8 after Holm correction over these comparisons. The uncertainty is conditional on the three tested missions and chosen settings. Generated task success is 100% for both K4 and L-BFGS-B, whereas executed success is 84.4% and 53.3%; Adam executes successfully on 71.1%. This distinction motivates reporting execution rather than generated success.
Planning endpoints in Figure 7d. Maze2D uses 553 available evaluations for unsteered, FO, and K1, and 501 for K4. Dense-obstacle and novel-obstacle collision rates use the paired reruns with 315 and 263 executions, respectively. Plan-arm error uses 380 evaluations per method, and shifted-goal success uses 1,440. The plot divides distance, error, or collision rate by the unsteered value within each task; shifted-goal success is first converted to failure rate. Thus one denotes the unsteered endpoint and smaller values are better, without averaging incompatible physical units. The accompanying source data retain all raw endpoints and counts.
General Guidance on Executed Obstacles
This planning comparison evaluates general guidance on the same simulator executions as spectral steering. Table 6 reports collision and goal-request error separately, because the multi-coordinate objective can trade off clearance against the requested goal change.
Table 6: Novel test-time obstacle, paired executions. Collision is physical; request error scores the goal move in generator space. Passes are campaign backward-pass units. Bold marks the best displayed number in each endpoint column; unsteered has the lowest request error here.
| Method | Passes | Collision % | Request error |
| Unsteered | 0 | 100.0 | 0.350 |
| FO | 45 | 79.5 | 0.522 |
| Spectral (K1) | 45 | 52.5 | 0.677 |
| Spectral (K4) | 180 | 11.8 | 0.417 |
| Universal Guidance, fixed | 16 | 9.5 | 0.756 |
| Universal Guidance, fixed | 256 | 1.1 | 0.442 |
Table 6.
On the dense-clearance condition (), K4 and the best guidance setting record 1.6% and 0.0% collisions respectively; the gap is small at this low base rate. The novel-obstacle condition supplies the discriminating comparison. Unsteered has lower median goal error than K4 here (0.350 versus 0.417), so the relative K4–guidance request advantage is not an absolute goal-accuracy gain. The 45-coordinate objective contains two goal coordinates, with the remainder for clearance and preservation. Request errors are heavy-tailed; comparisons use medians and paired win rates.
Optimizer Trajectories and Resource Accounting
Figure 15 puts reconstruction outcomes on a shared network-evaluation axis and distinguishes the evaluated image populations. The accounting below explains how the local inverse and optimizer updates contribute to that cost; the solver summary specifies the corresponding implementations.
Figure 15. Outcome against compute on one common denominator of network evaluations, on random-mask inpainting. The ladder is drawn on both instance subsets because the optimizer runs used a disjoint set (‡). The specialized inverse methods buy early reconstruction gains at lower network cost. Optimizer lines use the 50-image cohort; hollow optimizer markers are separate 20-image extension endpoints. The paired reconstruction-versus-residual trajectory is in Figure 6.
One Golub–Kahan iteration requires one VJP and one JVP through the remaining sampler. With the 16-iteration cap, one matrix-free correction uses up to 16 VJPs and 16 JVPs; corrections use up to of each. For the vision configuration with eight remaining denoising steps, this corresponds to 256 derivative-traversal equivalents per correction, versus 128 for 16 SGD/Adam steps. These counts exclude primal forward evaluations, checkpoint recomputation, and endpoint evaluation. Reported timings on the shared GPU are approximately 6.55 s per Krylov iteration and 3.44 s per SGD/Adam step. Magnitude-matched FO reuses K1’s computation in the directional control; this shared accounting is not its standalone cost.
Solver implementation
The same local inverse is evaluated with different derivative representations according to the output dimension and generator. The table maps the experiments to explicit or matrix-free implementations; the paragraphs below specify damping, numerical support, and the iterative stopping rule.
| Experimental setting | Jacobian and local solve |
| CelebA-HQ restoration; waveform localized EQ and bandwidth extension | Matrix-free JVPs/VJPs; 16-step Golub–Kahan projection and a direct damped solve in the projected space. |
| CIFAR and SD1.5 structured edits; mel-audio | Explicit reverse-mode Jacobian; reduced SVD and spectral filtering. |
| Executed robotics | Explicit Jacobian with batched reverse-mode rows sharing a forward graph; row-wise fallback; reduced SVD. |
| Early robot geometry diagnostics | Explicit sampler Jacobian composed with the objective Jacobian by the chain rule; reduced SVD. |
| Character motion generation | Explicit Jacobian through the remaining sampler and joint reconstruction; batched reverse-mode rows with a row-wise fallback; reduced SVD (Appendix E). |
| Custom trajectory control and standard 2D planning | Explicit reverse-mode Jacobian and reduced SVD; 2D planning uses batched rows with a row-wise fallback. |
Table 15.
Damping and support. In the audited implementations, “adaptive” means , refreshed at every local correction. Explicit solvers use and apply without hard rank truncation. The SD1.5 reachability diagnostic retains ; that campaign separately excludes (an absolute threshold). The common audio/control/robotics cores log numerical rank using ; this is a separate diagnostic. Named damping ablations vary the relative coefficient: the SD1.5 helper defines , and the inpainting runs use and the default. The SD1.5 fixed-damping control instead uses per objective and intervention time. None of these rules adds a damping line search.
Matrix-free solve. Starting from , Golub–Kahan bidiagonalization constructs , , with full reorthogonalization against the stored vectors of both bases. The cap is 16 iterations; early termination occurs only if a newly computed unnormalized basis vector has norm at most . There is no residual-based stopping tolerance or preconditioner. Basis storage and projected algebra use float64; derivative evaluations retain the model’s dtype. The implementation sets using an SVD of , then computes
No separate power iteration is used. The logged residual is that of these projected normal equations, not a stopping test or a full-space convergence certificate. A zero request returns a zero correction. The request-started projection is not used to establish full-Jacobian numerical reachability.
Universal Guidance Protocol
We distinguish selected guidance settings from an explicit instability audit. Table 7 records failures at one unstable setting per vision operator; it is not the selected-setting comparison in the main figure. The remaining text specifies the vision, audio, and control-and-planning protocols.
| Vision operator | UG below prior | Spectral below prior |
| Inpainting | 4/50 | 0/150 |
| 4× super-resolution | 50/50 | 0/150 |
| Deblurring | 40/50 | 0/150 |
| Multi-constraint | 50/50 | 0/150 |
Table 7. Failures below the unguided prior at an unstable guidance setting. Counts for Universal Guidance (UG) use one evaluated unstable setting per operator (); spectral counts pool its three evaluated rungs (). These are robustness counts, not the post-hoc best-setting PSNR comparison shown in Figure 5.
Figure 5 reports Universal Guidance alongside the vision baselines, using the selected settings from the frozen operator audit: fixed for inpainting, fixed for super-resolution, and normalized for deblurring and multi-constraint restoration.
On the 50 reference inpainting images, selected UG reaches 28.39 dB versus K4’s 28.32, with no significant paired difference. The dense Adam trajectory peaks at 25.60 dB at 54 steps (Figure 6).
The separate audio guidance comparison uses the unconstrained active-preservation setting. For control and planning, the character motion generation adaptation is described in Appendix § 15. The constrained-planning results use the same-loop reruns with all methods paired ( dense, novel obstacle); the separate UG comparisons use fixed and fixed , respectively. Figure 7d uses these paired reruns.
The audio comparison uses forward guidance plus self-recurrence, with scale selected on separate tuning examples. At , unconstrained Universal Guidance reaches 0.0501 dB error with latent norm 58.8, versus K1’s 0.4481 dB and norm 6.8; displacement matched, it reaches 1.5760 dB. At , the corresponding values are 0.0327 versus 0.5308 dB unconstrained and 1.5356 versus 0.5308 dB matched. This is evidence that nonlinear guidance can optimize the endpoint and that movement control changes the question.
Qualitative Results
The gallery complements the examples alongside the domain plots in Figure 2, Figure 5, and Figure 7. Each panel is an existing experiment output. The organizing question is whether the requested coordinates change while the named preservation coordinates and downstream behavior remain satisfactory.
Selection and interpretation. Captions identify recorded median, quantile, boundary, and high-effect examples. Video-derived control and planning panels use synchronized instants selected for maximum change-from-initial separation between FO and the highest displayed spectral rung. These are illustrations, not estimates of typical performance; printed metrics describe individual cases. Crops preserve aspect ratios. Task definitions and endpoints are in Appendix § 22; aggregate comparisons appear in the quantitative results.
Vision
Figures 16–17 show inverse problems whose loss is a measurement residual. Figure 18 shows paired-image and multi-region objectives built from regional color, agreement, and preservation coordinates. Printed metrics describe the displayed instances.

Figure 18. Paired-image and multi-region objectives from the vision extension.

Figure 16. Random-mask inpainting: a frozen optimizer comparison, an upper-quartile case, and a boundary case. Methods share the observation within each example; printed PSNR and error maps describe that instance.

Figure 17. Vision inverse-problem examples: super-resolution and deblurring. The reconstruction and error maps complement PSNR by locating residual errors. The upper-quartile, boundary, and median cases illustrate variation within the stored evaluations.
Audio
Figures 19, 20, and 21 organize the audio examples by objective support. Equalization rows expose the time–frequency cells that enter equation (11); restoration rows expose observed and missing frequency or time regions; transition rows expose the temporal quarters, contexts, and seams used by their dB objectives. Spectrograms locate changes in time and frequency. They are visual diagnostics; perceived quality requires listening evidence, and held-cell errors remain part of the quantitative evaluation.

Figure 19. Localized time–frequency equalization shown as spectrograms.
Vector (PDF)caption: Localized EQ automation: low-mid cut in the middle and high-band boost late in the clip.
Figure 19: Localized and automated audio equalization. Requests identify both edited and held time–frequency regions; Appendix D.1 reports their quantitative errors.

Figure 19. Known-filter equalization restoration.
Figure 19: Audio equalization (continued): restoration toward a supplied target spectrum.
Figure 20. Bandwidth-extension reconstruction. Missing-segment waveform inpainting. Figure 20: Audio restoration: bandwidth extension and missing-segment inpainting.

Figure 20. Crescendo temporal-envelope edit shown as spectrograms. Figure 20: Audio restoration and temporal control (continued): crescendo-envelope editing.


Figure 21. Crossfade from clip A to clip B across four temporal quarters. Interior high-band boost and low-band cut with both contexts and both seams held. Figure 21: Audio transition objectives. Dashed or boxed boundaries in the spectrograms mark the temporal regions entering the cost.
Robotics
Each row is a synchronized frame from the stored execution videos. The first two groups show position, orientation, checkpoint, and plane-preservation objectives (Figure 22 and 23). The third shows whole-arm clearance constraints, including cases where the first-order rollout contacts the obstacle while a spectral rollout remains clear (Figure 24). These manipulation examples were selected as high-effect illustrations. A contact-free unsteered rollout that fails to complete the reach does not establish task success.

Figure 22. Manipulation, top to bottom: Cartesian reach, XYZ-marker reach, and orientation hold.

Figure 24. Executed obstacle avoidance, top to bottom: unseen obstacle, lateral clearance, and narrow gap. Selected high-effect cases; contact-free execution must also complete the reach.

Figure 23. Manipulation, top to bottom: via-point reach and constant-height plane constraint.

Figure 23 (PDF p. 32). Manipulation (continued): planar-arm pose control.
Locomotion
Figure 25 retains the rollout renderings and their printed progress and posture outcomes. The cost requests forward progress over eight replanning windows while holding torso height and pitch. This source-declared high-effect example illustrates closed-loop transmission; the quantitative reference-to-execution limitations remain in Appendix C.

Figure 25. Forward progress over eight planning windows. All references are executed by the same objective-blind tracking controller.
Control and planning
The gallery starts with character motion generation, where the specification coordinates several joints over time. Canonical control and trajectory planning follow, showing how the same interface applies to dynamical states and routes.
Character motion generation
Figure 26 and Figure 27 show avoidance and reaching with the same camera and instance across methods.

Figure 26. Obstacle-avoidance character sequence. Five instants surround the unsteered motion’s deepest penetration. Rows show unsteered, magnitude-matched first-order, and Spectral (K4). A red sphere indicates that any skeleton segment intersects it. Figure 2 uses two instants from this same sequence.
Figure 27. Multi-objective character reaching. Columns show five requested instants; rows show unsteered, magnitude-matched first-order, and Spectral (K4), with a fixed camera. Stars mark requested wrist positions; the active waypoint is solid. Foot positions, zero arrival velocity, and bone lengths enter the objective; velocity and bone drift are separate diagnostics from the reported success criterion. The inherited small sphere is not a clearance objective for this task.
Canonical control
The pendulum request (Figure 28) specifies checkpoint angles and terminal angular velocity; CartPole requests a positive terminal cart displacement with upright, stationary terminal targets (Figure 29). The renderings are paired with full state traces, so a selected frame can be checked against the complete rollout. Frame selection follows the visual-separation rule above.

Figure 28. Pendulum reference tracking, shown on a source-declared high-effect example. The angular reference and full traces complement the selected rendering.

Figure 29. CartPole terminal-position request: selected illustrative rollout. CartPole terminal displacement with upright-angle and zero-velocity terminal coordinates. The upper strip renders the selected state, while the lower plots show the full state trajectories and the requested terminal cart position.
Trajectory planning
Figure 30 shows timed-waypoint tracking and obstacle avoidance with a double integrator. Black dots mark endpoints, orange stars waypoints, green crosses goals, and red disks obstacles. Frames maximize the change-from-initial separation between FO and Spectral (K4), following the gallery selection rule.
Figure 30. Four timed waypoints. Waypoint and goal with obstacle avoidance. Double-integrator trajectory objectives. Method order is fixed as Unsteered, First-order, Spectral (K1), and Spectral (K4).
Figure 31 shows a shifted Maze2D goal and a narrow passage. Path diagnostics report goal error, clearance, and smoothness; negative clearance denotes a violation. Paths are generated references unless execution is explicitly reported.
Figure 31. Goal-conditioned and obstacle-aware diffusion planning. Each trajectory panel uses the same method order: Unsteered, First-order, Spectral (K1), and Spectral (K4).
Task Objectives and Evaluation Endpoints
The following catalogue connects the common steering interface to the task-specific experiments and preceding gallery. It supplements the quantitative comparisons with the definition of what each method is asked to change, what is explicitly held, and how the outcome is evaluated. Parenthesized source keys identify the corresponding task campaigns.
Datasets and frozen diffusion models. The model names in Table 1 abbreviate the actual frozen checkpoints. Vision restoration in the main table uses google/ddpm-ema-celebahq-256 (DDPM EMA-256) [11] on CelebA-HQ validation images [33]. The structured vision edits use an additional model: stable-diffusion-v1-5/stable-diffusion-v1-5 (Stable Diffusion 1.5) [22], with generated images from the campaign’s frozen prompt/seed pools. This is the model in Figure 5(e), the spatial and composite editing experiments, and the paired-image/region-transfer qualitative examples. The table uses standard-scale edits; the guidance figure uses requests.
Waveform EQ and bandwidth extension use harmonai/maestro-150k (MAESTRO 150k) on MAESTRO v3 test excerpts [8]. EQ-16 and selective EQ use teticio/audio-diffusion-ddim-256 (Audio Diffusion), evaluated on generated log-mel spectrograms.
Temporal DDPM denotes a task-specific Diffuser-style temporal U-Net, trained before steering and then frozen. Panda reaching uses m1_generator_v1.pt; Panda pose and obstacle-aware reaching use m2_generator_v1.pt. Both HalfCheetah rows use loco_halfcheetah_generator_v2.pt. Execution uses MuJoCo [28]: Panda position servos or a custom tracking wrapper around Gymnasium’s HalfCheetah model [18]. The character extension uses a frozen text-conditioned MDM [27] (60 frames at 20 fps, 50 DDIM steps, classifier-free guidance 2.5), with 25 deterministic transitions remaining at the intervention, acting on the HumanML3D feature representation. We use the HumanML3D MDM checkpoint humanml_enc_512_50steps/model000750000.pt, with a 512-dimensional transformer encoder and eight layers.
The custom two-dimensional double-integrator experiments use distinct frozen checkpoints: di_generator.pt for one-waypoint control, ds_generator.pt for multi-waypoint tracking, and r1_generator_v5.pt for the point-mass cosine task. Sharing an architecture label does not imply sharing weights across these tasks.
Request, preservation, and outcome
For the frozen remaining sampler in equation 1, write and specify an absolute target . The nonlinear request loss optimized by the generic gradient comparators is
Coordinates use the units and any fixed scaling declared by the task; a pixel residual, a dB error, and a normalized physical-state error are not interchangeable quantities. A zero displacement holds a measured coordinate at its unsteered value. An absolute target such as terminal velocity zero instead specifies , unless the anchor is already stationary. Coordinates absent from receive no preservation constraint.
Spectral steering solves the damped local model in equation (4)–equation 5; denotes the number of local solves at the same diffusion time. Generic SGD, Adam, and L-BFGS optimize the nonlinear loss through the remaining sampler, adding a fixed displacement penalty where the source protocol specifies one (as in Appendix G), with the resource conventions in Appendices F and I. The magnitude-matched FO control isolates direction; the external guidance methods retain the operator-specific protocols and disclosed adaptations of Figure 5.
Request fidelity measures how well realizes , for example for nonzero . PSNR on unobserved pixels and physical execution provide separate tests of whether this realization is useful. The catalogue labels fidelity diagnostics separately from independent endpoints; neither an accurate request nor a convincing single example establishes a population-level quality or execution advantage.
Vision
The inverse-problem objectives compare a generated image after a known measurement operator with the stored observation. Structured edits instead reduce the image to analytic regional statistics. Background or boundary preservation is part of the loss only when those coordinates are explicitly included with zero requested change. The vision gallery mirrors these two groups: restoration strips first, then regional, paired-image, and multi-region edits. Table 8 specifies their measurement maps and endpoints.
| Task / source | Measurement map and target | Evaluation |
| Random-mask inpainting (V0) | Retained pixels in DDPM data units. Target: ; unobserved pixels are omitted, not preservation coordinates | PSNR on unobserved pixels |
| 4× super-resolution (V0) | DPS bicubic antialiased resizer . Target: | Full-image PSNR |
| Gaussian deblurring (V0) | Reflection-padded 61 × 61 Gaussian convolution . Target: | Full-image PSNR |
| Two-region restoration (V0) | . Target: Restore low-resolution ; preserve exactly observed | PSNR on |
| Multi-region color control (V0) | Mean CIELab coordinates in each named region, . Target: Fixed per-region color displacement; included preservation regions request zero change | Dense on preserved pixels |
| Spatial-statistic editing (V1) | under the smooth luminance weight . Target: Shift the weighted center and spreads; is calibrated from frozen pilot statistics | Objective fidelity |
| Combined color–geometry–texture (V1) | , where . Target: Pilot-scaled displacement in the raw objective coordinates; the apple large-edit illustration uses this objective at 4× scale | Objective fidelity and outside-region change |
| Paired compositing/appearance family (V2) | Analytic CIELab destination cells, luminance spread, inner/outer boundary rings, and background preservation blocks. Target: Reference-derived requested blocks; background entries are explicitly zero | Boundary and appearance diagnostics not optimized as semantic scores |
| Reference patch transfer (V3) | CIELab statistics over 20 destination edit coordinates and 56 background coordinates. Target: Match the supplied source patch in the destination rectangle; request zero change outside it | Dense outside-region |
| Two-image joint edit (V3) | Concatenated regional CIELab cells for two jointly steered images, followed by preserve cells from both images. Target: Drive the two named regions to agreement; request zero change over the remaining cells | Region disagreement and per-image collateral change |
Table 8. Vision objective and cost functions. Pixel-space inverse problems use the measurement residual itself; structured editing uses analytic color, spatial, boundary, and texture statistics.
The spatial-moment map uses normalized smooth luminance weights on an image grid in . In the combined objective, pilot standard deviations scale the requested direction before its normalization in the raw measurement space; they do not imply that every loss coordinate was whitened. The separate benchmark in Figure 5(e) uses the linear RGB box-average map, at diffusion window 10 and request scale (source V4). It is distinct from the combined color–geometry–texture apple illustration.
Audio
For waveform localized equalization, with STFT , four temporal segments and six log-frequency bands,
The other audio rows in Table 9 either use the same dB construction on a different partition or use the stated linear measurement operator directly. For a dB-cell task, the target is additive: requesting +6 dB means that the target coordinate is the unsteered value plus six. Cells named as “held” are included with a zero displacement, so leakage into them increases the same request loss. The spectrogram gallery shows where those requested and held regions lie in time and frequency.
| Task / source | Measurement map and target | Evaluation |
| Localized waveform EQ (A0) | equation 11, dB cells. Target: Primary request: +6 dB in one cell and −8 dB in another; remaining 22 cells request 0 dB | Requested-cell dB error; LSD and SI-SDR to a reference edit |
| Disjoint-cell EQ ladder (A0) | Per-cell dB on disjoint log-mel cells. Target: Alternating dB pattern at the declared scales; included hold cells request 0 change | Decoded-signal cell error and feasibility gate |
| Bandwidth extension (A1) | , a 601-tap Kaiser low-pass measurement. Target: | High-band LSD above the cutoff |
| Temporal inpainting (A1) | Observed samples of a centered binary temporal mask. Target: Match the observed samples; the missing interval is absent from | Gap-region SNR and splice-boundary discontinuity |
| Known-filter EQ restoration (A1) | 24 log-spaced band energies . Target: , restoring the clean band balance | LTAS distance and waveform metrics |
| Missing-segment boundary match (A0) | Eight boundary-band levels in dB. Target: Match the generated segment’s beginning/end to the adjacent clips; interior coordinates request 0 | Abrupt energy jump across the splice |
| Crossover between two clips (A3) | Four band levels in each of four temporal quarters, plus four broadband quarter levels (). Target: Ramp each band and broadband level from clip A’s ending profile to clip B’s opening profile | Bandwise ramp error and splice diagnostics |
| Context-preserving interior edit (A4) | Four interior band levels; four left/right context levels; two seam-continuity coordinates (). Target: Interior low band −6 dB and high band +6 dB; both contexts and both seams request no change | Interior error and leakage into contexts/seams |
| Crescendo envelope (A2) | , five equal time-bin mean-dB levels and global mean dB. Target: dB; middle and total energy request zero change | Realized temporal envelope; no listening preference |
Table 9. Audio objective and cost functions. All dB requests are literal changes in the stated time–frequency cells. No listening-study or perceptual-preference endpoint is claimed.
Executed manipulation and locomotion
The steering loss is applied to a generated reference trajectory, not directly to the MuJoCo rollout. The same objective-blind controller then executes every method’s reference. This separation is why Table 10 distinguishes requested reference coordinates from executed target error, contact, height drift, or world-frame progress.
| Task / source | Measurement map and target | Evaluation |
| Panda Cartesian reach (R0) | Position ; orientation error , ; terminal and . Target: Move position; preserve orientation and terminal linear/angular velocity | Executed Cartesian target error |
| Panda checkpoint/via-point reach (R1) | , with , or . Target: Independent checkpoint displacements or geometry-defined via-points | Executed closest-approach via error and final error |
| Reach with orientation hold (R0/R1) | . Target: Move position; request zero orientation change | Executed target error and tool rotation |
| Lateral move at constant height (R0/R1) | Terminal and four checkpoint heights . Target: Move laterally; hold start height at four times | Executed target error and maximum height drift |
| Unmodelled obstacle reach (R0) | with soft-minimum whole-arm clearance. Target: Request goal and clearance; preserve orientation | Contact-free execution and MuJoCo geom distance |
| Spatial locomotion target (R0/R1) | Midpoint/final , terminal , torso height, and pitch, each with its frozen physical scale. Target: Request intermediate/final position and speed; preserve height and pitch | Executed world-position error |
| Locomotion trajectory tracking (R0/R1) | Four checkpoints, terminal , torso height, and pitch, each frozen-scaled. Target: Request a smooth world-position profile; preserve posture | Executed trajectory RMSE |
| Progress and speed control (R0) | Progress , terminal speed , height , and pitch . Target: Request signed progress/speed change; preserve height and pitch | Executed progress or signed speed change |
Table 10. Table 10: Manipulation and locomotion objectives. The objective is evaluated on the generated reference; the endpoint is measured after an independent controller executes that reference in MuJoCo.
Control and planning
The catalogue first defines character motion generation requests, then the canonical control and trajectory-planning objectives. Both combine requested motion with explicitly represented preservation or terminal-state requirements.
Character motion generation
Character steering differentiates joint coordinates recovered by forward kinematics through the remaining motion sampler. Obstacle avoidance specifies a mid-clip pelvis shift around a 20 cm sphere, preserving final pelvis location and seven arm-bone lengths (). Reaching specifies five wrist waypoints, preserved ankle and hip positions at two additional frames, zero wrist arrival velocity, and bone lengths at waypoint frames (). Requested displacements use the unsteered subject frame and natural excursion scales. The reported conjunctive success criterion requires positional target accuracy, preservation of joint positions, and jerk below a population-calibrated threshold. Velocity, bone drift, and geometric collision are separate diagnostics, even when the corresponding quantities enter the steering objective. The full campaign definitions and reported thresholds are in Appendix E.
Canonical control and trajectory planning
Table 11 lists the canonical control and planning objectives. Here is evaluated on a generated state trajectory. Endpoint terms encode the requested final state; checkpoint terms encode the route; and velocity, posture, effort, and clearance entries specify either absolute targets or changes relative to the anchor. Collision and execution diagnostics are evaluated according to each task protocol, so satisfying a sparse endpoint request alone is not reported as obstacle-avoidance success.
| Task / source | Measurement map and target | Evaluation |
| Pendulum stabilization (C0) | . Target: Target ; control effort is preserved | Real-dynamics replay: angle, speed, return, and effort |
| Pendulum reference tracking (C0) | Wrapped angle errors at fixed intermediate checkpoints and , plus . Target: Reach the declared reference angles and finish stationary | Real-dynamics replay and tracking error |
| CartPole target position (C0) | divided by frozen training-distribution scales. Target: Target ; upright orientation and terminal velocities are explicit zeros | Open-loop true-dynamics survival and terminal errors |
| Double-integrator waypoints (P0) | , . Target: Hit all requested positions; finish at | Executed goal/waypoint error and task success |
| Maze2D goal and via-points (P0) | with A* reference waypoints. Target: Reach requested free-space waypoints and goal; finish stationary | Executed goal distance, wall collision, path length |
| Dense obstacle navigation (P0) | , . Target: Request the new goal and zero velocity/clearance violations; preserve smoothness | Executed collision, clearance, goal error |
| Planar-arm pose planning (P0) | ; dense body-clearance entries are added for the obstacle task. Target: Move the end effector; preserve orientation, terminal velocity, and clearance | Executed pose error and body collision |
Table 11. Control and planning objective and cost functions. State coordinates are normalized by the frozen physical scales named in the source protocols before entering the Euclidean request cost.
References
- [1]Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 843–852, 2023. URL https://openaccess.thecvf.com/content/CVPR2023W/GCV/html/Bansal_Universal_Guidance_for_Diffusion_Models_CVPRW_2023_paper.html.openaccess.thecvf.com/content/CVPR2023W/GCV/html/Bansal_Universal_Guidance_for_Diffusion_Models_CVPRW_2023_paper.html
- [2]Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman. D-Flow: Differentiating through flows for controlled generation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 3462–3483, 2024. URL https://proceedings.mlr.press/v235/ben-hamu24a.html.proceedings.mlr.press/v235/ben-hamu24a.html
- [3]Zander Blasingame and Chen Liu. Greed is good: A unifying perspective on guided generation. In Advances in Neural Information Processing Systems, volume 38, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/a565741b6a2b152323452215bd1f4185-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2025/hash/a565741b6a2b152323452215bd1f4185-Abstract-Conference.html
- [4]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin C. M. Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems, 2023. doi: 10.15607/RSS.2023.XIX.026. URL https://roboticsproceedings.org/rss19/p026.html.DOI
- [5]Hyungjin Chung, Jeongsol Kim, Michael Thompson McCann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=OnD9zGAGT0k.openreview.net/forum?id=OnD9zGAGT0k
- [6]Dhariwal, P., & Nichol, A. (2021). Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34. https://proceedings.neurips.cc/paper/2021/hash/49ad23d1ec9fa4bd8d77d02681df5cfa-Abstract.htmlproceedings.neurips.cc/paper/2021/hash/49ad23d1ec9fa4bd8d77d02681df5cfa-Abstract.html
- [7]Maximilian Du and Shuran Song. DynaGuide: Steering diffusion policies with active dynamic guidance. In Advances in Neural Information Processing Systems, volume 38, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/3ec6c6fc9065aa57785eb05dffe7c3db-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2025/hash/3ec6c6fc9065aa57785eb05dffe7c3db-Abstract-Conference.html
- [8]Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the maestro dataset. arXiv preprint arXiv:1810.12247, 2018.arxiv.org/abs/1810.12247
- [9]He, Y., Murata, N., Lai, C.-H., Takida, Y., Uesaka, T., Kim, D., Liao, W.-H., Mitsufuji, Y., Kolter, Z., Salakhutdinov, R., & al., . et . (2024). Manifold preserving guided diffusion. International Conference on Learning Representations, 2024, 44819–44850.
- [10]Liav Hen, Tom Tirer, Raja Giryes, and Shady Abu-Hussein. Jacobian-aware posterior sampling for inverse problems. Transactions on Machine Learning Research, 2026. URL https://arxiv.org/abs/2511.18471.
- [11]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.arxiv.org/abs/2006.11239
- [12]Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9902–9915, 2022. URL https://proceedings.mlr.press/v162/janner22a.html.proceedings.mlr.press/v162/janner22a.html
- [13]Wonsuhk Jung, Utkarsh Aashu Mishra, Nadun Ranawaka Arachchige, Yongxin Chen, Danfei Xu, and Shreyas Kousik. Joint model-based model-free diffusion for planning with constraints. In Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pp. 4328–4350, 2025. URL https://proceedings.mlr.press/v305/jung25d.html.proceedings.mlr.press/v305/jung25d.html
- [14]Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakorn, and Siyu Tang. Optimizing diffusion noise can serve as universal motion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1334–1345, 2024. URL https://openaccess.thecvf.com/content/CVPR2024/html/Karunratanakul_Optimizing_Diffusion_Noise_Can_Serve_As_Universal_Motion_Priors_CVPR_2024_paper.html.openaccess.thecvf.com/content/CVPR2024/html/Karunratanakul_Optimizing_Diffusion_Noise_Can_Serve_As_Universal_Motion_Priors_CVPR_2024_paper.html
- [15]Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems, volume 35, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/95504595b6169131b6ed6cd72eb05616-Abstract-Conference.html.proceedings.neurips.cc/paper_files/paper/2022/hash/95504595b6169131b6ed6cd72eb05616-Abstract-Conference.html
- [16]Kenneth Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of Applied Mathematics, 2(2):164–168, 1944. doi: 10.1090/qam/10666.DOI
- [17]Xingchao Liu, Lemeng Wu, Shujian Zhang, Chengyue Gong, Wei Ping, and Qiang Liu. FlowGrad: Controlling the output of generative ODEs with gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24335–24344, 2023. URL https://openaccess.thecvf.com/content/CVPR2023/html/Liu_FlowGrad_Controlling_the_Output_of_Generative_ODEs_With_Gradients_CVPR_2023_paper.html.DOI
- [18]Mandlekar, A., Xu, D., Wong, J., Soroush, N., Wang, C., Kulkarni, R., Fei-Fei, L., Savares, S., Zhu, Y., & Martín-Martín, R. (2021). What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.03298.arXiv
- [19]Donald W. Marquardt. An algorithm for least-squares estimation of nonlinear parameters. Journal of the Society for Industrial and Applied Mathematics, 11(2):431–441, 1963. doi: 10.1137/0111030.DOI
- [20]Novack, Z., Macauley, J., Berg-Kirkpatrick, T., & Bryan, N. J. (2024). DITTO: Diffusion inference-time t-optimization for music generation. Proceedings of the 41st International Conference on Machine Learning, 235, 38426–38447. https://proceedings.mlr.press/v235/novack24a.htmlproceedings.mlr.press/v235/novack24a.html
- [21]Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry. In Advances in Neural Information Processing Systems, volume 36, pp. 24129–24142, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/4bfcebedf7a2967c410b64670f27f904-Abstract.html.proceedings.neurips.cc/paper_files/paper/2023/hash/4bfcebedf7a2967c410b64670f27f904-Abstract.html
- [22]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 10674–10685. ieee, 2022.arxiv.org/abs/2112.10752
- [23]Seunghyeok Shin, Minwoo Kim, Dabin Kim, and Hongki Lim. Geometry-correct diffusion posterior sampling with denoiser-pullback curvature guidance and manifold-aligned damping. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2605.27990.
- [24]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.arxiv.org/abs/2010.02502
- [25]Song, J., Vahdat, A., Mardani, M., & Kautz, J. (n.d.). Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Representations. Retrieved https://research.nvidia.com/labs/genair/publication/song2023pseudoinverse/research.nvidia.com/labs/genair/publication/song2023pseudoinverse
- [26]Song, J., Zhang, Q., Yin, H., Mardani, M., Liu, M.-Y., Kautz, J., Chen, Y., & Vahdat, A. (n.d.). Loss-guided diffusion models for plug-and-play controllable generation. In Proceedings of the 40th International Conference on Machine Learning (Vol. 202, pp. 32483–32498). Retrieved https://proceedings.mlr.press/v202/song23k.htmlproceedings.mlr.press/v202/song23k.html
- [27]Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=SJ1kSyO2jwu.openreview.net/forum?id=SJ1kSyO2jwu
- [28]Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033, 2012. doi: 10.1109/IROS.2012.6386109.DOI
- [29]Andrew Wagenmaker, Yunchu Zhang, Mitsuhiko Nakamoto, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. In Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pp. 258–282, 2025. URL https://proceedings.mlr.press/v305/wagenmaker25a.html.proceedings.mlr.press/v305/wagenmaker25a.html
- [30]Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimization improves classifier guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7280–7290, 2023. URL https://openaccess.thecvf.com/content/ICCV2023/html/Wallace_End-to-End_Diffusion_Latent_Optimization_Improves_Classifier_Guidance_ICCV_2023_paper.html.openaccess.thecvf.com/content/ICCV2023/html/Wallace_End-to-End_Diffusion_Latent_Optimization_Improves_Classifier_Guidance_ICCV_2023_paper.html
- [31]Luran Wang, Chaoran Cheng, Yizhen Liao, Yanru Qu, and Ge Liu. Training free guided flow-matching with optimal control. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/19a94fdf9e1c5b387830e4c4fef6972a-Abstract-Conference.html.proceedings.iclr.cc/paper_files/paper/2025/hash/19a94fdf9e1c5b387830e4c4fef6972a-Abstract-Conference.html
- [32]Yinhua Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In International Conference on Learning Representations, 2023. URL https://mlanthology.org/iclr/2023/wang2023iclr-zeroshot/.mlanthology.org/iclr/2023/wang2023iclr-zeroshot
- [33]Xia, W., Yang, Y., Xue, J.-H., & Wu, B. (2021). Tedigan: Text-guided diverse face image generation and manipulation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2256–2265).
- [34]Xiao, W., Wang, J. (Tsun .-H., Gan, C., Hasani, R., Lechner, M., & Rus, D. (2025). SafeDiffuser: Safe planning with diffusion probabilistic models. International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2025/hash/f95606d8e870020085990d9650b4f2a1-Abstract-Conference.htmlproceedings.iclr.cc/paper_files/paper/2025/hash/f95606d8e870020085990d9650b4f2a1-Abstract-Conference.html
- [35]Lingxiao Yang, Shutong Ding, Yifan Cai, Jingyi Yu, Jingya Wang, and Ye Shi. Guidance with spherical gaussian constraint for conditional diffusion. arXiv preprint arXiv:2402.03201, 2024.arxiv.org/abs/2402.03201
- [36]Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free energy-guided conditional diffusion model. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 23117–23127. IEEE, 2023.arxiv.org/abs/2303.09833














