betaFeedback

WorksComputer Science

Learning Noise Optimization for Reward Alignment in Few-Step Generative Models

Abstract

Reward alignment for diffusion- and flow-based generative models is often formulated over stochastic trajectories, making it less suitable for few-step deterministic generators. Initial noise optimization provides a trajectory-free alternative, but existing gradient-free methods require costly per-sample search at inference. We introduce LeNO (Learning Noise Optimization), which amortizes reward-guided initial noise optimization into a learned residual transport while keeping the pretrained generator frozen. LeNO learns an anchor-conditioned distribution over noise corrections using Local GRPO and scalar rewards, eliminating iterative test-time optimization. For Gaussian initial noise with norm-preserving projection, we derive a tractable upper bound on the KL divergence between the corrected noise distribution and the original prior, providing a principled regularization objective. We evaluate LeNO on few-step text-to-image and protein structure generation, including expensive black-box designability rewards. Across both domains, LeNO improves target rewards with a single noise correction at inference while largely preserving generation quality and diversity.

Examples of generated images before and after applying learned noise corrections

Figure 1. LeNO learns reward-improving corrections directly in initial noise space. Each pair shows generations before (left) and after (right) applying the learned noise correction, with the pretrained generator kept frozen. The corrected noises produce images that better capture the desired visual attributes across diverse prompts.

Introduction

Reward-based post-training has emerged as a powerful paradigm for adapting pretrained generative models toward objectives such as human preference, visual quality, and task-specific utility. In diffusion and flow models, many recent approaches formulate generation as a sequential decision process and optimize the generative trajectory using reinforcement learning or stochastic control [2, 18, 16]. Recent generative models, however, increasingly rely on few-step or even one-step deterministic sampling [25, 34, 23, 8]. While highly efficient, these models provide only few step deterministic generation trajectory, making conventional trajectory-level policy optimization less natural.

A promising alternative is to move reward optimization from the generative trajectory to the initial noise space, as the initial noise strongly influences the generated output. This has motivated recent inference time methods that search for reward-favorable noises while keeping the pretrained generator frozen [6, 35, 14]. While these methods enable deterministic generators to be aligned with a given reward, including non-differentiable functions, they require an iterative noise-update process involving repeated generator executions and reward evaluations for each test sample. This becomes particularly costly when either generation or reward evaluation requires an expensive external pipeline.

This motivates amortizing reward-guided noise optimization into a learned mapping. Existing approaches, however, typically require either pre-collected offline data or differentiable reward functions. For example, GoldenNoise [37] learns from a pre-constructed dataset of favorable noise–sample pairs, while HyperNoise [7] learns to modify the initial noise through differentiable reward feedback. In contrast, directly learning sample-dependent noise corrections from online scalar rewards, without offline data or reward gradients, remains relatively underexplored.

To address these limitations, we introduce LeNO (Learning Noise Optimization), a framework that amortizes per-sample initial noise optimization for deterministic generators under differentiable or non-differentiable rewards. Rather than predicting an optimized noise sample from scratch, LeNO parameterizes the mapping in residual form and learns an anchor-dependent correction to the initial noise. To optimize this residual mapping using only scalar rewards, we formulate it as a stochastic transport that samples Gaussian residual actions around the predicted correction mean. These actions are trained with Local GRPO, where group-relative reward comparisons among corrections sampled from the same anchor provide the learning signal. We further provide a distributional interpretation of this residual parameterization. For Gaussian initial noise with anchor-norm-preserving projection, we show that a scaled squared norm of the predicted residual mean upper-bounds the KL divergence between the corrected noise distribution and the prior. This provides a tractable lower bound on the KL-regularized reward objective and directly motivates the residual regularization in Local GRPO. At inference, the learned transport generates a corrected initial noise with a single residual sample, eliminating per-sample optimization and additional reward evaluations.

We evaluate LeNO on few-step text-to-image generation and protein structure generation. In text-to-image generation, we consider direct optimization of multiple preference rewards. For proteins, we optimize a black-box designability objective whose evaluation requires sequence design and structure prediction, a setting where gradient-based amortization is inapplicable and per-sample search is costly. Across these settings, LeNO improves target rewards with only a lightweight residual correction at inference time, while retaining favorable held-out compositional performance and structural diversity. These results demonstrate that learned initial noise correction provides an efficient and broadly applicable alternative to repeated per-sample reward optimization.

Background

Reward-Tilted Noise Distribution. For a deterministic generator x=Gϕ(z,c)x = G_{\phi}(z, c) with initial noise z∼p0(z)z \sim p_0(z), the generated sample is fully determined by zz once the condition cc is fixed. Let r(⋅)r(\cdot) denote a scalar, potentially black-box reward function defined on the generated sample. Reward-based alignment can therefore be formulated directly in the initial noise space by tilting the noise prior toward high-reward regions:

...(1)... \tag*{(1)}
p⋆(z∣c)∝p0(z)exp⁡(r(Gϕ(z,c))λ),(2)p^\star(z \mid c) \propto p_0(z)\exp\left(\frac{r(G_\phi(z,c))}{\lambda}\right), \tag*{(2)}

where λ>0\lambda> 0 controls the trade-off between reward maximization and preservation of the original noise prior. This formulation provides a direct basis for initial noise optimization, where the pretrained generator is kept fixed and reward alignment is performed by searching for higher-reward noise samples.

Zeroth-Order Initial Noise Optimization. Reward optimization can be performed directly in the initial noise space without differentiating through either the generator or the reward function. Let

R(z;c):=r(Gϕ(z,c)),Rσ(z;c):=Eϵ∼N(0,I)[R(z+σϵ;c)].(3)\mathcal{R}(z;c) := r(G_\phi(z,c)), \qquad\mathcal{R}_\sigma(z;c) := \mathbb{E}_{\epsilon\sim\mathcal{N}(0,I)}[\mathcal{R}(z+\sigma\epsilon;c)]. \tag*{(3)}

The gradient g(z,c)g(z,c) of the Gaussian-smoothed reward can be estimated from centered reward-weighted perturbations as

g(z,c):=∇zRσ(z;c)≈1(K−1)σ∑i=1K(Ri−Rˉz)ϵi,(4)g(z,c) := \nabla_z\mathcal{R}_\sigma(z;c) \approx\frac{1}{(K-1)\sigma}\sum_{i=1}^{K}(R_i-\bar{R}_z)\epsilon_i, \tag*{(4)}

where Ri=R(z+σϵi;c)R_i=\mathcal{R}(z+\sigma\epsilon_i;c) and Rˉz=1K∑i=1KRi\bar{R}_z=\frac{1}{K}\sum_{i=1}^{K}R_i. This form underlies Natural Evolution Strategy style Wierstra et al., [28] gradient estimation and related gradient-free initial noise optimization methods, including approaches that derive reward-weighted perturbation updates from stochastic control perspectives [14]. Such methods avoid reward backpropagation, but require estimating a new local direction for each noise state through repeated generator and reward evaluations. LeNO instead seeks to amortize these state-dependent noise corrections into a learned residual transport, avoiding repeated optimization at inference time.

Policy Optimization for Deterministic Generative Models. Recent approaches have explored how to enable policy optimization for deterministic generative models by recovering a suitable source of stochasticity. Flow-GRPO Liu et al., [18], for example, converts the deterministic probability-flow ODE into an equivalent stochastic differential equation, thereby introducing stochastic transitions on which policy-gradient optimization can be performed. An alternative perspective preserves the deterministic sampling dynamics and instead constructs stochasticity through local trajectory neighborhoods.

Neighbor GRPO He et al., [10] follows this local-neighborhood perspective. Given a shared reference noise ϵ⋆\epsilon^\star, it samples neighboring initial conditions

ϵ(i)=1−σ2ϵ⋆+σδ(i),δ(i)∼N(0,I),(5)\epsilon^{(i)}=\sqrt{1-\sigma^2}\epsilon^\star+\sigma\delta^{(i)}, \qquad\delta^{(i)}\sim\mathcal{N}(0,I), \tag*{(5)}

and propagates them through the same deterministic ODE to construct a local trajectory neighborhood. Despite the similar shared-anchor perturbation geometry, Neighbor GRPO optimizes the generative model through a contrastive, distance-based surrogate policy defined over neighboring ODE trajectories. In contrast, LeNO keeps the generator frozen and learns a separate stochastic residual transport in the initial noise space, using local perturbations to obtain reward-based training signals. The two approaches are therefore complementary, as LeNO leaves the pretrained model unchanged and applies to any generator that exposes an initial noise input.

Methods

Noise-Space Residual Transport

We formulate amortized initial noise optimization as a stochastic residual transport for a frozen pretrained generator GϕG_\phi. Given an initial noise z∼p0z\sim p_0 and condition cc, we learn an anchor-dependent residual mean μθ(z,c)\mu_\theta(z,c) to improve the generated output using scalar reward feedback. The transport samples a residual action and applies it to the anchor:

a=ημθ(z,c)+σϵ,ϵ∼N(0,I),(6)a=\eta\mu_\theta(z,c)+\sigma\epsilon,\qquad\epsilon\sim\mathcal{N}(0,I), \tag*{(6)}
z′=Fθ(z,ϵ,c):=T(z,a),(7)z'=F_\theta(z,\epsilon,c):=\mathcal{T}(z,a), \tag*{(7)}

where η>0\eta>0 controls the correction scale and σ>0\sigma>0 is a fixed sampling scale. For Gaussian initial noise, we use the norm-preserving transition

T(z,a)=NormProj⁡(z+a;∥z∥2),NormProj⁡(x;r):=rx∥x∥2.(8)\mathcal{T}(z,a)=\operatorname{NormProj}(z+a;\lVert z\rVert_2),\qquad\operatorname{NormProj}(x;r):=r\frac{x}{\lVert x\rVert_2}. \tag*{(8)}

This updates the direction of the initial noise while preserving its norm. The stochastic transport induces the corrected noise distribution

z′=Fθ(z,ϵ,c)∼qθ(⋅∣c),(9)z' = F_\theta(z,\epsilon,c) \sim q_\theta(\cdot\mid c), \tag*{(9)}

where z∼p0z \sim p_0 and ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I) are independent.

At inference, we sample one corrected noise and generate Gϕ(z′,c)G_\phi(z',c). During training, we draw multiple actions per anchor and compare their rewards to learn the residual mean. The same conditional Gaussian structure provides both a tractable distributional KL bound and an action likelihood for reward-based learning. We next derive the bound, followed by the Local GRPO objective.

Distributional Target and KL Regularization

We aim to align the corrected noise distribution qθ(⋅∣c)q_\theta(\cdot\mid c) with the reward-tilted target

p∗(z∣c)=p0(z)exp⁡(R(z;c)/λ)Z(c),λ>0,(10)p^*(z \mid c) = \frac{p_0(z)\exp(\mathcal{R}(z;c)/\lambda)}{Z(c)}, \qquad\lambda> 0, \tag*{(10)}

where p0=N(0,I)p_0 = \mathcal{N}(0,I), R(z;c)=r(Gϕ(z,c))\mathcal{R}(z;c) = r(G_\phi(z,c)) and Z(c):=Ez∼p0[exp⁡(R(z;c)/λ)]<∞Z(c) := \mathbb{E}_{z\sim p_0}[\exp(\mathcal{R}(z;c)/\lambda)] < \infty. This target favors high-reward noises while retaining the prior as a reference.

Whenever the relevant expectations and divergences are finite,

DKL(qθ(⋅∣c)∥p∗(⋅∣c))=DKL(qθ(⋅∣c)∥p0)−1λEz∼qθ(⋅∣c)[R(z;c)]+log⁡Z(c).(11)D_{\mathrm{KL}}(q_\theta(\cdot\mid c)\Vert p^*(\cdot\mid c)) = D_{\mathrm{KL}}(q_\theta(\cdot\mid c)\Vert p_0) - \frac{1}{\lambda}\mathbb{E}_{z\sim q_\theta(\cdot\mid c)}[\mathcal{R}(z;c)] + \log Z(c). \tag*{(11)}

Since Z(c)Z(c) is independent of θ\theta, minimizing this KL divergence is equivalent to maximizing

L(qθ;c):=Ez∼qθ(⋅∣c)[R(z;c)]−λDKL(qθ(⋅∣c)∥p0).(12)\mathcal{L}(q_\theta;c) := \mathbb{E}_{z\sim q_\theta(\cdot\mid c)}[\mathcal{R}(z;c)] - \lambda D_{\mathrm{KL}}(q_\theta(\cdot\mid c)\Vert p_0). \tag*{(12)}

The reward term can be estimated from generated samples, but the prior-relative KL remains intractable because qθq_\theta is implicitly defined by the transport. We therefore bound this KL term using the conditional Gaussian action distributions.

Proposition 1. Let p0=N(0,I)p_0 = \mathcal{N}(0,I) and let qθq_\theta be a corrected noise distribution induced by the norm-preserving transition TT in Eq. (8). For σ>0\sigma> 0 and Ez∼p0∥μθ(z,c)∥22<∞\mathbb{E}_{z\sim p_0}\lVert\mu_\theta(z,c)\rVert_2^2 < \infty, the following upper bound holds:

DKL(qθ(⋅∣c)∥p0)≤Ez∼p0DKL(N(ημθ(z,c),σ2I)∥N(0,σ2I))=η22σ2Ez∼p0∥μθ(z,c)∥22.(13)\begin{aligned} D_{\mathrm{KL}}(q_\theta(\cdot\mid c)\Vert p_0) &\le\mathbb{E}_{z\sim p_0}D_{\mathrm{KL}}\left(\mathcal{N}(\eta\mu_\theta(z,c),\sigma^2 I)\Vert\mathcal{N}(0,\sigma^2 I)\right) \\ &= \frac{\eta^2}{2\sigma^2}\mathbb{E}_{z\sim p_0}\lVert\mu_\theta(z,c)\rVert_2^2. \tag*{(13)} \end{aligned}

The inequality follows from prior preservation under zero-mean residual actions, together with the data-processing inequality applied to the joint distributions of anchors and residual actions. The equality follows from the closed-form KL divergence between Gaussian distributions with identical covariance. A detailed proof is provided in Appendix § 8.

Under the conditions of Proposition 1, we obtain a tractable lower bound on the distributional objective:

Laux(θ;c):=Ez′∼qθ(⋅∣c)[R(z′;c)]−λη22σ2Ez∼p0∥μθ(z,c)∥22≤L(qθ;c).(14)\mathcal{L}_{\mathrm{aux}}(\theta;c) := \mathbb{E}_{z'\sim q_\theta(\cdot\mid c)}[\mathcal{R}(z';c)] - \frac{\lambda\eta^2}{2\sigma^2}\mathbb{E}_{z\sim p_0}\lVert\mu_\theta(z,c)\rVert_2^2 \le\mathcal{L}(q_\theta;c). \tag*{(14)}

The reward term can be estimated from sampled transport outputs, while the regularizer depends only on the predicted residual mean. Together with Eq. (11), this implies that maximizing Laux\mathcal{L}_{\mathrm{aux}} minimizes an upper bound on the KL divergence from qθq_\theta to the reward-tilted target p∗p^*. Using the same stochastic transport and noise scale during training and inference ensures that the bound applies to the corrected noise distribution used for generation.

Local GRPO Objective

Training Objective. We next optimize the auxiliary reward objective Laux(θ;c)\mathcal{L}_{\mathrm{aux}}(\theta;c) through a tractable Local GRPO surrogate. Rather than maximizing the expected reward directly, Local GRPO uses group-relative comparisons among samples from the stochastic residual transport to update the residual mean from scalar rewards, while preserving the regularization implied by the distributional KL bound.

At the start of each training iteration, we snapshot the current transport parameters as θold\theta_{\mathrm{old}} and hold them fixed throughout the subsequent GRPO updates on the sampled particles. For each anchor zz and condition cc, we draw KK residual actions from the conditional Gaussian distribution in Eq. (6). For i=1,…,Ki=1,\ldots,K,

a(i)=ημθold(z,c)+σϵi,(15)a^{(i)}=\eta\mu_{\theta_{\mathrm{old}}}(z,c)+\sigma\epsilon_i, \tag*{(15)}

where ϵi∼iidN(0,I)\epsilon_i \mathrel{\overset{\mathrm{iid}}{\sim}} \mathcal{N}(0,I). Each action produces a corrected noise z(i)=T(z,a(i))z^{(i)}=\mathcal{T}(z,a^{(i)}) with reward Ri=R(z(i);c)R_i=\mathcal{R}(z^{(i)};c). These particles are samples of the same stochastic transport used at inference, sharing the anchor and predicted residual mean while differing in their Gaussian perturbations. We compare their rewards within each group to construct the group-relative advantages:

Ai=Ri−RˉsR+ϵ,Rˉ=1K∑j=1KRj,(16)A_i=\frac{R_i-\bar{R}}{s_R+\epsilon},\qquad\bar{R}=\frac{1}{K}\sum_{j=1}^{K}R_j, \tag*{(16)}

where sRs_R is the within-group reward standard deviation and ϵ>0\epsilon>0 ensures numerical stability.

Algorithm 1 Training LeNO with Local GRPO
Require: Frozen generator GϕG_\phi, reward rr, residual transport FθF_\theta, group size KK, residual scale η\eta, exploration scale σ\sigma, regularization coefficient β\beta
1: for each training iteration do
2: θold←θ\theta_{\mathrm{old}}\leftarrow\theta
3: Sample condition cc and anchor z∼p0(z)z\sim p_0(z)
4: for i=1,…,Ki=1,\ldots,K do
5: Sample ϵi∼N(0,I)\epsilon_i\sim\mathcal{N}(0,I)
6: a(i)←ημθold(z,c)+σϵia^{(i)}\leftarrow\eta\mu_{\theta_{\mathrm{old}}}(z,c)+\sigma\epsilon_i
7: z(i)←T(z,a(i))z^{(i)}\leftarrow\mathcal{T}(z,a^{(i)})
8: Ri←r(Gϕ(z(i)),c)R_i\leftarrow r(G_\phi(z^{(i)}),c)
9: end for
10: Compute group-relative advantages AiA_i
11: Compute exploration likelihood ratios ρ(i)(θ)\rho^{(i)}(\theta)
12: Update θ\theta using Eq. (18)
13: end for

Table 1.

Let πθ(a∣z,c)\pi_\theta(a\mid z,c) denote the Gaussian action density specified by Eq. (6). For the sampled actions, we define the likelihood ratio

ρ(i)(θ)≔πθ(a(i)∣z,c)πθold(a(i)∣z,c).(17)\rho^{(i)}(\theta)\coloneqq\frac{\pi_\theta(a^{(i)}\mid z,c)}{\pi_{\theta_{\mathrm{old}}}(a^{(i)}\mid z,c)}. \tag*{(17)}

This ratio is evaluated on the residual actions before projection and does not require the density of the corrected noise marginal qθq_\theta. Holding the sampled actions and advantages fixed, our penalized local GRPO maximizes the following:

J(θ)=E[1K∑i=1Kmin⁡(ρ(i)(θ)Ai,clip⁡(ρ(i)(θ),1−ϵclip,1+ϵclip)Ai)−βη22σ2∥μθ(z,c)∥22],(18)\mathcal{J}(\theta)=\mathbb{E}\left[\frac{1}{K}\sum_{i=1}^{K}\min\left(\rho^{(i)}(\theta)A_i,\operatorname{clip}\left(\rho^{(i)}(\theta),1-\epsilon_{\mathrm{clip}},1+\epsilon_{\mathrm{clip}}\right)A_i\right)-\beta\frac{\eta^2}{2\sigma^2}\left\lVert\mu_\theta(z,c)\right\rVert_2^2\right], \tag*{(18)}

where the expectation is over training conditions, anchors z∼p0z\sim p_0, and particles sampled under θold\theta_{\mathrm{old}}. Here, ϵclip>0\epsilon_{\mathrm{clip}}>0 is the PPO clipping parameter and β>0\beta>0 controls regularization strength. In Eq. (18), the first term promotes residual actions with higher group-relative rewards, while the second regularizes the policy toward the zero-mean Gaussian reference through the conditional KL divergence, thereby linking to the distributional KL term in Eq. (14). In practice, we combine group-normalized rewards with PPO-style clipping to obtain a stable Local GRPO surrogate, where β\beta controls the strength of KL regularization relative to the normalized reward scale.

The local sampling procedure also admits a zeroth-order interpretation. Before norm projection, reward-weighted Gaussian perturbations provide an estimator of the gradient of the Gaussian-smoothed reward evaluated at the mean-shifted anchor, closely related to zeroth-order noise optimization methods such as ZeNO (Kim et al. [26]). Unlike such methods, which optimize the initial noise independently for each test-time sample, LeNO uses these local reward signals to learn an anchor-conditioned transport and thereby amortizes noise optimization across samples. See Appendix § 7 for more details of the derivation.

Why State-Dependent Noise Corrections? LeNO conditions the residual mean μθ(z,c)\mu_\theta(z,c) on the anchor zz itself, rather than solely on the condition cc. This design is motivated by the zeroth-order interpretation above: because local sampling estimates the reward-improvement direction in a neighborhood of a specific anchor, the resulting direction is generally anchor-dependent. In contrast, a correction conditioned only on cc cannot adapt to such local variation and instead captures only an average direction across anchors. Appendix § 9 formalizes this intuition using a simplified first-order surrogate of Laux\mathcal{L}_{\mathrm{aux}}. Under this surrogate, the optimal anchor-dependent correction follows the local reward-improvement direction at each anchor, whereas the optimal anchor-independent correction follows their population average. The resulting objective gap is proportional to the total variance of the local directions across anchors and vanishes only when these directions are identical for all anchors (Lemma 1). We empirically confirm the benefit of anchor-dependent conditioning in Figure 4.

Ablation for state-dependent training

Figure 4. Ablation for state-dependent training.

Experiments

Toy Experiment

We consider a four-component Gaussian mixture model where two different conditions, c1c_1 and c2c_2, select disjoint pairs of target modes with distinct reward values. The deterministic generator GϕG_\phi is defined as the probability-flow ODE of a linear interpolant between N(0,I)\mathcal{N}(0,I) and the target mixture. Figure 2(a) illustrates the initial noise samples and the corresponding reward landscape, where darker regions indicate higher rewards. As a baseline, Figure 2(b) directly maps the initial particles through the frozen flow ODE without any reward-based correction. For per-sample optimization, Figure 2(c) applies per-sample 30 step optimization, following ZeNO [14], with 32 perturbation particles per step, requiring 960 generator evaluations per sample. In contrast, LeNO in Figure 2(d) uses one-step residual correction with a lightweight 35K-parameter MLP trained for 400 iterations.

Toy experiment showing initial states, flow-ODE samples, per-sample optimization, amortized correction, and training progress

Figure 2. Toy Experiment. (a) Initial noise samples and the reward landscape of a four-component Gaussian mixture. (b) Samples obtained by directly passing the initial noise through the frozen probability-flow ODE. (c) Per-sample initial noise optimization with 30 step optimization. (d) Our amortized noise space correction with the residual transport. The learned correction successfully redirects samples toward the condition-specific high-reward modes with substantially lower inference time computation. (e) Corrected noise distribution over training.

Despite replacing iterative per-sample optimization with a learned corrector, our method consistently redirects samples toward the high-reward modes specified by each condition. While per-sample optimization improves rewards for each condition to 0.90 and 0.83, our amortized correction attains 0.84 and 0.86, showing that condition-dependent reward-improving corrections can be effectively amortized into a compact noise space transport with substantially reduced inference time computation. Figure 2(e) further visualizes the evolution of the learned correction during training. As training progresses, the transport gradually shifts the initial noise samples toward condition-dependent high-reward regions, increasing the average reward.

Text-to-Image Generation

Experimental Setup. We evaluate our method on text-to-image generation under two complementary reward-alignment settings: single reward optimization and compositional multi-reward optimization. Across all settings, the pretrained image generator GϕG_\phi remains frozen, and only the residual noise transport is optimized. We parameterize the residual correction μθ\mu_\theta using the same backbone architecture as the corresponding generator, with LoRA adapters injected into the model while keeping the underlying pretrained weights frozen. We use SDXL-Turbo [23] for the single-reward and SANA-Sprint [3] for the compositional multi-reward setting. For single-reward optimization, we train separate residual transports for three reward functions: aesthetic score [24], PickScore [15], and ImageReward [31]. For compositional multi-reward optimization, we optimize a composite reward consisting of ImageReward, PickScore, HPSv2 [30], and CLIPScore [12] proposed in ReNO [6] and use GenEval [9] solely as a held-out benchmark to evaluate compositional generalization. Full experimental details are provided in Appendix D.

Figure 3. Qualitative comparison of one-step generations before (left) and after LeNO noise correction (right). Top: SANA-Sprint with compositional multi-reward optimization. Bottom: SDXL-Turbo with PickScore optimization.

Baselines. We compare against both inference-time and amortized alignment methods. For inference time alignment, we include Best-of-NN, ψ\psi-Sampler [35], and ZeNO [14]. These methods require no additional model training, but improve reward through additional generator and reward evaluations at test time. In particular, ZeNO performs iterative gradient-free optimization independently for each initial noise, providing a direct per-sample counterpart to the optimization that our method aims to amortize. For amortized alignment, we compare with HyperNoise [7], which learns to modify the initial noise distribution through differentiable reward feedback. We additionally report GoldenNoise [37] as a reference baseline. Unlike the target reward alignment methods considered here, GoldenNoise is trained from an offline dataset of favorable noise-sample pairs rather than directly optimizing the evaluation reward. For the held-out GenEval comparison, we further include a broad set of previously reported text-to-image models and alignment methods as reference baselines [22, 20, 21, 1, 5].

Comparison with Baselines. Table 1 reports direct optimization results for aesthetic score, PickScore, and ImageReward. With a single residual correction, LeNO improves all three rewards over frozen SDXL-Turbo and achieves the best results among the evaluated amortized methods, outperforming HyperNoise without reward gradients. GoldenNoise, trained with different offline supervision, yields only marginal gains. We further evaluate cross-reward transfer, held-out quality metrics, and within-prompt diversity throughout training (Appendix E.1), which indicate that these gains are not obtained at the cost of perceptual quality or sample diversity, suggesting no evident reward hacking.

Method∇R\nabla RNFE ↓\downarrowAes. ↑\uparrowPick. ↑\uparrowImg. ↑\uparrow
SDXL-Turbo–15.33921.9320.761
Inference-Time
BoN×\times8005.71422.9931.201
ψ\psi-Sampler✓\checkmark8005.94523.1821.421
ZeNO×\times8005.75023.0381.260
Amortized
GoldenNoise1^{1}–15.35921.9080.762
HyperNoise✓\checkmark25.38221.9510.939
LeNO×\times25.46022.7151.019
LeNO + BoN×\times1005.95223.1911.444

Table 1. Single reward optimization results on text-to-image generation with SDXL-Turbo.

Inference-time methods achieve higher absolute rewards with substantially larger search budgets: Best-of-NN, ψ\psi-Sampler, and ZeNO use 800 generator evaluations per sample, compared with two NFEs for LeNO. LeNO amortizes this optimization into training, avoiding repeated reward evaluations at inference. Combining LeNO with Best-of-NN over 50 corrected candidates achieves competitive or better rewards than these 800-NFE baselines using only 100 NFEs, demonstrating more efficient inference-time scaling.

Table 2 evaluates transfer to GenEval, which is not directly optimized during multi-reward training. On SANA-Sprint, LeNO improves the mean score from 0.67 to 0.70, with gains in two-object generation, counting, position, and attribute binding. This is comparable to HyperNoise (0.71),

ModelParams (B)Time (s) ↓\downarrowMean ↑\uparrowSingle ↑\uparrowTwo ↑\uparrowCounting ↑\uparrowColors ↑\uparrowPosition ↑\uparrowAttribution ↑\uparrow
SD v2.10.81.90.500.980.510.440.850.070.17
SDXL2.66.90.550.980.740.390.850.150.23
DPO-SDXL2.66.90.590.990.840.490.870.130.24
Hyper-SDXL2.60.30.561.000.760.430.870.100.21
Flux-dev12.023.00.680.990.850.740.790.210.48
SD3-Medium2.04.40.701.000.900.720.870.310.66
SANA-Sprint0.60.20.671.000.680.600.850.400.47
+ HyperNoise0.90.30.711.000.750.670.840.480.53
+ LeNO0.90.30.701.000.740.660.850.410.54

Table 2. GenEval results for held-out compositional evaluation. LeNO and HyperNoise are trained from SANA-Sprint with the same multi-reward objective; GenEval is not used for training. HyperNoise requires reward gradients, whereas LeNO uses only scalar rewards.

despite using only scalar reward evaluations rather than reward gradients, and suggests transfer beyond the training reward ensemble.

State Dependency Ablation. We further evaluate whether conditioning the residual correction on the current initial noise is necessary for effective amortization by comparing the standard state-dependent model μθ(z,c)\mu_\theta(z,c) with a state-independent variant μθ(c)\mu_\theta(c). To preserve the same network architecture, the latter is implemented by feeding a fixed dummy noise zfixedz_{\mathrm{fixed}} in place of the current noise input, while keeping the actual initial noise used for generation and the Local GRPO training procedure unchanged. As shown in Figure 4, the state-dependent model continues to improve throughout training, whereas the state-independent variant shows only a modest initial gain before quickly saturating. This behavior aligns with the intuition from § 3.3, where a shared correction aggregates reward-improvement signals across the initial noise population, while μθ(z,c)\mu_\theta(z,c) can represent distinct local corrections for different noise states.

Exploration Scale Analysis. Figure 5 analyzes the effect of the exploration scale σ\sigma using SDXL-Turbo with PickScore as the training reward. We observe a clear trade-off between exploration and locality. When σ\sigma is too small (e.g., σ≤0.1\sigma\le0.1), the sampled particles remain close to the predicted correction, yielding weak reward contrast and slower improvement. Increasing σ\sigma broadens the explored neighborhood and strengthens the relative reward signal, with the best performance obtained around σ=0.2\sigma= 0.2. However, excessively large exploration degrades training; when σ\sigma exceeds approximately 1.21.2, the reward gain collapses and can even become negative. At this scale, the stochastic perturbation dominates the predicted correction, weakening the local correspondence between sampled particles and the reward landscape around the shared anchor. These results suggest that effective Local GRPO requires sufficient exploration to reveal reward differences while preserving a coherent local neighborhood for relative credit assignment.

Ablation for exploration scale $\sigma$

Figure 5. Ablation for exploration scale σ\sigma.

Protein Structure Generation

Protein backbone generation is the setting that most directly motivates LeNO. Evaluating a backbone requires designing compatible amino-acid sequences and predicting their folded structures through an external pipeline, so the reward is both expensive and non-differentiable with respect to the generator. Gradient-based amortization such as HyperNoise is therefore inapplicable, while persample reward-guided search must invoke the full pipeline many times for every test sample. LeNO instead uses this feedback only during training and applies a single learned correction to new samples without invoking the evaluator at inference.

We optimize designability using the standard self-consistency pipeline. For each generated backbone, ProteinMPNN (Dauparas et al., [4]) designs compatible amino-acid sequences, which are then refolded using ESMFold (Lin et al., [17]). A backbone is considered designable when the self-consistency RMSD (scRMSD) between the generated and refolded structures is below 22 Å. Each reward query thus involves multiple sequence design and structure prediction calls, which dominates the cost of generation itself.

We apply LeNO to a pretrained one-step Riemannian MeanFlow (RMF) (Woo et al., [29]) model while keeping the generator frozen, parameterizing the residual transport with LoRA adapters trained using the same Local GRPO objective. As the primary baselines, we compare against per-sample test-time ZeNO (Kim et al., [14]) applied to the same frozen RMF. We additionally report FrameDiff (Yim et al., [33]) and FrameFlow (Yim et al., [32]), on which RMF is built, as few-step reference models. During training, protein lengths are uniformly sampled from 60 to 128 residues, and all methods are evaluated over the same length range.

Results. Table 3 reports designability and diversity for few-step protein structure generation. FrameDiff and FrameFlow degrade substantially in the few-step regime, whereas LeNO markedly improves the designability of both RMF variants with a single residual correction. Notably, on RMF-M, LeNO outperforms per-sample ZeNO in both designability and scRMSD, even though ZeNO performs 30 optimization steps with 16 particles, i.e., 480 black-box reward queries per sample, each invoking ProteinMPNN and ESMFold. LeNO instead requires no reward queries at inference, matching or exceeding explicit per-sample search at a small fraction of its test-time cost. Its one-time training cost (1k iterations × 36 particles ≈ 36k queries) is thus amortized after generating roughly 75 backbones, beyond which LeNO is strictly cheaper than per-sample optimization. These gains come at a modest cost in structural diversity: MaxCluster (Herbert & Sternberg, [11]) remains comparable to the base models, while pairwise scTM increases slightly, suggesting that LeNO shifts the backbone distribution toward more designable regions without collapsing it. Qualitative examples are shown in Figure 6, and implementation details are provided in Appendix § 10.§ 2.

ModelNFE#Query< 2 ÅscRMSD (↓)Max. Cluster (↑)Pairwise scTM (↓)
FrameDiff500.096.190.540.24
FrameFlow500.046.530.680.22
RMF-M100.154.910.660.34
+ ZeNO4804800.283.540.650.35
+ LeNO200.313.270.640.38
RMF-L100.293.380.660.35
+ ZeNO4804800.402.600.710.36
+ LeNO200.422.950.660.38

Table 3. Comparison of the base models and LeNO. Results are averaged over lengths 60–128, with 10 backbones per length and 8 sequences per backbone.

Visualization of protein structures generated by the base RMF and LeNO

Figure 6. Visualization of protein structures generated by the base RMF and LeNO.

Conclusion

We presented LeNO as an efficient framework for learning reward-improving corrections directly in the initial noise space of frozen deterministic generators. Across text-to-image and protein structure generation, our results show that a lightweight stochastic residual transport can effectively amortize costly per-sample reward optimization while supporting non-differentiable black-box objectives. LeNO also maintains favorable held-out behavior, including compositional generalization in image generation and structural diversity in protein generation. A remaining limitation is that training still requires multiple generator and reward evaluations per anchor, which can be expensive when the reward pipeline itself is costly. Nevertheless, the learned correction introduces only modest inference overhead, making noise space optimization particularly attractive when extremely fast few-step or one-step generation is essential.

AI USE STATEMENT

In this work, we used generative AI tools for sentence-level editing of texts written by authors to improve readability. We have not used generative AI tools for other aspects, such as generating synthetic data, developing theoretical models, formulating mathematical claims or proposing research hypothesis. We have reviewed all AI-assisted work. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Relation to Inference-Time Zeroth-Order Estimator

We analyze the gradient of the expected reward under the stochastic residual transport, omitting the residual-magnitude regularizer and considering the additive transition before norm projection. Fix an anchor zz and condition cc, and define the mean-shifted anchor

z~:=z+ημθ(z,c).(19)\tilde{z} := z + \eta\mu_{\theta}(z,c). \tag*{(19)}

The expected reward under Gaussian action sampling is

JR(z;c):=Eϵ∼N(0,I)[R(z~+σϵ;c)]=Rσ(z~;c).(20)J_R(z;c) := \mathbb{E}_{\epsilon\sim\mathcal{N}(0,I)}\left[R(\tilde{z}+\sigma\epsilon;c)\right] = R_{\sigma}(\tilde{z};c). \tag*{(20)}

We assume the integrability conditions required for differentiating the Gaussian smoothing hold. For brevity, ∇μθ\nabla_{\mu_{\theta}} denotes differentiation with respect to the predicted residual mean μθ(z,c)\mu_{\theta}(z,c), rather than the network parameters.

By the chain rule and the Gaussian score identity,

∇μθJR(z;c)=η∇z~Rσ(z~;c)=ησEϵ[R(z~+σϵ;c)ϵ].(21)\begin{aligned} \nabla_{\mu_{\theta}}J_R(z;c) &= \eta\nabla_{\tilde{z}}R_{\sigma}(\tilde{z};c) \\ &= \frac{\eta}{\sigma}\mathbb{E}_{\epsilon}\left[R(\tilde{z}+\sigma\epsilon;c)\epsilon\right]. \tag*{(21)} \end{aligned}

Thus, the reward gradient with respect to the residual mean equals η\eta times the zeroth-order gradient of the smoothed reward at the mean-shifted anchor.

Since Eϵ[ϵ]=0\mathbb{E}_{\epsilon}[\epsilon]=0, a baseline independent of the sampled perturbation does not change the expected gradient. For K>1K>1 independent perturbations, define

Ri=R(z~+σϵi;c),Rˉ=1K∑i=1KRi.(22)R_i=R(\tilde{z}+\sigma\epsilon_i;c),\qquad\bar{R}=\frac{1}{K}\sum_{i=1}^{K}R_i. \tag*{(22)}

An unbiased centered estimator is

∇μθJR^=η(K−1)σ∑i=1K(Ri−Rˉ)ϵi.(23)\widehat{\nabla_{\mu_{\theta}}J_R}=\frac{\eta}{(K-1)\sigma}\sum_{i=1}^{K}(R_i-\bar{R})\epsilon_i. \tag*{(23)}

The factor 1/(K−1)1/(K-1) corrects for the dependence introduced by including RiR_i in Rˉ\bar{R}. Equivalently, this estimator uses the leave-one-out baseline b−i=(K−1)−1∑j≠iRjb_{-i}=(K-1)^{-1}\sum_{j\ne i}R_j, since

Ri−b−i=KK−1(Ri−Rˉ).(24)R_i-b_{-i}=\frac{K}{K-1}(R_i-\bar{R}). \tag*{(24)}

Equation 22 has the same form as Eq. (4), evaluated at z~\tilde{z} and scaled by η\eta.

Local GRPO instead uses standardized group-relative advantages,

Ai=Ri−RˉsR+ϵ.(25)A_i=\frac{R_i-\bar{R}}{s_R+\epsilon}. \tag*{(25)}

At the sampling policy θ=θold\theta=\theta_{\mathrm{old}}, the unclipped reward-gradient estimate with respect to the residual mean is

ηKσ∑i=1KAiϵi=K−1K(sR+ϵ)∇μθJR^.(26)\frac{\eta}{K\sigma}\sum_{i=1}^{K}A_i\epsilon_i=\frac{K-1}{K(s_R+\epsilon)}\widehat{\nabla_{\mu_{\theta}}J_R}. \tag*{(26)}

For a fixed sampled group, the multiplier is positive, so normalization preserves the direction of this local estimate. Across groups, however, normalization changes their relative weights and does not generally preserve the population reward gradient. Subsequent policy updates introduce likelihood-ratio weighting and potentially PPO clipping, so the resulting GRPO updates need not remain proportional to the zeroth-order estimate.

Proof of Proposition 1

Proposition 1. Let p0=N(0,I)p_0=\mathcal{N}(0,I) and let qθq_{\theta} be a corrected noise distribution induced by the norm-preserving transition TT in Eq. (8). For σ>0\sigma>0 and Ez∼p0∥μθ(z,c)∥2<∞\mathbb{E}_{z\sim p_0}\lVert\mu_{\theta}(z,c)\rVert^2<\infty, the following upper bound holds:

DKL(qθ(⋅∣c)∥p0)≤Ez∼p0DKL(N(ημθ(z,c),σ2I)∥N(0,σ2I))=η22σ2Ez∼p0∥μθ(z,c)∥2.(27)\begin{aligned} D_{\mathrm{KL}}(q_{\theta}(\cdot\mid c)\Vert p_0) &\le\mathbb{E}_{z\sim p_0}D_{\mathrm{KL}}\left(\mathcal{N}(\eta\mu_{\theta}(z,c),\sigma^2I)\Vert\mathcal{N}(0,\sigma^2I)\right) \\ &=\frac{\eta^2}{2\sigma^2}\mathbb{E}_{z\sim p_0}\lVert\mu_{\theta}(z,c)\rVert^2. \tag*{(27)} \end{aligned}

Proof. Fix a condition cc and σ>0\sigma> 0. Let p0=N(0,Id)p_0 = \mathcal{N}(0, I_d) and

T(z,a)=∥z∥2z+a∥z+a∥2.\mathcal{T}(z, a) = \|z\|_2 \frac{z + a}{\|z + a\|_2}.

We first show that zero-mean Gaussian actions preserve the prior, then apply data processing to establish the KL bound.

Preservation of the Gaussian prior. Let z,ϵ∼N(0,Id)z, \epsilon\sim\mathcal{N}(0, I_d) be independent. The output under zero-mean residual actions is

y=T(z,σϵ)=∥z∥2z+σϵ∥z+σϵ∥2.(28)y = \mathcal{T}(z, \sigma\epsilon) = \|z\|_2 \frac{z + \sigma\epsilon}{\|z + \sigma\epsilon\|_2}. \tag*{(28)}

The denominator is nonzero almost surely, and the transition may be defined arbitrarily on the null set where it vanishes.

Write z=ruz = ru, where r=∥z∥2r = \|z\|_2. For an isotropic Gaussian, uu is uniform on the unit sphere and independent of rr. Conditional on r>0r > 0, the vector ru+σϵru + \sigma\epsilon is rotationally invariant: for every orthogonal matrix QQ, the pairs (Qu,Qϵ)(Qu, Q\epsilon) and (u,ϵ)(u, \epsilon) have the same conditional distribution. Its normalized direction is therefore uniform on the unit sphere conditional on rr. Since this conditional distribution does not depend on rr, the output direction is independent of the radius.

The transition also preserves the radius exactly, ∥y∥2=r\|y\|_2 = r. Thus, yy has the same radial distribution and independent uniform direction as zz, giving

T(z,σϵ)∼N(0,Id)=p0.(29)\mathcal{T}(z, \sigma\epsilon) \sim\mathcal{N}(0, I_d) = p_0. \tag*{(29)}

Bounding the marginal KL divergence. Define the joint distributions of anchors and residual actions under the learned and zero-mean transports by

Pθ(dz,da∣c)=p0(dz)N(da;ημθ(z,c),σ2I),Pref(dz,da)=p0(dz)N(da;0,σ2I).(30)\begin{aligned} P_\theta(dz, da \mid c) &= p_0(dz)\mathcal{N}(da; \eta\mu_\theta(z,c), \sigma^2 I),\\ P_{\mathrm{ref}}(dz, da) &= p_0(dz)\mathcal{N}(da; 0, \sigma^2 I). \tag*{(30)} \end{aligned}

Both joint distributions share the anchor marginal p0p_0. The chain rule for KL divergence therefore gives

DKL(Pθ(⋅∣c)∥Pref)=Ez∼p0DKL(N(ημθ(z,c),σ2I)∥N(0,σ2I))=η22σ2Ez∼p0∥μθ(z,c)∥22.(31)\begin{aligned} D_{\mathrm{KL}}(P_\theta(\cdot\mid c)\|P_{\mathrm{ref}}) &= \mathbb{E}_{z\sim p_0}D_{\mathrm{KL}}\left(\mathcal{N}(\eta\mu_\theta(z,c),\sigma^2 I)\middle\|\mathcal{N}(0,\sigma^2 I)\right)\\ &= \frac{\eta^2}{2\sigma^2}\mathbb{E}_{z\sim p_0}\|\mu_\theta(z,c)\|_2^2. \tag*{(31)} \end{aligned}

The second equality follows from the Gaussian KL formula for identical covariances. The assumed finite second moment makes this divergence finite.

Applying the same map (z,a)↦T(z,a)(z,a)\mapsto\mathcal{T}(z,a) to Pθ(⋅∣c)P_\theta(\cdot\mid c) produces qθ(⋅∣c)q_\theta(\cdot\mid c). By Eq. (29), applying it to PrefP_{\mathrm{ref}} produces p0p_0. The data-processing inequality consequently yields

DKL(qθ(⋅∣c)∥p0)≤DKL(Pθ(⋅∣c)∥Pref)=η22σ2Ez∼p0∥μθ(z,c)∥22.(32)\begin{aligned} D_{\mathrm{KL}}(q_\theta(\cdot\mid c)\|p_0) &\leq D_{\mathrm{KL}}(P_\theta(\cdot\mid c)\|P_{\mathrm{ref}})\\ &= \frac{\eta^2}{2\sigma^2}\mathbb{E}_{z\sim p_0}\|\mu_\theta(z,c)\|_2^2. \tag*{(32)} \end{aligned}

This proves the proposition without requiring an invertible transition or evaluating its Jacobian.

From the reward-tilted target to the auxiliary objective. Let λ>0\lambda> 0 and assume 0<Z(c)<∞0 < Z(c) < \infty and Ez′∼qθ(⋅∣c)[∣R(z′;c)∣]<∞\mathbb{E}_{z'\sim q_\theta(\cdot\mid c)}[|R(z';c)|] < \infty. From the definition of p∗p^*,

log⁡qθ(z′∣c)p∗(z′∣c)=log⁡qθ(z′∣c)p0(z′)−R(z′;c)λ+log⁡Z(c).(33)\log\frac{q_\theta(z'\mid c)}{p^*(z'\mid c)} = \log\frac{q_\theta(z'\mid c)}{p_0(z')} - \frac{R(z';c)}{\lambda} + \log Z(c). \tag*{(33)}

Taking expectations under qθ(⋅∣c)q_\theta(\cdot\mid c) gives

DKL(qθ(⋅∣c)∥p∗(⋅∣c))=DKL(qθ(⋅∣c)∥p0)−1λEz′∼qθ(⋅∣c)[R(z′;c)]+log⁡Z(c).(34)D_{\mathrm{KL}}(q_\theta(\cdot\mid c)\|p^*(\cdot\mid c)) = D_{\mathrm{KL}}(q_\theta(\cdot\mid c)\|p_0) - \frac{1}{\lambda}\mathbb{E}_{z'\sim q_\theta(\cdot\mid c)}[R(z';c)] + \log Z(c). \tag*{(34)}

Since Z(c)Z(c) is independent of θ\theta, minimizing this divergence is equivalent to maximizing L(qθ;c)\mathcal{L}(q_\theta; c).

Applying Proposition 1 to the prior-relative KL term yields

L(qθ;c)=Ez′∼qθ(⋅∣c)[R(z′;c)]−λDKL(qθ(⋅∣c)∥p0)≥Ez′∼qθ(⋅∣c)[R(z′;c)]−λη22σ2Ez∼p0∥μθ(z,c)∥22=Laux(θ;c).(35)\begin{aligned} \mathcal{L}(q_\theta; c) = \mathbb{E}_{z'\sim q_\theta(\cdot\mid c)}[\mathcal{R}(z'; c)] - \lambda D_{\mathrm{KL}}(q_\theta(\cdot\mid c)\Vert p_0) \\ &\ge\mathbb{E}_{z'\sim q_\theta(\cdot\mid c)}[\mathcal{R}(z'; c)] - \frac{\lambda\eta^2}{2\sigma^2}\mathbb{E}_{z\sim p_0}\|\mu_\theta(z,c)\|_2^2 \\ &= \mathcal{L}_{\mathrm{aux}}(\theta; c). \tag*{(35)} \end{aligned}

This establishes Eq. (14). The same stochastic transport and sampling scale are used at inference, so the bound applies to the corrected noise distribution used for generation.

State-Dependent versus Shared Corrections

First-order surrogate. Fix a condition cc, consider the additive transition before projection, and set η=1\eta=1 for notational simplicity. We write μ\mu for an unrestricted correction function, independently of its neural parameterization. Under this transition, the corrected noise is z′=z+μ(z,c)+σϵz' = z + \mu(z,c) + \sigma\epsilon, and the expected reward over the Gaussian action equals the smoothed reward at the mean-shifted anchor (Eq. (20)). The auxiliary objective in Eq. (14) then reads

Laux(μ;c)=Ez∼p0[Rσ(z+μ(z,c);c)−λ2σ2∥μ(z,c)∥22].(36)\mathcal{L}_{\mathrm{aux}}(\mu; c) = \mathbb{E}_{z\sim p_0}\left[\mathcal{R}_\sigma(z+\mu(z,c);c) - \frac{\lambda}{2\sigma^2}\|\mu(z,c)\|_2^2\right]. \tag*{(36)}

Linearizing Rσ\mathcal{R}_\sigma around the anchor, Rσ(z+μ;c)≈Rσ(z;c)+g(z,c)⊤μ(z,c)\mathcal{R}_\sigma(z+\mu;c)\approx\mathcal{R}_\sigma(z;c)+g(z,c)^\top\mu(z,c) with g(z,c)=∇zRσ(z;c)g(z,c)=\nabla_z\mathcal{R}_\sigma(z;c) as in Eq. (4), and dropping the μ\mu-independent term gives

Jlin(μ;c):=Ez∼p0[g(z,c)⊤μ(z,c)−α2∥μ(z,c)∥22],α=λσ2.(37)J_{\mathrm{lin}}(\mu;c):=\mathbb{E}_{z\sim p_0}\left[g(z,c)^\top\mu(z,c)-\frac{\alpha}{2}\|\mu(z,c)\|_2^2\right],\qquad\alpha=\frac{\lambda}{\sigma^2}. \tag*{(37)}

The following statements are exact for this surrogate, rather than for the original nonlinear reward objective.

Lemma 1. Assume Ep0∥g(z,c)∥22<∞\mathbb{E}_{p_0}\|g(z,c)\|_2^2<\infty. Let Jstate∗J^*_{\mathrm{state}} and Jshared∗J^*_{\mathrm{shared}} denote the optimal values of Eq. (37) over square-integrable state-dependent corrections μ(z,c)\mu(z,c) and shared corrections μ(c)\mu(c), respectively. The optimal corrections are μstate∗(z,c)=g(z,c)/α\mu^*_{\mathrm{state}}(z,c)=g(z,c)/\alpha and μshared∗(c)=Ep0[g(z,c)]/α\mu^*_{\mathrm{shared}}(c)=\mathbb{E}_{p_0}[g(z,c)]/\alpha, and

ΔJ:=Jstate∗−Jshared∗=12αTr⁡(Cov⁡z∼p0[g(z,c)])≥0,(38)\Delta J:=J^*_{\mathrm{state}}-J^*_{\mathrm{shared}}=\frac{1}{2\alpha}\operatorname{Tr}\left(\operatorname{Cov}_{z\sim p_0}[g(z,c)]\right)\ge0, \tag*{(38)}

with equality if and only if g(z,c)g(z,c) is constant p0p_0-almost surely.

Proof. For each zz, completing the square gives

g⊤μ−α2∥μ∥22=12α∥g∥22−α2∥μ−gα∥22,(39)g^\top\mu-\frac{\alpha}{2}\|\mu\|_2^2=\frac{1}{2\alpha}\|g\|_2^2-\frac{\alpha}{2}\left\|\mu-\frac{g}{\alpha}\right\|_2^2, \tag*{(39)}

where we omit the arguments (z,c)(z,c). The pointwise maximizer is μstate∗=g/α\mu^*_{\mathrm{state}}=g/\alpha, which is square-integrable by the finite second-moment assumption, and Jstate∗=12αEp0∥g∥22J^*_{\mathrm{state}}=\frac{1}{2\alpha}\mathbb{E}_{p_0}\|g\|_2^2. For a shared correction μ(c)\mu(c), the objective reduces to Ep0[g]⊤μ(c)−α2∥μ(c)∥22\mathbb{E}_{p_0}[g]^\top\mu(c)-\frac{\alpha}{2}\|\mu(c)\|_2^2, and the same argument gives μshared∗=Ep0[g]/α\mu^*_{\mathrm{shared}}=\mathbb{E}_{p_0}[g]/\alpha and Jshared∗=12α∥Ep0[g]∥22J^*_{\mathrm{shared}}=\frac{1}{2\alpha}\|\mathbb{E}_{p_0}[g]\|_2^2. Subtracting,

ΔJ=12α(Ep0∥g∥22−∥Ep0[g]∥22)=12αTr⁡(Cov⁡p0[g])≥0,(40)\Delta J=\frac{1}{2\alpha}\left(\mathbb{E}_{p_0}\|g\|_2^2-\|\mathbb{E}_{p_0}[g]\|_2^2\right)=\frac{1}{2\alpha}\operatorname{Tr}\left(\operatorname{Cov}_{p_0}[g]\right)\ge0, \tag*{(40)}

with equality if and only if Cov⁡p0[g]=0\operatorname{Cov}_{p_0}[g]=0, i.e., gg is constant p0p_0-almost surely.

Implementation Details

All experiments are conducted using NVIDIA RTX 4090 GPUs with 24GB of memory.

Text-to-Image Generation

Hyperparameters. We use the same hyperparameters for all text-to-image Local GRPO experiments. We set the correction scale to η=0.3\eta= 0.3 and the exploration standard deviation to σ=0.2\sigma= 0.2, using 32 particles per group, 16 prompt anchors per update, and four GRPO epochs per rollout. Advantages are normalized using the group mean and standard deviation and clipped to [−5,5][-5, 5]. The PPO probability ratio is clipped to [0.8,1.2][0.8, 1.2], while the log-ratio is clipped to [−10,10][-10, 10] for numerical stability.

Single Reward Optimization. For single-reward optimization, we use SDXL-Turbo as the frozen generator and consider three reward functions: the aesthetic score predictor, PickScore, and ImageReward. We train a separate LeNO model for each reward using only the corresponding scalar reward signal. For a fair comparison, we also retrain HyperNoise independently for all three reward functions rather than reusing a single pretrained checkpoint. In each case, we follow the default HyperNoise training configuration and change only the target reward function. The Local GRPO hyperparameters and model-specific training configurations are summarized in Table 4 and Table 5, respectively.

HyperparameterValue
Correction scale η\eta0.3
Exploration std. σ\sigma0.2
KL regularization weight β\beta0.01
Global particle group size32
Prompt anchors per update16
GRPO epochs per rollout4
Advantage normalizationGroup mean/std
Advantage clipping[−5,5][-5, 5]
PPO ratio clipping[0.8,1.2][0.8, 1.2] (ϵ=0.2\epsilon= 0.2)
Log-ratio clipping[−10,10][-10, 10]

Table 4. GRPO training hyperparameters.

Single RewardMulti Reward
ModelHyperNoiseLeNOHyperNoiseLeNO
ModelSDXL-TurboSDXL-TurboSANA-SprintSANA-Sprint
Denoising Step1111
Learning rate1×10−31 \times10^{-3}5×10−55 \times10^{-5}1×10−31 \times10^{-3}5×10−55 \times10^{-5}
GradNorm Clipping1.01.01.01.0
LoRA rank1616128128
LoRA alpha1616256256
OptimizerSGDAdamSGDAdam
Batch size16161616
Accumulation steps3333
Training steps60k5k60k5k
Number of training prompts55k55k55k55k
Image size512×512512 \times512512×512512 \times5121024×10241024 \times10241024×10241024 \times1024

Table 5. Training configurations for HyperNoise and LeNO under single- and multi-reward optimization.

Compositional Multi-Reward Optimization. For compositional optimization, we follow the multi-reward formulation of ReNO [6] and use SANA-Sprint as the frozen generator. Specifically, the training reward combines ImageReward, HPSv2, PickScore, and CLIPScore, which provide complementary signals for human preference, perceptual quality, and text–image consistency. Following ReNO, we use the weighted objective

Rmulti=1.0 RImageReward+5.0 RHPSv2+0.05 RPickScore+1.0 RCLIPScore,(41)R_{\mathrm{multi}} = 1.0\,R_{\mathrm{ImageReward}} + 5.0\,R_{\mathrm{HPSv2}} + 0.05\,R_{\mathrm{PickScore}} + 1.0\,R_{\mathrm{CLIPScore}}, \tag*{(41)}

where the weighting accounts for the substantially different numerical scales of the individual reward models. The corresponding Local GRPO and training configurations are reported in Table 4 and Table 5.

To evaluate whether improvements transfer beyond the reward models used for training, we additionally report GenEval [9] as a held-out evaluation. The GenEval prompts and evaluator are not used during LeNO training, and the GenEval score is not included in the multi-reward objective. We evaluate the trained model only after optimization using the standard GenEval benchmark, which measures compositional text–image alignment across single-object generation, two-object composition, counting, color, spatial relations, and attribute binding. Thus, improvements on GenEval indicate that the learned noise correction generalizes beyond direct optimization of the four training rewards.

Protein Structure Generation

Generator and Residual Correction. We use the pretrained both RMF-M and RMF-L checkpoint at epoch 1600 as a frozen one-step backbone generator. For a protein of length NN, the initial state consists of residue-wise translations and rotations, X=(T,R)∈R0N×3×SO⁡(3)NX = (T, R) \in\mathbb{R}_0^{N \times3} \times\operatorname{SO}(3)^N, where the translation subspace has zero center of mass. Translations follow a centered Gaussian prior with standard deviation τT=10\tau_T = 10 Å, and rotations are sampled independently from the Haar measure on SO⁡(3)\operatorname{SO}(3). We parameterize the residual correction using the difference between LoRA-adapted and frozen RMF velocity fields:

(UT,UR)=fθLoRA(X;0,1)−f0(X;0,1).(42)(U_T, U_R) = f_{\theta}^{\mathrm{LoRA}}(X; 0, 1) - f_0(X; 0, 1). \tag*{(42)}

Rank-32 LoRA adapters with α=32\alpha= 32 are inserted into 284 trunk linear layers, giving approximately 17.2M trainable parameters. Only the adapters are optimized, and their output matrices are initialized to zero so that the initial predicted correction is zero.

Stochastic Residual Transport. Let C\mathcal{C} remove the translation center of mass and clip⁡c\operatorname{clip}_c limit each residue-wise vector to norm cc. We define the translation and rotation action means as

MT=C(clip⁡cT(ηTUT)),M_T = \mathcal{C}\left(\operatorname{clip}_{c_T}(\eta_T U_T)\right),
MR=clip⁡cR(ηRUR),(43)M_R = \operatorname{clip}_{c_R}(\eta_R U_R), \tag*{(43)}

with (ηT,ηR)=(0.30,0.15)(\eta_T, \eta_R) = (0.30, 0.15), cT=2.0c_T = 2.0 Å, and cR=0.10c_R = 0.10 rad. The stochastic transport samples residual actions

AT=MT+σTC(ϵT),A_T = M_T + \sigma_T \mathcal{C}(\epsilon_T),
AR=MR+σRϵR,(44)A_R = M_R + \sigma_R \epsilon_R, \tag*{(44)}

where ϵT\epsilon_T and ϵR\epsilon_R are independent standard Gaussian tensors, σT=2.0\sigma_T = 2.0 Å, and σR=0.20\sigma_R = 0.20 rad. The corrected initial frames are

T+=∥T∥FC(T+AT)∥C(T+AT)∥F,T^+ = \lVert T \rVert_F \frac{\mathcal{C}(T + A_T)}{\lVert\mathcal{C}(T + A_T) \rVert_F},
Ri+=RiExp⁡(A^R,i),(45)R_i^+ = R_i \operatorname{Exp}(\widehat{A}_{R,i}), \tag*{(45)}

where ⋅^\widehat{\cdot} maps a tangent vector to a skew-symmetric matrix. The translation update preserves the Frobenius norm of each centered Gaussian anchor while changing its direction, and the rotation update remains on SO⁡(3)\operatorname{SO}(3).

During training, we sample multiple independent action pairs for each anchor to construct group-relative advantages. At inference, we sample one action pair using the same σT\sigma_T and σR\sigma_R and apply a single residual transport step before generation.

Proposition 1 extends directly to the protein structure generation setting. The translation prior is an isotropic Gaussian on the zero-center-of-mass subspace, and the centered noise C(ϵT)C(\epsilon_T) lies in the same subspace, so the rotational-invariance argument applies within it. For rotations, RiR_i is Haar-distributed and independent of the zero-mean action, so RiExp⁡(σϵR,i)R_i \operatorname{Exp}(\sigma\epsilon_{R,i}) remains Haar-distributed by invariance of the Haar measure. Clipping acts only on the predicted mean and does not affect the argument.

Local GRPO Training. Each training iteration samples one initial state with N∼U{60,…,128}N \sim U\{60,\ldots,128\} and explores 36 local particles. Rewards are standardized within each group using the population standard deviation with ϵ=10−6\epsilon= 10^{-6}, and advantages are clipped to [−5,5][-5,5]. Exploration likelihoods are evaluated on the stored pre-projection residuals (AT,AR)(A_T, A_R). We perform two optimization epochs per group with surrogate clipping 0.20 and a Gaussian mean-shift KL regularizer toward zero residual correction, weighted by 5×10−35 \times10^{-3}. We use Adam with learning rate 2×10−52 \times10^{-5}, (β1,β2)=(0.9,0.999)(\beta_1,\beta_2) = (0.9, 0.999), ϵ=10−8\epsilon= 10^{-8}, no weight decay, and gradient-norm clipping at 1.0. We report the checkpoint at iteration 1k. The corresponding training configurations are reported in Table 6.

HyperparameterValue
Frozen generatorRMF-M / RMF-L
Generator sampling steps1
Residual transport steps1
Training protein lengthN∼U{60,…,128}N \sim U\{60,\ldots,128\}
LoRA rank / alpha32/32
Trainable parameters7.1M / 17.2M
Initial states per iteration1
Local particles per state36
Optimization epochs per group2
Gradient accumulation1
Surrogate clipping0.20
Advantage clipping[−5,5][-5,5]
KL coefficient5×10−35 \times10^{-3}
OptimizerAdam
Learning rate2×10−52 \times10^{-5}
Adam betas(0.9,0.999)(0.9, 0.999)
Weight decay0
GradNorm clipping1.0
Training steps1k
Sequences per backbone (train / eval.)4/8
Sequence sampling temperature0.1
Reward soft-min temperature1.0 A˚1.0\,\text{\AA}

Table 6. Training configuration for protein generation.

Designability Reward and Evaluation. Each corrected initial state is decoded into a backbone by the frozen one-step RMF generator. ProteinMPNN samples K=4K = 4 sequences per backbone at temperature 0.1, and ESMFold (facebook/esmfold_v1) refolds them. Given their aligned self-consistency RMSDs {dk}k=1K\{d_k\}_{k=1}^{K}, we maximize the negative normalized soft minimum:

R=τlog⁡(1K∑k=1Kexp⁡(−dk/τ)),τ=1.0 A˚.(46)\mathcal{R} = \tau\log\left(\frac{1}{K}\sum_{k=1}^{K}\exp(-d_k/\tau)\right), \qquad\tau= 1.0\,\text{\AA}. \tag*{(46)}

All particles within a local group share the same ProteinMPNN seed to reduce reward variance; no pLDDT term is included. For final evaluation, we sample eight sequences per backbone and classify it as designable when the minimum scRMSD is below 2 A˚2\,\text{\AA}.

Additional Experiments

Quality and Diversity Preservation

We investigate whether optimizing the initial noise for a single reward leads to diversity collapse or degradation in perceptual image quality. We evaluate the PickScore-, Aesthetic-, and ImageReward-optimized models on 100 held-out TempFlow prompts, using eight fixed initial-noise seeds per prompt. The same prompt–seed pairs are shared across all methods and checkpoints. Since the Aesthetic model has the shortest training horizon, we compare all models up to 800 training steps. The shaded regions in Figure 7 denote 95% confidence intervals obtained by paired bootstrap resampling over prompts.

Diversity and no-reference image quality throughout single-reward optimization

Figure 7. Diversity and no-reference image quality throughout single-reward optimization. LPIPS and DINO diversity are normalized by the unoptimized SDXL-Turbo baseline, while MUSIQ and CLIP-IQA are shown as absolute improvements over the baseline. All methods preserve nearly all of the baseline diversity, while consistently improving perceptual image quality. Shaded regions indicate 95% paired bootstrap confidence intervals over prompts.

We report two complementary diversity metrics and two no-reference image quality metrics:

  • LPIPS diversity ([36]): the average pairwise LPIPS distance among the eight images generated for the same prompt. A higher value indicates greater perceptual variation across initial-noise samples.

  • DINO diversity ([19]): the average pairwise cosine distance between DINOv2 image features. This metric captures diversity at a more semantic and content-level representation than pixel-level distances.

  • MUSIQ ([13]): a no-reference image quality assessment metric that measures overall perceptual quality without requiring reference images. Higher values indicate better image quality.

  • CLIP-IQA ([27]): a CLIP-based no-reference image quality metric that assesses perceptual quality using semantic visual representations. Higher values indicate better quality.

For the diversity metrics, we additionally report retention relative to the unoptimized SDXL-Turbo baseline. For the quality metrics, we report the absolute score together with its improvement over the baseline.

As shown in Figure 7 and Table 7, reward optimization does not induce diversity collapse. At step 800, PickScore retains 101.2% and 99.5% of the baseline LPIPS and DINO diversity, respectively. The Aesthetic model shows only a marginal reduction, retaining 99.2% LPIPS diversity and 99.7% DINO diversity. ImageReward increases both measures to 104.9% and 105.5%, respectively. Across the entire optimization trajectory, the minimum observed retention is 97.7% for LPIPS and 98.0% for DINO, indicating that the seed-dependent variation is largely preserved.

MethodLPIPS (ret.) ↑DINO div. (ret.) ↑MUSIQ (Δ) ↑CLIP-IQA (Δ) ↑
SDXL-Turbo0.5984 (100.0%)0.1952 (100.0%)70.96 (+0.00)0.6600 (+0.0000)
PickScore (step 800)0.6054 (101.2%)0.1942 (99.5%)72.67 (+1.72)0.7192 (+0.0592)
Aesthetic (step 800)0.5938 (99.2%)0.1947 (99.7%)72.51 (+1.55)0.7335 (+0.0734)
ImageReward (step 800)0.6275 (104.9%)0.2059 (105.5%)73.93 (+2.97)0.7306 (+0.0706)

Table 7. Diversity and image quality after 800 training steps. Values in parentheses denote diversity retention relative to SDXL-Turbo for LPIPS and DINO, and absolute improvement over SDXL-Turbo for MUSIQ and CLIP-IQA.

At the same time, all three optimized models improve both no-reference quality metrics. At step 800, MUSIQ increases by +1.72, +1.55, and +2.97 for PickScore, Aesthetic, and ImageReward, respectively. CLIP-IQA similarly improves by +0.0592, +0.0734, and +0.0706. These improvements appear throughout training rather than only at the final checkpoint. Overall, the results show that Local GRPO improves the target reward without sacrificing sample diversity and simultaneously improves no-reference perceptual quality. We note that these measurements characterize within-prompt diversity and perceptual quality, rather than distribution-level fidelity such as FID.

Cross-reward evaluation. To further examine whether gains on the target reward come at the expense of other preference signals, we evaluate each single-reward LeNO model on all three rewards using the same held-out prompt–seed pairs. As shown in Table 8, each model achieves its largest improvement on its own training reward (diagonal), while the two non-target rewards also increase in every case. For example, the PickScore-optimized model improves ImageReward by +0.160, and the ImageReward-optimized model improves both Aesthetic and PickScore. No non-target reward decreases on average, indicating that LeNO does not exploit idiosyncrasies of a single reward model but instead shifts generations toward outputs that are broadly preferred.

Training rewardEvaluation reward
Aesthetic ↑PickScore ↑ImageReward ↑
SDXL-Turbo (no training)6.117421.86080.9710
Aesthetic6.1796 (+0.0622)21.9728 (+0.1120)1.0297 (+0.0587)
PickScore6.1688 (+0.0514)22.3308 (+0.4701)1.1306 (+0.1596)
ImageReward6.1558 (+0.0384)21.9092 (+0.0484)1.1984 (+0.2274)

Table 8. Cross-reward evaluation of single-reward LeNO models on SDXL-Turbo. Rows denote the training reward and columns the evaluation reward. Values in parentheses denote improvement over SDXL-Turbo; shaded cells indicate the target reward.

Ablation for multi-step transport reward gain

Figure 8. Ablation for multi-step transport reward gain.

Multi-step Correction Ablation

We additionally investigate whether applying the learned noise correction over multiple residual steps provides finer-grained optimization than the single-step formulation used in the main experiments. We consider k∈{1,2,4}k \in\{1, 2, 4\} residual transport steps. For the multi-step variants, we use a branching strategy in which stochastic perturbations are introduced at only one of the kk steps, while the remaining steps follow deterministic residual transitions. Concretely, at the selected branching step, multiple local particles are generated around the predicted correction and evaluated using the terminal reward; before and after this branching point, the transport is applied deterministically. This preserves the local reward comparison used by Local GRPO without exponentially branching the trajectory at every step.

Figure 8 compares the reward gain throughout training for k=1,2k = 1, 2, and 44. All three settings exhibit nearly identical improvement curves, and increasing the number of residual steps does not lead to a clear additional gain. Although a multi-step transport is in principle more expressive and could progressively refine the corrected noise, we do not observe such a benefit in this setting. One possible explanation is that the single-step state-dependent correction already captures most of the useful local reward-improvement direction. Moreover, under the branching strategy, only one step receives stochastic local exploration in each branched trajectory, while the remaining steps compose deterministic transformations. Therefore, increasing kk does not necessarily provide additional independent reward-improvement signals. These results suggest that iterative refinement is not essential for the tasks considered here, motivating the use of the simpler single-step transport in our main experiments.

KL Regularization Ablation

We study the effect of the KL regularization weight β\beta, which controls how strongly the learned residual transport is constrained to remain close to the reference noise distribution. We evaluate β∈{0,0.01,0.1,1.0}\beta\in\{0, 0.01, 0.1, 1.0\} using ImageReward as the optimization objective.

As shown in Figure 9, a small amount of KL regularization improves both the final reward gain and training stability. Without KL regularization (β=0\beta= 0), LeNO still improves the reward substantially, but achieves a lower final gain (+0.1889 ±\pm 0.1038) and win rate (69%) than β=0.01\beta= 0.01, which obtains the best result (+0.2504 ±\pm 0.0887, 76% win rate). Increasing the KL regularization further reduces the attainable improvement: β=0.1\beta= 0.1 yields a gain of +0.1433 ±\pm 0.0543, while β=1.0\beta= 1.0 nearly suppresses the reward improvement entirely.

Ablation of the KL regularization weight $\beta$.

Figure 9. Ablation of the KL regularization weight β\beta.

These results suggest that the KL term plays more than a purely reward-regularization role. When its weight is zero, the residual transport can move the corrected-noise distribution more freely away from the reference prior, which may weaken the local structure assumed by the residual parameterterization and lead to less stable optimization. A small KL weight acts as a trust-region constraint, allowing sufficient movement toward high-reward regions while preventing excessively large distributional shifts. Conversely, overly strong regularization restricts the correction magnitude and prevents the transport from reaching high-reward noise states. Accordingly, an intermediate KL weight provides the best trade-off between reward improvement and preservation of the reference noise structure. This behavior is consistent with the reward-tilted target, which balances reward maximization against deviation from the reference noise distribution rather than optimizing reward alone.

β\betaImageReward GainWin Rate
00.188969%
0.010.250476%
0.10.143368%
1.00.011259%

Table 9. Effect of β\beta on ImageReward gain and win rate.

Generalization to a One-Step GAN Generator

We further evaluate whether LeNO is applicable beyond diffusion-based generators by applying it to DF-GAN [26], a one-step prompt-conditioned GAN. We use the same LeNO training setup as in the main experiments, keeping the pretrained generator frozen and optimizing only the residual correction from scalar reward evaluations. Since LeNO operates entirely in the initial noise space and does not require gradients through either the generator or the reward function, no modification of the generator architecture or its internal sampling process is required.

Figure 10 shows qualitative examples before and after LeNO correction to PickScore. Across different prompts, the corrected initial noises produce samples with improved reward-aligned visual characteristics while preserving the underlying generation capability of the frozen DF-GAN. Importantly, this experiment uses the same optimization procedure despite the substantially different generator family. The result therefore supports that LeNO is not tied to a particular diffusion formulation or denoising trajectory, but can be applied to deterministic prompt-conditioned generators whenever they expose an initial noise input and can be evaluated by a scalar reward oracle. This provides additional evidence for the black-box applicability of LeNO to both the generator and the reward function.

Figure 10. a white head and an orange beak; ~ have white markings on the secondaries; a white neck mottled white and gray bellow; has a yellow beak that curves downward. Qualitative comparison of DF-GAN generated samples before (left) and after LeNO noise correction (right).

Additional Results

Figure 11. Archive photo of Einstein; A rubber duck floating in space; A cup of coffee ... phoenix rooster ...; A frog wearing a pink kimono ... Additional qualitative comparison of SANA-Sprint optimized with compositional multi-rewards.

Figure 12. Additional qualitative comparison of SDXL-Turbo optimized with Image Rewards.

Figure 13. Additional qualitative comparison of SDXL-Turbo optimized with PickScore.

Additional qualitative self-consistency comparison with RMF at protein length $L=96$

Figure 14. Additional qualitative self-consistency comparison with RMF at protein length L=96L=96.

References

  1. [1]Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, et al. Flux. 1. k context: Flow matching for in-context image generation and editing in latent space. arXiv e-prints, arXiv–2506, 2025.DOI
  2. [2]Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations, volume 2024, pp. 4965–4987, 2024.arxiv.org/abs/2305.13301
  3. [3]Junsong Chen, Shuchen Xue, Yuyang Zhao, Jincheng Yu, Sayak Paul, Junyu Chen, Han Cai, Song Han, and Enze Xie. Sana-sprint: One-step diffusion with continuous-time consistency distillation. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16185–16195. IEEE, 2025.arxiv.org/abs/2503.09641
  4. [4]J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, P. J. Y. Leung, T. F. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, A. Kang, B. Sankaran, A. K. Bera, N. P. King, and D. Baker. Robust deep learning–based protein sequence design using proteinmpnn. Science, 378(6615):49–56, 2022. doi: 10.1126/science.add2187. URL https://www.science.org/doi/abs/10.1126/science.add2187.DOI
  5. [5]Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.arxiv.org/abs/2403.03206
  6. [6]Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. ReNO: Enhancing one-step text-to-image models through reward-based noise optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=MXY0qsGGeO.arxiv.org/abs/2406.04312
  7. [7]Luca Eyring, Shyamgopal Karthik, Alexey Dosovitskiy, Nataniel Ruiz, and Zeynep Akata. Noise hypernetworks: Amortizing test-time compute in diffusion models. Advances in Neural Information Processing Systems, 38:95429–95462, 2026.
  8. [8]Zhengyang Geng, Mingyang Deng, Xingjian Bai, Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505.13447, 2025.arxiv.org/abs/2505.13447
  9. [9]Dhruba Ghosh, Hannah Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=Wbr51vK331.arxiv.org/abs/2310.11513
  10. [10]Dailan He, Guanlin Feng, Xingtong Ge, Yazhe Niu, Yi Zhang, Bingqi Ma, Guanglu Song, Yu Liu, and Hongsheng Li. Neighbor grpo: Contrastive ode policy optimization aligns flow models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6033–6042, 2026.arxiv.org/abs/2511.16955
  11. [11]A. Herbert and M. Sternberg. Maxcluster: a tool for protein structure comparison and clustering. https://ic.ac.uk, 2008.
  12. [12]Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, pp. 7514–7528, 2021.
  13. [13]Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5128–5137. IEEE, 2021.arxiv.org/abs/2108.05997
  14. [14]Jeongsol Kim, Hongeun Kim, Jian Wang, and Jong Chul Ye. Gradient-free noise optimization for reward alignment in generative models. arXiv preprint arXiv:2605.11347, 2026.
  15. [15]Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=G5RwHpBUv0.
  16. [16]Jeongjae Lee and Jong Chul Ye. Pcpo: Proportionate credit policy optimization for aligning image generation models. arXiv preprint arXiv:2509.25774, 2025.
  17. [17]Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123–1130, 2023. doi: 10.1126/science.ade2574. URL https://www.science.org/doi/abs/10.1126/science.ade2574.
  18. [18]Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di ZHANG, and Wanli Ouyang. Flow-GRPO: Training flow matching models via online RL. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=oCBKGw5HNf.
  19. [19]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
  20. [20]Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
  21. [21]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023.arxiv.org/abs/2305.18290
  22. [22]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022.
  23. [23]Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision, pp. 87–103. Springer, 2024.arxiv.org/abs/2311.17042
  24. [24]Christoph Schuhmann. Improved aesthetic predictor. https://github.com/christophschuhmann/improved-aesthetic-predictor, 2022. GitHub repository.
  25. [25]Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.arxiv.org/abs/2303.01469
  26. [26]Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16515–16525, 2022.
  27. [27]Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 2555–2563, 2023.
  28. [28]Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15(27):949–980, 2014. URL http://jmlr.org/papers/v15/wierstra14a.html.
  29. [29]Dongyeop Woo, Marta Skreta, Seonghyun Park, Kirill Neklyudov, and Sungsoo Ahn. Riemannian meanflow, 2026. URL https://arxiv.org/abs/2602.07744.
  30. [30]Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023.arxiv.org/abs/2306.09341
  31. [31]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2304.05977
  32. [32]Jason Yim, Andrew Campbell, Andrew YK Foong, Michael Gastegger, José Jiménez-Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan S Veeling, Regina Barzilay, Tommi Jaakkola, et al. Fast protein backbone generation with se (3) flow matching. arXiv preprint arXiv:2310.05297, 2023a.arxiv.org/abs/2310.05297
  33. [33]Jason Yim, Brian L Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprint arXiv:2302.02277, 2023b.arxiv.org/abs/2302.02277
  34. [34]Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6613–6623, 2024.arxiv.org/abs/2311.18828
  35. [35]TaeHoon Yoon, Yunhong Min, Kyeongmin Yeo, and Minh hyuk Sung. $\psi$-sampler: Initial particle sampling for SMC-based inference-time reward alignment in score models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=slVqJAI5sT.arxiv.org/abs/2506.01320
  36. [36]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586–595, 2018.arxiv.org/abs/1801.03924
  37. [37]Zikai Zhou, Shitong Shao, Lichen Bai, Shifei Zhang, Zhiqiang Xu, Bo Han, and Zeke Xie. Golden noise for diffusion models: A learning framework. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 17688–17697. IEEE, 2025.arxiv.org/abs/2411.09502

Paper details

vlnc:W_64d2dava
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents