betaFeedback

WorksComputer Science

A General Framework for Inference-time Scaling and Steering of Diffusion Models

Abstract

Diffusion models produce impressive results in modalities ranging from images and video to protein design and text. However, generating samples with user-specified properties remains a challenge. Recent research proposes fine-tuning models to maximize rewards that capture desired properties, but these methods require expensive training and are prone to mode collapse. In this work, we present Feynman-Kac (FK) steering, an inference-time framework for steering diffusion models with reward functions. FK steering works by sampling a system of multiple interacting diffusion processes, called particles, and resampling particles at intermediate steps based on scores computed using functions called potentials. Potentials are defined using rewards for intermediate states and are selected such that a high value indicates that the particle will yield a high-reward sample. We explore various choices of potentials, intermediate rewards, and samplers. We evaluate FK steering on text-to-image and text diffusion models. For steering text-to-image models with a human preference reward, we find that FK steering a 0.8B parameter model outperforms a 2.6B parameter fine-tuned model on prompt fidelity, with faster sampling and no training. For steering text diffusion models with rewards for text quality and specific text attributes, we find that FK steering generates lower perplexity, more linguistically acceptable outputs and enables gradient-free control of attributes like toxicity. Our results demonstrate that inference-time scaling and steering of diffusion models - even with off-the-shelf rewards - can provide significant sample quality gains and controllability benefits. Code is available at https://github.com/zacharyhorvitz/Fk-Diffusion-Steering .

Introduction

Diffusion-based generative models [1] have led to advances in modeling images [2, 3], videos [4], and proteins [5], as well as promising results for text generation [6, 7, 8, 9, 10]. Despite these advances, diffusion models have failure modes. For example, there is a high failure rate for text-to-image models in terms of adherence to text prompts [11]. Additionally, adapting these models to produce samples that conform to user preferences remains a challenge.

One approach for making generative models adhere to user preferences is to encode preference via a reward function r(x0)r(x_0) and sample from a tilted distribution ptarget(x0)∝pθ(x0)exp⁡(r(x0))p_{\mathrm{target}}(x_0) \propto p_\theta(x_0)\exp(r(x_0)) [12], such that high-reward samples are up-weighted and low-reward samples are down-weighted. These reward functions can be human preference scores [13, 14], vision-language models [15] to score prompt fidelity, or likelihoods p(y∣x0)p(y \mid x_0) for an attribute yy [Wu et al., 2023b]. Current approaches for sampling from this tilted distribution can be categorized into (a) fine-tuning and (b) inference-time steering methods.

Black et al. [ [17], Domingo-Enrich et al. [ [18], Wallace et al. [ [19] explore fine-tuning of diffusion models with reward functions. However, fine-tuning requires expensive training and ties the model to a single reward. Alternatively, two common inference-time steering approaches are reward gradient-based guidance [3, 20] and best-of-n sampling. Best-of-n sampling can be used to guide any diffusion model with generic reward functions, however, it allocates a large amount of computation to samples that yield low rewards [21]. Gradient-based guidance presents an efficient alternative, but it is limited to differentiable reward functions and continuous-state diffusion models. Therefore, steering a diffusion model at inference-time with arbitrary rewards remains a challenge.

In this work, we present Feynman-Kac steering (FK STEERING), a flexible framework for steering diffusion-based generative models with arbitrary rewards that uses FK interacting particle system methods [22, 23]. We generalize previous works that define Feynman-Kac measures to conditionally sample diffusion models [16, 24, 25, 26]. FK STEERING enables guidance with arbitrary reward functions, differentiable or otherwise, for both discrete and continuous-state models. The approach makes use of a rare-event simulation method, Feynman-Kac interacting particle system (FK-IPS) [22, 23, 27, 28]. FK-IPS enables the generation of samples with high-rewards, which may be rare events under the original model pθ(x)p_\theta(x).

FK STEERING works by (a) sampling multiple interacting diffusion processes, called particles, (b) scoring these particles using functions called potentials, and (c) resampling the particles based on their potentials at intermediate steps during generation (see Figure 1). Potential functions are defined using intermediate rewards. Resampling with these intermediate rewards yields high-reward samples and eliminates lower reward particles. We present several ways of defining these intermediate rewards and potentials and empirically demonstrate that these new choices improve on traditional choices [16, 29]. By expanding the set of choices, users can find potentials that are better suited for their tasks. Remarkably, for a number of tasks, we see significant performance benefits for both image and text diffusion models with FK STEERING with as few as k=4k = 4 particles (see Figure 2). Additionally, we find that FK STEERING smaller diffusion models outperforms larger models, and their fine-tuned versions, using less compute.

Feynman-Kac diffusion steering (FK STEERING): inference, best-of-N, and FK steering particle trajectories

Figure 1. Feynman-Kac diffusion steering (FK STEERING) is a particle-based steering approach that generates multiple samples (particles) like best-of-n (importance sampling) approaches. In FK STEERING, particles are evaluated at intermediate steps, where they are scored with functions called potentials. Potentials are defined using intermediate rewards and are selected such that promising particles are resampled and poor samples are terminated. Additionally, potentials are selected such that resulting outputs are samples from the tilted distribution, x0∝pθ(x0)exp⁡(λr(x0))x_0 \propto p_\theta(x_0) \exp(\lambda r(x_0)).

Figure 2. With FK STEERING small models outperform bigger models with faster sampling and less compute. We compare stable diffusion text-to-image models and their fine-tuned versions, DPO [19] and DDPO [17], with FK STEERING. We use the ImageReward model [13] as the reward function and generate samples from the base model without reward gradients [20]. We also note that FK STEERING with k=2k = 2 outperforms the fine-tuned models. Left: The GenEval prompt fidelity scores [11] of the base models with FK STEERING and the fine-tuned models [19]. Without any training SDv2.1 with FK STEERING (k=3k = 3) outperforms a finetuned SDXL model with fewer FLOPS and less sampling time. Right: FLOPs for each configuration are on the x-axis.

Contributions. In summary, our methodological contributions are the following:

  • We propose Feynman-Kac diffusion steering as a framework for building particle-based approximations for sampling the tilted distribution pθ(x0)exp⁡(λr(x0))p_\theta(x_0)\exp(\lambda r(x_0)). FK STEERING can steer both continuous and discrete state-space models with generic reward functions. We also show that FK STEERING can be used to steer conditional models.

  • We show that particle-based sampling methods such as twisted diffusion sampler (TDS) [16] and Li et al. [ [29], are specific instances of FK interacting particle systems. We demonstrate that FK STEERING enables new choices of potentials, samplers, and reward models that can improve performance across many tasks.

Empirically, we demonstrate that FK STEERING:

  • Provides an alternative to fine-tuning when using sample quality rewards. We steer text-to-image latent diffusion models with an off-the-shelf reward model. FK STEERING with just k=4k = 4 particles outperforms fine-tuning on prompt fidelity and aesthetic quality, without making use of reward gradients. We also steer text diffusion models to generate higher quality samples that have more competitive linguistic acceptability and perplexity to those sampled from auto-regressive models.

  • Enables smaller models (0.8B parameters) to outperform larger models (2.6B parameters) on prompt-fidelity [11], using fewer FLOPs (see the right panel in Figure 2). Moreover, we show that using FK STEERING with fine-tuned models unlocks further performance benefits.

  • Outperforms fine-tuned models on prompt fidelity with just k=2k = 2 particles (see Figure 2).

  • Provides a valuable method for generating samples with (rare) specified attributes. As an example, the method can be used to generate toxic language to support efforts in responsible AI [32], including for red-teaming model behavior, and for fine-turning and steering generalist language models to recognize and reject toxic language input. On toxicity, without gradient guidance, FK STEERING can increase the toxicity rate of a text diffusion model from 0.3% to 64.7% with k=8k = 8 particles, and outperforms both gradient guidance and best-of-nn.

Overall, in all settings we consider, FK STEERING always improves performance, highlighting the benefits of inference-time scaling and steering of diffusion models.

Related Work

Current approaches to generate samples from the tilted distribution pθ(x0)exp⁡(λr(x0))p_{\theta}(x_0) \exp(\lambda r(x_0)) can be categorized into two types: (1) fine-tuning [17, 18, 33], and (2) inference-time steering approaches, such as universal guidance [3, 20] and particle-based approaches such as best-of-nn sampling and sequential Monte Carlo (SMC) [16].

Fine-tuning. Recent work [13, 17] proposes fine-tuning a diffusion model qfinetuneq_{\mathrm{finetune}} to maximize the reward Eqfinetune(x0)r(x0)\mathbb{E}_{q_{\mathrm{finetune}(x_0)}} r(x_0) without a Kullback-Leibler (KL) penalty. Domingo-Enrich et al. [ [18], Fan et al. [ [33] propose KL-regularized fine-tuning, and more recently Wallace et al. [ [19] proposes direct preference optimization for diffusion models. However, fine-tuning requires both allocating training resources and coupling a model to a specific reward function. Moreover, we show FK-IPS, with just k=3k = 3 particles, outperforms fine-tuning in several settings without requiring any kind of training.

Inference-time steering. For continuous-valued xtx_t and differentiable rewards, one can employ gradient-based steering methods such as classifier guidance [3], or more generally universal guidance [20], to define the twisted score, sθ(xt,t)+∇xtr(xt)s_\theta(x_t,t) + \nabla_{x_t}r(x_t), where sθs_\theta is the marginal score. However, the use of gradients limits steering to differentiable rewards and continuous-valued diffusion models.

FK STEERING builds on top of recent works that sample from Feynman-Kac path distributions for conditional sampling with diffusion models, either using particle-based sampling [16, 24, 34, 35, 36] or gradient-based sampling [25, 26].

In appendix § 10.§ 2, we show how TDS [16] and soft value-based decoding in diffusion models (SVDD) [29] are examples of FK interacting particle systems [22]. Our experiments demonstrate that generalizing beyond these methods to different choices of potentials, rewards, and samplers provide several improvements, such as higher reward samples.

TDS [16] uses twisted SMC (See Section 3 in Naesseth et al. [ [37]) for conditional sampling by targeting ptarget(x0∣y)∝pθ(x0)p(y∣x0)p_{\mathrm{target}}(x_0 \mid y) \propto p_\theta(x_0)p(y \mid x_0). The proposal generator for TDS uses classifier-guidance, restricting guidance to continuous state diffusion models and differentiable reward functions. In contrast, FK STEERING enables guidance with reward functions beyond differentiable likelihoods and generalizes to discrete state-spaces, including for text diffusion models.

More recently, Li et al. [ [29] propose SVDD, a derivative-free approach to guiding diffusion models for reward maximization, by targeting the distribution lim⁡λ→∞1Zpθ(x0)exp⁡(λr(x0))\lim_{\lambda\to\infty} \frac{1}{Z}p_\theta(x_0)\exp(\lambda r(x_0)). If there is a single sample x0∗x_0^* in the support of pθp_\theta that maximizes rr, then this distribution collapses to a point mass on x0∗x_0^*. SVDD uses nested importance sampling for sampling, see algorithm 5 in Naesseth et al. [ [37]. At each time step, SVDD samples kk states xtix_t^i from a diffusion model and selects one state xti∗x_t^{i^*} with the highest reward and makes kk copies of that state, therefore reducing diversity of samples x0ix_0^i.

Text Generation and SMC. The sampling procedure for traditional autoregressive language models poses a challenge for SMC approaches, since SMC typically requires estimating the reward of the full sequence given its prefix. Lew et al. [ [38] address this challenge by limiting to rewards calculated within a fixed look-ahead window. In contrast, Zhao et al. [ [32] learn intermediate twisting potentials to marginalize over the remaining elements of a particular partial sequence. In our work, we demonstrate that intermediate estimates from diffusion models, for attributes like toxicity, can be used even with off-the-shelf reward models by evaluating the reward models on intermediate denoised estimates (see Figure 5).

Figure 5. Correlation between rϕ(xt)r_{\phi}(x_t) and final state r(x0)r(x_0): Left: Correlations between r(x0)r(x_0) and r(x0=x^t)r(x_0 = \hat{x}_t) for several text-to-image diffusion models, where rr is a human preference score [13]. Right: Correlation with an off-the-shelf text toxicity classifier r(x0=x^t)r(x_0 = \hat{x}_t) and learned rewards rϕ(xt)r_{\phi}(x_t) on SSD-LM [7], a text diffusion model.

Feynman-Kac Steering of diffusion models

In this section, we present details of the Feynman-Kac steering (FK STEERING) for inference-time steering of diffusion models.

Diffusion Models

Diffusion-based generative models (DGMs) [1] are processes that are learned by reversing a forward noising process, q(xt)q(x_t). The noising process takes data x∼qdatax \sim q_{\mathrm{data}} and produces a noisy state xt∼q(xt∣x0=x)x_t \sim q(x_t \mid x_0 = x) such that at a terminal-time TT we have q(xT)=πpriorq(x_T) = \pi_{\mathrm{prior}}, where πprior\pi_{\mathrm{prior}} is the model prior. The noising process can be defined as a continuous-time Markov process [3, 39, 40, 41, 42], or discrete-time Markov chain [1, 43, 44, 45, 46]. For exposition, we focus on discrete time diffusions though the techniques are applicable to continuous time diffusions as noted below. A discrete time diffusion has a model given a context cc

pθ(xT,…,x0∣c)=πprior(xT)∏t∈[T−1,0]pθ(xt∣xt+1,c),(1)p_\theta(x_T,\ldots,x_0 \mid c)=\pi_{\mathrm{prior}}(x_T)\prod_{t\in[T-1,0]}p_\theta(x_t\mid x_{t+1},c), \tag*{(1)}

which involves iteratively sampling a path (xT,xT−1,…,x0)(x_T,x_{T-1},\ldots,x_0), where x0x_0 is the model sample. The model pθp_\theta can be trained by maximizing a lower bound on the model log-likelihood log⁡pθ(x0=x)\log p_\theta(x_0=x).

Most uses of generative models require samples with user-specified properties. In the next section, we describe a generic way to steer diffusion models towards such samples.

Steering Diffusion Models

One way to steer diffusion models is to encode user preferences in a reward model r(x0)r(x_0) and sample from a distribution that tilts the diffusion model’s generations pθ(x0)p_\theta(x_0) towards an exponential of the reward function r(x0)r(x_0):

ptarget(x0∣c)=1Zpθ(x0∣c)exp⁡(λr(x0,c)).(2)p_{\mathrm{target}}(x_0\mid c)=\frac{1}{Z}p_\theta(x_0\mid c)\exp(\lambda r(x_0,c)). \tag*{(2)}

The reward r(x0)r(x_0) can correspond to various objectives, including human preference scores [13, 14], vision-language models [15] that measure the prompt fidelity for text-to-image models, or likelihoods p(y∣x0)p(y\mid x_0) of a particular attribute yy.

One way to sample from the target distribution in eq. (2) is to generate kk particles {x0i}∼pθ(x0∣c)\{x_0^i\}\sim p_\theta(x_0\mid c) and then resample the particles based on the scores exp⁡(λ(r(x0i,c)))\exp(\lambda(r(x_0^i,c))). This procedure is known as importance sampling [47]. However, the target distribution favors samples that have higher reward, which may be rare under the model pθp_\theta. This suggests the use of simulation methods that better tilt towards rare events.

One broad class of rare-event simulation methods are FK-IPS approaches [22, 28] that use functions called potentials to tilt the transition kernels of the diffusion process to push samples towards paths that lead to higher rewards. In the next section, we describe FK STEERING, a general framework for inference-time steering of diffusion models using FK-IPS.

Feynman-Kac diffusion steering

We use FK-IPS to produce paths (xT,xT−1,…,x0)(x_T,x_{T-1},\ldots,x_0) with high-reward x0x_0 samples. FK-IPS defines a sequence of distributions pFK,t(xT,xT−1,…,xt)p_{\mathrm{FK},t}(x_T,x_{T-1},\ldots,x_t) by tilting the base distribution pθ(xT,xT−1,…,xt)p_\theta(x_T,x_{T-1},\ldots,x_t) using potentials GtG_t [22, 48]. The sequence of distributions pFK,tp_{\mathrm{FK},t} is built iteratively by tilting the transition kernels pθ(xt∣xt+1)p_\theta(x_t\mid x_{t+1}) with a potential Gt(xT,xT−1,…,xt)G_t(x_T,x_{T-1},\ldots,x_t). We start with pFK,T(xT)∝pθ(xT∣c)GT(xT,c)p_{\mathrm{FK},T}(x_T)\propto p_\theta(x_T\mid c)G_T(x_T,c) and then define the subsequent distributions as:

pFK,t(xT,…,xt∣c)=1Ztpθ(xT,xT−1,…,xt∣c){∏s=TtGs(xT,xT−1,…,xs,c)}(3)p_{\mathrm{FK},t}(x_T,\ldots,x_t\mid c)=\frac{1}{Z_t}p_\theta(x_T,x_{T-1},\ldots,x_t\mid c)\left\{\prod_{s=T}^{t}G_s(x_T,x_{T-1},\ldots,x_s,c)\right\} \tag*{(3)}
=1Ztpθ(xt∣xt+1,c)Gt(xT,…,xt,c)pθ(xT,…,xt+1∣c)∏s=Tt+1Gs(xT,…,xs,c)(4)=\frac{1}{Z_t}p_\theta(x_t\mid x_{t+1},c)G_t(x_T,\ldots,x_t,c)p_\theta(x_T,\ldots,x_{t+1}\mid c)\prod_{s=T}^{t+1}G_s(x_T,\ldots,x_s,c) \tag*{(4)}

where Zt=Epθ[∏s=TtGs]Z_t=\mathbb{E}_{p_\theta}\left[\prod_{s=T}^{t}G_s\right] is the normalization constant. The potentials GtG_t are selected to up-weight paths (xT,…,xt)(x_T,\ldots,x_t) that will ultimately yield high-reward samples x0x_0. We require that the product of the potentials GtG_t matches the exponential tilt of ptargetp_{\mathrm{target}}:

∏t=T0Gt(xT,…,xt,c)=exp⁡(λr(x0,c)).(5)\prod_{t=T}^{0}G_t(x_T,\ldots,x_t,c)=\exp(\lambda r(x_0,c)). \tag*{(5)}

Algorithm 1 Feynman-Kac Diffusion Steering

Input: Diffusion model pθ(x0:T∣c)p_\theta(x_{0:T}\mid c), reward model r(x0,c)r(x_0,c), proposal generator τ(xt∣xt+1,c)\tau(x_t\mid x_{t+1},c), potentials GtG_t, intermediate rewards rϕ(xt,c)r_\phi(x_t,c), number of particles kk

Returns: Samples {x0i}i=1k\{x_0^i\}_{i=1}^k

Sample xTi∼τ(xT∣c)x_T^i\sim\tau(x_T\mid c) for i∈[K]i\in[K]

Score, GTi=GT(xTi,c)G_T^i=G_T(x_T^i,c) for i∈[K]i\in[K]

for t∈{T,…,1}t\in\{T,\ldots,1\} do

Resample: Sample kk indices ati∼Multinomial⁡(xti,Gti)a_t^i\sim\operatorname{Multinomial}(x_t^i,G_t^i) and let xti=xtatix_t^i=x_t^{a_t^i} for i∈[K]i\in[K]

Propose: Sample xt−1i∼τ(xt−1∣xti,c)x_{t-1}^i\sim\tau(x_{t-1}\mid x_t^i,c) for i∈[K]i\in[K]

Re-weight: Compute weight Gt−1iG_{t-1}^i for i∈[K]i\in[K]:

Gt−1i=pθ(xt−1i∣xti,c)τ(xt−1i∣xti,c)Gt−1(xTi,…,xt−1i,c)G_{t-1}^i=\frac{p_\theta(x_{t-1}^i\mid x_t^i,c)}{\tau(x_{t-1}^i\mid x_t^i,c)}G_{t-1}(x_T^i,\ldots,x_{t-1}^i,c)

end for

Output: return samples {x0i}\{x_0^i\}

This choice ensures that pFK,0(xT,…,x0∣c)∝pθ(xT,…,x0∣c)exp⁡(λr(x0,c))p_{\mathrm{FK},0}(x_T,\ldots,x_0\mid c)\propto p_\theta(x_T,\ldots,x_0\mid c)\exp(\lambda r(x_0,c)); that is, sampling from x0∼pFK,0x_0\sim p_{\mathrm{FK},0} samples from the target that tilts towards higher rewards. Potential functions that satisfy this constraint are not unique.

Sampling from pFK,0p_{\mathrm{FK},0}. Direct sampling from pFK,0p_{\mathrm{FK},0} is intractable. However, targeting the intermediate distributions pFK,tp_{\mathrm{FK},t} supports sampling of the distribution pFK,0p_{\mathrm{FK},0} through the use of particle-based methods, such as SMC [22, 49], nested IS (see alg. 5 in Naesseth et al. [ [37]), diffusion Monte Carlo (DMC) [28]. SMC generates kk particles using a proposal generator τ(xt∣xt+1,…,xT,c)\tau(x_t\mid x_{t+1},\ldots,x_T,c) and at each transition step xt+1→xtx_{t+1}\to x_t scores the particles using the potential and the transition kernel importance weights:

Gti=pFK,t(xT,…,xt+1,xt∣c)pFK,t+1(xT,…,xt+1∣c)τ(xt∣xt+1,…,xT∣c)(6)G_t^i=\frac{p_{\mathrm{FK},t}(x_T,\ldots,x_{t+1},x_t\mid c)}{p_{\mathrm{FK},t+1}(x_T,\ldots,x_{t+1}\mid c)\tau(x_t\mid x_{t+1},\ldots,x_T\mid c)} \tag*{(6)}
=  Gt(xTi,…,xt+1i,xti,c)pθ(xti∣xt+1i,c)τ(xti∣xt+1i,…,xTi,c).(7)=\;G_t(x_T^i,\ldots,x_{t+1}^i,x_t^i,c)\frac{p_\theta(x_t^i\mid x_{t+1}^i,c)}{\tau(x_t^i\mid x_{t+1}^i,\ldots,x_T^i,c)}. \tag*{(7)}

The particles xtix_t^i are then resampled based on the scores GtiG_t^i. See Figure 1 for a visualization of the method and algorithm 1 for details. Particle-based approximations are consistent, that is as k→∞k\to\infty, the empirical distribution over the particles x0ix_0^i converges to ptarget(x0)p_{\mathrm{target}}(x_0), see theorem 3.19 in Del Moral and Miclo [ [50]. Next, we discuss different choices of the potentials GtG_t and proposal generators τ\tau.

Choosing the proposal generator τ\tau. For the proposal generator τ\tau, the simplest choice is to sample from the diffusion model’s transition kernel pθ(xt∣xt+1,c)p_\theta(x_t\mid x_{t+1},c). Alternatively, another choice is a transition kernel that tilts towards samples with high rewards, such as gradient-based guidance. We discuss choices in appendix § 10.§ 1.

Choosing the potential GtG_t. One choice of potentials is Gt=1G_t=1 for t≥1t\geq1 and G0=exp⁡(λ(r(x0,c)))G_0=\exp(\lambda(r(x_0,c))), which leads to importance sampling. However, importance sampling can require many particles to find high rewards [21] and does not take into account how a particle is likely to score during the generation process. To up-weight paths that yield high-reward samples, FK STEERING uses potentials that score particles using intermediate rewards rϕ(xt,c)r_\phi(x_t,c):

  • Difference: Gt(xt,xt+1,c)=exp⁡(λ(rϕ(xt,c)−rϕ(xt+1,c)))G_t(x_t,x_{t+1},c)=\exp(\lambda(r_\phi(x_t,c)-r_\phi(x_{t+1},c))) and GT=1G_T=1, similar to [16], prefers particles that have increasing rewards.

  • Max: Gt(xT,…,xt,c)=exp⁡(λmax⁡s=tTrϕ(xs,c))G_t(x_T,\ldots,x_t,c)=\exp(\lambda\max_{s=t}^{T}r_\phi(x_s,c)) and G0=exp⁡(λr(x0,c))(∏t=1TGt)−1G_0=\exp(\lambda r(x_0,c))(\prod_{t=1}^{T}G_t)^{-1} prefers particles that have the highest rewards.

Experiments

  • SUM: Gt(xT,…,xt)=exp⁡(λ∑s=tTrϕ(xs,c))G_t(\mathbf{x}_T,\ldots,\mathbf{x}_t)=\exp\left(\lambda\sum_{s=t}^{T}r_\phi(\mathbf{x}_s,\mathbf{c})\right) and G0=exp⁡(λr(x0,c))(∏t=1TGt)−1G_0=\exp(\lambda r(\mathbf{x}_0,\mathbf{c}))\left(\prod_{t=1}^{T}G_t\right)^{-1} selects particles that have the highest accumulated rewards.

Any choice of potentials that satisfies eq. (5) produce a consistent approximation of ptarget(x0)p_{\mathrm{target}}(\mathbf{x}_0). However, the number of particles required to produce high-reward samples depends on the choice of the potential. For instance, if the reward r(x0)r(\mathbf{x}_0) is bounded, then using the potential exp⁡(λ(rϕ(xt,c)−rϕ(xt+1,c)))\exp(\lambda(r_\phi(\mathbf{x}_t,\mathbf{c})-r_\phi(\mathbf{x}_{t+1},\mathbf{c}))) assigns low scores to particles that reach the maximum reward early in generation. In this setting, alternatives like the MAX potential may offer benefits.

Interval Resampling. For a typical diffusion process, the change between xt\mathbf{x}_t and xt+1\mathbf{x}_{t+1} is not substantial. Therefore, we can select potentials GtG_t such that we only resample at a few steps. We define a resampling schedule R={tr,…,t1}R=\{t_r,\ldots,t_1\}, where t1=0t_1=0. For t∉Rt\notin R, Gt=1G_t=1 and for tj∈Rt_j\in R, GtG_t is equal to a non-uniform potential, such as the max potential. This type of interval resampling encourages exploration and reduces sampling time and compute requirements.

Choosing intermediate rewards rϕ(xt,c)r_\phi(\mathbf{x}_t,\mathbf{c}). Any choice of the intermediate rewards rϕ(xt,c)r_\phi(\mathbf{x}_t,\mathbf{c}) yields consistent particle-based approximations. In this section, we discuss some choices of rϕr_\phi.

The ideal rewards for the intermediate state xt\mathbf{x}_t would require knowing the distribution of rewards of x0\mathbf{x}_0 generated from the state xt\mathbf{x}_t: pθ(r(x0)∣xt,c)p_\theta(r(\mathbf{x}_0)\mid\mathbf{x}_t,\mathbf{c}). With this distribution, reward functions rϕr_\phi could be chosen to yield high expected reward when reward variance is low, or based on the 10th percentile of the reward distribution to ensure good worst-case quality. Producing this distribution of rewards requires training with samples from the model. We will describe a few options that have different trade-offs in compute versus knowledge provided about the rewards for the sample x0\mathbf{x}_0.

  • Rewards at expected x0\mathbf{x}_0. Similar to Song et al. [ [3], Wu et al. [ [16], Bansal et al. [ [20], Li et al. [ [29], intermediate rewards can be defined by evaluating an off-the-shelf reward function at the diffusion model’s approximation of the expected sample x0\mathbf{x}_0 given the current state xt\mathbf{x}_t and context c\mathbf{c}: x^t≈Epθ(x0∣xt)[x0∣xt]\hat{\mathbf{x}}_t\approx\mathbb{E}_{p_\theta(\mathbf{x}_0\mid\mathbf{x}_t)}[\mathbf{x}_0\mid\mathbf{x}_t]. With this choice, the intermediate rewards are rϕ(xt,c)=r(x0=x^t,c)r_\phi(\mathbf{x}_t,\mathbf{c})=r(\mathbf{x}_0=\hat{\mathbf{x}}_t,\mathbf{c}).

  • Many-sample rϕr_\phi. Diffusion models provide a means to sample pθ(x0∣xt,c)p_\theta(\mathbf{x}_0\mid\mathbf{x}_t,\mathbf{c}). During inference, for each particle xti,j\mathbf{x}_t^{i,j}, we sample NN samples x0i,j∼pθ(x0∣xti,j,c)\mathbf{x}_0^{i,j}\sim p_\theta(\mathbf{x}_0\mid\mathbf{x}_t^{i,j},\mathbf{c}) and then use rϕ(xti,j,c)=log⁡1N∑j=1Nexp⁡(r(x0i,j,c))r_\phi(\mathbf{x}_t^{i,j},\mathbf{c})=\log\frac{1}{N}\sum_{j=1}^{N}\exp(r(\mathbf{x}_0^{i,j},\mathbf{c})) to summarize the empirical distribution of rewards.

  • Learned rϕr_\phi. When sampling from pθ(x0∣xt,c)p_\theta(\mathbf{x}_0\mid\mathbf{x}_t,\mathbf{c}) is expensive, we can leverage the fact that pθp_\theta is trained to approximate the inference process qq Sohl-Dickstein et al., [1]; Song et al. [3]. Because of this approximation, data samples can be used to train intermediate reward models rϕr_\phi. For instance, when r(x0)r(\mathbf{x}_0) is a classifier pθ(y∣x0)p_\theta(y\mid\mathbf{x}_0), then similar to Nichol et al. [ [51] we can train a classifier pϕ(y∣xt)p_\phi(y\mid\mathbf{x}_t). For more general rewards, we can use:

arg⁡min⁡ϕ Et∼U[0,T]Eqdata(x0∣c)q(xt∣x0)[∥aϕ(xt,c)−exp⁡(r(x0,c))∥22](8)\underset{\phi}{\arg\min}\ \mathbb{E}_{t\sim U[0,T]}\mathbb{E}_{q_{\mathrm{data}}(\mathbf{x}_0\mid\mathbf{c})q(\mathbf{x}_t\mid\mathbf{x}_0)}\left[\left\lVert a_\phi(\mathbf{x}_t,\mathbf{c})-\exp(r(\mathbf{x}_0,\mathbf{c}))\right\rVert_2^2\right] \tag*{(8)}

and define rϕ=log⁡aϕr_\phi=\log a_\phi. When pθ=qp_\theta=q, these objectives learn the reward, rϕ=log⁡Epθ(x0∣xt,c)[exp⁡(r(x0,c))]r_\phi=\log\mathbb{E}_{p_\theta(\mathbf{x}_0\mid\mathbf{x}_t,\mathbf{c})}[\exp(r(\mathbf{x}_0,\mathbf{c}))]. This choice of reward is used to define the potential GtG_t that leads to the local transitions that minimize the variance of the potential at each step (see theorem 10.1 in Chopin et al., [48]).

Continuous-time diffusions. While the presentation above is for discrete-time models, we note that FK STEERING can also be used to steer continuous-time diffusion models Song et al., [3]; Singhal et al. [40]. Sampling from continuous-time diffusion models is done using numerical methods such as Euler-Maruyama Särkkä and Solin, [52]. Numerical sampling methods involve defining a discretized grid {1,1−Δt,…,0}\{1,1-\Delta t,\ldots,0\} and then sampling from the transition kernel pθ(xt∣xt+Δt,c)p_\theta(\mathbf{x}_t\mid\mathbf{x}_{t+\Delta t},\mathbf{c}). Therefore, similar to discrete-time models, we can apply FK STEERING by tilting the transition kernels with potentials Gt(x1,x1−Δt,…,xt)G_t(\mathbf{x}_1,\mathbf{x}_{1-\Delta t},\ldots,\mathbf{x}_t) for t∈{0,Δt,…,1}t\in\{0,\Delta t,\ldots,1\}.

Experiments

To evaluate the efficacy of FK STEERING we conduct the following experiments:

  • FK STEERING for sample quality: This experiment steers text-to-image diffusion models and text diffusion models with off-the-shelf rewards that measure sample quality.

    • In the text-to-image setting, we run FK STEERING with a human preference reward model, ImageReward [13]. We evaluate on the heldout GenEval benchmark, a popular prompt fidelity evaluation.2

    • For text diffusion models, we use perplexity computed using GPT2 [53], a simple trigram language model [54], and linguistic acceptability classifier [55] for rewards.

  • Studying potential choices in FK STEERING: In this experiment, we explore the effect of using different choices of potentials on the rewards achieved by the samples x0ix_0^i.

  • Studying different choices of intermediate rewards: We examine the effectiveness of using different intermediate rewards with FK STEERING on control of two types of rare attribute:

    • For text diffusion models, we consider control of text toxicity, which occurs in 1% of base model samples.

    • For image diffusion models, we evaluate control of ImageNet class. There are 1000 classes in the dataset. In this experiment, we also use gradient-based guidance to tilt the transition kernel.

method for sample quality

Text-to-Image Diffusion Models. This experiment uses the stable diffusion [30, 31] family of text-to-image diffusion models pθ(x0∣c)p_\theta(x_0 \mid c), where cc is the text prompt. These models cover a range of model architectures [57] (Nichol and Dhariwal 2021) and inference processes [2, 58], see Table 8 for parameter counts and timings. As the reward we use the ImageReward human preference score model [13]. For the intermediate rewards, we evaluate the off-the-shelf reward model on the denoised state, rϕ(xt)=r(x0=x^t)r_\phi(x_t) = r(x_0 = \hat{x}_t) where x^t\hat{x}_t is the model’s approximation of Epθ[x0∣xt]\mathbb{E}_{p_\theta}[x_0 \mid x_t].

ModelParamsBase(k=1k = 1)Base(k=4k = 4)FK(k=4k = 4)FK(k=4k = 4, parallel)
SD v1.4/v1.5860M2.4s7.3s8.1s5.0s
SD v2.1865M4.6s15.6s17.4s9.1s
SDXL2.6B11.5s42.3s43.5s21.7s

Table 8. Parameter counts and timing. In this table, we provide inference timing for text-to-image diffusion models with FK STEERING. We include results for FK STEERING on a single NVIDIA-A100 GPU and a two-device parallel implementation.

For the proposal generator τ\tau, we use the base model itself. For sampling from the base model, we use classifier-free guidance [59] with guidance scale set to 7.5, the default choice from Hugging Face3, alongside the DDIM sampler [60] with η=1\eta= 1 and T=100T = 100 time-steps. We use λ=10\lambda= 10, k=4k = 4 and the resampling schedule [0,20,40,60,80][0, 20, 40, 60, 80] with the potential Gt(xT,…,xt)=exp⁡(λmax⁡s=tTrϕ(xs))G_t(x_T, \ldots, x_t) = \exp(\lambda\max_{s=t}^{T} r_\phi(x_s)), where t=0t = 0 is the terminal step.

As a benchmark for FK STEERING, we compare against best-of-N (BoN). Additionally, we consider publicly available models fine-tuned for prompt alignment and aesthetic quality. We consider DPO4 fine-tuned models, SD v1.5 and SDXL, [19, 61] and an RL fine-tuned SD v1.4 [17]5 with a vision-language model, LLAVA [15], as the reward for prompt alignment. We also explore FK STEERING the fine-tuned models.

We measure prompt alignment with the GenEval benchmark6 [11] and report ImageReward7 [13] and HPS [14] scores to measure aesthetic quality. For results with different sampling schedules and λ\lambda values, see appendix § 8.

In Table 1, we measure prompt fidelity and aesthetic quality of samples generated by FK STEERING with k=4k = 4. We measure prompt alignment by GenEVAL and aesthetic quality using ImageReward [13] and HPS scores. In Table 1 we report the performance of the highest reward particle generated by FK STEERING, and in Figure 4, we report average particle performance. We observe:

ModelSampler(λ\lambda, kk)GenEVAL* ↑\uparrowIR† ↑\uparrowHPS† ↑\uparrow
SD v1.4k=1k = 10.44080.2340.245
SD v1.4BoN(k=4k = 4)0.54600.8000.256
SD v1.4-DDPOk=1k = 10.43710.2630.241
SD v1.4FK(λ=10\lambda= 10, k=4k = 4)0.54920.9270.263
SD v1.5k=1k = 10.44830.1870.245
SD v1.5BoN(k=4k = 4)0.52390.7370.265
SD v1.5-DPOk=1k = 10.46710.3430.255
SD v1.5FK(λ=10\lambda= 10, k=4k = 4)0.54630.8980.263
SD v2.1k=1k = 10.51040.3720.253
SD v2.1BoN(k=4k = 4)0.61720.8880.263
SD v2.1FK(λ=10\lambda= 10, k=3k = 3)0.59870.8640.265
SD v2.1FK(λ=10\lambda= 10, k=4k = 4)0.62891.0060.268
SDXLk=1k = 10.55710.8710.289
SDXLBoN(k=4k = 4)0.63471.2360.296
SDXL-DPOk=1k = 10.58110.8590.296
SDXLFK(λ=10\lambda= 10, k=4k = 4)0.64891.2980.302

Table 1. Effect of FK STEERING prompt fidelity and human preference scores. GenEval scores * are computed using the GenEval prompts and ImageReward† and ImageReward and HPS scores† are computed on prompts provided by the ImageReward paper. As a baseline, we compare against Best-of-N with 4 independent samples. Across all models, FK STEERING improves both prompt fidelity as well as human preference alignment scores, beating BoN and fine-tuning. Interestingly, we note that BoN outperforms fine-tuning as well, showing the benefits of inference-time scaling.

Figure 4. Effect of scaling the number of particles. Left: GENEVAL scores for FK STEERING using IMAGEREWARD, average particle performance. Dashed lines indicate performance of fine-tuned baselines. Middle: Corresponding IMAGEREWARD scores. Right: Distribution of IMAGEREWARD scores for samples from SDv2.1 (0.8B) with and without FK STEERING, compared with SDXL (2.6B).

  • FK STEERING smaller models outperforms larger models. With k=4k = 4 FK STEERING SDv2.1 outperforms SDXL and its DPO [19] fine-tuned version, on GenEval scores and aesthetic quality with less sampling time: 11.5s versus 9.1s. See Figure 3 for samples.

FK steering improves prompt fidelity and sample quality

Figure 3. FK STEERING improves prompt fidelity and sample quality. In the first row, we plot an independent sample from SDXL [30]. In the middle and bottom rows, we plot the highest reward sample using FK STEERING with SDXL and SDv2.1 [31], here we use k=4k = 4 particles. We use a human preference score, ImageReward [13], as the reward function and generate samples from the base models, without any gradients from the reward. We observe that FK STEERING both SDXL and SDv2.1 improves their prompt fidelity and sample quality compared to the first row. Prompts are selected from the GenEval benchmark prompt set.

  • FK STEERING the base model outperforms fine-tuning the base model. FK STEERING with as few as k=4k = 4 particles can outperform fine-tuned models on both prompt fidelity and human preference alignment. Moreover, Figure 2 shows that the base model with just k=2k = 2 particles has a higher GenEval score than the DPO and DDPO fine-tuned models.

  • FK STEERING further improves fine-tuned models. In Table 2, we show that steering fine-tuned models with FK STEERING improves performance even further. For instance, the GenEval score of SDXL-DPO increases from 0.58 to 0.65, and similarly for SDv1.5, the GenEval score increases from 0.46 to 0.56.

ModelSampler(λ\lambda, kk)GenEval ↑\uparrowIR ↑\uparrowHPS ↑\uparrow
SD v1.5-DPOk=1k = 10.46710.3430.255
SD v1.5-DPOFK(λ=10\lambda= 10, k=4k = 4)0.57510.8850.276
SDXL-DPOk=1k = 10.58110.8590.296
SDXL-DPOFK(λ=10\lambda= 10, k=4k = 4)0.67551.1980.317

Table 2. Steering fine-tuned models. Here we sample from qfinetune(x0∣c)exp⁡(r(x0,c))q_{\mathrm{finetune}}(x_0 \mid\mathbf{c})\exp(r(x_0,\mathbf{c})) using FK STEERING, where qfinetuneq_{\mathrm{finetune}} are DPO fine-tuned models [19]. Note that all metrics are improved with FK STEERING.

  • FK STEERING can improve gradient-guidance. In Table 3, we show that FK STEERING SDv1.5 with k=4k = 4 outperforms gradient-guidance using the ImageReward model, using less sampling time. We note that using gradient-guidance for the proposal generator can improve performance even further, at the cost of increased sampling time and more compute.

ModelSamplerGenEvalIRHPSTime
SDv1.5k=1k = 10.440.1870.2452.4s
SDv1.5∇(k=1)\nabla(k = 1)0.450.6680.24520s
SDv1.5FK(k=4k = 4)0.540.8980.2638.1s
SDv1.5FK(∇\nabla, k=4k = 4)0.561.2900.26855s

Table 3. Comparison against gradient guidance. Here we note that FK STEERING with the model as the proposal generator outperforms gradient guidance, with faster sampling. We also note that FK STEERING can benefit from gradient guidance, albeit at the cost of more compute and sampling time.

  • Effect of scaling the number of particles. In Figure 4, we observe the effect of scaling the number of particles on prompt fidelity and human preference alignment metrics. We note that scaling the number of particles improves the prompt fidelity and human preference alignment scores of all particles for all models.

Text Diffusion Models. Next, we investigate steering with FK STEERING to improve the sample quality of text diffusion models, which generally underperform traditional autoregressive models on fluency metrics like perplexity [6, 9, 10, 44]. We consider three reward functions for improving text quality: perplexity computed with a simple trigram language model8, a classifier [55] trained on the Corpus of Linguistic Acceptability (CoLA) dataset [62], and perplexity computed by GPT2 [63]. For all choices of reward models, we define the intermediate rewards using rϕ(xt)=r(x0=x^t)r_{\phi}(x_t) = r(x_0 = \hat{x}_t) and use the potential Gt=exp⁡(λ(rϕ(xt)−rϕ(xt+1)))G_t = \exp(\lambda(r_{\phi}(x_t) - r_{\phi}(x_{t+1})))

We consider two base text diffusion models: SSD-LM [7] and MDLM [44] and use these models as the proposal generator τ\tau. SSD-LM is a continuous space diffusion model trained on noised word logits, while MDLM is a discrete diffusion model.

For FK STEERING, we resample 50 times during the inference process (every 10 steps for SSD-LM and every 20 for MDLM). For all models we use λ=10.0\lambda= 10.0, and return the highest scoring sample at t=0t = 0. Following Han et al. [ [7], we generate 20 continuations of length 50 using their 15 controllable generation prompts. In addition to FK STEERING, we evaluate base model performance and best-of-nn. As a baseline to compare quality improvements from scaling inference steps [8, 44], we also include results for SSD-LM with T=5000T = 5000 (versus T=500T = 500). We additionally include results for GPT2-Medium [63] as an auto-regressive baseline. We evaluate all methods using perplexity computed with GPT2-XL [63], CoLA acceptability, and distinct uni/bi/trigrams (Dist-1/2/3) [7, 64]. Additional details on our text experiments are included in B.

Table 4 contains our text sample quality evaluation results. We observe:

Model + Sampler(rr)kkPPL (GPT-XL) ↓\downarrowCoLA ↑\uparrowDist-1/2/3 ↑\uparrow
GPT2-medium114.187.654/89/94
SSD-LM123.268.346/83/92
SSD-LMT×10_{T\times10}118.876.646/81/90
FK(GPT2)411.080.040/73/86
FK(Trigram)414.177.441/76/88
FK(CoLA)417.4<u>95.7</u>45/80/90
BoN(GPT2)413.675.642/77/88
BoN(Trigram)415.971.943/78/89
BoN(CoLA)419.293.846/81/91
BoN(GPT2)8<u>11.2</u>80.341/74/86
BoN(Trigram)813.976.842/77/89
BoN(CoLA)818.497.246/82/91
MDLM185.328.957/91/94
FK(GPT2)449.039.852/86/92
FK(Trigram)440.337.050/87/93
FK(CoLA)473.6<u>69.8</u>59/88/91
BoN(GPT2)455.532.954/88/93
BoN(Trigram)452.130.153/89/93
BoN(CoLA)471.459.457/90/94
BoN(GPT2)846.937.253/87/92
BoN(Trigram)8<u>45.9</u>35.452/88/93
BoN(CoLA)868.273.158/91/94

Table 4. Table 4: Text sample quality results metrics. We sample texts of length 50 from all models and score perplexity with GPT2-XL and CoLA acceptability. Results are averaged over three seeds. Both SSD-LM and GPT-medium have 355 million parameters. MDLM is a smaller model with 170 million parameters.

  • FK STEERING significantly improves the perplexity and CoLA scores of both SSD-LM and MDLM. For all reward functions, FK STEERING (k=4k = 4) outperforms best-of-nn (n=4n = 4) on the corresponding target metric (perplexity or CoLA). For MDLM, trigram steering dramatically improves perplexity (37.2 vs 79.2), but is less effective at improving CoLA (35.3 vs 30.0).

  • FK STEERING outperforms best-of-nn. For all experiments, FK STEERING outperforms best-of-nn when

using the same number of particles. Notably, FK STEERING SSD-LM outperforms best-of-n with twice as many particles. We also note that FK STEERING on SSD-LM with T=500T = 500 improves on SSD-LM with 10× inference steps (T=5000T = 5000) for all fluency metrics.

Overall, our results demonstrate that FK STEERING with off-the-shelf rewards can enable sampling lower-perplexity, more linguistically acceptable text from diffusion models.

Studying different choices of potentials

In the previous section, we used two different choices of potentials: the max potential, exp⁡(λmax⁡s≥trϕ(xs))\exp(\lambda\max_{s \ge t} r_\phi(x_s)), for the text-to-image experiments and the difference potential, exp⁡(λ(rϕ(xt)−rϕ(xt+1)))\exp(\lambda(r_\phi(x_t)-r_\phi(x_{t+1}))), for the text quality experiment. However, as discussed in § 3, the choice of potential is not unique. This experiment studies different choices of the potential for steering text-to-image diffusion models. Similar to the previous section, we steer the stable diffusion text-to-image models with ImageReward, using the max, difference and the sum potential. For all potentials, we use λ=10\lambda= 10 with the [0,20,40,60,80][0, 20, 40, 60, 80] interval sampling schedule. We also note that ImageReward is bounded between [−2,2][-2, 2].

In Table 5, for all models considered, the prompt fidelity scores using the max potential are higher compared to the difference and sum potentials. The sum potential is worse on the smaller models. When limiting to the difference and max potentials, the difference potential can assign low scores for particles that achieve the maximum reward of 2 early in generation (because no further reward increase can be made after reaching the maximum). However, we observe that for the same value of λ\lambda and same number of particles, the max potential can lead to lower particle diversity than the difference potential (see appendix § 11 for samples). This is because resampling at intermediate steps with the max potential favors higher scoring particles more so than the difference potential.

PotentialkkSDv1.4SDv1.5SDv2.1SDXL
Max40.5400.5400.6160.633
Sum40.4960.4990.5690.613
Difference40.5250.5260.5780.603
Max80.5690.5610.6350.648
Sum80.5320.5170.5880.634
Difference80.5660.5530.6150.640

Table 5. Effect of different potentials on GenEval scores. Here we plot the average GenEval prompt fidelity score, averaged over all particles. Using the max potential outperforms the difference potential and the sum potential.

Studying different choices of rewards in method

In this experiment, we demonstrate the efficacy of FK STEERING for controllable generation, where we generate samples with attributes such as (a) toxic text for text diffusion models and (b) class-conditional image generation with 1000 classes in the dataset. In these experiments, we also compare against classifier-guidance. Controlling Text Toxicity. We consider the task of red-teaming toxicity, a rare attribute identified in only 1% of base SSD-LM samples and 0.3% of MDLM samples. Toxicity presents an undesirable behavior for language models. Here, we examine whether FK STEERING enables examining rare but dangerous model behavior, a critical factor considered before deploying systems [32]. In this experiment, we run FK STEERING with the base text diffusion models with SSD-LM [7] and MDLM models. We use the base model as the proposal generator. As a baseline, we compare against gradient guidance for SSD-LM. For reward, we use a popular toxicity classifier [65].10

In this experiment, we explore the effect of different choices of intermediate rewards:

  • For SSD-LM, we consider two choices: (1) the reward model evaluated at the denoised state and (2) the learned reward rϕr_\phi, trained using real data samples.

  • For MDLM, we use multiple samples x0i,j∼pθ(x0i,j∣xti)x_0^{i,j} \sim p_\theta(x_0^{i,j} \mid x_t^i) to compute the reward rϕ=log⁡1N∑j=1Nexp⁡(r(x0i,j))r_\phi= \log\frac{1}{N}\sum_{j=1}^{N}\exp(r(x_0^{i,j})) with (1) N=4N = 4 samples and (2) N=16N = 16 samples.

The sampling hyper-parameters and prompts are similar to section 4.1. In our evaluation, we also include results from an additional holdout toxicity classifier, trained on a multilingual mixture of toxicity datasets [66].11 Details are included in appendix § 9.

In Table 6, we observe the following:

Model + SamplerToxic ↑\uparrowToxic (Holdout) ↑\uparrowPPL (GPT2-XL) ↓\downarrowDist-1/2/3 ↑\uparrow
SSD-LM0.4%1.2%23.246/83/92
SSD-LM (∇\nabla guidance)22.3%22.6%40.353/89/94
MDLM0.3%1.9%85.357/91/94
SSD-LM (no gradients)
BoN(4)1.6%4.8%21.946/81/91
BoN(8)5.0%8.1%23.046/81/91
FK(k=4k = 4)8.4%14.0%22.545/81/91
FK(k=4k = 4, learned rϕr_{\phi})15.2%19.6%26.345/83/91
FK(k=8k = 8)25.0%29.7%23.945/81/91
FK(k=8k = 8, learned rϕr_{\phi})39.0%38.0%26.945/83/91
MDLM (discrete, no gradients)
BoN(4)2.2%6.7%83.857/90/93
BoN(8)3.7%10.8%84.657/90/93
FK(k=4k = 4)23.0%29.0%81.056/90/93
FK(k=4k = 4, many rϕr_{\phi})37.0%40.2%83.057/90/92
FK(k=8k = 8)53.4%48.3%74.356/89/92
FK(k=8k = 8, many rϕr_{\phi})64.7%51.7%82.957/89/92

Table 6. Toxicity results. We evaluate the toxicity of the generated samples with (a) the classifier used for steering and (b) a separate holdout classifier, we also report GPT2-XL perplexity. Results are averaged over three seeds.

  • Learned rewards improves controllability: With 8 particles, FK STEERING SSD-LM outperforms gradient guidance on fraction of generations labeled toxic (29.7% vs 22.6%) by the holdout classifier, and dramatically improves on perplexity (23.9 vs 40.3). Using improved intermediate rewards, learned

from real data, improves this fraction even further to 39%. Additionally, in table 11, we show that using learned reward gradient guidance with SS-LM with FK STEERING improves performance even further to 56.3%.

  • Using many-sample rϕr_{\phi} improves controllability: FK STEERING MDLM with k=8k = 8 achieves an even higher fraction of text labeled toxic 48.3%. FK STEERING outperforms best-of-nn sampling with both 4, 8 particles. Using improved intermediate rewards for MDLM more samples for intermediate rewards improves performance even further to 51.7% (64.7% on the guidance reward model).

Class-Conditional Image Generation. In this experiment, we steer a marginal diffusion model pθ(x0)p_{\theta}(x_0) to produce samples from one of 1000 different classes. Similar to Wu et al. [ [16], the reward is r(x0,y)=log⁡pθ(y∣x0)r(x_0, y) = \log p_{\theta}(y \mid x_0) and use gradient guidance for the proposal distribution.

We compare two potentials, the max potential and the difference potentials, along with two different reward models: one that uses the denoised state r(x0=x^t,y)r(x_0 = \hat{x}_t, y) and one that is trained on noisy states xt∼q(xt∣x0)x_t \sim q(x_t \mid x_0) where x0∼qdatax_0 \sim q_{\mathrm{data}} Nichol et al., [51]. This experiment uses pre-trained marginal diffusion model and classifiers from Nichol and Dhariwal [2021] and generate 256 × 256 resolution images. In

In this experiment, we explore the effect of two choices of rewards, learned and the reward evaluated at the denoised state [16]. We also explore the effect of different choices of potentials, the difference and the max potential. We observe that learning the reward improves performance significantly.

Table 7, we observe that learning rϕr_{\phi}, for both gradient guidance and potential computation, provides significant improvements over the reward evaluated at the denoised state.

Samplerrϕ(xt)r_{\phi}(x_t)Potentialkkp(y∣x0)p(y \mid x_0) Mean (Max)
FK STEERING (TDS [16])r(x0=x^t)r(x_0 = \hat{x}_t)Diff.440.59 (0.72)
FK STEERINGr(x0=x^t)r(x_0 = \hat{x}_t)Max440.65 (0.70)
FK STEERINGLearnedDiff.440.88 (0.94)
FK STEERINGLearnedMax440.88 (0.96)

Table 7. ImageNet class-conditional probabilities with different choices of rewards and potentials.

Better Rewards vs. More Particles. In the previous section, we observed that for the same number of particles, using either learned rewards or multiple samples improves performance. For instance, using k=8k = 8 with SSDLM without learning the intermediate rewards gets a toxicity rate of 25%, higher than the learned reward’s toxicity rate of 22.3%; using the learned rewards improves the toxicity rate to 39%. However, these rewards come with an added computational cost, either in training or more evaluations of the reward model, presenting a trade-off in terms of using extra computational resources either for more particles or for better intermediate rewards.

Conclusion

We introduce Feynman-Kac steering, a novel and efficient approach to inference-time steering of diffusion modeling. The method offers an extensible and scalable approach for improving diffusion models. FK STEERING flexibly steers diffusion models using Feynman-Kac interacting particle systems, a rare-event simulation technique. Our experiments demonstrate that FK STEERING can boost both the sample quality and controllability of image and text diffusion models, outperforming fine-tuning and other inference-time approaches. FK STEERING can be used in a “plug-and-play" fashion to improve diffusion models on various downstream tasks. We observe that using (a) either the difference or the max potential with intermediate rewards defined at the denoised state and (b) the base diffusion model as the proposal generator improves performance significantly. For instance, we find FK STEERING smaller diffusion models, with off-the-shelf rewards, outperforms larger models, while using less compute. However, we find that using learned rewards or rewards with many samples improves performance even further. Therefore, by validating different choices, such as potentials, rewards and samplers, a user can optimize performance for their task.

In our experiments, we find that scaling the number of particles is a natural mechanism for improving the performance of diffusion models. Notably, in our text-to-image experiments, even naive best-of-n with 4 particles outperforms fine-tuned models and improves prompt fidelity and aesthetic quality. FK STEERING improves on best-of-n by resampling using intermediate rewards during inference. We explore several choices for these intermediate rewards, which present several trade-offs, including sample diversity versus high rewards and spending more compute for better intermediate rewards.

Promising future directions include exploring the value of varying the numbers of particles at inference time, either by having a variable budget or allocating compute adaptively. For applications like protein design, dynamically assigning greater numbers of particles to promising regions could enable generating a large number of diverse candidates. We also note that FK STEERING can also improve preference learning algorithms [Zhang and Ranganath] by improving the sample generator. Additionally, we believe it is crucial to better understand the limits of inference-time particle scaling and the corresponding compute-performance trade-offs.

A limitation of FK STEERING and other inference scaling approaches is their reliance on the availability of strong reward functions. Therefore, advancing automated evaluation and reward modeling remains a critical area of research and can unlock further improvements for these methods.

Societal impact

Controllable generation methods such as FK STEERING can be applied to align language models with human preferences, including to improve their personalization or safety. Additionally, we show that FK STEERING can be used for automated red-teaming, which can inform model deployment. We recognize that any such method for controllable generation can be used to generate harmful samples by malicious actors. However, FK STEERING enables the research community to better understand properties of generative models and make them safer, which we believe will ultimately outweigh these harms.

Acknowledgments

The authors would like to acknowledge Stefan Andreas Baumann, Yunfan Zhang, Anshuk Uppal, Mark Goldstein, and Eric Horvitz for their valuable feedback.

This work was partly supported by the NIH/NHLBI Award R01HL148248, NSF Award 1922658 NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Science, NSF GAREER Award 2145542, ONR N00014-23-1-2634, and Apple. Additional support was provided by a Fellowship from the Columbia Center of AI Technology. This work was also supported by IITP with a grant funded by the MSIT of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research.

Text to Image Experiments

In this section, we explore the effect of λ\lambda and the resampling schedule on particle diversity for text-to-image generation. Similar to Domingo-Enrich et al. [ [18], we diversity of generations using the CLIP Radford et al., [68] encoder fθf_\theta, so given kk {x0i}i=1k\{x_0^i\}_{i=1}^k particles we measure:

CLIP-Div⁡({x0i}i=1k):=∑i=1k∑j=ik2k(k−1)∥fθ(x0i)−fθ(x0j)∥22.(9)\operatorname{CLIP\text{-}Div}\left(\{x_0^i\}_{i=1}^k\right) := \sum_{i=1}^{k}\sum_{j=i}^{k}\frac{2}{k(k-1)}\left\lVert f_\theta(x_0^i)-f_\theta(x_0^j)\right\rVert_2^2. \tag*{(9)}

Similar to section 4.1, we use the stable diffusion text-to-image models Rombach et al., [31] with the ImageReward human preference score Xu et al., [13] as the reward function. Here we use the difference potential.

We evaluate FK STEERING with different values of λ\lambda and different resampling schedules, [0, 20, 40, 60, 80] and [0, 70, 75, 80, 85, 90]. In Table 9, we observe that for all values of λ\lambda and the resampling schedule, the GenEval score of FK STEERING outperforms the base model. However, for lower values of λ\lambda, the CLIP diversity score is significantly higher, implying higher particle diversity. Similarly, in Table 10, we observe that for higher values of λ\lambda, the human preference scores are higher, while the particle diversity is lower.

ModelSamplerScheduleCLIP Div.GenEval Score
SD v1.4FK(k=4k = 4, λ=10\lambda= 10)5-30-50.14370.4814
SD v1.4FK(k=4k = 4, λ=10\lambda= 10)20-80-200.10500.5258
SD v1.4FK(k=4k = 4, λ=2\lambda= 2)5-30-50.23210.4975
SD v1.4FK(k=4k = 4, λ=2\lambda= 2)20-80-200.22390.4910
SD v1.4base (k=4k = 4)-0.31580.4408
SD v1.5FK(k=4k = 4, λ=10\lambda= 10)5-30-50.14590.4861
SD v1.5FK(k=4k = 4, λ=10\lambda= 10)20-80-200.10380.5224
SD v1.5FK(k=4k = 4, λ=2\lambda= 2)5-30-50.23300.4854
SD v1.5FK(k=4k = 4, λ=2\lambda= 2)20-80-200.22520.5114
SD v1.5base (k=4k = 4)-0.31150.4483
SD v2.1FK(k=4k = 4, λ=10\lambda= 10)5-30-50.12590.5523
SD v2.1FK(k=4k = 4, λ=10\lambda= 10)20-80-200.10610.5783
SD v2.1FK(k=4k = 4, λ=2\lambda= 2)5-30-50.20510.5607
SD v2.1FK(k=4k = 4, λ=2\lambda= 2)20-80-200.22130.5587
SD v2.1base (k=4k = 4)-0.29480.5104
SDXLFK(k=4k = 4, λ=10\lambda= 10)5-30-50.11820.6056
SDXLFK(k=4k = 4, λ=10\lambda= 10)20-80-200.10550.6034
SDXLFK(k=4k = 4, λ=2\lambda= 2)5-30-50.18160.5863
SDXLFK(k=4k = 4, λ=2\lambda= 2)20-80-200.21110.5857
SDXLbase (k=4k = 4)-0.28590.5571

Table 9. Effect of λ\lambda and resampling schedule on diversity. Here we report GenEval scores of all particles generation by FK STEERING to show that prompt fidelity increases for all particles. Moreover, we notice that lower values of λ\lambda can also be used to generate diverse particles.

ModelSamplerScheduleIR (Mean / Max)HPS (Mean / Max)CLIP Div.
SD v1.4base (k=4k = 4)-0.234 (0.800)0.245 (0.256)0.348
SD v1.4FK (k=4,λ=10.0k = 4, \lambda= 10.0)5-30-50.506 (0.783)0.251 (0.255)0.193
SD v1.4FK (k=4,λ=10.0k = 4, \lambda= 10.0)20-80-200.811 (0.927)0.258 (0.259)0.091
SD v1.4FK (k=4,λ=1.0k = 4, \lambda= 1.0)20-80-200.502 (0.763)0.252 (0.256)0.173
SD v1.4FK (k=4,λ=1.0k = 4, \lambda= 1.0)5-30-50.368 (0.723)0.248 (0.254)0.236
SD v2.1base (k=4k = 4)-0.372 (0.888)0.253 (0.263)0.318
SD v2.1FK (k=4,λ=1.0k = 4, \lambda= 1.0)5-30-50.582 (0.835)0.258 (0.261)0.180
SD v2.1FK (k=4,λ=10.0k = 4, \lambda= 10.0)20-80-200.891 (1.006)0.264 (0.266)0.087
SD v2.1FK (k=4,λ=1.0k = 4, \lambda= 1.0)20-80-200.579 (0.826)0.257 (0.261)0.164
SDXLbase (k=4k = 4)-0.871 (1.236)0.289 (0.296)0.248
SDXLFK (k=4,λ=10.0k = 4, \lambda= 10.0)5-30-51.032 (1.186)0.293 (0.295)0.123
SDXLFK (k=4,λ=10.0k = 4, \lambda= 10.0)20-80-201.211 (1.298)0.296 (0.297)0.071
Model + SamplerToxic ↑Toxic (Holdout) ↑PPL (GPT2-XL) ↓
SSD-LM0.4%1.2%23.2
SSD-LM (learned ∇ guidance)36.5%40.3%43.3
SSD-LM (∇ guidance)22.3%22.6%40.3
MDLM0.3%1.9%85.3
SSD-LM (no gradients)
BoN(4)1.6%4.8%21.9
BoN(8)5.0%8.1%23.0
FK(k=4k = 4)8.4%14.0%22.5
FK(k=4k = 4, learned rϕr_\phi)15.2%19.6%26.3
FK(k=8k = 8)25.0%29.7%23.9
FK(k=8k = 8, learned rϕr_\phi)39.0%38.0%26.9
SSD-LM (with gradients)
FK(k=4k = 4, gradients from learned rϕr_\phi)55.6%56.3%36.0

Table 10. Effect of λ\lambda and resampling schedule on diversity. Here we report ImageReward and HPS scores of all particles generation by FK STEERING to show that sample quality increases for all particles. Moreover, we notice that lower values of λ\lambda can also be used to generate diverse particles.

Text Experiments

For all text experiments, we use publicly available SSD-LM12, MDLM13, and GPT2-Medium14 checkpoints. For both text experiments, we generate sequences of length 50, conditioned on the prompts used by Han et al. [ [7] to evaluate controllable text generation. We generate 20 continuations for each of the 15 prompts.

Baselines

Following Han et al. [ [7], for SSD-LM we iteratively generate these continuations in blocks of 25. Except for our T=5000T = 5000 quality experiment, we default to T=500T = 500 for all SSD-LM experiments, and follow the multi-hot sampling procedure, with a top-p = 0.20 Han et al., [7]. For toxicity gradient guidance, we set the learning rate = 2000. For MDLM, we condition on each prompt by prefilling the prompt tokens at inference time. The model is trained to generate tokens in blocks of 1024. For consistency, we only consider the first 50 tokens of each generated sample, after re-tokenizing with the SSD-LM tokenizer. We use 1000 steps for all MDLM experiments. For the GPT2-Medium baseline, we generate all samples with top-p = 0.95 and temperature = 1.0.

method Details

For all FK STEERING text experiments, we set λ=10.0\lambda= 10.0 and use the difference of rewards potential. We resample 50 times for each inference: at every 10 steps for SSD-LM and every 20 steps for MDLM.12: https://huggingface.co/xhan77/ssd1m convert intermediate SSD-LM states to text, we sample tokens from the logit estimate, x^t\hat{x}_t, with top-p = 0.20. To convert intermediate MDLM states to text, we sample the masked tokens from the multinomial distribution given by x^t\hat{x}_t. By default, we sample one intermediate text for SSD-LM, and four texts for MDLM. Rewards are averaged over these samples. For Improved FK STEERING with MDLM, we sample and evaluate

16 intermediate texts, rather than 4.

For Improved FK STEERING with SSD-LM, we take the more involved approach of fine-tuning the off-the-shelf toxicity classifier on intermediate states, x^t\hat{x}_t. To build a training dataset, we used reward toxicity classifier to identify 26K non-toxic and 26K toxic texts from the OpenWebText corpus [69]. We then applied the SSD-LM forward process qq to noise the text to random timestep tt, and then use the base model to infer x^t\hat{x}_t. We then fine-tune the off-the-shelf reward classifier to estimate the toxicity probability of the original text given the intermediate text.

We fine-tune three reward models for different SSD-LM time-step ranges:

t∈[500,300),[300,200),[200,100)t \in[500, 300), [300, 200), [200, 100)

We train with batch size =16= 16 and learning rate =5e−7= 5e - 7, using a constant learning rate with 50 warm-up steps. We train with cross entropy loss, and use a weighting (0.99 non-toxic, 0.01 toxic), due to the rarity of toxicity in the original data distribution. For the gradient-based guidance baseline for SSD-LM, we use the default guidance scale from Han et al. [ [7].15

Feynman Kac IPS discussion

Choice of proposal distribution

Here we discuss various choices for twisting the transition kernel towards high reward samples:

  • Gradient-based guidance: For continuous-state models and differentiable rewards, we can use gradient’s from the reward [1, 3, 16, 20] to guide the sampling process. Suppose pθ(xt∣xt+1,c)=N(μθ(xt,c),σθ2Id)p_\theta(x_t \mid x_{t+1}, c) = \mathcal{N}(\mu_\theta(x_t, c), \sigma_\theta^2 I_d), then we can twist the transition kernel using reward gradients:

N(μθ(xt,c)+σθ2λ∇xtrϕ(xt,c),σθ2)(10)\mathcal{N}\left(\mu_\theta(x_t,c) + \sigma_\theta^2 \lambda\nabla_{x_t} r_\phi(x_t,c), \sigma_\theta^2\right) \tag*{(10)}

where rϕr_\phi is the intermediate reward, either learned or evaluated at the reward on the denoised state r(x0=x^t)r(x_0 = \hat{x}_t).

  • Discrete normalization: For discrete diffusion models, such as masked diffusion language model (MDLM) [44, 45], we can also estimate the normalization constant:

∑xtpθ(xt∣xt+1,c)Gt(xT,…,xt,c)(11)\sum_{x_t} p_\theta(x_t \mid x_{t+1}, c)G_t(x_T,\ldots,x_t,c) \tag*{(11)}

and sample from pFK(xt∣xt+1,…,xT)∝pθ(xt∣xt+1)Gt(xT,…,xt)p_{\mathrm{FK}}(x_t \mid x_{t+1},\ldots,x_T) \propto p_\theta(x_t \mid x_{t+1})G_t(x_T,\ldots,x_t). However, such methods for twisting the transition kernel can lead to increased sampling time compared to sampling from the base model pθp_\theta.

How existing work fits into method

TDS [16] uses SMC to do conditional sampling with a marginally trained model and a differentiable reward. They make the choices:

  • Potential. Gt(xt,xt+1)=exp⁡(λ(r(xt)−r(xt+1)))G_t(x_t,x_{t+1}) = \exp(\lambda(r(x_t)-r(x_{t+1}))), where the reward is computed on the denoised state r(xt)=r(x0=x^t)r(x_t)=r(x_0=\hat{x}_t).

  • Proposal generator. They use classifier-guidance to approximate the conditional score model sθ(xt,t,y)≈sθ(xt,t)+∇xtlog⁡pθ(y∣x0=x^t(xt,t))s_\theta(x_t,t,y) \approx s_\theta(x_t,t) + \nabla_{x_t}\log p_\theta(y \mid x_0=\hat{x}_t(x_t,t)) and use the following proposal generator τ(xt∣xt+1)\tau(x_t \mid x_{t+1}):

τ(xt∣xt+1)=N(Δt[f−gg⊤sθ(xt,t,y)],g(t)Δt)(12)\tau(x_t \mid x_{t+1}) = \mathcal{N}\left(\Delta t[f-gg^\top s_\theta(x_t,t,y)],g(t)\Delta t\right) \tag*{(12)}

FK STEERING allows for a more flexible use of potentials GtG_t, as well as proposal generators. For instance, Nichol et al. [ [51] show that conditionally trained scores outperform classifier-guidance even when the classifier is trained on noisy states xtx_t. However, as shown by Ghosh et al. [ [11], conditionally trained models still have failure modes. Therefore, we demonstrate how particle based methods can be used to improve the performance of conditionally trained models as well. Furthermore, FK STEERING allows these methods to be applied to discrete-space diffusions as well as non-differentiable rewards.

SVDD is a particle-based method which instead of using SMC, uses a nested importance sampling algorithm (see algorithm 5 of Naesseth et al. [ [37]). SVDD makes the following choices:

  • Potential. Similar to TDS, they use the potential Gt=exp⁡(λ(r(xt)−r(xt+1)))G_t=\exp(\lambda(r(x_t)-r(x_{t+1}))) where r(xt)r(x_t) can be off-the-shelf like TDS or learned from model samples.

  • Sampler. SVDD uses the base model as the proposal generator and generates kk samples at each step, selects a single sample using importance sampling and makes kk copies of it for the next step.

With λ=∞\lambda=\infty, SVDD is equivalent to doing best-of-nn at each step, since the authors recommend sampling from ptarget(x0)∝lim⁡λ→∞pθ(x0)exp⁡(λr(x0))p_{\mathrm{target}}(x_0) \propto\lim_{\lambda\to\infty}p_\theta(x_0)\exp(\lambda r(x_0)). We note that similar to SVDD, pFK,0p_{\mathrm{FK},0} can be sampled using nested importance sampling.

Adaptive Resampling

Following Wu et al. [ [16], Naesseth et al. [ [37], we use adaptive resampling to increase diversity of samples. Given kk particles xtix_t^i and their potentials GtiG_t^i, we define the effective sample size (ESS):

ESSt=1∑i=1k(G^ti)2(13)\mathrm{ESS}_t = \frac{1}{\sum_{i=1}^{k}(\hat{G}_t^i)^2} \tag*{(13)}

where G^\hat{G} refers to the normalized potentials and ESSt∈[1,k]\mathrm{ESS}_t \in[1,k]. If ESSt<k2\mathrm{ESS}_t < \frac{k}{2}, then we skip the resampling step. This encourages particle diversity.

FK STEERING samples

In this section, we show the effect of various sampling parameters, such as potentials, the temperature parameter λ\lambda, number of sampling steps, etc. on the diversity of samples. We use the stable diffusion XL-base (SDXL) as the base model and proposal generator and the ImageReward [13] human preference score model as the reward function. We also use adaptive resampling introduced in appendix § 10.§ 3. We compare FK STEERING against generating kk independent samples, using the same seed for generation, thus providing a counterfactual generation.

  • Effect of λ\lambda: The parameter λ\lambda is used to define the target distribution:

ptarget(x0)=1Zpθ(x0)exp⁡(λr(x0)),(14)p_{\mathrm{target}}(x_0) = \frac{1}{Z}p_{\theta}(x_0)\exp(\lambda r(x_0)), \tag*{(14)}

therefore, higher values of λ\lambda upweight higher reward samples x0x_0. Similarly, the potentials also use λ\lambda which affects resampling. We generate k=4k = 4 samples from the SDXL using FK STEERING as well as k=4k = 4 independent samples using the max potential. In Figure 6, we observe that using FK STEERING improves prompt fidelity, and higher values of λ\lambda improve fidelity at the cost of particle diversity.

A green stop sign in a red field

Figure 6. Effect on λ\lambda on diversity: In the top panel, we plot 4 independent samples from the base model and in the bottom 3 panels, we have the FK STEERING particles for varying values of λ\lambda. We observe that increasing λ\lambda leads to a decrease in diversity, at the cost of higher prompt fidelity and improved aesthetic quality, compared to the first row which has 4 independent samples. Caption: a green stop sign in a red field

  • Effect of potential: In Figure 7, we observe that FK STEERING with the max potential reduces diversity compared to the difference potential. Here we use λ=2\lambda= 2 and generate k=8k = 8 samples using the max and difference potential.

Eight independent stop-sign samples in the top row, with Diff and Max particle samples in the bottom two rows

Figure 7. Max versus Difference potential: In the top row, we plot 8 independent samples from the base model and in the bottom two rows, we have the FK STEERING particles for the max and difference potentials. Using the max potential reduces diversity compared to the difference potential. However, we note that by increasing the number of sampling steps, the diversity of the samples can be increased. Caption: a green stop sign in a red field

  • Effect of sampling steps. In Figure 7, we observe that diversity can be increased by increasing the number of sampling steps from 100 to 200. Here we use [180, 160, 140, 120, 0] and [80, 60, 40, 20, 0] as the resampling interval. We note that even if the samples x0x_0 share the same particle as parent, there is diversity in the final samples.

  • Effect of interval resampling: In Figure 8, we show that using interval resampling even with 100 sampling steps produces diversity in samples. For comparison, see Figure 9 for the independent versus FK STEERING generations.

Two images of a frisbee above a truck

Figure 8. Effect of interval resampling: While the overall diversity is reduced, using interval resampling encourages diversity. Caption: a photo of a frisbee above a truck

Two rows of eight images showing trucks with frisbees

Figure 9. Increased prompt fidelity: In this generation, we compare k=8k = 8 independent samples (top panel) versus k=8k = 8 samples from FK STEERING (bottom panel). FK STEERING selects samples which follow the prompt. Caption: a photo of a frisbee above a truck

References

  1. [1]Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  2. [2]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020.arxiv.org/abs/2006.11239
  3. [3]Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020a.arxiv.org/abs/2011.13456
  4. [4]Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022.
  5. [5]Nate Gruver, Samuel Stanton, Nathan C. Frey, Tim G. J. Rudner, Isidro Hötzel, Julien Lafrance-Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson. Protein design with guided discrete diffusion, 2023. URL https://arxiv.org/abs/2305.20009.
  6. [6]Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-Lm improves controllable text generation, 2022. URL https://arxiv.org/abs/2205.14217.
  7. [7]Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control, 2023. URL https://arxiv.org/abs/2210.17432.
  8. [8]Shanshan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models, 2023. URL https://arxiv.org/abs/2210.08933.
  9. [9]Ishaan Gulrajani and Tatsunori B. Hashimoto. Likelihood-based diffusion language models, 2023. URL https://arxiv.org/abs/2305.18619.
  10. [10]Zachary Horvitz, Ajay Patel, Chris Callison-Burch, Zhou Yu, and Kathleen McKeown. Paraguide: Guided diffusion paraphrasers for plug-and-play textual style transfer, 2024. URL https://arxiv.org/abs/2308.15459.
  11. [11]Dhruba Ghosh, Hannah Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2310.11513
  12. [12]Tomasz Korbak, Ethan Perez, and Christopher L Buckley. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275, 2022.arxiv.org/abs/2205.11275
  13. [13]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2304.05977
  14. [14]Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023a.arxiv.org/abs/2306.09341
  15. [15]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
  16. [16]Luhuan Wu, Brian Trippе, Christian Naesseth, David Blei, and John P Cunningham. Practical and asymptotically exact conditional sampling in diffusion models. Advances in Neural Information Processing Systems, 36, 2023b.arxiv.org/abs/2306.17775
  17. [17]Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023.arxiv.org/abs/2305.13301
  18. [18]Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint arXiv:2409.08861, 2024.
  19. [19]Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8228–8238, 2024.arxiv.org/abs/2311.12908
  20. [20]Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843–852, 2023.arxiv.org/abs/2302.07121
  21. [21]Sourav Chatterjee and Persi Diaconis. The sample size required in importance sampling. The Annals of Applied Probability, 28(2):1099–1135, 2018.arxiv.org/abs/1511.01437
  22. [22]Pierre Moral. Feynman-Kac formulae: genealogical and interacting particle systems with applications. Springer, 2004.DOI
  23. [23]Douglas Vestal, René Carmona, and Jean-Pierre Fouque. Interacting particle systems for the computation of cdo tranche spreads with rare defaults. 2008.
  24. [24]Brian L Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem. arXiv preprint arXiv:2206.04119, 2022.arxiv.org/abs/2206.04119
  25. [25]Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687, 2022.arxiv.org/abs/2209.14687
  26. [26]Yazid Janati, Badr Moufad, Alain Durmus, Eric Moulines, and Jimmy Olsson. Divide-and-conquer posterior sampling for denoising diffusion priors. Advances in Neural Information Processing Systems, 37:97408–97444, 2024.arxiv.org/abs/2403.11407
  27. [27]Pierre Del Moral and Josselin Garnier. Genealogical particle analysis of rare events. 2005.DOI
  28. [28]Martin Hairer and Jonathan Weare. Improved diffusion monte carlo. Communications on Pure and Applied Mathematics, 67(12):1995–2021, 2014.
  29. [29]Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Biancalani, Aviv Regev, Sergey Levine, and Masatoshi Uehara. Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding, 2024. URL https://arxiv.org/abs/2408.08252.
  30. [30]Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.arxiv.org/abs/2307.01952
  31. [31]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.arxiv.org/abs/2112.10752
  32. [32]Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language models via twisted sequential monte carlo, 2024a. URL https://arxiv.org/abs/2404.17546.
  33. [33]Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2305.16381
  34. [34]Gabriel Cardoso, Yazid Janati El Idrissi, Sylvain Le Corff, and Eric Moulines. Monte carlo guided diffusion for bayesian linear inverse problems. arXiv preprint arXiv:2308.07983, 2023.arxiv.org/abs/2308.07983
  35. [35]Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. In The Twelfth International Conference on Learning Representations, 2024.
  36. [36]Zheng Zhao, Ziwei Luo, Jens Sjölund, and Thomas B Schön. Conditional sampling within generative diffusion models. arXiv preprint arXiv:2409.09650, 2024b.arxiv.org/abs/2409.09650
  37. [37]Christian A Naesseth, Fredrik Lindsten, Thomas B Schön, et al. Elements of sequential monte carlo. Foundations and Trends® in Machine Learning, 12(3):307–392, 2019.arxiv.org/abs/1903.04797
  38. [38]Alexander K Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K Mansinghka. Sequential monte carlo steering of large language models using probabilistic programs. arXiv preprint arXiv:2306.03081, 2023.
  39. [39]Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2107.00630, 2021.arxiv.org/abs/2107.00630
  40. [40]Raghav Singhal, Mark Goldstein, and Rajesh Ranganath. Where to diffuse, how to diffuse, and how to get back: Automated learning for multivariate diffusions. In International conference on learning representations, 2023.arxiv.org/abs/2302.07261
  41. [41]Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.arxiv.org/abs/2210.02747
  42. [42]Raghav Singhal, Mark Goldstein, and Rajesh Ranganath. What’s the score? automated denoising score matching for nonlinear diffusions. In International conference on machine learning, 2024.arxiv.org/abs/2407.07998
  43. [43]Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34: 17981–17993, 2021.arxiv.org/abs/2107.03006
  44. [44]Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models, 2024.arxiv.org/abs/2406.07524
  45. [45]Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. arXiv preprint arXiv:2406.04329, 2024.arxiv.org/abs/2406.04329
  46. [46]Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266–28279, 2022.
  47. [47]Art Owen and Yi Zhou. Safe and effective importance sampling. Journal of the American Statistical Association, 95(449):135–143, 2000.DOI
  48. [48]Nicolas Chopin, Omiros Papaspiliopoulos, et al. An introduction to sequential Monte Carlo, volume 4. Springer, 2020.
  49. [49]Arnaud Doucet and Anthony Lee. Sequential monte carlo methods. In Handbook of graphical models, pages 165–188. CRC Press, 2018.
  50. [50]Pierre Del Moral and Laurent Miclo. Branching and interacting particle systems approximations of Feynman-Kac formulae with applications to non-linear filtering. Springer, 2000.
  51. [51]Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.arxiv.org/abs/2112.10741
  52. [52]Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
  53. [53]Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019a.
  54. [54]Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannanah Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377, 2024b.arxiv.org/abs/2401.17377
  55. [55]John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119–126, 2020.arxiv.org/abs/2005.05909
  56. [57]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.arxiv.org/abs/2212.09748
  57. [58]Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022.arxiv.org/abs/2206.00364
  58. [59]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.arxiv.org/abs/2207.12598
  59. [60]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020b.arxiv.org/abs/2010.02502
  60. [61]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2305.18290
  61. [62]Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018.arxiv.org/abs/1805.12471
  62. [63]Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019b. URL https://api.semanticscholar.org/CorpusID:160025533.
  63. [64]Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/N16-1014. URL https://aclanthology.org/N16-1014.DOI
  64. [65]Varvara Logacheva, Daryna Dementieva, Sergey Ustyantsev, Daniil Moskovskiy, David Dale, Irina Krotova, Nikita Semenov, and Alexander Panchenko. ParaDetox: Detoxification with parallel data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6804–6818, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.acl-long.469.
  65. [66]Daryna Dementieva, Daniil Moskovskiy, Nikolay Babakov, Abinew Ali Ayele, Naquee Rizwan, Frolian Schneider, Xintog Wang, Seid Muhie Yimam, Dmitry Ustalov, Elisei Stakovskii, Alisa Smirnova, Ashraf El-nagar, Animesh Mukherjee, and Alexander Panchenko. Overview of the multilingual text detoxification task at pan 2024. In Guglielmo Faggioli, Nicola Ferro, Petra Galuščáková, and Alba García Seco de Herrera, editors, Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum. CEUR-WS.org, 2024.
  66. [68]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
  67. [69]Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019.

Paper details

vlnc:W_cfmvsfp8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents