betaFeedback

WorksComputer Science

JaRS: Contrast-Free Localization of Memorized Regions in Diffusion Models via Jacobian Row Sums

Abstract

Diffusion models sometimes memorize and reproduce regions of their training images, raising privacy and copyright concerns. We aim to localize these regions. Existing contrast-based methods use the diagonal of the denoiser Jacobian, which reflects the posterior variance of each pixel. However, a small diagonal entry is ambiguous: it signals either memorization or averaging over many neighboring pixels in smooth or predictable regions. To resolve this ambiguity, these methods subtract a reference map, which requires a second evaluation with a null prompt or an earlier checkpoint. Unconditional models have no null prompt, and earlier checkpoints are unavailable in general. We show that Jacobian row sums (JaRS), unlike the diagonal, separate memorization from local averaging without a reference map. Pairing JaRS with an image-level detection score that uses the same statistic at an early denoising step yields JaRS+. Across Stable Diffusion v1.4 and v2.1, Realistic Vision v5.1, and an unconditional diffusion Transformer (DiT), JaRS+ outperforms contrast-based localization methods.

Introduction

Diffusion models [9, 28, 31, 32] have achieved remarkable success in image synthesis, yet they are known to memorize and replicate portions of their training data [4, 29]. While certain prompts cause text-to-image models to regenerate an entire training image almost verbatim, a far more common form of replication is local memorization: the model reproduces a specific region of a training image such as a foreground product, an interior layout, or a watermark—while synthesizing novel content in the rest of the image [30, 33]. Such partial copies raise significant copyright and privacy concerns. Because the memorized region is embedded within newly generated content, localizing the exact pixels that are copied tells us what the model has memorized and makes it possible to selectively remove the memorized region while preserving the surrounding generation.

Recently, Kim and Lee [15] showed that memorized regions can be localized directly from the local geometry of the denoiser. By Tweedie’s formula [7, 19, 24], the Jacobian JJ of the denoiser with respect to its noisy input is proportional to the posterior covariance of the clean image, and is directly related to the Hessian of the log density of the noisy marginal distribution. When a diffusion model memorizes a region, its posterior distribution over that region collapses [12, 15]; the posterior variance and hence the Jacobian become small, so the denoiser largely ignores perturbations to its noisy input. Kim and Lee [15] measure this collapse by using randomized Hutchinson probes [1, 10] to estimate the diagonal entries of the denoiser Jacobian (equivalently, the diagonal of the negative Hessian of the log-density). As Kim and Lee [15] note, however, a small diagonal entry JiiJ_{ii} is ambiguous on its own. Even in a model that generalizes without memorizing, the self-weight JiiJ_{ii} is small whenever a pixel lies in a smooth region such as a uniform background where the denoiser estimates each pixel by averaging over a neighborhood of NN pixels, assigning each pixel a weight of roughly 1/N1/N. To remove this data-driven curvature [27], Kim and Lee [15] rely on a contrast: they subtract a reference diagonal map computed either with an unconditional null prompt (Δh∅\Delta h_{\varnothing}) or with an earlier, less-trained checkpoint of the same model (Δhθ~\Delta h_{\tilde{\theta}}).

While contrast-based localization is effective on standard text-to-image models, requiring a second reference evaluation introduces fundamental limitation: A null prompt exists only for conditional models trained with classifier-free guidance [8], and is absent in unconditional models. Similarly, model developers rarely release intermediate training checkpoints alongside final weights. In this paper, we ask whether memorized regions can be separated from data-driven smoothing directly from the target model alone, without any reference prompt or secondary checkpoint. Our key insight is that while the diagonal entry JiiJ_{ii} cannot distinguish between a row whose weights are spread across NN neighboring pixels and a row whose entries vanish altogether, the row sum ∑jJij\sum_j J_{ij} separates the two.

Building on this principle, we introduce JaRS (Jacobian Row Sums), a contrast-free localization map that measures how far the per-channel row sums of the denoiser Jacobian fall below 1. Because the row sum is a linear functional of the Jacobian, JaRS requires no randomized probes: for a latent space with CC channels, the map is obtained from CC deterministic Jacobian-vector products (JVPs). Furthermore, to handle mixed evaluation settings that include both memorized and unmemorized generations where a localizer must output an empty mask on unmemorized images we propose JaRS+, a two-stage procedure that first determines whether an image contains memorized content using an early-step summary statistic of the JaRS map, and then returns the pixel-level JaRS map if memorization is detected. Our main contributions are summarized as follows:

  • We establish both theoretically via an analytical constant-region model and empirically that the row sums of the denoiser Jacobian concentrate near 1 on generated regions and drop sharply on memorized regions.

  • We propose JaRS and its two stage extension JaRS+.

  • Evaluated under the balanced protocol of Kim and Lee [15] across Stable Diffusion v1.4, Stable Diffusion v2.1, Realistic Vision v5.1, and an unconditional DiT-B/2 model, JaRS+ achieves the best localization mIoU in seven of ten settings, improving mIoU by up to 10.2 points on Stable Diffusion and from 0.666 to 0.869 on unconditional DiT.

  • We show that the early step detection score of JaRS+ identifies memorized prompts after just 5 of 50 sampling steps.

Related Work

Early studies demonstrated that diffusion models can memorize and regenerate training images with high fidelity. Carlini et al. [4] sampled from Stable Diffusion and from diffusion models trained on CIFAR-10 and flagged generations that were close, in Euclidean distance, to a training image. Somepalli et al. [29] instead measured replication in feature space, comparing generated and training images with copy-detection descriptors such as SSCD [22]. Webster [33] curated a benchmark of memorized prompts in Stable Diffusion and categorized them into three groups: matching verbatims (MV), which reproduce a training image paired with its training caption; retrieved verbatims (RV), where a prompt triggers a verbatim copy of a training image associated with a different caption; and template verbatims (TV), where the model reproduces a fixed spatial region (template) of a training image while generating diverse variations in the remaining regions across random seeds.

Subsequent works investigated the causes of memorization and proposed inference time detection and mitigation strategies. Somepalli et al. [30] linked replication to data duplication and text conditioning, proposing random token addition or replacement (RTA) to perturb memorized prompts. Wen et al. [34] observed that memorized prompts induce abnormally large text-conditional noise predictions and mitigated memorization by optimizing the prompt embedding at inference time, Chen et al. [6] re-anchored and searched prompts within classifier-free guidance to improve the privacy-utility trade-off, and Jeon et al. [12] optimized the initial Gaussian latent to steer trajectories away from sharp probability modes. However, all of these approaches operate at the level of the entire image or prompt.

Only recently has attention turned to pixel-level localization of memorized content. Chen et al. [5] argued that memorization should be analyzed locally and not only for whole images. They observed that at the final denoising step the cross-attention of the end-of-text token puts abnormally high weight on specific image patches (“Bright Ending”), and they used the resulting attention map to identify locally memorized regions. Interpreting memorization as a collapse of posterior p(x0∣xt)p(x_0 \mid x_t) [12], Kim and Lee [15] estimate the diagonal of the negative Hessian of log⁡pt(xt)\log p_t(x_t) at the final denoising step via Hutchinson-type probing [1, 10]. Because the raw diagonal also fires on smooth, easily predictable regions of ordinary images, Kim and Lee [15] subtracted a reference diagonal computed under the null prompt or an earlier training checkpoint, and established the localization benchmark that we adopt in our experiments. While we share their posterior collapse perspective, our work mainly departs from Kim and Lee [15] as we examine the row sums of the denoiser Jacobian rather than its diagonal which we believe eliminates the need for any contrast prompt or reference checkpoint.

Tweedie’s formula [7, 19, 24] connects the optimal denoiser to the score function ∇log⁡pt(xt)\nabla\log p_t(x_t), and differentiating it once more links the Jacobian of the denoiser to the posterior covariance Cov⁡[x0∣xt]\operatorname{Cov}[x_0 \mid x_t]. In classical and deep image denoising [13, 18, 20, 25], the Jacobian of a denoiser is interpreted as an input-dependent, locally adaptive linear filter; in particular, mean-preserving smoothing filters are row-stochastic, meaning that their row weights sum to 1 [18]. In diffusion posterior sampling for inverse problems, Boys et al. [3] multiplied the denoiser Jacobian by an all-ones vector as a computational shortcut to approximate the diagonal of Cov⁡[x0∣xt]\operatorname{Cov}[x_0 \mid x_t]. Jaziri et al. [11] studies the global leading singular value of the denoiser Jacobian along trajectories to characterize whole-image memorization basins.

Background and Preliminaries

We establish diffusion notation, link the denoiser Jacobian to the posterior, and review framework of Kim and Lee [15].

Setup

Let x0∈Rnx_0 \in\mathbb{R}^n denote a clean image (or the latent representation of an image) drawn from the data distribution p0p_0. A forward diffusion process progressively corrupts x0x_0 with Gaussian noise over time t∈[0,1]t \in[0,1] according to

xt=αtx0+σtϵ,ϵ∼N(0,I)(1)x_t = \alpha_t x_0 + \sigma_t \epsilon,\qquad\epsilon\sim\mathcal{N}(0,I) \tag*{(1)}

where αt,σt>0\alpha_t,\sigma_t > 0 define the noise schedule. Standard variance-preserving diffusion models such as Stable Diffusion [9, 26, 31] choose αt2+σt2=1\alpha_t^2 + \sigma_t^2 = 1 and train a neural network ϵθ(xt,t)\epsilon_\theta(x_t,t) to predict the injected noise ϵ\epsilon. Flow-matching models [16, 21] instead use the linear interpolation schedule αt=1−t\alpha_t = 1-t and σt=t\sigma_t=t, training a velocity network vθ(xt,t)v_\theta(x_t,t) to predict ϵ−x0\epsilon-x_0. In either formulation, given a noisy input xtx_t, the network produces a one-step estimate of the clean data point:

x^0(xt)=xt−σtϵθ(xt,t)αt(2)\hat{x}_0(x_t) = \frac{x_t-\sigma_t\epsilon_\theta(x_t,t)}{\alpha_t} \tag*{(2)}

Let pt(xt)p_t(x_t) denote the marginal density of xtx_t induced by Eq. (1). By Tweedie’s formula [7, 19, 24], the optimal noise predictor ϵ∗(xt,t)\epsilon^\ast(x_t,t) is

ϵ∗(xt,t)=−σt∇xtlog⁡pt(xt)\epsilon^\ast(x_t,t) = -\sigma_t \nabla_{x_t}\log p_t(x_t)

and the corresponding optimal denoiser is given by

E[x0∣xt]=xt+σt2∇xtlog⁡pt(xt)αt(3)\mathbb{E}[x_0 \mid x_t] = \frac{x_t+\sigma_t^2\nabla_{x_t}\log p_t(x_t)}{\alpha_t} \tag*{(3)}

To study the first-order sensitivity of the denoiser to perturbations in its noisy input xtx_t, we work with scaled denoiser D(xt)=αtx^0(xt)D(x_t)=\alpha_t\hat{x}_0(x_t). Differentiating D(xt)D(x_t) with respect to xtx_t yields the denoiser Jacobian J(xt)∈Rn×nJ(x_t)\in\mathbb{R}^{n\times n}:

J(xt)=∂D(xt)∂xt=I−σt∂ϵθ(xt,t)∂xt.(4)J(x_t)=\frac{\partial D(x_t)}{\partial x_t}=I-\sigma_t\frac{\partial\epsilon_\theta(x_t,t)}{\partial x_t}. \tag*{(4)}

When ϵθ\epsilon_\theta equals the optimal noise predictor ϵ∗\epsilon^\ast, substituting Eq. (3) into Eq. (4) and differentiating a second time via Tweedie’s second-order identity gives

J(xt)=I+σt2H(xt)=αt2σt2Cov⁡[x0∣xt](5)J(x_t)=I+\sigma_t^2H(x_t)=\frac{\alpha_t^2}{\sigma_t^2}\operatorname{Cov}[x_0\mid x_t] \tag*{(5)}

where H(xt)=∇xt2log⁡pt(xt)H(x_t)=\nabla_{x_t}^2\log p_t(x_t) is the Hessian of the log-density of the noisy marginal ptp_t. Additionally, one can see Jacobian as Adaptive linear filter. Locally around xtx_t, the ii-th row of the Jacobian, Ji,:=∂Di(xt)∂xtJ_{i,:}=\frac{\partial D_i(x_t)}{\partial x_t}, specifies how the denoised estimate at coordinate ii responds to infinitesimal changes at input coordinate j∈{1,…,n}j\in\{1,\ldots,n\}. For a bias-free network, the denoiser is exactly a locally adaptive linear filter, D(xt)=J(xt)xtD(x_t)=J(x_t)x_t, whose ii-th row Ji,:J_{i,:} contains the spatial averaging weights used to reconstruct pixel ii from its noisy neighborhood [13, 20]. For a general network, the rows still measure local sensitivity: ∑jJij\sum_j J_{ij} is the response of DiD_i to a uniform shift of its input. We verify in § 5.2 that, at intermediate noise, trained diffusion denoisers behave as mean-preserving filters on non-memorized content.

Curvature-Based Localization

We now connect the denoiser Jacobian to memorization. When a diffusion model memorizes a training image or a spatial region RR of it, the posterior p(x0∣xt)p(x_0\mid x_t) over RR collapses [12]. Because the denoised output Di(xt)D_i(x_t) for a memorized pixel i∈Ri\in R is fixed to the stored training content regardless of small perturbations to xtx_t, the posterior variance Var⁡(x0,i∣xt)\operatorname{Var}(x_{0,i}\mid x_t) vanishes. By Eq. (5), this implies Jii(xt)≈0J_{ii}(x_t)\approx0. Kim and Lee [15] turn this observation into a localization metric by examining the diagonal of the negative log-density Hessian −H(xt)-H(x_t), which by Eq. (5) is

σtdiag⁡(−H(xt))i=1−Jii(xt)σt(6)\sigma_t\operatorname{diag}(-H(x_t))_i=\frac{1-J_{ii}(x_t)}{\sigma_t} \tag*{(6)}

Kim and Lee [15] estimate σtdiag⁡(−H(xt))\sigma_t\operatorname{diag}(-H(x_t)) at the final denoising step (we follow their official implementation), marking pixels with large values of σtdiag⁡(−H(xt))\sigma_t\operatorname{diag}(-H(x_t)) (equivalently, small values of JiiJ_{ii}) as memorized. However, a small diagonal entry Jii(xt)J_{ii}(x_t) does not identify memorization. To see why, consider the following example.

Example 1. Suppose a denoiser reconstructs a smooth image region (such as a uniform wall or clear sky) of NN pixels by taking an unweighted average of the NN noisy pixels in that region Di(xt)=1N∑j∈Rxt,jD_i(x_t) = \frac{1}{N}\sum_{j\in R} x_{t,j} where i∈Ri \in R. The jacobian is given by Jij(xt)=1/NJ_{ij}(x_t) = 1/N for all j∈Rj \in R. Consequently, the diagonal is Jii(xt)=1/NJ_{ii}(x_t) = 1/N, which approaches 00 as the averaging neighborhood NN grows large. Thus, on a smooth region of size N≫1N \gg1, the diagonal entry Jii=1/NJ_{ii} = 1/N is nearly indistinguishable from the diagonal entry Jii=0J_{ii} = 0 of a memorized region where the denoiser outputs a stored constant Di(xt)=ciD_i(x_t) = c_i.

Following Ross et al. [27], Kim and Lee [15] refer to the large values of diag⁡(−H)i\operatorname{diag}(-H)_i as data-driven curvature, in contrast to memorization-driven curvature. As discussed in Example 1, raw diagonal (Eq. (6)) can’t differentiate between these two. To cancel the data-driven component, Kim and Lee [15] use a contrast, which assumes that a reference evaluation (∅\varnothing or θ~\tilde{\theta}) exhibits the same data-driven smoothing on generic regions, so that subtracting it cancels the background response and isolates the memorized region. For a conditional model ϵθ(xt,t,y)\epsilon_\theta(x_t,t,y) with prompt yy, they subtract the diagonal curvature computed either with null prompt ∅\varnothing,

(Δh∅)i=diag⁡(−Hθ(xt,t,y))i−diag⁡(−Hθ(xt,t,∅))i(7)(\Delta h_{\varnothing})_i = \operatorname{diag}(-H_\theta(x_t,t,y))_i - \operatorname{diag}(-H_\theta(x_t,t,\varnothing))_i \tag*{(7)}

or with a second, less-trained checkpoint θ~\tilde{\theta},

(Δhθ~)i=diag⁡(−Hθ(xt,t,y))i−diag⁡(−Hθ~(xt,t,y))i(8)(\Delta h_{\tilde{\theta}})_i = \operatorname{diag}(-H_\theta(x_t,t,y))_i - \operatorname{diag}(-H_{\tilde{\theta}}(x_t,t,y))_i \tag*{(8)}

Kim and Lee [15] also evaluate two score contrasts [34] given by

(Δs∅)i=(sθ(xt,t,y)−sθ(xt,t,∅))i2,(Δsθ~)i=(sθ(xt,t,y)−sθ~(xt,t,y))i2(9)\begin{aligned} (\Delta s_{\varnothing})_i &= \left(s_\theta(x_t,t,y)-s_\theta(x_t,t,\varnothing)\right)^2_i,\\ (\Delta s_{\tilde{\theta}})_i &= \left(s_\theta(x_t,t,y)-s_{\tilde{\theta}}(x_t,t,y)\right)^2_i \tag*{(9)} \end{aligned}

where sθ=−ϵθ/σts_\theta=-\epsilon_\theta/\sigma_t. Following the implementation of Kim and Lee [15], a 2D spatial map is formed by summing the coordinate-wise values across the CC channels at each latent pixel.

As noted in § 1, however, computing Eq. (7), Eq. (8), or Eq. (9) requires a reference evaluation that is unavailable for unconditional models or when suitable prememorization checkpoints are not released. This motivates a central question: Can we distinguish local smoothing from memorization using only the target model itself? Example 1 suggests a route: while the diagonal entry JiiJ_{ii} shrinks to 1/N1/N under local averaging, the sum of the entries across the row is ∑j∈RJij=N⋅(1/N)=1\sum_{j\in R} J_{ij}=N\cdot(1/N)=1, whereas under memorization (Jij=0J_{ij}=0) the row sum vanishes. We formalize this observation in the next section.

Jacobian Row Sums (JaRS)

We now present our contrast-free approach to localizing memorized regions. In § 4.1, we define the JaRS (Jacobian Row Sums) localization map. In § 4.2, we show how JaRS can be computed using Jacobian-vector products (we verify the predictions of this section on full denoiser Jacobians in § 5.2). Finally, we introduce our two-stage detection-and-localization pipeline JaRS+ in § 4.3.

Contrast-Free Localization

Consider xt∈Rnx_t\in\mathbb{R}^n with n=C×H×Wn=C\times H\times W, where we index coordinates by (c,i)(c,i) for channel c∈{1,…,C}c\in\{1,\ldots,C\} and spatial pixel location i∈{1,…,HW}i\in\{1,\ldots,HW\}. Let J(c,i),(c′,j)(xt)J_{(c,i),(c',j)}(x_t) denote the entry of the scaled denoiser Jacobian J(xt)J(x_t) in (4). Let 1C∈Rn\mathbf{1}_C\in\mathbb{R}^n denote the channel indicator vector.

Definition 1. At noise level tt, the Jacobian Row Sum (JaRS) score at spatial location i∈{1,…,HW}i\in\{1,\ldots,HW\} is defined as

st(i)=∑c=1C(1−∑j=1HWJ(c,i),(c,j)(xt))s_t(i)=\sum_{c=1}^{C}\left(1-\sum_{j=1}^{HW}J_{(c,i),(c,j)}(x_t)\right)
=C−∑c=1C(J(xt)1C)(c,i)(10)= C-\sum_{c=1}^{C}\left(J(x_t)\mathbf{1}_C\right)_{(c,i)} \tag*{(10)}

st(i)s_t(i) measures how far per-channel row sums fall below 1. From Eq. (5), each channel term can be written equivalently as

1−(J(xt)1C)(c,i)=−σt2(H(xt)1C)(c,i)1-\left(J(x_t)\mathbf{1}_C\right)_{(c,i)}=-\sigma_t^2\left(H(x_t)\mathbf{1}_C\right)_{(c,i)}

Thus, whereas Kim and Lee [15] take the diagonal of the negative log-density Hessian, JaRS takes its row sums, and the constant 1 per channel can be seen as a replacement for the contrast. By this identity, the row sum equals 1 when H(xt)1C=0H(x_t)\mathbf{1}_C=0, i.e., when log⁡pt\log p_t is flat along 1C\mathbf{1}_C: slightly brightening or darkening a generic region yields an equally plausible image.

As discussed in § 3.1 and Example 1, the diagonal shrinks as 1/N1/N for an NN-pixel average, whereas the row sum does not decrease as the averaging neighborhood widens. Therefore, on generated regions, (J(xt)1C)(c,i)\left(J(x_t)\mathbf{1}_C\right)_{(c,i)} is close to 1 and the JaRS score st(i)s_t(i) is close to 0. Conversely, when the posterior collapses onto a stored image x0∗x_0^*, ptp_t is locally close to N(αtx0∗,σt2I)\mathcal{N}(\alpha_t x_0^*,\sigma_t^2 I), so H(xt)≈−I/σt2H(x_t)\approx-I/\sigma_t^2: the denoiser ignores perturbations of xtx_t, (J(xt)1C)(c,i)\left(J(x_t)\mathbf{1}_C\right)_{(c,i)} drops toward 0, and st(i)s_t(i) rises toward CC.

The following proposition makes this precise in a simple analytical model of a homogeneous image region.

Proposition 1. Consider a single-channel region RR of NN pixels whose clean values share a common intensity level, x0,j=cx_{0,j}=c for all j∈Rj\in R, under a Gaussian prior $c\sim

N(μ,τ2)\mathcal{N}(\mu,\tau^2). Given a noisy observation xtx_t following Eq. (1), the optimal denoiser Jacobian satisfies, for every pixel i∈Ri \in R and j∈Rj \in R,

Jij(xt)=gN,g=Nαt2τ2σt2+Nαt2τ2∈[0,1)(11)J_{ij}(x_t)=\frac{g}{N},\qquad g=\frac{N\alpha_t^2\tau^2}{\sigma_t^2+N\alpha_t^2\tau^2}\in[0,1) \tag*{(11)}

Consequently, the diagonal entry at pixel i∈Ri \in R is Jii(xt)=g/NJ_{ii}(x_t)=g/N, whereas the row sum over RR is ∑j∈RJij(xt)=g\sum_{j\in R}J_{ij}(x_t)=g.

Proposition 1 illustrates two properties of the JaRS map: robustness to neighborhood size and dependence on noise schedule. First, regarding neighborhood size NN, the prior variance τ2\tau^2 quantifies the model’s uncertainty over RR prior to observing xtx_t. For generic, unmemorized regions where τ2\tau^2 is large, the row sum gg approaches 1 regardless of NN once Nαt2τ2≫σt2N\alpha_t^2\tau^2 \gg\sigma_t^2, while the diagonal entry JiiJ_{ii} scales as O(1/N)O(1/N). In contrast, for memorized regions, the prior collapses onto a stored value (τ2→0\tau^2 \to0), yielding g→0g \to0 and driving the JaRS score 1−g→11-g \to1.

Second, whereas Kim and Lee [15] evaluate curvature differences strictly at the final denoising step, Eq. (11) identifies the noise-to-signal ratio σt/αt\sigma_t/\alpha_t as the primary control knob. In the high-noise regime (σt2≫Nαt2τ2\sigma_t^2 \gg N\alpha_t^2\tau^2), g≈0g \approx0 even for unmemorized regions because xtx_t is nearly uninformative of x0x_0. At near-zero noise (σt2→0\sigma_t^2 \to0), g→1g \to1 everywhere, unless the memorized content is stored with effective variance smaller than σt2/(Nαt2)\sigma_t^2/(N\alpha_t^2). Between these extremes lies a broad intermediate noise window where g≈1g \approx1 on generated regions and g≪1g \ll1 on memorized regions; identifying and operating in this intermediate noise regime is a key step that takes us from final-step evaluation to JaRS and JaRS+.

Computation

A practical advantage of Definition 1 is that st(i)s_t(i) is a linear functional of the Jacobian J(xt)J(x_t). Unlike the diagonal diag⁡(J(xt))\operatorname{diag}(J(x_t)), which requires randomized Hutchinson probes [1, 10] to estimate, the row sums are obtained from the CC matrix-vector products J(xt)11,…,J(xt)1CJ(x_t)\mathbf{1}_1,\ldots,J(x_t)\mathbf{1}_C—one for each latent channel. Using Eq. (4), each product takes the form J(xt)1c=1c−σtJϵ(xt)1cJ(x_t)\mathbf{1}_c=\mathbf{1}_c-\sigma_tJ_\epsilon(x_t)\mathbf{1}_c, where Jϵ(xt)=∂ϵθ(xt,t,y)∂xtJ_\epsilon(x_t)=\frac{\partial\epsilon_\theta(x_t,t,y)}{\partial x_t} is the Jacobian of the noise predictor (with conditioning yy for conditional models, or omitted for unconditional models). We evaluate each directional derivative Jϵ(xt)1cJ_\epsilon(x_t)\mathbf{1}_c via a central finite difference:

Jϵ(xt)1c≈ϵθ(xt+h1c,t,y)−ϵθ(xt−h1c,t,y)2h(12)J_\epsilon(x_t)\mathbf{1}_c\approx\frac{\epsilon_\theta(x_t+h\mathbf{1}_c,t,y)-\epsilon_\theta(x_t-h\mathbf{1}_c,t,y)}{2h} \tag*{(12)}

For all our experiments, we used h=3×10−3h=3\times10^{-3}. Observe that computing the full JaRS map at a given timestep therefore requires 2C2C forward network evaluations, which can be run in a single batched forward pass without any backward pass or automatic differentiation graph (App. § 17).

JaRS+: Two-Stage Detection and Localization

Discussion after Proposition 1 tells us that row-sum map should be evaluated at intermediate time steps. In realistic settings that mix memorized and non-memorized generations, however, a localizer must also output an empty mask on images that contain no copied content. As seen in the left tail of the non-memorized histogram for Stable Diffusion v1.4 in Figure 2, certain highly predictable regions of ordinary images such as uniform studio backgrounds also exhibit low posterior variance at intermediate noise levels. If a single pixel threshold is applied across all images at step ℓ\ell without an image-level check, these confident background regions produce false-positive masks on non-memorized images.

Figure 2. Full denoiser Jacobian rows of Stable Diffusion v1.4 (step 15) and DiT-B/2 (step 35). Left: For one development image from each category, we show the generated image (copied region outlined in cyan; pixels outside a copied template are dimmed), the row Ji,:J_{i,:} of the full Jacobian at the marked pixel ‘+’ together with its sum, and the JaRS map (10). Right: Empirical distributions of the per-channel row sum ∑jJij\sum_j J_{ij} across all pixels of non-memorized images and whole-image copies, and inside the copied region of template verbatims and watermarked samples (10 development images per category; 7 retrieved verbatims). The vertical dotted line at 1 marks the row sum of a mean-preserving averaging filter.

Crucially, analyzing the JaRS map across the denoising trajectory reveals a two timescale separation between image-level detection and pixel-level localization. In conditional models, memorized content is locked in by the prompt in the earliest denoising steps (high noise), where the JaRS map already exhibits elevated values on memorized trajectories while ordinary trajectories remain dominated by the initial Gaussian noise; at intermediate noise steps, spatial boundaries between copied templates and generated backgrounds become sharp (Figure 5). Exploiting this two-step structure takes us from JaRS to JaRS+, a two-stage procedure that pairs an early image-level detection stage at step dd with a pixel-level localization stage at step ℓ\ell:

Figure 5. Evolution of the JaRS map sts_t across the 50-step sampling trajectory on the development split of the localization benchmark. Top: Image-level Stage 1 score zt=Q0.9(st)z_t = Q_{0.9}(s_t) for each category (median and interquartile range across development instances; category counts are listed in parentheses). Bottom: Pixel-level values of sts_t inside vs. outside the copied region for template verbatims (TV) and watermarked samples (WM), and across the full image for the remaining categories (median and interquartile range over 1,024 randomly sampled pixels per image). Vertical dotted lines mark the detection step dd (top) and localization step ℓ\ell (bottom) selected on the development split (§ 4.3). On the unconditional DiT, all 154 development samples are shown; at step 0 (t=1t = 1) of the flow-matching schedule, the scaled denoiser Jacobian vanishes identically (J=0J = 0), so st=Cs_t = C at every pixel.

  1. Stage 1 (Image-level detection): At denoising step dd, we compute the upsampled JaRS map stds_{t_d} and summarize it by its 90th percentile zd=Q0.9(std)z_d=Q_{0.9}(s_{t_d}). The generation is flagged as containing memorized content if and only if zd>τz_d>\tau, where τ\tau is a scalar detection threshold.

  2. Stage 2 (Pixel-level localization): If zd≤τz_d\le\tau, the image is declared unmemorized and JaRS+ outputs an identically zero map. If zd>τz_d>\tau, JaRS+ returns the JaRS localization map stℓs_{t_\ell} evaluated at step ℓ\ell.

We now describe how we select ℓ\ell, dd and τ\tau. For each model, the three hyperparameters are selected on a held-out development split.

  • Localization step ℓ\ell: Chosen from the candidate steps {5,10,…,45}\{5,10,\ldots,45\} of the 50-step sampler to maximize the JaRS mean IoU on the memorized development images.

  • Detection step dd: Chosen from {0,5,…,45}\{0,5,\ldots,45\} to maximize the image-level detection AUROC of zdz_d between memorized and non-memorized images on the balanced development split of the model’s broadest evaluation setting (§ 5.1)

  • Detection threshold τ\tau: Chosen from candidate quantiles (5%, 7.5%, …\ldots, 95%) of zdz_d on the same balanced development split to maximize the resulting two-stage development mIoU. When labeled memorized examples are not available to tune τ\tau, τ\tau can instead be set directly to the 95th percentile of zdz_d on ordinary non-memorized images (5% development false-positive rate), which we show in

Sec. D achieves comparable accuracy across all text-to-image models. All ties above are broken in favor of the earlier step. Selection of ℓ\ell, dd, τ\tau above is discussed for 50 sampling steps, but can extended easily to arbitrary number of sampling steps.

Experiments

We evaluate JaRS and JaRS+ across text-to-image diffusion models and unconditional Diffusion Transformer (DiT). We first describe our benchmarks, evaluation protocol, and baselines (§ 5.1), verify the behavior of row sums predicted in § 4.1 on full denoiser Jacobians (§ 5.2), and then present our main pixel-level localization results (§ 5.3). Finally, we evaluate prompt-level memorization detection (§ 5.4). In App. § 18, we further show that restricting the inference-time mitigation of Wen et al. [34] to the region localized by JaRS lowers the copy rate from 20.5% to 9.0% and the perceptual change outside the region from 0.708 to 0.158 LPIPS.

Benchmarks, Protocol, and Baselines

We adopt the memorization localization benchmark of Kim and Lee [15], which is built on the memorized prompts identified by Webster [33] across three settings: Stable Diffusion v1.4 (SD v1.4), Stable Diffusion v2.1-base(SD v2.1) [26], and Realistic Vision v5.1. The benchmark comprises three categories of generations: Template verbatims (TV), Matching and Retrieved verbatims (MV, RV) come under Global memorization and Non-memorized prompts (N). We generate all images and ground-truth masks using the official code released by Kim and Lee [15].

To evaluate localization in an unconditional setting where no null prompt exists and with exact ground-truth masks, we train a unconditional DiT (DiT-B/2 [21]) using flow matching [16] on the 70,000 FFHQ [14] face images at 256×256256 \times256 resolution. Prior to training, we inject two controlled forms of memorization into the dataset: Local memorization (WM): a fixed semi-transparent watermark is pasted at a random spatial location into 10% of the training images and Global memorization (Near-copies): 20 clean training images are duplicated 100 times each. Generated samples are labeled as watermarked (with the watermark footprint as the ground-truth mask), near-copies (all-ones mask), or non-memorized (empty mask), and we retain 300 watermarked, 300 non-memorized, and all 139 near-copy samples (App. § 9).

We follow the balanced evaluation protocol of Kim and Lee [15 App. F.2] across all settings reported in their work: TV (WM for DiT) evaluates localization exclusively on locally memorized images, TV+N (WM+N) adds non-memorized images, and All further adds globally memorized images, with each category subsampled once to the size of the smallest. Following Kim and Lee [15], every localization map is clipped at 0, transformed via x↦log⁡(1+x)x \mapsto\log(1+x), min-max normalized across the evaluated set, and binarized using the single threshold that maximizes the mean Intersection-over-Union (mIoU) across images.

We partition all prompts (along with their seeds and templates) and DiT samples into a 20% development split (indices divisible by 5) and 80% test split. All hyperparameters of JaRS+ (ℓ\ell, dd, and τ\tau; § 4.3) are selected on the development split, and all experimental results are reported on the test split. The development split selects (ℓ=15,d=5)(\ell= 15, d = 5) for SD v1.4 and SD v2.1, (ℓ=35,d=5)(\ell= 35, d = 5) for Realistic Vision, and (ℓ=35,d=30)(\ell= 35, d = 30) for the DiT (App. § 13).

We compare JaRS+ against the maps of Kim and Lee [15]: the contrast-free raw diagonal, the null-prompt contrast Δh∅\Delta h_{\varnothing} (Eq. (7)), the checkpoint contrast Δh~∅\Delta\tilde{h}_{\varnothing} (Eq. (8)), and the two score contrasts Δs∅\Delta s_{\varnothing} and Δs~∅\Delta\tilde{s}_{\varnothing} (Eq. (9)). On the DiT, only the checkpoint contrasts are applicable; we evaluate them at step 48, which matches the noise level of their published configuration, with the reference selected on the development split among five checkpoints saved during our training run (App. § 9). We also include the benchmark numbers reported by Kim and Lee [15] for Bright Ending [5].

Empirical Verification on Full Jacobians

To verify whether trained diffusion models follow the predicted behavior in § 4.1, we explicitly form the full n×nn \times n denoiser Jacobian J(xt)J(x_t) and inspect its rows directly (Figure 2). On the DiT (n=4,096n = 4{,}096), we materialize the exact Jacobian at ℓ=35\ell= 35 for 10 randomly selected development samples from each category: watermarked, near-copies, and non-memorized. On non-memorized samples, each row Ji,:J_{i,:} is a localized smoothing kernel centered at pixel ii, and the per-channel row sums concentrate sharply around 1 (median 0.97, with 80% of values in [0.87,1.05][0.87, 1.05]), confirming that the denoiser acts as a mean preserving local filter. The unmemorized background of watermarked samples behaves identically (median row sum 0.98). In contrast, inside the memorized watermark region, the row sums drop to a median of 0.48, and on whole-image near-copies they fall to 0.45. Across all pixels, the row sum separates watermark pixels from non-memorized pixels with an AU-ROC of 0.998.

The same phenomenon holds on SD v1.4 (n=16,384n = 16{,}384), where we compute the full Jacobian at the localization step ℓ=15\ell= 15 for 37 development images: 10 TV, 10 MV, all 7 RV of the development split, and 10 non-memorized images. The per-channel row sums have a median of 0.88 on non-memorized images and 0.89 outside the copied template, falling to 0.41 inside the template region, 0.54 on matching verbatims, and 0.58 on retrieved verbatims. App. § 10 repeats this analysis on SD v2.1 and compares the row sum with the ℓ1\ell_1 norm of the row.

Memorization Localization Results

Tab. 1 reports the quantitative localization results across all four models, and Figure 1 visualizes representative localization maps on SD v1.4, where the maps of Kim and Lee [15] are computed at the final sampling step, as in Table 1. On both SD v1.4 and v2.1, JaRS+ achieves the highest localization accuracy in the TV and TV+N settings despite using no reference prompt or checkpoint. Specifically, JaRS+ outperforms the strongest contrast baseline by +5.6 and +2.5 mIoU points on SD v1.4 and by +3.2 and +10.2 points on SD v2.1. In the All setting of SD v1.4, JaRS+ (0.940) comes within 0.005 of Δh∅\Delta h_{\varnothing} (0.945). When the step and the image-level gate of every method, including the percentile of JaRS+, are tuned on the same development split, JaRS+ remains best in eight of ten settings (App. E).

Qualitative comparison of localization maps

Figure 1. Qualitative comparison of localization maps on Stable Diffusion v1.4 for randomly sampled test instances, two per row: four template verbatims from distinct templates (rows 1–2), one global whole-image copy (bottom left), and one non-memorized prompt (bottom right), which has no copied region and hence an empty expected mask. All maps are displayed after the log⁡(1+⋅)\log(1+\cdot) transformation.

MethodReferenceSD v1.4 TVSD v1.4 TV+NSD v1.4 AllSD v2.1 TVSD v2.1 TV+NRealistic Vision TVRealistic Vision AllDiT (unconditional) WMDiT (unconditional) WM+NDiT (unconditional) All
Raw diagonal [15]–0.5940.5000.5240.6520.5000.6600.5300.2610.5000.640
Bright Ending† [5]–0.751––0.933–0.777––––
Δh∅\Delta h_{\varnothing} [15]null prompt0.9000.9500.9450.9450.8870.9300.951not available
Δhθ~\Delta h_{\tilde{\theta}} [15]checkpoint0.9130.9030.8840.9480.8760.9360.8800.4390.5000.666
Δs∅\Delta s_{\varnothing} [15]null prompt0.8260.9010.8980.7870.8090.8760.903not available
Δsθ~\Delta s_{\tilde{\theta}} [15]checkpoint0.8640.7300.6530.9210.8770.8600.6260.3760.5010.523
JaRS (ours)–0.9690.6190.6370.9800.5710.9490.6120.8360.8730.909
JaRS+ (ours)–0.9690.9750.9400.9800.9890.9290.8490.6380.7990.869

Table 1. Pixel-level localization of memorized regions on the test splits (mIoU) under the balanced protocol of Kim and Lee [15]. The baselines of Kim and Lee [15] are evaluated at the final sampling step (step 48 on the DiT, at the same noise level σt/αt\sigma_t/\alpha_t) using 16 Hutchinson probes; checkpoint references for Δhθ~\Delta h_{\tilde{\theta}} and Δsθ~\Delta s_{\tilde{\theta}} are SD v1.1 (for SD v1.4 and Realistic Vision), SD v2.0-base (for SD v2.1), and the 5k-step checkpoint, which is best on the DiT development split for both contrasts. JaRS+ requires no reference prompt or checkpoint, and its hyperparameters (ℓ,d,τ)(\ell, d, \tau) are chosen once per model on the development split; JaRS denotes the map at step ℓ\ell without the image-level gate (Stage 1) of JaRS+. In settings with non-memorized images, an mIoU of 0.500 corresponds to predicting an empty mask everywhere. †Numbers reported by Kim and Lee [15] on the full benchmark.

On the unconditional DiT, where the null-prompt contrast does not exist, JaRS+ outperforms the checkpoint contrast Δhθ~\Delta h_{\tilde{\theta}} across all three settings, raising mIoU from 0.439 to 0.638 on WM, from 0.500 to 0.799 on WM+N, and from 0.666 to 0.869 on All. Furthermore, evaluating Δhθ~\Delta h_{\tilde{\theta}} across all saved training checkpoints reveals sensitivity to the choice of reference: the 20k-step checkpoint would reach 0.503 on WM, and later checkpoints (100k and 150k steps) drop to 0.369 and 0.243 (full sweep in App. § 15).

On Realistic Vision, JaRS+ trails the contrast baselines: while JaRS+ is within 0.007 of Δhθ~\Delta h_{\widetilde{\theta}} on TV (0.929 vs. 0.936), it scores 0.849 on All compared to 0.951 for Δh∅\Delta h_{\varnothing}. This gap arises from the Stage 1 threshold rather than the JaRS map: with an oracle image-level detector, the same maps reach 0.945 on All (App. § 11). Removing Stage 1 (JaRS in Table 1) does not lower mIoU on TV and WM, but lowers it sharply on the text-to-image settings with non-memorized images (e.g., from 0.975 to 0.619 on TV+N of SD v1.4). On the DiT, removing Stage 1 raises mIoU (0.869 to 0.909 on All).

It is also instructive to compare JaRS+ with the raw diagonal row in Table 1, which evaluates the diagonal estimator of Kim and Lee [15] without a contrast: the raw diagonal achieves only 0.594 and 0.652 on TV and collapses to 0.500 on TV+N (performing no better than an empty mask). Replacing the diagonal with the row sum eliminates the data-driven smoothing. The noise level matters as well: at the final step, where the contrasts of Kim and Lee [15] are evaluated, the JaRS map itself is no better than an all-ones mask (0.563 and 0.648 on TV), whereas the step ℓ\ell selected on the development split lies on a broad high-accuracy plateau at intermediate noise on all four models (App. § 14).

Detecting Memorized Prompts and Samples

Beyond pixel-level maps, the Stage 1 score zd=Q0.9(std⁡t)z_d = Q_{0.9}(\operatorname{std}_{t}) of JaRS+ can directly serve as a standalone detector of whether a prompt (or unconditional sample) triggers memorization. Following the prompt-detection protocol of Kim and Lee [15 Sec. 5.2], each prompt is generated with four random seeds and scored by averaging zdz_d for JaRS+ (at the step dd of § 5.1) or the spatial mean of the final-step map for the baselines; on the DiT, individual samples are scored. The positive and negative sets, test split sizes, and results on Realistic Vision are given in App. § 16.

As shown in Table 2, the score contrasts of Kim and Lee [15] achieve the highest detection accuracy on SD v1.4 and the DiT, while on SD v2.1 JaRS+ and all four contrasts are within 0.003 AUC of one another. Nevertheless, JaRS+ remains competitive under much weaker requirements. First, it requires no reference evaluation: on the DiT, it outperforms both the raw diagonal and Δhθ~\Delta h_{\widetilde{\theta}}, and Δsθ~\Delta s_{\widetilde{\theta}} surpasses it only with a reference from 25%–75% of training, which is rarely available in practice, while no reference allows Δsθ~\Delta s_{\widetilde{\theta}} to localize memorized regions (at most 0.523 mIoU on All; App. § 15). Second, on Stable Diffusion, JaRS+ computes its score at step d=5d = 5 of 50, whereas the final-step maps of Kim and Lee [15] require the entire generation, allowing JaRS+ to flag memorized prompts before a copy is synthesized.

MethodSD v1.4SD v2.1DiT
AUCTPRAUCTPRAUCTPR
Bright Ending†[5]0.8860.3900.9450.877––
Raw diagonal [15]0.9060.5470.8710.0400.7470.308
Δh∅\Delta h_{\varnothing} [15]0.9690.9150.9950.966not available
Δhθ~\Delta h_{\widetilde{\theta}} [15]0.9510.8970.9940.9540.8670.363
Δs∅\Delta s_{\varnothing} [15]0.9690.9400.9940.966not available
Δsθ~\Delta s_{\widetilde{\theta}} [15]0.9570.9250.9930.9710.9430.500
JaRS+ (ours)0.9460.8730.9920.9710.9020.483

Table 2. Prompt- and sample-level memorization detection on the test split, measured by AUROC (‘AUC’) and true-positive rate at 1% false-positive rate (‘TPR’). On Stable Diffusion, the baselines of Kim and Lee [15] are our runs of their official code (compared to their reported numbers in App. § 16); on the DiT, the 50k- and 150k-step references of Δhθ~\Delta h_{\widetilde{\theta}} and Δsθ~\Delta s_{\widetilde{\theta}} are selected by development AUROC. †Reported by Kim and Lee [15] on all prompts.

The 0.023 AUC gap on SD v1.4 arises entirely from prompts that copy on only some or none of the four seeds: on prompts that produce a verified template or matching verbatim copy, JaRS+ and all four contrasts achieve 1.000 AUC (App. § 16).

Conclusion

We have shown that memorized regions in diffusion models can be localized accurately from the row sums of the denoiser Jacobian (JaRS) without any contrast prompt or reference checkpoint. Combining JaRS with an early-step image-level detection score yields JaRS+, which outperforms contrast-based localization and enables region-restricted mitigation (App. § 18).

Limitations and future work. JaRS evaluates the denoiser at an intermediate noise level of a multi-step trajectory; extending it to single-step generators, such as distilled models, is left for future work. Further, JaRS+ selects (ℓ,d,τ)(\ell, d, \tau) on a small labeled development split, whereas the published configuration of Kim and Lee [15] requires none on Stable Diffusion; τ\tau can be calibrated from non-memorized images alone (App. § 11), but ℓ\ell and dd still require memorized examples.

LLM Usage Statement

Large language models were used to assist with editing and polishing the prose of the manuscript and with writing utility scripts for data processing and plotting. All theoretical derivations, experimental designs, implementations, and empirical analyses were conceived, verified, and validated by the authors.

Proof of prop:smooth

Proof. Using Eq. (1), conditioned on cc, the noisy observations in RR are independent with xt,j∣c∼N(αtc,σt2)x_{t,j} \mid c \sim\mathcal{N}(\alpha_t c, \sigma_t^2) for each j∈Rj \in R. Using Eq. (2.116) of Bishop [2], the posterior distribution of c∣xtc \mid x_t is Gaussian with precision and mean given by

Λ=1τ2+Nαt2σt2,\Lambda= \frac{1}{\tau^2} + \frac{N\alpha_t^2}{\sigma_t^2},
E[c∣xt]=Λ−1(μτ2+αtσt2∑j∈Rxt,j)=σt2μ+αtτ2∑j∈Rxt,jσt2+Nαt2τ2.\mathbb{E}[c \mid x_t] = \Lambda^{-1}\left(\frac{\mu}{\tau^2} + \frac{\alpha_t}{\sigma_t^2}\sum_{j\in R}x_{t,j}\right) = \frac{\sigma_t^2\mu+\alpha_t\tau^2\sum_{j\in R}x_{t,j}}{\sigma_t^2+N\alpha_t^2\tau^2}.

Since x0,i=cx_{0,i}=c for every i∈Ri\in R, the scaled optimal denoiser is Di(xt)=αtE[x0,i∣xt]=αtE[c∣xt]D_i(x_t)=\alpha_t\mathbb{E}[x_{0,i}\mid x_t]=\alpha_t\mathbb{E}[c\mid x_t]. Differentiating Di(xt)D_i(x_t) with respect to xt,jx_{t,j} for any j∈Rj\in R yields

Jij(xt)=∂Di(xt)∂xt,j=αt2τ2σt2+Nαt2τ2=gN.J_{ij}(x_t)=\frac{\partial D_i(x_t)}{\partial x_{t,j}}=\frac{\alpha_t^2\tau^2}{\sigma_t^2+N\alpha_t^2\tau^2}=\frac{g}{N}.

Setting j=ij=i gives Jii(xt)=g/NJ_{ii}(x_t)=g/N, and summing over the NN pixels j∈Rj\in R gives ∑j∈RJij(xt)=N⋅(g/N)=g\sum_{j\in R}J_{ij}(x_t)=N\cdot(g/N)=g. □

Benchmark and Baseline Details

DiT benchmark. We train the unconditional DiT-B/2 [21] using flow matching [16] for 200k steps (batch size 128) on the 70,000 FFHQ [14] face images at 256×256256 \times256 resolution, encoded into 4×32×324 \times32 \times32 latents via the Stable Diffusion autoencoder. The planted watermark has a size of 80×3280 \times32 pixels and is pasted into 7,000 (10%) of the training images. We generate 12,000 samples from the trained model using 50 Euler steps. To label the generated samples, we compute the maximum normalized cross-correlation (NCC) of each sample against the watermark template (which scores ≥0.979\ge0.979 on watermarked training images and ≤0.28\le0.28 on clean training images). A generated sample is labeled as watermarked (WM) if its watermark NCC exceeds 0.6 (occurring in ∼4.5%\sim4.5\% of samples), with the exact watermark footprint at the detected location serving as its ground-truth mask. A sample within a root-mean-square pixel distance of 0.05 from any duplicated training image is labeled a near-copy with an all-ones mask, and a sample with watermark NCC below 0.3 and distance above 0.1 from all duplicated images is labeled non-memorized (N) with an empty mask. Following the scale of the Stable Diffusion benchmarks, we retain the first 300 watermarked samples, the first 300 non-memorized samples, and all 139 near-copies.

Evaluation protocol. Every map (ours and the baselines’) is bilinearly upsampled from latent resolution to the 256×256256 \times256 resolution of the ground-truth masks, as in the code of Kim and Lee [15], and the Stage 1 score s^d\hat{s}_d of JaRS+ is computed on the upsampled map. The binarization threshold is chosen among 1,001 uniformly spaced levels in [0,1][0,1]. The resulting test splits contain 278 (TV), 278 + 278 (TV+N), and 3×1863 \times186 (All) instances for SD v1.4; 471 (TV) and 400 + 400 (TV+N) for SD v2.1; 402 (TV) and 3×1173 \times117 (All) for Realistic Vision; and 242 (WM), 241 + 241 (WM+N), and 3×1023 \times102 (All) for the DiT.

Baselines. For Realistic Vision, where Kim and Lee [15] do not specify a reference checkpoint, we use SD v1.1. On the DiT, we saved intermediate checkpoints at 5k, 20k, 50k, 100k, and 150k steps during our own training run. Because the final sampling step of Stable Diffusion has noise-to-signal ratio σt/αt=0.041\sigma_t/\alpha_t=0.041 whereas the final step of the DiT reaches 0.020, we evaluate the baselines on the DiT at step 48 (σt/αt=0.042\sigma_t/\alpha_t=0.042), which matches the noise level of their published configuration without tuning on any split. To give the checkpoint contrasts the strongest fair baseline on the DiT, we evaluate all five saved checkpoints on the development split and select the best reference checkpoint separately for each method and task: by development All mIoU for localization (which selects the 5k-step checkpoint for both Δhθ~\Delta h_{\tilde{\theta}} and Δsθ~\Delta s_{\tilde{\theta}}) and by development AUROC for detection (which selects the 50k-step checkpoint for Δhθ~\Delta h_{\tilde{\theta}} and the 150k-step checkpoint for Δsθ~\Delta s_{\tilde{\theta}}).

Signed Row Sum versus the L1 Row Norm

A natural alternative to the signed row sum is the ℓ1\ell_1 norm ∑j∣J(c,i),(c,j)∣\sum_j |J_{(c,i),(c,j)}| row, which vanishes if and only if the entire row is zero. We evaluated this variant on the full Jacobians of § 5.2, together with 20 development images of SD v2.1 (10 template verbatims and 10 non-memorized images) at its localization step ℓ=15\ell= 15, replacing each row sum in (10) by the ℓ1\ell_1 norm. On SD v2.1, the row sums behave as on the other two models: their median is 0.86 on non-memorized images and 0.89 outside the copied template, and falls to 0.50 inside the template region. The ℓ1\ell_1 norm, in contrast, does not concentrate around a common value on non-memorized images (Figure 3, bottom; median 1.51 on the DiT and 1.13 and 1.12 on SD v1.4 and v2.1) and overlaps more with the memorized pixels. Table 3 confirms that the ℓ1\ell_1 variant separates memorized from non-memorized pixels less accurately than the row sum for every model and category except the DiT near-copies, where both are perfect. The gap is largest on the whole-image copies of SD v1.4 and on the DiT watermark. We therefore use the signed row sum throughout.

ModelMemorized pixelsRow sumℓ1\ell_1 norm
DiT-B/2Watermark0.9980.900
Near-copy1.0001.000
SD v1.4Template (TV)0.9800.908
Matching verbatim0.9170.772
Retrieved verbatim0.8620.574
SD v2.1Template (TV)0.9550.934

Table 3. Pixel-level AUROC between memorized and non-memorized pixels for the JaRS map (10) and its ℓ1\ell_1 variant, which replaces each row sum by the ℓ1\ell_1 norm of the row, on the development images of Figure 3.

Figure 3. Per-channel row sum ∑jJij\sum_j J_{ij} (top) and ℓ1\ell_1 row norm ∑j∣Jij∣\sum_j |J_{ij}| (bottom) of the full denoiser Jacobian at the localization step of each model (DiT-B/2 at step 35, SD v1.4 and SD v2.1 at step 15), across all pixels of non-memorized images and whole-image copies, and inside the copied region of watermarked samples and template verbatims. The development images are those of § 5.2, together with 10 template verbatims and 10 non-memorized images of SD v2.1. The vertical dotted line at 1 marks the row sum of a mean-preserving averaging filter.

Ablation of the Image-Level Detection Stage

Table 4 compares the default two-stage JaRS+ pipeline reported in the main paper against two natural variants on the test splits:

Detection VariantSD v1.4 TVSD v1.4 TV+NSD v1.4 AllSD v2.1 TVSD v2.1 TV+NRealistic Vision TVRealistic Vision AllDiT (unconditional) WMDiT (unconditional) WM+NDiT (unconditional) All
JaRS+ (tuned τ\tau, main paper)0.9690.9750.9400.9800.9890.9290.8490.6380.7990.869
JaRS+ (τ\tau at 5% dev FPR)0.9690.9230.9060.9800.9590.9440.8540.4630.7250.817
JaRS (no Stage 1, step ℓ\ell)0.9690.6190.6370.9800.5710.9490.6120.8360.8730.909

Table 4. Ablation of Stage 1 (image-level detection) variants on the test splits (mIoU). All hyperparameters are selected on the development split of each model. ‘JaRS+ (tuned τ\tau)’ chooses τ\tau to maximize development mIoU on the broadest balanced setting; ‘JaRS+ (τ\tau at 5% dev FPR)’ calibrates τ\tau using only non-memorized development images at the 95th percentile; and ‘JaRS’ returns the localization map stℓs_{t\ell} for every image without Stage 1 gating.

  1. JaRS+ (τ\tau at 5% dev FPR): Instead of selecting the threshold τ\tau to maximize development mIoU across both memorized and non-memorized development images, τ\tau is set to the 95th percentile of the detection score zdz_d computed exclusively on non-memorized development images (guaranteeing a 5% false-positive rate on the development split without requiring any labeled memorized examples to calibrate τ\tau).

  1. JaRS (no Stage 1): Returns the pixel-level JaRS map sts_t for every image without an image-level detection gate.

In settings that contain only memorized images (TV), JaRS achieves the highest performance (0.969, 0.980, and 0.949 on SD v1.4, SD v2.1, and Realistic Vision). In mixed text-to-image settings that include non-memorized prompts (TV+N and All), however, JaRS produces false-positive regions on confident backgrounds of unmemorized images, showing why Stage 1 in JaRS+ is essential. Notably, calibrating τ\tau at a 5% false-positive rate using only non-memorized development images remains competitive across all three text-to-image models and slightly improves over the mIoU-tuned threshold on Realistic Vision (TV: 0.944 vs. 0.929; All: 0.854 vs. 0.849), where the memorized development set is small. On the unconditional DiT, JaRS achieves the highest accuracy across all three settings (0.836 WM, 0.873 WM+N, and 0.909 All), because the 90th-percentile statistic zdz_d rejects a fraction of samples containing the small (4%-area) watermark.

To understand where JaRS+ loses accuracy on Realistic Vision All (§ 5.3), Table 6 breaks down the pass rates of the Stage 1 detector (zd>τz_d > \tau) by image category and compares the resulting mIoU against the same Stage 2 JaRS maps paired with an oracle image-level gate (‘Oracle Gate’). On Realistic Vision, the pixel-level JaRS maps at step ℓ=35\ell= 35 are highly accurate: with an oracle gate, they achieve 0.945 mIoU on All, nearly matching Δh∅\Delta h_{\varnothing} (0.951). The drop to 0.849 occurs because Stage 1 rejects 26.5% of the global whole-image copies on the test split. Inspecting the development split explains why: Realistic Vision has only 28 whole-image copies in its development split, on which zdz_d achieved a near-perfect development AUROC of 0.999, yielding a threshold τ\tau that proved overly conservative on the larger test split. On SD v1.4, where the development split contains 45 whole-image copies, Stage 1 accepts 92.5% of global copies and 100% of TV copies while rejecting 97.8% of non-memorized images, coming within 0.020 of the oracle gate (0.940 vs. 0.960).

ModelTVGlobalNon-mem.Oracle Gate (JaRS+)
SD v1.41.0000.9250.0220.960 (0.940)
Realistic Vision0.9740.7350.0510.945 (0.849)

Table 6. Disentangling Stage 1 (detection) and Stage 2 (localization) on the test All split. Columns 2–4 report the fraction of images in each category that pass the Stage 1 gate (zd>τz_d > \tau). Column 5 reports the mIoU of the Stage 2 JaRS localization maps when paired with an oracle image-level detector (with the actual two-stage JaRS+ mIoU in parentheses).

Tuning the Baselines in the Same Way as JaRS+

In Table 1, the baselines of Kim and Lee [15] are evaluated in their published configuration: at the final sampling step and without an image-level gate. JaRS+, in contrast, selects its steps and its threshold on the development split. To check that this difference does not drive our conclusions, we tune every method in the same way.

Protocol. For every method and model, we use the development split only, with the same candidates and selection rules for all methods.

  • Ungated. The step ℓ\ell of the map is chosen from {5,10,…,45,49}\{5,10,\ldots,45,49\} to maximize development mIoU on TV (WM on the DiT).

  • Gated (‘+’). Each method is given the two-stage form of JaRS+ (§ 4.3). The image-level score is a statistic qq of the method’s own map at step dd, with q∈{mean⁡,Q0.5,Q0.9,Q0.99}q \in\{\operatorname{mean}, Q_{0.5}, Q_{0.9}, Q_{0.99}\} and d∈{0,5,…,45,49}d \in\{0,5,\ldots,45,49\}. The pair (d,q)(d,q) maximizes development AUROC between memorized and non-memorized images, and τ\tau maximizes development mIoU, both on the broadest balanced setting, as for JaRS+. Ties are broken toward the earlier step.

JaRS+ follows the same rule, so here it may select a percentile other than 0.9. On the DiT, the checkpoint contrasts use the 5k-step reference of Table 1.

Results. Table 7 shows three findings. First, tuning the step changes little for the four contrasts on Stable Diffusion and Realistic Vision, where the development split selects their final step or a nearby one, but it strongly improves Δhθ~\Delta h_{\tilde{\theta}} on the DiT (0.666 to 0.849 on All). Second, the image-level gate improves every baseline whose map responds to non-memorized images (e.g., Δsθ~\Delta s_{\tilde{\theta}} on All of SD v1.4, from 0.668 to 0.912), yet JaRS+ remains best in eight of ten settings; the two exceptions, All on SD v1.4 and on Realistic Vision, are the settings where JaRS+ also trails in Tab. 1. Third, the development AUROC of almost every gate reaches 1.000, so the selected (d,q)(d,q) is often decided by the tie rule, and a gate can reject memorized test images (e.g., Δhθ~+\Delta h_{\tilde{\theta}}+ loses about 0.03 on TV of both SD models).

MethodSD v1.4 TVSD v1.4 TV+NSD v1.4 AllSD v2.1 TVSD v2.1 TV+NSD v2.1 AllRealistic Vision TVRealistic Vision AllDiT (unconditional) WMDiT (unconditional) WM+NDiT (unconditional) All
Ungated, step ℓ\ell tuned
Raw diagonal0.7790.6080.7200.8120.5750.8440.6490.3270.5000.632
Δh∅\Delta h_{\varnothing}0.9000.9500.9450.9450.8870.9300.951not available
Δhθ~\Delta h_{\tilde{\theta}}0.9130.9030.8840.9480.8760.9360.8800.7900.8600.849
Δs∅\Delta s_{\varnothing}0.8440.7090.7330.9110.5790.8760.903not available
Δsθ~\Delta s_{\tilde{\theta}}0.8850.7640.6680.9210.8770.8600.6260.6300.5700.417
JaRS0.9690.6190.6370.9800.5710.9490.6120.8360.8730.909
Gated (‘+’), (ℓ,d,q,τ)(\ell,d,q,\tau) tuned
Raw diagonal+0.7680.8770.9090.8110.8960.8440.9140.0380.5000.662
Δh∅+\Delta h_{\varnothing}+0.9000.9500.9450.9430.9670.9300.951not available
Δhθ~+\Delta h_{\tilde{\theta}}+0.8790.9360.9260.9180.9560.8950.9270.7900.8700.872
Δs∅+\Delta s_{\varnothing}+0.8410.9210.9190.9110.9460.8760.922not available
Δsθ~+\Delta s_{\tilde{\theta}}+0.8820.9410.9120.9210.9600.8470.8440.6060.8030.738
JaRS+0.9690.9830.9310.9800.9890.9440.8500.8190.8950.928

Table 7. Localization mIoU on the test splits when every method is tuned on the development split in the same way. Top: each map at its selected step ℓ\ell, without an image-level gate. Bottom: each method with its own image-level gate, selected as for JaRS+. Bold marks the best value in each column within each block.

Selection of the Detection Step dd

Tab. 5 reports the image-level detection AUROC of zd=Q0.9(std⁡td)z_d = Q_{0.9}(\operatorname{std}_{t_d}) across all candidate steps d∈{0,5,…,45}d \in\{0,5,\ldots,45\} on the balanced development splits of the three text-to-image models. Across all three models, image-level separation between memorized and non-memorized trajectories peaks in the earliest denoising steps (highest noise levels), where the text prompt immediately locks in memorized structure while non-memorized trajectories remain sensitive to the initial Gaussian noise. Accordingly, our selection rule chooses d=5d=5 on all three text-to-image models (on SD v2.1, steps 0 and 5 differ only in the fourth decimal place). By contrast, on the unconditional DiT, there is no text prompt to lock in the small watermark during the earliest steps; instead, the watermark emerges in the JaRS map at intermediate noise levels (Sec. G), and the development AUROC peaks at d=30d=30 (0.984).

The JaRS Map Across Noise Levels

Localization accuracy across noise levels. Figure 4 plots the test mIoU of the JaRS map sts_t as a function of the noise-to-signal ratio σt/αt\sigma_t/\alpha_t, which places the variance-preserving schedule of Stable Diffusion and the flow-matching schedule of the DiT on a unified axis. The empirical curves closely follow the theoretical prediction of Proposition 1. At the highest noise levels (σt/αt≫1\sigma_t/\alpha_t \gg1), the denoiser discards its noisy input everywhere (g→0g \to0), so on SD v1.4 and v2.1 the map scores 0.563 and 0.648 (the trivial accuracy of an all-ones mask). As noise decreases, mIoU rises sharply to a broad, stable plateau centered around σt/αt≈3\sigma_t/\alpha_t \approx3 for SD v1.4 and v2.1, and around σt/αt∈[0.3,1]\sigma_t/\alpha_t \in[0.3, 1] for Realistic Vision and the DiT—before declining as σt/αt→0\sigma_t/\alpha_t \to0, where g→1g \to1 everywhere. Notably, at the final sampling step (σt/αt≈0.04\sigma_t/\alpha_t \approx0.04), where Kim and Lee [15] evaluate their contrast maps, a single model’s Jacobian approaches the identity everywhere and the JaRS map returns to the trivial baseline (0.563 and 0.648). Identifying that contrast-free localization lives at intermediate σt/αt\sigma_t/\alpha_t rather than at the final step is thus a central benefit of our analysis: across all four models, the localization step ℓ\ell chosen on the development split (circled in Figure 4) lies squarely on the high-accuracy plateau, and on Stable Diffusion the localization map is available after just 15 of the 50 sampling steps.

Figure 4. Effect of noise level on localization accuracy. Test mIoU of the JaRS map sts_t on locally memorized instances (TV for text-to-image models, WM for the DiT) plotted against the noise-to-signal ratio σt/αt\sigma_t/\alpha_t at which the map is evaluated. Black circles mark the localization step ℓ\ell selected on the development split.

The JaRS map along the sampling trajectory. Figure 5 tracks the JaRS map sts_t across all 50 denoising steps for every development instance, directly illustrating the two-timescale separation between early image-level detection and intermediate pixel-level localization that motivates JaRS+ (§ 4.3).

Early image-level separation (Stage 1). On all three text-to-image models (Figure 5, top), the image-level score zt=Q0.9(st)z_t = Q_{0.9}(s_t) is systematically higher on memorized trajectories than on non-memorized trajectories at every step prior to the end of sampling, with the widest separation occurring in the earliest steps. On SD v1.4, for example, the median ztz_t on template verbatims is 3.46 at step 5 versus 1.52 on non-memorized images (compared to 1.07 vs. 0.42 by step 40). Retrieved verbatims (RV) begin at a lower median at step 0 (1.92 on SD v1.4 and 4.19 on Realistic Vision, compared to 3.42 and 5.28 for template verbatims) before catching up to the other memorized categories by step 5 on Realistic Vision and step 10 on SD v1.4. This trajectory behavior explains why the development detection AUROC in Table 5 is lower at step 0 than at step 5 and peaks at d=5d = 5 across all three text-to-image models. At the final step (49), as the noise level σt/αt\sigma_t/\alpha_t approaches zero, the median ztz_t of every category collapses to at most 0.12, matching the prediction of Proposition 1.

Candidate Detection Step dd051015202530354045
Stable Diffusion v1.40.9550.9880.9870.9680.9360.8870.8610.8500.8200.809
Stable Diffusion v2.10.99981.0000.9960.9740.9110.8260.7970.7860.7780.763
Realistic Vision v5.10.9160.9990.9830.9120.8690.8440.8300.8170.8070.814

Table 5. Development AUROC of the Stage 1 detection score zd=Q0.9(std⁡t)z_d = Q_{0.9}(\operatorname{std}_{t} ) across candidate denoising steps dd, evaluated on the balanced development split (‘All’ for SD v1.4 and Realistic Vision, ‘TV+N’ for SD v2.1). Bold indicates the selected step dd.

Intermediate pixel-level separation (Stage 2). The pixel-level curves (Figure 5, bottom) explain why accurate spatial localization requires waiting until an intermediate step ℓ>d\ell> d. In the earliest denoising steps, the unmemorized pixels outside the copied template of a TV image exhibit nearly the same elevated JaRS scores as the pixels inside the template: on SD v1.4 at step 0, the median score is 2.19 outside the template versus 2.66 inside. Thus, at very high noise, a memorized prompt elevates the JaRS map across the entire image—which provides a strong signal for image-level detection, but poor contrast for segmenting the copied region. By step ℓ=15\ell= 15 on SD v1.4 and SD v2.1, the pixels outside the template relax to the baseline level of ordinary non-memorized images (medians of 0.41 and 0.48, matching 0.47 and 0.44 on non-memorized images), whereas the pixels inside the copied template remain elevated (2.33 and 2.05), creating a sharp spatial boundary. On Realistic Vision, the unmemorized background relaxes more gradually and reaches the non-memorized level only around step 30 (0.70 vs. 0.67), consistent with its later selected localization step ℓ=35\ell= 35. Beyond step 20, the median score outside the template dips slightly below zero on SD v1.4 and SD v2.1 (reaching −0.16-0.16) where sharpening side lobes cause the per-channel row sums to slightly exceed 1; these negative values are clipped at 0 in the evaluation protocol (§ 5.1).

Trajectory dynamics on the unconditional DiT. On the unconditional DiT (Figure 5, rightmost column), global and local memorization emerge on different timescales. Whole-image near-copies separate from non-memorized samples almost immediately: the AUROC of ztz_t between near-copies and non-memorized samples reaches 0.960 at step 5 and 1.000 from step 15 onward. In contrast, the planted watermark covers only 3.8% of the image area on average, so it barely shifts the 90th spatial percentile ztz_t at step 30, the median z30z_{30} is 0.68 on watermarked samples versus 0.54 on non-memorized samples). Within each watermarked sample, pixel-level separation between the watermark and the unmemorized face background develops only at intermediate noise: the mean within-image pixel AUROC rises from 0.547 at step 0 and 0.769 at step 10 to 0.949 at step 15, and exceeds 0.986 from step 20 onward. Because no text prompt locks in the watermark at t≈1t \approx1, image-level separation between watermarked and non-memorized samples peaks later at step 30 (AUROC 0.925), which—combined with the near-copies—leads the development split to select d=30d = 30.

Sensitivity of Checkpoint Contrasts to the Reference Checkpoint

Kim and Lee [15] note that the reference checkpoint θ~\tilde{\theta} in Δhθ~\Delta h_{\tilde{\theta}} must be “relatively less overfitted” than the target model θ\theta, cautioning that overly early checkpoints fail to capture general data-driven curvature while overly late checkpoints already exhibit memorization-driven curvature. For the unconditional DiT (trained for 200k steps), we evaluated both Δhθ~\Delta h_{\tilde{\theta}} and Δsg\Delta s_g at step 48 (§ 5.1) across all five intermediate checkpoints saved during training (5k, 20k, 50k, 100k, and 150k steps). Table 8 reports the resulting development and test performance for both localization and sample-level detection.

Reference θ~\tilde{\theta}Localization (mIoU)Detection (AUC)
Dev AllWMWM+NAllDevTest
Δhθ~\Delta h_{\tilde{\theta}}5k (2.5%)0.6660.4390.5000.6660.8000.803
20k (10%)0.6630.5030.5000.6630.8590.860
50k (25%)0.6610.4570.5000.6550.8660.867
100k (50%)0.6340.3690.5000.6160.8640.864
150k (75%)0.5310.2430.5000.5260.8490.833
Δsθ~\Delta s_{\tilde{\theta}}5k (2.5%)0.5070.3760.5010.5230.8400.839
20k (10%)0.4730.4610.5040.4820.9000.903
50k (25%)0.4230.4770.5060.4340.9180.917
100k (50%)0.4020.5110.5150.4090.9320.932
150k (75%)0.3950.5390.5300.4020.9400.943
JaRS+ (no reference)–0.6380.7990.869–0.902

Table 8. Sensitivity of the checkpoint contrasts Δhθ~\Delta h_{\tilde{\theta}} and Δsθ~\Delta s_{\tilde{\theta}} [15] on the unconditional DiT (trained for 200k steps) across all saved intermediate checkpoints θ~\tilde{\theta} (fraction of training completed shown in parentheses), evaluated at step 48 (σt/αt=0.042\sigma_t/\alpha_t = 0.042, matching the final step of Stable Diffusion) with 16 Hutchinson probes. The ‘Dev’ columns are used to select the reference checkpoint for each task (bold indicates the selected checkpoint); the remaining columns report test-split performance. Sample-level detection follows the protocol of § 5.4.

For localization, the earliest checkpoint (5k steps) achieves the highest development All mIoU for both contrasts and is therefore used in Tab. 1. On WM alone, Δhθ~\Delta h_{\tilde{\theta}} peaks at the 20k checkpoint (0.503 mIoU, which is not identified by the development split) and degrades sharply as the reference checkpoint advances through training.

ing, dropping to 0.243 at 150k steps. Across all five checkpoints, neither Δhθ~\Delta h_{\tilde{\theta}} nor Δsθ~\Delta s_{\tilde{\theta}} separates watermarked samples from non-memorized samples on WM+N (reaching at most 0.530 mIoU), and every checkpoint falls well short of JaRS+ across all three localization settings (by at least 0.099 mIoU).

For sample-level detection, the optimal reference checkpoint differs both between the two contrasts and from the checkpoint that is best for localization: Δhθ~\Delta h_{\tilde{\theta}} peaks at the 50k checkpoint (0.867 test AUC) and remains below JaRS+ (0.902 AUC) for every reference, whereas Δsθ~\Delta s_{\tilde{\theta}} improves monotonically with later checkpoints from 0.839 AUC (5k) to 0.943 AUC (150k), matching JaRS+ at 20k steps (0.903) and surpassing it at 50k–150k. Crucially, the 150k checkpoint that yields the best detection accuracy for Δsθ~\Delta s_{\tilde{\theta}} yields the worst localization accuracy on All (0.402 mIoU), demonstrating that no single reference checkpoint serves both tasks well.

Prompt-Level Memorization Detection

Protocol. On Stable Diffusion, the positive set consists of the memorized prompts cataloged by Webster [33]: all 500 prompts for SD v1.4 and the 219 prompts labeled as verbatim copies for SD v2.1, while the negative set consists of the 500 non-memorized prompts from the localization benchmark. All metrics are reported on the held-out test split (400 memorized vs. 400 non-memorized prompts on SD v1.4, and 174 vs. 400 on SD v2.1). On the DiT, we score individual test samples (344 memorized watermark/near-copy samples vs. 241 non-memorized samples) using z30z_{30}. For JaRS+, the detection step dd is the one already selected for localization (App. F), with no re-tuning. For the baselines on Stable Diffusion, we follow the protocol described in Kim and Lee [15 Sec. 5.2] and run their official code. On SD v2.1, our numbers match theirs closely (0.995 and 0.994 AUC for Δh∅\Delta h_{\varnothing} and Δhθ~\Delta h_{\widetilde{\theta}}, vs. 0.995 and 0.996 reported). On SD v1.4, Δh∅\Delta h_{\varnothing} achieves 0.969 AUC on our test split (and 0.969 across all 1,000 prompts), compared to 0.997 reported in Kim and Lee [15]; similarly, Δhθ~\Delta h_{\widetilde{\theta}} achieves 0.951 vs. 0.989.

Per-category breakdown. Table 9 decomposes the prompt-level detection results of § 5.4 across the prompt subcategories verified by Kim and Lee [15]. On every prompt that produces a verified template verbatim (TV) or matching verbatim (MV) copy under the four evaluated seeds, JaRS+ and all four contrasts achieve an AUC of 1.000. The small gap in overall AUC on SD v1.4 arises solely from retrieved verbatims (RV, which copy under only one or two of the four seeds) and from candidate prompts (‘N-ver.’) in the catalog of Webster [33] that do not produce a copy under any of the four seeds. On the unconditional DiT, all methods detect whole-image near-copies with 1.000 AUC, and the methods differ only on watermarked samples: JaRS+ achieves 0.861 AUC, compared to 0.811 for Δhθ~\Delta h_{\widetilde{\theta}} (50k reference) and 0.919 for Δsθ~\Delta s_{\widetilde{\theta}} (150k reference). Overall, the DiT AUC is 0.902 with 95% CI [0.877, 0.924] for JaRS+, 0.867 [0.836, 0.895] for Δhθ~\Delta h_{\widetilde{\theta}}, and 0.943 [0.925, 0.960] for Δsθ~\Delta s_{\widetilde{\theta}}.

MethodOverall AUC [95% CI]TV (147)MV (44)RV (28)N-ver. (181)Overall AUC [95% CI]TV (145)N-ver. (29)
Raw diagonal [15]0.906 [0.882, 0.928]0.9600.9660.9270.8450.871 [0.842, 0.899]0.8820.819
Δh∅\Delta h_{\varnothing} [15]0.969 [0.955, 0.982]1.0001.0000.9990.9320.995 [0.989, 1.000]1.0000.972
Δhθ~\Delta h_{\tilde{\theta}} [15]0.951 [0.931, 0.970]1.0001.0000.9990.8920.994 [0.986, 0.999]1.0000.964
Δs∅\Delta s_{\varnothing} [15]0.969 [0.954, 0.983]1.0001.0001.0000.9310.994 [0.986, 0.999]1.0000.966
Δsθ~\Delta s_{\tilde{\theta}} [15]0.957 [0.939, 0.974]1.0001.0001.0000.9050.993 [0.986, 0.999]1.0000.961
JaRS+ (ours)0.946 [0.930, 0.962]1.0001.0000.9270.8930.992 [0.982, 0.999]1.0000.954

Table 9. Per-category breakdown of prompt-level detection on the test split (extending Table 2). Each category’s AUC is computed against all 400 non-memorized test prompts. Prompt subcategories follow the verification of Kim and Lee [15] across the four evaluated seeds (‘TV’ and ‘MV’: prompt copies on at least one of the four seeds; ‘RV’: retrieved verbatim; ‘N-ver.’: prompts from the list of Webster [33] that produce no verified copy under the four seeds; test prompt counts in parentheses). 95% confidence intervals are computed via 2,000 bootstrap resamples.

Realistic Vision. Because Kim and Lee [15] do not report prompt-level detection on Realistic Vision v5.1, we evaluate it under the same protocol. Since Realistic Vision is fine-tuned from Stable Diffusion v1.5, the positive set comprises the full catalog of 500 SD v1 memorized prompts from Webster [33].

As shown in Table 10, JaRS+ achieves 0.899 AUC, performing comparably to the checkpoint contrasts (Δhθ~\Delta h_{\widetilde{\theta}}: 0.907; Δsθ~\Delta s_{\widetilde{\theta}}: 0.918), trailing the null-prompt contrasts (Δh∅\Delta h_{\varnothing}: 0.937; Δs∅\Delta s_{\varnothing}: 0.960), and outperforming the raw diagonal by a wide margin (0.631 AUC). The per-category breakdown mirrors the pattern observed on SD v1.4: on prompts that generate a verified template verbatim (TV), JaRS+ and all four contrasts achieve at least 0.995 AUC, and the majority of the overall gap stems from the 198 candidate test prompts (‘N-ver.’) that produce no verified copy under the four evaluated seeds (0.807 vs. 0.918 for Δs∅\Delta s_{\varnothing}). Unlike on SD v1.4, however, JaRS+ also exhibits a small gap on matching verbatims (0.964 vs. 1.000 AUC). These correspond to the same whole-image copies that receive lower z5z_5 scores in the localization benchmark (Table 6), confirming that the Stage 1 score z5z_5 accounts for both behaviors on Realistic Vision.

MethodOverall AUC [95% CI]TPRTV (156)MV (35)RV (11)N-ver. (198)
Raw diagonal [15]0.631 [0.591, 0.670]0.0180.6340.7430.5380.613
Δh∅\Delta h_{\varnothing} [15]0.937 [0.917, 0.957]0.8651.0001.0001.0000.873
Δhθ~\Delta h_{\tilde{\theta}} [15]0.907 [0.882, 0.931]0.8251.0001.0001.0000.813
Δs∅\Delta s_{\varnothing} [15]0.960 [0.942, 0.976]0.9351.0001.0001.0000.918
Δsθ~\Delta s_{\tilde{\theta}} [15]0.918 [0.894, 0.941]0.7901.0000.9920.9680.838
JaRS+ (ours)0.899 [0.875, 0.922]0.6400.9950.9640.9770.807

Table 10. Prompt-level memorization detection on Realistic Vision v5.1 (test split), measured by AUROC (‘AUC’) and true-positive rate at a 1% false-positive rate (‘TPR’) across 400 memorized and 400 non-memorized test prompts. Prompt subcategories follow the verification of Kim and Lee [15] on Realistic Vision (test prompt counts in parentheses), with each category’s AUC computed against all 400 non-memorized test prompts. 95% confidence intervals are computed via 2,000 bootstrap resamples. The baselines of Kim and Lee [15] use 16 Hutchinson probes for the curvature maps and SD v1.1 as the checkpoint reference, matching Table 1.

Computation: Differentiation Mode, Precision, and Cost

In this section, we discuss the computation of the JaRS map. One might ask whether the row sums could instead be computed via reverse-mode automatic differentiation (backpropagation). Applying reverse-mode differentiation to 1cTD(xt)1_{\mathrm{c}}^{\mathsf{T}}D(x_t) yields the column sums J(xt)T1cJ(x_t)^{\mathsf{T}}1_{\mathrm{c}} rather than the row sums J(xt)1cJ(x_t)1_{\mathrm{c}}. Although the Bayes-optimal Jacobian (5) is symmetric, the Jacobian of a trained neural network is in general not symmetric. Forward directional derivatives (12) are therefore necessary to extract the true filter row sums. Furthermore, the forward passes in (12) must be evaluated in full single precision (fp32, with TensorFloat-32 disabled): reduced-precision convolutions corrupt the finite-difference quotient by roughly 20% on Stable Diffusion, whereas in fp32 the central difference (12) matches exact forward-mode automatic differentiation to within a relative error of 10−410^{-4}.

Table 11 reports the cost of every method in Table 1. A JaRS map requires 8 forward passes and no backward pass, compared with the equivalent of about 21 forward passes for the raw diagonal and 41 for Δh∅\Delta h_{\varnothing} and Δhθ~\Delta h_{\widetilde{\theta}} on Stable Diffusion (a backward pass with respect to the input costs about 1.2 forward passes). On average over the broader test setting, JaRS+ requires 3.1–3.4 times fewer FLOPs than Δh∅\Delta h_{\varnothing} and Δhθ~\Delta h_{\widetilde{\theta}} on Stable Diffusion, and 4.8 times fewer than Δhθ~\Delta h_{\widetilde{\theta}} on the DiT. The wall-clock comparison on Stable Diffusion is closer, because the finite differences in (12) require fp32 while the code of Kim and Lee [15] runs in fp16: JaRS+ takes 0.62–0.75 s per image against 0.69–0.81 s for Δh∅\Delta h_{\varnothing} and 0.86–0.99 s for Δhθ~\Delta h_{\widetilde{\theta}}, and it is slower than the raw diagonal (0.43–0.50 s). In fp32, Δh∅\Delta h_{\varnothing} and Δhθ~\Delta h_{\widetilde{\theta}} take 2.8–3.8 s. On the DiT, where all maps run in fp32, JaRS+ takes 0.049 s against 0.213 s for Δhθ~\Delta h_{\widetilde{\theta}}. The score contrasts are the cheapest maps, with two forward passes, but they are less accurate (Table 1). Every map costs a fraction of generating the image itself.

MethodStepPrecisionStable Diffusion TFLOPsSD v1.4 (s)SD v2.1 (s)RV (s)DiT TFLOPsTime (s)
Raw diagonallastfp16/fp32\mathrm{fp16}/\mathrm{fp32}16.60.490.430.501.470.106
Δh∅\Delta h_{\varnothing}lastfp16\mathrm{fp16}33.30.810.690.81––
Δhθ~\Delta h_{\widetilde{\theta}}lastfp16/fp32\mathrm{fp16}/\mathrm{fp32}33.30.990.860.992.950.213
Δs∅\Delta s_{\varnothing}lastfp16\mathrm{fp16}1.60.040.030.04––
Δsθ~\Delta s_{\widetilde{\theta}}lastfp16/fp32\mathrm{fp16}/\mathrm{fp32}1.60.050.040.050.090.012
JaRS (one map)ℓ\ellfp32\mathrm{fp32}6.40.460.410.460.370.029
JaRS+ (both stages)d,ℓd,\ellfp32\mathrm{fp32}12.90.910.830.920.740.058
JaRS+ (mean, broadest setting)d,ℓd,\ellfp32\mathrm{fp32}9.7–10.60.750.620.730.610.049
Generating the imageallfp16/bf16\mathrm{fp16}/\mathrm{bf16}80.31.801.691.842.300.329

Table 11. Cost of each localization map per test image on one NVIDIA A6000 GPU. FLOPs are counted with PyTorch’s FLOP counter; they do not depend on the image and are the same for the three Stable Diffusion models. Times are medians over 20 test prompts (test samples on the DiT) with three repetitions each. Every map is computed as in Tab. 1: on Stable Diffusion, the maps of Kim and Lee [15] with their official fp16 code (one forward pass that keeps the graph and 16 Hutchinson vector-Jacobian products), and on the DiT with our fp32 implementation (16 Rademacher probes with central-difference Jacobian-vector products); JaRS in fp32. ‘Last’ is step 49 of 50 on Stable Diffusion and step 48 on the DiT. JaRS+ computes the Stage 2 map only for images accepted by Stage 1: ‘both stages’ is the cost of an accepted image, and ‘mean’ averages over the broadest balanced test setting (All for SD v1.4, Realistic Vision, and the DiT; TV+N for SD v2.1), on which Stage 1 accepts 65%, 50%, 59%, and 67% of the images. Generating the image (50 steps, with classifier-free guidance on Stable Diffusion) is listed for reference.

The selection of (ℓ,d,τ)(\ell,d,\tau) on the development split is a one-time cost per model. It requires JaRS maps at the 10 candidate steps for the 206, 220, 161, and 153 development images used for selection on SD v1.4, SD v2.1, Realistic Vision, and the DiT: 16, 15, 12, and 1 minutes on one GPU, plus 6, 6, 5, and 1 minutes to generate the development images. The published configuration of Kim and Lee [15] requires no development split on Stable Diffusion. On the DiT, the reference checkpoint of the checkpoint contrasts is selected among five saved checkpoints on the development split (§ 5.1), which also requires these checkpoints to have been saved during training.

Region-Restricted Memorization Mitigation

Method

Existing inference-time mitigations prevent copying by modifying the generation globally, for instance by inserting random tokens into the prompt (RTA) [30], optimizing the prompt embedding to minimize the text-conditional noise prediction [34], or optimizing the initial noise latent [12]. While effective at suppressing the memorized template, a global intervention alters the entire image, destroying the unmemorized content that the user wished to keep.

Given the JaRS localization map stϵs_{t_\epsilon}, we can restrict any prompt-based mitigation exclusively to the copied region. Let R∈{0,1}H×WR \in\{0,1\}^{H \times W} be the binary latent mask obtained by thresholding stϵ(i)s_{t_\epsilon}(i) and dilating the mask by 2 latent pixels, and let z0z_0 denote the clean latent of the original generation. We resample the image using the inpainting schedule of RePaint [17], initializing the trajectory with fresh Gaussian noise inside RR. At each denoising step, the guided noise prediction applies the mitigated prompt conditioning y~\tilde{y} strictly inside RR while retaining the original prompt yy outside RR:

ϵ^=(1−R)⊙(ϵ∅+w(ϵy−ϵ∅))+R⊙(ϵ∅+w(ϵy~−ϵ∅))(13)\hat{\epsilon} = (1-R) \odot\left(\epsilon_{\varnothing} + w(\epsilon_y-\epsilon_{\varnothing})\right) + R \odot\left(\epsilon_{\varnothing} + w(\epsilon_{\tilde{y}}-\epsilon_{\varnothing})\right) \tag*{(13)}

where ϵ∅\epsilon_{\varnothing}, ϵy\epsilon_y, and ϵy~\epsilon_{\tilde{y}} denote the noise predictions under the null prompt, the original prompt yy, and the mitigated conditioning y~\tilde{y}, respectively, and ww is the classifier-free guidance scale. Following each denoising step to noise level t′t', the latent outside RR is replaced by the forward-noised original latent αt′z0+σt′ϵ\alpha_{t'} z_0 + \sigma_{t'}\epsilon (using a fixed noise realization ϵ\epsilon). This guarantees that the unmemorized region 1−R1-R remains faithful to the original generation while the memorized region RR is synthesized anew under y~\tilde{y}. In our experiments, we instantiate y~\tilde{y} with two standard global mitigations: RTA [30] and the optimized embedding of Wen et al. [34].

Experimental Setup and Results

We evaluate how effectively our localization maps enable targeted copy removal via the region-restricted resampling procedure of Sec. K.1 on the 278 test TV instances of Stable Diffusion v1.4. For each instance, we form the binary mask RR by thresholding the step 15 JaRS map at a threshold selected on the development split and dilating by 2 latent pixels; the resulting mask covers 99.9% of the groundtruth template TT on average (and at least 97.9% on every test instance). To compare against contrast-based localization, we also form RR in the same manner from the best-performing contrast on the development split, Δhθ~\Delta h_{\widetilde{\theta}} (which covers 97.6% of the template on average, but falls below 99% coverage on 9.4% of instances). We test two prompt modification strategies y~\widetilde{y} at default and strong settings: random token addition (RTA [30], inserting 4 or 8 random tokens) and prompt embedding optimization (Wen et al. [34],

stopping once the objective drops to 30% or 15% of its initial value). We evaluate four metrics:

  • Copied (↓): The fraction of generated images that still reproduce the ground-truth template under the benchmark verification criterion.

  • LPIPSRˉ_{\bar{R}} and LPIPSTˉ_{\bar{T}} (↓): The perceptual distance [35] between the mitigated image and the original generation outside the predicted mask RR and outside the ground-truth template TT, respectively (note that Tˉ\bar{T} evaluates the exact same pixels across all methods).

  • CLIP (↑): The CLIP similarity score [23] between the mitigated image and the original text prompt yy.

Table 12 presents the quantitative results, and qualitative examples are shown in Figure 6. When the prompt embedding optimization of Wen et al. [34] is restricted to the JaRS region RR, it simultaneously improves all metrics over whole-image mitigation: at the default setting, it reduces the copy rate from 20.5% to 9.0%, reduces perceptual change outside RR by 4.5× (LPIPSRˉ\mathrm{LPIPS}_{\bar{R}} from 0.708 to 0.158, and LPIPSTˉ\mathrm{LPIPS}_{\bar{T}} from 0.689 to 0.239), and improves prompt alignment by +1.8 CLIP points (22.2 vs. 20.4). Similarly, restricting RTA [30] to the JaRS region matches the whole-image copy rate (43.2%) while reducing background distortion by nearly 5× (LPIPSRˉ\mathrm{LPIPS}_{\bar{R}} from 0.444 to 0.094) and raising the CLIP score from 25.0 to 26.8.

Mitigation y~\widetilde{y}ScopeCopied ↓\downarrowLPIPSRˉ↓\mathrm{LPIPS}_{\bar{R}}\downarrowLPIPSTˉ↓\mathrm{LPIPS}_{\bar{T}}\downarrowCLIP ↑\uparrow
NoneOriginal image0.9750.0000.00029.0
RTA [30]Whole image0.4320.4440.43725.0
Whole image (strong)0.2450.6000.59121.3
Region RR, JaRS (ours)0.4320.0940.14826.8
Region RR, JaRS, strong (ours)0.2840.1210.18824.8
Region RR, Δhθ~\Delta h_{\widetilde{\theta}} [15]0.4710.0820.14626.9
Region RR, Δhθ~\Delta h_{\widetilde{\theta}}, strong [15]0.3060.1110.19524.8
Wen et al. [34]Whole image0.2050.7080.68920.4
Whole image (strong)0.1010.7990.77718.3
Region RR, JaRS (ours)0.0900.1580.23922.2
Region RR, JaRS, strong (ours)0.0430.1860.27620.4
Region RR, Δhθ~\Delta h_{\widetilde{\theta}} [15]0.1010.1550.26122.3
Region RR, Δhθ~\Delta h_{\widetilde{\theta}}, strong [15]0.0400.1850.30420.7

Table 12. Region-restricted vs. whole-image memorization mitigation on the 278 test TV instances of Stable Diffusion v1.4. ‘Copied’ is the fraction of mitigated images that still match the training template under the benchmark copy criterion; LPIPSRˉ\mathrm{LPIPS}_{\bar{R}} and LPIPSTˉ\mathrm{LPIPS}_{\bar{T}} measure perceptual change relative to the original truth template TT, respectively; and ‘CLIP’ measures image-prompt alignment. We compare global mitigation against restricting the intervention to the mask RR obtained from JaRS (contrast-free) or from Δhθ~\Delta h_{\widetilde{\theta}} [15].

Qualitative comparison showing Original, Region $R$, Wen et al., whole image, and Wen et al., region (ours) across six test instances; inset labels indicate copied or not copied and LPIPS$_{\bar{R}}$ distances

Figure 6. Qualitative comparison of whole-image vs. region-restricted mitigation on six randomly sampled test ‘TV’ instances of Stable Diffusion v1.4 (columns 1, 3, 4, and 5 share the same memorized bedroom template). Row 1 shows the original generation; Row 2 overlays the dilated region RR localized by JaRS (magenta corresponds to the unmemorized complement Rˉ\bar{R} that is preserved); Row 3 shows the result of applying the prompt-embedding optimization of Wen et al. [34] globally to the whole image; and Row 4 shows the result of restricting the same optimization to RR (Sec. K.1). Whole-image mitigation fails to remove the copied template in four of the six instances and heavily alters the unmemorized foreground rug whenever it succeeds. Restricting mitigation to RR removes the copied background template in five of the six instances while keeping the unmemorized region Rˉ\bar{R} intact (inset labels report whether the template remains copied and the LPIPSRˉ_{\bar{R}} distance to the original generation).

Two ablation checks clarify what drives these gains:

  1. Role of fresh noise inside RR: Re-initializing RR with fresh Gaussian noise during RePaint resampling is essential, as retaining the original noise inside RR under Wen et al. [34] leaves 34.5% of templates copied; how-

ever, fresh noise alone does not explain the improvement over whole-image mitigation, as running whole-image mitigation from fresh noise still leaves 24.8% (default) and 10.4% (strong) copied while altering the entire image.

  1. JaRS mask vs. Δhθ~\Delta h_{\tilde{\theta}} mask: Using the mask from JaRS performs as well as or better than using the mask from Δhθ~\Delta h_{\tilde{\theta}} without requiring a reference checkpoint. Under RTA, the JaRS mask removes significantly more copies than the Δhθ~\Delta h_{\tilde{\theta}} mask (43.2% vs. 47.1% copied; 15 instances remain copied only under Δhθ~\Delta h_{\tilde{\theta}} vs. 4 only under JaRS, McNemar’s test p=0.02p = 0.02), because Δhθ~\Delta h_{\tilde{\theta}} occasionally misses portions of the template.

The same benefits hold on Stable Diffusion v2.1: whole-image mitigation leaves 83.2% (RTA) and 26.5% (Wen et al. [34]) of instances copied with LPIPSRˉ\mathrm{LPIPS}_{\bar{R}} of 0.17 and 0.38, whereas restricting mitigation to the JaRS region reduces the copy rate to 9.1% and 0.6% while lowering LPIPSRˉ\mathrm{LPIPS}_{\bar{R}} to 0.12 and 0.13.

Qualitative Examples

Figure 6 visualizes six randomly drawn test instances from our mitigation experiment (Sec. K.2), comparing the global prompt-embedding optimization of Wen et al. [34] against restricting the same optimization to the region RR localized by JaRS.

References

  1. [1]【1】 Costas Bekas, Effrosyni Kokiopoulou, and Yousef Saad. An estimator for the diagonal of a matrix. Applied numerical mathematics, 57(11-12):1214–1229, 2007.DOI
  2. [2]Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, 1 edition, 2007.
  3. [3]Benjamin Boys, Mark Girolami, Jakiw Pidstrigach, Sebastian Reich, Alan Mosca, and O. Deniz Akyildiz. Tweedie moment projected diffusions for inverse problems. arXiv preprint arXiv:2310.06721, 2024.arxiv.org/abs/2310.06721
  4. [4]Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models, 2023.arxiv.org/abs/2301.13188
  5. [5]Chen Chen, Daochang Liu, Mubarak Shah, and Chang Xu. Exploring local memorization in diffusion models via bright ending attention. In International Conference on Learning Representations, pages 29827–29841, 2025.arxiv.org/abs/2410.21665
  6. [6]Chen Chen, Daochang Liu, Mubarak Shah, and Chang Xu. Enhancing privacy-utility trade-offs to mitigate memorization in diffusion models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8182–8191. IEEE, 2025.arxiv.org/abs/2504.18032
  7. [7]Bradley Efron. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496):1602–1614, 2011.pubmed.ncbi.nlm.nih.gov/22505788
  8. [8]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.arxiv.org/abs/2207.12598
  9. [9]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pages 6840–6851. Curran Associates, Inc., 2020.arxiv.org/abs/2006.11239
  10. [10]Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation, 18(3):1059–1076, 1989.
  11. [11]Achref Jaziri, Martin Mundt, Ahmed Imtiaz Humayun, and Visvanthan Ramesh. Spectral signatures of memorization basins in diffusion trajectories. arXiv preprint arXiv:2611.01775, 2026.arxiv.org/abs/2611.01775
  12. [12]Dongjae Jeon, Dueun Kim, and Albert No. Understanding and mitigating memorization in generative models via sharpness of probability landscapes. arXiv preprint arXiv:2412.04140, 2024.arxiv.org/abs/2412.04140
  13. [13]Zahra Kadkhodaie, Florentin Guth, Eero Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. In International Conference on Learning Representations, pages 46543–46567, 2024.
  14. [14]Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 4396–4405. IEEE, 2019.DOI
  15. [15]Gwangho Kim and Sungyoon Lee. Localizing memorized regions in diffusion models via coordinate-wise curvature differences. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2605.26756
  16. [16]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02747
  17. [17]Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 11451–11461. IEEE, 2022.arxiv.org/abs/2201.09865
  18. [18]Peyman Milanfar. A tour of modern image filtering: New insights and methods, both practical and theoretical. IEEE Signal Processing Magazine, 30(1):106–128, 2013.
  19. [19]Koichi Miyasawa. An empirical bayes estimator of the mean of a normal population. Bull. Inst. Internat. Statist. 38(181-188):1–2, 1961.
  20. [20]Sreyas Mohan, Zahra Kadkhodaie, Eero P. Simoncelli, and Carlos Fernandez-Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks. In International Conference on Learning Representations, 2020.
  21. [21]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4195–4205, 2023.arxiv.org/abs/2212.09748
  22. [22]Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthias Douze. A self-supervised descriptor for image copy detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14512–14522. IEEE, 2022.arxiv.org/abs/2202.10261
  23. [23]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
  24. [24]Herbert E Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pages 388–394. Springer, 1992.
  25. [25]Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (RED). SIAM journal on imaging sciences, 10(4):1804–1844, 2017.arxiv.org/abs/1611.02862
  26. [26]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022.arxiv.org/abs/2112.10752
  27. [27]Brendan Ross, Hamidreza Kamkari, Tongzi Wu, Rasa Hosseinzadeh, Zhaoyan Liu, George Stein, Jesse Cresswell, and Gabriel Loaiza-Ganem. A geometric framework for understanding memorization in generative models. In International Conference on Learning Representations, pages 34773–34805, 2025.arxiv.org/abs/2411.00113
  28. [28]Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, pages 2256–2265, Lille, France, 2015. PMLR.
  29. [29]Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6048–6058. IEEE, 2023. 1, 2arxiv.org/abs/2212.03860
  30. [30]Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36:47783–47803, 2023. 1, 2, 18, 19arxiv.org/abs/2305.20086
  31. [31]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. 1, 3arxiv.org/abs/2010.02502
  32. [32]Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. 1
  33. [33]Ryan Webster. A reproducible extraction of training images from diffusion models. arXiv preprint arXiv:2305.08694, 2023. 1, 2, 6, 16, 17, 18arxiv.org/abs/2305.08694
  34. [34]Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memorization in diffusion models. In The Twelfth International Conference on Learning Representations, 2024. 2, 4, 6, 18, 19, 20, 21arxiv.org/abs/2407.21720
  35. [35]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF conference on computer vision and pattern recognition, pages 586–595. IEEE, 2018. 19arxiv.org/abs/1801.03924

Paper details

vlnc:W_yj8jnf74
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Large language models

View statement
Looking for related papers…
Contents