WorksComputer Science
JaRS: Contrast-Free Localization of Memorized Regions in Diffusion Models via Jacobian Row Sums
Abstract
Diffusion models sometimes memorize and reproduce regions of their training images, raising privacy and copyright concerns. We aim to localize these regions. Existing contrast-based methods use the diagonal of the denoiser Jacobian, which reflects the posterior variance of each pixel. However, a small diagonal entry is ambiguous: it signals either memorization or averaging over many neighboring pixels in smooth or predictable regions. To resolve this ambiguity, these methods subtract a reference map, which requires a second evaluation with a null prompt or an earlier checkpoint. Unconditional models have no null prompt, and earlier checkpoints are unavailable in general. We show that Jacobian row sums (JaRS), unlike the diagonal, separate memorization from local averaging without a reference map. Pairing JaRS with an image-level detection score that uses the same statistic at an early denoising step yields JaRS+. Across Stable Diffusion v1.4 and v2.1, Realistic Vision v5.1, and an unconditional diffusion Transformer (DiT), JaRS+ outperforms contrast-based localization methods.
Introduction
Diffusion models [9, 28, 31, 32] have achieved remarkable success in image synthesis, yet they are known to memorize and replicate portions of their training data [4, 29]. While certain prompts cause text-to-image models to regenerate an entire training image almost verbatim, a far more common form of replication is local memorization: the model reproduces a specific region of a training image such as a foreground product, an interior layout, or a watermark—while synthesizing novel content in the rest of the image [30, 33]. Such partial copies raise significant copyright and privacy concerns. Because the memorized region is embedded within newly generated content, localizing the exact pixels that are copied tells us what the model has memorized and makes it possible to selectively remove the memorized region while preserving the surrounding generation.
Recently, Kim and Lee [15] showed that memorized regions can be localized directly from the local geometry of the denoiser. By Tweedie’s formula [7, 19, 24], the Jacobian of the denoiser with respect to its noisy input is proportional to the posterior covariance of the clean image, and is directly related to the Hessian of the log density of the noisy marginal distribution. When a diffusion model memorizes a region, its posterior distribution over that region collapses [12, 15]; the posterior variance and hence the Jacobian become small, so the denoiser largely ignores perturbations to its noisy input. Kim and Lee [15] measure this collapse by using randomized Hutchinson probes [1, 10] to estimate the diagonal entries of the denoiser Jacobian (equivalently, the diagonal of the negative Hessian of the log-density). As Kim and Lee [15] note, however, a small diagonal entry is ambiguous on its own. Even in a model that generalizes without memorizing, the self-weight is small whenever a pixel lies in a smooth region such as a uniform background where the denoiser estimates each pixel by averaging over a neighborhood of pixels, assigning each pixel a weight of roughly . To remove this data-driven curvature [27], Kim and Lee [15] rely on a contrast: they subtract a reference diagonal map computed either with an unconditional null prompt () or with an earlier, less-trained checkpoint of the same model ().
While contrast-based localization is effective on standard text-to-image models, requiring a second reference evaluation introduces fundamental limitation: A null prompt exists only for conditional models trained with classifier-free guidance [8], and is absent in unconditional models. Similarly, model developers rarely release intermediate training checkpoints alongside final weights. In this paper, we ask whether memorized regions can be separated from data-driven smoothing directly from the target model alone, without any reference prompt or secondary checkpoint. Our key insight is that while the diagonal entry cannot distinguish between a row whose weights are spread across neighboring pixels and a row whose entries vanish altogether, the row sum separates the two.
Building on this principle, we introduce JaRS (Jacobian Row Sums), a contrast-free localization map that measures how far the per-channel row sums of the denoiser Jacobian fall below 1. Because the row sum is a linear functional of the Jacobian, JaRS requires no randomized probes: for a latent space with channels, the map is obtained from deterministic Jacobian-vector products (JVPs). Furthermore, to handle mixed evaluation settings that include both memorized and unmemorized generations where a localizer must output an empty mask on unmemorized images we propose JaRS+, a two-stage procedure that first determines whether an image contains memorized content using an early-step summary statistic of the JaRS map, and then returns the pixel-level JaRS map if memorization is detected. Our main contributions are summarized as follows:
We establish both theoretically via an analytical constant-region model and empirically that the row sums of the denoiser Jacobian concentrate near 1 on generated regions and drop sharply on memorized regions.
We propose JaRS and its two stage extension JaRS+.
Evaluated under the balanced protocol of Kim and Lee [15] across Stable Diffusion v1.4, Stable Diffusion v2.1, Realistic Vision v5.1, and an unconditional DiT-B/2 model, JaRS+ achieves the best localization mIoU in seven of ten settings, improving mIoU by up to 10.2 points on Stable Diffusion and from 0.666 to 0.869 on unconditional DiT.
We show that the early step detection score of JaRS+ identifies memorized prompts after just 5 of 50 sampling steps.
Related Work
Early studies demonstrated that diffusion models can memorize and regenerate training images with high fidelity. Carlini et al. [4] sampled from Stable Diffusion and from diffusion models trained on CIFAR-10 and flagged generations that were close, in Euclidean distance, to a training image. Somepalli et al. [29] instead measured replication in feature space, comparing generated and training images with copy-detection descriptors such as SSCD [22]. Webster [33] curated a benchmark of memorized prompts in Stable Diffusion and categorized them into three groups: matching verbatims (MV), which reproduce a training image paired with its training caption; retrieved verbatims (RV), where a prompt triggers a verbatim copy of a training image associated with a different caption; and template verbatims (TV), where the model reproduces a fixed spatial region (template) of a training image while generating diverse variations in the remaining regions across random seeds.
Subsequent works investigated the causes of memorization and proposed inference time detection and mitigation strategies. Somepalli et al. [30] linked replication to data duplication and text conditioning, proposing random token addition or replacement (RTA) to perturb memorized prompts. Wen et al. [34] observed that memorized prompts induce abnormally large text-conditional noise predictions and mitigated memorization by optimizing the prompt embedding at inference time, Chen et al. [6] re-anchored and searched prompts within classifier-free guidance to improve the privacy-utility trade-off, and Jeon et al. [12] optimized the initial Gaussian latent to steer trajectories away from sharp probability modes. However, all of these approaches operate at the level of the entire image or prompt.
Only recently has attention turned to pixel-level localization of memorized content. Chen et al. [5] argued that memorization should be analyzed locally and not only for whole images. They observed that at the final denoising step the cross-attention of the end-of-text token puts abnormally high weight on specific image patches (“Bright Ending”), and they used the resulting attention map to identify locally memorized regions. Interpreting memorization as a collapse of posterior [12], Kim and Lee [15] estimate the diagonal of the negative Hessian of at the final denoising step via Hutchinson-type probing [1, 10]. Because the raw diagonal also fires on smooth, easily predictable regions of ordinary images, Kim and Lee [15] subtracted a reference diagonal computed under the null prompt or an earlier training checkpoint, and established the localization benchmark that we adopt in our experiments. While we share their posterior collapse perspective, our work mainly departs from Kim and Lee [15] as we examine the row sums of the denoiser Jacobian rather than its diagonal which we believe eliminates the need for any contrast prompt or reference checkpoint.
Tweedie’s formula [7, 19, 24] connects the optimal denoiser to the score function , and differentiating it once more links the Jacobian of the denoiser to the posterior covariance . In classical and deep image denoising [13, 18, 20, 25], the Jacobian of a denoiser is interpreted as an input-dependent, locally adaptive linear filter; in particular, mean-preserving smoothing filters are row-stochastic, meaning that their row weights sum to 1 [18]. In diffusion posterior sampling for inverse problems, Boys et al. [3] multiplied the denoiser Jacobian by an all-ones vector as a computational shortcut to approximate the diagonal of . Jaziri et al. [11] studies the global leading singular value of the denoiser Jacobian along trajectories to characterize whole-image memorization basins.
Background and Preliminaries
We establish diffusion notation, link the denoiser Jacobian to the posterior, and review framework of Kim and Lee [15].
Setup
Let denote a clean image (or the latent representation of an image) drawn from the data distribution . A forward diffusion process progressively corrupts with Gaussian noise over time according to
where define the noise schedule. Standard variance-preserving diffusion models such as Stable Diffusion [9, 26, 31] choose and train a neural network to predict the injected noise . Flow-matching models [16, 21] instead use the linear interpolation schedule and , training a velocity network to predict . In either formulation, given a noisy input , the network produces a one-step estimate of the clean data point:
Let denote the marginal density of induced by Eq. (1). By Tweedie’s formula [7, 19, 24], the optimal noise predictor is
and the corresponding optimal denoiser is given by
To study the first-order sensitivity of the denoiser to perturbations in its noisy input , we work with scaled denoiser . Differentiating with respect to yields the denoiser Jacobian :
When equals the optimal noise predictor , substituting Eq. (3) into Eq. (4) and differentiating a second time via Tweedie’s second-order identity gives
where is the Hessian of the log-density of the noisy marginal . Additionally, one can see Jacobian as Adaptive linear filter. Locally around , the -th row of the Jacobian, , specifies how the denoised estimate at coordinate responds to infinitesimal changes at input coordinate . For a bias-free network, the denoiser is exactly a locally adaptive linear filter, , whose -th row contains the spatial averaging weights used to reconstruct pixel from its noisy neighborhood [13, 20]. For a general network, the rows still measure local sensitivity: is the response of to a uniform shift of its input. We verify in § 5.2 that, at intermediate noise, trained diffusion denoisers behave as mean-preserving filters on non-memorized content.
Curvature-Based Localization
We now connect the denoiser Jacobian to memorization. When a diffusion model memorizes a training image or a spatial region of it, the posterior over collapses [12]. Because the denoised output for a memorized pixel is fixed to the stored training content regardless of small perturbations to , the posterior variance vanishes. By Eq. (5), this implies . Kim and Lee [15] turn this observation into a localization metric by examining the diagonal of the negative log-density Hessian , which by Eq. (5) is
Kim and Lee [15] estimate at the final denoising step (we follow their official implementation), marking pixels with large values of (equivalently, small values of ) as memorized. However, a small diagonal entry does not identify memorization. To see why, consider the following example.
Example 1. Suppose a denoiser reconstructs a smooth image region (such as a uniform wall or clear sky) of pixels by taking an unweighted average of the noisy pixels in that region where . The jacobian is given by for all . Consequently, the diagonal is , which approaches as the averaging neighborhood grows large. Thus, on a smooth region of size , the diagonal entry is nearly indistinguishable from the diagonal entry of a memorized region where the denoiser outputs a stored constant .
Following Ross et al. [27], Kim and Lee [15] refer to the large values of as data-driven curvature, in contrast to memorization-driven curvature. As discussed in Example 1, raw diagonal (Eq. (6)) can’t differentiate between these two. To cancel the data-driven component, Kim and Lee [15] use a contrast, which assumes that a reference evaluation ( or ) exhibits the same data-driven smoothing on generic regions, so that subtracting it cancels the background response and isolates the memorized region. For a conditional model with prompt , they subtract the diagonal curvature computed either with null prompt ,
or with a second, less-trained checkpoint ,
Kim and Lee [15] also evaluate two score contrasts [34] given by
where . Following the implementation of Kim and Lee [15], a 2D spatial map is formed by summing the coordinate-wise values across the channels at each latent pixel.
As noted in § 1, however, computing Eq. (7), Eq. (8), or Eq. (9) requires a reference evaluation that is unavailable for unconditional models or when suitable prememorization checkpoints are not released. This motivates a central question: Can we distinguish local smoothing from memorization using only the target model itself? Example 1 suggests a route: while the diagonal entry shrinks to under local averaging, the sum of the entries across the row is , whereas under memorization () the row sum vanishes. We formalize this observation in the next section.
Jacobian Row Sums (JaRS)
We now present our contrast-free approach to localizing memorized regions. In § 4.1, we define the JaRS (Jacobian Row Sums) localization map. In § 4.2, we show how JaRS can be computed using Jacobian-vector products (we verify the predictions of this section on full denoiser Jacobians in § 5.2). Finally, we introduce our two-stage detection-and-localization pipeline JaRS+ in § 4.3.
Contrast-Free Localization
Consider with , where we index coordinates by for channel and spatial pixel location . Let denote the entry of the scaled denoiser Jacobian in (4). Let denote the channel indicator vector.
Definition 1. At noise level , the Jacobian Row Sum (JaRS) score at spatial location is defined as
measures how far per-channel row sums fall below 1. From Eq. (5), each channel term can be written equivalently as
Thus, whereas Kim and Lee [15] take the diagonal of the negative log-density Hessian, JaRS takes its row sums, and the constant 1 per channel can be seen as a replacement for the contrast. By this identity, the row sum equals 1 when , i.e., when is flat along : slightly brightening or darkening a generic region yields an equally plausible image.
As discussed in § 3.1 and Example 1, the diagonal shrinks as for an -pixel average, whereas the row sum does not decrease as the averaging neighborhood widens. Therefore, on generated regions, is close to 1 and the JaRS score is close to 0. Conversely, when the posterior collapses onto a stored image , is locally close to , so : the denoiser ignores perturbations of , drops toward 0, and rises toward .
The following proposition makes this precise in a simple analytical model of a homogeneous image region.
Proposition 1. Consider a single-channel region of pixels whose clean values share a common intensity level, for all , under a Gaussian prior $c\sim
. Given a noisy observation following Eq. (1), the optimal denoiser Jacobian satisfies, for every pixel and ,
Consequently, the diagonal entry at pixel is , whereas the row sum over is .
Proposition 1 illustrates two properties of the JaRS map: robustness to neighborhood size and dependence on noise schedule. First, regarding neighborhood size , the prior variance quantifies the model’s uncertainty over prior to observing . For generic, unmemorized regions where is large, the row sum approaches 1 regardless of once , while the diagonal entry scales as . In contrast, for memorized regions, the prior collapses onto a stored value (), yielding and driving the JaRS score .
Second, whereas Kim and Lee [15] evaluate curvature differences strictly at the final denoising step, Eq. (11) identifies the noise-to-signal ratio as the primary control knob. In the high-noise regime (), even for unmemorized regions because is nearly uninformative of . At near-zero noise (), everywhere, unless the memorized content is stored with effective variance smaller than . Between these extremes lies a broad intermediate noise window where on generated regions and on memorized regions; identifying and operating in this intermediate noise regime is a key step that takes us from final-step evaluation to JaRS and JaRS+.
Computation
A practical advantage of Definition 1 is that is a linear functional of the Jacobian . Unlike the diagonal , which requires randomized Hutchinson probes [1, 10] to estimate, the row sums are obtained from the matrix-vector products —one for each latent channel. Using Eq. (4), each product takes the form , where is the Jacobian of the noise predictor (with conditioning for conditional models, or omitted for unconditional models). We evaluate each directional derivative via a central finite difference:
For all our experiments, we used . Observe that computing the full JaRS map at a given timestep therefore requires forward network evaluations, which can be run in a single batched forward pass without any backward pass or automatic differentiation graph (App. § 17).
JaRS+: Two-Stage Detection and Localization
Discussion after Proposition 1 tells us that row-sum map should be evaluated at intermediate time steps. In realistic settings that mix memorized and non-memorized generations, however, a localizer must also output an empty mask on images that contain no copied content. As seen in the left tail of the non-memorized histogram for Stable Diffusion v1.4 in Figure 2, certain highly predictable regions of ordinary images such as uniform studio backgrounds also exhibit low posterior variance at intermediate noise levels. If a single pixel threshold is applied across all images at step without an image-level check, these confident background regions produce false-positive masks on non-memorized images.
Figure 2. Full denoiser Jacobian rows of Stable Diffusion v1.4 (step 15) and DiT-B/2 (step 35). Left: For one development image from each category, we show the generated image (copied region outlined in cyan; pixels outside a copied template are dimmed), the row of the full Jacobian at the marked pixel ‘+’ together with its sum, and the JaRS map (10). Right: Empirical distributions of the per-channel row sum across all pixels of non-memorized images and whole-image copies, and inside the copied region of template verbatims and watermarked samples (10 development images per category; 7 retrieved verbatims). The vertical dotted line at 1 marks the row sum of a mean-preserving averaging filter.
Crucially, analyzing the JaRS map across the denoising trajectory reveals a two timescale separation between image-level detection and pixel-level localization. In conditional models, memorized content is locked in by the prompt in the earliest denoising steps (high noise), where the JaRS map already exhibits elevated values on memorized trajectories while ordinary trajectories remain dominated by the initial Gaussian noise; at intermediate noise steps, spatial boundaries between copied templates and generated backgrounds become sharp (Figure 5). Exploiting this two-step structure takes us from JaRS to JaRS+, a two-stage procedure that pairs an early image-level detection stage at step with a pixel-level localization stage at step :
Figure 5. Evolution of the JaRS map across the 50-step sampling trajectory on the development split of the localization benchmark. Top: Image-level Stage 1 score for each category (median and interquartile range across development instances; category counts are listed in parentheses). Bottom: Pixel-level values of inside vs. outside the copied region for template verbatims (TV) and watermarked samples (WM), and across the full image for the remaining categories (median and interquartile range over 1,024 randomly sampled pixels per image). Vertical dotted lines mark the detection step (top) and localization step (bottom) selected on the development split (§ 4.3). On the unconditional DiT, all 154 development samples are shown; at step 0 () of the flow-matching schedule, the scaled denoiser Jacobian vanishes identically (), so at every pixel.
Stage 1 (Image-level detection): At denoising step , we compute the upsampled JaRS map and summarize it by its 90th percentile . The generation is flagged as containing memorized content if and only if , where is a scalar detection threshold.
Stage 2 (Pixel-level localization): If , the image is declared unmemorized and JaRS+ outputs an identically zero map. If , JaRS+ returns the JaRS localization map evaluated at step .
We now describe how we select , and . For each model, the three hyperparameters are selected on a held-out development split.
Localization step : Chosen from the candidate steps of the 50-step sampler to maximize the JaRS mean IoU on the memorized development images.
Detection step : Chosen from to maximize the image-level detection AUROC of between memorized and non-memorized images on the balanced development split of the model’s broadest evaluation setting (§ 5.1)
Detection threshold : Chosen from candidate quantiles (5%, 7.5%, , 95%) of on the same balanced development split to maximize the resulting two-stage development mIoU. When labeled memorized examples are not available to tune , can instead be set directly to the 95th percentile of on ordinary non-memorized images (5% development false-positive rate), which we show in
Sec. D achieves comparable accuracy across all text-to-image models. All ties above are broken in favor of the earlier step. Selection of , , above is discussed for 50 sampling steps, but can extended easily to arbitrary number of sampling steps.
Experiments
We evaluate JaRS and JaRS+ across text-to-image diffusion models and unconditional Diffusion Transformer (DiT). We first describe our benchmarks, evaluation protocol, and baselines (§ 5.1), verify the behavior of row sums predicted in § 4.1 on full denoiser Jacobians (§ 5.2), and then present our main pixel-level localization results (§ 5.3). Finally, we evaluate prompt-level memorization detection (§ 5.4). In App. § 18, we further show that restricting the inference-time mitigation of Wen et al. [34] to the region localized by JaRS lowers the copy rate from 20.5% to 9.0% and the perceptual change outside the region from 0.708 to 0.158 LPIPS.
Benchmarks, Protocol, and Baselines
We adopt the memorization localization benchmark of Kim and Lee [15], which is built on the memorized prompts identified by Webster [33] across three settings: Stable Diffusion v1.4 (SD v1.4), Stable Diffusion v2.1-base(SD v2.1) [26], and Realistic Vision v5.1. The benchmark comprises three categories of generations: Template verbatims (TV), Matching and Retrieved verbatims (MV, RV) come under Global memorization and Non-memorized prompts (N). We generate all images and ground-truth masks using the official code released by Kim and Lee [15].
To evaluate localization in an unconditional setting where no null prompt exists and with exact ground-truth masks, we train a unconditional DiT (DiT-B/2 [21]) using flow matching [16] on the 70,000 FFHQ [14] face images at resolution. Prior to training, we inject two controlled forms of memorization into the dataset: Local memorization (WM): a fixed semi-transparent watermark is pasted at a random spatial location into 10% of the training images and Global memorization (Near-copies): 20 clean training images are duplicated 100 times each. Generated samples are labeled as watermarked (with the watermark footprint as the ground-truth mask), near-copies (all-ones mask), or non-memorized (empty mask), and we retain 300 watermarked, 300 non-memorized, and all 139 near-copy samples (App. § 9).
We follow the balanced evaluation protocol of Kim and Lee [15 App. F.2] across all settings reported in their work: TV (WM for DiT) evaluates localization exclusively on locally memorized images, TV+N (WM+N) adds non-memorized images, and All further adds globally memorized images, with each category subsampled once to the size of the smallest. Following Kim and Lee [15], every localization map is clipped at 0, transformed via , min-max normalized across the evaluated set, and binarized using the single threshold that maximizes the mean Intersection-over-Union (mIoU) across images.
We partition all prompts (along with their seeds and templates) and DiT samples into a 20% development split (indices divisible by 5) and 80% test split. All hyperparameters of JaRS+ (, , and ; § 4.3) are selected on the development split, and all experimental results are reported on the test split. The development split selects for SD v1.4 and SD v2.1, for Realistic Vision, and for the DiT (App. § 13).
We compare JaRS+ against the maps of Kim and Lee [15]: the contrast-free raw diagonal, the null-prompt contrast (Eq. (7)), the checkpoint contrast (Eq. (8)), and the two score contrasts and (Eq. (9)). On the DiT, only the checkpoint contrasts are applicable; we evaluate them at step 48, which matches the noise level of their published configuration, with the reference selected on the development split among five checkpoints saved during our training run (App. § 9). We also include the benchmark numbers reported by Kim and Lee [15] for Bright Ending [5].
Empirical Verification on Full Jacobians
To verify whether trained diffusion models follow the predicted behavior in § 4.1, we explicitly form the full denoiser Jacobian and inspect its rows directly (Figure 2). On the DiT (), we materialize the exact Jacobian at for 10 randomly selected development samples from each category: watermarked, near-copies, and non-memorized. On non-memorized samples, each row is a localized smoothing kernel centered at pixel , and the per-channel row sums concentrate sharply around 1 (median 0.97, with 80% of values in ), confirming that the denoiser acts as a mean preserving local filter. The unmemorized background of watermarked samples behaves identically (median row sum 0.98). In contrast, inside the memorized watermark region, the row sums drop to a median of 0.48, and on whole-image near-copies they fall to 0.45. Across all pixels, the row sum separates watermark pixels from non-memorized pixels with an AU-ROC of 0.998.
The same phenomenon holds on SD v1.4 (), where we compute the full Jacobian at the localization step for 37 development images: 10 TV, 10 MV, all 7 RV of the development split, and 10 non-memorized images. The per-channel row sums have a median of 0.88 on non-memorized images and 0.89 outside the copied template, falling to 0.41 inside the template region, 0.54 on matching verbatims, and 0.58 on retrieved verbatims. App. § 10 repeats this analysis on SD v2.1 and compares the row sum with the norm of the row.
Memorization Localization Results
Tab. 1 reports the quantitative localization results across all four models, and Figure 1 visualizes representative localization maps on SD v1.4, where the maps of Kim and Lee [15] are computed at the final sampling step, as in Table 1. On both SD v1.4 and v2.1, JaRS+ achieves the highest localization accuracy in the TV and TV+N settings despite using no reference prompt or checkpoint. Specifically, JaRS+ outperforms the strongest contrast baseline by +5.6 and +2.5 mIoU points on SD v1.4 and by +3.2 and +10.2 points on SD v2.1. In the All setting of SD v1.4, JaRS+ (0.940) comes within 0.005 of (0.945). When the step and the image-level gate of every method, including the percentile of JaRS+, are tuned on the same development split, JaRS+ remains best in eight of ten settings (App. E).

Figure 1. Qualitative comparison of localization maps on Stable Diffusion v1.4 for randomly sampled test instances, two per row: four template verbatims from distinct templates (rows 1–2), one global whole-image copy (bottom left), and one non-memorized prompt (bottom right), which has no copied region and hence an empty expected mask. All maps are displayed after the transformation.
| Method | Reference | SD v1.4 TV | SD v1.4 TV+N | SD v1.4 All | SD v2.1 TV | SD v2.1 TV+N | Realistic Vision TV | Realistic Vision All | DiT (unconditional) WM | DiT (unconditional) WM+N | DiT (unconditional) All |
| Raw diagonal [15] | – | 0.594 | 0.500 | 0.524 | 0.652 | 0.500 | 0.660 | 0.530 | 0.261 | 0.500 | 0.640 |
| Bright Ending† [5] | – | 0.751 | – | – | 0.933 | – | 0.777 | – | – | – | – |
| [15] | null prompt | 0.900 | 0.950 | 0.945 | 0.945 | 0.887 | 0.930 | 0.951 | not available | ||
| [15] | checkpoint | 0.913 | 0.903 | 0.884 | 0.948 | 0.876 | 0.936 | 0.880 | 0.439 | 0.500 | 0.666 |
| [15] | null prompt | 0.826 | 0.901 | 0.898 | 0.787 | 0.809 | 0.876 | 0.903 | not available | ||
| [15] | checkpoint | 0.864 | 0.730 | 0.653 | 0.921 | 0.877 | 0.860 | 0.626 | 0.376 | 0.501 | 0.523 |
| JaRS (ours) | – | 0.969 | 0.619 | 0.637 | 0.980 | 0.571 | 0.949 | 0.612 | 0.836 | 0.873 | 0.909 |
| JaRS+ (ours) | – | 0.969 | 0.975 | 0.940 | 0.980 | 0.989 | 0.929 | 0.849 | 0.638 | 0.799 | 0.869 |
Table 1. Pixel-level localization of memorized regions on the test splits (mIoU) under the balanced protocol of Kim and Lee [15]. The baselines of Kim and Lee [15] are evaluated at the final sampling step (step 48 on the DiT, at the same noise level ) using 16 Hutchinson probes; checkpoint references for and are SD v1.1 (for SD v1.4 and Realistic Vision), SD v2.0-base (for SD v2.1), and the 5k-step checkpoint, which is best on the DiT development split for both contrasts. JaRS+ requires no reference prompt or checkpoint, and its hyperparameters are chosen once per model on the development split; JaRS denotes the map at step without the image-level gate (Stage 1) of JaRS+. In settings with non-memorized images, an mIoU of 0.500 corresponds to predicting an empty mask everywhere. †Numbers reported by Kim and Lee [15] on the full benchmark.
On the unconditional DiT, where the null-prompt contrast does not exist, JaRS+ outperforms the checkpoint contrast across all three settings, raising mIoU from 0.439 to 0.638 on WM, from 0.500 to 0.799 on WM+N, and from 0.666 to 0.869 on All. Furthermore, evaluating across all saved training checkpoints reveals sensitivity to the choice of reference: the 20k-step checkpoint would reach 0.503 on WM, and later checkpoints (100k and 150k steps) drop to 0.369 and 0.243 (full sweep in App. § 15).
On Realistic Vision, JaRS+ trails the contrast baselines: while JaRS+ is within 0.007 of on TV (0.929 vs. 0.936), it scores 0.849 on All compared to 0.951 for . This gap arises from the Stage 1 threshold rather than the JaRS map: with an oracle image-level detector, the same maps reach 0.945 on All (App. § 11). Removing Stage 1 (JaRS in Table 1) does not lower mIoU on TV and WM, but lowers it sharply on the text-to-image settings with non-memorized images (e.g., from 0.975 to 0.619 on TV+N of SD v1.4). On the DiT, removing Stage 1 raises mIoU (0.869 to 0.909 on All).
It is also instructive to compare JaRS+ with the raw diagonal row in Table 1, which evaluates the diagonal estimator of Kim and Lee [15] without a contrast: the raw diagonal achieves only 0.594 and 0.652 on TV and collapses to 0.500 on TV+N (performing no better than an empty mask). Replacing the diagonal with the row sum eliminates the data-driven smoothing. The noise level matters as well: at the final step, where the contrasts of Kim and Lee [15] are evaluated, the JaRS map itself is no better than an all-ones mask (0.563 and 0.648 on TV), whereas the step selected on the development split lies on a broad high-accuracy plateau at intermediate noise on all four models (App. § 14).
Detecting Memorized Prompts and Samples
Beyond pixel-level maps, the Stage 1 score of JaRS+ can directly serve as a standalone detector of whether a prompt (or unconditional sample) triggers memorization. Following the prompt-detection protocol of Kim and Lee [15 Sec. 5.2], each prompt is generated with four random seeds and scored by averaging for JaRS+ (at the step of § 5.1) or the spatial mean of the final-step map for the baselines; on the DiT, individual samples are scored. The positive and negative sets, test split sizes, and results on Realistic Vision are given in App. § 16.
As shown in Table 2, the score contrasts of Kim and Lee [15] achieve the highest detection accuracy on SD v1.4 and the DiT, while on SD v2.1 JaRS+ and all four contrasts are within 0.003 AUC of one another. Nevertheless, JaRS+ remains competitive under much weaker requirements. First, it requires no reference evaluation: on the DiT, it outperforms both the raw diagonal and , and surpasses it only with a reference from 25%–75% of training, which is rarely available in practice, while no reference allows to localize memorized regions (at most 0.523 mIoU on All; App. § 15). Second, on Stable Diffusion, JaRS+ computes its score at step of 50, whereas the final-step maps of Kim and Lee [15] require the entire generation, allowing JaRS+ to flag memorized prompts before a copy is synthesized.
| Method | SD v1.4 | SD v2.1 | DiT | |||
| AUC | TPR | AUC | TPR | AUC | TPR | |
| Bright Ending†[5] | 0.886 | 0.390 | 0.945 | 0.877 | – | – |
| Raw diagonal [15] | 0.906 | 0.547 | 0.871 | 0.040 | 0.747 | 0.308 |
| [15] | 0.969 | 0.915 | 0.995 | 0.966 | not available | |
| [15] | 0.951 | 0.897 | 0.994 | 0.954 | 0.867 | 0.363 |
| [15] | 0.969 | 0.940 | 0.994 | 0.966 | not available | |
| [15] | 0.957 | 0.925 | 0.993 | 0.971 | 0.943 | 0.500 |
| JaRS+ (ours) | 0.946 | 0.873 | 0.992 | 0.971 | 0.902 | 0.483 |
Table 2. Prompt- and sample-level memorization detection on the test split, measured by AUROC (‘AUC’) and true-positive rate at 1% false-positive rate (‘TPR’). On Stable Diffusion, the baselines of Kim and Lee [15] are our runs of their official code (compared to their reported numbers in App. § 16); on the DiT, the 50k- and 150k-step references of and are selected by development AUROC. †Reported by Kim and Lee [15] on all prompts.
The 0.023 AUC gap on SD v1.4 arises entirely from prompts that copy on only some or none of the four seeds: on prompts that produce a verified template or matching verbatim copy, JaRS+ and all four contrasts achieve 1.000 AUC (App. § 16).
Conclusion
We have shown that memorized regions in diffusion models can be localized accurately from the row sums of the denoiser Jacobian (JaRS) without any contrast prompt or reference checkpoint. Combining JaRS with an early-step image-level detection score yields JaRS+, which outperforms contrast-based localization and enables region-restricted mitigation (App. § 18).
Limitations and future work. JaRS evaluates the denoiser at an intermediate noise level of a multi-step trajectory; extending it to single-step generators, such as distilled models, is left for future work. Further, JaRS+ selects on a small labeled development split, whereas the published configuration of Kim and Lee [15] requires none on Stable Diffusion; can be calibrated from non-memorized images alone (App. § 11), but and still require memorized examples.
LLM Usage Statement
Large language models were used to assist with editing and polishing the prose of the manuscript and with writing utility scripts for data processing and plotting. All theoretical derivations, experimental designs, implementations, and empirical analyses were conceived, verified, and validated by the authors.
Proof of prop:smooth
Proof. Using Eq. (1), conditioned on , the noisy observations in are independent with for each . Using Eq. (2.116) of Bishop [2], the posterior distribution of is Gaussian with precision and mean given by
Since for every , the scaled optimal denoiser is . Differentiating with respect to for any yields
Setting gives , and summing over the pixels gives . □
Benchmark and Baseline Details
DiT benchmark. We train the unconditional DiT-B/2 [21] using flow matching [16] for 200k steps (batch size 128) on the 70,000 FFHQ [14] face images at resolution, encoded into latents via the Stable Diffusion autoencoder. The planted watermark has a size of pixels and is pasted into 7,000 (10%) of the training images. We generate 12,000 samples from the trained model using 50 Euler steps. To label the generated samples, we compute the maximum normalized cross-correlation (NCC) of each sample against the watermark template (which scores on watermarked training images and on clean training images). A generated sample is labeled as watermarked (WM) if its watermark NCC exceeds 0.6 (occurring in of samples), with the exact watermark footprint at the detected location serving as its ground-truth mask. A sample within a root-mean-square pixel distance of 0.05 from any duplicated training image is labeled a near-copy with an all-ones mask, and a sample with watermark NCC below 0.3 and distance above 0.1 from all duplicated images is labeled non-memorized (N) with an empty mask. Following the scale of the Stable Diffusion benchmarks, we retain the first 300 watermarked samples, the first 300 non-memorized samples, and all 139 near-copies.
Evaluation protocol. Every map (ours and the baselines’) is bilinearly upsampled from latent resolution to the resolution of the ground-truth masks, as in the code of Kim and Lee [15], and the Stage 1 score of JaRS+ is computed on the upsampled map. The binarization threshold is chosen among 1,001 uniformly spaced levels in . The resulting test splits contain 278 (TV), 278 + 278 (TV+N), and (All) instances for SD v1.4; 471 (TV) and 400 + 400 (TV+N) for SD v2.1; 402 (TV) and (All) for Realistic Vision; and 242 (WM), 241 + 241 (WM+N), and (All) for the DiT.
Baselines. For Realistic Vision, where Kim and Lee [15] do not specify a reference checkpoint, we use SD v1.1. On the DiT, we saved intermediate checkpoints at 5k, 20k, 50k, 100k, and 150k steps during our own training run. Because the final sampling step of Stable Diffusion has noise-to-signal ratio whereas the final step of the DiT reaches 0.020, we evaluate the baselines on the DiT at step 48 (), which matches the noise level of their published configuration without tuning on any split. To give the checkpoint contrasts the strongest fair baseline on the DiT, we evaluate all five saved checkpoints on the development split and select the best reference checkpoint separately for each method and task: by development All mIoU for localization (which selects the 5k-step checkpoint for both and ) and by development AUROC for detection (which selects the 50k-step checkpoint for and the 150k-step checkpoint for ).
Signed Row Sum versus the L1 Row Norm
A natural alternative to the signed row sum is the norm row, which vanishes if and only if the entire row is zero. We evaluated this variant on the full Jacobians of § 5.2, together with 20 development images of SD v2.1 (10 template verbatims and 10 non-memorized images) at its localization step , replacing each row sum in (10) by the norm. On SD v2.1, the row sums behave as on the other two models: their median is 0.86 on non-memorized images and 0.89 outside the copied template, and falls to 0.50 inside the template region. The norm, in contrast, does not concentrate around a common value on non-memorized images (Figure 3, bottom; median 1.51 on the DiT and 1.13 and 1.12 on SD v1.4 and v2.1) and overlaps more with the memorized pixels. Table 3 confirms that the variant separates memorized from non-memorized pixels less accurately than the row sum for every model and category except the DiT near-copies, where both are perfect. The gap is largest on the whole-image copies of SD v1.4 and on the DiT watermark. We therefore use the signed row sum throughout.
| Model | Memorized pixels | Row sum | norm |
| DiT-B/2 | Watermark | 0.998 | 0.900 |
| Near-copy | 1.000 | 1.000 | |
| SD v1.4 | Template (TV) | 0.980 | 0.908 |
| Matching verbatim | 0.917 | 0.772 | |
| Retrieved verbatim | 0.862 | 0.574 | |
| SD v2.1 | Template (TV) | 0.955 | 0.934 |
Table 3. Pixel-level AUROC between memorized and non-memorized pixels for the JaRS map (10) and its variant, which replaces each row sum by the norm of the row, on the development images of Figure 3.
Figure 3. Per-channel row sum (top) and row norm (bottom) of the full denoiser Jacobian at the localization step of each model (DiT-B/2 at step 35, SD v1.4 and SD v2.1 at step 15), across all pixels of non-memorized images and whole-image copies, and inside the copied region of watermarked samples and template verbatims. The development images are those of § 5.2, together with 10 template verbatims and 10 non-memorized images of SD v2.1. The vertical dotted line at 1 marks the row sum of a mean-preserving averaging filter.
Ablation of the Image-Level Detection Stage
Table 4 compares the default two-stage JaRS+ pipeline reported in the main paper against two natural variants on the test splits:
| Detection Variant | SD v1.4 TV | SD v1.4 TV+N | SD v1.4 All | SD v2.1 TV | SD v2.1 TV+N | Realistic Vision TV | Realistic Vision All | DiT (unconditional) WM | DiT (unconditional) WM+N | DiT (unconditional) All |
| JaRS+ (tuned , main paper) | 0.969 | 0.975 | 0.940 | 0.980 | 0.989 | 0.929 | 0.849 | 0.638 | 0.799 | 0.869 |
| JaRS+ ( at 5% dev FPR) | 0.969 | 0.923 | 0.906 | 0.980 | 0.959 | 0.944 | 0.854 | 0.463 | 0.725 | 0.817 |
| JaRS (no Stage 1, step ) | 0.969 | 0.619 | 0.637 | 0.980 | 0.571 | 0.949 | 0.612 | 0.836 | 0.873 | 0.909 |
Table 4. Ablation of Stage 1 (image-level detection) variants on the test splits (mIoU). All hyperparameters are selected on the development split of each model. ‘JaRS+ (tuned )’ chooses to maximize development mIoU on the broadest balanced setting; ‘JaRS+ ( at 5% dev FPR)’ calibrates using only non-memorized development images at the 95th percentile; and ‘JaRS’ returns the localization map for every image without Stage 1 gating.
JaRS+ ( at 5% dev FPR): Instead of selecting the threshold to maximize development mIoU across both memorized and non-memorized development images, is set to the 95th percentile of the detection score computed exclusively on non-memorized development images (guaranteeing a 5% false-positive rate on the development split without requiring any labeled memorized examples to calibrate ).
JaRS (no Stage 1): Returns the pixel-level JaRS map for every image without an image-level detection gate.
In settings that contain only memorized images (TV), JaRS achieves the highest performance (0.969, 0.980, and 0.949 on SD v1.4, SD v2.1, and Realistic Vision). In mixed text-to-image settings that include non-memorized prompts (TV+N and All), however, JaRS produces false-positive regions on confident backgrounds of unmemorized images, showing why Stage 1 in JaRS+ is essential. Notably, calibrating at a 5% false-positive rate using only non-memorized development images remains competitive across all three text-to-image models and slightly improves over the mIoU-tuned threshold on Realistic Vision (TV: 0.944 vs. 0.929; All: 0.854 vs. 0.849), where the memorized development set is small. On the unconditional DiT, JaRS achieves the highest accuracy across all three settings (0.836 WM, 0.873 WM+N, and 0.909 All), because the 90th-percentile statistic rejects a fraction of samples containing the small (4%-area) watermark.
To understand where JaRS+ loses accuracy on Realistic Vision All (§ 5.3), Table 6 breaks down the pass rates of the Stage 1 detector () by image category and compares the resulting mIoU against the same Stage 2 JaRS maps paired with an oracle image-level gate (‘Oracle Gate’). On Realistic Vision, the pixel-level JaRS maps at step are highly accurate: with an oracle gate, they achieve 0.945 mIoU on All, nearly matching (0.951). The drop to 0.849 occurs because Stage 1 rejects 26.5% of the global whole-image copies on the test split. Inspecting the development split explains why: Realistic Vision has only 28 whole-image copies in its development split, on which achieved a near-perfect development AUROC of 0.999, yielding a threshold that proved overly conservative on the larger test split. On SD v1.4, where the development split contains 45 whole-image copies, Stage 1 accepts 92.5% of global copies and 100% of TV copies while rejecting 97.8% of non-memorized images, coming within 0.020 of the oracle gate (0.940 vs. 0.960).
| Model | TV | Global | Non-mem. | Oracle Gate (JaRS+) |
| SD v1.4 | 1.000 | 0.925 | 0.022 | 0.960 (0.940) |
| Realistic Vision | 0.974 | 0.735 | 0.051 | 0.945 (0.849) |
Table 6. Disentangling Stage 1 (detection) and Stage 2 (localization) on the test All split. Columns 2–4 report the fraction of images in each category that pass the Stage 1 gate (). Column 5 reports the mIoU of the Stage 2 JaRS localization maps when paired with an oracle image-level detector (with the actual two-stage JaRS+ mIoU in parentheses).
Tuning the Baselines in the Same Way as JaRS+
In Table 1, the baselines of Kim and Lee [15] are evaluated in their published configuration: at the final sampling step and without an image-level gate. JaRS+, in contrast, selects its steps and its threshold on the development split. To check that this difference does not drive our conclusions, we tune every method in the same way.
Protocol. For every method and model, we use the development split only, with the same candidates and selection rules for all methods.
Ungated. The step of the map is chosen from to maximize development mIoU on TV (WM on the DiT).
Gated (‘+’). Each method is given the two-stage form of JaRS+ (§ 4.3). The image-level score is a statistic of the method’s own map at step , with and . The pair maximizes development AUROC between memorized and non-memorized images, and maximizes development mIoU, both on the broadest balanced setting, as for JaRS+. Ties are broken toward the earlier step.
JaRS+ follows the same rule, so here it may select a percentile other than 0.9. On the DiT, the checkpoint contrasts use the 5k-step reference of Table 1.
Results. Table 7 shows three findings. First, tuning the step changes little for the four contrasts on Stable Diffusion and Realistic Vision, where the development split selects their final step or a nearby one, but it strongly improves on the DiT (0.666 to 0.849 on All). Second, the image-level gate improves every baseline whose map responds to non-memorized images (e.g., on All of SD v1.4, from 0.668 to 0.912), yet JaRS+ remains best in eight of ten settings; the two exceptions, All on SD v1.4 and on Realistic Vision, are the settings where JaRS+ also trails in Tab. 1. Third, the development AUROC of almost every gate reaches 1.000, so the selected is often decided by the tie rule, and a gate can reject memorized test images (e.g., loses about 0.03 on TV of both SD models).
| Method | SD v1.4 TV | SD v1.4 TV+N | SD v1.4 All | SD v2.1 TV | SD v2.1 TV+N | SD v2.1 All | Realistic Vision TV | Realistic Vision All | DiT (unconditional) WM | DiT (unconditional) WM+N | DiT (unconditional) All |
| Ungated, step tuned | |||||||||||
| Raw diagonal | 0.779 | 0.608 | 0.720 | 0.812 | 0.575 | 0.844 | 0.649 | 0.327 | 0.500 | 0.632 | |
| 0.900 | 0.950 | 0.945 | 0.945 | 0.887 | 0.930 | 0.951 | not available | ||||
| 0.913 | 0.903 | 0.884 | 0.948 | 0.876 | 0.936 | 0.880 | 0.790 | 0.860 | 0.849 | ||
| 0.844 | 0.709 | 0.733 | 0.911 | 0.579 | 0.876 | 0.903 | not available | ||||
| 0.885 | 0.764 | 0.668 | 0.921 | 0.877 | 0.860 | 0.626 | 0.630 | 0.570 | 0.417 | ||
| JaRS | 0.969 | 0.619 | 0.637 | 0.980 | 0.571 | 0.949 | 0.612 | 0.836 | 0.873 | 0.909 | |
| Gated (‘+’), tuned | |||||||||||
| Raw diagonal+ | 0.768 | 0.877 | 0.909 | 0.811 | 0.896 | 0.844 | 0.914 | 0.038 | 0.500 | 0.662 | |
| 0.900 | 0.950 | 0.945 | 0.943 | 0.967 | 0.930 | 0.951 | not available | ||||
| 0.879 | 0.936 | 0.926 | 0.918 | 0.956 | 0.895 | 0.927 | 0.790 | 0.870 | 0.872 | ||
| 0.841 | 0.921 | 0.919 | 0.911 | 0.946 | 0.876 | 0.922 | not available | ||||
| 0.882 | 0.941 | 0.912 | 0.921 | 0.960 | 0.847 | 0.844 | 0.606 | 0.803 | 0.738 | ||
| JaRS+ | 0.969 | 0.983 | 0.931 | 0.980 | 0.989 | 0.944 | 0.850 | 0.819 | 0.895 | 0.928 |
Table 7. Localization mIoU on the test splits when every method is tuned on the development split in the same way. Top: each map at its selected step , without an image-level gate. Bottom: each method with its own image-level gate, selected as for JaRS+. Bold marks the best value in each column within each block.
Selection of the Detection Step
Tab. 5 reports the image-level detection AUROC of across all candidate steps on the balanced development splits of the three text-to-image models. Across all three models, image-level separation between memorized and non-memorized trajectories peaks in the earliest denoising steps (highest noise levels), where the text prompt immediately locks in memorized structure while non-memorized trajectories remain sensitive to the initial Gaussian noise. Accordingly, our selection rule chooses on all three text-to-image models (on SD v2.1, steps 0 and 5 differ only in the fourth decimal place). By contrast, on the unconditional DiT, there is no text prompt to lock in the small watermark during the earliest steps; instead, the watermark emerges in the JaRS map at intermediate noise levels (Sec. G), and the development AUROC peaks at (0.984).
The JaRS Map Across Noise Levels
Localization accuracy across noise levels. Figure 4 plots the test mIoU of the JaRS map as a function of the noise-to-signal ratio , which places the variance-preserving schedule of Stable Diffusion and the flow-matching schedule of the DiT on a unified axis. The empirical curves closely follow the theoretical prediction of Proposition 1. At the highest noise levels (), the denoiser discards its noisy input everywhere (), so on SD v1.4 and v2.1 the map scores 0.563 and 0.648 (the trivial accuracy of an all-ones mask). As noise decreases, mIoU rises sharply to a broad, stable plateau centered around for SD v1.4 and v2.1, and around for Realistic Vision and the DiT—before declining as , where everywhere. Notably, at the final sampling step (), where Kim and Lee [15] evaluate their contrast maps, a single model’s Jacobian approaches the identity everywhere and the JaRS map returns to the trivial baseline (0.563 and 0.648). Identifying that contrast-free localization lives at intermediate rather than at the final step is thus a central benefit of our analysis: across all four models, the localization step chosen on the development split (circled in Figure 4) lies squarely on the high-accuracy plateau, and on Stable Diffusion the localization map is available after just 15 of the 50 sampling steps.
Figure 4. Effect of noise level on localization accuracy. Test mIoU of the JaRS map on locally memorized instances (TV for text-to-image models, WM for the DiT) plotted against the noise-to-signal ratio at which the map is evaluated. Black circles mark the localization step selected on the development split.
The JaRS map along the sampling trajectory. Figure 5 tracks the JaRS map across all 50 denoising steps for every development instance, directly illustrating the two-timescale separation between early image-level detection and intermediate pixel-level localization that motivates JaRS+ (§ 4.3).
Early image-level separation (Stage 1). On all three text-to-image models (Figure 5, top), the image-level score is systematically higher on memorized trajectories than on non-memorized trajectories at every step prior to the end of sampling, with the widest separation occurring in the earliest steps. On SD v1.4, for example, the median on template verbatims is 3.46 at step 5 versus 1.52 on non-memorized images (compared to 1.07 vs. 0.42 by step 40). Retrieved verbatims (RV) begin at a lower median at step 0 (1.92 on SD v1.4 and 4.19 on Realistic Vision, compared to 3.42 and 5.28 for template verbatims) before catching up to the other memorized categories by step 5 on Realistic Vision and step 10 on SD v1.4. This trajectory behavior explains why the development detection AUROC in Table 5 is lower at step 0 than at step 5 and peaks at across all three text-to-image models. At the final step (49), as the noise level approaches zero, the median of every category collapses to at most 0.12, matching the prediction of Proposition 1.
| Candidate Detection Step | 0 | 5 | 10 | 15 | 20 | 25 | 30 | 35 | 40 | 45 |
| Stable Diffusion v1.4 | 0.955 | 0.988 | 0.987 | 0.968 | 0.936 | 0.887 | 0.861 | 0.850 | 0.820 | 0.809 |
| Stable Diffusion v2.1 | 0.9998 | 1.000 | 0.996 | 0.974 | 0.911 | 0.826 | 0.797 | 0.786 | 0.778 | 0.763 |
| Realistic Vision v5.1 | 0.916 | 0.999 | 0.983 | 0.912 | 0.869 | 0.844 | 0.830 | 0.817 | 0.807 | 0.814 |
Table 5. Development AUROC of the Stage 1 detection score across candidate denoising steps , evaluated on the balanced development split (‘All’ for SD v1.4 and Realistic Vision, ‘TV+N’ for SD v2.1). Bold indicates the selected step .
Intermediate pixel-level separation (Stage 2). The pixel-level curves (Figure 5, bottom) explain why accurate spatial localization requires waiting until an intermediate step . In the earliest denoising steps, the unmemorized pixels outside the copied template of a TV image exhibit nearly the same elevated JaRS scores as the pixels inside the template: on SD v1.4 at step 0, the median score is 2.19 outside the template versus 2.66 inside. Thus, at very high noise, a memorized prompt elevates the JaRS map across the entire image—which provides a strong signal for image-level detection, but poor contrast for segmenting the copied region. By step on SD v1.4 and SD v2.1, the pixels outside the template relax to the baseline level of ordinary non-memorized images (medians of 0.41 and 0.48, matching 0.47 and 0.44 on non-memorized images), whereas the pixels inside the copied template remain elevated (2.33 and 2.05), creating a sharp spatial boundary. On Realistic Vision, the unmemorized background relaxes more gradually and reaches the non-memorized level only around step 30 (0.70 vs. 0.67), consistent with its later selected localization step . Beyond step 20, the median score outside the template dips slightly below zero on SD v1.4 and SD v2.1 (reaching ) where sharpening side lobes cause the per-channel row sums to slightly exceed 1; these negative values are clipped at 0 in the evaluation protocol (§ 5.1).
Trajectory dynamics on the unconditional DiT. On the unconditional DiT (Figure 5, rightmost column), global and local memorization emerge on different timescales. Whole-image near-copies separate from non-memorized samples almost immediately: the AUROC of between near-copies and non-memorized samples reaches 0.960 at step 5 and 1.000 from step 15 onward. In contrast, the planted watermark covers only 3.8% of the image area on average, so it barely shifts the 90th spatial percentile at step 30, the median is 0.68 on watermarked samples versus 0.54 on non-memorized samples). Within each watermarked sample, pixel-level separation between the watermark and the unmemorized face background develops only at intermediate noise: the mean within-image pixel AUROC rises from 0.547 at step 0 and 0.769 at step 10 to 0.949 at step 15, and exceeds 0.986 from step 20 onward. Because no text prompt locks in the watermark at , image-level separation between watermarked and non-memorized samples peaks later at step 30 (AUROC 0.925), which—combined with the near-copies—leads the development split to select .
Sensitivity of Checkpoint Contrasts to the Reference Checkpoint
Kim and Lee [15] note that the reference checkpoint in must be “relatively less overfitted” than the target model , cautioning that overly early checkpoints fail to capture general data-driven curvature while overly late checkpoints already exhibit memorization-driven curvature. For the unconditional DiT (trained for 200k steps), we evaluated both and at step 48 (§ 5.1) across all five intermediate checkpoints saved during training (5k, 20k, 50k, 100k, and 150k steps). Table 8 reports the resulting development and test performance for both localization and sample-level detection.
| Reference | Localization (mIoU) | Detection (AUC) | |||||
| Dev All | WM | WM+N | All | Dev | Test | ||
| 5k (2.5%) | 0.666 | 0.439 | 0.500 | 0.666 | 0.800 | 0.803 | |
| 20k (10%) | 0.663 | 0.503 | 0.500 | 0.663 | 0.859 | 0.860 | |
| 50k (25%) | 0.661 | 0.457 | 0.500 | 0.655 | 0.866 | 0.867 | |
| 100k (50%) | 0.634 | 0.369 | 0.500 | 0.616 | 0.864 | 0.864 | |
| 150k (75%) | 0.531 | 0.243 | 0.500 | 0.526 | 0.849 | 0.833 | |
| 5k (2.5%) | 0.507 | 0.376 | 0.501 | 0.523 | 0.840 | 0.839 | |
| 20k (10%) | 0.473 | 0.461 | 0.504 | 0.482 | 0.900 | 0.903 | |
| 50k (25%) | 0.423 | 0.477 | 0.506 | 0.434 | 0.918 | 0.917 | |
| 100k (50%) | 0.402 | 0.511 | 0.515 | 0.409 | 0.932 | 0.932 | |
| 150k (75%) | 0.395 | 0.539 | 0.530 | 0.402 | 0.940 | 0.943 | |
| JaRS+ (no reference) | – | 0.638 | 0.799 | 0.869 | – | 0.902 |
Table 8. Sensitivity of the checkpoint contrasts and [15] on the unconditional DiT (trained for 200k steps) across all saved intermediate checkpoints (fraction of training completed shown in parentheses), evaluated at step 48 (, matching the final step of Stable Diffusion) with 16 Hutchinson probes. The ‘Dev’ columns are used to select the reference checkpoint for each task (bold indicates the selected checkpoint); the remaining columns report test-split performance. Sample-level detection follows the protocol of § 5.4.
For localization, the earliest checkpoint (5k steps) achieves the highest development All mIoU for both contrasts and is therefore used in Tab. 1. On WM alone, peaks at the 20k checkpoint (0.503 mIoU, which is not identified by the development split) and degrades sharply as the reference checkpoint advances through training.
ing, dropping to 0.243 at 150k steps. Across all five checkpoints, neither nor separates watermarked samples from non-memorized samples on WM+N (reaching at most 0.530 mIoU), and every checkpoint falls well short of JaRS+ across all three localization settings (by at least 0.099 mIoU).
For sample-level detection, the optimal reference checkpoint differs both between the two contrasts and from the checkpoint that is best for localization: peaks at the 50k checkpoint (0.867 test AUC) and remains below JaRS+ (0.902 AUC) for every reference, whereas improves monotonically with later checkpoints from 0.839 AUC (5k) to 0.943 AUC (150k), matching JaRS+ at 20k steps (0.903) and surpassing it at 50k–150k. Crucially, the 150k checkpoint that yields the best detection accuracy for yields the worst localization accuracy on All (0.402 mIoU), demonstrating that no single reference checkpoint serves both tasks well.
Prompt-Level Memorization Detection
Protocol. On Stable Diffusion, the positive set consists of the memorized prompts cataloged by Webster [33]: all 500 prompts for SD v1.4 and the 219 prompts labeled as verbatim copies for SD v2.1, while the negative set consists of the 500 non-memorized prompts from the localization benchmark. All metrics are reported on the held-out test split (400 memorized vs. 400 non-memorized prompts on SD v1.4, and 174 vs. 400 on SD v2.1). On the DiT, we score individual test samples (344 memorized watermark/near-copy samples vs. 241 non-memorized samples) using . For JaRS+, the detection step is the one already selected for localization (App. F), with no re-tuning. For the baselines on Stable Diffusion, we follow the protocol described in Kim and Lee [15 Sec. 5.2] and run their official code. On SD v2.1, our numbers match theirs closely (0.995 and 0.994 AUC for and , vs. 0.995 and 0.996 reported). On SD v1.4, achieves 0.969 AUC on our test split (and 0.969 across all 1,000 prompts), compared to 0.997 reported in Kim and Lee [15]; similarly, achieves 0.951 vs. 0.989.
Per-category breakdown. Table 9 decomposes the prompt-level detection results of § 5.4 across the prompt subcategories verified by Kim and Lee [15]. On every prompt that produces a verified template verbatim (TV) or matching verbatim (MV) copy under the four evaluated seeds, JaRS+ and all four contrasts achieve an AUC of 1.000. The small gap in overall AUC on SD v1.4 arises solely from retrieved verbatims (RV, which copy under only one or two of the four seeds) and from candidate prompts (‘N-ver.’) in the catalog of Webster [33] that do not produce a copy under any of the four seeds. On the unconditional DiT, all methods detect whole-image near-copies with 1.000 AUC, and the methods differ only on watermarked samples: JaRS+ achieves 0.861 AUC, compared to 0.811 for (50k reference) and 0.919 for (150k reference). Overall, the DiT AUC is 0.902 with 95% CI [0.877, 0.924] for JaRS+, 0.867 [0.836, 0.895] for , and 0.943 [0.925, 0.960] for .
| Method | Overall AUC [95% CI] | TV (147) | MV (44) | RV (28) | N-ver. (181) | Overall AUC [95% CI] | TV (145) | N-ver. (29) |
| Raw diagonal [15] | 0.906 [0.882, 0.928] | 0.960 | 0.966 | 0.927 | 0.845 | 0.871 [0.842, 0.899] | 0.882 | 0.819 |
| [15] | 0.969 [0.955, 0.982] | 1.000 | 1.000 | 0.999 | 0.932 | 0.995 [0.989, 1.000] | 1.000 | 0.972 |
| [15] | 0.951 [0.931, 0.970] | 1.000 | 1.000 | 0.999 | 0.892 | 0.994 [0.986, 0.999] | 1.000 | 0.964 |
| [15] | 0.969 [0.954, 0.983] | 1.000 | 1.000 | 1.000 | 0.931 | 0.994 [0.986, 0.999] | 1.000 | 0.966 |
| [15] | 0.957 [0.939, 0.974] | 1.000 | 1.000 | 1.000 | 0.905 | 0.993 [0.986, 0.999] | 1.000 | 0.961 |
| JaRS+ (ours) | 0.946 [0.930, 0.962] | 1.000 | 1.000 | 0.927 | 0.893 | 0.992 [0.982, 0.999] | 1.000 | 0.954 |
Table 9. Per-category breakdown of prompt-level detection on the test split (extending Table 2). Each category’s AUC is computed against all 400 non-memorized test prompts. Prompt subcategories follow the verification of Kim and Lee [15] across the four evaluated seeds (‘TV’ and ‘MV’: prompt copies on at least one of the four seeds; ‘RV’: retrieved verbatim; ‘N-ver.’: prompts from the list of Webster [33] that produce no verified copy under the four seeds; test prompt counts in parentheses). 95% confidence intervals are computed via 2,000 bootstrap resamples.
Realistic Vision. Because Kim and Lee [15] do not report prompt-level detection on Realistic Vision v5.1, we evaluate it under the same protocol. Since Realistic Vision is fine-tuned from Stable Diffusion v1.5, the positive set comprises the full catalog of 500 SD v1 memorized prompts from Webster [33].
As shown in Table 10, JaRS+ achieves 0.899 AUC, performing comparably to the checkpoint contrasts (: 0.907; : 0.918), trailing the null-prompt contrasts (: 0.937; : 0.960), and outperforming the raw diagonal by a wide margin (0.631 AUC). The per-category breakdown mirrors the pattern observed on SD v1.4: on prompts that generate a verified template verbatim (TV), JaRS+ and all four contrasts achieve at least 0.995 AUC, and the majority of the overall gap stems from the 198 candidate test prompts (‘N-ver.’) that produce no verified copy under the four evaluated seeds (0.807 vs. 0.918 for ). Unlike on SD v1.4, however, JaRS+ also exhibits a small gap on matching verbatims (0.964 vs. 1.000 AUC). These correspond to the same whole-image copies that receive lower scores in the localization benchmark (Table 6), confirming that the Stage 1 score accounts for both behaviors on Realistic Vision.
| Method | Overall AUC [95% CI] | TPR | TV (156) | MV (35) | RV (11) | N-ver. (198) |
| Raw diagonal [15] | 0.631 [0.591, 0.670] | 0.018 | 0.634 | 0.743 | 0.538 | 0.613 |
| [15] | 0.937 [0.917, 0.957] | 0.865 | 1.000 | 1.000 | 1.000 | 0.873 |
| [15] | 0.907 [0.882, 0.931] | 0.825 | 1.000 | 1.000 | 1.000 | 0.813 |
| [15] | 0.960 [0.942, 0.976] | 0.935 | 1.000 | 1.000 | 1.000 | 0.918 |
| [15] | 0.918 [0.894, 0.941] | 0.790 | 1.000 | 0.992 | 0.968 | 0.838 |
| JaRS+ (ours) | 0.899 [0.875, 0.922] | 0.640 | 0.995 | 0.964 | 0.977 | 0.807 |
Table 10. Prompt-level memorization detection on Realistic Vision v5.1 (test split), measured by AUROC (‘AUC’) and true-positive rate at a 1% false-positive rate (‘TPR’) across 400 memorized and 400 non-memorized test prompts. Prompt subcategories follow the verification of Kim and Lee [15] on Realistic Vision (test prompt counts in parentheses), with each category’s AUC computed against all 400 non-memorized test prompts. 95% confidence intervals are computed via 2,000 bootstrap resamples. The baselines of Kim and Lee [15] use 16 Hutchinson probes for the curvature maps and SD v1.1 as the checkpoint reference, matching Table 1.
Computation: Differentiation Mode, Precision, and Cost
In this section, we discuss the computation of the JaRS map. One might ask whether the row sums could instead be computed via reverse-mode automatic differentiation (backpropagation). Applying reverse-mode differentiation to yields the column sums rather than the row sums . Although the Bayes-optimal Jacobian (5) is symmetric, the Jacobian of a trained neural network is in general not symmetric. Forward directional derivatives (12) are therefore necessary to extract the true filter row sums. Furthermore, the forward passes in (12) must be evaluated in full single precision (fp32, with TensorFloat-32 disabled): reduced-precision convolutions corrupt the finite-difference quotient by roughly 20% on Stable Diffusion, whereas in fp32 the central difference (12) matches exact forward-mode automatic differentiation to within a relative error of .
Table 11 reports the cost of every method in Table 1. A JaRS map requires 8 forward passes and no backward pass, compared with the equivalent of about 21 forward passes for the raw diagonal and 41 for and on Stable Diffusion (a backward pass with respect to the input costs about 1.2 forward passes). On average over the broader test setting, JaRS+ requires 3.1–3.4 times fewer FLOPs than and on Stable Diffusion, and 4.8 times fewer than on the DiT. The wall-clock comparison on Stable Diffusion is closer, because the finite differences in (12) require fp32 while the code of Kim and Lee [15] runs in fp16: JaRS+ takes 0.62–0.75 s per image against 0.69–0.81 s for and 0.86–0.99 s for , and it is slower than the raw diagonal (0.43–0.50 s). In fp32, and take 2.8–3.8 s. On the DiT, where all maps run in fp32, JaRS+ takes 0.049 s against 0.213 s for . The score contrasts are the cheapest maps, with two forward passes, but they are less accurate (Table 1). Every map costs a fraction of generating the image itself.
| Method | Step | Precision | Stable Diffusion TFLOPs | SD v1.4 (s) | SD v2.1 (s) | RV (s) | DiT TFLOPs | Time (s) |
| Raw diagonal | last | 16.6 | 0.49 | 0.43 | 0.50 | 1.47 | 0.106 | |
| last | 33.3 | 0.81 | 0.69 | 0.81 | – | – | ||
| last | 33.3 | 0.99 | 0.86 | 0.99 | 2.95 | 0.213 | ||
| last | 1.6 | 0.04 | 0.03 | 0.04 | – | – | ||
| last | 1.6 | 0.05 | 0.04 | 0.05 | 0.09 | 0.012 | ||
| JaRS (one map) | 6.4 | 0.46 | 0.41 | 0.46 | 0.37 | 0.029 | ||
| JaRS+ (both stages) | 12.9 | 0.91 | 0.83 | 0.92 | 0.74 | 0.058 | ||
| JaRS+ (mean, broadest setting) | 9.7–10.6 | 0.75 | 0.62 | 0.73 | 0.61 | 0.049 | ||
| Generating the image | all | 80.3 | 1.80 | 1.69 | 1.84 | 2.30 | 0.329 |
Table 11. Cost of each localization map per test image on one NVIDIA A6000 GPU. FLOPs are counted with PyTorch’s FLOP counter; they do not depend on the image and are the same for the three Stable Diffusion models. Times are medians over 20 test prompts (test samples on the DiT) with three repetitions each. Every map is computed as in Tab. 1: on Stable Diffusion, the maps of Kim and Lee [15] with their official fp16 code (one forward pass that keeps the graph and 16 Hutchinson vector-Jacobian products), and on the DiT with our fp32 implementation (16 Rademacher probes with central-difference Jacobian-vector products); JaRS in fp32. ‘Last’ is step 49 of 50 on Stable Diffusion and step 48 on the DiT. JaRS+ computes the Stage 2 map only for images accepted by Stage 1: ‘both stages’ is the cost of an accepted image, and ‘mean’ averages over the broadest balanced test setting (All for SD v1.4, Realistic Vision, and the DiT; TV+N for SD v2.1), on which Stage 1 accepts 65%, 50%, 59%, and 67% of the images. Generating the image (50 steps, with classifier-free guidance on Stable Diffusion) is listed for reference.
The selection of on the development split is a one-time cost per model. It requires JaRS maps at the 10 candidate steps for the 206, 220, 161, and 153 development images used for selection on SD v1.4, SD v2.1, Realistic Vision, and the DiT: 16, 15, 12, and 1 minutes on one GPU, plus 6, 6, 5, and 1 minutes to generate the development images. The published configuration of Kim and Lee [15] requires no development split on Stable Diffusion. On the DiT, the reference checkpoint of the checkpoint contrasts is selected among five saved checkpoints on the development split (§ 5.1), which also requires these checkpoints to have been saved during training.
Region-Restricted Memorization Mitigation
Method
Existing inference-time mitigations prevent copying by modifying the generation globally, for instance by inserting random tokens into the prompt (RTA) [30], optimizing the prompt embedding to minimize the text-conditional noise prediction [34], or optimizing the initial noise latent [12]. While effective at suppressing the memorized template, a global intervention alters the entire image, destroying the unmemorized content that the user wished to keep.
Given the JaRS localization map , we can restrict any prompt-based mitigation exclusively to the copied region. Let be the binary latent mask obtained by thresholding and dilating the mask by 2 latent pixels, and let denote the clean latent of the original generation. We resample the image using the inpainting schedule of RePaint [17], initializing the trajectory with fresh Gaussian noise inside . At each denoising step, the guided noise prediction applies the mitigated prompt conditioning strictly inside while retaining the original prompt outside :
where , , and denote the noise predictions under the null prompt, the original prompt , and the mitigated conditioning , respectively, and is the classifier-free guidance scale. Following each denoising step to noise level , the latent outside is replaced by the forward-noised original latent (using a fixed noise realization ). This guarantees that the unmemorized region remains faithful to the original generation while the memorized region is synthesized anew under . In our experiments, we instantiate with two standard global mitigations: RTA [30] and the optimized embedding of Wen et al. [34].
Experimental Setup and Results
We evaluate how effectively our localization maps enable targeted copy removal via the region-restricted resampling procedure of Sec. K.1 on the 278 test TV instances of Stable Diffusion v1.4. For each instance, we form the binary mask by thresholding the step 15 JaRS map at a threshold selected on the development split and dilating by 2 latent pixels; the resulting mask covers 99.9% of the groundtruth template on average (and at least 97.9% on every test instance). To compare against contrast-based localization, we also form in the same manner from the best-performing contrast on the development split, (which covers 97.6% of the template on average, but falls below 99% coverage on 9.4% of instances). We test two prompt modification strategies at default and strong settings: random token addition (RTA [30], inserting 4 or 8 random tokens) and prompt embedding optimization (Wen et al. [34],
stopping once the objective drops to 30% or 15% of its initial value). We evaluate four metrics:
Copied (↓): The fraction of generated images that still reproduce the ground-truth template under the benchmark verification criterion.
LPIPS and LPIPS (↓): The perceptual distance [35] between the mitigated image and the original generation outside the predicted mask and outside the ground-truth template , respectively (note that evaluates the exact same pixels across all methods).
CLIP (↑): The CLIP similarity score [23] between the mitigated image and the original text prompt .
Table 12 presents the quantitative results, and qualitative examples are shown in Figure 6. When the prompt embedding optimization of Wen et al. [34] is restricted to the JaRS region , it simultaneously improves all metrics over whole-image mitigation: at the default setting, it reduces the copy rate from 20.5% to 9.0%, reduces perceptual change outside by 4.5× ( from 0.708 to 0.158, and from 0.689 to 0.239), and improves prompt alignment by +1.8 CLIP points (22.2 vs. 20.4). Similarly, restricting RTA [30] to the JaRS region matches the whole-image copy rate (43.2%) while reducing background distortion by nearly 5× ( from 0.444 to 0.094) and raising the CLIP score from 25.0 to 26.8.
| Mitigation | Scope | Copied | CLIP | ||
| None | Original image | 0.975 | 0.000 | 0.000 | 29.0 |
| RTA [30] | Whole image | 0.432 | 0.444 | 0.437 | 25.0 |
| Whole image (strong) | 0.245 | 0.600 | 0.591 | 21.3 | |
| Region , JaRS (ours) | 0.432 | 0.094 | 0.148 | 26.8 | |
| Region , JaRS, strong (ours) | 0.284 | 0.121 | 0.188 | 24.8 | |
| Region , [15] | 0.471 | 0.082 | 0.146 | 26.9 | |
| Region , , strong [15] | 0.306 | 0.111 | 0.195 | 24.8 | |
| Wen et al. [34] | Whole image | 0.205 | 0.708 | 0.689 | 20.4 |
| Whole image (strong) | 0.101 | 0.799 | 0.777 | 18.3 | |
| Region , JaRS (ours) | 0.090 | 0.158 | 0.239 | 22.2 | |
| Region , JaRS, strong (ours) | 0.043 | 0.186 | 0.276 | 20.4 | |
| Region , [15] | 0.101 | 0.155 | 0.261 | 22.3 | |
| Region , , strong [15] | 0.040 | 0.185 | 0.304 | 20.7 |
Table 12. Region-restricted vs. whole-image memorization mitigation on the 278 test TV instances of Stable Diffusion v1.4. ‘Copied’ is the fraction of mitigated images that still match the training template under the benchmark copy criterion; and measure perceptual change relative to the original truth template , respectively; and ‘CLIP’ measures image-prompt alignment. We compare global mitigation against restricting the intervention to the mask obtained from JaRS (contrast-free) or from [15].

Figure 6. Qualitative comparison of whole-image vs. region-restricted mitigation on six randomly sampled test ‘TV’ instances of Stable Diffusion v1.4 (columns 1, 3, 4, and 5 share the same memorized bedroom template). Row 1 shows the original generation; Row 2 overlays the dilated region localized by JaRS (magenta corresponds to the unmemorized complement that is preserved); Row 3 shows the result of applying the prompt-embedding optimization of Wen et al. [34] globally to the whole image; and Row 4 shows the result of restricting the same optimization to (Sec. K.1). Whole-image mitigation fails to remove the copied template in four of the six instances and heavily alters the unmemorized foreground rug whenever it succeeds. Restricting mitigation to removes the copied background template in five of the six instances while keeping the unmemorized region intact (inset labels report whether the template remains copied and the LPIPS distance to the original generation).
Two ablation checks clarify what drives these gains:
Role of fresh noise inside : Re-initializing with fresh Gaussian noise during RePaint resampling is essential, as retaining the original noise inside under Wen et al. [34] leaves 34.5% of templates copied; how-
ever, fresh noise alone does not explain the improvement over whole-image mitigation, as running whole-image mitigation from fresh noise still leaves 24.8% (default) and 10.4% (strong) copied while altering the entire image.
JaRS mask vs. mask: Using the mask from JaRS performs as well as or better than using the mask from without requiring a reference checkpoint. Under RTA, the JaRS mask removes significantly more copies than the mask (43.2% vs. 47.1% copied; 15 instances remain copied only under vs. 4 only under JaRS, McNemar’s test ), because occasionally misses portions of the template.
The same benefits hold on Stable Diffusion v2.1: whole-image mitigation leaves 83.2% (RTA) and 26.5% (Wen et al. [34]) of instances copied with of 0.17 and 0.38, whereas restricting mitigation to the JaRS region reduces the copy rate to 9.1% and 0.6% while lowering to 0.12 and 0.13.
Qualitative Examples
Figure 6 visualizes six randomly drawn test instances from our mitigation experiment (Sec. K.2), comparing the global prompt-embedding optimization of Wen et al. [34] against restricting the same optimization to the region localized by JaRS.
References
- [1]【1】 Costas Bekas, Effrosyni Kokiopoulou, and Yousef Saad. An estimator for the diagonal of a matrix. Applied numerical mathematics, 57(11-12):1214–1229, 2007.DOI
- [2]Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, 1 edition, 2007.
- [3]Benjamin Boys, Mark Girolami, Jakiw Pidstrigach, Sebastian Reich, Alan Mosca, and O. Deniz Akyildiz. Tweedie moment projected diffusions for inverse problems. arXiv preprint arXiv:2310.06721, 2024.arxiv.org/abs/2310.06721
- [4]Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models, 2023.arxiv.org/abs/2301.13188
- [5]Chen Chen, Daochang Liu, Mubarak Shah, and Chang Xu. Exploring local memorization in diffusion models via bright ending attention. In International Conference on Learning Representations, pages 29827–29841, 2025.arxiv.org/abs/2410.21665
- [6]Chen Chen, Daochang Liu, Mubarak Shah, and Chang Xu. Enhancing privacy-utility trade-offs to mitigate memorization in diffusion models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8182–8191. IEEE, 2025.arxiv.org/abs/2504.18032
- [7]Bradley Efron. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496):1602–1614, 2011.pubmed.ncbi.nlm.nih.gov/22505788
- [8]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.arxiv.org/abs/2207.12598
- [9]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pages 6840–6851. Curran Associates, Inc., 2020.arxiv.org/abs/2006.11239
- [10]Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation, 18(3):1059–1076, 1989.
- [11]Achref Jaziri, Martin Mundt, Ahmed Imtiaz Humayun, and Visvanthan Ramesh. Spectral signatures of memorization basins in diffusion trajectories. arXiv preprint arXiv:2611.01775, 2026.arxiv.org/abs/2611.01775
- [12]Dongjae Jeon, Dueun Kim, and Albert No. Understanding and mitigating memorization in generative models via sharpness of probability landscapes. arXiv preprint arXiv:2412.04140, 2024.arxiv.org/abs/2412.04140
- [13]Zahra Kadkhodaie, Florentin Guth, Eero Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. In International Conference on Learning Representations, pages 46543–46567, 2024.
- [14]Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 4396–4405. IEEE, 2019.DOI
- [15]Gwangho Kim and Sungyoon Lee. Localizing memorized regions in diffusion models via coordinate-wise curvature differences. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2605.26756
- [16]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02747
- [17]Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 11451–11461. IEEE, 2022.arxiv.org/abs/2201.09865
- [18]Peyman Milanfar. A tour of modern image filtering: New insights and methods, both practical and theoretical. IEEE Signal Processing Magazine, 30(1):106–128, 2013.
- [19]Koichi Miyasawa. An empirical bayes estimator of the mean of a normal population. Bull. Inst. Internat. Statist. 38(181-188):1–2, 1961.
- [20]Sreyas Mohan, Zahra Kadkhodaie, Eero P. Simoncelli, and Carlos Fernandez-Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks. In International Conference on Learning Representations, 2020.
- [21]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4195–4205, 2023.arxiv.org/abs/2212.09748
- [22]Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthias Douze. A self-supervised descriptor for image copy detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14512–14522. IEEE, 2022.arxiv.org/abs/2202.10261
- [23]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
- [24]Herbert E Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pages 388–394. Springer, 1992.
- [25]Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (RED). SIAM journal on imaging sciences, 10(4):1804–1844, 2017.arxiv.org/abs/1611.02862
- [26]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022.arxiv.org/abs/2112.10752
- [27]Brendan Ross, Hamidreza Kamkari, Tongzi Wu, Rasa Hosseinzadeh, Zhaoyan Liu, George Stein, Jesse Cresswell, and Gabriel Loaiza-Ganem. A geometric framework for understanding memorization in generative models. In International Conference on Learning Representations, pages 34773–34805, 2025.arxiv.org/abs/2411.00113
- [28]Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, pages 2256–2265, Lille, France, 2015. PMLR.
- [29]Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6048–6058. IEEE, 2023. 1, 2arxiv.org/abs/2212.03860
- [30]Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36:47783–47803, 2023. 1, 2, 18, 19arxiv.org/abs/2305.20086
- [31]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. 1, 3arxiv.org/abs/2010.02502
- [32]Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. 1
- [33]Ryan Webster. A reproducible extraction of training images from diffusion models. arXiv preprint arXiv:2305.08694, 2023. 1, 2, 6, 16, 17, 18arxiv.org/abs/2305.08694
- [34]Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memorization in diffusion models. In The Twelfth International Conference on Learning Representations, 2024. 2, 4, 6, 18, 19, 20, 21arxiv.org/abs/2407.21720
- [35]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF conference on computer vision and pattern recognition, pages 586–595. IEEE, 2018. 19arxiv.org/abs/1801.03924



