betaFeedback

WorksComputer Science

Counterfactual Feature Steering for Adjacency-Preserving Identity Unlearning

Abstract

Identity unlearning in text-to-image diffusion models is increasingly important for privacy, safety, and ethical deployment, as it aims to remove the model's ability to generate specific individuals. In this work, we study identity unlearning with adjacency preservation, where the goal is to remove a target individual's identity while preserving related non-target identities. We propose Counterfactual Feature Steering (CFS), a method requiring one-time SAE training that performs bounded inference-time concept editing in the latent space of a sparse autoencoder attached to the text encoder. CFS derives a target direction by contrasting forget and retain distributions, then applies a per-token projection proportional to each token's current alignment with this direction, yielding counterfactual representations that suppress the target while limiting collateral damage. To evaluate adjacency preservation, we introduce the Adjacency-Weighted Unlearning Score (AWUS), which measures the tradeoff between target erasure and collateral damage, and extend it to face-specific identity similarity using ArcFace. Across 4,000 prompts derived from Celeb-100, spanning 100 celebrity identities, CFS achieves the strongest evaluated tradeoff between identity erasure of target identities and preservation of adjacent and generic concepts, while remaining stable across intervention strengths where other steering-based methods degrade. Independent ArcFace evaluation and a 10,000-caption COCO study further examine identity and general preservation. Quantitative nudity-suppression benchmarks and qualitative artistic-style examples provide evidence of transfer beyond identities on SD~1.4.

Introduction

Generative models have recently shown striking abilities across domains [18, 16], from natural language understanding to high-fidelity image synthesis. Text-to-image models [18], in particular, can translate simple prompts into rich visual content, but this capability also raises safety and privacy concerns: models trained on large web-scale datasets may reproduce sensitive, copyrighted, or personal data without consent. Mitigating these risks is important for both safety and legal compliance, including regulations such as European Union’s (EU) General Data Protection Regulation (GDPR) [17]. As a result, targeted model editing and unlearning have received increasing attention as ways to remove undesirable concepts while preserving broader model utility [9, 7].

Toward removing such undesirable concepts, a key challenge is first to locate them within the model’s internal representations. Mechanistic interpretability approaches—particularly Sparse Autoencoders (SAEs) [2, 10] offer a promising avenue for this [4, 15, 11]. SAEs are trained to decompose dense model activations into a set of sparse features, often aligning with more interpretable, monosemantic concepts [21, 4]. By operating in this sparse feature space, one can identify components associated with specific attributes (e.g., nudity or identity-related cues) and apply targeted interventions, such as penalizing or steering their activations. This enables selective unlearning or editing of concepts while minimizing unintended effects on the model’s broader capabilities [12, 4, 26].

While such steering-based approaches are appealing because they are lightweight and do not require access to training data or model retraining, existing methods typically apply a fixed intervention strength in the latent space to steer the model away from a target concept. This leads to two recurring structural issues. First, this strength is unbounded: increasing it often has limited impact up to a point before abruptly degrading outputs or causing representation collapse. Second, because the same strength is applied uniformly across prompts and tokens, the intervention may fail to distinguish cases where the target concept is strongly present from those where it is only weakly expressed, increasing the risk of unintended changes to semantically adjacent concepts.

This gives rise to the central problem we study in this paper: fine-grained unlearning, where the goal is not only to remove a target concept, but to do so while preserving semantically adjacent concepts. This distinction is especially important for identity unlearning. For example, removing one high-profile technology CEO should not cause the model to also suppress other visually or semantically related public figures because they occupy nearby regions in representation space. When identities share overlapping attributes, such as profession, social context, appearance, or prompt distribution—unlearning methods must avoid treating these shared features as part of the target concept. We refer to this challenge as adjacency preservation: the ability to erase a specified concept while maintaining the model’s ability to generate related but non-target concepts. Prior work has already investigated related collateral-erasure problems: FADE formalizes concept neighborhoods, while EraseBench measures spillover degradation across non-target concepts [23, 1]. We focus here on input-adaptive, inference-time identity suppression and evaluate whether it can preserve neighboring identities.

To address this gap, we take a different approach to concept steering. Rather than pushing representations away from a target concept in an unconstrained manner, we first measure how strongly the current activation aligns with that concept and then apply a controlled correction. Based on this idea, we propose Counterfactual Feature Steering (CFS), a method requiring one-time domain-level SAE training that performs bounded, input-adaptive projections in the latent space of a SAE attached to the text encoder. For each token, CFS estimates a scalar signal reflecting alignment with the target concept and applies a projection step that reduces this signal by moving the representation toward a retain distribution, while keeping the change minimal. This formulation yields several properties along the selected target axis: the intervention cannot cross the retain-side boundary for s∈[0,1]s \in[0,1], and it is selective and adaptive because tokens with nonpositive target alignment remain unchanged. At s=1s = 1 it is the minimum-Euclidean-norm projection onto the retain half-space in SAE coordinates; these geometric properties do not themselves guarantee preservation of all neighboring concepts or image attributes.

We evaluate CFS primarily in the context of identity unlearning, with a particular focus on preserving semantically adjacent concepts. To this end, we conduct experiments on a text-to-image generative model, Stable Diffusion, where we apply CFS to remove identities from the Celeb-100 dataset and analyze its effect on related identities (e.g., unlearning Barack Obama and evaluating its impact on Joe Biden). Since identity-level unlearning with explicit adjacency preservation has not been systematically studied, we introduce a suite of metrics to quantify overall unlearning utility. In particular, we measure target erasure while minimizing collateral damage to adjacent concepts. These effects are evaluated across multiple spaces, including representation space using CLIP, face embedding space using ArcFace, and perceptual similarity using LPIPS. Beyond identity unlearning, we evaluate CFS on nudity removal and provide qualitative style-editing examples. Our intended deployment is a hosted inference setting where a model provider controls the intervention and can activate target-specific directions without maintaining a separate weight-edited checkpoint; it is not a tamper-resistant removal mechanism for users controlling the serving code. We summarize our contributions as follows:

  • We identify adjacency preservation as a key challenge in fine-grained unlearning, showing that removing a target concept can inadvertently affect semantically related non-target concepts.

  • We propose Counterfactual Feature Steering (CFS), an SAE-based steering method with one-time domain-level training and bounded, input-adaptive projections in latent space.

  • We introduce adjacency-aware evaluation metrics for identity unlearning that jointly measure target erasure and collateral damage across CLIP, ArcFace, and LPIPS spaces.

  • We demonstrate that CFS improves fine-grained identity unlearning while better preserving adjacent identities, and provide quantitative nudity-suppression and qualitative style-editing.

Related Work

Concept unlearning in diffusion models. The problem of removing a concept from a pretrained diffusion model without full retraining has been approached from two directions. Weight-editing methods modify model parameters to suppress a target concept. ESD [7] fine-tunes the UNet cross-attention weights using a contrastive objective that pushes outputs away from the concept. UCE [8] derives a closed-form weight update that preserves manually specified anchor concepts. MACE [13] combines LoRA fine-tuning with segmentation-guided masks for mass concept erasure across many targets simultaneously. These methods are effective but require modifying and storing updated weights, making them slow and difficult to reverse or compose.

In contrast, inference-time steering methods operate on intermediate activations without modifying weights, making them lightweight and reversible. SLD [19] suppresses unsafe content by modifying the classifier-free guidance signal at each diffusion step. One Step Is Enough [21] applies a fixed mean concept vector to the UNet residual stream of SDXL-Turbo. SAeUron [4] identifies and multiplicatively suppresses concept-related SAE features in the UNet activations. Concept Steerers [12] uses a sparse autoencoder trained on the CLIP text encoder to identify concept directions and subtracts them at inference time. FADE [23] formalizes a Concept Neighborhood to preserve related concepts during erasure, and EraseBench [1] documents collateral degradation of visually and semantically related concepts. CFS differs in its inference-time, input-dependent projection mechanism and identity-specific preservation analysis.

Sparse autoencoders for mechanistic interpretability. Sparse autoencoders have emerged as a tool for decomposing dense neural network activations into interpretable, approximately monosemantic features [2, 22]. Originally developed for language models [10], they have been applied to diffusion model for concept identification [24, 25, 12, 4, 21].

Machine unlearning beyond diffusion models. Machine unlearning [3, 9] originally targeted classification models with the goal of removing the influence of specific training samples. More recent work addresses unlearning in large language models [6, 27] and biometric recognition systems [20]. In contrast, generative unlearning requires removing a distributed semantic concept that is not tied to a single data point but encoded across many prompts and model parameters, making the problem fundamentally different.

Counterfactual Feature Steering

Given a prompt, we construct a counterfactual representation by projecting dense activations from an internal layer of the text encoder into an SAE latent space. We then intervene on selected sparse features to reduce alignment with target concepts and decode the modified representation back into the model. This yields a bounded, input-dependent update: the intervention is applied only when the current SAE activation aligns with the target-concept direction, thereby suppressing the target concept while preserving unrelated information.

Our method consists of two main components: (i) identifying a direction in feature space that captures the target concept while avoiding features associated with retained concepts, and (ii) applying a per-token projection at inference time that edits SAE features along this direction to produce counterfactual representations. A schematic overview of the method is shown in Figure 1.

Counterfactual Feature Steering (CFS) overview

Figure 1. Counterfactual Feature Steering (CFS). (a) CFS computes sparse forget/retain statistics with a frozen K-SAE and derives an identity direction. (b) During generation, CFS projects the sparse code toward the retain distribution, decodes the sparse edit back to hidden space, and injects the resulting update into the frozen diffusion U-Net. The target identity is removed while adjacent and generic concepts are preserved.

Preliminaries

Text-conditional diffusion. Stable Diffusion 1.4 generates an image from a prompt pp by first tokenising the prompt into a fixed-length sequence of T=77T = 77 tokens (padding or truncating as necessary) and then encoding this sequence with a CLIP text encoder to produce hidden states h1:T∈RT×d\mathbf{h}_{1:T} \in\mathbb{R}^{T \times d}. The value T=77T = 77 is determined by CLIP’s maximum context length [16]. These embeddings condition a U-Net through cross-attention, and classifier-free guidance combines conditional and unconditional branches. We intervene at layer ℓ=9\ell= 9 of the CLIP text encoder. We select this layer by ablation (§ 4.4): layer 9 achieves the highest observed AWUS among layers tested (4, 6, 8, 9, 10, 11). This is a task- and backbone-specific empirical choice, not a universal layer-selection.

Sparse autoencoder representation. We train a k-sparse autoencoder (k-SAE) [14] on layer-ℓ\ell text-encoder activations [2, 10] collected from the full prompt corpus. The identity-domain SAE is trained once (approximately 2 hours in our reported setup) and reused across all eight identity targets; new targets require only cacheable no-gradient forget/retain statistics and do not require training a separate SAE or editing diffusion-model weights. The nudity and style experiments use their own reusable domain-level SAEs. Per-target cached statistics occupy approximately 925 KB. Computational accounting is detailed in Supplementary Appendix § 15. Each hidden state ht∈Rd\mathbf{h}_t \in\mathbb{R}^{d} is mapped to a sparse feature vector zt∈Rn\mathbf{z}_t \in\mathbb{R}^{n} with n≫dn \gg d, via

zt=TopK⁡(ReLU⁡(Wencht+b),k),h^t=ztWdec⊤.\mathbf{z}_t = \operatorname{TopK}\left(\operatorname{ReLU}\left(\mathbf{W}_{\mathrm{enc}}\mathbf{h}_t + \mathbf{b}\right), k\right), \qquad\hat{\mathbf{h}}_t = \mathbf{z}_t\mathbf{W}_{\mathrm{dec}}^{\top}.

Because the SAE decoder is linear in zt\mathbf{z}_t, differences in feature space map directly to differences in hidden space: Δht=ΔztWdec⊤\Delta\mathbf{h}_t = \Delta\mathbf{z}_t\mathbf{W}_{\mathrm{dec}}^{\top}. We exploit this property by defining the intervention in the sparse SAE space, where target-aligned features can be identified and edited, and then applying the corresponding decoded update directly in hidden space.

Target-concept Direction

Given a target concept cc, we partition the full prompt corpus into a forget set Pf={p:c∈p}\mathcal{P}_f = \{p : c \in p\} of prompts mentioning the target and a retain set Pr={p:c∉p}\mathcal{P}_r = \{p : c \notin p\} of all remaining prompts. Each prompt is tokenised into up to a fixed-length sequence of TT. Usually, most token positions encode generic concepts (articles, prepositions, style descriptors) with a small number encoding the target identity itself (e.g. the tokens barack and obama in “A realistic portrait of Barack Obama”).

Because all prompts in Pf\mathcal{P}_f share the target name and tokenise the target concept at similar positions, SAE features associated with the target identity are expected to activate systematically at those positions across Pf\mathcal{P}_f and remain less active at the same positions across Pr\mathcal{P}_r (where the prompt mentions different identities). The forget-retain contrast at each token position therefore isolates identity-specific features from generic content features. Figure H.1 in the supplementary appendix confirms this empirically: the highest-scoring features concentrate their activation at the name-bearing token positions.

Let V⊆{1,…,T}\mathcal{V} \subseteq\{1,\ldots,T\} denote the set of non-padding token positions for a given prompt. For each feature index ii and position t∈Vt \in\mathcal{V}, we compute the means and standard deviations of ztiz_{ti} across the forget and retain sets:

μtif=Ep∈Pf[zt(p)[i]],σtif=Std⁡p∈Pf[zti(p)],(1)\mu_{ti}^{f} = \mathbb{E}_{p\in\mathcal{P}_f}\left[z_{t}(p)[i]\right], \qquad\sigma_{ti}^{f} = \operatorname{Std}_{p\in\mathcal{P}_f}\left[z_{ti}(p)\right], \tag*{(1)}

and analogously μtir,σtir\mu_{ti}^{r},\sigma_{ti}^{r} over Pr\mathcal{P}_r.

We then score each feature by its discriminative power, averaged over valid token positions:

Si=1∣V∣∑t∈V(μtif−λrμtir)1∣V∣∑t∈V(σtif+σtir)+ε.(2)S_i = \frac{\frac{1}{|\mathcal{V}|}\sum_{t\in\mathcal{V}}\left(\mu_{ti}^{f}-\lambda_r\mu_{ti}^{r}\right)}{\frac{1}{|\mathcal{V}|}\sum_{t\in\mathcal{V}}\left(\sigma_{ti}^{f}+\sigma_{ti}^{r}\right)+\varepsilon}. \tag*{(2)}

The numerator favours features that fire strongly across the forget set and weakly across the retain set, modulated by a retain penalty λr\lambda_r. The denominator penalises features whose retain-set activations vary unpredictably from prompt to prompt: a feature with high σr\sigma^r does not have a stable “baseline” activation on non-target prompts, so subtracting it would induce inconsistent edits across non-target generations. We select the top-kk features and represent the resulting index set with I∗\mathcal{I}^*. Ablations over kk and λr\lambda_r are reported in § 4.4.

From the top-kk features we form a per-token direction in SAE space pointing from the retain centroid to the forget centroid:

dt=μtf−μtr∈Rn(coordinates outside I∗ set to zero),d^t=dt∥dt∥2.(3)\mathbf{d}_t=\mu_t^f-\mu_t^r\in\mathbb{R}^n \quad\text{(coordinates outside }\mathcal{I}^*\text{ set to zero)}, \qquad\hat{\mathbf{d}}_t=\frac{\mathbf{d}_t}{\lVert\mathbf{d}_t\rVert_2}. \tag*{(3)}

The unit vector d^t\hat{\mathbf{d}}_t encodes the feature-space deviation of the target concept from the retain distribution at token position tt.

Counterfactual Projection

At generation time, we apply a forward hook at the layer-ℓ\ell output of the conditional branch of the text encoder. For each token position tt, let ht\mathbf{h}_t be the dense hidden representation and zt=TopK⁡(ReLU⁡(Wencht+b),k)\mathbf{z}_t=\operatorname{TopK}(\operatorname{ReLU}(W_{\mathrm{enc}}\mathbf{h}_t+\mathbf{b}),k) its kk-SAE encoding. The hook performs three operations.

Step 1: Measure the alignment signal. We compute a scalar that captures how much of the concept direction is present in the sparse encoding at token position tt:

αt=max⁡((zt−μtr)⋅d^t,0)≥0.(4)\alpha_t=\max\left((\mathbf{z}_t-\mu_t^r)\cdot\hat{\mathbf{d}}_t,0\right)\ge0. \tag*{(4)}

Here d^t\hat{\mathbf{d}}_t is the unit target-concept axis: the dot product projects zt−μtr\mathbf{z}_t-\mu_t^r onto this axis to measure the target component. αt\alpha_t is large when the prompt strongly invokes the target at position tt and near zero otherwise. The clamp prevents amplifying the concept signal when the sparse activation is already towards the retain side.

Step 2: Projection in SAE space. We then define a counterfactual SAE code by removing an sαts\alpha_t-scaled component along the same direction:

ztcf=zt−s αt d^t,s∈[0,1].(5)\mathbf{z}_t^{\mathrm{cf}}=\mathbf{z}_t-s\,\alpha_t\,\hat{\mathbf{d}}_t,\qquad s\in[0,1]. \tag*{(5)}

The same target-concept axis d^t\hat{\mathbf{d}}_t is used twice: Eq. (4) measures the target component, and Eq. (5) subtracts that measured component along the same axis. The scalar ss controls the strength of the counterfactual intervention: s=0s=0 recovers the original representation, while larger values of ss produce increasingly counterfactual representations by shifting the SAE features away from the target-concept direction. At s=1s=1, this is the Euclidean projection onto the closed retain half-space Ht={u:(u−μtr)⊤d^t≤0}\mathcal{H}_t=\{u:(u-\mu_t^r)^\top\hat{\mathbf{d}}_t\le0\}: tokens already on the retain side receive no edit, while positively aligned tokens are projected onto its boundary. For intermediate strengths, the update interpolates between the original code and this projection.

Proposition 1 (Bounded counterfactual intervention). Let qt=(zt−μtr)⊤d^tq_t = (z_t - \mu_t^r)^\top\hat{d}_t and αt=[qt]+=max⁡(qt,0)\alpha_t = [q_t]_+ = \max(q_t, 0). For s∈[0,1]s \in[0, 1], the update ztcf=zt−sαtd^tz_t^{\mathrm{cf}} = z_t - s\alpha_t\hat{d}_t satisfies

(ztcf−μtr)⊤d^t=qt−s[qt]+={(1−s)qt,qt>0,qt,qt≤0.\left(z_t^{\mathrm{cf}}-\mu_t^r\right)^\top\hat{d}_t = q_t-s[q_t]_+ = \begin{cases} (1-s)q_t, & q_t>0,\\ q_t, & q_t\leq0. \end{cases}

Thus positive target-direction evidence decreases without crossing the retain-side boundary, while nonpositive evidence is left unchanged. At s=1s=1, ztcf=ΠHt−(zt)z_t^{\mathrm{cf}} = \Pi_{\mathcal{H}_t^-}(z_t), the unique minimum-Euclidean-distance projection of ztz_t onto Ht−\mathcal{H}_t^- in SAE coordinates.

Full proof is available in Supplementary Appendix § 7

Step 3: Decode to hidden space. Because the SAE decoder is linear, the SAE-space change zt−ztcf=sαtd^tz_t-z_t^{\mathrm{cf}}=s\alpha_t\hat{d}_t maps directly to a corresponding hidden-space update. Thus, rather than decoding the full counterfactual code, we can apply only the decoded intervention direction to the original hidden state:

htCFS=ht−sαt(d^tWdec⊤).(6)h_t^{\mathrm{CFS}} = h_t - s\alpha_t\left(\hat{d}_t W_{\mathrm{dec}}^\top\right). \tag*{(6)}

Here, d^tWdec⊤∈Rd\hat{d}_t W_{\mathrm{dec}}^\top\in\mathbb{R}^d is the hidden-space direction induced by the target-concept direction in the SAE latent space. These decoded directions are precomputed once per target, avoiding the need to explicitly construct the counterfactual code ztcfz_t^{\mathrm{cf}} at inference time.

Taken together, these steps implement CFS as a lightweight inference-time intervention: the target direction is identified once in SAE space, decoded once into hidden space, and then applied per token according to the live alignment score αt\alpha_t. As a result, prompts that do not activate the target direction receive little or no update, while prompts containing the target concept are steered toward a counterfactual representation with the target-aligned component suppressed.

Experiments & Results

Experimental setup

Model and baselines. We evaluate on Stable Diffusion 1.4 [18] and compare CFS against six baselines spanning both inference-time steering and weight-editing approaches: Concept Steerers [12] (SAE-based fixed-vector subtraction), SAeUron [4] (SAE feature suppression on the UNet), OneStep [21] (fixed mean concept vector), SLD [19] (inference-time CFG suppression), ESD [7] (UNet fine-tuning), and UCE [8] (closed-form weight editing).

Baseline evaluation protocol. All methods use the same identity targets, prompts, seeds, inference steps, and guidance settings in the reported comparison. CFS is evaluated at full steering strength s=1s = 1; strength-controlled baselines are reported at their best AWUS operating point over the evaluated strength sweeps, while ESD and UCE use the corresponding edited checkpoints. Fixed-full-strength CLIP results appear in Supplementary Appendix § 16. Complete implementation and evaluation details are documented in Supplementary Appendix § 17.

Identity unlearning targets. To our knowledge, no standard benchmark currently exists for identity unlearning while preserving adjacent identities. In this work, we study this setting through adjacency preservation, which requires maintaining each target identity’s associations with related identities after unlearning. To evaluate this problem, we construct a corpus of 4,000 prompts derived from CelebA-100, covering 100 celebrity identities across politics, sports, entertainment, and technology. From this corpus, we select eight target identities spanning diverse domains: Barack Obama, Joe Biden, Elon Musk, Bill Gates, Lionel Messi, Cristiano Ronaldo, Bradley Cooper, and Scarlett Johansson. For each target we define the forget set Pf\mathcal{P}_f as all prompts naming that identity, and the retain set Pr\mathcal{P}_r as all remaining prompts. We evaluate adjacency preservation using 4 adjacent identities per target, chosen based on semantic relatedness and image-space similarity (Supplementary Appendix § 10; Figure E.1; full list in Supplementary Appendix § 6), and 9 generic prompts.

Model-internal adjacency check. As a complementary analysis to semantic and CLIP-image-based peer selection, we average name-token representations from SD 1.4’s frozen layer-9 text encoder across prompts for 100 identities and compute pairwise cosine similarity. This analysis uses raw model conditioning representations rather than SAE features or generated-image evaluation scores. Examples include Ronaldo–Messi (0.460), Obama–Biden (0.394), Musk–Zuckerberg (0.401), Cooper–Gosling (0.421), and Johansson–Portman (0.506). For the 13 manually selected peers present in this identity corpus, the ranking achieves MRR 0.487. This establishes a model-grounded view of proximity, not a formal or unique definition of adjacency. Layer 9 was selected by the downstream intervention ablation, independently of this post-hoc adjacency analysis. Additional examples appear in Supplementary Appendix D.

Generalization experiments. Beyond identity, we further evaluate CFS on nudity unlearning using the I2P benchmark [19] and on style unlearning using prompts targeting Van Gogh. For nudity, we use an SAE trained on I2P prompts; for style, we use an SAE trained on the style-prompt corpus from Unlearn Canvas [28] and apply it at layer 9. We use the same CFS hyperparameters across all experiments, k=48k = 48 and λr=0.1\lambda_r = 0.1, without changing the projection rule. These experiments test cross-domain transfer on SD 1.4, which confirms the method’s generalization capabilities.

Evaluation metrics

To quantify whether unlearning a target identity also erases information about adjacent identities, we introduce the Adjacency-Weighted Unlearning Score (AWUS). At a high level, AWUS captures the tradeoff

AWUS=Target Erasure−Collateral Damage,\mathrm{AWUS} = \text{Target Erasure} - \text{Collateral Damage},

where the first term measures how effectively the target identity is erased, and the second penalizes unintended changes to adjacent and generic prompts. AWUS is computed in CLIP embedding space, so adjacency reflects semantic similarity between identities.

To capture identity-specific preservation beyond semantic similarity, we extend AWUS to a face-focused embedding space using ArcFace, yielding FaceAWUS. FaceAWUS uses the same target-erasure versus collateral-damage structure, but defines adjacency using facial identity similarity rather than CLIP-space semantic similarity. We further measure collateral forgetting using an LPIPS-gap metric, which quantifies how much more the target image changes than adjacent and generic images after unlearning. Together, AWUS, FaceAWUS, and LPIPS-gap provide complementary views across semantic, face-identity, and perceptual spaces: the target concept should change substantially, while adjacent and generic prompts should remain stable. CLIP-based scores and whole-image LPIPS can respond to backgrounds and other attributes as well as identity; ArcFace evaluation of detected faces provides complementary identity-specific evidence, but depends on the recognizability of the baseline face and on successful detection. Formal definitions are provided in Supplementary Appendix C.

Results

Main Results. Table 1 reports the three net utility metrics averaged over 8 celebrity targets. CFS achieves the highest AWUS (0.296), FaceAWUS (0.636), and LPIPS-gap (0.402) across all methods. Methods with higher raw erasure scores achieve them at 12–22× the collateral damage, which the net metrics correctly penalise.

MethodAWUS ↑FaceAWUS ↑LPIPS-gap ↑
SAeUron0.2330.3000.108
Concept Steerers0.279−0.0770.038
UCE0.1980.5430.163
ESD0.2620.1380.051
SLD0.0030.0930.022
One-Step0.1060.1360.033
CFS (Ours)0.2960.6360.402

Table 1. Identity unlearning: net utility metrics. We report the three metrics defined in Section 4.2; each rewards target erasure while penalising adjacency damage. Higher is better for all metrics. Bold: best. Per-target results are reported in Supplementary Appendix I.

Independent face-identity validation. To separate face-identity changes from changes to background and composition, we evaluate detected face regions with ArcFace under paired prompt/seed/generation settings. References are constructed from unmodified SD 1.4 generations, not from real celebrity photographs. At full CFS strength, mean target-face similarity is 0.160 (n=38n = 38), while adjacent identities retain similarity 0.760 (n=56n = 56). Face detection fails for 20.8% of target outputs and 3.4% of peer outputs. These results are consistent with greater target-face disruption than peer-face disruption, but detection failures and variability of baseline identity rendering constrain interpretation. The unmodified model’s self-consistency across three prompt templates ranges from approximately 0.15 to 0.66; most reported targets fall around 0.45–0.66. Lower-consistency targets are weaker tests of meaningful identity removal.

Preservation of general generation quality. We evaluate 10,000 COCO-30K captions at full CFS strength, averaging over all eight identity directions. Baseline and edited generations use identical captions, seeds, sampler, guidance, and inference steps; FID is calculated against the same corresponding real COCO images. Unmodified SD 1.4 has FID 15.71 and CLIPScore 27.39, versus FID 16.11 and CLIPScore 26.95 for CFS (changes of +0.40 and −0.44, respectively). This evaluates broad generation preservation beyond the selected identities.

Figure 2 provides a side-by-side qualitative comparison of target removal and collateral damage. The green columns show target prompts generation for the erased identity, Scarlett Johansson; the pink columns show adjacent identities, Emma Stone and Natalie Portman; and the light blue columns show generic person prompts generation after the identity removal. CFS removes the target identity while preserving adjacent and generic identities along with the scene attributes. In contrast, prior methods with high collateral damage introduce noticeable distortions in adjacent and generic identities, reflecting their lack of input adaptivity and feature disentanglement.

Adjacency preservation under identity concept removal

Figure 2. Adjacency preservation under identity concept removal. CFS removes the target identity while preserving adjacent and generic prompts.

Strength stability. Figure 3 shows that CFS remains stable across projection strengths. Qualitatively, identity is progressively removed as ss increases while pose, lighting, and background remain stable. Quantitatively, AWUS improves up to full projection, whereas subtraction-based methods often degrade at high strength due to unbounded edits. A face-space trade-off plot in Supplementary Appendix F further shows that CFS achieves high target-identity change with low adjacent-identity drift.

Qualitative sweep over steering strength and AWUS across steering strengths

Figure 3. Strength stability. Left: qualitative sweep over steering strength ss. Right: AWUS across strengths; dashed lines denote weight-editing methods without a test-time strength parameter.

Identity geometry and semantic alignment. Figure 4 analyzes why CFS preserves adjacent identities. Most measured identity directions have low pairwise cosine similarity, consistent with limited overlap along the chosen axes, but orthogonality is not guaranteed; Ronaldo–Messi illustrates a more entangled pair for which collateral effects remain possible. The alignment signal (Figure 4, right) responds to semantic descriptions: for Elon Musk, “CEO of Tesla” and “Founded SpaceX” reach 63% and 55% of the explicit-name signal, while Bill Gates remains low (bottom). Similarly, Obama-related circumlocutions activate the target direction more strongly than the adjacent identity Joe Biden (top panel). Together, these results show that CFS captures semantic identity signals while limiting activation on adjacent identities. A token-level heatmap in Supplementary Appendix H confirms that the alignment signal concentrates on identity-bearing tokens.

Identity-direction geometry and semantic alignment

Figure 4. Identity-direction geometry and semantic alignment. Left: pairwise cosine similarity between CFS identity directions. Right: alignment signals for explicit names, semantic circumlocutions, generic controls, and adjacent identities.

Additional qualitative results in Supplementary Appendix G show that CFS can compose two identity directions in a single generation and can erase identities from circumlocution prompts.

Generalisation: Nudity and Style unlearning. Although our primary focus is identity unlearning, CFS is concept-agnostic: the same projection mechanism can be applied to other concept unlearning tasks. We test this on two additional settings: nudity suppression and artistic-style removal.

Figure 5 shows that the same CFS mechanism extends beyond identity removal. In the unsafe-content examples, the intervention reduces the targeted visual content while largely preserving the coarse scene layout. In the style examples, CFS progressively removes recognizable Van Gogh-style features while maintaining the object and scene structure, such as the bedroom layout, sunflower arrangement, and wheat-field composition. These results suggest that CFS is not restricted to identity-specific directions; rather, it can operate on different concept directions learned in SAE feature space.

Generalisation beyond identity unlearning

Figure 5. Generalisation beyond identity unlearning. Top: nudity suppression. Bottom: Van Gogh-style removal. CFS reduces the target concept while preserving coarse scene structure.

Quantitative nudity-suppression transfer. Table 2 reports Nudity Detection Rate (%, lower is better) from the rebuttal experiments on I2P, Ring-a-Bell, and MMA-Diffusion. CFS outperforms inference-time SLD-MAX in all reported settings and gives the lowest rates on I2P and Ring-a-Bell; ESD has the lowest rate on MMA-Diffusion. These results support transfer to a safety-related concept on SD 1.4.

BenchmarkSD 1.4ESDSLD-MAXCFS
I2P17.802.871.740.47
Ring-a-Bell K7785.2620.0023.164.27
Ring-a-Bell K3887.3729.4732.637.42
Ring-a-Bell K1693.6835.7942.1112.63
MMA-Diffusion95.7012.7052.1017.48

Table 2. Nudity suppression across standard and adversarial prompt sets. Detection rate (%, lower is better); SD 1.4 is unmodified.

Ablation

We ablate the main CFS hyperparameters in a single-target setting using the identity “Elon Musk” to understand how feature sparsity, retain strength, and intervention layer affect unlearning utility. Table 3 shows that larger values of kk generally provide a stronger signal, while performance remains relatively stable across values of λr\lambda_r. The layer ablation further shows that intervention effectiveness varies across layers, with middle layers performing best, consistent with prior mechanistic interpretability findings on selecting effective layers for SAE training [22]. We also analyze the computational cost of CFS in Supplementary Appendix J; the reported per-image time is close to unmodified SD 1.4, with domain-level SAE training separately accounted for.

Table 3: Ablation studies. (Left) Hyperparameter ablation reports AWUS at s=1s = 1 for each (k,λr)(k, \lambda_r) pair in a single-target setting. (Right) Layer ablation reports AWUS across intervention layers.

top-kk0.10.20.40.60.81.0Layer ℓ\ellAWUS ↑\uparrow
50.0000.0000.0000.0000.0000.02840.167
100.0000.0000.0000.0000.0000.00360.245
200.0000.0000.0000.0000.0000.36080.311
300.0010.0010.0010.0010.0010.36690.375
480.3750.3740.3750.3500.3500.331100.302
640.3470.3470.3550.3390.3270.341110.288
960.3170.3080.3230.3530.3660.303
1000.3270.3270.3260.3470.3420.305
1500.3100.3390.3050.3020.3510.335
2000.3280.3070.3170.3340.3130.293

Table 3.

Necessity of the SAE representation. We compare the same bounded, input-adaptive projection in layer-9 SAE space with a projection applied directly to the raw layer-9 hidden states, keeping prompts, target identities, seeds, and strengths fixed. The raw-space method achieves slightly higher target erasure and AWUS, but SAE-space CFS has lower CLIP collateral drift and higher peer-face similarity (Table 4). Thus, the evidence supports the SAE’s contribution to selectivity rather than claiming it always improves every scalar metric.

Table 4: SAE-space versus raw hidden-space projection. Under matched experimental settings; this separate ablation should not be numerically conflated with the eight-target averages in Table 1.

MetricSAE-space CFSRaw hidden-space
Target erasure rate ↑\uparrow0.3870.399
CLIP collateral drift ↓\downarrow0.00650.0078
AWUS ↑\uparrow0.3810.391
Peer face similarity retained ↑\uparrow0.8480.722

Table 4.

Conclusion

In this work, we study the challenge of adjacency preservation within identity unlearning. To address this problem, we introduce Counterfactual Feature Steering (CFS), an inference-time identity-suppression method requiring one-time domain-level SAE training that identifies a direction in sparse feature space capturing the target concept while avoiding features associated with retained concepts. CFS then applies per-token projection at inference time, editing sparse features along this direction to produce counterfactual representations. To quantify whether unlearning a target identity also erases information about adjacent identities, we introduce the Adjacency-Weighted Unlearning Score (AWUS) and its extensions across complementary representation spaces. Using a corpus of 4,000 prompts derived from Celeb-100, covering 100 celebrity identities across politics, sports, entertainment, and technology, we find that CFS achieves the strongest tradeoff between target erasure and preservation for selected target identities. Moreover, CFS remains stable across steering strengths, whereas subtraction-based methods collapse under stronger interventions. We further show that the same projection rule yields quantitative and qualitative nudity and artistic-style suppression results.

Despite the growing importance of identity unlearning and the need to preserve non-target identities, including adjacent identities, a key limitation of this line of research is the lack of a systematic benchmark for evaluating this setting. While we make a best-effort attempt to curate adjacent identities using public resources such as Celeb-100, and further evaluate both semantic relatedness and image-space similarity, developing a dedicated benchmark for identity unlearning with adjacency preservation remains an important direction for future work. More broadly, CFS and related concept-steering methods depend on the quality of the underlying SAE representation. A deeper analysis of SAE learning and its ability to capture concept-specific features is therefore another important direction for future research.

Target and Adjacent Identity Configuration

Table A.1 lists the 8 target identities, their Adjacent identities used for adjacency evaluation, and the 9 shared generic prompts evaluated for all targets.

TargetAdjacent identities
Barack ObamaBiden, Trump, Modi, Lincoln
Joe BidenObama, Trump, Modi, Lincoln
Elon MuskBezos, Zuckerberg, Gates, Cook
Bill GatesMusk, Bezos, Zuckerberg, Cook
Bradley CooperReynolds, Gosling, Hemsworth, Jackman
Scarlett JohanssonHathaway, Stone, Portman, Kunis
Cristiano RonaldoMessi, Neymar, Modrić, Ramos
Lionel MessiRonaldo, Neymar, Modrić, Ramos
Generic prompts (shared)businessman, man in black suit, professional headshot, conference speaker, podium speaker, SpaceX rocket launch, formal portrait, person playing football, person on city street

Table A.1. Target identities and their adjacent sets.

Proof of Proposition 1

We restate proposition 1 mentioned in Section 3 with full proof. We use the notation of Section 3: zt∈Rnz_t \in\mathbb{R}^{n} is the SAE encoding, μr∈Rn\mu_r \in\mathbb{R}^{n} the per-token retain centroid restricted to I∗I^{*}, d^∈Rn\hat{d} \in\mathbb{R}^{n} the unit concept direction restricted to I∗I^{*} (with ∥d^∥2=1\lVert\hat{d}\rVert_{2}=1), αt=max⁡((zt−μr)⋅d^,0)≥0\alpha_t = \max((z_t-\mu_r)\cdot\hat{d},0) \ge0, and ztcf=zt−sαtd^z_t^{\mathrm{cf}} = z_t-s\alpha_t\hat{d} for s∈[0,1]s \in[0,1].

Proof. Write qt=(zt−μr)⊤d^q_t=(z_t-\mu_r)^{\top}\hat{d}. Since ∥d^∥2=1\lVert\hat{d}\rVert_{2}=1 and αt=[qt]+\alpha_t=[q_t]_{+}, substitution yields

(ztcf−μr)⊤d^=qt−sαt=qt−s[qt]+.(z_t^{\mathrm{cf}}-\mu_r)^{\top}\hat{d}=q_t-s\alpha_t=q_t-s[q_t]_{+}.

If qt>0q_t>0, this equals (1−s)qt≥0(1-s)q_t\ge0; in particular, s=1s=1 brings the positive component to zero without crossing the boundary. If qt≤0q_t\le0, then αt=0\alpha_t=0 and the code is unchanged, with its nonpositive signed component preserved. These cases prove the first claim. For s=1s=1, the closed half-space Ht−={u:(u−μr)⊤d^≤0}\mathcal{H}_{t}^{-}=\{u:(u-\mu_r)^{\top}\hat{d}\le0\} has Euclidean projection ΠHt−(zt)=zt−[qt]+d^\Pi_{\mathcal{H}_{t}^{-}}(z_t)=z_t-[q_t]_{+}\hat{d}. This is the standard orthogonal projection onto a closed convex half-space, and is the unique Euclidean-distance minimizer among points in Ht−\mathcal{H}_{t}^{-}. For s∈[0,1]s\in[0,1], CFS interpolates between ztz_t and that projection.

Evaluation metrics definition

For each configuration (method, target, strength), we generate images for the target prompts, adjacent prompts, and generic prompts. Let g0pg_{0}^{p} and gspg_{s}^{p} denote images generated from prompt pp by the unmodified model and the steered model at strength ss, respectively. We denote the set of target prompts by T\mathcal{T}, adjacent prompts by A\mathcal{A}, generic prompts by G\mathcal{G}, and all non-target prompts by N=A∪G\mathcal{N}=\mathcal{A}\cup\mathcal{G}.

CLIP-space metrics. Let CLIP⁡(p,s)=cos⁡(ψ(gsp),ψt(p))\operatorname{CLIP}(p,s)=\cos(\psi(g_{s}^{p}),\psi_{t}(p)) be the cosine similarity between the CLIP image embedding and the CLIP text embedding of prompt pp. Define the normalised score drop:

δ(p,s)=max⁡(0,CLIP⁡(p,0)−CLIP⁡(p,s)∣CLIP⁡(p,0)∣+ε).\delta(p,s)=\max\left(0,\frac{\operatorname{CLIP}(p,0)-\operatorname{CLIP}(p,s)}{\lvert\operatorname{CLIP}(p,0)\rvert+\varepsilon}\right).

With this, we define the Adjacency-Weighted Unlearning Score (AWUS):

AWUS⁡(s)=Ep∈T[δ(p,s)]−Ep∈N[δ(p,s)].\operatorname{AWUS}(s) = \mathbb{E}_{p \in T}[\delta(p,s)] - \mathbb{E}_{p \in N}[\delta(p,s)].

where the first term measures target erasure and the second measures collateral damage to adjacent and generic prompts. Higher AWUS is better. This formulation rewards target removal while penalising methods that achieve high erasure by damaging the surrounding representation manifold.

Face-space metric. CLIP was trained for image-text alignment rather than face recognition and underestimates identity-specific erasure when the steered image preserves pose and clothing but changes the face. We therefore use ArcFace embeddings [5] ϕ\phi and propose FaceAWUS:

FaceAWUS⁡(s)=Ep∈T[1−cos⁡(ϕ(gps),ϕ(gp0))]−Ep∈A[1−cos⁡(ϕ(gps),ϕ(gp0))].\operatorname{FaceAWUS}(s) = \mathbb{E}_{p \in T}\left[1-\cos\left(\phi(g_p^s),\phi(g_p^0)\right)\right] - \mathbb{E}_{p \in A}\left[1-\cos\left(\phi(g_p^s),\phi(g_p^0)\right)\right].

where the first term measures how much the target face changes, and the second measures how much adjacent faces change relative to the unmodified generation. Since ArcFace is meaningful for identity-bearing face prompts, FaceAWUS is computed over adjacent identity prompts. Higher FaceAWUS is better.

Perceptual metric. Finally, we measure perceptual selectivity using LPIPS:

LPIPS-gap⁡(s)=Ep∈T[LPIPS⁡(gps,gp0)]−Ep∈N[LPIPS⁡(gps,gp0)].\operatorname{LPIPS\text{-}gap}(s) = \mathbb{E}_{p \in T}\left[\operatorname{LPIPS}(g_p^s,g_p^0)\right] - \mathbb{E}_{p \in N}\left[\operatorname{LPIPS}(g_p^s,g_p^0)\right].

This metric measures how much more the target image changes than the adjacent and generic images. Higher LPIPS-gap indicates a more selective intervention.

Together, the three net utility metrics: AWUS, FaceAWUS and LPIPS-gap give a complete picture across different representation spaces (CLIP, ArcFace, perceptual): the target concept should change substantially, while adjacent and generic prompts should remain stable. Since a method may perform well in one space but poorly in another, reporting all three metrics helps prevent over-optimizing for any single notion of preservation.

Model-internal identity adjacency

Using frozen layer-9 SD 1.4 text-encoder name-token representations for 100 identities, we obtain the selected pairwise cosine similarities in Table D.1. This is separate from the CLIP-image-space adjacency analysis below and from SAE-based target-direction computation. The manually specified peer sets (Appendix § 6) are not redefined by this post-hoc analysis. For 13 peer identities represented in the 100-identity corpus, mean reciprocal rank is 0.487. For example, Obama’s four highest-ranking

PairCosine similarity
Ronaldo–Messi0.460
Obama–Biden0.394
Musk–Zuckerberg0.401
Cooper–Gosling0.421
Johansson–Portman0.506

Table D.1. Selected model-internal identity relationships. Cosine similarity in frozen SD 1.4 layer-9 text-encoder space.

neighbors are Michelle Obama (0.574), Joe Biden (0.394), Kerry Washington (0.338), and David Cameron (0.332). These model-internal rankings complement rather than replace the semantically selected peer lists used in evaluation.

Image-space identity adjacency

To make the choice of adjacent identities not subjective and intuitive, we compute pairwise image-space similarity between the unmodified baseline generations of each identity. Higher similarity indicates that two identities occupy nearby visual regions, motivating their use as adjacent identities. It is represented here by Figure E.1.

Pairwise identity similarity heatmap

Figure E.1. Pairwise image-space identity similarity. We compute cosine similarity between CLIP image embeddings of unmodified baseline generations for each identity. Higher values indicate visually closer identity prompts and provide an objective basis for selecting adjacent identities, e.g., Obama-Biden, Musk-Gates, Messi-Ronaldo, and Cooper-Johansson.

Figure F.1. Face-space trade-off. CFS achieves high target-identity change with low adjacent-identity change, while several baselines incur larger adjacent-face drift.

Additional trade-off analysis

Figure F.1 visualizes the face-space trade-off underlying FaceAWUS. A method that lies toward the top-left is ideal: large target-identity change with small adjacent-identity drift.

Compositional and semantic identity erasure

Figure G.1. Compositional and semantic identity erasure. Top: CFS simultaneously erases Elon Musk and Scarlett Johansson in a single generation. Bottom: CFS erases Elon Musk from a circumlocution prompt referring to him as “the CEO of Tesla”.

Figure H.1. Per-token alignment signal for Elon Musk. The alignment signal concentrates on identity-bearing name tokens and remains low on generic style or composition tokens.

Compositional and circumlocution examples

Figure G.1 provides additional qualitative examples showing that CFS can be composed across multiple target identities and can also respond to indirect semantic references rather than only explicit name tokens.

Token-level alignment

Figure H.1 shows where the CFS alignment signal appears within a prompt. The signal concentrates on the identity-bearing name tokens rather than generic style or composition tokens, supporting the claim that CFS targets identity-specific features.

Per-target Results

Figure I.1 shows per-target net utility across all three metric spaces. CFS achieves balanced performance across all 8 identities on AWUS, dominates on FaceAWUS, and encloses every other method on LPIPS-gap. Methods whose polygons collapse toward the centre on FaceAWUS (SAeUron, One-Step) performed such aggressive erasure that ArcFace could not detect a face in the steered output.

Runtime Comparison

Table J.1 reports generation-time overhead. Since CFS modifies the text-encoder representation once per prompt, rather than at every diffusion step, so its overhead is independent of the number of denoising steps. CFS achieves the best runtime among the evaluated unlearning methods, while remaining essentially as fast as the unmodified SD 1.4 baseline.

Figure I.1. Per-target net utility across three metric spaces. Each panel is a radar over the 8 celebrity identities, with one polygon per method. Larger area = better. CFS (red) achieves a balanced shape across identities on AWUS, dominates the perimeter on FaceAWUS, and visibly encloses every other method on every identity for LPIPS-gap. Missing ArcFace face detections are floored at the inner radius (negative extent); a polygon collapsing toward the centre on FaceAWUS indicates a method whose erasure was so aggressive that ArcFace could not detect a face in the steered image (e.g. SAEuron and OneStep on most targets). The dashed circle at 0 in the FaceAWUS and LPIPS-gap panels marks the zero-utility threshold: methods with polygons inside this circle on a given identity are doing more harm than good.

Methods/image ↓\downarrow
SD 1.4 (no erasure)9.52
CFS (ours)9.50
Concept Steerers9.96
SAeUron15.75
One-Step10.42
UCE18.75
SLD25.44
ESD (incl. fine-tune)21.23

Table J.1. Runtime comparison. Seconds per image on one A30 GPU using 50 DDIM steps, averaged over 20 runs after 3 warm-up runs. CFS modifies the text-encoder representation once per generation, so its cost is independent of the number of diffusion steps.

Fixed-full-strength CLIP-space comparison

The main comparison allows each strength-controlled baseline its reported AWUS operating point while CFS uses s=1s = 1. For transparency, Table K.1 reproduces the original separate supplementary table of CLIP-space scores at each method’s full-strength setting. Strength parameters are method-specific and not numerically calibrated across approaches. Face-space metrics are not available at this matched nominal operating-point comparison for every method.

MethodTER ↑\uparrowCDR ↓\downarrowAWUS ↑\uparrow
CFS (ours)0.3060.0110.296
Concept Steerers0.4790.4470.032
SAeUron0.4500.2550.195
UCE0.2350.0370.198
ESD0.3940.1320.262
SLD0.0100.025−0.015
One-Step0.4380.3370.101

Table K.1. Full-strength comparison in CLIP space. Scores averaged across eight celebrity targets; AWUS = TER − CDR.

Reproducibility and Implementation Details

Model, prompts, and concept directions. Our primary experiments use Stable Diffusion 1.4 with an intervention at layer 9 of its frozen CLIP text encoder. We construct 4,000 prompts derived from Celeb-100, spanning 100 identities, and evaluate eight target identities; for each target, the forget set contains prompts naming that target and the retain set contains the remaining prompts. We evaluate four designated adjacent identities per target and nine shared generic prompts. A domain-level kk-sparse autoencoder (SAE) is trained once on the corresponding text-encoder activations and reused for all eight identity targets. Per-target directions and retain centroids are computed from cached, no-gradient forget/retain statistics; the diffusion-model parameters remain unchanged. Nudity and artistic-style experiments use separately trained domain-level SAEs, rather than assuming that the identity SAE transfers unchanged. The feature-selection and projection equations appear in Section 3.

Controlled evaluations. For the main identity comparison, methods are evaluated with identical target sets, prompts, random seeds, inference steps, and guidance settings. CFS uses its full intervention strength s=1s = 1; other strength-controlled baselines use their best evaluated AWUS operating points, and weight-editing baselines use their target-specific edited checkpoints (Section 4.1). The additional face-specific evaluation uses ArcFace on detected faces, with identity references obtained from unsteered Stable Diffusion 1.4 generations rather than real celebrity photographs. The general-preservation evaluation uses 10,000 COCO-30K captions with matching seeds and generation settings; FID for the baseline and CFS is computed against the same corresponding COCO reference images. The safety transfer study reports nudity detection rates on I2P, Ring-a-Bell (K77, K38, K16), and MMA-Diffusion. The adjacency rankings use frozen layer-9 name-token representations (found this layer to be the most useful based on ablation), rather than the trained SAE or the generated-image evaluation metrics.

Computational accounting. The identity SAE requires approximately two hours of one-time training in the reported setup; each target’s cached statistics occupy approximately 925 KB. The controlled runtime comparison uses one NVIDIA A30 GPU, 50 DDIM steps, 20 measured runs following three warm-up runs, and reports 9.50 s/image for CFS versus 9.52 s/image for unmodified Stable Diffusion 1.4 (Appendix § 15).

References

  1. [1]Ibtihiel Amara, Ahmed Imtiaz Humayun, Ivana Kajic, Zarana Parekh, Natalie Harris, Sarah Young, Chirag Nagpal, Najoung Kim, Junfeng He, Cristina Nader Vasconcelos, et al. Erasebench: Understanding the ripple effects of concept erasure techniques. arXiv preprint arXiv:2501.09833, 1(3):6, 2025.
  2. [2]Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2(5):6, 2023.
  3. [3]Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE, 2015.DOI
  4. [4]Bartosz Cywiński and Kamil Deja. Sauron: Interpretable concept unlearning in diffusion models with sparse autoencoders. arXiv preprint arXiv:2501.18052, 2025.arxiv.org/abs/2501.18052
  5. [5]Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019.DOI
  6. [6]Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023.arxiv.org/abs/2310.02238
  7. [7]Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2426–2436, 2023.arxiv.org/abs/2303.07345
  8. [8]Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 5111–5120, 2024.arxiv.org/abs/2308.14761
  9. [9]Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1907.05012
  10. [10]Robert Huben, Hoagy Cunningham, Logan Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations, volume 2024, pages 7827–7845, 2024.arxiv.org/abs/2309.08600
  11. [11]Ronen Kamenetsky, Sara Dorfman, Daniel Garibi, Roni Paiss, Or Patashnik, and Daniel Cohen-Or. Saedit: Token-level control for continuous image editing via sparse autoencoder. arXiv preprint arXiv:2510.05081, 2025.arxiv.org/abs/2510.05081
  12. [12]Dahye Kim and Deepti Ghadivaram. Concept steerers: Leveraging k-sparse autoencoders for controllable generations. arXiv e-prints, pages arXiv–2501, 2025.arxiv.org/abs/2501.19066
  13. [13]Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6430–6440, 2024.arxiv.org/abs/2403.06135
  14. [14]Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv:1312.5663, 2013.arxiv.org/abs/1312.5663
  15. [15]Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata. Sparse autoencoders learn monosemantic features in vision-language models. arXiv preprint arXiv:2504.02821, 2025.arxiv.org/abs/2504.02821
  16. [16]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
  17. [17]Protection Regulation. Regulation (eu) 2016/679 of the european parliament and of the council. Regulation (eu), 679(2016):10–13, 2016.
  18. [18]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.arxiv.org/abs/2112.10752
  19. [19]Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22522–22531, 2023.arxiv.org/abs/2211.05105
  20. [20]Fnu Shivam, Nima Najafzadeh, Yenumula Reddy, and Prashnna Gyawali. Cure: Centroid-guided unsupervised representation erasure for facial recognition systems. arXiv preprint arXiv:2509.19562, 2025.arxiv.org/abs/2509.19562
  21. [21]Viacheslav Surkov, Chris Wendler, Antonio Mari, Mikhail Terekhov, Justin Deschenaux, Robert West, Caglar Gulcehre, and David Bau. One-step is enough: Sparse autoencoders for text-to-image diffusion models. arXiv preprint arXiv:2410.22366, 2024.arxiv.org/abs/2410.22366
  22. [22]Adly Templeton. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Anthropic, 2024.
  23. [23]Kartik Thakral, Tamar Glaser, Tal Hassner, Mayank Vatsa, and Richa Singh. Fine-grained erasure in text-to-image diffusion-based foundation models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9121–9130. IEEE, 2025.
  24. [24]Zhihua Tian, Sirun Nan, Ming Xu, Shengfang Zhai, Wenjie Qu, Jian Liu, Ruoxi Jia, and Jiaheng Zhang. Sparse autoencoder as a zero-shot classifier for concept erasing in text-to-image diffusion models. arXiv preprint arXiv:2503.09446, 2025.arxiv.org/abs/2503.09446
  25. [25]Berk Tinaz, Zalan Fabian, and Mahdi Soltanolkotabi. Emergence and evolution of interpretable concepts in diffusion models. 2025.arxiv.org/abs/2504.15473
  26. [26]Xu Wang, Zihao Li, Benyou Wang, Yan Hu, and Difan Zou. Model unlearning via sparse autoencoder subspace guided projections. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26541–26557, 2025.arxiv.org/abs/2505.24428
  27. [27]Yuanshun Yao and Xiaojun Xu. Large language model unlearning. Advances in Neural Information Processing Systems, 37:105425–105475, 2024.arxiv.org/abs/2310.10683
  28. [28]Yihua Zhang, Chongyu Fan, Yimeng Zhang, Yuguang Yao, Jinghan Jia, Jiancheng Liu, Gaoyuan Zhang, Gaowen Liu, Ramana Rao Kompella, Xiaoming Liu, et al. Unlearncanvas: Stylized image dataset for enhanced machine unlearning evaluation in diffusion models. arXiv preprint arXiv:2402.11846, 2024.arxiv.org/abs/2402.11846

Paper details

vlnc:W_22e3j8yr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents