betaFeedback

WorksComputer Science

Sample-Grained Approximate Unlearning with Provable Per-Sample Bounds

Abstract

Machine unlearning aims to erase the influence of a designated forget set from a trained model while preserving strong utility on the retain set. Approximate unlearning methods efficiently achieve this goal without full retraining, but their effectiveness is primarily assessed using dataset-level aggregate metrics. These aggregate metrics can mask deviations from retraining at the level of individual samples. In this work, we focus on per-sample unlearning effectiveness. We introduce SALA (Sample-grained Likelihoods ALignment), a plug-in optimization framework that enables approximate unlearning methods to operate at the level of individual samples by steering each example toward its correct target distribution. To quantitatively assess this behavior, we further propose the sample-grained distribution gap, a metric that captures per-sample discrepancies between approximate unlearning and exact retraining output distributions. We provide theoretical guarantees showing that SALA contracts this gap and yields a bounded per-sample discrepancy to retraining. To make this framework practical, we develop a low-cost estimation method for the member and non-member target distributions. Extensive experiments on classification and generation tasks demonstrate that SALA reduces the sample-grained distribution gap by up to 86.67%86.67\% when paired with existing approximate unlearning baselines, while also improving robustness to unlearning hyperparameters.

Introduction

Motivated by the legal data-deletion requirement known as the “right to be forgotten” [11], machine unlearning is introduced to remove the influence of a designated forget set from a trained model [2, 5, 14] while preserving utility on the retain set. The most direct way to achieve this goal is to retrain the model from scratch using only the retain set. This exact retraining procedure is the gold standard for unlearning. However, because exact retraining is often too expensive for deployment, recent work has increasingly shifted to approximate unlearning (AU) [15, 21], which updates a pretrained model with small compute overhead to closely match the result of exact retraining. Accordingly, the efficacy of AU is commonly evaluated by gap-to-retrain [4], which measures how close the AU model distribution is to the exact-retraining (ER) distribution under a chosen observable metric.

However, existing works usually rely on dataset-level average observables, such as prediction accuracy over a dataset, which can mask unlearning failures at the sample level. To capture per-sample unlearning performance, we measure distribution non-overlap (DNO2), defined as the discrepancy between the output distributions estimated over multiple runs of unlearning a given sample xx and exact retraining without xx, as illustrated in Figure 1. Figure 2a shows that state-of-the-art AU methods can achieve a small dataset-level gap-to-retrain, measured by 1−ToW1-\mathrm{ToW}3, while still exhibiting a large per-sample gap-to-retrain, i.e., DNO. Figure 2b further shows that this per-sample gap is associated with greater privacy leakage on forget samples, as evaluated here by the representative attack RULI [30].

Figure 1. Output log-odds distributions for a selected forgotten sample. We compare retraining with AU (i.e., SalUn [12]) with and without SALA under the score in Eq. (5). SALA moves the distribution closer to the retraining reference.

Gap-to-retrain evaluations for AU

Figure 2. Gap-to-retrain evaluations for AU. ToW measures comprehensive dataset-average gap (see (1)). Triangles denote AU results assisted by SALA.

These findings highlight a key research gap: sample-grained approximate unlearning remains insufficiently studied and far from well achieved. We address this gap with SALA (Sample-grained Likelihoods Alignment), a plug-in framework that equips AU with explicit per-sample targets. SALA assigns each sample a role-specific target distribution: retain samples align to Qin(⋅∣x)Q_{\mathrm{in}}(\cdot\mid x), which represents member behavior, while forget samples align to Qout(⋅∣x)Q_{\mathrm{out}}(\cdot\mid x), which represents non-member behavior. For each sample xx, SALA minimizes the sample’s zz-score to its assigned target distribution, driving the output toward the prescribed distribution.

Beyond empirically reducing the per-sample gap-to-retrain of AU methods, SALA provides a provable upper bound on this gap measured by DNO. Moreover, it reduces the strong hyperparameter sensitivity of existing AU methods, which are often tuned heuristically and can degrade sharply under poor configurations. SALA mitigates this issue through its zz-score, which serves as an explicit per-sample alignment signal. From this signal, we define InDist, the fraction of samples whose outputs lie within a preset zz-band around their assigned targets. InDist supports two complementary automated mechanisms: hyperparameter selection and a stop criterion. For the former, SALA runs a fast probe with only a few unlearning steps and chooses the configuration that maximizes InDist. For the latter, SALA stops once InDist exceeds a high threshold, such as 0.95. This improves AU robustness to hyperparameters and avoids unnecessary updates. Finally, since InDist imposes an explicit distributional constraint, it also yields a theoretical upper bound on SALA’s DNO.

However, constraining InDist in SALA requires access to the full original training set, which may be unrealistic in some settings. We therefore introduce a forget-target-only variant that aligns only forget samples to Qout(⋅∣x)Q_{\mathrm{out}}(\cdot\mid x), for AU methods with limited access to the retain set. When the underlying AU method satisfies the utility preservation condition in Eq. (16), this variant still yields an upper bound on DNO under a single InDist constraint on forget samples.

The main bottleneck of SALA is estimating Qin(⋅∣x)Q_{\mathrm{in}}(\cdot\mid x) and Qout(⋅∣x)Q_{\mathrm{out}}(\cdot\mid x) for each training sample xx. These targets can be estimated once during model development and reused across unlearning requests. One direct solution uses shadow-model estimation [13], which trains models under different forget-retain splits and aggregates their outputs. This pipeline resembles the memorization-estimation step used by AU methods such as SUM [40]. If compatible member and non-member score statistics already exist, SALA can reuse them for target estimation. However, shadow-model estimation is expensive and costs tens of standard training runs. To avoid this overhead, we propose an efficient estimator that obtains both targets and the model in a single training run. It uses a split–merge training scheme. In the middle stage, two models train on complementary subsets, so each sample is in-training for one model and out-of-training for the other. In the late stage, Federated Averaging merges the two models [23]. Finally, checkpoint ensembles collected during training then estimate Qin(⋅∣x)Q_{\mathrm{in}}(\cdot\mid x) and Qout(⋅∣x)Q_{\mathrm{out}}(\cdot\mid x).

In summary, this work makes the following contributions:

  • We propose a sample-grained approximate unlearning (AU) framework. It monitors and guides alignment to retraining at the individual-sample level.

  • SALA provides a theoretical upper bound on the sample-grained gap-to-retrain metric (i.e., DNO), yielding guarantees for AU at the sample level.

  • We introduce SMSE, which estimates per-sample member and non-member distributions with split–merge training and segment ensembles. SMSE reduces preparation from multiple shadow-model trainings to a single model training run.

  • We evaluate SALA on classification and generative tasks with diverse AU methods. Figure 2 shows that SALA reduces DNO by up to 86.67%, keeps a small dataset-level gap to retraining (i.e., 1-ToW), and remains robust without manual hyperparameter tuning.

Problem Definition

Machine Unlearning Notations

Machine unlearning studies how to update a trained model so that the influence of a designated subset of its training data is removed. Let fθf_\theta denote a model with parameters θ\theta, trained on a dataset D∼DD \sim\mathcal{D}. Upon a deletion request supported by the right to be forgotten (Article 17 of the GDPR [11]), a forget set Df⊆DD_f \subseteq D is specified and the remaining retain set is denoted as Dr=D∖DfD_r = D \setminus D_f. Let A:D→R\mathcal{A}: \mathcal{D} \to\mathcal{R} be a learning algorithm that maps a dataset to a model in the output space R\mathcal{R} (e.g., neural network parameters), and let U\mathcal{U} be an approximate unlearning (AU) algorithm. The retrained model is defined as fθ∗f_{\theta^*}, with θ∗=A(Dr)\theta^* = \mathcal{A}(D_r). The unlearned model is fθu=U(A(D),Df,D)f_{\theta_u} = \mathcal{U}(\mathcal{A}(D), D_f, D). AU aims to make θu\theta_u behave similarly to θ∗\theta^* while being substantially more efficient than rerunning A\mathcal{A} from scratch on DrD_r.

Evaluation for Approximate Unlearning (AU)

We evaluate approximate unlearning (AU) along four dimensions [4, 27]: (i) efficacy, i.e., how close the unlearned model is to the exact-retraining result and how much utility it keeps; (ii) privacy, i.e., whether deleted samples can be detected by inference attacks; and (iii) efficiency, i.e., the compute cost of unlearning, commonly summarized by run-time. In addition, we propose (iv) robustness, which captures AU sensitivity to hyperparameters and run-to-run variance.

Efficacy Metrics for AU

Efficacy is assessed at two levels: dataset-level and sample-grained. Most existing metrics are dataset-level and quantify the gap-to-retrain under a chosen observable. For an AU unlearned model fθuf_{\theta_u}, smaller gaps to the retrained model fθ∗f_{\theta^*} indicate higher unlearning efficacy.

Dataset-level metrics. This gap is measured between fθuf_{\theta_u} and fθ∗f_{\theta^*} on three datasets: test DtD_t, retain DrD_r, and forget DfD_f. For classification, the metrics are accuracy: test accuracy (TA), retain accuracy (RA), and forget accuracy (FA). For text generative tasks, the metrics are text perplexity (PPL): T-PPL, R-PPL, and F-PPL. These gaps are aggregated by the Tug-of-War (ToW) score [40]:

ToW⁡(M1,M2,M3)=∏i=13(1−d(Mi(θu),Mi(θ∗))Mi(θ∗)),(1)\operatorname{ToW}(M_1,M_2,M_3)=\prod_{i=1}^{3}\left(1-\frac{d(M_i(\theta_u),M_i(\theta^*))}{M_i(\theta^*)}\right), \tag*{(1)}

where Mi(θ)M_i(\theta) is a metric value of model fθf_\theta (e.g., TA/RA/FA or T-PPL/R-PPL/F-PPL), and d(⋅,⋅)d(\cdot,\cdot) is the absolute difference between two metric values.

Sample-grained metrics reveal finer gaps. Sample-grained metrics compare AU with exact retraining at the level of individual samples. For a fixed sample xx, they measure the discrepancy between the distribution of an observable score ss induced by the AU model output and the corresponding distribution under exact retraining. This paper introduces Distribution Non-Overlap (DNO) to quantify individual-sample distributional deviations of AU from retraining under both the retain and forget roles. Let Pr(s∣x)P_r(s \mid x) and Pf(s∣x)P_f(s \mid x) denote the estimated score distributions produced by AU when xx is treated as a retain sample (member) and a forget sample (non-member), respectively. Pr(s∣x)P_r(s \mid x) and Pf(s∣x)P_f(s \mid x) are estimated over NN independent runs in which xx repeatedly takes the corresponding role. Let Qin(s∣x)Q_{\mathrm{in}}(s \mid x) and Qout(s∣x)Q_{\mathrm{out}}(s \mid x) be their exact-retraining counterparts. For reporting, DNO summarizes these individual-sample discrepancies as

DNO=Ex12[(1−∫min⁡(Pr(s∣x),Qin(s∣x)) ds)+(1−∫min⁡(Pf(s∣x),Qout(s∣x)) ds)](2)\mathrm{DNO} = \mathbb{E}_{x}\frac{1}{2}\left[\left(1-\int\min(P_r(s \mid x),Q_{\mathrm{in}}(s \mid x))\,\mathrm{d}s\right)+\left(1-\int\min(P_f(s \mid x),Q_{\mathrm{out}}(s \mid x))\,\mathrm{d}s\right)\right] \tag*{(2)}

Smaller DNO indicates smaller sample-level distributional discrepancies from retraining on average.

Privacy Metrics for AU

This paper considers two AU privacy evaluation cases that differ in the attacker’s knowledge of the unlearning procedure: a legacy attacker that ignores unlearning, and an unlearning-aware attacker that knows the unlearning algorithm [18]. Both cases are evaluated via membership inference attack (MIA) [34], which takes forget samples as positives and held-out test samples as negatives. The legacy setting is reported as MIA efficacy [21]. The unlearning-aware setting is instantiated with RULI [30], a recent unlearning-specific MIA evaluation framework. For both settings, we use the best MIA accuracy as the primary metric. In both cases, the attacker estimates score densities from shadow models and applies the same likelihood-ratio rule; the hypotheses (H1,H0)(H_1,H_0) change with attacker knowledge. Specifically, the legacy case compares the member state (x∈D)(x \in D) against the non-member state (x∉D)(x \notin D), as in LiRA [6], while the unlearning-aware case compares the trained-then-unlearned state (x∈Df)(x \in D_f) against the never-trained state (x∉D)(x \notin D), using scores computed on the unlearned model, as in RULI [30]. Concretely,

Λ(x)=p(s(x)∣H1)p(s(x)∣H0), predict member if Λ(x)>τMI.(3)\Lambda(x)=\frac{p(s(x)\mid H_1)}{p(s(x)\mid H_0)},\ \text{predict member if }\Lambda(x)>\tau_{\mathrm{MI}}. \tag*{(3)}

For the legacy case, (H1,H0)=(x∈D, x∉D)(H_1,H_0)=(x \in D,\ x \notin D); for the unlearning-aware case, (H1,H0)=(x∈Df, x∉D)(H_1,H_0)=(x \in D_f,\ x \notin D) with score s(x)s(x) computed on the unlearned model.

Robustness Criteria

Robustness measures sensitivity to hyperparameter choices. Let K≜{arg⁡i}i=1n\mathcal{K}\triangleq\{\operatorname{arg}_i\}_{i=1}^{n} denote the set of evaluated hyperparameter settings. For an efficacy metric MM (e.g., ToW or DNO), robustness is measured by the worst–best range (i.e., fluctuation) over K\mathcal{K}:

Range⁡(M)=max⁡arg⁡∈KMarg⁡−min⁡arg⁡∈KMarg⁡.(4)\operatorname{Range}(M)=\max_{\operatorname{arg}\in\mathcal{K}}M_{\operatorname{arg}}-\min_{\operatorname{arg}\in\mathcal{K}}M_{\operatorname{arg}}. \tag*{(4)}

Smaller Range(M)(M) indicates higher robustness.

Sample-Grained Unlearning Framework

This section introduces sample-grained likelihood alignment (SALA), an auxiliary framework that improves approximate unlearning (AU) by explicitly aligning the predicted likelihood of each sample with its target distribution, as shown in Figure 3. For a forget sample x∈Dfx \in D_f, SALA aligns the prediction with the non-member distribution Qout(⋅∣x)Q_{\mathrm{out}}(\cdot\mid x). For a retain sample x∈Drx \in D_r, SALA aligns the prediction with the member distribution Qin(⋅∣x)Q_{\mathrm{in}}(\cdot\mid x). SALA measures this alignment with a per-sample zz-score, which both defines the guidance loss and serves as a sample-level stopping criterion (§ 3.2). When a sufficiently large fraction of samples satisfy this criterion, SALA bounds the per-sample gap to retraining, as measured by DNO (§ 3.3). We also provide a related SALA variant that constrains only forget samples and admits a DNO upper bound under additional conditions on the base unlearning algorithm (§ 3.4).

Figure 3. Overview of SALA. SALA augments a base unlearning UU with sample-wise distribution-alignment guidance losses, fast-probe hyperparameter selection, and sample-level stop criterion.

SALA Framework. Figure 3 shows the four-step SALA workflow. First, SALA prepares per-sample reference statistics for Qin(⋅∣x)Q_{\mathrm{in}}(\cdot\mid x) and Qout(⋅∣x)Q_{\mathrm{out}}(\cdot\mid x). It also identifies a hard-forget subset Dhf⊆DfD_{\mathrm{hf}}\subseteq D_f whose member and non-member distributions are more separable and thus harder to erase. Second, a fast probe screens candidate hyperparameters K≜{arg⁡i}i=1n\mathcal{K}\triangleq\{\operatorname{arg}_i\}_{i=1}^{n} with a few unlearning steps. The probe selects the candidate that best satisfies InDist (Eq. (8)), which measures the fraction of samples whose outputs fall within the target zz-band. Third, SALA uses the selected arg in the full unlearning stage and augments the base update rule U\mathcal{U} with a per-sample guidance loss. Finally, SALA monitors full-data InDist online and returns θu\theta_u once InDist exceeds a preset threshold. Appendix § 9 provides the pseudo-code.

Definitions

Score sθ(x)s_\theta(x). SALA uses an output score sθ(x)s_\theta(x) for distribution alignment, abbreviated as s(x)s(x) or ss when clear from context. For classification with logits gθ(x)∈RCg_\theta(x) \in\mathbb{R}^{C} and label yy, we define

sθ(x)≜log⁡pθ(y∣x)1−pθ(y∣x)(5)s_\theta(x) \triangleq\log\frac{p_\theta(y \mid x)}{1-p_\theta(y \mid x)} \tag*{(5)}

where pθ(y∣x)=softmax⁡(gθ(x))yp_\theta(y \mid x) = \operatorname{softmax}(g_\theta(x))_y is the predicted probability of the true class. The log-odds score is unbounded and retains distinctions when the probability is close to 0 or 1. Prior empirical evidence [6] shows that the score distribution for a fixed sample across runs is approximately Gaussian, motivating Gaussian modeling of the per-sample reference scores. For generation, we define the score as the per-token cross-entropy loss. We further validate this Gaussian approximation experimentally in Appendix § 14.1.

Per-sample reference distributions. Following prior empirical findings [6], we define the member and non-member target output distributions for each sample xx as

Qin(⋅∣x)≈N(μin(x),σin2(x)),Qout(⋅∣x)≈N(μout(x),σout2(x)),(6)Q_{\mathrm{in}}(\cdot\mid x) \approx\mathcal{N}\left(\mu_{\mathrm{in}}(x),\sigma_{\mathrm{in}}^{2}(x)\right), \quad Q_{\mathrm{out}}(\cdot\mid x) \approx\mathcal{N}\left(\mu_{\mathrm{out}}(x),\sigma_{\mathrm{out}}^{2}(x)\right), \tag*{(6)}

zz-score with shifted mean for distribution match. Let μQ(x)\mu_Q(x) and σQ(x)\sigma_Q(x) denote the mean and standard deviation of the target distribution for xx, where Q=QinQ = Q_{\mathrm{in}} for retain samples and Q=QoutQ = Q_{\mathrm{out}} for forget samples. To avoid the degenerate solution where sθ(x)s_\theta(x) matches μQ(x)\mu_Q(x) and the effective variance collapses, for each unlearning run, we compute the zz-score with a shifted mean:

zQr(x;θ)≜sθ(x)−μQr(x)σQ(x),μQr(x)≜μQ(x)+ϵ(x)σQ(x),(7)z_Q^r(x;\theta) \triangleq\frac{s_\theta(x)-\mu_Q^r(x)}{\sigma_Q(x)}, \qquad\mu_Q^r(x) \triangleq\mu_Q(x)+\epsilon(x)\sigma_Q(x), \tag*{(7)}

where ϵ(x)∼N(0,var⁡ϵ)\epsilon(x) \sim\mathcal{N}(0,\operatorname{var}_{\epsilon}) is a fixed random offset sampled once per xx for each run. Small ∣zQr(x;θ)∣\lvert z_Q^r(x;\theta)\rvert means the current output sθ(x)s_\theta(x) lies in a high-density region of the (regularized) target.

InDist metric. To explicitly monitor distribution alignment at the sample level, we define InDist as the fraction of samples whose outputs lie within a fixed zz-band:

InDist⁡(S;θ)≡Ex∼S[1(∣zQr(x;θ)∣<zcut)],(8)\operatorname{InDist}(S;\theta) \equiv\mathbb{E}_{x\sim S}\left[\mathbf{1}\left(\lvert z_Q^r(x;\theta)\rvert< z_{\mathrm{cut}}\right)\right], \tag*{(8)}

where S∈{Dr,Df,Dhf}S \in\{D_r, D_f, D_{hf}\} and zcutz_{\mathrm{cut}} is a fixed cutoff. InDist takes values in [0,1][0,1], and a larger value indicates that more samples fall within their target distributions.

SALA Hyper-parameter Fast Probe, Unlearning Guidance, and Stop Criterion

Hyper-parameter fast probe. Each candidate arg ∈K\in\mathcal{K} is evaluated by running only a few unlearning steps from the same pre-unlearning state and scoring coverage, e.g., InDist⁡(Dr;θ)+InDist⁡(Df;θ)+InDist⁡(Dhf;θ)\operatorname{InDist}(D_r;\theta)+\operatorname{InDist}(D_f;\theta)+\operatorname{InDist}(D_{hf};\theta). The best candidate with the highest score is then used for the unlearning, while all probe updates are discarded.

SALA guidance in unlearning. SALA augments the unlearning objective with the following sample-wise distribution-alignment loss:

LSALA(θ)=Ex[ω(x)zQr(x;θ)2].(9)\mathcal{L}_{\mathrm{SALA}}(\theta)=\mathbb{E}_{x}\left[\omega(x)z_Q^r(x;\theta)^2\right]. \tag*{(9)}

The zQr(x;θ)2z_Q^r(x;\theta)^2 term pulls outputs toward the target shifted mean. Sample weights are

ω(x)=λr1[x∈Dr]+λf(1+α1[x∈Dhf])1[x∈Df],(10)\omega(x)=\lambda_r\mathbf{1}[x\in D_r]+\lambda_f(1+\alpha\mathbf{1}[x\in D_{hf}])\mathbf{1}[x\in D_f], \tag*{(10)}

where λr,λf>0\lambda_r,\lambda_f>0 balance retain and forget guidance, and α≥0\alpha\geq0 emphasizes on hard forget samples.

InDist-guided stop criterion. Unlearning stops once coverage exceeds a target threshold τ\tau:

InDist⁡(S;θ)≥τ,∀S∈{Dr,Df,Dhf}(11)\operatorname{InDist}(S;\theta)\geq\tau,\forall S\in\{D_r,D_f,D_{hf}\} \tag*{(11)}

This makes the stopping decision explicit at the sample level and avoids ad-hoc step budgets.

Theoretical Guarantee for SALA’s DNO Upper Bound

This section derives an upper bound on DNO in (2) from the InDist constraint imposed by the SALA stop criterion. We first bound the overlap gap for each sample and then average over the set.

Setup. Fix a sample xx with target distribution QQ. Let the SALA-unlearned model distribution induce

X∼QSALA(⋅∣x)=N(μSALA(x),(σSALA(x))2).X \sim Q^{\mathrm{SALA}}(\cdot\mid x) = \mathcal{N}\left(\mu^{\mathrm{SALA}}(x),\left(\sigma^{\mathrm{SALA}}(x)\right)^2\right).

Define Δ(x)≜μSALA(x)−μQ(x)σQ(x)\Delta(x) \triangleq\frac{\mu^{\mathrm{SALA}}(x)-\mu_Q(x)}{\sigma_Q(x)}, ρ(x)≜σSALA(x)σQ(x)\rho(x) \triangleq\frac{\sigma^{\mathrm{SALA}}(x)}{\sigma_Q(x)}, η(x)≜X−μQ(x)σQ(x)\eta(x) \triangleq\frac{X-\mu_Q(x)}{\sigma_Q(x)}.

Theorem 3.1 (DNO upper bound from SALA). Assume that SALA makes the InDist constraint hold for xx:

PX∼QSALA(⋅∣x)(∣X−μQr(x)σQ(x)∣<Zcut)≥τ.(12)\mathbb{P}_{X\sim Q^{\mathrm{SALA}}(\cdot\mid x)}\left(\left|\frac{X-\mu_Q^r(x)}{\sigma_Q(x)}\right|<Z_{\mathrm{cut}}\right)\ge\tau. \tag*{(12)}

The mean shift in (7) induces variance expansion, which gives a non-degenerate lower bound on the effective scale:

ρmin⁡≜max⁡(0,var⁡ϵ−Cov⁡(ϵ(x),η(x))).\rho_{\min} \triangleq\max\left(0,\operatorname{var}_{\epsilon}-\operatorname{Cov}(\epsilon(x),\eta(x))\right).

If ρ(x)∈[ρmin⁡,Zcutq]\rho(x)\in\left[\rho_{\min},\frac{Z_{\mathrm{cut}}}{q}\right] with ρmin⁡>0\rho_{\min}>0, then the per-sample overlap gap satisfies

1−∫min⁡(QSALA(s∣x),Q(s∣x)) ds≤2(1−BCmin⁡).(13)1-\int\min\left(Q^{\mathrm{SALA}}(s\mid x),Q(s\mid x)\right)\,\mathrm{d}s\le\sqrt{2(1-\mathrm{BC}_{\min})}. \tag*{(13)}

where BCmin⁡\mathrm{BC}_{\min} is the worst-case Bhattacharyya coefficient [22] over the feasible (Δ,ρ)(\Delta,\rho) region:

BCmin⁡=min⁡ρ∈[ρmin⁡,Zcutq][2ρ1+ρ2exp⁡(−(Zcut−ρq)24(1+ρ2))],q≜Φ−1(1+τ2).(14)\mathrm{BC}_{\min}=\min_{\rho\in\left[\rho_{\min},\frac{Z_{\mathrm{cut}}}{q}\right]}\left[\sqrt{\frac{2\rho}{1+\rho^2}}\exp\left(-\frac{(Z_{\mathrm{cut}}-\rho q)^2}{4(1+\rho^2)}\right)\right],\qquad q\triangleq\Phi^{-1}\left(\frac{1+\tau}{2}\right). \tag*{(14)}

Consequently, the sample-averaged DNO in (2) is bounded by the same right-hand side as (13):

DNO≤2(1−BCmin⁡).(15)\mathrm{DNO}\le\sqrt{2(1-\mathrm{BC}_{\min})}. \tag*{(15)}

The proof of Theorem 3.1 is provided in Appendix § 10.

Instantiation. For a sanity check, a simplifying assumption Cov⁡(ϵ,η)=0\operatorname{Cov}(\epsilon,\eta)=0 is adopted. Under an easy-to-achieve setting (τ=0.95, Zcut=2, ϵ=clip⁡(N(0,1),±1.8))(\tau=0.95,\ Z_{\mathrm{cut}}=2,\ \epsilon=\operatorname{clip}(\mathcal{N}(0,1),\pm1.8)), and assuming exact access to Qin(⋅∣x)Q_{\mathrm{in}}(\cdot\mid x) and Qout(⋅∣x)Q_{\mathrm{out}}(\cdot\mid x), Theorem 3.1 yields a closed-form DNO upper bound of about 0.10. This bound matches the empirical DNO of SALA, about 0.11, in § 5.3.

SALA with Access Only to the Forget Set and Its DNO Guarantees

To accommodate unlearning settings with no access to retain samples, we further relax SALA by removing its dependence on the retain set in all stages where it is used, including the fast probe, the unlearning guidance, and the stop criterion. In other words, in this variant, SALA no longer explicitly optimizes any term involving x∈Drx\in D_r.

Even under this one-sided constraint, SALA still admits DNO guarantees as long as the base unlearning algorithm U\mathcal{U} provides a condition on DrD_r that can equivalently replace InDist⁡(Dr;θ)\operatorname{InDist}(D_r;\theta) criterion. Corollary 3.2 summarizes this claim, with the proof given in Appendix § 11.

Corollary 3.2 (DNO upper bound for SALA with forget-set access only). Assume that the base unlearning algorithm U\mathcal{U} satisfies the following retain-consistency condition:

Lr(θ)≜Ex∼Dr[(sθ(x)−sθ0(x)σin(x))2]≤(1−τ)Zcut2,(16)L_r(\theta)\triangleq\mathbb{E}_{x\sim D_r}\left[\left(\frac{s_\theta(x)-s_{\theta_0}(x)}{\sigma_{\mathrm{in}}(x)}\right)^2\right]\le(1-\tau)Z_{\mathrm{cut}}^2, \tag*{(16)}

where θ0\theta_0 denotes the model before unlearning. Then the DNO upper bound in Theorem 3.1 continues to hold for SALA with access only to the forget set.

Efficient Estimation of Per-Sample (Non-)Member Distributions

SALA requires the per-sample member and non-member output distributions in Eq. (6). A straightforward baseline is the shadow-model estimation [6], which trains multiple models on different forget-retain splits and aggregates their outputs. Compatible member and non-member score statistics retained from a shadow-model memorization pipeline, as used in SUM [40], can be reused to estimate these distributions. However, the shadow-model approach itself is generally too expensive. Therefore, the technical challenge for SALA is to estimate the per-sample distributions for all training samples at a cost comparable to a single training run, while also producing the target model itself.

Method: Split–Merge Training with Segment Ensemble (SMSE) Estimation

This paper proposes SMSE to estimate the distributions for every xx using one structured training run that also produces the final model. In standard training, estimating Qout(⋅∣x)Q_{\mathrm{out}}(\cdot\mid x) is challenging: every sample is always in-training, so no true non-member state exists. SMSE estimates non-member behavior by splitting the training set into two complementary halves and training two branches from shared warmup weights; each sample is excluded from one branch during half-data training. A merge stage then recovers a single final model via Federated Averaging [23]. To capture training randomness and approximate training-induced variability, SMSE uses an ensemble of late-stage checkpoints rather than many independent full retrains.

Overview. SMSE consists of four stages: (1) Split and warmup: the training set DD is split into two disjoint subsets AA and BB, followed by a warmup training on DD that initializes shared weights; (2) Half-data training: two models are trained independently, producing θA\theta_A from AA and θB\theta_B from BB; (3) Merge: θA\theta_A and θB\theta_B are merged using Federated Averaging [23], and the merged model is then fine-tuned on A∪BA \cup B; (4) Segment ensemble: checkpoints from the AA-model and BB-model are aggregated as an ensemble to estimate per-sample score moments. Appendix A provides the SMSE pseudo-code.

Segment-ensemble estimation of per-sample moments. Let sθ(x)s_\theta(x) denote the scalar score in (5). Let {θA(m)}m=1M\{\theta_A^{(m)}\}_{m=1}^{M} and {θB(m)}m=1M\{\theta_B^{(m)}\}_{m=1}^{M} be MM late-stage checkpoints from the two half-data trainings. For each sample xx, define role-specific checkpoints θt(m)(x)\theta_t^{(m)}(x) for t∈{in,out}t \in\{\mathrm{in},\mathrm{out}\}: if x∈Ax \in A, then θin(m)(x)=θA(m)\theta_{\mathrm{in}}^{(m)}(x) = \theta_A^{(m)} and θout(m)(x)=θB(m)\theta_{\mathrm{out}}^{(m)}(x) = \theta_B^{(m)}; if x∈Bx \in B, the two roles are reversed. SMSE then estimates

μ^t(x)=1M∑m=1Msθt(m)(x)(x),σ^t2(x)=1M∑m=1M(sθt(m)(x)(x)−μ^t(x))2,(17)\hat{\mu}_t(x) = \frac{1}{M}\sum_{m=1}^{M}s_{\theta_t^{(m)}(x)}(x), \qquad\hat{\sigma}_t^2(x) = \frac{1}{M}\sum_{m=1}^{M}\left(s_{\theta_t^{(m)}(x)}(x)-\hat{\mu}_t(x)\right)^2, \tag*{(17)}

for t∈{in,out}t \in\{\mathrm{in},\mathrm{out}\}. These estimates define Q^in(⋅∣x)\hat{Q}_{\mathrm{in}}(\cdot\mid x) and Q^out(⋅∣x)\hat{Q}_{\mathrm{out}}(\cdot\mid x) in (6) for all xx, at a cost close to a single training run.

Effect of Distribution Estimation Error on SALA’s DNO Bound in Theorem 3.1

The DNO bound in Theorem 3.1 assumes exact target distributions from multiple retrained shadow models. In practice, SALA uses lightweight SMSE estimates instead, which introduces distribution estimation error and thereby affects the bound. Next, we quantify that effect and derive the corresponding adjusted DNO bound.

The key point is that Theorem 3.1 still controls the discrepancy induced by SALA, but now with respect to the estimated target distribution Q^(⋅∣x)\hat{Q}(\cdot\mid x) by SMSE rather than the exact target distribution Q(⋅∣x)Q(\cdot\mid x). Consequently, the DNO with respect to Q^(⋅∣x)\hat{Q}(\cdot\mid x) can be decomposed into two parts: an estimation error term and a SALA-induced term. This is summarized in the following corollary.

Corollary 4.1 (Adjusted DNO Bound under Distribution Estimation Error). For a fixed sample xx and a fixed role (member or non-member), let Q(⋅∣x)Q(\cdot\mid x) denote the true target distribution, let Q^(⋅∣x)\hat{Q}(\cdot\mid x) denote its estimate, and let QSALA(⋅∣x)Q^{\mathrm{SALA}}(\cdot\mid x) denote the distribution induced by SALA. Then

DNO⁡(x)≜TV⁡(QSALA(⋅∣x),Q(⋅∣x))≤TV⁡(Q(⋅∣x),Q^(⋅∣x))⏟DNO⁡est(x)+TV⁡(QSALA(⋅∣x),Q^(⋅∣x))⏟DNO⁡Q^(x).(18)\operatorname{DNO}(x) \triangleq\operatorname{TV}\left(Q^{\mathrm{SALA}}(\cdot\mid x),Q(\cdot\mid x)\right) \leq\underbrace{\operatorname{TV}\left(Q(\cdot\mid x),\hat{Q}(\cdot\mid x)\right)}_{\operatorname{DNO}_{\mathrm{est}}(x)}+\underbrace{\operatorname{TV}\left(Q^{\mathrm{SALA}}(\cdot\mid x),\hat{Q}(\cdot\mid x)\right)}_{\operatorname{DNO}_{\hat{Q}}(x)}. \tag*{(18)}

*Here, DNO⁡est(x)\operatorname{DNO}_{\mathrm{est}}(x) is the distribution estimation error, which is explicitly bounded in Appendix D. DNO⁡Q^(x)\operatorname{DNO}_{\hat{Q}}(x) is the per-sample DNO component with respect to the estimated Q^(x)\hat{Q}(x). By Theorem 3.1 applied to Q^(⋅∣x)\hat{Q}(\cdot\mid x), we have DNO⁡Q^(⋅∣x)≤2(1−BC^min⁡)\operatorname{DNO}_{\hat{Q}}(\cdot\mid x) \leq\sqrt{2(1-\widehat{\mathrm{BC}}_{\min})}.

Experiments

We evaluate whether SALA enables sample-grained approximate unlearning. Specifically, we test whether dataset-level aggregate metrics hide per-sample deviation from exact retraining, whether SALA reduces this deviation under DNO while preserving utility, privacy, and efficiency, and whether SMSE provides accurate target estimates for SALA. Code is available at https://github.com/SALA-Unlearning/Sala-unlearning.

Experimental Setup

Datasets and models. We evaluate SALA on classification and generation tasks. For classification, we use CIFAR-10 [25] with ResNet-18 [19], and ImageNet-1000 [32] with ResNet-50. For generation, we use WikiText-103 [28] with GPT-2 [31]. Appendix E provides implementation details.

Unlearning setup and metrics. We evaluate three unlearning scenarios: random sample unlearning [2], class-wise unlearning [12] for classification, and sequence unlearning [30] for generation. Sequence unlearning uses contiguous text segments as forget sets. We evaluate unlearning from efficacy, privacy, and runtime perspectives, following § 2.2. DNO is computed over multiple unlearning tasks, with 64 tasks on CIFAR-10 and 16 tasks on ImageNet-1000 and WikiText-103. We compare against 12 approximate unlearning baselines on CIFAR-10. Due to memory constraints, we include five representative baselines on ImageNet-1000 and WikiText-103. All baselines use manually tuned hyperparameters, and Appendix § 13 provides the baseline configurations and search protocol.

SALA and SMSE configuration. SALA is an auxiliary objective for iterative unlearning, so we pair it with SalUn [12] in the main experiments. Results on other AU backbones are consistent and are reported in Appendix § 14.§ 7. We evaluate three variants: SALA with shadow-model targets, SALA-SMSE with SMSE-estimated targets, and SALA w/ DfD_f only, which constrains only the forget set. Appendix § 13 gives the full SALA and SMSE configurations, and Appendices § 14.3 and § 14.5 report the corresponding parameter ablations.

Evaluation of SMSE Estimation

We evaluate SMSE by comparing its estimates of (μin,σin,μout,σout)(\mu_{\mathrm{in}}, \sigma_{\mathrm{in}}, \mu_{\mathrm{out}}, \sigma_{\mathrm{out}}) in (6) against expensive shadow-model references and a lightweight single-run baseline. The shadow-model reference trains 64 models on CIFAR-10 and 16 models on ImageNet-1000 and WikiText-103 under different forget-retain splits, which incurs high computational cost. The single-run baseline uses a Laplace approximation to estimate (μin,σin)(\mu_{\mathrm{in}}, \sigma_{\mathrm{in}}) [9], sets μout(x)≈μin(x)−ΔμSI(x)\mu_{\mathrm{out}}(x) \approx\mu_{\mathrm{in}}(x) - \Delta\mu_{\mathrm{SI}}(x) using the self-influence shift ΔμSI(x)\Delta\mu_{\mathrm{SI}}(x), and obtains σout(x)\sigma_{\mathrm{out}}(x) through the same uncertainty propagation [3].

Results. We measure estimation quality by MSE, Pearson correlation, and DNOest\mathrm{DNO}_{\mathrm{est}} against shadow-model references, and report training cost relative to a standard training run. Table 1 shows that SMSE improves over Laplace+SI across all datasets and moments with 1.00× nominal training process. Appendix § 14.4 supplements the experiments with comprehensive computational costs. The gains are largest for non-member and variance estimates: on CIFAR-10, SMSE raises the correlation of μout\mu_{\mathrm{out}} from 0.41 to 0.92 and lowers DNOest\mathrm{DNO}_{\mathrm{est}} from 0.4580 to 0.1801. Figure 4 visualizes the per-sample agreement between SMSE estimates and shadow-model references, and Table 13 in Appendix § 14.6 reports the utility of SMSE-trained models. It indicates that SMSE preserves utility, with at most a 0.18% accuracy drop on vision tasks and a 0.99 PPL increase on WikiText-103 compared with standard training.

DatasetMethodμin\mu_{\mathrm{in}}μout\mu_{\mathrm{out}}σin\sigma_{\mathrm{in}}σout\sigma_{\mathrm{out}}DNOest↓_{\mathrm{est}}\downarrowCost↓\downarrow
MSE↓\downarrowPearson r↑\uparrowMSE↓\downarrowPearson r↑\uparrowMSE↓\downarrowPearson r↑\uparrowMSE↓\downarrowPearson r↑\uparrow
CIFAR-10Laplace + SI2.800.489.480.410.780.042.450.050.45801.24×
SMSE (Ours)1.290.791.900.920.670.580.520.630.18011.00×
ImageNet-1000Laplace + SI0.740.904.230.450.920.032.720.040.48351.08×
SMSE (Ours)0.280.980.310.980.590.620.670.520.10641.00×
WikiText-103Laplace + SI0.150.921.560.480.850.051.14-0.080.39131.16×
SMSE (Ours)0.00650.990.00670.990.470.650.560.680.09351.00×

Table 1. Estimation evaluation of per-sample member and non-member distribution parameters.

Figure 4. Per-sample agreement between SMSE-estimated moments and shadow-model references on CIFAR-10.

DatasetStandard TrainingSMSE
CIFAR-10 (ACC↑\mathrm{ACC} \uparrow)94.68±0.3594.68 \pm0.3594.79±0.1394.79 \pm0.13
ImageNet-1000 (ACC↑\mathrm{ACC} \uparrow)73.55±0.1373.55 \pm0.1373.37±0.2373.37 \pm0.23
WikiText-103 (PPL↓\mathrm{PPL} \downarrow)53.36±0.6153.36 \pm0.6154.35±1.0554.35 \pm1.05

Table 13. Utility of models produced by SMSE.

Comprehensive Evaluation for Unlearning Methods

Table 2 compares SALA with representative approximate unlearning baselines on random sample unlearning and sequence unlearning. Class-wise unlearning is reported in Appendix F.9. The results show that aggregate gap-to-retraining metrics alone are insufficient to characterize approximate unlearning. On CIFAR-10, SalUn reaches ToW = 0.95, but its DNO remains 0.9009 and its RULI remains 92.38%. The same pattern appears on ImageNet-1000, where SalUn reaches ToW = 0.95 but still has DNO = 0.9256 and RULI = 92.30%. Thus, dataset-level similarity to retraining does not necessarily eliminate sample-level deviation or unlearning-aware privacy leakage.

SALA consistently reduces the sample-grained distributional gap while preserving utility and efficiency. Across CIFAR-10, ImageNet-1000, and WikiText-103, SALA achieves the lowest DNO among approximate methods, with DNO = 0.1365, 0.1268, and 0.1048, respectively. It also keeps RULI close to random guessing, with 52.15%, 52.35%, and 50.92% on the three datasets. Across all three datasets, SALA reduces runtime by more than an order of magnitude relative to retraining. SALA-SMSE and SALA w/ DfD_f only have higher DNO than full SALA, which is consistent with estimation error and relaxed retain-side constraints, but they still keep DNO substantially below the corresponding SalUn baseline.

Robustness Evaluation for SALA and Baselines

Unlike § 5.3, which reports tuned configurations, this subsection evaluates unlearning robustness under a broad learning-rate sweep. We test SalUn [12], SCRUB [26], and ℓ1\ell_1 sparsity [21] with five log-spaced learning rates from 10−410^{-4} to 10−110^{-1} and fix the epoch budget to 10.

Results. Figure 5 shows that the performance of base unlearning methods changes sharply across the sweep. SalUn has a ToW range of 0.55, and SCRUB has a DNO range of 0.28, which indicates unstable dataset-level and sample-level unlearning quality. SALA reduces this variation through fast-probe hyper-parameter selection and InDist-based stopping. The red clouds show kernel density estimates of the baseline evaluation points and darker regions indicate higher density. In detail, with SALA and its variants, the ToW range drops to 0.03 for SalUn and 0.04 for SCRUB and ℓ1\ell_1 sparsity. Furthermore, the DNO range drops to 0.04 for SalUn and SCRUB and 0.06 for ℓ1\ell_1 sparsity. Overall, SALA stabilizes unlearning quality through automatic selection among the prescribed candidate configurations. Appendix § 14.2 reports the full results.

Figure 5. Robustness under hyperparameter sweeps for three unlearning methods. SALA substantially reduces performance fluctuation.

Exact unlearning removes designated forget data by ensuring that it never participates in training. Retraining from scratch on the retain set gives the gold standard. Sharding-based methods such as SISA [2] reduce deletion cost, but exact unlearning remains expensive at modern scale. This cost motivates approximate unlearning [15].

Approximate unlearning keeps the pretrained model and updates it to reduce the influence of the forget set. This family can be divided into certified and empirical routes, depending on whether the update gives a formal indistinguishability guarantee with respect to exact retraining. Certified unlearning pursues this goal in parameter space and often converts unlearning error into a boundable indistinguishability gap with tools related to differential privacy [17]. These guarantees usually require extra assumptions and calibrated noise injection, which can limit performance on large and complex models [7, 24, 29]. Since these conditions are often impractical, empirical approximate unlearning has become the default route. It supports flexible updates and can match retraining under empirical metrics, so it is widely used in large-scale settings such as large language models [10, 38]. Common designs include fine-tuning [16], gradient ascent [36, 39], relabeling objectives [8, 16], and influence-based unlearning [15, 37]. However, this flexibility leaves two practical gaps. Empirical approximate unlearning usually lacks a theory-backed upper bound on the gap to retraining, and its performance can change sharply with hyperparameters. These gaps motivate SALA, which stabilizes approximate unlearning and provides a computable upper bound on the per-sample gap to retraining.

Conclusion

Approximate unlearning can reduce dataset-level gap to retraining, but our experiments show that small dataset-level discrepancy can still hide large sample-level deviations. We address this issue with SALA, a plug-in framework that uses per-sample likelihood alignment to steer each sample toward its role-dependent target distribution. Under the InDist criterion, SALA gives a computable upper bound on the sample-grained distribution gap measured by DNO. We further introduce SMSE, a low-overhead estimator for the per-sample member and non-member target distributions required by SALA. Across classification and generation tasks, SALA reduces DNO by up to 86.67% and improves robustness to unlearning hyperparameters.

Acknowledgments and Disclosure of Funding

This work was supported by the Ministry of Science and Technology of the People’s Republic of China (National Key Research and Development Programme, Grant No: 2025YFE0200100), the National Natural Science Foundation of China (Grant No. 62502422), Guangdong Overseas Distinguished Scholars Program (Grant No: MS202600058), and the Research Grants Council (Grant No: 15208923, 15224124, and 25207224), Hong Kong SAR, China.

Algorithm

In Algorithm 1, z(x;θ)z(x;\theta) denotes the shifted zz-score in Eq. (7); in Algorithm 2, DA=AD_A=A and DB=BD_B=B. The experimental offset clipping and scale stabilization follow Appendix § 13.

Algorithm 1 SALA: sample-grained likelihood alignment

Require: θ\theta, D=Dr∪DfD=D_r\cup D_f; target distributions (μin/out(x),σin/out(x))(\mu_{\mathrm{in/out}}(x),\sigma_{\mathrm{in/out}}(x)); unlearning algorithm U(⋅;arg)\mathcal{U}(\mathord{\cdot};\mathrm{arg}); hyper-parameter candidates K\mathcal{K}; ZcutZ_{\mathrm{cut}}, τ\tau; λr\lambda_r, λf\lambda_f, α\alpha.

Ensure: θu\theta_u.

  1. Set (μ(x),σ(x))←(μin(x),σin(x))(\mu(x),\sigma(x))\leftarrow(\mu_{\mathrm{in}}(x),\sigma_{\mathrm{in}}(x)) for x∈Drx\in D_r; (μout(x),σout(x))(\mu_{\mathrm{out}}(x),\sigma_{\mathrm{out}}(x)) for x∈Dfx\in D_f

  2. Sample fixed ϵ(x)∼N(0,var⁡ϵ)\epsilon(x)\sim\mathcal{N}(0,\operatorname{var}_{\epsilon}), set μr(x)←μ(x)+ϵ(x)σ(x)\mu^r(x)\leftarrow\mu(x)+\epsilon(x)\sigma(x)

  3. Choose hard subset Dhf⊆DfD_{hf}\subseteq D_f (top-pp by hardness ≜∣μin(x)−μout(x)∣\triangleq\lvert\mu_{\mathrm{in}}(x)-\mu_{\mathrm{out}}(x)\rvert)

  4. Define InDist⁡(S;θ)=Ex∼S[1(∣z(x;θ)∣<Zcut)]\operatorname{InDist}(S;\theta)=\mathbb{E}_{x\sim S}[1(\lvert z(x;\theta)\rvert<Z_{\mathrm{cut}})]

  5. Store the pre-probe state θ0←θ\theta^0\leftarrow\theta

  6. for all arg∈K\mathrm{arg}\in\mathcal{K} do

  7. Initialize a probe copy θarg←θ0\theta_{\mathrm{arg}}\leftarrow\theta^0

  8. Run TfpT_{\mathrm{fp}} unlearning probe steps from θarg\theta_{\mathrm{arg}} with arg\mathrm{arg}

  9. score⁡fp(arg)←∑S∈{Dr,Df,Dhf}InDist⁡(S;θarg)\operatorname{score}_{\mathrm{fp}}(\mathrm{arg})\leftarrow\sum_{S\in\{D_r,D_f,D_{hf}\}}\operatorname{InDist}(S;\theta_{\mathrm{arg}})

  10. end for

  11. arg⋆←arg max⁡arg∈Kscore⁡fp(arg)\mathrm{arg}^{\star}\leftarrow\mathop{\operatorname{arg\,max}}\limits_{\mathrm{arg}\in\mathcal{K}}\operatorname{score}_{\mathrm{fp}}(\mathrm{arg})

  12. Reset θ←θ0\theta\leftarrow\theta^0

  13. while ∃S∈{Dr,Df,Dhf}\exists S\in\{D_r,D_f,D_{hf}\} s.t. InDist⁡(S;θ)<τ\operatorname{InDist}(S;\theta)<\tau do

  14. sample minibatch B⊆DB\subseteq D

  15. LSALA(θ)←1∣B∣∑x∈Bω(x)(sθ(x)−μr(x)σ(x))2\mathcal{L}_{\mathrm{SALA}}(\theta)\leftarrow\frac{1}{\lvert B\rvert}\sum_{x\in B}\omega(x)\left(\frac{s_{\theta}(x)-\mu^r(x)}{\sigma(x)}\right)^2

  16. where ω(x)=λr1[x∈Dr]+λf(1+α1[x∈Dhf])1[x∈Df]\omega(x)=\lambda_r1[x\in D_r]+\lambda_f(1+\alpha1[x\in D_{hf}])1[x\in D_f]

  17. θ←U(θ;arg⋆)\theta\leftarrow\mathcal{U}(\theta;\mathrm{arg}^{\star}) using (unlearning loss +LSALA+\mathcal{L}_{\mathrm{SALA}})

  18. end while

  19. return θu←θ\theta_u\leftarrow\theta

Algorithm 2 SMSE: split–merge segment ensemble estimation for member and non-member distributions

Require: Dataset DD; training algorithm A\mathcal{A}; warmup epoch ewe_w; merge finetune epoch eme_m; checkpoints per branch MM.

Ensure: {μ^in(x),σ^in(x),μ^out(x),σ^out(x)}x∈D,θ\{\hat{\mu}_{\mathrm{in}}(x),\hat{\sigma}_{\mathrm{in}}(x),\hat{\mu}_{\mathrm{out}}(x),\hat{\sigma}_{\mathrm{out}}(x)\}_{x\in D},\theta.

  1. Split DD into disjoint halves (A,B)(A,B); warmup θwarm←A(D;epochs=ew)\theta_{\mathrm{warm}}\leftarrow\mathcal{A}(D;\text{epochs}=e_w)

  2. Train branches: θA←A(A;init=θwarm)\theta_A\leftarrow\mathcal{A}(A;\text{init}=\theta_{\mathrm{warm}}), θB←A(B;init=θwarm)\theta_B\leftarrow\mathcal{A}(B;\text{init}=\theta_{\mathrm{warm}})

  3. Record late checkpoints {θA(m)}m=1M\{\theta_A^{(m)}\}_{m=1}^{M}, {θB(m)}m=1M\{\theta_B^{(m)}\}_{m=1}^{M}

  4. for all x∈Ax\in A (or x∈Bx\in B) do

  5. let br⁡(x)=A\operatorname{br}(x)=A if x∈Ax\in A else BB, and br⁡‾(x)\overline{\operatorname{br}}(x) be the other branch

  6. um←sθbr⁡‾(x)(m)(x)u_m\leftarrow s_{\theta_{\overline{\operatorname{br}}(x)}^{(m)}}(x), vm←sθbr⁡(x)(m)(x)v_m\leftarrow s_{\theta_{\operatorname{br}(x)}^{(m)}}(x) for m=1…Mm=1\ldots M

  7. μ^in(x)←1M∑mum\hat{\mu}_{\mathrm{in}}(x)\leftarrow\frac{1}{M}\sum_m u_m, σ^in2(x)←1M∑m(um−μ^in(x))2\hat{\sigma}_{\mathrm{in}}^2(x)\leftarrow\frac{1}{M}\sum_m(u_m-\hat{\mu}_{\mathrm{in}}(x))^2

  8. μ^out(x)←1M∑mvm\hat{\mu}_{\mathrm{out}}(x)\leftarrow\frac{1}{M}\sum_m v_m, σ^out2(x)←1M∑m(vm−μ^out(x))2\hat{\sigma}_{\mathrm{out}}^2(x)\leftarrow\frac{1}{M}\sum_m(v_m-\hat{\mu}_{\mathrm{out}}(x))^2

  9. end for

  10. σ^in(x)←σ^in2(x)\hat{\sigma}_{\mathrm{in}}(x)\leftarrow\sqrt{\hat{\sigma}_{\mathrm{in}}^2(x)}, σ^out(x)←σ^out2(x)\hat{\sigma}_{\mathrm{out}}(x)\leftarrow\sqrt{\hat{\sigma}_{\mathrm{out}}^2(x)} for all xx

  11. merge: θ←A(D;init=FedAvg⁡(θA,θB),epochs=em)\theta\leftarrow\mathcal{A}(D;\text{init}=\operatorname{FedAvg}(\theta_A,\theta_B),\text{epochs}=e_m)

  12. where FedAvg⁡(θA,θB)=∣DA∣∣DA∣+∣DB∣θA+∣DB∣∣DA∣+∣DB∣θB\operatorname{FedAvg}(\theta_A,\theta_B)=\frac{\lvert D_A\rvert}{\lvert D_A\rvert+\lvert D_B\rvert}\theta_A+\frac{\lvert D_B\rvert}{\lvert D_A\rvert+\lvert D_B\rvert}\theta_B

  13. return per-sample moments and θ\theta

Proof of Theorem 3.1

Proof. Step 1 (DNO term is total variation). For two densities QSALAQ^{\mathrm{SALA}}, QQ on the same space,

DNO=1−∫min⁡(QSALA,Q)=12∫∣QSALA−Q∣=TV(QSALA,Q).\mathrm{DNO}=1-\int\min(Q^{\mathrm{SALA}},Q)=\frac{1}{2}\int\lvert Q^{\mathrm{SALA}}-Q\rvert=\mathrm{TV}(Q^{\mathrm{SALA}},Q).

Step 2 (TV bound by Bhattacharyya coefficient). The Hellinger distance satisfies H2(QSALA,Q)=1−∫QSALAQ=1−BC(QSALA,Q)H^2(Q^{\mathrm{SALA}}, Q) = 1 - \int\sqrt{Q^{\mathrm{SALA}}Q} = 1 - \mathrm{BC}(Q^{\mathrm{SALA}}, Q) and TV(QSALA,Q)≤2H(QSALA,Q)\mathrm{TV}(Q^{\mathrm{SALA}}, Q) \le\sqrt{2} H(Q^{\mathrm{SALA}}, Q), hence

TV(QSALA,Q)≤2(1−BC(QSALA,Q)).(19)\mathrm{TV}(Q^{\mathrm{SALA}}, Q) \le\sqrt{2(1-\mathrm{BC}(Q^{\mathrm{SALA}}, Q))}. \tag*{(19)}

Step 3 (closed form BC for Gaussians). For Q=N(μ,σ2)Q=\mathcal{N}(\mu,\sigma^2) and QSALA=N(μSALA,(σSALA)2)Q^{\mathrm{SALA}}=\mathcal{N}(\mu^{\mathrm{SALA}},(\sigma^{\mathrm{SALA}})^2),

BC(QSALA,Q)=2ρ1+ρ2exp⁡(−Δ24(1+ρ2)),\mathrm{BC}(Q^{\mathrm{SALA}}, Q)=\sqrt{\frac{2\rho}{1+\rho^2}}\exp\left(-\frac{\Delta^2}{4(1+\rho^2)}\right),

with Δ\Delta, ρ\rho defined above.

Step 4 (InDist implies a feasible region for (Δ,ρ)(\Delta,\rho)). Under X∼QSALAX\sim Q^{\mathrm{SALA}},

Y≜X−μσ∼N(Δ,ρ2).Y \triangleq\frac{X-\mu}{\sigma}\sim\mathcal{N}(\Delta,\rho^2).

Constraint (12) becomes

Φ(Zcut−Δρ)−Φ(−Zcut−Δρ)≥τ.\Phi\left(\frac{Z_{\mathrm{cut}}-\Delta}{\rho}\right)-\Phi\left(\frac{-Z_{\mathrm{cut}}-\Delta}{\rho}\right)\ge\tau.

A sufficient (closed-form) condition is obtained by symmetry:

Φ(Zcut−∣Δ∣ρ)≥1+τ2,\Phi\left(\frac{Z_{\mathrm{cut}}-|\Delta|}{\rho}\right)\ge\frac{1+\tau}{2},

which is equivalent to

∣Δ∣≤Zcut−ρq,ρ≤ρmax⁡=Zcutq,(20)|\Delta|\le Z_{\mathrm{cut}}-\rho q,\qquad\rho\le\rho_{\max}=\frac{Z_{\mathrm{cut}}}{q}, \tag*{(20)}

with q=Φ−1((1+τ)/2)q=\Phi^{-1}((1+\tau)/2).

Step 5 (worst case over the region). For fixed ρ\rho, BC decreases with ∣Δ∣|\Delta|, so the smallest BC occurs at the boundary ∣Δ∣=Zcut−ρq|\Delta|=Z_{\mathrm{cut}}-\rho q. Minimizing over ρ∈[ρmin⁡,ρmax⁡]\rho\in[\rho_{\min},\rho_{\max}] yields (14). Plugging BC≥BCmin⁡\mathrm{BC}\ge\mathrm{BC}_{\min} into (19) proves (13).

If the same argument holds for both Qin(⋅∣x)Q_{\mathrm{in}}(\cdot\mid x) and Qout(⋅∣x)Q_{\mathrm{out}}(\cdot\mid x), each overlap gap is bounded by (13), and their averaged DNO is bounded as well.

Proof of Corollary 3.2

By Markov’s inequality,

Px∼Dr(∣sθ(x)−sθ0(x)σin(x)∣≥Zcut)≤Lr(θ)Zcut2≤1−τ.(21)\mathbb{P}_{x\sim D_r}\left(\left|\frac{s_\theta(x)-s_{\theta_0}(x)}{\sigma_{\mathrm{in}}(x)}\right|\ge Z_{\mathrm{cut}}\right)\le\frac{L_r(\theta)}{Z_{\mathrm{cut}}^2}\le1-\tau. \tag*{(21)}

Equivalently,

Px∼Dr(∣sθ(x)−sθ0(x)σin(x)∣<Zcut)≥τ.(22)\mathbb{P}_{x\sim D_r}\left(\left|\frac{s_\theta(x)-s_{\theta_0}(x)}{\sigma_{\mathrm{in}}(x)}\right|<Z_{\mathrm{cut}}\right)\ge\tau. \tag*{(22)}

Hence, the retain-side condition required by the DNO analysis is recovered without explicitly applying SALA to DrD_r. Therefore, the retain-side coverage requirement in Theorem 3.1 can be equivalently replaced by the condition in (16), and the same DNO upper bound continues to hold.

Explicit Bound of the Estimation Error

Bounding the estimation term DNOest\mathrm{DNO}_{\mathrm{est}}. DNOest\mathrm{DNO}_{\mathrm{est}} is bounded by the Bhattacharyya coefficient:

DNOest=TV(Q,Q^)≤2(1−BCmin⁡(Q,Q^)),(23)\mathrm{DNO}_{\mathrm{est}}=\mathrm{TV}(Q,\widehat{Q})\le\sqrt{2(1-\mathrm{BC}_{\min}(Q,\widehat{Q}))}, \tag*{(23)}

Since BC decreases with ∣μ−μ^∣|\mu-\widehat{\mu}|, the worst case of ∣μ−μ^∣|\mu-\widehat{\mu}| is defined as εμ\varepsilon_\mu. Let ρ≜σσ^\rho\triangleq\frac{\sigma}{\widehat{\sigma}} and Δmax⁡≜εμσ^\Delta_{\max}\triangleq\frac{\varepsilon_\mu}{\widehat{\sigma}}. Then ρ∈[ρL,ρU]\rho\in[\rho_L,\rho_U] with ρL=σLσ^\rho_L=\frac{\sigma_L}{\widehat{\sigma}} and ρU=σUσ^\rho_U=\frac{\sigma_U}{\widehat{\sigma}}, and

BCmin⁡(Q,Q^)=min⁡ρ∈[ρL,ρU][2ρ1+ρ2exp⁡(−Δmax⁡24(1+ρ2))].(24)\mathrm{BC}_{\min}(Q,\widehat{Q})=\min_{\rho\in[\rho_L,\rho_U]}\left[\sqrt{\frac{2\rho}{1+\rho^2}}\exp\left(-\frac{\Delta_{\max}^2}{4(1+\rho^2)}\right)\right]. \tag*{(24)}

Combining (23)–(24) gives an explicit bound for DNOest\mathrm{DNO}_{\mathrm{est}}.

Supplementary Experimental Setups

Dataset and model training. Training follows common setups in prior unlearning work on three benchmarks of increasing scale: CIFAR-10 (50,000 training images) with ResNet-18, ImageNet-1000 (1,281,167 training images) with ResNet-50, and WikiText-103 (1,165,029 training sequences) with GPT-2. CIFAR-10 trains for 150 epochs with SGD and learning rate 0.1. ImageNet-1000 trains for 90 epochs with SGD and learning rate 0.1. WikiText-103 trains for 22 epochs with AdamW and learning rate 5×10−45 \times10^{-4}.

Unlearning scenarios and evaluation protocol. We evaluate three unlearning scenarios. In random sample-wise unlearning, the forget set DfD_f contains individual training examples sampled across classes [2]. In class-wise unlearning, DfD_f contains all training samples from one or more target classes [12, 35]. In sequence unlearning, DfD_f contains contiguous target token sequences from the corpus [30]. For each random sample-wise unlearning task, we construct task-specific held-out, forget, and retain subsets from the original training split.

On CIFAR-10, we reserve 20% of the training samples as the task-level held-out subset and 20% as the forget subset, with the remaining 60% used as the retain subset. On ImageNet-1000 and WikiText-103, we reserve 33% of the training samples or sequences as the task-level held-out subset and 33% as the forget subset, with the remaining 34% used as the retain subset. These proportions provide observations in both membership roles over repeated tasks for estimating the per-sample score distributions. For class-wise unlearning, the forget set contains one entire target class, and the retain set contains all remaining classes. DNO is computed over 64 tasks on CIFAR-10 and 16 tasks on ImageNet-1000 and WikiText-103.

Baseline coverage. We compare SALA with direct update-based baselines and optimized approximate unlearning (AU) frameworks. The full set of 12 baselines is evaluated on CIFAR-10. Due to memory constraints, ImageNet-1000 and WikiText-103 use the five representative baselines reported in Table 2. All baselines use tuned hyperparameters to maximize their own performance under the same evaluation protocol.

MethodTA(%) (T-PPL)FA(%) (F-PPL)RA(%) (R-PPL)ToW ↓DNO ↓MIA efficacy (%)↓RULI (%)↓Time (s)↓
Retrain93.84±0.2193.84\pm0.2193.68±0.2893.68\pm0.28100.00±0.01100.00\pm0.011.00±0.001.00\pm0.000.0646±0.00220.0646\pm0.002250.57±0.2550.57\pm0.2550.00±0.0050.00\pm0.001625.53±5.331625.53\pm5.33
FT89.04±1.1389.04\pm1.1392.32±1.2092.32\pm1.2095.84±1.1095.84\pm1.100.90±0.010.90\pm0.010.3388±0.00530.3388\pm0.005352.09±0.2252.09\pm0.2270.01±0.7370.01\pm0.73242.01±42.25242.01\pm42.25
GA86.01±1.4286.01\pm1.4285.93±2.1085.93\pm2.1095.13±1.2095.13\pm1.200.80±0.010.80\pm0.010.6228±0.00540.6228\pm0.005451.04±0.5251.04\pm0.5287.69±0.8687.69\pm0.86149.30±2.02149.30\pm2.02
RL88.94±0.7188.94\pm0.7187.63±0.9087.63\pm0.9098.08±0.6098.08\pm0.600.87±0.020.87\pm0.020.6875±0.00590.6875\pm0.005955.10±0.2155.10\pm0.2182.35±14.3182.35\pm14.31321.32±7.70321.32\pm7.70
IU86.90±5.6186.90\pm5.6193.17±6.2593.17\pm6.2593.52±6.2693.52\pm6.260.86±0.030.86\pm0.030.3982±0.00900.3982\pm0.009052.35±2.1352.35\pm2.1364.52±5.1764.52\pm5.1741.59±0.5141.59\pm0.51
FF76.11±7.4276.11\pm7.4280.85±8.0880.85\pm8.0880.83±8.1180.83\pm8.110.57±0.030.57\pm0.030.8947±0.02340.8947\pm0.023453.30±0.1553.30\pm0.1590.39±8.4190.39\pm8.412408.39±105.192408.39\pm105.19
MinNorm-OG88.12±1.2888.12\pm1.2894.90±1.1994.90\pm1.1994.91±1.2294.91\pm1.220.88±0.020.88\pm0.020.6355±0.00750.6355\pm0.007557.31±0.4957.31\pm0.4981.81±1.7581.81\pm1.7560.20±7.7560.20\pm7.75
SEMU92.66±0.3492.66\pm0.3495.56±0.3595.56\pm0.3599.82±0.0399.82\pm0.030.96±0.010.96\pm0.010.7390±0.00500.7390\pm0.005053.53±0.4953.53\pm0.4982.42±0.8682.42\pm0.8648.15±5.7648.15\pm5.76
ℓ1\ell_1 sparsity88.18±3.4788.18\pm3.4791.94±4.4891.94\pm4.4895.80±2.9895.80\pm2.980.88±0.020.88\pm0.020.3376±0.00980.3376\pm0.009853.09±2.7253.09\pm2.7270.03±5.2470.03\pm5.24150.28±28.45150.28\pm28.45
SalUn92.36±0.4592.36\pm0.4595.29±0.5795.29\pm0.5798.61±0.3698.61\pm0.360.95±0.010.95\pm0.010.9009±0.00150.9009\pm0.001551.87±0.2151.87\pm0.2192.38±0.2892.38\pm0.28221.92±96.37221.92\pm96.37
RUM91.66±0.4791.66\pm0.4792.96±0.4992.96\pm0.4996.89±0.3996.89\pm0.390.94±0.010.94\pm0.010.7826±0.00620.7826\pm0.006250.88±0.2250.88\pm0.2285.35±1.3785.35\pm1.37375.29±83.05375.29\pm83.05
SFRon87.31±1.6487.31\pm1.6490.39±2.4590.39\pm2.4596.77±1.0596.77\pm1.050.87±0.010.87\pm0.010.4518±0.00910.4518\pm0.009154.23±0.9854.23\pm0.9876.75±2.0476.75\pm2.0467.97±0.7067.97\pm0.70
SCRUB90.29±2.3190.29\pm2.3194.06±3.1594.06\pm3.1595.74±3.0095.74\pm3.000.92±0.030.92\pm0.030.3644±0.00860.3644\pm0.008652.23±0.5752.23\pm0.5768.67±3.6268.67\pm3.62101.93±11.98101.93\pm11.98
SALA94.44±0.2894.44\pm0.2895.58±0.2195.58\pm0.21100.00±0.00100.00\pm0.000.97±0.020.97\pm0.020.1365±0.00550.1365\pm0.005552.42±0.1652.42\pm0.1652.15±1.3052.15\pm1.3053.22±3.2153.22\pm3.21
SALA-SMSE94.25±0.2794.25\pm0.2795.13±0.1795.13\pm0.1799.97±0.0199.97\pm0.010.98±0.010.98\pm0.010.2508±0.00560.2508\pm0.005654.28±0.3654.28\pm0.3655.97±0.5355.97\pm0.5356.47±5.3556.47\pm5.35
SALA w/ DfD_f only93.75±0.2993.75\pm0.2994.49±0.3094.49\pm0.3099.75±0.0899.75\pm0.080.99±0.010.99\pm0.010.2453±0.00420.2453\pm0.004253.37±0.3453.37\pm0.3456.75±2.4956.75\pm2.4959.35±6.2659.35\pm6.26
retrain71.26±0.4771.26\pm0.4771.27±0.5271.27\pm0.5289.45±0.4889.45\pm0.481.00±0.001.00\pm0.000.0277±0.00320.0277\pm0.003250.53±0.0950.53\pm0.0950.01±0.0150.01\pm0.011.42e5±5.47e31.42\mathrm{e}5\pm5.47\mathrm{e}3
FT68.36±0.3968.36\pm0.3972.65±0.6472.65\pm0.6480.89±0.1580.89\pm0.150.85±0.020.85\pm0.020.3497±0.00900.3497\pm0.009052.31±0.1452.31\pm0.1465.42±0.4265.42\pm0.421.93e4±7.52e21.93\mathrm{e}4\pm7.52\mathrm{e}2
GA75.71±0.4075.71\pm0.4087.11±0.4087.11\pm0.4087.22±0.3887.22\pm0.380.71±0.030.71\pm0.030.3931±0.00770.3931\pm0.007751.43±0.2551.43\pm0.2567.66±1.2167.66\pm1.212.92e3±2.12e22.92\mathrm{e}3\pm2.12\mathrm{e}2
RL74.64±1.3374.64\pm1.3374.95±0.2574.95\pm0.2576.28±1.8776.28\pm1.870.77±0.050.77\pm0.050.7048±0.00520.7048\pm0.005254.96±0.3554.96\pm0.3586.29±1.1586.29\pm1.152.76e4±1.10e42.76\mathrm{e}4\pm1.10\mathrm{e}4
ℓ1\ell_1 sparsity67.92±0.4367.92\pm0.4371.50±1.7071.50\pm1.7080.65±0.2880.65\pm0.280.86±0.030.86\pm0.030.3294±0.00930.3294\pm0.009352.20±0.1852.20\pm0.1869.40±1.5569.40\pm1.559.65e3±1.05e39.65\mathrm{e}3\pm1.05\mathrm{e}3
SalUn70.95±0.7070.95\pm0.7073.35±0.6573.35\pm0.6588.15±0.3688.15\pm0.360.95±0.010.95\pm0.010.9256±0.00200.9256\pm0.002052.05±0.1252.05\pm0.1292.30±2.8092.30\pm2.806.02e3±3.20e26.02\mathrm{e}3\pm3.20\mathrm{e}2
SALA71.13±1.6371.13\pm1.6372.33±0.6972.33\pm0.6988.36±1.1488.36\pm1.140.97±0.010.97\pm0.010.1268±0.00490.1268\pm0.004951.05±0.2551.05\pm0.2552.35±0.1752.35\pm0.173.08e3±1.56e23.08\mathrm{e}3\pm1.56\mathrm{e}2
SALA-SMSE71.79±4.4371.79\pm4.4373.92±3.3173.92\pm3.3188.06±1.1488.06\pm1.140.94±0.010.94\pm0.010.1845±0.00560.1845\pm0.005652.35±0.4452.35\pm0.4454.33±0.4754.33\pm0.472.72e3±3.32e22.72\mathrm{e}3\pm3.32\mathrm{e}2
SALA w/ DfD_f only71.49±0.9271.49\pm0.9273.31±1.4373.31\pm1.4388.25±0.8588.25\pm0.850.95±0.010.95\pm0.010.2247±0.00610.2247\pm0.006152.12±0.1352.12\pm0.1354.42±0.3554.42\pm0.352.84e3±1.65e22.84\mathrm{e}3\pm1.65\mathrm{e}2
Retrain63.72±5.4763.72\pm5.4763.73±5.5063.73\pm5.5056.86±6.1456.86\pm6.141.00±0.001.00\pm0.000.0178±0.00270.0178\pm0.002750.43±0.1350.43\pm0.1350.08±0.0450.08\pm0.046.14e4±4.35e36.14\mathrm{e}4\pm4.35\mathrm{e}3
FT61.25±0.4761.25\pm0.4758.85±0.5358.85\pm0.5356.85±0.4856.85\pm0.480.89±0.020.89\pm0.020.2496±0.00520.2496\pm0.005252.08±0.5252.08\pm0.5252.66±0.1152.66\pm0.111.72e3±1.52e21.72\mathrm{e}3\pm1.52\mathrm{e}2
GA60.73±0.4460.73\pm0.4458.76±0.5158.76\pm0.5155.85±0.4555.85\pm0.450.86±0.040.86\pm0.040.2421±0.00460.2421\pm0.004651.45±0.1351.45\pm0.1351.86±0.0951.86\pm0.093.99e3±3.54e23.99\mathrm{e}3\pm3.54\mathrm{e}2
RL63.80±0.1063.80\pm0.1060.61±4.2860.61\pm4.2859.88±3.8859.88\pm3.880.90±0.030.90\pm0.030.2783±0.00610.2783\pm0.006152.73±0.5252.73\pm0.5253.69±0.6353.69\pm0.636.02e2±9.47e16.02\mathrm{e}2\pm9.47\mathrm{e}1
ℓ1\ell_1 sparsity61.25±0.4761.25\pm0.4758.85±0.5358.85\pm0.5356.85±0.4856.85\pm0.480.89±0.030.89\pm0.030.2567±0.00830.2567\pm0.008351.35±0.4351.35\pm0.4352.66±0.1152.66\pm0.112.16e3±1.49e22.16\mathrm{e}3\pm1.49\mathrm{e}2
SalUn62.90±0.4962.90\pm0.4960.40±0.9360.40\pm0.9358.70±0.8558.70\pm0.850.90±0.030.90\pm0.030.3014±0.00700.3014\pm0.007052.10±0.2052.10\pm0.2058.40±1.4258.40\pm1.426.25e3±3.10e26.25\mathrm{e}3\pm3.10\mathrm{e}2
SALA62.18±0.9162.18\pm0.9161.18±0.9061.18\pm0.9057.24±0.8857.24\pm0.880.93±0.030.93\pm0.030.1048±0.00430.1048\pm0.004350.15±0.0850.15\pm0.0852.90±0.1452.90\pm0.146.42e2±2.53e26.42\mathrm{e}2\pm2.53\mathrm{e}2
SALA-SMSE62.70±0.7662.70\pm0.7660.90±0.9360.90\pm0.9359.40±0.9559.40\pm0.950.93±0.020.93\pm0.020.1965±0.00580.1965\pm0.005851.20±0.0851.20\pm0.0851.96±0.3351.96\pm0.337.90e2±1.62e27.90\mathrm{e}2\pm1.62\mathrm{e}2
SALA w/ DfD_f only63.63±0.4663.63\pm0.4661.29±0.8561.29\pm0.8558.12±0.7458.12\pm0.740.94±0.020.94\pm0.020.2217±0.00640.2217\pm0.006451.72±0.2151.72\pm0.2151.79±0.1451.79\pm0.147.69e2±1.86e27.69\mathrm{e}2\pm1.86\mathrm{e}2

Table 2. Comprehensive comparison of approximate unlearning methods.

Unlearning baselines. The evaluated baselines are configured as follows.

  • Fine-tuning (FT) [21]. The pretrained model fθf_\theta is further trained only on the retain set DrD_r. Forgetting emerges from catastrophic forgetting because samples in DfD_f are no longer reinforced. On CIFAR-10, we use learning rate 0.04 for 10 epochs. On ImageNet-1000, we use learning rate 10−510^{-5} for 15 epochs. On WikiText-103, we use AdamW with learning rate 5×10−55 \times10^{-5} and max steps 10,000.

  • Gradient Ascent (GA) [16, 36]. GA approximates unlearning by ascending the loss on DfD_f, which reverses part of the optimization signal induced by the deleted samples. On CIFAR-10, we use learning rate 0.04 for 8 epochs, with retain and forget weights 0.69 and 0.31, λ=20\lambda= 20, and gradient clipping threshold 5. On ImageNet-1000, we use learning rate 10−410^{-4} for 2 epochs. On WikiText-103, we use AdamW with learning rate 5×10−55 \times10^{-5} and max steps 10,000, with α=0.85\alpha= 0.85, ℓ2\ell_2 weight decay 0.02, and forget threshold τf=6.0\tau_f = 6.0.

  • Random Label (RL) [8, 16]. RL randomizes the labels of DfD_f and trains the model on corrupted targets, which disrupts the association between forget samples and their original labels. On CIFAR-10, we optimize the relabeling objective with learning rate 1.5×10−41.5 \times10^{-4} for 8 epochs. On ImageNet-1000, we use Adam with learning rate 5×10−35 \times10^{-3} for 2 epochs. On WikiText-103, we use AdamW with learning rate 5×10−55 \times10^{-5} and max steps 10,000, with amnesiac weight 0.35.

  • Influence Unlearning (IU) [21]. IU uses WoodFisher-style influence estimation to approximate how DfD_f contributes to the learned parameters, and then applies a compensatory parameter perturbation to remove this contribution from θ\theta. On CIFAR-10, we instantiate IU with the WoodFisher variant and set the WoodFisher coefficient to α=20∼50\alpha= 20 \sim50.

  • FisherForgetting (FF) [15]. FF perturbs parameters with Gaussian noise whose covariance is derived from the fourth root of the Fisher information matrix. This perturbation scrubs information associated with the forgotten data while approximately preserving retained behavior. On CIFAR-10, we set the noise-scale parameter to ϵ=7×10−8\epsilon= 7 \times10^{-8}.

  • MinNorm-OG [1]. MinNorm-OG solves for a minimum-norm parameter perturbation under an orthogonal-gradient constraint on DrD_r, followed by descent steps on the retain loss. On CIFAR-10, the retain-loss descent uses learning rate 0.001 for 5 epochs, and the regularization weight is α=1.8\alpha= 1.8 in the minimum-norm perturbation subproblem.

  • SEMU [33]. SEMU uses truncated singular value decomposition of forget-set gradients to identify a low-dimensional critical subspace of weights, and restricts unlearning updates to this compact projection. On CIFAR-10, the fine-tuning stage uses learning rate 0.01 for 3 epochs, and the truncated-rank hyperparameter defaults to 5.

  • ℓ1\ell_1 sparsity [21]. This method adds an ℓ1\ell_1 penalty to retain-set fine-tuning, so the unlearning trajectory favors sparse parameter changes. On CIFAR-10, we use learning rate 0.026 for 9 epochs and disable data augmentation, with ℓ1\ell_1 penalty weight 5×10−55 \times10^{-5}. On ImageNet-1000, we use learning rate 10−410^{-4} for 15 epochs. On WikiText-103, we use AdamW with learning rate 5×10−55 \times10^{-5} and max steps 10,000, with ℓ1\ell_1 penalty weight 2×10−52 \times10^{-5}.

  • SalUn [12]. SalUn uses a gradient-based saliency mask to localize updates on parameters that are most relevant to the forgetting objective. On CIFAR-10, we run saliency-guided unlearning with learning rate 9×10−49 \times10^{-4} for 14 epochs and use a precomputed saliency mask with ratio 0.2. On ImageNet-1000, we use Adam with learning rate 5×10−35 \times10^{-3} for 2 epochs. On WikiText-103, we use AdamW with learning rate 5×10−55 \times10^{-5} and max steps 10,000.

  • SCRUB [26]. SCRUB uses a teacher–student distillation objective. The student deviates from the teacher on DfD_f and stays close to the teacher on DrD_r. On CIFAR-10, we run stochastic gradient ascent–descent with learning rate 1.5×10−41.5 \times10^{-4} for 8 epochs.

  • SFRon [20]. SFRon decomposes gradient-based unlearning into weighted forgetting gradient ascent, retaining gradient descent, and saliency-guided weighting. It further uses remain-geometry information to modulate the update trajectory. On CIFAR-10, we use learning rate 0.0035 for 5 epochs and set the forgetting-update frequency to 1.

  • Refined-Unlearning Meta-algorithm (RUM) [40]. RUM first refines the forget set into more homogeneous subsets and then applies different unlearning strategies across these subsets. On CIFAR-10, we instantiate the pipeline with a fine-tuning stage at learning rate 0.1 followed by a SalUn stage at learning rate 5×10−45 \times10^{-4}. We use 5 fine-tuning epochs and 3 SalUn epochs.

Supplementary setups of SALA and SMSE. We provide the default hyperparameters used for SALA and SMSE. Unless otherwise specified, the main experiments pair SALA with SalUn because SALA acts as an auxiliary objective for iterative unlearning. Additional base unlearning methods are evaluated in Appendix § 14.§ 7.

SALA setup. We evaluate three variants: SALA with shadow-model targets, SALA-SMSE with SMSE-estimated targets, and SALA w/ DfD_f only, which constrains only the forget set. Unless otherwise specified, we set λf=1.0\lambda_f = 1.0, λr=0.2\lambda_r = 0.2, and α=2.0\alpha= 2.0. We use Zcut=2.0Z_{\mathrm{cut}} = 2.0, τ=0.95\tau= 0.95, ϵscale=1.0\epsilon_{\mathrm{scale}} = 1.0, and ϵclip=Zcut−0.2=1.8\epsilon_{\mathrm{clip}} = Z_{\mathrm{cut}} - 0.2 = 1.8. These settings provide stable guidance and keep the target perturbation within the theoretically motivated range. We ablate SALA weights in Appendix § 14.§ 3.1, and we ablate τ\tau and ZcutZ_{\mathrm{cut}} in Appendix § 14.§ 3.2.

SMSE setup. SMSE allocates 20% of the total epochs to warmup, 70% to half-data training, and 10% to fine-tuning after merging. We use segment ensemble size M=30M = 30. Checkpoints are recorded once per epoch on CIFAR-10 and ImageNet-1000, and eight times per epoch on WikiText-103. We ablate SMSE epoch allocation, segment ensemble size, and stochastic estimation method in Appendix § 14.§ 5.2, Appendix § 14.§ 5.3, and Appendix § 14.§ 5.4.

SMSE setup with σ^\hat{\sigma} stabilization. To tighten the adjusted DNO bound in (18), SMSE reduces the estimation term DNOest\mathrm{DNO}_{\mathrm{est}} by stabilizing the estimated scale σ^(x)\hat{\sigma}(x). The Gaussian-form analysis in Appendix § 12 shows that DNOest\mathrm{DNO}_{\mathrm{est}} increases when σ^(x)\hat{\sigma}(x) deviates too far from a plausible range of the true scale σ(x)\sigma(x); see Eq. (24). We therefore shrink each per-sample estimate toward a global scale and clip it to a safe interval:

σ^(x)←clip⁡(λσσ^(x)+(1−λσ)σˉ, σmin⁡, σmax⁡),(25)\hat{\sigma}(x) \leftarrow\operatorname{clip}\left(\lambda_{\sigma}\hat{\sigma}(x) + (1-\lambda_{\sigma})\bar{\sigma},\ \sigma_{\min},\ \sigma_{\max}\right), \tag*{(25)}

where λσ∈[0,1]\lambda_{\sigma} \in[0,1] controls the shrinkage strength, σˉ\bar{\sigma} is the global average of σ^(x)\hat{\sigma}(x) over the training set, and (σmin⁡,σmax⁡)(\sigma_{\min},\sigma_{\max}) prevent unrealistically small or large scales. In all experiments, we set the shrinkage coefficient to λσ=0.8\lambda_\sigma= 0.8, and clip the stabilized scale to [σmin⁡,σmax⁡]=[1e−2,5][\sigma_{\min}, \sigma_{\max}] = [1\mathrm{e}{-2}, 5]. This correction reduces variance in per-sample scale estimates and yields a tighter adjusted DNO estimate. We ablate σ^\hat{\sigma} stabilization in Appendix § 14.§ 5.1.

Implementation. CIFAR-10 runs on 2×2 \times RTX 4090 GPUs. ImageNet-1000 and WikiText-103 run on 8×8 \times RTX 4090 GPUs. All reported runtimes are measured under these hardware settings.

Hyperparameter selection. Each SALA variant evaluates ten candidate configurations using the fast probe and selects by InDist. For SalUn, the learning rates are {10−4,4×10−4,7×10−4,10−3,1.5×10−3,2.5×10−3,4×10−3,6×10−3,8×10−3,10−2}\{10^{-4}, 4 \times10^{-4}, 7 \times10^{-4}, 10^{-3}, 1.5 \times10^{-3}, 2.5 \times10^{-3}, 4 \times10^{-3}, 6 \times10^{-3}, 8 \times10^{-3}, 10^{-2}\}. For ℓ1\ell_1 sparsity, the pool combines {10−4,10−3,2.5×10−3,4×10−3,10−2}\{10^{-4}, 10^{-3}, 2.5 \times10^{-3}, 4 \times10^{-3}, 10^{-2}\} with regularization weights {5×10−5,10−4}\{5 \times10^{-5}, 10^{-4}\}. SCRUB uses the same five learning rates with teacher-KL weights {0.1,0.3}\{0.1, 0.3\}. Baselines use method-specific manual searches covering at least 10−410^{-4}–10−210^{-2} when learning rate is tuned, with approximately ten full runs per method and selection favoring the largest ToW at a practical runtime. SALA’s search budget consists of short probes; baseline tuning uses full unlearning runs.

Supplementary Experiments

Empirical Validation of the Gaussian Approximation

To empirically evaluate the Gaussian approximation of the per-sample reference distributions in Section 3, we examine how closely the scores follow a Gaussian distribution under each membership role. We evaluate log-odds on CIFAR-10 and ImageNet-1000 and per-token cross-entropy on WikiText-103, using the models and training settings in Appendix § 13.

Setup. This experiment uses 64, 32, and 32 models for CIFAR-10, ImageNet-1000, and WikiText-103, respectively. For each sample, we collect its scores across models separately for member and non-member roles, fit a Gaussian using the sample mean and standard deviation, and measure the fit using the Kolmogorov–Smirnov distance

Dgauss=sup⁡z∣Fn(z)−Φ(z−μ^σ^)∣(26)D_{\mathrm{gauss}} = \sup_z \left|F_n(z) - \Phi\left(\frac{z-\hat{\mu}}{\hat{\sigma}}\right)\right| \tag*{(26)}

where FnF_n is the empirical CDF, Φ\Phi is the standard Gaussian CDF, μ^\hat{\mu} is the sample mean, and σ^\hat{\sigma} is the sample standard deviation with one degree-of-freedom correction. A smaller DgaussD_{\mathrm{gauss}} indicates a closer fit. To interpret the distance at finite sample sizes, we also draw observations from a Gaussian distribution and apply the same fitting and distance computation.

Results. Table 3 compares the scores used by SALA with the finite-sample Gaussian references and alternative classification scores (i.e., the probability and the probability margin). It indicates that the log-odds and cross-entropy distances are close to their respective Gaussian references. These results empirically support approximating the per-sample log-odds and cross-entropy distributions as Gaussian.

DatasetScoreRoleMean DgaussD_{\mathrm{gauss}}Median DgaussD_{\mathrm{gauss}}Gaussian reference
CIFAR-10Log-oddsMember0.10980.1059
CIFAR-10Log-oddsNon-member0.11620.1116
CIFAR-10ProbabilityMember0.26460.2606
CIFAR-10ProbabilityNon-member0.29190.28880.1139
CIFAR-10Probability marginMember0.27140.2680
CIFAR-10Probability marginNon-member0.28900.2862
ImageNet-1000Log-oddsMember0.15140.1457
ImageNet-1000Log-oddsNon-member0.15240.1461
ImageNet-1000ProbabilityMember0.23260.2225
ImageNet-1000ProbabilityNon-member0.23670.22690.1651
ImageNet-1000Probability marginMember0.23300.2217
ImageNet-1000Probability marginNon-member0.22820.2157
WikiText-103Cross-entropyMember0.16290.15320.1651
WikiText-103Cross-entropyNon-member0.16900.1554

Table 3. Gaussian-fit evaluation of scalar scores. The last column gives the mean distance for the finite-sample Gaussian reference.

Supplementary experimental results of Section 5.4

To quantify the robustness observed in § 5.4, Table 4 reports the detailed results of 1 − ToW and DNO across the five settings. The SALA candidate pools and selection protocol are given in Appendix § 13. For all three baselines, adding SALA or its variants reduces the standard deviation of both metrics, indicating more stable unlearning quality.

MethodConfiguration1 − ToW95% intervalDNO95% interval
SalUnNo SALA0.2740±0.24090.2740 \pm0.2409[−0.0251,0.5731][-0.0251, 0.5731]0.9120±0.03770.9120 \pm0.0377[0.8652,0.9588][0.8652, 0.9588]
SALA0.0322±0.00330.0322 \pm0.0033[0.0280,0.0364][0.0280, 0.0364]0.1146±0.00760.1146 \pm0.0076[0.1051,0.1241][0.1051, 0.1241]
SALA-SMSE0.0218±0.00290.0218 \pm0.0029[0.0182,0.0254][0.0182, 0.0254]0.2060±0.01190.2060 \pm0.0119[0.1912,0.2208][0.1912, 0.2208]
SALA w/ DfD_f only0.0100±0.00160.0100 \pm0.0016[0.0080,0.0120][0.0080, 0.0120]0.2270±0.01350.2270 \pm0.0135[0.2102,0.2438][0.2102, 0.2438]
SCRUBNo SALA0.3180±0.15610.3180 \pm0.1561[0.1242,0.5118][0.1242, 0.5118]0.6780±0.12460.6780 \pm0.1246[0.5233,0.8327][0.5233, 0.8327]
SALA0.0322±0.00360.0322 \pm0.0036[0.0278,0.0366][0.0278, 0.0366]0.1094±0.00750.1094 \pm0.0075[0.1000,0.1188][0.1000, 0.1188]
SALA-SMSE0.0150±0.00220.0150 \pm0.0022[0.0122,0.0178][0.0122, 0.0178]0.1940±0.01600.1940 \pm0.0160[0.1742,0.2138][0.1742, 0.2138]
SALA w/ DfD_f only0.0080±0.00160.0080 \pm0.0016[0.0060,0.0100][0.0060, 0.0100]0.2290±0.01290.2290 \pm0.0129[0.2129,0.2451][0.2129, 0.2451]
ℓ1\ell_1 sparsityNo SALA0.3020±0.17710.3020 \pm0.1771[0.0821,0.5219][0.0821, 0.5219]0.4920±0.17210.4920 \pm0.1721[0.2783,0.7057][0.2783, 0.7057]
SALA0.0322±0.00300.0322 \pm0.0030[0.0284,0.0360][0.0284, 0.0360]0.1178±0.00730.1178 \pm0.0073[0.1087,0.1269][0.1087, 0.1269]
SALA-SMSE0.0180±0.00220.0180 \pm0.0022[0.0152,0.0208][0.0152, 0.0208]0.2110±0.01190.2110 \pm0.0119[0.1962,0.2258][0.1962, 0.2258]
SALA w/ DfD_f only0.0090±0.00160.0090 \pm0.0016[0.0070,0.0110][0.0070, 0.0110]0.2340±0.01290.2340 \pm0.0129[0.2179,0.2501][0.2179, 0.2501]

Table 4. Robustness statistics for Section 5.4.

Ablation Study of SALA

SALA weights

SALA introduces three weighting hyperparameters: λf\lambda_f (forget guidance weight), λr\lambda_r (retain guidance weight), and α\alpha (extra emphasis on hard-forget samples). In the default pipeline, early stopping is triggered by the InDist threshold, so these weights do not change the stopping criterion itself; instead, they mainly control (i) how quickly InDist reaches the threshold, and (ii) the trade-off between retaining and forgetting during the transient phase before termination. This ablation quantifies how sensitive the final outcome is to these weights and whether SALA remains stable across a broad, practical range.

Setup. A grid search is conducted with λf∈{0.3,1.0,2.0}\lambda_f \in\{0.3, 1.0, 2.0\}, λr∈{0.2,0.5,1.0}\lambda_r \in\{0.2, 0.5, 1.0\}, and α∈{0.0,1.0,2.0}\alpha\in\{0.0, 1.0, 2.0\}. For each α\alpha, a 3×33 \times3 heatmap is plotted over (λf,λr)(\lambda_f, \lambda_r). Three rows are reported: the first row is ToW (higher is better), the second row is DNO (lower is better), and the third row is runtime (lower is better). All other settings follow § 5.1, and results are averaged over the same unlearning task suite.

Results. Figure 6 shows that SALA is stable across a broad range of weighting choices. Across all values of α\alpha, ToW remains consistently high, mostly between 0.97 and 0.99, while DNO stays within a narrow range of approximately 0.11 to 0.14. This indicates that the final unlearning quality is not highly sensitive to moderate changes in λf\lambda_f, λr\lambda_r, or α\alpha once InDist-based stopping is used. The main variation appears in runtime mainly with λf\lambda_f and λr\lambda_r: smaller weights tend to slow convergence to about 75 to 84 seconds, while stronger guidance reduces it to roughly 47 to 58 seconds.

Figure 6. Ablation on SALA weights. Columns correspond to α\alpha. Within each panel, the heatmap axes are λf\lambda_f (y-axis) and λr\lambda_r (x-axis). Rows report ToW, DNO, and runtime, respectively.

Stop threshold and ZcutZ_{\mathrm{cut}} of SALA

We further examine the sensitivity of SALA to the InDist stopping threshold τ\tau and the cutoff ZcutZ_{\mathrm{cut}}. Overall, Table 5 shows that SALA remains stable across the tested stopping-threshold and cutoff settings. For the full SALA variant, DNO is consistently low at 0.12–0.13. The main effect of the sweep is on runtime rather than final unlearning quality. A smaller cutoff, Zcut=1Z_{\mathrm{cut}} = 1, leads to substantially longer runtime, while Zcut=2Z_{\mathrm{cut}} = 2 or 3 reaches the stopping condition much faster. Increasing τ\tau from 0.95 to 0.99 generally makes stopping stricter and increases runtime. The default setting, Zcut=2Z_{\mathrm{cut}} = 2 and τ=0.95\tau= 0.95, achieves a favorable balance, with low DNO, competitive leakage metrics, and short runtime.

MethodZcutZ_{\mathrm{cut}}τ\tauTA(%)FA(%)RA(%)ToW↑\uparrowDNO↓\downarrowMIA efficacy(%)↓\downarrowRULI(%)↓\downarrowTime(s)↓\downarrow
SALA10.9594.73±0.2394.73\pm0.2395.59±0.2095.59\pm0.20100.00±0.00100.00\pm0.000.97±0.010.97\pm0.010.1352.13±0.5752.13\pm0.5752.88±0.7452.88\pm0.74163.92±14.67163.92\pm14.67
SALA10.9994.73±0.2494.73\pm0.2494.69±0.2194.69\pm0.21100.00±0.00100.00\pm0.000.98±0.010.98\pm0.010.1352.09±0.7352.09\pm0.7353.55±0.6853.55\pm0.68209.64±9.61209.64\pm9.61
SALA20.9594.44±0.2894.44\pm0.2895.58±0.2195.58\pm0.21100.00±0.00100.00\pm0.000.97±0.010.97\pm0.010.1252.42±0.1652.42\pm0.1652.15±1.3052.15\pm1.3053.22±3.2153.22\pm3.21
SALA20.9994.59±0.2494.59\pm0.2495.15±0.2395.15\pm0.23100.00±0.00100.00\pm0.000.98±0.010.98\pm0.010.1252.23±0.4952.23\pm0.4952.56±0.5352.56\pm0.53101.37±10.29101.37\pm10.29
SALA30.9594.57±0.2694.57\pm0.2696.33±0.3696.33\pm0.36100.00±0.00100.00\pm0.000.95±0.020.95\pm0.020.1353.52±0.3553.52\pm0.3551.29±0.7251.29\pm0.7247.67±0.5247.67\pm0.52
SALA30.9994.54±0.2794.54\pm0.2795.82±0.4195.82\pm0.41100.00±0.00100.00\pm0.000.96±0.020.96\pm0.020.1353.44±0.6553.44\pm0.6551.36±0.4351.36\pm0.4355.83±7.6055.83\pm7.60
SALA+SMSE10.9594.55±0.2494.55\pm0.2496.06±0.2196.06\pm0.21100.00±0.00100.00\pm0.000.95±0.030.95\pm0.030.2453.34±0.4653.34\pm0.4656.48±0.3656.48\pm0.36204.28±10.30204.28\pm10.30
SALA+SMSE10.9994.56±0.2794.56\pm0.2796.06±0.2096.06\pm0.20100.00±0.00100.00\pm0.000.95±0.030.95\pm0.030.2453.32±0.2553.32\pm0.2556.50±0.7556.50\pm0.75212.64±2.57212.64\pm2.57
SALA+SMSE20.9594.25±0.2794.25\pm0.2795.13±0.1795.13\pm0.1799.97±0.0199.97\pm0.010.98±0.010.98\pm0.010.2154.28±0.3654.28\pm0.3655.97±0.5355.97\pm0.5356.47±5.3556.47\pm5.35
SALA+SMSE20.9994.35±0.2794.35\pm0.2795.63±0.2095.63\pm0.2099.99±0.0099.99\pm0.000.97±0.010.97\pm0.010.2253.50±0.2653.50\pm0.2655.83±0.6355.83\pm0.63135.14±9.56135.14\pm9.56
SALA+SMSE30.9594.37±0.2894.37\pm0.2895.86±0.2795.86\pm0.2799.99±0.0099.99\pm0.000.97±0.010.97\pm0.010.2854.15±0.4654.15\pm0.4653.52±0.7253.52\pm0.7248.52±1.1548.52\pm1.15
SALA+SMSE30.9994.35±0.2694.35\pm0.2695.52±0.2195.52\pm0.2199.99±0.0199.99\pm0.010.97±0.010.97\pm0.010.2553.96±0.6453.96\pm0.6454.88±0.8354.88\pm0.8374.81±5.4274.81\pm5.42
SALA w/ DfD_f only10.9593.96±0.3193.96\pm0.3195.70±0.2495.70\pm0.2499.75±0.1199.75\pm0.110.97±0.010.97\pm0.010.2753.80±0.3653.80\pm0.3656.69±1.1356.69\pm1.13139.11±26.55139.11\pm26.55
SALA w/ DfD_f only10.9993.96±0.2893.96\pm0.2895.69±0.2795.69\pm0.2799.77±0.1099.77\pm0.100.99±0.010.99\pm0.010.2653.78±0.4453.78\pm0.4456.71±0.7256.71\pm0.72135.68±24.88135.68\pm24.88
SALA w/ DfD_f only20.9593.75±0.2993.75\pm0.2994.49±0.3094.49\pm0.3099.75±0.0899.75\pm0.080.99±0.010.99\pm0.010.2353.37±0.3453.37\pm0.3456.75±2.4956.75\pm2.4959.35±6.2659.35\pm6.26
SALA w/ DfD_f only20.9993.90±0.2693.90\pm0.2694.85±0.6394.85\pm0.6399.77±0.1199.77\pm0.110.98±0.010.98\pm0.010.2453.79±0.6553.79\pm0.6556.07±0.4356.07\pm0.43129.29±29.09129.29\pm29.09
SALA w/ DfD_f only30.9593.92±0.2893.92\pm0.2895.37±0.5095.37\pm0.5099.85±0.0699.85\pm0.060.98±0.010.98\pm0.010.2653.77±0.3753.77\pm0.3754.29±0.8254.29\pm0.8274.74±18.0574.74\pm18.05
SALA w/ DfD_f only30.9993.91±0.3193.91\pm0.3195.22±0.4795.22\pm0.4799.80±0.0999.80\pm0.090.98±0.010.98\pm0.010.2453.76±0.4253.76\pm0.4254.98±0.7554.98\pm0.75118.27±25.01118.27\pm25.01

Table 5. Parameter sweep over ZcutZ_{\mathrm{cut}} and inDist threshold τ\tau for SALA variants.

The same trend holds for the relaxed SALA variants. SALA-SMSE remains stable across the sweep, with the default setting giving the lowest DNO among the SMSE configurations. Similarly, the DfD_f-only variant releases the retain-side constraint and therefore produces moderately higher DNO, but it does not cause excessive DNO inflation across the tested settings.

Computational and storage overhead of SMSE

We quantify the computational and storage overhead of estimating the per-sample target distributions, comparing SMSE with the shadow-model estimator evaluated in § 5.2.

Storage overhead. Both estimators store four FP32 statistics per sample, (μin\mu_{\mathrm{in}}, σin\sigma_{\mathrm{in}}, μout\mu_{\mathrm{out}}, σout\sigma_{\mathrm{out}}), adding 16NN bytes for NN samples. Table 6 reports this persistent storage alongside the size of one trained model. During estimation, models or checkpoints can be processed sequentially.

DatasetModelDistribution parameters
CIFAR-1042.700.763
ImageNet-100097.7819.55
WikiText-103201.9417.78

Table 6. Storage of the model and estimated distribution parameters (MiB).

Computational overhead. We measure computational overhead in two stages: training the models, and collecting their output scores to estimate the target-distribution parameters. SMSE obtains the trained model and checkpoint ensemble within the 1×\times training budget in Table 1. Table 7 compares nominal training work and measured time relative to regular training on the same dataset and hardware. SMSE’s training time remains close to regular training, while shadow-model estimation requires training multiple models.

MethodCIFAR-10 CostCIFAR-10 Training timeImageNet-1000 CostImageNet-1000 Training timeWikiText-103 CostWikiText-103 Training time
Regular training1×\times23.12 min1×\times42.50 h1×\times29.09 h
Shadow models32×\times13.21 h (34.28×\times)8×\times307.73 h (7.24×\times)8×\times195.82 h (6.73×\times)
SMSE1×\times20.72 min (0.90×\times)1×\times45.72 h (1.08×\times)1×\times30.58 h (1.05×\times)

Table 7. Training cost of shadow-model estimation and SMSE in § 5.2. Ratios in parentheses are relative to regular training.

After training, evaluating scores from KK models or checkpoints over NN samples requires O(K⌈N/B⌉Cfwd(B))O(K\lceil N/B\rceil C_{\mathrm{fwd}}(B)) work, where BB is the inference batch size and Cfwd(B)C_{\mathrm{fwd}}(B) is the forward-pass cost per batch. Computing the distribution statistics then requires O(NK)O(NK) work. Table 8 separates these two components: forward inference dominates, while the statistical reduction takes seconds. For SMSE, this additional estimation time is 3.59%–13.48% of a regular training run, on top of the training time in Table 7.

DatasetMethodForward inferenceStatisticsTotalTime ratio
CIFAR-10Shadow models51.90 s1.10 s53.00 s0.0382×\times
CIFAR-10SMSE48.70 s1.12 s49.82 s0.0359×\times
ImageNet-1000Shadow models1.72 h7.12 s1.72 h0.0405×\times
ImageNet-1000SMSE5.72 h26.54 s5.73 h0.1348×\times
WikiText-103Shadow models39.45 min6.43 s39.56 min0.0227×\times
WikiText-103SMSE2.19 h24.10 s2.20 h0.0756×\times

Table 8. Time to estimate distribution parameters from the trained models. Ratios are relative to regular training in Table 7.

Ablation Study of SMSE

stabilization of SMSE

SMSE estimates per-sample standard deviations σ^in(x)\hat{\sigma}_{\mathrm{in}}(x) and σ^out(x)\hat{\sigma}_{\mathrm{out}}(x) from a limited ensemble, which can produce overly small or overly dispersed values for a subset of samples. This matters because SALA uses σ^(x)\hat{\sigma}(x) to standardize scores and to form the InDist constraint; extreme σ^(x)\hat{\sigma}(x) values can inflate the effective mismatch term and lead to a systematic DNO offset (§ 4.2). To mitigate this, SMSE applies a sigma stabilization step that shrinks per-sample σ^(x)\hat{\sigma}(x) toward a global reference scale and clips it into a bounded range, preventing collapse and heavy tails. The ablation compares SALA-SMSE with and without sigma stabilization, keeping all other components fixed. Results are reported under the same unlearning task suite as the main experiments.

Results. Table 9 shows that sigma stabilization consistently improves sample-level alignment: it reduces DNO by about 0.01 and RULI accuracy by about 0.49% on average, with negligible change to ToW. This supports the interpretation in § 4.2: bounding σ^(x)\hat{\sigma}(x) directly reduces the estimation-induced term in the DNO decomposition and tightens the achievable DNO under SALA-SMSE.

MethodSigma StableTA(%)↑FA(%)↑RA(%)↑ToW↑DNO↓MIA efficacy(%)↓RULI(%)↓Time(s)↓
SALA-SMSE + Salun✓94.25±0.2795.13±0.1799.97±0.010.98±0.010.2154.28±0.3655.97±0.5356.47±5.35
SALA-SMSE + SCRUB✓94.01±0.2394.03±0.1699.97±0.020.98±0.010.2354.43±0.1655.25±0.2746.45±2.79
SALA-SMSE + ℓ1\ell_1✓93.13±0.3193.52±0.4299.99±0.000.99±0.010.2053.25±0.2752.93±0.31125.52±9.14
SALA-SMSE + Salun✗93.80±0.1994.74±0.22100.00±0.000.97±0.010.2354.74±0.2556.00±0.3176.73±7.15
SALA-SMSE + SCRUB✗93.76±0.2194.43±0.3599.97±0.020.97±0.000.2354.76±0.1956.47±0.2144.35±2.21
SALA-SMSE + ℓ1\ell_1 sparsity✗93.92±0.2794.14±0.3199.99±0.010.97±0.010.2354.41±0.0853.16±0.13108.70±9.15

Table 9. Ablation study of SMSE sigma stabilization mechanism

Epoch allocation of SMSE

We further study how the epoch allocation in SMSE affects per-sample distribution estimation. In the main experiments, the training budget is divided into 20% warmup, 70% half-data training, and 10% fine-tuning after merging. To examine the sensitivity of this schedule, we vary the warmup and merge allocations. Estimation quality is evaluated using the same metrics as in Table 1, including Pearson correlation, MSE for the four estimated moments and DNOest\mathrm{DNO}_{\mathrm{est}}. We also report the final test accuracy of each trained model.

Table 10 reports the results on CIFAR-10. Warmup mainly improves the calibration of per-sample moment estimates. Without warmup, SMSE obtains the worst DNOest\mathrm{DNO}_{\mathrm{est}} of 0.2464 and lower Pearson correlation for μin\mu_{\mathrm{in}}, although the final accuracy remains high. Introducing a moderate warmup substantially improves estimation quality: 10% warmup reduces DNOest\mathrm{DNO}_{\mathrm{est}} to 0.1999, while the main 20% warmup setting further reduces it to 0.1801.

Warmup Epoch Ratio (%)Merge Epoch Ratio (%)μin\mu_{\mathrm{in}} r↑r\uparrowμin\mu_{\mathrm{in}} MSE↓μout\mu_{\mathrm{out}} r↑r\uparrowμout\mu_{\mathrm{out}} MSE↓σin\sigma_{\mathrm{in}} r↑r\uparrowσin\sigma_{\mathrm{in}} MSE↓σout\sigma_{\mathrm{out}} r↑r\uparrowσout\sigma_{\mathrm{out}} MSE↓DNOest\mathrm{DNO}_{\mathrm{est}} ↓Final Acc.↑
0100.751.570.902.340.590.420.700.400.246494.87
10100.791.270.911.950.560.700.600.570.199994.49
2010 (Main setup)0.791.290.921.900.580.670.630.520.180194.79
20200.790.910.922.070.601.270.650.790.182294.78
2000.740.970.921.900.580.670.640.530.183283.59

Table 10. Ablation on warmup and merge scheduling in SMSE on CIFAR-10.

The merge stage has a different role. With the warmup ratio fixed at 20%, removing fine-tuning after merging sharply reduces final accuracy from 94.79% to 83.59%, while leaving the estimation metrics relatively competitive. Thus, merging is primarily needed to recover the predictive utility of the final model rather than to improve moment estimation alone. Overall, the main schedule with 20% warmup and 10% merge achieves the lowest DNOest\mathrm{DNO}_{\mathrm{est}} while maintaining high final accuracy, and is therefore used as the default SMSE configuration.

Segment ensemble size MM of SMSE

We further ablate the segment ensemble size MM, defined as the number of checkpoints per half-data training trajectory used to estimate per-sample score moments. In the main experiments, we set M=30M = 30. To examine sensitivity to this choice, we vary M∈{10,20,30,40}M \in\{10, 20, 30, 40\} and evaluate estimation quality using the same metrics as in Table 1.

Table 11 reports the results on CIFAR-10. Increasing MM from 10 generally improves estimation stability, particularly for the variance terms: the MSE of σout\sigma_{\mathrm{out}} decreases from 2.69 at M=10M = 10 to 0.52 at M=30M = 30. However, using a larger ensemble does not uniformly improve all metrics. Although M=40M = 40 achieves the best correlation for σin\sigma_{\mathrm{in}}, it worsens the MSE of all four moments compared with M=30M = 30. The main setting M=30M = 30 therefore provides the best overall balance, achieving the strongest mean estimation for μout\mu_{\mathrm{out}} and the lowest MSE for both μout\mu_{\mathrm{out}} and σout\sigma_{\mathrm{out}}.

MMμin\mu_{\mathrm{in}}μout\mu_{\mathrm{out}}σin\sigma_{\mathrm{in}}σout\sigma_{\mathrm{out}}
r↑r \uparrowMSE↓\mathrm{MSE} \downarrowr↑r \uparrowMSE↓\mathrm{MSE} \downarrowr↑r \uparrowMSE↓\mathrm{MSE} \downarrowr↑r \uparrowMSE↓\mathrm{MSE} \downarrow
100.771.300.892.470.531.060.732.69
200.781.190.902.240.570.420.701.38
30 (Main setup)0.791.290.921.900.580.670.630.52
400.771.700.922.080.600.850.650.58

Table 11. Ablation on the segment ensemble size MM in SMSE on CIFAR-10.

Estimation methods of SMSE

We examine alternative ways of constructing the stochastic ensemble used by SMSE for moment estimation. The default design uses a checkpoint ensemble, where checkpoints collected along the two half-data training trajectories are aggregated to estimate per-sample score moments. We compare this strategy with two lightweight alternatives that induce stochasticity from a single trained model, namely Monte Carlo dropout and repeated stochastic data augmentation.

Table 12 reports the results on CIFAR-10. The checkpoint ensemble provides the most balanced estimation quality across the four target moments. It achieves the strongest correlations for both mean statistics, with r=0.79r = 0.79 for μin\mu_{\mathrm{in}} and r=0.92r = 0.92 for μout\mu_{\mathrm{out}}, and also attains the lowest MSE for μout\mu_{\mathrm{out}} and σout\sigma_{\mathrm{out}}. Data augmentation produces competitive variance correlations, but its mean estimation is weaker and its σout\sigma_{\mathrm{out}} MSE remains higher than that of the checkpoint ensemble. Monte Carlo dropout achieves low MSE on some individual terms, but yields negative correlations for both σin\sigma_{\mathrm{in}} and σout\sigma_{\mathrm{out}}, indicating that its induced randomness does not preserve the variance structure required by SMSE.

Methodμin\mu_{\mathrm{in}}μout\mu_{\mathrm{out}}σin\sigma_{\mathrm{in}}σout\sigma_{\mathrm{out}}
r↑r \uparrowMSE↓\mathrm{MSE} \downarrowr↑r \uparrowMSE↓\mathrm{MSE} \downarrowr↑r \uparrowMSE↓\mathrm{MSE} \downarrowr↑r \uparrowMSE↓\mathrm{MSE} \downarrow
Checkpoint ensemble (Main setup)0.791.290.921.900.580.670.640.52
Dropout0.721.180.892.55-0.150.35-0.421.84
Data augmentation0.711.550.912.480.610.410.761.00

Table 12. Ablation on different stochastic estimation methods in SMSE on CIFAR-10.

Utility of Models Produced by SMSE

Experiments of SALA combined with various approximate unlearning algorithms

Results. Table 14 evaluates whether SALA can act as a stable plug-in mechanism for different approximate unlearning algorithms. Across SALUn, SCRUB, and ℓ1\ell_1 sparsity unlearning, SALA consistently improves the corresponding base unlearning methods: DNO drops by 0.78, 0.23, and 0.21, respectively, while RULI is reduced by 40.23%, 16.55%, and 17.98%. SALA also maintains nearly perfect ToW across all three base methods and keeps runtime far below full retraining.

MethodTA(%)↑FA(%)↑RA(%)↑ToW↑DNO↓MIA efficacy(%)↓RULI(%)↓Time(s)↓
Retrain93.84±0.2193.68±0.28100.00±0.011.00±0.000.0750.57±0.2550.00±0.001625.53±5.33
SalUn92.36±0.4595.29±0.5798.61±0.360.95±0.010.9051.87±0.2192.38±0.28221.92±96.37
SCRUB90.29±2.3194.06±3.1595.74±3.000.92±0.030.3452.23±0.5768.67±3.62101.93±11.98
ℓ1\ell_1 sparsity88.18±3.4791.94±4.4895.80±2.980.88±0.020.3253.09±2.7270.03±5.24150.28±28.45
SALA + SalUn94.44±0.2895.58±0.21100.00±0.000.97±0.020.1252.42±0.1652.15±1.3053.22±3.21
SALA + SCRUB94.35±0.1594.03±0.1499.99±0.000.99±0.010.1152.46±0.1152.12±0.0750.31±6.62
SALA + ℓ1\ell_194.31±0.1994.75±0.2799.99±0.010.98±0.010.1151.23±0.0252.05±0.1497.36±4.14
SALA-SMSE + SalUn94.25±0.2795.13±0.1799.97±0.010.98±0.010.2154.28±0.3655.97±0.5356.47±5.35
SALA-SMSE + SCRUB95.01±0.2394.03±0.1699.97±0.020.98±0.010.2354.43±0.1655.25±0.2746.45±2.79
SALA-SMSE + ℓ1\ell_193.13±0.3193.52±0.4299.99±0.000.99±0.010.2053.25±0.2752.93±0.31125.52±9.14
SALA w/ DfD_f only + SalUn93.75±0.2994.49±0.3099.75±0.080.99±0.010.2353.37±0.3456.75±2.4959.35±6.26
SALA w/ DfD_f only + SCRUB93.41±0.1693.89±0.3499.97±0.020.99±0.010.2353.43±0.5355.25±0.6451.45±5.24
SALA w/ DfD_f only + ℓ1\ell_193.13±0.5393.52±0.4999.99±0.000.99±0.010.2251.25±0.1553.03±0.28115.52±6.35

Table 14. Comprehensive evaluation of SALA-assisted approximate unlearning methods.

When using only DfD_f, DNO remains within 0.22 to 0.23, which is still lower than the corresponding base unlearning methods. Similarly, although SALA-SMSE uses an imperfect reference distribution with additional estimation error, its DNO remains within 0.20 to 0.23 and consistently improves over the baselines.

Sensitivity to the forget-set size

The main sample-wise and sequence unlearning experiments use forget ratios of 20% on CIFAR-10 and 33% on ImageNet-1000 and WikiText-103, providing observations in both membership roles over repeated tasks. We additionally evaluate 5%, 10%, and 50% on CIFAR-10, keeping the other settings fixed and using eight independent unlearning runs for each ratio. Table 15 reports the same utility, efficacy, privacy, and runtime metrics as Table 2. Across the three ratios, SALA and its variants obtain lower DNO and RULI than SalUn, with comparable or higher ToW.

MethodTA(%)FA(%)RA(%)ToWDNOMIA(%)RULI(%)Time(s)
Forget ratio: 5%
Retrain94.65±0.3794.33±0.21100.00±0.001.00±0.000.0651.15±0.0850.23±0.071457.39±10.74
SalUn94.45±0.0895.72±0.14100.00±0.000.96±0.010.5653.15±0.1474.42±0.5794.45±5.47
SALA94.10±0.2493.90±0.13100.00±0.000.99±0.000.0951.03±0.1152.14±0.2938.49±1.52
SALA-SMSE94.05±0.1394.21±0.18100.00±0.000.97±0.010.2452.14±0.1259.15±0.7534.08±2.47
SALA w/ DfD_f only93.91±0.1594.33±0.16100.00±0.000.97±0.010.2552.28±0.1459.87±0.6835.36±2.21
Forget ratio: 10%
Retrain94.12±0.1294.02±0.25100.00±0.001.00±0.000.0451.03±0.0950.10±0.351572.27±6.35
SalUn94.41±0.2594.43±0.11100.00±0.000.96±0.010.7654.35±0.0582.79±0.28115.72±19.15
SALA94.20±0.1593.78±0.47100.00±0.000.98±0.010.1050.79±0.3551.95±0.5136.20±1.13
SALA-SMSE93.95±0.1293.92±0.47100.00±0.000.98±0.000.2552.75±0.4158.37±1.5433.79±1.63
SALA w/ DfD_f only94.08±0.1494.26±0.39100.00±0.000.98±0.010.2652.52±0.3657.91±1.4234.27±1.81
Forget ratio: 50%
Retrain92.68±0.2592.53±0.18100.00±0.001.00±0.000.0751.10±0.0550.25±0.111625.53±5.33
SalUn89.88±0.4892.40±0.5498.05±0.380.95±0.000.9152.35±0.1094.30±2.22221.92±96.37
SALA93.71±0.2394.25±0.3099.99±0.010.97±0.010.1150.27±0.0452.34±0.3553.22±3.21
SALA-SMSE93.48±0.2093.49±0.2499.98±0.020.97±0.010.2452.27±0.1359.96±0.2263.90±3.59
SALA w/ DfD_f only93.62±0.1893.31±0.27100.00±0.000.96±0.010.2452.44±0.1559.71±0.2562.84±3.21

Table 15. CIFAR-10 results under different forget ratios.

SALA on Class-wise Unlearning

Main experiments focus on sample-wise unlearning (classification) and sequence unlearning (generation). This subsection extends SALA to class-wise unlearning to test whether per-sample distributional targets remain effective when the forget set is structured and highly correlated. Class-wise unlearning is evaluated on the classification benchmarks by selecting one target class as the forget set DfD_f and treating all remaining classes as the retain set DrD_r. SALA is applied on top of iterative baselines in the same way as in Section 5.3: forget-class samples are driven toward their Qout(⋅∣x)Q_{\mathrm{out}}(\cdot\mid x) targets, and retain-class samples are driven toward Qin(⋅∣x)Q_{\mathrm{in}}(\cdot\mid x) targets, with the same InDist-guided stopping rule. For class-wise unlearning, exact retraining yields FA = 0.00%, making the original ToW undefined if FA is included. We therefore compute ToW using only TA and RA, and report FA separately as the forgetting measure on the target class.

Why class-wise is different. Compared to random sample-wise deletion, class-wise deletion changes the distribution of the remaining data and induces a larger representation shift: many samples share class-level features, so forgetting one class can affect model outputs on neighboring classes. This makes dataset-level scores alone especially unreliable, and stresses whether SALA can maintain retain behavior while enforcing strong non-membership behavior on a coherent subset.

Results. As shown in Table 16, SALA remains compatible with class-wise unlearning and preserves the same qualitative benefits: it improves sample-level alignment on the forgotten class (lower DNO and lower RULI accuracy) while keeping ToW competitive.

MethodTA(%)↑FA(%)↑RA(%)↑ToW↑DNO↓MIA efficacy(%)↓RULI(%)↓Time(s)↓
Retrain91.87±2.090.00±0.00100.00±0.001.00±0.000.0350.62±0.2950.15±0.081627.24±12.10
Salun95.79±1.020.02±0.0999.78±0.130.96±0.010.4551.15±0.0977.90±2.8923.43±0.20
SCRUB94.43±1.150.00±0.0097.42±0.160.95±0.010.3951.36±0.0569.49±1.1473.39±7.47
ℓ1\ell_1 sparsity90.57±1.690.00±0.0099.92±0.040.99±0.000.2750.35±0.0466.45±2.1292.92±4.21
SALA + Salun91.31±1.720.00±0.0099.98±0.010.99±0.000.0650.41±0.1453.25±0.1662.47±3.35
SALA + SCRUB91.16±2.150.00±0.0099.99±0.000.99±0.010.0950.92±0.2552.76±0.2464.35±2.15
SALA + ℓ1\ell_1 sparsity90.69±1.720.00±0.00100.00±0.000.99±0.000.1150.34±0.1052.15±0.1674.57±6.41
SALA-SMSE + Salun91.35±1.430.00±0.00100.00±0.000.99±0.000.2750.54±0.1660.43±0.6569.62±4.47
SALA-SMSE + SCRUB91.15±3.430.00±0.00100.00±0.000.99±0.000.2550.35±0.2162.24±0.1666.48±2.26
SALA-SMSE + ℓ1\ell_1 sparsity92.23±1.470.00±0.0099.99±0.000.99±0.010.2750.31±0.1964.45±1.3170.92±4.43

Table 16. Comprehensive comparison of approximate unlearning methods in class-wise unlearning scenario.

Repeated deletion requests

To evaluate unlearning quality and cumulative cost when target distributions are reused across deletion requests, we compare Retrain, SALA, and SALA-SMSE over eight sequential requests on CIFAR-10. Each request removes 1,000 samples, totaling 8,000 deletions. SALA and SALA-SMSE estimate their target distributions once and reuse them across requests; Retrain trains on the remaining retain set after each request.

Cumulative cost. After the common initial model training, Table 17 counts target estimation at K=0K = 0 and adds the time to process each completed request. Shadow-model estimation dominates SALA’s cumulative time, which remains above repeated retraining after eight requests. With SMSE, the cumulative time for eight requests is 461.09 s, compared with 9,854.16 s for retraining.

MethodK=0K = 0K=1K = 1K=2K = 2K=3K = 3K=4K = 4K=5K = 5K=6K = 6K=7K = 7K=8K = 8
Retrain0.001,264.782,540.883,757.284,930.746,073.327,185.728,274.249,854.16
SALA46,211.8046,418.7146,618.7946,819.7747,020.8547,188.4047,369.0747,546.7747,715.27
SALA-SMSE49.82110.61166.30222.20281.67329.76376.10420.77461.09

Table 17. Cumulative time (seconds) over repeated deletion requests. K=0K = 0 records the target-estimation cost.

Unlearning quality. Table 18 compares performance after the first and eighth requests. After eight requests, both variants maintain ToW near 0.965 and retain accuracy above 99.5%. SALA achieves lower DNO, while SALA-SMSE maintains comparable ToW with substantially lower cumulative cost.

KKMethodTA(%)FA(%)RA(%)ToWDNORULI(%)
1Retrain94.57±0.1994.45±0.72100.00±0.0011.0000±0.00000.0000±0.000050.00±0.00
1SALA94.44±0.2694.01±0.9899.88±0.020.9871±0.00570.1725±0.064156.72±4.13
1SALA-SMSE94.34±0.2994.54±0.3999.96±0.010.9917±0.00370.3469±0.085059.44±7.62
8Retrain93.70±0.2593.77±0.37100.00±0.001.0000±0.00000.0000±0.000050.00±0.00
8SALA92.57±1.5194.73±1.6399.55±0.430.9643±0.01440.1486±0.079156.93±1.42
8SALA-SMSE94.33±0.2096.40±0.2199.97±0.010.9650±0.00520.2854±0.097557.38±0.99

Table 18. CIFAR-10 results after the first and eighth deletion requests.

Limitations and Broader Impact

Limitations. The main limitation of SALA is its dependence on per-sample target distributions, namely Qin(⋅∣x)Q_{\mathrm{in}}(\cdot\mid x) and Qout(⋅∣x)Q_{\mathrm{out}}(\cdot\mid x). When memorization statistics or shadow-model estimates have already been collected for a task or dataset, SALA can directly reuse them. Otherwise, these target distributions must be obtained through the proposed SMSE procedure, which requires a split–merge training schedule and checkpoint ensembles during model development. Thus, SALA imposes requirements on the training pipeline and is less convenient for models that have already been trained without saving the necessary statistics, though SMSE reduces this preparation cost to roughly one structured training run.

Broader impact. This additional training-time requirement is worthwhile because it enables approximate unlearning to be monitored and controlled at the individual-sample level. SALA helps reveal and reduce sample-level gap-to-retrain that can be hidden by dataset-level averaged metrics, thereby reducing privacy leakage after unlearning. More broadly, this work supports more reliable data deletion workflows under right-to-be-forgotten requirements. We expect SALA to help practitioners move from average-case unlearning evaluation towards privacy-preserving individuals and verifiable unlearning systems.

NeurIPS Paper Checklist

Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction explicitly state the paper’s main claims: SALA is a sample-grained approximate unlearning framework, DNO measures per-sample gap-to-retrain, SALA provides a computable upper bound on this gap, and SMSE estimates the required per-sample target distributions.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations in Appendix G.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper states the assumptions for each theoretical result, including the per-sample Gaussian score modeling, the InDist constraint, the scale feasibility condition, the retain-consistency condition for the forget-set-only variant, and the estimation-error decomposition. Theorem 3.1, Corollary 3.2, and Corollary 4.1 are accompanied by complete proofs or derivations in Appendices § 10, § 11 and § 12.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper discloses the datasets, models, unlearning scenarios, metrics, number of unlearning tasks, baseline coverage, and SALA/SMSE variants used to obtain the main experimental results. A complete code repository is provided with the implementation, configurations, and reproduction scripts needed to reproduce the main experiments.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example, (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open-source code at the repository linked in Section 5.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: The main text in § 5.1 describes the experimental setup, including datasets, models, unlearning scenarios, metrics, and evaluated variants, while Appendix § 13 provides detailed configurations.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report std values in the experimental tables, and Appendix § 13 states that these values denote one standard deviation over independent unlearning tasks and random seeds. The same paragraph also specifies the number of independent tasks used for DNO computation on CIFAR-10, ImageNet-1000, and WikiText-103.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix § 13, Implementation, reports the compute resources used for the experiments, including the GPU configurations for CIFAR-10, ImageNet-1000, and WikiText-103. It also states that all reported runtimes are measured under these hardware settings.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: The research conforms to the NeurIPS Code of Ethics. It uses standard public benchmarks, does not involve human subjects or private user data, and is intended to improve the evaluation and privacy protection of machine unlearning methods.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss broader impacts in Appendix § 15.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: [N/A]

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use standard existing assets, including public benchmark datasets, model architectures, and baseline methods, and properly credit their original creators through citations in the paper. The released repository further documents the assets used in the experiments and follows their corresponding terms of use.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper introduces new code assets for SALA and SMSE.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: [N/A]

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: [N/A]

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: [N/A]

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]Jacob L. Block, Aryan Mokhtari, and Sanjay Shakkottai. Machine unlearning under over-parameterization. CoRR, abs/2505.22601, 2025. doi: 10.48550/ARXIV.2505.22601. URL https://doi.org/10.48550/arXiv.2505.22601.
  2. [2]Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP), pages 141–159. IEEE, 2021.
  3. [3]Jonathan Brophy, Zayd Hammoudeh, and Daniel Lowd. Adapting and evaluating influence-estimation methods for gradient-boosted decision trees. Journal of Machine Learning Research, 24(154):1–48, 2023.arxiv.org/abs/2205.00359
  4. [4]Xavier F Cadet, Anastasia Borovykh, Mohammad Malekzadeh, Sara Ahmadi-Abhari, and Hamed Haddadi. Deep unlearn: Benchmarking machine unlearning. arXiv preprint arXiv:2410.01276, 2024.arxiv.org/abs/2410.01276
  5. [5]Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE, 2015.DOI
  6. [6]Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), pages 1897–1914. IEEE, 2022.arxiv.org/abs/2112.03570
  7. [7]Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li. Certified machine unlearning via noisy stochastic gradient descent. Advances in Neural Information Processing Systems, 37:38852–38887, 2024.arxiv.org/abs/2403.17105
  8. [8]Vikram S. Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan S. Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 7210–7217. AAAI Press, 2023. doi: 10.1609/AAAI.V37I6.25879.DOI
  9. [9]Erik Daxberger, Augustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux – effortless bayesian deep learning. In Advances in Neural Information Processing Systems, volume 34, 2021.arxiv.org/abs/2106.14806
  10. [10]Taha Entesari, Arman Hatami, Rinat Khaziev, Anil Ramakrishna, and Mahyar Fazlyab. Constrained entropic unlearning: A primal-dual framework for large language models. arXiv preprint arXiv:2506.05314, 2025.arxiv.org/abs/2506.05314
  11. [11]European Parliament and Council of the European Union. Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016. Official Journal of the European Union, 2016.
  12. [12]Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.12508
  13. [13]Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in neural information processing systems, 33: 2881–2891, 2020.arxiv.org/abs/2008.03703
  14. [14]Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1907.05012
  15. [15]Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9301–9309. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVPR42600.2020.00932.
  16. [16]Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11516–11524, 2021.arxiv.org/abs/2010.10981
  17. [17]Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3832–3842, 2020.arxiv.org/abs/1911.03030
  18. [18]Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 497–519. IEEE, 2025.arxiv.org/abs/2403.01218
  19. [19]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.90.DOI
  20. [20]Zhehao Huang, Xinwen Cheng, JingHao Zheng, Haoran Wang, Zhengbao He, Tao Li, and Xiaolin Huang. Unified gradient-based machine unlearning with remain geometry enhancement. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.arxiv.org/abs/2409.19732
  21. [21]Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.arxiv.org/abs/2304.04934
  22. [22]Thomas Kailath. The divergence and bhattacharyya distance measures in signal selection. IEEE transactions on communication technology, 15(1):52–60, 1967.DOI
  23. [23]Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends in machine learning, 14(1–2):1–210, 2021.arxiv.org/abs/1912.04977
  24. [24]Anastasia Koloskova, Youssef Allouah, Animesh Jha, Rachid Guerraoui, and Sanmi Koyejo. Certified unlearning for neural networks. arXiv preprint arXiv:2506.06985, 2025.arxiv.org/abs/2506.06985
  25. [25]Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
  26. [26]Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances in neural information processing systems, 36:1957–1987, 2023.
  27. [27]Xiang Li, Wenqi Wei, and Bhavani Thuraisingham. Mubox: A critical evaluation framework of deep machine unlearning [systematization of knowledge paper]. In Proceedings of the 30th ACM Symposium on Access Control Models and Technologies, pages 175–188, 2025.DOI
  28. [28]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.arxiv.org/abs/1609.07843
  29. [29]Siqiao Mu and Diego Klabjan. Rewind-to-delete: Certified machine unlearning for nonconvex functions. arXiv preprint arXiv:2409.09778, 2024.arxiv.org/abs/2409.09778
  30. [30]Nima Naderloui, Shenao Yan, Binghui Wang, Jie Fu, Wendy Hui Wang, Weiran Liu, and Yuan Hong. Rectifying privacy and efficacy measurements in machine unlearning: A new inference attack perspective. arXiv preprint arXiv:2506.13009, 2025.arxiv.org/abs/2506.13009
  31. [31]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  32. [32]Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.
  33. [33]Marcin Sendera, Lukasz Struski, Kamil Ksiazek, Kryspin Musiol, Jacek Tabor, and Dawid Damian Rymarczyk. SEMU: singular value decomposition for efficient machine unlearning. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. URL https://proceedings.mlr.press/v267/sendera25a.html.arxiv.org/abs/2502.07587
  34. [34]Nexhi Sula, Abhinav Kumar, Jie Hou, Han Wang, and Reza Tourani. Silver linings in the shadows: Harnessing membership inference for machine unlearning. arXiv preprint arXiv:2407.00866, 2024.arxiv.org/abs/2407.00866
  35. [35]Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 35(9):13046–13055, 2023.DOI
  36. [36]Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303–319. IEEE, 2022.arxiv.org/abs/2109.13398
  37. [37]Jing Wu and Mehrtash Harandi. Scissorhands: Scrub data influence via connection sensitivity in networks. In European Conference on Computer Vision, pages 367–384. Springer, 2024.arxiv.org/abs/2401.06187
  38. [38]Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. A closer look at machine unlearning for large language models. arXiv preprint arXiv:2410.08109, 2024.arxiv.org/abs/2410.08109
  39. [39]Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868, 2024.arxiv.org/abs/2404.05868
  40. [40]Kairan Zhao, Meghdad Kurmanji, George-Octavian Barbulescu, Eleni Triantafillou, and Peter Triantafillou. What makes unlearning hard and what to do about it. In Amir Globerson, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.

Paper details

vlnc:W_gf43hez4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents