betaFeedback

WorksComputer Science

Unlearning by Extrapolating a Shrink-and-Heal Lookahead

Abstract

Unlearning aims to make a large language model behave like one retrained without its forget set. Prevailing likelihood-suppression objectives, however, lack an explicit forgetting target: they continually reward lower forget-set likelihood, making forgetting sensitive to the optimization budget. We address this limitation by estimating the direction toward the oracle without training it. Our key idea is to shrink the fine-tuned model toward its pre-fine-tuning weights and briefly re-learn the retain set, thereby removing the forget-specific contribution; in the linearized regime, this shrink-and-heal lookahead moves the model a fixed fraction of the way to the retrained model, its unique fixed point. Our method, SHIFT, extrapolates the lookahead's per-token changes into distillation targets and iteratively refines them. Evaluated against eleven baselines under a unified OpenUnlearning protocol, SHIFT achieves the highest aggregate score across all TOFU settings and the closest oracle alignment, 3.7x closer on average than the next-best method; on MUSE, it preserves the most retain knowledge. Under larger optimization budgets, baselines overshoot the oracle, while SHIFT remains near it, even under relearning attacks.

Introduction

Large language models (LLMs) acquire substantial knowledge through fine-tuning, but may also memorize sensitive or unwanted training data [2]. When such data must be removed, retraining from scratch is often prohibitively expensive. This motivates machine unlearning, which aims to remove the influence of a forget set while preserving what the model learned from the remaining retain set [27]. Ideally, the unlearned model should behave like an oracle trained on the retain set alone. This oracle serves as the reference for evaluating unlearning in established benchmarks [29, 38], and has been advocated as the defining criterion for unlearning [49].

Most existing unlearning methods are built on gradient ascent over the forget set [20, 46], often combined with retain-set regularization to preserve model utility [26]. Subsequent work bounds or reshapes the ascent objective to prevent model collapse, such as NPO [52], SimNPO [11], and token reweighting [40, 45], extends the suppression to the model’s own high-confidence alternatives [24], or replaces it with a distillation target [4, 9, 7, 54].

Despite these refinements, a fundamental yet underexplored limitation remains in likelihood-suppression approaches: they lack an explicit target for forgetting. By rewarding lower forget-set likelihood, these objectives do not generally admit the oracle as a stationary point, allowing optimization to proceed beyond the desired forgetting level. Consequently, the extent of forgetting becomes sensitive to the learning rate and number of training steps. As illustrated in Figure 1(a), varying the optimization budget can shift the same method from insufficient forgetting to over-forgetting, where forget-set examples become less familiar than never-seen data, potentially exposing their former membership [16]. This sensitivity is not merely an optimization issue: the objective itself provides no criterion for stopping at the oracle.

Privacy leakage across learning rates and the shrink-and-heal lookahead

Figure 1. (a) Without a target, forgetting follows the budget. (b) The shrink-and-heal lookahead points to the oracle. Figure 1: (a) Privacy leakage on TOFU forget05 (Llama-3.2-1B) across learning rates under a fixed 10-epoch budget (0: oracle; positive: the forget set looks less familiar than never-seen data). Likelihood-suppression methods overshoot the oracle, whereas SHIFT stabilizes near it. (b) In the linearized regime (Prop. 1), shrinking θ0\theta_0 toward the base θb\theta_b and healing on the retain set removes a fraction ρ\rho of the forget-specific contribution; SHIFT extrapolates and iterates this step, which stops at the oracle θr\theta_r, whereas likelihood suppression has no reason to stop there.

An explicit target could address this limitation, and the best target is the oracle itself, which is exactly what one cannot afford to train. Our key observation is that a fine-tuned model already contains a cheap estimate of where the oracle lies. Specifically, shrinking the model toward its pre-fine-tuning weights partially reverses everything learned during fine-tuning, while briefly re-learning the retain set restores knowledge supported by the retained data; what remains removed is the contribution attributable to the forget set. We call this operation the shrink-and-heal lookahead. As shown in Figure 1(b), under linearized fine-tuning dynamics, it moves the model a fixed fraction of the way toward the oracle, which is its unique fixed point.

Building on this observation, we propose SHIFT (Shrink–heal Iteration toward the Fixed poinT), an unlearning framework that uses the lookahead to guide prediction-level updates. Rather than adopting the lookahead model directly, SHIFT measures its induced changes in forget-set predictions and extrapolates them to estimate an oracle-directed update. The resulting target is then distilled into the model on forget tokens, while its retain-set predictions are anchored to preserve utility. By repeatedly recomputing the lookahead from the updated model, SHIFT progressively corrects its approximation to the oracle through a Newton-style iteration.

We evaluate SHIFT on TOFU ([29]) and MUSE ([38]) against eleven baselines under a unified protocol. Across the TOFU settings, SHIFT achieves the closest forget-set statistics to the oracle, 3.7×3.7\times closer on average than the next-best method. It also obtains the highest aggregate score among methods that avoid over-forgetting. On MUSE, SHIFT preserves the most retain knowledge, while removing more memorization than NPO and SimNPO. Furthermore, SHIFT maintains forgetting behavior close to the oracle across different learning rates, demonstrating reduced sensitivity to the optimization budget.

Contributions. (1) We identify a fundamental limitation of likelihood-suppression unlearning: without an explicit target, its forgetting behavior depends heavily on the optimization budget (§ 3). (2) We introduce the shrink-and-heal lookahead and prove that, in the linearized regime, it steps a fixed fraction toward the oracle, which is its unique fixed point, and that distilling its extrapolated per-token change yields a Newton-style update toward the oracle (§ 4). (3) We develop SHIFT and demonstrate on TOFU and MUSE that it achieves the strongest oracle alignment and the best overall performance among methods that avoid over-forgetting (§ 5).

Related Work

Likelihood suppression. Early unlearning methods use gradient ascent on the forget set ([20, 46]), often combined with retain-set regularization ([26]). Subsequent approaches introduce broader or reweighted objectives ([52, 11, 40, 45]), refusal targets ([29]), representation misdirection ([25]), and constrained optimization ([10]). Recent work also explores adaptive stopping criteria based on the model’s own predictions ([6]). Despite their different mechanisms, they do not explicitly target the behavior of the retain-only oracle, which provides a principled reference for unlearning ([49, 16]).

Targets and distillation. Distillation-based methods introduce alternative targets to guide forgetting. UN-DIAL reduces the gold-token logit by a fixed offset [7], while “Who’s Harry Potter” subtracts the excess of a model reinforced on the forget data [9]. Other approaches construct teachers using incompetent models [4], in-context prompts [54], or assistants trained to memorize the forget set [21]. In contrast, SHIFT’s teacher is calibrated by the retain set and has the oracle as its fixed point.

Checkpoint arithmetic and resets. The closest related approach is MSA [36], which fine-tunes an earlier checkpoint on the forget set and subtracts the resulting parameter difference scaled by a tuned coefficient. SHIFT instead estimates the unlearning direction through shrinkage and retain-set recovery, without training on the forget set, and derives its extrapolation scale from the oracle fixed-point property. Related techniques, such as shrink-and-perturb [1] and rewinding [32] reset model weights, whereas SHIFT uses the reset to construct distillation targets. Unlike the Lookahead optimizer [51], which pulls weights toward a few fast steps, our lookahead is a model-level shrink-and-heal operation rather than an optimizer (see Appendix § 7 for further discussion).

Preliminaries and Motivation

Problem definition and evaluation. A base model θb\theta_b is fine-tuned on D=Df∪DrD = D_f \cup D_r, producing the target model θ0\theta_0, where DfD_f and DrD_r denote the forget and retain sets, respectively. Given θ0\theta_0, DfD_f, and DrD_r, an unlearning method aims to obtain a model θ\theta that behaves like the oracle θr\theta_r, defined as θb\theta_b fine-tuned on DrD_r alone. The oracle is unavailable during unlearning. For a sequence xx, let ℓθ(⋅∣x<t)\ell_\theta(\cdot\mid x_{<t}) denote the next-token log-probabilities at position tt. Most likelihood-suppression methods optimize an objective of the form:

−γLf(θ)+αLr(θ)(1)-\gamma\mathcal{L}_f(\theta) + \alpha\mathcal{L}_r(\theta) \tag*{(1)}

where Lf\mathcal{L}_f is the forget-set negative log-likelihood or its bounded transform, Lr\mathcal{L}_r preserves retain-set performance, and γ\gamma, α\alpha control their contributions. To evaluate unlearning, on TOFU [29], we follow OpenUnlearning [8] and report Mem., the harmonic mean of 1−1-extraction strength, 1−1-exact memorization, 1−1-paraphrased-answer probability, and 1−1-truth ratio on DfD_f; Util., the harmonic mean of model utility and fluency; and Agg., the harmonic mean of Mem. and Util. However, Agg. rewards forgetting monotonically even beyond the oracle, whose own Agg. on forget05 is 0.641. Since unlearning is to match the retained model [49] and excessive forgetting can itself reveal prior membership [16], we also measure distance to the oracle. Specifically, we compute the absolute difference Δ\Delta between the unlearned model and the oracle for four forget-set statistics: Mem., truth ratio (TR), extraction strength (ES), and ROUGE-L recall of the answers, and report their average. We further report privacy leakage (PL) that is the relative gap in membership-inference AUC between θ\theta and θr\theta_r [38]: PL is 0 for the oracle, close to −100-100 before unlearning, and positive under-over-forgetting. On MUSE [38] we report verbatim and knowledge memorization on DfD_f and knowledge memorization on DrD_r.

Motivation: forgetting without a target. To examine how far likelihood suppression forgets, we evaluate six such methods on TOFU forget05 using Llama-3.2-1B. Each method is trained for the standard 10 epochs across five learning rates from 5⋅10−65 \cdot10^{-6} to 5⋅10−55 \cdot10^{-5}. The results are shown in Figure 1(a), with detailed measurements in Appendix Table 7. Two key observations emerge:

  • Observation 1: forgetting is highly sensitive to the budget. Five of the six methods reach or surpass the oracle’s forgetting level within the tested learning rates. NPO’s privacy leakage increases from −100-100 to +52+52, and WGA’s from −100-100 to +54+54; GradDiff, RMU, and PDU already over-forget at a learning rate of 2⋅10−52 \cdot10^{-5}, and SimNPO never reaches the oracle. No method lands near the oracle for a reason internal to the method.

  • Observation 2: aggregate scores favor over-forgetting. The strongest baseline scores are achieved beyond the oracle: WGA reaches an Agg. of 0.692 at PL =+54= +54, while NPO peaks at PL =+37= +37. Thus, tuning for Agg. alone pushes a method past the oracle rather than toward it.

The underlying reason lies in the optimization objectives themselves. Their forget-set terms reward decreasing the likelihood of DfD_f without identifying the oracle as a stopping point; bounded variants such as NPO [52] attenuate the suppression but do not ensure convergence to the oracle. The extent of forgetting is thus governed by the number of training epochs and the learning rate. An unlearning objective should instead explicitly target an estimate of the oracle.

Shift: Unlearning toward an Estimated Oracle

The oracle provides a target for unlearning, but training it would defeat the purpose of unlearning. Instead, we use the fine-tuned model θ0\theta_0, the base model θb\theta_b, and the retain set DrD_r to estimate the direction toward the oracle at low computational cost. We first introduce the shrink-and-heal lookahead and analyze its fixed point (§ 4.1). We then extrapolate its prediction changes to construct distillation targets on forget tokens (§ 4.2) and iteratively refine these targets as the model evolves (§ 4.3). Figure 2 illustrates one round.

One round of SHIFT

Figure 2. One round of SHIFT. (1) Shrink toward the pre-fine-tuning weights and re-fit the retain set. (2) On forget answers, the lookahead lowers the memorized token and raises the alternatives the retain set supports. (3) Extrapolating by s/ρs/\rho gives the teacher, the oracle’s next-token distribution in the linearized regime. (4) Distill it on forget tokens, anchored to θ0\theta_0 on retain data; the next round looks ahead from the new model.

Shrink-and-Heal Lookahead

Fine-tuning displaces the base model by τ=θ0−θb\tau= \theta_0 - \theta_b that encodes both DrD_r and DfD_f. Pulling the model back toward θb\theta_b by a fraction ρ\rho removes a ρ\rho-share of everything that fine-tuning added. Re-fitting the retain set for a few steps then restores the component supported by DrD_r. What remains removed is the part of τ\tau that DrD_r cannot explain, i.e., what only DfD_f taught. We formalize this operation as the shrink-and-heal lookahead:

Π(θ)=Heal⁡k(θ−ρ(θ−θb)),(2)\Pi(\theta) = \operatorname{Heal}_k\left(\theta- \rho\left(\theta- \theta_b\right)\right), \tag*{(2)}

where Heal⁡k\operatorname{Heal}_k denotes kk optimization steps on DrD_r. To characterize this operation, we consider the linearized fine-tuning regime ([19, 30, 33]), which has recently been extended to attention networks initialized from pre-trained weights ([39]). In this model, the logits satisfy f(x;θ)=f(x;θb)+Jx(θ−θb)f(x;\theta) = f(x;\theta_b) + J_x(\theta- \theta_b), and gradient descent from θb\theta_b reaches the minimum-norm displacement that fits its data ([15]). Let JrJ_r stack the retain-set Jacobians and let Pr=Jr+JrP_r = J_r^{+}J_r project onto their row space. PrτP_r\tau and (I−Pr)τ(I - P_r)\tau represent the retain-supported and forget-specific components of the fine-tuning displacement, respectively.

Proposition 1 (The lookahead is a ρ\rho-step toward the oracle). If the heal runs to convergence, then

θr=θb+Prτ,(3)\theta_r = \theta_b + P_r\tau, \tag*{(3)}
Π(θ0)=θ0+ρ(θr−θ0).(4)\Pi(\theta_0) = \theta_0 + \rho(\theta_r - \theta_0). \tag*{(4)}

More generally, every θ=θb+τ′\theta= \theta_b + \tau' that fits DrD_r satisfies Π(θ)−θ=−ρ(I−Pr)τ′\Pi(\theta) - \theta= -\rho(I - P_r)\tau', so the oracle θr\theta_r is the unique fixed point of Π\Pi.

The heal moves only within the row space of JrJ_r, so it restores exactly the retain-supported component ρPrτ\rho P_r\tau of what the shrink removed; the net move −ρ(I−Pr)τ-\rho(I - P_r)\tau is a fraction ρ\rho of the forget-specific component of fine-tuning (Figure 1b); proof in Appendix § 8. The lookahead therefore points to the oracle without training it, and needs no knowledge of which parameters encode DfD_f. At the oracle, it produces no further change, which is the property that likelihood suppression lacks (§ 3).

What the lookahead does in practice. Real fine-tuning is not linear and the heal is short. We therefore examine the lookahead on TOFU forget10 with Llama-3.2-1B (ρ=12\rho= \frac{1}{2}, k=32k = 32; Appendix § 9) before building on it, comparing gold-token log-probabilities before and after the lookahead, extrapolated by 1/ρ1/\rho. (1) The lookahead isolates the forget set: its forget-specific drop is 1.9 nats per token, approximately 80% of the true gap between θ0\theta_0 and the oracle (2.45 nats, computed post hoc). (2) It also moves unrelated text: on never-seen authors, it raises the gold tokens by 0.96 nats, as shrinking makes θ0\theta_0 less over-confident in general, and this drift does not shrink as kk increases from 16 to 64.

Design requirements. These observations give three requirements. First, the lookahead should be read, not kept: it covers only a fraction ρ\rho of the way, and changes predictions everywhere, including where unlearning should not act. Second, its signal should act only where unlearning applies, the forget tokens, while the rest of the model’s behavior is held fixed. Third, the estimate should be refreshed: since the lookahead is accurate only locally, it should be recomputed from the updated model. The next two subsections instantiate them.

From Lookahead to Teacher

Reading the lookahead in output space. Following the first requirement, we never use Π(θ)\Pi(\theta) as a model, but instead only compare its predictions with those of the current model. Proposition 1 shows how to turn this comparison into the oracle’s predictions. Extrapolating the lookahead step by 1/ρ1/\rho lands on the oracle: θ0+1ρ(Π(θ0)−θ0)=θr\theta_0+\frac{1}{\rho}(\Pi(\theta_0)-\theta_0)=\theta_r. Since logits are linear in θ\theta, we can obtain:

f(x;θr)=f(x;θ0)+1ρ(f(x;Π(θ0))−f(x;θ0)).(5)f(x;\theta_r)=f(x;\theta_0)+\frac{1}{\rho}\left(f(x;\Pi(\theta_0))-f(x;\theta_0)\right). \tag*{(5)}

Therefore, two forward passes suffice to recover the oracle’s logits, without the oracle. In practice, we store log-probabilities rather than logits, as the softmax is invariant to the per-position normalization (see Lemma 1 in Appendix B).

Extrapolated teacher. At each answer position tt of a forget example, let ℓn,t\ell_{n,t} and ℓΠ,t\ell_{\Pi,t} be the log-probabilities of the current model θn\theta_n and its lookahead, restricted to VtV_t, the union of both models’ top-256 tokens. We form the teacher by:

log⁡pT,t=norm⁡Vt(ℓn,t+clip⁡κ(snρ(ℓΠ,t−ℓn,t))).(6)\log p_{T,t}=\operatorname{norm}_{V_t}\left(\ell_{n,t}+\operatorname{clip}_{\kappa}\left(\frac{s_n}{\rho}(\ell_{\Pi,t}-\ell_{n,t})\right)\right). \tag*{(6)}

which renormalizes the probabilities within VtV_t and leaves the current model’s mass outside VtV_t unchanged. The step sns_n scales the extrapolation: sn=ρs_n=\rho reproduces the lookahead, and sn=1s_n=1 takes the full step in Eq. (5). By Lemma 1, the teacher at sn=1s_n=1 is the oracle’s next-token distribution on VtV_t in the linearized regime. We clip each extrapolated change at κ=10\kappa=10 nats, since dividing by ρ\rho amplifies noise in the lookahead.

The change ℓΠ,t−ℓn,t\ell_{\Pi,t}-\ell_{n,t} does not only lower the memorized token; it also raises the alternatives that a model without Df\mathcal{D}_f would predict (Figure 2, panel 2). This distinction matters because suppressing only the gold token can transfer its probability mass to semantically related continuations [24]. Unlike methods that apply a fixed reduction to the gold-token logit [7], our teacher tells the model both how much to forget at each position and what to say instead, as estimated from the retain set.

Distillation and Iteration

Distillation with a retain anchor. Following the second requirement, the teacher acts only on forget tokens, and every retain example anchors the model to θ0\theta_0’s own predictive distribution:

L(θ)=γEDfKL⁡(pT∥pθ)+αEDrKL⁡(pθ0∥pθ).(7)\mathcal{L}(\theta)=\gamma\mathbb{E}_{\mathcal{D}_f}\operatorname{KL}(p_T\Vert p_\theta)+\alpha\mathbb{E}_{\mathcal{D}_r}\operatorname{KL}(p_{\theta_0}\Vert p_\theta). \tag*{(7)}

Unlike the likelihood-suppression objectives, this loss is minimized at the teacher, so additional optimization brings the model closer to the estimated oracle rather than past it. The KL anchor preserves the full distribution on retain data, not only its gold tokens [52, 7], and also holds the retain knowledge that a finite heal fails to restore (Appendix § 8). We cache θ0\theta_0’s top-64 tokens per retain position once, avoiding the need to keep a separate reference model in memory. Although the loss acts on outputs, in the linearized regime it carries out exactly the parameter-space step that the lookahead defines:

Proposition 2 (Distillation implements the step). Let θn\theta_n fit Dr\mathcal{D}_r, and let the teacher be (6) at step ss without clipping or truncation. Gradient descent on the logit-matching form of (7) from θn\theta_n converges to θn+sρ(Π(θn)−θn)\theta_n+\frac{s}{\rho}(\Pi(\theta_n)-\theta_n); at s=1s=1, this point is the oracle θr∗\theta_r^*.

The proof is provided in Appendix § 8. Both loss terms target the prediction outputs of the extrapolated parameter point, and gradient descent updates the parameters only within the row space of the training Jacobian, which contains the desired step.

Iterating the update. Following the third requirement, each round takes a new lookahead from the model it has just produced. In the linearized setting, this iteration provably converges to the oracle:

Corollary 1 (Iteration). Let θn+1=θn+sρ(Π(θn)−θn)\theta_{n+1}=\theta_n+\frac{s}{\rho}(\Pi(\theta_n)-\theta_n) with θ0=θ0\theta_0=\theta_0. Every iterate fits Dr\mathcal{D}_r, its retain-supported part Pr(θn−θb)P_r(\theta_n-\theta_b) stays equal to PrτP_r\tau, and its forget-specific part shrinks geometrically:

(I−Pr)(θn−θb)=(1−s)n(I−Pr)τ.(8)(I-P_r)(\theta_n-\theta_b)=(1-s)^n(I-P_r)\tau. \tag*{(8)}

Hence θn→θr\theta_n\to\theta_r for every s∈(0,2)s\in(0,2), and s=1s=1 reaches θr\theta_r in one step.

The proof is provided in Appendix § 8. The step s=1s = 1 is thus a Newton step on the fixed-point equation Π(θ)=θ\Pi(\theta) = \theta, and by Proposition 2 each round of distillation realizes one such step. On real models, each new lookahead corrects what the previous linear estimate missed. We use a fixed schedule s=(ρ,1,1,1)s = (\rho, 1, 1, 1): the first round takes the lookahead itself without extrapolation, as the lookahead moves θ0\theta_0 the most and extrapolating that first change is the least reliable step (the mean teacher drop at θ0\theta_0 is 0.62 nats at s=12s = \frac{1}{2} but 2.47 at s=1s = 1). The complete procedure is summarized in Algorithm 1.

What the idealization leaves out. Each gap between the analysis and practice is covered by one component. A finite heal restores retain knowledge only partially (Appendix § 8), and the lookahead drifts on unrelated text: the teacher therefore acts only on forget tokens, and the retain anchor holds the rest. Real fine-tuning is nonlinear, so the lookahead is accurate only locally: the iteration re-linearizes every round.

Experiments

Setup

Benchmarks and models. We use TOFU ([29]) with Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct on the forget01, forget05 and forget10 splits, with the OpenUnlearning target and oracle models ([8]); and MUSE-News ([38]) with Llama-2-7B. The bases θb\theta_b are the corresponding instruction-tuned (TOFU) or pre-trained (MUSE) checkpoints.

Baselines. We compare SHIFT against eleven unlearning baselines: GradAscent, GradDiff, NPO, SimNPO, DPO with refusal targets (IdkDPO) ([35]), RMU ([25]), UNDIAL ([7]), WGA ([40]), SatImp ([45]), CE-U ([44]) and PDU ([10]), with the OpenUnlearning implementations.

Protocol. Every baseline follows the OpenUnlearning training configuration ([8]) with its default hyperparameters: learning rate 10−510^{-5}, 10 epochs, effective batch 32 on TOFU, bf16 weights, one seed, and the final model (no checkpoint selection). SHIFT uses the same learning rate, batch and model selection, with ρ=12\rho= \frac{1}{2}, k=16k = 16, κ=10\kappa= 10, α=γ=1\alpha= \gamma= 1 and four rounds of four epochs. This budget is chosen once, on Llama-3.2-1B forget05, and used unchanged for every model, split and

benchmark in the main results. Appendix § 10 gives the implementation details.

Main Results

SHIFT ends closest to the oracle. The goal of unlearning is the retrained model, so we first ask how close each method ends to it. The results are reported in Table 1. SHIFT is the closest in every setting but Llama-3.1-8B forget01 and, on average, on each of the four statistics: within 0.019 of the oracle, 3.7× closer than the next method (NPO), with memorization and utility on average within 0.019 and 0.016 of the oracle’s (Appendix Table 6). TOFU’s own test against the oracle agrees: SHIFT has the highest forget quality on average (−2.07, against −3.29 for NPO and −3.30 for RMU). The baselines miss the oracle in one of two ways. Most stop short of it: NPO, UNDIAL, WGA, IdkDPO, SimNPO and SatImp leave Mem. 0.09–0.49 below the oracle’s. The others lose utility: Util. falls 0.16–0.56 below the oracle’s for RMU, GradDiff, PDU, GradAscent and CE-U.

MethodΔMem.↓\Delta\mathrm{Mem.}\downarrowΔTR↓\Delta\mathrm{TR}\downarrowΔES↓\Delta\mathrm{ES}\downarrowΔRL↓\Delta\mathrm{RL}\downarrowAvg.↓\mathrm{Avg.}\downarrowFQ↑\mathrm{FQ}\uparrow
GradAscent0.2480.2760.0660.2180.202−89.16
GradDiff0.2080.1950.0850.1310.155−47.38
NPO0.0910.0450.0590.0860.071−3.29
SimNPO0.4140.1070.4700.3390.332−11.49
IdkDPO0.2500.0710.2620.2750.215−6.67
RMU0.1580.0480.0880.1490.111−3.30
UNDIAL0.1310.0900.0910.0640.094−12.39
WGA0.2020.0890.1270.1480.141−11.37
SatImp0.4910.1100.7130.4500.441−12.26
CE-U0.2170.1260.0480.3020.173−41.69
PDU0.1360.0610.0390.1900.106−12.30
SHIFT (ours)0.0190.0210.0060.0290.019−2.07

Table 1. Distance to the oracle, ∣x−xretain∣\lvert x - x_{\mathrm{retain}}\rvert, for Mem., truth ratio (TR), extraction strength (ES) and ROUGE-L (RL) on the forget set, and TOFU’s forget quality (FQ: log⁡ρ\log\rho of a test against the oracle’s truth ratios; 0 is the oracle), averaged over the settings of the Avg. column of Table 2. Bold / underlined: best / second best.

SHIFT has the best aggregate score. SHIFT has the highest Agg. in six of the nine settings (Table 2), 0.009–0.076 above the strongest baseline, and the best average (0.633 vs. 0.561 for NPO). Two of the exceptions go to methods that over-forget there: RMU on Llama-3.2-3B forget10 (privacy leakage +30) and NPO on Llama-3.1-8B forget05 (+44). On Llama-3.1-8B forget10, UNDIAL leads (0.653 vs. 0.631), while SHIFT stays three times closer to the oracle (0.012 vs. 0.036). SHIFT earns its scores by matching the oracle: its memorization stays within 0.045 of the oracle’s in every setting, even where its Agg. exceeds the oracle’s own. No baseline is competitive everywhere: the strongest one changes across settings (PDU, RMU, NPO, GradAscent, UNDIAL), and one to four fall below 0.2 in each. Our baseline scores match those reported for tuned baselines (NPO: 0.55 and 0.57 on Llama-3.2-1B forget05 and forget10, against 0.54 and 0.58 in Li et al. [24]).

MethodLlama-3.2-1B forget01forget05forget10Llama-3.2-3B forget01forget05forget10Llama-3.1-8B forget01forget05forget10Avg.
Full0.1180.1540.1590.0240.0590.0630.0050.0160.0110.068
Retain0.6130.6410.6430.6320.6480.6500.6190.6410.6490.637
GradAscent0.4580.2140.0000.4090.5670.0000.6170.0000.0000.252
GradDiff0.4400.5270.5210.3780.4870.4820.5650.1110.4230.437
NPO0.4580.5480.5680.4190.5610.5840.5990.6660.6500.561
SimNPO0.2180.2290.2350.1510.1650.1810.2500.3660.3830.242
IdkDPO0.2940.4420.4880.1960.3970.4310.4780.5490.5310.423
RMU0.5090.5750.2210.2120.5390.6460.5330.6120.4470.477
UNDIAL0.4560.4800.4840.4460.5320.5460.6300.6490.6530.542
WGA0.4340.4880.5060.4050.4530.4850.5520.5890.4360.483
SatImp0.1390.1750.1820.0360.0800.0820.1060.1430.2070.128
CE-U0.5570.0000.0000.5380.0000.0000.2820.0000.0000.153
PDU0.5730.3350.4990.5590.0450.5200.0180.6310.3100.388
SHIFT (ours)0.6190.6230.6240.6350.6390.6340.6390.6500.6310.633

Table 2. Agg.↑ on TOFU, every method at the OpenUnlearning learning rate 10−510^{-5} (one seed). Agg. is the harmonic mean of Mem. (forgetting of DfD_f) and Util. (retained utility), so it is high only when a method both forgets and keeps utility. Bold / underlined: best / second-best Agg. of each setting. Full: before unlearning; Retain: the oracle. Mem. and Util. are in Appendix Table 6.

MUSE. On MUSE-News (results are reported in Table 3), where the forget set consists of raw text and the base is a pre-trained model, the methods that push verbatim memorization below the retrained oracle (GradAscent, GradDiff, UNDIAL, WGA) substantially degrade retain knowledge

MethodVerbMemKnowMem-FKnowMem-R
Target0.5790.6440.555
Retrain0.2020.3280.560
GradAscent0.0000.0000.000
GradDiff0.0220.3400.121
NPO0.4410.6270.496
SimNPO0.5190.6210.496
RMU0.2760.4610.476
UNDIAL0.1380.1300.204
WGA0.0790.2450.228
SHIFT (ours)0.2350.5720.539

Table 3. MUSE-News, Llama-2-7B, learning rate 10−510^{-5} (baselines: 10 epochs; SHIFT: 4 rounds of 4 epochs). Target: before unlearning; Retrain: retrained without the forget set (the oracle).

(KnowMem-R at most 0.23). In contrast, SHIFT achieves the highest retain-knowledge score (0.539, 97% of the target’s), while removing more memorization than NPO and SimNPO. It also comes closest to the retrained oracle in verbatim memorization and retain knowledge.

Analysis

More training does not overshoot. This analysis and the relearning attacks below use learning rate 2⋅10−52 \cdot10^{-5}, where the baselines’ 10 epochs carry twice the default update (SHIFT: k=32k = 32, two epochs a round). Figure 3 trains NPO, WGA, UNDIAL and SHIFT twice as long, the baselines for 20 epochs under their own warm-up and decay, and evaluates them along the way. NPO and WGA cross the oracle within a standard run: NPO settles at a privacy leakage of +24, and WGA keeps rising to +52, three times further from the oracle than at its closest. UNDI­AL never approaches the oracle (privacy leakage at most −84). SHIFT approaches the oracle and stays there: its distance is smallest at round 4 (0.007) and remains below 0.02 through round 8, and its privacy leakage rises toward 0 without crossing it, as expected of a loss minimized at its teacher rather than at an ever lower likelihood. Across learning rates (Figure 1(a), Appendix Table 7), every baseline but SimNPO overshoots to +40–+57 at 5⋅10−55 \cdot10^{-5}, whereas SHIFT stays within [−23, +6] from 2⋅10−52 \cdot10^{-5} up.

Training curves for NPO, WGA, UNDIAL, and SHIFT

Figure 3. What happens when training continues, on TOFU forget05 (Llama-3.2-1B, learning rate 2⋅10−52 \cdot10^{-5}): the baselines train for 20 epochs (warm-up, then linear decay over the 20 epochs) and SHIFT for 8 rounds. The dashed line marks the end of a standard 10-epoch run (step 60); SHIFT’s standard 4 rounds end at step 48. Top: distance to the oracle (log scale); bottom: privacy leakage (0 is the oracle, positive is over-forgetting). SHIFT: mean over three seeds, shaded from minimum to maximum; baselines: one run, evaluated every second epoch.

Training cost. SHIFT costs only modestly more than the baselines. Each round adds k=16k = 16 retain steps and two forward passes over DfD_f, and distillation runs no reference model, since the teacher and the retain anchor are cached (Algorithm 1). On one GPU, SHIFT trains on Llama-3.2-1B forget05 in 19 minutes against NPO’s 15.

Ablations. Table 4 removes one component at a time, in the setting of Table 2; every variant falls below SHIFT in Agg., Mem. and Util. alike. The heal keeps the target at the oracle: without it (k=0k = 0) the lookahead is a pure shrink toward θb\theta_b rather than a step toward the oracle (Proposition 1). At this budget the two end close (Agg. 0.617; Appendix Table 8 varies kk), but at twice the learning rate, removing the heal nearly doubles the distance to the oracle (Appendix Table 9). Extrapolation saves budget: the lookahead without extrapolation (s=ρs = \rho) moves in the right direction but more slowly (Corollary 1), reaching a lower Agg. (0.609) and a larger distance to the oracle (0.023) in the same rounds. Both directions of the teacher matter: keeping only the lowered tokens cuts Agg. to 0.563 and leaves the model nearly 4×4\times further from the oracle, since the raised alternatives are what moves the mass [41].

VariantAgg.↑Mem.↑Util.↑Δoracle↓\Delta_{\mathrm{oracle}}\downarrow
SHIFT0.6230.5580.7050.016
no heal (k=0k=0)0.6170.5530.6980.015
no extrapolation (s=ρs=\rho)0.6090.5370.7020.023
no raised tokens0.5630.4700.7030.060
Lookahead as the model, 1 round0.5200.4250.6680.084
Lookahead as the model, 4 rounds0.5590.5450.5730.013

Table 4. Ablation studies on TOFU forget05, Llama-3.2-1B (setting of Table 2; Δoracle\Delta_{\mathrm{oracle}} as in Table 1). Raised tokens drive the forgetting, extrapolation saves budget, the heal anchors the target (Table 9), and distillation keeps utility: only together are they best in Agg., Mem. and Util.

Reading beats committing: using the lookahead itself as the model, as resetting methods do [1, 32], either stops short (one round: Agg. 0.520) or, if iterated, loses utility (0.573 vs. 0.705).

The results of additional analyses, including learning-rate sensitivity (Table 7), heal-length sensitivity (Table 8), and ablations at a higher learning rate (Table 9), are provided in Appendix § 12. SHIFT is insensitive to the heal length kk (Table 8), so one setting can transfer to every other model and split without retuning.

Method5⋅10−65 \cdot10^{-6}10−510^{-5}2⋅10−52 \cdot10^{-5}3⋅10−53 \cdot10^{-5}5⋅10−55 \cdot10^{-5}
GradDiff0.353 / −98-980.527 / −54-540.539 / +49+490.293 / +57+570.094 / +57+57
NPO0.335 / −100-1000.548 / −70-700.621 / +8+80.638 / +37+370.631 / +52+52
SimNPO0.179 / −100-1000.229 / −100-1000.344 / −99-990.420 / −99-990.486 / −95-95
RMU0.446 / −98-980.575 / −41-410.182 / +50+500.099 / +50+500.085 / +46+46
WGA0.253 / −100-1000.488 / −94-940.634 / −28-280.681 / +36+360.692 / +54+54
PDU0.451 / −96-960.335 / −48-480.016 / +14+140.121 / +24+240.503 / +40+40
SHIFT (ours)0.427 / −97-970.603 / −74-740.633 / −23-230.628 / +6+60.596 / +6+6

Table 7. TOFU forget05, Llama-3.2-1B: Agg. / PL as the learning rate varies (seed 0; 10−510^{-5} is the OpenUnlearning default run of each baseline; SHIFT uses k=32k=32 and two epochs a round at every rate). SHIFT’s PL stays within [−23-23, +6+6] from 2⋅10−52 \cdot10^{-5} up; at 5⋅10−55 \cdot10^{-5} every baseline but SimNPO overshoots to +40+40–+57+57.

kkAgg.↑\uparrowMem.↑\uparrowUtil.↑\uparrowΔoracle↓\Delta_{\mathrm{oracle}}\downarrow
00.6170.5530.6980.015
80.6160.5460.7060.020
160.6230.5580.7050.016
320.6190.5510.7060.016

Table 8. TOFU forget05, Llama-3.2-1B, learning rate 10−510^{-5}, four rounds of four epochs (one seed): SHIFT with kk heal steps per round. Bold: the value of the main results. Δoracle\Delta_{\mathrm{oracle}}: as in Table 1.

VariantnnAgg.↑\uparrowMem.↑\uparrowUtil.↑\uparrowΔoracle↓\Delta_{\mathrm{oracle}}\downarrow
SHIFT30.633±\pm.0010.587±\pm.0010.686±\pm.0020.006
no heal (k=0k=0)30.632±\pm.0020.584±\pm.0010.689±\pm.0040.011
no extrapolation (s=ρs=\rho)30.619±\pm.0010.565±\pm.0010.684±\pm.003<u>0.007</u>
no raised tokens30.582±\pm.0000.506±\pm.0010.686±\pm.0010.041
Lookahead as the model, 1 round30.513±\pm.0010.419±\pm.0020.663±\pm.0010.089
Lookahead as the model, 4 rounds30.566±\pm.0010.537±\pm.0000.597±\pm.0020.017

Table 9. The ablations of Table 4 at learning rate 2⋅10−52 \cdot10^{-5} on TOFU forget05, Llama-3.2-1B (nn: seeds). Bold / underlined: closest / second closest to the oracle.

Robustness to relearning. Suppressed knowledge can return after fine-tuning on related data [28, 17, 5]. We attack the 2⋅10−52 \cdot10^{-5} models, relearning half of the forget05 authors and testing the other half (see details in Appendix § 13). The results are reported in Figure 4 and Appendix Table 11. After the attack, SHIFT is within 0.008 of the equally attacked oracle, 2.4×2.4\times closer than any baseline, whereas aggressive forgetting bounces back (RMU’s extraction strength rises from 0.029 to 0.195; Appendix Table 10), and SHIFT stays the closest at all six attack strengths (Figure 4).

Relearning attacks of increasing strength

Figure 4. Relearning attacks of increasing strength on Llama-3.2-1B forget05: distance to the equally attacked oracle (mean of three attack seeds). SHIFT is the closest at every strength (results for all baselines refer to Appendix Table 11).

Conclusion

Likelihood-suppression unlearning has no target, so its outcome is set by the optimization budget. The shrink-and-heal lookahead estimates this target from the model itself (its linearized fixed point is the retrained oracle); SHIFT distills it on forget tokens and iterates. Across three model sizes, it is on average the closest method to the oracle and has the highest average aggregate score, and it stays near the oracle where the baselines overshoot: unlearning should target an estimate of the oracle.

Limitations

SHIFT needs the pre-fine-tuning weights θb\theta_b; it applies to knowledge acquired in an identifiable fine-tuning stage (as in TOFU and MUSE), not to knowledge from pre-training, where no earlier checkpoint without it may exist. Our analysis assumes the linearized regime, a heal run to convergence and an exact teacher (no clipping or top-token truncation); we measured where real models depart from it (§ 4.1) but do not bound the error.

Ethics Statement

Unlearning is used to remove private or copyrighted content; methods that look successful on a single metric but over-forget are detectable by membership inference, which itself leaks which data were removed [16]. Calibrating forgetting to the retrained model is meant to reduce this leak. TOFU uses fictitious authors; MUSE uses public news articles. No new personal data were collected.

This appendix provides additional related work, theoretical proofs, implementation details, and experimental results supporting the main paper. Specifically, it includes:

  • Appendix A discusses extended related work: further discussion of unlearning objectives, oracle-based evaluation, distillation, robustness, and model arithmetic.

  • Appendix B provides proofs: complete proofs of Propositions 1 and 2, Corollary 1, and Lemma 1, along with an analysis of finite healing.

  • Appendix C measures the Lookahead: empirical analysis of the lookahead’s effects on forget-set and unseen examples.

  • Appendix D presents the implementation details, including training configurations, hyperparameters, and the complete algorithm.

  • Appendix E provides full TOFU results, including detailed memorization, utility, and aggregate scores across TOFU settings.

  • Appendix F reports additional analyses results, including learning-rate sensitivity, heal-length sensitivity, and ablations at larger learning rate.

  • Appendix G discusses relearning Attack, including attack setup and evaluation of oracle alignment under varying relearning strengths.

Extended Related Work

Likelihood suppression. Early unlearning methods use gradient ascent on the forget set ([20, 46]), often combined with retain-set regularization ([26]). Subsequent approaches introduce bounded or reweighted objectives ([52, 11, 40, 45]), refusal targets ([29]), representation misdirection ([25]), and constrained optimization ([10]). Recent work also explores adaptive stopping criteria based on the model’s own predictions ([6]). Despite their different mechanisms, they do not explicitly target the behavior of the retain-only oracle, which provides a principled reference for unlearning ([49, 16]).

Evaluation and scope. Recent position and evaluation work argues that unlearning claims should be judged against retraining ([49]) and that forgetting must generalize narrowly while retention generalizes broadly ([34]); robustness to relearning depends on how forget-and-retain-critical weights are separated ([3]). Our evaluation follows the first view: we measure the distance to the retrained model. Measuring the gap to the retrained model is standard beyond LLMs: forgetting has been defined as indistinguishability from a model trained without the data ([14]), image-classification unlearning reports the average gap to Retrain across accuracy and membership-inference metrics ([12]), and SCRUB adds a rewinding step for privacy so that the forget set is not forgotten beyond what retraining would give ([22]). For LLMs, TOFU’s forget quality tests whether the forget-set truth ratios are distributed as the retain model’s ([29]), and MUSE’s PrivLeak compares the membership-inference AUC with the retrained model’s, flagging both under- and over-unlearning ([38]). Our Δ\Delta applies this gap to each forget statistic, PL carries MUSE’s measure over to TOFU, and Table 1 reports TOFU’s forget quality.

Distillation and target distributions. Teacher–student unlearning uses an incompetent or adjusted teacher ([4, 22]); UNDIAL lowers the gold logit by a fixed amount ([7]); “Who’s Harry Potter” builds the teacher from a model reinforced on the forget data and subtracts its excess ([9]). SHIFT instead moves away from the fine-tuning that produced the knowledge, and calibrates the move by the retain set. Logit-difference methods combine the model with an assistant trained to remember the forget set ([21] (Huang et al. 2025)); emulated fine-tuning adds or removes the effect of fine-tuning at the logit level ([31]). DUET distills from an in-context prompted teacher ([54]), and bootstrapping methods suppress the model’s own high-confidence generations to avoid the squeezing of probability mass into rephrasings ([24]). Simulated oracle matching predicts the oracle’s outputs from data attribution ([13]); SHIFT predicts them from a cheap retain re-fit.

Robustness of unlearning. Suppressed knowledge can return after fine-tuning on related data ([28, 17, 5]), especially when the relearning data share their format with the forget set ([48]), or after low-bit quantization ([53]). Recent defenses target the answer margins ([47]), minor representation components ([42]), smoothness in parameter and representation space ([43]), or quantization explicitly ([37]), and distilling toward a target without the knowledge has been argued to remove it more durably ([23]). SHIFT is a distillation method of this kind, with a target estimated from the retain set.

Checkpoint arithmetic. The closest related approach is MSA ([36]), which also uses a checkpoint from before the forget set was seen, but in the opposite direction: it fine-tunes that checkpoint on DfD_f and subtracts the resulting vector, scaled by a tuned coefficient, from the final model. SHIFT never trains on DfD_f. It shrinks toward the checkpoint and re-fits DrD_r, so the forget direction is what DrD_r cannot restore, and its magnitude is set by the oracle (Proposition 1) rather than by that coefficient. The estimate is then read on forget tokens and distilled, instead of being subtracted in weight space.

Resetting weights. Shrink-and-perturb ([1]), rewinding to an earlier checkpoint and fine-tuning on retained data ([32]), Fisher-based scrubbing ([14]) and noising followed by distillation ([23]) all commit a reset. SAGE projects an unlearning update away from directions that damage retain activations ([50]), close in spirit to how our heal restores retain-supported directions. Our ablation that keeps the lookahead as the model (Table 4) belongs to this family; SHIFT reads the reset instead of keeping it, and derives a calibrated target from it. Task arithmetic negates a task vector to forget a task ([18]); Proposition 1 explains why shrinking and re-fitting the retain set isolates the forget-specific part of that vector without training on the forget set.

Proofs

Setup. Let fX(θ)∈Rmf_X(\theta) \in\mathbb{R}^m stack the logits of a set XX of training positions. In the linearized model,

fX(θ)=fX(θb)+JX(θ−θb).f_X(\theta) = f_X(\theta_b) + J_X(\theta-\theta_b).

Training to interpolation fits targets yXy_X (for the cross-entropy loss, logits that realize the fitted distributions; the argument only needs that the fitted values are reached). We use one standard fact about gradient descent ([15]):

Fact. Gradient descent on any loss of fXf_X, started from θ′\theta', moves only within the row space row⁡(JX)\operatorname{row}(J_X). If it fits yXy_X, it therefore converges to the closest fitting point:

θ′+JX+(yX−fX(θ′)).(9)\theta' + J_X^{+}(y_X-f_X(\theta')). \tag*{(9)}

For the retain set we write ur=yr−fr(θb)u_r = y_r-f_r(\theta_b) for the residual of the base model, Pr=Jr+JrP_r = J_r^{+}J_r for the projection onto row⁡(Jr)\operatorname{row}(J_r), and τ=θ0−θb\tau= \theta_0-\theta_b for what fine-tuning added.

Proof of Proposition 1. Step 1: the oracle. The target model fits DrD_r, so Jrτ=urJ_r\tau=u_r. The oracle is gradient descent on DrD_r from θb\theta_b, so by (F)

θr=θb+Jr+ur=θb+Jr+Jrτ=θb+Prτ,\theta_r = \theta_b + J_r^{+}u_r = \theta_b + J_r^{+}J_r\tau= \theta_b + P_r\tau,

which is Eq. (3).

Step 2: the shrink. The shrunk model θ′=θb+(1−ρ)τ\theta'=\theta_b+(1-\rho)\tau leaves a fraction ρ\rho of the retain set to be re-learned:

yr−fr(θ′)=ur−(1−ρ)Jrτ=ρur.y_r-f_r(\theta') = u_r-(1-\rho)J_r\tau= \rho u_r.

Step 3: the heal. By (F), healing from θ′\theta' adds Jr+(ρur)=ρPrτJ_r^{+}(\rho u_r)=\rho P_r\tau. Hence

Π(θ0)=θb+(1−ρ)τ+ρPrτ,\Pi(\theta_0)=\theta_b+(1-\rho)\tau+\rho P_r\tau,
Π(θ0)−θ0=−ρ(I−Pr)τ=ρ(θr−θ0),\Pi(\theta_0)-\theta_0=-\rho(I-P_r)\tau=\rho(\theta_r-\theta_0),

where the last equality uses θr−θ0=Prτ−τ\theta_r-\theta_0=P_r\tau-\tau from Step 1. This is Eq. (4).

Step 4: the fixed point. Every lookahead fits DrD_r, because the heal runs to convergence, so it suffices to consider θ=θb+τ′\theta=\theta_b+\tau' with Jrτ′=urJ_r\tau'=u_r. Steps 2–3 then apply with τ′\tau' in place of τ\tau.

Π(θ)−θ=−ρ(I−Pr)τ′.\Pi(\theta)-\theta=-\rho(I-P_r)\tau'.

A fixed point has (I−Pr)τ′=0(I-P_r)\tau'=0, i.e., τ′∈row⁡(Jr)\tau'\in\operatorname{row}(J_r) with Jrτ′=urJ_r\tau'=u_r. The only such vector is τ′=Jr+ur=Prτ\tau'=J_r^{+}u_r=P_r\tau, so the fixed point is θr\theta_r, and it is unique. This completes the proof. □

Proof of Proposition 2. Let JDJ_D stack the Jacobians of all training positions of Df∪DrD_f\cup D_r, write τn=θn−θb\tau_n=\theta_n-\theta_b, and let

θ⋆=θn+sρ(Π(θn)−θn)=θn−s(I−Pr)τn,\theta^\star=\theta_n+\frac{s}{\rho}\bigl(\Pi(\theta_n)-\theta_n\bigr)=\theta_n-s(I-P_r)\tau_n,

where the second form is Proposition 1. We show that distillation from θn\theta_n lands exactly on θ⋆\theta^\star. Step 1: the targets are the outputs of θ⋆\theta^\star. On Df\mathcal{D}_f, linearity (and Lemma 1 for log-probabilities) gives

ff(θn)+sρ(ff(Π(θn))−ff(θn))=ff(θ⋆).f_f(\theta_n) + \frac{s}{\rho}\left(f_f(\Pi(\theta_n)) - f_f(\theta_n)\right) = f_f(\theta^\star).

On Dr\mathcal{D}_r, the anchor’s target is fr(θ0)=yrf_r(\theta_0) = y_r, and fr(θ⋆)=fr(θn)=yrf_r(\theta^\star) = f_r(\theta_n) = y_r because Jr(I−Pr)=0J_r(I-P_r)=0.

Step 2: the step lies in row⁡(JD)\operatorname{row}(J_{\mathcal{D}}). The target model is the minimum-norm fit of D\mathcal{D}, so τ∈row⁡(JD)\tau\in\operatorname{row}(J_{\mathcal{D}}). Each iterate is τn+1=(1−s)τn+sPrτn\tau_{n+1}=(1-s)\tau_n+sP_r\tau_n with Prτn∈row⁡(Jr)⊆row⁡(JD)P_r\tau_n\in\operatorname{row}(J_r)\subseteq\operatorname{row}(J_{\mathcal{D}}), so by induction from τ0=τ\tau_0=\tau every τn\tau_n lies in row⁡(JD)\operatorname{row}(J_{\mathcal{D}}), and hence so does θ⋆−θn=−s(I−Pr)τn\theta^\star-\theta_n=-s(I-P_r)\tau_n.

Step 3: conclusion. By (F), gradient descent on the logit-matching loss from θn\theta_n converges to

θn+JD+(fD(θ⋆)−fD(θn))=θn+JD+JD(θ⋆−θn)=θ⋆,\theta_n+J_{\mathcal{D}}^{+}\left(f_{\mathcal{D}}(\theta^\star)-f_{\mathcal{D}}(\theta_n)\right) =\theta_n+J_{\mathcal{D}}^{+}J_{\mathcal{D}}(\theta^\star-\theta_n)=\theta^\star,

since JD+JDJ_{\mathcal{D}}^{+}J_{\mathcal{D}} is the identity on row⁡(JD)\operatorname{row}(J_{\mathcal{D}}). At s=1s=1, Corollary 1 gives θ⋆=θr\theta^\star=\theta_r. The KL form of the loss is minimized by any logits that equal the targets up to a constant per position, and those leave the predictive distributions unchanged.

Proof of Corollary 1. Write τn=θn−θb\tau_n=\theta_n-\theta_b. If θn\theta_n fits Dr\mathcal{D}_r, Proposition 1 gives

θn+1=θn+sρ(Π(θn)−θn)=θn−s(I−Pr)τn.\theta_{n+1}=\theta_n+\frac{s}{\rho}\left(\Pi(\theta_n)-\theta_n\right) =\theta_n-s(I-P_r)\tau_n.

The step lies in the null space of JrJ_r, so Jrτn+1=Jrτn=urJ_r\tau_{n+1}=J_r\tau_n=u_r and θn+1\theta_{n+1} still fits Dr\mathcal{D}_r; by induction from θ0=θ0\theta_0=\theta_0, every iterate does. Projecting the step onto row⁡(Jr)\operatorname{row}(J_r) and its complement,

Prτn+1=Prτn,P_r\tau_{n+1}=P_r\tau_n,
(I−Pr)τn+1=(1−s)(I−Pr)τn.(I-P_r)\tau_{n+1}=(1-s)(I-P_r)\tau_n.

Unrolling from τ0=τ\tau_0=\tau,

Prτn=Prτ,(I−Pr)τn=(1−s)n(I−Pr)τ.P_r\tau_n=P_r\tau,\qquad(I-P_r)\tau_n=(1-s)^n(I-P_r)\tau.

For s∈(0,2)s\in(0,2) we have ∣1−s∣<1\lvert1-s\rvert<1, so τn→Prτ\tau_n\to P_r\tau, i.e., θn→θr\theta_n\to\theta_r; for s=1s=1 the forget-specific part vanishes after one step.

Lemma 1 (Normalization invariance). For logits z,z′z,z' with log-probabilities ℓ=z−lse⁡(z)\ell=z-\operatorname{lse}(z) and ℓ′=z′−lse⁡(z′)\ell'=z'-\operatorname{lse}(z'), and any β\beta,

softmax⁡(ℓ+β(ℓ′−ℓ))=softmax⁡(z+β(z′−z))\operatorname{softmax}\left(\ell+\beta(\ell'-\ell)\right)=\operatorname{softmax}\left(z+\beta(z'-z)\right)

Proof of Lemma 1. Substituting the definitions,

ℓ+β(ℓ′−ℓ)=z+β(z′−z)−[(1−β)lse⁡(z)+βlse⁡(z′)].\ell+\beta(\ell'-\ell)=z+\beta(z'-z)-\left[(1-\beta)\operatorname{lse}(z)+\beta\operatorname{lse}(z')\right].

The bracket is the same across tokens, and softmax is invariant to adding a constant to all logits.

A finite heal. With kk heal steps of size η\eta on the squared loss, the healed component along an eigenvector viv_i of Jr⊤JrJ_r^\top J_r with eigenvalue λi\lambda_i is a fraction 1−(1−ηλi)k1-(1-\eta\lambda_i)^k of its shrunk part. The lookahead step therefore becomes

Π(θ0)−θ0=−ρ(I−Pr)τ−ρ∑i(1−ηλi)kvivi⊤Prτ.\Pi(\theta_0)-\theta_0=-\rho(I-P_r)\tau -\rho\sum_i(1-\eta\lambda_i)^k v_i v_i^\top P_r\tau.

The first term is the step of Proposition 1. The second is the retain knowledge that the heal did not restore; it lies in retain-supported directions, which is what the retain anchor of Eq. (7) counteracts (§ 4.3).

Measuring The Lookahead

Before building the method, we measured the lookahead at θ0\theta_0 (TOFU forget10, Llama-3.2-1B, ρ=12\rho=\frac{1}{2}). For each answer position, dt=−(ℓΠ−ℓ0)(yt)/ρd_t=-(\ell_\Pi-\ell_0)(y_t)/\rho is the extrapolated (s=1s=1) drop of the gold token’s log-probability: positive when the lookahead lowers the gold token, negative when it raises it. On never-seen holdout authors (positions 200–300 of holdout10) we fit a band of dd for each confidence decile of ℓ0(yt)\ell_0(y_t), and call a position active when its dd falls outside the band. RR is the gated forget-set drop at s=1s=1 (forget minus null).

Heal steps kkdˉ\bar d holdoutdˉ\bar d forgetRRActive F/null/test
16−0.876-0.876+1.414+1.4141.580.26/0.05/0.056
32−0.955-0.955+1.489+1.4891.930.31/0.05/0.058
64−0.933-0.933+1.542+1.5421.840.30/0.05/0.060

Table 5. The lookahead at θ0\theta_0: mean extrapolated drop dˉ\bar d (nats/token) on never-seen and forget authors, gated forget-specific drop RR, and the fraction of active positions on the forget set, the band’s own authors (null) and a second set of never-seen authors (test).

The lookahead lowers the gold tokens of the forget set (dˉ>1.4\bar d>1.4 nats) and raises them on never-seen authors (by 0.88–0.96 nats). The band transfers to the second set of never-seen authors (5.6–6.0% active, as designed), where the gated drop is 0.04–0.08 nats/token. For reference, θ0\theta_0’s log-probability of the gold forget tokens exceeds the retrained model’s by 2.45 nats/token (computed post hoc; the method never uses the retrained model). The heal does not remove the drift on holdout authors between 16 and 64 steps, so a longer heal would not remove it either; this is why SHIFT confines the teacher to forget tokens rather than trying to make the lookahead exact.

Implementation Details

Implementation Setup

Environment. Llama-3.2-1B runs on one NVIDIA GeForce RTX 5060 Ti (16 GB) or A100 (40 GB), Llama-3.2-3B on one A100, and Llama-3.1-8B and the MUSE model on four A100 GPUs with DeepSpeed ZeRO-3 (optimizer states on the CPU). All code builds on OpenUnlearning (Dorna et al. [8]) with PyTorch 2.4.1, Transformers 4.51.3 and DeepSpeed 0.15.4.

Training configuration. Every method follows the OpenUnlearning configuration. On TOFU we use AdamW (8-bit paged optimizer states), an effective batch size of 32, a learning rate of 10−510^{-5} with one epoch of linear warm-up and linear decay, weight decay 0.01, 10 epochs and bf16 weights; with Llama-3.1-8B every method keeps the input embeddings frozen. On MUSE we use an effective batch size of 64, a learning rate of 10−510^{-5} with a constant schedule, 10 epochs and no weight decay. Each method is run once (seed 0) and its final model is evaluated, with no checkpoint selection.

Hyperparameter Settings

Every baseline keeps the OpenUnlearning defaults below (γ\gamma: weight of the forget loss, α\alpha: weight of the retain loss, NLL on the retain set unless noted); we do not tune them per setting.

  • GradAscent. No method-specific hyperparameter.

  • GradDiff. γ=α=1\gamma= \alpha= 1.

  • NPO. β=0.1\beta= 0.1, γ=α=1\gamma= \alpha= 1.

  • SimNPO. β=4.5\beta= 4.5, margin δ=0\delta= 0, γ=0.125\gamma= 0.125, α=1\alpha= 1.

  • IdkDPO. β=0.1\beta= 0.1 with refusal answers as the preferred responses, γ=α=1\gamma= \alpha= 1.

  • RMU. Steering coefficient 2 at layer 7, all parameters trained, retain loss on hidden representations, γ=α=1\gamma= \alpha= 1.

  • UNDIAL. β=10\beta= 10, γ=1\gamma= 1, α=0\alpha= 0.

  • WGA. β=1\beta= 1, γ=α=1\gamma= \alpha= 1.

  • SatImp. β1=5\beta_1 = 5, β2=1\beta_2 = 1, γ=0.1\gamma= 0.1, α=1\alpha= 1.

  • CE-U. The first answer token is not trained.

  • PDU. Primal-dual updates with dual step size 1 and 5 warm-up epochs, retain tolerance ε=0\varepsilon= 0, γ=α=1\gamma= \alpha= 1.

  • SHIFT. Lookahead: ρ=12\rho= \frac{1}{2} and k=16k = 16 heal steps on retain answers (32 QA pairs per step, learning rate 10−510^{-5}). Teacher: the top-256 tokens of the model and the lookahead at each position (top-128 on MUSE), clip κ=10\kappa= 10, step sizes s=(12,1,1,1)s = (\frac{1}{2}, 1, 1, 1). Distillation: γ=α=1\gamma= \alpha= 1, a retain anchor on θ0\theta_0’s top-64 next-token distributions, learning rate 10−510^{-5} with a short warm-up in each round, four rounds of four epochs.

Algorithm 1 SHIFT
Require: target $\theta_0$, base $\theta_b$, $D_f$, $D_r$; $\rho$, $k$, $\kappa$, $s_0, \ldots, s_{N-1}$
1: cache $p_{\theta_0}$ on $D_r$ ▷ retain anchor
2: for $n = 0, \ldots, N - 1$ do
3: $\Pi \leftarrow \operatorname{Heal}_k(\theta_n - \rho(\theta_n - \theta_b))$ ▷ {xref}`Eq. (2)`
4: for all answer positions $t$ of $D_f$ do
5: $\delta_t \leftarrow \ell_{\Pi,t} - \ell_{n,t}$ ▷ on top tokens $V_t$
6: $p_{T,t} \leftarrow \operatorname{norm}_{V_t}\left(\ell_{n,t} + \operatorname{clip}_{\kappa}\left(\frac{s_n}{\rho}\delta_t\right)\right)$ ▷ {xref}`Eq. (6)`
7: cache $p_T$ on $D_f$; free $\Pi$
8: $\theta_{n+1} \leftarrow \operatorname{distill}$ from $\theta_n$ on $\mathcal{L}$ ▷ {xref}`Eq. (7)`
9: return $\theta_N$

Full TOFU Results

Table 6 complements Table 2 with the two components of Agg. SHIFT’s memorization and utility both stay close to the oracle’s, whereas on average every baseline gives up one of the two: most leave the forget set memorized (low Mem.), and GradAscent and CE-U collapse (Util. near 0).

MethodLlama-3.2-1B Agg.↑\uparrowLlama-3.2-1B Mem.↑\uparrowLlama-3.2-1B Util.↑\uparrowLlama-3.2-3B Agg.↑\uparrowLlama-3.2-3B Mem.↑\uparrowLlama-3.2-3B Util.↑\uparrowLlama-3.1-8B Agg.↑\uparrowLlama-3.1-8B Mem.↑\uparrowLlama-3.1-8B Util.↑\uparrow
FORGET 1%
Full0.1180.0640.7170.0240.0120.7630.0050.0020.738
Retain0.6130.5430.7030.6320.5410.7610.6190.5330.738
GradAscent0.4580.3350.7230.4090.2810.7500.6170.5420.716
GradDiff0.4400.3170.7180.3780.2510.7600.5650.4750.698
NPO0.4580.3360.7170.4190.2900.7500.5990.5040.737
SimNPO0.2180.1290.7250.1510.0840.7460.2500.1500.762
IdkDPO0.2940.1850.7120.1960.1120.7620.4780.3650.693
RMU0.5090.4140.6600.2120.1230.7680.5330.4190.734
UNDIAL0.4560.3350.7180.4460.3160.759<u>0.630</u>0.5380.759
WGA0.4340.3100.7230.4050.2770.7530.5520.4360.751
SatImp0.1390.0770.7220.0360.0190.7560.1060.0570.762
CE-U0.5570.4660.6920.5380.4170.7610.2820.7960.171
PDU<u>0.573</u>0.4880.692<u>0.559</u>0.4590.7140.0180.8100.009
SHIFT (ours)0.6190.5420.7200.6350.5460.7580.6390.5650.736
FORGET 5%
Full0.1540.0860.7070.0590.0300.7480.0160.0080.723
Retain0.6410.5770.7220.6480.5690.7510.6410.5670.736
GradAscent0.2140.6350.129<u>0.567</u>0.5010.6530.0000.9830.000
GradDiff0.5270.4750.5910.4870.3990.6240.1110.9880.059
NPO0.5480.5070.5970.5610.4770.6810.6660.6090.736
SimNPO0.2290.1360.7090.1650.0930.7410.3660.2440.734
IdkDPO0.4420.3250.6890.3970.2720.7340.5490.4530.696
RMU<u>0.575</u>0.5240.6380.5390.4210.7480.6120.7290.527
UNDIAL0.4800.3690.6840.5320.4110.7510.6490.5730.748
WGA0.4880.3700.7150.4530.3240.7480.5890.7330.493
SatImp0.1750.1000.7100.0800.0420.7400.1430.0800.713
CE-U0.0000.7850.0000.0000.7890.0000.0000.7880.000
PDU0.3350.6400.2270.0450.7370.0230.6310.6020.664
SHIFT (ours)0.6230.5580.7050.6390.5560.752<u>0.650</u>0.6120.693
FORGET 10%
Full0.1590.0890.7060.0630.0330.7520.0110.0060.727
Retain0.6430.5840.7150.6500.5740.7490.6490.5730.748
GradAscent0.0000.9920.0000.0000.9680.0000.0000.9820.000
GradDiff0.5210.4940.5510.4820.4740.4900.4230.9880.269
NPO<u>0.568</u>0.5460.5920.5840.5020.697<u>0.650</u>0.5970.714
SimNPO0.2350.1410.7090.1810.1030.7480.3830.2610.715
IdkDPO0.4880.3750.6990.4310.3030.7430.5310.4210.720
RMU0.2210.7100.1310.6460.6440.6480.4470.7730.314
UNDIAL0.4840.3740.6860.5460.4260.7570.6530.5830.742
WGA0.5060.3920.7110.4850.3600.7430.4360.7710.304
SatImp0.1820.1040.7090.0820.0440.7430.2070.1210.699
CE-U0.0000.7850.0000.0000.7940.0000.0000.9900.000
PDU0.4990.4100.6380.5200.4340.6470.3100.8020.192
SHIFT (ours)0.6240.5560.710<u>0.634</u>0.5500.7480.6310.5810.691

Table 6. Agg., Mem., and Util. on TOFU, all methods at learning rate 10−510^{-5} (one seed). Bold / <u>underlined</u>: best / second-best Agg. for each model and split; we do not mark Mem. or Util. alone, since collapsing maximizes Mem. (GradAscent and CE-U reach Util. ≈0\approx0).

Additional Analyses

Learning rate. Table 7 varies the learning rate of every method on Llama-3.2-1B forget05. At the default 10−510^{-5} the baselines stop short of the oracle (privacy leakage far below 0); from 2⋅10−52 \cdot10^{-5} up most of them cross it, while SHIFT stays near it.

Heal length. Table 8 varies the number of heal steps kk on Llama-3.2-1B forget05, with everything else as in Table 2. The results change little with kk: Agg. stays between 0.616 and 0.623 and the distance to the oracle between 0.015 and 0.020; k=16k = 16, the value of the main results, has the highest Agg.

Ablations at the larger learning rate. Table 9 repeats Table 4 at learning rate 2⋅10−52 \cdot10^{-5} (k=32k = 32, two epochs a round, three seeds). With the larger update, the heal matters: without it the distance to the oracle nearly doubles (0.011 vs. 0.006), and SHIFT is the closest variant.

Relearning Attack

The attack fine-tunes the unlearned model on the first half of the forget05 authors (TOFU has 20

consecutive questions per author) with learning rate 10−510^{-5} for one epoch after 3 warmup steps (batch 8, fp32 master weights, 8-bit AdamW), and evaluates the second half, which it never sees. Results are averaged over three attack seeds, which change the order of the relearning batches. Table 11 repeats the attack at learning rates 10−510^{-5} and 2⋅10−52 \cdot10^{-5} for one to three epochs (Figure 4). Stronger attacks also erode what the full model itself knows about the held-out half, so every method moves toward the oracle, but SHIFT stays the closest at every strength.

Methodlr 10−5lr\ 10^{-5} 1 ep.lr 10−5lr\ 10^{-5} 2 ep.lr 10−5lr\ 10^{-5} 3 ep.lr 2⋅10−5lr\ 2 \cdot10^{-5} 1 ep.lr 2⋅10−5lr\ 2 \cdot10^{-5} 2 ep.lr 2⋅10−5lr\ 2 \cdot10^{-5} 3 ep.
GradAscent0.0390.0170.0160.0170.0220.014
GradDiff0.0380.0480.0230.0290.0210.013
NPO<u>0.019</u><u>0.005</u>0.020<u>0.012</u>0.015<u>0.010</u>
SimNPO0.1600.0920.0650.0550.0380.032
IdkDPO0.0250.0290.0260.020<u>0.009</u>0.011
RMU0.1040.0770.0480.0510.0340.027
UNDIAL0.0490.0370.0200.0200.0210.021
WGA0.0300.0470.0180.0210.0200.015
SatImp0.1980.1110.0770.0610.0360.024
CE-U0.0450.0220.0400.0380.0310.027
PDU0.0330.0330.0320.0360.0160.018
SHIFT (ours)0.0080.0020.0120.0090.0080.008

Table 11. Distance to the equally attacked oracle (mean absolute difference of ES and ROUGE-L on the held-out half) across attack strengths, three attack seeds each. Bold / <u>underlined</u>: closest / second closest.

Use of AI Assistants

In preparing this paper, we made limited use of AI Assistants such as ChatGPT only to help refine wording, correct grammar, and enhance clarity. All core research contributions, including conceptual development, theoretical analysis, methodology design and implementation, experimental evaluations, and paper organization and composition, were conducted entirely by the authors.

MethodBefore attack ESBefore attack RLAfter attack ESAfter attack RLΔ\Deltaoracle↓\downarrow
Full0.6840.8310.3160.5370.214
Retain0.0600.3720.0700.353–
GradAscent0.0290.0000.0850.2900.039
GradDiff0.0480.2440.1110.3890.038
NPO0.0710.2880.0760.3210.019
SimNPO0.3010.5910.2530.4890.160
IdkDPO0.1100.1540.1140.3610.025
RMU0.0290.1160.1950.4360.104
UNDIAL0.0470.3100.1030.4180.049
WGA0.0620.3050.1120.3710.030
SatImp0.5920.7850.3080.5120.198
CE-U0.0290.0010.0780.2710.045
PDU0.0290.0730.1230.3400.033
SHIFT (ours)0.0560.3650.0650.3650.008

Table 10. Relearning attack on TOFU forget05 (Llama-3.2-1B, models unlearned at 2⋅10−52 \cdot10^{-5}): extraction strength (ES) and ROUGE-L (RL) on the held-out half of the forget authors, before and after relearning on the other half (mean of three attack seeds). Δ\Deltaoracle: mean absolute difference of ES and RL from the relearned Retain model; bold / <u>underlined</u>: closest / second closest.

References

  1. [1]Jordan T. Ash and Ryan P. Adams. 2020. On warm-starting neural network training. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/1910.08475
  2. [2]Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In USENIX Security Symposium.arxiv.org/abs/2012.07805
  3. [3]Yi Chen, Hanna Hsieh, Shuhong Liu, Chuanbo Hua, Zihan Ma, Kun Wang, and Joo-Young Kim. 2026. Distance is not enough: Forget-retain alignment gap predicts LLM relearning robustness. arXiv preprint arXiv:2608.25429.DOI
  4. [4]Vikram S. Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan Kankanhalli. 2023. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In AAAI Conference on Artificial Intelligence, volume 37, pages 7210–7217.
  5. [5]Aghyad Deeb and Fabien Roger. 2024. Do unlearning methods remove information from language model weights? arXiv preprint arXiv:2410.08827.arxiv.org/abs/2410.08827
  6. [6]Federico Di Gennaro, Alexander Shevchenko, and Fanny Yang. 2026. Fast unlearning at scale via margin self-correction. arXiv preprint arXiv:2606.02920.arxiv.org/abs/2606.02920
  7. [7]Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. 2025. UNDIAL: Self-distillation with adjusted logits for robust unlearning in large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8827–8840.arxiv.org/abs/2402.10052
  8. [8]Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, J. Zico Kolter, Zachary C. Lipton, and Pratyush Maini. 2025. OpenUnlearning: Accelerating LLM unlearning via unified benchmarking of methods and metrics. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track.arxiv.org/abs/2506.12618
  9. [9]Ronen Eldan and Mark Russinovich. 2023. Who’s Harry Potter? approximate unlearning in LLMs. arXiv preprint arXiv:2310.02238.arxiv.org/abs/2310.02238
  10. [10]Taha Entesari, Arman Hatami, Rinat Khaziev, Anil Ramakrishna, and Mahyar Fazlyab. 2025. Constrained entropic unlearning: A primal-dual framework for large language models. In Advances in Neural Information Processing Systems (NeurIPS).
  11. [11]Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. 2025. Simplicity prevails: Rethinking negative preference optimization for LLM unlearning. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2410.07163
  12. [12]Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. 2024. SalUn: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2310.12508
  13. [13]Kristian Georgiev, Roy Rinberg, Sung Min Park, Shivam Garg, Andrew Ilyas, Aleksander Madry, and Seth Neel. 2025. Machine unlearning via simulated oracle matching. In International Conference on Learning Representations (ICLR).
  14. [14]Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).arxiv.org/abs/1911.04933
  15. [15]Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. 2018. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning (ICML).arxiv.org/abs/1802.08246
  16. [16]Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. 2025. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy. In IEEE Conference on Secure and Trustworthy Machine Learning (SaTML).arxiv.org/abs/2403.01218
  17. [17]Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. 2025. Unlearning or obfuscating? jogging the memory of unlearned LLMs via benign relearning. In International Conference on Learning Representations (ICLR). James Y. Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2025. Offset unlearning for large language models. Transactions on Machine Learning Research.
  18. [18]Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hanneleh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2212.04089
  19. [19]Arthur Jacot, Franck Gabriel, and Clément Hongler. 2018. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/1806.07572
  20. [20]Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajunagen Logeswaran, and Minjoon Seo. 2023. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389–14408.
  21. [21]Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Rao Kompella, Sijia Liu, and Shiyu Chang. 2024. Reversing the forget-retain objectives: An efficient LLM unlearning framework from logit difference. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2406.08607
  22. [22]Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. 2023. Towards unbounded machine unlearning. In Advances in Neural Information Processing Systems (NeurIPS).
  23. [23]Bruce W. Lee, Addie Foote, Alex Infanger, Leni Shor, Harish Kamath, Jacob Goldman-Wetzler, Bryce Woodworth, Alex Cloud, and Alexander Matt Turner. 2025. Distillation robustifies unlearning. In Advances in Neural Information Processing Systems (NeurIPS).
  24. [24]Kemou Li, Qizhou Wang, Yue Wang, Fengpeng Li, Jun Liu, Bo Han, and Jiantao Zhou. 2026. LLM unlearning with LLM beliefs. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2510.19422
  25. [25]Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew Bo Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, and 27 others. 2024. The WMDP benchmark: Measuring and reducing malicious use with unlearning. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 28525–28550. PMLR.arxiv.org/abs/2403.03218
  26. [26]Bo Liu, Qiang Liu, and Peter Stone. 2022. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243–254. PMLR.arxiv.org/abs/2203.12817
  27. [27]Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuyang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. 2025. Rethinking machine unlearning for large language models. Nature Machine Intelligence, 7(2):181–194.
  28. [28]Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. 2024. Eight methods to evaluate robust unlearning in LLMs. arXiv preprint arXiv:2402.16835.arxiv.org/abs/2402.16835
  29. [29]Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. 2024. TOFU: A task of fictitious unlearning for LLMs. In Conference on Language Modeling (COLM).arxiv.org/abs/2401.06121
  30. [30]Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. 2023. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning (ICML).arxiv.org/abs/2210.05643
  31. [31]Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D. Manning. 2024. An emulator for fine-tuning large language models using small language models. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2310.12962
  32. [32]Siqiao Mu and Diego Klabjan. 2025. Rewind-to-delete: Certified machine unlearning for nonconvex functions. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2409.09778
  33. [33]Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. 2023. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Advances in Neural Information Processing Systems (NeurIPS).
  34. [34]Amit Peleg, Naman Deep Singh, Naama Pearl, Bibhabasu Mohapatra, and Matthias Hein. 2026. Forget narrowly, retain broadly: Unlearning as an asymmetric generalization problem. arXiv preprint arXiv:2607.09236.arxiv.org/abs/2607.09236
  35. [35]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2305.18290
  36. [36]Keivan Rezaei, Mehrdad Saberi, Abhilasha Ravichander, and Soheil Feizi. 2026. Revisiting the past: Data unlearning with model state history. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2506.20941
  37. [37]Saisab Sadhu, Pratinav Seth, and Vinay Kumar Sankarapu. 2026. Forgetting that sticks: Quantization-permanent unlearning via circuit attribution. arXiv preprint arXiv:2605.15138.arxiv.org/abs/2605.15138
  38. [38]Weijia Shi, Jaechang Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah Smith, and Chiyuan Zhang. 2025. Muse: Machine unlearning six-way evaluation for language models. In International Conference on Learning Representations, volume 2025, pages 27797–27818.arxiv.org/abs/2407.06460
  39. [39]Zexuan Sun and Garyesh Raskutti. 2026. A theoretical framework for LLM fine-tuning using early stopping for non-random initialization. arXiv preprint arXiv:2602.13942.arxiv.org/abs/2602.13942
  40. [40]Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, and Kilian Q. Weinberger. 2025. Rethinking LLM unlearning objectives: A gradient perspective and go beyond. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2502.19301
  41. [41]Jiawei Wu and Doudou Zhou. 2026. Unlearning what matters: Token-level attribution for precise language model unlearning. arXiv preprint arXiv:2605.00364.arxiv.org/abs/2605.00364
  42. [42]Zeguan Xiao, Xuanzhe Xu, Yun Chen, Yong Wang, Jian Yang, Yanqing Hu, and Guanhua Chen. 2026. Robust llm unlearning against relearning attacks: The minor components in representations matter. arXiv preprint arXiv:2605.11685.arxiv.org/abs/2605.11685
  43. [43]Han Yan, Zheyuan Liu, and Meng Jiang. 2026. Dual-space smoothness for robust and balanced LLM unlearning. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2509.23362
  44. [44]Bo Yang. 2025. CE-U: Cross entropy unlearning. arXiv preprint arXiv:2503.01224.arxiv.org/abs/2503.01224
  45. [45]Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. 2025. Exploring criteria of loss reweighting to enhance LLM unlearning. In International Conference on Machine Learning (ICML).arxiv.org/abs/2505.11953
  46. [46]Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024. Large language model unlearning. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2310.10683
  47. [47]Xiangyu Yin, Jiaxu Liu, Zhen Chen, and Chih-Hong Cheng. 2026. Crossing the margin cliff: Toward relearn-robust LLM unlearning via margin calibration. arXiv preprint arXiv:2607.27836.arxiv.org/abs/2607.27836
  48. [48]Sangyeon Yoon, Hyesoo Hong, Wonje Jeung, and Albert No. 2026a. Rethinking benign relearning: Syntax as the hidden driver of unlearning failures. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2602.03379
  49. [49]Sangyeon Yoon, Yeachan Jun, and Albert No. 2026b. Position: The term “machine unlearning” is overused in LLMs. In International Conference on Machine Learning (ICML), Position Paper Track.arxiv.org/abs/2606.27379
  50. [50]Jingyuan Zhang, Yucheng Bai, Peixi Wen, Zhehao Huang, Zhengbao He, Hanling Tian, Xinwen Cheng, Haiyin Ran, and Xiaolin Huang. 2026. SAGE: Retain-aware post-hoc sanitization of final unlearning vector. arXiv preprint arXiv:2606.18309.arxiv.org/abs/2606.18309
  51. [51]Michael R. Zhang, James Lucas, Geoffrey Hinton, and Jimmy Ba. 2019. Lookahead optimizer: k steps forward, 1 step back. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/1907.08610
  52. [52]Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. In Conference on Language Modeling (COLM).arxiv.org/abs/2404.05868
  53. [53]Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. 2025. Catastrophic failure of LLM unlearning via quantization. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2410.16454
  54. [54]Yisheng Zhong, Zhengbang Yang, and Zhuangdi Zhu. 2026. DUET: Distilled LLM unlearning from an efficiently contextualized teacher. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2601.21283

Paper details

vlnc:W_9f4jzpf6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

ChatGPT

View statement
Looking for related papers…
Contents