WorksComputer Science
Unlearning by Extrapolating a Shrink-and-Heal Lookahead
Abstract
Unlearning aims to make a large language model behave like one retrained without its forget set. Prevailing likelihood-suppression objectives, however, lack an explicit forgetting target: they continually reward lower forget-set likelihood, making forgetting sensitive to the optimization budget. We address this limitation by estimating the direction toward the oracle without training it. Our key idea is to shrink the fine-tuned model toward its pre-fine-tuning weights and briefly re-learn the retain set, thereby removing the forget-specific contribution; in the linearized regime, this shrink-and-heal lookahead moves the model a fixed fraction of the way to the retrained model, its unique fixed point. Our method, SHIFT, extrapolates the lookahead's per-token changes into distillation targets and iteratively refines them. Evaluated against eleven baselines under a unified OpenUnlearning protocol, SHIFT achieves the highest aggregate score across all TOFU settings and the closest oracle alignment, 3.7x closer on average than the next-best method; on MUSE, it preserves the most retain knowledge. Under larger optimization budgets, baselines overshoot the oracle, while SHIFT remains near it, even under relearning attacks.
Introduction
Large language models (LLMs) acquire substantial knowledge through fine-tuning, but may also memorize sensitive or unwanted training data [2]. When such data must be removed, retraining from scratch is often prohibitively expensive. This motivates machine unlearning, which aims to remove the influence of a forget set while preserving what the model learned from the remaining retain set [27]. Ideally, the unlearned model should behave like an oracle trained on the retain set alone. This oracle serves as the reference for evaluating unlearning in established benchmarks [29, 38], and has been advocated as the defining criterion for unlearning [49].
Most existing unlearning methods are built on gradient ascent over the forget set [20, 46], often combined with retain-set regularization to preserve model utility [26]. Subsequent work bounds or reshapes the ascent objective to prevent model collapse, such as NPO [52], SimNPO [11], and token reweighting [40, 45], extends the suppression to the model’s own high-confidence alternatives [24], or replaces it with a distillation target [4, 9, 7, 54].
Despite these refinements, a fundamental yet underexplored limitation remains in likelihood-suppression approaches: they lack an explicit target for forgetting. By rewarding lower forget-set likelihood, these objectives do not generally admit the oracle as a stationary point, allowing optimization to proceed beyond the desired forgetting level. Consequently, the extent of forgetting becomes sensitive to the learning rate and number of training steps. As illustrated in Figure 1(a), varying the optimization budget can shift the same method from insufficient forgetting to over-forgetting, where forget-set examples become less familiar than never-seen data, potentially exposing their former membership [16]. This sensitivity is not merely an optimization issue: the objective itself provides no criterion for stopping at the oracle.

Figure 1. (a) Without a target, forgetting follows the budget. (b) The shrink-and-heal lookahead points to the oracle. Figure 1: (a) Privacy leakage on TOFU forget05 (Llama-3.2-1B) across learning rates under a fixed 10-epoch budget (0: oracle; positive: the forget set looks less familiar than never-seen data). Likelihood-suppression methods overshoot the oracle, whereas SHIFT stabilizes near it. (b) In the linearized regime (Prop. 1), shrinking toward the base and healing on the retain set removes a fraction of the forget-specific contribution; SHIFT extrapolates and iterates this step, which stops at the oracle , whereas likelihood suppression has no reason to stop there.
An explicit target could address this limitation, and the best target is the oracle itself, which is exactly what one cannot afford to train. Our key observation is that a fine-tuned model already contains a cheap estimate of where the oracle lies. Specifically, shrinking the model toward its pre-fine-tuning weights partially reverses everything learned during fine-tuning, while briefly re-learning the retain set restores knowledge supported by the retained data; what remains removed is the contribution attributable to the forget set. We call this operation the shrink-and-heal lookahead. As shown in Figure 1(b), under linearized fine-tuning dynamics, it moves the model a fixed fraction of the way toward the oracle, which is its unique fixed point.
Building on this observation, we propose SHIFT (Shrink–heal Iteration toward the Fixed poinT), an unlearning framework that uses the lookahead to guide prediction-level updates. Rather than adopting the lookahead model directly, SHIFT measures its induced changes in forget-set predictions and extrapolates them to estimate an oracle-directed update. The resulting target is then distilled into the model on forget tokens, while its retain-set predictions are anchored to preserve utility. By repeatedly recomputing the lookahead from the updated model, SHIFT progressively corrects its approximation to the oracle through a Newton-style iteration.
We evaluate SHIFT on TOFU ([29]) and MUSE ([38]) against eleven baselines under a unified protocol. Across the TOFU settings, SHIFT achieves the closest forget-set statistics to the oracle, closer on average than the next-best method. It also obtains the highest aggregate score among methods that avoid over-forgetting. On MUSE, SHIFT preserves the most retain knowledge, while removing more memorization than NPO and SimNPO. Furthermore, SHIFT maintains forgetting behavior close to the oracle across different learning rates, demonstrating reduced sensitivity to the optimization budget.
Contributions. (1) We identify a fundamental limitation of likelihood-suppression unlearning: without an explicit target, its forgetting behavior depends heavily on the optimization budget (§ 3). (2) We introduce the shrink-and-heal lookahead and prove that, in the linearized regime, it steps a fixed fraction toward the oracle, which is its unique fixed point, and that distilling its extrapolated per-token change yields a Newton-style update toward the oracle (§ 4). (3) We develop SHIFT and demonstrate on TOFU and MUSE that it achieves the strongest oracle alignment and the best overall performance among methods that avoid over-forgetting (§ 5).
Related Work
Likelihood suppression. Early unlearning methods use gradient ascent on the forget set ([20, 46]), often combined with retain-set regularization ([26]). Subsequent approaches introduce broader or reweighted objectives ([52, 11, 40, 45]), refusal targets ([29]), representation misdirection ([25]), and constrained optimization ([10]). Recent work also explores adaptive stopping criteria based on the model’s own predictions ([6]). Despite their different mechanisms, they do not explicitly target the behavior of the retain-only oracle, which provides a principled reference for unlearning ([49, 16]).
Targets and distillation. Distillation-based methods introduce alternative targets to guide forgetting. UN-DIAL reduces the gold-token logit by a fixed offset [7], while “Who’s Harry Potter” subtracts the excess of a model reinforced on the forget data [9]. Other approaches construct teachers using incompetent models [4], in-context prompts [54], or assistants trained to memorize the forget set [21]. In contrast, SHIFT’s teacher is calibrated by the retain set and has the oracle as its fixed point.
Checkpoint arithmetic and resets. The closest related approach is MSA [36], which fine-tunes an earlier checkpoint on the forget set and subtracts the resulting parameter difference scaled by a tuned coefficient. SHIFT instead estimates the unlearning direction through shrinkage and retain-set recovery, without training on the forget set, and derives its extrapolation scale from the oracle fixed-point property. Related techniques, such as shrink-and-perturb [1] and rewinding [32] reset model weights, whereas SHIFT uses the reset to construct distillation targets. Unlike the Lookahead optimizer [51], which pulls weights toward a few fast steps, our lookahead is a model-level shrink-and-heal operation rather than an optimizer (see Appendix § 7 for further discussion).
Preliminaries and Motivation
Problem definition and evaluation. A base model is fine-tuned on , producing the target model , where and denote the forget and retain sets, respectively. Given , , and , an unlearning method aims to obtain a model that behaves like the oracle , defined as fine-tuned on alone. The oracle is unavailable during unlearning. For a sequence , let denote the next-token log-probabilities at position . Most likelihood-suppression methods optimize an objective of the form:
where is the forget-set negative log-likelihood or its bounded transform, preserves retain-set performance, and , control their contributions. To evaluate unlearning, on TOFU [29], we follow OpenUnlearning [8] and report Mem., the harmonic mean of extraction strength, exact memorization, paraphrased-answer probability, and truth ratio on ; Util., the harmonic mean of model utility and fluency; and Agg., the harmonic mean of Mem. and Util. However, Agg. rewards forgetting monotonically even beyond the oracle, whose own Agg. on forget05 is 0.641. Since unlearning is to match the retained model [49] and excessive forgetting can itself reveal prior membership [16], we also measure distance to the oracle. Specifically, we compute the absolute difference between the unlearned model and the oracle for four forget-set statistics: Mem., truth ratio (TR), extraction strength (ES), and ROUGE-L recall of the answers, and report their average. We further report privacy leakage (PL) that is the relative gap in membership-inference AUC between and [38]: PL is 0 for the oracle, close to before unlearning, and positive under-over-forgetting. On MUSE [38] we report verbatim and knowledge memorization on and knowledge memorization on .
Motivation: forgetting without a target. To examine how far likelihood suppression forgets, we evaluate six such methods on TOFU forget05 using Llama-3.2-1B. Each method is trained for the standard 10 epochs across five learning rates from to . The results are shown in Figure 1(a), with detailed measurements in Appendix Table 7. Two key observations emerge:
Observation 1: forgetting is highly sensitive to the budget. Five of the six methods reach or surpass the oracle’s forgetting level within the tested learning rates. NPO’s privacy leakage increases from to , and WGA’s from to ; GradDiff, RMU, and PDU already over-forget at a learning rate of , and SimNPO never reaches the oracle. No method lands near the oracle for a reason internal to the method.
Observation 2: aggregate scores favor over-forgetting. The strongest baseline scores are achieved beyond the oracle: WGA reaches an Agg. of 0.692 at PL , while NPO peaks at PL . Thus, tuning for Agg. alone pushes a method past the oracle rather than toward it.
The underlying reason lies in the optimization objectives themselves. Their forget-set terms reward decreasing the likelihood of without identifying the oracle as a stopping point; bounded variants such as NPO [52] attenuate the suppression but do not ensure convergence to the oracle. The extent of forgetting is thus governed by the number of training epochs and the learning rate. An unlearning objective should instead explicitly target an estimate of the oracle.
Shift: Unlearning toward an Estimated Oracle
The oracle provides a target for unlearning, but training it would defeat the purpose of unlearning. Instead, we use the fine-tuned model , the base model , and the retain set to estimate the direction toward the oracle at low computational cost. We first introduce the shrink-and-heal lookahead and analyze its fixed point (§ 4.1). We then extrapolate its prediction changes to construct distillation targets on forget tokens (§ 4.2) and iteratively refine these targets as the model evolves (§ 4.3). Figure 2 illustrates one round.

Figure 2. One round of SHIFT. (1) Shrink toward the pre-fine-tuning weights and re-fit the retain set. (2) On forget answers, the lookahead lowers the memorized token and raises the alternatives the retain set supports. (3) Extrapolating by gives the teacher, the oracle’s next-token distribution in the linearized regime. (4) Distill it on forget tokens, anchored to on retain data; the next round looks ahead from the new model.
Shrink-and-Heal Lookahead
Fine-tuning displaces the base model by that encodes both and . Pulling the model back toward by a fraction removes a -share of everything that fine-tuning added. Re-fitting the retain set for a few steps then restores the component supported by . What remains removed is the part of that cannot explain, i.e., what only taught. We formalize this operation as the shrink-and-heal lookahead:
where denotes optimization steps on . To characterize this operation, we consider the linearized fine-tuning regime ([19, 30, 33]), which has recently been extended to attention networks initialized from pre-trained weights ([39]). In this model, the logits satisfy , and gradient descent from reaches the minimum-norm displacement that fits its data ([15]). Let stack the retain-set Jacobians and let project onto their row space. and represent the retain-supported and forget-specific components of the fine-tuning displacement, respectively.
Proposition 1 (The lookahead is a -step toward the oracle). If the heal runs to convergence, then
More generally, every that fits satisfies , so the oracle is the unique fixed point of .
The heal moves only within the row space of , so it restores exactly the retain-supported component of what the shrink removed; the net move is a fraction of the forget-specific component of fine-tuning (Figure 1b); proof in Appendix § 8. The lookahead therefore points to the oracle without training it, and needs no knowledge of which parameters encode . At the oracle, it produces no further change, which is the property that likelihood suppression lacks (§ 3).
What the lookahead does in practice. Real fine-tuning is not linear and the heal is short. We therefore examine the lookahead on TOFU forget10 with Llama-3.2-1B (, ; Appendix § 9) before building on it, comparing gold-token log-probabilities before and after the lookahead, extrapolated by . (1) The lookahead isolates the forget set: its forget-specific drop is 1.9 nats per token, approximately 80% of the true gap between and the oracle (2.45 nats, computed post hoc). (2) It also moves unrelated text: on never-seen authors, it raises the gold tokens by 0.96 nats, as shrinking makes less over-confident in general, and this drift does not shrink as increases from 16 to 64.
Design requirements. These observations give three requirements. First, the lookahead should be read, not kept: it covers only a fraction of the way, and changes predictions everywhere, including where unlearning should not act. Second, its signal should act only where unlearning applies, the forget tokens, while the rest of the model’s behavior is held fixed. Third, the estimate should be refreshed: since the lookahead is accurate only locally, it should be recomputed from the updated model. The next two subsections instantiate them.
From Lookahead to Teacher
Reading the lookahead in output space. Following the first requirement, we never use as a model, but instead only compare its predictions with those of the current model. Proposition 1 shows how to turn this comparison into the oracle’s predictions. Extrapolating the lookahead step by lands on the oracle: . Since logits are linear in , we can obtain:
Therefore, two forward passes suffice to recover the oracle’s logits, without the oracle. In practice, we store log-probabilities rather than logits, as the softmax is invariant to the per-position normalization (see Lemma 1 in Appendix B).
Extrapolated teacher. At each answer position of a forget example, let and be the log-probabilities of the current model and its lookahead, restricted to , the union of both models’ top-256 tokens. We form the teacher by:
which renormalizes the probabilities within and leaves the current model’s mass outside unchanged. The step scales the extrapolation: reproduces the lookahead, and takes the full step in Eq. (5). By Lemma 1, the teacher at is the oracle’s next-token distribution on in the linearized regime. We clip each extrapolated change at nats, since dividing by amplifies noise in the lookahead.
The change does not only lower the memorized token; it also raises the alternatives that a model without would predict (Figure 2, panel 2). This distinction matters because suppressing only the gold token can transfer its probability mass to semantically related continuations [24]. Unlike methods that apply a fixed reduction to the gold-token logit [7], our teacher tells the model both how much to forget at each position and what to say instead, as estimated from the retain set.
Distillation and Iteration
Distillation with a retain anchor. Following the second requirement, the teacher acts only on forget tokens, and every retain example anchors the model to ’s own predictive distribution:
Unlike the likelihood-suppression objectives, this loss is minimized at the teacher, so additional optimization brings the model closer to the estimated oracle rather than past it. The KL anchor preserves the full distribution on retain data, not only its gold tokens [52, 7], and also holds the retain knowledge that a finite heal fails to restore (Appendix § 8). We cache ’s top-64 tokens per retain position once, avoiding the need to keep a separate reference model in memory. Although the loss acts on outputs, in the linearized regime it carries out exactly the parameter-space step that the lookahead defines:
Proposition 2 (Distillation implements the step). Let fit , and let the teacher be (6) at step without clipping or truncation. Gradient descent on the logit-matching form of (7) from converges to ; at , this point is the oracle .
The proof is provided in Appendix § 8. Both loss terms target the prediction outputs of the extrapolated parameter point, and gradient descent updates the parameters only within the row space of the training Jacobian, which contains the desired step.
Iterating the update. Following the third requirement, each round takes a new lookahead from the model it has just produced. In the linearized setting, this iteration provably converges to the oracle:
Corollary 1 (Iteration). Let with . Every iterate fits , its retain-supported part stays equal to , and its forget-specific part shrinks geometrically:
Hence for every , and reaches in one step.
The proof is provided in Appendix § 8. The step is thus a Newton step on the fixed-point equation , and by Proposition 2 each round of distillation realizes one such step. On real models, each new lookahead corrects what the previous linear estimate missed. We use a fixed schedule : the first round takes the lookahead itself without extrapolation, as the lookahead moves the most and extrapolating that first change is the least reliable step (the mean teacher drop at is 0.62 nats at but 2.47 at ). The complete procedure is summarized in Algorithm 1.
What the idealization leaves out. Each gap between the analysis and practice is covered by one component. A finite heal restores retain knowledge only partially (Appendix § 8), and the lookahead drifts on unrelated text: the teacher therefore acts only on forget tokens, and the retain anchor holds the rest. Real fine-tuning is nonlinear, so the lookahead is accurate only locally: the iteration re-linearizes every round.
Experiments
Setup
Benchmarks and models. We use TOFU ([29]) with Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct on the forget01, forget05 and forget10 splits, with the OpenUnlearning target and oracle models ([8]); and MUSE-News ([38]) with Llama-2-7B. The bases are the corresponding instruction-tuned (TOFU) or pre-trained (MUSE) checkpoints.
Baselines. We compare SHIFT against eleven unlearning baselines: GradAscent, GradDiff, NPO, SimNPO, DPO with refusal targets (IdkDPO) ([35]), RMU ([25]), UNDIAL ([7]), WGA ([40]), SatImp ([45]), CE-U ([44]) and PDU ([10]), with the OpenUnlearning implementations.
Protocol. Every baseline follows the OpenUnlearning training configuration ([8]) with its default hyperparameters: learning rate , 10 epochs, effective batch 32 on TOFU, bf16 weights, one seed, and the final model (no checkpoint selection). SHIFT uses the same learning rate, batch and model selection, with , , , and four rounds of four epochs. This budget is chosen once, on Llama-3.2-1B forget05, and used unchanged for every model, split and
benchmark in the main results. Appendix § 10 gives the implementation details.
Main Results
SHIFT ends closest to the oracle. The goal of unlearning is the retrained model, so we first ask how close each method ends to it. The results are reported in Table 1. SHIFT is the closest in every setting but Llama-3.1-8B forget01 and, on average, on each of the four statistics: within 0.019 of the oracle, 3.7× closer than the next method (NPO), with memorization and utility on average within 0.019 and 0.016 of the oracle’s (Appendix Table 6). TOFU’s own test against the oracle agrees: SHIFT has the highest forget quality on average (−2.07, against −3.29 for NPO and −3.30 for RMU). The baselines miss the oracle in one of two ways. Most stop short of it: NPO, UNDIAL, WGA, IdkDPO, SimNPO and SatImp leave Mem. 0.09–0.49 below the oracle’s. The others lose utility: Util. falls 0.16–0.56 below the oracle’s for RMU, GradDiff, PDU, GradAscent and CE-U.
| Method | ||||||
| GradAscent | 0.248 | 0.276 | 0.066 | 0.218 | 0.202 | −89.16 |
| GradDiff | 0.208 | 0.195 | 0.085 | 0.131 | 0.155 | −47.38 |
| NPO | 0.091 | 0.045 | 0.059 | 0.086 | 0.071 | −3.29 |
| SimNPO | 0.414 | 0.107 | 0.470 | 0.339 | 0.332 | −11.49 |
| IdkDPO | 0.250 | 0.071 | 0.262 | 0.275 | 0.215 | −6.67 |
| RMU | 0.158 | 0.048 | 0.088 | 0.149 | 0.111 | −3.30 |
| UNDIAL | 0.131 | 0.090 | 0.091 | 0.064 | 0.094 | −12.39 |
| WGA | 0.202 | 0.089 | 0.127 | 0.148 | 0.141 | −11.37 |
| SatImp | 0.491 | 0.110 | 0.713 | 0.450 | 0.441 | −12.26 |
| CE-U | 0.217 | 0.126 | 0.048 | 0.302 | 0.173 | −41.69 |
| PDU | 0.136 | 0.061 | 0.039 | 0.190 | 0.106 | −12.30 |
| SHIFT (ours) | 0.019 | 0.021 | 0.006 | 0.029 | 0.019 | −2.07 |
Table 1. Distance to the oracle, , for Mem., truth ratio (TR), extraction strength (ES) and ROUGE-L (RL) on the forget set, and TOFU’s forget quality (FQ: of a test against the oracle’s truth ratios; 0 is the oracle), averaged over the settings of the Avg. column of Table 2. Bold / underlined: best / second best.
SHIFT has the best aggregate score. SHIFT has the highest Agg. in six of the nine settings (Table 2), 0.009–0.076 above the strongest baseline, and the best average (0.633 vs. 0.561 for NPO). Two of the exceptions go to methods that over-forget there: RMU on Llama-3.2-3B forget10 (privacy leakage +30) and NPO on Llama-3.1-8B forget05 (+44). On Llama-3.1-8B forget10, UNDIAL leads (0.653 vs. 0.631), while SHIFT stays three times closer to the oracle (0.012 vs. 0.036). SHIFT earns its scores by matching the oracle: its memorization stays within 0.045 of the oracle’s in every setting, even where its Agg. exceeds the oracle’s own. No baseline is competitive everywhere: the strongest one changes across settings (PDU, RMU, NPO, GradAscent, UNDIAL), and one to four fall below 0.2 in each. Our baseline scores match those reported for tuned baselines (NPO: 0.55 and 0.57 on Llama-3.2-1B forget05 and forget10, against 0.54 and 0.58 in Li et al. [24]).
| Method | Llama-3.2-1B forget01 | forget05 | forget10 | Llama-3.2-3B forget01 | forget05 | forget10 | Llama-3.1-8B forget01 | forget05 | forget10 | Avg. |
| Full | 0.118 | 0.154 | 0.159 | 0.024 | 0.059 | 0.063 | 0.005 | 0.016 | 0.011 | 0.068 |
| Retain | 0.613 | 0.641 | 0.643 | 0.632 | 0.648 | 0.650 | 0.619 | 0.641 | 0.649 | 0.637 |
| GradAscent | 0.458 | 0.214 | 0.000 | 0.409 | 0.567 | 0.000 | 0.617 | 0.000 | 0.000 | 0.252 |
| GradDiff | 0.440 | 0.527 | 0.521 | 0.378 | 0.487 | 0.482 | 0.565 | 0.111 | 0.423 | 0.437 |
| NPO | 0.458 | 0.548 | 0.568 | 0.419 | 0.561 | 0.584 | 0.599 | 0.666 | 0.650 | 0.561 |
| SimNPO | 0.218 | 0.229 | 0.235 | 0.151 | 0.165 | 0.181 | 0.250 | 0.366 | 0.383 | 0.242 |
| IdkDPO | 0.294 | 0.442 | 0.488 | 0.196 | 0.397 | 0.431 | 0.478 | 0.549 | 0.531 | 0.423 |
| RMU | 0.509 | 0.575 | 0.221 | 0.212 | 0.539 | 0.646 | 0.533 | 0.612 | 0.447 | 0.477 |
| UNDIAL | 0.456 | 0.480 | 0.484 | 0.446 | 0.532 | 0.546 | 0.630 | 0.649 | 0.653 | 0.542 |
| WGA | 0.434 | 0.488 | 0.506 | 0.405 | 0.453 | 0.485 | 0.552 | 0.589 | 0.436 | 0.483 |
| SatImp | 0.139 | 0.175 | 0.182 | 0.036 | 0.080 | 0.082 | 0.106 | 0.143 | 0.207 | 0.128 |
| CE-U | 0.557 | 0.000 | 0.000 | 0.538 | 0.000 | 0.000 | 0.282 | 0.000 | 0.000 | 0.153 |
| PDU | 0.573 | 0.335 | 0.499 | 0.559 | 0.045 | 0.520 | 0.018 | 0.631 | 0.310 | 0.388 |
| SHIFT (ours) | 0.619 | 0.623 | 0.624 | 0.635 | 0.639 | 0.634 | 0.639 | 0.650 | 0.631 | 0.633 |
Table 2. Agg.↑ on TOFU, every method at the OpenUnlearning learning rate (one seed). Agg. is the harmonic mean of Mem. (forgetting of ) and Util. (retained utility), so it is high only when a method both forgets and keeps utility. Bold / underlined: best / second-best Agg. of each setting. Full: before unlearning; Retain: the oracle. Mem. and Util. are in Appendix Table 6.
MUSE. On MUSE-News (results are reported in Table 3), where the forget set consists of raw text and the base is a pre-trained model, the methods that push verbatim memorization below the retrained oracle (GradAscent, GradDiff, UNDIAL, WGA) substantially degrade retain knowledge
| Method | VerbMem | KnowMem-F | KnowMem-R |
| Target | 0.579 | 0.644 | 0.555 |
| Retrain | 0.202 | 0.328 | 0.560 |
| GradAscent | 0.000 | 0.000 | 0.000 |
| GradDiff | 0.022 | 0.340 | 0.121 |
| NPO | 0.441 | 0.627 | 0.496 |
| SimNPO | 0.519 | 0.621 | 0.496 |
| RMU | 0.276 | 0.461 | 0.476 |
| UNDIAL | 0.138 | 0.130 | 0.204 |
| WGA | 0.079 | 0.245 | 0.228 |
| SHIFT (ours) | 0.235 | 0.572 | 0.539 |
Table 3. MUSE-News, Llama-2-7B, learning rate (baselines: 10 epochs; SHIFT: 4 rounds of 4 epochs). Target: before unlearning; Retrain: retrained without the forget set (the oracle).
(KnowMem-R at most 0.23). In contrast, SHIFT achieves the highest retain-knowledge score (0.539, 97% of the target’s), while removing more memorization than NPO and SimNPO. It also comes closest to the retrained oracle in verbatim memorization and retain knowledge.
Analysis
More training does not overshoot. This analysis and the relearning attacks below use learning rate , where the baselines’ 10 epochs carry twice the default update (SHIFT: , two epochs a round). Figure 3 trains NPO, WGA, UNDIAL and SHIFT twice as long, the baselines for 20 epochs under their own warm-up and decay, and evaluates them along the way. NPO and WGA cross the oracle within a standard run: NPO settles at a privacy leakage of +24, and WGA keeps rising to +52, three times further from the oracle than at its closest. UNDIAL never approaches the oracle (privacy leakage at most −84). SHIFT approaches the oracle and stays there: its distance is smallest at round 4 (0.007) and remains below 0.02 through round 8, and its privacy leakage rises toward 0 without crossing it, as expected of a loss minimized at its teacher rather than at an ever lower likelihood. Across learning rates (Figure 1(a), Appendix Table 7), every baseline but SimNPO overshoots to +40–+57 at , whereas SHIFT stays within [−23, +6] from up.

Figure 3. What happens when training continues, on TOFU forget05 (Llama-3.2-1B, learning rate ): the baselines train for 20 epochs (warm-up, then linear decay over the 20 epochs) and SHIFT for 8 rounds. The dashed line marks the end of a standard 10-epoch run (step 60); SHIFT’s standard 4 rounds end at step 48. Top: distance to the oracle (log scale); bottom: privacy leakage (0 is the oracle, positive is over-forgetting). SHIFT: mean over three seeds, shaded from minimum to maximum; baselines: one run, evaluated every second epoch.
Training cost. SHIFT costs only modestly more than the baselines. Each round adds retain steps and two forward passes over , and distillation runs no reference model, since the teacher and the retain anchor are cached (Algorithm 1). On one GPU, SHIFT trains on Llama-3.2-1B forget05 in 19 minutes against NPO’s 15.
Ablations. Table 4 removes one component at a time, in the setting of Table 2; every variant falls below SHIFT in Agg., Mem. and Util. alike. The heal keeps the target at the oracle: without it () the lookahead is a pure shrink toward rather than a step toward the oracle (Proposition 1). At this budget the two end close (Agg. 0.617; Appendix Table 8 varies ), but at twice the learning rate, removing the heal nearly doubles the distance to the oracle (Appendix Table 9). Extrapolation saves budget: the lookahead without extrapolation () moves in the right direction but more slowly (Corollary 1), reaching a lower Agg. (0.609) and a larger distance to the oracle (0.023) in the same rounds. Both directions of the teacher matter: keeping only the lowered tokens cuts Agg. to 0.563 and leaves the model nearly further from the oracle, since the raised alternatives are what moves the mass [41].
| Variant | Agg.↑ | Mem.↑ | Util.↑ | |
| SHIFT | 0.623 | 0.558 | 0.705 | 0.016 |
| no heal () | 0.617 | 0.553 | 0.698 | 0.015 |
| no extrapolation () | 0.609 | 0.537 | 0.702 | 0.023 |
| no raised tokens | 0.563 | 0.470 | 0.703 | 0.060 |
| Lookahead as the model, 1 round | 0.520 | 0.425 | 0.668 | 0.084 |
| Lookahead as the model, 4 rounds | 0.559 | 0.545 | 0.573 | 0.013 |
Table 4. Ablation studies on TOFU forget05, Llama-3.2-1B (setting of Table 2; as in Table 1). Raised tokens drive the forgetting, extrapolation saves budget, the heal anchors the target (Table 9), and distillation keeps utility: only together are they best in Agg., Mem. and Util.
Reading beats committing: using the lookahead itself as the model, as resetting methods do [1, 32], either stops short (one round: Agg. 0.520) or, if iterated, loses utility (0.573 vs. 0.705).
The results of additional analyses, including learning-rate sensitivity (Table 7), heal-length sensitivity (Table 8), and ablations at a higher learning rate (Table 9), are provided in Appendix § 12. SHIFT is insensitive to the heal length (Table 8), so one setting can transfer to every other model and split without retuning.
| Method | |||||
| GradDiff | 0.353 / | 0.527 / | 0.539 / | 0.293 / | 0.094 / |
| NPO | 0.335 / | 0.548 / | 0.621 / | 0.638 / | 0.631 / |
| SimNPO | 0.179 / | 0.229 / | 0.344 / | 0.420 / | 0.486 / |
| RMU | 0.446 / | 0.575 / | 0.182 / | 0.099 / | 0.085 / |
| WGA | 0.253 / | 0.488 / | 0.634 / | 0.681 / | 0.692 / |
| PDU | 0.451 / | 0.335 / | 0.016 / | 0.121 / | 0.503 / |
| SHIFT (ours) | 0.427 / | 0.603 / | 0.633 / | 0.628 / | 0.596 / |
Table 7. TOFU forget05, Llama-3.2-1B: Agg. / PL as the learning rate varies (seed 0; is the OpenUnlearning default run of each baseline; SHIFT uses and two epochs a round at every rate). SHIFT’s PL stays within [, ] from up; at every baseline but SimNPO overshoots to –.
| Agg. | Mem. | Util. | ||
| 0 | 0.617 | 0.553 | 0.698 | 0.015 |
| 8 | 0.616 | 0.546 | 0.706 | 0.020 |
| 16 | 0.623 | 0.558 | 0.705 | 0.016 |
| 32 | 0.619 | 0.551 | 0.706 | 0.016 |
Table 8. TOFU forget05, Llama-3.2-1B, learning rate , four rounds of four epochs (one seed): SHIFT with heal steps per round. Bold: the value of the main results. : as in Table 1.
| Variant | Agg. | Mem. | Util. | ||
| SHIFT | 3 | 0.633.001 | 0.587.001 | 0.686.002 | 0.006 |
| no heal () | 3 | 0.632.002 | 0.584.001 | 0.689.004 | 0.011 |
| no extrapolation () | 3 | 0.619.001 | 0.565.001 | 0.684.003 | <u>0.007</u> |
| no raised tokens | 3 | 0.582.000 | 0.506.001 | 0.686.001 | 0.041 |
| Lookahead as the model, 1 round | 3 | 0.513.001 | 0.419.002 | 0.663.001 | 0.089 |
| Lookahead as the model, 4 rounds | 3 | 0.566.001 | 0.537.000 | 0.597.002 | 0.017 |
Table 9. The ablations of Table 4 at learning rate on TOFU forget05, Llama-3.2-1B (: seeds). Bold / underlined: closest / second closest to the oracle.
Robustness to relearning. Suppressed knowledge can return after fine-tuning on related data [28, 17, 5]. We attack the models, relearning half of the forget05 authors and testing the other half (see details in Appendix § 13). The results are reported in Figure 4 and Appendix Table 11. After the attack, SHIFT is within 0.008 of the equally attacked oracle, closer than any baseline, whereas aggressive forgetting bounces back (RMU’s extraction strength rises from 0.029 to 0.195; Appendix Table 10), and SHIFT stays the closest at all six attack strengths (Figure 4).

Figure 4. Relearning attacks of increasing strength on Llama-3.2-1B forget05: distance to the equally attacked oracle (mean of three attack seeds). SHIFT is the closest at every strength (results for all baselines refer to Appendix Table 11).
Conclusion
Likelihood-suppression unlearning has no target, so its outcome is set by the optimization budget. The shrink-and-heal lookahead estimates this target from the model itself (its linearized fixed point is the retrained oracle); SHIFT distills it on forget tokens and iterates. Across three model sizes, it is on average the closest method to the oracle and has the highest average aggregate score, and it stays near the oracle where the baselines overshoot: unlearning should target an estimate of the oracle.
Limitations
SHIFT needs the pre-fine-tuning weights ; it applies to knowledge acquired in an identifiable fine-tuning stage (as in TOFU and MUSE), not to knowledge from pre-training, where no earlier checkpoint without it may exist. Our analysis assumes the linearized regime, a heal run to convergence and an exact teacher (no clipping or top-token truncation); we measured where real models depart from it (§ 4.1) but do not bound the error.
Ethics Statement
Unlearning is used to remove private or copyrighted content; methods that look successful on a single metric but over-forget are detectable by membership inference, which itself leaks which data were removed [16]. Calibrating forgetting to the retrained model is meant to reduce this leak. TOFU uses fictitious authors; MUSE uses public news articles. No new personal data were collected.
This appendix provides additional related work, theoretical proofs, implementation details, and experimental results supporting the main paper. Specifically, it includes:
Appendix A discusses extended related work: further discussion of unlearning objectives, oracle-based evaluation, distillation, robustness, and model arithmetic.
Appendix B provides proofs: complete proofs of Propositions 1 and 2, Corollary 1, and Lemma 1, along with an analysis of finite healing.
Appendix C measures the Lookahead: empirical analysis of the lookahead’s effects on forget-set and unseen examples.
Appendix D presents the implementation details, including training configurations, hyperparameters, and the complete algorithm.
Appendix E provides full TOFU results, including detailed memorization, utility, and aggregate scores across TOFU settings.
Appendix F reports additional analyses results, including learning-rate sensitivity, heal-length sensitivity, and ablations at larger learning rate.
Appendix G discusses relearning Attack, including attack setup and evaluation of oracle alignment under varying relearning strengths.
Extended Related Work
Likelihood suppression. Early unlearning methods use gradient ascent on the forget set ([20, 46]), often combined with retain-set regularization ([26]). Subsequent approaches introduce bounded or reweighted objectives ([52, 11, 40, 45]), refusal targets ([29]), representation misdirection ([25]), and constrained optimization ([10]). Recent work also explores adaptive stopping criteria based on the model’s own predictions ([6]). Despite their different mechanisms, they do not explicitly target the behavior of the retain-only oracle, which provides a principled reference for unlearning ([49, 16]).
Evaluation and scope. Recent position and evaluation work argues that unlearning claims should be judged against retraining ([49]) and that forgetting must generalize narrowly while retention generalizes broadly ([34]); robustness to relearning depends on how forget-and-retain-critical weights are separated ([3]). Our evaluation follows the first view: we measure the distance to the retrained model. Measuring the gap to the retrained model is standard beyond LLMs: forgetting has been defined as indistinguishability from a model trained without the data ([14]), image-classification unlearning reports the average gap to Retrain across accuracy and membership-inference metrics ([12]), and SCRUB adds a rewinding step for privacy so that the forget set is not forgotten beyond what retraining would give ([22]). For LLMs, TOFU’s forget quality tests whether the forget-set truth ratios are distributed as the retain model’s ([29]), and MUSE’s PrivLeak compares the membership-inference AUC with the retrained model’s, flagging both under- and over-unlearning ([38]). Our applies this gap to each forget statistic, PL carries MUSE’s measure over to TOFU, and Table 1 reports TOFU’s forget quality.
Distillation and target distributions. Teacher–student unlearning uses an incompetent or adjusted teacher ([4, 22]); UNDIAL lowers the gold logit by a fixed amount ([7]); “Who’s Harry Potter” builds the teacher from a model reinforced on the forget data and subtracts its excess ([9]). SHIFT instead moves away from the fine-tuning that produced the knowledge, and calibrates the move by the retain set. Logit-difference methods combine the model with an assistant trained to remember the forget set ([21] (Huang et al. 2025)); emulated fine-tuning adds or removes the effect of fine-tuning at the logit level ([31]). DUET distills from an in-context prompted teacher ([54]), and bootstrapping methods suppress the model’s own high-confidence generations to avoid the squeezing of probability mass into rephrasings ([24]). Simulated oracle matching predicts the oracle’s outputs from data attribution ([13]); SHIFT predicts them from a cheap retain re-fit.
Robustness of unlearning. Suppressed knowledge can return after fine-tuning on related data ([28, 17, 5]), especially when the relearning data share their format with the forget set ([48]), or after low-bit quantization ([53]). Recent defenses target the answer margins ([47]), minor representation components ([42]), smoothness in parameter and representation space ([43]), or quantization explicitly ([37]), and distilling toward a target without the knowledge has been argued to remove it more durably ([23]). SHIFT is a distillation method of this kind, with a target estimated from the retain set.
Checkpoint arithmetic. The closest related approach is MSA ([36]), which also uses a checkpoint from before the forget set was seen, but in the opposite direction: it fine-tunes that checkpoint on and subtracts the resulting vector, scaled by a tuned coefficient, from the final model. SHIFT never trains on . It shrinks toward the checkpoint and re-fits , so the forget direction is what cannot restore, and its magnitude is set by the oracle (Proposition 1) rather than by that coefficient. The estimate is then read on forget tokens and distilled, instead of being subtracted in weight space.
Resetting weights. Shrink-and-perturb ([1]), rewinding to an earlier checkpoint and fine-tuning on retained data ([32]), Fisher-based scrubbing ([14]) and noising followed by distillation ([23]) all commit a reset. SAGE projects an unlearning update away from directions that damage retain activations ([50]), close in spirit to how our heal restores retain-supported directions. Our ablation that keeps the lookahead as the model (Table 4) belongs to this family; SHIFT reads the reset instead of keeping it, and derives a calibrated target from it. Task arithmetic negates a task vector to forget a task ([18]); Proposition 1 explains why shrinking and re-fitting the retain set isolates the forget-specific part of that vector without training on the forget set.
Proofs
Setup. Let stack the logits of a set of training positions. In the linearized model,
Training to interpolation fits targets (for the cross-entropy loss, logits that realize the fitted distributions; the argument only needs that the fitted values are reached). We use one standard fact about gradient descent ([15]):
Fact. Gradient descent on any loss of , started from , moves only within the row space . If it fits , it therefore converges to the closest fitting point:
For the retain set we write for the residual of the base model, for the projection onto , and for what fine-tuning added.
Proof of Proposition 1. Step 1: the oracle. The target model fits , so . The oracle is gradient descent on from , so by (F)
which is Eq. (3).
Step 2: the shrink. The shrunk model leaves a fraction of the retain set to be re-learned:
Step 3: the heal. By (F), healing from adds . Hence
where the last equality uses from Step 1. This is Eq. (4).
Step 4: the fixed point. Every lookahead fits , because the heal runs to convergence, so it suffices to consider with . Steps 2–3 then apply with in place of .
A fixed point has , i.e., with . The only such vector is , so the fixed point is , and it is unique. This completes the proof. □
Proof of Proposition 2. Let stack the Jacobians of all training positions of , write , and let
where the second form is Proposition 1. We show that distillation from lands exactly on . Step 1: the targets are the outputs of . On , linearity (and Lemma 1 for log-probabilities) gives
On , the anchor’s target is , and because .
Step 2: the step lies in . The target model is the minimum-norm fit of , so . Each iterate is with , so by induction from every lies in , and hence so does .
Step 3: conclusion. By (F), gradient descent on the logit-matching loss from converges to
since is the identity on . At , Corollary 1 gives . The KL form of the loss is minimized by any logits that equal the targets up to a constant per position, and those leave the predictive distributions unchanged.
Proof of Corollary 1. Write . If fits , Proposition 1 gives
The step lies in the null space of , so and still fits ; by induction from , every iterate does. Projecting the step onto and its complement,
Unrolling from ,
For we have , so , i.e., ; for the forget-specific part vanishes after one step.
Lemma 1 (Normalization invariance). For logits with log-probabilities and , and any ,
Proof of Lemma 1. Substituting the definitions,
The bracket is the same across tokens, and softmax is invariant to adding a constant to all logits.
A finite heal. With heal steps of size on the squared loss, the healed component along an eigenvector of with eigenvalue is a fraction of its shrunk part. The lookahead step therefore becomes
The first term is the step of Proposition 1. The second is the retain knowledge that the heal did not restore; it lies in retain-supported directions, which is what the retain anchor of Eq. (7) counteracts (§ 4.3).
Measuring The Lookahead
Before building the method, we measured the lookahead at (TOFU forget10, Llama-3.2-1B, ). For each answer position, is the extrapolated () drop of the gold token’s log-probability: positive when the lookahead lowers the gold token, negative when it raises it. On never-seen holdout authors (positions 200–300 of holdout10) we fit a band of for each confidence decile of , and call a position active when its falls outside the band. is the gated forget-set drop at (forget minus null).
| Heal steps | holdout | forget | Active F/null/test | |
| 16 | 1.58 | 0.26/0.05/0.056 | ||
| 32 | 1.93 | 0.31/0.05/0.058 | ||
| 64 | 1.84 | 0.30/0.05/0.060 |
Table 5. The lookahead at : mean extrapolated drop (nats/token) on never-seen and forget authors, gated forget-specific drop , and the fraction of active positions on the forget set, the band’s own authors (null) and a second set of never-seen authors (test).
The lookahead lowers the gold tokens of the forget set ( nats) and raises them on never-seen authors (by 0.88–0.96 nats). The band transfers to the second set of never-seen authors (5.6–6.0% active, as designed), where the gated drop is 0.04–0.08 nats/token. For reference, ’s log-probability of the gold forget tokens exceeds the retrained model’s by 2.45 nats/token (computed post hoc; the method never uses the retrained model). The heal does not remove the drift on holdout authors between 16 and 64 steps, so a longer heal would not remove it either; this is why SHIFT confines the teacher to forget tokens rather than trying to make the lookahead exact.
Implementation Details
Implementation Setup
Environment. Llama-3.2-1B runs on one NVIDIA GeForce RTX 5060 Ti (16 GB) or A100 (40 GB), Llama-3.2-3B on one A100, and Llama-3.1-8B and the MUSE model on four A100 GPUs with DeepSpeed ZeRO-3 (optimizer states on the CPU). All code builds on OpenUnlearning (Dorna et al. [8]) with PyTorch 2.4.1, Transformers 4.51.3 and DeepSpeed 0.15.4.
Training configuration. Every method follows the OpenUnlearning configuration. On TOFU we use AdamW (8-bit paged optimizer states), an effective batch size of 32, a learning rate of with one epoch of linear warm-up and linear decay, weight decay 0.01, 10 epochs and bf16 weights; with Llama-3.1-8B every method keeps the input embeddings frozen. On MUSE we use an effective batch size of 64, a learning rate of with a constant schedule, 10 epochs and no weight decay. Each method is run once (seed 0) and its final model is evaluated, with no checkpoint selection.
Hyperparameter Settings
Every baseline keeps the OpenUnlearning defaults below (: weight of the forget loss, : weight of the retain loss, NLL on the retain set unless noted); we do not tune them per setting.
GradAscent. No method-specific hyperparameter.
GradDiff. .
NPO. , .
SimNPO. , margin , , .
IdkDPO. with refusal answers as the preferred responses, .
RMU. Steering coefficient 2 at layer 7, all parameters trained, retain loss on hidden representations, .
UNDIAL. , , .
WGA. , .
SatImp. , , , .
CE-U. The first answer token is not trained.
PDU. Primal-dual updates with dual step size 1 and 5 warm-up epochs, retain tolerance , .
SHIFT. Lookahead: and heal steps on retain answers (32 QA pairs per step, learning rate ). Teacher: the top-256 tokens of the model and the lookahead at each position (top-128 on MUSE), clip , step sizes . Distillation: , a retain anchor on ’s top-64 next-token distributions, learning rate with a short warm-up in each round, four rounds of four epochs.
Algorithm 1 SHIFT
Require: target $\theta_0$, base $\theta_b$, $D_f$, $D_r$; $\rho$, $k$, $\kappa$, $s_0, \ldots, s_{N-1}$
1: cache $p_{\theta_0}$ on $D_r$ ▷ retain anchor
2: for $n = 0, \ldots, N - 1$ do
3: $\Pi \leftarrow \operatorname{Heal}_k(\theta_n - \rho(\theta_n - \theta_b))$ ▷ {xref}`Eq. (2)`
4: for all answer positions $t$ of $D_f$ do
5: $\delta_t \leftarrow \ell_{\Pi,t} - \ell_{n,t}$ ▷ on top tokens $V_t$
6: $p_{T,t} \leftarrow \operatorname{norm}_{V_t}\left(\ell_{n,t} + \operatorname{clip}_{\kappa}\left(\frac{s_n}{\rho}\delta_t\right)\right)$ ▷ {xref}`Eq. (6)`
7: cache $p_T$ on $D_f$; free $\Pi$
8: $\theta_{n+1} \leftarrow \operatorname{distill}$ from $\theta_n$ on $\mathcal{L}$ ▷ {xref}`Eq. (7)`
9: return $\theta_N$
Full TOFU Results
Table 6 complements Table 2 with the two components of Agg. SHIFT’s memorization and utility both stay close to the oracle’s, whereas on average every baseline gives up one of the two: most leave the forget set memorized (low Mem.), and GradAscent and CE-U collapse (Util. near 0).
| Method | Llama-3.2-1B Agg. | Llama-3.2-1B Mem. | Llama-3.2-1B Util. | Llama-3.2-3B Agg. | Llama-3.2-3B Mem. | Llama-3.2-3B Util. | Llama-3.1-8B Agg. | Llama-3.1-8B Mem. | Llama-3.1-8B Util. |
| FORGET 1% | |||||||||
| Full | 0.118 | 0.064 | 0.717 | 0.024 | 0.012 | 0.763 | 0.005 | 0.002 | 0.738 |
| Retain | 0.613 | 0.543 | 0.703 | 0.632 | 0.541 | 0.761 | 0.619 | 0.533 | 0.738 |
| GradAscent | 0.458 | 0.335 | 0.723 | 0.409 | 0.281 | 0.750 | 0.617 | 0.542 | 0.716 |
| GradDiff | 0.440 | 0.317 | 0.718 | 0.378 | 0.251 | 0.760 | 0.565 | 0.475 | 0.698 |
| NPO | 0.458 | 0.336 | 0.717 | 0.419 | 0.290 | 0.750 | 0.599 | 0.504 | 0.737 |
| SimNPO | 0.218 | 0.129 | 0.725 | 0.151 | 0.084 | 0.746 | 0.250 | 0.150 | 0.762 |
| IdkDPO | 0.294 | 0.185 | 0.712 | 0.196 | 0.112 | 0.762 | 0.478 | 0.365 | 0.693 |
| RMU | 0.509 | 0.414 | 0.660 | 0.212 | 0.123 | 0.768 | 0.533 | 0.419 | 0.734 |
| UNDIAL | 0.456 | 0.335 | 0.718 | 0.446 | 0.316 | 0.759 | <u>0.630</u> | 0.538 | 0.759 |
| WGA | 0.434 | 0.310 | 0.723 | 0.405 | 0.277 | 0.753 | 0.552 | 0.436 | 0.751 |
| SatImp | 0.139 | 0.077 | 0.722 | 0.036 | 0.019 | 0.756 | 0.106 | 0.057 | 0.762 |
| CE-U | 0.557 | 0.466 | 0.692 | 0.538 | 0.417 | 0.761 | 0.282 | 0.796 | 0.171 |
| PDU | <u>0.573</u> | 0.488 | 0.692 | <u>0.559</u> | 0.459 | 0.714 | 0.018 | 0.810 | 0.009 |
| SHIFT (ours) | 0.619 | 0.542 | 0.720 | 0.635 | 0.546 | 0.758 | 0.639 | 0.565 | 0.736 |
| FORGET 5% | |||||||||
| Full | 0.154 | 0.086 | 0.707 | 0.059 | 0.030 | 0.748 | 0.016 | 0.008 | 0.723 |
| Retain | 0.641 | 0.577 | 0.722 | 0.648 | 0.569 | 0.751 | 0.641 | 0.567 | 0.736 |
| GradAscent | 0.214 | 0.635 | 0.129 | <u>0.567</u> | 0.501 | 0.653 | 0.000 | 0.983 | 0.000 |
| GradDiff | 0.527 | 0.475 | 0.591 | 0.487 | 0.399 | 0.624 | 0.111 | 0.988 | 0.059 |
| NPO | 0.548 | 0.507 | 0.597 | 0.561 | 0.477 | 0.681 | 0.666 | 0.609 | 0.736 |
| SimNPO | 0.229 | 0.136 | 0.709 | 0.165 | 0.093 | 0.741 | 0.366 | 0.244 | 0.734 |
| IdkDPO | 0.442 | 0.325 | 0.689 | 0.397 | 0.272 | 0.734 | 0.549 | 0.453 | 0.696 |
| RMU | <u>0.575</u> | 0.524 | 0.638 | 0.539 | 0.421 | 0.748 | 0.612 | 0.729 | 0.527 |
| UNDIAL | 0.480 | 0.369 | 0.684 | 0.532 | 0.411 | 0.751 | 0.649 | 0.573 | 0.748 |
| WGA | 0.488 | 0.370 | 0.715 | 0.453 | 0.324 | 0.748 | 0.589 | 0.733 | 0.493 |
| SatImp | 0.175 | 0.100 | 0.710 | 0.080 | 0.042 | 0.740 | 0.143 | 0.080 | 0.713 |
| CE-U | 0.000 | 0.785 | 0.000 | 0.000 | 0.789 | 0.000 | 0.000 | 0.788 | 0.000 |
| PDU | 0.335 | 0.640 | 0.227 | 0.045 | 0.737 | 0.023 | 0.631 | 0.602 | 0.664 |
| SHIFT (ours) | 0.623 | 0.558 | 0.705 | 0.639 | 0.556 | 0.752 | <u>0.650</u> | 0.612 | 0.693 |
| FORGET 10% | |||||||||
| Full | 0.159 | 0.089 | 0.706 | 0.063 | 0.033 | 0.752 | 0.011 | 0.006 | 0.727 |
| Retain | 0.643 | 0.584 | 0.715 | 0.650 | 0.574 | 0.749 | 0.649 | 0.573 | 0.748 |
| GradAscent | 0.000 | 0.992 | 0.000 | 0.000 | 0.968 | 0.000 | 0.000 | 0.982 | 0.000 |
| GradDiff | 0.521 | 0.494 | 0.551 | 0.482 | 0.474 | 0.490 | 0.423 | 0.988 | 0.269 |
| NPO | <u>0.568</u> | 0.546 | 0.592 | 0.584 | 0.502 | 0.697 | <u>0.650</u> | 0.597 | 0.714 |
| SimNPO | 0.235 | 0.141 | 0.709 | 0.181 | 0.103 | 0.748 | 0.383 | 0.261 | 0.715 |
| IdkDPO | 0.488 | 0.375 | 0.699 | 0.431 | 0.303 | 0.743 | 0.531 | 0.421 | 0.720 |
| RMU | 0.221 | 0.710 | 0.131 | 0.646 | 0.644 | 0.648 | 0.447 | 0.773 | 0.314 |
| UNDIAL | 0.484 | 0.374 | 0.686 | 0.546 | 0.426 | 0.757 | 0.653 | 0.583 | 0.742 |
| WGA | 0.506 | 0.392 | 0.711 | 0.485 | 0.360 | 0.743 | 0.436 | 0.771 | 0.304 |
| SatImp | 0.182 | 0.104 | 0.709 | 0.082 | 0.044 | 0.743 | 0.207 | 0.121 | 0.699 |
| CE-U | 0.000 | 0.785 | 0.000 | 0.000 | 0.794 | 0.000 | 0.000 | 0.990 | 0.000 |
| PDU | 0.499 | 0.410 | 0.638 | 0.520 | 0.434 | 0.647 | 0.310 | 0.802 | 0.192 |
| SHIFT (ours) | 0.624 | 0.556 | 0.710 | <u>0.634</u> | 0.550 | 0.748 | 0.631 | 0.581 | 0.691 |
Table 6. Agg., Mem., and Util. on TOFU, all methods at learning rate (one seed). Bold / <u>underlined</u>: best / second-best Agg. for each model and split; we do not mark Mem. or Util. alone, since collapsing maximizes Mem. (GradAscent and CE-U reach Util. ).
Additional Analyses
Learning rate. Table 7 varies the learning rate of every method on Llama-3.2-1B forget05. At the default the baselines stop short of the oracle (privacy leakage far below 0); from up most of them cross it, while SHIFT stays near it.
Heal length. Table 8 varies the number of heal steps on Llama-3.2-1B forget05, with everything else as in Table 2. The results change little with : Agg. stays between 0.616 and 0.623 and the distance to the oracle between 0.015 and 0.020; , the value of the main results, has the highest Agg.
Ablations at the larger learning rate. Table 9 repeats Table 4 at learning rate (, two epochs a round, three seeds). With the larger update, the heal matters: without it the distance to the oracle nearly doubles (0.011 vs. 0.006), and SHIFT is the closest variant.
Relearning Attack
The attack fine-tunes the unlearned model on the first half of the forget05 authors (TOFU has 20
consecutive questions per author) with learning rate for one epoch after 3 warmup steps (batch 8, fp32 master weights, 8-bit AdamW), and evaluates the second half, which it never sees. Results are averaged over three attack seeds, which change the order of the relearning batches. Table 11 repeats the attack at learning rates and for one to three epochs (Figure 4). Stronger attacks also erode what the full model itself knows about the held-out half, so every method moves toward the oracle, but SHIFT stays the closest at every strength.
| Method | 1 ep. | 2 ep. | 3 ep. | 1 ep. | 2 ep. | 3 ep. |
| GradAscent | 0.039 | 0.017 | 0.016 | 0.017 | 0.022 | 0.014 |
| GradDiff | 0.038 | 0.048 | 0.023 | 0.029 | 0.021 | 0.013 |
| NPO | <u>0.019</u> | <u>0.005</u> | 0.020 | <u>0.012</u> | 0.015 | <u>0.010</u> |
| SimNPO | 0.160 | 0.092 | 0.065 | 0.055 | 0.038 | 0.032 |
| IdkDPO | 0.025 | 0.029 | 0.026 | 0.020 | <u>0.009</u> | 0.011 |
| RMU | 0.104 | 0.077 | 0.048 | 0.051 | 0.034 | 0.027 |
| UNDIAL | 0.049 | 0.037 | 0.020 | 0.020 | 0.021 | 0.021 |
| WGA | 0.030 | 0.047 | 0.018 | 0.021 | 0.020 | 0.015 |
| SatImp | 0.198 | 0.111 | 0.077 | 0.061 | 0.036 | 0.024 |
| CE-U | 0.045 | 0.022 | 0.040 | 0.038 | 0.031 | 0.027 |
| PDU | 0.033 | 0.033 | 0.032 | 0.036 | 0.016 | 0.018 |
| SHIFT (ours) | 0.008 | 0.002 | 0.012 | 0.009 | 0.008 | 0.008 |
Table 11. Distance to the equally attacked oracle (mean absolute difference of ES and ROUGE-L on the held-out half) across attack strengths, three attack seeds each. Bold / <u>underlined</u>: closest / second closest.
Use of AI Assistants
In preparing this paper, we made limited use of AI Assistants such as ChatGPT only to help refine wording, correct grammar, and enhance clarity. All core research contributions, including conceptual development, theoretical analysis, methodology design and implementation, experimental evaluations, and paper organization and composition, were conducted entirely by the authors.
| Method | Before attack ES | Before attack RL | After attack ES | After attack RL | oracle |
| Full | 0.684 | 0.831 | 0.316 | 0.537 | 0.214 |
| Retain | 0.060 | 0.372 | 0.070 | 0.353 | – |
| GradAscent | 0.029 | 0.000 | 0.085 | 0.290 | 0.039 |
| GradDiff | 0.048 | 0.244 | 0.111 | 0.389 | 0.038 |
| NPO | 0.071 | 0.288 | 0.076 | 0.321 | 0.019 |
| SimNPO | 0.301 | 0.591 | 0.253 | 0.489 | 0.160 |
| IdkDPO | 0.110 | 0.154 | 0.114 | 0.361 | 0.025 |
| RMU | 0.029 | 0.116 | 0.195 | 0.436 | 0.104 |
| UNDIAL | 0.047 | 0.310 | 0.103 | 0.418 | 0.049 |
| WGA | 0.062 | 0.305 | 0.112 | 0.371 | 0.030 |
| SatImp | 0.592 | 0.785 | 0.308 | 0.512 | 0.198 |
| CE-U | 0.029 | 0.001 | 0.078 | 0.271 | 0.045 |
| PDU | 0.029 | 0.073 | 0.123 | 0.340 | 0.033 |
| SHIFT (ours) | 0.056 | 0.365 | 0.065 | 0.365 | 0.008 |
Table 10. Relearning attack on TOFU forget05 (Llama-3.2-1B, models unlearned at ): extraction strength (ES) and ROUGE-L (RL) on the held-out half of the forget authors, before and after relearning on the other half (mean of three attack seeds). oracle: mean absolute difference of ES and RL from the relearned Retain model; bold / <u>underlined</u>: closest / second closest.
References
- [1]Jordan T. Ash and Ryan P. Adams. 2020. On warm-starting neural network training. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/1910.08475
- [2]Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In USENIX Security Symposium.arxiv.org/abs/2012.07805
- [3]Yi Chen, Hanna Hsieh, Shuhong Liu, Chuanbo Hua, Zihan Ma, Kun Wang, and Joo-Young Kim. 2026. Distance is not enough: Forget-retain alignment gap predicts LLM relearning robustness. arXiv preprint arXiv:2608.25429.DOI
- [4]Vikram S. Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan Kankanhalli. 2023. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In AAAI Conference on Artificial Intelligence, volume 37, pages 7210–7217.
- [5]Aghyad Deeb and Fabien Roger. 2024. Do unlearning methods remove information from language model weights? arXiv preprint arXiv:2410.08827.arxiv.org/abs/2410.08827
- [6]Federico Di Gennaro, Alexander Shevchenko, and Fanny Yang. 2026. Fast unlearning at scale via margin self-correction. arXiv preprint arXiv:2606.02920.arxiv.org/abs/2606.02920
- [7]Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. 2025. UNDIAL: Self-distillation with adjusted logits for robust unlearning in large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8827–8840.arxiv.org/abs/2402.10052
- [8]Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, J. Zico Kolter, Zachary C. Lipton, and Pratyush Maini. 2025. OpenUnlearning: Accelerating LLM unlearning via unified benchmarking of methods and metrics. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track.arxiv.org/abs/2506.12618
- [9]Ronen Eldan and Mark Russinovich. 2023. Who’s Harry Potter? approximate unlearning in LLMs. arXiv preprint arXiv:2310.02238.arxiv.org/abs/2310.02238
- [10]Taha Entesari, Arman Hatami, Rinat Khaziev, Anil Ramakrishna, and Mahyar Fazlyab. 2025. Constrained entropic unlearning: A primal-dual framework for large language models. In Advances in Neural Information Processing Systems (NeurIPS).
- [11]Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. 2025. Simplicity prevails: Rethinking negative preference optimization for LLM unlearning. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2410.07163
- [12]Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. 2024. SalUn: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2310.12508
- [13]Kristian Georgiev, Roy Rinberg, Sung Min Park, Shivam Garg, Andrew Ilyas, Aleksander Madry, and Seth Neel. 2025. Machine unlearning via simulated oracle matching. In International Conference on Learning Representations (ICLR).
- [14]Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).arxiv.org/abs/1911.04933
- [15]Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. 2018. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning (ICML).arxiv.org/abs/1802.08246
- [16]Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. 2025. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy. In IEEE Conference on Secure and Trustworthy Machine Learning (SaTML).arxiv.org/abs/2403.01218
- [17]Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. 2025. Unlearning or obfuscating? jogging the memory of unlearned LLMs via benign relearning. In International Conference on Learning Representations (ICLR). James Y. Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2025. Offset unlearning for large language models. Transactions on Machine Learning Research.
- [18]Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hanneleh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2212.04089
- [19]Arthur Jacot, Franck Gabriel, and Clément Hongler. 2018. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/1806.07572
- [20]Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajunagen Logeswaran, and Minjoon Seo. 2023. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389–14408.
- [21]Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Rao Kompella, Sijia Liu, and Shiyu Chang. 2024. Reversing the forget-retain objectives: An efficient LLM unlearning framework from logit difference. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2406.08607
- [22]Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. 2023. Towards unbounded machine unlearning. In Advances in Neural Information Processing Systems (NeurIPS).
- [23]Bruce W. Lee, Addie Foote, Alex Infanger, Leni Shor, Harish Kamath, Jacob Goldman-Wetzler, Bryce Woodworth, Alex Cloud, and Alexander Matt Turner. 2025. Distillation robustifies unlearning. In Advances in Neural Information Processing Systems (NeurIPS).
- [24]Kemou Li, Qizhou Wang, Yue Wang, Fengpeng Li, Jun Liu, Bo Han, and Jiantao Zhou. 2026. LLM unlearning with LLM beliefs. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2510.19422
- [25]Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew Bo Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, and 27 others. 2024. The WMDP benchmark: Measuring and reducing malicious use with unlearning. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 28525–28550. PMLR.arxiv.org/abs/2403.03218
- [26]Bo Liu, Qiang Liu, and Peter Stone. 2022. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243–254. PMLR.arxiv.org/abs/2203.12817
- [27]Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuyang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. 2025. Rethinking machine unlearning for large language models. Nature Machine Intelligence, 7(2):181–194.
- [28]Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. 2024. Eight methods to evaluate robust unlearning in LLMs. arXiv preprint arXiv:2402.16835.arxiv.org/abs/2402.16835
- [29]Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. 2024. TOFU: A task of fictitious unlearning for LLMs. In Conference on Language Modeling (COLM).arxiv.org/abs/2401.06121
- [30]Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. 2023. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning (ICML).arxiv.org/abs/2210.05643
- [31]Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D. Manning. 2024. An emulator for fine-tuning large language models using small language models. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2310.12962
- [32]Siqiao Mu and Diego Klabjan. 2025. Rewind-to-delete: Certified machine unlearning for nonconvex functions. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2409.09778
- [33]Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. 2023. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Advances in Neural Information Processing Systems (NeurIPS).
- [34]Amit Peleg, Naman Deep Singh, Naama Pearl, Bibhabasu Mohapatra, and Matthias Hein. 2026. Forget narrowly, retain broadly: Unlearning as an asymmetric generalization problem. arXiv preprint arXiv:2607.09236.arxiv.org/abs/2607.09236
- [35]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2305.18290
- [36]Keivan Rezaei, Mehrdad Saberi, Abhilasha Ravichander, and Soheil Feizi. 2026. Revisiting the past: Data unlearning with model state history. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2506.20941
- [37]Saisab Sadhu, Pratinav Seth, and Vinay Kumar Sankarapu. 2026. Forgetting that sticks: Quantization-permanent unlearning via circuit attribution. arXiv preprint arXiv:2605.15138.arxiv.org/abs/2605.15138
- [38]Weijia Shi, Jaechang Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah Smith, and Chiyuan Zhang. 2025. Muse: Machine unlearning six-way evaluation for language models. In International Conference on Learning Representations, volume 2025, pages 27797–27818.arxiv.org/abs/2407.06460
- [39]Zexuan Sun and Garyesh Raskutti. 2026. A theoretical framework for LLM fine-tuning using early stopping for non-random initialization. arXiv preprint arXiv:2602.13942.arxiv.org/abs/2602.13942
- [40]Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, and Kilian Q. Weinberger. 2025. Rethinking LLM unlearning objectives: A gradient perspective and go beyond. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2502.19301
- [41]Jiawei Wu and Doudou Zhou. 2026. Unlearning what matters: Token-level attribution for precise language model unlearning. arXiv preprint arXiv:2605.00364.arxiv.org/abs/2605.00364
- [42]Zeguan Xiao, Xuanzhe Xu, Yun Chen, Yong Wang, Jian Yang, Yanqing Hu, and Guanhua Chen. 2026. Robust llm unlearning against relearning attacks: The minor components in representations matter. arXiv preprint arXiv:2605.11685.arxiv.org/abs/2605.11685
- [43]Han Yan, Zheyuan Liu, and Meng Jiang. 2026. Dual-space smoothness for robust and balanced LLM unlearning. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2509.23362
- [44]Bo Yang. 2025. CE-U: Cross entropy unlearning. arXiv preprint arXiv:2503.01224.arxiv.org/abs/2503.01224
- [45]Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. 2025. Exploring criteria of loss reweighting to enhance LLM unlearning. In International Conference on Machine Learning (ICML).arxiv.org/abs/2505.11953
- [46]Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024. Large language model unlearning. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2310.10683
- [47]Xiangyu Yin, Jiaxu Liu, Zhen Chen, and Chih-Hong Cheng. 2026. Crossing the margin cliff: Toward relearn-robust LLM unlearning via margin calibration. arXiv preprint arXiv:2607.27836.arxiv.org/abs/2607.27836
- [48]Sangyeon Yoon, Hyesoo Hong, Wonje Jeung, and Albert No. 2026a. Rethinking benign relearning: Syntax as the hidden driver of unlearning failures. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2602.03379
- [49]Sangyeon Yoon, Yeachan Jun, and Albert No. 2026b. Position: The term “machine unlearning” is overused in LLMs. In International Conference on Machine Learning (ICML), Position Paper Track.arxiv.org/abs/2606.27379
- [50]Jingyuan Zhang, Yucheng Bai, Peixi Wen, Zhehao Huang, Zhengbao He, Hanling Tian, Xinwen Cheng, Haiyin Ran, and Xiaolin Huang. 2026. SAGE: Retain-aware post-hoc sanitization of final unlearning vector. arXiv preprint arXiv:2606.18309.arxiv.org/abs/2606.18309
- [51]Michael R. Zhang, James Lucas, Geoffrey Hinton, and Jimmy Ba. 2019. Lookahead optimizer: k steps forward, 1 step back. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/1907.08610
- [52]Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. In Conference on Language Modeling (COLM).arxiv.org/abs/2404.05868
- [53]Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. 2025. Catastrophic failure of LLM unlearning via quantization. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2410.16454
- [54]Yisheng Zhong, Zhengbang Yang, and Zhuangdi Zhu. 2026. DUET: Distilled LLM unlearning from an efficiently contextualized teacher. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2601.21283