Introduction
Modern machine learning increasingly uses sensitive, copyrighted, or safety-critical data. As a result, machine unlearning, the post-hoc removal of a data’s influence on the learned model, has become a practical and regulatory concern [2 , 7 ] . The gold standard for machine unlearning is conceptually simple: retrain the model on the retained data after deleting the forget set. For large models and frequent deletion requests, however, exact retraining is usually too expensive, so practical unlearning methods seek more efficient approximations to this retrained gold standard model.
One family of approximations uses Newton-style updates to move directly toward the retrained optimum, but these methods depend on Hessian information and are often not scalable to large models [15 , 18 , 39 ] . The dominant practical alternative is gradient-based unlearning: gradient ascent (GA) increases loss on the forget set, while GradDiff, NPO, SimNPO, IMU, and projected variants add retain losses, per-sample weights, or projections to preserve utility [8 , 9 , 17 , 24 , 38 , 40 ] . These methods are usually tuned and compared through forget and retain losses.
Recent work has established that loss-space, parameter-space agreement with retraining, and even finite output-imitation tests are insufficient evaluation criteria for unlearning [21 , 37 ] . We verify this empirically in Figure 1 : at matched forget loss or similar weights, different gradient-based unlearners can have very different output-space behavior and forget performance, and they can all differ substantially from the retrained model. This motivates output-space deviation from retraining as the evaluation surface, which leads to the open question we study here: what governs the structure of this deviation for gradient-based unlearners? We show that the answer is a leverage geometry induced by the retrained model, and existing gradient-based variants control only parts of this geometry.
Figure 1. Loss and weights are not behavior. MNIST[20 ] with multi-layer perceptron (MLP) model for class forgetting with 10 seeds for various gradient-based methods and a Newton step unlearning. From left to right: forget cross-entropy (matched to the retrained reference, dashed), weight distance ∥ θ − θ r ∗ ∥ 2 \lVert\theta-\theta_r^*\rVert_2 ∥ θ − θ r ∗ ∥ 2 , KL(p θ r ∗ ∥ p θ p_{\theta_r^*}\lVert p_\theta p θ r ∗ ∥ p θ ) on D f D_f D f , and Top-1 forget accuracy. Setup is detailed in Appendix § 10 .
Our key contributions are:
Unified gap formula. We derive closed-form output-space gaps for gradient-based unlearners, exact in linear regression and extended to nonlinear models under a local generalized Gauss-Newton (GGN) approximation [31 ] . This gap shows a fundamental objective mismatch between common gradient-based methods and the gold standard.
Two-leverage decomposition. Our derivation reveals two important components of the gap: forget self-leverage, which captures the uniqueness of forget samples under the retrained model geometry, and retain cross-leverage, which captures how strongly forget-directed changes transfer into retain outputs.
Forget-set diagnostics. We validate each method’s gap formula on MNIST and CIFAR-10, derive a closed-form sufficient rule for ρ \rho ρ , and empirically validate precomputable leverage diagnostics for LLM scale.
Related Work
Machine unlearning methods
Early formulations of machine unlearning formalized data deletion through exact retraining, certified removal, or influence-style approximations to retraining [2 , 15 , 18 ] . Newton-style methods are attractive because they target the retrained optimum directly, but they require Hessian or inverse-Hessian operations and rely on local approximations that can be costly or fragile at modern scale [11 , 39 ] . A related line linearizes the network around a reference point (pretrained or core-trained weights) and forgets exactly in the resulting linear, typically squared-loss, model. Examples include NTK-based scrubbing [12 ] , mixed-privacy forgetting [13 ] , and parameter-efficient variants [22 ] , made practical by fast empirical-NTK approximations [30 , 35 ] . Within that linear model they are exact retraining, i.e. the Newton Step row of Table 1 , so their output gap to θ r ∗ \theta_r^* θ r ∗ is governed by linearization error and the locality limits of § 4.3 . While Newton-style methods are limited in practice, the retrained-model Hessian they use as a preconditioner is the same geometric object that, as we show, implicitly governs the output-space gap of gradient-based unlearners.
Table 1. Theoretical output gaps from θ r ∗ \theta_r^* θ r ∗ in the linear setting. Each row assumes that its displayed inverse exists. The GradDiff, Pure GA, fixed-weight IMU, fixed-projector ProjGD, and Newton Step rows are exact for the displayed linear objectives; NPO and SimNPO are frozen-weight local forms, with weights evaluated at θ r ∗ \theta_r^* θ r ∗ . The geometry column distinguishes methods that route forget perturbations through H r − 1 H_r^{-1} H r − 1 (top group: GradDiff, NPO, SimNPO, fixed-projector ProjGD) from those that use raw Euclidean Gram structure (Pure GA, IMU). Newton Step recovers θ r ∗ \theta_r^* θ r ∗ exactly in linear regression. Proofs in Appendix § 9 .6.
Most practical large language model (LLM) unlearning methods instead optimize surrogate objectives. GA on the forget loss is the simplest example [17 ] ; GradDiff adds retain descent to improve utility preservation [24 ] ; NPO and SimNPO stabilize GradDiff-style unlearning through preference-style losses and reweighting [8 , 38 ] ; IMU introduces influence-guided sample weights [9 ] ; and GU-style projected methods reduce retain drift by projecting updates away from retain-gradient directions [40 ] . Another LLM baseline is Representation Misdirection Unlearning (RMU) [23 ] , but it operates on intermediate representations rather than the output-loss gradient-based family analyzed in this work.
Limits of loss-space and parameter-space evaluation
Several lines of existing work show from different angles that loss, output imitation on a limited evaluation set, and parameter agreement are not enough to certify unlearning. In general prediction problems, predictive multiplicity shows that competing models with similar loss can assign conflicting predictions [28 ] . In unlearning, robustness studies show that superficial “forgetting” can be reversed by relearning or adversarial evaluation [5 , 16 , 21 , 25 , 26 ] . Other work argues that parameter-space or algorithmic equivalence criteria are themselves delicate: verification-error analyses motivate proximity to retraining, while auditable definitions and retrain-equivalence impossibility results expose limits of parameter- or path-oblivious formulations [33 , 34 , 37 ] . In the adjacent setting of catastrophic forgetting, Shenfeld et al. [32 ] show that the forward KL to the original model on the new task predicts the loss of prior-task performance. We share the output-space viewpoint, but measure per-example gaps to the retrained reference and decompose them by forget–retain geometry.
Retain-forget entanglement and geometry
The closest work to ours studies why gradient-based unlearning fails when forget and retain data interact. Mavrothalassitis et al. [29 ] attribute failures of ascent-based methods to statistical dependence between forget and retain sets. Other recent analyses study model resistance, stability, semantic entanglement, gradient conflict, and retain-gradient geometry [1 , 3 , 4 , 36 , 40 ] . These works diagnose retain-forget interaction through loss, gradient alignment, empirical instability, coherence, or semantic similarity. The main distinction of our work is the focus on output space: if an unlearned model is meant to approximate gold-standard retraining, then its outputs, not only its losses or parameters, are the relevant behavioral object. Our unified analysis reveals the matrices through which this interaction propagates to outputs: forget self-leverage and retain cross-leverage under the retrained model geometry.
Problem Setup
In this section, we formally define the unlearning and the gold-standard retraining objectives. We consider a training set D = D r ∪ D f \mathcal{D} = \mathcal{D}_r \cup\mathcal{D}_f D = D r ∪ D f with ∣ D ∣ = n |\mathcal{D}| = n ∣ D ∣ = n and ∣ D f ∣ = m |\mathcal{D}_f| = m ∣ D f ∣ = m , where D f \mathcal{D}_f D f is the forget set whose influence is to be removed and D r = D ∖ D f \mathcal{D}_r = \mathcal{D} \setminus\mathcal{D}_f D r = D ∖ D f is the retain set . Each sample ( x i , y i ) (x_i, y_i) ( x i , y i ) pairs an input x i x_i x i with a label y i y_i y i . A model f ( θ , x ) ∈ R c f(\theta, x) \in\mathbb{R}^{c} f ( θ , x ) ∈ R c parameterized by θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d is trained on D \mathcal{D} D under the ℓ 2 \ell_2 ℓ 2 -regularized objective
L ( θ ) = 1 n ∑ i = 1 n ℓ ( θ , x i , y i ) + λ 2 ∥ θ ∥ 2 , λ > 0 , (1) \mathcal{L}(\theta) = \frac{1}{n}\sum_{i=1}^{n}\ell(\theta, x_i, y_i) + \frac{\lambda}{2}\|\theta\|^2,\qquad\lambda> 0, \tag*{(1)} L ( θ ) = n 1 i = 1 ∑ n ℓ ( θ , x i , y i ) + 2 λ ∥ θ ∥ 2 , λ > 0 , (1)
where λ \lambda λ is the regularization strength and ℓ ( ⋅ ) \ell(\cdot) ℓ ( ⋅ ) is the loss function. The full-data optimum is θ ⋆ = arg min θ L ( θ ) \theta^\star= \arg\min_\theta\mathcal{L}(\theta) θ ⋆ = arg min θ L ( θ ) .
The gold standard. The gold-standard unlearned model is the minimizer of an analogous objective restricted to D r \mathcal{D}_r D r . We adopt the full-data-normalized retain objective
R ( θ ) : = 1 n ∑ i ∈ D r ℓ ( θ , x i , y i ) + λ 2 ∥ θ ∥ 2 , θ r ⋆ : = arg min θ R ( θ ) , (2) \mathcal{R}(\theta) := \frac{1}{n}\sum_{i\in\mathcal{D}_r}\ell(\theta, x_i, y_i) + \frac{\lambda}{2}\|\theta\|^2,\qquad\theta_r^\star:= \arg\min_\theta\mathcal{R}(\theta), \tag*{(2)} R ( θ ) := n 1 i ∈ D r ∑ ℓ ( θ , x i , y i ) + 2 λ ∥ θ ∥ 2 , θ r ⋆ := arg θ min R ( θ ) , (2)
which has the same minimizer as the standard 1 n − m \frac{1}{n-m} n − m 1 -normalized retain objective 1 n − m ∑ i ∈ D r ℓ ( θ , x i , y i ) + λ r 2 ∥ θ ∥ 2 \frac{1}{n-m}\sum_{i\in\mathcal{D}_r}\ell(\theta, x_i, y_i) + \frac{\lambda_r}{2}\|\theta\|^2 n − m 1 ∑ i ∈ D r ℓ ( θ , x i , y i ) + 2 λ r ∥ θ ∥ 2 with a scaled regularization strength λ r = n n − m λ \lambda_r = \frac{n}{n-m}\lambda λ r = n − m n λ . We refer to θ r ⋆ \theta_r^\star θ r ⋆ as the gold standard : any unlearning method ideally returns a model whose induced behavior on arbitrary inputs matches that of θ r ⋆ \theta_r^\star θ r ⋆ . Define also the full-data-normalized forget loss
F ( θ ) : = 1 n ∑ j ∈ D f ℓ ( θ , x j , y j ) , (3) \mathcal{F}(\theta) := \frac{1}{n}\sum_{j\in\mathcal{D}_f}\ell(\theta, x_j, y_j), \tag*{(3)} F ( θ ) := n 1 j ∈ D f ∑ ℓ ( θ , x j , y j ) , (3)
so that L = R + F \mathcal{L} = \mathcal{R} + \mathcal{F} L = R + F .
Gradient-objective unlearning. Most existing optimization-based unlearners minimize a controlled retain–forget trade-off
θ ρ G D i f f : = arg min θ R ( θ ) − ρ F ( θ ) , ρ > 0 , (4) \theta_{\rho}^{\mathrm{GDiff}} := \underset{\theta}{\arg\min}\ \mathcal{R}(\theta) - \rho\mathcal{F}(\theta), \qquad\rho> 0, \tag*{(4)} θ ρ GDiff := θ arg min R ( θ ) − ρ F ( θ ) , ρ > 0 , (4)
recovering retraining at ρ = 0 \rho= 0 ρ = 0 and adding active forget-ascent for ρ > 0 \rho> 0 ρ > 0 . Variants such as NPO and SimNPO replace F \mathcal{F} F by a per-sample reweighted version, IMU incorporates per-sample influence weighting on the forget term, and Pure GA omits R \mathcal{R} R entirely. Newton-style updates instead approximate θ r ⋆ \theta_r^\star θ r ⋆ in a single second-order step. Throughout, we treat θ r ⋆ \theta_r^\star θ r ⋆ as the reference solution and ask how each method’s output behavior deviates from it. For nonconvex models the minimizers in (2) and (4) are not unique. We read them as local stationary points. In our MNIST and CIFAR-10 experiments θ r ⋆ \theta_r^\star θ r ⋆ is the warm-start retrain from θ ⋆ \theta^\star θ ⋆ (on TOFU, the released retain model), and in the fixed-point validation of § 5.2 each method’s solution is the stationary point on the branch through θ r ⋆ \theta_r^\star θ r ⋆ . For a parameter θ \theta θ and evaluation inputs X a X_a X a , we call f ( θ , X a ) − f ( θ r ⋆ , X a ) f(\theta, X_a) - f(\theta_r^\star, X_a) f ( θ , X a ) − f ( θ r ⋆ , X a ) the output-space gap of θ \theta θ on X a X_a X a , and summarize its size by per-coordinate RMS or output KL.
Output-Space Gap Analysis
In this section, we derive closed-form expressions for the output-space gap between gradient-based unlearners and the gold-standard retrained model θ r ⋆ \theta_r^\star θ r ⋆ , first under a linear squared-loss setting and then under a local nonlinear approximation. Our following analysis reveals the key role of the retrained-model-geometry-based leverage matrices in structuring this gap, and it subsumes various gradient-based methods under a unified framework parametrized by geometry, reweighting, and projection.
Linear squared-loss analysis
We first analyze the output-space gap in a linear regression setting, which admits a closed-form solution and clean geometric interpretation that reveals two key factors influencing the gap. To make the gap explicit, we begin with the squared-loss linear setting f ( θ , x ) = x T θ f(\theta, x) = x^{\mathsf{T}}\theta f ( θ , x ) = x T θ , for x ∈ R d x \in\mathbb{R}^{d} x ∈ R d , θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d , and ℓ ( θ , x , y ) = 1 2 ( y − x T θ ) 2 \ell(\theta, x, y) = \frac{1}{2}(y - x^{\mathsf{T}}\theta)^2 ℓ ( θ , x , y ) = 2 1 ( y − x T θ ) 2 . Let X r ∈ R ( n − m ) × d X_r \in\mathbb{R}^{(n-m)\times d} X r ∈ R ( n − m ) × d and X f ∈ R m × d X_f \in\mathbb{R}^{m\times d} X f ∈ R m × d be the retain and forget data matrices, respectively, with labels y r ∈ R n − m y_r \in\mathbb{R}^{n-m} y r ∈ R n − m , y f ∈ R m y_f \in\mathbb{R}^{m} y f ∈ R m . The full-data objective is
L ( θ ) = 1 2 n ∥ y r − X r θ ∥ 2 2 + 1 2 n ∥ y f − X f θ ∥ 2 2 + λ 2 ∥ θ ∥ 2 2 . (5) \mathcal{L}(\theta) = \frac{1}{2n}\lVert y_r - X_r\theta\rVert_2^2 + \frac{1}{2n}\lVert y_f - X_f\theta\rVert_2^2 + \frac{\lambda}{2}\lVert\theta\rVert_2^2. \tag*{(5)} L ( θ ) = 2 n 1 ∥ y r − X r θ ∥ 2 2 + 2 n 1 ∥ y f − X f θ ∥ 2 2 + 2 λ ∥ θ ∥ 2 2 . (5)
Define the retain-Hessian and the two natural forget residuals , evaluated at the full-data optimum and the retrained optimum, respectively, as
H r : = 1 n X r T X r + λ I d , e f + : = X f θ ⋆ − y f , e f − : = X f θ r ⋆ − y f . (6) H_r := \frac{1}{n}X_r^{\mathsf{T}}X_r + \lambda I_d,\qquad e_f^+ := X_f\theta^\star- y_f,\qquad e_f^- := X_f\theta_r^\star- y_f. \tag*{(6)} H r := n 1 X r T X r + λ I d , e f + := X f θ ⋆ − y f , e f − := X f θ r ⋆ − y f . (6)
The residual e f + e_f^+ e f + is precomputable from θ ⋆ \theta^\star θ ⋆ alone, while e f − e_f^- e f − is oracle-evaluable from the retrained model. For any arbitrary evaluation data X a X_a X a , we define the leverage matrix
K a f : = X a H r − 1 X f T , (7) K_{af} := X_a H_r^{-1}X_f^{\mathsf{T}}, \tag*{(7)} K a f := X a H r − 1 X f T , (7)
where as Theorem 1 shows, the leverage matrix K a f K_{af} K a f captures the influence of unlearning the forget samples on the output on arbitrary data X a X_a X a under the retrained model geometry. Two special cases of the leverage matrix are (i) forget self-leverage K f f = X f H r − 1 X f T K_{ff} = X_f H_r^{-1}X_f^{\mathsf{T}} K f f = X f H r − 1 X f T : its diagonal entries capture the uniqueness of individual forget samples under this geometry, while its off-diagonal entries capture overlap among forget samples; and (ii) retain cross-leverage K r f = X r H r − 1 X f T K_{rf} = X_r H_r^{-1}X_f^{\mathsf{T}} K r f = X r H r − 1 X f T which measures how forget-directed changes transfer into outputs on the retain set.
The two residuals e f + e_f^+ e f + and e f − e_f^- e f − are not independent quantities; they are linked by an explicit invertible transformation through K f f K_{ff} K f f , which we use throughout to translate between equivalent forms.
Lemma 1 (Forget residual conversion). Suppose H r H_r H r is positive definite. Then the two forget residuals defined in (6) are linked by an explicit invertible transformation:
e f − = ( I m + 1 n K f f ) e f + , e f + = ( I m + 1 n K f f ) − 1 e f − . (8) e_f^- = \left(I_m + \frac{1}{n}K_{ff}\right)e_f^+,\qquad e_f^+ = \left(I_m + \frac{1}{n}K_{ff}\right)^{-1}e_f^-. \tag*{(8)} e f − = ( I m + n 1 K f f ) e f + , e f + = ( I m + n 1 K f f ) − 1 e f − . (8) In particular, e f + = 0 ⟺ e f − = 0 e_f^+ = 0 \Longleftrightarrow e_f^- = 0 e f + = 0 ⟺ e f − = 0 .
The proof is in Appendix A.2; it is a one-line consequence of the normal-equation difference between θ ⋆ \theta^{\star} θ ⋆ and θ r ⋆ \theta_r^{\star} θ r ⋆ , evaluated at X f X_f X f .
Theorem 1 (Retrained output deviation). Suppose H r H_r H r is positive definite. Then for any evaluation data X a X_a X a , the output gap of the retrained model from the full-data optimum admits two equivalent closed forms:
X a θ r ⋆ − X a θ ⋆ = 1 n K a f e f + (9) X_a\theta_r^{\star}-X_a\theta^{\star}=\frac{1}{n}K_{af}e_f^{+} \tag*{(9)} X a θ r ⋆ − X a θ ⋆ = n 1 K a f e f + (9) = 1 n K a f ( I m + 1 n K f f ) − 1 e f − , (10) =\frac{1}{n}K_{af}\left(I_m+\frac{1}{n}K_{ff}\right)^{-1}e_f^{-}, \tag*{(10)} = n 1 K a f ( I m + n 1 K f f ) − 1 e f − , (10) where the second form follows from the first by Lemma 1.
The proof is at Appendix A.3. (25) reveals that K a f K_{af} K a f captures how much of the forget residual e f + e_f^{+} e f + is transferred to the change in output on X a X_a X a when we retrain on D r D_r D r , and the e f − e_f^{-} e f − form is its dual through K f f K_{ff} K f f .
Theorem 2 (Linear GradDiff output gap). Suppose H r − ρ n X f T X f H_r-\frac{\rho}{n}X_f^{\mathsf T}X_f H r − n ρ X f T X f is positive definite (equivalently ρ n λ max ( K f f ) < 1 \frac{\rho}{n}\lambda_{\max}(K_{ff})<1 n ρ λ m a x ( K f f ) < 1 , where λ max ( ⋅ ) \lambda_{\max}(\cdot) λ m a x ( ⋅ ) captures the maximum eigenvalue, see Appendix A.5). Then for any evaluation data X a X_a X a , the output gap of GradDiff from θ r ⋆ \theta_r^{\star} θ r ⋆ has the closed form
X a θ ρ G D i f f − X a θ r ⋆ = ρ n K a f ( I m − ρ n K f f ) − 1 ( I m + 1 n K f f ) e f + . (11) X_a\theta_{\rho}^{\mathrm{GDiff}}-X_a\theta_r^{\star}
=\frac{\rho}{n}K_{af}\left(I_m-\frac{\rho}{n}K_{ff}\right)^{-1}\left(I_m+\frac{1}{n}K_{ff}\right)e_f^{+}. \tag*{(11)} X a θ ρ GDiff − X a θ r ⋆ = n ρ K a f ( I m − n ρ K f f ) − 1 ( I m + n 1 K f f ) e f + . (11) Equivalently, in terms of the retrained-model forget residual e f − = X f θ r ⋆ − y f e_f^{-}=X_f\theta_r^{\star}-y_f e f − = X f θ r ⋆ − y f ,
X a θ ρ G D i f f − X a θ r ⋆ = ρ n K a f ( I m − ρ n K f f ) − 1 e f − . (12) X_a\theta_{\rho}^{\mathrm{GDiff}}-X_a\theta_r^{\star}
=\frac{\rho}{n}K_{af}\left(I_m-\frac{\rho}{n}K_{ff}\right)^{-1}e_f^{-}. \tag*{(12)} X a θ ρ GDiff − X a θ r ⋆ = n ρ K a f ( I m − n ρ K f f ) − 1 e f − . (12)
The proof is at Appendix A.4. The two forms (11) and (12) are equivalent by the residual conversion identity in Lemma 1. Eq. (12) makes the structural decomposition cleanest: the retrained-model forget residual e f − e_f^{-} e f − is amplified by the inverse stability factor ( I m − ρ n K f f ) − 1 \left(I_m-\frac{\rho}{n}K_{ff}\right)^{-1} ( I m − n ρ K f f ) − 1 , which captures how forget-directed perturbations are amplified under the retrained model geometry, and then transferred to the output gap on X a X_a X a by the leverage matrix K a f K_{af} K a f . Eq. (11), on the other hand, reveals that the gap is computable entirely from θ ⋆ \theta^{\star} θ ⋆ without any retraining since we can compute the retain Hessian H r = H − H f H_r=H-H_f H r = H − H f , where H = 1 n X T X + λ I d H=\frac{1}{n}X^{\mathsf T}X+\lambda I_d H = n 1 X T X + λ I d is the full Hessian and H f = 1 n X f T X f H_f=\frac{1}{n}X_f^{\mathsf T}X_f H f = n 1 X f T X f is the forget Hessian, making (11) an a priori diagnostic that bounds the output-space gap on any candidate forget set D f D_f D f before any unlearning is run. We adopt the Eq. (12) form in the following analysis for readability; all bounds and takeaways translate immediately to e f + e_f^{+} e f + via the same identity.
The two corollaries below summarize the practical norm bound and the separation between forget and retain output deviations.
Corollary 1 (Stability of the effective residual). Under the assumption of Theorem 2,
∥ X a θ ρ G D i f f − X a θ r ⋆ ∥ 2 ≤ ρ n ∥ K a f ∥ 2 ∥ e f − ∥ 2 1 − ρ n λ max ( K f f ) . (13) \left\|X_a\theta_{\rho}^{\mathrm{GDiff}}-X_a\theta_r^{\star}\right\|_2
\leq\frac{\rho}{n}\frac{\left\|K_{af}\right\|_2\left\|e_f^{-}\right\|_2}{1-\frac{\rho}{n}\lambda_{\max}(K_{ff})}. \tag*{(13)} X a θ ρ GDiff − X a θ r ⋆ 2 ≤ n ρ 1 − n ρ λ m a x ( K f f ) ∥ K a f ∥ 2 e f − 2 . (13) or equivalently, in terms of the precomputable full-data residual,
∥ X a θ ρ G D i f f − X a θ r ⋆ ∥ 2 ≤ ρ n ∥ K a f ∥ 2 ( 1 + 1 n λ max ( K f f ) ) ∥ e f + ∥ 2 1 − ρ n λ max ( K f f ) . (14) \left\|X_a\theta_{\rho}^{\mathrm{GDiff}}-X_a\theta_r^{\star}\right\|_2
\leq\frac{\rho}{n}
\frac{\left\|K_{af}\right\|_2\left(1+\frac{1}{n}\lambda_{\max}(K_{ff})\right)\left\|e_f^{+}\right\|_2}
{1-\frac{\rho}{n}\lambda_{\max}(K_{ff})}. \tag*{(14)} X a θ ρ GDiff − X a θ r ⋆ 2 ≤ n ρ 1 − n ρ λ m a x ( K f f ) ∥ K a f ∥ 2 ( 1 + n 1 λ m a x ( K f f ) ) e f + 2 . (14)
Corollary 2 (Forget and retain output deviations). Under the assumption of Theorem 2, the GradDiff output gap on the forget and retain sets is
X f θ ρ G D i f f − X f θ r ⋆ = ρ n K f f ( I m − ρ n K f f ) − 1 e f − , (15) X_f\theta_{\rho}^{\mathrm{GDiff}}-X_f\theta_r^{\star}
=\frac{\rho}{n}K_{ff}\left(I_m-\frac{\rho}{n}K_{ff}\right)^{-1}e_f^{-}, \tag*{(15)} X f θ ρ GDiff − X f θ r ⋆ = n ρ K f f ( I m − n ρ K f f ) − 1 e f − , (15) X r θ ρ G D i f f − X r θ r ⋆ = ρ n K r f ( I m − ρ n K f f ) − 1 e f − . (16) X_r\theta_{\rho}^{\mathrm{GDiff}}-X_r\theta_r^{\star}
=\frac{\rho}{n}K_{rf}\left(I_m-\frac{\rho}{n}K_{ff}\right)^{-1}e_f^{-}. \tag*{(16)} X r θ ρ GDiff − X r θ r ⋆ = n ρ K r f ( I m − n ρ K f f ) − 1 e f − . (16) The corresponding e f + e_f^{+} e f + forms follow from Lemma 1.
Corollary 2 delivers the central design principle of this paper: K f f K_{ff} K f f and K r f K_{rf} K r f capture qualitatively different geometric information about the forget set and act asymmetrically on the output gap. At a high level, K f f = X f H r − 1 X f ⊤ K_{ff}=X_fH_r^{-1}X_f^\top K f f = X f H r − 1 X f ⊤ is an intrinsic property of the forget set under the retrained-model geometry: its diagonal entries measure how distinctive each forget sample is once the retain data is accounted for, and its eigenvalues control how strongly forget-directed perturbations are amplified by ( I − ρ n K f f ) − 1 \left(I-\frac{\rho}{n}K_{ff}\right)^{-1} ( I − n ρ K f f ) − 1 . By contrast, K r f = X r H r − 1 X f ⊤ K_{rf}=X_rH_r^{-1}X_f^\top K r f = X r H r − 1 X f ⊤ is a cross-set alignment quantity, capturing how forget-directed updates project onto retain directions; it carries no information about the forget set in isolation. In the next section, we show that most existing gradient-based methods only partially account for the leverage geometry: they control the effective forget residual through reweighting or projection, but they do not fully eliminate the underlying forget self-leverage or retain cross-leverage.
Unified method view
So far our analysis has focused on the GradDiff method. In this section, we extend the analysis to a broader class of gradient-based unlearning methods. Each method differs only in (i) its effective leverage matrix (retrained-model Hessian H r − 1 H_r^{-1} H r − 1 vs. Euclidean λ − 1 I \lambda^{-1}I λ − 1 I ) and (ii) a per-sample reweighting matrix or fixed projector. Define the diagonal weighting matrices
W N P O − = diag ( σ ( z j − ) ) j = 1 m , W S i m N P O − = diag ( σ ( z ~ j − ) ) j = 1 m , A I M U = diag ( α j ) j = 1 m , (17) W_{\mathrm{NPO}}^{-}=\operatorname{diag}\left(\sigma(z_j^{-})\right)_{j=1}^{m},\qquad W_{\mathrm{SimNPO}}^{-}=\operatorname{diag}\left(\sigma(\widetilde{z}_j^{-})\right)_{j=1}^{m},\qquad A_{\mathrm{IMU}}=\operatorname{diag}(\alpha_j)_{j=1}^{m}, \tag*{(17)} W NPO − = diag ( σ ( z j − ) ) j = 1 m , W SimNPO − = diag ( σ ( z j − ) ) j = 1 m , A IMU = diag ( α j ) j = 1 m , (17)
where σ ( t ) = ( 1 + e − t ) − 1 \sigma(t)=(1+e^{-t})^{-1} σ ( t ) = ( 1 + e − t ) − 1 is the logistic sigmoid, z j − , z ~ j − z_j^{-},\widetilde{z}_j^{-} z j − , z j − are the NPO/SimNPO logit arguments evaluated at θ r ∗ \theta_r^* θ r ∗ , and α j = ∥ H θ ∗ − 1 ∇ θ ℓ ( θ ∗ , x j , y j ) ∥ 2 \alpha_j=\left\|H_{\theta^*}^{-1}\nabla_\theta\ell(\theta^*,x_j,y_j)\right\|_2 α j = H θ ∗ − 1 ∇ θ ℓ ( θ ∗ , x j , y j ) 2 is IMU’s per-sample influence norm, where H θ ∗ H_{\theta^*} H θ ∗ is the full-data Hessian at θ ∗ \theta^* θ ∗ . Explicitly, for the squared-loss surrogate, z j − = − β N P O ( ℓ j ( θ r ∗ ) − ℓ j ( θ r e f ) ) z_j^{-}=-\beta_{\mathrm{NPO}}\left(\ell_j(\theta_r^*)-\ell_j(\theta_{\mathrm{ref}})\right) z j − = − β NPO ( ℓ j ( θ r ∗ ) − ℓ j ( θ ref ) ) and z ~ j − = − β S i m N P O ℓ j ( θ r ∗ ) \widetilde{z}_j^{-}=-\beta_{\mathrm{SimNPO}}\ell_j(\theta_r^*) z j − = − β SimNPO ℓ j ( θ r ∗ ) . We adopt the convention that the (Sim)NPO objectives use the prefactor ρ / β \rho/\beta ρ / β in front of the inner log σ ( ⋅ ) \log\sigma(\cdot) log σ ( ⋅ ) term (Appendix § 9 .6), so that the inverse-temperature β > 0 \beta>0 β > 0 that arises from differentiating the inner sigmoid cancels with the prefactor and the effective coefficient appearing in Table 1 is exactly ρ \rho ρ with the β \beta β -free weights above; under any other convention β \beta β would simply be absorbed into ρ \rho ρ . Let also G f f = X f X f ⊤ G_{ff}=X_fX_f^\top G f f = X f X f ⊤ and G a f = X a X f ⊤ G_{af}=X_aX_f^\top G a f = X a X f ⊤ be Euclidean Gram matrices. For ProjGD, let P r P_r P r be a fixed projector that removes retain-gradient-aligned directions from the forget gradient, and define the projected leverage matrices as
K a f P : = X a H r − 1 P r X f ⊤ , K f f P : = X f H r − 1 P r X f ⊤ . K_{af}^{P}:=X_aH_r^{-1}P_rX_f^\top,\qquad K_{ff}^{P}:=X_fH_r^{-1}P_rX_f^\top. K a f P := X a H r − 1 P r X f ⊤ , K f f P := X f H r − 1 P r X f ⊤ .
Proposition 1 (Unified linear gaps ). For the fixed-weight or frozen-local linear surrogate specified above, whenever the displayed inverse exists, each method’s stationary output gap from θ r ∗ \theta_r^* θ r ∗ admits the closed form in Table 1 . Invertibility ensures a unique stationary solution, while-definite Hessians additionally ensure unique minima for the quadratic objectives.
Table 1 shows that all methods share the same basic structure: an effective forget residual is formed by an inverse term on the forget set, then transferred to evaluation outputs through a method-specific geometry. GradDiff, NPO, SimNPO, and fixed-projector ProjGD use the retrained-Hessian geometry through K K K -matrices, while Pure GA and IMU use Euclidean Gram matrices G G G . Reweighting and projection change the inverse term and transfer map, but they do not remove the output-space gap in general. This distinction governs which leverage diagnostic should predict each method’s behavior.
Concretely, for a fixed forget residual:
Forget self-leverage governs amplification on any evaluation set. In Eq. (12) , K f f K_{ff} K f f enters the inverse stability factor ( I m − ρ n K f f ) − 1 \left(I_{m}-\frac{\rho}{n}K_{ff}\right)^{-1} ( I m − n ρ K f f ) − 1 . At fixed ρ \rho ρ , increasing ∥ K f f ∥ 2 \lVert K_{ff}\rVert_{2} ∥ K f f ∥ 2 increases its worst-case amplification within the stable region; realized gaps also depend on residual alignment and the evaluation map. On the forget set, K f f K_{ff} K f f additionally appears in the prefactor of (15).
Retain cross-leverage governs transfer to retain outputs. For fixed K f f K_{ff} K f f and forget residual, the retain-gap bound scales linearly with ∥ K r f ∥ 2 \lVert K_{rf}\rVert_{2} ∥ K r f ∥ 2 , while the realized gap depends on alignment with the effective residual. The forget gap does not involve K r f K_{rf} K r f , and gaps on other evaluation sets depend on their respective K a f K_{af} K a f .
Local nonlinear approximation
In this section, we move away from the linear setting and show that the same structure persists for nonlinear models under a local approximation around θ r ⋆ \theta_{r}^{\star} θ r ⋆ . Define for each forget sample j j j the per-sample Jacobian, output-gradient, and output-space Hessian
J j − = ∇ θ f ( θ r ⋆ , x j ) , δ j − = ∇ z ℓ ( z , y j ) ∣ z = f ( θ r ⋆ , x j ) , B j − = ∇ z 2 ℓ ( z , y j ) ∣ z = f ( θ r ⋆ , x j ) , (18) J_{j}^{-}=\nabla_{\theta}f(\theta_{r}^{\star},x_{j}),\qquad\delta_{j}^{-}=\left.\nabla_{z}\ell(z,y_{j})\right|_{z=f(\theta_{r}^{\star},x_{j})},\qquad B_{j}^{-}=\left.\nabla_{z}^{2}\ell(z,y_{j})\right|_{z=f(\theta_{r}^{\star},x_{j})}, \tag*{(18)} J j − = ∇ θ f ( θ r ⋆ , x j ) , δ j − = ∇ z ℓ ( z , y j ) ∣ z = f ( θ r ⋆ , x j ) , B j − = ∇ z 2 ℓ ( z , y j ) z = f ( θ r ⋆ , x j ) , (18)
with stacked counterparts J f − ∈ R m c × d J_{f}^{-}\in\mathbb{R}^{mc\times d} J f − ∈ R m c × d , δ f − ∈ R m c \delta_{f}^{-}\in\mathbb{R}^{mc} δ f − ∈ R m c , B f − ∈ R m c × m c B_{f}^{-}\in\mathbb{R}^{mc\times mc} B f − ∈ R m c × m c (block-diagonal), where c c c is the output dimension. Let H r − H_{r}^{-} H r − be the positive-definite local retain curvature used by the approximation (the exact Hessian in an exact Taylor model, or the retain GGN curvature in the experiments), and define the nonlinear leverage matrices
K a f N L = J a − ( H r − ) − 1 ( J f − ) T , K f f N L = J f − ( H r − ) − 1 ( J f − ) T . (19) K_{af}^{\mathrm{NL}}=J_{a}^{-}(H_{r}^{-})^{-1}(J_{f}^{-})^{\mathsf{T}},\qquad K_{ff}^{\mathrm{NL}}=J_{f}^{-}(H_{r}^{-})^{-1}(J_{f}^{-})^{\mathsf{T}}. \tag*{(19)} K a f NL = J a − ( H r − ) − 1 ( J f − ) T , K f f NL = J f − ( H r − ) − 1 ( J f − ) T . (19)
Also define the forget GGN curvature and the local stability matrix
H f , G : = 1 n ( J f − ) T B f − J f − , M ρ − : = H r − − ρ H f , G . (20) H_{f,G}:=\frac{1}{n}(J_{f}^{-})^{\mathsf{T}}B_{f}^{-}J_{f}^{-},\qquad M_{\rho}^{-}:=H_{r}^{-}-\rho H_{f,G}. \tag*{(20)} H f , G := n 1 ( J f − ) T B f − J f − , M ρ − := H r − − ρ H f , G . (20)
Proposition 2 (Local nonlinear GradDiff gap). Suppose R \mathcal{R} R and F \mathcal{F} F are twice continuously differentiable near θ r ⋆ \theta_{r}^{\star} θ r ⋆ , and that M ρ − M_{\rho}^{-} M ρ − is positive definite. The local GGN/Taylor predictor for the GradDiff parameter gap is
Δ ^ ρ : = ρ ( M ρ − ) − 1 ∇ F ( θ r ⋆ ) = ρ n ( H r − ) − 1 ( J f − ) T ( I m c − ρ n B f − K f f N L ) − 1 δ f − . (21) \widehat{\Delta}_{\rho}:=\rho(M_{\rho}^{-})^{-1}\nabla F(\theta_{r}^{\star})=\frac{\rho}{n}(H_{r}^{-})^{-1}(J_{f}^{-})^{\mathsf{T}}\left(I_{mc}-\frac{\rho}{n}B_{f}^{-}K_{ff}^{\mathrm{NL}}\right)^{-1}\delta_{f}^{-}. \tag*{(21)} Δ ρ := ρ ( M ρ − ) − 1 ∇ F ( θ r ⋆ ) = n ρ ( H r − ) − 1 ( J f − ) T ( I m c − n ρ B f − K f f NL ) − 1 δ f − . (21) Consequently, for any evaluation set A A A with stacked Jacobian J a − J_{a}^{-} J a − , the first-order predicted output gap is
Δ ^ f A : = J a − Δ ^ ρ = ρ n K a f N L ( I m c − ρ n B f − K f f N L ) − 1 δ f − , (22) \widehat{\Delta}_{f}^{A}:=J_{a}^{-}\widehat{\Delta}_{\rho}=\frac{\rho}{n}K_{af}^{\mathrm{NL}}\left(I_{mc}-\frac{\rho}{n}B_{f}^{-}K_{ff}^{\mathrm{NL}}\right)^{-1}\delta_{f}^{-}, \tag*{(22)} Δ f A := J a − Δ ρ = n ρ K a f NL ( I m c − n ρ B f − K f f NL ) − 1 δ f − , (22) with the nonlinear evaluation remainder controlled by the local-error discussion below.
The proof (Appendix A.7) is a Taylor expansion of the stationarity condition combined with the same push-through identity used in Theorem 2. Equation (22) preserves every interpretive feature of the linear case: the residual term δ f − \delta_{f}^{-} δ f − measures forget mispredictions at θ r ⋆ \theta_{r}^{\star} θ r ⋆ , the leverage K a f N L K_{af}^{\mathrm{NL}} K a f NL transfers them to evaluation outputs, and the effective-residual factor becomes ill-conditioned as the smallest singular value of I m c − ( ρ / n ) B f − K f f N L I_{mc}-(\rho/n)B_{f}^{-}K_{ff}^{\mathrm{NL}} I m c − ( ρ / n ) B f − K f f NL approaches zero. Analogous local approximations for the other methods in Table 1 are obtained by the same expansion and are stated in Appendix A.9.
Approximation error. Equation (22) is a linear-response approximation to how the GradDiff stationary point moves under a forget-set reweighting. This is the same perturbative object studied by influence functions and the infinitesimal jackknife; finite-sample error bounds for such linearized reweighting approximations require local Hessian invertibility and Lipschitz/Taylor control of the gradients [10 , 18 ] . In our notation, the error has the same form: it is amplified by the local conditioning of M ρ − M_{\rho}^{-} M ρ − , is second order in the displacement when exact Hessians are used, and gains an additional first-order term when the exact Hessian is replaced by a GGN curvature. A representative bound and derivation are given in Appendix § 9 .§ 14 ; we use it only to justify the locality of the diagnostic, not as a new tight guarantee.
Our formulas give exact stationary equations in linear regression, but introduce local approximations for nonlinear models. Moreover, practical implementations use stochastic finite-step updates, retain subsets, approximate Hessians, and sometimes dynamically updated weights or projectors. The approximation-error discussion above shows how these local errors enter the output gap; Section 5 shows that the leverage matrices remain informative in practice despite such implementation noise.
Experiments
In this section, we demonstrate how one may apply the leverage geometry analysis to understand and predict the behavior of unlearning methods in practice. We experiment on MNIST with a small MLP to (i) measure the leverage geometry (K f f , K r f , e f K_{ff}, K_{rf}, e^f K f f , K r f , e f ) that § 4.1 predicts to govern the output-space gap, in two qualitatively different forgetting regimes, and (ii) compare the five unlearners’ output-space distance to the gold-standard retrained model in both regimes. The motivating evidence that loss and weight-space distance are not faithful evaluation surfaces is summarized in Figure 1 ; this section quantifies the geometry that underlies that dissociation. A scalable θ ∗ \theta^* θ ∗ -only leverage score at LLM scale on TOFU is reported in Appendix § 11 , and additional validation of the exact linear formula, the pointwise nonlinear predictor, and the stability bound appears in Appendix § 12 . Full reproducibility details are in Appendix § 10 . Output deviations are reported as per-coordinate logit root-mean-square (RMS), E ∥ Δ z ∥ 2 2 / c \sqrt{\mathbb{E}\lVert\Delta z\rVert_2^2/c} E ∥ Δ z ∥ 2 2 / c , where Δ z \Delta z Δ z is the output logits difference measured against the reference θ r ∗ \theta_r^* θ r ∗ .
Class vs. random forgetting: leverage geometry and method behavior
We construct two regimes of forget set on MNIST. In the class regime, D f D_f D f contains all instances of one class in the 4,000-sample training subset (so ∣ D f ∣ ≈ 350 \lvert D_f\rvert\approx350 ∣ D f ∣ ≈ 350 –490, reflecting natural class-size variation across the seed’s subsample rather than a strict 350–490 samples per class); in the random regime, D f D_f D f is a uniformly random training subsample of size ∣ D f ∣ = 200 \lvert D_f\rvert= 200 ∣ D f ∣ = 200 . For each forget set we form θ r ∗ \theta_r^* θ r ∗ by fixed-step retraining and estimate the leverage geometry using a retain-normalized GGN surrogate (Appendix § 10 ). The four iterative unlearners take 500 full-batch Adam steps; Newton Step takes one update. Aggregates are over 10 classes × \times × 3 seeds (class regime) and 100 random sets × \times × 3 seeds (random regime).
The bottom block of Table 2 reports the central observable, R M S f \mathrm{RMS}_f RMS f and R M S r \mathrm{RMS}_r RMS r from θ r ∗ \theta_r^* θ r ∗ . Three takeaways follow from this comparison.
Table 2. Class vs. random forgetting on MNIST. Top: mean leverage estimates using retain-normalized GGN curvature at θ r ∗ \theta_r^* θ r ∗ . Bottom: per-coordinate logit RMS gaps from θ r ∗ \theta_r^* θ r ∗ on D f D_f D f and D r D_r D r and their ratio. Class-regime forget accuracy is zero for GradDiff, NPO, SimNPO, and ProjGD, and 0.005 ± 0.020 0.005 \pm0.020 0.005 ± 0.020 for Newton Step. Loss ranges are summarized in the text; setup details in Appendix § 10 .
(i) The forget channel is consistently larger than the retain channel. For every method and every regime, R M S f ≥ R M S r \mathrm{RMS}_f \ge\mathrm{RMS}_r RMS f ≥ RMS r , with ratios 1.0–1.55. Equations (15) –(16) distinguish these channels: K f f K_{ff} K f f enters the forget gap both as a prefactor and inside the inverse stability factor, while K r f K_{rf} K r f transfers the effective residual to retain outputs. The observed ordering also depends on residual alignment.
(ii) Class forgetting concentrates the residual; the leverage matrices grow much less. Class-regime forget CE is similar for the four iterative methods (4.368–4.404), all with zero forget accuracy, yet R M S f \mathrm{RMS}_f RMS f differs by about 58× between SimNPO (0.009) and ProjGD (0.509). Newton has lower CE (2.972), so its larger gap illustrates accuracy mismatch only. We summarize these metrics here and in Appendix § 10 rather than add columns to Table 2 . Going from random to class forgetting, ∥ e f ∥ \lVert e_f\rVert ∥ e f ∥ grows by 4.59×, whereas the surrogate estimates of λ max ( K f f ) \lambda_{\max}(K_{ff}) λ m a x ( K f f ) and ∥ K r f ∥ F \lVert K_{rf}\rVert_F ∥ K r f ∥ F grow by 1.57× and 1.32×: the residual changes much more than the leverage norms. The iterative methods (GradDiff, NPO, SimNPO) scale R M S f \mathrm{RMS}_f RMS f by 1.8–3.8×, consistent with the residual ratio modulated by geometry and method weights. SimNPO separates from NPO mainly in the class regime. There the forget CE near θ f ∗ \theta_f^* θ f ∗ is large (≈ 4.4 \approx4.4 ≈ 4.4 vs. 0.33 for random sets), so the default β = 2.5 \beta=2.5 β = 2.5 saturates its weights (w ˉ ≈ 10 − 4 \bar{w}\approx10^{-4} w ˉ ≈ 1 0 − 4 ), switching the forget term off and leaving the iterate near θ r ∗ \theta_r^* θ r ∗ , whereas NPO at β = 0.1 \beta=0.1 β = 0.1 keeps nearly uniform O ( 1 ) O(1) O ( 1 ) weights and so behaves like GradDiff with a rescaled ρ \rho ρ . Swapping the two β \beta β values swaps the behavior (Table 11 ). ProjGD and the Newton step have larger output gaps; Appendix § 12 discusses their behavior. Because Eq. (14) bounds the gap from θ ∗ \theta^* θ ∗ quantities alone, the leverage geometry can still serve as an a-priori predictive diagnostic when a retrained reference is unavailable.
Table 11. Table 11: β \beta β -swap control on MNIST. The four corners of the (method, β \beta β ) square under the protocol of Table 9, scored with uniform weights (10 seeds). w ˉ ( θ r ∗ ) \bar{w}(\theta_r^*) w ˉ ( θ r ∗ ) is the mean per-example forget weight of the reference in the deployed 2 σ ( ⋅ ) 2\sigma(\cdot) 2 σ ( ⋅ ) convention, 2 σ ( β log p θ ∗ ′ − log p θ r ∗ ) 2\sigma(\beta\log p_{\theta^*}'-\log p_{\theta_r^*}) 2 σ ( β log p θ ∗ ′ − log p θ r ∗ ) for NPO and 2 σ ( β log p θ r ∗ ) 2\sigma(\beta\log p_{\theta_r^*}) 2 σ ( β log p θ r ∗ ) for SimNPO. At β = 0.1 \beta=0.1 β = 0.1 both methods agree with the predictor equally well. At β = 2.5 \beta=2.5 β = 2.5 both collapse in the class regime. There the weights saturate (w ˉ ∼ 10 − 4 \bar{w}\sim10^{-4} w ˉ ∼ 1 0 − 4 when C E f ( θ r ∗ ) ≈ 4.4 \mathrm{CE}_f(\theta_r^*)\approx4.4 CE f ( θ r ∗ ) ≈ 4.4 ), the effective forget term vanishes, and the endpoint stays at θ r ∗ \theta_r^* θ r ∗ up to optimizer noise, so the forget CE equals the reference value and RelErr ≫ 1 \gg1 ≫ 1 . Swapping β \beta β swaps the behavior, so the NPO/SimNPO difference comes from β \beta β -induced weight saturation and not from the reference-based vs. reference-free form.
(iii) Geometry motivates a per-set ρ \rho ρ rule. The stability condition of Theorem 2 requires s : = ( ρ / η ) λ max ( K f f ) < 1 s := (\rho/\eta)\lambda_{\max}(K_{ff}) < 1 s := ( ρ / η ) λ m a x ( K f f ) < 1 , motivating ρ = s η / λ max ( K f f ) \rho= s\eta/\lambda_{\max}(K_{ff}) ρ = sη / λ m a x ( K f f ) . In Table 2 , we use s = 0.3 s=0.3 s = 0.3 with the surrogate kernels described above. This is an empirical heuristic: the per-set ρ \rho ρ varies by an order of magnitude across random forget sets, yet optimization is stable across all 330 forget sets without per-set grid search. The positive-definiteness guarantee applies to the consistently normalized, unprojected frozen GGN models in Appendix § 10 ; locality up to s = 2 s=2 s = 2 remains empirical. Saturated SimNPO weights motivate the weighted kernel in § 5.2 .
LLM-scale evidence. On TOFU with Llama-3.2-1B-Instruct, K K K -channel scores computed once at θ ∗ \theta^* θ ∗ in a LoRA chart rank, before any unlearning run, the retain examples that GradDiff, NPO, and SimNPO will disturb most (Spearman 0.26–0.49 over three seeds, far above the permutation null) and, less sharply, the forget examples they will treat least like retraining (0.14–0.29, null for NPO on forget01; Appendix § 11 ). They are a cheap per-example signal of unlearning difficulty and spillover, on par with per-example loss and gradient-norm baselines. With curvature restricted to a LoRA chart and early-stopped endpoints they do not yet predict the size of the gap. Whether richer curvature estimates or endpoints nearer stationarity improve these diagnostics remains open.
Local gap validation and controls
We validate the local GradDiff, NPO, and SimNPO predictors on MNIST and a head over frozen CIFAR-10 features (Appendix § 10 ). Their stationary endpoints are reached by Newton continuation from θ r ∗ \theta_r^* θ r ∗ with s = 0.1 s = 0.1 s = 0.1 , and the corresponding predictors are compared with actual logit gaps on D f \mathcal{D}_f D f and 256 retain anchors (Table 3 ). These endpoints lie within 5 × 10 − 3 5 \times10^{-3} 5 × 1 0 − 3 of θ r ∗ \theta_r^* θ r ∗ in parameter norm. Predictions track both channels and datasets (P f K = 0.89 P_f^K = 0.89 P f K = 0.89 –0.98 0.98 0.98 , P r K = 0.72 P_r^K = 0.72 P r K = 0.72 –0.97 0.97 0.97 , relative ℓ 2 \ell_2 ℓ 2 error 0.25 0.25 0.25 –0.84 0.84 0.84 ) and beat the Euclidean control throughout. As a control, ProjGD uses projected-gradient dynamics with a recomputed projector and is compared with the unprojected predictor ((22) ): their directions correlate (P f K = 0.79 P_f^K = 0.79 P f K = 0.79 –0.97 0.97 0.97 ), but the actual gap is about 7 7 7 –15 × 15\times 15 × smaller. Pure GA and IMU provide controls outside the local regime: their endpoints have output KL 0.7 0.7 0.7 –3 3 3 nats from θ r ∗ \theta_r^* θ r ∗ , versus at most 10 − 5 10^{-5} 1 0 − 5 for the retain-anchored methods, and their predictors are uninformative (Section 6).
Table 3. Local gap validation and controls. Per-logit Pearson correlation between predicted and actual gaps on D f \mathcal{D}_f D f (f f f ) and 256 retain anchors (r r r ), using retain-GGN (P K P^K P K ) or Euclidean-Gram (P G P^G P G ) geometry; mean± \pm ± sd over 10 seeds at s = 0.1 s=0.1 s = 0.1 . GradDiff/NPO/SimNPO test the local predictors of Table 4 ; NPO/SimNPO predictors freeze reference weights, and SimNPO uses the weighted ρ \rho ρ rule. ProjGD compares its projected-gradient endpoint with the unprojected predictor; Pure GA is a nonlocal control.
The local validation concerns the stationary branch through the specified retrained reference; initialization at the original model can reach a different branch where the predictor fails, as demonstrated in Appendix § 10 .
Limitations and Discussion
Our analysis is exact for linear regression, but nonlinear approximations and practical implementations introduce additional error. Although Appendix § 9 .§ 14 analyzes this error and § 5.2 and Appendix § 12 validate the approximation empirically, the leverage matrices should be read as output-space risk diagnostics rather than exact predictors, and they are local to the specified retrained reference and to the stationary branch through it (§ 3 , Appendix § 10 ). Relatedly, Pure GA and IMU have no term anchoring their iterates near θ ∗ \theta^* θ ∗ . Their rows in Table 1 remain exact but contain − X a θ a ∗ -X_a\theta_a^* − X a θ a ∗ , so their gap is of the order of the reference outputs and the local predictor is uninformative at their converged endpoints under either geometry (§ 5.2 ), while their early-stopped operating points (Figure 1 ) are not stationary points and lie outside our analysis. Computing the full gap is also infeasible for large models: Appendix § 11 .§ 2 shows that LoRA-space diagonal-GGN approximations of K f f K_{ff} K f f and K r f K_{rf} K r f are informative on TOFU, but their modest correlations leave better estimators as an open problem.
These limitations suggest future methods that account for both forget self-leverage and retain cross-leverage when choosing ρ \rho ρ , reweighting forget samples, or projecting updates, rather than optimizing losses alone. The precomputable e f + e_f^+ e f + form of Theorem 1 also suggests LLM-scale audits that combine released-model forget residuals with sketched or low-rank K f f K_{ff} K f f estimates.
The broader impact is primarily diagnostic: output-space auditing can improve privacy, data-removal, copyright, and safety evaluations, but approximate diagnostics should complement behavioral evaluation, privacy auditing, and transparent assumptions rather than serve as compliance certificates.
Conclusion
We studied gradient-based machine unlearning through the output-space gap to gold-standard retraining. In linear regression, this gap admits a closed form; under a local GGN approximation, the same structure extends to nonlinear models. The resulting framework unifies many common gradient-based unlearners and identifies two leverage matrices that govern how forget-set residuals propagate: forget self-leverage, which controls behavior on the forget set, and retain cross-leverage, which controls transfer into retain outputs. Together, these matrices provide a sharper language for comparing existing methods, auditing forget sets, and designing better unlearning algorithms.
Acknowledgments and Disclosure of Funding
This work received no third-party funding, and we declare no competing interests.
Proofs and Detailed Derivations
Auxiliary push-through identity
Lemma 2 (Push-through identity). Let A A A , U U U , C C C , and V V V be conformable matrices. If A A A and I − C V A − 1 U I-CVA^{-1}U I − C V A − 1 U are invertible, then A − U C V A-UCV A − U C V is invertible and
( A − U C V ) − 1 U = A − 1 U ( I − C V A − 1 U ) − 1 . (A-UCV)^{-1}U=A^{-1}U(I-CVA^{-1}U)^{-1}. ( A − U C V ) − 1 U = A − 1 U ( I − C V A − 1 U ) − 1 .
Proof. By Sylvester’s determinant identity, I − A − 1 U C V I-A^{-1}UCV I − A − 1 U C V is invertible whenever I − C V A − 1 U I-CVA^{-1}U I − C V A − 1 U is invertible; hence A − U C V = A ( I − A − 1 U C V ) A-UCV=A(I-A^{-1}UCV) A − U C V = A ( I − A − 1 U C V ) is invertible. Start from
( A − U C V ) A − 1 U = U − U C V A − 1 U = U ( I − C V A − 1 U ) . (A-UCV)A^{-1}U=U-UCVA^{-1}U=U(I-CVA^{-1}U). ( A − U C V ) A − 1 U = U − U C V A − 1 U = U ( I − C V A − 1 U ) .
Multiplying on the left by ( A − U C V ) − 1 (A-UCV)^{-1} ( A − U C V ) − 1 and on the right by ( I − C V A − 1 U ) − 1 (I-CVA^{-1}U)^{-1} ( I − C V A − 1 U ) − 1 gives the result. □ \square □
Proof of Lemma 1
In the linear squared-loss setting the full-data objective is
L ( θ ) = 1 2 n ∥ y r − X r θ ∥ 2 2 + 1 2 n ∥ y f − X f θ ∥ 2 2 + λ 2 ∥ θ ∥ 2 2 , \mathcal{L}(\theta)=\frac{1}{2n}\lVert y_r-X_r\theta\rVert_2^2+\frac{1}{2n}\lVert y_f-X_f\theta\rVert_2^2+\frac{\lambda}{2}\lVert\theta\rVert_2^2, L ( θ ) = 2 n 1 ∥ y r − X r θ ∥ 2 2 + 2 n 1 ∥ y f − X f θ ∥ 2 2 + 2 λ ∥ θ ∥ 2 2 ,
whose stationarity condition ∇ L ( θ ⋆ ) = 0 \nabla\mathcal{L}(\theta^\star)=0 ∇ L ( θ ⋆ ) = 0 reads
( H r + 1 n X f T X f ) θ ⋆ = b r + 1 n X f T y f , b r : = 1 n X r T y r . (23) \left(H_r+\frac{1}{n}X_f^\mathsf{T}X_f\right)\theta^\star=b_r+\frac{1}{n}X_f^\mathsf{T}y_f,\qquad b_r:=\frac{1}{n}X_r^\mathsf{T}y_r. \tag*{(23)} ( H r + n 1 X f T X f ) θ ⋆ = b r + n 1 X f T y f , b r := n 1 X r T y r . (23)
The retrained optimum satisfies ∇ R ( θ r ⋆ ) = 0 \nabla R(\theta_r^\star)=0 ∇ R ( θ r ⋆ ) = 0 , i.e. H r θ r ⋆ = b r H_r\theta_r^\star=b_r H r θ r ⋆ = b r . Rearranging (23),
H r θ ⋆ = b r − 1 n X f T ( X f θ ⋆ − y f ) = b r − 1 n X f T e f + . H_r\theta^\star=b_r-\frac{1}{n}X_f^\mathsf{T}(X_f\theta^\star-y_f)=b_r-\frac{1}{n}X_f^\mathsf{T}e_f^+. H r θ ⋆ = b r − n 1 X f T ( X f θ ⋆ − y f ) = b r − n 1 X f T e f + .
Subtracting from H r θ r ⋆ = b r H_r\theta_r^\star=b_r H r θ r ⋆ = b r and using positive-definiteness of H r H_r H r gives the parameter shift
H r ( θ r ⋆ − θ ⋆ ) = 1 n X f T e f + ⟹ θ r ⋆ − θ ⋆ = 1 n H r − 1 X f T e f + . (24) H_r(\theta_r^\star-\theta^\star)=\frac{1}{n}X_f^\mathsf{T}e_f^+\quad\Longrightarrow\quad\theta_r^\star-\theta^\star=\frac{1}{n}H_r^{-1}X_f^\mathsf{T}e_f^+. \tag*{(24)} H r ( θ r ⋆ − θ ⋆ ) = n 1 X f T e f + ⟹ θ r ⋆ − θ ⋆ = n 1 H r − 1 X f T e f + . (24)
Evaluating (24) at X f X_f X f ,
e f − − e f + = X f ( θ r ⋆ − θ ⋆ ) = 1 n X f H r − 1 X f T e f + = 1 n K f f e f + . e_f^--e_f^+=X_f(\theta_r^\star-\theta^\star)=\frac{1}{n}X_fH_r^{-1}X_f^\mathsf{T}e_f^+=\frac{1}{n}K_{ff}e_f^+. e f − − e f + = X f ( θ r ⋆ − θ ⋆ ) = n 1 X f H r − 1 X f T e f + = n 1 K f f e f + .
Rearranging gives e f − = ( I m + 1 n K f f ) e f + e_f^-=(I_m+\frac{1}{n}K_{ff})e_f^+ e f − = ( I m + n 1 K f f ) e f + . Since K f f K_{ff} K f f is positive semidefinite, I m + 1 n K f f ⪰ I m I_m+\frac{1}{n}K_{ff}\succeq I_m I m + n 1 K f f ⪰ I m is invertible, yielding the inverse direction e f + = ( I m + 1 n K f f ) − 1 e f − e_f^+=(I_m+\frac{1}{n}K_{ff})^{-1}e_f^- e f + = ( I m + n 1 K f f ) − 1 e f − and the equivalence e f + = 0 ⟺ e f − = 0 e_f^+\!=0\Longleftrightarrow e_f^-\!=0 e f + = 0 ⟺ e f − = 0 . □ \square □
Proof of Theorem 1
Left-multiplying the parameter shift (24) (established in the proof of Lemma 1) by X a X_a X a and using the definition K a f = X a H r − 1 X f ⊤ K_{af}=X_aH_r^{-1}X_f^\top K a f = X a H r − 1 X f ⊤ yields (9):
X a θ r ⋆ − X a θ ⋆ = 1 n K a f e f + . (25) X_a\theta_r^\star-X_a\theta^\star=\frac{1}{n}K_{af}e_f^+. \tag*{(25)} X a θ r ⋆ − X a θ ⋆ = n 1 K a f e f + . (25)
The equivalent retrained-residual form (10) follows by substituting Lemma 1 into (9).
Proof of Theorem 2
In the linear squared-loss setting,
R ( θ ) = 1 2 n ∥ y r − X r θ ∥ 2 2 + λ 2 ∥ θ ∥ 2 2 , F ( θ ) = 1 2 n ∥ y f − X f θ ∥ 2 2 . \mathcal{R}(\theta)=\frac{1}{2n}\lVert y_r-X_r\theta\rVert_2^2+\frac{\lambda}{2}\lVert\theta\rVert_2^2,\qquad\mathcal{F}(\theta)=\frac{1}{2n}\lVert y_f-X_f\theta\rVert_2^2. R ( θ ) = 2 n 1 ∥ y r − X r θ ∥ 2 2 + 2 λ ∥ θ ∥ 2 2 , F ( θ ) = 2 n 1 ∥ y f − X f θ ∥ 2 2 .
Hence
∇ R ( θ ) = ( 1 n X r ⊤ X r + λ I ) θ − 1 n X r ⊤ y r = H r θ − b r , ∇ F ( θ ) = 1 n X f ⊤ ( X f θ − y f ) , \nabla\mathcal{R}(\theta)=\left(\frac{1}{n}X_r^\top X_r+\lambda I\right)\theta-\frac{1}{n}X_r^\top y_r=H_r\theta-b_r,\qquad\nabla\mathcal{F}(\theta)=\frac{1}{n}X_f^\top(X_f\theta-y_f), ∇ R ( θ ) = ( n 1 X r ⊤ X r + λ I ) θ − n 1 X r ⊤ y r = H r θ − b r , ∇ F ( θ ) = n 1 X f ⊤ ( X f θ − y f ) ,
where b r : = 1 n X r ⊤ y r b_r:=\frac{1}{n}X_r^\top y_r b r := n 1 X r ⊤ y r . The first-order condition for θ ρ G D i f f \theta_\rho^{\mathrm{GDiff}} θ ρ GDiff is
∇ R ( θ ρ G D i f f ) − ρ ∇ F ( θ ρ G D i f f ) = 0. \nabla\mathcal{R}(\theta_\rho^{\mathrm{GDiff}})-\rho\nabla\mathcal{F}(\theta_\rho^{\mathrm{GDiff}})=0. ∇ R ( θ ρ GDiff ) − ρ ∇ F ( θ ρ GDiff ) = 0.
Substituting the gradients and collecting the terms involving θ ρ G D i f f \theta_\rho^{\mathrm{GDiff}} θ ρ GDiff gives
( H r − ρ n X f ⊤ X f ) θ ρ G D i f f = b r − ρ n X f ⊤ y f , \left(H_r-\frac{\rho}{n}X_f^\top X_f\right)\theta_\rho^{\mathrm{GDiff}}=b_r-\frac{\rho}{n}X_f^\top y_f, ( H r − n ρ X f ⊤ X f ) θ ρ GDiff = b r − n ρ X f ⊤ y f ,
or, writing
M ρ : = H r − ρ n X f ⊤ X f , M_\rho:=H_r-\frac{\rho}{n}X_f^\top X_f, M ρ := H r − n ρ X f ⊤ X f ,
M ρ θ ρ G D i f f = b r − ρ n X f ⊤ y f . M_\rho\theta_\rho^{\mathrm{GDiff}}=b_r-\frac{\rho}{n}X_f^\top y_f. M ρ θ ρ GDiff = b r − n ρ X f ⊤ y f .
The retrained optimum satisfies ∇ R ( θ r ⋆ ) = 0 \nabla\mathcal{R}(\theta_r^\star)=0 ∇ R ( θ r ⋆ ) = 0 , so H r θ r ⋆ = b r H_r\theta_r^\star=b_r H r θ r ⋆ = b r . Therefore
M ρ θ r ⋆ = ( H r − ρ n X f ⊤ X f ) θ r ⋆ = b r − ρ n X f ⊤ X f θ r ⋆ . M_\rho\theta_r^\star=\left(H_r-\frac{\rho}{n}X_f^\top X_f\right)\theta_r^\star=b_r-\frac{\rho}{n}X_f^\top X_f\theta_r^\star. M ρ θ r ⋆ = ( H r − n ρ X f ⊤ X f ) θ r ⋆ = b r − n ρ X f ⊤ X f θ r ⋆ .
Subtracting this identity from the equation for θ ρ G D i f f \theta_\rho^{\mathrm{GDiff}} θ ρ GDiff yields
M ρ ( θ ρ G D i f f − θ r ⋆ ) = ( b r − ρ n X f ⊤ y f ) − ( b r − ρ n X f ⊤ X f θ r ⋆ ) = ρ n X f ⊤ ( X f θ r ⋆ − y f ) . M_\rho(\theta_\rho^{\mathrm{GDiff}}-\theta_r^\star)=\left(b_r-\frac{\rho}{n}X_f^\top y_f\right)-\left(b_r-\frac{\rho}{n}X_f^\top X_f\theta_r^\star\right)=\frac{\rho}{n}X_f^\top(X_f\theta_r^\star-y_f). M ρ ( θ ρ GDiff − θ r ⋆ ) = ( b r − n ρ X f ⊤ y f ) − ( b r − n ρ X f ⊤ X f θ r ⋆ ) = n ρ X f ⊤ ( X f θ r ⋆ − y f ) .
Using e f − = X f θ r ⋆ − y f e_f^-=X_f\theta_r^\star-y_f e f − = X f θ r ⋆ − y f , this becomes
M ρ ( θ ρ G D i f f − θ r ⋆ ) = ρ n X f ⊤ e f − . M_\rho(\theta_\rho^{\mathrm{GDiff}}-\theta_r^\star)=\frac{\rho}{n}X_f^\top e_f^-. M ρ ( θ ρ GDiff − θ r ⋆ ) = n ρ X f ⊤ e f − .
The assumption M ρ ≻ 0 M_\rho\succ0 M ρ ≻ 0 ensures that we can invert this system, so
θ ρ G D i f f − θ r ⋆ = ρ n M ρ − 1 X f ⊤ e f − . \theta_\rho^{\mathrm{GDiff}}-\theta_r^\star=\frac{\rho}{n}M_\rho^{-1}X_f^\top e_f^-. θ ρ GDiff − θ r ⋆ = n ρ M ρ − 1 X f ⊤ e f − .
Applying Lemma 2 with A = H r A=H_r A = H r , U = X f ⊤ U=X_f^\top U = X f ⊤ , C = ρ n I m C=\frac{\rho}{n}I_m C = n ρ I m , and V = X f V=X_f V = X f ,
M ρ − 1 X f ⊤ = H r − 1 X f ⊤ ( I m − ρ n X f H r − 1 X f ⊤ ) − 1 = H r − 1 X f ⊤ ( I m − ρ n K f f ) − 1 . M_\rho^{-1}X_f^\top=H_r^{-1}X_f^\top\left(I_m-\frac{\rho}{n}X_fH_r^{-1}X_f^\top\right)^{-1}=H_r^{-1}X_f^\top\left(I_m-\frac{\rho}{n}K_{ff}\right)^{-1}. M ρ − 1 X f ⊤ = H r − 1 X f ⊤ ( I m − n ρ X f H r − 1 X f ⊤ ) − 1 = H r − 1 X f ⊤ ( I m − n ρ K f f ) − 1 .
Substituting this expression into the parameter gap gives
θ ρ G D i f f − θ r ⋆ = ρ n H r − 1 X f ⊤ ( I m − ρ n K f f ) − 1 e f − . \theta_\rho^{\mathrm{GDiff}}-\theta_r^\star=\frac{\rho}{n}H_r^{-1}X_f^\top\left(I_m-\frac{\rho}{n}K_{ff}\right)^{-1}e_f^-. θ ρ GDiff − θ r ⋆ = n ρ H r − 1 X f ⊤ ( I m − n ρ K f f ) − 1 e f − .
Finally, left-multiplying by X a X_a X a and using K a f = X a H r − 1 X f ⊤ K_{af}=X_aH_r^{-1}X_f^\top K a f = X a H r − 1 X f ⊤ gives
X a ( θ ρ G D i f f − θ r ⋆ ) = ρ n K a f ( I m − ρ n K f f ) − 1 e f − , X_a(\theta_\rho^{\mathrm{GDiff}}-\theta_r^\star)=\frac{\rho}{n}K_{af}\left(I_m-\frac{\rho}{n}K_{ff}\right)^{-1}e_f^-, X a ( θ ρ GDiff − θ r ⋆ ) = n ρ K a f ( I m − n ρ K f f ) − 1 e f − ,
which is (12). The equivalent full-data-residual form (11) follows by substituting Lemma 1:
X a ( θ ρ G D i f f − θ r ⋆ ) = ρ n K a f ( I m − ρ n K f f ) − 1 ( I m + 1 n K f f ) e f + . X_a(\theta_\rho^{\mathrm{GDiff}}-\theta_r^\star)=\frac{\rho}{n}K_{af}\left(I_m-\frac{\rho}{n}K_{ff}\right)^{-1}\left(I_m+\frac{1}{n}K_{ff}\right)e_f^+. X a ( θ ρ GDiff − θ r ⋆ ) = n ρ K a f ( I m − n ρ K f f ) − 1 ( I m + n 1 K f f ) e f + .
Proof of Corollary 1
The norm bound (13) follows from (12) and submultiplicativity of ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 , using ∥ ( I − ρ n K f f ) − 1 ∥ 2 = ( 1 − ρ n λ max ( K f f ) ) − 1 \left\lVert\left(I-\frac{\rho}{n}K_{ff}\right)^{-1}\right\rVert_2=\left(1-\frac{\rho}{n}\lambda_{\max}(K_{ff})\right)^{-1} ( I − n ρ K f f ) − 1 2 = ( 1 − n ρ λ m a x ( K f f ) ) − 1 when K f f ⪰ 0 K_{ff}\succeq0 K f f ⪰ 0 and ρ n λ max ( K f f ) < 1 \frac{\rho}{n}\lambda_{\max}(K_{ff})<1 n ρ λ m a x ( K f f ) < 1 . The equivalent bound (14) follows analogously from (11) together with ∥ I m + 1 n K f f ∥ 2 = 1 + 1 n λ max ( K f f ) \left\lVert I_m+\frac{1}{n}K_{ff}\right\rVert_2=1+\frac{1}{n}\lambda_{\max}(K_{ff}) I m + n 1 K f f 2 = 1 + n 1 λ m a x ( K f f ) since K f f ⪰ 0 K_{ff}\succeq0 K f f ⪰ 0 . The stability equivalence is obtained by conjugating H r − ρ n X f T X f ≻ 0 H_r-\frac{\rho}{n}X_f^{\mathsf{T}}X_f\succ0 H r − n ρ X f T X f ≻ 0 by H r − 1 / 2 H_r^{-1/2} H r − 1/2 and noting that the nonzero eigenvalues of H r − 1 / 2 X f T X f H r − 1 / 2 H_r^{-1/2}X_f^{\mathsf{T}}X_fH_r^{-1/2} H r − 1/2 X f T X f H r − 1/2 coincide with those of K f f K_{ff} K f f . □ \square □
Proof of Proposition 1 (per-method)
All five non-trivial rows are obtained by the same recipe used for GradDiff in Appendix A.4: (i) write the method’s regularized linear objective and its first-order stationarity condition, (ii) subtract the corresponding stationarity equation for θ r ⋆ \theta_r^\star θ r ⋆ to obtain a linear system in θ − θ r ⋆ \theta-\theta_r^\star θ − θ r ⋆ , and (iii) apply the push-through identity (Lemma 2) to express the result in the leverage-matrix form of Table 1 . We give each row in turn.
GradDiff (recap). Objective R ( θ ) − ρ F ( θ ) \mathcal{R}(\theta)-\rho\mathcal{F}(\theta) R ( θ ) − ρ F ( θ ) , stationarity ( H r − ρ n X f T X f ) θ = b r − ρ n X f T y f \left(H_r-\frac{\rho}{n}X_f^{\mathsf{T}}X_f\right)\theta=b_r-\frac{\rho}{n}X_f^{\mathsf{T}}y_f ( H r − n ρ X f T X f ) θ = b r − n ρ X f T y f where b r : = 1 n X r T y r b_r:=\frac{1}{n}X_r^{\mathsf{T}}y_r b r := n 1 X r T y r and H r θ r ⋆ = b r H_r\theta_r^\star=b_r H r θ r ⋆ = b r . Subtracting and applying Lemma 2 with A = H r A=H_r A = H r , U = X f T U=X_f^{\mathsf{T}} U = X f T , C = ρ n I m C=\frac{\rho}{n}I_m C = n ρ I m , V = X f V=X_f V = X f gives the GradDiff row (full derivation in Appendix A.4).
NPO and SimNPO. The (Sim)NPO objectives replace the unweighted forget term − ρ F -\rho\mathcal{F} − ρ F by a per-sample preference-style loss
N P O : R ( θ ) − ρ β ⋅ 1 n ∑ j ∈ D f log σ ( β ( ℓ j ( θ ) − ℓ j ( θ r e f ) ) ) , \mathrm{NPO}:\quad\mathcal{R}(\theta)-\frac{\rho}{\beta}\cdot\frac{1}{n}\sum_{j\in D_f}\log\sigma\left(\beta\left(\ell_j(\theta)-\ell_j(\theta_{\mathrm{ref}})\right)\right), NPO : R ( θ ) − β ρ ⋅ n 1 j ∈ D f ∑ log σ ( β ( ℓ j ( θ ) − ℓ j ( θ ref ) ) ) ,
S i m N P O : R ( θ ) − ρ β ⋅ 1 n ∑ j ∈ D f log σ ( β ℓ ~ j ( θ ) ) , \mathrm{SimNPO}:\quad\mathcal{R}(\theta)-\frac{\rho}{\beta}\cdot\frac{1}{n}\sum_{j\in D_f}\log\sigma\left(\beta\widetilde{\ell}_j(\theta)\right), SimNPO : R ( θ ) − β ρ ⋅ n 1 j ∈ D f ∑ log σ ( β ℓ j ( θ ) ) ,
where ℓ j ( θ ) = 1 2 ( y j − x j T θ ) 2 \ell_j(\theta)=\frac{1}{2}(y_j-x_j^{\mathsf{T}}\theta)^2 ℓ j ( θ ) = 2 1 ( y j − x j T θ ) 2 is the per-sample squared loss and ℓ ~ j \widetilde{\ell}_j ℓ j is its length-normalized variant; θ r e f \theta_{\mathrm{ref}} θ ref is the NPO reference parameter. Differentiating,
∇ θ [ − ρ β log σ ( β t j ( θ ) ) ] = − ρ σ ( − β t j ( θ ) ) ∇ θ t j ( θ ) , \nabla_\theta\left[-\frac{\rho}{\beta}\log\sigma\left(\beta t_j(\theta)\right)\right]=-\rho\sigma\left(-\beta t_j(\theta)\right)\nabla_\theta t_j(\theta), ∇ θ [ − β ρ log σ ( β t j ( θ ) ) ] = − ρ σ ( − β t j ( θ ) ) ∇ θ t j ( θ ) ,
which produces a per-sample weight w j ( θ ) = σ ( z j ( θ ) ) w_j(\theta)=\sigma(z_j(\theta)) w j ( θ ) = σ ( z j ( θ )) on the standard squared-loss gradient (the inverse-temperature factor β \beta β from differentiating the inner log σ ( β ⋅ ) \log\sigma(\beta\cdot) log σ ( β ⋅ ) exactly cancels with the ρ / β \rho/\beta ρ / β prefactor of the objective, so the effective forget coefficient in the linear stationarity equation below is ρ \rho ρ and the diagonal weights are β \beta β -free, matching Section 4.2; under any alternative prefactor convention β \beta β would simply be absorbed into ρ \rho ρ ). Freezing the weights at θ r ⋆ \theta_r^\star θ r ⋆ yields the linear stationarity condition
( H r − ρ n X f T W − X f ) θ = b r − ρ n X f T W − y f , \left(H_r-\frac{\rho}{n}X_f^{\mathsf{T}}W^-X_f\right)\theta=b_r-\frac{\rho}{n}X_f^{\mathsf{T}}W^-y_f, ( H r − n ρ X f T W − X f ) θ = b r − n ρ X f T W − y f ,
with W − = W N P O − W^-=W^-_{\mathrm{NPO}} W − = W NPO − or W S i m N P O − W^-_{\mathrm{SimNPO}} W SimNPO − . Subtracting the θ r ⋆ \theta_r^\star θ r ⋆ equation H r θ r ⋆ = b r H_r\theta_r^\star=b_r H r θ r ⋆ = b r gives ( H r − ρ n X f T W − X f ) ( θ − θ r ⋆ ) = ρ n X f T W − e f \left(H_r-\frac{\rho}{n}X_f^{\mathsf{T}}W^-X_f\right)(\theta-\theta_r^\star)=\frac{\rho}{n}X_f^{\mathsf{T}}W^-e_f ( H r − n ρ X f T W − X f ) ( θ − θ r ⋆ ) = n ρ X f T W − e f . Applying Lemma 2 with C = ρ n W − C=\frac{\rho}{n}W^- C = n ρ W − and left-multiplying by X a X_a X a yields the (Sim)NPO row.
Pure GA. The objective drops R \mathcal{R} R and keeps only ridge regularization and forget ascent: − ρ F ( θ ) + λ 2 ∥ θ ∥ 2 -\rho\mathcal{F}(\theta)+\frac{\lambda}{2}\lVert\theta\rVert^2 − ρ F ( θ ) + 2 λ ∥ θ ∥ 2 . The first-order condition is ( λ I d − ρ n X f T X f ) θ = − ρ n X f T y f \left(\lambda I_d-\frac{\rho}{n}X_f^{\mathsf{T}}X_f\right)\theta=-\frac{\rho}{n}X_f^{\mathsf{T}}y_f ( λ I d − n ρ X f T X f ) θ = − n ρ X f T y f . Lemma 2 with A = λ I d A=\lambda I_d A = λ I d , U = X f T U=X_f^{\mathsf{T}} U = X f T , C = ρ n I m C=\frac{\rho}{n}I_m C = n ρ I m , V = X f V=X_f V = X f gives θ G A ⋆ = ρ n λ X f T ( I m − ρ n λ K f f ) − 1 y f \theta_{\mathrm{GA}}^\star=\frac{\rho}{n\lambda}X_f^{\mathsf{T}}\left(I_m-\frac{\rho}{n\lambda}K_{ff}\right)^{-1}y_f θ GA ⋆ = nλ ρ X f T ( I m − nλ ρ K f f ) − 1 y f . Subtracting θ r ⋆ \theta_r^\star θ r ⋆ and left-multiplying by X a X_a X a yields the Pure-GA row.
IMU. IMU rescales each forget term by a per-sample influence weight α j \alpha_j α j , giving the objective − ρ ⋅ 1 n ∑ j ∈ D f α j ℓ j ( θ ) + λ 2 ∥ θ ∥ 2 -\rho\cdot\frac{1}{n}\sum_{j\in D_f}\alpha_j\ell_j(\theta)+\frac{\lambda}{2}\lVert\theta\rVert^2 − ρ ⋅ n 1 ∑ j ∈ D f α j ℓ j ( θ ) + 2 λ ∥ θ ∥ 2 (without R \mathcal{R} R , matching the Pure-GA setting). The stationarity condition below is the Pure-GA system with the influence diagonal A I M U A_{\mathrm{IMU}} A IMU inserted: ( λ I d − ρ n X f T A I M U X f ) θ = − ρ n X f T A I M U y f \left(\lambda I_d-\frac{\rho}{n}X_f^{\mathsf{T}}A_{\mathrm{IMU}}X_f\right)\theta=-\frac{\rho}{n}X_f^{\mathsf{T}}A_{\mathrm{IMU}}y_f ( λ I d − n ρ X f T A IMU X f ) θ = − n ρ X f T A IMU y f . The same push-through application with C = ρ n A I M U C=\frac{\rho}{n}A_{\mathrm{IMU}} C = n ρ A IMU yields the IMU row. This is a fixed-weight local approximation; in the experiments the diagonal is instantiated once from θ ⋆ \theta^\star θ ⋆ using the final-layer proxy in Appendix B.
ProjGD. The fixed-projector method applies P r P_r P r to the forget gradient at every step, so the stationarity condition is ∇ R ( θ ) − ρ P r ∇ F ( θ ) = 0 \nabla R(\theta)-\rho P_r\nabla F(\theta)=0 ∇ R ( θ ) − ρ P r ∇ F ( θ ) = 0 . In linear regression this gives ( H r − ρ n P r X f T X f ) θ = b r − ρ n P r X f T y f \left(H_r-\frac{\rho}{n}P_rX_f^{\mathsf T}X_f\right)\theta=b_r-\frac{\rho}{n}P_rX_f^{\mathsf T}y_f ( H r − n ρ P r X f T X f ) θ = b r − n ρ P r X f T y f . Subtracting the θ r ⋆ \theta_r^\star θ r ⋆ equation and applying Lemma 2 with A = H r A=H_r A = H r , U = P r X f T U=P_rX_f^{\mathsf T} U = P r X f T , C = ρ n I m C=\frac{\rho}{n}I_m C = n ρ I m , V = X f V=X_f V = X f yields the ProjGD row, with K a f P K_{af}^{P} K a f P and K f f P K_{ff}^{P} K f f P in place of K a f K_{af} K a f and K f f K_{ff} K f f .
Newton Step. Newton step solves θ ⋆ + H r − 1 ∇ F ( θ ⋆ ) = θ r ⋆ \theta^\star+H_r^{-1}\nabla F(\theta^\star)=\theta_r^\star θ ⋆ + H r − 1 ∇ F ( θ ⋆ ) = θ r ⋆ exactly in linear regression with squared loss, using the same push-through identity as Theorem 1 [39 ] . Hence the gap is zero.
Proof of Proposition 2
Let Δ ρ = θ ρ G D i f f − θ r ⋆ \Delta_\rho=\theta_\rho^{\mathrm{GDiff}}-\theta_r^\star Δ ρ = θ ρ GDiff − θ r ⋆ . The first-order condition for θ ρ G D i f f \theta_\rho^{\mathrm{GDiff}} θ ρ GDiff is ∇ R ( θ ρ G D i f f ) − ρ ∇ F ( θ ρ G D i f f ) = 0 \nabla R(\theta_\rho^{\mathrm{GDiff}})-\rho\nabla F(\theta_\rho^{\mathrm{GDiff}})=0 ∇ R ( θ ρ GDiff ) − ρ ∇ F ( θ ρ GDiff ) = 0 . Using ∇ R ( θ r ⋆ ) = 0 \nabla R(\theta_r^\star)=0 ∇ R ( θ r ⋆ ) = 0 and replacing the local gradients by their first-order GGN/Taylor model gives
H r Δ ρ − ρ g f − ρ H f , G Δ ρ ≈ 0 , H_r\Delta_\rho-\rho g_f-\rho H_{f,\mathrm{G}}\Delta_\rho\approx0, H r Δ ρ − ρ g f − ρ H f , G Δ ρ ≈ 0 ,
where g f = ∇ F ( θ r ⋆ ) = 1 n ( J f ) T δ f g_f=\nabla F(\theta_r^\star)=\frac{1}{n}(J_f)^{\mathsf T}\delta_f g f = ∇ F ( θ r ⋆ ) = n 1 ( J f ) T δ f and H f , G = 1 n ( J f ) T B f J f H_{f,\mathrm{G}}=\frac{1}{n}(J_f)^{\mathsf T}B_fJ_f H f , G = n 1 ( J f ) T B f J f . Solving this local linear system gives
Δ ^ ρ = ρ ( H r − ρ H f , G ) − 1 g f . \widehat{\Delta}_\rho=\rho\left(H_r-\rho H_{f,\mathrm{G}}\right)^{-1}g_f. Δ ρ = ρ ( H r − ρ H f , G ) − 1 g f .
Applying the same push-through identity with A = H r A=H_r A = H r , U = ( J f ) T U=(J_f)^{\mathsf T} U = ( J f ) T , C = ρ n B f C=\frac{\rho}{n}B_f C = n ρ B f , and V = J f V=J_f V = J f yields
Δ ^ ρ = ρ n ( H r ) − 1 ( J f ) T ( I m c − ρ n B f K f f N L ) − 1 δ f . \widehat{\Delta}_\rho=\frac{\rho}{n}(H_r)^{-1}(J_f)^{\mathsf T}\left(I_{mc}-\frac{\rho}{n}B_fK_{ff}^{\mathrm{NL}}\right)^{-1}\delta_f. Δ ρ = n ρ ( H r ) − 1 ( J f ) T ( I m c − n ρ B f K f f NL ) − 1 δ f .
Left-multiplying by J a J_a J a gives the displayed output predictor in (22).
A local nonlinear approximation-error bound
This appendix records a representative local bound for the approximation-error discussion in Section 4.3. Let Δ ρ = θ ρ G D i f f − θ r ⋆ \Delta_\rho=\theta_\rho^{\mathrm{GDiff}}-\theta_r^\star Δ ρ = θ ρ GDiff − θ r ⋆ be any stationary GradDiff displacement in a neighborhood of θ r ⋆ \theta_r^\star θ r ⋆ , and assume ∥ ( M ρ − ) − 1 ∥ 2 ≤ γ \left\|(M_\rho^-)^{-1}\right\|_2\le\gamma ( M ρ − ) − 1 2 ≤ γ . Suppose that for all displacements Δ \Delta Δ in this neighborhood,
∥ ∇ R ( θ r ⋆ + Δ ) − H r − Δ ∥ 2 ≤ ε r ∥ Δ ∥ 2 + L r 2 ∥ Δ ∥ 2 2 , \left\|\nabla R(\theta_r^\star+\Delta)-H_r^-\Delta\right\|_2\le\varepsilon_r\|\Delta\|_2+\frac{L_r}{2}\|\Delta\|_2^2, ∇ R ( θ r ⋆ + Δ ) − H r − Δ 2 ≤ ε r ∥Δ ∥ 2 + 2 L r ∥Δ ∥ 2 2 ,
∥ ∇ F ( θ r ⋆ + Δ ) − ∇ F ( θ r ⋆ ) − H f , G Δ ∥ 2 ≤ ε f ∥ Δ ∥ 2 + L f 2 ∥ Δ ∥ 2 2 , \left\|\nabla F(\theta_r^\star+\Delta)-\nabla F(\theta_r^\star)-H_{f,\mathrm{G}}\Delta\right\|_2\le\varepsilon_f\|\Delta\|_2+\frac{L_f}{2}\|\Delta\|_2^2, ∥ ∇ F ( θ r ⋆ + Δ ) − ∇ F ( θ r ⋆ ) − H f , G Δ ∥ 2 ≤ ε f ∥Δ ∥ 2 + 2 L f ∥Δ ∥ 2 2 ,
and that the evaluation map satisfies
∥ f ( θ r ⋆ + Δ , A ) − f ( θ r ⋆ , A ) − J a Δ ∥ 2 ≤ L a 2 ∥ Δ ∥ 2 2 . \left\|f(\theta_r^\star+\Delta,A)-f(\theta_r^\star,A)-J_a\Delta\right\|_2\le\frac{L_a}{2}\|\Delta\|_2^2. ∥ f ( θ r ⋆ + Δ , A ) − f ( θ r ⋆ , A ) − J a Δ ∥ 2 ≤ 2 L a ∥Δ ∥ 2 2 .
Then the output-space predictor Δ f ^ A : = J a Δ ^ ρ \widehat{\Delta f}_A:=J_a\widehat{\Delta}_\rho Δ f A := J a Δ ρ obeys
∥ f ( θ ρ G D i f f , A ) − f ( θ r ⋆ , A ) − Δ f ^ A ∥ 2 ≤ ∥ J a ∥ 2 γ ( ( ε r + ρ ε f ) ∥ Δ ρ ∥ 2 + L r + ρ L f 2 ∥ Δ ρ ∥ 2 2 ) + L a 2 ∥ Δ ρ ∥ 2 2 . \left\|f(\theta_\rho^{\mathrm{GDiff}},A)-f(\theta_r^\star,A)-\widehat{\Delta f}_A\right\|_2\le\|J_a\|_2\gamma\left((\varepsilon_r+\rho\varepsilon_f)\|\Delta_\rho\|_2+\frac{L_r+\rho L_f}{2}\|\Delta_\rho\|_2^2\right)+\frac{L_a}{2}\|\Delta_\rho\|_2^2. f ( θ ρ GDiff , A ) − f ( θ r ⋆ , A ) − Δ f A 2 ≤ ∥ J a ∥ 2 γ ( ( ε r + ρ ε f ) ∥ Δ ρ ∥ 2 + 2 L r + ρ L f ∥ Δ ρ ∥ 2 2 ) + 2 L a ∥ Δ ρ ∥ 2 2 .
For exact Hessians, ε r = ε f = 0 \varepsilon_r=\varepsilon_f=0 ε r = ε f = 0 under ordinary Lipschitz-Hessian smoothness, by which we mean that ∇ 2 R \nabla^2R ∇ 2 R and ∇ 2 F \nabla^2F ∇ 2 F are L r L_r L r - and L f L_f L f -Lipschitz on the neighborhood, so a standard Taylor argument removes the linear remainder and leaves only the L r , L f L_r,L_f L r , L f second-order terms. With GGN curvatures, ε r \varepsilon_r ε r and ε f \varepsilon_f ε f record the omitted model-Hessian terms.
Let g f − = ∇ F ( θ r ⋆ ) g_f^-=\nabla F(\theta_r^\star) g f − = ∇ F ( θ r ⋆ ) and define the local-model remainders
r r ( Δ ) : = ∇ R ( θ r ⋆ + Δ ) − H r − Δ , r_r(\Delta):=\nabla R(\theta_r^\star+\Delta)-H_r^-\Delta, r r ( Δ ) := ∇ R ( θ r ⋆ + Δ ) − H r − Δ ,
r f ( Δ ) : = ∇ F ( θ r ⋆ + Δ ) − g f − − H f , G Δ . r_f(\Delta):=\nabla F(\theta_r^\star+\Delta)-g_f^--H_{f,\mathrm{G}}\Delta. r f ( Δ ) := ∇ F ( θ r ⋆ + Δ ) − g f − − H f , G Δ.
Since Δ ρ \Delta_\rho Δ ρ is stationary for GradDiff,
0 = ∇ R ( θ r ⋆ + Δ ρ ) − ρ ∇ F ( θ r ⋆ + Δ ρ ) = M ρ − Δ ρ − ρ g f − + r r ( Δ ρ ) − ρ r f ( Δ ρ ) . \begin{aligned}
0&=\nabla R(\theta_r^\star+\Delta_\rho)-\rho\nabla F(\theta_r^\star+\Delta_\rho)\\
&=M_\rho^-\Delta_\rho-\rho g_f^-+r_r(\Delta_\rho)-\rho r_f(\Delta_\rho).
\end{aligned} 0 = ∇ R ( θ r ⋆ + Δ ρ ) − ρ ∇ F ( θ r ⋆ + Δ ρ ) = M ρ − Δ ρ − ρ g f − + r r ( Δ ρ ) − ρ r f ( Δ ρ ) .
The predictor satisfies M ρ − Δ ^ ρ = ρ g f − M_\rho^-\widehat{\Delta}_\rho=\rho g_f^- M ρ − Δ ρ = ρ g f − . Subtracting the two equations gives
Δ ρ − Δ ^ ρ = ( M ρ − ) − 1 ( − r r ( Δ ρ ) + ρ r f ( Δ ρ ) ) . \Delta_\rho-\widehat{\Delta}_\rho=(M_\rho^-)^{-1}\left(-r_r(\Delta_\rho)+\rho r_f(\Delta_\rho)\right). Δ ρ − Δ ρ = ( M ρ − ) − 1 ( − r r ( Δ ρ ) + ρ r f ( Δ ρ ) ) .
Using the assumed remainder bounds and ∥ ( M ρ − ) − 1 ∥ 2 ≤ γ \lVert(M_{\rho}^{-})^{-1}\rVert_{2}\le\gamma ∥( M ρ − ) − 1 ∥ 2 ≤ γ ,
∥ Δ ρ − Δ ^ ρ ∥ 2 ≤ γ ( ( ε r + ρ ε f ) ∥ Δ ρ ∥ 2 + L r + ρ L f 2 ∥ Δ ρ ∥ 2 2 ) . \lVert\Delta_{\rho}-\widehat{\Delta}_{\rho}\rVert_{2}\le\gamma\left((\varepsilon_{r}+\rho\varepsilon_{f})\lVert\Delta_{\rho}\rVert_{2}+\frac{L_{r}+\rho L_{f}}{2}\lVert\Delta_{\rho}\rVert_{2}^{2}\right). ∥ Δ ρ − Δ ρ ∥ 2 ≤ γ ( ( ε r + ρ ε f ) ∥ Δ ρ ∥ 2 + 2 L r + ρ L f ∥ Δ ρ ∥ 2 2 ) .
Finally,
∥ f ( θ ρ G D i f f , A ) − f ( θ r ⋆ , A ) − J a Δ ^ ρ ∥ 2 ≤ ∥ J a ∥ 2 ∥ Δ ρ − Δ ^ ρ ∥ 2 + ∥ f ( θ r ⋆ + Δ ρ , A ) − f ( θ r ⋆ , A ) − J a Δ ρ ∥ 2 , \begin{aligned}
\lVert f(\theta_{\rho}^{\mathrm{GDiff}},A)-f(\theta_{r}^{\star},A)-J_{a}\widehat{\Delta}_{\rho}\rVert_{2}
&\le\lVert J_{a}\rVert_{2}\lVert\Delta_{\rho}-\widehat{\Delta}_{\rho}\rVert_{2}\\
&\quad+\lVert f(\theta_{r}^{\star}+\Delta_{\rho},A)-f(\theta_{r}^{\star},A)-J_{a}\Delta_{\rho}\rVert_{2},
\end{aligned} ∥ f ( θ ρ GDiff , A ) − f ( θ r ⋆ , A ) − J a Δ ρ ∥ 2 ≤ ∥ J a ∥ 2 ∥ Δ ρ − Δ ρ ∥ 2 + ∥ f ( θ r ⋆ + Δ ρ , A ) − f ( θ r ⋆ , A ) − J a Δ ρ ∥ 2 ,
and the evaluation-map remainder bound gives (25).
Local nonlinear forms for the other methods
The local nonlinear rows are obtained from Table 1 by replacing linear features with Jacobians and inserting the output-Hessian/GGN factor B ~ f \widetilde{B}_{f} B f . For NPO and SimNPO, weights are frozen at θ r ⋆ \theta_{r}^{\star} θ r ⋆ and their derivatives are omitted. Scalar per-sample weights are lifted to output blocks as W ~ = W ⊗ I c \widetilde{W}=W\otimes I_{c} W = W ⊗ I c and A ~ = A ⊗ I c \widetilde{A}=A\otimes I_{c} A = A ⊗ I c . For fixed-projector ProjGD and the Euclidean methods, define
K a f N L , P : = J a − ( H r − ) − 1 P r ( J f − ) T , K f f N L , P : = J f − ( H r − ) − 1 P r ( J f − ) T , K_{af}^{\mathrm{NL},P}:=J_{a}^{-}(H_{r}^{-})^{-1}P_{r}(J_{f}^{-})^{\mathsf{T}},\qquad
K_{ff}^{\mathrm{NL},P}:=J_{f}^{-}(H_{r}^{-})^{-1}P_{r}(J_{f}^{-})^{\mathsf{T}}, K a f NL , P := J a − ( H r − ) − 1 P r ( J f − ) T , K f f NL , P := J f − ( H r − ) − 1 P r ( J f − ) T ,
G a f N L : = J a − ( J f − ) T , G f f N L : = J f − ( J f − ) T . G_{af}^{\mathrm{NL}}:=J_{a}^{-}(J_{f}^{-})^{\mathsf{T}},\qquad
G_{ff}^{\mathrm{NL}}:=J_{f}^{-}(J_{f}^{-})^{\mathsf{T}}. G a f NL := J a − ( J f − ) T , G f f NL := J f − ( J f − ) T .
For the Euclidean rows, let
η f : = δ f − − B f − J f − θ r ⋆ . \eta_{f}:=\delta_{f}^{-}-B_{f}^{-}J_{f}^{-}\theta_{r}^{\star}. η f := δ f − − B f − J f − θ r ⋆ .
The shifted residual η f \eta_{f} η f appears in the Pure GA and IMU rows because their local linearization replaces the regression target y f y_{f} y f of Table 1 by its linearized analog δ f − − B f − J f − θ r ⋆ \delta_{f}^{-}-B_{f}^{-}J_{f}^{-}\theta_{r}^{\star} δ f − − B f − J f − θ r ⋆ ; this is the per-sample object that plays the role of − y f -y_{f} − y f when the loss is taken to second order in the model output rather than in the parameter directly. Whenever the displayed inverse exists, the leading-order local output gaps are summarized in Table 4 . All quantities are evaluated at θ r ⋆ \theta_{r}^{\star} θ r ⋆ . The Pure GA and IMU rows are written in Euclidean-Gram form; because they do not use the retrained-Hessian K K K geometry, the K K K -channel pointwise diagnostic in Appendix D is not the predictor for those rows.
Table 4. Local nonlinear counterparts of Table 1; notation as defined in Section 4.3. NPO/SimNPO use weights frozen at θ r ⋆ \theta_{r}^{\star} θ r ⋆ , omitting their derivatives. The Newton Step row omits higher-order and GGN-mismatch residuals.
Experimental Setup Details
This appendix gives the full reproducibility detail for the experiments of Section 5: data, model, hardware, software, hyperparameters, and per-experiment protocols.
Hardware and software. All MNIST/MLP experiments run in float64 on a single NVIDIA H100 NVL 96GB GPU within a Linux + CUDA ≥ 11.8 \ge11.8 ≥ 11.8 environment (PyTorch 2). The figures below were measured by re-running one seed of every experiment in its final configuration on an otherwise idle GPU with our implementation and scaling by the number of seeds. A single iterative MNIST unlearning run (full-batch Adam, 500 steps) takes a few seconds, and the Newton step is a single closed-form update. One (forget set, seed) cell of Table 2 , comprising the θ r ⋆ \theta_{r}^{\star} θ r ⋆ solve, the leverage geometry, and all five unlearners, takes about one minute, so the regime comparison (330 cells, 100 × 3 random and 10 × 3 class) totals ∼5 GPU-hours. The matched-CE experiments of Figure 1 and Table 5 (10 seeds, ρ \rho ρ bisection for six methods) take ∼0.2 GPU-hours. The per-example validations of Appendix § 12 take about 4 (MNIST) and 2 (CIFAR-10) minutes per seed. The fixed-point validation of Table 3 takes ∼2.9 GPU-hours on MNIST and ∼1.9 on CIFAR-10 for 10 seeds, dominated by the polishing of θ r ∗ \theta_r^* θ r ∗ and the dense fp64 curvature solves. The TOFU experiments train no base model. The oracle-proxy analysis of Appendix § 11 .§ 1 consumes the saved per-checkpoint per-example evaluations of one trajectory of the public open-unlearning recipe. The seed study of Appendix § 11 .§ 2 uses fifteen ten-epoch unlearning trajectories of that recipe with per-epoch evaluation (6–49 minutes each, 6.9 GPU-hours in total), a one-time pass on Llama-3.2-1B-Instruct that caches the LoRA-space diagonal GGN over the 4,000 TOFU training sequences (∼7 minutes), and K f f / K r f K_{ff}/K_{rf} K f f / K r f scoring that takes under two minutes per split. We do not separately enumerate compute used by preliminary or discarded experiments. We plan to open-source our code at https://github.com/xingjianz/loss-is-not-behavior; the code is not yet publicly available. Asset versions and licenses. MNIST [20 ] (CC BY-SA 3.0), CIFAR-10 [19 ] together with the pretrained ResNet-18 released with the NeurIPS 2023 Machine Unlearning Challenge starting kit (https://github.com/unlearning-challenge/starting-kit), the TOFU dataset [27 ] (MIT), the open-unlearning library [6 ] (MIT), Llama-3.2-1B-Instruct [14 ] (Llama 3.2 Community License), PyTorch (BSD-3-Clause), and Hugging Face Transformers/PEFT (Apache-2.0); all are used within their licensed terms. Random seeds are explicit per run; all stochastic data subsampling, weight initialization, and SGD/Adam mini-batching is reproducible from the seed.
Table 5. Random-forget aggregate results on MNIST/MLP (∣ D f ∣ = 200 |\mathcal{D}_f| = 200 ∣ D f ∣ = 200 , 10 seeds). The tunable gradient methods are matched to the retrained forget cross-entropy; Newton Step has no ρ \rho ρ knob and is reported as its single second-order step. Setup in Appendix § 10 ; class-forget version in Figure 1 .
Data: MNIST. We use the standard MNIST dataset of 28 × 28 28 \times28 28 × 28 grayscale digits (60 000 training, 10 000 test), accessed locally as a NumPy archive. Per seed, we draw a random training subsample of n t r = 4000 n_{\mathrm{tr}} = 4000 n tr = 4000 images and a random test subsample of n t e = 2000 n_{\mathrm{te}} = 2000 n te = 2000 images, normalize pixels to [ 0 , 1 ] [0, 1] [ 0 , 1 ] , and standardize features per-coordinate using the training-subset mean and standard deviation (an additive 10 − 6 10^{-6} 1 0 − 6 floor on the std). Labels are one-hot encoded for squared/CE losses.
Model and base training. The MLP has architecture 784 → 24 → 10 784 \to24 \to10 784 → 24 → 10 with tanh hidden activation, a softmax output head, and biases in both layers (d = 19 , 090 d = 19,090 d = 19 , 090 ). We initialize weights from a zero-mean Gaussian with variance equal to inverses and biases to zero, using the run seed. Base models are trained for 40 epochs with SGD (learning rate 0.1, batch size 256, momentum 0.9), without L-BFGS polishing. The matched-CE motivation experiments use λ = 10 − 3 \lambda= 10^{-3} λ = 1 0 − 3 ; the regime comparison and fixed-point validation use λ = 10 − 2 \lambda= 10^{-2} λ = 1 0 − 2 . The initial models are trained for 20 epochs from θ ∗ \theta^* θ ∗ on the retain set, with learning rate 0.05, batch size 64, momentum 0.9, and mini-batch seed seed + 1. The regime comparison replaces this reference and the fixed-point validation polishes it as specified below.
Unlearners and optimization. The motivation and regime experiments start unlearning from θ ∗ \theta^* θ ∗ ; the fixed-point experiments start from the polished θ r ∗ \theta_r^* θ r ∗ . The iterative methods optimize full-batch objectives. GradDiff minimizes retain CE minus ρ \rho ρ times forget CE plus ℓ 2 \ell_2 ℓ 2 regularization. NPO uses reference log-probabilities from θ ∗ \theta^* θ ∗ with β = 0.1 \beta= 0.1 β = 0.1 ; SimNPO uses the reference-free loss with β = 2.5 \beta= 2.5 β = 2.5 and margin δ = 0 \delta= 0 δ = 0 . The β \beta β values follow the open-unlearning library defaults. Pure GA drops the retain term and ascends forget CE with only the ℓ 2 \ell_2 ℓ 2 anchor. IMU uses the same Pure-GA objective but multiplies each forget CE by a fixed weight equal to the norm of the final-layer CE gradient at θ ∗ \theta^* θ ∗ , normalized to mean one; this is the experimental proxy for the ideal influence norm in Table 1 . ProjGD applies the GradDiff direction after projecting out the span of 64 retain-anchor CE gradients in the Euclidean metric, with the surrogate subspace recomputed along the trajectory. These choices differ from the closed-form theory rows, which solve stationary fixed-weight or frozen-local surrogate objectives.
The matched-CE protocols use deterministic full-batch SGD-style runners during ρ \rho ρ bisection: 120 steps per evaluation, lr 0.05 for GradDiff/NPO/SimNPO/ProjGD and lr 0.005 for Pure GA/IMU. The four iterative methods in § 5.1 instead use full-batch Adam (β 1 = 0.9 \beta_1 = 0.9 β 1 = 0.9 , β 2 = 0.999 \beta_2 = 0.999 β 2 = 0.999 , ϵ = 10 − 8 \epsilon= 10^{-8} ϵ = 1 0 − 8 , lr δ × 10 − 3 \delta\times10^{-3} δ × 1 0 − 3 ) for 500 steps with λ = 10 − 2 \lambda= 10^{-2} λ = 1 0 − 2 ; no convergence-based stopping is used. Per-method ρ \rho ρ -bracket centers for matched-CE tuning were calibrated once at the start of the project as ρ 0 ∈ { G r a d D i f f : 5 , N P O : 5 , S i m N P O : 30 , P r o j G D : 5 , P u r e G A : 2 , I M U : 0.5 } \rho_0 \in\{\mathrm{GradDiff}: 5, \mathrm{NPO}: 5, \mathrm{SimNPO}: 30, \mathrm{ProjGD}: 5, \mathrm{Pure\ GA}: 2, \mathrm{IMU}: 0.5\} ρ 0 ∈ { GradDiff : 5 , NPO : 5 , SimNPO : 30 , ProjGD : 5 , Pure GA : 2 , IMU : 0.5 } . The Newton step is a single closed-form second-order update from θ ∗ \theta^* θ ∗ using the local GGN curvature; it has no ρ \rho ρ knob.
The MNIST per-coordinate logit RMS is ( 1 / ∣ A ∣ ) ∑ i ∈ A ∥ f ( θ , x i ) − f ( θ ∗ , x i ) ∥ 2 2 / c \sqrt{(1/|A|)\sum_{i\in A}\|f(\theta,x_i)-f(\theta^*,x_i)\|_2^2}/c ( 1/∣ A ∣ ) ∑ i ∈ A ∥ f ( θ , x i ) − f ( θ ∗ , x i ) ∥ 2 2 / c with c = 10 c = 10 c = 10 ; TOFU instead uses per-example sequence-NLL deviation from the retain95 oracle.
Class-forget protocol (Figure 1). The forget set is the entire instance of a single class in the seed’s training subsample (rotated across seeds: seed i → i \to i → class i m o d 10 i \mathop{\mathrm{mod}} 10 i mod 10 , so ∣ D f ∣ |\mathcal{D}_f| ∣ D f ∣ varies between ∼ 350 \sim350 ∼ 350 and ∼ 490 \sim490 ∼ 490 depending on which class lands in that seed’s 4000-sample subset). For each (seed, method) pair we tune ρ \rho ρ by geometric bisection on the bracket [ ρ 0 / 16 , 16 ρ 0 ] [\rho_0/16, 16\rho_0] [ ρ 0 /16 , 16 ρ 0 ] until the achieved forget CE on D f \mathcal{D}_f D f falls within ± 10 % \pm10\% ± 10% of CE ( θ ∗ , D f ) \operatorname{CE}(\theta^*,\mathcal{D}_f) CE ( θ ∗ , D f ) , capped at 12 bisection steps. Because Newton step has no ρ \rho ρ knob, we report it without the matching constraint and flag it with hatched bars in Figure 1 . Figure 1 displays four observables: forget CE, weight distance ∥ θ − θ r ∗ ∥ 2 \|\theta-\theta_r^*\|_2 ∥ θ − θ r ∗ ∥ 2 , KL ( p θ ∗ ∥ p θ ) \operatorname{KL}(p_{\theta^*}\|p_\theta) KL ( p θ ∗ ∥ p θ ) on D f \mathcal{D}_f D f , and top-1 forget accuracy. The full observable set — additionally including top-2 forget accuracy, confidence margin, held-out test accuracy, and a balanced-accuracy MIA based on a logistic attack on per-example loss (anchor batch from the retain set, equal forget/retain balance) — appears in the random-forget aggregate Table 5 . Means and standard deviations are over 10 seeds. The random-forget protocol uses the same recipe but with D f \mathcal{D}_f D f a uniformly random subsample of size 200 from the seed’s training subsample.
Stability-fraction protocol (Section 5.1). For each forget set, we form the reference θ r ∗ \theta_r^* θ r ∗ by 500 full-batch Adam steps on the full-data-normalized R \mathcal{R} R from θ ∗ \theta^* θ ∗ , at learning rate 5 × 10 − 3 5 \times10^{-3} 5 × 1 0 − 3 and λ = 10 − 2 \lambda= 10^{-2} λ = 1 0 − 2 . This fixed-step reference is not polished to a prescribed gradient tolerance. Curvature is estimated at this reference as
H ~ r = 1 n − m ∑ i ∈ D r J i T B i J i + λ I . \widetilde{H}_r = \frac{1}{n-m}\sum_{i\in\mathcal{D}_r} J_i^{\mathsf{T}}B_iJ_i+\lambda I. H r = n − m 1 i ∈ D r ∑ J i T B i J i + λ I .
This retain-normalized GGN is a surrogate for the full-n n n curvature in the theory. The K K K quantities in Table 2 denote estimates J d H ~ r − 1 J f T J_d\widetilde{H}_r^{-1}J_f^{\mathsf{T}} J d H r − 1 J f T : the self-leverage block uses all m m m forget examples, and the cross-leverage block uses 200 retain anchors and all m m m forget examples. We set ρ = 0.3 n / λ max ( K f f ) \rho= 0.3n/\lambda_{\max}(K_{ff}) ρ = 0.3 n / λ m a x ( K f f ) for GradDiff, NPO, SimNPO, and ProjGD as a heuristic motivated by the linear stability rule. Each takes 500 full-batch Adam steps from θ ∗ \theta^* θ ∗ ; Newton Step takes one update. The class regime uses m = 350 m = 350 m = 350 –490 490 490 (mean ≈ 400 \approx400 ≈ 400 ); the random regime uses m = 200 m = 200 m = 200 . Aggregates are over 10 × \times × 3 = 30 class runs and 100 × \times × 3 = 300 random runs. Output deviations are per-coordinate logit RMS on D f \mathcal{D}_f D f and D r \mathcal{D}_r D r . Mean ρ \rho ρ is 1.14 × 10 − 3 1.14 \times10^{-3} 1.14 × 1 0 − 3 (random) and 6.69 × 10 − 4 6.69 \times10^{-4} 6.69 × 1 0 − 4 (class), varying by an order of magnitude across random forget sets. Across all five methods, mean forget CE ranges from 0.329 to 0.359 in the random regime. Class-regime CE is 4.368–4.404 for the four iterative methods and 2.972 for Newton Step.
GGN stability rule. For unprojected GradDiff and frozen-weight (Sim)NPO GGN models formed with the same full-n n n normalization as the retain objective, the softmax-curvature bound ∥ B j ∥ 2 ≤ 1 2 \|B_j\|_2 \le\frac{1}{2} ∥ B j ∥ 2 ≤ 2 1 bounds the effective kernel eigenvalues by ( w max / 2 ) λ max ( K f f N L ) (w_{\max}/2)\lambda_{\max}(K_{ff}^{\mathrm{NL}}) ( w m a x /2 ) λ m a x ( K f f NL ) , where w max w_{\max} w m a x is the largest sample weight. It equals 1 for GradDiff and is at most 1 under the table’s σ \sigma σ convention or 2 under the deployed 2 σ 2\sigma 2 σ convention, equivalent after rescaling ρ \rho ρ . Thus s < 1 s < 1 s < 1 suffices for positive definiteness of these frozen GGN models under the unweighted rule; locality up to s = 2 s = 2 s = 2 remains an empirical observation. Table 2 uses the differently normalized surrogate above, so its s s s budget is not a direct certificate for the implemented objectives. Saturated SimNPO weights barely move the iterate under this rule, motivating the weighted kernel in § 5.2 .
CIFAR-10 over frozen features (Table 3 and Table 10). For CIFAR-10 [19 ] we use the ResNet-18 pretrained on CIFAR-10 that was released with the NeurIPS 2023 Machine Unlearning Challenge starting kit (test accuracy 88.6%) and extract its frozen 512-dimensional penultimate features. The backbone is held fixed and is not the target of unlearning. The unlearning analysis runs on a 2-layer MLP head of architecture 512 → 24 → 10 512 \to24 \to10 512 → 24 → 10 (tanh hidden activation, softmax output, d = 12,562 d = 12{,}562 d = 12 , 562 parameters), so the feature vector plays the role of the input x x x and the MNIST pipeline applies unchanged. Per seed we draw 4000 training and 2000 test examples, standardize the features per coordinate, train the head to obtain θ ⋆ \theta^{\star} θ ⋆ , and form θ r ⋆ \theta_{r}^{\star} θ r ⋆ exactly as for MNIST, with ∣ D f ∣ = 200 |\mathcal{D}_{f}| = 200 ∣ D f ∣ = 200 in the random regime and one full class (∣ D f ∣ ≈ 400 |\mathcal{D}_{f}| \approx400 ∣ D f ∣ ≈ 400 ) in the class regime.
Fixed-point validation protocol (§ 5.2 ). Per dataset, regime, and seed, the reference θ r ⋆ \theta_{r}^{\star} θ r ⋆ is the 20-epoch warm-start retrain above, polished to a stationary point of R \mathcal{R} R under the normalization of (2) with λ = 10 − 2 \lambda= 10^{-2} λ = 1 0 − 2 . The SGD retrain alone is not stationary, so we polish it with multi-restart L-BFGS, which reduces ∥ ∇ R ∥ 2 \lVert\nabla\mathcal{R}\rVert_{2} ∥ ∇ R ∥ 2 below 10 − 7 10^{-7} 1 0 − 7 and brings it to the stationary point in the basin of θ ⋆ \theta^{\star} θ ⋆ . The retain curvature is the exact dense fp64 GGN H r = 1 n ∑ i ∈ D r J i T B i J i + λ I H_{r} = \frac{1}{n}\sum_{i \in\mathcal{D}_{r}} J_{i}^{\mathsf{T}}B_{i}J_{i} + \lambda I H r = n 1 ∑ i ∈ D r J i T B i J i + λ I at θ r ⋆ \theta_{r}^{\star} θ r ⋆ . The stability rule sets ρ = s n / λ max ( K f f ) \rho= s n/\lambda_{\max}(K_{ff}) ρ = s n / λ m a x ( K f f ) with s = 0.1 s = 0.1 s = 0.1 . For SimNPO the method-weighted kernel W 1 / 2 K f f W 1 / 2 W^{1/2}K_{ff}W^{1/2} W 1/2 K f f W 1/2 replaces K f f K_{ff} K f f , because under the uniform rule its saturated class-regime weights (w ˉ ≈ 10 − 4 \bar{w} \approx10^{-4} w ˉ ≈ 1 0 − 4 ) leave the endpoint at θ r ⋆ \theta_{r}^{\star} θ r ⋆ . For GradDiff, NPO, SimNPO, Pure GA, and IMU, stationary endpoints are located by damped Newton continuation from θ r ⋆ \theta_{r}^{\star} θ r ⋆ ; NPO and SimNPO optimize their deployed objectives with parameter-dependent weights. ProjGD is run as projected-gradient dynamics from θ r ⋆ \theta_{r}^{\star} θ r ⋆ (1000 steps, learning rate 0.005, 64 retain anchors). The branch through θ r ⋆ \theta_{r}^{\star} θ r ⋆ is not the only stationary point near θ ⋆ \theta^{\star} θ ⋆ . For MNIST seed 0, a GradDiff run started at θ ⋆ \theta^{\star} θ ⋆ and converged with L-BFGS at the same ρ \rho ρ ends 1.8 (random) and 2.5 (class) from the continuation endpoint in parameter norm, at a stationary point whose forget-logit RMS from θ r ⋆ \theta_{r}^{\star} θ r ⋆ (0.26 and 0.39) the local predictor does not describe (P f ⋆ ≈ 0 P_{f}^{\star} \approx0 P f ⋆ ≈ 0 ), while the same run started at θ r ⋆ \theta_{r}^{\star} θ r ⋆ returns to the continuation endpoint within 5 × 10 − 5 5 \times10^{-5} 5 × 1 0 − 5 . The formulas describe the branch through the specified reference, and a deployed run inherits them only insofar as it converges to that branch. GradDiff, NPO, and SimNPO predictions use their rows of Table 4, with NPO/SimNPO weights frozen at θ r ⋆ \theta_{r}^{\star} θ r ⋆ and their derivatives omitted. ProjGD is compared with the unprojected predictor (22); Pure GA/IMU are evaluated with simplified K K K - and Euclidean–Gram controls that omit the shifted-residual and reference-offset terms of their theoretical rows. Predictions are compared with the actual logit gap on D f \mathcal{D}_{f} D f and on 256 random retain anchors by per-logit Pearson correlation and relative ℓ 2 \ell_{2} ℓ 2 error. The Euclidean control replaces H r − 1 H_{r}^{-1} H r − 1 by λ − 1 I \lambda^{-1}I λ − 1 I in the prediction; for NPO/SimNPO it uses uniform forget weights, so these comparisons change both geometry and weighting. The shuffled-anchor control permutes the anchor rows of the prediction (it gives ∣ P ∣ ≤ 0.20 |P| \le0.20 ∣ P ∣ ≤ 0.20 on every row). The output KL quoted in § 5.2 is the mean per-example KL ( p θ r ⋆ ∥ p θ ) \operatorname{KL}(p_{\theta_{r}^{\star}} \Vert p_{\theta}) KL ( p θ r ⋆ ∥ p θ ) at the endpoint. Aggregates are over 10 seeds per dataset and regime.
TOFU/Llama setup (Appendix § 11 ). All TOFU runs use the open-unlearning full-finetune recipe [6 ] on TOFU [27 ] with Llama-3.2-1B-Instruct [14 ] as the base model (learning rate 10 − 5 10^{-5} 1 0 − 5 , 10 epochs, per-epoch checkpoints, library-default β \beta β for NPO and SimNPO). The released retain95 (retain99) model is the oracle θ r ⋆ \theta_{r}^{\star} θ r ⋆ for forget05 (forget01). Two analyses are reported.
The oracle-NLL forget-channel proxy (Appendix § 11 .§ 1 ) uses one GradDiff forget05 trajectory and the score s N L L ( i ) = avg_loss θ r ⋆ ( x i ) s_{\mathrm{NLL}}(i) = \operatorname{avg\_loss}_{\theta_{r}^{\star}}(x_{i}) s NLL ( i ) = avg_loss θ r ⋆ ( x i ) , the per-token NLL of the gold answer under the retain95 oracle. This score is not an explicit Hessian leverage estimate; it is a scalable oracle-evaluable surrogate for the forget residual term that appears together with K f f K_{ff} K f f in Eq. (22) . We sort all 200 forget05 examples by s N L L s_{\mathrm{NLL}} s NLL , split them into quartiles, and compute the per-bin mean ∣ Δ i ∣ |\Delta_{i}| ∣ Δ i ∣ at every saved checkpoint, where Δ i \Delta_{i} Δ i is the per-example sequence-NLL deviation of the unlearned model from the oracle.
The seed study with θ ⋆ \theta^{\star} θ ⋆ -only scores (Appendix § 11 .§ 2 ) covers { forget01 , forget05 } × { GradDiff , NPO } \{\text{forget01},\text{forget05}\} \times\{\text{GradDiff},\text{NPO}\} { forget01 , forget05 } × { GradDiff , NPO } and forget05 SimNPO with three optimizer seeds each. The three seeds are the trainer seeds 0, 1, 2 of the recipe with no other change, and the seed-0 GradDiff forget05 trajectory is the one used in Appendix § 11 .§ 1 . The scores are computed once at θ ⋆ \theta^{\star} θ ⋆ and never see θ r ⋆ \theta_{r}^{\star} θ r ⋆ or the trajectories. We restrict the parameter space to a LoRA adapter (r = 8 r=8 r = 8 , target modules all-linear, 5.6 × 10 6 5.6 \times10^{6} 5.6 × 1 0 6 trainable parameters) and approximate the curvature by H ^ = diag ( 1 N G G N ^ ) + λ I \widehat{H} = \operatorname{diag}\left(\frac{1}{N}\widehat{\mathrm{GGN}}\right) + \lambda I H = diag ( N 1 GGN ) + λ I with λ = 0.01 \lambda= 0.01 λ = 0.01 , where the diagonal uses one Monte-Carlo label sample per training sequence on D f ∪ D r \mathcal{D}_{f} \cup\mathcal{D}_{r} D f ∪ D r , followed by a Woodbury low-rank correction built from S = 4 S=4 S = 4 MC pseudo-gradients on D f \mathcal{D}_{f} D f . With g i g_{i} g i the LoRA-chart gradient of the answer NLL of example i i i at θ ⋆ \theta^{\star} θ ⋆ , the forget score is g j T H ^ − 1 g j g_{j}^{\mathsf{T}}\widehat{H}^{-1}g_{j} g j T H − 1 g j on the 200 (forget05 or 40 (forget01) forget examples, and the retain score is ∑ j ∈ D f ∣ g i T H ^ − 1 g j ∣ \sum_{j \in\mathcal{D}_f} |g_i^{\mathsf{T}}\widehat{H}^{-1}g_j| ∑ j ∈ D f ∣ g i T H − 1 g j ∣ on the 400 probes of the standard TOFU retain_perturbed split. Each score is compared with the per-example ∣ N L L u − N L L r e t ∣ |\mathrm{NLL}_u - \mathrm{NLL}_{\mathrm{ret}}| ∣ NLL u − NLL ret ∣ at every per-epoch checkpoint except checkpoint 0, by Spearman correlation, AUC, and the Q4/Q1 ratio of mean gaps. Checkpoints are aggregated by the median within a trajectory and then averaged across seeds, with hierarchical-bootstrap confidence intervals (B = 1,000 B=1{,}000 B = 1 , 000 , outer level trajectories, inner level examples) and within-checkpoint permutation nulls (1,000 permutations). Cheap baselines are the NLL at θ ⋆ \theta^\star θ ⋆ (forget) and the gradient norm ∥ g i ∥ 2 \lVert g_i\rVert_2 ∥ g i ∥ 2 or the NLL at θ ⋆ \theta^\star θ ⋆ (retain). Table 8 re-analyzes the same trajectories against the NLL movement from θ ⋆ \theta^\star θ ⋆ with cosine, Pearson, and a fitted scale, and repeats the scoring on a half-frozen LoRA chart (A A A frozen, exact GGN over the B B B coordinates).
Table 6. Per-example forget deviation ∣ Δ i ∣ |\Delta_i| ∣ Δ i ∣ grouped by oracle-NLL proxy quartile along the GradDiff trajectory on TOFU/Llama-3.2-1B-Instruct (forget05, ∣ D f ∣ = 200 |\mathcal{D}_f| = 200 ∣ D f ∣ = 200 , 50 examples per quartile). Q1/Q4 contain the lowest/highest proxy scores. Setup in Appendix § 10 . The rows are successive checkpoints of one trajectory, not independent runs. The step-0 Spearman is close to a tautology, because at step 0 Δ i = N L L θ ⋆ ( x i ) − N L L θ ⋆ ( x i ) \Delta_i = \mathrm{NLL}_{\theta^\star}(x_i) - \mathrm{NLL}_{\theta^\star}(x_i) Δ i = NLL θ ⋆ ( x i ) − NLL θ ⋆ ( x i ) while the proxy is N L L θ ⋆ ( x i ) \mathrm{NLL}_{\theta^\star}(x_i) NLL θ ⋆ ( x i ) and N L L θ ⋆ ( x i ) \mathrm{NLL}_{\theta^\star}(x_i) NLL θ ⋆ ( x i ) is small (mean 0.13 vs. 2.5). Independent seeds and θ ⋆ \theta^\star θ ⋆ -only scores are in Tables 7 and 8.
LLM-scale diagnostics on TOFU/Llama
We ask whether the K K K -channel quantities of Section 5 remain informative in the LLM regime, where explicit GGN leverage on the full parameter space is intractable. All runs use the open-unlearning recipe on TOFU with Llama-3.2-1B-Instruct [6 , 14 , 27 ] and the released retain models as θ r ⋆ \theta_r^\star θ r ⋆ . Appendix § 11 .§ 1 reports an oracle-NLL proxy for the forget-side residual channel along one GradDiff trajectory. Appendix § 11 .§ 2 reports θ ⋆ \theta^\star θ ⋆ -only LoRA-space scores for the forget and retain channels against five method/split combinations with three optimizer seeds each. Both protocols are described in Appendix § 10 .
Oracle-NLL forget-channel proxy
Equation (22) couples the forget self-leverage with the residual term δ f − \delta_f^- δ f − ; for next-token CE, the per-token NLL of the gold answer under θ ⋆ \theta^\star θ ⋆ is not a direct estimate of K f , f K_{f,f} K f , f , but it is an oracle-evaluable proxy for this residual-weighted forget channel. We compute s N L L ( x i ) = avg θ ⋆ ( x i ) s_{\mathrm{NLL}}(x_i) = \operatorname{avg}_{\theta^\star}(x_i) s NLL ( x i ) = avg θ ⋆ ( x i ) for each forget example, sort examples by it, and split them into four equal-sized quartile groups. Because this score uses only forget-example oracle NLL, it probes the forget-side channel rather than retain cross-leverage.
Observation. The qualitative prediction holds for this proxy: the high-score quartile (Q4) separates from the low-score one (Q1) throughout the trajectory, with a Q4/Q1 ratio of 1.3–3.1× at intermediate checkpoints and a per-example Spearman(s N L L , ∣ Δ ∣ s_{\mathrm{NLL}}, |\Delta| s NLL , ∣Δ∣ ) that starts at 0.99 at step 0 and decays to ~0.1 as the forget channel saturates (Table 6). The diagnostic remains informative at LLM scale despite the absence of an explicit Hessian, optimization noise, and the move from squared loss to next-token CE. We note that this proxy still requires the retrained reference θ r ⋆ \theta_r^\star θ r ⋆ ; a fully deployable LLM-scale variant would build on the precomputable e f + e_f^+ e f + form of Theorem 1 (computed only at θ ⋆ \theta^\star θ ⋆ ), which we discuss as future work in Section 6.
θ ∗ \theta^\ast θ ∗ -only K f f K_{ff} K f f and K r f K_{rf} K r f scores over independent seeds
The Section 5.1 regime comparison hinges on K f f K_{ff} K f f and K r f K_{rf} K r f being computable. To check that this is not specific to small networks, and to remove the dependence on a single trajectory and on the oracle, we compute both channels once at θ ∗ \theta^\ast θ ∗ in a LoRA chart (r = 8 r=8 r = 8 , 5.6 × 10 6 5.6 \times10^6 5.6 × 1 0 6 coordinates) with a diagonal-plus-Woodbury GGN, and compare them with the realized per-example gaps ∣ N L L u − N L L r e t ∣ |\mathrm{NLL}_u-\mathrm{NLL}_{\mathrm{ret}}| ∣ NLL u − NLL ret ∣ of GradDiff, NPO, and SimNPO over three optimizer seeds on forget01 and forget05 (full protocol in Appendix B). The forget score is the residual-weighted self-leverage g j T H ^ − 1 g j g_j^\mathsf{T}\hat{H}^{-1}g_j g j T H ^ − 1 g j and the retain score is the cross-leverage row sum ∑ j ∈ D f ∣ g i T H ^ − 1 g j ∣ \sum_{j\in\mathcal{D}_f}|g_i^\mathsf{T}\hat{H}^{-1}g_j| ∑ j ∈ D f ∣ g i T H ^ − 1 g j ∣ , both from gradients at θ ∗ \theta^\ast θ ∗ alone, so this is the precomputable form anticipated in Section 6.
Table 7. TOFU/Llama-3.2-1B: θ ∗ \theta^\ast θ ∗ -only K K K -channel scores vs. realized per-example gaps over independent optimizer seeds. Unlearning uses the open-unlearning full-finetune recipe (lr 10 − 5 10^{-5} 1 0 − 5 , 10 epochs, 8–10 per-epoch checkpoints per trajectory) for {forget01, forget05} × \times × {GradDiff, NPO} and forget05 SimNPO, with 3 optimizer seeds each. The reference is the released retain99/95 model. Scores are computed once at θ ∗ \theta^\ast θ ∗ in the LoRA chart (r = 8 r=8 r = 8 , diagonal GGN + MC-Woodbury H ^ \hat{H} H ^ , λ = 0.01 \lambda=0.01 λ = 0.01 ) without access to θ r ∗ \theta_r^\ast θ r ∗ or to the trajectories. With g i g_i g i the LoRA-chart gradient of the answer NLL of example i i i at θ ∗ \theta^\ast θ ∗ , the forget score is g j T H ^ − 1 g j g_j^\mathsf{T}\hat{H}^{-1}g_j g j T H ^ − 1 g j and the retain score is ∑ j ∈ D f ∣ g i T H ^ − 1 g j ∣ \sum_{j\in\mathcal{D}_f}|g_i^\mathsf{T}\hat{H}^{-1}g_j| ∑ j ∈ D f ∣ g i T H ^ − 1 g j ∣ . Actual gaps are the per-example ∣ N L L u − N L L r e t ∣ |\mathrm{NLL}_u-\mathrm{NLL}_{\mathrm{ret}}| ∣ NLL u − NLL ret ∣ on the forget set (top) and on 400 retain anchors (bottom). Per-epoch checkpoints (checkpoint 0 excluded) are aggregated by the median within a trajectory and then averaged across seeds, with hierarchical-bootstrap CIs (B = 1,000 B=1{,}000 B = 1 , 000 ). The Δ \Delta Δ columns subtract the Spearman of a cheap per-example baseline, namely the NLL at θ ∗ \theta^\ast θ ∗ on the forget channel and the gradient norm (or the NLL where marked “loss”) on the retain channel. The null columns give the 95% interval of the within-checkpoint permutation null (1,000 permutations) and the two-sided p p p of the observed Spearman. Token KL is the mean answer-token KL( p θ ∗ ∥ p θ u ) (p_{\theta^\ast}\Vert p_{\theta_u}) ( p θ ∗ ∥ p θ u ) at the final checkpoint. The scores identify examples with the largest NLL deviations from retraining on both channels, above the permutation null in every cell except the NPO forget01 forget channel, and are comparable to the cheap per-example baselines.
Observation. The θ ∗ \theta^\ast θ ∗ -only scores carry usable per-example signal at LLM scale. The retain score ranks the retain probes that the unlearners disturb most, with Spearman 0.26–0.49 and Q4/Q1 gap ratios of 1.3–3.7, far outside the permutation null (± 0.02 \pm0.02 ± 0.02 ), on both splits and for all three methods (SimNPO on forget05 only), across independent seeds. The forget score ranks the forget examples whose treatment departs most from retraining, with Spearman 0.14–0.29 and Q4/Q1 ratios of 1.1–1.8, and is at the null only for NPO on forget01 (p = 0.99 p=0.99 p = 0.99 ). Both scores are available before any unlearning run, so they can flag which examples will be hard to unlearn faithfully and where an evaluation of the unlearned model should look. Two limits are visible in the same table. The scores are comparable to, not clearly better than, the per-example loss or gradient norm at θ ∗ \theta^\ast θ ∗ (margins between -0.17 and +0.06), and they are not calibrated predictions of retraining-gap magnitude. A separate diagnostic in Table 8 evaluates the profile and scale of NLL movement from θ ∗ \theta^\ast θ ∗ on the same trajectories. The per-example profile of the NLL movement away from θ ∗ \theta^\ast θ ∗ scores (cosine above its permutation null on every row, centered Pearson 0.16–0.58 forget and 0.19–0.39 retain), but the relative error after fitting one scale stays at 0.6–0.8 and the fitted scale varies by 27–65% across checkpoints and seeds. Several differences from the local validation may contribute. The curvature is a diagonal-plus-rank-4 GGN on a LoRA chart of 5.6 × 10 6 5.6 \times10^{6} 5.6 × 1 0 6 of the 1.2 × 10 9 1.2 \times10^{9} 1.2 × 1 0 9 parameters, the endpoints are early-stopped ten-epoch runs, and these scores target sequence NLL rather than the logit gaps of § 5.2 . These endpoints sit 0.07–1.2 nats of retain KL from a cold-start θ r ∗ \theta_r^* θ r ∗ , far beyond the ≤ 10 − 5 \le10^{-5} ≤ 1 0 − 5 -nat regime of the positive local validation in § 5.2 . Computing the scores on a half-frozen LoRA chart with an exact GGN on the active half changes little. These comparisons do not isolate the effects of chart restriction, curvature approximation, endpoint nonstationarity, or target choice; whether richer curvature or more local endpoints improve prediction remains open. The output-space KL of the final checkpoints (0.07–1.2 nats on the retain probes) confirms that these recipes do not leave the retain channel untouched.
Table 8. TOFU/Llama-3.2-1B: per-example profile of the NLL movement from θ ∗ \theta^* θ ∗ against the θ ∗ \theta^* θ ∗ -only scores. Predictor-only re-analysis of the trajectories of Table 7 (3 seeds, 7–10 checkpoints each) with the same forget and retain scores, which carry no method weights W − W^- W − and no ρ / η \rho/\eta ρ / η factor. The actual quantity is the per-example sequence-NLL movement nll ( ckpt ) − nll ( θ ∗ ) \operatorname{nll}(\operatorname{ckpt})-\operatorname{nll}(\theta^*) nll ( ckpt ) − nll ( θ ∗ ) (absolute value on the retain channel). This is the movement away from the original model and not the gap to retraining, so the test asks only whether the per-example profile of that movement is predicted up to one scale. P P P is the per-example Pearson correlation (its permutation null is 0). Cosine is the cosine between the two vectors (median over checkpoints per seed, mean± \pm ± sd over seeds) and perm. null is its within-checkpoint permutation baseline (500 permutations), so the profile is predicted only where Cosine exceeds its null. RelErr is min s ∥ s k − Δ ∥ 2 / ∥ Δ ∥ 2 \min_s\|s\,k-\Delta\|_2/\|\Delta\|_2 min s ∥ s k − Δ ∥ 2 /∥Δ ∥ 2 after fitting the single free scale, and scale CV is the coefficient of variation of that fitted scale (within-seed / pooled). The second block computes the scores on the half-frozen LoRA chart (A A A frozen, exact GGN over the active B B B coordinates, S = 4 S=4 S = 4 vocabulary-dimension MC pseudo-gradients). NPO/SimNPO rows use the unweighted scores and are therefore conservative checks, not calibrated W − W^- W − predictions.
Additional Experimental Validations
These additional experiments test the framework along axes complementary to Section 5 and are referenced from there but not relied on for the main claims.
ProjGD and Newton-step behavior in the regime comparison. ProjGD has nearly unchanged retain RMS across regimes (0.438 → \to → 0.440) but larger output gaps than GradDiff, NPO, and SimNPO. Its projection removes components along sampled retain-loss gradients; this does not guarantee unchanged retain logits, and the regime comparison does not isolate the cause of its larger gaps. The Newton step, which recovers θ r ∗ \theta_r^* θ r ∗ exactly in the linear case (Table 1 ), shows the largest nonlinear output gaps (0.46 → \to → 0.73 on the forget set). This illustrates the limits of the local quadratic approximation. The comparison does not separately identify GGN mismatch, higher-order error, or imperfect reference stationarity.
Linear closed-form sanity check (Theorem 2). On a synthetic ridge regression problem we compare the analytical gap of Eq. (12) to the numerical solution of the GradDiff stationarity equations. The relative error is < 10 − 10 <10^{-10} < 1 0 − 10 across all evaluation sets and all values of ρ \rho ρ inside the stability region, confirming that the closed form is exact in the linear setting.
Full local validation and controls (Eq. (22)). Tables 9 and 10 expand the local GradDiff/NPO/SimNPO validation and the ProjGD and Pure GA/IMU comparisons of Table 3 , under the protocol of § 5.2 . Panel A adds the held-out test channel K a f K_{af} K a f (P k a = 0.80 P^a_k=0.80 P k a = 0.80 –0.96 on MNIST and 0.71–0.95 on CIFAR-10 for GradDiff, NPO, and ProjGD), the relative ℓ 2 \ell_2 ℓ 2 error of the predicted gap vector, the output-space KL to θ r ⋆ \theta_r^\star θ r ⋆ , and the IMU rows, which behave like Pure GA. For Pure GA and IMU the correlations are small or negative (between − 0.51 -0.51 − 0.51 and 0.32 0.32 0.32 ) under both the K K K -geometry and the Euclidean-Gram predictor, and their endpoints sit 0.7–3 nats of KL away from θ r ⋆ \theta_r^\star θ r ⋆ , so these rows are outside the local regime as discussed in Section 6. Panel B sweeps the removal scale s ∈ { 0.05 , 0.2 , 0.5 , 1 , 2 } s \in\{0.05, 0.2, 0.5, 1, 2\} s ∈ { 0.05 , 0.2 , 0.5 , 1 , 2 } . P f K P_f^K P f K stays at or above 0.96 (MNIST) and 0.88 (CIFAR-10) up to s = 2 s=2 s = 2 , while the endpoint displacement from θ r ⋆ \theta_r^\star θ r ⋆ grows by more than an order of magnitude and the class-regime forget cross-entropy rises. Table 11 is the β \beta β -swap control referenced in § 5.1 .
Table 9 (PDF p. 24). Panel B: agreement across the normalized removal scale s s s (GradDiff / NPO), ρ = s n / λ max ( K f f ) \rho=s\,n/\lambda_{\max}(K_{ff}) ρ = s n / λ m a x ( K f f ) .
Table 9 : MNIST: full local gap validation and controls. The setting is a 2-layer tanh MLP (784 → \to → 24 → \to → 10, d = 19,090 d=19{,}090 d = 19 , 090 ), n = 4000 n=4000 n = 4000 , λ = 10 − 2 \lambda=10^{-2} λ = 1 0 − 2 , exact fp64 throughout. Protocol (Appendix § 10 ). The warm retrain is polished to stationarity under the full-n n n normalization and H r H_r H r is the exact fp64 GGN. GradDiff, NPO, SimNPO, Pure GA, and IMU endpoints are tracked from θ r ⋆ \theta_r^\star θ r ⋆ by Newton continuation. ProjGD uses projected-gradient dynamics and the unprojected predictor. GradDiff/NPO/SimNPO test the local predictors of Table 4 ; Pure GA and IMU are controls outside the local regime. Mean± \pm ± sd over 10 seeds. At s = 0.1 s=0.1 s = 0.1 the GradDiff endpoint lies 0.0008 (random) / 0.0040 (class) from θ r ⋆ \theta_r^\star θ r ⋆ in parameter norm, versus ∥ θ ⋆ − θ r ⋆ ∥ 2 = 3.39 \lVert\theta^\star-\theta_r^\star\rVert_2=3.39 ∥ θ ⋆ − θ r ⋆ ∥ 2 = 3.39 / 4.06, illustrating the locality of the retain-anchored validation. Columns are as in Table 3 . P a K P_a^K P a K is the per-logit Pearson correlation on the held-out test subsample (K a f K_{af} K a f channel), RelErr is the relative ℓ 2 \ell_2 ℓ 2 error of the predicted gap vector, and KL is the output-space KL(p θ r ⋆ ∥ p θ u p_{\theta_r^\star}\Vert p_{\theta_u} p θ r ⋆ ∥ p θ u ) at the endpoint. The shuffled-anchor control gives ∣ P ∣ ≤ 0.20 \lvert P\rvert\leq0.20 ∣ P ∣ ≤ 0.20 . †method-specific ( K , W − ) (K,W^-) ( K , W − ) scoring with frozen reference weights for NPO/SimNPO. ‡test channel scored with uniform weights (for NPO the weights at θ r ⋆ \theta_r^\star θ r ⋆ are nearly uniform), not stored for SimNPO. Panel B repeats the construction at larger removal scales. P f K P_f^K P f K stays at or above 0.96 up to s = 2 s=2 s = 2 , while the endpoint moves farther from θ r ⋆ \theta_r^\star θ r ⋆ and the class-regime forget cross-entropy rises. Here s s s is normalized by λ max ( K f f ) \lambda_{\max}(K_{ff}) λ m a x ( K f f ) alone, so s = 1 s=1 s = 1 is the squared-loss boundary of Corollary 1. Under cross-entropy the softmax curvature B f − B_f^- B f − enters the stability factor of Eq. (22) , and the tracked fixed point continues past s = 1 s=1 s = 1 .
Table 9. Panel A: agreement at the operating point s = 0.1 s=0.1 s = 0.1 . Panel B: agreement across the normalized removal scale s s s (GradDiff / NPO), ρ = s n / λ max ( K f f ) \rho=s\,n/\lambda_{\max}(K_{ff}) ρ = s n / λ m a x ( K f f ) .
Stability divergence as s → 1 s\to1 s → 1 (Corollary 1). Sweeping the stability fraction s = ρ n λ max ( K f f ) ∈ { 0.05 , 0.1 , 0.2 , 0.4 , 0.6 , 0.8 , 0.9 , 0.95 } s=\frac{\rho}{n}\lambda_{\max}(K_{ff})\in\{0.05,0.1,0.2,0.4,0.6,0.8,0.9,0.95\} s = n ρ λ m a x ( K f f ) ∈ { 0.05 , 0.1 , 0.2 , 0.4 , 0.6 , 0.8 , 0.9 , 0.95 } , the predicted and actual logit deviations increase as the inverse factor approaches the stability boundary. The norm bound of Eq. (13) has the correct denominator but is extremely loose in absolute magnitude, as expected from a worst-case operator-norm argument.
Table 10 (PDF p. 25). Panel B: agreement across the normalized removal scale s s s (GradDiff / NPO), ρ = s n / λ max ( K f f ) \rho= s\,n/\lambda_{\max}(K_{ff}) ρ = s n / λ m a x ( K f f )
Table 10 : CIFAR-10: full local gap validation and controls. The setting is a 2-layer tanh MLP head (512 → 24 → 10 512 \to24 \to10 512 → 24 → 10 , d = 12,562 d=12{,}562 d = 12 , 562 ) over frozen ResNet-18 penultimate features, n = 4000 n=4000 n = 4000 , λ = 10 − 2 \lambda=10^{-2} λ = 1 0 − 2 , exact fp64 throughout. Protocol (Appendix B). The warm retrain is polished to stationarity under the full-n n n normalization and H r H_r H r is the exact fp64 GGN. GradDiff, NPO, SimNPO, Pure GA, and IMU endpoints are tracked from θ r ∗ \theta_r^* θ r ∗ by Newton continuation. ProjGD uses projected-gradient dynamics and the unprojected predictor. GradDiff/NPO/SimNPO test the local predictors of Table 4 ; Pure GA and IMU are controls outside the local regime. Mean± \pm ± sd over 10 seeds. At s = 0.1 s=0.1 s = 0.1 the GradDiff endpoint lies 0.0011 (random) / 0.0015 (class) from θ r ∗ \theta_r^* θ r ∗ in parameter norm, versus ∥ θ ∗ − θ r ∗ ∥ 2 = 0.91 \lVert\theta^*-\theta_r^*\rVert_2=0.91 ∥ θ ∗ − θ r ∗ ∥ 2 = 0.91 / 1.70, illustrating the locality of the retain-anchored validation. Columns are as in Table 3 . P a K P_a^K P a K is the per-logit Pearson correlation on the held-out test subsample (K a f K_{af} K a f channel), RelErr is the relative ℓ 2 \ell_2 ℓ 2 error of the predicted gap vector, and KL is the output-space KL( p θ r ∗ ∥ p θ u ) (p_{\theta_r^*}\Vert p_{\theta_u}) ( p θ r ∗ ∥ p θ u ) at the endpoint. The shuffled-anchor control gives ∣ P ∣ ≤ 0.18 \lvert P\rvert\leq0.18 ∣ P ∣ ≤ 0.18 . † method-specific ( K , W − ) (K,W^{-}) ( K , W − ) scoring with frozen reference weights for NPO/SimNPO. ‡ test-channel scored with uniform weights (for the weights at θ r ∗ \theta_r^* θ r ∗ are nearly uniform), not stored for SimNPO. Panel B repeats the construction at larger removal scales. P f K P_f^K P f K stays at or above 0.88 up to s = 2 s=2 s = 2 , while the endpoint moves farther from θ r ∗ \theta_r^* θ r ∗ and the class-regime forget cross-entropy rises. Here s s s is normalized by λ max ( K f f ) \lambda_{\max}(K_{ff}) λ m a x ( K f f ) alone, so s = 1 s=1 s = 1 is the squared-loss boundary of Corollary 1. Under cross-entropy the softmax curvature B f B_f B f enters the stability factor of (22) , and the tracked fixed point continues past s = 1 s=1 s = 1 .
Table 10. Panel A: agreement at the operating point s = 0.1 s = 0.1 s = 0.1
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction are that we provide a unified view of machine unlearning methods, and that we provide a theoretical analysis of the output gaps of these methods. These claims are supported by the content of the paper.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes discussions of the limitations of the proposed methods, including assumptions made for theoretical analysis, practical approximation errors, and potential practicality issues.
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides a complete set of assumptions and proofs for all theoretical results, either in the main text or in the supplementary material.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix § 10 describes the datasets, model architectures, training procedures, and evaluation metrics used in the experiments.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We plan to open-source our code; it is not yet publicly available. The datasets are public, and Appendix B describes the experimental protocols.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: Appendix B describes the datasets, model architectures, training procedures, and evaluation metrics used in the experiments.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports results across multiple runs with error bars that capture variability across runs for the main experiments.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Appendix B reports the detailed resource requirements for all experiments.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: The research conducted in the paper adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses potential societal impacts.
Guidelines:
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification: The paper poses no such risks.
Guidelines:
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper properly credits all creators and original owners of assets used, and the licenses and terms of use are explicitly mentioned and respected.
Guidelines:
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [N/A]
Justification: The paper does not release new assets.
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: The paper does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.