FisherEdit: Fisher Information Guided Knowledge Editing for Large Language Models

Introduction

Large language models (LLMs) encode extensive factual knowledge during pretraining, and this knowledge must often be revised as facts become outdated, incorrect, or inconsistent with new evidence [33, 36, 10, 46, 38]. Knowledge editing addresses this need by updating specific facts without full retraining, while aiming to preserve unrelated behavior [50, 42]. This goal is especially difficult in sequential editing, where repeated weight changes can accumulate, interfere with one another, and degrade the model’s general abilities [49, 45]. Maintaining reliable behavior under repeated edits, therefore, remains a central challenge for knowledge editing.

A prominent approach is direct weight editing. Locate-then-edit methods [27 b] identify where factual associations are stored and update the corresponding weights. Because repeated updates can interfere with previously edited or preserved knowledge, recent methods add preservation constraints to restrict how the model changes. AlphaEdit [8] is a representative example: it uses a null-space projection to steer edits away from activation directions associated with prior knowledge. This projection-based view has become influential in sequential editing and has motivated several follow-up methods [48, 18].

Given AlphaEdit’s influence, we revisit it to examine what its null-space projection actually contributes. AlphaEdit offers a clean geometric intuition: edits are projected away from activation directions associated with prior knowledge, so the corresponding knowledge should be preserved. However, this intuition only partially explains its empirical behavior. As illustrated in Figure 2, the default projector remains close to the identity and removes only a small portion of the activation space, leaving most update directions largely unchanged. Our more systematic analysis further shows that preservation in AlphaEdit is strongly shaped by update regularization, not projection alone. This motivates us to rethink preservation in sequential editing as a problem of designing more precise update regularization.

The key issue is how to constrain edits without removing the flexibility needed to encode new facts. Existing preservation mechanisms often operate at a coarse level: hard subspace constraints remove entire directions [25, 48, 18], while regularization-based methods typically constrain the update through global magnitude, complexity, or spectral properties [15, 23, 52]. While effective, these constraints do not directly distinguish parameters by their importance to general behavior. Parameter sensitivity in LLMs is highly uneven: some weight perturbations strongly affect general predictions, while others have limited impact [9, 22]. Effective preservation should therefore be importance-aware, imposing stronger constraints on sensitive parameters while leaving less critical parameters available for editing. Following this principle, we propose FisherEdit, a

Fisher-guided regularization method for sequential knowledge editing. It uses a diagonal Fisher estimate to measure parameter sensitivity and weight the update penalties accordingly. This constrains sensitive weights more strongly while allowing less sensitive weights to adapt to new facts. Our contributions are as follows:

• We re-examine AlphaEdit and find that its preservation benefit comes mainly from update regularization, with little contribution from the default null-space projection. Strengthening the projection trades edit efficacy for locality.

• Based on our analysis, we rethink preservation in sequential editing as regularization and introduce FisherEdit, a theoretically motivated method that uses Fisher geometry to derive an importance-aware, element-wise update penalty, together with a specialized row-wise closed-form solver.

• Across backbones and benchmarks, FisherEdit achieves state-of-the-art performance while better preserving general abilities on downstream tasks and reducing representation- and parameter-space drift relative to prior methods.

Related Work

Knowledge editing (KE) updates factual associations in LLMs or reroutes queries to corrected knowledge, without full retraining or broad degradation of unrelated behavior [50, 42]. Existing methods can be broadly grouped into learning-based editors and direct weight-editing methods.

Learning-based editors train auxiliary modules to produce, route, or store edits for a frozen base model. Representative methods include KnowledgeEditor [7], MEND [30], MALMEN [40], SERAC [31] and RLEdit [21]. These methods require additional training, and their effectiveness depends on the editor’s capacity and training distribution.

Direct weight-editing methods modify target-model weights directly. Knowledge Neurons [6] and ROME [27] localize factual associations in transformer MLPs, while MEMIT [28] extends this to batched least-squares editing. Subsequent work adds constraints to preserve prior knowledge [13, 53] (Meng et al. 2025). Among projection-based methods, AlphaEdit [8] uses the null space of preserved activation keys; EvoEdit [25] evolves it during editing; and KDE [48] decouples past and current knowledge through orthogonal projection. CrispEdit [18] instead hard-thresholds a K-FAC curvature spectrum, restricting edits to the approximate null space of capability curvature. Such hard projections can remove both prior-knowledge and edit-relevant signals, trading efficacy for locality. FisherEdit instead uses a soft, element-wise Fisher penalty, retaining all update directions while constraining important parameters more strongly.

Connection to continual-learning regularization. Importance-weighted regularization is classical in continual learning: EWC [20] penalizes movement of parameters with large diagonal Fisher, and SI ([51]), MAS ([1]), and Meta-CL ([44]) derive related online importance estimates. FisherEdit adapts this principle to least-squares editing, where updates are computed in closed form rather than through SGD. Its Fisher penalty is motivated by a local second-order approximation to predictive KL divergence (§ 4.4).

Rethinking Null-Space Projection

We now examine what drives preservation in AlphaEdit ([8]). After introducing its editing objective, we use controlled ablations to separate the effects of null-space projection and update regularization.

Preliminaries

Following ([27 b]), we edit the MLP down-projection matrix $\mathbf{W} \in\mathbb{R}^{d \times m}$. For a batch of $n$ edits, let $\mathbf{K} \in\mathbb{R}^{m \times n}$ denote the edit keys, $\mathbf{V} \in\mathbb{R}^{d \times n}$ the target values. The goal is to find an update $\Delta\in\mathbb{R}^{d \times m}$ such that the edited weights map edit keys to the target values:

Equivalently, $\Delta$ should fit the residual left by the current weights:

In sequential editing, the update should also avoid changing previous edits and unrelated knowledge. Let $\mathbf{K}_p \in\mathbb{R}^{m \times n_p}$ denote historical edit keys from previous batches, $\mathbf{K}_1 \in\mathbb{R}^{m \times n_1}$ denote current edit keys, and $\mathbf{K}_0 \in\mathbb{R}^{m \times n_0}$ denote preserved keys sampled from a background corpus. A standard locate-then-edit objective is

Term (a) fits the new edits, while terms (b) and (c) penalize changes along historical-edit and preserved-knowledge directions, respectively.

AlphaEdit ([8]) replaces the penalty on $\mathbf{K}_0$ with a null-space projector $\mathbf{P}_{\tau}$. Specifically, $\mathbf{P}_{\tau}$ is computed from the eigenspectrum of $\mathbf{K}_0\mathbf{K}_0^{\mathsf{T}}$: directions whose eigenvalues exceed a threshold $\tau$ are treated as occupied, and the remaining subspace is used for editing. AlphaEdit solves

The following analysis asks whether the projection $\mathbf{P}_{\tau}$, the regularization strength $\lambda$, or the threshold $\tau$ is responsible for AlphaEdit’s empirical preservation behavior.

What Actually Drives Preservation?

Finding 1: The Projector $\mathbf{P}_{\tau}$ is near-identity. If the null-space mechanism is the main source of preservation, the projector should remove a substantial set of directions associated with prior knowledge. We first inspect the projector used by AlphaEdit under its default setting. As shown in Figure 2(a), the matrix is almost identical to the identity: its diagonal entries are close to one, and its off-diagonal entries are negligible. This behavior comes from the spectrum of $\mathbf{K}_0\mathbf{K}_0^{\mathsf{T}}$. Figure 2(b) shows that only a small number of singular directions exceed the default threshold $\tau$, while the vast majority fall below it and are kept for editing. Thus, the projected-out subspace covers only a tiny fraction of the activation space. In practice, $\mathbf{P}_{\tau}$ leaves most update directions unchanged.

We then ask whether this near-identity projector affects editing outcomes. We compare AlphaEdit with a variant that disables the projector while keeping the same regularization strength. As shown in Figure 3, the two variants largely overlap across Efficacy, Generalization, and Specificity metrics over the working range of $\lambda$. This suggests that, under the default setting, AlphaEdit’s projector contributes little to its empirical preservation behavior.

Finding 2: Regularization is the main control knob. If the projector changes little, the remaining candidate explanation is the update regularizer. To isolate its effect, we keep the projector fixed. and vary only the regularization strength $\lambda$. Figure 4 shows that this single change produces large shifts across all edit-success metrics. With weak regularization, the update becomes unstable and performance collapses; with stronger regularization, performance is substantially restored. These results suggest that much of AlphaEdit’s robustness comes from controlling update magnitude through regularization, with limited contribution from the default projection.

Finding 3: Stronger projection only moves along a trade-off. One possible explanation is that the default threshold $\tau$ makes the projector too weak. We therefore lower $\tau$, forcing $P_\tau$ to remove more activation directions. If the null-space interpretation were sufficient, a stronger projection should improve preservation without harming edit success. The results show a different pattern. As $\tau$ decreases, specificity improves, but efficacy and generalization drop sharply (Figure 1). Thus, stronger projection does not recover a better protected subspace; it simply trades editing success for locality. This happens because the removed directions are not purely preserved-knowledge directions. They can also contain information needed to realize the edit. Removing them therefore protects some unrelated behavior, but weakens the edit itself.

Takeaway. These findings suggest a different design principle for sequential editing: preservation should act on parameters, not entire activation directions. Hard projection can remove directions needed for the new edit, while uniform shrinkage ignores which weights matter most for general behavior. A better constraint should distinguish sensitive weights from safer ones. This motivates FisherEdit, which applies a Fisher-weighted, element-wise penalty directly to the update.

Method

Fisher-Guided Editing Objective

Motivated by the analysis above, our FisherEdit shifts preservation from activation-space projection to parameter-space regularization. We keep the standard edit-fitting objective, but penalize each parameter update according to its Fisher importance:

Here, $\mathbf{F}\in\mathbb{R}^{d\times m}$ is the diagonal Fisher Information matrix and $\odot$ denotes element-wise multiplication. This term makes changes to high-Fisher parameters more costly while leaving low-Fisher parameters more flexible for encoding new facts. Compared with the objectives in Eqns. (3)(4), FisherEdit keeps the edit-fitting and historical-edit terms, but replaces the preserved-key penalty with a Fisher-weighted penalty on the update itself.

Solving the Fisher-Weighted Objective

Eqn. (5) is a quadratic objective that admits a closed-form solution. Because the Fisher penalty is element-wise, we can solve for each row of the update independently. Let $\mathbf{G}=K_{1}K_{1}^{\mathsf{T}}+K_{p}K_{p}^{\mathsf{T}}$, and let $\delta_i$, $e_i$, and $f_i$ denote the $i$-th rows of $\Delta$, $E$, and $\mathbf{F}$, respectively. With $\mathbf{D}_i=\operatorname{diag}(f_i)$, setting the gradient of Eqn. (5) to zero gives $\delta_i(\mathbf{G}+\lambda\mathbf{D}_i)=e_iK_{1}^{\mathsf{T}}$ for each row, yielding

The shared matrix $\mathbf{G}$ captures the constraints from current and historical edit keys, while the row-specific $\mathbf{D}_i$ weights the update according to parameter importance.

Directly evaluating Eqn. (6) requires an $m \times m$ matrix inverse for each row. Since $m$ is the MLP intermediate dimension, this becomes expensive at LLM scale. We therefore exploit the low-rank structure $G = UU^{\top}$, where $U = [K_1 \mid K_p] \in\mathbb{R}^{m \times r}$ and $r = n_1 + n_p$. Applying the Woodbury identity [11] gives the equivalent update

Since $D_i$ is diagonal, $(\lambda D_i)^{-1}$ is computed by element-wise division. The remaining linear system has size $r \times r$, reducing the cost when the number of edit keys is much smaller than the MLP dimension ($r \ll m$). All $d$ row-wise systems can be solved using batched linear algebra, with an overall complexity of $O(dmr^2 + dr^3)$. The full derivation is provided in Appendix § 13.

Estimating the Fisher Diagonal

The solver above assumes access to the Fisher diagonal $\mathbf{F}$ for the edited weight block. We estimate this quantity once before editing and reuse it throughout the edit sequence. This keeps the editing procedure efficient while giving each parameter a fixed importance score measured from the model’s original predictive behavior.

We use the empirical Fisher diagonal. Given a held-out calibration set $\mathcal{D} = \{(x_t, y_t)\}_{t=1}^{N}$, the Fisher value for parameter $W_{ij}^{(\ell)}$ in layer $\ell$ is estimated as

where $\mathcal{L}$ is the cross-entropy loss. We compute this estimate only for the MLP down-projection matrices that are later edited, rather than for all model parameters.

For calibration, we follow Šliogeris et al. [39] and use $N=1{,}000$ examples sampled from MMLU [17]. The calibration data is separate from the edit requests and is used only to estimate parameter sensitivity. Each example is processed individually, so the squared gradients correspond to per-example empirical Fisher estimates rather than gradients averaged over a batch.

Theoretical Interpretation

In this section, we explain why the element-wise Fisher regularizer in Eqn. (5) provides an effective regularization principle for model editing. Specifically, we relate it to the locality cost of an update $\Delta$, measured by the KL divergence between the pre-edit and post-edit predictive distributions on a reference set $\mathcal{D}_{\mathrm{ref}}$:

To establish this connection, we consider a second-order expansion of $\mathcal{L}_{\mathrm{loc}}$ around $\Delta= 0$. Under the regularity conditions in Appendix § 14.§ 2, its value and gradient vanish at this point, and its Hessian equals the Fisher matrix $\mathbf{F}^{(S)}$ restricted to the edited parameters. Writing $u = \operatorname{vec}(\Delta)$, we obtain

The full $md \times md$ Fisher is intractable at LLM scale. Following Kirkpatrick et al. [20], we use a diagonal approximation, estimated in practice by the empirical Fisher in Eqn. (8):

which recovers the form of the regularizer in Eqn. (5), with the constant absorbed into $\lambda$. FisherEdit therefore uses empirical parameter sensitivities to approximate the local cost of changing the pre-edit distribution, providing a tractable regularization principle for preserving model behavior. The full Taylor expansion and derivation are provided in Appendix § 14.

Experiments

We conduct comprehensive experiments to answer two core questions: (1) How does FisherEdit improve the overall performance of editing? (2) Does FisherEdit protect the general performance?

Experiment Setup

Datasets and Models. We evaluate FisherEdit on three widely used model editing benchmarks: CounterFact ([27]), Ripple ([5]), and EVOKE ([53]). Experiments are conducted under a sequential editing setting with 1,000 edits, using three representative backbones: Llama-3-8B-Instruct (Graff et al., 2024), Qwen2.5-7B-Instruct (Owen et al., 2025), and Olmo-3-7B-Instruct ([32]). To assess the impact on general capabilities, we further evaluate the edited models on HumanEval ([3]), ARC-Challenge/Easy ([4]), IFEval ([54]), WinoGrande ([37]) and GPQA ([35]), and report WikiText ([29]) perplexity after sequential editing on CounterFact.

Baselines and Metrics. We compare FisherEdit with twelve baselines, including the learning-based and direct weight-editing methods discussed in § 2, as well as FT [55], GRACE [16], WISE [41], AnyEdit [19], and UltraEdit [14]. For editing performance, we report efficacy (Eff), generalization (Gen), specificity (Spe), and their unweighted average (Avg). For general performance, we report task accuracy and WikiText perplexity.

Editing Performance

Table 1 reports CounterFact results across three backbones. FisherEdit achieves the highest Avg in all three settings, with gains over AlphaEdit of +1.10 on Llama, +2.37 on Qwen, and +1.39 on Olmo. Since Efficacy is already close to saturated for both AlphaEdit and FisherEdit, the main differences appear in Gen and Spe. This indicates that FisherEdit improves the quality of the update primarily by better balancing paraphrase transfer and neighborhood preservation.

Method	Llama-3-8B-Instruct Eff	Gen	Spe	Avg	Qwen2.5-7B-Instruct Eff	Gen	Spe	Avg	Olmo-3-7B-Instruct Eff	Gen	Spe	Avg
FT	65.02	58.05	41.63	54.90	73.23	62.76	40.52	58.84	95.17	70.87	56.12	74.05
ROME	33.84	36.51	68.16	46.17	37.33	40.92	65.52	47.92	36.10	39.12	69.32	48.18
MEMIT	48.00	47.64	52.52	49.39	64.20	62.41	55.88	60.83	71.38	69.53	55.45	65.45
MEND	48.41	48.41	51.05	49.29	1.13	0.67	1.15	0.98	38.97	43.13	62.18	48.09
GRACE	21.84	26.15	78.13	42.04	37.23	41.94	64.55	47.91	39.17	42.00	64.09	48.42
RECT	33.94	36.87	70.05	46.95	36.20	40.61	65.87	47.56	36.71	39.89	68.39	48.33
PRUNE	35.48	37.48	67.88	46.95	37.33	40.92	65.52	47.92	36.00	38.71	69.47	48.06
WISE	43.59	42.31	61.55	49.15	44.21	45.23	61.76	50.40	39.28	42.10	63.80	48.39
AnyEdit	52.00	47.33	51.72	50.35	69.53	48.05	56.34	57.97	63.48	51.28	48.29	54.35
RLEdit	24.20	27.43	75.63	42.42	35.79	38.20	66.42	46.80	22.25	29.94	75.63	42.61
UltraEdit	52.41	53.12	46.77	50.77	60.61	42.05	69.57	57.41	43.38	45.33	57.57	48.76
AlphaEdit	99.79	90.15	73.25	87.73	100.00	64.15	66.36	76.84	99.48	75.84	72.76	82.69
FisherEdit (ours)	99.90	90.92	75.68	88.83	99.90	71.08	66.66	79.21	99.38	79.53	73.33	84.08

Table 2 further evaluates whether the same trend holds across datasets on a fixed backbone. On Olmo-3-7B-Instruct, FisherEdit obtains the best Avg on CounterFact, Ripple, and EVOKE. The improvements over AlphaEdit are consistent: +1.39, +3.01, and +0.47 points, respectively. These results support the claim that Fisher-weighted regularization provides a robust preservation mechanism across both model and benchmark variations.

Method	COUNTERFACT Eff	Gen	Spe	Avg	RIPPLE Eff	Gen	Spe	Avg	EVOKE Eff	Gen	Spe	Avg
FT	95.17	70.87	56.12	74.05	76.10	59.87	63.79	66.59	94.10	79.48	47.70	73.76
ROME	36.10	39.12	69.32	48.18	42.60	44.01	67.30	51.30	33.50	34.22	65.58	44.43
MEMIT	71.38	69.53	55.45	65.45	48.40	45.99	69.85	54.75	64.80	63.81	51.16	59.92
MEND	38.97	43.13	62.18	48.09	32.90	34.94	74.62	47.48	37.20	37.32	63.73	46.08
GRACE	39.17	42.00	64.09	48.42	50.20	48.39	60.89	53.16	35.40	35.80	63.52	44.91
RECT	36.71	39.89	68.39	48.33	41.00	41.91	68.93	50.61	34.00	34.42	65.66	44.69
PRUNE	36.00	38.71	69.47	48.06	43.80	43.93	67.17	51.63	33.30	34.21	65.59	44.37
WISE	39.28	42.10	63.80	48.39	56.20	56.71	55.06	55.99	36.00	35.96	62.77	44.91
AnyEdit	63.48	51.28	48.29	54.35	50.80	49.85	82.60	61.08	68.10	58.90	40.54	55.85
RLEdit	22.25	29.94	75.63	42.61	6.90	10.03	82.55	33.16	17.10	18.83	78.51	38.15
UltraEdit	43.38	45.33	57.57	48.76	48.00	49.04	57.03	51.36	48.90	48.88	51.19	49.66
AlphaEdit	99.48	75.84	72.76	82.69	96.70	75.10	33.39	68.40	99.90	92.47	70.14	87.50
FisherEdit (ours)	99.38	79.53	73.33	84.08	97.30	74.66	42.28	71.41	99.90	92.94	71.08	87.97

We also evaluate FisherEdit’s computational efficiency by measuring wall-clock time and peak GPU memory across the editing pipeline, with a detailed comparison to AlphaEdit in Appendix § 9.§ 5. The results show that FisherEdit substantially reduces preprocessing time and overall runtime including preprocessing, while using nearly the same peak GPU memory.

General Performance

General domains. Preserving general capabilities beyond the edited facts is crucial, yet is often overlooked by prior editing methods. From Figure 5(a), FisherEdit maintains stable performance across all six downstream tasks, covering code generation, science, commonsense reasoning, instruction following, and challenging knowledge-intensive question answering. In contrast, AlphaEdit and RECT exhibit noticeable drops on HumanEval and GPQA. These results suggest that the Fisher-diagonal regularizer helps mitigate collateral damage while still enabling effective edits.

Language modeling. We further evaluate the impact of editing on general language modeling by reporting WikiText perplexity after sequentially editing $N \in\{200, 400, 600, 800, 1000\}$ CounterFact samples. As shown in Figure 5(b), FisherEdit better preserves general text modeling ability than competing methods during sequential editing. Its WikiText perplexity remains consistently low across all edit counts, increasing only mildly from 10.826 after 200 edits to 11.110 after 1,000 edits. By contrast, AlphaEdit, RECT, and PRUNE show larger increases. These results indicate that FisherEdit maintains the most stable general-language perplexity throughout the editing process.

Ablation Study

Fisher Matrix. To examine whether the Fisher diagonal captures model-specific information, we replace $\mathbf{F}$ in Eq. (5) with an all-ones matrix (Uniform), which reduces the objective to $\ell_2$ regularization, or i.i.d. samples from Uniform[0, 1] (Random), while keeping all other hyperparameters fixed. From Table 3, replacing $\mathbf{F}$ with Uniform largely preserves rewrite and paraphrase performance, which are mainly controlled by $\lambda$ as discussed in § 3, but substantially weakens neighborhood locality. Meanwhile, Random degrades all three metrics. These results indicate that FisherEdit benefits from the structured information encoded in the Fisher diagonal, rather than from generic non-uniform reweighting.

	Eff	Gen	Spe	Avg
Fisher	99.58	91.28	75.13	88.66
Uniform	99.28	91.48	72.22	87.66
Random	95.89	85.74	69.94	83.86

Top-Fisher masking. The Fisher diagonal is heavy-tailed and model-dependent: in layer 4, its largest 1% of cached entries carry 16.8%, 52.6%, and 30.4% of the total mass for Llama-3-8B-Instruct, Qwen2.5-7B-Instruct, and Olmo-3-7B-Instruct, respectively (a uniform matrix would carry 1%). To test whether the high-Fisher coordinates are important for preservation, we rerun the Llama-3-8B-Instruct/CounterFact setting with 1,000 sequential edits while selectively removing or retaining the highest-Fisher coordinates. For mask-top, entries at the top $q$ percent are replaced by that layer’s median; for keep-top, only those entries are retained and all others are replaced by the median. As shown in Table 4, masking the top 1% decreases Specificity by 1.65 points, eliminating approximately 48% of the Full-Fisher–Uniform Specificity gap and 63% of the Average gap. Conversely, keeping only the top 10% recovers approximately 60% of the Specificity advantage and 70% of the Average advantage, while the full diagonal remains strongest. These results provide direct evidence that the proposed Fisher diagonal plays a fundamental role in FisherEdit’s gains, beyond uniform $\ell_2$ regularization alone.

	Eff	Gen	Spe	Avg
Full Fisher	99.90	90.92	75.68	88.83
Mask top 1%	99.49	90.87	74.03	88.13
Mask top 10%	99.38	91.03	73.50	87.97
Mask top 30%	99.49	89.54	74.33	87.79
Keep top 10%	99.69	91.49	74.29	88.49
Uniform	99.90	91.02	72.22	87.71

Number of Calibration Examples. We vary the number of examples $n$ used to estimate the Fisher diagonal while keeping the editing configuration fixed. As shown in Table 5, FisherEdit is largely insensitive to the estimation sample size: the average score remains within a narrow range, from 87.94 to 88.67, and the neighborhood score changes only mildly. Even with only $n=50$ examples, the Fisher constraint retains most of its benefit, suggesting that the regularizer does not require a large reference set to provide effective protection.

$n$	Eff	Gen	Spe	Avg
50	99.79	89.84	74.18	87.94
100	99.69	92.15	73.48	88.44
500	99.79	91.12	75.10	88.67
1000	99.58	91.28	75.13	88.66
2000	99.79	90.76	74.66	88.40

Conclusion

In this work, we addressed parameter interference and capability degradation in sequential knowledge editing with FisherEdit, a Fisher-guided framework that replaces coarse activation-space projection with importance-aware parameter regularization and computes updates through an efficient row-wise closed-form solver. Theoretical analysis and empirical results show that FisherEdit reduces distributional drift and achieves state-of-the-art editing performance while better preserving general capabilities across backbones and benchmarks.

AI Use Statement

AI assistants were used only for language polishing, LaTeX checking, and coding/debugging assistance. All scientific claims, experiments, analyses, and final writing were verified and controlled by the authors.

Reproducibility Statement

The editing algorithm, datasets, evaluation metrics, and hyperparameters are specified in Appendices § 7 and § 8. Appendices § 13 and § 14 provide the derivations, and Appendix § 9.§ 1 reports results across random seeds.

FisherEdit Algorithm Flow

Algorithm 1 FisherEdit sequential editing

Require: pretrained model $\mathbf{W}$, edited layers $\mathcal{L}$, edit stream $\{\mathcal{B}_t\}_{t=1}^{T}$, diagonal Fisher $\mathbf{F}$, regularization strength $\lambda$

1. Initialize historical key memory $\mathbf{K}_p^{(\ell)} \leftarrow\varnothing$ for each layer $\ell\in\mathcal{L}$

2. for $t = 1$ to $T$ do

3. Format the current batch prompts and compute the target representation $\mathbf{z}^{\ast}$ for each edit request

4. for each layer $\ell\in\mathcal{L}$ do

5. Trace the current lookup keys $\mathbf{K}_1^{(\ell)}$ and compute the remaining residual $\mathbf{E}^{(\ell)}$

6. Retrieve historical keys $\mathbf{K}_p^{(\ell)}$ from memory

7. Form $\mathbf{U}^{(\ell)} = [\mathbf{K}_1^{(\ell)}, \mathbf{K}_p^{(\ell)}]$ and $\mathbf{E}^{\prime(\ell)} = [\mathbf{E}^{(\ell)}, 0]$

8. Solve the Fisher-weighted row-wise update $\Delta^{(\ell)}$ by Eqn. (7)

9. Update the layer weights: $\mathbf{W}^{(\ell)} \leftarrow\mathbf{W}^{(\ell)} + \Delta^{(\ell)}$

10. Append the current batch keys to memory: $\mathbf{K}_p^{(\ell)} \leftarrow[\mathbf{K}_p^{(\ell)}, \mathbf{K}_1^{(\ell)}]$

11. end for

12. end for

13. return the edited model

Experimental Details

Evaluation Metrics

For an edit request $(s_i, o_i) \rightarrow o_i^{\ast}$, where $s_i$ is the subject, $o_i$ is the original object, and $o_i^{\ast}$ is the edited target object, let $q_i^{\mathrm{edit}}$ denote the rewrite prompt used to state the edited fact, $Q_i^{\mathrm{para}}$ the set of paraphrase prompts that express the same edited fact, and $Q_i^{\mathrm{loc}}$ the set of neighborhood prompts used to test preserved knowledge. After applying the edit sequence, we evaluate all metrics with the post-edit model $p_{\mathbf{W}+\Delta}$.

For a prompt $q$ and a candidate answer $a = (a_1, \ldots, a_T)$, define the average target negative log-likelihood

Lower $\ell_{\Delta}(a \mid q)$ means that the edited model assigns higher probability to answer $a$ under prompt $q$.

Efficacy. Efficacy measures whether the edited model recalls the new target on the original rewrite prompt. A request is counted as successful when the new answer has lower average NLL than the original answer:

Generalization. Generalization measures whether the same update transfers to paraphrased prompts. For each request, we average the same new-versus-original comparison over its paraphrase set:

Specificity. Specificity measures whether unrelated or neighboring facts are preserved. Let $o_i^{\mathrm{loc}}(q)$ be the correct preserved answer for a neighborhood prompt $q$. A neighborhood prompt is counted as successful when the preserved answer remains more likely than the edit target:

Dataset Details

Main editing datasets. CounterFact [27] is a factual knowledge editing benchmark in which each request rewrites a subject–relation association from an original object to a counterfactual target object. Each record also provides paraphrase prompts for testing whether the edited fact generalizes beyond the original wording, as well as neighborhood prompts for testing whether unrelated facts are preserved. We use the AKEW [43] version of CounterFact and remove records whose aligned AKEW annotations are unavailable, leaving 975 valid edit requests. RippleEdits [5] evaluates whether an edit has appropriate ripple effects on related queries while preserving other relation-specific knowledge about the edited subject. In our experiments, we use the random subset of RippleEdits and evaluate the first 1,000 records. EVOKE [53] is designed to evaluate overfitting in model editing by combining direct rewrite prompts with augmented paraphrase prompts and locality prompts. We use the main subset of EVOKE and evaluate the first 1,000 records. All main editing results are reported under sequential editing with 100 edit requests per batch. The edited model is not reset between batches; after processing the full edit stream, we evaluate the final edited model on all requests.

Downstream datasets. To measure whether editing damages general capabilities beyond the edited facts, we evaluate the edited models on five downstream benchmarks. HumanEval [3] evaluates code generation by asking models to complete Python programming problems, and we report pass@1. ARC-Challenge and ARC-Easy [4] evaluate multiple-choice science question answering, with ARC-Challenge containing the harder subset and ARC-Easy containing the easier subset; we report normalized accuracy. IFEval [54] evaluates instruction-following behavior by checking whether model outputs satisfy verifiable prompt-level constraints, and we report strict prompt accuracy. WinoGrande [37] evaluates commonsense reasoning through adversarial Winograd-style pronoun resolution examples, and we report accuracy. GPQA [35] evaluates difficult graduate-level, Google-proof multiple-choice questions; we use the Diamond zero-shot setting and report normalized accuracy. We additionally report WikiText [29] perplexity to measure general language-modeling quality after sequentially applying CounterFact edits.

Hyperparameter Configurations

All experiments use instruction-tuned backbones: Llama-3-8B-Instruct, Qwen2.5-7B-Instruct, and Olmo-3-7B-Instruct. Models are loaded with HuggingFace AutoModelForCausalLM and evaluated with their corresponding tokenizers. Because the backbones are instruction-tuned, both editing prompts and evaluation prompts are serialized with each tokenizer’s chat template. We run the entire editing pipeline in fp32, including offline precomputation, and use bf16 for downstream inference. We edit the MLP down-projection matrices model.layers.{}.mlp.down_proj at layers 4–8, use the last subject token as the fact lookup token, and optimize target values for 25 gradient steps with learning rate 0.1 and weight decay 0.5. The target loss layer to calculate $V$ is 31 for Llama-3-8B-Instruct and Olmo-3-7B-Instruct, and 27 for Qwen2.5-7B-Instruct. We set $\lambda= 10^{-3}$ across all benchmarks and backbones. For Fisher diagonal estimation, we estimate the empirical Fisher diagonal once from 1,000 MMLU examples, computing per-example squared cross-entropy gradients only for the edited MLP down-projection weights. The resulting Fisher tensors are cached and reused throughout the sequential edit stream.

Additional Experimental Results

Seed Variance and Significance Testing

Table 1 reports single runs under the benchmark’s native edit order. To quantify run-to-run variability, we repeat the experiment with three random seeds for both FisherEdit and AlphaEdit. Each seed reshuffles the edit stream, i.e. the composition and ordering of the sequential batches, and reseeds all stochastic components including the target-value optimization. Both methods see exactly the same cases under matched seeds, so the comparison is paired.

Method	Eff	Gen	Spe	Avg
AlphaEdit	99.69 ± 0.10	90.17 ± 0.75	73.75 ± 0.30	87.87 ± 0.28
FisherEdit	99.79 ± 0.21	92.68 ± 0.70	73.91 ± 0.16	88.80 ± 0.28

The largest gap is in Generalization, where FisherEdit leads by 2.51 points on average. A two-sided Welch’s $t$-test on the three seed-level Generalization scores gives $p = 0.013$. The Average advantage is also consistent rather than seed-specific: FisherEdit is higher in all three paired runs, by +0.76 to +1.03 points (mean +0.93). Efficacy is near saturation for both methods and Specificity is comparable, so the seed-level picture matches the single-run trend in § 5.2.

Results on Mistral-7B-Instruct

To check that the conclusion is not tied to the three backbones of § 5, we extend the evaluation on Mistral-7B-Instruct-v0.3 on both CounterFact and Ripple, against the six baselines from Table 1 and Table 2.

Method	\multicolumn{4}{c	}{COUNTERFACT}	\multicolumn{4}{c	}{RIPPLE}
	Eff	Gen	Spe	Avg	Eff	Gen	Spe	Avg
MEMIT	50.26	50.26	49.74	50.09	53.80	53.58	49.49	52.29
RECT	29.85	31.54	75.46	45.61	31.50	32.38	76.15	46.68
PRUNE	32.41	32.36	71.19	45.32	32.50	33.81	73.24	46.52
RLEdit	48.00	48.26	52.91	49.72	49.30	47.19	55.62	50.70
UltraEdit	54.26	54.15	45.83	51.41	67.60	69.38	34.58	57.19
AlphaEdit	99.90	89.18	76.91	88.66	96.40	76.36	45.67	72.81
FisherEdit (ours)	99.90	90.10	77.12	89.04	95.70	75.99	51.12	74.27

FisherEdit attains the best Average on both benchmarks. On CounterFact it matches AlphaEdit’s saturated Efficacy while improving Generalization and Specificity (88.66 → 89.04 Avg). On Ripple it trades a small amount of rewrite and paraphrase accuracy for a 5.45-point Specificity gain, giving +1.46 Avg. Overall, it extends the backbone-level consistency of our conclusion to four instruction-tuned model families.

AlphaEdit under Matched Regularization Strength

To verify that FisherEdit’s gain does not come merely from using a weaker regularization strength than AlphaEdit’s published $\ell_2 = 1$, we sweep AlphaEdit over $\ell_2 \in\{1, 10^{-1}, 10^{-2}, 10^{-3}\}$ on all three backbones in Table 8.

Backbone	Method	$\ell_2$	Eff	Gen	Spe	Avg
Llama	AlphaEdit	1	99.79	90.15	73.25	87.73
Llama	AlphaEdit	$10^{-1}$	99.79	91.44	74.17	88.47
Llama	AlphaEdit	$10^{-2}$	99.79	91.18	74.56	88.51
Llama	AlphaEdit	$10^{-3}$	99.90	90.77	74.95	88.54
Llama	FisherEdit	$10^{-3}$	99.90	90.92	75.68	88.83
Qwen	AlphaEdit	1	100.00	64.15	66.36	76.84
Qwen	AlphaEdit	$10^{-1}$	99.79	65.79	66.38	77.32
Qwen	AlphaEdit	$10^{-2}$	100.00	65.08	66.19	77.09
Qwen	AlphaEdit	$10^{-3}$	99.28	65.44	62.55	75.76
Qwen	FisherEdit	$10^{-3}$	99.90	71.08	66.66	79.21
Olmo	AlphaEdit	1	99.48	75.84	72.76	82.69
Olmo	AlphaEdit	$10^{-1}$	99.59	76.46	72.87	82.97
Olmo	AlphaEdit	$10^{-2}$	99.59	76.41	72.95	82.98
Olmo	AlphaEdit	$10^{-3}$	99.59	76.62	73.08	83.10
Olmo	FisherEdit	$10^{-3}$	99.38	79.53	73.33	84.08

Three orders of magnitude in $\ell_2$ move AlphaEdit’s Average by at most 0.81 points, and the best setting is model-dependent. FisherEdit stays ahead of every tested coefficient, by +0.29, +1.89, and +0.98 Average points on Llama, Qwen, and Olmo. Tuning a scalar strength therefore does not reproduce what the Fisher diagonal provides.

Fisher Drift and Online Fisher Updating

FisherEdit estimates the Fisher diagonal once before editing and reuses it throughout the edit stream. Because the edited weights change during sequential editing, parameter importance could in principle drift, and re-estimating the Fisher online might then help. We examine both questions under the Llama-3-8B-Instruct / CounterFact setting of Table 1 with 1,000 sequential edits.

Parameter-importance drift. We save the model after 1,000 sequential edits and re-estimate its Fisher diagonal on the same held-out calibration set used before editing. We then compare the pre- and post-editing diagonals at each edited layer using log-Pearson and Spearman correlation, cosine similarity, and the overlap of the top-1% and top-10% coordinate sets.

As Table 9 shows, the importance structure remains strongly aligned after 1,000 edits: Spearman correlation is 0.86–0.90, log-Pearson correlation is 0.87–0.91, cosine similarity is 0.82–0.98, and 72–75% of the highest-Fisher 10% of parameters remain in that set. Both the ranking and the membership of the high-Fisher parameter sets are therefore largely stable under sequential editing with FisherEdit.

Layer	Pearson (log)	Spearman	Cosine	Top-1% overlap	Top-10% overlap
4	0.878	0.874	0.824	0.61	0.73
5	0.871	0.859	0.947	0.61	0.72
6	0.883	0.864	0.979	0.65	0.72
7	0.911	0.893	0.981	0.71	0.75
8	0.913	0.897	0.919	0.71	0.75

Online Fisher variant. To test whether online re-estimation nevertheless helps, we implement a variant that updates the Fisher after every edit batch. Let $\mathbf{F}_t$ be the Fisher used for batch $t$. After applying the edits of batch $t$, we run the updated model on the just-edited prompt/target pairs, compute per-example squared cross-entropy gradients for the edited down-projection weights, and average them into a local diagonal $\mathbf{F}_{\mathrm{local},t}$. The Fisher for the next batch is

where $\gamma\in[0,1]$ controls how quickly the estimate adapts. Setting $\gamma= 1$ exactly recovers the fixed Fisher used in the paper, while smaller $\gamma$ gives more weight to the newly computed local Fisher. All other editing settings are unchanged.

Table 10 shows that online updating provides no improvement: both online variants fall slightly below the fixed Fisher on all four metrics, with Average decreasing by 0.42–0.52 points. They also require an additional forward and backward pass over every edit batch, adding roughly 95–97 seconds per 100 edits, about 45% of the per-batch editing time reported in Table 11. Together with the limited drift in Table 9, these results support the fixed Fisher as the more effective and more efficient choice.

Fisher	Eff	Gen	Spe	Avg	Extra time / batch
Fixed ($\gamma= 1$)	99.90	90.92	75.68	88.83	0
Online ($\gamma= 0.99$)	99.59	90.21	75.14	88.31	$\sim$95 s
Online ($\gamma= 0.9$)	99.79	90.56	74.87	88.41	$\sim$97 s

Phase	AlphaEdit	FisherEdit	Diff.
Wall-clock time
Preprocessing (min)	648.0	17.7	−97.3%
Editing / 100 edits (s)	195.7	213.0	+8.9%
Evaluation (s)	328.4	318.7	−3.0%
End-to-end (min)	680.4	53.4	−92.1%
Peak memory
Peak GPU memory (GiB)	35.19	34.94	−0.7%

Computational Cost

We instrument every phase of the pipeline and report wall-clock time and peak allocated GPU memory on a single RTX 6000 Ada, under the same setting of Table 1.

FisherEdit substantially lowers the cost of constructing the preservation constraint. AlphaEdit requires second-moment computation over 100k Wikipedia samples and layerwise SVD, taking 10.80 hours in our experiment. FisherEdit estimates a diagonal Fisher matrix once from 1,000 MMLU examples and completes preprocessing in 17.70 minutes. Thus, FisherEdit reduces preprocessing time by 36.6$\times$ and end-to-end runtime by 12.8$\times$ for 1,000 sequential edits, without introducing inference-time overhead.

The trade-off is an 8.9% increase in per-batch editing time (195.7 versus 213.0 seconds). This overhead is caused by the Fisher-weighted row-wise solve, while the Woodbury formulation keeps the computation efficient and peak GPU memory nearly unchanged.

Hidden-Space Drift Analysis

To better understand how FisherEdit changes internal representations, we visualize the hidden-state movement induced by FisherEdit and AlphaEdit. We use the EVOKE sequential editing run on Olmo-3-7B-Instruct and select the first and last edited layers, layers 4 and 8. For each edit request, we record the last-token hidden state of the rewrite prompt and ten neighborhood prompts before and after editing. Each point in Figure 6 corresponds to one edit request: the x-axis measures the $\ell_2$ movement of the rewrite prompt, while the y-axis measures the average $\ell_2$ movement of its neighborhood prompts. Lower neighborhood movement indicates less collateral change in nearby but unrelated contexts.

At layer 4, FisherEdit produces a clearly smaller movement than AlphaEdit, indicating that the Fisher-weighted constraint already limits unintended representation drift at the first edited layer. At layer 8, the advantage becomes less pronounced but remains visible. This attenuation is expected, since later edited layers receive representations that have already been affected by preceding edited layers, making the hidden-space comparison noisier. Overall, the case study suggests that FisherEdit better controls hidden representation drift, especially at the early stage of the edited layer stack.

Weight-Space View of Regularization

To complement the projection analysis in § 3, we examine how AlphaEdit’s weight updates evolve during sequential editing. Figure 7 reports the total update magnitude per batch. For each edit batch, we aggregate the five edited layers by $\lVert\Delta_t\rVert_F = \left(\sum_{\ell}\lVert\Delta_t^{(\ell)}\rVert_F^2\right)^{1/2}$. It shows that AlphaEdit’s update scale is sharply controlled by $\lambda$ during sequential editing. With extremely weak regularization ($\lambda= 10^{-5}$), the update magnitude grows by many orders of magnitude as editing proceeds. At $\lambda= 10^{-4}$, the trajectory no longer shows sustained exponential growth, but it exhibits large transient spikes, suggesting a near-unstable boundary. For $\lambda\ge10^{-3}$, the update magnitude remains bounded throughout the edit stream. This weight-space view refines the main finding in § 3 that AlphaEdit’s behavior is governed primarily by regularization strength, while the near-identity projector does not by itself prevent unstable updates.

Full-Layer Visualization of the Null-Space Projector

Figure 8 summarizes the full-layer projectors for the edited backbones. Across layers, $P_{\tau}$ remains nearly diagonal and close to identity, indicating that AlphaEdit removes only a limited part of the activation space.

Derivation of Eqn. (6)

For the Fisher-weighted objective in Eqn. (5), define

Taking the derivative with respect to $\widetilde{\Delta}$ gives

Setting the derivative to zero and rearranging yields

Let $G=K_1K_1^{\mathsf{T}}+K_pK_p^{\mathsf{T}}$. Because the Fisher penalty is element-wise, this normal equation decomposes across rows. Let $\widetilde{\delta}_i$, $e_i$, and $f_i$ denote the $i$-th rows of $\widetilde{\Delta}$, $E$, and $F$, respectively, and define $D_i=\operatorname{diag}(f_i)$. The $i$-th row therefore satisfies

which gives the closed-form row-wise solution

At the optimum, $\widetilde{\delta}_i=\delta_i$, recovering Eqn. (6).

For the efficient implementation, concatenate the current and historical keys as $U = [K_1 \mid K_p] \in\mathbb{R}^{m \times r}$, with $r = n_1 + n_p$, so that $G = UU^{\mathsf{T}}$. The Woodbury identity gives

Substituting this expression into Eqn. (6) yields Eqn. (7), replacing the $m \times m$ inverse with diagonal scaling and an $r \times r$ linear solve.

Full Derivation of §4.4

In the following, we give the full derivation of the theoretical interpretation in §4.4. Specifically, we show that the element-wise Fisher regularizer in Eqn. (5) corresponds to a computable diagonal approximation of the second-order Taylor expansion of the locality KL objective around $\Delta= 0$, restricted to the edited parameter subspace.

Setup and Notation

Let $\mathbf{W}$ denote the pre-edit value of the edited parameter block and $\Delta$ the update. Following Eqn. (9), the single-sample locality cost is

and the reference-distribution objective is $\mathcal{L}_{\mathrm{loc}}(\Delta) = \mathbb{E}_{x \sim\mathcal{D}_{\mathrm{ref}}}[\ell_x(\Delta)]$.

According to Eqn. (23), we have

The first term depends only on the pre-edit distribution $p_{\mathbf{W}}(\cdot)$ and is therefore constant with respect to $\Delta$. Thus, we can write

where $\operatorname{const}(x) = \sum_y p_{\mathbf{W}}(y \mid x) \log p_{\mathbf{W}}(y \mid x)$. Therefore, only the cross-entropy term contributes derivatives with respect to $\Delta$.

Regularity Conditions

The derivation assumes the following standard regularity conditions on the model distribution:

• Smoothness. $p_{\mathbf{W}}(y \mid x)$ is twice continuously differentiable in the edited parameters $\mathbf{W}$.

• Parameter-independent support. The support $\{y : p_{\mathbf{W}}(y \mid x) > 0\}$ does not depend on $\mathbf{W}$. This holds for softmax outputs since every token has positive probability.

• Interchange of differentiation and summation/expectation. Derivatives with respect to $w$ can be interchanged with the summation over $y$ and the expectation over $x \sim\mathcal{D}_{\mathrm{ref}}$.

• Finite Fisher moments. $\mathbb{E}_{y \sim p_{\mathbf{W}}(\cdot\mid x)}\lVert\nabla_{\mathbf{W}} \log p_{\mathbf{W}}(y \mid x)\rVert^2 < \infty$, and the corresponding Hessian expectation exists.

Under these conditions, the information equality between the KL Hessian and the Fisher information holds [2].

The Taylor Expansion of Single-Sample Locality Cost

Next, we consider the Taylor expansion of Eqn. (25) around $\Delta= 0$. Let $u = \operatorname{vec}(\Delta)$. We have,

Zeroth order terms. Setting $\Delta= 0$ in Eqn. (23) gives $\ell_x(0) = \operatorname{KL}(p_W \mathbin{\|} p_W) = 0$.

First order terms. Differentiating Eqn. (25) with respect to $\Delta$ gives

Let $\overline{W} = W + \Delta$. Since $\operatorname{vec}(\overline{W}) = \operatorname{vec}(W) + \operatorname{vec}(\Delta)$, the Jacobian $\partial\operatorname{vec}(\overline{W})/\partial\operatorname{vec}(\Delta)$ is the identity. Therefore,

For $\Delta= 0$, by Eqn. (27) and Eqn. (28), we have

By swapping summation and differentiation, and using $\sum_y p_W(y \mid x) = 1$, we can derive

Therefore, we have $\nabla_\Delta\ell_x(0) = 0$.

Second order terms. Let $u := \operatorname{vec}(\Delta) \in\mathbb{R}^{md}$ for $\Delta\in\mathbb{R}^{d \times m}$. Treat $\ell_x$ as a scalar function of $u$ and expand it around $u = 0$. From the derivation above, both the zeroth-order and first-order terms vanish. Thus,

where

To compute $H_x$, we differentiate Eqn. (25) twice with respect to $u$. Since $p_W(y \mid x)$ and $\operatorname{const}(x)$ do not depend on $\Delta$, we have

Let $\overline{W} = W + \Delta$. As shown in Eqn. (28), $\operatorname{vec}(\overline{W})$ depends on $u = \operatorname{vec}(\Delta)$ through an identity Jacobian. Therefore,

Substituting this into the Hessian gives

Define the single-sample Fisher information matrix restricted to the edited block as

where

Under the conditions in Appendix § 14.§ 2, $\mathbf{H}_x = \mathbf{F}_x$. To see this, we differentiate $\nabla_{\operatorname{vec}(\mathbf{W})}\log p_{\mathbf{W}} = p_{\mathbf{W}}^{-1}\nabla_{\operatorname{vec}(\mathbf{W})}p_{\mathbf{W}}$ once more:

Multiplying Eqn. (38) by $p_{\mathbf{W}}(y \mid x)$ and summing over $y$ gives

The first term is zero because

For the second term, using $\nabla p_{\mathbf{W}} = p_{\mathbf{W}}\nabla\log p_{\mathbf{W}}$, we obtain

Therefore,

which is the information equality.

Averaging over the Reference Distribution

Substituting $\mathbf{H}_x = \mathbf{F}_x$ into Eqn. (31) and taking the expectation over $x \sim\mathcal{D}_{\mathrm{ref}}$,

where

is the Fisher information matrix restricted to the edited parameter subspace. This restriction is not cosmetic: the edit only perturbs the weights of a single MLP down-projection block, so the Taylor expansion lives on that subspace and $\mathbf{F}^{(S)}$ is the corresponding Fisher block rather than the full-model Fisher.

Diagonal Approximation and Recovery of Eqn. (5)

For $\Delta\in\mathbb{R}^{d \times m}$, the full block $\mathbf{F}^{(S)}$ has size $(md) \times(md)$. At LLM scale ($md$ on the order of $10^7$), storing $\mathbf{F}^{(S)}$ or solving the associated quadratic program in closed form is infeasible. We therefore adopt the standard diagonal approximation

and reshape $f$ into a matrix $\widetilde{\mathbf{F}} \in\mathbb{R}^{d \times m}$ of the same shape as $\Delta$. The quadratic form collapses to an element-wise penalty:

Combining Eqns. (43) and (46),

which is exactly the regularizer appearing in Eqn. (5).

Case Study

We present representative paired outputs from Ripple. Each example reports two prompts from the same edited sample: a paraphrase of the edited fact and a neighborhood prompt about a nearby fact that should remain unchanged. Across these paired examples, FisherEdit consistently produces the edited answer while preserving the unrelated local fact.

Case 1

Edit / Neighborhood. John Masly: occupation researcher $\to$ unemployed; gender should remain male. Outputs. FisherEdit: unemployed / male. AlphaEdit: unknown / unknown. AnyEdit: “John Masly is a historian.” / “John P Masly is male.” MEMIT: unrelated edit continuation / “male and female farmers, also genderfarmersqueepea”.

Case 2

Edit / Neighborhood. Sharifah Mazlina binti Syed Abdul Kadir: occupation explorer $\to$ music professor; citizenship should remain Malaysia. Outputs. FisherEdit: music professor / Malaysia. AlphaEdit: “music teacher and calligrapher” / Iran. AnyEdit: “Minister of Women’s Empowerment and Family Development in” / “citizen of Malaysia.” MEMIT: corrupted text on both prompts.

Case 3

Edit / Neighborhood. Anthony Deakin: occupation researcher $\to$ art dealer; alma mater should remain University of Liverpool. Outputs. FisherEdit: art dealer / University of Liverpool. AlphaEdit: art dealer / “known only to a few”. AnyEdit: “Royal Navy officer and politician.” / University of Sheffield. MEMIT: degenerate continuations on both prompts.

Future Work

Scaling to longer edit streams. An important direction is to extend FisherEdit to substantially longer editing sequences by improving the scalability of its solver and historical-key memory. In Eqn. (7), $r = n_1 + n_p$ counts both the current and accumulated historical edit keys, and the row-wise solve costs $O(dm r^2 + d r^3)$ per edited weight matrix. Since Algorithm 1 appends new keys after each batch, $n_p$ grows throughout the stream even when the batch size is fixed. The computational advantage of the Woodbury formulation relies on $r \ll m$, motivating further improvements to the row-wise solver as historical constraints accumulate. Future work could explore incremental updates to the row-wise factorizations and warm-started iterative solvers that reuse computations across successive edit batches, with the aim of reducing editing latency while preserving earlier edits.

Knowledge editing in reasoning models. FisherEdit’s parameter-importance regularization provides a natural basis for updating factual knowledge while protecting the reasoning processes that use it. The method requires locate-then-edit MLP locations and a Fisher diagonal estimated on a calibration set, without architectural modifications. The calibration set serves as an adaptable control knob: estimating the Fisher diagonal directly on complex reasoning traces, such as multi-step mathematical or logical trajectories, would penalize changes to parameters with high sensitivity on these traces. This offers a way to direct preservation toward reasoning-related behavior while allowing factual updates through less sensitive parameters. Future work can investigate this separation between factual recall and reasoning, and develop sequential-editing evaluations that jointly assess successful knowledge replacement, the use of updated facts in multi-step solutions, and preservation of reasoning ability.

Complementarity with retrieval and deployment scenarios. Parameter editing and retrieval-based approaches are complementary. When retrieved context conflicts with memorized knowledge, models may over-rely on stale parametric memory [24], and their use of external evidence depends on its coherence and persuasiveness [47]. Editing updates the fact at its source, so the correction can remain available when retrieval misses, ranks poorly, is only partially relevant, or is disabled. This motivates combining FisherEdit with retrieval-augmented generation or agentic search to improve consistency between parametric knowledge and external evidence. On-device, offline, and privacy-sensitive deployments also provide useful settings for exploration, particularly when maintaining a retrieval corpus and pipeline is impractical. Once applied, FisherEdit introduces no additional inference-time computation; latency- and cost-sensitive serving can therefore benefit from factual corrections that do not require a retrieval step on every query.