WorksComputer Science
Adaptive Localization of Knowledge Negation for Continual LLM Unlearning
Abstract
With the growing deployment of large language models (LLMs) across diverse domains, concerns regarding their safety have grown substantially. LLM unlearning has emerged as a pivotal approach to removing harmful or unlawful content while maintaining utility. Despite increasing interest, the challenges of continual unlearning, which is common in real-world scenarios, remain underexplored. Successive unlearning tasks often lead to intensified utility degradation. To effectively unlearn targeted knowledge while preserving LLM utility, it is essential to minimize changes in model parameters by selectively updating those linked to the target knowledge, thereby ensuring other knowledge remains unaffected. Building on the task vector framework, we propose a new method named ALKN (Adaptive Localization of Knowledge Negation), which uses dynamic masking to sparsify training gradients and adaptively adjusts unlearning intensity based on inter-task relationships. Comprehensive experiments across three well-established LLM unlearning datasets demonstrate that our approach consistently outperforms baseline methods in both unlearning effectiveness and utility retention under continual unlearning settings.
Introduction
Large Language Models (LLMs) exhibit exceptional capabilities in processing general knowledge, positioning themselves as a promising framework on the path towards artificial general intelligence ([27, 36, 3, 49]). However, reliance on extensive web data introduces risks of memorizing sensitive information, including private data, copyrighted material, and harmful content, hindering lawful deployments of LLMs ([37] (Ji et al. 2024)). These factors continuously spur growing interest in research on LLM unlearning, which aims to remove specified sensitive information from LLMs while retaining their general knowledge and capabilities, i.e., preserving model utility. It focuses on ensuring that the model cannot reproduce or accurately respond to target data associated with harmful outcomes or legal issues ([22]).
Although LLM unlearning has received much interest, an important problem of LLM continual unlearning remains underexplored. In real-world applications, unlearning requests from users or regulators often arrive continuously, requiring models to unlearn multiple pieces of data over time. For example, we consider a website equipped with a dedicated LLM that has been fine-tuned using user data from the platform, as shown in Figure 1. Periodically, users may request to delete their accounts and have their personal data removed, or regulators may report sensitive information in the LLM that needs to be removed. In these cases, the LLM needs to continuously perform unlearning tasks to ensure compliance with the requests from users and regulators.
Figure 1. An illustrative example of the LLM continual unlearning scenario. An LLM is integrated into a technical blog website, where some users occasionally close their accounts and request the deletion of their blog contents.
Maintaining model utility represents a critical challenge in LLM unlearning, which is particularly pronounced in the continual unlearning scenario [31]. Representative baseline approaches to LLM unlearning use gradient ascent optimization (GA) and its variants [12, 43], inherently affecting the overall utility of the model due to the extreme modification of model parameters. The issue of utility deterioration is further magnified in the context of continual unlearning, which can be attributed to two factors. (I) Each unlearning task contributes to a decline in the model utility of the LLM. And such decline accumulates progressively with an increasing number of unlearning tasks. We refer to this phenomenon as accumulative decline. (II) After completing preceding unlearning tasks, the memory retention of the LLM regarding data from subsequent tasks may be disrupted, making them “partially forgotten” even before they are explicitly unlearned. Employing a fixed unlearning intensity without adjustment for these partially forgotten tasks can easily result in over-unlearning, further exacerbating the decline in model utility, which we refer to as cascading degradation. While accumulative decline stems from the progressive loss of utility over multiple unlearning tasks, cascading degradation arises from the unintended over-unlearning of partially forgotten data, amplifying the utility loss in subsequent tasks. These two issues combined exacerbate the utility degradation problem, making continual unlearning in LLMs a significantly more challenging scenario compared to one-time unlearning.
In this paper, we propose ALKN (Adaptive Localization of Knowledge Negation) to mitigate the problem of utility degradation in LLM continual unlearning. ALKN is built on the task vector framework, which first fine-tunes a model to reinforce knowledge from unlearning data and then negates the target knowledge by subtracting the parameter offsets from the original model. [15]. Task vectors are trained with a lower-bounded loss, which inherently reduces the risk of excessive unlearning compared to GA. To further address the accumulative decline and cascading degradation issues in continual unlearning, we propose three key modules incorporated into the fine-tuning process of task vectors. (I) Entropic soft label loss strategically adjusts unlearning intensity via adaptive training objectives, reducing cascading degradation. (II) Dynamic gradient sparsity facilitates selective fine-tuning of different parameter sets for distinct tasks, minimizing accumulative decline. (III) Adaptive parameter modulation enables parameter-specific learning rates to further mitigate utility loss. In summary, the proposed modules, along with the task vector method, effectively address the catastrophic utility decline commonly observed in LLM continual unlearning by dynamically localizing model parameters and adaptively refining gradients.
We conducted comprehensive experiments to validate the effectiveness of our proposed method. To objectively assess the utility of models, we introduced Training Data Evaluation Corpus (TRAVIS) generated using the membership inference attacks (MIA) method [30], designed to reconstruct knowledge in pretraining data of LLMs across a diverse range of topics. Testing on diverse training data provides a more comprehensive assessment of the overall utility of LLMs and also highlights precise utility decline relative to their pre-unlearning performance [32]. Experiments are performed across multiple models and datasets, demonstrating that ALKN effectively retains the general utility of the model while achieving effective unlearning of sequential tasks. After sufficient unlearning, our method retains over 95% of its model utility, whereas some baseline methods completely lose their utility. We summarize our key contributions as follows:
We consider a practical yet challenging LLM continual unlearning setting, conducting theoretical and experimental studies on the unique issues of accumulative decline and cascading degradation in this context.
We introduce a novel method, ALKN, which adaptively modulates unlearning gradients to effectively minimize the utility degradation that is particularly severe in LLM continual unlearning scenarios.
To rigorously evaluate unlearning methods, we construct an evaluation corpus, TRAVIS, and perform extensive experiments across multiple benchmark datasets. The experimental results validate the effectiveness and demonstrate the superiority of ALKN compared to state-of-the-art baselines.
Problem Setup and Preliminary
In this section, we formulate the setting of continual unlearning, introduce the baseline methods GA and task vectors. We also demonstrate the intrinsic causes and negative impact of the cascading degradation issue.
LLM continual unlearning
LLM unlearning. This paper explores the concept of unlearning in causal LLMs. For a model , where denotes the model parameters, the probability of the -th token given the first tokens of an input text is computed by the model followed by a softmax function, formalized as . The objective of LLM unlearning is to eliminate the influence of a specified target dataset, , along with any related generative capabilities, from the model [1]. Sample may take the form of plain text or question-answer pairs. Crucially, the unlearning process needs to preserve the knowledge and utility of the model on other data.
LLM continual unlearning. In the context of LLM continual unlearning, the goal extends to managing a sequence of unlearning tasks. Beginning with an initial LLM characterized by pre-trained parameters , the model is required to sequentially unlearn datasets , where each corresponds to the -th unlearning task. During the training of a specific task, the data of other tasks is not accessible. LLM continual unlearning faces the issue of severe degradation of the model utility on non-target data during successive unlearning processes. To safeguard the general utility of the model throughout this iterative process, a retain set can be employed to mitigate unintended degradation of utility.
Gradient Ascent and Task Vectors
A straightforward approach to addressing the continual unlearning of LLMs is to directly apply existing LLM unlearning methods for each task, such as leveraging gradient ascent-based techniques or task vector methods.
Gradient Ascent. GA is one of the most fundamental fine-tuning-based approaches for unlearning and often serves as a baseline or a foundational module in many existing methods [43]. While the model acquires knowledge during training through gradient descent, GA facilitates the unlearning process by optimizing in the opposite direction. This is achieved by minimizing the negative cross-entropy loss of predicting the next token on the unlearning dataset:
For simplicity, the normalization of the loss by token length is omitted. Unlike gradient descent loss, loss of GA lacks a lower bound, which hinders its ability to converge. Training with GA could lead to significant deviations in model parameters from their pre-training values, which severely degrades model utility. The NPO method is also a gradient ascent-based approach. Compared to GA, the gradient in NPO is scaled by a regularization term that gradually diminishes over the course of training, alleviating the problem of over-unlearning to some extent [47].
Task Vectors. Introduced by Ilharco et al. [15], the task vector method manipulates the ability of a pre-trained model to address downstream tasks through fine-tuning and the arithmetic operation of model parameters. When applied to LLM unlearning, the method enables the selective removal of knowledge related to a specific dataset . Specifically, a pre-trained model is fine-tuned on the unlearning dataset , resulting in a model with enhanced knowledge of . The difference between the fine-tuned parameters and the original parameters, , constitutes the task vector for . By subtracting this task vector from the original parameters , a modified model is obtained that has unlearned the knowledge from . Additionally, a scalar is introduced to control the magnitude of the task vector:
We theoretically analyze the convergence of the GA and task vector method in Appendix § 9. In the case of logistic regression unlearning, GA maintains a gradient magnitude above a positive constant and fails to converge. In contrast, the task vector method is proven to converge, indicating that it is less prone to over-unlearning and may cause less utility degradation in LLMs. While the task vector method mitigates utility loss to some extent during a single unlearning task, it remains susceptible to accumulative decline and cascading degradation issues in continual unlearning scenarios.
Cascading Degradation
LLM continual unlearning encounters more complex challenges than one-time unlearning, one of which is cascading degradation, where the interaction between unlearning tasks amplifies the utility degradation. To elaborate, the utility decline during preceding unlearning tasks can also impair the memory retention of the LLM regarding the data of subsequent tasks. Besides, in real-world applications, unlearning data is often highly homogeneous, with requests involving similar content.As seen in Figure 1, data from different users may share similar structures and content. This homogeneity intensifies the impact of preceding unlearning tasks on subsequent tasks. Consequently, the model partially forgets data of subsequent tasks before explicit unlearning, yet these tasks still need to be performed. Unlearning such partially forgotten data with the same optimization process as preceding tasks can easily lead to over-unlearning, amplifying the utility degradation. We refer to this phenomenon as cascading degradation.
To empirically validate the causes and harm of the cascading degradation, we conducted experiments using a baseline approach, NPO+RT on the TOFU dataset, where the LLM model was made to sequentially unlearn five tasks. As shown in Figure 2(a), we examined the model’s prediction probabilities on the data of the -th task before unlearning it (▮ Probability After Previous Unlearning). The results reveal a substantial decrease in these probabilities compared to their initial values (▮ Probability Before Any Unlearning), indicating that the model had already partially forgotten the -th task after unlearning the first tasks. To validate the detrimental effects of cascading degradation, we compared continual unlearning with independent Figure 2. The comparative experiments on the TOFU dataset using the baseline method to sequentially unlearn five disjoint subdatasets. (a) The unlearning of preceding tasks results in the partial forgetting of subsequent tasks, as reflected in the reduced conditional probabilities of task data predicted by the model. (b) Under comparable unlearning performance, continual unlearning exhibits significantly greater utility degradation than independent unlearning due to the effects of cascading degradation.
Figure 2. (a) Prediction probability under three conditions and model utility.
Figure 2b. (b) Rouge-F and utility change under two unlearning scenarios.
unlearning, where the model independently unlearns the data of each task starting from pre-trained parameters, as shown in Figure 2(b). The utility changes in the figure denote utility declines in one unlearning task. We ensure that the level of unlearning (Rouge-F) achieved by independent unlearning is similar to that of continual unlearning, enabling a fair comparison of the changes in model utility. Under equivalent conditions, continual unlearning with cascading degradation leads to a greater and increasingly severe decline in model utility compared to independently unlearning each task. We also compare the performance of joint unlearning with continual unlearning in Appendix § 15 to further validate the harm of cascading degradation.
To theoretically substantiate the above conclusion, we consider a toy example where a logistic model performs unlearning with the GA algorithm on a binary classification problem with inputs being and labels being . Suppose there are two unlearning tasks with their dataset being correlated. Our objective is to verify whether the first unlearning task influences the second task.
Proposition 2.1. Consider two correlated datasets and , each composed of , satisfying: , and , where is a positive constant. The logistic model starts with an initial parameter . We compare two optimization scenarios:
The model successively unlearns on and , yielding intermediate and final parameters: and .
Starting from , the model directly unlearns on , yielding parameters .
The parameter changes in the two scenarios during unlearning on satisfy:
where is a positive constant depending on the datasets, is iteration steps, and is the number of samples.
Please refer to Appendix § 10 for the proof. Proposition 2.1 demonstrates that during the process of continual unlearning, conducting a preceding unlearning task causes subsequent related tasks to experience larger parameter changes in their own unlearning. In the context of LLMs, this phenomenon may exacerbate the decline in model utility.
Method
In continual unlearning for LLMs, there is severe deterioration in model utility. To effectively mitigate the utility decline, the key is reducing the overall changes to the model’s parameters under the condition of effective unlearning. Utilizing the task vector method can mitigate utility decline to some extent, but the accumulative decline and cascading degradation issues still exist. Therefore, we further propose three modules to alleviate these issues. Entropic soft label loss utilizes generated labels to flexibly adjust the unlearning objective. Dynamic gradient sparsity applies a learnable mask to the gradients and encourages different tasks to localize distinct model parameters. Adaptive parameter modulation employs a lower learning rate for parameters related to already unlearned tasks. These modules collaboratively address the challenges of accumulative decline and cascading degradation by mitigating task interference and adaptively regulating the intensity of unlearning. The diagram of these modules is shown in Figure 3. The overall procedure of training is in Algorithm 1.

Figure 3. The diagram illustrating the training process of the intermediate model for the -th unlearning task. The training loss is computed using entropic soft labels § 3.1. Gradients derived from the loss are masked using a binary mask with an underlying vector , dynamically updated based on , and § 3.2. The masked gradients are then employed to update the model parameters with an adaptive learning rate § 3.3.
Fine-tuning with Entropic Soft Labels
To perform the -th unlearning task, we employ knowledge negation from the current model with the task vector method. This process begins by initializing a model with parameters , which is then fine-tuned on the unlearning dataset . The fine-tuning step yields the intermediate model , reinforcing the knowledge of the model specific to the -th dataset [15]. Subsequently, the knowledge associated with the -th task is removed using model weight arithmetic: .
The fine-tuning process for task vector training typically leverages gradient descent optimization with a cross-entropy loss, which quantifies the discrepancy between the next-token prediction labels and the model’s output probability distribution conditioned on preceding tokens:
where . In the remainder of this paper, is denoted as for simplicity.
However, employing cross-entropy loss in continual unlearning scenarios poses the risk of over-unlearning. As demonstrated in § 2.3, interaction between unlearning tasks could result in a task being partially forgotten even before its data is explicitly unlearned. The predicted probability for the data of the task gets notably low. In such cases, the cross-entropy loss, which enforces alignment between the output of the model and a one-hot label in the token space, produces gradients as follows:
When is low, the gradient magnitude may increase, causing large parameter updates that negatively impact the utility of the LLM. This issue is theoretically illustrated in Proposition 2.1. Besides, one-hot labels in cross-entropy loss are unnecessarily difficult objectives to optimize towards since the goal of task vector fine-tuning is only to enhance the knowledge of the model regarding the target data. Training with one-hot labels forces redundant parameter adjustments of the LLM model, leading to significant utility decline.
To mitigate this issue, we propose Entropic Soft Labels (ESL) to preserve model utility during task vector fine-tuning. We introduce soft labels that increase the entropy of the target distribution in the loss. The soft label is derived from the initial predicted probability for a training sample and is defined as , where is the sigmoid function and is a scaling factor that controls the entropy of labels. By leveraging these soft labels, the cross-entropy loss is reformulated as:
where represents the soft label for the next token. mitigates the cascading degradation issue by generating adaptive optimization goals for partially forgotten data.
Dynamic Gradient Sparsity
In continual unlearning scenarios, we aim to fine-tune different sets of model parameters for each task. This approach helps prevent cumulative parameter divergence from their initial values as the number of tasks increases, thereby reducing the risk of accumulative decline. Moreover, minimizing changes to model parameters during unlearning is essential for maintaining utility. We strive to identify parameters that significantly influence the unlearning objective and focus on adjusting these parameters during training. This would enable effective unlearning while making as few adjustments to the model parameters as possible.
To achieve these goals, we propose to apply learnable masks to model gradients during the fine-tuning of the model on the unlearning set , allowing for precise and efficient parameter adjustments. Inspired by von Oswald et al. [34], for the -th task, the optimization of the model parameters is given by:
where represents a dynamic underlying vector, from which a binary mask is derived using the threshold function . This function outputs 1 for parameters where the corresponding value of exceeds the threshold ; otherwise it outputs 0. The threshold is determined by the percentile of and increases gradually throughout the training process. Additionally, denotes the adaptive learning rate, which is detailed in § 3.3.
Mask Value Updating. The application of gradient masking is designed to achieve two purposes: (I) To encourage different tasks to adjust distinct sets of model parameters during unlearning, thereby reducing the overall parameter changes. (II) To dynamically identify model parameters that are important to the unlearning objective while also being beneficial to utility preservation. To achieve these goals, the underlying vector is dynamically learned from the data, allowing the mask to adapt to task-specific requirements during training. For objective (I), the current mask is encouraged to differ from the masks used in previous tasks. Specifically, let represent the set of parameters that underwent substantial adjustments in the first tasks. The mask for the current task is encouraged to minimize its overlap with . For objective (II), the mask is refined to preserve model utility. The cross-entropy loss on is used as a proxy to evaluate the retained utility of models. The underlying vector of the mask is optimized to minimize this utility loss. The sum of these two objectives with a scaling factor is given by:
By substituting into Equation (7) and differentiating this objective with respect to , we obtain the iterative update rule for :
where
The derivation is detailed in Appendix § 11. According to the update rule of , the elements will increase where the gradient signs are consistent across both the unlearning objective and the utility objective. And the corresponding parameters will be selectively activated. In other words, the proposed method selectively activates model parameters, aligning the gradient directions of the unlearning and utility preservation objectives. Furthermore, elements that have a greater impact on both objectives—i.e., those with larger gradient magnitudes—lead to faster changes in the corresponding values of , resulting in quicker activation or freezing of the model parameters.
The proposed method localizes distinct parameters for different unlearning tasks. By avoiding cumulative changes to parameters, we alleviate the accumulative decline in utility of the LLM. And the gradient sparsity helps preserve the overall model utility, further mitigating the issue.
In contrast to methods that rely on static model parameter masks, our proposed approach dynamically adjusts the mask throughout the training process. This dynamic tuning ensures that each model parameter is trained within an appropriate range: parameters with minimal impact on the unlearning objective undergo limited adjustments in the early stages and are subsequently masked, while parameters with significant impact receive more extensive training. This strategy enables balanced training across the model, preventing excessive adjustments to all parameters or overly drastic changes to a small subset. Moreover, our method leverages the underlying vector to aggregate the influence of data on the mask over time, mitigating biases that may arise from extreme values in a single training iteration. By accumulating data-driven insights, the mask adapts more robustly to task-specific requirements, enhancing the stability and effectiveness of the unlearning process.
Adaptive Parameter Modulation
To further mitigate the decline in model utility, we propose adaptive parameter modulation during fine-tuning . The gradient masks obtained from our training process represent the relationship between the model parameters and the features of each task. Consequently, during the training of the current task, we apply a reduced learning rate to the parameters previously activated, thereby preventing the re-unlearning of features already forgotten and alleviating the cascading degradation of utility. Specifically, let denote the model parameters that have undergone significant adjustments in the first tasks. We apply an adaptive learning rate to model parameters while fine-tuning on the -th task as follows:
where denotes a low learning rate for model parameters that are crucial for previous tasks and denotes a relatively high learning rate.
Experiments
We conduct extensive experiments on various datasets and models to validate the effectiveness of the proposed method.
Experimental Settings
Datasets and models. We conduct experiments on three datasets: TOFU, MUSE News, and WHP, each representing scenarios relevant to real-world applications such as privacy preservation and copyright protection. To simulate a continual unlearning setting, we divide the data designated for unlearning into multiple disjoint subsets based on the structure of the datasets. These subsets, either from different parts of the same dataset or a mix of datasets, are treated as sequential unlearning tasks. The experimental details for each dataset are as follows. 1) TOFU ([25]) dataset consists of generated fictional author information. We group the information of four authors into one unlearning task, resulting in five or more continual unlearning tasks. 2) MUSE News ([31])
Algorithm 1 Continual unlearning method ALKN
Input: Sequential unlearning datasets , retain set , initial model parameters , hyperparameters and , learning rates and , updating steps for each unlearning task.
Initialize the overall mask .
for to do
Initialize parameters .
Initialize the underlying vector .
Calculate .
for to do
Update model parameters with Equation 7.
Update the underlying vector with Equation 9.
Tune with percentiles of as in Appendix E.2.
end for
.
.
end for
Output: Final parameters .
comprises BBC news articles and includes four predefined unlearning subsets that serve as continual unlearning tasks. 3) WHP (Who’s Harry Potter) ([7, 31]) involves the unlearning of original text from the Harry Potter series. We treat the original text of three Harry Potter books as three separate unlearning tasks. Following recent works ([25, 18]), we use Llama-2-7B and Phi-1.5 as target models.
Metrics. Unlearning algorithms require evaluation from two perspectives: unlearning performance and model utility, both measured using various metrics. Metrics for unlearning performance include: 1) Rouge-L for unlearning (F-Rouge) measures similarity between the text generated by the model and the correct text ([25, 31]). 2) Probability (F-Prob) evaluates the conditional probability of model outputs. 3) Membership inference attack (MIA) calculates Min-K% Prob ([30]) for unlearning samples to detect whether the post-unlearning model retains the unlearning set. 4) Forget Quality (FQ) quantifies the difference in prediction distributions on the forget set between the post-unlearning model and a retrained model ([25]).
Metrics for model utility include: 1) R-Rouge and 2) R-Prob are also applied to the retain set or holdout set to evaluate the model utility. 3) T-Rouge calculates the Rouge-L recall of completing sentences in the proposed TRAVIS dataset. Testing on inferred training data provides a more accurate assessment of the impact of unlearning algorithms on the pre-existing knowledge of the model. Additionally, TRAVIS is generated without thematic constraints, resulting in a broad coverage of content, which enables a more comprehensive evaluation of performance. 4) Model Utility (MU) represents the harmonic mean of the above metrics calculated across multiple retain sets in the TOFU dataset ([25]), including the T-Rouge. It provides an aggregate measure of the model utility.
Baselines. Given the limited research on LLM continual unlearning, the proposed method is primarily compared with prominent existing unlearning approaches. These algorithms are applied sequentially to unlearn the data for each task, thereby achieving continual unlearning. Unlearning methods include GA ([43]), NPO ([47]), and Task Vectors (TV) ([15]), as introduced earlier. We also include DPO ([47]), SO-PPO, and SO-NPO ([19]) methods. Besides, EUL ([6]) incorporates lightweight unlearning layers into the model and explicitly considers scenarios involving multiple unlearning tasks. WAGLE ([18]) optimizes only the parameters that are crucial to unlearning and retaining objectives during the unlearning process. When applying these methods in continual unlearning scenario, the learning rate is tuned progressively to mitigate catastrophic collapse as the number of tasks increases. Retaining objectives are also leveraged with an auxiliary retain dataset to help preserve the model utility in addition to conducting unlearning optimization on the target data. For example, the Retain Loss (RT) is employed as , where gradient descent is performed on the retain dataset to optimize the model. Similarly, KL divergence loss (KL), defined as , is used to ensure that the logits of the model outputs on the retain dataset remain similar to those of the initial model ([47]).
Experiments on Privacy-Preservation Unlearning
In the TOFU dataset, unlearning algorithms are tested with personal information of fictitious authors, where each unlearning task corresponds to different authors. Table 1 presents the unlearning and utility performance of various methods across five continual tasks, showing that utility declines to varying degrees as tasks increase. A Rouge score of approximately 0.35 on the unlearning set is considered sufficient for effective unlearning ([25]). NPO+RT achieves reasonable unlearning performance in the first four tasks but suffers over-unlearning in the last, with model utility dropping below 0.2. Conversely, GA+RT struggles with insufficient unlearning in earlier tasks, followed by over-unlearning later. NPO, EUL, and WAGLE exhibit severe under-unlearning, with F-Rouge reductions of less than 15% in the first task compared to the original model. Although NPO and EUL achieve the highest utility in the third and fourth tasks, this stems from insufficient unlearning. GA+KL, NPO+KL, and SKU avoid over-unlearning but suffer progressively lower utility. In contrast, our method effectively balances unlearning and utility, achieving sufficient unlearning while minimizing the decline in model utility. Notably, it not only achieves the best performance across all metrics on the first two tasks but also surpasses baseline methods from the perspective of multiple tasks. After completing all five unlearning tasks, our method exhibits less than a 0.06 reduction in utility, effectively mitigating the cumulative utility degradation typically observed in continual unlearning scenario. This highlights its significantly lower impact on utility compared to other approaches.
| Methods | Task 1 F-Rouge↓ | Task 1 FQ↑ | Task 1 MU↑ | Task 2 F-Rouge↓ | Task 2 FQ↑ | Task 2 MU↑ | Task 3 F-Rouge↓ | Task 3 FQ↑ | Task 3 MU↑ | Task 4 F-Rouge↓ | Task 4 FQ↑ | Task 4 MU↑ | Task 5 F-Rouge↓ | Task 5 FQ↑ | Task 5 MU↑ |
| Original | 0.9614 | 3.2e-16 | 0.5237 | 0.9723 | 7.4e-16 | 0.5237 | 0.9827 | 4.2e-15 | 0.5237 | 0.9895 | 7.5e-16 | 0.5237 | 0.9847 | 5.0e-17 | 0.5237 |
| GA+RT | 0.5723 | 7.9e-14 | 0.4766 | 0.4047 | 9.5e-13 | 0.4427 | 0.3524 | 1.3e-4 | 0.3494 | 0.3095 | 6.5e-5 | 0.2253 | 0.1027 | 9.7e-2 | 0.0812 |
| GA+KL | 0.3963 | 7.8e-4 | 0.3955 | 0.3896 | 1.5e-5 | 0.4270 | 0.3923 | 9.1e-3 | 0.3578 | 0.4311 | 5.2e-5 | 0.2898 | 0.3108 | 5.9e-4 | 0.2059 |
| NPO | 0.8430 | 1.9e-16 | 0.4836 | 0.7322 | 4.6e-15 | 0.4675 | 0.6091 | 8.5e-9 | 0.4651 | 0.4893 | 5.4e-9 | 0.4257 | 0.4033 | 8.9e-6 | 0.3817 |
| NPO+RT | 0.3803 | 9.4e-3 | 0.4823 | 0.3931 | 7.6e-5 | 0.4470 | 0.3747 | 4.8e-4 | 0.3814 | 0.3524 | 6.8e-3 | 0.3087 | 0.3234 | 7.9e-3 | 0.1841 |
| NPO+KL | 0.4109 | 6.0e-5 | 0.4729 | 0.4359 | 1.0e-6 | 0.4465 | 0.4018 | 6.3e-4 | 0.3957 | 0.3891 | 9.9e-6 | 0.3350 | 0.3651 | 8.5e-5 | 0.2789 |
| DPO+RT | 0.4510 | 8.9e-10 | 0.5043 | 0.4091 | 4.3e-8 | 0.4432 | 0.3983 | 4.6e-5 | 0.3602 | 0.3821 | 3.1e-4 | 0.3406 | 0.3771 | 3.9e-3 | 0.3359 |
| TV | 0.3957 | 4.8e-4 | 0.4825 | 0.3719 | 5.2e-3 | 0.4578 | 0.3573 | 8.9e-3 | 0.3866 | 0.3371 | 9.8e-3 | 0.3253 | 0.3028 | 4.2e-2 | 0.2963 |
| SKU | 0.4279 | 3.0e-5 | 0.4927 | 0.4055 | 5.8e-4 | 0.4100 | 0.3839 | 6.0e-4 | 0.3330 | 0.3805 | 6.5e-5 | 0.3024 | 0.3346 | 7.2e-3 | 0.2216 |
| EUL | 0.7885 | 2.4e-15 | 0.4839 | 0.5562 | 2.9e-12 | 0.4662 | 0.5264 | 7.2e-12 | 0.4551 | 0.5171 | 1.6e-13 | 0.4537 | 0.5058 | 8.0e-12 | 0.3938 |
| SO-NPO | 0.4525 | 7.3e-8 | 0.4935 | 0.4316 | 6.2e-6 | 0.4532 | 0.4168 | 9.9e-7 | 0.3725 | 0.4153 | 5.0e-5 | 0.3472 | 0.4002 | 4.1e-4 | 0.3001 |
| WAGLE | 0.7667 | 2.0e-15 | 0.4731 | 0.5031 | 4.8e-9 | 0.4669 | 0.4593 | 7.7e-6 | 0.4314 | 0.3860 | 2.5e-5 | 0.3400 | 0.3731 | 3.5e-4 | 0.3013 |
| Ours w/o ESL | 0.3627 | 3.5e-2 | 0.5016 | 0.3320 | 3.7e-2 | 0.4413 | 0.3067 | 7.3e-2 | 0.4170 | 0.2684 | 1.4e-1 | 0.3807 | 0.2344 | 5.7e-1 | 0.3351 |
| Ours w/o DGS | 0.3827 | 5.4e-3 | 0.5048 | 0.3642 | 1.6e-3 | 0.4559 | 0.3536 | 6.2e-3 | 0.4437 | 0.3297 | 4.0e-3 | 0.4296 | 0.3073 | 7.9e-1 | 0.3893 |
| Ours w/o APM | 0.3703 | 1.4e-2 | 0.5128 | 0.3464 | 8.2e-3 | 0.4784 | 0.3320 | 4.7e-2 | 0.4524 | 0.3043 | 7.3e-2 | 0.4367 | 0.2517 | 2.5e-1 | 0.3768 |
| Ours | 0.3703 | 1.4e-2 | 0.5128 | 0.3671 | 2.0e-3 | 0.4853 | 0.3681 | 2.1e-3 | 0.4627 | 0.3452 | 1.3e-4 | 0.4578 | 0.3314 | 1.1e-2 | 0.4534 |
Table 1. Unlearning performance (F-Rouge, FQ) and model utility (MU) on the TOFU dataset for the Phi model.
Figure 4. Model utility (MU) and changes of F-Rouge of five unlearning tasks on TOFU dataset for the Llama2 model.
Ablation study. We present the ablation study results in Table 1, illustrating the impact of each module on the overall performance of the model. When the entropic soft label (ESL) module is removed, the model exhibits significant cascading degradation, resulting in severe utility decline during the unlearning of later tasks. The dynamic gradient sparsity (DGS) module plays a critical role in identifying model parameters that balance utility retention and effective unlearning. Without the DGS module, the overall model utility decreases, and the cumulative utility degradation becomes more pronounced. Similarly, the absence of the adaptive parameter modulation (APM) module leads to a sustained decline in utility over the course of unlearning.
Figure 4 illustrates the unlearning results for the Llama2 model on the TOFU dataset. The persistent issue of utility degradation is evident, as the data points of the same color representing the performance of successive tasks are generally arranged sequentially from up to down, reflecting the cumulative effect of unlearning tasks on model utility. For the latter tasks, NPO+RT and GA+RT suffer a drastic decline in model utility. While WAGLE exhibits poor unlearning effectiveness with low changes of F-Rouge. Notably, the data points corresponding to our method are clustered more tightly across the five tasks, indicating a more stable and controlled utility decline in the continual unlearning scenario. Furthermore, these points are positioned nearer to the upright corner of the plot compared to other methods, signifying a superior trade-off between effective unlearning and maintaining model utility. By achieving a high degree of unlearning with minimal impact on utility, our approach demonstrates clear advantages over baseline methods.
Experiments on Copyright-Protection Unlearning
The WHP and MUSE News datasets are designed to evaluate the unlearning of verbatim memory of news articles and Harry Potter books by LLMs. We conducted experiments with the Llama2 model independently on each dataset and further explored the effects of alternating tasks between the two datasets. Please refer to Appendix § 15 for more results.
The results on the WHP dataset are shown in Table 2 and Figure 7 (b). Utilizing the TRAVIS dataset provides the evaluation of general utility wider than domain knowledge in retain dataset. In the WHP dataset, the sequential tasks involve unlearning data from several books in the Harry Potter series. Due to the strong interconnections between the data across these tasks, unlearning earlier tasks inevitably impacts subsequent tasks, leading to pronounced cascading degradation. For baseline methods, model utility declines largely after completing the unlearning. In contrast, our method effectively mitigates the cascading degradation issue, demonstrating improved preservation of model utility in continual unlearning scenarios.
| Methods | Unlearning | Utility | ||
| F-Rouge | MIA | R-Rouge | T-Rouge | |
| Original | 0.9821 | 0.5206 | 0.6940 | 0.7643 |
| NPO+RT | 0.1559 | 0.2607 | 0.0 | 0.0 |
| NPO+KL | 0.1620 | 0.2452 | 0.0 | 0.0 |
| GA+RT | 0.1855 | 0.2674 | 0.1172 | 0.1975 |
| SO-NPO | 0.3080 | 0.3892 | 0.4109 | 0.4536 |
| WAGLE | 0.2946 | 0.3745 | 0.3721 | 0.4594 |
| Ours | 0.1790 | 0.2729 | 0.4586 | 0.5617 |
Table 2. Final model utility metrics and average unlearning performance metrics over three tasks on the WHP dataset.
Figure 7. (a) R-Rouge across four continual unlearning tasks on the MUSE News dataset. (b) T-Rouge across three continual unlearning tasks on the WHP dataset.
Evaluation of TRAVIS Dataset Sensitivity
The TRAVIS dataset consists of a wide variety of topics, enabling a comprehensive evaluation of the model’s overall utility. As the dataset is generated by the LLM, it provides an accurate baseline for evaluating the model’s original utility. To investigate the sensitivity of the TRAVIS dataset to variations in model performance, we conducted a synthetic experiment by introducing a small amount of random Gaussian noise to the model parameters to simulate destruction of model utility. We compared the Rouge scores of the TRAVIS dataset against those of the TOFU and WHP datasets to assess their relative sensitivity to model utility changes. As presented in Table 3, the TRAVIS dataset is the most sensitive to changes in model utility. Notably, the TRAVIS Rouge score exhibits a decline when the noise standard deviation reaches 0.5%, whereas the TOFU and WHP datasets remain largely unaffected until higher noise levels.
| Noise Std(%) | TRAVIS | TOFU | WHP |
| Original | 0.7643 | 0.8965 | 0.6940 |
| 0.1% | 0.7643 | 0.8965 | 0.6940 |
| 0.5% | 0.7576 | 0.8965 | 0.6940 |
| 1.0% | 0.7425 | 0.8965 | 0.6940 |
| 2.0% | 0.7320 | 0.8958 | 0.6940 |
Table 3. Rouge Score Sensitivity to Gaussian Noise in Model Parameters.
Conclusion
In this study, we explore the challenges of continual unlearning in LLMs, which is a practical and complicated setting. Specifically, the issues of accumulative decline and cascading degradation exacerbate the utility deterioration problem for LLM unlearning methods. To mitigate these issues, we propose ALKN, a method that minimizes adjustments to model parameters by identifying the model parameters that are crucial to the knowledge representation of each task and tuning these parameters with adaptive intensity. The proposed method is built on the task vector framework with three novel modules applied in the fine-tuning process, dynamically localizing model parameters and adaptively refining gradients. Besides, we construct an evaluation corpus, TRAVIS, designed to test the overall utility of LLMs comprehensively. The experimental results demonstrate that the proposed method effectively preserves the utility of LLMs while accomplishing sufficient unlearning, outperforming various baseline methods.
Acknowledgments
This work is funded by the National Science and Technology Major Project (No. 2022ZD0114903) and the Natural Science Foundation of China (NSFC. No. 62176132). Sen Cui would like to acknowledge the financial support received from Shunmu Tsinghua scholar program. Qizhou Wang and Bo Han were supported by the NSFC General Program No. 62376235, GDST Basic Research Fund Nos. 2022A1515011652 and 2024A1515012399. This research was partially supported by Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo.
Impact Statement
We propose a novel method for continual unlearning in large language models that is designated to protect personal privacy and copyright. It can also be utilized to prevent harmful responses from LLMs. For similar reasons, the proposed method also does not involve storing training data. The constructed corpus TRAVIS is generated by the Phi model and the Llama2 model, only presenting general knowledge. It is manually filtered to guarantee no private or sensitive data is included.
Related Work
Machine unlearning studies the problem of efficiently removing specific data from machine learning models, ensuring compliance with privacy regulations like GDPR [4, 39, 46, 26, 48]. Machine unlearning methods are generally categorized into exact unlearning [33, 2] and approximate unlearning. The latter modifies the model or dataset to simulate the removal effect without full retraining [8, 9, 10] (Wu et al. 2020). There is a line of work that employs Newton’s method to remove the influence of target data involving calculating second-order derivatives of model parameters [20, 11, 16]. However, the approach is computationally prohibitive for LLMs. In addition, precisely evaluating LLM unlearning algorithms, as is commonly done in the field of machine unlearning, is often impractical because obtaining a fully retrained model for comparison is rarely feasible [29].
Among the studies on LLM unlearning, some focus on establishing evaluation metric [35, 38, 42, 24, 45] (Wu et al. 2024), while others emphasize the design of effective algorithms. With trivial unlearning methods, the process often leads to excessive unlearning, where the performance of the model on non-target data is unintentionally degraded. The primary objective of LLM unlearning is to preserve the model utility on normal data while effectively unlearning the target information. NPO is proposed to mitigate the issue of unbounded optimization loss of GA [47]. Eldan & Russinovich [7] suggest replacing sensitive keywords with generic terms, enabling the model to produce generic predictions through reinforcement bootstrapping. The constructed generic outputs enable unlearning by fine-tuning the LLM model on them. Ji et al. (2024) propose ULD, which reverses the optimization direction during the training of an assistant model. SKU incorporates an object function that embeds robustness and utility retention into task vector training [23], offering a more efficient approach to unlearning. WAGLE seeks to locate the specific LLM parameters that are most critical for both unlearning and retaining objectives and fine-tuning exclusively on these parameters [18]. Several studies have also examined the impact of unlearning techniques on various modules within transformer models [14]. Despite the substantial body of research on LLM unlearning, the practical challenge of continual unlearning remains underexplored and warrants further investigation.
Convergence of GA and the task vector method
To compare the convergence of GA and the task vector method, we consider a standard binary classification problem with training data , where and . A logistic model is used to predict the probabilities , where represents the sigmoid function [28].
Convergence of the task vector method
Theorem B.1. Using the task vector algorithm with a learning rate to optimize the aforementioned logistic regression problem can guarantee convergence. Let denotes the parameters after the -th iteration of optimizing. The loss function strictly decreases:
and converges to its optimal value:
Proof. The fine-tuning process of the task vector method utilizes the gradient descent algorithm with the following loss function:
The function is L-smooth. Thus the following inequality holds:
Substituting yields:
With , we get a strict decrease in :
With lower bound , the loss function converges to its optimal value:
Non-convergence of the GA method
Theorem B.2. Optimizing the aforementioned logistic regression problem using the GA algorithm will result in the norm of the gradient remaining greater than a certain constant:
where denotes the parameters after the -th iteration of optimizing and is a positive constant only depending on the training data and initial parameters . And the objective function strictly decreasing to negative infinity:
Proof. The loss function of GA is:
The gradient with respect to the parameters is:
We define , , and denotes . Besides, we define .
Inspired by Zhang et al. [47], we use the induction method to proceed.
Case 1: . Since is a finite value, we can derive:
where and are two positive constants. We assume there exists a constant such that , and satisfying:
We obtain:
We set and , then we have:
Then we derive :
Substracting , we obtain:
Comparing it with , we can derive:
Case 2: . We suppose:
for . And we suppose:
for . With this condition, we can derive the same conclusion for as in Case 1:
and the same conclusion for :
From above, we prove that and for all . We can further derive that:
and
By incorporating the condition , we can obtain:
where is a positive constant only depending on the training data and initial parameters .
Furthermore, since is a concave function, it satisfies the following inequality:
then , and we can obtain:
Proof of Proposition 2.1
As defined for the GA, the loss function and its gradient are given as follows:
where follows the definition in Appendix § 9.§ 2, as do , and .
Independently training on .
In this scenario, the model is directly trained on starting from the initial parameter for iterations with a learning rate , resulting in the updated parameters :
As in Appendix § 9.§ 2, we can prove that:
Continuously training on and .
In this scenario, the model is first trained on starting from , yielding parameters :
The training on affects the model predictions on :
Because , we assume there exists a constant such that , and satisfying:
Therefore, we can obtain:
Thus the effect of training on demonstrates as:
Then the model starts training on with initial parameters being , yielding updated parameters :
Since the inequality in Equation (45), we can derive:
Using the induction method as in Appendix § 9.§ 2, we can prove that:
Therefore, .
Finally, we derive the conclusion:
therefore,
In this proposition, we consider a two-task scenario for simplicity. When there are multiple tasks, we can also similarly derive such conclusions:
Corollary C.1. For multiple tasks, the conclusion of Proposition 2.1 holds. Suppose there are preceding datasets and a current dataset . If one of the preceding datasets is correlated with , satisfying: , then the previous unlearning results in larger changes of parameters in the unlearning of :
Derivation of Equation 9
According to the chain rule, the gradient of Equation (8) with respect to is given by:
For computational efficiency, we approximate the gradient of on with . And is derived by substituting into Equation (7):
For simplicity, we omit and approximate using straight-through estimation, which consists in taking this derivative equal to the identity (von Oswald et al., [34]). Then we obtain the gradient of Equation (8) with respect to :
The updating of is given by substracting the gradient:
Implementation details
Optimization
The Adam optimizer is employed for the training of all methods. We employ the learning rate of 3e-5 for the TOFU dataset and 3e-6 for the MUSE News and WHP datasets. Most baselines easily suffer from the issue of over-unlearning while conducting second or third tasks with constant learning rate. Thus, we apply learning rate tuning, decaying the learning rate by 50% after each unlearning task. For experiments in Figure 2 and Figure 8 which require alignment of unlearning performance, learning rates of different methods are tuned to achieve aimed unlearning performance. The hyperparameters and of our method are set as 10 and 0.8. is set as the deviation of the unlearning gradients . is initialized with zero vectors in the beginning of each training.
Figure 8. Rouge-F and model utility under two unlearning scenarios.
Weight decay ratio is set as 0.01 for all experiments. For TOFU dataset, batch size is set as 32 and models are trained for 5 epochs. For MUSE News dataset, batch size is 6 and number of epochs is 2. For WHP dataset, batch size and epoch number are 6 and 10.
Tuning of gradient mask threshold
The threshold determines the proportion of model parameters that are activated. The threshold is determined by the percentiles of the underlying vector , and it increases gradually throughout the training process. Specifically, 100% parameters are activated in the beginning of each training, while 30% parameters are activated in the end. With this approach, parameters with smaller values undergo minor fine-tuning in the early stages of training, while parameters with larger values receive more extensive fine-tuning over additional steps. Compared to setting a large threshold in the beginning of training, this method balances the training of all model parameters, preventing excessive adjustments to a small subset of activated parameters to achieve the unlearning objective.
Calculating the percentiles of can be computationally expensive for models with a large number of parameters since it involves sorting the parameters. To address this, we employ an efficient estimation method. Calculating the maximum and minimum values of a list has a computational complexity of . By identifying the top 15% largest and smallest values, we can mitigate the influence of outliers. The estimated quantiles are then obtained through interpolation between these extreme values after excluding outliers.
Discussions regarding parameters masking in LLMs unlearning
We argue that an efficient way to preserve the utility of LLM during the unlearning process is to minimize changes to its parameters. In contrast to methods that directly calculate the mask using gradient magnitude, our proposed dynamic gradient sparsity method in § 3.2 learns the mask values from the data. And the mask is dynamically tuned along with the training process. In this way, each model parameter receives an appropriate training range: parameters with a relatively small impact on the training objective undergo minor adjustments in the early stages, after which they are masked, while parameters with a larger impact are trained more extensively. This approach allows for appropriate training of the model as a whole, avoiding large adjustments to all parameters or drastic changes to a small subset. Besides, the proposed method leverages the underlying vector to accumulate the influence of data on the mask, thereby avoiding biases introduced by extreme values during a single training iteration.
Improving Memory Efficiency
Although effective, the iterative approach in Equation (9) requires storing during the fine-tuning process of the -th task. To reduce memory usage, we propose an alternative method.
To reduce memory overhead during training, can be replaced with a lower-precision approximation, . For elements with absolute values exceeding a certain threshold, which have a significant impact on the retain set, their signs +1 or −1 are preserved, while the remaining elements are set to 0. During the iteration of , calculation of is estimated with:
where is a large positive constant. This method preserves both the signs and magnitude information of while significantly reducing memory consumption.
Experimental Setup
Dataset
TOFU ([25]) dataset consists of information about 200 fictional authors presented in a question-answer pair format. Specific authors’ information was designated as unlearning data, while the remaining authors’ data were retained to assist in training and testing. Besides, there are also question-answer pairs about Real Authors and World Facts. We grouped the information of four authors into one unlearning task, resulting in five or more continual unlearning tasks.
MUSE News ([31]) comprises BBC news articles published after August 2023. This dataset includes four predefined unlearning subsets that serve as continual unlearning tasks. Additionally, it contains a retain set that does not overlap with the unlearning sets.
WHP (Who’s Harry Potter) ([7, 31]) involves the unlearning of original data from the Harry Potter series. In contrast, retain data consists of knowledge related to the books, which aligns with real-world requirements for copyright protection. We treated the original texts of three Harry Potter books as three separate unlearning tasks. To evaluate the performance of algorithms in scenarios involving unrelated continual tasks, we constructed a sequence of continual unlearning tasks by interleaving multiple unlearning tasks from the MUSE News and WHP datasets.
Metrics
Rouge-L for unlearning (F-Rouge) measures the similarity between the text generated by the model and the correct text. Specifically, given a question or the beginning of the unlearning text, the Rouge-L recall is computed by comparing the output of models with the correct answer or remaining text. A lower Rouge-L value indicates better unlearning performance ([25, 31]).
Probability (F-Prob) evaluates the conditional probability of the model outputs in the TOFU dataset. To ensure comparability, this probability is normalized by the length of the output text ([25]).
Membership inference attack (MIA) uses the state-of-the-art Min-K% Prob method for LLMs ([30]) to detect whether data in the unlearning set remains part of the training data for the post-unlearning model. A lower probability of the unlearning set belonging to the training data reflects a more effective unlearning process.
Forget Quality (FQ) quantifies the difference in prediction distributions on the forget set between the post-unlearning model and a retrained model (fine-tuned on retrained data without exposure to the forget data). This metric assesses the degree to which the unlearning algorithm eliminates traces of the forget set in the model ([25]).
R-Rouge and R-Prob are also applied to the retain set or holdout set to evaluate the impact of the unlearning algorithm on the overall model utility. Higher values for these metrics indicate less performance degradation.
T-Rouge calculates the Rouge-L recall of completing sentences in the proposed TRAVIS dataset. Testing on inferred training data provides a more accurate assessment of the impact of unlearning algorithms on the pre-existing knowledge of the model. Additionally, TRAVIS is generated without thematic constraints, resulting in a broad coverage of content, which enables a more comprehensive evaluation of performance.
Model Utility (MU) represents the harmonic mean of the above metrics calculated across multiple retain sets in the TOFU dataset ([25]), including TRAVIS dataset. It provides an aggregate measure of the model utility.
Baseline methods
GA ([43]) unlearns target data by optimizing in the opposite direction of gradient descent. This is achieved by minimizing the negative cross-entropy loss of predicting the next token on the unlearning dataset:
NPO ([47]) is also a gradient ascent-based approach. Compared to GA, the gradient in NPO is scaled by a regularization term that gradually diminishes over the course of training, alleviating the problem of over-unlearning to some extent. The loss objective of NPO is:
Task Vectors (TV) ([15]) method manipulates the ability of a pre-trained model to address downstream tasks through fine-tuning and the arithmetic operation of model parameters. When applied to LLM unlearning, the method enables the selective removal of knowledge related to a specific dataset . Specifically, a pre-trained model is fine-tuned on the unlearning dataset , resulting in a model with enhanced knowledge of . The difference between the fine-tuned parameters and the original parameters, , constitutes the task vector for . By subtracting this task vector from the original parameters , a modified model is obtained that has unlearned the knowledge from .
DPO is a preference optimization approach. For a preference dataset, DPO encourages the model to produce one output over the other between a pair. In the unlearning scenario, it encourages the model to produce responses like ”I don’t know” instead of generating original sensitive text ([47]).
SKU ([23]) is a method based on task vectors. In the fine-tuning process, SKU not only reinforces the target knowledge into the model, but also eliminates knowledge of retain set from the model, incorporating a utility preservation objective. Besides, it includes a robustness objective during the fine-tuning process.
EUL incorporates lightweight unlearning layers into the model to enable efficient unlearning. This method explicitly considers scenarios involving multiple unlearning tasks ([6]).
SO-PO and SO-NPO introduce a second-order optimization framework for LLM unlearning, leveraging Hessian matrix approximations to address the challenges of applying Newton’s method to LLMs ([19]).
WAGLE identifies model parameters that benefit both unlearning and retaining objectives before training begins, optimizing only these parameters during the unlearning process. This approach is utilized to minimize the impact of unlearning on model utility ([18]).
When applying these methods in continual unlearning scenarios, the learning rate is tuned progressively to mitigate catastrophic collapse as tasks continue.
TRAVIS Dataset
To construct a high-quality text corpus, we adopted the methodology of Membership Inference Attack (MIA) ([44]) and applied it to two pre-trained language models, Phi-1.5 and Llama2-7b. The approach began with the generation of 5,000 sentences using the models. Subsequently, the MIA technique was employed to identify 250 candidate sentences with the highest probabilities from the generated set ([5]). After deduplication and filtering based on sentence similarity, the candidate set was further refined. More than 150 high-quality sentences were manually selected to ensure both diversity and authenticity in the dataset.
To enhance the variety of sentence prefixes and enable the models to process a broader range of contextual information, we incorporated a large collection of prefix data sourced from the internet following the work of Carlini et al. [5]. Specifically, we extracted samples from a subset of the Common Crawl dataset, cleaned the data by removing HTML tags and JavaScript code, and performed deduplication, resulting in approximately 50MB of clean text data. From this corpus, we randomly selected 5 to 10 context tokens and employed a Top-n sampling strategy to generate sequences based on the model’s probability distribution. The sequences with the highest likelihood were selected, and sentence lengths were capped at a maximum of 1024 tokens to ensure relevance and appropriate length.
The MIA methodology was applied next, using the perplexity of each sentence as a measure of training data likelihood. Perplexity was calculated with the following formula:
where denotes a sequence of tokens. Low perplexity indicates that the model holds high confidence in completing the sequence.
However, relying solely on MIA for membership inference introduces potential risks, as the model might assign higher likelihoods to unremarkable or low-quality samples, such as repeated substrings. To address this issue, we introduced several quality control mechanisms to ensure the integrity of the generated text. Specifically, we calculated the zlib entropy of each sample, measured perplexity within a sliding window, and computed the ratio of perplexity before and after lowercasing the text to assess its complexity and information density. These metrics allowed us to filter out low-quality samples and ensure that the selected texts exhibited a high degree of linguistic fluency and semantic richness.
During the selection process, samples were ranked according to the three quality metrics, and the top 100 were chosen. After deduplication, Sentence-BERT was used to compute cosine similarity between sentences, and those with similarity scores above 0.7 were excluded. Cosine similarity was calculated by:
where denote the embeddings of two sentences. This process yielded approximately 250 diverse and high-quality sentences. In the final manual filtering step, short, highly repetitive, or logically incoherent sentences were discarded, while longer substrings with richer semantics and more varied content were retained. The resulting dataset contains more than 150 high-quality sentences per model, meeting our established quality standards. For dataset construction, the first 70% of each sentence was designated as the prompt, while the remaining 30% served as the ground truth, thus creating the final training dataset.
This dataset offers two significant advantages. First, it accurately evaluates the impact of unlearning algorithms on the original knowledge retained by models. This aligns closely with the definition of efficiency in the context of machine unlearning. Second, the broad thematic coverage of the dataset enables evaluations of model performance across diverse contexts, ranging from everyday conversations to specialized topics. This diversity not only allows the dataset to comprehensively assess the language generation capabilities of the model, but also provides insights into its generalization ability when exposed to different background knowledge. Consequently, the dataset serves as a versatile and reliable foundation for evaluating model performance in a wide range of scenarios.
| Phi Model | |
| 1 | The Concept and Examples Recursion is a powerful programming concept that involves a function calling itself in its definition. This allows the program to be written in a more concise and elegant way, and provides a elegant solution to problems that would otherwise require multiple lines of code. For example: def factorial(n): if n == 0: return 1 return n * factorial( n-1 ) print( factory( 10 ) ) # Prints: 10 * 9 * 8 * 7 * 6 * 5 * 4 * 3 * 2 * 1 = 3628800 |
| 2 | Bullying is a pervasive problem that affects many people, especially children. It can have serious long-term effects on mental health, self-esteem, and academic performance. |
| 3 | Financial independence is a goal that many individuals aspire to achieve, but the path to attaining it can be challenging. One of the most effective ways to build financial independence early on is through saving. |
| 4 | As the world continues to grapple with the effects of climate change, many are turning to creative solutions to help mitigate its impact. From art installations that draw attention to environmental issues to community-based initiatives that promote sustainability, individuals and organizations alike are stepping up to the challenge. |
| 5 | Reasons to do exercise regularly: 1.To maintain a healthy weight: Regular exercise helps to burn excess calories and prevent weight gain. |
Table 4. Parts of examples in TRAVIS dataset (Phi Model)
More experimental results
Mixed dataset.
We argue that the higher the data relevance between unlearning tasks, the more severe the cascading degradation phenomenon. To evaluate the performance of our method in scenarios with low data relevance in continual unlearning, we conduct experiments mixing tasks from the MUSE News and WHP datasets. In these experiments, the model is sequentially tasked to unlearn data from the two datasets, and the results are shown in Figure 5. After completing the first MUSE News task, the model proceeds to unlearn the second WHP task, during which all methods exhibit a slight decline in model utility. The decline is less severe compared to unlearning consecutive WHP tasks. Similarly, our method maintains stable model utility when unlearning unrelated tasks sequentially. When moving to the third task—another MUSE News task—baseline methods experienced significant performance degradation due to the cascading degradation effect initiated by the first task. In contrast, our method demonstrates its robustness by effectively mitigating this phenomenon and maintaining model utility.
Figure 5. T-Rouge across seven continual unlearning tasks on the mixed dataset of MUSE News and WHP. Different background color represents tasks from different datasets.
| Llama Model | |
| 1 | The city of Los Angeles also offers a wide range of public services, including healthcare, education, and transportation, all of which are critical in maintaining a high quality of life for city residents. |
| 2 | There are a few things you can do to make sure you get the best possible experience from your interactions with the Russian culture and people. 1. Learn as much as you Can: |
| 3 | So, if you want to use the power of positive thinking to improve your life, start small and build up gradually. Believe in yourself and your abilities, and don’t worry about criticism as it’s an integral part of growth. |
| 4 | A New York City-based non-profit has been instrumental in fostering greater transparence in the city’s supplementary food system. This has involved working with foodstores, restaurants, and community organizations to promote greater understanding and access to healthier foods. |
| 5 | The global COVID-19 pandemic has significantly influenced the worldwide popularity of online platforms and digital services. |
Table 5. Parts of examples in TRAVIS dataset (Llama Model)
Experiments on the TOFU dataset
We present maximum and minimum model utility across 5 unlearning tasks in Figure 6. And the maximum F-Rouge is shown in Table 6, illustrating the performance of the least forgotten task after unlearning. Our proposed method achieves the lowest F-Rouge, outperforming baseline methods in terms of unlearning performance. Figure 6 compares the ability of retaining model utility among different methods. For Max MU, the proposed method attains the highest value (0.5574), followed by WAGLE (0.5486) and SO-NPO (0.5427). Similarly, for Min MU, the proposed method leads with a score of 0.5082, exceeding WAGLE (0.4721) and SO-NPO (0.4503), demonstrating superior retention of utility after unlearning all tasks. These results validate the proposed approach’s ability to balance effective unlearning and high model utility.
| Methods | NPO+RT | GA+RT | NPO+KL | SO-NPO | WAGLE | Ours |
| Max F-Rouge↓ | 0.3398 | 0.3167 | 0.4538 | 0.3614 | 0.4621 | 0.3067 |
Table 6. Max F-Rouge across five unlearning tasks on TOFU dataset for Llama2 model.
Figure 6. Max/Min Model Utility (MU) across five unlearning tasks on TOFU dataset for Llama2 model.
Experiments on the MUSE News dataset
MUSE News. Figure 7 (a) illustrates the utility and unlearning results on the MUSE News dataset, which exhibits more severe over-unlearning compared to the TOFU dataset. Specifically, for baseline methods like NPO+RT and GA+RT, the utility drops to zero during the third or fourth unlearning task. This can be attributed to the nature of the dataset, which requires verbatim unlearning of textual data rather than conceptual unlearning. Verbatim unlearning poses a greater challenge and directly disrupts the generative capabilities of the models. Moreover, the initial memorization of the model regarding the news content is relatively shallow; thus, high intensity and continuous unlearning can easily lead to excessive unlearning.
Our method, leveraging entropic soft labels, adaptively adjusts the unlearning intensity based on the prediction probabilities of the models for each sample. It ensures complete unlearning while dynamically modulating the unlearning intensity and selecting model parameters to avoid redundant unlearning. As a result, our approach maintains model utility effectively, with negligible performance degradation after completing four unlearning tasks.
Experiments on the WHP dataset
Figure 7 (b) illustrates the T-Rouge performance of various methods across three continual unlearning tasks on the WHP dataset. T-Rouge measures model utility on the TRAVIS dataset, which includes diverse genres, with higher values of T-Rouge indicating better retention of general knowledge. The ”Original” baseline, shown as the dashed line, represents the model’s performance without unlearning. Among the methods, our proposed approach demonstrates superior performance, maintaining the highest T-Rouge scores across all three tasks, indicating minimal degradation in utility. While competing methods like WAGLE and SO-NPO exhibit better retention than NPO+RT and GA+RT, they still show significant drops in performance, especially as tasks progress. As shown in Table 2, our method achieves comparable unlearning performance as other baselines. The consistently higher T-Rouge scores of the proposed method reflect its ability to balance effective unlearning with the retention of general knowledge across diverse tasks, surpassing all baseline methods. This highlights its efficacy in handling challenging continual unlearning scenarios.
Comparison between continual unlearning and one-time unlearning
To further validate the harmful effect of cascading degradation, we compare the model utility under continual unlearning and one-time unlearning scenarios in Figure 8. For one-time unlearning, all the data for current and previous unlearning tasks is jointly used for unlearning, which is only for experimental purposes and unrealistic in practical applications.
The Rouge-F in these two scenarios are aligned for a fair comparison of model utility. As shown in the figure, continual unlearning results in a significantly more drastic decline in utility, while one-time unlearning results in a mild decrease in utility as the number of tasks increases. This difference indicates the severe damage caused by cascading degradation in utility.
References
- [1]Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., and Tan, K. E. Digital forgetting in large language models: A survey of unlearning methods. CoRR, abs/2404.02062, 2024.
- [2]Brophy, J. and Lowd, D. Machine unlearning for random forests. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 1092–1104. PMLR, 2021.arxiv.org/abs/2009.05567
- [3]Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.arxiv.org/abs/2005.14165
- [4]Cao, Y. and Yang, J. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, SP 2015, San Jose, CA, USA, May 17-21, 2015, pp. 463–480. IEEE Computer Society, 2015.
- [5]Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D., Erlingsson, Ú., Oprea, A., and Raffel, C. Extracting training data from large language models. In 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, pp. 2633–2650. USENIX Association, 2021.
- [6]Chen, J. and Yang, D. Unlearn what you want to forget: Efficient unlearning for llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 12041–12052. Association for Computational Linguistics, 2023.
- [7]Eldan, R. and Russinovich, M. Who’s harry potter? approximate unlearning in llms. CoRR, abs/2310.02238, 2023.arxiv.org/abs/2310.02238
- [8]Golatkar, A., Achille, A., and Soatto, S. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX, volume 12374 of Lecture Notes in Computer Science, pp. 383–398. Springer, 2020.arxiv.org/abs/2003.02960
- [9]Golatkar, A., Achille, A., Ravichandran, A., Polito, M., and Soatto, S. Mixed-privacy forgetting in deep networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 792–801. Computer Vision Foundation / IEEE, 2021.arxiv.org/abs/2012.13431
- [10]Graves, L., Nagisetty, V., and Ganesh, V. Amnesiac machine learning. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 11516–11524. AAAI Press, 2021.arxiv.org/abs/2010.10981
- [11]Guo, C., Goldstein, T., Hannun, A., and Van Der Maaten, L. Certified data removal from machine learning models. arXiv preprint arXiv:1911.03030, 2019.
- [12]Han, B., Niu, G., Yu, X., Yao, Q., Xu, M., Tsang, I. W., and Sugiyama, M. SIGUA: forgetting may make learning with noisy labels more robust. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 4006–4016. PMLR, 2020.
- [14]Hong, Y., Zou, Y., Hu, L., Zeng, Z., Wang, D., and Yang, H. Dissecting fine-tuning unlearning in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 3933–3941. Association for Computational Linguistics, 2024b.
- [15]Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.arxiv.org/abs/2212.04089
- [16]Izzo, Z., Smart, M. A., Chaudhuri, K., and Zou, J. Approximate data deletion from machine learning models. In The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pp. 2008–2016. PMLR, 2021.
- [18]Jia, J., Liu, J., Zhang, Y., Ram, P., Baracaldo, N., and Liu, S. WAGLE: strategic weight attribution for effective and modular unlearning in large language models. CoRR, abs/2410.17509, 2024a.
- [19]Jia, J., Zhang, Y., Zhang, Y., Liu, J., Runwal, B., Diffenderfer, J., Kailkhura, B., and Liu, S. SOUL: unlocking the power of second-order optimization for LLM unlearning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 4276–4292. Association for Computational Linguistics, 2024b.
- [20]Koh, P. W. and Liang, P. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885–1894. PMLR, 2017.
- [22]Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, Thaker, P., Hu, S., Kale, N., Maurya, Y., Wu, Z. S., and P., Xu, X., Yao, Y., Li, H., Varshney, K. R., Bansal, M., Smith, V. Position: LLM unlearning benchmarks are Koyejo, S., and Liu, Y. Rethinking machine unlearn- weak measures of progress. CoRR, abs/2410.02879, ing for large language models. CoRR, abs/2402.08787, 2024. 2024a.
- [23]Liu, Z., Dou, G., Tan, Z., Tian, Y., and Jiang, M. Towards safer large language models through machine unlearn- ing. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 1817–1829. Association for Computational Linguistics, 2024b.arxiv.org/abs/2402.10058
- [24]Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadfield- Menell, D. Eight methods to evaluate robust unlearning in llms. CoRR, abs/2402.16835, 2024.arxiv.org/abs/2402.16835
- [25]Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and Kolter, J. Z. TOFU: A task of fictitious unlearning for llms. CoRR, abs/2401.06121, 2024.arxiv.org/abs/2401.06121
- [26]Niu, G., du Plessis, M. C., Sakai, T., Ma, Y., and Sugiyama, M. Theoretical comparisons of positive-unlabeled learning against positive-negative learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 1199–1207, 2016.arxiv.org/abs/1603.03130
- [27]Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Canton-Ferrer, C., Graffitiori, A., Xiong, W., Défoss ez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T., and Synnaeve, G. Code llama: Open foundation models for code. CoRR, abs/2308.12950, 2023.arxiv.org/abs/2308.12950
- [28]Schmidt, M. Cpsc 540: Machine learning. Convergence of Gradient Descent, 2017.
- [29]Scholten, Y., Günnemann, S., and Schwinn, L. A probabilistic perspective on unlearning and alignment for large language models. CoRR, abs/2410.03523, 2024.
- [30]Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a.arxiv.org/abs/2310.16789
- [31]Shi, W., Lee, J., Huang, Y., Malladi, S., Zhao, J., Holtzman, A., Liu, D., Zettlemoyer, L., Smith, N. A., and Zhang, C. MUSE: machine unlearning six-way evaluation for language models. CoRR, abs/2407.06460, 2024b.arxiv.org/abs/2407.06460
- [32]Thaker, P., Hu, S., Kale, N., Maurya, Y., Wu, Z. S., and Smith, V. Position: LLM unlearning benchmarks are weak measures of progress. CoRR, abs/2410.02879, 2024.arxiv.org/abs/2410.02879
- [33]Ullah, E., Mai, T., Rao, A., Rossi, R. A., and Arora, R. Machine unlearning via algorithmic stability. In Conference on Learning Theory, COLT 2021, 15-19 August 2021, Boulder, Colorado, USA, volume 134 of Proceedings of Machine Learning Research, pp. 4126–4142. PMLR, 2021.arxiv.org/abs/2102.13179
- [34]von Oswald, J., Zhao, D., Kobayashi, S., Schug, S., Caccia, M., Zuc het, N., and Sacramento, J. Learning where to learn: Gradient sparsity in meta and continual learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 5250–5263, 2021.arxiv.org/abs/2110.14402
- [35]Wang, Q., Han, B., Yang, P., Zhu, J., Liu, T., and Sugiyama, M. Unlearning with control: Assessing real-world utility for large language model unlearning. CoRR, abs/2406.09179, 2024a.
- [36]Wang, Q., Lin, Y., Chen, Y., Schmidt, L., Han, B., and Zhang, T. A sober look at the robustness of clips to spurious features. In Advances in Neural Information Processing Systems, 2024b.arxiv.org/abs/2403.11497
- [37]Wang, Q., Han, B., Yang, P., Zhu, J., Liu, T., and Sugiyama, M. Towards effective evaluations and comparison for llm unlearning methods. In International Conference on Learning Representations, 2025a.arxiv.org/abs/2406.09179
- [38]Wang, Q., Zhou, J. P., Zhou, Z., Shin, S., Han, B., and Weinberger, K. Q. Rethinking llm unlearning objectives: A gradient perspective and go beyond. In International Conference on Learning Representations, 2025b.arxiv.org/abs/2502.19301
- [39]Wang, Y., Wang, Q., Liu, F., Huang, W., Du, Y., Du, X., and Han, B. Gru: Mitigating the trade-off between unlearning and retention for large language models. In International Conference on Machine Learning, 2025c.
- [42]Yang, P., Wang, Q., Huang, Z., Liu, T., Zhang, C., and Han, B. Exploring criteria of loss reweighting to enhance llm unlearning. In International Conference on Machine Learning, 2025.arxiv.org/abs/2505.11953
- [43]Yao, Y., Xu, X., and Liu, Y. Large language model unlearning. CoRR, abs/2310.10683, 2023.arxiv.org/abs/2310.10683
- [44]Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 31st IEEE Computer Security Foundations Symposium, CSF 2018, Oxford, United Kingdom, July 9-12, 2018, pp. 268–282. IEEE Computer Society, 2018.arxiv.org/abs/1709.01604
- [45]Yuan, X., Pang, T., Du, C., Chen, K., Zhang, W., and Lin, M. A closer look at machine unlearning for large language models. CoRR, abs/2410.08109, 2024.arxiv.org/abs/2410.08109
- [46]Zhang, H., Nakamura, T., Isohara, T., and Sakurai, K. A review on machine unlearning. SN Comput. Sci., 4(4): 337, 2023.DOI
- [47]Zhang, R., Lin, L., Bai, Y., and Mei, S. Negative preference optimization: From catastrophic collapse to effective unlearning. CoRR, abs/2404.05868, 2024a.arxiv.org/abs/2404.05868
- [48]Zhang, Z., Zhang, J., Yao, H., Niu, G., and Sugiyama, M. On unsupervised prompt learning for classification with black-box language models. CoRR, abs/2410.03124, 2024b.
- [49]Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J., and Wen, J. A survey of large language models. CoRR, abs/2303.18223, 2023.DOI







