WorksComputer Science
Online Label-Free Gradient Matching for Continual Language Model Training
Abstract
Language models in customer-facing products must be continually updated, since the data they serve shifts over time. As retraining on the full accumulated data is impractical, each update should instead run on a small set of examples carrying the training signal of all data seen so far. The strongest current methods build this set by gradient matching, which synthesizes examples whose training gradients match those of the full data. However, they require ground-truth labels, run once on a fixed snapshot rather than on a growing stream, and are slow on large data. We introduce OLGA, which lifts these limitations. It distills unlabeled data by drawing the gradient matching targets from the model's own next-token predictions. Across five text classification benchmarks with phi-1.5, it improves accuracy by 10 percentage points over the strongest selection baseline using as few as five examples. OLGA also distills online data streams by keeping a running rehearsal memory of synthetic examples, which is re-distilled together with each new batch as it arrives. In the online setting, it outperforms the strongest selection-based continual baseline by 6.6 stream-AUC percentage points. Finally, our method scales to large data by pairing distillation with selection in a hybrid pipeline, letting selection handle the mild stage of compression and distillation the extreme one. It cuts runtime by 30% versus distillation alone at similar accuracy, and beats the best selection-only pipeline by up to 12.6 percentage points.
Introduction
Language models are increasingly deployed in customer-facing products, where the traffic they serve keeps changing. User preferences move, new topics appear, and safety requirements shift, so a model that was accurate at release gradually falls out of step with the data it now sees ([10]). Keeping it current means updating it repeatedly rather than once, and the data available at each update grows with every cycle. Retraining on every example accumulated so far is the obvious remedy and also the least affordable one, because a full pass costs more as the data grows while the oldest examples contribute least. What a practical update loop needs instead is a small set of examples, rebuilt at each cycle, that still carries the training signal of everything seen so far.
Most existing methods build that set by selecting a well-chosen subset of the real examples. This works well when the budget is generous, but once only a few examples can be kept, it becomes hard to preserve a strong training signal. Dataset distillation with gradient matching addresses this by synthesizing examples whose training gradient matches the gradient of the real data ([46]). Because a synthetic example does not have to be a real sentence, it can capture what many real examples teach the model, so the set can be kept to a few examples.
Three obstacles stand in the way of using gradient matching to keep a language model current. First, it needs ground-truth labels, which rules out the unlabeled data that makes up most of what a deployed system collects. For images, recent work replaces the labels with targets from a frozen model trained without them ([11, 7]), but text distillation methods still rely on labels ([25]). Second, it runs once over a fixed snapshot, so it never has to fold an arriving batch into an existing set. Methods that do this exist for images ([5]), but they need labels and nudge pixel values in small steps, which does not work for the discrete tokens of text. Finally, the cost of matching gradients grows with the amount of data it compresses, making it least affordable exactly when data is plentiful. For images, discarding part of the real data before distilling cuts this cost [42, 22], but these methods rank examples by their loss against a label and have only been studied on a fixed snapshot. As a result, no existing method compresses an unlabeled text stream into a small training set at a cost that stays affordable as the stream grows.
We remove all three obstacles at once. To drop the labels, we keep them out of the objective and take the matching target from the model’s own predictions instead. To move from a snapshot to a stream, we keep a running rehearsal memory of synthetic examples and merge each arriving batch into it with the same gradient matching objective. To control the cost, we introduce a hybrid pipeline in which cheap unsupervised selection runs ahead of expensive distillation, since the two are strong at different amounts of compression.
We make the following contributions.
We introduce a label-free gradient matching objective whose compact distilled set trains a model 10 points more accurate than the best real subset of the same size.
We extend text distillation to streaming data, updating a rehearsal memory as each batch arrives, and beat the best real memory of the same size by 6.6 stream-AUC points.
We design a hybrid pipeline in which cheap selection shrinks each batch before distillation, cutting distillation runtime by 30% compared with distilling the full batch.
Background and Related Work
For much of the past decade, progress in deep learning came from scale, with larger models trained on more data consistently outperforming smaller ones. As models have grown to billions of parameters and corpora to trillions of tokens, however, the bottleneck has shifted from data availability to compute and time. Deployed models must also be updated ever more often as user preferences, knowledge, and safety requirements change, and retraining on the full corpus at every update is prohibitively expensive. The question is therefore no longer how to collect more data, but how to select or construct the smallest set of examples that preserves the training signal of the full corpus.
Coreset selection. Coreset selection keeps the most representative slice of the real training set without modifying any examples. Geometry-based algorithms formalize this as a coverage problem, where k-center greedy [33] picks the point farthest from the selected set, while herding [29] and prototypical selection [34] sequentially choose samples that best approximate the per-class feature centroid. Training-oriented variants use model-dependent signals such as gradient norms (GraNd) or L2 prediction error (EL2N) [27] to prioritize samples near the decision boundary rather than near the class mean. Despite this diversity, all coreset methods share a hard representational ceiling, since a text sequence is an atomic unit that cannot be perturbed below the token level and no selection rule can manufacture variance that does not already exist in the chosen sample. This ceiling becomes binding at extreme compression, the regime where synthetic examples become necessary.
Dataset distillation. Dataset distillation compresses a large training set into a handful of synthetic examples, optimized so that a model trained on them behaves like one trained on the original data. Wang et al. (2018) first framed this as bi-level meta-learning, updating the synthetic examples through an unrolled inner training loop, but the memory cost and numerical instability of unrolling restrict it to shallow networks trained for only a few steps. Later work replaces the unroll with cheaper surrogate objectives. Trajectory matching [3] aligns the parameter path traced by training on synthetic data with paths recorded on real data, at the cost of pre-computing and caching thousands of such expert trajectories. Distribution matching [45] instead matches per-class feature embeddings of real and synthetic examples, which is cheap but ignores the task loss and can leave class boundaries misaligned. Gradient matching [46] targets the task loss directly by finding synthetic examples whose per-step parameter gradient matches the real gradient in cosine distance. It needs neither long unrolls nor stored trajectories and scales to larger architectures more readily, making it the dominant paradigm. All three directions, however, assume that gradients flow through a continuous input space, which holds for images but breaks down for discrete text tokens, where a gradient step on the input has no direct meaning.
Dataset distillation for language models. Early work on distilling text avoided discreteness by optimizing continuous word embeddings instead of tokens [18], but the results are model-specific, meaning a synthetic embedding created with BERT cannot be used to fine-tune RoBERTa, and projecting the optimized embeddings back to the nearest vocabulary token yields incoherent sequences that carry little training signal. Maekawa et al. (2024) addressed this with DiLM, which trains a proxy generative model to synthesize readable text, routing gradients through the discrete bottleneck via generation probabilities and producing sequences that transfer across architectures. Training a full proxy language model as a distillation scaffold is itself expensive, however, and the proxy may memorize sensitive training-set content. GradMM [23] avoids the proxy by applying the Alternating Direction Method of Multipliers (ADMM) [2], in which gradient matching proceeds in continuous embedding space while a periodic projection step snaps each embedding back to its nearest vocabulary token, enforcing hard discreteness throughout optimization. This produces human-readable, cross-architecture synthetic text, and it is the offline supervised technique we build on here. Like every text distillation method so far, however, it places ground-truth labels in the matching objective, which ties the distilled data to a single task and excludes the unlabeled data that makes up most of what language models are trained on.
Continual learning and catastrophic forgetting. In continual learning, a model must keep absorbing a non-stationary stream of data without losing what it has already learned. The core difficulty is catastrophic interference, in which training on new material abruptly overwrites earlier knowledge [21, 4], and it arises whether the stream introduces new tasks, new domains, or new classes [38]. Architecture-based methods counter it by dedicating parameters to each task [31, 20], but presuppose known task boundaries and a growing parameter budget. Regularization-based methods penalize changes to weights that matter for past tasks [8, 44, 15], but weaken over long streams as their accumulated constraints stiffen the network. The dominant remedy is rehearsal, which replays a small memory of past examples alongside incoming data, with iCaRL [29] as its canonical instance and a strong baseline for nearly a decade. Because the memory is the method, much work concerns how to curate it, from fixed rules such as reservoir sampling, herding, or class-balanced storage [28] to gradient-based rules that keep the examples whose gradients best span [1] or best approximate [37] the gradient of all data seen so far. Every such scheme still chooses among real examples, so the memory inherits the representational ceiling of coreset selection. That ceiling binds hardest where rehearsal matters most: as the stream grows and the per-class budget shrinks, a few stored examples can no longer carry the training signal of the task they stand for.
Continual distillation as rehearsal memory. This ceiling motivates compressing the memory rather than selecting it, using dataset distillation to synthesize a handful of maximally informative rehearsal examples. Distilled Replay keeps as little as one synthetic pattern per class [30], Zhao et al. (2021) showed that gradient-matched synthetic memories alleviate forgetting in a task-incremental image setting, and later methods refine the buffer by carrying cross-task information [16], mixing distilled with real exemplars [9], or condensing prototypical exemplars [24]. Two limitations run through this line of work. First, these methods optimize continuous pixel tensors with objectives that do not carry over to discrete tokens. Continual learning for text has therefore sidestepped distillation in favor of generative rehearsal, sampling pseudo-examples from an autoencoder [43] or prompting the model to synthesize its own rehearsal set [6], which re-incurs the expense and memorization risk of a separate generator. Second, nearly all of them are offline, distilling each task once from a complete snapshot. The one exception, SSD [5], folds each arriving batch into a synthetic image memory by matching gradients against the labeled batch. How to fold an arriving batch into a synthetic memory of discrete tokens, with or without labels, has not been studied.
Methodology
To address these obstacles, we introduce Online Label-free Gradient Matching (OLGA), in which each component removes one obstacle (Figure 1). We begin in § 3.1 with the offline supervised method that OLGA builds on. § 3.2 then drops the need for labels, § 3.3 extends the method to streaming data, and § 3.4 speeds it up with a hybrid design.
Figure 1. Overview of the components of OLGA. (a) Synthetic text is generated by matching its training gradient to that of real data, alternating with a step that maps it back to real tokens. (b) A frozen language model’s next-token predictions replace ground-truth labels, so gradient matching works on unlabeled data. (c) A running rehearsal memory extends offline distillation to an online data stream. Each new batch is distilled together with the memory, which the model then retrains on. (d) Our hybrid design first uses cheap selection to shrink each batch, then distills what remains into the few examples kept, which cuts runtime on large data.
Gradient matching with the supervised objective
The method we start from is GradMM (Nguyen et al., [23]), which produces readable and architecture-agnostic examples by solving a discrete gradient matching optimization. Let denote the parameters of a frozen pretrained language model and a set of examples , each pairing a prompt with a response. The loss of the model on , written , is the average of the individual example losses . The real examples that serve as a matching target are called anchors and written , and the synthetic set produced from them is , held to a budget of examples. GradMM searches for a synthetic set whose fine-tuning gradient matches the anchor gradient at the pretrained parameters,
where compares gradient vectors by direction rather than scale, and is the set of readable token sequences. GradMM solves this with ADMM (Boyd et al., [2]), which alternates a few steps of first-order optimization in continuous embedding space with a projection back onto real vocabulary tokens (Figure 1(a)). We keep this solver unchanged and focus instead on making the optimization work without labels, on a data stream, and at lower runtime.
In the supervised setup, the model is asked to emit a verbalizer token , a single word standing in for the class of the example, at a fixed position after the prompt, and the loss is the cross-entropy of that single token, . Since every example of a class is trained toward the same token, their gradients add up and add up rather than cancel. The anchor gradient thus lies in a few class directions, which a handful of synthetic examples can reproduce closely.
Gradient matching with a label-free objective
While the supervised loss needs each example’s class to know which verbalizer token the model should emit, customer-facing platforms mostly receive unannotated examples for which a single class label is hard to pin down (Zhao et al., [47]). Such data is rarely unstructured, though, and can be partitioned into groups by pre-logged metadata or through unsupervised clustering (Wan et al., [40]). We use this grouping, together with the model’s internal next-token probabilities, to design a label-free loss that preserves inter-group separation (Figure 1(b)).
Let denote the probability the frozen model puts on a next token after a prompt , and let and be its averages over one group and over all anchors. The pointwise mutual information between a token and a group,
measures how much more the model expects inside one group than across the data as a whole. Ranking the vocabulary by equation (2) and keeping the highest-scoring tokens gives a set of the tokens that best separate group from the rest, and is the only hyperparameter the construction adds. Normalizing the group probabilities across turns them into a target , and for the group-distribution loss is the soft cross-entropy toward it,
Substituting into equation (1) replaces the class label with a soft group distribution read off the model’s own knowledge, while preserving the separation between groups. § 4.5 compares it against several alternatives.
Online gradient matching with rehearsal memory
The objective of equation (1) suits an offline setup, where a single fixed dataset is available all at once. A deployed system rarely works this way, since its data arrives as a sequence of blocks , with the data seen through step [13]. In this online setting, we want a compact synthetic set of at most examples that, at each step, best represents everything seen so far, so that fine-tuning on it tracks the model trained on all data through step . We write this set as , the one whose gradient at matches that of the full history:
Computing directly is impractical in a stream, since it requires keeping the entire history and re-distilling it from scratch at every step, so both storage and per-step cost grow without bound as the stream lengthens. We approximate it by solving a separate distillation problem for each incoming block and merging the results into a running rehearsal memory through the same gradient-matching objective. Each block is first compressed on its own into a synthetic set by the offline problem of equation (1), and the memory carries these forward. It starts from the first distilled block and, at every later step, is rebuilt by distilling the union of the new set and the previous memory back down to ,
We call this a cascaded merge (Figure 1(c)). At every step we fine-tune on , which stands in for the we cannot compute. Each step distills only the new block and the current memory instead of the whole history, so its cost is the same at every step and the total grows linearly with the stream length . Since the merge is independent of the loss, it applies to both the supervised and the label-free objectives. We compare the cascaded merge against alternative merge strategies in § 4.5.
Hybrid hierarchical compression
Selection and distillation work best in different regimes. When the budget is a large fraction of the data, cheap coreset selection is enough, since a well-chosen real subset still covers most of the training signal. As the budget shrinks toward a handful of examples, no real subset can carry that signal, whereas a single synthetic example can combine signal drawn from many real ones. Viewing compression as an information bottleneck [36, 48], we expect selection to suffice while it is wide and distillation to be needed once it narrows.
We therefore compress in two stages (Figure 1(d)). At every compression step, offline on a static pool or at each step of the stream, a coreset selector first cuts the incoming batch of examples down to a smaller pool of , and gradient matching then distills that pool into the examples actually kept, so . We call this the hybrid pipeline and compare it against selection-only and distillation-only alternatives in § 4.4.
Experiments
Experimental setup
Data. We use five text classification datasets: SST-2 ([35]), Rotten Tomatoes ([26]), RT-Polarity ([26]), IMDB ([17]), and TwitterEmotion ([32]). They span two domains, movie reviews and tweets, inputs from a single sentence to a full review, and training pools from 1,000 examples to tens of thousands. Every experiment uses all five, except the streaming hybrid, whose ten steps of 1,000 examples only SST-2 and IMDB can supply. Appendix A1 details the data partitioning and stream construction.
Model. Following GradMM ([23]), we use phi-1.5 ([14]) for our experiments. During distillation, we only match the gradients of the output layer to avoid recomputing the gradients of all 1.3B parameters at each step, and we run the ADMM solver for 30 outer iterations of 50 inner steps at a step size of 0.008. During fine-tuning, we train 200 steps on a linear schedule, evaluating every 50 steps and keeping the best checkpoint, with the learning rate picked per dataset from . While we use phi-1.5 for both distillation and fine-tuning, two different models can be used in general, without any change to the method, because what distillation returns is plain text rather than weights.
Compared methods and metrics. In the offline setup, our main metric is accuracy after fine-tuning, and we compare against five competing methods: k-center greedy ([33]) takes the example farthest from those already picked; herding ([29]) takes the ones that keep the running mean closest to the average; prototypical ([34]) takes the ones closest to that average; uncertainty ([12]) takes the ones the model is least sure about; and random picks uniformly. In the online setup, we focus on the area under the accuracy-versus-step curve (AUC), which rewards being right at every step rather than only the last, and we compare against four competing methods: k-center greedy, herding, and prototypical from above, and reservoir sampling ([39]), which keeps a uniform sample of all data seen. In both settings we also report three reference points. An oracle fine-tunes on all available real data, and supervised GradMM distills with the ground-truth labels that our label-free method never sees. Together they give upper bounds, while the zero-shot model with no fine-tuning gives the lower bound.
Offline label-free gradient matching beats competing methods
We benchmark OLGA, setting for its label-free group distribution objective, against the supervised objective (GradMM) and the five competing methods in the offline setup, where the full data pool is available at once. Each task starts from 50 real anchors, which we distill into a compact synthetic set of 5 examples and fine-tune phi-1.5 on, evaluating on the held-out validation split of each dataset. This is a 90% reduction, which creates the extreme-compression regime we target.
Figure 2(a) reports classification accuracy on each dataset. OLGA is the strongest label-free method, beating the strongest competing method on four of the five datasets and by 10.2 points on average. The gap is widest on IMDB, where OLGA scores 89.5 against 72.1 for the strongest competitor, and narrowest on Rotten Tomatoes, at 4.1 points (74.5 against 70.4). The one exception is TwitterEmotion, which is hard for every method, and where OLGA reaches just 56.5, on par with the competing methods. Overall, even without labels, gradient matching keeps enough signal to compress a dataset into a handful of synthetic examples that train a model better than any subset of real examples can. Table 1 shows two distilled examples next to the real examples they borrow their phrasing from, and Appendix A5 one for each of the other datasets. Appendix A2 lists exact values for Figure 2(a).

Figure 2. Across five datasets, OLGA beats every competing method on average in both the offline and online setups, and comes close to the supervised upper bound in the online one. (a) Offline methods reduce 50 real examples to 5, and we report the held-out accuracy of phi-1.5 fine-tuned on them. (b) Online data arrives over a sequence of steps, each method keeps a memory of only 5 examples, and we report stream-AUC, the area under the accuracy-versus-step curve.
| TwitterEmotion (joy) | SST-2 (negative) | |
| Synthetic | I am feeling1 so much2 better after my recent health surgery yesterday and I can feel3 more | teacher was giving a long1 and boring and2 not interesting and3 clear lecture to the students because the teacher was reading from a textbook |
| Real | 1. i am feeling amazing mostly normal i am going to a pre thanksgiving celebration with our friends from that time we were in softball<br>2. i feel like we have so much to be thankful for<br>3. i can feel superior on that point | 1. it ai n’t art, by a long shot,<br>2. a shame that stealing harvard is too busy getting in its own way to be anything but frustrating, boring, and forgettable.<br>3. a more confused, less interesting and more sloppily made film could possibly come down the road in 2002 |
Table 1. Two synthetic examples distilled by OLGA, one per dataset. Each is generated as a mix of phrases from real examples, three of which are listed below it. A shared phrase has the same color in both places, and its superscript gives the number of the real example it comes from.
Online gradient matching with memory beats continual selection
We benchmark OLGA, setting for the rehearsal memory and cascaded merge budgets, across all five datasets in an online setup of steps (8 for IMDB). At each step, 50 anchors arrive and are distilled into 5 synthetic examples, which are then folded into the memory by the cascaded merge. At each step, phi-1.5 is fine-tuned on the resulting memory and evaluated on the held-out validation set.
Figure 2(b) reports the online stream-AUC of the fine-tuned phi-1.5 on each dataset. Averaged over the five datasets, Online GradMM and OLGA finish 8.5 and 6.6 points above k-center, the strongest competing method overall at 67.0. The two are within two points of each other, so removing the labels costs little. Across datasets the biggest gain for both comes on RT-Polarity, roughly 10 points over the best competing method there. The shortfalls are all on the label-free side, where OLGA only draws level with the best competing method on TwitterEmotion and comes in 4.2 points behind it on SST-2. Appendix A3 lists exact values for Figure 2(b).
Hybrid compression pipeline
We explore whether a mix of selection and distillation exists that lets cheap selection take the early, easy part of compression and leaves only the extreme end to distillation, and how such a mix compares against alternatives. In our setup, a batch of real examples arrives, selection trims it to , and distillation turns those into the examples we keep and train on. This achieves a 99.5% compression rate overall. We call this two-stage route Hybrid, and compare it against two alternatives. Direct drops the middle stage and goes from straight to in a single step. Double keeps both stages but runs the second one the same way as the first. Across all three routes, the distillation stage runs either GradMM or OLGA, and selection is done by uniform sampling.
Hybrid pipeline in offline setting. Table 2 compares the accuracy of the three routes in the offline setting, where the whole compression happens once over a static pool, and Table 2 compares what each route costs in time. With labels, the hybrid pipeline is the strongest route, averaging 86.0 percent across the five datasets, ahead of GradMM-Direct at 82.7 and GradMM-Double at 73.9, while costing less than distilling the whole batch directly or in two steps. This confirms that selecting early, while the compression is still easy, and distilling only at the extreme end is the more effective way through the bottleneck, and the cheaper one too. Without labels, the same funnel becomes a trade-off rather than a gain: OLGA-Hybrid falls short of OLGA-Direct by only 0.9 points on average while running 30% faster, which puts the two routes on par and makes the choice between them a balance of accuracy against time. What holds for both objectives is the lead over selection: GradMM-Hybrid and OLGA-Hybrid beat the selection-only routes on average, 86.0 and 80.0 against 73.4 for Selection-Double and 66.6 for Selection-Direct.
| Method | Route | Offline accuracy | Online stream-AUC | Runtime (s) | |||||||
| SST-2 | Rotten T. | IMDB | RT-Pol. | TwitEmo. | Avg. | SST-2 | IMDB | Avg. | |||
| GradMM | Hybrid | 88.2 | 83.9 | 92.3 | 82.1 | 83.4 | 86.0 (+3.3) | 76.0 | 80.8 | 78.4 (+3.1) | 436.1 (−29.9%) |
| Direct | 87.4 | 76.6 | 90.6 | 81.3 | 77.6 | 82.7 | 72.1 | 78.5 | 75.3 | 622.1 | |
| Double | 79.7 | 71.1 | 80.7 | 67.0 | 70.8 | 73.9 (−8.8) | — | — | — | 2073.6 (+233.3%) | |
| OLGA | Hybrid | 87.8 | 81.2 | 89.0 | 84.6 | 57.3 | 80.0 (−0.9) | 71.2 | 78.4 | 74.8 (+2.1) | 447.5 (−30.1%) |
| Direct | 88.6 | 85.0 | 90.9 | 82.6 | 57.4 | 80.9 | 67.7 | 77.7 | 72.7 | 639.9 | |
| Double | 67.4 | 71.2 | 87.6 | 76.1 | 58.9 | 72.2 (−8.7) | — | — | — | 2086.4 (+226.1%) | |
| Selection | Double | 82.5 | 73.6 | 89.8 | 52.9 | 68.0 | 73.4 (+6.8) | 69.6 | 75.3 | 72.5 (+1.4) | 5.5 (+14.6%) |
| Direct | 81.9 | 54.7 | 87.9 | 50.3 | 58.1 | 66.6 | 67.2 | 75.0 | 71.1 | 4.8 |
Table 2. Comparison of the hybrid, direct, and double routes in the offline and online settings across 5 datasets. At each time step, the data is compressed in stages from to to . The offline setting runs a single step (), and the online setting runs steps. Runtime is the wall-clock time of one offline distillation run on SST-2. On average, the hybrid route beats the direct and double routes for both objectives in both settings, except offline without labels, where OLGA-Hybrid is on par with OLGA-Direct. Bold marks the best result in each column, and underline marks the reference that each colored difference is measured against.
Hybrid pipeline in online setting. We extend the comparison to the online setting, where a new batch of examples arrives at each of steps. After each distillation, the resulting examples are merged into the running rehearsal memory by the cascaded merge. We report only SST-2 and IMDB, since they are the only datasets large enough to supply a stream of this length.
The online half of Table 2 shows that GradMM-Hybrid performs best, with an average stream-AUC of 78.4. GradMM-Hybrid beats GradMM-Direct by 3.1 points and OLGA-Hybrid beats OLGA-Direct by 2.1 points, while distilling only 50 examples at each step instead of 1,000. Every route that uses distillation also beats selection alone on average, and the best of them leads the strongest selection-only route (72.5) by 5.9 points.
Ablations
Alternative label-free objectives. The group-distribution loss introduced in OLGA is not the only way to turn the model’s own probabilities into a label-free signal. We compare it against two alternatives (Figure 3(a)). The pseudo-token signal appends the prompt “In a single word:” and takes the most likely next token as its target, letting the model label itself in one word. The self-information signal finds the token in each sequence that the frozen model predicted least well and takes it as the target, on the view that this token carries most of what makes the example distinctive. Appendix A6 describes both signals further and gives their exact formulations.
Figure 3. (a) How each gradient matching objective builds its loss from the model’s next-token distribution; the dots mark which position it reads. (b) Three ways to fold a new batch into the memory: compress it first, pool it with the memory, or weight the two by hand.
Figure 4 compares the three signals. At , group-distribution averages 75.5 percent across the five datasets, ahead of pseudo-token (74.1) and self-information (72.2), and it is the steadiest as the budget grows. The tokens each signal aims at help explain why. Pseudo-token aims at only 18 distinct tokens, spread over just 3.1 bits of entropy, so its training signal keeps pushing in the same few directions, while self-information spreads over 47 tokens (5.5 bits) and group-distribution over 100 (6.6 bits). The alternatives are also up to five times slower to run.
Figure 4. Comparison of alternative label-free objectives. (Left) Accuracy against the number of examples , per dataset and on average. (Right) Number of distinct tokens each signal aims at, and runtime relative to group-distribution. Group-distribution leads at , stays steady at larger budgets, and runs fastest.
Alternative selectors for the hybrid pipeline. The hybrid routes use uniform sampling by default, but any selection method can take its place. We repeat the offline GradMM-Hybrid route with k-center, the strongest selection baseline of Figure 2(b). On average, uniform sampling is both more accurate and faster than k-center (Table 3). This indicates that naive sampling, which keeps the original distribution of the data, is preferable in the hybrid pipeline, since distillation is able to synthesize higher-quality text from it. Appendix A7 reports the per-dataset numbers.
| Selector | Avg. accuracy | SST-2 runtime (s) |
| Uniform (ours) | 86.0 | 436.1 |
| k-center | 82.1 | 564.8 |
Table 3. Effect of the selector on the offline hybrid pipeline, averaged over the five datasets.
Alternative merge strategies. The cascaded merge is only one way to use distillation to fold an arriving batch into the running memory. We compare it against two alternatives (Figure 3(b)). The one-pass merge appends the old memory to the raw new batch and distills the two together in a single step. The history-weighted merge does the same single pass, but weights the old and new gradients by how much data each one stands for.
Table 4 extends the experiment of § 4.3 for OLGA with the alternative merge strategies, at steps. The cascaded merge beats both on average, at 72.0 against 70.2 for one-pass and 69.8 for history-weighted. Appendix A8 reports the per-dataset performance.
| Merge strategy | Avg. stream-AUC |
| Cascaded (ours) | 72.0 |
| One-pass | 70.2 |
| History-weighted | 69.8 |
Table 4. Comparison of online memory-merging strategies, averaged over the five datasets.
Conclusion
Keeping a deployed language model current calls for updates that run on a handful of examples carrying the training signal of all data seen so far. OLGA extends gradient matching for text in three directions: removing the need for labels, following a data stream through a running rehearsal memory, and cutting cost with a hybrid pipeline that selects before it distills. Across five text classification datasets, it outperforms every competing method on average in both the offline and online settings, and our experiments show that each of the three directions matters. Together, they make continual updating of a language model accurate, affordable, and independent of ground-truth labels, offering a practical path toward keeping deployed language model applications current.
AI Use Statement
The research ideas and hypotheses, the method and its formulation in Section 3, the design of the experiments, the implementation of the methods, and the analysis and interpretation of the results are all our own work. Generative AI tools were used only in a supporting role. AI coding agents also wrote secondary engineering code, namely experiment scripts, job launchers, and the plotting code behind our figures, and large language models were used to edit the paper for readability. No generative AI tools were used to generate, clean, or reformat datasets, for translation, or for qualitative data analysis. The synthetic examples reported in the paper are the output of our method run with phi-1.5, not text produced by an AI assistant. We have reviewed all AI-assisted work. We reviewed every piece of code the agents produced. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.
Reproducibility Statement
All experiments use publicly available datasets (SST-2, Rotten Tomatoes, RT-Polarity, IMDB, and TwitterEmotion) and the open-source phi-1.5 model. We will release our full implementation publicly on GitHub upon acceptance, and it reproduces all results reported in the paper.
Ethics Statement
We have read and adhere to the ICLR Code of Ethics. Our experiments use only publicly available text classification datasets and an open-source language model, and involve no human subjects and no data collection of our own. Because the synthetic examples our method produces are built from phrases of the underlying data, they may carry over biases present in that data, which should be considered before deploying the method.
Appendix
Data partitioning and construction of data streams
Each dataset is partitioned into a training split and a validation split. SST-2, Rotten Tomatoes, and TwitterEmotion come with these splits by default, and we use them as provided, while IMDB and RT-Polarity do not, so we construct a fixed split for them once. Within each validation split, we hold out 100 examples, chosen with a fixed seed, as a source pool, and the remaining examples form the evaluation set, which serves every method, step, and experiment, and on which no experiment trains. The offline experiments draw their real anchors from the source pool, and the offline hybrid, which needs a much larger batch, draws it from the training split together with the source pool.
The online experiments stream the training split instead. We fix one random permutation of it and feed it in consecutive disjoint blocks, one block per step. The online hybrid streams the full official training split in the same way but balanced across groups, so each step receives a fresh batch of examples. Only SST-2 and IMDB have training splits large enough for this.
Budgets , , and are counted per group throughout the paper. All five tasks are binary, so each stage actually handles twice the stated number of examples; the offline hybrid batch, for instance, holds examples of each group. Since the budgets before and after each compression stage double together, the compression rates are unchanged, and for simplicity we leave the number of groups out of the notation.
Exact values for the offline results
Bar charts make the offline comparison easy to scan but hard to read exactly, so here we list the values behind § 4.2 in full. Each method reduces 50 real examples to 5, phi-1.5 is fine-tuned on the result, and Table 5 reports its accuracy on the held-out validation split of each dataset, together with its gap to the best selection method.
| Method | IMDB | SST-2 | Rotten Tom. | RT-Polarity | TwitEmo. | Avg. |
| Oracle | 92.3 (+20.2) | 88.2 (+12.9) | 84.7 (+14.3) | 83.2 (+12.8) | 79.4 (+21.9) | 85.6 (+20.3) |
| GradMM [23] | 91.4 (+19.3) | 87.3 (+12.0) | 83.7 (+13.3) | 78.0 (+7.6) | 78.6 (+21.1) | 83.8 (+18.5) |
| OLGA (ours) | 89.5 (+17.4) | 82.3 (+7.0) | 74.5 (+4.1) | 74.6 (+4.2) | 56.5 (−1.0) | 75.5 (+10.2) |
| Prototypical [34] | 66.3 (−5.8) | 65.4 (−9.9) | 68.5 (−1.9) | 68.9 (−1.5) | 57.5 | 65.3 |
| k-center [33] | 72.1 | 75.3 | 60.1 (−10.3) | 60.2 (−10.2) | 55.4 (−2.1) | 64.6 (−0.7) |
| Herding [29] | 65.3 (−6.8) | 61.2 (−14.1) | 69.8 (−0.6) | 69.7 (−0.7) | 56.3 (−1.2) | 64.5 (−0.8) |
| Random | 55.4 (−16.7) | 52.9 (−22.4) | 70.4 | 70.4 | 56.2 (−1.3) | 61.1 (−4.2) |
| Uncertainty [12] | 60.4 (−11.7) | 55.8 (−19.5) | 65.0 (−5.4) | 64.5 (−5.9) | 54.0 (−3.5) | 59.9 (−5.4) |
| Zero-shot | 64.0 (−8.1) | 59.4 (−15.9) | 64.4 (−6.0) | 64.3 (−6.1) | 43.5 (−14.0) | 59.1 (−6.2) |
Table 5. Exact values for Figure 2(a), the offline accuracy of phi-1.5 on each dataset. The upper block gives the supervised upper bounds and the lower block the label-free methods. OLGA beats the best selection method on four of the five datasets and by 10.2 points on average. Bold marks the best label-free result in each column, and underline marks the reference that each colored difference is measured against.
Exact values for the online results
As with the offline results, we list the exact values behind the online comparison of § 4.3 here. Data arrives over steps (8 for IMDB), each method keeps a memory of 5 examples, and Table 6 reports the resulting stream-AUC on each dataset, together with each entry’s gap to the best competing method.
Table 6: Exact values for Figure 2(b), the online stream-AUC of -1.5 on each dataset. Oracle-accumulate keeps all data seen so far and gives the upper bound. OLGA beats the best competing method by 6.6 points on average and comes within two points of supervised GradMM. Bold marks the best result in each column apart from the oracle, and underline marks the reference that each colored difference is measured against.
| Method | IMDB ↑ | SST-2 ↑ | TwitEmo. ↑ | Rotten Tom. ↑ | RT-Polarity ↑ | Avg. ↑ |
| Oracle-accumulate | 92.1 (+4.8) | 88.4 (+10.0) | 87.2 (+27.5) | 84.5 (+20.1) | 84.3 (+20.0) | 87.3 (+18.2) |
| Online GradMM (ours) | 85.4 (+3.8) | 79.2 (+0.8) | 65.3 (+5.6) | 73.2 (+8.8) | 74.3 (+10.0) | 75.5 (+8.5) |
| OLGA (ours) | 85.1 (+3.5) | 74.2 (−4.2) | 59.6 (−0.1) | 74.1 (+9.7) | 74.9 (+10.6) | 73.6 (+6.6) |
| k-center (Sener & Savarese, [33]) | 73.6 (−8.0) | 78.4 | 58.8 (−0.9) | 61.3 (−3.1) | 62.9 (−1.4) | 67.0 |
| Reservoir (Vitter, [39]) | 81.6 | 70.3 (−8.1) | 59.7 | 62.1 (−2.3) | 52.3 (−12.0) | 65.2 (−1.8) |
| Herding (Rebuffi et al., [29]) | 76.1 (−5.5) | 67.7 (−10.7) | 50.5 (−9.2) | 61.3 (−3.1) | 57.1 (−7.2) | 62.5 (−4.5) |
| Prototypical (Snell et al., [34]) | 80.4 (−1.2) | 78.0 (−0.4) | 47.3 (−12.4) | 55.3 (−9.1) | 54.7 (−9.6) | 63.1 (−3.9) |
| Zero-shot | 64.0 (−17.6) | 59.4 (−19.0) | 43.5 (−16.2) | 64.4 | 64.3 | 59.1 (−7.9) |
Table 6.
Extending the offline results to examples
Distillation is designed for extreme compression, so it is worth checking how the picture changes when the budget is less tight. We repeat the offline comparison of § 4.2 at examples, with the same methods and setup as Table 5. Table 7 gives the per-dataset results. The conclusions of § 4.2 carry over, as OLGA remains the strongest label-free method on average, 1.7 points ahead of the best selection method, and supervised GradMM stays close to the oracle. The margin is narrower than at , however, since selection improves faster as the budget grows and already overtakes OLGA on three of the five datasets at . Distillation is strongest under extreme compression while selection catches up as the budget grows, which is the split the hybrid pipeline is built on and is consistent with its results in Table 2.
Table 7: Offline accuracy of -1.5 on each dataset at , the counterpart to Table 5. The upper block gives the supervised upper bounds and the lower block the label-free methods. OLGA stays ahead of the best selection method by 1.7 points on average, but leads on only two of the five datasets. Bold marks the best label-free result in each column, and underline marks the reference that each colored difference is measured against.
| Method | IMDB ↑ | SST-2 ↑ | Rotten Tom. ↑ | RT-Polarity ↑ | TwitEmo. ↑ | Avg. ↑ |
| Oracle | 92.3 (+11.9) | 88.2 (+6.2) | 84.7 (+10.2) | 83.2 (+9.1) | 79.4 (+16.9) | 85.6 (+12.9) |
| GradMM (Nguyen et al., [23]) | 90.6 (+10.2) | 87.2 (+5.2) | 81.6 (+7.1) | 82.6 (+8.5) | 78.9 (+16.4) | 84.2 (+11.5) |
| OLGA (ours) | 89.0 (+8.6) | 78.9 (−3.1) | 71.0 (−4.4) | 76.4 (+2.3) | 57.6 (−4.9) | 74.4 (+1.7) |
| Prototypical (Snell et al., [34]) | 78.8 (−1.6) | 80.1 (−1.9) | 72.2 (−2.3) | 72.0 (−2.1) | 60.2 (−2.3) | 72.7 |
| Herding (Rebuffi et al., [29]) | 77.6 (−2.8) | 78.8 (−3.2) | 71.5 (−3.0) | 71.8 (−2.3) | 58.4 (−4.1) | 71.6 (−1.1) |
| Uncertainty (Lewis & Gale, [12]) | 70.4 (−10.0) | 70.5 (−11.5) | 73.0 (−1.5) | 72.8 (−1.3) | 62.5 | 69.8 (−2.9) |
| k-center (Sener & Savarese, [33]) | 80.4 | 82.0 | 61.1 (−13.4) | 61.6 (−12.5) | 61.0 (−1.5) | 69.2 (−3.5) |
| Random | 68.3 (−12.1) | 66.1 (−15.9) | 74.5 | 74.1 | 62.1 (−0.4) | 69.0 (−3.7) |
| Zero-shot | 64.0 (−16.4) | 59.4 (−22.6) | 64.4 (−10.1) | 64.3 (−9.8) | 43.5 (−19.0) | 59.1 (−13.6) |
Table 7.
What a distilled example looks like
Because OLGA returns plain text rather than weights, its synthetic examples can be read directly. OLGA optimizes each synthetic example so that its training gradient matches that of many real examples at once, and then maps it back to real tokens. The text it settles on therefore reads as a blend of those real examples, taking a phrase from one and another phrase from a different one, and joining them into a sentence that appears nowhere in the data. Tables 8 to 10 show one such example for each of the remaining datasets, in the style of Table 1. In each table, the synthetic example comes first and the real examples it draws from are listed below it. A phrase shared between the two is highlighted in both places, and its superscript gives the number of the real example it comes from.
| Synthetic | IMDLEG is is great1 because it is a well made2 and entertaining story that keeps you3 engaged from beginning4 onwards and the well done production and the talented and well5 done people who6 are involved in making the show are truly a great and7 a |
| Real | 1. This is just a short comment but I stumbled onto this movie by chance and I loved it. The acting is great, the story is simple and touching . . . Seek it out. |
| 2. I thought this was one of those really great films to see with a bunch of close friends. . . . Although not a new concept this was a very well made film. | |
| 3. Powers Boothe turns in a stellar performance as 1970’s cult figure Jim Jones of the Peoples Temple. . . . The movie keeps you riveted and is a must see for anyone. | |
| 4. A funny comedy from beginning to end! . . . Creative story line with a very talented cast. I thoroughly enjoyed it! | |
| 5. terribly underrated with matt dillon and tom skerritt, good backdrop for solid story . . . well acted and well cast, tommy lee jones and bruce dern make you hate them with passion | |
| 6. This movie will always be a Broadway and Movie classic, as long as there are still people who sing, dance, and act. | |
| 7. A great and truly independent film that hit most of my emotions and carried me into another world. . . . |
Table 8. Synthetic example distilled by OLGA for IMDB (positive), with the seven real examples it borrows its phrasing from. A shared phrase is highlighted in both places, and its superscript gives the number of the real example it comes from.
| Synthetic | The new movie was boring and had no interesting or1 well developed action or a good2 one of the actors3 in the story4 |
| Real | 1. what the director can’t do is make either of val kilmer’s two personas interesting or worth caring about. |
| 2. the techno tux is good for a few laughs, as are chan and hewitt, but when such a good design turns out to be a cheap knockoff . . . | |
| 3. sadly, though many of the actors throw off a spark or two when they first appear, they can’t generate enough heat in this cold vacuum of a comedy . . . | |
| 4. the story drifts so inexorably into cliches about tortured (and torturing) artists . . . |
Table 9. Synthetic example distilled by OLGA for Rotten Tomatoes (negative), with the four real examples it borrows its phrasing from. A shared phrase is highlighted in both places, and its superscript gives the number of the real example it comes from.
| Synthetic | Amazing movie review provided by critics is a great1 example of how2 a single positive sentence can make a3 significant impact on the |
| Real | 1. morton is a great actress portraying a complex character, but morvern callar grows less compelling the farther it meanders from its shocking start. |
| 2. a truly moving experience, and a perfect example of how art – when done right – can help heal, clarify, and comfort. | |
| 3. australian actor/director john polson and award-winning english cinematographer giles nuttgens make a terrific effort at disguising the obvious with energy and innovation. |
Table 10. Synthetic example distilled by OLGA for RT-Polarity (positive), with the three real examples it borrows its phrasing from. A shared phrase is highlighted in both places, and its superscript gives the number of the real example it comes from.
Alternative label-free gradient signals
The group-distribution loss equation (3) is one of several ways to read a matching target off the frozen model. Here we give the exact definitions of the two alternatives compared against it in § 4.5, pseudo-token and self-information. Both are built like and , as a per-example cross-entropy toward a target the data supplies. Each substitutes into equation (1) in place of , and the same loss forms both the real and the synthetic gradient.
Pseudo-token. Let be the fixed suffix “In a single word:” and write for the prompt with that suffix appended. The loss is the cross-entropy toward the token the frozen model finds most likely to come next,
The model is asked to name the example in one word and then trained to keep saying that word. Like this puts all the weight on a single token, but the token is whatever the model volunteers rather than a class name we supply.
Self-information. No suffix is added here. Writing with response tokens , the quantity says how unexpected the frozen model found the token at position . The loss is the cross-entropy at the position where it is largest,
The reasoning is that the token the model predicts least well is the one that carries most of what makes the example distinctive.
Two differences from equation (3) follow from these definitions. Both targets above are single tokens, where the group-distribution target spreads over the tokens of ; and both depend on the individual example, where the group-distribution target depends only on the group it belongs to. For the real anchors neither difference costs anything, since every target is computed once under before matching starts. For the synthetic set, the second one does, because the synthetic sequences keep changing while the solver runs, so a target read off a sequence has to be found again as it changes, whereas the group-distribution target is fixed from the outset. This is what separates the three runtimes in Figure 4, and self-information is the most expensive of them because finding its target means scoring every position of every sequence inside the matching loop.
Alternative selectors for the hybrid pipeline
The first stage of the hybrid pipeline only has to thin out the incoming batch, so any selection method can run it. As in § 4.5, we replace uniform sampling in the first stage of the offline GradMM-Hybrid route with k-center, the strongest selection method of Figure 2(b), and keep everything else fixed. Table 11 gives the per-dataset results. Uniform sampling is ahead of k-center on all five datasets, from 2.0 points on IMDB to 8.3 on TwitterEmotion, and is also faster to run. This supports keeping the original distribution of the data at the selection stage and leaving the rest of the compression to distillation.
| Selector | SST-2 ↑ | Rotten Tomatoes ↑ | IMDB ↑ | RT-Polarity ↑ | TwitterEmotion ↑ | Avg. ↑ | SST-2 runtime (s) ↓ |
| Uniform (ours) | 88.2 | 83.9 | 92.3 | 82.1 | 83.4 | 86.0 | 436.1 |
| k-center | 84.2 | 80.9 | 90.3 | 79.8 | 75.1 | 82.1 | 564.8 |
Table 11. Per-dataset version of Table 3, the offline accuracy of the GradMM-Hybrid route with uniform sampling or k-center as its selector. Uniform sampling is more accurate on every dataset and faster to run. Bold marks the better selector in each column.
Alternative merge strategies
The running memory can absorb each arriving batch in more than one way. As in Section 4.5, we compare the cascaded merge against two alternatives for folding each batch into the memory of OLGA, the one-pass and the history-weighted merge, over steps. Table 12 gives the per-dataset stream-AUC. The cascaded merge scores highest on SST-2, Rotten Tomatoes, and RT-Polarity, while one-pass leads on TwitterEmotion and history-weighted on IMDB. Each alternative wins on one dataset at the cost of the others, which leaves the cascaded merge ahead on average, at 72.0 against 70.2 and 69.8.
| Merge strategy | SST-2 ↑ | Rotten Tomatoes ↑ | IMDB ↑ | RT-Polarity ↑ | TwitterEmotion ↑ | Avg. ↑ |
| Cascaded (ours) | 61.8 | 85.4 | 69.1 | 69.5 | 73.9 | 72.0 |
| One-pass | 60.4 | 78.7 | 69.3 | 62.5 | 80.1 | 70.2 |
| History-weighted | 57.2 | 85.3 | 72.4 | 68.1 | 66.0 | 69.8 |
Table 12. Per-dataset version of Table 4, the online stream-AUC of OLGA under each merge strategy. The cascaded merge leads on three of the five datasets and on average. Bold marks the best result in each column.
Memory learnability diagnostics
This section asks whether a cheap property of the memory can tell in advance how well the model will do on the stream, without fine-tuning on it. We measure two such properties of each method’s memory on every dataset. Feature discriminability measures how well the memory’s examples separate the classes in the model’s feature space, and gradient contrast measures how different the training gradients of the classes are. Figure 5 plots each against stream-AUC. Neither one predicts stream-AUC on its own. The least-squares fits explain almost none of the variation, with of 0.03 for feature discriminability and 0.08 for gradient contrast, and memories with nearly the same score on either property end up tens of points apart in stream-AUC. The gradient-matching memories and the oracle do sit in the high-accuracy region, but so do some selection memories whose scores are very different from theirs. How well a memory trains the model therefore cannot be read off a single cheap property, and the choice of method, rather than a proxy score, has to drive how the memory is built.
Figure 5. Feature discriminability (left) and gradient contrast (right) of each memory against its stream-AUC, one point per dataset and method. Neither property predicts how well a memory trains the model.
References
- [1]Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, volume 32, 2019.
- [2]Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1):1–122, 2011.
- [4]Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135, 1999.
- [5]Jianyang Gu, Kai Wang, Wei Jiang, and Yang You. Summarizing stream data for memory-constrained online continual learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(11):12217–12225, 2024.arxiv.org/abs/2305.16645
- [6]Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1416–1428, 2024.arxiv.org/abs/2403.01244
- [7]Siddharth Joshi, Jiayi Ni, and Baharan Mirzasoleiman. Dataset distillation via knowledge distillation: Towards efficient self-supervised pre-training of deep networks. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2410.02116
- [8]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrej A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.DOI
- [9]Jiangtao Kong, Jiacheng Shi, Ashley Gao, Shaohan Hu, Tianyi Zhou, and Huajie Shao. Hybrid memory replay: Blending real and distilled data for class incremental learning. arXiv preprint arXiv:2410.15372, 2024.arxiv.org/abs/2410.15372
- [10]Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. Mind the gap: Assessing temporal generalization in neural language models. In Advances in Neural Information Processing Systems, volume 34, pp. 29348–29363, 2021.arxiv.org/abs/2102.01951
- [11]Dong Bok Lee, Seanie Lee, Joonho Ko, Kenji Kawaguchi, Juho Lee, and Sung Ju Hwang. Self-supervised dataset distillation for transfer learning. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2310.06511
- [12]David D. Lewis and William A. Gale. A sequential algorithm for training text classifiers. In SIGIR ’94, pp. 3–12. Springer, 1994.arxiv.org/abs/cmp-lg/9407020
- [13]Jeffrey Li, Mohammadreza Armandpour, Seyed Iman Mirzadeh, Sachin Mehta, Vaishaal Shankar, Raviteja Vemulapalli, Samy Bengio, Oncel Tuzel, Mehrdad Farajtabar, Hadi Pouransari, and Fartash Faghri. TiC-LM: A web-scale benchmark for time-continual LLM pretraining. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 32231–32273, 2025.arxiv.org/abs/2504.02107
- [14]Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need II: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023.arxiv.org/abs/2309.05463
- [15]Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947, 2018.
- [16]Wenyang Liao, Quanziang Wang, Yichen Wu, Renzhen Wang, and Deyu Meng. Data-Distill-Net: A data distillation approach tailored for reply-based continual learning. arXiv preprint arXiv:2505.20135, 2025.arxiv.org/abs/2505.20135
- [17]Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150, 2011.
- [18]Aru Maekawa, Naoki Kobayashi, Kotaro Funakoshi, and Manabu Okumura. Dataset distillation with attention labels for fine-tuning BERT. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 119–127, 2023.
- [20]Arun Mallya and Svetlana Lazebnik. PackNet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7765–7773, 2018.arxiv.org/abs/1711.05769
- [21]Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pp. 109–165. Elsevier, 1989.DOI
- [22]Brian B. Moser, Federico Raue, Tobias C. Nauen, Stanislav Frolov, and Andreas Dengel. Distill the best, ignore the rest: A study in latent dataset distillation on core-sets. In 2025 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE, 2025.DOI
- [23]Dang Nguyen, Zeman Li, Mohammadhossein Bateni, Vahab Mirrokni, Meisam Razaviyayn, and Baharan Mirzasoleiman. Synthetic text generation for training large language models via gradient matching. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 46008–46025. PMLR, 2025.
- [24]Minh-Duong Nguyen, Thien-Thanh Dao, Le-Tuan Nguyen, Dung D. Le, and Kok-Seng Wong. Memory-efficient continual learning with prototypical exemplar condensation. arXiv preprint arXiv:2603.13804, 2026.arxiv.org/abs/2603.13804
- [25]Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Victor Zacarias, Edson Bollis, Lucas Pellicer, Rosimeire Pereira Costa, Anna Helena Reali Costa, and Artur Jordao. Technical report on text dataset distillation. arXiv preprint arXiv:2512.03967, 2025.arxiv.org/abs/2512.03967
- [26]Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pp. 115–124, 2005.DOI
- [27]Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. In Advances in Neural Information Processing Systems, volume 34, pp. 20596–20607, 2021.arxiv.org/abs/2107.07075
- [28]Ameya Prabhu, Philip H. S. Torr, and Puneet K. Dokania. GDumb: A simple approach that questions our progress in continual learning. In Computer Vision – ECCV 2020, pp. 524–540. Springer, 2020.DOI
- [29]Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001–2010, 2017.arxiv.org/abs/1611.07725
- [30]Andrea Rosasco, Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Distilled replay: Overcoming forgetting through synthetic samples. In Continual Semi-Supervised Learning, pp. 104–117. Springer, 2022.arxiv.org/abs/2103.15851
- [31]Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.arxiv.org/abs/1606.04671
- [32]Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3687–3697, 2018.DOI
- [33]Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
- [34]Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1703.05175
- [35]Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642, 2013.DOI
- [36]Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
- [37]Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. GCR: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 99–108, 2022.arxiv.org/abs/2111.11210
- [38]Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019.arxiv.org/abs/1904.07734
- [39]Jeffrey S. Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software, 11(1):37–57, 1985.DOI
- [40]Mengting Wan, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, Chirag Shah, Ryen W. White, Longqi Yang, Reid Andersen, Georg Büscher, Dhruv Joshi, and Nagu Rangan. TnT-LLM: Text mining at scale with large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5836–5847, 2024.arxiv.org/abs/2403.12173
- [42]Yue Xu, Yong-Lu Li, Kaitong Cui, Ziyu Wang, Cewu Lu, Yu-Wing Tai, and Chi-Keung Tang. Distill gold from massive ores: Bi-level data pruning towards efficient dataset distillation. In Computer Vision – ECCV 2024, pp. 240–257. Springer, 2024.arxiv.org/abs/2305.18381
- [43]Min Zeng, Haiqin Yang, Wei Xue, Qifeng Liu, and Yike Guo. Dirichlet continual learning: Tackling catastrophic forgetting in NLP. In Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, volume 244 of Proceedings of Machine Learning Research, pp. 4096–4108. PMLR, 2024.
- [44]Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3987–3995. PMLR, 2017.arxiv.org/abs/1703.04200
- [45]Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514–6523, 2023.arxiv.org/abs/2110.04181
- [46]Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021.
- [47]Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. WildChat: 1M ChatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2405.01470
- [48]Zheng Zhou, Wenquan Feng, Qiaosheng Zhang, Shuchang Lyu, Qi Zhao, and Guangliang Cheng. ROME is forged in adversity: Robust distilled datasets via information bottleneck. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 78687–78713. PMLR, 2025.



