betaFeedback

WorksComputer Science

Implicit Assumptions About Tasks in Continual Learning

Abstract

Continual learning is the study of how a model accumulates knowledge from data received over time, rather than all at once. In practice, this is typically achieved by dividing a dataset into tasks and presenting them sequentially. Explanations of the resulting measurements take into account factors such as the similarity between segments, the rate at which the data changes and the order of presentation. Each of these factors can be defined in terms of the underlying data, the partition imposed on it, or the learner that encounters it. These levels do not necessarily coincide, and it is usually left implicit which level a reported quantity refers to. We separate these and identify which properties acquire a value at each level. In controlled teacher–student experiments, we demonstrate that partitioning the same underlying trajectory at different levels of granularity, changing the order of presentation, and varying the similarity structure of the underlying data all result in different measured outcomes. We review representative benchmarks to determine which of these properties are stated. Where the underlying data cannot be inspected, as in foundation models built on undisclosed pre-training corpora, its properties are inherited by the model's initial state, and the training applied on top of it can be described.

Introduction

Continual learning (CL) refers to a model’s ability to update its knowledge and skills as new data arrives over time, as well as retaining knowledge from past data [44, 45]. Performance metrics, such as transfer and forgetting, are evaluated on a model trained across a data or experience stream, which is a realized ordered sequence of functions to be learnt (Figure 1(a)). Typically, this stream is split into disjoint parts, tasks, and the learner is sequentially permitted access to only one task at a time during training. Transfer then measures how the model performs as it transitions from task to task, and forgetting measures how well it performs on previous tasks as it learns newer ones.

One stream, many partitions

Figure 1. One stream, many partitions. (a) A stream is a realized, ordered sequence. Colour denotes content of experience; nearby colours indicate related experience. (b) A task cut produces a task sequence. Segment width = task scope. (c) Change within a task depends on scope. (d) Similarity between tasks depends on scope: coarse segments span heterogeneous stretches and overlap; fine segments are narrow and largely disjoint. S(KN,CN)S(K_N, C_N) is the similarity of two tasks under a partition into NN tasks. (e) The same segments in a different order are a different stream, since a stream is a realized order. None of (c)-(e) is a property of the stream alone; each is dependent on stream & cut. Everything here is learner independent. Figure 2 shows what changes once a learner is present.

This standard approach requires deciding how, or whether, to define a task cut, the partition of a data stream into separate tasks. In practice the strategies for laying boundaries between tasks vary widely. In classification scenarios, tasks can be disjoint subsets of classes to be learnt [57, 40] or different input transformations with a shared label space [13], while reinforcement learning typically defines tasks as new goals or skills to be acquired by an agent [54, 30]. Alternatively, streaming benchmarks vary the distribution of data to be modeled continuously, and either leave the task cut undefined [7, 48] or impose it by time period [29]. Wherever drawn, the cut fixes how much experience falls within each task and how much the learnt function changes inside it (Figure 1(b,c)), how similar the resulting tasks are to one another (Figure 1d), and the order in which they are presented, which need not follow the stream’s own progression (Figure 1(e)).

Classical CL taxonomies do not account for the task cut, instead differentiating between scenarios based on the availability of task identity during training and testing as well as whether the label space is shared or expanded across tasks [44]. These differences have guided most experimental approaches to CL over the past years, yielding many insightful and practically useful results. However, it has become increasingly clear that the task cut cannot be ignored when attempting to understand forgetting and transfer in CL models as changing the boundaries of tasks changes the outcome [11]. A stream and a cut are in turn meaningless without a learner attempting to model the data presented. Here again, scope, or the history a CL learner has access to, similarity, now within the representational space of the learner rather than the functional space of the stream, and change, in terms of the update rules applied to the learner, are critical to the final forgetting or transfer exhibited.

In summary, a (final) CL setting is a realized stream, a task cut on it, and a learner moving through the stream. Notions such as scope, similarity, and change are defined differently at these three levels and affect forgetting and transfer differently. Benchmarks usually define settings near the stream (classes, domains, transforms). However, much of the theory that predicts forgetting quantitatively works near the learner: NTK overlap [9], representational similarity [39], feature/readout decomposition [24] (Hiratani 2024), node re-use vs activation [25], representational geometry [33, 21].

Contributions. In this work, we describe critical properties of the three levels at which CL operates and provide a framework by which to compare different benchmarks and experimental paradigms. Specifically we:

  • a) Separate and conceptualize the properties such as scope, order, similarity and change at the stream, task and learner levels (§ 2, Table 1).

  • b) Show that in a controlled teacher–student setting partitioning the trajectory differently, reordering

PropertyStream (§2.1)Task Cut (§2.2)Learner (§2.3)
ScopeSupport of the realized stream (data)Support of a task/segment induced by the partitionSupport of the learner’s trajectory in function space. Access: the learner’s filtration
OrderRealized order at sample or batch levelOrder of the task labels induced by the partitionWhich learner state encounters which segment
SimilarityA function of time points, D(t,t′)D(t,t'), or an aggregateIntra- and inter-task similarityDistance in hypothesis or representation space
ChangeGradual change or drift over timeDiscrete change across task boundaries, and intra-task change induced by partitionState displacement or learning dynamics induced by the update rule

Table 1. Properties of the realized stream, task discretization, and learner.

the resulting tasks, and varying the distance the stream traverses each change measured forgetting (§ 2, Figure 2).

A learner enters the setting

Figure 2. A learner enters the setting. (a) A learner occupies one point in a stream at a time. Its lifetime scope is the breadth of experience the stream covers as a whole, of which only the past is available to it and the future is unobserved. (b) The same stream under a single partition. Similarity is defined for every pair of tasks and computable from the data once the boundaries are fixed. What a learner retains of each task is a separate quantity (§ 2.3). (c–e) A teacher–student stream in which the teacher’s weights follow a fixed trajectory, with the student trained on tasks sampled along it. Curves show mean squared test error on each task after the whole stream has been traversed, shaded by standard error over five initialisations, with task position on the horizontal axis. Full setup in Sec. B. (c) Partition granularity, N∈{5,10,50}N \in\{5, 10, 50\}, which alters similarity between tasks and change within them together. (d) Presentation order at N=50N = 50. Reversing the stream reverses the profile of retained error across it, and shuffling removes its structure. (e) Distance traversed by the teacher across the stream at N=50N = 50, which sets the overall similarity between its regions.

c) Derive the consequence for attribution and for measurement metrics themselves, which are defined over segments (§ 2.4)

d) Review representative benchmarks against these properties, showing which are specified, which are left implicit, and where the omissions make reported comparisons unsupported (§ 3).

We do not claim a complete description and formalization of the listed properties. Extending the experiments to standard benchmarks and to learner-level properties, and specifying and testing the consequences we draw for foundation models, are left to future work. Complexity and impossibility results exist for the continual learning problem as usually posed [23, 36], taking the partition as given. Defining tasks is also contested outside ML [14].

Continual Learning: The three levels

Prior CL work has explored numerous properties of how the data is presented to the learner such as order, similarity, or rate of change (or drift). These analyses are often conducted on a predetermined partition of the dataset, and hence fail to distinguish whether the conclusions hold for the actual data stream or merely for the selected partition. Furthermore, a subset of literature also demonstrates these properties to be heavily dependent on the actual learning rule and underlying hypothesis space [8, 52]. Outside of the task-dependent framework the literature investigating these properties is scattered across different sub-fields of CL. Therefore, we introduce a three-level taxonomy – stream, task, and learner, with the distinction of the properties shown in Table 1.

In this section, we introduce our taxonomy, re-contextualizing related work and demonstrating practical implications of our framework in a simplified toy experimental setting. Inspired by the student-teacher setup used in Lee et al. [ [24], we train a 2-layer ReLU MLP, the student, to reproduce the outputs of a smaller two-layer ReLU MLP, the teacher. To specify a continual task stream, we gradually shift the weights of the teacher MLP along a smooth trajectory that can be specified arbitrarily through a series of linearly interpolated way-points in weight space. Since local shifts in weight space result in locally bounded changes to the teacher’s function, this simulates a realistic continuously shifting task stream while also enabling control over the stream’s scope, resolution and order. For simplicity, we use only plain Adam optimization on the student, focusing these experiments mostly on how task cuts and modifying properties affect what the learner retains across the stream in a simple student network. Details regarding the implementation of every experiment can be found in Appendix B, with results shown in Figure 2(c)-(Figure 2e) and Figure 3.

Final MSE test loss across normalized stream time for changing teacher parameters and stream scope

Figure 3. (a) Changing different parameters in the teacher has different effects on forgetting in the student. (b) Changing the stream scope (i.e., the intrinsic dimensionality of the teacher’s weight trajectory) affects forgetting in the learner.

Stream

Learning is sequential for a non-methodological reason because a learner encounters experience in one order and cannot revisit it except through what it has retained in its parameters, in context, or in an explicit buffer [50]. We use the term stream for a realized, ordered sequence, not the distribution it could have been drawn from (Figure 1(a)). Many regimes are specified distributionally, as multiple epochs over a dataset, or repeated draws from a task distribution. However, the learner only encounters this distribution through this realized stream and the final outcome depends on the order of data which has been available.

Scope of the realized stream. The scope describes the stream’s data support, i.e. the breadth of distinct experience the stream covers as a whole. Benchmarks name the source a stream is drawn from, but no standard measure of its diversity exists, so the scope is bounded without being quantified, while influencing the lower and upper bounds of transfer and forgetting. In empirical CL literature, scope ranges from a single digit dataset (e.g. Split-MNIST) to a mixture of generalist and specialized datasets as in continual pre-training of foundational models [19]. Our teacher-student setup allows for changing the scope of the task stream by increasing the intrinsic dimensionality across which the weights of the teacher move through the stream. Changes in stream scope affecting the learner’s retention are demonstrated in Appendix § 7.

Realized order. The order describes which experience precedes which (e.g., the ordering of samples or batches). Order arises from the environment’s dynamics in natural settings, from the learner’s own behavior in embodied and interactive settings, and from the experimenter in benchmarks. Stream-level order is rarely remarked on by CL literature, but rather accounted for implicitly, aggregating statistics over multiple random number generator seeds, inducing different data point orderings. In our teacher-student setup, we can switch the order of a task stream by traversing it in the forward or reverse direction. Results to this effect (along with results regarding a shuffled presentation once a task cut is decided) are shown in Figure 2(d). Other changes to how waypoints in the teacher’s weight space are traversed would necessarily alter the underlying nature of the stream itself.

Similarity. Stream-level similarity measures how different one sample or region is from another, ignoring the observer and learner. This requires a metric on experience itself, which can then be used to make comparisons. The choice of space this metric lives in (parameter, data, function) needs to be further given, and Mitchell et al. [2025] note that it is sometimes forced, as when differing architectures rule out parameter-space distance. Webb et al. [2016] make the abstraction explicit, defining drift against an unspecified distance function and suggesting Hellinger or KL as instances. These measures are often asymmetric, how informative one region is about another need not equal the reverse.

Change. The temporal geometry of the similarity can be abrupt, gradual, cyclic, compositional and not necessarily evenly distributed along the stream. In concept drift literature, this has been described as the drift rate at some time step tt [53]. The effective learning regime depends jointly on change and similarity [15]. In our setup, we are able to manipulate the overall similarity along with the rate of change within the stream by increasing the maximum distance that is traversed by the teacher weights along the stream trajectory. Figure 2(e) demonstrates that increasing this distance, equivalent to decreasing the similarity of any region in the task stream to any other region, increases the final test loss across all tasks, our measure of retained performance.

Partitioning and tasks

A stream comes with no predetermined notion of tasks or boundaries. In fully online settings none are required, since the properties above describe the stream as a whole. However, partitioning the stream into segments can often be indispensable for prediction and evaluation. In benchmark construction the choice is explicit. Two MNIST digits are one such segment, the next two another. In natural settings however there is no innate partition, and segments a researcher describes are a retrospective device. Task-free CL methods can automatically locate boundaries [1, 45], but still require a distinction criterion supplied by the experimenter, with different criteria locating different boundaries in the same underlying stream. There is also no privileged granularity: locomotion, running, or extending a leg are defensible units for the same behaviour. Segments carrying the same label form an equivalence class we call a task. The label is not given by the stream and requires either an external (annotation, class partition, designed task set) or internal (measured similarity under a threshold) criterion. The scope of a task is the support of the subset of the realized stream associated with the task (Figure 1(b)).

Task similarity and change. With the task cuts determined, inter-task similarity becomes well-defined and measurable by distributional [2, 31] or functional distance [24] (Hiratani 2024). Different granularities of partitioning render different inter-task similarities and change (Figure 1(c,Figure 1d)). This model-agnostic notion should not be conflated with the model-dependent similarity discussed in § 2.3.

Task order. Once tasks exist, the order of their instances can be manipulated, yielding a different stream (Figure 1(e)). Inter-task order is how instances of different tasks interleave. Intra-task order or exposure is how instances of one task are distributed across the stream, massed, spaced or recurring. Blocked (contiguous per task) and interleaved (tasks alternate) are two points on this axis, which also includes spacing and number of traversals. Prior literature has already documented the effect of reordering [5, 35], and of task similarity [24] (Hiratani 2024) on forgetting. However, order and task similarity cannot be cleanly manipulated independently from each other. Ramasesh et al. [ [39] demonstrate that the preceding tasks determine whether high inter-task similarity leads to more or less forgetting. Lin et al. [ [28] analytically describe the expected forgetting and generalization error as a function of both similarity and order, while Li and Hiratani [ [27] provide an analytical reordering scheme based on the similarity structure. Task similarity also affects the performance of interleaved and blocked presentations, coinciding with human learning behavior [34].

In our setup, we vary the granularity of the task partition on a fixed teacher trajectory under a constant compute budget, effectively changing both inter-task similarities and intra-task change. Figure 2(c) shows that simply discretizing the task stream more finely can alter the profile of retained loss across the stream, such that more dissimilar regions to the final task exhibit a higher test loss after training while more similar ones (even if they occur much earlier in the task stream) are preserved better than under coarser discretizations. Figure 2(d) further investigates the effect of task order, revealing again a dependence of the shape of forgetting across the task stream on the order in which tasks were presented. Randomly shuffling the order removes any relation between a region’s position in the stream and how well it is retained. In contrast, reversing the order of task presentation clearly reverses the shape of forgetting across regions of the stream.

The learner

With a learning agent in the picture, the relevant domain shifts from data to the hypothesis space [47] where the stream carves a learning trajectory, an ordered distribution over this space. New concepts emerge, such as capacity, inductive biases, architecture, optimizer, optional pretrained representations, and the choice of updated component (e.g., in-weight vs in-context adaptation [10], drifting representation vs stable readout [46]). Since each decomposition is built for different purposes, they do not easily translate across domains, e.g., in language the representation and the task-solving component are not separable as in vision. As with the stream and task level, we can map scope, order, similarity, and change to the learner:

Scope and access. The learner’s scope is the support of the stream induced trajectory (Figure 2). The learner’s access includes the accessible stream history (filtration) of the stream and possibly cuts/task identities until the present time. Prior literature categorizes the accessibility to task identity as task, class and domain-incremental setups [18, 44]. For learners accessing the task identity, the task support is consequential: for instance, Prabhu et al. [ [38] report continual prompt-tuning methods collapse as the number of classes per task is reduced to one.

Agent-side similarity, order and change. Similarity for the agent refers to model and trajectory dependent measures like representational or NTK overlap [9, 39]. This combines the agent properties with the task cut, therefore cannot be reported independently of both. Can also be viewed as distributional distance on the function space induced by a metric e.g., gradient or parameter overlap. The update rule specifies the change in the trajectory in response to the stream and possibly the cuts. Under replay, samples retained from earlier segments are re-inserted into the sequence the update rule receives, so the realized stream of Sec. 2.1 and the data the learner trains on come apart. Two settings that share a stream and a cut, but present the segments in a different order, place each segment against a different accumulated learner state.

Effective capacity and difficulty. The regime in which the learner is operating is a consequence of the stream’s demands relative to the capacity. This can be a result of parameter count relative to the stream or due to hard (scope restriction) or soft (modifying tails) inductive biases. Under a learning strategy that uses boundaries only for evaluation, this is fixed by stream and architecture alone. As an example, in overparameterised settings the network accommodates the entire stream, so no region of it strains capacity. What then varies across conditions is not the stream but the task construction, and it is that similarity the forgetting results are reported against [39, 24, 25] (Hiratani 2024). A related composite of effective capacity and dataset is difficulty, conceptualizing how well the task can be represented in the model’s function space. Concretely, the minimum KL divergence between the optimum likelihood and the true task distribution. The same task can be trivial for one learner and hard for another, which is why difficulty cannot be reported by only a benchmark [43].

A crucial metric that allows the learner to optimise its strategy is predictability, the ability to anticipate what comes next: the future trajectory bounded by the stream’s properties, but realized only relative to what this learner has retained and can potentially extrapolate from. Predictability determines the worth of retention as where the future is unrelated to the past, retention can cost capacity without benefit, where it is closely related retention is directly helpful. Figure 2(a) shows the sequential nature of learning, standing at one point in the stream with past behind and future unobserved.

The attribution consequence

What follows is that an outcome in CL is a joint function of the setting and the learner agent.

outcome⁡=f(stream, cut, access⏟setting; agentt),agentt=g(agent0, stream<t, access<t).\operatorname{outcome}=f\left(\underbrace{\mathrm{stream},\ \mathrm{cut},\ \mathrm{access}}_{\text{setting}};\ \mathrm{agent}_{t}\right),\qquad\mathrm{agent}_{t}=g\left(\mathrm{agent}_{0},\ \mathrm{stream}_{<t},\ \mathrm{access}_{<t}\right).

The setting therefore matters directly, as the conditions of measurement, and indirectly, through the learner those conditions produced. Two studies sharing a stream and cut but ordering segments differently are evaluating two different learners. Even when holding the method fixed, results are not directly comparable across settings, so a statement like "method A beats method B" compares learners whose difference is not exhausted by A versus B. For example, Ibrahim et al. [ [19] show that what they define as a stronger shift (Pile→German Common Crawl) requires more replay than what they define as a weaker one (Pile→SlimPajama) to mitigate forgetting to the same extent, so a reported percentage bundles a stream property with the learner’s objective and accumulated state.

Forgetting, backward and forward transfer are computed from an accuracy matrix whose indices are the cut. The consequence of this is that per-sample loss and accuracy change with the cut only if the learning strategy is affected. This predicts that where the update rule does not see the cut, loss and accuracy should be invariant while forgetting and transfer are not, since the latter are defined over segments chosen after training. Boundaries are therefore necessary for metrics even where they are not necessary for training. In a fully online setting they are not a training requirement. Standard CL scenarios differ in how they access the cut: Single-head online SGD only sees the cuts as annotations. Multi-head task-IL uses the cut to select the active head and routes gradients by it. Class-IL fixes the output space by it. Boundary-triggered operations like EWC Fisher computation [22] and LwF teacher snapshot [26] fire on it, changing the objective at the boundary. Replay is the strongest case, since the cut decides what enters and leaves the buffer and therefore which data the learner sees at all. This yields the prediction that where two cuts induce the same schedule and evaluation reference, results should agree. Consistent with this, Filat et al. [2026] hold stream, model and budget fixed and see forgetting and backward transfer move with the split.

Given a fixed stream, cuts can be chosen to favor retention, plasticity, or whatever the objective demands. Task construction is consequential even where the construction also alters what each task contains: Jhajj and Lin [2025] report roughly 9.8 percentage points more forgetting under relation-based than under random partitioning of a knowledge graph, and conclude independently that protocols should report their partitioning strategy. There is no existing selection procedure for the best way to partition tasks. Filat et al. [2026] propose a measure of how sensitive a stream is to where its boundaries fall, but do not use it to choose them, and Li and Hiratani [2025], Bell and Lawrence [2022] optimise order with the task set already given. Any such procedure depends on the objective, a cut minimising interference does not have to be the same as one maximising transfer.

What Benchmarks Record

We reviewed CL benchmarks against the properties of § 2, recording for each property whether the benchmark’s construction specifies it (Sec. A). Benchmarks built by splitting a classification dataset into class subsets, or by transforming its inputs, are inexpensive, reproducible and comparable across papers [57, 40, 13]. Benchmarks specify a procedure for constructing a stream rather than a realized stream, holding many factors fixed by construction, which is what makes them controlled instruments. Much of what is reliably known about forgetting and transfer was obtained with them. Which data are drawn on is fixed, but in synthetic benchmarks the assignment of classes to tasks and the order in which tasks are presented are commonly drawn per run and averaged over seeds, so two studies reporting results on Split-CIFAR100 need not have trained on the same task sequence. Benchmarks built from temporal data fix the order instead [29]. Split-CIFAR-100 is run with anywhere from 2 to 50 classes per task [40, 44], and its standard random split does not state how closely the classes within a task resemble one another. A method performing well on it may be performing well on closely related or on unrelated tasks. In line with these observations, Verwimp et al. [2023] survey benchmarks and conclude that most represent extreme rather than typical CL problems; Lin et al. [2021] report that protocols training and testing on i.i.d. data inflate measured performance; Jhajj and Lin [2025] find that how tasks are constructed affects measured difficulty and recommend that protocols state their partitioning strategy.

Access. Whether task identity is supplied, at which stage, and whether transitions between tasks are signaled is fixed by every benchmark we reviewed, though not always stated. The three often cited standard scenarios [44] focus on task identity at test time, and class-incremental learning additionally requires discrimination across all classes seen so far. For Permuted MNIST, the scenario follows only from the output layer shared across tasks, which Van de Ven et al. [ [44] classify as domain-incremental.

Similarity and scope. Split benchmarks state how classes are grouped, while permutation benchmarks draw a random input permutation for each task [22], which can therefore differ between studies. While a reader can infer that some pairs of tasks are more closely related than others, they cannot determine by how much or in which respect, since inputs may overlap where labels do not. Where specified, Liu et al. [ [30] declares which axis shifts, and Ibrahim et al. [ [19] additionally grade their shifts as weak or strong. The only quantitative measure, forward transfer between every pair of tasks [54], is defined for a reference agent and therefore lies at the learner level (§ 2.3). Scope is conveyed similarly, where the number of classes per task is known but not comparable across benchmarks. The exception among the reviewed is Liu et al. [ [30], which declares per suite which factors vary, though without a magnitude. An instruction-tuning dataset [51] and a discrimination between two handwritten digits [57] are each called one task.

Change and exposure. Among the benchmarks we reviewed, these two properties are not varied independently. Most benchmarks with abrupt transitions present experience in blocks that can be revisited. The other combinations occur in Lin et al. [ [29] and Yao et al. [ [56], which present gradually drifting data in blocks, while CLAD-C and CL-Bench contain sudden shifts within an online stream [48, 3]. No reviewed benchmark varies one property while holding the other fixed on the same stream, so that an outcome cannot be attributed to either alone.

To conclude, comparing methods on a fixed benchmark does reveal differences in how learners behave. However, what such a comparison establishes is local to that benchmark, and to generalize the behavior, a reader would need values for all properties on which the settings differ.

Foundation Models and Beyond

In foundation models the properties of § 2 are harder to report, since the division into tasks may have no counterpart in how the model was trained. Surveys divide CL for foundation models into stages, contiguous training periods under one objective and one data source: continual pre-training, domain-adaptive pre-training, continual fine-tuning and alignment [42, 55], or continual pre-training, fine-tuning and compositionality [4]. Naming them mirrors the operation of § 2.2, with the surveys positioning the divisions differently. The stages mix two criteria: continual and domain-adaptive pre-training change the data source, which is a cut in the sense of § 2.2, while alignment changes the objective and therefore the update rule, which Table 1 places at the learner level. Forgetting reported across a stage boundary confounds a cut-level with a learner-level change. Alongside stages, Shi et al. [ [42] classify shifts during continual pre-training as temporal, content-level or language-level. A temporal shift may span a month or a decade and a content shift may cover adjacent domains or unrelated ones, so two studies reporting the same type of shift may have faced very different problems. Two settings have to be separated. Where continual pre-training is itself the object of study, experimenters choose the corpora, their mixture and order [19], stream-level properties remain reportable, and the constraint of a limited replay buffer is replaced by compute budget, since i.i.d. mixing reduces the throughput of novel data [37]. Where a released checkpoint is the starting point, as in continued multilingual pre-training [41] or evaluation of deployed models [51], the pre-training corpus is usually unpublished [49], and where corpus and order are released the order is a shuffle [6], so even fully disclosed data carry no meaningful stream-level order. Here the stream-level properties are fixed before the study begins and enter it through agent0_0. The adaptation stream and its cut are specifiable, while the scope of experience the checkpoint covers remains a black-box assumption on the learner state, on which any reported forgetting is conditioned (§ 2.4). Reusing the checkpoint holds that scope fixed across studies, which makes agent-side similarity, defined relative to a learner and a cut (§ 2.3), comparable between them. Ibrahim et al. [ [19] match full re-training when a new corpus arrives as a mixture of domains but report poor results when the same corpus is consumed one domain at a time, and find that the stronger shift requires more replay than the one they classify as weaker. Continued multilingual pre-training still degrades general-purpose benchmarks, requiring further alignment stages [41]. Proposals for the field’s future lie at the learner level, whether compositionality, the recombination of components [4], or memory operations deciding what to store, when to consolidate or retrieve and what to discard [10]. Asawa et al. [2026] report naive in-context learning outperforming systems built around such operations on tasks whose instances are constructed to share a latent structure. What a memory policy should store depends on the predictability of the stream it runs on (§ 2.3).

Outlook. Any system that keeps learning after an initial training phase learns continually. As a learner is a product of the setting in which it was trained (§ 2.4), attributing a performance difference to a method requires knowledge of that setting, which is why we ask for the properties of § 2 to be routinely reported. Together they describe a space of learning problems. A robot restricted to loading a dishwasher in a kitchen and a model deployed across changing domains both learn continually, over different scopes of experience. What a learner should retain and what it should revise depends on how that experience is structured, and the relation is not necessarily monotonic [39, 25] (Hiratani 2024). Such a description would also help the learner to estimate its lifetime scope, to then decide what and how much to retain. Revising that estimate changes which experiences it treats as similar and how its capacity is best spent. In interactive and agentic settings the stream is generated in part by the learner’s own behavior, so setting and learner are jointly determined rather than composed as in § 2.4. Scope becomes a property of the policy, and two methods evaluated on the same agentic benchmark traverse different streams. The field lacks a means of predicting expected forgetting and transfer for a given stream, cut and learner. Currently, a replay fraction that works on one corpus pair, or a method that performs well on one split of CIFAR, gives little generalisation. Hinder et al. [2020] formalise how a data distribution evolves over time, which supplies the stream alone. Mitchell et al. [2025] propose a different approach, algorithmic invariance to permutations that carry no information; we consider the two approaches complementary as identifying which permutations these are requires the specification we request.

Acknowledgements

This project was supported by a European Research Council (ERC) grant (GA no. 101045094, project "GROW-AI"), and by Helmholtz Association’s project-oriented funding programme (PoF 2, Topic 3). Aspects of this project originated from interactive discussions and hands-on working groups at the CapoCaccia Neuromorphic Intelligence Workshop (https://capocaccia.cc), which the authors gratefully acknowledge.

Benchmark review: full entries

We selected benchmarks that are widely used in CL and that together span classical synthetic partitions, purpose-built streams, interactive settings and foundation-model training. The selection is illustrative rather than exhaustive, and the entries below describe each benchmark’s standard protocol.

Table 2: Benchmark families against the properties of Sec. 2. Superscripts denote the status of the entry (✓ specified; ◦ not stated by the benchmark but our reading of its description, judged recoverable from the released data and not computed here; ? unspecified and not recoverable from the protocol, with a marker alone indicating that the protocol gives no value). For Similarity, ✓ marks that the benchmark states how tasks were constructed, which constrains the kind of relatedness without providing a magnitude. Two of the reviewed benchmarks provide more information. Ibrahim et al. [ [19] grade their two shifts as weak or strong, and Wołczyk et al. [ [54] report forward transfer between every pair of tasks for a reference agent, a learner-level quantity that they also use to set the task order. For Scope, a class count fixes how many labels fall in a task without stating the breadth of experience they cover, and is recorded as ?; only Liu et al. [ [30] is marked with ✓, for declaring per suite which factors are held fixed and which vary. The Order/Exposure column records the presentation regime, while the realized ordering within it is typically drawn per run rather than fixed by the benchmark. The last two columns, Task identity at test time and Boundaries, are two facets of the learner’s access. Each entry is recorded at whichever level the benchmark specifies.

BenchmarkScopeOrder/ExposureSimilarityChangeTask identityBoundaries
Classical synthetic partitions
Split MNIST [Zenke et al., [57]; Van de Ven et al., [44]]2 classes per task ?blocked ✓label-disjoint, pairs fixed or per run ✓abrupt ✓given or withheld ✓given ✓
Split CIFAR-100 [Rebuffi et al., [40]; Van de Ven et al., [44]]2–50 classes per task ?blocked ✓label-disjoint, grouping per run ✓abrupt ✓given or withheld ✓given ✓
Permuted MNIST [Goodfellow et al., [13]; Kirkpatrick et al., [22]]50–61 layers ?blocked ✓permutation fixed or per run ✓abrupt ✓given or withheld ✓given ✓
Purpose-built streams
CORe50 [Lomonaco and Maltoni, [32]]scenario-dependent ?shuffled batch order ✓same objects across sessions ✓abrupt at switch ✓none (single head) ✓batch-marked ✓
CLAD [Kumar et al., 2023]6 classes ?online, chronological ✓class and domain shift ✓gradual + abrupt ✓none ✓none ✓
CLEAR [Lin et al., [29]]11 classes ?offline per bucket ✓concepts persist ✓gradual ✓none ✓buckets imposed ✓
CLO-Cai [Cai et al., [7]]712 locations ?online, single-pass ✓not stated ?unspecified ?none ✓none ✓
Wild-Time [Yao et al., [56]]varies by dataset ?bit protocol ✓not stated ?gradual, timestamped ✓none ✓timestamps ✓
Interactive / embodied
Continual World [Wołczyk et al., [54]]manipulation goal ?sequential ✓shared spaces, pairwise transfer ✓abrupt at switch ✓given (multi-head) ✓given ✓
LIBERO [Liu et al., [30]]fixed factors per suite ✓sequential ✓declared axes (3 of 4 suites) ✓abrupt at switch ✓language instruction ✓given ✓
Foundation-model era
TRACE [Wang et al., [51]]one dataset = one task ?blocked, fixed order ✓heterogeneous ?abrupt dataset switch ✓in-context ✓given ✓
Continual pretraining [Ibrahim et al., [19]]corpus ?shuffled, single pass ✓shift graded weak/strong ✓abrupt corpus switch ✓none ✓dataset-marked ✓
Continual SLM [Fini et al., [12]]split-dependent ?blocked, many epochs ✓not stated ?test split type ✓setting-dependent ✓required ✓
CL-Bench [Asawa et al., [3]]domain task ?online, episodic ✓latent structure shared ✓task-specific shifts ✓none ✓instance-marked ✓

Table 2. Benchmark families against the properties of Sec. 2. Superscripts denote the status of the entry (✓ specified; ◦ not stated by the benchmark but our reading of its description, judged recoverable from the released data and not computed here; ? unspecified and not recoverable from the protocol, with a marker alone indicating that the protocol gives no value). For Similarity, ✓ marks that the benchmark states how tasks were constructed, which constrains the kind of relatedness without providing a magnitude. Two of the reviewed benchmarks provide more information. Ibrahim et al. [ [19] grade their two shifts as weak or strong, and Wołczyk et al. [ [54] report forward transfer between every pair of tasks for a reference agent, a learner-level quantity that they also use to set the task order. For Scope, a class count fixes how many labels fall in a task without stating the breadth of experience they cover, and is recorded as ?; only Liu et al. [ [30] is marked with ✓, for declaring per suite which factors are held fixed and which vary. The Order/Exposure column records the presentation regime, while the realized ordering within it is typically drawn per run rather than fixed by the benchmark. The last two columns, Task identity at test time and Boundaries, are two facets of the learner’s access. Each entry is recorded at whichever level the benchmark specifies.

Toy experiments

To investigate how continuous task streams and the task cut affect a learner’s forgetting, we deploy a simple synthetic teacher-student toy setup. In real-world CL settings, data streams are often high-dimensional, complex shifts of data distributions generated by opaque processes. This teacher-student setup attempts to model such continuous distribution shifts through explicitly configured, continuous weight trajectories Θ(t)\Theta(t) of a reference teacher network fΘ(t)f_{\Theta(t)}. The teacher fΘ(t)f_{\Theta(t)} yields a task stream that can be discretized by sampling target datasets (Xn,Yn)(X_n, Y_n) along the weight trajectories, where properties such as the trajectory distance (reflective of the stream scope), resolution of the discretization (which influences the per-task scope and per-task change), presentation order, and parameter subspace alignment (which allows independent manipulation of the label space, input space or representational space of the teacher network) are controllable and can be evaluated.

Teacher architecture and weight trajectory construction

The teacher network fΘ(t):R10→R2f_{\Theta(t)}: \mathbb{R}^{10} \to\mathbb{R}^{2} is a multi-layer perceptron (MLP) with 2 hidden layers of 32 ReLU neurons. This results in a total parameter dimension of D=1474D = 1474. Parameters evolve continuously over normalized stream time t∈[0,1]t \in[0, 1] according to low-dimensional keypoint trajectories w(t)∈Rd\mathbf{w}(t) \in\mathbb{R}^{d} (d≪Dd \ll D). To lift these waypoints into the full teacher weight space RD\mathbb{R}^{D}, we project them using P∈RD×dP \in\mathbb{R}^{D \times d} (with unit-norm columns) sampled from N(0,ID)\mathcal{N}(0, I_D), defining the parameter evolution as Θ(t)=Θ0+Pw(t)\Theta(t) = \Theta_0 + P\mathbf{w}(t), where Θ0\Theta_0 is also sampled from N(0,ID)\mathcal{N}(0, I_D).

Stream discretization

To evaluate the impact of discretization coarseness, a continuous teacher weight trajectory is partitioned into NN discrete tasks sampled at tn=n/(N−1)t_n = n/(N - 1). For each task nn, the teacher fΘ(tn)f_{\Theta(t_n)} is passed inputs xnx_n sampled from N(0,I10)\mathcal{N}(0, I_{10}) and generates outputs yny_n which are used as labels to train the learner. The learner’s training data budget across the stream is fixed at 10, 000 samples, allocated evenly across the sampled tasks, so that the total data budget and the total number of gradient steps are constant across discretization levels. In addition, we use exactly the same underlying weight trajectory for every discretization level (ensured by always using the same random seed during trajectory generation). We evaluate discretization levels N∈{5,10,50}N \in\{5, 10, 50\}, corresponding to 2,000, 1, 000, and 200 training samples per task. For testing, a test set of 200 samples is independently generated for each task nn.

Figure 2(c) shows the results of changing the task discretization averaged over student networks generated with 5 different seeds.

Task presentation order

For a fixed discretization of N=50N = 50 and using the same training and test setup as in the previous section, we evaluate three task presentation orders:

  • Forward: The same sequence as the stream trajectory n=0,1,…,N−1n = 0, 1, \ldots, N - 1.

  • Reverse: The inverted trajectory sequence n=N−1,N−2,…,0n = N - 1, N - 2, \ldots, 0.

  • Shuffled: A random permutation of tasks.

Figure 2(d) covers the results from this experiment, again over 5 different seeds for the student network.

Selective teacher shift

To determine which components of the teacher’s function most affect forgetting in the student, we selectively restrict the parameter trajectory shifts to specific weight layers in the teacher while freezing non-targeted parameters at their initial values Θ0\Theta_0:

  • Input-Hidden: Only layer 1 weight matrix W(1)W^{(1)} and bias b(1)b^{(1)} evolve over stream.

  • Hidden-Hidden: Only intermediate layers change while input and output layers are fixed.

  • Hidden-Output: Only output layer weight matrix W(3)W^{(3)} and bias b(3)b^{(3)} change over the stream.

Fig. 3a demonstrates the effect of changing the label space by shifting Hidden-Output weights versus shifting the input or hidden representational space of the teacher. Consistent with readout-similarity effects in Lee et al. [ [24], we see that shifts in the label space cause the learner to forget more than shifts in the input or hidden representational spaces of the teacher.

Stream scope

To test how the scope or the support of the task stream influences student forgetting, we expand the intrinsic dimensionality of the weight trajectory manifold w(t)w(t) from 1D to 3D. Added dimensions are generated by random Brownian motion multiplied by a linear decay envelope (1−t)(1-t) to make sure that the trajectory ends at the exact same parameter target at t=1t=1 as it started with at t=0t=0. We made this choice to visualize how similarity between tasks does not necessarily depend on temporal adjacency in the stream; however other choices could have been made.

fig:3b demonstrates the effect on the learner of changing the teacher weight trajectory’s intrinsic dimensionality, and thus the stream scope. The only clear qualitative trend that shines through is that the higher dimensional trajectories seem harder to retain than the simple 1D trajectory projected into the teacher’s weight space. We acknowledge that increasing the intrinsic dimensionality of the trajectory does not change scope in isolation, but also lengthens the path the teacher traverses, alters pairwise distances between tasks and changes the local rate of drift. The observed effect should therefore be read as the effect of this joint change rather than of scope alone.

Student architecture and evaluation

The student network is an MLP with layers [10,128,128,2][10, 128, 128, 2] trained with Adam (η=0.01\eta=0.01, batch size 32) for 10 epochs per task, evaluated across 5 independent random initializations. We report only the final stream loss (LN−1,nL_{N-1,n}), which is the MSE test loss on task nn evaluated after the student model has completed training on the complete task stream.

References

  1. [1]Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 11246–11255. IEEE, 2019.arxiv.org/abs/1812.03596
  2. [2]David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33:21428–21439, 2020.
  3. [3]Parth Asawa, Christopher M Glaze, Gabriel Orlanski, Ramya Ramakrishnan, Benji Xu, Asim Biswal, Vincent Sunn Chen, Frederic Sala, Matei Zaharia, and Joseph E Gonzalez. Continual learning bench: Evaluating frontier ai systems in real-world stateful environments. arXiv preprint arXiv:2606.05661, 2026.arxiv.org/abs/2606.05661
  4. [4]Jack Bell, Luigi Quarantello, Eric Nuert ey Coleman, Lanpei Li, Malio Li, Mauro Madeddu, Elia Piccoli, and Vincenzo Lomonaco. The future of continual learning in the era of foundation models: Three key directions. arXiv preprint arXiv:2506.03320, 2025.arxiv.org/abs/2506.03320
  5. [5]Samuel J Bell and Neil D Lawrence. The effect of task ordering in continual learning. arXiv preprint arXiv:2205.13323, 2022.arxiv.org/abs/2205.13323
  6. [6]Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International conference on machine learning, pages 2397–2430. PMLR, 2023.arxiv.org/abs/2304.01373
  7. [7]Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribution shifts: An empirical study with visual data. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8261–8270. IEEE, 2021.arxiv.org/abs/2108.09020
  8. [8]Hung-Jen Chen, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. Mitigating forgetting in online continual learning via instance-aware parameterization. Advances in Neural Information Processing Systems, 33:17466–17477, 2020.
  9. [9]Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rausseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference on Artificial Intelligence and Statistics, pages 1072–1080. PMLR, 2021.arxiv.org/abs/2010.04003
  10. [10]Vaggelis Dorovatas, Malte Schwerin, Andrew D Bagdanov, Lucas Caccia, Antonio Carta, Laurent Charlin, Barbara Hammer, Tyler L Hayes, Timm Hess, Christopher Kanan, et al. Position: Modular memory is the key to continual learning agents. arXiv preprint arXiv:2603.01761, 2026.arxiv.org/abs/2603.01761
  11. [11]Nicolao Filat, Ahmed Hussain, Konstantinos Kalogiannis, and Elena Bourceanu. Temporal task-ification in streaming continual learning: A source of evaluation instability. arXiv preprint arXiv:2604.21930, 2026.arxiv.org/abs/2604.21930
  12. [12]Enrico Fini, Victor G Turrisi Da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual learners. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9611–9620. IEEE, 2022.arxiv.org/abs/2112.04215
  13. [13]Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.arxiv.org/abs/1312.6211
  14. [14]Davood Gozli. What is a task? In Davood Gozli, editor, Experimental Psychology and Human Agency, pages 83–111. Springer Verlag, 2019.DOI
  15. [15]Jacopo Graldi, Alessandro Breccia, Giulia Lanzillotta, Thomas Hofmann, and Lorenzo Noci. The importance of being lazy: Scaling limits of continual learning. arXiv preprint arXiv:2506.16884, 2025.arxiv.org/abs/2506.16884
  16. [18]Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines, 2019. URL https://arxiv.org/abs/1810.12488.
  17. [19]Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. arXiv preprint arXiv:2403.08763, 2024.arxiv.org/abs/2403.08763
  18. [21]W Jeffrey Johnston and Stefano Fusi. Modular representations emerge in neural networks trained to perform context-dependent tasks. bioRxiv, 2024.DOI
  19. [22]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521–3526, 2017.DOI
  20. [23]Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and is np-hard. In International Conference on Machine Learning, pages 5327–5337. PMLR, 2020.arxiv.org/abs/2006.05188
  21. [24]Sebastian Lee, Sebastian Goldt, and Andrew Saxe. Continual learning in the teacher-student setup: Impact of task similarity. In International Conference on Machine Learning, pages 6109–6119. PMLR, 2021.arxiv.org/abs/2107.04384
  22. [25]Sebastian Lee, Stefano Sarao Mannelli, Claudia Clopath, Sebastian Goldt, and Andrew Saxe. Maslow’s hammer for catastrophic forgetting: Node re-use vs node activation. arXiv preprint arXiv:2205.09029, 2022.arxiv.org/abs/2205.09029
  23. [26]Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., 40(12):2935–2947, December 2018. ISSN 0162-8828. doi: 10.1109/TPAMI.2017.2773081. URL https://doi.org/10.1109/TPAMI.2017.2773081.
  24. [27]Ziyan Li and Naoki Hiratani. Optimal task order for continual learning of multiple tasks. arXiv preprint arXiv:2502.03350, 2025.arxiv.org/abs/2502.03350
  25. [28]Sen Lin, Peizhong Ju, Yingbin Liang, and Ness Shroff. Theory on forgetting and generalization of continual learning. In International Conference on Machine Learning, pages 21078–21100. PMLR, 2023.arxiv.org/abs/2302.05836
  26. [29]Zhiqui Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on real-world imagery. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 2), 2021.arxiv.org/abs/2201.06289
  27. [30]Bo Liu, Yifeng Zhu, Changkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776–44791, 2023.arxiv.org/abs/2306.03310
  28. [31]Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea Soltoggio, and Soheil Kolouri. Wasserstein task embedding for measuring task similarities. Neural Networks, 181:106796, 2025.
  29. [32]Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In Conference on robot learning, pages 17–26. PMLR, 2017.arxiv.org/abs/1705.03550
  30. [33]Nicholas Menghi, W Jeffrey Johnston, Simone Vigano’, Max Andreas Bosse Hinrichs, Burkhard Maess, Stefano Fusi, and Christian F Doeller. The effects of task similarity during representation learning in brains and neural networks. Nature Communications, 2025a.DOI
  31. [34]Nicholas Menghi, Simone Vigano’, WJ Johnston, Salma Elnagar, S Fusi, and FC Doeller. Impact of task similarity and training regimes on cognitive transfer and interference. bioRxiv, pages 2025–09, 2025b.
  32. [35]Rupert Mitchell, Antonio Alliegro, Raffaello Camoriano, Dustin Carrión-Ojeda, Antonio Carta, Georgia Chальvatzaki, Nikhil Churamani, Carlo D’Eramo, Samin Hamidi, Robin Hesse, et al. Continual learning should move beyond incremental classification. arXiv preprint arXiv:2502.11927, 2025.arxiv.org/abs/2502.11927
  33. [36]Binghui Peng and Andrej Risteski. Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions. Advances in Neural Information Processing Systems, 35: 28414–28427, 2022.arxiv.org/abs/2203.14383
  34. [37]Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. Online continual learning without the storage constraint, 2023. URL https://arxiv.org/abs/2305.09253.
  35. [38]Ameya Prabhu, Shiven Sinha, Ponnurangam Kumaraguru, Philip H Torr, Ozan Sener, and Puneet K Dokania. Randumb: Random representations outperform online continually learned representations. Advances in Neural Information Processing Systems, 37:37988–38006, 2024.DOI
  36. [39]Vinay V Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv preprint arXiv:2007.07400, 2020.arxiv.org/abs/2007.07400
  37. [40]Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE conference on computer vision and pattern recognition (CVPR), pages 5533–5542. IEEE, 2017.arxiv.org/abs/1611.07725
  38. [41]Ricardo Rei, Nuno M Guerreiro, José Pombal, João Alves, Amin Farajian, Pedro Teixeirinha, and André FT Martins. Tower+: Bridging generality and translation specialization in multilingual llms. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29614–29635, 2026.
  39. [42]Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. Continual learning of large language models: A comprehensive survey. ACM Computing Surveys, 58(5):1–42, 2025.arxiv.org/abs/2404.16789
  40. [43]Bohan Tang, Huidong Liang, Keyue Jiang, and Xiaowen Dong. On the importance of task complexity in evaluating llm-based multi-agent systems, 2025. URL https://arxiv.org/abs/2510.04311.
  41. [44]Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185–1197, 2022.DOI
  42. [45]Gido M. van de Ven, Nicholas Soures, and Dhireesha Kudithipudi. Continual learning and catastrophic forgetting, page 153–168. Elsevier, 2025. ISBN 9780443157554. doi: 10.1016/b978-0-443-15754-7.00073-0. URL http://dx.doi.org/10.1016/B978-0-443-15754-7.00073-0.
  43. [46]Suzanne van der Veldt, Gido M van de Ven, Sanne Moorman, and Guillaume Etter. Learning continually with representational drift. Trends Open, 2025.
  44. [47]Vladimir Vapnik. The Nature of Statistical Learning Theory, volume 8, pages 1–15. Springer science & business media, 01 2000. ISBN 978-1-4419-3160-3. doi: 10.1007/978-1-4757-3264-1_1.
  45. [48]Eli Verwimp, Kuo Yang, Sarah Parisiot, Lanqing Hong, Steven McDonagh, Eduardo Pérez-Pellitero, Matthias De Lange, and Tinne Tuytelaars. Clad: A realistic continual learning benchmark for autonomous driving. Neural Networks, 161:659–669, 2023.DOI
  46. [49]Alexander Wan, Kevin Klyman, Sayash Kapoor, Nestor Maslej, Shayne Longpre, Betty Xiong, Percy Liang, and Rishi Bommasani. The 2025 foundation model transparency index. arXiv preprint arXiv:2512.10169, 2025.arxiv.org/abs/2512.10169
  47. [50]Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, 46(8):5362–5383, 2024.DOI
  48. [51]Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, et al. Trace: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.06762, 2023.arxiv.org/abs/2310.06762
  49. [52]Xulong Wang, Yu Zhang, Tong Liu, Zhipeng Yuan, Kang Liu, Vitaveska Lanfrānchi, and Po Yang. Spot: An efficient training-free task similarity quantification method for continual learning. Pattern Recognition Letters, 197:168–174, 2025. ISSN 0167-8655. doi: https://doi.org/10.1016/j.patrec.2025.07.018. URL https://www.sciencedirect.com/science/article/pii/S0167865525002703.
  50. [53]Geoffrey J Webb, Roy Hyde, Hong Cao, Hai Long Nguyen, and Francois Petitjean. Characterizing concept drift. Data Mining and Knowledge Discovery, 30(4):964–994, 2016.DOI
  51. [54]Maciej Wołczyk, Michał Zając, Razvan Pascanu, Łukasz Kuciński, and Piotr Miłos. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems, 34:28496–28510, 2021.
  52. [55]Yutao Yang, Jie Zhou, Xuanwen Ding, Tianyu Huai, Shunyu Liu, Qin Chen, Yuan Xie, and Liang He. Recent advances of foundation language models-based continual learning: A survey. ACM Computing Surveys, 57(5):1–38, 2025.arxiv.org/abs/2405.18653
  53. [56]Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei W Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. Advances in Neural Information Processing Systems, 35:10309–10324, 2022.arxiv.org/abs/2211.14238
  54. [57]Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. Pmlr, 2017.arxiv.org/abs/1703.04200

Paper details

vlnc:W_zkjcvpus
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents