WorksComputer Science
DuoCL: Dual-Path Online Continual Learning
Abstract
Online continual learning (OCL) requires learning from a data stream whose arrival rate is independent of model training speed. When training cannot keep pace, computationally expensive gradient-based methods update less frequently, limiting adaptation. Analytic methods reduce this overhead, but their frozen backbones cannot adapt representations to changing distributions. We propose DuoCL, a dual-path OCL framework whose fast analytic path accumulates feature statistics from every incoming minibatch while its slow path periodically adapts the backbone. The slow path adjusts its sample count to the available compute budget without interrupting fast-path learning. Feature alignment compensates for drift in the accumulated classifier statistics. Across four datasets and three computational budgets, DuoCL improves online accuracy by an average of 9.67 percentage points over the strongest gradient-based baseline in each setting. DuoCL consistently outperforms analytic learning under distribution shifts, with gains of up to 4.00 percentage points on CIFAR-C, and achieves higher accuracy than prior methods across disjoint class-incremental, domain-incremental, and Si-Blurry settings.
Introduction
Continual learning (CL) [14, 16, 47] aims to enable machine learning models to learn incrementally from non-stationary data streams while mitigating catastrophic forgetting [16, 42, 57] of previously acquired knowledge. In conventional offline CL, models are trained on a sequence of tasks, typically allowing multiple training epochs over each task’s data before proceeding to the next task [31, 55]. In contrast, online continual learning (OCL) processes a continuous stream of data, with model updates performed under limited computational and memory budgets [10].
Existing OCL methods primarily focus on mitigating catastrophic forgetting through experience replay [13, 28], regularization [1, 8, 9], knowledge distillation [34], and sampling strategies [3, 4, 68]. However, these methods have different computational requirements, which can limit how frequently models update when data arrives independently of model training time. A recent benchmark [18] demonstrates that computationally expensive methods may miss learning opportunities when model training cannot keep up with the data arrival rate, with a simple baseline outperforming more sophisticated OCL methods.
Recent work [75] has explored approaches to support rapid adaptation under computational constraints. System-level approaches employ fine-grained pipeline parallelism to process high-frequency data streams, using gradient compensation to mitigate stale gradients. Other methods reduce training computation through selective [59] and adaptive layer freezing [58], restricting gradient-based updates to selected backbone layers. Analytic CL methods [6, 17, 43, 48, 61] [20] offer a different approach by freezing the backbone and updating the classifier in closed form using accumulated feature statistics. However, restricting backbone updates can limit representation plasticity, particularly when pretrained features cannot adequately capture the incoming data distribution. This motivates a central question: Can a model keep pace with the data arrival rate while periodically adapting its backbone representations under limited computational and memory budgets?
We propose DuoCL, a dual-path OCL framework that combines continuous updates to analytic statistics with periodic gradient-based backbone adaptation (Fig. 1). The fast path accumulates feature statistics for every incoming minibatch, while the slow path adapts the backbone using recent samples. Coordinating these paths under limited computational and memory budgets requires determining when slow-path training occurs and how backbone updates are incorporated without interrupting fast-path learning. DuoCL addresses this through a bounded buffer of recent samples and a periodic slow-path update schedule. However, backbone adaptation changes the feature representations underlying previously accumulated classifier statistics, introducing representation drift. To mitigate this drift, DuoCL employs feature alignment to maintain consistency between the updated backbone and the analytic classifier. Our main contributions are as follows:
Figure 1. Comparison of OCL approaches under a computational budget. (a) Frozen analytic methods update a closed-form classifier from accumulated feature statistics and for every incoming minibatch at low cost, but the fixed backbone limits adaptation. (b) Gradient-based OCL methods adapt the backbone, but must skip minibatches for training when their computational demand exceeds the budget (). (c) DuoCL accumulates feature statistics for every incoming minibatch through the fast path, while the slow path periodically adapts the backbone using samples retained from the previous window and aligns and with the updated features.
We propose DuoCL, a dual-path OCL framework that combines continuous analytic statistics updates with periodic gradient-based backbone adaptation.
We develop a budget-aware slow path that supports periodic backbone adaptation under computational and memory constraints using bounded buffers, with feature alignment to mitigate representation drift between the updated backbone and analytic classifier statistics.
Across four datasets and three computational budgets, DuoCL improves online accuracy by an average of 9.67 percentage points over the strongest gradient-based baseline in each setting.
DuoCL consistently outperforms analytic learning under distribution shifts, with gains of up to 4.00 percentage points on CIFAR-C, and achieves higher accuracy than prior methods across disjoint class-incremental, domain-incremental, and Si-Blurry settings.
Related Work
Compute-constrained online continual learning. OCL methods mitigate forgetting through replay, regularization, distillation, and sample selection [2, 4, 9, 13, 34], or adapt prompts and low-rank adapters of pretrained models online [46, 66] (Fig. 1b). These methods typically assume that training on each minibatch completes before the next minibatch arrives. [18] evaluates OCL methods without this assumption and shows that a method times more expensive than experience replay can train on only one in every minibatches. Under such computational constraints, simple replay often outperforms more sophisticated methods [15, 18, 51, 71]. Budget-aware methods reduce the cost of each gradient update through layer freezing [58, 59] or throughput-oriented replay [64], while system-level approaches schedule retraining [7, 29, 30] or distribute backpropagation across additional pipeline workers [75]. These approaches still learn from incoming minibatches through backpropagation, so minibatches are either skipped or processed with additional computational resources. In contrast, DuoCL learns from every incoming minibatch by accumulating feature statistics using features already computed during inference, incurring little additional computation, and restricts backpropagation to a slow path whose budget is determined by the data arrival rate and device throughput (§ 4.1).
Analytic continual learning. Analytic CL methods replace iterative classifier training with closed-form ridge-regression or discriminant solutions computed from accumulated feature statistics [22, 43, 45, 48, 76]. These methods are exemplar-free and, for a fixed feature extractor, equivalent to joint training on all observed data [76]. Recursive formulations update the classifier online for every minibatch [53, 77], and the same statistics have been used in federated CL [6, 17, 61] [20]. Such methods perform strongly under limited online budgets [52, 53], but they rely on a frozen pretrained backbone, since updating it would invalidate the accumulated statistics. They therefore cannot adapt their representations when the incoming data deviate from the pretraining distribution. DuoCL addresses the limitation by adapting the backbone throughout the stream while aligning the accumulated statistics with backbone updates.
Combining analytic and gradient-based learning. Fast and slow learners motivated by complementary learning systems theory [41] typically train both components with backpropagation, differing in learning rate or objective [5, 49, 70]. Other methods combine a training-free or analytic classifier with gradient-based tuning [72, 74], while drift compensation methods correct stored class statistics after backbone updates [19, 39, 56, 69], including through @page 2/12 starts_mid=0 ends_mid=1
Figure 1 (PDF p. 2). Comparison of OCL approaches under a computational budget. (a) Frozen analytic methods update a closed-form classifier from accumulated feature statistics and for every incoming minibatch at low cost, but the fixed backbone limits adaptation. (b) Gradient-based OCL methods adapt the backbone, but must skip minibatches for training when their computational demand exceeds the budget (). (c) DuoCL accumulates feature statistics for every incoming minibatch through the fast path, while the slow path periodically adapts the backbone using samples retained from the previous window and aligns and with the updated features.
We propose DuoCL, a dual-path OCL framework that combines continuous analytic statistics updates with periodic gradient-based backbone adaptation.
We develop a budget-aware slow path that supports periodic backbone adaptation under computational and memory constraints using bounded buffers, with feature alignment to mitigate representation drift between the updated backbone and analytic classifier statistics.
Across four datasets and three computational budgets, DuoCL improves online accuracy by an average of 9.67 percentage points over the strongest gradient-based baseline in each setting.
DuoCL consistently outperforms analytic learning under distribution shifts, with gains of up to 4.00 percentage points on CIFAR-C, and achieves higher accuracy than prior methods across disjoint class-incremental, domain-incremental, and Si-Blurry settings.
Related Work
Compute-constrained online continual learning. OCL methods mitigate forgetting through replay, regularization, distillation, and sample selection [2, 4, 9, 13, 34], or adapt prompts and low-rank adapters of pretrained models online [46, 66] (Fig. 1b). These methods typically assume that training on each minibatch completes before the next minibatch arrives. [18] evaluates OCL methods without this assumption and shows that a method times more expensive than experience replay can train on only one in every minibatches. Under such computational constraints, simple replay often outperforms more sophisticated methods [15, 18, 51, 71]. Budget-aware methods reduce the cost of each gradient update through layer freezing [58, 59] or throughput-oriented replay [64], while system-level approaches schedule retraining [7, 29, 30] or distribute backpropagation across additional pipeline workers [75]. These approaches still learn from incoming minibatches through backpropagation, so minibatches are either skipped or processed with additional computational resources. In contrast, DuoCL learns from every incoming minibatch by accumulating feature statistics using features already computed during inference, incurring little additional computation, and restricts backpropagation to a slow path whose budget is determined by the data arrival rate and device throughput (§ 4.1).
Analytic continual learning. Analytic CL methods replace iterative classifier training with closed-form ridge-regression or discriminant solutions computed from accumulated feature statistics [22, 43, 45, 48, 76]. These methods are exemplar-free and, for a fixed feature extractor, equivalent to joint training on all observed data [76]. Recursive formulations update the classifier online for every minibatch [53, 77], and the same statistics have been used in federated CL [6, 17, 61] [20]. Such methods perform strongly under limited online budgets [52, 53], but they rely on a frozen pretrained backbone, since updating it would invalidate the accumulated statistics. They therefore cannot adapt their representations when the incoming data deviate from the pretraining distribution. DuoCL addresses the limitation by adapting the backbone throughout the stream while aligning the accumulated statistics with backbone updates.
Combining analytic and gradient-based learning. Fast and slow learners motivated by complementary learning systems theory [41] typically train both components with backpropagation, differing in learning rate or objective [5, 49, 70]. Other methods combine a training-free or analytic classifier with gradient-based tuning [72, 74], while drift compensation methods correct stored class statistics after backbone updates [19, 39, 56, 69], including through a closed-form least-squares transformation of second-order statistics [24, 54]. These methods are designed for task-based CL, where the backbone is trained for multiple epochs on each task and the statistics are corrected once per task using the full task data. SIESTA [21] updates its output layer online and consolidates the full network in periodic offline sleep phases, which assume scheduled downtime with no incoming data. To our knowledge, DuoCL is the first to combine analytic and gradient-based learning concurrently within a single-pass data stream. The fast path accumulates feature statistics for every incoming minibatch, the slow path adapts the backbone within the computational budget in every window, and a closed-form transformation estimated from a small set of retained samples aligns the accumulated classifier statistics with the backbone after each update, without requiring task boundaries, multiple training epochs, or interruptions to the stream.
Method
Problem Definition
Online continual learning. Following [10, 18], we consider a data stream arriving at samples per second and grouped into minibatches of samples. Each minibatch arrival defines a time step , with consecutive arrivals separated by seconds. At step , the model predicts for the incoming minibatch , drawn from a distribution that may change over time. The ground-truth labels are then revealed to evaluate the predictions, after which the model may learn from subject to a fixed computational budget and a memory budget of at most stored samples. Online performance is evaluated on every arrival, including minibatches skipped for training. Figure 2 illustrates the stream and an example training schedule.
Figure 2. Problem Definition: Minibatches of samples arrive every seconds. With , training on spans three arrival intervals, so and are skipped for training. Prediction continues on every arrival.
Computational budget. Minibatches arrive independently of training progress, so the computation that training can use between consecutive arrivals depends on both the stream rate and the device throughput. For a device providing FLOP/s for training, this computational budget per time step is
We specify directly to express different stream rates and hardware capabilities through a common computational budget. Let denote the training FLOPs a method requires per time step to learn from every incoming minibatch. We characterize its demand relative to this budget by
Update frequency. When , a method’s training procedure can keep pace with the stream. When , queueing every incoming minibatch for training would create a growing backlog that eventually exceeds the fixed memory budget. Maintaining this training procedure therefore requires skipping some minibatches. The average number of minibatches skipped per update is
Predictions continue on every arrival using the latest available model, with parameter updates taking effect only after their computation completes.
DuoCL Overview
As illustrated in Figure 3, DuoCL consists of two coordinated learning paths: a fast path for analytic classifier learning and a slow path for periodic backbone adaptation. For each incoming minibatch, the backbone extracts features, which the fast path incorporates into accumulated classifier statistics. The analytic classifier is periodically reconstructed from these statistics. The slow path is triggered every incoming minibatches, with each set of minibatches forming a window. During each window, the slow path adapts the backbone using samples retained from the previous window, while a new subset of samples is collected for the next update. Since backbone weight updates change the feature representations, DuoCL aligns the accumulated classifier statistics and updates the classifier.
Figure 3. Overview of DuoCL over two consecutive windows of minibatches. The fast path predicts every minibatch and updates and after label revelation. Reservoir sampling collects up to current-window samples in , while the slow path adapts the backbone using previous-window samples in . At the next window boundary, feature alignment updates the accumulated statistics and the classifier is rebuilt.
Fast Path: Online Analytic Learning
In OCL, predictions must be made for every incoming minibatch, even when the computational and memory budgets prevent training on all incoming samples. Since online prediction already requires extracting backbone features, DuoCL’s fast path reuses these features and the subsequently revealed labels to accumulate feature statistics from every minibatch.
To achieve this, we adopt the ridge-regression formulation used in analytic continual learning [43, 45, 48]. We maintain two accumulated statistics: a Gram matrix that captures second-order feature information, and a matrix that accumulates class-wise feature information. For each incoming minibatch containing samples, let denote the extracted features, where is the feature dimension. Predictions are made using the latest available classifier : . After observing the ground-truth labels, let denote the one-hot encoding of the labels, where is the number of classes. We update the accumulated statistics as and , where and are initialized to zero. Each column of accumulates features belonging to a particular class, while captures second-order feature statistics used to construct the classifier. The resulting ridge-regression classifier has the closed-form solution
where is the regularization coefficient. While and can be updated incrementally using the extracted features, reconstructing requires solving a linear system, with computational complexity for a conventional dense solve. Performing this operation after every arrival can be expensive under high data arrival rates. To reduce this cost, DuoCL updates and for every incoming minibatch but reconstructs once every minibatches. This amortizes the classifier reconstruction cost while allowing every incoming minibatch to contribute to the accumulated statistics. Unlike conventional frozen-backbone analytic learning, DuoCL also periodically adapts the backbone, requiring the accumulated statistics to be aligned with the updated representations, as in § 4.5.
Slow Path: Periodic Backbone Adaptation
Although the fast path incorporates every incoming minibatch into the analytic classifier statistics, it does not adapt the backbone. Performing backpropagation on every incoming minibatch can exceed the available computational budget. To address this, DuoCL periodically adapts the backbone using a bounded subset of recent samples.
Budget-Aware Sample Selection and Scheduling. The slow path is triggered every incoming minibatches, with each group of minibatches forming a window. During each window, collects incoming samples, while the slow path adapts the backbone using samples retained in from the previous window. We first determine the number of samples retained in each buffer based on the available computational and memory budgets. With samples per minibatch, each window contains samples and provides a total computational budget of . The slow path performs gradient steps using retained samples. Let denote the profiled FLOPs required for one minibatch-equivalent of slow-path training. The retained sample count is constrained by
where denotes the available computation budget over a window. Since both buffers can coexist, their combined storage must also satisfy . Accordingly, we determine the maximum retained sample count as
Since each window contains samples, which may exceed the buffer size , we employ uniform reservoir sampling to select a subset for backbone adaptation. The first samples are stored in . For each subsequent sample , we retain it with probability , replacing a uniformly selected sample in the buffer. This ensures that every sample in the window has an equal probability of being retained while the buffer size remains bounded by . Meanwhile, the slow path performs gradient steps using samples from . At the end of the window, the backbone update is applied, the previous samples are discarded, and the buffers are updated as and . Thus, DuoCL performs backbone adaptation using recently retained samples without maintaining a persistent historical replay buffer.
Slow-Path Training Objective. Let denote the retained samples and their corresponding labels in . For backbone adaptation, we employ three complementary loss terms:
where and control the contributions of the auxiliary and drift losses, respectively. The main cross-entropy loss adapts the backbone features using the existing analytic classifier:
where denotes the features of the retained samples, denotes their labels, and is a fixed copy of the analytic classifier. During slow-path training, gradients propagate through the classifier to update the backbone, while remains unchanged.
To provide an additional supervised signal for backbone adaptation, we employ a training-only auxiliary classifier , with loss
However, backbone updates can also change predictions for previously observed classes. To limit these changes, we employ drift regularization that penalizes deviations from the logits computed before backbone adaptation:
where denotes the features before adaptation and denotes previously observed classes absent from the retained samples. To maintain a consistent loss scale across updates, we normalize the drift penalty by the number of retained samples and previously observed classes when applying its weight . The drift loss is zero when is empty. For CNN-based backbones, the slow path directly updates the backbone parameters. For Transformer-based backbones, we employ low-rank adaptation (LoRA) [27], keeping the pre-trained backbone weights frozen while updating the LoRA adapters in the query and value projections. Both configurations use the same slow-path training objective.
Drift-Aware Feature Alignment
Slow-path backbone adaptation changes the feature representations used to accumulate the classifier statistics and . Consequently, directly reusing these statistics with the updated backbone can introduce inconsistencies between the classifier and the new feature space. Recomputing the statistics from all previously observed samples is impractical under the computational and memory constraints of OCL. To address this, inspired by prior work on feature space drift compensation [24, 67], DuoCL estimates a feature transformation using the retained samples and aligns the accumulated statistics with the updated backbone.
For the retained samples , we extract features before and after backbone adaptation: and , where and denote the backbone parameters before and after adaptation, respectively. We estimate an old-to-new feature transformation using regularized least squares:
where is the regularization coefficient that encourages the transformation to remain close to the identity matrix. This prevents excessive changes to the feature space when estimating the transformation from a limited number of retained samples. The optimization admits the closed-form solution
Rather than recomputing the classifier statistics from previously observed samples, we transform the accumulated statistics into the updated feature space: and . The transformation is applied to the accumulated statistics, including minibatches processed while the slow-path update was pending. Finally, the analytic classifier is reconstructed from the aligned statistics using Eq. (4). This allows DuoCL to retain information accumulated from the incoming stream while periodically adapting its backbone representations.
Experiment
Experimental Setup
Dataset and Baseline
Our evaluation covers ten widely used image classification benchmarks. CIFAR-10 and CIFAR-100 [33] are standard image classification benchmarks. CLEAR-10 and CLEAR-100 [37] capture the natural temporal evolution of visual concepts, while CLOC [10] provides a large-scale stream with naturally occurring distribution shifts. CIFAR-C [25] evaluates robustness under common image corruptions. ImageNet-R [26] and ImageNet-S [63] evaluate visual distribution shifts through artistic renditions and sketches, respectively. CUB-200 [62] provides a fine-grained classification benchmark covering 200 bird species. CORe50 [38] evaluates continual object recognition across sessions with varying viewpoints and environments.
We compare DuoCL against representative baselines: ER [13], ACE [9], LwF [34], RWalk [11], PoLRS [10], MIR [2], GSS [4], and aL-SAR [58]. For pretrained ViT-based comparisons, we compare against AGEM [12], ER [13], MIR [2], GDumb [50], DER++ [8], PCR [36], LODE (with DER++ [8]) [35], EMA (with DER++ [8] and with RAR [73]) [60], EWC++ [11], L2P [65], MVP [46], Online-LoRA [66], and FGH [44]. Additional implementation details, including dataset splits and hyperparameter configurations, are provided in the supplementary material.
Metrics
We evaluate DuoCL using three metrics: (i) average online accuracy [18], the percentage of incoming samples correctly classified before learning their labels, including samples skipped for training; (ii) final accuracy [40, 66], computed after training on the final task; and (iii) area under the curve of accuracy [32], which measures model performance throughout training. We report for the online setting, for disjoint class-incremental and domain-incremental settings, and both and for Si-Blurry. See Appendix for detailed definitions.
Results
Impact of Data Arrival Rate
To evaluate performance under varying data arrival rates, we vary the computational budget available for training per incoming minibatch, , as defined in § 4.1. At fixed training throughput and minibatch size , a lower corresponds to a higher data arrival rate . We consider three computational budgets: High, Medium, and Low, corresponding to , , and the FLOPs required for one backbone forward pass, respectively, as shown in Table 2. We profile the training FLOPs of each baseline and report the resulting compute ratios and skipped mini-
| Budget | CIFAR-10/100 | CLEAR-10/100 |
| High (3× FP) | 2.22 | 3,138.92 |
| Medium (2× FP) | 1.48 | 2,092.61 |
| Low (1× FP) | 0.74 | 1,046.31 |
Table 2. Available computational budget (GFLOPs per arrival). FP denotes one backbone forward pass on an incoming minibatch.
batches in Table 3. aL-SAR [58] explicitly accounts for computational cost through adaptive layer freezing. The memory budgets used in these experiments are provided in the appendix. For all methods in Table 1, we use an ImageNet-1K-pretrained ResNet-18 [23] with a batch size of 10 for CIFAR-10 and CIFAR-100, and an ImageNet-1K-pretrained ResNet-50 [23] with a batch size of 128 for CLEAR-10 and CLEAR-100. Here, UB represents an unconstrained variant of DuoCL that retains all samples in each window and performs slow-path adaptation without computational restrictions. Table 1 shows the average online accuracy of all methods under three computational budgets. Under the Low budget, existing OCL methods skip multiple minibatches during training, limiting their learning opportunities. In contrast, DuoCL continuously updates its accumulated feature statistics and , while periodically adapting the backbone using buffered samples. As a result, DuoCL outperforms the strongest existing OCL baseline by 8.53, 17.04, 12.06, and 21.26 percentage points on CIFAR-10, CIFAR-100, CLEAR-10, and CLEAR-100, respectively, under the Low budget, with an average improvement of 9.67 percentage points across all datasets and budgets. Furthermore, DuoCL achieves accuracy within 0.79 percentage points of UB across all settings, suggesting that continuous fast-path learning combined with reservoir-based slow-path adaptation can approach unconstrained performance under limited computational and memory budgets.
| Method | CIFAR-10 | CIFAR-100 | CLEAR-10 | CLEAR-100 | |
| ER [13] | High | 64.35 | 28.56 | 85.25 | 73.06 |
| Medium | 61.18 | 25.04 | 80.66 | 66.89 | |
| Low | 54.04 | 19.43 | 69.09 | 52.19 | |
| ACE [9] | High | 60.99 | 24.03 | 87.04 | 76.63 |
| Medium | 57.70 | 21.85 | 83.18 | 72.36 | |
| Low | 51.98 | 17.54 | 74.99 | 62.66 | |
| LwF [34] | High | 63.17 | 9.90 | 82.48 | 69.58 |
| Medium | 58.25 | 8.15 | 78.39 | 62.24 | |
| Low | 51.31 | 6.43 | 65.71 | 44.71 | |
| RWalk [11] | High | 57.63 | 9.16 | 78.01 | 60.56 |
| Medium | 54.11 | 8.19 | 69.05 | 51.90 | |
| Low | 46.83 | 5.87 | 51.67 | 30.55 | |
| PoLRS [10] | High | 45.47 | 17.48 | 62.72 | 48.68 |
| Medium | 44.57 | 14.46 | 59.60 | 45.71 | |
| Low | 41.81 | 11.11 | 18.47 | 21.23 | |
| MIR [2] | High | 49.25 | 8.18 | 72.29 | 50.53 |
| Medium | 44.90 | 7.22 | 61.04 | 37.84 | |
| Low | 39.02 | 4.85 | 39.05 | 17.36 | |
| GSS [4] | High | 40.26 | 9.67 | 51.43 | 30.60 |
| Medium | 34.31 | 7.93 | 31.66 | 18.82 | |
| Low | 28.18 | 6.09 | 16.07 | 4.76 | |
| aL-SAR [58] | High | 61.62 | 23.96 | 84.69 | 65.58 |
| Medium | 60.17 | 22.48 | 83.73 | 63.33 | |
| Low | 55.73 | 18.92 | 80.57 | 58.07 | |
| DuoCL (Ours) | High | 64.80 | 36.64 | 92.70 | 83.96 |
| Medium | 64.68 | 36.60 | 92.67 | 83.94 | |
| Low | 64.26 | 36.47 | 92.63 | 83.92 | |
| UB | 65.05 | 36.86 | 92.70 | 84.00 | |
| Improvement | High | +0.45 | +8.08 | +5.66 | +7.34 |
| Medium | +3.51 | +11.56 | +8.94 | +11.59 | |
| Low | +8.53 | +17.04 | +12.06 | +21.26 |
Table 1. Average online accuracy (%, ↑) under three computational budgets, averaged over three seeds. Improvement denotes the absolute percentage-point gain of DuoCL over the strongest existing OCL baseline for each dataset and budget. Best and second-best results, excluding the unconstrained UB, are shown in bold and <u>underlined</u>, respectively.
Table 3. Compute ratio () and skipped minibatches per update () under three computational budgets.
Impact of Distribution Shift
To demonstrate the importance of dual-path learning under distribution shifts commonly encountered in OCL, we compare DuoCL with Analytic, a variant that uses only the fast path without backbone adaptation. We first evaluate both methods using ResNet-18 on a CIFAR-C stream consisting
of clean images followed by Gaussian noise and contrast corruptions at severity level 4. On CIFAR-C, DuoCL improves test accuracy over Analytic by 3.61 and 4.00 percentage points under Gaussian noise and contrast corruption, respectively (Figure 4). On CLOC, using ResNet-50, DuoCL consistently achieves higher monthly online accuracy (Figure 5), supporting the benefit of backbone adaptation under naturally evolving distributions.
Figure 4. Accuracy under distribution shifts on CIFAR-C.
Figure 5. Comparison of online accuracy between DuoCL and Analytic on the CLOC dataset under naturally occurring temporal distribution shifts.
Performance under Different OCL Settings
We evaluate DuoCL under disjoint class-incremental learning (CIL), where new classes are introduced sequentially across tasks and do not reappear in subsequent tasks. We consider CIFAR-100, ImageNet-R, ImageNet-S, and CUB-200 with the CIL setup using ViT-B/16 and ViT-S/16 backbones. We set the memory budget to samples and the available computational budget to the FLOPs corresponding to one training pass per incoming minibatch, consistent with prior work [66]. The first three datasets are divided into 10 tasks, while CUB-200 uses 5 tasks. UB represents the offline upper bound obtained through supervised fine-tuning on the complete dataset with i.i.d. samples. DuoCL achieves the highest final accuracy among OCL methods in seven of eight comparisons, improving upon Analytic by up to 3.08 percentage points (Table 5). Under domain-incremental learning (DIL), the input distribution changes while the class set remains fixed. On CORe50, DuoCL exceeds Online-LoRA [66] by 1.81 and 1.58 percentage points with ViT-B/16 and ViT-S/16, respectively (Table 6).
| Backbone | Method | CIFAR-100 | ImageNet-R | ImageNet-S | CUB-200 |
| ViT-B/16 | AGEM [12] | 12.67 ± 1.87 | 5.60 ± 2.74 | 0.16 ± 0.04 | 10.84 ± 1.57 |
| ER [13] | 44.85 ± 1.83 | 40.99 ± 3.96 | 30.21 ± 0.70 | 31.66 ± 0.83 | |
| EWC++ [11] | 10.61 ± 0.74 | 3.86 ± 2.02 | 0.32 ± 0.28 | 26.14 ± 3.46 | |
| MIR [2] | 48.36 ± 3.11 | 41.51 ± 2.99 | 30.33 ± 3.81 | 31.64 ± 2.97 | |
| GDumb [50] | 41.00 ± 19.97 | 8.87 ± 1.36 | 1.65 ± 0.22 | 9.09 ± 1.03 | |
| PCR [36] | 48.48 ± 0.15 | 46.11 ± 3.03 | 38.75 ± 0.22 | 41.11 ± 1.43 | |
| DER++ [8] | 36.64 ± 6.11 | 30.90 ± 8.04 | 6.47 ± 0.06 | 26.61 ± 1.27 | |
| LODE (DER++) [35] | 44.29 ± 1.48 | 42.20 ± 6.46 | 9.97 ± 2.29 | 39.20 ± 4.25 | |
| EMA (DER++) [60] | 42.28 ± 4.36 | 41.75 ± 1.98 | 16.88 ± 2.23 | 35.26 ± 3.31 | |
| EMA (RAR) [60] | 47.10 ± 0.82 | 30.04 ± 0.33 | 14.06 ± 0.37 | 33.34 ± 1.11 | |
| Online-LoRA [66] | 49.40 ± 1.36 | 48.18 ± 0.63 | 47.06 ± 0.24 | 41.46 ± 0.31 | |
| FGH [44] | 76.94 ± 0.89 | 64.85 ± 0.71 | 55.71 ± 2.96 | 75.22 ± 1.55 | |
| Analytic | 85.55 ± 0.18 | 68.24 ± 0.12 | 59.12 ± 0.30 | 82.86 ± 0.56 | |
| DuoCL (Ours) | 87.78 ± 0.16 | 71.31 ± 0.14 | 57.98 ± 0.49 | 82.91 ± 0.54 | |
| UB | 89.50 ± 0.04 | 76.78 ± 0.44 | 63.82 ± 0.02 | 82.81 ± 1.07 | |
| ViT-S/16 | AGEM [12] | 7.43 ± 2.15 | 2.35 ± 0.87 | 2.75 ± 2.86 | 1.40 ± 0.17 |
| ER [13] | 31.91 ± 2.06 | 32.73 ± 0.20 | 19.53 ± 1.44 | 21.81 ± 3.02 | |
| EWC++ [11] | 6.80 ± 2.13 | 1.32 ± 0.83 | 4.08 ± 3.24 | 2.07 ± 0.54 | |
| MIR [2] | 29.08 ± 1.14 | 34.73 ± 0.29 | 13.96 ± 1.95 | 22.95 ± 1.12 | |
| GDumb [50] | 10.87 ± 4.94 | 5.33 ± 1.09 | 2.90 ± 0.32 | 3.28 ± 0.99 | |
| PCR [36] | 32.89 ± 1.47 | 21.96 ± 0.27 | 14.37 ± 0.95 | 22.28 ± 2.73 | |
| DER++ [8] | 17.67 ± 4.04 | 22.17 ± 4.27 | 18.15 ± 0.66 | 29.53 ± 2.37 | |
| LODE (DER++) [35] | 28.65 ± 3.06 | 31.65 ± 0.72 | 17.59 ± 0.84 | 26.81 ± 0.89 | |
| EMA (DER++) [60] | 12.76 ± 0.65 | 20.89 ± 0.35 | 12.93 ± 0.13 | 35.79 ± 5.27 | |
| EMA (RAR) [60] | 19.21 ± 2.16 | 16.11 ± 0.35 | 14.50 ± 2.71 | 34.53 ± 1.04 | |
| Online-LoRA [66] | 32.16 ± 0.24 | 33.21 ± 0.50 | 22.45 ± 0.43 | 37.41 ± 1.16 | |
| FGH [44] | 74.01 ± 1.22 | 52.39 ± 4.37 | 48.09 ± 2.28 | 76.46 ± 0.65 | |
| Analytic | 80.08 ± 0.06 | 54.70 ± 0.07 | 46.89 ± 0.21 | 83.22 ± 0.67 | |
| DuoCL (Ours) | 82.62 ± 0.11 | 57.78 ± 0.01 | 49.93 ± 0.18 | 83.82 ± 0.56 | |
| UB | 86.55 ± 0.01 | 69.94 ± 0.34 | 59.28 ± 0.11 | 73.91 ± 1.16 |
Table 5. Final accuracy (, %, ) under disjoint CIL. Best and second-best non-UB results are shown in bold and _underlined_, respectively.
| Method | ViT-B/16 | ViT-S/16 |
| AGEM [12] | 80.15 ± 2.97 | 78.22 ± 3.51 |
| ER [13] | 85.85 ± 1.35 | 78.99 ± 3.85 |
| EWC++ [11] | 78.65 ± 6.51 | 79.03 ± 4.54 |
| MIR [2] | 74.35 ± 4.07 | 86.49 ± 0.81 |
| GDumb [50] | 77.20 ± 3.49 | 75.64 ± 2.92 |
| PCR [36] | 87.16 ± 0.73 | 75.20 ± 1.48 |
| DER++ [8] | 81.88 ± 7.06 | 89.33 ± 0.62 |
| LODE (DER++) [35] | 77.02 ± 2.22 | 83.48 ± 5.84 |
| L2P [65] | 87.97 ± 0.37 | 86.47 ± 0.23 |
| MVP [46] | 84.82 ± 0.54 | 79.85 ± 0.33 |
| Online-LoRA [66] | 93.71 ± 0.01 | 90.96 ± 0.02 |
| Analytic | 93.57 ± 0.00 | 89.19 ± 0.00 |
| DuoCL (Ours) | 95.52 ± 0.08 | 92.54 ± 0.14 |
| UB | 95.60 ± 0.01 | 93.56 ± 0.01 |
Table 6. Final accuracy (, %, ) under domain-incremental learning on CORE50 [38]. Best and second-best non-UB results are shown in bold and _underlined_, respectively.
We further evaluate DuoCL in the stochastic incremental-blurry (Si-Blurry) setting [46], where class distributions change stochastically, and classes overlap across tasks. We consider CIFAR-100, ImageNet-R, and ImageNet-S, each divided into five tasks, using the same computational and memory budgets. DuoCL outperforms
Analytic across all 12 and comparisons (Table 4). On CIFAR-100, final accuracy improves by 4.93 and 5.69 percentage points with ViT-B/16 and ViT-S/16, respectively.
| Backbone | Method | CIFAR-100 | CIFAR-100 | ImageNet-R | ImageNet-R | ImageNet-S | ImageNet-S |
| ViT-B/16 | L2P [65] | 43.01 ± 9.37 | 39.86 ± 2.28 | 22.71 ± 1.86 | 27.08 ± 2.49 | 10.02 ± 0.42 | 13.58 ± 4.04 |
| MVP [46] | 47.52 ± 9.74 | 44.49 ± 0.93 | 27.79 ± 0.62 | 31.64 ± 1.77 | 10.68 ± 0.45 | 13.99 ± 1.73 | |
| Online-LoRA [66] | 60.12 ± 5.79 | 61.70 ± 6.29 | 45.05 ± 1.59 | 48.00 ± 6.01 | 30.81 ± 2.09 | 30.22 ± 4.36 | |
| Analytic | 69.86 ± 6.03 | 81.70 ± 0.00 | 59.28 ± 0.31 | 58.82 ± 0.07 | 47.20 ± 2.50 | 44.85 ± 0.24 | |
| DuoCL (Ours) | 74.74 ± 5.29 | 86.63 ± 0.16 | 60.95 ± 0.29 | 61.65 ± 0.39 | 48.53 ± 2.70 | 46.15 ± 0.71 | |
| Offline UB | 89.50 ± 0.04 | 76.78 ± 0.44 | 63.82 ± 0.02 | ||||
| ViT-S/16 | L2P [65] | 37.82 ± 12.19 | 30.88 ± 1.39 | 24.31 ± 1.83 | 21.83 ± 2.13 | 2.00 ± 0.12 | 3.61 ± 1.08 |
| MVP [46] | 40.31 ± 9.52 | 35.55 ± 2.11 | 27.04 ± 1.09 | 26.67 ± 3.70 | 2.27 ± 0.14 | 3.72 ± 0.77 | |
| Online-LoRA [66] | 52.84 ± 7.97 | 58.72 ± 1.44 | 39.47 ± 1.93 | 36.61 ± 4.63 | 15.35 ± 0.92 | 20.18 ± 1.84 | |
| Analytic | 64.70 ± 7.51 | 77.93 ± 0.00 | 44.85 ± 0.68 | 43.56 ± 0.24 | 33.35 ± 2.17 | 32.06 ± 0.31 | |
| DuoCL (Ours) | 68.75 ± 7.24 | 83.62 ± 0.35 | 47.69 ± 0.62 | 48.90 ± 0.79 | 36.49 ± 2.40 | 35.88 ± 0.78 | |
| UB | 86.55 ± 0.01 | 69.94 ± 0.34 | 59.28 ± 0.11 |
Table 4. Performance comparison under Si-Blurry CIL (%, ). Each dataset reports area under the accuracy curve () and final accuracy (). Best and second-best non-UB results are shown in bold and _underlined_, respectively.
Ablation on Individual Components
Figure 6 presents the component ablation on CIFAR-100 under Si-Blurry using ViT-B/16. The fast-path-only baseline achieves 81.70% final accuracy, increasing to 85.29% with slow-path adaptation using the main cross-entropy loss, 85.95% with drift regularization, and 86.63% with the auxiliary loss, demonstrating the contribution of each component.

Figure 6. Component ablation on Si-Blurry CIFAR-100.
Conclusion
We presented DuoCL, a dual-path OCL framework that combines continuous analytic learning with periodic backbone adaptation under computational and memory constraints. DuoCL improves average online accuracy by 9.67 percentage points over existing baselines and demonstrates the benefits of backbone adaptation under distribution shifts and diverse OCL settings.
LLM Usage Statement
OpenAI’s ChatGPT and Anthropic’s Claude were used to assist with editing throughout the manuscript, including grammar and clarity, as well as figure refinement, code implementation, and managing experiments. The authors reviewed and verified the final content and take full responsibility for the submitted work.
References
- [1]Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Computer Vision – ECCV 2018, pages 144–161, Cham, 2018. Springer International Publishing. 1arxiv.org/abs/1711.09601
- [2]Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019. 2, 6, 7, 8
- [3]Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with minimal interfered retrieval. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019. 1
- [4]Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019. 1, 2, 6, 7
- [5]Elnaz Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. In International Conference on Learning Representations, 2022. 2arxiv.org/abs/2201.12604
- [6]Jebacyril Ariockiaraj, Dhruv Parikh, Jayashree Adivarahan, Rajgopal Kannan, and Viktor Prasanna. Accurate and resource-efficient federated continual learning, 2026. 1, 2
- [7]Romil Bhardwaj, Zhengxu Xia, Ganesh Ananthanarayanan, Junchen Jiang, Yuanchao Shu, Nikolaos Kariakaris, Kevin Hsieh, Paramvir Bahl, and Ion Stoica. Ekya: Continuous learning of video analytics models on edge compute servers. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 119–135, Renton, WA, 2022. USENIX Association. 2
- [8]Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and SIMONE CALDERARA. Dark experience for general continual learning: a strong, simple baseline. In
- [9]Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In International Conference on Learning Representations, 2022. 1, 2, 6, 7
- [10]Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data . In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8261–8270, Los Alamitos, CA, USA, 2021. IEEE Computer Society. 1, 3, 6, 7
- [11]Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Computer Vision – ECCV 2018, pages 556–572, Cham, 2018. Springer International Publishing. 6, 7, 8DOI
- [12]Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019. 6, 8arxiv.org/abs/1812.00420
- [13]Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, P. Dokania, P. Torr, and M. Ranzato. Continual learning with tiny episodic memories. In Workshop on multi-task and lifelong reinforcement learning, 2019. 1, 2, 6, 7, 8
- [14]Zhiyuan Chen and Bing Liu. Lifelong machine learning, second edition. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12:1–207, 2018. 1DOI
- [15]Botos Csaba, Wenxuan Zhang, Matthias Müller, Ser-Nam Lim, Mohamed Elhoseiny, Philip Torr, and Adel Bibi. Label delay in online continual learning. In Advances in Neural Information Processing Systems, pages 119976–120012. Curran Associates, Inc., 2024. 2arxiv.org/abs/2312.00923
- [16]Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021. 1DOI
- [17]Eros Fanì, Raffaello Camoriano, Barbara Caputo, and Marco Cicone. Accelerating heterogeneous federated learning with closed-form classifiers. In Proceedings of the 41st International Conference on Machine Learning, pages 13029–13048. PMLR, 2024. 1, 2
- [18]Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip H.S. Torr, and Bernard Ghanem. Real-time evaluation in online continual learning: A new hope. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11888–11897, 2023. 1, 2, 3, 6
- [19]Dipam Goswami, Albin Soutif-Cormerais, Yuyang Liu, Sandesh Kamath, Bartlomiej Twardowski, and Joost van de Weijer. Resurrecting old classes with new data for exemplar-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28525–28534, 2024. 2 [20] Zenghao Guan, Guojun Zhu, Yucan Zhou, Wu Liu, Weiping Wang, Jiebo Luo, and Xiaoyan Gu. Enhancing federated class-incremental learning via spatial-temporal statistics aggregation. In Proceedings of the ACM Web Conference 2026, pages 5356–5367, 2026. 1, 2
- [21]Md Yousuf Harun, Jhair Gallardo, Tyler L. Hayes, Ronald Kemker, and Christopher Kanan. SIESTA: Efficient online continual learning with sleep. Transactions on Machine Learning Research, 2023. 3arxiv.org/abs/2303.10725
- [22]Tyler L. Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 887–896, 2020. 2arxiv.org/abs/1909.01520
- [23]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6arxiv.org/abs/1512.03385
- [24]Run He, Di Fang, Yicheng Xu, Yawen Cui, Ming Li, Cen Chen, Ziqian Zeng, and Huiping Zhuang. Semantic shift estimation via dual-projection and classifier reconstruction for exemplar-free class-incremental learning. In Forty-second International Conference on Machine Learning, 2025. 3, 5arxiv.org/abs/2503.05423
- [25]Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019. 6
- [26]Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8320–8329, 2021. 6
- [27]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. 5arxiv.org/abs/2106.09685
- [28]David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018. 1arxiv.org/abs/1802.10269
- [29]Mehrdad Khani, Pouya Hamadanian, Arash Nasr-Esfahany, and Mohammad Alizadeh. Real-time video inference on edge devices via adaptive model streaming. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4552–4562, 2021. 2arxiv.org/abs/2006.06628
- [30]Mehrdad Khani, Ganesh Ananthanarayanan, Kevin Hsieh, Junchen Jiang, Ravi Netravali, Yuanchao Shu, Mohammad Alizadeh, and Victor Bahl. RECL: Responsive Resource-Efficient continuous learning for video analytics. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 917–932, Boston, MA, 2023. USENIX Association. 2
- [31]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. 1DOI
- [32]Hyunseo Koh, Dahyun Kim, Jung-Woo Ha, and Jonghyun Choi. Online continual learning on class incremental blurry task configuration with anytime inference. In ICLR, 2022. 6arxiv.org/abs/2110.10031
- [33]Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. 6
- [34]Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017. 1, 2, 6, 7
- [35]Yan-Shuo Liang and Wu-Jun Li. Loss decoupling for task-agnostic continual learning. In Advances in Neural Information Processing Systems, pages 11151–11167. Curran Associates, Inc., 2023. 6, 8
- [36]Huiwei Lin, Baoquan Zhang, Shanshan Feng, Xutao Li, and Yunming Ye. Pcr: Proxy-based contrastive replay for online class-incremental continual learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24246–24255, 2023. 6, 8arxiv.org/abs/2304.04408
- [37]Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on real-world imagery. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. 6arxiv.org/abs/2201.06289
- [38]Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In Proceedings of the 1st Annual Conference on Robot Learning, pages 17–26. PMLR, 2017. 6, 8arxiv.org/abs/1705.03550
- [39]Simone Magistri, Tomaso Trinci, Albin Soutif-Cormerais, Joost van de Weijer, and Andrew Bagdanov. Elastic feature consolidation for cold start exemplar-free incremental learning. In International Conference on Learning Representations, pages 31909–31931, 2024.
- [40]Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning . In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3584–3594, Los Alamitos, CA, USA, 2021. IEEE Computer Society. 6arxiv.org/abs/2103.13885
- [41]James L McClelland, Bruce L McNaughton, and Randall C O’Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995. 2
- [42]Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. pages 109–165. Academic Press, 1989. 1DOI
- [43]Mark D McDonnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, and Anton Van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36:12022–12053, 2023. 1, 2, 3
- [44]Nicolas Michel, Maorong Wang, Jiangpeng He, and Toshihiko Yamasaki. From offline to online memory-free and task-free continual learning via fine-grained hypergradients. Transactions on Machine Learning Research, 2026. 6, 8arxiv.org/abs/2502.18762
- [45]Saleh Momeni, Changnan Xiao, and Bing Liu. Anacp: Toward upper-bound continual learning via analytic contrastive projection. In Advances in Neural Information Processing Systems, pages 72923–72944. Curran Associates, Inc., 2025. 2, 3arxiv.org/abs/2511.13880
- [46]Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, and Gyeong-Moon Park. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11697–11707, 2023. 2, 6, 7, 8arxiv.org/abs/2308.09303
- [47]German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71, 2019. 1DOI
- [48]Liangzu Peng, Juan Elenter, Joshua Agterberg, Alejandro Ribeiro, and Rene Vidal. Lornapc: Low-rank random features and pre-trained models for bridging theory and practice in continual learning. In The Thirteenth International Conference on Learning Representations, 2025. 1, 2, 3arxiv.org/abs/2410.00645
- [49]Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. In Advances in Neural Information Processing Systems, pages 16131–16144. Curran Associates, Inc., 2021. 2arxiv.org/abs/2110.00175
- [50]Ameya Prabhu, Philip H. S. Torr, and Puneet K. Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision – ECCV 2020, pages 524–540, Cham, 2020. Springer International Publishing. 6, 8DOI
- [51]Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet K. Dokania, Philip H.S. Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3698–3707, 2023. 2arxiv.org/abs/2303.11165
- [52]Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. Online continual learning without the storage constraint, 2023. 2arxiv.org/abs/2305.09253
- [53]Ameya Prabhu, Shiven Sinha, Ponnurangam Kumaraguru, Philip Torr, Ozan Sener, and Puneet K. Dokania. Random representations outperform online continually learned representations. In Advances in Neural Information Processing Systems, pages 37988–38006. Curran Associates, Inc., 2024. 2arxiv.org/abs/2402.08823
- [54]Xuan Rao, Simian Xu, Zheng Li, Bo Zhao, Derong Liu, Mingming Ha, and Cesare Alippi. Compensating distribution drifts in continual learning with pre-trained vision transformers. Proceedings of the AAAI Conference on Artificial Intelligence, 40(30):25090–25098, 2026. 3DOI
- [55]Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5533–5542, 2017. 1arxiv.org/abs/1611.07725
- [56]Grzegorz Rypeś, Sebastian Cygert, Tomasz Trzcinski, and Bartł omiej Twardowski. Task-recency bias strikes back: Adapting covariances in exemplar-free class incremental learning. In Advances in Neural Information Processing Systems, pages 63628–63289. Curran Associates, Inc., 2024. 2
- [57]Jonathan Schwarz, Wojciech Czarnecki, Elena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International conference on machine learning, pages 4528–4537. PMLR, 2018. 1arxiv.org/abs/1805.06370
- [58]Minhyuk Seo, Hyunseo Koh, and Jonghyun Choi. Budgeted online continual learning by adaptive layer freezing and frequency-based sampling. In International Conference on Learning Representations, pages 47499–47533, 2025. 1, 2, 6arxiv.org/abs/2410.15143
- [59]Amelia Sorrenti, Giovanni Bellitto, Federica Proietto Salan-itri, Matteo Pennisi, Concetto Spampinato, and Simone Palazzo. Selective freezing for efficient continual learning. In 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 3542–3551, 2023. 1, 2DOI
- [60]Albin Soutif-Cormerais, Antonio Carta, and Joost van de Weijer. Improving online continual learning performance and stability with temporal ensembles. In Proceedings of The 2nd Conference on Lifelong Learning Agents, pages 828–845. PMLR, 2023. 6, 8
- [61]Jianheng Tang, Jingyu He, Kejia Fan, Run He, Jingchao Wang, Anfeng Liu, Houbing Herbert Song, Leye Wang, Zhanxing Zhu, Huiping Zhuang, and Yunhuai Liu. Acfl: Achieving spatio-temporal invariance to data heterogeneity in federated continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings, pages 7768–7778, 2026. 1, 2DOI
- [62]C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-ucsd birds-200-2011. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 6
- [63]Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P. Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506–10518, 2019. 6
- [64]Xinrui Wang, Chuanxing Geng, Wenhai Wan, Shao-yuan Li, and Songcan Chen. Forgetting, ignorance or myopia: Revisiting key challenges in online continual learning. In Advances in Neural Information Processing Systems, pages 58341–58375. Curran Associates, Inc., 2024. 2arxiv.org/abs/2409.19245
- [65]Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Thomas Pfister. Learning to prompt for continual learning. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 139–149, 2022. 6, 8arxiv.org/abs/2112.08654
- [66]Xiwen Wei, Guihong Li, and Radu Marculescu. Online-lora: Task-free online continual learning via low rank adaptation. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6634–6645, 2025. 2, 6, 7, 8arxiv.org/abs/2411.05663
- [67]Hongye Xu and Bartosz Krawczyk. Two-way is better than one: Bidirectional alignment with cycle consistency for exemplar-free class-incremental learning. In The Fourteenth International Conference on Learning Representations, 2026. 5arxiv.org/abs/2606.05675
- [68]Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. In International Conference on Learning Representations, 2022. 1arxiv.org/abs/2106.01085
- [69]Lu Yu, Bartlomiej Twardowski, Xiailei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2arxiv.org/abs/2004.00440
- [70]Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 19091–19101, 2023. 2arxiv.org/abs/2303.05118
- [71]Wenxuan Zhang, Youssef Mohamed, Bernard Ghanem, Philip Torr, Adel Bibi, and Mohamed Elhoseiny. Continual learning on a diet: Learning from sparsely labeled streams under constrained computation. In International Conference on Learning Representations, pages 31516–31532, 2024. 2arxiv.org/abs/2404.12766
- [72]Xueyi Zhang, Chengwei Zhang, Zheng Li, Xiyu Wang, Siqi Cai, Mingrui Lao, Yanming Guo, and Huiping Zhuang. Rep deep & machine learning: Exemplar-free continual video action recognition via slow-fast collaborative learning. Proceedings of the AAAI Conference on Artificial Intelligence, 40(42):36075–36083, 2026. 2DOI
- [73]Yaqian Zhang, Bernhard Pfahringer, Eibe Frank, Albert Bifet, Nick Jin Sean Lim, and Yunzhe Jia. A simple but strong baseline for online continual learning: Repeated augmented rehearsal. In Advances in Neural Information Processing Systems, pages 14771–14783. Curran Associates, Inc., 2022. 6
- [74]Linglan Zhao, Xuerui Zhang, Ke Yan, Shouhong Ding, and Weiran Huang. Safe: Slow and fast parameter-efficient tuning for continual learning with pre-trained models. In Advances in Neural Information Processing Systems, pages 113772–113796. Curran Associates, Inc., 2024. 2arxiv.org/abs/2411.02175
- [75]Yuhao Zhou, Yuxin Tian, Jindi Lv, Mingjia Shi, Yuanxi Li, Qing Ye, Shuhao Zhang, and Jiancheng Lv. Ferret: An efficient online continual learning framework under varying memory constraints. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4850–4861, 2025. 1, 2arxiv.org/abs/2503.12053
- [76]HUIPING ZHUANG, Zhenyu Weng, Hongxin Wei, RENCHUNZI XIE, Kar-Ann Toh, and Zhiping Lin. Acil: Analytic class-incremental learning with absolute memorization and privacy protection. In Advances in Neural Information Processing Systems, pages 11602–11614. Curran Associates, Inc., 2022. 2arxiv.org/abs/2205.14922
- [77]Huiping Zhuang, Yuchen Liu, Run He, Kai Tong, Ziqian Zeng, Cen Chen, Yi Wang, and Lap-Pui Chau. Foal: Forward-only online analytic learning with fast training and low memory footprint in class incremental learning. In Advances in Neural Information Processing Systems, pages 41517–41538. Curran Associates, Inc., 2024. 2arxiv.org/abs/2403.15751




