WorksComputer Science
Variance Restoration over Covariance Rank: Memory-Budgeted Analytic Continual Learning
Abstract
Analytic continual learning replaces iterative gradient updates with closed-form ridge regression over frozen pre-trained representations, summarizing all past sessions in two running sufficient statistics: the feature autocorrelation matrix and the class cross-correlation matrix . Lifting backbone features into a wide random code of dimension substantially improves linear separability across tasks, but forces to grow quadratically as . Under a fixed persistent memory envelope \Bcal, a learner faces a fundamental structural trade-off: store exact pairwise co-activations inside a narrower feature code, or expand to a wide code while compressing cross-coordinate interactions. We observe that the coordinate self-variances v_t = \diag(Σ_t) and class statistics grow only linearly with , whereas off-diagonal interactions dominate memory. We introduce \method (Coordinate-Restored Analytic Continual Learning), which retains exactly, maintains off-diagonal co-activations through a streaming rank- factor , restores exact diagonal variances via \Sighat_t = F_t^\top F_t + \Diag(v_t - \diag(F_t^\top F_t)), and solves the full normal equations via the Woodbury identity. Across five class-incremental benchmarks on frozen ViT-B/16, \method matches dense full-covariance readouts within points while cutting persistent state by ( vs.\ MiB). Under equal memory envelopes of – MiB, prioritizing expansion width over covariance rank outperforms exact narrow readouts in of configurations by up to points.
Introduction
Class-incremental learning (CIL) requires a recognition system to assimilate new categories sequentially without reprocessing historical data or degrading previously acquired decision boundaries [1, 2]. Whereas rehearsal, weight-regularization, and parameter-isolation paradigms treat catastrophic forgetting as an optimization artifact of stochastic gradient descent, analytic continual learning reformulates incremental adaptation as exact recursive least squares over a frozen backbone [3, 4, 5, 6]. Because the cumulative ridge solution depends on past sessions solely through two additive sufficient statistics—the feature autocorrelation and the feature–label cross-correlation —analytic learners achieve order-invariant updates without storing raw exemplars.
To unlock strong linear separability from frozen features , recent representation-matching pipelines project embeddings into a high-dimensional random space of width via dense nonlinear projections [7, 8, 9] or biologically inspired sparse TopK expansions [10, 11]. However, widening the readout code shifts the bottleneck from compute to persistent memory: at , storing a dense FP32 autocorrelation matrix requires 381.5 MiB, whereas the class cross-correlation for classes requires only 3.8 MiB. When multiple incremental heads coexist on an edge accelerator or share a single frozen vision foundation model, this covariance footprint dictates the maximum affordable width.
Constrained by a fixed memory envelope , an analytic learner must choose between two competing designs that prior work has never disentangled: exact covariance at narrow width ( chosen so ) versus compressed covariance at wide width ( paired with a rank- approximation of ). Existing sketching methods such as Frequent Directions (FD) [12], Robust FD (RFD) [13, 14], and LoRanPAC [15] study spectral compression at a fixed dimension, while expansion methods vary without bounding memory [7, 10]. Because evaluations across papers simultaneously alter backbones, regularization rules, and memory budgets, it remains unclear whether a fixed byte budget should buy more coordinates or higher covariance fidelity.
Figure 1 reveals the structural asymmetry that governs this choice. In the cumulative normal equations, both the class cross-correlation ( scalars) and the coordinate self-variances ( scalars) grow linearly with width; only the off-diagonal co-activations scale quadratically. Yet when is compressed into a streaming rank- factor , repeated spectral truncation erodes 20%–33% of the median coordinate self-variance across a ten-session stream (Figure 3a), leaving off-subspace feature directions with zero data curvature.
Figure 1. The width–interaction allocation trade-off in analytic continual learning. (a) Dense autocorrelation storage () scales quadratically as , whereas CORE-ACL () scales linearly with width and reduces persistent memory by 90.7% at . (b) Under fixed memory envelopes ( MiB) with GCV regularization, reallocating bytes from covariance rank to expansion width consistently improves stage-average accuracy over exact narrow ridge, except when rank falls below a dataset-specific floor ( on CIFAR-100). (c) Stage-average accuracy saturates rapidly as interaction rank grows, matching the 385 MiB full-covariance ceiling (dotted lines) already at (35.9 MiB).
Figure 3. Mechanism diagnostics. (a) Median fraction of coordinate self-variance captured by the rank- factor across sessions; CORE-ACL restores the diagonal to 1.0 exactly. (b) Session-1 CUB-200 accuracy across forced penalties : small triggers blow-up in unreserved LR and uncalibrated CORE-ACL, whereas CORE-ACL selects (star). (c) Cumulative accuracy trajectory across 10 CIFAR-100 sessions.
Built on this insight, we propose CORE-ACL (Coordinate-Restored Analytic Continual Learning). CORE-ACL maintains exactly in space, summarizes off-diagonal interactions via a two-stage rank- factor , and restores the exact diagonal variance through [16]. Applying the Woodbury identity yields the exact full-width ridge classifier in time without materializing any matrix. Furthermore, we uncover a critical pathology in full-width sketched readouts: when the ridge penalty is selected via hold-out validation on an untruncated subset () and refitted on the full session (), off-subspace label projections are amplified by [17]. Our rank-calibrated hold-out rule (CORE-ACL*) matches the truncation ratio during validation, eliminating this collapse.
Our main contributions are summarized as follows:
We formulate the memory-budgeted width–interaction problem for analytic continual learning and isolate when expansion width outweighs covariance rank under matched storage, backbones, and class orders.
We propose CORE-ACL, a variance-restored streaming ridge solver that preserves exact per-coordinate variances and class cross-correlations while compressing only off-diagonal interactions into a rank- factor.
We establish Loewner positivity, Frobenius diagonal optimality, and a nullspace amplification theorem that explains hold-out refit collapse and motivates our rank-calibrated selection rule (CORE-ACL*).
Across CIFAR-100, CUB-200, VTAB-50, ImageNet-R, and ImageNet-A, CORE-ACL matches dense full-covariance readouts within 0.09 points using 90.7% less persistent state, while outperforming state-matched narrow exact ridge across diverse backbones and stream horizons.
Related Work
Pre-trained continual learning and parameter isolation. With strong pre-trained Vision Transformers (ViTs) [18, 19], exemplar-free CIL has shifted toward lightweight adaptation modules [1, 2]. Prompt-tuning approaches such as L2P [20], DualPrompt [21], and CODA-Prompt [22] steer frozen attention layers via task-queried prompt pools. Subspace and adapter methods—including EASE [23], InfLoRA [24], MOS [25], MoA [26], and rank-wise expert routing [27, 28]—allocate task-specific low-rank subspaces to limit gradient interference. Recent metric and transport frameworks further align frozen visual branches across sessions [29, 30, 31]. Unlike these methods, which modify backbone features via iterative gradient descent, CORE-ACL operates in the purely analytic regime with a fixed feature map and optimizes how the classifier’s sufficient statistics are budgeted.
Analytic continual learners and random feature lifting. Starting from ACIL [3], recursive least-squares updates have enabled exemplar-free continual learning with exact equivalence to joint ridge regression. Subsequent works introduced dual-stream compensation (DS-AL) [4], generalized objectives (GACL) [5], forward-only projections (F-OAL) [32], and representation enhancement (REAL) [6], with recent extensions to keyword spotting, autonomous driving, continual graphs, and long-tailed spectral rectification [33, 34, 35, 36, 37]. In parallel, RanPAC [7] and RanDumb [8] showed that lifting frozen features into random coordinates before analytic fitting rivals complex prompt-tuning pipelines. Fly-CL [10, 38] replaced dense projections with sparse TopK Kenyon-cell expansions, while multi-view boosting (MVP) [9], guided projections (SCL-MGSM) [11], analytic contrastive projection (AnaCP) [39], class-level statistical estimators [40], and temporal-spectral reconstruction [41] further enriched lifted representations. However, these methods either retain a dense autocorrelation matrix or compress it at a single fixed width without evaluating equal-memory narrow exact alternatives.
Streaming matrix sketches and low-rank-plus-diagonal models. Deterministic matrix sketching via Frequent Directions (FD) [12] and Robust FD (RFD) [13] approximates streaming Gram matrices by shrinking singular values, and streaming ridge sketches bound regression suboptimality [14]. LoRanPAC [15] applies continual truncated SVD to lifted features and restricts the classifier to the retained subspace. Diagonal-corrected low-rank covariances have classic roots in sparse Gaussian processes [16]. We unify these perspectives by treating spectral truncation, shrinkage, and exact variance restoration as competing allocations under a strict byte budget .
Preliminaries and Problem Formulation
Analytic Readouts on Lifted Representations
We study exemplar-free class-incremental learning across sequential sessions with disjoint label sets. Session arrives with samples , where denotes one-hot targets over the total class vocabulary . A frozen backbone extracts embeddings that are lifted into an -dimensional sparse code via a fixed random projection with non-zero Gaussian entries per row followed by TopK activation [10]:
Stacking the lifted session features into , the cumulative ridge classifier at session with regularization penalty is given in closed form by
At test time, an input is assigned to . Because raw samples are discarded immediately after session , the learner’s persistent continuation state consists solely of the sufficient statistics required to compute when session arrives.
The Memory-Budgeted Allocation Problem
The persistent state decomposes into two structurally distinct components according to their scaling in the expansion width :
Linear per-coordinate statistics : The class cross-correlation ( scalars) and the coordinate self-variance vector ( scalars). At and , occupies only 3.85 MiB in FP32.
Quadratic cross-coordinate interactions : The off-diagonal co-activation entries of , which require scalars (381.47 MiB in dense FP32 at ).
Given a fixed persistent memory budget (in bytes or FP32 scalars), an exact narrow learner selects the largest width satisfying and solves Eq. (2) without approximation. Conversely, a wide factored learner fixes a wide dimension and approximates with a compressed representation whose total persistent state fits within . Our central question is how should be partitioned between coordinate width and interaction rank , and what structural invariants must preserve so that wide compression reliably beats narrow exactness.
CORE-ACL: Coordinate-Restored Analytic Learning
CORE-ACL exploits the linear scaling of by storing both without loss while compressing only the quadratic off-diagonal interactions into a rank- factor (), as depicted in Figure 2.
Figure 2. Overview of CORE-ACL at session . I. Frozen backbone features are lifted into a wide sparse random code . II. Under persistent memory budget , CORE-ACL splits state maintenance into an exact linear pathway for coordinate self-variances and class cross-correlations , and a rank- spectral merge pathway for off-diagonal interactions . III. Exact diagonal variance restoration repairs the truncated covariance into , and a Woodbury normal-equation solve yields the full-width classifier .
Two-Stage Streaming Update
Upon receiving session and labels , CORE-ACL updates the exact linear statistics:
For any matrix with economy singular value decomposition , define the rank- spectral truncation operator with , so that is the best rank- Frobenius approximation to . CORE-ACL compresses the current session and merges it with the historical factor :
Each session discards a positive semidefinite (PSD) session tail and a PSD merge tail . Consequently, the persistent state requires exactly scalars—scaling linearly in and remaining strictly independent of the number of sessions .
Exact Variance Restoration and Woodbury Readout
Let denote the cumulative discarded covariance and its diagonal. Because is tracked exactly, CORE-ACL computes the exact per-coordinate variance deficit directly from its state and forms the variance-restored covariance operator:
Algorithm 1: CORE-ACL (and CORE-ACL) session update
Input: Frozen , sparse lift , state , session data , rank , penalty grid
Output: Updated state and readout
Solve by Cholesky;
return ;
In exact arithmetic, , meaning 100% of every coordinate’s self-energy is preserved and the approximation error is purely off-diagonal. Defining the diagonal preconditioner , the Woodbury identity solves the normal equations via a compact Cholesky factorization:
Crucially, because feeds the entire cross-correlation matrix into , the classifier is not restricted to , in contrast to subspace-projected PCR or LoRAN-PAC [15].
Regularization Selection and Rank Calibration
We evaluate two standard penalty selection paradigms for choosing . Under task-local Generalized Cross-Validation (GCV) [17], minimizes the leave-one-out proxy on the current session’s singular spectrum without splitting data. Under hold-out validation and refit, 90% of session ( samples) updates a temporary state to score on the remaining 10%, after which the learner refits on all samples. As we prove in Theorem 1, if on session 1, the validation state is exact () whereas the refit state is truncated (), causing small choices to blow up. Our rank-calibrated rule, CORE-ACL⋆, truncates the validation factor at proportional rank during SELECTPENALTY, exposing off-subspace sensitivity before is locked in.
Theoretical Analysis
We now formalize the algebraic invariants of variance restoration, its regularization equivalence, and the mechanism of off-subspace amplification. Complete proofs and extensions are provided in Appendix § 8.
Proposition 1 (Loewner Positivity and Optimal Diagonal Restoration). In exact arithmetic under Eqs. (3)–(5), for every session : (i) , , , and ; (ii) the residual error satisfies and ; and (iii) for any fixed factor , is the unique minimizer of .*
Proposition 1 guarantees that is strictly positive definite for all and that achieves the largest possible Frobenius reduction over unrestored low-rank ridge (, reduction ) and over isotropic shift (, reduction ).
Proposition 2 (Coordinate-Adaptive Regularization View). For fixed and , the readout uniquely minimizes the coordinate-weighted objective
Consequently, isotropic restoration simply reparameterizes unrestored low-rank ridge via , whereas CORE-ACL imposes a heterogeneous coordinate penalty tailored to each coordinate’s discarded variance.
Next, let denote the orthogonal projector onto , and decompose the class cross-correlation into in-subspace and nullspace components: .
Theorem 1 (Nullspace Amplification under Truncating Refit). For unrestored low-rank ridge at penalty , the classifier decomposes orthogonally as
A label component aligned with a retained singular vector of singular value is scaled by , whereas any nullspace component in is scaled by —a relative amplification of . Whenever the factor is exact (), . Under CORE-ACL, any unit vector receives Rayleigh curvature
Theorem 1 pinpoints why hold-out validation can catastrophically fail on session 1: when , the temporary validation factor is exact (), so tiny penalties () incur zero nullspace error. Upon refitting on all samples, truncation creates a non-zero that is multiplied by . Early in the stream, captures of , so is small and Eq. (9) stays near . By contrast, methods with an explicit spectral shrinkage floor (such as RFD) cap relative amplification at , and CORE-ACL⋆ prevents the tiny from being chosen by enforcing during validation.
Experiments
Benchmarks and Evaluation Protocol. We evaluate on five standard exemplar-free CIL benchmarks: 10-session splits of CIFAR-100 [42], CUB-200-2011 [43], ImageNet-R [44], and ImageNet-A [45], plus a 5-session split of VTAB-50 [46] (partition details in Appendix § 12). Following the pre-trained representation protocol of RanPAC and Fly-CL [7, 10], we use a frozen ViT-B/16 backbone (augreg2-in21k_ft_in1k, ) [18] and lift features to coordinates with non-zeros per row and TopK sparsity . We report stage-average accuracy and final accuracy , together with persistent state size in MiB ( bytes, FP32 storage, FP64 normal-equation solve).
Comparison with State-of-the-Art Methods
Table 1 compares CORE-ACL (, GCV penalty selection on ) against prompt-tuning, adapter, and analytic continual learners across all five benchmarks. Three findings stand out:
Near-lossless compression of the full lifted covariance: Compared with Fly-CL [10], which stores the exact Gram matrix of the identical sparse expansion (385.3 MiB), CORE-ACL (35.9 MiB)
Table 1. Benchmark comparison on frozen ViT-B/16 across five class-incremental datasets (mean over 5 seeds). We report stage-average accuracy (%), final accuracy (%), and persistent continuation state (MiB, on CIFAR-100). Best overall accuracy is bold; best among compact-state analytic methods (< 60 MiB) is _underlined_.
| Method | CIFAR-100 () | CIFAR-100 () | CUB-200 () | CUB-200 () | VTAB-50 () | VTAB-50 () | ImageNet-R () | ImageNet-R () | ImageNet-A () | ImageNet-A () | State MiB |
| Prompt-tuning, subspace, and adapter continual learners | |||||||||||
| L2P [20] | 87.96 | 81.82 | 77.72 | 66.88 | 83.82 | 73.41 | 72.34 | 65.18 | 49.15 | 41.08 | 327.8 |
| DualPrompt [21] | 88.62 | 83.19 | 79.48 | 69.65 | 84.11 | 75.70 | 71.62 | 64.75 | 53.42 | 45.20 | 328.6 |
| CODA-Prompt [22] | 90.41 | 85.34 | 82.64 | 74.12 | 87.90 | 80.45 | 75.18 | 68.92 | 56.84 | 48.76 | 331.2 |
| EASE [23] | 91.74 | 86.91 | 86.92 | 81.62 | 90.68 | 84.94 | 76.08 | 70.14 | 63.85 | 55.92 | 48.3 |
| InfLoRA [24] | 90.84 | 85.04 | 79.38 | 67.41 | 86.51 | 77.72 | 74.65 | 68.10 | 59.12 | 50.84 | 102.1 |
| Analytic continual learners and lifted covariance matchers | |||||||||||
| ACIL [3] | 92.14 | 89.08 | 89.02 | 83.12 | 94.93 | 92.38 | 73.82 | 67.45 | 62.18 | 54.36 | 518.2 |
| DS-AL [4] | 92.31 | 89.18 | 89.64 | 84.05 | 95.08 | 92.64 | 74.56 | 68.32 | 63.45 | 55.80 | 524.5 |
| GACL [5] | 92.46 | 89.25 | 90.12 | 84.68 | 95.16 | 92.78 | 75.12 | 69.04 | 64.28 | 56.72 | 518.2 |
| REAL [6] | 92.58 | 89.34 | 90.85 | 85.74 | 95.28 | 92.96 | 75.64 | 69.78 | 65.40 | 57.95 | 518.2 |
| RanPAC [7] | 92.88 | 89.00 | 92.54 | 88.96 | 95.19 | 93.02 | 76.52 | 70.89 | 70.18 | 62.68 | 385.3 |
| LoRanPAC [15] | 93.29 | 89.79 | 92.75 | 89.42 | 95.36 | 93.33 | 77.66 | 73.06 | 70.82 | 64.01 | 4218.3 |
| Fly-CL (dense ) [10] | 92.83 | 88.82 | 91.48 | 87.12 | 95.06 | 92.51 | 76.74 | 71.14 | 69.85 | 62.08 | 385.3 |
| Compact-state analytic learners (< 60 MiB persistent memory) | |||||||||||
| F-OAL [32] | 92.02 | 87.35 | 90.34 | 85.12 | 95.92 | 94.06 | 74.90 | 68.85 | 64.08 | 56.42 | 8.4 |
| CORE-ACL () | _92.81_ | _88.78_ | _91.52_ | _87.26_ | 95.07 | 92.52 | _76.65_ | _71.02_ | _69.78_ | _61.96_ | 35.9 |
Table 1.
stays within 0.09 points in (−0.02, +0.04, +0.01, −0.09, −0.07 across the five benchmarks) and within 0.14 points in , cutting persistent state by 90.7%.
Superior accuracy over classical analytic and prompt baselines: Under a single-backbone feature stream, CORE-ACL surpasses classical recursive ridge learners (ACIL, DS-AL, GACL, REAL) by +0.67 to + 2.50 points on CUB-200 and by +4.38 to + 7.60 points on ImageNet-A, while outperforming EASE and CODA-Prompt across all five datasets.
Honest positioning against unbudgeted and multi-block upper bounds: LoRanPAC [15] achieves the highest overall accuracy on four datasets, but its materialized implementation consumes 879–4,218 MiB (24–117× larger than CORE-ACL). Conversely, F-OAL [32] uses only 8.4 MiB and leads on VTAB-50 by fusing features from all 12 ViT blocks, but trails CORE-ACL by 0.79, 1.18, 1.75, and 5.70 points on CIFAR-100, CUB-200, ImageNet-R, and ImageNet-A.
Iso-Memory Width vs. Covariance Rank Allocation
To isolate the allocation question directly, we fix the persistent memory budget at MiB and compare an Exact Narrow readout (largest rank fitting in ) against wide factored readouts (, rank filling ). Under GCV selection (Figure 1b and Appendix Table 6), trading covariance rank for expansion width improves in 17 of 18 width–rank cells across CIFAR-100, CUB-200, and VTAB-50:
Fine-grained and multi-domain streams favor maximal width: On CUB-200 and VTAB-50 at the tightest 16 MiB budget, pushing width to (leaving ranks and ) beats Exact Narrow () by +2.19 [+1.68, +2.70] and +1.72 [+1.33, +2.11] points.
Rank starvation creates an interior optimum on dense spectra: On CIFAR-100 at 16 MiB, full width (, ) suffers from rank starvation and drops −0.62 [−0.85, −0.39] points below Exact Narrow (90.52% vs. 91.14%), whereas an intermediate allocation (, ) gains +0.78 [+0.48, +1.08] points (91.92%). Even against a stronger Packed Exact Narrow baseline that stores only the upper triangle (widening by ), CORE-ACL maintains a clear lead of +1.06 to +1.65 points on CUB-200 and VTAB-50 (Appendix § 13.§ 5).
Table 2 evaluates the same MiB envelope when is chosen by hold-out validation and refit. On CIFAR-100, variance-corrected rules (RFD, FD+var, CORE-ACL, CORE-ACL*) all achieve significant gains of +0.57 to + 0.60 points over Exact Narrow, recovering > 80% of the unbudgeted wide ceiling (92.35%), whereas unrestored LR (+0.07) does not. On CUB-200, however, standard hold-out validation triggers the Theorem 1 pathology: because Session-1 fit splits () barely fit inside while refits () exceed , standard validation picks on two seeds and collapses Unrestored LR (86.12%), FD (86.34%), and uncalibrated CORE-ACL (88.68%). RFD avoids collapse via its isotropic shrinkage floor , and CORE-ACL* eliminates the collapse at the source by calibrating the validation truncation rank (), restoring CUB-200 accuracy to 91.89% (+0.45 over Exact Narrow).
| Readout Rule | CIFAR-100 | CIFAR-100 | CUB-200 | CUB-200 | VTAB-50 | VTAB-50 |
| Exact Narrow () | 91.62 | 0.00 | 91.44 | 0.00 | 93.81 | 0.00 |
| Unrestored LR | 91.69 | 86.12 | 94.21 | |||
| Frequent Directions | 91.81 | 86.34 | 94.02 | |||
| Robust FD (RFD) | 92.22 | 91.88 | 94.56 | |||
| FD + Exact Var. | 92.19 | 91.58 | 94.35 | |||
| CORE-ACL (std. val.) | 92.20 | 88.68 | 94.40 | |||
| CORE-ACL (ours) | 92.21 | 91.89 | 94.41 | |||
| Unbudgeted Wide | 92.35 | +0.73 | 92.02 | +0.58 | 94.88 | +1.07 |
Table 2. Iso-memory comparison at MiB under hold-out validation and refit (3 seeds, in %). Unrestored LR, FD, and RFD receive rows to match the variance scalars of CORE-ACL. CORE-ACL applies rank-calibrated hold-out selection.
Mechanism and Component Ablations
What must the state retain? Table 3 ablates the components of under a shared GCV grid. Discarding off-diagonal interactions entirely (Diagonal only, ) degrades by 9.17, 27.90, and 58.25 points across the three benchmarks, confirming that cross-coordinate co-activations are indispensable under standard regularization scales. Restricting the readout to the retained subspace (Subspace PCR, which projects ) drops VTAB-50 stage-average accuracy by 5.09 points (89.98% vs. 95.07%) while slightly raising final accuracy (92.82% vs. 92.52%), demonstrating that off-subspace cross-correlations are vital during early sessions before the rank- factor spans the full class geometry. Restoring exact coordinate variances in CORE-ACL improves CIFAR-100 by +0.63 points in and +1.69 points in over Unrestored LR on the exact same factor .
| State Configuration | CIFAR-100 | CIFAR-100 | CUB-200 | CUB-200 | VTAB-50 | VTAB-50 |
| Diagonal (, ) only | 83.64 | 77.89 | 63.62 | 52.94 | 36.82 | 31.78 |
| Subspace PCR () | 92.19 | 87.76 | 91.42 | 86.48 | 89.98 | 92.82 |
| Unrestored LR (, ) | 92.18 | 87.09 | 91.50 | 87.34 | 95.04 | 92.48 |
| Frequent Directions (FD) | 91.80 | 85.82 | 91.64 | 87.44 | 95.08 | 92.61 |
| Robust FD (RFD) | 92.65 | 88.49 | 91.31 | 86.72 | 95.06 | 92.55 |
| CORE-ACL (, , ) | 92.81 | 88.78 | 91.52 | 87.26 | 95.07 | 92.52 |
Table 3. Ablation of retained sufficient-statistic components (, , shared GCV grid, 5 seeds). Reporting both and (%) exposes the intermediate-vs.-final trade-off of span restriction.
Variance erosion and nullspace amplification. Figure 3a tracks the median variance capture ratio across sessions . As new classes rotate the feature distribution, a fixed rank retains a steadily shrinking share of coordinate variance, falling to 0.671 on CIFAR-100 at (0.795 at ). Without variance restoration, this lost energy directly weakens diagonal conditioning and causes the 8.18-point late-session accuracy deficit seen in Figure 4a. Meanwhile, Figure 3b visualizes Theorem 1 on Session 1 of CUB-200: although off-subspace label energy accounts for of , forcing amplifies until , collapsing accuracy to 21.4% (LR) and 49.8% (CORE-ACL). CORE-ACL detects this amplification on the pre-truncated validation split and selects (92.4%).
Figure 4. Stream fidelity, solver speedup, and cross-backbone transfer. (a) Session-wise gap relative to dense full covariance. (b) Normal-equation solve latency and temporary buffer on RTX 3090 (15.0× speedup and memory reduction). (c) Paired gains of CORE-ACL over state-matched Exact Narrow across ResNet-50, DINOv2-g/14, and 20-session horizons.
Generalization and Computational Efficiency
Backbone and horizon generality. Figure 4c and Appendix Table 7 verify that the width-over-rank allocation holds across diverse visual backbones and stream lengths at matched 36 MiB state. On ResNet-50 [47], where raw features are less linearly separable, CORE-ACL beats state-matched Exact Narrow by +0.66, +3.38, and +0.43 points on CIFAR-100, CUB-200, and VTAB-50. On DINOv2 ViT-g/14 () [19], gains remain consistently positive (+0.39, +0.94, +0.64), and on longer 20- and 50-session streams CORE-ACL matches dense Fly-CL within 0.06 points (92.76% vs. 92.80% on 50-session CIFAR-100).
Compute vs. memory trade-off. As shown in Figure 4b, the Woodbury normal-equation solve in Eq. (6) takes 4.9 ms and 102 MiB of temporary buffer versus 73.4 ms and 1,532 MiB for dense Cholesky factorization—a 15.0× reduction in both latency and solver workspace. End-to-end session fitting on CIFAR-100 takes 11.4 s per session for CORE-ACL (dominated by the session SVD ) versus 24.6 s for RFD and 0.8 s for Exact Narrow (Appendix § 13.§ 6). Thus, CORE-ACL specifically optimizes persistent intersession memory and normal-equation solve cost rather than feature-extraction or SVD wall-clock time.
Conclusion
We investigated how a fixed persistent memory budget in analytic continual learning should be allocated between feature expansion width and covariance interaction fidelity. Because coordinate self-variances and class cross-correlations scale linearly with width whereas off-diagonal co-activations scale quadratically, CORE-ACL preserves the linear statistics exactly, compresses interactions into a streaming rank- factor, and restores exact diagonal variances via a fast Woodbury readout. Across five CIL benchmarks, CORE-ACL matches dense 385 MiB covariance readouts within 0.09 points in just 35.9 MiB, and demonstrates that spending bytes on width consistently beats narrow exactness once rank starvation and nullspace refit amplification (CORE-ACL) are controlled.
Mathematical Proofs and Extended Theory
Throughout this appendix, we assume exact arithmetic, a fixed feature lifting map , a class vocabulary of size , and ridge regularization parameter . Rank-deficient session matrices and empty initial factors are permitted. For any matrix , let denote the number of singular rows retained by , and write where is the stacked factor at session . The session truncation residual and the merge truncation residual are defined as
Both and are positive semidefinite (PSD) by the Eckart–Young–Mirsky theorem, since each equals a sum of outer products of discarded right singular vectors weighted by squared singular values.
Proof of Proposition 1 (Loewner Positivity, Operator Sandwich, and Optimal Diagonal Restoration)
Proof. We establish by induction over sessions . At , and . Suppose . Because discards only non-negative singular modes, . By block-matrix multiplication,
Applying to again drops a PSD tail , yielding . Consequently, the cumulative discarded covariance satisfies
Because every diagonal entry of a PSD matrix is non-negative, for all . Thus is a sum of two PSD matrices and satisfies . Since is accumulated without approximation, .
For part (ii), write the approximation residual as . Since , we immediately have . Conversely, since , we have . Because , the extreme eigenvalues of lie in , which proves .
For part (iii), fix and consider any candidate diagonal vector . Decomposing , note that has a strictly zero diagonal while is purely diagonal. Hence , and the Pythagorean identity in Frobenius norm gives
which is uniquely minimized at . Setting yields an excess Frobenius error of , and restricting to an isotropic shift yields a minimum at with excess Frobenius error .
Proof of Proposition 2 (Coordinate-Adaptive Regularization Equivalence)
Proof. Expanding the Frobenius norm in Eq. (7) trace-wise gives
Since by Proposition 1, the objective is strictly convex in . Setting its matrix derivative yields the unique global minimizer . When is replaced by an isotropic scalar , the system matrix becomes , which coincides with the un-restored low-rank ridge operator .
Proof of Theorem 1 and Nullspace Stabilization Guarantees
Proof. Let have economy SVD with positive singular values (). The orthogonal projector onto is . Because , its inverse acts as on and as on . Applying this inverse to gives Eq. (8), where the two terms lie in orthogonal subspaces column by column. A unit component along is scaled by whereas a unit component in is scaled by , giving the relative ratio . If the factor is exact (), then , which contains the column space of , so . Finally, for any unit vector , , so .
Proposition 3 (Why Spectral Shrinkage Floors Prevent Refit Blow-Up). *Consider a spectral sketch with an isotropic floor , (as in RFD [13]). For any penalty , off-subspace components in are scaled by , capping the relative amplification against mode at
In particular, even as , the amplification remains bounded by , whereas for unrestored low-rank ridge () it diverges as .
Proposition 4 (Rank-Calibrated Validation Exposes Nullspace Sensitivity). *Suppose session 1 has and . Standard validation at capacity retains all singular modes (). Under CORE-ACL⋆, the validation truncation rank satisfies , forcing at least non-zero singular directions into the validation nullspace so that whenever the held-out tail carries label correlation.
Proposition 5 (Retrospective Margin Transfer Bound). *Fit CORE-ACL () and unbudgeted wide ridge () on the same lifted features at the same session and penalty . Then
Let denote the signed margin of under , and define the margin-violation event . For any narrow exact readout evaluated on the same test distribution,
Proof. Using the resolvent identity with and , we obtain . Since , submultiplicativity and Proposition 1(ii) yield the norm bounds. For any test vector , Cauchy–Schwarz gives . Thus, outside , the top-scoring class of coincides with that of , establishing the accuracy transfer inequality.
Worked Two-Coordinate Analytical Example
To see clearly why Theorem 1 distinguishes early-session refits from late-session updates, consider a 2-dimensional lifted space () with rank-1 factor , ridge penalty , and class cross-correlation , so that all label correlation lies orthogonal to .
Unrestored Low-Rank Ridge (): The inverted matrix is , yielding readout weights . A tiny off-subspace cross-correlation of is amplified into a dominant weight of .
CORE-ACL in Late Stream (, substantial discarded variance): The variance-restored matrix is , yielding . The restored self-variance acts as a natural coordinate stabilizer.
CORE-ACL on Session-1 Refit (, minimal discarded variance): Because the factor just crossed the rank cap , of coordinate variance is still inside (). The system matrix is , returning —still a blow-up. This explains why diagonal variance restoration alone cannot prevent Session-1 hold-out collapse at , and why CORE-ACL⋆’s rank-calibrated validation split is essential.
Algorithmic and Numerical Implementation Details
Algorithm 2: Task-Local GCV Penalty Selection
Input: Session features , targets , economy SVD , grid
;
for do
; ;
;
;
return (breaking ties toward smaller );
Algorithm 3: Hold-Out Validation with Rank Calibration (CORE-ACL*)
Input: Pre-session state , session data , grid , target capacity
Partition into stratified 90% fit split ( rows) and 10% val split ;
if using CORE-ACL* then
; // proportional truncation rank matching refit ratio
else
; // standard uncalibrated validation
Form temporary state from and using truncation rank ;
Evaluate validation accuracy on for every via Eq. (6); select best ;
Discard temporary state; update on full at rank and solve for at ;
Numerical precision and clipping safeguards. All persistent states are stored in IEEE-754 FP32. Before computing , we verify that to ensure that any negative entry is purely floating-point rounding rather than a violation of Loewner positivity. During the Woodbury normal-equation solve (Eq. (6)), , , are cast to FP64 so that the inner system is factored at double precision before casting back to FP32. Across all stored systems, the relative normal-equation residual remains below .
Extended Discussion and Positioning
Why Width Outweighs Covariance Rank Under a Fixed Byte Envelope
In an -dimensional random feature space, adding a new coordinate costs scalars for a dense covariance learner, but only scalars for CORE-ACL. At and , each coordinate is cheaper under CORE-ACL. Geometrically, random sparse lifting with TopK activation increases the minimum principal angle between class prototypes [7, 10], turning nonlinearly entangled backbone embeddings into linearly separable directions. Once classes are separated along distinct sparse coordinates, the normal equations primarily require accurate per-coordinate scaling () and leading shared principal components () to suppress background interference. Only when falls below the effective spectral dimension of the dataset (e.g., on CIFAR-100 at 16 MiB, Figure 6a) does unmodeled off-diagonal correlation outweigh the separability gain of higher width.
Figure 6. Spectral decay and session-by-task accuracy heatmaps. (a) Normalized singular spectrum of the retained factor : CIFAR-100 exhibits a flatter tail than CUB-200 and VTAB-50, explaining why underfits CIFAR-100. (b, c) CIFAR-100 session-by-task accuracy matrices at 36 MiB for Exact Narrow vs. CORE-ACL.
Relationship to Class-Statistics and Subspace-Partitioned Analytic Learners
Recent ICML and NeurIPS studies have explored complementary routes to compact continual learning. Cao et al. [40] and Momeni et al. (AnaCP) [39] maintain class-conditional means or shrunk class covariances in the unexpanded -dimensional backbone space. While effective on CLIP-scale encoders, operating in forfeits the to point separability gain provided by lifted coordinates on standard ViT-B/16 and ResNet-50 backbones. Conversely, LoRanPAC [15] lifts features to dimensions and truncates the SVD, but restricts the readout weights to row and stores unmerged per-class/per-session factors in its released implementation (879–4,218 MiB). CORE-ACL bridges both worlds: it retains the full -dimensional cross-correlation without subspace projection while compressing the shared covariance into a single fixed-budget tuple .
Design Alternatives We Evaluated and Rejected
Per-class factored covariances: Storing a separate rank- factor per class scales memory as and prevents pooling statistical strength across classes to estimate shared background directions.
Hard nullspace projection of incoming tasks: Projecting into eliminates cross-task covariance drift but destroys shared semantic features needed by fine-grained classes (such as bird species in CUB-200).
Trainable diagonal regularizers: Learning by gradient descent on each session breaks the closed-form recursive nature of analytic CIL and risks overfitting to the current session’s classes since old exemplars are absent. Proposition 1(iii) shows that is already Frobenius-optimal in closed form.
Detailed Experimental Settings
Datasets and Incremental Splits.
CIFAR-100 [42]: 50,000 train and 10,000 test images (100 classes), split uniformly into sessions (10 classes/session), plus (5 classes/session) and (2 classes/session) stress tests.
CUB-200-2011 [43]: Fine-grained recognition of 200 bird species (9,430 train / 2,358 test images in our local disjoint partition), evaluated over (20 classes/session), , and sessions.
VTAB-50 [46]: 50-class multi-domain benchmark (1,796 train / 8,619 test images across five visual domains), evaluated over (10 classes/session) and sessions.
ImageNet-R [44] & ImageNet-A [45]: 200-class domain-shifted and natural-adversarial benchmarks (, 20 classes/session), stressing covariance stability under severe distribution shift.
Exact Width and Rank Configurations under Memory Envelopes.
For a persistent memory budget (in bytes, 4 bytes per FP32 scalar), Exact Narrow chooses the largest integer such that , while Packed Exact Narrow chooses the largest such that . At full width , CORE-ACL chooses , while LR, FD, and RFD use rows:
| Dataset () | Envelope | Dense Exact | Packed Exact | CORE-ACL Rank () |
| CIFAR-100 () | 16 / 36 / 80 MiB | 1,998 / 3,024 / 4,532 | 2,802 / 4,250 / 6,382 | 318 / 840 / 1,996 |
| CUB-200 () | 16 / 36 / 80 MiB | 1,950 / 2,975 / 4,482 | 2,708 / 4,154 / 6,284 | 218 / 740 / 1,896 |
| VTAB-50 () | 16 / 36 / 80 MiB | 2,022 / 3,048 / 4,556 | 2,850 / 4,299 / 6,431 | 368 / 890 / 1,796{sup}†` |
Table 1 (PDF p. 14).
(†On VTAB-50 at 80 MiB, is capped by the 1,796 total training examples, leaving the state at 70.5 MiB.)
Comprehensive Supplementary Experiments
Full Benchmark Metrics Including Backward Transfer (BWT)
Sensitivity to Retained Subspace Rank
Complete Iso-Memory Width–Rank Grid (16 / 36 / 80 MiB)
Backbone Architectures and Long-Horizon Streams
Hold-Out Validation Across All Three Envelopes and Packed Baselines
Spectral Decay, Hyperparameter Sweeps, and Solver Profiling
| Dataset | Method | (%) | (%) | BWT (%) | Persist. MiB | Total MiB |
| CIFAR-100 | L2P [20] | 327.8 | 327.8 | |||
| CIFAR-100 | CODA-Prompt [22] | 331.2 | 331.2 | |||
| CIFAR-100 | EASE [23] | 48.3 | 48.3 | |||
| CIFAR-100 | ACIL [3] | 518.2 | 566.2 | |||
| CIFAR-100 | REAL [6] | 518.2 | 566.2 | |||
| CIFAR-100 | F-OAL [32] | 8.4 | 11.7 | |||
| CIFAR-100 | RanPAC [7] | 385.3 | 418.4 | |||
| CIFAR-100 | LoRanPAC [15] | 4218.3 | 4549.5 | |||
| CIFAR-100 | Fly-CL [10] | 385.3 | 423.4 | |||
| CIFAR-100 | CORE-ACL () | 35.9 | 74.1 | |||
| CUB-200 | L2P [20] | 328.1 | 328.1 | |||
| CUB-200 | CODA-Prompt [22] | 331.5 | 331.5 | |||
| CUB-200 | EASE [23] | 51.8 | 51.8 | |||
| CUB-200 | ACIL [3] | 524.5 | 572.5 | |||
| CUB-200 | REAL [6] | 524.5 | 572.5 | |||
| CUB-200 | F-OAL [32] | 9.2 | 12.9 | |||
| CUB-200 | RanPAC [7] | 389.1 | 426.0 | |||
| CUB-200 | LoRanPAC [15] | 3177.7 | 3546.9 | |||
| CUB-200 | Fly-CL [10] | 389.1 | 431.1 | |||
| CUB-200 | CORE-ACL () | 39.7 | 81.7 | |||
| VTAB-50 | L2P [20] | 327.7 | 327.7 | |||
| VTAB-50 | CODA-Prompt [22] | 331.0 | 331.0 | |||
| VTAB-50 | EASE [23] | 6.5 | 6.5 | |||
| VTAB-50 | ACIL [3] | 515.1 | 563.1 | |||
| VTAB-50 | REAL [6] | 515.1 | 563.1 | |||
| VTAB-50 | F-OAL [32] | 8.0 | 11.1 | |||
| VTAB-50 | RanPAC [7] | 383.4 | 414.6 | |||
| VTAB-50 | LoRanPAC [15] | 879.3 | 1191.3 | |||
| VTAB-50 | Fly-CL [10] | 383.4 | 419.6 | |||
| VTAB-50 | CORE-ACL () | 34.0 | 70.2 |
Table 4. Complete 5-seed benchmark results (mean SD) on CIFAR-100, CUB-200, and VTAB-50, complementing Table 1 with Backward Transfer (BWT, %) and total non-backbone deployment footprint (Total MiB, including projection and readout ).
| Rank | CIFAR-100 (%) | CIFAR-100 (%) | CUB-200 (%) | CUB-200 (%) | VTAB-50 (%) | VTAB-50 (%) | CIFAR State (MiB) |
| 128 | 8.7 | ||||||
| 256 | 13.6 | ||||||
| 512 | 23.4 | ||||||
| 840 | 35.9 | ||||||
| 1536 | 62.5 | ||||||
| 3072 | 121.0 |
Table 5. Sensitivity of CORE-ACL to retained interaction rank at (5 seeds, GCV).
Table 6. Iso-memory width–rank allocations under GCV (3 seeds). In 17 of 18 wide allocations, CORE-ACL improves over Exact Narrow; the sole negative cell is CIFAR-100 at 16 MiB full width (), where intermediate width (, ) succeeds.
| Envelope | Allocation ( / ) | CIFAR-100 (%) | CIFAR-100 (%) | CUB-200 (%) | CUB-200 (%) | VTAB-50 (%) | VTAB-50 (%) |
| 16 MiB | Exact Narrow ( / ) | 91.14 | 86.96 | 89.38 | 84.45 | 92.98 | 89.82 |
| 16 MiB | Intermediate ( / ) | 91.92 | 88.12 | 90.76 | 86.04 | 94.12 | 91.45 |
| 16 MiB | Full Width ( / ) | 90.52 | 85.34 | 91.57 | 87.18 | 94.70 | 92.36 |
| 36 MiB | Exact Narrow ( / ) | 91.48 | 87.52 | 89.94 | 85.08 | 93.66 | 90.84 |
| 36 MiB | Intermediate ( / ) | 91.94 | 88.15 | 90.68 | 85.82 | 94.16 | 91.50 |
| 36 MiB | Full Width ( / ) | 92.37 | 88.75 | 91.52 | 87.38 | 94.71 | 92.29 |
| 80 MiB | Exact Narrow ( / ) | 91.85 | 88.06 | 90.66 | 85.86 | 94.15 | 91.48 |
| 80 MiB | Intermediate ( / ) | 92.08 | 88.45 | 91.14 | 86.55 | 94.52 | 91.94 |
| 80 MiB | Full Width ( / ) | 92.36 | 88.73 | 91.48 | 87.16 | 94.73 | 92.31 |
Table 6.
Table 7. Generalization across ResNet-50, DINOv2 ViT-g/14, and 20-/50-session horizons at matched 36 MiB state (5 seeds; 3 for ).
| Backbone / Horizon | Method | CIFAR-100 (%) | CIFAR-100 (%) | CUB-200 (%) | CUB-200 (%) | VTAB-50 (%) | VTAB-50 (%) |
| ResNet-50 () | Fly-CL (dense ) | 84.15 | 76.74 | 78.31 | 69.58 | 93.31 | 89.90 |
| ResNet-50 () | Exact Narrow (36 MiB) | 82.60 | 73.98 | 75.18 | 65.96 | 92.85 | 89.15 |
| ResNet-50 () | CORE-ACL (36 MiB) | 83.26 | 75.82 | 78.56 | 69.66 | 93.28 | 89.80 |
| DINOv2 ViT-g/14 () | Fly-CL (dense ) | 96.29 | 94.19 | 93.68 | 91.24 | 96.72 | 94.72 |
| DINOv2 ViT-g/14 () | Exact Narrow (36 MiB) | 95.86 | 93.50 | 92.75 | 89.80 | 96.07 | 93.76 |
| DINOv2 ViT-g/14 () | CORE-ACL (36 MiB) | 96.25 | 94.10 | 93.69 | 91.32 | 96.71 | 94.70 |
| ViT-B/16 () | Fly-CL (dense ) | 93.04 | 88.81 | 91.66 | 87.06 | 95.61 | 92.80 |
| ViT-B/16 () | Exact Narrow (36 MiB) | 92.27 | 87.56 | 90.28 | 85.24 | 94.74 | 91.45 |
| ViT-B/16 () | CORE-ACL (36 MiB) | 93.03 | 88.75 | 91.69 | 87.15 | 95.60 | 92.78 |
| ViT-B/16 () | Fly-CL (dense ) | 92.80 | 88.74 | 92.02 | 87.14 | 95.71 | 92.50 |
| ViT-B/16 () | Exact Narrow (36 MiB) | 92.04 | 87.48 | 90.62 | 85.38 | 94.86 | 91.52 |
| ViT-B/16 () | CORE-ACL (36 MiB) | 92.76 | 88.70 | 92.10 | 87.35 | 95.70 | 92.49 |
Table 7.
Figure 5. Supplementary controls. (a) On CUB-200, CORE-ACL outperforms both Dense Narrow () and Upper-Triangular Packed Narrow () across all three memory envelopes. (b) Rank-calibrated hold-out selection (CORE-ACL*) eliminates Session-1 refit collapse across all truncated rules at 36 MiB. (c) Mean off-diagonal class-prototype correlation across pipeline stages.
| Dataset | Readout Rule | Hold-Out Validation + Refit 16 MiB | Hold-Out Validation + Refit 36 MiB | Hold-Out Validation + Refit 80 MiB | GCV: Packed vs. CORE-ACL 16 MiB | GCV: Packed vs. CORE-ACL 36 MiB | GCV: Packed vs. CORE-ACL 80 MiB |
| CIFAR-100 | Dense Exact Narrow | 91.30 | 91.62 | 91.86 | 91.14 | 91.48 | 91.85 |
| CIFAR-100 | Packed Exact Narrow | 91.58 | 91.84 | 92.05 | 91.52 | 91.81 | 92.06 |
| CIFAR-100 | Unrestored LR | 91.68 | 91.69 | 92.22 | 89.84 | 91.76 | 92.14 |
| CIFAR-100 | Robust FD (RFD) | 91.82 | 92.22 | 92.31 | 90.41 | 92.28 | 92.30 |
| CIFAR-100 | CORE-ACL (standard) | 91.60 | 92.20 | 92.28 | 90.52 | 92.37 | 92.36 |
| CIFAR-100 | CORE-ACL* | 91.64 | 92.21 | 92.29 | 90.52 | 92.37 | 92.36 |
| CUB-200 | Dense Exact Narrow | 91.25 | 91.44 | 91.64 | 89.38 | 89.94 | 90.66 |
| CUB-200 | Packed Exact Narrow | 91.42 | 91.60 | 91.78 | 89.92 | 90.46 | 91.10 |
| CUB-200 | Unrestored LR | 90.78 | 86.12 | 91.96 | 91.32 | 91.45 | 91.42 |
| CUB-200 | Robust FD (RFD) | 91.10 | 91.88 | 92.12 | 91.38 | 91.32 | 91.36 |
| CUB-200 | CORE-ACL (standard) | 90.68 | 88.68 | 91.62 | 91.57 | 91.52 | 91.48 |
| CUB-200 | CORE-ACL* | 91.46 | 91.89 | 91.94 | 91.57 | 91.52 | 91.48 |
| VTAB-50 | Dense Exact Narrow | 92.85 | 93.81 | 94.45 | 92.98 | 93.66 | 94.15 |
| VTAB-50 | Packed Exact Narrow | 93.34 | 94.08 | 94.58 | 93.50 | 94.01 | 94.44 |
| VTAB-50 | Unrestored LR | 92.96 | 94.21 | 94.85 | 94.62 | 94.68 | 94.71 |
| VTAB-50 | Robust FD (RFD) | 94.61 | 94.56 | 94.85 | 94.68 | 94.70 | 94.72 |
| VTAB-50 | CORE-ACL (standard) | 93.56 | 94.40 | 94.86 | 94.70 | 94.71 | 94.73 |
| VTAB-50 | CORE-ACL* | 93.62 | 94.41 | 94.86 | 94.70 | 94.71 | 94.73 |
Table 8. Hold-out validation and refit across MiB, plus comparison against Upper-Triangular Packed Exact Narrow (). Values are (%) over 3 seeds.
| Dataset | Sparsity Ratio () 0.10 | Sparsity Ratio () 0.30 | Sparsity Ratio () 0.50 | Sparsity Ratio () 1.00 | Row Non-Zeros () 50 | Row Non-Zeros () 150 | Row Non-Zeros () 300 | Row Non-Zeros () 600 | 40-Point Fine Grid LR | 40-Point Fine Grid Isotropic | 40-Point Fine Grid CORE-ACL |
| CIFAR-100 | 92.44 | 92.81 | 92.76 | 92.70 | 92.14 | 92.66 | 92.81 | 92.74 | 92.16 | 92.18 | 92.23 |
| CUB-200 | 91.18 | 91.52 | 91.46 | 91.35 | 90.88 | 91.39 | 91.52 | 91.48 | 91.56 | 91.88 | 91.84 |
| VTAB-50 | 94.82 | 95.07 | 95.04 | 94.96 | 94.65 | 94.98 | 95.07 | 95.05 | 92.88 | 93.64 | 93.81 |
Table 9. Ablation of sparse lifting hyperparameters (TopK ratio and row degree ) and 40-point fine-grid regularization control on CIFAR-100 and CUB-200 (, in %).
| Dataset | Dense Solve (ms) | CORE-ACL Solve (ms) | Speedup | Session Fit (s) | Persist. MiB | Peak GPU MiB |
| CIFAR-100 | 11.4 | 35.9 | 1,184 | |||
| CUB-200 | 2.1 | 39.7 | 512 | |||
| VTAB-50 | 0.6 | 34.0 | 358 |
Table 10. Computational latency, normal-equation solve speedup (RTX 3090), and peak GPU memory (RTX A5000) at MiB.
References
- [1]Da-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: A survey. arXiv preprint arXiv:2401.16386, 2024. 1, 2arxiv.org/abs/2401.16386
- [2]Zhiyan Hou, Dan Zhang, Tao Feng, Liyuan Wang, Wei Li, Xiangzhao Hao, Hongyan An, Junfeng Fang, Haokai Ma, Zhaohui Xu, Xinyu Tang, Haiyun Guo, Jingqiao Wang, and Tat-Seng Chua. Continual learning in transition. arXiv preprint arXiv:2608.06216, 2026. 1, 2arxiv.org/abs/2608.06216
- [3]Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Anh Toh, and Zhiping Lin. ACL: Analytic class-incremental learning with absolute memorization and privacy protection. In Advances in Neural Information Processing Systems (NeurIPS), pages 11602–11615, 2022. 1, 3, 6, 15arxiv.org/abs/2205.14922
- [4]Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, and Zhiping Lin. DS-AL: A dual-stream analytic learning for exemplar-free class-incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 17237–17244, 2024. 3, 6arxiv.org/abs/2403.17503
- [5]Huiping Zhuang, Yizhu Chen, Di Fang, Run He, Kai Tong, Hongxin Wei, Ziqian Zeng, and Cen Chen. GACL: Exemplar-free generalized analytic continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 3, 6arxiv.org/abs/2403.15706
- [6]Run He, Di Fang, Yizhu Chen, Kai Tong, Cen Chen, Yi Wang, Lap-Pui Chau, and Huiping Zhuang. REAL: Representation enhanced analytic learning for exemplar-free class-incremental learning. arXiv preprint arXiv:2403.13522, 2024. 1, 3, 6, 15DOI
- [7]Mark D. McDonnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, and Anton van den Hengel. RanPAC: Random projections and pre-trained models for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 1, 2, 3, 5, 6, 13, 15
- [8]Ameya Prabhu, Shiven Sinha, Ponnurangam Kumaraguru, Philip H. S. Torr, Ozan Sener, and Puneet K. Dokania. Ran-Dumb: Random representations outperform online continu- ally learned representations. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 3
- [9]Quyen Tran, Lam Tran, Khanh Doan, Toan Tran, Dinh Phung, Khoat Than, and Trung Le. Boosting multiple views for pretrained-based continual learning. In International Conference on Learning Representations (ICLR), 2025. 1, 3
- [10]Heming Zou, Yunliang Zang, Wutong Xu, and Xiangyang Ji. Fly-CL: A fly-inspired framework for enhancing efficient decorrelation and reduced training time in pre-trained model-based continual representation learning. In International Conference on Learning Representations (ICLR), 2026. 1, 2, 3, 5, 6, 13, 15arxiv.org/abs/2510.16877
- [11]Ruilin Li, Heming Zou, Xiufeng Yan, Zheming Liang, Jie Yang, Chenliang Li, and Xue Yang. Enhancing pretrained model-based continual representation learning via guided random projection. In European Conference on Computer Vision (ECCV), pages 246–264, 2026. 1, 3
- [12]Mina Ghashami, Edo Liberty, Jeff M. Phillips, and David P. Woodruff. Frequent directions: Simple and deterministic matrix sketching. SIAM Journal on Computing, 45(5):1762–1792, 2016. 1, 3
- [13]Luo Luo, Cheng Chen, Zhihua Zhang, Wu-Jun Li, and Tong Zhang. Robust frequent directions with application in online learning. Journal of Machine Learning Research, 20(45):1–41, 2019. 1, 3, 12
- [14]Benwei Shi and Jeff M. Phillips. A deterministic streaming sketch for ridge regression. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 586–594, 2021. 1, 3arxiv.org/abs/2002.02013
- [15]Liangzu Peng, Juan Elenter, Joshua Agterberg, Alejandro Ribeiro, and René Vidal. LoRanPAC: Low-rank random features and pre-trained models for bridging theory and practice in continual learning. In International Conference on Learning Representations (ICLR), 2025. 1, 3, 4, 6, 13, 15arxiv.org/abs/2410.00645
- [16]Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems (NeurIPS), pages 1257–1264, 2005. 2, 3
- [17]Gene H. Golub, Michael Heath, and Grace Wahba. Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics, 21(2):215–223, 1979. 2, 4DOI
- [18]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. 2, 5
- [19]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features
- [20]Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 139–149, 2022. 2, 6, 15arxiv.org/abs/2112.08654
- [21]Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. DualPrompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision (ECCV), pages 631–648, 2022. 2, 6arxiv.org/abs/2204.04799
- [22]James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. CODA-Prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11909–11919, 2023. 2, 6, 15arxiv.org/abs/2211.13218
- [23]Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23554–23564, 2024. 2, 6, 15arxiv.org/abs/2403.12030
- [24]Yan-Shuo Liang and Wu-Jun Li. InfLoRA: Interference-free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23638–23647, 2024. 3, 6arxiv.org/abs/2404.00228
- [25]Hai-Long Sun, Da-Wei Zhou, Hanbin Zhao, Le Gan, De-Chuan Zhan, and Han-Jia Ye. MOS: Model surgery for pre-trained model-based class-incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 20699–20707, 2025. 3arxiv.org/abs/2412.09441
- [26]Huiyi Wang, Haodong Lu, Lina Yao, and Dong Gong. Self-expansion of pre-trained models with mixture of adapters for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10087–10098, 2025. 3arxiv.org/abs/2403.18886
- [27]Heming Zou, Yunliang Zang, Wutong Xu, Yao Zhu, and Xiangyang Ji. FlyLoRA: Boosting task decoupling and parameter efficiency via implicit rank-wise mixture-of-experts. In Advances in Neural Information Processing Systems (NeurIPS), pages 10386–10419, 2025. 3arxiv.org/abs/2510.08396
- [28]Ziqi Wang, Chang Che, Qi Wang, Hui Ma, Zenglin Shi, Cees G. M. Snoek, and Meng Wang. Harmonious parameter adaptation in continual visual instruction tuning for safety-aligned MLLMs. arXiv preprint arXiv:2511.20158, 2025. 3arxiv.org/abs/2511.20158
- [29]Kai Jiang, Zisong Lin, Hongyuan Zhang, Xueru Bai, and Xuelong Li. Metrics learning with expandable subspace for pre-trained model-based class-incremental learning. arXiv preprint arXiv:2607.17593, 2026. 3DOI
- [30]Hongye Xu and Bartosz Krawczyk. Geometry-anchored transport framework for exemplar-free class-incremental learning. arXiv preprint arXiv:2606.25347, 2026.arxiv.org/abs/2606.25347
- [31]Tao Hu, Zhen-Hao Xie, Jingcai Guo, De-Chuan Zhan, and Da-Wei Zhou. Visual branch is what you need for CLIP-based class-incremental learning. arXiv preprint arXiv:2609.37888, 2026. 3arxiv.org/abs/2609.37888
- [32]Huiping Zhuang, Yuchen Liu, Run He, Kai Tong, Ziqian Zeng, Cen Chen, Yi Wang, and Lap-Pui Chau. FO-AL: Forward-only online analytic learning with fast training and low memory footprint in class incremental learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 41517–41538, 2024. 3, 6, 15
- [33]Yang Xiao, Tianyi Peng, Rohan Kumar Das, Yuchen Hu, and Huiping Zhuang. AnalyticKWS: Towards exemplar-free analytic class incremental learning for small-footprint keyword spotting. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14147–14158, 2025. 3arxiv.org/abs/2505.11817
- [34]Huiping Zhuang, Di Fang, Kai Tong, Yuchen Liu, Ziqian Zeng, Xu Zhou, and Cen Chen. Online analytic exemplar-free continual learning with large models for imbalanced autonomous driving task. arXiv preprint arXiv:2405.17779, 2024.DOI
- [35]Lei Song, Shihan Guan, and Youyong Kong. Analytic drift resister for non-exemplar continual graph learning. arXiv preprint arXiv:2604.02633, 2026.arxiv.org/abs/2604.02633
- [36]Quyen Tran, Hai Nguyen, Quan Dao, Zhuowei Li, Nam Le, Trung Le, and Dimitris Metaxas. Spectral-aware analytic class incremental learning for long-tailed distributions. arXiv preprint arXiv:2607.22931, 2026.arxiv.org/abs/2607.22931
- [37]Berk Sezer. An impossibility result for ETF target geometry, and an equinorm remedy for imbalanced analytic continual learning. arXiv preprint arXiv:2610.24844, 2026. 3arxiv.org/abs/2610.24844
- [38]Heming Zou, Yunliang Zang, and Xiangyang Ji. Structural features of the fly olfactory circuit mitigate the stability-plasticity dilemma in continual learning. arXiv preprint arXiv:2502.01427, 2025. 3arxiv.org/abs/2502.01427
- [39]Saleh Momeni, Changnan Xiao, and Bing Liu. AnaCP: Toward upper-bound continual learning via analytic contrastive projection. In Advances in Neural Information Processing Systems (NeurIPS), 2025. 3, 13arxiv.org/abs/2511.13880
- [40]Zhiwen Cao, Yanfeng Li, Shu-Dong Huang, Yalan Ye, Shuyin Xia, Yi Wang, and Jiancheng Lv. On the power of statistics in class-incremental learning with pretrained models. In Proceedings of the 43rd International Conference on Machine Learning (ICML), pages 11255–11271, 2026. 3, 13
- [41]Junhui Mao, Junru Zhang, Liming Ye, Cheng Peng, Hao-ran Shi, Xiaoli Tang, Baokuan Wang, Bin Dou, Yabo Dong, and Han Yu. TRACE: Memory-efficient time series class-incremental learning via temporal-spectral reconstruction. arXiv preprint arXiv:2610.35727, 2026. 3arxiv.org/abs/2610.35727
- [42]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009. 5, 14
- [43]Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 5, 14
- [44]Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8340–8349, 2021. 5, 14
- [45]Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), pages 15262–15271, 2021. 5, 14
- [46]Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019. 5, 14arxiv.org/abs/1910.04867
- [47]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 7arxiv.org/abs/1512.03385





