betaFeedback

WorksComputer Science

CaLLM: Continual Adaptation of LLMs via Non-Parametric Adapter Routing

Abstract

Continual adaptation of large language models (LLMs) to evolving task distributions while mitigating catastrophic forgetting remains an open problem. Existing strategies often struggle to scale or lack a robust mechanism for zero-shot task-to-expert assignment. We introduce CaLLM, a framework that resolves these tensions by combining metric-based meta-learning with modular parameter isolation via lightweight adapters. First, CaLLM projects each input batch into a task-agnostic embedding space to find its nearest prototype. It then routes the batch to the specific adapter specialized for that data distribution. This formulation provides (i)~controllable cross-task interference by design, (ii)~constant computational complexity regardless of task count, (iii)~seamless integration of novel distributions through prototype expansion, and (iv)~regime agnosticism as the local routing decision enables a unified approach to both online and offline settings. Our experiments demonstrate that this routing mechanism discovers semantically coherent task groupings while facilitating constructive interference between similar tasks, and keeping overhead at a minimum. Across diverse and fine-grained continual learning benchmarks, CaLLM consistently outperforms traditional isolation and replay methods, establishing prototype-based modular routing as a principled and scalable approach to lifelong adaptation of LLMs.

Introduction

Large language models (LLMs) have become remarkable generalists, but real-world deployment often requires them to acquire task-specific expertise without sacrificing their broader capabilities. The same underlying model needs to be continually adapted to different use cases ranging from closely related to radically different, without retraining from scratch. However, the standard practice of sequential fine-tuning typically triggers catastrophic forgetting [16], where adaptation to a new domain erodes performance on previously learned ones. A recent survey [27] formalizes this challenge along two axes: vertical continuity, moving from broad pretraining to task-specific fine-tuning, and horizontal continuity, adapting to shifting tasks and data distributions over time. Without careful management, adaptation along either of these dimensions typically results in severe performance degradation.

This problem is particularly pronounced for aligned, instruction-tuned LLMs. Wang et al. [32] demonstrate that even parameter-efficient methods such as LoRA [12] can severely degrade an aligned model’s general abilities and instruction-following quality after only a few rounds of sequential adaptation. Classical continual learning (CL) strategies such as regularization, experience replay, and orthogonal subspace isolation address specific aspects of this problem but they entail fundamental trade-offs. Regularization methods collapse when the model must transition between radically different domains (e.g., from natural language to code). Replay requires growing memory buffers that become unscalable as the task stream lengthens. Meanwhile, full retraining after each new task is computationally prohibitive, and real-world data often arrives as a continuous, non-stationary stream rather than the neatly delineated task blocks assumed in most CL benchmarks. Ideally, CL methods should adapt efficiently to new tasks in both online and offline settings.

We present CaLLM (Continual Adaptation of LLMs), a modular CL method that sidesteps these limitations through two interlocking components. First, CaLLM maintains a pool of modular experts (e.g., lightweight adapters implemented as LoRAs), each learning to specialize to a distinct region of the observed input distribution, ensuring that new knowledge is internalized without interfering with prior knowledge. Second, a lightweight non-parametric router, inspired by Prototypical Networks (Snell et al., [29]), maps each incoming batch to its most relevant adapter by comparing its (frozen-backbone) embeddings against stored prototypes. This mechanism dynamically and efficiently manages the adapter pool: it spawns new adapters when the data is sufficiently novel and re-purposes the least-used modules when the pool reaches full capacity. While Prototypical Networks use class-mean embeddings for few-shot classification, CaLLM uses distributional prototypes for task-level routing. This approach inherits the non-parametric simplicity of metric learning, where no gradient-based router optimization is required, and the model absorbs novel distributions simply by initializing new prototypes. A core architectural advantage of CaLLM is its constant computational overhead: only one adapter is active per forward pass, while the remaining adapters reside on disk.

We evaluate CaLLM on two complementary benchmarks: TRACE-8, which tests robustness to radical cross-domain shifts, and CITB-1911, which tests scalability on a long horizon dialogue stream, under offline (task-incremental) and online protocols. CaLLM outperforms CL baselines on the majority of tasks in both benchmarks, with the largest gains observed on specialized domains where the pretrained backbone is weakest. Furthermore, a systematic analysis of pool capacity reveals that moderate pooling outperforms both full isolation and single-adapter training in online settings. In this regime, we observe that interleaved gradients from semantically grouped tasks function as implicit replay, fostering constructive interference that enhances generalization without the overhead of a physical replay buffer.

In summary, our primary contribution is the complete CaLLM system, which unifies non-parametric routing with dynamic adapter isolation. Specifically:

  • We design and validate an end-to-end modular framework that achieves robust performance while maintaining a constant computational overhead by activating only one adapter per forward pass.

  • We introduce a training-free routing mechanism that scales dynamically to multiple tasks without requiring gradient-based router optimization.

  • We provide a comprehensive evaluation under both task-incremental and long-horizon online streaming scenarios, uncovering how constrained pool capacities foster constructive cross-task transfer.

The remainder of this paper is organized as follows. Section § 2 contextualizes our work in the CL literature, before section § 3 details the CaLLM method. Section § 4 presents our empirical analysis, including router representations, task-incremental and online results, and pool capacity effects. Section § 5 summarizes our findings and future directions.

Related Work

Continual learning foundations. CL aims to accumulate knowledge from non-stationary streams without catastrophic forgetting. In this paradigm, each newly introduced task represents an inherently out-of-distribution shift relative to the historical training data seen by the model so far. Existing methods generally utilize replay (Rolnick et al., [24]; Lopez-Paz & Ranzato, [20]), regularization (Farajtabar et al., [6]), or parameter isolation (Rusu et al., [25]; Kang et al., [15]). Beyond architectural design, evaluation is a critical aspect. Recent studies advocate for per-iteration metrics to detect the “stability gap”—a transient but severe forgetting phase during training (De Lange et al., [3])—and transferability-aware embeddings to leverage inter-task relationships (Wu et al., [37]). Online Continual Learning (OCL) (Cai et al., [1]) further emphasizes learning efficacy via a single-pass “test-then-train” protocol. This regime requires optimizing for immediate adaptation rather than long-term convergence, tracking performance through temporal moving averages instead of static end-of-task scalars.

Continual learning for LLMs. Adapting LLMs involves navigating vertical (pretrain-to-fine-tune) and horizontal (temporal shift) continuity. We focus on Continual Fine-Tuning (CFT), evaluated via benchmarks like TRACE (cross-domain robustness) and CITB (long-horizon instruction streams). LoRA is the primary vehicle for CFT, with methods managing interference via subspace isolation, enforcing orthogonality through geometric constraints (Wang et al., [31]; Liang & Li, [17]) or gradient projection (Qiao et al., [23]). TreeLoRA (Qian et al., [22]) organises per-layer LoRAs in a gradient-similarity tree, exploring task relationships via lower-confidence-bound bandits. However, these often sum branches with equal weight at inference, failing to select the most relevant parameters. While GainLoRA (Liang & Li, [18]) and SwitchCIT (Wu et al., [36]) introduce gating or switch networks, they require specialized optimization. Conversely, replay-based methods (Scialom et al., [26]; Wang et al., [33]; He et al., [10]) maintain performance but add memory overhead. Notably, recent work suggests that much degradation in LLMs stems from disrupted task alignment rather than true knowledge loss (Ma et al., [21]).

Routing and mixture-of-experts approaches. Sparse Mixture-of-Experts (MoE) architectures decouple model capacity from per-input computation by routing each token to a small subset of parallel experts [7, 14]. Learned routers discover interpretable data partitions aligned with latent clusters [4]. In parameter-efficient fine-tuning (PEFT), LoRAMoE [5] and MoRAL [38] use balancing losses to preserve world knowledge, while MoA [8], SEE [35], and MLLM-CL [40] assigns specific LoRA modules to different tasks or modalities, routing them at inference time to minimize interference. Despite their modularity, these methods generally rely on gradient-based gating and specialized training objectives and they are highly susceptible to routing drift and catastrophic forgetting within the router itself. Furthermore, they require complex balancing and regularization losses to prevent expert collapse. CaLLM overcomes this by using a non-parametric metric-based router, effectively decoupling router scaling from gradient optimization.

Prototypical networks and metric-based routing. CaLLM’s router is grounded in metric-based meta-learning [30], drawing inspiration from Prototypical Networks. By computing distances to centroids in a learned embedding space, this non-parametric bias enables robust adaptation without inner-loop optimization. While prototypes have traditionally been used for low-level class-incremental classification [2, 41], CaLLM transposes this to high-level system routing. Rather than using embeddings to represent a single class, CaLLM uses embedding prototypes to capture the geometric boundaries of entire multi-task streams. This allows the model to map task-level representation to isolated parameter spaces that are assigned to specific adapters. This yields a training-free router that inherits the modularity of MoE but avoids the overhead of learned gating. By combining parameter isolation with an online, no-replay constraint, CaLLM provides a scalable alternative that handles novel tasks via prototype expansion and bounds memory through an repurposing policy.

Continual Adaptation of LLMs (CaLLM)

CaLLM addresses catastrophic forgetting through a prototype-based routing mechanism coupled with dynamic, task-specific parameter-efficient adapters. The central idea is to maintain a pool of lightweight adapter modules, each specializing in a region of the task embedding space. As illustrated in Figure 1, the system pipeline integrates four primary processes: (i) a non-parametric router that extracts embeddings from incoming data and maps data batches to adapters; (ii) an adapter pool that manages a collection of PP parameter-efficient adapters and their associated prototypes; (iii) a model trainer that performs localized SFT on selected adapters; and (iv) an evaluation pipeline that assesses performance across task streams to quantify learning progress and monitor for catastrophic forgetting. By routing data based on semantic similarity or spawning new adapters for sufficiently novel inputs, CaLLM decouples task capacity, allowing selective reuse or isolation as the task distribution evolves.

CaLLM system overview (training + evaluation)

Figure 1. CaLLM system overview (training + evaluation). Each new data batch is embedded with a frozen base model (using mean-pooling). The non-parametric router manages a dynamic pool of specialized adapters, each mapped to a prototype. Based on semantic similarity between the new embedding and existing prototypes, the router triggers the reuse, creation, or repurposing of an adapter. The selected adapter is then trained or finetuned and the pool is updated. The diagram further details how this routing mechanism integrates with the distinct training and evaluation loops.

CaLLM operates in two modes: Batched (CaLLM-B), where tasks arrive sequentially, and Online (CaLLM-O), where small, randomly interleaved chunks from multiple tasks arrive in a non-stationary stream. Both share routing, training, and evaluation components but differ in data presentation order, evaluation protocol, and CL metric computation.

Adapter Pool

CaLLM maintains a pool P={(ℓi,pi,ci)}i=1∣P∣\mathcal{P} = \{(\ell_i, \mathbf{p}_i, c_i)\}_{i=1}^{|\mathcal{P}|} of parameter-efficient adapters, with maximum capacity PP. Each adapter ℓi\ell_i is paired with a prototype embedding pi∈Rd\mathbf{p}_i \in\mathbb{R}^{d} that summarizes the data distribution it was trained on and a usage counter cic_i. The proposed system is both backbone- and adapter-agnostic, supporting any Transformer-based LLM and any PEFT method that can be independently instantiated, saved, and loaded (e.g., LoRA [12], (IA)3 [19], or adapter modules [11]). We use the terms expert and adapter interchangeably here, since each expert in the pool is realized as a parameter-efficient adapter. In our experiments we use LoRA as the default adapter, where each ℓi\ell_i consists of low-rank decompositions ΔW=BA\Delta W = BA injected into all linear projections in each Transformer block, parameterized with rank rr and scaling factor α=2r\alpha= 2r. The modified forward pass for a projection WW becomes:

h′=Wx+αrBiAix,(1)\mathbf{h}' = W\mathbf{x} + \frac{\alpha}{r}B_iA_i\mathbf{x}, \tag*{(1)}

where Ai∈Rdin×rA_i \in\mathbb{R}^{d_{\mathrm{in}}\times r} and Bi∈Rr×doutB_i \in\mathbb{R}^{r\times d_{\mathrm{out}}} are the trainable low-rank factors of the currently loaded adapter ℓi\ell_i. All backbone parameters remain frozen throughout training and only the active adapter’s parameters are updated. This ensures that (i) the base model’s representations remain stable, and (ii) adapters from different tasks do not interfere with one another at the parameter level. Because only one adapter is active per forward pass while other reside in secondary storage, CaLLM ensures fixed-cost modularity: training FLOPs and GPU memory remain constant regardless of pool size ∣P∣|\mathcal{P}|. Notably, routing does not compromise this property. Its additional pass through the frozen backbone is encoder-only, reuses the already-loaded model, and is invoked once per task (or once per chunk in the online setting) rather than once per query. Consequently, routing contributes only about 0.5% of training FLOPs and less than 0.1% of per-query inference cost (see Appendix § 7.§ 2 for quantitative results).

Non-Parametric Router

The router maps incoming data to the most appropriate adapter in the pool via two stages: embedding extraction and prototype matching.

Embedding extraction. Given a batch of nn input prompts, the router extracts a distributional representation ee by performing a forward pass through the frozen backbone with all adapters disabled. Let hjt(L)h_{jt}^{(L)} denote the hidden state at position tt of sequence jj in the final transformer layer LL. We compute a single embedding by mean-pooling over valid (non-padding) tokens and averaging across the batch:

e=1n∑j=1n∑t=1sjmjt⋅hjt(L)∑t=1sjmjt,(2)\mathbf{e} = \frac{1}{n}\sum_{j=1}^{n}\frac{\sum_{t=1}^{s_j}m_{jt}\cdot\mathbf{h}_{jt}^{(L)}}{\sum_{t=1}^{s_j}m_{jt}}, \tag*{(2)}

where mjt∈{0,1}m_{jt} \in\{0,1\} is the attention mask and sjs_j is the sequence length. Input sequences are tokenized with truncation at a configurable maximum length. By using the frozen backbone without adapters, the embeddings reflect the base model’s representation of the input distribution, providing a stable reference frame that works from t=0t = 0 and does not drift as adapters are trained.

Prototype matching. The router computes the cosine distance between the new embedding ee and each stored prototype pip_i:

di=1−e⋅pi∥e∥∥pi∥,i=1,…,∣P∣.(3)d_i = 1 - \frac{e \cdot p_i}{\lVert e\rVert\lVert p_i\rVert}, \quad i = 1,\ldots,|\mathcal{P}|. \tag*{(3)}

The adapter selection then follows a three-way decision rule governed by a distance threshold η\eta:

  1. Reuse (dmin⁡≤ηd_{\min} \le\eta): Load the closest adapter ℓarg⁡min⁡idi\ell_{\arg\min_i d_i}, update its prototype via an exponential moving average with equal weighting pi←12(pi+e)p_i \leftarrow\frac{1}{2}(p_i + e), and increment its usage counter cic_i.

  1. Create (dmin⁡>ηd_{\min} > \eta and ∣P∣<P|\mathcal{P}| < P): Initialize a new adapter from the base weights, add it to the pool with prototype pnew=ep_{\mathrm{new}} = e and usage counter cnew=1c_{\mathrm{new}} = 1.

  1. Repurpose (dmin⁡>ηd_{\min} > \eta and ∣P∣=P|\mathcal{P}| = P): Repurpose the least-used adapter slot (by usage count cic_i), reassign its prototype to ee, reset its usage counter, and overwrite its weights during subsequent fine-tuning. Although repurposing can overwrite a rare adapter, our capacity analysis shows the router consistently groups similar tasks together, preventing such loss in practice.

During evaluation, the router operates in read-only mode. It finds the closest prototype and returns the associated adapter without creating or modifying prototypes. The threshold η\eta controls the granularity of the pool, lower values encourage more adapters (finer specialization), while higher values promote sharing across related distributions. We analyze the effect of pool capacity in § 4.5.

When no fixed threshold is provided, an adaptive threshold is computed as half the minimum pairwise cosine distance between the closest prototype and all others:

ηadapt=12min⁡j≠i∗(1−pi∗⋅pj∥pi∗∥∥pj∥),(4)\eta_{\mathrm{adapt}} = \frac{1}{2}\min_{j \ne i^*}\left(1 - \frac{p_{i^*} \cdot p_j}{\lVert p_{i^*}\rVert\lVert p_j\rVert}\right), \tag*{(4)}

where i∗=arg⁡min⁡idii^* = \arg\min_i d_i is the closest prototype to the incoming embedding. This ensures the threshold tightens as the embedding space becomes more densely populated. When only one prototype exists, a default threshold of 0.1 is used.

Training and Evaluation Protocol

Each training step applies SFT on the selected adapter using the standard causal language modeling cross-entropy loss:

L=−∑t=1∣y∣log⁡Pθadapt(yt∣y<t,x),(5)\mathcal{L} = -\sum_{t=1}^{|y|} \log P_{\theta_{\mathrm{adapt}}}(y_t \mid y_{<t}, x), \tag*{(5)}

where xx is the input prompt, y=(y1,…,y∣y∣)y = (y_1, \ldots, y_{|y|}) is the target sequence, and θadapt\theta_{\mathrm{adapt}} denotes the adapter parameters. Optimization hyperparameters (e.g. optimizer, learning rate schedule, gradient clipping, numerical precision) as well as input formatting are all configurable and decoupled from the core method. Specific settings used in our experiments are reported in § 4.

CaLLM supports two CL regimes that share the training objective above but differ in data presentation, evaluation timing, and metric computation. Full pseudocode for both protocols is provided in Appendix § 7.§ 1.

Batched task-incremental protocol. In Batched mode, tasks arrive sequentially as complete datasets D1,D2,…,DT\mathcal{D}_1, \mathcal{D}_2, \ldots, \mathcal{D}_T, each with train, evaluation, and test splits (Algorithm 1). For each task with dataset Dt\mathcal{D}_t, the system first extracts an embedding from a sample of the training split and queries the router (with prototype updates enabled) to select or create an adapter. The selected adapter is then loaded onto the base model and fine-tuned on the full training split using the SFT objective (Eq. (5)), with the evaluation split used for early stopping and checkpoint selection. After training, the adapter is saved to disk.

Evaluation proceeds in two phases. First, the trained adapter is evaluated on the current task’s test set to measure adaptation quality. Second, to assess backward transfer, all previously seen tasks j<tj < t are re-evaluated: for each prior task, the router operates in read-only mode (no prototype updates) to find the best-matching adapter under the current prototype set, and that adapter is loaded and evaluated on task jj’s test set. Backward transfer is then computed as the average normalized performance change across prior tasks.

Online continual learning protocol. In Online mode, data arrives as a stream of small chunks randomly interleaved across multiple tasks in a non-stationary distribution. (Algorithm 2). Each chunk contains a fixed number of examples (chunk_size) from a single task.

Following the test-then-train paradigm, the protocol begins with the router extracting a chunk’s embedding eie_i to select, create, or repurpose an adapter according to the logic described above. Once the parameter subspace is determined, the chunk is immediately evaluated using the selected adapter, or the frozen backbone if a new adapter is being initialized, prior to any gradient updates. This pre-update score provides an unbiased measure of the system’s current knowledge and is used to update the continual learning metrics defined below. After evaluation, the selected adapter is fine-tuned on the input chunk via SFT (Eq. (5)). This single-pass procedure ensures structural consistency and strictly avoids post-update evaluation to prevent information leakage, allowing CaLLM to satisfy the core assumptions of online learning while tracking knowledge accumulation in real-time.

Continual learning metrics. We report the following measures to quantify plasticity and stability across the task stream:

  • Average Performance: In the batched setting, after learning task tt, we compute the normalized average over all tasks seen so far (each evaluated with its best-matching adapter via read-only routing): Avgt=1t∑j=1tRj,t/sj\mathrm{Avg}_{t} = \frac{1}{t}\sum_{j=1}^{t} R_{j,t}/s_j, where Rj,tR_{j,t} is the performance on task jj after learning task tt and sjs_j is the metric scale. This captures overall knowledge retention and adaptation quality at each point in the task sequence. In the online setting, the equivalent is the Online Moving Average (OMA), a running normalized average updated at each step with the latest pre-update evaluation: OMAt=1t∑k=1tRkpre/sk\mathrm{OMA}_{t} = \frac{1}{t}\sum_{k=1}^{t} R_{k}^{\mathrm{pre}}/s_k, where sks_k is the task-specific metric scale at step kk. As a moving average over the stream, OMA tracks how the system’s overall performance evolves in real-time, reflecting its ability to generalize to new distributions.

  • Backward Transfer (BWT): Measures the impact of learning new tasks on previously learned ones. In the batched setting, after learning task tt: BWTt=1t−1∑j=1t−1(Rj,t−Rj,j)/sj\mathrm{BWT}_{t} = \frac{1}{t-1}\sum_{j=1}^{t-1}(R_{j,t}-R_{j,j})/s_j, where Rj,jR_{j,j} is the performance immediately after learning task jj and sjs_j is the metric scale. In the online setting, BWT is computed as: BWTj,t=(Rjpre,t−Rjpre,prev)/sj\mathrm{BWT}_{j,t} = (R_{j}^{\mathrm{pre},t}-R_{j}^{\mathrm{pre,prev}})/s_j, where Rjpre,prevR_{j}^{\mathrm{pre,prev}} is the pre-update score from the most recent prior observation of task jj. Unlike the batched variant above, online BWT measures the change between consecutive observations of the same task rather than from the initial post-training score.

Experiments & Results

We organize our experimental analysis around a progression from defining setup to understanding mechanism, followed by performance evaluation and stress-testing.

Experimental Setup

Benchmarks. We evaluate CaLLM on two complementary benchmarks that test different axes of CL difficulty:

  1. TRACE-8 Wang et al. [32] (Diversity & Shift) comprises 8 heterogeneous tasks spanning domain-specific QA (ScienceQA, FOMC, MeetingBank), multilingual processing (C-STANCE, 20Minuten), code completion (Py150), and math reasoning (NumGLUE-cm/ds) testing robustness to radical distribution shifts. We utilize the variant with 5000 training examples per task.

  1. CITB-19 Zhang et al. [39] (Scale & Granularity) consists of 19 dialogue-centric tasks derived from Super-NaturalInstructions Wang et al. [34], categorized into Dialogue State Tracking (4), Dialogue Generation (11), and Intent Identification (4). This suite is used to test long-horizon streams and fine-grained intra-domain discrimination. To ensure training stability, we utilize a refined version of this benchmark augmented with 3000 high-quality synthetic examples per task (see Appendix § 6.§ 5 for the generation and filtering protocol). This is the dataset version used in all our experiments.

Task performance is measured using standard metrics appropriate to each task type: accuracy for classification and multiple-choice tasks, ROUGE-L for summarization and all CITB-19 tasks, SARI for text simplification, and EDIM (edit-distance fuzzy match ratio) for code completion. In label-free stream settings, metric assignment is handled automatically via LLM-based task classification (Appendix § 6.6).

All input data is formatted using an Alpaca-style instruction template (Instruction / Response / <eos>), ensuring a uniform interface across diverse task types. Following the protocols defined in § 3.3, the Batched setting processes tasks as complete sequential blocks, while the Online setting uses a non-stationary stream of max_chunks steps. Each step consists of chunk_size examples, with chunks randomly interleaved across tasks using a fixed seed to ensure reproducibility.

Baselines. We compare CaLLM against five representative CL strategies: Sequential Adapter Fine-tuning (Seq-FT), a single LoRA adapter fine-tuned sequentially on the task stream; Inference-only, the zero-shot performance of the frozen backbone; Elastic Weight Consolidation (EWC) Kirkpatrick et al. [16], a regularization-based method penalizing shifts in important weights; Experience Replay (ER) Rolnick et al. [24], a replay-based method with a fixed memory buffer; Orthogonal LoRA (oLoRA) Wang et al. [31], a parameter-efficient CL baseline that learns tasks in orthogonal low-rank subspaces. TreeLoRA Qian et al. [22]: layer-wise LoRAs organised by a hierarchical gradient-similarity tree with bandit-based task-relationship exploration.

All experiments use gemma-3-12b-it (4-bit quantized) as the backbone, selected for its superior zero-shot performance across all tasks (see Appendix § 6.§ 1). Appendix § 6.§ 2 shows implementation details and hyperparameter configuration. The code will be openly released upon paper acceptance.

Router Representations: Why Modular Routing Works

CaLLM’s ability to mitigate interference in task-agnostic settings depends on the router’s capacity to recognize distributional boundaries from input features alone. Figure 2 visualizes 2D projections of the TRACE-8 benchmark. The router produces well-separated clusters, with radical domain shifts (e.g., code vs. multilingual text) pushed far apart in embedding space. This geometric separation is expected given the large distributional gaps between TRACE tasks, but it confirms that the router’s lightweight architecture and a base-model representation is sufficient to capture these boundaries without additional finetuning.

Router embedding space 2D projection for TRACE-8

Figure 2. Router Embedding Space (2D Projection) for TRACE-8. t-SNE projections of the router’s embeddings, colored by task identity.

Similarly, on CITB-19 (Figure 9(b) in Appendix § 9.§ 2), the router discriminates between 19 fine-grained dialogue sub-tasks, such as dialogue state tracking and intent identification, despite their shared conversational format. This intrinsic geometric separation allows for reliable, task-agnostic routing. When pool capacity PP is sufficient, each cluster maps to a dedicated adapter, achieving full parameter isolation. The consequences of restricting PP and forcing cluster merges are analyzed in § 4.5.

Two-panel t-SNE projections of router embeddings for TRACE-8 and CITB-19

Figure 9. Router Embedding Space (2D Projection). t-SNE projections of the router’s embeddings, coloured by task identity. (a) TRACE-8: 8 tasks spanning diverse domains. (b) CITB-19: 19 dialogue sub-tasks.

Task-Incremental Results

We first evaluate CaLLM’s end-to-end performance in the standard batched task-incremental setting, where tasks arrive one at a time and the system trains on each complete dataset sequentially.

Final performance by individual task for CaLLM-B and other baselines for TRACE benchmark is shown in Table 1. CaLLM-B achieves the highest score among all baseline methods on 4 of 8 tasks. The most significant gains over the inference-only baseline appear on specialized domains where the pre-trained backbone is weakest: code completion (Py150, +2.33 in EDIM), science reasoning (ScienceQA, 0.35 → 0.81 in accuracy), and math (NumGLUE-ds, 6× improvement in accuracy). TreeLoRA’s wins concentrate on the two arithmetic splits, where its hierarchical gradient-similarity tree consolidates closely related skills (mean accuracy 0.60 across NumGLUE-cm/ds, versus 0.28 for CaLLM-B), but it collapses on Py150 (5.52 vs. 17.24 in EDIM), where the code-generation gradient signature is dissimilar from the natural language tasks and the tree potentially mis-routes the adapter, a failure mode that CaLLM-B’s prototype router avoids by treating each task as a first-class cluster. Conversely, on C-STANCE and 20Minuten, performance is largely inherited from the backbone; on the former, fine-tuning slightly degrades pre-existing competence, while on the latter, all methods fall within a narrow 2-point SARI band. Because 20Minuten is the final task in the sequence, no method suffers from forgetting, narrowing the performance gap between CaLLM-B and baselines. Notably, CL baselines underperform Inference-Only on several tasks (e.g., C-STANCE, FOMC, MeetingBank, Py150). We attribute this to Inference-Only’s strong large-scale backbone—whose pre-trained competence makes it a strong baseline rather than a trivial lower bound—and to its avoidance of the catastrophic forgetting that erodes CL baselines as the task stream grows (Appendix § 6.§ 3 shows this gap persists even with per-baseline hyperparameter tuning). CaLLM-B avoids this trade-off by routing task-specific updates to isolated experts, preserving the backbone’s zero-shot competence while enabling specialization.

TaskMetricInf-OnlySeq-FTEWCERoLoRATreeLoRA 2CaLLM-B
C-STANCEacc (0–1)0.620.030.280.040.090.410.43
FOMCacc (0–1)0.640.300.310.140.060.510.68
MeetingBankROUGE-L (0–1)0.210.060.100.100.160.110.23
Py150EDIM (0–100)14.9111.9610.5512.989.815.5217.24
ScienceQAacc (0–1)0.350.130.610.190.000.720.81
NumGLUE-cmacc (0–1)0.050.140.120.060.040.670.15
NumGLUE-dsacc (0–1)0.070.280.150.130.050.540.42
20MinutenSARI (0–100)39.2238.0640.0639.6339.9141.0539.49
Avg. perf.normalized (0–1)0.290.160.260.150.150.390.41
BWT(higher is better)–-0.21-0.10-0.18-0.11–0.01

Table 1. Per-Task Final Performance on TRACE-8 (Batched). Final scores after training on all tasks sequentially.

Figure 3 shows the evolution of average normalized per-task performance across the sequential task stream both for TRACE-8 and CITB-19 datasets, while a detailed per-task performance breakdown can be seen in Table 7 and Table 8 in Appendix § 8.2.1. On TRACE-8, baselines struggle with radical distribution shifts. EWC’s performance declines significantly upon encountering tasks that require more complex reasoning (Py150 at Task-4), while ER and oLoRA exhibits a steady downward trend in later tasks. In contrast, CaLLM-B demonstrates robust stability throughout the entire sequence, by routing incompatible distributions (e.g., code vs. math) to separate experts, as shown in Figure 2(a). CaLLM-B finishes the 8-task sequence with an average normalized score of 0.44, substantially outperforming the Inference-Only baseline (0.31) and ER (0.31). We additionally verified that using Gemma 3’s native chat template for the Inference-Only baseline does not yield a stronger competitor, making our reported Inf-Only a conservative comparison (Appendix § 8.§ 3). The narrow variance bands (±1 SD) confirm CaLLM-B’s robustness to stochasticity. On CITB-19 (Figure 3b), the challenge shifts to sequence length. While Seq-FT degrades rapidly due to interference within a single adapter, CaLLM-B maintains a stable trajectory across all 19 tasks. By instantiating new experts as needed (P≈19P \approx19), the system prevents parameter overloading. The tight variance bands across all 19 steps confirm that this stability is not degrading even when sequence task length increases. CaLLM-B achieves the highest score among CL methods on 12 of 19 tasks, demonstrating broad competence across fine-grained CITB dialogue subtypes (see Table 9, Appendix § 8.2.2). However, fine-tuning on CITB-19 does not improve over Inference-Only for any method, likely because dialogue tasks are within the natural competence of a 12B instruction-tuned model (see Appendix § 6.§ 4).

Task-incremental learning performance for TRACE-8 and CITB-19

Figure 3. Task-Incremental Learning Performance (mean ±1 std). Average normalized score as tasks are added sequentially, for CaLLM-B and all baselines. (a) TRACE-8. (b) CITB-19.

TaskC-STANCEFOMCMeetingBankPy150ScienceQANumGLUE-cmNumGLUE-ds20Minuten
C-STANCE0.420.420.420.420.410.420.420.42
FOMC–0.560.550.560.560.560.560.56
MeetingBank––0.220.230.230.230.230.23
Py150–––16.9416.9516.8517.1517.12
ScienceQA––––0.560.570.560.56
NumGLUE-cm–––––0.190.190.19
NumGLUE-ds––––––0.300.30
20Minuten–––––––39.59

Table 7. Per-Task Performance Table on TRACE-8 for CaLLM-B. Results are reported for a single random seed. Metrics used are accuracy (C-STANCE, FOMC, ScienceQA, NumGLUE-cm/ds), ROUGE-L (MeetingBank), EDIM (Py150), and SARI (20Minuten).

Tasktask1590task1713task565task1730task573task294task1500task1603task576task639task1714task1501task1729task574task848task611task766task1384task1600
task15900.410.500.550.440.530.450.520.440.460.460.400.430.470.380.560.550.440.460.53
task1713–0.600.520.560.530.590.590.510.560.530.550.640.520.530.520.490.560.490.54
task565––0.370.410.380.390.330.310.370.330.290.290.210.360.370.290.450.290.33
task1730–––0.240.150.210.180.140.180.180.200.210.180.210.190.130.200.140.18
task573––––0.450.430.470.490.480.440.550.560.450.440.340.500.560.600.51
task294–––––0.110.110.120.090.100.120.130.150.130.130.110.160.090.14
task1500––––––0.090.070.130.100.130.060.140.090.160.050.130.110.14
task1603–––––––0.700.800.760.620.700.740.760.610.660.650.590.67
task576––––––––0.080.100.090.070.070.100.070.070.070.070.08
task639–––––––––0.250.290.200.250.220.230.240.180.230.23
task1714––––––––––0.090.100.060.040.070.050.090.080.05
task1501–––––––––––0.720.800.840.720.720.840.920.76
task1729––––––––––––0.080.080.200.150.080.120.04
task574–––––––––––––0.090.040.080.080.090.09
task848––––––––––––––0.270.270.230.280.29
task611–––––––––––––––0.310.280.230.21
task766––––––––––––––––0.180.210.25
task1384–––––––––––––––––0.040.04
task1600––––––––––––––––––0.30

Table 8. Per-Task Performance Table on CITB-19 for CaLLM-B. Results are reported for a single random seed. The metric used is ROUGE-L for all tasks.

TaskInf-OnlySeq-FTEWCERoLoRACaLLM-B
Diplomacy (T1590)0.840.140.050.200.100.31
ConvAI3-17130.360.050.070.140.180.28
CIRCA (T565)0.080.100.040.090.280.40
P.Chat Choose (T1730)0.020.240.020.090.390.45
AirDial. Class. (T573)0.920.050.070.080.270.36
Story Motiv. (T294)0.340.090.070.090.490.25
DSTC3 Class. (T1500)0.120.050.000.030.120.44
SMCalFlow-16030.680.120.060.120.110.47
Curiosity Dial. (T576)0.250.280.030.040.290.28
MultiWOZ (T639)0.140.070.100.080.320.15
ConvAI3-17140.200.110.020.230.120.25
DSTC3 Ans. (T1501)0.060.050.030.070.310.41
P.Chat Gen. (T1729)0.820.160.080.070.180.09
AirDial. Gen. (T574)0.220.110.050.090.160.20
PubMedQA (T848)0.080.170.040.050.060.21
Mutual Dial. (T611)0.840.050.120.190.360.26
Craigslist (T766)0.210.130.050.120.110.36
DealOrNo (T1384)0.070.120.100.100.260.38
SMCalFlow-16000.260.360.190.280.500.35

Table 9. Per-Task Final Performance on CITB-19 (Batched). Scores measured after all 19 tasks have been trained sequentially; all tasks use ROUGE-L (0–1) averaged over 3 seeds.

We also analyzed the BWT dynamics across both benchmarks in Figure 7 (Appendix § 8.2.1) in the batched scenario. Across both TRACE-8 and CITB-19, CaLLM-B is the only method that consistently maintains a positive, or close to 0, BWT, with a very low variance across different random seeds. Conversely, all competing baselines exhibit a steady degradation in BWT as more tasks are introduced to the stream. This stark contrast demonstrates that CaLLM successfully learn new tasks without forgetting the previously acquired knowledge.

Figure 7. Task-Incremental BWT Comparison Across Baselines (mean ±\pm std). CaLLM-B maintains a BWT score close to zero throughout the stream on both benchmarks.

These gains come at negligible additional compute. Because only one adapter is active per forward pass, CaLLM consumes the same 7339 TFLOPs of training compute as single-adapter Seq-FT. The router adds only a small increment: Using the standard 2N2N FLOPs-per-token estimate with N≈12×109N \approx12 \times10^{9} for gemma-3-12b-it, a routing call over n≈20n \approx20 prompts of ∼10\sim10 tokens costs ≈4.8\approx4.8 TFLOPs, so across TRACE-8’s eight tasks the router consumes ≈38.4\approx38.4 TFLOPs (only ≈0.5%\approx0.5\% of training). At inference this amortizes over all evaluation queries within a task, giving a per-query overhead ≪0.1%\ll0.1\% (see Appendix § 7.§ 2 for further details).

Online Streaming Dynamics

Having established CaLLM’s task-incremental performance, we now examine the fine-grained dynamics of online learning: How does the router behave over time as the data stream unfolds, and how does expert specialization emerge without supervision? Unlike the batched setting, we do not report multi-seed variance here expecting the OMA metric averaging over 500 evaluation steps to provide inherent statistical stability.

Figure 4 summarises CaLLM-O’s behavior over the TRACE online stream. The top panel (Figure 4a) tracks the OMA performance whereas the bottom panel (Figure 4b) zooms into a representative 100-chunk window (chunks 200–300) of the task ×\times expert assignment heatmap, where each row is a task, each column a chunk, and color encodes the assigned expert. The full 500-chunk heatmap is provided in Appendix § 9.§ 3 showing this behavior persists for the whole stream.

CaLLM Online Learning Dynamics

Figure 4. CaLLM Online Learning Dynamics (TRACE Stream, P=8P=8). (a) OMA. (b) Task ×\times expert assignment heatmap; aggregate adapter timeline on top and per-task routing below. See Appendix § 9.§ 3 for the full 500-chunk view.

Knowledge Accumulation. The OMA curve (Figure 4a) rises steadily from ∼0.29\sim0.29 to ∼0.40\sim0.40 over the 500-chunk stream, confirming that CaLLM-O accumulates knowledge faster than it forgets, even in the absence of explicit task boundaries. After brief initial volatility during the first ∼50\sim50 chunks, where the router is still forming its expert assignments and initialized experts building up knowledge over tasks, the trajectory stabilizes into a monotonic upward trend, indicating that new tasks are absorbed without disrupting previously learned ones.

Expert Specialization. The zoomed heatmap (Figure 4b) reveals consistent task–expert matching within the displayed window. The top strip shows the aggregate adapter timeline with distinct color band showing how frequently the adapter changes through the stream. The per-task rows confirm that each task is reliably routed to the same expert when it reappears in the stream, indicating that the router functions as a distribution-sensitive switch rather than randomly cycling through experts.

The Role of Pool Capacity

The router embedding analysis (§ 4.2) showed that the router is capable of separating task clusters both in diverse and fine-grained task distributions. When pool capacity is unconstrained, each cluster maps to a unique expert, achieving full isolation. But in practice, memory budgets are limited and the number of tasks to be seen is not always known in advance. We now investigate what happens as pool size PP is progressively reduced, revealing a spectrum of routing behaviors from perfect isolation to system collapse.

Expert Assignment Dynamics in Task-Incremental Learning

Figure 5 visualizes the evolution of expert assignments and the corresponding BWT curves on TRACE-8 as pool size varies from P=8P=8 to P=2P=2 (see Appendix § 9.§ 2 for extended plots).

Router behavior under pool capacity constraints for TRACE-8 in Batched scenario

Figure 5. Router behavior under pool capacity constraints for TRACE-8 in Batched scenario.

  • Ideal Isolation (P=8P=8): The router assigns unique experts to each task, resulting in zero BWT loss. This confirms that the router’s embedding clusters (Figure 2(a)) translate directly into correct expert assignment.

  • Constructive Clustering (P=6P=6): Semantically related tasks (e.g., NumGLUE-cm/ds) are co-routed to Expert 5, while other tasks retain dedicated experts. Because these merged tasks share similar gradient directions, parameter sharing is beneficial rather than destructive. The BWT curve remains relatively stable with a slight dip at Step 7 when the merge occurs.

  • Destructive Overload (P=4P=4): A “super-expert” absorbs disparate tasks (QA, Stance, Math), leading to conflicting updates and a sharp BWT drop to −0.19-0.19. Notably, the router still identifies Py150 (code) as maximally distinct, preserving its isolation in a separate expert even under severe constraints.

  • System Collapse (P=2P=2): All tasks merge into one expert and the BWT curve declines to −0.25-0.25 as catastrophic forgetting becomes unavoidable. This represents the lower bound of the capacity spectrum and illustrates the fundamental trade-off between memory efficiency and interference prevention.

Pool capacity constraints in the online scenario

The batched analysis reveals a clear capacity-stability trade-off which we investigate here for the online scenario. We evaluate CaLLM-O over 500-chunk streams with varying pool sizes on TRACE (P∈{8,6,4}P \in\{8,6,4\}) and CITB (P∈{19,10,3}P \in\{19,10,3\}), against SeqFT (P=1P=1), revealing a non-monotonic relation between performance and capacity.

Stability over capacity. On TRACE, P=4P=4 achieves the highest final OMA (0.42) and the best BWT, outperforming both P=8P=8 (0.39) and P=6P=6 (0.38) (see Appendix § 8.§ 1 for full plots). Routing stability—the consistency of task-to-expert assignment—outweighs raw capacity (see Appendix § 9.§ 3.1 Fig. 12). At P=4P=4, the router converges on four semantically coherent groups: (1) C-STANCE; (2) quantitative tasks (NumGLUE-ds, NumGLUE-cm, Py150); (3) FOMC; (4) long-form understanding (ScienceQA, MeetingBank, 20Minuten). Interleaved chunks within each group provide implicit replay via alternating gradient signals, fostering constructive interference. Conversely, P=6P=6 suffers from task displacement: 20Minuten is scattered across multiple experts, preventing specialization. P=8P=8 achieves perfect isolation but forgoes the co-assignment regularization that benefits the streaming regime.

Similarity and forgetting. In more homogeneous streams like CITB-19, performance is less sensitive to capacity; since similar tasks tolerate compression well, though smaller pools still cause deeper forgetting spikes. Task similarity ultimately determines how sensitive a stream is to pool size, diverse streams need larger, more carefully tuned pools.

Conclusion

We presented CaLLM, a modular continual learning method for LLMs that pairs a bounded pool of lightweight adapters with a non-parametric, prototype-based router operating entirely in the frozen backbone’s embedding space, requiring no gradient-based training or task labels at inference. Incoming data are matched to the nearest expert via cosine distance, with new experts spawned or repurposed automatically when the data is sufficiently novel. Our experiments on TRACE-8 and CITB-19 demonstrate that CaLLM is effective across both batched and online CL settings, where it reliably discovers distributional boundaries from input features alone, and translates this geometric separation into accurate, task-agnostic expert assignments. Although CITB-19 shows that the router can separate fine-grained tasks that share a language and format, we do not test whether it can keep tasks together when they overlap conceptually but differ lexically (e.g. the same task across languages), which we leave to future evaluation. Notably, we observe that limiting the adapter pool size can boost performance by forming semantically coherent groups and using interleaved gradients as implicit replay, without requiring memory buffers.

Future directions include integrating online AutoML to adapt currently fixed hyperparameters as the stream unfolds, explicitly merging related adapters to consolidate shared knowledge, and extending prototype-based routing to multi-modal streams spanning vision, language, and other modalities simultaneously.

Appendix

Backbone selection

Table 2 compares zero-shot performance across three candidate backbones – Llama-3.1-8B-Instruct, gemma-3-12b-it, and Qwen3-8B– on the TRACE-8 benchmark. While all models struggle on the NumGLUE (math tasks), confirming the necessity of domain-specific adaptation, gemma-3-12b-it consistently exhibits the strongest generalist prior. It achieves the highest zero-shot accuracy across all eight tasks, with marked advantages on C-STANCE (0.62 vs. 0.20), FOMC (0.64 vs. 0.14), and Py150 (14.91 vs. 6.42). Selecting gemma-3-12b-it as our primary backbone ensures that our results evaluate the efficacy of the CaLLM framework rather than compensating for a weak base-model initialization.

TaskMetricLlama-3.1-8B-Instructgemma-3-12b-itQwen3-8B
C-STANCEacc (0–1)0.200.620.20
FOMCacc (0–1)0.140.640.04
MeetingBankROUGE-L (0–1)0.050.210.04
Py150EDIM (0–100)6.4214.910.00
ScienceQAacc (0–1)0.090.350.07
NumGLUE-cmacc (0–1)0.010.050.00
NumGLUE-dsacc (0–1)0.020.070.00
20MinutenSARI (0–100)36.239.2237.0

Table 2. Inference-only (Zero-Shot) Performance on TRACE-8. gemma-3-12b-it achieves the highest score on all 8 tasks. All models show low performance on math, motivating the need for continual adaptation.

Implementation details and hyperparameters

As detailed in Appendix § 6.§ 1, all experiments use gemma-3-12b-it (4-bit quantized) as the primary backbone. We maintain this fixed model for all comparisons, though CaLLM is backbone-agnostic and compatible with any Transformer-based LLM. The router utilizes the same frozen backbone as a feature extractor, where the last hidden states are mean-pooled over valid tokens to obtain a sample embedding and then averaged across the input batch to produce a single representation for routing.

LoRA adapters are trained using the HuggingFace SFTTrainer with an 8-bit AdamW optimizer, gradient clipping at norm 1.0, and bfloat16 mixed precision without sequence packing. Additional training constants include a maximum sequence length of 1024, temperature of 1.0, LoRA dropout of 0.05, and a warmup ratio of 0.05. In the online setting, we use max_chunks=500\mathit{max\_chunks} = 500 and chunk_size=40\mathit{chunk\_size} = 40, with chunks generated via random resampling. All input follows the Alpaca prompt style detailed in §4.1, and the router uses the adaptive threshold defined in Eq. (4).

Batched experiments are reported as the mean and standard deviation across three random seeds with optimal configurations selected based on final mean validation performance. Online experiments are instead performed as a single run due to the significant computational cost of 500-chunk streams. We rely on the OMA metric, which averages over 500 evaluation steps, to provide inherent statistical stability. Table 3 summarizes the specific hyperparameters selected via grid search over learning rate ∈{10−4,5×10−4,10−3}\in\{10^{-4}, 5 \times10^{-4}, 10^{-3}\}, LoRA rank ∈{4,8,16}\in\{4, 8, 16\}, and epochs ∈{3,5}\in\{3, 5\}. To mitigate the substantial computational overhead of tuning long-horizon streams, hyperparameter optimization for the online setting was restricted to a truncated proxy window of 200 evaluation steps under a standard pre-update (test-than-train) evaluation protocol. All experiments were conducted on a single NVIDIA RTX A6000 GPU with 40GB of VRAM using Unsloth ([9]) and the trl, peft and transformers libraries from HuggingFace.

SettingLRrrα\alphaEpochsBatchGASLR Sched.
CaLLM-B, TRACE1×10−41 \times10^{-4}16323164cosine
CaLLM-B, CITB1×10−31 \times10^{-3}8165254cosine
CaLLM-O, TRACE3×10−33 \times10^{-3}481102linear
CaLLM-O, CITB1×10−31 \times10^{-3}881201cosine

Table 3. CaLLM hyperparameters per scenario. All runs use the HuggingFace SFTTrainer with 8-bit AdamW, gradient clipping at norm 1.0, and bfloat16 mixed precision. LR = learning rate, rr / α\alpha = LoRA rank / scaling factor, Batch = batch size, GAS = gradient accumulation steps, LR Sched. = learning rate schedule.

Details about the baselines

To evaluate CaLLM, we compare it against several representative strategies from the CL literature, adapted for the PEFT regime. To ensure a fair and rigorous comparison, all baselines share the same core architectural parameters as CaLLM and the same LoRA configuration (rank, alpha, and dropout). We then utilize the optimal hyperparameters (such as regularization strength or buffer size) specific to each method as reported in their respective original papers, unless stated otherwise.

To understand the potential impact of this design choice on the fairness of baseline comparisons, we conducted an ablation study of hyperparameter search on selected continual learning baselines, namely ER and EWC. The search follows the CaLLM search space over learning rate and number of training epochs, using a grid search over learning rate ∈{10−4,5×10−4,10−3}\in\{10^{-4}, 5 \times10^{-4}, 10^{-3}\} and epochs ∈{3,5}\in\{3, 5\}. The ablation focuses on the TRACE benchmark, due to its higher variation across tasks and smaller number of tasks. Results of this search are presented in Table 4, including the original hyperparameter values and performance results. As shown in the table, the search selects a hyperparameter combination different from the originally chosen values for each baseline (where the per-task performance does not always improve compared with the original setting). However, the magnitude of the performance difference is small, and may also be explained by the inherent randomness of LLM generations. Importantly, this does not change the ordering of baseline–method comparisons for any task in the analysis, supporting the original design choice.

MethodConfig.LREpochsC-STANCEFOMCMeetingBankPy150ScienceQANumGLUE-cmNumGLUE-ds20Minuten
EROriginal1×10−41 \times10^{-4}30.040.140.1012.980.190.060.1339.63
ERTuned5×10−45 \times10^{-4}50.110.190.1213.040.220.090.1539.58
EWCOriginal1×10−41 \times10^{-4}30.280.310.1010.550.610.120.1540.06
EWCTuned5×10−45 \times10^{-4}30.170.350.0910.610.550.210.1139.01

Table 4. Baseline hyperparameter ablation on TRACE-8 (Batched). Original rows use the configuration from the main comparison; tuned rows are selected from the same learning-rate/epoch grid used for CaLLM while keeping LoRA rank, scaling factor, and dropout fixed.

  • Inference only is a simple baseline where the base model (e.g. gemma-3-12b-it) is used during inference in a continual/online way, every time a new task/chunk is presented to the model.

  • Sequential Adapter Fine-Tuning (Seq-FT): A single LoRA adapter is fine-tuned sequentially on the task stream. This represents the lower bound for stability, as it is highly susceptible to catastrophic forgetting.

  • Elastic Weight Consolidation (EWC) [16]: A regularization-based approach that penalizes changes to parameters that were important for previous tasks. Importance is estimated via the diagonal of the Fisher Information Matrix. We apply EWC to the LoRA weights specifically.

  • Experience Replay (ER) [24]: A memory buffer stores a subset of samples from previous tasks. During the adaptation to a new task, each training batch is augmented with replayed samples from the buffer to preserve past knowledge. In our experiments, we use a fixed buffer size of 1000 samples.

  • Orthogonal LoRA (OLoRA) [31]: A subspace isolation method that constrains the LoRA weight updates to be orthogonal to the pre-trained weights or previous task directions, aiming to prevent gradient-based interference. To ensure compatibility with the Gemma backbone and avoid out-of-memory (OOM) errors, we modified the original OLoRA implementation as follows: (i) reduced the evaluation batch size to 4, instead of using the batch size, and (ii) offloaded the orthogonal loss calculation to the CPU. For the TRACE-8 online stream, we utilized batch_size=8batch\_size = 8, r=4r = 4, α=8\alpha= 8. For CITB, we used batch_size=2batch\_size = 2, r=4r = 4, α=8\alpha= 8, and gradient accumulation step of 8.

  • K-D Tree of Low-Rank Adapters (TreeLoRA) [22]: Introduces layer-wise adapters to construct hierarchical tree based on the criteria of gradient similarity, exploring task relationships via lower-confidence-bound bandits.

gemma-3-12b-it training data and CITB overlap

Gemma 3 models are well-suited for text-generation tasks such as question answering, summarization, and reasoning. The 12B model used in our work was trained on 12T tokens consisting of web documents (in over 140 languages), code, mathematics, and images. The pre-training data is not released openly and there is limited information about the data sources. Additionally, the IT variant has been instruction-tuned using knowledge distillation from larger teacher models (likely Gemini models).

CITB is built on top of the Super-NaturalInstructions dataset Wang et al. [34]. In this work, we focus on the InstrDialog task stream, which consists of 19 dialogue-related tasks spanning three categories: dialogue state tracking (4 tasks), dialogue generation (11 tasks), and intent identification (4 tasks). All tasks are evaluated using the ROUGE-L metric.

While there is no clear information about the data used by Gemma 3 for pre-training or fine-tuning, it is plausible that the Super-NaturalInstructions dataset (on which CITB is based) was included in the training data of either Gemma 3 or the Gemini teacher models. This would explain why LoRA fine-tuning on CITB does not improve over the inference-only baseline. Despite this, all other baselines suffer from sequential fine-tuning, suggesting that CaLLM maintains the original performance even when the data has already been seen.

To investigate potential data contamination, we applied the method of Shi et al. [28], which computes per-token probabilities under the hypothesis that “seen” examples have fewer low-probability outliers with respect to “unseen” words. We compared 500 CITB samples against 309 “old” Wikipedia pages (likely within Gemma 3’s training data) and 97 Wikipedia pages created in the following 60 days (unlikely to be in the training data). The results are as follows:

  • Member mean: 0.005722

  • Non-member mean: 0.003475

  • CITB mean: 0.002848

From these results we can conclude that CITB has an average probability closer to the non-member samples than the (assumed) member samples, suggesting that CITB is likely not in the training data of Gemma 3.

This appears to contradict our earlier hypothesis that Gemma 3 has already seen CITB data. Indeed, Figure 3b shows that fine-tuning with CITB data does not improve over the inference-only baseline. However, it is important to note that the method of Shi et al. [28] is only an empirical heuristic, and the candidate member and non-member sets are based on assumptions about Wikipedia page creation dates rather than ground-truth training data membership.

Synthetic data generation for CITB

The original CITB benchmark, defined as InstrDialog in the original paper Zhang et al. [39], consists of 19 tasks derived from Super-NaturalInstructions, detailed in Table 5.

CategoryTask TypeCount
Dialogue State TrackingSlot Filling, State Update4
Dialogue GenerationResponse Gen, Clarification11
Intent IdentificationClassification, Intent Detection4
Total19

Table 5. The 19 dialogue tasks in the CITB InstrDialog stream.

Each task contains only 500 training instances, which is insufficient for effectively training LoRA adapters and creates an imbalance when compared to the TRACE-8 benchmark (5000 instances per task). To ensure a fair comparison and improve adaptation stability, we generated synthetic training data for each CITB task. Inspired by self-synthesized rehearsal strategies Huang et al. [13], we utilized the frozen gemma-3-12b-it model as a data generator. We employed a few-shot prompting strategy, providing the model with two randomly sampled real instances in-context and tasking it to generate a novel example following the same Alpaca-style format. By iterating this process with randomized in-context examples, we produced an initial pool of 6000 synthetic instances per task. To ensure data quality and distributional consistency, we applied a two-stage refinement pipeline:

  1. Deduplication: We removed all exact string matches and near-duplicates to ensure diversity within the synthetic pool.

  2. Centroid-based filtering: To retain only high-quality, representative samples, we converted the synthetic instances into embeddings using the frozen base model. We then applied K-means clustering to the embeddings of each task. Finally, we calculated the Euclidean distance between each instance and its respective cluster centroid, selecting the 3000 instances closest to the centroid for the final training set.

This procedure resulted in a refined CITB dataset containing 3000 examples per task. This refined version maintains the semantic integrity of the original tasks while providing sufficient signal for parameter-efficient fine-tuning, and it serves as the standard CITB data stream used for all experiments and baseline comparisons presented in this paper.

Automatic Metric Selection

When ground-truth metric labels are unavailable (i.e., in label-free stream configurations), CaLLM dispatches an automatic metric selection procedure at the start of each task’s evaluation phase. This adds only a negligible time overhead from a single short inference call and no memory overhead, as it reuses the already-loaded backbone. The procedure works as follows.

Model. The same base LLM used for task inference (gemma-3-12b-it, 4-bit quantized) serves as the zero-shot classifier for metric selection. This avoid the overhead of loading an auxiliary model and the system simply executes a single generation call before the model training and evaluation on each task.

Meta-prompt template. A single representative prompt is sampled from the task’s test split, truncated to 700 characters to manage context, and embedded into a structured meta-prompt. The prompt instructs the LLM to categorize the task based on its linguistic and structural features (e.g., presence of code-specific keywords, multiple-choice indicators, or summarization requests). The model is constrained to a closed set of responses: ACCURACY, ROUGE, SARI, or EDIM.

Analyze the following task prompt and select the best evaluation metric.
RULES:
1.  Code completion tasks (contains ‘code’, ‘python’, ‘complete the code’,
etc.)  → Use EDIM
2.  Summarization tasks (contains ‘summarize’, ‘summary’, ‘write a
summary’, etc.)  → Use ROUGE
3.  Multiple choice questions (contains ‘A)’, ‘B)’, ‘C)’, ‘choose’, etc.)
→ Use ACCURACY
4.  Text simplification tasks (contains ‘simplify’, ‘simplified version’,
etc.)  → Use SARI
5.  Default for all other tasks → Use ACCURACY
Task prompt: {prompt}
Respond with ONLY the metric name.  Choose one:  ACCURACY, ROUGE, SARI,
or EDIM
### Response:

Parsing and fallback. To ensure robustness, the model’s output is parsed for the first occurrence of a valid metric keyword. If the model fails to produce a recognized metric name within its response, the system defaults to ACCURACY as a conservative baseline. Once a metric is selected, it is cached for the duration of the task’s residency in the stream to prevent redundant computation.

Algorithms and Efficiency

Algorithms (pseudocode)

Algorithm 1: CaLLM-Batched

Input: Task sequence D1,…,DT\mathcal{D}_1,\ldots,\mathcal{D}_T; threshold θ\theta (fixed or adaptive, Eq. (4)); pool capacity PP

Output: Performance table M\mathbf{M}

Initialize pool P=∅\mathcal{P} = \varnothing

for t=1t = 1 to TT do

// Route (training mode: prototype updates enabled)

e←EMBED⁡(sample⁡(Dttrain))\mathbf{e} \leftarrow\operatorname{EMBED}(\operatorname{sample}(\mathcal{D}^{\mathrm{train}}_t))

i∗,d∗←arg min⁡i:ℓi∈P dii^*, d^* \leftarrow\underset{i:\ell_i\in\mathcal{P}}{\operatorname{arg\,min}}\ d_i (Eq. (3); d∗=∞d^* = \infty if P=∅\mathcal{P} = \varnothing)

if d∗≤θd^* \leq\theta then

at←ℓi∗a_t \leftarrow\ell_{i^*}; pi∗←12(pi∗+e)\mathbf{p}_{i^*} \leftarrow\frac{1}{2}(\mathbf{p}_{i^*} + \mathbf{e}); ci∗←ci∗+1c_{i^*} \leftarrow c_{i^*} + 1 // reuse

else if ∣P∣<P|\mathcal{P}| < P then

at←NEWADAPTER⁡(); P←P∪{(at,e,1)}a_t \leftarrow\operatorname{NEWADAPTER}();\ \mathcal{P} \leftarrow\mathcal{P} \cup\{(a_t,\mathbf{e},1)\} // create

else

imin⁡←arg min⁡i cii_{\min} \leftarrow\underset{i}{\operatorname{arg\,min}}\ c_i; at←ℓimin⁡a_t \leftarrow\ell_{i_{\min}}; pimin⁡←e\mathbf{p}_{i_{\min}} \leftarrow\mathbf{e}; cimin⁡←1c_{i_{\min}} \leftarrow1 // repurpose

end if

// Train (overwrites adapter weights in the repurposing case)

SFT⁡(Dttrain,Dteval,adapter⁡=at)\operatorname{SFT}(\mathcal{D}^{\mathrm{train}}_t,\mathcal{D}^{\mathrm{eval}}_t,\operatorname{adapter}=a_t)

// Evaluate current task

LOADADAPTER⁡(at); M[t,t]←EVALUATE⁡(Dttest)\operatorname{LOADADAPTER}(a_t);\ \mathbf{M}[t,t] \leftarrow\operatorname{EVALUATE}(\mathcal{D}^{\mathrm{test}}_t)

// Backward transfer on all previous tasks (read-only routing)

if t>1t > 1 then

for j=1j = 1 to t−1t - 1 do

ej←EMBED⁡(sample⁡(Djtest))\mathbf{e}_j \leftarrow\operatorname{EMBED}(\operatorname{sample}(\mathcal{D}^{\mathrm{test}}_j))

aj←ℓarg min⁡i di(ej)a_j \leftarrow\ell_{\underset{i}{\operatorname{arg\,min}}\ d_i(\mathbf{e}_j)} // read-only: no prototype updates

LOADADAPTER⁡(aj); M[j,t]←EVALUATE⁡(Djtest)\operatorname{LOADADAPTER}(a_j);\ \mathbf{M}[j,t] \leftarrow\operatorname{EVALUATE}(\mathcal{D}^{\mathrm{test}}_j)

end for

M[BWT,t]←1t−1∑j=1t−1(M[j,t]−M[j,j])/sj\mathbf{M}[\mathrm{BWT},t] \leftarrow\frac{1}{t-1}\sum_{j=1}^{t-1}\left(\mathbf{M}[j,t]-\mathbf{M}[j,j]\right)/s_j

else M[BWT,t]←0\mathbf{M}[\mathrm{BWT},t] \leftarrow0

end if

M[Avg,t]←1t∑i=1tM[i,t]/si\mathbf{M}[\mathrm{Avg},t] \leftarrow\frac{1}{t}\sum_{i=1}^{t}\mathbf{M}[i,t]/s_i

end for

Algorithm 2: CaLLM-Online

Input: Chunk stream (τk,Dk)(\tau_k,\mathcal{D}_k) for k=1,…,Nk = 1,\ldots,N; threshold θ\theta (fixed or adaptive); pool capacity PP

Output: Performance table M\mathbf{M} with per-step Forgetting, BWT, and OMA

Initialize: pool P=∅\mathcal{P} = \varnothing, last ={}= \{\}, Srun=0S_{\mathrm{run}} = 0, c=0c = 0

for each step k=1,…,Nk = 1,\ldots,N, receiving (τ,D)(\tau,\mathcal{D}) do

// Route (single pass: prototype updates enabled)

e←EMBED⁡(sample⁡(D))\mathbf{e} \leftarrow\operatorname{EMBED}(\operatorname{sample}(\mathcal{D}))

i∗,d∗←arg min⁡i:ℓi∈P dii^*, d^* \leftarrow\underset{i:\ell_i\in\mathcal{P}}{\operatorname{arg\,min}}\ d_i (d∗=∞d^* = \infty if P=∅\mathcal{P} = \varnothing)

if d∗≤θd^* \leq\theta then

a←ℓi∗a \leftarrow\ell_{i^*}; pi∗←12(pi∗+e)\mathbf{p}_{i^*} \leftarrow\frac{1}{2}(\mathbf{p}_{i^*} + \mathbf{e}); ci∗←ci∗+1c_{i^*} \leftarrow c_{i^*} + 1 // reuse

else if ∣P∣<P|\mathcal{P}| < P then

a←NEWADAPTER⁡(); P←P∪{(a,e,1)}a \leftarrow\operatorname{NEWADAPTER}();\ \mathcal{P} \leftarrow\mathcal{P} \cup\{(a,\mathbf{e},1)\} // create

else

imin⁡←arg min⁡i cii_{\min} \leftarrow\underset{i}{\operatorname{arg\,min}}\ c_i; a←ℓimin⁡a \leftarrow\ell_{i_{\min}}; pimin⁡←e\mathbf{p}_{i_{\min}} \leftarrow\mathbf{e}; cimin⁡←1c_{i_{\min}} \leftarrow1 // repurpose

end if

// Pre-update evaluation (test-then-train)

LOADADAPTER⁡(a); pre⁡←EVALUATE⁡(D); M[τ(pre⁡),k]←pre⁡\operatorname{LOADADAPTER}(a);\ \operatorname{pre} \leftarrow\operatorname{EVALUATE}(\mathcal{D});\ \mathbf{M}[\tau(\operatorname{pre}),k] \leftarrow\operatorname{pre}

// BWT (defined only when task has a prior observation)
if τ ∈ last then
M[BWT, k] ← (pre − last[τ]) / sτ
end if
last[τ] ← pre
Srun ← Srun + pre / sτ ; c ← c + 1; M[OMA, k] ← Srun / c
// Train on same adapter, same chunk
SFT(D, adapter=a)
end for

Computational and Memory Efficiency

A practical CL system must not only preserve performance but also remain efficient as the expert pool grows. Table 6 reports training FLOPs, throughput, and peak GPU memory for CaLLM at varying pool sizes alongside baselines, measured on the full TRACE-8 batched pipeline (gemma-3-12b-it, single NVIDIA RTX A6000 48 GB GPU, CUDA 12.4).

MethodPool PPTrain TFLOPsSamples/sPeak GPU (GB)Wall (h)# Adapters
CaLLM-B27,3393.8112.1560.02
CaLLM-B47,3393.8112.1559.34
CaLLM-B67,3393.6712.1559.76
CaLLM-B87,3393.8712.1559.58
Seq-FT17,3393.5312.1568.21
EWC†1——12.4876.91
ER†1——12.10125.81
oLoRA†8——25.2075.18

Table 6. Compute and Memory Overhead on TRACE-8 (Batched). CaLLM consumes the same training FLOPs as single-adapter Seq-FT regardless of pool size, with constant GPU memory. ER doubles wall-clock time due to replay, and oLoRA requires 2.1× more GPU memory for orthogonal projections. †EWC, ER, and oLoRA use custom training loops that do not report total flos; EWC’s FLOPs are comparable to CaLLM (same data/rank), ER’s are higher due to replay augmentation, and oLoRA’s are lower due to smaller rank (r=4r=4).

Training compute. The central result is that CaLLM’s training cost is independent of pool size: all four configurations (P=2,4,6,8P = 2, 4, 6, 8) consume exactly 7,339 TFLOPs. This is a direct consequence of the modular architecture—only one adapter is active per forward pass, so the computational graph is identical regardless of how many adapters exist in the pool. Training throughput is also stable across pool sizes (3.67–3.87 samples/s), confirming that adapter switching introduces negligible latency.

CaLLM and Seq-FT share identical training FLOPs (7,339 TFLOPs), as expected—both perform the same number of gradient updates on the same data with the same adapter rank. The difference lies entirely in where gradients are routed, not in how many are computed. CaLLM achieves slightly higher throughput (3.81–3.87 vs. 3.53 samples/s) and lower wall-clock time (59–60 h vs. 68 h), likely because isolated adapters encounter simpler optimization landscapes than a shared adapter accumulating all tasks.

Routing overhead. Unlike the baselines, CaLLM performs an additional forward pass through the frozen backbone for prototype matching (§ 3.2). Because this routing pass is encoder-only, i.e., no decoding, over a short input and reuses the already-loaded backbone, its marginal cost is small relative to the per-task training and inference compute. Using the standard transformer estimate of approximately 2N2N FLOPs per forward token, where N≈12×109N \approx12 \times10^{9} for gemma-3-12b-it, and noting that each routing call processes n≈20n \approx20 prompts of approximately 10 tokens each, one routing call costs 2⋅(12×109)⋅20⋅10≈4.8×10122 \cdot(12 \times10^{9}) \cdot20 \cdot10 \approx4.8 \times10^{12} FLOPs, i.e., 4.8 TFLOPs. Over the full TRACE-8 batched run, with one routing call per task across 8 tasks, the router consumes 8⋅4.8≈38.48 \cdot4.8 \approx38.4 TFLOPs, or roughly (38.4/7339)×100≈0.5%(38.4/7339) \times100 \approx0.5\% of the 7339 TFLOPs reported above for training. This is consistent with the observation that CaLLM and Seq-FT report identical training TFLOPs in Table 6: the router’s contribution falls below the reporting precision. The same logic applies at inference time, where each routing call is amortized over all subsequent evaluation queries within a task, giving per-query overhead ≪0.1%\ll0.1\%, and during the online continual-learning phase, where routing is invoked once per chunk rather than per query. The routing module is, moreover, a tunable component: a smaller encoder or cached prompt embeddings could reduce this already-small overhead further, which we leave to future work.

Baseline comparison. Among the baselines that explicitly address forgetting, each pays a distinct overhead. EWC trains on the same data with the same adapter rank, so its per-step FLOPs are expected to be comparable to CaLLM; the additional wall-clock cost (76.9 h) comes from Fisher information computation after each task. ER augments each training batch with replay buffer samples, which increases the effective data budget and thus is expected to incur higher total training FLOPs—this is reflected in its 2.1× wall-clock overhead (125.8 h). oLoRA uses a smaller rank (r=4r=4 vs. r=16r=16), which should reduce per-step FLOPs, but must maintain orthogonal projection matrices across all subspaces simultaneously, making it the most memory-intensive baseline at 25.2 GB (2.1× CaLLM).

These results confirm that CaLLM achieves its anti-forgetting benefits at zero additional training compute relative to naïve sequential fine-tuning, while the baselines that attempt to mitigate forgetting through regularization, replay, or orthogonality constraints each pay a substantial cost in time, memory, or both.

Memory management. CaLLM maintains a constant 12.15 GB GPU memory footprint regardless of pool size, because exactly one adapter is loaded on GPU at any time. Inactive adapters are offloaded to CPU memory and swapped in on demand, so the GPU overhead does not grow with the number of adapters in the pool. The maximum pool capacity Pmax⁡P_{\max} can be set explicitly or computed automatically based on available system memory according to Eq. (6):

P=⌊MavailSadapter⌋−1,(6)P = \left\lfloor\frac{M_{\mathrm{avail}}}{S_{\mathrm{adapter}}} \right\rfloor- 1, \tag*{(6)}

where MavailM_{\mathrm{avail}} is the available system (CPU) memory and SadapterS_{\mathrm{adapter}} is the on-disk size of a single adapter (both in GB); the −1 reserves one slot for the adapter currently resident on GPU. In the online regime, peak GPU memory remains flat at 12.01 GB—virtually identical to the batched setting—confirming that this constant-memory property holds across both training protocols.

Online regime. The online streaming configuration uses lighter adapters (r=4r=4, α=8\alpha=8) and single-epoch chunk updates, so its total FLOPs are not directly comparable to the batched runs above. A notable characteristic of the online setting is that evaluation dominates the runtime: out of 19.3 h total wall-clock time over 500 chunks, only 3.6 h (18.4%) is spent on training while 13.9 h (72.3%) is consumed by the per-chunk test-then-train evaluation loop. This suggests that evaluation scheduling—rather than training efficiency—is the primary lever for reducing online deployment cost, and motivates future work on sparse or amortized evaluation strategies.

Extended Quantitative Results

Online Pool Constraints: OMA and BWT Curves

Online Pool Constraints: TRACE OMA, TRACE BWT, CITB OMA, and CITB BWT

Figure 6. Online Pool Constraints (500-chunk stream). OMA and BWT for CaLLM-O and the SeqFT baseline. Top row (TRACE-8): (a) OMA and (b) BWT for pool sizes P∈{4,6,8}P \in\{4, 6, 8\}. Bottom row (CITB-19): (c) OMA and (d) BWT for pool sizes P∈{3,10,19}P \in\{3, 10, 19\}. Knowledge accumulation (OMA) is successful across all configurations; performance is non-monotonic with respect to pool size on TRACE, while CITB shows comparable cumulative scores but diverging forgetting profiles.

Extended Batched Performance

Per-Task Batched Performance

Table 7 and Table 8 show the per-task performance on TRACE-8 and CITB-19 for the CaLLM-B baseline as each task is sequentially presented to the model, capturing the immediate performance right after all training data for that specific task has been processed. These tables are complementary to the results illustrated in Figure 3, demonstrating that performance on individual tasks is steadily maintained as the stream length increases. This stability is further supported by Figure 7, which shows that CaLLM consistently achieves a BWT score of nearly zero, confirming minimal catastrophic forgetting across the streaming timeline.

CITB Extended Batched Performance

Table 9 disaggregates the final CITB-19 results by individual task, analogous to Table 1 for TRACE-8. All 19 tasks use ROUGE-L (0–1) as the evaluation metric. Scores are averaged across 3 seeds for all methods except EWC (2 seeds available). CaLLM-B achieves the highest score among all continual learning methods on 13 of 19 tasks. On the remaining 6, oLoRA leads on individual tasks but trails CaLLM-B in mean performance (0.24 vs. 0.31), indicating that its per-task wins come at the cost of broader forgetting elsewhere. The high Inference-Only scores on tasks such

as Diplomacy (T1590), P.Chat Gen. (T1729), and Mutual Dial. (T611) reflect strong backbone priors that sequential fine-tuning erodes for all methods.

Inference-Only baseline: prompt-template sensitivity

To test the robustness of the performance to the prompt template, we repeated the Inference-Only baseline on both TRACE-8 and CITB-19 using Gemma 3’s native chat format (<start_of_turn>user ...<end_of_turn><start_of_turn>model), keeping every other hyperparameter identical to the Alpaca-format run reported in the main paper. We present the per-task comparison in Figure 8. On TRACE-8, the Alpaca template is strictly stronger or equal on every task (mean normalized score 0.31 vs. 0.21), with the largest drops on structured-output tasks (Py150, ScienceQA, NumGLUE-cm/ds). On CITB-19 the two templates are tied in aggregate (mean ROUGE-L 0.34 vs. 0.35) but redistribute mass between generation-style tasks (which lose 0.4–0.8), and classification-/choice-style tasks, which gain a comparable amount. We therefore retain the Alpaca-format Inf-Only as the headline baseline as it is the more conservative comparison on TRACE-8 and equivalent on CITB-19.

Two-panel line plots comparing Alpaca and Gemma 3 chat templates for TRACE-8 and CITB-19

Figure 8. Prompt-template sensitivity of the Inference-Only baseline. Per-task final performance under the Alpaca template (blue, used in the main paper) and Gemma 3’s native chat template (red). (a) TRACE-8 (Batched): normalized scores (EDIM, SARI divided by 100); Alpaca dominates on every task (mean 0.31 vs. 0.21). (b) CITB-19 (Batched): ROUGE-L; the two templates are tied in aggregate (mean 0.34 vs. 0.35) but redistribute mass across generation- and classification-style tasks. Tasks are shown in original training/streaming order.

Extended Router Analysis

This section provides a deep dive into the router’s internal logic, visualizing how it represents tasks and how capacity constraints (PP) dictate expert assignment.

Task Separability in the Embedding Space

Figure 9 illustrates the inherent separability of the task distributions within the router’s embedding space. Using t-SNE projections of the frozen backbone’s mean-pooled features, we observe that diverse tasks (both for TRACE-8 and CITB-19) form well-defined, distant clusters.

Prototype Dynamics and Capacity Progression

Figure 10 demonstrate how the router dynamically partitions the embedding space under varying capacity constrains, studied in §4.5.1, for TRACE-8. Each panel overlays PP prototypes (yellow stars) on the task-cluster structure; grey lines connect each data point to its nearest prototype. The prototype positions determine which tasks share an expert during training.

Router embedding space with prototypes at P=8

(a) P=8P=8: perfect isolation, one prototype per task cluster.

Router embedding space with prototypes at P=6

(b) P=6P=6: NumGLUE-cm/ds share a prototype—constructive merge, negligible BWT.

Router embedding space with prototypes at P=4

(c) P=4P=4: one prototype absorbs 5 diverse tasks—destructive overload, BWT = −0.19-0.19.

Router embedding space with prototypes at P=2

(d) P=2P=2: system collapse, 7 of 8 tasks share one prototype, BWT = −0.25-0.25.

Figure 10. Router Embedding Space with Prototypes (TRACE-8, t-SNE). Yellow stars are prototype positions; grey lines show nearest-prototype assignment. The transition from P=8P=8 to P=2P=2 maps directly onto the isolation → constructive clustering → destructive overload → collapse progression described in § 4.5.1.

At P=8P=8 (Figure 10a), every prototype settles at the centroid of one task cluster, giving zero-interference isolation. At P=6P=6 (Figure 10b), the two NumGLUE math tasks and the C-STANCE dataset are pulled toward a shared prototype, a constructive merge that incurs negligible BWT. At P=4P=4 (Figure 10c), one prototype absorbs five semantically diverse tasks on the right side of the space, producing the conflicting gradients that drive destructive overload. Even here, Py150 (code) retains its own isolated prototype (in red), confirming the router’s sensitivity to the most extreme distributional gap. At P=2P=2 (Figure 10d), only 20Minuten keeps a dedicated prototype; all other tasks collapse onto a single expert.

Online Streaming Heatmap

While the previous plots show the router assignment in the Batched scenario, Figure 11 shows the complete 500-chunk task×\timesexpert assignment heatmap for the TRACE online stream. Each row corresponds to a task, each column to a chunk, and colour encodes the expert ID assigned by the router. The “null” entries (dark blue) indicate chunks in which the corresponding task did not appear in the interleaved stream. Despite the sparsity, the per-task colour patterns are stable throughout the full stream, confirming the persistent task—expert binding reported in § 4.4. The zoomed view (chunks 200–300) used in the main text (Figure 4(b)) was selected as a representative window that minimises null-cell density and maximises readability.

Full Expert-Assignment Heatmap

Figure 11. Full Expert-Assignment Heatmap (TRACE, 500 chunks, P=8P=8). Each row is a task; each column a chunk; colour encodes the assigned expert ID. Dark blue (“null”) cells indicate chunks where that task was absent from the stream. The per-task colour patterns remain stable across the full 500-chunk stream, confirming persistent task–expert binding.

Aggregate Online Routing Matrices

The temporal stability observed in the full stream is summarized numerically in Figure 12. These matrices for TRACE (P∈{4,6,8}P \in\{4,6,8\}) show the fraction of chunks routed to each expert over a 500-chunk stream. As analyzed in § 4.5.2, these matrices reveal that at P=4P=4, the router settles into highly stable, semantically aligned groups that outperform the P=6P=6 configuration due to reduced task displacement.

Router Assignment Matrices on TRACE (Online)

Figure 12. Router Assignment Matrices on TRACE (Online). Pool sizes P∈{4,6,8}P \in\{4,6,8\}. Each cell shows the fraction of chunks from that task routed to each expert.

References

  1. [1]Zipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribution shifts: An empirical study with visual data. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8281–8290, 2021.arxiv.org/abs/2108.09020
  2. [2]Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non-stationary data streams. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8250–8259, 2021.arxiv.org/abs/2009.00919
  3. [3]Matthias De Lange, Gido M. van de Ven, and Tinne Tuytelaars. Continual evaluation for lifelong learning: Identifying the stability gap. In International Conference on Learning Representations (ICLR), 2023.
  4. [4]Nishanth Dikkala, Nikhil Ghosh, Raghu Meka, Rina Panigrahy, Nikhil Vyas, and Xin Wang. On the benefits of learning to route in mixture-of-experts models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9376–9396, 2023.DOI
  5. [5]Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1932–1945, Bangkok, Thailand, August 2024. Association for Computational Linguistics.DOI
  6. [6]Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
  7. [7]William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–40, 2022.arxiv.org/abs/2101.03961
  8. [8]Wenfeng Feng, Chuzhan Hao, Yewei Zhang, Yu Han, and Hao Wang. Mixture-of-LoRAs: An efficient multitask tuning for large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), pp. 11371–11380, 2024.
  9. [9]Daniel Han, Michael Han, and Unsloth team. Unsloth. https://github.com/unslothai/unsloth, 2023.
  10. [10]Jinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao, Ming Tang, and Jinqiao Wang. SEEKR: Selective attention-guided knowledge retention for continual learning of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3254–3266, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
  11. [11]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attarian, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.arxiv.org/abs/1902.00751
  12. [12]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.arxiv.org/abs/2106.09685
  13. [13]Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. arXiv preprint arXiv:2403.01244, 2024.arxiv.org/abs/2403.01244
  14. [14]Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.arxiv.org/abs/2401.04088
  15. [15]Haeyong Kang, Jaehong Yoon, Sultan Rizky Madjid, Sung Ju Hwang, and Chang D. Yoo. Forget-free continual learning with soft-winning subnetworks. arXiv preprint arXiv:2303.14962, 2023.arxiv.org/abs/2303.14962
  16. [16]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.DOI
  17. [17]Yan-Shuo Liang and Wu-Jun Li. InfLoRA: Interference-free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23638–23647, 2024.arxiv.org/abs/2404.00228
  18. [18]Yan-Shuo Liang and Wu-Jun Li. Gated integration of low-rank adaptation for continual learning of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2505.15424
  19. [19]Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Phang, Lingling Tian, Colin Raffel, and Mohit Bansal. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2205.05638
  20. [20]David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  21. [21]Xinbei Ma, Zhuo Tang, Zhuowei Chen, Cheng Peng, and Lei Clifton. Spurious forgetting in continual learning of language models. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2501.13453
  22. [22]Yu-Yang Qian, Yuan-Ze Xu, Zhen-Yu Zhang, Peng Zhao, and Zhi-Hua Zhou. TreeLoRA: Efficient continual learning via layer-wise LoRAs guided by a hierarchical gradient-similarity tree. arXiv preprint arXiv:2506.10355, 2025.arxiv.org/abs/2506.10355
  23. [23]Fuli Qiao, Guizhong Liu, et al. Learn more, but bother less: Parameter efficient continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.DOI
  24. [24]David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
  25. [25]Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.arxiv.org/abs/1606.04671
  26. [26]Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.arxiv.org/abs/2205.12393
  27. [27]Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. Continual learning of large language models: A comprehensive survey. ACM Computing Surveys, 2025.arxiv.org/abs/2404.16789
  28. [28]Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023.arxiv.org/abs/2310.16789
  29. [29]Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4077–4087, 2017.arxiv.org/abs/1703.05175
  30. [30]Anna Vettoruzzo, Mohamed-Rafik Bouguelia, Joaquin Vanschoren, Thorstein Rögnvaldsson, and K. C. Santosh. Advances and challenges in meta-learning: A technical review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(7):4763–4779, 2024.DOI
  31. [31]Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10658–10671, Singapore, December 2023a. Association for Computational Linguistics.arxiv.org/abs/2310.14152
  32. [32]Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, and Xuanjing Huang. TRACE: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.06762, 2023b.arxiv.org/abs/2310.06762
  33. [33]Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. InsCL: A data-efficient continual learning paradigm for fine-tuning large language models with instructions. arXiv preprint arXiv:2403.14555, 2024.
  34. [34]Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamalakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravneesh Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannanneh Hajishirzi, and Daniel Khashabi. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. arXiv preprint arXiv:2204.07705, 2022.arxiv.org/abs/2204.07705
  35. [35]Zhilin Wang, Yafu Li, Xiaoye Ou, and Yu Cheng. SEE: Continual fine-tuning with sequential ensemble of experts. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 7418–7432. Association for Computational Linguistics, 2025.arxiv.org/abs/2504.06664
  36. [36]Xinbo Wu, Max Hartman, Vidhata Arjun Jayaraman, and Lav R. Varshney. SwitchCIT: Switching for continual instruction tuning of large language models. arXiv preprint arXiv:2407.11780, 2024.arxiv.org/abs/2407.11780
  37. [37]Yanru Wu, Jianning Wang, Xiangyu Chen, Enming Zhang, Hanbing Liu, Yang Tan, and Yang Li. Exploiting task relationships in continual learning via transferability-aware task embeddings. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2502.11609
  38. [38]Shu Yang, Muhammad Asif Ali, Cheng-Long Wang, Lijie Hu, and Di Wang. MoRAL: MoE augmented LoRA for LLMs’ lifelong learning. arXiv preprint arXiv:2402.11260, 2024.arxiv.org/abs/2402.11260
  39. [39]Zihan Zhang, Meng Fang, Ling Chen, and Mohammad-Reza Namazi-Rad. CITB: A benchmark for continual instruction tuning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9443–9455. Singapore, December 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.findings-emnlp.633/.arxiv.org/abs/2310.14510
  40. [40]Hongbo Zhao, Fei Zhu, Rundong Wang, Gaofeng Meng, and Zhaoxiang Zhang. MLLM-CL: Continual learning for multimodal large language models. arXiv preprint arXiv:2506.05453, 2025.arxiv.org/abs/2506.05453
  41. [41]Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zheng-Jun Zha. Self-promoted prototype refinement for few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6801–6810, 2021.arxiv.org/abs/2107.08918

Paper details

vlnc:W_5ca8qvad
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents