Generalizing Multimodal Foundational Models in Context-Driven Class Incremental Learning

Introduction

Class-Incremental Learning (CIL), the task of sequentially learning new classes without succumbing to the catastrophic forgetting of previously learned ones [88], has gained significant interest due to its diverse applications in real-world scenarios, including smart devices, autonomous driving, etc [43, 77, 78, 79]. Existing literature has primarily focused on strategies such as knowledge distillation [10, 18, 19], parameter-efficient adapters [9, 27, 60, 71], and exemplar-based memory replay [11, 16, 59]. While these approaches have yielded significant advancements in unimodal settings, their efficacy in the context of Multimodal Foundational Models (MFMs) remains largely underexplored. Leveraging cross-modal interactions, particularly the temporal alignment between audio and visual modalities, has been shown to drastically enhance the robustness of latent representations [6, 51]. However, incrementally leveraging these dense, pre-trained multimodal spaces without disrupting the inter-modal alignment poses a unique and formidable challenge [44, 72].

Despite the success of MFMs [1, 2, 41, 61] in low-level vision tasks [76], virtual assistants [42], autonomous driving [25], and medical imaging [5], MFM-based CIL still faces a stability-plasticity dilemma, where the model struggles to balance between learning and integrating new information (plasticity) while retaining previously acquired knowledge (stability) [21]. Nevertheless, few recent studies [8, 36, 73, 80, 86] have explored the use of foundational multimodality, focusing exclusively on Vision-Language foundational models in the class-incremental settings. In contrast, other works [28, 56, 58] incorporate audio-visual inputs through independent feature extractors, and often yield unreliable representations.

A natural question arises: are MFMs even suitable for incremental learning? As these models are pretrained on large datasets, they may already possess information relevant to both base and incremental classes. We argue that this apparent tension dissolves once representation priors are distinguished from task knowledge. A frozen encoder supplies generic features, but not a classifier over a specific label set. Catastrophic forgetting arises in the trainable components, whose decision boundaries drift as new classes are introduced. To validate this, we conduct a preliminary study, visualized in Figure 1, on the AVE-CI dataset across four incremental tasks. We visualize image-specific audio and visual features and observe that, with a standard audio-visual feature extractor (🟧), both modalities overlap and become cluttered as the task progresses. This overlap prevents the model from capturing fine-grained semantic cues, as reflected in Figure 1, through a significant drop in accuracy and high forgetting as it progresses across tasks. Conversely, a multimodal foundational feature extractor (🟦) yields significantly clearer class clusters. Yet, stronger features are not sufficient; a sequential linear setting (🟪) on the frozen MFM features, lacking any guidance or distillation, still degrades sharply across tasks (Figure 1, § 4.2.2). This confirms that strong priors alone do not solve incremental recognition. This situates our work within the established paradigm of pre-trained-model-based CIL, where the central question is not whether to leverage frozen foundations, but how to adapt them incrementally without disrupting their cross-modal alignment. This dual observation motivates us to leverage MFMs in CIL and to actively organize them for incremental learning rather than naively concatenating them.

Hence, we introduce a unified framework that generalizes multimodal foundational models for CIL without requiring full model fine-tuning. We leverage the frozen representations of state-of-the-art MFMs, specifically SAM-Audio [63], ImageBind [23], and Qwen2.5-Omni-3B [75], to extract robust, high-dimensional audio and visual features. To strategically bridge these modalities, we introduce Attention-based Contextual Guidance that uses extracted audio representations as dynamic context to guide visual features within a transformer-based attention block. Unlike prior audio-guided attention, which targets static single-task settings [35], the incremental setting requires representations to remain stable as new classes arrive; we therefore use audio as task-stable guidance. This results in sample-specific

multimodal attended features that capture the fine-grained correlations needed to distinguish novel classes. To mitigate catastrophic forgetting, we further implement the distillation loss (at both the feature and logit levels), constraining the current task’s attended features to remain consistent with those of the previous task and preserving the structural integrity of the shared latent space. Finally, a lightweight classifier operates on these stable, integrated representations.

We summarize our key contributions as: ❶ We generalize frozen multimodal foundational models (MFMs), specifically SAM-Audio, ImageBind, and Qwen2.5-Omni-3B, to audio-visual class-incremental learning, and we establish that strong MFM priors do not by themselves solve incremental recognition, motivating a targeted adaptation rather than naive feature reuse; ❷ We adapt cross-modal attention, using audio as a task-stable contextual guidance for visual representations in the incremental setting, and show this design generalizes across three distinct MFMs; ❸ We employ feature- and logit-level distillation to balance the current task’s representations with the previous task’s state, and further reduce forgetting; ❹ We conduct extensive experiments on three benchmark datasets, demonstrating consistent improvement over both unimodal and multimodal CIL baselines.

Related Works

Class-Incremental Learning (CIL)

Class-Incremental Learning (CIL) aims to extend a model’s recognition capabilities by continuously learning on new, unknown classes without forgetting the previously learned ones [29, 37, 38, 69]. Existing works primarily address the stability-plasticity dilemma in adapting to new classes while mitigating catastrophic forgetting. Parameter regularization-based methods [3, 7, 12] penalize weights of critical parameters to indicate their importance. Knowledge distillation-based approaches [10, 18, 19] preserve the trade-offs by maintaining consistency between current and previous model states and minimizing the KL divergence between their respective output distributions. Exemplar-based (or Memory Replay) methods [4, 11, 13, 14, 16, 43] store a subset of historical exemplar samples to periodically re-train the model, and ensure the decision boundaries remain intact. Lastly, architecture-based methods [24, 33, 45, 77] dynamically expand the network’s capacity or isolate the task-specific parameters to mitigate inter-task interference.

Multimodal Foundational Models

Recent advancements in multimodal foundation models [1, 2, 61] have demonstrated remarkable breakthroughs across a spectrum of downstream tasks, including dense prediction [74, 76, 84], image recognition [83], and agentic reasoning [65]. By aligning disparate modalities into a unified latent space through large-scale pretraining, these architectures excel at capturing cross-modal semantic associations [70]. Some notable milestones are SAM-Audio [63], ImageBind [23], and Qwen2.5-Omni [75]. These multimodal foundational models share a common latent space across modalities and present a unified representation by synchronizing them. However, despite their zero-shot capabilities, these models are fundamentally constrained by a static learning paradigm. When deployed in non-static environments requiring Class-Incremental Learning (CIL), they remain acutely vulnerable to catastrophic forgetting [49].

Multimodal Foundational Models in CIL

The emergence of foundational multimodal learning represents a significant paradigm shift in Class-Incremental Learning (CIL) [17, 34, 54, 57, 81, 82, 89]. Rather than training task-specific models from scratch, contemporary approaches focus on the sequential adaptation of pre-trained foundations, such as CLIP [22, 30, 31, 87] to non-stationary data distributions [40]. In this setting, the primary objective is to leverage the robust, zero-shot priors of these models while incrementally expanding their semantic understanding without inducing representational drift or degrading the original cross-modal alignment [20, 50, 52, 66, 86]. To navigate the stability-plasticity dilemma, recent works have explored several high-impact strategies, such as memory-based rehearsal for experience replay, Parameter-Efficient Fine-Tuning (PEFT), LoRA [48] for low-rank adaptation, and prompt tuning alongside knowledge distillation. Recent analytic and prototype-based approaches built on frozen pretrained models such as RanPAC [53], GACL [90], and AnaCPL [55] have shown that catastrophic forgetting can be strongly mitigated via closed-form or analytic classifiers. We propose to leverage the foundational multimodal priors and their representations for incremental adaptation, thereby enabling sequential task mastery and retaining prior learning.

Proposed Method

Multimodal Foundational Models (MFMs)

For our work, we leverage three recent foundational multimodal models, namely: SAM-Audio, ImageBind, and Qwen2.5-Omni-3B, to extract audio and visual features.

SAM-Audio is a large scale model that extends the “Segment Anything” series—originally for images/videos—to the audio domain. The video encoder of the SAM-Audio is based on the perception encoder (PE), a vision encoder trained with large-scale contrastive vision-language learning on over 100 million videos. We also leverage the audio encoder of the SAM-Audio, which is based on the DAC-like autoencoder structure with a Variational Autoencoder (VAE) bottleneck layer. The audio is encoded into a sequence at 25 Hz.

ImageBind is another pioneering multimodal foundational model that unifies six different sensory modalities into a single shared embedding space. In ImageBind, both the visual and audio encoders utilize Vision Transformer (ViT) architectures. The model aligns these by treating audio as a 2D image and mapping both into a shared embedding space anchored by visual data. The visual encoder is based on the ViT-H with approximately 630 million parameters. The audio input is first converted into a 2D mel-spectrogram, which is then passed through the ViT-B-based audio encoder. It uses a patch size of 16 and a stride of 10, treating the spectrogram exactly like a 2D image signal.

Qwen2.5-Omni-3B is an end-to-end multimodal foundation model. The visual encoder handles both static images and video frames and is based on a Vision Transformer (ViT) encoder with approximately 675 million parameters. The audio encoder is derived from Whisper-large-v3, a state-of-the-art model for speech and sound recognition. Raw audio is resampled to 16 kHz and converted into 128-channel mel-spectrograms using a 25ms window and a 10-ms hop.

For all three models, we only leverage their audio and visual encoders to extract the corresponding features, keeping the weights of all three encoders frozen.

Problem Formulation and Overview

The incremental learning process is structured as a sequence of tasks $t \in\{0, 1, ..., T\}$. Each task $t$ introduces a new set of classes $C_t$ that are semantically distinct. The training data available for task $t$ is the dataset $D_t$. Following [56], we adopt the updated training dataset for the task $t$, i.e.

$D'_t = D_t \cup M_{t-1}$, where $M_{t-1}$ are the exemplar samples, comprising samples from the classes learned so far $C_{t-1}$. These exemplar samples are randomly selected from $C_{t-1}$. Each sample from $D'_t$ consists of a video sample $x$ and its corresponding ground-truth label $y$. Let $f_t$ denote the training pipeline for the task $t$. At the base task $t = 0$, we consider the video sample $(x,y) \in D_0$ and extract the corresponding video frames $x_v$ and audio signal $x_a$. The $x_v$ and $x_a$ are passed through the visual $\psi_v$ and audio $\psi_a$ encoder to get the corresponding visual $z_v$ and audio $z_a$ encoded features, respectively. These encoded features are fused using attention-based contextual guidance, where the audio-encoded features $z_a$ are used as contextual guidance for visual-encoded features $z_v$ to finally get the contextually-guided features $z'_v$. These features are finally passed through the classifier to get the predicted label $\hat{y}$. At the incremental task $t > 0$, we utilize $D'_t$, comprising of both current task’s samples $D_t$ and exemplar samples $M_{t-1}$. For the incremental task, we adopt the same training strategy used in the base task.

In summary, our proposed approach is divided into three major components: (a) Foundational Multimodal Feature Extraction, that extracts the audio and visual features from MFM’s encoders; (b) Attention-based Contextual Guidance, that uses the extracted audio features as contextual guidance for the corresponding visual features in transformer-based attention and provides sample-specific multimodal attended features; and (c) Classifier, that predicts the logits from the multimodal attended features. The proposed architecture is visualized in Figure 2.

Foundational Multimodal Feature Extraction

As mentioned before, the proposed approach aims to leverage learned audio and visual representations from various foundational multimodal models in an incremental setting. For our approach, we leverage SAM-Audio ($sam$), ImageBind ($ibind$), and Qwen2.5-Omni-3B ($qwen$) as backbones for both the audio and visual modalities. Specifically, we leverage the audio $\psi_a$ and visual $\psi_v$ encoders of the aforementioned models and pass the audio signal $x_a \in\mathbb{R}^{k}$ and visual frames $x_v \in\mathbb{R}^{T_v \times C \times H \times W}$ as inputs to extract the corresponding audio $z_a$ and visual $z_v$ features:

where $model \in\{sam, ibind, qwen\}$, visual features $z_v^{model} \in\mathbb{R}^{T_v \times d^{model}}$, and audio features $z_a^{model} \in\mathbb{R}^{T_a \times d^{model}}$. $T_a^{model}$ and $T_v$ are the audio chunks and visual frames for different multimodal foundational models (MFMs), respectively. $C$, $H$, and $W$ are the number of channels, height and width of each frame, respectively. $d^{model}$ denotes the hidden dimension for different MFMs. We specify the values of $T_v$, $T_a$, and $d$ in Table 1. The extracted features are passed through the attention-based contextual guidance module to leverage the audio representation as context for effective video recognition.

Models (model)	$d^{model}$	$T_v$	$T_a^{model}$
SAM-Audio	1024	16	59
ImageBind	1024	16	1
Qwen2.5-Omni-3B	2048	16	251

Attention-based Contextual Guidance

The audio $z_a^{model}$ and visual $z_v^{model}$ features from foundational multimodal model ($model \in\{sam, ibind, qwen\}$) represent the sample-wise information. To fuse these features, we perform a preliminary analysis (as illustrated in Table 4) that naively concatenates the audio and visual representations (linear probe); we observe a substantial performance gap. Hence, we propose an attention-based contextual guidance module. This module leverages effective guidance from audio representations to enhance the target object in visual representations. Prior audio-guided attention [35] addresses static, single-task video-text retrieval, where representations need not persist across tasks; our design prioritizes the stability required as new classes arrive. Specifically, the guided attention module is a single-layer attention block comprising layer normalization, Multi-Head Attention (MHA), Feed-Forward Network (FFN), and Multi-Head Self-Attention (MHSA). Below, we provide a detailed breakdown of the pipeline.

Methods	AVE-CI		VS100-CI		K-S-CI
	$A_m \uparrow$	$F_{\mathrm{avg}} \downarrow$	$A_m \uparrow$	$F_{\mathrm{avg}} \downarrow$	$A_m \uparrow$	$F_{\mathrm{avg}} \downarrow$
Linear Probe (SAM-Audio)	77.62	3.43	61.31	4.75	79.71	2.87
Ours (SAM-Audio)	88.68	0.42	79.64	0.34	84.01	0.43
Linear Probe (Qwen2.5)	72.30	9.13	73.36	6.15	72.84	7.31
Ours (Qwen2.5)	86.14	4.18	82.40	1.20	81.85	3.84
Linear Probe (ImageBind)	80.17	4.47	70.22	8.47	71.23	7.54
Ours (ImageBind)	91.48	0.52	84.94	0.06	86.29	0.44

First, we pass the normalized visual representations, along with the audio representations (as contextual guidance), into the Multi-Head Attention (MHA) module. In this module, the normalized visual representations serve as the query (Q), while the normalized audio representations serve as the key (K) and value (V), enabling each visual token to attend to the audio context. This setting allows the audio representations to serve as contextual guidance that enhances the visual features, and the attended output preserves the visual sequence length $T_v$, keeping the residual connection well-defined. Following this, the output is fed into a Feed-Forward Network (FFN) with another residual connection. This fusion process is formulated as:

where LN denotes the layer normalization. MHA denotes the single-layer attention and is formulated as:

where $Q = z_v^{model}W_Q$; $K = z_a^{model}W_K$; $V = z_a^{model}W_V$, and $W_Q$, $W_K$, $W_V \in\mathbb{R}^{d^{model}\times d^{model}}$. The output representation $\bar{z}$ is further refined to enhance overall visual representations. Specifically, $\bar{z}$ is passed through the Multi-Head Self-Attention (MHSA) module with another residual connection followed by FFN and final residual connection to produce the refined visual representation $z_v^{\prime model}$. This process is formulated as:

where MHSA is the single-layer self-attention operation, similar to MHA as $\operatorname{MHSA}(X) = \operatorname{MHA}(X,X,X)$.

Objective Function

Our proposed strategy employs cross-entropy $\mathcal{L}_{ce}$ for class-label prediction. The loss is formulated as:

where $\hat{y}_i$, and $y_i$ denotes the predicted and ground-truth labels respectively. Additionally, we use the feature distillation loss $\mathcal{L}_{fd}$ between the attended visual features of the current and the previous task (on the exemplar samples $\mathcal{M}_{t-1}$) to teach $z_{v,t}^{\prime model}$ to mimic $z_{v,t-1}^{\prime model}$. It is formulated as:

The last part of our final objective function is the task-wise knowledge distillation $\mathcal{L}_{kd}$, where we evaluate the Kullback-Leibler (KL) divergence between the logits of classes in the current ($\hat{y}_t^{C_s}$) and previous ($\hat{y}_{t-1}^{C_s}$) tasks which helps preserve past knowledge and prevent the learned knowledge from being biased by other tasks.

To sum it up, our overall objective function is formulated as: $\mathcal{L}_{tot} = \mathcal{L}_{ce} + \lambda_{fd} * \mathcal{L}_{fd} + \lambda_{kd} * \mathcal{L}_{kd}$, where $\lambda_{fd}$ and $\lambda_{kd}$ are the hyperparameters for feature and knowledge distillation losses, respectively.

Experiments

Experimental Details

Datasets. To ensure fair comparison, we follow AV-CIL [56] and evaluate our proposed approach on three class-incremental learning (CIL) datasets: AVE-Class-Incremental (AVE-CI) dataset [67], which comprises over 4,000 video sequences covering 28 distinct event categories. We divide the 28 classes into four sequential tasks, each containing 7 classes. The Kinetics-Sounds-Class-Incremental (K-S-CI) dataset [39] consists of 23,000 samples spanning 30 audio-visual classes. The dataset is divided into 5 incremental tasks, each containing 6 classes. The VGGSound100-Class-Incremental (VS100-CI) dataset [15] is a large-scale benchmark comprising 60,000 samples across 100 classes. These classes are distributed across 10 incremental tasks, with 10 classes per task, providing a more rigorous test of long-term catastrophic forgetting.

Evaluation Metrics. We employ two standard metrics used in CIL: Mean Accuracy and Average Forgetting.

Mean Accuracy ($A_m$). This metric provides a global measure of the model’s performance across all observed categories. After the model has been trained on $T$ sequential tasks, the Mean Accuracy is defined as the mean of the cumulative accuracies recorded at each incremental task:

where $A_t$ is the test accuracy of all seen classes after completing the training on the current task $t$ (i.e., $\bigcup_{i=1}^{t} C_i$).

Average Forgetting $F_{avg}$. To evaluate the impact of catastrophic forgetting on previously learned knowledge, we employ the Average Forgetting. For a specific task $t$, the forgetting $F_t^i$ regarding a previous task $i < t$ is defined as the maximum decrease in accuracy observed for that task up to the current state. The Average Forgetting after $T$ tasks is expressed as:

where $A_{k,i}$ is the test accuracy of the i-th task after training on the k-th task.

Implementation Details. We implemented our framework using the PyTorch framework on a single workstation with a NVIDIA A100 GPU. In this work, we leverage three different foundational models SAM-Audio [63], ImageBind [23], and Qwen2.5-Omni-3B [75] for their pre-trained audio and visual encoders. During training, we freeze the visual and audio encoders of all the foundational models and train the rest of the pipeline. We use the SGD optimizer with a learning rate of 5e-5, weight decay of 1e-4, and a Cosine Annealing scheduler. For all three datasets, we set the maximum number of training epochs in each incremental task to 200. Following AV-CIL [56], we set exemplar memory size to 340 for AVE-CI, 500 for K-S-CI, and 1500 for VS100-CI. $\lambda_{fd}$, and $\lambda_{kd}$ are set to 1.0. The Attention-based Contextual Guidance module uses a single-layer block with 16 attention heads and an FFN of hidden dimension $d^{model}$. The classifier is a single fully connected layer that maps the attended features to the number of seen classes, expanded incrementally as new tasks arrive. Additional details are provided in Table 1.

Results and Analysis:

Table 2. Comparison of our proposed approach with previous works on different datasets. † represents results that are produced for fair comparison. Best results (%) are marked in Bold.

Methods	Exemplar	AVE-CI $A_m \uparrow$	AVE-CI $F_{avg} \downarrow$	VS100-CI $A_m \uparrow$	VS100-CI $F_{avg} \downarrow$	K-S-CI $A_m \uparrow$	K-S-CI $F_{avg} \downarrow$
Fine-tuning	❌	42.40	70.99	26.21	89.37	41.18	89.62
LwF [46]	❌	58.07	26.90	59.34	23.01	65.54	16.55
iCaRL-NME [62]	✓	56.15	11.71	56.19	12.80	64.51	18.70
iCaRL-FC [62]	✓	65.88	26.08	64.22	29.94	55.54	40.57
SS-IL [4]	✓	61.94	22.49	69.20	9.75	69.71	10.53
AFC-NME [38]	✓	68.46	14.18	61.41	23.30	69.13	27.30
AFC-LSC [38]	✓	65.21	28.11	57.76	29.64	67.02	33.56
AV-CIL [56]	✓	74.04	7.63	72.80	5.49	73.06	6.48
MMAL [81]	❌	76.71	-	76.19	-	78.98	-
CIGNT [54]	❌	75.78	6.88	73.11	5.67	74.08	6.45
MMECN [47]	✓	75.07	6.51	74.37	6.36	72.63	7.44
PromptCL [64]	✓	71.11	3.12	74.14	5.47	73.34	5.70
CMCDR [85]	❌	74.63	14.02	-	-	75.31	9.86
Ours	❌	77.21	3.08	77.76	5.12	79.85	6.02
Ours	✓	78.67	1.05	79.12	4.96	81.21	5.26

Table 3. Comparison of our proposed approach on different MFMs with AV-CIL [56]. † represents results that are produced for fair settings. Best results (%) are marked in Bold.

Comparison with State-of-the-Art

We evaluate our proposed framework along two cues: (a) a backbone-matched evaluation; (b) a generalized evaluation on the three multimodal foundational models. Across both cues, we leverage corresponding extracted features and hyperparameters, and we randomly select exemplar samples for both ours and AV-CIL (using published configuration).

Backbone-matched comparison. For fair comparison, following AV-CIL [56], we leverage the AudioMAE [32] and VideoMAE [68] encoders in our pipeline. As summarized in Table 2, our approach consistently outperforms existing baselines by a substantial margin across all datasets. Specifically, we observe a consistent improvement of +0.5%, +1.57%, and +0.87% on AVE-CI, VS100-CI, and K-S-CI, respectively, from MMAL [81] (under the non-exemplar setting), and against the AV-CIL baseline (for exemplar setting), the margin widens to +4.63%, +6.32%, and +8.15%.

This indicates that even without strong pretrained features, our contextual guidance and dual distillation yield a better stability-plasticity trade-off than the previous state of the art.

Generalization across MFMs. To generalize across different multimodal foundational models (MFMs), we comprehensively compare our approach against AV-CIL [56] on three recent MFMs, SAM-Audio, ImageBind, and Qwen2.5 Omni. From Table 3, we observe that our approach shows significant improvement across all datasets. The gains are largest on ImageBind, where we improve by +5.68%, +7.24%, and +8.33% in accuracy on AVE-CI, VS100-CI, and K-S-CI with similar improvement holding for SAM-Audio and Qwen2.5 as well. Furthermore, our ImageBind-based approach achieves a mean accuracy gain of $\approx3\%$ over the other two MFMs across the three datasets. When compared to AV-CIL, our approach improves by $\approx7\%$ in mean accuracy on the AVE-CI and K-S-CI datasets.

Methods	AVE-CI $A_m \uparrow$	AVE-CI $F_{avg} \downarrow$	VS100-CI $A_m \uparrow$	VS100-CI $F_{avg} \downarrow$	K-S-CI $A_m \uparrow$	K-S-CI $F_{avg} \downarrow$
AV-CIL† (SAM-Audio)	84.38	2.24	74.04	4.11	80.21	1.45
Ours (SAM-Audio)	86.88	0.42	79.64	0.34	84.01	0.43
AV-CIL† (Qwen2.5)	82.98	5.91	80.43	5.40	79.32	4.28
Ours (Qwen2.5)	86.14	4.18	82.40	1.20	81.85	3.84
AV-CIL† (ImageBind)	85.03	3.04	77.70	7.72	77.96	7.09
Ours (ImageBind)	91.48	0.52	84.94	0.06	86.29	0.44

To further validate the stability of our approach, we provide a task-wise test accuracy analysis in Figure 3, comparing our approach against AV-CIL on ImageBind backbone. While AV-CIL suffers from performance decay as the number of tasks increases, our proposed approach maintains nearly consistent performance across tasks.

Analysis on Backbone Contributions: Figure 4 visualizes different backbones and the resulting Ours–AV-CIL margin. We observe that, with the same conventional (Audio/Video MAE) backbone, our method already surpasses AV-CIL by +4.63% on AVE-CI. This improvement is attributable solely to the architecture, since both methods use the same features, and is observed across other MFM-based backbones as well. Furthermore, the performance improvement is not biased toward our method. AV-CIL’s configuration transfers well to MFM features, and in several settings, its improvement exceeds ours (eg VS100/Qwen: +7.63 vs +3.28; K-S/SAM: +7.15 vs +2.80). This margin indicates that the improvement is from the proposed contextual guidance and distillation rather than from backbone-specific tuning.

Impact of MFMs in Linear Setting

In this section, we evaluate the impact of MFMs by training a linear probe on each MFM (naive concatenation). We train a single linear classifier on the frozen audio-visual features, task by task, using only cross-entropy, without contextual guidance or distillation. As shown in Table 4, the linear probe performs poorly across all backbones and datasets, reaching only $\approx73\%$ mean accuracy on ImageBind (far below our full method $\approx87\%$) and degrading sharply across tasks (Figure 1). This indicates that without proper guidance, the frozen backbone’s representations are susceptible to inter-modality interference. Notably, the probe performs poorly even against AV-CIL under the same conditions, indicating that frozen MFM features are not directly usable for incremental learning without targeted adaptation. This confirms that strong priors alone do not solve the task.

Robustness to Task Sequences

To evaluate the robustness of our proposed approach, we conduct a comprehensive analysis across varying incremental task sequences for all three datasets. For the AVE-CI dataset, we introduce additional 2-task settings (4 classes per task and 2 classes per task). For K-S-CI, we introduce additional 3-task settings (15 classes per task, 5 classes per task, and 2 classes per task). Finally, for VS100-CI, we introduce three additional task settings (50 classes per task, 20 classes per task, and 5 classes per task). The results are summarized in Figure 5. Our proposed approach demonstrates remarkable stability, maintaining high performance regardless of whether the model encounters large tasks or many small, frequent tasks. This consistent performance across diverse task sequences underscores the architectural robustness of our approach. A detailed task-wise comparative analysis with AV-CIL [56] is presented in the supplementary file.

$L_{ce}$	$L_{kd}$	$L_{fd}$	SAM-Audio $A_m \uparrow$	SAM-Audio $F_{\mathrm{avg}} \downarrow$	ImageBind $A_m \uparrow$	ImageBind $F_{\mathrm{avg}} \downarrow$	Qwen2.5 $A_m \uparrow$	Qwen2.5 $F_{\mathrm{avg}} \downarrow$
✓	✗	✗	87.26	4.29	86.55	3.18	83.20	7.32
✓	✓	✗	87.34	2.30	87.60	2.80	83.94	5.45
✓	✗	✓	87.79	1.32	87.78	1.02	84.49	5.01
✓	✓	✓	88.68	0.42	91.48	0.52	86.14	4.18

Effectiveness of Individual Loss Functions

To analyze the contributions of each component, we conduct an ablation study on the AVE-CI benchmark. We specifi-

Table 6 presents the sensitivity analysis of distillation hyperparameters $\lambda_{fd}$ and $\lambda_{kd}$ by comparing the accuracy ($A_m$) for different MFM-based backbones. Bold denotes the adopted configuration.

Sensitivity Analysis of Hyperparameters

To analyze the contributions of our distillation components, we conduct a study over $\lambda_{fd}$ and $\lambda_{kd}$. These values balance the trade-off between preserving feature representations and transferring logit-level knowledge. The results,

cal-ly evaluate the impact of Knowledge Distillation ($\mathcal{L}_{kd}$) and Feature Distillation ($\mathcal{L}_{fd}$) relative to the standard Cross-Entropy ($\mathcal{L}_{ce}$) baseline. The results are summarized in Table 5. Comparing the baseline ($\mathcal{L}_{ce}$) to the inclusion of $\mathcal{L}_{kd}$ shows a consistent improvement in Mean Accuracy (from 86.55% to 87.60% for ImageBind), and forgetting (from 3.18% to 2.80%). $\mathcal{L}_{kd}$ serves as a regularizer that maintains the decision boundaries of previously learned classes, hence preventing the classifier from over-fitting to the distribution of the current task. Introducing $\mathcal{L}_{fd}$ provides further improvement than $\mathcal{L}_{kd}$ alone. While $\mathcal{L}_{kd}$ manages label-space consistency, $\mathcal{L}_{fd}$ ensures feature stability (reducing forgetting from 3.18% to 1.02%). The combination results in a jump in accuracy (91.48%) and a drop in forgetting (0.52%). We observe a similar trend in other MFMs as well.

Sensitivity Analysis of Hyperparameters

To analyze the contributions of our distillation components, we conduct a study over $\lambda_{fd}$ and $\lambda_{kd}$. These values balance the trade-off between preserving feature representations and transferring logit-level knowledge. The results, summarized in Table 6, provide insights into the stability-plasticity balance in our framework. We observe that, when $\lambda_{fd}$ and $\lambda_{kd}$ are both set to 1.0 provides the best overall trade-off, achieving the highest Mean Accuracy in five of the six columns. The only exception is K-S-CI with SAM-Audio, where $\lambda_{fd}/\lambda_{kd}$ = 0.2/1.0 attains 84.50 vs 84.01 at the adopted setting. We retain 1.0/1.0 as it is optimal for the remaining configurations and best on average. For a given backbone and dataset, the variance between configurations remains small, indicating low sensitivity to the weights.

$\lambda_{fd}$	$\lambda_{kd}$	\multicolumn{3}{c}{AVE-CI}	\multicolumn{3}{c}{K-S-CI}
		SAM-Audio	ImageBind	Qwen2.5	SAM-Audio	ImageBind	Qwen2.5
0.2	1.0	88.04	88.66	82.92	84.50	84.88	81.52
0.4	1.0	87.90	89.33	82.68	84.36	85.13	81.52
0.6	1.0	88.06	89.48	82.41	84.07	85.11	81.49
0.8	1.0	87.92	89.36	82.57	83.72	85.05	81.46
1.0	1.0	88.68	91.48	86.14	84.01	86.29	81.85
0.5	0.8	88.14	89.40	82.81	84.35	85.10	81.53
0.5	0.6	86.06	89.51	82.83	84.28	85.47	80.57
0.5	0.4	88.09	89.39	82.80	84.33	85.67	80.64
0.5	0.2	88.44	89.96	82.68	84.04	85.61	81.71

Conclusion

In this work, we addressed the challenge of Class-Incremental Learning (CIL) by leveraging Multimodal Foundational Models (MFMs). By leveraging the frozen priors of MFMs such as SAM-Audio, ImageBind, and Qwen2.5-Omni, we propose a unified, shared latent strategy and present a generic CIL system for audio-visual class recognition. Furthermore, we employ an Attention-based Contextual Guidance module that uses audio representations as context, followed by dual distillation, for effective class recognition. Empirical results across multiple benchmarks show that our method significantly outperforms current baselines.

Acknowledgement

This work was started during the internship of Avi Gupta at Dolby Laboratories and was later supported in part by the Infosys Centre for Artificial Intelligence, IIIT-Delhi, and in part by the Institute Fellowship of the IIIT-Delhi. A preliminary version of this work was presented at the ICML workshop [26].

Introduction

In this section, we present the additional analysis on our proposed “Generalizing Multimodal Foundational Models in Context-Driven Class Incremental Learning”. We first present the training pipeline of our proposed work (Algorithm 1), followed by ablation studies on individual modalities, analysis across multiple runs, and task-wise performance across different settings.

Algorithm 1 Training Pipeline

1. Input: Training dataset for current task $t$, $D'_t = D_t \cup\mathcal{M}_{t-1}$; total tasks $T$.

2. // model $\in\{sam, ibind, qwen\}$

3. for $t \in\{0,1,\ldots,T\}$ do

4. &nbsp;&nbsp;&nbsp;&nbsp;for $x_{a,i}, x_{v,i}, y_i \in D'_t$ do

5. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// — Foundational Multimodal Feature Extraction —

6. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$z_{a,i}^{model}, z_{v,i}^{model} \leftarrow\operatorname{Features}(x_a,x_v;model)$

7. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// — Attention-based Contextual Guidance —

8. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$z_{v,i}^{\prime model} \leftarrow\operatorname{Guided\ Attention}(z_{v,i}^{model}, z_{a,i}^{model})$

9. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// — Classifier —

10. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\hat{y}_i \leftarrow\operatorname{Classifier}(z_{v,i}^{\prime model})$

11. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if $t=0$ then

12. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\operatorname{Loss} \leftarrow\mathcal{L}_{ce}$

13. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else

14. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\operatorname{Loss} \leftarrow\mathcal{L}_{tot}$

15. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end if

16. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\operatorname{Loss}(\hat{y}_i,y_i)$

17. &nbsp;&nbsp;&nbsp;&nbsp;end for

18. end for

Effectiveness of Individual Modality

To assess the necessity of a multimodal framework for class-incremental learning, we conduct an ablation study by isolating the contributions of audio and visual inputs. In Table 7, unimodal baselines demonstrate a significant performance deficit, failing to capture the complex semantic nuances. Furthermore, we observe a performance improvement through the integration of both modalities. For instance, in the SAM-Audio backbone, fusing audio and video yields a significant improvement in Mean Accuracy over the best-performing unimodal variant. As for Qwen2.5, we observe a high unimodal performance for both audio (67.75%) and visual-only (69.84%) counterparts. Nevertheless, across these varying priors, the dual-modality configuration consistently emerges as optimal, achieving a peak accuracy of 86.14%.

Audio	Video	SAM-Audio	Qwen2.5
✓	✗	30.75	67.75
✗	✓	38.98	69.84
✓	✓	88.68	86.14

Analysis over Multiple Runs

For robustness across different random runs, we have run three independent random seeds on AVE-CI, VS100-CI, and K-S-CI for all three MFMs and the conventional AudioMAE/VideoMAE backbone, and report mean and standard deviation in Table 8. The main paper reports the single run, and the supplementary presents the seed-averaged values. The standard deviations are small, and the Ours–AV-CIL margin exceeds the combined variance at every backbone, indicating the improvement is robust to task ordering rather than an artifact of a particular seed.

Methods	AVE-CI		VS100-CI		K-S-CI
	AV-CIL	Ours	AV-CIL	Ours	AV-CIL	Ours
	$A_m$	$F_{avg}$	$A_m$	$F_{avg}$	$A_m$	$F_{avg}$
AudioMAE/VideoMAE	74.04	7.63	78.06±1.17	1.12±0.45	72.80	5.49
	79.88±1.37	4.65±0.55	73.06	6.48	80.24±1.32	5.45±0.76
SAM-Audio	84.38	2.24	88.72±0.67	0.43±0.03	74.04	4.11
	79.11±0.76	0.37±0.08	80.21	1.45	83.34±1.75	0.54±0.12
ImageBind	85.80	3.04	91.25±0.51	0.55±0.06	77.70	7.72
	84.01±1.21	0.10±0.06	77.96	7.09	86.11±0.55	0.76±0.54
Qwen2.5	82.98	5.91	86.97±1.12	4.66±0.97	80.43	5.40
	82.50±0.24	1.34±0.32	79.32	4.28	82.20±0.44	3.54±0.40

Task-wise Performance for different settings

We present the task-wise performance (Table 10-Table 15) across diverse task sequences (briefly described in {$\S$}4.2 of the paper). We evaluate the corresponding performance across the three foundational models and compare against AV-CIL. Our proposed method consistently outperforms the AV-CIL baseline across all three datasets in both traditional and task-wise learning settings. Among the backbones, ImageBind establishes the highest performance; in the traditional Kinetics-CI setting, it achieves 96.77% on the initial task and sustains 81.05% by the final step. On the AVE-CI dataset, our method preserves near-perfect accuracy on the initial task across all backbones. Even in the highly challenging 20-task split of VGGSound100-CI, our approach prevents the severe performance collapse observed in the baseline, keeping late-stage task accuracies firmly in the 65% to 75% range.

Steps	SAMAudio		ImageBind		Qwen2.5
	AV-CIL	Ours	AV-CIL	Ours	AV-CIL	Ours
Step 0	86.15	90.39	91.34	95.20	90.38	92.30
Step 1	85.23	91.43	87.14	95.71	83.33	86.19
Step 2	83.01	86.35	84.75	86.67	81.58	83.80
Step 3	83.14	86.55	79.90	88.32	76.64	82.23

Steps	SAMAudio		ImageBind		Qwen2.5
	AV-CIL	Ours	AV-CIL	Ours	AV-CIL	Ours
6 Tasks (5 classes per task)
Step 0	89.19	95.82	92.74	98.12	84.35	95.18
Step 1	81.42	86.09	77.58	86.75	70.90	83.28
Step 2	70.77	76.04	76.39	85.62	69.68	76.04
Step 3	79.37	81.75	72.00	82.91	66.76	79.35
Step 4	74.86	79.23	66.15	80.85	61.66	77.31
Step 5	72.66	78.14	63.80	79.21	59.66	73.75
2 Tasks (15 classes per task)
Step 0	84.04	87.34	85.11	91.70	71.82	80.54
Step 1	77.89	81.05	76.48	83.81	62.67	76.10
15 Tasks (2 classes per task)
Step 0	90.00	99.15	97.67	99.78	88.84	99.15
Step 1	87.59	94.26	86.75	96.72	68.31	88.11
Step 2	85.97	87.63	73.39	94.35	74.27	79.03
Step 3	85.09	86.60	75.46	88.60	74.36	80.40
Step 4	78.92	88.28	72.41	88.28	72.78	81.09
Step 5	79.28	86.88	67.84	87.40	67.00	78.96
Step 6	69.61	79.44	66.83	82.67	68.56	76.33
Step 7	67.19	76.60	63.92	83.69	62.61	75.24
Step 8	72.88	80.68	60.82	81.63	63.35	75.22
Step 9	67.90	77.48	59.19	81.37	63.90	74.53
Step 10	69.78	78.90	61.61	81.50	65.70	74.75
Step 11	66.59	77.09	58.99	77.80	61.90	71.20
Step 12	68.40	77.92	60.83	80.18	61.80	71.63
Step 13	67.93	77.52	56.41	79.00	63.51	71.03
Step 14	64.39	74.92	57.07	76.97	59.95	67.82

Compute/Memory

Our framework extracts audio/visual features from the frozen encoders of MFMs, which are computed offline; later, incremental training and inference are performed on these features via a contextual guidance module and a linear classifier. Table 9 reports memory overhead for each backbone. The trainable parameters for MFM-based backbones are $\approx$70M, their FLOPs are on the order of $\approx$1.0GFLOPs, and peak training VRAM is $\approx$3.5GB is determined by the shallow trainable module rather than the encoders. The exemplar buffer stores features rather than raw media, significantly reducing per-buffer storage. The exemplar cost (averaged across different MFM backbones) for AVE-CI (340 samples) is 255MB, VS100-CI (1500 samples) is 1.13GB, and K-S-CI (500 samples) is 375MB. This design makes the recurring incremental cost lightweight, a practical advantage for continual deployment.

Backbone	Frozen Params	Trainable Params	GFLOPs	Latency	VRAM
SAMAudio	≈500M	27.31M	1.06G	4.11ms	1.58GB
ImageBind	≈700M	27.31M	0.54G	5.84ms	1.41GB
Qwen2.5	≈1.5B	109.15M	2.17G	6.12ms	5.61GB

Steps	SAMAudio		ImageBind		Qwen2.5
	AV-CIL	Ours	AV-CIL	Ours	AV-CIL	Ours
Step 0	83.14	91.33	92.33	93.34	82.54	92.94
Step 1	81.84	85.70	86.30	89.02	80.93	86.20
Step 2	78.06	83.03	84.10	87.46	80.47	87.19
Step 3	76.45	82.02	81.12	86.91	77.0	83.59
Step 4	76.36	80.61	78.80	86.24	75.31	83.35
Step 5	72.69	77.36	75.62	82.96	71.35	81.25
Step 6	70.05	75.65	73.18	82.43	70.31	79.88
Step 7	68.93	75.21	71.33	81.96	67.25	78.51
Step 8	67.15	72.82	67.27	79.64	65.01	75.93
Step 9	65.76	72.60	66.86	79.46	64.14	75.20

Steps	SAMAudio AV-CIL	SAMAudio Ours	ImageBind AV-CIL	ImageBind Ours	Qwen2.5 AV-CIL	Qwen2.5 Ours
Step 0	89.59	95.97	93.14	96.77	92.74	94.09
Step 1	80.05	85.97	77.97	85.71	77.77	81.95
Step 2	80.23	80.85	76.78	84.75	77.57	80.24
Step 3	79.27	79.42	73.98	83.17	76.46	78.64
Step 4	74.23	77.83	67.93	81.05	72.04	74.31

Steps	SAMAudio AV-CIL	SAMAudio Ours	ImageBind AV-CIL	ImageBind Ours	Qwen2.5 AV-CIL	Qwen2.5 Ours
7 Tasks (4 classes per task)
Step 0	86.55	99.2	90.00	98.89	88.00	98.28
Step 1	86.36	70.0	77.00	80.0	55.45	63.63
Step 2	85.45	86.93	75.34	91.47	78.07	78.98
Step 3	83.83	88.07	74.36	92.59	62.30	68.72
Step 4	81.15	82.62	74.59	86.88	68.69	71.48
Step 5	85.18	83.57	73.00	86.40	69.89	70.82
Step 6	83.91	82.49	71.98	85.53	65.38	73.10
2 Tasks (14 classes per task)
Step 0	85.71	94.76	82.38	93.80	78.10	80.05
Step 1	79.34	85.03	73.50	89.85	69.44	79.95

Steps	SAMAudio		ImageBind		Qwen2.5
	AV-CIL	Ours	AV-CIL	Ours	AV-CIL	Ours
5 Tasks (20 classes per task)
Step 0	84.36	91.74	91.74	93.66	82.35	88.78
Step 1	79.38	80.51	81.04	83.38	76.25	72.92
Step 2	73.20	75.65	76.93	79.28	71.62	85.89
Step 3	70.17	75.16	71.44	77.76	67.83	82.46
Step 4	66.87	73.54	67.02	75.98	64.35	90.04
2 Tasks (50 classes per task)
Step 0	79.94	87.33	87.13	91.19	78.33	82.48
Step 1	68.70	72.74	70.32	77.55	63.93	81.80
20 Tasks (5 classes per task)
Step 0	78.71	85.48	77.90	87.90	80.32	82.56
Step 1	83.55	88.50	87.50	89.52	81.33	83.59
Step 2	83.94	88.02	88.29	90.71	73.27	77.74
Step 3	81.04	83.08	83.59	87.92	70.94	80.50
Step 4	80.34	81.16	83.01	86.80	70.42	79.49
Step 5	77.86	78.20	81.76	83.84	71.48	78.05
Step 6	79.69	79.45	82.04	84.97	69.75	77.46
Step 7	76.35	77.20	79.15	82.53	64.94	76.18
Step 8	77.25	77.20	79.07	83.72	66.85	76.03
Step 9	76.08	75.70	78.88	82.98	64.67	75.03
Step 10	73.64	73.17	76.29	79.62	63.02	74.51
Step 11	71.59	72.23	74.71	78.71	61.19	71.93
Step 12	69.75	69.78	73.55	76.71	69.96	71.41
Step 13	67.84	70.10	71.11	76.52	69.33	71.07
Step 14	67.96	68.16	70.47	75.87	66.83	69.86
Step 15	58.81	67.74	69.43	74.81	65.57	92.95
Step 16	57.35	65.06	67.71	72.40	64.03	78.25
Step 17	56.35	65.10	65.86	72.60	64.30	79.68
Step 18	55.54	63.78	64.77	70.90	63.46	78.23
Step 19	54.81	64.06	63.78	71.93	62.40	75.28