betaFeedback

WorksComputer Science

Learning What the True Labels Would Teach: Posterior Analytic Class-Incremental Learning with Unlabeled Data

Abstract

In class-incremental learning with unlabeled data, each task brings only class names and unlabeled images, so CLIP's zero-shot predictions replace labels. Existing methods train on hard pseudo-labels, but a wrong one gives the true class zero probability, which self-training often confirms. We argue that each image should carry its label posterior, as an analytic classifier trained on posteriors equals the true-label one in expectation. The zero-shot prediction is not this posterior, as its class proportions confound name bias with true imbalance and it ignores other images. We propose Posterior Analytic Class-Incremental Learning (PACIL), which corrects the class prior with a prior selection rule and refines each distribution by a fixed-prior von Mises-Fisher mixture on whitened features. These soft labels train the classifier and replace labels to select its fusion weight with zero-shot CLIP by closed-form leave-one-out cross-validation. We prove that posterior targets minimize the mean-square gap to the true-label prediction and that their argmax widens this gap only through ambiguous images. The fusion weight selected with soft labels loses at most their distance to the posteriors in expected correct decisions. PACIL has the best average incremental accuracy among methods trained without labels on all 18 settings.

Introduction

Vision-language models such as CLIP [1] classify images of any named class, so their zero-shot predictions can supervise models on unlabeled images. Pseudo-labeling methods reduce each prediction to one class or a candidate set [2, 3], while transductive methods keep the whole distribution [4, 5] but only to label the images they are fitted on, not future ones. What target each image should carry when a classifier is trained on such predictions remains open.

This question is critical in class-incremental learning (CIL) [6], where a model learns disjoint classes in sequence and must recognize all seen classes without revisiting old

data. In CIL with unlabeled data [7, 8], each task brings only its new class names and unlabeled images. This arises when categories change faster than annotation, as for retail products [9] or camera-trap wildlife surveys, whose 3.2 million images would take over eight years of full-time work to label [10]. Earlier images may also be withheld for privacy [11], and each target is then fixed once its task ends. Existing methods, from CLIP-based CIL on zero-shot argmax labels [8, 12, 13, 14] to N2L [7] with self-training, all use hard pseudo-labels. Figure 1a measures how often the true label is discarded. A hard label discards it for most images, and keeping more top-scored labels lowers this rate slowly. N2L refines the hard labels three times, which may introduce confirmation bias [15]. With two to ten labels kept, it discards more true labels than zero-shot CLIP, as fitting a wrong label suppresses the true one. Only the full distribution discards no true label. The target of each image should therefore keep it and give the true label as much weight as the observations support.

Figure 1. Discarded true labels and average accuracy. (a) Training images (%) whose true label is discarded when each keeps only its top-scored labels, on the first task of Aircraft (half-base), for zero-shot CLIP, N2L after three self-training refinements and PACIL. (b) Average accuracy Aˉ\bar{A} of the methods of Table 1 on its zero-base (Z) and half-base (H) settings, each axis spanning the lowest (inner ring) to the highest value (outer ring). Numbers are the Aˉ\bar{A} of ZS-CLIP (gray), N2L (blue), and PACIL (red).

We argue that this target is the label posterior given all observations. Soft targets in distillation [16] and expectation-maximization (EM) [17] rest on the same idea, which becomes exact for analytic learning [11, 18], the exemplar-free learner we build on. Its solution is linear in the targets, so training on the posteriors yields the conditional mean of the classifier that the true labels would give. A hard label departs from this target on every ambiguous image, even when its argmax is correct. Because the posteriors also give the probability that each decision is correct, they can replace labels when choosing the weight that fuses the classifier with zero-shot CLIP. However, keeping the whole zero-shot distribution rather than its argmax is not enough. Its class proportions mix the name bias of CLIP with true imbalance [19, 20, 21] in shares that vary across datasets and tasks, and it ignores the other images of the task. Without labels or old images, three questions arise. (i) Which class proportions should the prior of each new task follow? (ii) How far should the images of a task revise each distribution when old classes are kept only as statistics? (iii) How much should predictions over all seen classes trust the analytic classifier rather than zero-shot CLIP?

We answer these questions with Posterior Analytic Class-Incremental Learning (PACIL), whose soft labels estimate the posteriors. For (i), a prior selection rule tests on the images of each task whether the skew of its zero-shot class proportions is name bias or true imbalance, which text alone cannot separate. For (ii), the selected distribution is a fixed per-image prior of a von Mises-Fisher (vMF) mixture fitted by EM on features whitened with statistics of all seen classes, so the images revise each distribution but not the class proportions. For (iii), closed-form leave-one-out (LOO) cross-validation picks the fusion weight of each task with the most correct decisions expected under the soft labels. In our analysis, rounding the posteriors to their argmax adds exactly the squared change this causes in the prediction, a cost that comes only from ambiguous posteriors. The distance of the soft labels to the posteriors bounds both the extra prediction error and what the selected fusion weight can lose in expected correct decisions. Prior selection and EM are therefore designed to reduce this distance before a task ends and its targets are fixed.

Our contributions are as follows. We show that hard pseudo-labels discard plausible classes and that their errors persist because targets are fixed after each task, and we therefore cast labeling as posterior estimation. We prove that posteriors are mean-square optimal targets for analytic learning and that they yield a label-free fusion criterion whose loss is bounded by the soft-label error. We propose PACIL, an exemplar-free method that estimates these posteriors and attains the best average incremental accuracy among label-free methods on every setting, especially where zero-shot CLIP is weakest (Figure 1b).

Related Work

Class-Incremental Learning with CLIP. Most CLIP-based CIL methods tune CLIP or added modules by gradient descent on the labeled images of each task while limiting forgetting [12, 14, 22, 23, 24, 25]. Analytic learning replaces gradient training by a recursively updated ridge regression on frozen features, reaching the joint-training solution without exemplars [11, 18, 26, 27]. RAIL [13] and VILA [28] fuse such a learner with zero-shot CLIP by a fixed rule, and AnytimeCL [29] weights a tuned model per class by labeled accuracy. All of these methods need labeled tasks. Class-incremental novel class discovery [30, 31] and unsupervised CIL [32] learn from unlabeled images without class names, and name-only continual learning [33] retrieves web images. N2L [7] introduced CIL with unlabeled data and trains an analytic learner on zero-shot argmax labels refined by self-training. CrossWorld-CL [8] tunes prompts and adapters on confident hard pseudo-labels in a similar setting, replaying retrieved ImageNet images. PACIL trains an analytic learner on estimated label posteriors and selects its fusion weight with them.

Learning from Unlabeled Images with CLIP. Methods that adapt CLIP to unlabeled images differ in how much of its zero-shot distribution they keep. Pseudo-labeling methods keep one class per image [2, 34, 35] or a candidate set [3] and tune CLIP’s prompts on them. Transductive methods keep the whole distribution and refine it over a set of test images, and test-time methods [36, 37] do so over a stream. Martin et al. [4] fit a Dirichlet mixture to CLIP’s probability vectors and re-estimate the class proportions of each batch. TransCLIP [38] fits a Gaussian mixture with shared covariance to the image features, regularized by a neighborhood graph and a KL term toward the zero-shot prediction. MOON [5] uses vMF components whose mean directions shrink toward the text embeddings. Without graph terms, both multiply each zero-shot prediction by a class-conditional likelihood, the form of our E-step.

Frolic [39] removes an estimated label bias under a uniform test prior and fuses CLIP with a feature-space classifier at a weight equalizing their average confidences. These methods either predict the images they are fitted on or train on hard labels. Unlike Martin et al. [4], TransCLIP [38] and Frolic [39], PACIL fixes the prior of each image from the prior selection rule and whitens features with the within class scatter of all seen classes.

Class Priors and Label Bias

Class priors are usually corrected with labels or a known source prior. Under label shift, Saerens et al. [40] re-estimate the target priors by EM and test whether they differ from the source prior. BBSE [41] inverts a confusion matrix estimated on labeled source data. GLA [20] fits the name bias of zero-shot CLIP as a per-class logit offset on labeled data. It adds the debiased logits to those of a fine-tuned model with equal weights, which is Bayes optimal if the two are independent given the label. Without labels, zero-shot predictions and pseudo-labels are rebalanced toward a uniform [42, 43, 44], given [45] or estimated marginal [36]. Wang et al. [46] estimate such a marginal in partial-label learning [47] and refine candidate-set labels toward it by Sinkhorn scaling. PACIL asks the question of Saerens et al. [40] without a source prior, namely whether the skew of the zero-shot marginal reflects the data. Since logits alone cannot separate name bias from true imbalance, it selects the raw or the debiased prior

from the images. For fusion, GLA’s equal weights rely on conditional independence, which fails because the targets of PACIL come from zero-shot outputs. PACIL therefore selects the fusion weight by closed-form LOO.

Method

As illustrated in Figure 2, for each task, PACIL estimates the posterior of the label of every unlabeled image, as the targets of the classifier and in place of labels when the fusion weight is chosen. Section 3.1 fixes the setting and shows why posteriors are the right targets for the analytic classifier. The next three subsections answer questions (i) to (iii) of Sec. 1 in the order of Figure 2. Class-prior correction (Sec. 3.2) chooses the class proportions that the labels of a new task follow. Posterior label-distribution estimation (Sec. 3.3) then refines each label distribution with the images of the task, and soft-label analytic learning and fusion (Sec. 3.4) trains the classifier on these soft labels and selects its fusion weight after each task.

Figure 2. Overview of PACIL on task tt. (1) Class-prior correction: the prior selection rule chooses PtselP_t^{\mathrm{sel}} between the raw and the debiased zero-shot distribution. (2) Posterior label-distribution estimation: EM of a vMF mixture on whitened features keeps PtselP_t^{\mathrm{sel}} as a fixed per-image prior and combines it with a tempered likelihood into the soft labels qiq_i. (3) Soft-label analytic learning and fusion: the analytic classifier WtW_t is fitted to qiq_i, and closed-form LOO selects the weight ωt\omega_t that fuses it with zero-shot CLIP.

Problem Formulation

Setting. Following N2L [7], a model learns tasks t=1,…,Tt = 1,\ldots,T in sequence, as when a shop or a camera-trap survey adds newly named products or species from unlabeled images. Task tt brings ktk_t new classes Yt\mathcal{Y}_t, disjoint from those of other tasks, with only their names and a set Dt\mathcal{D}_t of ntn_t unlabeled images drawn from these classes. No label yi∈Yty_i \in\mathcal{Y}_t is revealed, even in the first task, which holds a tenth of the classes in the zero-base setting (T=10T = 10) and half in the half-base setting (T=6T = 6). No earlier image or per-image feature is kept, and after task tt the model classifies test images over all KtK_t seen classes Y1:t=Y1∪⋯∪Yt\mathcal{Y}_{1:t} = \mathcal{Y}_1 \cup\cdots\cup\mathcal{Y}_t without task identity. A frozen CLIP model with image encoder fimgf_{\mathrm{img}} and text encoder ftextf_{\mathrm{text}} gives the feature xi∈Rd\boldsymbol{x}_i \in\mathbb{R}^d of image ii and the embedding zc=nrm⁡(ftext(prompt⁡(c)))\boldsymbol{z}_c = \operatorname{nrm}(f_{\mathrm{text}}(\operatorname{prompt}(c))) of class cc, where nrm⁡(v)=v/∥v∥\operatorname{nrm}(\boldsymbol{v}) = \boldsymbol{v}/\lVert\boldsymbol{v}\rVert and prompt⁡(c)\operatorname{prompt}(c) inserts the class name into the first prompt template. With CLIP’s logit scale ss and the logits ℓic=snrm⁡(xi)⊤zc\ell_{ic} = s\operatorname{nrm}(\boldsymbol{x}_i)^\top\boldsymbol{z}_c, the raw zero-shot distribution of image ii, row piraw\boldsymbol{p}_i^{\mathrm{raw}} of Praw∈Rnt×ktP^{\mathrm{raw}} \in\mathbb{R}^{n_t \times k_t}, is

Picraw=exp⁡(ℓic)∑c′∈Ytexp⁡(ℓic′),c∈Yt.(1)P_{ic}^{\mathrm{raw}} = \frac{\exp(\ell_{ic})}{\sum_{c'\in\mathcal{Y}_t}\exp(\ell_{ic'})}, \qquad c \in\mathcal{Y}_t. \tag*{(1)}

Analytic Classifier. Like RanPAC [18], RAIL [13] and N2L, PACIL uses an analytic classifier, a ridge regression on random features ϕ(x)=max⁡(R⊤x,0)∈RD\phi(\boldsymbol{x}) = \max(R^\top\boldsymbol{x}, 0) \in\mathbb{R}^D with a fixed random matrix R∈Rd×DR \in\mathbb{R}^{d \times D}. Each image ii of task tt enters with its features ϕi=ϕ(xi)\phi_i = \phi(\boldsymbol{x}_i), a weight mi>0m_i > 0 and a target QiQ_i in the probability simplex Δkt−1\Delta^{k_t-1}, and after each task the classifier is updated in closed form [11],

At=At−1+∑i∈Dtmiϕiϕi⊤,Wt=(At+λID)−1Ct,A_t = A_{t-1} + \sum_{i\in\mathcal{D}_t}m_i\phi_i\phi_i^\top,\qquad W_t = (A_t+\lambda I_D)^{-1}C_t,
Ct=[Ct−1,0]+∑i∈DtmiϕiQi⊤.(2)C_t = [C_{t-1},0] + \sum_{i\in\mathcal{D}_t}m_i\phi_iQ_i^\top. \tag*{(2)}

where At∈RD×DA_t \in\mathbb{R}^{D\times D} and Ct∈RD×KtC_t \in\mathbb{R}^{D\times K_t} accumulate weighted moments (A0=0A_0 = 0, C0C_0 empty), 00 is a D×ktD \times k_t zero block for the new classes, λ>0\lambda> 0 is the ridge coefficient and Wt∈RD×KtW_t \in\mathbb{R}^{D\times K_t} is the classifier, the ridge solution on all seen images although none is stored. Hard-label methods such as N2L set QiQ_i to the one-hot vector ec\boldsymbol{e}_c of a pseudo-label cc.

Posterior targets. The true label yiy_i of each image is unknown, and the classifier Wt⋆W_t^\star that the true labels would give is Eq. (2) with Qi=eyiQ_i = \boldsymbol{e}_{y_i}. Let O\mathcal{O} denote everything observed, the images and class names of all seen tasks, and let the weights mim_i be fixed. As WtW_t is linear in its targets,

E[Wt⋆∣O]=(At+λID)−1∑imiϕiE[eyi∣O]⊤,(3)\mathbb{E}[W_t^\star\mid\mathcal{O}] = (A_t+\lambda I_D)^{-1}\sum_i m_i\phi_i\mathbb{E}[\boldsymbol{e}_{y_i}\mid\mathcal{O}]^\top, \tag*{(3)}

where the sum runs over all seen images and E[eyi∣O]\mathbb{E}[\boldsymbol{e}_{y_i}\mid\mathcal{O}] is the posterior distribution of yiy_i. Training on the posterior of each label therefore gives the conditional mean of the classifier that the true labels would give. Labeling becomes posterior estimation, and Secs. 3.2 and 3.3 estimate the posterior of each image of task tt as a soft label qi∈Δkt−1q_i \in\Delta^{k_t-1}.

Class-Prior Correction

The class proportions of a new task are unknown. Let pˉt=1nt∑iPiraw∈Δkt−1\bar{p}_t = \frac{1}{n_t}\sum_i P_i^{\mathrm{raw}} \in\Delta^{k_t-1} be the class marginal of the zero-shot distribution and uu the uniform marginal with entries 1/kt1/k_t. The marginal pˉt\bar{p}_t departs from uu for two reasons that call for opposite actions. Name bias should be removed, since CLIP disfavors some class names, such as an aircraft variant named without its manufacturer, and gives these classes too little mass. True imbalance should be kept, since in datasets such as SUN the classes differ in size and the skew of pˉt\bar{p}_t reflects the data. Both act on the logits as one offset per class, and the text alone cannot tell them apart. PACIL therefore builds one candidate prior for each cause and selects between them from the images.

Two candidate priors. The raw candidate is PrawP^{\mathrm{raw}} itself. The debiased candidate removes any class-level offset η∈Rkt\boldsymbol{\eta} \in\mathbb{R}^{k_t} from the logits. Starting from η=0\boldsymbol{\eta} = 0, Sinkhorn iterations [48] alternate, for every c∈Ytc \in\mathcal{Y}_t,

P(η)ic=exp⁡(ℓic+ηc)∑c′∈Ytexp⁡(ℓic′+ηc′),P(\boldsymbol{\eta})_{ic} = \frac{\exp(\ell_{ic}+\eta_c)}{\sum_{c'\in\mathcal{Y}_t}\exp(\ell_{ic'}+\eta_{c'})},
ηc←ηc+log⁡ntkt−log⁡∑i∈DtP(η)ic,(4)\eta_c \leftarrow\eta_c + \log\frac{n_t}{k_t} - \log\sum_{i\in\mathcal{D}_t}P(\boldsymbol{\eta})_{ic}, \tag*{(4)}

until every class has mass nt/ktn_t/k_t. As P(0)=PrawP(0) = P^{\mathrm{raw}}, the iterations move from the raw distribution to the debiased distribution Pbal=P(η)P^{\mathrm{bal}} = P(\boldsymbol{\eta}), the KL projection of PrawP^{\mathrm{raw}} onto distributions with equal class masses [49].

Prior selection rule. The prior selection rule tests whether the images reproduce the imbalance of pˉt\bar{p}_t under a prior that carries none. It runs the estimator of Sec. 3.3 from the debiased prior without tempering, qibal=EM⁡(Pbal,1)q_i^{\mathrm{bal}} = \operatorname{EM}(P^{\mathrm{bal}},1), which gives the likelihood its full weight, and takes the class marginal qˉt=1nt∑iqibal\bar{q}_t = \frac{1}{n_t}\sum_i q_i^{\mathrm{bal}}. Since PbalP^{\mathrm{bal}} has a uniform marginal and the estimator never re-estimates the prior of an image, any departure of qˉt\bar{q}_t from uu comes from the images alone. The rule regresses this departure on that of the zero-shot marginal and selects the prior,

ρt=⟨qˉt−u,pˉt−u⟩∥pˉt−u∥2,Ptsel={Praw,ρt>12,Pbal,otherwise,(5)\rho_t = \frac{\langle\bar{q}_t-u,\bar{p}_t-u\rangle}{\lVert\bar{p}_t-u\rVert^2},\qquad P_t^{\mathrm{sel}} = \begin{cases} P^{\mathrm{raw}}, & \rho_t > \frac{1}{2},\\ P^{\mathrm{bal}}, & \text{otherwise}, \end{cases} \tag*{(5)}

with ρt=0\rho_t = 0 if pˉt=u\bar{p}_t = u. The threshold is not tuned, since ρt>12\rho_t > \frac{1}{2} exactly when qˉt\bar{q}_t is closer to pˉt\bar{p}_t than to uu. PselP^{\mathrm{sel}} fixes the class proportions of the task, but each of its rows rests on the zero-shot logits of a single image.

Posterior Label-Distribution Estimation

Images of a class tend to lie close together, the cluster assumption [50], which allows the images of a task to correct each other’s label distributions. Fitting a standard mixture model to CLIP features faces two obstacles. Cosines between CLIP features mix class differences with within-class variation such as pose and background, and such a mixture would re-estimate the class proportions that § 3.2 has fixed. PACIL therefore fits a mixture with fixed per-image priors on whitened features, whose statistics cover all seen classes without storing their images.

Whitening with accumulated statistics. PACIL compares images on the unit sphere after the whitening map

x~i=nrm⁡(L−1V⊤(xi−xˉ))∈Sdp−1,(6)\widetilde{x}_i=\operatorname{nrm}\left(L^{-1}V^{\top}(x_i-\bar{x})\right)\in\mathbb{S}^{d_p-1}, \tag*{(6)}

which centers xix_i at the mean xˉ\bar{x} of all seen features, keeps its coordinates along the top dpd_p principal directions V∈Rd×dpV\in\mathbb{R}^{d\times d_p} of their total covariance, and applies L−1L^{-1} to make the within-class spread equal in every direction, so that cosines on Sdp−1\mathbb{S}^{d_p-1} become Mahalanobis cosines [51, 52]. Here LL is the Cholesky factor of V⊤ΣwVV^{\top}\Sigma_wV, and Σw∈Rd×d\Sigma_w\in\mathbb{R}^{d\times d} is the within-class covariance after shrinkage toward a multiple of the identity with the oracle approximating shrinkage (OAS) coefficient [53], which keeps L−1L^{-1} stable.

With few images per task, all with uncertain labels, Σw\Sigma_w is pooled over all seen classes. Earlier images enter only through statistics accumulated under their frozen soft labels, the second moment GG, the sum gg and the number NN of their features, and for each class cc the soft sum hch_c, a column of HH, and the soft count νc\nu_c. A run of EM merges the ntn_t current images and their current soft labels qq into these statistics, and a prime marks the merged values, G′=G+∑ixixi⊤G'=G+\sum_i x_i x_i^{\top}, g′=g+∑ixig'=g+\sum_i x_i and N′=N+ntN'=N+n_t, with hc′=hc+∑iqicxih'_c=h_c+\sum_i q_{ic}x_i and νc′=νc+∑iqic\nu'_c=\nu_c+\sum_i q_{ic} for each new class c∈Ytc\in\mathcal{Y}_t and hc′=hch'_c=h_c and νc′=νc\nu'_c=\nu_c for earlier classes. They give xˉ=g′/N′\bar{x}=g'/N', the directions VV as the top eigenvectors of G′/N′−xˉxˉ⊤G'/N'-\bar{x}\bar{x}^{\top}, and

Σw∝G′−∑c:νc′>0hc′hc′⊤νc′,(7)\Sigma_w\propto G'-\sum_{c:\nu'_c>0}\frac{h'_c h_c^{\prime\top}}{\nu'_c}, \tag*{(7)}

the soft within-class scatter of all seen images, whose scale cancels in OAS and in Eq. (6), so no earlier image is stored.

Fixed-prior vMF mixture. On this sphere each class c∈Ytc\in\mathcal{Y}_t is a von Mises–Fisher (vMF) component with density proportional to exp⁡(κμc⊤x~)\exp(\kappa\mu_c^{\top}\widetilde{x}), mean direction μc∈Sdp−1\mu_c\in\mathbb{S}^{d_p-1} and a concentration κ>0\kappa>0 shared by all classes. The shared κ\kappa cancels the normalizers, so the likelihood has no class-specific term, unlike a shared-covariance Gaussian whose term −12∥μc∥2-\frac{1}{2}\lVert\mu_c\rVert^2 favors classes with short means. For a prior P∈{Pbal,Ptsel}P\in\{P_{\mathrm{bal}},P_t^{\mathrm{sel}}\}, the mixing weights of image ii are fixed to Pi⋅P_{i\cdot}. The class proportions chosen in § 3.2 enter every E-step and are never re-estimated. Starting from qi⋅=Pi⋅q_{i\cdot}=P_{i\cdot}, EM alternates between an M-step for the mean directions μc\mu_c and the concentration κ\kappa, and an E-step for the soft labels,

μc=nrm⁡(∑iqicx~i),κ=rˉ(dp−rˉ2)1−rˉ2,\mu_c=\operatorname{nrm}\left(\sum_i q_{ic}\widetilde{x}_i\right),\qquad \kappa=\frac{\bar{r}(d_p-\bar{r}^{2})}{1-\bar{r}^{2}},
qic=Picexp⁡(κμc⊤x~i/τ)∑c′∈YtPic′exp⁡(κμc′⊤x~i/τ),(8)q_{ic}=\frac{P_{ic}\exp(\kappa\mu_c^{\top}\widetilde{x}_i/\tau)} {\sum_{c'\in\mathcal{Y}_t}P_{ic'}\exp(\kappa\mu_{c'}^{\top}\widetilde{x}_i/\tau)}, \tag*{(8)}

where rˉ=1nt∑c∥∑iqicx~i∥∈[0,1)\bar{r}=\frac{1}{n_t}\sum_c\left\lVert\sum_i q_{ic}\widetilde{x}_i\right\rVert\in[0,1) is the mean resultant length and κ\kappa is the approximation of [54] to its maximum-likelihood value. The E-step has the form of the assignment step of TransCLIP [38] without its graph term. The temperature τ≥1\tau\geq1 tempers the likelihood, so that small cosine gaps, scaled by the fitted κ\kappa, cannot override the text prior, and τ=1\tau=1 gives the untempered posterior. EM runs in two passes, the second recomputing Σw\Sigma_w, LL and x~i\widetilde{x}_i from the first-pass posterior with xˉ\bar{x} and VV fixed, since the prior alone gives only a coarse within-class estimate. This defines q=EM⁡(P,τ)q=\operatorname{EM}(P,\tau). The soft labels of task tt are q=EM⁡(Ptsel,τ)q=\operatorname{EM}(P_t^{\mathrm{sel}},\tau), after which (xi,qi)(x_i,q_i) are added to (G,g,N,H,ν)(G,g,N,H,\nu) and never revised.

Soft-Label Analytic Learning and Fusion

The soft labels of a task are used twice before they are discarded. They first train the analytic classifier and then, in place of labels, select the weight that fuses it with zero-shot CLIP in predictions over all seen classes.

Soft-label analytic learning. Following Eq. (3), the target of image ii is its soft label padded with zeros on the old classes, Qi=(0Kt−1⊤,qi⊤)⊤Q_i=(\mathbf{0}_{K_{t-1}}^{\top},q_i^{\top})^{\top}. For any output o∈RKto\in\mathbb{R}^{K_t}, Ey∼Qi∥o−ey∥2=∥o−Qi∥2+1−∥Qi∥2\mathbb{E}_{y\sim Q_i}\lVert o-e_y\rVert^2=\lVert o-Q_i\rVert^2+1-\lVert Q_i\rVert^2, so the squared error of Eq. (2) fits the label distribution rather than its argmax. To keep large classes from dominating, the weights balance classes by their soft counts n^c=∑iqic\hat{n}_c=\sum_iq_{ic},

mi=∑c∈Ytqicfc,fc=ntKeffmax⁡(n^c,1),(9)m_i=\sum_{c\in\mathcal{Y}_t}q_{ic}f_c,\qquad f_c=\frac{n_t}{K_{\mathrm{eff}}\max(\hat{n}_c,1)}, \tag*{(9)}

where Keff=max⁡(∣{c:n^c≥1}∣,1)K_{\mathrm{eff}}=\max(\lvert\{c:\hat{n}_c\geq1\}\rvert,1) is the number of classes with at least unit mass, and fcf_c equals the class-balancing factor of N2L for one-hot labels.

Fusion weight selection. At test time, PACIL predicts the class in Y1:t\mathcal{Y}_{1:t} with the largest fused score

Ft(x)=(1−ωt)pzs(x)+ωtWt⊤ϕ(x),(10)F_t(x)=(1-\omega_t)p^{\mathrm{zs}}(x)+\omega_t W_t^{\top}\phi(x), \tag*{(10)}

where pzs(x)∈ΔKt−1p^{\mathrm{zs}}(x)\in\Delta^{K_t-1} is the zero-shot distribution over all seen classes with each class embedding averaged over all prompt templates, and ωt∈[0,1]\omega_t\in[0,1] weights the classifier. The right weight changes across datasets and tasks. Choosing it requires predictions that, like test predictions, come from a classifier that has not seen the image. The fitted output ri=Wt⊤ϕir_i=W_t^{\top}\phi_i does not qualify, since each image has pulled WtW_t toward its own target. PACIL instead uses the closed-form LOO output of ridge regression [55, 56],

r~i=ri−hiQi1−hi,hi=miϕi⊤(At+λID)−1ϕi,(11)\widetilde{r}_i=\frac{r_i-h_iQ_i}{1-h_i},\qquad h_i=m_i\phi_i^{\top}(A_t+\lambda I_D)^{-1}\phi_i, \tag*{(11)}

where hi∈[0,1)h_i\in[0,1) is the leverage of image ii and r~i∈RKt\widetilde{r}_i\in\mathbb{R}^{K_t} is the output at xix_i of the classifier re-solved with $m_i=0. After training on task tt, the soft labels score the fused LOO decisions in place of the labels,

c^i(ω)=arg⁡max⁡c∈Y1:t[(1−ω)pczs(xi)+ωr~ic],ωt=arg⁡max⁡ω∈Ω∑i∈DtQi,c^i(ω).(12)\begin{aligned} \hat{c}_i(\omega) &= \arg\max_{c \in\mathcal{Y}_{1:t}} \left[(1-\omega)p_c^{\mathrm{zs}}(x_i)+\omega\tilde{r}_{ic}\right],\\ \omega_t &= \arg\max_{\omega\in\Omega}\sum_{i\in D_t} Q_{i,\hat{c}_i(\omega)}. \tag*{(12)} \end{aligned}

where pczs(xi)p_c^{\mathrm{zs}}(x_i) is the softmax of ℓic\ell_{ic} over all seen classes c∈Y1:tc \in\mathcal{Y}_{1:t} and Ω={0,0.01,…,1}\Omega= \{0, 0.01, \ldots, 1\}, with ties resolved by their median. Each decision earns the probability that its soft label gives to the chosen class, so the criterion counts the correct decisions expected under the soft labels, and a decision for an old class earns nothing. Across tasks, PACIL stores only (G,g,N,H,ν)(G, g, N, H, \nu) and (At,Ct)(A_t, C_t), whose size does not grow with the number of seen images, and Algorithm 1 in Supp. B lists every step.

Theoretical analysis. Supp. A justifies both uses of the soft labels, the first in squared error and the second in expected correct decisions. As targets, the posteriors minimize the mean-square gap to the true-label prediction, and any other targets add at most a multiple of their squared distance to them. Their argmax adds exactly the squared change it causes in the prediction. PACIL therefore trains on soft labels, and § 3.2 and § 3.3 aim to bring them close to the posteriors. In place of labels, the soft labels make the criterion of Eq. (12) the expected number of correct LOO decisions up to their error, which bounds what ωt\omega_t can lose.

Experiments

We first describe the setup (§ 4.1), then compare PACIL with exemplar-free methods on six datasets (§ 4.2) and ablate its three modules (§ 4.3). The supplement compares PACIL on Food, ImageNet-R and SUN and with exemplar-based methods (Supp. C.1), tests its robustness to τ\tau, λ\lambda, the CLIP backbone (Supp. C.2) and noisy task boundaries (Supp. C.3), gives N2L two components of PACIL (Supp. C.4) and PACIL the soft labels of transductive methods (Supp. C.5), and reports pseudo-label quality, per-task accuracy, and training and inference cost (Supp. C.6 to C.8).

Experimental Setup

Datasets. We use the nine datasets with the class subsets and train/test splits adopted from N2L [7], PROOF [24] and ENGINE [14]. They cover fine-grained categories in Aircraft [57], Cars [58], CUB [59] and Food [60], generic objects in CIFAR100 [61], distribution shift in ImageNet-R [62] and ObjectNet [63], human actions in UCF [64] and scenes in SUN [65]. SUN has 300 classes, CUB, ObjectNet and ImageNet-R have 200, and the others have 100. BbInckB b I n c k denotes a first task of bb classes followed by tasks of kk classes, with b=0b = 0 meaning that the first task also has kk classes. The zero-base setting splits all classes evenly into 10 tasks (B0Inc10, B0Inc20 or B0Inc30 for 100, 200 or 300 classes). The half-base setting puts half of the classes in the first task and splits the other half evenly into 5 tasks (B50Inc10, B100Inc20 or B150Inc30). Each task, including the first, provides only the names of its new classes and their unlabeled training images.

Evaluation Metrics. Following N2L, after each task tt the model classifies the test images of all classes seen so far without task identity. Its accuracy is At=1∣T1:t∣∑(x,y)∈T1:tI[y^t(x)=y]A_t = \frac{1}{|\mathcal{T}_{1:t}|}\sum_{(x,y)\in\mathcal{T}_{1:t}}\mathbb{I}[\hat{y}_t(x)=y], where T1:t\mathcal{T}_{1:t} is the set of test images xx with labels yy in the seen classes Y1:t\mathcal{Y}_{1:t}, y^t(x)∈Y1:t\hat{y}_t(x) \in\mathcal{Y}_{1:t} is the class predicted after task tt, and the indicator I[⋅]\mathbb{I}[\cdot] is 1 if its argument holds and 0 otherwise. We report the last accuracy ATA_T after the final task TT and the average incremental accuracy Aˉ=1T∑t=1TAt\bar{A} = \frac{1}{T}\sum_{t=1}^{T} A_t, both in percent.

Comparison Methods. We compare with the exemplar-free methods evaluated by N2L. ZS-CLIP [1] classifies by class names without training. CODA-Prompt [66], MoE-Adapter [22], RAPF [23], ENGINE [14] and RAIL [13] are designed for labeled tasks, so N2L trains them on the argmax of the zero-shot prediction of each image as its pseudo-label. CODA-Prompt tunes prompts of the image encoder, MoE-Adapter, RAPF and ENGINE (without its re-ranking) adapt CLIP by gradient descent, and RAIL fits an analytic classifier. N2L [7] starts from the same pseudo-labels, refines them in three iterations with a ridge classifier on dimension-reduced features, and weights each image by its class frequency and confidence. We also compare with the exemplar-based PROOF [24] and CLAP4CLIP [67], which store 20 images per class of earlier tasks (Supp. C.1). N2L+Label and PACIL+Label replace the pseudo-labels of N2L and the soft labels of PACIL by ground-truth labels, and serve as reference upper bounds that are not ranked.

Implementation Details. PACIL uses the frozen CLIP ViT-B/16 of N2L, pre-trained on LAION-400M [68], and its random features with D=5000D = 5000. As in N2L, class names are embedded with the first prompt template of each dataset during training and with the ensemble of all its templates at test time. All datasets share one hyperparameter setting, τ=4\tau= 4, λ=10\lambda= 10 and dp=128d_p = 128, and after training on each task, ωt\omega_t is selected from {0,0.01,…,1}\{0, 0.01, \ldots, 1\} by the closed-form LOO criterion of Eq. (12). Images are encoded in batches of 64. Each result is the mean of three runs with seeds 1993, 1994 and 1995 on one NVIDIA A100 GPU with PyTorch. The best and second-best results in each column are marked in bold and _underlined_, respectively. We will release the code upon the manuscript acceptance.

Main Results

Table 1 covers the six datasets on which N2L evaluates all baselines. Among the methods trained without labels, PACIL has the best Aˉ\bar{A} on all 12 settings and the best ATA_T on 11 of them. N2L is the strongest compared method throughout, and in the Mean columns PACIL improves on it by 3.19% and 2.49% in Aˉ\bar{A} on the two blocks and by 4.00% and 1.80% in ATA_T, the latter held down by ObjectNet, with Aˉ\bar{A} gains of 2.83% and 2.85% on average in the zero-base and half-base settings. The gains over N2L are largest on Aircraft, 7.62% and 7.06% in Aˉ\bar{A} and 9.49% and 7.38% in ATA_T on B0Inc10 and B50Inc10. On this dataset zero-shot CLIP is the weakest, with an ATA_T of 17.16%, and ground-truth labels raise the Aˉ\bar{A} of N2L the most, by 22.65% and 29.49%. The pseudo-labels of N2L are least reliable here. This is the case that motivates PACIL in Sec. 1, since the zero-shot distribution keeps plausible classes that the argmax discards. PACIL falls behind only in ATA_T on ObjectNet B100Inc20, 0.87% below N2L, and on ObjectNet B0Inc20 it gains 3.75% in Aˉ\bar{A} but 0.03% in ATA_T. ObjectNet has the second weakest zero-shot CLIP and varies rotation, background and viewpoint by design, so both the zero-shot prior and the feature clusters used by EM are weak, and the early gain of PACIL fades by the last task.

MethodVenueCUB B0Inc20 Aˉ\bar{A}CUB B0Inc20 ATA_TCUB B100Inc20 Aˉ\bar{A}CUB B100Inc20 ATA_TObjectNet B0Inc20 Aˉ\bar{A}ObjectNet B0Inc20 ATA_TObjectNet B100Inc20 Aˉ\bar{A}ObjectNet B100Inc20 ATA_TUCF B0Inc10 Aˉ\bar{A}UCF B0Inc10 ATA_TUCF B50Inc10 Aˉ\bar{A}UCF B50Inc10 ATA_TMean Aˉ\bar{A}Mean ATA_T
ZS-CLIPICML’2175.4763.7269.0663.7238.4326.4331.1226.4375.8867.7971.6867.7960.2752.65
CODA-PromptCVPR’2367.4051.9153.2437.6438.2526.2326.2219.2281.6175.1867.0658.8755.6344.84
MoE-AdapterCVPR’2473.6963.0768.2963.0940.1428.7032.4528.5375.3567.3672.2869.0760.3753.30
RAPFECCV’2472.0858.1462.2756.0239.1625.8828.9324.7085.9981.1075.7074.3160.6953.36
RAILNeurIPS’2481.6473.9373.0870.9139.8035.1331.2130.1990.1889.9081.0584.2766.1664.06
ENGINEICCV’2577.0665.0770.8564.9244.5731.2434.6229.9687.8584.4682.3081.0466.2159.45
N2LICLR’2683.4176.4875.1673.4049.3141.5941.4238.6595.0093.2986.4187.8771.7968.55
PACILOurs86.7180.3878.7677.2153.0641.6241.8837.7895.9294.7289.3590.4074.2870.35
N2L+LabelICLR’2686.4379.0582.2979.0653.1845.2749.7145.2698.7497.7598.3797.7578.1274.02
PACIL+LabelOurs87.4381.2383.8981.2458.8946.1951.6746.1598.6797.4098.2297.3679.8074.93

Table 1 (PDF p. 7). Comparison with exemplar-free methods on six datasets. Aˉ\bar{A} and ATA_T in %, and Mean averages the six settings of each block.

The +Label rows measure the cost of the missing labels. PACIL+Label is above PACIL by 5.29% and 9.79% in Aˉ\bar{A} on average in the zero-base and half-base settings, and by 3.81% and 7.76% in ATA_T. PACIL+Label reaches nearly the same ATA_T in both settings, whereas PACIL is lower in the half-base setting on all six datasets. Its first task must then be labeled as a 50-way or 100-way problem instead of a 10-way or 20-way one, and labels that enter the classifier are never corrected. The gap is largest on Aircraft, 20.04% and 28.68% in Aˉ\bar{A}, and below 1% in both metrics on CIFAR100 B0Inc10 and CUB B0Inc20, where PACIL is also above N2L+Label. PACIL+Label is in turn above N2L+Label in both metrics on 10 of the 12 settings, all except UCF.

Ablation Study

Class-Prior Correction. We replace the prior selection rule of Eq. (5) by PrawP^{\mathrm{raw}} or PbalP^{\mathrm{bal}} on every task (PrawP^{\mathrm{raw}} only, PbalP^{\mathrm{bal}} only). Each fails on a different dataset in Figure 3. PrawP^{\mathrm{raw}} only loses 5.26% in Aˉ\bar{A} on Aircraft B0Inc10, where the classes are balanced and the skew of the zero-shot marginal is name bias. PbalP^{\mathrm{bal}} only loses 4.07% on SUN B0Inc30, where the classes differ in size and debiasing erases a real imbalance. The prior selection rule matches the better fixed prior on both, and on UCF B0Inc10 it switches priors across tasks and exceeds both. Its gain over the better fixed prior is small, however, none on Aircraft and SUN and 0.18% on UCF, so the rule mainly guards against a wrong fixed prior.

Figure 3. Ablation of class-prior correction. Aˉ\bar{A} (%) on the zero-base settings of Aircraft, SUN and UCF with PrawP^{\mathrm{raw}} or PbalP^{\mathrm{bal}} fixed on every task and with the prior selection rule of PACIL.

Posterior Label-Distribution Estimation. Keeping the prior selection rule, we change only how PtselP_t^{\mathrm{sel}} becomes the soft labels. Hard labels replaces the label posteriors with hard pseudo-labels, w/o EM trains directly on qi=Piselq_i=P_i^{\mathrm{sel}}, and w/o whitening runs EM with L=IL=I in Eq. (6). All three are below PACIL in both metrics on the eight settings of Figure 4. Removing EM costs up to 4.18% in Aˉ\bar{A} on UCF B50Inc10, in line with the cluster assumption that the images of a task correct each other’s distributions (§ 3.3). Without whitening, EM loses up to 2.41% on ObjectNet B100Inc20, whose images vary in pose and background, but about 0.2% on UCF B50Inc10 and Aircraft B0Inc10. Taking the hard labels lowers Aˉ\bar{A} further on all eight settings, most on Aircraft B0Inc10 with 1.99%, where the distribution keeps plausible classes that the weak zero-shot argmax misses. Keeping the whole distribution helps before EM is run, and EM then brings it closer to the posterior of each label. In ATA_T, the argmax is better than w/o EM on both CUB settings, by 0.95% and 0.54%.

Figure 4. Ablation of posterior label-distribution estimation. Change in (a) Aˉ\bar{A} and (b) ATA_T (%) relative to PACIL when hard pseudo-labels replace the label posteriors (Hard labels), the final EM is skipped (w/o EM) or run with L=IL=I (w/o whitening), on the zero-base (Zero) and half-base (Half) settings of four datasets.

Fusion Weight Selection. We fix ωt=0.8\omega_t=0.8 on every task as N2L does, which leaves the classifier unchanged. Selecting ωt\omega_t by Eq. (12) is better on every setting, by 0.71% in Aˉ\bar{A} and 1.16% in ATA_T on average (Figure 5). Across the 144 tasks, the selected ωt\omega_t ranges from 0.72 to 1.00 with a mean of 0.90, and it is above the fixed 0.8 of N2L on 134 of them (Figure 5c). The gain is small on ImageNet-R, 0.16% and 0.14% in Aˉ\bar{A}, whose mean ωt\omega_t of about 0.84 and 0.86 is nearest 0.8. It is largest on ObjectNet, 3.62% and 2.48% in Aˉ\bar{A}, though its mean ωt\omega_t is near 0.90, so ObjectNet is more sensitive to the weight than in need of a different one.

Figure 5. Ablation of fusion weight selection. (a, b) Gain of PACIL over the fixed ωt=0.8\omega_t=0.8 of N2L in Aˉ\bar{A} and ATA_T (%) on all 18 settings. (c) ωt\omega_t selected on each task (dots, mean over three runs) and its mean per setting (ticks). Bottom rows give the means over all settings, and in (c) the line spans the range of ωt\omega_t.

Conclusion

When class-incremental learning has only class names and unlabeled images, each image should carry its label posterior as its training target. An analytic classifier is linear in its targets, so training it on posteriors recovers the true-label classifier in expectation. PACIL estimates them with a prior selection rule and a fixed-prior vMF mixture on whitened features, then uses the soft labels as targets and to select the fusion weight by closed-form LOO. In our analysis, no targets beat the posteriors in mean square, and rounding them to hard labels adds error only through ambiguous images. The error of the soft labels bounds the loss of the selected weight. PACIL leads methods trained without labels in average incremental accuracy, most clearly where zero-shot CLIP is weakest, and its lead over N2L persists across six CLIP backbones. Its corrections are limited by how well CLIP relates class names to images, and a large gap to labeled training remains. Language descriptions or queried labels [69] could narrow this gap, and future work should also admit unlabeled images from unknown classes.

Theoretical Analysis

Without labels, PACIL must decide what its analytic classifier is trained on and how much its prediction trusts this classifier. Theorem S1 answers both questions for the prediction that PACIL deploys after each task, and Lemma S1 shows that the leave-one-out (LOO) outputs that select the fusion weight come from classifiers that have not seen the image, just as test predictions do. Section A.1 formulates the problem and states both, Sec. A.2 discusses what they imply for PACIL, and Sec. A.3 proves them.

Problem Formulation and Main Theorem

After task tt, every seen image jj has its features ϕj\phi_j, its weight mj>0m_j > 0 and its soft label QjQ_j, padded with zeros on the other classes of Y1:t\mathcal{Y}_{1:t} and frozen after its task. For targets Θ=(Θj)j\Theta= (\Theta_j)_j, one vector Θj∈RKt\Theta_j \in\mathbb{R}^{K_t} per seen image, the classifier fitted to them is

W(Θ)=(At+λID)−1∑jmjϕjΘj⊤,(13)W(\Theta) = (A_t + \lambda I_D)^{-1} \sum_j m_j \phi_j \Theta_j^\top, \tag*{(13)}

and the classifier of Eq. (2) is Wt=W(Q)W_t = W(Q) by Step 1 of the proof. Given everything observed, the only unknowns are the true labels. For any joint law of images and labels, we therefore condition on the σ\sigma-algebra Ft\mathcal{F}_t generated by the observations O\mathcal{O} after task tt and by RR, and write Et\mathbb{E}_t for the expectation given Ft\mathcal{F}_t. Every quantity that PACIL computes, such as QQ, the weights, the zero-shot scores and ωt\omega_t, is then fixed, while the label of image jj has the posterior

eˉj=Et[eyj],eˉjc=Pr⁡(yj=c∣Ft).(14)\bar e_j = \mathbb{E}_t[e_{y_j}], \qquad\bar e_{jc} = \Pr(y_j = c \mid\mathcal{F}_t). \tag*{(14)}

With the classifier fitted to Θ\Theta, Eq. (10) predicts

FΘ(x)=(1−ωt)pˉzs(x)+ωtW(Θ)⊤ϕ(x),(15)F_\Theta(x) = (1-\omega_t)\bar p^{\mathrm{zs}}(x) + \omega_t W(\Theta)^\top\phi(x), \tag*{(15)}

so PACIL predicts with Ft=FQF_t = F_Q, and the true labels ey=(eyj)je_y = (e_{y_j})_j would give FeyF_{e_y} with the same mjm_j and ωt\omega_t. We compare targets by their risk Rx(Θ)R_x(\Theta) at an input xx, and write st(x)s_t(x) for the leverage of xx and σt2(x)\sigma_t^2(x) for the conditional variance of the output that the true labels would give, which no choice of targets affects,

Rx(Θ)=Et∥FΘ(x)−Fey(x)∥2,st(x)=ϕ(x)⊤(At+λID)−1ϕ(x),σt2(x)=Et∥(W(ey)−W(eˉ))⊤ϕ(x)∥2.(16)\begin{aligned} R_x(\Theta) &= \mathbb{E}_t\|F_\Theta(x)-F_{e_y}(x)\|^2,\\ s_t(x) &= \phi(x)^\top(A_t+\lambda I_D)^{-1}\phi(x),\\ \sigma_t^2(x) &= \mathbb{E}_t\|(W(e_y)-W(\bar e))^\top\phi(x)\|^2. \tag*{(16)} \end{aligned}

For the fusion weight, let c^i(ω)\hat c_i(\omega) be the decision of Eq. (12) for a current image ii, so that ωt\omega_t maximizes the criterion J(ω)J(\omega) and Ncor(ω)N_{\mathrm{cor}}(\omega) counts the correct decisions, where

J(ω)=∑i∈DtQi,c^i(ω),Ncor(ω)=∑i∈DtI[c^i(ω)=yi].(17)\begin{aligned} J(\omega) &= \sum_{i\in D_t} Q_{i,\hat c_i(\omega)},\\ N_{\mathrm{cor}}(\omega) &= \sum_{i\in D_t}\mathbb{I}[\hat c_i(\omega)=y_i]. \tag*{(17)} \end{aligned}

Let ω⋆∈Ω\omega^\star\in\Omega maximize Et[Ncor(ω)]\mathbb{E}_t[N_{\mathrm{cor}}(\omega)], and let SS hold the current images that ωt\omega_t and ω⋆\omega^\star decide differently.

Theorem S1 (Keep the posterior, not the argmax). (a) The posteriors give the conditional mean, Et[Fey(x)]=Feˉ(x)\mathbb{E}_t[F_{e_y}(x)] = F_{\bar e}(x), and for every input xx and all Ft\mathcal{F}_t-measurable targets Θ\Theta,

Rx(Θ)=ωt2(∥(W(Θ)−W(eˉ))⊤ϕ(x)∥2+σt2(x))≤ωt2(σt2(x)+st(x)∑jmj∥Θj−eˉj∥2),(18)R_x(\Theta) = \omega_t^2\left(\| (W(\Theta)-W(\bar e))^\top\phi(x)\|^2+\sigma_t^2(x)\right) \le\omega_t^2\left(\sigma_t^2(x)+s_t(x)\sum_j m_j\|\Theta_j-\bar e_j\|^2\right), \tag*{(18)}

with st(x)≤∥ϕ(x)∥2/λs_t(x)\le\|\phi(x)\|^2/\lambda, so the posteriors attain the smallest risk, Rx(eˉ)=ωt2σt2(x)R_x(\bar e)=\omega_t^2\sigma_t^2(x). (b) Let Q^ij=ecj\hat Q_{ij}=e_{c_j}, with cj∈arg⁡max⁡cQjcc_j\in\arg\max_c Q_{jc} be the hard label of image jj, and let

δx=(W(Q^)−W(Q))⊤ϕ(x),\delta_x=(W(\hat Q)-W(Q))^\top\phi(x),
ϵx=∥(W(Q)−W(eˉ))⊤ϕ(x)∥(19)\epsilon_x=\|(W(Q)-W(\bar e))^\top\phi(x)\| \tag*{(19)}

be the change of the output that the argmax causes, which needs no label, and the error of the soft labels at xx. Then

∣Rx(Q^)−Rx(Q)−ωt2∥δx∥2∣≤2ωt2∥δx∥ϵx,|R_x(\hat Q)-R_x(Q)-\omega_t^2\|\delta_x\|^2|\le2\omega_t^2\|\delta_x\|\epsilon_x,
ϵx2≤st(x)∑jmj∥Qj−eˉj∥2,(20)\epsilon_x^2\le s_t(x)\sum_jm_j\|Q_j-\bar e_j\|^2, \tag*{(20)}

so Rx(Q^)−Rx(Q)=ωt2∥δx∥2R_x(\hat Q)-R_x(Q)=\omega_t^2\|\delta_x\|^2 when Q=eˉQ=\bar e. (c) For every ω∈Ω\omega\in\Omega, the criterion splits as

J(ω)=Et[Ncor(ω)]+∑i∈Dt(Qi−eˉi)c^i(ω),(21)J(\omega)=\mathbb{E}_t[N_{\mathrm{cor}}(\omega)]+\sum_{i\in D_t}(Q_i-\bar e_i)_{\hat c_i(\omega)}, \tag*{(21)}

and the selected weight satisfies

0≤Et[Ncor(ω⋆)−Ncor(ωt)]≤∑i∈S∥Qi−eˉi∥1.(22)0\le\mathbb{E}_t[N_{\mathrm{cor}}(\omega^\star)-N_{\mathrm{cor}}(\omega_t)] \le\sum_{i\in S}\|Q_i-\bar e_i\|_1. \tag*{(22)}

Lemma S1 (Closed-form LOO). For λ>0\lambda>0, the leverage hih_i of Eq. (11) lies in [0,1)[0,1), and r~i\tilde r_i is the output at xix_i of the classifier of Eq. (2) re-solved with mi=0m_i=0, all other seen images keeping their weights and targets.

Discussion

By (a), the posteriors are the targets that bring the deployed prediction closest, in mean square, to the one that the true labels would give, and other targets add at most ωt2st(x)\omega_t^2 s_t(x) times their weighted squared distance to the posteriors. The sum runs over all seen images, and a soft label is frozen after its task while its image is discarded, so an error made on one task stays in the bound for every later task. PACIL therefore estimates the posteriors, with the prior selection rule of § 3.2 and the EM of § 3.3, before it freezes the soft labels for all later tasks. By (b), if the soft labels are the posteriors, replacing them by their argmax raises the risk by exactly ωt2∥δx∥2\omega_t^2 \lVert\delta_x\rVert^2, the squared change of the prediction. If they are estimates of the posteriors, the argmax raises the risk whenever ωt>0\omega_t > 0 and ϵx<∥δx∥/2\epsilon_x < \lVert\delta_x\rVert/2. The argmax discards the ambiguity of each posterior, since a one-hot target stays at squared distance

∥ec−eˉj∥2=1−2eˉjc+∥eˉj∥2≥(1−eˉjc)2(23)\lVert e_c-\bar e_j\rVert^2 = 1-2\bar e_{jc}+\lVert\bar e_j\rVert^2 \ge(1-\bar e_{jc})^2 \tag*{(23)}

from the posterior for every class cc, which is large on ambiguous images, whereas a soft label can match the posterior. Section C.6 measures how far the soft labels of PACIL and the hard pseudo-labels of zero-shot CLIP [1] and N2L [7] are from the true labels.

By Lemma S1, the LOO output of an image is the output of a classifier that has not seen it, like a test prediction, and costs no re-solving. By (c), the criterion of Eq. (12) is the expected number of correct LOO decisions plus an error term that vanishes when the soft labels are the posteriors. Against the weight ω⋆\omega^\star with the most correct decisions in expectation, the selected weight then loses at most the L1L_1 distance of the soft labels to the posteriors, summed over the images that the two weights decide differently.

Parts (a) and (b) compare predictions in squared error and part (c) counts correct LOO decisions on the current images, so none of them bounds the test accuracy, and the argmax can be more accurate, as for ATA_T on CUB in Figure 4. The results hold for any soft labels computed from the observations, including those of the prior selection rule and of EM with τ=4\tau= 4. They state what these modules should aim at, not that they reach it. The reference prediction FeyF_{e_y} keeps the weights mjm_j that Eq. (9) computes from the soft labels. The LOO of Lemma S1 removes an image only from the ridge step, since the soft labels and weights computed with this image are kept.

Proofs

Proof of Theorem S1. Given FtF_t, the labels follow a probability vector over the finitely many label configurations, EtE_t is the expectation under this vector, and every FtF_t-measurable quantity is a constant under it. We write Bt=At+λIDB_t=A_t+\lambda I_D and ϕj=ϕ(xj)\phi_j=\phi(x_j), and jj runs over all seen images.

Step 1: the recursion is ridge regression on all seen images. By induction on tt, the recursion of Eq. (2) gives

At=∑jmjϕjϕjT,Ct=∑jmjϕjQjT.(24)A_t=\sum_j m_j\phi_j\phi_j^{\mathsf T},\qquad C_t=\sum_j m_j\phi_jQ_j^{\mathsf T}. \tag*{(24)}

Written over all classes, Ct−1C_{t-1} has zero columns for the classes of task tt, as every soft label is zero outside the classes of its own task, so the zero block of Eq. (2) loses nothing. Hence Wt=Bt−1Ct=W(Q)W_t=B_t^{-1}C_t=W(Q), although no seen image is stored once its task ends.

Step 2: conditional mean and risk. W(Θ)W(\Theta) is linear in Θ\Theta with the Ft\mathcal{F}_t-measurable coefficients mjBt−1ϕjm_jB_t^{-1}\phi_j, so Et[W(ey)]=W(eˉ)E_t[W(e_y)]=W(\bar e), and Et[Fey(x)]=Feˉ(x)E_t[F_{e_y}(x)]=F_{\bar e}(x) because ωt\omega_t and ptz(x)p_t^z(x) are constants. With o=W(Θ)Tϕ(x)o=W(\Theta)^{\mathsf T}\phi(x) and o⋆=W(ey)Tϕ(x)o^\star=W(e_y)^{\mathsf T}\phi(x), the zero-shot term cancels in

FΘ(x)−Fey(x)=ωt(o−o⋆).(25)F_\Theta(x)-F_{e_y}(x)=\omega_t(o-o^\star). \tag*{(25)}

As oˉ=W(eˉ)Tϕ(x)=Et[o⋆]\bar o=W(\bar e)^{\mathsf T}\phi(x)=E_t[o^\star], the cross term of

∥o−o⋆∥2=∥o−oˉ∥2+∥oˉ−o⋆∥2+2(o−oˉ)T(oˉ−o⋆)(26)\lVert o-o^\star\rVert^2=\lVert o-\bar o\rVert^2+\lVert\bar o-o^\star\rVert^2+2(o-\bar o)^{\mathsf T}(\bar o-o^\star) \tag*{(26)}

has conditional mean zero, so

Rx(Θ)=ωt2(∥o−oˉ∥2+σt2(x)),(27)R_x(\Theta)=\omega_t^2\left(\lVert o-\bar o\rVert^2+\sigma_t^2(x)\right), \tag*{(27)}

which is the identity of (a) and is smallest at Θ=eˉ\Theta=\bar e.

Step 3: target errors pass through the leverage. As BtB_t is symmetric, the target errors enter the output at xx as

o−oˉ=∑jmjaj(Θj−eˉj),aj=ϕjTb,(28)o-\bar o=\sum_j m_j a_j(\Theta_j-\bar e_j),\qquad a_j=\phi_j^{\mathsf T}b, \tag*{(28)}

where b=Bt−1ϕ(x)b=B_t^{-1}\phi(x), so that st(x)=ϕ(x)Tbs_t(x)=\phi(x)^{\mathsf T}b. The triangle and Cauchy–Schwarz inequalities give

∥o−oˉ∥2≤(∑jmjaj2)(∑jmj∥Θj−eˉj∥2).(29)\lVert o-\bar o\rVert^2\le\left(\sum_j m_j a_j^2\right)\left(\sum_j m_j\lVert\Theta_j-\bar e_j\rVert^2\right). \tag*{(29)}

The inner product of Btb=ϕ(x)B_tb=\phi(x) with bb gives

st(x)=∑jmjaj2+λ∥b∥2,(30)s_t(x)=\sum_j m_j a_j^2+\lambda\lVert b\rVert^2, \tag*{(30)}

so ∑jmjaj2≤st(x)\sum_j m_j a_j^2\le s_t(x), which proves the upper bound of (a). The same identity gives λ∥b∥2≤st(x)\lambda\lVert b\rVert^2\le s_t(x), and Cauchy–Schwarz gives st(x)2≤∥ϕ(x)∥2∥b∥2s_t(x)^2\le\lVert\phi(x)\rVert^2\lVert b\rVert^2, so

λst(x)2≤∥ϕ(x)∥2λ∥b∥2≤∥ϕ(x)∥2st(x),(31)\lambda s_t(x)^2\le\lVert\phi(x)\rVert^2\lambda\lVert b\rVert^2\le\lVert\phi(x)\rVert^2s_t(x), \tag*{(31)}

and st(x)≤∥ϕ(x)∥2/λs_t(x)\le\lVert\phi(x)\rVert^2/\lambda as st(x)>0s_t(x)>0.

Step 4: posterior versus argmax. The error of the soft labels at xx is ϵx=∥ζ∥\epsilon_x=\lVert\zeta\rVert with ζ=(W(Q)−W(eˉ))Tϕ(x)\zeta=(W(Q)-W(\bar e))^{\mathsf T}\phi(x), and applying Eq. (S15) to Q^\hat Q and QQ gives

Rx(Q^)−Rx(Q)=ωt2(∥δx+ζ∥2−∥δx∥2)=ωt2(∥δx∥2+2δxTζ).(32)\begin{aligned} R_x(\hat Q)-R_x(Q)=\omega_t^2\left(\lVert\delta_x+\zeta\rVert^2-\lVert\delta_x\rVert^2\right) \\ &=\omega_t^2\left(\lVert\delta_x\rVert^2+2\delta_x^{\mathsf T}\zeta\right). \tag*{(32)} \end{aligned}

@page 13/24 starts_mid=0 ends_mid=1

Discussion

By (a), the posteriors are the targets that bring the deployed prediction closest, in mean square, to the one that the true labels would give, and other targets add at most ωt2st(x)\omega_t^2 s_t(x) times their weighted squared distance to the posteriors. The sum runs over all seen images, and a soft label is frozen after its task while its image is discarded, so an error made on one task stays in the bound for every later task. PACIL therefore estimates the posteriors, with the prior selection rule of § 3.2 and the EM of § 3.3, before it freezes the soft labels for all later tasks. By (b), if the soft labels are the posteriors, replacing them by their argmax raises the risk by exactly ωt2∥δx∥2\omega_t^2 \lVert\delta_x\rVert^2, the squared change of the prediction. If they are estimates of the posteriors, the argmax raises the risk whenever ωt>0\omega_t > 0 and ϵx<∥δx∥/2\epsilon_x < \lVert\delta_x\rVert/2. The argmax discards the ambiguity of each posterior, since a one-hot target stays at squared distance

∥ec−eˉj∥2=1−2eˉjc+∥eˉj∥2≥(1−eˉjc)2(33)\lVert e_c-\bar e_j\rVert^2 = 1-2\bar e_{jc}+\lVert\bar e_j\rVert^2 \ge(1-\bar e_{jc})^2 \tag*{(33)}

from the posterior for every class cc, which is large on ambiguous images, whereas a soft label can match the posterior. Section C.6 measures how far the soft labels of PACIL and the hard pseudo-labels of zero-shot CLIP [1] and N2L [7] are from the true labels.

By Lemma S1, the LOO output of an image is the output of a classifier that has not seen it, like a test prediction, and costs no re-solving. By (c), the criterion of Eq. (12) is the expected number of correct LOO decisions plus an error term that vanishes when the soft labels are the posteriors. Against the weight ω⋆\omega^\star with the most correct decisions in expectation, the selected weight then loses at most the L1L_1 distance of the soft labels to the posteriors, summed over the images that the two weights decide differently.

Parts (a) and (b) compare predictions in squared error and part (c) counts correct LOO decisions on the current images, so none of them bounds the test accuracy, and the argmax can be more accurate, as for ATA_T on CUB in Figure 4. The results hold for any soft labels computed from the observations, including those of the prior selection rule and of EM with τ=4\tau= 4. They state what these modules should aim at, not that they reach it. The reference prediction FeyF_{e_y} keeps the weights mjm_j that Eq. (9) computes from the soft labels. The LOO of Lemma S1 removes an image only from the ridge step, since the soft labels and weights computed with this image are kept.

Proofs

Proof of Theorem S1. Given FtF_t, the labels follow a probability vector over the finitely many label configurations, EtE_t is the expectation under this vector, and every FtF_t-measurable quantity is a constant under it. We write Bt=At+λIDB_t=A_t+\lambda I_D and ϕj=ϕ(xj)\phi_j=\phi(x_j), and jj runs over all seen images.

Step 1: the recursion is ridge regression on all seen images. By induction on tt, the recursion of Eq. (2) gives

At=∑jmjϕjϕjT,Ct=∑jmjϕjQjT.(34)A_t=\sum_j m_j\phi_j\phi_j^{\mathsf T},\qquad C_t=\sum_j m_j\phi_jQ_j^{\mathsf T}. \tag*{(34)}

Written over all classes, Ct−1C_{t-1} has zero columns for the classes of task tt, as every soft label is zero outside the classes of its own task, so the zero block of Eq. (2) loses nothing. Hence Wt=Bt−1Ct=W(Q)W_t=B_t^{-1}C_t=W(Q), although no seen image is stored once its task ends.

Step 2: conditional mean and risk. W(Θ)W(\Theta) is linear in Θ\Theta with the Ft\mathcal{F}_t-measurable coefficients mjBt−1ϕjm_jB_t^{-1}\phi_j, so Et[W(ey)]=W(eˉ)E_t[W(e_y)]=W(\bar e), and Et[Fey(x)]=Feˉ(x)E_t[F_{e_y}(x)]=F_{\bar e}(x) because ωt\omega_t and ptz(x)p_t^z(x) are constants. With o=W(Θ)Tϕ(x)o=W(\Theta)^{\mathsf T}\phi(x) and o⋆=W(ey)Tϕ(x)o^\star=W(e_y)^{\mathsf T}\phi(x), the zero-shot term cancels in

FΘ(x)−Fey(x)=ωt(o−o⋆).(35)F_\Theta(x)-F_{e_y}(x)=\omega_t(o-o^\star). \tag*{(35)}

As oˉ=W(eˉ)Tϕ(x)=Et[o⋆]\bar o=W(\bar e)^{\mathsf T}\phi(x)=E_t[o^\star], the cross term of

∥o−o⋆∥2=∥o−oˉ∥2+∥oˉ−o⋆∥2+2(o−oˉ)T(oˉ−o⋆)(36)\lVert o-o^\star\rVert^2=\lVert o-\bar o\rVert^2+\lVert\bar o-o^\star\rVert^2+2(o-\bar o)^{\mathsf T}(\bar o-o^\star) \tag*{(36)}

has conditional mean zero, so

Rx(Θ)=ωt2(∥o−oˉ∥2+σt2(x)),(37)R_x(\Theta)=\omega_t^2\left(\lVert o-\bar o\rVert^2+\sigma_t^2(x)\right), \tag*{(37)}

which is the identity of (a) and is smallest at Θ=eˉ\Theta=\bar e.

Step 3: target errors pass through the leverage. As BtB_t is symmetric, the target errors enter the output at xx as

o−oˉ=∑jmjaj(Θj−eˉj),aj=ϕjTb,(38)o-\bar o=\sum_j m_j a_j(\Theta_j-\bar e_j),\qquad a_j=\phi_j^{\mathsf T}b, \tag*{(38)}

where b=Bt−1ϕ(x)b=B_t^{-1}\phi(x), so that st(x)=ϕ(x)Tbs_t(x)=\phi(x)^{\mathsf T}b. The triangle and Cauchy–Schwarz inequalities give

∥o−oˉ∥2≤(∑jmjaj2)(∑jmj∥Θj−eˉj∥2).(39)\lVert o-\bar o\rVert^2\le\left(\sum_j m_j a_j^2\right)\left(\sum_j m_j\lVert\Theta_j-\bar e_j\rVert^2\right). \tag*{(39)}

The inner product of Btb=ϕ(x)B_tb=\phi(x) with bb gives

st(x)=∑jmjaj2+λ∥b∥2,(40)s_t(x)=\sum_j m_j a_j^2+\lambda\lVert b\rVert^2, \tag*{(40)}

so ∑jmjaj2≤st(x)\sum_j m_j a_j^2\le s_t(x), which proves the upper bound of (a). The same identity gives λ∥b∥2≤st(x)\lambda\lVert b\rVert^2\le s_t(x), and Cauchy–Schwarz gives st(x)2≤∥ϕ(x)∥2∥b∥2s_t(x)^2\le\lVert\phi(x)\rVert^2\lVert b\rVert^2, so

λst(x)2≤∥ϕ(x)∥2λ∥b∥2≤∥ϕ(x)∥2st(x),(41)\lambda s_t(x)^2\le\lVert\phi(x)\rVert^2\lambda\lVert b\rVert^2\le\lVert\phi(x)\rVert^2s_t(x), \tag*{(41)}

and st(x)≤∥ϕ(x)∥2/λs_t(x)\le\lVert\phi(x)\rVert^2/\lambda as st(x)>0s_t(x)>0.

Step 4: posterior versus argmax. The error of the soft labels at xx is ϵx=∥ζ∥\epsilon_x=\lVert\zeta\rVert with ζ=(W(Q)−W(eˉ))Tϕ(x)\zeta=(W(Q)-W(\bar e))^{\mathsf T}\phi(x), and applying Eq. (S15) to Q^\hat Q and QQ gives

Rx(Q^)−Rx(Q)=ωt2(∥δx+ζ∥2−∥δx∥2)=ωt2(∥δx∥2+2δxTζ).(42)\begin{aligned} R_x(\hat Q)-R_x(Q)=\omega_t^2\left(\lVert\delta_x+\zeta\rVert^2-\lVert\delta_x\rVert^2\right) \\ &=\omega_t^2\left(\lVert\delta_x\rVert^2+2\delta_x^{\mathsf T}\zeta\right). \tag*{(42)} \end{aligned}

and ∣δxTζ∣≤∥δx∥ϵx|\delta x^{\mathrm{T}}\zeta| \le\|\delta x\|\epsilon_x gives (b). The bound on ϵx2\epsilon_x^2 is Step 3 with Θ=Q\Theta= Q, and Q=eˉQ = \bar{e} gives ζ=0\zeta= 0. In Eq. (S11), the identity holds as ∥ec∥=1\|e_c\| = 1 and ecTeˉj=eˉjce_c^{\mathrm{T}}\bar{e}_j = \bar{e}_{jc}, and the inequality follows from ∥eˉj∥2≥eˉjc2\|\bar{e}_j\|^2 \ge\bar{e}_{jc}^2, as 1−2eˉjc+eˉjc2=(1−eˉjc)21 - 2\bar{e}_{jc} + \bar{e}_{jc}^2 = (1 - \bar{e}_{jc})^2.

Step 5: the fusion criterion. As each decision c^i(ω)\hat{c}_i(\omega) is Ft\mathcal{F}_t-measurable, it is correct with probability

eˉi,c^i(ω)=Qi,c^i(ω)−ξi,c^i(ω),ξi=Qi−eˉi,(43)\bar{e}_{i,\hat{c}_i(\omega)} = Q_{i,\hat{c}_i(\omega)} - \xi_{i,\hat{c}_i(\omega)}, \qquad\xi_i = Q_i - \bar{e}_i, \tag*{(43)}

and summing over ii gives Eq. (21). This identity at ωt\omega_t and ω⋆\omega^\star, with J(ωt)≥J(ω⋆)J(\omega_t) \ge J(\omega^\star), gives

Et[Ncor(ω⋆)−Ncor(ωt)]≤∑i∈S(ξi,c^i(ωt)−ξi,c^i(ω⋆)),(44)\mathbb{E}_t\left[N_{\mathrm{cor}}(\omega^\star) - N_{\mathrm{cor}}(\omega_t)\right] \le\sum_{i\in\mathcal{S}}\left(\xi_{i,\hat{c}_i(\omega_t)} - \xi_{i,\hat{c}_i(\omega^\star)}\right), \tag*{(44)}

since the images decided alike contribute the same term at both weights. Each summand involves two different classes aa and bb and is at most ∣ξia∣+∣ξib∣≤∥ξi∥1|\xi_{ia}|+|\xi_{ib}| \le\|\xi_i\|_1. The lower bound holds because ω⋆\omega^\star maximizes Et[Ncor(ω)]\mathbb{E}_t[N_{\mathrm{cor}}(\omega)]. □

Proof of Lemma S1. Setting the weight of image ii to zero replaces BtB_t and CtC_t by

B−=Bt−miϕiϕiT,C−=Ct−miϕiQiT,(45)B_- = B_t - m_i\phi_i\phi_i^{\mathrm{T}}, \qquad C_- = C_t - m_i\phi_i Q_i^{\mathrm{T}}, \tag*{(45)}

and B−⪰λIDB_- \succeq\lambda I_D is positive definite. Let w=B−−1ϕiw = B_-^{-1}\phi_i and h−=miϕiTw≥0h_- = m_i\phi_i^{\mathrm{T}}w \ge0. Then

Btw=(1+h−)ϕi,B_t w = (1+h_-)\phi_i,
hi=miϕiTBt−1ϕi=h−1+h−∈[0,1),(46)h_i = m_i\phi_i^{\mathrm{T}}B_t^{-1}\phi_i = \frac{h_-}{1+h_-} \in[0,1), \tag*{(46)}

since Bt−1ϕi=w/(1+h−)B_t^{-1}\phi_i = w/(1+h_-). For v=Bt−1ϕiv = B_t^{-1}\phi_i, we have B−v=(1−hi)ϕiB_-v = (1-h_i)\phi_i, hence B−−1ϕi=v/(1−hi)B_-^{-1}\phi_i = v/(1-h_i), and as B−B_- is symmetric, the re-solved classifier B−−1C−B_-^{-1}C_- outputs at xix_i

C−TB−−1ϕi=CtTv−mi(ϕiTv)Qi1−hi=ri−hiQi1−hi=r~i,(47)\begin{aligned} C_-^{\mathrm{T}}B_-^{-1}\phi_i = \frac{C_t^{\mathrm{T}}v - m_i(\phi_i^{\mathrm{T}}v)Q_i}{1-h_i} \\ &= \frac{r_i-h_iQ_i}{1-h_i} = \tilde{r}_i, \tag*{(47)} \end{aligned}

since ri=WtTϕi=CtTvr_i = W_t^{\mathrm{T}}\phi_i = C_t^{\mathrm{T}}v and hi=miϕiTvh_i = m_i\phi_i^{\mathrm{T}}v. □

More Implementation Details

This section lists the datasets and task splits of our experiments and gives the full algorithm of PACIL.

Datasets and task splits. Table S1 lists the class subsets, the image counts and the two settings of each dataset. The counts are those of the features extracted in our runs, which use the train and test splits of N2L [7].

DatasetClassesTrainTestSettings
Aircraft1006,6673,333B0Inc10, B50Inc10
Cars1004,1354,083B0Inc10, B50Inc10
CIFAR10010050,00010,000B0Inc10, B50Inc10
CUB2009,4302,358B0Inc20, B100Inc20
ObjectNet20026,5096,628B0Inc20, B100Inc20
UCF10010,5532,639B0Inc10, B50Inc10
Food10079,98820,012B0Inc10, B50Inc10
ImageNet-R20024,0006,000B0Inc20, B100Inc20
SUN30072,87018,179B0Inc30, B150Inc30

Table S1. Datasets and task splits. Train and Test are the numbers of training and test images, and BbbInckk is a first task of bb classes followed by tasks of kk classes each.

Algorithm 1 PACIL on task tt
Require: class names of Y1:t\mathcal{Y}_{1:t}, features Xt∈Rnt×dX_t \in\mathbb{R}^{n_t\times d}, statistics (G,g,N,H,ν)(G,g,N,H,\nu) and (A,C)(A,C), fixed $R
Ensure: classifier WtW_t, fusion weight ωt\omega_t, updated statistics
1: Text embeddings zcz_c from the first prompt template, c∈Y1:tc \in\mathcal{Y}_{1:t}
2: PrawP^{\mathrm{raw}} by Eq. (1), PbalP^{\mathrm{bal}} by Sinkhorn (Eq. (4))
3: qbal←EM⁡(Xt,Pbal,1)q^{\mathrm{bal}} \leftarrow\operatorname{EM}(X_t,P^{\mathrm{bal}},1)
4: ρt\rho_t from the marginals of PrawP^{\mathrm{raw}} and qbalq^{\mathrm{bal}}, select PtselP_t^{\mathrm{sel}} (Eq. (5))
5: q←EM⁡(Xt,Ptsel,τ)q \leftarrow\operatorname{EM}(X_t,P_t^{\mathrm{sel}},\tau) with τ=4\tau=4
6: Update (G,g,N,H,ν)(G,g,N,H,\nu) with XtX_t and qq
7: Φ←max⁡(XtR,0)\Phi\leftarrow\max(X_tR,0), weights mm by Eq. (9)
8: Update (A,C)(A,C) and solve WtW_t by Cholesky, λ=10\lambda=10 (Eq. (2))
9: LOO predictions (Eq. (11)), ωt\omega_t by Eq. (12) with pizp_i^{z} from zcz_c, c∈Y1:tc \in\mathcal{Y}_{1:t}
10: Discard XtX_t, Φ\Phi and qq
Inference: prompt-ensemble embeddings zˉc\bar{z}_c, c∈Y1:tc \in\mathcal{Y}_{1:t}, predict with Eq. (10)
11: function EM(X,P,τ)(X,P,\tau)
12: xˉ\bar{x} and VV (dp=128d_p=128) from the statistics merged with XX
13: qq and PP
14: for pass =1,2=1,2 do
15: Σw\Sigma_w by Eq. (7), OAS shrinkage, Cholesky factor LL, whitened x~\tilde{x}
16: Alternate the M-step and the E-step of Eq. (8) for at most 21 iterations, until the mean of log⁡Zi\log Z_i, with ZiZ_i the denominator of the E-step, changes by less than 10−510^{-5}
17: end for
18: return qq
19: end function

Table 1 (PDF p. 14).

Algorithm. Algorithm 1 gives the steps that PACIL runs on task tt. Lines 1 and 2 compute the raw zero-shot distribution PrawP^{\mathrm{raw}} with the first prompt template and its debiased version PbalP^{\mathrm{bal}} with Sinkhorn iterations. Lines 3 and 4 form the prior selection rule, which runs EM from PbalP^{\mathrm{bal}} without tempering and keeps PrawP^{\mathrm{raw}} only if the images reproduce its class imbalance. Line 5 runs EM again from the selected prior with τ=4\tau=4, and line 6 adds the images and their soft labels to the whitening statistics. Lines 7 to 9 fit the analytic classifier to the soft labels with class-balancing weights and select ωt\omega_t from its closed-form LOO outputs. Line 10 then discards the features and soft labels of the task, so later tasks see it only through the accumulated statistics. Each EM call computes the mean and the principal directions once and makes two passes, each re-estimating the within-class covariance and whitening before the vMF iterations. Because EM only reads the statistics, the EM run of the prior selection rule leaves no trace in them.

Additional Experiments

More Comparisons

Food, ImageNet-R and SUN. Table S2 completes Table 1 with the three datasets that N2L [7] reports in its Tab. 6, which gives only ZS-CLIP [1], ENGINE [14], N2L, ENGINE+Label and N2L+Label for them. PACIL is the best of the methods trained without labels in both metrics on all six settings, and its gains over N2L range from 0.42% to 1.45% on these datasets, where the refined pseudo-labels of N2L are between 85.70% and 96.22% correct. PACIL+Label is above PACIL by 0.23% to 2.90% in Aˉ\bar{A} and below N2L+Label in both metrics on both settings of SUN, the dataset with the most classes. In the Mean columns, PACIL is 0.94% and 0.88% above N2L in Aˉ\bar{A} and ATA_T, far less than on the two blocks of Table 1, and ground-truth labels add only 1.24% and 0.85% to its Mean.

MethodVenueAircraft B0Inc10 Aˉ\bar{A}Aircraft B0Inc10 ATA_TAircraft B50Inc10 Aˉ\bar{A}Aircraft B50Inc10 ATA_TCars B0Inc10 Aˉ\bar{A}Cars B0Inc10 ATA_TCars B50Inc10 Aˉ\bar{A}Cars B50Inc10 ATA_TCIFAR100 B0Inc10 Aˉ\bar{A}CIFAR100 B0Inc10 ATA_TCIFAR100 B50Inc10 Aˉ\bar{A}CIFAR100 B50Inc10 ATA_TMean Aˉ\bar{A}Mean ATA_T
ZS-CLIPICML’2126.6117.1621.6617.1682.9076.7378.7476.7381.8171.3876.4971.3861.3755.09
CODA-PromptCVPR’2326.5617.7918.1112.2879.6969.0367.8156.5979.3667.7466.9454.6456.4146.35
MoE-AdapterCVPR’2426.7317.9222.1917.5283.1677.0279.2177.7281.8671.5376.5771.4961.6255.53
RAPFECCV’2429.0719.5220.8318.4683.7773.2675.7371.5684.4776.0877.3074.3361.8655.54
RAILNeurIPS’2436.2333.5923.7525.9988.6484.6882.7282.1387.3480.3781.4478.8966.6964.28
ENGINEICCV’2534.7725.4126.9423.5686.9078.7682.6779.9385.1577.1179.8976.1566.0560.15
N2LICLR’2643.7340.2129.6932.4292.3887.5086.4285.4587.8081.1382.9280.3070.4967.84
PACILOurs51.3549.7036.7539.8092.7389.4088.5387.9588.8282.7183.8781.4973.6871.84
N2L+LabelICLR’2666.3856.3159.1856.3193.5789.1590.9289.1488.5281.9285.2381.9280.6375.79
PACIL+LabelOurs71.3961.7865.4361.7894.2791.2192.1791.1289.6883.5586.5083.5583.2478.83

Table 1. Comparison with exemplar-free methods on six datasets. Aˉ\bar{A} and ATA_T in %, and Mean averages the six settings of each block.

MethodVenueFood B0Inc10 Aˉ\bar{A}Food B0Inc10 ATA_TFood B50Inc10 Aˉ\bar{A}Food B50Inc10 ATA_TImageNet-R B0Inc20 Aˉ\bar{A}ImageNet-R B0Inc20 ATA_TImageNet-R B100Inc20 Aˉ\bar{A}ImageNet-R B100Inc20 ATA_TSUN B0Inc30 Aˉ\bar{A}SUN B0Inc30 ATA_TSUN B150Inc30 Aˉ\bar{A}SUN B150Inc30 ATA_TMean Aˉ\bar{A}Mean ATA_T
ZS-CLIPICML’2187.8681.9984.7881.9983.1276.6279.2376.6279.4572.1474.9972.1481.5776.92
ENGINEICCV’2588.3081.7884.7281.5584.1476.9379.2676.5683.0476.1778.0675.1082.9278.02
N2LICLR’2690.5885.5087.1885.0186.0081.0681.8980.2984.3878.8579.1577.4084.8681.35
PACILOurs91.1386.2587.9485.6686.8681.4882.8280.9985.4680.1480.5878.8585.8082.23
ENGINE+LabelICCV’2588.5281.8585.0881.5984.6177.1279.9676.7883.5176.2178.8375.2083.4278.13
N2L+LabelICLR’2690.9585.8488.3885.8486.6181.5283.8881.5187.4281.5084.3181.5086.9382.95
PACIL+LabelOurs91.3686.4988.8486.5087.4982.0784.3282.1286.7280.6383.4880.6487.0483.08

Table S2. Comparison on Food, ImageNet-R and SUN. Aˉ\bar{A} and ATA_T in %, and Mean averages the six settings.

Exemplar-based methods. Table S3 compares PACIL with two exemplar-based methods, PROOF [24] and CLAP4CLIP [67], which keep 20 training images per class of earlier tasks. N2L ran both in CIL with unlabeled data, training them on pseudo-labels from zero-shot CLIP, as its other baselines, and their results, as well as those of ZS-CLIP, are quoted from tbl:4 of [7], which covers the 12 settings of Aircraft, Cars, CIFAR100, CUB, ObjectNet and UCF. PACIL stores no image and is above both methods in Aˉ\bar{A} and ATA_T on all 12 settings, although only by 0.03% and 0.01% on ObjectNet B100Inc20, where it is level with CLAP4CLIP. Averaged over the 12 settings, it reaches 73.98% in Aˉ\bar{A} and 71.10% in ATA_T, against 68.71% and 64.79% for CLAP4CLIP, the stronger of the two exemplar-based methods. The Mean columns show this margin on both blocks, 5.85% and 4.69% in Aˉ\bar{A} and 7.46% and 5.15% in ATA_T over CLAP4CLIP.

MethodVenueAircraft B0Inc10 Aˉ\bar{A}Aircraft B0Inc10 ATA_TAircraft B50Inc10 Aˉ\bar{A}Aircraft B50Inc10 ATA_TCars B0Inc10 Aˉ\bar{A}Cars B0Inc10 ATA_TCars B50Inc10 Aˉ\bar{A}Cars B50Inc10 ATA_TCIFAR100 B0Inc10 Aˉ\bar{A}CIFAR100 B0Inc10 ATA_TCIFAR100 B50Inc10 Aˉ\bar{A}CIFAR100 B50Inc10 ATA_TMean Aˉ\bar{A}Mean ATA_T
ZS-CLIPICML’2126.6117.1621.6617.1682.9076.7378.7476.7381.8171.3876.4971.3861.3755.09
CLAP4CLIPNeurIPS’2441.1535.8126.8828.4390.0286.4679.6875.8886.3479.1182.9080.5767.8364.38
PROOFTPAMI’2538.1033.6228.1028.1589.2385.0983.8383.0384.8577.4779.1776.3667.2163.95
PACILOurs51.3549.7036.7539.8092.7389.4088.5387.9588.8282.7183.8781.4973.6871.84
MethodVenueCUB B0Inc20 Aˉ\bar{A}CUB B0Inc20 ATA_TCUB B100Inc20 Aˉ\bar{A}CUB B100Inc20 ATA_TObjectNet B0Inc20 Aˉ\bar{A}ObjectNet B0Inc20 ATA_TObjectNet B100Inc20 Aˉ\bar{A}ObjectNet B100Inc20 ATA_TUCF B0Inc10 Aˉ\bar{A}UCF B0Inc10 ATA_TUCF B50Inc10 Aˉ\bar{A}UCF B50Inc10 ATA_TMean Aˉ\bar{A}Mean ATA_T
ZS-CLIPICML’2175.4763.7269.0663.7238.4326.4331.1226.4375.8867.7971.6867.7960.2752.65
CLAP4CLIPNeurIPS’2482.4574.8575.7472.7149.0038.5541.8537.7787.1083.8281.3883.5069.5965.20
PROOFTPAMI’2580.5073.9873.7471.8345.4634.9335.3632.1689.4186.0081.2381.7667.6263.44
PACILOurs86.7180.3878.7677.2153.0641.6241.8837.7895.9294.7289.3590.4074.2870.35

Table S3. Comparison with exemplar-based methods on six datasets. Aˉ\bar{A} and ATA_T in %, and Mean averages the six settings of each block. CLAP4CLIP and PROOF are trained on zero-shot pseudo-labels and store 20 images per class of earlier tasks, while ZS-CLIP and PACIL store none.

Robustness to Hyperparameters and Backbones

Temperature τ\tau. Figure S1 reports Aˉ\bar{A} and ATA_T on the 18 settings when the temperature τ\tau of the final EM of every task takes the values 2, 4, 6, 8 and 10. All other hyperparameters are those of § 4.1, the debiased run of the prior selection rule keeps τ=1\tau= 1 (§ 3.2), and each point is the mean of three runs with seeds 1993, 1994 and 1995 at that value of τ\tau. Over this range, Aˉ\bar{A} and ATA_T each vary by less than 1% on 15 of the 18 settings, and Aˉ\bar{A} varies by at most 0.18% on CIFAR100 B0Inc10, Cars B0Inc10, ImageNet-R B0Inc20 and both Food settings. Averaged over the 18 settings, τ=4\tau= 4 gives the highest Aˉ\bar{A} and ATA_T of the five values, 77.92% and 74.81%, and the other four values are lower by at most 0.25% and 0.18%. On single settings, τ=4\tau= 4 is within 0.30% of the best of the five values on 16 of the 18 settings in each metric. Choosing τ\tau for each dataset by the mean Aˉ\bar{A} of the other eight datasets selects τ=4\tau= 4 for every dataset except ObjectNet, where it selects τ=6\tau= 6, and lowers the mean Aˉ\bar{A} over the 18 settings by 0.06%. A single τ\tau shared by all datasets therefore costs little compared with a per-dataset choice, and it is chosen without labeled data.

Figure S1. Sensitivity to the temperature τ\tau of the final EM. (a) Aˉ\bar{A} and (b) ATA_T (%) on the zero-base and half-base settings of the nine datasets for τ∈{2,4,6,8,10}\tau\in\{2, 4, 6, 8, 10\}, with all other hyperparameters fixed. PACIL uses τ=4\tau= 4, and each point is the mean over three runs.

For fixed class directions and concentration, dividing the exponent of Eq. (8) by τ\tau divides by τ\tau every shift that the images apply to the log-odds of the prior PtselP_t^{\mathrm{sel}}. With a small τ\tau, small differences between cosines, scaled by the large fitted concentration, override the prior. The soft labels then become sharp and follow the vMF clusters even where these clusters do not match the classes, and they enter the statistics of the closed-form classifier without later revision.

With a large τ\tau, every qiq_i moves toward PtselP_t^{\mathrm{sel}}, and the soft labels become flatter and keep fewer of the corrections that the images make to the zero-shot prior. The mean Aˉ\bar{A} of the nine zero-base settings, whose tasks hold 10 to 30 classes, decreases from 81.40% at τ=2\tau= 2 to 80.95% at τ=10\tau= 10. The mean Aˉ\bar{A} of the nine half-base settings, whose first task holds 50 to 150 classes, is 74.08% at τ=2\tau= 2 and between 74.39% and 74.53% for τ≥4\tau\ge4. This is consistent with more pairs of similar classes in an EM run over more classes, where strong image evidence moves more images to a wrong class. On ObjectNet B0Inc20, Aˉ\bar{A} falls from 54.13% at τ=2\tau= 2 to 51.57% at τ=10\tau= 10, and τ=4\tau= 4 is 1.07% below the best value. On Aircraft B50Inc10, whose first task holds 50 aircraft variants, Aˉ\bar{A} rises from 35.22% at τ=2\tau= 2 to 37.59% at τ=8\tau= 8, 0.84% above τ=4\tau= 4, and ATA_T is 0.92% higher at τ=10\tau= 10 than at τ=4\tau= 4. SUN B150Inc30 also prefers τ≥6\tau\ge6, by at most 0.30% in Aˉ\bar{A} and 0.32% in ATA_T. These settings would gain from a τ\tau chosen per dataset, but such a choice needs labeled data from each dataset.

Ridge coefficient λ\lambda. Figure S2 reports Aˉ\bar{A} and ATA_T on the 18 settings for 13 values of the ridge coefficient λ\lambda of Eq. (2) from 0.1 to 100, with τ=4\tau= 4 and all other hyperparameters as in § 4.1, and each point is the mean of three runs with seeds 1993, 1994 and 1995. The coefficient λ\lambda enters only the solve of WtW_t and the leverages of Eq. (11). The soft labels and the weights mim_i are the same for every λ\lambda, and only WtW_t and ωt\omega_t change. Averaged over the 18 settings, Aˉ\bar{A} rises from 77.28% at λ=0.1\lambda= 0.1, the value used by N2L [7], to between 77.86% and 77.93% for λ\lambda from 2.5 to 10, and falls to 77.34% at λ=100\lambda= 100. ATA_T follows the same course, from 74.25% to between 74.80% and 74.83% and down to 73.95%. For λ\lambda from 1 to 10, Aˉ\bar{A} and ATA_T each vary by less than 0.5% on 15 of the 18 settings.

Figure S2. Sensitivity to the ridge coefficient λ\lambda. (a) Aˉ\bar{A} and (b) ATA_T (%) on the zero-base and half-base settings of the nine datasets for 13 values of λ\lambda from 0.1 to 100, with all other hyperparameters fixed. PACIL uses λ=10\lambda= 10, and each point is the mean over three runs.

λ\lambda is the weight of the penalty λ∥W∥F2\lambda\lVert W\rVert_{F}^{2} of a weighted ridge regression whose data terms have weights mim_i summing to about ntn_t per task. A small λ\lambda barely damps the directions of AtA_t with small eigenvalues, so WtW_t follows the soft label of each image closely. It also raises the leverages hih_i, and the LOO predictions that select ωt\omega_t then amplify the fitted residuals by a larger factor 1/(1−hi)1/(1-h_i). From λ=10\lambda= 10 to 0.1, Aˉ\bar{A} falls by 2.66% on Aircraft B0Inc10 and 2.58% on ObjectNet B0Inc20. A large λ\lambda shrinks WtW_t most along the directions of AtA_t with small eigenvalues, and λWt\lambda W_t tends to CtC_t, a sum of features weighted by the soft labels that ignores the correlations in AtA_t, which can remove the fine distinctions between similar classes. From λ=10\lambda= 10 to 100, Aˉ\bar{A} falls on 17 of the 18 settings and ATA_T on all 18, most on Aircraft, by 2.08% and 1.62% in Aˉ\bar{A}, and on UCF, by 1.21% and 1.17%, whose Aˉ\bar{A} peaks at λ=2.5\lambda= 2.5 and λ=1\lambda= 1. ObjectNet B0Inc20 is the only exception, as it rises to 53.35% at λ=25\lambda= 25 and stays within 0.05% of this value up to λ=100\lambda= 100.

We use λ=10\lambda= 10 because it lies on the plateau of both means and loses least on the settings that depart most from it. Its mean Aˉ\bar{A} and ATA_T are 0.01% and 0.02% below the best of the 13 values, reached at λ=7.5\lambda= 7.5 and at λ=5\lambda= 5 and 7.5. Among the 13 values it has the smallest largest shortfall in Aˉ\bar{A} from the best value of a setting, 0.38% on Aircraft B50Inc10, and it is within 0.20% of that best value on 16 settings in Aˉ\bar{A} and on 15 in ATA_T, where its largest shortfall is 0.58%, also on Aircraft B50Inc10. As the weights mim_i sum to about ntn_t per task in both methods, this value also compares directly with the 0.1 of N2L (§ 3.4).

CLIP backbones. Figure S3 compares N2L and PACIL on all 18 settings with ViT-B/32, ViT-B/16 and ViT-L/14, each with LAION-400M weights (OpenCLIP laion400m_e32) and with OpenAI weights. Both methods are re-run with the same code, splits and seeds, and only the backbone changes. PACIL keeps τ=4\tau= 4, λ=10\lambda= 10, dp=128d_p = 128 and D=5000D = 5000 on every backbone without tuning, and N2L keeps its own hyperparameters. PACIL is above N2L in Aˉ\bar{A} on 105 of the 108 pairs of backbone and setting, and in ATA_T on 98. Averaged over the 18 settings, its gain in Aˉ\bar{A} is 2.10%, 2.24% and 2.19% on the LAION backbones from B/32 to L/14 and 2.25%, 2.24% and 1.74% on the OpenAI backbones, and its gain in ATA_T lies between 1.78% and 2.39% on all six. The gain therefore does not shrink with scale, except on OpenAI ViT-L/14, where N2L is most accurate, with a mean Aˉ\bar{A} of 82.07%.

Figure S3. CLIP backbones. (a) Aˉ\bar{A} and (b) ATA_T (%) of N2L and PACIL with ViT-B/32, ViT-B/16 and ViT-L/14 (left to right within each group), each with LAION-400M and OpenAI weights, on the nine datasets in the zero-base (left) and half-base (right) settings. Both methods are re-run in the same setting.

A larger backbone gives better zero-shot predictions, which raises the accuracy of N2L and leaves less to correct where accuracy is high. With OpenAI weights, the gain in Aˉ\bar{A} falls from B/32 to L/14 from 0.23% to 0.07% on Food B0Inc10 and from 2.10% to 0.32% on ImageNet-R B0Inc20. On datasets that stay hard at every scale it holds or grows, as on ObjectNet B100Inc20, from 0.44% to 2.39% with LAION weights and from 1.04% to 2.40% with OpenAI weights, which is consistent with EM gaining from better image features while zero-shot labels leave errors to correct. Aircraft B0Inc10 shows both cases, as the gain rises from 6.91% to 10.34% with LAION weights and falls from 5.41% to 4.45% with OpenAI weights. N2L is higher on three pairs in Aˉ\bar{A}, by at most 0.33% (UCF B50Inc10 with LAION ViT-L/14, SUN B150Inc30 and CIFAR100 B50Inc10 with OpenAI ViT-B/32), and on ten pairs in ATA_T, by at most 0.54%. Seven of these ten are on Food and ImageNet-R with OpenAI weights, where the gain of PACIL in Aˉ\bar{A} is also at most 0.61%.

Noisy Task Boundaries

Unlabeled images rarely respect task boundaries. We test whether PACIL remains effective when the unlabeled data of a task also contain images of earlier classes. Following the protocol of N2L [7], from the second task on, images drawn from the training sets of earlier tasks are added to the unlabeled images of task tt until they make up a fraction pp of them, with p=20%p = 20\% or 50%50\%. The learner is not told which images they are and can only name them among the classes of task tt, so no correct label is available for them. In a zero-base setting 9 of the 10 tasks are noisy, whereas in a half-base setting the clean first task holds half of the classes. Table S4 compares PACIL with RAIL [13], ENGINE [14] and N2L under this protocol at both noise levels.

Table S4. Comparison under noisy task boundaries. Aˉ\bar{A} and ATA_T in %, and Mean averages the six settings of each block. From the second task on, a fraction pp of the unlabeled training images of each task comes from earlier tasks, and the results without noise are in Table 1.

ppMethodAircraft B0Inc10 Aˉ\bar{A}Aircraft B0Inc10 ATA_TAircraft B50Inc10 Aˉ\bar{A}Aircraft B50Inc10 ATA_TCars B0Inc10 Aˉ\bar{A}Cars B0Inc10 ATA_TCars B50Inc10 Aˉ\bar{A}Cars B50Inc10 ATA_TCIFAR100 B0Inc10 Aˉ\bar{A}CIFAR100 B0Inc10 ATA_TCIFAR100 B50Inc10 Aˉ\bar{A}CIFAR100 B50Inc10 ATA_TMean Aˉ\bar{A}Mean ATA_T
20%RAIL35.1931.7123.0624.6087.9582.8182.3781.0287.0179.9581.1078.3866.1163.08
ENGINE33.9524.7226.5122.5686.4578.3282.1679.1884.7276.6179.5275.8665.5559.54
N2L43.1138.7029.1330.5191.0685.2686.0084.2387.6880.8182.7880.0769.9666.60
PACIL49.5046.9636.3539.1991.2988.3788.2487.7688.3982.2283.7381.3572.9270.98
50%RAIL33.4628.8321.9721.8185.5678.7481.5278.9986.5579.2080.6777.6964.9660.88
ENGINE33.3623.2525.6321.3985.4476.5680.8375.9083.7275.6878.6474.5064.6057.88
N2L40.4936.0027.4527.7589.3281.6185.1081.5887.0280.2782.3979.2368.6364.41
PACIL43.1640.5832.9034.1489.5585.7386.7484.6387.1380.9582.8979.8870.4067.65
ppMethodCUB B0Inc20 Aˉ\bar{A}CUB B0Inc20 ATA_TCUB B100Inc20 Aˉ\bar{A}CUB B100Inc20 ATA_TObjectNet B0Inc20 Aˉ\bar{A}ObjectNet B0Inc20 ATA_TObjectNet B100Inc20 Aˉ\bar{A}ObjectNet B100Inc20 ATA_TUCF B0Inc10 Aˉ\bar{A}UCF B0Inc10 ATA_TUCF B50Inc10 Aˉ\bar{A}UCF B50Inc10 ATA_TMean Aˉ\bar{A}Mean ATA_T
20%RAIL80.2571.6172.7270.0238.6133.6630.2928.7488.5488.3380.4583.2165.1462.60
ENGINE75.6863.6070.2064.1944.0130.3434.2829.3986.9484.5482.0480.6465.5358.78
N2L82.1074.2874.4671.7648.1040.8440.9637.7393.1591.2185.8586.4370.7767.04
PACIL84.2678.1678.5176.3653.2841.5742.8839.3395.5393.8589.0689.9073.9269.86
50%RAIL78.3868.4571.2066.3334.4829.5328.0924.9886.3185.7578.4479.5862.8259.10
ENGINE74.3961.7969.2262.8242.7229.7533.5328.4184.2882.8380.7578.6364.1557.37
N2L79.8670.5272.8068.5446.4837.0939.0534.9490.8088.8684.7584.9968.9664.16
PACIL80.0872.9776.0473.2847.6437.1641.1736.9091.2689.3687.8988.6670.6866.39

PACIL has the best Aˉ\bar{A} and ATA_T on all 12 settings at both noise levels, and every Mean column ranks the four methods in the same order at both levels. At p=20%p = 20\% its Mean gain over N2L in Aˉ\bar{A}, 2.96% and 3.15% on the two blocks, is as large as without noise in Table 1, where it is 3.19% and 2.49%. At p=50%p = 50\% it shrinks to 1.77% and 1.72%, while the gain in ATA_T remains 3.24% and 2.23%. The drop comes mostly from the zero-base settings, where the gain in Aˉ\bar{A} falls from 2.83% to 0.81% while it stays at 2.68% in the half-base settings. Soft labels lie closer to the true labels than hard labels on the images of the current classes (Section C.6), and also on an image of an earlier class, for which no current class is correct, since a soft label qiq_i is at squared distance 1+∥qi∥21 + \lVert q_i\rVert^2 from its true label against 2 for a hard label. This lowers the target error that each such image brings into the classifier and is consistent with the unchanged lead at p=20%p = 20\%. From no noise to p=50%p = 50\%, the Mean Aˉ\bar{A} of PACIL falls by 3.28% and 3.60%, more than the 1.86% and 2.83% of N2L, while the falls in ATA_T, 4.19% and 3.96% against 3.43% and 4.39%, are of similar size. The larger fall in Aˉ\bar{A} comes from the part of the gain that PACIL draws from the images. EM corrects the zero-shot distributions by fitting one component per current class to the images of a task (§ 3.3) and is worth up to 4.18% of Aˉ\bar{A} without noise (Figure 4), whereas the self-training of N2L adds little after its first round (Section C.6). Images of earlier classes fit none of these components, and at p=50%p = 50\% in a zero-base setting they outnumber the images of each current class several times over, so they can pull the components away from the classes that EM should recover. Such images therefore cost PACIL more than they cost N2L, yet it stays above N2L on every setting. These images fall outside the assumption that a task brings images of its own classes, and a component for them is left to future work.

N2L with Components of PACIL

N2L [7] uses the ridge coefficient λ=0.1\lambda= 0.1 and a fixed fusion weight ω=0.8\omega= 0.8, and its results in Figure S4 are our reproduction with its official code. Two components of PACIL carry over to N2L without its soft labels, λ=10\lambda= 10 and the LOO choice of ωt\omega_t in Eq. (12), which then scores the LOO decisions with the hard pseudo-labels of N2L and keeps its sample weights. We add them to this reproduction one at a time and together, to test whether the gain of PACIL comes only from them. Averaged over the 18 settings, λ=10\lambda= 10 changes Aˉ\bar{A} and ATA_T by 0.24% and −0.10%-0.10\%, and the LOO choice by 0.13% and 0.36%. Neither helps everywhere, as λ=10\lambda= 10 lowers Aˉ\bar{A} on both UCF settings, by up to 0.86%, and the LOO choice gains most on ObjectNet, by 1.43% and 1.11% in Aˉ\bar{A}. Together they raise Aˉ\bar{A} and ATA_T by 0.61% and 0.64%, more than the sum of their separate effects. With λ=10\lambda= 10 the leverages are smaller, so the LOO outputs amplify the fitted residuals less (Sec. C.2), and the selected ωt\omega_t rises from 0.85 to 0.91 on average.

Figure S4. N2L with components of PACIL. (a) Aˉ\bar{A} and (b) ATA_T (%) of N2L, N2L with the ridge coefficient λ=10\lambda= 10, N2L with the LOO choice of the fusion weight of Eq. (12), N2L with both, and PACIL, on the nine datasets in the zero-base (left) and half-base (right) settings. N2L is our reproduction with its official code [7], with λ=0.1\lambda= 0.1 and a fixed ω=0.8\omega= 0.8, and its three variants are run with the same code. Bars are means over three seeds, error bars show standard deviation, and the axes do not start at zero.

PACIL stays 1.63% and 1.75% above the strongest variant and is higher on 16 of the 18 settings in Aˉ\bar{A} and on 17 in

ATA_T. The two components thus explain about a quarter of the gain of PACIL over N2L, and most of it comes from how PACIL estimates and keeps the posterior of each label. The remaining gap is largest on Aircraft, by 6.23% and 6.73% in Aˉ\bar{A} and by 7.72% and 7.13% in ATA_T, where N2L mislabels more than 40% of the training images and the argmax of the soft labels of PACIL alone brings N2L 37% to 66% of the gain in Aˉ\bar{A} (Sec. C.6). On ObjectNet, the two components raise Aˉ\bar{A} of N2L the most, by 4.20% and 1.93%, and the strongest variant is level with PACIL in Aˉ\bar{A} on B0Inc20 and above it on B100Inc20, by 0.36% in Aˉ\bar{A} and 0.54% in ATA_T. ObjectNet B0Inc20 is also the only setting where PACIL itself gains from a λ\lambda above 10 (Sec. C.2).

Comparison with Transductive Methods

Transductive methods also estimate a class distribution for each unlabeled image, but only to predict these images (§ 2). We replace the soft labels qiq_i of each task by the

distributions that TransCLIP [38], StatA [70], MOON [5] and EM-Dirichlet [4] compute on DtD_t from the same features and class embeddings with their zero-shot hyperparameters, and keep the rest of PACIL unchanged. As references, we also train on PrawP^{\mathrm{raw}} and PtselP_t^{\mathrm{sel}} without EM, and on TransCLIP and MOON with PtselP_t^{\mathrm{sel}} as their zero-shot input.

In Figure S6, PACIL is above all four methods in both metrics. Averaged over the 18 settings, it is 2.34% to 2.94% above TransCLIP, StatA and MOON in Aˉ\bar{A} and 2.22% to 2.69% in ATA_T, and it is lower only on the two SUN settings and in ATA_T on Food B50Inc10, by at most 0.31%. EM-Dirichlet falls below even PrawP^{\mathrm{raw}}, as it expects a batch to hold only a few classes. The gaps come first from the class prior, as TransCLIP, StatA and MOON keep PrawP^{\mathrm{raw}} as the prior of each image. PtselP_t^{\mathrm{sel}} alone, without EM, is above all four methods in both metrics, and as their zero-shot input it narrows the gap of TransCLIP and MOON to PACIL from 2.34% to 1.49% and from 2.94% to 0.73% in Aˉ\bar{A}. The remaining gap lies mainly in the likelihood, as PACIL fits vMF components to features whitened with the within-class scatter of all seen classes (§ 3.3), whereas TransCLIP uses Gaussians with a shared diagonal covariance and MOON does not whiten. Whitening with the statistics of the current task alone is enough to lower Aˉ\bar{A} of PACIL by 0.37%, which is half of its lead over MOON with PtselP_t^{\mathrm{sel}}.

Figure S6. Comparison with transductive methods. Change in (a) Aˉ\bar{A} and (b) ATA_T (%) relative to PACIL when its soft labels are replaced by PrawP^{\mathrm{raw}} or PtselP_t^{\mathrm{sel}} without EM or by the distributions of transductive methods, averaged over the nine datasets in the zero-base and the half-base settings. All sources share the classifier and fusion of PACIL, +Psel+P^{\mathrm{sel}} gives a method PtselP_t^{\mathrm{sel}} as its zero-shot input, and the bars of EM-Dirichlet are labeled with their values.

Pseudo-Label Quality

Accuracy. Figure S5 compares the pseudo-label accuracy of N2L [7] at zero-shot and after each of its three refinement rounds with that of PACIL on Aircraft and CUB, in the zero-base and half-base settings. Accuracy is the fraction of training images whose label is the true class, pooled over all tasks of a setting, and ground-truth labels are used only for the measurements of this subsection. PACIL keeps a label distribution qiq_i for each image, and to compare it with the hard labels of N2L we take its argmax. This comparison is conservative for PACIL, since its classifier is fitted to the whole qiq_i, and an image whose argmax is wrong keeps part of its probability mass on the true class.

Figure S5. Pseudo-labels of PACIL and N2L [7] on Aircraft and CUB. (a) Pseudo-label accuracy (%) on the training images, pooled over all tasks, for N2L at zero-shot (ZS) and after each refinement round and for the argmax of the final soft labels of PACIL, with zero-base settings on top and half-base settings at the bottom. (b) Gain over N2L in Aˉ\bar{A} and ATA_T (%) of N2L with its pseudo-labels replaced by the argmax of the PACIL soft labels, and of PACIL.

Both methods start from the same zero-shot labels, and PACIL is above N2L after round 3 on all four settings, by 9.73% and 8.31% on Aircraft and 2.78% and 3.75% on CUB. N2L gains most in round 1, by up to 5.48% on Aircraft and 3.14% on CUB, and rounds 2 and 3 together add at most 0.59%. The margins are largest on Aircraft, where N2L is least accurate and mislabels more than 40% of the images after round 3. Self-training on hard labels corrects few errors after round 1 and can overwrite correct ones, while PACIL corrects the class prior of the whole distribution and refines it by EM over the images of each task, which raises the accuracy on Aircraft from 53.85% and 39.36% at zero-shot to 69.00% and 50.92%.

Figure S5 separates the effect of these labels from the rest of PACIL. Replacing the pseudo-labels of N2L by the argmax of the PACIL soft labels, with the rest of N2L unchanged, raises Aˉ\bar{A} by 3.58% and 4.66% on Aircraft and by 1.47% and 1.34% on CUB. This is 37% to 66% of the gain of PACIL in Aˉ\bar{A}, and 47% to 70% in ATA_T. The rest comes from the other differences between the two methods, such as training on the whole qiq_i rather than its argmax and selecting ωt\omega_t on each task by closed-form LOO.

Squared error. The bounds of Theorem S1 depend on how close the targets of the classifier are to the label posteriors, which cannot be observed. Figure S7 therefore

Figure S7. Squared error of the pseudo-labels. MSE of the zero-shot argmax (grey circles), the hard pseudo-labels of N2L [7] after round 3 (blue squares) and the soft labels qq of PACIL (red diamonds), averaged over the training images of all tasks of each setting in (a) the zero-base and (b) the half-base settings. The last row is the mean over the nine datasets. KK differs across datasets, so markers are compared within a row.

measures how far three targets are from the true labels. They are the zero-shot argmax, which is also the starting label of N2L, the hard pseudo-labels of N2L after its three refinement rounds, and the soft labels qiq_i of PACIL. A hard label enters as the one-hot vector of its class, and the distance of a target pip_i for an image ii of a task with KK classes is the mean squared error ∥pi−eyi∥2/K\lVert p_i-e_{y_i}\rVert^2/K, the Brier score averaged over these classes, so a hard label scores 2/K when it is wrong and 0 when it is right. It is averaged over the training images of all tasks.

The soft labels of PACIL have the lowest MSE in all 18 settings. Averaged over the 18 settings, their MSE is 0.0123, against 0.0169 for N2L and 0.0216 for the zero-shot argmax. The refinement of N2L lowers the MSE of the zero-shot argmax by 22% on average, but the MSE of qiq_i is 9% to 46% below that of N2L in single settings. SUN is the exception to this ordering, as the refinement there moves the hard labels away from the true labels, from 0.0044 to 0.0050 and from 0.0033 to 0.0039, in line with the accuracy of these labels, which falls from 93.42% to 92.46% and from 86.77% to 85.70% over the three rounds.

This ordering reflects the distance to the posteriors that Theorem S1 needs. For any target pip_i computed from the observations, the identity of § 3.4 gives

E∥pi−eyi∥2=∥pi−eˉi∥2+1−∥eˉi∥2,(48)\mathbb{E}\lVert p_i-e_{y_i}\rVert^2=\lVert p_i-\bar{e}_i\rVert^2+1-\lVert\bar{e}_i\rVert^2, \tag*{(48)}

whose last two terms do not depend on the target. The Brier score against the true labels therefore ranks targets by their squared distance to the posteriors, and its expectation is smallest at pi=eˉip_i=\bar{e}_i. In a zero-base setting every image has the same KK, so the MSE ranks them in the same way. A hard label also stays at squared distance at least (1−eˉic)2(1-\bar{e}_{ic})^2 from the posterior on every image by Eq. (S11), even when its class is correct, whereas a soft label can match the posterior. Figure S7 thus shows that qiq_i comes closer to the posteriors than both kinds of hard labels, and this distance is what the bound of Theorem S1(a) depends on.

Accuracy after Each Task

Figure S8 and Figure S9 show the accuracy on all seen classes after each task, with N2L [7] re-run from its official code. In the zero-base settings, PACIL is higher on 81 of the 90 tasks and level on 2, by 2.25% on average. It is lower only on the first four tasks of Cars, by at most 1.25%, and on the first task of CIFAR100, CUB and ImageNet-R, by at most 0.27%. In the half-base settings it is higher on all 54 tasks, by 2.22% on average and by at least 0.22%.

Figure S8. Accuracy (%) after each task in the zero-base settings, against the number of seen classes. Accuracy is measured on the test images of all seen classes with the fused prediction. N2L [7] is re-run with its official code, since its paper reports no per-task accuracy. Lines are means over three seeds, and shaded bands show standard deviation.

Figure S9. Accuracy (%) after each task in the half-base settings, where the first task holds half of the classes. N2L [7] is re-run with its official code, lines are means over three seeds, and shaded bands show standard deviation.

Averaged over the datasets, the zero-base gap grows from 0.56% after the first task to 2.61% after the last, consistent with the errors of frozen pseudo-labels adding up over the tasks (Theorem S1). In the half-base settings, the gap is 2.26% after the first task, which holds half of the classes. The gain is largest on the fine-grained Aircraft, 9.49% and 7.32% at the last task, where zero-shot CLIP [1] has an ATA_T of 17.16% (Table 1). It is smallest on Food and ImageNet-R, 0.58% to 0.71%, where zero-shot CLIP reaches 81.99% and 76.62% (Table S2). There the zero-shot argmax is mostly right, and a posterior concentrated on one class is close to its argmax by Eq. (S11), so keeping the whole distribution changes little.

Training and Inference Cost

Table S5 reports the measured cost of training and inference. All methods read the same pre-extracted CLIP features and run the 18 settings with three seeds each on one A100 GPU, so the comparison covers the learning step itself. N2L [7] is re-implemented here in our code base following its official implementation, which inverts the 5000×50005000 \times5000 matrix of its classifier with NumPy on the CPU, and N2L†^{\dagger} computes the same inverse on the GPU.

N2LN2L†^{\dagger}PACIL
Training time (s)22.612.812.16
labeling2.361.901.18
classifier update20.120.780.85
Inference time (s)7.617.337.33
Training memory (GB)1.452.543.23
Aˉ\bar{A} (%)75.6475.6477.92
ATA_T (%)72.3272.3274.81

Table S5. Measured training and inference cost. Means over the 18 settings with three runs each, on one A100 GPU with the same pre-extracted ViT-B/16 features. Times are per setting and summed over its tasks. Memory is the peak

PACIL takes 2.16 s on average to train on one setting, of which 1.18 s is labeling. Labeling is cheap because each task runs EM only twice, once for the prior selection rule and once for the final run, on dp=128d_p = 128 whitened dimensions, where one iteration costs O(ntktdp)O(n_t k_t d_p). N2L takes 22.61 s, almost all of it in the inverse on the CPU, since moving this step to the GPU reduces its classifier update from 20.12 s to 0.78 s and its training time to 2.81 s. The classifier update of PACIL costs O(ntD2)O(n_t D^2) for the statistics and for the closed-form LOO, plus one D×DD \times D Cholesky factorization, the same order as the inverse of N2L†. With the same kind of solver, the training time of PACIL is therefore on par with that of N2L.

At inference, both methods only add the random features and two matrix products, with the classifier and with the text embeddings, to the CLIP forward pass, and their inference time is the same, 7.33 s per setting and about 0.32 ms per image. PACIL keeps its statistics in double precision, so its peak training memory is 3.23 GB on average against 2.54 GB for N2L†. N2L itself uses only 1.45 GB because its inverse runs on the CPU, outside the measured GPU memory. Across tasks, PACIL carries only (G,g,N,H,ν)(G, g, N, H, \nu) and (At,Ct)(A_t, C_t), which take 8(d2+dKt+Kt+d+1)+8(D2+DKt)8(d^2 + dK_t + K_t + d + 1) + 8(D^2 + DK_t) bytes and do not grow with the number of seen images. They are dominated by the D×DD \times D matrix AtA_t, which N2L also keeps.

References

  1. [1]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, pages 8748–8763, 2021. 1, 6, 13, 15, 23arxiv.org/abs/2103.00020
  2. [2]M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Mateusz Kozinski, Horst Possegger, Rogerio Feris, and Horst Bischof. LaFter: Label-free tuning of zero-shot classifier using language and unlabeled image collections. In Advances in Neural Information Processing Systems, pages 5765–5777, 2023. 1, 2arxiv.org/abs/2305.18287
  3. [3]Jiahan Zhang, Qi Wei, Feng Liu, and Lei Feng. Candidate pseudolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data. In Proceedings of the International Conference on Machine Learning, pages 60004–60020, 2024. 1, 2arxiv.org/abs/2406.10502
  4. [4]Ségolène Martin, Yunshi Huang, Freshteh Shakeri, Jean-Christophe Pesquet, and Ismail Ben Ayed. Transductive zero-shot and few-shot CLIP. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28816–28826, 2024. 1, 2, 3, 21arxiv.org/abs/2405.18437
  5. [5]Jiazhen Huang, Zhiming Liu, Changhu Wang, Wei Ju, Ziyue Qiao, and Xiao Luo. Von Mises-Fisher mixture model with dynamic shrinkage for realistic test-time transduction. In Proceedings of the International Conference on Machine Learning, pages 47178–47211, 2026. 1, 2, 21arxiv.org/abs/2607.15851
  6. [6]Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Class-incremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9851–9873, 2024. 1DOI
  7. [7]Qiwei Li, Xiaocheng Yang, and Jiahua Zhou. Naming to learn: Class incremental learning for vision-language model with unlabeled data. In International Conference on Learning Representations, pages 100800–100824, 2026. 1, 2, 3, 6, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24
  8. [8]Hari Chandana Kuchibhotla, K. S. Ananth, and Vineeth N. Balasubramanian. Annotation-free class-incremental learning. arXiv, abs/2511.19344, 2025. 1, 2arxiv.org/abs/2511.19344
  9. [9]Niels Griffioen, Nevena Ranković, Federico Zamberlan, and Monisha Punith. Efficient annotation reduction with active learning for computer vision-based retail product recognition. Journal of Computational Social Science, 7(1):1039–1070, 2024. 1DOI
  10. [10]Mohammad Sadegh Norouzzadeh, Anh Nguyen, Margaret Kosmala, Alexandra Swanson, Meredith S. Palmer, Craig Packer, and Jeff Clune. Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning. Proceedings of the National Academy of Sciences, 115(25):E5716–E5725, 2018. 2DOI
  11. [11]Huiping Zhuang, Zhenyu Wang, Hongxin Wei, Rencunzi Xie, Kar-Ann Toh, and Zhiping Lin. ACIL: Analytic class-incremental learning with absolute memorization and privacy protection. In Advances in Neural Information Processing Systems, pages 11602–11614, 2022. 2, 4arxiv.org/abs/2205.14922
  12. [12]Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19125–19136, 2023. 2arxiv.org/abs/2303.06628
  13. [13]Yicheng Xu, Yuxin Chen, Jiahao Nie, Yusong Wang, Huiping Zhuang, and Manabu Okumura. Advancing cross-domain discriminability in continual learning of vision-language models. In Advances in Neural Information Processing Systems, pages 51552–51576, 2024. 2, 4, 6, 19arxiv.org/abs/2406.18868
  14. [14]Da-Wei Zhou, Kai-Wen Li, Jingyi Ning, Han-Jia Ye, Lijun Zhang, and De-Chuan Zhan. External knowledge injection for CLIP-based class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3314–3325, 2025. 2, 6, 15, 19arxiv.org/abs/2503.08510
  15. [15]Eric Arazo, Diego Ortego, Paul Albert, Noel E. O’Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In Proceedings of the International Joint Conference on Neural Networks, pages 1–8, 2020. 2arxiv.org/abs/1908.02983
  16. [16]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 2arxiv.org/abs/1503.02531
  17. [17]Kamal Nigam, Andrew Kachites McCallum, Sebastian Thrun, and Tom Mitchell. Text classification from labeled and unlabeled documents using EM. Machine Learning, 39(2–3):103–134, 2000. 2DOI
  18. [18]Mark D. McDonnell, Dong Gong, Amin Parvenah, Ehsan Abbasnejad, and Anton van den Hengel. RanPAC: Random projections and pre-trained models for continual learning. In Advances in Neural Information Processing Systems, pages 12022–12053, 2023. 2, 4
  19. [19]Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanah, James Caverlee, and Shu Kong. The neglected tails in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12988–12997, 2024. 2arxiv.org/abs/2401.12425
  20. [20]Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. In Advances in Neural Information Processing Systems, pages 64663–64680, 2023. 3arxiv.org/abs/2310.08106
  21. [21]Wei Tang, Zuo-Zheng Wang, Kun Zhang, Tong Wei, and Min-Ling Zhang. Unleashing the power of vision-language models for long-tailed multi-label visual recognition. arXiv preprint arXiv:2511.20641, 2025. 2arxiv.org/abs/2511.20641
  22. [22]Jiazou Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23219–23230, 2024. 2, 6arxiv.org/abs/2403.11549
  23. [23]Linlan Huang, Xusheng Cao, Haori Lu, and Xiailei Liu. Class-incremental learning with CLIP: Adaptive representation adjustment and parameter fusion. In Proceedings of the European Conference on Computer Vision, pages 214–231, 2024. 6arxiv.org/abs/2407.14143
  24. [24]Da-Wei Zhou, Yuanhan Zhang, Yan Wang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(6): 4489–4504, 2025. 6, 15DOI
  25. [25]Mao-Lin Luo, Zi-Hao Zhou, Tong Wei, and Min-Ling Zhang. LADA: Scalable label-specific CLIP adapter for continual learning. In Proceedings of the International Conference on Machine Learning, pages 41604–41619, 2025. 2arxiv.org/abs/2505.23271
  26. [26]Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, and Zhiping Lin. DS-AL: A dual-stream analytic learning for exemplar-free class-incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 17237–17244, 2024. 2arxiv.org/abs/2403.17503
  27. [27]Huiping Zhuang, Yizhu Chen, Di Fang, Run He, Kai Tong, Hongxin Wei, Ziqian Zeng, and Cen Chen. GACL: Exemplar-free generalized analytic continual learning. In Advances in Neural Information Processing Systems, pages 83024–83047, 2024. 2arxiv.org/abs/2403.15706
  28. [28]Binyu Zhao, Wei Zhang, Xingrui Yu, Zhaonian Zou, and Ivor Tsang. Advancing analytic class-incremental learning through vision-language calibration. In Proceedings of the International Conference on Machine Learning, pages 162951–162970, 2026. 2arxiv.org/abs/2602.13670
  29. [29]Zhen Zhu, Yiming Gong, and Derek Hoiem. Anytime continual learning for open vocabulary classification. In Proceedings of the European Conference on Computer Vision, pages 269–285, 2024. 2arxiv.org/abs/2409.08518
  30. [30]Subhankar Roy, Mingxuan Liu, Zhun Zhong, Nicu Sebe, and Elisa Ricci. Class-incremental novel class discovery. In Proceedings of the European Conference on Computer Vision, pages 317–333, 2022. 2
  31. [31]Shijie Ma, Fei Zhu, Zhun Zhong, Wenzhuo Liu, Xu-Yao Zhang, and Cheng-Lin Liu. Happy: A debiased learning framework for continual generalized category discovery. In Advances in Neural Information Processing Systems, pages 50850–50875, 2024. 2arxiv.org/abs/2410.06535
  32. [32]Donghao Sun, Xi Wang, Xu Yang, Kun Wei, and Cheng Deng. Dual-space semantic synergy distillation for continual learning of unlabeled streams. In Advances in Neural Information Processing Systems, pages 21303–21325, 2025. 2DOI
  33. [33]Ameya Prabhu, Hasan Abed Al Kader Hammoud, Ser-Nam Lim, Bernard Ghanem, Philip Torr, and Adel Bibi. From categories to classifiers: Name-only continual learning by exploring the web. In Proceedings of the Conference on Lifelong Learning Agents, pages 534–559, 2024. 2arxiv.org/abs/2311.11293
  34. [34]Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649, 2022. 2arxiv.org/abs/2204.03649
  35. [35]Junnan Li, Silvio Savarese, and Steven C. H. Hoi. Masked unsupervised self-training for label-free image classification. In International Conference on Learning Representations, 2023. 2arxiv.org/abs/2206.02967
  36. [36]Qiyuan Dai and Sibei Yang. Free on the fly: Enhancing flexibility in test-time adaptation with online EM. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9538–9548, 2025. 2, 3arxiv.org/abs/2507.06973
  37. [37]Lihua Zhou, Mao Ye, Shuaifeng Li, Nianxin Li, Xiatian Zhu, Lei Deng, Hongbin Liu, and Zhen LeI. Bayesian test-time adaptation for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 29999–30009, 2025. 2
  38. [38]Maxime Zanella, Benoît Gérin, and Ismail Ben Ayed. Boosting vision-language models with transduction. In Advances in Neural Information Processing Systems, pages 62223–62256, 2024. 2, 3, 5, 21arxiv.org/abs/2406.01837
  39. [39]Xingyu Xu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. In Advances in Neural Information Processing Systems, pages 2001–2025, 2024. 3
  40. [40]Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. Neural Computation, 14(1):21–41, 2002. 3
  41. [41]Zachary C. Lipton, Yu-Xiang Wang, and Alexander J. Smola. Detecting and correcting for label shift with black box predictors. In Proceedings of the International Conference on Machine Learning, pages 3122–3130, 2018. 3arxiv.org/abs/1802.03916
  42. [42]Xudong Wang, Zhirong Wu, Long Lian, and Stella X. Yu. Debiased learning from naturally imbalanced pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14647–14657, 2022. 3arxiv.org/abs/2201.01490
  43. [43]Adian Liusie, Potswaee Manakul, and Mark J. F. Gales. Mitigating world bias in zero-shot prompt-based classifiers. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023, pages 327–335, 2023.
  44. [44]Qi Qian, Yuanhong Xu, and Juhua Hu. Intra-modal proxy learning for zero-shot visual categorization with CLIP. In Advances in Neural Information Processing Systems, pages 25461–25474, 2023. 3arxiv.org/abs/2310.19752
  45. [45]Changho Shin, Jitian Zhao, Sonia Cromp, Harit Vishwakarma, and Frederic Sala. OTTER: Effortless label distribution adaptation of zero-shot models. In Advances in Neural Information Processing Systems, pages 44064–44101, 2024. 3arxiv.org/abs/2404.08461
  46. [46]Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. SoLar: Sinkhorn label refinery for imbalanced partial-label learning. In Advances in Neural Information Processing Systems, pages 8104–8117, 2022. 3arxiv.org/abs/2209.10365
  47. [47]Kuang He, Wei Tang, Tong Wei, and Min-Ling Zhang. Tuning the right foundation models is what you need for partial label learning. arXiv preprint arXiv:2506.05027, 2025. 3arxiv.org/abs/2506.05027
  48. [48]Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, pages 2292–2300, 2013. 4
  49. [49]Imre Csiszár. I-divergence geometry of probability distributions and minimization problems. The Annals of Probability, 3(1):146–158, 1975. 4DOI
  50. [50]Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien, editors. Semi-Supervised Learning. MIT Press, Cambridge, MA, 2006. 4
  51. [51]Dipam Goswami, Yuyang Liu, Bartlomiej Twardowski, and Joost van de Weijer. FeCAM: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. In Advances in Neural Information Processing Systems, pages 6582–6595, 2023. 5arxiv.org/abs/2309.14062
  52. [52]Ameya Prabhu, Shiven Sinha, Ponnurangam Kumarguru, Philip Torr, Ozan Sener, and Puneet K. Dokania. Ran-Dumb: Random representations outperform online continually learned representations. In Advances in Neural Information Processing Systems, pages 37988–38006, 2024. 5DOI
  53. [53]Yilun Chen, Ami Wiesel, Yonina C. Eldar, and Alfred O. Hero. Shrinkage algorithms for MMSE covariance estimation. IEEE Transactions on Signal Processing, 58(10):5016–5029, 2010. 5DOI
  54. [54]Arindam Banerjee, Inderjit S. Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering on the unit hypersphere using von Mises-Fisher distributions. Journal of Machine Learning Research, 6(46):1345–1382, 2005. 5DOI
  55. [55]David M. Allen. The relationship between variable selection and data augmentation and a method for prediction. Technometrics, 16(1):125–127, 1974. 5DOI
  56. [56]Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, NY, 2nd edition, 2009. 5
  57. [57]Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013. 6arxiv.org/abs/1306.5151
  58. [58]Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 554–561, 2013. 6DOI
  59. [59]Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 6
  60. [60]Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. In Proceedings of the European Conference on Computer Vision, pages 446–461, 2014. 6DOI
  61. [61]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 6
  62. [62]Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8349, 2021. 6
  63. [63]Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfruend, Joshua Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems, pages 9448–9458, 2019. 6
  64. [64]Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human action classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 6arxiv.org/abs/1212.0402
  65. [65]Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3485–3492, 2010. 6DOI
  66. [66]James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. CODA-Prompt: COntinual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909–11919, 2023. 6arxiv.org/abs/2211.13218
  67. [67]Saurav Jha, Dong Gong, and Lina Yao. CLAP4CLIP: Continual learning with probabilistic finetuning for vision-language models. In Advances in Neural Information Processing Systems, pages 129146–129186, 2024. 6, 15arxiv.org/abs/2403.19137
  68. [68]Christoph Schuhmann, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 6arxiv.org/abs/2111.02114
  69. [69]Kai Gan, Bo Ye, Min-Ling Zhang, and Tong Wei. Semi-supervised CLIP adaptation by enforcing semantic and trapezoidal consistency. In The Thirteenth International Conference on Learning Representations, 2025. 8
  70. [70]Maxime Zanella, Clément Fuchs, Christophe De Vleeschouwer, and Ismail Ben Ayed. Realistic test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 25103–25112, 2025. 21arxiv.org/abs/2501.03729

Paper details

vlnc:W_bst8sgp6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents