WorksComputer Science
Learning What the True Labels Would Teach: Posterior Analytic Class-Incremental Learning with Unlabeled Data
Abstract
In class-incremental learning with unlabeled data, each task brings only class names and unlabeled images, so CLIP's zero-shot predictions replace labels. Existing methods train on hard pseudo-labels, but a wrong one gives the true class zero probability, which self-training often confirms. We argue that each image should carry its label posterior, as an analytic classifier trained on posteriors equals the true-label one in expectation. The zero-shot prediction is not this posterior, as its class proportions confound name bias with true imbalance and it ignores other images. We propose Posterior Analytic Class-Incremental Learning (PACIL), which corrects the class prior with a prior selection rule and refines each distribution by a fixed-prior von Mises-Fisher mixture on whitened features. These soft labels train the classifier and replace labels to select its fusion weight with zero-shot CLIP by closed-form leave-one-out cross-validation. We prove that posterior targets minimize the mean-square gap to the true-label prediction and that their argmax widens this gap only through ambiguous images. The fusion weight selected with soft labels loses at most their distance to the posteriors in expected correct decisions. PACIL has the best average incremental accuracy among methods trained without labels on all 18 settings.
Introduction
Vision-language models such as CLIP [1] classify images of any named class, so their zero-shot predictions can supervise models on unlabeled images. Pseudo-labeling methods reduce each prediction to one class or a candidate set [2, 3], while transductive methods keep the whole distribution [4, 5] but only to label the images they are fitted on, not future ones. What target each image should carry when a classifier is trained on such predictions remains open.
This question is critical in class-incremental learning (CIL) [6], where a model learns disjoint classes in sequence and must recognize all seen classes without revisiting old
data. In CIL with unlabeled data [7, 8], each task brings only its new class names and unlabeled images. This arises when categories change faster than annotation, as for retail products [9] or camera-trap wildlife surveys, whose 3.2 million images would take over eight years of full-time work to label [10]. Earlier images may also be withheld for privacy [11], and each target is then fixed once its task ends. Existing methods, from CLIP-based CIL on zero-shot argmax labels [8, 12, 13, 14] to N2L [7] with self-training, all use hard pseudo-labels. Figure 1a measures how often the true label is discarded. A hard label discards it for most images, and keeping more top-scored labels lowers this rate slowly. N2L refines the hard labels three times, which may introduce confirmation bias [15]. With two to ten labels kept, it discards more true labels than zero-shot CLIP, as fitting a wrong label suppresses the true one. Only the full distribution discards no true label. The target of each image should therefore keep it and give the true label as much weight as the observations support.
Figure 1. Discarded true labels and average accuracy. (a) Training images (%) whose true label is discarded when each keeps only its top-scored labels, on the first task of Aircraft (half-base), for zero-shot CLIP, N2L after three self-training refinements and PACIL. (b) Average accuracy of the methods of Table 1 on its zero-base (Z) and half-base (H) settings, each axis spanning the lowest (inner ring) to the highest value (outer ring). Numbers are the of ZS-CLIP (gray), N2L (blue), and PACIL (red).
We argue that this target is the label posterior given all observations. Soft targets in distillation [16] and expectation-maximization (EM) [17] rest on the same idea, which becomes exact for analytic learning [11, 18], the exemplar-free learner we build on. Its solution is linear in the targets, so training on the posteriors yields the conditional mean of the classifier that the true labels would give. A hard label departs from this target on every ambiguous image, even when its argmax is correct. Because the posteriors also give the probability that each decision is correct, they can replace labels when choosing the weight that fuses the classifier with zero-shot CLIP. However, keeping the whole zero-shot distribution rather than its argmax is not enough. Its class proportions mix the name bias of CLIP with true imbalance [19, 20, 21] in shares that vary across datasets and tasks, and it ignores the other images of the task. Without labels or old images, three questions arise. (i) Which class proportions should the prior of each new task follow? (ii) How far should the images of a task revise each distribution when old classes are kept only as statistics? (iii) How much should predictions over all seen classes trust the analytic classifier rather than zero-shot CLIP?
We answer these questions with Posterior Analytic Class-Incremental Learning (PACIL), whose soft labels estimate the posteriors. For (i), a prior selection rule tests on the images of each task whether the skew of its zero-shot class proportions is name bias or true imbalance, which text alone cannot separate. For (ii), the selected distribution is a fixed per-image prior of a von Mises-Fisher (vMF) mixture fitted by EM on features whitened with statistics of all seen classes, so the images revise each distribution but not the class proportions. For (iii), closed-form leave-one-out (LOO) cross-validation picks the fusion weight of each task with the most correct decisions expected under the soft labels. In our analysis, rounding the posteriors to their argmax adds exactly the squared change this causes in the prediction, a cost that comes only from ambiguous posteriors. The distance of the soft labels to the posteriors bounds both the extra prediction error and what the selected fusion weight can lose in expected correct decisions. Prior selection and EM are therefore designed to reduce this distance before a task ends and its targets are fixed.
Our contributions are as follows. We show that hard pseudo-labels discard plausible classes and that their errors persist because targets are fixed after each task, and we therefore cast labeling as posterior estimation. We prove that posteriors are mean-square optimal targets for analytic learning and that they yield a label-free fusion criterion whose loss is bounded by the soft-label error. We propose PACIL, an exemplar-free method that estimates these posteriors and attains the best average incremental accuracy among label-free methods on every setting, especially where zero-shot CLIP is weakest (Figure 1b).
Related Work
Class-Incremental Learning with CLIP. Most CLIP-based CIL methods tune CLIP or added modules by gradient descent on the labeled images of each task while limiting forgetting [12, 14, 22, 23, 24, 25]. Analytic learning replaces gradient training by a recursively updated ridge regression on frozen features, reaching the joint-training solution without exemplars [11, 18, 26, 27]. RAIL [13] and VILA [28] fuse such a learner with zero-shot CLIP by a fixed rule, and AnytimeCL [29] weights a tuned model per class by labeled accuracy. All of these methods need labeled tasks. Class-incremental novel class discovery [30, 31] and unsupervised CIL [32] learn from unlabeled images without class names, and name-only continual learning [33] retrieves web images. N2L [7] introduced CIL with unlabeled data and trains an analytic learner on zero-shot argmax labels refined by self-training. CrossWorld-CL [8] tunes prompts and adapters on confident hard pseudo-labels in a similar setting, replaying retrieved ImageNet images. PACIL trains an analytic learner on estimated label posteriors and selects its fusion weight with them.
Learning from Unlabeled Images with CLIP. Methods that adapt CLIP to unlabeled images differ in how much of its zero-shot distribution they keep. Pseudo-labeling methods keep one class per image [2, 34, 35] or a candidate set [3] and tune CLIP’s prompts on them. Transductive methods keep the whole distribution and refine it over a set of test images, and test-time methods [36, 37] do so over a stream. Martin et al. [4] fit a Dirichlet mixture to CLIP’s probability vectors and re-estimate the class proportions of each batch. TransCLIP [38] fits a Gaussian mixture with shared covariance to the image features, regularized by a neighborhood graph and a KL term toward the zero-shot prediction. MOON [5] uses vMF components whose mean directions shrink toward the text embeddings. Without graph terms, both multiply each zero-shot prediction by a class-conditional likelihood, the form of our E-step.
Frolic [39] removes an estimated label bias under a uniform test prior and fuses CLIP with a feature-space classifier at a weight equalizing their average confidences. These methods either predict the images they are fitted on or train on hard labels. Unlike Martin et al. [4], TransCLIP [38] and Frolic [39], PACIL fixes the prior of each image from the prior selection rule and whitens features with the within class scatter of all seen classes.
Class Priors and Label Bias
Class priors are usually corrected with labels or a known source prior. Under label shift, Saerens et al. [40] re-estimate the target priors by EM and test whether they differ from the source prior. BBSE [41] inverts a confusion matrix estimated on labeled source data. GLA [20] fits the name bias of zero-shot CLIP as a per-class logit offset on labeled data. It adds the debiased logits to those of a fine-tuned model with equal weights, which is Bayes optimal if the two are independent given the label. Without labels, zero-shot predictions and pseudo-labels are rebalanced toward a uniform [42, 43, 44], given [45] or estimated marginal [36]. Wang et al. [46] estimate such a marginal in partial-label learning [47] and refine candidate-set labels toward it by Sinkhorn scaling. PACIL asks the question of Saerens et al. [40] without a source prior, namely whether the skew of the zero-shot marginal reflects the data. Since logits alone cannot separate name bias from true imbalance, it selects the raw or the debiased prior
from the images. For fusion, GLA’s equal weights rely on conditional independence, which fails because the targets of PACIL come from zero-shot outputs. PACIL therefore selects the fusion weight by closed-form LOO.
Method
As illustrated in Figure 2, for each task, PACIL estimates the posterior of the label of every unlabeled image, as the targets of the classifier and in place of labels when the fusion weight is chosen. Section 3.1 fixes the setting and shows why posteriors are the right targets for the analytic classifier. The next three subsections answer questions (i) to (iii) of Sec. 1 in the order of Figure 2. Class-prior correction (Sec. 3.2) chooses the class proportions that the labels of a new task follow. Posterior label-distribution estimation (Sec. 3.3) then refines each label distribution with the images of the task, and soft-label analytic learning and fusion (Sec. 3.4) trains the classifier on these soft labels and selects its fusion weight after each task.
Figure 2. Overview of PACIL on task . (1) Class-prior correction: the prior selection rule chooses between the raw and the debiased zero-shot distribution. (2) Posterior label-distribution estimation: EM of a vMF mixture on whitened features keeps as a fixed per-image prior and combines it with a tempered likelihood into the soft labels . (3) Soft-label analytic learning and fusion: the analytic classifier is fitted to , and closed-form LOO selects the weight that fuses it with zero-shot CLIP.
Problem Formulation
Setting. Following N2L [7], a model learns tasks in sequence, as when a shop or a camera-trap survey adds newly named products or species from unlabeled images. Task brings new classes , disjoint from those of other tasks, with only their names and a set of unlabeled images drawn from these classes. No label is revealed, even in the first task, which holds a tenth of the classes in the zero-base setting () and half in the half-base setting (). No earlier image or per-image feature is kept, and after task the model classifies test images over all seen classes without task identity. A frozen CLIP model with image encoder and text encoder gives the feature of image and the embedding of class , where and inserts the class name into the first prompt template. With CLIP’s logit scale and the logits , the raw zero-shot distribution of image , row of , is
Analytic Classifier. Like RanPAC [18], RAIL [13] and N2L, PACIL uses an analytic classifier, a ridge regression on random features with a fixed random matrix . Each image of task enters with its features , a weight and a target in the probability simplex , and after each task the classifier is updated in closed form [11],
where and accumulate weighted moments (, empty), is a zero block for the new classes, is the ridge coefficient and is the classifier, the ridge solution on all seen images although none is stored. Hard-label methods such as N2L set to the one-hot vector of a pseudo-label .
Posterior targets. The true label of each image is unknown, and the classifier that the true labels would give is Eq. (2) with . Let denote everything observed, the images and class names of all seen tasks, and let the weights be fixed. As is linear in its targets,
where the sum runs over all seen images and is the posterior distribution of . Training on the posterior of each label therefore gives the conditional mean of the classifier that the true labels would give. Labeling becomes posterior estimation, and Secs. 3.2 and 3.3 estimate the posterior of each image of task as a soft label .
Class-Prior Correction
The class proportions of a new task are unknown. Let be the class marginal of the zero-shot distribution and the uniform marginal with entries . The marginal departs from for two reasons that call for opposite actions. Name bias should be removed, since CLIP disfavors some class names, such as an aircraft variant named without its manufacturer, and gives these classes too little mass. True imbalance should be kept, since in datasets such as SUN the classes differ in size and the skew of reflects the data. Both act on the logits as one offset per class, and the text alone cannot tell them apart. PACIL therefore builds one candidate prior for each cause and selects between them from the images.
Two candidate priors. The raw candidate is itself. The debiased candidate removes any class-level offset from the logits. Starting from , Sinkhorn iterations [48] alternate, for every ,
until every class has mass . As , the iterations move from the raw distribution to the debiased distribution , the KL projection of onto distributions with equal class masses [49].
Prior selection rule. The prior selection rule tests whether the images reproduce the imbalance of under a prior that carries none. It runs the estimator of Sec. 3.3 from the debiased prior without tempering, , which gives the likelihood its full weight, and takes the class marginal . Since has a uniform marginal and the estimator never re-estimates the prior of an image, any departure of from comes from the images alone. The rule regresses this departure on that of the zero-shot marginal and selects the prior,
with if . The threshold is not tuned, since exactly when is closer to than to . fixes the class proportions of the task, but each of its rows rests on the zero-shot logits of a single image.
Posterior Label-Distribution Estimation
Images of a class tend to lie close together, the cluster assumption [50], which allows the images of a task to correct each other’s label distributions. Fitting a standard mixture model to CLIP features faces two obstacles. Cosines between CLIP features mix class differences with within-class variation such as pose and background, and such a mixture would re-estimate the class proportions that § 3.2 has fixed. PACIL therefore fits a mixture with fixed per-image priors on whitened features, whose statistics cover all seen classes without storing their images.
Whitening with accumulated statistics. PACIL compares images on the unit sphere after the whitening map
which centers at the mean of all seen features, keeps its coordinates along the top principal directions of their total covariance, and applies to make the within-class spread equal in every direction, so that cosines on become Mahalanobis cosines [51, 52]. Here is the Cholesky factor of , and is the within-class covariance after shrinkage toward a multiple of the identity with the oracle approximating shrinkage (OAS) coefficient [53], which keeps stable.
With few images per task, all with uncertain labels, is pooled over all seen classes. Earlier images enter only through statistics accumulated under their frozen soft labels, the second moment , the sum and the number of their features, and for each class the soft sum , a column of , and the soft count . A run of EM merges the current images and their current soft labels into these statistics, and a prime marks the merged values, , and , with and for each new class and and for earlier classes. They give , the directions as the top eigenvectors of , and
the soft within-class scatter of all seen images, whose scale cancels in OAS and in Eq. (6), so no earlier image is stored.
Fixed-prior vMF mixture. On this sphere each class is a von Mises–Fisher (vMF) component with density proportional to , mean direction and a concentration shared by all classes. The shared cancels the normalizers, so the likelihood has no class-specific term, unlike a shared-covariance Gaussian whose term favors classes with short means. For a prior , the mixing weights of image are fixed to . The class proportions chosen in § 3.2 enter every E-step and are never re-estimated. Starting from , EM alternates between an M-step for the mean directions and the concentration , and an E-step for the soft labels,
where is the mean resultant length and is the approximation of [54] to its maximum-likelihood value. The E-step has the form of the assignment step of TransCLIP [38] without its graph term. The temperature tempers the likelihood, so that small cosine gaps, scaled by the fitted , cannot override the text prior, and gives the untempered posterior. EM runs in two passes, the second recomputing , and from the first-pass posterior with and fixed, since the prior alone gives only a coarse within-class estimate. This defines . The soft labels of task are , after which are added to and never revised.
Soft-Label Analytic Learning and Fusion
The soft labels of a task are used twice before they are discarded. They first train the analytic classifier and then, in place of labels, select the weight that fuses it with zero-shot CLIP in predictions over all seen classes.
Soft-label analytic learning. Following Eq. (3), the target of image is its soft label padded with zeros on the old classes, . For any output , , so the squared error of Eq. (2) fits the label distribution rather than its argmax. To keep large classes from dominating, the weights balance classes by their soft counts ,
where is the number of classes with at least unit mass, and equals the class-balancing factor of N2L for one-hot labels.
Fusion weight selection. At test time, PACIL predicts the class in with the largest fused score
where is the zero-shot distribution over all seen classes with each class embedding averaged over all prompt templates, and weights the classifier. The right weight changes across datasets and tasks. Choosing it requires predictions that, like test predictions, come from a classifier that has not seen the image. The fitted output does not qualify, since each image has pulled toward its own target. PACIL instead uses the closed-form LOO output of ridge regression [55, 56],
where is the leverage of image and is the output at of the classifier re-solved with $m_i=0. After training on task , the soft labels score the fused LOO decisions in place of the labels,
where is the softmax of over all seen classes and , with ties resolved by their median. Each decision earns the probability that its soft label gives to the chosen class, so the criterion counts the correct decisions expected under the soft labels, and a decision for an old class earns nothing. Across tasks, PACIL stores only and , whose size does not grow with the number of seen images, and Algorithm 1 in Supp. B lists every step.
Theoretical analysis. Supp. A justifies both uses of the soft labels, the first in squared error and the second in expected correct decisions. As targets, the posteriors minimize the mean-square gap to the true-label prediction, and any other targets add at most a multiple of their squared distance to them. Their argmax adds exactly the squared change it causes in the prediction. PACIL therefore trains on soft labels, and § 3.2 and § 3.3 aim to bring them close to the posteriors. In place of labels, the soft labels make the criterion of Eq. (12) the expected number of correct LOO decisions up to their error, which bounds what can lose.
Experiments
We first describe the setup (§ 4.1), then compare PACIL with exemplar-free methods on six datasets (§ 4.2) and ablate its three modules (§ 4.3). The supplement compares PACIL on Food, ImageNet-R and SUN and with exemplar-based methods (Supp. C.1), tests its robustness to , , the CLIP backbone (Supp. C.2) and noisy task boundaries (Supp. C.3), gives N2L two components of PACIL (Supp. C.4) and PACIL the soft labels of transductive methods (Supp. C.5), and reports pseudo-label quality, per-task accuracy, and training and inference cost (Supp. C.6 to C.8).
Experimental Setup
Datasets. We use the nine datasets with the class subsets and train/test splits adopted from N2L [7], PROOF [24] and ENGINE [14]. They cover fine-grained categories in Aircraft [57], Cars [58], CUB [59] and Food [60], generic objects in CIFAR100 [61], distribution shift in ImageNet-R [62] and ObjectNet [63], human actions in UCF [64] and scenes in SUN [65]. SUN has 300 classes, CUB, ObjectNet and ImageNet-R have 200, and the others have 100. denotes a first task of classes followed by tasks of classes, with meaning that the first task also has classes. The zero-base setting splits all classes evenly into 10 tasks (B0Inc10, B0Inc20 or B0Inc30 for 100, 200 or 300 classes). The half-base setting puts half of the classes in the first task and splits the other half evenly into 5 tasks (B50Inc10, B100Inc20 or B150Inc30). Each task, including the first, provides only the names of its new classes and their unlabeled training images.
Evaluation Metrics. Following N2L, after each task the model classifies the test images of all classes seen so far without task identity. Its accuracy is , where is the set of test images with labels in the seen classes , is the class predicted after task , and the indicator is 1 if its argument holds and 0 otherwise. We report the last accuracy after the final task and the average incremental accuracy , both in percent.
Comparison Methods. We compare with the exemplar-free methods evaluated by N2L. ZS-CLIP [1] classifies by class names without training. CODA-Prompt [66], MoE-Adapter [22], RAPF [23], ENGINE [14] and RAIL [13] are designed for labeled tasks, so N2L trains them on the argmax of the zero-shot prediction of each image as its pseudo-label. CODA-Prompt tunes prompts of the image encoder, MoE-Adapter, RAPF and ENGINE (without its re-ranking) adapt CLIP by gradient descent, and RAIL fits an analytic classifier. N2L [7] starts from the same pseudo-labels, refines them in three iterations with a ridge classifier on dimension-reduced features, and weights each image by its class frequency and confidence. We also compare with the exemplar-based PROOF [24] and CLAP4CLIP [67], which store 20 images per class of earlier tasks (Supp. C.1). N2L+Label and PACIL+Label replace the pseudo-labels of N2L and the soft labels of PACIL by ground-truth labels, and serve as reference upper bounds that are not ranked.
Implementation Details. PACIL uses the frozen CLIP ViT-B/16 of N2L, pre-trained on LAION-400M [68], and its random features with . As in N2L, class names are embedded with the first prompt template of each dataset during training and with the ensemble of all its templates at test time. All datasets share one hyperparameter setting, , and , and after training on each task, is selected from by the closed-form LOO criterion of Eq. (12). Images are encoded in batches of 64. Each result is the mean of three runs with seeds 1993, 1994 and 1995 on one NVIDIA A100 GPU with PyTorch. The best and second-best results in each column are marked in bold and _underlined_, respectively. We will release the code upon the manuscript acceptance.
Main Results
Table 1 covers the six datasets on which N2L evaluates all baselines. Among the methods trained without labels, PACIL has the best on all 12 settings and the best on 11 of them. N2L is the strongest compared method throughout, and in the Mean columns PACIL improves on it by 3.19% and 2.49% in on the two blocks and by 4.00% and 1.80% in , the latter held down by ObjectNet, with gains of 2.83% and 2.85% on average in the zero-base and half-base settings. The gains over N2L are largest on Aircraft, 7.62% and 7.06% in and 9.49% and 7.38% in on B0Inc10 and B50Inc10. On this dataset zero-shot CLIP is the weakest, with an of 17.16%, and ground-truth labels raise the of N2L the most, by 22.65% and 29.49%. The pseudo-labels of N2L are least reliable here. This is the case that motivates PACIL in Sec. 1, since the zero-shot distribution keeps plausible classes that the argmax discards. PACIL falls behind only in on ObjectNet B100Inc20, 0.87% below N2L, and on ObjectNet B0Inc20 it gains 3.75% in but 0.03% in . ObjectNet has the second weakest zero-shot CLIP and varies rotation, background and viewpoint by design, so both the zero-shot prior and the feature clusters used by EM are weak, and the early gain of PACIL fades by the last task.
| Method | Venue | CUB B0Inc20 | CUB B0Inc20 | CUB B100Inc20 | CUB B100Inc20 | ObjectNet B0Inc20 | ObjectNet B0Inc20 | ObjectNet B100Inc20 | ObjectNet B100Inc20 | UCF B0Inc10 | UCF B0Inc10 | UCF B50Inc10 | UCF B50Inc10 | Mean | Mean |
| ZS-CLIP | ICML’21 | 75.47 | 63.72 | 69.06 | 63.72 | 38.43 | 26.43 | 31.12 | 26.43 | 75.88 | 67.79 | 71.68 | 67.79 | 60.27 | 52.65 |
| CODA-Prompt | CVPR’23 | 67.40 | 51.91 | 53.24 | 37.64 | 38.25 | 26.23 | 26.22 | 19.22 | 81.61 | 75.18 | 67.06 | 58.87 | 55.63 | 44.84 |
| MoE-Adapter | CVPR’24 | 73.69 | 63.07 | 68.29 | 63.09 | 40.14 | 28.70 | 32.45 | 28.53 | 75.35 | 67.36 | 72.28 | 69.07 | 60.37 | 53.30 |
| RAPF | ECCV’24 | 72.08 | 58.14 | 62.27 | 56.02 | 39.16 | 25.88 | 28.93 | 24.70 | 85.99 | 81.10 | 75.70 | 74.31 | 60.69 | 53.36 |
| RAIL | NeurIPS’24 | 81.64 | 73.93 | 73.08 | 70.91 | 39.80 | 35.13 | 31.21 | 30.19 | 90.18 | 89.90 | 81.05 | 84.27 | 66.16 | 64.06 |
| ENGINE | ICCV’25 | 77.06 | 65.07 | 70.85 | 64.92 | 44.57 | 31.24 | 34.62 | 29.96 | 87.85 | 84.46 | 82.30 | 81.04 | 66.21 | 59.45 |
| N2L | ICLR’26 | 83.41 | 76.48 | 75.16 | 73.40 | 49.31 | 41.59 | 41.42 | 38.65 | 95.00 | 93.29 | 86.41 | 87.87 | 71.79 | 68.55 |
| PACIL | Ours | 86.71 | 80.38 | 78.76 | 77.21 | 53.06 | 41.62 | 41.88 | 37.78 | 95.92 | 94.72 | 89.35 | 90.40 | 74.28 | 70.35 |
| N2L+Label | ICLR’26 | 86.43 | 79.05 | 82.29 | 79.06 | 53.18 | 45.27 | 49.71 | 45.26 | 98.74 | 97.75 | 98.37 | 97.75 | 78.12 | 74.02 |
| PACIL+Label | Ours | 87.43 | 81.23 | 83.89 | 81.24 | 58.89 | 46.19 | 51.67 | 46.15 | 98.67 | 97.40 | 98.22 | 97.36 | 79.80 | 74.93 |
Table 1 (PDF p. 7). Comparison with exemplar-free methods on six datasets. and in %, and Mean averages the six settings of each block.
The +Label rows measure the cost of the missing labels. PACIL+Label is above PACIL by 5.29% and 9.79% in on average in the zero-base and half-base settings, and by 3.81% and 7.76% in . PACIL+Label reaches nearly the same in both settings, whereas PACIL is lower in the half-base setting on all six datasets. Its first task must then be labeled as a 50-way or 100-way problem instead of a 10-way or 20-way one, and labels that enter the classifier are never corrected. The gap is largest on Aircraft, 20.04% and 28.68% in , and below 1% in both metrics on CIFAR100 B0Inc10 and CUB B0Inc20, where PACIL is also above N2L+Label. PACIL+Label is in turn above N2L+Label in both metrics on 10 of the 12 settings, all except UCF.
Ablation Study
Class-Prior Correction. We replace the prior selection rule of Eq. (5) by or on every task ( only, only). Each fails on a different dataset in Figure 3. only loses 5.26% in on Aircraft B0Inc10, where the classes are balanced and the skew of the zero-shot marginal is name bias. only loses 4.07% on SUN B0Inc30, where the classes differ in size and debiasing erases a real imbalance. The prior selection rule matches the better fixed prior on both, and on UCF B0Inc10 it switches priors across tasks and exceeds both. Its gain over the better fixed prior is small, however, none on Aircraft and SUN and 0.18% on UCF, so the rule mainly guards against a wrong fixed prior.
Figure 3. Ablation of class-prior correction. (%) on the zero-base settings of Aircraft, SUN and UCF with or fixed on every task and with the prior selection rule of PACIL.
Posterior Label-Distribution Estimation. Keeping the prior selection rule, we change only how becomes the soft labels. Hard labels replaces the label posteriors with hard pseudo-labels, w/o EM trains directly on , and w/o whitening runs EM with in Eq. (6). All three are below PACIL in both metrics on the eight settings of Figure 4. Removing EM costs up to 4.18% in on UCF B50Inc10, in line with the cluster assumption that the images of a task correct each other’s distributions (§ 3.3). Without whitening, EM loses up to 2.41% on ObjectNet B100Inc20, whose images vary in pose and background, but about 0.2% on UCF B50Inc10 and Aircraft B0Inc10. Taking the hard labels lowers further on all eight settings, most on Aircraft B0Inc10 with 1.99%, where the distribution keeps plausible classes that the weak zero-shot argmax misses. Keeping the whole distribution helps before EM is run, and EM then brings it closer to the posterior of each label. In , the argmax is better than w/o EM on both CUB settings, by 0.95% and 0.54%.
Figure 4. Ablation of posterior label-distribution estimation. Change in (a) and (b) (%) relative to PACIL when hard pseudo-labels replace the label posteriors (Hard labels), the final EM is skipped (w/o EM) or run with (w/o whitening), on the zero-base (Zero) and half-base (Half) settings of four datasets.
Fusion Weight Selection. We fix on every task as N2L does, which leaves the classifier unchanged. Selecting by Eq. (12) is better on every setting, by 0.71% in and 1.16% in on average (Figure 5). Across the 144 tasks, the selected ranges from 0.72 to 1.00 with a mean of 0.90, and it is above the fixed 0.8 of N2L on 134 of them (Figure 5c). The gain is small on ImageNet-R, 0.16% and 0.14% in , whose mean of about 0.84 and 0.86 is nearest 0.8. It is largest on ObjectNet, 3.62% and 2.48% in , though its mean is near 0.90, so ObjectNet is more sensitive to the weight than in need of a different one.
Figure 5. Ablation of fusion weight selection. (a, b) Gain of PACIL over the fixed of N2L in and (%) on all 18 settings. (c) selected on each task (dots, mean over three runs) and its mean per setting (ticks). Bottom rows give the means over all settings, and in (c) the line spans the range of .
Conclusion
When class-incremental learning has only class names and unlabeled images, each image should carry its label posterior as its training target. An analytic classifier is linear in its targets, so training it on posteriors recovers the true-label classifier in expectation. PACIL estimates them with a prior selection rule and a fixed-prior vMF mixture on whitened features, then uses the soft labels as targets and to select the fusion weight by closed-form LOO. In our analysis, no targets beat the posteriors in mean square, and rounding them to hard labels adds error only through ambiguous images. The error of the soft labels bounds the loss of the selected weight. PACIL leads methods trained without labels in average incremental accuracy, most clearly where zero-shot CLIP is weakest, and its lead over N2L persists across six CLIP backbones. Its corrections are limited by how well CLIP relates class names to images, and a large gap to labeled training remains. Language descriptions or queried labels [69] could narrow this gap, and future work should also admit unlabeled images from unknown classes.
Theoretical Analysis
Without labels, PACIL must decide what its analytic classifier is trained on and how much its prediction trusts this classifier. Theorem S1 answers both questions for the prediction that PACIL deploys after each task, and Lemma S1 shows that the leave-one-out (LOO) outputs that select the fusion weight come from classifiers that have not seen the image, just as test predictions do. Section A.1 formulates the problem and states both, Sec. A.2 discusses what they imply for PACIL, and Sec. A.3 proves them.
Problem Formulation and Main Theorem
After task , every seen image has its features , its weight and its soft label , padded with zeros on the other classes of and frozen after its task. For targets , one vector per seen image, the classifier fitted to them is
and the classifier of Eq. (2) is by Step 1 of the proof. Given everything observed, the only unknowns are the true labels. For any joint law of images and labels, we therefore condition on the -algebra generated by the observations after task and by , and write for the expectation given . Every quantity that PACIL computes, such as , the weights, the zero-shot scores and , is then fixed, while the label of image has the posterior
With the classifier fitted to , Eq. (10) predicts
so PACIL predicts with , and the true labels would give with the same and . We compare targets by their risk at an input , and write for the leverage of and for the conditional variance of the output that the true labels would give, which no choice of targets affects,
For the fusion weight, let be the decision of Eq. (12) for a current image , so that maximizes the criterion and counts the correct decisions, where
Let maximize , and let hold the current images that and decide differently.
Theorem S1 (Keep the posterior, not the argmax). (a) The posteriors give the conditional mean, , and for every input and all -measurable targets ,
with , so the posteriors attain the smallest risk, . (b) Let , with be the hard label of image , and let
be the change of the output that the argmax causes, which needs no label, and the error of the soft labels at . Then
so when . (c) For every , the criterion splits as
and the selected weight satisfies
Lemma S1 (Closed-form LOO). For , the leverage of Eq. (11) lies in , and is the output at of the classifier of Eq. (2) re-solved with , all other seen images keeping their weights and targets.
Discussion
By (a), the posteriors are the targets that bring the deployed prediction closest, in mean square, to the one that the true labels would give, and other targets add at most times their weighted squared distance to the posteriors. The sum runs over all seen images, and a soft label is frozen after its task while its image is discarded, so an error made on one task stays in the bound for every later task. PACIL therefore estimates the posteriors, with the prior selection rule of § 3.2 and the EM of § 3.3, before it freezes the soft labels for all later tasks. By (b), if the soft labels are the posteriors, replacing them by their argmax raises the risk by exactly , the squared change of the prediction. If they are estimates of the posteriors, the argmax raises the risk whenever and . The argmax discards the ambiguity of each posterior, since a one-hot target stays at squared distance
from the posterior for every class , which is large on ambiguous images, whereas a soft label can match the posterior. Section C.6 measures how far the soft labels of PACIL and the hard pseudo-labels of zero-shot CLIP [1] and N2L [7] are from the true labels.
By Lemma S1, the LOO output of an image is the output of a classifier that has not seen it, like a test prediction, and costs no re-solving. By (c), the criterion of Eq. (12) is the expected number of correct LOO decisions plus an error term that vanishes when the soft labels are the posteriors. Against the weight with the most correct decisions in expectation, the selected weight then loses at most the distance of the soft labels to the posteriors, summed over the images that the two weights decide differently.
Parts (a) and (b) compare predictions in squared error and part (c) counts correct LOO decisions on the current images, so none of them bounds the test accuracy, and the argmax can be more accurate, as for on CUB in Figure 4. The results hold for any soft labels computed from the observations, including those of the prior selection rule and of EM with . They state what these modules should aim at, not that they reach it. The reference prediction keeps the weights that Eq. (9) computes from the soft labels. The LOO of Lemma S1 removes an image only from the ridge step, since the soft labels and weights computed with this image are kept.
Proofs
Proof of Theorem S1. Given , the labels follow a probability vector over the finitely many label configurations, is the expectation under this vector, and every -measurable quantity is a constant under it. We write and , and runs over all seen images.
Step 1: the recursion is ridge regression on all seen images. By induction on , the recursion of Eq. (2) gives
Written over all classes, has zero columns for the classes of task , as every soft label is zero outside the classes of its own task, so the zero block of Eq. (2) loses nothing. Hence , although no seen image is stored once its task ends.
Step 2: conditional mean and risk. is linear in with the -measurable coefficients , so , and because and are constants. With and , the zero-shot term cancels in
As , the cross term of
has conditional mean zero, so
which is the identity of (a) and is smallest at .
Step 3: target errors pass through the leverage. As is symmetric, the target errors enter the output at as
where , so that . The triangle and Cauchy–Schwarz inequalities give
The inner product of with gives
so , which proves the upper bound of (a). The same identity gives , and Cauchy–Schwarz gives , so
and as .
Step 4: posterior versus argmax. The error of the soft labels at is with , and applying Eq. (S15) to and gives
@page 13/24 starts_mid=0 ends_mid=1
Discussion
By (a), the posteriors are the targets that bring the deployed prediction closest, in mean square, to the one that the true labels would give, and other targets add at most times their weighted squared distance to the posteriors. The sum runs over all seen images, and a soft label is frozen after its task while its image is discarded, so an error made on one task stays in the bound for every later task. PACIL therefore estimates the posteriors, with the prior selection rule of § 3.2 and the EM of § 3.3, before it freezes the soft labels for all later tasks. By (b), if the soft labels are the posteriors, replacing them by their argmax raises the risk by exactly , the squared change of the prediction. If they are estimates of the posteriors, the argmax raises the risk whenever and . The argmax discards the ambiguity of each posterior, since a one-hot target stays at squared distance
from the posterior for every class , which is large on ambiguous images, whereas a soft label can match the posterior. Section C.6 measures how far the soft labels of PACIL and the hard pseudo-labels of zero-shot CLIP [1] and N2L [7] are from the true labels.
By Lemma S1, the LOO output of an image is the output of a classifier that has not seen it, like a test prediction, and costs no re-solving. By (c), the criterion of Eq. (12) is the expected number of correct LOO decisions plus an error term that vanishes when the soft labels are the posteriors. Against the weight with the most correct decisions in expectation, the selected weight then loses at most the distance of the soft labels to the posteriors, summed over the images that the two weights decide differently.
Parts (a) and (b) compare predictions in squared error and part (c) counts correct LOO decisions on the current images, so none of them bounds the test accuracy, and the argmax can be more accurate, as for on CUB in Figure 4. The results hold for any soft labels computed from the observations, including those of the prior selection rule and of EM with . They state what these modules should aim at, not that they reach it. The reference prediction keeps the weights that Eq. (9) computes from the soft labels. The LOO of Lemma S1 removes an image only from the ridge step, since the soft labels and weights computed with this image are kept.
Proofs
Proof of Theorem S1. Given , the labels follow a probability vector over the finitely many label configurations, is the expectation under this vector, and every -measurable quantity is a constant under it. We write and , and runs over all seen images.
Step 1: the recursion is ridge regression on all seen images. By induction on , the recursion of Eq. (2) gives
Written over all classes, has zero columns for the classes of task , as every soft label is zero outside the classes of its own task, so the zero block of Eq. (2) loses nothing. Hence , although no seen image is stored once its task ends.
Step 2: conditional mean and risk. is linear in with the -measurable coefficients , so , and because and are constants. With and , the zero-shot term cancels in
As , the cross term of
has conditional mean zero, so
which is the identity of (a) and is smallest at .
Step 3: target errors pass through the leverage. As is symmetric, the target errors enter the output at as
where , so that . The triangle and Cauchy–Schwarz inequalities give
The inner product of with gives
so , which proves the upper bound of (a). The same identity gives , and Cauchy–Schwarz gives , so
and as .
Step 4: posterior versus argmax. The error of the soft labels at is with , and applying Eq. (S15) to and gives
and gives (b). The bound on is Step 3 with , and gives . In Eq. (S11), the identity holds as and , and the inequality follows from , as .
Step 5: the fusion criterion. As each decision is -measurable, it is correct with probability
and summing over gives Eq. (21). This identity at and , with , gives
since the images decided alike contribute the same term at both weights. Each summand involves two different classes and and is at most . The lower bound holds because maximizes . □
Proof of Lemma S1. Setting the weight of image to zero replaces and by
and is positive definite. Let and . Then
since . For , we have , hence , and as is symmetric, the re-solved classifier outputs at
since and . □
More Implementation Details
This section lists the datasets and task splits of our experiments and gives the full algorithm of PACIL.
Datasets and task splits. Table S1 lists the class subsets, the image counts and the two settings of each dataset. The counts are those of the features extracted in our runs, which use the train and test splits of N2L [7].
| Dataset | Classes | Train | Test | Settings |
| Aircraft | 100 | 6,667 | 3,333 | B0Inc10, B50Inc10 |
| Cars | 100 | 4,135 | 4,083 | B0Inc10, B50Inc10 |
| CIFAR100 | 100 | 50,000 | 10,000 | B0Inc10, B50Inc10 |
| CUB | 200 | 9,430 | 2,358 | B0Inc20, B100Inc20 |
| ObjectNet | 200 | 26,509 | 6,628 | B0Inc20, B100Inc20 |
| UCF | 100 | 10,553 | 2,639 | B0Inc10, B50Inc10 |
| Food | 100 | 79,988 | 20,012 | B0Inc10, B50Inc10 |
| ImageNet-R | 200 | 24,000 | 6,000 | B0Inc20, B100Inc20 |
| SUN | 300 | 72,870 | 18,179 | B0Inc30, B150Inc30 |
Table S1. Datasets and task splits. Train and Test are the numbers of training and test images, and BInc is a first task of classes followed by tasks of classes each.
| Algorithm 1 PACIL on task |
| Require: class names of , features , statistics and , fixed $R |
| Ensure: classifier , fusion weight , updated statistics |
| 1: Text embeddings from the first prompt template, |
| 2: by Eq. (1), by Sinkhorn (Eq. (4)) |
| 3: |
| 4: from the marginals of and , select (Eq. (5)) |
| 5: with |
| 6: Update with and |
| 7: , weights by Eq. (9) |
| 8: Update and solve by Cholesky, (Eq. (2)) |
| 9: LOO predictions (Eq. (11)), by Eq. (12) with from , |
| 10: Discard , and |
| Inference: prompt-ensemble embeddings , , predict with Eq. (10) |
| 11: function EM |
| 12: and () from the statistics merged with |
| 13: and |
| 14: for pass do |
| 15: by Eq. (7), OAS shrinkage, Cholesky factor , whitened |
| 16: Alternate the M-step and the E-step of Eq. (8) for at most 21 iterations, until the mean of , with the denominator of the E-step, changes by less than |
| 17: end for |
| 18: return |
| 19: end function |
Table 1 (PDF p. 14).
Algorithm. Algorithm 1 gives the steps that PACIL runs on task . Lines 1 and 2 compute the raw zero-shot distribution with the first prompt template and its debiased version with Sinkhorn iterations. Lines 3 and 4 form the prior selection rule, which runs EM from without tempering and keeps only if the images reproduce its class imbalance. Line 5 runs EM again from the selected prior with , and line 6 adds the images and their soft labels to the whitening statistics. Lines 7 to 9 fit the analytic classifier to the soft labels with class-balancing weights and select from its closed-form LOO outputs. Line 10 then discards the features and soft labels of the task, so later tasks see it only through the accumulated statistics. Each EM call computes the mean and the principal directions once and makes two passes, each re-estimating the within-class covariance and whitening before the vMF iterations. Because EM only reads the statistics, the EM run of the prior selection rule leaves no trace in them.
Additional Experiments
More Comparisons
Food, ImageNet-R and SUN. Table S2 completes Table 1 with the three datasets that N2L [7] reports in its Tab. 6, which gives only ZS-CLIP [1], ENGINE [14], N2L, ENGINE+Label and N2L+Label for them. PACIL is the best of the methods trained without labels in both metrics on all six settings, and its gains over N2L range from 0.42% to 1.45% on these datasets, where the refined pseudo-labels of N2L are between 85.70% and 96.22% correct. PACIL+Label is above PACIL by 0.23% to 2.90% in and below N2L+Label in both metrics on both settings of SUN, the dataset with the most classes. In the Mean columns, PACIL is 0.94% and 0.88% above N2L in and , far less than on the two blocks of Table 1, and ground-truth labels add only 1.24% and 0.85% to its Mean.
| Method | Venue | Aircraft B0Inc10 | Aircraft B0Inc10 | Aircraft B50Inc10 | Aircraft B50Inc10 | Cars B0Inc10 | Cars B0Inc10 | Cars B50Inc10 | Cars B50Inc10 | CIFAR100 B0Inc10 | CIFAR100 B0Inc10 | CIFAR100 B50Inc10 | CIFAR100 B50Inc10 | Mean | Mean |
| ZS-CLIP | ICML’21 | 26.61 | 17.16 | 21.66 | 17.16 | 82.90 | 76.73 | 78.74 | 76.73 | 81.81 | 71.38 | 76.49 | 71.38 | 61.37 | 55.09 |
| CODA-Prompt | CVPR’23 | 26.56 | 17.79 | 18.11 | 12.28 | 79.69 | 69.03 | 67.81 | 56.59 | 79.36 | 67.74 | 66.94 | 54.64 | 56.41 | 46.35 |
| MoE-Adapter | CVPR’24 | 26.73 | 17.92 | 22.19 | 17.52 | 83.16 | 77.02 | 79.21 | 77.72 | 81.86 | 71.53 | 76.57 | 71.49 | 61.62 | 55.53 |
| RAPF | ECCV’24 | 29.07 | 19.52 | 20.83 | 18.46 | 83.77 | 73.26 | 75.73 | 71.56 | 84.47 | 76.08 | 77.30 | 74.33 | 61.86 | 55.54 |
| RAIL | NeurIPS’24 | 36.23 | 33.59 | 23.75 | 25.99 | 88.64 | 84.68 | 82.72 | 82.13 | 87.34 | 80.37 | 81.44 | 78.89 | 66.69 | 64.28 |
| ENGINE | ICCV’25 | 34.77 | 25.41 | 26.94 | 23.56 | 86.90 | 78.76 | 82.67 | 79.93 | 85.15 | 77.11 | 79.89 | 76.15 | 66.05 | 60.15 |
| N2L | ICLR’26 | 43.73 | 40.21 | 29.69 | 32.42 | 92.38 | 87.50 | 86.42 | 85.45 | 87.80 | 81.13 | 82.92 | 80.30 | 70.49 | 67.84 |
| PACIL | Ours | 51.35 | 49.70 | 36.75 | 39.80 | 92.73 | 89.40 | 88.53 | 87.95 | 88.82 | 82.71 | 83.87 | 81.49 | 73.68 | 71.84 |
| N2L+Label | ICLR’26 | 66.38 | 56.31 | 59.18 | 56.31 | 93.57 | 89.15 | 90.92 | 89.14 | 88.52 | 81.92 | 85.23 | 81.92 | 80.63 | 75.79 |
| PACIL+Label | Ours | 71.39 | 61.78 | 65.43 | 61.78 | 94.27 | 91.21 | 92.17 | 91.12 | 89.68 | 83.55 | 86.50 | 83.55 | 83.24 | 78.83 |
Table 1. Comparison with exemplar-free methods on six datasets. and in %, and Mean averages the six settings of each block.
| Method | Venue | Food B0Inc10 | Food B0Inc10 | Food B50Inc10 | Food B50Inc10 | ImageNet-R B0Inc20 | ImageNet-R B0Inc20 | ImageNet-R B100Inc20 | ImageNet-R B100Inc20 | SUN B0Inc30 | SUN B0Inc30 | SUN B150Inc30 | SUN B150Inc30 | Mean | Mean |
| ZS-CLIP | ICML’21 | 87.86 | 81.99 | 84.78 | 81.99 | 83.12 | 76.62 | 79.23 | 76.62 | 79.45 | 72.14 | 74.99 | 72.14 | 81.57 | 76.92 |
| ENGINE | ICCV’25 | 88.30 | 81.78 | 84.72 | 81.55 | 84.14 | 76.93 | 79.26 | 76.56 | 83.04 | 76.17 | 78.06 | 75.10 | 82.92 | 78.02 |
| N2L | ICLR’26 | 90.58 | 85.50 | 87.18 | 85.01 | 86.00 | 81.06 | 81.89 | 80.29 | 84.38 | 78.85 | 79.15 | 77.40 | 84.86 | 81.35 |
| PACIL | Ours | 91.13 | 86.25 | 87.94 | 85.66 | 86.86 | 81.48 | 82.82 | 80.99 | 85.46 | 80.14 | 80.58 | 78.85 | 85.80 | 82.23 |
| ENGINE+Label | ICCV’25 | 88.52 | 81.85 | 85.08 | 81.59 | 84.61 | 77.12 | 79.96 | 76.78 | 83.51 | 76.21 | 78.83 | 75.20 | 83.42 | 78.13 |
| N2L+Label | ICLR’26 | 90.95 | 85.84 | 88.38 | 85.84 | 86.61 | 81.52 | 83.88 | 81.51 | 87.42 | 81.50 | 84.31 | 81.50 | 86.93 | 82.95 |
| PACIL+Label | Ours | 91.36 | 86.49 | 88.84 | 86.50 | 87.49 | 82.07 | 84.32 | 82.12 | 86.72 | 80.63 | 83.48 | 80.64 | 87.04 | 83.08 |
Table S2. Comparison on Food, ImageNet-R and SUN. and in %, and Mean averages the six settings.
Exemplar-based methods. Table S3 compares PACIL with two exemplar-based methods, PROOF [24] and CLAP4CLIP [67], which keep 20 training images per class of earlier tasks. N2L ran both in CIL with unlabeled data, training them on pseudo-labels from zero-shot CLIP, as its other baselines, and their results, as well as those of ZS-CLIP, are quoted from tbl:4 of [7], which covers the 12 settings of Aircraft, Cars, CIFAR100, CUB, ObjectNet and UCF. PACIL stores no image and is above both methods in and on all 12 settings, although only by 0.03% and 0.01% on ObjectNet B100Inc20, where it is level with CLAP4CLIP. Averaged over the 12 settings, it reaches 73.98% in and 71.10% in , against 68.71% and 64.79% for CLAP4CLIP, the stronger of the two exemplar-based methods. The Mean columns show this margin on both blocks, 5.85% and 4.69% in and 7.46% and 5.15% in over CLAP4CLIP.
| Method | Venue | Aircraft B0Inc10 | Aircraft B0Inc10 | Aircraft B50Inc10 | Aircraft B50Inc10 | Cars B0Inc10 | Cars B0Inc10 | Cars B50Inc10 | Cars B50Inc10 | CIFAR100 B0Inc10 | CIFAR100 B0Inc10 | CIFAR100 B50Inc10 | CIFAR100 B50Inc10 | Mean | Mean |
| ZS-CLIP | ICML’21 | 26.61 | 17.16 | 21.66 | 17.16 | 82.90 | 76.73 | 78.74 | 76.73 | 81.81 | 71.38 | 76.49 | 71.38 | 61.37 | 55.09 |
| CLAP4CLIP | NeurIPS’24 | 41.15 | 35.81 | 26.88 | 28.43 | 90.02 | 86.46 | 79.68 | 75.88 | 86.34 | 79.11 | 82.90 | 80.57 | 67.83 | 64.38 |
| PROOF | TPAMI’25 | 38.10 | 33.62 | 28.10 | 28.15 | 89.23 | 85.09 | 83.83 | 83.03 | 84.85 | 77.47 | 79.17 | 76.36 | 67.21 | 63.95 |
| PACIL | Ours | 51.35 | 49.70 | 36.75 | 39.80 | 92.73 | 89.40 | 88.53 | 87.95 | 88.82 | 82.71 | 83.87 | 81.49 | 73.68 | 71.84 |
| Method | Venue | CUB B0Inc20 | CUB B0Inc20 | CUB B100Inc20 | CUB B100Inc20 | ObjectNet B0Inc20 | ObjectNet B0Inc20 | ObjectNet B100Inc20 | ObjectNet B100Inc20 | UCF B0Inc10 | UCF B0Inc10 | UCF B50Inc10 | UCF B50Inc10 | Mean | Mean |
| ZS-CLIP | ICML’21 | 75.47 | 63.72 | 69.06 | 63.72 | 38.43 | 26.43 | 31.12 | 26.43 | 75.88 | 67.79 | 71.68 | 67.79 | 60.27 | 52.65 |
| CLAP4CLIP | NeurIPS’24 | 82.45 | 74.85 | 75.74 | 72.71 | 49.00 | 38.55 | 41.85 | 37.77 | 87.10 | 83.82 | 81.38 | 83.50 | 69.59 | 65.20 |
| PROOF | TPAMI’25 | 80.50 | 73.98 | 73.74 | 71.83 | 45.46 | 34.93 | 35.36 | 32.16 | 89.41 | 86.00 | 81.23 | 81.76 | 67.62 | 63.44 |
| PACIL | Ours | 86.71 | 80.38 | 78.76 | 77.21 | 53.06 | 41.62 | 41.88 | 37.78 | 95.92 | 94.72 | 89.35 | 90.40 | 74.28 | 70.35 |
Table S3. Comparison with exemplar-based methods on six datasets. and in %, and Mean averages the six settings of each block. CLAP4CLIP and PROOF are trained on zero-shot pseudo-labels and store 20 images per class of earlier tasks, while ZS-CLIP and PACIL store none.
Robustness to Hyperparameters and Backbones
Temperature . Figure S1 reports and on the 18 settings when the temperature of the final EM of every task takes the values 2, 4, 6, 8 and 10. All other hyperparameters are those of § 4.1, the debiased run of the prior selection rule keeps (§ 3.2), and each point is the mean of three runs with seeds 1993, 1994 and 1995 at that value of . Over this range, and each vary by less than 1% on 15 of the 18 settings, and varies by at most 0.18% on CIFAR100 B0Inc10, Cars B0Inc10, ImageNet-R B0Inc20 and both Food settings. Averaged over the 18 settings, gives the highest and of the five values, 77.92% and 74.81%, and the other four values are lower by at most 0.25% and 0.18%. On single settings, is within 0.30% of the best of the five values on 16 of the 18 settings in each metric. Choosing for each dataset by the mean of the other eight datasets selects for every dataset except ObjectNet, where it selects , and lowers the mean over the 18 settings by 0.06%. A single shared by all datasets therefore costs little compared with a per-dataset choice, and it is chosen without labeled data.
Figure S1. Sensitivity to the temperature of the final EM. (a) and (b) (%) on the zero-base and half-base settings of the nine datasets for , with all other hyperparameters fixed. PACIL uses , and each point is the mean over three runs.
For fixed class directions and concentration, dividing the exponent of Eq. (8) by divides by every shift that the images apply to the log-odds of the prior . With a small , small differences between cosines, scaled by the large fitted concentration, override the prior. The soft labels then become sharp and follow the vMF clusters even where these clusters do not match the classes, and they enter the statistics of the closed-form classifier without later revision.
With a large , every moves toward , and the soft labels become flatter and keep fewer of the corrections that the images make to the zero-shot prior. The mean of the nine zero-base settings, whose tasks hold 10 to 30 classes, decreases from 81.40% at to 80.95% at . The mean of the nine half-base settings, whose first task holds 50 to 150 classes, is 74.08% at and between 74.39% and 74.53% for . This is consistent with more pairs of similar classes in an EM run over more classes, where strong image evidence moves more images to a wrong class. On ObjectNet B0Inc20, falls from 54.13% at to 51.57% at , and is 1.07% below the best value. On Aircraft B50Inc10, whose first task holds 50 aircraft variants, rises from 35.22% at to 37.59% at , 0.84% above , and is 0.92% higher at than at . SUN B150Inc30 also prefers , by at most 0.30% in and 0.32% in . These settings would gain from a chosen per dataset, but such a choice needs labeled data from each dataset.
Ridge coefficient . Figure S2 reports and on the 18 settings for 13 values of the ridge coefficient of Eq. (2) from 0.1 to 100, with and all other hyperparameters as in § 4.1, and each point is the mean of three runs with seeds 1993, 1994 and 1995. The coefficient enters only the solve of and the leverages of Eq. (11). The soft labels and the weights are the same for every , and only and change. Averaged over the 18 settings, rises from 77.28% at , the value used by N2L [7], to between 77.86% and 77.93% for from 2.5 to 10, and falls to 77.34% at . follows the same course, from 74.25% to between 74.80% and 74.83% and down to 73.95%. For from 1 to 10, and each vary by less than 0.5% on 15 of the 18 settings.
Figure S2. Sensitivity to the ridge coefficient . (a) and (b) (%) on the zero-base and half-base settings of the nine datasets for 13 values of from 0.1 to 100, with all other hyperparameters fixed. PACIL uses , and each point is the mean over three runs.
is the weight of the penalty of a weighted ridge regression whose data terms have weights summing to about per task. A small barely damps the directions of with small eigenvalues, so follows the soft label of each image closely. It also raises the leverages , and the LOO predictions that select then amplify the fitted residuals by a larger factor . From to 0.1, falls by 2.66% on Aircraft B0Inc10 and 2.58% on ObjectNet B0Inc20. A large shrinks most along the directions of with small eigenvalues, and tends to , a sum of features weighted by the soft labels that ignores the correlations in , which can remove the fine distinctions between similar classes. From to 100, falls on 17 of the 18 settings and on all 18, most on Aircraft, by 2.08% and 1.62% in , and on UCF, by 1.21% and 1.17%, whose peaks at and . ObjectNet B0Inc20 is the only exception, as it rises to 53.35% at and stays within 0.05% of this value up to .
We use because it lies on the plateau of both means and loses least on the settings that depart most from it. Its mean and are 0.01% and 0.02% below the best of the 13 values, reached at and at and 7.5. Among the 13 values it has the smallest largest shortfall in from the best value of a setting, 0.38% on Aircraft B50Inc10, and it is within 0.20% of that best value on 16 settings in and on 15 in , where its largest shortfall is 0.58%, also on Aircraft B50Inc10. As the weights sum to about per task in both methods, this value also compares directly with the 0.1 of N2L (§ 3.4).
CLIP backbones. Figure S3 compares N2L and PACIL on all 18 settings with ViT-B/32, ViT-B/16 and ViT-L/14, each with LAION-400M weights (OpenCLIP laion400m_e32) and with OpenAI weights. Both methods are re-run with the same code, splits and seeds, and only the backbone changes. PACIL keeps , , and on every backbone without tuning, and N2L keeps its own hyperparameters. PACIL is above N2L in on 105 of the 108 pairs of backbone and setting, and in on 98. Averaged over the 18 settings, its gain in is 2.10%, 2.24% and 2.19% on the LAION backbones from B/32 to L/14 and 2.25%, 2.24% and 1.74% on the OpenAI backbones, and its gain in lies between 1.78% and 2.39% on all six. The gain therefore does not shrink with scale, except on OpenAI ViT-L/14, where N2L is most accurate, with a mean of 82.07%.
Figure S3. CLIP backbones. (a) and (b) (%) of N2L and PACIL with ViT-B/32, ViT-B/16 and ViT-L/14 (left to right within each group), each with LAION-400M and OpenAI weights, on the nine datasets in the zero-base (left) and half-base (right) settings. Both methods are re-run in the same setting.
A larger backbone gives better zero-shot predictions, which raises the accuracy of N2L and leaves less to correct where accuracy is high. With OpenAI weights, the gain in falls from B/32 to L/14 from 0.23% to 0.07% on Food B0Inc10 and from 2.10% to 0.32% on ImageNet-R B0Inc20. On datasets that stay hard at every scale it holds or grows, as on ObjectNet B100Inc20, from 0.44% to 2.39% with LAION weights and from 1.04% to 2.40% with OpenAI weights, which is consistent with EM gaining from better image features while zero-shot labels leave errors to correct. Aircraft B0Inc10 shows both cases, as the gain rises from 6.91% to 10.34% with LAION weights and falls from 5.41% to 4.45% with OpenAI weights. N2L is higher on three pairs in , by at most 0.33% (UCF B50Inc10 with LAION ViT-L/14, SUN B150Inc30 and CIFAR100 B50Inc10 with OpenAI ViT-B/32), and on ten pairs in , by at most 0.54%. Seven of these ten are on Food and ImageNet-R with OpenAI weights, where the gain of PACIL in is also at most 0.61%.
Noisy Task Boundaries
Unlabeled images rarely respect task boundaries. We test whether PACIL remains effective when the unlabeled data of a task also contain images of earlier classes. Following the protocol of N2L [7], from the second task on, images drawn from the training sets of earlier tasks are added to the unlabeled images of task until they make up a fraction of them, with or . The learner is not told which images they are and can only name them among the classes of task , so no correct label is available for them. In a zero-base setting 9 of the 10 tasks are noisy, whereas in a half-base setting the clean first task holds half of the classes. Table S4 compares PACIL with RAIL [13], ENGINE [14] and N2L under this protocol at both noise levels.
Table S4. Comparison under noisy task boundaries. and in %, and Mean averages the six settings of each block. From the second task on, a fraction of the unlabeled training images of each task comes from earlier tasks, and the results without noise are in Table 1.
| Method | Aircraft B0Inc10 | Aircraft B0Inc10 | Aircraft B50Inc10 | Aircraft B50Inc10 | Cars B0Inc10 | Cars B0Inc10 | Cars B50Inc10 | Cars B50Inc10 | CIFAR100 B0Inc10 | CIFAR100 B0Inc10 | CIFAR100 B50Inc10 | CIFAR100 B50Inc10 | Mean | Mean | |
| 20% | RAIL | 35.19 | 31.71 | 23.06 | 24.60 | 87.95 | 82.81 | 82.37 | 81.02 | 87.01 | 79.95 | 81.10 | 78.38 | 66.11 | 63.08 |
| ENGINE | 33.95 | 24.72 | 26.51 | 22.56 | 86.45 | 78.32 | 82.16 | 79.18 | 84.72 | 76.61 | 79.52 | 75.86 | 65.55 | 59.54 | |
| N2L | 43.11 | 38.70 | 29.13 | 30.51 | 91.06 | 85.26 | 86.00 | 84.23 | 87.68 | 80.81 | 82.78 | 80.07 | 69.96 | 66.60 | |
| PACIL | 49.50 | 46.96 | 36.35 | 39.19 | 91.29 | 88.37 | 88.24 | 87.76 | 88.39 | 82.22 | 83.73 | 81.35 | 72.92 | 70.98 | |
| 50% | RAIL | 33.46 | 28.83 | 21.97 | 21.81 | 85.56 | 78.74 | 81.52 | 78.99 | 86.55 | 79.20 | 80.67 | 77.69 | 64.96 | 60.88 |
| ENGINE | 33.36 | 23.25 | 25.63 | 21.39 | 85.44 | 76.56 | 80.83 | 75.90 | 83.72 | 75.68 | 78.64 | 74.50 | 64.60 | 57.88 | |
| N2L | 40.49 | 36.00 | 27.45 | 27.75 | 89.32 | 81.61 | 85.10 | 81.58 | 87.02 | 80.27 | 82.39 | 79.23 | 68.63 | 64.41 | |
| PACIL | 43.16 | 40.58 | 32.90 | 34.14 | 89.55 | 85.73 | 86.74 | 84.63 | 87.13 | 80.95 | 82.89 | 79.88 | 70.40 | 67.65 | |
| Method | CUB B0Inc20 | CUB B0Inc20 | CUB B100Inc20 | CUB B100Inc20 | ObjectNet B0Inc20 | ObjectNet B0Inc20 | ObjectNet B100Inc20 | ObjectNet B100Inc20 | UCF B0Inc10 | UCF B0Inc10 | UCF B50Inc10 | UCF B50Inc10 | Mean | Mean | |
| 20% | RAIL | 80.25 | 71.61 | 72.72 | 70.02 | 38.61 | 33.66 | 30.29 | 28.74 | 88.54 | 88.33 | 80.45 | 83.21 | 65.14 | 62.60 |
| ENGINE | 75.68 | 63.60 | 70.20 | 64.19 | 44.01 | 30.34 | 34.28 | 29.39 | 86.94 | 84.54 | 82.04 | 80.64 | 65.53 | 58.78 | |
| N2L | 82.10 | 74.28 | 74.46 | 71.76 | 48.10 | 40.84 | 40.96 | 37.73 | 93.15 | 91.21 | 85.85 | 86.43 | 70.77 | 67.04 | |
| PACIL | 84.26 | 78.16 | 78.51 | 76.36 | 53.28 | 41.57 | 42.88 | 39.33 | 95.53 | 93.85 | 89.06 | 89.90 | 73.92 | 69.86 | |
| 50% | RAIL | 78.38 | 68.45 | 71.20 | 66.33 | 34.48 | 29.53 | 28.09 | 24.98 | 86.31 | 85.75 | 78.44 | 79.58 | 62.82 | 59.10 |
| ENGINE | 74.39 | 61.79 | 69.22 | 62.82 | 42.72 | 29.75 | 33.53 | 28.41 | 84.28 | 82.83 | 80.75 | 78.63 | 64.15 | 57.37 | |
| N2L | 79.86 | 70.52 | 72.80 | 68.54 | 46.48 | 37.09 | 39.05 | 34.94 | 90.80 | 88.86 | 84.75 | 84.99 | 68.96 | 64.16 | |
| PACIL | 80.08 | 72.97 | 76.04 | 73.28 | 47.64 | 37.16 | 41.17 | 36.90 | 91.26 | 89.36 | 87.89 | 88.66 | 70.68 | 66.39 |
PACIL has the best and on all 12 settings at both noise levels, and every Mean column ranks the four methods in the same order at both levels. At its Mean gain over N2L in , 2.96% and 3.15% on the two blocks, is as large as without noise in Table 1, where it is 3.19% and 2.49%. At it shrinks to 1.77% and 1.72%, while the gain in remains 3.24% and 2.23%. The drop comes mostly from the zero-base settings, where the gain in falls from 2.83% to 0.81% while it stays at 2.68% in the half-base settings. Soft labels lie closer to the true labels than hard labels on the images of the current classes (Section C.6), and also on an image of an earlier class, for which no current class is correct, since a soft label is at squared distance from its true label against 2 for a hard label. This lowers the target error that each such image brings into the classifier and is consistent with the unchanged lead at . From no noise to , the Mean of PACIL falls by 3.28% and 3.60%, more than the 1.86% and 2.83% of N2L, while the falls in , 4.19% and 3.96% against 3.43% and 4.39%, are of similar size. The larger fall in comes from the part of the gain that PACIL draws from the images. EM corrects the zero-shot distributions by fitting one component per current class to the images of a task (§ 3.3) and is worth up to 4.18% of without noise (Figure 4), whereas the self-training of N2L adds little after its first round (Section C.6). Images of earlier classes fit none of these components, and at in a zero-base setting they outnumber the images of each current class several times over, so they can pull the components away from the classes that EM should recover. Such images therefore cost PACIL more than they cost N2L, yet it stays above N2L on every setting. These images fall outside the assumption that a task brings images of its own classes, and a component for them is left to future work.
N2L with Components of PACIL
N2L [7] uses the ridge coefficient and a fixed fusion weight , and its results in Figure S4 are our reproduction with its official code. Two components of PACIL carry over to N2L without its soft labels, and the LOO choice of in Eq. (12), which then scores the LOO decisions with the hard pseudo-labels of N2L and keeps its sample weights. We add them to this reproduction one at a time and together, to test whether the gain of PACIL comes only from them. Averaged over the 18 settings, changes and by 0.24% and , and the LOO choice by 0.13% and 0.36%. Neither helps everywhere, as lowers on both UCF settings, by up to 0.86%, and the LOO choice gains most on ObjectNet, by 1.43% and 1.11% in . Together they raise and by 0.61% and 0.64%, more than the sum of their separate effects. With the leverages are smaller, so the LOO outputs amplify the fitted residuals less (Sec. C.2), and the selected rises from 0.85 to 0.91 on average.
Figure S4. N2L with components of PACIL. (a) and (b) (%) of N2L, N2L with the ridge coefficient , N2L with the LOO choice of the fusion weight of Eq. (12), N2L with both, and PACIL, on the nine datasets in the zero-base (left) and half-base (right) settings. N2L is our reproduction with its official code [7], with and a fixed , and its three variants are run with the same code. Bars are means over three seeds, error bars show standard deviation, and the axes do not start at zero.
PACIL stays 1.63% and 1.75% above the strongest variant and is higher on 16 of the 18 settings in and on 17 in
. The two components thus explain about a quarter of the gain of PACIL over N2L, and most of it comes from how PACIL estimates and keeps the posterior of each label. The remaining gap is largest on Aircraft, by 6.23% and 6.73% in and by 7.72% and 7.13% in , where N2L mislabels more than 40% of the training images and the argmax of the soft labels of PACIL alone brings N2L 37% to 66% of the gain in (Sec. C.6). On ObjectNet, the two components raise of N2L the most, by 4.20% and 1.93%, and the strongest variant is level with PACIL in on B0Inc20 and above it on B100Inc20, by 0.36% in and 0.54% in . ObjectNet B0Inc20 is also the only setting where PACIL itself gains from a above 10 (Sec. C.2).
Comparison with Transductive Methods
Transductive methods also estimate a class distribution for each unlabeled image, but only to predict these images (§ 2). We replace the soft labels of each task by the
distributions that TransCLIP [38], StatA [70], MOON [5] and EM-Dirichlet [4] compute on from the same features and class embeddings with their zero-shot hyperparameters, and keep the rest of PACIL unchanged. As references, we also train on and without EM, and on TransCLIP and MOON with as their zero-shot input.
In Figure S6, PACIL is above all four methods in both metrics. Averaged over the 18 settings, it is 2.34% to 2.94% above TransCLIP, StatA and MOON in and 2.22% to 2.69% in , and it is lower only on the two SUN settings and in on Food B50Inc10, by at most 0.31%. EM-Dirichlet falls below even , as it expects a batch to hold only a few classes. The gaps come first from the class prior, as TransCLIP, StatA and MOON keep as the prior of each image. alone, without EM, is above all four methods in both metrics, and as their zero-shot input it narrows the gap of TransCLIP and MOON to PACIL from 2.34% to 1.49% and from 2.94% to 0.73% in . The remaining gap lies mainly in the likelihood, as PACIL fits vMF components to features whitened with the within-class scatter of all seen classes (§ 3.3), whereas TransCLIP uses Gaussians with a shared diagonal covariance and MOON does not whiten. Whitening with the statistics of the current task alone is enough to lower of PACIL by 0.37%, which is half of its lead over MOON with .
Figure S6. Comparison with transductive methods. Change in (a) and (b) (%) relative to PACIL when its soft labels are replaced by or without EM or by the distributions of transductive methods, averaged over the nine datasets in the zero-base and the half-base settings. All sources share the classifier and fusion of PACIL, gives a method as its zero-shot input, and the bars of EM-Dirichlet are labeled with their values.
Pseudo-Label Quality
Accuracy. Figure S5 compares the pseudo-label accuracy of N2L [7] at zero-shot and after each of its three refinement rounds with that of PACIL on Aircraft and CUB, in the zero-base and half-base settings. Accuracy is the fraction of training images whose label is the true class, pooled over all tasks of a setting, and ground-truth labels are used only for the measurements of this subsection. PACIL keeps a label distribution for each image, and to compare it with the hard labels of N2L we take its argmax. This comparison is conservative for PACIL, since its classifier is fitted to the whole , and an image whose argmax is wrong keeps part of its probability mass on the true class.
Figure S5. Pseudo-labels of PACIL and N2L [7] on Aircraft and CUB. (a) Pseudo-label accuracy (%) on the training images, pooled over all tasks, for N2L at zero-shot (ZS) and after each refinement round and for the argmax of the final soft labels of PACIL, with zero-base settings on top and half-base settings at the bottom. (b) Gain over N2L in and (%) of N2L with its pseudo-labels replaced by the argmax of the PACIL soft labels, and of PACIL.
Both methods start from the same zero-shot labels, and PACIL is above N2L after round 3 on all four settings, by 9.73% and 8.31% on Aircraft and 2.78% and 3.75% on CUB. N2L gains most in round 1, by up to 5.48% on Aircraft and 3.14% on CUB, and rounds 2 and 3 together add at most 0.59%. The margins are largest on Aircraft, where N2L is least accurate and mislabels more than 40% of the images after round 3. Self-training on hard labels corrects few errors after round 1 and can overwrite correct ones, while PACIL corrects the class prior of the whole distribution and refines it by EM over the images of each task, which raises the accuracy on Aircraft from 53.85% and 39.36% at zero-shot to 69.00% and 50.92%.
Figure S5 separates the effect of these labels from the rest of PACIL. Replacing the pseudo-labels of N2L by the argmax of the PACIL soft labels, with the rest of N2L unchanged, raises by 3.58% and 4.66% on Aircraft and by 1.47% and 1.34% on CUB. This is 37% to 66% of the gain of PACIL in , and 47% to 70% in . The rest comes from the other differences between the two methods, such as training on the whole rather than its argmax and selecting on each task by closed-form LOO.
Squared error. The bounds of Theorem S1 depend on how close the targets of the classifier are to the label posteriors, which cannot be observed. Figure S7 therefore
Figure S7. Squared error of the pseudo-labels. MSE of the zero-shot argmax (grey circles), the hard pseudo-labels of N2L [7] after round 3 (blue squares) and the soft labels of PACIL (red diamonds), averaged over the training images of all tasks of each setting in (a) the zero-base and (b) the half-base settings. The last row is the mean over the nine datasets. differs across datasets, so markers are compared within a row.
measures how far three targets are from the true labels. They are the zero-shot argmax, which is also the starting label of N2L, the hard pseudo-labels of N2L after its three refinement rounds, and the soft labels of PACIL. A hard label enters as the one-hot vector of its class, and the distance of a target for an image of a task with classes is the mean squared error , the Brier score averaged over these classes, so a hard label scores 2/K when it is wrong and 0 when it is right. It is averaged over the training images of all tasks.
The soft labels of PACIL have the lowest MSE in all 18 settings. Averaged over the 18 settings, their MSE is 0.0123, against 0.0169 for N2L and 0.0216 for the zero-shot argmax. The refinement of N2L lowers the MSE of the zero-shot argmax by 22% on average, but the MSE of is 9% to 46% below that of N2L in single settings. SUN is the exception to this ordering, as the refinement there moves the hard labels away from the true labels, from 0.0044 to 0.0050 and from 0.0033 to 0.0039, in line with the accuracy of these labels, which falls from 93.42% to 92.46% and from 86.77% to 85.70% over the three rounds.
This ordering reflects the distance to the posteriors that Theorem S1 needs. For any target computed from the observations, the identity of § 3.4 gives
whose last two terms do not depend on the target. The Brier score against the true labels therefore ranks targets by their squared distance to the posteriors, and its expectation is smallest at . In a zero-base setting every image has the same , so the MSE ranks them in the same way. A hard label also stays at squared distance at least from the posterior on every image by Eq. (S11), even when its class is correct, whereas a soft label can match the posterior. Figure S7 thus shows that comes closer to the posteriors than both kinds of hard labels, and this distance is what the bound of Theorem S1(a) depends on.
Accuracy after Each Task
Figure S8 and Figure S9 show the accuracy on all seen classes after each task, with N2L [7] re-run from its official code. In the zero-base settings, PACIL is higher on 81 of the 90 tasks and level on 2, by 2.25% on average. It is lower only on the first four tasks of Cars, by at most 1.25%, and on the first task of CIFAR100, CUB and ImageNet-R, by at most 0.27%. In the half-base settings it is higher on all 54 tasks, by 2.22% on average and by at least 0.22%.
Figure S8. Accuracy (%) after each task in the zero-base settings, against the number of seen classes. Accuracy is measured on the test images of all seen classes with the fused prediction. N2L [7] is re-run with its official code, since its paper reports no per-task accuracy. Lines are means over three seeds, and shaded bands show standard deviation.
Figure S9. Accuracy (%) after each task in the half-base settings, where the first task holds half of the classes. N2L [7] is re-run with its official code, lines are means over three seeds, and shaded bands show standard deviation.
Averaged over the datasets, the zero-base gap grows from 0.56% after the first task to 2.61% after the last, consistent with the errors of frozen pseudo-labels adding up over the tasks (Theorem S1). In the half-base settings, the gap is 2.26% after the first task, which holds half of the classes. The gain is largest on the fine-grained Aircraft, 9.49% and 7.32% at the last task, where zero-shot CLIP [1] has an of 17.16% (Table 1). It is smallest on Food and ImageNet-R, 0.58% to 0.71%, where zero-shot CLIP reaches 81.99% and 76.62% (Table S2). There the zero-shot argmax is mostly right, and a posterior concentrated on one class is close to its argmax by Eq. (S11), so keeping the whole distribution changes little.
Training and Inference Cost
Table S5 reports the measured cost of training and inference. All methods read the same pre-extracted CLIP features and run the 18 settings with three seeds each on one A100 GPU, so the comparison covers the learning step itself. N2L [7] is re-implemented here in our code base following its official implementation, which inverts the matrix of its classifier with NumPy on the CPU, and N2L computes the same inverse on the GPU.
| N2L | N2L | PACIL | |
| Training time (s) | 22.61 | 2.81 | 2.16 |
| labeling | 2.36 | 1.90 | 1.18 |
| classifier update | 20.12 | 0.78 | 0.85 |
| Inference time (s) | 7.61 | 7.33 | 7.33 |
| Training memory (GB) | 1.45 | 2.54 | 3.23 |
| (%) | 75.64 | 75.64 | 77.92 |
| (%) | 72.32 | 72.32 | 74.81 |
Table S5. Measured training and inference cost. Means over the 18 settings with three runs each, on one A100 GPU with the same pre-extracted ViT-B/16 features. Times are per setting and summed over its tasks. Memory is the peak
PACIL takes 2.16 s on average to train on one setting, of which 1.18 s is labeling. Labeling is cheap because each task runs EM only twice, once for the prior selection rule and once for the final run, on whitened dimensions, where one iteration costs . N2L takes 22.61 s, almost all of it in the inverse on the CPU, since moving this step to the GPU reduces its classifier update from 20.12 s to 0.78 s and its training time to 2.81 s. The classifier update of PACIL costs for the statistics and for the closed-form LOO, plus one Cholesky factorization, the same order as the inverse of N2L†. With the same kind of solver, the training time of PACIL is therefore on par with that of N2L.
At inference, both methods only add the random features and two matrix products, with the classifier and with the text embeddings, to the CLIP forward pass, and their inference time is the same, 7.33 s per setting and about 0.32 ms per image. PACIL keeps its statistics in double precision, so its peak training memory is 3.23 GB on average against 2.54 GB for N2L†. N2L itself uses only 1.45 GB because its inverse runs on the CPU, outside the measured GPU memory. Across tasks, PACIL carries only and , which take bytes and do not grow with the number of seen images. They are dominated by the matrix , which N2L also keeps.
References
- [1]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, pages 8748–8763, 2021. 1, 6, 13, 15, 23arxiv.org/abs/2103.00020
- [2]M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Mateusz Kozinski, Horst Possegger, Rogerio Feris, and Horst Bischof. LaFter: Label-free tuning of zero-shot classifier using language and unlabeled image collections. In Advances in Neural Information Processing Systems, pages 5765–5777, 2023. 1, 2arxiv.org/abs/2305.18287
- [3]Jiahan Zhang, Qi Wei, Feng Liu, and Lei Feng. Candidate pseudolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data. In Proceedings of the International Conference on Machine Learning, pages 60004–60020, 2024. 1, 2arxiv.org/abs/2406.10502
- [4]Ségolène Martin, Yunshi Huang, Freshteh Shakeri, Jean-Christophe Pesquet, and Ismail Ben Ayed. Transductive zero-shot and few-shot CLIP. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28816–28826, 2024. 1, 2, 3, 21arxiv.org/abs/2405.18437
- [5]Jiazhen Huang, Zhiming Liu, Changhu Wang, Wei Ju, Ziyue Qiao, and Xiao Luo. Von Mises-Fisher mixture model with dynamic shrinkage for realistic test-time transduction. In Proceedings of the International Conference on Machine Learning, pages 47178–47211, 2026. 1, 2, 21arxiv.org/abs/2607.15851
- [6]Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Class-incremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9851–9873, 2024. 1DOI
- [7]Qiwei Li, Xiaocheng Yang, and Jiahua Zhou. Naming to learn: Class incremental learning for vision-language model with unlabeled data. In International Conference on Learning Representations, pages 100800–100824, 2026. 1, 2, 3, 6, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24
- [8]Hari Chandana Kuchibhotla, K. S. Ananth, and Vineeth N. Balasubramanian. Annotation-free class-incremental learning. arXiv, abs/2511.19344, 2025. 1, 2arxiv.org/abs/2511.19344
- [9]Niels Griffioen, Nevena Ranković, Federico Zamberlan, and Monisha Punith. Efficient annotation reduction with active learning for computer vision-based retail product recognition. Journal of Computational Social Science, 7(1):1039–1070, 2024. 1DOI
- [10]Mohammad Sadegh Norouzzadeh, Anh Nguyen, Margaret Kosmala, Alexandra Swanson, Meredith S. Palmer, Craig Packer, and Jeff Clune. Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning. Proceedings of the National Academy of Sciences, 115(25):E5716–E5725, 2018. 2DOI
- [11]Huiping Zhuang, Zhenyu Wang, Hongxin Wei, Rencunzi Xie, Kar-Ann Toh, and Zhiping Lin. ACIL: Analytic class-incremental learning with absolute memorization and privacy protection. In Advances in Neural Information Processing Systems, pages 11602–11614, 2022. 2, 4arxiv.org/abs/2205.14922
- [12]Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19125–19136, 2023. 2arxiv.org/abs/2303.06628
- [13]Yicheng Xu, Yuxin Chen, Jiahao Nie, Yusong Wang, Huiping Zhuang, and Manabu Okumura. Advancing cross-domain discriminability in continual learning of vision-language models. In Advances in Neural Information Processing Systems, pages 51552–51576, 2024. 2, 4, 6, 19arxiv.org/abs/2406.18868
- [14]Da-Wei Zhou, Kai-Wen Li, Jingyi Ning, Han-Jia Ye, Lijun Zhang, and De-Chuan Zhan. External knowledge injection for CLIP-based class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3314–3325, 2025. 2, 6, 15, 19arxiv.org/abs/2503.08510
- [15]Eric Arazo, Diego Ortego, Paul Albert, Noel E. O’Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In Proceedings of the International Joint Conference on Neural Networks, pages 1–8, 2020. 2arxiv.org/abs/1908.02983
- [16]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 2arxiv.org/abs/1503.02531
- [17]Kamal Nigam, Andrew Kachites McCallum, Sebastian Thrun, and Tom Mitchell. Text classification from labeled and unlabeled documents using EM. Machine Learning, 39(2–3):103–134, 2000. 2DOI
- [18]Mark D. McDonnell, Dong Gong, Amin Parvenah, Ehsan Abbasnejad, and Anton van den Hengel. RanPAC: Random projections and pre-trained models for continual learning. In Advances in Neural Information Processing Systems, pages 12022–12053, 2023. 2, 4
- [19]Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanah, James Caverlee, and Shu Kong. The neglected tails in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12988–12997, 2024. 2arxiv.org/abs/2401.12425
- [20]Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. In Advances in Neural Information Processing Systems, pages 64663–64680, 2023. 3arxiv.org/abs/2310.08106
- [21]Wei Tang, Zuo-Zheng Wang, Kun Zhang, Tong Wei, and Min-Ling Zhang. Unleashing the power of vision-language models for long-tailed multi-label visual recognition. arXiv preprint arXiv:2511.20641, 2025. 2arxiv.org/abs/2511.20641
- [22]Jiazou Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23219–23230, 2024. 2, 6arxiv.org/abs/2403.11549
- [23]Linlan Huang, Xusheng Cao, Haori Lu, and Xiailei Liu. Class-incremental learning with CLIP: Adaptive representation adjustment and parameter fusion. In Proceedings of the European Conference on Computer Vision, pages 214–231, 2024. 6arxiv.org/abs/2407.14143
- [24]Da-Wei Zhou, Yuanhan Zhang, Yan Wang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(6): 4489–4504, 2025. 6, 15DOI
- [25]Mao-Lin Luo, Zi-Hao Zhou, Tong Wei, and Min-Ling Zhang. LADA: Scalable label-specific CLIP adapter for continual learning. In Proceedings of the International Conference on Machine Learning, pages 41604–41619, 2025. 2arxiv.org/abs/2505.23271
- [26]Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, and Zhiping Lin. DS-AL: A dual-stream analytic learning for exemplar-free class-incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 17237–17244, 2024. 2arxiv.org/abs/2403.17503
- [27]Huiping Zhuang, Yizhu Chen, Di Fang, Run He, Kai Tong, Hongxin Wei, Ziqian Zeng, and Cen Chen. GACL: Exemplar-free generalized analytic continual learning. In Advances in Neural Information Processing Systems, pages 83024–83047, 2024. 2arxiv.org/abs/2403.15706
- [28]Binyu Zhao, Wei Zhang, Xingrui Yu, Zhaonian Zou, and Ivor Tsang. Advancing analytic class-incremental learning through vision-language calibration. In Proceedings of the International Conference on Machine Learning, pages 162951–162970, 2026. 2arxiv.org/abs/2602.13670
- [29]Zhen Zhu, Yiming Gong, and Derek Hoiem. Anytime continual learning for open vocabulary classification. In Proceedings of the European Conference on Computer Vision, pages 269–285, 2024. 2arxiv.org/abs/2409.08518
- [30]Subhankar Roy, Mingxuan Liu, Zhun Zhong, Nicu Sebe, and Elisa Ricci. Class-incremental novel class discovery. In Proceedings of the European Conference on Computer Vision, pages 317–333, 2022. 2
- [31]Shijie Ma, Fei Zhu, Zhun Zhong, Wenzhuo Liu, Xu-Yao Zhang, and Cheng-Lin Liu. Happy: A debiased learning framework for continual generalized category discovery. In Advances in Neural Information Processing Systems, pages 50850–50875, 2024. 2arxiv.org/abs/2410.06535
- [32]Donghao Sun, Xi Wang, Xu Yang, Kun Wei, and Cheng Deng. Dual-space semantic synergy distillation for continual learning of unlabeled streams. In Advances in Neural Information Processing Systems, pages 21303–21325, 2025. 2DOI
- [33]Ameya Prabhu, Hasan Abed Al Kader Hammoud, Ser-Nam Lim, Bernard Ghanem, Philip Torr, and Adel Bibi. From categories to classifiers: Name-only continual learning by exploring the web. In Proceedings of the Conference on Lifelong Learning Agents, pages 534–559, 2024. 2arxiv.org/abs/2311.11293
- [34]Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649, 2022. 2arxiv.org/abs/2204.03649
- [35]Junnan Li, Silvio Savarese, and Steven C. H. Hoi. Masked unsupervised self-training for label-free image classification. In International Conference on Learning Representations, 2023. 2arxiv.org/abs/2206.02967
- [36]Qiyuan Dai and Sibei Yang. Free on the fly: Enhancing flexibility in test-time adaptation with online EM. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9538–9548, 2025. 2, 3arxiv.org/abs/2507.06973
- [37]Lihua Zhou, Mao Ye, Shuaifeng Li, Nianxin Li, Xiatian Zhu, Lei Deng, Hongbin Liu, and Zhen LeI. Bayesian test-time adaptation for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 29999–30009, 2025. 2
- [38]Maxime Zanella, Benoît Gérin, and Ismail Ben Ayed. Boosting vision-language models with transduction. In Advances in Neural Information Processing Systems, pages 62223–62256, 2024. 2, 3, 5, 21arxiv.org/abs/2406.01837
- [39]Xingyu Xu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. In Advances in Neural Information Processing Systems, pages 2001–2025, 2024. 3
- [40]Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. Neural Computation, 14(1):21–41, 2002. 3
- [41]Zachary C. Lipton, Yu-Xiang Wang, and Alexander J. Smola. Detecting and correcting for label shift with black box predictors. In Proceedings of the International Conference on Machine Learning, pages 3122–3130, 2018. 3arxiv.org/abs/1802.03916
- [42]Xudong Wang, Zhirong Wu, Long Lian, and Stella X. Yu. Debiased learning from naturally imbalanced pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14647–14657, 2022. 3arxiv.org/abs/2201.01490
- [43]Adian Liusie, Potswaee Manakul, and Mark J. F. Gales. Mitigating world bias in zero-shot prompt-based classifiers. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023, pages 327–335, 2023.
- [44]Qi Qian, Yuanhong Xu, and Juhua Hu. Intra-modal proxy learning for zero-shot visual categorization with CLIP. In Advances in Neural Information Processing Systems, pages 25461–25474, 2023. 3arxiv.org/abs/2310.19752
- [45]Changho Shin, Jitian Zhao, Sonia Cromp, Harit Vishwakarma, and Frederic Sala. OTTER: Effortless label distribution adaptation of zero-shot models. In Advances in Neural Information Processing Systems, pages 44064–44101, 2024. 3arxiv.org/abs/2404.08461
- [46]Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. SoLar: Sinkhorn label refinery for imbalanced partial-label learning. In Advances in Neural Information Processing Systems, pages 8104–8117, 2022. 3arxiv.org/abs/2209.10365
- [47]Kuang He, Wei Tang, Tong Wei, and Min-Ling Zhang. Tuning the right foundation models is what you need for partial label learning. arXiv preprint arXiv:2506.05027, 2025. 3arxiv.org/abs/2506.05027
- [48]Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, pages 2292–2300, 2013. 4
- [49]Imre Csiszár. I-divergence geometry of probability distributions and minimization problems. The Annals of Probability, 3(1):146–158, 1975. 4DOI
- [50]Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien, editors. Semi-Supervised Learning. MIT Press, Cambridge, MA, 2006. 4
- [51]Dipam Goswami, Yuyang Liu, Bartlomiej Twardowski, and Joost van de Weijer. FeCAM: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. In Advances in Neural Information Processing Systems, pages 6582–6595, 2023. 5arxiv.org/abs/2309.14062
- [52]Ameya Prabhu, Shiven Sinha, Ponnurangam Kumarguru, Philip Torr, Ozan Sener, and Puneet K. Dokania. Ran-Dumb: Random representations outperform online continually learned representations. In Advances in Neural Information Processing Systems, pages 37988–38006, 2024. 5DOI
- [53]Yilun Chen, Ami Wiesel, Yonina C. Eldar, and Alfred O. Hero. Shrinkage algorithms for MMSE covariance estimation. IEEE Transactions on Signal Processing, 58(10):5016–5029, 2010. 5DOI
- [54]Arindam Banerjee, Inderjit S. Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering on the unit hypersphere using von Mises-Fisher distributions. Journal of Machine Learning Research, 6(46):1345–1382, 2005. 5DOI
- [55]David M. Allen. The relationship between variable selection and data augmentation and a method for prediction. Technometrics, 16(1):125–127, 1974. 5DOI
- [56]Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, NY, 2nd edition, 2009. 5
- [57]Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013. 6arxiv.org/abs/1306.5151
- [58]Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 554–561, 2013. 6DOI
- [59]Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 6
- [60]Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. In Proceedings of the European Conference on Computer Vision, pages 446–461, 2014. 6DOI
- [61]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 6
- [62]Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8349, 2021. 6
- [63]Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfruend, Joshua Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems, pages 9448–9458, 2019. 6
- [64]Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human action classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 6arxiv.org/abs/1212.0402
- [65]Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3485–3492, 2010. 6DOI
- [66]James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. CODA-Prompt: COntinual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909–11919, 2023. 6arxiv.org/abs/2211.13218
- [67]Saurav Jha, Dong Gong, and Lina Yao. CLAP4CLIP: Continual learning with probabilistic finetuning for vision-language models. In Advances in Neural Information Processing Systems, pages 129146–129186, 2024. 6, 15arxiv.org/abs/2403.19137
- [68]Christoph Schuhmann, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 6arxiv.org/abs/2111.02114
- [69]Kai Gan, Bo Ye, Min-Ling Zhang, and Tong Wei. Semi-supervised CLIP adaptation by enforcing semantic and trapezoidal consistency. In The Thirteenth International Conference on Learning Representations, 2025. 8
- [70]Maxime Zanella, Clément Fuchs, Christophe De Vleeschouwer, and Ismail Ben Ayed. Realistic test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 25103–25112, 2025. 21arxiv.org/abs/2501.03729













