WorksComputer Science
GRAFT: Graph-Regularized Prototype Fields for Realistic Test-Time Adaptation of VLMs
Abstract
Unlabeled test data can improve the zero-shot classifier of a vision-language model, yet transductive and test-time adaptation methods often fall below the zero-shot baseline when only a few vocabulary classes are present in a batch or when streams are correlated. We trace this failure to a shared design: every class owns independent parameters estimated from its own samples, so absent or rare classes are left at biased text anchors or fitted to foreign samples. We propose Graph-Regularized Anchor-displacement Field Transduction (GRAFT), which models the adapted prototypes as text anchors plus a displacement field that is partially pooled over a semantic class graph through a Gaussian Markov random field prior, together with an explicit class-presence model. Inference is closed-form and gradient-free on cached features. Anchor-only shrinkage and a single global shift are the two extreme settings of the prior; absent classes are provably harmonic extensions of the present ones, and the pooling strength can be chosen by empirical Bayes. Under the realistic batch and online protocols on eleven benchmarks and four ImageNet shifts, GRAFT improves over the zero-shot model in every setting and over anchor-based transduction where few classes are present.
Introduction
Vision-language models (VLMs) such as CLIP [31] and SigLIP [48] classify an image by comparing its embedding with the text embeddings of class names. The resulting zero-shot classifier is general but frozen, and a growing family of methods adapts it from the unlabeled test data itself, transductively on a batch [23, 27, 40, 46] or online on a stream [10, 21, 49, 51], without training and on cached features.
These methods are evaluated on batches that contain every class of the vocabulary in comparable numbers, whereas
a photo album or a user session contains a handful of the classes that the vocabulary names, and streams are temporally correlated. Under such conditions transductive methods lose tens of accuracy points with respect to the zero-shot model they are meant to improve [47], and a controlled benchmark reports marginal gains, poor transfer across backbones and degraded calibration [36]. The remedy proposed in [47] and refined in [18] is a statistical anchor: each class prototype is shrunk towards its text embedding with a strength that grows with the number of test samples the class receives. Anchoring prevents collapse but treats every class in isolation: a class that receives no samples stays at its text anchor, although that anchor is displaced from the image distribution by the modality gap and by the domain of the test data [24, 35], and a class with one or two samples is adapted from those samples alone. The problem addressed here is therefore to adapt all prototypes of a large vocabulary from an unlabeled batch in which most classes are absent or rare, without collapsing on the absent ones and without discarding what the present ones reveal about them.
The displacement between the text anchor of a class and the centroid of its images contains a component shared by all classes, the modality and domain offset, and a component that varies smoothly with the semantics of the class. GRAFT turns this structure into a prior. The adapted prototypes are written as and the displacement field receives a Gaussian Markov random field prior whose precision is , where is the Laplacian of a nearest-neighbour graph over the text embeddings. The term reproduces the anchor-only shrinkage of [18, 47]. The term couples the classes, so that the displacement of a well-observed class propagates to its semantic neighbours. The maximum a posteriori field has a closed form: a graph filter that returns the empirical displacement for well-observed classes and the harmonic extension of the neighbours’ displacements for absent ones (Figure 1c). Classes that receive no samples are additionally removed from the competition by a Dirichlet-regularised estimate of the class proportions, which treats the vocabulary as a label-shift problem [12, 34]. All updates are closed form on cached features, and the prior strengths can be selected by maximising the marginal likelihood of the unlabeled batch, which removes the dataset-specific tuning criticised in [36].

Figure 1. Per-class adaptation on a realistic batch (conceptual illustration). Classes A and B are present, C is absent, and all images are shifted from their anchors. (a) Per-class fitting lets C capture samples of B. (b) Anchor-only shrinkage leaves C at its biased anchor. (c) GRAFT lets C inherit B’s displacement and removes it through the presence model.
Contributions.
We formulate transductive VLM adaptation as the estimation of a displacement field over the class vocabulary with a graph-structured prior, and derive closed-form block-coordinate inference for batches and streams (§ 4); anchor-only shrinkage and a single global shift are its two extreme settings, and absent classes receive the harmonic extension of the present-class displacements.
We combine the field with an explicit class-presence model and an empirical Bayes rule for the prior strength, so that the vocabulary is treated as a label-shift problem and no pooling strength has to be tuned per dataset.
We evaluate GRAFT under the realistic batch and online protocols on eleven benchmarks and four ImageNet shifts, with architecture-matched controls that isolate the graph coupling and the presence model.
Related Work
Unlabeled adaptation of vision-language models. Transductive methods classify an unlabeled batch jointly: TransCLIP [46] fits a Gaussian mixture regularised by a KL term to the zero-shot posterior and a sample-graph Laplacian, EM-Dirichlet [27] works on the simplex, ZLaP [40] and ECALP [23] propagate labels over image and prompt graphs, and LATA [2] and Conf-OT [38] target conformal prediction. Online methods adapt over a stream with feature caches [21, 50, 51], evolving prototypes [49], per-class Gaussians [10] or online label distributions [13, 30]; episodic methods act on single images [8, 37, 41, 45]. The realistic protocols with a variable number of effective classes and correlated streams were introduced in [47], which showed that these methods collapse under them and proposed StatA, a shrinkage of each class’s Gaussian parameters towards its text embedding with strength . MOON [18] derives this shrinkage from penalised likelihood on the hypersphere and makes it confidence dependent. Both anchor every class to its own text embedding and use no information from other classes; their mean update is the case of GRAFT. A unified benchmark further reports marginal gains, poor transfer to SigLIP and worse calibration [36].
Structure over classes and label shift. Class graphs propagate classifier weights to unseen classes in zero-shot learning [20], propagate prototypes along a category hierarchy [25] and select correlated classes in CLIP test-time adaptation [17]. Laplacian regularisation over samples is standard in transductive few-shot learning [52, 55]. We are not aware of prior work that regularises the displacement of VLM prototypes from their text anchors with a Gaussian Markov random field over the class graph. The modality gap literature [24, 35] supports the premise that much of the displacement is shared. Adjusting a classifier to new class priors from unlabeled data is classical [12, 34]; Frolic [54] corrects the label bias of CLIP without labels, and open-vocabulary classifiers degrade as distractor classes are added [3].
Preliminaries
Zero-shot classification. A VLM provides an image encoder and a text encoder mapping into a common space . For a vocabulary of class names , prompt ensembling yields unit-norm text anchors , collected in . A test image gives vectors , collected in . The zero-shot posterior is
with logit scale (100 for CLIP). We write for the probability simplex and for the all-ones vector.
Unlabeled adaptation. Given and only, the goal is to output posteriors more accurate and at least as well calibrated as , for a batch processed at once or for a stream of batches predicted as they arrive. Following [47], a batch contains effective classes, often far fewer than the vocabulary, and streams are correlated.
Mixture-model transduction and its anchors. TransCLIP [46] and StatA [47] model with a -component Gaussian mixture and minimise, over assignments and parameters, a penalised negative log-likelihood with a KL anchor to and a Laplacian term over a nearest-neighbour graph of the samples. StatA additionally shrinks the mean of class towards its anchor,
with the soft count and the responsibility-weighted mean. Equation (2) is the posterior mean of under an independent Gaussian prior centred at : a class with keeps even when every present class has moved by a common offset, a class with small is estimated from its own few samples only, and the mixture weights are implicitly uniform, so absent classes keep competing for samples. GRAFT addresses the first two points by coupling the classes through a graph prior and the third by estimating the class proportions.
Method
GRAFT keeps the generative view of transduction [46, 47] but changes the object that is adapted. Instead of independent prototypes it estimates a displacement field anchored at the text embeddings, , under a prior that couples semantically related classes, together with the class proportions and a shared covariance. Figure 2 shows the components: the frozen encoders with the zero-shot prior of Eq. (1) and the two graphs (§ 4.1), the likelihood and the graph-regularised field (§ 4.2), the class-presence model (§ 4.3), the objective (§ 4.4), the closed-form inference (§ 4.5) and the empirical Bayes rule for the prior strength (§ 4.6).

Figure 2. Overview of GRAFT. Frozen encoders give features and anchors , which define the zero-shot prior , the sample graph (from , images to images) and the semantic class graph (from , classes to classes). Closed-form updates alternate the E-step over , using and , with M-steps for the presence , the displacement field pooled over and the shared covariance; are the adapted prototypes and the prediction.
Sample graph and semantic class graph
Two graphs enter the model, and they are built from different sources. The sample graph , inherited from [46, 47], is data driven: it connects each test image to its nearest neighbours in feature space with weight and encodes the manifold assumption that neighbouring images share a label. It is recomputed for every batch and says nothing about classes that have no image in the batch.
The semantic class graph, introduced in this work, is built from the vocabulary only. Its nodes are the classes and its edges are derived from the text anchors produced by the frozen text encoder, so that it depends on the class names and the prompts, not on the images, the batch or the responsibilities. Let be the cosine similarity between anchors. Class is connected to the classes with the largest , , with the heat-kernel weight
with bandwidth . The neighbourhoods are symmetrised by union, , and the normalised Laplacian is with . is positive semi-definite with spectrum in and is computed once per vocabulary. Because it is built from text semantics, every class has neighbours, including those that receive no sample in a batch, so the field below is defined on the whole vocabulary. The graph expresses the hypothesis, that classes with similar names are displaced from their anchors in similar directions.
Graph-regularised displacement field
Likelihood. Each class is a Gaussian with mean and a diagonal covariance shared across classes. As in TransCLIP, the likelihood is tempered by , which acts as a covariance inflation and keeps the responsibilities soft on -dimensional features:
Given responsibilities , the shared variance has the closed-form M-step .
How the graph defines the field. The displacement field assigns to every class of the vocabulary the vector by which its prototype departs from its text anchor. The class graph enters through a Gaussian Markov random field prior [33] on this field, which factorises over feature dimensions,
with . The two terms of the precision have distinct roles. The term pulls every displacement towards zero, i.e., every prototype towards its own text anchor; this is the anchoring of StatA and MOON. The term is the coupling introduced here: it penalises the Dirichlet energy , so that displacements of classes joined by an edge are asked to agree, with a strength given by the semantic weight . The conditional of the GMRF makes the role of the graph explicit: under Eq. (5), the prior mean of given the displacements of all other classes is
a weighted average of the neighbours’ displacements on the class graph, shrunk towards zero by the anchor. A class therefore inherits, as a priori, the displacement of the classes whose names are closest to its own. Dividing by expresses and in units of sample counts.
How the field is regularised and updated. Three forces act on every displacement. The data term of the likelihood pulls towards the empirical displacement of the images currently assigned to class , with weight equal to the soft count (here , and when ); the anchor term pulls it towards zero with weight ; and the coupling term pulls it towards the neighbour average of Eq. (6) with weight . Writing , the maximum a posteriori field given the responsibilities balances the three and is obtained for every feature dimension from a linear system,
Equation [7] is a graph filter applied to the empirical displacements (Figure 3). Writing for the normalised adjacency and for the diagonal part, it expands as the convergent series

Figure 3. Graph-pooled displacement field on a toy class graph (node size: ). (a) Empirical displacements. (b) Independent shrinkage () leaves absent classes at their anchors. (c) Eq. (7) transfers displacements along the graph.
in which the evidence-weighted empirical displacements are diffused along the class graph: the term carries the displacement of present classes to classes edges away, attenuated at every hop by . Hence a class with keeps its empirical displacement, a class with few samples mixes its own noisy estimate with its neighbours’ displacements in proportion to against , and a class with no sample receives only what diffuses from
its neighbours. The field is re-estimated at every outer iteration: the responsibilities determine and , Eq. (7) gives , the prototypes enter the likelihood of Eq. (4), and the next E-step updates (§ 4.5); the systems are solved jointly by preconditioned conjugate gradients.
Special cases. With , Eq. (7) reduces per dimension to , the count-based anchor shrinkage of StatA (Eq. (2)) and MOON [18], which are therefore the graph-free member of the family. With , the Laplacian of the complete graph and , every class receives the count-weighted mean displacement , a single global shift of all prototypes. GRAFT interpolates between these extremes along the semantic neighbourhoods.
Absent classes. Let be the absent classes, their complement and . Since the rows of Eq. (7) in have and , the solution satisfies , so
is the harmonic extension of the present-class field, and solves Eq. (7) on with replaced by its Kron reduction [7]: absent classes affect the present-class prototypes only through the effective graph, never through data (Figure 1c).
Posterior variance. Given , the posterior covariance of is ; since , , the variance under independent shrinkage. Proofs are in the supplementary material.
Smoothness hypothesis. None of these statements guarantees higher accuracy than zero-shot inference. The benefit of the coupling rests on a testable hypothesis, class-level smoothness (H1): the oracle displacement field has a Dirichlet energy on substantially smaller than on a degree-matched random graph. If H1 fails, GRAFT reduces to anchor-only shrinkage.
Class presence as label shift
The mixture weights are free parameters with a symmetric Dirichlet prior . Their M-step is the smoothed count
and they enter the E-step as , which is the classical label-shift correction of a calibrated classifier [34]: the zero-shot posterior implicitly assumes uniform class priors, and Eq. (10) replaces them by the estimate obtained from the batch. For a batch with classes, absent classes receive while present ones receive : with , a present class with sixteen samples is more than thirty times more likely a priori than an absent one in a batch of 64 over 1000 classes. The pseudo-count keeps finite and lets a class absent from earlier online batches re-enter the stream.
Objective
Collecting the terms, GRAFT minimises over
where the first line is the variational free energy of the mixture with the text prior of [46], and the second line contains the inherited sample-graph term, the field prior of Eq. (5) and the presence prior. With and without the terms this is the StatA objective up to its per-class covariances; with in addition, the unanchored mixture.
Inference
The objective is minimised by block-coordinate descent over four blocks (pseudocode in the supplementary material). The E-step is the mean-field update with the Laplacian term linearised at the current , a majorise-minimise step [19, 46, 55]:
iterated a few times. The M-steps are the presence update of Eq. (10), the field of Eq. (7) and the shared variance of § 4.2. Each block update is the exact minimiser of Eq. (11) over its block when , so decreases monotonically and converges; with the E-step minimises a majoriser of , as in [46]. All quantities depend on the data only through the sufficient statistics , and , which makes the online variant immediate: the statistics are accumulated over the stream with an optional forgetting factor , and the same M-steps are applied after every batch. The prediction for a batch is its posterior after adaptation, as in the transductive protocol of [47]; a strictly inductive variant predicts before the statistics are updated. Per outer iteration the E-step costs and the field with conjugate gradients. No backward pass is executed and the VLM is untouched after feature extraction.
Empirical Bayes selection of the pooling strength
Fixed are used for all datasets in the main protocol, but the model also admits a label-free rule: under the model, the empirical displacements of the present classes are Gaussian with a covariance that depends on through the Schur complement of onto the present classes, and their marginal likelihood is maximised over a small logarithmic grid at every M-step whenever at least three classes are present. The derivation is given in the supplementary material.
Experiments
Datasets
We follow the eleven-dataset suite of TransCLIP and StatA [46, 47] with the CoOp test splits [53]: ImageNet [6], SUN397 [44], FGVC-Aircraft [26], EuroSAT [14], Stanford Cars [22], Food101 [1], Oxford Pets [29], Flowers102 [28], Caltech101 [9], DTD [5] and UCF101 [39], plus the ImageNet distribution shifts ImageNet-V2 [32], ImageNet-Sketch [43], ImageNet-A [16] and ImageNet-R [15]. Class names and prompt templates are those of CoOp and StatA. Features are extracted once per backbone and cached. No training split is used. Batch adaptation follows the seven settings of [47]: a task draws classes uniformly from , or (very low, low, medium) with images, from , or (medium, high, very high) with , or uses the full test set, with 1000 tasks per dataset. Online adaptation streams batches of 128 whose class composition follows Dirichlet() slot proportions, (low to high correlation) or one class after another (separate), with 100 runs per dataset. The vocabulary sizes range from 10 (EuroSAT) to 1000 (ImageNet) and the test sets from 1,692 (DTD) to 50,000 images (ImageNet).
Baselines
We compare with zero-shot CLIP, the episodic MTA [45], the transductive EM-Dirichlet [27], ZLaP [40], TransCLIP [46], StatA [47] and MOON [18], and the online TENT [42], TDA [21], DMN [51] and OGA [10], all run with default hyper-parameters through the StatA harness into which GRAFT is plugged as a drop-in solver, so that samplers, features and prompts are identical. To attribute gains to the proposed mechanism rather than to the likelihood or the inherited terms, four architecture-matched controls share our implementation: anchor-only shrinkage (, the StatA/MOON mechanism in our likelihood), complete pooling (a single global shift: , complete graph, ), presence only ( without displacement) and the unanchored mixture (, no ).
Implementation details
Inherited constants follow TransCLIP and StatA and are never tuned: , , , , outer iterations with 3 inner E-step sweeps. The new constants are fixed once for all datasets and backbones: (one pseudo-count, as in StatA), , neighbours, , . The linear systems are solved exactly for and by 30 preconditioned conjugate-gradient iterations otherwise. Backbones: CLIP ViT-B/16 (main), ViT-L/14 and RN50 [31], OpenCLIP [4] ViT-B/16 trained on DataComp [11], and SigLIP ViT-B/16 [48], the backbone on which [36] found most TTA methods to fail. All runs use a single GPU. Seeds and task indices are fixed and shared across methods. We report top-1 accuracy averaged over tasks and datasets, the worst-decile accuracy, expected calibration error and time per task in supplementary material.
Results
Realistic batches
Table 1 and Figure 4 compare GRAFT with the state-of-the-art methods and the architecture-matched controls across the seven batch settings. First, the unanchored transductive methods collapse as the number of effective classes shrinks: TransCLIP falls from 70.3 with all classes to 38.9 in the very-low setting and ZLaP from 66.4 to 27.5, whereas EM-Dirichlet, the strongest method at , collapses in the opposite direction, to 29.5 on the full set. The unanchored control of our implementation reproduces the first pattern (42.4 to 45.4 at ). Second, anchoring removes the collapse but adds little beyond it: StatA, MOON and the anchor-only control lie within 1.2 points of each other in every setting, so the count-dependent shrinkage they share is what prevents collapse. Third, pooling the field across classes adds a margin on top of anchoring that is largest where absent and rare classes dominate and vanishes where every class is well observed: GRAFT improves over the anchor-only control by 1.9, 1.6 and 1.1 points in the very-low, low and medium settings with , by 0.9, 0.7 and 0.5 points with , and by 0.3 points on the full set, where Sec. 4.2 predicts that the two mechanisms coincide; against MOON the margins are 0.7 to 0.1 points. The presence model alone reaches 67.4 in the very-low setting, about a third of the gain over zero-shot, and the complete-graph control, a single shift of all prototypes, stays below the anchor-only control in every setting, so the gain comes from pooling along the semantic neighbourhoods rather than from a global shift. Empirical Bayes selection of matches the fixed strengths within 0.1 point, and standard deviations over tasks are 0.2 to 0.4 points for the anchored methods, well below these margins.
Figure 4. Top-1 accuracy versus the number of effective classes per batch (eleven-dataset mean, error bars: standard deviation over tasks); the dotted line is the zero-shot model.
Online streams
Table 2 reports the online protocol. The cache-based TDA and DMN are strongest on weakly correlated streams (67.7 and at ) but gain little on strongly correlated ones ( and at ), whereas the anchored OGA, StatA and MOON improve with correlation (, and at ), since correlated batches contain few classes with many samples each. GRAFT follows the trend of the anchored methods and adds , and points over StatA at medium and high correlation and on the separate stream, and at low correlation ( to over MOON). The inductive variant, which predicts each batch before updating the statistics, retains most of the margin (, and against , and ).
Table 1. Batch adaptation with a variable number of effective classes: top-1 accuracy (%) averaged over 1000 tasks and eleven datasets, CLIP ViT-B/16.
Table 2. Online adaptation: top-1 accuracy (%) over 100 runs and eleven datasets, batches of 128, CLIP ViT-B/16, for stream correlation and the separate-class stream.

Figure 5. Per-dataset accuracy in the low setting for the zero-shot model, StatA and GRAFT.
Per-dataset results and distribution shifts
Table Table 3 reports every baseline on each dataset in the low setting and Fig. 5 plots the zero-shot model, StatA and GRAFT. GRAFT is first on eight of the eleven datasets and second on SUN397, EuroSAT and Flowers102, where EM-Dirichlet, strong at but collapsing with large batches, leads by , and points. Against StatA, GRAFT improves on every dataset. The margin is largest on the fine-grained vocabularies with many confusable classes, Stanford Cars (), FGVC-Aircraft () and SUN397 (), and smallest on the coarse ones, Caltech101 (), EuroSAT () and UFC101 (). Table 4 extends the comparison to the ImageNet distribution shifts with the 1000-class vocabulary. The ranking is unchanged under the four shifts; StatA improves over zero-shot by to points, and the gain of GRAFT over StatA grows from points on ImageNet to to points on the shifted sets, where the displacement from the text anchors, and with it the shared component that the graph propagates, is largest.
| Method | IN | SUN | Airc. | Euro. | Cars | Food | Pets | Flow. | Calt. | DTD | UCF | Avg. |
| CLIP zero-shot [31] | 66.7 | 62.6 | 24.8 | 47.8 | 65.3 | 86.1 | 89.1 | 71.3 | 93.3 | 44.4 | 66.7 | 65.3 |
| MTA [45] | 68.0 | 64.2 | 27.0 | 49.5 | 66.4 | 86.8 | 89.6 | 72.6 | 93.7 | 47.0 | 67.9 | 66.6 |
| EM-Dirichlet [27] | 70.2 | 69.3 | 35.1 | 57.3 | 70.3 | 88.0 | 91.0 | 76.1 | 94.1 | 50.6 | 71.4 | 70.3 |
| ZLaP [40] | 35.4 | 30.7 | 13.3 | 28.4 | 40.3 | 48.4 | 47.7 | 38.2 | 44.9 | 26.6 | 33.3 | 35.2 |
| TransCLIP [46] | 43.2 | 35.8 | 16.8 | 32.6 | 44.4 | 50.8 | 48.8 | 45.9 | 62.5 | 24.2 | 39.5 | 40.4 |
| StatA [47] | 71.0 | 66.0 | 35.0 | 52.3 | 69.6 | 87.4 | 90.7 | 74.2 | 94.0 | 50.9 | 71.1 | 69.3 |
| MOON [18] | 71.7 | 66.9 | 36.3 | 52.6 | 71.0 | 87.9 | 91.2 | 75.0 | 94.3 | 51.6 | 71.5 | 70.0 |
| GRAFT | 72.3 | 67.9 | 37.6 | 53.0 | 72.4 | 88.5 | 91.6 | 75.7 | 94.6 | 52.3 | 71.9 | 70.7 |
Table 3. Per-dataset top-1 accuracy (%) in the low setting (, ), CLIP ViT-B/16.
| Method | IN | V2 | S | A | R |
| CLIP zero-shot | 66.7 | 60.8 | 46.2 | 47.8 | 74.0 |
| TransCLIP | 43.2 | 35.5 | 29.1 | 28.9 | 42.8 |
| StatA | 71.0 | 66.2 | 51.1 | 51.9 | 79.4 |
| MOON | 71.7 | 66.8 | 51.7 | 52.6 | 79.7 |
| GRAFT | 72.3 | 68.3 | 53.1 | 54.3 | 81.9 |
Table 4. ImageNet (IN) and its shifts V2, Sketch (S), A and R in the low setting, top-1 accuracy (%), CLIP ViT-B/16.

Figure 6. Posterior mass assigned to classes absent from the batch versus the number of effective classes (ImageNet, ).
Qualitative results
Figure 7 shows test images from batches of the low set- @page 7/10 starts_mid=1 ends_mid=1
Figure 7. Qualitative predictions in the low setting: ground truth (GT) and the predictions of the zero-shot model (ZS), StatA and GRAFT. Correct predictions in green, wrong ones in red.
Table 1 (PDF p. 7). Batch adaptation with a variable number of effective classes: top-1 accuracy (%) averaged over 1000 tasks and eleven datasets, CLIP ViT-B/16.
Table 2 (PDF p. 7). Online adaptation: top-1 accuracy (%) over 100 runs and eleven datasets, batches of 128, CLIP ViT-B/16, for stream correlation and the separate-class stream.

Figure 5 (PDF p. 7). Per-dataset accuracy in the low setting for the zero-shot model, StatA and GRAFT.
and at ) but gain little on strongly correlated ones ( and at ), whereas the anchored OGA, StatA and MOON improve with correlation (, and at ), since correlated batches contain few classes with many samples each. GRAFT follows the trend of the anchored methods and adds , and points over StatA at medium and high correlation and on the separate stream, and at low correlation ( to over MOON). The inductive variant, which predicts each batch before updating the statistics, retains most of the margin (, and against , and ).
Per-dataset results and distribution shifts
Table Table 3 reports every baseline on each dataset in the low setting and Fig. 5 plots the zero-shot model, StatA and GRAFT. GRAFT is first on eight of the eleven datasets and second on SUN397, EuroSAT and Flowers102, where EM-Dirichlet, strong at but collapsing with large batches, leads by , and points. Against StatA, GRAFT improves on every dataset. The margin is largest on the fine-grained vocabularies with many confusable classes, Stanford Cars (), FGVC-Aircraft () and SUN397 (), and smallest on the coarse ones, Caltech101 (), EuroSAT () and UFC101 (). Table 4 extends the comparison to the ImageNet distribution shifts with the 1000-class vocabulary. The ranking is unchanged under the four shifts; StatA improves over zero-shot by to points, and the gain of GRAFT over StatA grows from points on ImageNet to to points on the shifted sets, where the displacement from the text anchors, and with it the shared component that the graph propagates, is largest.

Figure 6 (PDF p. 7). Posterior mass assigned to classes absent from the batch versus the number of effective classes (ImageNet, ).
Qualitative results
Figure 7 shows test images from batches of the low setshot model, StatA and GRAFT. The typical correction is a sample of a present class that the zero-shot model assigns to a semantically close class absent from the batch (an Egyptian cat for a tabby cat). StatA recovers some of these samples when the present class has enough evidence to move its prototype, and GRAFT recovers the remaining ones because the absent neighbour is removed through the presence model and displaced consistently with the present class. The failure case, a stone wall predicted as a cliff by all three methods, has no close neighbour in the batch. Figure 6 quantifies the mechanism on ImageNet batches of 64: with a single present class the zero-shot model assigns 31.0% of its posterior mass to classes that are not in the batch, StatA 11.5% and GRAFT 4.1%. With fifty present classes the values converge to 8.1%, 3.0% and 1.2%.
Ablation Studies
Components. Table 5 adds the components one at a time. Without anchor, coupling or presence the mixture reaches
| EB | Low () | All | ||||
| zero-shot | 65.2 | 65.2 | ||||
| − | − | − | − | ✓ | 43.3±0.5 | 70.3 |
| ✓ | − | − | − | ✓ | 55.8±0.5 | 70.3 |
| − | ✓ | − | − | ✓ | 69.1±0.3 | 69.9 |
| ✓ | ✓ | − | − | ✓ | 69.6±0.3 | 69.9 |
| − | ✓ | ✓ | − | ✓ | 70.2±0.3 | 70.2 |
| ✓ | ✓ | ✓ | − | − | 70.4±0.3 | 68.7 |
| ✓ | ✓ | ✓ | − | ✓ | 70.7±0.3 | 70.2 |
| ✓ | ✓ | ✓ | ✓ | ✓ | 70.8±0.3 | 70.2 |
Table 5. Component ablation, top-1 accuracy (%): presence, anchor, graph coupling, EB empirical Bayes, sample graph.
43.3 in the low setting, 22 points below the zero-shot model, but 70.3 on the full set, where every class is well observed. The presence model alone raises the low-setting result to 55.8, the anchor term alone to 69.1, and the two together to 69.6, the anchor-only regime of StatA and MOON. The graph coupling adds 1.1 points on top (70.7), the margin that distinguishes GRAFT from the anchored methods, and empirical Bayes a further 0.1. The inherited sample graph contributes 0.3 points in the low setting but 1.5 on the full set, where the anchor and the coupling change the result by at most 0.4 points.
Prior and graph. We vary , and the number of neighbours , and replace the text kNN graph by a complete graph and by a degree-matched random graph (table and sensitivity curves in the supplementary material). Accuracy in the low setting varies by at most 0.6 points for between 1 and 16 and by 0.3 points for between 3 and 10, so neither is a sensitive hyper-parameter; removing the anchor () costs 1.7 points and removing the coupling () 1.6. The random graph (69.2) performs like no coupling (69.1) and the complete graph (68.8) is below both, so pooling helps only when it follows the semantic neighbourhoods, the control that H1 of § 4.2 requires. H1 itself is tested with labels (supplementary material): the energy ratio of the oracle displacement field on the text graph against degree-matched random graphs lies between 0.25 and 0.75 on all datasets, and the datasets with the highest ratios have the smallest margins in Table 3. Likelihood and optimisation variants change the result by at most 0.5 points.
Conclusion
We traced the collapse of unlabeled VLM adaptation on realistic batches to independent per-class parameters and replaced them by a displacement field over the vocabulary with a Gaussian Markov random field prior on a semantic class graph. GRAFT is closed form, contains anchor-only shrinkage and a global shift as extreme settings, and improves over anchored transduction on realistic batches.
LLM Usage Statement
Generative AI tools were used for language editing of the manuscript and to assist with the literature search. They were not used to generate experimental results, numerical values, or the statements and proofs of the theoretical results, all of which were derived by the authors. The authors take full responsibility for the final content of this work, including all text, claims and artifacts.
References
- [1]Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. In ECCV, 2014. 6DOI
- [2]Behzad Bozorgtabar, Dwarikanath Mahapatra, Sudipta Roy, Muzammal Naseer, Imran Razzak, and Zongyuan Ge. LATA: Laplacian-assisted transductive adaptation for conformal uncertainty in medical VLMs. In CVPR, 2026. 2arxiv.org/abs/2602.17535
- [3]Ziliang Chen, Yulu Li, Liangda Fang, Jusheng Zhang, Yongsen Zheng, Quanlong Guan, and Xipeng Chen. Vocabulary scaling law: Tuning open-vocabulary predictors for their openness. In CVPR, 2026. 2
- [4]Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023. 6DOI
- [5]Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014. 6arxiv.org/abs/1311.3618
- [6]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 5
- [7]Florian Dörfler and Francesco Bullo. Kron reduction of graphs with applications to electrical networks. IEEE Transactions on Circuits and Systems I, 60(1):150–163, 2013. 5arxiv.org/abs/1102.2950
- [8]Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, and Elisa Ricci. Frustratingly easy test-time adaptation of vision-language models. In NeurIPS, 2024. 2arxiv.org/abs/2405.18330
- [9]Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories. In CVPR Workshops, 2004. 6
- [10]Clément Fuchs, Maxime Zanella, and Christophe De Vleeschouwer. Online gaussian test-time adapta- tion of vision-language models. In CVPR Workshops, 2025. 1, 2, 6, 7arxiv.org/abs/2501.04352
- [11]Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannan Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. DataComp: In search of the next generation of multimodal datasets. In NeurIPS, 2023. 6arxiv.org/abs/2304.14108
- [12]Saurabh Garg, Yifan Wu, Sivaram Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. In NeurIPS, 2020. 2arxiv.org/abs/2003.07554
- [13]Pengxiao Han, Changkun Ye, Yanshuo Wang, Jinguang Tong, Miaohua Zhang, Xuesong Li, Jie Hong, and Lars Petersson. Label shift aware adaptation for online zero-shot learning with contrastive language-image pre-training (CLIP). arXiv preprint arXiv:2606.15169, 2026. 2arxiv.org/abs/2606.15169
- [14]Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 6
- [15]Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021. 6
- [16]Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021. 6
- [17]Fanding Huang, Jingyan Jiang, Qinting Jiang, Hebei Li, Faisal Nadeem Khan, and Zhi Wang. COSMIC: Clique-oriented semantic multi-space integration for robust CLIP test-time adaptation. In CVPR, 2025. 2arxiv.org/abs/2503.23388
- [18]Jiazhen Huang, Zhiming Liu, Changhu Wang, Wei Ju, Ziyue Qiao, and Xiao Luo. Von mises-fisher mixture model with dynamic shrinkage for realistic test-time transduction. In ICML, 2026. 1, 2, 4, 6, 7, 8arxiv.org/abs/2607.15851
- [19]David R Hunter and Kenneth Lange. A tutorial on MM algorithms. The American Statistician, 58(1):30–37, 2004. 5DOI
- [20]Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P. Xing. Rethinking knowledge graph propagation for zero-shot learning. In CVPR, 2019. 2arxiv.org/abs/1805.11724
- [21]Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In CVPR, 2024. 1, 2, 6, 7arxiv.org/abs/2403.18293
- [22]Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In ICCV Workshops, 2013. 6DOI
- [23]Yushu Li, Yongyi Su, Adam Goodge, Kui Jia, and Xun Xu. Efficient and context-aware label propagation for zero/few- shot training-free adaptation of vision-language model. In ICLR, 2025. 1, 2
- [24]Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, 2022. 1, 2arxiv.org/abs/2203.02053
- [25]Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Prototype propagation networks (PPN) for weakly-supervised few-shot learning on category graph. In IJCAI, 2019. 2arxiv.org/abs/1905.04042
- [26]Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013. 6arxiv.org/abs/1306.5151
- [27]Ségolène Martin, Yunshi Huang, Fereshteh Shakeri, Jean-Christophe Pesquet, and Ismail Ben Ayed. Transductive zero-shot and few-shot CLIP. In CVPR, 2024. 1, 2, 6, 7, 8arxiv.org/abs/2405.18437
- [28]Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008. 6DOI
- [29]Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C V Jawahar. Cats and dogs. In CVPR, 2012. 6
- [30]Qi Qian and Juhua Hu. Online zero-shot classification with CLIP. In ECCV, 2024. 2arxiv.org/abs/2408.13320
- [31]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 6, 7, 8arxiv.org/abs/2103.00020
- [32]Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In ICML, 2019. 6arxiv.org/abs/1902.10811
- [33]Håvard Rue and Leonhard Held. Gaussian Markov Random Fields: Theory and Applications. Chapman and Hall/CRC, 2005. 3
- [34]Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. Neural Computation, 14(1):21–41, 2002. 2, 5
- [35]Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language models. In ICLR, 2025. 1, 2arxiv.org/abs/2404.07983
- [36]Lijun Sheng, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. The illusion of progress? a critical look at test-time adaptation for vision-language models. In NeurIPS, 2025. 1, 2, 6arxiv.org/abs/2506.24000
- [37]Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS, 2022. 2
- [38]Julio Silva-Rodríguez, Ismail Ben Ayed, and Jose Dolz. Conformal prediction for zero-shot models. In CVPR, 2025. 2
- [39]Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes fromarxiv.org/abs/1212.0402
- [40]Vladan Stojnić, Yannis Kalantidis, and Giorgos Tolias. Label propagation for zero-shot classification with vision-language models. In CVPR, 2024. 1, 2, 6, 7, 8arxiv.org/abs/2404.04072
- [41]Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. Just shift it: Test-time prototype shifting for zero-shot generalization with vision-language models. In WACV, 2025. 2arxiv.org/abs/2403.12952
- [42]Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021. 6, 7arxiv.org/abs/2006.10726
- [43]Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019. 6
- [44]Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010. 6DOI
- [45]Maxime Zanella and Ismail Ben Ayed. On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning? In CVPR, 2024. 2, 6, 7, 8arxiv.org/abs/2405.02266
- [46]Maxime Zanella, Benoît Gérin, and Ismail Ben Ayed. Boosting vision-language models with transduction. In NeurIPS, 2024. 1, 2, 3, 5, 6, 7, 8arxiv.org/abs/2406.01837
- [47]Maxime Zanella, Clément Fuchs, Christophe De Vleeschouwer, and Ismail Ben Ayed. Realistic test-time adaptation of vision-language models. In CVPR, 2025. 1, 2, 3, 5, 6, 7, 8arxiv.org/abs/2501.03729
- [48]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 1, 6arxiv.org/abs/2303.15343
- [49]Ce Zhang, Simon Stepputis, Katia Sycara, and Yaqi Xie. Dual prototype evolving for test-time generalization of vision-language models. In NeurIPS, 2024. 1, 2arxiv.org/abs/2410.12790
- [50]Taolin Zhang, Jinpeng Wang, Hanqu Guo, Tao Dai, Bin Chen, and Shu-Tao Xia. BoostAdapter: Improving vision-language test-time adaptation via regional bootstrapping. In NeurIPS, 2024. 2arxiv.org/abs/2410.15430
- [51]Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision-language models. In CVPR, 2024. 1, 2, 6, 7arxiv.org/abs/2403.17589
- [52]Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NeurIPS, 2003. 2
- [53]Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022. 5DOI
- [54]Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. In NeurIPS, 2024. 2arxiv.org/abs/2410.19294
- [55]Imtiaz Ziko, Dounia Zou, Eric Granger, and Ismail Ben Ayed. Laplacian regularized few-shot learning. In ICML, 2020. 2, 5arxiv.org/abs/2006.15486

