betaFeedback

WorksComputer Science

GRAFT: Graph-Regularized Prototype Fields for Realistic Test-Time Adaptation of VLMs

Abstract

Unlabeled test data can improve the zero-shot classifier of a vision-language model, yet transductive and test-time adaptation methods often fall below the zero-shot baseline when only a few vocabulary classes are present in a batch or when streams are correlated. We trace this failure to a shared design: every class owns independent parameters estimated from its own samples, so absent or rare classes are left at biased text anchors or fitted to foreign samples. We propose Graph-Regularized Anchor-displacement Field Transduction (GRAFT), which models the adapted prototypes as text anchors plus a displacement field that is partially pooled over a semantic class graph through a Gaussian Markov random field prior, together with an explicit class-presence model. Inference is closed-form and gradient-free on cached features. Anchor-only shrinkage and a single global shift are the two extreme settings of the prior; absent classes are provably harmonic extensions of the present ones, and the pooling strength can be chosen by empirical Bayes. Under the realistic batch and online protocols on eleven benchmarks and four ImageNet shifts, GRAFT improves over the zero-shot model in every setting and over anchor-based transduction where few classes are present.

Introduction

Vision-language models (VLMs) such as CLIP [31] and SigLIP [48] classify an image by comparing its embedding with the text embeddings of class names. The resulting zero-shot classifier is general but frozen, and a growing family of methods adapts it from the unlabeled test data itself, transductively on a batch [23, 27, 40, 46] or online on a stream [10, 21, 49, 51], without training and on cached features.

These methods are evaluated on batches that contain every class of the vocabulary in comparable numbers, whereas

a photo album or a user session contains a handful of the classes that the vocabulary names, and streams are temporally correlated. Under such conditions transductive methods lose tens of accuracy points with respect to the zero-shot model they are meant to improve [47], and a controlled benchmark reports marginal gains, poor transfer across backbones and degraded calibration [36]. The remedy proposed in [47] and refined in [18] is a statistical anchor: each class prototype is shrunk towards its text embedding with a strength that grows with the number of test samples the class receives. Anchoring prevents collapse but treats every class in isolation: a class that receives no samples stays at its text anchor, although that anchor is displaced from the image distribution by the modality gap and by the domain of the test data [24, 35], and a class with one or two samples is adapted from those samples alone. The problem addressed here is therefore to adapt all prototypes of a large vocabulary from an unlabeled batch in which most classes are absent or rare, without collapsing on the absent ones and without discarding what the present ones reveal about them.

The displacement between the text anchor of a class and the centroid of its images contains a component shared by all classes, the modality and domain offset, and a component that varies smoothly with the semantics of the class. GRAFT turns this structure into a prior. The adapted prototypes are written as μk=tk+dk\mu_k = t_k + d_k and the displacement field D=[d1;…;dK]D = [d_1; \ldots; d_K] receives a Gaussian Markov random field prior whose precision is αI+λLT\alpha I + \lambda L_T, where LTL_T is the Laplacian of a nearest-neighbour graph over the text embeddings. The α\alpha term reproduces the anchor-only shrinkage of [18, 47]. The λ\lambda term couples the classes, so that the displacement of a well-observed class propagates to its semantic neighbours. The maximum a posteriori field has a closed form: a graph filter that returns the empirical displacement for well-observed classes and the harmonic extension of the neighbours’ displacements for absent ones (Figure 1c). Classes that receive no samples are additionally removed from the competition by a Dirichlet-regularised estimate of the class proportions, which treats the vocabulary as a label-shift problem [12, 34]. All updates are closed form on cached features, and the prior strengths can be selected by maximising the marginal likelihood of the unlabeled batch, which removes the dataset-specific tuning criticised in [36].

Per-class adaptation on a realistic batch

Figure 1. Per-class adaptation on a realistic batch (conceptual illustration). Classes A and B are present, C is absent, and all images are shifted from their anchors. (a) Per-class fitting lets C capture samples of B. (b) Anchor-only shrinkage leaves C at its biased anchor. (c) GRAFT lets C inherit B’s displacement and removes it through the presence model.

Contributions.

  • We formulate transductive VLM adaptation as the estimation of a displacement field over the class vocabulary with a graph-structured prior, and derive closed-form block-coordinate inference for batches and streams (§ 4); anchor-only shrinkage and a single global shift are its two extreme settings, and absent classes receive the harmonic extension of the present-class displacements.

  • We combine the field with an explicit class-presence model and an empirical Bayes rule for the prior strength, so that the vocabulary is treated as a label-shift problem and no pooling strength has to be tuned per dataset.

  • We evaluate GRAFT under the realistic batch and online protocols on eleven benchmarks and four ImageNet shifts, with architecture-matched controls that isolate the graph coupling and the presence model.

Related Work

Unlabeled adaptation of vision-language models. Transductive methods classify an unlabeled batch jointly: TransCLIP [46] fits a Gaussian mixture regularised by a KL term to the zero-shot posterior and a sample-graph Laplacian, EM-Dirichlet [27] works on the simplex, ZLaP [40] and ECALP [23] propagate labels over image and prompt graphs, and LATA [2] and Conf-OT [38] target conformal prediction. Online methods adapt over a stream with feature caches [21, 50, 51], evolving prototypes [49], per-class Gaussians [10] or online label distributions [13, 30]; episodic methods act on single images [8, 37, 41, 45]. The realistic protocols with a variable number of effective classes and correlated streams were introduced in [47], which showed that these methods collapse under them and proposed StatA, a shrinkage of each class’s Gaussian parameters towards its text embedding with strength nk/(nk+α)n_k/(n_k + \alpha). MOON [18] derives this shrinkage from penalised likelihood on the hypersphere and makes it confidence dependent. Both anchor every class to its own text embedding and use no information from other classes; their mean update is the λ=0\lambda= 0 case of GRAFT. A unified benchmark further reports marginal gains, poor transfer to SigLIP and worse calibration [36].

Structure over classes and label shift. Class graphs propagate classifier weights to unseen classes in zero-shot learning [20], propagate prototypes along a category hierarchy [25] and select correlated classes in CLIP test-time adaptation [17]. Laplacian regularisation over samples is standard in transductive few-shot learning [52, 55]. We are not aware of prior work that regularises the displacement of VLM prototypes from their text anchors with a Gaussian Markov random field over the class graph. The modality gap literature [24, 35] supports the premise that much of the displacement is shared. Adjusting a classifier to new class priors from unlabeled data is classical [12, 34]; Frolic [54] corrects the label bias of CLIP without labels, and open-vocabulary classifiers degrade as distractor classes are added [3].

Preliminaries

Zero-shot classification. A VLM provides an image encoder fvf_v and a text encoder ftf_t mapping into a common space Rd\mathbb{R}^d. For a vocabulary of KK class names {ck}k=1K\{c_k\}_{k=1}^{K}, prompt ensembling yields unit-norm text anchors tk=ft(ck)/∥ft(ck)∥t_k = f_t(c_k)/\lVert f_t(c_k)\rVert, collected in T∈RK×dT \in\mathbb{R}^{K \times d}. A test image IiI_i gives vectors xi=fv(Ii)/∥fv(Ii)∥x_i = f_v(I_i)/\lVert f_v(I_i)\rVert, collected in X∈RN×dX \in\mathbb{R}^{N \times d}. The zero-shot posterior is

y^ik=exp⁡(τxiTtk)∑lexp⁡(τxiTtl)(1)\hat{y}_{ik} = \frac{\exp(\tau x_i^\mathsf{T} t_k)}{\sum_l \exp(\tau x_i^\mathsf{T} t_l)} \tag*{(1)}

with logit scale τ\tau (100 for CLIP). We write ΔK\Delta^K for the probability simplex and 1\mathbf{1} for the all-ones vector.

Unlabeled adaptation. Given XX and TT only, the goal is to output posteriors zi∈ΔKz_i \in\Delta^K more accurate and at least as well calibrated as y^i\hat{y}_i, for a batch processed at once or for a stream of batches predicted as they arrive. Following [47], a batch contains Keff≤KK_{\mathrm{eff}} \le K effective classes, often far fewer than the vocabulary, and streams are correlated.

Mixture-model transduction and its anchors. TransCLIP [46] and StatA [47] model xix_i with a KK-component Gaussian mixture and minimise, over assignments zz and parameters, a penalised negative log-likelihood with a KL anchor to y^\hat{y} and a Laplacian term over a nearest-neighbour graph of the samples. StatA additionally shrinks the mean of class kk towards its anchor,

μk=βkxˉk+(1−βk)tk,βk=nknk+α,(2)\mu_k = \beta_k \bar{x}_k + (1-\beta_k)t_k,\qquad\beta_k = \frac{n_k}{n_k+\alpha}, \tag*{(2)}

with nk=∑izikn_k = \sum_i z_{ik} the soft count and xˉk\bar{x}_k the responsibility-weighted mean. Equation (2) is the posterior mean of μk\mu_k under an independent Gaussian prior centred at tkt_k: a class with nk=0n_k = 0 keeps μk=tk\mu_k = t_k even when every present class has moved by a common offset, a class with small nkn_k is estimated from its own few samples only, and the mixture weights are implicitly uniform, so absent classes keep competing for samples. GRAFT addresses the first two points by coupling the classes through a graph prior and the third by estimating the class proportions.

Method

GRAFT keeps the generative view of transduction [46, 47] but changes the object that is adapted. Instead of KK independent prototypes it estimates a displacement field D∈RK×dD \in\mathbb{R}^{K \times d} anchored at the text embeddings, μk=tk+dk\mu_k = t_k+d_k, under a prior that couples semantically related classes, together with the class proportions π∈ΔK\pi\in\Delta^K and a shared covariance. Figure 2 shows the components: the frozen encoders with the zero-shot prior of Eq. (1) and the two graphs (§ 4.1), the likelihood and the graph-regularised field (§ 4.2), the class-presence model (§ 4.3), the objective (§ 4.4), the closed-form inference (§ 4.5) and the empirical Bayes rule for the prior strength (§ 4.6).

Overview of GRAFT

Figure 2. Overview of GRAFT. Frozen encoders give features XX and anchors TT, which define the zero-shot prior y^\hat{y}, the sample graph WSW_S (from XX, images to images) and the semantic class graph LTL_T (from TT, classes to classes). Closed-form updates alternate the E-step over zz, using y^\hat{y} and WSW_S, with M-steps for the presence π\pi, the displacement field DD pooled over LTL_T and the shared covariance; μ=T+D\mu= T + D are the adapted prototypes and zz the prediction.

Sample graph and semantic class graph

Two graphs enter the model, and they are built from different sources. The sample graph Ws∈RN×NW_s \in\mathbb{R}^{N \times N}, inherited from [46, 47], is data driven: it connects each test image to its msm_s nearest neighbours in feature space with weight (Ws)ii′=xiTxi′(W_s)_{ii'} = x_i^\mathsf{T}x_{i'} and encodes the manifold assumption that neighbouring images share a label. It is recomputed for every batch and says nothing about classes that have no image in the batch.

The semantic class graph, introduced in this work, is built from the vocabulary only. Its nodes are the KK classes and its edges are derived from the text anchors TT produced by the frozen text encoder, so that it depends on the class names and the prompts, not on the images, the batch or the responsibilities. Let S=TTTS = TT^\mathsf{T} be the cosine similarity between anchors. Class kk is connected to the mm classes with the largest SklS_{kl}, l≠kl \ne k, with the heat-kernel weight

wkl=exp⁡((Skl−1)/ρ)∈(0,1],(3)w_{kl} = \exp\left((S_{kl}-1)/\rho\right) \in(0,1], \tag*{(3)}

with bandwidth ρ\rho. The neighbourhoods are symmetrised by union, W=max⁡(W,WT)W = \max(W,W^\mathsf{T}), and the normalised Laplacian is LT=I−D−1/2WD−1/2L_T = I-D^{-1/2}WD^{-1/2} with D=diag⁡(W1)D = \operatorname{diag}(W\mathbf{1}). LT∈RK×KL_T \in\mathbb{R}^{K \times K} is positive semi-definite with spectrum in [0,2][0,2] and is computed once per vocabulary. Because it is built from text semantics, every class has neighbours, including those that receive no sample in a batch, so the field below is defined on the whole vocabulary. The graph expresses the hypothesis, that classes with similar names are displaced from their anchors in similar directions.

Graph-regularised displacement field

Likelihood. Each class is a Gaussian with mean μk=tk+dk\mu_k = t_k+d_k and a diagonal covariance Σ=diag⁡(σ2)\Sigma= \operatorname{diag}(\sigma^2) shared across classes. As in TransCLIP, the likelihood is tempered by κ\kappa, which acts as a covariance inflation and keeps the responsibilities soft on dd-dimensional features:

ℓik(D,σ)=−12κ[∑j=1d(xij−tkj−dkj)2σj2+∑j=1dlog⁡σj2].(4)\ell_{ik}(D,\sigma) = -\frac{1}{2\kappa}\left[\sum_{j=1}^{d}\frac{(x_{ij}-t_{kj}-d_{kj})^2}{\sigma_j^2}+\sum_{j=1}^{d}\log\sigma_j^2\right]. \tag*{(4)}

Given responsibilities zz, the shared variance has the closed-form M-step σj2=1N∑i,kzik(xij−μkj)2\sigma_j^2 = \frac{1}{N}\sum_{i,k}z_{ik}(x_{ij}-\mu_{kj})^2.

How the graph defines the field. The displacement field D∈RK×dD \in\mathbb{R}^{K \times d} assigns to every class kk of the vocabulary the vector dkd_k by which its prototype departs from its text anchor. The class graph enters through a Gaussian Markov random field prior [33] on this field, which factorises over feature dimensions,

p(D)∝exp⁡(−12∑j=1dd⋅jTQd⋅j),Q=αIK+λLTκσˉ2,(5)p(D) \propto\exp\left(-\frac{1}{2}\sum_{j=1}^{d}d_{\cdot j}^{\mathsf{T}}Qd_{\cdot j}\right),\qquad Q=\frac{\alpha I_K+\lambda L_T}{\kappa\bar{\sigma}^2}, \tag*{(5)}

with σˉ2=1d∑jσj2\bar{\sigma}^2 = \frac{1}{d}\sum_j \sigma_j^2. The two terms of the precision QQ have distinct roles. The term αIK\alpha I_K pulls every displacement towards zero, i.e., every prototype towards its own text anchor; this is the anchoring of StatA and MOON. The term λLT\lambda L_T is the coupling introduced here: it penalises the Dirichlet energy ∑k∼lwkl∥dk/Dk−dl/Dl∥2\sum_{k\sim l}w_{kl}\lVert d_k/\sqrt{D_k}-d_l/\sqrt{D_l}\rVert^2, so that displacements of classes joined by an edge are asked to agree, with a strength given by the semantic weight wklw_{kl}. The conditional of the GMRF makes the role of the graph explicit: under Eq. (5), the prior mean of dkd_k given the displacements of all other classes is

E[dk∣d−k]=λα+λ∑l∼kwklDkDldl,(6)\mathbb{E}[d_k\mid d_{-k}] = \frac{\lambda}{\alpha+\lambda}\sum_{l\sim k}\frac{w_{kl}}{\sqrt{D_kD_l}}d_l, \tag*{(6)}

a weighted average of the neighbours’ displacements on the class graph, shrunk towards zero by the anchor. A class therefore inherits, as a priori, the displacement of the classes whose names are closest to its own. Dividing by κσˉ2\kappa\bar{\sigma}^2 expresses α\alpha and λ\lambda in units of sample counts.

How the field is regularised and updated. Three forces act on every displacement. The data term of the likelihood pulls dkd_k towards the empirical displacement ek=xˉk−tke_k = \bar{x}_k-t_k of the images currently assigned to class kk, with weight equal to the soft count nk=∑izikn_k = \sum_i z_{ik} (here xˉk=∑izikxi/nk\bar{x}_k = \sum_i z_{ik}x_i/n_k, and ek=0e_k = 0 when nk=0n_k = 0); the anchor term pulls it towards zero with weight α\alpha; and the coupling term pulls it towards the neighbour average of Eq. (6) with weight λ\lambda. Writing Nz=diag⁡(n1,…,nK)N_z = \operatorname{diag}(n_1,\ldots,n_K), the maximum a posteriori field given the responsibilities balances the three and is obtained for every feature dimension jj from a K×KK \times K linear system,

d:j=(Nz+sj(αIK+λLT))−1Nze:j,sj=σj2σˉ2.(7)d_{:j} = \left(N_z+s_j(\alpha I_K+\lambda L_T)\right)^{-1}N_z e_{:j}, \qquad s_j = \frac{\sigma_j^2}{\bar{\sigma}^2}. \tag*{(7)}

Equation [7] is a graph filter applied to the empirical displacements (Figure 3). Writing A~=D−1/2WD−1/2\widetilde{A}=D^{-1/2}WD^{-1/2} for the normalised adjacency and M=Nz+sj(α+λ)IKM=N_z+s_j(\alpha+\lambda)I_K for the diagonal part, it expands as the convergent series

Graph-pooled displacement field on a toy class graph

Figure 3. Graph-pooled displacement field on a toy class graph (node size: nkn_k). (a) Empirical displacements. (b) Independent shrinkage (λ=0\lambda= 0) leaves absent classes at their anchors. (c) Eq. (7) transfers displacements along the graph.

d:j=∑t≥0(sjλM−1A~)tM−1Nze:j,(8)d_{:j}=\sum_{t\geq0}\left(s_j\lambda M^{-1}\widetilde{A}\right)^tM^{-1}N_z e_{:j}, \tag*{(8)}

in which the evidence-weighted empirical displacements Nze:jN_z e_{:j} are diffused along the class graph: the term tt carries the displacement of present classes to classes tt edges away, attenuated at every hop by sjλ/(nl+sj(α+λ))s_j\lambda/(n_l+s_j(\alpha+\lambda)). Hence a class with nk≫α+λn_k\gg\alpha+\lambda keeps its empirical displacement, a class with few samples mixes its own noisy estimate with its neighbours’ displacements in proportion to nkn_k against λ\lambda, and a class with no sample receives only what diffuses from

its neighbours. The field is re-estimated at every outer iteration: the responsibilities zz determine nkn_k and eke_k, Eq. (7) gives DD, the prototypes μ=T+D\mu= T + D enter the likelihood of Eq. (4), and the next E-step updates zz (§ 4.5); the dd systems are solved jointly by preconditioned conjugate gradients.

Special cases. With λ=0\lambda= 0, Eq. (7) reduces per dimension to dkj=nknk+αsjekjd_{kj} = \frac{n_k}{n_k+\alpha s_j}e_{kj}, the count-based anchor shrinkage of StatA (Eq. (2)) and MOON [18], which are therefore the graph-free member of the family. With α=0\alpha= 0, LTL_T the Laplacian of the complete graph and λ→∞\lambda\to\infty, every class receives the count-weighted mean displacement d=∑knkek/∑knkd = \sum_k n_k e_k/\sum_k n_k, a single global shift of all prototypes. GRAFT interpolates between these extremes along the semantic neighbourhoods.

Absent classes. Let A={k:nk=0}A = \{k : n_k = 0\} be the absent classes, PP their complement and Qλ=αI+λLTQ_\lambda= \alpha I + \lambda L_T. Since the rows of Eq. (7) in AA have nk=0n_k = 0 and ek=0e_k = 0, the solution satisfies Qλ,AADA+Qλ,APDP=0Q_{\lambda,AA}D_A + Q_{\lambda,AP}D_P = 0, so

DA=−Qλ,AA−1Qλ,APDP(9)D_A = -Q_{\lambda,AA}^{-1}Q_{\lambda,AP}D_P \tag*{(9)}

is the harmonic extension of the present-class field, and DPD_P solves Eq. (7) on PP with QλQ_\lambda replaced by its Kron reduction Qλ/Qλ,AAQ_\lambda/Q_{\lambda,AA} [7]: absent classes affect the present-class prototypes only through the effective graph, never through data (Figure 1c).

Posterior variance. Given zz, the posterior covariance of d:jd_{:j} is κσj2(Nz+sjQλ)−1\kappa\sigma_j^2(Nz+s_jQ_\lambda)^{-1}; since λLT⪰0\lambda L_T \succeq0, Var⁡(dkj)≤κσj2/(nk+sjα)\operatorname{Var}(d_{kj}) \le\kappa\sigma_j^2/(n_k+s_j\alpha), the variance under independent shrinkage. Proofs are in the supplementary material.

Smoothness hypothesis. None of these statements guarantees higher accuracy than zero-shot inference. The benefit of the coupling rests on a testable hypothesis, class-level smoothness (H1): the oracle displacement field ek⋆=E[x∣y=k]−tke_k^\star= \mathbb{E}[x \mid y = k] - t_k has a Dirichlet energy on LTL_T substantially smaller than on a degree-matched random graph. If H1 fails, GRAFT reduces to anchor-only shrinkage.

Class presence as label shift

The mixture weights π\pi are free parameters with a symmetric Dirichlet prior Dir⁡(c+1)\operatorname{Dir}(c+1). Their M-step is the smoothed count

πk=nk+cN+Kc,(10)\pi_k = \frac{n_k+c}{N+Kc}, \tag*{(10)}

and they enter the E-step as log⁡πk\log\pi_k, which is the classical label-shift correction of a calibrated classifier [34]: the zero-shot posterior implicitly assumes uniform class priors, and Eq. (10) replaces them by the estimate obtained from the batch. For a batch with Keff≪KK_{\mathrm{eff}} \ll K classes, absent classes receive πk≈c/(N+Kc)\pi_k \approx c/(N+Kc) while present ones receive nk/(N+Kc)n_k/(N+Kc): with c=12c=\frac{1}{2}, a present class with sixteen samples is more than thirty times more likely a priori than an absent one in a batch of 64 over 1000 classes. The pseudo-count keeps log⁡πk\log\pi_k finite and lets a class absent from earlier online batches re-enter the stream.

Objective

Collecting the terms, GRAFT minimises over (z,D,σ,π)(z,D,\sigma,\pi)

J=∑i,kzik[log⁡zik−log⁡πk−λylog⁡y^ik−ℓik(D,σ)]−λL2ms∑i,i′(Ws)ii′ziTzi′+12∑jd:jTQd:j−c∑klog⁡πk,(11)\begin{aligned} J ={}& \sum_{i,k} z_{ik}\left[\log z_{ik}-\log\pi_k-\lambda_y\log\hat{y}_{ik}-\ell_{ik}(D,\sigma)\right] \\ &- \frac{\lambda_L}{2m_s}\sum_{i,i'}(W_s)_{ii'}z_i^\mathsf{T}z_{i'}+\frac{1}{2}\sum_j d_{:j}^\mathsf{T}Qd_{:j}-c\sum_k\log\pi_k, \tag*{(11)} \end{aligned}

where the first line is the variational free energy of the mixture with the text prior y^λy\hat{y}^{\lambda_y} of [46], and the second line contains the inherited sample-graph term, the field prior of Eq. (5) and the presence prior. With λ=0\lambda=0 and without the π\pi terms this is the StatA objective up to its per-class covariances; with α=λL=0\alpha=\lambda_L=0 in addition, the unanchored mixture.

Inference

The objective is minimised by block-coordinate descent over four blocks (pseudocode in the supplementary material). The E-step is the mean-field update with the Laplacian term linearised at the current zz, a majorise-minimise step [19, 46, 55]:

zik∝πky^ikλyexp⁡(ℓik+λL2ms[(Ws+WsT)z]ik),(12)z_{ik} \propto\pi_k\hat{y}_{ik}^{\lambda_y}\exp\left(\ell_{ik}+\frac{\lambda_L}{2m_s}\left[(W_s+W_s^\mathsf{T})z\right]_{ik}\right), \tag*{(12)}

iterated a few times. The M-steps are the presence update of Eq. (10), the field of Eq. (7) and the shared variance of § 4.2. Each block update is the exact minimiser of Eq. (11) over its block when λL=0\lambda_L=0, so JJ decreases monotonically and converges; with λL>0\lambda_L>0 the E-step minimises a majoriser of JJ, as in [46]. All quantities depend on the data only through the sufficient statistics n=zT1n=z^\mathsf{T}\mathbf{1}, S=zTXS=z^\mathsf{T}X and q=∑ixi2q=\sum_i x_i^2, which makes the online variant immediate: the statistics are accumulated over the stream with an optional forgetting factor ϱ\varrho, and the same M-steps are applied after every batch. The prediction for a batch is its posterior zz after adaptation, as in the transductive protocol of [47]; a strictly inductive variant predicts before the statistics are updated. Per outer iteration the E-step costs O(NKd+NmsK)O(NKd+Nm_sK) and the field O(ncKd(1+m))O(n_cKd(1+m)) with conjugate gradients. No backward pass is executed and the VLM is untouched after feature extraction.

Empirical Bayes selection of the pooling strength

Fixed (α,λ)(\alpha,\lambda) are used for all datasets in the main protocol, but the model also admits a label-free rule: under the model, the empirical displacements of the present classes are Gaussian with a covariance that depends on (α,λ)(\alpha,\lambda) through the Schur complement of QQ onto the present classes, and their marginal likelihood is maximised over a small logarithmic grid at every M-step whenever at least three classes are present. The derivation is given in the supplementary material.

Experiments

Datasets

We follow the eleven-dataset suite of TransCLIP and StatA [46, 47] with the CoOp test splits [53]: ImageNet [6], SUN397 [44], FGVC-Aircraft [26], EuroSAT [14], Stanford Cars [22], Food101 [1], Oxford Pets [29], Flowers102 [28], Caltech101 [9], DTD [5] and UCF101 [39], plus the ImageNet distribution shifts ImageNet-V2 [32], ImageNet-Sketch [43], ImageNet-A [16] and ImageNet-R [15]. Class names and prompt templates are those of CoOp and StatA. Features are extracted once per backbone and cached. No training split is used. Batch adaptation follows the seven settings of [47]: a task draws KeffK_{\mathrm{eff}} classes uniformly from [1,4][1,4], [2,10][2,10] or [5,25][5,25] (very low, low, medium) with B=64B=64 images, from [5,25][5,25], [25,50][25,50] or [50,100][50,100] (medium, high, very high) with B=1000B=1000, or uses the full test set, with 1000 tasks per dataset. Online adaptation streams batches of 128 whose class composition follows Dirichlet(γ\gamma) slot proportions, γ∈{0.1,0.01,0.001}\gamma\in\{0.1,0.01,0.001\} (low to high correlation) or one class after another (separate), with 100 runs per dataset. The vocabulary sizes range from 10 (EuroSAT) to 1000 (ImageNet) and the test sets from 1,692 (DTD) to 50,000 images (ImageNet).

Baselines

We compare with zero-shot CLIP, the episodic MTA [45], the transductive EM-Dirichlet [27], ZLaP [40], TransCLIP [46], StatA [47] and MOON [18], and the online TENT [42], TDA [21], DMN [51] and OGA [10], all run with default hyper-parameters through the StatA harness into which GRAFT is plugged as a drop-in solver, so that samplers, features and prompts are identical. To attribute gains to the proposed mechanism rather than to the likelihood or the inherited terms, four architecture-matched controls share our implementation: anchor-only shrinkage (λ=0\lambda=0, the StatA/MOON mechanism in our likelihood), complete pooling (a single global shift: α=0\alpha=0, complete graph, λ→∞\lambda\to\infty), presence only (π\pi without displacement) and the unanchored mixture (α=λ=0\alpha=\lambda=0, no π\pi).

Implementation details

Inherited constants follow TransCLIP and StatA and are never tuned: κ=50\kappa=50, λy=1\lambda_y=1, λL=1\lambda_L=1, ms=3m_s=3, Tmax⁡=10T_{\max}=10 outer iterations with 3 inner E-step sweeps. The new constants are fixed once for all datasets and backbones: α=1\alpha=1 (one pseudo-count, as in StatA), λ=4\lambda=4, m=5m=5 neighbours, ρ=0.05\rho=0.05, c=12c=\frac{1}{2}. The linear systems are solved exactly for K≤256K \le256 and by 30 preconditioned conjugate-gradient iterations otherwise. Backbones: CLIP ViT-B/16 (main), ViT-L/14 and RN50 [31], OpenCLIP [4] ViT-B/16 trained on DataComp [11], and SigLIP ViT-B/16 [48], the backbone on which [36] found most TTA methods to fail. All runs use a single GPU. Seeds and task indices are fixed and shared across methods. We report top-1 accuracy averaged over tasks and datasets, the worst-decile accuracy, expected calibration error and time per task in supplementary material.

Results

Realistic batches

Table 1 and Figure 4 compare GRAFT with the state-of-the-art methods and the architecture-matched controls across the seven batch settings. First, the unanchored transductive methods collapse as the number of effective classes shrinks: TransCLIP falls from 70.3 with all classes to 38.9 in the very-low setting and ZLaP from 66.4 to 27.5, whereas EM-Dirichlet, the strongest method at B=64B=64, collapses in the opposite direction, to 29.5 on the full set. The unanchored control of our implementation reproduces the first pattern (42.4 to 45.4 at B=64B=64). Second, anchoring removes the collapse but adds little beyond it: StatA, MOON and the anchor-only control lie within 1.2 points of each other in every setting, so the count-dependent shrinkage they share is what prevents collapse. Third, pooling the field across classes adds a margin on top of anchoring that is largest where absent and rare classes dominate and vanishes where every class is well observed: GRAFT improves over the anchor-only control by 1.9, 1.6 and 1.1 points in the very-low, low and medium settings with B=64B=64, by 0.9, 0.7 and 0.5 points with B=1000B=1000, and by 0.3 points on the full set, where Sec. 4.2 predicts that the two mechanisms coincide; against MOON the margins are 0.7 to 0.1 points. The presence model alone reaches 67.4 in the very-low setting, about a third of the gain over zero-shot, and the complete-graph control, a single shift of all prototypes, stays below the anchor-only control in every setting, so the gain comes from pooling along the semantic neighbourhoods rather than from a global shift. Empirical Bayes selection of (α,λ)(\alpha,\lambda) matches the fixed strengths within 0.1 point, and standard deviations over tasks are 0.2 to 0.4 points for the anchored methods, well below these margins.

Figure 4. Top-1 accuracy versus the number of effective classes per batch (eleven-dataset mean, error bars: standard deviation over tasks); the dotted line is the zero-shot model.

Online streams

Table 2 reports the online protocol. The cache-based TDA and DMN are strongest on weakly correlated streams (67.7 and 67.267.2 at γ=0.1\gamma=0.1) but gain little on strongly correlated ones (66.866.8 and 66.366.3 at γ=0.001\gamma=0.001), whereas the anchored OGA, StatA and MOON improve with correlation (69.669.6, 69.569.5 and 69.969.9 at γ=0.001\gamma=0.001), since correlated batches contain few classes with many samples each. GRAFT follows the trend of the anchored methods and adds 1.21.2, 1.31.3 and 1.11.1 points over StatA at medium and high correlation and on the separate stream, and 0.60.6 at low correlation (0.20.2 to 0.90.9 over MOON). The inductive variant, which predicts each batch before updating the statistics, retains most of the margin (69.569.5, 70.370.3 and 69.869.8 against 70.170.1, 70.870.8 and 70.270.2).

MethodVery low B=64B=64Low B=64B=64Medium B=64B=64Medium B=1000B=1000High B=1000B=1000Very high B=1000B=1000All full
CLIP zero-shot [31]65.2±0.665.2 \pm0.665.2±0.565.2 \pm0.565.2±0.465.2 \pm0.465.2±0.265.2 \pm0.265.2±0.265.2 \pm0.265.2±0.165.2 \pm0.165.265.2
MTA [45]66.6±0.666.6 \pm0.666.6±0.566.6 \pm0.566.6±0.466.6 \pm0.466.6±0.266.6 \pm0.266.6±0.266.6 \pm0.266.6±0.166.6 \pm0.166.666.6
EM-Dirichlet [27]68.5±0.768.5 \pm0.770.3±0.670.3 \pm0.667.5±0.567.5 \pm0.564.4±0.464.4 \pm0.445.3±0.645.3 \pm0.633.6±0.733.6 \pm0.729.529.5
ZLaP [40]27.5±1.127.5 \pm1.135.2±0.935.2 \pm0.944.7±0.844.7 \pm0.841.5±0.541.5 \pm0.552.2±0.452.2 \pm0.458.4±0.358.4 \pm0.366.466.4
TransCLIP [46]38.9±1.038.9 \pm1.040.4±0.940.4 \pm0.942.7±0.842.7 \pm0.856.5±0.456.5 \pm0.462.0±0.362.0 \pm0.364.4±0.264.4 \pm0.270.3
StatA [47]70.4±0.470.4 \pm0.469.3±0.369.3 \pm0.367.4±0.367.4 \pm0.369.7±0.269.7 \pm0.269.8±0.269.8 \pm0.269.0±0.269.0 \pm0.269.969.9
MOON [18]71.3±0.471.3 \pm0.470.0±0.370.0 \pm0.367.9±0.367.9 \pm0.370.1±0.270.1 \pm0.270.1±0.270.1 \pm0.269.3±0.269.3 \pm0.270.170.1
Unanchored mixture (α=λ=0\alpha=\lambda=0, no π\pi)42.4±0.642.4 \pm0.643.3±0.543.3 \pm0.545.4±0.545.4 \pm0.557.7±0.357.7 \pm0.362.8±0.362.8 \pm0.364.5±0.264.5 \pm0.270.370.3
Presence only (π\pi, no field)67.4±0.567.4 \pm0.566.7±0.466.7 \pm0.465.3±0.465.3 \pm0.466.9±0.266.9 \pm0.266.6±0.266.6 \pm0.265.7±0.265.7 \pm0.266.066.0
Anchor-only shrinkage (λ=0\lambda=0)70.1±0.470.1 \pm0.469.1±0.369.1 \pm0.367.3±0.367.3 \pm0.369.6±0.269.6 \pm0.269.7±0.269.7 \pm0.269.0±0.269.0 \pm0.269.969.9
Complete pooling (global shift)69.6±0.469.6 \pm0.468.8±0.368.8 \pm0.366.8±0.366.8 \pm0.368.8±0.268.8 \pm0.268.6±0.268.6 \pm0.267.6±0.267.6 \pm0.268.368.3
GRAFT (fixed α\alpha, λ\lambda)72.0±0.472.0 \pm0.470.7±0.370.7 \pm0.368.4±0.368.4 \pm0.370.5±0.270.5 \pm0.270.4±0.270.4 \pm0.269.5±0.269.5 \pm0.270.270.2
GRAFT + empirical Bayes72.1±0.472.1 \pm0.470.8±0.370.8 \pm0.368.5±0.368.5 \pm0.370.6±0.270.6 \pm0.270.5±0.270.5 \pm0.269.5±0.269.5 \pm0.270.270.2

Table 1. Batch adaptation with a variable number of effective classes: top-1 accuracy (%) averaged over 1000 tasks and eleven datasets, CLIP ViT-B/16.

MethodLowMediumHighSeparate
CLIP zero-shot [31]65.2±0.365.2 \pm0.365.2±0.365.2 \pm0.365.2±0.365.2 \pm0.365.2±0.365.2 \pm0.3
TENT [42]65.8±0.465.8 \pm0.465.5±0.565.5 \pm0.565.3±0.665.3 \pm0.664.5±0.864.5 \pm0.8
TDA [21]67.7±0.467.7 \pm0.467.1±0.567.1 \pm0.566.8±0.666.8 \pm0.666.6±0.766.6 \pm0.7
DMN [51]67.2±0.467.2 \pm0.466.5±0.566.5 \pm0.566.3±0.666.3 \pm0.665.8±0.765.8 \pm0.7
OGA [10]67.5±0.567.5 \pm0.569.1±0.569.1 \pm0.569.6±0.669.6 \pm0.669.1±0.769.1 \pm0.7
StatA [47]67.0±0.567.0 \pm0.568.9±0.568.9 \pm0.569.5±0.669.5 \pm0.669.1±0.769.1 \pm0.7
MOON [18]67.4±0.567.4 \pm0.569.4±0.569.4 \pm0.569.9±0.569.9 \pm0.569.4±0.669.4 \pm0.6
GRAFT (transductive)67.6±0.567.6 \pm0.570.1±0.470.1 \pm0.470.8±0.570.8 \pm0.570.2±0.670.2 \pm0.6
GRAFT (inductive)67.2±0.567.2 \pm0.569.5±0.469.5 \pm0.470.3±0.570.3 \pm0.569.8±0.669.8 \pm0.6

Table 2. Online adaptation: top-1 accuracy (%) over 100 runs and eleven datasets, batches of 128, CLIP ViT-B/16, for stream correlation γ∈{0.1,0.01,0.001}\gamma\in\{0.1, 0.01, 0.001\} and the separate-class stream.

Per-dataset accuracy in the low setting

Figure 5. Per-dataset accuracy in the low setting for the zero-shot model, StatA and GRAFT.

Per-dataset results and distribution shifts

Table Table 3 reports every baseline on each dataset in the low setting and Fig. 5 plots the zero-shot model, StatA and GRAFT. GRAFT is first on eight of the eleven datasets and second on SUN397, EuroSAT and Flowers102, where EM-Dirichlet, strong at B=64B=64 but collapsing with large batches, leads by 1.41.4, 4.34.3 and 0.40.4 points. Against StatA, GRAFT improves on every dataset. The margin is largest on the fine-grained vocabularies with many confusable classes, Stanford Cars (2.82.8), FGVC-Aircraft (2.62.6) and SUN397 (1.91.9), and smallest on the coarse ones, Caltech101 (0.60.6), EuroSAT (0.70.7) and UFC101 (0.80.8). Table 4 extends the comparison to the ImageNet distribution shifts with the 1000-class vocabulary. The ranking is unchanged under the four shifts; StatA improves over zero-shot by 4.14.1 to 5.45.4 points, and the gain of GRAFT over StatA grows from 1.31.3 points on ImageNet to 2.02.0 to 2.52.5 points on the shifted sets, where the displacement from the text anchors, and with it the shared component that the graph propagates, is largest.

MethodINSUNAirc.Euro.CarsFoodPetsFlow.Calt.DTDUCFAvg.
CLIP zero-shot [31]66.762.624.847.865.386.189.171.393.344.466.765.3
MTA [45]68.064.227.049.566.486.889.672.693.747.067.966.6
EM-Dirichlet [27]70.269.335.157.370.388.091.076.194.150.671.470.3
ZLaP [40]35.430.713.328.440.348.447.738.244.926.633.335.2
TransCLIP [46]43.235.816.832.644.450.848.845.962.524.239.540.4
StatA [47]71.066.035.052.369.687.490.774.294.050.971.169.3
MOON [18]71.766.936.352.671.087.991.275.094.351.671.570.0
GRAFT72.367.937.653.072.488.591.675.794.652.371.970.7

Table 3. Per-dataset top-1 accuracy (%) in the low setting (B=64B=64, Keff∈[2,10]K_{\mathrm{eff}} \in[2, 10]), CLIP ViT-B/16.

MethodINV2SAR
CLIP zero-shot66.760.846.247.874.0
TransCLIP43.235.529.128.942.8
StatA71.066.251.151.979.4
MOON71.766.851.752.679.7
GRAFT72.368.353.154.381.9

Table 4. ImageNet (IN) and its shifts V2, Sketch (S), A and R in the low setting, top-1 accuracy (%), CLIP ViT-B/16.

Posterior mass assigned to absent classes

Figure 6. Posterior mass assigned to classes absent from the batch versus the number of effective classes (ImageNet, B=64B=64).

Qualitative results

Figure 7 shows test images from batches of the low set- @page 7/10 starts_mid=1 ends_mid=1

Figure 7. Qualitative predictions in the low setting: ground truth (GT) and the predictions of the zero-shot model (ZS), StatA and GRAFT. Correct predictions in green, wrong ones in red.

MethodVery low B=64B=64Low B=64B=64Medium B=64B=64Medium B=1000B=1000High B=1000B=1000Very high B=1000B=1000All full
CLIP zero-shot [31]65.2±0.665.2 \pm0.665.2±0.565.2 \pm0.565.2±0.465.2 \pm0.465.2±0.265.2 \pm0.265.2±0.265.2 \pm0.265.2±0.165.2 \pm0.165.265.2
MTA [45]66.6±0.666.6 \pm0.666.6±0.566.6 \pm0.566.6±0.466.6 \pm0.466.6±0.266.6 \pm0.266.6±0.266.6 \pm0.266.6±0.166.6 \pm0.166.666.6
EM-Dirichlet [27]68.5±0.768.5 \pm0.770.3±0.670.3 \pm0.667.5±0.567.5 \pm0.564.4±0.464.4 \pm0.445.3±0.645.3 \pm0.633.6±0.733.6 \pm0.729.529.5
ZLaP [40]27.5±1.127.5 \pm1.135.2±0.935.2 \pm0.944.7±0.844.7 \pm0.841.5±0.541.5 \pm0.552.2±0.452.2 \pm0.458.4±0.358.4 \pm0.366.466.4
TransCLIP [46]38.9±1.038.9 \pm1.040.4±0.940.4 \pm0.942.7±0.842.7 \pm0.856.5±0.456.5 \pm0.462.0±0.362.0 \pm0.364.4±0.264.4 \pm0.270.3
StatA [47]70.4±0.470.4 \pm0.469.3±0.369.3 \pm0.367.4±0.367.4 \pm0.369.7±0.269.7 \pm0.269.8±0.269.8 \pm0.269.0±0.269.0 \pm0.269.969.9
MOON [18]71.3±0.471.3 \pm0.470.0±0.370.0 \pm0.367.9±0.367.9 \pm0.370.1±0.270.1 \pm0.270.1±0.270.1 \pm0.269.3±0.269.3 \pm0.270.170.1
Unanchored mixture (α=λ=0\alpha=\lambda=0, no π\pi)42.4±0.642.4 \pm0.643.3±0.543.3 \pm0.545.4±0.545.4 \pm0.557.7±0.357.7 \pm0.362.8±0.362.8 \pm0.364.5±0.264.5 \pm0.270.370.3
Presence only (π\pi, no field)67.4±0.567.4 \pm0.566.7±0.466.7 \pm0.465.3±0.465.3 \pm0.466.9±0.266.9 \pm0.266.6±0.266.6 \pm0.265.7±0.265.7 \pm0.266.066.0
Anchor-only shrinkage (λ=0\lambda=0)70.1±0.470.1 \pm0.469.1±0.369.1 \pm0.367.3±0.367.3 \pm0.369.6±0.269.6 \pm0.269.7±0.269.7 \pm0.269.0±0.269.0 \pm0.269.969.9
Complete pooling (global shift)69.6±0.469.6 \pm0.468.8±0.368.8 \pm0.366.8±0.366.8 \pm0.368.8±0.268.8 \pm0.268.6±0.268.6 \pm0.267.6±0.267.6 \pm0.268.368.3
GRAFT (fixed α\alpha, λ\lambda)72.0±0.472.0 \pm0.470.7±0.370.7 \pm0.368.4±0.368.4 \pm0.370.5±0.270.5 \pm0.270.4±0.270.4 \pm0.269.5±0.269.5 \pm0.270.270.2
GRAFT + empirical Bayes72.1±0.472.1 \pm0.470.8±0.370.8 \pm0.368.5±0.368.5 \pm0.370.6±0.270.6 \pm0.270.5±0.270.5 \pm0.269.5±0.269.5 \pm0.270.270.2

Table 1 (PDF p. 7). Batch adaptation with a variable number of effective classes: top-1 accuracy (%) averaged over 1000 tasks and eleven datasets, CLIP ViT-B/16.

MethodLowMediumHighSeparate
CLIP zero-shot [31]65.2±0.365.2 \pm0.365.2±0.365.2 \pm0.365.2±0.365.2 \pm0.365.2±0.365.2 \pm0.3
TENT [42]65.8±0.465.8 \pm0.465.5±0.565.5 \pm0.565.3±0.665.3 \pm0.664.5±0.864.5 \pm0.8
TDA [21]67.7±0.467.7 \pm0.467.1±0.567.1 \pm0.566.8±0.666.8 \pm0.666.6±0.766.6 \pm0.7
DMN [51]67.2±0.467.2 \pm0.466.5±0.566.5 \pm0.566.3±0.666.3 \pm0.665.8±0.765.8 \pm0.7
OGA [10]67.5±0.567.5 \pm0.569.1±0.569.1 \pm0.569.6±0.669.6 \pm0.669.1±0.769.1 \pm0.7
StatA [47]67.0±0.567.0 \pm0.568.9±0.568.9 \pm0.569.5±0.669.5 \pm0.669.1±0.769.1 \pm0.7
MOON [18]67.4±0.567.4 \pm0.569.4±0.569.4 \pm0.569.9±0.569.9 \pm0.569.4±0.669.4 \pm0.6
GRAFT (transductive)67.6±0.567.6 \pm0.570.1±0.470.1 \pm0.470.8±0.570.8 \pm0.570.2±0.670.2 \pm0.6
GRAFT (inductive)67.2±0.567.2 \pm0.569.5±0.469.5 \pm0.470.3±0.570.3 \pm0.569.8±0.669.8 \pm0.6

Table 2 (PDF p. 7). Online adaptation: top-1 accuracy (%) over 100 runs and eleven datasets, batches of 128, CLIP ViT-B/16, for stream correlation γ∈{0.1,0.01,0.001}\gamma\in\{0.1, 0.01, 0.001\} and the separate-class stream.

Per-dataset accuracy in the low setting

Figure 5 (PDF p. 7). Per-dataset accuracy in the low setting for the zero-shot model, StatA and GRAFT.

and 67.267.2 at γ=0.1\gamma=0.1) but gain little on strongly correlated ones (66.866.8 and 66.366.3 at γ=0.001\gamma=0.001), whereas the anchored OGA, StatA and MOON improve with correlation (69.669.6, 69.569.5 and 69.969.9 at γ=0.001\gamma=0.001), since correlated batches contain few classes with many samples each. GRAFT follows the trend of the anchored methods and adds 1.21.2, 1.31.3 and 1.11.1 points over StatA at medium and high correlation and on the separate stream, and 0.60.6 at low correlation (0.20.2 to 0.90.9 over MOON). The inductive variant, which predicts each batch before updating the statistics, retains most of the margin (69.569.5, 70.370.3 and 69.869.8 against 70.170.1, 70.870.8 and 70.270.2).

Per-dataset results and distribution shifts

Table Table 3 reports every baseline on each dataset in the low setting and Fig. 5 plots the zero-shot model, StatA and GRAFT. GRAFT is first on eight of the eleven datasets and second on SUN397, EuroSAT and Flowers102, where EM-Dirichlet, strong at B=64B=64 but collapsing with large batches, leads by 1.41.4, 4.34.3 and 0.40.4 points. Against StatA, GRAFT improves on every dataset. The margin is largest on the fine-grained vocabularies with many confusable classes, Stanford Cars (2.82.8), FGVC-Aircraft (2.62.6) and SUN397 (1.91.9), and smallest on the coarse ones, Caltech101 (0.60.6), EuroSAT (0.70.7) and UFC101 (0.80.8). Table 4 extends the comparison to the ImageNet distribution shifts with the 1000-class vocabulary. The ranking is unchanged under the four shifts; StatA improves over zero-shot by 4.14.1 to 5.45.4 points, and the gain of GRAFT over StatA grows from 1.31.3 points on ImageNet to 2.02.0 to 2.52.5 points on the shifted sets, where the displacement from the text anchors, and with it the shared component that the graph propagates, is largest.

Posterior mass assigned to absent classes

Figure 6 (PDF p. 7). Posterior mass assigned to classes absent from the batch versus the number of effective classes (ImageNet, B=64B=64).

Qualitative results

Figure 7 shows test images from batches of the low setshot model, StatA and GRAFT. The typical correction is a sample of a present class that the zero-shot model assigns to a semantically close class absent from the batch (an Egyptian cat for a tabby cat). StatA recovers some of these samples when the present class has enough evidence to move its prototype, and GRAFT recovers the remaining ones because the absent neighbour is removed through the presence model and displaced consistently with the present class. The failure case, a stone wall predicted as a cliff by all three methods, has no close neighbour in the batch. Figure 6 quantifies the mechanism on ImageNet batches of 64: with a single present class the zero-shot model assigns 31.0% of its posterior mass to classes that are not in the batch, StatA 11.5% and GRAFT 4.1%. With fifty present classes the values converge to 8.1%, 3.0% and 1.2%.

Ablation Studies

Components. Table 5 adds the components one at a time. Without anchor, coupling or presence the mixture reaches

π\piα\alphaλ\lambdaEBWsW_sLow (B=64B=64)All
zero-shot65.265.2
−−−−✓43.3±0.570.3
✓−−−✓55.8±0.570.3
−✓−−✓69.1±0.369.9
✓✓−−✓69.6±0.369.9
−✓✓−✓70.2±0.370.2
✓✓✓−−70.4±0.368.7
✓✓✓−✓70.7±0.370.2
✓✓✓✓✓70.8±0.370.2

Table 5. Component ablation, top-1 accuracy (%): π\pi presence, α\alpha anchor, λ\lambda graph coupling, EB empirical Bayes, WsW_s sample graph.

43.3 in the low setting, 22 points below the zero-shot model, but 70.3 on the full set, where every class is well observed. The presence model alone raises the low-setting result to 55.8, the anchor term alone to 69.1, and the two together to 69.6, the anchor-only regime of StatA and MOON. The graph coupling adds 1.1 points on top (70.7), the margin that distinguishes GRAFT from the anchored methods, and empirical Bayes a further 0.1. The inherited sample graph contributes 0.3 points in the low setting but 1.5 on the full set, where the anchor and the coupling change the result by at most 0.4 points.

Prior and graph. We vary α\alpha, λ\lambda and the number of neighbours mm, and replace the text kNN graph by a complete graph and by a degree-matched random graph (table and sensitivity curves in the supplementary material). Accuracy in the low setting varies by at most 0.6 points for λ\lambda between 1 and 16 and by 0.3 points for mm between 3 and 10, so neither is a sensitive hyper-parameter; removing the anchor (α=0\alpha=0) costs 1.7 points and removing the coupling (λ=0\lambda=0) 1.6. The random graph (69.2) performs like no coupling (69.1) and the complete graph (68.8) is below both, so pooling helps only when it follows the semantic neighbourhoods, the control that H1 of § 4.2 requires. H1 itself is tested with labels (supplementary material): the energy ratio of the oracle displacement field on the text graph against degree-matched random graphs lies between 0.25 and 0.75 on all datasets, and the datasets with the highest ratios have the smallest margins in Table 3. Likelihood and optimisation variants change the result by at most 0.5 points.

Conclusion

We traced the collapse of unlabeled VLM adaptation on realistic batches to independent per-class parameters and replaced them by a displacement field over the vocabulary with a Gaussian Markov random field prior on a semantic class graph. GRAFT is closed form, contains anchor-only shrinkage and a global shift as extreme settings, and improves over anchored transduction on realistic batches.

LLM Usage Statement

Generative AI tools were used for language editing of the manuscript and to assist with the literature search. They were not used to generate experimental results, numerical values, or the statements and proofs of the theoretical results, all of which were derived by the authors. The authors take full responsibility for the final content of this work, including all text, claims and artifacts.

References

  1. [1]Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. In ECCV, 2014. 6DOI
  2. [2]Behzad Bozorgtabar, Dwarikanath Mahapatra, Sudipta Roy, Muzammal Naseer, Imran Razzak, and Zongyuan Ge. LATA: Laplacian-assisted transductive adaptation for conformal uncertainty in medical VLMs. In CVPR, 2026. 2arxiv.org/abs/2602.17535
  3. [3]Ziliang Chen, Yulu Li, Liangda Fang, Jusheng Zhang, Yongsen Zheng, Quanlong Guan, and Xipeng Chen. Vocabulary scaling law: Tuning open-vocabulary predictors for their openness. In CVPR, 2026. 2
  4. [4]Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023. 6DOI
  5. [5]Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014. 6arxiv.org/abs/1311.3618
  6. [6]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 5
  7. [7]Florian Dörfler and Francesco Bullo. Kron reduction of graphs with applications to electrical networks. IEEE Transactions on Circuits and Systems I, 60(1):150–163, 2013. 5arxiv.org/abs/1102.2950
  8. [8]Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, and Elisa Ricci. Frustratingly easy test-time adaptation of vision-language models. In NeurIPS, 2024. 2arxiv.org/abs/2405.18330
  9. [9]Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories. In CVPR Workshops, 2004. 6
  10. [10]Clément Fuchs, Maxime Zanella, and Christophe De Vleeschouwer. Online gaussian test-time adapta- tion of vision-language models. In CVPR Workshops, 2025. 1, 2, 6, 7arxiv.org/abs/2501.04352
  11. [11]Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannan Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. DataComp: In search of the next generation of multimodal datasets. In NeurIPS, 2023. 6arxiv.org/abs/2304.14108
  12. [12]Saurabh Garg, Yifan Wu, Sivaram Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. In NeurIPS, 2020. 2arxiv.org/abs/2003.07554
  13. [13]Pengxiao Han, Changkun Ye, Yanshuo Wang, Jinguang Tong, Miaohua Zhang, Xuesong Li, Jie Hong, and Lars Petersson. Label shift aware adaptation for online zero-shot learning with contrastive language-image pre-training (CLIP). arXiv preprint arXiv:2606.15169, 2026. 2arxiv.org/abs/2606.15169
  14. [14]Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 6
  15. [15]Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021. 6
  16. [16]Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021. 6
  17. [17]Fanding Huang, Jingyan Jiang, Qinting Jiang, Hebei Li, Faisal Nadeem Khan, and Zhi Wang. COSMIC: Clique-oriented semantic multi-space integration for robust CLIP test-time adaptation. In CVPR, 2025. 2arxiv.org/abs/2503.23388
  18. [18]Jiazhen Huang, Zhiming Liu, Changhu Wang, Wei Ju, Ziyue Qiao, and Xiao Luo. Von mises-fisher mixture model with dynamic shrinkage for realistic test-time transduction. In ICML, 2026. 1, 2, 4, 6, 7, 8arxiv.org/abs/2607.15851
  19. [19]David R Hunter and Kenneth Lange. A tutorial on MM algorithms. The American Statistician, 58(1):30–37, 2004. 5DOI
  20. [20]Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P. Xing. Rethinking knowledge graph propagation for zero-shot learning. In CVPR, 2019. 2arxiv.org/abs/1805.11724
  21. [21]Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In CVPR, 2024. 1, 2, 6, 7arxiv.org/abs/2403.18293
  22. [22]Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In ICCV Workshops, 2013. 6DOI
  23. [23]Yushu Li, Yongyi Su, Adam Goodge, Kui Jia, and Xun Xu. Efficient and context-aware label propagation for zero/few- shot training-free adaptation of vision-language model. In ICLR, 2025. 1, 2
  24. [24]Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, 2022. 1, 2arxiv.org/abs/2203.02053
  25. [25]Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Prototype propagation networks (PPN) for weakly-supervised few-shot learning on category graph. In IJCAI, 2019. 2arxiv.org/abs/1905.04042
  26. [26]Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013. 6arxiv.org/abs/1306.5151
  27. [27]Ségolène Martin, Yunshi Huang, Fereshteh Shakeri, Jean-Christophe Pesquet, and Ismail Ben Ayed. Transductive zero-shot and few-shot CLIP. In CVPR, 2024. 1, 2, 6, 7, 8arxiv.org/abs/2405.18437
  28. [28]Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008. 6DOI
  29. [29]Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C V Jawahar. Cats and dogs. In CVPR, 2012. 6
  30. [30]Qi Qian and Juhua Hu. Online zero-shot classification with CLIP. In ECCV, 2024. 2arxiv.org/abs/2408.13320
  31. [31]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 6, 7, 8arxiv.org/abs/2103.00020
  32. [32]Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In ICML, 2019. 6arxiv.org/abs/1902.10811
  33. [33]Håvard Rue and Leonhard Held. Gaussian Markov Random Fields: Theory and Applications. Chapman and Hall/CRC, 2005. 3
  34. [34]Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. Neural Computation, 14(1):21–41, 2002. 2, 5
  35. [35]Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language models. In ICLR, 2025. 1, 2arxiv.org/abs/2404.07983
  36. [36]Lijun Sheng, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. The illusion of progress? a critical look at test-time adaptation for vision-language models. In NeurIPS, 2025. 1, 2, 6arxiv.org/abs/2506.24000
  37. [37]Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS, 2022. 2
  38. [38]Julio Silva-Rodríguez, Ismail Ben Ayed, and Jose Dolz. Conformal prediction for zero-shot models. In CVPR, 2025. 2
  39. [39]Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes fromarxiv.org/abs/1212.0402
  40. [40]Vladan Stojnić, Yannis Kalantidis, and Giorgos Tolias. Label propagation for zero-shot classification with vision-language models. In CVPR, 2024. 1, 2, 6, 7, 8arxiv.org/abs/2404.04072
  41. [41]Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. Just shift it: Test-time prototype shifting for zero-shot generalization with vision-language models. In WACV, 2025. 2arxiv.org/abs/2403.12952
  42. [42]Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021. 6, 7arxiv.org/abs/2006.10726
  43. [43]Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019. 6
  44. [44]Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010. 6DOI
  45. [45]Maxime Zanella and Ismail Ben Ayed. On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning? In CVPR, 2024. 2, 6, 7, 8arxiv.org/abs/2405.02266
  46. [46]Maxime Zanella, Benoît Gérin, and Ismail Ben Ayed. Boosting vision-language models with transduction. In NeurIPS, 2024. 1, 2, 3, 5, 6, 7, 8arxiv.org/abs/2406.01837
  47. [47]Maxime Zanella, Clément Fuchs, Christophe De Vleeschouwer, and Ismail Ben Ayed. Realistic test-time adaptation of vision-language models. In CVPR, 2025. 1, 2, 3, 5, 6, 7, 8arxiv.org/abs/2501.03729
  48. [48]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 1, 6arxiv.org/abs/2303.15343
  49. [49]Ce Zhang, Simon Stepputis, Katia Sycara, and Yaqi Xie. Dual prototype evolving for test-time generalization of vision-language models. In NeurIPS, 2024. 1, 2arxiv.org/abs/2410.12790
  50. [50]Taolin Zhang, Jinpeng Wang, Hanqu Guo, Tao Dai, Bin Chen, and Shu-Tao Xia. BoostAdapter: Improving vision-language test-time adaptation via regional bootstrapping. In NeurIPS, 2024. 2arxiv.org/abs/2410.15430
  51. [51]Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision-language models. In CVPR, 2024. 1, 2, 6, 7arxiv.org/abs/2403.17589
  52. [52]Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NeurIPS, 2003. 2
  53. [53]Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022. 5DOI
  54. [54]Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. In NeurIPS, 2024. 2arxiv.org/abs/2410.19294
  55. [55]Imtiaz Ziko, Dounia Zou, Eric Granger, and Ismail Ben Ayed. Laplacian regularized few-shot learning. In ICML, 2020. 2, 5arxiv.org/abs/2006.15486

Paper details

vlnc:W_j4d22f7c
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents