betaFeedback

WorksComputer Science

VIBRA: Backward-Compatible Cross-Modal Retrieval with Consensus Barycenters

Abstract

Backward-compatible retrieval enables a new encoder to query the gallery that an old model has already encoded, so that a cross-modal retrieval system can be upgraded without re-encoding billions of items. Existing compatible methods either fit the old query and stay near the accuracy of the old model, or run the old and the new encoder for every query. We propose VIBRA, which matches relevance on the old gallery, corrects the errors of a single model with the barycenter of several views, which is itself a query in the old space, and trains the new encoder to inherit this consensus, so that models absent at deployment still shape every query. We show that the barycenter gains margin over the average of its views exactly when their hardest irrelevant items differ, and we characterize the share of this gain that one encoder can reach. On five benchmarks with five upgrades among six CLIP, MetaCLIP and SigLIP2 backbones, VIBRA is the best compatible method in all 50 settings, improving Recall@1 by up to 5.6 points over the strongest baseline and by up to 18.0 points over the old model, while adding less than 1 ms to the latency of the new encoder.

Introduction

Backward compatibility allows a cross-modal retrieval system to adopt a new encoder while its billions of gallery items keep the embeddings of the old model. Vision and language models such as CLIP [15, 26] encode every gallery item once and answer each query by a nearest neighbor search over the stored vectors [16]. These models improve quickly through new data, architectures and training objectives [39, 43], yet two independently trained models produce embeddings that cannot be compared with each other [36]. Without compatibility, adopting a new model requires re-encoding the whole gallery, known as backfilling (Figure 1(a)), which costs hundreds of GPU hours for

Figure 1. Four ways to upgrade the query encoder. Only VIBRA (d) keeps the gallery and one encoder per query while improving accuracy.

billions of items and is impossible once the raw items are gone [30]. In cross-modal search the old gallery even holds the other modality, so a new image encoder must be compared with captions that only the old text encoder has seen.

Existing compatible methods avoid backfilling and can be grouped into three categories (Figure 1), which train the new model for compatibility, map its queries to the old query, or run both encoders for every query. Backward-compatible training, introduced by BCT [36] and extended by later work [27, 51], trains the new model with a compatibility loss toward the old one, and query side maps [20, 29, 41] keep a pretrained new model frozen and translate its queries into the old space (Figure 1(b)).

XBT [14] tunes the new encoder through a projection pretrained on text. These methods fit the old query of each input, so they inherit the errors of the old model and their accuracy stays close to it. SLERP [30] interpolates between the aligned new query and the old query and gains accuracy, but it runs the old encoder next to the new one (Figure 1(c)), which raises the latency of every query.

Our key insight is that both limitations stem from the training target. A target defined by the old query reproduces at best the ranking of the old model, including its errors, whereas a target defined by relevance, the gallery items that a query should retrieve, can exceed it. Several models correct one another when they make errors on different items, so one encoder can learn from a teacher that combines the new, the old and an auxiliary encoder.

To address both limitations, we propose VIBRA, which matches relevance on the old gallery, corrects the errors of a single model with the barycenter of several views, itself a query in the old space, and lets the new encoder inherit this consensus, so that models absent at deployment still shape every query (Figure 2). Relevance matching derives the ideal query from the optimality condition of a listwise retrieval loss and estimates relevance by mixing the paired caption with the ranking of the new model. Consensus correction pools the queries of the new, the old and an auxiliary encoder into a view barycenter, with weights that a noise model of the views fixes before training. Consensus inheritance trains a low rank update of the new encoder, steered toward the auxiliary view, to reproduce the barycenter and merges it into the encoder, so that deployment runs one encoder.

Figure 2. Overview of VIBRA. (a) Training workflow. § 3.2 fits the stage 1 query q0q_0 with the relevance loss LRC\mathcal{L}_{\mathrm{RC}}; § 3.3 pools the views qvq_v offline into the teacher TT with weights cvc_v; § 3.4 trains a low rank update ΔW\Delta W so that qθq_\theta matches TT, reaching the share ΠLG\Pi_{\mathcal{L}G} of the gain GG. (b) Retrieval example, in which an image query and a text query each run one encoder and retrieve the correct item from the old gallery.

Our analysis shows that the margin of the barycenter exceeds the average margin of its views exactly when their hardest irrelevant items differ, and it gives the share of this gain that one encoder can reach. Pooling therefore corrects errors that the views make on different items, no weighting corrects errors that they share, and one encoder recovers the part of this gain that lies in the directions it can reach.

In experiments on five benchmarks and five upgrades among six CLIP, MetaCLIP and SigLIP2 backbones, VIBRA ranks first among compatible methods in all 50 settings, with Recall@1 gains of up to 5.6 points over the strongest baseline and up to 18.0 points over the old model, at a latency within 1 ms of the new encoder. On average it raises Recall@1 by 6.5 points over the old model and by 2.3 points over SLERP, which runs two encoders.

Our contributions are as follows.

  • We propose VIBRA, a single encoder approach to cross-modal compatibility that learns a barycenter of several encoders derived from the retrieval objective, while leaving the old gallery unchanged (§ 3).

  • We develop a theory of the compatible target, which is defined by relevance on the old gallery (§ 3.2), gains a dispersion margin over its views exactly when their hardest irrelevant items differ (Proposition 1), and is inherited by one encoder up to a projection of this gain (§ 3.4).

  • We evaluate VIBRA on five benchmarks and five upgrades, where it is the best compatible method in all 50 settings, improves Recall@1 by up to 5.6 points over the strongest baseline and by up to 18.0 points over the old model, and keeps the latency of a single encoder (§ 4).

Related Work

Backward-compatible training. Backward-compatible training makes a new model comparable with features that an old model has already computed [8, 36]. BCT [36] trains the new embedding model with an influence loss through the classifier of the old model, so that its queries can be compared directly with old gallery features. Later methods improve the balance between compatibility and the accuracy of the new model through class center alignment [21], adversarial learning [23], basis transformation [51], stationary classifiers [3, 4] or orthogonal transformations [28], while others lower the cost of backfilling itself [13, 27, 34, 48, 50]. These methods address image retrieval with class labels and usually train the new model from scratch, whereas we fine-tune an existing pretrained vision and language model across modalities without re-encoding any gallery item.

Query side maps. Query side maps keep the new model frozen and translate its queries into the old space. Drift-Adapter [41] fits Procrustes, low rank affine or residual MLP adapters on paired embeddings, λ\lambda-orth [29] learns a transformation with relaxed orthogonality, and latent space translation [12, 20, 22] estimates the map from semantically aligned anchors; related adapters address continual text retrievers [9] and embeddings of multimodal large language models [5]. All of them fit the old query of the same input, so their objective contains no mechanism that corrects the errors of the old model (§ 3.2), and they can only use information that the output of the new encoder already contains (§ 3.4).

Cross-modal compatibility. Cross-modal compatibility extends backward compatibility to vision and language models, whose gallery holds the other modality. XBT [14] introduced it with a projection module pretrained on text and a low rank update of the new encoder. TaCA [49] inserts a compatible adapter into an upgraded visual encoder, and SLERP [30] interpolates on the sphere between an aligned new query and the old query, which requires both encoders at query time. VIBRA keeps the deployment constraint of XBT, one query encoder and an untouched gallery, and also adapts the new encoder with a low rank update, but derives what the update learns from the retrieval objective and from several views.

Soft targets and distillation. Soft targets and distillation transfer knowledge from other models into the model being trained. ALBEF [18] mixes one hot labels with the similarity distribution of a momentum copy of the model to soften noisy image and text pairs; our stage 1 target has the same form, but its soft ranking comes from another model, the new one, it is matched on the old gallery, and the mixture follows from the optimality condition of the retrieval objective (§ 3.2). Beyond matching the outputs or features of a teacher [10, 31], relational and ranking distillation [25, 37] transfer the structure among items, and ensembles and several teachers [17, 45] are combined with weights that are usually set by hand; for vision and language models, distillation has mainly served compression [42, 44]. In VIBRA, the teacher lives in the old space and the student must stay compatible with a frozen gallery; the teacher is a barycenter of views, the form that a single query can reproduce, and its weights follow from the noise of the views, as in the combination of forecasts [2].

Method

Overview

We upgrade the query encoder of a retrieval system whose gallery must stay as it is. The gallery G={gy}G = \{g_y\} holds unit vectors gy=ϕold(y)∈Rdg_y = \phi_{\mathrm{old}}(y) \in\mathbb{R}^{d} computed by the old model ϕold\phi_{\mathrm{old}} for every item yy. A new model ψ\psi gives features z=ψ(x)∈Rdz = \psi(x) \in\mathbb{R}^{d} of a query xx. Both ϕold\phi_{\mathrm{old}} and ψ\psi denote the image or text tower that matches their input. We build a query map q(x)q(x) on the unit sphere Sd−1\mathbb{S}^{d-1} that runs one network, initialized from ψ\psi, and ranks the gallery by the score

sq(x,y)=⟨q(x),gy⟩s_q(x,y) = \langle q(x), g_y\rangle; we write unit⁡(v)=v/∥v∥\operatorname{unit}(v) = v/\lVert v\rVert. Training may run any encoder on a set D\mathcal{D} of image and caption pairs, whereas deployment runs only qq. We describe image queries and text queries in § 3.4.

VIBRA has three components, one per subsection (Figure 2). Relevance matching replaces the imitation of the old query (§ 3.2), a barycenter of views corrects the errors of a single model (§ 3.3), and the new encoder is trained on that barycenter so that one encoder suffices (§ 3.4).

Relevance Matching with Ranking Consistency

The stage 1 query matches relevance on the old gallery through the loss of Eq. (1) (Figure 2(a), left). Query side maps imitate the old query, and imitation cannot exceed the old model. Procrustes alignment, Drift-Adapter, LST and λ\lambda-orth fit the new query to the old query ϕold(x)\phi_{\mathrm{old}}(x) of the same input. A perfect fit gives q(x)=ϕold(x)q(x) = \phi_{\mathrm{old}}(x) and therefore exactly the ranking and the recall of the old model on its own gallery (Old →\rightarrow old): every item the old model ranks wrongly with confidence stays wrong. An imperfect fit can land somewhat above or below the old model, but the objective itself contains no mechanism that moves the query toward relevant items the old model misses. To exceed the old model, the target must be defined by the relevance of the gallery items.

The retrieval objective defines such a target through the relevance of the gallery items. Recall at any rank is piecewise constant in qq, so we work with its standard listwise surrogate [6]. Let p∗(⋅∣x)p^{*}(\cdot\mid x) be the relevance distribution of query xx, which spreads unit mass over the gallery items relevant to xx, and let pq(y∣x)=exp⁡(sq(x,y)/τ)∑y′exp⁡(sq(x,y′)/τ)p_q(y \mid x) = \frac{\exp(s_q(x,y)/\tau)}{\sum_{y'}\exp(s_q(x,y')/\tau)} be the distribution that a query induces over the gallery, with temperature τ\tau. The cross entropy L(q;x)=−∑yp∗(y∣x)log⁡pq(y∣x)\mathcal{L}(q;x)=-\sum_y p^{*}(y\mid x)\log p_q(y\mid x) is convex in qq, and its gradient 1τ(Ey∼pq[gy]−Ey∼p∗[gy])\frac{1}{\tau}\left(\mathbb{E}_{y\sim p_q}[g_y]-\mathbb{E}_{y\sim p^{*}}[g_y]\right) vanishes exactly at a query q⋆q^{\star} with

Ey∼pq⋆(⋅∣x)[gy]=Ey∼p∗(⋅∣x)[gy],\mathbb{E}_{y\sim p_{q^{\star}}(\cdot\mid x)}[g_y]=\mathbb{E}_{y\sim p^{*}(\cdot\mid x)}[g_y],

that is, when the barycenter of the gallery weighted by the query coincides with the barycenter of the relevant items; for unit queries the condition holds for the component tangent to the sphere. The optimal query is thus determined by relevance measured on the old gallery, and we adopt this condition as the target. During training the gallery is a pool of KK candidate captions per image encoded by the old model, and every distribution below is taken over this pool.

We estimate the unobserved relevance p∗p^{*} by mixing two estimates that a training pair offers, whose errors differ. The label δyx\delta_{y_x}, which puts all mass on the caption yxy_x paired with xx, is correct about that caption but treats every other relevant caption as irrelevant. The ranking of the new model, pnew(y∣x)=softmax⁡y(⟨ψ(x),ψ(y)⟩/τ)p_{\mathrm{new}}(y\mid x)=\operatorname{softmax}_y(\langle\psi(x),\psi(y)\rangle/\tau) over the same items, recognizes such captions but carries the errors of the new model. We therefore use the mixture pˉλ=(1−λ)δyx+λpnew\bar{p}_{\lambda}=(1-\lambda)\delta_{y_x}+\lambda p_{\mathrm{new}} with a weight λ∈[0,1]\lambda\in[0,1]. Because the cross entropy is linear in the target distribution, the loss equals, up to a constant independent of qq,

LRC(q;x)=(1−λ)CE⁡(δyx,pq)+λKL⁡(pnew∥pq),(1)\mathcal{L}_{\mathrm{RC}}(q;x)=(1-\lambda)\operatorname{CE}(\delta_{y_x},p_q)+\lambda\operatorname{KL}(p_{\mathrm{new}}\mathbin{\|}p_q), \tag*{(1)}

the contrastive loss [40] on the label plus a ranking consistency term toward the new model, both measured on the old gallery.

We fit this loss with a linear head on the frozen new features and anchor the head with the Procrustes map [32]. We minimize (1) with a linear head f(z)=unit⁡(Wz+b)f(z)=\operatorname{unit}(Wz+b) with weights WW and bias bb on the frozen new features and use λ=12\lambda=\frac{1}{2}, the equal weighting of the two terms. At a stationary point the residual of the optimality condition for pˉλ\bar{p}_{\lambda}, projected onto the tangent space of the sphere and weighted by 1/∥Wz+b∥1/\lVert Wz+b\rVert, is uncorrelated with every coordinate of zz; the head therefore matches the estimated relevance on average, in the least squares sense over linear functions of zz. The head is fit on finite candidate pools and can distort the geometry of the new space, whereas the orthogonal Procrustes map, the matrix P∈Rd′×dP\in\mathbb{R}^{d'\times d} with orthonormal rows or columns that best maps the new features of the training images onto their old image embeddings, only rotates and projects that space. We therefore average the head with PP, which serves as an anchor,

q0(z)=unit⁡(βf(z)+(1−β)unit⁡(PTz))q_0(z)=\operatorname{unit}\left(\beta f(z)+(1-\beta)\operatorname{unit}(P^{\mathsf{T}}z)\right)

with a fixed weight β\beta. The stage 1 query q0q_0 uses only the new encoder, and so it keeps the errors of the new model, which the next section addresses.

Consensus Correction with View Barycenters

The teacher is the barycenter of three views, Eq. (3), which corrects the errors that the views make on different items (Figure 2(a), middle). Views from other encoders can correct the errors of a single model. Any query estimated from one model ranks wrongly whatever that model ranks wrongly with confidence. Other encoders see the same image differently, and during training they are available. We call a view any unit query in the old space: the stage 1 head alone, qnew=f(z)q_{\mathrm{new}}=f(z), which we index as view 0, the old query qold=ϕold(x)q_{\mathrm{old}}=\phi_{\mathrm{old}}(x), and auxiliary encoders ψaux\psi_{\mathrm{aux}} mapped to the old space by ridge regression onto the old image embeddings, qaux=unit⁡(Rψaux(x)+e)q_{\mathrm{aux}}=\operatorname{unit}(R\psi_{\mathrm{aux}}(x)+e) with ridge weights RR and offset ee. Using views to remove the errors of one model raises two questions: which targets combine several views and can still be represented by a single query, and which errors of the views such a target can correct.

Every weighted fusion of view scores is reproduced by a single query, the view barycenter. A query interacts with the gallery only through its scores. Fusing the scores of the views with weights cvc_v in the simplex gives, by linearity, ∑vcvsv(x,y)=(∑vcvqv,gy)\sum_v c_v s_v(x,y)=(\sum_v c_vq_v,g_y) with sv=⟨qv,gy⟩s_v=\langle q_v,g_y\rangle, so every weighted fusion of view scores is reproduced exactly by a single query. For ∑vcvqv≠0\sum_v c_vq_v\ne0, its normalized form is the view barycenter, the point of the sphere with the smallest weighted squared chordal distance to the views,

Bc(x)=arg⁡min⁡u∈Sd−1∑vcv∥u−qv(x)∥2=unit⁡(∑vcvqv(x)).(2)B_c(x)=\underset{u\in\mathbb{S}^{d-1}}{\arg\min}\sum_v c_v\lVert u-q_v(x)\rVert^2=\operatorname{unit}\left(\sum_v c_vq_v(x)\right). \tag*{(2)}

Normalization rescales all scores of a query by the same factor, so the barycenter ranks the gallery exactly as the fused scores do. A target that combines several views and is representable by one query therefore takes the form of a view barycenter, one vector per training image. It remains to determine which errors the views it corrects. Whether the barycenter corrects an error depends on which irrelevant items each view ranks above y+y^{+} (Figure 3).

Consensus correction on the sphere

Figure 3. Consensus correction on the sphere. BcB_c projects the mean of the views (hollow point) onto the sphere and ranks y+y^+ first inside the green cone. (a) Views share their hardest irrelevant caption, so D=0D = 0 and BcB_c fails. (b) Their hardest irrelevant captions differ, so D>0D > 0 and BcB_c succeeds while every view fails.

Proposition 1 (Barycentric margin). Fix a query with a relevant item y+y^{+} and irrelevant items N\mathcal{N}. For each view let Δv(n)=sv(y+)−sv(n)\Delta_v(n)=s_v(y^{+})-s_v(n) for n∈Nn\in\mathcal{N}, let Δv,(k)\Delta_{v,(k)} be its kkth smallest value and mv=Δv,(1)m_v=\Delta_{v,(1)} the margin of the view, and let ΔB=∑vcvΔv\Delta_B=\sum_v c_v\Delta_v be the margin of the barycenter; up to a positive factor, with margin mB=ΔB,(1)m_B=\Delta_{B,(1)}. Then: (i) the margin of the barycenter decomposes as

mB=∑vcvmv+D,0≤D≤∑v≠0cvγv(n^),m_B=\sum_v c_vm_v+D,\qquad0\le D\le\sum_{v\ne0}c_v\gamma_v(\hat{n}),

where γv(n)=Δv(n)−mv≥0\gamma_v(n)=\Delta_v(n)-m_v\ge0 and n^\hat{n} is the hardest irrelevant item of view 0, and D>0D>0 if and only if the views

with cv>0c_v > 0 do not share their hardest irrelevant item; (ii) for all integers kv≥1k_v \ge1 with ∑v(kv−1)+1=k\sum_v (k_v - 1) + 1 = k, ΔB,(k)≥∑vcvΔv,(kv)\Delta_{B,(k)} \ge\sum_v c_v \Delta_{v,(k_v)}, so y+y^+ is among the top kk of the barycenter whenever the right side is positive.

Proof sketch. (i) For every irrelevant item nn, ΔB(n)=∑vcvΔv(n)≥∑vcvmv\Delta_B(n) = \sum_v c_v\Delta_v(n) \ge\sum_v c_vm_v, and taking the minimum over nn gives D≥0D \ge0. Evaluating ΔB\Delta_B at n^\hat{n}, where view 0 attains its margin, gives the upper bound. If n∗n^* attains mBm_B, then D=∑vcv(Δv(n∗)−mv)D = \sum_v c_v(\Delta_v(n^*) - m_v) is a sum of nonnegative terms, which vanishes exactly when n∗n^* is the hardest item of every view with cv>0c_v > 0. (ii) View vv has at most kv−1k_v - 1 items with Δv<Δv,(kv)\Delta_v < \Delta_{v,(k_v)}, so at most k−1k - 1 items lie below some threshold, and among the kk items with the smallest barycentric gaps one lies above all of them. The full proof is in Section S3 of the supplementary material. □

The proposition identifies the quantity that decides whether pooling corrects errors. Pooling never lowers the average margin of the views, and it adds the dispersion gain DD exactly when the views do not share their hardest irrelevant item. In that case the barycenter can rank y+y^+ first even when some or all views rank it lower, since a positive DD can outweigh negative view margins. Conversely, if every view ranks y+y^+ below one common irrelevant item ncn_c, then ΔB(nc)=∑vcvΔv(nc)<0\Delta_B(n_c) = \sum_v c_v\Delta_v(n_c) < 0 for all weights, so no barycenter of these views corrects the error. Part (ii) extends the same statement from the first rank to recall at rank kk. The proposition thus prescribes the views: they must differ in their hardest irrelevant items, which the three views of the new, the old and an auxiliary model trained on other data provide, and three views suffice, since the bound on DD does not grow with the number of views and the influence of a further view decays as 1/(V+1)1/(V + 1) with VV views.

A noise model of the views sets the weights of the barycenter and predicts which views help. Proposition 1 turns the choice of views into a criterion: a candidate view can correct the new view only on queries where its hardest irrelevant item differs from that of the new view, and since the proposition holds for any weights, the weights only need to reflect the reliability of each view. We model the score of view vv as sv(x,y)=avr(x,y)+bv(x)+εv(x,y)s_v(x,y) = a_v r(x,y) + b_v(x) + \varepsilon_v(x,y), where r∈{0,1}r \in\{0,1\} marks relevance, av>0a_v > 0 is the separation of the view, bv(x)b_v(x) is an offset shared by all items of a query, and εv\varepsilon_v is noise with zero mean. The rescaled noises of the views have the covariance matrix Σuv=Cov⁡(εu/au,εv/av)\Sigma_{uv} = \operatorname{Cov}(\varepsilon_u/a_u,\varepsilon_v/a_v) over the irrelevant items, assumed equal for all queries. The fused score ∑vcvsv\sum_v c_vs_v separates relevant from irrelevant items by ∑vcvav\sum_v c_va_v; writing wv∝cvavw_v \propto c_va_v for the weights normalized to this separation, the fusion with the smallest noise variance [2] has

w∗=Σ−111TΣ−11,cv∝wv∗/av,w^* = \frac{\Sigma^{-1}\mathbf{1}}{\mathbf{1}^{\mathsf{T}}\Sigma^{-1}\mathbf{1}}, \qquad c_v \propto w_v^*/a_v,

where 1\mathbf{1} is the vector of ones. We use these weights when all entries of w∗w^* are nonnegative, so that cc lies in the simplex, and otherwise the minimizer over nonnegative weights. For view 0 and one extra view whose rescaled noises have standard deviations σ0\sigma_0 and σa\sigma_a and correlation ρ\rho, the relative reduction of noise variance is

Γ=1−σa2(1−ρ2)σ02+σa2−2ρσ0σa.\Gamma= 1 - \frac{\sigma_a^2(1-\rho^2)}{\sigma_0^2+\sigma_a^2-2\rho\sigma_0\sigma_a}.

The gain depends on the correlation and the noise ratio together; with nonnegative weights an extra view lowers the noise of view 0 exactly when ρ<σ0/σa\rho< \sigma_0/\sigma_a. Both cc and Γ\Gamma are estimated from held out pairs before training, so the views and their weights are fixed in advance. We use three views, qnewq_{\mathrm{new}}, qoldq_{\mathrm{old}} and LAION CLIP ViT-H/14 [7, 33] as qauxq_{\mathrm{aux}}, and the teacher

T(x)=Bc(x)=unit⁡(cnewqnew+coldqold+cauxqaux).(3)T(x) = B_c(x) = \operatorname{unit}\left(c_{\mathrm{new}}q_{\mathrm{new}} + c_{\mathrm{old}}q_{\mathrm{old}} + c_{\mathrm{aux}}q_{\mathrm{aux}}\right). \tag*{(3)}

Inheriting the Consensus in a Single Encoder

We train the new encoder itself to produce the barycenter (Figure 2(a), right), since the teacher needs the old and the auxiliary encoders, which deployment forbids. A view that is a function of zz, such as the head ff or the Procrustes map, can be pooled at deployment at no cost; a view that needs another encoder cannot. Let ψθ\psi_\theta be the new encoder with a low rank update ΔW=BA\Delta W = BA [11] of the attention query and value projections of the image tower, whose factors form θ\theta, zθ=ψθ(x)z_\theta= \psi_\theta(x), and qθ=qθ(zθ)q_\theta= q_\theta(z_\theta) the stage 1 query function, with ff and PP frozen, applied to the adapted features. The student minimizes

Ex∼D[∥qθ(x)−T(x)∥2+μ(1−cos⁡(h(zθ),ψaux(x)))].(4)\mathbb{E}_{x\sim\mathcal{D}}\left[\lVert q_\theta(x)-T(x)\rVert^2+\mu\left(1-\cos\left(h(z_\theta),\psi_{\mathrm{aux}}(x)\right)\right)\right]. \tag*{(4)}

where μ\mu weights the auxiliary term and hh is a linear head trained jointly and discarded after training. The target, the trainable family and the auxiliary term of (4) follow from the analysis below.

Matching the barycenter as a vector keeps the ranking of the teacher in every gallery and regresses onto all views at once. If ∥qθ(x)−T(x)∥≤δ\lVert q_\theta(x)-T(x)\rVert\le\delta, every score against a unit gallery vector moves by at most δ\delta, so the student keeps the first ranked item of the teacher whenever the teacher margin exceeds 2δ2\delta, in every gallery, including galleries never seen in training. A loss on rankings constrains only the candidate pools seen in training. Matching the vector also matches all views at once: for every unit query qq, with qˉ=∑vcvqv\bar{q} = \sum_v c_v q_v,

∥q−T∥2=∑vcv∥q−qv∥2−κ(x)∥qˉ(x)∥\lVert q-T\rVert^2 = \frac{\sum_v c_v\lVert q-q_v\rVert^2-\kappa(x)}{\lVert\bar{q}(x)\rVert}

where κ=2(1−∥qˉ∥)\kappa=2(1-\lVert\bar{q}\rVert) depends only on the query, so the first term of (4) is a joint regression onto the views, weighted per query by 1/∥qˉ(x)∥1/\lVert\bar{q}(x)\rVert, which is largest when the views disagree, that is, where Proposition 1 locates the dispersion gain.

The first term of (4) is also a least squares problem over the student family, which shows what a single encoder can reach and why the encoder itself must be adapted. Writing the teacher as T=q0+GT=q_0+G, where GG is the gain of the barycenter over the stage 1 query from which the student starts, stacking the outputs of all training images into one vector, and taking the family to first order and without the unit norm constraint as q0+Lq_0+\mathcal{L}, with L\mathcal{L} the span of the Jacobian columns of the trainable parameters, the projection theorem gives the minimizer q0+ΠLGq_0+\Pi_{\mathcal{L}}G, with ΠL\Pi_{\mathcal{L}} the orthogonal projection onto L\mathcal{L}, which captures the fraction

η=∥ΠLG∥2/∥G∥2\eta=\lVert\Pi_{\mathcal{L}}G\rVert^2/\lVert G\rVert^2

of the gain, the share of GG that the family explains. A head on the frozen features zz spans only maps of zz, while the gain comes from views whose errors differ from those of the new model, which such maps express poorly. Adapting the encoder enlarges L\mathcal{L} in the directions that matter. The same result implies that recovery is partial whenever GG has a component outside L\mathcal{L}, and that the student can only learn a gain that the barycenter shows on the training images.

The second term of (4) steers the adaptation toward the auxiliary view. It asks the adapted features to predict the native embedding of the auxiliary encoder through the head hh, which is initialized by ridge regression from zz. It adds no module at deployment, since hh is discarded, and reaches the query only through the features, where it favors the directions of L\mathcal{L} that encode the auxiliary view, from which part of the gain GG comes.

Text queries need no student. Both available views are functions of the new text features tt: the stage 1 text query q0txt(t)q_0^{\mathrm{txt}}(t), built as in § 3.2 with old images as the gallery, and a map MM trained contrastively on captions of the training corpus, whose target is the old text embedding of the same caption, so that it needs captions only. The text query is their barycenter qtxt=unit⁡(αq0txt(t)+(1−α)unit⁡(M(t)))q^{\mathrm{txt}}=\operatorname{unit}(\alpha q_0^{\mathrm{txt}}(t)+(1-\alpha)\operatorname{unit}(M(t))) with a fixed weight α\alpha; since neither view needs another encoder, it is computed directly at deployment.

Training has three stages: fit the stage 1 heads with (1) and the text map MM; estimate the weights cc on held out pairs and compute the teacher (3) offline for every training image; train the auxiliary head and only the low rank factors of the encoder with (4), which keeps the upgrade cheap. At deployment an image query runs the new image encoder with the update merged into its weights, followed by q0q_0, and a text query runs the new text encoder followed by the text barycenter. The cost is one forward pass of the new encoder plus a small map, the same as any single encoder baseline.

Experiments

Setup. We evaluate five upgrades of a deployed encoder, always against the old gallery. We upgrade the old CLIP ViT-B/32 [26] to CLIP ViT-L/14, CLIP ViT-B/16, SigLIP2 SO400M/14 [39, 47] and MetaCLIP ViT-L/14 [43], and the old MetaCLIP ViT-B/32 to SigLIP2 SO400M/14. All training uses CC3M [35] only, with 100K images for the student and 2.35M captions for the text map. All experiments run on a single NVIDIA A100 GPU.

Benchmarks. We evaluate image to text and text to image retrieval by Recall@1 on Flickr30k [46] and COCO [19] with the large galleries of [14], NoCaps [1], CxC [24] and the English part of XM3600 [38].

Baselines. We compare with five compatible methods. Drift-Adapter [41], LST [20] and λ\lambda-orth [29] are query side maps, XBT [14] fine-tunes the new encoder, and SLERP [30] runs the old and the new encoder for every query.

Hyperparameters. All hyperparameters are fixed on held out pairs before any test evaluation. We set K=1024K=1024, λ=12\lambda=\frac{1}{2}, β=0.75\beta=0.75, α=0.75\alpha=0.75, τ=0.03\tau=0.03 and μ=1\mu=1, take the teacher weights c∝w∗/ac\propto w^*/a derived from the noise model on held out CC3M pairs, and train LoRA of rank 8 for 2 epochs at learning rate 10−410^{-4}.

Accuracy

VIBRA is the best compatible method in all 50 combinations of model pair, dataset and direction (Table 1 and Tables S1 and S2 of the supplementary material). It leads the strongest other compatible method by 0.1 to 3.6 points from image to text and by 0.3 to 5.6 points from text to image, and Old → old by 1.5 to 18.0 points. This includes SLERP, which runs two encoders per query, and XBT, which also adapts the new encoder.

MethodImage → textText → image
Flickr30kCOCONoCapsCxCXM3600Flickr30kCOCONoCapsCxCXM3600
CLIP ViT-B/32 → CLIP ViT-L/14
Old → old40.728.671.251.653.321.814.545.332.338.7
New → new48.734.473.458.054.128.218.747.838.242.5
Drift-Adapter [41]42.429.167.452.445.921.714.143.931.938.6
XBT [14]42.5{sup}†`30.7{sup}†`75.0{sup}†`50.8{sup}‡`50.8{sup}‡`22.4{sup}†`15.6{sup}†`48.0{sup}†`33.9{sup}‡`41.6{sup}‡`
SLERP [30]48.0{sup}†`33.4{sup}†`74.0{sup}†`56.055.623.3{sup}†`15.3{sup}†`46.6{sup}†`33.840.5
LST [20]40.428.667.352.445.920.913.742.930.937.1
λ-orth [29]35.625.764.049.547.221.213.844.431.440.3
VIBRA (ours)51.636.076.159.656.125.917.350.836.845.1
CLIP ViT-B/32 → SigLIP2 SO400M/14
Old → old40.728.671.251.653.321.814.545.332.338.7
New → new69.150.488.572.465.851.535.170.156.858.8
Drift-Adapter [41]50.634.374.258.252.924.215.746.934.739.9
XBT [14]38.5{sup}‡`27.9{sup}‡`71.6{sup}‡`51.8{sup}‡`54.2{sup}‡`23.2{sup}‡`16.2{sup}‡`49.7{sup}‡`35.7{sup}‡`43.4{sup}‡`
SLERP [30]51.0{sup}†`37.4{sup}†`78.6{sup}†`61.758.925.8{sup}†`17.2{sup}†`50.6{sup}†`36.543.2
LST [20]46.131.571.655.349.922.915.044.933.438.4
λ-orth [29]42.230.470.655.453.825.516.550.235.743.6
VIBRA (ours)54.038.579.562.260.830.320.356.241.248.7
MetaCLIP ViT-B/32 → SigLIP2 SO400M/14
Old → old43.333.077.956.759.228.019.355.940.149.9
New → new69.150.488.572.465.851.535.170.156.858.8
Drift-Adapter [41]58.540.981.064.558.826.818.253.338.645.9
XBT [14]49.0{sup}‡`34.4{sup}‡`79.8{sup}‡`59.0{sup}‡`57.8{sup}‡`22.7{sup}‡`17.0{sup}‡`53.1{sup}‡`37.6{sup}‡`45.9{sup}‡`
SLERP [30]59.742.584.266.362.429.720.557.941.850.9
LST [20]56.539.179.662.957.925.717.351.437.044.4
λ-orth [29]49.337.378.661.359.224.016.752.636.045.8
VIBRA (ours)61.343.384.466.565.530.021.058.942.551.7

Table 1. Recall@1 (%) against the old gallery. Gray: references. {sup}† published; {sup}‡ our reproduction of XBT. Bold: best compatible method.

VIBRA closes much of the gap to re-encoding the gallery. On the two CLIP upgrades it matches or exceeds New → new in 16 of 20 cells, including 7 of the 10 cells of CLIP ViT-B/32 → CLIP ViT-L/14 in Table 1, although New → new reencodes the whole gallery. On the three larger upgrades New → new stays ahead by 0.3 to 21.5 points; VIBRA recovers 45% to 95% of the image to text gain that re-encoding brings over Old → old, and 9% to 69% of the text to image gain, since text queries use no student. Pooled over all settings, VIBRA obtains 55.0% of the update gain of BCT [36], against 36.1% for SLERP (Table S9 of the supplementary material).

Latency

VIBRA lies above the frontier of the baselines on every dataset and in both directions (Figure 4). At equal or lower latency, VIBRA is ahead by 3.4 to 9.0 points from image to text and by 0.8 to 6.1 points from text to image.

Figure 4. Recall@1 against query latency at batch size 1. Each marker is one new backbone, and the dashed line is the frontier of the baselines. Points toward the upper left are better.

The latency of VIBRA is that of the new encoder. It stays within 0.5 ms of the new encoder for image queries, where the low rank update is merged, and is 0.4 to 0.9 ms above it for text queries, which add the text map, while SLERP adds the old encoder and costs 7.4 to 8.2 ms more per query than VIBRA. The gallery size matters little: NoCaps, whose galleries are a quarter of the size of those of COCO, is faster by about 0.5 ms. Table S3 of the supplementary material gives latency and throughput for all five pairs. Training VIBRA takes less than one A100 hour, 19 to 27 times less than XBT and as much as re about two million images (Table S8 of the supplementary material).

PairOld → oldNew → newDrift-AdapterXBTSLERPLSTλ-orthVIBRA
I→T: CLIP ViT-B/32 → CLIP ViT-L/149.0 / 860316.6 / 71116.6 / 71016.8 / 68825.2 / 65916.5 / 70916.5 / 70916.6 / 709
I→T: CLIP ViT-B/32 → CLIP ViT-B/169.0 / 86008.8 / 27079.0 / 27029.1 / 256017.6 / 20838.9 / 26938.9 / 26938.9 / 2693
I→T: CLIP ViT-B/32 → SigLIP2 SO400M/148.9 / 858819.0 / 54918.6 / 54718.1 / 59027.1 / 51718.8 / 54618.7 / 54618.9 / 546
I→T: CLIP ViT-B/32 → MetaCLIP ViT-L/148.8 / 860116.2 / 71216.2 / 71116.4 / 68724.4 / 66016.2 / 71016.3 / 71016.2 / 710
I→T: MetaCLIP ViT-B/32 → SigLIP2 SO400M/148.9 / 858719.1 / 54919.0 / 54718.3 / 59027.4 / 51719.0 / 54618.9 / 54618.9 / 546
T→I: CLIP ViT-B/32 → CLIP ViT-L/149.1 / 268989.2 / 190419.5 / 172269.6 / 1895418.4 / 112989.5 / 190189.4 / 1909910.0 / 18844
T→I: CLIP ViT-B/32 → CLIP ViT-B/168.9 / 271558.9 / 271469.3 / 261239.3 / 2609218.2 / 135509.1 / 268699.1 / 267829.7 / 25377
T→I: CLIP ViT-B/32 → SigLIP2 SO400M/149.0 / 2665716.7 / 251716.7 / 252316.7 / 252226.1 / 232116.4 / 252316.5 / 252417.0 / 2519
T→I: CLIP ViT-B/32 → MetaCLIP ViT-L/149.1 / 269289.2 / 190289.5 / 190669.4 / 1774818.4 / 113849.3 / 188669.3 / 190969.8 / 18842
T→I: MetaCLIP ViT-B/32 → SigLIP2 SO400M/149.0 / 2688616.5 / 251716.7 / 252316.6 / 252125.8 / 231816.3 / 252316.3 / 252416.9 / 2519

Table S3. Query latency on the Large COCO galleries: ms per query at batch size 1 / queries per second at batch size 256. One A100-SXM4-40GB, bf16, exact top 10 search over the old gallery (175,680 captions for image → text, 35,136 images for text → image); preprocessing and tokenization excluded.

Table S8. Training cost in A100 hours (bf16, preprocessing and tokenization excluded, one configuration and one seed), counting the encoding of the training data with every encoder a method needs and the optimization, and the cost of a full backfill with the new model.

MethodB/32 →\to L/14B/32 →\to B/16B/32 →\to SO400MB/32 →\to MetaCLIP L/14MetaCLIP B/32 →\to SO400M
XBT [14], paper scale10.034.1818.9410.0318.93
VIBRA (ours)0.440.220.710.440.71
Full backfill, per 1M images0.390.100.510.390.51
Full backfill, per 1B images388101506389506
Gallery where backfill costs as much as VIBRA1.1M2.2M1.4M1.1M1.4M

Consensus Analysis

The barycenter corrects errors where Proposition 1 locates the dispersion gain (Table 2). We split the test queries by the condition of the proposition, whether the three views share their hardest irrelevant item. The dispersion gain DD is positive on 88.4% to 93.0% of the queries, and on these queries the barycenter corrects 10.0% to 13.4% of the errors of the new view, 2.5 to 3 times as often as on the queries with D=0D = 0.

L/14B/16SO400MMC L/14MC→SO
Queries with D>0D > 0 (%)91.290.193.092.088.4
Errors corrected, D>0D > 0 (%)11.310.913.412.510.0
Errors corrected, D=0D = 0 (%)4.24.24.54.24.0
Net R@1R@1 gain (points)4.74.65.25.31.2

Table 2. Consensus correction split by the condition of Proposition 1 (image → text, one positive caption per image). Errors corrected: share of the errors of the new view that the barycenter corrects. Columns name the new model for CLIP ViT-B/32; MC→SO: MetaCLIP ViT-B/32 → SigLIP2 SO400M/14.

The condition of the proposition is exact on the test sets. On the queries whose views share their hardest irrelevant item the measured DD is zero up to numerical precision, and when all three views are wrong on such a query the barycenter corrects none of them, the case of Figure 3(a). Overall the barycenter raises Recall@1 over the new view on every pair, by 1.2 to 5.3 points. § 10 of the supplementary material gives further statistics.

Ablation

Each component of VIBRA improves on the previous one, and relevance matching contributes most (Table 3, mean over the five datasets). Starting from the Procrustes map, which imitates the old query, relevance matching raises Recall@1 by 2.4 to 10.8 points, 5.4 on average; training the new encoder on the consensus teacher adds 0.3 to 2.4 points, and the auxiliary term another 0.1 to 1.0 points.

L/14B/16SO400MMC L/14MC→SOMean
Procrustes map48.848.646.850.956.350.3
+ Relevance matching52.451.257.653.363.855.7
+ Consensus teacher54.852.658.655.264.157.1
VIBRA (all three)55.853.258.855.864.257.6

Table 3. Ablation of the components (image → text R@1, %); each row adds one component. Columns as in Table 2.

Every component helps on every model pair. Both estimates of relevance are needed, and the teacher of three views exceeds every single view and the teacher of two views (Table S4 and Table S7 of the supplementary material).

Table S4. Mixing weight λ\lambda of the stage 1 target (stage 1 query q0q_0; mean R@1 over the five sets, image →\to text / text →\to image, %). λ=0\lambda= 0 uses the label only and λ=1\lambda= 1 the ranking of the new model only; λ∗\lambda^* is the minimum variance weight estimated on COCO DEV.

Pairλ=0\lambda= 00.250.50.751λ∗\lambda^*
B/32 →\to L/1448.9 / 33.250.8 / 33.952.4 / 34.453.1 / 34.251.7 / 32.253.0 / 34.0 (0.82)
B/32 →\to B/1647.1 / 32.949.7 / 33.751.2 / 33.951.9 / 33.450.6 / 30.951.9 / 33.1 (0.81)
B/32 →\to SO400M56.0 / 37.256.9 / 37.857.6 / 38.557.8 / 39.055.9 / 38.857.5 / 39.1 (0.84)
B/32 →\to MetaCLIP L/1451.9 / 36.152.5 / 36.653.3 / 37.153.0 / 37.051.6 / 36.553.0 / 37.1 (0.83)
MetaCLIP B/32 →\to SO400M62.6 / 38.763.4 / 39.363.8 / 40.064.0 / 40.462.9 / 40.063.9 / 40.4 (0.84)

Table S4.

Table S7. Full ablation, extending Table 3 of the main paper. Contribution of each component (image →\to text R@1R@1, mean over the five datasets, %; students with seed 0). Stage 1: the Procrustes map alone (β=0\beta= 0), the head alone (β=1\beta= 1) and q0q_0. Single views and teachers are queries in the old space; the teacher of two views drops the auxiliary encoder. Students: trained on the teacher of two views without the auxiliary term, on the teacher of three views without the auxiliary term (μ=0\mu= 0), and VIBRA.

PairProcrustesHeadq0q_0OldH/142 views3 views2 views3 views, μ=0\mu= 0VIBRA
B/32 →\to L/1448.850.252.449.150.256.056.654.854.855.8
B/32 →\to B/1648.649.651.249.150.254.355.852.752.653.2
B/32 →\to SO400M46.852.957.649.150.258.859.758.858.658.8
B/32 →\to MetaCLIP L/1450.950.953.349.150.256.857.455.555.255.8
MetaCLIP B/32 →\to SO400M56.362.363.854.054.564.564.964.564.164.2

Sensitivity

The stage 1 query is insensitive to its hyperparameters around the deployed values. We vary one hyperparameter at a time around the deployed setting (Figure 5, mean over the five datasets). Temperatures from 0.01 to 0.03 stay within 1.3 points and only τ=0.1\tau= 0.1 loses 1.7 to 4.2, and over KK from 256 to 4096 Recall@1 varies by at most 1.5 points. Every β\beta between 0.5 and 0.75 is within 1.4 points of the best (Table S5 of the supplementary material), and the minimum variance weight λ∗\lambda^{*}, computed on COCO DEV without tuning, falls at 0.81 to 0.84 and stays within 0.3 points of the best λ\lambda (Table S4).

Figure 5. Sensitivity of image → text R@1 to one hyperparameter at a time.

Table S5. Weight β\beta of the stage 1 head in q0=unit⁡(βf(z)+(1−β)unit⁡(P⊤z))q_0 = \operatorname{unit}(\beta f(z) + (1 - \beta)\operatorname{unit}(P^\top z)) (mean R@1 over the five sets, image →\to text / text →\to image, %). β=0\beta= 0 is the Procrustes map alone, β=1\beta= 1 the head alone; the deployed value is β=0.75\beta= 0.75.

Pairβ=0\beta= 00.250.50.751
B/32 →\to L/1448.8 / 30.352.1 / 33.153.2 / 34.552.4 / 34.450.2 / 32.9
B/32 →\to B/1648.6 / 28.851.0 / 32.052.1 / 33.751.2 / 33.949.6 / 33.1
B/32 →\to SO400M46.8 / 32.855.3 / 36.759.0 / 38.657.6 / 38.552.9 / 36.8
B/32 →\to MetaCLIP L/1450.9 / 35.053.9 / 36.854.5 / 37.553.3 / 37.150.9 / 35.5
MetaCLIP B/32 →\to SO400M56.3 / 37.261.4 / 39.463.7 / 40.163.8 / 40.062.3 / 38.7

Table S5.

The student is insensitive to the auxiliary weight, the rank of the update and the amount of training data. Doubling μ\mu changes the student by at most 0.6 points. Ranks 4 and 16 are within 0.1 points of rank 8, so a larger update brings no gain, and students trained on 25K or 50K images with the same number of steps are within 0.2 points of the one trained on 100K.

Conclusion

VIBRA upgrades the query encoder of a cross-modal retrieval system without re-encoding the gallery and with one encoder per query. It derives its training target from the retrieval objective, pools the views of several models into a barycenter that a single query reproduces, and transfers this target into the new encoder. On five benchmarks with five upgrades, it is the best compatible method in all 50 settings, improving Recall@1 by up to 5.6 points over the strongest baseline and by up to 18.0 points over the old model, with less than 1 ms added to the latency of the new encoder.

LLM Usage Statement

In this work, we used a large language model assistant to implement the method, the baselines and the evaluation in code, to design and give feedback on experiments, including the preregistered sensitivity and ablation studies, to check derivations and assist in writing the proofs of the supplementary material, and to help interpret the results. We have not used it to generate synthetic data sets, and translation and qualitative data analysis are not applicable to this work. Additionally, we used it to suggest experimental parameters, to create and modify the figures, to search for and summarize related work, to draft and edit parts of the text.

We have reviewed all AI assisted work. The authors defined the problem and the method, inspected the code and ran every experiment, checked each reported number against the experiment outputs, verified the derivations and proofs by hand, and read the cited papers to confirm the statements made about them. We take responsibility for the final content of this work, including text, claims and artifacts produced with the aid of generative AI.

Supplementary Material

Section, equation and proposition numbers without the prefix S refer to the main paper.

Full Retrieval Results

Tables S1 and S2 extend the R@1 results of the main paper to R@5 and R@10 for every model pair, dataset and method.

Query Latency

Table S3 lists the latency and throughput of every method on the Large COCO galleries for all five model pairs.

Proofs

Throughout, gallery vectors and views are unit vectors, the weights cc lie in the simplex, qˉ=∑vcvqv\bar{q} = \sum_v c_vq_v, and N\mathcal{N} is a finite set with at least kk elements. For a function ξ\xi on N\mathcal{N}, ξ(k)\xi_{(k)} denotes its kkth smallest value. We use two elementary facts about order statistics: if ξ(n)≤ζ(n)\xi(n) \le\zeta(n) for all nn, then ξ(k)≤ζ(k)\xi_{(k)} \le\zeta_{(k)}; and for every subset S⊆NS \subseteq\mathcal{N} with ∣S∣=k|S| = k, ξ(k)≤max⁡n∈Sξ(n)\xi_{(k)} \le\max_{n \in S} \xi(n).

Relevance matching

Optimality condition. With ℓy=⟨q,gy⟩/τ\ell_y = \langle q, g_y\rangle/\tau and ∑yp∗(y∣x)=1\sum_y p^*(y \mid x) = 1, the cross entropy equals −∑yp∗(y∣x)ℓy+log⁡∑yeℓy-\sum_y p^*(y \mid x)\ell_y + \log\sum_y e^{\ell_y}. The first term is linear in qq and the second is a log sum exp of linear functions of qq, which is convex. Its gradient is 1τ∑yp∗(y∣x)gy\frac{1}{\tau}\sum_y p^*(y \mid x)g_y, which gives the stated gradient, and a convex differentiable function is minimized exactly where its gradient vanishes, which gives the condition of Section 3.2 of the main paper. Under the constraint ∥q∥=1\|q\| = 1, a stationary point satisfies (I−qq⊤)∇qL=0(I - qq^\top)\nabla_qL = 0, so the condition holds for the component tangent to the sphere.

Loss (1). The cross entropy is linear in its first argument, so CE⁡(pˉλ,pq)=(1−λ)CE⁡(δyx,pq)+λCE⁡(pnew,pq)\operatorname{CE}(\bar{p}_\lambda,p_q) = (1-\lambda)\operatorname{CE}(\delta_{y_x},p_q) + \lambda\operatorname{CE}(p_{\mathrm{new}},p_q), and CE⁡(pnew,pq)=KL⁡(pnew∥pq)+H(pnew)\operatorname{CE}(p_{\mathrm{new}},p_q) = \operatorname{KL}(p_{\mathrm{new}}\|p_q) + H(p_{\mathrm{new}}), where the entropy H(pnew)H(p_{\mathrm{new}}) does not depend on qq.

Stationarity of the linear head. Let u=Wz+bu = Wz + b and q=unit⁡(u)q = \operatorname{unit}(u), so that ∂q/∂u=J=(I−qq⊤)/∥u∥\partial q/\partial u = J = (I - qq^\top)/\|u\|. By the chain rule, ∇WExL=1τEx[J(Epq[g]−Epˉ[g])z⊤]\nabla_W\mathbb{E}_x\mathcal{L} = \frac{1}{\tau}\mathbb{E}_x\left[J\left(\mathbb{E}_{p_q}[g] - \mathbb{E}_{\bar{p}}[g]\right)z^\top\right]. At a stationary point this matrix vanishes: the residual of the optimality condition, projected onto the tangent space and rescaled by 1/∥u∥1/\|u\|, is uncorrelated with every coordinate of zz.

View barycenters

Closed form (2). For a unit vector uu, ∥u−qv∥2=2−2⟨u,qv⟩\|u-q_v\|^2 = 2 - 2\langle u,q_v\rangle, hence ∑vcv∥u−qv∥2=2−2⟨u,qˉ⟩\sum_v c_v\|u-q_v\|^2 = 2 - 2\langle u,\bar{q}\rangle. By the Cauchy Schwarz inequality, ⟨u,qˉ⟩≤∥qˉ∥\langle u,\bar{q}\rangle\le\|\bar{q}\| with equality only at u=unit⁡(qˉ)u = \operatorname{unit}(\bar{q}) when qˉ≠0\bar{q} \ne0. The score identity ⟨q,gˉy⟩=∑vcvsv(x,y)\langle q,\bar{g}_y\rangle= \sum_v c_vs_v(x,y) is linearity of the inner product, and dividing by ∥qˉ∥>0\|\bar{q}\| > 0 does not change the ranking.

Proof of Proposition 1. Since cv≥0c_v \ge0 and the scores of the barycenter are ⟨q,gˉ⟩/∥qˉ∥\langle q,\bar{g}\rangle/\|\bar{q}\|, its gaps are ΔB/∥qˉ∥\Delta_B/\|\bar{q}\|, which have the same signs and order as ΔB\Delta_B. The rank of y+y^+ is one plus the number of irrelevant items with ΔB(n)≤0\Delta_B(n) \le0, so y+y^+ is among the top kk exactly when ΔB,(k)>0\Delta_{B,(k)} > 0.

(i) For every nn, ΔB(n)=∑vcvΔv(n)≥∑vcvmv\Delta_B(n) = \sum_v c_v\Delta_v(n) \ge\sum_v c_vm_v, so mB≥∑vcvmvm_B \ge\sum_v c_vm_v and D≥0D \ge0. Evaluating at n^\hat{n}, where Δ0(n^)=m0\Delta_0(\hat{n}) = m_0, gives mB≤ΔB(n^)=∑vcv(mv+γv(n^))=∑vcvmv+∑v≠0cvγv(n^)m_B \le\Delta_B(\hat{n}) = \sum_v c_v(m_v + \gamma_v(\hat{n})) = \sum_v c_vm_v + \sum_{v\ne0}c_v\gamma_v(\hat{n}), which is the upper bound. Let n∗n^* attain mBm_B. Then D=∑vcv(Δv(n∗)−mv)D = \sum_v c_v(\Delta_v(n^*)-m_v) is a sum of nonnegative terms, so D=0D = 0 implies Δv(n∗)=mv\Delta_v(n^*) = m_v for every vv with cv>0c_v > 0, that is, n∗n^* is the hardest item of all these views. Conversely, if one item n0n_0 is the hardest for all of them, mB≤ΔB(n0)=∑vcvmvm_B \le\Delta_B(n_0) = \sum_v c_vm_v and therefore D=0D = 0.

(ii) Let Ev={n∈N:Δv(n)<Δv,(kv)}E_v = \{n \in\mathcal{N} : \Delta_v(n) < \Delta_{v,(k_v)}\}. By definition of the order statistic, ∣Ev∣≤kv−1|E_v| \le k_v - 1, so ∣⋃vEv∣≤∑v(kv−1)=k−1\left|\bigcup_v E_v\right| \le\sum_v(k_v - 1) = k - 1. Let SS be a set of kk items with the smallest values of ΔB\Delta_B. Since ∣S∣=k|S| = k, some n′∈Sn' \in S lies outside every EvE_v, so Δv(n′)≥Δv,(kv)\Delta_v(n') \ge\Delta_{v,(k_v)} for all vv and, as cv≥0c_v \ge0, ΔB(n′)≥∑vcvΔv,(kv)\Delta_B(n') \ge\sum_v c_v\Delta_{v,(k_v)}. Finally ΔB,(k)=max⁡n∈SΔB(n)≥ΔB(n′)\Delta_{B,(k)} = \max_{n\in S}\Delta_B(n) \ge\Delta_B(n').

Effect of an additional view. With VV views of equal weight, adding view jj gives ΔjV+1(n)=(VΔV(n)+Δj(n))/(V+1)\Delta^{V+1}_j(n) = (V\Delta^V(n) + \Delta_j(n))/(V+1). Since ΔV+1(n)≥(VΔV(n)+Δj,(1))/(V+1)\Delta^{V+1}(n) \ge(V\Delta^V(n) + \Delta_{j,(1)})/(V+1), the first fact on order statistics gives Δ(k)V+1−Δ(k)V≥(Δj,(1)−Δ(k)V)/(V+1)\Delta^{V+1}_{(k)} - \Delta^V_{(k)} \ge(\Delta_{j,(1)} - \Delta^V_{(k)})/(V+1). With SkS_k the kk items with the smallest ΔV\Delta^V, the second fact gives Δ(k)V+1≤max⁡n∈Sk(VΔV(n)+Δj(n))/(V+1)≤(VΔ(k)V+max⁡n∈SkΔj(n))/(V+1)\Delta^{V+1}_{(k)} \le\max_{n\in S_k}(V\Delta^V(n) + \Delta_j(n))/(V+1) \le(V\Delta^V_{(k)} + \max_{n\in S_k}\Delta_j(n))/(V+1). Both changes are of order 1/(V+1)1/(V+1).

Minimum variance weights. Under the noise model, ∑vwv(sv−bv)/av=r+∑vwvεv/av\sum_v w_v(s_v-b_v)/a_v = r + \sum_v w_v\varepsilon_v/a_v whenever ∑vwv=1\sum_v w_v = 1, with noise variance w⊤Σww^\top\Sigma w. For positive definite Σ\Sigma, minimizing w⊤Σww^\top\Sigma w subject to 1⊤w=1\mathbf{1}^\top w = 1 with a Lagrange multiplier gives Σw=μ1\Sigma w = \mu\mathbf{1}, hence w∗=Σ−11/(1⊤Σ−11)w^* = \Sigma^{-1}\mathbf{1}/(\mathbf{1}^\top\Sigma^{-1}\mathbf{1}) with variance V∗=1/(1⊤Σ−11)≤∑vvV^* = 1/(\mathbf{1}^\top\Sigma^{-1}\mathbf{1}) \le\sum_v v for every vv. The fused score equals ⟨∑v(wv/av)qv,gy⟩\left\langle\sum_v (w_v/a_v)q_v,g_y \right\rangle minus a term constant in yy, so the barycenter weights are cv∝wv∗/avc_v \propto w_v^*/a_v. For two views, Σ=(σδ2ρσδσaρσδσaσa2)\Sigma= \begin{pmatrix} \sigma_\delta^2 & \rho\sigma_\delta\sigma_a \\ \rho\sigma_\delta\sigma_a & \sigma_a^2 \end{pmatrix} and 1ΓΣ−11Γ=(σδ2+σa2−2ρσδσa)/(σδ2σa2(1−ρ2)){}^{1}\Gamma\Sigma^{-1}{}^{1}\Gamma= (\sigma_\delta^2+\sigma_a^2-2\rho\sigma_\delta\sigma_a)/(\sigma_\delta^2\sigma_a^2(1-\rho^2)), which gives Γ=1−V∗/σδ2\Gamma= 1 - V^*/\sigma_\delta^2.

MethodFlickr30k R@1Flickr30k R@5Flickr30k R@10COCO R@1COCO R@5COCO R@10NoCaps R@1NoCaps R@5NoCaps R@10CxC R@1CxC R@5CxC R@10XM3600 R@1XM3600 R@5XM3600 R@10
CLIP ViT-B/32 → CLIP ViT-L/14
Old → old40.764.773.728.650.459.971.291.996.351.677.485.553.379.186.5
New → new48.773.081.234.456.065.473.493.597.458.081.788.454.178.886.9
Drift-Adapter [41]42.466.876.029.150.159.367.490.195.452.476.884.845.972.681.8
XBT [14]42.5†66.0†74.9†30.7†52.6†62.3†75.0†93.3†97.3†50.8‡76.0‡85.0‡50.8‡77.2‡85.6‡
SLERP [30]48.0†71.8†80.3†33.4†55.4†64.8†74.0†93.3†96.9†56.080.688.155.681.388.2
LST [20]40.464.974.228.649.558.967.389.495.352.476.585.045.972.481.9
λ-orth [29]35.659.669.625.746.456.064.087.994.149.574.584.147.274.383.5
VIBRA (ours)51.675.783.736.058.667.976.194.497.959.683.090.156.181.989.4
CLIP ViT-B/32 → CLIP ViT-B/16
Old → old40.764.773.728.650.459.971.291.996.351.677.485.553.379.186.5
New → new44.468.677.430.952.462.071.891.996.854.178.486.955.180.187.6
Drift-Adapter [41]42.166.475.728.649.859.268.690.795.552.177.085.449.676.685.3
XBT [14]35.1‡58.2‡67.9‡24.4‡44.6‡54.2‡66.7‡89.9‡95.0‡47.2‡73.3‡82.6‡48.3‡75.9‡84.6‡
SLERP [30]45.269.878.531.453.462.673.693.197.055.479.587.254.781.087.8
LST [20]40.664.674.228.048.958.567.790.395.651.176.885.049.875.684.5
λ-orth [29]32.656.266.423.744.053.563.287.293.446.572.982.247.574.783.5
VIBRA (ours)47.071.980.233.055.164.573.992.796.957.081.188.154.881.789.5
CLIP ViT-B/32 → SigLIP2 SO400M/14
Old → old40.764.773.728.650.459.971.291.996.351.677.485.553.379.186.5
New → new69.188.493.150.472.680.188.598.599.672.490.795.365.887.593.0
Drift-Adapter [41]50.674.181.934.356.465.574.293.497.358.281.888.652.978.086.0
XBT [14]38.5‡62.0‡71.4‡27.9‡48.9‡59.0‡71.6‡92.8‡96.7‡51.8‡77.6‡86.0‡54.2‡80.1‡87.2‡
SLERP [30]51.0†75.3†83.5†37.4†60.4†69.3†78.6†95.3†98.3†61.784.291.058.983.990.5
LST [20]46.170.078.531.553.462.871.691.796.055.379.687.249.976.184.3
λ-orth [29]42.266.776.130.452.061.670.691.996.255.480.187.753.879.587.2
VIBRA (ours)54.077.785.238.561.470.479.595.598.262.284.591.460.884.891.4
CLIP ViT-B/32 → MetaCLIP ViT-L/14
Old → old40.764.773.728.650.459.971.291.996.351.677.485.553.379.186.5
New → new57.180.387.642.464.973.582.496.899.065.586.592.362.285.891.8
Drift-Adapter [41]42.666.675.928.749.859.368.690.695.651.676.484.847.072.781.4
XBT [14]36.5‡60.2‡69.9‡26.9‡47.4‡57.1‡70.3‡91.8‡96.4‡50.6‡77.0‡85.0‡53.1‡79.1‡86.6‡
SLERP [30]49.273.281.434.256.565.775.193.797.257.981.288.755.380.388.1
LST [20]43.267.476.329.751.260.668.490.295.452.877.786.147.474.182.7
λ-orth [29]37.261.871.727.148.357.865.889.494.951.576.385.348.576.584.8
VIBRA (ours)49.673.881.935.758.267.776.993.797.659.382.889.857.682.989.6
MetaCLIP ViT-B/32 → SigLIP2 SO400M/14
Old → old43.367.276.033.055.364.777.994.597.656.780.888.459.283.990.7
New → new69.188.493.150.472.680.188.598.599.672.490.795.365.887.593.0
Drift-Adapter [41]58.580.987.640.963.572.281.095.398.064.586.192.058.883.690.1
XBT [14]49.0‡72.1‡80.4‡34.4‡56.7‡65.8‡79.8‡95.5‡98.4‡59.0‡82.6‡89.4‡57.8‡81.8‡87.9‡
SLERP [30]59.781.287.842.565.073.784.297.399.066.387.493.262.486.592.1
LST [20]56.579.286.539.161.870.779.695.197.862.984.591.257.982.990.0
λ-orth [29]49.373.281.837.359.769.078.694.797.761.384.490.959.284.391.1
VIBRA (ours)61.383.189.343.365.974.284.497.599.266.587.892.965.588.693.7

Table S1. Image → text retrieval against the old gallery (R@1 / R@5 / R@10, %). The old gallery is never reencoded. Every method except SLERP encodes the query with one network; SLERP encodes the query with one network; SLERP runs the old and the new encoder. Gray rows are references: Old → old keeps the old model, New → new reencodes the whole gallery. †published number. ‡our reproduction of XBT at the scale of its paper, used where the paper reports no number. Bold marks the best compatible method.

MethodFlickr30k R@1Flickr30k R@5Flickr30k R@10COCO R@1COCO R@5COCO R@10NoCaps R@1NoCaps R@5NoCaps R@10CxC R@1CxC R@5CxC R@10XM3600 R@1XM3600 R@5XM3600 R@10
CLIP ViT-B/32 → CLIP ViT-L/14
Old → old21.841.651.114.530.238.945.375.084.632.358.870.038.766.576.5
New → new28.249.659.018.735.944.447.876.685.238.263.974.242.569.277.9
Drift-Adapter [41]21.741.651.114.129.938.543.974.383.831.958.369.538.666.676.4
XBT [14]22.4†42.7†52.4†15.6†32.3†41.3†48.0†79.0†88.2†33.9‡60.6‡71.8‡41.6‡70.7‡80.0‡
SLERP [30]23.3†43.8†53.3†15.3†31.6†40.3†46.6†76.5†85.5†33.860.471.540.568.978.6
LST [20]20.940.449.913.729.137.742.973.383.230.957.568.837.165.575.8
λ-orth [29]21.240.650.013.829.137.544.474.584.231.457.468.540.368.077.6
VIBRA (ours)25.947.256.917.334.944.050.880.288.636.863.874.445.173.381.3
CLIP ViT-B/32 → CLIP ViT-B/16
Old → old21.841.651.114.530.238.945.375.084.632.358.870.038.766.576.5
New → new24.945.454.916.533.041.646.876.185.034.661.172.042.268.577.0
Drift-Adapter [41]20.439.849.213.628.737.142.972.983.030.657.368.836.364.374.4
XBT [14]20.9‡40.7‡50.6‡14.3‡30.5‡39.4‡46.3‡77.7‡87.2‡32.9‡59.7‡71.0‡41.2‡70.1‡79.8‡
SLERP [30]22.342.451.814.630.439.245.675.384.732.459.370.338.666.276.7
LST [20]19.838.948.313.228.236.442.172.182.330.056.668.335.563.573.7
λ-orth [29]21.340.750.113.829.137.544.274.484.131.557.368.540.067.977.6
VIBRA (ours)25.246.556.116.834.243.349.679.588.036.463.374.043.972.280.8
CLIP ViT-B/32 → SigLIP2 SO400M/14
Old → old21.841.651.114.530.238.945.375.084.632.358.870.038.766.576.5
New → new51.573.280.435.156.565.070.191.095.156.880.287.358.882.689.2
Drift-Adapter [41]24.244.654.315.732.641.446.976.084.934.761.272.139.968.377.8
XBT [14]23.2‡44.2‡54.2‡16.2‡33.4‡42.6‡49.7‡80.9‡89.4‡35.7‡62.9‡73.7‡43.4‡72.6‡81.6‡
SLERP [30]25.8†47.2†56.8†17.2†34.8†44.0†50.6†79.7†87.6†36.563.574.443.271.480.4
LST [20]22.943.152.615.031.239.944.973.983.233.459.670.538.466.676.8
λ-orth [29]25.546.155.616.533.542.450.278.587.135.762.172.543.671.380.3
VIBRA (ours)30.352.862.420.339.348.656.284.091.141.268.077.948.776.684.8
CLIP ViT-B/32 → MetaCLIP ViT-L/14
Old → old21.841.651.114.530.238.945.375.084.632.358.870.038.766.576.5
New → new40.963.872.326.947.356.460.986.292.348.773.682.552.478.285.7
Drift-Adapter [41]24.545.555.015.933.042.148.778.386.935.062.073.242.471.480.6
XBT [14]21.9‡42.2‡52.2‡15.1‡31.7‡40.7‡47.4‡78.8‡88.1‡33.9‡61.0‡72.4‡42.8‡71.8‡81.3‡
SLERP [30]27.148.758.417.535.244.451.680.788.837.364.375.044.873.281.8
LST [20]23.644.153.615.532.341.247.677.386.234.461.372.542.270.579.9
λ-orth [29]23.844.453.915.431.940.748.377.886.734.560.971.442.471.580.4
VIBRA (ours)28.650.860.518.937.647.054.382.890.239.666.677.248.176.584.5
MetaCLIP ViT-B/32 → SigLIP2 SO400M/14
Old → old28.049.759.119.337.647.055.983.690.740.167.077.149.976.284.3
New → new51.573.280.435.156.565.070.191.095.156.880.287.358.882.689.2
Drift-Adapter [41]26.848.057.518.236.345.653.381.088.838.665.275.845.973.282.5
XBT [14]22.7‡43.3‡53.0‡17.0‡34.6‡43.8‡53.1‡82.8‡90.7‡37.6‡64.4‡75.4‡45.9‡74.5‡82.8‡
SLERP [30]29.751.761.020.539.548.857.984.891.541.868.478.550.977.785.3
LST [20]25.746.756.017.334.843.751.479.487.437.063.573.944.471.981.1
λ-orth [29]24.044.253.716.733.742.752.680.488.536.062.173.245.873.282.1
VIBRA (ours)30.052.161.521.040.249.658.985.492.042.568.879.051.778.485.9

Table S2. Text → image retrieval against the old gallery (R@1 / R@5 / R@10, %). The old gallery is never reencoded. Every method except SLERP encodes the query with one network; SLERP runs the old and the new encoder. Gray rows are references: Old → old keeps the old model, New → new reencodes the whole gallery. †published number. ‡our reproduction of XBT at the scale of its paper, used where the paper reports no number. Bold marks the best compatible method.

With weight t∈[0,1]t \in[0,1] on the extra view, the variance v(t)=(1−t)2σ02+t2σa2+2t(1−t)ρσ0σav(t) = (1-t)^2\sigma_0^2 + t^2\sigma_a^2 + 2t(1-t)\rho\sigma_0\sigma_a is a convex quadratic with v′(0)=2σ0(ρσa−σ0)v'(0) = 2\sigma_0(\rho\sigma_a - \sigma_0), so a small positive weight lowers the variance of view 0 exactly when

ρ<σ0/σa\rho< \sigma_0/\sigma_a.

Inheriting the consensus

Margin preservation. By the Cauchy Schwarz inequality, ∣⟨qθ−T,g⟩∣≤∥qθ−T∥ ∥g∥≤δ|\langle q_\theta-T,g\rangle| \le\|q_\theta-T\|\,\|g\| \le\delta for every unit gg. Let y+y^+ be the item that the teacher ranks first and mTm_T its margin. For every other item yy, sqθ(y+)−sqθ(y)≥sT(y+)−sT(y)−2δ≥mT−2δs_{q_\theta}(y^+) - s_{q_\theta}(y) \ge s_T(y^+) - s_T(y) - 2\delta\ge m_T - 2\delta, which is positive when δ<mT/2\delta< m_T/2.

Barycentric regression. For every vector uu, expanding the squares gives ∑vcv∥u−qv∥2=∥u−qˉ∥2+∑vcv∥qv−qˉ∥2\sum_v c_v\|u-q_v\|^2 = \|u-\bar q\|^2+\sum_v c_v\|q_v-\bar q\|^2, and with unit views ∑vcv∥qv−qˉ∥2=1−∥qˉ∥2\sum_v c_v\|q_v-\bar q\|^2 = 1-\|\bar q\|^2. For a unit qq, ∥q−T∥2=2−2⟨q,qˉ⟩/∥qˉ∥\|q-T\|^2 = 2-2\langle q,\bar q\rangle/\|\bar q\| and ∥q−qˉ∥2=1−2⟨q,qˉ⟩+∥qˉ∥2\|q-\bar q\|^2 = 1-2\langle q,\bar q\rangle+\|\bar q\|^2. Eliminating ⟨q,qˉ⟩\langle q,\bar q\rangle gives ∥q−T∥2=(∥q−qˉ∥2−(1−∥qˉ∥)2)/∥qˉ∥\|q-T\|^2 = \bigl(\|q-\bar q\|^2-(1-\|\bar q\|)^2\bigr)/\|\bar q\|, and substituting ∥q−qˉ∥2=∑vcv∥q−qv∥2−(1−∥qˉ∥2)\|q-\bar q\|^2 = \sum_v c_v\|q-q_v\|^2-(1-\|\bar q\|^2) gives the stated identity, since (1−∥qˉ∥)+(1−∥qˉ∥)2=2(1−∥qˉ∥)(1-\|\bar q\|)+(1-\|\bar q\|)^2=2(1-\|\bar q\|).

Projection onto the student family. Let H\mathcal{H} be the Hilbert space of square integrable maps x↦Rdx\mapsto\mathbb{R}^d under the training distribution, with ⟨u1,u2⟩=Ex⟨u1(x),u2(x)⟩\langle u_1,u_2\rangle=\mathbb{E}_x\langle u_1(x),u_2(x)\rangle. For q=q0+lq=q_0+l with ll in a closed subspace L\mathcal{L}, the first term of (4) equals ∥l−G∥2\|l-G\|^2. By the projection theorem, L\mathcal{L} contains a unique minimizer ΠLG\Pi_{\mathcal{L}}G, and G−ΠLGG-\Pi_{\mathcal{L}}G is orthogonal to L\mathcal{L}. By Pythagoras, ∥G∥2=∥ΠLG∥2+∥G−ΠLG∥2\|G\|^2=\|\Pi_{\mathcal{L}}G\|^2+\|G-\Pi_{\mathcal{L}}G\|^2, so η=∥ΠLG∥2/∥G∥2=1−∥G−ΠLG∥2/∥G∥2\eta=\|\Pi_{\mathcal{L}}G\|^2/\|G\|^2=1-\|G-\Pi_{\mathcal{L}}G\|^2/\|G\|^2, the coefficient of determination of the least squares fit of GG within L\mathcal{L}. This treats the family as affine and ignores the normalization of the query; for LoRA it applies to the first order expansion q0+J(θ−θ0)q_0+J(\theta-\theta_0).

Stage 1 Sweeps

Tables S4 and S5 give the full sweeps of the mixing weight λ\lambda of the stage 1 target and the weight β\beta of q0q_0, summarized in Section 4.4 of the main paper.

Empirical Analysis of the Theory

Proposition 1 makes two exact claims that can be checked query by query: the barycentric margin is never below the weighted average of the view margins, and it never exceeds the ceiling given by the spread of the other views at the hardest item of view 0. Both held for every test query of every pair. Table S6 quantifies the correction. Under this protocol the barycenter of the three views raises R@1 over the new view by 1.3 to 5.3 points and rescues 6.0% to 9.4% of the queries. Close to 60% of the rescued queries, 3.4% to 5.7% of all queries, have a nonpositive average view margin and are rescued only through the dispersion gain DD, and 1.0% to 1.7% of all queries are ranked correctly by the barycenter although every view ranks them wrongly. All three views share their hardest irrelevant item on 7.0% to 11.6% of the queries; when such a query is ranked wrongly by every view, no weighting rescues it. Splitting the queries by this condition confirms the proposition: on the queries with a shared hardest item the measured DD is zero up to numerical precision (at most 7×10−87\times10^{-8}) and none of those that every view ranks wrongly is rescued, whereas DD is positive on more than 99.9% of the remaining queries, where the barycenter corrects 10.0% to 13.4% of the errors of the new view, against 4.0% to 4.5% on the queries with a shared hardest item.

The noise model of Section 3.3 of the main paper is estimated on CC3M DEV only. In a preregistered test over 40 combinations of a model pair and a candidate view, its predicted gain Γ\Gamma ranks the views by their measured test gain (Spearman 0.38, one sided permutation p=0.007p=0.007; 0.61 on average within a pair), whereas the correlation pp alone does not (0.08). The same model overestimates the absolute size of the gains by a factor of three to four, because it treats the noise of different irrelevant items as independent while the errors of the views concentrate on a few shared hard items, the situation in which Proposition 1 predicts no dispersion gain. Training the student on the teacher with the derived weights c∝w∗/ac\propto w^*/a gives image → text R@1 on par with a fixed rule without estimation, the nested midpoint unit⁡(qnew+unit⁡(qold+qaux))\operatorname{unit}(q_{\mathrm{new}} + \operatorname{unit}(q_{\mathrm{old}} + q_{\mathrm{aux}})) with vector weights of roughly 12,14,14\frac{1}{2}, \frac{1}{4}, \frac{1}{4}: over the 25 combinations of pair and set, the mean difference is −0.05-0.05 points (mean over three seeds), positive in 12.

Full Ablation

The full ablation adds the head alone, the single views, and the teachers and students of two views to the ablation of the main paper (Table S7). Each part of the stage 1 query contributes: the Procrustes map alone is 2.5 to 10.9 points below q0q_0 and the head alone 1.5 to 4.7 points. The two estimates of relevance are both needed (Table S4): the label alone (λ=0\lambda= 0) is 1.2 to 4.1 points below λ=12\lambda= \frac{1}{2} and the ranking of the new model alone (λ=1\lambda= 1) 0.6 to 1.7 points below.

The barycenter of three views is the strongest target, and the auxiliary term carries the third view into the student. The teacher of three views improves on q0q_0 by 1.1 to 4.5 points and on the teacher of two views by 0.4 to 1.4 points, and every teacher exceeds each of its views. Without the auxiliary term, the student of three views (μ=0\mu= 0) is within 0.5 points of the student of two views, although its teacher is better, and adding it gives 0.1 to 1.0 points. On MetaCLIP B/32 →\to SO400M, where the teacher gains least, the student of two views is 0.3 points ahead of VIBRA.

Training Cost

VIBRA trains 19 to 27 times faster than XBT, the other method that trains the new encoder for compatibility across modalities (Table S8). VIBRA needs 0.22 to 0.71 A100 hours, spent on encoding its training data with the old, new and auxiliary encoders and on the student, while XBT at the scale of its paper needs 4.2 to 18.9 hours. Query side maps fit in minutes on cached features.

Training VIBRA costs as much as reencoding 1.1M to 2.2M images, and this cost does not grow with the gallery. A full backfill with the new model costs 101 to 506 A100 hours per billion images.

Update Gain

VIBRA obtains more than half of the gain of reencoding the gallery without reencoding it (Table S9). We report the update gain of BCT [36], the improvement of a method over Old →\to old divided by the improvement of New →\to new, on the mean R@1 over the five pairs and five datasets, so that 0% keeps the old model and 100% matches reencoding. VIBRA reaches 55.0%, with 68.4% from image to text and 43.3% from text to image.

Table S9. Update gain [36] (%), the share of the R@1R@1 gain of reencoding that a method obtains without reencoding, over all pairs and datasets.

MethodEncodersImage →\to textText →\to imageAll
Drift-Adapter [41]114.00.46.8
LST [20]14.2−7.5−2.0
λ\lambda-orth [29]1−17.63.0−6.7
XBT [14]1−7.06.20.0
SLERP [30]254.419.936.1
VIBRA (ours)168.443.355.0

Methods with one encoder obtain almost none of this gain, and SLERP obtains less with two encoders. Query side maps and XBT range from −6.7%-6.7\% to 6.8%, as expected from methods that fit the old query, and SLERP reaches 36.1% while it runs the old encoder for every query. The gap is largest from text to image, where VIBRA obtains 43.3% and SLERP 19.9%.

Table S6. Consensus correction on the test sets, extending Table 2 of the main paper (image →\to text, one designated positive caption per image, other positives removed from the negatives; mean over the five sets, %). Rescued: the barycenter ranks the positive first while the new view does not. Needing DD: rescued although the weighted average of the view margins is not positive. All views wrong: rescued although every view ranks the positive lower. Shared: all three views have the same hardest irrelevant item.

PairNew viewBarycenterCeilingRescuedNeeding DDAll views wrongShared
B/32 →\to L/1423.027.738.08.04.71.28.7
B/32 →\to B/1622.927.537.17.64.41.19.9
B/32 →\to SO400M24.329.544.39.45.71.77.0
B/32 →\to MetaCLIP L/1423.028.340.89.05.21.48.0
MetaCLIP B/32 →\to SO400M32.333.640.66.03.41.011.6

References

  1. [1]Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8947–8956. IEEE, 2019. 6DOI
  2. [2]John M Bates and Clive WJ Granger. The combination of forecasts. Journal of the operational research society, 20(4): 451–468, 1969. 3, 5
  3. [3]Niccolò Biondi, Federico Pernici, Matteo Bruni, and Alberto Del Bimbo. Cores: Compatible representations via stationarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):9567–9582, 2023. 2DOI
  4. [4]Niccolò Biondi, Federico Pernici, Simone Ricci, and Alberto Del Bimbo. Stationary representations: Optimally approximating compatibility and implications for improved model replacements. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28793–28804. IEEE, 2024. 2
  5. [5]Jaeseok Byun, Gukyeong Kwon, Han-Kai Hsu, Meher Gitika Karumuri, Zhikang Zhang, Hao Yang, and Davide Modolo. Multi-modal knowledge preserving adapter for embedding backward compatibility. In European Conference on Computer Vision, pages 473–490. Springer, 2026. 2arxiv.org/abs/2609.16875
  6. [6]Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th International Conference on Machine Learning, page 129–136, New York, NY, USA, 2007. Association for Computing Machinery. 4DOI
  7. [7]Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2829. IEEE, 2023. 5DOI
  8. [8]Rahul Duggal, Hao Zhou, Shuo Yang, Yuanjun Xiong, Wei Xia, Zhuowen Tu, and Stefano Soatto. Compatibility-aware heterogeneous visual search. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10718–10727, 2021. 2arxiv.org/abs/2105.06047
  9. [9]Dipam Goswami, Liying Wang, Bartlomiej Twardowski, and Joost van de Weijer. Query drift compensation: Enabling compatibility in continual learning of retrieval embedding models. arXiv preprint arXiv:2506.00037, 2025. 2arxiv.org/abs/2506.00037
  10. [10]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 3arxiv.org/abs/1503.02531
  11. [11]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 5arxiv.org/abs/2106.09685
  12. [12]Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis, 2024. 2arxiv.org/abs/2405.07987
  13. [13]Florian Jaeckle, Fartash Faghri, Ali Farhadi, Oncel Tuzel, and Hadi Pouransari. Fastfill: Efficient compatible model update. arXiv preprint arXiv:2303.04766, 2023. 2arxiv.org/abs/2303.04766
  14. [14]Young Kyun Jang and Ser-nam Lim. Towards cross-modal backward-compatible representation learning for vision-language models. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1783–1792. IEEE, 2025. 1, 2, 6, 7, 13, 14, 17arxiv.org/abs/2405.14715
  15. [15]Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR, 2021. 1arxiv.org/abs/2102.05918
  16. [16]Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE transactions on big data, 7(3):535–547, 2019. 1arxiv.org/abs/1702.08734
  17. [17]Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. Advances in neural information processing systems, 7, 1994. 3
  18. [18]Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. Advances in neural information processing systems, 34:9694–9705, 2021. 3
  19. [19]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 6
  20. [20]Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. Latent space translation via semantic alignment. Advances in Neural Information Processing Systems, 36:55394–55414, 2023. 1, 2, 6, 7, 13, 14, 17arxiv.org/abs/2311.00664
  21. [21]Qiang Meng, Chixiang Zhang, Xiaoqiang Xu, and Feng Zhou. Learning compatible embeddings. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9919–9928. IEEE, 2021. 2
  22. [22]Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430, 2022. 2arxiv.org/abs/2209.15430
  23. [23]Tan Pan, Furong Xu, Xudong Yang, Sifeng He, Chen Jiang, Qingpei Guo, Feng Qian, Xiaobo Zhang, Yuan Cheng, Lei Yang, et al. Boundary-aware backward-compatible representation via adversarial learning in image retrieval. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15201–15210. IEEE, 2023. 2
  24. [24]Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Waters, and Yinfei Yang. Crisscross captions: Extended intramodal and intermodal semantic similarity judgments for ms-coco. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2855–2870, 2021. 6
  25. [25]Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3962–3971. IEEE, 2019. 3arxiv.org/abs/1904.05068
  26. [26]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 1, 6
  27. [27]Vivek Ramanujan, Pavan Kumar Anasosalu Vasu, Ali Farhadi, Oncel Tuzel, and Hadi Pouransari. Forward compatible training for large-scale embedding retrieval systems. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19364–19373. IEEE, 2022. 1, 2arxiv.org/abs/2112.02805
  28. [28]Simone Ricci, Niccolò Biondi, Federico Pernici, and Alberto Del Bimbo. Backward-compatible aligned representations via an orthogonal transformation layer. In European Conference on Computer Vision, pages 451–464. Springer, 2024. 2
  29. [29]Simone Ricci, Niccolò Biondi, Federico Pernici, Ioannis Patras, and Alberto Del Bimbo. λ-orthogonality regularization for compatible representation learning. Advances in Neural Information Processing Systems, 38:29036–29063, 2025. 1, 2, 6, 7, 13, 14, 17DOI
  30. [30]Simone Ricci, Niccolò Biondi, and Federico Pernici. Spherical interpolation for backward-compatible multimodal representations. arXiv preprint arXiv:2609.39836, 2026. 1, 2, 6, 7, 13, 14, 17arxiv.org/abs/2609.39836
  31. [31]Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, 2015. 3DOI
  32. [32]Peter H. Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10, 1966. 4DOI
  33. [33]Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: an open large-scale dataset for training next generation image-text models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2022. Curran Associates Inc. 5
  34. [34]Seonguk Seo, Mustafa Gokhan Uzunbas, Bohyung Han, Sara Cao, Joena Zhang, Taipeng Tian, and Ser-Nam Lim. Online backfilling with no regret for large-scale image retrieval. arXiv preprint arXiv:2301.03767, 4, 2023. 2
  35. [35]Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018. 6DOI
  36. [36]Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward-compatible representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6367–6376. IEEE, 2020. 1, 2, 7, 16, 17arxiv.org/abs/2003.11942
  37. [37]Jiaxi Tang and Ke Wang. Ranking distillation: Learning compact ranking models with high performance for recommender system. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2289–2298, 2018. 3
  38. [38]Ashish V Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 715–729, 2022. 6
  39. [39]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdoulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. 1, 6arxiv.org/abs/2502.14786
  40. [40]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019. 4arxiv.org/abs/1807.03748
  41. [41]Harshil Vejendla. Drift-adapter: A practical approach to near zero-downtime embedding model upgrades in vector databases. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15949–15960, 2025. 1, 2, 6, 7, 13, 14, 17arxiv.org/abs/2509.23471
  42. [42]Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang, et al. Tyniclip: Clip distillation via affinity mimicking and weight inheritance. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 21913–21923. IEEE, 2023. 3arxiv.org/abs/2309.12314
  43. [43]Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In International Conference on Learning Representations, pages 47812–47831, 2024. 1, 6arxiv.org/abs/2309.16671
  44. [44]Chuangchuang Yang, Zhulin An, Libo Huang, Junyu Bi, Xinqiang Yu, Han Yang, Boyu Diao, and Yongjun Xu. Clip-kd: An empirical study of clip model distillation. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15952–15962. IEEE, 2024. 3arxiv.org/abs/2307.12732
  45. [45]Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1285–1294, 2017. 3DOI
  46. [46]Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2:67–78, 2014. 6
  47. [47]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941–11952, 2023. 6arxiv.org/abs/2303.15343
  48. [48]Binjie Zhang, Yixiao Ge, Yantao Shen, Yu Li, Chun Yuan, Xuyuan Xu, Yexin Wang, and Ying Shan. Hot-refresh model upgrades with regression-alleviating compatible training in image retrieval. arXiv preprint arXiv:2201.09724, 2022. 2arxiv.org/abs/2201.09724
  49. [49]Binjie Zhang, Yixiao Ge, Xuyuan Xu, Ying Shan, and Mike Zheng Shou. Taca: Upgrading your visual foundation model with task-agnostic compatible adapter. arXiv preprint arXiv:2306.12642, 2023. 2arxiv.org/abs/2306.12642
  50. [50]Binjie Zhang, Shupeng Su, Yixiao Ge, Xuyuan Xu, Yexin Wang, Chun Yuan, Mike Zheng Shou, and Ying Shan. Darwinian model upgrades: Model evolving with selective compatibility. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3393–3400, 2023. 2arxiv.org/abs/2210.06954
  51. [51]Yifei Zhou, Zilu Li, Abhinav Shrivastava, Hengshuang Zhao, Antonio Torralba, Taiping Tian, and Ser-Nam Lim. Bt^2: Backward-compatible training with basis transformation. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11195–11204. IEEE, 2023. 1, 2arxiv.org/abs/2211.03989

Paper details

vlnc:W_pbewtc4z
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

a large language model assistant

View statement
Looking for related papers…
Contents