WorksComputer Science
VIBRA: Backward-Compatible Cross-Modal Retrieval with Consensus Barycenters
Abstract
Backward-compatible retrieval enables a new encoder to query the gallery that an old model has already encoded, so that a cross-modal retrieval system can be upgraded without re-encoding billions of items. Existing compatible methods either fit the old query and stay near the accuracy of the old model, or run the old and the new encoder for every query. We propose VIBRA, which matches relevance on the old gallery, corrects the errors of a single model with the barycenter of several views, which is itself a query in the old space, and trains the new encoder to inherit this consensus, so that models absent at deployment still shape every query. We show that the barycenter gains margin over the average of its views exactly when their hardest irrelevant items differ, and we characterize the share of this gain that one encoder can reach. On five benchmarks with five upgrades among six CLIP, MetaCLIP and SigLIP2 backbones, VIBRA is the best compatible method in all 50 settings, improving Recall@1 by up to 5.6 points over the strongest baseline and by up to 18.0 points over the old model, while adding less than 1 ms to the latency of the new encoder.
Introduction
Backward compatibility allows a cross-modal retrieval system to adopt a new encoder while its billions of gallery items keep the embeddings of the old model. Vision and language models such as CLIP [15, 26] encode every gallery item once and answer each query by a nearest neighbor search over the stored vectors [16]. These models improve quickly through new data, architectures and training objectives [39, 43], yet two independently trained models produce embeddings that cannot be compared with each other [36]. Without compatibility, adopting a new model requires re-encoding the whole gallery, known as backfilling (Figure 1(a)), which costs hundreds of GPU hours for
Figure 1. Four ways to upgrade the query encoder. Only VIBRA (d) keeps the gallery and one encoder per query while improving accuracy.
billions of items and is impossible once the raw items are gone [30]. In cross-modal search the old gallery even holds the other modality, so a new image encoder must be compared with captions that only the old text encoder has seen.
Existing compatible methods avoid backfilling and can be grouped into three categories (Figure 1), which train the new model for compatibility, map its queries to the old query, or run both encoders for every query. Backward-compatible training, introduced by BCT [36] and extended by later work [27, 51], trains the new model with a compatibility loss toward the old one, and query side maps [20, 29, 41] keep a pretrained new model frozen and translate its queries into the old space (Figure 1(b)).
XBT [14] tunes the new encoder through a projection pretrained on text. These methods fit the old query of each input, so they inherit the errors of the old model and their accuracy stays close to it. SLERP [30] interpolates between the aligned new query and the old query and gains accuracy, but it runs the old encoder next to the new one (Figure 1(c)), which raises the latency of every query.
Our key insight is that both limitations stem from the training target. A target defined by the old query reproduces at best the ranking of the old model, including its errors, whereas a target defined by relevance, the gallery items that a query should retrieve, can exceed it. Several models correct one another when they make errors on different items, so one encoder can learn from a teacher that combines the new, the old and an auxiliary encoder.
To address both limitations, we propose VIBRA, which matches relevance on the old gallery, corrects the errors of a single model with the barycenter of several views, itself a query in the old space, and lets the new encoder inherit this consensus, so that models absent at deployment still shape every query (Figure 2). Relevance matching derives the ideal query from the optimality condition of a listwise retrieval loss and estimates relevance by mixing the paired caption with the ranking of the new model. Consensus correction pools the queries of the new, the old and an auxiliary encoder into a view barycenter, with weights that a noise model of the views fixes before training. Consensus inheritance trains a low rank update of the new encoder, steered toward the auxiliary view, to reproduce the barycenter and merges it into the encoder, so that deployment runs one encoder.
Figure 2. Overview of VIBRA. (a) Training workflow. § 3.2 fits the stage 1 query with the relevance loss ; § 3.3 pools the views offline into the teacher with weights ; § 3.4 trains a low rank update so that matches , reaching the share of the gain . (b) Retrieval example, in which an image query and a text query each run one encoder and retrieve the correct item from the old gallery.
Our analysis shows that the margin of the barycenter exceeds the average margin of its views exactly when their hardest irrelevant items differ, and it gives the share of this gain that one encoder can reach. Pooling therefore corrects errors that the views make on different items, no weighting corrects errors that they share, and one encoder recovers the part of this gain that lies in the directions it can reach.
In experiments on five benchmarks and five upgrades among six CLIP, MetaCLIP and SigLIP2 backbones, VIBRA ranks first among compatible methods in all 50 settings, with Recall@1 gains of up to 5.6 points over the strongest baseline and up to 18.0 points over the old model, at a latency within 1 ms of the new encoder. On average it raises Recall@1 by 6.5 points over the old model and by 2.3 points over SLERP, which runs two encoders.
Our contributions are as follows.
We propose VIBRA, a single encoder approach to cross-modal compatibility that learns a barycenter of several encoders derived from the retrieval objective, while leaving the old gallery unchanged (§ 3).
We develop a theory of the compatible target, which is defined by relevance on the old gallery (§ 3.2), gains a dispersion margin over its views exactly when their hardest irrelevant items differ (Proposition 1), and is inherited by one encoder up to a projection of this gain (§ 3.4).
We evaluate VIBRA on five benchmarks and five upgrades, where it is the best compatible method in all 50 settings, improves Recall@1 by up to 5.6 points over the strongest baseline and by up to 18.0 points over the old model, and keeps the latency of a single encoder (§ 4).
Related Work
Backward-compatible training. Backward-compatible training makes a new model comparable with features that an old model has already computed [8, 36]. BCT [36] trains the new embedding model with an influence loss through the classifier of the old model, so that its queries can be compared directly with old gallery features. Later methods improve the balance between compatibility and the accuracy of the new model through class center alignment [21], adversarial learning [23], basis transformation [51], stationary classifiers [3, 4] or orthogonal transformations [28], while others lower the cost of backfilling itself [13, 27, 34, 48, 50]. These methods address image retrieval with class labels and usually train the new model from scratch, whereas we fine-tune an existing pretrained vision and language model across modalities without re-encoding any gallery item.
Query side maps. Query side maps keep the new model frozen and translate its queries into the old space. Drift-Adapter [41] fits Procrustes, low rank affine or residual MLP adapters on paired embeddings, -orth [29] learns a transformation with relaxed orthogonality, and latent space translation [12, 20, 22] estimates the map from semantically aligned anchors; related adapters address continual text retrievers [9] and embeddings of multimodal large language models [5]. All of them fit the old query of the same input, so their objective contains no mechanism that corrects the errors of the old model (§ 3.2), and they can only use information that the output of the new encoder already contains (§ 3.4).
Cross-modal compatibility. Cross-modal compatibility extends backward compatibility to vision and language models, whose gallery holds the other modality. XBT [14] introduced it with a projection module pretrained on text and a low rank update of the new encoder. TaCA [49] inserts a compatible adapter into an upgraded visual encoder, and SLERP [30] interpolates on the sphere between an aligned new query and the old query, which requires both encoders at query time. VIBRA keeps the deployment constraint of XBT, one query encoder and an untouched gallery, and also adapts the new encoder with a low rank update, but derives what the update learns from the retrieval objective and from several views.
Soft targets and distillation. Soft targets and distillation transfer knowledge from other models into the model being trained. ALBEF [18] mixes one hot labels with the similarity distribution of a momentum copy of the model to soften noisy image and text pairs; our stage 1 target has the same form, but its soft ranking comes from another model, the new one, it is matched on the old gallery, and the mixture follows from the optimality condition of the retrieval objective (§ 3.2). Beyond matching the outputs or features of a teacher [10, 31], relational and ranking distillation [25, 37] transfer the structure among items, and ensembles and several teachers [17, 45] are combined with weights that are usually set by hand; for vision and language models, distillation has mainly served compression [42, 44]. In VIBRA, the teacher lives in the old space and the student must stay compatible with a frozen gallery; the teacher is a barycenter of views, the form that a single query can reproduce, and its weights follow from the noise of the views, as in the combination of forecasts [2].
Method
Overview
We upgrade the query encoder of a retrieval system whose gallery must stay as it is. The gallery holds unit vectors computed by the old model for every item . A new model gives features of a query . Both and denote the image or text tower that matches their input. We build a query map on the unit sphere that runs one network, initialized from , and ranks the gallery by the score
; we write . Training may run any encoder on a set of image and caption pairs, whereas deployment runs only . We describe image queries and text queries in § 3.4.
VIBRA has three components, one per subsection (Figure 2). Relevance matching replaces the imitation of the old query (§ 3.2), a barycenter of views corrects the errors of a single model (§ 3.3), and the new encoder is trained on that barycenter so that one encoder suffices (§ 3.4).
Relevance Matching with Ranking Consistency
The stage 1 query matches relevance on the old gallery through the loss of Eq. (1) (Figure 2(a), left). Query side maps imitate the old query, and imitation cannot exceed the old model. Procrustes alignment, Drift-Adapter, LST and -orth fit the new query to the old query of the same input. A perfect fit gives and therefore exactly the ranking and the recall of the old model on its own gallery (Old old): every item the old model ranks wrongly with confidence stays wrong. An imperfect fit can land somewhat above or below the old model, but the objective itself contains no mechanism that moves the query toward relevant items the old model misses. To exceed the old model, the target must be defined by the relevance of the gallery items.
The retrieval objective defines such a target through the relevance of the gallery items. Recall at any rank is piecewise constant in , so we work with its standard listwise surrogate [6]. Let be the relevance distribution of query , which spreads unit mass over the gallery items relevant to , and let be the distribution that a query induces over the gallery, with temperature . The cross entropy is convex in , and its gradient vanishes exactly at a query with
that is, when the barycenter of the gallery weighted by the query coincides with the barycenter of the relevant items; for unit queries the condition holds for the component tangent to the sphere. The optimal query is thus determined by relevance measured on the old gallery, and we adopt this condition as the target. During training the gallery is a pool of candidate captions per image encoded by the old model, and every distribution below is taken over this pool.
We estimate the unobserved relevance by mixing two estimates that a training pair offers, whose errors differ. The label , which puts all mass on the caption paired with , is correct about that caption but treats every other relevant caption as irrelevant. The ranking of the new model, over the same items, recognizes such captions but carries the errors of the new model. We therefore use the mixture with a weight . Because the cross entropy is linear in the target distribution, the loss equals, up to a constant independent of ,
the contrastive loss [40] on the label plus a ranking consistency term toward the new model, both measured on the old gallery.
We fit this loss with a linear head on the frozen new features and anchor the head with the Procrustes map [32]. We minimize (1) with a linear head with weights and bias on the frozen new features and use , the equal weighting of the two terms. At a stationary point the residual of the optimality condition for , projected onto the tangent space of the sphere and weighted by , is uncorrelated with every coordinate of ; the head therefore matches the estimated relevance on average, in the least squares sense over linear functions of . The head is fit on finite candidate pools and can distort the geometry of the new space, whereas the orthogonal Procrustes map, the matrix with orthonormal rows or columns that best maps the new features of the training images onto their old image embeddings, only rotates and projects that space. We therefore average the head with , which serves as an anchor,
with a fixed weight . The stage 1 query uses only the new encoder, and so it keeps the errors of the new model, which the next section addresses.
Consensus Correction with View Barycenters
The teacher is the barycenter of three views, Eq. (3), which corrects the errors that the views make on different items (Figure 2(a), middle). Views from other encoders can correct the errors of a single model. Any query estimated from one model ranks wrongly whatever that model ranks wrongly with confidence. Other encoders see the same image differently, and during training they are available. We call a view any unit query in the old space: the stage 1 head alone, , which we index as view 0, the old query , and auxiliary encoders mapped to the old space by ridge regression onto the old image embeddings, with ridge weights and offset . Using views to remove the errors of one model raises two questions: which targets combine several views and can still be represented by a single query, and which errors of the views such a target can correct.
Every weighted fusion of view scores is reproduced by a single query, the view barycenter. A query interacts with the gallery only through its scores. Fusing the scores of the views with weights in the simplex gives, by linearity, with , so every weighted fusion of view scores is reproduced exactly by a single query. For , its normalized form is the view barycenter, the point of the sphere with the smallest weighted squared chordal distance to the views,
Normalization rescales all scores of a query by the same factor, so the barycenter ranks the gallery exactly as the fused scores do. A target that combines several views and is representable by one query therefore takes the form of a view barycenter, one vector per training image. It remains to determine which errors the views it corrects. Whether the barycenter corrects an error depends on which irrelevant items each view ranks above (Figure 3).

Figure 3. Consensus correction on the sphere. projects the mean of the views (hollow point) onto the sphere and ranks first inside the green cone. (a) Views share their hardest irrelevant caption, so and fails. (b) Their hardest irrelevant captions differ, so and succeeds while every view fails.
Proposition 1 (Barycentric margin). Fix a query with a relevant item and irrelevant items . For each view let for , let be its th smallest value and the margin of the view, and let be the margin of the barycenter; up to a positive factor, with margin . Then: (i) the margin of the barycenter decomposes as
where and is the hardest irrelevant item of view 0, and if and only if the views
with do not share their hardest irrelevant item; (ii) for all integers with , , so is among the top of the barycenter whenever the right side is positive.
Proof sketch. (i) For every irrelevant item , , and taking the minimum over gives . Evaluating at , where view 0 attains its margin, gives the upper bound. If attains , then is a sum of nonnegative terms, which vanishes exactly when is the hardest item of every view with . (ii) View has at most items with , so at most items lie below some threshold, and among the items with the smallest barycentric gaps one lies above all of them. The full proof is in Section S3 of the supplementary material. □
The proposition identifies the quantity that decides whether pooling corrects errors. Pooling never lowers the average margin of the views, and it adds the dispersion gain exactly when the views do not share their hardest irrelevant item. In that case the barycenter can rank first even when some or all views rank it lower, since a positive can outweigh negative view margins. Conversely, if every view ranks below one common irrelevant item , then for all weights, so no barycenter of these views corrects the error. Part (ii) extends the same statement from the first rank to recall at rank . The proposition thus prescribes the views: they must differ in their hardest irrelevant items, which the three views of the new, the old and an auxiliary model trained on other data provide, and three views suffice, since the bound on does not grow with the number of views and the influence of a further view decays as with views.
A noise model of the views sets the weights of the barycenter and predicts which views help. Proposition 1 turns the choice of views into a criterion: a candidate view can correct the new view only on queries where its hardest irrelevant item differs from that of the new view, and since the proposition holds for any weights, the weights only need to reflect the reliability of each view. We model the score of view as , where marks relevance, is the separation of the view, is an offset shared by all items of a query, and is noise with zero mean. The rescaled noises of the views have the covariance matrix over the irrelevant items, assumed equal for all queries. The fused score separates relevant from irrelevant items by ; writing for the weights normalized to this separation, the fusion with the smallest noise variance [2] has
where is the vector of ones. We use these weights when all entries of are nonnegative, so that lies in the simplex, and otherwise the minimizer over nonnegative weights. For view 0 and one extra view whose rescaled noises have standard deviations and and correlation , the relative reduction of noise variance is
The gain depends on the correlation and the noise ratio together; with nonnegative weights an extra view lowers the noise of view 0 exactly when . Both and are estimated from held out pairs before training, so the views and their weights are fixed in advance. We use three views, , and LAION CLIP ViT-H/14 [7, 33] as , and the teacher
Inheriting the Consensus in a Single Encoder
We train the new encoder itself to produce the barycenter (Figure 2(a), right), since the teacher needs the old and the auxiliary encoders, which deployment forbids. A view that is a function of , such as the head or the Procrustes map, can be pooled at deployment at no cost; a view that needs another encoder cannot. Let be the new encoder with a low rank update [11] of the attention query and value projections of the image tower, whose factors form , , and the stage 1 query function, with and frozen, applied to the adapted features. The student minimizes
where weights the auxiliary term and is a linear head trained jointly and discarded after training. The target, the trainable family and the auxiliary term of (4) follow from the analysis below.
Matching the barycenter as a vector keeps the ranking of the teacher in every gallery and regresses onto all views at once. If , every score against a unit gallery vector moves by at most , so the student keeps the first ranked item of the teacher whenever the teacher margin exceeds , in every gallery, including galleries never seen in training. A loss on rankings constrains only the candidate pools seen in training. Matching the vector also matches all views at once: for every unit query , with ,
where depends only on the query, so the first term of (4) is a joint regression onto the views, weighted per query by , which is largest when the views disagree, that is, where Proposition 1 locates the dispersion gain.
The first term of (4) is also a least squares problem over the student family, which shows what a single encoder can reach and why the encoder itself must be adapted. Writing the teacher as , where is the gain of the barycenter over the stage 1 query from which the student starts, stacking the outputs of all training images into one vector, and taking the family to first order and without the unit norm constraint as , with the span of the Jacobian columns of the trainable parameters, the projection theorem gives the minimizer , with the orthogonal projection onto , which captures the fraction
of the gain, the share of that the family explains. A head on the frozen features spans only maps of , while the gain comes from views whose errors differ from those of the new model, which such maps express poorly. Adapting the encoder enlarges in the directions that matter. The same result implies that recovery is partial whenever has a component outside , and that the student can only learn a gain that the barycenter shows on the training images.
The second term of (4) steers the adaptation toward the auxiliary view. It asks the adapted features to predict the native embedding of the auxiliary encoder through the head , which is initialized by ridge regression from . It adds no module at deployment, since is discarded, and reaches the query only through the features, where it favors the directions of that encode the auxiliary view, from which part of the gain comes.
Text queries need no student. Both available views are functions of the new text features : the stage 1 text query , built as in § 3.2 with old images as the gallery, and a map trained contrastively on captions of the training corpus, whose target is the old text embedding of the same caption, so that it needs captions only. The text query is their barycenter with a fixed weight ; since neither view needs another encoder, it is computed directly at deployment.
Training has three stages: fit the stage 1 heads with (1) and the text map ; estimate the weights on held out pairs and compute the teacher (3) offline for every training image; train the auxiliary head and only the low rank factors of the encoder with (4), which keeps the upgrade cheap. At deployment an image query runs the new image encoder with the update merged into its weights, followed by , and a text query runs the new text encoder followed by the text barycenter. The cost is one forward pass of the new encoder plus a small map, the same as any single encoder baseline.
Experiments
Setup. We evaluate five upgrades of a deployed encoder, always against the old gallery. We upgrade the old CLIP ViT-B/32 [26] to CLIP ViT-L/14, CLIP ViT-B/16, SigLIP2 SO400M/14 [39, 47] and MetaCLIP ViT-L/14 [43], and the old MetaCLIP ViT-B/32 to SigLIP2 SO400M/14. All training uses CC3M [35] only, with 100K images for the student and 2.35M captions for the text map. All experiments run on a single NVIDIA A100 GPU.
Benchmarks. We evaluate image to text and text to image retrieval by Recall@1 on Flickr30k [46] and COCO [19] with the large galleries of [14], NoCaps [1], CxC [24] and the English part of XM3600 [38].
Baselines. We compare with five compatible methods. Drift-Adapter [41], LST [20] and -orth [29] are query side maps, XBT [14] fine-tunes the new encoder, and SLERP [30] runs the old and the new encoder for every query.
Hyperparameters. All hyperparameters are fixed on held out pairs before any test evaluation. We set , , , , and , take the teacher weights derived from the noise model on held out CC3M pairs, and train LoRA of rank 8 for 2 epochs at learning rate .
Accuracy
VIBRA is the best compatible method in all 50 combinations of model pair, dataset and direction (Table 1 and Tables S1 and S2 of the supplementary material). It leads the strongest other compatible method by 0.1 to 3.6 points from image to text and by 0.3 to 5.6 points from text to image, and Old → old by 1.5 to 18.0 points. This includes SLERP, which runs two encoders per query, and XBT, which also adapts the new encoder.
| Method | Image → text | Text → image | |||||||||
| Flickr30k | COCO | NoCaps | CxC | XM3600 | Flickr30k | COCO | NoCaps | CxC | XM3600 | ||
| CLIP ViT-B/32 → CLIP ViT-L/14 | |||||||||||
| Old → old | 40.7 | 28.6 | 71.2 | 51.6 | 53.3 | 21.8 | 14.5 | 45.3 | 32.3 | 38.7 | |
| New → new | 48.7 | 34.4 | 73.4 | 58.0 | 54.1 | 28.2 | 18.7 | 47.8 | 38.2 | 42.5 | |
| Drift-Adapter [41] | 42.4 | 29.1 | 67.4 | 52.4 | 45.9 | 21.7 | 14.1 | 43.9 | 31.9 | 38.6 | |
| XBT [14] | 42.5{sup}†` | 30.7{sup}†` | 75.0{sup}†` | 50.8{sup}‡` | 50.8{sup}‡` | 22.4{sup}†` | 15.6{sup}†` | 48.0{sup}†` | 33.9{sup}‡` | 41.6{sup}‡` | |
| SLERP [30] | 48.0{sup}†` | 33.4{sup}†` | 74.0{sup}†` | 56.0 | 55.6 | 23.3{sup}†` | 15.3{sup}†` | 46.6{sup}†` | 33.8 | 40.5 | |
| LST [20] | 40.4 | 28.6 | 67.3 | 52.4 | 45.9 | 20.9 | 13.7 | 42.9 | 30.9 | 37.1 | |
| λ-orth [29] | 35.6 | 25.7 | 64.0 | 49.5 | 47.2 | 21.2 | 13.8 | 44.4 | 31.4 | 40.3 | |
| VIBRA (ours) | 51.6 | 36.0 | 76.1 | 59.6 | 56.1 | 25.9 | 17.3 | 50.8 | 36.8 | 45.1 | |
| CLIP ViT-B/32 → SigLIP2 SO400M/14 | |||||||||||
| Old → old | 40.7 | 28.6 | 71.2 | 51.6 | 53.3 | 21.8 | 14.5 | 45.3 | 32.3 | 38.7 | |
| New → new | 69.1 | 50.4 | 88.5 | 72.4 | 65.8 | 51.5 | 35.1 | 70.1 | 56.8 | 58.8 | |
| Drift-Adapter [41] | 50.6 | 34.3 | 74.2 | 58.2 | 52.9 | 24.2 | 15.7 | 46.9 | 34.7 | 39.9 | |
| XBT [14] | 38.5{sup}‡` | 27.9{sup}‡` | 71.6{sup}‡` | 51.8{sup}‡` | 54.2{sup}‡` | 23.2{sup}‡` | 16.2{sup}‡` | 49.7{sup}‡` | 35.7{sup}‡` | 43.4{sup}‡` | |
| SLERP [30] | 51.0{sup}†` | 37.4{sup}†` | 78.6{sup}†` | 61.7 | 58.9 | 25.8{sup}†` | 17.2{sup}†` | 50.6{sup}†` | 36.5 | 43.2 | |
| LST [20] | 46.1 | 31.5 | 71.6 | 55.3 | 49.9 | 22.9 | 15.0 | 44.9 | 33.4 | 38.4 | |
| λ-orth [29] | 42.2 | 30.4 | 70.6 | 55.4 | 53.8 | 25.5 | 16.5 | 50.2 | 35.7 | 43.6 | |
| VIBRA (ours) | 54.0 | 38.5 | 79.5 | 62.2 | 60.8 | 30.3 | 20.3 | 56.2 | 41.2 | 48.7 | |
| MetaCLIP ViT-B/32 → SigLIP2 SO400M/14 | |||||||||||
| Old → old | 43.3 | 33.0 | 77.9 | 56.7 | 59.2 | 28.0 | 19.3 | 55.9 | 40.1 | 49.9 | |
| New → new | 69.1 | 50.4 | 88.5 | 72.4 | 65.8 | 51.5 | 35.1 | 70.1 | 56.8 | 58.8 | |
| Drift-Adapter [41] | 58.5 | 40.9 | 81.0 | 64.5 | 58.8 | 26.8 | 18.2 | 53.3 | 38.6 | 45.9 | |
| XBT [14] | 49.0{sup}‡` | 34.4{sup}‡` | 79.8{sup}‡` | 59.0{sup}‡` | 57.8{sup}‡` | 22.7{sup}‡` | 17.0{sup}‡` | 53.1{sup}‡` | 37.6{sup}‡` | 45.9{sup}‡` | |
| SLERP [30] | 59.7 | 42.5 | 84.2 | 66.3 | 62.4 | 29.7 | 20.5 | 57.9 | 41.8 | 50.9 | |
| LST [20] | 56.5 | 39.1 | 79.6 | 62.9 | 57.9 | 25.7 | 17.3 | 51.4 | 37.0 | 44.4 | |
| λ-orth [29] | 49.3 | 37.3 | 78.6 | 61.3 | 59.2 | 24.0 | 16.7 | 52.6 | 36.0 | 45.8 | |
| VIBRA (ours) | 61.3 | 43.3 | 84.4 | 66.5 | 65.5 | 30.0 | 21.0 | 58.9 | 42.5 | 51.7 |
Table 1. Recall@1 (%) against the old gallery. Gray: references. {sup}† published; {sup}‡ our reproduction of XBT. Bold: best compatible method.
VIBRA closes much of the gap to re-encoding the gallery. On the two CLIP upgrades it matches or exceeds New → new in 16 of 20 cells, including 7 of the 10 cells of CLIP ViT-B/32 → CLIP ViT-L/14 in Table 1, although New → new reencodes the whole gallery. On the three larger upgrades New → new stays ahead by 0.3 to 21.5 points; VIBRA recovers 45% to 95% of the image to text gain that re-encoding brings over Old → old, and 9% to 69% of the text to image gain, since text queries use no student. Pooled over all settings, VIBRA obtains 55.0% of the update gain of BCT [36], against 36.1% for SLERP (Table S9 of the supplementary material).
Latency
VIBRA lies above the frontier of the baselines on every dataset and in both directions (Figure 4). At equal or lower latency, VIBRA is ahead by 3.4 to 9.0 points from image to text and by 0.8 to 6.1 points from text to image.
Figure 4. Recall@1 against query latency at batch size 1. Each marker is one new backbone, and the dashed line is the frontier of the baselines. Points toward the upper left are better.
The latency of VIBRA is that of the new encoder. It stays within 0.5 ms of the new encoder for image queries, where the low rank update is merged, and is 0.4 to 0.9 ms above it for text queries, which add the text map, while SLERP adds the old encoder and costs 7.4 to 8.2 ms more per query than VIBRA. The gallery size matters little: NoCaps, whose galleries are a quarter of the size of those of COCO, is faster by about 0.5 ms. Table S3 of the supplementary material gives latency and throughput for all five pairs. Training VIBRA takes less than one A100 hour, 19 to 27 times less than XBT and as much as re about two million images (Table S8 of the supplementary material).
| Pair | Old → old | New → new | Drift-Adapter | XBT | SLERP | LST | λ-orth | VIBRA |
| I→T: CLIP ViT-B/32 → CLIP ViT-L/14 | 9.0 / 8603 | 16.6 / 711 | 16.6 / 710 | 16.8 / 688 | 25.2 / 659 | 16.5 / 709 | 16.5 / 709 | 16.6 / 709 |
| I→T: CLIP ViT-B/32 → CLIP ViT-B/16 | 9.0 / 8600 | 8.8 / 2707 | 9.0 / 2702 | 9.1 / 2560 | 17.6 / 2083 | 8.9 / 2693 | 8.9 / 2693 | 8.9 / 2693 |
| I→T: CLIP ViT-B/32 → SigLIP2 SO400M/14 | 8.9 / 8588 | 19.0 / 549 | 18.6 / 547 | 18.1 / 590 | 27.1 / 517 | 18.8 / 546 | 18.7 / 546 | 18.9 / 546 |
| I→T: CLIP ViT-B/32 → MetaCLIP ViT-L/14 | 8.8 / 8601 | 16.2 / 712 | 16.2 / 711 | 16.4 / 687 | 24.4 / 660 | 16.2 / 710 | 16.3 / 710 | 16.2 / 710 |
| I→T: MetaCLIP ViT-B/32 → SigLIP2 SO400M/14 | 8.9 / 8587 | 19.1 / 549 | 19.0 / 547 | 18.3 / 590 | 27.4 / 517 | 19.0 / 546 | 18.9 / 546 | 18.9 / 546 |
| T→I: CLIP ViT-B/32 → CLIP ViT-L/14 | 9.1 / 26898 | 9.2 / 19041 | 9.5 / 17226 | 9.6 / 18954 | 18.4 / 11298 | 9.5 / 19018 | 9.4 / 19099 | 10.0 / 18844 |
| T→I: CLIP ViT-B/32 → CLIP ViT-B/16 | 8.9 / 27155 | 8.9 / 27146 | 9.3 / 26123 | 9.3 / 26092 | 18.2 / 13550 | 9.1 / 26869 | 9.1 / 26782 | 9.7 / 25377 |
| T→I: CLIP ViT-B/32 → SigLIP2 SO400M/14 | 9.0 / 26657 | 16.7 / 2517 | 16.7 / 2523 | 16.7 / 2522 | 26.1 / 2321 | 16.4 / 2523 | 16.5 / 2524 | 17.0 / 2519 |
| T→I: CLIP ViT-B/32 → MetaCLIP ViT-L/14 | 9.1 / 26928 | 9.2 / 19028 | 9.5 / 19066 | 9.4 / 17748 | 18.4 / 11384 | 9.3 / 18866 | 9.3 / 19096 | 9.8 / 18842 |
| T→I: MetaCLIP ViT-B/32 → SigLIP2 SO400M/14 | 9.0 / 26886 | 16.5 / 2517 | 16.7 / 2523 | 16.6 / 2521 | 25.8 / 2318 | 16.3 / 2523 | 16.3 / 2524 | 16.9 / 2519 |
Table S3. Query latency on the Large COCO galleries: ms per query at batch size 1 / queries per second at batch size 256. One A100-SXM4-40GB, bf16, exact top 10 search over the old gallery (175,680 captions for image → text, 35,136 images for text → image); preprocessing and tokenization excluded.
Table S8. Training cost in A100 hours (bf16, preprocessing and tokenization excluded, one configuration and one seed), counting the encoding of the training data with every encoder a method needs and the optimization, and the cost of a full backfill with the new model.
| Method | B/32 L/14 | B/32 B/16 | B/32 SO400M | B/32 MetaCLIP L/14 | MetaCLIP B/32 SO400M |
| XBT [14], paper scale | 10.03 | 4.18 | 18.94 | 10.03 | 18.93 |
| VIBRA (ours) | 0.44 | 0.22 | 0.71 | 0.44 | 0.71 |
| Full backfill, per 1M images | 0.39 | 0.10 | 0.51 | 0.39 | 0.51 |
| Full backfill, per 1B images | 388 | 101 | 506 | 389 | 506 |
| Gallery where backfill costs as much as VIBRA | 1.1M | 2.2M | 1.4M | 1.1M | 1.4M |
Consensus Analysis
The barycenter corrects errors where Proposition 1 locates the dispersion gain (Table 2). We split the test queries by the condition of the proposition, whether the three views share their hardest irrelevant item. The dispersion gain is positive on 88.4% to 93.0% of the queries, and on these queries the barycenter corrects 10.0% to 13.4% of the errors of the new view, 2.5 to 3 times as often as on the queries with .
| L/14 | B/16 | SO400M | MC L/14 | MC→SO | |
| Queries with (%) | 91.2 | 90.1 | 93.0 | 92.0 | 88.4 |
| Errors corrected, (%) | 11.3 | 10.9 | 13.4 | 12.5 | 10.0 |
| Errors corrected, (%) | 4.2 | 4.2 | 4.5 | 4.2 | 4.0 |
| Net gain (points) | 4.7 | 4.6 | 5.2 | 5.3 | 1.2 |
Table 2. Consensus correction split by the condition of Proposition 1 (image → text, one positive caption per image). Errors corrected: share of the errors of the new view that the barycenter corrects. Columns name the new model for CLIP ViT-B/32; MC→SO: MetaCLIP ViT-B/32 → SigLIP2 SO400M/14.
The condition of the proposition is exact on the test sets. On the queries whose views share their hardest irrelevant item the measured is zero up to numerical precision, and when all three views are wrong on such a query the barycenter corrects none of them, the case of Figure 3(a). Overall the barycenter raises Recall@1 over the new view on every pair, by 1.2 to 5.3 points. § 10 of the supplementary material gives further statistics.
Ablation
Each component of VIBRA improves on the previous one, and relevance matching contributes most (Table 3, mean over the five datasets). Starting from the Procrustes map, which imitates the old query, relevance matching raises Recall@1 by 2.4 to 10.8 points, 5.4 on average; training the new encoder on the consensus teacher adds 0.3 to 2.4 points, and the auxiliary term another 0.1 to 1.0 points.
| L/14 | B/16 | SO400M | MC L/14 | MC→SO | Mean | |
| Procrustes map | 48.8 | 48.6 | 46.8 | 50.9 | 56.3 | 50.3 |
| + Relevance matching | 52.4 | 51.2 | 57.6 | 53.3 | 63.8 | 55.7 |
| + Consensus teacher | 54.8 | 52.6 | 58.6 | 55.2 | 64.1 | 57.1 |
| VIBRA (all three) | 55.8 | 53.2 | 58.8 | 55.8 | 64.2 | 57.6 |
Table 3. Ablation of the components (image → text R@1, %); each row adds one component. Columns as in Table 2.
Every component helps on every model pair. Both estimates of relevance are needed, and the teacher of three views exceeds every single view and the teacher of two views (Table S4 and Table S7 of the supplementary material).
Table S4. Mixing weight of the stage 1 target (stage 1 query ; mean R@1 over the five sets, image text / text image, %). uses the label only and the ranking of the new model only; is the minimum variance weight estimated on COCO DEV.
| Pair | 0.25 | 0.5 | 0.75 | 1 | ||
| B/32 L/14 | 48.9 / 33.2 | 50.8 / 33.9 | 52.4 / 34.4 | 53.1 / 34.2 | 51.7 / 32.2 | 53.0 / 34.0 (0.82) |
| B/32 B/16 | 47.1 / 32.9 | 49.7 / 33.7 | 51.2 / 33.9 | 51.9 / 33.4 | 50.6 / 30.9 | 51.9 / 33.1 (0.81) |
| B/32 SO400M | 56.0 / 37.2 | 56.9 / 37.8 | 57.6 / 38.5 | 57.8 / 39.0 | 55.9 / 38.8 | 57.5 / 39.1 (0.84) |
| B/32 MetaCLIP L/14 | 51.9 / 36.1 | 52.5 / 36.6 | 53.3 / 37.1 | 53.0 / 37.0 | 51.6 / 36.5 | 53.0 / 37.1 (0.83) |
| MetaCLIP B/32 SO400M | 62.6 / 38.7 | 63.4 / 39.3 | 63.8 / 40.0 | 64.0 / 40.4 | 62.9 / 40.0 | 63.9 / 40.4 (0.84) |
Table S4.
Table S7. Full ablation, extending Table 3 of the main paper. Contribution of each component (image text , mean over the five datasets, %; students with seed 0). Stage 1: the Procrustes map alone (), the head alone () and . Single views and teachers are queries in the old space; the teacher of two views drops the auxiliary encoder. Students: trained on the teacher of two views without the auxiliary term, on the teacher of three views without the auxiliary term (), and VIBRA.
| Pair | Procrustes | Head | Old | H/14 | 2 views | 3 views | 2 views | 3 views, | VIBRA | |
| B/32 L/14 | 48.8 | 50.2 | 52.4 | 49.1 | 50.2 | 56.0 | 56.6 | 54.8 | 54.8 | 55.8 |
| B/32 B/16 | 48.6 | 49.6 | 51.2 | 49.1 | 50.2 | 54.3 | 55.8 | 52.7 | 52.6 | 53.2 |
| B/32 SO400M | 46.8 | 52.9 | 57.6 | 49.1 | 50.2 | 58.8 | 59.7 | 58.8 | 58.6 | 58.8 |
| B/32 MetaCLIP L/14 | 50.9 | 50.9 | 53.3 | 49.1 | 50.2 | 56.8 | 57.4 | 55.5 | 55.2 | 55.8 |
| MetaCLIP B/32 SO400M | 56.3 | 62.3 | 63.8 | 54.0 | 54.5 | 64.5 | 64.9 | 64.5 | 64.1 | 64.2 |
Sensitivity
The stage 1 query is insensitive to its hyperparameters around the deployed values. We vary one hyperparameter at a time around the deployed setting (Figure 5, mean over the five datasets). Temperatures from 0.01 to 0.03 stay within 1.3 points and only loses 1.7 to 4.2, and over from 256 to 4096 Recall@1 varies by at most 1.5 points. Every between 0.5 and 0.75 is within 1.4 points of the best (Table S5 of the supplementary material), and the minimum variance weight , computed on COCO DEV without tuning, falls at 0.81 to 0.84 and stays within 0.3 points of the best (Table S4).
Figure 5. Sensitivity of image → text R@1 to one hyperparameter at a time.
Table S5. Weight of the stage 1 head in (mean R@1 over the five sets, image text / text image, %). is the Procrustes map alone, the head alone; the deployed value is .
| Pair | 0.25 | 0.5 | 0.75 | 1 | |
| B/32 L/14 | 48.8 / 30.3 | 52.1 / 33.1 | 53.2 / 34.5 | 52.4 / 34.4 | 50.2 / 32.9 |
| B/32 B/16 | 48.6 / 28.8 | 51.0 / 32.0 | 52.1 / 33.7 | 51.2 / 33.9 | 49.6 / 33.1 |
| B/32 SO400M | 46.8 / 32.8 | 55.3 / 36.7 | 59.0 / 38.6 | 57.6 / 38.5 | 52.9 / 36.8 |
| B/32 MetaCLIP L/14 | 50.9 / 35.0 | 53.9 / 36.8 | 54.5 / 37.5 | 53.3 / 37.1 | 50.9 / 35.5 |
| MetaCLIP B/32 SO400M | 56.3 / 37.2 | 61.4 / 39.4 | 63.7 / 40.1 | 63.8 / 40.0 | 62.3 / 38.7 |
Table S5.
The student is insensitive to the auxiliary weight, the rank of the update and the amount of training data. Doubling changes the student by at most 0.6 points. Ranks 4 and 16 are within 0.1 points of rank 8, so a larger update brings no gain, and students trained on 25K or 50K images with the same number of steps are within 0.2 points of the one trained on 100K.
Conclusion
VIBRA upgrades the query encoder of a cross-modal retrieval system without re-encoding the gallery and with one encoder per query. It derives its training target from the retrieval objective, pools the views of several models into a barycenter that a single query reproduces, and transfers this target into the new encoder. On five benchmarks with five upgrades, it is the best compatible method in all 50 settings, improving Recall@1 by up to 5.6 points over the strongest baseline and by up to 18.0 points over the old model, with less than 1 ms added to the latency of the new encoder.
LLM Usage Statement
In this work, we used a large language model assistant to implement the method, the baselines and the evaluation in code, to design and give feedback on experiments, including the preregistered sensitivity and ablation studies, to check derivations and assist in writing the proofs of the supplementary material, and to help interpret the results. We have not used it to generate synthetic data sets, and translation and qualitative data analysis are not applicable to this work. Additionally, we used it to suggest experimental parameters, to create and modify the figures, to search for and summarize related work, to draft and edit parts of the text.
We have reviewed all AI assisted work. The authors defined the problem and the method, inspected the code and ran every experiment, checked each reported number against the experiment outputs, verified the derivations and proofs by hand, and read the cited papers to confirm the statements made about them. We take responsibility for the final content of this work, including text, claims and artifacts produced with the aid of generative AI.
Supplementary Material
Section, equation and proposition numbers without the prefix S refer to the main paper.
Full Retrieval Results
Tables S1 and S2 extend the R@1 results of the main paper to R@5 and R@10 for every model pair, dataset and method.
Query Latency
Table S3 lists the latency and throughput of every method on the Large COCO galleries for all five model pairs.
Proofs
Throughout, gallery vectors and views are unit vectors, the weights lie in the simplex, , and is a finite set with at least elements. For a function on , denotes its th smallest value. We use two elementary facts about order statistics: if for all , then ; and for every subset with , .
Relevance matching
Optimality condition. With and , the cross entropy equals . The first term is linear in and the second is a log sum exp of linear functions of , which is convex. Its gradient is , which gives the stated gradient, and a convex differentiable function is minimized exactly where its gradient vanishes, which gives the condition of Section 3.2 of the main paper. Under the constraint , a stationary point satisfies , so the condition holds for the component tangent to the sphere.
Loss (1). The cross entropy is linear in its first argument, so , and , where the entropy does not depend on .
Stationarity of the linear head. Let and , so that . By the chain rule, . At a stationary point this matrix vanishes: the residual of the optimality condition, projected onto the tangent space and rescaled by , is uncorrelated with every coordinate of .
View barycenters
Closed form (2). For a unit vector , , hence . By the Cauchy Schwarz inequality, with equality only at when . The score identity is linearity of the inner product, and dividing by does not change the ranking.
Proof of Proposition 1. Since and the scores of the barycenter are , its gaps are , which have the same signs and order as . The rank of is one plus the number of irrelevant items with , so is among the top exactly when .
(i) For every , , so and . Evaluating at , where , gives , which is the upper bound. Let attain . Then is a sum of nonnegative terms, so implies for every with , that is, is the hardest item of all these views. Conversely, if one item is the hardest for all of them, and therefore .
(ii) Let . By definition of the order statistic, , so . Let be a set of items with the smallest values of . Since , some lies outside every , so for all and, as , . Finally .
Effect of an additional view. With views of equal weight, adding view gives . Since , the first fact on order statistics gives . With the items with the smallest , the second fact gives . Both changes are of order .
Minimum variance weights. Under the noise model, whenever , with noise variance . For positive definite , minimizing subject to with a Lagrange multiplier gives , hence with variance for every . The fused score equals minus a term constant in , so the barycenter weights are . For two views, and , which gives .
| Method | Flickr30k R@1 | Flickr30k R@5 | Flickr30k R@10 | COCO R@1 | COCO R@5 | COCO R@10 | NoCaps R@1 | NoCaps R@5 | NoCaps R@10 | CxC R@1 | CxC R@5 | CxC R@10 | XM3600 R@1 | XM3600 R@5 | XM3600 R@10 |
| CLIP ViT-B/32 → CLIP ViT-L/14 | |||||||||||||||
| Old → old | 40.7 | 64.7 | 73.7 | 28.6 | 50.4 | 59.9 | 71.2 | 91.9 | 96.3 | 51.6 | 77.4 | 85.5 | 53.3 | 79.1 | 86.5 |
| New → new | 48.7 | 73.0 | 81.2 | 34.4 | 56.0 | 65.4 | 73.4 | 93.5 | 97.4 | 58.0 | 81.7 | 88.4 | 54.1 | 78.8 | 86.9 |
| Drift-Adapter [41] | 42.4 | 66.8 | 76.0 | 29.1 | 50.1 | 59.3 | 67.4 | 90.1 | 95.4 | 52.4 | 76.8 | 84.8 | 45.9 | 72.6 | 81.8 |
| XBT [14] | 42.5† | 66.0† | 74.9† | 30.7† | 52.6† | 62.3† | 75.0† | 93.3† | 97.3† | 50.8‡ | 76.0‡ | 85.0‡ | 50.8‡ | 77.2‡ | 85.6‡ |
| SLERP [30] | 48.0† | 71.8† | 80.3† | 33.4† | 55.4† | 64.8† | 74.0† | 93.3† | 96.9† | 56.0 | 80.6 | 88.1 | 55.6 | 81.3 | 88.2 |
| LST [20] | 40.4 | 64.9 | 74.2 | 28.6 | 49.5 | 58.9 | 67.3 | 89.4 | 95.3 | 52.4 | 76.5 | 85.0 | 45.9 | 72.4 | 81.9 |
| λ-orth [29] | 35.6 | 59.6 | 69.6 | 25.7 | 46.4 | 56.0 | 64.0 | 87.9 | 94.1 | 49.5 | 74.5 | 84.1 | 47.2 | 74.3 | 83.5 |
| VIBRA (ours) | 51.6 | 75.7 | 83.7 | 36.0 | 58.6 | 67.9 | 76.1 | 94.4 | 97.9 | 59.6 | 83.0 | 90.1 | 56.1 | 81.9 | 89.4 |
| CLIP ViT-B/32 → CLIP ViT-B/16 | |||||||||||||||
| Old → old | 40.7 | 64.7 | 73.7 | 28.6 | 50.4 | 59.9 | 71.2 | 91.9 | 96.3 | 51.6 | 77.4 | 85.5 | 53.3 | 79.1 | 86.5 |
| New → new | 44.4 | 68.6 | 77.4 | 30.9 | 52.4 | 62.0 | 71.8 | 91.9 | 96.8 | 54.1 | 78.4 | 86.9 | 55.1 | 80.1 | 87.6 |
| Drift-Adapter [41] | 42.1 | 66.4 | 75.7 | 28.6 | 49.8 | 59.2 | 68.6 | 90.7 | 95.5 | 52.1 | 77.0 | 85.4 | 49.6 | 76.6 | 85.3 |
| XBT [14] | 35.1‡ | 58.2‡ | 67.9‡ | 24.4‡ | 44.6‡ | 54.2‡ | 66.7‡ | 89.9‡ | 95.0‡ | 47.2‡ | 73.3‡ | 82.6‡ | 48.3‡ | 75.9‡ | 84.6‡ |
| SLERP [30] | 45.2 | 69.8 | 78.5 | 31.4 | 53.4 | 62.6 | 73.6 | 93.1 | 97.0 | 55.4 | 79.5 | 87.2 | 54.7 | 81.0 | 87.8 |
| LST [20] | 40.6 | 64.6 | 74.2 | 28.0 | 48.9 | 58.5 | 67.7 | 90.3 | 95.6 | 51.1 | 76.8 | 85.0 | 49.8 | 75.6 | 84.5 |
| λ-orth [29] | 32.6 | 56.2 | 66.4 | 23.7 | 44.0 | 53.5 | 63.2 | 87.2 | 93.4 | 46.5 | 72.9 | 82.2 | 47.5 | 74.7 | 83.5 |
| VIBRA (ours) | 47.0 | 71.9 | 80.2 | 33.0 | 55.1 | 64.5 | 73.9 | 92.7 | 96.9 | 57.0 | 81.1 | 88.1 | 54.8 | 81.7 | 89.5 |
| CLIP ViT-B/32 → SigLIP2 SO400M/14 | |||||||||||||||
| Old → old | 40.7 | 64.7 | 73.7 | 28.6 | 50.4 | 59.9 | 71.2 | 91.9 | 96.3 | 51.6 | 77.4 | 85.5 | 53.3 | 79.1 | 86.5 |
| New → new | 69.1 | 88.4 | 93.1 | 50.4 | 72.6 | 80.1 | 88.5 | 98.5 | 99.6 | 72.4 | 90.7 | 95.3 | 65.8 | 87.5 | 93.0 |
| Drift-Adapter [41] | 50.6 | 74.1 | 81.9 | 34.3 | 56.4 | 65.5 | 74.2 | 93.4 | 97.3 | 58.2 | 81.8 | 88.6 | 52.9 | 78.0 | 86.0 |
| XBT [14] | 38.5‡ | 62.0‡ | 71.4‡ | 27.9‡ | 48.9‡ | 59.0‡ | 71.6‡ | 92.8‡ | 96.7‡ | 51.8‡ | 77.6‡ | 86.0‡ | 54.2‡ | 80.1‡ | 87.2‡ |
| SLERP [30] | 51.0† | 75.3† | 83.5† | 37.4† | 60.4† | 69.3† | 78.6† | 95.3† | 98.3† | 61.7 | 84.2 | 91.0 | 58.9 | 83.9 | 90.5 |
| LST [20] | 46.1 | 70.0 | 78.5 | 31.5 | 53.4 | 62.8 | 71.6 | 91.7 | 96.0 | 55.3 | 79.6 | 87.2 | 49.9 | 76.1 | 84.3 |
| λ-orth [29] | 42.2 | 66.7 | 76.1 | 30.4 | 52.0 | 61.6 | 70.6 | 91.9 | 96.2 | 55.4 | 80.1 | 87.7 | 53.8 | 79.5 | 87.2 |
| VIBRA (ours) | 54.0 | 77.7 | 85.2 | 38.5 | 61.4 | 70.4 | 79.5 | 95.5 | 98.2 | 62.2 | 84.5 | 91.4 | 60.8 | 84.8 | 91.4 |
| CLIP ViT-B/32 → MetaCLIP ViT-L/14 | |||||||||||||||
| Old → old | 40.7 | 64.7 | 73.7 | 28.6 | 50.4 | 59.9 | 71.2 | 91.9 | 96.3 | 51.6 | 77.4 | 85.5 | 53.3 | 79.1 | 86.5 |
| New → new | 57.1 | 80.3 | 87.6 | 42.4 | 64.9 | 73.5 | 82.4 | 96.8 | 99.0 | 65.5 | 86.5 | 92.3 | 62.2 | 85.8 | 91.8 |
| Drift-Adapter [41] | 42.6 | 66.6 | 75.9 | 28.7 | 49.8 | 59.3 | 68.6 | 90.6 | 95.6 | 51.6 | 76.4 | 84.8 | 47.0 | 72.7 | 81.4 |
| XBT [14] | 36.5‡ | 60.2‡ | 69.9‡ | 26.9‡ | 47.4‡ | 57.1‡ | 70.3‡ | 91.8‡ | 96.4‡ | 50.6‡ | 77.0‡ | 85.0‡ | 53.1‡ | 79.1‡ | 86.6‡ |
| SLERP [30] | 49.2 | 73.2 | 81.4 | 34.2 | 56.5 | 65.7 | 75.1 | 93.7 | 97.2 | 57.9 | 81.2 | 88.7 | 55.3 | 80.3 | 88.1 |
| LST [20] | 43.2 | 67.4 | 76.3 | 29.7 | 51.2 | 60.6 | 68.4 | 90.2 | 95.4 | 52.8 | 77.7 | 86.1 | 47.4 | 74.1 | 82.7 |
| λ-orth [29] | 37.2 | 61.8 | 71.7 | 27.1 | 48.3 | 57.8 | 65.8 | 89.4 | 94.9 | 51.5 | 76.3 | 85.3 | 48.5 | 76.5 | 84.8 |
| VIBRA (ours) | 49.6 | 73.8 | 81.9 | 35.7 | 58.2 | 67.7 | 76.9 | 93.7 | 97.6 | 59.3 | 82.8 | 89.8 | 57.6 | 82.9 | 89.6 |
| MetaCLIP ViT-B/32 → SigLIP2 SO400M/14 | |||||||||||||||
| Old → old | 43.3 | 67.2 | 76.0 | 33.0 | 55.3 | 64.7 | 77.9 | 94.5 | 97.6 | 56.7 | 80.8 | 88.4 | 59.2 | 83.9 | 90.7 |
| New → new | 69.1 | 88.4 | 93.1 | 50.4 | 72.6 | 80.1 | 88.5 | 98.5 | 99.6 | 72.4 | 90.7 | 95.3 | 65.8 | 87.5 | 93.0 |
| Drift-Adapter [41] | 58.5 | 80.9 | 87.6 | 40.9 | 63.5 | 72.2 | 81.0 | 95.3 | 98.0 | 64.5 | 86.1 | 92.0 | 58.8 | 83.6 | 90.1 |
| XBT [14] | 49.0‡ | 72.1‡ | 80.4‡ | 34.4‡ | 56.7‡ | 65.8‡ | 79.8‡ | 95.5‡ | 98.4‡ | 59.0‡ | 82.6‡ | 89.4‡ | 57.8‡ | 81.8‡ | 87.9‡ |
| SLERP [30] | 59.7 | 81.2 | 87.8 | 42.5 | 65.0 | 73.7 | 84.2 | 97.3 | 99.0 | 66.3 | 87.4 | 93.2 | 62.4 | 86.5 | 92.1 |
| LST [20] | 56.5 | 79.2 | 86.5 | 39.1 | 61.8 | 70.7 | 79.6 | 95.1 | 97.8 | 62.9 | 84.5 | 91.2 | 57.9 | 82.9 | 90.0 |
| λ-orth [29] | 49.3 | 73.2 | 81.8 | 37.3 | 59.7 | 69.0 | 78.6 | 94.7 | 97.7 | 61.3 | 84.4 | 90.9 | 59.2 | 84.3 | 91.1 |
| VIBRA (ours) | 61.3 | 83.1 | 89.3 | 43.3 | 65.9 | 74.2 | 84.4 | 97.5 | 99.2 | 66.5 | 87.8 | 92.9 | 65.5 | 88.6 | 93.7 |
Table S1. Image → text retrieval against the old gallery (R@1 / R@5 / R@10, %). The old gallery is never reencoded. Every method except SLERP encodes the query with one network; SLERP encodes the query with one network; SLERP runs the old and the new encoder. Gray rows are references: Old → old keeps the old model, New → new reencodes the whole gallery. †published number. ‡our reproduction of XBT at the scale of its paper, used where the paper reports no number. Bold marks the best compatible method.
| Method | Flickr30k R@1 | Flickr30k R@5 | Flickr30k R@10 | COCO R@1 | COCO R@5 | COCO R@10 | NoCaps R@1 | NoCaps R@5 | NoCaps R@10 | CxC R@1 | CxC R@5 | CxC R@10 | XM3600 R@1 | XM3600 R@5 | XM3600 R@10 |
| CLIP ViT-B/32 → CLIP ViT-L/14 | |||||||||||||||
| Old → old | 21.8 | 41.6 | 51.1 | 14.5 | 30.2 | 38.9 | 45.3 | 75.0 | 84.6 | 32.3 | 58.8 | 70.0 | 38.7 | 66.5 | 76.5 |
| New → new | 28.2 | 49.6 | 59.0 | 18.7 | 35.9 | 44.4 | 47.8 | 76.6 | 85.2 | 38.2 | 63.9 | 74.2 | 42.5 | 69.2 | 77.9 |
| Drift-Adapter [41] | 21.7 | 41.6 | 51.1 | 14.1 | 29.9 | 38.5 | 43.9 | 74.3 | 83.8 | 31.9 | 58.3 | 69.5 | 38.6 | 66.6 | 76.4 |
| XBT [14] | 22.4† | 42.7† | 52.4† | 15.6† | 32.3† | 41.3† | 48.0† | 79.0† | 88.2† | 33.9‡ | 60.6‡ | 71.8‡ | 41.6‡ | 70.7‡ | 80.0‡ |
| SLERP [30] | 23.3† | 43.8† | 53.3† | 15.3† | 31.6† | 40.3† | 46.6† | 76.5† | 85.5† | 33.8 | 60.4 | 71.5 | 40.5 | 68.9 | 78.6 |
| LST [20] | 20.9 | 40.4 | 49.9 | 13.7 | 29.1 | 37.7 | 42.9 | 73.3 | 83.2 | 30.9 | 57.5 | 68.8 | 37.1 | 65.5 | 75.8 |
| λ-orth [29] | 21.2 | 40.6 | 50.0 | 13.8 | 29.1 | 37.5 | 44.4 | 74.5 | 84.2 | 31.4 | 57.4 | 68.5 | 40.3 | 68.0 | 77.6 |
| VIBRA (ours) | 25.9 | 47.2 | 56.9 | 17.3 | 34.9 | 44.0 | 50.8 | 80.2 | 88.6 | 36.8 | 63.8 | 74.4 | 45.1 | 73.3 | 81.3 |
| CLIP ViT-B/32 → CLIP ViT-B/16 | |||||||||||||||
| Old → old | 21.8 | 41.6 | 51.1 | 14.5 | 30.2 | 38.9 | 45.3 | 75.0 | 84.6 | 32.3 | 58.8 | 70.0 | 38.7 | 66.5 | 76.5 |
| New → new | 24.9 | 45.4 | 54.9 | 16.5 | 33.0 | 41.6 | 46.8 | 76.1 | 85.0 | 34.6 | 61.1 | 72.0 | 42.2 | 68.5 | 77.0 |
| Drift-Adapter [41] | 20.4 | 39.8 | 49.2 | 13.6 | 28.7 | 37.1 | 42.9 | 72.9 | 83.0 | 30.6 | 57.3 | 68.8 | 36.3 | 64.3 | 74.4 |
| XBT [14] | 20.9‡ | 40.7‡ | 50.6‡ | 14.3‡ | 30.5‡ | 39.4‡ | 46.3‡ | 77.7‡ | 87.2‡ | 32.9‡ | 59.7‡ | 71.0‡ | 41.2‡ | 70.1‡ | 79.8‡ |
| SLERP [30] | 22.3 | 42.4 | 51.8 | 14.6 | 30.4 | 39.2 | 45.6 | 75.3 | 84.7 | 32.4 | 59.3 | 70.3 | 38.6 | 66.2 | 76.7 |
| LST [20] | 19.8 | 38.9 | 48.3 | 13.2 | 28.2 | 36.4 | 42.1 | 72.1 | 82.3 | 30.0 | 56.6 | 68.3 | 35.5 | 63.5 | 73.7 |
| λ-orth [29] | 21.3 | 40.7 | 50.1 | 13.8 | 29.1 | 37.5 | 44.2 | 74.4 | 84.1 | 31.5 | 57.3 | 68.5 | 40.0 | 67.9 | 77.6 |
| VIBRA (ours) | 25.2 | 46.5 | 56.1 | 16.8 | 34.2 | 43.3 | 49.6 | 79.5 | 88.0 | 36.4 | 63.3 | 74.0 | 43.9 | 72.2 | 80.8 |
| CLIP ViT-B/32 → SigLIP2 SO400M/14 | |||||||||||||||
| Old → old | 21.8 | 41.6 | 51.1 | 14.5 | 30.2 | 38.9 | 45.3 | 75.0 | 84.6 | 32.3 | 58.8 | 70.0 | 38.7 | 66.5 | 76.5 |
| New → new | 51.5 | 73.2 | 80.4 | 35.1 | 56.5 | 65.0 | 70.1 | 91.0 | 95.1 | 56.8 | 80.2 | 87.3 | 58.8 | 82.6 | 89.2 |
| Drift-Adapter [41] | 24.2 | 44.6 | 54.3 | 15.7 | 32.6 | 41.4 | 46.9 | 76.0 | 84.9 | 34.7 | 61.2 | 72.1 | 39.9 | 68.3 | 77.8 |
| XBT [14] | 23.2‡ | 44.2‡ | 54.2‡ | 16.2‡ | 33.4‡ | 42.6‡ | 49.7‡ | 80.9‡ | 89.4‡ | 35.7‡ | 62.9‡ | 73.7‡ | 43.4‡ | 72.6‡ | 81.6‡ |
| SLERP [30] | 25.8† | 47.2† | 56.8† | 17.2† | 34.8† | 44.0† | 50.6† | 79.7† | 87.6† | 36.5 | 63.5 | 74.4 | 43.2 | 71.4 | 80.4 |
| LST [20] | 22.9 | 43.1 | 52.6 | 15.0 | 31.2 | 39.9 | 44.9 | 73.9 | 83.2 | 33.4 | 59.6 | 70.5 | 38.4 | 66.6 | 76.8 |
| λ-orth [29] | 25.5 | 46.1 | 55.6 | 16.5 | 33.5 | 42.4 | 50.2 | 78.5 | 87.1 | 35.7 | 62.1 | 72.5 | 43.6 | 71.3 | 80.3 |
| VIBRA (ours) | 30.3 | 52.8 | 62.4 | 20.3 | 39.3 | 48.6 | 56.2 | 84.0 | 91.1 | 41.2 | 68.0 | 77.9 | 48.7 | 76.6 | 84.8 |
| CLIP ViT-B/32 → MetaCLIP ViT-L/14 | |||||||||||||||
| Old → old | 21.8 | 41.6 | 51.1 | 14.5 | 30.2 | 38.9 | 45.3 | 75.0 | 84.6 | 32.3 | 58.8 | 70.0 | 38.7 | 66.5 | 76.5 |
| New → new | 40.9 | 63.8 | 72.3 | 26.9 | 47.3 | 56.4 | 60.9 | 86.2 | 92.3 | 48.7 | 73.6 | 82.5 | 52.4 | 78.2 | 85.7 |
| Drift-Adapter [41] | 24.5 | 45.5 | 55.0 | 15.9 | 33.0 | 42.1 | 48.7 | 78.3 | 86.9 | 35.0 | 62.0 | 73.2 | 42.4 | 71.4 | 80.6 |
| XBT [14] | 21.9‡ | 42.2‡ | 52.2‡ | 15.1‡ | 31.7‡ | 40.7‡ | 47.4‡ | 78.8‡ | 88.1‡ | 33.9‡ | 61.0‡ | 72.4‡ | 42.8‡ | 71.8‡ | 81.3‡ |
| SLERP [30] | 27.1 | 48.7 | 58.4 | 17.5 | 35.2 | 44.4 | 51.6 | 80.7 | 88.8 | 37.3 | 64.3 | 75.0 | 44.8 | 73.2 | 81.8 |
| LST [20] | 23.6 | 44.1 | 53.6 | 15.5 | 32.3 | 41.2 | 47.6 | 77.3 | 86.2 | 34.4 | 61.3 | 72.5 | 42.2 | 70.5 | 79.9 |
| λ-orth [29] | 23.8 | 44.4 | 53.9 | 15.4 | 31.9 | 40.7 | 48.3 | 77.8 | 86.7 | 34.5 | 60.9 | 71.4 | 42.4 | 71.5 | 80.4 |
| VIBRA (ours) | 28.6 | 50.8 | 60.5 | 18.9 | 37.6 | 47.0 | 54.3 | 82.8 | 90.2 | 39.6 | 66.6 | 77.2 | 48.1 | 76.5 | 84.5 |
| MetaCLIP ViT-B/32 → SigLIP2 SO400M/14 | |||||||||||||||
| Old → old | 28.0 | 49.7 | 59.1 | 19.3 | 37.6 | 47.0 | 55.9 | 83.6 | 90.7 | 40.1 | 67.0 | 77.1 | 49.9 | 76.2 | 84.3 |
| New → new | 51.5 | 73.2 | 80.4 | 35.1 | 56.5 | 65.0 | 70.1 | 91.0 | 95.1 | 56.8 | 80.2 | 87.3 | 58.8 | 82.6 | 89.2 |
| Drift-Adapter [41] | 26.8 | 48.0 | 57.5 | 18.2 | 36.3 | 45.6 | 53.3 | 81.0 | 88.8 | 38.6 | 65.2 | 75.8 | 45.9 | 73.2 | 82.5 |
| XBT [14] | 22.7‡ | 43.3‡ | 53.0‡ | 17.0‡ | 34.6‡ | 43.8‡ | 53.1‡ | 82.8‡ | 90.7‡ | 37.6‡ | 64.4‡ | 75.4‡ | 45.9‡ | 74.5‡ | 82.8‡ |
| SLERP [30] | 29.7 | 51.7 | 61.0 | 20.5 | 39.5 | 48.8 | 57.9 | 84.8 | 91.5 | 41.8 | 68.4 | 78.5 | 50.9 | 77.7 | 85.3 |
| LST [20] | 25.7 | 46.7 | 56.0 | 17.3 | 34.8 | 43.7 | 51.4 | 79.4 | 87.4 | 37.0 | 63.5 | 73.9 | 44.4 | 71.9 | 81.1 |
| λ-orth [29] | 24.0 | 44.2 | 53.7 | 16.7 | 33.7 | 42.7 | 52.6 | 80.4 | 88.5 | 36.0 | 62.1 | 73.2 | 45.8 | 73.2 | 82.1 |
| VIBRA (ours) | 30.0 | 52.1 | 61.5 | 21.0 | 40.2 | 49.6 | 58.9 | 85.4 | 92.0 | 42.5 | 68.8 | 79.0 | 51.7 | 78.4 | 85.9 |
Table S2. Text → image retrieval against the old gallery (R@1 / R@5 / R@10, %). The old gallery is never reencoded. Every method except SLERP encodes the query with one network; SLERP runs the old and the new encoder. Gray rows are references: Old → old keeps the old model, New → new reencodes the whole gallery. †published number. ‡our reproduction of XBT at the scale of its paper, used where the paper reports no number. Bold marks the best compatible method.
With weight on the extra view, the variance is a convex quadratic with , so a small positive weight lowers the variance of view 0 exactly when
.
Inheriting the consensus
Margin preservation. By the Cauchy Schwarz inequality, for every unit . Let be the item that the teacher ranks first and its margin. For every other item , , which is positive when .
Barycentric regression. For every vector , expanding the squares gives , and with unit views . For a unit , and . Eliminating gives , and substituting gives the stated identity, since .
Projection onto the student family. Let be the Hilbert space of square integrable maps under the training distribution, with . For with in a closed subspace , the first term of (4) equals . By the projection theorem, contains a unique minimizer , and is orthogonal to . By Pythagoras, , so , the coefficient of determination of the least squares fit of within . This treats the family as affine and ignores the normalization of the query; for LoRA it applies to the first order expansion .
Stage 1 Sweeps
Tables S4 and S5 give the full sweeps of the mixing weight of the stage 1 target and the weight of , summarized in Section 4.4 of the main paper.
Empirical Analysis of the Theory
Proposition 1 makes two exact claims that can be checked query by query: the barycentric margin is never below the weighted average of the view margins, and it never exceeds the ceiling given by the spread of the other views at the hardest item of view 0. Both held for every test query of every pair. Table S6 quantifies the correction. Under this protocol the barycenter of the three views raises R@1 over the new view by 1.3 to 5.3 points and rescues 6.0% to 9.4% of the queries. Close to 60% of the rescued queries, 3.4% to 5.7% of all queries, have a nonpositive average view margin and are rescued only through the dispersion gain , and 1.0% to 1.7% of all queries are ranked correctly by the barycenter although every view ranks them wrongly. All three views share their hardest irrelevant item on 7.0% to 11.6% of the queries; when such a query is ranked wrongly by every view, no weighting rescues it. Splitting the queries by this condition confirms the proposition: on the queries with a shared hardest item the measured is zero up to numerical precision (at most ) and none of those that every view ranks wrongly is rescued, whereas is positive on more than 99.9% of the remaining queries, where the barycenter corrects 10.0% to 13.4% of the errors of the new view, against 4.0% to 4.5% on the queries with a shared hardest item.
The noise model of Section 3.3 of the main paper is estimated on CC3M DEV only. In a preregistered test over 40 combinations of a model pair and a candidate view, its predicted gain ranks the views by their measured test gain (Spearman 0.38, one sided permutation ; 0.61 on average within a pair), whereas the correlation alone does not (0.08). The same model overestimates the absolute size of the gains by a factor of three to four, because it treats the noise of different irrelevant items as independent while the errors of the views concentrate on a few shared hard items, the situation in which Proposition 1 predicts no dispersion gain. Training the student on the teacher with the derived weights gives image → text R@1 on par with a fixed rule without estimation, the nested midpoint with vector weights of roughly : over the 25 combinations of pair and set, the mean difference is points (mean over three seeds), positive in 12.
Full Ablation
The full ablation adds the head alone, the single views, and the teachers and students of two views to the ablation of the main paper (Table S7). Each part of the stage 1 query contributes: the Procrustes map alone is 2.5 to 10.9 points below and the head alone 1.5 to 4.7 points. The two estimates of relevance are both needed (Table S4): the label alone () is 1.2 to 4.1 points below and the ranking of the new model alone () 0.6 to 1.7 points below.
The barycenter of three views is the strongest target, and the auxiliary term carries the third view into the student. The teacher of three views improves on by 1.1 to 4.5 points and on the teacher of two views by 0.4 to 1.4 points, and every teacher exceeds each of its views. Without the auxiliary term, the student of three views () is within 0.5 points of the student of two views, although its teacher is better, and adding it gives 0.1 to 1.0 points. On MetaCLIP B/32 SO400M, where the teacher gains least, the student of two views is 0.3 points ahead of VIBRA.
Training Cost
VIBRA trains 19 to 27 times faster than XBT, the other method that trains the new encoder for compatibility across modalities (Table S8). VIBRA needs 0.22 to 0.71 A100 hours, spent on encoding its training data with the old, new and auxiliary encoders and on the student, while XBT at the scale of its paper needs 4.2 to 18.9 hours. Query side maps fit in minutes on cached features.
Training VIBRA costs as much as reencoding 1.1M to 2.2M images, and this cost does not grow with the gallery. A full backfill with the new model costs 101 to 506 A100 hours per billion images.
Update Gain
VIBRA obtains more than half of the gain of reencoding the gallery without reencoding it (Table S9). We report the update gain of BCT [36], the improvement of a method over Old old divided by the improvement of New new, on the mean R@1 over the five pairs and five datasets, so that 0% keeps the old model and 100% matches reencoding. VIBRA reaches 55.0%, with 68.4% from image to text and 43.3% from text to image.
Table S9. Update gain [36] (%), the share of the gain of reencoding that a method obtains without reencoding, over all pairs and datasets.
Methods with one encoder obtain almost none of this gain, and SLERP obtains less with two encoders. Query side maps and XBT range from to 6.8%, as expected from methods that fit the old query, and SLERP reaches 36.1% while it runs the old encoder for every query. The gap is largest from text to image, where VIBRA obtains 43.3% and SLERP 19.9%.
Table S6. Consensus correction on the test sets, extending Table 2 of the main paper (image text, one designated positive caption per image, other positives removed from the negatives; mean over the five sets, %). Rescued: the barycenter ranks the positive first while the new view does not. Needing : rescued although the weighted average of the view margins is not positive. All views wrong: rescued although every view ranks the positive lower. Shared: all three views have the same hardest irrelevant item.
| Pair | New view | Barycenter | Ceiling | Rescued | Needing | All views wrong | Shared |
| B/32 L/14 | 23.0 | 27.7 | 38.0 | 8.0 | 4.7 | 1.2 | 8.7 |
| B/32 B/16 | 22.9 | 27.5 | 37.1 | 7.6 | 4.4 | 1.1 | 9.9 |
| B/32 SO400M | 24.3 | 29.5 | 44.3 | 9.4 | 5.7 | 1.7 | 7.0 |
| B/32 MetaCLIP L/14 | 23.0 | 28.3 | 40.8 | 9.0 | 5.2 | 1.4 | 8.0 |
| MetaCLIP B/32 SO400M | 32.3 | 33.6 | 40.6 | 6.0 | 3.4 | 1.0 | 11.6 |
References
- [1]Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8947–8956. IEEE, 2019. 6DOI
- [2]John M Bates and Clive WJ Granger. The combination of forecasts. Journal of the operational research society, 20(4): 451–468, 1969. 3, 5
- [3]Niccolò Biondi, Federico Pernici, Matteo Bruni, and Alberto Del Bimbo. Cores: Compatible representations via stationarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):9567–9582, 2023. 2DOI
- [4]Niccolò Biondi, Federico Pernici, Simone Ricci, and Alberto Del Bimbo. Stationary representations: Optimally approximating compatibility and implications for improved model replacements. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28793–28804. IEEE, 2024. 2
- [5]Jaeseok Byun, Gukyeong Kwon, Han-Kai Hsu, Meher Gitika Karumuri, Zhikang Zhang, Hao Yang, and Davide Modolo. Multi-modal knowledge preserving adapter for embedding backward compatibility. In European Conference on Computer Vision, pages 473–490. Springer, 2026. 2arxiv.org/abs/2609.16875
- [6]Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th International Conference on Machine Learning, page 129–136, New York, NY, USA, 2007. Association for Computing Machinery. 4DOI
- [7]Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2829. IEEE, 2023. 5DOI
- [8]Rahul Duggal, Hao Zhou, Shuo Yang, Yuanjun Xiong, Wei Xia, Zhuowen Tu, and Stefano Soatto. Compatibility-aware heterogeneous visual search. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10718–10727, 2021. 2arxiv.org/abs/2105.06047
- [9]Dipam Goswami, Liying Wang, Bartlomiej Twardowski, and Joost van de Weijer. Query drift compensation: Enabling compatibility in continual learning of retrieval embedding models. arXiv preprint arXiv:2506.00037, 2025. 2arxiv.org/abs/2506.00037
- [10]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 3arxiv.org/abs/1503.02531
- [11]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 5arxiv.org/abs/2106.09685
- [12]Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis, 2024. 2arxiv.org/abs/2405.07987
- [13]Florian Jaeckle, Fartash Faghri, Ali Farhadi, Oncel Tuzel, and Hadi Pouransari. Fastfill: Efficient compatible model update. arXiv preprint arXiv:2303.04766, 2023. 2arxiv.org/abs/2303.04766
- [14]Young Kyun Jang and Ser-nam Lim. Towards cross-modal backward-compatible representation learning for vision-language models. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1783–1792. IEEE, 2025. 1, 2, 6, 7, 13, 14, 17arxiv.org/abs/2405.14715
- [15]Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR, 2021. 1arxiv.org/abs/2102.05918
- [16]Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE transactions on big data, 7(3):535–547, 2019. 1arxiv.org/abs/1702.08734
- [17]Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. Advances in neural information processing systems, 7, 1994. 3
- [18]Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. Advances in neural information processing systems, 34:9694–9705, 2021. 3
- [19]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 6
- [20]Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. Latent space translation via semantic alignment. Advances in Neural Information Processing Systems, 36:55394–55414, 2023. 1, 2, 6, 7, 13, 14, 17arxiv.org/abs/2311.00664
- [21]Qiang Meng, Chixiang Zhang, Xiaoqiang Xu, and Feng Zhou. Learning compatible embeddings. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9919–9928. IEEE, 2021. 2
- [22]Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430, 2022. 2arxiv.org/abs/2209.15430
- [23]Tan Pan, Furong Xu, Xudong Yang, Sifeng He, Chen Jiang, Qingpei Guo, Feng Qian, Xiaobo Zhang, Yuan Cheng, Lei Yang, et al. Boundary-aware backward-compatible representation via adversarial learning in image retrieval. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15201–15210. IEEE, 2023. 2
- [24]Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Waters, and Yinfei Yang. Crisscross captions: Extended intramodal and intermodal semantic similarity judgments for ms-coco. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2855–2870, 2021. 6
- [25]Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3962–3971. IEEE, 2019. 3arxiv.org/abs/1904.05068
- [26]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 1, 6
- [27]Vivek Ramanujan, Pavan Kumar Anasosalu Vasu, Ali Farhadi, Oncel Tuzel, and Hadi Pouransari. Forward compatible training for large-scale embedding retrieval systems. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19364–19373. IEEE, 2022. 1, 2arxiv.org/abs/2112.02805
- [28]Simone Ricci, Niccolò Biondi, Federico Pernici, and Alberto Del Bimbo. Backward-compatible aligned representations via an orthogonal transformation layer. In European Conference on Computer Vision, pages 451–464. Springer, 2024. 2
- [29]Simone Ricci, Niccolò Biondi, Federico Pernici, Ioannis Patras, and Alberto Del Bimbo. λ-orthogonality regularization for compatible representation learning. Advances in Neural Information Processing Systems, 38:29036–29063, 2025. 1, 2, 6, 7, 13, 14, 17DOI
- [30]Simone Ricci, Niccolò Biondi, and Federico Pernici. Spherical interpolation for backward-compatible multimodal representations. arXiv preprint arXiv:2609.39836, 2026. 1, 2, 6, 7, 13, 14, 17arxiv.org/abs/2609.39836
- [31]Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, 2015. 3DOI
- [32]Peter H. Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10, 1966. 4DOI
- [33]Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: an open large-scale dataset for training next generation image-text models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2022. Curran Associates Inc. 5
- [34]Seonguk Seo, Mustafa Gokhan Uzunbas, Bohyung Han, Sara Cao, Joena Zhang, Taipeng Tian, and Ser-Nam Lim. Online backfilling with no regret for large-scale image retrieval. arXiv preprint arXiv:2301.03767, 4, 2023. 2
- [35]Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018. 6DOI
- [36]Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward-compatible representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6367–6376. IEEE, 2020. 1, 2, 7, 16, 17arxiv.org/abs/2003.11942
- [37]Jiaxi Tang and Ke Wang. Ranking distillation: Learning compact ranking models with high performance for recommender system. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2289–2298, 2018. 3
- [38]Ashish V Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 715–729, 2022. 6
- [39]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdoulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. 1, 6arxiv.org/abs/2502.14786
- [40]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019. 4arxiv.org/abs/1807.03748
- [41]Harshil Vejendla. Drift-adapter: A practical approach to near zero-downtime embedding model upgrades in vector databases. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15949–15960, 2025. 1, 2, 6, 7, 13, 14, 17arxiv.org/abs/2509.23471
- [42]Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang, et al. Tyniclip: Clip distillation via affinity mimicking and weight inheritance. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 21913–21923. IEEE, 2023. 3arxiv.org/abs/2309.12314
- [43]Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In International Conference on Learning Representations, pages 47812–47831, 2024. 1, 6arxiv.org/abs/2309.16671
- [44]Chuangchuang Yang, Zhulin An, Libo Huang, Junyu Bi, Xinqiang Yu, Han Yang, Boyu Diao, and Yongjun Xu. Clip-kd: An empirical study of clip model distillation. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15952–15962. IEEE, 2024. 3arxiv.org/abs/2307.12732
- [45]Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1285–1294, 2017. 3DOI
- [46]Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2:67–78, 2014. 6
- [47]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941–11952, 2023. 6arxiv.org/abs/2303.15343
- [48]Binjie Zhang, Yixiao Ge, Yantao Shen, Yu Li, Chun Yuan, Xuyuan Xu, Yexin Wang, and Ying Shan. Hot-refresh model upgrades with regression-alleviating compatible training in image retrieval. arXiv preprint arXiv:2201.09724, 2022. 2arxiv.org/abs/2201.09724
- [49]Binjie Zhang, Yixiao Ge, Xuyuan Xu, Ying Shan, and Mike Zheng Shou. Taca: Upgrading your visual foundation model with task-agnostic compatible adapter. arXiv preprint arXiv:2306.12642, 2023. 2arxiv.org/abs/2306.12642
- [50]Binjie Zhang, Shupeng Su, Yixiao Ge, Xuyuan Xu, Yexin Wang, Chun Yuan, Mike Zheng Shou, and Ying Shan. Darwinian model upgrades: Model evolving with selective compatibility. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3393–3400, 2023. 2arxiv.org/abs/2210.06954
- [51]Yifei Zhou, Zilu Li, Abhinav Shrivastava, Hengshuang Zhao, Antonio Torralba, Taiping Tian, and Ser-Nam Lim. Bt^2: Backward-compatible training with basis transformation. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11195–11204. IEEE, 2023. 1, 2arxiv.org/abs/2211.03989



