betaFeedback

WorksComputer Science

Permute-then-Adapt: Weak-to-Strong Contrastive Image-Text Adaptation

Abstract

Image-text alignment models such as CLIP are typically trained with contrastive learning, where unpaired examples are treated as strict negatives. While this assumption introduces noise by penalising semantically related pairs, existing solutions often rely on heuristic similarity thresholds that lack statistical grounding and sensitivity to dataset-specific noise distributions. In this paper, we propose Permute-then-Adapt (PTA), a framework that leverages efficient weak models to identify missed positives while controlling Type-I error relative to a conditional permutation reference, up to calibration approximation error. Unlike standard distillation or arbitrary thresholding, PTA estimates a random-pairing reference distribution from the weak teacher's similarity scores and uses its upper tail, determined by αα, to select auxiliary positives. The calibrated positive set is then trained against by a single multi-positive contrastive objective: each anchor maximises the total probability mass it assigns to its positive set. Offline calibration with the weak model adds negligible training overhead. Experiments show gains over heuristic soft-label approaches in object recognition, cross-modal retrieval, and data efficiency. Project repository: https://github.com/JinhaoLee/Permute-then-Adapt.

Introduction

Recent advances in multimodal learning [37, 19, 32, 48, 7, 51, 42] have demonstrated the immense potential of image-text alignment models for understanding and connecting visual and textual information. Models like CLIP [37] have shown remarkable capabilities in zero-shot recognition [6], cross-modal retrieval [30], and serving as foundational components for larger multimodal systems [44]. These models achieve their performance through contrastive learning on large-scale image-text pairs, where the core training objective is to maximise similarity between matched pairs while minimising similarity between unmatched ones.

However, the assumption that all unmatched pairs are completely unrelated is a fundamental limitation in contrastive learning, as it forces the model to treat semantically valid overlaps as false negatives [4, 47]. Prior work uses similarity-based filtering to curate image–text training data [7], or learns soft alignment targets through teacher supervision and momentum distillation [18, 28]. These strategies motivate dataset-adaptive calibration of auxiliary-positive selection. A fixed threshold is inherently brittle: it fails to adapt to the varying signal-to-noise ratios across different datasets or model architectures, leading to either missed positives or the introduction of noise. This reveals a critical gap: the lack of a mechanism to distinguish potential signals from the null distribution of random pairings without manual tuning. As shown in Figure 1, a fixed threshold that works for a general domain like COCO fails catastrophically on fine-grained tasks (e.g., Stanford Cars), accepting 12× more noise due to the domain’s naturally higher baseline similarities. We address this by proposing a statistically grounded framework. Our method uses permutation testing to estimate the conditional random-pairing null and select auxiliary positives with Type-I error control relative to that reference, up to calibration approximation error.

Figure 1. The risk of uncalibrated heuristics. While a fixed threshold (red dashed) works on COCO (Left), it fails on fine-grained Stanford Cars (Right), where the random-pairing null distribution shifts rightward. The fixed threshold accepts 12× more false positives from the null on Cars. In contrast, PTA (green solid) estimates a dataset-specific permutation null and adapts its cutoff to control Type-I error relative to that reference at level α\alpha, up to calibration approximation error.

In this paper, we propose a framework for enhancing image-text alignment by leveraging the semantic knowledge of efficient weak models. Our motivation stems from the insight that weak-to-strong generalisation [3] (e.g., smaller CLIP variants), despite their limitations in zero-shot accuracy, encode rich signals about semantic relatedness that are computationally cheap to access. However, raw similarity scores are noisy; relying on a single fixed threshold to extract them is brittle and fails to generalise across domains. To rigorously separate this signal from noise, we introduce Permute-then-Adapt (PTA), a weak-to-strong framework that replaces heuristic thresholding with statistical hypothesis testing. Inspired by the permutation test [15], PTA estimates a dataset-specific reference distribution of weak-teacher similarities under random image–text pairings and selects auxiliary positives whose scores exceed its calibrated upper-tail threshold. This approach requires no additional validation data and adapts naturally to the intrinsic noise floor of different datasets. The resulting calibrated positive set is then trained against by a single multi-positive contrastive objective: for each anchor we maximise the total probability mass assigned to its positive set, where the positive set is the ground-truth diagonal augmented with the calibrated off-diagonal pairs. We validate our approach through extensive experiments across image classification and retrieval benchmarks, demonstrating that our statistically grounded guidance consistently outperforms heuristic baselines.

Our main contributions can be summarised as follows: (i) Permutation-calibrated auxiliary positive selection. We use semantically irrelevant permutations to calibrate a dataset-adaptive cutoff, providing Type-I error control relative to the conditional random-pairing reference at significance level α\alpha, up to Monte Carlo and percentile approximation error. (ii) PTA multi-positive objective. We train against the calibrated set with a single positive-set marginal-likelihood loss. (iii) Through extensive experiments on multiple zero-shot image classification benchmarks and two cross-modal retrieval datasets, we show that PTA consistently outperforms existing weakly supervised baselines and tolerates moderate inference-time teacher dropout.

Related work

Vision-language models. Since the introduction of CLIP [37], numerous models building upon its multimodal representation learning framework have emerged. Subsequent work has explored architectural improvements and training-recipe refinements, such as ALIGN [19] which scales up the training data and model size, EVA-CLIP [41] which improves the training stability and efficiency of large-scale CLIP through optimised initialisations and learning rates, and Florence [49] which incorporates additional visual reasoning capabilities. BLIP [29] uses caption bootstrapping for unified vision–language understanding and generation, while LiT [50] trains a text encoder against a locked pretrained image encoder. FG-CLIP [46] explicitly targets fine-grained visual–textual alignment. PTA is orthogonal to this line of work: rather than scaling the model or replacing the backbone, it provides a post-hoc refinement that exploits a calibrated, dataset-adaptive selection of auxiliary positives and could in principle be applied on top of any of these architectures.

CLIP fine-tuning. Recent advancements in fine-tuning CLIP models have led to various parameter-efficient methods aimed at enhancing performance on downstream tasks. The Context Optimization (CoOp) [55] approach introduces learnable text prompts, enabling the model to adapt to specific tasks by optimizing these prompts during training. Building upon this, Conditional Prompt Learning (CoCoOp) [54] extends CoOp by conditioning the prompts on image instances, thereby improving generalisation across diverse visual concepts. Alternatively, CLIP-Adapter [9] incorporates lightweight residual-style adapters into CLIP’s architecture, facilitating the learning of task-specific features without extensive retraining. Tip-Adapter [53] further streamlines this process by constructing a key-value cache model from the few-shot training set, enabling efficient adaptation with minimal or no training. Additionally, methods like Visual Prompt Tuning (VPT) [20] add learnable prompt tokens in the input space, fine-tuning the model effectively for various tasks. These strategies aim to improve CLIP’s adaptability and performance while maintaining computational efficiency.

Bootstrapping and pseudo-labelling. Several works attempt to mitigate noise in image-text alignment through bootstrapping or pseudo-labelling. BLIP [29] employs caption bootstrapping and multi-stage training to refine noisy web-crawled supervision. More recent studies propose pseudo-labelling strategies to relax one-hot supervision, such as Cross-Modal Similarity Regulation [21], multi-label pseudo-labelling [47], and soft-label alignment [18]. These methods highlight the importance of leveraging softer or multiple alignment signals. Our approach shares this motivation but differs by introducing a statistically grounded mechanism (permutation testing) to identify semantically meaningful extra positives with controlled false alarm rates.

Knowledge distillation and weak-to-strong supervision. Knowledge distillation techniques, such as ALBEF [28], employ momentum encoders to refine alignment signals from auxiliary models. Weak-to-strong transfer has also been studied more broadly [3], emphasising how weaker models guide stronger ones. For CLIP classification, CPL [27] learns class prototypes under weak-model supervision. PTA builds upon this weak-to-strong philosophy but departs from direct distillation: instead of transferring continuous similarity scores, it adaptively thresholds weak-model signals using a permutation-derived null distribution, producing statistically validated supervision.

Mitigating false negatives. False negatives, i.e. unpaired image–text instances that are semantically related, have recently received direct attention in vision–language pretraining. For instance, MAFA [4] proposes an efficient connection mining process to convert some hard false negatives into positives, and smooths the contrastive ITC loss to reduce over-penalisation. Empirically, MAFA shows that doing this yields substantial downstream gains, especially under hard-negative sampling strategies. PTA distinguishes itself by employing a permutation-based null distribution to statistically validate and promote high-scoring unpaired tuples as auxiliary positives for weak-to-strong supervision, and by reusing the same calibrated mask to clean false negatives from the contrastive loss. Related approaches mitigate noisy supervision by modifying contrastive objectives or learning softer alignment targets. PyramidCLIP [10], SoftCLIP [11], and CLIP-PSD [1] follow this direction. PTA calibrates auxiliary-positive selection against a permutation reference. AdaCL [34] focuses on clone negatives captions that are semantically close but not exact matches by adaptively reshaping the positive logit, but still treats all non-matching pairs as negatives. NCU [14] studies post-hoc unlearning of noisy correspondences in pretrained CLIP, addressing both false positives and false negatives, and can operate with a subset of the original pretraining data. We instead study post-hoc weak-to-strong refinement on a small, clean downstream dataset with correct positives, where in-batch sampling introduces hard false negatives into the contrastive loss.

Preliminaries

Dual encoder models. Dual encoder models [37] belong to a class of neural architectures designed to learn representations of two different modalities in a shared embedding space. These models consist of two separate encoders that process different input types (e.g., images and text) and project them into a common semantic space where similarity can be directly computed. Formally, we define: an image encoder fimage:X→Rdf_{\mathrm{image}}:\mathcal{X}\to\mathbb{R}^{d} that maps images to a dd-dimensional embedding, and a text encoder ftext:Z→Rdf_{\mathrm{text}}:\mathcal{Z}\to\mathbb{R}^{d} that maps text to the same dd-dimensional space.

Image-text alignment. Image-text alignment aims to capture semantic correspondences between visual and textual content [37, 26]. Given a dataset D={(Ii,Ti)}i=1N\mathcal{D}=\{(I_i,T_i)\}_{i=1}^{N} of image-text pairs, where Ii∈XI_i\in\mathcal{X} represents an image and Ti∈ZT_i\in\mathcal{Z} represents its corresponding text description, the goal is to learn a mapping that reflects semantic similarity in a shared embedding space. For notational brevity, we denote the similarity score assigned by a model ff to an image-text pair (Ii,Tj)(I_i,T_j) as the inner product of their normalised embeddings: f(Ii,Tj)=⟨fimage(Ii),ftext(Tj)⟩f(I_i,T_j)=\langle f_{\mathrm{image}}(I_i),f_{\mathrm{text}}(T_j)\rangle.

We formulate the alignment task using contrastive learning on the full dataset D\mathcal{D}. The objective aligns embeddings by maximising the similarity score f(Ii,Ti)f(I_i,T_i) for matched pairs while minimising it for mismatched pairs (e.g., IiI_i and TjT_j where i≠ji\ne j).

To ensure proper gradient scaling, a learnable temperature parameter τ\tau is introduced. For each anchor ii, the image-to-text and text-to-image contrastive losses are defined as:

Li2t(i)=−log⁡exp⁡(f(Ii,Ti)/τ)∑j=1Nexp⁡(f(Ii,Tj)/τ),Lt2i(i)=−log⁡exp⁡(f(Ii,Ti)/τ)∑j=1Nexp⁡(f(Ij,Ti)/τ).(1)\mathcal{L}_{i2t}^{(i)}=-\log\frac{\exp(f(I_i,T_i)/\tau)}{\sum_{j=1}^{N}\exp(f(I_i,T_j)/\tau)},\qquad \mathcal{L}_{t2i}^{(i)}=-\log\frac{\exp(f(I_i,T_i)/\tau)}{\sum_{j=1}^{N}\exp(f(I_j,T_i)/\tau)}. \tag*{(1)}

The total loss

Lclip=12N∑i=1N(Li2t(i)+Lt2i(i))(2)\mathcal{L}^{\mathrm{clip}}=\frac{1}{2N}\sum_{i=1}^{N}\left(\mathcal{L}_{i2t}^{(i)}+\mathcal{L}_{t2i}^{(i)}\right) \tag*{(2)}

is minimised over D\mathcal{D} to align embeddings, with negative pairs enforcing discriminative separation.

Image-text alignment with weak models

We introduce our method to improve a strong model’s image-text alignment using weak models.

Our method: Permute-then-Adapt

In this section, we will discuss the problem setup and our method. Figure 2 demonstrates our method visually. The overall algorithm can be found in Algorithm 1.

Figure 2. Overview of Permute-then-Adapt (PTA). Given paired image–text samples, PTA uses a frozen weak teacher to estimate a permutation null distribution and dataset-adaptive threshold. This threshold selects calibrated off-diagonal auxiliary positives, which join ground-truth diagonal pairs to form positive sets. The student is optimized with a positive-set marginal likelihood objective.

A key limitation of standard contrastive learning (Equation (2)) is its hard-negative assumption: all non-paired tuples (Ii,Tj)(I_i,T_j) with i≠ji \ne j are treated as negatives, even though some may be semantically related. A natural idea is to threshold teacher ff similarity scores to convert such tuples into auxiliary positives; however, fixed cutoffs [7] are brittle across datasets and model architectures.

Problem setup. We consider post-hoc refinement of a pretrained dual-encoder (the student fsf^s, also denoted the strong model) on a paired dataset D\mathcal{D}. Given a frozen, lightweight teacher ff (also denoted the weak model3), our goal is to identify a subset of non-paired tuples (Ii,Tj)(I_i,T_j) with i≠ji \ne j as auxiliary positives and use them to enhance the image-text alignment of the student fsf^s.

Weak teacher similarities. A data-calibrated threshold QQ defines pseudo-labels for non-paired tuples. Standard contrastive learning encodes supervision via the dataset pairing indicator

lD(Ii,Tj):=I[i=j],(3)l_{\mathcal{D}}(I_i,T_j) := \mathbb{I}[i=j], \tag*{(3)}

Unpaired tuples receive the hard label lD=0l_{\mathcal{D}}=0, which we replace with a weak pseudo-label lf(⋅,⋅;Q)l_f(\cdot,\cdot;Q):

lf(I,T;Q)={1if f(I,T)>Q,0otherwise,(4)l_f(I,T;Q) = \begin{cases} 1 & \text{if } f(I,T)>Q,\\ 0 & \text{otherwise,} \end{cases} \tag*{(4)}

where ff is a frozen weak teacher that scores arbitrary image–text pairs, and QQ selects high-scoring tuples as auxiliary positives. Assigning the weak pseudo-label lf(I,T;Q)l_f(I,T;Q) gives the adaptation set

Dadaptf(Q)={(Ii,Tj,lf(Ii,Tj;Q))}i,j=1N.(5)\mathcal{D}^{f}_{\mathrm{adapt}}(Q)=\{(I_i,T_j,l_f(I_i,T_j;Q))\}_{i,j=1}^{N}. \tag*{(5)}

We can then fine-tune the student using Dadaptf(Q)\mathcal{D}^{f}_{\mathrm{adapt}}(Q)4. A key difficulty is that Dadaptf(Q)\mathcal{D}^{f}_{\mathrm{adapt}}(Q) is highly sensitive to the choice of QQ. Fixed similarity thresholds [7] are not comparable across datasets or model architectures: the same numeric value of QQ may correspond to very different signal-to-noise ratios, leading to either missed auxiliary positives or excessive noise (as shown in Figure 1). To make selection principled and dataset-adaptive, we calibrate QQ via permutation testing at significance level α\alpha, providing Type-I error control relative to the conditional permutation reference, up to Monte Carlo and percentile approximation error [15].

Permute: estimating the null distribution

Permutation-based null. Selecting QQ requires a reference distribution of similarity scores for irrelevant image–text pairs. Under the independence null H0:I⊥TH_0 : I \perp T, text permutation preserves the observed marginals but breaks associations, yielding a valid (conditional) permutation null [13, 15].

Interpretation. The permutation null is intentionally not a generative model of hard semantic negatives, and our framework does not require that real images and texts are independent. Instead, the null serves as an empirical reference distribution of weak-similarity scores under random mismatching, evaluated on the same dataset and the same weak teacher used downstream. PTA then promotes off-diagonal pairs whose weak similarity is implausibly large relative to this reference; semantically related hard negatives, which would otherwise be penalised by the standard contrastive loss, are precisely the right-tail deviations we wish to rescue from strict negation. The null thus provides a conservative, dataset-adaptive baseline without assuming that semantic clustering is absent.

Semantically irrelevant perturbations. Shuffling text in the paired dataset D={(Ii,Ti)}i=1N\mathcal{D}=\{(I_i,T_i)\}_{i=1}^{N} yields an empirical permutation null sample of weak similarity scores. Let GN\mathcal{G}_N denote the permutation group on [N]={1,…,N}[N]=\{1,\ldots,N\}, with elements σ:[N]→[N]\sigma:[N]\to[N] acting on the text tuple T=(T1,…,TN)T=(T_1,\ldots,T_N) via σT:=(Tσ(1),…,Tσ(N))\sigma T:=(T_{\sigma(1)},\ldots,T_{\sigma(N)}). Under semantically irrelevant image–text permutations, we draw RR random permutations {σr}r=1R∼Unif⁡(GN)\{\sigma_r\}_{r=1}^{R}\sim\operatorname{Unif}(\mathcal{G}_N) and collect the resulting null similarity scores:

Sperm:={f(Ii,Tσr(i)):i∈[N], r∈[R]}.(6)\mathcal{S}_{\mathrm{perm}} := \{f(I_i,T_{\sigma_r(i)}) : i\in[N],\ r\in[R]\}. \tag*{(6)}

This empirical permutation null sample retains repeated scores; we threshold auxiliary positives with Type-I error control relative to that reference, up to calibration approximation error.

Proposition 4.1 (Validity of the permutation null [12, 24, 15]). Under the independence null H0:I⊥TH_0: I \perp T, suppose text assignments are exchangeable conditional on the observed image and text marginals M\mathcal{M}. All matchings {(Ii,Tσ(i))}i=1N\{(I_i,T_{\sigma(i)})\}_{i=1}^{N} are then equally likely under H0H_0. For a fixed teacher ff, a uniformly sampled matching and image index yield the conditional random-pairing score CDF

F0(t)=1N2∑i,j=1N1{f(Ii,Tj)≤t}.F_0(t) = \frac{1}{N^2}\sum_{i,j=1}^{N}\mathbf{1}\{f(I_i,T_j)\le t\}.

Independent uniform text permutations provide a Monte Carlo approximation to this reference distribution. When calibration uses a subset, apply the statements below with N=NcalN=N_{\mathrm{cal}} and M\mathcal{M} denoting its observed marginals.

Conditional permutation testing provides the theoretical background [24]; the bound below explicitly accounts for calibration approximation error.

Definition 4.2 (Permutation-Calibrated Selection). Given a significance level α∈(0,1)\alpha\in(0,1), define

QPTA:=percentile⁡1−α(Sperm).Q_{\mathrm{PTA}} := \operatorname{percentile}_{1-\alpha}(S_{\mathrm{perm}}).

The selection rule assigns a pseudo-label according to

ℓfp(I,T):={1if f(I,T)>QPTA,0otherwise.(7)\ell_f^p(I,T) := \begin{cases} 1 & \text{if } f(I,T)>Q_{\mathrm{PTA}},\\ 0 & \text{otherwise}. \end{cases} \tag*{(7)}

Corollary 4.3 (Type-I error control). Condition on M\mathcal{M}, the fixed teacher ff, and the sampled permutations Π\Pi. Let J,KJ,K be independent uniform indices in [N][N], independent of Π\Pi, so that (IJ,TK)(I_J,T_K) is a random pair from the same observed marginals. Write Q=QPTAQ=Q_{\mathrm{PTA}}, and let F^R\widehat{F}_R be the empirical CDF of SpermS_{\mathrm{perm}}. Define the finite-calibration approximation error

εcal:=∥F^R−F0∥∞+[1−F^R(Q)−α]+,\varepsilon_{\mathrm{cal}} := \lVert\widehat{F}_R-F_0\rVert_{\infty}+\left[1-\widehat{F}_R(Q)-\alpha\right]_+,

where [x]+=max⁡{x,0}[x]_+=\max\{x,0\}. Then the Type-I error relative to the conditional random-pairing reference satisfies

Pr⁡J,K(ℓfp(IJ,TK)=1∣M,f,Π)≤min⁡{1,α+εcal}.(8)\Pr_{J,K}\left(\ell_f^p(I_J,T_K)=1\mid\mathcal{M},f,\Pi\right)\le\min\{1,\alpha+\varepsilon_{\mathrm{cal}}\}. \tag*{(8)}

Indeed, the left-hand side equals 1−F0(Q)1-F_0(Q), which is at most 1−F^R(Q)+∥F^R−F0∥∞1-\widehat{F}_R(Q)+\lVert\widehat{F}_R-F_0\rVert_{\infty}. The first error term is the Monte Carlo reference approximation; the second accounts for percentile discretisation or interpolation and is zero for a generalised-inverse empirical quantile.

Thus, α\alpha specifies the tolerated Type-I error relative to the conditional permutation reference, up to the stated calibration approximation error. This reference control does not assert semantic precision of the selected auxiliary positives.

Adapt: enhance image-text alignment

Following this selection procedure, we collect the permutation-calibrated pseudo-labels into a training set of additional positives for fine-tuning the strong model. Let K⊂[N]\mathcal{K}\subset[N] be a set of indices with ∣K∣=B|\mathcal{K}|=B defining a mini-batch. We use paired samples from D\mathcal{D} together with off-diagonal PTA-labelled pairs mined within the batch:

Dbatch:={(Ik,Tk):k∈K},DbatchPTA:={(Ik,Tm,ℓfp(Ik,Tm)):k,m∈K, k≠m}.(9)\mathcal{D}_{\mathrm{batch}}:=\{(I_k,T_k):k\in\mathcal{K}\},\quad\mathcal{D}_{\mathrm{batch}}^{\mathrm{PTA}}:=\{(I_k,T_m,\ell_f^p(I_k,T_m)):k,m\in\mathcal{K},\ k\ne m\}. \tag*{(9)}

By construction, DbatchPTA⊂Dadaptf(Q)\mathcal{D}_{\mathrm{batch}}^{\mathrm{PTA}}\subset\mathcal{D}_{\mathrm{adapt}}^f(Q). Define the PTA indicator for an in-batch pair as

ℓkmPTA:=ℓfp(Ik,Tm)∈{0,1},(k≠m),(10)\ell_{km}^{\mathrm{PTA}}:=\ell_f^p(I_k,T_m)\in\{0,1\},\qquad(k\ne m), \tag*{(10)}

and set ℓkkPTA:=0\ell_{kk}^{\mathrm{PTA}}:=0 for completeness. We then form the final target mask

ℓ~kmPTA:=I[k=m]∨ℓkmPTA,ℓ~kmPTA∈{0,1}.(11)\widetilde{\ell}_{km}^{\mathrm{PTA}}:=\mathbb{I}[k=m]\vee\ell_{km}^{\mathrm{PTA}},\qquad\widetilde{\ell}_{km}^{\mathrm{PTA}}\in\{0,1\}. \tag*{(11)}

Algorithm 1 Permute-then-Adapt (PTA) Training Framework

  1. Input: Teacher ff, Student fsf^{s}, Dataset D\mathcal{D}, significance level α\alpha.

  2. Stage 1: Calibration (Offline)

  3. Sample NcalN_{\mathrm{cal}} image-text pairs from D\mathcal{D} and apply RR random text permutations to obtain the empirical null sample Sperm\mathcal{S}_{\mathrm{perm}}.

  4. Determine threshold QPTA←percentile⁡1−α(Sperm)Q_{\mathrm{PTA}} \leftarrow\operatorname{percentile}_{1-\alpha}(\mathcal{S}_{\mathrm{perm}}).

  5. Stage 2: Adaptation (Online)

  6. for mini-batch B={(Ik,Tk)}k=1B\mathcal{B} = \{(I_k,T_k)\}_{k=1}^{B} do

  7. // Calibrated positive set

  8. Compute teacher similarity matrix Sweak∈RB×B\mathbf{S}_{\mathrm{weak}} \in\mathbb{R}^{B \times B}.

  9. For each anchor kk, form Pk←{k}∪{m≠k:Sweak(Ik,Tm)>QPTA}\mathcal{P}_k \leftarrow\{k\} \cup\{m \ne k : S_{\mathrm{weak}}(I_k,T_m) > Q_{\mathrm{PTA}}\} (Eq. (12)).

  10. // Single-equation multi-positive contrastive loss

  11. Compute LPTA\mathcal{L}^{\mathrm{PTA}}: for each anchor, the negative log of the student’s total probability mass on Pk\mathcal{P}_k (Eq. (13)); standard CLIP InfoNCE is recovered when Pk={k}\mathcal{P}_k = \{k\}.

  12. Update Student: θ←θ−η∇θLPTA\theta\leftarrow\theta- \eta\nabla_{\theta}\mathcal{L}^{\mathrm{PTA}}.

  13. end for

That is, each anchor always includes its ground-truth match (k=mk = m) and includes additional in-batch positives selected by PTA via the permutation-calibrated threshold. In contrast to standard CLIP supervision, which treats only the diagonal pair as positive, PTA promotes a selected subset of off-diagonal in-batch pairs to positives.

Positive-set marginal likelihood objective. PTA trains the student with a positive-set marginal likelihood [8, 52]: each anchor maximises the total probability mass it assigns to its calibrated positive set. For an image anchor IkI_k with positive set

Pk={k}∪{m≠k:ℓkmPTA=1},(12)\mathcal{P}_k = \{k\} \cup\{m \ne k : \ell^{\mathrm{PTA}}_{km} = 1\}, \tag*{(12)}

the image-to-text loss is

ℓi2t(k)=−log⁡∑m∈Pkexp⁡(fs(Ik,Tm)/τ)∑m′=1Bexp⁡(fs(Ik,Tm′)/τ),(13)\ell^{(k)}_{\mathrm{i2t}} = -\log\frac{\sum_{m \in\mathcal{P}_k} \exp(f^{s}(I_k,T_m)/\tau)}{\sum_{m'=1}^{B} \exp(f^{s}(I_k,T_{m'})/\tau)}, \tag*{(13)}

with the symmetric text-to-image counterpart ℓt2i(m)\ell^{(m)}_{\mathrm{t2i}}. Averaging both directions over the batch gives

LPTA=12(1B∑k=1Bℓi2t(k)+1B∑m=1Bℓt2i(m)).(14)\mathcal{L}^{\mathrm{PTA}} = \frac{1}{2}\left(\frac{1}{B}\sum_{k=1}^{B}\ell^{(k)}_{\mathrm{i2t}} + \frac{1}{B}\sum_{m=1}^{B}\ell^{(m)}_{\mathrm{t2i}}\right). \tag*{(14)}

When Pk={k}\mathcal{P}_k = \{k\} this objective reduces to the standard CLIP InfoNCE loss; the only hyperparameter is the significance level α\alpha controlling which off-diagonal pairs join Pk\mathcal{P}_k.

Experiments

We evaluate PTA on zero-shot classification and Karpathy-split cross-modal retrieval, compare it with heuristic weak supervision, and examine how calibrated positives enter the objective. Table 1 reports five-seed mean±\pmSD; additional ablations examine teacher quality and calibration.

MethodImageNetIN-V2IN-AIN-RCarsDTDC-100C-10FoodCUBAircraftFlowersPetsAvg.
SA66.23<br>±0.0559.49<br>±0.1337.15<br>±0.2771.82<br>±0.2259.38<br>±0.2343.36<br>±0.2367.46<br>±0.4190.40<br>±0.4084.55<br>±0.1353.30<br>±0.2419.86<br>±0.4863.27<br>±0.1287.11<br>±0.1761.80<br>±0.06
VPL65.78<br>±0.0558.74<br>±0.0740.89<br>±0.3172.55<br>±0.1458.29<br>±0.1543.17<br>±0.2965.88<br>±0.5189.80<br>±0.2282.58<br>±0.1449.88<br>±0.3120.15<br>±0.1762.07<br>±0.4187.63<br>±0.2061.34<br>±0.08
FT65.02<br>±0.0557.76<br>±0.0640.49<br>±0.3072.15<br>±0.1656.87<br>±0.3342.19<br>±0.2765.05<br>±0.5789.63<br>±0.2581.80<br>±0.0848.05<br>±0.2218.83<br>±0.4361.39<br>±0.5187.31<br>±0.2160.50<br>±0.06
MD65.68<br>±0.1058.53<br>±0.2140.80<br>±0.2572.57<br>±0.0359.28<br>±0.2342.75<br>±0.3665.34<br>±0.3788.89<br>±0.2382.23<br>±0.0850.42<br>±0.1519.66<br>±0.3562.96<br>±0.2487.53<br>±0.1761.28<br>±0.07
WSA65.77<br>±0.0758.61<br>±0.2041.28<br>±0.2872.97<br>±0.0859.28<br>±0.2042.71<br>±0.2165.36<br>±0.4989.37<br>±0.1982.38<br>±0.1150.04<br>±0.3519.58<br>±0.4762.03<br>±0.5987.65<br>±0.1761.31<br>±0.08
PTA (ours)66.63<br>±0.0759.84<br>±0.2040.34<br>±0.2773.57<br>±0.1360.54<br>±0.2743.32<br>±0.2667.64<br>±0.3890.90<br>±0.4184.64<br>±0.1052.58<br>±0.2022.21<br>±0.5263.51<br>±0.3788.00<br>±0.2862.59<br>±0.06

Table 1. Zero-shot classification accuracy (%). CLIP ViT-B/16 students are fine-tuned on COCO train2014 with a CLIP ViT-B/32 teacher. Upper and lower rows show the mean and sample SD over seeds 41–45. PTA uses finite-rank calibration (implementation details). Best means are bold.

Datasets. Our experiments leverage the Microsoft COCO [30] (Common Objects in Context) 2014 dataset. To evaluate generalisation, we test on natural-image object-recognition tasks spanning CIFAR-10 [23], CIFAR-100 [23], Food-101 [2], ImageNet [6] together with its ImageNet-V2/A/R variants [38, 17, 16], Describable Textures (DTD) [5], StanfordCars [22], CUB-200 [43], FGVC Aircraft [31], Flowers-102 [33], and Oxford Pets [35], which cover diverse domains. For cross-modal retrieval, we use Flickr30K [36] and MS-COCO [30], standard benchmarks for image-text matching.

Baselines. We compare our method against a comprehensive set of baselines: (1) CLIP supervised fine-tuning (SFT) baselines. These provide reference points for direct adaptation of student. (2) Weak-to-Strong Augmentation (WSA). Proposed by Burns et al. [3], WSA incrementally augments positives discovered by weaker models in multi-stage training. This is the closest prior to our weak-to-strong framework. (3) Soft Align (SA). Following recent work on soft-label alignment [18], we use similarity SA: Soft Align; VPL: VL PseudoLabel; FT: Fixed Threshold; MD: Momentum Distillation; WSA: Weak-to-Strong Augmentation. Avg. is the 13-benchmark mean; its SD uses five seed-wise averages.

scores from a weaker model to generate soft positive labels, replacing the binary supervision in the contrastive loss. (4) VL PseudoLabel (VPL). Inspired by multi-label pseudo-labeling approaches for vision-language models [47], this baseline allows each image to have multiple captions marked positive if predicted above a confidence threshold. (5) Fixed Threshold (FT). As a naive baseline, we apply a constant threshold (e.g., 0.3) on weak model similarity scores to decide additional positives [7]. This tests whether our expectation-based statistical control is necessary. (6) Momentum Distillation (MD). Following ALBEF [28], we maintain a momentum-updated copy of the weak model as a teacher, and minimize KL divergence between the teacher’s and student’s similarity distributions.

Implementation details. Unless otherwise specified, a frozen CLIP ViT-B/32 teacher guides a CLIP ViT-B/16 student on COCO train2014. We train for up to 20 epochs using AdamW (learning rate 10−410^{-4} / 4096, weight decay 10−410^{-4}, 10% warmup, cosine decay), BF16 mixed precision, and gradient clipping on one GPU (Appendix § 8.10.1). Legacy single-seed runs tune the last four vision blocks. The Table 1 runs use the full vision tower and last two text blocks, B=512B = 512, seeds {41,42,43,44,45}\{41, 42, 43, 44, 45\}, and the checkpoint with the best in-domain validation recall. For PTA in Table 1, calibration uses 5,000 records in batches of 1,000 at α=0.01\alpha= 0.01, discarding fixed-point scores and retaining repetitions. Given MM retained scores, we use the kk-th order statistic with k=⌈(M+1)(1−α)⌉k = \lceil(M+1)(1-\alpha)\rceil and strict selection s>Qs > Q. This finite-rank variant retains the positive-set objective in Eq. (13). Other experiments retain the original empirical 99th-percentile calibration unless stated otherwise.

Main results. Table 1 reports five-seed zero-shot classification on 13 benchmarks after fine-tuning on COCO. PTA achieves the highest average accuracy, 62.59%, and the best mean on 10 of the 13 tasks. SA is strongest on DTD and CUB-200, while WSA is strongest on ImageNet-A.

PTA exceeds SA, the strongest baseline by average accuracy, by 0.80 percentage points (62.59% versus 61.80%). Relative to SA, PTA gains 3.19 points on ImageNet-A and 2.35 on FGVC Aircraft, while trailing by 0.04 on DTD and 0.72 on CUB-200. These differences are computed from unrounded five-seed means.

Table 2 reports cross-modal retrieval on Flickr30k and MS-COCO. On MS-COCO, PTA raises image-to-text recall@1 from 22.69% to 25.24% and text-to-image recall@1 from 21.94% to 25.06%, gains of 2.55 and 3.12 percentage points over SFT. Recall@5 gains are 4.24 and 4.44 points. On Flickr30k, recall@1 gains are 1.40 points for image-to-text and 3.50 for text-to-image. PTA achieves the highest text-to-image recall@1, while WSA remains stronger on image-to-text recall@1, showing that gains vary with retrieval direction.

MethodFlickr30kMS-COCO
Image to TextText to ImageImage to TextText to Image
R@1R@5R@1R@5R@1R@5R@1R@5
SFT81.0095.6079.7095.3022.6941.6921.9441.29
SA81.9096.4081.7096.0023.6543.6922.2642.06
VPL81.3096.1081.5095.8024.9145.3524.2444.81
FT83.0096.2081.3096.4023.9143.7223.8443.69
MD82.7096.2081.2096.3023.8943.7323.6943.58
WSA83.4096.2081.4096.3023.9743.6723.8543.72
PTA (ours)82.4096.6083.2096.3025.2445.9325.0645.73

Table 2. Cross-modal retrieval (recall@1/5). All systems are fine-tuned on COCO train2014; best scores are bold. SFT: supervised fine-tuning. Other abbreviations follow Table 1.

How weak can the teacher be? We progressively corrupt the CLIP-B/32 teacher at inference time with dropout rates {0,0.1,0.3,0.5,0.8}\{0, 0.1, 0.3, 0.5, 0.8\}, keeping the CLIP-B/16 student architecture and training hyperparameters fixed. Figure 3 compares the teacher’s standalone accuracy with the PTA-trained student’s benchmark average. At dropout 0.3, teacher accuracy falls from 61.00% to 39.00%, whereas student accuracy falls from 65.60% to 64.63% (0.97 percentage points). Student accuracy is less sensitive to this moderate corruption than the teacher’s standalone accuracy. Severe corruption still matters: at dropout 0.8, teacher accuracy is 3.54% and student accuracy falls to 59.79%, 5.81 points below its uncorrupted-teacher counterpart. Thus, the student tolerates moderate degradation but remains dependent on useful teacher information. Inference-time dropout is a controlled corruption proxy; these results do not establish robustness to every source of teacher error. Full per-task results and a separate calibration-only diagnostic are in Appendix § 8.§ 10.

Figure 3. Sensitivity to weak-teacher corruption. Teacher and PTA student accuracy under inference-time teacher dropout, with the student setup fixed. Points reproduce reported benchmark averages; per-task results are in Appendix § 8.§ 10.

Mechanism decomposition. Table 3 reports independent ablation runs with matched SFT controls, varying how the calibrated mask is consumed: (i) a CN-only variant that uses the mask only to remove false negatives from the contrastive denominator, and (ii) an asymmetric variant that keeps the full multi-positive numerator only on the image→text direction. CN-only accounts for roughly 71% of the full PTA gain (+1.08 ± 0.06); admitting the calibrated pairs into the numerator (full PTA) buys the remaining ~0.45.

VariantAvg. gain (pp)
SFT (ceiling)–
CN-only (App. A.1.1)+1.076 ±0.061
Asymmetric (t→i: CE)+1.147 ±0.035
PTA (full)+1.524 ±0.108

Table 3. PTA mechanism decomposition. Independent ablations of Eq. (13): paired accuracy gains (pp) over matched SFT controls on 13 benchmarks, mean ± SD over five seeds (three for asymmetric).

CN-only retains the observed match as the sole numerator term while removing selected off-diagonal pairs from the denominator. Full PTA retains the full normalization and maximizes probability mass over the calibrated positive set, without requiring equal probability for each candidate. The asymmetric result is consistent with a benefit from applying the positive-set objective in both directions.

Empirical calibration across datasets and teachers. Across 100 resampling seeds on COCO and Flickr30k with CLIP ViT-B/32 and ViT-B/16 teachers, held-out random-pairing exceedance at α=0.01\alpha= 0.01 is 1.005–1.023% using M=5,000M = 5,000 calibration scores. Larger budgets reduce resampling variability. These global-permutation diagnostics are separate from training-time minibatch calibration; dependent pooled scores and finite cached pools limit their interpretation. These empirical rates do not establish exact ϵ\epsilon control or semantic precision. Full protocols and results are in Appendix § 8.6; additional permutation-stability results are in Appendix § 8.5.

Visual grounding. Figure 4 compares Grad-CAM [39] maps for SA and PTA using the same three queries. PTA can reduce conflicting supervision on potentially related pairs through calibrated auxiliary positives, encouraging the student to focus on query-relevant visual regions. Full queries and retrieval rankings are in Appendix § 8.§ 13.

Figure 4. Grad-CAM visualisations. Query images and SA/PTA maps for three selected examples. Full query texts are in Appendix § 8.§ 13.

Training efficiency. Figure 5 compares per-step throughput relative to SFT. Under fixed teacher preprocessing, PTA-Cache reuses frozen embeddings and scores candidates by matrix multiplication, avoiding repeated teacher encoding. It retains the thresholding and positive-set training rules while achieving 95–99% of SFT throughput. The trade-off is preprocessing and embedding storage; cache construction is excluded. Details are in Appendix § 8.§ 12.

Figure 5. Training throughput. Ranges are not confidence intervals. Online values use inverse slowdown; cache construction excluded.

Sensitivity to α\alpha. The highest reported ImageNet accuracy occurs at α=0.01\alpha= 0.01 (Figure 6).

Figure 6. Sensitivity to α\alpha. ImageNet top-1 accuracy at the tested α\alpha values. Points are reported estimates; dashed: no PTA.

Additional experiments. The appendix covers clean-negative ablations (Appendix § 8.1.1); calibration budgets and α\alpha (Appendices § 8.§ 2–§ 8.§ 3); low-scoring matched pairs (Appendix § 8.4); permutation stability and held-out calibration (Appendices § 8.5–§ 8.6); threshold rules (Appendix § 8.§ 9.4); teacher-quality diagnostics (Appendix § 8.§ 10); data scaling and cached training (Appendices § 8.§ 11–§ 8.§ 12); and retrieval examples with full queries (Appendix § 8.§ 13).

Limitations. Experiments cover CLIP-based dual encoders, natural-image classification, and image–text retrieval, with scaling up to three million CC3M pairs. Multilingual corpora, specialised domains, and larger pretraining regimes remain to be evaluated. PTA retains observed matches as positives and selects auxiliary positives within each minibatch. Repairing incorrect original pairings or searching broader candidate pools would require extending the reference construction and evaluating the associated computational trade-offs.

Conclusion

PTA uses a weak teacher to select auxiliary positives for image–text adaptation. Permutation calibration provides Type-I error control relative to the conditional permutation reference, up to calibration approximation error. A frozen dual-encoder teacher enables offline calibration. A single multi-positive objective maximises probability mass over the calibrated positive set and recovers CLIP InfoNCE when auxiliary positives are absent. PTA achieves the highest average classification accuracy among the evaluated weak-supervision methods and improves retrieval over supervised fine-tuning. Under inference-time dropout, student accuracy is less sensitive to moderate corruption than teacher accuracy, while severe corruption still degrades student performance.

Acknowledgments and Disclosure of Funding

We thank The University of Melbourne’s Research Computing Services and the Petascale Campus for providing computational resources.

Funding. Jinhao Li is supported by the Melbourne Research Scholarship. Feng Liu is supported by the ARC with grant numbers DP230101540, DE240101089 and DP260103996.

The authors declare no competing interests.

Appendix

Additional experiments

Clean-negatives ablation

The mechanism-decomposition study in Table 3 compares the full PTA objective (Eq. (13)) against a clean-negatives only variant. The CN-only variant uses the calibrated mask solely to remove false negatives from the contrastive denominator, restricting the numerator to the ground-truth diagonal:

\ell_{\mathrm{i2t}}^{\mathrm{CN},(k)}=-\log\frac{\exp(f^{s}(I_k,T_k)/\tau)}{\exp(f^{s}(I_k,T_k)/\tau)+\sum_{m\ne k}(1-\ellambda_{km}^{\mathrm{PTA}})\exp(f^{s}(I_k,T_m)/\tau)} \tag*{(15)}

with the symmetric text-to-image counterpart. CN-only therefore isolates the contribution of not penalising calibrated positives from the contribution of pulling them into the numerator. Empirically, CN-only recovers ∼71%\sim71\% of the full PTA gain in classification (Table 3); the remaining ∼0.45\sim0.45 comes from admitting calibrated pairs into the numerator, with no additional hyperparameters.

Ablation on the number of permutations

The permutation null is a Monte-Carlo estimate, so the calibrated threshold QQ inherits sampling variability from the size of SpermS_{\mathrm{perm}}. Table 4 reports the across-seed variability of QQ as we vary the number of permuted scores from 1,000 to 20,000. With only 1,000 scores the threshold is noticeably noisy across seeds, but at our default of 5,000 scores (R=5R=5 permutations of 1,000 pairs) the across-seed range has dropped below 0.3 in scaled-similarity units, and further increases to 10,000 or 20,000 produce only marginal stabilisation. We therefore use 5,000 permuted scores throughout the main results; the calibration cost is negligible relative to a single training step.

# permuted scoresAcross-seed range of QQStd of QQ
1,0008.50–9.000.25
5,000 (default)8.44–8.690.13
10,0008.49–8.620.07
20,0008.51–8.570.03

Table 4. Threshold stability. Range and standard deviation of QQ across seeds {41,42,43}\{41, 42, 43\} for different numbers of permuted scores in SpermS_{\mathrm{perm}}. Threshold units are cosine similarity · exp⁡(logit_scale)\exp(\mathrm{logit\_scale}). The default is 5,000 scores from R=5R=5 random permutations of 1,000 pairs.

Ablation on sensitivity to α\alpha

We study how the permutation-test significance level α\alpha (used to accept weakly-matched pairs as additional positives) affects downstream performance. Figure 6 reports ImageNet top-1 accuracy after fine-tuning with different α\alpha. Overall, performance exhibits a clear inverted-U trend: overly permissive thresholds (large α\alpha) admit noisy positives and reduce accuracy, while overly conservative thresholds (small α\alpha) accept too few positives and diminish the benefit of PTA. In our setting, α=10−2\alpha= 10^{-2} achieves the best accuracy, and α∈{5×10−2,10−2}\alpha\in\{5 \times10^{-2}, 10^{-2}\} remains competitive, indicating that PTA is not overly sensitive within a moderate range. We therefore set α=10−2\alpha= 10^{-2} as the default in all main experiments.

What happens to ground-truth pairs below QQ

A natural question is whether the calibration mistakenly excludes some true matched pairs whose weak similarity falls below the threshold. On a diagnostic over 10,000 COCO training pairs evaluated with the learned threshold QQ, around 5.6% of the diagonal pairs fall below QQ under the weak model. The PTA selection rule preserves these pairs by construction. Specifically, each anchor’s positive set always includes its original matched example; the threshold QQ determines which off-diagonal pairs are added as auxiliary positives. Low teacher scores therefore do not remove the original image–text pairs from supervision during student adaptation.

Monte Carlo stability of global permutation calibration

We fix n=5,000n = 5,000 COCO train2014 image–caption records and their frozen CLIP B/32 features, using the first caption per image and fixed preprocessing. This calibration-only diagnostic evaluates global permutations separately from the batch-wise training implementation. For each R∈{1,10,100}R \in\{1, 10, 100\}, we pool scores from RR uniform permutations of the complete calibration set, discard fixed-point positions, and retain repeated scores with equal weight. We use the linear empirical (1−α)(1-\alpha) quantile and the strict decision s>Qs > Q. We use 30 permutation seeds (41–70); per repetition, the RR settings share permutation prefixes. The calibration features, evaluation pairs, and probe batches remain fixed. Reference exceedance is measured on 10,000 pairs whose image and text sources come from separate held-out record blocks. Auxiliary selection is measured on off-diagonal pairs in ten disjoint probe batches of 512 records. The calibration, probe, and reference record sets are disjoint.

Table 6 shows that increasing RR reduces conditional Monte Carlo variability. At α=0.01\alpha= 0.01, the threshold standard deviation is approximately 8.1 times smaller for R=100R = 100 than for R=1R = 1, while mean reference exceedance remains near 1%. This experiment assesses calibration stability, not student accuracy or semantic precision. Reusing the same records does not create nRnR independent calibration samples or establish exact α\alpha control.

α\alphaRRQQReference exceedance (%)Auxiliary selection (%)
0.005128.262 ±\pm 0.3590.462 ±\pm 0.0670.451 ±\pm 0.077
0.0051028.152 ±\pm 0.1420.473 ±\pm 0.0240.471 ±\pm 0.031
0.00510028.151 ±\pm 0.0480.468 ±\pm 0.0080.471 ±\pm 0.011
0.010126.520 ±\pm 0.2940.969 ±\pm 0.0940.932 ±\pm 0.106
0.0101026.384 ±\pm 0.1261.007 ±\pm 0.0360.977 ±\pm 0.046
0.01010026.387 ±\pm 0.0361.008 ±\pm 0.0110.975 ±\pm 0.013

Table 6. Global-permutation calibration on fixed features (n=5,000n = 5{,}000). Values are mean ±\pm sample standard deviation across 30 permutation seeds. QQ is in scaled-similarity units. Rates are percentages, with their standard deviations in percentage points. Variability is conditional on the fixed records and features; reference exceedance measures random-pair selection, not semantic error.

Empirical calibration across datasets and teachers

We evaluate the random-pairing reference calibration on the COCO Captions and Flickr30k training splits using pretrained CLIP ViT-B/32 and ViT-B/16 teachers in evaluation mode. We retain one caption per image. For each of 100 seeds (41–140), we randomly repartition a fixed cached feature pool into image-disjoint calibration and held-out pools, and independently draw one uniform caption permutation within each pool. Fixed-point positions are removed without modifying the permutation. The calibration pool contains 20,064 image–caption records; MM denotes the number of retained permuted scores used for the threshold, rather than the number of independent observations. We take the first MM retained scores and set QQ to their linearly interpolated empirical (1−α)(1-\alpha)-quantile. We measure the fraction of held-out random-pairing scores satisfying S>QS > Q, using 20,000 scores per seed on COCO and 8,000 on Flickr30k.

Table 7 shows larger mean exceedance and variability at small calibration budgets. With M=20,000M = 20{,}000, the mean exceedance ranges from 0.490% to 0.503% at α=0.005\alpha= 0.005 and from 0.985% to 1.007% at α=0.01\alpha= 0.01 across the four settings. Increasing MM also reduces the observed resampling variability. These measurements provide empirical calibration diagnostics within the tested feature pools; they do not establish exact α\alpha control, conditional guarantees for an observed calibration sample, or semantic precision. In particular, the permuted scores share records, and the resampling repetitions reuse a finite cached pool. This diagnostic uses a global permutation within each pool and is separate from the minibatch calibration used in the training experiments.

α\alphaMMCOCO ViT-B/32COCO ViT-B/16Flickr30k ViT-B/32Flickr30k ViT-B/16
0.0051001.258 ±\pm 1.1661.170 ±\pm 0.8691.384 ±\pm 1.2481.176 ±\pm 0.945
0.0052000.909 ±\pm 0.7420.904 ±\pm 0.6840.951 ±\pm 0.8170.942 ±\pm 0.707
0.0055000.643 ±\pm 0.3300.631 ±\pm 0.3420.595 ±\pm 0.3210.605 ±\pm 0.309
0.0051,0000.560 ±\pm 0.2500.548 ±\pm 0.2220.581 ±\pm 0.2930.572 ±\pm 0.220
0.0055,0000.518 ±\pm 0.1090.509 ±\pm 0.1010.509 ±\pm 0.1360.525 ±\pm 0.121
0.00510,0000.515 ±\pm 0.0890.508 ±\pm 0.0940.496 ±\pm 0.1080.519 ±\pm 0.094
0.00520,0000.503 ±\pm 0.0700.503 ±\pm 0.0740.490 ±\pm 0.0950.502 ±\pm 0.084
0.0101001.766 ±\pm 1.3681.675 ±\pm 1.0762.071 ±\pm 1.5841.846 ±\pm 1.285
0.0102001.416 ±\pm 0.9071.388 ±\pm 0.8241.480 ±\pm 1.0671.460 ±\pm 0.869
0.0105001.097 ±\pm 0.4891.123 ±\pm 0.4781.152 ±\pm 0.4781.146 ±\pm 0.512
0.0101,0001.027 ±\pm 0.3261.027 ±\pm 0.2961.087 ±\pm 0.3831.087 ±\pm 0.320
0.0105,0001.005 ±\pm 0.1591.006 ±\pm 0.1531.023 ±\pm 0.1741.009 ±\pm 0.173
0.01010,0001.009 ±\pm 0.1401.002 ±\pm 0.1391.006 ±\pm 0.1400.995 ±\pm 0.135
0.01020,0001.007 ±\pm 0.0971.005 ±\pm 0.1101.005 ±\pm 0.1290.985 ±\pm 0.124

Table 7. Held-out random-pairing exceedance rate (%) for the linear-quantile calibration rule. Entries are mean ±\pm sample standard deviation over 100 resampling seeds; standard deviations are in percentage points. MM is the number of retained permuted calibration scores. These rates concern random pairing, rather than semantic false positives.

Ablation on thresholding strategies

PTA uses a calibrated rule to select additional in-batch positives based on weak similarities. Here we compare several thresholding strategies under an identical weak-to-strong setting, and report their average zero-shot accuracy (Table 5). The variants are: (i) SFT. Standard supervised fine-tuning on COCO without any weak-similarity filtering, (ii) FT. A single global cosine-similarity cutoff is tuned on COCO validation and reused for all batches, (iii) Naive percentile. The cutoff is set to the 99th percentile of weak similarities on ground-truth COCO pairs and kept fixed during training, and (iv) PTA. Our method derives the cutoff from the permuted null distribution of weak similarities at a chosen significance level α\alpha, yielding a dataset- and teacher-adaptive threshold with explicit control over the tolerated false-alarm rate. Table 5 summarises the seven-dataset averages for these strategies. Both FT and naive percentile improve over the SFT baseline, showing that even simple heuristics can benefit from weak similarities. However, PTA achieves the highest average accuracy among the tested thresholding strategies in this setting.

MethodAvg. accuracy (%)Δ\Delta vs. SFT
SFT62.420.00
FT63.080.66
Naive percentile64.592.17
PTA65.603.18

Table 5. Thresholding ablation. Average accuracy (%) on a legacy benchmark suite at B=4096B=4096 with the last four vision blocks unfrozen (single seed); included to mirror prior protocol.

Additional details on weak-teacher quality

Figure 3 in the main text summarises the teacher dropout study. Table 9 and Table 10 give the per-task teacher and student accuracies, including Caltech101 [25] and SUN397 [45]. We vary inference-time dropout in CLIP-B/32 over {0,0.1,0.3,0.5,0.8}\{0, 0.1, 0.3, 0.5, 0.8\} while holding the CLIP-B/16 student architecture and training hyperparameters fixed. The calibration-only diagnostic below uses a separate protocol and does not attribute the student accuracy changes to one calibration metric.

Table 9: Zero-shot classification accuracy (%) of CLIP-B/32 with different dropout rates during inference. Higher dropout rates simulate progressively weaker models.

DropoutCaltech101CIFAR10CIFAR100DTDFlowers102Food101SUN397ImageNetAverage
0.059.5858.3842.6636.5260.6088.7180.5561.0061.00
0.152.7750.2236.0639.7451.6689.2677.3656.7256.72
0.328.3121.7426.1726.3638.9381.0850.4339.0039.00
0.57.324.4410.9612.5110.0233.3713.0913.1013.10
0.80.110.502.079.771.0610.320.973.543.54

Table 9.

Table 10: Classification accuracy (%) after fine-tuning CLIP-B/16 using PTA with CLIP-B/32 weak models at different dropout rates. All use identical hyperparameters on COCO Captions.

Weak DropoutCaltech101CIFAR10CIFAR100DTDFlowers102Food101SUN397ImageNetAverage
0.064.1460.4044.5750.5166.4589.3883.7765.6065.60
0.164.1359.4744.5250.9466.5889.3683.9865.5765.57
0.362.8056.0644.3151.1765.5689.4783.0464.6364.63
0.558.7447.1044.7349.7562.4388.7277.1261.2361.23
0.856.8447.7840.5346.5160.8187.7478.3059.7959.79

Table 10.

Calibration diagnostics under teacher corruption

To complement the weak-teacher accuracy study, we conduct a separate calibration-only diagnostic with the pretrained CLIP-B/32 teacher. We vary its inference-time dropout rate over {0,0.1,0.3,0.5,0.8}\{0, 0.1, 0.3, 0.5, 0.8\}, keeping the weights fixed. The teacher is first set to evaluation mode; modules with dropout are then selectively activated at the requested rate. Scores are 100 times the cosine similarity of normalized image and text embeddings.

We fix the first 5,064 eligible COCO train2014 images in image-ID order, using the first annotated caption per image. For each of five paired seeds (41–45), we encode this fixed sample and draw one uniform global text permutation. We discard fixed points and retain the first M=5,000M = 5{,}000 mismatched scores. This diagnostic uses the kk-th order statistic as QQ, where k=⌈(M+1)(1−α)⌉k = \lceil(M + 1)(1 - \alpha) \rceil; the two α\alpha settings reuse the same score vector. The order-statistic rule specifies the practical inclusion threshold and does not establish an exact finite-sample guarantee for these dependent pooled scores.

We evaluate selection on a separate, fixed set of 5,120 COCO val2014 images, arranged into ten seed-shuffled batches of size B=512B = 512. The first caption per image defines each batch score matrix. The auxiliary selection rate is the fraction of its B(B−1)B(B - 1) off-diagonal scores strictly above QQ, aggregated over the ten batches. Known-positive recall uses separate encodings of all 25,610 annotated captions for these images to measure the fraction of corresponding image–caption scores strictly above QQ. This raw-score recall is distinct from the unconditional inclusion of observed paired positives in the training mask. Across dropout conditions, images, captions, mismatch permutations, and batch orders are paired by seed. The fixed images are not resampled across seeds.

Table 8 shows that the threshold changes with teacher corruption, but not monotonically. At α=0.01\alpha= 0.01, known-positive recall falls from 91.51% without dropout to 1.16% at dropout 0.8, while the off-diagonal selection rate remains 1.12%. Thus, recalibration does not restore matching information lost under severe teacher corruption. The auxiliary selection rate measures selection yield, not the semantic precision of the selected pairs. These calibration-only results complement the student accuracy study; they do not attribute its accuracy changes to a single calibration metric.

α\alphaDropoutQQAuxiliary selection (%)Known-positive recall (%)
0.0050.027.698 ± 0.6230.589 ± 0.14580.72 ± 5.44
0.0050.127.509 ± 0.5370.581 ± 0.13974.49 ± 4.96
0.0050.326.827 ± 0.3760.540 ± 0.12246.70 ± 5.12
0.0050.527.756 ± 0.1570.510 ± 0.1274.88 ± 0.88
0.0050.825.418 ± 0.1780.633 ± 0.1460.65 ± 0.16
0.0100.025.990 ± 0.5991.143 ± 0.25591.51 ± 2.49
0.0100.126.110 ± 0.5671.056 ± 0.23885.68 ± 3.64
0.0100.325.741 ± 0.2901.059 ± 0.18660.36 ± 3.36
0.0100.527.269 ± 0.1331.041 ± 0.1937.63 ± 0.98
0.0100.824.909 ± 0.1011.119 ± 0.1621.16 ± 0.19

Table 8. Calibration-only weak-teacher corruption diagnostic. Entries are mean ± sample standard deviation across five paired seeds. Auxiliary selection and known-positive recall are percentages; their standard deviations are in percentage points. No student is trained in this experiment.

Ablation on data scaling

To evaluate whether PTA acts as a data-efficient refinement mechanism rather than merely adding computation, we conduct a scaling experiment on CC3M [40]. We conducted a pretraining on a CLIP-B/16 student for one epoch under identical hyperparameters, varying only the number of CC3M image–text pairs used: 1.0M, 2.0M, and 3.0M. All baselines and PTA use the same weak teacher (CLIP-B/32) and the same training setup; the only difference is how additional positive signals are extracted. Figure 7 shows the average accuracy over downstream benchmarks as a function of the number of CC3M pairs. The supervised baseline exhibits clear diminishing returns: increasing the dataset from 1.0M to 2.0M pairs yields a +2.91 point gain, whereas the next million (from 2.0M to 3.0M) adds only +1.40 points. On the same 3M examples, PTA improves performance by a further +1.25 points over the supervised baseline. These results highlight that PTA yields non-trivial improvements even when the training data is fixed and relatively small. Rather than requiring additional multimodal pairs, PTA provides a statistically calibrated way to reuse in-batch negatives as auxiliary positives, improving alignment in a data-efficient manner.

Figure 7. PTA data efficiency on CC3M.

PTA-Cache for efficient training

To eliminate the dominant weak-model overhead, we implement a cached variant of PTA that precomputes weak embeddings once and reuses them throughout training, as the weak model is fixed and their embedding should be fixed as well. Concretely, before fine-tuning we run the weak CLIP encoders over the entire training set to compute and store embeddings. During training, for each mini-batch we gather the corresponding cached vectors by index, transfer them to GPU, and compute weak similarities via a single matrix multiplication; all permutation-based thresholding and positive selection is then applied directly without any additional encoder forward pass. In practice this removes the ∼\sim1.3–1.4×\times slowdown of PTA-online (which adds one extra CLIP forward per step) and yields per-step throughput within 1–5% of the SFT ceiling, since the remaining costs are lightweight.

Table 11: Recorded training allocations for Table 1. Each job uses one GPU. Counts in parentheses indicate how many of the five seeds ran on each GPU model. Elapsed time is the per-job range; GPU-hours sum the five allocations, using unrounded times.

MethodRunsGPU mix across runsElapsed/run (h)GPU-hours
SA5L40S (5)1.08–1.355.99
VPL5L40S (5)1.34–1.988.15
FT5L40S (4), H100 (1)0.99–1.626.81
MD5L40S (5)1.43–1.747.99
WSA5L40S (5)1.44–1.677.78
PTA5A100 (2), H100 (3)1.00–2.147.24
Total30Mixed hardware–43.97

Table 11.

Compute resources and recorded runtimes

We recover hardware and elapsed times from allocation records and the corresponding training logs and environment records. The 30 recorded training jobs underlying Table 1 each use one GPU, eight CPU cores, and an 80 GB host-memory allocation. The GPU models are NVIDIA A100 80GB PCIe, H100 80GB HBM3, and L40S 48GB; these memory values denote nominal device capacities. The five seeds per method follow the COCO protocol in the main text, with batch size 512, up to 20 epochs, and validation-based early stopping.

Timing scope. Job time is the allocation duration, including setup, calibration where applicable, training, scheduled validation, and saving; queueing and separate downstream evaluation are excluded. GPU-hours equal allocated GPUs multiplied by elapsed hours. The 30 jobs total 43.97 GPU-hours (4.06 A100, 4.17 H100, and 35.74 L40S). PTA’s Trainer runtime, excluding initialization and calibration but including validation, is 53.93–87.80 minutes; its five allocations sum to 7.24 GPU-hours. Hardware and stopping epochs differ, so these are cost records rather than controlled speed comparisons.

Additional cost examples. Two separate one-epoch CC3M runs use one A100 80GB GPU, eight CPU cores, 160 GB host memory, batch size 4096, and seed 42, updating the full vision tower and last two text blocks; PTA uses α=0.005\alpha= 0.005. SFT and PTA allocations are 1:38:14 and 2:06:25, respectively (3.74 A100 GPU-hours combined). This configuration differs from the scaling runs in Figure 7.

Visualisation

The main text presents Grad-CAM visualisations in Figure 4. Here we provide complementary retrieval examples and the full text queries used in the visualisations.

Qualitative retrieval

Figure 8 compares the top-three retrieval results for three selected queries. Green outlines identify the matching image, ranked third by SA and first by PTA in each example.

Figure 8. Qualitative retrieval. Three selected examples. Within each top-3 strip, ranks run left to right; green outlines mark the matching image, ranked third by SA and first by PTA.

Queries for the main-text Grad-CAM visualisations

The full text queries in Figure 4, from top to bottom, are:

  1. Airplane: A small white plane sitting on top of an airport runway.

  2. Sheep: Some very cute sheep in a big field.

  3. Motorcycle: A parked motorcycle sitting on the side of the road.

NeurIPS Paper Checklist

  1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the contributions.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

  1. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitation is discussed in the paper.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

  1. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] Justification: Yes, we have included it.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:We have included detailed implementation details.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]

Justification: We will release code upon publication.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: We have included it.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: It has been reported.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix § 8.10.1 reports verified hardware, GPU and host-memory allocations, and recorded times for the 30 training jobs underlying Table 1, plus two historical CC3M cost examples.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: We reviewed the NeurIPS Code of Ethics and confirm this research complies.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: By providing a computationally efficient and statistically robust mechanism for weak-to-strong alignment, our framework lowers resource barriers to training advanced multimodal systems, though care is needed to ensure promoting uncurated auxiliary pairings does not inadvertently amplify existing societal biases present in the training data.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: This work studies contrastive adaptation of existing image–text encoders using existing datasets and does not release data or models that pose a high risk for misuse.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Asset sources, versions, and available licenses and usage terms are documented in the project repository’s third-party notices.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: We do not introduce new datasets or models.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: We do not involve it.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: We do not.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: Polish scripts.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]Alex Andonian, Shixing Chen, and Raffay Hamid. Robust cross-modal representation learning with progressive self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16430–16441, 2022.arxiv.org/abs/2204.04588
  2. [2]Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pages 446–461. Springer, 2014.DOI
  3. [3]Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., & al., . et . (2023). Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In arXiv preprint.arXiv
  4. [4]Jaeseok Byun, Dohoon Kim, and Taesup Moon. Mafa: Managing false negatives for vision-language pretraining. In CVPR, 2024.arxiv.org/abs/2312.06112
  5. [5]Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014.arxiv.org/abs/1311.3618
  6. [6]Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255.
  7. [7]Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023.arxiv.org/abs/2309.17425
  8. [8]Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. Advances in neural information processing systems, 33:10948–10960, 2020.arxiv.org/abs/2007.08929
  9. [9]Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581–595, 2024.arxiv.org/abs/2110.04544
  10. [10]Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramidclip: Hierarchical feature alignment for vision-language model pretraining. Advances in neural information processing systems, 35:35959–35970, 2022.arxiv.org/abs/2204.14095
  11. [11]Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang, Wei Liu, and Xing Sun. Softclip: Softer cross-modal alignment makes clip stronger. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1860–1868, 2024.arxiv.org/abs/2303.17561
  12. [12]Phillip Good. Permutation, parametric and bootstrap tests of hypotheses. Springer, 2005.
  13. [13]Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf, and Alex Smola. A kernel statistical test of independence. In Advances in neural information processing systems, 2007.
  14. [14]Haochen Han, Alex Jinpeng Wang, Peijun Ye, and Fangming Liu. Unlearning the noisy correspondence makes clip more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4518–4528, 2025.arxiv.org/abs/2507.03434
  15. [15]Jesse Hemerik and Jelle Goeman. Exact testing with random permutations. Test, 27(4):811–825, 2018.DOI
  16. [16]Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., & Gilmer, J. (2021). The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV.
  17. [17]Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021). Natural adversarial examples. CVPR.
  18. [18]Hailang Huang, Zhijie Nie, Ziqiao Wang, and Ziyu Shang. Cross-modal and uni-modal soft-label alignment for image-text retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18298–18306, 2024.DOI
  19. [19]Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR, 2021.arxiv.org/abs/2102.05918
  20. [20]Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022.arxiv.org/abs/2203.12119
  21. [21]Chaoya Jiang, Wei Ye, Haiyang Xu, Songfang Huang, Fei Huang, and Shikun Zhang. Vision language pre-training by contrastive learning with cross-modal similarity regulation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14660–14679. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.acl-long.819. URL https://aclanthology.org/2023.acl-long.819/.DOI
  22. [22]Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, June 2013.DOI
  23. [23]Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. https://cave.cs.toronto.edu/kriz/learning-features-2009-TR.pdfcave.cs.toronto.edu/kriz/learning-features-2009-TR.pdf
  24. [24]Lehmann, E. L., & Romano, J. P. (2005). Testing statistical hypotheses.
  25. [25]Fei-Fei Li, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pattern Recognition Workshop, page 178. IEEE, 2004. doi: 10.1109/CVPR.2004.383. URL https://doi.org/10.1109/CVPR.2004.383.
  26. [26]Jinhao Li, Haopeng Li, Sarah Monazam Erfani, Lei Feng, James Bailey, and Feng Liu. Visual-text cross alignment: Refining the similarity score in vision-language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 28018–28039. PMLR, 2024. URL https://proceedings.mlr.press/v235/li24ag.html.proceedings.mlr.press/v235/li24ag.html
  27. [27]Jinhao Li, Sarah Monazam Erfani, Lei Feng, James Bailey, and Feng Liu. Exploring weak-to-strong generalization for CLIP-based classification. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=quE8gD0egf.openreview.net/forum?id=quE8gD0egf
  28. [28]Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705, 2021.arxiv.org/abs/2107.07651
  29. [29]Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022.arxiv.org/abs/2201.12086
  30. [30]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.arxiv.org/abs/1405.0312
  31. [31]Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.arxiv.org/abs/1306.5151
  32. [32]Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In European conference on computer vision, pages 529–544. Springer, 2022.arxiv.org/abs/2112.12750
  33. [33]Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.DOI
  34. [34]Pan, R., Dong, J., & Yang, H. (2025). Discovering clone negatives via adaptive contrastive learning for image-text matching. The Thirteenth International Conference on Learning Representations.
  35. [35]Parkhi, O. M., Vedaldi, A., Zisserman, A., & Jawahar, C. V. (2012). Cats and dogs. IEEE Conference on Computer Vision and Pattern Recognition.
  36. [36]Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.arxiv.org/abs/1505.04870
  37. [37]Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., & al., . et . (2021). Learning transferable visual models from natural language supervision. International Conference on Machine Learning, 8748–8763.
  38. [38]Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389–5400. PMLR, 2019.arxiv.org/abs/1902.10811
  39. [39]Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.DOI
  40. [40]Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565. Association for Computational Linguistics, 2018. doi: 10.18653/v1/P18-1238. URL https://aclanthology.org/P18-1238/.DOI
  41. [41]Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023.arxiv.org/abs/2303.15389
  42. [42]Tschannen, M., Gritsenko, A., Wang, X., Naeem, M. F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafá, B., & al., . et . (2025). Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. In arXiv preprint.arXiv
  43. [43]Wah, C., Branson, S., Welinder, P., Perona, P., & Belongie, S. (2011). The Caltech-UCSD Birds-200-2011 dataset. In Technical Report CNS-TR-2011-001.
  44. [44]Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., & al., . et . (2024). Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. In arXiv preprint. 2409.12191
  45. [45]Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Conference on Computer Vision and Pattern Recognition, pages 3485–3492. IEEE, 2010. doi: 10.1109/CVPR.2010.5539970. URL https://doi.org/10.1109/CVPR.2010.5539970.
  46. [46]Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. Fg-clip: Fine-grained visual and textual alignment. arXiv preprint arXiv:2505.05071, 2025.arXiv
  47. [47]Xin Xing, Zhexiao Xiong, Abby Stylianou, Srikumar Sastry, Liyu Gong, and Nathan Jacobs. Vision-language pseudo-labels for single-positive multi-label learning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 7799–7808. IEEE, 2024.arxiv.org/abs/2310.15985
  48. [48]Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309.16671, 2023.arXiv
  49. [49]Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., & al., . et . (2021). Florence: A new foundation model for computer vision. In arXiv preprint. 2111.11432
  50. [50]Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18123–18133, 2022.arxiv.org/abs/2111.07991
  51. [51]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023.arxiv.org/abs/2303.15343
  52. [52]Jiahan Zhang, Qi Wei, Feng Liu, and Lei Feng. Candidate pseudolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data. arXiv preprint arXiv:2406.10502, 2024.arXiv
  53. [53]Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930, 2021.arXiv
  54. [54]Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816–16825, 2022.arxiv.org/abs/2203.05557
  55. [55]Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022.DOI

Paper details

vlnc:W_4ya6gt74
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

LLMs

View statement
Looking for related papers…
Contents