WorksComputer Science
ERCLIP: Small-Data Adaptation of CLIP for Free-Length Image-Text Retrieval
Abstract
CLIP's 77-token text encoder truncates detailed image descriptions. We present ERCLIP, a small-data adaptation of CLIP for long-caption image–text retrieval. During training, a long caption is split into sentences that the CLIP text encoder encodes independently; the sentence embeddings are fused by mean pooling or a gated residual Transformer scorer and receive their own contrastive loss. At inference ERCLIP is an unmodified CLIP dual encoder: test captions are still read through the 77-token window, and only training sees the full caption. We find that this fine-tuning erodes CLIP's zero-shot classification, most when the fusion output is not a combination of CLIP embeddings, and add a frozen-CLIP preservation loss on the image and short-caption embeddings. With 20k pairs and one epoch, ERCLIP improves long-caption R@1 over CLIP by 6.5–11.2 points, closing 62–77% of the gap to LongCLIP with 2% of its long captions. Preservation recovers 1.5–2.1 ImageNet zero-shot points (30–42% of the loss) and adds up to 2.4 points of COCO image-to-text R@1, at no inference cost.
Introduction
CLIP dual encoders align images and text in a shared embedding space and support both retrieval and zero-shot recognition [16]. Their text encoders, however, read at most 77 tokens, so detailed descriptions are truncated and the discarded facts never reach the embedding. Long-context methods extend the positional embeddings, match features across caption lengths, or learn token selection [25, 22, 14, 21]. They rely on large long-caption corpora and rarely report what the adaptation does to the tasks CLIP already solves, a cost documented for classification fine-tuning [20, 7] but not for long-caption retrieval.
We study the opposite regime: 20k image–caption pairs, 2% of the long captions used by LongCLIP, and one epoch on one GPU. ERCLIP splits each long caption into sentences that fit the original encoder, fuses their embeddings, and aligns the result with the image. The sentence pathway exists only in training, so the adapted model keeps CLIP’s architecture, 77-token context, and inference cost; at test time long captions are still truncated, but the part the encoder reads is better aligned with the image. We observe that this adaptation erodes zero-shot classification by an amount that depends on the fusion module and grows with training, and we limit it with a gated residual fusion initialised at mean pooling and a frozen-CLIP preservation loss.
Our contributions are (i) a sentence-level scheme that adapts CLIP to long captions from 20k pairs without changing its inference; (ii) evidence that the fusion module sets the cost to zero-shot ability, which motivates gated residual fusion and a label-free preservation loss; and (iii) a joint evaluation of long-caption retrieval, short-caption retrieval, and zero-shot classification at one fixed budget. The ingredients are not new (§ 2); our focus is their behaviour under scarce data.
Related Work
Vision–language pretraining. CLIP learns image and text encoders contrastively [16]; follow-up work changes the data, objective, or supervision [4, 23, 10, 24, 9, 8], all at a scale far beyond our budget.
Long-caption retrieval. LongCLIP stretches the positional embeddings to 248 tokens and aligns the principal components of short-caption features [25]. SmartCLIP and TULIP learn structured selection or relative positional encodings for long text [22, 14], and HiMo-CLIP models semantic hierarchy and monotonicity [21]. All are trained on large corpora, e.g., 1M ShareGPT4V captions for LongCLIP. LongCLIP’s ablations include aligning an image with both its long and short caption, and anchoring short-text features to the frozen CLIP encoder, and find both inferior to its own strategy [25]. ERCLIP’s objective is close to these variants, and we do not claim the losses as new. We study them in a different setting: a 77-token encoder that reaches content beyond its window only through sentence-level fusion during training, an anchor on the image as well as the short caption, and 20k pairs for one epoch. We ask how much of the full-scale gain this recovers, and at what cost, without large data or compute.
Preserving pretrained behaviour. Fine-tuning CLIP degrades its zero-shot ability. WiSE-FT recovers much of it by weight interpolation [20], LP-FT shows that full fine-tuning distorts pretrained features [7], and FLYP retains more by fine-tuning with the contrastive pretraining loss [2]; continual learning earlier used distillation to a frozen copy [11] or parameter regularisation [5]. These works fine-tune for and preserve classification; we fine-tune for long-caption retrieval, a different task.
Method
ERCLIP fine-tunes the CLIP encoders and (embedding dimension ) on triplets of an image , a short caption , and a long caption (Figure 1).

Figure 1. ERCLIP training graph. Green encoders are trainable; the two text encoders share weights. Sentences are encoded into and fused as , with weights from the Transformer scorer (yellow). A frozen copy supplies for . At inference only the green encoders remain: images pass through and captions, truncated to 77 tokens, through ; sentence split, fusion, teacher, and losses (orange) are discarded.
Sentence-Level Encoding and Fusion
is split on sentence-final periods into ordered sentences, each tokenised separately and encoded by the shared text encoder, , so that every sentence fits the pretrained encoder’s window. The embeddings are fused into one long-caption embedding.
Mean pooling. adds no parameters and weights every sentence equally.
Transformer scorer. A single T5-style Transformer block [17] (8 heads) contextualises , where is the -th row of CLIP’s text positional embedding and encodes sentence order. An MLP scores each contextual state :
The scorer re-weights the original rather than emitting , so stays inside the convex hull of CLIP sentence embeddings.
Gated residual fusion. ERCLIP-v1 uses directly, so its randomly initialised scorer co-adapts with the backbone from arbitrary weights. We instead start from mean pooling with a scalar gate initialised to zero:
The model starts as mean pooling, and the scorer acts only as far as grows. As an ablation, v1 also uses an LSTM [3] whose final hidden state is , an output not formed from CLIP embeddings.
Training Objective
Let , , and be the -normalised image, short-caption, and fused long-caption embeddings of a batch of size . We use the symmetric InfoNCE loss with a fixed temperature and label smoothing ,
with and : two in-batch contrastive terms sharing the image embedding, not a triplet loss. Since every sentence passes through the that also encodes short captions and class prompts, the long-caption gradient moves the representation zero-shot classification relies on. ERCLIP-v1 optimises only , which costs 3–6 ImageNet points with Transformer fusion and 12–14 with the LSTM (§ 4.2).
Frozen-CLIP preservation. Before training we copy the model into a frozen teacher [11], which gives reference embeddings for the same image and short caption, and penalise the cosine distance to them:
Only the image and short-caption embeddings, the interface of zero-shot classification, are anchored. The long-caption embedding stays free, since pretrained CLIP has no good long-caption representation to anchor it to (Table 1). Unlike weight interpolation after training [20], anchors features during training and needs no labels or extra data.
| Model | ViT | Urban1K I2T | Urban1K T2I | ShareGPT4V I2T | ShareGPT4V T2I | DOCCI I2T | DOCCI T2I | Long-DCI I2T | Long-DCI T2I | Avg. I2T | Avg. T2I |
| LongCLIP | B/16 | 79.30 | 79.40 | 94.60 | 93.40 | 63.18 | 71.50 | 42.20 | 48.46 | 69.82 | 73.19 |
| LongCLIP | L/14 | 82.50 | 86.10 | 97.20 | 97.30 | 66.58 | 78.56 | 46.53 | 54.28 | 73.20 | 79.06 |
| SmartCLIP | B/16 | 90.50 | 87.20 | 98.70 | 98.00 | 77.38 | 78.08 | 54.80 | 54.49 | 80.34 | 79.44 |
| SmartCLIP | L/14 | 93.30 | 90.10 | 97.90 | 98.50 | 81.70 | 82.58 | 59.14 | 60.27 | 83.01 | 82.86 |
| TULIP | B/16 | 88.10 | 86.60 | 98.60 | 98.60 | – | – | 50.20 | 50.60 | – | – |
| TULIP | L/14 | 90.10 | 91.10 | 99.00 | 99.00 | – | – | 55.70 | 56.40 | – | – |
| CLIP | B/32 | 61.20 | 46.60 | 83.30 | 77.10 | 58.04 | 53.40 | 34.79 | 30.60 | 59.33 | 51.93 |
| ERCLIP M | B/32 | 61.40 | 65.40 | 83.70 | 81.00 | 58.66 | 60.24 | 38.54 | 39.88 | 60.57 | 61.63 |
| ERCLIP M+P | B/32 | 62.30 | 66.40 | 84.00 | 82.60 | 59.58 | 61.58 | 39.65 | 40.94 | 61.38 | 62.88 |
| ERCLIP RT+P | B/32 | 62.50 | 65.40 | 84.70 | 82.00 | 58.78 | 61.22 | 38.92 | 40.61 | 61.22 | 62.31 |
| CLIP | B/16 | 67.50 | 53.30 | 84.50 | 79.60 | 60.76 | 57.10 | 36.91 | 33.73 | 62.42 | 55.93 |
| ERCLIP M | B/16 | 69.30 | 75.10 | 86.30 | 83.10 | 63.20 | 67.48 | 41.86 | 44.76 | 65.17 | 67.61 |
| ERCLIP M+P | B/16 | 68.60 | 75.70 | 87.50 | 85.30 | 63.24 | 67.16 | 41.57 | 45.34 | 65.23 | 68.38 |
| ERCLIP RT+P | B/16 | 69.30 | 73.40 | 87.80 | 85.70 | 62.84 | 67.24 | 41.90 | 45.51 | 65.46 | 67.96 |
| CLIP | L/14 | 68.20 | 56.00 | 84.20 | 83.50 | 65.70 | 63.16 | 36.98 | 35.06 | 63.77 | 59.43 |
| ERCLIP M | L/14 | 78.30 | 81.30 | 89.00 | 88.00 | 71.62 | 75.28 | 47.59 | 51.12 | 71.63 | 73.92 |
| ERCLIP M+P | L/14 | 76.40 | 79.90 | 89.50 | 89.30 | 72.14 | 74.02 | 48.46 | 51.10 | 71.62 | 73.58 |
| ERCLIP RT+P | L/14 | 75.40 | 80.00 | 90.40 | 89.70 | 71.82 | 73.54 | 47.91 | 51.17 | 71.38 | 73.60 |
Table 1. Long-caption retrieval (Recall@1, %). CLIP is the pretrained model without fine-tuning. ERCLIP rows are trained for 1 epoch on 20k pairs: M is mean pooling, M+P adds , and RT+P is residual Transformer fusion with (shaded). Grey rows are published results of methods trained on larger long-caption corpora and are given for reference only. “–”: not available.
Inference. Fusion and teacher are used only in training. At test time ERCLIP is the adapted dual encoder with CLIP’s 77-token context, so longer captions are truncated exactly as in CLIP. “Free-length” thus refers to training, where every sentence reaches the encoder; the gains in Table 1 come from better alignment of the part of the caption the encoder reads, not from reading all of it. Sentence-wise encoding at test time is left to future work. ERCLIP keeps the parameters and latency of its backbone.
Experiments
Data and metrics. We train on the first 20k pairs, in COCO image-id order, of the 50k-image pool shared by COCO and ShareGPT4V [12, 1], with the first COCO caption as the short caption and the ShareGPT4V description as the long one. We report Recall@1 (R@1) for image-to-text (I2T) and text-to-image (T2I) retrieval on Urban1K, ShareGPT4V-1k, DOCCI, and Long-DCI [25, 15, 19, 14] and on the 5k images of COCO val2017, and zero-shot accuracy on CIFAR-10/100 and ImageNet [6, 18] with the CLIP prompt ensemble. Only Urban1K overlaps with our training data: 61 of its images are in our training set and 170 in our pool. For comparability with published results we report the full Urban1K set; on the 830 clean images every run’s gain over CLIP changes by at most 1.2 points and the ranking of the variants is unchanged.
Implementation. We fine-tune all parameters of the ViT-B/32, B/16, and L/14 CLIP backbones with AdamW [13] (learning rate , weight decay 0.05, 200 warm-up steps, cosine decay), batch size 64, mixed precision, and on one A100 GPU (312 steps per epoch). We compare M (mean pooling, ), M+P (mean pooling, ), and RT+P (residual Transformer, ). Every run is evaluated after each of five epochs, and all tables report epoch 1, a budget fixed in advance (§ 4.3). Each configuration is a single run, so we treat differences below about one point between variants as ties and draw conclusions only from effects with the same sign on all three backbones.
Long-Caption Retrieval
Table 1 shows that one epoch on 20k pairs raises CLIP’s four-benchmark average by 6.5 (B/32), 7.6 (B/16), and 11.2 points (L/14), mostly in text-to-image retrieval. Preservation does not trade retrieval away: M+P gains 1.0 point over M on B/32, M+P and RT+P gain 0.4 and 0.3 points on B/16, and on L/14 all three are within 0.3 points. The residual Transformer does not surpass mean pooling, as expected from a gate that departs from the mean only when the data support it. Methods trained on large corpora (grey rows) remain clearly stronger: SmartCLIP leads the best ERCLIP variant by 13.1 (B/16) and 10.2 (L/14) average points, and LongCLIP by 4.7 and 3.4. They differ in data (1M+ vs. 20k captions), compute, and test-time context (248+ vs. 77 tokens), and none has been retrained on our 20k pairs, so these rows are a reference rather than a same-budget baseline. One epoch on 20k pairs closes 62% (B/16) and 77% (L/14) of the CLIP-to-LongCLIP gap. Much of the alignment is thus reachable with little data, though this does not show greater data efficiency.
Effect on Basic Tasks
Table 2 places long-caption retrieval next to the tasks CLIP already solves. ERCLIP-v1 shows the zero-shot degradation known for classification fine-tuning [20] under a retrieval objective. Mean and Transformer fusion cost 3.2–6.1 ImageNet points on every backbone while COCO retrieval improves, as expected since the short captions come from COCO. The LSTM loses 12.4–14.3 ImageNet and 2.1–7.7 CIFAR-100 points and has the worst long-caption retrieval. This ordering follows how far each fusion output may move from the CLIP space: mean pooling and the scorer combine CLIP sentence embeddings, while the LSTM emits a new vector. We do not measure drift directly, so this is a pattern consistent across backbones, not a measured relation.
Figure 2 isolates the matched rows of Table 2. Preservation acts most clearly on short-caption alignment: COCO I2T rises over M in every run, by 1.3–2.4 points with M+P and 1.1–2.3 with RT+P, while long-caption R@1 stays within −0.3 to +1.0 points of M and COCO T2I within 0.6. Anchoring thus restores the short-caption interface without competing with the long-caption loss. The fusion module matters mainly for classification. With mean pooling, CIFAR-10 stays within −0.2 to +0.7 points of M. RT+P is 1.3–2.3 points below M on B/32 and B/16 at epoch 1 but reaches M+P by epoch 2 (Figure 3), suggesting that the new scorer briefly perturbs the shared encoder before its gate settles; on L/14 it is 1.3 points above M. Under a one-epoch budget M+P is therefore the safest variant. On ImageNet, the hardest zero-shot test, one epoch without preservation costs 3.8 (L/14) to 6.1 (B/16) points, and recovers 1.5–2.1 of them on every backbone and fusion type (30–42%); CIFAR-100 improves over M in every run. Preservation reduces the erosion but does not remove it, and it is not an advantage over full-scale training: LongCLIP reports zero-shot accuracy on par with or above ours while being stronger on long captions [25]. Its role is to make small-data adaptation safe.
| ViT | Model | Long avg. I2T | Long avg. T2I | COCO I2T | COCO T2I | Zero-shot cls. C10 | Zero-shot cls. C100 | Zero-shot cls. IN | |
| B/32 | CLIP (no fine-tuning) | – | 59.33 | 51.93 | 50.00 | 30.37 | 89.76 | 64.31 | 63.34 |
| ERCLIP-v1 LSTM | × | 45.99 | 44.23 | 45.94 | 34.32 | 86.42 | 56.60 | 49.48 | |
| ERCLIP-v1 Transformer | × | 60.33 | 62.32 | 53.34 | 39.31 | 88.13 | 64.89 | 58.04 | |
| ERCLIP M | × | 60.57 | 61.63 | 54.40 | 39.16 | 87.75 | 62.45 | 57.51 | |
| ERCLIP M+P | ✓ | 61.38 | 62.88 | 56.40 | 39.74 | 87.51 | 63.42 | 59.64 | |
| ERCLIP RT+P | ✓ | 61.22 | 62.31 | 55.46 | 39.68 | 86.49 | 63.66 | 59.43 | |
| B/16 | CLIP (no fine-tuning) | – | 62.42 | 55.93 | 51.72 | 32.66 | 90.81 | 67.22 | 68.35 |
| ERCLIP-v1 LSTM | × | 51.27 | 51.17 | 50.68 | 38.38 | 85.51 | 59.87 | 54.09 | |
| ERCLIP-v1 Transformer | × | 65.94 | 67.68 | 59.98 | 44.46 | 89.22 | 67.66 | 62.57 | |
| ERCLIP M | × | 65.17 | 67.61 | 59.00 | 44.28 | 91.61 | 67.92 | 62.30 | |
| ERCLIP M+P | ✓ | 65.23 | 68.38 | 61.36 | 44.51 | 91.83 | 68.73 | 64.09 | |
| ERCLIP RT+P | ✓ | 65.46 | 67.96 | 61.34 | 44.36 | 89.34 | 68.02 | 64.15 | |
| L/14 | CLIP (no fine-tuning) | – | 63.77 | 59.43 | 56.10 | 35.33 | 95.70 | 76.44 | 75.53 |
| ERCLIP-v1 LSTM | × | 57.34 | 59.34 | 55.40 | 44.53 | 93.60 | 74.34 | 63.13 | |
| ERCLIP-v1 Transformer | × | 70.94 | 73.91 | 65.62 | 50.35 | 94.86 | 77.02 | 72.30 | |
| ERCLIP M | × | 71.63 | 73.92 | 65.24 | 50.55 | 94.64 | 76.94 | 71.76 | |
| ERCLIP M+P | ✓ | 71.62 | 73.58 | 66.58 | 49.95 | 95.37 | 78.29 | 73.36 | |
| ERCLIP RT+P | ✓ | 71.38 | 73.60 | 66.36 | 49.96 | 95.92 | 78.83 | 73.26 |
Table 2. Table 2. Long-caption retrieval versus the tasks CLIP already solves. Long avg. is Recall@1 averaged over the four long-caption benchmarks, COCO is 5k-test Recall@1, and classification is zero-shot accuracy on CIFAR-10/100 (C10/C100) and ImageNet (IN). The lower block of each backbone is the matched 1-epoch comparison.
Figure 2. Change of M+P and RT+P relative to M (epoch 1). Colour: fusion; marker: backbone.
Figure 3. Long-caption R@1 (four-benchmark average, top) and zero-shot CIFAR-10 accuracy (bottom) over five epochs. Each panel uses its own scale. The shaded epoch is the budget reported in all tables, and the dashed line is the pretrained model. Colour denotes the fusion type; hollow markers are runs without preservation.
Training Length
Longer training does not help (Figure 3). From epoch 1 to 5, long-caption R@1 falls by 0.4–1.2 points for every variant while COCO R@1 rises by 0.5–1.2 points on the B backbones and is flat on L/14: the model drifts toward the COCO caption style. Without preservation CIFAR-10 also falls, by 1.2–1.8 points; with it the B/32 decline shrinks to 0.5. Preservation limits this drift but not the loss of long-caption alignment, so a fixed early budget is needed. We have not tested whether the decline persists with a learned temperature, larger batches, or a schedule matched to a longer budget, so one epoch is an empirical choice for our configuration.
Ablations
Components. In the matched block of Table 2, CLIPM adds sentence-level training with , MM+P adds only , and M+PRT+P changes only the fusion. Not included are RT without and an -only run, which would separate the long-caption loss from fine-tuning on COCO captions; the v1 rows are not matched and show only the direction of the fusion effect.
Preservation weight and data size. Table 3 uses a one-epoch schedule that fully decays the learning rate, so its rows compare only with each other. Raising from 0 to 2 raises ImageNet monotonically (58.2 to 60.8), while long-caption and COCO R@1 rise by up to 1.3 points and plateau for and CIFAR-10 stays within 0.4 points. We use , untuned on the test sets. Long-caption R@1 saturates early (56.3 at 1k, 61.9 at 10k, within 0.4 to 50k) while COCO R@1 rises (40.8 to 49.3) and ImageNet falls (63.5 to 59.7): beyond about 10k pairs extra data mainly shifts the model toward COCO captions. We use k.
| Long avg. | COCO | Zero-shot | ||||
| I2T | T2I | I2T | T2I | C10 | IN | |
| 0.0 | 60.36 | 61.44 | 53.96 | 39.07 | 88.72 | 58.16 |
| 0.25 | 61.45 | 62.32 | 54.78 | 39.42 | 88.71 | 59.29 |
| 0.5 | 61.75 | 62.71 | 55.74 | 39.61 | 88.87 | 59.67 |
| 1.0 | 61.60 | 62.49 | 56.20 | 39.48 | 88.52 | 60.06 |
| 2.0 | 61.56 | 62.33 | 56.10 | 39.20 | 88.65 | 60.76 |
| 1k | 60.08 | 52.50 | 51.22 | 30.45 | 89.92 | 63.48 |
| 5k | 59.81 | 58.85 | 53.00 | 36.45 | 90.19 | 61.08 |
| 10k | 62.37 | 61.52 | 55.22 | 38.14 | 89.00 | 60.91 |
| 20k | 61.60 | 62.49 | 56.20 | 39.48 | 88.52 | 60.06 |
| 50k (all) | 61.43 | 63.12 | 57.42 | 41.19 | 88.10 | 59.72 |
Table 3. Ablations on ViT-B/32 (mean pooling, 1 epoch): preservation weight with k pairs, and training-set size with . These runs fully decay the learning rate within the epoch and are therefore not directly comparable to Table 2. Long avg. is Recall@1 averaged over the four long-caption benchmarks; IN is ImageNet zero-shot accuracy.
Discussion and Conclusion
With 20k pairs and one epoch, ERCLIP gains 6.5–11.2 points of long-caption R@1 without changing CLIP’s inference, and preservation recovers 30–42% of the ImageNet loss. The fusion choice, not only the objective, sets the cost of adaptation.
When to use it. With a million long captions and several GPUs, LongCLIP or SmartCLIP is the better choice: they read the full caption and retrieve better. ERCLIP fits when only thousands of long captions and one GPU are available, e.g., in a specialised domain, and the result must remain a standard CLIP checkpoint that keeps short-caption retrieval and zero-shot classification usable. Its inference cost equals CLIP’s, a small advantage over LongCLIP, whose extra cost is confined to a 248-token text pass; we did not benchmark it.
Limitations. Test captions are truncated to 77 tokens and sentence-wise inference is not evaluated; no baseline is retrained on our 20k pairs; results are single-seed; the one-epoch budget holds for one optimisation setting; the link between drift and zero-shot loss is inferred rather than measured; and all data are general-domain.
Acknowledgement
This publication has emanated from research supported in part by research grants from Science Foundation Ireland under grant numbers SFI/13/RC/2106 P2 and 18/CRT/6223, and co-funded by the European Regional Development Fund.
References
- [1]Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., Lin, D.: Sharegpt4v: Improving large multi-modal models with better captions. In: European Conference on Computer Vision. pp. 370–387. Springer (2024)arxiv.org/abs/2311.12793
- [2]Goyal, S., Kumar, A., Garg, S., Kolter, Z., Raghunathan, A.: Finetune like you pretrain: Improved finetuning of zero-shot vision models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 19338–19347 (2023)arxiv.org/abs/2212.00638
- [3]Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)pubmed.ncbi.nlm.nih.gov/9377276
- [4]Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference on machine learning. pp. 4904–4916. PMLR (2021)
- [5]Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabsk-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114(13), 3521–3526 (2017)
- [6]Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. (2009)
- [7]Kumar, A., Raghunathan, A., Jones, R., Ma, T., Liang, P.: Fine-tuning can distort pretrained features and underperform out-of-distribution. In: International Conference on Learning Representations (2022)arxiv.org/abs/2202.10054
- [8]Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023)arxiv.org/abs/2301.12597
- [9]Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: ICML (2022)arxiv.org/abs/2201.12086
- [10]Li, Y., Liang, F., Zhao, L., Cui, Y., Ouyang, W., Shao, J., Yu, F., Yan, J.: Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In: International Conference on Learning Representationsarxiv.org/abs/2110.05208
- [11]Li, Z., Hoiem, D.: Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(12), 2935–2947 (2018)
- [12]Lin, T.Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C.L., Dollár, P.: Microsoft coco: Common objects in context (2015), https://arxiv.org/abs/1405.0312
- [13]Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)arxiv.org/abs/1711.05101
- [14]Najdenkoska, I., Derakhshani, M.M., Asano, Y.M., Noord, N.V., Worring, M., Snoek, C.G.M.: TULIP: Token-length upgraded CLIP. In: The Thirteenth International Conference on Learning Representations (2025), https://openreview.net/forum?id=r9oqH0doHfarxiv.org/abs/2410.10034
- [15]Onoe, Y., Rane, S., Berger, Z., Bitton, Y., Cho, J., Garg, R., Ku, A., Parekh, Z., Pont-Tuset, J., Tanzer, G., Wang, S., Bald ridge, J.: DOCCI: Descriptions of Connected and Contrasting Images. In: ECCV (2024)arxiv.org/abs/2404.19753
- [16]Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)arxiv.org/abs/2103.00020
- [17]Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21(140), 1–67 (2020)arxiv.org/abs/1910.10683
- [18]Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision 115(3), 211–252 (2015)
- [19]Urbanek, J., Bordes, F., Astolfi, P., Williamson, M., Sharma, V., Romero-Soriano, A.: A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 26700–26709 (June 2024)arxiv.org/abs/2312.08578
- [20]Wortsman, M., Ilharco, G., Kim, J.W., Li, M., Kornblith, S., Roelofs, R., Lopes, R.G., Hajishirzi, H., Farhadi, A., Namkoong, H., Schmidt, L.: Robust fine-tuning of zero-shot models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7959–7971 (2022)arxiv.org/abs/2109.01903
- [21]Wu, R., Chen, P., Shen, F., Zhao, S., Hui, Q., Gao, H., Lu, T., Liu, Z., Zhao, F., Wang, K., et al.: Himo-clip: Modeling semantic hierarchy and monotonicity in vision-language alignment. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 40, pp. 26974–26982 (2026)
- [22]Xie, S., Lingjing, L., Zheng, Y., Yao, Y., Tang, Z., Xing, E.P., Chen, G., Zhang, K.: Smartclip: Modular vision-language alignment with identification guarantees. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 29780–29790 (2025)
- [23]Yang, K., Deng, J., An, X., Li, J., Feng, Z., Guo, J., Yang, J., Liu, T.: ALIP: Adaptive Language-Image Pre-Training with Synthetic Caption. In: International Conference on Computer Vision. pp. 2922–2931 (2023). https://doi.org/10.1109/ICCV51070.2023.00273, https://mlanthology.org/iccv/2023/yang2023iccv-alip/
- [24]Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975–11986 (2023)arxiv.org/abs/2303.15343
- [25]Zhang, B., Zhang, P., Dong, X., Zang, Y., Wang, J.: Long-clip: Unlocking the long-text capability of clip. In: European conference on computer vision. pp. 310–325. Springer (2024)arxiv.org/abs/2403.15378

