betaFeedback

WorksComputer Science

Where Does Alignment Emerge and What Does It Mean? A Closer Look at the Platonic Representation Hypothesis

Abstract

The Platonic Representation Hypothesis (PRH) suggests that neural networks are converging toward similar representation spaces. As language models become more capable, their representations are aligning with those of vision models. However, recent work has questioned whether this trend continues to hold for newer language models. Moreover, it has been shown that representation similarity metrics can be confounded by model depth and width, requiring appropriate baselines for interpretation. In this work, we first investigate where alignment emerges between unimodal models across model layers and representation dimensions. Next, we find that the PRH capability-alignment trend persists for newer models as long as the captions are sufficiently clean and informative. Finally, we contextualize cross-modal alignment, introducing unimodal and multimodal baselines on various datasets. Our results show that the cross-modal representational alignment of unimodal models can surpass that of explicitly aligned vision-language models, but remains substantially behind the alignment observed for models of the same modality. Code is available at https://github.com/ExplainableML/prh-closer-look.

Introduction

Images and textual descriptions provide two views of the same underlying world. The Platonic Representation Hypothesis (PRH) [12] posits that sufficiently capable vision and language models may therefore learn similar abstractions despite being trained on different modalities. This insight has motivated a series of methods [11, 20, 29, 36, 37, 39] that align pretrained unimodal models with lightweight alignment layers and limited paired data instead of training for cross-modal alignment from scratch [28, 35, 38].

Recent work has refined the PRH. Koepke et al. [ [15] show that some findings strongly depend on the evaluation protocol and that performance-alignment trends observed for earlier language models no longer extend to newer model families. Gröger et al. [ [8] argue that similarity scores are difficult to interpret without suitable baselines, and thus propose calibrating them against a null hypothesis representing the absence of alignment. Moreover, these investigations into the PRH commonly report the maximum similarity across all layer pairs using the full representation of models, obscuring at which layers and dimensions alignment emerges.

We take a closer look at the PRH along three axes. First, we analyze where alignment emerges across model layers and dimensions under compression via principal component analysis (PCA). Second, we examine how the relationship between language model capability and cross-modal alignment is affected by caption characteristics. Third, we contextualize the cross-modal alignment of unimodal vision and language models beyond a null baseline. To our knowledge, no prior work has provided a comparison with randomly initialized models, explicitly aligned vision-language models (VLMs), and model pairs from the same modality.

Experimental setup

Models. We evaluate a broad set of unimodal models. On the vision side, we consider DINOv2 ViT-L [24], DINOv3 ViT-L [32] and MAE ViT-L [9]. On the language side, we evaluate 21 generative models from the BLOOMZ [22], Llama [34, 19, 21], Mistral [14], Qwen [26, 27] (Qwen Team 2024), Gemma [6, 7] (Gemma Team 2024), and DeepSeek [3] families. Additionally, we include the text embedding models Qwen3-Embedding-8B [40] and NV-Embed-v2 [16]. We compare their alignment against explicitly aligned VLMs such as CLIP ViT-L/14 [28, 13, 30], SigLIP 2 So400M [35] and Qwen3-VL-Embedding-8B [17]. For the explicitly aligned VLMs, we extract image and text representations from the joint embedding space after the final projection layer. The text encoders of CLIP and SigLIP 2 support a maximum context length of 77 and 64 tokens, respectively. For captions exceeding these limits, we split the text into chunks that respect the context window, embed each chunk independently, and finally average these chunk representations.

Datasets. We evaluate alignment on image-text datasets with different caption properties: WIT [33] with short and noisy captions from Wikipedia, CC3M [31] with short and synthetically created DreamLIP captions [41], COCO [18] with concise human-written captions, and DOCCI [23] with long and fine-grained human-written captions. To investigate the effect of caption verbosity for the same set of images, we additionally construct shortened versions of the DOCCI captions. Specifically, we prompt Qwen3-8B [27] to summarize each caption to approximately 10%, 25%, 50% or 75% of its original length while preserving the most salient semantic content. The prompt is provided in Appendix § 13 along with a DOCCI sample and its newly generated captions.

Metric. We measure alignment using mutual kk-nearest-neighbor overlap [12] and its calibrated variant proposed by Gröger et al. [2026]. For each dataset, we sample N=1024N = 1024 image-caption pairs and compare the indices of the k=10k = 10 nearest images in vision space with those of the nearest captions in language space. Koepke et al. [2026] find that alignment decreases for fixed kk with increasing gallery size, but also show that it remains stable when the relative neighborhood size k/Nk/N is preserved. An ablation study on kk and NN is provided in Appendix § 12. We use the CLS token for vision models and mean-pooled representations for language models and apply L2L_2 normalization.

Implementation. All experiments can be performed on a single NVIDIA A100 GPU with 80 GB memory. We use the LM Evaluation Harness framework [4] to obtain the MMLU and ARC Challenge accuracies as well as the GSM8K exact match scores for the generative language models in Section 3.2.

Evaluation

Where does alignment emerge across layers and dimensions?

Prior work [12, 15, 8] reports only the maximum alignment across layer pairs. Figure 1 instead shows the complete layer-wise alignment between two models on WIT. Cross-modal alignment emerges mainly in later layers, once more abstract semantic representations have formed, and peaks at a mutual kk-NN score of 0.143 between DINOv3 layer 21 and Qwen3-8B layer 30. Within-modality alignment is much stronger. Vision models form three broad block-diagonal regions above 0.3, peaking at 0.538 among later layers. A generative language model (Qwen3-8B) and a language embedding model (NV-Embed-v2) reach a mutual kk-NN score around 0.5 across many early-to-middle layer pairs. Notably, final-layer representations never reach maximum alignment and may even be far from it. More layer-wise alignment results are presented in Appendix § 7.

Heatmaps with mutual $k$-NN alignment between every layer on WIT

Figure 1. Heatmaps with mutual kk-NN alignment between every layer on WIT.

We next investigate whether alignment is localized not only to specific layers, but also within a small subspace of the representations. Specifically, we compress the vision and language representations via PCA before computing mutual kk-NN on WIT. PCA is fit independently for each layer of each model, with separate transformations for the language and vision representations. Figure 2 shows that retaining 40% of the variance preserves or slightly improves alignment for most models, suggesting that lower-variance directions mainly capture modality-specific or noisy information. However, very strong compression causes alignment to collapse, indicating that the shared structure remains distributed across multiple principal components. In Figure 2b, we plot alignment against the number of dimensions retained after PCA. This allows us to investigate the effective dimensionality of the subspaces in which the observed alignment resides. In particular, the results suggest that the cross-modal alignment is largely captured within a relatively small subspace of approximately 100 dimensions, compared to the original language spaces with 1,536–5,120 dimensions. Additional results on DOCCI and COCO are provided in Appendix C.

Figure 2. Effect of PCA compression on alignment between DINOv2 and language models on WIT.

Together, these results indicate that cross-modal alignment is localized both across depth and within the representation space: only particular layers exhibit strong correspondence, and within those layers, the correspondence can be captured by relatively low-dimensional subspaces.

Does higher alignment indicate stronger model performance?

The PRH originally suggested that stronger language models exhibit greater representational alignment with vision models on WIT [12]. However, this trend does not extend to newer model families on the same dataset [15]. Following Koepke et al. [ [15], we examine the relationship between five-shot MMLU accuracy [10] and alignment with DINOv3 across datasets with different caption characteristics. For each dataset, we independently sample 1024 image-caption pairs across five runs and report the mean and standard deviation of the mutual k-NN score with k=10k = 10. Since the original WIT subset provided by Huh et al. [2024] contains only 1024 samples, its evaluation corresponds to a single run.

Figure 3 compares the performance-alignment relationship on WIT, CC3M, and DOCCI using both original and summarized captions. We observe a weak to moderate positive correlation for maximally aligned layers on CC3M (Pearson r=0.26r = 0.26, p=0.25p = 0.25) and WIT (r=0.46r = 0.46, p=0.03p = 0.03), consistent with the findings of Koepke et al. [ [15]. In contrast, DOCCI exhibits a stronger positive correlation when considering the maximally aligned layers (r=0.66r = 0.66, p=0.001p = 0.001), which remains significant with calibrated mutual k-NN (r=0.66r = 0.66, p=0.001p = 0.001). Importantly, this trend persists when DOCCI captions are compressed to 10% of their original length, with r=0.59r = 0.59 (p=0.005p = 0.005) for the maximally aligned layers. Caption verbosity alone therefore does not explain the performance-alignment trend. Instead, the substantially weaker trend on the short and noisy WIT and CC3M captions, together with its persistence on original and summarized DOCCI captions, suggests that sufficiently clean and visually informative data is important for revealing shared structure between vision and language representations.

Figure 3. Performance of language models on MMLU vs. alignment with vision models.

To assess whether these MMLU findings generalize to other measures of language-model capability, Figure 4 relates alignment with DINOv3 on DOCCI to four-shot ARC Challenge accuracy [1] and five-shot GSM8K exact-match accuracy [2]. ARC performance correlates strongly with alignment at the maximally aligned layers (Pearson r=0.73r = 0.73, p=0.0002p = 0.0002), and the correlation remains significant after calibration (Pearson r=0.73r = 0.73, p=0.0002p = 0.0002). This is in line with the MMLU results and shows that the performance-alignment trend is not specific to a single benchmark. Solving MMLU and ARC requires factual and semantic knowledge about the world that may be represented similarly by both vision and language models. In contrast, GSM8K primarily evaluates mathematical reasoning, which is less directly grounded in the semantic knowledge shared with visual representations, and its performance does not correlate significantly with alignment. Therefore, cross-modal alignment is not a universal proxy for language-model capability, but may be more closely associated with capabilities that rely on semantic knowledge. Results for ARC and GSM8K with alignment measured on the remaining caption datasets are provided in Appendix § 9.

Figure 4. Performance of LLMs on ARC and GSM8K vs. alignment with vision models on DOCCI.

How to interpret the magnitude of cross-modal alignment?

Alignment scores are difficult to interpret in isolation. A score such as 0.3 does not indicate by itself whether two representation spaces are weakly or strongly aligned. Therefore, we explicitly compare four regimes: randomly initialized models, pairs of unimodal vision and language models as the main object of study, explicitly aligned VLMs as cross-modal references, and pairs of unimodal models within one modality.

As displayed in Figure 5, the cross-modal alignment of unimodal models on COCO and DOCCI reaches a mutual kk-NN score above 0.3, surpassing VLMs such as CLIP and SigLIP 2, but remaining below Qwen3-VL-Embedding-8B. For VLMs, we use the final embeddings in the shared image-text space, using the maximum across layer pairs instead yields nearly identical scores (Appendix § 10). Notably, Qwen3-VL-Embedding-8B reports state-of-the-art performance on multimodal retrieval benchmarks and achieves the highest mutual kk-NN cross-modal alignment. This shows that strong downstream retrieval performance is compatible with high cross-modal neighborhood agreement. The cross-modal alignment of unimodal models remains far below a mutual kk-NN score of 0.5 observed for models of the same modality. This gap indicates that unimodal vision and language models learn partially shared representations that support cross-modal neighborhood agreement, but remain substantially less similar than models operating on the same modality.

Figure 5. Alignment for different models and evaluation protocols: DINOv2/v3 (D2/D3), Qwen3-Emb-8B (Q3), NV-Embed-v2 (NV), CLIP, SigLIP 2, Qwen3-VL-Emb-8B (Q3-VL) (mean and standard deviation across 3 runs).

The DOCCI caption variants provide a controlled view of how caption detail affects this comparison while keeping the underlying images fixed. Shortening the captions reduces absolute cross-modal alignment, but leaves the relative ordering of the model pairs largely unchanged. In particular,

DINOv2/v3 paired with Qwen3-Embedding-8B or NV-Embed-v2 continue to surpass CLIP and SigLIP 2. Caption detail therefore affects the magnitude of alignment without changing our main contextual conclusions. Results across all DOCCI caption lengths are provided in Appendix § 11.

The high similarity between independently pretrained unimodal models is consistent with prior work: Zhang et al. [ [39] showed that their representations can be mapped into a shared embedding space using lightweight alignment layers and limited paired data, outperforming CLIP.

Conclusion

We revisited the PRH through a fine-grained analysis of alignment between unimodal vision and language models. Cross-modal alignment emerges in later model layers and is preserved under substantial dimensionality reduction. Moreover, we find that performance-alignment trends of the PRH persist for newer language models on image-text datasets with sufficiently informative captions. Finally, our baseline analysis shows that unimodal vision and language models can reach the representational alignment of explicitly aligned VLMs, but remain far below models of the same modality.

Limitations. The conclusions of our empirical work are limited to the evaluated models, datasets, and alignment metrics. We focus on cross-modal alignment between unimodal encoders from the vision and language domain; whether our findings generalize to other modalities remains open. Moreover, mutual kk-NN captures only one notion of representational alignment, and its absolute scores depend on the metric hyperparameters, representation extraction choices, and evaluation data. We mitigate these dependencies by demonstrating that our findings persist with calibrated mutual kk-NN and by interpreting the raw scores relative to a wide set of baselines.

Acknowledgments and Disclosure of Funding

This work was partially funded by the ERC (853489 - DEXIM) and the Alfried Krupp von Bohlen und Halbach Foundation, which we thank for their generous support. The authors gratefully acknowledge the scientific support and resources of the AI service infrastructure LRZ AI Systems provided by the Leibniz Supercomputing Centre (LRZ) of the Bavarian Academy of Sciences and Humanities (BAdW), funded by Bayerisches Staatsministerium für Wissenschaft und Kunst (StMWK). The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gausscentre.eu) for funding this project by providing computing time on the Supercomputer JUPITER at Jülich Supercomputing Centre (JSC). This work was supported by the EuroHPC JU and the Gauss Centre for Supercomputing (GCS) through funding by the European Commission, German Federal Ministry of Research, Technology and Space (BMFTR), the Ministry of Culture and Science of the State of North Rhine-Westphalia (MKW). This work benefited from Hi! PARIS and State funding managed by the French National Research Agency (ANR) under the France 2030 program, reference ANR-23-IACL-0005.

Broader impacts

This work complements prior research on the Platonic Representation Hypothesis by identifying where alignment emerges across layers and dimensions, showing that its correlation with model performance re-emerges for sufficiently clean and informative captions, and contextualizing its magnitude against a broad set of baselines. While our study is diagnostic and does not introduce any new model, our findings may help practitioners select encoders and representations for efficient cross-modal alignment, potentially reducing paired supervision and computation. The evaluated models and image-text datasets may encode societal and cultural biases that aggregate alignment scores cannot reveal. Therefore, downstream applications will require task-specific evaluation.

Layer-wise alignment analysis

Complementing the analysis in Section 3.1, we provide additional heatmaps with layer-wise alignment scores for more model combinations on WIT in Figure 6 and on DOCCI in Figure 7. Here, we also include layer-wise alignment scores between two generative language models for the language-language regime. While mutual kk-NN alignment between a generative model and a language embedding model is primarily concentrated in early-to-intermediate layer pairs, alignment between two generative language models often peaks in later layers.

Figure 6. Heatmaps with mutual kk-NN alignment between every layer on WIT.

Figure 7. Heatmaps with mutual kk-NN alignment between every layer on DOCCI.

Compression of representations via PCA

Figure 8 extends the PCA compression analysis from Section 3.1 to DOCCI. We report the mean and standard deviation across 5 runs. Figure 8a shows that, on DOCCI, a larger fraction of the variance must be retained to preserve alignment, while alignment remains largely preserved when compressing the language representations to 100 dimensions (Figure 8b). Figure 9 shows the results on COCO.

(a) Explained variance.

(b) Number of dimensions.

Figure 9. Effect of PCA compression on alignment between DINOv2 and language models on COCO.

(a) Explained variance.

(b) Number of dimensions.

Figure 8. Effect of PCA compression on alignment between DINOv2 and language models on DOCCI.

Performance-alignment trends

Section 3.2 analyzes the relationship between language-model performance on ARC and GSM8K and cross-modal alignment only on DOCCI. Figure 10 extends the ARC analysis to WIT, CC3M, and summarized DOCCI captions. The positive relationship observed on original DOCCI remains visible, although more weakly, on WIT and summarized DOCCI when considering the maximally aligned layers. In contrast, CC3M exhibits little consistent relationship, and final-layer alignment is generally less predictive.

Figure 10. Performance on ARC vs. alignment with DINOv3.

Figure 11 presents the corresponding results for GSM8K. Unlike MMLU and ARC, GSM8K accuracy exhibits no consistent positive correlation with cross-modal alignment. Therefore, cross-modal alignment should not be interpreted as a general measure of language-model capability. Instead, its relationship with performance is benchmark-dependent and may primarily reflect capabilities associated with semantic structure shared across vision and language, which mathematical reasoning on GSM8K probes less directly.

Figure 11. Performance on GSM8K vs. alignment with DINOv3.

Layer selection for VLM baselines

In Section 3.3, we represent explicitly aligned VLMs using their final layer representations in the shared image-text space. These are the representations employed in downstream tasks, such as retrieval, and directly optimized for cross-modal alignment during training. In contrast, unimodal model pairs are evaluated at their maximally aligned layer pair. To verify that this difference in evaluation does not disadvantage the VLMs, Table 1 compares the alignment obtained at the projection layer with the maximum alignment across all layer pairs. While the projection layer does not always achieve the highest score, its alignment remains consistently close to the maximum. The largest difference is 0.016 for SigLIP 2 on COCO, while all other differences are at most 0.006. Consequently, using the maximum alignment instead of the projection-layer alignment does not alter the relative ordering of the regimes in Figure 5. The layer-wise alignment patterns of the VLMs are visualized in Figure 12.

VLMWIT FinalWIT MaxCOCO FinalCOCO MaxDOCCI 100% FinalDOCCI 100% Max
CLIP0.1990.1990.317 ± 0.0050.317 ± 0.0050.292 ± 0.0050.293 ± 0.003
SigLIP 20.0790.0790.285 ± 0.0040.301 ± 0.0050.258 ± 0.0010.263 ± 0.006
Q3-VL0.2180.2240.407 ± 0.0020.407 ± 0.0020.445 ± 0.0030.445 ± 0.003

Table 1. Alignment of explicitly aligned VLMs at the final layer pair (shared embedding space) and at the maximally aligned layer pair. Mean and standard deviation over three runs for COCO and DOCCI. WIT consists of a single run.

Figure 12. Heatmaps with mutual kk-NN alignment between every layer on DOCCI.

Contextualization with varying DOCCI captions

Section 3.3 compares the alignment of unimodal models against a set of baselines on original DOCCI captions and selected summarized variants. Figure 13 complements this analysis by comparing the contextualization baselines across a larger set of DOCCI caption lengths while keeping the underlying images fixed. As captions become more concise, the absolute cross-modal alignment measured by mutual kk-NN decreases. However, the relative ordering between unimodal vision-language pairs, explicitly aligned VLMs, and within-modality pairs remains largely unchanged. In particular, DINOv2/v3 paired with Qwen3-Embedding-8B or NV-Embed-v2 remain more strongly aligned than CLIP and SigLIP 2 across caption lengths.

Figure 13. Alignment for different models and evaluation protocols with DOCCI captions of varying verbosity: DINOv2/v3 (D2/D3), Qwen3-Emb-8B (Q3), NV-Embed-v2 (NV), CLIP, SigLIP 2, Qwen3-VL-Emb-8B (Q3-VL).

Mutual kk-NN for varying kk and NN

We investigate the sensitivity of mutual kk-NN alignment to the choice of the neighborhood size kk and the gallery size NN using the DOCCI dataset.

First, we fix the gallery size to N=1024N = 1024 and compare k∈{1,5,10,100}k \in\{1, 5, 10, 100\} in Figure 14. While the alignment scores increase with larger kk, the relative ordering of the alignment regimes remains unchanged, suggesting that the observed differences are robust to the choice of kk.

Figure 14. Alignment for different models and evaluation protocols with N=1024N = 1024 DOCCI samples and varying kk.

Second, we examine the effect of varying kk while keeping the ratio kN≈0.01\frac{k}{N} \approx0.01 fixed, considering (k,N)∈{(1,102),(5,512),(10,1024)}(k, N) \in\{(1, 102), (5, 512), (10, 1024)\} (see Figure 15). For each gallery size (N<1024N < 1024), we randomly subsample from the same pool of 1024 images and repeat the experiment five times, reporting the mean and standard deviation across subsamples. For N=1024N = 1024, we use the full set of images and report a single score. The alignment scores remain relatively stable across these settings, which is in line with Koepke et al. [ [15], who find that alignment is stable when the neighborhood size scales with the gallery size.

Figure 15. Alignment for different models and evaluation protocols on DOCCI samples with varying NN and kk but constant ratio kN\frac{k}{N}.

DOCCI summarization prompt and example

We provide the prompt for summarizing DOCCI captions to a fraction of their original word count in Figure 16. An example from DOCCI with its original caption and newly generated captions of varying length is displayed in Figure 17.

Prompt for summarizing detailed DOCCI captions
You are compressing a detailed image description to {target_ratio} of its
original length.

Your goal is to reduce the LEVEL OF VISUAL DETAIL in the description.
At shorter target lengths, the output should describe substantially fewer
visual details, not merely express the same details using fewer words.
Your task is to rewrite the description as a shorter caption of {target_words}
words while preserving
the important visually grounded information. The {target_words} target is a
strict length requirement, not a suggestion!

IMPORTANT RULES:
- Use ONLY information contained in the original description.
- Do NOT add, infer, or invent information.
- Keep the remaining text grammatically complete and natural.
- Do NOT create sentence fragments.
- Do NOT remove function words merely to save space.
- At shorter target lengths, it is EXPECTED that many secondary details
will be completely omitted.
- Prioritize the most salient subjects and their primary actions.
- The target length is {target_words} words.
- Your caption HAS TO be LONGER than {target_words - 10} and SHORTER than {
  target_words + 10}.
- Write a natural, self-contained image caption.
- A simple caption containing only the main subject and action is preferable
to a dense caption containing many secondary details.
- Output ONLY the compressed caption. Do not explain your changes.

The original description contains approximately {original_words} words.

Original description:
{caption}

Compressed caption (~{target_words} words):

Figure 16. Prompt for the summarization of detailed DOCCI captions with Qwen3-8B to a fraction of their original word count.

Example of progressively compressed DOCCI captions for the same image, showing an original caption and 75%, 50%, 25%, and 10% compressed captions alongside a teddy bear photograph

Figure 17. Example of progressively compressed DOCCI captions for the same image.

References

  1. [1]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
  2. [2]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  3. [3]DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633–638, 2025.DOI
  4. [4]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPoFi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness. https://zenodo.org/records/12608602, July 2024.
  5. [6]Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Riviere, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, et al. Gemma: Open models based on Gemini research and technology. arXiv preprint arXiv:2403.08295, 2024a.
  6. [7]Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024b.
  7. [8]Fabian Gröger, Shuo Wen, and Maria Brbić. Revisiting the Platonic representation hypothesis: An Aristotelian view. In International Conference on Machine Learning (ICML), 2026.arxiv.org/abs/2602.14486
  8. [9]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2111.06377
  9. [10]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021.
  10. [11]Weiquan Huang, Aoqi Wu, Yifan Yang, Xufang Luo, Yuqing Yang, Usman Naseem, Chunyu Wang, Qi Dai, Xiyang Dai, Dongdong Chen, Chong Luo, Lili Qiu, and Liang Hu. LLM2CLIP: Powerful language model unlocks richer cross-modality representation. In AAAI Conference on Artificial Intelligence (AAAI), 2026.arxiv.org/abs/2411.04997
  11. [12]Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The Platonic representation hypothesis. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2405.07987
  12. [13]Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hananneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, 2021. Version 0.1. URL: https://doi.org/10.5281/zenodo.5143773.
  13. [14]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.arxiv.org/abs/2310.06825
  14. [15]A. Sophia Koepke, Daniil Zverev, Shiry Ginosar, and Alexei A. Efros. Back into Plato’s cave: Examining cross-modal representational convergence at scale. arXiv preprint arXiv:2604.18572, 2026.arxiv.org/abs/2604.18572
  15. [16]Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NV-Embed: Improved techniques for training LLMs as generalist embedding models. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2405.17428
  16. [17]Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Qwen3-VL-Embedding and Qwen3-VL-Reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720, 2026.arxiv.org/abs/2601.04720
  17. [18]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.arxiv.org/abs/1405.0312
  18. [19]Llama Team, Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  19. [20]Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Ankit Singh, and Noel E. O’Connor. Harnessing frozen unimodal encoders for flexible multimodal alignment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2409.19425
  20. [21]Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/, 2024.
  21. [22]Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. In Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
  22. [23]Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. DOCCI: Descriptions of connected and contrasting images. In European Conference on Computer Vision (ECCV), 2024.arxiv.org/abs/2404.19753
  23. [24]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaäddin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), 2024.arxiv.org/abs/2304.07193
  24. [26]Qwen Team, An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
  25. [27]Qwen Team, An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengeng Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
  26. [28]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2103.00020
  27. [29]Simon Roschmann, Paul Krzakala, Sonia Mazelet, Quentin Bouniot, and Zeynep Akata. SOTALign: Semi-supervised alignment of unimodal vision and language models via optimal transport. In International Conference on Machine Learning (ICML), 2026.arxiv.org/abs/2602.23353
  28. [30]Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W. Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R. Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems (NeurIPS), 2022. Datasets and Benchmarks Track.
  29. [31]Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguistics (ACL), 2018.DOI
  30. [32]Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungyun Yi, Michaël Ramamonjisoa, et al. DINOv3. Transactions on Machine Learning Research (TMLR), 2026.arxiv.org/abs/2508.10104
  31. [33]Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. WIT: Wikipedia-based image text dataset for multimodal multilingual machine learning. In International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2021.DOI
  32. [34]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.arxiv.org/abs/2302.13971
  33. [35]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Abdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025.arxiv.org/abs/2502.14786
  34. [36]Noël Vouitsis, Zhaoyan Liu, Satya Krishna Gorti, Valentin Villecroze, Jesse C. Cresswell, Guangwei Yu, Gabriel Loaiza-Ganem, and Maksims Volkovs. Data-efficient multimodal fusion on a single GPU. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.arxiv.org/abs/2312.10144
  35. [37]Amitai Yacobi, Nir Ben-Ari, Ronen Talmon, and Uri Shaham. Learning shared representations from unpaired data. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2505.21524
  36. [38]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.arxiv.org/abs/2303.15343
  37. [39]Le Zhang, Qian Yang, and Aishwarya Agrawal. Assessing and learning alignment of unimodal vision and language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025a.arxiv.org/abs/2412.04616
  38. [40]Yanzhao Zhang, Minxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 Embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025b.arxiv.org/abs/2506.05176
  39. [41]Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shu ailei Ma, Xin Jin, Wei Chen, and Yujun Shen. DreamLIP: Language-image pre-training with long captions. In European Conference on Computer Vision (ECCV), 2024.arxiv.org/abs/2403.17007

Paper details

vlnc:W_bxeczt2b
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 12

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents