betaFeedback

WorksComputer Science

Unimodal Augmentation-based SSL Objectives Induce Cross-Modal Alignment

Abstract

Recent works have observed implicit cross-modal alignment between independently trained unimodal models. However, the mechanisms underlying the emergence of this phenomenon remain unresolved. In this paper, we demonstrate that unimodal augmentation-based self-supervised objectives implicitly induce cross-modal alignment. We show that low unimodal losses force high cross-modal alignment, whereas a high unimodal loss prevents such alignment. We theoretically connect the widely used InfoNCE loss to local CKA, the popular measure of cross-modal alignment. We further provide extensive empirical evidence supporting this claim, both in controlled settings and across 53 unconstrained frozen text, vision, and audio unimodal encoders. In both cases, unimodal augmentation-based objectives control cross-modal alignment. Overall, our results provide a theoretical and empirical foundation for understanding the emergence of implicit cross-modal alignment by establishing a direct connection between unimodal training objectives and cross-modal structure.

Introduction

Recent discoveries in representation learning have provided empirical evidence that independently trained unimodal vision and language models converge toward a unified representation, particularly in terms of their local neighborhood structure ([29, 19]). This phenomenon is known as the Platonic Representation Hypothesis (PRH), and has been further supported by extensive subsequent work ([38, 1, 26]), including extensions to additional modalities ([14, 39, 15, 47]) and practical applications ([75, 76, 77, 20, 31, 22, 61, 40]).

However, the understanding of the causes underlying this emergent alignment remains limited. Most works either investigate the effect of scaling-related attributes of model size and capacity or the amount of training data on the strength of this phenomenon ([68] (Lobov et al. 2025)).

Preprint. or empirically address this question with explicit cross-modal factors. For example, shared input statistics and architectural biases [32, 65], or overlap in datasets and downstream tasks [39].

Meanwhile, Augmentation-based Self-Supervised Learning (Aug-SSL) has emerged as a leading training paradigm for training large models [21]. Popular models such as DINO for vision [52], Sentence-BERT for text [58], and BYOL-A for audio [49], have all been trained via Aug-SSL. While its power in unimodal settings is well-studied [71, 4], to the best of our knowledge, its effect on implicit cross-modal alignment has remained unexplored.

In this paper, we show that emergent cross-modal alignment (CMA) is a direct consequence of unimodal Aug-SSL objectives. That is, the unimodal objectives implicitly induce cross-modal alignment, even in the absence of explicit cross-modal supervision. The augmentations act as view-specific information removers, filtering out view-specific variation. The encoders then steer each embedding space toward the shared semantic structure underlying the modalities, thereby promoting cross-modal similarity. To the best of our knowledge, this is the first work to explain implicit cross-modal alignment via unimodal properties, connecting them both theoretically and in practice.

We begin with a theoretical investigation, directly connecting the widely used InfoNCE objective [51] to local CKA, a popular measure of CMA [10]. We prove that low unimodal Aug-SSL losses force high CMA (Thm. 1), whereas even a single high unimodal loss restricts the alignment (Thm. 2). These results hold independently of data size, model architecture, and other factors previously proposed to explain the emergence of cross-modal alignment.

We further provide extensive empirical evidence for these results in real-world settings, both in controlled experiments and across large-scale pretrained models and modalities. For example, Figure 1 shows that pre-trained text augmentation-based models consistently achieve higher CMA with a vision model than models of comparable size trained with next-token prediction (NTP). Moreover, augmentation-based models reach high CMA with substantially fewer parameters.

Cross-modal text-vision alignment across text models and model sizes

Figure 1. Augmentation-based objectives induce cross-modal alignment. Cross-modal text-vision alignment (mutual kk-nearest neighbors) as a function of model size (number of parameters) across different text models to DINOv2. Notably, augmentation-based models achieve higher cross-modal alignment for a given model size, and yield superior alignment even with significantly fewer parameters.

Contributions. Our contributions are summarized as follows: (1) We show that unimodal self-supervised objectives implicitly drive cross-modal alignment; (2) We theoretically establish a connection between the unimodal InfoNCE loss and cross-modal alignment as measured by local CKA; and (3) We empirically demonstrate that the unimodal Aug-SSL loss predicts cross-modal alignment across controlled experiments and large-scale pretrained models.

Related Work

Implicit Cross-Modal Alignment. Numerous studies have documented the emergence of implicit CMA in vision-language models [29, 38, 1, 45, 16, 26]. Subsequent work has shown that similar alignment phenomena arise across a broader range of modalities, including video-text [53], astronomical data [14], and scientific data [40, 15], among others. Beyond documenting its existence, several studies have examined how CMA depends on properties of the data and training setup, attributing its emergence to factors such as dataset and task overlap [39] or shared data characteristics [65]. However, these explanations remain empirical and do not establish a theoretical account of why CMA emerges.

Theory of Implicit CMA. Several recent works have developed theoretical perspectives on implicit cross-modal alignment. Lu et al. [44] formalize the alignment hypothesis using category theory. Insulla et al. [30] establish a connection between representation alignment and downstream task performance. However, these do not characterize the mechanisms underlying its emergence. Other studies investigate potential mechanisms that may give rise to this phenomenon [5, 79, 78]; however, their analyses rely on strong linearity assumptions about the data or learned representations, limiting their applicability to the highly nonlinear models used in practice. Lobashev [43] further assumes that the two modalities share input statistics, imposing an additional restriction on the data-generating process. In contrast, this paper provides a theoretical account of how implicit cross-modal alignment emerges directly from the unimodal training objective in general settings.

Augmentation-based SSL. Self-supervised learning (SSL) has emerged as a dominant training paradigm across a wide range of domains, including vision [9, 52, 25], language [11, 17, 42], audio [2, 27, 60, 49], video [66, 6], and scientific data [73]. In particular, augmentation-based SSL provides a simple and effective mechanism for learning from large numbers of self-generated training examples through different transformations of the same underlying samples. Prior works have investigated the properties induced by SSL training within individual modalities [71, 4]. Conversely, other works have demonstrated the effectiveness of explicit cross-modal SSL objectives for learning aligned representations [57, 74]. In contrast, this paper considers the implications of SSL on implicit cross-modal alignment, without any cross-modal objective.

Theoretical Analysis: From Unimodal InfoNCE to CMA

This section presents our theoretical results connecting unimodal InfoNCE and cross-modal alignment (CMA). We first formally define the setup, CMA, and unimodal augmentation-based self-supervised learning (Aug-SSL) (§ 3.1). We then establish their connection through two complementary results (§ 3.2). In Thm. 1, we show that low unimodal losses force high CMA. In Thm. 2, we show that a high unimodal loss for even one model restricts CMA, independently of other factors such as model architecture and training data size. Together, these results establish that unimodal augmentation-based training is a fundamental driver of implicit CMA. For brevity, we defer the complete derivations and proofs to App. A.

Setup

Notations (visualized in Figure 2a). Let M,U,V\mathcal{M}, \mathcal{U}, \mathcal{V} be manifolds representing shared semantics and view-specific information, respectively. Observations are generated by gX:M×U→Xg_{X}: \mathcal{M} \times\mathcal{U} \to\mathcal{X} and gY:M×V→Yg_{Y}: \mathcal{M} \times\mathcal{V} \to\mathcal{Y}, where X⊆Rdx\mathcal{X} \subseteq\mathbb{R}^{d_{x}}, Y⊆Rdy\mathcal{Y} \subseteq\mathbb{R}^{d_{y}}. Two trainable unimodal encoders, eX:X→EXe_{X}: \mathcal{X} \to\mathcal{E}_{X}, eY:Y→EYe_{Y}: \mathcal{Y} \to\mathcal{E}_{Y}, map the observations to normalized embedding spaces EX⊆Sdx−1\mathcal{E}_{X} \subseteq\mathbb{S}^{d_{x}-1}, EY⊆Sdy−1\mathcal{E}_{Y} \subseteq\mathbb{S}^{d_{y}-1}, respectively. Throughout, 0<τ≤10 < \tau\leq1 denotes the fixed temperature of the training objective.

Figure 2. (a) Theoretical setup visualization. Three manifolds M\mathcal{M}, U\mathcal{U}, and V\mathcal{V} generate two observation spaces X\mathcal{X} and Y\mathcal{Y} with shared semantics M\mathcal{M}. Two trainable encoders then map the observations to the corresponding embedding spaces EX\mathcal{E}_{X} and EY\mathcal{E}_{Y}. (b) Correlation between CKA-RBF and mkNN. mkNN is highly correlated with CKA-RBF across different choices of kk and the corresponding σ\sigma. Both Spearman (ρs\rho_{s}) and Pearson (rr) correlations are close to 1.

The phenomenon of implicit CMA dictates that independently trained unimodal encoders naturally develop highly aligned representations. To measure the alignment between the embedding spaces EX\mathcal{E}_X and EY\mathcal{E}_Y, we use the well-established Centered Kernel Alignment (CKA) metric [35], following recent works on implicit cross-modal alignment [29, 19]. We denote the observations x=gX(m,u)x = g_X(m,u), x′=gX(m′,u′)x' = g_X(m',u'), y=gY(m,v)y = g_Y(m,v), y′=gY(m′,v′)y' = g_Y(m',v'). That is, xx and yy share the same underlying semantics mm, and likewise x′x' and y′y' share m′m'. In practice, this shared semantic structure is obtained through paired data. The CKA is then often defined using the Hilbert-Schmidt Independence Criterion (HSIC), which is

HSIC⁡(EX,EY)=E[kXc(eX(x),eX(x′)) kYc(eY(y),eY(y′))]\operatorname{HSIC}(\mathcal{E}_X,\mathcal{E}_Y)=\mathbb{E}\left[k_X^c(e_X(x),e_X(x'))\,k_Y^c(e_Y(y),e_Y(y'))\right]

where kck^c is the doubly centered kernel kc=k−E[k∣x]−E[k∣x′]+E[k]k^c = k - \mathbb{E}[k|x] - \mathbb{E}[k|x'] + \mathbb{E}[k]. CKA then measures the correlation between the centered kernels, giving:

CKA⁡(EX,EY)=HSIC⁡(EX,EY)HSIC⁡(EX,EX)HSIC⁡(EY,EY)∈[0,1],(1)\operatorname{CKA}(\mathcal{E}_X,\mathcal{E}_Y)=\frac{\operatorname{HSIC}(\mathcal{E}_X,\mathcal{E}_Y)}{\sqrt{\operatorname{HSIC}(\mathcal{E}_X,\mathcal{E}_X)\operatorname{HSIC}(\mathcal{E}_Y,\mathcal{E}_Y)}}\in[0,1], \tag*{(1)}

Throughout the paper, we use the CKA-RBF version of CKA [35], with the Gaussian kernel k(z,z′)=e−∥z−z′∥2/2σ2k(z,z')=e^{-\lVert z-z'\rVert^2/2\sigma^2}, and denote the kernel variances σX2=HSIC⁡(EX,EX)\sigma_X^2=\operatorname{HSIC}(\mathcal{E}_X,\mathcal{E}_X), σY2=HSIC⁡(EY,EY)\sigma_Y^2=\operatorname{HSIC}(\mathcal{E}_Y,\mathcal{E}_Y). In particular, we assume σX,σY>0\sigma_X,\sigma_Y>0 so the CKA is defined.

The Gaussian kernel emphasizes local structure, making this measure particularly suitable for studying the local nature of implicit cross-modal alignment observed in recent work [19]. Moreover, as shown in Figure 2b, CKA-RBF is strongly correlated with the mutual kk-nearest-neighbors (mkNN) metric used in recent works, with the kernel bandwidth σ\sigma playing a role analogous to the neighborhood parameter kk (see App. § 17).

Augmentation-based SSL. We define an augmentation as a resampling of both the semantics and the view, while preserving the underlying semantic structure. Specifically:

x+:=aug⁡(x)=gX(m+,u+),m+∼P(m,⋅),u+∼νU independent(2)x^+ := \operatorname{aug}(x)=g_X(m^+,u^+),\qquad m^+\sim P(m,\cdot),\qquad u^+\sim\nu_U\ \text{independent} \tag*{(2)}

where P(m,⋅):=J(m,⋅) μP(m,\cdot):=J(m,\cdot)\,\mu is a Markov kernel on M\mathcal{M} with density JJ. This definition aligns with prior theoretical studies of augmentation-based SSL [70], while not restricting the augmentation to exactly preserve the semantics. We exclude the degenerate case J≡1J\equiv1, in which the augmentation resamples the semantics independently and carries no signal.

An Aug-SSL model is trained to maximize the similarity between augmented samples, while preventing collapse to a constant representation, for instance by contrasting against randomly sampled observations. A popular objective for this purpose is the InfoNCE loss [51], which has been widely adopted in vision [9, 24], text [58, 17], audio [49], and other domains [54, 73]. Let x,x′∈Xx,x'\in\mathcal{X} denote observations and let x+=aug⁡(x)x^+=\operatorname{aug}(x) be an augmentation of xx. The KK-negative InfoNCE loss is

ℓX(K)=E[−log⁡exp⁡(⟨eX(x),eX(x+)⟩/τ)exp⁡(⟨eX(x),eX(x+)⟩/τ)+∑k=1Kexp⁡(⟨eX(x),eX(xk)⟩/τ)],(3)\ell_X^{(K)}=\mathbb{E}\left[-\log\frac{\exp\left(\langle e_X(x),e_X(x^+)\rangle/\tau\right)}{\exp\left(\langle e_X(x),e_X(x^+)\rangle/\tau\right)+\sum_{k=1}^{K}\exp\left(\langle e_X(x),e_X(x_k)\rangle/\tau\right)}\right], \tag*{(3)}

where KK is a hyperparameter controlling the number of negative samples. Following Wang & Isola [71] (Thm. 1), we use its population form:

NX=lim⁡K→∞(ℓX(K)−log⁡K)=−1τE[⟨eX(x),eX(x+)⟩]+Ex[log⁡Ex′[e⟨eX(x),eX(x′)⟩/τ]].N_X=\lim_{K\to\infty}\left(\ell_X^{(K)}-\log K\right)=-\frac{1}{\tau}\mathbb{E}\left[\langle e_X(x),e_X(x^+)\rangle\right]+\mathbb{E}_x\left[\log\mathbb{E}_{x'}\left[e^{\langle e_X(x),e_X(x')\rangle/\tau}\right]\right].

For convenience, we normalize the objective as follows, with both ends attainable (Lemma 8):

LX=NX+1/τRτ∈[0,1],L_X=\frac{N_X+1/\tau}{R_\tau}\in[0,1],

where Rτ=2τ+log⁡cosh⁡1τR_\tau=\frac{2}{\tau}+\log\cosh\frac{1}{\tau}. LYL_Y is analogously defined for Y\mathcal{Y}.

Connecting InfoNCE and local CKA

We now present the theoretical results that formalize the connection between the unimodal Aug-SSL losses (InfoNCE) and CMA under the setup above. The two results establish complementary directions of this relationship. Specifically, Thm. 1 shows that low losses in both modalities guarantee high CMA, while Thm. 2 shows that high loss in either modality prevents alignment.

Implicit alignment. The bound in Thm. 1 utilizes the concentration strength of the augmentations, ∥J−1∥L2=:χ\lVert J-1\rVert_{L^2} =: \chi, where J(m,m+)J(m,m^+) is the likelihood ratio of m+m^+ being an augmentation of mm versus an independent sample. Thus, χ\chi quantifies the semantic information retained by augmentations: χ=0\chi=0 when J≡1J\equiv1 and augmentations ignore the original semantics, and χ=∞\chi=\infty when they preserve them exactly. We consider the practical regime 0<χ<∞0<\chi<\infty, where augmentations alter but retain semantics. Together with the InfoNCE temperature τ\tau and CKA-RBF scale σ\sigma, this allows us to lower-bound CKA in terms of the unimodal losses.

References

  1. [1]Santiago Acevedo, Andrea Mascaretti, Riccardo Rende, Matéo Mahaut, Marco Baroni, and Alessandro Laio. A quantitative analysis of semantic information in deep representations of text and images. arXiv preprint arXiv:2505.17101, 2025.arxiv.org/abs/2505.17101
  2. [2]Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460, 2020.arxiv.org/abs/2006.11477
  3. [3]Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International conference on machine learning, pp. 1298–1312. PMLR, 2022.arxiv.org/abs/2202.03555
  4. [4]Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. Advances in Neural Information Processing Systems, 35:26671–26685, 2022.arxiv.org/abs/2205.11508
  5. [5]Kiril Bangachev, Guy Bresler, and Yury Polyanskiy. Representation alignment rests on linear structure. arXiv preprint arXiv:2605.28870, 2026.arxiv.org/abs/2605.28870
  6. [6]Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471, 2024.arxiv.org/abs/2404.08471
  7. [7]Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Afzal Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International conference on machine learning, pp. 2397–2430. PMLR, 2023.arxiv.org/abs/2304.01373
  8. [8]Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6): 1505–1518, 2022.DOI
  9. [9]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PmLR, 2020.arxiv.org/abs/2002.05709
  10. [10]Nello Cristianini, John Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. Advances in neural information processing systems, 14, 2001.
  11. [11]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186, 2019.arxiv.org/abs/1810.04805
  12. [12]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.arxiv.org/abs/2010.11929
  13. [13]Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736–740. IEEE, 2020.arxiv.org/abs/1910.09387
  14. [14]Kshitij Duraphe, Michael J Smith, Shashwat Sourav, John F Wu, et al. The platonic universe: Do foundation models see the same sky? arXiv preprint arXiv:2509.19453, 2025.arxiv.org/abs/2509.19453
  15. [15]Sathya Edamadaka, Soojung Yang, Ju Li, and Rafael Gómez-Bombarelli. Universally converging representations of matter across scientific foundation models. arXiv preprint arXiv:2512.03750, 2025.arxiv.org/abs/2512.03750
  16. [16]David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language-free visual representation learning. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1–13. IEEE, 2025.arxiv.org/abs/2504.01017
  17. [17]Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 conference on empirical methods in natural language processing, pp. 6894–6910, 2021.arxiv.org/abs/2104.08821
  18. [18]Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. URL https://github.com/openlm-research/open_llama.
  19. [19]Fabian Gröger, Shuo Wen, and Maria Brbić. Revisiting the platonic representation hypothesis: An aristotelian view. arXiv preprint arXiv:2602.14486, 2026a.arxiv.org/abs/2602.14486
  20. [20]Fabian Gröger, Shuo Wen, Huyen Le, and Maria Brbic. With limited data for multimodal alignment, let the structure guide you. Advances in Neural Information Processing Systems, 38:151747–151776, 2026b.
  21. [21]Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self-supervised learning: Algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9052–9071, 2024.arxiv.org/abs/2301.05712
  22. [22]Sharut Gupta, Shobhita Sundaram, Chenyu Wang, Stefanie Jegelka, and Phillip Isola. Better together: Leveraging unpaired multimodal data for stronger unimodal models. In International Conference on Learning Representations, volume 2026, pp. 73970–74025, 2026.
  23. [23]Emman Haider, Daniel Perez-Becker, Thomas Portet, Piyush Madan, Amit Garg, Atabak Ashfaq, David Majercak, Wen Wen, Dongwoo Kim, Ziyi Yang, et al. Phi-3 safety post-training: Aligning language models with a" break-fix" cycle. arXiv preprint arXiv:2407.13833, 2024.
  24. [24]Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 9726–9735. IEEE, 2020.arxiv.org/abs/1911.05722
  25. [25]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 15979–15988. IEEE, 2022.arxiv.org/abs/2111.06377
  26. [26]Zoe Wanying He, Sean Trott, and Meenakshi Khosla. Seeing through words, speaking through pixels: Deep representational alignment between vision and language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 35645–35660, 2025.
  27. [27]Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460, 2021.
  28. [28]Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in neural information processing systems, 35:28708–28720, 2022.arxiv.org/abs/2207.06405
  29. [29]Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024.arxiv.org/abs/2405.07987
  30. [30]Francesco Maria Gabriele Insulla, Shuo Huang, and Lorenzo Rosasco. Towards a learning theory of representation alignment. In International Conference on Learning Representations, volume 2025, pp. 37594–37608, 2025.
  31. [31]Rishi Jha, Collin Zhang, Vitaly Shmatikov, and John Morris. Harnessing the universal geometry of embeddings. Advances in Neural Information Processing Systems, 38:45963–45987, 2026.arxiv.org/abs/2505.12540
  32. [32]Chaitanya Kapoor, Sudhanshu Srivastava, and Meenakshi Khosla. Bridging critical gaps in convergent learning: How representational alignment evolves across layers, training, and distribution shifts. Advances in Neural Information Processing Systems, 38:140352–140366, 2026.
  33. [33]Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. *Advances in neural information processing systems*, 33:18661–18673, 2020.arxiv.org/abs/2004.11362
  34. [34]Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pp. 119–132, 2019.DOI
  35. [35]Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In *International conference on machine learning*, pp. 3519–3529. PMLR, 2019.arxiv.org/abs/1905.00414
  36. [36]Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. 2022.arxiv.org/abs/2211.05100
  37. [37]Kuang-Huei Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. *Advances in Neural Information Processing Systems*, 34:19538–19552, 2021.arxiv.org/abs/2109.12909
  38. [38]Jiaang Li, Yova Kementchedjhieva, Constanza Fierro, and Anders Søgaard. Do vision and language models share concepts? a vector space alignment study. *Transactions of the Association for Computational Linguistics*, 12:1232–1249, 2024.arxiv.org/abs/2302.06555
  39. [39]Zeyu Michael Li, Hung Anh Vu, Damilola Awofisayo, and Emily Wenger. Causes and consequences of representational similarity in machine learning models. *arXiv preprint arXiv:2505.13899*, 2025.arxiv.org/abs/2505.13899
  40. [40]Zhenzhu Li and Aron Walsh. Platonic representation of foundation machine learning interatomic potentials. *Nature Machine Intelligence*, pp. 1–11, 2026.DOI
  41. [41]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In *European conference on computer vision*, pp. 740–755. Springer, 2014.arxiv.org/abs/1405.0312
  42. [42]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. *arXiv preprint arXiv:1907.11692*, 2019.arxiv.org/abs/1907.11692
  43. [43]Alexander Lobashev. An information-geometric view of the platonic hypothesis. In *NeurIPS 2025 Workshop on Symmetry and Geometry in Neural Representations*, 2025.
  44. [44]Jianglin Lu, Hailing Wang, Kuo Yang, Yitian Zhang, Simon Jenni, and Yun Fu. The indra representation hypothesis for multimodal alignment. *Advances in Neural Information Processing Systems*, 38:156980–157000, 2026.
  45. [45]JW Miller. Testing the platonic representation hypothesis via representation-controlled tokenization. 2026.
  46. [46]Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schölkopf. Kernel mean embedding of distributions: A review and beyond. *Foundations and Trends® in Machine Learning*, 10(1-2):1–141, 2017. ISSN 1935-8245. doi: 10.1561/2200000060. URL http://dx.doi.org/10.1561/2200000060.
  47. [47]Jerry Ngo and Yoon Kim. What do language models hear? probing for auditory representations in language models. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 5435–5448, 2024.arxiv.org/abs/2402.16998
  48. [48]Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In *Findings of the association for computational linguistics: ACL 2022*, pp. 1864–1874, 2022.arxiv.org/abs/2108.08877
  49. [49]Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Self-supervised learning for general-purpose audio representation. In *2021 International Joint Conference on Neural Networks (IJCNN)*, pp. 1–8. IEEE, 2021.arxiv.org/abs/2103.06695
  50. [50]Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Exploring pre-trained general-purpose audio representations. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:137–151, 2022.
  51. [51]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.arxiv.org/abs/1807.03748
  52. [52]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
  53. [53]Maks Ovsjanikov, Viorica Patraucean, Leonidas Guibas, Tyler Zhu, and Tengda Han. Dynamic reflections: Probing video representations with text alignment. In International Conference on Learning Representations, volume 2026, pp. 71224–71250, 2026.
  54. [54]Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 6960–6970. IEEE, 2021.arxiv.org/abs/2008.03800
  55. [55]Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Daiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115.
  56. [56]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  57. [57]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
  58. [58]Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp. 3982–3992, 2019.arxiv.org/abs/1908.10084
  59. [59]Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.
  60. [60]Aaqib Saeed, David Grangier, and Neil Zeghidour. Contrastive learning of general-purpose audio representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3875–3879. IEEE, 2021.arxiv.org/abs/2010.10915
  61. [61]Dominik Schnaus, Nikita Araslanov, and Daniel Cremers. It’s a (blind) match! towards vision-language correspondence without parallel data. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24983–24992. IEEE, 2025.arxiv.org/abs/2503.24129
  62. [62]Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 2443–2449, 2021.DOI
  63. [63]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreyev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Carbune, Michelle Casbon, et al. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026.
  64. [64]Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36:48382–48402, 2023.arxiv.org/abs/2306.00984
  65. [65]Megan Tjandrasuwita, Chanakya Ekbote, Liu Ziyin, and Paul Pu Liang. Understanding the emergence of multimodal representation alignment. arXiv preprint arXiv:2502.16282, 2025.arxiv.org/abs/2502.16282
  66. [66]Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078–10093, 2022.
  67. [67]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https://arxiv.org/abs/2302.13971.
  68. [68]Ali Hussaini Umar and Alessandro Laio. Signal-to-noise ratio and sample size govern representational alignment in neural networks. arXiv preprint arXiv:2605.26973, 2026.
  69. [69]Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1805.10965
  70. [70]Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451–16467, 2021.arxiv.org/abs/2106.04619
  71. [71]Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning, pp. 9929–9939. PMLR, 2020.arxiv.org/abs/2005.10242
  72. [72]Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776–5788, 2020.arxiv.org/abs/2002.10957
  73. [73]Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279–287, 2022.DOI
  74. [74]Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4563–4567. IEEE, 2022.
  75. [75]Amitai Yacobi, Nir Ben-Ari, Ronen Talmon, and Uri Shaham. Learning shared representations from unpaired data. Advances in Neural Information Processing Systems, 38:46634–46666, 2026.arxiv.org/abs/2505.21524
  76. [76]Lauren Hyoseo Yoon, Yisong Yue, and Been Kim. Escaping plato’s cave: Jam for aligning independently trained vision and language models. arXiv preprint arXiv:2507.01201, 2025.arxiv.org/abs/2507.01201
  77. [77]Le Zhang, Qian Yang, and Aishwarya Agrawal. Assessing and learning alignment of unimodal vision and language models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14604–14614. IEEE, 2025.arxiv.org/abs/2412.04616
  78. [78]Liu Ziyin and Isaac Chuang. Proof of a perfect platonic representation hypothesis. arXiv preprint arXiv:2507.01098, 2025.arxiv.org/abs/2507.01098
  79. [79]Liu Ziyin, Yizhou Xu, and Isaac Chuang. Neural thermodynamics: Entropic forces in deep and universal representation learning. Advances in Neural Information Processing Systems, 38:56057–56089, 2026.arxiv.org/abs/2505.12387

Paper details

vlnc:W_gc3u744e
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents