betaFeedback

WorksComputer Science

Provable Cross-Modal Transfer of Hierarchical Concept Structure in CLIP Embeddings

Abstract

Contrastive vision-language models such as CLIP are trained to align images and text, not to represent taxonomies. Yet empirical studies show that CLIP embeddings exhibit hierarchical structure aligned with the hierarchy of semantic concepts. A theoretical characterization of why such hierarchy emerges from a non-hierarchical objective remains open. In this paper, we formalize this phenomenon as Hierarchical Angular Separation (HAS), a geometric property whereby each non-leaf concept embedding is more similar to its descendants than to off-branch concepts. Empirically, we show that HAS holds approximately in raw images and randomly initialized CLIP image embeddings, while in text embeddings it holds approximately only after contrastive learning. To theoretically explain this asymmetry, we analyze an unconstrained features model in which the image embeddings are fixed and only the text embeddings are trained. At the global minimizer, the text-side HAS violation percentage at each node is at most the image-side HAS violation and near-violation percentage. The key mechanism is the hierarchical co-occurrence structure of image-text pairs, which biases text embeddings toward root-centered image prototypes. For a linear contrastive loss, we derive a closed-form solution that makes this mechanism explicit. We extend the analysis to the InfoNCE loss and establish analogous guarantees under sufficient regularization. Synthetic experiments validate the theory and show that HAS is transferred even beyond the regimes covered by our analysis.

Introduction

Given a collection of image-text pairs, Contrastive Language-Image Pretraining (CLIP) learns text and image encoders by aligning paired embeddings and separating unpaired ones. The learned representations enable strong transfer across tasks such as image classification, visual question answering, and multimodal retrieval [1, 2, 3].

One possible explanation for this empirical success is that CLIP embeddings implicitly encode rich semantic and geometric structure. In particular, embeddings of semantic concepts have been shown to organize into linear subspaces [4, 5] and to reflect hierarchical relationships aligned with noun hierarchies in WordNet [6, 7, 8, 9]. For example, a cat and a dog are both mammals, and a mammal and a bird are both animals. Recent work has attempted to characterize such hierarchical structure through geometric conditions, for example via orthogonality relations between embeddings of semantic concepts [10, 11]. However, these conditions fail to capture a fundamental property of semantic hierarchies: a concept should be more similar to its descendants than to concepts from unrelated branches [12, 13, 14, 7]. Empirical studies further suggest that CLIP image embeddings encode hierarchy through separated subspaces [9], but this evidence does not quantify the extent of such structure across models and scales, nor does it provide a principled explanation of its origin.

A natural hypothesis is that hierarchical structure is learned through contrastive learning: if one modality encodes semantic relationships, alignment should transfer them to the other. Our experiments show that the two modalities differ. Approximate hierarchical structure is already present in raw-pixel image data and this structure is preserved by a randomly initialized CLIP encoder, implying that the CLIP embedding space natively exhibits hierarchy prior to any training. In contrast, raw text consists of discrete symbols whose semantic relationships are not reflected in their randomly initialized embeddings; any geometric structure must therefore be learned through training.

These observations raise a central question: how does contrastive learning transform and propagate hierarchical structure across modalities? In particular, why does alignment amplify and transfer structure from image to text embeddings, rather than suppress it? While the origin of hierarchical structure in image data remains open, our focus in this paper is on understanding how contrastive learning transfers such structure to other modalities once it is present in one modality.

To study this phenomenon, we formalize hierarchical structure as a geometric property called Hierarchical Angular Separation (HAS\mathcal{H}\mathcal{A}\mathcal{S}): given a taxonomy of concepts, an ancestor has higher cosine similarity to each of its descendants concepts than to any of its off-branch concepts. We quantify violations from this ordering using the HAS\mathcal{H}\mathcal{A}\mathcal{S} violation percentage (HAS\mathcal{H}\mathcal{A}\mathcal{S}-Viol): the percentage of off-branch concepts that violate the HAS\mathcal{H}\mathcal{A}\mathcal{S} condition. Here, off-branch concepts exclude both the concept’s subtree and its ancestors. Our experiments across CLIP models of varying scales show that image embeddings natively exhibit low HAS\mathcal{H}\mathcal{A}\mathcal{S}-Viol at random initialization, whereas text embeddings begin with high HAS\mathcal{H}\mathcal{A}\mathcal{S}-Viol and achieve low values only after contrastive training.

These empirical observations motivate the need for a theoretical analysis. However, analyzing contrastive learning theoretically is challenging due to the non-convexity of the optimization problem. To address this, we adopt the Unconstrained Features Model (UFM) [15, 16, 17], which abstracts away the deep network by optimizing directly over last-layer features. Within this framework, we show that hierarchical structure in image embeddings transfers to text embeddings under contrastive learning. The key mechanism, illustrated in Figure 1, is the hierarchical co-occurrence structure of image-text pairs: although each image corresponds to a specific concept (e.g., dog), its paired text may correspond to that concept or to one of its hypernyms (e.g., mammal, animal). This structure induces a bias that propagates hierarchical relationships from image to text embeddings.

Image not extracted

Figure 1. How contrastive learning transfers hierarchy from image to text embeddings. Top-left: the image prototypes are arranged so that each concept is closer to its descendants than to concepts in other subtrees. The blue and red dotted lines indicate the boundary of the dog and cat subtrees, respectively. Top-right: each image of a labrador is a variation of the labrador prototype, and its caption may pair with any concept on the path from the root to that leaf, so an image can be captioned labrador, dog or animal. Bottom-left: the pairing also means that the caption dog pairs with images of dog or any of its descendants, so the average of these images approaches the mean of the dog subtree. Bottom-right: contrastive learning pulls each text embedding toward the corresponding subtree mean, so the text embeddings inherit the arrangement of the image prototypes.

Paper Contributions. In this paper, we empirically characterize the hierarchical structure of CLIP embeddings and theoretically explain how contrastive learning transfers this structure from image to text embeddings:

  1. Empirical characterization of the hierarchical structure. Using WordNet, we show that text embeddings have high HAS\mathcal{H}\mathcal{A}\mathcal{S}-Viol at random initialization and low HAS\mathcal{H}\mathcal{A}\mathcal{S}-Viol after pretraining, while image embeddings already have low HAS\mathcal{H}\mathcal{A}\mathcal{S}-Viol at initialization. This reveals a systematic asymmetry: hierarchy is learned in text but partly present in image embeddings prior to training.

  1. Hierarchical co-occurrence transfers image geometry to text at the linear-loss optimum. Under the UFM with hierarchical co-occurrence, we derive a closed-form minimizer of a linear contrastive loss [18], showing that optimal text embeddings align with root-centered image prototypes. The text-side HAS\mathcal{H}\mathcal{A}\mathcal{S}-Viol at each anchor is at most the image-side percentage of off-branch nodes that violate or nearly violate HAS\mathcal{H}\mathcal{A}\mathcal{S}.

  1. Extension to InfoNCE and robustness beyond theory. We establish an analogous bound on text HAS\mathcal{H}\mathcal{A}\mathcal{S}-Viol for the InfoNCE loss used in CLIP under sufficient regularization. Furthermore, we empirically show that transfer occurs even when the regularization condition does not hold.

Related Work

Unconstrained Features Model. The UFM was originally used to explain Neural Collapse [19]. Subsequent work showed that the Simplex Equiangular Tight Frame is the unique optimal geometry for cross-entropy with weight decay [15, 16, 17] and for supervised contrastive losses [20, 21]. However, these analyses focus on classification with mutually exclusive classes. We extend the UFM to contrastive learning with hierarchically structured image-text pairings and characterize the geometry that emerges at global minimizers.

Geometry of Representations in Foundation Models. Recent work has documented geometric structure in pretrained embedding spaces. One line of work shows that semantic concepts form linear or compositional subspaces [22, 23], exhibit hierarchical organization [10, 9, 7], or give rise to phenomena such as the modality gap [24, 25]. Relatedly, Zhao and Thrampoulidis Zhao and Thrampoulidis [26] infer a semantic hierarchy from sign patterns in a learned concept basis for next-token prediction. Nava and Wyart Nava and Wyart [27] derive coarse-to-fine spectral organization from hierarchy-dependent word co-occurrence in word2vec-style embeddings. We instead study a given tree of object categories and explain how HAS\mathcal{H}\mathcal{A}\mathcal{S} transfers from image to text embeddings during contrastive pretraining.

Learning Theory with Hierarchy Models. Another line of work studies how hierarchical latent structure shapes learning and generation, including hierarchical concept learning in language models [28, 29], emergent properties under hierarchical contrastive learning [30], and phase transitions in diffusion models signaling an underlying latent hierarchy [31, 32]. Closest to our work, several papers [31, 33] analyze the Random Hierarchy Model, a supervised setting in which the sample for a class is generated by a hierarchical composition of latent features. However, these works study compositional hierarchies of features, while we instead study hierarchies over object categories.

Hierarchical Representation Learning. Traditionally, hierarchical information is explicitly encoded into models through specialized architectures or objectives. A popular approach is to embed hierarchies in hyperbolic space, exploiting its exponential volume growth [34, 12, 13, 14, 35]. In contrast, we analyze when implicit hierarchical structure transfers in self-supervised learning.

Contrastive Learning Theory. Theoretical analyses have connected contrastive objectives to spectral decompositions of augmentation graphs [36, 37], and have shown that contrastive objectives can invert the data-generating process [38]. A separate line of work provides theoretical and empirical evidence that self-supervised representations cluster at multiple hierarchical levels [39, 40]. By contrast, we analyze how the contrastive loss interacts with hierarchical co-occurrence structures.

Hierarchical Angular Separation and Its Modality Asymmetry in CLIP

We first define a hierarchical similarity ordering for embeddings, Hierarchical Angular Separation (HAS\mathcal{H}AS). In this definition, each concept embedding is closer to its descendant concept embeddings than to any of its off-branch concepts. Pretraining creates a modality asymmetry in CLIP’s hierarchical structure: text embeddings acquire HAS\mathcal{H}AS after training, whereas image embeddings exhibit HAS\mathcal{H}AS approximately at initialization and strengthened HAS\mathcal{H}AS after training. We formalize HAS\mathcal{H}AS in Section 3.1 and measure it across CLIP variants in Section 3.2.

Definition of Hierarchical Angular Separation

Datasets of natural images and language often contain semantic concepts that can be organized into hierarchies. For example, a cat and a dog are both mammals, and a mammal and a bird are both animals. Below, we introduce the notation used to describe such hierarchies.

Hierarchy. Let T=(V,E)\mathcal{T} = (V,E) be a rooted tree with ∣V∣=M|V| = M nodes, leaf set Vleaf⊆VV_{\mathrm{leaf}} \subseteq V, and root node root∈V\mathrm{root} \in V. Fix an enumeration of the nodes and identify VV with [M]:={1,…,M}[M] := \{1,\ldots,M\}. Each u∈Vu \in V corresponds to a synset (e.g., mammal). For u∈Vu \in V, let Anc⁡(u)\operatorname{Anc}(u) denote its ancestors, Desc⁡(u)\operatorname{Desc}(u) its descendants, and Sub⁡(u):={u}∪Desc⁡(u)\operatorname{Sub}(u) := \{u\} \cup\operatorname{Desc}(u) the subtree rooted at uu.

Concept Embeddings. For each node k∈Vk \in V, we associate a concept embedding hk∈Rd\mathbf{h}_k \in\mathbb{R}^d, and write the embedding of the hierarchy T\mathcal{T} as the collection {hk:k∈V}\{\mathbf{h}_k : k \in V\}. In practice, concept embeddings are obtained using text and image encoders in two common ways: (1) instance aggregation [22, 41, 42], which averages all image embeddings zj∈Rd\mathbf{z}_j \in\mathbb{R}^d corresponding to a node u∈Vu \in V (e.g., averaging over all dog images to estimate the concept embedding for dog); (2) direct concept encoding [1, 23], which encodes the concept string (e.g., image of a dog) to obtain a text embedding.

Hierarchical Angular Separation. We define HAS\mathcal{H}AS as a Euclidean similarity formalization of hierarchy based on angular separation, applicable to either image or text embeddings. To make this precise, for non-zero x,y∈Rd\mathbf{x}, \mathbf{y} \in\mathbb{R}^d, we define cos⁡(x,y):=x⊤y/(∥x∥∥y∥)\cos(\mathbf{x},\mathbf{y}) := \mathbf{x}^{\top}\mathbf{y}/(\lVert\mathbf{x}\rVert\lVert\mathbf{y}\rVert).

Definition 3.1 (Hierarchical Angular Separation (HAS\mathcal{H}AS)). The concept embeddings {hk}k∈V\{\mathbf{h}_k\}_{k\in V} of hierarchy T\mathcal{T} (obtained by either instance aggregation or direct concept encoding) satisfy HAS\mathcal{H}AS if for every non-leaf node u∈Vu \in V, every descendant v∈Desc⁡(u)v \in\operatorname{Desc}(u), and every node outside the subtree and not an ancestor, w∉Sub⁡(u)∪Anc⁡(u)w \notin\operatorname{Sub}(u) \cup\operatorname{Anc}(u), we have

cos⁡(hu,hv)>cos⁡(hu,hw).(1)\cos(\mathbf{h}_u,\mathbf{h}_v) > \cos(\mathbf{h}_u,\mathbf{h}_w). \tag*{(1)}

Nguyen et al. [9] use instance aggregation to estimate concept embeddings, and claim to have observed subtree separation in pretrained CLIP image embeddings without specifying how.

HAS\mathcal{H}AS differs from prior hierarchy-learning objectives that prescribe hierarchical structure during training. Hyperbolic entailment cones [12, 13, 14] and Poincaré embeddings [34] optimize non-Euclidean embeddings with losses designed to enforce hierarchical relations. In contrast, and as we compare in detail in Appendix § 7, we define HAS\mathcal{H}AS descriptively rather than prescriptively: Eq. (1) is a Euclidean cosine inequality measured in the learned CLIP space, where the observed angles determine the descendant cone around huh_u rather than the training objective imposing it. To quantify how far a set of embeddings is from satisfying Definition 3.1, we use the following per-node metric.

Definition 3.2 (HAS\mathcal{H}AS Violation Percentage). For a non-root non-leaf node uu, we first define the descendant threshold, the minimum cosine similarity between uu and its least similar descendant, as su:=min⁡v∈Desc⁡(u)cos⁡(hu,hv)s_u := \min_{v\in\operatorname{Desc}(u)} \cos(\mathbf{h}_u,\mathbf{h}_v). The HAS\mathcal{H}AS violation percentage at uu is then the percentage of the set of off-branch (not in the subtree and not an ancestor) nodes {w}\{w\} that has a larger cosine similarity with uu than sus_u

HAS-Viol⁡(u):=∣{w∉Sub⁡(u)∪Anc⁡(u):cos⁡(hu,hw)≥su}∣∣{w∉Sub⁡(u)∪Anc⁡(u)}∣×100%.(2)\mathcal{H}AS\text{-}\operatorname{Viol}(u) := \frac{|\{w \notin\operatorname{Sub}(u) \cup\operatorname{Anc}(u) : \cos(\mathbf{h}_u,\mathbf{h}_w) \ge s_u\}|}{|\{w \notin\operatorname{Sub}(u) \cup\operatorname{Anc}(u)\}|} \times100\%. \tag*{(2)}

HAS-Viol⁡(u)=0\mathcal{H}AS\text{-}\operatorname{Viol}(u) = 0 for every non-root non-leaf u∈Vu \in V if and only if HAS\mathcal{H}AS (Definition 3.1) holds. We are now ready to measure HAS\mathcal{H}AS in CLIP embeddings.

Hierarchical Angular Separation in CLIP embeddings

We measure HAS\mathcal{H}AS-Viol on the ImageNet-1K training split [43] and the WordNet noun hierarchy [6].1 We evaluate both the text and image embeddings of two pretrained CLIP ViT-L/14 checkpoints (OpenAI CLIP [1] and OpenCLIP [44] trained on DataComp-1B [45]) and a randomly initialized ViT-L/14. To reduce the effect of outliers, we replace the descendant threshold su=min⁡v∈Desc⁡(u)cos⁡(hu,hv)s_u = \min_{v \in\operatorname{Desc}(u)} \cos(h_u,h_v) by its 95th percentile over Desc⁡(u)\operatorname{Desc}(u). Corresponding results on ViT-B/32, ViT-B/16, ViT-L/14@336px, SigLIP2 [46], MetaCLIP2 [47], and on the CUB-200 taxonomy [48], are deferred to Appendix § 16.§ 1.

Text embeddings: HAS\mathcal{H}AS is acquired through pretraining. In Figure 2 (top), both pretrained text models have low violation percentages across depths, whereas the randomly initialized text model has substantially larger violations across the tree. Hence, for text concept embeddings, HAS\mathcal{H}AS is a learned property of contrastive pretraining. Additional results estimating concept embeddings by instance aggregation reach the same conclusion (Appendix § 16.§ 1).

Image not extracted

Figure 2. Distribution of HAS\mathcal{H}AS-Viol vs depth on ViT-L/14 CLIP embeddings over non-root and non-leaf nodes. Top: text embeddings show that HAS\mathcal{H}AS is acquired through pretraining. Bottom: image embeddings show approximate HAS\mathcal{H}AS at random initialization and stronger HAS\mathcal{H}AS after pretraining.

Image embeddings: HAS\mathcal{H}AS is present in raw pixels and strengthened by pretraining. In Figure 2 (bottom), pretrained image embeddings have violation percentages concentrated near zero across depths. The randomly initialized ViT-L/14 has substantially lower violations than randomly initialized text embeddings, although its violations remain non-negligible.

In Table 1, raw pixels and randomly initialized image embeddings separate from their own hierarchy-shuffled nulls, whereas i.i.d. Gaussian vectors do not. Approximate image-side HAS\mathcal{H}AS therefore comes from raw image data, remains after a randomly initialized encoder, and strengthens with contrastive pretraining, while text-side HAS\mathcal{H}\mathcal{A}\mathcal{S} is acquired only through pretraining.

RepresentationHAS\mathcal{H}AS-Viol (%) ↓HAS\mathcal{H}AS-Viol shuffled (%) ↓
Raw pixels26.5150.95 ± 0.29
Randomly initialized ViT-L/1427.88 ± 0.2456.80 ± 0.42
Pretrained OpenAI CLIP ViT-L/142.9532.93 ± 0.48
i.i.d. Gaussian vectors (d=768d = 768)76.02 ± 0.9776.33 ± 0.93

Table 1. Mean image-side HAS\mathcal{H}AS-Viol. Each representation is paired with its own hierarchy-shuffled null. Null values report mean ± standard deviation over five hierarchy shuffles.

Provable Transfer of Hierarchical Angular Separation from Image to Text Embeddings

Motivated by the observed empirical asymmetry, we study theoretically how contrastive learning transfers pre-existing image geometry to text. We use an Unconstrained Features Model (UFM) in which the image embeddings are fixed and only the text embeddings are learned. Under hierarchical co-occurrence, each text concept aggregates images from its descendant leaves, and contrastive learning therefore aligns the text embedding with the corresponding root-centered image prototype. We first derive this mechanism in closed form for a linear contrastive loss (§ 4.2), then show that regularized symmetric InfoNCE inherits it under common assumptions (§ 4.3). We summarize the relation between these theoretical results in Figure 3. We also show that the transfer is robust to weaker regularization than required by theory (§ 4.4).

Image not extracted

Figure 3. Overview of the HAS\mathcal{H}\mathcal{A}\mathcal{S} transfer results. Under assumptions on the data model, both losses decompose the text embeddings into prototypes plus perturbations. Bounding the perturbations relative to the prototype norms gives a bound on text-side violations in terms of image-side margins.

Setup and Notation

Hierarchical Image Prototypes and Samples. Continuing the notation of § 3.1, we consider a tree T=(V,E)\mathcal{T} = (V, E) together with an embedding {μk}k∈V\{\mu_k\}_{k \in V} of its nodes, as illustrated in Figure 1 (top-left). Each vector μk∈Rd\mu_k \in\mathbb{R}^{d}, for k∈Vk \in V, is called an image prototype.

Our transfer analysis compares the image-side margin of each prototype triplet with the perturbations introduced by sampling and optimization. Because image embedding instances deviate from their prototypes, we model the resulting finite-sample error as follows. We sample exactly nn images per leaf, for a total of N=n∣Vleaf∣N = n|V_{\mathrm{leaf}}| images.

Assumption 4.1 (Prototype variation model). The image embeddings {zj}j=1N\{z_j\}_{j=1}^{N} are sampled as zj=μyj+εjz_j = \mu_{y_j} + \varepsilon_j, where yj∈Vleafy_j \in V_{\mathrm{leaf}} is the leaf node associated with image jj and {εj}j=1N\{\varepsilon_j\}_{j=1}^{N} are independent, zero-mean, σ\sigma-sub-Gaussian random vectors in Rd\mathbb{R}^{d}.

Hierarchical captions determine each concept’s positive images. For each node u∈Vu \in V, let Path⁡(u)\operatorname{Path}(u) denote the root-to-uu path. Each image j∈[N]j \in[N] is paired with a fixed observed caption node cj∈Path⁡(yj)c_j \in\operatorname{Path}(y_j) (Figure 1, top-right). For example, an image from the leaf labrador may be captioned as labrador, dog, mammal, or animal.

For each node k∈Vk \in V, define its empirical positive set and caption count by

Pk:={j∈[N]:cj=k},Nk:=∣Pk∣.(3)\mathcal{P}_k := \{j \in[N] : c_j = k\}, \qquad N_k := |\mathcal{P}_k|. \tag*{(3)}

As shown in Figure 1 (bottom-left), the positive set of a non-leaf concept such as dog can pool images across descendant leaves such as labrador and corgi.

Hierarchical co-occurrence distinguishes our UFM from non-hierarchical UFM analyses. Prior UFM work for cross-entropy or supervised contrastive learning assumes mutually exclusive class labels and analyzes the resulting Simplex ETF geometry [15, 16, 20, 21]. Here, each image still has one caption, but that caption may be any concept along Path⁡(yj)\operatorname{Path}(y_j) rather than only the mutually exclusive leaf yjy_j. The resulting dependencies across nodes allow each optimum to depend on subtree prototype averages, making HAS\mathcal{H}\mathcal{A}\mathcal{S}, rather than a flat simplex, the relevant geometry.

The UFM optimizes only the text embeddings. Following the Unconstrained Features Model (UFM) approach [16], we fix the observed image embeddings Z=[z1,…,zN]∈Rd×NZ = [z_1,\ldots,z_N] \in\mathbb{R}^{d \times N} and caption labels {cj}j=1N\{c_j\}_{j=1}^{N}, and optimize the text concept embeddings A=[a1,…,aM]∈Rd×MA = [a_1,\ldots,a_M] \in\mathbb{R}^{d \times M}. For a regularization coefficient λ>0\lambda> 0, the objective is

min⁡ALcontr(A;Z)+λ2∥A∥F2.(4)\min_A \mathcal{L}_{\mathrm{contr}}(A;Z) + \frac{\lambda}{2}\lVert A\rVert_F^2. \tag*{(4)}

Remark 4.2 (Relation to CLIP’s normalized formulation). In practice, CLIP normalizes embeddings to the unit sphere, which bounds logits and guarantees finite minimizers. We instead use unnormalized embeddings with L2L_2 regularization, following the standard UFM approach [16, 20, 21]. This substitution is motivated by tractability: normalization introduces non-convexity, which makes analysis more complex. The regularization coefficient λ\lambda prevents unbounded growth of embedding norms as a normalization operator does.

Next, we study the geometry of A∗A^* at the global minimizers of the contrastive objective (4). In § 4.2, we first analyze when the contrastive loss is a linear contrastive loss, which admits an exact minimizer in closed form. We use the closed-form minimizer to identify which image-side triplet orderings are transferred and thereby bound the text-side HAS\mathcal{H}\mathcal{A}\mathcal{S} violation percentage.

Linear Contrastive Loss Analysis

To get a closed-form analysis of HAS\mathcal{H}\mathcal{A}\mathcal{S} transfer in a contrastive learning setting, we first consider the linear contrastive loss of Ji et al. [18]. The symmetric linear contrastive loss Llin(A)\mathcal{L}^{\mathrm{lin}}(A) is

References

  1. [1]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.arxiv.org/abs/2103.00020
  2. [2]Aditya Chattopadhyay, Kwan Ho Ryan Chan, and René Vidal. Bootstrapping variational information pursuit with large language and vision models for interpretable image classification. In ICLR, 2024.
  3. [3]Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. In NeurIPS, 2023.arxiv.org/abs/2309.17133
  4. [4]Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P. Calmon, and Himabindu Lakkaraju. Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE). In NeurIPS, 2024.arxiv.org/abs/2402.10376
  5. [5]Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting CLIP’s Image Representation via Text-Based Decomposition. In ICLR, 2024.
  6. [6]George A. Miller. WordNet: A Lexical Database for English. Communications of the ACM, 1995.
  7. [7]Morris Alper and Hadar Averbuch-Elor. Emergent Visual-Semantic Hierarchies in Image-Text Representations. In ECCV, 2024.arxiv.org/abs/2407.08521
  8. [8]Vladimir Zajgrajew, Hubert Baniecki, and Przemyslaw Biecek. Interpreting CLIP with Hierarchical Sparse Autoencoders. In ICML, 2025.
  9. [9]Nghia Nguyen, Tianjiao Ding, and René Vidal. Hierarchical Concept Embedding & Pursuit for Interpretable Image Classification. In CVPR, 2026.arxiv.org/abs/2602.11448
  10. [10]Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The Geometry of Categorical and Hierarchical Concepts in Large Language Models. In ICLR, 2025.arxiv.org/abs/2406.01506
  11. [11]Yibo Jiang, Bryon Aragam, and Victor Veitch. Uncovering Meanings of Embeddings via Partial Orthogonality. In NeurIPS, 2023.arxiv.org/abs/2310.17611
  12. [12]Octavian-Eugen Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic Entailment Cones for Learning Hierarchical Embeddings. In ICML, 2018.arxiv.org/abs/1804.01882
  13. [13]Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedantam. Hyperbolic Image-text Representations. In ICML, 2023.arxiv.org/abs/2304.09172
  14. [14]Avik Pal, Max van Spengler, Guido Maria D’Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes. Compositional Entailment Learning for Hyperbolic Vision-Language Models. In ICLR, 2025.
  15. [15]Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A Geometric Analysis of Neural Collapse with Unconstrained Features. In NeurIPS, 2021.arxiv.org/abs/2105.02375
  16. [16]Dustin G. Mixon, Hans Parshall, and Jianzong Pi. Neural Collapse with Unconstrained Features. Sampling Theory, Signal Processing, and Data Analysis, 2022.arxiv.org/abs/2011.11619
  17. [17]Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An Unconstrained Layer-Peeled Perspective on Neural Collapse. In ICLR, 2022.arxiv.org/abs/2110.02796
  18. [18]Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The Power of Contrast for Feature Learning: A Theoretical Analysis. Journal of Machine Learning Research, 2023.
  19. [19]Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of Neural Collapse during the terminal phase of Deep Learning training. Proceedings of the National Academy of Sciences, 2020.DOI
  20. [20]Florian Graf, Christoph Hofer, Marc Niethammer, and Roland Kwitt. Dissecting Supervised Contrastive Learning. In ICML, 2021.arxiv.org/abs/2102.08817
  21. [21]Tina Behnia and Christos Thrampoulidis. Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features. In ISIT, 2024.arxiv.org/abs/2402.18884
  22. [22]Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models. In ICML, 2024.arxiv.org/abs/2311.03658
  23. [23]Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parminder Bhatia, and Stefano Soatto. Linear Spaces of Meanings: Compositional Structures in Vision-Language Models. In ICCV, 2023.arxiv.org/abs/2302.14383
  24. [24]Meir Yossef Levi and Guy Gilboa. The Double-Ellipsoid Geometry of CLIP. In ICML, 2025.arxiv.org/abs/2411.14517
  25. [25]Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. In NeurIPS, 2022.arxiv.org/abs/2203.02053
  26. [26]Yize Zhao and Christos Thrampoulidis. Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations. arXiv preprint arXiv:2505.08348, 2025.arxiv.org/abs/2505.08348
  27. [27]Andres Naya and Matthieu Wýart. Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence. arXiv preprint arXiv:2605.23821, 2026.
  28. [28]Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 1, Learning Hierarchical Language Structures. Transactions on Machine Learning Research, 2025.
  29. [29]Lingjing Kong, Guangyi Chen, Biwei Huang, Eric P. Xing, Yuejie Chi, and Kun Zhang. Learning Discrete Concepts in Latent Hierarchical Models. In NeurIPS, 2024.arxiv.org/abs/2406.00519
  30. [30]Jianyang Gu, Samuel Stevens, Elizabeth G. Campolongo, Matthew J. Thompson, Net Zhang, Jiaman Wu, Andrei Kopanev, Zheda Mai, Alexander E. White, James Balhoff, Wasila Dahdul, Daniel Rubenstein, Hilmar Lapp, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su. BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning. In NeurIPS, 2025.
  31. [31]Antonio Sclocchi, Alessandro Favero, and Matthieu Wyart. A phase transition in diffusion models reveals the hierarchical nature of data. Proceedings of the National Academy of Sciences, 2025.DOI
  32. [32]Antonio Sclocchi, Alessandro Favero, Noam Itzhak Levi, and Matthieu Wyart. Probing the latent hierarchical structure of data via diffusion models. Journal of Statistical Mechanics: Theory and Experiment, 2025.arxiv.org/abs/2410.13770
  33. [33]Francesco Cagnetta, Leonardo Petrini, Umberto M. Tomasini, Alessandro Favero, and Matthieu Wyart. How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model. Physical Review X, 2024.DOI
  34. [34]Maximilian Nickel and Douwe Kiela. Poincaré Embeddings for Learning Hierarchical Representations. In NeurIPS, 2017.arxiv.org/abs/1705.08039
  35. [35]Ziwei Wang, Sameera Ramasinghe, Chenchen Xu, Julien Monteil, Loris Bazzani, and Thalaiyasingam Ajanthan. Learning Visual Hierarchies in Hyperbolic Space for Image Retrieval. In ICCV, 2025.arxiv.org/abs/2411.17490
  36. [36]Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. In NeurIPS, 2021.arxiv.org/abs/2106.04156
  37. [37]Ryumei Nakada, Halil Ibrahim Gullu, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data. In AISTATS, 2023.arxiv.org/abs/2302.06232
  38. [38]Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive Learning Inverts the Data Generating Process. In ICML, 2021.arxiv.org/abs/2102.08850
  39. [39]Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding Self-Supervised Learning with Dual Deep Networks. arXiv preprint arXiv:2010.00578, 2020.arxiv.org/abs/2010.00578
  40. [40]Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse Engineering Self-Supervised Learning. In NeurIPS, 2023.
  41. [41]Tom Wollschläger, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan Günnemann, and Johannes Gasteiger. The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence. In ICML, 2025.arxiv.org/abs/2502.17420
  42. [42]Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Darshan Thaker, Aditya Chattopadhyay, Chris Callison-Burch, and René Vidal. PaCE: Parsimonious Concept Engineering for Large Language Models. In NeurIPS, 2024.arxiv.org/abs/2406.04331
  43. [43]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
  44. [44]Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023.DOI
  45. [45]Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander J Ratner, Shuran Song, Hannanah Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. DataComp: In search of the next generation of multimodal datasets. In NeurIPS, 2023.
  46. [46]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harm sen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025.arxiv.org/abs/2502.14786
  47. [47]Yung-Sung Chuang, Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, Xinlei Chen, Zhuang Liu, Saining Xie, Wen-tau Yih, Shang-Wen Li, and Hu Xu. Meta CLIP 2: A worldwide scaling recipe. In NeurIPS, 2025.arxiv.org/abs/2507.22062
  48. [48]Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
  49. [49]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv:1807.03748, 2018.arxiv.org/abs/1807.03748
  50. [50]Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. Representation Tradeoffs for Hyperbolic Embeddings. In ICML, 2018.
  51. [51]Aaron B. Adcock, Blair D. Sullivan, and Michael W. Mahoney. Tree-like structure in large social and information networks. In ICDM, 2013.DOI
  52. [52]Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T. Toshev, and Vaishaal Shankar. Data filtering networks. In ICLR, 2024.arxiv.org/abs/2309.17425
  53. [53]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.

Paper details

vlnc:W_fw4ht2xn
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents