Introduction
Given a collection of image-text pairs, Contrastive Language-Image Pretraining (CLIP) learns text and image encoders by aligning paired embeddings and separating unpaired ones. The learned representations enable strong transfer across tasks such as image classification, visual question answering, and multimodal retrieval [1 , 2 , 3 ] .
One possible explanation for this empirical success is that CLIP embeddings implicitly encode rich semantic and geometric structure. In particular, embeddings of semantic concepts have been shown to organize into linear subspaces [4 , 5 ] and to reflect hierarchical relationships aligned with noun hierarchies in WordNet [6 , 7 , 8 , 9 ] . For example, a cat and a dog are both mammals, and a mammal and a bird are both animals. Recent work has attempted to characterize such hierarchical structure through geometric conditions, for example via orthogonality relations between embeddings of semantic concepts [10 , 11 ] . However, these conditions fail to capture a fundamental property of semantic hierarchies: a concept should be more similar to its descendants than to concepts from unrelated branches [12 , 13 , 14 , 7 ] . Empirical studies further suggest that CLIP image embeddings encode hierarchy through separated subspaces [9 ] , but this evidence does not quantify the extent of such structure across models and scales, nor does it provide a principled explanation of its origin.
A natural hypothesis is that hierarchical structure is learned through contrastive learning: if one modality encodes semantic relationships, alignment should transfer them to the other. Our experiments show that the two modalities differ. Approximate hierarchical structure is already present in raw-pixel image data and this structure is preserved by a randomly initialized CLIP encoder, implying that the CLIP embedding space natively exhibits hierarchy prior to any training. In contrast, raw text consists of discrete symbols whose semantic relationships are not reflected in their randomly initialized embeddings; any geometric structure must therefore be learned through training.
These observations raise a central question: how does contrastive learning transform and propagate hierarchical structure across modalities? In particular, why does alignment amplify and transfer structure from image to text embeddings, rather than suppress it? While the origin of hierarchical structure in image data remains open, our focus in this paper is on understanding how contrastive learning transfers such structure to other modalities once it is present in one modality.
To study this phenomenon, we formalize hierarchical structure as a geometric property called Hierarchical Angular Separation (H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S ): given a taxonomy of concepts, an ancestor has higher cosine similarity to each of its descendants concepts than to any of its off-branch concepts. We quantify violations from this ordering using the H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S violation percentage (H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol): the percentage of off-branch concepts that violate the H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S condition. Here, off-branch concepts exclude both the concept’s subtree and its ancestors. Our experiments across CLIP models of varying scales show that image embeddings natively exhibit low H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol at random initialization, whereas text embeddings begin with high H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol and achieve low values only after contrastive training.
These empirical observations motivate the need for a theoretical analysis. However, analyzing contrastive learning theoretically is challenging due to the non-convexity of the optimization problem. To address this, we adopt the Unconstrained Features Model (UFM) [15 , 16 , 17 ] , which abstracts away the deep network by optimizing directly over last-layer features. Within this framework, we show that hierarchical structure in image embeddings transfers to text embeddings under contrastive learning. The key mechanism, illustrated in Figure 1 , is the hierarchical co-occurrence structure of image-text pairs: although each image corresponds to a specific concept (e.g., dog), its paired text may correspond to that concept or to one of its hypernyms (e.g., mammal, animal). This structure induces a bias that propagates hierarchical relationships from image to text embeddings.
Figure 1. How contrastive learning transfers hierarchy from image to text embeddings. Top-left: the image prototypes are arranged so that each concept is closer to its descendants than to concepts in other subtrees. The blue and red dotted lines indicate the boundary of the dog and cat subtrees, respectively. Top-right: each image of a labrador is a variation of the labrador prototype, and its caption may pair with any concept on the path from the root to that leaf, so an image can be captioned labrador, dog or animal. Bottom-left: the pairing also means that the caption dog pairs with images of dog or any of its descendants, so the average of these images approaches the mean of the dog subtree. Bottom-right: contrastive learning pulls each text embedding toward the corresponding subtree mean, so the text embeddings inherit the arrangement of the image prototypes.
Paper Contributions. In this paper, we empirically characterize the hierarchical structure of CLIP embeddings and theoretically explain how contrastive learning transfers this structure from image to text embeddings:
Empirical characterization of the hierarchical structure. Using WordNet, we show that text embeddings have high H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol at random initialization and low H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol after pretraining, while image embeddings already have low H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol at initialization. This reveals a systematic asymmetry: hierarchy is learned in text but partly present in image embeddings prior to training.
Hierarchical co-occurrence transfers image geometry to text at the linear-loss optimum. Under the UFM with hierarchical co-occurrence, we derive a closed-form minimizer of a linear contrastive loss [18 ] , showing that optimal text embeddings align with root-centered image prototypes. The text-side H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol at each anchor is at most the image-side percentage of off-branch nodes that violate or nearly violate H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S .
Extension to InfoNCE and robustness beyond theory. We establish an analogous bound on text H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol for the InfoNCE loss used in CLIP under sufficient regularization. Furthermore, we empirically show that transfer occurs even when the regularization condition does not hold.
Related Work
Unconstrained Features Model. The UFM was originally used to explain Neural Collapse [19 ] . Subsequent work showed that the Simplex Equiangular Tight Frame is the unique optimal geometry for cross-entropy with weight decay [15 , 16 , 17 ] and for supervised contrastive losses [20 , 21 ] . However, these analyses focus on classification with mutually exclusive classes. We extend the UFM to contrastive learning with hierarchically structured image-text pairings and characterize the geometry that emerges at global minimizers.
Geometry of Representations in Foundation Models. Recent work has documented geometric structure in pretrained embedding spaces. One line of work shows that semantic concepts form linear or compositional subspaces [22 , 23 ] , exhibit hierarchical organization [10 , 9 , 7 ] , or give rise to phenomena such as the modality gap [24 , 25 ] . Relatedly, Zhao and Thrampoulidis Zhao and Thrampoulidis [26 ] infer a semantic hierarchy from sign patterns in a learned concept basis for next-token prediction. Nava and Wyart Nava and Wyart [27 ] derive coarse-to-fine spectral organization from hierarchy-dependent word co-occurrence in word2vec-style embeddings. We instead study a given tree of object categories and explain how H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S transfers from image to text embeddings during contrastive pretraining.
Learning Theory with Hierarchy Models. Another line of work studies how hierarchical latent structure shapes learning and generation, including hierarchical concept learning in language models [28 , 29 ] , emergent properties under hierarchical contrastive learning [30 ] , and phase transitions in diffusion models signaling an underlying latent hierarchy [31 , 32 ] . Closest to our work, several papers [31 , 33 ] analyze the Random Hierarchy Model, a supervised setting in which the sample for a class is generated by a hierarchical composition of latent features. However, these works study compositional hierarchies of features, while we instead study hierarchies over object categories.
Hierarchical Representation Learning. Traditionally, hierarchical information is explicitly encoded into models through specialized architectures or objectives. A popular approach is to embed hierarchies in hyperbolic space, exploiting its exponential volume growth [34 , 12 , 13 , 14 , 35 ] . In contrast, we analyze when implicit hierarchical structure transfers in self-supervised learning.
Contrastive Learning Theory. Theoretical analyses have connected contrastive objectives to spectral decompositions of augmentation graphs [36 , 37 ] , and have shown that contrastive objectives can invert the data-generating process [38 ] . A separate line of work provides theoretical and empirical evidence that self-supervised representations cluster at multiple hierarchical levels [39 , 40 ] . By contrast, we analyze how the contrastive loss interacts with hierarchical co-occurrence structures.
Hierarchical Angular Separation and Its Modality Asymmetry in CLIP
We first define a hierarchical similarity ordering for embeddings, Hierarchical Angular Separation (H A S \mathcal{H}AS H A S ). In this definition, each concept embedding is closer to its descendant concept embeddings than to any of its off-branch concepts. Pretraining creates a modality asymmetry in CLIP’s hierarchical structure: text embeddings acquire H A S \mathcal{H}AS H A S after training, whereas image embeddings exhibit H A S \mathcal{H}AS H A S approximately at initialization and strengthened H A S \mathcal{H}AS H A S after training. We formalize H A S \mathcal{H}AS H A S in Section 3.1 and measure it across CLIP variants in Section 3.2.
Definition of Hierarchical Angular Separation
Datasets of natural images and language often contain semantic concepts that can be organized into hierarchies. For example, a cat and a dog are both mammals, and a mammal and a bird are both animals. Below, we introduce the notation used to describe such hierarchies.
Hierarchy. Let T = ( V , E ) \mathcal{T} = (V,E) T = ( V , E ) be a rooted tree with ∣ V ∣ = M |V| = M ∣ V ∣ = M nodes, leaf set V l e a f ⊆ V V_{\mathrm{leaf}} \subseteq V V leaf ⊆ V , and root node r o o t ∈ V \mathrm{root} \in V root ∈ V . Fix an enumeration of the nodes and identify V V V with [ M ] : = { 1 , … , M } [M] := \{1,\ldots,M\} [ M ] := { 1 , … , M } . Each u ∈ V u \in V u ∈ V corresponds to a synset (e.g., mammal). For u ∈ V u \in V u ∈ V , let Anc ( u ) \operatorname{Anc}(u) Anc ( u ) denote its ancestors, Desc ( u ) \operatorname{Desc}(u) Desc ( u ) its descendants, and Sub ( u ) : = { u } ∪ Desc ( u ) \operatorname{Sub}(u) := \{u\} \cup\operatorname{Desc}(u) Sub ( u ) := { u } ∪ Desc ( u ) the subtree rooted at u u u .
Concept Embeddings. For each node k ∈ V k \in V k ∈ V , we associate a concept embedding h k ∈ R d \mathbf{h}_k \in\mathbb{R}^d h k ∈ R d , and write the embedding of the hierarchy T \mathcal{T} T as the collection { h k : k ∈ V } \{\mathbf{h}_k : k \in V\} { h k : k ∈ V } . In practice, concept embeddings are obtained using text and image encoders in two common ways: (1) instance aggregation [22 , 41 , 42 ] , which averages all image embeddings z j ∈ R d \mathbf{z}_j \in\mathbb{R}^d z j ∈ R d corresponding to a node u ∈ V u \in V u ∈ V (e.g., averaging over all dog images to estimate the concept embedding for dog); (2) direct concept encoding [1 , 23 ] , which encodes the concept string (e.g., image of a dog) to obtain a text embedding.
Hierarchical Angular Separation. We define H A S \mathcal{H}AS H A S as a Euclidean similarity formalization of hierarchy based on angular separation, applicable to either image or text embeddings. To make this precise, for non-zero x , y ∈ R d \mathbf{x}, \mathbf{y} \in\mathbb{R}^d x , y ∈ R d , we define cos ( x , y ) : = x ⊤ y / ( ∥ x ∥ ∥ y ∥ ) \cos(\mathbf{x},\mathbf{y}) := \mathbf{x}^{\top}\mathbf{y}/(\lVert\mathbf{x}\rVert\lVert\mathbf{y}\rVert) cos ( x , y ) := x ⊤ y / (∥ x ∥ ∥ y ∥) .
Definition 3.1 (Hierarchical Angular Separation (H A S \mathcal{H}AS H A S )). The concept embeddings { h k } k ∈ V \{\mathbf{h}_k\}_{k\in V} { h k } k ∈ V of hierarchy T \mathcal{T} T (obtained by either instance aggregation or direct concept encoding) satisfy H A S \mathcal{H}AS H A S if for every non-leaf node u ∈ V u \in V u ∈ V , every descendant v ∈ Desc ( u ) v \in\operatorname{Desc}(u) v ∈ Desc ( u ) , and every node outside the subtree and not an ancestor, w ∉ Sub ( u ) ∪ Anc ( u ) w \notin\operatorname{Sub}(u) \cup\operatorname{Anc}(u) w ∈ / Sub ( u ) ∪ Anc ( u ) , we have
cos ( h u , h v ) > cos ( h u , h w ) . (1) \cos(\mathbf{h}_u,\mathbf{h}_v) > \cos(\mathbf{h}_u,\mathbf{h}_w). \tag*{(1)} cos ( h u , h v ) > cos ( h u , h w ) . (1)
Nguyen et al. [9 ] use instance aggregation to estimate concept embeddings, and claim to have observed subtree separation in pretrained CLIP image embeddings without specifying how.
H A S \mathcal{H}AS H A S differs from prior hierarchy-learning objectives that prescribe hierarchical structure during training. Hyperbolic entailment cones [12 , 13 , 14 ] and Poincaré embeddings [34 ] optimize non-Euclidean embeddings with losses designed to enforce hierarchical relations. In contrast, and as we compare in detail in Appendix § 7 , we define H A S \mathcal{H}AS H A S descriptively rather than prescriptively: Eq. (1) is a Euclidean cosine inequality measured in the learned CLIP space, where the observed angles determine the descendant cone around h u h_u h u rather than the training objective imposing it. To quantify how far a set of embeddings is from satisfying Definition 3.1, we use the following per-node metric.
Definition 3.2 (H A S \mathcal{H}AS H A S Violation Percentage). For a non-root non-leaf node u u u , we first define the descendant threshold , the minimum cosine similarity between u u u and its least similar descendant, as s u : = min v ∈ Desc ( u ) cos ( h u , h v ) s_u := \min_{v\in\operatorname{Desc}(u)} \cos(\mathbf{h}_u,\mathbf{h}_v) s u := min v ∈ Desc ( u ) cos ( h u , h v ) . The H A S \mathcal{H}AS H A S violation percentage at u u u is then the percentage of the set of off-branch (not in the subtree and not an ancestor) nodes { w } \{w\} { w } that has a larger cosine similarity with u u u than s u s_u s u
H A S - Viol ( u ) : = ∣ { w ∉ Sub ( u ) ∪ Anc ( u ) : cos ( h u , h w ) ≥ s u } ∣ ∣ { w ∉ Sub ( u ) ∪ Anc ( u ) } ∣ × 100 % . (2) \mathcal{H}AS\text{-}\operatorname{Viol}(u) := \frac{|\{w \notin\operatorname{Sub}(u) \cup\operatorname{Anc}(u) : \cos(\mathbf{h}_u,\mathbf{h}_w) \ge s_u\}|}{|\{w \notin\operatorname{Sub}(u) \cup\operatorname{Anc}(u)\}|} \times100\%. \tag*{(2)} H A S - Viol ( u ) := ∣ { w ∈ / Sub ( u ) ∪ Anc ( u )} ∣ ∣ { w ∈ / Sub ( u ) ∪ Anc ( u ) : cos ( h u , h w ) ≥ s u } ∣ × 100%. (2)
H A S - Viol ( u ) = 0 \mathcal{H}AS\text{-}\operatorname{Viol}(u) = 0 H A S - Viol ( u ) = 0 for every non-root non-leaf u ∈ V u \in V u ∈ V if and only if H A S \mathcal{H}AS H A S (Definition 3.1) holds. We are now ready to measure H A S \mathcal{H}AS H A S in CLIP embeddings.
Hierarchical Angular Separation in CLIP embeddings
We measure H A S \mathcal{H}AS H A S -Viol on the ImageNet-1K training split [43 ] and the WordNet noun hierarchy [6 ] .1 We evaluate both the text and image embeddings of two pretrained CLIP ViT-L/14 checkpoints (OpenAI CLIP [1 ] and OpenCLIP [44 ] trained on DataComp-1B [45 ] ) and a randomly initialized ViT-L/14. To reduce the effect of outliers, we replace the descendant threshold s u = min v ∈ Desc ( u ) cos ( h u , h v ) s_u = \min_{v \in\operatorname{Desc}(u)} \cos(h_u,h_v) s u = min v ∈ Desc ( u ) cos ( h u , h v ) by its 95th percentile over Desc ( u ) \operatorname{Desc}(u) Desc ( u ) . Corresponding results on ViT-B/32, ViT-B/16, ViT-L/14@336px, SigLIP2 [46 ] , MetaCLIP2 [47 ] , and on the CUB-200 taxonomy [48 ] , are deferred to Appendix § 16 .§ 1 .
Text embeddings: H A S \mathcal{H}AS H A S is acquired through pretraining. In Figure 2 (top), both pretrained text models have low violation percentages across depths, whereas the randomly initialized text model has substantially larger violations across the tree. Hence, for text concept embeddings, H A S \mathcal{H}AS H A S is a learned property of contrastive pretraining. Additional results estimating concept embeddings by instance aggregation reach the same conclusion (Appendix § 16 .§ 1 ).
Figure 2. Distribution of H A S \mathcal{H}AS H A S -Viol vs depth on ViT-L/14 CLIP embeddings over non-root and non-leaf nodes. Top : text embeddings show that H A S \mathcal{H}AS H A S is acquired through pretraining. Bottom : image embeddings show approximate H A S \mathcal{H}AS H A S at random initialization and stronger H A S \mathcal{H}AS H A S after pretraining.
Image embeddings: H A S \mathcal{H}AS H A S is present in raw pixels and strengthened by pretraining. In Figure 2 (bottom), pretrained image embeddings have violation percentages concentrated near zero across depths. The randomly initialized ViT-L/14 has substantially lower violations than randomly initialized text embeddings, although its violations remain non-negligible.
In Table 1 , raw pixels and randomly initialized image embeddings separate from their own hierarchy-shuffled nulls, whereas i.i.d. Gaussian vectors do not. Approximate image-side H A S \mathcal{H}AS H A S therefore comes from raw image data, remains after a randomly initialized encoder, and strengthens with contrastive pretraining, while text-side H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S is acquired only through pretraining.
Table 1. Mean image-side H A S \mathcal{H}AS H A S -Viol. Each representation is paired with its own hierarchy-shuffled null. Null values report mean ± standard deviation over five hierarchy shuffles.
Provable Transfer of Hierarchical Angular Separation from Image to Text Embeddings
Motivated by the observed empirical asymmetry, we study theoretically how contrastive learning transfers pre-existing image geometry to text. We use an Unconstrained Features Model (UFM) in which the image embeddings are fixed and only the text embeddings are learned. Under hierarchical co-occurrence, each text concept aggregates images from its descendant leaves, and contrastive learning therefore aligns the text embedding with the corresponding root-centered image prototype. We first derive this mechanism in closed form for a linear contrastive loss (§ 4.2 ), then show that regularized symmetric InfoNCE inherits it under common assumptions (§ 4.3 ). We summarize the relation between these theoretical results in Figure 3 . We also show that the transfer is robust to weaker regularization than required by theory (§ 4.4 ).
Figure 3. Overview of the H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S transfer results. Under assumptions on the data model, both losses decompose the text embeddings into prototypes plus perturbations. Bounding the perturbations relative to the prototype norms gives a bound on text-side violations in terms of image-side margins.
Setup and Notation
Hierarchical Image Prototypes and Samples. Continuing the notation of § 3.1 , we consider a tree T = ( V , E ) \mathcal{T} = (V, E) T = ( V , E ) together with an embedding { μ k } k ∈ V \{\mu_k\}_{k \in V} { μ k } k ∈ V of its nodes, as illustrated in Figure 1 (top-left). Each vector μ k ∈ R d \mu_k \in\mathbb{R}^{d} μ k ∈ R d , for k ∈ V k \in V k ∈ V , is called an image prototype .
Our transfer analysis compares the image-side margin of each prototype triplet with the perturbations introduced by sampling and optimization. Because image embedding instances deviate from their prototypes, we model the resulting finite-sample error as follows. We sample exactly n n n images per leaf, for a total of N = n ∣ V l e a f ∣ N = n|V_{\mathrm{leaf}}| N = n ∣ V leaf ∣ images.
Assumption 4.1 (Prototype variation model ). The image embeddings { z j } j = 1 N \{z_j\}_{j=1}^{N} { z j } j = 1 N are sampled as z j = μ y j + ε j z_j = \mu_{y_j} + \varepsilon_j z j = μ y j + ε j , where y j ∈ V l e a f y_j \in V_{\mathrm{leaf}} y j ∈ V leaf is the leaf node associated with image j j j and { ε j } j = 1 N \{\varepsilon_j\}_{j=1}^{N} { ε j } j = 1 N are independent, zero-mean, σ \sigma σ -sub-Gaussian random vectors in R d \mathbb{R}^{d} R d .
Hierarchical captions determine each concept’s positive images. For each node u ∈ V u \in V u ∈ V , let Path ( u ) \operatorname{Path}(u) Path ( u ) denote the root-to-u u u path. Each image j ∈ [ N ] j \in[N] j ∈ [ N ] is paired with a fixed observed caption node c j ∈ Path ( y j ) c_j \in\operatorname{Path}(y_j) c j ∈ Path ( y j ) (Figure 1 , top-right). For example, an image from the leaf labrador may be captioned as labrador, dog, mammal, or animal.
For each node k ∈ V k \in V k ∈ V , define its empirical positive set and caption count by
P k : = { j ∈ [ N ] : c j = k } , N k : = ∣ P k ∣ . (3) \mathcal{P}_k := \{j \in[N] : c_j = k\}, \qquad N_k := |\mathcal{P}_k|. \tag*{(3)} P k := { j ∈ [ N ] : c j = k } , N k := ∣ P k ∣. (3)
As shown in Figure 1 (bottom-left), the positive set of a non-leaf concept such as dog can pool images across descendant leaves such as labrador and corgi.
Hierarchical co-occurrence distinguishes our UFM from non-hierarchical UFM analyses. Prior UFM work for cross-entropy or supervised contrastive learning assumes mutually exclusive class labels and analyzes the resulting Simplex ETF geometry [15 , 16 , 20 , 21 ] . Here, each image still has one caption, but that caption may be any concept along Path ( y j ) \operatorname{Path}(y_j) Path ( y j ) rather than only the mutually exclusive leaf y j y_j y j . The resulting dependencies across nodes allow each optimum to depend on subtree prototype averages, making H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S , rather than a flat simplex, the relevant geometry.
The UFM optimizes only the text embeddings. Following the Unconstrained Features Model (UFM) approach [16 ] , we fix the observed image embeddings Z = [ z 1 , … , z N ] ∈ R d × N Z = [z_1,\ldots,z_N] \in\mathbb{R}^{d \times N} Z = [ z 1 , … , z N ] ∈ R d × N and caption labels { c j } j = 1 N \{c_j\}_{j=1}^{N} { c j } j = 1 N , and optimize the text concept embeddings A = [ a 1 , … , a M ] ∈ R d × M A = [a_1,\ldots,a_M] \in\mathbb{R}^{d \times M} A = [ a 1 , … , a M ] ∈ R d × M . For a regularization coefficient λ > 0 \lambda> 0 λ > 0 , the objective is
min A L c o n t r ( A ; Z ) + λ 2 ∥ A ∥ F 2 . (4) \min_A \mathcal{L}_{\mathrm{contr}}(A;Z) + \frac{\lambda}{2}\lVert A\rVert_F^2. \tag*{(4)} A min L contr ( A ; Z ) + 2 λ ∥ A ∥ F 2 . (4)
Remark 4.2 (Relation to CLIP’s normalized formulation). In practice, CLIP normalizes embeddings to the unit sphere, which bounds logits and guarantees finite minimizers. We instead use unnormalized embeddings with L 2 L_2 L 2 regularization, following the standard UFM approach [16 , 20 , 21 ] . This substitution is motivated by tractability: normalization introduces non-convexity, which makes analysis more complex. The regularization coefficient λ \lambda λ prevents unbounded growth of embedding norms as a normalization operator does.
Next, we study the geometry of A ∗ A^* A ∗ at the global minimizers of the contrastive objective (4) . In § 4.2 , we first analyze when the contrastive loss is a linear contrastive loss, which admits an exact minimizer in closed form. We use the closed-form minimizer to identify which image-side triplet orderings are transferred and thereby bound the text-side H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S violation percentage.
Linear Contrastive Loss Analysis
To get a closed-form analysis of H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S transfer in a contrastive learning setting, we first consider the linear contrastive loss of Ji et al. [18 ] . The symmetric linear contrastive loss L l i n ( A ) \mathcal{L}^{\mathrm{lin}}(A) L lin ( A ) is
− 1 2 ∑ j = 1 N [ a c j T z j − 1 M − 1 ∑ k ∈ [ M ] k ≠ c j a k T z j + a c j T z j − 1 N − 1 ∑ j ′ ∈ [ N ] j ′ ≠ j a c j T z j ′ ] + λ 2 ∥ A ∥ F 2 . (5) -\frac{1}{2}\sum_{j=1}^{N}\left[a_{c_j}^{\mathsf{T}}z_j-\frac{1}{M-1}\sum_{\substack{k\in[M]\\k\ne c_j}}a_k^{\mathsf{T}}z_j+a_{c_j}^{\mathsf{T}}z_j-\frac{1}{N-1}\sum_{\substack{j'\in[N]\\j'\ne j}}a_{c_j}^{\mathsf{T}}z_{j'}\right]+\frac{\lambda}{2}\lVert A\rVert_F^2. \tag*{(5)} − 2 1 j = 1 ∑ N a c j T z j − M − 1 1 k ∈ [ M ] k = c j ∑ a k T z j + a c j T z j − N − 1 1 j ′ ∈ [ N ] j ′ = j ∑ a c j T z j ′ + 2 λ ∥ A ∥ F 2 . (5)
Solving the first-order condition for Eq. (5) gives a unique minimizer. For each node k ∈ V k\in V k ∈ V with N k > 0 N_k>0 N k > 0 , define the empirical mean of the images captioned k k k as z ˉ k : = 1 N k ∑ j ∈ P k z j \bar{z}_k:=\frac{1}{N_k}\sum_{j\in P_k}z_j z ˉ k := N k 1 ∑ j ∈ P k z j , and define the global empirical image mean z ˉ : = 1 N ∑ j = 1 N z j \bar{z}:=\frac{1}{N}\sum_{j=1}^{N}z_j z ˉ := N 1 ∑ j = 1 N z j . The optimal text embedding for k k k is a weighted combination of these two means (Proposition D.1 and Corollary D.2 in Appendix § 10 ). The transfer proof therefore reduces to identifying the prototype component of each z ˉ k \bar{z}_k z ˉ k . We first make this identification exact with two simplifying assumptions: leaf-uniform coverage makes each mean uniform over descendant leaves, and prototype consistency identifies that average with the corresponding node prototype.
Assumption 4.3 (Leaf-uniform caption coverage). Every non-root node k ∈ V ∖ { r o o t } k\in V\setminus\{\mathrm{root}\} k ∈ V ∖ { root } has N k > 0 N_k>0 N k > 0 , and its set P k P_k P k contains the same number of images from each leaf in Sub ( k ) ∩ V l e a f \operatorname{Sub}(k)\cap V_{\mathrm{leaf}} Sub ( k ) ∩ V leaf .
Assumption 4.4 (Prototype consistency). For every node k ∈ V k\in V k ∈ V , the average of the prototypes over leaves in Sub ( k ) ∩ V l e a f \operatorname{Sub}(k)\cap V_{\mathrm{leaf}} Sub ( k ) ∩ V leaf is exactly μ k \mu_k μ k .
Using Assumptions 4.1, 4.3, and 4.4, we can write z ˉ k = μ k + ϵ ˉ k \bar{z}_k=\mu_k+\bar{\epsilon}_k z ˉ k = μ k + ϵ ˉ k and z ˉ = μ r o o t + ϵ ˉ \bar{z}=\mu_{\mathrm{root}}+\bar{\epsilon} z ˉ = μ root + ϵ ˉ , where ϵ ˉ k : = 1 N k ∑ j ∈ P k ϵ j \bar{\epsilon}_k:=\frac{1}{N_k}\sum_{j\in P_k}\epsilon_j ϵ ˉ k := N k 1 ∑ j ∈ P k ϵ j and the global noise average is ϵ ˉ : = 1 N ∑ j = 1 N ϵ j \bar{\epsilon}:=\frac{1}{N}\sum_{j=1}^{N}\epsilon_j ϵ ˉ := N 1 ∑ j = 1 N ϵ j , yielding the following prototype decomposition for the minimizer. The proof is deferred to Appendix § 10 .
Corollary 4.5 (Prototype Decomposition for the Linear Contrastive Loss). Suppose Assumptions 4.1, 4.3, and 4.4 hold. For each non-root k k k , define
ξ k l i n : = N ( 1 2 ( M − 1 ) + 1 2 ( N − 1 ) ) N k ( 1 + 2 M − 1 + 2 N − 1 ) . \xi_k^{\mathrm{lin}}:=\frac{N\left(\frac{1}{2(M-1)}+\frac{1}{2(N-1)}\right)}{N_k\left(1+\frac{2}{M-1}+\frac{2}{N-1}\right)}. ξ k lin := N k ( 1 + M − 1 2 + N − 1 2 ) N ( 2 ( M − 1 ) 1 + 2 ( N − 1 ) 1 ) . Then every such k k k satisfies
a k l i n , ∗ = N k λ ( 1 + 1 2 ( M − 1 ) + 1 2 ( N − 1 ) ) ( μ k + ϵ ˉ k − ξ k l i n ( μ r o o t + ϵ ˉ ) ) . (6) a_k^{\mathrm{lin},*}=\frac{N_k}{\lambda}\left(1+\frac{1}{2(M-1)}+\frac{1}{2(N-1)}\right)\left(\mu_k+\bar{\epsilon}_k-\xi_k^{\mathrm{lin}}\left(\mu_{\mathrm{root}}+\bar{\epsilon}\right)\right). \tag*{(6)} a k lin , ∗ = λ N k ( 1 + 2 ( M − 1 ) 1 + 2 ( N − 1 ) 1 ) ( μ k + ϵ ˉ k − ξ k lin ( μ root + ϵ ˉ ) ) . (6)
In Corollary 4.5, a k l i n , ∗ a_k^{\mathrm{lin},*} a k lin , ∗ is a positive rescaling of μ k \mu_k μ k plus a perturbation (Figure 1 , bottom-right). The perturbation consists of the deterministic root-centering term − ξ k l i n μ r o o t -\xi_k^{\mathrm{lin}}\mu_{\mathrm{root}} − ξ k lin μ root , which shifts each prototype along the root direction (Figure 4 ), and the stochastic variation term ϵ ˉ k − ξ k l i n ϵ ˉ \bar{\epsilon}_k-\xi_k^{\mathrm{lin}}\bar{\epsilon} ϵ ˉ k − ξ k lin ϵ ˉ , which is bounded with high probability by Assumption 4.1. Positive rescaling preserves cosine similarity, so transfer depends on how much the perturbation changes each prototype triplet margin. We therefore bound the perturbation relative to the prototype norms and identify which triplet orderings remain intact. Let ρ min : = min k ∈ V ∖ { r o o t } ∥ μ k ∥ \rho_{\min}:=\min_{k\in V\setminus\{\mathrm{root}\}}\|\mu_k\| ρ m i n := min k ∈ V ∖ { root } ∥ μ k ∥ denote the minimum non-root prototype norm. With ξ k l i n \xi_k^{\mathrm{lin}} ξ k lin from Corollary 4.5, define ξ max l i n : = max k ∈ V ∖ { r o o t } ∣ ξ k l i n ∣ \xi_{\max}^{\mathrm{lin}}:=\max_{k\in V\setminus\{\mathrm{root}\}}|\xi_k^{\mathrm{lin}}| ξ m a x lin := max k ∈ V ∖ { root } ∣ ξ k lin ∣ and δ r o o t : = ξ max l i n ∥ μ r o o t ∥ \delta_{\mathrm{root}}:=\xi_{\max}^{\mathrm{lin}}\|\mu_{\mathrm{root}}\| δ root := ξ m a x lin ∥ μ root ∥ , the largest root-centering perturbation norm.
Figure 4. Optimal text geometry for a stylized hierarchy. Each colored triangle is a regular simplex of child concepts, with its parent (diamond) at the centroid. The optimal text directions in (b) satisfy a k l i n , ∗ ∝ μ k − ξ k l i n μ r o o t a_k^{\mathrm{lin},*} \propto\mu_k-\xi_k^{\mathrm{lin}}\mu_{\mathrm{root}} a k lin , ∗ ∝ μ k − ξ k lin μ root , where ξ k l i n \xi_k^{\mathrm{lin}} ξ k lin is the root-centering coefficient. Each child simplex remains centered at its parent, and H A S \mathcal{HAS} H A S is preserved.
To compare violations across modalities, for a non-root non-leaf anchor u ∈ V u\in V u ∈ V , let H A S \mathcal{HAS} H A S -Violt x t ( u ) ^{\mathrm{txt}}(u) txt ( u ) denote H A S \mathcal{HAS} H A S -Viol evaluated on the optimal text embeddings. For non-zero non-root image prototypes and τ ≥ 0 \tau\geq0 τ ≥ 0 , let H A S \mathcal{HAS} H A S -Violτ i m g ( u ) ^{\mathrm{img}}_{\tau}(u) τ img ( u ) denote the same metric evaluated on the prototypes with their threshold s u s_u s u lowered to s u − τ s_u-\tau s u − τ . Equivalently, H A S \mathcal{HAS} H A S -Violτ i m g ( u ) ^{\mathrm{img}}_{\tau}(u) τ img ( u ) is the percentage of off-branch nodes at u u u whose worst-case triplet margin on the image prototypes is at most τ \tau τ .
Theorem 4.6 (H A S \mathcal{HAS} H A S Transfer for the Linear Contrastive Loss). Suppose Assumption 4.1 (sub-Gaussian prototype variation), Assumption 4.3 (leaf-uniform caption coverage), and Assumption 4.4 (prototype consistency) hold, and suppose ρ min > 0 \rho_{\min}>0 ρ m i n > 0 . Let N min : = min k ∈ V ∖ { r o o t } N k N_{\min}:=\min_{k\in V\setminus\{\mathrm{root}\}}N_k N m i n := min k ∈ V ∖ { root } N k be the minimum non-root caption count. There exists a universal constant C > 0 C>0 C > 0 such that the following holds for every ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) . Define the combined error term
D ϵ l i n : = δ r o o t + C σ ( d + log ( M − 1 ) + log ( 4 / ϵ ) N min + ξ max l i n d + log ( 4 / ϵ ) N min ) . (7) D_{\epsilon}^{\mathrm{lin}}:=\delta_{\mathrm{root}}+C\sigma\left(\sqrt{\frac{d+\log(M-1)+\log(4/\epsilon)}{N_{\min}}}+\xi_{\max}^{\mathrm{lin}}\sqrt{\frac{d+\log(4/\epsilon)}{N_{\min}}}\right). \tag*{(7)} D ϵ lin := δ root + C σ N m i n d + log ( M − 1 ) + log ( 4/ ϵ ) + ξ m a x lin N m i n d + log ( 4/ ϵ ) . (7) If D ϵ l i n < ρ min / 4 D_{\epsilon}^{\mathrm{lin}}<\rho_{\min}/4 D ϵ lin < ρ m i n /4 , then, with probability at least 1 − ϵ 1-\epsilon 1 − ϵ , every non-root non-leaf anchor node u u u satisfies
H A S - V i o l t x t ( u ) ≤ H A S - V i o l τ ϵ l i n i m g ( u ) , τ ϵ l i n : = 8 D ϵ l i n ρ min . (8) \mathcal{HAS}\text{-}\mathrm{Viol}^{\mathrm{txt}}(u)\leq\mathcal{HAS}\text{-}\mathrm{Viol}^{\mathrm{img}}_{\tau_{\epsilon}^{\mathrm{lin}}}(u),\qquad\tau_{\epsilon}^{\mathrm{lin}}:=\frac{8D_{\epsilon}^{\mathrm{lin}}}{\rho_{\min}}. \tag*{(8)} H A S - Viol txt ( u ) ≤ H A S - Viol τ ϵ lin img ( u ) , τ ϵ lin := ρ m i n 8 D ϵ lin . (8)
We defer the proof to Appendix G. Under the assumptions, every image-side triplet with margin greater than τ ϵ l i n \tau_{\epsilon}^{\mathrm{lin}} τ ϵ lin remains correctly ordered on the text side. Only triplets with image-side margin at most τ ϵ l i n \tau_{\epsilon}^{\mathrm{lin}} τ ϵ lin can violate text-side H A S \mathcal{HAS} H A S . We now turn to the InfoNCE loss [49 ] .
InfoNCE Loss Analysis
In the same fixed-image setup, we test whether the transfer mechanism exposed by the linear loss persists under regularized symmetric InfoNCE [49 ] :
L ( A ) : = ∑ j = 1 N 1 2 ( − a c j T z j + log ∑ k = 1 M e a c k T z j − a c j T z j + log ∑ j ′ = 1 N e a c j ′ T z j ′ ) + λ 2 ∥ A ∥ F 2 . (9) \mathcal{L}(A):=\sum_{j=1}^{N}\frac{1}{2}\left(-a_{c_j}^{\mathsf{T}}z_j+\log\sum_{k=1}^{M}e^{a_{c_k}^{\mathsf{T}}z_j}-a_{c_j}^{\mathsf{T}}z_j+\log\sum_{j'=1}^{N}e^{a_{c_{j'}}^{\mathsf{T}}z_{j'}}\right)+\frac{\lambda}{2}\|A\|_{F}^{2}. \tag*{(9)} L ( A ) := j = 1 ∑ N 2 1 − a c j T z j + log k = 1 ∑ M e a c k T z j − a c j T z j + log j ′ = 1 ∑ N e a c j ′ T z j ′ + 2 λ ∥ A ∥ F 2 . (9)
The objective is λ \lambda λ -strongly convex, so its unique minimizer A n c e , ∗ \mathbf{A}^{\mathrm{nce},*} A nce , ∗ satisfies the softmax fixed-point equation (Proposition E.2). Because this equation has no closed form, we analyze A n c e , ∗ \mathbf{A}^{\mathrm{nce},*} A nce , ∗ as λ → ∞ \lambda\to\infty λ → ∞ .
Proposition 4.7 (Large-Regularization Expansion). As λ → ∞ \lambda\to\infty λ → ∞ , for any k k k with N k > 0 N_k > 0 N k > 0 ,
a k n c e , ∗ = N k λ z ˉ k − 1 λ ( N 2 M + N k 2 ) z ˉ + O ( λ − 2 ) . (10) a_k^{\mathrm{nce},*} = \frac{N_k}{\lambda}\bar{z}_k - \frac{1}{\lambda}\left(\frac{N}{2M}+\frac{N_k}{2}\right)\bar{z} + O(\lambda^{-2}). \tag*{(10)} a k nce , ∗ = λ N k z ˉ k − λ 1 ( 2 M N + 2 N k ) z ˉ + O ( λ − 2 ) . (10)
As λ \lambda λ grows, A n c e , ∗ \mathbf{A}^{\mathrm{nce},*} A nce , ∗ shrinks and the softmax weights approach uniform. The fixed-point equation then yields Eq. (10). We defer the proof to Appendix E. Substituting z ˉ k = μ k + ϵ ˉ k \bar{z}_k = \mu_k+\bar{\epsilon}_k z ˉ k = μ k + ϵ ˉ k and z ˉ = μ r o o t + ϵ ˉ \bar{z}=\mu_{\mathrm{root}}+\bar{\epsilon} z ˉ = μ root + ϵ ˉ into Eq. (10) gives the same prototype decomposition as the linear-loss minimizer.
Corollary 4.8 (Prototype Decomposition for the Large-Regularization Expansion). Suppose Assumptions 4.1, 4.3, and 4.4 hold. For each non-root k k k , define ξ k n c e : = ( N 2 M + N k 2 ) / N k \xi_k^{\mathrm{nce}} := \left(\frac{N}{2M}+\frac{N_k}{2}\right)/N_k ξ k nce := ( 2 M N + 2 N k ) / N k . Then every such k k k satisfies
a k n c e , ∗ = N k λ ( μ k + ϵ ˉ k − ξ k n c e ( μ r o o t + ϵ ˉ ) ) + O ( λ − 2 ) . (11) a_k^{\mathrm{nce},*} = \frac{N_k}{\lambda}\left(\mu_k+\bar{\epsilon}_k-\xi_k^{\mathrm{nce}}\left(\mu_{\mathrm{root}}+\bar{\epsilon}\right)\right)+O(\lambda^{-2}). \tag*{(11)} a k nce , ∗ = λ N k ( μ k + ϵ ˉ k − ξ k nce ( μ root + ϵ ˉ ) ) + O ( λ − 2 ) . (11)
We defer the proof to Appendix E.4. The leading term has the same form as Corollary 4.5 with ξ k n c e \xi_k^{\mathrm{nce}} ξ k nce in place of ξ k l i n \xi_k^{\mathrm{lin}} ξ k lin , so the same margin argument gives the following H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S transfer guarantee for InfoNCE. For ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) , define D ϵ n c e D_\epsilon^{\mathrm{nce}} D ϵ nce by recomputing the error term in Eq. (7) with ξ k n c e \xi_k^{\mathrm{nce}} ξ k nce in place of ξ k l i n \xi_k^{\mathrm{lin}} ξ k lin throughout, including in ξ max \xi_{\max} ξ m a x and δ r o o t \delta_{\mathrm{root}} δ root , and with log ( 8 / ϵ ) \log(8/\epsilon) log ( 8/ ϵ ) in place of log ( 4 / ϵ ) \log(4/\epsilon) log ( 4/ ϵ ) .
Theorem 4.9 (H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S Transfer for InfoNCE). Suppose Assumptions 4.1, 4.3, and 4.4 hold and ρ min > 0 \rho_{\min}>0 ρ m i n > 0 , and fix ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) . If 0 < D ϵ n c e < ρ min / 8 0 < D_\epsilon^{\mathrm{nce}} < \rho_{\min}/8 0 < D ϵ nce < ρ m i n /8 , then there is a deterministic threshold λ 0 ( ϵ ) > 0 \lambda_0(\epsilon)>0 λ 0 ( ϵ ) > 0 such that, for every λ ≥ λ 0 ( ϵ ) \lambda\ge\lambda_0(\epsilon) λ ≥ λ 0 ( ϵ ) , with probability at least 1 − ϵ 1-\epsilon 1 − ϵ , every non-root non-leaf anchor node u u u satisfies Eq. (8) with τ ϵ l i n \tau_\epsilon^{\mathrm{lin}} τ ϵ lin replaced by τ ϵ n c e : = 16 D ϵ n c e / ρ min \tau_\epsilon^{\mathrm{nce}} := 16D_\epsilon^{\mathrm{nce}}/\rho_{\min} τ ϵ nce := 16 D ϵ nce / ρ m i n .
Thus, regularized InfoNCE inherits the linear-loss transfer mechanism. The large-λ \lambda λ condition is required for the analysis but not for the phenomenon: in § 4.4 we show that H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S transfer persists under substantially weaker regularization. We defer the proof to Appendix H.
Remark 4.10 (Connection to the modality gap). CLIP embeds images and text in separated regions of the shared space, which is known as the modality gap [25 ] . Our analysis offers an explanation to this phenomenon. In Eqs. (6) and (11) , each optimal text embedding subtracts a multiple of the global image mean μ r o o t \mu_{\mathrm{root}} μ root . Instead of aligning with the global image mean, the text embeddings subtract it, so the two modalities remain separated at the text optimum.
Extensions beyond the basic transfer model. In Appendix L, we quantify how caption imbalance, inexact prototype consistency, and ancestor omission affect the linear-loss transfer guarantee. Caption imbalance and prototype inconsistency add deterministic error terms, while independent ancestor omissions reduce the effective sample size and introduce random imbalance. In Appendix K, we analyze one exact image-side update after the text-side update under the linear loss, and show that averaging over images sharing the same caption reduces stochastic variation in the updated image embeddings.
Uncaptioned concepts limit hierarchical transfer. In our fixed-image UFM, both regularized losses treat non-root concepts absent from positive captions symmetrically. Their unique minimizers therefore assign these concepts identical text embeddings (Proposition L.4). If an anchor has an uncaptioned descendant and an uncaptioned off-branch concept, the cosine similarity between their embeddings cannot satisfy the H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S condition. Thus, hierarchical image geometry and the caption co-occurrences are both necessary for the successful transfer of hierarchical structure. Our result therefore resonates with the view that data coverage is a key factor in successful compositional generalization.
Limitations. Our analysis explains how pre-existing image geometry transfers to text under a simplified model of contrastive learning. For tractability, we fix the image embeddings and replace CLIP’s unit-norm constraints with quadratic regularization (Remark 4.2). The transfer guarantees require perturbations small relative to the prototype norms; for InfoNCE, we additionally require sufficiently large λ \lambda λ . In § 4.4 , we observe transfer under weaker regularization, but a theoretical guarantee in that regime remains open. Beyond these fixed-image guarantees, our analysis of one image-side update does not explain the full joint training dynamics or how approximate H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S becomes exact. Explaining why hierarchical geometry is already present in image data also remains outside the scope of our transfer analysis. We discuss further limitations in Appendix B.
Synthetic Validation at Small Regularization
We test whether H A S \mathcal{H}AS H A S transfer persists beyond the sufficient conditions used in our InfoNCE analysis. In Theorem 4.9, we guarantee transfer only for sufficiently large regularization λ \lambda λ and sufficiently small perturbations. On a synthetic image-prototype hierarchy, we minimize the InfoNCE objective (9) with L-BFGS over λ ∈ [ 10 − 5 , 10 2 ] \lambda\in[10^{-5}, 10^{2}] λ ∈ [ 1 0 − 5 , 1 0 2 ] . At each minimizer, we report H A S \mathcal{H}AS H A S -Viol and prototype alignment , the minimum over non-root k k k of cos ( a k n c e , ∗ , μ k ) \cos(a_{k}^{\mathrm{nce},*}, \mu_k) cos ( a k nce , ∗ , μ k ) . The full setup is in Appendix § 16 .§ 3 .
H A S \mathcal{H}AS H A S transfer occurs under weaker regularization than required by the analysis. As λ \lambda λ increases, violations vanish at the minimizer before prototype alignment saturates (Figure 5 b). The convergence trace confirms that zero violations persist at the minimizer found by L-BFGS (Figure 5 a). H A S \mathcal{H}AS H A S requires preserving similarity orderings, so transfer can occur before text embeddings align with image prototypes.
Figure 5. Synthetic validation of H A S \mathcal{H}AS H A S transfer at small regularization with the InfoNCE loss. (a) L-BFGS convergence at λ = 10 − 3 \lambda= 10^{-3} λ = 1 0 − 3 : the InfoNCE loss without the regularization term and the text-side H A S \mathcal{H}AS H A S -Viol against the training step. (b) At the minimizer, text-side H A S \mathcal{H}AS H A S -Viol reaches zero at smaller λ \lambda λ than is needed for prototype alignment to saturate.
Transfer also occurs outside the sufficient perturbation bound: the root norm in this experiment violates the bound, yet H A S \mathcal{H}AS H A S -Viol reaches zero. Violations also remain small across different root norms and variation strengths (Appendix § 16 .§ 3 ).
To test whether transfer still depends on hierarchical co-occurrence, we permute hypernym–image pairings while preserving caption frequencies. Permutation substantially increases text violations under both the linear and InfoNCE losses (Appendix § 16 .§ 5 ), so caption frequencies alone does not imply transfer.
Conclusion
In this paper, we studied how concept embeddings learned without explicit hierarchical supervision encode the underlying concept hierarchy. We measured this property with Hierarchical Angular Separation (H A S \mathcal{H}AS H A S ): for each non-leaf concept, a node has a higher cosine similarity to its descendant concepts than to concepts outside its subtree. We first showed empirically that H A S \mathcal{H}AS H A S holds approximately in CLIP text embeddings only after pretraining, while it is already approximately present in image embeddings at random initialization. To explain this phenomenon, we analyzed an Unconstrained Features Model in which we freeze the image embeddings and train the text embeddings using either a linear contrastive loss or the InfoNCE loss with sufficient regularization. Under hierarchical co-occurrence, we proved that the text-side H A S \mathcal{H}AS H A S violation percentage at each anchor is bounded by the image-side H A S \mathcal{H}AS H A S violation and near-violation percentage. Numerical experiments on synthetic concept hierarchies confirm these predictions and indicate that H A S \mathcal{H}AS H A S is induced even with weaker regularization than theoretically required. Our results suggest that multimodal contrastive learning transfers hierarchical structure already present in image geometry to text embeddings. Whether this mechanism extends beyond hierarchy to sparse, low-rank, or other compositional structures, and why image data exhibit hierarchy in the first place, remain open.
Acknowledgments and Disclosure of Funding
The authors thank Ryan Chan, Ryan Pilgrim, Uday Kiran Tadipatri, Buyun Liang, Kyle Poe, Leandro Palma, Yi Ma, and anonymous reviewers for their helpful comments and suggestions. This work was funded by Simons Foundation grant 814201, NSF grant 2031985, and University of Pennsylvania startup funds.
Formalizations of Hierarchy in Embedding Spaces
Several prior works have formalized how an embedding of a rooted tree encodes the hierarchy. We group these formalizations into two categories:
Prescriptive formalizations, which incorporate an explicit geometric constraint (e.g., a cone with a chosen angle) into the training objective to enforce it.
Descriptive formalizations, which characterize a property of the learned embeddings without building it into the loss.
Our Definition 3.1 is descriptive: We state a pairwise cosine-similarity inequality to be verified for trained embeddings. We compare with the closest prior work in Appendix § 7 .§ 1 and note other formalizations in Appendix § 7 .§ 2 .
Hyperbolic Entailment Cones
The most comparable prior formalization is the hyperbolic entailment cone line of work [12 , 13 , 14 ] . Each non-root embedding x x x is assigned a convex cone C x C_x C x with a half-angle ψ ( x ) \psi(x) ψ ( x ) specified as a function of x x x ,2 and the loss penalizes violations of cone inclusion as descendants of u u u must lie inside C a u \mathcal{C}_{a_u} C a u , while non-descendants must lie outside. The angle is thus determined in advance and enforced as a training margin.
H A S \mathcal{H}AS H A S can be viewed as a Euclidean version of this condition, but descriptive rather than prescriptive : the pairwise inequality cos ( a u , a v ) > cos ( a u , a w ) \cos(a_u, a_v) > \cos(a_u, a_w) cos ( a u , a v ) > cos ( a u , a w ) is equivalent to strict cone containment with the angle determined by the nearest off-branch embedding, so the angle is determined by the data rather than imposed by the loss.
Other Formalizations
Poincaré embeddings. Nickel and Kiela [34 ] learn embeddings in the Poincaré ball via a soft-ranking loss, yielding a hierarchical ordering of concepts (general concepts near the origin, specific ones near the boundary) without an explicit descendant-vs-non-descendant inequality. In Definition 3.1 , we make the inequality explicit.
Hyperbolic tree distortion and δ \delta δ -hyperbolicity. A separate line of work formalizes hierarchy at the metric level: Sala et al. [50 ] bound the distortion of the tree metric under a hyperbolic embedding, and Adcock et al. [51 ] characterize tree-likeness via Gromov’s δ \delta δ -hyperbolicity. These are metric (distance-based) formalizations. In contrast, H A S \mathcal{H}AS H A S is a directional (angle-based) one and makes no claim about metric distortion.
Limitations of our analysis
Our analysis isolates how hierarchical image-caption co-occurrence transfers existing image geometry to text. We discuss the scope of this mechanism and the assumptions that limit its application to CLIP training.
Fixed image embeddings. The fixed-image UFM captures the direction of transfer suggested by the observed modality asymmetry: hierarchical geometry is already approximately present in image embeddings, and text embeddings acquire it through contrastive learning. By optimizing text concept embeddings directly, we can identify hierarchical image-caption co-occurrence as the mechanism that aligns text embeddings with root-centered image prototypes. The model therefore explains how an alignment objective can transfer hierarchical structure without explicit supervision. The limitation is that assuming fixed image embeddings does not capture joint training dynamics. We also replace CLIP’s unit-norm embeddings with unnormalized embeddings and quadratic regularization for tractability (Remark 4.2).
Image embedding optimization analysis. The image-side extension in Appendix § 17 identifies noise reduction within caption groups after an exact text update followed by an exact image update under the linear loss. How that mechanism affects subtree-aggregated image prototypes during joint gradient descent, and how the initial image-side hierarchy arises, remain open.
Approximate H A S \mathcal{H}AS H A S guarantees and empirical measurements. The main theorems bound text-side violations using the image-side percentage of off-branch nodes whose worst-case triplet margin is at most the transfer threshold. The guarantees require small perturbations relative to the prototype norms. This sufficient condition fails for the real image prototypes examined in Appendix § 16 .§ 4 , so the main theorems give no guarantee for the observed transfer there. Deriving tighter guarantees, therefore, remains an open problem.
Sufficient regularization for InfoNCE. The InfoNCE guarantee requires sufficiently large regularization to control the remainder in the expansion around uniform softmax weights. Experiments in Appendix § 16 .§ 3 show HAS transfer at weaker regularization, but do not extend the theorem to that regime. A guarantee for weaker regularization, or for CLIP’s normalized objective, remains open.
Supporting Lemmas
Softmax expansions
Lemma C.1 (Softmax partial derivatives). For K ≥ 1 K \ge1 K ≥ 1 , define softmax : R K → R K \mathbb{R}^{K} \to\mathbb{R}^{K} R K → R K by
( softmax ( s ) ) j : = e s j ∑ r = 1 K e s r , j ∈ [ K ] . (12) \left(\operatorname{softmax}(\mathbf{s})\right)_{j} := \frac{e^{s_j}}{\sum_{r=1}^{K} e^{s_r}}, \qquad j \in[K]. \tag*{(12)} ( softmax ( s ) ) j := ∑ r = 1 K e s r e s j , j ∈ [ K ] . (12)
Then, for every j , ℓ ∈ [ K ] j,\ell\in[K] j , ℓ ∈ [ K ] ,
∂ ∂ s ℓ ( softmax ( s ) ) j = ( softmax ( s ) ) j ( δ j ℓ − ( softmax ( s ) ) ℓ ) . (13) \frac{\partial}{\partial s_{\ell}}\left(\operatorname{softmax}(\mathbf{s})\right)_j=\left(\operatorname{softmax}(\mathbf{s})\right)_j\left(\delta_{j\ell}-\left(\operatorname{softmax}(\mathbf{s})\right)_{\ell}\right). \tag*{(13)} ∂ s ℓ ∂ ( softmax ( s ) ) j = ( softmax ( s ) ) j ( δ j ℓ − ( softmax ( s ) ) ℓ ) . (13)
where δ j ℓ \delta_{j\ell} δ j ℓ is the Kronecker delta, equal to 1 when j = ℓ j=\ell j = ℓ and 0 otherwise.
Proof. Write
p j ( s ) : = ( softmax ( s ) ) j = e s j Z ( s ) , Z ( s ) : = ∑ r = 1 K e s r . (14) p_j(\mathbf{s}):=\left(\operatorname{softmax}(\mathbf{s})\right)_j=\frac{e^{s_j}}{Z(\mathbf{s})},\qquad Z(\mathbf{s}):=\sum_{r=1}^{K}e^{s_r}. \tag*{(14)} p j ( s ) := ( softmax ( s ) ) j = Z ( s ) e s j , Z ( s ) := r = 1 ∑ K e s r . (14)
Differentiating with respect to s ℓ s_{\ell} s ℓ and using the quotient rule gives
∂ p j ∂ s ℓ = ( ∂ s ℓ e s j ) Z ( s ) − e s j ∂ s ℓ Z ( s ) Z ( s ) 2 = δ j ℓ e s j Z ( s ) − e s j e s ℓ Z ( s ) 2 = e s j Z ( s ) ( δ j ℓ − e s ℓ Z ( s ) ) = softmax ( s ) j ( δ j ℓ − softmax ( s ) ℓ ) . (15) \begin{aligned}
\frac{\partial p_j}{\partial s_{\ell}}&=\frac{(\partial_{s_{\ell}}e^{s_j})Z(\mathbf{s})-e^{s_j}\partial_{s_{\ell}}Z(\mathbf{s})}{Z(\mathbf{s})^2}\\
&=\frac{\delta_{j\ell}e^{s_j}Z(\mathbf{s})-e^{s_j}e^{s_{\ell}}}{Z(\mathbf{s})^2}\\
&=\frac{e^{s_j}}{Z(\mathbf{s})}\left(\delta_{j\ell}-\frac{e^{s_{\ell}}}{Z(\mathbf{s})}\right)\\
&=\operatorname{softmax}(\mathbf{s})_j\left(\delta_{j\ell}-\operatorname{softmax}(\mathbf{s})_{\ell}\right). \tag*{(15)}
\end{aligned} ∂ s ℓ ∂ p j = Z ( s ) 2 ( ∂ s ℓ e s j ) Z ( s ) − e s j ∂ s ℓ Z ( s ) = Z ( s ) 2 δ j ℓ e s j Z ( s ) − e s j e s ℓ = Z ( s ) e s j ( δ j ℓ − Z ( s ) e s ℓ ) = softmax ( s ) j ( δ j ℓ − softmax ( s ) ℓ ) . (15)
Lemma C.2 (Quantitative softmax deviation from uniform). For K ≥ 1 K\ge1 K ≥ 1 , define softmax : R K → R K \operatorname{softmax}:\mathbb{R}^{K}\to\mathbb{R}^{K} softmax : R K → R K as in Lemma C.1. Then, for every s ∈ R K \mathbf{s}\in\mathbb{R}^{K} s ∈ R K and every j ∈ [ K ] j\in[K] j ∈ [ K ] ,
∣ softmax ( s ) j − 1 K ∣ ≤ 2 ∥ s ∥ . (16) \left|\operatorname{softmax}(\mathbf{s})_j-\frac{1}{K}\right|\le2\|\mathbf{s}\|. \tag*{(16)} softmax ( s ) j − K 1 ≤ 2∥ s ∥. (16) Proof. Let f j ( s ) : = softmax ( s ) j f_j(\mathbf{s}):=\operatorname{softmax}(\mathbf{s})_j f j ( s ) := softmax ( s ) j . By Lemma C.1,
∇ f j ( s ) = f j ( s ) ( e j − softmax ( s ) ) , (17) \nabla f_j(\mathbf{s})=f_j(\mathbf{s})(\mathbf{e}_j-\operatorname{softmax}(\mathbf{s})), \tag*{(17)} ∇ f j ( s ) = f j ( s ) ( e j − softmax ( s )) , (17) where e j \mathbf{e}_j e j is the j j j th standard basis vector in R K \mathbb{R}^{K} R K . Since 0 ≤ f j ( s ) ≤ 1 0\le f_j(\mathbf{s})\le1 0 ≤ f j ( s ) ≤ 1 and ∥ softmax ( s ) ∥ ≤ 1 \|\operatorname{softmax}(\mathbf{s})\|\le1 ∥ softmax ( s ) ∥ ≤ 1 , we have ∥ ∇ f j ( s ) ∥ ≤ 2 \|\nabla f_j(\mathbf{s})\|\le2 ∥∇ f j ( s ) ∥ ≤ 2 for every s ∈ R K \mathbf{s}\in\mathbb{R}^{K} s ∈ R K . By the fundamental theorem of calculus along the segment { τ s : τ ∈ [ 0 , 1 ] } \{\tau\mathbf{s}:\tau\in[0,1]\} { τ s : τ ∈ [ 0 , 1 ]} ,
f j ( s ) − f j ( 0 ) = ∫ 0 1 ∇ f j ( τ s ) T s d τ . (18) f_j(\mathbf{s})-f_j(\mathbf{0})=\int_0^1\nabla f_j(\tau\mathbf{s})^{\mathsf{T}}\mathbf{s}\,d\tau. \tag*{(18)} f j ( s ) − f j ( 0 ) = ∫ 0 1 ∇ f j ( τ s ) T s d τ . (18) Therefore
∣ f j ( s ) − f j ( 0 ) ∣ ≤ ∫ 0 1 ∥ ∇ f j ( τ s ) ∥ ∥ s ∥ d τ ≤ 2 ∥ s ∥ . (19) \left|f_j(\mathbf{s})-f_j(\mathbf{0})\right|\le\int_0^1\|\nabla f_j(\tau\mathbf{s})\|\,\|\mathbf{s}\|\,d\tau\le2\|\mathbf{s}\|. \tag*{(19)} ∣ f j ( s ) − f j ( 0 ) ∣ ≤ ∫ 0 1 ∥∇ f j ( τ s ) ∥ ∥ s ∥ d τ ≤ 2∥ s ∥. (19) Since f j ( 0 ) = 1 / K f_j(\mathbf{0})=1/K f j ( 0 ) = 1/ K , this proves Eq. (16).
Lemma C.3 (Softmax near the zero vector). For K ≥ 1 K\ge1 K ≥ 1 , define softmax : R K → R K \operatorname{softmax}:\mathbb{R}^{K}\to\mathbb{R}^{K} softmax : R K → R K by
( softmax ( s ) ) j : = e s j ∑ ℓ = 1 K e s ℓ , j ∈ [ K ] . (20) \left(\operatorname{softmax}(\mathbf{s})\right)_j:=\frac{e^{s_j}}{\sum_{\ell=1}^{K}e^{s_{\ell}}},\qquad j\in[K]. \tag*{(20)} ( softmax ( s ) ) j := ∑ ℓ = 1 K e s ℓ e s j , j ∈ [ K ] . (20) Let ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ denote the Euclidean norm on R K \mathbb{R}^{K} R K . Then ( softmax ( 0 ) ) j = 1 K \left(\operatorname{softmax}(\mathbf{0})\right)_j=\frac{1}{K} ( softmax ( 0 ) ) j = K 1 for every j ∈ [ K ] j\in[K] j ∈ [ K ] . Moreover, if s → 0 \mathbf{s}\to\mathbf{0} s → 0 , then
( softmax ( s ) ) j = 1 K + 1 K ( s j − 1 K ∑ ℓ = 1 K s ℓ ) + O ( ∥ s ∥ 2 ) , (21) \left(\operatorname{softmax}(\mathbf{s})\right)_j=\frac{1}{K}+\frac{1}{K}\left(s_j-\frac{1}{K}\sum_{\ell=1}^{K}s_{\ell}\right)+O(\|\mathbf{s}\|^2), \tag*{(21)} ( softmax ( s ) ) j = K 1 + K 1 ( s j − K 1 ℓ = 1 ∑ K s ℓ ) + O ( ∥ s ∥ 2 ) , (21) uniformly over j ∈ [ K ] j\in[K] j ∈ [ K ] . In particular, if ∥ s ∥ = O ( ε ) \|\mathbf{s}\|=O(\varepsilon) ∥ s ∥ = O ( ε ) , then
( softmax ( s ) ) j = 1 K + O ( ε ) , (22) \left(\operatorname{softmax}(\mathbf{s})\right)_j=\frac{1}{K}+O(\varepsilon), \tag*{(22)} ( softmax ( s ) ) j = K 1 + O ( ε ) , (22) uniformly over $j\in[K].
Proof. At s = 0 \mathbf{s} = 0 s = 0 , all exponentials equal 1 1 1 , so
( softmax ( 0 ) ) j = 1 ∑ ℓ = 1 K 1 = 1 K . (23) (\operatorname{softmax}(\mathbf{0}))_j = \frac{1}{\sum_{\ell=1}^{K} 1} = \frac{1}{K}. \tag*{(23)} ( softmax ( 0 ) ) j = ∑ ℓ = 1 K 1 1 = K 1 . (23)
Since softmax is smooth on R K \mathbb{R}^{K} R K , Taylor’s theorem around 0 \mathbf{0} 0 with f j ( s ) : = ( softmax ( s ) ) j f_j(\mathbf{s}) := (\operatorname{softmax}(\mathbf{s}))_j f j ( s ) := ( softmax ( s ) ) j gives
f j ( s ) = f j ( 0 ) + ∇ f j ( 0 ) T s + O ( ∥ s ∥ 2 ) . (24) f_j(\mathbf{s}) = f_j(\mathbf{0}) + \nabla f_j(\mathbf{0})^{\mathsf{T}}\mathbf{s} + O(\lVert\mathbf{s}\rVert^2). \tag*{(24)} f j ( s ) = f j ( 0 ) + ∇ f j ( 0 ) T s + O (∥ s ∥ 2 ) . (24)
Equivalently,
( softmax ( s ) ) j = ( softmax ( 0 ) ) j + ∑ ℓ = 1 K ∂ f j ∂ s ℓ ( 0 ) s ℓ + O ( ∥ s ∥ 2 ) . (25) (\operatorname{softmax}(\mathbf{s}))_j = (\operatorname{softmax}(\mathbf{0}))_j + \sum_{\ell=1}^{K} \frac{\partial f_j}{\partial s_\ell}(\mathbf{0})s_\ell+ O(\lVert\mathbf{s}\rVert^2). \tag*{(25)} ( softmax ( s ) ) j = ( softmax ( 0 ) ) j + ℓ = 1 ∑ K ∂ s ℓ ∂ f j ( 0 ) s ℓ + O (∥ s ∥ 2 ) . (25)
By Lemma C.1,
∂ ∂ s ℓ ( softmax ( s ) ) j = ( softmax ( s ) ) j ( δ j ℓ − ( softmax ( s ) ) ℓ ) , (26) \frac{\partial}{\partial s_\ell}(\operatorname{softmax}(\mathbf{s}))_j = (\operatorname{softmax}(\mathbf{s}))_j\left(\delta_{j\ell} - (\operatorname{softmax}(\mathbf{s}))_\ell\right), \tag*{(26)} ∂ s ℓ ∂ ( softmax ( s ) ) j = ( softmax ( s ) ) j ( δ j ℓ − ( softmax ( s ) ) ℓ ) , (26)
so at s = 0 \mathbf{s} = \mathbf{0} s = 0 we obtain
∂ f j ∂ s ℓ ( 0 ) = 1 K ( δ j ℓ − 1 K ) . (27) \frac{\partial f_j}{\partial s_\ell}(\mathbf{0}) = \frac{1}{K}\left(\delta_{j\ell} - \frac{1}{K}\right). \tag*{(27)} ∂ s ℓ ∂ f j ( 0 ) = K 1 ( δ j ℓ − K 1 ) . (27)
Therefore the linear term in the Taylor expansion is
∇ f j ( 0 ) T s = ∑ ℓ = 1 K 1 K ( δ j ℓ − 1 K ) s ℓ = 1 K ( s j − 1 K ∑ ℓ = 1 K s ℓ ) . (28) \begin{aligned}
\nabla f_j(\mathbf{0})^{\mathsf{T}}\mathbf{s}
&= \sum_{\ell=1}^{K} \frac{1}{K}\left(\delta_{j\ell} - \frac{1}{K}\right)s_\ell\\
&= \frac{1}{K}\left(s_j - \frac{1}{K}\sum_{\ell=1}^{K}s_\ell\right). \tag*{(28)}
\end{aligned} ∇ f j ( 0 ) T s = ℓ = 1 ∑ K K 1 ( δ j ℓ − K 1 ) s ℓ = K 1 ( s j − K 1 ℓ = 1 ∑ K s ℓ ) . (28)
Moreover,
∣ ∇ f j ( 0 ) T s ∣ ≤ 1 K ( ∣ s j ∣ + 1 K ∣ ∑ ℓ = 1 K s ℓ ∣ ) ≤ 1 K ( ∥ s ∥ + 1 K ∥ 1 ∥ ∥ s ∥ ) = 1 K ( 1 + 1 K ) ∥ s ∥ , (29) \left|\nabla f_j(\mathbf{0})^{\mathsf{T}}\mathbf{s}\right| \leq\frac{1}{K}\left(\left|s_j\right| + \frac{1}{K}\left|\sum_{\ell=1}^{K}s_\ell\right|\right)
\leq\frac{1}{K}\left(\lVert\mathbf{s}\rVert+ \frac{1}{K}\lVert\mathbf{1}\rVert\lVert\mathbf{s}\rVert\right) = \frac{1}{K}\left(1 + \frac{1}{\sqrt{K}}\right)\lVert\mathbf{s}\rVert, \tag*{(29)} ∇ f j ( 0 ) T s ≤ K 1 ( ∣ s j ∣ + K 1 ℓ = 1 ∑ K s ℓ ) ≤ K 1 ( ∥ s ∥ + K 1 ∥ 1 ∥ ∥ s ∥ ) = K 1 ( 1 + K 1 ) ∥ s ∥ , (29)
where 1 ∈ R K \mathbf{1} \in\mathbb{R}^{K} 1 ∈ R K is the all-ones vector. Thus the linear term is O ( ∥ s ∥ ) O(\lVert\mathbf{s}\rVert) O (∥ s ∥) . Substituting f j ( 0 ) = 1 K f_j(\mathbf{0}) = \frac{1}{K} f j ( 0 ) = K 1 and this expression for ∇ f j ( 0 ) T s \nabla f_j(\mathbf{0})^{\mathsf{T}}\mathbf{s} ∇ f j ( 0 ) T s into the Taylor expansion yields Eq. (21). If ∥ s ∥ = O ( ε ) \lVert\mathbf{s}\rVert= O(\varepsilon) ∥ s ∥ = O ( ε ) , then the linear term is O ( ε ) O(\varepsilon) O ( ε ) and the remainder is O ( ε 2 ) O(\varepsilon^2) O ( ε 2 ) , which gives the final claim.
Prototype geometry and margin transfer
Lemma C.4 (Balanced Descendant Prototype Averaging). Assume the tree is balanced with branching factor b b b , and the prototypes { μ k } \{\mu_k\} { μ k } are constructed as in Appendix I. Then for every node k k k , the average of the leaf prototypes in the subtree rooted at k k k is exactly μ k \mu_k μ k .
Proof. The claim is immediate for leaves. Now let k k k be an internal node with children v 1 , … , v b v_1,\ldots,v_b v 1 , … , v b . By construction,
μ v j = μ k + U d j , j = 1 , … , b , (30) \mu_{v_j} = \mu_k + U d_j,\qquad j = 1,\ldots,b, \tag*{(30)} μ v j = μ k + U d j , j = 1 , … , b , (30)
where the embedded simplex offsets satisfy ∑ j = 1 b d j = 0 \sum_{j=1}^{b}d_j = 0 ∑ j = 1 b d j = 0 . Hence the child average is
1 b ∑ j = 1 b μ v j = μ k + U ( 1 b ∑ j = 1 b d j ) = μ k . (31) \frac{1}{b}\sum_{j=1}^{b}\mu_{v_j} = \mu_k + U\left(\frac{1}{b}\sum_{j=1}^{b}d_j\right) = \mu_k. \tag*{(31)} b 1 j = 1 ∑ b μ v j = μ k + U ( b 1 j = 1 ∑ b d j ) = μ k . (31)
Because the tree is balanced, each child subtree contains the same number of leaves. Applying the induction hypothesis within each child subtree therefore shows that the leaf-descendant average of k k k is the same average of the child averages, hence equal to μ k \mu_k μ k .
Lemma C.5 (Cosine stability under small perturbations). Let x , y ∈ R d x,y \in\mathbb{R}^{d} x , y ∈ R d be nonzero, let e x , e y ∈ R d e_x,e_y \in\mathbb{R}^{d} e x , e y ∈ R d , and set
ρ : = min { ∥ x ∥ , ∥ y ∥ } . (32) \rho:= \min\{\lVert x\rVert,\lVert y\rVert\}. \tag*{(32)} ρ := min {∥ x ∥ , ∥ y ∥} . (32) If
max { ∥ e x ∥ , ∥ e y ∥ } ≤ δ < ρ , (33) \max\{\lVert e_x\rVert,\lVert e_y\rVert\} \le\delta< \rho, \tag*{(33)} max {∥ e x ∥ , ∥ e y ∥} ≤ δ < ρ , (33) then
∣ cos ( x + e x , y + e y ) − cos ( x , y ) ∣ ≤ 4 δ ρ . (34) \left|\cos(x+e_x,y+e_y)-\cos(x,y)\right| \le\frac{4\delta}{\rho}. \tag*{(34)} ∣ cos ( x + e x , y + e y ) − cos ( x , y ) ∣ ≤ ρ 4 δ . (34)
Proof. For any nonzero z ∈ R d z \in\mathbb{R}^{d} z ∈ R d and perturbation e e e with ∥ e ∥ < ∥ z ∥ \lVert e\rVert< \lVert z\rVert ∥ e ∥ < ∥ z ∥ , define ν ( z ) : = z / ∥ z ∥ \nu(z):=z/\lVert z\rVert ν ( z ) := z / ∥ z ∥ . Then
∥ ν ( z + e ) − ν ( z ) ∥ = ∥ z + e ∥ z + e ∥ − z + e ∥ z ∥ + z + e ∥ z ∥ − z ∥ z ∥ ∥ ≤ ∥ z + e ∥ z + e ∥ − z + e ∥ z ∥ ∥ + ∥ z + e ∥ z ∥ − z ∥ z ∥ ∥ = ∣ ∥ z ∥ − ∥ z + e ∥ ∣ ∥ z ∥ + ∥ e ∥ ∥ z ∥ ≤ ∥ e ∥ ∥ z ∥ + ∥ e ∥ ∥ z ∥ = 2 ∥ e ∥ ∥ z ∥ , (35) \begin{aligned}
\lVert\nu(z+e)-\nu(z)\rVert
&=\left\lVert\frac{z+e}{\lVert z+e\rVert}-\frac{z+e}{\lVert z\rVert}+\frac{z+e}{\lVert z\rVert}-\frac{z}{\lVert z\rVert}\right\rVert\\
&\le\left\lVert\frac{z+e}{\lVert z+e\rVert}-\frac{z+e}{\lVert z\rVert}\right\rVert+\left\lVert\frac{z+e}{\lVert z\rVert}-\frac{z}{\lVert z\rVert}\right\rVert\\
&=\frac{\left|\lVert z\rVert-\lVert z+e\rVert\right|}{\lVert z\rVert}+\frac{\lVert e\rVert}{\lVert z\rVert}\\
&\le\frac{\lVert e\rVert}{\lVert z\rVert}+\frac{\lVert e\rVert}{\lVert z\rVert}\\
&=\frac{2\lVert e\rVert}{\lVert z\rVert}, \tag*{(35)}
\end{aligned} ∥ ν ( z + e ) − ν ( z )∥ = ∥ z + e ∥ z + e − ∥ z ∥ z + e + ∥ z ∥ z + e − ∥ z ∥ z ≤ ∥ z + e ∥ z + e − ∥ z ∥ z + e + ∥ z ∥ z + e − ∥ z ∥ z = ∥ z ∥ ∣ ∥ z ∥ − ∥ z + e ∥ ∣ + ∥ z ∥ ∥ e ∥ ≤ ∥ z ∥ ∥ e ∥ + ∥ z ∥ ∥ e ∥ = ∥ z ∥ 2 ∥ e ∥ , (35)
where the first inequality is the triangle inequality after adding and subtracting ( z + e ) / ∥ z ∥ (z+e)/\lVert z\rVert ( z + e ) / ∥ z ∥ , and the second inequality is the reverse triangle inequality
∣ ∥ z ∥ − ∥ z + e ∥ ∣ ≤ ∥ e ∥ . (36) \left|\lVert z\rVert-\lVert z+e\rVert\right|\le\lVert e\rVert. \tag*{(36)} ∣ ∥ z ∥ − ∥ z + e ∥ ∣ ≤ ∥ e ∥ . (36)
Applying this bound with ( z , e ) = ( x , e x ) (z,e)=(x,e_x) ( z , e ) = ( x , e x ) and ( z , e ) = ( y , e y ) (z,e)=(y,e_y) ( z , e ) = ( y , e y ) gives
∥ ν ( x + e x ) − ν ( x ) ∥ ≤ 2 δ ρ , ∥ ν ( y + e y ) − ν ( y ) ∥ ≤ 2 δ ρ . (37) \lVert\nu(x+e_x)-\nu(x)\rVert\le\frac{2\delta}{\rho},\qquad\lVert\nu(y+e_y)-\nu(y)\rVert\le\frac{2\delta}{\rho}. \tag*{(37)} ∥ ν ( x + e x ) − ν ( x )∥ ≤ ρ 2 δ , ∥ ν ( y + e y ) − ν ( y )∥ ≤ ρ 2 δ . (37)
Therefore
∣ cos ( x + e x , y + e y ) − cos ( x , y ) ∣ = ∣ ν ( x + e x ) ⊤ ν ( y + e y ) − ν ( x ) ⊤ ν ( y ) ∣ = ∣ ν ( x + e x ) ⊤ ν ( y + e y ) − ν ( x ) ⊤ ν ( y + e y ) + ν ( x ) ⊤ ν ( y + e y ) − ν ( x ) ⊤ ν ( y ) ∣ ≤ ∣ ( ν ( x + e x ) − ν ( x ) ) ⊤ ν ( y + e y ) ∣ + ∣ ν ( x ) ⊤ ( ν ( y + e y ) − ν ( y ) ) ∣ ≤ ∥ ν ( x + e x ) − ν ( x ) ∥ ∥ ν ( y + e y ) ∥ + ∥ ν ( x ) ∥ ∥ ν ( y + e y ) − ν ( y ) ∥ = ∥ ν ( x + e x ) − ν ( x ) ∥ + ∥ ν ( y + e y ) − ν ( y ) ∥ ≤ 4 δ ρ . (38) \begin{aligned}
\left|\cos(x+e_x,y+e_y)-\cos(x,y)\right|
&=\left|\nu(x+e_x)^{\top}\nu(y+e_y)-\nu(x)^{\top}\nu(y)\right|\\
&=\left|\nu(x+e_x)^{\top}\nu(y+e_y)-\nu(x)^{\top}\nu(y+e_y)+\nu(x)^{\top}\nu(y+e_y)-\nu(x)^{\top}\nu(y)\right|\\
&\le\left|(\nu(x+e_x)-\nu(x))^{\top}\nu(y+e_y)\right|+\left|\nu(x)^{\top}(\nu(y+e_y)-\nu(y))\right|\\
&\le\lVert\nu(x+e_x)-\nu(x)\rVert\lVert\nu(y+e_y)\rVert+\lVert\nu(x)\rVert\lVert\nu(y+e_y)-\nu(y)\rVert\\
&=\lVert\nu(x+e_x)-\nu(x)\rVert+\lVert\nu(y+e_y)-\nu(y)\rVert\\
&\le\frac{4\delta}{\rho}. \tag*{(38)}
\end{aligned} ∣ cos ( x + e x , y + e y ) − cos ( x , y ) ∣ = ν ( x + e x ) ⊤ ν ( y + e y ) − ν ( x ) ⊤ ν ( y ) = ν ( x + e x ) ⊤ ν ( y + e y ) − ν ( x ) ⊤ ν ( y + e y ) + ν ( x ) ⊤ ν ( y + e y ) − ν ( x ) ⊤ ν ( y ) ≤ ( ν ( x + e x ) − ν ( x ) ) ⊤ ν ( y + e y ) + ν ( x ) ⊤ ( ν ( y + e y ) − ν ( y )) ≤ ∥ ν ( x + e x ) − ν ( x )∥ ∥ ν ( y + e y )∥ + ∥ ν ( x )∥ ∥ ν ( y + e y ) − ν ( y )∥ = ∥ ν ( x + e x ) − ν ( x )∥ + ∥ ν ( y + e y ) − ν ( y )∥ ≤ ρ 4 δ . (38)
where the first inequality is the triangle inequality and the second inequality is Cauchy–Schwarz.
In Theorems 4.6 and 4.9, we compare two embedding families at a common anchor, so each quantity defined below is written with a superscript for the family h = { h k } k ∈ V ∖ { r o o t } h=\{h_k\}_{k\in V\setminus\{\mathrm{root}\}} h = { h k } k ∈ V ∖ { root } of nonzero vectors it is evaluated on. The first of the two is the quantity that must be positive in Definition 3.1.
Definition C.6 (Admissible triplet and triplet margin). Write O u : = V ∖ ( Sub ( u ) ∪ Anc ( u ) ) O_u:=V\setminus(\operatorname{Sub}(u)\cup\operatorname{Anc}(u)) O u := V ∖ ( Sub ( u ) ∪ Anc ( u )) for the off-branch set at u u u , which forms the denominator of Eq. (2) , and call ( u , v , w ) (u,v,w) ( u , v , w ) an admissible triplet if u ∈ V ∖ { r o o t } u\in V\setminus\{\mathrm{root}\} u ∈ V ∖ { root } is non-leaf, v ∈ Desc ( u ) v\in\operatorname{Desc}(u) v ∈ Desc ( u ) , and w ∈ O u w\in O_u w ∈ O u . The triplet margin of an admissible triplet under h h h is
γ u v w h : = cos ( h u , h v ) − cos ( h u , h w ) . (39) \gamma^{h}_{uvw}:=\cos(h_u,h_v)-\cos(h_u,h_w). \tag*{(39)} γ uv w h := cos ( h u , h v ) − cos ( h u , h w ) . (39)
When admissible triplets exist for a family of nonzero image prototypes, define their global margin by
γ H A S : = min ( u , v , w ) a d m i s s i b l e γ u v w μ . (40) \gamma_{\mathcal{H}AS} := \min_{(u,v,w)\ \mathrm{admissible}} \gamma_{uvw}^{\mu}. \tag*{(40)} γ H A S := ( u , v , w ) admissible min γ uv w μ . (40)
The second definition is Definition 3.2 with its descendant threshold lowered by a slack.
Definition C.7 (Slack violation percentage). Let s u h : = min v ∈ Desc ( u ) cos ( h u , h v ) s_u^h := \min_{v \in\operatorname{Desc}(u)} \cos(h_u,h_v) s u h := min v ∈ Desc ( u ) cos ( h u , h v ) be the descendant threshold s u s_u s u of Definition 3.2 evaluated on h h h . For a non-root non-leaf node u u u and a slack τ ≥ 0 \tau\ge0 τ ≥ 0 , the slack-τ \tau τ violation percentage is
H A S - Viol τ h ( u ) : = ∣ { w ∈ O u : cos ( h u , h w ) ≥ s u h − τ } ∣ ∣ O u ∣ × 100 % . (41) \mathcal{H}AS\text{-}\operatorname{Viol}_{\tau}^{h}(u) := \frac{\left|\left\{w \in O_u : \cos(h_u,h_w) \ge s_u^h-\tau\right\}\right|}{|O_u|} \times100\%. \tag*{(41)} H A S - Viol τ h ( u ) := ∣ O u ∣ { w ∈ O u : cos ( h u , h w ) ≥ s u h − τ } × 100%. (41) and we drop the subscript at τ = 0 \tau=0 τ = 0 , writing H A S - Viol h ( u ) : = H A S - Viol 0 h ( u ) \mathcal{H}AS\text{-}\operatorname{Viol}^{h}(u) := \mathcal{H}AS\text{-}\operatorname{Viol}_{0}^{h}(u) H A S - Viol h ( u ) := H A S - Viol 0 h ( u ) .
At τ = 0 \tau=0 τ = 0 the threshold is unchanged, so H A S - Viol h ( u ) \mathcal{H}AS\text{-}\operatorname{Viol}^{h}(u) H A S - Viol h ( u ) is exactly the violation percentage H A S - Viol ( u ) \mathcal{H}AS\text{-}\operatorname{Viol}(u) H A S - Viol ( u ) of Definition 3.2 for the family h h h . In the main text the superscript is a mnemonic rather than a symbol: H A S - Viol i m g ( u ) \mathcal{H}AS\text{-}\operatorname{Viol}^{\mathrm{img}}(u) H A S - Viol img ( u ) is H A S - Viol μ ( u ) \mathcal{H}AS\text{-}\operatorname{Viol}^{\mu}(u) H A S - Viol μ ( u ) on the image prototypes, and H A S - Viol t x t ( u ) \mathcal{H}AS\text{-}\operatorname{Viol}^{\mathrm{txt}}(u) H A S - Viol txt ( u ) is H A S - Viol h ( u ) \mathcal{H}AS\text{-}\operatorname{Viol}^{h}(u) H A S - Viol h ( u ) with h h h the optimal text embeddings under consideration.
The two definitions are connected through the identity min v ∈ Desc ( u ) γ u v w μ = s u h − cos ( h u , h w ) \min_{v\in\operatorname{Desc}(u)}\gamma_{uvw}^{\mu}=s_u^h-\cos(h_u,h_w) min v ∈ Desc ( u ) γ uv w μ = s u h − cos ( h u , h w ) , which holds for each fixed off-branch node w w w . The percentage H A S - Viol τ h ( u ) \mathcal{H}AS\text{-}\operatorname{Viol}_{\tau}^{h}(u) H A S - Viol τ h ( u ) is therefore the percentage of off-branch nodes w w w whose worst-case triplet margin at anchor u u u is at most τ \tau τ .
Lemma C.8 is the deterministic core shared by Theorems 4.6 and 4.9. It assumes nothing about the prototype variation model, the observed captions, or the global margin γ H A S \gamma_{\mathcal{H}AS} γ H A S , and all randomness in either theorem enters only through the bound η \eta η on the perturbations.
Lemma C.8 (Per-triplet H A S \mathcal{H}AS H A S margin preservation). Let ν = { ν k } k ∈ V ∖ { r o o t } \boldsymbol{\nu}=\{\nu_k\}_{k\in V\setminus\{\mathrm{root}\}} ν = { ν k } k ∈ V ∖ { root } be reference vectors with minimum norm ρ min ν : = min k ≠ r o o t ∥ ν k ∥ > 0 \rho_{\min}^{\nu}:=\min_{k\ne\mathrm{root}}\|\nu_k\|>0 ρ m i n ν := min k = root ∥ ν k ∥ > 0 and, for each non-root node k k k , let a ~ k = c k ( ν k + e k ) \tilde{a}_k=c_k(\nu_k+e_k) a ~ k = c k ( ν k + e k ) for some scaling c k > 0 c_k>0 c k > 0 and perturbation e k ∈ R d e_k\in\mathbb{R}^{d} e k ∈ R d . Let η ≥ 0 \eta\ge0 η ≥ 0 satisfy
max k ≠ r o o t ∥ e k ∥ ≤ η < ρ min ν , (42) \max_{k\ne\mathrm{root}}\|e_k\|\le\eta<\rho_{\min}^{\nu}, \tag*{(42)} k = root max ∥ e k ∥ ≤ η < ρ m i n ν , (42) and set τ : = 8 η / ρ min ν \tau:=8\eta/\rho_{\min}^{\nu} τ := 8 η / ρ m i n ν . Then:
(i) the family a ~ : = { a ~ k } k ≠ r o o t \tilde{a}:=\{\tilde{a}_k\}_{k\ne\mathrm{root}} a ~ := { a ~ k } k = root satisfies a ~ k ≠ 0 \tilde{a}_k\ne0 a ~ k = 0 for every non-root node k k k , so the cosines in part (ii) and the violation percentage in part (iii) are well defined;
(ii) every admissible triplet ( u , v , w ) (u,v,w) ( u , v , w ) with γ u v w ν > τ \gamma_{uvw}^{\nu}>\tau γ uv w ν > τ satisfies cos ( a ~ u , a ~ v ) > cos ( a ~ u , a ~ w ) \cos(\tilde{a}_u,\tilde{a}_v)>\cos(\tilde{a}_u,\tilde{a}_w) cos ( a ~ u , a ~ v ) > cos ( a ~ u , a ~ w ) ;
(iii) H A S - Viol a ~ ( u ) ≤ H A S - Viol τ ν ( u ) \mathcal{H}AS\text{-}\operatorname{Viol}^{\tilde{a}}(u)\le\mathcal{H}AS\text{-}\operatorname{Viol}_{\tau}^{\nu}(u) H A S - Viol a ~ ( u ) ≤ H A S - Viol τ ν ( u ) for every non-root non-leaf node u u u .
Proof. (i) By Eq. (42) and the reverse triangle inequality, ∥ ν k + e k ∥ ≥ ρ min ν − η > 0 \|\nu_k+e_k\|\ge\rho_{\min}^{\nu}-\eta>0 ∥ ν k + e k ∥ ≥ ρ m i n ν − η > 0 for every non-root node k k k ; since c k > 0 c_k>0 c k > 0 , this gives a ~ k ≠ 0 \tilde{a}_k\ne0 a ~ k = 0 . The ν k \nu_k ν k are nonzero by hypothesis, so every cosine in parts (ii) and (iii) is taken between nonzero vectors.
(ii) Fix an admissible triplet ( u , v , w ) (u,v,w) ( u , v , w ) with γ u v w ν > τ \gamma_{uvw}^{\nu}>\tau γ uv w ν > τ . Positive rescaling leaves cosines unchanged, so
cos ( a ~ u , a ~ v ) = cos ( ν u + e u , ν v + e v ) , (43) \cos(\tilde{a}_u,\tilde{a}_v)=\cos(\nu_u+e_u,\nu_v+e_v), \tag*{(43)} cos ( a ~ u , a ~ v ) = cos ( ν u + e u , ν v + e v ) , (43)
and similarly with v v v replaced by w w w . By Eq. (42) , we can apply Lemma C.5 to the pairs ( u , v ) (u,v) ( u , v ) and ( u , w ) (u,w) ( u , w ) with δ = η \delta=\eta δ = η and pairwise minimum norm at least ρ min ν \rho_{\min}^{\nu} ρ m i n ν , so each of the two cosines differs from its reference counterpart by at most 4 η / ρ min ν 4\eta/\rho_{\min}^{\nu} 4 η / ρ m i n ν . Therefore
cos ( a ~ u , a ~ v ) − cos ( a ~ u , a ~ w ) ≥ cos ( ν u , ν v ) − cos ( ν u , ν w ) − 8 η ρ min ν = γ u v w ν − τ > 0. (44) \cos(\tilde{a}_u,\tilde{a}_v)-\cos(\tilde{a}_u,\tilde{a}_w)\ge\cos(\nu_u,\nu_v)-\cos(\nu_u,\nu_w)-\frac{8\eta}{\rho_{\min}^{\nu}}=\gamma_{uvw}^{\nu}-\tau>0. \tag*{(44)} cos ( a ~ u , a ~ v ) − cos ( a ~ u , a ~ w ) ≥ cos ( ν u , ν v ) − cos ( ν u , ν w ) − ρ m i n ν 8 η = γ uv w ν − τ > 0. (44)
(iii) Fix a non-root non-leaf node u u u and suppose w w w is counted in H A S - Viol a ~ ( u ) \mathcal{H}AS\text{-}\operatorname{Viol}^{\tilde{a}}(u) H A S - Viol a ~ ( u ) , i.e., cos ( a ~ u , a ~ w ) ≥ s u a ~ \cos(\tilde{a}_u,\tilde{a}_w)\ge s_u^{\tilde{a}} cos ( a ~ u , a ~ w ) ≥ s u a ~ . Let v ⋆ ∈ Desc ( u ) v^\star\in\operatorname{Desc}(u) v ⋆ ∈ Desc ( u ) attain the minimum defining s u a ~ s_u^{\tilde{a}} s u a ~ . Then the H A S \mathcal{H}AS H A S inequality fails for the admissible triplet ( u , v ⋆ , w ) (u,v^\star,w) ( u , v ⋆ , w ) , so the contrapositive of part (ii) gives γ u v ⋆ w ν ≤ τ \gamma_{uv^\star w}^{\nu}\le\tau γ u v ⋆ w ν ≤ τ . Therefore
s u ν − cos ( ν u , ν w ) = min v ∈ Desc ( u ) γ u v w ν ≤ γ u v ⋆ w ν ≤ τ ⟹ cos ( ν u , ν w ) ≥ s u ν − τ . (45) s_u^{\nu}-\cos(\nu_u,\nu_w)=\min_{v\in\operatorname{Desc}(u)}\gamma_{uvw}^{\nu}\le\gamma_{uv^\star w}^{\nu}\le\tau\quad\Longrightarrow\quad\cos(\nu_u,\nu_w)\ge s_u^{\nu}-\tau. \tag*{(45)} s u ν − cos ( ν u , ν w ) = v ∈ Desc ( u ) min γ uv w ν ≤ γ u v ⋆ w ν ≤ τ ⟹ cos ( ν u , ν w ) ≥ s u ν − τ . (45)
Thus, the same w w w is counted in H A S - Viol τ ν ( u ) \mathcal{H}AS\text{-}\operatorname{Viol}_{\tau}^{\nu}(u) H A S - Viol τ ν ( u ) . Both percentages share the denominator ∣ O u ∣ |O_u| ∣ O u ∣ , so the inclusion of the counted sets yields the claim. □ \square □
Applied with ν = μ \nu= \mu ν = μ , part (iii) is the transfer inequality of Eq. (8). In Appendix § 12 , we derive the perturbation bound η \eta η for each loss.
Caption bias and sampling
The optimizer forms for both losses involve the same difference of empirical means. In Lemma C.10, we separate that difference into prototype, caption-bias, and variation components, and in Corollary C.11, we give the conditions under which the caption-bias component vanishes.
Definition C.9 (Empirical caption prototype and caption bias). Fix prototypes { μ k } k ∈ V ⊂ R d \{\mu_k\}_{k\in V} \subset\mathbb{R}^d { μ k } k ∈ V ⊂ R d , leaf labels y j ∈ V l e a f y_j \in V_{\mathrm{leaf}} y j ∈ V leaf , and caption labels c j ∈ V c_j \in V c j ∈ V for every j ∈ [ N ] j \in[N] j ∈ [ N ] , with N k > 0 N_k > 0 N k > 0 for every non-root node k k k . The captions need not be path-consistent. Recall from § 4.1 that P k : = { j ∈ [ N ] : c j = k } P_k := \{j \in[N] : c_j = k\} P k := { j ∈ [ N ] : c j = k } is the positive set of node k k k and N k : = ∣ P k ∣ N_k := |P_k| N k := ∣ P k ∣ . The empirical caption prototype of node k k k and the global prototype mean are
μ ˉ k : = 1 N k ∑ j ∈ P k μ y j , μ ˉ : = 1 N ∑ j = 1 N μ y j , (46) \bar{\mu}_k := \frac{1}{N_k}\sum_{j\in P_k}\mu_{y_j}, \qquad\bar{\mu} := \frac{1}{N}\sum_{j=1}^{N}\mu_{y_j}, \tag*{(46)} μ ˉ k := N k 1 j ∈ P k ∑ μ y j , μ ˉ := N 1 j = 1 ∑ N μ y j , (46) with associated caption biases b k : = μ ˉ k − μ k b_k := \bar{\mu}_k - \mu_k b k := μ ˉ k − μ k and b : = μ ˉ − μ r o o t b := \bar{\mu} - \mu_{\mathrm{root}} b := μ ˉ − μ root .
Lemma C.10 (Empirical-mean decomposition). Fix prototypes, leaf labels, and caption labels as in Definition C.9, and let ξ k ∈ R \xi_k \in\mathbb{R} ξ k ∈ R for every non-root node k k k . Let z j = μ y j + ϵ j z_j = \mu_{y_j} + \epsilon_j z j = μ y j + ϵ j with arbitrary ϵ j ∈ R d \epsilon_j \in\mathbb{R}^d ϵ j ∈ R d for every j ∈ [ N ] j \in[N] j ∈ [ N ] , and define ϵ ˉ k : = N k − 1 ∑ j ∈ P k ϵ j \bar{\epsilon}_k := N_k^{-1}\sum_{j\in P_k}\epsilon_j ϵ ˉ k := N k − 1 ∑ j ∈ P k ϵ j for every non-root node k k k and ϵ ˉ : = N − 1 ∑ j = 1 N ϵ j \bar{\epsilon} := N^{-1}\sum_{j=1}^{N}\epsilon_j ϵ ˉ := N − 1 ∑ j = 1 N ϵ j . Then, for every non-root node k k k ,
z ˉ k = μ k + b k + ϵ ˉ k , z ˉ = μ r o o t + b + ϵ ˉ , (47) \bar{z}_k = \mu_k + b_k + \bar{\epsilon}_k, \qquad\bar{z} = \mu_{\mathrm{root}} + b + \bar{\epsilon}, \tag*{(47)} z ˉ k = μ k + b k + ϵ ˉ k , z ˉ = μ root + b + ϵ ˉ , (47) and, for every non-root node k k k ,
z ˉ k − ξ k z ˉ = μ k − ξ k μ r o o t + b k − ξ k b + ϵ ˉ k − ξ k ϵ ˉ . (48) \bar{z}_k - \xi_k\bar{z} = \mu_k - \xi_k\mu_{\mathrm{root}} + b_k - \xi_k b
+ \bar{\epsilon}_k - \xi_k\bar{\epsilon}. \tag*{(48)} z ˉ k − ξ k z ˉ = μ k − ξ k μ root + b k − ξ k b + ϵ ˉ k − ξ k ϵ ˉ . (48) Proof. Averaging z j = μ y j + ϵ j z_j = \mu_{y_j} + \epsilon_j z j = μ y j + ϵ j over P k P_k P k and over [ N ] [N] [ N ] gives Eq. (47) by Definition C.9. Subtracting ξ k z ˉ \xi_k\bar{z} ξ k z ˉ gives Eq. (48) .
Lemma C.10 is an identity, so it holds for any observed captions; Appendix § 12 uses it in that generality and adds the caption bias to the error term. The main theorems instead work in the symmetric setting of § 4.1 , where the bias vanishes.
Corollary C.11 (Zero caption bias under symmetric captions). Fix prototypes, leaf labels, and caption labels as in Definition C.9, suppose the captions are path-consistent, and suppose Assumption 4.3 (leaf-uniform caption coverage) and Assumption 4.4 (prototype consistency) hold. Then μ ˉ k = μ k \bar{\mu}_k = \mu_k μ ˉ k = μ k , and hence b k = 0 b_k = 0 b k = 0 , for every non-root node k k k . If exactly n ∈ N > 0 n \in\mathbb{N}_{>0} n ∈ N > 0 images are sampled from each leaf, then also μ ˉ = μ r o o t \bar{\mu} = \mu_{\mathrm{root}} μ ˉ = μ root and b = 0 b = 0 b = 0 .
Proof. Path-consistency gives y j ∈ Sub ( k ) y_j \in\operatorname{Sub}(k) y j ∈ Sub ( k ) for every j ∈ P k j \in P_k j ∈ P k , and by Assumption 4.3 each such leaf contributes equally many images. Hence μ ˉ k \bar{\mu}_k μ ˉ k is the uniform average of the leaf prototypes in Sub ( k ) \operatorname{Sub}(k) Sub ( k ) , which equals μ k \mu_k μ k by Assumption 4.4, so b k = 0 b_k = 0 b k = 0 . With exactly n n n images per leaf, μ ˉ \bar{\mu} μ ˉ is the uniform average of all leaf prototypes, which equals μ r o o t \mu_{\mathrm{root}} μ root by the same assumption at the root, so b = 0 b = 0 b = 0 .
Concentration
Lemma C.12 (Concentration of finitely many empirical variation means). Let K \mathcal{K} K be a finite index set. Let { ϵ j } j = 1 N \{\epsilon_j\}_{j=1}^{N} { ϵ j } j = 1 N be independent centered random vectors in R d \mathbb{R}^d R d . Assume there exists σ > 0 \sigma> 0 σ > 0 such that, for every j ∈ [ N ] j \in[N] j ∈ [ N ] and every unit vector u ∈ S d − 1 u \in\mathbb{S}^{d-1} u ∈ S d − 1 , the scalar u T ϵ j u^\mathsf{T}\epsilon_j u T ϵ j is sub-Gaussian with Orlicz norm at most σ \sigma σ . For each k ∈ K k \in\mathcal{K} k ∈ K , let I k ⊆ [ N ] I_k \subseteq[N] I k ⊆ [ N ] with s k : = ∣ I k ∣ > 0 s_k := |I_k| > 0 s k := ∣ I k ∣ > 0 , and define
ϵ ˉ k : = 1 s k ∑ j ∈ I k ϵ j , ϵ ˉ : = 1 N ∑ j = 1 N ϵ j (49) \bar{\epsilon}_k := \frac{1}{s_k}\sum_{j\in I_k}\epsilon_j, \qquad\bar{\epsilon} := \frac{1}{N}\sum_{j=1}^{N}\epsilon_j \tag*{(49)} ϵ ˉ k := s k 1 j ∈ I k ∑ ϵ j , ϵ ˉ := N 1 j = 1 ∑ N ϵ j (49)
For each k ∈ K k \in\mathcal{K} k ∈ K , let ξ k ∈ R \xi_k \in\mathbb{R} ξ k ∈ R , and define
ξ max : = max k ∈ K ∣ ξ k ∣ , s min : = min k ∈ K s k . (50) \xi_{\max} := \max_{k\in\mathcal{K}} |\xi_k|,\qquad s_{\min} := \min_{k\in\mathcal{K}} s_k. \tag*{(50)} ξ m a x := k ∈ K max ∣ ξ k ∣ , s m i n := k ∈ K min s k . (50)
Then there exists a universal constant C > 0 C > 0 C > 0 such that, for every u ≥ 0 u \ge0 u ≥ 0 , with probability at least 1 − 4 e − u 1-4e^{-u} 1 − 4 e − u ,
max k ∈ K ∥ ϵ ˉ k − ξ k ϵ ˉ ∥ ≤ C σ ( d + log ∣ K ∣ + u s min + ξ max d + u N ) . (51) \max_{k\in\mathcal{K}}\|\bar{\epsilon}_k-\xi_k\bar{\epsilon}\| \le C\sigma\left(\sqrt{\frac{d+\log|\mathcal{K}|+u}{s_{\min}}}+\xi_{\max}\sqrt{\frac{d+u}{N}}\right). \tag*{(51)} k ∈ K max ∥ ϵ ˉ k − ξ k ϵ ˉ ∥ ≤ C σ s m i n d + log ∣ K ∣ + u + ξ m a x N d + u . (51)
Proof. For each fixed k ∈ K k \in\mathcal{K} k ∈ K , apply the scalar Hoeffding-type tail bound for averages of independent centered sub-Gaussian random variables to u ⊤ ϵ ˉ k u^\top\bar{\epsilon}_k u ⊤ ϵ ˉ k for u ∈ S d − 1 u \in\mathbb{S}^{d-1} u ∈ S d − 1 . A standard 1 / 2 1/2 1/2 -net argument on S d − 1 \mathbb{S}^{d-1} S d − 1 then gives a universal constant C > 0 C > 0 C > 0 such that, for every t ≥ 0 t \ge0 t ≥ 0 ,
Pr ( ∥ ϵ ˉ k ∥ > C σ d + t s k ) ≤ 2 e − t . (52) \Pr\left(\|\bar{\epsilon}_k\| > C\sigma\sqrt{\frac{d+t}{s_k}}\right) \le2e^{-t}. \tag*{(52)} Pr ( ∥ ϵ ˉ k ∥ > C σ s k d + t ) ≤ 2 e − t . (52)
The same argument with I k = [ N ] I_k=[N] I k = [ N ] and s k = N s_k=N s k = N gives
Pr ( ∥ ϵ ˉ ∥ > C σ d + t N ) ≤ 2 e − t . (53) \Pr\left(\|\bar{\epsilon}\| > C\sigma\sqrt{\frac{d+t}{N}}\right) \le2e^{-t}. \tag*{(53)} Pr ( ∥ ϵ ˉ ∥ > C σ N d + t ) ≤ 2 e − t . (53)
Apply the first bound with t = u + log ∣ K ∣ t=u+\log|\mathcal{K}| t = u + log ∣ K ∣ and take a union bound over k ∈ K k \in\mathcal{K} k ∈ K . Then, with probability at least 1 − 2 e − u 1-2e^{-u} 1 − 2 e − u ,
max k ∈ K ∥ ϵ ˉ k ∥ ≤ C σ d + log ∣ K ∣ + u s min . (54) \max_{k\in\mathcal{K}}\|\bar{\epsilon}_k\| \le C\sigma\sqrt{\frac{d+\log|\mathcal{K}|+u}{s_{\min}}}. \tag*{(54)} k ∈ K max ∥ ϵ ˉ k ∥ ≤ C σ s m i n d + log ∣ K ∣ + u . (54)
The second bound with t = u t=u t = u gives, with probability at least 1 − 2 e − u 1-2e^{-u} 1 − 2 e − u ,
∥ ϵ ˉ ∥ ≤ C σ d + u N . (55) \|\bar{\epsilon}\| \le C\sigma\sqrt{\frac{d+u}{N}}. \tag*{(55)} ∥ ϵ ˉ ∥ ≤ C σ N d + u . (55)
Another union bound shows that both displayed bounds hold simultaneously with probability at least 1 − 4 e − u 1-4e^{-u} 1 − 4 e − u . On that event,
max k ∈ K ∥ ϵ ˉ k − ξ k ϵ ˉ ∥ ≤ max k ∈ K ∥ ϵ ˉ k ∥ + ξ max ∥ ϵ ˉ ∥ , (56) \max_{k\in\mathcal{K}}\|\bar{\epsilon}_k-\xi_k\bar{\epsilon}\| \le\max_{k\in\mathcal{K}}\|\bar{\epsilon}_k\|+\xi_{\max}\|\bar{\epsilon}\|, \tag*{(56)} k ∈ K max ∥ ϵ ˉ k − ξ k ϵ ˉ ∥ ≤ k ∈ K max ∥ ϵ ˉ k ∥ + ξ m a x ∥ ϵ ˉ ∥ , (56)
where we used the triangle inequality and the bound ∣ ξ k ∣ ≤ ξ max |\xi_k| \le\xi_{\max} ∣ ξ k ∣ ≤ ξ m a x for every k ∈ K k \in\mathcal{K} k ∈ K . Substituting Eqs. (54) and (55) into this estimate gives Eq. (51). □ \square □
Lemma C.13 (Maximum feature norm under sub-Gaussian variation). Suppose Assumption 4.1 holds, and define μ l e a f , max : = max ℓ ∈ V l e a f ∥ μ ℓ ∥ \mu_{\mathrm{leaf},\max}:=\max_{\ell\in V_{\mathrm{leaf}}}\|\mu_\ell\| μ leaf , m a x := max ℓ ∈ V leaf ∥ μ ℓ ∥ . Then there exists a universal constant C > 0 C > 0 C > 0 such that, for every ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) , with probability at least 1 − ϵ 1-\epsilon 1 − ϵ ,
max j ∈ [ N ] ∥ z j ∥ ≤ μ l e a f , max + C σ d + log ( 2 N / ϵ ) . (57) \max_{j\in[N]}\|z_j\| \le\mu_{\mathrm{leaf},\max}+C\sigma\sqrt{d+\log(2N/\epsilon)}. \tag*{(57)} j ∈ [ N ] max ∥ z j ∥ ≤ μ leaf , m a x + C σ d + log ( 2 N / ϵ ) . (57)
Proof. For each fixed j ∈ [ N ] j \in[N] j ∈ [ N ] , the same Hoeffding bound used in the proof of Lemma C.12 gives
Pr ( ∥ ϵ j ∥ > C σ d + u ) ≤ 2 e − u \Pr\left(\|\epsilon_j\| > C\sigma\sqrt{d+u}\right) \le2e^{-u} Pr ( ∥ ϵ j ∥ > C σ d + u ) ≤ 2 e − u
for a universal constant C > 0 C > 0 C > 0 . Taking u = log ( 2 N / ϵ ) u=\log(2N/\epsilon) u = log ( 2 N / ϵ ) and union bounding over j ∈ [ N ] j \in[N] j ∈ [ N ] gives max j ∈ [ N ] ∥ ϵ j ∥ ≤ C σ d + log ( 2 N / ϵ ) \max_{j\in[N]}\|\epsilon_j\| \le C\sigma\sqrt{d+\log(2N/\epsilon)} max j ∈ [ N ] ∥ ϵ j ∥ ≤ C σ d + log ( 2 N / ϵ ) with probability at least 1 − ϵ 1-\epsilon 1 − ϵ . Absorbing the factor 2 into the universal constant and using ∥ z j ∥ ≤ ∥ μ y j ∥ + ∥ ϵ j ∥ ≤ μ l e a f , max + ∥ ϵ j ∥ \|z_j\| \le\|\mu_{y_j}\|+\|\epsilon_j\| \le\mu_{\mathrm{leaf},\max}+\|\epsilon_j\| ∥ z j ∥ ≤ ∥ μ y j ∥ + ∥ ϵ j ∥ ≤ μ leaf , m a x + ∥ ϵ j ∥ proves the claim. □ \square □
Proofs for Intermediate Results in Linear Contrastive Loss Analysis
Closed-Form Minimizers for the Linear Contrastive Loss
The same label encoding governs the image-side update in Appendix K, so we state both minimizers together.
Proposition D.1 (Closed-Form Minimizers for the Linear Contrastive Loss with One Side Fixed). Define the assignment matrix Y ∈ { 0 , 1 } M × N Y \in\{0,1\}^{M \times N} Y ∈ { 0 , 1 } M × N via Y k j : = 1 [ c j = k ] Y_{kj} := \mathbb{1}[c_j = k] Y k j := 1 [ c j = k ] . Then define the label encoding matrix B ∈ R M × N B \in\mathbb{R}^{M \times N} B ∈ R M × N by
B k j : = 1 2 ( Y k j − 1 − Y k j M − 1 ) + 1 2 ( Y k j − N k − Y k j N − 1 ) . (58) B_{kj} := \frac{1}{2}\left(Y_{kj} - \frac{1-Y_{kj}}{M-1}\right) + \frac{1}{2}\left(Y_{kj} - \frac{N_k-Y_{kj}}{N-1}\right). \tag*{(58)} B k j := 2 1 ( Y k j − M − 1 1 − Y k j ) + 2 1 ( Y k j − N − 1 N k − Y k j ) . (58) Then the unique minimizer of Eq. (5) is
A l i n , ∗ = 1 λ Z B T . (59) A^{\mathrm{lin},*} = \frac{1}{\lambda} ZB^{\mathsf{T}}. \tag*{(59)} A lin , ∗ = λ 1 Z B T . (59) For any fixed A ∈ R d × M A \in\mathbb{R}^{d \times M} A ∈ R d × M and λ Z > 0 \lambda_Z > 0 λ Z > 0 , the image-side quadratic − tr ( A T Z B T ) + λ Z 2 ∥ Z ∥ F 2 -\operatorname{tr}(A^{\mathsf{T}}ZB^{\mathsf{T}}) + \frac{\lambda_Z}{2}\lVert Z\rVert_F^2 − tr ( A T Z B T ) + 2 λ Z ∥ Z ∥ F 2 has unique minimizer λ Z − 1 A B \lambda_Z^{-1}AB λ Z − 1 A B over Z ∈ R d × N Z \in\mathbb{R}^{d \times N} Z ∈ R d × N .
We first rewrite Eq. (5) as
L l i n ( A ) = − tr ( A T Z B T ) + λ 2 ∥ A ∥ F 2 , (60) \mathcal{L}^{\mathrm{lin}}(A) = -\operatorname{tr}(A^{\mathsf{T}}ZB^{\mathsf{T}}) + \frac{\lambda}{2}\lVert A\rVert_F^2, \tag*{(60)} L lin ( A ) = − tr ( A T Z B T ) + 2 λ ∥ A ∥ F 2 , (60) with B B B as in Eq. (58) .
Proof. Write the two terms in Eq. (5) as L i 2 t l i n ( A ) \mathcal{L}^{\mathrm{lin}}_{i2t}(A) L i 2 t lin ( A ) and L t 2 i l i n ( A ) \mathcal{L}^{\mathrm{lin}}_{t2i}(A) L t 2 i lin ( A ) . Since Y k j = 1 [ c j = k ] Y_{kj} = \mathbb{1}[c_j = k] Y k j = 1 [ c j = k ] , the image-to-text term satisfies
L i 2 t l i n ( A ) = − 1 2 ∑ j = 1 N a c j T z j + 1 2 ( M − 1 ) ∑ j = 1 N ∑ k ∈ [ M ] k ≠ c j a k T z j = − ∑ k = 1 M ∑ j = 1 N 1 2 ( Y k j − 1 − Y k j M − 1 ) a k T z j . (61) \begin{aligned}
\mathcal{L}^{\mathrm{lin}}_{i2t}(A)
&= -\frac{1}{2}\sum_{j=1}^{N} a_{c_j}^{\mathsf{T}}z_j + \frac{1}{2(M-1)}\sum_{j=1}^{N}\sum_{\substack{k \in[M]\\k\ne c_j}} a_k^{\mathsf{T}}z_j \\
&= -\sum_{k=1}^{M}\sum_{j=1}^{N}\frac{1}{2}\left(Y_{kj}-\frac{1-Y_{kj}}{M-1}\right)a_k^{\mathsf{T}}z_j. \tag*{(61)}
\end{aligned} L i 2 t lin ( A ) = − 2 1 j = 1 ∑ N a c j T z j + 2 ( M − 1 ) 1 j = 1 ∑ N k ∈ [ M ] k = c j ∑ a k T z j = − k = 1 ∑ M j = 1 ∑ N 2 1 ( Y k j − M − 1 1 − Y k j ) a k T z j . (61) For the text-to-image term, fix k ∈ [ M ] k \in[M] k ∈ [ M ] and j ∈ [ N ] j \in[N] j ∈ [ N ] . The logit a k T z j a_k^{\mathsf{T}}z_j a k T z j appears once in the positive part when c j = k c_j = k c j = k , and it appears in the negative pool once for each j ′ ∈ [ N ] ∖ { j } j' \in[N] \setminus\{j\} j ′ ∈ [ N ] ∖ { j } with c j ′ = k c_{j'} = k c j ′ = k . Therefore the coefficient of a k T z j a_k^{\mathsf{T}}z_j a k T z j in L t 2 i l i n ( A ) \mathcal{L}^{\mathrm{lin}}_{t2i}(A) L t 2 i lin ( A ) is
− 1 2 Y k j + 1 2 ( N − 1 ) ∑ j ′ ∈ [ N ] j ′ ≠ j Y k j ′ = − 1 2 ( Y k j − N k − Y k j N − 1 ) , (62) -\frac{1}{2}Y_{kj} + \frac{1}{2(N-1)}\sum_{\substack{j' \in[N]\\j' \ne j}}Y_{kj'} = -\frac{1}{2}\left(Y_{kj}-\frac{N_k-Y_{kj}}{N-1}\right), \tag*{(62)} − 2 1 Y k j + 2 ( N − 1 ) 1 j ′ ∈ [ N ] j ′ = j ∑ Y k j ′ = − 2 1 ( Y k j − N − 1 N k − Y k j ) , (62) because ∑ j ′ ∈ [ N ] j ′ ≠ j Y k j ′ = N k − Y k j \sum_{\substack{j' \in[N]\\j' \ne j}}Y_{kj'} = N_k-Y_{kj} ∑ j ′ ∈ [ N ] j ′ = j Y k j ′ = N k − Y k j by the definition of N k N_k N k . Hence
L t 2 i l i n ( A ) = − ∑ k = 1 M ∑ j = 1 N 1 2 ( Y k j − N k − Y k j N − 1 ) a k T z j . (63) \mathcal{L}^{\mathrm{lin}}_{t2i}(A) = -\sum_{k=1}^{M}\sum_{j=1}^{N}\frac{1}{2}\left(Y_{kj}-\frac{N_k-Y_{kj}}{N-1}\right)a_k^{\mathsf{T}}z_j. \tag*{(63)} L t 2 i lin ( A ) = − k = 1 ∑ M j = 1 ∑ N 2 1 ( Y k j − N − 1 N k − Y k j ) a k T z j . (63) Adding the two terms and using Eq. (58) gives
L l i n ( A ) = − ∑ k = 1 M ∑ j = 1 N B k j a k T z j + λ 2 ∥ A ∥ F 2 . (64) \mathcal{L}^{\mathrm{lin}}(A) = -\sum_{k=1}^{M}\sum_{j=1}^{N}B_{kj}a_k^{\mathsf{T}}z_j + \frac{\lambda}{2}\lVert A\rVert_F^2. \tag*{(64)} L lin ( A ) = − k = 1 ∑ M j = 1 ∑ N B k j a k T z j + 2 λ ∥ A ∥ F 2 . (64) Finally,
tr ( A T Z B T ) = ∑ k = 1 M a k T ( ∑ j = 1 N B k j z j ) = ∑ k = 1 M ∑ j = 1 N B k j a k T z j , (65) \operatorname{tr}(A^{\mathsf{T}}ZB^{\mathsf{T}}) = \sum_{k=1}^{M}a_k^{\mathsf{T}}\left(\sum_{j=1}^{N}B_{kj}z_j\right) = \sum_{k=1}^{M}\sum_{j=1}^{N}B_{kj}a_k^{\mathsf{T}}z_j, \tag*{(65)} tr ( A T Z B T ) = k = 1 ∑ M a k T ( j = 1 ∑ N B k j z j ) = k = 1 ∑ M j = 1 ∑ N B k j a k T z j , (65) so the previous display is exactly Eq. (60) . Differentiating with respect to A A A yields
∇ A L l i n ( A ) = λ A − Z B T . (66) \nabla_A\mathcal{L}^{\mathrm{lin}}(A) = \lambda A - ZB^{\mathsf{T}}. \tag*{(66)} ∇ A L lin ( A ) = λ A − Z B T . (66) Setting the gradient to zero gives Eq. (59) , and the quadratic regularizer makes the minimizer unique. For the image-side quadratic, ∇ Z = λ Z Z − A B \nabla_Z = \lambda_Z Z - AB ∇ Z = λ Z Z − A B ; setting this gradient to zero gives the stated image-side minimizer, unique because λ Z > 0 \lambda_Z > 0 λ Z > 0 . □ \square □
Proof of Corollary D.2 (Empirical-Mean Expression)
Corollary D.2 (Empirical-Mean Expression for the Linear Minimizer). Every node k ∈ [ M ] k \in[M] k ∈ [ M ] with N k > 0 N_k > 0 N k > 0 satisfies
a k l i n , ∗ = N k λ ( 1 + 1 2 ( M − 1 ) + 1 2 ( N − 1 ) ) z ˉ k − N λ ( 1 2 ( M − 1 ) + N k 2 ( N − 1 ) ) z ˉ . (67) a_k^{\mathrm{lin},*} = \frac{N_k}{\lambda}\left(1+\frac{1}{2(M-1)}+\frac{1}{2(N-1)}\right)\bar{z}_k-\frac{N}{\lambda}\left(\frac{1}{2(M-1)}+\frac{N_k}{2(N-1)}\right)\bar{z}. \tag*{(67)} a k lin , ∗ = λ N k ( 1 + 2 ( M − 1 ) 1 + 2 ( N − 1 ) 1 ) z ˉ k − λ N ( 2 ( M − 1 ) 1 + 2 ( N − 1 ) N k ) z ˉ . (67)
Proof. Fix a node k ∈ [ M ] k \in[M] k ∈ [ M ] with N k > 0 N_k > 0 N k > 0 , and write
P k : = { j ∈ [ N ] : c j = k } . (68) \mathcal{P}_k := \{j \in[N] : c_j = k\}. \tag*{(68)} P k := { j ∈ [ N ] : c j = k } . (68)
By Eq. (58) ,
B k j = { γ k + , j ∈ P k , − γ k − , j ∉ P k , (69) B_{kj} =
\begin{cases}
\gamma_k^{+}, & j \in\mathcal{P}_k,\\
-\gamma_k^{-}, & j \notin\mathcal{P}_k,
\end{cases} \tag*{(69)} B k j = { γ k + , − γ k − , j ∈ P k , j ∈ / P k , (69)
where
γ k + : = 1 − N k − 1 2 ( N − 1 ) , γ k − : = 1 2 ( M − 1 ) + N k 2 ( N − 1 ) . (70) \gamma_k^{+} := 1-\frac{N_k-1}{2(N-1)}, \qquad\gamma_k^{-} := \frac{1}{2(M-1)}+\frac{N_k}{2(N-1)}. \tag*{(70)} γ k + := 1 − 2 ( N − 1 ) N k − 1 , γ k − := 2 ( M − 1 ) 1 + 2 ( N − 1 ) N k . (70)
Applying Proposition D.1,
a k l i n , ∗ = 1 λ ∑ j = 1 N B k j z j = γ k + λ ∑ j ∈ P k z j − γ k − λ ∑ j ∉ P k z j . (71) \begin{aligned}
a_k^{\mathrm{lin},*}=\frac{1}{\lambda}\sum_{j=1}^{N}B_{kj}z_j \\
&=\frac{\gamma_k^{+}}{\lambda}\sum_{j\in\mathcal{P}_k}z_j-\frac{\gamma_k^{-}}{\lambda}\sum_{j\notin\mathcal{P}_k}z_j. \tag*{(71)}
\end{aligned} a k lin , ∗ = λ 1 j = 1 ∑ N B k j z j = λ γ k + j ∈ P k ∑ z j − λ γ k − j ∈ / P k ∑ z j . (71)
By definition,
∑ j ∉ P k z j = ∑ j = 1 N z j − ∑ j ∈ P k z j , (72) \sum_{j\notin\mathcal{P}_k}z_j=\sum_{j=1}^{N}z_j-\sum_{j\in\mathcal{P}_k}z_j, \tag*{(72)} j ∈ / P k ∑ z j = j = 1 ∑ N z j − j ∈ P k ∑ z j , (72)
so
a k l i n , ∗ = γ k + + γ k − λ ∑ j ∈ P k z j − γ k − λ ∑ j = 1 N z j . (73) a_k^{\mathrm{lin},*}=\frac{\gamma_k^{+}+\gamma_k^{-}}{\lambda}\sum_{j\in\mathcal{P}_k}z_j-\frac{\gamma_k^{-}}{\lambda}\sum_{j=1}^{N}z_j. \tag*{(73)} a k lin , ∗ = λ γ k + + γ k − j ∈ P k ∑ z j − λ γ k − j = 1 ∑ N z j . (73)
By definition of z ˉ k \bar{z}_k z ˉ k ,
∑ j ∈ P k z j = N k z ˉ k . (74) \sum_{j\in\mathcal{P}_k}z_j=N_k\bar{z}_k. \tag*{(74)} j ∈ P k ∑ z j = N k z ˉ k . (74)
∑ j = 1 N z j = N z ˉ . (75) \sum_{j=1}^{N}z_j=N\bar{z}. \tag*{(75)} j = 1 ∑ N z j = N z ˉ . (75)
Substituting these identities and using
γ k + + γ k − = 1 + 1 2 ( M − 1 ) + 1 2 ( N − 1 ) (76) \gamma_k^{+}+\gamma_k^{-}=1+\frac{1}{2(M-1)}+\frac{1}{2(N-1)} \tag*{(76)} γ k + + γ k − = 1 + 2 ( M − 1 ) 1 + 2 ( N − 1 ) 1 (76)
yields
a k l i n , ∗ = N k λ ( 1 + 1 2 ( M − 1 ) + 1 2 ( N − 1 ) ) z ˉ k − N λ ( 1 2 ( M − 1 ) + N k 2 ( N − 1 ) ) z ˉ , (77) a_k^{\mathrm{lin},*}=\frac{N_k}{\lambda}\left(1+\frac{1}{2(M-1)}+\frac{1}{2(N-1)}\right)\bar{z}_k-\frac{N}{\lambda}\left(\frac{1}{2(M-1)}+\frac{N_k}{2(N-1)}\right)\bar{z}, \tag*{(77)} a k lin , ∗ = λ N k ( 1 + 2 ( M − 1 ) 1 + 2 ( N − 1 ) 1 ) z ˉ k − λ N ( 2 ( M − 1 ) 1 + 2 ( N − 1 ) N k ) z ˉ , (77)
which is Eq. (67) .
Proof of Corollary 4.5
Corollary 4.5 (Prototype Decomposition for the Linear Contrastive Loss). Suppose Assumptions 4.1, 4.3, and 4.4 hold. For each non-root k k k , define ξ k l i n : = N ( 1 2 ( M − 1 ) + N k 2 ( N − 1 ) ) N k ( 1 + 1 2 ( M − 1 ) + 1 2 ( N − 1 ) ) \xi_k^{\mathrm{lin}} := \frac{N\left(\frac{1}{2(M-1)}+\frac{N_k}{2(N-1)}\right)}{N_k\left(1+\frac{1}{2(M-1)}+\frac{1}{2(N-1)}\right)} ξ k lin := N k ( 1 + 2 ( M − 1 ) 1 + 2 ( N − 1 ) 1 ) N ( 2 ( M − 1 ) 1 + 2 ( N − 1 ) N k ) . Then every such k k k satisfies
a k l i n , ∗ = N k λ ( 1 + 1 2 ( M − 1 ) + 1 2 ( N − 1 ) ) ( μ k + ε ˉ k − ξ k l i n ( μ r o o t + ε ˉ ) ) . (78) a_k^{\mathrm{lin},*}=\frac{N_k}{\lambda}\left(1+\frac{1}{2(M-1)}+\frac{1}{2(N-1)}\right)\left(\mu_k+\bar{\varepsilon}_k-\xi_k^{\mathrm{lin}}\left(\mu_{\mathrm{root}}+\bar{\varepsilon}\right)\right). \tag*{(78)} a k lin , ∗ = λ N k ( 1 + 2 ( M − 1 ) 1 + 2 ( N − 1 ) 1 ) ( μ k + ε ˉ k − ξ k lin ( μ root + ε ˉ ) ) . (78)
Proof. The observed captions are path-consistent by the setup in § 4.1 , and the same setup has exactly n n n images per leaf. Under these hypotheses, Assumptions 4.3 and 4.4, and the decomposition in Assumption 4.1, we can apply Lemma C.10 and Corollary C.11. By Assumption 4.3, we also have N k > 0 N_k>0 N k > 0 for every non-root node k k k , so we can apply Corollary D.2. Factor its empirical-mean expression using ξ k l i n \xi_k^{\mathrm{lin}} ξ k lin from Corollary 4.5, and substitute Eq. (48) with b k = b = 0 b_k=b=0 b k = b = 0 to obtain Eq. (6).
Proofs of Intermediate Results in InfoNCE Loss Analysis
Proof of Lemma E.1 (Strong Convexity)
Lemma E.1 (Strong Convexity). The objective L ( A ) \mathcal{L}(A) L ( A ) in Eq. (9) is λ \lambda λ -strongly convex in A A A . Therefore, it has a unique global minimizer A n c e , ∗ A^{\mathrm{nce},*} A nce , ∗ .
Proof. First, each logit a k T z j a_k^{\mathsf{T}}z_j a k T z j is linear in A A A . Then, the term − ( linear in A ) + log ∑ exp ( linear in A ) -(\text{linear in }A)+\log\sum\exp(\text{linear in }A) − ( linear in A ) + log ∑ exp ( linear in A ) is convex as the composition of log-sum-exp with linear maps. Finally, composition with the sum over j ∈ [ N ] j\in[N] j ∈ [ N ] and the regularizer λ 2 ∥ A ∥ F 2 \frac{\lambda}{2}\lVert A\rVert_F^2 2 λ ∥ A ∥ F 2 yield λ \lambda λ -strong convexity.
Proof of Proposition E.2 (Stationarity Condition)
Proposition E.2 (Stationarity Condition). Define the softmax weights q k ( j ) ( A ) : = e a k T z j ∑ k ′ = 1 M e a k ′ T z j q_k^{(j)}(A):=\frac{e^{a_k^{\mathsf{T}}z_j}}{\sum_{k'=1}^{M}e^{a_{k'}^{\mathsf{T}}z_j}} q k ( j ) ( A ) := ∑ k ′ = 1 M e a k ′ T z j e a k T z j and r j ( j ) ( A ) : = e a c j T z j ∑ m = 1 N e a c j T z m r_j^{(j)}(A):=\frac{e^{a_{c_j}^{\mathsf{T}}z_j}}{\sum_{m=1}^{N}e^{a_{c_j}^{\mathsf{T}}z_m}} r j ( j ) ( A ) := ∑ m = 1 N e a c j T z m e a c j T z j . The unique minimizer A n c e , ∗ A^{\mathrm{nce},*} A nce , ∗ of Eq. (9) satisfies, for each k ∈ [ M ] k\in[M] k ∈ [ M ] ,
a k n c e , ∗ = 1 λ ( ∑ j : c j = k z j − 1 2 ∑ j = 1 N q k ( j ) ( A n c e , ∗ ) z j − 1 2 ∑ j : c j = k ∑ j ′ = 1 N r j ′ ( j ) ( A n c e , ∗ ) z j ′ ) . (79) a_k^{\mathrm{nce},*}=\frac{1}{\lambda}\left(\sum_{j:c_j=k}z_j-\frac{1}{2}\sum_{j=1}^{N}q_k^{(j)}(A^{\mathrm{nce},*})z_j-\frac{1}{2}\sum_{j:c_j=k}\sum_{j'=1}^{N}r_{j'}^{(j)}(A^{\mathrm{nce},*})z_{j'}\right). \tag*{(79)} a k nce , ∗ = λ 1 j : c j = k ∑ z j − 2 1 j = 1 ∑ N q k ( j ) ( A nce , ∗ ) z j − 2 1 j : c j = k ∑ j ′ = 1 ∑ N r j ′ ( j ) ( A nce , ∗ ) z j ′ . (79)
Proof. First, we compute the image-to-text gradients. The total i2t loss is
L i 2 t = ∑ j = 1 N L j i 2 t , (80) \mathcal{L}^{\mathrm{i2t}}=\sum_{j=1}^{N}\mathcal{L}^{\mathrm{i2t}}_j, \tag*{(80)} L i2t = j = 1 ∑ N L j i2t , (80)
where
L j i 2 t = − a c j T z j + log ∑ k ′ e a k ′ T z j . (81) \mathcal{L}^{\mathrm{i2t}}_j=-a_{c_j}^{\mathsf{T}}z_j+\log\sum_{k'}e^{a_{k'}^{\mathsf{T}}z_j}. \tag*{(81)} L j i2t = − a c j T z j + log k ′ ∑ e a k ′ T z j . (81)
Each term depends on a k a_k a k both through the numerator (when c j = k c_j=k c j = k ) and the log-sum-exp denominator:
∇ a k L j i 2 t = − 1 [ c j = k ] z j + e a k T z j ∑ k ′ e a k ′ T z j z j = ( − 1 [ c j = k ] + q k ( j ) ) z j . (82) \begin{aligned}
\nabla_{a_k}\mathcal{L}^{\mathrm{i2t}}_j=-\mathbf{1}[c_j=k]z_j+\frac{e^{a_k^{\mathsf{T}}z_j}}{\sum_{k'}e^{a_{k'}^{\mathsf{T}}z_j}}z_j \\
&=\left(-\mathbf{1}[c_j=k]+q_k^{(j)}\right)z_j. \tag*{(82)}
\end{aligned} ∇ a k L j i2t = − 1 [ c j = k ] z j + ∑ k ′ e a k ′ T z j e a k T z j z j = ( − 1 [ c j = k ] + q k ( j ) ) z j . (82)
Summing over j ∈ [ N ] j\in[N] j ∈ [ N ] gives
∇ a k L i 2 t = ∑ j = 1 N ( − 1 [ c j = k ] + q k ( j ) ) z j = − ∑ j : c j = k z j + ∑ j = 1 N q k ( j ) z j . (83) \nabla_{a_k}\mathcal{L}^{\mathrm{i2t}}=\sum_{j=1}^{N}\left(-\mathbf{1}[c_j=k]+q_k^{(j)}\right)z_j=-\sum_{j:c_j=k}z_j+\sum_{j=1}^{N}q_k^{(j)}z_j. \tag*{(83)} ∇ a k L i2t = j = 1 ∑ N ( − 1 [ c j = k ] + q k ( j ) ) z j = − j : c j = k ∑ z j + j = 1 ∑ N q k ( j ) z j . (83)
Next, we compute the text-to-image gradients. The total t2i loss is
L t 2 i = ∑ j = 1 N L j t 2 i . (84) \mathcal{L}^{\mathrm{t2i}} = \sum_{j=1}^{N} \mathcal{L}^{\mathrm{t2i}}_j . \tag*{(84)} L t2i = j = 1 ∑ N L j t2i . (84)
where
L j t 2 i = − a c j ⊤ z j + log ∑ j ′ = 1 N e a c j ′ ⊤ z j . (85) \mathcal{L}^{\mathrm{t2i}}_j = -a_{c_j}^{\top}z_j + \log\sum_{j'=1}^{N} e^{a_{c_{j'}}^{\top}z_j}. \tag*{(85)} L j t2i = − a c j ⊤ z j + log j ′ = 1 ∑ N e a c j ′ ⊤ z j . (85)
Since a k a_k a k appears in L j t 2 i \mathcal{L}^{\mathrm{t2i}}_j L j t2i only when c j = k c_j = k c j = k :
∇ a k L j t 2 i = 1 [ c j = k ] ( − z j + ∑ j ′ = 1 N e a c j ′ ⊤ z j z j ′ ∑ m = 1 N e a c j ⊤ z m ) = 1 [ c j = k ] ( − z j + ∑ j ′ = 1 N r j ′ ( j ) z j ′ ) . (86) \begin{aligned}
\nabla_{a_k}\mathcal{L}^{\mathrm{t2i}}_j
= \mathbf{1}[c_j = k]\left(-z_j + \frac{\sum_{j'=1}^{N}e^{a_{c_{j'}}^{\top}z_j}z_{j'}}{\sum_{m=1}^{N}e^{a_{c_j}^{\top}z_m}}\right) \\
&= \mathbf{1}[c_j = k]\left(-z_j + \sum_{j'=1}^{N}r_{j'}^{(j)}z_{j'}\right). \tag*{(86)}
\end{aligned} ∇ a k L j t2i = 1 [ c j = k ] − z j + ∑ m = 1 N e a c j ⊤ z m ∑ j ′ = 1 N e a c j ′ ⊤ z j z j ′ = 1 [ c j = k ] − z j + j ′ = 1 ∑ N r j ′ ( j ) z j ′ . (86)
Summing over j ∈ [ N ] j \in[N] j ∈ [ N ] gives
∇ a k L t 2 i = ∑ j : c j = k ( − z j + ∑ j ′ = 1 N r j ′ ( j ) z j ′ ) . (87) \nabla_{a_k}\mathcal{L}^{\mathrm{t2i}}
= \sum_{j:c_j=k}\left(-z_j + \sum_{j'=1}^{N}r_{j'}^{(j)}z_{j'}\right). \tag*{(87)} ∇ a k L t2i = j : c j = k ∑ − z j + j ′ = 1 ∑ N r j ′ ( j ) z j ′ . (87)
Finally, we combine Eqs. (83) and (87) and add the regularization gradient λ a k \lambda a_k λ a k . Since A n c e , ∗ A^{\mathrm{nce},*} A nce , ∗ is the unique minimizer, it satisfies
∇ a k L = 1 2 ( ∇ a k L i 2 t + ∇ a k L t 2 i ) + λ a k n c e , ∗ = 0. (88) \nabla_{a_k}\mathcal{L}
= \frac{1}{2}\left(\nabla_{a_k}\mathcal{L}^{\mathrm{i2t}}+\nabla_{a_k}\mathcal{L}^{\mathrm{t2i}}\right)+\lambda a_k^{\mathrm{nce},*}=0. \tag*{(88)} ∇ a k L = 2 1 ( ∇ a k L i2t + ∇ a k L t2i ) + λ a k nce , ∗ = 0. (88)
Substituting Eqs. (83) and (87) and rearranging yields Eq. (79) . □
Proof of Proposition 4.7 (Large-Regularization Expansion)
Proposition 4.7 (Large-Regularization Expansion). As λ → ∞ \lambda\to\infty λ → ∞ , for any k k k with N k > 0 N_k > 0 N k > 0 ,
a k n c e , ∗ = N k λ z ˉ k − 1 λ ( N 2 M + N k 2 ) z ˉ + O ( λ − 2 ) . (89) a_k^{\mathrm{nce},*} = \frac{N_k}{\lambda}\bar{z}_k - \frac{1}{\lambda}\left(\frac{N}{2M}+\frac{N_k}{2}\right)\bar{z}+O(\lambda^{-2}). \tag*{(89)} a k nce , ∗ = λ N k z ˉ k − λ 1 ( 2 M N + 2 N k ) z ˉ + O ( λ − 2 ) . (89)
Proof. Let Z max : = max j ∈ [ N ] ∥ z j ∥ Z_{\max}:=\max_{j\in[N]}\|z_j\| Z m a x := max j ∈ [ N ] ∥ z j ∥ . Since ∑ k = 1 M q k ( j ) ( A n c e , ∗ ) = 1 \sum_{k=1}^{M}q_k^{(j)}(A^{\mathrm{nce},*})=1 ∑ k = 1 M q k ( j ) ( A nce , ∗ ) = 1 and ∑ j ′ = 1 N r j ′ ( j ) ( A n c e , ∗ ) = 1 \sum_{j'=1}^{N}r_{j'}^{(j)}(A^{\mathrm{nce},*})=1 ∑ j ′ = 1 N r j ′ ( j ) ( A nce , ∗ ) = 1 , we obtain from Eq. (79)
∥ a k n c e , ∗ ∥ ≤ 1 λ ( N k Z max + 1 2 ∑ j = 1 N q k ( j ) ( A n c e , ∗ ) Z max + 1 2 ∑ j ∈ P k ∑ j ′ = 1 N r j ′ ( j ) ( A n c e , ∗ ) Z max ) ≤ 1 λ ( N k + N 2 + N k 2 ) Z max = O ( λ − 1 ) . (90) \begin{aligned}
\left\|a_k^{\mathrm{nce},*}\right\|
\leq\frac{1}{\lambda}\left(N_kZ_{\max}+\frac{1}{2}\sum_{j=1}^{N}q_k^{(j)}(A^{\mathrm{nce},*})Z_{\max}+\frac{1}{2}\sum_{j\in\mathcal{P}_k}\sum_{j'=1}^{N}r_{j'}^{(j)}(A^{\mathrm{nce},*})Z_{\max}\right) \\
&\leq\frac{1}{\lambda}\left(N_k+\frac{N}{2}+\frac{N_k}{2}\right)Z_{\max}=O(\lambda^{-1}). \tag*{(90)}
\end{aligned} a k nce , ∗ ≤ λ 1 N k Z m a x + 2 1 j = 1 ∑ N q k ( j ) ( A nce , ∗ ) Z m a x + 2 1 j ∈ P k ∑ j ′ = 1 ∑ N r j ′ ( j ) ( A nce , ∗ ) Z m a x ≤ λ 1 ( N k + 2 N + 2 N k ) Z m a x = O ( λ − 1 ) . (90)
Therefore every logit ( a ℓ n c e , ∗ ) ⊤ z j (a_\ell^{\mathrm{nce},*})^{\top}z_j ( a ℓ nce , ∗ ) ⊤ z j is O ( λ − 1 ) O(\lambda^{-1}) O ( λ − 1 ) , uniformly over ( ℓ , j ) ∈ [ M ] × [ N ] (\ell,j)\in[M]\times[N] ( ℓ , j ) ∈ [ M ] × [ N ] . Hence the i2t logit vectors
s ( j ) : = ( ( a 1 n c e , ∗ ) ⊤ z j , … , ( a M n c e , ∗ ) ⊤ z j ) ∈ R M s^{(j)}:=\left((a_1^{\mathrm{nce},*})^{\top}z_j,\ldots,(a_M^{\mathrm{nce},*})^{\top}z_j\right)\in\mathbb{R}^{M} s ( j ) := ( ( a 1 nce , ∗ ) ⊤ z j , … , ( a M nce , ∗ ) ⊤ z j ) ∈ R M
and the t2i logit vectors
u ( j ) : = ( ( a c j ′ n c e , ∗ ) ⊤ z j ) j ′ ∈ [ N ] ∈ R N u^{(j)}:=\left((a_{c_{j'}}^{\mathrm{nce},*})^{\top}z_j\right)_{j'\in[N]}\in\mathbb{R}^{N} u ( j ) := ( ( a c j ′ nce , ∗ ) ⊤ z j ) j ′ ∈ [ N ] ∈ R N
have all entries’ magnitudes of order O ( λ − 1 ) O(\lambda^{-1}) O ( λ − 1 ) , and therefore satisfy ∥ s ( j ) ∥ , ∥ u ( j ) ∥ = O ( λ − 1 ) \|s^{(j)}\|,\|u^{(j)}\|=O(\lambda^{-1}) ∥ s ( j ) ∥ , ∥ u ( j ) ∥ = O ( λ − 1 ) , uniformly over the relevant indices. In particular, these norms converge to 0 0 0 as λ → ∞ \lambda\to\infty λ → ∞ , so for all sufficiently large λ \lambda λ they lie in a fixed neighborhood of 0 0 0 where we can apply Lemma C.3. Applying that lemma with K = M K=M K = M and K = N K=N K = N , respectively, gives
q k ( j ) ( A n c e , ∗ ) = 1 M + O ( λ − 1 ) , r j ′ ( j ) ( A n c e , ∗ ) = 1 N + O ( λ − 1 ) , (91) q_k^{(j)}(A^{\mathrm{nce},*})=\frac{1}{M}+O(\lambda^{-1}),\qquad r_{j'}^{(j)}(A^{\mathrm{nce},*})=\frac{1}{N}+O(\lambda^{-1}), \tag*{(91)} q k ( j ) ( A nce , ∗ ) = M 1 + O ( λ − 1 ) , r j ′ ( j ) ( A nce , ∗ ) = N 1 + O ( λ − 1 ) , (91)
uniformly over the relevant indices. Substituting these expansions into Eq. (79) gives
a k n c e , ∗ = 1 λ ( ∑ j ∈ P k z j − 1 2 M ∑ j = 1 N z j − N k 2 N ∑ j = 1 N z j ) + O ( λ − 2 ) = N k λ z ˉ k − 1 λ ( N 2 M + N k 2 ) z ˉ + O ( λ − 2 ) , (92) a_k^{\mathrm{nce},*} = \frac{1}{\lambda}\left(\sum_{j\in P_k} z_j-\frac{1}{2M}\sum_{j=1}^{N}z_j-\frac{N_k}{2N}\sum_{j=1}^{N}z_j\right)+O(\lambda^{-2})
= \frac{N_k}{\lambda}\bar{z}_k-\frac{1}{\lambda}\left(\frac{N}{2M}+\frac{N_k}{2}\right)\bar{z}+O(\lambda^{-2}), \tag*{(92)} a k nce , ∗ = λ 1 j ∈ P k ∑ z j − 2 M 1 j = 1 ∑ N z j − 2 N N k j = 1 ∑ N z j + O ( λ − 2 ) = λ N k z ˉ k − λ 1 ( 2 M N + 2 N k ) z ˉ + O ( λ − 2 ) , (92)
which is Eq. (10).
Proof of Corollary 4.8 (Prototype Decomposition)
Proof. Recall the per-node global-mean coefficient
ξ k n c e : = N 2 M + N k 2 N k , k ∈ V ∖ { r o o t } . (93) \xi_k^{\mathrm{nce}}:=\frac{\frac{N}{2M}+\frac{N_k}{2}}{N_k},\qquad k\in V\setminus\{\mathrm{root}\}. \tag*{(93)} ξ k nce := N k 2 M N + 2 N k , k ∈ V ∖ { root } . (93)
The observed captions are path-consistent by the setup in Section 4.1, and the same setup has exactly n n n images per leaf. Under these hypotheses, Assumptions 4.3 and 4.4, and the decomposition in Assumption 4.1, we can apply Lemma C.10 and Corollary C.11. By Assumption 4.3, N k > 0 N_k>0 N k > 0 for every non-root node k k k , so we can apply Proposition 4.7. Factor Eq. (10) using Eq. (93) , and substitute Eq. (48) with b k = b = 0 b_k=b=0 b k = b = 0 to obtain Eq. (11) .
Quantitative Remainder for the InfoNCE Expansion
The asymptotic expansion identifies the leading empirical means; transfer at a specified regularization level also requires a quantitative remainder bound.
Lemma E.3 (Quantitative remainder for the large-λ \lambda λ expansion). Let Z max : = max j ∈ [ N ] ∥ z j ∥ Z_{\max}:=\max_{j\in[N]}\lVert z_j\rVert Z m a x := max j ∈ [ N ] ∥ z j ∥ , assume N k > 0 N_k>0 N k > 0 for every non-root node k k k , and let N min : = min k ∈ V ∖ { r o o t } N k N_{\min}:=\min_{k\in V\setminus\{\mathrm{root}\}}N_k N m i n := min k ∈ V ∖ { root } N k . For each non-root node k ∈ V ∖ { r o o t } k\in V\setminus\{\mathrm{root}\} k ∈ V ∖ { root } , with ξ k n c e \xi_k^{\mathrm{nce}} ξ k nce as in Eq. (93) , define the prototype-decomposition residual r k , λ ∈ R d r_{k,\lambda}\in\mathbb{R}^{d} r k , λ ∈ R d by
r k , λ : = λ N k a k n c e , ∗ − ( z ˉ k − ξ k n c e z ˉ ) . (94) r_{k,\lambda}:=\frac{\lambda}{N_k}a_k^{\mathrm{nce},*}-\left(\bar{z}_k-\xi_k^{\mathrm{nce}}\bar{z}\right). \tag*{(94)} r k , λ := N k λ a k nce , ∗ − ( z ˉ k − ξ k nce z ˉ ) . (94)
Under the hypotheses of Corollary 4.8, z ˉ k − ξ k n c e z ˉ = μ ~ k + ϵ ˉ k − ξ k n c e ϵ ˉ \bar{z}_k-\xi_k^{\mathrm{nce}}\bar{z}=\widetilde{\mu}_k+\bar{\epsilon}_k-\xi_k^{\mathrm{nce}}\bar{\epsilon} z ˉ k − ξ k nce z ˉ = μ k + ϵ ˉ k − ξ k nce ϵ ˉ with μ ~ k : = μ k − ξ k n c e μ r o o t \widetilde{\mu}_k:=\mu_k-\xi_k^{\mathrm{nce}}\mu_{\mathrm{root}} μ k := μ k − ξ k nce μ root , so that, equivalently,
a k n c e , ∗ = N k λ ( μ ~ k + ϵ ˉ k − ξ k n c e ϵ ˉ + r k , λ ) . (95) a_k^{\mathrm{nce},*}=\frac{N_k}{\lambda}\left(\widetilde{\mu}_k+\bar{\epsilon}_k-\xi_k^{\mathrm{nce}}\bar{\epsilon}+r_{k,\lambda}\right). \tag*{(95)} a k nce , ∗ = λ N k ( μ k + ϵ ˉ k − ξ k nce ϵ ˉ + r k , λ ) . (95)
Then there exists a deterministic constant C r e m = C r e m ( M , N , N min ) C_{\mathrm{rem}}=C_{\mathrm{rem}}(M,N,N_{\min}) C rem = C rem ( M , N , N m i n ) , depending only on M M M , N N N , and N min N_{\min} N m i n , such that for every λ > 0 \lambda>0 λ > 0 ,
max k ∈ V ∖ { r o o t } ∥ r k , λ ∥ ≤ C r e m Z max 3 λ . (96) \max_{k\in V\setminus\{\mathrm{root}\}}\lVert r_{k,\lambda}\rVert\leq\frac{C_{\mathrm{rem}}Z_{\max}^{3}}{\lambda}. \tag*{(96)} k ∈ V ∖ { root } max ∥ r k , λ ∥ ≤ λ C rem Z m a x 3 . (96)
Proof. We first bound the difference between the KKT identity and its uniform-softmax approximation. For every non-root node k ∈ V ∖ { r o o t } k\in V\setminus\{\mathrm{root}\} k ∈ V ∖ { root } , define
ρ k , λ : = 1 2 λ ∑ j = 1 N ( q k ( j ) ( A n c e , ∗ ) − 1 M ) z j − 1 2 λ ∑ j ∈ P k ∑ j ′ = 1 N ( r j ′ ( j ) ( A n c e , ∗ ) − 1 N ) z j ′ . (97) \rho_{k,\lambda}:=\frac{1}{2\lambda}\sum_{j=1}^{N}\left(q_k^{(j)}(A^{\mathrm{nce},*})-\frac{1}{M}\right)z_j-\frac{1}{2\lambda}\sum_{j\in P_k}\sum_{j'=1}^{N}\left(r_{j'}^{(j)}(A^{\mathrm{nce},*})-\frac{1}{N}\right)z_{j'}. \tag*{(97)} ρ k , λ := 2 λ 1 j = 1 ∑ N ( q k ( j ) ( A nce , ∗ ) − M 1 ) z j − 2 λ 1 j ∈ P k ∑ j ′ = 1 ∑ N ( r j ′ ( j ) ( A nce , ∗ ) − N 1 ) z j ′ . (97)
Substituting Eq. (97) into the stationarity condition in Eq. (79) gives
a k n c e , ∗ = N k λ z ˉ k − 1 λ ( N 2 M + N k 2 ) z ˉ + ρ k , λ . (98) a_k^{\mathrm{nce},*}=\frac{N_k}{\lambda}\bar{z}_k-\frac{1}{\lambda}\left(\frac{N}{2M}+\frac{N_k}{2}\right)\bar{z}+\rho_{k,\lambda}. \tag*{(98)} a k nce , ∗ = λ N k z ˉ k − λ 1 ( 2 M N + 2 N k ) z ˉ + ρ k , λ . (98)
From Eq. (79) and the identities ∑ k q k ( j ) ( A n c e , ∗ ) = 1 \sum_k q_k^{(j)}(A^{\mathrm{nce},*})=1 ∑ k q k ( j ) ( A nce , ∗ ) = 1 , ∑ j ′ r j ′ ( j ) ( A n c e , ∗ ) = 1 \sum_{j'}r_{j'}^{(j)}(A^{\mathrm{nce},*})=1 ∑ j ′ r j ′ ( j ) ( A nce , ∗ ) = 1 , we have
max ℓ ∈ [ M ] ∥ a ℓ n c e , ∗ ∥ ≤ 1 λ ( N + N 2 + N 2 ) Z max = 2 N Z max λ . (99) \max_{\ell\in[M]}\lVert a_{\ell}^{\mathrm{nce},*}\rVert\leq\frac{1}{\lambda}\left(N+\frac{N}{2}+\frac{N}{2}\right)Z_{\max}=\frac{2NZ_{\max}}{\lambda}. \tag*{(99)} ℓ ∈ [ M ] max ∥ a ℓ nce , ∗ ∥ ≤ λ 1 ( N + 2 N + 2 N ) Z m a x = λ 2 N Z m a x . (99)
For each image index j ∈ [ N ] j \in[N] j ∈ [ N ] , each entry of the i2t logit vector s ( j ) s^{(j)} s ( j ) has magnitude at most ∥ a ℓ n c e , ∗ ∥ ∥ z j ∥ ≤ 2 N Z max 2 / λ \lVert a_{\ell}^{\mathrm{nce},*} \rVert\lVert z_j \rVert\le2N Z_{\max}^{2}/\lambda ∥ a ℓ nce , ∗ ∥ ∥ z j ∥ ≤ 2 N Z m a x 2 / λ . For each j ∈ [ N ] j \in[N] j ∈ [ N ] , each entry of the t2i logit vector u ( j ) u^{(j)} u ( j ) has the same upper bound. Hence
∥ s ( j ) ∥ ≤ M ⋅ 2 N Z max 2 λ , ∥ u ( j ) ∥ ≤ N ⋅ 2 N Z max 2 λ . (100) \lVert s^{(j)} \rVert\le\sqrt{M}\cdot\frac{2N Z_{\max}^{2}}{\lambda}, \qquad\lVert u^{(j)} \rVert\le\sqrt{N}\cdot\frac{2N Z_{\max}^{2}}{\lambda}. \tag*{(100)} ∥ s ( j ) ∥ ≤ M ⋅ λ 2 N Z m a x 2 , ∥ u ( j ) ∥ ≤ N ⋅ λ 2 N Z m a x 2 . (100)
Applying Lemma C.2 with K = M K=M K = M and K = N K=N K = N , respectively, gives
∣ q k ( j ) ( A n c e , ∗ ) − 1 M ∣ ≤ 2 ∥ s ( j ) ∥ ≤ c 1 ( M , N ) Z max 2 λ , (101) \left\lvert q_k^{(j)}(A^{\mathrm{nce},*})-\frac{1}{M}\right\rvert\le2\lVert s^{(j)}\rVert\le c_1(M,N)\frac{Z_{\max}^{2}}{\lambda}, \tag*{(101)} q k ( j ) ( A nce , ∗ ) − M 1 ≤ 2 ∥ s ( j ) ∥ ≤ c 1 ( M , N ) λ Z m a x 2 , (101)
∣ r j , j ′ ( i ) ( A n c e , ∗ ) − 1 N ∣ ≤ 2 ∥ u ( j ) ∥ ≤ c 1 ( M , N ) Z max 2 λ , (102) \left\lvert r_{j,j'}^{(i)}(A^{\mathrm{nce},*})-\frac{1}{N}\right\rvert\le2\lVert u^{(j)}\rVert\le c_1(M,N)\frac{Z_{\max}^{2}}{\lambda}, \tag*{(102)} r j , j ′ ( i ) ( A nce , ∗ ) − N 1 ≤ 2 ∥ u ( j ) ∥ ≤ c 1 ( M , N ) λ Z m a x 2 , (102)
uniformly over the relevant indices, where c 1 ( M , N ) : = 4 N M + N c_1(M,N):=4N\sqrt{M+N} c 1 ( M , N ) := 4 N M + N . Using the fact that ∥ z j ∥ ≤ Z max \lVert z_j\rVert\le Z_{\max} ∥ z j ∥ ≤ Z m a x and Eqs. (101) –(102) ,
∥ ρ k , λ ∥ ≤ 1 2 λ ( N + N k N ) ⋅ c 1 ( M , N ) Z max 2 λ ⋅ Z max ≤ c 2 ( M , N ) Z max 3 λ 2 , (103) \lVert\rho_{k,\lambda}\rVert\le\frac{1}{2\lambda}\left(N+N_kN\right)\cdot c_1(M,N)\frac{Z_{\max}^{2}}{\lambda}\cdot Z_{\max}\le\frac{c_2(M,N)Z_{\max}^{3}}{\lambda^{2}}, \tag*{(103)} ∥ ρ k , λ ∥ ≤ 2 λ 1 ( N + N k N ) ⋅ c 1 ( M , N ) λ Z m a x 2 ⋅ Z m a x ≤ λ 2 c 2 ( M , N ) Z m a x 3 , (103)
with c 2 ( M , N ) : = N ( 1 + N ) c 1 ( M , N ) / 2 c_2(M,N):=N(1+N)c_1(M,N)/2 c 2 ( M , N ) := N ( 1 + N ) c 1 ( M , N ) /2 .
By Eq. (93) , the coefficient of z ˉ \bar z z ˉ in Eq. (98) equals N k ξ k n c e / λ N_k\xi_k^{\mathrm{nce}}/\lambda N k ξ k nce / λ , and therefore we can rewrite Eq. (98) as
a k n c e , ∗ = N k λ ( z ˉ k − ξ k n c e z ˉ ) + ρ k , λ . (104) a_k^{\mathrm{nce},*}=\frac{N_k}{\lambda}\left(\bar z_k-\xi_k^{\mathrm{nce}}\bar z\right)+\rho_{k,\lambda}. \tag*{(104)} a k nce , ∗ = λ N k ( z ˉ k − ξ k nce z ˉ ) + ρ k , λ . (104)
Comparing this identity with the definition of r k , λ r_{k,\lambda} r k , λ in Eq. (94) gives
N k λ r k , λ = ρ k , λ , (105) \frac{N_k}{\lambda}r_{k,\lambda}=\rho_{k,\lambda}, \tag*{(105)} λ N k r k , λ = ρ k , λ , (105)
and therefore ∥ r k , λ ∥ ≤ c 2 ( M , N ) Z max 3 / ( N k λ ) \lVert r_{k,\lambda}\rVert\le c_2(M,N)Z_{\max}^{3}/(N_k\lambda) ∥ r k , λ ∥ ≤ c 2 ( M , N ) Z m a x 3 / ( N k λ ) . Since N k ≥ N min N_k\ge N_{\min} N k ≥ N m i n for every non-root node k ∈ V ∖ { r o o t } k\in V\setminus\{\mathrm{root}\} k ∈ V ∖ { root } ,
max k ∈ V ∖ { r o o t } ∥ r k , λ ∥ ≤ C r e m Z max 3 λ , (106) \max_{k\in V\setminus\{\mathrm{root}\}}\lVert r_{k,\lambda}\rVert\le\frac{C_{\mathrm{rem}}Z_{\max}^{3}}{\lambda}, \tag*{(106)} k ∈ V ∖ { root } max ∥ r k , λ ∥ ≤ λ C rem Z m a x 3 , (106)
where C r e m ( M , N , N min ) : = c 2 ( M , N ) N min C_{\mathrm{rem}}(M,N,N_{\min}):=\frac{c_2(M,N)}{N_{\min}} C rem ( M , N , N m i n ) := N m i n c 2 ( M , N ) .
Shared Transfer Argument
For λ > 0 \lambda>0 λ > 0 , write { a k } \{a_k\} { a k } for the minimizer under consideration, A l i n , ∗ A^{\mathrm{lin},*} A lin , ∗ of Eq. (5) for the linear loss and A n c e , ∗ A^{\mathrm{nce},*} A nce , ∗ of Eq. (9) for InfoNCE. For both losses a k a_k a k is a positive multiple of the empirical-mean difference z ˉ k − ξ k z ˉ \bar z_k-\xi_k\bar z z ˉ k − ξ k z ˉ : exactly so for the linear loss, and up to a remainder bounded by Lemma E.3 for InfoNCE. The two transfer proofs therefore share the same steps, which we present next. We allow a caption bias throughout, so that the main theorems are the special case of zero bias and the relaxations of Appendix L need only a bound on it.
In Lemma F.1 below, we need only a bound on the perturbation in a representation a k = c k ( μ k + perturbation ) a_k=c_k\left(\mu_k+\text{perturbation}\right) a k = c k ( μ k + perturbation ) with c k > 0 c_k>0 c k > 0 . It has three components, the caption bias, the sampling noise, and the finite regularization, which we define in turn below.
Recall from Definition C.9 that b k = μ ˉ k − μ k b_k=\bar\mu_k-\mu_k b k = μ ˉ k − μ k and b = μ ˉ − μ r o o t b=\bar\mu-\mu_{\mathrm{root}} b = μ ˉ − μ root , the gaps between the caption-averaged prototypes and the true ones; by Eq. (48) the bias contributes b k − ξ k b b_k-\xi_kb b k − ξ k b to the empirical-mean difference. The transfer bound uses that contribution only through an upper bound on its largest norm, so for fixed observed captions with non-root coverage and a fixed family of coefficients { ξ k ∈ R } k ∈ V ∖ { r o o t } \{\xi_k\in\mathbb{R}\}_{k\in V\setminus\{\mathrm{root}\}} { ξ k ∈ R } k ∈ V ∖ { root } , we call δ b i a s ≥ 0 \delta_{\mathrm{bias}}\ge0 δ bias ≥ 0 a caption bias bound if
max k ≠ r o o t ∥ b k − ξ k b ∥ ≤ δ b i a s . (107) \max_{k\ne\mathrm{root}}\lVert b_k-\xi_kb\rVert\le\delta_{\mathrm{bias}}. \tag*{(107)} k = root max ∥ b k − ξ k b ∥ ≤ δ bias . (107)
The smallest caption bias bound is the left-hand side itself. We allow larger values so that we can apply Lemma F.1 directly with an upper bound on this maximum in Appendix L.
The error terms D ε l i n D_{\varepsilon}^{\mathrm{lin}} D ε lin and D ε n c e D_{\varepsilon}^{\mathrm{nce}} D ε nce of the main theorems carry over unchanged. Throughout, ξ k \xi_k ξ k are the coefficients of the loss under consideration, from Corollary 4.5 for the linear loss and Eq. (93) for InfoNCE, and ξ max \xi_{\max} ξ m a x , N min N_{\min} N m i n , and δ r o o t \delta_{\mathrm{root}} δ root are evaluated at the observed captions and those coefficients, and δ b i a s \delta_{\mathrm{bias}} δ bias is a caption bias bound for them. Write D ϵ l i n D^{\mathrm{lin}}_{\epsilon} D ϵ lin for the expression in Eq. (7) and D ϵ n c e D^{\mathrm{nce}}_{\epsilon} D ϵ nce for its InfoNCE counterpart defined before Theorem 4.9, using the universal constant from Lemma C.12 . Both expressions are defined for any observed captions with non-root coverage, regardless of caption symmetry, so we use the same D ϵ l i n D^{\mathrm{lin}}_{\epsilon} D ϵ lin and D ϵ n c e D^{\mathrm{nce}}_{\epsilon} D ϵ nce in the biased setting here and in the symmetric setting of the main theorems.
For InfoNCE we also choose a regularization threshold that makes the large-λ \lambda λ remainder no larger than D ϵ n c e D^{\mathrm{nce}}_{\epsilon} D ϵ nce . For ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) and D ϵ n c e > 0 D^{\mathrm{nce}}_{\epsilon} > 0 D ϵ nce > 0 , define
R ϵ : = max ℓ ∈ V l e a f ∥ μ ℓ ∥ + C max σ d + log ( 4 N / ϵ ) , λ 0 ( ϵ ) : = C r e m R ϵ 3 D ϵ n c e , (108) R_{\epsilon} := \max_{\ell\in V_{\mathrm{leaf}}} \lVert\mu_{\ell} \rVert+ C_{\max}\sigma\sqrt{d+\log(4N/\epsilon)}, \qquad\lambda_{0}(\epsilon) := \frac{C_{\mathrm{rem}}R_{\epsilon}^{3}}{D^{\mathrm{nce}}_{\epsilon}}, \tag*{(108)} R ϵ := ℓ ∈ V leaf max ∥ μ ℓ ∥ + C m a x σ d + log ( 4 N / ϵ ) , λ 0 ( ϵ ) := D ϵ nce C rem R ϵ 3 , (108)
where C max > 0 C_{\max} > 0 C m a x > 0 is the universal constant from Lemma C.13 and C r e m = C r e m ( M , N , N min ) C_{\mathrm{rem}} = C_{\mathrm{rem}}(M,N,N_{\min}) C rem = C rem ( M , N , N m i n ) is the deterministic constant from Lemma E.3. In particular, R ϵ R_{\epsilon} R ϵ is independent of the captions, whereas λ 0 ( ϵ ) \lambda_{0}(\epsilon) λ 0 ( ϵ ) depends on the observed captions through C r e m C_{\mathrm{rem}} C rem and D ϵ n c e D^{\mathrm{nce}}_{\epsilon} D ϵ nce .
Recall the minimum non-root prototype norm ρ min : = min k ≠ r o o t ∥ μ k ∥ \rho_{\min} := \min_{k \ne\mathrm{root}} \lVert\mu_k \rVert ρ m i n := min k = root ∥ μ k ∥ , which sets the scale of the threshold below. For ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) , the total error term D ϵ b i a s D^{\mathrm{bias}}_{\epsilon} D ϵ bias and the margin threshold τ ϵ b i a s \tau^{\mathrm{bias}}_{\epsilon} τ ϵ bias it induces are
D ϵ b i a s : = { D ϵ l i n + δ b i a s , linear contrastive loss , 2 D ϵ n c e + δ b i a s , InfoNCE loss , τ ϵ b i a s : = 8 D ϵ b i a s ρ min . (109) D^{\mathrm{bias}}_{\epsilon} :=
\begin{cases}
D^{\mathrm{lin}}_{\epsilon}+\delta_{\mathrm{bias}}, & \text{linear contrastive loss},\\
2D^{\mathrm{nce}}_{\epsilon}+\delta_{\mathrm{bias}}, & \text{InfoNCE loss},
\end{cases}
\qquad
\tau^{\mathrm{bias}}_{\epsilon} := \frac{8D^{\mathrm{bias}}_{\epsilon}}{\rho_{\min}}. \tag*{(109)} D ϵ bias := { D ϵ lin + δ bias , 2 D ϵ nce + δ bias , linear contrastive loss , InfoNCE loss , τ ϵ bias := ρ m i n 8 D ϵ bias . (109)
The caption bias enters additively in both cases, and the factor 2 in the InfoNCE case appears because D ϵ n c e D^{\mathrm{nce}}_{\epsilon} D ϵ nce bounds each of two terms, the perturbation from the empirical means and the large-λ \lambda λ remainder.
In Lemma F.1 , we assume neither Assumption 4.3 nor Assumption 4.4, and nothing about the global margin γ H A S \gamma_{\mathcal{H}\mathcal{A}\mathcal{S}} γ H A S ; the captions enter only through δ b i a s \delta_{\mathrm{bias}} δ bias .
Lemma F.1 (HAS transfer under biased captions). Suppose Assumption 4.1 (sub-Gaussian prototype variation) holds, ρ min > 0 \rho_{\min} > 0 ρ m i n > 0 , and the observed captions are deterministic with N k > 0 N_k > 0 N k > 0 for every non-root node k k k . Fix ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) , λ > 0 \lambda> 0 λ > 0 , and a caption bias bound δ b i a s \delta_{\mathrm{bias}} δ bias , and suppose D ϵ b i a s < ρ min / 4 D^{\mathrm{bias}}_{\epsilon} < \rho_{\min}/4 D ϵ bias < ρ m i n /4 ; in the InfoNCE case suppose also D ϵ n c e > 0 D^{\mathrm{nce}}_{\epsilon} > 0 D ϵ nce > 0 and λ ≥ λ 0 ( ϵ ) \lambda\ge\lambda_0(\epsilon) λ ≥ λ 0 ( ϵ ) . Then, with probability at least 1 − ϵ 1-\epsilon 1 − ϵ , all of the following hold simultaneously:
(i) a k ≠ 0 a_k \ne0 a k = 0 for every non-root node k k k ;
(ii) every admissible triplet ( u , v , w ) (u,v,w) ( u , v , w ) with γ u v w > τ ϵ b i a s \gamma_{uvw} > \tau^{\mathrm{bias}}_{\epsilon} γ uv w > τ ϵ bias satisfies cos ( a u , a v ) > cos ( a u , a w ) \cos(a_u,a_v) > \cos(a_u,a_w) cos ( a u , a v ) > cos ( a u , a w ) ;
(iii) H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Violt x t ( u ) ≤ H A S ^{\mathrm{txt}}(u) \le\mathcal{H}\mathcal{A}\mathcal{S} txt ( u ) ≤ H A S -Violτ ϵ b i a s i m g ( u ) ^{\mathrm{img}}_{\tau^{\mathrm{bias}}_{\epsilon}}(u) τ ϵ bias img ( u ) for every non-root non-leaf node u u u , with H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Violt x t ( u ) ^{\mathrm{txt}}(u) txt ( u ) computed on { a k } k ≠ r o o t \{a_k\}_{k\ne\mathrm{root}} { a k } k = root .
Proof. Step 1: Decompose the optimal embeddings. Fix the observed captions and the coefficients for the loss under consideration. For every non-root node k k k , set
e k : = − ξ k μ r o o t + e ˉ k − ξ k e ˉ , b ~ k : = b k − ξ k b . (110) e_k := -\xi_k\mu_{\mathrm{root}}+\bar e_k-\xi_k\bar e,\qquad\widetilde b_k := b_k-\xi_k b. \tag*{(110)} e k := − ξ k μ root + e ˉ k − ξ k e ˉ , b k := b k − ξ k b . (110)
Under coverage, we apply Corollary D.2 for the linear loss and Lemma E.3 for InfoNCE to write a k a_k a k as a positive multiple of z ˉ k − ξ k z ˉ \bar z_k-\xi_k\bar z z ˉ k − ξ k z ˉ plus the remainder. Under Assumption 4.1 , every sample satisfies z j = μ y j + e j z_j=\mu_{y_j}+e_j z j = μ y j + e j , which is the form required by Lemma C.10 . Substituting Eq. (48) into those empirical-mean expressions gives
a k = c k ( μ k + e k + b ~ k + r k , λ ) , (111) a_k = c_k\left(\mu_k+e_k+\widetilde b_k+r_{k,\lambda}\right), \tag*{(111)} a k = c k ( μ k + e k + b k + r k , λ ) , (111)
where c k = N k M ( 1 + 1 2 ( M − 1 ) + 1 2 ( N k − 1 ) ) c_k = \dfrac{N_k}{M}\left(1+\dfrac{1}{2(M-1)}+\dfrac{1}{2(N_k-1)}\right) c k = M N k ( 1 + 2 ( M − 1 ) 1 + 2 ( N k − 1 ) 1 ) and r k , λ = 0 r_{k,\lambda}=0 r k , λ = 0 for the linear loss, whereas c k = N k / λ c_k=N_k/\lambda c k = N k / λ and r k , λ r_{k,\lambda} r k , λ is defined by Eq. (94) for InfoNCE. In both cases, coverage and λ > 0 \lambda>0 λ > 0 give c k > 0 c_k>0 c k > 0 , and by Eq. (107) , max k ≠ r o o t ∥ b ~ k ∥ ≤ δ b i a s \max_{k\ne\mathrm{root}}\lVert\widetilde b_k\rVert\le\delta_{\mathrm{bias}} max k = root ∥ b k ∥ ≤ δ bias .
Step 2: Bound the stochastic deviation. For a failure probability p ∈ ( 0 , 1 ) p\in(0,1) p ∈ ( 0 , 1 ) , define
η p : = C σ ( d + log ( M − 1 ) + log ( 4 / p ) N min + ξ max d + log ( 4 / p ) N ) , (112) \eta_p := C\sigma\left(\sqrt{\frac{d+\log(M-1)+\log(4/p)}{N_{\min}}}+\xi_{\max}\sqrt{\frac{d+\log(4/p)}{N}}\right), \tag*{(112)} η p := C σ N m i n d + log ( M − 1 ) + log ( 4/ p ) + ξ m a x N d + log ( 4/ p ) , (112)
and let E p : = { max k ≠ r o o t ∥ ϵ ˉ k − ξ k ϵ ˉ ∥ ≤ η p } \mathcal{E}_p := \{\max_{k\ne\mathrm{root}} \|\bar{\epsilon}_k-\xi_k\bar{\epsilon}\| \le\eta_p\} E p := { max k = root ∥ ϵ ˉ k − ξ k ϵ ˉ ∥ ≤ η p } . The observed captions are deterministic and have coverage, so the index sets P k P_k P k are fixed and nonempty, and the variations ϵ j \epsilon_j ϵ j are independent, centered, and sub-Gaussian by Assumption 4.1. Apply Lemma C.12 with K = V ∖ { r o o t } K=V\setminus\{\mathrm{root}\} K = V ∖ { root } , I k = P k I_k=P_k I k = P k for every k ∈ K k\in K k ∈ K , the fixed coefficients ξ k \xi_k ξ k , and u = log ( 4 / p ) u=\log(4/p) u = log ( 4/ p ) . Since ∣ K ∣ = M − 1 |K|=M-1 ∣ K ∣ = M − 1 and min k ∈ K ∣ I k ∣ = N min \min_{k\in K}|I_k|=N_{\min} min k ∈ K ∣ I k ∣ = N m i n , we obtain Pr ( E p ) ≥ 1 − p \Pr(\mathcal{E}_p)\ge1-p Pr ( E p ) ≥ 1 − p from Eq. (51) . On E p \mathcal{E}_p E p , Eq. (110) and the triangle inequality give
max k ≠ r o o t ∥ e k ∥ ≤ δ r o o t + η p . (113) \max_{k\ne\mathrm{root}}\|e_k\|\le\delta_{\mathrm{root}}+\eta_p. \tag*{(113)} k = root max ∥ e k ∥ ≤ δ root + η p . (113)
For the linear loss, choose p = ϵ p=\epsilon p = ϵ . Then δ r o o t + η ϵ = D ϵ l i n \delta_{\mathrm{root}}+\eta_\epsilon=D^{\mathrm{lin}}_\epsilon δ root + η ϵ = D ϵ lin and the remainder is zero, so the total perturbation in Eq. (111) is at most D ϵ l i n + δ b i a s = D ϵ b i a s D^{\mathrm{lin}}_\epsilon+\delta_{\mathrm{bias}}=D^{\mathrm{bias}}_\epsilon D ϵ lin + δ bias = D ϵ bias with probability at least 1 − ϵ 1-\epsilon 1 − ϵ .
Step 3: Control the InfoNCE remainder. For InfoNCE, choose p = ϵ / 2 p=\epsilon/2 p = ϵ /2 in Step 2, so δ r o o t + η ϵ / 2 = D ϵ n c e \delta_{\mathrm{root}}+\eta_{\epsilon/2}=D^{\mathrm{nce}}_\epsilon δ root + η ϵ /2 = D ϵ nce . Under Assumption 4.1, we can also apply Lemma C.13 with failure probability p = ϵ / 2 p=\epsilon/2 p = ϵ /2 . Thus F ϵ : = { max j ∈ [ N ] ∥ z j ∥ ≤ R ϵ } \mathcal{F}_\epsilon:=\{\max_{j\in[N]}\|z_j\|\le R_\epsilon\} F ϵ := { max j ∈ [ N ] ∥ z j ∥ ≤ R ϵ } has probability at least 1 − ϵ / 2 1-\epsilon/2 1 − ϵ /2 . On F ϵ \mathcal{F}_\epsilon F ϵ , by Lemma E.3 and Eq. (108) , for every λ ≥ λ 0 ( ϵ ) \lambda\ge\lambda_0(\epsilon) λ ≥ λ 0 ( ϵ ) ,
max k ≠ r o o t ∥ r k , λ ∥ ≤ C r e m R ϵ 3 λ ≤ D ϵ n c e . (114) \max_{k\ne\mathrm{root}}\|r_{k,\lambda}\|\le\frac{C_{\mathrm{rem}}R_\epsilon^3}{\lambda}\le D^{\mathrm{nce}}_\epsilon. \tag*{(114)} k = root max ∥ r k , λ ∥ ≤ λ C rem R ϵ 3 ≤ D ϵ nce . (114)
The union bound gives Pr ( E ϵ / 2 ∩ F ϵ ) ≥ 1 − ϵ \Pr(\mathcal{E}_{\epsilon/2}\cap\mathcal{F}_\epsilon)\ge1-\epsilon Pr ( E ϵ /2 ∩ F ϵ ) ≥ 1 − ϵ ; independence of these events is not needed. On their intersection, the total perturbation is at most 2 D ϵ n c e + δ b i a s = D ϵ b i a s 2D^{\mathrm{nce}}_\epsilon+\delta_{\mathrm{bias}}=D^{\mathrm{bias}}_\epsilon 2 D ϵ nce + δ bias = D ϵ bias .
Step 4: Apply the per-triplet transfer bound. For either loss, we have established an event of probability at least 1 − ϵ 1-\epsilon 1 − ϵ on which
max k ≠ r o o t ∥ e k + b ~ k + r k , λ ∥ ≤ D ϵ b i a s < ρ min / 4 < ρ min . (115) \max_{k\ne\mathrm{root}}\|e_k+\tilde b_k+r_{k,\lambda}\|\le D^{\mathrm{bias}}_\epsilon<\rho_{\min}/4<\rho_{\min}. \tag*{(115)} k = root max ∥ e k + b ~ k + r k , λ ∥ ≤ D ϵ bias < ρ m i n /4 < ρ m i n . (115)
Apply Lemma C.8 with reference vectors ν k = μ k \nu_k=\mu_k ν k = μ k , the positive scalings c k c_k c k in Eq. (111) , and the total perturbations e k + b ~ k + r k , λ e_k+\tilde b_k+r_{k,\lambda} e k + b ~ k + r k , λ . With η = D ϵ b i a s \eta=D^{\mathrm{bias}}_\epsilon η = D ϵ bias its threshold is τ = 8 D ϵ b i a s / ρ min = τ ϵ b i a s \tau=8D^{\mathrm{bias}}_\epsilon/\rho_{\min}=\tau^{\mathrm{bias}}_\epsilon τ = 8 D ϵ bias / ρ m i n = τ ϵ bias , so parts (i), (ii), and (iii) of that lemma are claims (i), (ii), and (iii). □ \square □
Proof of Hierarchical Angular Separation Transfer for the Linear Contrastive Loss
Theorem 4.6 (H A S HAS H A S Transfer for the Linear Contrastive Loss). Suppose Assumption 4.1 (sub-Gaussian prototype variation), Assumption 4.3 (leaf-uniform caption coverage), and Assumption 4.4 (prototype consistency) hold, and suppose ρ min > 0 \rho_{\min}>0 ρ m i n > 0 . Let N min : = min k ∈ V ∖ { r o o t } N k N_{\min}:=\min_{k\in V\setminus\{\mathrm{root}\}}N_k N m i n := min k ∈ V ∖ { root } N k be the minimum non-root caption count. There exists a universal constant C > 0 C>0 C > 0 such that the following holds for every ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) . Define the combined error term
D ϵ l i n : = δ r o o t + C σ ( d + log ( M − 1 ) + log ( 4 / ϵ ) N min + ξ max l i n d + log ( 4 / ϵ ) N ) . (116) D^{\mathrm{lin}}_\epsilon:=\delta_{\mathrm{root}}+C\sigma\left(\sqrt{\frac{d+\log(M-1)+\log(4/\epsilon)}{N_{\min}}}+\xi_{\max}^{\mathrm{lin}}\sqrt{\frac{d+\log(4/\epsilon)}{N}}\right). \tag*{(116)} D ϵ lin := δ root + C σ N m i n d + log ( M − 1 ) + log ( 4/ ϵ ) + ξ m a x lin N d + log ( 4/ ϵ ) . (116) If D ϵ l i n < ρ min / 4 D^{\mathrm{lin}}_\epsilon<\rho_{\min}/4 D ϵ lin < ρ m i n /4 , then, with probability at least 1 − ϵ 1-\epsilon 1 − ϵ , every non-root non-leaf anchor node u u u satisfies
H A S - V i o l t x t ( u ) ≤ H A S - V i o l τ ϵ l i n i m g ( u ) , τ ϵ l i n : = 8 D ϵ l i n ρ min . (117) \mathcal{H}\mathcal{A}\mathcal{S}\text{-}\mathrm{Viol}^{\mathrm{txt}}(u)\le\mathcal{H}\mathcal{A}\mathcal{S}\text{-}\mathrm{Viol}^{\mathrm{img}}_{\tau^{\mathrm{lin}}_\epsilon}(u),\qquad\tau^{\mathrm{lin}}_\epsilon:=\frac{8D^{\mathrm{lin}}_\epsilon}{\rho_{\min}}. \tag*{(117)} H A S - Viol txt ( u ) ≤ H A S - Viol τ ϵ lin img ( u ) , τ ϵ lin := ρ m i n 8 D ϵ lin . (117) Proof. Step 1: Establish zero caption bias. The observed captions are path-consistent by the setup in Section 4.1, and the same setup has exactly n n n images per leaf. By Corollary C.1 together with Assumptions 4.3 and 4.4, b k = 0 b_k=0 b k = 0 for every non-root node k k k and b = 0 b=0 b = 0 . Therefore, by Eq. (107) , δ b i a s = 0 \delta_{\mathrm{bias}}=0 δ bias = 0 is a caption bias bound for the linear-loss coefficients.
Step 2: Bound the error term. We obtain non-root coverage from Assumption 4.3 and the required variation model from Assumption 4.1, and we assume ρ min > 0 \rho_{\min}>0 ρ m i n > 0 and D ϵ l i n < ρ min / 4 D^{\mathrm{lin}}_\epsilon<\rho_{\min}/4 D ϵ lin < ρ m i n /4 in the theorem. With δ b i a s = 0 \delta_{\mathrm{bias}}=0 δ bias = 0 , we have D ϵ b i a s = D ϵ l i n < ρ min / 4 D^{\mathrm{bias}}_\epsilon=D^{\mathrm{lin}}_\epsilon<\rho_{\min}/4 D ϵ bias = D ϵ lin < ρ m i n /4 by Eq. (109) .
Step 3: Apply the shared transfer lemma. We can apply Lemma E.1 with τ ϵ b i a s = 8 D ϵ l i n / ρ min = τ ϵ l i n \tau^{\mathrm{bias}}_\epsilon=8D^{\mathrm{lin}}_\epsilon/\rho_{\min}=\tau^{\mathrm{lin}}_\epsilon τ ϵ bias = 8 D ϵ lin / ρ m i n = τ ϵ lin . Its part (iii) gives Eq. (8) with probability at least 1 − ϵ 1-\epsilon 1 − ϵ ; parts (i) and (ii) also establish nonzero optimal embeddings and preservation of every admissible triplet with margin greater than τ ϵ l i n \tau^{\mathrm{lin}}_\epsilon τ ϵ lin . □ \square □
Proof of Hierarchical Angular Separation Transfer for the InfoNCE Loss
Theorem 4.9 (HAS Transfer for InfoNCE). *Suppose Assumptions 4.1, 4.3, and 4.4 hold and ρ min > 0 \rho_{\min} > 0 ρ m i n > 0 , and fix ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) . If 0 < D ϵ n c e < ρ min / 8 0 < D_{\epsilon}^{\mathrm{nce}} < \rho_{\min}/8 0 < D ϵ nce < ρ m i n /8 , then there is a deterministic threshold λ 0 ( ϵ ) > 0 \lambda_0(\epsilon) > 0 λ 0 ( ϵ ) > 0 such that, for every λ ≥ λ 0 ( ϵ ) \lambda\ge\lambda_0(\epsilon) λ ≥ λ 0 ( ϵ ) , with probability at least 1 − ϵ 1-\epsilon 1 − ϵ , every non-root non-leaf anchor node u u u satisfies Eq. (8) with τ ϵ n c e \tau_{\epsilon}^{\mathrm{nce}} τ ϵ nce replaced by τ ϵ n c e : = 16 D ϵ n c e / ρ min \tau_{\epsilon}^{\mathrm{nce}} := 16D_{\epsilon}^{\mathrm{nce}}/\rho_{\min} τ ϵ nce := 16 D ϵ nce / ρ m i n .
Proof. Step 1: Establish zero caption bias. The observed captions are path-consistent by the setup in Section 4.1, and the same setup has exactly n n n images per leaf. By Corollary C.11 together with Assumptions 4.3 and 4.4, b k = 0 b_k = 0 b k = 0 for every non-root node k k k and b = 0 b = 0 b = 0 . Therefore, by Eq. (107) , δ b i a s = 0 \delta_{\mathrm{bias}} = 0 δ bias = 0 is a caption bias bound for the InfoNCE coefficients.
Step 2: Bound the error term. We obtain non-root coverage from Assumption 4.3 and the required variation model from Assumption 4.1. The theorem hypothesis 0 < D ϵ n c e < ρ min / 8 0 < D_{\epsilon}^{\mathrm{nce}} < \rho_{\min}/8 0 < D ϵ nce < ρ m i n /8 gives D ϵ b i a s = 2 D ϵ n c e < ρ min / 4 D_{\epsilon}^{\mathrm{bias}} = 2D_{\epsilon}^{\mathrm{nce}} < \rho_{\min}/4 D ϵ bias = 2 D ϵ nce < ρ m i n /4 and makes the deterministic threshold in Eq. (108) positive and finite.
Step 3: Apply the shared transfer lemma. For every λ ≥ λ 0 ( ϵ ) \lambda\ge\lambda_0(\epsilon) λ ≥ λ 0 ( ϵ ) , we can therefore apply Lemma F.1 with τ ϵ b i a s = 16 D ϵ n c e / ρ min = τ ϵ n c e \tau_{\epsilon}^{\mathrm{bias}} = 16D_{\epsilon}^{\mathrm{nce}}/\rho_{\min} = \tau_{\epsilon}^{\mathrm{nce}} τ ϵ bias = 16 D ϵ nce / ρ m i n = τ ϵ nce . Its part (iii) gives Eq. (8) at τ ϵ n c e \tau_{\epsilon}^{\mathrm{nce}} τ ϵ nce with probability at least 1 − ϵ 1-\epsilon 1 − ϵ ; parts (i) and (ii) also establish nonzero optimal embeddings and preservation of every admissible triplet with margin greater than τ ϵ n c e \tau_{\epsilon}^{\mathrm{nce}} τ ϵ nce .
Step-by-step construction of a Hierarchical Concept Embedding
In this section, we provide an explicit construction for the hierarchical prototypes { μ k } k ∈ V \{\mu_k\}_{k\in V} { μ k } k ∈ V used in our theoretical analysis, introduced by Nguyen et al. [9 ] . Throughout this section the tree T \mathcal{T} T of Section 3.1 is balanced with depth L ≥ 2 L \ge2 L ≥ 2 and branching factor b ≥ 2 b \ge2 b ≥ 2 , so every non-leaf node has b b b children and there are m = b L m = b^L m = b L leaves. The construction has the following properties: (i) the displacement μ k − μ p \mu_k-\mu_p μ k − μ p from each strict ancestor p p p to a non-root node k k k is orthogonal to μ p \mu_p μ p , (ii) sibling prototypes { μ u ( j ) } j = 1 b \{\mu_{u^{(j)}}\}_{j=1}^{b} { μ u ( j ) } j = 1 b form a regular simplex around their parent μ u \mu_u μ u , and (iii) every child μ u ( j ) \mu_{u^{(j)}} μ u ( j ) is located in a fixed half-angle cone around its parent μ u \mu_u μ u . The prototypes satisfy the averaging property in Lemma C.4, while the angle schedule in Proposition I.1 also ensures a positive HAS margin.
Fix an internal node u ∈ V u \in V u ∈ V at depth l : = d ( u ) ≥ 0 l := d(u) \ge0 l := d ( u ) ≥ 0 with children u ( 1 ) , … , u ( b ) u^{(1)},\ldots,u^{(b)} u ( 1 ) , … , u ( b ) , and write π 0 ( u ) , π 1 ( u ) , … , π l ( u ) \pi_0(u),\pi_1(u),\ldots,\pi_l(u) π 0 ( u ) , π 1 ( u ) , … , π l ( u ) for the ancestors of u u u along the root-to-u u u path, with π l ( u ) = u \pi_l(u)=u π l ( u ) = u . Assume the ancestor prototypes μ π 0 ( u ) , … , μ π l ( u ) \mu_{\pi_0(u)},\ldots,\mu_{\pi_l(u)} μ π 0 ( u ) , … , μ π l ( u ) have already been constructed and stack them into the matrix
A u : = [ μ π 0 ( u ) μ π 1 ( u ) ⋯ μ π l ( u ) ] ∈ R d × ( l + 1 ) . (118) A_u := \begin{bmatrix} \mu_{\pi_0(u)} & \mu_{\pi_1(u)} & \cdots& \mu_{\pi_l(u)} \end{bmatrix} \in\mathbb{R}^{d\times(l+1)}. \tag*{(118)} A u := [ μ π 0 ( u ) μ π 1 ( u ) ⋯ μ π l ( u ) ] ∈ R d × ( l + 1 ) . (118)
The construction below produces the b b b child prototypes { μ u ( j ) } j = 1 b \{\mu_{u^{(j)}}\}_{j=1}^{b} { μ u ( j ) } j = 1 b and proceeds inductively starting from any nonzero root prototype μ r o o t ∈ R d \mu_{\mathrm{root}} \in\mathbb{R}^{d} μ root ∈ R d .
Geometric requirements
The child prototypes are required to satisfy the following three conditions, where θ l ∈ ( 0 , π / 2 ) \theta_l \in(0,\pi/2) θ l ∈ ( 0 , π /2 ) is a fixed cone half-angle for depth-l l l children.
(R1) Hierarchical orthogonality. For every ancestor π k ( u ) \pi_k(u) π k ( u ) , k = 0 , … , l k=0,\ldots,l k = 0 , … , l , and every child j = 1 , … , b j=1,\ldots,b j = 1 , … , b ,
( μ u ( j ) − μ π k ( u ) ) T μ π k ( u ) = 0. (119) \left(\mu_{u^{(j)}}-\mu_{\pi_k(u)}\right)^{\mathsf{T}}\mu_{\pi_k(u)}=0. \tag*{(119)} ( μ u ( j ) − μ π k ( u ) ) T μ π k ( u ) = 0. (119)
(R2) Regular ( b − 1 ) (b-1) ( b − 1 ) -simplex on siblings. With the parent μ u \mu_u μ u as the simplex centroid,
( μ u ( j ) − μ u ) T ( μ u ( k ) − μ u ) = { λ l 2 , j = k , − λ l 2 / ( b − 1 ) , j ≠ k , (120) \left(\mu_{u^{(j)}}-\mu_u\right)^{\mathsf{T}}\left(\mu_{u^{(k)}}-\mu_u\right)
=
\begin{cases}
\lambda_l^2, & j=k,\\
-\lambda_l^2/(b-1), & j\ne k,
\end{cases} \tag*{(120)} ( μ u ( j ) − μ u ) T ( μ u ( k ) − μ u ) = { λ l 2 , − λ l 2 / ( b − 1 ) , j = k , j = k , (120)
for a scale λ l > 0 \lambda_l > 0 λ l > 0 chosen below.
(R3) Cone half-angle around the parent. For every j = 1 , … , b j=1,\ldots,b j = 1 , … , b ,
∠ ( μ u ( j ) , μ u ) = θ l . (121) \angle\left(\mu_{u^{(j)}},\mu_u\right)=\theta_l. \tag*{(121)} ∠ ( μ u ( j ) , μ u ) = θ l . (121)
Feasible subspace and depth–dimension condition
The ancestor orthogonality conditions in (R1) intersect in the affine subspace
V u : = { x ∈ R d : A u ⊤ x = z u } , z u : = [ ∥ μ π 0 ( u ) ∥ 2 , … , ∥ μ π l ( u ) ∥ 2 ] ⊤ . (122) \mathcal{V}_u := \left\{x \in\mathbb{R}^d : A_u^\top x = z_u\right\}, \qquad z_u := \left[\left\lVert\mu_{\pi_0(u)}\right\rVert^2,\ldots,\left\lVert\mu_{\pi_l(u)}\right\rVert^2\right]^\top. \tag*{(122)} V u := { x ∈ R d : A u ⊤ x = z u } , z u := [ μ π 0 ( u ) 2 , … , μ π l ( u ) 2 ] ⊤ . (122)
When the ancestor prototypes are linearly independent, dim V u = d − ( l + 1 ) \dim\mathcal{V}_u = d - (l+1) dim V u = d − ( l + 1 ) . Embedding a regular ( b − 1 ) (b-1) ( b − 1 ) -simplex inside V u \mathcal{V}_u V u for every internal node up to depth L − 1 L-1 L − 1 therefore requires the depth–dimension condition
d ≥ L + b − 1 , (123) d \ge L+b-1, \tag*{(123)} d ≥ L + b − 1 , (123)
which makes the trade-off explicit: L L L dimensions are absorbed by the ancestor-orthogonality constraints down to depth L − 1 L-1 L − 1 , while the regular ( b − 1 ) (b-1) ( b − 1 ) -simplex of siblings requires b − 1 b-1 b − 1 additional dimensions [9 ] .
Limits on hierarchy depth. Eq. (123) is necessary but not sufficient. Because each child is separated from its parent by a fixed angle θ l \theta_l θ l (see (R3)), the children’s cones must be nested inside their parent’s cone. This, in turn, forces the half-angles { θ l } l = 0 L − 1 \{\theta_l\}_{l=0}^{L-1} { θ l } l = 0 L − 1 to shrink monotonically with depth and to satisfy a recursive feasibility constraint coupling θ l \theta_l θ l , θ l − 1 \theta_{l-1} θ l − 1 , and the branching factor b b b . As L L L increases, the admissible cone angles collapse and the inter-prototype margin γ H A S \gamma_{\mathit{HAS}} γ HAS degrades, establishing an effective upper bound on the depth L L L for a fixed embedding dimension d d d while preserving H A S \mathit{HAS} HAS . For the full analysis on the precise depth-angle-dimension trade-off and the resulting bound on L L L , we refer the reader to Nguyen et al. [9 ] . The dimension condition in Eq. (123) makes the orthogonal simplex construction feasible, but a positive H A S \mathit{HAS} HAS margin also requires control of the angles across depths. In Proposition I.1, we give a sufficient decay schedule for this construction.
Construction
We construct { μ u ( j ) } j = 1 b \{\mu_{u(j)}\}_{j=1}^{b} { μ u ( j ) } j = 1 b in five steps.
Origin in V u \mathcal{V}_u V u [ for (R1) ]. Take the parent itself, g u : = μ u g_u := \mu_u g u := μ u . Hierarchical orthogonality of μ u \mu_u μ u to all its ancestors (inductively) gives μ u ⊤ μ π k ( u ) = ∥ μ π k ( u ) ∥ 2 \mu_u^\top\mu_{\pi_k(u)} = \left\lVert\mu_{\pi_k(u)}\right\rVert^2 μ u ⊤ μ π k ( u ) = μ π k ( u ) 2 for every k ≤ l k \le l k ≤ l , so g u ∈ V u g_u \in\mathcal{V}_u g u ∈ V u .
Orthonormal basis of the offset directions [ for (R1) ]. Because d − l − 1 ≥ b − 1 d-l-1 \ge b-1 d − l − 1 ≥ b − 1 for every internal node at depth l ≤ L − 1 l \le L-1 l ≤ L − 1 by Eq. (123) , choose U u ∈ R d × ( b − 1 ) U_u \in\mathbb{R}^{d\times(b-1)} U u ∈ R d × ( b − 1 ) with A u ⊤ U u = 0 A_u^\top U_u = 0 A u ⊤ U u = 0 and U u ⊤ U u = I b − 1 U_u^\top U_u = I_{b-1} U u ⊤ U u = I b − 1 , e.g. by taking any b − 1 b-1 b − 1 left singular vectors spanning ker ( A u ⊤ ) \operatorname{ker}(A_u^\top) ker ( A u ⊤ ) . Any vector in g u + range ( U u ) g_u+\operatorname{range}(U_u) g u + range ( U u ) lies in V u \mathcal{V}_u V u and therefore satisfies (R1).
Canonical regular simplex in R b − 1 \mathbb{R}^{b-1} R b − 1 [ for (R2) ]. Choose vectors d ~ 1 , … , d ~ b ∈ R b − 1 \tilde{d}_1,\ldots,\tilde{d}_b \in\mathbb{R}^{b-1} d ~ 1 , … , d ~ b ∈ R b − 1 forming a centered regular simplex:
∑ j = 1 b d ~ j = 0 , d ~ j ⊤ d ~ k = 1 [ j = k ] − 1 b , j , k ∈ [ b ] . (124) \sum_{j=1}^{b}\tilde{d}_j = 0, \qquad\tilde{d}_j^\top\tilde{d}_k = \mathbf{1}[j=k]-\frac{1}{b}, \qquad j,k \in[b]. \tag*{(124)} j = 1 ∑ b d ~ j = 0 , d ~ j ⊤ d ~ k = 1 [ j = k ] − b 1 , j , k ∈ [ b ] . (124)
Scale to enforce the cone condition [ for (R3) ]. Set
λ l : = ∥ μ u ∥ tan θ l , d j : = λ l ( b − 1 ) / b d ~ j , j ∈ [ b ] , (125) \lambda_l := \left\lVert\mu_u\right\rVert\tan\theta_l, \qquad d_j := \frac{\lambda_l}{\sqrt{(b-1)/b}}\tilde{d}_j, \qquad j \in[b], \tag*{(125)} λ l := ∥ μ u ∥ tan θ l , d j := ( b − 1 ) / b λ l d ~ j , j ∈ [ b ] , (125)
so that ∥ d j ∥ = λ l \left\lVert d_j\right\rVert= \lambda_l ∥ d j ∥ = λ l and d j ⊤ d k = − λ l 2 / ( b − 1 ) d_j^\top d_k = -\lambda_l^2/(b-1) d j ⊤ d k = − λ l 2 / ( b − 1 ) for j ≠ k j \ne k j = k , satisfying (R2).
Embed and translate.
μ u ( j ) : = μ u + U u d j , j = 1 , … , b . (126) \mu_{u(j)} := \mu_u+U_ud_j, \qquad j=1,\ldots,b. \tag*{(126)} μ u ( j ) := μ u + U u d j , j = 1 , … , b . (126)
Properties
Compatibility with the assumptions. Iterating the construction from the root to all leaves gives a set of prototypes { μ k } k ∈ V \{\mu_k\}_{k\in V} { μ k } k ∈ V where (i) for every non-root node k k k and strict ancestor p p p , the displacement μ k − μ p \mu_k-\mu_p μ k − μ p is orthogonal to μ p \mu_p μ p , and (ii) the children of any internal node form a centered regular simplex around their parent. Property (i) yields ∥ μ k ∥ > 0 \left\lVert\mu_k\right\rVert>0 ∥ μ k ∥ > 0 for every non-root k k k provided the depth-wise scales ∥ μ u ∥ tan θ l \left\lVert\mu_u\right\rVert\tan\theta_l ∥ μ u ∥ tan θ l are positive, so ρ min > 0 \rho_{\min}>0 ρ m i n > 0 . Under the angle schedule in Proposition I.1, the prototypes also have γ H A S > 0 \gamma_{\mathit{HAS}}>0 γ HAS > 0 . Property (ii) is exactly the centered-simplex hypothesis used in Lemma C.4, so the descendant average of leaf prototypes equals the ancestor prototype. As this holds for every internal node, the resulting set of prototypes is consistent with Assumption 4.4.
The centroid property holds for any choice of the angles, whereas a positive H A S \mathcal{H}AS H A S margin needs separation between descendants and off-branch nodes.
Proposition I.1 (Positive H A S \mathcal{H}AS H A S margin under geometric angle decay). Suppose the prototypes are constructed as in Appendix I.3 with 0 < θ 0 < π / 2 0 < \theta_0 < \pi/2 0 < θ 0 < π /2 and some q q q such that
0 < q ≤ 1 / 4 , (127) 0 < q \le1/4, \tag*{(127)} 0 < q ≤ 1/4 , (127) θ r + 1 ≤ q θ r ( r ∈ { 0 , … , L − 2 } ) . (128) \theta_{r+1} \le q\theta_r \qquad(r \in\{0,\ldots,L-2\}). \tag*{(128)} θ r + 1 ≤ q θ r ( r ∈ { 0 , … , L − 2 }) . (128) Then the global margin γ H A S \gamma_{\mathcal{H}AS} γ H A S of Eq. (40) is positive, so the prototype family satisfies H A S \mathcal{H}AS H A S .
Proof. Fix an admissible triplet ( u , v , w ) (u,v,w) ( u , v , w ) , let p p p be the lowest common ancestor of u u u and w w w , and write l : = d ( p ) l := d(p) l := d ( p ) . The nodes u u u and w w w lie below distinct children a a a and a ′ a' a ′ of p p p . By Eqs. (120) and (121) , their sibling angle α l : = ∠ ( μ a , μ a ′ ) \alpha_l := \angle(\mu_a,\mu_{a'}) α l := ∠ ( μ a , μ a ′ ) satisfies
cos α l = μ a ⊤ μ a ′ ∥ μ a ∥ ∥ μ a ′ ∥ (definition of α l ) = ∥ μ p ∥ 2 + ( μ a − μ p ) ⊤ ( μ a ′ − μ p ) ∥ μ p ∥ 2 + λ l 2 (by Eqs. (115) and (116)) = ∥ μ p ∥ 2 − λ l 2 / ( b − 1 ) ∥ μ p ∥ 2 + λ l 2 (by Eq. (116)) = 1 − λ l 2 / ( ( b − 1 ) ∥ μ p ∥ 2 ) 1 + λ l 2 / ∥ μ p ∥ 2 (dividing by ∥ μ p ∥ 2 ) = 1 − tan 2 θ l / ( b − 1 ) 1 + tan 2 θ l (by Eq. (121)) = cos 2 θ l − sin 2 θ l b − 1 (multiplying by cos 2 θ l ) < cos θ l (since 0 < cos θ l < 1 and b ≥ 2 ) ⟺ α l > θ l (since cosine decreases on [ 0 , π ] ). (129) \begin{aligned}
\cos\alpha_l
&= \frac{\mu_a^{\top}\mu_{a'}}{\lVert\mu_a\rVert\lVert\mu_{a'}\rVert}
&&\text{(definition of }\alpha_l\text{)}\\
&= \frac{\lVert\mu_p\rVert^2+(\mu_a-\mu_p)^{\top}(\mu_{a'}-\mu_p)}{\lVert\mu_p\rVert^2+\lambda_l^2}
&&\text{(by Eqs. (115) and (116))}\\
&= \frac{\lVert\mu_p\rVert^2-\lambda_l^2/(b-1)}{\lVert\mu_p\rVert^2+\lambda_l^2}
&&\text{(by Eq. (116))}\\
&= \frac{1-\lambda_l^2/((b-1)\lVert\mu_p\rVert^2)}{1+\lambda_l^2/\lVert\mu_p\rVert^2}
&&\text{(dividing by }\lVert\mu_p\rVert^2\text{)}\\
&= \frac{1-\tan^2\theta_l/(b-1)}{1+\tan^2\theta_l}
&&\text{(by Eq. (121))}\\
&= \cos^2\theta_l-\frac{\sin^2\theta_l}{b-1}
&&\text{(multiplying by }\cos^2\theta_l\text{)}\\
&< \cos\theta_l
&&\text{(since }0<\cos\theta_l<1\text{ and }b\ge2\text{)}\\
&\Longleftrightarrow\alpha_l>\theta_l
&&\text{(since cosine decreases on }[0,\pi]\text{).} \tag*{(129)}
\end{aligned} cos α l = ∥ μ a ∥ ∥ μ a ′ ∥ μ a ⊤ μ a ′ = ∥ μ p ∥ 2 + λ l 2 ∥ μ p ∥ 2 + ( μ a − μ p ) ⊤ ( μ a ′ − μ p ) = ∥ μ p ∥ 2 + λ l 2 ∥ μ p ∥ 2 − λ l 2 / ( b − 1 ) = 1 + λ l 2 / ∥ μ p ∥ 2 1 − λ l 2 / (( b − 1 ) ∥ μ p ∥ 2 ) = 1 + tan 2 θ l 1 − tan 2 θ l / ( b − 1 ) = cos 2 θ l − b − 1 sin 2 θ l < cos θ l ⟺ α l > θ l (definition of α l ) (by Eqs. (115) and (116)) (by Eq. (116)) (dividing by ∥ μ p ∥ 2 ) (by Eq. (121)) (multiplying by cos 2 θ l ) (since 0 < cos θ l < 1 and b ≥ 2 ) (since cosine decreases on [ 0 , π ] ). (129)
Set T l : = ∑ j = l + 1 L − 1 θ j T_l := \sum_{j=l+1}^{L-1}\theta_j T l := ∑ j = l + 1 L − 1 θ j . Applying the angle triangle inequality first through u u u and then through w w w gives
α l ≤ ∠ ( μ a , μ u ) + ∠ ( μ u , μ a ′ ) ≤ ∠ ( μ a , μ u ) + ∠ ( μ u , μ w ) + ∠ ( μ w , μ a ′ ) . \begin{aligned}
\alpha_l &\le\angle(\mu_a,\mu_u)+\angle(\mu_u,\mu_{a'})\\
&\le\angle(\mu_a,\mu_u)+\angle(\mu_u,\mu_w)+\angle(\mu_w,\mu_{a'}).
\end{aligned} α l ≤ ∠ ( μ a , μ u ) + ∠ ( μ u , μ a ′ ) ≤ ∠ ( μ a , μ u ) + ∠ ( μ u , μ w ) + ∠ ( μ w , μ a ′ ) .
By Eq. (121) and repeated angle triangle inequalities along the three ancestor–descendant paths,
∠ ( μ u , μ a ) ≤ ∑ j = l + 1 d ( u ) − 1 θ j ≤ T l , ∠ ( μ w , μ a ′ ) ≤ ∑ j = l + 1 d ( w ) − 1 θ j ≤ T l , ∠ ( μ u , μ v ) ≤ ∑ j = d ( u ) d ( v ) − 1 θ j ≤ T l . \begin{aligned}
\angle(\mu_u,\mu_a)&\le\sum_{j=l+1}^{d(u)-1}\theta_j\le T_l,\\
\angle(\mu_w,\mu_{a'})&\le\sum_{j=l+1}^{d(w)-1}\theta_j\le T_l,\\
\angle(\mu_u,\mu_v)&\le\sum_{j=d(u)}^{d(v)-1}\theta_j\le T_l.
\end{aligned} ∠ ( μ u , μ a ) ∠ ( μ w , μ a ′ ) ∠ ( μ u , μ v ) ≤ j = l + 1 ∑ d ( u ) − 1 θ j ≤ T l , ≤ j = l + 1 ∑ d ( w ) − 1 θ j ≤ T l , ≤ j = d ( u ) ∑ d ( v ) − 1 θ j ≤ T l .
Rearranging the sibling-angle bound and applying these path bounds gives
∠ ( μ u , μ w ) ≥ α l − ∠ ( μ u , μ a ) − ∠ ( μ w , μ a ′ ) ≥ α l − T l − ∠ ( μ w , μ a ′ ) (since ∠ ( μ u , μ a ) ≤ T l ) ≥ α l − 2 T l (since ∠ ( μ w , μ a ′ ) ≤ T l ) ∠ ( μ u , μ w ) − ∠ ( μ u , μ v ) ≥ α l − 2 T l − ∠ ( μ u , μ v ) ≥ α l − 3 T l (since ∠ ( μ u , μ v ) ≤ T l ). (130) \begin{aligned}
\angle(\mu_u,\mu_w)
&\ge\alpha_l-\angle(\mu_u,\mu_a)-\angle(\mu_w,\mu_{a'})\\
&\ge\alpha_l-T_l-\angle(\mu_w,\mu_{a'})
&&\text{(since }\angle(\mu_u,\mu_a)\le T_l\text{)}\\
&\ge\alpha_l-2T_l
&&\text{(since }\angle(\mu_w,\mu_{a'})\le T_l\text{)}\\
\angle(\mu_u,\mu_w)-\angle(\mu_u,\mu_v)
&\ge\alpha_l-2T_l-\angle(\mu_u,\mu_v)\\
&\ge\alpha_l-3T_l
&&\text{(since }\angle(\mu_u,\mu_v)\le T_l\text{).} \tag*{(130)}
\end{aligned} ∠ ( μ u , μ w ) ∠ ( μ u , μ w ) − ∠ ( μ u , μ v ) ≥ α l − ∠ ( μ u , μ a ) − ∠ ( μ w , μ a ′ ) ≥ α l − T l − ∠ ( μ w , μ a ′ ) ≥ α l − 2 T l ≥ α l − 2 T l − ∠ ( μ u , μ v ) ≥ α l − 3 T l (since ∠ ( μ u , μ a ) ≤ T l ) (since ∠ ( μ w , μ a ′ ) ≤ T l ) (since ∠ ( μ u , μ v ) ≤ T l ). (130)
It remains to show that 3 T l < α l 3T_l < \alpha_l 3 T l < α l . Since q > 0 q > 0 q > 0 by Eq. (127) , iterating Eq. (128) gives θ l + m ≤ q m θ l \theta_{l+m} \le q^m\theta_l θ l + m ≤ q m θ l for every m ∈ { 1 , … , L − l − 1 } m \in\{1,\ldots,L-l-1\} m ∈ { 1 , … , L − l − 1 } . Thus
3 T l = 3 ∑ j = l + 1 L − 1 θ j (definition of T l ) ≤ 3 θ l ∑ m = 1 L − l − 1 q m (by xref‘Eq. (124)‘) ≤ 3 θ l ∑ m = 1 ∞ q m (since q > 0 by xref‘Eq. (123)‘) = 3 q 1 − q θ l (geometric series) ≤ θ l (since q ≤ 1 / 4 by xref‘Eq. (123)‘) < α l (by xref‘Eq. (125)‘). (131) \begin{aligned}
3T_l &= 3\sum_{j=l+1}^{L-1}\theta_j && \text{(definition of }T_l\text{)}\\
&\le3\theta_l\sum_{m=1}^{L-l-1}q^m && \text{(by {xref}`Eq. (124)`)}\\
&\le3\theta_l\sum_{m=1}^{\infty}q^m && \text{(since }q > 0\text{ by {xref}`Eq. (123)`)}\\
&= \frac{3q}{1-q}\theta_l && \text{(geometric series)}\\
&\le\theta_l && \text{(since }q \le1/4\text{ by {xref}`Eq. (123)`)}\\
&< \alpha_l && \text{(by {xref}`Eq. (125)`).} \tag*{(131)}
\end{aligned} 3 T l = 3 j = l + 1 ∑ L − 1 θ j ≤ 3 θ l m = 1 ∑ L − l − 1 q m ≤ 3 θ l m = 1 ∑ ∞ q m = 1 − q 3 q θ l ≤ θ l < α l (definition of T l ) (by xref ‘Eq. (124)‘) (since q > 0 by xref ‘Eq. (123)‘) (geometric series) (since q ≤ 1/4 by xref ‘Eq. (123)‘) (by xref ‘Eq. (125)‘). (131)
Thus the angle margin is positive. Cosine decreases with angle on [ 0 , π ] [0,\pi] [ 0 , π ] , so the triplet margin is positive. Since L ≥ 2 L \ge2 L ≥ 2 and b ≥ 2 b \ge2 b ≥ 2 , admissible triplets exist, and their margin is positive.
Additional Experiments
All experiments are run on a server equipped with NVIDIA RTX A5000 GPUs (24 GB memory each). The CLIP empirical evaluations of § 3.2 use publicly released CLIP checkpoints over the ImageNet-1K training set and the WordNet hierarchy. The synthetic experiments of Appendix § 16 .§ 3 run on a single A5000.
Hierarchical Angular Separation on CLIP embeddings
Controls and robustness checks for approximate image-side H A S \mathcal{H}AS H A S
Table 1 measures every representation under the disjoint-split protocol of Appendix § 16 .§ 2 : we partition the images in each ImageNet-1K leaf into even and odd halves, estimate every node embedding from each half separately by descendant-leaf-weighted instance aggregation, and average H A S \mathcal{H}AS H A S -Viol over the two directions. For raw pixels, we apply CLIP’s resize, center-crop, and channel-normalization preprocessing without an encoder and flatten each image into a 150,528-dimensional vector. The Gaussian control draws independent isotropic Gaussian vectors in R 768 \mathbb{R}^{768} R 768 , normalizes each to unit length, and uses the same per-leaf sample counts. Every condition uses the WordNet hierarchy, anchor set, and 95th-percentile descendant threshold of § 3.2 , and its hierarchy-shuffled null reassigns leaf groups within that hierarchy while preserving the representation and the aggregation procedure.
For random ViT, the original-assignment value reports mean ± \pm ± sample standard deviation over three weight initializations, while the null pools five shuffles per initialization. For Gaussian vectors, the original-assignment value reports mean ± \pm ± standard deviation over 40 independent draws, while the null pools five shuffles for each of the first five draws. The raw-pixel and pretrained CLIP nulls use five shuffles of their fixed representations.
The randomly initialized row of Table 1 averages three ViT-L/14 random initializations, so it leaves open whether approximate H A S \mathcal{H}AS H A S at random initialization depends on the architecture. We therefore repeat the measurement across visual backbones, with at least three independent random initializations each.
Table 2. Approximate image-side H A S \mathcal{H}AS H A S at random initialization across visual backbones, measured with the disjoint-split protocol. Each backbone is evaluated with S S S independent random initializations, and both columns report the mean ± \pm ± sample standard deviation across those samplings; each sampling’s null is itself a mean over five hierarchy shuffles. The ViT-L/14 row is the randomly initialized row of Table 1 .
Across architectures and random seeds, the violation percentage of every backbone is far below its own shuffled null.
Robustness to the prototype construction and the taxonomy
Image-side H A S \mathcal{H}AS H A S could depend on two choices in the setup of § 3.2 : the descendant-leaf weighting used to build prototypes, and the WordNet taxonomy itself. We repeat the disjoint-split measurement with each choice replaced in turn. First, we replace the descendant-leaf mean by a uniform mean over child prototypes, so that every child contributes equally regardless of how many images it has. Second, we replace ImageNet-1K and WordNet by CUB-200 [48 ] with its own bird taxonomy, an independent hierarchy of different depth and branching. Chance level depends on both choices, so we compare each condition against its own hierarchy-shuffled null rather than against a common baseline.
Table 3. Image-side H A S \mathcal{H}AS H A S is robust to the prototype construction and to the taxonomy. Each condition is paired with its own hierarchy-shuffled null (mean ± \pm ± standard deviation over five shuffles). The first row restates the ViT-L/14 measurements of Table 1 at the precision used there; its randomly initialized entries are the seed-0 initialization rather than the mean over three initializations reported there.
In all three conditions, the violation percentage of the pretrained encoder is roughly an order of magnitude below its own null, while that of the randomly initialized encoder is roughly half of its null. Neither the prototype construction nor the choice of taxonomy therefore changes the conclusion: pretraining accounts for roughly the same share of the achievable separation in every setting, and approximate separation at random initialization appears in every setting.
Instance aggregation on the text side (ViT-L/14). Figure 6 repeats the text experiment of § 3.2 estimating concept embeddings by instance aggregation instead of direct concept encoding, again on ViT-L-14, and reaches the same conclusion: the median violation percentages are concentrated near zero for both pretrained models and substantially larger for the randomly initialized model.
Figure 6. H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S violation percentage by depth on ViT-L-14 CLIP text embeddings (instance aggregation): H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S appears with pretraining.
ViT-H/14 results. We further repeat the experiment on ViT-H/14, comparing OpenCLIP trained on DFN-5B [52 ] against a randomly initialized ViT-H/14. Figure 7 groups text embeddings (top; instance aggregation) and image embeddings (bottom; instance aggregation) and reproduces the same qualitative pattern.
Figure 7. H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S violation percentage by depth on ViT-H/14 CLIP embeddings. Top : text embeddings (instance aggregation) – H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S appears with pretraining. Bottom : image embeddings (instance aggregation): approximate H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S appears at random initialization and strengthens after pretraining.
ViT-B/16 results. We additionally repeat the main-text experiment on a smaller backbone, ViT-B/16, comparing OpenAI CLIP [1 ] and OpenCLIP [44 ] trained on DataComp-1B [45 ] against a randomly initialized ViT-B/16. Figure 8 (text via direct concept encoding on top, image via instance aggregation on bottom) and Figure 9 (text, instance aggregation) reproduce the same qualitative pattern as on ViT-L/14: text H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S appears with pretraining, while image embeddings show approximate H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S at random initialization.
Figure 8. H A S \mathcal{H}_{AS} H A S violation percentage by depth on ViT-B/16 CLIP embeddings. Top : text embeddings (direct concept encoding): H A S \mathcal{H}_{AS} H A S appears with pretraining. Bottom : image embeddings (instance aggregation): approximate H A S \mathcal{H}_{AS} H A S appears at random initialization and strengthens after pretraining.
Figure 9. H A S \mathcal{H}_{AS} H A S violation percentage by depth on ViT-B/16 CLIP text embeddings (instance aggregation): H A S \mathcal{H}_{AS} H A S appears with pretraining.
ViT-B/32 results. We further repeat the experiment on ViT-B/32, comparing OpenAI CLIP and OpenCLIP (DataComp-1B) against a randomly initialized ViT-B/32. Figure 10 groups text emembeddings (top; direct concept encoding) and image embeddings (bottom; instance aggregation) and reproduces the same qualitative pattern.
Figure 10. H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S violation percentage by depth on ViT-B/32 CLIP embeddings. Top: text embeddings (direct concept encoding): H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S appears with pretraining. Bottom: image embeddings (instance aggregation): approximate H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S appears at random initialization and strengthens after pretraining.
ViT-L/14@336px results. We also evaluate the higher-resolution ViT-L/14@336px backbone, for which only the OpenAI checkpoint ships at 336px in OpenCLIP; we compare it against a randomly initialized ViT-L/14@336px. Figure 11 groups text embeddings (top; direct concept encoding) and image embeddings (bottom; instance aggregation) and reproduces the same qualitative pattern.
Figure 11. H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S violation percentage by depth on ViT-L/14@336px CLIP embeddings. Top: text embeddings (direct concept encoding): H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S appears with pretraining. Bottom: image embeddings (instance aggregation): approximate H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S appears at random initialization and strengthens after pretraining.
SigLIP2 results. We evaluate SigLIP2 (ViT-B/16 at 256px), comparing the pretrained SigLIP2 checkpoint against a randomly initialized ViT-B/16 at 256px. Figure 12 (text via direct concept encoding on top, image via instance aggregation on bottom) reproduces the same qualitative pattern: text H A S \mathcal{H}_{AS} H A S appears with pretraining, while image embeddings show approximate H A S \mathcal{H}_{AS} H A S at random initialization.
Figure 12. H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S violation percentage by depth on SigLIP2 embeddings. Top : text embeddings (direct concept encoding): H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S appears with pretraining. Bottom : image embeddings (instance aggregation): approximate H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S appears at random initialization and strengthens after pretraining.
MetaCLIP2 results. We evaluate MetaCLIP2 (ViT-H/14 at 224px), comparing the pretrained MetaCLIP2 worldwide checkpoint against a randomly initialized ViT-H/14. Figure 13 (text via direct concept encoding on top, image via instance aggregation on bottom) reproduces the same qualitative pattern: text H A S \mathcal{H}_{AS} H A S appears with pretraining, while image embeddings show approximate H A S \mathcal{H}_{AS} H A S at random initialization.
Figure 13. H A S \mathcal{H}AS H A S violation percentage by depth on MetaCLIP2 embeddings. Top: text embeddings (direct concept encoding): H A S \mathcal{H}AS H A S appears with pretraining. Bottom: image embeddings (instance aggregation): approximate H A S \mathcal{H}AS H A S appears at random initialization and strengthens after pretraining.
Disjoint-Split Control against Instance Overlap for H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S in Image Embeddings
In hierarchical instance aggregation, ancestor prototypes naturally share image instances with their descendants, creating an inherent source of correlated sampling noise. When prototypes are estimated by averaging instance vectors over descendant leaves, every image contributing to a descendant node also enters the sample mean of all its ancestors. This shared sample overlap inflates anchor–descendant cosine similarities and narrows the apparent descendant cone, potentially producing low H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S violation rates even in representations lacking semantic hierarchy. Because standard hierarchy shuffling permutes leaf assignments across the tree while preserving this internal parent–descendant overlap, a shuffled null alone cannot detect this artifact.
To eliminate sample overlap while preserving tree aggregation, we evaluate H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S using a split-sample protocol. For each ImageNet training leaf, we partition the available images into even and odd splits, rebuilding all internal node prototypes independently within each split. We then evaluate anchor prototypes from one split against descendant and off-branch prototypes from the other split, and average H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol across both split directions. Under this procedure, every anchor prototype and its comparison targets are estimated from strictly disjoint sets of images. We employ this disjoint-split protocol for every representation and its matched hierarchy-shuffled null in Table 1 .
To measure the precise impact of instance overlap, we conduct a paired experiment comparing shared-instance and disjoint-split evaluations alongside an uninformative Gaussian baseline. In Table 4 , we evaluate raw pixels, randomly initialized ViT-L/14, pretrained CLIP, and isotropic Gaussian noise vectors (z j ∼ N ( 0 , I d ) z_j \sim\mathcal{N}(0, I_d) z j ∼ N ( 0 , I d ) , normalized to unit length). Because Gaussian vectors are distributed independently of the semantic taxonomy, any apparent separation under shared instances isolates the purely geometric artifact of sample reuse, while their behavior under disjoint splits establishes the uninformative chance level. For random ViT and Gaussian vectors, reported values and spreads indicate means and sample standard deviations across three weight initializations and 40 independent Gaussian draws, respectively; raw pixels and pretrained CLIP report single evaluations paired with five hierarchy permutations.
Table 4. Mean image-side H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol (%) under shared-instance versus disjoint-split instance aggregation, paired with matched hierarchy-shuffled nulls. Spreads (±) denote sample standard deviations across random seeds or permutations (details in text). Disjoint splitting eliminates the apparent separation of the Gaussian noise baseline while preserving the hierarchical geometry of genuine visual representations.
The paired results in Table 4 demonstrate that instance overlap artificially induces apparent separation for unstructured noise, but does not account for the hierarchical structure observed in visual representations. For isotropic Gaussian vectors, shared instance aggregation produces an artificially low violation rate (1.19%), which is indistinguishable from its matched shuffled null (1.19%). Under disjoint splits, where sample overlap is eliminated, Gaussian violations jump to chance level (76.02%), again matching the shuffled baseline (76.24%) and unmasking the shared-instance artifact. In sharp contrast, all three visual representations (raw pixels, randomly initialized ViT-L/14, and pretrained CLIP) remain largely unaffected by disjoint splitting, maintaining violation rates far below their respective shuffled nulls under both protocols.
This geometric stability holds across all hierarchy levels, ruling out sample overlap as the source of approximate image-side H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S . In Figure 14 , the depth-resolved violation profiles for ViT-L/14 under disjoint splits mirror those under shared-instance aggregation across all depths. While the disjoint-split protocol eliminates sample sharing between anchor and comparison prototypes, parent prototypes remain averages over descendant leaf distributions. The measured separation therefore reflects genuine consistency across visual subtrees and the hierarchical structure of the data distribution, rather than a statistical artifact of instance reuse.
(a) Randomly initialized.
(b) OpenAI.
(c) OpenCLIP (DataComp-1B).
(d) Randomly initialized.
(e) OpenAI.
(f) OpenCLIP (DataComp-1B).
Figure 14. H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S violation percentages by depth on ViT-L/14 CLIP image embeddings remain similar under the disjoint-split control. Top: disjoint-split control, where parent prototypes and comparison prototypes are estimated from disjoint image instances. Bottom: original instance-aggregation evaluation.
:::
Numerical Validation of H A S \mathcal{H}AS H A S Transfer at Small Regularization
In Theorem 4.9, we guarantee approximate H A S \mathcal{H}AS H A S transfer only for sufficiently large λ \lambda λ , but leave open whether transfer persists outside the asymptotic regime used in its proof. We test this possibility on a synthetic image-prototype hierarchy by minimizing Eq. (9) across a sweep of λ \lambda λ . We verify whether transfer happens outside the large-λ \lambda λ regime, whether the optimal text embeddings approach the root-centered prototype decomposition of Corollary 4.8 as λ \lambda λ increases, and whether the sufficient transfer condition is conservative. We report H A S \mathcal{H}AS H A S -Viol at the minimizer found by L-BFGS and define prototype alignment as the minimum, over non-root nodes k k k , of the cosine between a k n c e , ∗ a_k^{\mathrm{nce},*} a k nce , ∗ and μ k \mu_k μ k .
Experimental setup. For convenience, we restate the data generating process from Assumption 4.1 as
z j = μ y j + ε j , E [ ε j ] = 0 , j ∈ [ N ] . (132) z_j = \mu_{y_j} + \varepsilon_j,\qquad\mathbb{E}[\varepsilon_j] = 0,\quad j \in[N]. \tag*{(132)} z j = μ y j + ε j , E [ ε j ] = 0 , j ∈ [ N ] . (132)
We instantiate Eq. (132) with isotropic Gaussian variation ε j ∼ N ( 0 , σ 2 I d ) \varepsilon_j \sim\mathcal{N}(0,\sigma^2 I_d) ε j ∼ N ( 0 , σ 2 I d ) at depth L = 5 L = 5 L = 5 , branching factor 3, variation parameter σ = 10 − 5 \sigma= 10^{-5} σ = 1 0 − 5 , feature-dimension-to-number-of-nodes ratio d / ∣ V ∣ = 1 d/|V| = 1 d /∣ V ∣ = 1 , and 150 images per leaf, using prototypes generated by geometric-decay sibling angles (θ 0 = 85 ∘ \theta_0 = 85^\circ θ 0 = 8 5 ∘ ). We optimize the symmetric InfoNCE objective (temperature 1, no learnable temperature) with L-BFGS to find the global minimizer. We assign captions deterministically, cycling each leaf’s images through its root-to-leaf path; we call this the balanced-caption setup.
Transfer precedes the asymptotic regime. Sweeping λ ∈ [ 10 − 5 , 10 2 ] \lambda\in[10^{-5}, 10^2] λ ∈ [ 1 0 − 5 , 1 0 2 ] separates the onset of transfer from the accuracy of the large-λ \lambda λ expansion. H A S \mathcal{H}AS H A S -Viol reaches zero before prototype alignment saturates (Figure 5 b). The convergence trace confirms that zero H A S \mathcal{H}AS H A S -Viol holds at the minimizer found by L-BFGS rather than only at an early iterate (Figure 5 a).
The large-λ \lambda λ expansion predicts the limiting geometry. Prototype alignment increases with λ \lambda λ and approaches one in the large-λ \lambda λ regime (Figure 5 b). The trend is consistent with the directional prediction of Corollary 4.8 , while the earlier disappearance of violations shows that transfer does not require near-perfect prototype alignment.
The sufficient transfer condition is conservative. Under Theorem 4.9, transfer is certified when D ε n c e < ρ min γ H A S / 16 D_{\varepsilon}^{\mathrm{nce}} < \rho_{\min}\gamma_{\mathcal{H}AS}/16 D ε nce < ρ m i n γ H A S /16 . Both tested root norms, ∥ μ r o o t ∥ ∈ { 10 − 4 , 10 − 2 } \lVert\mu_{\mathrm{root}}\rVert\in\{10^{-4},10^{-2}\} ∥ μ root ∥ ∈ { 1 0 − 4 , 1 0 − 2 } , violate this condition yet produce nearly the same transition (Figure 5 b and Figure 15 ). At fixed λ = 10 − 3 \lambda= 10^{-3} λ = 1 0 − 3 , violations remain small across the root-norm sweep and increase only under substantially larger shifts (Figure 16 ). Across the noise sweep at λ ∈ { 10 − 3 , 10 − 2 , 10 − 1 } \lambda\in\{10^{-3},10^{-2},10^{-1}\} λ ∈ { 1 0 − 3 , 1 0 − 2 , 1 0 − 1 } , degradation appears first under the weakest regularization and later under all three regularization values (Figure 17 ). We therefore find that the theorem’s transfer condition is sufficient but not necessary in this synthetic setting.
Figure 15. λ \lambda λ sweep at root norm ∥ μ r o o t ∥ = 10 − 2 \lVert\mu_{\mathrm{root}}\rVert= 10^{-2} ∥ μ root ∥ = 1 0 − 2 in the same balanced-caption setup as Figure 5 b. The x-axis is λ \lambda λ ; the left y-axis is the final H A S \mathcal{H}AS H A S -Viol averaged over non-leaf nodes, and the right y-axis is the final minimum prototype alignment over non-root nodes. Although this root norm violates the sufficient transfer condition D ϵ n c e < ρ min γ H A S / 16 D_{\epsilon}^{\mathrm{nce}} < \rho_{\min}\gamma_{\mathcal{H}AS}/16 D ϵ nce < ρ m i n γ H A S /16 by a factor of 161 on this set of prototypes, because D ϵ n c e ≥ ξ max ∥ μ r o o t ∥ D_{\epsilon}^{\mathrm{nce}} \geq\xi_{\max}\lVert\mu_{\mathrm{root}}\rVert D ϵ nce ≥ ξ m a x ∥ μ root ∥ and ξ max = 2.50 \xi_{\max} = 2.50 ξ m a x = 2.50 , the drop in violation percentage is nearly unchanged: it again reaches zero by λ = 10 − 3 \lambda= 10^{-3} λ = 1 0 − 3 , while prototype alignment saturates just below 1, reflecting the remaining root prototype μ r o o t \mu_{\mathrm{root}} μ root .
Figure 16. Root-norm sweep at λ = 10 − 3 \lambda= 10^{-3} λ = 1 0 − 3 in the same balanced-caption setup as Appendix § 16 .§ 3 . The x-axis is the root norm ∥ μ r o o t ∥ \lVert\mu_{\mathrm{root}}\rVert ∥ μ root ∥ , and the y-axis is the final H A S \mathcal{H}AS H A S -Viol averaged over non-leaf nodes. The violation percentage remains nearly flat at small root norm, then departs strongly from near-zero once the root norm is large, with non-monotone behavior in the extreme large-root-norm regime.
Figure 17. σ \sigma σ -robustness sweep in the same balanced-caption setup as Appendix § 16 .§ 3 , with root norm ∥ μ r o o t ∥ = 10 − 4 \lVert\mu_{\mathrm{root}}\rVert= 10^{-4} ∥ μ root ∥ = 1 0 − 4 and λ ∈ { 10 − 3 , 10 − 2 , 10 − 1 } \lambda\in\{10^{-3}, 10^{-2}, 10^{-1}\} λ ∈ { 1 0 − 3 , 1 0 − 2 , 1 0 − 1 } . Left: H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol. Middle: global minimum branch margin. Right: minimum non-root prototype alignment. All three regularization levels remain stable through σ = 3 × 10 − 3 \sigma= 3 \times10^{-3} σ = 3 × 1 0 − 3 . At σ = 10 − 2 \sigma= 10^{-2} σ = 1 0 − 2 , only λ = 10 − 3 \lambda= 10^{-3} λ = 1 0 − 3 has a nonzero violation percentage (0.014%) and a negative global minimum margin; the other two settings retain zero violations and positive margins. At σ = 3 × 10 − 2 \sigma= 3 \times10^{-2} σ = 3 × 1 0 − 2 , every λ \lambda λ has a negative global minimum margin and a violation percentage between 0.37% and 0.38%.
Predicted and Realized Slack on Real ViT-L/14 Prototypes
We use these experiments to compare the guarantee of Theorem 4.6 with the behavior of its exact linear-loss minimizer on real prototypes. We first evaluate the theorem’s threshold τ ϵ l i n = 8 D ϵ l i n / ρ min \tau_{\epsilon}^{\mathrm{lin}} = 8D_{\epsilon}^{\mathrm{lin}}/\rho_{\min} τ ϵ lin = 8 D ϵ lin / ρ m i n , then measure the realized transfer and explain the gap between them. Although the theorem’s bound gives no information in this setting, approximate H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S still transfers.
We use the ImageNet-1K / WordNet image prototypes of § 3.2 (1375 nodes and N = 1,281,167 N = 1{,}281{,}167 N = 1 , 281 , 167 training images) from three ViT-L/14 backbones: randomly initialized, OpenAI CLIP, and OpenCLIP trained on DataComp-1B. Holding the image prototypes fixed at Z Z Z , we compute the closed-form text-side minimizer A l i n , ∗ A_{\mathrm{lin},*} A lin , ∗ of Proposition D.1 . Each leaf prototype is its exact empirical image mean: we neither inject nor fit noise. The calculation in this subsection is therefore deterministic given the prototypes and caption counts.
The inequality required by the theorem does not hold. For this calculation, the reference vector is the empirical prototype itself, μ k = z ˉ k \mu_k = \bar{z}_k μ k = z ˉ k , taken as the descendant-leaf-weighted instance aggregation of § 3.2 . The stochastic part of the perturbation therefore vanishes, leaving e k = − ξ k l i n z ˉ e_k = -\xi_k^{\mathrm{lin}}\bar{z} e k = − ξ k lin z ˉ . The theorem’s error term consequently reduces to the root-centering contribution alone, D = ξ max l i n ∥ z ˉ ∥ D = \xi_{\max}^{\mathrm{lin}}\lVert\bar{z}\rVert D = ξ m a x lin ∥ z ˉ ∥ , where ξ k l i n ∈ [ 0.50 , 8.54 ] \xi_k^{\mathrm{lin}} \in[0.50, 8.54] ξ k lin ∈ [ 0.50 , 8.54 ] is fixed by the ImageNet caption counts and shared across encoders.
Using expected counts has a negligible effect on the prototypes or violation percentages. We use N k = ∑ ℓ n ℓ / ∣ path ( ℓ ) ∣ N_k = \sum_{\ell} n_\ell/ \lvert\operatorname{path}(\ell)\rvert N k = ∑ ℓ n ℓ / ∣ path ( ℓ )∣ under Assumption 4.3 rather than a sampled caption assignment, which makes the calculation deterministic. Under that assumption the empirical mean z ˉ k \bar{z}_k z ˉ k of the images captioned k k k weights each descendant leaf ℓ \ell ℓ by n ℓ / ∣ path ( ℓ ) ∣ n_\ell/\lvert\operatorname{path}(\ell)\rvert n ℓ / ∣ path ( ℓ )∣ instead of n ℓ n_\ell n ℓ . When we rebuild every internal prototype under that weighting, the prototypes and the text-side violation percentages of Table 6 remain essentially unchanged.
Table 6. Caption-permutation ablations for (a) the closed-form linear-loss text optimum on fixed real image prototypes and (b) the synthetic InfoNCE setting of Appendix § 16 .§ 3 . Permuted entries are mean ± standard deviation over five permutation seeds. (a) Real prototypes with the linear contrastive loss. (b) Synthetic prototypes with the InfoNCE loss.
For every backbone, the inequality D < ρ min / 4 D < \rho_{\min}/4 D < ρ m i n /4 required by Theorem 4.6 does not hold, so we obtain no guarantee from the theorem for these prototypes. Moreover, the resulting threshold exceeds the largest possible cosine-margin difference, which is 2, so the bound does not certify any triplet. The mismatch is structural: the stochastic term is zero here, and the universal constant C C C multiplies only that term.
The sufficient bound is loose, but approximate H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S still transfers. We therefore measure transfer descriptively rather than certify it. For each anchor, we find the smallest slack that satisfies Eq. (8), then report the largest of these anchor-specific values. This worst-anchor slack is the single post hoc slack needed for the inequality to hold at every anchor.
The quantity H A S -Viol τ i m g ( u ) \mathcal{H}\mathcal{A}\mathcal{S}\text{-Viol}_{\tau}^{\mathrm{img}}(u) H A S -Viol τ img ( u ) is Definition 3.2 evaluated on the image prototypes with the descendant radius s u μ s_u^{\mu} s u μ relaxed by τ \tau τ . The theoretical bound and the empirical metric of § 3.2 therefore have the same units: a percentage of off-branch nodes per anchor. The same bound holds for the percentile-based descendant radius used there for outlier robustness, provided both sides replace the minimum by the same order statistic, the r r r -th smallest descendant cosine for a fixed r r r . If w w w is counted on the text side, then at least r r r descendants v v v satisfy cos ( a ~ u , a ~ v ) ≤ cos ( a ~ u , a ~ w ) \cos(\tilde{a}_u,\tilde{a}_v) \le\cos(\tilde{a}_u,\tilde{a}_w) cos ( a ~ u , a ~ v ) ≤ cos ( a ~ u , a ~ w ) .
By part (ii) of Lemma C.8, γ u v w ≤ τ \gamma_{uvw} \le\tau γ uv w ≤ τ for each of them, so at least r r r descendants satisfy cos ( μ u , μ v ) ≤ cos ( μ u , μ w ) + τ \cos(\mu_u,\mu_v) \le\cos(\mu_u,\mu_w) + \tau cos ( μ u , μ v ) ≤ cos ( μ u , μ w ) + τ , and w w w is counted on the image side.
As in § 3.2 , we compute the 95th percentile of the descendant angles with linear interpolation rather than as an exact order statistic. Both sides have the same rank, but neither radius is one of the measured cosines, so the order-statistic argument applies only approximately. When we recompute both sides with the order statistic, every conclusion in this appendix remains intact.
To assess whether individual image-side violations persist after transfer, we pool the 505,938 (anchor, off-branch node) pairs scored by Definition 3.2 over the 375 anchors. We compare the probability that such a pair violates H \mathcal{H} H AS in the learned text embeddings with the conditional probability that it violates H \mathcal{H} H AS on the text side given that it violates H \mathcal{H} H AS on the image side.
Table 5 shows that the theorem’s bound is loose for these prototypes. Although D < ρ min / 4 D < \rho_{\min}/4 D < ρ m i n /4 does not hold for any backbone, Eq. (8) holds at every anchor with a post hoc slack well below the largest possible cosine-margin difference of 2. The worst-anchor slack is comparable to the median image margin for the two pretrained backbones and exceeds it for the randomly initialized encoder.
Table 5. The inequality required by Theorem 4.6 and measured approximate H \mathcal{H} H AS transfer on the same real prototypes. The inequality D < ρ min / 4 D < \rho_{\min}/4 D < ρ m i n /4 does not hold for any of the three backbones, so we obtain no guarantee from the theorem here. We evaluate the image-side rows on the fixed prototypes and the text-side rows on the closed-form linear-loss optimum A l i n , ∗ A_{\mathrm{lin},*} A lin , ∗ . For the worst-anchor slack, one value of τ \tau τ must satisfy Eq. (8) at every anchor, so we report the largest anchor-specific slack. This slack is post hoc. For the last three rows, we pool the 505,938 (anchor, off-branch node) pairs over the 375 anchors. For the two H \mathcal{H} H AS-Viol rows, we report per-anchor percentages measured from all training images per node in one pass, not with the disjoint even/odd split of Table 1 .
For the randomly initialized encoder, the image and text geometries are most similar because their mean violation percentages nearly match. The worst-anchor slack shows that this agreement is not uniform across anchors.
The slack bound is loose because the perturbation is structured. The realized perturbation e k = − ξ k l i n z ⃗ e_k=-\xi_k^{\mathrm{lin}}\vec{z} e k = − ξ k lin z is rank one: every text embedding moves along the single direction − z ⃗ -\vec{z} − z , differing only in magnitude. By contrast, in Lemma C.5 we bound perturbations in arbitrary directions by 4 η / ρ min 4\eta/\rho_{\min} 4 η / ρ m i n per cosine and 8 η / ρ min 8\eta/\rho_{\min} 8 η / ρ m i n for the cosine difference in Definition 3.1. In the worst case, the perturbations at u u u , v v v , and w w w point in directions that maximize the change in the cosine difference. Here the parallel shifts largely cancel in the cosine difference. The gap between the theorem’s bound and the observed transfer is therefore due to the worst-case bound, which allows perturbations in any direction, not a failure of the transfer mechanism.
To obtain a useful a priori threshold for these real embeddings, we may need an analysis that accounts for the perturbations sharing one direction, which we leave open.
Image-side violations persist, but new text-side violations also arise. If image- and text-side violations were independent, the conditional and marginal text-side violation probabilities would be equal. Instead, their ratio exceeds one for every backbone in Table 5 . Thus an image-side violation is more likely to remain after transfer than an arbitrary triplet is to violate on the text side.
A high conditional violation probability does not mean that text-side violations occur only where image-side ones do. On the trained backbones, the per-anchor text violation percentage substantially exceeds the image percentage, so many text-side violations have no image-side counterpart. We conclude only that, where Theorem 4.6 gives no guarantee, substantial approximate H A S \mathcal{H}AS H A S structure remains after transfer while new violations also arise.
Caption Permutation Disrupts Approximate H A S \mathcal{H}AS H A S Transfer
We test whether approximate H A S \mathcal{H}AS H A S transfer depends on caption–image pairings or only on the marginal caption distribution. Under the path-consistent setup of § 4.1 , each caption co-occurs only with images from its descendant leaves.
Using the ImageNet-1K / WordNet prototypes from three ViT-L/14 backbones in Appendix § 16 .§ 4 , we compute the closed-form linear-loss text minimizer A l i n , ∗ A_{\mathrm{lin},*} A lin , ∗ from Proposition D.1 with path-consistent captions and with hypernyms randomly permuted across images. The permutation preserves the image prototypes and marginal caption distribution but disrupts path consistency. We repeat the ablation in the synthetic InfoNCE setting of Appendix § 16 .§ 3 , varying λ \lambda λ to test whether stronger regularization compensates for the permutation.
In Table 6 (a), permutation more than doubles the text-side violation percentage for every backbone, raising it above half. In Table 6 (b), increasing λ \lambda λ reduces the path-consistent violation percentage to zero, but not the permuted percentage. Thus, in both settings, preserving the marginal caption distribution alone does not preserve approximate H A S \mathcal{H}AS H A S transfer.
Text-side H A S \mathcal{H}AS H A S is strongest with path-consistent captions under joint CLIP training
In Appendix § 16 .§ 5 , we fix the image embeddings and permute hypernym captions to test whether text-side H A S \mathcal{H}AS H A S depends on their image pairings. Here we train both CLIP encoders on ImageNet and compare caption pairings.
Training procedure. We train CLIP ViT-B/32 [1 , 53 ] from scratch with OpenCLIP [44 ] on the 1,281,167 ImageNet-1K training images for 32 epochs. We use AdamW (β 1 = 0.9 \beta_1 = 0.9 β 1 = 0.9 , β 2 = 0.98 \beta_2 = 0.98 β 2 = 0.98 , ϵ = 10 − 6 \epsilon= 10^{-6} ϵ = 1 0 − 6 ), learning rate 10 − 3 10^{-3} 1 0 − 3 , and weight decay 0.1. We use the AdamW schedule: 2024, 2000 warmup steps, and a learned temperature. For each image, we fix one caption across epochs by placing a hierarchy node’s name in one of the 80 ImageNet prompt templates of Radford et al. [1 ] .
Caption pairings. We compare the following caption pairings:
path-consistent : draw uniformly from the image’s leaf and its ancestors in the WordNet hierarchy of § 3.2 , following § 4.1 ;
leaf-only : choose the image’s leaf;
After every epoch, we measure H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol using the 375 anchors and 95th-percentile descendant threshold of § 3.2 . We average the 80 prompt templates for each text concept and aggregate 50 fixed training images per leaf with descendant-leaf weighting for each image concept. In the left panel of Figure 18 , the path-consistent pairing has the lowest H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol from the first epoch onward. Text-side violations fall less under leaf-only captions and remain high under permutation.
Figure 18. Text-side H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S -Viol is lowest with path-consistent captions during joint CLIP training. We plot text (left) and image (right, log scale) H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S violation percentages (lower is better). Path-consistent captions pair an image with a leaf or ancestor; leaf-only captions only pair with the leaf; in the permuted pairing, we shuffle hypernyms among images at the same depth.
Alternating Optimization: The Image-Side Update
The fixed-image results explain how image-side approximate H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S transfers to text, but not how an image-side update changes variation among images. We therefore analyze an exact image update after the text-side update. Images sharing a caption then receive the same embedding; their input variation enters through caption-group averages, the global image mean, and a common shift weighted by caption counts. In Proposition K.3 , we bound the stochastic deviation of these updated embeddings.
We analyze one exact text-side step followed by one exact image-side step under the linear contrastive loss, with ridge regularization on both sides. We use the linear loss because the InfoNCE image-side step has no closed form.
L a l t ( A , Z ) : = − tr ( A T Z B T ) + λ 2 ∥ A ∥ F 2 + λ Z 2 ∥ Z ∥ F 2 , (133) \mathcal{L}^{\mathrm{alt}}(A,Z) := -\operatorname{tr}(A^{\mathsf{T}}ZB^{\mathsf{T}}) + \frac{\lambda}{2}\lVert A\rVert_{F}^{2} + \frac{\lambda_Z}{2}\lVert Z\rVert_{F}^{2}, \tag*{(133)} L alt ( A , Z ) := − tr ( A T Z B T ) + 2 λ ∥ A ∥ F 2 + 2 λ Z ∥ Z ∥ F 2 , (133)
Here λ , λ Z > 0 \lambda, \lambda_Z > 0 λ , λ Z > 0 and B B B is the label-encoding matrix of Eq. (58) ; by Eq. (60) , the first two terms are exactly the regularized linear contrastive loss of Eq. (5) . The cross-modal term is bilinear in ( A , Z ) (A,Z) ( A , Z ) , and the objective is strongly convex in either argument when the other is fixed. We obtain the exact minimizer of each step from Proposition D.1 .
To obtain the image-side update, we first analyze the same label-encoding matrix B B B used in the A A A -step by columns, splitting each entry into a matched-caption contribution and an off-caption coefficient shared across image columns. Set κ : = 1 + 1 2 ( M − 1 ) + 1 2 ( N − 1 ) \kappa:= 1 + \frac{1}{2(M-1)} + \frac{1}{2(N-1)} κ := 1 + 2 ( M − 1 ) 1 + 2 ( N − 1 ) 1 and, for k ∈ [ M ] k \in[M] k ∈ [ M ] , γ k : = 1 2 ( M − 1 ) + N k 2 ( N − 1 ) \gamma_k := \frac{1}{2(M-1)} + \frac{N_k}{2(N-1)} γ k := 2 ( M − 1 ) 1 + 2 ( N − 1 ) N k . Expanding Eq. (58) and collecting terms in Y k j Y_{kj} Y k j gives
B k j = Y k j + Y k j − 1 2 ( M − 1 ) + Y k j − N k 2 ( N − 1 ) = κ Y k j − γ k , ( k , j ) ∈ [ M ] × [ N ] . (134) \begin{aligned}
B_{kj} &= Y_{kj} + \frac{Y_{kj}-1}{2(M-1)} + \frac{Y_{kj}-N_k}{2(N-1)} \\
&= \kappa Y_{kj} - \gamma_k,\qquad(k,j) \in[M] \times[N]. \tag*{(134)}
\end{aligned} B k j = Y k j + 2 ( M − 1 ) Y k j − 1 + 2 ( N − 1 ) Y k j − N k = κ Y k j − γ k , ( k , j ) ∈ [ M ] × [ N ] . (134)
The indicator Y k j Y_{kj} Y k j selects the paired text embedding, while summing − γ k -\gamma_k − γ k against the text embeddings gives the uniform and caption-frequency-weighted means in the common shift below.
Fix A ∈ R d × M A \in\mathbb{R}^{d \times M} A ∈ R d × M . Applying Proposition D.1 to Eq. (133) gives the image-side minimizer
Z l i n , ∗ ( A ) = 1 λ Z A B . (135) Z^{\mathrm{lin},*}(A) = \frac{1}{\lambda_Z}AB. \tag*{(135)} Z lin , ∗ ( A ) = λ Z 1 A B . (135)
To express its columns, define the uniform and caption-frequency-weighted text means
a ˉ : = 1 M ∑ k = 1 M a k , a ˉ c a p t : = 1 N ∑ k = 1 M N k a k = 1 N ∑ j = 1 N a c j . (136) \bar{a} := \frac{1}{M}\sum_{k=1}^{M} a_k,\qquad\bar{a}_{\mathrm{capt}} := \frac{1}{N}\sum_{k=1}^{M}N_k a_k = \frac{1}{N}\sum_{j=1}^{N}a_{c_j}. \tag*{(136)} a ˉ := M 1 k = 1 ∑ M a k , a ˉ capt := N 1 k = 1 ∑ M N k a k = N 1 j = 1 ∑ N a c j . (136)
Using Eq. (134) and the means in Eq. (136) , the j j j -th column of Eq. (135) is
z j l i n , ∗ ( A ) = 1 λ Z ∑ k = 1 M B k j a k = 1 λ Z ( κ a c j − ∑ k = 1 M γ ˉ k a k ) = 1 λ Z [ κ a c j − M 2 ( M − 1 ) a ˉ − N 2 ( N − 1 ) a ˉ c a p t ] (137) \begin{aligned}
z_j^{\mathrm{lin},*}(A) &= \frac{1}{\lambda_Z}\sum_{k=1}^{M}B_{kj}a_k \\
&= \frac{1}{\lambda_Z}\left(\kappa a_{c_j}-\sum_{k=1}^{M}\bar{\gamma}_k a_k\right) \\
&= \frac{1}{\lambda_Z}\left[\kappa a_{c_j}-\frac{M}{2(M-1)}\bar{a}-\frac{N}{2(N-1)}\bar{a}_{\mathrm{capt}}\right] \tag*{(137)}
\end{aligned} z j lin , ∗ ( A ) = λ Z 1 k = 1 ∑ M B k j a k = λ Z 1 ( κ a c j − k = 1 ∑ M γ ˉ k a k ) = λ Z 1 [ κ a c j − 2 ( M − 1 ) M a ˉ − 2 ( N − 1 ) N a ˉ capt ] (137)
for every j ∈ [ N ] j \in[N] j ∈ [ N ] .
Thus images with the same caption receive the same embedding, even before imposing the text-side optimum.
In Eq. (137) , the positive term favors the text embedding paired with image j j j , while the two subtracted means play the role of the global image mean z ˉ \bar{z} z ˉ in Corollary D.2. To simplify this image-side update after the text step, we first show that its uniform text mean vanishes.
Lemma K.1 (Global centering at the A A A -step optimum). Every column of B B B sums to zero: ∑ k = 1 M B k j = 0 \sum_{k=1}^{M} B_{kj}=0 ∑ k = 1 M B k j = 0 for every j ∈ [ N ] j \in[N] j ∈ [ N ] . Consequently, for every Z Z Z , the A A A -step optimum of Proposition D.1 satisfies
∑ k = 1 M a k l i n , ∗ = 0. (138) \sum_{k=1}^{M}a_k^{\mathrm{lin},*}=0. \tag*{(138)} k = 1 ∑ M a k lin , ∗ = 0. (138)
Proof. Fix j ∈ [ N ] j \in[N] j ∈ [ N ] . Each image has exactly one caption, so ∑ k = 1 M Y k j = 1 \sum_{k=1}^{M}Y_{kj}=1 ∑ k = 1 M Y k j = 1 ; also ∑ k = 1 M N k = N \sum_{k=1}^{M}N_k=N ∑ k = 1 M N k = N . Therefore, by Eq. (134) ,
∑ k = 1 M B k j = κ − M 2 ( M − 1 ) − N 2 ( N − 1 ) = 0 , (139) \sum_{k=1}^{M}B_{kj}=\kappa-\frac{M}{2(M-1)}-\frac{N}{2(N-1)}=0, \tag*{(139)} k = 1 ∑ M B k j = κ − 2 ( M − 1 ) M − 2 ( N − 1 ) N = 0 , (139)
Thus B T 1 = 0 B^{\mathsf{T}}\mathbf{1}=0 B T 1 = 0 with 1 ∈ R M \mathbf{1}\in\mathbb{R}^{M} 1 ∈ R M the all-ones vector. Since ∑ k = 1 M a k l i n , ∗ = A l i n , ∗ 1 \sum_{k=1}^{M}a_k^{\mathrm{lin},*}=A^{\mathrm{lin},*}\mathbf{1} ∑ k = 1 M a k lin , ∗ = A lin , ∗ 1 , we obtain Eq. (138) from Proposition D.1.
At the A A A -step optimum, we remove the uniform mean from Eq. (137) using Lemma K.1. The remaining terms give an affine copy of the text embeddings.
Corollary K.2 (Image-side update as a scaled, shifted text embedding). Let A = A l i n , ∗ A=A^{\mathrm{lin},*} A = A lin , ∗ for arbitrary fixed Z Z Z . With κ \kappa κ from Eq. (134) , define the caption-mean shift
e c a p t : = − N 2 ( N − 1 ) κ a ˉ c a p t . (140) e_{\mathrm{capt}} := -\frac{N}{2(N-1)\kappa}\bar{a}_{\mathrm{capt}}. \tag*{(140)} e capt := − 2 ( N − 1 ) κ N a ˉ capt . (140) Then the Z Z Z -step optimum is a scaled, shifted copy of the caption’s text embedding:
z j l i n , ∗ = κ λ Z ( a c j l i n , ∗ + e c a p t ) (141) z_j^{\mathrm{lin},*}=\frac{\kappa}{\lambda_Z}\left(a_{c_j}^{\mathrm{lin},*}+e_{\mathrm{capt}}\right) \tag*{(141)} z j lin , ∗ = λ Z κ ( a c j lin , ∗ + e capt ) (141) for every j ∈ [ N ] j \in[N] j ∈ [ N ] .
Proof. By Lemma K.1, a ˉ = 0 \bar{a}=0 a ˉ = 0 at A = A l i n , ∗ A=A^{\mathrm{lin},*} A = A lin , ∗ . Substituting this and Eq. (140) into Eq. (137) gives Eq. (141) .
This update makes images with the same caption identical. The remaining stochastic deviation comes from caption-group averages and a common caption-count-weighted shift; in Proposition K.3, we bound it after removing the update’s positive scale.
Proposition K.3 (Caption-group averaging in the image update). Suppose Assumptions 4.1, 4.3, and 4.4 hold. Starting from Z Z Z , compute the exact text-side minimizer A l i n , ∗ A^{\mathrm{lin},*} A lin , ∗ and then the exact image-side minimizer Z l i n , ∗ Z^{\mathrm{lin},*} Z lin , ∗ . Fix the observed leaf and caption labels, and let E ε \mathbb{E}_{\varepsilon} E ε average over the independent variation vectors of Assumption 4.1. For each non-root k k k and any j ∈ P k j \in\mathcal{P}_k j ∈ P k , define the normalized stochastic deviation
s k : = λ λ Z κ 2 N k ( z j l i n , ∗ − E ε z j l i n , ∗ ) , (142) s_k := \frac{\lambda\lambda_Z}{\kappa^2 N_k}\left(z_j^{\mathrm{lin},*}-\mathbb{E}_{\varepsilon}z_j^{\mathrm{lin},*}\right), \tag*{(142)} s k := κ 2 N k λ λ Z ( z j lin , ∗ − E ε z j lin , ∗ ) , (142) where κ \kappa κ is from Eq. (134) and the choice of j ∈ P k j \in\mathcal{P}_k j ∈ P k does not matter by Eq. (141) . The effect of the common caption-count-weighted shift is measured by
ω : = ∑ k = 1 M N k 3 ( N − 1 ) N min . (143) \omega:= \frac{\sqrt{\sum_{k=1}^{M}N_k^3}}{(N-1)\sqrt{N_{\min}}}. \tag*{(143)} ω := ( N − 1 ) N m i n ∑ k = 1 M N k 3 . (143) With ξ max l i n \xi_{\max}^{\mathrm{lin}} ξ m a x lin defined before Theorem 4.6, there exists a universal constant C > 0 C>0 C > 0 such that, for every ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) , with probability at least 1 − ϵ 1-\epsilon 1 − ϵ ,
max k ≠ r o o t ∥ s k ∥ ≤ C σ ( ( 1 + ω ) d + log ( M − 1 ) + log ( 8 / ϵ ) N min + ξ max l i n d + log ( 8 / ϵ ) N ) . (144) \max_{k\ne\mathrm{root}}\|s_k\| \le C\sigma\left((1+\omega)\sqrt{\frac{d+\log(M-1)+\log(8/\epsilon)}{N_{\min}}}+\xi_{\max}^{\mathrm{lin}}\sqrt{\frac{d+\log(8/\epsilon)}{N}}\right). \tag*{(144)} k = root max ∥ s k ∥ ≤ C σ ( 1 + ω ) N m i n d + log ( M − 1 ) + log ( 8/ ϵ ) + ξ m a x lin N d + log ( 8/ ϵ ) . (144)
Proof. Step 1: Identify the centered image update. We first write the common shift e c a p t e_{\mathrm{capt}} e capt in terms of the input images. For each image j ∈ [ N ] j \in[N] j ∈ [ N ] , define its caption-count weight by
h j : = N c j − 1 κ ∑ k = 1 M N k γ k = N c j − 1 κ ( N 2 ( M − 1 ) + ∑ k = 1 M N k 2 2 ( N − 1 ) ) . (145) \begin{aligned}
h_j := N_{c_j}-\frac{1}{\kappa}\sum_{k=1}^{M}N_k\gamma_k \\
&= N_{c_j}-\frac{1}{\kappa}\left(\frac{N}{2(M-1)}+\frac{\sum_{k=1}^{M}N_k^2}{2(N-1)}\right). \tag*{(145)}
\end{aligned} h j := N c j − κ 1 k = 1 ∑ M N k γ k = N c j − κ 1 ( 2 ( M − 1 ) N + 2 ( N − 1 ) ∑ k = 1 M N k 2 ) . (145)
The common shift in Eq. (140) uses the N k / N N_k/N N k / N -weighted text mean of Eq. (136) . The text-side minimizer of Proposition D.1 and the label encoding in Eq. (134) apply to every k ∈ [ M ] k \in[M] k ∈ [ M ] , including nodes with N k = 0 N_k=0 N k = 0 . Substituting them into the weighted mean gives
λ a ˉ c a p t = 1 N ∑ k = 1 M N k λ a k l i n , ∗ (by xref‘Eq. (132)‘) = κ N ∑ k = 1 M N k ∑ i ∈ P k z i − ( ∑ k = 1 M N k γ k ) z ˉ (by xref‘Eqs. (59) and (130)‘) = κ N ∑ j = 1 N N c j z j − 1 N ∑ j = 1 N ( ∑ k = 1 M N k γ k ) z j (since j ∈ P c j and z ˉ = 1 N ∑ j = 1 N z j ) = κ N ∑ j = 1 N ( N c j − 1 κ ∑ k = 1 M N k γ k ) z j (collecting terms in z j ) = κ N ∑ j = 1 N h j z j (by xref‘Eq. (141)‘). (146) \begin{aligned}
\lambda\bar{a}_{\mathrm{capt}}=\frac{1}{N}\sum_{k=1}^{M}N_k\lambda a_k^{\mathrm{lin},*}
\qquad\text{(by {xref}`Eq. (132)`)} \\
&=\frac{\kappa}{N}\sum_{k=1}^{M}N_k\sum_{i\in\mathcal{P}_k}z_i-\left(\sum_{k=1}^{M}N_k\gamma_k\right)\bar{z}
\qquad\text{(by {xref}`Eqs. (59) and (130)`)} \\
&=\frac{\kappa}{N}\sum_{j=1}^{N}N_{c_j}z_j-\frac{1}{N}\sum_{j=1}^{N}\left(\sum_{k=1}^{M}N_k\gamma_k\right)z_j\qquad\text{(since }j\in\mathcal{P}_{c_j}\text{ and }\bar{z}=\frac{1}{N}\sum_{j=1}^{N}z_j\text{)} \\
&=\frac{\kappa}{N}\sum_{j=1}^{N}\left(N_{c_j}-\frac{1}{\kappa}\sum_{k=1}^{M}N_k\gamma_k\right)z_j\qquad\text{(collecting terms in }z_j\text{)} \\
&=\frac{\kappa}{N}\sum_{j=1}^{N}h_jz_j
\qquad\text{(by {xref}`Eq. (141)`).} \tag*{(146)}
\end{aligned} λ a ˉ capt = N 1 k = 1 ∑ M N k λ a k lin , ∗ (by xref ‘Eq. (132)‘) = N κ k = 1 ∑ M N k i ∈ P k ∑ z i − ( k = 1 ∑ M N k γ k ) z ˉ (by xref ‘Eqs. (59) and (130)‘) = N κ j = 1 ∑ N N c j z j − N 1 j = 1 ∑ N ( k = 1 ∑ M N k γ k ) z j (since j ∈ P c j and z ˉ = N 1 j = 1 ∑ N z j ) = N κ j = 1 ∑ N ( N c j − κ 1 k = 1 ∑ M N k γ k ) z j (collecting terms in z j ) = N κ j = 1 ∑ N h j z j (by xref ‘Eq. (141)‘). (146)
Substituting this expression into Eq. (140) gives
e c a p t = − N 2 ( N − 1 ) κ a ˉ c a p t = − 1 2 ( N − 1 ) λ ∑ j = 1 N h j z j . (147) \begin{aligned}
e_{\mathrm{capt}}=-\frac{N}{2(N-1)\kappa}\bar{a}_{\mathrm{capt}} \\
&=-\frac{1}{2(N-1)\lambda}\sum_{j=1}^{N}h_jz_j. \tag*{(147)}
\end{aligned} e capt = − 2 ( N − 1 ) κ N a ˉ capt = − 2 ( N − 1 ) λ 1 j = 1 ∑ N h j z j . (147)
Fix a non-root k k k and j ∈ P k j \in\mathcal{P}_k j ∈ P k , so c j = k c_j = k c j = k . Substituting this shift and the text-side empirical-mean expression of Corollary D.2 into Eq. (141) gives
z j l i n , ∗ = κ λ Z ( a k l i n , ∗ + e c a p t ) z_j^{\mathrm{lin},*} = \frac{\kappa}{\lambda_Z}\left(a_k^{\mathrm{lin},*}+e_{\mathrm{capt}}\right) z j lin , ∗ = λ Z κ ( a k lin , ∗ + e capt )
(by Eq. (141) with c j = k c_j=k c j = k )
= κ λ Z [ κ N k λ ( z ˉ k − ξ k l i n z ˉ ) − 1 2 ( N − 1 ) λ ∑ i = 1 N h i z i ] = \frac{\kappa}{\lambda_Z}\left[\frac{\kappa N_k}{\lambda}\left(\bar{z}_k-\xi_k^{\mathrm{lin}}\bar{z}\right)-\frac{1}{2(N-1)\lambda}\sum_{i=1}^{N}h_i z_i\right] = λ Z κ [ λ κ N k ( z ˉ k − ξ k lin z ˉ ) − 2 ( N − 1 ) λ 1 i = 1 ∑ N h i z i ]
(by Eqs. (67) and (147) )
= κ 2 N k λ λ Z ( z ˉ k − ξ k l i n z ˉ − 1 2 κ ( N − 1 ) N k ∑ i = 1 N h i z i ) . (148) = \frac{\kappa^2N_k}{\lambda\lambda_Z}\left(\bar{z}_k-\xi_k^{\mathrm{lin}}\bar{z}-\frac{1}{2\kappa(N-1)N_k}\sum_{i=1}^{N}h_i z_i\right). \tag*{(148)} = λ λ Z κ 2 N k ( z ˉ k − ξ k lin z ˉ − 2 κ ( N − 1 ) N k 1 i = 1 ∑ N h i z i ) . (148)
(factoring out κ 2 N k / ( λ λ Z ) \kappa^2N_k/(\lambda\lambda_Z) κ 2 N k / ( λ λ Z ) ).
The path-consistent captions and equal image counts across leaves from § 4.1 , together with Assumptions 4.1, 4.3, and 4.4, allow us to apply Lemma C.10 and Corollary C.11. With ξ k l i n \xi_k^{\mathrm{lin}} ξ k lin from Corollary 4.5, substituting Eq. (47) with b k = b = 0 b_k=b=0 b k = b = 0 and z i = μ y i + ϵ i z_i=\mu_{y_i}+\epsilon_i z i = μ y i + ϵ i for every i ∈ [ N ] i\in[N] i ∈ [ N ] gives
λ λ Z κ 2 N k z j l i n , ∗ = ( μ k + ϵ ˉ k ) − ξ k l i n ( μ r o o t + ϵ ˉ ) (149) \frac{\lambda\lambda_Z}{\kappa^2N_k}z_j^{\mathrm{lin},*}=(\mu_k+\bar{\epsilon}_k)-\xi_k^{\mathrm{lin}}(\mu_{\mathrm{root}}+\bar{\epsilon}) \tag*{(149)} κ 2 N k λ λ Z z j lin , ∗ = ( μ k + ϵ ˉ k ) − ξ k lin ( μ root + ϵ ˉ ) (149)
(by Eqs. (47) and (148) )
− 1 2 κ ( N − 1 ) N k ∑ i = 1 N h i ( μ y i + ϵ i ) (150) -\frac{1}{2\kappa(N-1)N_k}\sum_{i=1}^{N}h_i(\mu_{y_i}+\epsilon_i) \tag*{(150)} − 2 κ ( N − 1 ) N k 1 i = 1 ∑ N h i ( μ y i + ϵ i ) (150)
= μ k − ξ k l i n μ r o o t − 1 2 κ ( N − 1 ) N k ∑ i = 1 N h i μ y i (151) =\mu_k-\xi_k^{\mathrm{lin}}\mu_{\mathrm{root}}-\frac{1}{2\kappa(N-1)N_k}\sum_{i=1}^{N}h_i\mu_{y_i} \tag*{(151)} = μ k − ξ k lin μ root − 2 κ ( N − 1 ) N k 1 i = 1 ∑ N h i μ y i (151)
(grouping prototype and noise terms)
+ ( ϵ ˉ k − ξ k l i n ϵ ˉ − 1 2 κ ( N − 1 ) N k ∑ i = 1 N h i ϵ i ) . (152) +\left(\bar{\epsilon}_k-\xi_k^{\mathrm{lin}}\bar{\epsilon}-\frac{1}{2\kappa(N-1)N_k}\sum_{i=1}^{N}h_i\epsilon_i\right). \tag*{(152)} + ( ϵ ˉ k − ξ k lin ϵ ˉ − 2 κ ( N − 1 ) N k 1 i = 1 ∑ N h i ϵ i ) . (152)
The parenthesized noise term has zero mean by Assumption 4.1, so it equals s k s_k s k in Eq. (142) . Thus the exact prototype decomposition is
z j l i n , ∗ = κ 2 N k λ λ Z ( μ k − ξ k l i n μ r o o t − 1 2 κ ( N − 1 ) N k ∑ i = 1 N h i μ y i + s k ) . (153) z_j^{\mathrm{lin},*}=\frac{\kappa^2N_k}{\lambda\lambda_Z}\left(\mu_k-\xi_k^{\mathrm{lin}}\mu_{\mathrm{root}}-\frac{1}{2\kappa(N-1)N_k}\sum_{i=1}^{N}h_i\mu_{y_i}+s_k\right). \tag*{(153)} z j lin , ∗ = λ λ Z κ 2 N k ( μ k − ξ k lin μ root − 2 κ ( N − 1 ) N k 1 i = 1 ∑ N h i μ y i + s k ) . (153)
Step 2: Bound the caption-group and global means. By Assumption 4.3, N k ≥ N min > 0 N_k\ge N_{\min}>0 N k ≥ N m i n > 0 for every non-root k k k , and by Assumption 4.1, the variations are independent, centered, and sub-Gaussian. Apply Lemma C.12 with K = V ∖ { r o o t } \mathcal{K}=V\setminus\{\mathrm{root}\} K = V ∖ { root } , I k = P k I_k=\mathcal{P}_k I k = P k for k ∈ K k\in\mathcal{K} k ∈ K , coefficients { ξ k l i n } k ∈ K \{\xi_k^{\mathrm{lin}}\}_{k\in\mathcal{K}} { ξ k lin } k ∈ K , and u = log ( 8 / ϵ ) u=\log(8/\epsilon) u = log ( 8/ ϵ ) . Since ∣ K ∣ = M − 1 |\mathcal{K}|=M-1 ∣ K ∣ = M − 1 and min k ∈ K ∣ I k ∣ = N min \min_{k\in\mathcal{K}}|I_k|=N_{\min} min k ∈ K ∣ I k ∣ = N m i n , with probability at least 1 − ϵ / 2 1-\epsilon/2 1 − ϵ /2 ,
max k ≠ r o o t ∥ ϵ ˉ k − ξ k l i n ϵ ˉ ∥ ≤ C σ ( d + log ( M − 1 ) + log ( 8 / ϵ ) N min + ξ max l i n d + log ( 8 / ϵ ) N ) . (154) \max_{k\ne\mathrm{root}}\left\|\bar{\epsilon}_k-\xi_k^{\mathrm{lin}}\bar{\epsilon}\right\|
\le C\sigma\left(\sqrt{\frac{d+\log(M-1)+\log(8/\epsilon)}{N_{\min}}}+\xi_{\max}^{\mathrm{lin}}\sqrt{\frac{d+\log(8/\epsilon)}{N}}\right). \tag*{(154)} k = root max ϵ ˉ k − ξ k lin ϵ ˉ ≤ C σ N m i n d + log ( M − 1 ) + log ( 8/ ϵ ) + ξ m a x lin N d + log ( 8/ ϵ ) . (154)
Step 3: Bound the common weighted shift and combine the events. We first control the caption-count weights. By Eq. (145) , N c j − h j N_{c_j}-h_j N c j − h j is nonnegative and takes the same value for every j ∈ [ N ] j\in[N] j ∈ [ N ] . Since ∑ k = 1 M N k = N \sum_{k=1}^{M}N_k=N ∑ k = 1 M N k = N , Cauchy–Schwarz gives N 2 ≤ M ∑ k = 1 M N k 2 N^2\le M\sum_{k=1}^{M}N_k^2 N 2 ≤ M ∑ k = 1 M N k 2 . Hence
∑ j = 1 N ( N c j + h j ) = ∑ j = 1 N [ 2 N c j − 1 κ ( N 2 ( M − 1 ) + ∑ k = 1 M N k 2 2 ( N − 1 ) ) ] \begin{aligned}
\sum_{j=1}^{N}(N_{c_j}+h_j) \\
&=\sum_{j=1}^{N}\left[2N_{c_j}-\frac{1}{\kappa}\left(\frac{N}{2(M-1)}+\frac{\sum_{k=1}^{M}N_k^2}{2(N-1)}\right)\right]
\end{aligned} j = 1 ∑ N ( N c j + h j ) = j = 1 ∑ N [ 2 N c j − κ 1 ( 2 ( M − 1 ) N + 2 ( N − 1 ) ∑ k = 1 M N k 2 ) ]
(by Eq. (145) )
= 2 ∑ k = 1 M N k 2 − 1 κ ( N 2 2 ( M − 1 ) + N ∑ k = 1 M N k 2 2 ( N − 1 ) ) =2\sum_{k=1}^{M}N_k^2-\frac{1}{\kappa}\left(\frac{N^2}{2(M-1)}+\frac{N\sum_{k=1}^{M}N_k^2}{2(N-1)}\right) = 2 k = 1 ∑ M N k 2 − κ 1 ( 2 ( M − 1 ) N 2 + 2 ( N − 1 ) N ∑ k = 1 M N k 2 )
(since N k N_k N k images have c j = k c_j=k c j = k )
≥ ( 2 − N 2 ( M − 1 ) − N 2 ( N − 1 ) ) ∑ k = 1 M N k 2 \ge\left(2-\frac{N}{2(M-1)}-\frac{N}{2(N-1)}\right)\sum_{k=1}^{M}N_k^2 ≥ ( 2 − 2 ( M − 1 ) N − 2 ( N − 1 ) N ) k = 1 ∑ M N k 2
(since κ ≥ 1 \kappa\ge1 κ ≥ 1 and N 2 ≤ M ∑ k N k 2 N^2\le M\sum_kN_k^2 N 2 ≤ M ∑ k N k 2 )
≥ 0 (155) \ge0 \tag*{(155)} ≥ 0 (155)
(since M ≥ 2 M\ge2 M ≥ 2 and N ≥ 2 N\ge2 N ≥ 2 ). Therefore, with w : = ( h 1 , … , h N ) w := (h_1,\ldots,h_N) w := ( h 1 , … , h N ) ,
∥ w ∥ 2 = ∑ j = 1 N N c j 2 − ∑ j = 1 N ( N c j − h j ) ( N c j + h j ) (difference of squares) ≤ ∑ j = 1 N N c j 2 (by Eq. (151), since N c j − h j ≥ 0 is constant) = ∑ k = 1 M N k 3 (since N k images have c j = k ). (156) \begin{aligned}
\lVert w\rVert^2 = \sum_{j=1}^{N}N_{c_j}^2-\sum_{j=1}^{N}(N_{c_j}-h_j)(N_{c_j}+h_j) \qquad\text{(difference of squares)} \\
&\leq\sum_{j=1}^{N}N_{c_j}^2 \qquad\text{(by Eq. (151), since } N_{c_j}-h_j\geq0 \text{ is constant)} \\
&= \sum_{k=1}^{M}N_k^3 \qquad\text{(since } N_k \text{ images have } c_j=k\text{).} \tag*{(156)}
\end{aligned} ∥ w ∥ 2 = j = 1 ∑ N N c j 2 − j = 1 ∑ N ( N c j − h j ) ( N c j + h j ) (difference of squares) ≤ j = 1 ∑ N N c j 2 (by Eq. (151), since N c j − h j ≥ 0 is constant) = k = 1 ∑ M N k 3 (since N k images have c j = k ). (156)
The Hoeffding and net argument in the proof of Lemma C.12 applies to the fixed weights w w w and gives Pr ( ∥ ∑ j = 1 N h j ε j ∥ > C σ ∥ w ∥ d + t ) ≤ 2 e − t \Pr\left(\left\lVert\sum_{j=1}^{N}h_j\varepsilon_j\right\rVert>C\sigma\lVert w\rVert\sqrt{d+t}\right)\leq2e^{-t} Pr ( ∑ j = 1 N h j ε j > C σ ∥ w ∥ d + t ) ≤ 2 e − t for every t > 0 t>0 t > 0 . Taking t = log ( 4 / ϵ ) t=\log(4/\epsilon) t = log ( 4/ ϵ ) makes the probability of exceeding this bound at most ϵ / 2 \epsilon/2 ϵ /2 . On the complementary event,
max k ≠ r o o t ∥ 1 2 κ ( N − 1 ) N k ∑ j = 1 N h j ε j ∥ ≤ 1 2 ( N − 1 ) N min ∥ ∑ j = 1 N h j ε j ∥ (since κ ≥ 1 and N k ≥ N min ) ≤ C σ ∥ w ∥ d + log ( 4 / ϵ ) 2 ( N − 1 ) N min (by the Hoeffding bound with t = log ( 4 / ϵ ) ) ≤ C σ ∑ k = 1 M N k 3 d + log ( 4 / ϵ ) 2 ( N − 1 ) N min (by Eq. (152)) = C 2 σ ω d + log ( 4 / ϵ ) N min (by Eq. (139)). (157) \begin{aligned}
\max_{k\ne\mathrm{root}}\left\lVert\frac{1}{2\kappa(N-1)N_k}\sum_{j=1}^{N}h_j\varepsilon_j\right\rVert\\
&\leq\frac{1}{2(N-1)N_{\min}}\left\lVert\sum_{j=1}^{N}h_j\varepsilon_j\right\rVert\qquad\text{(since }\kappa\geq1 \text{ and } N_k\geq N_{\min}\text{)} \\
&\leq\frac{C\sigma\lVert w\rVert\sqrt{d+\log(4/\epsilon)}}{2(N-1)N_{\min}} \qquad\text{(by the Hoeffding bound with } t=\log(4/\epsilon)\text{)} \\
&\leq\frac{C\sigma\sqrt{\sum_{k=1}^{M}N_k^3}\sqrt{d+\log(4/\epsilon)}}{2(N-1)N_{\min}} \qquad\text{(by Eq. (152))} \\
&= \frac{C}{2}\sigma\omega\sqrt{\frac{d+\log(4/\epsilon)}{N_{\min}}} \qquad\text{(by Eq. (139)).} \tag*{(157)}
\end{aligned} k = root max 2 κ ( N − 1 ) N k 1 j = 1 ∑ N h j ε j ≤ 2 ( N − 1 ) N m i n 1 j = 1 ∑ N h j ε j (since κ ≥ 1 and N k ≥ N m i n ) ≤ 2 ( N − 1 ) N m i n C σ ∥ w ∥ d + log ( 4/ ϵ ) (by the Hoeffding bound with t = log ( 4/ ϵ ) ) ≤ 2 ( N − 1 ) N m i n C σ ∑ k = 1 M N k 3 d + log ( 4/ ϵ ) (by Eq. (152)) = 2 C σ ω N m i n d + log ( 4/ ϵ ) (by Eq. (139)). (157)
Since log ( 4 / ϵ ) ≤ log ( M − 1 ) + log ( 8 / ϵ ) \log(4/\epsilon)\leq\log(M-1)+\log(8/\epsilon) log ( 4/ ϵ ) ≤ log ( M − 1 ) + log ( 8/ ϵ ) , a union bound over the two failure events gives Eq. (140) with probability at least 1 − ϵ 1-\epsilon 1 − ϵ .
In Lemma C.13 , we bound per-instance input noise at scale C σ d + log ( 2 N / ϵ ) C\sigma\sqrt{d+\log(2N/\epsilon)} C σ d + log ( 2 N / ϵ ) . The caption-group term in Eq. (140) is smaller by order N min / ( 1 + ω ) \sqrt{N_{\min}}/(1+\omega) N m i n / ( 1 + ω ) , up to logarithmic factors. The separate N − 1 / 2 N^{-1/2} N − 1/2 term comes from the global image mean.
The result concerns one exact alternating step under the linear loss, so it is an analytic surrogate for joint gradient descent rather than a model of CLIP training. The bound controls stochastic deviation around the caption-grouped prototype decomposition, not the deterministic prototype margin: reduced variation can protect a small positive margin, but the bound alone does not certify reversal of a negative margin. Whether the same contraction holds for the subtree-aggregated prototypes of § 3.2 , which mix caption groups along root-to-node paths, remains open.
Relaxing the Caption-Symmetry Assumptions
In Lemma F.1 , H A S \mathcal{H}AS H A S transfer under observed captions depends on a bound on caption bias. In Appendix § 18 .§ 1 , we bound two sources of that bias, caption imbalance and inexact prototype consistency, and give transfer conditions for both losses. We then examine partial ancestor omission, which creates random caption imbalance, in Appendix § 18 .§ 2 . Finally, in Appendix § 18 .§ 3 , we analyze total omission, for which a caption mean is undefined. Throughout this section, we sample exactly n n n images per leaf as in § 4.1 . In Appendix § 16 .§ 5 , we support these bounds with empirical results: permuting which images match each caption increases the caption bias well outside the small-bias regime of Corollary L.2 , and approximate H A S \mathcal{H}AS H A S transfer fails.
Caption Imbalance and Inexact Prototype Consistency
Caption imbalance changes how leaf prototypes contribute to a concept’s caption mean. For example, dog captions drawn mostly from dalmatians overrepresent one descendant leaf; off-path captions add images from outside the dog’s subtree. Inexact prototype consistency adds bias even when captions are leaf-uniform. To bound these effects, for every non-root node k k k , split its positive set into
P k o n : = { j ∈ P k : y j ∈ Sub ( k ) } , P k o f f : = P k ∖ P k o n , (158) \mathcal{P}^{\mathrm{on}}_k := \{j \in\mathcal{P}_k : y_j \in\operatorname{Sub}(k)\}, \qquad\mathcal{P}^{\mathrm{off}}_k := \mathcal{P}_k \setminus\mathcal{P}^{\mathrm{on}}_k, \tag*{(158)} P k on := { j ∈ P k : y j ∈ Sub ( k )} , P k off := P k ∖ P k on , (158)
where L k : = Sub ( k ) ∩ V l e a f L_k := \operatorname{Sub}(k) \cap V_{\mathrm{leaf}} L k := Sub ( k ) ∩ V leaf denotes the leaf descendants of each node k ∈ V k \in V k ∈ V , and n k ℓ o n : = ∣ { j ∈ P k o n : y j = ℓ } ∣ n^{\mathrm{on}}_{k\ell} := |\{j \in\mathcal{P}^{\mathrm{on}}_k : y_j = \ell\}| n k ℓ on := ∣ { j ∈ P k on : y j = ℓ } ∣ for ℓ ∈ L k \ell\in L_k ℓ ∈ L k counts its on-path captions from leaf ℓ \ell ℓ . A nonempty, leaf-uniform on-path set has prototype mean 1 ∣ L k ∣ ∑ ℓ ∈ L k μ ℓ \frac{1}{|L_k|}\sum_{\ell\in L_k}\mu_\ell ∣ L k ∣ 1 ∑ ℓ ∈ L k μ ℓ , which equals μ k \mu_k μ k under Assumption 4.4. When every non-root node has N k > 0 N_k > 0 N k > 0 , define the caption imbalance by
δ c a p : = max k ≠ r o o t [ 1 N k ∑ ℓ ∈ L k ∣ n k ℓ o n − ∣ P k o n ∣ ∣ L k ∣ ∣ ∥ μ ℓ − μ k ∥ + 1 N k ∑ j ∈ P k o f f ∥ μ y j − μ k ∥ ] , (159) \delta_{\mathrm{cap}} := \max_{k \ne\mathrm{root}} \left[\frac{1}{N_k}\sum_{\ell\in L_k}\left|n^{\mathrm{on}}_{k\ell}-\frac{|\mathcal{P}^{\mathrm{on}}_k|}{|L_k|}\right|\|\mu_\ell-\mu_k\|+\frac{1}{N_k}\sum_{j\in\mathcal{P}^{\mathrm{off}}_k}\|\mu_{y_j}-\mu_k\|\right], \tag*{(159)} δ cap := k = root max N k 1 ℓ ∈ L k ∑ n k ℓ on − ∣ L k ∣ ∣ P k on ∣ ∥ μ ℓ − μ k ∥ + N k 1 j ∈ P k off ∑ ∥ μ y j − μ k ∥ , (159)
and the prototype mismatch by
δ c o n s : = max k ∈ V ∥ 1 ∣ L k ∣ ∑ ℓ ∈ L k μ ℓ − μ k ∥ . (160) \delta_{\mathrm{cons}} := \max_{k \in V}\left\|\frac{1}{|L_k|}\sum_{\ell\in L_k}\mu_\ell-\mu_k\right\|. \tag*{(160)} δ cons := k ∈ V max ∣ L k ∣ 1 ℓ ∈ L k ∑ μ ℓ − μ k . (160)
The maximum in δ c o n s \delta_{\mathrm{cons}} δ cons includes the root because its prototype mismatch contributes to the global caption bias.
Lemma L.1 (Caption bias under imbalanced captions and inexact prototypes). Suppose exactly n n n images are sampled per leaf and the observed captions satisfy N k > 0 N_k>0 N k > 0 for every non-root node k k k . Write b k : = μ ˉ k − μ k b_k := \bar{\mu}_k-\mu_k b k := μ ˉ k − μ k for the caption bias of each non-root node k k k and b : = μ ˉ − μ r o o t b := \bar{\mu}-\mu_{\mathrm{root}} b := μ ˉ − μ root for the global caption bias, as in Definition C.9. Then
∥ b ∥ ≤ δ c o n s , max k ≠ r o o t ∥ b k ∥ ≤ δ c a p + δ c o n s . (161) \|b\| \le\delta_{\mathrm{cons}}, \qquad\max_{k\ne\mathrm{root}}\|b_k\| \le\delta_{\mathrm{cap}}+\delta_{\mathrm{cons}}. \tag*{(161)} ∥ b ∥ ≤ δ cons , k = root max ∥ b k ∥ ≤ δ cap + δ cons . (161) In particular, δ c a p + ( 1 + ξ max ) δ c o n s \delta_{\mathrm{cap}}+(1+\xi_{\max})\delta_{\mathrm{cons}} δ cap + ( 1 + ξ m a x ) δ cons is a caption bias bound for any coefficients { ξ k ∈ R } k ∈ V ∖ { r o o t } \{\xi_k\in\mathbb{R}\}_{k\in V\setminus\{\mathrm{root}\}} { ξ k ∈ R } k ∈ V ∖ { root } , where ξ max : = max k ≠ r o o t ∣ ξ k ∣ \xi_{\max}:=\max_{k\ne\mathrm{root}}|\xi_k| ξ m a x := max k = root ∣ ξ k ∣ . Under Assumption 4.4, b = 0 b=0 b = 0 and δ c a p \delta_{\mathrm{cap}} δ cap is a caption bias bound for any such coefficients.
Proof. Balanced image sampling gives
b = μ ˉ − μ r o o t b=\bar{\mu}-\mu_{\mathrm{root}} b = μ ˉ − μ root
(by Definition C.9)
= 1 ∣ V l e a f ∣ ∑ ℓ ∈ V l e a f μ ℓ − μ r o o t =\frac{1}{|V_{\mathrm{leaf}}|}\sum_{\ell\in V_{\mathrm{leaf}}}\mu_\ell-\mu_{\mathrm{root}} = ∣ V leaf ∣ 1 ℓ ∈ V leaf ∑ μ ℓ − μ root
(since each leaf contributes n n n images).
Since L r o o t = V l e a f L_{\mathrm{root}}=V_{\mathrm{leaf}} L root = V leaf , this is the root term in Eq. (156), so ∥ b ∥ ≤ δ c o n s \|b\|\le\delta_{\mathrm{cons}} ∥ b ∥ ≤ δ cons . For a non-root node k k k with P k o n ≠ ∅ \mathcal{P}^{\mathrm{on}}_k\ne\varnothing P k on = ∅ , grouping images by their leaf and adding and subtracting the centroid of the descendant leaf prototypes give
1 ∣ P k o n ∣ ∑ j ∈ P k o n ( μ y j − μ k ) = ∑ ℓ ∈ L k n k ℓ o n ∣ P k o n ∣ ( μ ℓ − μ k ) = ∑ ℓ ∈ L k ( n k ℓ o n ∣ P k o n ∣ − 1 ∣ L k ∣ ) ( μ ℓ − μ k ) + ( 1 ∣ L k ∣ ∑ ℓ ∈ L k μ ℓ − μ k ) . (162) \begin{aligned}
\frac{1}{|\mathcal{P}^{\mathrm{on}}_k|}\sum_{j\in\mathcal{P}^{\mathrm{on}}_k}(\mu_{y_j}-\mu_k)
=\sum_{\ell\in L_k}\frac{n^{\mathrm{on}}_{k\ell}}{|\mathcal{P}^{\mathrm{on}}_k|}(\mu_\ell-\mu_k) \\
&=\sum_{\ell\in L_k}\left(\frac{n^{\mathrm{on}}_{k\ell}}{|\mathcal{P}^{\mathrm{on}}_k|}-\frac{1}{|L_k|}\right)(\mu_\ell-\mu_k)+\left(\frac{1}{|L_k|}\sum_{\ell\in L_k}\mu_\ell-\mu_k\right). \tag*{(162)}
\end{aligned} ∣ P k on ∣ 1 j ∈ P k on ∑ ( μ y j − μ k ) = ℓ ∈ L k ∑ ∣ P k on ∣ n k ℓ on ( μ ℓ − μ k ) = ℓ ∈ L k ∑ ( ∣ P k on ∣ n k ℓ on − ∣ L k ∣ 1 ) ( μ ℓ − μ k ) + ( ∣ L k ∣ 1 ℓ ∈ L k ∑ μ ℓ − μ k ) . (162)
By Definition C.9 and the split in Eq. (154),
b k = 1 N k ∑ j ∈ P k o n ( μ y j − μ k ) + 1 N k ∑ j ∈ P k o f f ( μ y j − μ k ) . (163) b_k=\frac{1}{N_k}\sum_{j\in\mathcal{P}^{\mathrm{on}}_k}(\mu_{y_j}-\mu_k)+\frac{1}{N_k}\sum_{j\in\mathcal{P}^{\mathrm{off}}_k}(\mu_{y_j}-\mu_k). \tag*{(163)} b k = N k 1 j ∈ P k on ∑ ( μ y j − μ k ) + N k 1 j ∈ P k off ∑ ( μ y j − μ k ) . (163)
For P k o n = ∅ \mathcal{P}^{\mathrm{on}}_k = \varnothing P k on = ∅ , the first sum in Eq. (163) is zero. Otherwise, substituting Eq. (162) into that sum and applying the triangle inequality give
∥ b k ∥ ≤ 1 N k ∑ ℓ ∈ L k ∣ n k ℓ o n − ∣ P k o n ∣ ∣ L k ∣ ∣ ∥ μ ℓ − μ k ∥ + 1 N k ∑ j ∈ P k o f f ∥ μ y j − μ k ∥ + ∣ P k o n ∣ N k ∥ 1 ∣ L k ∣ ∑ ℓ ∈ L k μ ℓ − μ k ∥ ≤ δ c a p + δ c o n s , \begin{aligned}
\lVert b_k\rVert&\le\frac{1}{N_k}\sum_{\ell\in L_k}\left|n^{\mathrm{on}}_{k\ell}-\frac{|\mathcal{P}^{\mathrm{on}}_k|}{|L_k|}\right|\lVert\mu_\ell-\mu_k\rVert+\frac{1}{N_k}\sum_{j\in\mathcal{P}^{\mathrm{off}}_k}\lVert\mu_{y_j}-\mu_k\rVert\\
&\quad+\frac{|\mathcal{P}^{\mathrm{on}}_k|}{N_k}\left\lVert\frac{1}{|L_k|}\sum_{\ell\in L_k}\mu_\ell-\mu_k\right\rVert\\
&\le\delta_{\mathrm{cap}}+\delta_{\mathrm{cons}},
\end{aligned} ∥ b k ∥ ≤ N k 1 ℓ ∈ L k ∑ n k ℓ on − ∣ L k ∣ ∣ P k on ∣ ∥ μ ℓ − μ k ∥ + N k 1 j ∈ P k off ∑ ∥ μ y j − μ k ∥ + N k ∣ P k on ∣ ∣ L k ∣ 1 ℓ ∈ L k ∑ μ ℓ − μ k ≤ δ cap + δ cons ,
since the first two terms are the bracket in Eq. (159) at node k k k and ∣ P k o n ∣ ≤ N k |\mathcal{P}^{\mathrm{on}}_k|\le N_k ∣ P k on ∣ ≤ N k . The same bound holds when P k o n = ∅ \mathcal{P}^{\mathrm{on}}_k=\varnothing P k on = ∅ , because only the off-path sum then remains. Taking the maximum over non-root k k k gives the second bound in Eq. (161) . Since ∣ ξ k ∣ ≤ ξ max |\xi_k|\le\xi_{\max} ∣ ξ k ∣ ≤ ξ m a x for every non-root k k k , the triangle inequality gives
max k ≠ r o o t ∥ b k − ξ k b ∥ ≤ max k ≠ r o o t ∥ b k ∥ + ξ max ∥ b ∥ ≤ δ c a p + ( 1 + ξ max ) δ c o n s , \max_{k\ne\mathrm{root}}\lVert b_k-\xi_k b\rVert\le\max_{k\ne\mathrm{root}}\lVert b_k\rVert+\xi_{\max}\lVert b\rVert\le\delta_{\mathrm{cap}}+(1+\xi_{\max})\delta_{\mathrm{cons}}, k = root max ∥ b k − ξ k b ∥ ≤ k = root max ∥ b k ∥ + ξ m a x ∥ b ∥ ≤ δ cap + ( 1 + ξ m a x ) δ cons ,
so δ c a p + ( 1 + ξ max ) δ c o n s \delta_{\mathrm{cap}}+(1+\xi_{\max})\delta_{\mathrm{cons}} δ cap + ( 1 + ξ m a x ) δ cons is a caption bias bound by Eq. (107) . Under Assumption 4.4, δ c o n s = 0 \delta_{\mathrm{cons}}=0 δ cons = 0 , so b = 0 b=0 b = 0 and this value reduces to δ c a p \delta_{\mathrm{cap}} δ cap for any coefficients.
By Lemma L.1 , we can use δ c a p + ( 1 + ξ max ) δ c o n s \delta_{\mathrm{cap}}+(1+\xi_{\max})\delta_{\mathrm{cons}} δ cap + ( 1 + ξ m a x ) δ cons as the additive bias value in Lemma F.1 . The next corollary gives the resulting transfer conditions. Within this bound, the uneven-coverage term vanishes for leaf-uniform on-path captions, the off-path term vanishes for path-consistent captions, and δ c o n s \delta_{\mathrm{cons}} δ cons vanishes under Assumption 4.4.
Corollary L.2 (HAS transfer under imbalanced captions and inexact prototypes). Suppose Assumption 4.1 holds, exactly n n n images are sampled per leaf, ρ min > 0 \rho_{\min}>0 ρ m i n > 0 , and the observed captions are deterministic with N k > 0 N_k>0 N k > 0 for every non-root node k k k . Assumption 4.4 need not hold. Fix ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) and λ > 0 \lambda>0 λ > 0 . For the loss under consideration, compute ξ max \xi_{\max} ξ m a x from its coefficients, assume the corresponding case below, and let τ \tau τ denote its threshold.
Linear loss. Assume D ϵ l i n + δ c a p + ( 1 + ξ max ) δ c o n s < ρ min / 4 D^{\mathrm{lin}}_{\epsilon}+\delta_{\mathrm{cap}}+(1+\xi_{\max})\delta_{\mathrm{cons}}<\rho_{\min}/4 D ϵ lin + δ cap + ( 1 + ξ m a x ) δ cons < ρ m i n /4 and set
τ : = 8 ( D ϵ l i n + δ c a p + ( 1 + ξ max ) δ c o n s ) ρ min . (164) \tau:=\frac{8\left(D^{\mathrm{lin}}_{\epsilon}+\delta_{\mathrm{cap}}+(1+\xi_{\max})\delta_{\mathrm{cons}}\right)}{\rho_{\min}}. \tag*{(164)} τ := ρ m i n 8 ( D ϵ lin + δ cap + ( 1 + ξ m a x ) δ cons ) . (164) InfoNCE. Assume D ϵ n c e > 0 D^{\mathrm{nce}}_{\epsilon}>0 D ϵ nce > 0 , λ ≥ λ 0 ( ϵ ) \lambda\ge\lambda_0(\epsilon) λ ≥ λ 0 ( ϵ ) , and 2 D ϵ n c e + δ c a p + ( 1 + ξ max ) δ c o n s < ρ min / 4 2D^{\mathrm{nce}}_{\epsilon}+\delta_{\mathrm{cap}}+(1+\xi_{\max})\delta_{\mathrm{cons}}<\rho_{\min}/4 2 D ϵ nce + δ cap + ( 1 + ξ m a x ) δ cons < ρ m i n /4 , and set
τ : = 8 ( 2 D ϵ n c e + δ c a p + ( 1 + ξ max ) δ c o n s ) ρ min . (165) \tau:=\frac{8\left(2D^{\mathrm{nce}}_{\epsilon}+\delta_{\mathrm{cap}}+(1+\xi_{\max})\delta_{\mathrm{cons}}\right)}{\rho_{\min}}. \tag*{(165)} τ := ρ m i n 8 ( 2 D ϵ nce + δ cap + ( 1 + ξ m a x ) δ cons ) . (165) Then, with probability at least 1 − ϵ 1-\epsilon 1 − ϵ , all of the following hold simultaneously:
a k ≠ 0 a_k\ne0 a k = 0 for every non-root node k k k ;
every admissible triplet ( u , v , w ) (u,v,w) ( u , v , w ) with γ u v w > τ \gamma_{uvw}>\tau γ uv w > τ satisfies cos ( a u , a v ) > cos ( a u , a w ) \cos(a_u,a_v)>\cos(a_u,a_w) cos ( a u , a v ) > cos ( a u , a w ) ;
H A S \mathit{HAS} HAS -Violt x t ( u ) ≤ H A S ^{\mathrm{txt}}(u)\le\mathit{HAS} txt ( u ) ≤ HAS -Violi m g ( u ) ^{\mathrm{img}}(u) img ( u ) for every non-root non-leaf node u u u , with H A S \mathit{HAS} HAS -Violt x t ( u ) ^{\mathrm{txt}}(u) txt ( u ) computed on { a k } k ≠ r o o t \{a_k\}_{k\ne\mathrm{root}} { a k } k = root .
Proof. By Lemma L.1 , δ c a p + ( 1 + ξ max ) δ c o n s \delta_{\mathrm{cap}}+(1+\xi_{\max})\delta_{\mathrm{cons}} δ cap + ( 1 + ξ m a x ) δ cons is a caption bias bound, so we can take δ b i a s \delta_{\mathrm{bias}} δ bias equal to it. By Eq. (109) , D ϵ b i a s D^{\mathrm{bias}}_{\epsilon} D ϵ bias then equals D ϵ l i n + δ b i a s D^{\mathrm{lin}}_{\epsilon}+\delta_{\mathrm{bias}} D ϵ lin + δ bias for the linear loss and 2 D ϵ n c e + δ b i a s 2D^{\mathrm{nce}}_{\epsilon}+\delta_{\mathrm{bias}} 2 D ϵ nce + δ bias for InfoNCE. In both cases, it is less than ρ min / 4 \rho_{\min}/4 ρ m i n /4 by assumption, and τ b i a s \tau^{\mathrm{bias}} τ bias equals the threshold τ \tau τ in Eq. (164) or Eq. (165) . The remaining InfoNCE hypotheses, D ϵ n c e > 0 D^{\mathrm{nce}}_{\epsilon}>0 D ϵ nce > 0 and λ ≥ λ 0 ( ϵ ) \lambda\ge\lambda_0(\epsilon) λ ≥ λ 0 ( ϵ ) , also hold by assumption. Applying Lemma F.1 with this δ b i a s \delta_{\mathrm{bias}} δ bias therefore gives parts (i)–(iii) at τ \tau τ with probability at least 1 − ϵ 1-\epsilon 1 − ϵ .
Partial Ancestor Omission
The deterministic bound above applies to each realized caption assignment. To study the random imbalance caused by ancestor omission, we fix the image sample and independently retain each occurrence of a concept caption with the same probability across leaves. Conditional on the retained count, the positive set is a uniform sample of the original leaf-uniform captions. We can therefore bound its expected bias separately from the effect of a smaller N k N_k N k on the stochastic transfer term.
Lemma L.3 (Caption bias under partial ancestor omission). Suppose Assumption 4.4 holds, and fix a non-root node k k k with subtree radius R k : = max ℓ ∈ L k ∥ μ ℓ − μ k ∥ R_k := \max_{\ell\in L_k}\lVert\mu_\ell- \mu_k \rVert R k := max ℓ ∈ L k ∥ μ ℓ − μ k ∥ . Suppose that, before omission, the positive set of k k k contains equally many images from each leaf in L k L_k L k , and that each of these images independently keeps caption k k k with the same probability. Then, for every m ≥ 1 m \ge1 m ≥ 1 with Pr ( N k = m ) > 0 \Pr(N_k = m) > 0 Pr ( N k = m ) > 0 , the caption bias of the retained positive set satisfies
E [ b k ∣ N k = m ] = 0 , E [ ∥ b k ∥ ∣ N k = m ] ≤ R k m . (166) \mathbb{E}[b_k \mid N_k = m] = 0,\qquad\mathbb{E}[\lVert b_k \rVert\mid N_k = m] \le\frac{R_k}{\sqrt{m}}. \tag*{(166)} E [ b k ∣ N k = m ] = 0 , E [∥ b k ∥ ∣ N k = m ] ≤ m R k . (166)
Proof. Condition on N k = m N_k = m N k = m , and let P k f u l l \mathcal{P}_k^{\mathrm{full}} P k full be the positive set of k k k before omission. Since every image keeps caption k k k independently with the same probability, the retained positive set P k \mathcal{P}_k P k is a uniformly random m m m -subset of P k f u l l \mathcal{P}_k^{\mathrm{full}} P k full . Write x j : = μ y j − μ k x_j := \mu_{y_j} - \mu_k x j := μ y j − μ k for j ∈ P k f u l l j \in\mathcal{P}_k^{\mathrm{full}} j ∈ P k full , so that ∥ x j ∥ ≤ R k \lVert x_j\rVert\le R_k ∥ x j ∥ ≤ R k and
1 ∣ P k f u l l ∣ ∑ j ∈ P k f u l l x j = 1 ∣ L k ∣ ∑ ℓ ∈ L k μ ℓ − μ k (equal image counts across L k ) = 0 (by Assumption 4.4). \begin{aligned}
\frac{1}{\lvert\mathcal{P}_k^{\mathrm{full}}\rvert}\sum_{j\in\mathcal{P}_k^{\mathrm{full}}}x_j
=
\frac{1}{\lvert L_k\rvert}\sum_{\ell\in L_k}\mu_\ell-\mu_k
\qquad\text{(equal image counts across }L_k\text{)} \\
&=0
\qquad\text{(by Assumption 4.4).}
\end{aligned} ∣ P k full ∣ 1 j ∈ P k full ∑ x j = ∣ L k ∣ 1 ℓ ∈ L k ∑ μ ℓ − μ k (equal image counts across L k ) = 0 (by Assumption 4.4).
Hence
E [ b k ∣ N k = m ] = E [ 1 m ∑ j ∈ P k x j ∣ N k = m ] (by Definition C.9) = 1 ∣ P k f u l l ∣ ∑ j ∈ P k f u l l x j (each image is retained with probability m / ∣ P k f u l l ∣ ) = 0 (by the zero average above). \begin{aligned}
\mathbb{E}[b_k\mid N_k=m]
=
\mathbb{E}\left[
\left.
\frac{1}{m}\sum_{j\in\mathcal{P}_k}x_j
\right|N_k=m
\right]
\qquad\text{(by Definition C.9)} \\
&=
\frac{1}{\lvert\mathcal{P}_k^{\mathrm{full}}\rvert}
\sum_{j\in\mathcal{P}_k^{\mathrm{full}}}x_j
\qquad\text{(each image is retained with probability }m/\lvert\mathcal{P}_k^{\mathrm{full}}\rvert\text{)} \\
&=0
\qquad\text{(by the zero average above).}
\end{aligned} E [ b k ∣ N k = m ] = E m 1 j ∈ P k ∑ x j N k = m (by Definition C.9) = ∣ P k full ∣ 1 j ∈ P k full ∑ x j (each image is retained with probability m / ∣ P k full ∣ ) = 0 (by the zero average above).
Sampling without replacement has variance at most that of sampling with replacement, so
E [ ∥ b k ∥ ∣ N k = m ] ≤ E [ ∥ b k ∥ 2 ∣ N k = m ] (by Jensen’s inequality) ≤ 1 m ⋅ 1 ∣ P k f u l l ∣ ∑ j ∈ P k f u l l ∥ x j ∥ 2 (variance with replacement, zero average) ≤ R k m (since ∥ x j ∥ ≤ R k ). \begin{aligned}
\mathbb{E}[\lVert b_k\rVert\mid N_k=m]
\le
\sqrt{\mathbb{E}[\lVert b_k\rVert^2\mid N_k=m]}
\qquad\text{(by Jensen’s inequality)} \\
&\le
\sqrt{
\frac{1}{m}\cdot\frac{1}{\lvert\mathcal{P}_k^{\mathrm{full}}\rvert}
\sum_{j\in\mathcal{P}_k^{\mathrm{full}}}\lVert x_j\rVert^2
}
\qquad\text{(variance with replacement, zero average)} \\
&\le
\frac{R_k}{\sqrt{m}}
\qquad\text{(since }\lVert x_j\rVert\le R_k\text{).}
\end{aligned} E [∥ b k ∥ ∣ N k = m ] ≤ E [∥ b k ∥ 2 ∣ N k = m ] (by Jensen’s inequality) ≤ m 1 ⋅ ∣ P k full ∣ 1 j ∈ P k full ∑ ∥ x j ∥ 2 (variance with replacement, zero average) ≤ m R k (since ∥ x j ∥ ≤ R k ).
Omission can also lower N k N_k N k , which affects the stochastic transfer term through the realized N min N_{\min} N m i n and ξ max \xi_{\max} ξ m a x . If the resulting labels remain path-consistent, are independent of the image variations, and keep N k > 0 N_k > 0 N k > 0 for every non-root node k k k , we can condition on them and apply Corollary L.2. The off-path term in δ c a p \delta_{\mathrm{cap}} δ cap is then zero, as is δ c o n s \delta_{\mathrm{cons}} δ cons under Assumption 4.4.
With leaf-dependent retention, E [ b k ∣ N k = m ] \mathbb{E}[b_k\mid N_k=m] E [ b k ∣ N k = m ] need not vanish. For each realized path-consistent assignment, we can still use Lemma L.1 to bound the caption bias by the δ c a p \delta_{\mathrm{cap}} δ cap of that assignment, whose off-path term is zero.
Total Caption Omission: Collapse of Uncaptioned Concepts
When N k = 0 N_k = 0 N k = 0 , the caption mean is undefined, so the transfer corollary does not apply. We analyze the optimizer directly to determine how uncaptioned concepts affect H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S .
Proposition L.4 (Collapse of uncaptioned concepts). Fix the image features and observed captions, let λ > 0 \lambda> 0 λ > 0 , and define the uncaptioned non-root concepts by
U : = { k ∈ V ∖ { r o o t } : N k = 0 } . (167) U := \{k \in V \setminus\{\mathrm{root}\}: N_k = 0\}. \tag*{(167)} U := { k ∈ V ∖ { root } : N k = 0 } . (167) Under either the linear contrastive loss of Eq. (5) or the InfoNCE loss of Eq. (9) , every pair of distinct concepts in U U U has equal optimal text embeddings. For every non-root non-leaf node u u u , write
O u : = V ∖ ( Sub ( u ) ∪ Anc ( u ) ) \mathcal{O}_u := V \setminus(\operatorname{Sub}(u) \cup\operatorname{Anc}(u)) O u := V ∖ ( Sub ( u ) ∪ Anc ( u )) for its off-branch set. If the optimal text embeddings are nonzero at every non-root node, then every such u u u with U ∩ Desc ( u ) ≠ ∅ U \cap\operatorname{Desc}(u) \ne\varnothing U ∩ Desc ( u ) = ∅ satisfies
H A S - Viol t x t ( u ) ≥ ∣ U ∩ O u ∣ ∣ O u ∣ × 100 % . (168) \mathcal{H}\mathcal{A}\mathcal{S}\text{-}\operatorname{Viol}^{\mathrm{txt}}(u) \ge\frac{\lvert U \cap\mathcal{O}_u\rvert}{\lvert\mathcal{O}_u\rvert} \times100\%. \tag*{(168)} H A S - Viol txt ( u ) ≥ ∣ O u ∣ ∣ U ∩ O u ∣ × 100%. (168)
In particular, H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S cannot hold if some non-root non-leaf node u u u has both U ∩ Desc ( u ) ≠ ∅ U \cap\operatorname{Desc}(u) \ne\varnothing U ∩ Desc ( u ) = ∅ and U ∩ O u ≠ ∅ U \cap\mathcal{O}_u \ne\varnothing U ∩ O u = ∅ . Under the linear contrastive loss, every uncaptioned concept k k k also has the exact embedding
a k l i n , ∗ = − N 2 ( M − 1 ) λ z ˉ . (169) \mathbf{a}^{\mathrm{lin},*}_k = -\frac{N}{2(M-1)\lambda}\,\bar{\mathbf{z}}. \tag*{(169)} a k lin , ∗ = − 2 ( M − 1 ) λ N z ˉ . (169)
Proof. Fix either loss and write { a m } m ∈ [ M ] \{\mathbf{a}_m\}_{m\in[M]} { a m } m ∈ [ M ] for its optimal text embeddings. Let k ≠ k ′ k \ne k' k = k ′ belong to U U U . Exchanging a k \mathbf{a}_k a k and a k ′ \mathbf{a}_{k'} a k ′ leaves the objective unchanged. Neither index occurs in a positive term or a text-to-image term, the image-to-text terms sum symmetrically over concept indices, and the ridge penalty assigns the same coefficient to both embeddings. The linear loss has a unique minimizer by Proposition D.1, and the InfoNCE loss has a unique minimizer by Lemma E.1. The swapped optimizer must therefore equal the original optimizer, so a k = a k ′ \mathbf{a}_k = \mathbf{a}_{k'} a k = a k ′ .
Suppose every non-root a m \mathbf{a}_m a m is nonzero, and fix a non-root non-leaf node u u u with U ∩ Desc ( u ) ≠ ∅ U \cap\operatorname{Desc}(u) \ne\varnothing U ∩ Desc ( u ) = ∅ . Choose v ∈ U ∩ Desc ( u ) v \in U \cap\operatorname{Desc}(u) v ∈ U ∩ Desc ( u ) . For every w ∈ U ∩ O u w \in U \cap\mathcal{O}_u w ∈ U ∩ O u , the equality above gives a w = a v \mathbf{a}_w = \mathbf{a}_v a w = a v , and therefore
s u t x t ≤ cos ( a u , a v ) = cos ( a u , a w ) , s_u^{\mathrm{txt}} \le\cos(\mathbf{a}_u,\mathbf{a}_v) = \cos(\mathbf{a}_u,\mathbf{a}_w), s u txt ≤ cos ( a u , a v ) = cos ( a u , a w ) ,
where s u t x t s_u^{\mathrm{txt}} s u txt is the descendant threshold in Definition C.7 at zero slack. Thus every w ∈ U ∩ O u w \in U \cap\mathcal{O}_u w ∈ U ∩ O u is counted in H A S - Viol t x t ( u ) \mathcal{H}\mathcal{A}\mathcal{S}\text{-}\operatorname{Viol}^{\mathrm{txt}}(u) H A S - Viol txt ( u ) , which proves Eq. (164). If U ∩ Desc ( u ) U \cap\operatorname{Desc}(u) U ∩ Desc ( u ) and U ∩ O u U \cap\mathcal{O}_u U ∩ O u are both nonempty, the right-hand side of Eq. (164) is positive, so H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S cannot hold. If any non-root optimal text embedding is zero, the required cosine is undefined, so H A S \mathcal{H}\mathcal{A}\mathcal{S} H A S cannot hold in that case either.
For the additional linear characterization, fix an uncaptioned concept k k k . Then Y k j = N k = 0 Y_{kj} = N_k = 0 Y k j = N k = 0 for every j ∈ [ N ] j \in[N] j ∈ [ N ] , so Eq. (58) gives
B k j = 1 2 ( 0 − 1 M − 1 ) + 1 2 ( 0 − 0 N − 1 ) = − 1 2 ( M − 1 ) . B_{kj} = \frac{1}{2}\left(0-\frac{1}{M-1}\right) + \frac{1}{2}\left(0-\frac{0}{N-1}\right)
= -\frac{1}{2(M-1)}. B k j = 2 1 ( 0 − M − 1 1 ) + 2 1 ( 0 − N − 1 0 ) = − 2 ( M − 1 ) 1 .
By Proposition D.1, the optimal embedding is therefore
a k l i n , ∗ = 1 λ ∑ j = 1 N B k j z j (by Eq. (59)) = − 1 2 ( M − 1 ) λ ∑ j = 1 N z j (substituting B k j ) = − N 2 ( M − 1 ) λ z ˉ (by the definition of z ˉ ), \begin{aligned}
\mathbf{a}^{\mathrm{lin},*}_k &= \frac{1}{\lambda}\sum_{j=1}^{N} B_{kj}\mathbf{z}_j && \text{(by Eq. (59))}\\
&= -\frac{1}{2(M-1)\lambda}\sum_{j=1}^{N}\mathbf{z}_j && \text{(substituting }B_{kj}\text{)}\\
&= -\frac{N}{2(M-1)\lambda}\,\bar{\mathbf{z}} && \text{(by the definition of }\bar{\mathbf{z}}\text{),}
\end{aligned} a k lin , ∗ = λ 1 j = 1 ∑ N B k j z j = − 2 ( M − 1 ) λ 1 j = 1 ∑ N z j = − 2 ( M − 1 ) λ N z ˉ (by Eq. (59)) (substituting B k j ) (by the definition of z ˉ ),
which is Eq. (165).