Compositionality with Primitives: A Unified View from Euclidean to Hyperbolic Vision-Language Models

Introduction

Recognizing a sliced banana after seeing only sliced bread and whole bananas requires disentangling the attributes sliced and whole from the objects bread and banana, and reasoning over them independently. This ability reflects the broader principle of compositionality, a fundamental cognitive skill of humans clearly arising in language [14], whereby complex concepts are constructed from simpler, reusable ones [24]. At its core, it underpins the ability to generalize robustly to novel compositions without requiring direct supervision [14].

In Machine Vision, this paradigm is formalized as compositional zero-shot learning (CZSL) [20, 21]: given training images depicting a closed set of attribute-object pairs, the aim is to enable a model to recognize novel compositions of known concepts without having seen them during training. Open-world CZSL [18] extended this setting to the full Cartesian product of attributes and objects, significantly increasing task difficulty, and accounting for the long-tailed distribution of real-world generalization. Existing solutions typically attempt to learn compositional representations using conditional formulations of attribute-object representations [9, 35, 36], or learning compositional soft-prompt [17, 22], visual proxies [41] or invariant representations [42].

Recent work on Geodesically Decomposable Embeddings (GDE) [2] provides a different perspective: vision embeddings of pre-trained vision-language models already exhibit compositional semantic structures. These can be disentangled to perform compositional zero-shot classification. By projecting representations into the tangent space at the Fréchet mean of the embedding distribution of a dataset and performing a factorization step, attribute and object representations emerge as vector directions that embed basic visual concepts, called primitives. These can be recombined by vector arithmetic in the tangent space, thus synthesizing the representation of unseen object-attribute pairs and enabling their recognition. However, the focus in [2] was limited to CLIP spherical embeddings [28]. It remains unclear whether this view of compositionality extends to other space geometries, i.e. spaces with flat (Euclidean) or negative (hyperbolic) curvature, and whether explicit composition of primitives into pairs is necessary to achieve open-world compositional generalization. This question gains particular relevance for hyperbolic models [5, 23], where embeddings lie on the Lorentz space and representations encode compositional hierarchy in a tree-like fashion.

In this work, we place primitives at the center of the analysis and revisit compositional reasoning in VLMs. By means of novel compositional scoring functions, we demonstrate that primitives can be used directly to perform compositional inference without the need of explicitly constructing (unseen) pair representations. We demonstrate improved open-world compositional generalization, across Euclidean, spherical, and hyperbolic representation spaces, which hold different geometric and data distribution properties (see Figure 1). To perform this extensive investigation, we build upon the GDE framework and leverage its formulation beyond the original spherical setting of CLIP [28], to a broad family of models across geometries. We investigate Euclidean DINOv2 [1, 19] and hyperbolic VLMs (MERU [5], HyCoCLIP [23]), by utilizing corresponding manifold mapping functions between these geometries and the tangent space.

We also develop a formulation of primitives directly on the embedding manifold as the Fréchet means of samples sharing an attribute or object, and call them intrinsic primitives. We thus study manifold-native primitives as well as tangent-space GDE primitives, investigating how latent space geometry influences compositional abilities of models.

Across experiments, we find that compositional structure is a general emergent property of pre-trained models, showing through both GDE and intrinsic primitive representations across several CZSL benchmarks (MIT-States, VAW-CZSL, Clothing16K, UT-Zappos50K), with stronger evidence for datasets that contain samples with higher semantic variance. We further show that primitive-based inference is beneficial for improving recognition under group bias in the data through experiments on benchmark datasets for group robustness (Waterbirds, CelebA) using solely the object primitives, decoupled from the attribute primitives. The observed substantial and consistent increase of worst group accuracy indicates that biases and spurious correlations induced by attributes can be removed with primitive-based reasoning. The contributions of this work are therefore summarized as:

• (i) Primitives as the basis for open-world compositional learning and generalization. We show that pretrained models consistently organize representation into primitives, either as tangent-space directions or intrinsic points on manifolds, which encode basic visual concepts of attributes and objects. We develop new compositional scoring functions and demonstrate with extensive experiments that primitives support direct CZSL inference and open-world generalization, making explicit attribute-object pair construction unnecessary.

• (ii) Understanding of compositional power of pretrained models in relation to manifold geometry and dataset intrinsic variance. We find that, while compositional structure is geometry-agnostic, its task-wise optimal representation depends on the underlying geometry: tangent-space primitives are sufficient in Euclidean and spherical spaces, whereas hyperbolic representations benefit from intrinsic primitives aligned with their space geometry. We further shed light on the role that CZSL datasets and their intrinsic variance have in factorizing compositional primitives in pretrained models.

Related Work

Compositional zero-shot learning in VLMs. Early compositional zero-shot learning (CZSL) methods modeled attribute-object relations through shared embedding spaces [10, 38]. More recent studies have shifted attention to the compositional properties of Vision-Language models and highlighted that contrastive VLMs inherently struggle with text-based composition during standard inference, often behaving as a “bag of words” [34, 39]. This vulnerability is further underscored by benchmarks like SugarCrepe [8], which demonstrate that VLMs frequently rely on statistical hacks rather than genuine compositionality. In order to improve the compositional reasoning abilities of VLMs, subsequent research has explored adaptation strategies based on soft-prompt tuning [17, 22], or learning conditional attribute-object representations, where attribute frequency is coupled with the most likely co-occurring objects [9, 35, 36].

In [2], it was hypothesised that the pre-trained vision encoder of VLMs natively encodes compositional structures, induced by language-based pre-training. Thus, Geodesically Decomposable Embeddings (GDE) was proposed to factorize primitive representation directions in the tangent space at the Fréchet mean of the embeddings distribution, enabling training-free compositional reasoning via constructing unseen combination of primitives. The focus was restricted to spherical (CLIP-like) embeddings and to explicit construction of object-attribute pair representations for CZSL.

In this work, we fundamentally revisit the assumption that explicit pair re-composition is necessary for CZSL inference. We formulate new scoring functions for compositional classification that operate directly on independent object and attribute primitives. We show that leveraging these primitive representations is sufficient to recognize unseen compositions, bypassing pair recombination entirely. Furthermore, we expand the GDE framework to work across different geometries of the latent space (i.e. Euclidean, spherical and hyperbolic) and demonstrate the emergence of compositional structures in a broad family of visual encoders.

Our work differs from fine-tuning [17, 22] and conditional attribute learning [9, 35, 36], as we do not add learnable components and study the embedding capabilities of pretrained VLMs. It also shares a philosophical connection with Concept Bottleneck Models (CBMs) [13], which enforce intermediate, primitive-based concept representations for downstream inference. However, while CBMs require explicit bottleneck layers or fine-tuning with concept supervision, we analyze the compositional structure that inherently emerges within the embedding space of pretrained models.

Compositionality in hyperbolic VLMs. Most vision-language models operate on the unit hypersphere using cosine similarity. In contrast, hyperbolic models embed representations on the Lorentz model [5, 23, 27, 29, 37], where radius reflects semantic specificity, and geometry captures hierarchical structure: abstract concepts are placed closer to the origin, while more complex detailed concepts lie further away. HyCoCLIP [23] further incorporates compositional entailment supervision to align scene-object compositions with the hierarchical organization of the embedding space. While prior work has primarily focused on retrieval, entailment, and hierarchy preservation, we instead investigate how hyperbolic geometry shapes compositional reasoning. We show that primitives naturally align with the manifold structure, and that for group robustness intrinsic primitives provide a more semantically-coherent representation than tangent-space approximations in hyperbolic settings.

Compositionality in vision-only DINO. Compositional capabilities are not exclusive to cross-modal vision-language models. Self-supervised vision-only foundation models, such as DINOv2 [19], capture exceptional fine-grained, localized spatial semantics without using any language-signal during training. Recent efforts have explored post-hoc mechanisms to interface language with these frozen visual encoders. For instance, Talk2DINO [1] demonstrated that text embeddings can be aligned, via a learned projection, onto DINO patch-level features to guide open-vocabulary tasks. These findings suggest that a structured semantic organization emerges within the token embeddings of vision-only pre-trained encoders. Understanding the geometry underlying this intrinsic compositionality is therefore crucial, as such models provide powerful representations that are not influenced by language biases. In this work, we apply the GDE framework to DINOv2 and show that compositional structure naturally emerges in its (Euclidean) latent space, despite the absence of language supervision.

Preliminaries

In this section, we provide a brief overview of the mathematical background relevant to the notion of compositional embeddings and the GDE framework. Moreover, we provide the formalization of open-world CZSL and pair-based scoring functions used for recognition, and cover the foundational alignment of DINOv2 patch embeddings with text embeddings of CLIP done via the Talk2DINO framework.

Geodesically Decomposable Embeddings

VLMs encode images $x \in\mathcal{X}$ and their associated caption $y \in\mathcal{Y}$ into shared embedding spaces via specific encoders $\phi_{im}:\mathcal{X} \to\mathbb{R}^{d}$, $\phi_{tx}:\mathcal{Y} \to\mathbb{R}^{d}$. Let $\mathcal{M} \subset\mathbb{R}^{d}$ be the target $d$-dimensional representation manifold, with $\mathbf{u} \in\mathcal{M}$ being a generic point on the manifold. The text prompts can be created by a general template function $g:\mathbb{L}^{n} \to\mathcal{Y}\ \ \forall n \in\mathbb{N}$ where $\mathbb{L}^{n}$ are the $n$ concepts to make a template for.

In the Geodesically Decomposable Embeddings (GDE) framework [2], visual representations from pre-trained VLMs are decomposed into primitive directions corresponding to objects and attributes, which can then be recombined for CZSL. Given $N$ samples belonging to a dataset, their intrinsic (Fréchet) mean is $\boldsymbol{\mu} = \mathop{\operatorname{arg\,min}}_{\mathbf{u}\in\mathcal{M}} \sum_{i=1,\ldots,N} d_{\mathcal{M}}(\mathbf{u},\mathbf{u}_{i})^{2}$, where $d_{\mathcal{M}}$ denotes the geodesic distance on the manifold. Generally, a Riemannian gradient descent approach with a finite number of iterations is employed to compute the Fréchet mean on any manifold geometry (see Algorithm 1 in the supplementary material). Let $T_{\boldsymbol{\mu}}\mathcal{M}$ be the tangent space to the embedding manifold at $\boldsymbol{\mu}$, and $\mathbf{t}_{i}=\operatorname{Log}_{\boldsymbol{\mu}}(\mathbf{u}_{i}) \in T_{\boldsymbol{\mu}}\mathcal{M}$ the projection of an embedding vector to the tangent space via the logarithmic map. The function is locally invertible, and its inverse is the exponential map $\mathbf{u}_{i}=\operatorname{Exp}_{\boldsymbol{\mu}}(\mathbf{t}_{i}) \in\mathcal{M}$. The exact form of the map functions depends on the manifold geometry detailed in § 4.

Given a set of attributes $\mathcal{A}=\{a_{1},a_{2},\ldots,a_{n}\}$ and objects $\mathcal{O}=\{o_{1},o_{2},\ldots,o_{m}\}$, GDE estimates an independent, isolated primitive semantic direction $\mathbf{v}_{c}$ for any concept $c \in\mathcal{A}\cup\mathcal{O}$ via a weighted average of the projected tangent vectors containing that concept:

where $y_{i}$ denotes the attribute or object label associated with sample $i$, and each sample is labelled by a compositional pair $(a_{i},o_{i})\in\mathcal{A}\times\mathcal{O}$. Following [2], each embedding is weighted according to its alignment with the corresponding attribute–object text representation

where $\tau$ is a temperature parameter and $s_{\mathcal{M}}(\cdot,\cdot)$ is a similarity function defined as a geometry-specific inner product.

Open-World CZSL and Pair-based Scoring

In the open-world CZSL (OW-CZSL) setting [18], models are evaluated over the full Cartesian product $\mathcal{A}\times\mathcal{O}$ of possible compositions at test time, regardless of which attribute-object pairs were observed during training. This removes the disambiguation advantage of closed candidate sets and provides a stronger test for compositional generalization. For the zero-shot inference baseline, we prompt VLMs using a pre-defined text template in the form of ‘a photo of an {attribute} {object}’, which is embedded via the text encoder and used to score the vision embeddings $\phi_{im}(x)$. The scores computed using the function $s_{\mathcal{M}}$ are conventionally converted into probabilities through a softmax over the valid label space. The baseline Zero-Shot Pair paradigm is based on the scoring function (Eq. (2)), which calculates the probability assigned to an attribute-object pair and the prediction is obtained by selecting the label with max-probability (Eq. (3)).

Rather than using the text prompts, the GDE paper [2] proposed to combine all possible attribute-object primitive pairs and re-project them from the tangent space to the embedding manifold. For the GDE Pair paradigm, compositional recognition is achieved by using the scoring function:

Here we raise the question of whether exhaustive pair re-combination of primitives is necessary for OW-CZSL or if scores derived from single concept primitive are sufficient and perhaps better instead. However, we must first define scoring with primitives and their computation.

Primitive-based CZSL

The core focus of this work is the representation of visual concept as primitives (i.e. isolated attributes and objects representations), that form fundamental building blocks of compositional generalization. Rather than evaluating compositionality via pre-defined textual or visual pairs, we investigate how individual primitives emerge within different manifold geometries. Specifically, we study two distinct approaches for extracting these representations: intrinsic primitives computed directly on the underlying manifold via Riemannian operations, and tangent primitives extracted as localized linear vector directions via the Geodesically Decomposable Embeddings (GDE) framework. Under both paradigms, inference is performed zero-shot by scoring test images directly against these standalone primitives in their native mathematical domains. This eliminates the need for explicit primitive re-composition, currently used as standard in GDE. Figure 2 illustrates the differences between intrinsic primitives, GDE tangent primitives, and GDE-Pair compositions on the hyperbolic geometry as a representative case.

Intrinsic Primitives via Fréchet Means

We define an intrinsic primitive as the manifold mean of samples sharing the same attribute or object label. Given a set of embeddings $\{f(x_i)\}_{i=1}^{N}\subset\mathcal{M}$, and a concept $c$, it is computed as the Fréchet mean of the subset, $\mathcal{S}_c$, associated with that concept:

where $d_{\mathcal{M}}$ denotes the geodesic distance on the manifold. We compute separate intrinsic primitives for attributes and objects, using the Riemannian gradient descent for nonEuclidean spaces and the arithmetic mean for Euclidean ones, yielding respectively $\mu_a$ for each attribute $a \in\mathcal{A}$ and $\mu_o$ for each object $o \in\mathcal{O}$.

Given a test image $x$, denoting its embedding as $\phi^{x}_{\mathrm{im}}$ for brevity, its similarity to the primitives is computed directly on the underlying manifold using the geometry-specific similarity function $s_{\mathcal{M}}$. In particular, for a reference pair of object $(o)$ and attribute $(a)$ and text template $g_c := g(c)\;\; c \in\{a,o\}$ with its corresponding text embedding $\phi^{c}_{\mathrm{tx}}$, the Zero Shot Primitive score is defined as:

In contrast, our scoring function that uses intrinsic primitive individually does not rely on the text prompt embeddings:

The similarity functions used for the underlying spherical, Euclidean and hyperbolic manifolds are respectively

GDE Primitives on any Manifold

We exploit the GDE framework on different manifold geometries by integrating the logarithmic map $\operatorname{Log}_{\mu}:\mathcal{M}\to T_{\mu}\mathcal{M}$ and the exponential map $\operatorname{Exp}_{\mu}:T_{\mu}\mathcal{M}\to\mathcal{M}$ specifically defined for spherical, Euclidean and Lorentz spaces.

For Euclidean manifolds $\mathbb{R}^{d}$, the tangent space at every point is naturally identified with the manifold itself. In this case, the logarithmic and exponential maps reduce to ordinary subtraction and addition:

For spherical manifolds $\mathbb{S}^{d-1}$, the logarithmic and exponential maps are:

For hyperbolic models, with embeddings lying on the Lorentz space, $\mathbb{H}^{d+1}$ the logarithmic and exponential maps are:

where $\lVert\mathbf{t}\rVert_{\mathcal{L}} = \sqrt{\langle\mathbf{t},\mathbf{t}\rangle_{\mathcal{L}}}$.

The primitive directions $(\mathbf{v}_a,\mathbf{v}_o)$, obtained via the GDE framework, and the projected test image representation $\mathbf{t}_x = \operatorname{Log}_{\mu}(\phi_{\mathrm{im}}(x))$ all lie in the same tangent space $T_{\mu}\mathcal{M}$, where relationships are linear. We thus, in contrast to Eq. (4), define scoring functions for CZSL using individual GDE primitives, in which attribute and object scores are computed via standard cosine similarity in the tangent space as:

bypassing manifold projections entirely during inference.

Experiments

Setup

Datasets. We evaluate compositional recognition on four datasets spanning varying levels of semantic diversity and attribute-object coupling. MIT-States [10] contains 115 attributes paired with 245 objects, yielding 28,175 open-world compositions. VAW-CZSL [26, 31] is substantially larger, with 440 attributes and 541 objects (238,140 candidate compositions) sampled from images with larger scene context. UT-Zappos [38] is focused on footwear with 16 material attributes and 12 shoe types, while Clothing16K provides a broader clothing-domain benchmark with greater intra-category variability. Together, these datasets allow us to study compositionality under both highly diverse semantic settings (MIT-States, VAW-CZSL) and more constrained cases where sample variance is small and attributes are tightly coupled to object identity (UT-Zappos, Clothing16K).

Models and baselines. We investigate models operating in Euclidean, spherical, and hyperbolic embedding spaces. Spherical models include CLIP: ViT-B/16 (GRIT dataset) and ViT-L/14 (OpenAI). For Euclidean representations, we use DINOv2 with Talk2DINO [1] bridge to the text embedding of CLIP. This preserves the native Euclidean embeddings of DINOv2 and enables to compute baseline results with CLIP-style text prompts (see Appendix § 9). We also experiment with MERU [5] and HyCoCLIP [23] hyperbolic models. CLIP-B, MERU-B, and HyCoCLIP-B share the same ViT backbone and are pretrained on the GRIT corpus [25], thus allowing to attribute performance differences primarily to the underlying manifold geometry.

Prompts and inference paradigms. We test the compositional zero-shot learning performance of the considered vision-language models under different settings, namely:

(i) ZS-Pair: direct zero-shot classification using pair-based text prompts A photo of an {attribute} {object} as defined in Eq. (2);

(ii) ZS-Prim: primitive-level zero-shot inference using text prompts with attribute-focus (A photo of an {attribute}) and object-focus (A photo of an object) {object}) using the scoring function defined in Eq. (6);

• GDE-Pair: composing pair representations by adding visual primitives in the tangent space as $v_{a,o}=v_a+v_o$, and scoring the composed vector as defined in [2];

• GDE-Prim: scoring the GDE object and attribute primitives independently using our function defined in Eq. (10);

• Int-Prim: scoring the intrinsic object and attribute primitives independently using our function defined in Eq. (7).

Metrics. We compute the unseen pair accuracy (UPA) that evaluates the generalization to novel attribute-object compositions, and the seen pair accuracy (SPA) that measures the recognition performance on compositions observed during training (we report SPA in Appendix § 14). Models often exhibit a bias toward seen compositions, especially in the open-world setting where all feasible compositions are considered candidate predictions. Evaluating only one regime can provide a misleading picture of overall performance. Hence, as standard practice in OW-CZSL [18], we report the best harmonic mean of seen and unseen compositions:

that penalizes models that perform well on only one subset and therefore captures the ability to maintain strong recognition on both seen and unseen compositions simultaneously.

Results: CZSL via Primitives

Primitives improve open-world generalization. In Table 1 we reports the Harmonic Mean (HM) and Unseen Pair Accuracy (UPA) results of the CZSL performance of the considered pretrained models. Across models and datasets, replacing GDE pair composition with primitive-based inference consistently improves both metrics, indicating that explicit pair re-composition is not necessary for compositional generalization. On MIT-States, GDE primitives improve HM over GDE pairs by +1.53 for CLIP ViT-L/14, +1.66 for DINOv2, +1.89 for CLIP-B, +2.99 for MERU-B, and +3.66 for HyCoCLIP-B. The gains are often larger for UPA, including +3.85 for MERU-B and +5.04 for HyCoCLIP-B, indicating that primitives specifically strengthen open-world generalization. Notably, DINOv2 also benefits from primitive decomposition despite weak zero-shot pair prediction performance, improving HM by +1.65 on MIT-States and +0.90 on VAW-CZSL. As DINOv2 is trained without image-text contrastive supervision, this suggests that compositional semantics intrinsically emerge also from large-scale vision-only self-supervision. Across geometries, intrinsic primitives provide the most stable formulation, particularly in hyperbolic space, consistently with representing visual concept directly on the embedding manifold rather than through tangent-space approximations. Overall, primitive representations provide a stronger tool for open-world compositional generalization than explicit pair composition.

Model	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.
MIT-States
CLIP ViT-L/14	8.03	10.56	9.37	10.90	10.40	8.04	13.03	8.62	11.98	11.75
DINOv2 ViT-L/14	3.00	3.69	6.56	8.22	8.04	2.88	4.78	6.26	9.33	9.42
CLIP-B (GRIT)	4.61	4.70	4.75	6.64	6.38	4.24	5.36	4.80	7.18	7.12
MERU-B	4.68	4.74	3.58	6.57	6.36	4.40	5.54	3.40	7.25	7.40
HyCoCLIP-B	5.26	6.53	4.28	7.94	7.54	4.67	8.20	3.71	8.75	8.76
VAW-CZSL
CLIP ViT-L/14	0.74	1.51	1.18	1.77	1.77	0.79	1.59	1.32	2.07	2.22
DINOv2 ViT-L/14	0.72	1.22	1.30	2.05	2.20	0.77	1.57	1.63	2.60	2.81
CLIP-B (GRIT)	0.26	0.72	0.96	1.52	1.49	0.41	0.96	1.30	2.13	1.95
MERU-B	0.36	0.77	0.86	1.49	1.34	0.49	0.96	1.00	1.97	1.83
HyCoCLIP-B	0.29	1.14	0.85	1.47	1.40	0.33	1.20	1.08	2.09	1.87
Clothing16k
CLIP ViT-L/14	91.67	85.33	58.81	92.23	92.11	94.84	91.26	63.77	95.40	95.20
DINOv2 ViT-L/14	21.89	27.09	13.95	33.71	33.37	22.34	34.98	15.95	43.51	44.94
CLIP-B (GRIT)	87.64	79.19	64.60	89.27	88.92	89.94	83.36	72.17	93.22	92.59
MERU-B	86.25	76.99	67.57	88.73	88.07	90.14	83.09	74.35	92.32	92.52
HyCoCLIP-B	87.01	87.29	70.47	88.84	88.31	89.18	91.13	75.31	92.89	92.52

Dataset structure affects compositional reasoning. UT-

Zappos is a consistent exception to the advantages observed for primitive-based CZSL inference on the broader CZSL datasets. The dataset covers a narrow semantic domain and its samples have strongly coupled attribute-object relationship, with many attributes appearing across a restricted set of shoe categories (see Figure 6 in the Appendix). We therefore test whether effectiveness of primitive-based compositional inference depends on the semantic spread of the embedding distribution: we investigate the intrinsic dataset variance and causally link its value to compositional generalization performance. For embeddings $\{x_i\}_{i=1}^{N}$ from dataset $\mathcal{D}$ on manifold $\mathcal{M}$, we compute the intrinsic variance

where $\mu_{\mathcal{D}}$ is the Fréchet mean and $d_{\mathcal{M}}(\cdot,\cdot)$ is the geodesic distance. Figure 3 reveals a similar pattern across Euclidean, spherical, and hyperbolic representations: MIT-States, VAW-CZSL, and Clothing16K have substantially greater intrinsic variance than UT-Zappos. This suggests that primitive decomposition benefits from representations spanning sufficiently diverse semantic contexts, whereas low-variance distributions allow pair representations to exploit stronger dataset-specific attribute-object co-occurrence patterns.

We study the role of intrinsic dataset variance as a causal factor to explain compositional reasoning limitations. We apply a geometry-specific whitening operation (detailed in Appendix § 12) to the embeddings of the considered models of the UTZappos dataset, and denote the resulting representations as UTZ-W. Whitening has the effect of increasing the intrinsic variance of UTZ-W (similarly to that of the broader datasets) compared to that of the original UT-Zappos.

We report CZSL results before and after whitening in

Table 2: the HM and UPA values increase noticeably for both GDE- and intrinsic-primitive compositional inference, achieving comparable or largely superior performance in some cases than pair-based inference.

Model	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.
Harmonic Mean (%, ↑) / After Whitening
CLIP ViT-L/14	11.76 / –	9.61 / –	16.41 / 16.69	8.04 / 13.75	9.33 / 22.80
DINOv2 ViT-L/14	2.81 / –	0.38 / –	12.06 / 13.64	3.20 / 12.40	2.49 / 16.17
CLIP-B (GRIT)	2.33 / –	1.61 / –	10.81 / 13.53	9.04 / 12.79	8.95 / 14.65
MERU-B	3.38 / –	3.80 / –	10.39 / 12.79	8.99 / 8.80	9.36 / 16.32
HyCoCLIP-B	4.34 / –	3.94 / –	8.50 / 13.01	5.89 / 8.14	5.86 / 15.11
Unseen Pair Accuracy (%, ↑) / After Whitening
CLIP ViT-L/14	21.89 / –	29.19 / –	29.46 / 36.81	18.03 / 24.43	22.85 / 36.83
DINOv2 ViT-L/14	9.84 / –	11.85 / –	21.95 / 28.34	12.64 / 19.78	16.08 / 28.42
CLIP-B (GRIT)	16.45 / –	11.63 / –	16.50 / 21.01	13.11 / 15.44	13.91 / 21.21
MERU-B	8.41 / –	8.51 / –	16.45 / 25.17	11.26 / 15.55	14.60 / 25.12
HyCoCLIP-B	11.95 / –	14.49 / –	16.45 / 23.69	12.16 / 16.08	14.01 / 24.80

After whitening, intrinsic primitives achieve the highest HM for every model and the highest UPA for four of five models, falling only 0.05 points below the best MERU-B UPA results using pair-based inference. These results provide causal evidence that low intrinsic variance of the dataset (and its embedding distribution) contributes to the original failure mode of compositional inference. UT-Zappos is thus not a failure of primitive inference: its low-variance representation suppresses decomposition effectiveness.

Statistical Separation of GDE Primitives

A key premise of primitive-based inference is that the decomposition isolates object and attribute information into distinct semantic components. We test this by measuring the statistical dependence between object, attribute, and pair representations using Pearson cross-correlation (CORR) and mutual information (MI). We perform this analysis on the GDE primitives. The results show that, across datasets and model families, the dependence and mutual information between object and attribute primitive representations is consistently substantially lower than those of either object and pair or attribute and pair representations. For example, on MIT-States object and attribute primitive CORR remains between 0.444 and 0.473 across models, compared to 0.825–0.895 for object primitive and pair representation, and the MI estimates are 13.0 versus 119.4–320.7. We observe similar results across the considered CZSL datasets and model representations. We report full results in Appendix § 11.

The higher dependence that we consistently observe between the primitive representations and the full pair representation are expected and confirm the disentanglement quality of the factorization that we perform. Object and attribute primitives retain information from the composition from which they are derived, while exhibiting weak dependence on one another. These results provide evidence that primitive-based decomposition, as we study it, recovers a factorized semantic structure in which object and attribute information is meaningfully separated rather than simply reproducing the original pair representation, and thus providing a solid tool for compositional reasoning.

Group Robustness via Primitives

Pre-trained vision-language models achieve strong zero-shot performance under distribution shift, but remain sensitive to spurious correlations and subgroup imbalance [40]. Existing approaches to group robustness typically rely on group annotations, additional optimization, group-specific prompts, adaptation, or class reweighting [7, 11, 12, 30, 32, 33]. We instead ask whether the compositional structure already encoded in and emerging from pretrained representations can itself be used for robust inference. The results of the statistical disentanglement indicate that decomposed primitives separate object and attribute semantics, allowing inference to retain information relevant to the object while suppressing attribute directions that may induce spurious correlations. We thus evaluate decomposed primitive for robustness to group bias on Waterbirds [30] and CelebA [16]. We do not use group annotations, additional data, or further optimization. Specifically, we classify images using only the object primitives, retaining only the object-score component of Eq. (6), Eq. (7) and Eq. (10) for zero-shot, intrinsic, and GDE primitives, respectively. Table 3 shows that this simple inference-time decomposition consistently improves group robustness. Both GDE and intrinsic object primitives improve worst-group accuracy over zero-shot object inference while largely preserving, and in several cases improving, average accuracy. These gains arise without an explicit de-biasing objective or fairness supervision: they come solely from isolating semantic structure already present in the pretrained representations. This suggests that compositional factorization is not only useful for interpreting representations, but also provides a practical mechanism for reducing reliance on spurious attributes at inference time.

Model	Zero Shot			GDE-Primitive			Intrinsic Primitive
	Avg. Acc. $\uparrow$	WG. Acc. $\uparrow$	Gap $\downarrow$	Avg. Acc. $\uparrow$	WG. Acc. $\uparrow$	Gap $\downarrow$	Avg. Acc. $\uparrow$	WG. Acc. $\uparrow$	Gap $\downarrow$
Waterbirds
CLIP ViT-L/14	80.9	48.8	32.1	90.8	85.7	5.13	90.8	85.7	5.17
DINOv2 ViT-L/14	69.1	42.4	26.7	85.5	48.0	37.5	85.0	49.4	35.6
CLIP-B (GRIT)	58.4	36.9	21.5	77.1	64.4	12.7	77.0	64.4	12.7
MERU-B	58.3	31.7	26.6	63.0	39.0	24.0	77.1	62.6	14.6
HyCoCLIP-B	53.9	20.3	33.7	62.6	38.3	24.3	77.1	62.6	14.5
CelebA
CLIP ViT-L/14	75.3	69.2	6.16	87.9	86.9	1.00	87.8	86.8	1.00
DINOv2 ViT-L/14	13.7	0.07	13.6	77.7	73.1	4.54	77.4	73.8	3.57
CLIP-B (GRIT)	56.6	29.8	26.9	81.9	74.8	7.08	81.7	74.7	6.97
MERU-B	35.2	11.5	23.6	69.1	59.6	9.58	84.1	80.3	3.80
HyCoCLIP-B	34.3	6.46	27.8	69.4	57.3	12.1	84.6	79.7	4.95

Intrinsic primitives yield better group robustness in hyperbolic spaces. The group robustness gains are especially pronounced for MERU and HyCoCLIP, where intrinsic object primitives yield large improvements in both average and worst-group accuracy. We conjecture that this advantage is related to the radial organization of semantic abstraction in the hyperbolic space. Intrinsic object primitives lie closer to the origin than compound object–attribute representations, consistent with their encoding broader, less attribute-specific concepts. GDE primitives, by contrast, are obtained through a local linear approximation around the Fréchet mean and can therefore remain more influenced by the local neighborhood structure induced by dominant attribute or background correlations. This provides a plausible geometric explanation for the stronger robustness of intrinsic primitives that lay directly on the manifold. The radial organization of these representations is further explored in Appendix § 15.

Conclusion

This work establishes primitive decomposition as a unifying principle for compositional zero-shot learning (CZSL) across Euclidean, spherical, and hyperbolic vision–language models. By systematically analyzing models with diverse manifold geometries, we demonstrate that compositional structure consistently emerges in pretrained vision embeddings in the form of primitive visual concept representations, which can be directly exploited for compositional zero-shot inference. Crucially, we show that explicit attribute-object pair re-composition is not necessary for CZSL: independently reasoning over attribute and object primitives, with specifically-designed scoring functions, can effectively support compositional open-world generalization.

We demonstrate primitives as a novel paradigm for CZSL inference with results being consistent for the tangent space GDE primitives and intrinsic primitives. Across experiments, GDE primitives provide robust and transferable primitive representations, consistently outperforming pair-based compositions for open-world CZSL while remaining simple and geometry-agnostic. Intrinsic primitives offer a complementary, geometry-aware refinement, particularly effective in hyperbolic spaces, where operating directly on the manifold improves stability by avoiding local linearization errors. We further demonstrate benefits of primitive-based inference in group robustness tasks, where object-primitive representations allow to perform unbiased inference without model re-training, avoiding spurious correlations induced by attribute characteristics.

Finally, this work provides the first comprehensive and unified view of primitive-based CZSL, demonstrating it as an emerging property of existing pretrained models.

Acknowledgments

This publication is part of the project ROBUST: Trustworthy AI-based Systems for Sustainable Growth with project number KICH3.LTP.20.006, which is (partly) financed by the Dutch Research Council (NWO), Siemens Healthineers, and the Dutch Ministry of Economic Affairs and Climate Policy (EZK) under the program LTP KIC 2020–2023.

This publication is part of the project ReVision with file number VI.Vidi.243.176 of the research programme Vidi ENW which is (partly) financed by the Dutch Research Council (NWO) under the grant https://doi.org/10.61686/ERNED50610.

T.T. acknowledges the EU project ELSA-European Lighthouse on Secure and Safe AI, under the EU’s Horizon Europe Research and Innovation Programme (grant agreement no. 101070617)

This publication is part of the project HyperVision with file number VI.Vidi.243.085 of the research programme Vidi ENW which is (partly) financed by the Dutch Research Council (NWO) under the grant https://doi.org/10.61686/DCECG94893.

Riemannian Gradient Descent

Given a collection of manifold-valued embeddings $\{f(x_i)\}_{i=1}^{N} \subset\mathcal{M}$, we seek a representative point that minimizes the sum of squared geodesic distances to all samples. This point is known as the intrinsic mean (or Fréchet mean) and is defined as

where $d_{\mathcal{M}}(\cdot,\cdot)$ denotes the geodesic distance on the manifold $\mathcal{M}$. Unlike Euclidean averaging, optimization on a curved manifold requires moving along tangent spaces using the logarithmic and exponential maps. At an iterate $\mu^{(t)} \in\mathcal{M}$, each sample $f(x_i)$ is first mapped to the tangent space $T_{\mu^{(t)}}\mathcal{M}$ via the logarithmic map $\operatorname{Log}_{\mu^{(t)}}(f(x_i))$. The average tangent vector:

corresponds to the Riemannian gradient direction of the Fréchet objective. The iterate is then updated by mapping this tangent vector back onto the manifold using the exponential map:

where $\eta> 0$ is the step size. Algorithm 1 summarizes the complete optimization procedure.

Algorithm 1 Intrinsic Mean Computation

Require: Embeddings $\{f(x_i)\}_{i=1}^{N} \subset\mathcal{M}$, initial point $\mu^{(0)} \in\mathcal{M}$, step size $\eta$

1. for $t = 0,1,\ldots,T-1$ do

2. $\mathbf{v}^{(t)} \leftarrow\frac{1}{N}\sum_{i=1}^{N}\operatorname{Log}_{\mu^{(t)}}(f(x_i))$

3. $\mu^{(t+1)} \leftarrow\operatorname{Exp}_{\mu^{(t)}}\left(\eta\mathbf{v}^{(t)}\right)$

4. end for

5. return $\mu^{(T)}$

Dense Feature Extraction via Talk2DINO

Unlike CLIP, MERU, or HyCoCLIP that inherently produce aligned image-text features from joint text pairs, vision-only models like DINOv2 yield dense patch embeddings without text alignment. To construct a competitive text-conditioned compositional baseline for DINOv2 that matches the open-world zero-shot capabilities of vision-language models, we interface its Euclidean representation space via Talk2DINO [1]. Given an image $x$ and a textual query $q$, Talk2DINO produces patch embeddings $\{p_m\}_{m=1}^{M}$ together with a query embedding $q$. The spatial relevance map and the final pooled image representation are computed respectively as

where $\tau$ is a temperature parameter. This routing mechanism yields a single, unified Euclidean image representation conditioned dynamically on the queried attribute-object composition while strictly preserving DINOv2’s original underlying geometry.

DINO Embeddings: Euclidean or Spherical?

Talk2DINO provides a language wrapper to DINO, without modifying its native visual embeddings. This enables us to compute for DINO the for fair comparison. The core compositionality analysis is performed on the native DINO vision embedding and confirms that properly addressing manifold geometry is key to effective decomposition: Table 4 shows that reducing the embeddings to spherical space harms performance, due to wrong manifold assumption for primitive extraction.

Dataset	$\mathcal{M}$	Harmonic Mean (GDE Pair)	Harmonic Mean (GDE Prim.)	Harmonic Mean (Int. Prim.)	Unseen Pair Accuracy (GDE Pair)	Unseen Pair Accuracy (GDE Prim.)	Unseen Pair Accuracy (Int. Prim.)
MIT	euc	6.56	8.22	8.04	6.26	9.33	9.42
MIT	sph	2.17	4.70	4.85	1.87	4.61	4.75
AWA	euc	1.30	2.05	2.20	1.63	2.60	2.81
AWA	sph	0.28	0.68	0.72	0.28	0.65	0.71

GDE Primitive Concept Separation

To assess whether the GDE primitives capture concept structure that is separable in the model’s embedding space, we measure the statistical dependence between each pair of primitives using two complementary estimators: the Pearson cross correlation (CORR), and a canonical correlation estimator of mutual information (MI) computed with the bench-mi mutual information package [4]. CORR captures linear alignment between two sets of vectors, while MI additionally captures non-linear dependence structure that a purely linear estimator would miss. We report both for the three primitive pairings: object-attribute, object-pair, and attribute-pair. Our analysis is limited to GDE primitives.

Across datasets and models, in Table 5, we observe trends consistent with our expectations: the object-attribute pairing is consistently the lowest correlated and carries the least mutual information relative to object-pair and attribute-pair. This suggests that object and attribute representations are comparatively more disentangled from one another, whereas information about objects and attributes is each more entangled with pairwise (relational) structure than with each other.

		CLIP ViT-L/14		DINOv2 ViT-L/14		CLIP-B (GRIT)		HyCoCLIP-B		MERU-B
		CORR	MI	CORR	MI	CORR	MI	CORR	MI	CORR	MI
MIT-States	object-attribute	0.453	13.0	0.456	13.0	0.444	13.0	0.453	13.0	0.473	13.0
MIT-States	object-pair	0.825	191.0	0.895	320.7	0.831	148.7	0.839	119.7	0.843	119.4
MIT-States	attribute-pair	0.530	77.9	0.516	129.0	0.511	44.5	0.513	44.2	0.531	44.3
VAW-CZSL	object-attribute	0.187	7.2	0.183	7.2	0.195	6.9	0.191	6.9	0.173	6.9
VAW-CZSL	object-pair	0.575	38.1	0.713	61.0	0.565	28.1	0.565	26.6	0.576	29.9
VAW-CZSL	attribute-pair	0.203	14.9	0.228	13.2	0.215	16.5	0.213	11.0	0.203	10.9
UT-Zappos	object-attribute	0.098	0.8	0.090	0.8	0.067	0.8	0.067	0.8	0.069	0.8
UT-Zappos	object-pair	0.872	4.8	0.930	5.1	0.896	5.5	0.909	5.6	0.929	18.0
UT-Zappos	attribute-pair	0.338	4.0	0.174	3.4	0.264	3.0	0.241	3.0	0.208	3.1
Clothing16k	object-attribute	0.404	2.7	0.454	2.7	0.419	2.7	0.427	2.7	0.419	2.7
Clothing16k	object-pair	0.945	73.1	0.992	73.0	0.962	72.7	0.953	73.2	0.948	73.0
Clothing16k	attribute-pair	0.606	83.1	0.488	83.2	0.581	83.4	0.612	83.7	0.609	83.2

We also note that object-pair estimates are consistently higher than attribute-pair estimates across datasets and models. This points to embeddings being generally more aligned with object-level information than with attribute-level information, and suggests attribute-awareness may be an area where current embeddings are comparatively weaker, a direction that could benefit from further study.

Non-Euclidean Whitening

For Euclidean features, whitening can be applied directly in the ambient space. For spherical and hyperbolic features, we instead perform whitening in a tangent space, following the general principle of Principal Geodesic Analysis (PGA) [6]. The intrinsic mean $\mu$ is computed as described in Algorithm 1, after which the embeddings are mapped to the corresponding tangent space using the logarithmic map.

Spherical Whitening

For spherical embeddings $\{f(x_i)\}_{i=1}^{N} \subset\mathbb{S}^{d-1}$, we first compute the intrinsic mean $\mu$ and map each embedding to $T_{\mu}\mathbb{S}^{d-1}$ using $\operatorname{Log}_{\mu}$, as defined in Eq. (9). Whitening is then performed on the resulting tangent vectors.

A spherical-specific restriction arises from the cut locus. For the unit sphere, the logarithmic representation around $\mu$ is unique only for tangent vectors with norm smaller than $\pi$. Arbitrarily rescaling a tangent vector beyond this radius causes the exponential map to wrap around the sphere. Distinct tangent vectors can then map to the same or nearby spherical locations, which may destroy semantic information encoded by their relative positions. We therefore constrain

the norm of the projected tangent vector to remain within the injectivity radius with rescaling. This follows the same general tangent-space construction used in spherical PCA methods spherep [15].

Hyperboloid Whitening

For hyperbolic embeddings $\{f(x_i)\}_{i=1}^{N} \subset\mathbb{H}^{d+1}$, we again begin by computing an intrinsic mean $\mu$. Unlike the spherical case, the ambient hyperboloid coordinates are equipped with the Lorentz metric rather than the Euclidean metric. We therefore first apply a Lorentz isometry that maps $\mu$ to the canonical hyperbolic origin $\mathbf{o}$. This preserves hyperbolic distances while placing the tangent representation in a canonical coordinate system. Related use of hyperbolic isometries appears in HoroPCA [3]. After this transformation, the embeddings are mapped

Algorithm 3 Hyperboloid whitening
Require: Embeddings $\{f(x_i)\}_{i=1}^{N} \subset \mathbb{H}^{d+1}$
1: Compute intrinsic mean $\mu$ using Algorithm 1
2: Compute Lorentz isometry $B_{\mu}$ such that $B_{\mu}(\mu)=\mathbf{o}$
3: for $i=1,\ldots,N$ do
4: $\mathbf{u}_i \leftarrow B_{\mu}(f(x_i))$
5: $\mathbf{v}_i \leftarrow \operatorname{Log}_{\mathbf{o}}(\mathbf{u}_i)$
6: end for
7: Compute covariance $\mathbf{C}$ of $\{\mathbf{v}_i\}_{i=1}^{N}$
8: Compute whitening transform $\mathbf{W} \leftarrow \mathbf{C}^{-1/2}$
9: for $i=1,\ldots,N$ do
10: $\hat{\mathbf{v}}_i \leftarrow \mathbf{W}\mathbf{v}_i$
11: $\hat{f}(x_i) \leftarrow \operatorname{Exp}_{\mathbf{o}}(\hat{\mathbf{v}}_i)$
12: end for
13: return $\{\hat{f}(x_i)\}_{i=1}^{N}$

to $T_{\mathbf{o}}\mathbb{H}^{d+1}$ using the logarithmic map as defined in Eq. 10. At the canonical origin, the tangent coordinates provide an ordinary Euclidean representation of the local Riemannian metric, and standard covariance whitening can therefore be applied. The whitened tangent vectors are finally mapped back to the hyperboloid using the exponential map.

In contrast to the sphere, hyperbolic space has no finite cut locus. The tangent radius is therefore not subject to the same wrapping ambiguity.

Primitive Weighting Ablation

GDE primitive sample weights are scalar, carrying only confidence rather than semantic information. In Table 6, we provide additional results that ablate this design inherited from the baseline framework.

Dataset	Model	Harmonic Mean (%, ↑) Weighted	Harmonic Mean (%, ↑) Uniform	Unseen Pair Accuracy (%, ↑) Weighted	Unseen Pair Accuracy (%, ↑) Uniform
MIT	CLIP ViT-L/14	10.90	10.39	11.98	11.47
MIT	DINOv2 ViT-L/14	8.22	8.02	9.33	9.10
MIT	CLIP-B (GRIT)	6.64	6.12	7.18	6.62
MIT	MERU-B	6.57	6.59	7.25	7.14
MIT	HyCoCLIP-B	7.94	7.83	8.75	8.62
VAW	CLIP ViT-L/14	1.77	1.64	2.07	2.03
VAW	DINOv2 ViT-L/14	2.05	2.11	2.60	2.64
VAW	CLIP-B (GRIT)	1.52	1.37	2.13	2.01
VAW	MERU-B	1.49	1.43	1.97	1.87
VAW	HyCoCLIP-B	1.47	1.41	2.09	2.03

Extended Open-World CZSL Metrics

This section reports additional evaluation metrics for open-world compositional zero-shot learning beyond the harmonic mean discussed in the main paper. In addition to composed pair performance, we separately evaluate object-level, attribute-level. These metrics provide a more detailed view of compositional behaviour. Object and attribute accuracies isolate recognition performance on the underlying semantic primitives, helping distinguish whether improvements arise from better primitive identification or from stronger compositional reasoning.

Table 7 reports object and attribute accuracies across all datasets and geometries. Consistent with the main results, primitive-based inference generally improves both object and attribute recognition, particularly on datasets with greater semantic diversity such as MIT-States and VAW-CZSL. These gains indicate that primitive decomposition improves not only pair retrieval, but also the separability of the underlying semantic factors. Table 8 reports seen and overall pair accuracies.

Model	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.
MIT-States
CLIP ViT-L/14	47.33	47.33	43.79	43.68	43.94	15.76	20.05	18.38	20.57	20.68
DINOv2 ViT-L/14	32.70	29.70	40.13	40.25	40.22	8.49	11.97	16.00	18.45	18.41
CLIP-B (GRIT)	36.59	35.81	33.91	35.50	35.58	10.35	13.48	14.09	17.41	16.97
MERU-B	37.31	36.34	34.03	35.39	35.51	10.57	13.22	11.71	17.48	17.07
HyCoCLIP-B	39.79	39.72	37.41	39.01	39.15	10.95	16.26	12.00	18.09	17.71
VAW-CZSL
CLIP ViT-L/14	18.66	19.18	24.49	25.99	25.33	3.70	5.97	5.40	6.69	6.37
DINOv2 ViT-L/14	20.77	20.80	30.07	30.88	30.88	4.24	5.38	5.82	8.06	7.85
CLIP-B (GRIT)	13.94	15.58	20.22	22.29	21.78	3.19	4.39	5.55	6.72	6.32
MERU-B	14.54	15.60	20.91	22.41	22.13	2.92	4.38	5.20	6.80	6.03
HyCoCLIP-B	15.96	19.12	21.77	22.96	22.61	2.64	4.75	4.81	6.45	6.19
UT-Zappos
CLIP ViT-L/14	50.03	63.90	60.33	57.93	59.75	18.53	17.54	25.91	19.73	21.69
DINOv2 ViT-L/14	33.22	12.32	56.04	55.11	56.21	13.90	13.25	25.91	6.35	5.66
CLIP-B (GRIT)	36.65	29.89	49.83	47.39	50.62	11.77	10.16	18.50	23.58	22.10
MERU-B	21.86	17.19	50.48	51.41	50.27	13.62	16.64	18.91	21.45	22.10
HyCoCLIP-B	34.45	25.57	46.88	48.70	47.67	12.94	21.79	16.09	15.41	15.61
Clothing16k
CLIP ViT-L/14	97.10	93.06	84.50	97.48	97.39	95.11	93.29	64.28	95.17	95.11
DINOv2 ViT-L/14	78.76	65.63	84.32	79.81	78.99	22.12	36.89	11.34	32.35	32.20
CLIP-B (GRIT)	93.11	92.82	87.72	94.02	93.82	93.26	85.56	71.90	95.46	95.22
MERU-B	93.14	93.55	88.31	94.20	94.37	91.71	84.09	76.65	94.35	93.88
HyCoCLIP-B	92.73	92.88	89.25	94.08	94.55	93.20	94.32	78.23	95.43	94.52

Model	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.	ZS Pair	ZS Prim.	GDE Pair	GDE Prim.	Int. Prim.
MIT-States
CLIP ViT-L/14	29.03	22.23	31.18	25.21	25.88	15.76	20.05	18.38	20.57	20.68
DINOv2 ViT-L/14	16.64	9.87	27.23	20.63	19.96	8.49	11.97	16.00	18.45	18.41
CLIP-B (GRIT)	21.72	16.93	22.18	18.99	18.82	10.35	13.48	14.09	17.41	16.97
MERU-B	22.61	16.76	19.58	18.91	19.20	10.57	13.22	11.71	17.48	17.07
HyCoCLIP-B	23.87	15.50	21.97	21.34	20.97	10.95	16.26	12.00	18.09	17.71
VAW-CZSL
CLIP ViT-L/14	3.76	4.97	3.67	4.87	4.67	3.70	5.97	5.40	6.69	6.37
DINOv2 ViT-L/14	3.64	4.21	3.62	5.44	5.14	4.24	5.38	5.82	8.06	7.85
CLIP-B (GRIT)	2.12	2.80	2.88	3.37	3.39	3.19	4.39	5.55	6.72	6.32
MERU-B	2.31	3.22	2.49	3.79	3.44	2.92	4.38	5.20	6.80	6.03
HyCoCLIP-B	2.31	3.96	2.53	3.77	3.69	2.64	4.75	4.81	6.45	6.19
UT-Zappos
CLIP ViT-L/14	13.49	8.80	29.03	19.16	21.70	18.53	17.54	25.91	19.73	21.69
DINOv2 ViT-L/14	2.25	0.29	18.48	10.36	7.82	13.90	13.25	25.91	6.35	5.66
CLIP-B (GRIT)	1.76	1.76	19.35	22.09	22.39	11.77	10.16	18.50	23.58	22.10
MERU-B	3.23	4.11	21.41	22.78	23.07	13.62	16.64	18.91	21.45	22.10
HyCoCLIP-B	9.48	3.71	17.69	11.24	13.88	12.94	21.79	16.09	15.41	15.61
Clothing16k
CLIP ViT-L/14	98.72	91.30	87.21	98.98	98.72	92.41	86.90	58.86	92.76	92.62
DINOv2 ViT-L/14	48.59	42.46	52.94	52.17	57.54	19.22	28.42	9.00	30.65	30.53
CLIP-B (GRIT)	97.19	90.03	92.07	97.19	97.44	87.11	79.20	65.48	89.83	89.42
MERU-B	97.70	93.09	93.09	97.95	97.44	85.64	78.35	69.47	88.95	88.69
HyCoCLIP-B	97.19	94.12	93.86	97.95	97.44	86.55	87.81	71.26	89.75	89.33

Hierarchy of Hyperbolic Intrinsic Primitives

Hyperbolic embeddings encode hierarchy through radial distance from the origin, where larger radii correspond to more specific concepts. Driven by the strong group robustness results, and to further deepen the analysis of the properties of the intrinsic primitives, we examine the distributions of the radius of the representations of attributes, objects, and attribute-object compositions on the manifold of hyperbolic models.

We plot the computed distributions for HyCoCLIP in Figure 4 and for MERU in Figure 5 for the considered datasets. The embeddings of attribute-object compositions consistently occupy larger radii than their constituent primitives, consistently with the expected semantic hierarchy in hyperbolic spaces: compositions such as blue car are more specific than either primitives alone.

For MIT-States, VAW-CZSL, UT-Zappos, and Cloth-ing1K, attribute primitives tend to lie closer to the origin and thus represent more general concepts, shared across several objects. Object primitives instead occupy deeper regions of the hierarchy, configuring as slightly more specific and complex semantic concepts in the considered datasets. In contrast, Waterbirds and CelebA data shows the opposite trend, with attributes occupying larger radii than objects. This suggests that attributes (e.g., backgrounds in Waterbirds or facial characteristics in CelebA) add complexity and further specifications to the object concepts in the dataset, possibly inducing biases and dominating the semantic structure of the data. The results of this analysis align with those achieved for group robustness tests where intrinsic primitives show exceptional de-biasing performance for hyperbolic models: the use of object primitives allows to disentangle the complexity induced by the attributes that cause bias and spurious correlations arising in the zero-shot predictions.

From a general perspective, these observations support the effectiveness of intrinsic primitive inference. Primitive embeddings occupy denser and better-covered regions of the representation space, whereas compositions correspond to deeper and sparser concepts. By decomposing representations into primitives rather than relying on locally estimated GDE primitives, intrinsic primitive means remain closer to the training distribution and better capture the dominant semantic structure of the dataset. This leads to more stable compositional representations and improved robustness under distribution shifts.

Example Images for Datasets

We provide qualitative examples from each dataset used in our compositional zero-shot learning experiments in Figure 6. These samples are drawn from the test splits and illustrate the diversity of attribute-object compositions present in each domain.

Overall, most datasets exhibit substantial variation in both attributes and objects, reflecting the combinatorial nature of the task. In particular, Clothing16K contains a broad range of clothing categories and attributes, including different garment types, textures, and visual appearances, as well as a wider variety of poses and viewpoints. This results in higher visual diversity and more challenging compositional generalization settings.

In contrast, UT-Zappos is considerably more constrained, consisting exclusively of footwear categories. Moreover, the visual presentation of objects in this dataset tends to be relatively consistent, with limited variation in pose and viewpoint. This reduced geometric and semantic diversity aligns with our observation of lower compositional variability and helps explain its comparatively different behavior in our results.