betaFeedback

WorksComputer Science

Closing the Modality Gap via Simplex-Constrained Representations

Abstract

While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent modality gap. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere, Sd−1S^{d-1}, to the probability simplex, Δd−1Δ^{d-1}. Although both manifolds share the same intrinsic dimension, the simplex acts as a structural prior; its nonnegative, unit-mass constraints serve as a cross-modal calibration that forces both modalities to ground themselves in the same coordinate system. By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid ℓ2\ell_2 gap by 97–99% across five multimodal retrieval benchmarks. This confirms that simplicial geometry induces genuine semantic alignment, matching or exceeding the retrieval performance of standard unconstrained models.

§: https://github.com/ServiceNow/retreever/tree/main/experiments/modality_gap

Correspondence: [email protected]

Introduction

Contrastive multimodal models embed diverse data streams, such as text and audio Wu et al. [34]; Elizalde et al. [7] or text and images Radford et al. [24]; Jia et al. [14], into a shared latent space. Despite objectives designed to align matched pairs, these models often converge to representations in which modality-wise embedding distributions remain systematically offset, a phenomenon known as the modality gap Liang et al. [20]. The practical consequences of this gap remain debated. Several studies report that large modality gaps can miscalibrate cross-modal similarity, degrading retrieval and distorting zero-shot decision boundaries Zhang et al. [37]; Schrodi et al. [27]. At the same time, a nonzero gap may encode modality-specific structure, and indiscriminately collapsing it can perturb within-modality geometry and harm downstream performance Liang et al. [20]; Ramasinghe et al. [25]; Schrodi et al. [27]. Driven by this tension, a range of strategies have been proposed to manage cross-modal separation, including modifying the contrastive objective Fahim et al. [8], architectural fusion via shared attention layers Singh et al. [29]; Chen et al. [4], adversarial alignment with gradient reversal Sofer et al. [30], and alternative embeddings motivated by hierarchical structures Ramasinghe et al. [25]; Desai et al. [5].

We take a geometric view of the modality gap and show that output geometry alone can close it dramatically, without modifying the pretrained backbone and with fewer than 1% trainable parameters. Our key move is to replace the default spherical embedding domain Sd−1S^{d-1} with a simplex-constrained one, Δd−1={p∈R≥0d:∑ipi=1}\Delta^{d-1} = \{p \in\mathbb{R}_{\ge0}^{d} : \sum_i p_i = 1\}. Although the sphere and simplex are both (d−1)(d-1)-dimensional, they encode very different symmetries. On the sphere, cosine similarity is invariant to rotations and sign flips, leaving many equivalent solutions in which modalities can match pairs while remaining distributionally separated. On the simplex, these symmetries largely vanish: embeddings are nonnegative and sum to one, forcing both modalities to allocate a shared probability budget over the same coordinates. With softmax adapters, the representation also becomes invariant to global logit shifts, providing an additional cross-modal normalization.

Concretely, we freeze LAION-CLAP Elizalde et al. [7] for audio–text and FLAVA Singh et al. [29] for image–text, attach lightweight shared adapters, and train with an in-batch InfoNCE objective (as in DPR)van den Oord et al. (2018); Karpukhin et al. [15], changing only the scoring function: cosine similarity for Euclidean outputs versus negative Total Variation distance for simplex outputs produced by a softmax projection. To isolate the effect of geometry from architectural changes, we evaluate matched-capacity baselines. We start from frozen backbones and train lightweight Linear and MLP adapters with Euclidean outputs, and we also train Linear/MLP adapters of the same architecture whose outputs are followed by a softmax projection to the simplex. To disentangle output geometry from shared cross-attention, we use an Euclidean variant of ReTreever Gupta et al. [11] (we call it XAttnXAttn) that keeps the same learned-query cross-attention but outputs cosine-scored Euclidean embeddings, and compare it to ReTreever’s simplex outputs Gupta et al. [11]. Finally, to contrast coarse-to-fine strategies across geometries, we include MRL as an Euclidean baseline Kusupati et al. [18] with Retreever.

Our contributions are as follows: (i) We propose simplex-constrained adapters for multimodal retrieval and show that they reduce the centroid ℓ2\ell_2 modality gap by 97–99% across five audio–text and image–text benchmarks while matching or improving NDCG@10. (ii) We develop a geometry-adjusted gap measure, RelGap, which normalizes centroid separation by intra-modality spread, enabling fair comparison of modality gap across Euclidean and simplicial embedding domains. (iii) Using matched-capacity and matched-form baselines, we isolate the effect of output geometry and show that simplex constraints (rather than additional architectural capacity) drive gap closure. (iv) We analyze coarse-to-fine behavior and dimensional scaling, showing that Euclidean gaps grow with representation dimension whereas simplex gaps remain small across resolutions.

Related Work

Modality gap. Liang et al. [20] systematically characterized the modality gap in CLIP and connected it to geometric effects such as the cone effect and initialization-induced asymmetries. Subsequent work has refined both the diagnosis and the proposed mechanisms: Fahim et al. [8] formalize a contrastive view of the gap, Schrodi et al. [27] relate it to information imbalance and compositional failures, and Yaras et al. [35] further links initialization and optimization dynamics to persistent separation. Empirically, the gap has been associated with miscalibrated cross-modal similarities that can hurt retrieval Ma et al. [22], distort zero-shot decision boundaries Zhang et al. [37], and hinder compositional generalization Schrodi et al. [27], though its impact can be regime-dependent and remains an active point of debate.

Managing or reducing cross-modal separation. Prior approaches intervene either in the training objective or the architecture. On the objective side, methods modify contrastive learning to explicitly encourage alignment or reweight supervision Fahim et al. [8], use adversarial signals such as gradient reversal to suppress modality-specific features Sofer et al. [30], or propose structured/group-wise alignment strategies Grassucci et al. [10]. On the architectural side, fusion models introduce shared cross-modal processing, e.g., shared transformer layers or attention blocks, as in UNITER Chen et al. [4], FLAVA Singh et al. [29], VATT Akbari et al. [2], and BLIP-2 Li et al. [19]. A complementary line adds lightweight adapters or projection heads on top of frozen backbones (e.g., CLIP-Adapter Gao et al. [9] and parameter-efficient tuning methods such as LoRA Houlsby et al. [12]; Hu et al. [13]), aiming to improve alignment with minimal trainable parameters. Finally, hyperbolic spaces have been explored to better capture hierarchical structure Ramasinghe et al. [25]; Desai et al. [5].

Simplex and distributional representations. Simplex-valued representations naturally model nonnegative mixture weights, with roots in compositional data analysis Aitchison [1] and topic models such as LDA Blei et al. [3]. In deep learning, simplex/Dirichlet parameterizations appear in uncertainty modeling Sensoy et al. [28] and in neural collapse, where late-stage features exhibit simplex-like structure Papyan et al. [23]. Most relevant here, ReTreever uses softmax-normalized coarse-to-fine simplex embeddings for retrieval and is competitive with Euclidean multi-resolution baselines such as MRL Gupta et al. [11]; Kusupati et al. [18].

Method

Adapters and output domains

Let f(⋅)f(\cdot) denote a frozen multimodal encoder (audio–text or image–text). For an input uu (text or audio/image), the encoder produces a pooled embedding hu∈Rd\mathbf{h}_u \in\mathbb{R}^d. We attach a lightweight shared adapter gθg_\theta (either linear or a 2-layer MLP) to obtain logits zu=gθ(hu)\mathbf{z}_u = g_\theta(\mathbf{h}_u). We study two output domains using the same adapter architecture, isolating the effect of geometry from adapter capacity. Euclidean (spherical) outputs. We form normalized embeddings eu=zu/∥zu∥2∈Sd−1\mathbf{e}_u = \mathbf{z}_u/\lVert\mathbf{z}_u\rVert_2 \in S^{d-1} and score pairs with cosine similarity, scos(u,v)=eu⊤evs_{\mathrm{cos}}(u,v) = \mathbf{e}_u^\top\mathbf{e}_v. Simplex-constrained outputs. We project logits to the simplex via softmax, pu=softmax⁡(zu)∈Δd−1\mathbf{p}_u = \operatorname{softmax}(\mathbf{z}_u) \in\Delta^{d-1}, and score pairs with a Total Variation–based score, sTV(u,v)=−TV⁡(pu,pv)=−12∥pu−pv∥1s_{\mathrm{TV}}(u,v) = -\operatorname{TV}(\mathbf{p}_u,\mathbf{p}_v) = -\frac{1}{2}\lVert\mathbf{p}_u-\mathbf{p}_v\rVert_1. Beyond Linear/MLP adapters, we evaluate cross-attention and multi-resolution baselines. XAttn learns 100 shared 1024-d queries that cross-attend to token-level representations and are mean-pooled into a cosine-scored Euclidean embedding, while ReTreever uses similar mechanism to produce multi-resolution simplex embeddings via softmax Gupta et al. [11]. We also train MRL Kusupati et al. [18], which jointly optimizes multiple prefix slices of a single Euclidean embedding under the same retrieval objective.

Training objective

We train the adapter(s) using a symmetric in-batch contrastive loss (DPR-style) Karpukhin et al. [15]. Let a minibatch contain paired examples {(ti,xi)}i=1B\{(t_i,x_i)\}_{i=1}^{B}, where tt is text and xx is audio or image. Given a scoring function s(⋅,⋅)s(\cdot,\cdot) (cosine for Euclidean outputs, negative TV for simplex), we optimize

L=12(LNCEt→x+LNCEx→t),\mathcal{L} = \frac{1}{2}\left(\mathcal{L}_{\mathrm{NCE}}^{t\to x} + \mathcal{L}_{\mathrm{NCE}}^{x\to t}\right),

where:

LNCEt→x=−1B∑i=1Blog⁡exp⁡(s(ti,xi))∑j=1Bexp⁡(s(ti,xj)),LNCEx→t=−1B∑i=1Blog⁡exp⁡(s(xi,ti))∑j=1Bexp⁡(s(xi,tj)).\mathcal{L}_{\mathrm{NCE}}^{t\to x} = -\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(s(t_i,x_i))}{\sum_{j=1}^{B}\exp(s(t_i,x_j))}, \qquad\mathcal{L}_{\mathrm{NCE}}^{x\to t} = -\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(s(x_i,t_i))}{\sum_{j=1}^{B}\exp(s(x_i,t_j))}.

Sphere vs. simplex: symmetries and distance scaling

Switching the output domain from the sphere to the simplex changes both (i) the invariances of the representation and (ii) the baseline behavior of distances in high dimensions. These geometric properties are important for interpreting modality gaps across domains.

Symmetries and calibration. With Euclidean outputs, we normalize embeddings and use cosine similarity, so representations lie on the unit sphere Sd−1S^{d-1}. Cosine similarity is invariant to orthogonal transformations (rotations/reflections) and sign flips, which implies a large family of equivalent solutions: matched pairs can be aligned while modality-wise distributions remain offset. With simplex outputs, embeddings are constrained to Δd−1={p∈R≥0d:∑ipi=1}\Delta^{d-1} = \{\mathbf{p} \in\mathbb{R}_{\ge0}^d : \sum_i p_i = 1\}, which removes most of this freedom by enforcing nonnegativity and unit mass. As a result, both modalities must express representations as comparable allocations of a shared probability budget over the same coordinates. When the simplex embedding is produced via p=softmax⁡(z)\mathbf{p} = \operatorname{softmax}(\mathbf{z}), the representation is also invariant to global logit shifts: softmax⁡(z+c1)=softmax⁡(z)\operatorname{softmax}(\mathbf{z}+c\mathbf{1}) = \operatorname{softmax}(\mathbf{z}), providing an additional cross-modal normalization.

Baseline distances. On Sd−1S^{d-1}, random unit vectors become nearly orthogonal as dd increases, and pairwise distances concentrate around a constant: for x,y∼Unif⁡(Sd−1)\mathbf{x},\mathbf{y} \sim\operatorname{Unif}(S^{d-1}), E∥x−y∥22=2\mathbb{E}\lVert\mathbf{x}-\mathbf{y}\rVert_2^2 = 2 and ∥x−y∥2≈2\lVert\mathbf{x}-\mathbf{y}\rVert_2 \approx\sqrt{2} in high dimension. On the simplex, typical distances depend on how dense the probability vectors are. A convenient reference model is the symmetric Dirichlet distribution p,q∼iidDir⁡(α,…,α)\mathbf{p},\mathbf{q} \overset{\mathrm{iid}}{\sim} \operatorname{Dir}(\alpha,\ldots,\alpha), for which E∥p−q∥22=2(α+1)αd+1≈2d\mathbb{E}\lVert\mathbf{p}-\mathbf{q}\rVert_2^2 = \frac{2(\alpha+1)}{\alpha d+1} \approx\frac{2}{d}. This highlights two regimes: (i) dense simplex embeddings (fixed α\alpha) where typical distances shrink as O(1/d)O(1/\sqrt{d}), and (ii) sparse embeddings (α=c/d\alpha= c/d with dα=cd\alpha= c held constant) where distances plateau at ≈2/(c+1)\approx2/(c+1), analogous to the spherical concentration phenomenon.

Estimating the regime and comparing gaps across domains. To connect this characterization to learned simplex embeddings, we estimate a concentration parameter via moment matching. Let s=E∥p∥22s = \mathbb{E}\lVert\mathbf{p}\rVert_2^2 for learned p∈Δd−1\mathbf{p} \in\Delta^{d-1}; then α^=1−ss\hat{\alpha} = \frac{1-s}{s}. We use α^\hat{\alpha} to determine whether learned simplex representations lie in a dense (contracting) or sparse (plateauing) regime as resolution/dimension increases (see § 6.4). Finally, because raw centroid distances can be influenced by the baseline distance scale of the embedding domain, we report a geometry-adjusted ReLGAP metric (Eq. (1)) that normalizes centroid separation by intra-modality spread, enabling fairer gap comparisons between Euclidean and simplex outputs.

Experimental Setup

Baselines

Linear/MLP Adapters We use Linear and 2-layer MLP adapters in Euclidean space, and the same adapters followed by a softmax projection to the simplex. These paired baselines keep architecture fixed and isolate the effect of output geometry.

Cross-attention adapters XAttn uses ReTreever’s shared cross-attention mechanism, in which a fixed set of learned queries attends over token representations, but produces Euclidean embeddings scored with cosine similarity. ReTreever uses the same cross-attention backbone but outputs simplex-valued representations via softmax and is scored with a TV-based measure Gupta et al. [11]. Comparing XAttn to ReTreever isolates the effect of simplex output constraints from the effect of shared cross-attention.

Coarse-to-fine representations MRL Kusupati et al. [18] provides a multi-resolution Euclidean baseline (Matryoshka embeddings) enabling coarse-to-fine retrieval in Euclidean space. ReTreever provides the corresponding coarse-to-fine retrieval mechanism on the simplex Gupta et al. [11]. We use this pairing to study how geometry affects gap scaling with effective dimension (see § 6.4).

Experimental Setup

Datasets. We evaluate on five cross-modal retrieval benchmarks spanning audio–text and image–text alignment. For audio–text, we use Clotho Drossos et al. [6], AudioCaps Kim et al. [16], and SoundDescs Koepke et al. [17]. For image–text, we use MS-COCO Captions Lin et al. [21] and Flickr30k Young et al. [36]. All datasets consist of paired items with bidirectional retrieval (text↔audio or text↔image).

Backbones and adapters. For audio–text experiments we freeze LAION-CLAP Wu et al. [33], and for image–text we freeze FLAVA Singh et al. [29]. On top of each frozen backbone we evaluate seven configurations: the frozen encoder without any adapter; Linear or MLP adapters with Euclidean outputs scored by cosine similarity; Linear/MLP adapters followed by a softmax projection to the simplex and scored with a TV-based measure; XAttn (shared cross-attention with Euclidean outputs and cosine similarity); and ReTreever (shared cross-attention with multi-resolution simplex outputs) Gupta et al. [11]. For coarse-to-fine analysis (§ 6.4), we additionally compare MRL Kusupati et al. [18] (multi-resolution Euclidean) against ReTreever. All adapters are trained for 200K steps with AdamW (lr =4×10−4= 4 \times10^{-4}), batch size 64, and a learnable temperature τ\tau, totaling fewer than 1% trainable parameters.

Metrics. We measure the modality gap using centroid ℓ2\ell_2, silhouette Rousseeuw [26], and a geometry-adjusted relative gap metric. Centroid ℓ2\ell_2 is ∥μA−μB∥2\lVert\boldsymbol{\mu}_A-\boldsymbol{\mu}_B\rVert_2, where μA\boldsymbol{\mu}_A and μB\boldsymbol{\mu}_B are modality centroids. Silhouette summarizes cluster separability via the standard per-point coefficient (b−a)/max⁡(a,b)(b-a)/\max(a,b), averaged over points, where aa is the mean intra-modality distance and bb is the mean nearest-modality distance. To make gaps comparable across Euclidean and simplex outputs, we report RelGap, which normalizes centroid separation by intra-modality spread:

RelGap⁡(A,B)=∥μA−μB∥212(E∥xA−μA∥22+E∥xB−μB∥22)(1)\operatorname{RelGap}(A,B)=\frac{\lVert\mu_A-\mu_B\rVert_2}{\sqrt{\frac{1}{2}\left(\mathbb{E}\lVert x_A-\mu_A\rVert_2^2+\mathbb{E}\lVert x_B-\mu_B\rVert_2^2\right)}} \tag*{(1)}

We report retrieval quality using NDCG@10 for both directions (text→\toaudio/audio→\totext or text→\toimage/image→\totext).

Results

Simplex projection collapses the gap while preserving retrieval

Table 1–Table 2 and Figure 2 summarize results across five benchmarks (audio–text and image–text). Across all datasets, switching an otherwise identical adapter from Euclidean output to a softmax projection onto the simplex collapses the modality gap by two orders of magnitude, without sacrificing retrieval.

AdapterClotho Gap ℓ2\ell_2↓Clotho Gap RG↓Clotho Gap Sil↓Clotho Retrieval T2A↑Clotho Retrieval A2T↑SoundDescs Gap ℓ2\ell_2↓SoundDescs Gap RG↓SoundDescs Gap Sil↓SoundDescs Retrieval T2A↑SoundDescs Retrieval A2T↑AudioCaps Gap ℓ2\ell_2↓AudioCaps Gap RG↓AudioCaps Gap Sil↓AudioCaps Retrieval T2A↑AudioCaps Retrieval A2T↑
CLAP.640.748.197.259.436.488.521.117.273.250.363.380.067.596.507
Linear.514.689.167.373.453.458.554.133.410.418.366.468.139.584.439
MLP.323.353.064.358.436.220.264.033.520.514.266.285.049.587.448
XAttn.450.443.126.353.432.309.331.051.464.456.297.321.053.592.452
Lin.+Sfx†.007.468.127.382.470.021.196.022.437.436.012.217.030.584.440
MLP+Sfx†.011.336.089.361.438.010.179.018.507.499.006.180.012.597.451
ReTreever.020.181.026.381.464.012.097.006.535.542.011.113.000.590.444

Table 1. Audio-text modality gap and retrieval. Sfx = softmax. Simplex methods shaded. Modality gap measured as Centroid ℓ2\ell_2, RelGap (RG, Eq. (1)), silhouette (Sil) (lower is better (↓)), Retrieval as NDCG@10 (higher is better (↑)). Best per column in bold. †Lin.+Sfx and MLP+Sfx: per dataset, the better of the paper’s configuration (L2-normalised adapter input, no logit norm) and the same adapter trained on un-normalised input with a LayerNorm on the logits before the softmax; the other rows are unchanged.

AdapterCOCO Gap ℓ2↓\ell_2\downarrowCOCO Gap RG↓\downarrowCOCO Gap Sil.↓\downarrowCOCO Retrieval T2I↑\uparrowCOCO Retrieval I2T↑\uparrowFlickr30k Gap ℓ2↓\ell_2\downarrowFlickr30k Gap RG↓\downarrowFlickr30k Gap Sil.↓\downarrowFlickr30k Retrieval T2I↑\uparrowFlickr30k Retrieval I2T↑\uparrow
FLAVA.8101.00.336.630.529.731.891.285.801.695
Linear.647.732.212.711.644.543.598.151.858.817
MLP.411.438.088.688.615.246.277.036.828.792
XAttn.615.666.182.708.641.527.563.136.857.819
Lin.+Sfx†.021.475.113.691.593.017.328.071.844.787
MLP+Sfx†.012.239.029.686.606.022.324.052.824.783
ReTreever.007.221.026.694.613.005.129.008.837.799

Table 2. Image-text modality gap and retrieval (FLAVA encoder). Sfx = softmax. Simplex methods shaded. Modality Gap measured by Centroid ℓ2\ell_2, RelGap (RG, Eq. (1)), silhouette (Sil) and Retrieval as NDCG@10. Best per column in bold. †Lin.+Sfx and MLP+Sfx: the same adapters trained on un-normalised adapter input (no L2 normalisation), no norm on the logits; the other rows are unchanged.

Figure 2. Modality gap (centroid ℓ2\ell_2) vs. avg. retrieval NDCG@10. Softmax methods (blue) cluster at low gap with competitive retrieval; non-softmax (red) spread across higher gaps.

On frozen backbones, centroid ℓ2\ell_2 gaps are large (0.36–0.81 across datasets). Adding a softmax to the same Linear/MLP adapter reduces centroid ℓ2\ell_2 to 0.005–0.020, corresponding to a 97–99% reduction. For example, COCO drops from 0.810 (frozen) to 0.009 (MLP+Sfx), and Flickr30k drops from 0.731 (frozen) to 0.007 (MLP+Sfx). Figure 2 shows the same pattern geometrically: softmax (simplex) methods cluster at low gap while remaining competitive in average NDCG@10.

Because the absolute distance scale differs across output domains, we also report RelGap (Eq. (1)), which normalizes centroid separation by intra-modality spread. Simplex methods consistently achieve the lowest RelGap, indicating that cross-modal alignment improves beyond any uniform rescaling of distances. For instance, on SoundDescs, RelGap drops from 0.521 (frozen) to 0.179 (MLP+Sfx) and to 0.097 (ReTreever), and on AudioCaps it drops from 0.380 (frozen) to 0.180 (MLP+Sfx) and to 0.113 (ReTreever).

Finally, gap reduction does not trade off with retrieval quality. On Clotho, Lin.+Sfx improves T2A NDCG@10 from 0.259 to 0.382, and on SoundDescs, ReTreever achieves the best retrieval scores (0.535/0.542) while maintaining a very small gap. Overall, simplex constraints remove modality-specific distributional offsets while preserving (and sometimes improving) the neighborhood structure relevant for retrieval.

Separating cross-attention from output geometry

Separating cross-attention from output geometry

To separate architectural effects from output geometry, we compare Euclidean and simplex models that share cross-attention. XAttn and ReTreever use similar shared cross-attention mechanism, but XAttn outputs Euclidean embeddings (cosine) whereas ReTreever outputs simplex embeddings (softmax, TV-based scoring).

Across datasets, XAttn retains substantial gaps comparable to other Euclidean baselines, while ReTreever achieves dramatically smaller centroid gaps and RELGap. For example, on SoundDescs, XAttn has centroid ℓ2=0.309\ell_2 = 0.309 and RELGap = 0.331, whereas ReTreever reaches 0.012 and 0.097. On Flickr30k, XAttn has centroid ℓ2=0.527\ell_2 = 0.527 and RELGap = 0.563, whereas ReTreever reaches 0.005 and 0.129. Figure 4 shows the same trend on SoundDescs and AudioCaps: methods with Euclidean outputs (including large cross-attention models such as WAVE-7B Tang et al. [31]) retain large relative gaps, while simplex outputs consistently reduce them. This isolates output geometry as the dominant factor in gap closure, beyond any effect introduced by shared attention layers.

Figure 4. Relative gap (RelGap, Eq. (1)) across adapter types on SoundDescs (left) and AudioCaps (right). Non-softmax methods (red) vs. softmax methods (blue). Lower is better. Softmax methods consistently reduce RelGap.

Coarse-to-fine scaling: Euclidean gaps grow with dimension, simplex gaps stay small

Figure 3 compares MRL (Euclidean coarse-to-fine) and ReTreever (simplex coarse-to-fine) across representation dimensions. MRL exhibits increasing modality gaps as resolution/dimension grows, while ReTreever maintains small gaps across resolutions. On AudioCaps, MRL’s centroid gap grows from 0.032 at 2-d to 0.534 at 256-d (17×), whereas ReTreever remains below 0.06 throughout.

Figure 3. Coarse-to-fine modality gap: MRL (Euclidean, red) vs. ReTreever (simplex, blue) across representation dimensions. Euclidean gaps grow; simplex gaps stay flat.

Learned simplex embeddings lie in the sparse regime

Distance behavior on the simplex depends on the density of the probability vectors (§ 3.3). We therefore estimate the Dirichlet concentration α^\hat{\alpha} from learned embeddings (Figure 1b). On SoundDescs, α^\hat{\alpha} decreases from ∼18 at d=2d = 2 to ∼0.06 at d=1024d = 1024, falling below α=1\alpha= 1 by d=16d = 16. This places high-dimensional ReTreever embeddings in the sparse regime, where baseline distances plateau rather than contract (Figure 1a). Despite this, the modality gap remains very small and continues to reamin very low across resolutions (Figure 3), and RELGap remains lowest for simplex methods. This supports the interpretation that gap closure reflects improved cross-modal alignment rather than a purely geometric change in baseline distance scale.

Figure 1. (a) Expected ℓ2\ell_2 distance vs. dimension. Sphere: plateau at 2\sqrt{2}. Dense simplex (α=1\alpha= 1): shrinks as O(1/d)O(1/\sqrt{d}). Sparse simplex (α=c/d\alpha= c/d): plateau at 2/(c+1)\sqrt{2/(c+1)}. (b) Dirichlet α^\hat{\alpha} estimated from learned ReTreever embeddings (SoundDescs). As dimension grows, α^≪1\hat{\alpha} \ll1, placing embeddings in the sparse regime where distances plateau rather than contract.

Conclusions

We presented a geometric approach to reducing the modality gap by projecting embeddings onto the probability simplex. Although Sd−1S^{d-1} and Δd−1\Delta^{d-1} have the same intrinsic dimension, simplex constraints (nonnegativity and unit mass) impose a shared coordinate system that acts as cross-modal calibration. Across five benchmarks, simplex projections reduce the centroid ℓ2\ell_2 gap by 97–99% while matching or improving retrieval, with RELGap improving as well. Comparisons to Euclidean cross-attention baselines (XAttn and WAVE-7B) indicate that shared cross-attention by itself does not reliably eliminate the gap, and that the simplex projection plays a central role in the reductions we observe. Finally, coarse-to-fine analysis confirms the expected scaling behavior across resolutions and supports that gap reduction reflects alignment rather than a distance-scale artifact. Simplex embeddings also provide a convenient compositional and interpretable representation, and enable information-theoretic similarity measures (e.g., JS, Hellinger, KL) as alternatives to cosine distance.

References

  1. [1]John Aitchison. The statistical analysis of compositional data. Journal of the Royal Statistical Society: Series B (Methodological), 44(2):139–160, 1982.
  2. [2]Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. VATT: Transformers for multimodal self-supervised learning from raw video, audio and text. In Advances in Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2104.11178
  3. [3]David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, 2003.
  4. [4]Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: Universal image-text representation learning. In European Conference on Computer Vision (ECCV), 2020.arxiv.org/abs/1909.11740
  5. [5]Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Ramakrishna Vedantam. Hyperbolic image-text representations. In International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2304.09172
  6. [6]Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.arxiv.org/abs/1910.09387
  7. [7]Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP: Learning audio concepts from natural language supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.arxiv.org/abs/2206.04769
  8. [8]Abrar Fahim, Alex Murphy, and Alona Fyshe. It’s not a modality gap: Characterizing and addressing the contrastive gap. arXiv preprint arXiv:2405.18570, 2024.arxiv.org/abs/2405.18570
  9. [9]Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. CLIP-Adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132: 581–595, 2024.arxiv.org/abs/2110.04544
  10. [10]Eleonora Grassucci, Giordano Cicchetti, Emanuele Frasca, Aurelio Uncini, and Danilo Comminiello. Closing the modality gap aligns group-wise semantics. arXiv preprint arXiv:2601.18525, 2026.arxiv.org/abs/2601.18525
  11. [11]Shubham Gupta, Zichao Li, Tianyi Chen, Cem Subakan, Siva Reddy, Perouz Taslakian, and Valentina Zantedeschi. Hierarchical retrieval at scale: Bridging transparency and efficiency, 2026. URL https://arxiv.org/abs/2502.07971.
  12. [12]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attarian, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (ICML), 2019.arxiv.org/abs/1902.00751
  13. [13]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2106.09685
  14. [14]Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. Proceedings of the International Conference on Machine Learning (ICML), 2021.
  15. [15]Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.arxiv.org/abs/2004.04906
  16. [16]Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pp. 119–132, 2019.DOI
  17. [17]A. Sophia Koepke, Andreea-Maria Oncescu, João F. Henriques, Zeynep Akata, and Samuel Albanie. Audio retrieval with natural language queries: A benchmark study. IEEE Transactions on Multimedia, 24:4469–4480, 2022.DOI
  18. [18]Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  19. [19]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2301.12597
  20. [20]Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2203.02053
  21. [21]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.arxiv.org/abs/1405.0312
  22. [22]Xiang Ma, Xue-mei Li, Lexin Fang, and Caiming Zhang. Bridging the modality gap: Dimension information alignment and sparse spatial constraint for image-text matching. In Proceedings of the 32nd ACM International Conference on Multimedia, 2024.arxiv.org/abs/2410.16853
  23. [23]Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020.DOI
  24. [24]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2103.00020
  25. [25]Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, and Ajanthan Thalaiyasingam. Accept the modality gap: An exploration in the hyperbolic space. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.DOI
  26. [26]Peter J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53–65, 1987.DOI
  27. [27]Simon Schrodi, David T. Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language representation learning. In International Conference on Learning Representations (ICLR), 2025.
  28. [28]Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In Advances in Neural Information Processing Systems (NeurIPS), 2018.arxiv.org/abs/1806.01768
  29. [29]Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  30. [30]Amit Sofer, Yoav Goldman, and Shlomo E. Chazan. Pull it together: Reducing the modality gap in contrastive learning. 2025.
  31. [31]Changli Tang, QinFan Xiao, Ke Mei, Tianyi Wang, Fengyun Rao, and Chao Zhang. WAVE: Learning unified and versatile audio-visual embeddings with multimodal LLM. arXiv preprint arXiv:2509.21990, 2025.
  32. [33]Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023a.
  33. [34]Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023b.
  34. [35]Can Yaras, Siyi Chen, Peng Wang, and Qing Qiu. Explaining and mitigating the modality gap in contrastive multimodal learning. In Conference on Parsimony and Learning (CPAL), 2025.
  35. [36]Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics (TACL), 2:67–78, 2014.
  36. [37]Yuhui Zhang, Jeff Z. HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Diagnosing and rectifying vision models using language. In International Conference on Learning Representations (ICLR), 2023.

Paper details

vlnc:W_avmjmgg8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents