betaFeedback

WorksComputer Science

Fuse To Align: Multimodal Representation Learning for Retrieval via Hypergraph Regularization

Abstract

Multimodal retrieval must integrate complementary evidence from views that may be missing, noisy, or unequally informative. When a sample provides limited reliable evidence, learning from its views alone can hinder the acquisition of useful retrieval representations. Semantically related training samples may provide complementary observations, yet existing volume-based objectives jointly align modalities within each sample without explicitly enabling information exchange across samples. We therefore propose a training-time hypergraph fusion framework that connects observed modalities both within samples and across semantically related samples, allowing representation learning to draw on evidence beyond an individual sample while leaving the retrieval pipeline unchanged at inference. The framework includes HyVol as a special case and introduces two configurations: wHyVol learns view contribution weights and symmetric semantic affinities, whereas wRHG additionally models typed interactions and participant roles through relational propagation. Missing modalities are omitted rather than imputed, and the hypergraph is discarded after training, incurring no additional inference cost. Experiments on eleven zero-shot video-text and audio-text retrieval benchmarks demonstrate broad improvements with both GRAM and HyperGRAM. Across 17 video benchmark-modality settings evaluated in both retrieval directions, wRHG outperforms GRAM in all 34 comparisons and HyperGRAM in 33 of 34. Mean text-to-video R@1 increases from 63.4 to 68.2 with GRAM and from 64.8 to 68.4 with HyperGRAM, while the largest video-to-text gain reaches 13.3 percentage points on ActivityNet. Under missing-modality evaluation, both weighted configurations retain positive text-to-video gains over GRAM across seven benchmarks at every evaluated missingness rate, even when 90% of clips lack one modality.

Introduction

Humans interpret the world by integrating information across multiple sensory modalities. For example, visual input can influence auditory perception, while language provides contextual cues that help unify sensory experiences. Psychologists refer to this process as multisensory integration, through which individuals reconstruct events from both overlapping and modality-specific sensory cues. Multimodal representation learning presents an analogous challenge because images, sounds, speech, and text capture complementary aspects of the same scenario. Effective representation learning must therefore identify shared features across modalities while preserving the distinctive information provided by each modality.

Text-to-video retrieval requires a model to identify, among thousands of clips, the clip described by a given caption. Each clip provides multiple views of the same event, including frames, a soundtrack, and subtitles, but real-world collections rarely contain all views at consistently high quality. Consider two clips described as “a man explains how to tune a guitar”. In the first clip, the frames depict the guitar, and the subtitles transcribe the explanation. In the second, the subtitles are absent, and the soundtrack contains only background music, leaving the frames as the sole reliable source of evidence. Such missing, noisy, and unequally informative views are common, requiring a retrieval model to integrate them into a single relevance score.

Current multimodal retrieval models evaluate each clip independently. Contrastive foundation models such as CLIP [1] measure the cosine similarity between two embeddings, while their multi-view extensions either align each view with an anchor modality or fuse multiple views into a single representation [2, 3, 4, 5]. Geometric objectives extend this approach by jointly scoring all views of a clip using the volume spanned with the caption in GRAM [6], the area of a triangle in TRI-ANGLE [7], the dominant singular value in PMRL [8], or a combination of Euclidean and hyperbolic volumes in HYPERGRAM [9]. Nevertheless, in all these methods, a clip’s score depends exclusively on the caption and the clip’s own views. Consequently, in the example above, the second clip must be matched using only one reliable view, while the first clip, despite depicting the same event, is treated solely as a negative during training [10, 11].

Our central premise is that fusion should exploit complementary evidence both within a sample and across semantically related training samples. When a clip contains missing or uninformative views, restricting fusion to its own observations limits the evidence available for representation learning. Related clips may contain useful observations of similar events, providing an additional source of supervision through information exchange. This motivates a fusion mechanism that jointly models cross-modal dependencies within clips and semantic associations across clips.

We propose a training-time hypergraph feature fusion framework that captures both forms of interaction. Each observed candidate modality forms a vertex, while hyperedges connect modalities belonging to the same candidate and observations from semantically related candidates. Feature propagation along these hyperedges integrates contextual evidence into the modality-specific representations, which remain separate inputs to the retrieval objective. Learned contribution weights regulate this fusion, while relational propagation distinguishes interaction types and participant roles. Missing views remain absent rather than being reconstructed. The hypergraph operates only during training, allowing the encoders to learn from fused evidence while preserving independent candidate encoding and the original retrieval procedure at inference. Figure 1 contrasts this design with existing methods, which score each clip alone, and shows that the hypergraph is removed at test time.

Figure 1. Overview of the proposed framework. (a) Existing methods score each clip alone: a clip’s score depends only on the caption and the clip’s own views, and every other clip in the batch, even a related one, is pushed apart as a negative. (b) Our training-time hypergraph: observed views form vertices, document hyperedges group the views of each clip, and caption-guided semantic hyperedges connect related clips through same-modality and cross-modality links; missing views have no vertex and are not imputed, and each vertex receives a learned reliability weight. (c) At test time the hypergraph is removed: each clip is encoded and scored independently with the original backbone score, so the inference cost equals that of the backbone.

Our framework differs from conventional fusion strategies in three respects. First, it integrates evidence both within individual clips and across semantically related training clips. Second, unlike approaches that reconstruct missing views [10], it operates only on observed modalities, enabling information exchange without imputation. Third, fusion is confined to training: the learned encoders retain their benefits, while inference requires neither a hypergraph nor interactions among evaluation candidates. The framework encompasses HyVol [12] as a special case, where document and semantic neighborhood hyperedges support additive feature fusion. Our weighted and relational configurations investigate how to regulate individual-view contributions and related-sample contributions, and how different interactions should participate in the fusion.

HyVol does not explicitly model differences in view reliability or distinguish modality-specific interaction types. A silent soundtrack receives the same weight as the frames of the corresponding clip; related clips are weighted by an attention score that ranks the neighbors of every clip in the same order [13]; and a link between the frames of two clips is not distinguished from a link between the frames of one clip and the subtitles of another. Moreover, only the refined embeddings are trained, although retrieval subsequently uses the unrefined embeddings. Information exchange across clips therefore raises two questions that remain unresolved in HyVol: how much should each view and each related clip contribute, and what type of relationship connects them?

Weighted HyVol (wHyVol) addresses the first question through two learned weights. The first is a view-specific reliability weight predicted solely from the corresponding view embedding. In the guitar example, the background music in the second clip may receive a low weight, while its frames receive a high weight, causing the clip representation to be determined primarily by reliable evidence. The reliability weight therefore provides an adaptive fusion mechanism for imbalanced views: each view’s contribution depends on its estimated quality rather than its modality. The second weight represents the affinity between two related clips; it is computed jointly from their captions and is symmetric. Consequently, a clip receives substantial information from a close semantic match but little information from a weak or incidental match. Unlike the attention mechanism in HyVol, this affinity can induce different neighbour rankings for different clips.

The weighted relational hypergraph (wRHG) addresses the second question. It retains both weights introduced by wHyVol but replaces untyped links with typed relations. A same-modality link connects corresponding views from two related clips, such as their frames; a cross-modality link connects different views, such as the frames of one clip and the subtitles of another; and a whole-clip link connects all observed views from both clips. Each relation type has distinct learned parameters, and the message received by a view through a link depends jointly on all other views associated with that link. Thus, a clip with missing subtitles can learn from a related clip’s subtitles through a cross-modality link, while the model learns the relative importance of each relation type.

In wHyVol and wRHG, a missing view has no corresponding vertex in the hypergraph; consequently, neither model imputes nor replaces missing views. These configurations also train the unrefined embeddings used at test time. As with HyVol, their hypergraph modules operate exclusively during training. We integrate HyVol, wHyVol, and wRHG with two volume-based backbones, GRAM and HYPERGRAM, which share the pretrained vision–audio–language foundation encoders (VAST [4]). We evaluate all configurations zero-shot on eleven text–video and text–audio retrieval benchmarks. Averaged across the 17 text-to-video settings from the eight video benchmarks, the R@1 of GRAM increases from 63.4 to 68.2 with wRHG (and to 67.3 with HyVol), while that of HYPERGRAM increases from 64.8 to 68.4. The gains are consistent across datasets: wRHG improves the R@1 of GRAM in all 17 text-to-video and all 17 video-to-text settings, and improves that of HYPERGRAM in 16 and 17 settings, respectively. The weighted variants also retain gains under missing-modality evaluation, with consistent improvements over GRAM and benchmark-dependent results with HYPERGRAM. Our main contributions are as follows.

  • We formulate multi-view feature fusion within and across semantically related samples through a common training-time hypergraph framework, with HyVol recovered as a special case.

  • We introduce two new instances of this framework: a weighted hypergraph (wHyVol), which learns the contribution of each view and related clip, and a weighted relational hypergraph (wRHG), which models distinct interaction types through relation-specific parameters and participant roles. Both methods handle missing views without imputation and incur no additional inference cost.

  • We evaluate the proposed methods using two backbones and eleven zero-shot benchmarks—eight for text–video retrieval and three for text–audio retrieval—and conduct a missing-modality study across seven video benchmarks.

Building on HyVol [12], this article introduces a general framework together with wHyVol and wRHG, expands the evaluation from six to eleven benchmarks, and extends the missing-modality study from four to seven video benchmarks. The remainder of this article is organized as follows. § 2 reviews the related literature. § 3 presents the general hypergraph framework, recovers HyVol as a special case, and introduces wHyVol and wRHG. § 4 describes the datasets and evaluation protocol and reports the zero-shot and missing-modality results. § 6 discusses these results, § 7 outlines the limitation of the approach, and § 8 concludes the article.

Related Work

Our research builds upon multimodal representation learning, higher-order geometric alignment, and hypergraph-based information fusion. We examine these foundational concepts alongside methods that address missing and imbalanced modalities, inter-sample structural relationships in retrieval tasks, and relation-aware message passing.

Contrastive Vision–Language and Multimodal Foundations

CLIP [1] established contrastive pretraining on image–text pairs, whereby representations of matched pairs are brought closer together while all other pairs in the batch serve as negatives. Subsequent work developed larger and openly available backbones [14]. More recent approaches have developed large-scale video–language models pretrained on extensive video corpora [15, 16, 17, 18, 19, 20]. Contrastive learning has also been applied to audio–text retrieval [21, 22] and to text–video retrieval through multigranular alignment and text-conditioned objectives.

When more than two modalities are available, existing methods typically either fuse streams into a single representation or align them within a shared embedding space. VALOR [5] fuses video and audio, while VAST [4] additionally incorporates subtitles. ImageBind [2] and LanguageBind [3] instead align modalities using images and language as anchors, respectively. Alignment may also precede explicit cross-modal fusion [23]. These approaches do not directly score joint agreement among a candidate’s individual streams. Our models retain the modality-specific VAST embeddings and evaluate their joint agreement with the query.

Higher-Order Geometric Alignment

Recent methods assess multimodal agreement through joint geometric objectives. GRAM [6] uses the Gramian volume spanned by the query and candidate modality embeddings, while TRIANGLE [7] measures the triangle area formed by text, video, and audio embeddings. PMRL [8] promotes rank-one alignment by maximising the dominant singular value of the embedding matrix, and HYPERGRAM [9] combines Euclidean and hyperbolic volumes. These methods score each candidate using only its own embeddings. HYVOL [12] introduces information exchange across candidates through a training-time hypergraph while retaining the GRAM and HYPERGRAM scoring functions. Our framework includes HYVOL as a special case and investigates learned contribution weights, semantic affinities, and typed relational propagation.

Learning with Missing and Imbalanced Modalities

Real-world collections often contain missing modalities and streams that differ in informativeness. Existing methods address missing views through reconstruction [10] or fusion mechanisms designed to accommodate absent inputs [11]. Our framework instead excludes missing modalities from the hypergraph and the Gram matrix, so retrieval depends only on observed streams. Learned vertex weights predicted from individual modality embeddings regulate their contributions during training-time fusion.

Hypergraphs and Inter-Sample Structure in Retrieval

Hypergraphs represent higher-order relations by connecting multiple vertices within a single hyperedge. Hypergraph neural networks exchange features between vertices and hyperedges [24], while AllSet [25] learns set-based aggregation functions. Applications include cross-modal retrieval through bipartite hypergraphs [26]. Attention mechanisms learn interaction weights [27], although the original additive attention score has a static neighbor-ranking property [13], motivating the pair-dependent affinity kernel in wHyVol. In retrieval, inter-item structure is also exploited through test-time methods such as k-reciprocal re-ranking [28] and querybank normalisation [29]. By contrast, HyVol [12] and our configurations exchange information only during training, allowing independent candidate encoding without additional inference cost.

Relational and Knowledge Hypergraphs

Relational graph convolutional networks [30] assign distinct transformations to different edge types. Knowledge hypergraphs extend this principle to facts involving more than two arguments, where the position of each entity carries semantic information. HypE [31] embeds entities using position-specific filters; StarE [32] represents hyper-relational facts with qualifiers; and G-MPNN [33] performs message passing over multi-relational ordered hypergraphs. HC-MPNN [34] combines relation embeddings with positional encodings and constructs the message to each member of a fact from all remaining members. Its expressive power for link prediction has been characterised for relations of arbitrary arity.

Graphs with multiple edge types have also been applied to multimodal fusion. For conversational emotion recognition, MMGCN [35] connects utterances through modality-specific and cross-modal edges. For unaligned multimodal language sequences, MTAG [36] constructs a graph whose edges depend on node modalities and temporal order. In each setting, the data determine the edges or facts, such as a knowledge base, conversational order, or temporal order. In WRHG, by contrast, facts are constructed within each training batch according to caption similarity, and each fact’s relation type identifies the modalities and candidates that it connects.

Relational Hypergraph Learning

Multimodal representation learning must capture dependencies at two complementary levels: cross-modal associations within each candidate and semantic correspondences across candidates. The former integrates complementary evidence from the modalities observed for a candidate, while the latter lets its representation incorporate relevant information from semantically related candidates in the same training batch. We model both forms of dependency using a weighted hypergraph in which each vertex represents an observed candidate–modality pair. Hyperedge construction determines which vertices interact, learned weights control their contributions, and the propagation operator specifies how the resulting information is aggregated.

The framework supports two propagation mechanisms: weighted additive aggregation and relation-conditioned multiplicative interaction. Both mechanisms refine candidate embed­dings under the original retrieval objective without imposing a specific similarity function or representation geometry. HYVOL [12] is recovered as a special case by choosing a specific hyperedge construction, weighting scheme, and additive propagation operator. The hypergraph is used only during training; at inference, the model retains the trained encoders and follows the original retrieval procedure without constructing or propagating the hypergraph. Figure 2 summarises the framework.

Figure 2. Overview of training-time hypergraph refinement. Each observed modality becomes a weighted vertex, and stop-gradient captions connect semantically related candidates through mutual top-kk neighbors. Information is then exchanged either by weighted additive propagation (wHyVol) or by typed relational messages (wRHG), and the refined embeddings are scored by the unchanged GRAM/HYPERGRAM objective. The hypergraph is used only during training and is removed at inference.

Problem Formulation

Let a mini-batch comprise NN matched query–item pairs {(ci,yi)}i=1N\{(c_i,y_i)\}_{i=1}^{N}, where cic_i is a caption query and yiy_i is its matched retrieval item. We use candidate to denote any retrieval item evaluated as a potential match for a query. Each candidate yiy_i comprises observations from a nonempty subset Oi⊆MO_i \subseteq\mathcal{M} of the supported modalities. The modality set M\mathcal{M} may include visual, audio, subtitle, or other available streams, and M=∣M∣M = |\mathcal{M}| denotes the number of supported modalities. Subtitles are candidate-side observations and are therefore distinct from the caption used as the retrieval query.

A query encoder fqf_q and modality-specific encoders {fm}m∈M\{f_m\}_{m\in\mathcal{M}} produce dd-dimensional representations in a common embedding space. Each encoder ends with a learned linear projection to this shared space. Although these encoders are parameterised separately, the retrieval objective aligns their outputs for cross-modal comparison. The resulting normalised embeddings are qi=norm⁡(fq(ci))q_i = \operatorname{norm}(f_q(c_i)) and xim=norm⁡(fm(yim))x_i^m = \operatorname{norm}(f_m(y_i^m)). Here, norm⁡(x)=x/max⁡(∥x∥2,ϵn)\operatorname{norm}(x) = x/\max(\lVert x\rVert_2,\epsilon_n), where ϵn>0\epsilon_n > 0 is a numerical-stability constant. When ∥x∥2>ϵn\lVert x\rVert_2 > \epsilon_n, this operation produces a unit-norm embedding, and inner products between the resulting embeddings correspond to cosine similarities.

Using these embeddings, we represent each observed candidate–modality pair as a vertex: V={(i,m):1≤i≤N,m∈Oi}\mathcal{V} = \{(i,m): 1 \le i \le N, m \in O_i\}. Accordingly, the number of vertices is nv=∑i=1N∣Oi∣n_v = \sum_{i=1}^{N}|O_i|. This construction preserves modality identity while allowing different observations of the same candidate to participate in distinct interactions. Because vertices are instantiated only for observed modalities, missing modalities require neither imputation nor masking; they are absent from both propagation and pooling.

Query embeddings are not propagated as vertex features. Instead, stop-gradient query embeddings are used only to define semantic associations among candidates. This design allows caption semantics to guide the construction of the training-time hypergraph without injecting query information directly into candidate representations. Consequently, each candidate can be encoded independently of the test query at inference.

Semantic Connectivity and Learned Weights

Cross-candidate interactions are restricted to semantically related training instances. Let ti=sg⁡(qi)t_i = \operatorname{sg}(q_i) denote the stop-gradient caption embedding, where sg⁡\operatorname{sg} is the stop-gradient operator. For candidate ii, let K(i)\mathcal{K}(i) contain the indices of keffk_{\mathrm{eff}} other candidates with the most similar caption embeddings under the inner product tiTtjt_i^{\mathsf{T}}t_j. We retain an association only when this neighbourhood relation is reciprocal:

Sij=I[j∈K(i)]I[i∈K(j)]FijDij,Sii=0.(1)S_{ij} = \mathbb{I}[j \in\mathcal{K}(i)]\mathbb{I}[i \in\mathcal{K}(j)]F_{ij}D_{ij}, \qquad S_{ii}=0. \tag*{(1)}

Here, I[⋅]\mathbb{I}[\cdot] is the indicator function, Fij=FjiF_{ij}=F_{ji} is an optional similarity filter, and Dij=DjiD_{ij}=D_{ji} is a symmetric training-time dropout mask. In the absence of filtering or dropout, the corresponding factors are set to one. The reciprocity requirement eliminates one-sided associations supported by only one neighbourhood ranking.

In the implementation, keff=min⁡(k,max⁡(2,⌊N/4⌋))k_{\mathrm{eff}}=\min(k,\max(2,\lfloor N/4\rfloor)), where kk is the requested neighbourhood size, and semantic connections are omitted when N≤keff+1N \le k_{\mathrm{eff}}+1. The optional filter retains similarities above μ+ησ\mu+\eta\sigma, where μ\mu and σ\sigma are the mean and standard deviation of the off-diagonal caption similarities and η>0\eta>0 controls selectivity. These choices determine the candidate associations available to all relation families.

Connectivity and contribution strength are treated separately. A learned projection determines the weight of each retained candidate pair, while a modality-dependent predictor estimates the contribution of an individual vertex.

ai=norm⁡(Wati),ωij=Sijexp⁡(aiTaj−1τa),\mathbf{a}_i = \operatorname{norm}(W_a t_i), \qquad\omega_{ij}=S_{ij}\exp\left(\frac{\mathbf{a}_i^{\mathsf{T}}\mathbf{a}_j-1}{\tau_a}\right),
τa=exp⁡(ξ),βim=2sigmoid⁡(wvTxim+bv).(2)\tau_a=\exp(\xi), \qquad\beta_i^m = 2\operatorname{sigmoid}\left(\mathbf{w}_v^{\mathsf{T}}x_i^m+b_v\right). \tag*{(2)}

The parameters are Wa∈Rd×dW_a \in\mathbb{R}^{d\times d}, ξ∈R\xi\in\mathbb{R}, wv∈Rd\mathbf{w}_v \in\mathbb{R}^{d}, and bv∈Rb_v \in\mathbb{R}. The connection weights satisfy ωij=ωji\omega_{ij}=\omega_{ji} and lie in [0,1][0,1] in exact arithmetic, while βim∈(0,2)\beta_i^m \in(0,2). Unlike a neighbourhood softmax, the exponential kernel does not require the outgoing weights of a candidate to sum to one. It therefore preserves an absolute measure of affinity within the learned caption space.

We initialise WaW_a to the identity, τa\tau_a to 0.50.5, and wv\mathbf{w}_v and bvb_v to zero. Consequently, all observed vertices initially have unit weight. The weighting parameters receive gradients from the training objective, whereas the detached caption inputs do not transmit these gradients to the query encoder. Vertex weights remain differentiable with respect to their candidate features.

A Common Hypergraph Propagation Formulation

A hyperedge is represented by an indexed interaction e=(Ve,re,ωe,πe)e=(\mathcal{V}_e,r_e,\omega_e,\pi_e). Its participating vertices form Ve⊆V\mathcal{V}_e \subseteq\mathcal{V}, rer_e identifies its relation type, ωe\omega_e is a scalar interaction weight, and πe(v)\pi_e(v) assigns a participant role to vertex vv. Different indexed interactions may share the same vertex set while carrying different relation labels or role assignments.

Let E(v)\mathcal{E}(v) denote the hyperedges incident to vv, and let hv(ℓ)∈Rd\mathbf{h}_v^{(\ell)} \in\mathbb{R}^{d} be its representation at layer ℓ\ell. We express refinement through a message function Mℓ\mathcal{M}_\ell, an aggregation operator Aℓ\mathcal{A}_\ell, and an update function Uℓ\mathcal{U}_\ell:

me→v(ℓ)=Mℓ(v,e,{hu(ℓ)}u∈Ve),\mathbf{m}_{e\to v}^{(\ell)}=\mathcal{M}_\ell\left(v,e,\{\mathbf{h}_u^{(\ell)}\}_{u\in\mathcal{V}_e}\right),
mv(ℓ)=Aℓ({me→v(ℓ)}e∈E(v)),\mathbf{m}_v^{(\ell)}=\mathcal{A}_\ell\left(\{\mathbf{m}_{e\to v}^{(\ell)}\}_{e\in\mathcal{E}(v)}\right),
hv(ℓ+1)=hv(ℓ)+tanh⁡(γℓ)Uℓ(hv(ℓ),mv(ℓ)).(3)\mathbf{h}_v^{(\ell+1)}=\mathbf{h}_v^{(\ell)}+\tanh(\gamma_\ell)\mathcal{U}_\ell\left(\mathbf{h}_v^{(\ell)},\mathbf{m}_v^{(\ell)}\right). \tag*{(3)}

The learned scalar γℓ\gamma_\ell modulates the sign and strength of the residual correction. Hyperedge weights, memberships, relation labels, and participant roles enter through the message and aggregation functions.

This formulation separates the interaction structure from the mechanism that combines its features. Additive propagation constructs weighted hyperedge summaries and redistributes them to their members. Relational propagation instead constructs recipient-specific messages whose components depend jointly on the other participants. Both are instances of Eq. (3).

Weighted Additive Hypergraph

The additive configuration uses document and semantic neighbourhood hyperedges. A document hyperedge connects all observed modalities of one candidate. A semantic hyperedge connects the observed modalities of an anchor candidate and those of its retained semantic neighbours. With rows indexed by observed vertices, the corresponding incidence matrices are

H(j,m),idoc=1[j=i]βjm,H(j,m),isem=1[j=i]+ωji,H=[Hdoc∣Hsem].(4)H^{\mathrm{doc}}_{(j,m),i}=\mathbb{1}[j=i]\beta_j^m, \qquad H^{\mathrm{sem}}_{(j,m),i}=\mathbb{1}[j=i]+\omega_{ji}, \qquad H=\left[H^{\mathrm{doc}}\mid H^{\mathrm{sem}}\right]. \tag*{(4)}

Document membership reflects learned modality importance. Semantic membership assigns unit weight to each observed anchor modality and a shared connection weight to each observed modality of a neighbour. Consequently, a neighbour with more observed modalities contributes more total membership mass; subsequent degree normalisation operates on this total mass.

The semantic block is omitted when the batch contains no retained semantic connections. Otherwise, one semantic hyperedge is included for every candidate, including anchor-only hyperedges for candidates without neighbours. This convention is relevant because repeated or overlapping hyperedges affect the normalisation of incident messages.

Let Z(0)∈Rnv×dZ^{(0)} \in\mathbb{R}^{n_v \times d} stack the encoder embeddings as rows. Define De=diag⁡(H⊤1)D_e=\operatorname{diag}(H^\top\mathbf{1}) and Dv=diag⁡(H1)D_v=\operatorname{diag}(H\mathbf{1}), where 1\mathbf{1} is an all-ones vector of the appropriate dimension. The additive layer is

C(ℓ)=Dv−1H GELU⁡(De−1H⊤Z(ℓ)WV(ℓ))WE(ℓ),(5)C^{(\ell)} = D_v^{-1} H\,\operatorname{GELU}\left(D_e^{-1}H^\top Z^{(\ell)}W_V^{(\ell)}\right)W_E^{(\ell)}, \tag*{(5)}
Z(ℓ+1)=Z(ℓ)+tanh⁡(γℓ)ϕℓ(C(ℓ)).Z^{(\ell+1)} = Z^{(\ell)} + \tanh(\gamma_\ell)\phi_\ell(C^{(\ell)}).

Here, WV(ℓ),WE(ℓ)∈Rd×dW_V^{(\ell)}, W_E^{(\ell)} \in\mathbb{R}^{d \times d} are learned projections, and ϕℓ\phi_\ell is GELU in intermediate layers and the identity in the final layer. Degree entries are floored at 10−610^{-6} before inversion.

The first aggregation computes a weighted average of the vertices in each hyperedge. The second combines the resulting hyperedge representations at each vertex. The residual path preserves the incoming feature while allowing within-candidate and cross-candidate context to modify it. Hyperedge types affect connectivity, but this operator shares its feature transformations across types.

Relational Hypergraph

The relational configuration makes the interaction type explicit through typed hyperedge families. Within a candidate, a single document hyperedge connects all of its observed modalities and therefore already covers every pair of them, so no separate within-candidate pairwise hyperedges are formed.

Cross-candidate hyperedges are constructed for pairs with ωij>0\omega_{ij} > 0. Same-modality hyperedges connect matching modality types across the two candidates. Cross-modality hyperedges connect different modality types; for each unordered modality pair, both assignments to the two candidates are included when the corresponding observations are available. A joint-candidate hyperedge contains all observed modalities from both candidates. Within-candidate hyperedges have weight ωe=1\omega_e = 1, and cross-candidate hyperedges have weight ωe=ωij\omega_e = \omega_{ij}. Hyperedges with fewer than two observed vertices are discarded.

The role map πe\pi_e identifies modality type. For joint-candidate hyperedges, it additionally identifies which of the two candidates supplies the observation. These two candidate positions use disjoint role indices. Both candidate orders are instantiated, ensuring that the construction does not privilege either endpoint of the undirected semantic association.

For relational propagation, the initial state is hv(0)=dxv\mathbf{h}_v^{(0)} = \sqrt{d}\mathbf{x}_v. Let pπe(u)(ℓ)∈Rd\mathbf{p}_{\pi_e(u)}^{(\ell)} \in\mathbb{R}^{d} and rre(ℓ)∈Rd\mathbf{r}_{r_e}^{(\ell)} \in\mathbb{R}^{d} denote learned role and relation embeddings. The contribution of participant uu and the aggregated message to recipient vv are

ae,u(ℓ)=βu(αℓhu(ℓ)+(1−αℓ)pπe(u)(ℓ)),\mathbf{a}_{e,u}^{(\ell)} = \beta_u\left(\alpha_\ell\mathbf{h}_u^{(\ell)} + (1-\alpha_\ell)\mathbf{p}_{\pi_e(u)}^{(\ell)}\right),
mv(ℓ)=∑e∈E(v)ωerre(ℓ)⊙⨀u∈Ve∖{v}ae,u(ℓ).(6)\mathbf{m}_v^{(\ell)} = \sum_{e\in\mathcal{E}(v)} \omega_e\mathbf{r}_{r_e}^{(\ell)} \odot\bigodot_{u\in V_e\setminus\{v\}}\mathbf{a}_{e,u}^{(\ell)}. \tag*{(6)}

The symbol ⊙\odot denotes elementwise multiplication. The learned scalar αℓ\alpha_\ell is initialised to 0.5 and is unconstrained, so the feature–role combination is affine rather than necessarily convex. Vertex weights βu\beta_u are computed from the encoder outputs and reused across layers.

Excluding the recipient from the product makes each message depend on the other participants in the interaction. For two-vertex hyperedges, the product contains one factor. For larger hyperedges, it jointly combines several participants, allowing the message to depend on their coordinate-wise interactions. Relation embeddings modulate these products by interaction type, while role embeddings distinguish contributions across modalities and candidate positions.

Messages are summed over incident hyperedges without degree normalisation and are incorporated through

hv(ℓ+1)=hv(ℓ)+tanh⁡(γℓ)σℓ(LN⁡ℓ(zv(ℓ))),\mathbf{h}_v^{(\ell+1)} = \mathbf{h}_v^{(\ell)} + \tanh(\gamma_\ell)\sigma_\ell\left(\operatorname{LN}_\ell(\mathbf{z}_v^{(\ell)})\right),
zv(ℓ)=Dropout⁡(Wℓ[mv(ℓ)∥hv(ℓ)]+bℓ),(7)\mathbf{z}_v^{(\ell)} = \operatorname{Dropout}\left(W_\ell[\mathbf{m}_v^{(\ell)} \Vert\mathbf{h}_v^{(\ell)}] + \mathbf{b}_\ell\right), \tag*{(7)}

where ∥\Vert denotes concatenation, Wℓ∈Rd×2dW_\ell\in\mathbb{R}^{d \times2d}, bℓ∈Rd\mathbf{b}_\ell\in\mathbb{R}^{d}, and LN⁡ℓ\operatorname{LN}_\ell is layer normalisation. The activation σℓ\sigma_\ell is ReLU in intermediate layers and the identity in the final layer.

Concatenating the current feature with its aggregated message allows the update to depend on both local information and relational context. An observed vertex without incident hyperedges receives a zero message but may still undergo this local transformation. When the entire relational hypergraph is empty, the implementation bypasses refinement. We use at most two propagation layers.

Retrieval Supervision and Inference

After LL layers, let uim\mathbf{u}_i^m denote the final unnormalised modality representation. For additive propagation, it is the corresponding row of Z(L)Z^{(L)}; for relational propagation, it is h(i,m)(L)/d\mathbf{h}_{(i,m)}^{(L)}/\sqrt{d}. The module produces normalised modality embeddings and a pooled candidate representation:

xˉim=norm⁡(uim),\bar{\mathbf{x}}_i^m = \operatorname{norm}(\mathbf{u}_i^m),
h~i=Wpool∑m∈Oiβimuimmax⁡(∑m∈Oiβim,ϵp),\tilde{\mathbf{h}}_i = W_{\mathrm{pool}}\frac{\sum_{m\in\mathcal{O}_i}\beta_i^m\mathbf{u}_i^m}{\max\left(\sum_{m\in\mathcal{O}_i}\beta_i^m,\epsilon_p\right)},
hidoc=norm⁡(h~i).(8)\mathbf{h}_i^{\mathrm{doc}} = \operatorname{norm}(\tilde{\mathbf{h}}_i). \tag*{(8)}

Here, Wpool∈Rd×dW_{\mathrm{pool}} \in\mathbb{R}^{d \times d} is learned and ϵp=10−6\epsilon_p = 10^{-6}. Pooling includes only observed modalities and uses the same learned vertex weights as the corresponding weighted refinement configuration.

Let sbs_b denote the unscaled compatibility function of retrieval backbone bb, where larger values indicate stronger compatibility. Refinement changes the embeddings supplied to this function while preserving its form:

Φij(b)=sb(qi,{xˉjm}m∈Oj)ϑb,ϑb>0.(9)\Phi_{ij}^{(b)} = \frac{s_b\left(\mathbf{q}_i,\{\bar{\mathbf{x}}_j^m\}_{m\in\mathcal{O}_j}\right)}{\vartheta_b}, \qquad\vartheta_b > 0. \tag*{(9)}

The temperature convention of the backbone is retained; if no temperature scaling is used, ϑb=1\vartheta_b = 1.

For a Euclidean Gramian-volume backbone, sb=−det⁡(Aij⊤Aij)s_b = -\sqrt{\det(A_{ij}^{\top}A_{ij})}, where AijA_{ij} contains the query and observed candidate embeddings as columns. Other backbones retain their corresponding compatibility functions. Thus, volume scoring is one instantiation of the framework rather than a requirement of hypergraph refinement. With one designated positive pair per batch index, the training objective is

Lret(b)=−12N∑i=1N[log⁡exp⁡(Φii(b))∑j=1Nexp⁡(Φij(b))+log⁡exp⁡(Φii(b))∑j=1Nexp⁡(Φji(b))],L(b)=Lret(b)+∑a∈AbλaLa(b).(10)\begin{aligned} \mathcal{L}_{\mathrm{ret}}^{(b)} &= -\frac{1}{2N}\sum_{i=1}^{N}\left[\log\frac{\exp(\Phi_{ii}^{(b)})}{\sum_{j=1}^{N}\exp(\Phi_{ij}^{(b)})}+\log\frac{\exp(\Phi_{ii}^{(b)})}{\sum_{j=1}^{N}\exp(\Phi_{ji}^{(b)})}\right],\\ \mathcal{L}^{(b)} &= \mathcal{L}_{\mathrm{ret}}^{(b)}+\sum_{a\in\mathcal{A}_{b}}\lambda_{a}\mathcal{L}_{a}^{(b)}. \tag*{(10)} \end{aligned}

The set Ab\mathcal{A}_{b} indexes the auxiliary objectives retained from the selected training recipe, and λa≥0\lambda_{a} \ge0 controls their contributions. Semantic connectivity does not alter the positive assignments: related candidates exchange information during refinement but remain contrastive negatives unless they are designated matches.

For the HYVOL recipe, the auxiliary terms are pooled caption–candidate alignment, variance regularisation, and the backbone’s matching loss, with their original definitions [12]. The pooled representation in Eq. (8) supplies the candidate feature for auxiliary alignment. These auxiliary choices are separate from the hypergraph propagation operator and are held fixed when comparing refinement configurations under a common training recipe.

At inference, we replace refined embeddings with the encoder outputs xjx_j in Eq. (9). The hypergraph, propagation layers, and auxiliary pooling head are not evaluated. Consequently, relational context influences retrieval through the encoder parameters learned during training, without test-time neighbourhood construction or interaction among test candidates.

Special Cases and Instantiations

The framework separates hyperedge structure, contribution weights, and propagation. The following choices recover HYVOL and define the weighted and relational configurations.

HYVOL [12] is obtained by selecting document hyperedges and caption-guided semantic neighbourhood hyperedges, assigning unit document memberships, and using its original learned semantic attention. Specifically, detached caption features are projected and scored by an additive attention function; a softmax over retained mutual neighbours produces attention coefficients, which are subsequently symmetrised. These coefficients determine semantic memberships, together with unit anchor memberships.

We choose the message and aggregation functions in Eq. (3) as HYVOL’s vertex-to-hyperedge and hyperedge-to-vertex additive operators, with its original degree conventions and gated residual update. Relation labels and participant roles do not explicitly parameterize these messages. Selecting the original neighborhood dropout, auxiliary objectives, and volume-based scorer completes the HYVOL special case.

Weighted hypergraph (WHYVOL): The weighted configuration uses the neighbourhood hyperedges in Eq. (4) and additive propagation in Eq. (5). Learned vertex weights regulate within-candidate contributions, while the symmetric exponential kernel in Eq. (2) regulates cross-candidate connections.

Unlike HYVOL’s original attention rule, these connection weights are not obtained through a neighborhood softmax.

Relational hypergraph (wRHG): The relational configuration uses four typed hyperedge families. Within a single candidate, the document hyperedge connects every observed modalities of that candidate; since it already contains every pair of these modalities (e.g., video with audio), separate modality-pair hyperedges are not required. Between two related candidates ii and jj, i.e., candidates whose captions are mutual top-kk neighbours, three families are used: same-modality hyperedges connect matching modalities (e.g., video of ii with video of jj), cross-modality hyperedges connect different modalities (e.g., video of ii with audio of jj), and the joint-candidate hyperedge contains all observed modalities of both candidates. Messages over these hyperedges are formed multiplicatively by Eq. (6), and vertex features are updated by Eq. (7). Each family has its own relation embedding, and each participating modality has its own role embedding, so the model distinguishes, for example, a video–video same-modality interaction from a video–audio cross-modality interaction, which additive propagation would combine through the same transformation.

These configurations are instances of the common propagation formulation rather than successive restrictions of a single multiplicative operator. HYVOL is recovered by selecting its complete structural and computational specification. Restricting relation families alone changes the available interactions but does not convert multiplicative propagation into HYVOL’s additive aggregation.

Experiments

We evaluate whether training-time hypergraph fusion improves multimodal retrieval across different backbones, benchmarks, and modality configurations. Our experiments assess zero-shot retrieval performance and robustness to missing modalities, comparing HYVOL, WHYVOL, and wRHG with their corresponding backbones and existing retrieval methods. We first describe the datasets and experimental protocol, then present and analyze the results.

Datasets

We describe the training corpus and evaluation benchmarks used to assess retrieval performance across different modality configurations.

Training data

All models trained in this work are initialized from the VAST checkpoint [4] and trained on the 150K-clip subset of VAST-27M used by GRAM [6] and HYPERGRAM [9]. Each clip has five visual captions and audio captions, a subtitle containing the speech transcript, and one omni-modal caption that integrates visual, acoustic, and speech information. We use the omni-modal caption as the training text to provide supervision that captures complementary information across the candidate modalities. The annotation file for the subset lists 150, 154 clips, of which 136, 694 were successfully recovered with both video and audio because online video availability changed. These recovered clips constitute the effective training corpus for all models trained in this work.

For GRAM [6], TRIANGLE [7], and PMRL [8], we evaluated the checkpoints released by their authors. The exact set of clips available during the original training of these checkpoints cannot be reconstructed. We therefore distinguish comparisons against released checkpoints from comparisons between models trained on our common accessible subset.

Evaluation benchmarks

Table 1 summarises the eleven evaluation benchmarks: eight for video–text retrieval and three for audio–text retrieval. These benchmarks differ in their temporal structure, annotation granularity, and available modalities, enabling evaluation across short videos, multi-shot narratives, long-video excerpts, and sound recordings. We do not use any target benchmarks for training in this work, and we perform all evaluations without target-specific adaptation. When several alternative captions describe an item, we select the first caption in the annotation file. DiDeMo and ActivityNet instead use paragraph queries constructed from the sentences associated with each video, as detailed below. All methods we evaluate use the same queries and candidate collections.

DatasetStreamsModeText queryTrainValTestFramesAudio (10 s)Max caption length
MSR-VTT [37]VV, AA, SST–V, T–VA, T–VAScaption9,000–1,0008140
DiDeMo [38]VV, AAT–V, T–VAparagraph8,3941,0651,0038270
ActivityNet [39]VV, AAT–V, T–VAparagraph10,009–4,9178270
VATEX [40]VV, AA, SST–V, T–VA, T–VAScaption14,060–4318140
Shot2Story [41]VV, AAT–V, T–VAvideo summary14,0161,9824,0238170
MSVD [42]VVT–Vcaption1,2001006708–40
FLARE [43]VV, AAT–V, T–VAvision / unified caption––5,0008170
VALOR-32K [5]VV, AAT–V, T–VAaudio-visual caption25,3273,5003,4918170
AudioCaps [44]VV, AAT–AVaudio caption––7008140
Clotho [45]AAT–Acaption3,8391,0451,045–140
SoundDescs [46]AAT–Adescription23,0854,9474,947–270

Table 1. Statistics of the evaluation benchmarks. TT, VV, AA, and SS denote text, video, audio, and subtitles. Mode lists the modalities compared with the text, with the primary modality first: T–V, T–VA, and T–VAS on the video benchmarks, and T–A and T–AV on the audio benchmarks. Retrieval is evaluated in both directions: text-to-video and video-to-text on the video benchmarks, and text-to-audio and audio-to-text on the audio benchmarks. Text query is the annotation used on the text side. Train and Val give the split sizes; we do not use them, since all evaluations are zero-shot. Test is the number of evaluated items. Frames is the number of video frames and Audio the number of 10 s audio segments sampled per clip. Max caption length is the maximum number of text tokens kept per query; longer queries are truncated.

Video–text retrieval. MSR-VTT [37] contains short videos paired with natural-language descriptions. We evaluate on the 1,000-video test split used by VAST [4] and GRAM [6]. We evaluate DiDeMo [38] and ActivityNet [39] as paragraph-to-video retrieval tasks by concatenating the annotated sentences associated with each video into a single query. These settings assess retrieval using descriptions that can encompass multiple events within a video. Following VAST and GRAM, we evaluate on 1,003 videos for DiDeMo and 4,917 videos for ActivityNet.

VATEX [40] supports evaluation with visual, audio, and subtitle information. We follow the test split adopted by VAST and GRAM; however, when we collected the data, only 431 videos from this split remained accessible. All methods we evaluate use this identical subset. We therefore treat published results from a different video collection as contextual references rather than directly matched comparisons. Shot2Story20K [41] contains multi-shot videos paired with summary descriptions. Its 4,023-video test split allows us to examine retrieval when a query describes content distributed across several shots. MSVD [42] contains short videos with multiple textual descriptions, and we evaluate its 670-video test split. Since audio is unavailable in our MSVD evaluation data, only the T–V setting is reported.

FLARE [43] provides clips extracted from longer videos, together with vision, audio, and unified audiovisual captions. We evaluate a fixed subset of 5,000 clips, using vision captions for T–V and unified captions for T–VA. These settings differ in both query content and candidate modalities; therefore, their performance differences do not isolate the contribution of audio.

VALOR-32K [5] consists of approximately 10 s clips accompanied by captions describing both visible content and audible events. We evaluate its 3,491-item test split. Its audiovisual annotations complement benchmarks whose descriptions primarily emphasize visual content.

Audio–text retrieval. AudioCaps [44] pairs audio events with natural-language descriptions. We use the same 700-item test split as VAST [4] and GRAM [6]. The corresponding video is available alongside the audio, allowing evaluation in the T–AV setting.

Clotho v2.1 [45] contains audio recordings with five captions per recording. We evaluate its 1,045-item evaluation split using the first caption associated with each recording. SoundDescs [46] contains sound-effect recordings paired with textual descriptions, and we evaluate on 4,947 test items. Both Clotho and SoundDescs are evaluated in the T–A setting, assessing retrieval from acoustic information alone.

Compared Methods and Implementation Details

Compared methods

We evaluate the proposed refinement framework with two retrieval backbones, GRAM3 [6] and HYPERGRAM4 [9]. GRAM measures multimodal compatibility through Gramian volumes, whereas HYPERGRAM combines Euclidean and hyperbolic volume-based representations. Evaluating both backbones allows us to examine whether the benefits of hypergraph refinement persist across these two geometric scoring formulations. In each case, the refinement module operates on the candidate representations during training while retaining the backbone’s retrieval scoring function.

For each backbone, we compare the model without hypergraph refinement against three configurations: HyVol5 [12], the weighted hypergraph configuration WHyVol, and the weighted relational configuration WRHG. HyVol provides the reference configuration with within-candidate and caption-guided semantic hyperedges. The WHyVol configuration introduces learned vertex contributions and symmetric semantic affinity weights within additive propagation. The WRHG configuration additionally distinguishes relation types and participant roles through relational propagation. These comparisons evaluate the full configurations; § 5.0.4 also examines the hypergraph’s sensitivity to neighborhood size kk.

We additionally compare against TRIANGLE6 [7] and PMRL7 [8]. TRIANGLE offers an alternative volume-based retrieval approach using triangular area, while PMRL provides an additional multimodal representation-learning baseline. We include results from other published retrieval systems to contextualize performance within the broader literature.

Comparison protocol

We distinguish between methods evaluated using released checkpoints, methods trained in this work, and results taken from prior publications. For GRAM, TRIANGLE, and PMRL, we use the checkpoints released by their authors. Since a pretrained HYPERGRAM checkpoint is unavailable, we train HYPERGRAM using the authors’ public implementation and training recipe. All models trained in this work start from the VAST initialization and use the effective training corpus described in § 4.1.1.

All checkpoints evaluated by us are tested on the same benchmark items and query annotations within each retrieval setting. We evaluate GRAM using its retrieval protocol, and we use TRIANGLE’s released evaluation code. Any method-specific scoring or post-processing is retained and must therefore be considered when interpreting comparisons across methods.

The common evaluation data remove differences arising from query selection and candidate availability. However, the released checkpoints may differ from models trained in this work in their effective training samples and optimization history. Comparisons against these checkpoints therefore measure performance relative to the available reference systems, rather than isolating the effect of a single architectural component.

We report published scores separately as reference results because their evaluation collections, input processing, or training conditions may differ from ours. This distinction is particularly relevant for benchmarks affected by video availability, including the reduced VATEX test subset.

Input processing and model configuration

Backbone encoders. All compared models share the modality encoders of VAST [4]: EVA-CLIP ViT-g/148 for video frames, BEATs9 for audio, and BERT-base10 for both captions and subtitles. We remove the VAST fusion layers, project each stream to a 512-dimensional space, and ℓ2\ell_2-normalize it so that the backbone score acts directly on the per-modality embeddings. The backbone temperature is loaded from its checkpoint and kept trainable.

Training recipe. HyVol, wHyVol and wRHG share the same optimization and data settings on both backbones. Starting from the VAST checkpoint, we train each model for one epoch (533 steps) on the recovered training corpus with a global batch of 256 clips, sampling two frames and one 10 s audio segment per clip. No encoder is frozen. We use AdamW with β=(0.9,0.98)\beta=(0.9, 0.98), weight decay 0.01 and gradient-norm clipping at 2.0. The learning rate is 2×10−52 \times10^{-5} for the backbone, 5×10−75 \times10^{-7} for the EVA-CLIP visual encoder and 5×10−45 \times10^{-4} for the hypergraph module, with linear warm-up over the first 10% of the steps followed by linear decay.

Hypergraph and loss settings. We split the batch across four GPUs, and each GPU builds its hypergraph from its own 64 clips, so each GPU’s batch also forms the pool of semantic neighbors. All three configurations use k=12k=12, which gives keff=12k_{\mathrm{eff}}=12 in Eq. (1), no similarity filter, a semantic-edge dropout of 0.3, and L=2L=2 propagation layers. The auxiliary weights in Eq. (10) are λdoc=1.0\lambda_{\mathrm{doc}} = 1.0 for pooled caption–candidate alignment, λreg=0.1\lambda_{\mathrm{reg}} = 0.1 for variance regularisation (with stability constant 10−410^{-4}), and λitm=0.1\lambda_{\mathrm{itm}} = 0.1 for the matching loss. The volume-based retrieval and alignment losses use label smoothing of 0.1. Table 8 in Appendix C lists all hyperparameters.

Backbone
InitializationVAST foundation checkpoint
Vision encoderEVA-CLIP ViT-g/14
Audio encoderBEATs
Text encoderBERT-base (caption and subtitle)
Projection dimension dd512, followed by ℓ2\ell_2 normalisation
Trainable modulesall encoders, projections, hypergraph, and heads
Hypergraph construction
Neighbours kk / adaptive cap12/⌊Bshard/4⌋=1612 / \lfloor B_{\mathrm{shard}}/4 \rfloor= 16
Neighbourhoodmutual top-kk cosine on stop-gradient captions
Similarity filterdisabled
Semantic-edge dropout0.3
Relation families (wRHG)document, same-modality, cross-modality, joint-candidate
Learned weights
Semantic weightssoftmax attention (HyVol); kernel of Eq. (2) (wHyVol, wRHG)
Affinity temperature τa\tau_a0.5, learned (wHyVol, wRHG)
Affinity projection WaW_aidentity initialisation (wHyVol, wRHG)
Vertex weight βim\beta_i^mfixed at 1 (HyVol); learned in (0, 2), initialised to 1 (wHyVol, wRHG)
Propagation
Operatoradditive (HyVol, wHyVol); relational (wRHG)
Layers LL2
ActivationGELU (HyVol, wHyVol); ReLU (wRHG); identity in the final layer
Mixing αℓ\alpha_\ell, Eq. (6) (wRHG)initialised to 0.5, learned
Losses
Retrieval objectivebackbone volume loss, symmetric cross-entropy
Backbone temperatureloaded from the checkpoint, learned
Label smoothing0.1 (volume-based)
Alignment weight λdoc\lambda_{\mathrm{doc}}1.0
Variance weight λreg/ϵ\lambda_{\mathrm{reg}} / \epsilon0.1/10−40.1 / 10^{-4}
Matching weight λitm\lambda_{\mathrm{itm}}0.1
HYPERGRAM backbone
Curvatureinitialised to 1, learned
Euclidean / hyperbolic weightsinitialised to 0.5 / 0.5, learned
Hyperbolic gradient clip1.0
Optimisation
OptimiserAdamW, β=(0.9,0.98)\beta=(0.9, 0.98)
Learning rate2×10−52 \times10^{-5} backbone; 5×10−75 \times10^{-7} visual encoder; 5×10−45 \times10^{-4} hypergraph
Weight decay / gradient clip0.01 / 2.0
Schedulerlinear; warm-up ratio 0.1
Batch size256 global; 64 per GPU
Epochs / steps1 / 533
Training frames / audio2 / one 10 s segment
Precision / parallelismfp16 / DDP over four A100 GPUs
Seed50 (single run)
Model selection and inference
ValidationMSR-VTT 1k, T-VAS, ten times per epoch
Selection criteriontext-to-video R@1 of the matching score
Sampled frames8
Audio segments1 or 2 of 10 s, per benchmark
Maximum query length40 or 70 tokens, per benchmark
Hypergraph at test timeremoved; encoder outputs scored directly
Test-time missingness rates rr0, 25, 50, 75, 90%

Table 8. Hyperparameters. One configuration is used for both backbones and all eleven benchmarks, without benchmark-specific tuning; settings that differ between configurations name the configurations they apply to.

Differences across configurations. The three configurations differ only in their hypergraph weighting and propagation, as defined in § 3.7. For all three configurations on HYPERGRAM, we retain the authors’ hybrid-geometry settings: a learnable curvature initialized to 1, learnable Euclidean and hyperbolic weights initialized to 0.5, and hyperbolic gradient clipping at 1.0.

Hypergraph at training and inference. We apply hypergraph refinement to the observed candidate modalities during training. Training captions guide the construction of semantic neighborhoods but are not included as candidate vertices. Missing modalities are excluded from the hypergraph, so they contribute neither vertex features nor propagated messages. At inference, we discard the hypergraph, propagation layers, and auxiliary pooling head, and retrieve scores using the corresponding backbone’s scoring function in Eq. (9). Consequently, the refinement configurations introduce no additional hypergraph computation during retrieval and require no neighborhood construction over evaluation candidates.

Input processing. Table 1 lists the sampling used at test time. Eight frames are sampled for every candidate that contains video, and audio is represented by one or two 10 s segments depending on the benchmark. Queries are truncated to 40 or 70 tokens, which accommodates the different clip durations and description lengths of the benchmarks.

Checkpoint selection. We validate ten times during the epoch on the MSR-VTT 1k split in the T–VAS setting and keep the checkpoint with the highest text-to-video R@1 of the matching score. MSR-VTT is therefore the only benchmark used for model selection; the other ten benchmarks are used exclusively for testing. The neighbourhood size kk was likewise chosen from the sweep in § 5.0.4, which uses four of the evaluation benchmarks; a single value is used for all benchmarks and configurations.

Hardware and runs. All models are trained on four NVIDIA A100 GPUs with fp16 mixed precision, while hypergraph aggregation is computed in float32 for numerical stability. Following GRAM [6] and HYPERGRAM [9], which report results from a single training run, we train each configuration once.

Zero-Shot Retrieval Performance

We evaluate retrieval performance across the eleven benchmarks described in § 4.1.2. Table 2 and Table 3 report text-to-video and video-to-text retrieval, respectively, while Table 5 presents both directions of audio–text retrieval. We first examine improvements over the corresponding backbone, then compare refinement configurations and their behavior across modalities. All improvements discussed below are absolute differences in R@1, expressed in percentage points.

DatasetModeTRIANGLE*GRAM*+HYVOL+WHYVOL+WRHGHYPERGRAM†+HYVOL+WHYVOL+WRHG
MSR-VTT [37]T–V–51.952.6 (+0.7)53.4 (+1.5)53.0 (+1.1)50.852.5 (+1.7)54.3 (+3.5)54.2 (+3.4)
T–VA50.552.054.0 (+2.0)54.9 (+2.9)54.7 (+2.7)51.754.2 (+2.5)55.0 (+3.3)55.1 (+3.4)
T–VAS–51.855.8 (+4.0)56.8 (+5.0)57.4 (+5.6)52.356.1 (+3.8)57.0 (+4.7)57.3 (+5.0)
DiDeMo [38]T–V–51.353.8 (+2.5)54.1 (+2.8)54.0 (+2.7)50.053.6 (+3.6)54.3 (+4.3)54.0 (+4.0)
T–VA51.050.654.6 (+4.0)53.8 (+3.2)54.1 (+3.5)50.454.2 (+3.8)54.6 (+4.2)54.6 (+4.2)
ActivityNet [39]T–V–55.556.2 (+0.7)59.6 (+4.1)59.6 (+4.1)56.156.7 (+0.6)59.5 (+3.4)59.9 (+3.8)
T–VA56.056.456.9 (+0.5)60.8 (+4.4)60.7 (+4.3)56.857.8 (+1.0)60.5 (+3.7)60.9 (+4.1)
VATEX [40]T–V–87.292.1 (+4.9)91.9 (+4.7)92.3 (+5.1)88.492.6 (+4.2)92.3 (+3.9)92.3 (+3.9)
T–VA88.988.493.0 (+4.6)93.5 (+5.1)93.5 (+5.1)88.693.0 (+4.4)93.7 (+5.1)93.5 (+4.9)
T–VAS–88.494.0 (+5.6)93.5 (+5.1)94.0 (+5.6)89.893.3 (+3.5)93.5 (+3.7)93.5 (+3.7)
Shot2Story [41]T–V–70.575.1 (+4.6)75.6 (+5.1)76.7 (+6.2)75.576.0 (+0.5)76.3 (+0.8)76.5 (+1.0)
T–VA75.870.574.8 (+4.3)76.3 (+5.8)77.4 (+6.9)75.074.6 (−0.4)75.9 (+0.9)76.6 (+1.6)
MSVD [42]T–V–70.475.8 (+5.4)74.3 (+3.9)73.6 (+3.2)69.675.8 (+6.2)75.5 (+5.9)75.7 (+6.1)
FLARE [43]T–V–47.452.1 (+4.7)52.6 (+5.2)54.1 (+6.7)52.353.4 (+1.1)53.9 (+1.6)53.8 (+1.5)
T–VA56.146.052.0 (+6.0)51.7 (+5.7)53.9 (+7.9)52.952.1 (−0.8)52.1 (−0.8)52.8 (−0.1)
VALOR-32K [5]T–V–64.671.5 (+6.9)70.2 (+5.6)70.2 (+5.6)65.671.7 (+6.1)70.6 (+5.0)70.4 (+4.8)
T–VA75.975.280.5 (+5.3)80.4 (+5.2)80.8 (+5.6)75.980.9 (+5.0)80.5 (+4.6)80.9 (+5.0)

Table 2. Zero-shot text-to-video retrieval (R@1) on eight video benchmarks. For FLARE, T–V uses the vision captions and T–VA the unified captions. The compared models TRIANGLE [7], GRAM [6] and HYPERGRAM [9] are all volume-based: they score a clip by the area of a triangle, the Gramian volume, and a combination of Euclidean and hyperbolic volumes, respectively; +HYVOL, +WHYVOL and +WRHG denote the backbone trained with HyVol [12], weighted HyVol and the weighted relational hypergraph, respectively. ⋆Authors’ released checkpoint (TRIANGLE, GRAM), evaluated by us on the same test data: GRAM with its retrieval protocol, TRIANGLE with its released evaluation code. †Trained by us with the authors’ released code and recipe (HYPERGRAM). Numbers in parentheses give the change with respect to the backbone (GRAM or HYPERGRAM); green marks an improvement, red a drop. A dash (–) marks a setting that is not available. Bold marks the best result within each backbone group.

DatasetModeBaseline: TRIANGLE⋆^\starBaseline: GRAM⋆^\starGRAM backbone: +HyVolGRAM backbone: +wHyVolGRAM backbone: +wRHGHyperGRAM backbone: HyperGRAM†^\daggerHyperGRAM backbone: +HyVolHyperGRAM backbone: +wHyVolHyperGRAM backbone: +wRHG
MSR-VTT [37]T–V–48.754.5 (+5.8)55.1 (+6.4)56.5 (+7.8)48.754.2 (+5.5)55.2 (+6.5)55.9 (+7.2)
T–VA49.749.355.8 (+6.5)56.3 (+7.0)57.2 (+7.9)49.856.6 (+6.8)56.5 (+6.7)57.2 (+7.4)
T–VAS–50.158.1 (+8.0)58.0 (+7.9)58.0 (+7.9)50.959.2 (+8.3)58.8 (+7.9)59.3 (+8.4)
DiDeMo [38]T–V–49.053.8 (+4.8)54.0 (+5.0)54.4 (+5.4)50.753.8 (+3.1)54.6 (+3.9)54.8 (+4.1)
T–VA49.449.054.1 (+5.1)55.5 (+6.5)55.0 (+6.0)50.854.7 (+3.9)55.1 (+4.3)55.5 (+4.7)
ActivityNet [39]T–V–49.356.2 (+6.9)61.4 (+12.1)61.6 (+12.3)50.555.8 (+5.3)61.1 (+10.6)61.3 (+10.8)
T–VA52.349.756.6 (+6.9)62.7 (+13.0)63.0 (+13.3)50.956.4 (+5.5)62.8 (+11.9)63.0 (+12.1)
VATEX [40]T–V–84.792.3 (+7.6)92.6 (+7.9)92.6 (+7.9)86.392.4 (+6.1)92.3 (+6.0)92.1 (+5.8)
T–VA86.586.592.8 (+6.3)94.0 (+7.5)93.7 (+7.2)88.492.9 (+4.5)93.5 (+5.1)93.3 (+4.9)
T–VAS–86.593.8 (+7.3)94.4 (+7.9)94.4 (+7.9)87.893.9 (+6.1)94.0 (+6.2)94.0 (+6.2)
Shot2Story [41]T–V–66.477.6 (+11.2)75.7 (+9.3)76.4 (+10.0)73.078.1 (+5.1)75.9 (+2.9)75.8 (+2.8)
T–VA74.866.277.6 (+11.4)75.7 (+9.5)76.3 (+10.1)73.178.1 (+5.0)76.2 (+3.1)76.0 (+2.9)
MSVD [42]T–V–69.074.5 (+5.5)74.6 (+5.6)74.5 (+5.5)67.074.6 (+7.6)74.6 (+7.6)75.2 (+8.2)
FLARE [43]T–V–47.855.6 (+7.8)55.7 (+7.9)55.8 (+8.0)52.456.4 (+4.0)56.4 (+4.0)55.9 (+3.5)
T–VA55.046.855.5 (+8.7)54.6 (+7.8)55.0 (+8.2)52.656.4 (+3.8)55.9 (+3.3)55.8 (+3.2)
VALOR-32K [5]T–V–61.471.8 (+10.4)69.9 (+8.5)70.2 (+8.8)62.572.5 (+10.0)71.4 (+8.9)71.5 (+9.0)
T–VA74.771.281.6 (+10.4)81.1 (+9.9)80.8 (+9.6)72.482.2 (+9.8)82.0 (+9.6)81.8 (+9.4)

Table 3. Zero-shot video-to-text retrieval (R@1) on eight video benchmarks. For FLARE, T–V uses the vision captions and T–VA the unified captions. The compared methods, evaluation settings and notation are described in the caption of Table 2. Bold marks the best result within each backbone group.

DatasetModeTRIANGLE⋆^{\star}GRAM backbone: GRAM⋆^{\star}+HYVOL+WHYVOL+WRHGHYPERGRAM backbone: HYPERGRAM†^{\dagger}+HYVOL+WHYVOL+WRHG
Text-to-audio
AudioCaps [44]T–AV37.433.435.7 (+2.3)35.2 (+1.8)35.4 (+2.0)32.436.1 (+3.7)36.9 (+4.5)36.8 (+4.4)
Clotho [45]T–A–11.911.0 (−0.9)13.2 (+1.3)12.3 (+0.4)12.911.5 (−1.4)12.3 (−0.6)13.0 (+0.1)
SoundDescs [46]T–A–3.64.6 (+1.0)4.5 (+0.9)4.5 (+0.9)3.94.5 (+0.6)4.2 (+0.3)4.1 (+0.2)
Audio-to-text
AudioCaps [44]T–AV38.233.433.7 (+0.3)33.8 (+0.4)34.9 (+1.5)33.236.4 (+3.2)37.5 (+4.3)36.2 (+3.0)
Clotho [45]T–A–11.512.5 (+1.0)15.0 (+3.5)14.0 (+2.5)12.112.4 (+0.3)13.7 (+1.6)13.6 (+1.5)
SoundDescs [46]T–A–4.34.7 (+0.4)4.5 (+0.2)4.3 (0.0)4.24.6 (+0.4)4.4 (+0.2)4.3 (+0.1)

Table 5. Zero-shot text-to-audio and audio-to-text retrieval (R@1). The compared methods, evaluation settings and notation are described in the caption of Table 2. Bold marks the best result within each backbone group.

Video–text retrieval

Performance across backbones and retrieval directions. Hypergraph refinement improves retrieval across a broad range of benchmarks. With GRAM, wRHG outperforms the backbone in all 17 text-to-video settings and all 17 video-to-text settings. With HYPERGRAM, it improves 16 of the 17 text-to-video settings and all 17 video-to-text settings. These results show that the benefits extend across both backbone formulations and retrieval directions, although their magnitude depends on the benchmark and available modalities. Figure 3 visualizes these results in the T–VA setting, with the minimum and maximum R@1 of each benchmark shown below its name.

Figure 3. Zero-shot text-to-video and video-to-text retrieval (R@1) with the GRAM backbone (top) and the HYPERGRAM backbone (bottom) on seven video benchmarks, all evaluated in the T–VA setting. Each axis is scaled independently to the range of R@1 values on that benchmark; the minimum and maximum R@1 are shown in parentheses below its name, so distances are comparable only within an axis. The exact values are the T–VA rows of Table 2 and Table 3.

ActivityNet exhibits particularly substantial improvements. In the T–VA setting, GRAM +wRHG increases text-to-video R@1 from 56.4 to 60.7 and video-to-text R@1 from 49.7 to 63.0. HYPERGRAM +wRHG shows a similar pattern, increasing the corresponding scores from 56.8 to 60.9 and from 50.9 to 63.0. The larger gains in video-to-text retrieval indicate that the effect of refinement is not symmetric across retrieval directions. Although both directions use the same compatibility scores, ranking candidates for each textual query and ranking descriptions for each video impose different retrieval conditions.

Effect of weighting and relational modeling. The weighted configurations improve on HyVol [12] in several settings. On ActivityNet T–VA, replacing HYVOL with WHYVOL increases GRAM text-to-video R@1 from 56.9 to 60.8 and video-to-text R@1 from 56.6 to 62.7. With HYPERGRAM, the corresponding scores increase from 57.8 to 60.5 and from 56.4 to 62.8. The consistent pattern across backbones indicates that the weighted configuration is particularly effective on ActivityNet.

Relational modeling provides further benefits in selected settings. On Shot2Story text-to-video retrieval with GRAM, wRHG reaches 76.7 in T–V and 77.4 in T–VA, compared with 75.6 and 76.3 for WHYVOL. On MSR-VTT video-to-text retrieval in T–V, wRHG improves over WHYVOL from 55.1 to 56.5 with GRAM and from 55.2 to 55.9 with HYPERGRAM. These comparisons identify settings in which explicit modeling of distinct interactions provides improvements over weighted additive propagation.

The ordering of the configurations is nevertheless not uniform. HYVOL [12] achieves the highest text-to-video score on MSVD for both backbones, reaching 75.8. It also outperforms the weighted configurations on Shot2Story video-to-text retrieval and VALOR-32K video-to-text retrieval. Thus, the additional weighting and relational mechanisms provide benchmark-dependent benefits rather than consistent improvement across settings. § 5.0.4 examines one such design choice, the neighbourhood size, under otherwise fixed settings.

Influence of available modalities. Additional candidate streams can improve retrieval when they provide information relevant to the query. On MSR-VTT, GRAM + wRHG increases text-to-video R@1 from 53.0 with video alone to 54.7 with video and audio, and to 57.4 when subtitles are also included. HYPERGRAM + wRHG follows the same trend, achieving 54.2, 55.1, and 57.3, respectively.

The contribution of audio is particularly pronounced on VALOR-32K, whose descriptions cover both visible and audible events. Adding audio increases text-to-video R@1 from 70.2 to 80.8 for GRAM + wRHG and from 70.4 to 80.9 for HYPERGRAM + wRHG. This pattern is consistent with the audiovisual content of the queries and illustrates the importance of evaluating fusion under annotations that describe the available streams.

Additional modalities do not guarantee higher performance. On FLARE, HYPERGRAM +wRHG obtains 52.8 in T–VA, slightly below the backbone score of 52.9. Moreover, FLARE uses different descriptions for T–V and T–VA, so we cannot attribute their difference solely to the inclusion of audio. These results highlight the dependence of retrieval performance on the combination of candidate information, query content, and backbone.

Audio–text retrieval

Table 5 shows that refinement also benefits audio–text retrieval, though the gains are less uniform than on the video benchmarks. On AudioCaps, all three configurations improve over their corresponding backbones in both retrieval directions. HYPERGRAM +wHyVol yields the strongest results, increasing text-to-audio R@1 from 32.4 to 36.9 and audio-to-text R@1 from 33.2 to 37.5. HYPERGRAM +wRHG reaches 36.8 and 36.2, respectively, indicating that relational propagation offers no additional advantage over weighted additive propagation in this setting.

Clotho reveals a stronger dependence on the configuration and retrieval direction. For text-to-audio retrieval, HyVol [12] reduces R@1 from 11.9 to 11.0 with GRAM and from 12.9 to 11.5 with HYPERGRAM. The weighted configurations recover part or all of this loss: GRAM +wHyVol reaches 13.2, while HYPERGRAM +wRHG reaches 13.0. In audio-to-text retrieval, wHyVol achieves the highest score for both backbones, increasing R@1 from 11.5 to 15.0 with GRAM and from 12.1 to 13.7 with HYPERGRAM.

On SoundDescs, improvements are small by comparison, and HyVol achieves the highest scores among the refinement configurations for both backbones and retrieval directions. For example, GRAM +HyVol increases text-to-audio R@1 from 3.6 to 4.6, while HYPERGRAM +HyVol increases it from 3.9 to 4.5. The low absolute recall indicates that this benchmark remains challenging under the evaluated training and retrieval protocol.

Unlike AudioCaps, Clotho and SoundDescs contain only audio on the candidate side. The results therefore assess transfer to unimodal candidates rather than the direct benefit of combining multiple streams at inference. They demonstrate that training-time refinement can benefit audio-only retrieval, while also showing that the more elaborate configurations are not consistently preferable in this setting.

Comparison with existing methods

Table 6 compares video retrieval results with those of existing methods, grouped by available candidate modalities. Both GRAM +wRHG and HYPERGRAM +wRHG outperform the re-evaluated PMRL checkpoint [8] in every reported setting in this table. On ActivityNet T–VA, for example, PMRL obtains 54.1 for text-to-video retrieval and 48.6 for video-to-text retrieval, compared with 60.9 and 63.0 for HYPERGRAM +wRHG. On MSR-VTT T–VAS, HYPERGRAM +wRHG achieves 57.3 and 59.3, compared with 54.3 and 52.9 for PMRL.

The comparison with Triangle [7] is more mixed. On VALOR-32K T–VA, GRAM +wRHG reaches 80.8 for text-to-video retrieval, compared with 75.9 for Triangle. Conversely, Triangle obtains 56.1 on FLARE T–VA, exceeding the corresponding scores of 53.9 and 52.8 for GRAM +wRHG and HYPERGRAM +wRHG. On AudioCaps, Triangle also remains stronger than the refinement configurations, achieving 37.4 for text-to-audio retrieval and 38.2 for audio-to-text retrieval.

Table 7 provides additional context for AudioCaps text-to-audio retrieval. In the T–AV setting, GRAM +wRHG and HYPERGRAM +wRHG achieve 35.4 and 36.8, respectively, compared with 34.4 for the re-evaluated PMRL checkpoint. Published results obtained with other pretraining corpora or evaluation procedures are included as reference values. They establish the broader performance context but do not constitute controlled comparisons of fusion mechanisms.

MethodT–AT–AV
AVFIC§ [21]8.710.6
ImageBind§ [2]9.3–
LanguageBind§ [3]19.7–
VIP-ANT§ [22]27.7–
VAST§ [4]–32.1
PMRL⋆ [8]–34.4
GRAM+WRHG (ours)–35.4
HYPERGRAM+WRHG (ours)–36.8

Table 7. Comparison with published text-to-audio retrieval results (R@1) on AudioCaps [44]. §Published numbers (reference only); ⋆released checkpoint evaluated by us.

Neighbourhood-Size Sensitivity

The neighbourhood size kk in Eq. (1) determines how many caption neighbours each candidate considers before the mutual-neighbour test is applied. Table 4 varies kk over {4,8,12,16}\{4, 8, 12, 16\} for GRAM +HyVol while keeping all other settings fixed. Since each GPU builds its hypergraph from 64 clips, keff=min⁡(k,16)k_{\mathrm{eff}} = \min(k, 16), and k=16k = 16 is the largest neighbourhood a batch can support.

DatasetModeT→V: k=4k=4T→V: k=8k=8T→V: k=12k=12T→V: k=16k=16V→T: k=4k=4V→T: k=8k=8V→T: k=12k=12V→T: k=16k=16
MSR-VTT [37]T–V50.052.152.651.147.752.254.554.4
T–VA50.652.554.052.149.051.855.855.7
T–VAS53.254.355.854.751.953.758.157.6
DiDeMo [38]T–V50.051.853.852.548.052.653.853.8
T–VA50.351.454.652.448.452.754.153.3
ActivityNet [39]T–V53.355.956.254.947.850.656.249.2
T–VA53.956.756.955.448.450.256.649.8
VATEX [40]T–V87.791.992.191.991.492.192.391.6
T–VA89.091.993.093.391.792.192.892.3
T–VAS90.192.694.093.593.393.793.893.0

Table 4. Sensitivity to the number of semantic neighbours kk. Values are R@1 for text-to-video (T→V) and video-to-text (V→T) retrieval; bold marks the best tested kk in each row, and k=12k=12 is used in all main experiments. The reported configuration uses HYVOL on top of the GRAM backbone.

Increasing kk from 4 to 12 improves R@1 in every setting, and the effect is most pronounced for video-to-text retrieval. Averaged over the modality sets of each benchmark, the video-to-text gain is 6.6 points on MSR-VTT, 5.8 on DiDeMo, and 8.3 on ActivityNet; on VATEX, where R@1 already exceeds 90, it is 0.8 points. The corresponding text-to-video gains are smaller but consistent, for example from 53.2 to 55.8 on MSR-VTT T–VAS and from 50.3 to 54.6 on DiDeMo T–VA. A further increase to k=16k = 16 brings no average benefit and markedly degrades video-to-text retrieval on ActivityNet, where R@1 falls from 56.2 to 49.2 in T–V and from 56.6 to 49.8 in T–VA. Across the 20 settings, k=12k = 12 yields the best or tied-best result in 19; the only exception is VATEX T–VA text-to-video retrieval, where k=16k = 16 is 0.3 points higher. We therefore adopt k=12k = 12 for all configurations and benchmarks.

Robustness to Missing Modalities

We further evaluate retrieval when individual clips lose part of their multimodal information. Unlike the complete-modality comparisons, this setting introduces heterogeneous observations within the same collection: some clips retain all available streams, while others are represented by a reduced modality set. Figure 4–7 examine whether improvements obtained through training-time hypergraph refinement persist under these conditions.

Figure 4. Zero-shot text-to-video retrieval under missing modalities with the GRAM backbone on seven video benchmarks. Each panel shows the R@1 gain of +HYVOL, +WHYVOL and +WRHG over GRAM (dashed line at zero) as a fraction rr of the gallery clips that lose one randomly chosen modality; the per-clip masks are fixed and identical for all methods, and r=0r=0 is the standard setting. MSR-VTT and VATEX use the T–VAS modality set, and the other benchmarks use T–VA.

Evaluation protocol

The evaluation covers seven video benchmarks: MSR-VTT, DiDeMo, ActivityNet, VATEX, Shot2Story, FLARE, and VALOR-32K. MSR-VTT and VATEX use the T–VAS setting, while the remaining benchmarks use T–VA. MSVD is excluded because its evaluation configuration contains only video.

For a fraction rr of the clips, one randomly selected modality is removed. We consider missingness rates of 0%, 25%, 50%, 75%, and 90%, with 0% corresponding to the standard evaluation setting. The per-clip masks are fixed and identical across methods, ensuring that performance differences are measured under the same availability conditions. Each affected clip retains two modalities in T–VAS and one modality in T–VA. Thus, the missingness rate specifies the proportion of incomplete clips rather than the proportion of all modality observations removed.

The affected clips serve as gallery items in text-to-video retrieval and as queries in video-to-text retrieval. Figures 4 and 5 report results with GRAM, while Figures 6 and 7 report results with HYPERGRAM. Each curve shows the difference in R@1 between a refinement configuration and its corresponding backbone at the same missingness rate. Positive values therefore indicate an advantage over the backbone under incomplete observations; they do not imply that absolute retrieval performance is unchanged as modalities become unavailable. Table 9 in Appendix D lists the underlying R@1 values for each method compared at each missingness rate.

Results with Gram

Figure 4 shows that wHyVoL and wRHG retain positive text-to-video gains over GRAM on all seven benchmarks at every evaluated missingness rate. Their advantage therefore persists even when 90% of the clips lose one modality. Although the gains generally become smaller as missingness increases, the weighted configurations remain beneficial across the evaluated collections.

ActivityNet clearly distinguishes the weighted configurations from HyVol [12]. Without missing modalities, the T–VA gains over GRAM are 4.4 percentage points for wHyVoL and 4.3 points for wRHG, compared with 0.5 points for

This ordering does not hold uniformly. On DiDeMo, HyVol provides a larger text-to-video gain than either weighted configuration at the nonzero missingness rates. On VALOR-32K, all three configurations remain above GRAM throughout the sweep, and their gains do not decrease monotonically. These differences indicate that the relative benefit of weighting depends on the benchmark as well as the amount of missing information.

For video-to-text retrieval, Figure 5 shows positive gains for all three refinement configurations across all evaluated missingness rates. The gains generally narrow as more clips become incomplete. On ActivityNet, the complete-modality improvements of 13.0 points for wHyVol and 13.3 points for wRHG decrease with missingness, but both configurations remain above HyVol throughout the sweep. On VATEX, the weighted configurations likewise retain a clearer advantage at high missingness rates. Overall, the GRAM results show that refinement remains beneficial under incomplete observations, while the magnitude and ordering of the improvements vary across datasets.

Figure 5. Zero-shot video-to-text retrieval under missing modalities with the GRAM backbone: R@1 gain over GRAM (dashed line at zero). The protocol is the same as in Figure 4.

HyVol. As shown in Figure 4, both weighted configurations remain above the backbone throughout the missingness sweep, whereas HyVol falls below at 25%, 50%, and 75% missingness. The weighted configurations therefore maintain their advantage as audiovisual observations become incomplete.

A similar distinction appears on MSR-VTT. In the complete T–VAS setting, wHyVol and wRHG improve over GRAM by 5.0 and 5.6 points, respectively, compared with 4.0 points for HyVol. At 90% missingness, both weighted configurations remain positive, whereas HyVol falls slightly below the backbone. On VATEX, the weighted configurations also retain a larger advantage than HyVol at the highest missingness rate.

Results with HyperGram

The HyperGRAM results in Figure 6 and Figure 7 vary more across benchmarks. On MSR-VTT and ActivityNet, both weighted configurations retain positive gains in both retrieval directions at every evaluated missingness rate. However, Shot2Story and FLARE include conditions where refinement offers little benefit or underperforms the backbone.

Figure 6. Zero-shot text-to-video retrieval under missing modalities with the HYPERGRAM backbone: R@1 gain over HYPERGRAM (dashed line at zero). The protocol is the same as in Figure 4.

Figure 7. Zero-shot video-to-text retrieval under missing modalities with the HYPERGRAM backbone: R@1 gain over HYPERGRAM (dashed line at zero). The protocol is the same as in Figure 4.

On ActivityNet text-to-video retrieval, wHyVol and wRHG improve over HyperGRAM by 3.7 and 4.1 points in the complete T–VA setting, compared with 1.0 point for HyVol. Figure 6 shows that both weighted configurations remain above the backbone at all four nonzero missingness rates, whereas HYVOL falls below it. The video-to-text results in Figure 7 show the same preference for the weighted configurations: their initial gains of 11.9 and 12.1 points decrease with missingness but remain positive, while HYVOL falls below HYPERGRAM at 90% missingness.

The Shot2Story results establish a limitation. In text-to-video retrieval, wRHG’s advantage becomes negligible at 90% missingness, while HYVOL and wHYVOL underperform the backbone at several rates. In video-to-text retrieval, all three configurations fall below HYPERGRAM at the highest missingness rate. The improvement observed with complete observations therefore does not consistently transfer to severely incomplete clips on this benchmark.

VATEX illustrates the importance of retrieval direction. All three configurations remain above HYPERGRAM in text-to-video retrieval, although their relative ordering changes across the sweep. In video-to-text retrieval, wRHG maintains a positive margin throughout, while wHYVOL’s advantage approaches zero at 90% missingness. HYVOL falls below the backbone at 75% and 90% missingness. The relational configuration therefore provides the most persistent video-to-text advantage on this benchmark.

FLARE also exhibits non-monotonic behavior. In text-to-video retrieval, all three configurations underperform HYPERGRAM at 25%, 50%, and 75% missingness, then show gains at 90%. In video-to-text retrieval, all three fall below the backbone at 75% missingness and recover positive gains at 90%. Since these curves report differences from a backbone whose performance also changes with missingness, such recoveries do not imply an increase in absolute recall.

Taken together, Figure 4–Figure 7 show that the weighted configurations frequently preserve their advantage under incomplete observations, with particularly consistent results for GRAM. The exceptions with HYPERGRAM demonstrate that this advantage is not universal. Moreover, the hypergraph is absent during evaluation: the observed benefits arise through the learned encoder parameters, rather than through reconstruction of missing streams or information exchange between evaluation clips.

Discussion

Why training-time exchange helps. The refinement never touches the test-time score, yet all three configurations improve both backbones on almost every video setting. The only

path from the hypergraph to retrieval is the encoder weights, so the gains indicate that the encoders learn more useful per-modality embeddings when related clips can supply evidence during training. Several of the largest gains occur where the backbone is weak; for example, in ActivityNet video-to-text, GRAM obtains 49.7 R@1 and WRHG 63.0, and in FLARE in the T–VA setting, WRHG raises GRAM’s text-to-video R@1 from 46.0 to 53.9. In this sense the hypergraph acts as a fusion mechanism that operates across samples during learning rather than across modalities at inference.

Weighting versus relational structure. The two extensions address different weaknesses of HyVOL, and the results separate them only partly. The weighted configuration produces the clearest change: on ActivityNet it raises video-to-text R@1 by about six points over HyVOL with both backbones, and it keeps the gain under missing modalities where HyVOL falls below the backbone. Typed relations add a further but smaller improvement, mainly on Shot2Story and MSR-VTT, and on several benchmarks wHyVoL and wRHG are within a few tenths of a point of each other. A plausible reading is that deciding how much a view or a neighbour contributes matters more for low-quality data than deciding what kind of link connects them. HyVOL remains the best configuration for MSVD text-to-video and for SoundDesc, so the added weighting and typing are not uniformly beneficial.

MethodMSR-VTT [37] T→VMSR-VTT [37] V→TDiDeMo [38] T→VDiDeMo [38] V→TActivityNet [39] T→VActivityNet [39] V→TVATEX [40] T→VVATEX [40] V→T
Text–video (T–V)
HiTeA§ [16]34.446.843.256.5––––
ImageBind§ [2]36.836.8––––––
InternVideo-L§ [17]40.739.631.533.530.731.449.569.5
LanguageBind§ [3]44.840.939.939.841.039.1––
mPLUG-2§ [18]47.1–45.7–––––
Norton§ [47]10.7–––––––
OmniVL§ [19]42.034.640.633.3––––
TVTSv2§ [15]38.238.234.634.6––––
UMT§ [48]33.333.334.034.031.931.9––
UMT-L§ [48]40.740.748.624.941.939.4––
ViCLIP§ [49]42.441.318.427.915.124.0––
VideoCoCa§ [50]34.334.3––34.534.553.2–
VideoPrism-b§ [20]51.450.2––49.647.962.576.2
PMRL⋆ [8]51.349.653.050.553.848.087.785.4
GRAM+WRHG (ours)53.056.554.054.459.661.692.392.6
HYPERGRAM+WRHG (ours)54.255.954.054.859.961.392.392.1
Text–video–audio (T–VA)
VAST§ [4]49.343.749.548.251.446.880.077.1
PMRL⋆ [8]50.951.052.549.354.148.688.986.1
GRAM+WRHG (ours)54.757.254.155.060.763.093.593.7
HYPERGRAM+WRHG (ours)55.157.254.655.560.963.093.593.3
Text–video–audio–subtitle (T–VAS)
VAST§ [4]50.749.0––––82.178.7
PMRL⋆ [8]54.352.9––––89.687.0
GRAM+WRHG (ours)57.458.0––––94.094.4
HYPERGRAM+WRHG (ours)57.359.3––––93.594.0

Table 6. Comparison with published zero-shot retrieval results (R@1) on the four benchmarks for which such results exist. Rows are grouped by modality set, and methods are compared only within the same group. §Numbers reported in the original papers under their own pretraining data and protocols (reference only). ⋆Authors’ released checkpoint, evaluated by us on the same test data as our models. A dash (–) marks a result that is not available. Bold marks the best result within each modality group.

Asymmetry between retrieval directions. Gains are generally larger for video-to-text than for text-to-video retrieval,

reaching 13.3 points on ActivityNet against 4.3 in the other direction. The backbone scores are the same in both directions, but the rankings are formed over different sets. One possible explanation is that the refinement acts on the video embeddings, which are the side exposed to related clips during training, so a better-organised video space benefits most directly the direction in which videos act as queries.

Fusion under missing and unequal modalities. The missing-modality study shows that the benefit of training-time fusion largely survives when views disappear at test time, although the models never see incomplete clips during training. With GRAM, both weighted configurations remain above the backbone on all seven benchmarks at every rate, suggesting that learned vertex weights reduce the embeddings’ dependence on any single stream. The HYPERGRAM results are less regular, and on two benchmarks with long or multi-shot videos some conditions show no benefit, possibly because removing a stream there changes which modality carries the most information. On the audio benchmarks, gains are smaller and less consistent, which indicates that exchange between related clips is most useful when each clip offers several complementary streams to fuse.

Cost and applicability. Since no part of the hypergraph survives training, the refined models have the same parameters at inference, the same encoding cost and the same scoring function as their backbones, unlike test-time re-ranking and query-bank methods [28, 29]. The approach can therefore be added to an existing volume-based retrieval system without changing its deployment; the only extra cost is a small hypergraph module during the same one-epoch training.

Limitations

Backbone coverage. All compared models use the VAST encoders. We apply our models (HYVOL, WHYVOL, and WRHG) to the two volume-based backbones, GRAM and HYPERGRAM. TRIANGLE [7] is evaluated exclusively as a baseline because it is designed for exactly three modalities. Its score corresponds to the area spanned by three embeddings, the three-vector case of the GRAM volume; it therefore cannot incorporate subtitles as a fourth stream, and its underlying principle is already represented by GRAM. PMRL [8] is also evaluated exclusively as a baseline because it adopts a different formulation: it aligns modalities by maximizing the leading singular value of their stacked embeddings, ranks candidates using a single fused representation, and assumes complete modality availability. In contrast, our models refine separate per-modality embeddings. Integrating them with PMRL would therefore require adapting its objective rather than directly incorporating our module. Extensions to PMRL and to encoders pretrained on other corpora remain directions for future research.

Training-time refinement only. Our models employ the hypergraph feature fusion exclusively during training, with its influence incorporated into the encoder weights. At inference,

each clip is encoded and scored independently using the original backbone. Although this design preserves the inference cost, it constrains the model’s adaptability. Specifically, the deployed model cannot exploit gallery-specific structure, adapt to individual queries, or account for the particular pattern of missing streams encountered at test time. Such adaptation would require an explicit inference-time component, such as a lightweight graph constructed over the gallery, which lies beyond the scope of this work.

Training scale and single runs. Following GRAM, PMRL, and HYPERGRAM, we perform one epoch of continued pre-training on the 150K-clip subset of VAST-27M, of which 136,694 clips remained recoverable. Our conclusions are therefore restricted to this continued-pretraining regime and do not extend to training on the full corpus. Consistent with these studies, which do not report variability across repeated training runs, we train each configuration once using a fixed seed. All runs begin from the same pretrained checkpoint and differ only in the small hypergraph module. We therefore assess the consistency of improvements across eleven benchmarks and two backbones, rather than quantify variability across repeated runs.

Simulated missing modalities. None of the compared backbones evaluates retrieval with missing modalities. We therefore adopt the protocol introduced in our conference version [12], which, consistent with prior robustness studies [11], removes one randomly selected stream per clip at test time. This controlled protocol ensures identical evaluation conditions across methods. However, it only approximates the structured patterns of missing data encountered in real-world collections, in which entire sources or recording conditions may be absent.

Checkpoint selection. Following the released training configurations of GRAM, TRIANGLE, PMRL, and HYPERGRAM, we select the checkpoint by validating on MSR-VTT, which is also one of our evaluation benchmarks. We never use the remaining ten benchmarks during training or model selection, and the gains on them are similar to those on MSR-VTT, indicating that our conclusions do not depend on this choice. A held-out validation set drawn from the training distribution would make the evaluation fully independent of model selection.

Conclusion

We presented a hypergraph refinement framework for multimodal retrieval that models information exchange among observed modalities within a candidate and across semantically related training candidates. The framework includes HYVOL as a special case and accommodates two additional configurations: WHYVOL, which weights modality contributions and semantic connections, and WRHG, which additionally distinguishes relation types and participant roles. Missing modalities are naturally excluded from the hypergraph without imputation. We restrict refinement to training, while inference retains the corresponding backbone’s encoder outputs and retrieval scoring function.

Evaluation on 11 zero-shot benchmarks demonstrates improvements across video–text and audio–text retrieval. With GRAM, WRHG improves over the backbone in all 17 text-to-video settings and all 17 video-to-text settings. With HYPERGRAM, it improves 16 of the 17 text-to-video settings and all 17 video-to-text settings. The missing-modality evaluation on seven video benchmarks further shows that the weighted configurations often retain their advantage when individual clips lose an observed stream.

The results also identify the boundaries of these benefits. No refinement configuration is uniformly strongest across datasets and retrieval directions. HYVOL remains competitive or superior in several settings, audio-only retrieval exhibits smaller and less consistent gains, and some incomplete-modality conditions favor the unrefined HYPERGRAM backbone. These findings support training-time hypergraph refinement as an effective approach to improving retrieval across evaluated settings, while showing that the value of weighting and relational modeling depends on the available information, backbone, and task.

CRediT authorship contribution statement

Anindya Nag: Conceptualization, Investigation, Methodology, Writing - Original draft preparation. Ambuj Mehrish: Methodology, Validation, Writing - Original draft preparation, Writing - Review & Editing. Sebastiano Vascon: Supervision, Methodology, Conceptualization, Writing - Review & Editing.

Funding

This work was supported by the DoE 2023-2027 (MUR, AIS.DIP.ECCELLENZA2023_27.FF project).

Acknowledgements

This work was supported by the European Union’s Horizon Europe research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 101205348 (CASPER). This work was supported by the DoE 2023-2027 (MUR, AIS.DIP.ECCELLENZA2023_27.FF project). We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium, through the EuroHPC AI Factories “AI for Science and Collaborative EU Projects” Access call (proposal No. EHPC-AIF2026SC01-041). We further acknowledge the CINECA award under the ISCRA initiative (Class C projects IsCd5-CASPER-A and IsCd5_GMEG), for the availability of high-performance computing resources and support.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The code and trained weights of HYVOL, our conference model, are available here. The trained weights of WHYVOL and WRHG for both backbones are available here, and their code will be released upon publication. All datasets used in this work are publicly available from their original sources.

Use of Large Language Models

We used large language models only to assist with the writing, namely to correct grammar, refine wording and readability, and shorten the text where needed. The authors originated the research idea, method design, experimental protocol, analyses, and all reported results. The authors checked every suggested change against their experimental records and accept full responsibility for the content of this paper.

Reproducibility

We have made the code and trained weights of HyVol publicly available here. The trained weights of wHyVol and wRHG for both backbones are available here, and we will release the corresponding training and evaluation code, configuration files, and the scripts that produce our tables and figures upon publication. We use one configuration per variant for all eleven benchmarks (eight text–video and three text–audio retrieval tasks) and do not tune the learning rate, neighbourhood size, or schedule for any individual benchmark. We train all models on the same 150K-clip subset of VAST-27M, of which we could download 136,694 clips with both video and audio.

For GRAM, TRIANGLE and PMRL, we use the checkpoints made available by their authors. Since no checkpoint is available for HYPERGRAM, we train it ourselves with the authors’ public code and their published training recipe. Every evaluated method is initialised from the VAST foundation model and tested on identical test data. Published numbers from other works are reported only as reference values.

Implementation details

Table 8 lists the complete configuration of HyVol, wHyVol, and wRHG, complementing the summary in § 4.2.3. The three configurations share the encoders, optimisation schedule, and data settings, and differ only in how the hypergraph is weighted and propagated: wHyVol and wRHG learn vertex and pair weights, and wRHG additionally uses typed relational messages. The same values are used for both backbones and all eleven benchmarks without benchmark-specific tuning, and the hypergraph is used only during training.

Additional Results and Analysis

Table 9 reports text-to-video and video-to-text R@1 on seven video benchmarks as the share of gallery clips missing one modality grows from 0% to 90%. Recall falls for every method as clips become incomplete, yet the refined models generally remain clearly ahead of their backbones. On VALOR-32K video-to-text retrieval with GRAM, for example, wRHG leads by 9.1, 8.8, and 7.0 points at 25%, 50%, and 75% missingness, and its advantage persists at 6.6 points even when 90% of the clips lack one modality.

DatasetMethodT→V 0%T→V 25%T→V 50%T→V 75%T→V 90%V→T 0%V→T 25%V→T 50%V→T 75%V→T 90%
MSR-VTT [37] (T–VAS)GRAM⋆51.849.244.041.439.550.146.542.438.136.4
MSR-VTT [37] (T–VAS)+HYVOL55.8 (+4.0)52.7 (+3.5)47.2 (+3.2)41.6 (+0.2)39.3 (−0.2)58.1 (+8.0)54.1 (+7.6)48.0 (+5.6)42.6 (+4.5)39.4 (+3.0)
MSR-VTT [37] (T–VAS)+wHYVOL56.8 (+5.0)53.7 (+4.5)48.0 (+4.0)43.9 (+2.5)41.2 (+1.7)58.0 (+7.9)53.6 (+7.1)47.9 (+5.5)43.1 (+5.0)40.7 (+4.3)
MSR-VTT [37] (T–VAS)+wRHG57.4 (+5.6)53.5 (+4.3)48.5 (+4.5)44.4 (+3.0)41.3 (+1.8)58.0 (+7.9)53.6 (+7.1)48.6 (+6.2)43.1 (+5.0)40.2 (+3.8)
MSR-VTT [37] (T–VAS)HYPERGRAM†52.350.245.441.239.250.947.643.339.136.7
MSR-VTT [37] (T–VAS)+HYVOL56.1 (+3.8)52.3 (+2.1)47.2 (+1.8)42.3 (+1.1)39.9 (+0.7)59.2 (+8.3)54.7 (+7.1)47.9 (+4.6)42.6 (+3.5)39.5 (+2.8)
MSR-VTT [37] (T–VAS)+wHYVOL57.0 (+4.7)53.2 (+3.0)48.9 (+3.5)43.3 (+2.1)40.8 (+1.6)58.8 (+7.9)54.3 (+6.7)47.9 (+4.6)42.5 (+3.4)40.2 (+3.5)
MSR-VTT [37] (T–VAS)+wRHG57.3 (+5.0)53.7 (+3.5)48.9 (+3.5)43.8 (+2.6)41.6 (+2.4)59.3 (+8.4)54.3 (+6.7)48.3 (+5.0)43.5 (+4.4)40.9 (+4.2)
DiDeMo [38] (T–VA)GRAM⋆50.645.540.636.633.549.042.536.732.128.6
DiDeMo [38] (T–VA)+HYVOL54.6 (+4.0)47.5 (+2.0)42.4 (+1.8)38.1 (+1.5)35.2 (+1.7)54.1 (+5.1)47.5 (+5.0)41.1 (+4.4)36.1 (+4.0)31.2 (+2.6)
DiDeMo [38] (T–VA)+wHYVOL53.8 (+3.2)47.2 (+1.7)42.0 (+1.4)38.0 (+1.4)34.0 (+0.5)55.5 (+6.5)47.1 (+4.6)40.5 (+3.8)35.1 (+3.0)31.3 (+2.7)
DiDeMo [38] (T–VA)+wRHG54.1 (+3.5)46.9 (+1.4)41.5 (+0.9)36.9 (+0.3)33.8 (+0.3)55.0 (+6.0)47.4 (+4.9)40.8 (+4.1)35.9 (+3.8)31.8 (+3.2)
DiDeMo [38] (T–VA)HYPERGRAM†50.445.039.936.632.850.843.737.633.629.9
DiDeMo [38] (T–VA)+HYVOL54.2 (+3.8)47.8 (+2.8)41.9 (+2.0)38.8 (+2.2)35.1 (+2.3)54.7 (+3.9)48.2 (+4.5)40.5 (+2.9)35.6 (+2.0)31.0 (+1.1)
DiDeMo [38] (T–VA)+wHYVOL54.6 (+4.2)48.1 (+3.1)42.5 (+2.6)37.7 (+1.1)34.3 (+1.5)55.1 (+4.3)46.8 (+3.1)39.9 (+2.3)34.6 (+1.0)30.5 (+0.6)
DiDeMo [38] (T–VA)+wRHG54.6 (+4.2)48.1 (+3.1)42.5 (+2.6)38.6 (+2.0)34.5 (+1.7)55.5 (+4.7)47.2 (+3.5)40.8 (+3.2)34.7 (+1.1)31.0 (+1.1)
ActivityNet [39] (T–VA)GRAM⋆56.450.444.438.134.249.743.938.431.728.2
ActivityNet [39] (T–VA)+HYVOL56.9 (+0.5)49.0 (−1.4)43.5 (−0.9)37.8 (−0.3)34.2 (0.0)56.6 (+6.9)48.5 (+4.6)41.6 (+3.2)32.9 (+1.2)29.0 (+0.8)
ActivityNet [39] (T–VA)+wHYVOL60.8 (+4.4)54.3 (+3.9)47.7 (+3.3)40.9 (+2.8)36.5 (+2.3)62.7 (+13.0)53.8 (+9.9)44.9 (+6.5)36.3 (+4.6)31.2 (+3.0)
ActivityNet [39] (T–VA)+wRHG60.7 (+4.3)54.1 (+3.7)47.9 (+3.5)40.7 (+2.6)36.6 (+2.4)63.0 (+13.3)54.2 (+10.3)45.4 (+7.0)36.7 (+5.0)31.7 (+3.5)
ActivityNet [39] (T–VA)HYPERGRAM†56.850.945.238.935.250.944.639.132.628.9
ActivityNet [39] (T–VA)+HYVOL57.8 (+1.0)48.7 (−2.2)43.7 (−1.5)38.0 (−0.9)34.1 (−1.1)56.4 (+5.5)48.7 (+4.1)41.3 (+2.2)32.6 (0.0)28.3 (−0.6)
ActivityNet [39] (T–VA)+wHYVOL60.5 (+3.7)54.2 (+3.3)48.0 (+2.8)40.9 (+2.0)36.5 (+1.3)62.8 (+11.9)53.7 (+9.1)45.0 (+5.9)36.6 (+4.0)31.4 (+2.5)
ActivityNet [39] (T–VA)+wRHG60.9 (+4.1)54.5 (+3.6)48.3 (+3.1)41.1 (+2.2)36.8 (+1.6)63.0 (+12.1)53.9 (+9.3)45.0 (+5.9)36.4 (+3.8)31.3 (+2.4)
VATEX [40] (T–VAS)GRAM⋆88.480.774.567.764.586.579.173.567.364.0
VATEX [40] (T–VAS)+HYVOL94.0 (+5.6)85.8 (+5.1)78.0 (+3.5)70.8 (+3.1)66.4 (+1.9)93.8 (+7.3)84.9 (+5.8)77.3 (+3.8)69.8 (+2.5)65.2 (+1.2)
VATEX [40] (T–VAS)+wHYVOL93.5 (+5.1)84.9 (+4.2)77.0 (+2.5)70.5 (+2.8)65.7 (+1.2)94.4 (+7.9)86.1 (+7.0)78.4 (+4.9)71.5 (+4.2)67.5 (+3.5)
VATEX [40] (T–VAS)+wRHG94.0 (+5.6)85.6 (+4.9)77.3 (+2.8)71.0 (+3.3)68.4 (+3.9)94.4 (+7.9)86.1 (+7.0)78.4 (+4.9)71.2 (+3.9)67.5 (+3.5)
VATEX [40] (T–VAS)HYPERGRAM†89.881.774.767.363.887.880.574.769.465.9
VATEX [40] (T–VAS)+HYVOL93.3 (+3.5)85.4 (+3.7)78.2 (+3.5)70.8 (+3.5)66.8 (+3.0)93.9 (+6.1)83.5 (+3.0)74.9 (+0.2)68.2 (−1.2)63.3 (−2.6)
VATEX [40] (T–VAS)+wHYVOL93.5 (+3.7)85.8 (+4.1)77.3 (+2.6)68.9 (+1.6)65.2 (+1.4)94.0 (+6.2)84.7 (+4.2)76.8 (+2.1)70.8 (+1.4)65.9 (0.0)
VATEX [40] (T–VAS)+wRHG93.5 (+3.7)85.2 (+3.5)77.0 (+2.3)70.5 (+3.2)67.5 (+3.7)94.0 (+6.2)85.4 (+4.9)77.3 (+2.6)71.7 (+2.3)68.2 (+2.3)
Shot2Story [41] (T–VA)GRAM⋆70.562.454.046.342.266.258.250.742.937.9
Shot2Story [41] (T–VA)+HYVOL74.8 (+4.3)66.4 (+4.0)58.5 (+4.5)50.2 (+3.9)45.2 (+3.0)77.6 (+11.4)67.1 (+8.9)56.9 (+6.2)47.6 (+4.7)41.9 (+4.0)
Shot2Story [41] (T–VA)+wHYVOL76.3 (+5.8)66.7 (+4.3)57.5 (+3.5)49.2 (+2.9)44.4 (+2.2)75.7 (+9.5)66.2 (+8.0)56.3 (+5.6)47.1 (+4.2)41.5 (+3.6)
Shot2Story [41] (T–VA)+wRHG77.4 (+6.9)68.3 (+5.9)59.0 (+5.0)50.5 (+4.2)45.0 (+2.8)76.3 (+10.1)66.8 (+8.6)56.7 (+6.0)47.8 (+4.9)42.0 (+4.1)
Shot2Story [41] (T–VA)HYPERGRAM†75.067.458.550.645.273.165.656.447.742.3
Shot2Story [41] (T–VA)+HYVOL74.6 (−0.4)66.5 (−0.9)58.4 (−0.1)50.4 (−0.2)44.8 (−0.4)78.1 (+5.0)67.4 (+1.8)57.1 (+0.7)48.0 (+0.3)42.2 (−0.1)
Shot2Story [41] (T–VA)+wHYVOL75.9 (+0.9)67.3 (−0.1)58.5 (0.0)50.2 (−0.4)44.9 (−0.3)76.2 (+3.1)66.4 (+0.8)56.5 (+0.1)47.6 (−0.1)41.6 (−0.7)
Shot2Story [41] (T–VA)+wRHG76.6 (+1.6)67.8 (+0.4)59.5 (+1.0)50.7 (+0.1)45.2 (0.0)76.0 (+2.9)66.2 (+0.6)56.5 (+0.1)47.6 (−0.1)41.5 (−0.8)
FLARE [43] (T–VA)GRAM⋆46.041.637.232.030.046.841.635.929.425.9
FLARE [43] (T–VA)+HYVOL52.0 (+6.0)47.6 (+6.0)41.9 (+4.7)36.3 (+4.3)33.4 (+3.4)55.5 (+8.7)48.6 (+7.0)40.6 (+4.7)33.2 (+3.8)28.8 (+2.9)
FLARE [43] (T–VA)+wHYVOL51.7 (+5.7)45.6 (+4.9)39.7 (+2.5)34.9 (+2.9)32.4 (+2.4)54.6 (+7.8)47.7 (+6.1)40.5 (+4.6)33.1 (+3.7)28.9 (+3.0)
FLARE [43] (T–VA)+wRHG53.9 (+7.9)48.3 (+6.7)41.9 (+4.7)36.5 (+4.5)33.8 (+3.8)55.0 (+8.2)48.1 (+6.5)40.7 (+4.8)33.0 (+3.6)29.1 (+3.2)
FLARE [43] (T–VA)HYPERGRAM†52.949.743.938.233.652.648.341.434.428.6
FLARE [43] (T–VA)+HYVOL52.1 (−0.8)47.6 (−2.1)42.4 (−1.5)36.7 (−1.5)34.1 (+0.5)56.4 (+3.8)48.9 (+1.5)41.5 (+0.1)33.7 (−0.7)29.4 (+0.8)
FLARE [43] (T–VA)+wHYVOL52.1 (−0.8)47.4 (−2.3)41.4 (−2.5)36.2 (−2.0)33.7 (+0.1)55.9 (+3.3)48.7 (+0.4)41.1 (−0.3)33.2 (−1.2)29.2 (+0.6)
FLARE [43] (T–VA)+wRHG52.8 (−0.1)47.9 (−1.8)42.7 (−1.2)37.3 (−0.9)34.1 (+0.5)55.8 (+3.2)48.7 (+0.4)40.8 (−0.6)33.1 (−1.3)29.1 (+0.5)
VALOR-32K [5] (T–VA)GRAM⋆75.265.656.147.241.971.263.054.045.440.4
VALOR-32K [5] (T–VA)+HYVOL80.5 (+5.3)71.2 (+5.6)62.2 (+6.1)53.3 (+6.1)47.4 (+5.5)81.6 (+10.4)72.0 (+9.0)62.0 (+8.0)52.0 (+6.6)46.6 (+6.2)
VALOR-32K [5] (T–VA)+wHYVOL80.4 (+5.2)70.9 (+5.3)61.2 (+5.1)51.8 (+4.6)47.5 (+5.6)81.1 (+9.9)71.3 (+8.3)62.2 (+8.2)51.6 (+6.2)46.4 (+6.0)
VALOR-32K [5] (T–VA)+wRHG80.8 (+5.6)71.4 (+5.8)61.7 (+5.6)53.3 (+6.1)48.0 (+6.1)80.8 (+9.6)72.1 (+9.1)62.8 (+8.8)52.4 (+7.0)47.0 (+6.6)
VALOR-32K [5] (T–VA)HYPERGRAM†75.967.257.648.243.572.465.156.247.342.1
VALOR-32K [5] (T–VA)+HYVOL80.9 (+5.0)71.3 (+4.1)62.5 (+4.9)53.5 (+5.3)46.7 (+4.1)82.2 (+9.8)72.5 (+7.4)62.5 (+6.3)52.5 (+5.3)46.9 (+4.8)
VALOR-32K [5] (T–VA)+wHYVOL80.5 (+4.6)71.0 (+3.8)60.9 (+3.3)51.5 (+3.3)45.5 (+2.0)82.0 (+9.6)72.2 (+7.1)62.6 (+6.4)52.1 (+4.8)46.6 (+4.5)
VALOR-32K [5] (T–VA)+wRHG80.9 (+5.0)71.5 (+4.3)62.2 (+4.6)52.6 (+4.4)46.7 (+3.2)81.8 (+9.4)72.4 (+7.3)62.9 (+6.7)52.5 (+5.2)47.1 (+5.0)

Table 9. Retrieval under missing modalities (R@1) on the seven video benchmarks. A fraction rr of the gallery clips loses one randomly chosen modality; the per-clip masks are fixed and identical across methods, and r=0r=0 reproduces the main protocol. MSR-VTT and VATEX use the T–VAS modality set, and the other benchmarks use T–VA. Numbers in parentheses give the change with respect to the backbone (GRAM or HYPERGRAM); green marks an improvement, red a drop, and gray no change.

References

  1. [1]A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, in: International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, 2021, pp. 8748–8763.arxiv.org/abs/2103.00020
  2. [2]R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, I. Misra, ImageBind: One embedding space to bind them all, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 15180–15190.arxiv.org/abs/2305.05665
  3. [3]B. Zhu, B. Lin, M. Ning, Y. Yan, J. Cui, H. Wang, Y. Pang, W. Jiang, J. Zhang, Z. Li, et al., LanguageBind: Extending video-language pretraining to n-modality by language-based semantic alignment, in: International Conference on Learning Representations, 2024.arxiv.org/abs/2310.01852
  4. [4]S. Chen, H. Li, Q. Wang, Z. Zhao, M. Sun, X. Zhu, J. Liu, VAST: A vision-audio-subtitle-text omni-modality foundation model and dataset, Advances in Neural Information Processing Systems 36 (2023) 72842–72866.arxiv.org/abs/2305.18500
  5. [5]J. Liu, S. Chen, X. He, L. Guo, X. Zhu, W. Wang, J. Tang, VALOR: Vision-audio-language omni-perception pre-training model and dataset, IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (2025) 708–724.
  6. [6]G. Cicchetti, E. Grassucci, L. Sigillo, D. Comminello, Gramian multimodal representation learning and alignment, in: International Conference on Learning Representations, 2025.arxiv.org/abs/2412.11959
  7. [7]G. Cicchetti, E. Grassucci, D. Comminiello, A TRI-ANGLE enables multimodal alignment beyond cosine similarity, Advances in Neural Information Processing Systems 38 (2025) 124144–124162.DOI
  8. [8]X. Liu, X. Xia, S.-K. Ng, T.-S. Chua, Principled multimodal representation learning, IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (2026) 9114–9128.DOI
  9. [9]S. Na, F. Jiang, Q. Zhou, W. Zhong, T. M. Dang, Y. Guo, H. Ma, C. Li, W. An, J. Huang, Hyperbolic Gramian volumes for multimodal alignment, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 37756–37765.
  10. [10]M. Ma, J. Ren, L. Zhao, S. Tulyakov, C. Wu, X. Peng, SMIL: Multimodal learning with severely missing modality, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021, pp. 2302–2310.arxiv.org/abs/2103.05677
  11. [11]M. Ma, J. Ren, L. Zhao, D. Testuggine, X. Peng, Are multimodal transformers robust to missing modality?, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2022, pp. 18156–18165.arxiv.org/abs/2204.05454
  12. [12]A. Nag, A. Mehrish, S. Vascon, Hypergraph-regularized Gramian volumes for multimodal retrieval, in: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2027. ArXiv:2609.15320.arxiv.org/abs/2609.15320
  13. [13]S. Brody, U. Alon, E. Yahav, How attentive are graph attention networks?, in: International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2105.14491
  14. [14]Q. Sun, Y. Fang, L. Wu, X. Wang, Y. Cao, EVA-CLIP: Improved training techniques for CLIP at scale, ArXiv preprint: arXiv:2303.15389 (2023).arxiv.org/abs/2303.15389
  15. [15]Z. Zeng, Y. Ge, Z. Tong, X. Liu, S.-T. Xia, Y. Shan, TVSTv2: Learning out-of-the-box spatiotemporal visual representations at scale, ArXiv preprint: arXiv:2305.14173 (2023).arxiv.org/abs/2305.14173
  16. [16]Q. Ye, G. Xu, M. Yan, H. Xu, Q. Qian, J. Zhang, F. Huang, HiTeA: Hierarchical temporal-aware video-language pre-training, in: IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 15359–15370.arxiv.org/abs/2212.14546
  17. [17]Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang, S. Xing, G. Chen, J. Pan, J. Yu, Y. Wang, L. Wang, Y. Qiao, InternVideo: General video foundation models via generative and discriminative learning, ArXiv preprint: arXiv:2212.03191 (2022).arxiv.org/abs/2212.03191
  18. [18]H. Xu, Q. Ye, M. Yan, Y. Shi, J. Ye, Y. Xu, C. Li, B. Bi, Q. Qian, W. Wang, et al., mPLUG-2: A modularized multi-modal foundation model across text, image and video, in: International Conference on Machine Learning, PMLR, 2023, pp. 38728–38748.arxiv.org/abs/2302.00402
  19. [19]J. Wang, D. Chen, Z. Wu, C. Luo, L. Zhou, Y. Zhao, Y. Xie, C. Liu, Y.-G. Jiang, L. Yuan, OmniVL: One foundation model for image-language and video-language tasks, in: Advances in Neural Information Processing Systems, volume 35, 2022, pp. 5696–5710.arxiv.org/abs/2209.07526
  20. [20]L. Zhao, N. B. Gundavarapu, L. Yuan, H. Zhou, S. Yan, J. J. Sun, L. Friedman, R. Qian, T. Weyand, Y. Zhao, R. Hornung, F. Schroff, M.-H. Yang, D. A. Ross, H. Wang, H. Adam, M. Sirotenko, T. Liu, B. Gong, VideoPrism: A foundational visual encoder for video understanding, in: International Conference on Machine Learning, 2024, pp. 60785–60811.arxiv.org/abs/2402.13217
  21. [21]A. Nagrani, P. H. Seo, B. Seybold, A. Hauth, S. Manén, C. Sun, C. Schmid, Learning audio-video modalities from image captions, in: European Conference on Computer Vision, 2022, pp. 407–426.arxiv.org/abs/2204.00679
  22. [22]Y. Zhao, J. Hessel, Y. Yu, X. Lu, R. Zellers, Y. Choi, Connecting the dots between audio and text without parallel data through visual knowledge transfer, in: Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2022, pp. 4492–4507.arxiv.org/abs/2112.08995
  23. [23]J. Li, R. R. Selvaraju, A. D. Gotmare, S. R. Joty, C. Xiong, S. C. H. Hoi, Align before fuse: Vision and language representation learning with momentum distillation, in: Advances in Neural Information Processing Systems, volume 34, 2021, pp. 9694–9705.
  24. [24]Y. Feng, H. You, Z. Zhang, R. Ji, Y. Gao, Hypergraph neural networks, in: AAAI Conference on Artificial Intelligence, 2019, pp. 3558–3565.arxiv.org/abs/1809.09401
  25. [25]E. Chien, C. Pan, J. Peng, O. Milenkovic, You are AllSet: A multiset function framework for hypergraph neural networks, in: International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2106.13264
  26. [26]M. Li, S. Zhou, Y. Chen, C. Huang, Y. Jiang, EduCross: Dual adversarial bipartite hypergraph learning for cross-modal retrieval in multimodal educational slides, Information Fusion 109 (2024) 102428. doi:10.1016/j.inffus.2024.102428.DOI
  27. [27]P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio, Graph attention networks, in: International Conference on Learning Representations (ICLR), 2018.arxiv.org/abs/1710.10903
  28. [28]Z. Zhong, L. Zheng, D. Cao, S. Li, Re-ranking person re-identification with k-reciprocal encoding, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1318–1327.arxiv.org/abs/1701.08398
  29. [29]S.-V. Bogolin, I. Croitoru, H. Jin, Y. Liu, S. Albanie, Cross modal retrieval with querybank normalisation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5194–5205.arxiv.org/abs/2112.12777
  30. [30]M. Schlichtkrull, T. N. Kipf, P. Bloem, R. van den Berg, I. Titov, M. Welling, Modeling relational data with graph convolutional networks, in: The Semantic Web (ESWC), 2018, pp. 593–607.arxiv.org/abs/1703.06103
  31. [31]B. Fatemi, P. Taslakian, D. Vazquez, D. Poole, Knowledge hypergraphs: Prediction beyond binary relations, in: International Joint Conference on Artificial Intelligence (IJCAI), 2020, pp. 2191–2197.
  32. [32]M. Galkin, P. Trivedi, G. Maheshwari, R. Usbeck, J. Lehmann, Message passing for hyper-relational knowledge graphs, in: Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 7346–7359.arxiv.org/abs/2009.10847
  33. [33]N. Yadati, Neural message passing for multi-relational ordered and recursive hypergraphs, in: Advances in Neural Information Processing Systems (NeurIPS), 2020.
  34. [34]X. Huang, M. Romero Orth, P. Barceló, M. M. Bronstein, I. i. Ceylan, Link prediction with relational hypergraphs, arXiv preprint arXiv:2402.04062 (2024).arxiv.org/abs/2402.04062
  35. [35]J. Hu, Y. Liu, J. Zhao, Q. Jin, MMGCN: Multimodal fusion via deep graph convolution network for emotion recognition in conversation, in: Annual Meeting of the Association for Computational Linguistics (ACL), 2021, pp. 5666–5675.arxiv.org/abs/2107.06779
  36. [36]J. Yang, Y. Wang, R. Yi, Y. Zhu, A. Rehman, A. Zadeh, S. Poria, L.-P. Morency, MTAG: Modal-temporal attention graph for unaligned human multimodal language sequences, in: Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021, pp. 1009–1021.arxiv.org/abs/2010.11985
  37. [37]J. Xu, T. Mei, T. Yao, Y. Rui, MSR-VTT: A large video description dataset for bridging video and language, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 5288–5296.
  38. [38]L. A. Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, B. Russell, Localizing moments in video with natural language, in: IEEE International Conference on Computer Vision (ICCV), 2017, pp. 5804–5813.arxiv.org/abs/1708.01641
  39. [39]F. Caba Heilbron, V. Escorcia, B. Ghanem, J. Carlos Niebles, ActivityNet: A large-scale video benchmark for human activity understanding, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 961–970.DOI
  40. [40]X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, W. Y. Wang, VaTeX: A large-scale, high-quality multilingual dataset for video-and-language research, in: IEEE/CVF International Conference on Computer Vision, 2019, pp. 4581–4591.
  41. [41]M. Han, L. Yang, X. Chang, H. Wang, Shot2Story20K: A new benchmark for comprehensive understanding of multi-shot videos, arXiv preprint arXiv:2312.10300 (2023).
  42. [42]D. L. Chen, W. B. Dolan, Collecting highly parallel data for paraphrase evaluation, in: Annual Meeting of the Association for Computational Linguistics (ACL), 2011, pp. 190–200.
  43. [43]Q. You, H. Liang, M. Chen, B. Zeng, M. Qiang, Z. Wong, W. Zhang, FLARE: Full-modality long-video audiovisual retrieval benchmark with user-simulated queries, arXiv preprint arXiv:2605.10228 (2026).arxiv.org/abs/2605.10228
  44. [44]C. D. Kim, B. Kim, H. Lee, G. Kim, AudioCaps: Generating Captions for Audios in The Wild, in: NAACL-HLT, 2019, pp. 119–132.DOI
  45. [45]K. Drossos, S. Lipping, T. Virtanen, Clotho: An audio captioning dataset, in: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 736–740.arxiv.org/abs/1910.09387
  46. [46]A. S. Koepke, A.-M. Oncescu, J. F. Henriques, Z. Akata, S. Albanie, Audio retrieval with natural language queries: A benchmark study, IEEE Transactions on Multimedia 25 (2023) 2675–2685.
  47. [47]Y. Lin, J. Zhang, Z. Huang, J. Liu, Z. Wen, X. Peng, Multi-granularity correspondence learning from long-term noisy videos, in: International Conference on Learning Representations, 2024.arxiv.org/abs/2401.16702
  48. [48]K. Li, Y. Wang, Y. Li, Y. Wang, Y. He, L. Wang, Y. Qiao, Unmasked teacher: Towards training-efficient video foundation models, in: IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 19891–19903.arxiv.org/abs/2303.16058
  49. [49]Y. Wang, Y. He, Y. Li, K. Li, J. Yu, X. Ma, X. Li, G. Chen, X. Chen, Y. Wang, P. Luo, Z. Liu, Y. Wang, L. Wang, Y. Qiao, InterVideo: A large-scale video-text dataset for multimodal understanding and generation, in: International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2307.06942
  50. [50]S. Yan, T. Zhu, Z. Wang, Y. Cao, M. Zhang, S. Ghosh, Y. Wu, J. Yu, VideoCoCa: Video-text modeling with zero-shot transfer from contrastive captioners, 2022. ArXiv preprint: arXiv:2212.04979.arxiv.org/abs/2212.04979

Paper details

vlnc:W_jprd2vct
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 13

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

large language models

View statement
Looking for related papers…
Contents