WorksComputer Science
Fuse To Align: Multimodal Representation Learning for Retrieval via Hypergraph Regularization
Abstract
Multimodal retrieval must integrate complementary evidence from views that may be missing, noisy, or unequally informative. When a sample provides limited reliable evidence, learning from its views alone can hinder the acquisition of useful retrieval representations. Semantically related training samples may provide complementary observations, yet existing volume-based objectives jointly align modalities within each sample without explicitly enabling information exchange across samples. We therefore propose a training-time hypergraph fusion framework that connects observed modalities both within samples and across semantically related samples, allowing representation learning to draw on evidence beyond an individual sample while leaving the retrieval pipeline unchanged at inference. The framework includes HyVol as a special case and introduces two configurations: wHyVol learns view contribution weights and symmetric semantic affinities, whereas wRHG additionally models typed interactions and participant roles through relational propagation. Missing modalities are omitted rather than imputed, and the hypergraph is discarded after training, incurring no additional inference cost. Experiments on eleven zero-shot video-text and audio-text retrieval benchmarks demonstrate broad improvements with both GRAM and HyperGRAM. Across 17 video benchmark-modality settings evaluated in both retrieval directions, wRHG outperforms GRAM in all 34 comparisons and HyperGRAM in 33 of 34. Mean text-to-video R@1 increases from 63.4 to 68.2 with GRAM and from 64.8 to 68.4 with HyperGRAM, while the largest video-to-text gain reaches 13.3 percentage points on ActivityNet. Under missing-modality evaluation, both weighted configurations retain positive text-to-video gains over GRAM across seven benchmarks at every evaluated missingness rate, even when 90% of clips lack one modality.
Introduction
Humans interpret the world by integrating information across multiple sensory modalities. For example, visual input can influence auditory perception, while language provides contextual cues that help unify sensory experiences. Psychologists refer to this process as multisensory integration, through which individuals reconstruct events from both overlapping and modality-specific sensory cues. Multimodal representation learning presents an analogous challenge because images, sounds, speech, and text capture complementary aspects of the same scenario. Effective representation learning must therefore identify shared features across modalities while preserving the distinctive information provided by each modality.
Text-to-video retrieval requires a model to identify, among thousands of clips, the clip described by a given caption. Each clip provides multiple views of the same event, including frames, a soundtrack, and subtitles, but real-world collections rarely contain all views at consistently high quality. Consider two clips described as “a man explains how to tune a guitar”. In the first clip, the frames depict the guitar, and the subtitles transcribe the explanation. In the second, the subtitles are absent, and the soundtrack contains only background music, leaving the frames as the sole reliable source of evidence. Such missing, noisy, and unequally informative views are common, requiring a retrieval model to integrate them into a single relevance score.
Current multimodal retrieval models evaluate each clip independently. Contrastive foundation models such as CLIP [1] measure the cosine similarity between two embeddings, while their multi-view extensions either align each view with an anchor modality or fuse multiple views into a single representation [2, 3, 4, 5]. Geometric objectives extend this approach by jointly scoring all views of a clip using the volume spanned with the caption in GRAM [6], the area of a triangle in TRI-ANGLE [7], the dominant singular value in PMRL [8], or a combination of Euclidean and hyperbolic volumes in HYPERGRAM [9]. Nevertheless, in all these methods, a clip’s score depends exclusively on the caption and the clip’s own views. Consequently, in the example above, the second clip must be matched using only one reliable view, while the first clip, despite depicting the same event, is treated solely as a negative during training [10, 11].
Our central premise is that fusion should exploit complementary evidence both within a sample and across semantically related training samples. When a clip contains missing or uninformative views, restricting fusion to its own observations limits the evidence available for representation learning. Related clips may contain useful observations of similar events, providing an additional source of supervision through information exchange. This motivates a fusion mechanism that jointly models cross-modal dependencies within clips and semantic associations across clips.
We propose a training-time hypergraph feature fusion framework that captures both forms of interaction. Each observed candidate modality forms a vertex, while hyperedges connect modalities belonging to the same candidate and observations from semantically related candidates. Feature propagation along these hyperedges integrates contextual evidence into the modality-specific representations, which remain separate inputs to the retrieval objective. Learned contribution weights regulate this fusion, while relational propagation distinguishes interaction types and participant roles. Missing views remain absent rather than being reconstructed. The hypergraph operates only during training, allowing the encoders to learn from fused evidence while preserving independent candidate encoding and the original retrieval procedure at inference. Figure 1 contrasts this design with existing methods, which score each clip alone, and shows that the hypergraph is removed at test time.
Figure 1. Overview of the proposed framework. (a) Existing methods score each clip alone: a clip’s score depends only on the caption and the clip’s own views, and every other clip in the batch, even a related one, is pushed apart as a negative. (b) Our training-time hypergraph: observed views form vertices, document hyperedges group the views of each clip, and caption-guided semantic hyperedges connect related clips through same-modality and cross-modality links; missing views have no vertex and are not imputed, and each vertex receives a learned reliability weight. (c) At test time the hypergraph is removed: each clip is encoded and scored independently with the original backbone score, so the inference cost equals that of the backbone.
Our framework differs from conventional fusion strategies in three respects. First, it integrates evidence both within individual clips and across semantically related training clips. Second, unlike approaches that reconstruct missing views [10], it operates only on observed modalities, enabling information exchange without imputation. Third, fusion is confined to training: the learned encoders retain their benefits, while inference requires neither a hypergraph nor interactions among evaluation candidates. The framework encompasses HyVol [12] as a special case, where document and semantic neighborhood hyperedges support additive feature fusion. Our weighted and relational configurations investigate how to regulate individual-view contributions and related-sample contributions, and how different interactions should participate in the fusion.
HyVol does not explicitly model differences in view reliability or distinguish modality-specific interaction types. A silent soundtrack receives the same weight as the frames of the corresponding clip; related clips are weighted by an attention score that ranks the neighbors of every clip in the same order [13]; and a link between the frames of two clips is not distinguished from a link between the frames of one clip and the subtitles of another. Moreover, only the refined embeddings are trained, although retrieval subsequently uses the unrefined embeddings. Information exchange across clips therefore raises two questions that remain unresolved in HyVol: how much should each view and each related clip contribute, and what type of relationship connects them?
Weighted HyVol (wHyVol) addresses the first question through two learned weights. The first is a view-specific reliability weight predicted solely from the corresponding view embedding. In the guitar example, the background music in the second clip may receive a low weight, while its frames receive a high weight, causing the clip representation to be determined primarily by reliable evidence. The reliability weight therefore provides an adaptive fusion mechanism for imbalanced views: each view’s contribution depends on its estimated quality rather than its modality. The second weight represents the affinity between two related clips; it is computed jointly from their captions and is symmetric. Consequently, a clip receives substantial information from a close semantic match but little information from a weak or incidental match. Unlike the attention mechanism in HyVol, this affinity can induce different neighbour rankings for different clips.
The weighted relational hypergraph (wRHG) addresses the second question. It retains both weights introduced by wHyVol but replaces untyped links with typed relations. A same-modality link connects corresponding views from two related clips, such as their frames; a cross-modality link connects different views, such as the frames of one clip and the subtitles of another; and a whole-clip link connects all observed views from both clips. Each relation type has distinct learned parameters, and the message received by a view through a link depends jointly on all other views associated with that link. Thus, a clip with missing subtitles can learn from a related clip’s subtitles through a cross-modality link, while the model learns the relative importance of each relation type.
In wHyVol and wRHG, a missing view has no corresponding vertex in the hypergraph; consequently, neither model imputes nor replaces missing views. These configurations also train the unrefined embeddings used at test time. As with HyVol, their hypergraph modules operate exclusively during training. We integrate HyVol, wHyVol, and wRHG with two volume-based backbones, GRAM and HYPERGRAM, which share the pretrained vision–audio–language foundation encoders (VAST [4]). We evaluate all configurations zero-shot on eleven text–video and text–audio retrieval benchmarks. Averaged across the 17 text-to-video settings from the eight video benchmarks, the R@1 of GRAM increases from 63.4 to 68.2 with wRHG (and to 67.3 with HyVol), while that of HYPERGRAM increases from 64.8 to 68.4. The gains are consistent across datasets: wRHG improves the R@1 of GRAM in all 17 text-to-video and all 17 video-to-text settings, and improves that of HYPERGRAM in 16 and 17 settings, respectively. The weighted variants also retain gains under missing-modality evaluation, with consistent improvements over GRAM and benchmark-dependent results with HYPERGRAM. Our main contributions are as follows.
We formulate multi-view feature fusion within and across semantically related samples through a common training-time hypergraph framework, with HyVol recovered as a special case.
We introduce two new instances of this framework: a weighted hypergraph (wHyVol), which learns the contribution of each view and related clip, and a weighted relational hypergraph (wRHG), which models distinct interaction types through relation-specific parameters and participant roles. Both methods handle missing views without imputation and incur no additional inference cost.
We evaluate the proposed methods using two backbones and eleven zero-shot benchmarks—eight for text–video retrieval and three for text–audio retrieval—and conduct a missing-modality study across seven video benchmarks.
Building on HyVol [12], this article introduces a general framework together with wHyVol and wRHG, expands the evaluation from six to eleven benchmarks, and extends the missing-modality study from four to seven video benchmarks. The remainder of this article is organized as follows. § 2 reviews the related literature. § 3 presents the general hypergraph framework, recovers HyVol as a special case, and introduces wHyVol and wRHG. § 4 describes the datasets and evaluation protocol and reports the zero-shot and missing-modality results. § 6 discusses these results, § 7 outlines the limitation of the approach, and § 8 concludes the article.
Related Work
Our research builds upon multimodal representation learning, higher-order geometric alignment, and hypergraph-based information fusion. We examine these foundational concepts alongside methods that address missing and imbalanced modalities, inter-sample structural relationships in retrieval tasks, and relation-aware message passing.
Contrastive Vision–Language and Multimodal Foundations
CLIP [1] established contrastive pretraining on image–text pairs, whereby representations of matched pairs are brought closer together while all other pairs in the batch serve as negatives. Subsequent work developed larger and openly available backbones [14]. More recent approaches have developed large-scale video–language models pretrained on extensive video corpora [15, 16, 17, 18, 19, 20]. Contrastive learning has also been applied to audio–text retrieval [21, 22] and to text–video retrieval through multigranular alignment and text-conditioned objectives.
When more than two modalities are available, existing methods typically either fuse streams into a single representation or align them within a shared embedding space. VALOR [5] fuses video and audio, while VAST [4] additionally incorporates subtitles. ImageBind [2] and LanguageBind [3] instead align modalities using images and language as anchors, respectively. Alignment may also precede explicit cross-modal fusion [23]. These approaches do not directly score joint agreement among a candidate’s individual streams. Our models retain the modality-specific VAST embeddings and evaluate their joint agreement with the query.
Higher-Order Geometric Alignment
Recent methods assess multimodal agreement through joint geometric objectives. GRAM [6] uses the Gramian volume spanned by the query and candidate modality embeddings, while TRIANGLE [7] measures the triangle area formed by text, video, and audio embeddings. PMRL [8] promotes rank-one alignment by maximising the dominant singular value of the embedding matrix, and HYPERGRAM [9] combines Euclidean and hyperbolic volumes. These methods score each candidate using only its own embeddings. HYVOL [12] introduces information exchange across candidates through a training-time hypergraph while retaining the GRAM and HYPERGRAM scoring functions. Our framework includes HYVOL as a special case and investigates learned contribution weights, semantic affinities, and typed relational propagation.
Learning with Missing and Imbalanced Modalities
Real-world collections often contain missing modalities and streams that differ in informativeness. Existing methods address missing views through reconstruction [10] or fusion mechanisms designed to accommodate absent inputs [11]. Our framework instead excludes missing modalities from the hypergraph and the Gram matrix, so retrieval depends only on observed streams. Learned vertex weights predicted from individual modality embeddings regulate their contributions during training-time fusion.
Hypergraphs and Inter-Sample Structure in Retrieval
Hypergraphs represent higher-order relations by connecting multiple vertices within a single hyperedge. Hypergraph neural networks exchange features between vertices and hyperedges [24], while AllSet [25] learns set-based aggregation functions. Applications include cross-modal retrieval through bipartite hypergraphs [26]. Attention mechanisms learn interaction weights [27], although the original additive attention score has a static neighbor-ranking property [13], motivating the pair-dependent affinity kernel in wHyVol. In retrieval, inter-item structure is also exploited through test-time methods such as k-reciprocal re-ranking [28] and querybank normalisation [29]. By contrast, HyVol [12] and our configurations exchange information only during training, allowing independent candidate encoding without additional inference cost.
Relational and Knowledge Hypergraphs
Relational graph convolutional networks [30] assign distinct transformations to different edge types. Knowledge hypergraphs extend this principle to facts involving more than two arguments, where the position of each entity carries semantic information. HypE [31] embeds entities using position-specific filters; StarE [32] represents hyper-relational facts with qualifiers; and G-MPNN [33] performs message passing over multi-relational ordered hypergraphs. HC-MPNN [34] combines relation embeddings with positional encodings and constructs the message to each member of a fact from all remaining members. Its expressive power for link prediction has been characterised for relations of arbitrary arity.
Graphs with multiple edge types have also been applied to multimodal fusion. For conversational emotion recognition, MMGCN [35] connects utterances through modality-specific and cross-modal edges. For unaligned multimodal language sequences, MTAG [36] constructs a graph whose edges depend on node modalities and temporal order. In each setting, the data determine the edges or facts, such as a knowledge base, conversational order, or temporal order. In WRHG, by contrast, facts are constructed within each training batch according to caption similarity, and each fact’s relation type identifies the modalities and candidates that it connects.
Relational Hypergraph Learning
Multimodal representation learning must capture dependencies at two complementary levels: cross-modal associations within each candidate and semantic correspondences across candidates. The former integrates complementary evidence from the modalities observed for a candidate, while the latter lets its representation incorporate relevant information from semantically related candidates in the same training batch. We model both forms of dependency using a weighted hypergraph in which each vertex represents an observed candidate–modality pair. Hyperedge construction determines which vertices interact, learned weights control their contributions, and the propagation operator specifies how the resulting information is aggregated.
The framework supports two propagation mechanisms: weighted additive aggregation and relation-conditioned multiplicative interaction. Both mechanisms refine candidate embeddings under the original retrieval objective without imposing a specific similarity function or representation geometry. HYVOL [12] is recovered as a special case by choosing a specific hyperedge construction, weighting scheme, and additive propagation operator. The hypergraph is used only during training; at inference, the model retains the trained encoders and follows the original retrieval procedure without constructing or propagating the hypergraph. Figure 2 summarises the framework.
Figure 2. Overview of training-time hypergraph refinement. Each observed modality becomes a weighted vertex, and stop-gradient captions connect semantically related candidates through mutual top- neighbors. Information is then exchanged either by weighted additive propagation (wHyVol) or by typed relational messages (wRHG), and the refined embeddings are scored by the unchanged GRAM/HYPERGRAM objective. The hypergraph is used only during training and is removed at inference.
Problem Formulation
Let a mini-batch comprise matched query–item pairs , where is a caption query and is its matched retrieval item. We use candidate to denote any retrieval item evaluated as a potential match for a query. Each candidate comprises observations from a nonempty subset of the supported modalities. The modality set may include visual, audio, subtitle, or other available streams, and denotes the number of supported modalities. Subtitles are candidate-side observations and are therefore distinct from the caption used as the retrieval query.
A query encoder and modality-specific encoders produce -dimensional representations in a common embedding space. Each encoder ends with a learned linear projection to this shared space. Although these encoders are parameterised separately, the retrieval objective aligns their outputs for cross-modal comparison. The resulting normalised embeddings are and . Here, , where is a numerical-stability constant. When , this operation produces a unit-norm embedding, and inner products between the resulting embeddings correspond to cosine similarities.
Using these embeddings, we represent each observed candidate–modality pair as a vertex: . Accordingly, the number of vertices is . This construction preserves modality identity while allowing different observations of the same candidate to participate in distinct interactions. Because vertices are instantiated only for observed modalities, missing modalities require neither imputation nor masking; they are absent from both propagation and pooling.
Query embeddings are not propagated as vertex features. Instead, stop-gradient query embeddings are used only to define semantic associations among candidates. This design allows caption semantics to guide the construction of the training-time hypergraph without injecting query information directly into candidate representations. Consequently, each candidate can be encoded independently of the test query at inference.
Semantic Connectivity and Learned Weights
Cross-candidate interactions are restricted to semantically related training instances. Let denote the stop-gradient caption embedding, where is the stop-gradient operator. For candidate , let contain the indices of other candidates with the most similar caption embeddings under the inner product . We retain an association only when this neighbourhood relation is reciprocal:
Here, is the indicator function, is an optional similarity filter, and is a symmetric training-time dropout mask. In the absence of filtering or dropout, the corresponding factors are set to one. The reciprocity requirement eliminates one-sided associations supported by only one neighbourhood ranking.
In the implementation, , where is the requested neighbourhood size, and semantic connections are omitted when . The optional filter retains similarities above , where and are the mean and standard deviation of the off-diagonal caption similarities and controls selectivity. These choices determine the candidate associations available to all relation families.
Connectivity and contribution strength are treated separately. A learned projection determines the weight of each retained candidate pair, while a modality-dependent predictor estimates the contribution of an individual vertex.
The parameters are , , , and . The connection weights satisfy and lie in in exact arithmetic, while . Unlike a neighbourhood softmax, the exponential kernel does not require the outgoing weights of a candidate to sum to one. It therefore preserves an absolute measure of affinity within the learned caption space.
We initialise to the identity, to , and and to zero. Consequently, all observed vertices initially have unit weight. The weighting parameters receive gradients from the training objective, whereas the detached caption inputs do not transmit these gradients to the query encoder. Vertex weights remain differentiable with respect to their candidate features.
A Common Hypergraph Propagation Formulation
A hyperedge is represented by an indexed interaction . Its participating vertices form , identifies its relation type, is a scalar interaction weight, and assigns a participant role to vertex . Different indexed interactions may share the same vertex set while carrying different relation labels or role assignments.
Let denote the hyperedges incident to , and let be its representation at layer . We express refinement through a message function , an aggregation operator , and an update function :
The learned scalar modulates the sign and strength of the residual correction. Hyperedge weights, memberships, relation labels, and participant roles enter through the message and aggregation functions.
This formulation separates the interaction structure from the mechanism that combines its features. Additive propagation constructs weighted hyperedge summaries and redistributes them to their members. Relational propagation instead constructs recipient-specific messages whose components depend jointly on the other participants. Both are instances of Eq. (3).
Weighted Additive Hypergraph
The additive configuration uses document and semantic neighbourhood hyperedges. A document hyperedge connects all observed modalities of one candidate. A semantic hyperedge connects the observed modalities of an anchor candidate and those of its retained semantic neighbours. With rows indexed by observed vertices, the corresponding incidence matrices are
Document membership reflects learned modality importance. Semantic membership assigns unit weight to each observed anchor modality and a shared connection weight to each observed modality of a neighbour. Consequently, a neighbour with more observed modalities contributes more total membership mass; subsequent degree normalisation operates on this total mass.
The semantic block is omitted when the batch contains no retained semantic connections. Otherwise, one semantic hyperedge is included for every candidate, including anchor-only hyperedges for candidates without neighbours. This convention is relevant because repeated or overlapping hyperedges affect the normalisation of incident messages.
Let stack the encoder embeddings as rows. Define and , where is an all-ones vector of the appropriate dimension. The additive layer is
Here, are learned projections, and is GELU in intermediate layers and the identity in the final layer. Degree entries are floored at before inversion.
The first aggregation computes a weighted average of the vertices in each hyperedge. The second combines the resulting hyperedge representations at each vertex. The residual path preserves the incoming feature while allowing within-candidate and cross-candidate context to modify it. Hyperedge types affect connectivity, but this operator shares its feature transformations across types.
Relational Hypergraph
The relational configuration makes the interaction type explicit through typed hyperedge families. Within a candidate, a single document hyperedge connects all of its observed modalities and therefore already covers every pair of them, so no separate within-candidate pairwise hyperedges are formed.
Cross-candidate hyperedges are constructed for pairs with . Same-modality hyperedges connect matching modality types across the two candidates. Cross-modality hyperedges connect different modality types; for each unordered modality pair, both assignments to the two candidates are included when the corresponding observations are available. A joint-candidate hyperedge contains all observed modalities from both candidates. Within-candidate hyperedges have weight , and cross-candidate hyperedges have weight . Hyperedges with fewer than two observed vertices are discarded.
The role map identifies modality type. For joint-candidate hyperedges, it additionally identifies which of the two candidates supplies the observation. These two candidate positions use disjoint role indices. Both candidate orders are instantiated, ensuring that the construction does not privilege either endpoint of the undirected semantic association.
For relational propagation, the initial state is . Let and denote learned role and relation embeddings. The contribution of participant and the aggregated message to recipient are
The symbol denotes elementwise multiplication. The learned scalar is initialised to 0.5 and is unconstrained, so the feature–role combination is affine rather than necessarily convex. Vertex weights are computed from the encoder outputs and reused across layers.
Excluding the recipient from the product makes each message depend on the other participants in the interaction. For two-vertex hyperedges, the product contains one factor. For larger hyperedges, it jointly combines several participants, allowing the message to depend on their coordinate-wise interactions. Relation embeddings modulate these products by interaction type, while role embeddings distinguish contributions across modalities and candidate positions.
Messages are summed over incident hyperedges without degree normalisation and are incorporated through
where denotes concatenation, , , and is layer normalisation. The activation is ReLU in intermediate layers and the identity in the final layer.
Concatenating the current feature with its aggregated message allows the update to depend on both local information and relational context. An observed vertex without incident hyperedges receives a zero message but may still undergo this local transformation. When the entire relational hypergraph is empty, the implementation bypasses refinement. We use at most two propagation layers.
Retrieval Supervision and Inference
After layers, let denote the final unnormalised modality representation. For additive propagation, it is the corresponding row of ; for relational propagation, it is . The module produces normalised modality embeddings and a pooled candidate representation:
Here, is learned and . Pooling includes only observed modalities and uses the same learned vertex weights as the corresponding weighted refinement configuration.
Let denote the unscaled compatibility function of retrieval backbone , where larger values indicate stronger compatibility. Refinement changes the embeddings supplied to this function while preserving its form:
The temperature convention of the backbone is retained; if no temperature scaling is used, .
For a Euclidean Gramian-volume backbone, , where contains the query and observed candidate embeddings as columns. Other backbones retain their corresponding compatibility functions. Thus, volume scoring is one instantiation of the framework rather than a requirement of hypergraph refinement. With one designated positive pair per batch index, the training objective is
The set indexes the auxiliary objectives retained from the selected training recipe, and controls their contributions. Semantic connectivity does not alter the positive assignments: related candidates exchange information during refinement but remain contrastive negatives unless they are designated matches.
For the HYVOL recipe, the auxiliary terms are pooled caption–candidate alignment, variance regularisation, and the backbone’s matching loss, with their original definitions [12]. The pooled representation in Eq. (8) supplies the candidate feature for auxiliary alignment. These auxiliary choices are separate from the hypergraph propagation operator and are held fixed when comparing refinement configurations under a common training recipe.
At inference, we replace refined embeddings with the encoder outputs in Eq. (9). The hypergraph, propagation layers, and auxiliary pooling head are not evaluated. Consequently, relational context influences retrieval through the encoder parameters learned during training, without test-time neighbourhood construction or interaction among test candidates.
Special Cases and Instantiations
The framework separates hyperedge structure, contribution weights, and propagation. The following choices recover HYVOL and define the weighted and relational configurations.
HYVOL [12] is obtained by selecting document hyperedges and caption-guided semantic neighbourhood hyperedges, assigning unit document memberships, and using its original learned semantic attention. Specifically, detached caption features are projected and scored by an additive attention function; a softmax over retained mutual neighbours produces attention coefficients, which are subsequently symmetrised. These coefficients determine semantic memberships, together with unit anchor memberships.
We choose the message and aggregation functions in Eq. (3) as HYVOL’s vertex-to-hyperedge and hyperedge-to-vertex additive operators, with its original degree conventions and gated residual update. Relation labels and participant roles do not explicitly parameterize these messages. Selecting the original neighborhood dropout, auxiliary objectives, and volume-based scorer completes the HYVOL special case.
Weighted hypergraph (WHYVOL): The weighted configuration uses the neighbourhood hyperedges in Eq. (4) and additive propagation in Eq. (5). Learned vertex weights regulate within-candidate contributions, while the symmetric exponential kernel in Eq. (2) regulates cross-candidate connections.
Unlike HYVOL’s original attention rule, these connection weights are not obtained through a neighborhood softmax.
Relational hypergraph (wRHG): The relational configuration uses four typed hyperedge families. Within a single candidate, the document hyperedge connects every observed modalities of that candidate; since it already contains every pair of these modalities (e.g., video with audio), separate modality-pair hyperedges are not required. Between two related candidates and , i.e., candidates whose captions are mutual top- neighbours, three families are used: same-modality hyperedges connect matching modalities (e.g., video of with video of ), cross-modality hyperedges connect different modalities (e.g., video of with audio of ), and the joint-candidate hyperedge contains all observed modalities of both candidates. Messages over these hyperedges are formed multiplicatively by Eq. (6), and vertex features are updated by Eq. (7). Each family has its own relation embedding, and each participating modality has its own role embedding, so the model distinguishes, for example, a video–video same-modality interaction from a video–audio cross-modality interaction, which additive propagation would combine through the same transformation.
These configurations are instances of the common propagation formulation rather than successive restrictions of a single multiplicative operator. HYVOL is recovered by selecting its complete structural and computational specification. Restricting relation families alone changes the available interactions but does not convert multiplicative propagation into HYVOL’s additive aggregation.
Experiments
We evaluate whether training-time hypergraph fusion improves multimodal retrieval across different backbones, benchmarks, and modality configurations. Our experiments assess zero-shot retrieval performance and robustness to missing modalities, comparing HYVOL, WHYVOL, and wRHG with their corresponding backbones and existing retrieval methods. We first describe the datasets and experimental protocol, then present and analyze the results.
Datasets
We describe the training corpus and evaluation benchmarks used to assess retrieval performance across different modality configurations.
Training data
All models trained in this work are initialized from the VAST checkpoint [4] and trained on the 150K-clip subset of VAST-27M used by GRAM [6] and HYPERGRAM [9]. Each clip has five visual captions and audio captions, a subtitle containing the speech transcript, and one omni-modal caption that integrates visual, acoustic, and speech information. We use the omni-modal caption as the training text to provide supervision that captures complementary information across the candidate modalities. The annotation file for the subset lists 150, 154 clips, of which 136, 694 were successfully recovered with both video and audio because online video availability changed. These recovered clips constitute the effective training corpus for all models trained in this work.
For GRAM [6], TRIANGLE [7], and PMRL [8], we evaluated the checkpoints released by their authors. The exact set of clips available during the original training of these checkpoints cannot be reconstructed. We therefore distinguish comparisons against released checkpoints from comparisons between models trained on our common accessible subset.
Evaluation benchmarks
Table 1 summarises the eleven evaluation benchmarks: eight for video–text retrieval and three for audio–text retrieval. These benchmarks differ in their temporal structure, annotation granularity, and available modalities, enabling evaluation across short videos, multi-shot narratives, long-video excerpts, and sound recordings. We do not use any target benchmarks for training in this work, and we perform all evaluations without target-specific adaptation. When several alternative captions describe an item, we select the first caption in the annotation file. DiDeMo and ActivityNet instead use paragraph queries constructed from the sentences associated with each video, as detailed below. All methods we evaluate use the same queries and candidate collections.
| Dataset | Streams | Mode | Text query | Train | Val | Test | Frames | Audio (10 s) | Max caption length |
| MSR-VTT [37] | , , | T–V, T–VA, T–VAS | caption | 9,000 | – | 1,000 | 8 | 1 | 40 |
| DiDeMo [38] | , | T–V, T–VA | paragraph | 8,394 | 1,065 | 1,003 | 8 | 2 | 70 |
| ActivityNet [39] | , | T–V, T–VA | paragraph | 10,009 | – | 4,917 | 8 | 2 | 70 |
| VATEX [40] | , , | T–V, T–VA, T–VAS | caption | 14,060 | – | 431 | 8 | 1 | 40 |
| Shot2Story [41] | , | T–V, T–VA | video summary | 14,016 | 1,982 | 4,023 | 8 | 1 | 70 |
| MSVD [42] | T–V | caption | 1,200 | 100 | 670 | 8 | – | 40 | |
| FLARE [43] | , | T–V, T–VA | vision / unified caption | – | – | 5,000 | 8 | 1 | 70 |
| VALOR-32K [5] | , | T–V, T–VA | audio-visual caption | 25,327 | 3,500 | 3,491 | 8 | 1 | 70 |
| AudioCaps [44] | , | T–AV | audio caption | – | – | 700 | 8 | 1 | 40 |
| Clotho [45] | T–A | caption | 3,839 | 1,045 | 1,045 | – | 1 | 40 | |
| SoundDescs [46] | T–A | description | 23,085 | 4,947 | 4,947 | – | 2 | 70 |
Table 1. Statistics of the evaluation benchmarks. , , , and denote text, video, audio, and subtitles. Mode lists the modalities compared with the text, with the primary modality first: T–V, T–VA, and T–VAS on the video benchmarks, and T–A and T–AV on the audio benchmarks. Retrieval is evaluated in both directions: text-to-video and video-to-text on the video benchmarks, and text-to-audio and audio-to-text on the audio benchmarks. Text query is the annotation used on the text side. Train and Val give the split sizes; we do not use them, since all evaluations are zero-shot. Test is the number of evaluated items. Frames is the number of video frames and Audio the number of 10 s audio segments sampled per clip. Max caption length is the maximum number of text tokens kept per query; longer queries are truncated.
Video–text retrieval. MSR-VTT [37] contains short videos paired with natural-language descriptions. We evaluate on the 1,000-video test split used by VAST [4] and GRAM [6]. We evaluate DiDeMo [38] and ActivityNet [39] as paragraph-to-video retrieval tasks by concatenating the annotated sentences associated with each video into a single query. These settings assess retrieval using descriptions that can encompass multiple events within a video. Following VAST and GRAM, we evaluate on 1,003 videos for DiDeMo and 4,917 videos for ActivityNet.
VATEX [40] supports evaluation with visual, audio, and subtitle information. We follow the test split adopted by VAST and GRAM; however, when we collected the data, only 431 videos from this split remained accessible. All methods we evaluate use this identical subset. We therefore treat published results from a different video collection as contextual references rather than directly matched comparisons. Shot2Story20K [41] contains multi-shot videos paired with summary descriptions. Its 4,023-video test split allows us to examine retrieval when a query describes content distributed across several shots. MSVD [42] contains short videos with multiple textual descriptions, and we evaluate its 670-video test split. Since audio is unavailable in our MSVD evaluation data, only the T–V setting is reported.
FLARE [43] provides clips extracted from longer videos, together with vision, audio, and unified audiovisual captions. We evaluate a fixed subset of 5,000 clips, using vision captions for T–V and unified captions for T–VA. These settings differ in both query content and candidate modalities; therefore, their performance differences do not isolate the contribution of audio.
VALOR-32K [5] consists of approximately 10 s clips accompanied by captions describing both visible content and audible events. We evaluate its 3,491-item test split. Its audiovisual annotations complement benchmarks whose descriptions primarily emphasize visual content.
Audio–text retrieval. AudioCaps [44] pairs audio events with natural-language descriptions. We use the same 700-item test split as VAST [4] and GRAM [6]. The corresponding video is available alongside the audio, allowing evaluation in the T–AV setting.
Clotho v2.1 [45] contains audio recordings with five captions per recording. We evaluate its 1,045-item evaluation split using the first caption associated with each recording. SoundDescs [46] contains sound-effect recordings paired with textual descriptions, and we evaluate on 4,947 test items. Both Clotho and SoundDescs are evaluated in the T–A setting, assessing retrieval from acoustic information alone.
Compared Methods and Implementation Details
Compared methods
We evaluate the proposed refinement framework with two retrieval backbones, GRAM3 [6] and HYPERGRAM4 [9]. GRAM measures multimodal compatibility through Gramian volumes, whereas HYPERGRAM combines Euclidean and hyperbolic volume-based representations. Evaluating both backbones allows us to examine whether the benefits of hypergraph refinement persist across these two geometric scoring formulations. In each case, the refinement module operates on the candidate representations during training while retaining the backbone’s retrieval scoring function.
For each backbone, we compare the model without hypergraph refinement against three configurations: HyVol5 [12], the weighted hypergraph configuration WHyVol, and the weighted relational configuration WRHG. HyVol provides the reference configuration with within-candidate and caption-guided semantic hyperedges. The WHyVol configuration introduces learned vertex contributions and symmetric semantic affinity weights within additive propagation. The WRHG configuration additionally distinguishes relation types and participant roles through relational propagation. These comparisons evaluate the full configurations; § 5.0.4 also examines the hypergraph’s sensitivity to neighborhood size .
We additionally compare against TRIANGLE6 [7] and PMRL7 [8]. TRIANGLE offers an alternative volume-based retrieval approach using triangular area, while PMRL provides an additional multimodal representation-learning baseline. We include results from other published retrieval systems to contextualize performance within the broader literature.
Comparison protocol
We distinguish between methods evaluated using released checkpoints, methods trained in this work, and results taken from prior publications. For GRAM, TRIANGLE, and PMRL, we use the checkpoints released by their authors. Since a pretrained HYPERGRAM checkpoint is unavailable, we train HYPERGRAM using the authors’ public implementation and training recipe. All models trained in this work start from the VAST initialization and use the effective training corpus described in § 4.1.1.
All checkpoints evaluated by us are tested on the same benchmark items and query annotations within each retrieval setting. We evaluate GRAM using its retrieval protocol, and we use TRIANGLE’s released evaluation code. Any method-specific scoring or post-processing is retained and must therefore be considered when interpreting comparisons across methods.
The common evaluation data remove differences arising from query selection and candidate availability. However, the released checkpoints may differ from models trained in this work in their effective training samples and optimization history. Comparisons against these checkpoints therefore measure performance relative to the available reference systems, rather than isolating the effect of a single architectural component.
We report published scores separately as reference results because their evaluation collections, input processing, or training conditions may differ from ours. This distinction is particularly relevant for benchmarks affected by video availability, including the reduced VATEX test subset.
Input processing and model configuration
Backbone encoders. All compared models share the modality encoders of VAST [4]: EVA-CLIP ViT-g/148 for video frames, BEATs9 for audio, and BERT-base10 for both captions and subtitles. We remove the VAST fusion layers, project each stream to a 512-dimensional space, and -normalize it so that the backbone score acts directly on the per-modality embeddings. The backbone temperature is loaded from its checkpoint and kept trainable.
Training recipe. HyVol, wHyVol and wRHG share the same optimization and data settings on both backbones. Starting from the VAST checkpoint, we train each model for one epoch (533 steps) on the recovered training corpus with a global batch of 256 clips, sampling two frames and one 10 s audio segment per clip. No encoder is frozen. We use AdamW with , weight decay 0.01 and gradient-norm clipping at 2.0. The learning rate is for the backbone, for the EVA-CLIP visual encoder and for the hypergraph module, with linear warm-up over the first 10% of the steps followed by linear decay.
Hypergraph and loss settings. We split the batch across four GPUs, and each GPU builds its hypergraph from its own 64 clips, so each GPU’s batch also forms the pool of semantic neighbors. All three configurations use , which gives in Eq. (1), no similarity filter, a semantic-edge dropout of 0.3, and propagation layers. The auxiliary weights in Eq. (10) are for pooled caption–candidate alignment, for variance regularisation (with stability constant ), and for the matching loss. The volume-based retrieval and alignment losses use label smoothing of 0.1. Table 8 in Appendix C lists all hyperparameters.
| Backbone | |
| Initialization | VAST foundation checkpoint |
| Vision encoder | EVA-CLIP ViT-g/14 |
| Audio encoder | BEATs |
| Text encoder | BERT-base (caption and subtitle) |
| Projection dimension | 512, followed by normalisation |
| Trainable modules | all encoders, projections, hypergraph, and heads |
| Hypergraph construction | |
| Neighbours / adaptive cap | |
| Neighbourhood | mutual top- cosine on stop-gradient captions |
| Similarity filter | disabled |
| Semantic-edge dropout | 0.3 |
| Relation families (wRHG) | document, same-modality, cross-modality, joint-candidate |
| Learned weights | |
| Semantic weights | softmax attention (HyVol); kernel of Eq. (2) (wHyVol, wRHG) |
| Affinity temperature | 0.5, learned (wHyVol, wRHG) |
| Affinity projection | identity initialisation (wHyVol, wRHG) |
| Vertex weight | fixed at 1 (HyVol); learned in (0, 2), initialised to 1 (wHyVol, wRHG) |
| Propagation | |
| Operator | additive (HyVol, wHyVol); relational (wRHG) |
| Layers | 2 |
| Activation | GELU (HyVol, wHyVol); ReLU (wRHG); identity in the final layer |
| Mixing , Eq. (6) (wRHG) | initialised to 0.5, learned |
| Losses | |
| Retrieval objective | backbone volume loss, symmetric cross-entropy |
| Backbone temperature | loaded from the checkpoint, learned |
| Label smoothing | 0.1 (volume-based) |
| Alignment weight | 1.0 |
| Variance weight | |
| Matching weight | 0.1 |
| HYPERGRAM backbone | |
| Curvature | initialised to 1, learned |
| Euclidean / hyperbolic weights | initialised to 0.5 / 0.5, learned |
| Hyperbolic gradient clip | 1.0 |
| Optimisation | |
| Optimiser | AdamW, |
| Learning rate | backbone; visual encoder; hypergraph |
| Weight decay / gradient clip | 0.01 / 2.0 |
| Scheduler | linear; warm-up ratio 0.1 |
| Batch size | 256 global; 64 per GPU |
| Epochs / steps | 1 / 533 |
| Training frames / audio | 2 / one 10 s segment |
| Precision / parallelism | fp16 / DDP over four A100 GPUs |
| Seed | 50 (single run) |
| Model selection and inference | |
| Validation | MSR-VTT 1k, T-VAS, ten times per epoch |
| Selection criterion | text-to-video R@1 of the matching score |
| Sampled frames | 8 |
| Audio segments | 1 or 2 of 10 s, per benchmark |
| Maximum query length | 40 or 70 tokens, per benchmark |
| Hypergraph at test time | removed; encoder outputs scored directly |
| Test-time missingness rates | 0, 25, 50, 75, 90% |
Table 8. Hyperparameters. One configuration is used for both backbones and all eleven benchmarks, without benchmark-specific tuning; settings that differ between configurations name the configurations they apply to.
Differences across configurations. The three configurations differ only in their hypergraph weighting and propagation, as defined in § 3.7. For all three configurations on HYPERGRAM, we retain the authors’ hybrid-geometry settings: a learnable curvature initialized to 1, learnable Euclidean and hyperbolic weights initialized to 0.5, and hyperbolic gradient clipping at 1.0.
Hypergraph at training and inference. We apply hypergraph refinement to the observed candidate modalities during training. Training captions guide the construction of semantic neighborhoods but are not included as candidate vertices. Missing modalities are excluded from the hypergraph, so they contribute neither vertex features nor propagated messages. At inference, we discard the hypergraph, propagation layers, and auxiliary pooling head, and retrieve scores using the corresponding backbone’s scoring function in Eq. (9). Consequently, the refinement configurations introduce no additional hypergraph computation during retrieval and require no neighborhood construction over evaluation candidates.
Input processing. Table 1 lists the sampling used at test time. Eight frames are sampled for every candidate that contains video, and audio is represented by one or two 10 s segments depending on the benchmark. Queries are truncated to 40 or 70 tokens, which accommodates the different clip durations and description lengths of the benchmarks.
Checkpoint selection. We validate ten times during the epoch on the MSR-VTT 1k split in the T–VAS setting and keep the checkpoint with the highest text-to-video R@1 of the matching score. MSR-VTT is therefore the only benchmark used for model selection; the other ten benchmarks are used exclusively for testing. The neighbourhood size was likewise chosen from the sweep in § 5.0.4, which uses four of the evaluation benchmarks; a single value is used for all benchmarks and configurations.
Hardware and runs. All models are trained on four NVIDIA A100 GPUs with fp16 mixed precision, while hypergraph aggregation is computed in float32 for numerical stability. Following GRAM [6] and HYPERGRAM [9], which report results from a single training run, we train each configuration once.
Zero-Shot Retrieval Performance
We evaluate retrieval performance across the eleven benchmarks described in § 4.1.2. Table 2 and Table 3 report text-to-video and video-to-text retrieval, respectively, while Table 5 presents both directions of audio–text retrieval. We first examine improvements over the corresponding backbone, then compare refinement configurations and their behavior across modalities. All improvements discussed below are absolute differences in R@1, expressed in percentage points.
| Dataset | Mode | TRIANGLE* | GRAM* | +HYVOL | +WHYVOL | +WRHG | HYPERGRAM† | +HYVOL | +WHYVOL | +WRHG |
| MSR-VTT [37] | T–V | – | 51.9 | 52.6 (+0.7) | 53.4 (+1.5) | 53.0 (+1.1) | 50.8 | 52.5 (+1.7) | 54.3 (+3.5) | 54.2 (+3.4) |
| T–VA | 50.5 | 52.0 | 54.0 (+2.0) | 54.9 (+2.9) | 54.7 (+2.7) | 51.7 | 54.2 (+2.5) | 55.0 (+3.3) | 55.1 (+3.4) | |
| T–VAS | – | 51.8 | 55.8 (+4.0) | 56.8 (+5.0) | 57.4 (+5.6) | 52.3 | 56.1 (+3.8) | 57.0 (+4.7) | 57.3 (+5.0) | |
| DiDeMo [38] | T–V | – | 51.3 | 53.8 (+2.5) | 54.1 (+2.8) | 54.0 (+2.7) | 50.0 | 53.6 (+3.6) | 54.3 (+4.3) | 54.0 (+4.0) |
| T–VA | 51.0 | 50.6 | 54.6 (+4.0) | 53.8 (+3.2) | 54.1 (+3.5) | 50.4 | 54.2 (+3.8) | 54.6 (+4.2) | 54.6 (+4.2) | |
| ActivityNet [39] | T–V | – | 55.5 | 56.2 (+0.7) | 59.6 (+4.1) | 59.6 (+4.1) | 56.1 | 56.7 (+0.6) | 59.5 (+3.4) | 59.9 (+3.8) |
| T–VA | 56.0 | 56.4 | 56.9 (+0.5) | 60.8 (+4.4) | 60.7 (+4.3) | 56.8 | 57.8 (+1.0) | 60.5 (+3.7) | 60.9 (+4.1) | |
| VATEX [40] | T–V | – | 87.2 | 92.1 (+4.9) | 91.9 (+4.7) | 92.3 (+5.1) | 88.4 | 92.6 (+4.2) | 92.3 (+3.9) | 92.3 (+3.9) |
| T–VA | 88.9 | 88.4 | 93.0 (+4.6) | 93.5 (+5.1) | 93.5 (+5.1) | 88.6 | 93.0 (+4.4) | 93.7 (+5.1) | 93.5 (+4.9) | |
| T–VAS | – | 88.4 | 94.0 (+5.6) | 93.5 (+5.1) | 94.0 (+5.6) | 89.8 | 93.3 (+3.5) | 93.5 (+3.7) | 93.5 (+3.7) | |
| Shot2Story [41] | T–V | – | 70.5 | 75.1 (+4.6) | 75.6 (+5.1) | 76.7 (+6.2) | 75.5 | 76.0 (+0.5) | 76.3 (+0.8) | 76.5 (+1.0) |
| T–VA | 75.8 | 70.5 | 74.8 (+4.3) | 76.3 (+5.8) | 77.4 (+6.9) | 75.0 | 74.6 (−0.4) | 75.9 (+0.9) | 76.6 (+1.6) | |
| MSVD [42] | T–V | – | 70.4 | 75.8 (+5.4) | 74.3 (+3.9) | 73.6 (+3.2) | 69.6 | 75.8 (+6.2) | 75.5 (+5.9) | 75.7 (+6.1) |
| FLARE [43] | T–V | – | 47.4 | 52.1 (+4.7) | 52.6 (+5.2) | 54.1 (+6.7) | 52.3 | 53.4 (+1.1) | 53.9 (+1.6) | 53.8 (+1.5) |
| T–VA | 56.1 | 46.0 | 52.0 (+6.0) | 51.7 (+5.7) | 53.9 (+7.9) | 52.9 | 52.1 (−0.8) | 52.1 (−0.8) | 52.8 (−0.1) | |
| VALOR-32K [5] | T–V | – | 64.6 | 71.5 (+6.9) | 70.2 (+5.6) | 70.2 (+5.6) | 65.6 | 71.7 (+6.1) | 70.6 (+5.0) | 70.4 (+4.8) |
| T–VA | 75.9 | 75.2 | 80.5 (+5.3) | 80.4 (+5.2) | 80.8 (+5.6) | 75.9 | 80.9 (+5.0) | 80.5 (+4.6) | 80.9 (+5.0) |
Table 2. Zero-shot text-to-video retrieval (R@1) on eight video benchmarks. For FLARE, T–V uses the vision captions and T–VA the unified captions. The compared models TRIANGLE [7], GRAM [6] and HYPERGRAM [9] are all volume-based: they score a clip by the area of a triangle, the Gramian volume, and a combination of Euclidean and hyperbolic volumes, respectively; +HYVOL, +WHYVOL and +WRHG denote the backbone trained with HyVol [12], weighted HyVol and the weighted relational hypergraph, respectively. ⋆Authors’ released checkpoint (TRIANGLE, GRAM), evaluated by us on the same test data: GRAM with its retrieval protocol, TRIANGLE with its released evaluation code. †Trained by us with the authors’ released code and recipe (HYPERGRAM). Numbers in parentheses give the change with respect to the backbone (GRAM or HYPERGRAM); green marks an improvement, red a drop. A dash (–) marks a setting that is not available. Bold marks the best result within each backbone group.
| Dataset | Mode | Baseline: TRIANGLE | Baseline: GRAM | GRAM backbone: +HyVol | GRAM backbone: +wHyVol | GRAM backbone: +wRHG | HyperGRAM backbone: HyperGRAM | HyperGRAM backbone: +HyVol | HyperGRAM backbone: +wHyVol | HyperGRAM backbone: +wRHG |
| MSR-VTT [37] | T–V | – | 48.7 | 54.5 (+5.8) | 55.1 (+6.4) | 56.5 (+7.8) | 48.7 | 54.2 (+5.5) | 55.2 (+6.5) | 55.9 (+7.2) |
| T–VA | 49.7 | 49.3 | 55.8 (+6.5) | 56.3 (+7.0) | 57.2 (+7.9) | 49.8 | 56.6 (+6.8) | 56.5 (+6.7) | 57.2 (+7.4) | |
| T–VAS | – | 50.1 | 58.1 (+8.0) | 58.0 (+7.9) | 58.0 (+7.9) | 50.9 | 59.2 (+8.3) | 58.8 (+7.9) | 59.3 (+8.4) | |
| DiDeMo [38] | T–V | – | 49.0 | 53.8 (+4.8) | 54.0 (+5.0) | 54.4 (+5.4) | 50.7 | 53.8 (+3.1) | 54.6 (+3.9) | 54.8 (+4.1) |
| T–VA | 49.4 | 49.0 | 54.1 (+5.1) | 55.5 (+6.5) | 55.0 (+6.0) | 50.8 | 54.7 (+3.9) | 55.1 (+4.3) | 55.5 (+4.7) | |
| ActivityNet [39] | T–V | – | 49.3 | 56.2 (+6.9) | 61.4 (+12.1) | 61.6 (+12.3) | 50.5 | 55.8 (+5.3) | 61.1 (+10.6) | 61.3 (+10.8) |
| T–VA | 52.3 | 49.7 | 56.6 (+6.9) | 62.7 (+13.0) | 63.0 (+13.3) | 50.9 | 56.4 (+5.5) | 62.8 (+11.9) | 63.0 (+12.1) | |
| VATEX [40] | T–V | – | 84.7 | 92.3 (+7.6) | 92.6 (+7.9) | 92.6 (+7.9) | 86.3 | 92.4 (+6.1) | 92.3 (+6.0) | 92.1 (+5.8) |
| T–VA | 86.5 | 86.5 | 92.8 (+6.3) | 94.0 (+7.5) | 93.7 (+7.2) | 88.4 | 92.9 (+4.5) | 93.5 (+5.1) | 93.3 (+4.9) | |
| T–VAS | – | 86.5 | 93.8 (+7.3) | 94.4 (+7.9) | 94.4 (+7.9) | 87.8 | 93.9 (+6.1) | 94.0 (+6.2) | 94.0 (+6.2) | |
| Shot2Story [41] | T–V | – | 66.4 | 77.6 (+11.2) | 75.7 (+9.3) | 76.4 (+10.0) | 73.0 | 78.1 (+5.1) | 75.9 (+2.9) | 75.8 (+2.8) |
| T–VA | 74.8 | 66.2 | 77.6 (+11.4) | 75.7 (+9.5) | 76.3 (+10.1) | 73.1 | 78.1 (+5.0) | 76.2 (+3.1) | 76.0 (+2.9) | |
| MSVD [42] | T–V | – | 69.0 | 74.5 (+5.5) | 74.6 (+5.6) | 74.5 (+5.5) | 67.0 | 74.6 (+7.6) | 74.6 (+7.6) | 75.2 (+8.2) |
| FLARE [43] | T–V | – | 47.8 | 55.6 (+7.8) | 55.7 (+7.9) | 55.8 (+8.0) | 52.4 | 56.4 (+4.0) | 56.4 (+4.0) | 55.9 (+3.5) |
| T–VA | 55.0 | 46.8 | 55.5 (+8.7) | 54.6 (+7.8) | 55.0 (+8.2) | 52.6 | 56.4 (+3.8) | 55.9 (+3.3) | 55.8 (+3.2) | |
| VALOR-32K [5] | T–V | – | 61.4 | 71.8 (+10.4) | 69.9 (+8.5) | 70.2 (+8.8) | 62.5 | 72.5 (+10.0) | 71.4 (+8.9) | 71.5 (+9.0) |
| T–VA | 74.7 | 71.2 | 81.6 (+10.4) | 81.1 (+9.9) | 80.8 (+9.6) | 72.4 | 82.2 (+9.8) | 82.0 (+9.6) | 81.8 (+9.4) |
Table 3. Zero-shot video-to-text retrieval (R@1) on eight video benchmarks. For FLARE, T–V uses the vision captions and T–VA the unified captions. The compared methods, evaluation settings and notation are described in the caption of Table 2. Bold marks the best result within each backbone group.
| Dataset | Mode | TRIANGLE | GRAM backbone: GRAM | +HYVOL | +WHYVOL | +WRHG | HYPERGRAM backbone: HYPERGRAM | +HYVOL | +WHYVOL | +WRHG |
| Text-to-audio | ||||||||||
| AudioCaps [44] | T–AV | 37.4 | 33.4 | 35.7 (+2.3) | 35.2 (+1.8) | 35.4 (+2.0) | 32.4 | 36.1 (+3.7) | 36.9 (+4.5) | 36.8 (+4.4) |
| Clotho [45] | T–A | – | 11.9 | 11.0 (−0.9) | 13.2 (+1.3) | 12.3 (+0.4) | 12.9 | 11.5 (−1.4) | 12.3 (−0.6) | 13.0 (+0.1) |
| SoundDescs [46] | T–A | – | 3.6 | 4.6 (+1.0) | 4.5 (+0.9) | 4.5 (+0.9) | 3.9 | 4.5 (+0.6) | 4.2 (+0.3) | 4.1 (+0.2) |
| Audio-to-text | ||||||||||
| AudioCaps [44] | T–AV | 38.2 | 33.4 | 33.7 (+0.3) | 33.8 (+0.4) | 34.9 (+1.5) | 33.2 | 36.4 (+3.2) | 37.5 (+4.3) | 36.2 (+3.0) |
| Clotho [45] | T–A | – | 11.5 | 12.5 (+1.0) | 15.0 (+3.5) | 14.0 (+2.5) | 12.1 | 12.4 (+0.3) | 13.7 (+1.6) | 13.6 (+1.5) |
| SoundDescs [46] | T–A | – | 4.3 | 4.7 (+0.4) | 4.5 (+0.2) | 4.3 (0.0) | 4.2 | 4.6 (+0.4) | 4.4 (+0.2) | 4.3 (+0.1) |
Table 5. Zero-shot text-to-audio and audio-to-text retrieval (R@1). The compared methods, evaluation settings and notation are described in the caption of Table 2. Bold marks the best result within each backbone group.
Video–text retrieval
Performance across backbones and retrieval directions. Hypergraph refinement improves retrieval across a broad range of benchmarks. With GRAM, wRHG outperforms the backbone in all 17 text-to-video settings and all 17 video-to-text settings. With HYPERGRAM, it improves 16 of the 17 text-to-video settings and all 17 video-to-text settings. These results show that the benefits extend across both backbone formulations and retrieval directions, although their magnitude depends on the benchmark and available modalities. Figure 3 visualizes these results in the T–VA setting, with the minimum and maximum R@1 of each benchmark shown below its name.
Figure 3. Zero-shot text-to-video and video-to-text retrieval (R@1) with the GRAM backbone (top) and the HYPERGRAM backbone (bottom) on seven video benchmarks, all evaluated in the T–VA setting. Each axis is scaled independently to the range of R@1 values on that benchmark; the minimum and maximum R@1 are shown in parentheses below its name, so distances are comparable only within an axis. The exact values are the T–VA rows of Table 2 and Table 3.
ActivityNet exhibits particularly substantial improvements. In the T–VA setting, GRAM +wRHG increases text-to-video R@1 from 56.4 to 60.7 and video-to-text R@1 from 49.7 to 63.0. HYPERGRAM +wRHG shows a similar pattern, increasing the corresponding scores from 56.8 to 60.9 and from 50.9 to 63.0. The larger gains in video-to-text retrieval indicate that the effect of refinement is not symmetric across retrieval directions. Although both directions use the same compatibility scores, ranking candidates for each textual query and ranking descriptions for each video impose different retrieval conditions.
Effect of weighting and relational modeling. The weighted configurations improve on HyVol [12] in several settings. On ActivityNet T–VA, replacing HYVOL with WHYVOL increases GRAM text-to-video R@1 from 56.9 to 60.8 and video-to-text R@1 from 56.6 to 62.7. With HYPERGRAM, the corresponding scores increase from 57.8 to 60.5 and from 56.4 to 62.8. The consistent pattern across backbones indicates that the weighted configuration is particularly effective on ActivityNet.
Relational modeling provides further benefits in selected settings. On Shot2Story text-to-video retrieval with GRAM, wRHG reaches 76.7 in T–V and 77.4 in T–VA, compared with 75.6 and 76.3 for WHYVOL. On MSR-VTT video-to-text retrieval in T–V, wRHG improves over WHYVOL from 55.1 to 56.5 with GRAM and from 55.2 to 55.9 with HYPERGRAM. These comparisons identify settings in which explicit modeling of distinct interactions provides improvements over weighted additive propagation.
The ordering of the configurations is nevertheless not uniform. HYVOL [12] achieves the highest text-to-video score on MSVD for both backbones, reaching 75.8. It also outperforms the weighted configurations on Shot2Story video-to-text retrieval and VALOR-32K video-to-text retrieval. Thus, the additional weighting and relational mechanisms provide benchmark-dependent benefits rather than consistent improvement across settings. § 5.0.4 examines one such design choice, the neighbourhood size, under otherwise fixed settings.
Influence of available modalities. Additional candidate streams can improve retrieval when they provide information relevant to the query. On MSR-VTT, GRAM + wRHG increases text-to-video R@1 from 53.0 with video alone to 54.7 with video and audio, and to 57.4 when subtitles are also included. HYPERGRAM + wRHG follows the same trend, achieving 54.2, 55.1, and 57.3, respectively.
The contribution of audio is particularly pronounced on VALOR-32K, whose descriptions cover both visible and audible events. Adding audio increases text-to-video R@1 from 70.2 to 80.8 for GRAM + wRHG and from 70.4 to 80.9 for HYPERGRAM + wRHG. This pattern is consistent with the audiovisual content of the queries and illustrates the importance of evaluating fusion under annotations that describe the available streams.
Additional modalities do not guarantee higher performance. On FLARE, HYPERGRAM +wRHG obtains 52.8 in T–VA, slightly below the backbone score of 52.9. Moreover, FLARE uses different descriptions for T–V and T–VA, so we cannot attribute their difference solely to the inclusion of audio. These results highlight the dependence of retrieval performance on the combination of candidate information, query content, and backbone.
Audio–text retrieval
Table 5 shows that refinement also benefits audio–text retrieval, though the gains are less uniform than on the video benchmarks. On AudioCaps, all three configurations improve over their corresponding backbones in both retrieval directions. HYPERGRAM +wHyVol yields the strongest results, increasing text-to-audio R@1 from 32.4 to 36.9 and audio-to-text R@1 from 33.2 to 37.5. HYPERGRAM +wRHG reaches 36.8 and 36.2, respectively, indicating that relational propagation offers no additional advantage over weighted additive propagation in this setting.
Clotho reveals a stronger dependence on the configuration and retrieval direction. For text-to-audio retrieval, HyVol [12] reduces R@1 from 11.9 to 11.0 with GRAM and from 12.9 to 11.5 with HYPERGRAM. The weighted configurations recover part or all of this loss: GRAM +wHyVol reaches 13.2, while HYPERGRAM +wRHG reaches 13.0. In audio-to-text retrieval, wHyVol achieves the highest score for both backbones, increasing R@1 from 11.5 to 15.0 with GRAM and from 12.1 to 13.7 with HYPERGRAM.
On SoundDescs, improvements are small by comparison, and HyVol achieves the highest scores among the refinement configurations for both backbones and retrieval directions. For example, GRAM +HyVol increases text-to-audio R@1 from 3.6 to 4.6, while HYPERGRAM +HyVol increases it from 3.9 to 4.5. The low absolute recall indicates that this benchmark remains challenging under the evaluated training and retrieval protocol.
Unlike AudioCaps, Clotho and SoundDescs contain only audio on the candidate side. The results therefore assess transfer to unimodal candidates rather than the direct benefit of combining multiple streams at inference. They demonstrate that training-time refinement can benefit audio-only retrieval, while also showing that the more elaborate configurations are not consistently preferable in this setting.
Comparison with existing methods
Table 6 compares video retrieval results with those of existing methods, grouped by available candidate modalities. Both GRAM +wRHG and HYPERGRAM +wRHG outperform the re-evaluated PMRL checkpoint [8] in every reported setting in this table. On ActivityNet T–VA, for example, PMRL obtains 54.1 for text-to-video retrieval and 48.6 for video-to-text retrieval, compared with 60.9 and 63.0 for HYPERGRAM +wRHG. On MSR-VTT T–VAS, HYPERGRAM +wRHG achieves 57.3 and 59.3, compared with 54.3 and 52.9 for PMRL.
The comparison with Triangle [7] is more mixed. On VALOR-32K T–VA, GRAM +wRHG reaches 80.8 for text-to-video retrieval, compared with 75.9 for Triangle. Conversely, Triangle obtains 56.1 on FLARE T–VA, exceeding the corresponding scores of 53.9 and 52.8 for GRAM +wRHG and HYPERGRAM +wRHG. On AudioCaps, Triangle also remains stronger than the refinement configurations, achieving 37.4 for text-to-audio retrieval and 38.2 for audio-to-text retrieval.
Table 7 provides additional context for AudioCaps text-to-audio retrieval. In the T–AV setting, GRAM +wRHG and HYPERGRAM +wRHG achieve 35.4 and 36.8, respectively, compared with 34.4 for the re-evaluated PMRL checkpoint. Published results obtained with other pretraining corpora or evaluation procedures are included as reference values. They establish the broader performance context but do not constitute controlled comparisons of fusion mechanisms.
Table 7. Comparison with published text-to-audio retrieval results (R@1) on AudioCaps [44]. §Published numbers (reference only); ⋆released checkpoint evaluated by us.
Neighbourhood-Size Sensitivity
The neighbourhood size in Eq. (1) determines how many caption neighbours each candidate considers before the mutual-neighbour test is applied. Table 4 varies over for GRAM +HyVol while keeping all other settings fixed. Since each GPU builds its hypergraph from 64 clips, , and is the largest neighbourhood a batch can support.
| Dataset | Mode | T→V: | T→V: | T→V: | T→V: | V→T: | V→T: | V→T: | V→T: |
| MSR-VTT [37] | T–V | 50.0 | 52.1 | 52.6 | 51.1 | 47.7 | 52.2 | 54.5 | 54.4 |
| T–VA | 50.6 | 52.5 | 54.0 | 52.1 | 49.0 | 51.8 | 55.8 | 55.7 | |
| T–VAS | 53.2 | 54.3 | 55.8 | 54.7 | 51.9 | 53.7 | 58.1 | 57.6 | |
| DiDeMo [38] | T–V | 50.0 | 51.8 | 53.8 | 52.5 | 48.0 | 52.6 | 53.8 | 53.8 |
| T–VA | 50.3 | 51.4 | 54.6 | 52.4 | 48.4 | 52.7 | 54.1 | 53.3 | |
| ActivityNet [39] | T–V | 53.3 | 55.9 | 56.2 | 54.9 | 47.8 | 50.6 | 56.2 | 49.2 |
| T–VA | 53.9 | 56.7 | 56.9 | 55.4 | 48.4 | 50.2 | 56.6 | 49.8 | |
| VATEX [40] | T–V | 87.7 | 91.9 | 92.1 | 91.9 | 91.4 | 92.1 | 92.3 | 91.6 |
| T–VA | 89.0 | 91.9 | 93.0 | 93.3 | 91.7 | 92.1 | 92.8 | 92.3 | |
| T–VAS | 90.1 | 92.6 | 94.0 | 93.5 | 93.3 | 93.7 | 93.8 | 93.0 |
Table 4. Sensitivity to the number of semantic neighbours . Values are R@1 for text-to-video (T→V) and video-to-text (V→T) retrieval; bold marks the best tested in each row, and is used in all main experiments. The reported configuration uses HYVOL on top of the GRAM backbone.
Increasing from 4 to 12 improves R@1 in every setting, and the effect is most pronounced for video-to-text retrieval. Averaged over the modality sets of each benchmark, the video-to-text gain is 6.6 points on MSR-VTT, 5.8 on DiDeMo, and 8.3 on ActivityNet; on VATEX, where R@1 already exceeds 90, it is 0.8 points. The corresponding text-to-video gains are smaller but consistent, for example from 53.2 to 55.8 on MSR-VTT T–VAS and from 50.3 to 54.6 on DiDeMo T–VA. A further increase to brings no average benefit and markedly degrades video-to-text retrieval on ActivityNet, where R@1 falls from 56.2 to 49.2 in T–V and from 56.6 to 49.8 in T–VA. Across the 20 settings, yields the best or tied-best result in 19; the only exception is VATEX T–VA text-to-video retrieval, where is 0.3 points higher. We therefore adopt for all configurations and benchmarks.
Robustness to Missing Modalities
We further evaluate retrieval when individual clips lose part of their multimodal information. Unlike the complete-modality comparisons, this setting introduces heterogeneous observations within the same collection: some clips retain all available streams, while others are represented by a reduced modality set. Figure 4–7 examine whether improvements obtained through training-time hypergraph refinement persist under these conditions.
Figure 4. Zero-shot text-to-video retrieval under missing modalities with the GRAM backbone on seven video benchmarks. Each panel shows the R@1 gain of +HYVOL, +WHYVOL and +WRHG over GRAM (dashed line at zero) as a fraction of the gallery clips that lose one randomly chosen modality; the per-clip masks are fixed and identical for all methods, and is the standard setting. MSR-VTT and VATEX use the T–VAS modality set, and the other benchmarks use T–VA.
Evaluation protocol
The evaluation covers seven video benchmarks: MSR-VTT, DiDeMo, ActivityNet, VATEX, Shot2Story, FLARE, and VALOR-32K. MSR-VTT and VATEX use the T–VAS setting, while the remaining benchmarks use T–VA. MSVD is excluded because its evaluation configuration contains only video.
For a fraction of the clips, one randomly selected modality is removed. We consider missingness rates of 0%, 25%, 50%, 75%, and 90%, with 0% corresponding to the standard evaluation setting. The per-clip masks are fixed and identical across methods, ensuring that performance differences are measured under the same availability conditions. Each affected clip retains two modalities in T–VAS and one modality in T–VA. Thus, the missingness rate specifies the proportion of incomplete clips rather than the proportion of all modality observations removed.
The affected clips serve as gallery items in text-to-video retrieval and as queries in video-to-text retrieval. Figures 4 and 5 report results with GRAM, while Figures 6 and 7 report results with HYPERGRAM. Each curve shows the difference in R@1 between a refinement configuration and its corresponding backbone at the same missingness rate. Positive values therefore indicate an advantage over the backbone under incomplete observations; they do not imply that absolute retrieval performance is unchanged as modalities become unavailable. Table 9 in Appendix D lists the underlying R@1 values for each method compared at each missingness rate.
Results with Gram
Figure 4 shows that wHyVoL and wRHG retain positive text-to-video gains over GRAM on all seven benchmarks at every evaluated missingness rate. Their advantage therefore persists even when 90% of the clips lose one modality. Although the gains generally become smaller as missingness increases, the weighted configurations remain beneficial across the evaluated collections.
ActivityNet clearly distinguishes the weighted configurations from HyVol [12]. Without missing modalities, the T–VA gains over GRAM are 4.4 percentage points for wHyVoL and 4.3 points for wRHG, compared with 0.5 points for
This ordering does not hold uniformly. On DiDeMo, HyVol provides a larger text-to-video gain than either weighted configuration at the nonzero missingness rates. On VALOR-32K, all three configurations remain above GRAM throughout the sweep, and their gains do not decrease monotonically. These differences indicate that the relative benefit of weighting depends on the benchmark as well as the amount of missing information.
For video-to-text retrieval, Figure 5 shows positive gains for all three refinement configurations across all evaluated missingness rates. The gains generally narrow as more clips become incomplete. On ActivityNet, the complete-modality improvements of 13.0 points for wHyVol and 13.3 points for wRHG decrease with missingness, but both configurations remain above HyVol throughout the sweep. On VATEX, the weighted configurations likewise retain a clearer advantage at high missingness rates. Overall, the GRAM results show that refinement remains beneficial under incomplete observations, while the magnitude and ordering of the improvements vary across datasets.
Figure 5. Zero-shot video-to-text retrieval under missing modalities with the GRAM backbone: R@1 gain over GRAM (dashed line at zero). The protocol is the same as in Figure 4.
HyVol. As shown in Figure 4, both weighted configurations remain above the backbone throughout the missingness sweep, whereas HyVol falls below at 25%, 50%, and 75% missingness. The weighted configurations therefore maintain their advantage as audiovisual observations become incomplete.
A similar distinction appears on MSR-VTT. In the complete T–VAS setting, wHyVol and wRHG improve over GRAM by 5.0 and 5.6 points, respectively, compared with 4.0 points for HyVol. At 90% missingness, both weighted configurations remain positive, whereas HyVol falls slightly below the backbone. On VATEX, the weighted configurations also retain a larger advantage than HyVol at the highest missingness rate.
Results with HyperGram
The HyperGRAM results in Figure 6 and Figure 7 vary more across benchmarks. On MSR-VTT and ActivityNet, both weighted configurations retain positive gains in both retrieval directions at every evaluated missingness rate. However, Shot2Story and FLARE include conditions where refinement offers little benefit or underperforms the backbone.
Figure 6. Zero-shot text-to-video retrieval under missing modalities with the HYPERGRAM backbone: R@1 gain over HYPERGRAM (dashed line at zero). The protocol is the same as in Figure 4.
Figure 7. Zero-shot video-to-text retrieval under missing modalities with the HYPERGRAM backbone: R@1 gain over HYPERGRAM (dashed line at zero). The protocol is the same as in Figure 4.
On ActivityNet text-to-video retrieval, wHyVol and wRHG improve over HyperGRAM by 3.7 and 4.1 points in the complete T–VA setting, compared with 1.0 point for HyVol. Figure 6 shows that both weighted configurations remain above the backbone at all four nonzero missingness rates, whereas HYVOL falls below it. The video-to-text results in Figure 7 show the same preference for the weighted configurations: their initial gains of 11.9 and 12.1 points decrease with missingness but remain positive, while HYVOL falls below HYPERGRAM at 90% missingness.
The Shot2Story results establish a limitation. In text-to-video retrieval, wRHG’s advantage becomes negligible at 90% missingness, while HYVOL and wHYVOL underperform the backbone at several rates. In video-to-text retrieval, all three configurations fall below HYPERGRAM at the highest missingness rate. The improvement observed with complete observations therefore does not consistently transfer to severely incomplete clips on this benchmark.
VATEX illustrates the importance of retrieval direction. All three configurations remain above HYPERGRAM in text-to-video retrieval, although their relative ordering changes across the sweep. In video-to-text retrieval, wRHG maintains a positive margin throughout, while wHYVOL’s advantage approaches zero at 90% missingness. HYVOL falls below the backbone at 75% and 90% missingness. The relational configuration therefore provides the most persistent video-to-text advantage on this benchmark.
FLARE also exhibits non-monotonic behavior. In text-to-video retrieval, all three configurations underperform HYPERGRAM at 25%, 50%, and 75% missingness, then show gains at 90%. In video-to-text retrieval, all three fall below the backbone at 75% missingness and recover positive gains at 90%. Since these curves report differences from a backbone whose performance also changes with missingness, such recoveries do not imply an increase in absolute recall.
Taken together, Figure 4–Figure 7 show that the weighted configurations frequently preserve their advantage under incomplete observations, with particularly consistent results for GRAM. The exceptions with HYPERGRAM demonstrate that this advantage is not universal. Moreover, the hypergraph is absent during evaluation: the observed benefits arise through the learned encoder parameters, rather than through reconstruction of missing streams or information exchange between evaluation clips.
Discussion
Why training-time exchange helps. The refinement never touches the test-time score, yet all three configurations improve both backbones on almost every video setting. The only
path from the hypergraph to retrieval is the encoder weights, so the gains indicate that the encoders learn more useful per-modality embeddings when related clips can supply evidence during training. Several of the largest gains occur where the backbone is weak; for example, in ActivityNet video-to-text, GRAM obtains 49.7 R@1 and WRHG 63.0, and in FLARE in the T–VA setting, WRHG raises GRAM’s text-to-video R@1 from 46.0 to 53.9. In this sense the hypergraph acts as a fusion mechanism that operates across samples during learning rather than across modalities at inference.
Weighting versus relational structure. The two extensions address different weaknesses of HyVOL, and the results separate them only partly. The weighted configuration produces the clearest change: on ActivityNet it raises video-to-text R@1 by about six points over HyVOL with both backbones, and it keeps the gain under missing modalities where HyVOL falls below the backbone. Typed relations add a further but smaller improvement, mainly on Shot2Story and MSR-VTT, and on several benchmarks wHyVoL and wRHG are within a few tenths of a point of each other. A plausible reading is that deciding how much a view or a neighbour contributes matters more for low-quality data than deciding what kind of link connects them. HyVOL remains the best configuration for MSVD text-to-video and for SoundDesc, so the added weighting and typing are not uniformly beneficial.
| Method | MSR-VTT [37] T→V | MSR-VTT [37] V→T | DiDeMo [38] T→V | DiDeMo [38] V→T | ActivityNet [39] T→V | ActivityNet [39] V→T | VATEX [40] T→V | VATEX [40] V→T |
| Text–video (T–V) | ||||||||
| HiTeA§ [16] | 34.4 | 46.8 | 43.2 | 56.5 | – | – | – | – |
| ImageBind§ [2] | 36.8 | 36.8 | – | – | – | – | – | – |
| InternVideo-L§ [17] | 40.7 | 39.6 | 31.5 | 33.5 | 30.7 | 31.4 | 49.5 | 69.5 |
| LanguageBind§ [3] | 44.8 | 40.9 | 39.9 | 39.8 | 41.0 | 39.1 | – | – |
| mPLUG-2§ [18] | 47.1 | – | 45.7 | – | – | – | – | – |
| Norton§ [47] | 10.7 | – | – | – | – | – | – | – |
| OmniVL§ [19] | 42.0 | 34.6 | 40.6 | 33.3 | – | – | – | – |
| TVTSv2§ [15] | 38.2 | 38.2 | 34.6 | 34.6 | – | – | – | – |
| UMT§ [48] | 33.3 | 33.3 | 34.0 | 34.0 | 31.9 | 31.9 | – | – |
| UMT-L§ [48] | 40.7 | 40.7 | 48.6 | 24.9 | 41.9 | 39.4 | – | – |
| ViCLIP§ [49] | 42.4 | 41.3 | 18.4 | 27.9 | 15.1 | 24.0 | – | – |
| VideoCoCa§ [50] | 34.3 | 34.3 | – | – | 34.5 | 34.5 | 53.2 | – |
| VideoPrism-b§ [20] | 51.4 | 50.2 | – | – | 49.6 | 47.9 | 62.5 | 76.2 |
| PMRL⋆ [8] | 51.3 | 49.6 | 53.0 | 50.5 | 53.8 | 48.0 | 87.7 | 85.4 |
| GRAM+WRHG (ours) | 53.0 | 56.5 | 54.0 | 54.4 | 59.6 | 61.6 | 92.3 | 92.6 |
| HYPERGRAM+WRHG (ours) | 54.2 | 55.9 | 54.0 | 54.8 | 59.9 | 61.3 | 92.3 | 92.1 |
| Text–video–audio (T–VA) | ||||||||
| VAST§ [4] | 49.3 | 43.7 | 49.5 | 48.2 | 51.4 | 46.8 | 80.0 | 77.1 |
| PMRL⋆ [8] | 50.9 | 51.0 | 52.5 | 49.3 | 54.1 | 48.6 | 88.9 | 86.1 |
| GRAM+WRHG (ours) | 54.7 | 57.2 | 54.1 | 55.0 | 60.7 | 63.0 | 93.5 | 93.7 |
| HYPERGRAM+WRHG (ours) | 55.1 | 57.2 | 54.6 | 55.5 | 60.9 | 63.0 | 93.5 | 93.3 |
| Text–video–audio–subtitle (T–VAS) | ||||||||
| VAST§ [4] | 50.7 | 49.0 | – | – | – | – | 82.1 | 78.7 |
| PMRL⋆ [8] | 54.3 | 52.9 | – | – | – | – | 89.6 | 87.0 |
| GRAM+WRHG (ours) | 57.4 | 58.0 | – | – | – | – | 94.0 | 94.4 |
| HYPERGRAM+WRHG (ours) | 57.3 | 59.3 | – | – | – | – | 93.5 | 94.0 |
Table 6. Comparison with published zero-shot retrieval results (R@1) on the four benchmarks for which such results exist. Rows are grouped by modality set, and methods are compared only within the same group. §Numbers reported in the original papers under their own pretraining data and protocols (reference only). ⋆Authors’ released checkpoint, evaluated by us on the same test data as our models. A dash (–) marks a result that is not available. Bold marks the best result within each modality group.
Asymmetry between retrieval directions. Gains are generally larger for video-to-text than for text-to-video retrieval,
reaching 13.3 points on ActivityNet against 4.3 in the other direction. The backbone scores are the same in both directions, but the rankings are formed over different sets. One possible explanation is that the refinement acts on the video embeddings, which are the side exposed to related clips during training, so a better-organised video space benefits most directly the direction in which videos act as queries.
Fusion under missing and unequal modalities. The missing-modality study shows that the benefit of training-time fusion largely survives when views disappear at test time, although the models never see incomplete clips during training. With GRAM, both weighted configurations remain above the backbone on all seven benchmarks at every rate, suggesting that learned vertex weights reduce the embeddings’ dependence on any single stream. The HYPERGRAM results are less regular, and on two benchmarks with long or multi-shot videos some conditions show no benefit, possibly because removing a stream there changes which modality carries the most information. On the audio benchmarks, gains are smaller and less consistent, which indicates that exchange between related clips is most useful when each clip offers several complementary streams to fuse.
Cost and applicability. Since no part of the hypergraph survives training, the refined models have the same parameters at inference, the same encoding cost and the same scoring function as their backbones, unlike test-time re-ranking and query-bank methods [28, 29]. The approach can therefore be added to an existing volume-based retrieval system without changing its deployment; the only extra cost is a small hypergraph module during the same one-epoch training.
Limitations
Backbone coverage. All compared models use the VAST encoders. We apply our models (HYVOL, WHYVOL, and WRHG) to the two volume-based backbones, GRAM and HYPERGRAM. TRIANGLE [7] is evaluated exclusively as a baseline because it is designed for exactly three modalities. Its score corresponds to the area spanned by three embeddings, the three-vector case of the GRAM volume; it therefore cannot incorporate subtitles as a fourth stream, and its underlying principle is already represented by GRAM. PMRL [8] is also evaluated exclusively as a baseline because it adopts a different formulation: it aligns modalities by maximizing the leading singular value of their stacked embeddings, ranks candidates using a single fused representation, and assumes complete modality availability. In contrast, our models refine separate per-modality embeddings. Integrating them with PMRL would therefore require adapting its objective rather than directly incorporating our module. Extensions to PMRL and to encoders pretrained on other corpora remain directions for future research.
Training-time refinement only. Our models employ the hypergraph feature fusion exclusively during training, with its influence incorporated into the encoder weights. At inference,
each clip is encoded and scored independently using the original backbone. Although this design preserves the inference cost, it constrains the model’s adaptability. Specifically, the deployed model cannot exploit gallery-specific structure, adapt to individual queries, or account for the particular pattern of missing streams encountered at test time. Such adaptation would require an explicit inference-time component, such as a lightweight graph constructed over the gallery, which lies beyond the scope of this work.
Training scale and single runs. Following GRAM, PMRL, and HYPERGRAM, we perform one epoch of continued pre-training on the 150K-clip subset of VAST-27M, of which 136,694 clips remained recoverable. Our conclusions are therefore restricted to this continued-pretraining regime and do not extend to training on the full corpus. Consistent with these studies, which do not report variability across repeated training runs, we train each configuration once using a fixed seed. All runs begin from the same pretrained checkpoint and differ only in the small hypergraph module. We therefore assess the consistency of improvements across eleven benchmarks and two backbones, rather than quantify variability across repeated runs.
Simulated missing modalities. None of the compared backbones evaluates retrieval with missing modalities. We therefore adopt the protocol introduced in our conference version [12], which, consistent with prior robustness studies [11], removes one randomly selected stream per clip at test time. This controlled protocol ensures identical evaluation conditions across methods. However, it only approximates the structured patterns of missing data encountered in real-world collections, in which entire sources or recording conditions may be absent.
Checkpoint selection. Following the released training configurations of GRAM, TRIANGLE, PMRL, and HYPERGRAM, we select the checkpoint by validating on MSR-VTT, which is also one of our evaluation benchmarks. We never use the remaining ten benchmarks during training or model selection, and the gains on them are similar to those on MSR-VTT, indicating that our conclusions do not depend on this choice. A held-out validation set drawn from the training distribution would make the evaluation fully independent of model selection.
Conclusion
We presented a hypergraph refinement framework for multimodal retrieval that models information exchange among observed modalities within a candidate and across semantically related training candidates. The framework includes HYVOL as a special case and accommodates two additional configurations: WHYVOL, which weights modality contributions and semantic connections, and WRHG, which additionally distinguishes relation types and participant roles. Missing modalities are naturally excluded from the hypergraph without imputation. We restrict refinement to training, while inference retains the corresponding backbone’s encoder outputs and retrieval scoring function.
Evaluation on 11 zero-shot benchmarks demonstrates improvements across video–text and audio–text retrieval. With GRAM, WRHG improves over the backbone in all 17 text-to-video settings and all 17 video-to-text settings. With HYPERGRAM, it improves 16 of the 17 text-to-video settings and all 17 video-to-text settings. The missing-modality evaluation on seven video benchmarks further shows that the weighted configurations often retain their advantage when individual clips lose an observed stream.
The results also identify the boundaries of these benefits. No refinement configuration is uniformly strongest across datasets and retrieval directions. HYVOL remains competitive or superior in several settings, audio-only retrieval exhibits smaller and less consistent gains, and some incomplete-modality conditions favor the unrefined HYPERGRAM backbone. These findings support training-time hypergraph refinement as an effective approach to improving retrieval across evaluated settings, while showing that the value of weighting and relational modeling depends on the available information, backbone, and task.
CRediT authorship contribution statement
Anindya Nag: Conceptualization, Investigation, Methodology, Writing - Original draft preparation. Ambuj Mehrish: Methodology, Validation, Writing - Original draft preparation, Writing - Review & Editing. Sebastiano Vascon: Supervision, Methodology, Conceptualization, Writing - Review & Editing.
Funding
This work was supported by the DoE 2023-2027 (MUR, AIS.DIP.ECCELLENZA2023_27.FF project).
Acknowledgements
This work was supported by the European Union’s Horizon Europe research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 101205348 (CASPER). This work was supported by the DoE 2023-2027 (MUR, AIS.DIP.ECCELLENZA2023_27.FF project). We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium, through the EuroHPC AI Factories “AI for Science and Collaborative EU Projects” Access call (proposal No. EHPC-AIF2026SC01-041). We further acknowledge the CINECA award under the ISCRA initiative (Class C projects IsCd5-CASPER-A and IsCd5_GMEG), for the availability of high-performance computing resources and support.
Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Data availability
The code and trained weights of HYVOL, our conference model, are available here. The trained weights of WHYVOL and WRHG for both backbones are available here, and their code will be released upon publication. All datasets used in this work are publicly available from their original sources.
Use of Large Language Models
We used large language models only to assist with the writing, namely to correct grammar, refine wording and readability, and shorten the text where needed. The authors originated the research idea, method design, experimental protocol, analyses, and all reported results. The authors checked every suggested change against their experimental records and accept full responsibility for the content of this paper.
Reproducibility
We have made the code and trained weights of HyVol publicly available here. The trained weights of wHyVol and wRHG for both backbones are available here, and we will release the corresponding training and evaluation code, configuration files, and the scripts that produce our tables and figures upon publication. We use one configuration per variant for all eleven benchmarks (eight text–video and three text–audio retrieval tasks) and do not tune the learning rate, neighbourhood size, or schedule for any individual benchmark. We train all models on the same 150K-clip subset of VAST-27M, of which we could download 136,694 clips with both video and audio.
For GRAM, TRIANGLE and PMRL, we use the checkpoints made available by their authors. Since no checkpoint is available for HYPERGRAM, we train it ourselves with the authors’ public code and their published training recipe. Every evaluated method is initialised from the VAST foundation model and tested on identical test data. Published numbers from other works are reported only as reference values.
Implementation details
Table 8 lists the complete configuration of HyVol, wHyVol, and wRHG, complementing the summary in § 4.2.3. The three configurations share the encoders, optimisation schedule, and data settings, and differ only in how the hypergraph is weighted and propagated: wHyVol and wRHG learn vertex and pair weights, and wRHG additionally uses typed relational messages. The same values are used for both backbones and all eleven benchmarks without benchmark-specific tuning, and the hypergraph is used only during training.
Additional Results and Analysis
Table 9 reports text-to-video and video-to-text R@1 on seven video benchmarks as the share of gallery clips missing one modality grows from 0% to 90%. Recall falls for every method as clips become incomplete, yet the refined models generally remain clearly ahead of their backbones. On VALOR-32K video-to-text retrieval with GRAM, for example, wRHG leads by 9.1, 8.8, and 7.0 points at 25%, 50%, and 75% missingness, and its advantage persists at 6.6 points even when 90% of the clips lack one modality.
| Dataset | Method | T→V 0% | T→V 25% | T→V 50% | T→V 75% | T→V 90% | V→T 0% | V→T 25% | V→T 50% | V→T 75% | V→T 90% |
| MSR-VTT [37] (T–VAS) | GRAM⋆ | 51.8 | 49.2 | 44.0 | 41.4 | 39.5 | 50.1 | 46.5 | 42.4 | 38.1 | 36.4 |
| MSR-VTT [37] (T–VAS) | +HYVOL | 55.8 (+4.0) | 52.7 (+3.5) | 47.2 (+3.2) | 41.6 (+0.2) | 39.3 (−0.2) | 58.1 (+8.0) | 54.1 (+7.6) | 48.0 (+5.6) | 42.6 (+4.5) | 39.4 (+3.0) |
| MSR-VTT [37] (T–VAS) | +wHYVOL | 56.8 (+5.0) | 53.7 (+4.5) | 48.0 (+4.0) | 43.9 (+2.5) | 41.2 (+1.7) | 58.0 (+7.9) | 53.6 (+7.1) | 47.9 (+5.5) | 43.1 (+5.0) | 40.7 (+4.3) |
| MSR-VTT [37] (T–VAS) | +wRHG | 57.4 (+5.6) | 53.5 (+4.3) | 48.5 (+4.5) | 44.4 (+3.0) | 41.3 (+1.8) | 58.0 (+7.9) | 53.6 (+7.1) | 48.6 (+6.2) | 43.1 (+5.0) | 40.2 (+3.8) |
| MSR-VTT [37] (T–VAS) | HYPERGRAM† | 52.3 | 50.2 | 45.4 | 41.2 | 39.2 | 50.9 | 47.6 | 43.3 | 39.1 | 36.7 |
| MSR-VTT [37] (T–VAS) | +HYVOL | 56.1 (+3.8) | 52.3 (+2.1) | 47.2 (+1.8) | 42.3 (+1.1) | 39.9 (+0.7) | 59.2 (+8.3) | 54.7 (+7.1) | 47.9 (+4.6) | 42.6 (+3.5) | 39.5 (+2.8) |
| MSR-VTT [37] (T–VAS) | +wHYVOL | 57.0 (+4.7) | 53.2 (+3.0) | 48.9 (+3.5) | 43.3 (+2.1) | 40.8 (+1.6) | 58.8 (+7.9) | 54.3 (+6.7) | 47.9 (+4.6) | 42.5 (+3.4) | 40.2 (+3.5) |
| MSR-VTT [37] (T–VAS) | +wRHG | 57.3 (+5.0) | 53.7 (+3.5) | 48.9 (+3.5) | 43.8 (+2.6) | 41.6 (+2.4) | 59.3 (+8.4) | 54.3 (+6.7) | 48.3 (+5.0) | 43.5 (+4.4) | 40.9 (+4.2) |
| DiDeMo [38] (T–VA) | GRAM⋆ | 50.6 | 45.5 | 40.6 | 36.6 | 33.5 | 49.0 | 42.5 | 36.7 | 32.1 | 28.6 |
| DiDeMo [38] (T–VA) | +HYVOL | 54.6 (+4.0) | 47.5 (+2.0) | 42.4 (+1.8) | 38.1 (+1.5) | 35.2 (+1.7) | 54.1 (+5.1) | 47.5 (+5.0) | 41.1 (+4.4) | 36.1 (+4.0) | 31.2 (+2.6) |
| DiDeMo [38] (T–VA) | +wHYVOL | 53.8 (+3.2) | 47.2 (+1.7) | 42.0 (+1.4) | 38.0 (+1.4) | 34.0 (+0.5) | 55.5 (+6.5) | 47.1 (+4.6) | 40.5 (+3.8) | 35.1 (+3.0) | 31.3 (+2.7) |
| DiDeMo [38] (T–VA) | +wRHG | 54.1 (+3.5) | 46.9 (+1.4) | 41.5 (+0.9) | 36.9 (+0.3) | 33.8 (+0.3) | 55.0 (+6.0) | 47.4 (+4.9) | 40.8 (+4.1) | 35.9 (+3.8) | 31.8 (+3.2) |
| DiDeMo [38] (T–VA) | HYPERGRAM† | 50.4 | 45.0 | 39.9 | 36.6 | 32.8 | 50.8 | 43.7 | 37.6 | 33.6 | 29.9 |
| DiDeMo [38] (T–VA) | +HYVOL | 54.2 (+3.8) | 47.8 (+2.8) | 41.9 (+2.0) | 38.8 (+2.2) | 35.1 (+2.3) | 54.7 (+3.9) | 48.2 (+4.5) | 40.5 (+2.9) | 35.6 (+2.0) | 31.0 (+1.1) |
| DiDeMo [38] (T–VA) | +wHYVOL | 54.6 (+4.2) | 48.1 (+3.1) | 42.5 (+2.6) | 37.7 (+1.1) | 34.3 (+1.5) | 55.1 (+4.3) | 46.8 (+3.1) | 39.9 (+2.3) | 34.6 (+1.0) | 30.5 (+0.6) |
| DiDeMo [38] (T–VA) | +wRHG | 54.6 (+4.2) | 48.1 (+3.1) | 42.5 (+2.6) | 38.6 (+2.0) | 34.5 (+1.7) | 55.5 (+4.7) | 47.2 (+3.5) | 40.8 (+3.2) | 34.7 (+1.1) | 31.0 (+1.1) |
| ActivityNet [39] (T–VA) | GRAM⋆ | 56.4 | 50.4 | 44.4 | 38.1 | 34.2 | 49.7 | 43.9 | 38.4 | 31.7 | 28.2 |
| ActivityNet [39] (T–VA) | +HYVOL | 56.9 (+0.5) | 49.0 (−1.4) | 43.5 (−0.9) | 37.8 (−0.3) | 34.2 (0.0) | 56.6 (+6.9) | 48.5 (+4.6) | 41.6 (+3.2) | 32.9 (+1.2) | 29.0 (+0.8) |
| ActivityNet [39] (T–VA) | +wHYVOL | 60.8 (+4.4) | 54.3 (+3.9) | 47.7 (+3.3) | 40.9 (+2.8) | 36.5 (+2.3) | 62.7 (+13.0) | 53.8 (+9.9) | 44.9 (+6.5) | 36.3 (+4.6) | 31.2 (+3.0) |
| ActivityNet [39] (T–VA) | +wRHG | 60.7 (+4.3) | 54.1 (+3.7) | 47.9 (+3.5) | 40.7 (+2.6) | 36.6 (+2.4) | 63.0 (+13.3) | 54.2 (+10.3) | 45.4 (+7.0) | 36.7 (+5.0) | 31.7 (+3.5) |
| ActivityNet [39] (T–VA) | HYPERGRAM† | 56.8 | 50.9 | 45.2 | 38.9 | 35.2 | 50.9 | 44.6 | 39.1 | 32.6 | 28.9 |
| ActivityNet [39] (T–VA) | +HYVOL | 57.8 (+1.0) | 48.7 (−2.2) | 43.7 (−1.5) | 38.0 (−0.9) | 34.1 (−1.1) | 56.4 (+5.5) | 48.7 (+4.1) | 41.3 (+2.2) | 32.6 (0.0) | 28.3 (−0.6) |
| ActivityNet [39] (T–VA) | +wHYVOL | 60.5 (+3.7) | 54.2 (+3.3) | 48.0 (+2.8) | 40.9 (+2.0) | 36.5 (+1.3) | 62.8 (+11.9) | 53.7 (+9.1) | 45.0 (+5.9) | 36.6 (+4.0) | 31.4 (+2.5) |
| ActivityNet [39] (T–VA) | +wRHG | 60.9 (+4.1) | 54.5 (+3.6) | 48.3 (+3.1) | 41.1 (+2.2) | 36.8 (+1.6) | 63.0 (+12.1) | 53.9 (+9.3) | 45.0 (+5.9) | 36.4 (+3.8) | 31.3 (+2.4) |
| VATEX [40] (T–VAS) | GRAM⋆ | 88.4 | 80.7 | 74.5 | 67.7 | 64.5 | 86.5 | 79.1 | 73.5 | 67.3 | 64.0 |
| VATEX [40] (T–VAS) | +HYVOL | 94.0 (+5.6) | 85.8 (+5.1) | 78.0 (+3.5) | 70.8 (+3.1) | 66.4 (+1.9) | 93.8 (+7.3) | 84.9 (+5.8) | 77.3 (+3.8) | 69.8 (+2.5) | 65.2 (+1.2) |
| VATEX [40] (T–VAS) | +wHYVOL | 93.5 (+5.1) | 84.9 (+4.2) | 77.0 (+2.5) | 70.5 (+2.8) | 65.7 (+1.2) | 94.4 (+7.9) | 86.1 (+7.0) | 78.4 (+4.9) | 71.5 (+4.2) | 67.5 (+3.5) |
| VATEX [40] (T–VAS) | +wRHG | 94.0 (+5.6) | 85.6 (+4.9) | 77.3 (+2.8) | 71.0 (+3.3) | 68.4 (+3.9) | 94.4 (+7.9) | 86.1 (+7.0) | 78.4 (+4.9) | 71.2 (+3.9) | 67.5 (+3.5) |
| VATEX [40] (T–VAS) | HYPERGRAM† | 89.8 | 81.7 | 74.7 | 67.3 | 63.8 | 87.8 | 80.5 | 74.7 | 69.4 | 65.9 |
| VATEX [40] (T–VAS) | +HYVOL | 93.3 (+3.5) | 85.4 (+3.7) | 78.2 (+3.5) | 70.8 (+3.5) | 66.8 (+3.0) | 93.9 (+6.1) | 83.5 (+3.0) | 74.9 (+0.2) | 68.2 (−1.2) | 63.3 (−2.6) |
| VATEX [40] (T–VAS) | +wHYVOL | 93.5 (+3.7) | 85.8 (+4.1) | 77.3 (+2.6) | 68.9 (+1.6) | 65.2 (+1.4) | 94.0 (+6.2) | 84.7 (+4.2) | 76.8 (+2.1) | 70.8 (+1.4) | 65.9 (0.0) |
| VATEX [40] (T–VAS) | +wRHG | 93.5 (+3.7) | 85.2 (+3.5) | 77.0 (+2.3) | 70.5 (+3.2) | 67.5 (+3.7) | 94.0 (+6.2) | 85.4 (+4.9) | 77.3 (+2.6) | 71.7 (+2.3) | 68.2 (+2.3) |
| Shot2Story [41] (T–VA) | GRAM⋆ | 70.5 | 62.4 | 54.0 | 46.3 | 42.2 | 66.2 | 58.2 | 50.7 | 42.9 | 37.9 |
| Shot2Story [41] (T–VA) | +HYVOL | 74.8 (+4.3) | 66.4 (+4.0) | 58.5 (+4.5) | 50.2 (+3.9) | 45.2 (+3.0) | 77.6 (+11.4) | 67.1 (+8.9) | 56.9 (+6.2) | 47.6 (+4.7) | 41.9 (+4.0) |
| Shot2Story [41] (T–VA) | +wHYVOL | 76.3 (+5.8) | 66.7 (+4.3) | 57.5 (+3.5) | 49.2 (+2.9) | 44.4 (+2.2) | 75.7 (+9.5) | 66.2 (+8.0) | 56.3 (+5.6) | 47.1 (+4.2) | 41.5 (+3.6) |
| Shot2Story [41] (T–VA) | +wRHG | 77.4 (+6.9) | 68.3 (+5.9) | 59.0 (+5.0) | 50.5 (+4.2) | 45.0 (+2.8) | 76.3 (+10.1) | 66.8 (+8.6) | 56.7 (+6.0) | 47.8 (+4.9) | 42.0 (+4.1) |
| Shot2Story [41] (T–VA) | HYPERGRAM† | 75.0 | 67.4 | 58.5 | 50.6 | 45.2 | 73.1 | 65.6 | 56.4 | 47.7 | 42.3 |
| Shot2Story [41] (T–VA) | +HYVOL | 74.6 (−0.4) | 66.5 (−0.9) | 58.4 (−0.1) | 50.4 (−0.2) | 44.8 (−0.4) | 78.1 (+5.0) | 67.4 (+1.8) | 57.1 (+0.7) | 48.0 (+0.3) | 42.2 (−0.1) |
| Shot2Story [41] (T–VA) | +wHYVOL | 75.9 (+0.9) | 67.3 (−0.1) | 58.5 (0.0) | 50.2 (−0.4) | 44.9 (−0.3) | 76.2 (+3.1) | 66.4 (+0.8) | 56.5 (+0.1) | 47.6 (−0.1) | 41.6 (−0.7) |
| Shot2Story [41] (T–VA) | +wRHG | 76.6 (+1.6) | 67.8 (+0.4) | 59.5 (+1.0) | 50.7 (+0.1) | 45.2 (0.0) | 76.0 (+2.9) | 66.2 (+0.6) | 56.5 (+0.1) | 47.6 (−0.1) | 41.5 (−0.8) |
| FLARE [43] (T–VA) | GRAM⋆ | 46.0 | 41.6 | 37.2 | 32.0 | 30.0 | 46.8 | 41.6 | 35.9 | 29.4 | 25.9 |
| FLARE [43] (T–VA) | +HYVOL | 52.0 (+6.0) | 47.6 (+6.0) | 41.9 (+4.7) | 36.3 (+4.3) | 33.4 (+3.4) | 55.5 (+8.7) | 48.6 (+7.0) | 40.6 (+4.7) | 33.2 (+3.8) | 28.8 (+2.9) |
| FLARE [43] (T–VA) | +wHYVOL | 51.7 (+5.7) | 45.6 (+4.9) | 39.7 (+2.5) | 34.9 (+2.9) | 32.4 (+2.4) | 54.6 (+7.8) | 47.7 (+6.1) | 40.5 (+4.6) | 33.1 (+3.7) | 28.9 (+3.0) |
| FLARE [43] (T–VA) | +wRHG | 53.9 (+7.9) | 48.3 (+6.7) | 41.9 (+4.7) | 36.5 (+4.5) | 33.8 (+3.8) | 55.0 (+8.2) | 48.1 (+6.5) | 40.7 (+4.8) | 33.0 (+3.6) | 29.1 (+3.2) |
| FLARE [43] (T–VA) | HYPERGRAM† | 52.9 | 49.7 | 43.9 | 38.2 | 33.6 | 52.6 | 48.3 | 41.4 | 34.4 | 28.6 |
| FLARE [43] (T–VA) | +HYVOL | 52.1 (−0.8) | 47.6 (−2.1) | 42.4 (−1.5) | 36.7 (−1.5) | 34.1 (+0.5) | 56.4 (+3.8) | 48.9 (+1.5) | 41.5 (+0.1) | 33.7 (−0.7) | 29.4 (+0.8) |
| FLARE [43] (T–VA) | +wHYVOL | 52.1 (−0.8) | 47.4 (−2.3) | 41.4 (−2.5) | 36.2 (−2.0) | 33.7 (+0.1) | 55.9 (+3.3) | 48.7 (+0.4) | 41.1 (−0.3) | 33.2 (−1.2) | 29.2 (+0.6) |
| FLARE [43] (T–VA) | +wRHG | 52.8 (−0.1) | 47.9 (−1.8) | 42.7 (−1.2) | 37.3 (−0.9) | 34.1 (+0.5) | 55.8 (+3.2) | 48.7 (+0.4) | 40.8 (−0.6) | 33.1 (−1.3) | 29.1 (+0.5) |
| VALOR-32K [5] (T–VA) | GRAM⋆ | 75.2 | 65.6 | 56.1 | 47.2 | 41.9 | 71.2 | 63.0 | 54.0 | 45.4 | 40.4 |
| VALOR-32K [5] (T–VA) | +HYVOL | 80.5 (+5.3) | 71.2 (+5.6) | 62.2 (+6.1) | 53.3 (+6.1) | 47.4 (+5.5) | 81.6 (+10.4) | 72.0 (+9.0) | 62.0 (+8.0) | 52.0 (+6.6) | 46.6 (+6.2) |
| VALOR-32K [5] (T–VA) | +wHYVOL | 80.4 (+5.2) | 70.9 (+5.3) | 61.2 (+5.1) | 51.8 (+4.6) | 47.5 (+5.6) | 81.1 (+9.9) | 71.3 (+8.3) | 62.2 (+8.2) | 51.6 (+6.2) | 46.4 (+6.0) |
| VALOR-32K [5] (T–VA) | +wRHG | 80.8 (+5.6) | 71.4 (+5.8) | 61.7 (+5.6) | 53.3 (+6.1) | 48.0 (+6.1) | 80.8 (+9.6) | 72.1 (+9.1) | 62.8 (+8.8) | 52.4 (+7.0) | 47.0 (+6.6) |
| VALOR-32K [5] (T–VA) | HYPERGRAM† | 75.9 | 67.2 | 57.6 | 48.2 | 43.5 | 72.4 | 65.1 | 56.2 | 47.3 | 42.1 |
| VALOR-32K [5] (T–VA) | +HYVOL | 80.9 (+5.0) | 71.3 (+4.1) | 62.5 (+4.9) | 53.5 (+5.3) | 46.7 (+4.1) | 82.2 (+9.8) | 72.5 (+7.4) | 62.5 (+6.3) | 52.5 (+5.3) | 46.9 (+4.8) |
| VALOR-32K [5] (T–VA) | +wHYVOL | 80.5 (+4.6) | 71.0 (+3.8) | 60.9 (+3.3) | 51.5 (+3.3) | 45.5 (+2.0) | 82.0 (+9.6) | 72.2 (+7.1) | 62.6 (+6.4) | 52.1 (+4.8) | 46.6 (+4.5) |
| VALOR-32K [5] (T–VA) | +wRHG | 80.9 (+5.0) | 71.5 (+4.3) | 62.2 (+4.6) | 52.6 (+4.4) | 46.7 (+3.2) | 81.8 (+9.4) | 72.4 (+7.3) | 62.9 (+6.7) | 52.5 (+5.2) | 47.1 (+5.0) |
Table 9. Retrieval under missing modalities (R@1) on the seven video benchmarks. A fraction of the gallery clips loses one randomly chosen modality; the per-clip masks are fixed and identical across methods, and reproduces the main protocol. MSR-VTT and VATEX use the T–VAS modality set, and the other benchmarks use T–VA. Numbers in parentheses give the change with respect to the backbone (GRAM or HYPERGRAM); green marks an improvement, red a drop, and gray no change.
References
- [1]A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, in: International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, 2021, pp. 8748–8763.arxiv.org/abs/2103.00020
- [2]R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, I. Misra, ImageBind: One embedding space to bind them all, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 15180–15190.arxiv.org/abs/2305.05665
- [3]B. Zhu, B. Lin, M. Ning, Y. Yan, J. Cui, H. Wang, Y. Pang, W. Jiang, J. Zhang, Z. Li, et al., LanguageBind: Extending video-language pretraining to n-modality by language-based semantic alignment, in: International Conference on Learning Representations, 2024.arxiv.org/abs/2310.01852
- [4]S. Chen, H. Li, Q. Wang, Z. Zhao, M. Sun, X. Zhu, J. Liu, VAST: A vision-audio-subtitle-text omni-modality foundation model and dataset, Advances in Neural Information Processing Systems 36 (2023) 72842–72866.arxiv.org/abs/2305.18500
- [5]J. Liu, S. Chen, X. He, L. Guo, X. Zhu, W. Wang, J. Tang, VALOR: Vision-audio-language omni-perception pre-training model and dataset, IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (2025) 708–724.
- [6]G. Cicchetti, E. Grassucci, L. Sigillo, D. Comminello, Gramian multimodal representation learning and alignment, in: International Conference on Learning Representations, 2025.arxiv.org/abs/2412.11959
- [7]G. Cicchetti, E. Grassucci, D. Comminiello, A TRI-ANGLE enables multimodal alignment beyond cosine similarity, Advances in Neural Information Processing Systems 38 (2025) 124144–124162.DOI
- [8]X. Liu, X. Xia, S.-K. Ng, T.-S. Chua, Principled multimodal representation learning, IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (2026) 9114–9128.DOI
- [9]S. Na, F. Jiang, Q. Zhou, W. Zhong, T. M. Dang, Y. Guo, H. Ma, C. Li, W. An, J. Huang, Hyperbolic Gramian volumes for multimodal alignment, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 37756–37765.
- [10]M. Ma, J. Ren, L. Zhao, S. Tulyakov, C. Wu, X. Peng, SMIL: Multimodal learning with severely missing modality, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021, pp. 2302–2310.arxiv.org/abs/2103.05677
- [11]M. Ma, J. Ren, L. Zhao, D. Testuggine, X. Peng, Are multimodal transformers robust to missing modality?, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2022, pp. 18156–18165.arxiv.org/abs/2204.05454
- [12]A. Nag, A. Mehrish, S. Vascon, Hypergraph-regularized Gramian volumes for multimodal retrieval, in: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2027. ArXiv:2609.15320.arxiv.org/abs/2609.15320
- [13]S. Brody, U. Alon, E. Yahav, How attentive are graph attention networks?, in: International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2105.14491
- [14]Q. Sun, Y. Fang, L. Wu, X. Wang, Y. Cao, EVA-CLIP: Improved training techniques for CLIP at scale, ArXiv preprint: arXiv:2303.15389 (2023).arxiv.org/abs/2303.15389
- [15]Z. Zeng, Y. Ge, Z. Tong, X. Liu, S.-T. Xia, Y. Shan, TVSTv2: Learning out-of-the-box spatiotemporal visual representations at scale, ArXiv preprint: arXiv:2305.14173 (2023).arxiv.org/abs/2305.14173
- [16]Q. Ye, G. Xu, M. Yan, H. Xu, Q. Qian, J. Zhang, F. Huang, HiTeA: Hierarchical temporal-aware video-language pre-training, in: IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 15359–15370.arxiv.org/abs/2212.14546
- [17]Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang, S. Xing, G. Chen, J. Pan, J. Yu, Y. Wang, L. Wang, Y. Qiao, InternVideo: General video foundation models via generative and discriminative learning, ArXiv preprint: arXiv:2212.03191 (2022).arxiv.org/abs/2212.03191
- [18]H. Xu, Q. Ye, M. Yan, Y. Shi, J. Ye, Y. Xu, C. Li, B. Bi, Q. Qian, W. Wang, et al., mPLUG-2: A modularized multi-modal foundation model across text, image and video, in: International Conference on Machine Learning, PMLR, 2023, pp. 38728–38748.arxiv.org/abs/2302.00402
- [19]J. Wang, D. Chen, Z. Wu, C. Luo, L. Zhou, Y. Zhao, Y. Xie, C. Liu, Y.-G. Jiang, L. Yuan, OmniVL: One foundation model for image-language and video-language tasks, in: Advances in Neural Information Processing Systems, volume 35, 2022, pp. 5696–5710.arxiv.org/abs/2209.07526
- [20]L. Zhao, N. B. Gundavarapu, L. Yuan, H. Zhou, S. Yan, J. J. Sun, L. Friedman, R. Qian, T. Weyand, Y. Zhao, R. Hornung, F. Schroff, M.-H. Yang, D. A. Ross, H. Wang, H. Adam, M. Sirotenko, T. Liu, B. Gong, VideoPrism: A foundational visual encoder for video understanding, in: International Conference on Machine Learning, 2024, pp. 60785–60811.arxiv.org/abs/2402.13217
- [21]A. Nagrani, P. H. Seo, B. Seybold, A. Hauth, S. Manén, C. Sun, C. Schmid, Learning audio-video modalities from image captions, in: European Conference on Computer Vision, 2022, pp. 407–426.arxiv.org/abs/2204.00679
- [22]Y. Zhao, J. Hessel, Y. Yu, X. Lu, R. Zellers, Y. Choi, Connecting the dots between audio and text without parallel data through visual knowledge transfer, in: Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2022, pp. 4492–4507.arxiv.org/abs/2112.08995
- [23]J. Li, R. R. Selvaraju, A. D. Gotmare, S. R. Joty, C. Xiong, S. C. H. Hoi, Align before fuse: Vision and language representation learning with momentum distillation, in: Advances in Neural Information Processing Systems, volume 34, 2021, pp. 9694–9705.
- [24]Y. Feng, H. You, Z. Zhang, R. Ji, Y. Gao, Hypergraph neural networks, in: AAAI Conference on Artificial Intelligence, 2019, pp. 3558–3565.arxiv.org/abs/1809.09401
- [25]E. Chien, C. Pan, J. Peng, O. Milenkovic, You are AllSet: A multiset function framework for hypergraph neural networks, in: International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2106.13264
- [26]M. Li, S. Zhou, Y. Chen, C. Huang, Y. Jiang, EduCross: Dual adversarial bipartite hypergraph learning for cross-modal retrieval in multimodal educational slides, Information Fusion 109 (2024) 102428. doi:10.1016/j.inffus.2024.102428.DOI
- [27]P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio, Graph attention networks, in: International Conference on Learning Representations (ICLR), 2018.arxiv.org/abs/1710.10903
- [28]Z. Zhong, L. Zheng, D. Cao, S. Li, Re-ranking person re-identification with k-reciprocal encoding, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1318–1327.arxiv.org/abs/1701.08398
- [29]S.-V. Bogolin, I. Croitoru, H. Jin, Y. Liu, S. Albanie, Cross modal retrieval with querybank normalisation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5194–5205.arxiv.org/abs/2112.12777
- [30]M. Schlichtkrull, T. N. Kipf, P. Bloem, R. van den Berg, I. Titov, M. Welling, Modeling relational data with graph convolutional networks, in: The Semantic Web (ESWC), 2018, pp. 593–607.arxiv.org/abs/1703.06103
- [31]B. Fatemi, P. Taslakian, D. Vazquez, D. Poole, Knowledge hypergraphs: Prediction beyond binary relations, in: International Joint Conference on Artificial Intelligence (IJCAI), 2020, pp. 2191–2197.
- [32]M. Galkin, P. Trivedi, G. Maheshwari, R. Usbeck, J. Lehmann, Message passing for hyper-relational knowledge graphs, in: Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 7346–7359.arxiv.org/abs/2009.10847
- [33]N. Yadati, Neural message passing for multi-relational ordered and recursive hypergraphs, in: Advances in Neural Information Processing Systems (NeurIPS), 2020.
- [34]X. Huang, M. Romero Orth, P. Barceló, M. M. Bronstein, I. i. Ceylan, Link prediction with relational hypergraphs, arXiv preprint arXiv:2402.04062 (2024).arxiv.org/abs/2402.04062
- [35]J. Hu, Y. Liu, J. Zhao, Q. Jin, MMGCN: Multimodal fusion via deep graph convolution network for emotion recognition in conversation, in: Annual Meeting of the Association for Computational Linguistics (ACL), 2021, pp. 5666–5675.arxiv.org/abs/2107.06779
- [36]J. Yang, Y. Wang, R. Yi, Y. Zhu, A. Rehman, A. Zadeh, S. Poria, L.-P. Morency, MTAG: Modal-temporal attention graph for unaligned human multimodal language sequences, in: Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021, pp. 1009–1021.arxiv.org/abs/2010.11985
- [37]J. Xu, T. Mei, T. Yao, Y. Rui, MSR-VTT: A large video description dataset for bridging video and language, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 5288–5296.
- [38]L. A. Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, B. Russell, Localizing moments in video with natural language, in: IEEE International Conference on Computer Vision (ICCV), 2017, pp. 5804–5813.arxiv.org/abs/1708.01641
- [39]F. Caba Heilbron, V. Escorcia, B. Ghanem, J. Carlos Niebles, ActivityNet: A large-scale video benchmark for human activity understanding, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 961–970.DOI
- [40]X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, W. Y. Wang, VaTeX: A large-scale, high-quality multilingual dataset for video-and-language research, in: IEEE/CVF International Conference on Computer Vision, 2019, pp. 4581–4591.
- [41]M. Han, L. Yang, X. Chang, H. Wang, Shot2Story20K: A new benchmark for comprehensive understanding of multi-shot videos, arXiv preprint arXiv:2312.10300 (2023).
- [42]D. L. Chen, W. B. Dolan, Collecting highly parallel data for paraphrase evaluation, in: Annual Meeting of the Association for Computational Linguistics (ACL), 2011, pp. 190–200.
- [43]Q. You, H. Liang, M. Chen, B. Zeng, M. Qiang, Z. Wong, W. Zhang, FLARE: Full-modality long-video audiovisual retrieval benchmark with user-simulated queries, arXiv preprint arXiv:2605.10228 (2026).arxiv.org/abs/2605.10228
- [44]C. D. Kim, B. Kim, H. Lee, G. Kim, AudioCaps: Generating Captions for Audios in The Wild, in: NAACL-HLT, 2019, pp. 119–132.DOI
- [45]K. Drossos, S. Lipping, T. Virtanen, Clotho: An audio captioning dataset, in: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 736–740.arxiv.org/abs/1910.09387
- [46]A. S. Koepke, A.-M. Oncescu, J. F. Henriques, Z. Akata, S. Albanie, Audio retrieval with natural language queries: A benchmark study, IEEE Transactions on Multimedia 25 (2023) 2675–2685.
- [47]Y. Lin, J. Zhang, Z. Huang, J. Liu, Z. Wen, X. Peng, Multi-granularity correspondence learning from long-term noisy videos, in: International Conference on Learning Representations, 2024.arxiv.org/abs/2401.16702
- [48]K. Li, Y. Wang, Y. Li, Y. Wang, Y. He, L. Wang, Y. Qiao, Unmasked teacher: Towards training-efficient video foundation models, in: IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 19891–19903.arxiv.org/abs/2303.16058
- [49]Y. Wang, Y. He, Y. Li, K. Li, J. Yu, X. Ma, X. Li, G. Chen, X. Chen, Y. Wang, P. Luo, Z. Liu, Y. Wang, L. Wang, Y. Qiao, InterVideo: A large-scale video-text dataset for multimodal understanding and generation, in: International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2307.06942
- [50]S. Yan, T. Zhu, Z. Wang, Y. Cao, M. Zhang, S. Ghosh, Y. Wu, J. Yu, VideoCoCa: Video-text modeling with zero-shot transfer from contrastive captioners, 2022. ArXiv preprint: arXiv:2212.04979.arxiv.org/abs/2212.04979






