betaFeedback

WorksComputer Science

BrainBind: Bridging Synchronized EEG and fMRI through Long-Form Naturalistic Video Representations

Abstract

Real-world vision is dynamic and scene-rich, while EEG and fMRI resolve it at complementary temporal and spatial scales. Most neural decoding systems, however, align one modality in isolation and mostly against static images or short clips. We study the synchronized case: EEG and fMRI are recorded during the same long-form naturalistic viewing, giving a shared stimulus but also recurring narrative context and many semantically similar candidates. We present BrainBind, a framework that maps synchronized EEG and fMRI into frozen long-form video embedding spaces and treats those spaces as common visual-semantic references, not literal neural codes. On CineBrain, a five-seed Recall@K 2 x 4 matrix across two video anchors and four system configurations shows distinct retrieval profiles. With Omni, fMRI organizes the full candidate list most strongly (MRR = 23.82, nDCG@10 = 25.31), while Joint-Dual gives the highest mean label-aware R@1 (13.52%) with a distinct rank-wide profile. The complementary episode-disjoint evaluation quantifies the contribution of repeated stimulus identity and narrative context to the random split. Across these experiments, proxy choice and system configuration produce distinct top-rank and rank-wide retrieval profiles within the present synchronized dataset. The code is available at https://github.com/Forainest789/Brainbind.

Introduction

Foundation visual representations have become an effective interface between brain activity and natural stimuli [5, 39, 41, 45]. Recent systems align fMRI or EEG with pretrained visual-semantic spaces before retrieval or reconstruction [2, 3, 10, 14, 29, 31, 32, 33]. EEG has fast temporal resolution but weak spatial localization; fMRI localizes cortical regions but integrates slow hemodynamic responses. These complementary measurements are nevertheless usually examined separately. This alignment strategy replaces a difficult pixel-level target with a structured representation learned from large-scale visual data. When EEG and fMRI are recorded synchronously during the same stimulus, their alignment to a common reference space tests complementary measurements while holding the dataset, task, and stimulus set fixed.

Long-form naturalistic video makes this question both useful and difficult. Nearby clips can share characters, locations, actions, and narrative context [17, 18]; consequently, a random sample split may test fine-grained discrimination within recurring context as much as generalization to a new episode. The candidate pool is also semantically dense, so top-rank retrieval and rank-wide organization need not tell the same story. We therefore separate these views with Recall@K, MRR, mAP, and nDCG, and pair the standard random split with correspondence controls and an episode-disjoint evaluation.

We introduce BrainBind, a representation-and-evaluation framework for synchronized EEG and fMRI under long-form naturalistic viewing (Figure 1). BrainBind maps each neural input to a frozen video embedding space and treats that space as a shared visual-semantic reference geometry. The reference makes modality inputs, system configurations, and proxy anchors comparable under the same stimulus and retrieval protocol; it is not presented as a literal neural code. We evaluate BrainBind on CineBrain [11], using a five-seed Recall@K 2×42 \times4 matrix spanning two video anchors and four system configurations, together with geometry diagnostics, targeted S4AE analyses, correspondence controls, an episode-disjoint split, a three-seed loss/temperature ablation, and selective two-subject held-out-training checks.

Overview of BrainBind

Figure 1. Overview of BrainBind. Synchronized EEG and fMRI responses are mapped into a frozen long-form video proxy space, which provides a shared visual-semantic geometry for comparing modality-specific and fused neural–video retrieval. The dashed reconstruction path denotes the optional L1L_1 reconstruction loss LrecL_{\mathrm{rec}} used only in designated diagnostic configurations; the core systems use contrastive-only training. The rightmost branch is retrieval-conditioned text-to-video visualization rather than direct neural reconstruction.

On the random split, fMRI with the Omni anchor produces the strongest MRR, mAP, and nDCG@10, indicating the strongest rank-wide organization. The complete Joint-Dual system instead produces the highest mean R@1R@1 while remaining below fMRI on those rank-wide metrics, revealing a precision–breadth trade-off. EEG-only alignment remains above control levels and exhibits a retrieval profile distinct from fMRI in the same reference space. The choice of proxy also matters: spectral and cross-anchor analyses associate the broader Omni geometry with stronger neural alignment. The sharp drop under episode-disjoint evaluation then exposes the contribution of recurring narrative context to random-split retrieval.

Contributions. BrainBind is a representation and evaluation framework for synchronized EEG–fMRI alignment under long-form naturalistic video. Its contributions are threefold:

Shared neural–video proxy formulation. We cast naturalistic neural decoding as stimulus-aligned neural-to-proxy representation learning, using frozen long-form video embeddings as a common visual-semantic geometry that organizes heterogeneous neural measurements without assuming literal proxy-to-neural correspondence.

Proxy and system-configuration study. Across two video anchors, four system configurations, and five Recall@K seeds, we show that proxy choice and complete-system design yield distinct top-rank and rank-wide profiles: fMRI gives the strongest rank-wide organization, whereas Joint-Dual gives the highest mean label-aware R@1R@1 on the random split.

Ranking-aware evaluation and geometry diagnostics. We complement Recall@K with MRR, mAP, and nDCG@10, and augment retrieval evaluation with random-brain and shuffled controls, a three-seed loss and temperature ablation, selective held-out training checks, episode disjoint evaluation, embedding-geometry diagnostics, and qualitative retrieval conditioned visualization, enabling joint analysis of top-rank retrieval and rank-wide organization.

Related Work

fMRI decoding for static and dynamic visual stimuli. A large body of work aligns fMRI to pretrained visual or semantic embeddings as a precursor to retrieval or generation. For static images, MindEye [31] and MindEye2 [32] align fMRI to CLIP-like embeddings and apply diffusion priors for reconstruction on large-scale fMRI datasets [1]; related systems explore semantic alignment [26, 34, 42] and low-data regimes [32]. For dynamic stimuli, Mind-Video [7], NeuroClips [14], Brain Netflix [9], Animate Your Thoughts [29] extend this paradigm to fMRI-driven video synthesis with diffusion-based decoders. These systems primarily study fMRI in isolation; BrainBind instead asks how synchronized EEG and fMRI behave under the same long-form video geometry.

EEG decoding into visual representation spaces. EEG-based systems leverage visual and textual embedding spaces for image generation, classification, and retrieval. DreamDiffusion [2] synthesizes images from EEG via diffusion priors; BrainVis [10] and EEG-CLIP [3] bridge EEG to CLIP-like geometries. NICE-EEG [33] establishes zero-shot EEG-to-image retrieval on THINGS-EEG [12], and ATM [24] introduces adaptive masked transformers for EEG-visual decoding. Self-supervised EEG pretraining [6, 21, 23] provides general-purpose temporal backbones. These studies are predominantly single-modality and often use static-image stimuli. BrainBind addresses the controlled interaction of synchronized EEG, fMRI, and frozen video proxies under long-form dynamic stimuli.

Multimodal foundation embeddings as stimulus geometries. Contrastive representation learning organizes heterogeneous modalities in shared metric spaces [4, 19, 30, 35], and ImageBind [13] extends this principle to image, video, audio, text, and other sensory signals. Retrieval-oriented video embeddings, including Omni-style models [8, 44] and temporally rich video encoders [38, 43], encode objects, actions, scene layout, and temporal continuity. For neural decoding, such embeddings serve not only as supervision targets but also as reference geometries with different spectral and neighborhood structure. BrainBind explicitly compares proxy anchors and treats their geometry as an experimental design variable, relating retrieval differences to PCA spectral statistics and CCA analysis without equating either proxy with a neural code.

Temporal modeling, contrastive learning, and ranking metrics. Transformers [36] and structured state-space models [15, 16] provide complementary inductive biases for sequence modeling, and InfoNCE [4, 19, 35] is the standard contrastive alignment objective, with supervised and factorized extensions for richer positive structure [22, 25, 40]. BrainBind uses these established components as controlled implementations. Its targeted S4AE analyses examine state-space configurations alongside Transformer encoders; they are not isolated tests of temporal mixing because capacity and, in one diagnostic, the reconstruction objective also change. Beyond Recall@K, ranking-aware metrics such as MRR, mAP, and nDCG [20] are essential for naturalistic retrieval where semantically proximate candidates require analysis of the full ranked list in addition to top-1 accuracy.

Method

Problem formulation. Each sample is a tuple (XEEG,XfMRI,V)(\mathbf{X}^{\mathrm{EEG}}, \mathbf{X}^{\mathrm{fMRI}}, \mathbf{V}), where XEEG∈RC×T\mathbf{X}^{\mathrm{EEG}} \in\mathbb{R}^{C \times T} is an EEG window, XfMRI\mathbf{X}^{\mathrm{fMRI}} is the corresponding fMRI observation after temporal alignment for the hemodynamic response, and V\mathbf{V} is the matched video. The goal is to learn modality-specific encoders that map EEG and fMRI into a common video-grounded embedding space supporting bidirectional neural-to-video and video-to-neural retrieval.

Long-form video proxy space. For anchor aa (Omni or ImageBind), a frozen video encoder FaF_a produces a pooled clip embedding. We retain its native dimension and apply ℓ2\ell_2 normalization:

zS=norm⁡2(pool⁡(Fa(V))).(1)\mathbf{z}^{S} = \operatorname{norm}_2(\operatorname{pool}(F_a(\mathbf{V}))). \tag*{(1)}

The neural output mapping matches the selected native target space (2,048-D Omni or 1,024-D ImageBind); no video-side cross-anchor projection is used. Thus the native proxy geometry analyzed below is also the geometry used for contrastive alignment. We use zS\mathbf{z}^{S} solely as a structured stimulus reference. The experiment tests whether its geometric organization provides a useful inductive bias for neural alignment; individual proxy dimensions carry no assumed neural meaning.

Frozen reference design. We keep the video proxy encoder frozen throughout training. This makes the proxy geometry an experimental variable rather than a trainable component, and it isolates changes in neural-to-proxy alignment from changes in the reference space itself. The stable reference delegates subject-specific adaptation to the neural encoders and preserves comparability across modalities, fusion designs, and anchors.

Neural encoders. Each modality m∈{EEG,fMRI}m \in\{\mathrm{EEG}, \mathrm{fMRI}\} is encoded by a modality-specific encoder EmE_m followed by a projection head gmg_m, producing an ℓ2\ell_2-normalized embedding zm\mathbf{z}^m. EEG-only, fMRI-only, and Both use Transformer encoders [36] with 512 hidden units, 8 heads, GELU, dropout 0.1, and pre-norm. In Both, each modality is encoded and mean-pooled before one cross-attention module fuses the pair ahead of the final proxy-space mapping.

Joint-Dual uses two independent S4AE encoders with mean fusion. Each S4AE branch applies a TemporalMLP and stacked blocks that pair a structured state-space time mixer [15] with a channel-mixing MLP, both under residual connections and layer normalization, before bottleneck projection. A mirrored decoder reconstructs the input when reconstruction is enabled. For EEG these blocks mix along the sampled time axis; in the fMRI configuration the five TRs are input channels and the structured mixer operates along the voxel axis. The primary matrix is therefore a controlled system comparison across anchors and complete system configurations, not a backbone-matched fusion ablation.

Proxy alignment and multimodal fusion. For a batch of BB matched pairs, define the directed InfoNCE term

ℓ(m,S)=−1B∑i=1Blog⁡exp⁡(zimTziS/τ)∑j=1Bexp⁡(zimTzjS/τ)(2)\ell(m,S) = -\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(z_i^{m\mathsf{T}}z_i^S/\tau)}{\sum_{j=1}^{B}\exp(z_i^{m\mathsf{T}}z_j^S/\tau)} \tag*{(2)}

and optimize the symmetric loss LNCE(m,S)=12[ℓ(m,S)+ℓ(S,m)]\mathcal{L}_{\mathrm{NCE}}(m,S) = \frac{1}{2}[\ell(m,S) + \ell(S,m)] with τ=0.05\tau= 0.05. The core matrix uses paired positives during training; the SupCon ablations use all samples sharing the same stim_label as positives. Joint-Dual aligns the mean-fused embedding and both modality-specific embeddings with the same video anchor:

LJD=λfLNCE(fused,S)+λm(LNCE(EEG,S)+LNCE(fMRI,S)),(3)\mathcal{L}_{\mathrm{JD}} = \lambda_f\mathcal{L}_{\mathrm{NCE}}(\mathrm{fused},S) + \lambda_m(\mathcal{L}_{\mathrm{NCE}}(\mathrm{EEG},S) + \mathcal{L}_{\mathrm{NCE}}(\mathrm{fMRI},S)), \tag*{(3)}

with λf=1.0\lambda_f = 1.0 and λm=0.5\lambda_m = 0.5, down-weighting the modality-specific terms to prevent them from dominating the fused objective and preserving complementary alignment paths for each modality.

S4AE diagnostics and reconstruction. S4AE denotes the backbone; reconstruction is enabled only for designated diagnostic configurations. The core Joint-Dual model and its reduced-width retest are contrastive-only (α=0\alpha= 0). In the targeted EEG–ImageBind S4AE diagnostic, reconstruction is enabled for the active modality:

LS4AE=βLNCE+αLrec,Lrec=1∣Xm∣∥X^m−Xm∥1.(4)\mathcal{L}_{\mathrm{S4AE}} = \beta\mathcal{L}_{\mathrm{NCE}} + \alpha\mathcal{L}_{\mathrm{rec}}, \quad\mathcal{L}_{\mathrm{rec}} = \frac{1}{\lvert X^m\rvert}\lVert\widehat{X}^m - X^m\rVert_1. \tag*{(4)}

All reported configurations set β=1\beta= 1. The EEG–ImageBind diagnostic sets α=0.25\alpha= 0.25; the core and reduced-width Joint-Dual systems set α=0\alpha= 0. The reported S4AE rows provide targeted configuration evidence across width, modality, and objective choices.

Given an EEG or fMRI window, the S4AE branch first lifts the input with a lightweight TemporalMLP, then applies stacked blocks that combine a structured state-space time mixer with a channel-mixing MLP. For EEG the structured mixer operates along sampled time; for fMRI the five TRs are input channels and the 18,926 voxels form the mixed sequence axis, so we describe the fMRI operation as structured state-space mixing rather than temporal mixing. When reconstruction is enabled, mirrored decoder blocks map the latent sequence back to the active neural input.

Evaluation protocol. All candidates are ranked by cosine similarity. We compute Recall@K (K∈{1,5,10,20}K \in\{1,5,10,20\}), MRR, mean average precision (mAP), and nDCG@10 [20] from the same ranked list. All candidates with the same stim_label are positives. This label-aware protocol reflects the repeated-observation structure of naturalistic EEG–fMRI data and is applied identically to every learned configuration and label-aware control. The one-to-one shuffled-gallery control is defined separately below. We also report effective rank, kk-nearest-neighbor density, and a densest-1% neighborhood audit to characterize the learned neural embedding geometry.

Experiments

Dataset, protocol, and experimental matrix. We evaluate on CineBrain [11], a publicly available synchronized EEG–fMRI dataset collected during long-form naturalistic audiovisual viewing with HRF-compensated fMRI alignment. A fixed 60/20/20 random sample split is shared across all configurations, and retrieval is evaluated over the full evaluation split. With approximately four observations per stimulus, the expected probability that a test sample shares its exact segment with training is 1−0.253=93.6%1 - 0.25^3 = 93.6\%. We use this random split as an in-distribution retrieval setting and complement it with episode-disjoint evaluation. All eight core configurations have five-seed Recall@K results: two frozen proxy anchors, Omni [44] and ImageBind [13], crossed with EEG-only, fMRI-only, cross-attention fusion (Both), and Joint-Dual. The first three systems use Transformer encoders; Joint-Dual uses two S4AE branches and mean fusion. In the label-aware setting, all same-stim_label candidates are treated as positives, and R@K, MRR, mAP, and nDCG@10 are computed from the same ranked list. EEG–ImageBind MRR, mAP, and nDCG@10 use four available seeds because one run lacks ranking outputs; all other core ranking aggregates use five. We additionally report random-brain and shuffled-video controls, targeted S4AE configuration diagnostics, a three-seed episode-disjoint loss and temperature ablation, selective two-subject held-out-training checks, and an episode-disjoint stress test.

Each sample pairs a video segment with its corresponding EEG and fMRI windows under the dataset-provided temporal alignment, including hemodynamic-response compensation for fMRI. This common temporal reference keeps stimulus alignment fixed across modality, anchor, fusion, and backbone comparisons. BrainBind treats frozen visual representations as proxy stimulus geometries: structured anchors for neural alignment, not literal neural codes.

For each retrieval direction, brain-to-video uses a neural embedding as the query against the video proxy gallery, while video-to-brain reverses this direction. The theoretical chance and random-brain rows use label-aware repeated positives. The shuffled-video row instead evaluates one-to-one row correspondence after independently permuting the video gallery. All controls retain the same gallery size and ranking metrics. The episode-disjoint split assigns each season–episode group to one partition, with 19,444/6,480/6,480 as train, validation, and test pairs.

Implementation details. Neural outputs match the selected anchor’s native dimension (2,048 for Omni and 1,024 for ImageBind) and are ℓ2\ell_2-normalized. Transformer branches use 512 hidden units, 8 heads, GELU, dropout 0.1, and pre-norm; fMRI inputs are 5 TR observations projected from 18,926 voxels; EEG uses temporal stride 8. Both applies one cross-attention module after modality-specific encoding and mean pooling. Joint-Dual uses independent S4AE branches, mean fusion, and contrastive-only training. All runs use AdamW [28] with cosine annealing [27], mixed precision, gradient clipping, and contrastive training. Checkpoints are selected by validation mean R@1. Statistical comparisons use two-sided paired t-tests over five matched random-split seeds; p-values are unadjusted. Three-seed diagnostics are interpreted descriptively. Additional architecture, control, and compute details are in the supplementary.

Training and compute

The primary Transformer systems contain 47.8–49.9M parameters; Joint-Dual–Omni has 293.8M. All runs use two NVIDIA A100 80GB GPUs, PyTorch 2.5.1 with CUDA 12.1, and the Omni Transformer runs use bf16 mixed precision. Table 2 reports measured training duration and validation recall for the runs with recorded timing.

ExperimentRunsAvg. epochsSpeed (it/s)Duration (min/run)Best val mean R@1 (%)
EEG–Omni Transformer550.00.067103.505.17
fMRI–Omni Transformer550.00.08283.1422.33
Both–Omni Transformer550.00.063111.3020.69
Joint–Dual–Omni 2×S4AE550.00.012585.2216.54
EEG–Omni S4AE-matched524.40.04182.441.10
fMRI–Omni S4AE-matched546.80.027357.9816.76
Joint–Dual–ImageBind 2×S4AE523.00.027117.3011.12
Reduced J-Dual–Omni S4AE250.00.014508.3018.87

Table 2. Training duration and throughput for runs with recorded timing.

Experiment familyBackboneTotal parameters
EEG / fMRI / Both, OmniTransformer [36]49,907,712
EEG / fMRI / Both, ImageBindTransformer [36]47,809,536
Joint-Dual, Omni2×S4AE (structured SSM) [15]293,835,134
Joint-Dual, ImageBind2×S4AE (structured SSM) [15]191,595,902
EEG S4AE-matched, OmniS4AE (structured SSM) [15]118,541,167
fMRI S4AE-matched, OmniS4AE (structured SSM) [15]118,541,167
Joint-Dual S4AE-matched, OmniS4AE (structured SSM) [15]142,754,222

Table 1. Model parameter counts for the reported configurations.

Joint-Dual–Omni is nearly six times larger than the fMRI-only Omni Transformer. This capacity difference positions the result as a system-level comparison: Joint-Dual more often places a label-positive candidate first, while fMRI-only organizes broader candidate neighborhoods more strongly. A capacity-matched control is a natural extension of this analysis.

Results

Random-split retrieval reveals asymmetric modality strength

Tables 3 and Table 4 summarize the label-aware bidirectional retrieval matrix. With Omni, fMRI reaches mean R@1 12.71, MRR 23.82, mAP 21.37, and nDCG@10 25.31, making it the strongest standalone modality across the ranked list. EEG aligned to the same proxy reaches mean R@1 1.91 in a 6,480 neural–video sample-pair pool where candidates share characters, scenes, and narrative context. Its absolute retrieval is modest, yet its separation from random-brain and shuffled-video controls indicates measurable neural–video correspondence on the random split. The episode-disjoint experiment below tests how much remains when recurring narrative context no longer crosses partitions.

AnchorInputFusionBackboneR@1MRRmAPnDCG@10
OmniEEGsingleTransformer1.91±0.286.60±0.586.03±0.496.06±0.75
OmnifMRIsingleTransformer12.71±0.2423.82±0.2921.37±0.1925.31±0.20
OmniBothcross-attnDual-Transformer11.23±0.3021.91±0.1819.61±0.1123.22±0.20
OmniJ-Dualmean2×S4AE13.52±0.1118.69±0.3515.09±0.3916.52±0.51
ImageBindEEGsingleTransformer0.61±0.052.83±0.0612.71±0.0612.20±0.061
ImageBindfMRIsingleTransformer6.12±0.5213.27±0.6711.41±0.5312.99±0.72
ImageBindBothcross-attnDual-Transformer4.76±0.4711.23±0.689.78±0.5310.94±0.70
ImageBindJ-Dualmean2×S4AE1.43±0.224.42±0.313.86±0.213.74±0.28

Table 4. Ranking-aware evaluation for the controlled retrieval matrix. Mean R@1, MRR, mAP, and nDCG@10 are averaged across the s2v and v2s directions.

AnchorInputR@1R@5R@10R@20
OmniEEG1.51±0.136.09±0.9011.26±1.5520.16±1.82
OmnifMRI9.81±0.1627.70±0.6442.71±0.3159.85±0.53
OmniBoth8.41±0.3025.08±0.4039.34±0.4156.04±0.69
OmniJ-Dual10.28±0.2417.47±0.5424.20±1.1733.35±1.78
ImageBindEEG0.65±0.052.61±0.114.77±0.139.23±0.26
ImageBindfMRI3.90±0.4212.90±0.8220.97±1.0332.68±0.86
ImageBindBoth3.29±0.2010.49±0.5917.94±0.9028.96±1.18
ImageBindJ-Dual0.99±0.123.52±0.246.76±0.5011.99±0.85

Table 3a. Brain-to-video retrieval (s2v)

AnchorInputR@1R@5R@10R@20
OmniEEG2.32±0.4510.16±1.3918.38±1.8730.65±2.54
OmnifMRI15.61±0.4540.08±0.6256.29±0.5272.20±0.57
OmniBoth14.05±0.6238.04±0.2753.47±0.6069.16±0.64
OmniJ-Dual16.75±0.2626.19±0.7033.46±1.1643.81±1.64
ImageBindEEG0.56±0.063.07±0.225.97±0.5511.31±0.43
ImageBindfMRI8.35±0.6624.05±1.3834.90±1.2448.18±0.95
ImageBindBoth6.23±0.7620.30±1.6330.93±1.7544.32±1.55
ImageBindJ-Dual1.87±0.356.90±0.6311.65±0.6718.89±0.83

Table 3b. Video-to-brain retrieval (v2s)

tbl:3: Label-aware CineBrain bidirectional retrieval matrix. Values are percentages, reported as mean±std across five seeds. Both directions use the same evaluation pair pool with query and gallery roles reversed. EEG and fMRI use single-modality Transformers; Both uses cross-attention fusion with a Dual-Transformer; J-Dual uses mean fusion with two S4AE branches. Best R@1 values within each anchor and direction are bold.

J-Dual changes the retrieval profile across ranks

Joint-Dual–Omni achieves the highest mean R@1 (13.52) in the main matrix, exceeding fMRI-only by +0.81 pp (p=8×10−4p = 8 \times10^{-4}). fMRI–Omni, however, provides stronger rank-wide organization: it leads on MRR, mAP, and nDCG@10 by 5.13, 6.28, and 8.79 pp respectively (p=8.5×10−7p = 8.5 \times10^{-7}, 9.7×10−79.7 \times10^{-7}, and 1.2×10−61.2 \times10^{-6}). Joint-Dual therefore places a label-positive candidate first more often, whereas fMRI-only organizes positives more strongly across the ranked list. We describe this system-level pattern as a precision–breadth trade-off. Joint-Dual uses two S4AE branches instead of the fMRI-only Transformer, is about six times larger (294M vs. 50M parameters), and optimizes fused and modality-specific contrastive terms; the reported gain therefore characterizes the complete Joint-Dual system. Architecture-, capacity-, and objective-matched controls provide a focused direction for isolating

individual contributions. Full matched-seed comparisons and signed effect differences are reported in Table 5.

ComparisonMetricΔ ppppInterpretation
Both vs. fMRI (Omni)mean R@1-1.480.0012System-level difference
Both vs. fMRI (Omni)nDCG@10-2.093.3×10−53.3 \times10^{-5}Rank-structure difference
J-Dual vs. fMRI (Omni)mean R@1+0.810.0008Top-rank difference
J-Dual vs. fMRI (Omni)MRR-5.138.5×10−78.5 \times10^{-7}Rank-organization difference
J-Dual vs. fMRI (Omni)mAP-6.289.7×10−79.7 \times10^{-7}Candidate-space difference
J-Dual vs. fMRI (Omni)nDCG@10-8.791.2×10−61.2 \times10^{-6}Rank-structure difference
fMRI ImageBind vs. fMRI OmninDCG@10-12.321.3×10−61.3 \times10^{-6}Anchor-associated difference

Table 5. Paired seed-level comparisons. Positive Δ indicates the first configuration is higher. The Joint-Dual rows quantify the difference between precise matching and rank-wide organization; the anchor row quantifies the consistent Omni advantage.

The embedding diagnostics in Figure 2 connect the 2D projections to high-dimensional neighborhood statistics. fMRI–Omni exhibits the strongest rank-wide profile: the steepest Top-KK curve on the 3,000-sample diagnostic subset (R@5 44.43, R@10 61.10) and the lowest median first-positive rank (7). Both–Omni remains close to fMRI–Omni but slightly weakens rank-wide organization. The R@1 ordering in this diagnostic subset differs from the full evaluation matrix: fMRI–Omni reaches 15.2%, Both–Omni 12.9%, J-Dual–Omni 11.1%, and EEG–Omni 3.6%. J-Dual–Omni has a median first-positive rank of 32 on this subset. Its highest mean R@1 in Table 4 refers to the full evaluation pool averaged over seeds, not to the subset shown in Figure 2. EEG–Omni exhibits stronger local density non-uniformity. These diagnostics characterize the displayed subset; the full-pool aggregates support the primary system comparisons.

Embedding-geometry diagnostics for Omni-aligned brain representations

Figure 2. Embedding-geometry diagnostics for Omni-aligned brain representations. The top block compares retrieval-colored 2D projections (t-SNE and UMAP) for the four Omni-aligned brain embeddings on a 3,000-sample diagnostic subset. These subset metrics are distinct from the full-pool, multi-seed aggregates in Tables 3 and Table 4. The bottom block reports rank-space and high-dimensional neighborhood statistics: Top-K retrieval curve, median first-positive rank, 20-NN cosine distance, density Gini coefficient, and Top-5 retrieval within the densest 1% neighborhoods. Projections are qualitative; metric panels are computed in retrieval rank space or the original high-dimensional embedding space. Higher Gini values indicate stronger local density non-uniformity rather than a quality ranking.

Proxy-anchor geometry is associated with alignment quality

Omni outperforms ImageBind across all configurations. For fMRI, Omni achieves s2v R@1 of 9.81 versus 3.90 for ImageBind; the nDCG@10 gap is 12.32 pp (p=1.3×10−6p = 1.3 \times10^{-6}). Figure 3 and the supplementary cross-anchor analysis characterize the corresponding geometric difference. Native-space spectral analysis shows that ImageBind concentrates 94.9% of its variance in the top 256 PCA components, whereas Omni distributes its variance more broadly across components (86.9% in the top 256), yielding a higher effective rank (1,112 vs. 426) and lower feature redundancy. This broader spectrum is consistent with the stronger Omni retrieval results. Exploratory CCA, applied after independently fitted 256-dimensional PCA representations and evaluated on the same 8,100 slices, has mean correlation 0.88 across the first 64 components. This in-sample correspondence motivates future multi-anchor analysis. The consistent Omni advantage across all eight systems supports a systematic relationship between proxy-anchor geometry and alignment quality for both EEG and fMRI.

Cross-anchor geometry of the two frozen video proxies

Figure 3. Cross-anchor geometry of the two frozen video proxies. ImageBind concentrates more variance in fewer principal components; Omni has broader spectral support and lower redundancy. Independently fitted PCA followed by in-sample CCA characterizes their linear correspondence.

Temporal and S4AE diagnostics characterize modeling behavior

A temporal-order check reverses, shifts, and TR-shuffles the EEG sequence at test time. Each perturbation changes retrieval by at most 0.5 percentage points, showing that the current mean-pooled EEG Transformer emphasizes order-robust neural–video correspondence. This result motivates sequence-aware extensions for explicitly modeling temporal order.

The targeted S4AE rows add two configuration checks. The core Joint-Dual model already uses two S4AE branches and contrastive-only training. A reduced-width contrastive-only retest reaches mean R@1R@1 14.38 over three seeds, compared with 13.52 for the larger five-seed core system. This configuration-dependent pattern motivates matched width analysis. An EEG–ImageBind S4AE configuration with reconstruction reaches mean R@1R@1 0.15, below its Transformer counterpart at 0.61. Together, these rows identify promising S4AE configurations and motivate a fuller modality-by-anchor study.

Loss and temperature ablations under the episode-disjoint split

We additionally complete two three-seed ablations under the episode-disjoint protocol. The first varies the EEG-only S4AE objective: InfoNCE versus SupCon, crossed with contrastive-only versus reconstruction-assisted training. SupCon defines positives by shared stimulus labels across neural and video views. In the contrastive-only setting, SupCon improves the EEG baseline across recall and ranking metrics, including mean R@1R@1 0.782% versus 0.532% and mean R@20R@20 1.258% versus 0.977%. The reconstruction-assisted SupCon variant reaches the highest mean R@1R@1 (0.136 ± 0.072%) but has larger seed variation and is not consistently stronger at R@5R@5, R@10R@10, or R@20R@20 than the contrastive-only SupCon row.

The second ablation uses a separate EEG Transformer configuration, holds the SupCon reconstruction-assisted objective fixed, reconstructs the mean-pooled 512-dimensional projected EEG input, and varies the contrastive temperature on the episode split. The configured default τ\tau=0.05 is the strongest three-seed setting on every reported metric: mean R@1R@1 0.180%, R@5R@5 0.486%, R@10R@10 0.936%, R@20R@20 1.674%, MRR 0.614%, and mAP 0.486%. Both ablations use three coupled model/split-seed runs, providing complementary configuration evidence across episode assignments.

Table 6 and Table 7 report the completed loss and temperature sweeps.

ObjectiveR@1R@5R@10R@20MRRmAP
InfoNCE, contrastive-only0.080±0.0250.080\pm0.0250.345±0.0470.345\pm0.0470.532±0.0780.532\pm0.0780.977±0.1050.977\pm0.1050.393±0.0280.393\pm0.0280.343±0.0090.343\pm0.009
SupCon, contrastive-only0.111±0.0380.111\pm0.0380.448±0.0080.448\pm0.0080.782±0.0470.782\pm0.0471.258±0.0881.258\pm0.0880.477±0.0230.477\pm0.0230.376±0.0300.376\pm0.030
InfoNCE, reconstruction-assisted0.080±0.0250.080\pm0.0250.316±0.0740.316\pm0.0740.689±0.1030.689\pm0.1031.175±0.1711.175\pm0.1710.435±0.0570.435\pm0.0570.391±0.0200.391\pm0.020
SupCon, reconstruction-assisted0.136±0.0720.136\pm0.0720.399±0.0980.399\pm0.0980.648±0.1670.648\pm0.1671.052±0.1611.052\pm0.1610.464±0.0810.464\pm0.0810.375±0.0280.375\pm0.028

Table 6. EEG-only S4AE loss ablation on the episode-disjoint split (percentages; mean±\pmstd over three seeds).

τ\tauR@1R@5R@10R@20MRRmAP
0.010.010.087±0.0090.087\pm0.0090.370±0.0270.370\pm0.0270.777±0.0620.777\pm0.0621.350±0.1241.350\pm0.1240.465±0.0070.465\pm0.0070.387±0.0240.387\pm0.024
0.050.050.180±0.0580.180\pm0.0580.486±0.0340.486\pm0.0340.936±0.0850.936\pm0.0851.674±0.1151.674\pm0.1150.614±0.0250.614\pm0.0250.486±0.0360.486\pm0.036
0.100.100.098±0.0350.098\pm0.0350.322±0.0660.322\pm0.0660.746±0.1030.746\pm0.1031.394±0.1801.394\pm0.1800.465±0.0900.465\pm0.0900.370±0.0660.370\pm0.066
0.200.200.123±0.0410.123\pm0.0410.396±0.0850.396\pm0.0850.671±0.0270.671\pm0.0271.168±0.1201.168\pm0.1200.471±0.0030.471\pm0.0030.367±0.0630.367\pm0.063

Table 7. EEG SupCon reconstruction-assisted Transformer temperature sweep on the episode-disjoint split (percentages; mean±\pmstd over three seeds).

Correspondence controls and selective held-out-training checks

Table 8 reports the correspondence controls, and Table 9 reports three-seed held-out-training checks for fMRI–Omni and Joint-Dual–Omni with subjects sub-0001 and sub-0002 excluded from training. The label-aware random-brain and one-to-one shuffled-gallery controls remain at their respective retrieval floors under both anchors, supporting the neural–visual correspondence measured by the learned systems.

SettingSplit / pools2v@1v2s@1Mean R@1MRRnDCG@10
Label-aware theoretical chance64800.0260.0260.0260.0260.0260.0260.2220.2220.0810.081
Random brain, Omni64800.0150.0150.0150.0150.0150.0150.1490.1490.0520.052
Shuffled video, Omni64800.0000.0000.0000.0000.0000.0000.1250.1250.0450.045
Random brain, ImageBind64800.0150.0150.0000.0000.0080.0080.1510.1510.0520.052
Shuffled video, ImageBind64800.0000.0000.0000.0000.0000.0000.1290.1290.0540.054

Table 8. Retrieval controls (percentages). Theoretical chance and random-brain rows are label-aware; shuffled-video rows use one-to-one row correspondence after permuting the video gallery.

ModelSubjectR@1R@5R@10R@20MRRmAP
fMRI–Omnisub-00015.234±5.2395.234\pm5.23916.501±13.14816.501\pm13.14824.103±15.74524.103\pm15.74531.956±14.96731.956\pm14.96711.083±8.14511.083\pm8.14511.085±8.14611.085\pm8.146
fMRI–Omnisub-00020.050±0.0370.050\pm0.0370.197±0.0530.197\pm0.0530.367±0.0930.367\pm0.0930.768±0.1570.768\pm0.1570.288±0.0610.288\pm0.0610.288±0.0610.288\pm0.061
J-Dual–Omnisub-000124.150±1.94424.150\pm1.94442.934±1.37942.934\pm1.37947.909±0.72547.909\pm0.72549.792±0.24149.792\pm0.24132.203±1.65232.203\pm1.65232.203±1.65332.203\pm1.653
J-Dual–Omnisub-00020.069±0.0120.069\pm0.0120.274±0.0590.274\pm0.0590.505±0.1200.505\pm0.1200.860±0.1940.860\pm0.1940.320±0.0190.320\pm0.0190.320±0.0190.320\pm0.019

Table 9. Selective three-seed held-out-training checks (percentages; mean±\pmstd over seeds). Twenty percent of each held-out subject’s samples are used for checkpoint selection; the remaining samples form the test gallery.

The held-out-training results are strongly subject-dependent. Joint-Dual–Omni is consistently strong for sub-0001 (24.150 ± 1.944% mean R@1R@1) but remains close to the floor for sub-0002 (0.069 ± 0.012%). fMRI–Omni shows the same direction of subject shift, with much higher seed variance on sub-0001 (5.234 ± 5.239%). This selective matrix covers only two subjects and two Omni models, and checkpoint selection still uses a validation subset from the held-out participant. It remains a held-out-training check and provides no population-level cross-subject estimate.

Low-capacity probes calibrate frozen-proxy readability

Shallow linear and MLP probes map EEG or fMRI features directly into the same frozen video proxy spaces under the random split using a 2,000-candidate diagnostic gallery. Table 10 shows that Omni readouts exceed ImageBind readouts, and fMRI readouts exceed EEG readouts within this probe protocol.

AnchorInputProbeR@1s2v@1v2s@1s2v@5v2s@5s2v@10v2s@10
ImageBindEEGLinear0.550.500.601.151.902.053.80
OmniEEGLinear1.050.951.153.203.656.205.60
ImageBindEEGMLP0.800.800.802.452.303.004.00
OmniEEGMLP1.231.151.304.054.556.807.35
ImageBindfMRIMLP18.5817.2019.9527.7535.4534.3043.15
OmnifMRIMLP22.8520.6025.1047.2552.3058.4064.95

Table 10. Low-capacity probe results on a 2,000-candidate CineBrain random-split gallery (percentages). Linear and MLP probes use shallow projections into the frozen video proxy space and the same label-aware bidirectional retrieval protocol as BrainBind.

The probes reproduce the main anchor and modality trends: Omni is more readable than ImageBind for EEG (linear 1.05 vs. 0.55; MLP 1.23 vs. 0.80), and the fMRI MLP probe reaches 22.85 mean R@1R@1 with Omni versus 18.58 with ImageBind. These within-gallery results show that the frozen proxies are readily accessible to shallow neural readouts. A separate 6,480-candidate EEG–Omni evaluation in the supplementary material places the linear and MLP probes at 0.32 and 0.40 mean R@1R@1, versus 1.91 for the full encoder under the same gallery size.

Episode-disjoint evaluation exposes context dependence

As a complementary generalization test, we group all clips by season–episode identifier and assign every clip from an episode to exactly one split. The resulting 30 disjoint episodes contain 19,444/6,480/6,480 train/validation/test sample pairs. Retrieval drops sharply under this stricter protocol (Table 11), showing that repeated stimulus identity and episode continuity contribute materially to random-split retrieval. The label-aware R@1R@1 chance floor is 0.069%: EEG is close to this floor, fMRI is higher, and Joint-Dual has the largest mean at every reported KK. Each of the three runs couples its model seed to the episode-assignment seed, capturing both initialization and split variation in the reported descriptive statistics.

ModelnnR@1R@1R@5R@5R@10R@10R@20R@20
Chance floor–0.0690.3470.6931.381
EEG30.075±0.0290.075 \pm0.0290.363±0.1090.363 \pm0.1090.579±0.1820.579 \pm0.1821.224±0.3231.224 \pm0.323
fMRI30.129±0.0730.129 \pm0.0730.417±0.2500.417 \pm0.2500.908±0.2440.908 \pm0.2441.641±0.3731.641 \pm0.373
Both30.126±0.0930.126 \pm0.0930.453±0.2000.453 \pm0.2000.893±0.4620.893 \pm0.4621.402±0.5851.402 \pm0.585
J-Dual30.216±0.0510.216 \pm0.0510.615±0.0830.615 \pm0.0831.026±0.1091.026 \pm0.1091.988±0.0541.988 \pm0.054

Table 11. Episode-disjoint label-aware retrieval (percentages; mean±std over three coupled model/split-seed runs). No season–episode group appears in more than one split; the chance row reflects the repeated-positive structure.

Retrieval-conditioned visualization provides qualitative examples

As a focused interpretability aid, the fMRI–Omni embedding retrieves a text description. A prompt-only diagnostic uses the first 32 test-loader samples as both queries and gallery candidates (chance 1/32=3.1%1/32 = 3.1\%). In a separate generation pass, retrieved text from a 16-query run conditions the fixed Wan2.1-T2V-1.3B generator [37]; ten outputs were retained. Figure 1 and the supplementary show representative generated frames from this proxy-mediated pathway, not direct neural reconstruction.

Discussion

Video proxy geometry as a shared neural reference. Within CineBrain, frozen long-form video embeddings provide a common retrieval geometry for synchronized EEG and fMRI. Their role here is a stimulus reference. The 12.32 pp fMRI nDCG@10 advantage of Omni over ImageBind (p=1.3×10−6p = 1.3 \times10^{-6}) coincides with Omni’s broader effective spectrum (effective rank 1,112 vs. 426) and lower redundancy. The association is observational. The consistent Omni advantage across all eight systems shows that proxy choice is associated with alignment quality. Selecting and characterizing the target video space is therefore a central design decision in neural alignment.

Modality- and system-specific profiles. The modalities and systems expose different retrieval profiles. fMRI-only provides the strongest rank-wide organization, while EEG-only carries a measurable, distinct in-domain retrieval profile alongside fMRI’s stronger rank-wide organization. The temporal-order perturbation result suggests that the current EEG encoder prioritizes order-robust features. Joint-Dual places a label-positive candidate first more often than fMRI-only as a complete S4AE, capacity, and multi-branch system. Its rank-wide metrics make system designs that combine top-rank precision with broader candidate organization a promising direction.

Relation to static-image benchmarks. BrainBind evaluates long-form synchronized video, complementing established paradigms such as THINGS-EEG and the NSD with MindEye, which primarily rely on isolated, static images or brief trial epochs. While static benchmarks isolate fine-grained visual representations without temporal confounding, continuous naturalistic viewing introduces complex narrative flow, multi-scale temporal dependencies, and recurring semantic contexts that fundamentally alter the neural decoding landscape. Direct cross-benchmark score comparisons can therefore be misleading due to structural differences in stimulus density, continuous hemodynamic integration, and evaluation dynamically. Consequently, we position our baseline controls, ranking-aware metrics, and visual embedding diagnostics strictly within the naturalistic video protocol on CineBrain, offering a calibrated framework for evaluating synchronized multimodal brain representations under dynamic real-world conditions.

Implications for evaluation. The results also clarify how synchronized neural decoding should be evaluated. The random split measures fine-grained in-distribution retrieval with repeated stimulus identity and narrative context, while the episode-disjoint split complements it with unseen-episode evaluation. Reporting top-rank and rank-wide metrics under both protocols reveals precision and candidate-space organization within a unified evaluation framework.

Scope and future directions. The current CineBrain study establishes a synchronized long-form video benchmark evaluated under two complementary splitting protocols. Under the four-observation structure, approximately 93.6% of random-split test samples share exact segment identities with the training set, capturing model performance when exposed to recurring dynamic scenes. Conversely, the episode-disjoint evaluation strips away this overlap to quantify true out-of-domain generalization. Initial held-out checks across two subjects and three random seeds provide an early snapshot of inter-subject variability, while hyperparameter ablations over contrastive losses (e.g., SupCon) and temperature parameters optimize representation learning within specific model families. Furthermore, our qualitative generation panel converts neural retrieval embeddings into descriptive text prompts to condition generative video models (such as Wan2.1), creating an interpretable semantic bridge alongside quantitative retrieval matrices. Future work will build upon this foundation by implementing parameter-matched fMRI controls to isolate modality-specific representation capacity, scaling to complete cross-subject folds, and integrating multi-positive contrastive objectives to accommodate non-unique semantic matches in continuous video streams.

Conclusion

We introduced BrainBind, a stimulus-aligned framework that compares synchronized EEG, fMRI, and fused systems within frozen long-form video geometries. On CineBrain, the 2×42 \times4 system matrix shows: fMRI–Omni provides the strongest rank-wide organization, EEG–Omni remains above controls, and Joint-Dual–Omni most often places a label-positive candidate first with a distinct rank-wide profile. Omni outperforms ImageBind throughout the matrix, linking proxy choice to alignment quality, and the episode-disjoint drop reveals the contribution of recurring narrative context to random-split scores. Episode-disjoint loss and temperature ablations further identify SupCon and τ=0.05\tau=0.05 as stronger EEG configurations under this stricter protocol, while two-subject held-out checks underscore the subject dependence that remains to be characterized. Correspondence controls and geometry diagnostics support the observed synchronized neural–video alignment within the scope of the present dataset. The current evidence points to three concrete next steps: a capacity-matched fMRI-only control, complete subject folds, and multi-positive objectives for related clips. These tests would separate multimodal information from capacity, estimate population-level subject variability, and align the objective with the repeated-label structure of naturalistic video.

BrainBind: Supplementary Material

Overview

This supplement provides extended protocol details, S4AE configuration results, correspondence controls, held-out low-capacity probes, cross-anchor geometry, qualitative retrieval-conditioned visualization, selective held-out-training checks, and hardware details that support the main paper. The primary retrieval, ranking, statistical, ablation, probe, and training tables are reported in the main text to avoid duplication.

Experimental Protocol

This section specifies the data alignment, split construction, and retrieval protocol used for all reported experiments.

Dataset and Temporal Alignment

All experiments use CineBrain [11], a synchronized EEG–fMRI dataset collected during naturalistic audiovisual viewing. Each sample pairs a video segment with its corresponding EEG and fMRI windows under the dataset-provided temporal alignment protocol, including hemodynamic-response compensation for fMRI windows. This keeps every model variant under a common temporal reference and enables configuration-level comparisons across modality, proxy anchor, fusion implementation, and backbone.

BrainBind treats frozen visual representations as proxy stimulus geometries: structured anchors for neural alignment, not literal neural codes. The proxy space supplies a structured visual-semantic reference within which heterogeneous neural measurements can be associated with complex video scenes. It evaluates how EEG and fMRI responses project into a shared video-grounded space, using the proxy embedding as a stable stimulus reference for neural alignment.

Evaluation Splits

The main retest matrix uses a fixed 60/20/20 random split. All configurations use identical partitions, enabling configuration-level comparisons across neural modality, proxy anchor, fusion implementation, and backbone. With approximately four observations per stimulus, the expected probability that a random-split test sample shares its exact segment with training is 1−0.43=93.6%1-0.4^3 = 93.6\%. This split provides an in-distribution retrieval setting, complemented by the episode-disjoint evaluation below.

We further report selective held-out-training checks for fMRI–Omni and Joint-Dual–Omni with subjects sub-0001 and sub-0002 excluded from training. For each held-out subject, 20% of that participant’s samples are used for validation and the remaining 80% for test, with seeds 42, 123, and 456. This asks whether video-grounded alignment remains measurable when a participant is absent from the training set. The two reported subjects provide an initial view of subject variation and motivate complete LOSO folds.

We additionally use an episode-disjoint split to test sensitivity to repeated narrative context. All clips sharing a season–episode identifier are assigned to exactly one partition. The resulting 30 disjoint episodes contain 19,444/6,480/6,480 train/validation/test samples, so no episode appears in more than one split. The three runs couple model and episode-assignment seeds 42, 123, and 456, incorporating both initialization and split variation.

Retrieval Evaluation

Retrieval is evaluated over the full evaluation split. In the label-aware setting, all candidates with the same stim_label are positives, and candidates with different stim_label values are negatives. Brain-to-video retrieval (s2v) uses neural embeddings as queries against the video proxy gallery; video-to-brain retrieval (v2s) reverses the direction.

All metrics (R@1\mathrm{R}@1, R@5\mathrm{R}@5, R@10\mathrm{R}@10, R@20\mathrm{R}@20, mean R@K\mathrm{R}@K, MRR, mAP, and nDCG@10) are computed from the ranked list for each query direction. This unified protocol allows the evaluation to capture both top-rank matching and the global organization of the candidate space. R@1\mathrm{R}@1 measures whether any label-positive candidate appears first, while MRR, mAP, nDCG, and higher-rank recall describe how positives are structured throughout the retrieval list. Core Recall aggregates use five seeds. EEG–ImageBind MRR, mAP, and nDCG@10 use four available seeds because one run lacks ranking outputs; every other core ranking aggregate uses five.

S4AE Architecture and Targeted Configuration Diagnostics

S4AE denotes the backbone family; reconstruction is enabled only in designated diagnostic configurations. The core Joint-Dual systems already contain independent EEG and fMRI S4AE branches and use contrastive-only training. We additionally report a reduced-width Joint-Dual retest and an EEG–ImageBind S4AE configuration. These targeted rows differ in width and objective and extend the primary eight-system matrix without completing a matched factorial comparison.

Architecture. Given an EEG or fMRI window, a lightweight TemporalMLP maps the input to the model width before stacked structured state-space and channel-mixing blocks. Normalization and residual connections wrap both mixers, and the aggregated bottleneck is projected into the selected proxy space. For EEG, the structured mixer operates along sampled time. In the reported fMRI branch, the five TRs are input channels and the 18,926 voxels form the mixed sequence axis; we therefore use the neutral term structured state-space mixing for fMRI. When reconstruction is enabled, mirrored decoder blocks map the latent sequence back to the active neural input.

Objective. The optional joint objective for an active modality mm is

LS4AE=βLNCE+αLrec,Lrec=1∣Xm∣∥X^m−Xm∥1.(5)\mathcal{L}_{\mathrm{S4AE}}=\beta\mathcal{L}_{\mathrm{NCE}}+\alpha\mathcal{L}_{\mathrm{rec}},\qquad\mathcal{L}_{\mathrm{rec}}=\frac{1}{\lvert X^{m}\rvert}\lVert\hat{X}^{m}-X^{m}\rVert_{1}. \tag*{(5)}

All reported configurations set β=1\beta= 1. The core Joint-Dual and reduced-width Joint-Dual retest set α=0\alpha= 0 and have no reconstruction loss. The EEG–ImageBind diagnostic uses joint contrastive and reconstruction training with α=0.25\alpha= 0.25. In the SupCon ablations, positives share the same stim_label; the Transformer decoder predicts the mean-pooled 512-dimensional projected EEG input.

Interpretation. The S4AE rows are configuration diagnostics that reveal useful architecture, width, and objective choices and motivate a fuller modality-by-anchor grid.

Additional S4AE Configuration Results

Table 12 and Table 13 report the reduced-width Joint-Dual–Omni configuration over three seeds and the EEG–ImageBind S4AE configuration over five seeds. They remain separate from the primary matrix to present targeted evidence over backbone, width, objective, modality, and anchor choices.

Anchor / inputDirectionnnR@1R@5R@10R@20
Omni / J-Duals2v310.08±0.3217.44±0.5122.96±1.1330.92±1.25
Omni / J-Dualv2s318.68±0.4227.14±0.8135.06±1.4744.98±2.04
ImageBind / EEGs2v50.09±0.060.30±0.090.51±0.100.95±0.18
ImageBind / EEGv2s50.20±0.030.82±0.071.49±0.112.69±0.30

Table 12. Targeted S4AE configurations on CineBrain bidirectional retrieval. Values are percentages; nn is the number of seeds. The reduced-width J-Dual configuration uses mean fusion and contrastive-only training. The single-modality EEG–ImageBind configuration jointly optimizes contrastive alignment and reconstruction. Widths, objectives, anchors, and seed sets differ, so these are configuration diagnostics rather than a factorial backbone ablation.

AnchorInputFusionConfigurationnnMean R@1MRRmAPnDCG@10
OmniJ-Dualmeanreduced S4AE, contrastive314.38±0.3519.14±0.5615.28±0.5216.89±0.66
ImageBindEEGsingleS4AE + reconstruction50.15±0.030.59±0.050.50±0.040.39±0.04

Table 13.

Table 13: Ranking-aware metrics for the targeted S4AE configurations. Values are percentages, mean±std over nn seeds.

Takeaway. With Omni, the reduced-width contrastive-only Joint-Dual model reaches mean R@1R@1 14.38 over three seeds, compared with 13.52 for the larger five-seed core system. Joint S4AE training with EEG–ImageBind reaches 0.15, below the Transformer row at 0.61. These results provide useful capacity and configuration checks; the differing seed sets and objectives support configuration-level rather than single-factor interpretation.

Chance and Shuffled-Feature Controls

The control experiments evaluate retrieval when neural–video correspondence is removed or replaced by theoretical chance structure. All values are reported as percentages. These controls are important because video embeddings can contain intrinsic clustering structure; therefore, a valid neural–video retrieval signal should remain separated from random-brain and shuffled-video baselines. The shuffled-video rows use unshuffled frozen video features as queries and a seed-fixed permutation of the same features as the gallery, with exact row identity as the positive definition; they do not use trained neural embeddings.

Omni Controls

Table 14 reports the Omni control results. The one-to-one theoretical chance row corresponds to exact pair matching in the full candidate pool. The label-aware theoretical chance row accounts for repeated same-stim_label positives. Random-brain and shuffled-video controls remain at their respective retrieval floors, providing label-aware and proxy-only empirical references for the learned neural–video results.

Controls2v@1s2v@5s2v@10s2v@20v2s@1v2s@5v2s@10v2s@20
One-to-one theoretical chance0.0150.0770.1540.3090.0150.0770.1540.309
Label-aware theoretical chance0.0260.1310.2620.5230.0260.1310.2620.523
Label-aware random brain0.0150.0460.0770.1230.0150.0930.1390.401
Shuffled video features0.0000.0620.1080.2620.0000.0460.1230.355

Table 14. Omni chance and shuffled-feature controls. Label-aware rows use same-stim_label positives; the shuffled-video row reports the one-to-one shuffled-feature control.

ImageBind Controls

Table 15 reports the corresponding ImageBind controls. As with Omni, random-brain and shuffled-video rows remain at the retrieval floor. The learned ImageBind systems remain clearly separated from both the label-aware random-brain and one-to-one proxy-only references.

Controls2v@1s2v@5s2v@10s2v@20v2s@1v2s@5v2s@10v2s@20
One-to-one theoretical chance0.0150.0770.1540.3090.0150.0770.1540.309
Label-aware theoretical chance0.0260.1310.2620.5230.0260.1310.2620.523
Label-aware random brain0.0150.0620.1390.2780.0000.0770.1080.231
Shuffled video features0.0000.0310.0770.2930.0000.1080.2310.309

Table 15. ImageBind chance and shuffled-feature controls. Label-aware rows use same-stim_label positives; the shuffled-video row reports the one-to-one shuffled-feature control.

Ranking-Aware Controls

Table 16 reports ranking-aware control metrics. The control rows remain near the theoretical floor for both anchors, while the main BrainBind configurations remain well above these baselines. This validates that the reported retrieval signal is not explained by intrinsic clustering of video embeddings, random neural structure, or label-aware bookkeeping.

AnchorControlMean R@1Mean MRRMean mAPMean nDCG@10
OmniLabel-aware theoretical chance0.0260.2220.1550.081
OmniLabel-aware random brain0.0150.1490.1480.052
OmniShuffled video features0.0000.1250.1250.045
ImageBindLabel-aware theoretical chance0.0260.2220.1550.081
ImageBindLabel-aware random brain0.0080.1510.1540.052
ImageBindShuffled video features0.0000.1290.1290.054

Table 16. Ranking-aware chance and control metrics. The label-aware theoretical chance accounts for multiple same-label positives per query. The shuffled-video rows use the one-to-one shuffled-feature control.

Takeaway. Across both anchors and both retrieval directions, random-brain and shuffled-video controls remain at their respective retrieval floors. The separation between BrainBind results and these controls is large in absolute terms (fMRI–Omni mean R@1R@1 of 12.71 versus a label-aware floor of 0.026) and consistent across recall and ranking metrics. The control separation is especially informative for EEG: above-chance retrieval tracks preserved neural–video correspondence, while randomized neural inputs remain at floor level in the same anchor.

Low-Capacity Probe Baselines

The main paper reports the low-capacity probe matrix on a CineBrain random-split gallery with 2,000 candidates. For held-out EEG with Omni, we additionally report probe runs on the test gallery with 6,480 candidates, enabling a same-gallery comparison with the full encoder.

InputProbeMean R@1s2v@1v2s@1s2v@5v2s@5s2v@10v2s@10
EEGLinear0.320.290.351.021.681.872.87
EEGMLP0.400.290.511.362.282.273.60

Table 17. Held-out test results for completed EEG–Omni probe runs. Values are percentages. The probes and full EEG–Omni encoder use the same 6,480-candidate test gallery and label-aware retrieval protocol.

Held-out test calibration. The completed EEG–Omni test runs show that shallow readouts recover measurable video-aligned information from EEG. The MLP probe improves over the linear probe in mean R@1 (0.40 vs. 0.32) and higher-rank recall, indicating that a small nonlinear readout helps extract EEG-proxy correspondence. The full BrainBind EEG–Omni encoder in the main matrix reaches 1.91 mean R@1, substantially above both low-capacity probes. This gap supports the contribution of the learned neural encoder and contrastive proxy alignment beyond a shallow mapping from EEG features. Alongside the random-brain and shuffled-video controls, the probe baselines show that BrainBind’s retrieval signal reflects measurable neural–video association and not a trivial readout from proxy-only structure.

Takeaway. Low-capacity probes provide an interpretable baseline for proxy-space readability. They reproduce two central trends from the main experiments: Omni is more readable than ImageBind, and fMRI exposes stronger video-aligned structure than EEG. At the same time, the held-out EEG–Omni comparison shows that BrainBind’s full neural encoder improves substantially over shallow linear or MLP readouts. These results strengthen the interpretation that BrainBind benefits from learned neural-to-proxy alignment while preserving the anchor and modality effects observed in the controlled retrieval matrix.

Cross-Anchor Proxy-Space Characterization

Why this matters. The main paper identifies anchor geometry as a first-class design axis: every Omni configuration outperforms its ImageBind counterpart, and Omni exceeds ImageBind by 12.32 pp in fMRI nDCG@10. To examine the associated geometry, we characterize the two surrogate-anchor spaces on 8,100 time-aligned video clip slices. For each slice, frozen visual embeddings from ImageBind [13] and Omni [44] retain their native 1,024- and 2,048-dimensional representations for spectral analysis. PCA is then fitted independently to each anchor with 256 components before canonical correlation analysis. These native frozen representations are also the contrastive target spaces: the neural output mapping matches the 1,024-D ImageBind or 2,048-D Omni, and the 256-D PCA reduction is used only for the cross-anchor analysis.

Spectral structure. The top 256 principal components explain 94.9% of ImageBind variance and 86.9% of Omni variance. ImageBind exhibits stronger low-dimensional concentration, indicating that a larger portion of its feature variance is compressed into fewer dominant directions. Omni shows a substantially higher effective rank (1111.7 vs. 425.7), indicating a broader spectrum and lower redundancy, and a lower mean absolute feature–feature correlation. This association offers a geometric hypothesis for the empirical retrieval gap: Omni exposes a broader and less redundant long-form video geometry that may be more readable by the neural encoders and is consistent with its retrieval advantage.

Exploratory linear correspondence. CCA fitted after the independent PCA reductions yields an in-sample mean correlation of 0.88 across the first 64 canonical components; the top 10 exceed 0.95 and the top 5 approach 0.98. Because the same 8,100 slices are used for fitting and evaluation, these values provide descriptive evidence of linear cross-anchor correspondence and motivate held-out multi-anchor analysis.

Takeaway. The geometric results contextualize the main paper’s anchor finding. Omni’s broader spectrum and lower redundancy make a richer geometry available to neural alignment and are consistent with the Omni advantage across the evaluated configurations. The in-sample CCA fit further shows that a strong linear correspondence can be estimated between the two anchors. BrainBind therefore treats anchor choice as an explicit experimental variable and motivates held-out multi-anchor fusion as a natural extension.

Qualitative Retrieval-Conditioned Visualization

In addition to retrieval metrics, we provide retrieval-conditioned visualizations as an exploratory interpretability aid. The prompt-only diagnostic uses the first 32 test-loader samples as both queries and gallery candidates. A separate 16-query generation pass retrieves text in Omni space and supplies it to Wan2.1-T2V-1.3B; ten generated videos were retained, and Figure 4 shows representative frames from three. Because the fixed generator receives retrieved text rather than neural signals directly, these outputs are proxy-mediated semantic visualizations rather than direct brain reconstruction.

Qualitative retrieval-conditioned visualization via the Omni pipeline

Figure 4. Qualitative retrieval-conditioned visualization via the Omni pipeline. The fMRI embedding retrieves text in Omni space, and the fixed Wan2.1-T2V-1.3B generator [37] produces the displayed frames. The panel shows three of ten retained outputs from the 16-query generation pass. It visualizes a neural→proxy→text→video pathway rather than direct neural reconstruction.

Omni pipeline: neural →\to proxy →\to text →\to Wan2.1

Given a time-aligned neural window, we encode EEG and/or fMRI through the BrainBind neural encoder to obtain an embedding in the Omni proxy space. We then retrieve a short textual description from the same proxy space and use the retrieved text as the prompt for Wan2.1-T2V-1.3B [37]:

Neural→Omni→Text→Wan2.1.(6)\mathrm{Neural} \to\mathrm{Omni} \to\mathrm{Text} \to\mathrm{Wan2.1}. \tag*{(6)}

All downstream components are fixed, yielding an interpretable pathway from neural activity through proxy embeddings to semantic descriptions. Retrieval and generation remain separate fixed stages.

The qualitative examples show visual content associated with the retrieved descriptions, including indoor environments, conversation settings, object context, and human presence. They illustrate the semantic pathway exposed by retrieval-conditioned generation.

Relation to Static-Image Neural Decoding Benchmarks

Long-form naturalistic video complements static-image benchmarks through a different candidate pool, stimulus regime, and target definition. THINGS-EEG and NSD/MindEye protocols [1, 12, 24, 31, 33] typically use isolated images and short stimulus epochs, whereas BrainBind uses synchronized EEG and fMRI windows from long-form videos with semantically proximate candidates. We therefore report BrainBind’s absolute metrics with random-brain and shuffled-video controls. Chance multipliers across these incommensurate protocols would not support a valid comparison.

Study / familySignalStimulus / supervisionOutput targetScale interpretation
MindEye2 [32]fMRINSD static natural imagesImage retrieval and diffusion reconstructionStrong low-data static-image reference; the target is an image embedding/generator, not video dynamics.
MinD-Video [7]fMRIDynamic visual stimuliVideo reconstructionDirect fMRI-to-video generation reference; it does not study synchronized EEG–fMRI proxy retrieval.
NeuroClips [14]fMRINatural video clipsSmooth fMRI-to-video reconstructionHigh-fidelity video-generation reference optimized for perceptual synthesis, not full-split ranking.
Brain Netflix [9]fMRIScaled natural video viewingVideo reconstruction from brain activityClosest large-scale video-generation reference; it focuses on fMRI synthesis and data scaling.
Animate Your Thoughts / Mind-Animator [29]fMRIDynamic natural visionDecoupled video reconstructionRecent fMRI-to-video generation reference; its target is reconstruction, not shared EEG–fMRI retrieval geometry.
BrainBind (ours)EEG + fMRICineBrain long-form naturalistic videoBidirectional retrieval, proxy diagnostics, and retrieval-conditioned visualizationStudies whether synchronized EEG and fMRI can be organized in one frozen long-form video reference geometry.

Table 18. Video-generation and static-image external references as scale context. These studies are not direct leaderboard comparisons: they differ in stimulus regime, signal modality, and output target. They define the landscape against which BrainBind’s distinct contribution is positioned.

Selective Held-Out-Training Checks

Table 19 reports three-seed held-out-training checks for fMRI–Omni and Joint-Dual–Omni with subjects sub-0001 and sub-0002 excluded from training. For each held-out subject, 20% of that participant’s samples are used for checkpoint selection and the remaining 80% form the test gallery. These selective checks provide a focused two-participant view of subject variation and complement the random- and episode-split evaluations; complete LOSO folds are a direct extension.

ModelHeld-out subjectSeedPairsMean R@1R@5R@10R@20MRRmAP
fMRI–Omnisub-000142432111.24731.48641.86548.55420.36820.372
fMRI–Omnisub-000112343212.80011.12018.58427.8297.7377.738
fMRI–Omnisub-000145643211.6556.89711.86119.4865.1455.145
fMRI–Omnisub-00024243200.0350.1850.3590.6710.2700.270
fMRI–Omnisub-000212343200.0930.2550.4630.9490.3560.356
fMRI–Omnisub-000245643200.0230.1500.2780.6830.2370.237
J-Dual–Omnisub-000142432121.95141.35647.07249.51430.31430.314
J-Dual–Omnisub-0001123432124.85543.53248.32249.93132.91432.914
J-Dual–Omnisub-0001456432125.64243.91348.33449.93133.38033.382
J-Dual–Omnisub-00024243200.0580.3240.6020.9840.3420.342
J-Dual–Omnisub-000212343200.0690.2890.5440.9610.3150.316
J-Dual–Omnisub-000245643200.0810.2080.3700.6370.3040.304

Table 19. Selective held-out-training checks (percentages). Mean R@1, R@5, R@10, R@20, MRR, and mAP average the neural-to-video and video-to-neural directions.

Across the three seeds, Joint-Dual–Omni reaches mean R@1 24.150 ± 1.944% for sub-0001 but only 0.069 ± 0.012% for sub-0002. fMRI–Omni shows the same subject direction with substantially higher variance on sub-0001 (5.234 ± 5.239%) and a near-floor mean for sub-0002 (0.050 ± 0.037%). The large subject difference, together with the fMRI–Omni seed spread on sub-0001, motivates subject-wise reporting and complete participant folds.

Training and Compute Details

This section records hardware and precision; parameter counts and measured training duration are reported in the main paper.

Hardware and Software

PropertyValue
GPUs2× NVIDIA A100 80GB PCIe
GPU memory80 GB per GPU
CUDA12.1
cuDNN90100
PyTorch2.5.1+cu121
Launch styleaccelerate launch --num_processes=2

Table 20. Hardware and software environment.

Archived execution metadata records bf16 for the reported Omni core runs and fp16 for ImageBind startup. The Omni artifacts record local batch 24 without distributed embedding gathering; released YAML files provide current reproduction defaults.

References

  1. [1]Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. A massive 7t fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1):116–126, 2022.DOI
  2. [2]Yunpeng Bai, Xintao Wang, Yan-Pei Cao, Yixiao Ge, Chun Yuan, and Ying Shan. Dreamdiffusion: High-quality EEG-to-image generation with temporal masked signal modeling and CLIP alignment. In European Conference on Computer Vision (ECCV), pages 472–488, 2024.DOI
  3. [3]Xuh"ao Cao, Peiliang Gong, Liying Zhang, and Daoqiang Zhang. EEG-CLIP: A transformer-based framework for EEG-guided image generation. Neural Networks, 194:108167, 2026. doi: 10.1016/j.neunet.2025.108167.DOI
  4. [4]T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020.arxiv.org/abs/2002.05709
  5. [5]Ye Chen, Peter Beech, Ziwei Yin, Shanshan Jia, Jiayi Zhang, Zhaofei Yu, and Jian K. Liu. Decoding dynamic visual scenes across the brain hierarchy. PLOS Computational Biology, 20(8):e1012297, 2024.DOI
  6. [6]Yuqi Chen, Kan Ren, Kaitaao Song, Yansen Wang, Yifan Wang, Dongsheng Li, and Lili Qiu. EEGFormer: Towards transferable and interpretable large-scale EEG foundation model. arXiv preprint arXiv:2401.10278, 2024.arxiv.org/abs/2401.10278
  7. [7]Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou. Cinematic mindscapes: High-quality video reconstruction from brain activity. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2305.11675
  8. [8]Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. NV-Retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.15831, 2024.
  9. [9]C. Fosco, B. Lahner, B. Pan, A. Andonian, E. Josephs, A. Lascelles, and A. Oliva. Brain netflix: Scaling data to reconstruct videos from brain signals. In European Conference on Computer Vision (ECCV), 2024.DOI
  10. [10]H. Fu, Z. Shen, J. J. Chin, and H. Wang. Brainvis: Exploring the bridge between brain and visual signals via image reconstruction. arXiv preprint arXiv:2312.14871, 2024.DOI
  11. [11]Jianxiong Gao, Yichang Liu, Baofeng Yang, Jianfeng Feng, and Yanwei Fu. Cinebrain: A large-scale multi-modal audiovisual brain dataset for brain-conditioned video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 36224–36234, 2026.
  12. [12]Alessandro T. Gifford, Kshitij Dwivedi, Gemma Roig, and Radoslaw M. Cichy. A large and rich EEG dataset for modeling human visual object recognition. NeuroImage, 264:119754, 2022.
  13. [13]R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Imagebind: One embedding space to bind them all. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.arxiv.org/abs/2305.05665
  14. [14]Zixuan Guo, Guangyue Bai, Qi Zhang, Zhongwei Wan, Duoqian Miao, Shoujin Wang, Lei Zhu, Changwei Wang, Rongtao Xu, Liang Hu, Ke Liu, and Yu Zhang. Neuroclips: Towards high-fidelity and smooth fMRI-to-video reconstruction. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 51655–51683, 2024.
  15. [15]A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2111.00396
  16. [16]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.arxiv.org/abs/2312.00752
  17. [17]U. Hasson, J. Chen, and C. J. Honey. Hierarchical process memory: Memory as an integral component of information processing. Trends in Cognitive Sciences, 19(6):304–313, 2015.DOI
  18. [18]Uri Hasson, Rafael Malach, and David J. Heeger. Reliability of cortical activity during natural stimulation. Trends in Cognitive Sciences, 14(1):40–48, 2010.DOI
  19. [19]K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.arxiv.org/abs/1911.05722
  20. [20]Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422–446, 2002.
  21. [21]Wei-Bang Jiang, Li-Ming Zhao, and Bao-Liang Lu. Large brain model for learning generic representations with tremendous EEG data in BCI. In International Conference on Learning Representations (ICLR), 2024.
  22. [22]P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2004.11362
  23. [23]Demetres Kostas, Stephane Aroca-Ouellette, and Frank Rudzicz. BENDR: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data. Frontiers in Human Neuroscience, 15:653659, 2021. doi: 10.3389/fnhum.2021.653659.
  24. [24]Dongyang Li, Chen Wei, Shiying Li, Jiachen Zou, and Quanying Liu. Visual decoding and reconstruction via EEG embeddings with guided diffusion. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024.
  25. [25]P. P. Liang, Z. Deng, M. Ma, J. Zou, L.-P. Morency, and R. Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2306.05268
  26. [26]Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, and Nanning Zheng. Brainclip: Bridging brain and visual-linguistic representation via CLIP for generic natural visual stimulus decoding. arXiv preprint arXiv:2302.12971, 2023.
  27. [27]Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR), 2017.
  28. [28]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
  29. [29]Yizhuo Lu, Changde Du, Chong Wang, Xuanliu Zhu, Liuyun Jiang, Xujin Li, and Huiguang He. Animate your thoughts: Reconstruction of dynamic natural vision from human brain activity. In International Conference on Learning Representations (ICLR), 2025.
  30. [30]A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
  31. [31]Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth A. Norman, and Tanishq Abraham. Reconstructing the mind’s eye: fMRI-to-image with contrastive learning and diffusion priors. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023.arxiv.org/abs/2305.18274
  32. [32]Paul Steven Scotti, Mihir Tripathy, Cesar Torrico, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A. Norman, and Tanishq Mathew Abraham. MindEye2: Shared-subject models enable fMRI-to-image with 1 hour of data. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 44038–44059. PMLR, 2024.
  33. [33]Yonghao Song, Bingchuan Liu, Xiang Li, Nanlin Shi, Yijun Wang, and Xiaorong Gao. Decoding natural images from EEG for object recognition. In International Conference on Learning Representations (ICLR), 2024.
  34. [34]Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  35. [35]A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  36. [36]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  37. [37]Wan Team, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models, 2025.
  38. [38]Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022.
  39. [39]Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral Cortex, pages 1–25, 2017.
  40. [40]J. Wu, S. Atito, Z. Feng, S. Mo, J. Kittler, and M. Awais. Rethinking positive pairs in contrastive learning. arXiv preprint arXiv:2410.18200, 2024.arxiv.org/abs/2410.18200
  41. [41]Rining Wu, Feixiang Zhou, Ziwei Yin, and Jian K. Liu. Aligning neuronal coding of dynamic visual scenes with foundation vision models. In European Conference on Computer Vision (ECCV), 2024.
  42. [42]Weihao Xia, Raoul de Charette, Cengiz Öztireli, and Jing-Hao Xue. Umbrae: Unified multimodal brain decoding. In European Conference on Computer Vision (ECCV), 2024.arxiv.org/abs/2404.07202
  43. [43]Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In EMNLP, 2021.
  44. [44]M. Xu, W. Zhou, Y. Babakhin, G. Moreira, R. Ak, R. Osmulski, B. Liu, E. Oldridge, and B. Schifferer. Omni-embed-nemotron: A unified multimodal retrieval model for text, image, audio, and video. arXiv preprint arXiv:2510.03458, 2025.
  45. [45]Yichen Zhang, Shanshan Jia, Yajing Zheng, Zhaofei Yu, Yonghong Tian, Siwei Ma, Tiejun Huang, and Jian K. Liu. Reconstruction of natural visual scenes from neural spikes with deep neural networks. Neural Networks, 125:190–302, 2020. doi: 10.1016/j.neunet.2020.01.033.

Paper details

vlnc:W_pkdpxkpx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents