WorksComputer Science
STAR-VF: Spatio-Temporal Aware Representation Condensation for Data-Efficient Video Fusion
Abstract
Video fusion aims to integrate complementary cues from heterogeneous sources, producing visually informative videos with coherent spatio-temporal structures. Jointly learning cross-modal complementarity and temporal dependencies typically relies on large-scale, high-quality video datasets, whose storage and repeated use during optimization incur substantial costs, making data-efficient learning highly desirable. To address this challenge, we rethink video fusion from the perspective of dataset condensation and propose STAR-VF, the first framework that condenses spatio-temporal representations for data-efficient video fusion. STAR-VF condenses the original training set into a compact set of learnable spatio-temporal representations, which are decoded into synthetic video clips and optimized using feature-level supervision provided by a frozen proxy model. The resulting condensed data preserves fusion-relevant spatial and temporal cues without requiring ground-truth fused videos as supervision. To adapt fusion learning to condensed representations, we equip the final fusion model with flow-free inter-frame temporal modeling and query-driven intra-frame aggregation, enabling direct temporal interaction and adaptive selection of fusion-relevant information in representation space. Extensive experiments on infrared-visible, multi-exposure, multi-focus, and medical video fusion benchmarks demonstrate that STAR-VF, trained with only approximately 1/16 of the original temporal training set, achieves fusion quality comparable to methods trained on the full data. Further analyses validate the contributions of learned condensation and flow-free temporal modeling, and demonstrate the robustness of the condensed supervision across different proxy encoders and downstream fusion models. Project page: https://zju-xyc.github.io/STAR-VF-Project-Page/.
CCS Concepts
Computing methodologies → Computer vision tasks.
Keywords
Video fusion, heterogeneous input, dataset condensation, data-efficient learning, spatio-temporal representation compression
ACM Reference Format:
Yuchen Xian, Yunqiu Xu, Guosheng Lin, and Yang He. 2026. STAR-VF: Spatio-Temporal Aware Representation Condensation for Data-Efficient Video Fusion. In Proceedings of the 34th ACM International Conference on Multimedia (MM ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 21 pages. https://doi.org/10.1145/3767308.3836033
Introduction
Video fusion [7, 13, 25, 33, 37, 39, 41] is an important yet under-explored task that aims to integrate video sequences from different sensors or vary acquisition conditions into a unified and temporally consistent output, yielding a more informative representation for dynamic scenes. Compared to static image fusion [8, 12, 16, 19, 20, 24, 31, 32, 33, 34, 35, 36], video fusion is more challenging, as it must jointly integrate complementary heterogeneous information at the frame level while preserving content stability and temporal structural consistency across frames.
Despite recent advances in video fusion, prior works [7, 25, 39, 41] still follow a full-data training paradigm, learning spatial complementarity and temporal dependency from complete aligned video pairs. While effective, this paradigm implicitly assumes continuous access to large-scale raw spatiotemporal training data. For video fusion, however, the challenge lies not only in the larger scale and higher training cost of videos, but also in the dual nature of the supervision required for effective fusion: the model must capture complementary information across heterogeneous input sources while preserving temporal coherence over time [7, 11, 39]. Once the training data is compressed, both signals can be degraded simultaneously, weakening the model’s ability to learn reliable fusion patterns [14]. Therefore, the central question for dataset compression in video fusion is not whether less data can be used, but whether the spatio-temporal supervision essential to effective fusion can still be preserved after substantial data reduction.
as a viable data-efficient learning paradigm for video fusion. The main contributions of this paper are as follows:
We first revisit video fusion from a data-efficient learning perspective and approach it through the lens of dataset condensation, aiming to enable efficient training while retaining fusion-critical spatio-temporal information.
We propose STAR-VF, a new video fusion framework with spatio-temporal aware representation compression, which learns a condensed training source and adapts fusion to condensed representations via flow-free inter-frame temporal modeling and query-driven intra-frame aggregation.
We evaluate STAR-VF on four video fusion tasks, validating that training on condensed data (1/1 of the original dataset) still achieves performance competitive with full-data-trained state-of-the-art approaches.

Figure 1. Comparison of temporal alignment paradigms under video fusion dataset condensation. Conventional flow-based alignment is not naturally suited to condensed synthetic clips, whereas our method models order-aware temporal dependency in compact representation space.
Therefore, applying dataset condensation to video fusion introduces three key challenges. ❶ Conventional temporal alignment based on optical flow estimation [2, 26] and feature warping [42] is designed for raw frames or dense features, making it poorly matched to compact condensed representations. ❷ In the condensed setting, fusion-critical cues are no longer expected to be uniformly distributed over all elements, so uniform aggregation can easily dilute the most informative components. ❸ After substantial data reduction, the condensed training source must still preserve the fusion-relevant supervision required for both cross-source complementarity and temporal coherence. However, these challenges remains largely underexplored. Existing dataset condensation methods are primarily designed for static tasks such as image classification [4, 22]. Therefore, making dataset condensation truly applicable to video fusion requires not only compressing the training data, but also developing a fusion framework that preserves spatio-temporal supervision and remains compatible with condensed representations.
To address this issue, we propose STAR-VF, a Spatio-Temporal Aware Representation condensation framework for data-efficient Video Fusion. STAR-VF compresses the original training set into a small set of learnable compact spatio-temporal representations, which are decoded into synthetic video clips under feature-level supervision from a frozen proxy network to preserve fusion-relevant signals. Built on this condensed set, the final fusion model is further designed to operate effectively on compact representations, with flow-free inter-frame temporal modeling for direct temporal dependency learning in compact space and query-driven intra-frame aggregation for adaptive extraction of fusion-critical intra-frame information, shown in Figure 1. Extensive experiments on four representative video fusion tasks show that, using only condensed synthetic data amounting to 1/1 of the original dataset, STAR-VF still achieves performance close to full-data training. These results suggest that spatio-temporal representation condensation can serve
Related Work
Video Fusion. Video fusion extends image fusion from static scenes to dynamic temporal sequences. While existing image fusion works, including CNN- [24, 34], Transformer- [19], GAN- [20], and diffusion-based [35, 40] approaches, have shown strong ability in multi-source information integration, they are primarily designed for single-frame settings and do not explicitly account for temporal dependency [3]. Directly applying image fusion models frame by frame to videos often causes flickering, motion discontinuity, and inter-frame inconsistency. Recent video fusion works [7, 25, 33, 39, 41] therefore place greater emphasis on temporal modeling. UniVF [39] introduces multi-frame learning with flow-guided warping, but suffers from considerable computational overhead. MambaVF [41] alleviates this issue by replacing explicit motion alignment with state-space-based temporal modeling. TemCoCo [7] further improves fused quality and temporal consistency through visual-semantic collaboration. However, these methods still rely on full-data training pipelines and original video representations, leaving compact condensed representations largely unexplored.
Dataset Condensation. Existing dataset condensation methods can be broadly divided into three categories based on their matching objectives. In performance matching, DD [28] represents the meta-based line by learning synthetic samples through bilevel optimization, while RFAD [17] improves kernel-based performance matching with random feature approximation to reduce computational cost. In parameter matching, IDC [9] belongs to the single-step line and improves condensation efficiency through synthetic-data parameterization under limited storage, whereas MTT [1] extends this idea to multi-step parameter matching by aligning training trajectories over multiple optimization steps, thereby alleviating the limitation of single-step matching, which only constrains local updates. In distribution matching, IDM [38] represents the single-layer line and improves naive feature distribution matching by addressing feature-number imbalance and unreliable embeddings for distance computation, while CAFE [27] further advances to multi-layer matching by aligning real and synthetic features across various scales, thus providing stronger global supervision and better discriminative preservation. Although these methods have achieved strong performance, most existing dataset condensation studies are still primarily developed and evaluated on image classification benchmarks [6], leaving limited exploration of multimodal video fusion, where condensed data should preserve not only cross-modal complementarity but also temporal dependency in compact spatiotemporal representations.
Preliminaries
Problem Formulation
Given a pair of source videos from two modalities, denoted by , each modality-specific video for is written as
Here, denotes the -th input frame, where is the channel number of modality , is the spatial resolution, and is the full video length.
The goal of multi-source video fusion is to generate a fused video from . Unlike static image fusion, multi-source video fusion must preserve both cross-modal complementarity within each frame and temporal coherence across neighboring frames. Accordingly, the task naturally involves two coupled aspects: intra-frame information aggregation and inter-frame temporal dependency modeling. The original training set is denoted by , which consists of paired source videos.
In data-efficient scenario, the fusion model is not applied to the full sequence at once. Instead, both training and inference are conducted on a local temporal window of length . Therefore, denotes the full video length of a sample, whereas denotes the effective temporal context visible to the model at each step. This distinction is important because the subsequent temporal modeling is window-based rather than full-sequence based.
Condensed Training Set
Instead of directly training on the full dataset , we consider a condensed training set containing learnable multi-source samples. Each condensed sample is parameterized as a pair of learnable compact spatiotemporal canvases , where each modality-specific canvas for satisfies
Here, denotes the grid size and denotes the spatial size of each grid cell. Under this parameterization, one condensed sample contains compact frame units. To recover video-like inputs from the compact canvases, we introduce a canvas decoder , which maps each to a decoded clip:
Here, each decoded frame lies in , matching the channel dimensionality and spatial resolution of the original input frame. These decoded clips serve as the synthetic training source for subsequent fusion learning. Accordingly, defines the condensed-data branch used only during training, while only the final fusion model is retained at test time.
Challenges under Condensed Representations
Dataset condensation changes not only the amount of training data, but also the representation form on which fusion is learned. Instead of operating on natural video streams directly, the model is trained on decoded clips reconstructed from compact spatiotemporal canvases.
Under this setting, video fusion faces the following three challenges. Challenge 1: Temporal modeling mismatch. Conventional video fusion often relies on optical flow estimation and feature warping to establish cross-frame correspondence. However, such designs are developed for raw video frames or dense feature maps. In the condensed setting considered here, training is performed on decoded clips reconstructed from compact representations. As a result, conventional flow-based temporal alignment is not naturally matched to the representation form of condensed data. Challenge 2: Intra-frame information dilution. In condensed representations, fusion-relevant cues are no longer expected to be uniformly distributed over all elements. Instead, they are more likely to concentrate in a sparse informative subset. Consequently, uniform aggregation can dilute the most useful information for fusion, making it difficult for the model to emphasize the elements that are truly critical. Challenge 3: Supervision preservation under data compression. Effective video fusion depends on two coupled types of supervision: cross-modal complementarity within each frame and temporal coherence across neighboring frames. Once the training set is compressed into a small number of synthetic samples, both signals may be weakened simultaneously. Therefore, the last challenge is to preserve the fusion-relevant spatio-temporal supervision after substantial data reduction.
These three challenges show that dataset condensation for multi-source video fusion is not merely a data reduction problem. It also requires a fusion pipeline that remains compatible with condensed spatio-temporal representations.
STAR-VF
Overview
As shown in Figure 2, STAR-VF contains a condensed-data branch and a final fusion model. The condensed set is

Figure 2. Overview of the proposed STAR-VF framework. In Stage I, learnable compact spatio-temporal canvases and the canvas decoder are jointly optimized under feature-level supervision from a frozen proxy feature extractor, yielding the learned condensed set and decoder . In Stage II, the learned condensed set is decoded into synthetic temporal samples to train the final fusion model with flow-free temporal alignment and query-driven intra-frame aggregation. During inference, only the trained fusion model is used, without the condensed set or proxy model.
where . A modality-conditioned decoder maps the cells of each canvas, in row-major order , to . Let be the number of paired temporal units in the original set. The training-source ratio is . The canvases are optimized variables rather than selected clips, and converts their cells into full-resolution training inputs. We use and choose such that . This ratio measures temporal training units rather than spatial downsampling.
Training has two stages. Stage I optimizes and by matching statistics extracted by a frozen proxy. Stage II fixes the condensed branch and trains on paired synthetic clips. For each target time , the final model encodes a local window, models inter-frame dependency, aggregates informative elements from both modalities, and reconstructs the fused center frame. At inference, only is retained to process real video pairs.
Flow-Free Inter-Frame Temporal Modeling
Decoded synthetic frames do not provide reliable motion fields for feature warping. We therefore model temporal dependency directly in representation space, allowing neighboring frames to interact without optical-flow estimation. This module uses the center frame as the query and the complete local window as temporal memory.
For a neighborhood of length , the modality-specific encoder produces
where is the number of tokens. Let be the number of heads and . For head ,
with . For token , M-RoPE uses and to encode temporal order and spatial position. We concatenate the encoded keys and values over the window:
both in . Temporal attention is normalized over all key tokens:
Joint normalization lets tokens from different frames compete while keeping the response scale independent of . M-RoPE retains their temporal order and spatial location.
The enhanced center-frame representation is
Query-Driven Intra-Frame Aggregation
Fusion-relevant cues are not distributed uniformly across the tokens, so direct pooling may suppress localized or modality-specific details. For each modality, learnable queries summarize the enhanced representation into aggregation slots. With , we compute
| Method | Train Ratio | IVF (VTMOT [43]) | MVF (Harvard [23]) | MEF (YouTube-HDR [29]) | MFF (DAVIS [21]) | ||||||||||||||||
| CC↑ | EN↑ | SSIM↑ | BiSWE↓ | MS2R↓ | CC↑ | EN↑ | SSIM↑ | BiSWE↓ | MS2R↓ | CC↑ | EN↑ | SSIM↑ | BiSWE↓ | MS2R↓ | CC↑ | EN↑ | SSIM↑ | BiSWE↓ | MS2R↓ | ||
| UniVF [39] | 1 | 0.65 | 6.81 | 0.65 | 3.95 | 0.35 | 0.78 | 4.27 | 0.75 | 29.92 | 1.23 | 0.96 | 7.25 | 0.65 | 6.44 | 0.34 | 0.98 | 7.41 | 0.90 | 6.04 | 1.11 |
| VideoFusion [25] | 1 | 0.62 | 6.59 | 0.62 | 3.14 | 0.59 | 0.78 | 4.91 | 0.69 | 19.71 | 1.31 | 0.91 | 6.82 | 0.58 | 9.98 | 1.82 | 0.96 | 7.34 | 0.82 | 9.07 | 6.13 |
| TemCoCo [7] | 1 | 0.63 | 6.68 | 0.60 | 6.26 | 0.59 | 0.77 | 5.40 | 0.24 | 28.41 | 1.57 | 0.90 | 7.34 | 0.53 | 11.62 | 1.70 | 0.97 | 7.45 | 0.84 | 12.34 | 4.08 |
| MambaVF [41] | 1 | - | - | 0.65 | 3.91 | 0.36 | - | - | 0.76 | 29.61 | 1.27 | - | - | - | 6.96 | 0.16 | - | - | - | 8.29 | - |
| STAR-VF (ours) | ∼1/16 | 0.64 | 6.73 | 0.65 | 3.91 | 0.35 | 0.80 | 4.37 | 0.76 | 28.36 | 1.26 | 0.91 | 7.28 | 0.55 | 10.23 | 1.28 | 0.96 | 7.42 | 0.91 | 5.53 | 1.18 |
Table 1. Quantitative comparison on IVF (VTMOT), MVF (Harvard), MEF (YouTube-HDR), and MFF (DAVIS). The training-data ratio denotes the fraction of the original temporal training units used to train each method. STAR-VF uses only approximately 1/16 of the original training source, yet achieves fusion quality comparable to the full-data baselines. The best and second-best results are highlighted in red and blue, respectively.

Figure 1 (PDF p. 5).
| Two-Stage Training Pipeline for Video Fusion | |
| Require: | training set , condensed set size , and window length |
| Ensure: | |
| 1: | Initialize and |
| Stage I: Condensation | |
| 2: | for each iteration do |
| 3: | Sample a real clip pair from ▷ real supervision source |
| 4: | Sample and decode for ▷ synthetic clips from compact canvases |
| 5: | Compute and update ▷ learn condensed branch |
| 6: | end for |
| 7: | ▷ freeze for Stage II |
| 8: | Initialize |
| Stage II: Fusion Training | |
| 9: | for each iteration do |
| 10: | Decode synthetic clips from |
| 11: | Construct with length ▷ local temporal window |
| 12: | Predict ▷ center-frame fusion |
| 13: | Compute and update ▷ train final fusion model |
| 14: | end for |
| 15: | |
| 16: | return |
Table Algorithm 1.
The row-wise attention weights allow each slot to select a different subset of compact elements. This produces a structured summary for each modality while retaining non-uniform information selection.
The modality features are concatenated along the feature dimension and reconstructed by :
Condensed Set Learning and Final Fusion Training
Stage I: Condensed Set Learning. For each sampled real clip and decoded synthetic clip, a frozen proxy [39] extracts multi-level features
Following multi-level feature matching [27], we use
Here matches mean proxy feature maps, matches channel mean and standard deviation, and matches spatial mean and standard deviation on a grid. Each term is summed over modalities and local proxy levels after pooling over sampled clips and frames. applies the same moment matching in pixel space [5, 16], preventing feature matching from drifting toward implausible image statistics. Stage I therefore preserves representative spatial statistics in a compact paired source.
Stage II: Final Fusion Model Training. After fixing , we train from scratch on local windows sampled from the decoded clips:
The fusion objective is
We use the intensity and gradient objectives of [34], SSIM [30], and the local temporal-consistency objective of [39] without modification. The first three terms supervise complementary source content in each fused frame, while regularizes changes across adjacent predictions.
Experiments
Experimental Setup
Datasets and Metrics. We evaluate STAR-VF on four video-fusion tasks: infrared-visible fusion (IVF) on VTMOT [43], medical video fusion (MVF) on Harvard [23], multi-exposure fusion (MEF) on YouTube-HDR [29], and multi-focus fusion (MFF) on DAVIS [21]. These datasets are used under their corresponding fusion tasks following the UniVF protocol, without additional custom pairing, alignment, or split construction. We compare with the dedicated video-fusion methods UniVF [39], VideoFusion [25], TemCoCo [7], and MambaVF [41].
We report the image-level metrics CC, EN, and SSIM used in IVIF-ZOO [15], together with the video-level temporal metrics BiSWE and MS2R from VF-Bench [39]. Higher CC, EN, and SSIM and lower BiSWE and MS2R indicate better performance. Efficiency is measured by parameters, FLOPs, latency, and peak GPU memory.
Implementation Details. In Stage I, compact canvases are initialized by sampling and spatially interpolating real frames. A frozen UniVF model provides proxy features, and the condensed branch is optimized for 10,000 iterations using Adam [10] with and a learning rate of . In Stage II, the condensed branch is fixed and a fusion model with and four Transformer blocks is trained from scratch for 20,000 iterations. We use aggregation queries and the centered 3-frame window . AdamW [18], a cosine schedule from to , and a batch size of 8 are used. We empirically set , , and to 0.5, 0.1, 0.1 and 0.5, respectively.
Main Results
Quantitative Results. Table 1 shows that STAR-VF, trained with the dataset-specific synthetic budgets in Table 2 (PDF p. 6), remains competitive with full-data methods. It ranks first in CC and SSIM on MVF and in SSIM and BiSWE on MFF, while remaining close to the best MS2R, showing that a compact training source retains useful per-frame fusion cues and temporal regularity.
| Dataset | Task | Train seqs | ||||
| VTMOT | IVF | 75 | 22,500 | 88 | 1,408 | 1/16.0 |
| Harvard | MVF | 49 | 1,308 | 6 | 96 | 1/13.6 |
| YouTube-HDR | MEF | 450 | 66,595 | 261 | 4,176 | 1/15.9 |
| DAVIS | MFF | 120 | 8,195 | 32 | 512 | 1/16.0 |
Table 2 (PDF p. 6). Training-source budgets used by STAR-VF.
Qualitative Results. Figure 3 shows that STAR-VF preserves salient structures and complementary details across all four tasks, including pedestrian contours on VTMOT, tissue boundaries on Harvard, highlight and shadow details on YouTube-HDR, and focus transitions on DAVIS. The enlarged regions expose local structural and brightness differences. In Figure 4, STAR-VF also exhibits fewer brightness fluctuations and more consistent colors than the compared methods.

Figure 3. Qualitative comparison on four representative benchmarks. For each example, we show the input pair, our method, the full-size variant trained on the original data, and competing video fusion baselines (VideoFusion, UniVF, and TemCoCo). The cyan boxes indicate enlarged local regions for detailed comparison.

Figure 4. Frame-by-frame qualitative comparison on a representative multi-source video fusion sequence.
Training-Source Efficiency. Table 2 (PDF p. 6) reports the actual budget on each benchmark, where . All settings use , , and . STAR-VF reduces the training source to approximately of the original temporal units on three datasets and on Harvard, substantially lowering the data required for fusion training.

Figure 5. Visual comparison between the full dataset and the learned condensed dataset.
Computational Efficiency. Table 3 compares the deployed fusion architectures; the condensed branch and proxy are absent at inference. Relative to UniVF, STAR-VF reduces parameters from 9.16M to 0.32M. At , FLOPs and latency decrease from 934.7G and 531.4 ms to 184.0G and 110.5 ms; at , they decrease from 6139G and 820.7 ms to 1066.6G and 468.5 ms. These inference gains arise from the final lightweight architecture and are distinct from the reduction in training-data budget.
Table 3. Computational efficiency comparison.
Frame-Wise Consistency. We extract luminance trajectories from aligned regions of interest and apply min–max normalization to remove scale differences. Figure 6 shows that STAR-VF follows the shared temporal trend with smaller local deviations, indicating a more stable fused appearance.

Figure 6. Alignment consistency comparison across video fusion methods, evaluated by frame-wise aligned ROI luminance trajectories after min–max normalization
Ablation Studies
Effect of Flow-Free Temporal Modeling. We replace VideoFusion’s RAFT-based alignment and warping with M-RoPE attention while keeping the full training set and fusion setting unchanged. Table 4 shows comparable fusion quality and temporal consistency on IVF and MFF, indicating that explicit flow estimation can be removed without performance loss while preserving effective temporal interactions across adjacent frames.
| Method | CC↑ | EN↑ | SSIM↑ | BiSWE↓ | MS2R↓ |
| UniVF [39] | 0.65 | 6.81 | 0.65 | 3.95 | 0.35 |
| w/o condensation | 0.64 | 6.73 | 0.64 | 3.91 | 0.36 |
| Ours (w/o RAFT) | 0.64 | 6.73 | 0.65 | 3.91 | 0.35 |
Table 4. Ablation of flow-free temporal modeling on IVF.
Effect of Condensation Grid Size. Table 5 shows that the grid achieves the lowest MS2R (0.35) and lower BiSWE (3.91), providing the best balance between compactness and temporal stability.
| Grid | #Frames | CC↑ | EN↑ | SSIM↑ | BiSWE↓ | MS2R↓ |
| 1×1 (full) | 1 | 0.64 | 6.73 | 0.64 | 3.91 | 0.36 |
| 2×2 | 4 | 0.90 | 7.25 | 0.87 | 5.07 | 0.36 |
| 3×3 | 9 | 0.89 | 7.82 | 0.88 | 5.10 | 0.36 |
| 4×4 (ours) | 16 | 0.64 | 6.73 | 0.65 | 3.91 | 0.35 |
| 5×5 | 25 | 0.90 | 6.62 | 0.88 | 5.17 | 0.36 |
Table 5. Ablation on condensation grid size on IVF.
Attribution to Learned Condensation. Under the same architecture and 1/16 budget on VTMOT, random compact sources achieve 5.64±0.79 EN, 0.48±0.12 CC, and 0.51±0.08 SSIM, compared with 6.73, 0.64, and 0.65 for learned condensation. This consistent improvement under an identical data budget confirms that the gains come from optimized condensation rather than arbitrary data reduction or source-size effects.
Proxy Robustness and Transferability. Table 6 shows that condensed sources generated with ResNet-18 and VGG-16 remain effective for both STAR-VF and UniVF, demonstrating robustness to proxy choice and transferability across fusion models.
| Downstream | Source/Proxy | EN↑ | CC↑ | SSIM↑ | BiSWE↓ | MS2R↓ |
| UniVF | Full | 6.814 | 0.654 | 0.652 | 3.950 | 0.353 |
| UniVF | Default | 6.370 | 0.625 | 0.626 | 5.505 | 0.379 |
| UniVF | ResNet-18 | 6.320 | 0.601 | 0.626 | 5.101 | 0.382 |
| UniVF | VGG-16 | 6.510 | 0.660 | 0.628 | 5.205 | 0.375 |
| STAR-VF | Default | 6.731 | 0.641 | 0.648 | 3.920 | 0.356 |
| STAR-VF | ResNet-18 | 6.720 | 0.591 | 0.635 | 3.586 | 0.355 |
| STAR-VF | VGG-16 | 6.709 | 0.643 | 0.646 | 3.445 | 0.354 |
Table 6. Proxy robustness and downstream transferability on VTMOT. “Full” denotes UniVF trained on the complete real set; all other rows use a 1/16 condensed source.
Conclusion
In this paper, we introduced STAR-VF, the first spatio-temporal representation condensation framework for data-efficient video fusion. Across four representative tasks, STAR-VF achieves performance comparable to full-data-trained methods using only a 1/16 training-source budget, demonstrating that compact synthetic representations can preserve fusion-relevant spatial and temporal cues. Controlled comparisons further show that the improvement arises from learned condensation rather than random compact construction or model-size reduction. Experiments with different proxy networks also indicate that the condensed supervision is transferable and not tied to a specific feature extractor or downstream model. Since the condensed branch and proxy are used only during training, inference operates directly on real multi-source videos using the final fusion model. Nevertheless, performance remains bounded by the capacity of the condensed source, and a fixed budget may become insufficient as the diversity of original data increases.
Acknowledgments
This research is supported in part by the RIE2025 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) (Award I2301E0026), administered by ASTAR, as well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL). This research is also supported in part by the ASTAR Career Development Fund (CDF) under Grant C243512011.
References
Problem Setup and Compact Representation
This section expands the formulation in § 2 and the condensed-set definition in § 3.1. We distinguish the temporal extent of a video from the context used for one prediction, explain why video-fusion condensation differs from static condensation, and make the compact training source and its decoder explicit.
Full Sequences and Local Fusion Windows
For modality , a paired input sequence and the corresponding fused output are
where and is the sequence length. The target is defined over all frames, but the model predicts each center frame from a neighborhood of length :
Sliding the operator in Eq. (A-2) over produces the full output in Eq. (A-1). Thus, is a property of the data sequence, whereas is the temporal receptive field of one fusion decision. This distinction also separates output scope from computational scope: STAR-VF generates a full video while using a bounded amount of context at each time step.
The local formulation matches both training sources. For real videos, is sampled from the original paired sequences. For condensed data, the same construction is applied to ordered frames decoded from a compact canvas. The final model therefore sees the same input type in Stage II and at inference; only the source of the frames changes. Boundary positions can be handled by the same padding or replication rule used by the implementation, without changing the operator itself.
Local windows also make the temporal supervision scale explicit. The condensed branch does not need to reproduce an unrestricted video history before the fusion model can learn temporal regularity. It needs to provide enough ordered support to form the neighborhoods consumed by . The relation between and the useful temporal range is analyzed in Section B.3.
Why Video-Fusion Condensation Requires a Structured Source
Static condensation usually represents each synthetic example as an independent image and preserves supervision associated with one spatial input. Video fusion adds two coupled axes. Within a frame, the two modalities contribute different and spatially non-uniform evidence; across frames, the output should remain stable while preserving genuine scene changes. A condensed source must therefore retain cross-modal complementarity and temporal organization at the same time. Compressing each frame independently can preserve local appearance yet discard the ordering needed to construct informative temporal neighborhoods.
The representation used for temporal learning also changes after condensation. Real videos contain natural pixel-space continuity, whereas decoded synthetic clips are optimized to preserve downstream supervision under a limited budget. Their useful information may be represented correctly in feature space even when their pixelwise motion does not follow the assumptions required by a conventional optical-flow estimator. A temporal operator tied to dense image-space correspondence can consequently become misaligned with the source used in Stage II. STAR-VF addresses this by decoding frame-compatible clips but establishing temporal dependency after encoding, as detailed in Section B.
Compression further concentrates informative cues. Salient structure, texture, illumination, and modality-specific contrast need not be distributed uniformly among the retained tokens. A global average or uniform pooling step can weaken precisely the localized evidence that distinguishes the two modalities. The query-driven aggregation in Section B.4 lets multiple learned slots select complementary subsets of the enhanced center-frame representation instead of assigning equal importance to all elements.
These requirements are linked rather than independent. The compact source must expose ordered, modality-matched inputs; the temporal module must operate on a representation compatible with those inputs; and the aggregation module must prevent the retained evidence from being diluted before reconstruction. This chain motivates the canvas-decoder pair below and the feature-level objective in Section C.
Condensed Source and Canvas Decoder
STAR-VF replaces the original training set with learnable paired samples . For modality , each canvas is arranged as a grid and decoded in row-major order:
The modality argument in allows one decoder design to produce outputs with the statistics and channel layout expected by each source modality. The grid contains ordered units, so the decoded output can be sampled as a short clip rather than as an unordered collection of images.
The pair specifies both what compact support is retained and how that support is realized as frame-shaped training data. In Stage I, decoded frames can be passed through the same frozen proxy as real frames. In Stage II, decoded clips can be grouped into windows of the form in Eq. (A-2). The decoder is therefore part of the training representation, not a visualization module: without it, the latent canvas would be incompatible with both the proxy feature extractor and the frame-based fusion network.
Let be the number of paired temporal units in the original training set. The effective training-source ratio is . This definition counts decoded temporal units rather than pixels. It separates data condensation from spatial downsampling: the decoded frames retain the input resolution required by the final model, while the number of learned training units is reduced. The experiments use , as reported in Table 2 (PDF p. 6).
Once Stage II is complete, neither nor is needed for inference. The deployed system is operating on real paired videos. The condensed branch changes the cost and composition of training but does not add a decoding step to the test-time path.
Temporal Ordering and Window Coverage
The row-major mapping in Eq. (A-3) gives each grid cell a unique temporal index . The two modalities use the same indexing rule, so forms a paired temporal unit. This shared order is required for fusion: modality complementarity is defined at a common index, and temporal neighborhoods are constructed by advancing that index consistently in both decoded clips. The spatial arrangement of cells inside is a compact parameterization; after decoding, their row-major indices provide the sequence order used by Stage II.
For a decoded clip of length and an odd local window length , every center index can produce a window when the boundary rule in Section A.1 is applied. If only fully contained windows are retained, one clip provides center positions. Thus, affects not only the number of decoded frames but also the diversity of local temporal contexts available from each canvas. With canvases, the number of candidate centers scales linearly with and approximately quadratically with , before any stochastic sampling used during training.
Ordering does not imply that adjacent grid cells must resemble neighboring image patches. The decoder maps each compact unit to a full-resolution frame, and temporal supervision is supplied by the ordered windows built from those outputs. This separates storage geometry from prediction geometry: the canvas uses a two-dimensional layout to share a compact decoder, whereas the fusion model operates on full-resolution frames and token coordinates. Consequently, the spatial coordinates in M-RoPE refer to positions within decoded frames, not positions of cells inside the canvas.
This distinction prevents two common ambiguities. First, is not the resolution of a synthetic frame; every decoded frame has resolution . Second, is the temporal support per condensed sample, not the length of a real test video. The real sequence can contain any supported by the sliding inference procedure, while the compact source only needs to supply local training contexts of length .
Canvas Capacity and Grid Size
A direct synthetic-video parameterization would learn full-resolution frames per modality. A frame-wise compact alternative would learn an independent tensor for every synthetic time step. STAR-VF stores the same compact support in one structured canvas.
Their parameter counts are
With , . The gain over independent compact units is therefore structural rather than a claim of fewer scalar variables: all units belong to one jointly optimized object and are interpreted by a shared decoder. Relative to direct full-resolution clips, the reduction factor per modality is for the same number of temporal units.
The grid size controls three quantities simultaneously: latent capacity, decoded clip length, and the number of windows available to Stage II. Small grids impose stronger compression but provide fewer temporal configurations. Larger grids provide more synthetic support and allow the decoder to store more variation, but weaken compactness and can allocate capacity to redundant patterns. Consequently, increasing need not improve the final model monotonically.
The ablation in Table 5 follows this trade-off, and Figure 7 shows its geometric interpretation. The setting supplies enough ordered units for local temporal training while maintaining the target data budget. The selected value is therefore determined by downstream performance under compression, not by treating the canvas as an image whose spatial resolution should simply be maximized.

Figure 7. Illustration of the capacity–compactness trade-off controlled by the grid size . Increasing enlarges both the latent canvas size and the decoded synthetic clip length , thereby increasing the effective supervision budget available to Stage II. At the same time, a larger weakens the compression effect by making the condensed source more flexible. The default setting in STAR-VF is selected as a balanced operating point under this trade-off.
Flow-Free Temporal Modeling and Selective Aggregation
The final model first establishes temporal dependency within each modality and then extracts a compact set of fusion-relevant elements from the enhanced center-frame representation. Both operations are defined in feature space, which keeps them compatible with decoded condensed clips and preserves a direct path from modality-specific evidence to reconstruction.
Representation Compatibility and Flow Sensitivity
Temporal modeling under condensation can be implemented with flow-based warping, 3D convolution, recurrent or state-space propagation, or local attention. These choices differ mainly in the assumptions imposed on the input representation. Flow-based warping expects recoverable image-space correspondence. A 3D convolution uses fixed local kernels over space and time. Sequential propagation compresses previous observations into a state and introduces an asymmetric update order. Local attention instead relates the center frame to all encoded tokens in a bounded neighborhood and lets the relevance weights depend on their content and position.
Decoded clips are optimized to preserve fusion-relevant statistics, not dense optical-flow fields. If a decoded frame is written conceptually as , where denotes decoding variation, a conventional alignment pipeline would use
Appearance variation first affects the displacement estimate and is then propagated through the warp. Since the Stage I objective in Eq. (14) does not supervise dense correspondence, this introduces an interface whose assumptions are not enforced during condensation. It can also suppress useful modality-specific patterns if a locally plausible displacement is estimated from a synthetic appearance change rather than real motion.
STAR-VF removes that interface. Each frame is encoded independently, and temporal interaction is applied directly to the resulting tokens. The model can then use a neighboring token because it is relevant to the center representation, without first expressing that relation as a pixel displacement. This choice does not discard spatial organization: token position is included by M-RoPE in Section B.2.
Joint Spatio-Temporal Attention with M-RoPE
For every , the modality-specific encoder produces , where is the number of tokens. The center frame supplies the query, while all frames in the window supply keys and values. For head , the projections are
This center-query construction keeps the output aligned with the target frame. Neighboring frames act as temporal memory rather than as additional output positions.
For token , M-RoPE uses the coordinate and applies temporal and spatial rotations through . The encoded keys and unrotated values are concatenated across the frames into . Attention is then
The softmax in Eq. (B-3) is normalized over all memory tokens, not separately over frames. Tokens from different times therefore compete in one distribution, while the response scale remains independent of window length. This is the same computation formalized in Eq. (7)–(9).
The position vector carries both temporal order and spatial location. Content-only attention could assign the same relation to visually similar tokens before and after the center time, a temporal-only position term would distinguish order but omit where a token occurs in the frame. M-RoPE retains both coordinates, so the similarity score can distinguish preceding from following evidence and preserve the spatial geometry required by fusion. The resulting is an enhanced center-frame representation with information selected from the whole local window.
The tensor shapes also clarify what is and is not compressed by the temporal block. A query head contains center-frame tokens, while its key and value memories contain tokens. Multiplication by therefore produces a score matrix. Each row describes how one center token draws evidence from every spatial position and time step in the local memory. Multiplication by returns , so the number and spatial indexing of center tokens are preserved through temporal enhancement.
Joint normalization over tokens is important for comparisons across frames. If attention were normalized separately inside each frame and the frame responses were summed, every frame would contribute a unit-normalized response regardless of its relevance, and the output magnitude would grow with unless another weighting stage were added. The normalization in Eq. (B-3) instead allocates one probability mass across the entire memory. A nearby frame can dominate when it contains decisive evidence, while an uninformative frame can receive little weight.
M-RoPE enters only the query-key similarity. Values remain content features and are mixed according to the position-aware weights. This keeps the retrieved evidence in the learned feature basis while using to determine where it should be retrieved from. The final projection combines the heads and returns the original channel dimension , making directly compatible with the aggregation projections in Eq. (B-5).
Local Temporal Scope
The effect of restricting attention to can be expressed by comparing a full-sequence response with the omitted contribution:
The local approximation is accurate when is small relative to the evidence inside the window. This condition is consistent with the evaluated tasks, where temporal quality is dominated by adjacent motion, local structural continuity, and short-range illumination change. It also matches the synthetic source, whose ordered support is intentionally compact.
The choice of balances information coverage against computation and relevance. A short window can miss useful motion or complementary evidence. A much longer window increases attention cost and introduces distant tokens whose relation to the center may be weak or ambiguous. Joint normalization can then distribute probability mass over redundant context. The selected local scope focuses the operator on the range most directly supervised by the decoded clips.
Because Eq. (A-2) is evaluated at every , local attention still produces a full-sequence result. Its limitation is temporal reach, not output length. Scenes with long occlusion, delayed cross-modal evidence, or large non-local displacement can yield a non-negligible residual; these cases are discussed in Section E.2.
Query-Driven Intra-Frame Aggregation
Temporal attention enhances center-frame tokens, but those tokens do not contribute equally to fusion. Some carry boundaries or fine texture, others capture thermal saliency or exposure detail, and many describe regions already represented adequately by the other modality. Uniform pooling would compress these elements with fixed weights and can dilute localized evidence after the already aggressive reduction of the training source.
STAR-VF uses learned aggregation queries per modality. After linear projection, each query attends over all enhanced tokens:
The row-wise softmax in Eq. (B-5) gives every slot its own selection pattern. One slot may emphasize high-contrast structure, another may collect smooth intensity evidence, and another may respond to localized modality-specific content. The roles are learned from the fusion objective rather than assigned manually. This produces a structured summary instead of collapsing all tokens into one global vector.
The two modality summaries retain separate parameters until aggregation is complete, so evidence is selected within its native representation before cross-modal reconstruction. They are then concatenated along the feature dimension and passed to , exactly as defined in Eq. (10)–(12). This ordering supports complementarity: each modality contributes several selected elements, and the decoder reconstructs the fused center frame from their joint representation.
Using multiple slots also avoids forcing heterogeneous cues into a single bottleneck. A one-vector summary would require structure, texture, intensity, and modality-specific saliency to share the same attention pattern. With rows, can retain several evidence types while remaining substantially smaller than the -token input when . The output is therefore compact in token count but not uniform in content. The aggregation queries are model parameters learned in Stage II; they are distinct from the condensed canvases optimized in Stage I.
Cross-modal interaction occurs after each modality has completed temporal enhancement and selective summarization. This preserves the meaning of modality-specific tokens during selection and gives the reconstruction decoder matched sets of slots. Concatenation along the feature dimension pairs the summaries without averaging them, allowing to combine corroborating evidence or retain a detail that appears strongly in only one source. The reconstruction objective in Eq. (16) supplies the signal that determines which slot patterns are useful.
The temporal and intra-frame modules therefore solve different selection problems. Eq. (B-3) determines which tokens across neighboring frames improve the center representation. Eq. (B-5) determines which elements of that enhanced representation should survive the compact summary used for fusion. Keeping these stages distinct preserves temporal order without forcing the reconstruction module to process every token uniformly.
Condensation Objectives and Two-Stage Training
Stage I learns a compact source that preserves statistics useful to the downstream fusion model. Stage II freezes that source and trains the final architecture with the same fusion objective used for real video inputs. This section details the exact matching terms and the boundary between the two stages.
Feature-Space Supervision
A direct pixel objective would require every decoded frame to reproduce a particular real frame even when several appearances provide equivalent training signals. Under a 1/16 source budget, this would spend limited capacity on sample-specific pixel configurations rather than on statistics shared by the task. STAR-VF instead uses a frozen proxy to map real and synthetic frames to a common feature hierarchy:
The proxy remains fixed, so the representation against which the canvases and decoder are optimized does not drift. Matching multiple levels covers complementary properties: shallower activations retain edges, contrast, and modality-specific texture, while deeper activations capture region organization and salient structure. The condensed source is not required to match each real sample one-to-one; it matches pooled statistics across sampled clips and frames.
Exact Multi-Level Matching Objective
The Stage I loss is the same objective given in Eq. (14):
Here, compares mean proxy feature maps after alignment to a spatial support. It preserves average feature organization instead of reducing the representation immediately to a single global statistic. The channel term compares the mean and standard deviation of every channel, retaining activation strength and dispersion. The spatial terms compare mean and standard deviation within grids for , preserving coarse and intermediate spatial layouts. All terms are accumulated over modalities and proxy levels after pooling the sampled real and synthetic units.
The three feature statistics are complementary. A global channel match can reproduce the distribution of responses while placing them in incorrect regions. A spatial mean can preserve layout but omit the variation carried by feature dispersion. The proxy-map term retains finer organization, while the and statistics stabilize structure at broader scales. Their combination constrains what is present, how strongly it is represented, and where it occurs without requiring a one-to-one pairing between real and synthetic frames.
Operationally, the real side aggregates statistics over the sampled batch, frame indices, and spatial support appropriate to each term. The synthetic side applies the same reductions to decoded units. A mean statistic aligns the center of the response distribution, while a standard-deviation statistic aligns its spread. Computing both prevents a constant synthetic activation from matching a non-trivial real distribution through its mean alone. Applying the reductions at several spatial granularities similarly prevents a globally correct statistic from masking a regionally incorrect arrangement.
The objective is symmetric with respect to the two source modalities in its aggregation structure, but the proxy features and decoded frames remain modality-specific. This is important when the sources have different channel counts or appearance statistics: the loss does not force infrared and visible inputs, for example, to become identical. Instead, each synthetic modality is matched to the corresponding real modality before the two are combined by the fusion model. Pairing is preserved through the common indices and introduced in Eq. (A-3).
Feature extractors are intentionally tolerant to some low-level changes. Without an image-space constraint, the decoder could match proxy statistics while drifting in brightness, contrast, or modality-specific channel distribution. applies the corresponding moment matching in pixel space and anchors the output of to a stable appearance regime. This term is especially relevant at the transition to Stage II, where the final encoders receive decoded images rather than proxy features. It prevents the final model from learning around avoidable artifacts of the condensed branch.
The weights , , and balance different statistics; they do not introduce a separate target task. Together, the terms in Eq. (C-2) preserve the spatial, channel-wise, and low-level properties needed by the fixed input interface defined in Eq. (A-3).
Two-Stage Optimization
Jointly updating the condensed source and the final model would make the training distribution and its consumer change simultaneously. The source could adapt to transient weaknesses of the current fusion network, while the network could adapt to transient decoder artifacts. STAR-VF separates the problems:
Stage I determines which supervision survives compression. Its optimization variables are the canvases and the modality-conditioned decoder, while the proxy is fixed. After convergence, is frozen. Stage II initializes the final fusion model from scratch and determines how the retained supervision is converted into fused outputs. This separation ensures that performance is attributable to a reusable training source rather than a private co-adaptation between one synthetic set and one intermediate model state.
Stage II builds ordered local windows from and applies the full architecture in Algorithm 1. The prediction loss remains the intensity, gradient, SSIM, and temporal combination in Eq. (16). The first three components preserve complementary content in each output frame; the temporal component regularizes changes across adjacent predictions. Condensation changes the training examples supplied to this objective, not the definition of successful fusion.
The gradient boundary between the stages is explicit. During Stage I, gradients pass from the matching statistics through the frozen proxy to and , but proxy parameters are not updated. During Stage II, decoded clips are treated as fixed data: gradients update the encoders, temporal attention, aggregation queries, and reconstruction decoder inside , but do not modify or . At inference, neither set of Stage I variables participates in the forward pass.
This schedule also makes evaluation reproducible across model initializations. A single learned source can be frozen and used to train a fresh fusion model under the same protocol as alternative compact sources. The random-source comparison in Section D.1 follows this separation; differences in the final metrics can therefore be attributed to the training source rather than to simultaneous adaptation of that source during Stage II.
Compatibility across Training and Inference
The learned source is compatible with the final model at three levels. First, interface compatibility follows from Eq. (A-3): decoded frames have the modality-specific channels and spatial resolution of real inputs, and their ordering supports the same local-window construction. No synthetic-only encoder or test-time adapter is introduced.
Second, representation compatibility follows from Eq. (C-2). The proxy-map, channel, spatial, and color terms preserve the statistics encountered by the early and intermediate stages of the final model. The synthetic frames can differ from particular real samples while remaining inside a useful training regime. Query-driven aggregation then selects non-uniform evidence from each modality without assuming that synthetic content is evenly distributed.
Third, objective compatibility follows from keeping unchanged. The model trained on condensed windows is evaluated on real windows with the same input-output relation in Eq. (A-2). At inference, the frozen condensed branch is removed; the real paired sequence is encoded, temporally enhanced, selectively aggregated, and reconstructed by . The efficiency gain is therefore a reduction in training data, while the deployed architecture remains the fusion network described in § 3.
The transition between stages is summarized as a stable sequence: learn a task-aligned source against a fixed proxy, freeze that source, train a fresh model against a fixed fusion objective, and retain only the model. This boundary also clarifies the limitation in Section E.1: Stage II can exploit the supervision preserved by Stage I but cannot recreate information absent from the condensed source.
Additional Experimental Results
The reported metrics describe complementary aspects of video fusion. EN, CC, and SSIM primarily measure information content, agreement, and structural quality at the frame level, with larger values preferred. BiSWE and MS2R characterize temporal behavior, with smaller values preferred in the reported protocol. A temporal score should therefore be interpreted together with spatial quality: an output can vary little because it is stable, or because useful detail has been smoothed away. The comparisons below use the full metric profile to distinguish those cases.
Random Initialization vs. Learned Condensed Data
Table 7 compares five randomly constructed compact sources with the learned condensed source under the same data budget and Stage II protocol. Across the random runs, EN, CC, and SSIM have means of 5.64, 0.48, and 0.51, with standard deviations of 0.79, 0.12, and 0.08. The learned source reaches 6.73, 0.64, and 0.65, respectively. It therefore lies near the strong end of the random distribution on all three image-level metrics rather than benefiting on only one measure.
The individual runs also show why a fixed compact budget is insufficient by itself. Random_2 obtains EN 4.76, CC 0.43, and SSIM 0.46, whereas Random_3 reaches 6.80, 0.67, and 0.65. Such dispersion means that the quality of the retained support depends strongly on the sampled initialization when no condensation objective organizes it. Learned canvases reduce this uncertainty by optimizing the spatial, channel, and color statistics that are later consumed by the final model.
Figure 8 shows the individual-run distribution after direction-aligned normalization, while Figure 9 summarizes its mean and variance. The visualizations place the learned source relative to the distribution rather than relative to a single selected seed. This matters because an isolated favorable run can conceal the instability of random compact construction.

Figure 8. Distribution of random initialization across five metrics, with the condensation result overlaid as a reference point. For comparability across metrics with different scales and optimization directions, all values are transformed into direction-aligned normalized scores. A wider spread of the random runs indicates stronger seed sensitivity and lower stability.

Figure 9. Mean-and-variance summary of random initialization across different metrics. Error bars denote the standard deviation over random runs, while the star marks the condensation result. The comparison shows that random initialization exhibits larger performance variability, whereas condensation provides a more stable and consistently strong outcome.
The temporal metrics should be read jointly with the spatial metrics. Some random runs obtain lower BiSWE or MS2R, but these cases coincide with markedly weaker EN, CC, or SSIM. A low temporal-variation score can result from outputs that change little because informative detail has been suppressed. The learned source instead provides a balanced point: it preserves substantially stronger spatial content while keeping temporal behavior within the range of the compact baselines. The comparison therefore isolates the effect of optimized condensation from the effect of training on a small source alone.
Additional Baselines on VTMOT
Table 8 extends the VTMOT comparison with representative image-fusion methods. GIFNet obtains the highest CC, while SwinFusion obtains the highest EN and SSIM. These results confirm that single-frame models can retain strong per-frame complementary information on this dataset. Their image-level strengths provide an informative reference for evaluating whether explicit temporal modeling sacrifices spatial quality.
Table 8. Quantitative comparison on IVF (VTMOT [43]). Red and Blue indicate the best and second-best results. Notably, STAR-VF is trained using only 1/16 of the original training data, yet still achieves competitive performance on VTMOT.
The temporal columns reveal a different pattern. GIFNet and TC-MoA obtain BiSWE values of 5.52 and 4.69, respectively, even though their frame-wise metrics are competitive. CDDFuse and SwinFusion also do not dominate the spatial and temporal measures simultaneously. Processing every frame independently leaves temporal regularity to emerge indirectly from similar inputs; it does not explicitly connect the prediction at to neighboring evidence.
STAR-VF reaches CC 0.64, EN 6.73, and SSIM 0.65 together with BiSWE 3.91 and MS2R 0.35, using 1/16 of the original training data. Its purpose is not to maximize one column in isolation but to retain a competitive spatial-temporal profile under a condensed source. Read together with Table 7, the result supports two distinct conclusions: learned condensation is more reliable than random compact construction, and flow-free temporal modeling preserves video-level regularity that strong image baselines do not enforce directly.
| Model | EN | CC | SSIM | BiSWE | MS2R |
| Random_1 | 5.27 | 0.38 | 0.45 | 2.65 | 0.28 |
| Random_2 | 4.76 | 0.43 | 0.46 | 2.83 | 0.29 |
| Random_3 | 6.80 | 0.67 | 0.65 | 4.03 | 0.35 |
| Random_4 | 5.32 | 0.41 | 0.47 | 2.94 | 0.31 |
| Random_5 | 6.04 | 0.51 | 0.52 | 3.05 | 0.33 |
| Random Mean | 5.64 | 0.48 | 0.51 | 3.10 | 0.31 |
| Random Std | 0.79 | 0.12 | 0.08 | 0.54 | 0.03 |
| Condensation | 6.73 | 0.64 | 0.65 | 3.92 | 0.36 |
Table 7. Comparison between random initialization and condensation on VTMOT. Higher is better for EN, CC, and SSIM, while lower is better for BiSWE and MS2R.
The two comparisons answer different controls. Random initialization holds the architecture and compact budget fixed, isolating whether the learned source contains organized supervision. Image-fusion baselines change the architecture and use frame-wise processing, testing whether strong per-frame methods already provide the temporal balance observed for STAR-VF. The learned source improves the first control, while explicit temporal dependency addresses the second. Neither table alone establishes both effects; their joint reading connects the condensed representation to the temporal model without conflating them.
Metric differences near the top of a column are also best read as trade-offs rather than as a single total ordering. SwinFusion has slightly higher EN and SSIM, and GIFNet has higher CC, whereas their temporal profiles differ from STAR-VF. The result claimed here is therefore competitive preservation under substantial data reduction, supported by simultaneous spatial and temporal measurements. This interpretation is consistent with the capacity analysis in Section A.5: the condensed representation is selected to preserve a balanced downstream signal rather than to reproduce every property of the full training set.
Limitations and Future Directions
Dependence on Condensed Representation Quality
The final model is trained only on frames produced by the learned pair . Information omitted in Stage I cannot be recovered by Stage II, so the quality of the condensed representation sets an upper bound on downstream performance. A fixed source budget may underrepresent rare structures, unusual modality interactions, or motion patterns that occupy little probability mass in the original data. The proxy statistics in Eq. (C-2) preserve broad properties of the training distribution, but they do not guarantee that every low-frequency event receives an individual synthetic counterpart.
This dependence becomes more important as the original distribution grows in diversity while and remain fixed. The decoder must then express more modes with the same number of learned units. Increasing the grid or the number of canvases can raise capacity, but it also changes and weakens the intended compression. An adaptive budget that allocates support according to motion, modality discrepancy, or sample rarity is a direct extension of the current fixed-ratio design.
Proxy choice also affects which equivalences are encouraged. A proxy that is insensitive to a fusion-critical pattern may allow the synthetic source to omit it even if the matching objective is minimized accurately. The multi-level and color terms reduce dependence on one representation depth, yet the condensed source remains task-aligned through the statistics selected for Stage I. Evaluating alternative frozen proxies and cross-dataset transfer would clarify how broadly one learned source can support different fusion architectures.
These limitations suggest concrete diagnostic signatures. Loss of rare spatial content would appear as locally weakened boundaries or modality-specific details even when average metrics remain strong. Insufficient variation in the condensed source would appear as high sensitivity to scene type or as a gap between common and uncommon motion patterns. A proxy mismatch would appear when Stage I statistics are well matched but Stage II performance remains weak across independent model initializations. Separating these cases requires inspecting both decoded samples and downstream outputs rather than relying only on the final mean score.
The two-stage structure helps localize such failures. If multiple final models trained from the same source fail similarly, the bottleneck is likely the retained supervision or decoder. If performance varies widely across Stage II initializations while the source is fixed, optimization of the final model becomes the more likely cause. This decomposition does not eliminate the representation ceiling, but it provides a clear experimental boundary for improving the appropriate stage.
Temporal Scope and Non-Local Motion
The residual in Eq. (B-4) becomes significant when useful evidence lies outside . Long occlusion, abrupt exposure change, large displacement, delayed modality complementarity, and temporally sparse events can all produce this condition. The output may remain locally smooth while missing a structure visible only in a more distant frame. This is a consequence of the bounded receptive field rather than of the absence of temporal modeling.
Highly non-local motion can also challenge token relevance within a short window. M-RoPE preserves temporal order and spatial coordinates, but it does not enlarge the observed interval. If an object crosses a large part of the frame or reappears after several missing observations, the available tokens may not contain a reliable chain of local evidence. Increasing is one solution, although the quadratic attention cost and the number of weakly related memory tokens also grow.
Hierarchical windows, sparse long-range memory, or adaptive neighborhood selection could extend temporal reach while preserving the local operator as the first stage. Such designs would need a corresponding condensed source with enough ordered support to train the added pathway. The representation and temporal backbone should therefore be expanded together rather than increasing sequence length in only one branch.
Future Extensions
Three directions follow from the preceding limitations. First, budget-aware condensation can vary or according to the diversity of a dataset or sample group. A simple extension could allocate more units to clusters with large motion or strong cross-modal discrepancy while preserving a global ratio. A more general formulation could learn the allocation jointly with the canvases under an explicit storage constraint.
Second, Stage I can incorporate additional task-aligned statistics. Temporal feature moments, cross-modal correlation, or a lightweight sequence-level proxy could supplement the current spatial and color terms when the target domain contains broader dynamics. These additions should preserve the stable two-stage boundary: the proxy and matching criteria define the source, and the final fusion model is still trained afterward from a fixed initialization.
Third, the final model can adopt hierarchical attention, recurrent memory, or stronger video backbones to cover longer dependencies. The query-driven aggregation remains applicable because it operates on the enhanced center representation rather than on a specific temporal architecture. Together, these directions extend the amount and range of supervision represented by each stage while retaining the central principle of STAR-VF: learn a compact, frame-compatible supervisory source first, then train the deployed fusion model on that fixed source.
A stronger temporal backbone also changes what the condensed source must preserve. A model with long-range memory can use evidence that a short-window model never observes, but only if Stage I supplies ordered variation from which that dependency can be learned. Expanding the backbone without expanding the temporal diversity of can therefore leave the additional capacity under-supervised.
Conversely, increasing without a model able to use the resulting contexts can spend the budget on redundant decoded units. Future scaling should coordinate the source budget, window construction, and temporal receptive field.
Transfer and Scaling Considerations
The current source is learned for a fixed task, modality pair, and proxy. Transfer to another fusion backbone is most plausible when the new model consumes the same frame interface and benefits from the same spatial and channel statistics. The two-stage design makes this question measurable: keep fixed, train several architectures from independent initializations, and compare their performance with models trained on the corresponding real data. A small and consistent gap would indicate that the condensed source captures architecture-independent supervision; a gap that grows for one backbone would indicate that its learning requirements are not represented adequately by the current matching objective.
Cross-dataset transfer is stricter because both appearance statistics and temporal patterns can change. Direct reuse is suitable when the target domain shares the modality semantics, resolution interface, and main motion regime. Larger shifts may require adapting the canvases, the decoder, or both. Initializing from an existing condensed source and continuing Stage I on a target-domain sample is one practical extension: it preserves the compact parameterization while allowing proxy and color statistics to move toward the new distribution. The amount of adaptation required can be measured against training a target source from scratch under the same .
Budget comparisons should report temporal units and model performance together. Because counts , two settings can have the same nominal ratio but organize it differently: more canvases provide greater sample diversity, while larger grids provide longer within-sample support. These allocations are not equivalent for a temporal model. Holding the ratio fixed while varying would separate diversity from sequence length and provide a more complete scaling analysis than changing alone.
The same principle applies beyond the evaluated modality pairs. A new source type can be incorporated when produces the required channel format and the proxy supplies meaningful features for that modality. The condensation statistics should then be validated for the cues that drive the new fusion task rather than copied unchanged. This preserves the framework-level separation between compact source learning and final-model training while allowing the representation, proxy, and budget allocation to be specialized to the target domain.
Practical Budget Selection
A practical deployment can select the condensed budget by treating downstream quality and source size as a validation curve. Begin from a small pair that can form the required windows, learn the source with a fixed Stage I protocol, and train the same Stage II architecture for every candidate. The preferred point is the smallest ratio whose spatial-temporal profile remains within the desired tolerance of the full-data model. This criterion directly measures the purpose of condensation and avoids choosing a grid solely from reconstruction appearance.
When additional budget is available, increasing and increasing should be tested separately. More canvases can represent distinct scenes or modality relationships, whereas a larger grid lengthens the ordered support inside each sample. Datasets dominated by diverse scene content may benefit more from ; datasets dominated by longer motion patterns may benefit more from . A controlled sweep at fixed can reveal which form of capacity is limiting before the total ratio is increased.
Validation should preserve the two-stage boundary used for the final experiment. Every source candidate should be frozen before Stage II, and the final model should be initialized independently. Using several Stage II seeds distinguishes a source that consistently retains supervision from a favorable model initialization. Reporting the mean together with variation is particularly useful near the budget knee, where small sources may produce similar average scores but different training stability.
Finally, budget selection should include both per-frame and temporal measures. A setting that preserves EN, CC, or SSIM but degrades BiSWE or MS2R may lack ordered support; a setting with smooth temporal scores but weaker spatial measures may be over-compressed. Applying the joint interpretation introduced in Section D yields a compact source that supports the complete video-fusion objective rather than one metric family alone.
References
- [1]George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. 2022. Dataset distillation by matching training trajectories. In CVPR Workshops.
- [2]Kelvin C.K. Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy. 2022. BasicVSR++: Improving Video Super-Resolution With Enhanced Propagation and Alignment. In CVPR.arxiv.org/abs/2104.13371
- [3]Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Jiahao Wang, Zhe Chen, Zhiqi Li, Tong Lu, and Limin Wang. 2026. Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding. IJCV (2026).arxiv.org/abs/2403.09626
- [4]Jianrong Ding, Zhanyu Liu, Guanjie Zheng, Haiming Jin, and Linghe Kong. 2024. CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting. In NeurIPS.arxiv.org/abs/2406.02131
- [5]Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In CVPR.
- [6]Jiahui Geng, Zongxiang Chen, Yuandou Wang, Herbert Woitseschlaeger, Sonja Schimmler, Ruben Mayer, Zhiming Zhao, and Chumming Rong. 2023. A survey on dataset distillation: Approaches, applications and future directions. In IJCAI.arxiv.org/abs/2305.01975
- [7]Meiqi Gong, Hao Zhang, Xunpeng Yi, Linfeng Tang, and Jiayi Ma. 2025. Temcoco: Temporally consistent multi-modal video fusion with visual-semantic collaboration. In ICCV.
- [8]Dingli Hua, Qingmao Chen, Zhiliang Wu, Yifan Zuo, Wenying Wen, and Yuming Fang. 2025. Perceptual Transform Fusion of Infrared and Visible Images. IEEE TCSVT (2025).DOI
- [9]Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. 2022. Dataset condensation via efficient synthetic-data parameterization. In ICML.arxiv.org/abs/2205.14959
- [10]Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.arxiv.org/abs/1412.6980
- [11]Kunchang Li, Xinhao Li, Yi Wang, Yinanh He, Yali Wang, Limin Wang, and Yu Qiao. 2024. VideoMamba: State Space Model for Efficient Video Understanding. In ECCV.
- [12]Xingyuan Li, Soncheng Du, Yang Zou, HaoYuan Xu, Zhiying Jiang, and Jinyuan Liu. 2026. Unifusion: A unified image fusion framework with robust representation and source-aware preservation. In CVPR.arxiv.org/abs/2603.14214
- [13]Xingyuan Li, Haoyuan Xu, Shulin Li, Xiang Chen, Zhiying Jiang, and Jinyuan Liu. 2026. DRFusion: Drift-Resilient Temporally Consistent Infrared-Visible Video Fusion. arXiv preprint arXiv:2605.25775 (2026).arxiv.org/abs/2605.25775
- [14]Zhe Li, Hadrén Reynaud, and Bernhard Kainz. 2025. Leveraging Multi-Modal Information to Enhance Dataset Distillation. In BMVC Workshops.
- [15]Jinyuan Liu, Guanyao Wu, Zhu Liu, Di Wang, Zhiying Jiang, Long Ma, Wei Zhong, Xin Fan, and Risheng Liu. 2024. Infrared and visible image fusion: From data compatibility to task adaption. IEEE TPAMI (2024).DOI
- [16]Jinyuan Liu, Bowei Zhang, Qingyun Mei, Xingyuan Li, Yang Zou, Zhiying Jiang, Long Ma, Risheng Liu, and Xin Fan. 2025. DCDFM: Discriminative Cross-Dimensional Evolutionary Learning for Infrared and Visible Image Fusion. In CVPR.
- [17]Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. 2022. Efficient dataset distillation using random feature approximation. In NeurIPS.arxiv.org/abs/2210.12067
- [18]Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
- [19]Jiayi Ma, Linfeng Tang, Fan Fan, Jun Huang, Xiaoguang Mei, and Yong Ma. 2022. SwinFusion: Cross-domain long-range learning for general image fusion via swin transformer. IEEE/CAA Journal of Automatica Sinica (2022).DOI
- [20]Jiayi Ma, Wei Yu, Pengwei Liang, Chang Li, and Junjun Jiang. 2019. FusionGAN: A generative adversarial network for infrared and visible image fusion. Information Fusion (2019).DOI
- [21]Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. 2017. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675 (2017).
- [22]Ding Qj, Jian Li, Junyao Gao, Shuguang Dou, Ying Tai, Jianlong Hu, Bo Zhao, Yibiao Wang, Chengjie Wang, and Cairong Zhao. 2025. Towards Universal Dataset Distillation via Task-Driven Diffusion. In CVPR.DOI
- [23]D Summers. 2003. Harvard Whole Brain Atlas: www. med. harvard. edu/AANLIB/home. html. Journal of Neurology, Neurosurgery & Psychiatry (2003).DOI
- [24]Han Tang, Bin Xiao, Weisheng Li, and Guoyin Wang. 2018. Pixel Convolutional Neural Network for Multi-Focus Image Fusion. Information Sciences (2018).DOI
- [25]Linfeng Tang, Yeda Wang, Meiqi Gong, Zizhuo Li, Yuxin Deng, Xunpeng Yi, Chunya Li, Han Xu, Hao Zhang, and Jiayi Ma. 2026. VideoFusion: A Spatio-temporal Collaborative Network for Multi-modal Video Fusion. In CVPR.arxiv.org/abs/2503.23359
- [26]Zachary Teed and Jia Deng. 2020. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In ECCV.arxiv.org/abs/2003.12039
- [27]Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. 2022. CAFE: Learning to Condense Dataset by Aligning Features. In CVPR.arxiv.org/abs/2203.01531
- [28]Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. 2018. Dataset Distillation. arXiv preprint arXiv:1811.10959 (2018).arxiv.org/abs/1811.10959
- [29]Yilin Wang, Joongwon Yim, Neil Birkbeck, and Balu Adusumilli. 2024. YouTube SFV+HDR Quality Dataset. In ICIP.arxiv.org/abs/2406.05305
- [30]Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simonc... 2004. Image Quality Assessment: from Error Visibility to Structural Similarity. IEEE TIP (2004).pubmed.ncbi.nlm.nih.gov/15376593
- [31]Zhiliang Wu, Kang Zhang, Hanyu Xuan, Xia Yuan, and Chunxia Zhao. 2023. Divide-and-conquer model based on wavelet domain for multi-focus image fusion. Signal Processing: Image Communication (2023).DOI
- [32]Yuchen Xian, Yunqiu Xu, Yang He, and Yi Yang. 2026. From 2D Grids to 1D Tokens: Reforming Shared Representations for Multimodal Image Fusion. In ICML.arxiv.org/abs/2606.12303
- [33]Housheng Xie, Meng Sang, Yukuang Zhang, Yang Yang, Shan Zhao, and Jianbo Zhong. 2024. RCVS: A Unified Registration and Fusion Framework for Video Streams. IEEE TMM (2024).DOI
- [34]Han Xu, Jiayi Ma, Junjun Jiang, Xiaojie Guo, and Haibin Ling. 2020. U2Fusion: A Unified Unsupervised Image Fusion Network. IEEE TPAMI (2020).DOI
- [35]Xunpeng Yi, Linfeng Tang, Hao Zhang, Han Xu, and Jiayi Ma. 2024. Diff-IF: Multi-modality image fusion via diffusion model with fusion knowledge prior. Information Fusion (2024).DOI
- [36]Guanghui Yue, Wentao Li, Cheng Zhao, Zhiliang Wu, Tianwei Zhou, Qiuiping Jiang, and Runmin Cong. 2025. Text-Guided Semantic Alignment Network With Spatial-Frequency Interaction for Infrared-Visible Image Fusion Under Extreme Illumination. IEEE TIP (2025).DOI
- [37]Cheng Zhao, Tianyun Song, Zhiliang Wu, Tianfu Wang, Moncef Gabbouj, Guanghui Yue, Baiying Lei, and Wei Zhou. 2026. Uncertainty-Guided Spatiotemporal Consistency Fusion Network for Infrared-Visible Video Fusion under Extremely Low-Light Conditions. IEEE TIP (2026).DOI
- [38]Ganglong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. 2023. Improved distribution matching for dataset condensation. In CVPR.arxiv.org/abs/2307.09742
- [39]Xiziang Zhao, Haowen Bai, Bingxin Ke, Yukun Cui, Lilun Deng, Yulun Zhang, Kai Zhang, and Konrad Schindler. 2025. A Unified Solution to Video Fusion: From Multi-Frame Learning to Benchmarking. In NeurIPS.
- [40]Xiziang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, and Luc Van Gool. 2023. DDFM: Disentangling Diffusion Model for Multi-Modality Image Fusion. In ICCV.arxiv.org/abs/2303.06840
- [41]Xiziang Zhao, Yukun Cui, Lilun Deng, Haowen Bai, Haotong Qin, Tao Feng, and Konrad Schindler. 2026. MambaVF: State Space Model for Efficient Video Fusion. arXiv preprint arXiv:2602.06017 (2026).arxiv.org/abs/2602.06017
- [42]Kun Zhou, Wenbo Li, Liying Lu, Xiaoguang Han, and Jiangbo Lu. 2022. Revisiting Temporal Alignment for Video Restoration. In CVPR.
- [43]Yabin Zhu, Qianwu Wang, Chenglong Li, Jin Tang, Chengjie Gu, and Zhixiang Huang. 2025. Visible-Thermal Multiple Object Tracking: Large-scale Video Dataset and Progressive Fusion Approach. PR (2025).arxiv.org/abs/2408.00969