WorksComputer Science
V-STORM: A Structure-Aware Grounded Chain of Thought Dataset for Faithful and Fine-Grained Spatio-Temporal Video Reasoning
Abstract
While MLLMs can often identify what happens in a video, they still struggle with fine-grained spatio-temporal reasoning, which requires precise grounding when and where the supporting evidence appears and composing such evidence into a coherent reasoning chain. This challenge is exacerbated by the lack of scalable data construction and evaluation pipelines that provide granular, verifiable supervision linking reasoning steps to explicit spatio-temporal evidence.To address this gap, we present V-STORM, a structure-aware framework that transforms videos into Spatio-Temporal Scene Graphs (STSGs) and derives Grounded Chain of Thought (GCoT) from validated evidence paths, making each reasoning step traceable to structured and verifiable spatio-temporal evidence.Under this framework, we construct a 260K-sample training set and a 3K-sample human-verified benchmark, both featuring GCoT annotations that link reasoning to explicit spatio-temporal evidence.Experiments reveal a clear separation between evidence localization and evidence composition: current MLLMs can often identify relevant spatio-temporal cues, yet still fail to integrate them into correct multi-step reasoning, exposing a bottleneck that answer-only evaluation largely obscures. Moreover, fine-tuning with V-STORM substantially improves both answer accuracy and grounding faithfulness, validating its effectiveness as supervision for faithful and evidence-grounded video reasoning. Our dataset, benchmark, and source code will be publicly released.
Introduction
Recent advances in multi-modal large language models (MLLMs) have significantly improved video-level understanding and question answering ([2, 14, 13]). However, these models still struggle with fine-grained spatio-temporal reasoning. This capability requires not only answering what happens in a video, but also explicitly grounding when and where the supporting evidence lies and correctly integrating such evidence into a coherent reasoning process. While recent studies have begun to enhance fine-grained temporal ([11]) or spatial ([28]) understanding, and emerging works explore evidence-grounded reasoning ([20, 8, 22, 16]), these efforts are often limited to isolated aspects or model-level designs, lacking systematic support for jointly modeling spatio-temporal grounding with structured, verifiable reasoning.
Fine-grained spatio-temporal reasoning involves three interdependent capabilities: (i) temporal grounding identifying when relevant events occur; (ii) spatial grounding localizing the involved entities; and (iii) compositional grounding integrating grounded observations into multi-step chains. These capabilities must be evaluated separately: accurate localization does not guarantee correct reasoning. In high-stakes settings like surveillance or embodied intelligence, plausible answers are insufficient without grounding in the right evidence. For example, a model may correctly report that a person entered a restricted area, but ground it to the wrong time window or individual—making the decision unreliable and hard to audit.
A major bottleneck is the lack of scalable data that jointly provides fine-grained grounding, explicit reasoning supervision, and verifiable evidence. As Table 1 summarizes, most existing video reasoning datasets suit evaluation but offer no step-level supervision for training evidence-grounded models ([12, 26, 6] (Fu et al. 2025; Qi et al. 2025)). A few ([15, 28]) provide high-quality spatial or temporal annotations, yet rely on costly expert labeling and offer limited reasoning depth. Video-Espresso ([11]) scales data via LLM generation, but focuses on high-level reasoning without fine-grained spatio-temporal grounding or verifiable supervision. Thus, existing resources insufficiently support faithful fine-grained spatio-temporal reasoning, where models must both locate and compose evidence into multi-step chains.
Table 1: Comparison of representative video reasoning datasets. We compare existing datasets across scale, grounding, reasoning capability, and construction paradigm. Compositional ST-Reasoning denotes multi-hop spatio-temporal reasoning ability. GCoT uses language-based spatio-temporal reasoning traces, while MMCoT denotes interleaved multimodal CoT.
| Dataset | Training Set | Grounding | Compositional | Annotation | Construction | Video-Free | ||
| Scale | Spatial | Temporal | ST-Reasoning | CoT | Verifiable | Paradigm | Extension | |
| Video-MME((Fu et al. 2025)) | – | – | – | – | – | – | Human | – |
| MotionBench([12]) | – | – | – | – | – | – | Human | – |
| NExT-QA([26]) | – | – | – | – | – | – | Human | – |
| ECBench([6]) | – | – | – | – | – | – | Human | – |
| VCR-Bench((Qi et al. 2025)) | – | ✓ | ✓ | – | GCoT | – | LLM+Human | – |
| E.T. Bench([15]) | – | – | ✓ | – | – | – | Human | – |
| EOC-Bench([28]) | – | ✓ | – | – | – | – | Human | – |
| Video-Espresso([11]) | 203k | ✓ | – | – | MMCoT | – | LLM | – |
| V-STORM (Ours) | 260k | ✓ | ✓ | ✓ | GCoT | ✓ | Programmatic | ✓ |
Table 1.
To address these limitations, we propose V-STORM (Figure 1), a structure-aware framework formulating video reasoning as a graph-grounded process for scalable and evidence-aligned data construction. The key component of V-STORM is the Spatio-Temporal Scene Graph (STSG), which organizes objects, trajectories, and dynamic relations into a unified and reusable representation. Unlike generation-based approaches that directly synthesize reasoning chains, STSG provides an explicit intermediate structure that preserves fine-grained spatial and temporal evidence. Built from complementary perception and MLLMs, a single STSG supports diverse reasoning paths and downstream tasks without repeatedly processing the original video. Based on this representation, V-STORM performs graph-guided question generation, where QA pairs and Grounded Chain-of-Thought (GCoT) annotations are derived from valid evidence paths. Consequently, each reasoning step is associated with explicit temporal and spatial evidence, enabling more reliable supervision and systematic evaluation.
Figure 1. Overview of V-STORM. (a) Construction Pipeline: Unlike traditional methods, V-STORM employs a structure-aware framework with multi-model collaboration to generate complex tasks and verifiable CoT annotations. (b) Task Taxonomy: We design 11 tasks spanning Perception, Reasoning, and Combination dimensions. (c) Examples: Sample queries demonstrating fine-grained grounding (specific timestamps and coordinates) in both questions and GCoT. (d) Statistics: Dataset distribution highlighting the scale (260k), diversity, and high-quality verification.
Based on this framework, we construct the V-STORM Dataset with over 260K training samples and a 3.3K human-verified benchmark. The benchmark introduces a grounding-aware evaluation protocol that separately assesses perception, evidence localization, and compositional reasoning, revealing distinct failure modes of current MLLMs.
Our contributions are summarized as follows:
We propose V-STORM, a structure-aware framework that ensures scalable and evidence-grounded video reasoning supervision through graph-based modeling.
We introduce the V-STORM Dataset, containing 260K training samples and a 3.3K human-verified benchmark with Grounded Chain-of-Thought annotations for fine-grained spatio-temporal reasoning.
We develop a grounding-aware evaluation protocol that disentangles perception, evidence localization, and reasoning, enabling systematic analysis of MLLM capabilities and limitations.
Experiments show that current MLLMs can effectively localize evidence for relatively simple reasoning, but struggle with composing multiple grounded observations. V-STORM provides effective structured supervision improving both reasoning accuracy and evidence grounding.
Related Work
Video Datasets. Existing video datasets generally fall into two categories. Benchmark-oriented datasets ([12, 26, 6] (Fu et al. 2025)) focus on high-level video understanding across coarse video-level supervision, while localization-centric datasets ([11, 28, 15]) provide fine-grained temporal or spatial annotations but typically support limited reasoning complexity. Recent efforts enrich annotations or introduce more challenging reasoning tasks, yet no existing benchmark jointly supports structured reasoning with verifiable fine-grained spatio-temporal grounding.
Video-LLMs. Recent Video-LLMs, including the Qwen-VL series ([21, 2 a]) and InternVL series ([5, 23]), have substantially improved video understanding through stronger visual encoders, longer context modeling, and video-native architectures. Inference-time approaches such as Chain-of-Frames ([10]) and Temporal-CoT ([31]) further enhance temporal reasoning by explicitly organizing intermediate reasoning steps over video frames.
Grounded Reasoning and Thinking-with-Videos. Recent work stresses traceable reasoning. TreeBench ([20]) shows the need to assess reasoning–evidence alignment, while recent video-reasoning methods ([16, 22, 8]) embed spatio-temporal evidence in reasoning. Yet these efforts target model design or training, and lack datasets providing reasoning traces, grounding labels, and sound metrics.
The V-STORM Framework
V-STORM formulates fine-grained video reasoning as a graph-grounded data construction process. Questions, answers, and CoT annotations are derived from explicit spatio-temporal evidence rather than generated directly from raw videos or captions. Based on videos, V-STORM generates question-answer pairs together with grounded Chain-of-Thought (GCoT) and corresponding spatio-temporal evidence annotations, enabling both reasoning supervision and evidence verification.
The framework consists of three stages: (i) STSG Construction transforms raw videos into structured spatio-temporal evidence; (ii) Programmatic Generation samples evidence paths from the STSG and renders them into question-answer pairs with GCoT annotations; and (iii) Traceable Verification validates the generated samples against the underlying evidence structure. Ultimately, V-STORM reduces the risk of ungrounded reasoning chains and enables systematic evaluation of both evidence localization and reasoning composition. The complete formal problem formulation is provided in the Supplementary Material, Sec. 2. Furthermore, to facilitate review and future reproducibility, we provide the anonymized source code in the Supplementary Material source code package, along with detailed descriptions of each file’s functionality in Supplementary Material Sec. 3.
Hybrid Perception & STSG Construction
Fine-grained spatio-temporal reasoning requires both accurate visual grounding and reliable semantic abstraction. Existing methods typically specialize in only one aspect: trackers such as SAM2 provide stable localization but limited semantics, while VLMs enable open-vocabulary understanding yet often suffer from unreliable long-term grounding. To leverage their complementary strengths, we propose a hierarchical graph construction pipeline that separates low-level perception from high-level semantic abstraction, progressively organizing visual information from frame-level observations to segment-level structures and finally a unified video-level STSG.
Hybrid Perception. Input video is divided into temporal segments . Within each segment, a semantic proposer (Florence-2) detects open-vocabulary object candidates on keyframes, while SAM2 propagates them into temporally consistent trajectories. An MLLM is further employed to resolve ambiguous identities when it cannot be reliably determined from low-level visual cues. Trajectories with insufficient temporal support are discarded, yielding a set of verified spatio-temporal elements .
Frame-level STSG Instantiation. Verified trajectories are instantiated as frame scene graphs , where each node represents an object with grounded attributes and each edge denotes a visually verified spatial relation. This atomic representation preserves faithful frame-level evidence without introducing global abstraction.
Hierarchical STSG Aggregation. Frame scene graphs are first aggregated within each temporal segment to form Segment-STSGs by propagating identities only inside short temporal windows, capturing object evolution while avoiding unreliable long-range association. The resulting Segment-STSGs are then conservatively merged into a global Video-STSG through high-confidence cross-segment matching. Finally, we build a reference index that links high-level events to their underlying atomic observations, enabling all subsequent question generation and verification to be traced back to validated visual evidence.
Programmatic Question Generation & Verification
Existing video question answering (QA) datasets typically rely on end-to-end generation, which often produces visually ungrounded questions and unverifiable reasoning chains. In contrast, we generate each sample directly from the STSG, ensuring that questions, reasoning steps, and answers are explicitly grounded in spatio-temporal evidence rather than free-form language priors.
As illustrated in Figure 2, our generation process follows three stages: (i) Programmatic Sampling extracts a structured reasoning path from the STSG; (ii) QA Instantiation converts the sampled path into a QA pair with reasoning and metadata; and (iii) Traceable Verification checks faithfulness against visual evidence and filters inconsistent samples.
Figure 2. V-STORM Data Construction Pipeline. The pipeline consists of three stages: (i) Hybrid Perception & STSG Construction, (ii) Programmatic Question Generation, and (iii) Traceable Verification. The STSG serves as the core asset that connects raw video to generated structured samples.
Evidence Sampling. We first sample an evidence path from the STSG . Each path consists of a sequence of connected objects, events, and relations with explicit spatial and temporal evidence. As shown in Stage 2 of Figure 2, the sampled paths can represent different reasoning types, including (i) atomic facts, (ii) temporal event sequences, and (iii) hierarchical event relations. This allows the same sampling strategy to support both simple perception tasks and more complex reasoning tasks. To construct challenging multiple-choice questions, we also sample visually plausible but incorrect candidates from the STSG as distractors.
QA Instantiation. Given a sampled evidence path , we employ an LLM as a structure-preserving translator to generate a Question–Reasoning–Answer triplet together with grounding metadata and distractors , yielding . The translator is constrained by rather than free-form generation, ensuring that each reasoning step follows the evidence graph. The correct answer is rendered from , while distractors are programmatically sampled from alternative STSG elements that are visually plausible but evidence-inconsistent. Finally, grounding metadata links all referenced entities to their original spatio-temporal locations, making every QA instance fully traceable.
Traceable Verification. Finally, we use the grounding metadata to retrieve corresponding frames, clips, and object trajectories as visual evidence. A verifier VLLM jointly examines the visual evidence and generated pair for factual consistency. Samples with inconsistent reasoning or
incorrect grounding are filtered, ensuring the final dataset is evidence-aligned and free from hallucinated annotations.
The V-STORM Dataset
To systematically investigate the performance of MLLMs in fine-grained spatiotemporal localization and reasoning tasks, we constructed the V-STORM Dataset. This dataset comprises two distinct components: a rigorously human-annotated Benchmark for evaluation, and a large-scale Training Split constructed for model training. While distinguished by their annotation precision and construction methodologies, both splits share a unified data structure and task definition, thereby supporting large-scale training and rigorous evaluation within a single consistent paradigm. Examples of tasks are provided in the Supplementary Material, Sec. 5.
Scope & Coverage
V-STORM is built from several public large-scale video datasets, including SAV [18], ActivityNet (Yu et al. 2019), Panda [24], WebVid [3], and InternVid [25]. A unified sampling frame rate and a maximum pixel budget are applied to ensure consistent perceptual granularity and manageable inference.
As shown in Figure 3, the V-STORM dataset spans both short and medium-length videos, supporting fine-grained spatiotemporal localization while preserving enough context for event-level reasoning. The V-STORM dataset covers diverse real-world domains such as sports, traffic, cooking, animals, natural scenery, home life, and music, providing a rich semantic foundation for studying fine-grained video reasoning that demands precise spatio-temporal perception and cross-temporal reasoning.
Figure 3. Distribution of video lengths in the Training Split and Benchmark, and video topics in the Benchmark.
Capability Taxonomy & Case Study
To facilitate fine-grained spatio-temporal reasoning, V-STORM provides Grounded Chain-of-Thought (GCoT) annotations that embed spatio-temporal grounding into each reasoning step. As Figure 1(c) illustrates, GCoT adopts a step-by-step natural language format, with every intermediate step tied to concrete temporal segments and spatial regions. This design guides models to explicitly utilize fine-grained visual evidence during multi-step reasoning, rather than relying solely on abstract linguistic patterns.
In GCoT, the temporal and spatial references are not generated freely but are derived from the spatio-temporal annotations defined during data construction. By coupling each reasoning step with explicit grounding metadata, the reasoning process becomes directly traceable to verifiable visual evidence. This formulation encourages models to not only produce logically coherent explanations, but also to perform reasoning that is consistently anchored in correct temporal and spatial locations.
Data Verification & Quality Control
V-STORM employs rigorous quality control to ensure reliability. For the Benchmark split, eight trained experts manually inspected every sample, verifying question formulation, answer correctness, and spatio-temporal consistency. For the Training split, we used metadata-guided automated verification (Figure 2 stage 3), supplemented by stratified human auditing. As Table 2 shows, this hybrid strategy achieves stable logical and annotation accuracy across categories, ensuring high-quality scaling without exhaustive manual effort.
| Question Category | Samples | Before | After | Imp. (Δ) |
| Frame-Level Perception | 100 | 82% | 98% | +16% |
| Segment-Level Perception | 100 | 78% | 96% | +18% |
| Temporal Reasoning | 100 | 71% | 92% | +21% |
| Compositional Reasoning | 100 | 64% | 94% | +30% |
| Overall / Average | 400 | 73.8% | 95.0% | +21.2% |
Table 2. Human audit results on the training split.

Figure 4. Definition of Answer Accuracy, Reasoning Score, and Grounding Score.
Experiments
This section empirically evaluates V-STORM’s design rationality and validates its effectiveness as both a unified benchmark and a training resource. Our experiments address three questions: (i) whether V-STORM effectively discriminates MLLMs’ fine-grained spatio-temporal perception and reasoning; (ii) the performance characteristics of mainstream models on this benchmark; and (iii) whether V-STORM’s training split provides effective supervision for model training. Notably, this section introduces no new model architectures or algorithms; all experiments solely verify the dataset’s diagnostic value and utility.
Evaluation Setup
Evaluation Protocol. The V-STORM benchmark adopts a unified multiple-choice QA format. Besides selecting the correct answer, our prompt explicitly instructs models to generate a Grounded Chain-of-Thought (GCoT) by citing fine-grained spatio-temporal evidence, including relevant timestamps and object locations, for each reasoning step. This protocol encourages evidence-grounded reasoning rather than answer-only prediction, enabling joint evaluation of answer correctness, reasoning quality, and grounding faithfulness. To ensure fair comparison, all models are evaluated under a unified zero-shot prompting protocol with a standardized output format for automatic parsing and scoring. Abstract prompt templates are provided in Figure 5 and the full version is provided in Supplementary Material, Sec. 4.1.
Figure 5. Inference Prompt (Top) and Failure Cases on the V-STORM benchmark. Middle: the model recognizes the correct subject but selects a wrong region due to semantic granularity mismatching. Bottom: the model correctly grounds the object at first but fails to track it later.
Evaluation Metrics. We evaluate models using three complementary metrics: Answer Accuracy, Reasoning Score, and Grounding Score. Accuracy measures task completion; Reasoning Score assesses logical consistency of generated reasoning chains via Qwen; Grounding Score evaluates alignment of cited evidence with annotated timestamps and object locations. These metrics disentangle answer correctness, reasoning quality, and evidence localization. To validate our scoring pipeline, we conduct a preliminary agreement study on a sample set by comparing Qwen with ChatGPT and Gemini. All three models show high inter-model agreement with a quadratic weighted kappa of 0.8841. We further verify the scoring quality via human expert review. Detailed metric definitions are in the Supplementary Material, Sec. 4.2.
Test Settings. We evaluate representative open-source and proprietary multimodal video models, including the Qwen3-VL [1], GPT [13], Gemini [19], VideoLLaMA3 [30], and InternVL [23] series. To ensure fair comparison, all models are evaluated under a unified maximum pixel budget, video sampling rate (), and prompt template, except Gemini-3-Flash, which uses its default setting (FPS = 1). Open-source models use greedy decoding (temperature = 0) with a maximum generation length of 2048 tokens.
Pipeline Reliability Validation
We further evaluate whether errors accumulate during the hierarchical STSG construction process through stage-wise human auditing. As shown in Table 4, high audited accuracies are consistently maintained across frame-level node grounding (92%), relation construction (88%), segment-level aggregation (91%), and video-level integration (95%), suggesting that the proposed pipeline preserves reliable spatio-temporal grounding throughout successive construction stages. In addition, the metadata-guided verification strategy improves the audited accuracy of generated training samples from 73.8% to 95.0% (Table 2), indicating that noisy or inconsistent samples are effectively filtered before inclusion in the final training set. Together, these results demonstrate that V-STORM provides a reliable and traceable pipeline for large-scale evidence-grounded data construction.
| Metric | FSG-Node | FSG-Edge | Segment STSG | Video STSG |
| Acc | 92% | 88% | 91% | 95% |
Table 4. Human audit accuracy (Acc) across STSG construction stages: node category/bbox, edge relation, segment trajectory/event, and video-level identity/event integration.
Benchmark Results of Existing Models
Table 3 summarizes the performance of representative MLLMs on the V-STORM Benchmark. Existing MLLMs achieve strong performance on coarse semantic perception, but their accuracy consistently declines as tasks require fine-grained temporal localization or compositional reasoning. Although leading commercial models outperform open-source counterparts overall, all models exhibit substantial performance drops on tasks requiring precise spatio-temporal understanding, indicating that current MLLMs still struggle with fine-grained grounded video reasoning.
| Model | FPS | Frame-Level Perception (Acc) | Segment-Level Perception (Acc) | Temporal Reasoning Acc | Temporal Reasoning R | Temporal Reasoning G | Comp. Reasoning Acc | Comp. Reasoning R | Comp. Reasoning G | Overall Acc | Overall R | Overall G |
| GPT-5.2 | 2 | 75.95 | 90.36 | 79.12 | 3.25 | 3.58 | 41.71 | 2.63 | 3.75 | 69.91 | 2.80 | 3.70 |
| Qwen3.6-plus | 2 | 73.42 | 91.30 | 80.93 | 3.43 | 3.97 | 40.36 | 2.75 | 3.96 | 69.38 | 2.93 | 3.97 |
| Gemini3-Flash | 1 | 69.71 | 87.81 | 75.52 | 3.02 | 3.61 | 34.84 | 2.51 | 4.06 | 64.97 | 2.65 | 3.94 |
| Qwen3-VL-2B-Instruct | 2 | 58.99 | 77.41 | 63.66 | 2.31 | 3.07 | 19.85 | 1.64 | 3.14 | 52.83 | 1.82 | 3.22 |
| Qwen3-VL-4B-Instruct | 2 | 65.44 | 90.64 | 79.64 | 3.18 | 3.67 | 27.00 | 2.16 | 3.82 | 62.81 | 2.44 | 3.78 |
| InternVL3.5-4B-Instruct | 2 | 59.49 | 84.22 | 68.56 | 3.01 | 2.80 | 23.80 | 3.27 | 3.65 | 56.99 | 2.06 | 3.43 |
| VideoLLaMA3-7B | 2 | 46.20 | 72.50 | 58.51 | 0.69 | 1.48 | 15.99 | 0.66 | 1.38 | 46.35 | 0.67 | 1.40 |
| Qwen3-VL-8B-Instruct | 2 | 68.48 | 91.40 | 78.87 | 2.82 | 2.69 | 27.38 | 2.33 | 3.86 | 63.81 | 2.20 | 3.55 |
| Qwen3-VL-32B-Instruct | 2 | 71.27 | 92.82 | 80.67 | 3.36 | 3.92 | 32.27 | 2.36 | 4.06 | 66.72 | 2.63 | 4.03 |
| Qwen3-VL-2BI+V-STORM | 2 | 76.84 | 88.94 | 75.52 | 3.11 | 3.77 | 73.28 | 3.48 | 4.61 | 79.42 | 3.38 | 4.38 |
Table 3. Performance of representative commercial and open-source multimodal models on the V-STORM Benchmark. Results are reported across four capability categories using accuracy (Acc) as the primary metric, with auxiliary Reasoning (R) and Grounding (G) scores for evidence-aligned spatio-temporal reasoning. Best scores are bold, and second-best are underlined.
A closer examination of the auxiliary metrics further reveals that model failures vary with reasoning complexity. For Temporal Reasoning tasks, high Answer Accuracy is consistently accompanied by high Grounding scores, suggesting that current MLLMs can reliably identify the required spatio-temporal evidence for relatively simple reasoning. However, in Compositional Reasoning, Answer Accuracy drops markedly despite maintaining strong Grounding performance. This observation indicates that current models are often able to locate the correct evidence, yet still struggle to integrate multiple grounded observations into coherent reasoning chains. These results demonstrate that V-STORM effectively separates evidence localization from reasoning ability, enabling diagnosis of failure modes that remain hidden under conventional answer-only evaluation.
Effectiveness of V-STORM-260K
To evaluate V-STORM-260K as a supervision resource, we fine-tune Qwen3-VL-2B-Instruct and compare it with representative commercial and open-source MLLMs on the V-STORM Benchmark. As shown in Table 3, V-STORM fine-tuning substantially improves fine-grained spatio-temporal reasoning. The resulting 2B model achieves an Overall Accuracy of 79.42, outperforming all evaluated models despite its substantially smaller scale. Compositional Reasoning accuracy increases from 19.85 to 73.28, indicating strong task-relevant supervision for multi-step reasoning. The Grounding Score improves from 3.22 to 4.38, the highest among evaluated models, demonstrating improved alignment between predictions and supporting spatio-temporal evidence. These results demonstrate that V-STORM-260K effectively improves both answer accuracy and evidence grounding, with particularly substantial gains in compositional reasoning.
Figure 5 presents representative failure cases of existing models on V-STORM. In the top case, the model identifies the correct subject and produces a region covering the target, but fails to capture the required semantic granularity by selecting the full body instead of the face. In the bottom case, the model successfully grounds the object in the initial scene but loses its spatio-temporal correspondence after a temporal offset, leading to incorrect tracking and reasoning. These examples demonstrate that current MLLMs may achieve plausible semantic understanding while still struggling with fine-grained evidence alignment and temporal consistency.
Robustness and Generalization Analysis
We further evaluate our approach from two complementary perspectives: generalization to out-of-distribution (OOD) settings and the effects of different reasoning supervision strategies, providing additional insights into model behavior under varying conditions.
OOD Generalization. We further evaluate the generalization capability of models trained on V-STORM under out-of-distribution (OOD) settings. As shown in Table 5, models trained with our V-STORM-260K maintain stable performance across diverse evaluation distributions, indicating strong generalization beyond the training domain. V-STORM yields consistent improvements across representative benchmarks, particularly those requiring fine-grained spatio-temporal understanding and reasoning. Notably, we observe gains on VideoRefer-Bench (Yuan et al. [29]) and NExT-QA (Xiao et al. [26]), which emphasize spatio-temporal grounding, as well as on EOC-Bench (Yuan et al. [28]), which focuses on object-centric temporal modeling. In addition, performance improvements on the reasoning-oriented MMBench-Video (Fang et al. [7]) further demonstrate the effectiveness of V-STORM.
| Method | VideoRefer | NExT-QA | MMBench-V | EOC |
| Qwen3-VL-2B | 67.20 | 74.88 | 1.3078 | 40.82 |
| + VSTORM | 68.60 | 75.92 | 1.3163 | 42.32 |
Table 5. Performance on OOD benchmarks.
Impact of Reasoning Supervision. We further investigate different reasoning supervision strategies, including coarse-grained CoT (CCoT) and our GCoT, using the same V-STORM subset (~95k) and Qwen3-VL-2B-Instruct as the baseline model. As shown in Table 6, SFT without explicit reasoning supervision already substantially improves performance, particularly on complex reasoning tasks (22.11→70.84), demonstrating the value of task-specific spatio-temporal supervision. Both CCoT and GCoT provide additional improvements, suggesting that explicit reasoning supervision further benefits model learning. Notably, GCoT consistently achieves the best performance across all evaluated metrics, outperforming SFT without CoT by 1.97 points in Overall Accuracy and CCoT by 1.27 points. The gains are particularly pronounced on fine-grained perception (+3.54 over SFT without CoT) and complex reasoning (+2.35 and +1.98 over SFT without CoT and CCoT, respectively). These results suggest that fine-grained spatio-temporal grounding provides more effective reasoning supervision than coarse-grained reasoning traces, while the relatively modest improvement on coarse-grained tasks indicates that structured reasoning supervision is particularly beneficial for more demanding fine-grained and compositional reasoning.
| Strategy | FG | CG | General | Complex | Overall |
| baseline | 60.25 | 86.77 | 73.91 | 22.11 | 58.08 |
| sft w/o CoT (95k) | 72.66 | 87.52 | 72.94 | 70.84 | 76.87 |
| sft w/ CCoT (95k) | 74.43 | 87.43 | 74.23 | 71.21 | 77.57 |
| sft w/ GCoT (95k) | 76.20 | 87.71 | 75.26 | 73.19 | 78.84 |
| GCoT vs Non-CoT | +3.54 | +0.19 | +2.32 | +2.35 | +1.97 |
| GCoT vs CCoT | +1.77 | +0.28 | +1.03 | +1.98 | +1.27 |
Table 6. Impact of different reasoning supervision strategies on various spatio-temporal tasks. (FG: Fine-grained Task, CG: Coarse-grained Task, CCoT: Coarse-grained CoT)
Conclusion
In this paper, we present V-STORM, a structure-aware framework for scalable construction of fine-grained video reasoning data. By leveraging Spatio-Temporal Scene Graphs (STSGs) as a unified intermediate representation, V-STORM supports graph-grounded question generation with verifiable spatio-temporal evidence. Based on this framework, we construct a training set and a human-verified benchmark for evaluating fine-grained perception, evidence localization, and compositional reasoning. Our results demonstrate the value of graph-grounded supervision for learning evidence-aware video reasoning while enabling systematic evaluation of reasoning capabilities. Experiments reveal that although current MLLMs can accurately localize relevant evidence, composing multiple grounded observations into correct reasoning remains a major challenge. We hope V-STORM serves as a valuable benchmark and training resource for future research on fine-grained video reasoning.
References
- [1]Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; Cheng, Z.; Deng, L.; Ding, W.; Gao, C.; Ge, C.; et al. 2025a. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631.arxiv.org/abs/2511.21631
- [2]Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025b. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923.arxiv.org/abs/2502.13923
- [3]Bain, M.; Nagrani, A.; Varol, G.; and Zisserman, A. 2021. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, 1728–1738.arxiv.org/abs/2104.00650
- [5]Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 24185–24198.
- [6]Dang, R.; Yuan, Y.; Zhang, W.; Xin, Y.; Zhang, B.; Li, L.; Wang, L.; Zeng, Q.; Li, X.; and Bing, L. 2025. ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark. In Proceedings of the Computer Vision and Pattern Recognition Conference, 24593–24602.arxiv.org/abs/2501.05031
- [7]Fang, X.; Mao, K.; Duan, H.; Zhao, X.; Li, Y.; Lin, D.; and Chen, K. 2024. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37: 89098–89124.arxiv.org/abs/2406.14515
- [8]Feng, K.; Gong, K.; Li, B.; Guo, Z.; Wang, Y.; Peng, T.; Wu, J.; Zhang, X.; Wang, B.; and Yue, X. 2025. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776.arxiv.org/abs/2503.21776
- [10]Ghazanfari, S.; Croce, F.; Flammarion, N.; Krishnamurthy, P.; Khorram, F.; and Garg, S. 2025. Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning. arXiv preprint arXiv:2506.00318.arxiv.org/abs/2506.00318
- [11]Han, S.; Huang, W.; Shi, H.; Zhuo, L.; Su, X.; Zhang, S.; Zhou, X.; Qi, X.; Liao, Y.; and Liu, S. 2025. VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection. In Proceedings of the Computer Vision and Pattern Recognition Conference, 26181–26191.arxiv.org/abs/2411.14794
- [12]Hong, W.; Cheng, Y.; Yang, Z.; Wang, W.; Wang, L.; Gu, X.; Huang, S.; Dong, Y.; and Tang, J. 2025. MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models. In Proceedings of the Computer Vision and Pattern Recognition Conference, 8450–8460.arxiv.org/abs/2501.02955
- [13]Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2401.21276.arxiv.org/abs/2410.21276
- [14]Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual instruction tuning. Advances in neural information processing systems, 36: 34892–34916.
- [15]Liu, Y.; Ma, Z.; Qi, Z.; Wu, Y.; Shan, Y.; and Chen, C. W. 2024. Et bench: Towards open-ended event-level video-language understanding. volume 37, 32076–32110.arxiv.org/abs/2409.18111
- [16]Meng, J.; Li, X.; Wang, H.; Tan, Y.; Zhang, T.; Kong, L.; Tong, Y.; Wang, A.; Teng, Z.; Wang, Y.; and Wang, Z. 2026. Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence. arXiv:2510.20579.arxiv.org/abs/2510.20579
- [18]Ravi, N.; Gabeur, V.; Hu, Y.-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; Rädle, R.; Rolland, C.; Gustafson, L.; et al. 2024. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714.arxiv.org/abs/2408.00714
- [19]Team, G.; Anil, R.; Borgéaud, S.; Alayrac, J.-B.; Yu, J.; Sorićut, R.; Schalkwyk, J.; Dai, A. M.; Hauth, A.; Millican, K.; et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
- [20]Wang, H.; Li, X.; Huang, Z.; Wang, A.; Wang, J.; Zhang, T.; Zheng, J.; Bai, S.; Kang, Z.; Feng, J.; et al. 2025a. Traceable evidence enhanced visual grounded reasoning: Evaluation and methodology. arXiv preprint arXiv:2507.09799.
- [21]Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.arxiv.org/abs/2409.12191
- [22]Wang, Q.; Yu, Y.; Yuan, Y.; Mao, R.; and Zhou, T. 2025b. Videorft: Incentivizing video reasoning capability in mllms via reinforced fine-tuning. arXiv preprint arXiv:2505.12434.arxiv.org/abs/2505.12434
- [23]Wang, W.; Gao, Z.; Gu, L.; Pu, H.; Cui, L.; Wei, X.; Liu, Z.; Jing, L.; Ye, S.; Shao, J.; et al. 2025c. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.arxiv.org/abs/2508.18265
- [24]Wang, X.; Zhang, X.; Zhu, Y.; Guo, Y.; Yuan, X.; Xiang, L.; Wang, Z.; Ding, G.; Brady, D.; Dai, Q.; et al. 2020. Panda: A gigapixel-level human-centric video dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 3268–3278.arxiv.org/abs/2003.04852
- [25]Wang, Y.; He, Y.; Li, Y.; Li, K.; Yu, J.; Ma, X.; Li, X.; Chen, G.; Chen, X.; Wang, Y.; et al. 2023. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942.arxiv.org/abs/2307.06942
- [26]Xiao, J.; Shang, X.; Yao, A.; and Chua, T.-S. 2021. NeXT-QA: Next Phase of Question-Answering to Explaining Temporal Actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9777–9786.arxiv.org/abs/2105.08276
- [28]Yuan, Y.; Dang, R.; Li, L.; Li, W.; Jiao, D.; Li, X.; Zhao, D.; Wang, F.; Zhang, W.; Xiao, J.; et al. 2025a. EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World? arXiv preprint arXiv:2506.05287.arxiv.org/abs/2506.05287
- [29]Yuan, Y.; Zhang, H.; Li, W.; Cheng, Z.; Zhang, B.; Li, L.; Li, X.; Zhao, D.; Zhang, W.; Zhuang, Y.; et al. 2025b. Videorefer suite: Advancing spatial-temporal object understanding with video llm. In Proceedings of the Computer Vision and Pattern Recognition Conference, 18970–18980.arxiv.org/abs/2501.00599
- [30]Zhang, B.; Li, K.; Cheng, Z.; Hu, Z.; Yuan, Y.; Chen, G.; Leng, S.; Jiang, Y.; Zhang, H.; Li, X.; et al. 2025a. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106.arxiv.org/abs/2501.13106
- [31]Zhang, S.; Hao, X.; Tang, Y.; Zhang, L.; Wang, P.; Wang, Z.; Ma, H.; and Zhang, S. 2025b. Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thought. In Proceedings of the 33rd ACM International Conference on Multimedia, 12745–12752.arxiv.org/abs/2506.08817



