betaFeedback

WorksComputer Science

VTG-RAG: Long-Video Reasoning via Query-Adaptive Coarse-to-Fine Sampling and Temporal Graph Retrieval

Abstract

Long-video reasoning requires a system to localize sparse evidence over extended temporal sequences, preserve temporal structure, and generate answers despite substantial multimodal distractors. Existing video retrieval-augmented generation methods typically build indexes from uniformly sampled video segments. In long-video settings, this design readily introduces irrelevant context and is unreliable for questions with explicit temporal constraints. We propose VTG-RAG, a framework for long-video question answering that combines query-adaptive coarse-to-fine sampling with temporal graph retrieval. First, a two-stage sampling strategy selects core segments with high mean query relevance or high intra-segment relevance variance. The framework then integrates subtitles, speech transcripts, visual descriptions, entities, and relations from these segments into a temporally ordered multimodal graph. Finally, it performs temporally filtered, two-pass retrieval and reasoning across frame, text, and graph evidence spaces. A lot of experiments have proven the effectiveness of the VTG-RAG. Specifically, we evaluate the framework on Video-MME and LongVideoBench. On LongVideoBench, the reported Qwen2.5-VL-32B and 72B results improve from 48.16% to 50.56% and from 52.13% to 53.32%, respectively, with 24 answer-input frames. These correspond to gains of 2.40 and 1.19 percentage points. The ablation experiments of VTG-RAG further support that the contributions of each component are very important.

*Corresponding author(s). E-mail(s): link;

Introduction

Long video reasoning is becoming an essential capability in the field of visual language models, supporting applications ranging from movie and sports analysis to surveillance and egocentric assistance [24, 32]. Unlike static images or short clips, long videos distribute visual and linguistic evidence across extended timelines, such that answering a question may require locating a brief event and relating it to observations occurring far earlier or later [17]. Directly processing all frames is often impractical because long video sequences can exceed the context capacity of current multimodal large language models and incur substantial computational and memory costs [19, 20]. Meanwhile, aggressive visual compression may discard fine-grained yet task-critical cues [16, 31]. Long-video reasoning therefore requires more than simply extending the context window: a system must localize sparse question-relevant evidence, preserve temporal relationships, and integrate information across distant segments into a compact representation for reliable reasoning [8, 25].

Recent long-context vision-language models and video retrieval-augmented generation methods have substantially advanced long-video understanding [18, 21, 22, 30]. Nevertheless, long-video understanding and moment-retrieval pipelines commonly begin with uniformly sampled frames or fixed-length clips before subsequent aggregation, retrieval, or reasoning [1, 4]. This strategy treats sampling as a fixed preprocessing step and allocates similar computational resources to temporal regions with substantially different relevance. Increasing the number of sampled frames may reduce the risk of missing evidence, but it also expands the context and introduces additional distractors. Conversely, using fewer uniformly sampled frames may discard decisive evidence before indexing, which downstream retrievers and answer models cannot recover [9]. This limitation cannot be fully addressed by selecting segments solely according to their average similarity to the question. Even when a segment appears only moderately relevant as a whole, it may contain a few question-critical frames whose sparse signals are overwhelmed by the surrounding irrelevant content [12, 13]. Moreover, when uniformly sampled segments are directly converted into multimodal chunks or graph nodes, irrelevant content propagates into the retrieval structure and reduces its information density. Effective long-video reasoning therefore requires sampling to be optimized jointly with downstream evidence organization. Existing sampling compression methods reduce redundant visual input [28, 34], while graph-based retrieval methods organize dispersed evidence for multi-hop reasoning [3, 7, 33]. However, these components are typically studied separately, leaving unresolved how question-dependent sampling should determine the compact, temporally structured evidence used by downstream retrieval and reasoning.

To address this gap, we propose VTG-RAG, which consists of query-adaptive coarse-to-fine sampling with temporal graph retrieval. As illustrated in Figure 1, the framework first scores video segments using both mean query relevance and intra-segment relevance variance. It then applies a second sampling stage to refine the candidate segments most likely to contain critical evidence. The retained segments are subsequently organized into a temporal multimodal graph that encodes visual descriptions, transcripts, entities, relations, and event order. Finally, evidence is jointly retrieved from frame, text, and graph spaces, with temporal filtering applied when an explicit query time is available.

Figure 1. Motivation for VTG-RAG. Uniform segmentation treats all temporal intervals equally, introducing irrelevant or temporally invalid evidence into long-video question answering. In contrast, VTG-RAG combines query-adaptive coarse-to-fine sampling with temporal graph retrieval to construct a compact and ordered evidence context, applying timestamp-based filtering when a query time is available

In this way, VTG-RAG converts a long and redundant video stream into a compact, structured, and question-relevant evidence space for downstream reasoning.

We evaluate VTG-RAG on datasets including Video-MME and LongVideoBench, which cover multimodal video understanding and long-context video question answering, respectively [5, 26]. Under matched visual-frame budgets, the framework produces accuracy improvements over the corresponding direct baselines across the evaluated vision–language backbones, including Qwen2.5-VL [2] models ranging from 7B to 72B parameters. The efficiency measurements further show that the retrieved evidence requires fewer tokens than the corresponding direct-inference contexts. Componentwise ablations on Video-MME show that performance decreases when query-adaptive sampling, temporal graph retrieval, or two-pass reasoning is removed. Together, these results show that efficient sampling and evidence organization jointly improve long-video reasoning.

The main contributions of this work are as follows:

  • We present a selection-before-indexing pipeline in which query-adaptive video sampling determines the evidence used for multimodal graph construction.

  • We connect the resulting graph with frame and text retrieval, followed by working-memory-guided evidence completion and answer generation.

  • We report benchmark comparisons and grouped ablations that characterize the complete pipeline under the evaluated configurations, while distinguishing these observations from untested mechanism-level claims.

Related Work

Video Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) is originally introduced for text-based, knowledge-intensive tasks. Its central idea is to explicitly retrieve external evidence to compensate for limitations in parametric memory and reduce hallucinations during generation [11]. With the development of multimodal large language models, this paradigm has been extended to video understanding. It addresses challenges in long-video settings, including excessively long contexts, dense visual tokens, and sparsely distributed critical evidence. MovieChat, MovieChat+, and LongVA improve long-video understanding through sparse memory modeling, question-aware compression, and long-context transfer, respectively [21, 22, 30]. VideoRAG and related studies further formulate long-video question answering explicitly as video retrieval-augmented generation. They support answer generation by constructing retrievable intermediate representations or directly retrieving relevant visual evidence from video corpora [10, 18]. However, most existing video RAG methods still use uniformly segmented clips or flat multimodal chunks as their basic indexing units. They neither screen critical temporal intervals before indexing nor explicitly model temporal structure. Consequently, these methods remain susceptible to noisy context in long-video question answering that requires sparse evidence localization, cross-segment aggregation, and temporally consistent reasoning.

Graph-Based Visual Understanding and Temporal Reasoning

Graph structures have been widely used in retrieval-augmented generation and visual understanding to represent complex relations and long-range dependencies that flat retrieval cannot capture. GraphRAG, LightRAG, and KG2RAG organize dispersed evidence into structured retrieval spaces suitable for multi-hop reasoning. They use entity graphs, lightweight graph-enhanced indexing, and knowledge-graph-guided context expansion, respectively [3, 7, 33]. In video understanding, EgoGraph further shows that temporal knowledge graphs can model entity interactions, event order, and state evolution across segments, thereby improving visual reasoning over long temporal spans [23]. These studies demonstrate that graph structures can strengthen relational representation and temporal modeling in complex visual reasoning. Existing approaches, however, generally assume that upstream video segments are already relatively clean. If a graph is constructed directly from many uniformly segmented and noisy video clips, its size can grow rapidly while its information density decreases, weakening the benefits of structured retrieval.

Evidence Compression

Efficiency improvements can target either model complexity or input redundancy. Lu et al. investigate layer pruning and post-pruning fine-tuning to reduce the computational overhead of large language models [15]. Unlike model compression, evidence compression reduces redundant input while retaining task-relevant information for downstream reasoning.

For textual inputs, Gao et al. formulate prompt compression as redundancy-aware pruning on a multiplex graph that captures local dependencies and global semantic relations [6]. Their method uses Lévy walks to identify informative nodes, illustrating a structure-aware approach to input compression. For visual inputs, FOCUS formulates keyframe selection in long videos as an upper-confidence-bound exploration problem, enabling training-free frame selection under a constrained visual-token budget [34]. FocusGraph further combines graph-structured scene representations with hierarchical evidence selection. It first identifies relevant clips and then refines keyframes within each clip for embodied long-video question answering [28].

These studies address complementary aspects of efficient inference and evidence selection. Our focus is on selecting sparse question-relevant video evidence and organizing it for temporally consistent reasoning. VTG-RAG therefore couples query-adaptive coarse-to-fine sampling with temporal graph retrieval. The retained evidence is organized by entities, relations, and temporal order before multi-space retrieval and reasoning.

Method

Figure 2 presents the overall architecture of VTG-RAG. Given a long video VV, a question qq, and candidate answers O={o1,…,oK}\mathcal{O} = \{o_1, \ldots, o_K\}, where KK is the number of answer options, the objective is to generate a video-grounded answer y^\hat{y}. When a question time tqt_q is available, retrieval is designed to exclude timestamped documents later than tqt_q, subject to the limitations described below. In Video-MME, time generally corresponds to the video playback timeline. For datasets that do not provide an explicit question time, such as LongVideoBench, the system retrieves evidence over the full video timeline. To this end, VTG-RAG follows three stages. First, query-adaptive sampling ranks temporal segments using their mean relevance and within-segment relevance variance, and then refines the top candidates into a compact set of core segments. Second, visual descriptions and transcripts from the retained segments are fused into multimodal chunks and organized as a temporal graph containing entities, relations, and event order. Finally, query rewriting and reasoning planning guide temporally constrained retrieval across frame, text, and graph spaces. The retrieved evidence is compressed into working memory and supplemented through a second retrieval pass before final answer generation. The central challenge is therefore to identify a compact yet sufficiently informative evidence set from a long, multimodal, and noisy video stream while preserving temporal consistency.

Figure 2. Overall architecture of VTG-RAG for long-video question answering. The framework comprises three modules. Query-Adaptive Coarse-to-Fine sampling combines variance-aware coarse scoring with candidate refinement to identify a compact set of core segments. Temporal graph construction integrates visual descriptions, transcripts, entities, relations, and temporal links into a multimodal graph. Time-constrained retrieval and reasoning combines query rewriting, reasoning planning, multi-space retrieval, and working memory to retrieve supporting evidence and generate the final answer

Query-Adaptive Coarse-to-Fine Sampling

Our sampler follows the two-stage optimistic selection principle of FOCUS [34]. We use a variance-thresholded exploration bonus to retain candidate segments, then resample these candidates and rank them by their refined mean relevance. The bonus is a heuristic adaptation of this selection principle, not a newly derived statistical confidence bound. Its role here is to determine which segments supply evidence for graph construction.

Temporal Segment Generation. In the current implementation, the input video is partitioned into fixed-length, non-overlapping 16-second segments {s1,…,sN}\{s_1, \ldots, s_N\}. From each segment sis_i, the system randomly samples kck_c frames to form the sequence

Fi={fi,1,…,fi,kc}F_i = \{f_{i,1}, \ldots, f_{i,k_c}\}. Each FiF_i is the basic unit for subsequent scoring, representation, and graph construction.

First-stage Coarse-grained Scoring. For each segment sis_i, the system uses an image–text similarity model to score the relevance of the kck_c sampled frames in FiF_i to question qq. The current implementation uses a SigLIP-based local similarity scorer [29]. Let the resulting similarity scores be {ri,1,…,ri,kc}\{r_{i,1}, \ldots, r_{i,k_c}\}. The mean score of segment ii is defined as

μi=1kc∑j=1kcri,j,(1)\mu_i = \frac{1}{k_c}\sum_{j=1}^{k_c} r_{i,j}, \tag*{(1)}

which measures the segment’s overall match to the query. We further define the within-segment variance as

σi2=1kc∑j=1kc(ri,j−μi)2,(2)\sigma_i^2 = \frac{1}{k_c}\sum_{j=1}^{k_c}(r_{i,j}-\mu_i)^2, \tag*{(2)}

which characterizes the variation in relevance within the segment. Unlike mean-only ranking, our approach also considers intra-segment relevance variance. A segment with moderate mean relevance may still contain a few highly relevant frames and therefore exhibit a large σi2\sigma_i^2. Ranking only by μi\mu_i may therefore discard segments containing decisive evidence at an early stage. We consequently define a variance-aware optimistic score as

s~i=μi+bi,(3)\tilde{s}_i = \mu_i + b_i, \tag*{(3)}

where bib_i is a variance-dependent exploration bonus that compensates for the potential under-ranking of segments containing sparse but critical evidence. Intuitively, bib_i measures whether a segment merits retention for further examination. Even when its mean relevance μi\mu_i is not among the highest, a segment may contain decisive evidence if a few sampled frames are highly relevant to the question. It should therefore receive an additional ranking bonus. Specifically, when the intra-segment variance σi2\sigma_i^2 does not exceed the threshold τ\tau, we set

bi=0;(4)b_i = 0; \tag*{(4)}

when σi2>τ\sigma_i^2 > \tau, we define

bi=α(2ln⁡T⋅σi2ni+3ln⁡Tni),(5)b_i = \alpha\left(\sqrt{\frac{2\ln T \cdot\sigma_i^2}{n_i}} + \frac{3\ln T}{n_i}\right), \tag*{(5)}

Here, nin_i is the number of frames sampled from segment ii, and T=NkcT = Nk_c is the nominal first-stage sampling budget for NN segments. The ln⁡T\ln T factor adjusts the bonus according to the global sampling scale. The square-root term captures the principal contribution of relevance variance, whereas 3ln⁡Tni\frac{3\ln T}{n_i} provides an additional conservative correction when the segment has been sparsely sampled. Thus, bib_i quantifies the extent to which a segment may be under-ranked by its mean score alone and allows the first stage to retain potentially critical segments for further examination. The coefficient α\alpha controls the strength of the bonus. For fixed TT and nin_i, and conditional on σi2>τ\sigma_i^2 > \tau, bib_i increases with σi2\sigma_i^2. Low variance indicates relatively uniform relevance and warrants little or no bonus. High variance suggests that a few sampled frames may carry critical evidence and therefore yields a larger bonus. For fixed TT and σi2\sigma_i^2, bib_i decreases as nin_i increases because denser sampling yields a more stable relevance estimate. In other words, σi2\sigma_i^2 measures the within-segment heterogeneity associated with potentially critical evidence, and bib_i converts this heterogeneity into a ranking bonus. The system ultimately ranks all segments by s~i\tilde{s}_i and retains a candidate set subject to ratio and count constraints for subsequent second-stage refinement and retrieval-based reasoning.

Second-stage Candidate Refinement

The second stage processes the candidate segments retained after ranking by the first-stage optimistic score. A second round of refinement sampling is performed only on these candidates. This design first retains potentially critical segments at relatively low cost and then concentrates additional computation on promising candidates, thereby controlling computational overhead.

The first-stage score is deliberately exploratory to avoid prematurely discarding potentially critical segments. However, the first stage estimates relevance from only a few sampled frames. It therefore favors recall and efficiency but may be sensitive to individual high-scoring frames. To obtain a more robust relevance estimate, VTG-RAG performs an additional round of sampling on the candidate segments retained after the first stage. Up to mm previously unsampled frames are drawn without replacement. Their relevance to the question is computed, and the new observations are combined with the first-stage samples to calculate a refined mean. Suppose that the first stage yields kck_c local similarity scores {ri,1,…,ri,kc}\{r_{i,1},\ldots,r_{i,k_c}\} and that the second stage samples an additional mm frames with scores {ri,1′,…,ri,m′}\{r'_{i,1},\ldots,r'_{i,m}\}. The refined mean is defined as

μ^i=1kc+m(∑j=1kcri,j+∑t=1mri,t′).(6)\hat{\mu}_i = \frac{1}{k_c + m}\left(\sum_{j=1}^{k_c} r_{i,j} + \sum_{t=1}^{m} r'_{i,t}\right). \tag*{(6)}

Unlike the first stage, the second stage no longer uses the optimistic score. Instead, it reranks the candidate segments according to the refined mean μ^i\hat{\mu}_i and uses this ranking to determine the final set of core segments. The first stage preserves potentially relevant segments to reduce the risk of missing critical evidence. The second stage increases sampling density and reduces the influence of occasional high-scoring frames, producing the final core-segment ranking. Only the selected core segments proceed to visual-description generation, transcript alignment, temporal graph construction, and retrieval-based reasoning.

Temporal Graph Construction

Entity-centric temporal graphs are an established representation for long-video understanding, as illustrated by EgoGraph [23]. We do not claim Person, Location, Object, and Event types or same-type entity fusion as independently novel mechanisms. Here, these mechanisms organize the evidence supplied by the preceding question-dependent segment selection stage.

Multimodal Representation of Core Segments. For each retained segment, VTG-RAG extracts two types of textual evidence. First, it selects the kcapk_{\mathrm{cap}} sampled frames with the highest query-similarity scores and generates their visual descriptions using the configured local vision–language model. Second, it incorporates ASR-derived transcripts as speech evidence. For LongVideoBench, the textual evidence consists of these transcripts and generated visual descriptions; official subtitles are not used by VTG-RAG. The two sources are then fused into a segment-level chunk:

ci=merge⁡(caption⁡i,transcript⁡i),(7)c_i = \operatorname{merge}(\operatorname{caption}_i,\operatorname{transcript}_i), \tag*{(7)}

where each chunk stores the segment identifier, start and end times, and fused textual content. This representation preserves both what occurred and when it occurred and serves as the basic node unit for subsequent temporal graph construction.

Chunk-level Temporal Backbone. After sorting the core chunks by start time, the system adds them as nodes in a directed multigraph. For each adjacent pair, it inserts a NEXT_IN_TIME edge from cic_i to ci+1c_{i+1}. This temporal backbone is essential for answering questions about event order, repeated mentions, and state transitions.

Entity and Relation Extraction. For each chunk, the system extracts entities and relations from the fused descriptive text. In the current implementation, entity types are restricted to Person, Location, Object and Event. Each extracted entity may carry attribute information, and each relation is stored together with its supporting evidence and confidence score. When the remote extraction model is unavailable, the system falls back to a rule-based extractor to maintain pipeline robustness.

Entity Fusion. The same person, location, object, or event may be mentioned repeatedly across different chunks. To reduce duplication and accumulate evidence over time, VTG-RAG fuses entities of the same type. The system first normalizes each entity to a canonical textual form and then computes embedding similarity. It merges a mention into an existing same-type node when their cosine similarity exceeds a threshold. The node then records the mention’s timestamp. Otherwise, the system creates a new node. The resulting graph is converted into retrievable evidence comprising visual descriptions, subtitles or transcripts, descriptions of temporal order, entity nodes, edges, and relations.

Time-constrained Retrieval and Reasoning

Query Rewriting and Reasoning Planning. Consider the original question: What is the role of the man who wears a T-shirt with the words “BLUE GHOST” on it? VTG-RAG rewrites it for cross-modal retrieval as follows: Find the video segments showing the man wearing a T-shirt with the words “BLUE GHOST”. Identify the surrounding scene, the man’s actions and interactions, and any spoken or visual clues that indicate his role. It then formulates the corresponding reasoning plan. First, the system retrieves segments showing the T-shirt bearing the words “BLUE GHOST” and locates the time range in which the man appears. Second, it uses the subtitles and visual descriptions to determine his actions, speech, and relationships with other people in the scene. Third, it retrieves entities, events, and adjacent segments associated with the man from the temporal graph to establish his identity or responsibility. Finally, it compares the evidence with the candidate answers and selects the role that is best supported. In the multiple-choice setting, the system also injects discriminative cues derived from the candidate answers to make subsequent evidence retrieval more sensitive to differences among the options. This process transforms the original question into a unified intermediate representation that supports both retrieval and reasoning, thereby providing a more explicit target for subsequent evidence aggregation.

Multi-space Retrieval with Temporal Filtering. The retriever supports an optional task-supplied query-time cutoff tqt_q. Candidate documents with known timestamps are filtered using their end time, or their start time when no end time is available, before ranking in each retrieval pass. Documents without timestamps are not excluded by this rule.

When no cutoff is supplied, the video duration is used as the temporal reference, leaving the full timeline eligible. The temporal-proximity score then acts as a recency preference rather than a causality constraint. LongVideoBench is evaluated in this full-timeline setting.

Retrieval proceeds across frame descriptions, transcripts, and graph-derived evidence, and the results are merged into a shared ranking. No dedicated prefix-only evaluation is reported, so the current results do not validate prevention of future-information leakage. Filtering document timestamps alone cannot exclude future-derived information already incorporated into fused graph entities or summaries.

Working-memory Compression and Second-pass Retrieval. Evidence obtained in the first retrieval pass is not sent directly to the answer-generation module. Instead, it is compressed into a compact working memory that retains the most relevant timeline summary, key facts, changes in entity states, and evidence for or against each candidate answer. Rather than answering in a single pass, VTG-RAG uses this working memory as an intermediate reasoning state to drive a second, complementary retrieval pass. This pass fills evidence gaps and suppresses distractors introduced during the initial retrieval. The answer model then generates the final prediction from the retrieved context, reasoning plan, and working memory, using timestamp-filtered retrieval when tqt_q is available. This two-pass mechanism implements coarse recall followed by targeted evidence completion within the reasoning loop.

Experiments

Datasets and Experimental Settings

We evaluate VTG-RAG on two benchmarks: Video-MME [5] and LongVideoBench [26]. Video-MME results are reported separately with and without official subtitles. For LongVideoBench, we use the official labeled validation set containing 753 videos and 1,337 multiple-choice questions. Since no explicit query-time cutoff is provided, retrieval covers the full video timeline.

For Video-MME, the two subtitle conditions indicate whether official subtitles are provided. ASR-derived transcripts remain available to VTG-RAG in both conditions. Therefore, the setting without official subtitles is not a visual-only evaluation.

For LongVideoBench, the direct baseline receives uniformly sampled video frames and official subtitles, without ASR-derived transcripts or generated visual descriptions. VTG-RAG uses ASR-derived transcripts and generated visual descriptions for evidence construction and retrieval. The paired configurations use the same vision–language backbone and answer-input frame count, but their textual inputs are not identical.

Videos are divided into non-overlapping 16-second segments. A SigLIP-based image–text scorer supports query-adaptive coarse-to-fine sampling. For LongVideoBench, each paired comparison uses the same vision–language backbone and answer-input frame count. The direct baseline receives uniformly sampled video frames and official subtitles. VTG-RAG instead uses selected video frames, ASR-derived transcripts, and generated visual descriptions, without official subtitles. The two pipelines therefore use different sources of textual evidence. Unless otherwise specified, the retrieved-text budget is 2.0K tokens; configuration-specific budgets are reported in Table 2.

ModelTextSizeFramesAcc. (%)Gain (pp)TokensAcc./1M tokens
LVLMs
LongVA–7B824.2–16.7M1.45
LongVA + VTG-RAG1.8K7B840.1+15.910.4M3.86
LLaVA–7B1618.8–11.3M1.65
LLaVA + VTG-RAG2.0K7B1635.2+16.44.8M7.33
InternVL3–8B1623.2–12.7M1.83
InternVL3-8B + VTG-RAG2.0K8B1636.8+13.66.8M5.41
Qwen2.5-VL–7B2434.9–21.6M1.62
Qwen2.5-VL + VTG-RAG2.1K7B2446.0+11.18.9M5.17

Table 2. Performance and token efficiency of different vision–language models on LongVideoBench. Text denotes the retrieved-text budget of VTG-RAG, and Gain denotes the absolute accuracy improvement over the direct baseline using the same model. Direct baselines use official subtitles, whereas VTG-RAG uses ASR-derived transcripts and generated visual descriptions without official subtitles. The Text column reports the retrieved-text budget of VTG-RAG; “–” does not indicate the absence of subtitle input for the direct baselines

Baselines

We compared VTG-RAG with LightRAG [7], VideoRAG [18], KG2^{2}RAG [33], and LongVA [30].

  • KG2^{2}RAG introduces knowledge graphs into retrieval-augmented generation. It extracts entities and relations from textual corpora, constructs a structured knowledge graph, and expands relevant evidence along the graph during retrieval. Although this organization benefits multi-hop question answering, the graph is

designed primarily for static textual knowledge and does not explicitly model video-segment order or cross-segment event evolution.

  • LongVA transfers the long-context capability of language models to visual understanding. It processes large numbers of video frames or visual tokens to model long videos end to end, providing direct access to information distributed over long temporal spans. However, it relies primarily on a large visual context for global reasoning and does not actively select question-relevant segments or construct structured evidence before inference.

  • VideoRAG combines static knowledge graphs with video content. It is developed for lecture videos, which generally exhibit less visual variation than egocentric videos, where head motion can produce substantial viewpoint changes. Its applicability to extremely long egocentric videos may therefore be limited.

  • LightRAG is a text-based retrieval-augmented generation method. It divides a long context into chunks and extracts nodes and edges from each chunk. This graph-based retrieval approach is particularly suitable for query-focused summarization. Our setting follows a related idea by tracking recurring entities across temporally separated segments in extremely long videos.

Under the configurations considered here, these methods do not provide the explicit temporal modeling targeted by VTG-RAG. In contrast, VTG-RAG constructs a temporally ordered multimodal graph for explicit temporal reasoning. Unless otherwise specified, SigLIP2 is used as the embedding model for all retrieval-based comparison methods to ensure a fair comparison. We also use the same question-answering model, Qwen2.5-VL-7B, for all retrieval-based methods.

Analysis of The Main Results. Table 1 shows that VTG-RAG obtained an Avg score of 42.34, higher than the three graph-based baselines shown. Because the systems differ in multiple components, this comparison alone does not isolate the effects of query-adaptive coarse-to-fine sampling or temporal graph retrieval. Table 2 presents the performance and efficiency of VTG-RAG with different vision–language backbones on LongVideoBench. For each backbone, we match the model parameter count and answer-input frame count. The direct baseline uses uniformly sampled frames and official subtitles, whereas VTG-RAG uses selected frames, ASR-derived transcripts, and generated visual descriptions without official subtitles. This comparison therefore evaluates complete pipelines with different textual evidence sources. With LongVA, LLaVA, InternVL3, and Qwen2.5-VL, VTG-RAG yields absolute accuracy gains of 15.9, 16.4, 13.6, and 11.1 percentage points, respectively. These gains are observed across the tested architectures, parameter counts, and visual-frame budgets, although the available results do not establish generalization beyond these settings.

ModelSizew/o official subtitlesw/ official subtitlesAvgΔOurs\Delta_{\mathrm{Ours}}
Proprietary LVLMs
Gemini 1.5 Pro–42.6948.4645.58–
GPT-4o–53.6259.1756.40–
Open-source LVLMs
LLaVA-OV7B35.0835.5235.30–
LongVA7B30.1633.2931.72–
InternVL38B28.9429.8229.38–
Graph-based methods
LightRAG–37.8040.7839.293.05
KG2RAG–32.4532.1632.3110.03
VideoRAG–33.7733.7733.778.57
Ours (Qwen2.5-VL)7B40.7843.8942.34/

Table 1. Comparison of the main results on Video-MME. ΔOurs\Delta_{\mathrm{Ours}} denotes the absolute improvement of VTG-RAG in Avg over the corresponding method. The subtitle conditions refer to the availability of official subtitles. VTG-RAG uses ASR-derived transcripts in both conditions; the condition without official subtitles is therefore not visual-only.

VTG-RAG uses only 1.8K–2.1K tokens of retrieved textual evidence and achieved higher accuracy under a fixed visual-frame budget. Under the evaluated input configurations, VTG-RAG uses fewer reported tokens than the corresponding direct baselines. Across the four completed settings, reported token use decreases from 11.3M–21.6M to 4.8M–10.4M, an average reduction of approximately 50%. Correspondingly, accuracy per 1M tokens increases from 1.45–1.83 to 3.86–7.33 percentage points per million tokens. Thus, the reported settings show concurrent improvements in answer accuracy and token-use efficiency.

The number of frames affects baseline performance, but more frames do not necessarily yield higher accuracy. For example, the Qwen2.5-VL direct baseline achieves the highest accuracy with 24 frames. LongVA, LLaVA, and InternVL3 still differ substantially when using 8 or 16 frames. These results indicate that frame count alone does not determine performance within the evaluated settings. The video-understanding capability of the backbone and its use of evidence may also have contributed. The paired settings use the same number of answer-input frames, so the accuracy gains do not arise from increasing this frame count. However, the textual evidence sources differ between the pipelines. These results therefore support the evaluated system configurations, but do not isolate the contributions of query-adaptive sampling or temporal graph retrieval.

Table 2 reports the comparisons using 7B and 8B backbones. Results for larger Qwen2.5-VL models are reported in § 4.3. Within these settings, VTG-RAG improves Qwen2.5-VL from 34.9% to 46.0% and InternVL3-8B from 23.2% to 36.8%. These observations suggest that Query-Adaptive Coarse-to-Fine sampling and temporal graph retrieval may complement the visual and language modeling capacity of the tested backbones.

Scaling to Larger Models

On the reported LongVideoBench configurations with 24 answer-input frames, Qwen2.5-VL-32B improves from 48.16% to 50.56%, a gain of 2.40 percentage points. Qwen2.5-VL-72B improves from 52.13% to 53.32%, a gain of 1.19 percentage points.

The larger model attains higher absolute accuracy, but its gain from VTG-RAG is smaller. These two configurations do not establish a monotonic relationship between model size and pipeline benefit. Statistical significance and robustness to sampling randomness remain to be established.

ModelRetrieved textSizeFramesAccuracy (%)Gain (pp)
Qwen2.5-VL–32B2448.16–
Qwen2.5-VL + VTG-RAG2.1K32B2450.56+2.40
Qwen2.5-VL–72B2452.13–
Qwen2.5-VL + VTG-RAG2.1K72B2453.32+1.19

Table 3. Scaling performance of VTG-RAG with larger Qwen2.5-VL models on the official labeled validation set of LongVideoBench. All configurations used 24 visual frames and the same evaluation protocol. Retrieved text denotes the textual-evidence budget. Gain is the absolute accuracy improvement, in percentage points, over the direct baseline with the same model size and visual-frame budget

Ablation Experiments

We conduct three grouped ablations to examine the contributions of query-adaptive sampling, graph retrieval, and the two-pass reasoning pipeline. Table 4 reports the results on Video-MME Long with subtitles. These variants evaluate module-level changes rather than isolate every operation within each module.

ConfigurationLong w/ subtitles (%)
Full model43.8
– w/o query-adaptive sampling40.7
– w/o graph retrieval42.1
– w/o two-pass reasoning42.9

Table 4. Core ablation experiments for VTG-RAG. All results are reported under the Video-MME Long setting with subtitles.

Role of Query-Adaptive Sampling. To evaluate the contribution of Query-Adaptive sampling, we construct a uniformly sampled variant. This variant replaces Query-Adaptive sampling in the first stage with uniform segment selection while retaining all other modules. Compared with the full model, its accuracy on Video-MME decreases from 43.8 to 40.7. This decrease is consistent with question-guided segment retention helping allocate computation to relevant context and retain key evidence.

Role of Temporal Graph Retrieval. To examine temporal graph retrieval, we construct a variant without graph retrieval. This variant retains frame-level visual evidence and textual transcripts but omits graph retrieval based on entity relations and temporal dependencies. Removing the temporal graph reduces performance on Video-MME to 42.1. This degradation is consistent with structured temporal information contributing to the aggregate performance of the full model.

Role of Two-pass Reasoning. To assess the contribution of two-pass reasoning, we further remove working-memory compression and second-pass retrieval. The model instead generates an answer directly from the first-pass retrieval results. This variant achieves 42.9 on Video-MME, compared with 43.8 for the full model. The result is consistent with working-memory compression and second-pass retrieval helping organize initial evidence and suppress redundant or distracting information.

Effect of Retrieval Frame Count Across Model Scales

To investigate how the retrieval frame count affects models of different scales, we evaluate Qwen2.5-VL-7B, Qwen2.5-VL-32B, and Qwen2.5-VL-72B using frame budgets of F∈{8,16,24}F \in\{8, 16, 24\}, while keeping all other settings fixed. As shown in Figure 3, increasing the number of retrieved frames consistently improves accuracy across all three model scales. Specifically, the accuracy of Qwen2.5-VL-7B increases from 32.4% at F=8F = 8 to 35.2% at F=16F = 16 and 46.0% at F=24F = 24. Similar trends are observed for Qwen2.5-VL-32B, whose accuracy rises from 33.2% to 36.4% and 50.6%, and for Qwen2.5-VL-72B, which improves from 36.8% to 42.9% and 53.3%. Overall, increasing the frame budget from 8 to 24 yields absolute gains of 13.6, 17.4, and 16.5 percentage points for the 7B, 32B, and 72B models, respectively. Larger models also consistently outperform smaller models under the same frame budget, indicating that increased model capacity facilitates more effective use of the retrieved visual evidence. The largest performance improvements generally occur when the frame count increases from 16 to 24, suggesting that smaller frame budgets may omit evidence required for long-video reasoning. Therefore, we use 24 retrieved frames in the main experiments, as this setting achieves the best performance among the evaluated frame budgets.

Figure 3. Effect of the retrieval frame count on Qwen2.5-VL models at different scales

Discussion

The central contribution of VTG-RAG is the coupling of query-adaptive evidence compression with structured temporal retrieval. The reported gains suggest that long-video question answering may depend on more than the strength of the retriever or answer model. Performance may also depend on actively controlling which video regions enter the reasoning pipeline.

The framework also embodies a design principle that may be useful for long-video RAG: temporal structure should be represented in the evidence structure and enforced during retrieval. VTG-RAG implements this principle at two levels. First, temporal information is encoded explicitly in the graph through chunk timestamps and NEXT IN TIME edges. Second, timestamped documents are filtered during retrieval using the timestamp ≤tq\le t_q criterion. As noted above, this document-level filter does not guarantee that fused graph evidence is free from future-derived information. This design could be useful for question-answering settings with causal boundaries, including EgoLifeQA and other settings in which an explicit question time is available [27].

The LongVideoBench comparison does not isolate the contribution of additional ASR and caption-derived evidence from that of evidence selection and organization.

The current system nevertheless has several limitations. First, graph quality depends directly on the quality of visual captions and entity–relation extraction. Weak captions or noisy extraction can make the graph sparse or even mislead retrieval. Second, the current graph schema is deliberately lightweight, which facilitates reproduction but provides limited support for abstract representations of complex events. Third, front-end sampling relies primarily on image–text similarity. It may therefore miss a critical segment when the decisive evidence lies mainly in linguistic detail rather than visual change. Future work could investigate stronger event-centric graph schemas, uncertainty-aware entity fusion, finer-grained subtitle–vision alignment, and adaptive segment lengths.

Another direction is to reduce the cost of annotating and validating textual evidence used in graph construction. AutoAnnotator coordinates large and small language models for efficient text annotation [14]. Adapting this collaborative strategy to video-derived descriptions and entity–relation annotations remains a direction for future investigation.

Conclusion

We introduce VTG-RAG, a framework that combines query-adaptive coarse-to-fine sampling and temporal graph retrieval for long-video question answering. The central idea is to compress a long video into a small set of question-relevant core segments before graph construction. The system then jointly retrieves frame, text, and graph evidence, applying an explicit temporal filter when a question time is available. This design retains sparse but important evidence while reducing irrelevant context during downstream reasoning. First, we develop a query-adaptive coarse-to-fine sampling strategy. It combines segment–question relevance with within-segment variance in relevance scores to locate informative segments along a long timeline and compress redundant context. Second, we construct a temporally ordered multimodal graph with entity relations. The graph unifies visual descriptions, subtitles or transcripts, and entity relations within a structured evidence space to support cross-segment evidence integration. Third, we design time-constrained multi-space retrieval and two-pass reasoning. Under the stated conditions, this process jointly retrieves frame, text, and graph evidence and uses working memory for evidence completion and answer inference. Across the evaluated Video-MME and LongVideoBench settings, VTG-RAG improves accuracy relative to the selected graph-based or paired direct baselines. The completed LongVideoBench settings also show higher token efficiency. These findings are limited to the reported datasets, backbones, and frame budgets; statistical significance remains to be established.

Statements and Declarations

Funding

This work was supported by the National Key R&D Program of China (2025YFA1510900), the Baima Lake Laboratory Joint Fund of the Zhejiang Provincial Natural Science Foundation of China (LBMHH25F020002), the Yangtze River Delta Science and Technology Innovation Community Joint Research Project (2025CSJGG01000 and 2026ZY03003), the Applied Basic Research Program of Zhejiang Province (2026C02A1264), and the National Natural Science Foundation of China (62673439).

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Author contributions

Yao Lu: Conceptualization, Methodology, Software, Writing - original draft, Visualization. Yunhao Wang: Conceptualization, Methodology, Writing - review & editing. Yaxin Gao: Validation, Formal analysis, Investigation. Feifei Teng: Conceptualization, Writing - original draft, Visualization. Chenxiao Liu: Conceptualization, Visualization. Zhe Tang: Investigation, Data curation. Jiaheng Wei: Investigation, Visualization. Zhaowei Zhu: Visualization, Writing - review & editing, Supervision. Shanqing Yu: Resources, Writing - review & editing, Supervision. Qi Xuan: Resources, Funding acquisition.

Data availability

The study uses the Video-MME and LongVideoBench benchmarks described in Refs. [5, 26]. The benchmark data and access instructions are available through the official project repositories at https://github.com/MME-Benchmarks/Video-MME and https://github.com/longvideobench/LongVideoBench, respectively. Access and reuse are subject to the conditions and licences specified by the original providers.

Code availability

The implementation is not currently deposited in a public repository. The code required to reproduce the reported experiments is available from the corresponding author upon reasonable request to support verification and reproducibility.

References

  1. [1]Afham M, Shukla SN, Poursaeed O, et al (2023) Revisiting kernel temporal segmentation as an adaptive tokenizer for long-form video understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshopsarxiv.org/abs/2309.11569
  2. [2]Bai S, Chen K, Liu X, et al (2025) Qwen2.5-VL technical report. arXiv preprint, URL https://arxiv.org/abs/2502.13923, arXiv:2502.13923
  3. [3]Edge D, Trinh H, Cheng N, et al (2024) From local to global: A graph RAG approach to query-focused summarization. arXiv preprint, URL https://arxiv.org/abs/2404.16130, arXiv:2404.16130
  4. [4]Fang X, Liu D, Fang W, et al (2024) Fewer steps, better performance: Efficient cross-modal clip trimming for video moment retrieval using language. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 1735–1743
  5. [5]Fu C, Dai Y, Luo Y, et al (2024) Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. arXiv preprint, URL https://arxiv.org/abs/2405.21075, arXiv:2405.21075
  6. [6]Gao Y, Lu Y, Deng J, et al (2026) Mapping text to multiplex graph: Prompt compression as Lévy walk-guided graph pruning. Knowledge-Based Systems 351:116661arxiv.org/abs/2607.01241
  7. [7]Guo Z, Xia L, Yu Y, et al (2024) LightRAG: Simple and fast retrieval-augmented generation. arXiv preprint, URL https://arxiv.org/abs/2410.05779, arXiv:2410.05779
  8. [8]Hannan T, Islam MM, Gu J, et al (2025) ReVisionLLM: Recursive vision-language model for temporal grounding in hour-long videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitionarxiv.org/abs/2411.14901
  9. [9]Hu K, Gao F, Nie X, et al (2025) M-LLM based video frame selection for efficient video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 13702–13712arxiv.org/abs/2502.19680
  10. [10]Jeong S, Kim K, Baek J, et al (2025) VideoRAG: Retrieval-augmented generation over video corpus. arXiv preprint, URL https://arxiv.org/abs/2501.05874, arXiv:2501.05874
  11. [11]Lewis P, Perez E, Piktus A, et al (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systemsarxiv.org/abs/2005.11401
  12. [12]Li Y, Xiao J, Feng C, et al (2023) Discovering spatio-temporal rationales for video question answering. In: Proceedings of the IEEE/CVF International Conference on Computer Visionarxiv.org/abs/2307.12058
  13. [13]Liu S, Zhao C, Xu T, et al (2025) BOLT: Boost large vision-language model without training for long-form video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitionarxiv.org/abs/2503.21483
  14. [14]Lu Y, Ji Z, Du J, et al (2025) AutoAnnotator: A collaborative annotation framework for large and small language models. Transactions on Machine Learning Research
  15. [15]Lu Y, Cheng H, Fang Y, et al (2026) Reassessing layer pruning in LLMs: New insights and methods. In: International Conference on Learning Representations
  16. [16]Qian R, Dong X, Zhang P, et al (2024) Streaming long video understanding with large language models. In: Advances in Neural Information Processing Systemsarxiv.org/abs/2405.16009
  17. [17]Ren S, Yao L, Li S, et al (2024) TimeChat: A time-sensitive multimodal large language model for long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitionarxiv.org/abs/2312.02051
  18. [18]Ren X, Xu L, Xia L, et al (2025) VideoRAG: Retrieval-augmented generation with extreme long-context videos. arXiv preprint, URL https://arxiv.org/abs/2502.01549, arXiv:2502.01549
  19. [19]Shen X, Xiong Y, Zhao C, et al (2025) LongVU: Spatiotemporal adaptive compression for long video-language understanding. In: Proceedings of the 42nd International Conference on Machine Learningarxiv.org/abs/2410.17434
  20. [20]Shu Y, Liu Z, Zhang P, et al (2025) Video-XL: Extra-long vision language model for hour-scale video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitionarxiv.org/abs/2409.14485
  21. [21]Song E, Chai W, Wang G, et al (2023) MovieChat: From dense token to sparse memory for long video understanding. arXiv preprint, URL https://arxiv.org/abs/2307.16449, arXiv:2307.16449
  22. [22]Song E, Chai W, Ye T, et al (2024) MovieChat+: Question-aware sparse memory for long video question answering. arXiv preprint, URL https://arxiv.org/abs/2404.17176, arXiv:2404.17176
  23. [23]Sun S, Han K, Huang Y, et al (2026) EgoGraph: Temporal knowledge graph for egocentric video understanding. arXiv preprint, URL https://arxiv.org/abs/2602.23709, arXiv:2602.23709
  24. [24]Wang W, He Z, Hong W, et al (2025) LVBench: An extreme long video understanding benchmark. In: Proceedings of the IEEE/CVF International Conference on Computer Visionarxiv.org/abs/2406.08035
  25. [25]Wang Z, Yu S, Stengel-Eskin E, et al (2025) VideoTree: Adaptive tree-based video representation for LLM reasoning on long videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitionarxiv.org/abs/2405.19209
  26. [26]Wu H, Li D, Chen B, et al (2024) LongVideoBench: A benchmark for long-context interleaved video-language understanding. arXiv preprint, URL https://arxiv.org/abs/2407.15754, arXiv:2407.15754
  27. [27]Yang J, Liu S, Guo H, et al (2025) EgoLife: Towards egocentric life assistant. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitionarxiv.org/abs/2503.03803
  28. [28]Zemskova T, Andryushenko S, Obrubov I, et al (2026) FocusGraph: Graph-structured frame selection for embodied long video question answering. arXiv preprint, URL https://arxiv.org/abs/2603.04349, arXiv:2603.04349
  29. [29]Zhai X, Mustafa B, Kolesnikov A, et al (2023) Sigmoid loss for language image pre-training. In: Proceedings of the IEEE/CVF International Conference on Computer Visionarxiv.org/abs/2303.15343
  30. [30]Zhang P, Zhang K, Li B, et al (2024) Long context transfer from language to vision. arXiv preprint, URL https://arxiv.org/abs/2406.16852, arXiv:2406.16852
  31. [31]Zhang Z, Pang Z, Chen S, et al (2025) One token per highly selective frame: Towards extreme compression for long video understanding. In: Advances in Neural Information Processing Systems
  32. [32]Zhou J, Shu Y, Zhao B, et al (2025) MLVU: Benchmarking multi-task long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitionarxiv.org/abs/2406.04264
  33. [33]Zhu W, Wang H, Lin Y, et al (2025) Knowledge graph-guided retrieval augmented generation. arXiv preprint, URL https://arxiv.org/abs/2502.06864, arXiv:2502.06864
  34. [34]Zhu Z, Xu H, Luo Y, et al (2025) FOCUS: Efficient keyframe selection for long video understanding. arXiv preprint, URL https://arxiv.org/abs/2510.27280, arXiv:2510.27280

Paper details

vlnc:W_3b8aj625
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents