betaFeedback

WorksComputer Science

DiffGround: Semantic Difference-Guided Zero-Shot Video Question Grounding

Abstract

Video Question Grounding (VideoQG) requires both accurate question answering and precise temporal localization. Existing supervised methods rely on costly temporal annotations, while existing zero-shot approaches are primarily designed for Video Question Answering (VQA) and are not optimized for temporal grounding. We propose DiffGround, an action-driven zero-shot framework built upon frozen Multimodal Large Language Models (MLLMs). Our framework first predicts an answer-relevant action cue that transforms the input question into an explicit semantic target for grounding. Action Cue converts a reasoningoriented question into an action-centric retrieval query. Guided by the action cue, a Vision-Language Model (VLM) efficiently selects informative keyframes for reasoning. Finally, we introduce Semantic Diff Refinement, a training-free temporal refinement method that exploits semantic differences between neighboring SigLIP2 embeddings to recover complete action segments from sparse predictions. Extensive experiments on NExT-GQA and STAR show that DiffGround consistently outperforms existing zero-shot and supervised methods, establishing a new state of the art for Video Question Grounding.

Introduction

Video Question Grounding (VideoQG) is an emerging multimodal task that requires models to not only answer questions about video content but also accurately localize the temporal segments supporting their answers. Compared with conventional Video Question Answering (VQA), VideoQG requires both semantic reasoning and fine-grained temporal localization, making it a fundamental challenge for video understanding with applications in video retrieval, human-computer interaction, and embodied AI.

Existing approaches can be broadly categorized into supervised and zero-shot methods, both of which suffer from important limitations. Supervised methods rely on expensive temporal annotations and often exhibit a trade-off between VQA accuracy and grounding precision. More recently, Multimodal Large Language Models (MLLMs) have enabled zero-shot VideoQG without additional training. However, existing methods are primarily designed for VQA rather than temporal grounding, limiting their ability to accurately localize actions.

We identify two fundamental challenges underlying existing zero-shot approaches. First, they directly use the input question as the retrieval query for frame selection. However, natural language questions are designed for reasoning rather than visual retrieval, containing interrogative expressions, temporal references, and contextual descriptions that provide limited guidance for identifying relevant visual evidence. As a result, current methods tend to retrieve frames containing the correct objects instead of the target action, leading to object-centric rather than action-centric frame selection [18]. Second, sparse keyframe predictions alone are insufficient for accurate temporal localization. Existing approaches often compensate with dense frame sampling or multiple MLLM calls, resulting in substantial computational cost while still failing to recover complete action boundaries.

To address these challenges, we propose DiffGround, an action-driven zero-shot Video Question Grounding framework, illustrated in Figure 1. DiffGround first performs an offline action segmentation by measuring semantic differences between neighboring Vision-Language Model (VLM) embeddings, producing reusable action segments for each video. Given a question, a frozen MLLM performs an initial reasoning step to generate an Action Cue, a concise action-centric description that serves as an explicit retrieval target. Guided by both the question and the predicted action cue, the VLM efficiently identifies informative keyframes. Finally, Semantic Diff Refinement expands sparse grounding predictions using the precomputed semantic transitions to recover complete action boundaries.

Overview of the proposed DiffGround framework

Figure 1. Overview of the proposed DiffGround framework. Given a video and a question, DiffGround first performs offline, question-agnostic action segment preprocessing to partition the video into semantically coherent action segments. A frozen MLLM then predicts an action cue describing the target action with a single MLLM call. Guided by the question and predicted action cue, a Vision-Language Model (VLM) ranks video frames to select the most informative keyframes for Video Question Answering (VQA). Finally, Semantic Diff Refinement expands the initial grounding prediction by leveraging SigLIP2 semantic differences to align temporal boundaries with the complete action segment, producing more accurate Video Question Grounding (VideoQG) results.

Our framework is built upon three key innovations. First, we introduce an Action Cue Prediction module that converts reasoning-oriented questions into concise action-centric representations, substantially improving frame retrieval quality with only a single MLLM inference. Second, we propose a VLM-guided keyframe selection strategy that jointly matches the question and predicted action cue, enabling efficient action-aware frame retrieval. Third, we introduce Semantic Diff Refinement, a training-free temporal refinement method that exploits semantic differences between adjacent SigLIP2 embeddings to recover complete action segments. Compared with conventional motion-based segmentation methods, Semantic Diff directly captures semantic transitions, operates in the semantic embedding space rather than relying on low-level motion cues, and requires minimal hyperparameter tuning. Since visual embeddings are extracted once and reused across multiple questions associated with the same video, the preprocessing is performed entirely offline, significantly reducing online inference cost.

Extensive experiments on NExT-GQA [16] and STAR [15] demonstrate that DiffGround consistently outperforms existing zero-shot and supervised approaches, establishing a new state of the art in both Video Question Grounding and Video Question Answering.

Related Works

Caption and Agent based Methods

Recent zero-shot frameworks leverage textual intermediate representations to facilitate long-video understanding. LLoVi [21] and MoReVQA [7] first generate dense video captions and index them into textual memories for retrieval-based reasoning. Meanwhile, agent-based frameworks, such as VideoAgent [13] and BOLT [6], formulate video understanding as an iterative planning process, where an MLLM repeatedly queries localized video segments through multiple reasoning steps.

These approaches demonstrate strong zero-shot reasoning capability by exploiting the language understanding ability of MLLMs. However, they generally require numerous MLLM calls during either offline indexing or online planning, leading to considerable computational overhead. Moreover, because temporal reasoning is performed primarily through textual representations, the resulting temporal localization is constrained by the granularity of generated captions, making precise boundary localization challenging. In contrast, our approach directly operates on visual representations while requiring only a single MLLM inference for each query.

Training-based Methods

Training-based Video Question Grounding methods can be broadly categorized into weakly supervised and temporally supervised approaches.

Weakly supervised methods, such as CRA [2] and TOGA [3], are trained solely with video-level question-answer pairs. While avoiding dense temporal annotations, they typically rely on coarse visual-textual correlations and often struggle to precisely localize action boundaries.

Temporally supervised approaches explicitly exploit temporal annotations to improve grounding performance. Representative methods include ActPrompt [14], which introduces action-centric prompts to guide temporal attention, and fully supervised frameworks such as Grounded-VideoLLM [12], VideoStreaming [8], and SeViLA [19], which fine-tune large video-language models using dense temporal annotations.

Unlike these methods, DiffGround is a training-free framework that requires no parameter updates or temporal annotations. Instead, it performs action-aware grounding through Action Cue Prediction and Semantic Diff Refinement, enabling strong generalization to reasoning-oriented VideoQG tasks.

Frame Sampling-based Methods

To reduce the computational cost of long-video understanding, recent zero-shot methods perform sparse frame sampling before MLLM inference. Representative approaches include TStar [18], Adaptive Key Frame Sampling [10], and Q-Frame [22], which select informative frames based on spatial saliency, cross-attention, or vision-language similarity. By avoiding dense video rendering and task-specific fine-tuning, these methods provide an efficient solution for zero-shot VideoQA.

However, existing frame sampling strategies are primarily designed for question answering rather than temporal grounding. First, they retrieve discrete keyframes without explicitly recovering continuous temporal spans. Consequently, temporal boundaries must be inferred from sparse observations, often leading to inaccurate localization. Second, their retrieval strategies remain largely object-centric, emphasizing visual entities instead of action dynamics. As a result, they may successfully identify relevant objects while missing the temporal evolution of the target action. Our proposed DiffGround addresses both challenges through Action Cue Prediction for action-aware frame retrieval and Semantic Diff Refinement for recovering complete action segments from sparse predictions.

Method

Overview

Given a video V\mathcal{V} and a question qq, DiffGround predicts both the answer and its supporting temporal segment through an offline preprocessing stage and an online grounding stage, as illustrated in Figure 2.

Overview of the proposed DiffGround framework

Figure 2. Overview of the proposed DiffGround framework. DiffGround consists of three stages. First, a frozen MLLM predicts an Action Cue from the input question and uniformly sampled video frames, converting a reasoning-oriented question into an action-centric retrieval query. Second, a frozen VLM (SigLIP2) jointly matches the question and Action Cue to retrieve the Top-KK most informative keyframes. Meanwhile, an offline Semantic Diff Segmentation module partitions the video into semantically coherent action segments. Finally, the retrieved keyframes are used for Video Question Answering, and the predicted timestamps are refined by the precomputed semantic segments to recover complete temporal boundaries for Video Question Grounding.

During offline preprocessing, we partition the video into semantically coherent action segments by measuring semantic differences between neighboring SigLIP2 [11] visual embeddings. These action segments are computed once for each video and reused for all subsequent questions.

During online inference, a frozen MLLM first performs an initial reasoning step to generate an Action Cue, a concise action-centric description of the target event. The Action Cue is combined with the original question to retrieve informative keyframes using a Vision-Language Model (VLM). The selected keyframes are then fed into the MLLM to predict the final answer. Finally, Semantic Diff Refinement aligns the retrieved keyframes with the precomputed action segments to recover accurate temporal boundaries.

Offline Semantic Diff Segmentation

The Top-KK keyframes selected by the VLM provide sparse observations of the target action but do not directly reveal its complete temporal extent. We therefore seek to partition each video into semantically coherent action segments, allowing sparse frame predictions to be naturally expanded into complete temporal intervals during inference.

Instead, we propose Semantic Diff Refinement, a training-free temporal refinement framework that leverages semantic transitions in a Vision-Language Model (VLM) embedding space. The key observation is that modern VLMs encode high-level semantic information, allowing changes between neighboring embeddings to naturally indicate scene transitions and action boundaries. Since the visual embeddings are computed once for each video, the entire refinement process is performed offline and reused for all subsequent questions.

Semantic Diff Feature Extraction Given an input video V\mathcal{V}, we first partition it into NN consecutive one-second clips,

C={c1,c2,…,cN}.(1)\mathcal{C} = \{c_1, c_2, \ldots, c_N\}. \tag*{(1)}

For each clip cic_i, we uniformly sample kk frames,

Fi={fi,1,…,fi,k},\mathcal{F}_i = \{f_{i,1}, \ldots, f_{i,k}\},

and extract their visual embeddings using the frozen SigLIP2 vision encoder,

vi,j=fVLM(fi,j)∈Rd.(2)v_{i,j} = f_{\mathrm{VLM}}(f_{i,j}) \in\mathbb{R}^{d}. \tag*{(2)}

Instead of explicitly estimating motion, we characterize temporal changes directly in the VLM embedding space. For the ii-th clip, we define its semantic displacement as

si=vi,k−vi,1k−1,(3)s_i = \frac{v_{i,k} - v_{i,1}}{k - 1}, \tag*{(3)}

where vi,1v_{i,1} and vi,kv_{i,k} denote the embeddings of the first and last sampled frames, respectively. The resulting sequence

S={s1,s2,…,sN}\mathcal{S} = \{s_1, s_2, \ldots, s_N\}

provides a compact representation of semantic changes across consecutive video clips.

Adaptive Greedy Semantic Segmentation Given the sequence S\mathcal{S}, our objective is to partition the video into semantically coherent action segments. Rather than specifying the number of segments manually, we first estimate the target number of segments using semantic change-point detection. For each pair of adjacent clip representations, we compute

di=∥si+1−si∥2.(4)d_i = \lVert s_{i+1} - s_i \rVert_2. \tag*{(4)}

Large values of did_i indicate significant semantic transitions. We therefore estimate an adaptive threshold,

τ=μD+ασD,(5)\tau= \mu_D + \alpha\sigma_D, \tag*{(5)}

Algorithm 1 Adaptive Greedy Semantic Diff Segmentation

Require: Semantic clip features S={s1,…,sN}S = \{s_1,\ldots,s_N\}, threshold coefficient α\alpha

Ensure: Semantic action segments A\mathcal{A}

Stage I: Estimate Target Segment Number

1: for i=1i = 1 to N−1N - 1 do 2: di←∥si+1−si∥2d_i \leftarrow\lVert s_{i+1} - s_i \rVert_2 3: end for 4: τ←μD+ασD\tau\leftarrow\mu_D + \alpha\sigma_D 5: M←∣{di>τ}∣+1M \leftarrow|\{d_i > \tau\}| + 1

Stage II: Greedy Semantic Merging

6: Initialize A={(ci,si)}i=1N\mathcal{A} = \{(c_i,s_i)\}_{i=1}^{N} 7: while ∣A∣>M|\mathcal{A}| > M do 8: i∗←arg⁡max⁡icos⁡(A[i].f,A[i+1].f)i^* \leftarrow\arg\max_i \cos(\mathcal{A}[i].f,\mathcal{A}[i+1].f) 9: Merge A[i∗]\mathcal{A}[i^*] and A[i∗+1]\mathcal{A}[i^* + 1] 10: Update feature using duration-weighted average 11: Remove A[i∗+1]\mathcal{A}[i^* + 1] 12: end while 13: return A\mathcal{A}

where μD\mu_D and σD\sigma_D denote the mean and standard deviation of {di}\{d_i\}, respectively. The target number of action segments is then estimated as

M=∣{di>τ}∣+1.(6)M = |\{d_i > \tau\}| + 1. \tag*{(6)}

Starting from NN atomic clips, we greedily merge neighboring segments with the highest semantic similarity. Specifically, the adjacent pair

i∗=arg⁡max⁡icos⁡(si,si+1)(7)i^* = \arg\max_i \cos(s_i,s_{i+1}) \tag*{(7)}

is merged during each iteration. The feature of the merged segment is updated using a duration-weighted average to preserve semantic consistency. The merging process continues until the number of remaining segments equals the estimated target value MM.

Algorithm 1 summarizes the complete adaptive greedy segmentation procedure.

Action Cue Prediction

Existing zero-shot methods directly use the input question as the retrieval query for frame selection. However, natural language questions are designed for reasoning rather than visual retrieval, often containing interrogative phrases, temporal references, and contextual descriptions that dilute the retrieval signal. Consequently, the retrieved frames frequently correspond to relevant objects instead of the target action.

To address this issue, we introduce an Action Cue Prediction module that performs an initial reasoning step before frame retrieval. Given the input question qq and a small set of uniformly sampled frames Funiform\mathcal{F}_{\mathrm{uniform}}, a frozen MLLM generates a concise action-centric description:

c=fMLLM(q,Funiform),(8)c=f_{\mathrm{MLLM}}(q,\mathcal{F}_{\mathrm{uniform}}), \tag*{(8)}

where cc denotes the predicted Action Cue. Unlike the original question, which is optimized for reasoning, the Action Cue explicitly describes the target action (e.g., “The boy in red touches the child”), providing a semantically focused retrieval target for subsequent frame selection.

VLM-guided Keyframe Selection

Given a set of candidate frames F={fi}i=1T\mathcal{F}=\{f_i\}_{i=1}^{T}, we encode each frame using the SigLIP2 vision encoder,

F=VLMvision(F)∈RT×d.(9)F=\mathrm{VLM}_{\mathrm{vision}}(\mathcal{F})\in\mathbb{R}^{T\times d}. \tag*{(9)}

The question and Action Cue are independently encoded using the SigLIP2 text encoder,

Qq=VLMtext(q),(10)Q_q=\mathrm{VLM}_{\mathrm{text}}(q), \tag*{(10)}
Qc=VLMtext(c),(11)Q_c=\mathrm{VLM}_{\mathrm{text}}(c), \tag*{(11)}

and combined into a unified representation,

Q=Qq+Qc.(12)Q=Q_q+Q_c. \tag*{(12)}

Frame relevance is computed using cosine similarity,

S=QFT,(13)S=QF^{T}, \tag*{(13)}

and the Top-KK highest-scoring frames are selected for downstream VQA. The Action Cue complements the original question by emphasizing action semantics, while the question preserves contextual information. Their combination provides a stronger retrieval signal than either representation alone.

Online Semantic Diff Refinement

The semantic segmentation described above is performed entirely offline and shared by all questions associated with the same video.

During online inference, the VLM first retrieves the Top-KK keyframes using the unified query representation described in § 3.4. As illustrated in Figure 2, the selected keyframes are then fed into a second MLLM call to jointly predict the answer and the supporting keyframe indices. Each predicted keyframe is subsequently mapped to its corresponding precomputed semantic action segment. Instead of returning isolated frame timestamps, DiffGround expands the sparse keyframe predictions to the complete action segments, producing temporally coherent grounding results with negligible additional computation.

Figure 3 illustrates a representative example. Besides separating two distinct scenes (“playing piano” and “woman with child”), Semantic Diff further identifies the brief “touch child” interaction as an independent action segment. Although the retrieved keyframe corresponds to only a single instant, Semantic Diff Refinement successfully recovers the entire temporal interval (38.8–40.9 s), enabling accurate Video Question Grounding.

Visualization of the proposed Semantic Diff segmentation

Figure 3. Visualization of the proposed Semantic Diff segmentation. By measuring semantic transitions between neighboring clips, our offline preprocessing partitions the video into semantically coherent action segments. Besides separating distinct scenes (e.g., piano playing versus woman and child), it further identifies fine-grained action boundaries, isolating the brief “touch child” interaction (38.8–40.9 s) from surrounding activities. During online inference, Semantic Diff Refinement aligns the retrieved keyframe with the corresponding action segment, producing accurate temporal grounding.

To the best of our knowledge, DiffGround is the first zero-shot Video Question Grounding framework to leverage semantic differences in Vision-Language Model (VLM) embeddings for action segmentation. Since semantic features are precomputed offline, Semantic Diff efficiently captures both scene transitions and fine-grained action boundaries with negligible online overhead.

Experiments

Datasets

We evaluate our method on two standard benchmarks: the NExT-GQA test set [16] and the STAR validation set [15], as its test set is not publicly released. Detailed statistics for both datasets are presented in Table 1.

DatasetVid.Que.Seg.Seg. Dur.(s)Vid. Dur. (s)Ratio (S./V.)
NExT-GQA990555366006.739.50.2
STAR9147098709811.930.00.4

Table 1. Statistic of the evaluation sets. Note: We utilize NExT-GQA [16]’s test set and STAR [15]’s validation set for evaluation. By employing a column-width resizing wrapper, this table maintains a consistent style and layout alignment with our main quantitative results.

NExT-GQA [16]. NExT-GQA selects long videos with an average length of 40 seconds. Each video has multiple questions, and the answers’ grounding can overlap. The videos include a sequence of atomic events, and the questions involve interaction between multiple actors and objects. Questions are of two

types: causal why/how questions and temporal questions with when/before/after clauses. Causal questions, such as Why are there two men standing at the center island and holding their camera?, require localizing the evidence of the answer, which is Recording for this question. Temporal questions, such as What did the boy do after the green man walked past him?, require understanding the temporal evolution of the events to generate an answer such as Look at man in green. The train set consists of 3,870 videos and 34,132 QA pairs without the grounding annotations. The test set consists of 990 videos with 5,553 QA pairs. Though the dataset provides multiple-choice answers, we generate open-ended responses without observing the options.

STAR [15]. STAR consists of about 60K situated reasoning questions with programs and answers, 240K candidate choices, and 22K trimmed situation video clips. Situation video clips in the benchmark are sourced from human activity videos, which record the dynamic interaction processes of human actions and surrounding environments in daily-life scenes. It also provides about 144K situation hypergraphs as structured situation abstraction. Designed questions cover four types of skills for situated reasoning. The dataset includes annotated questions with answers and options. Each question answering corresponds to a specific program for reasoning logic. To connect situation abstraction and reasoning diagnosis for question-answering, STAR provides situation hypergraphs tied with executable programs. Then situations, questions, and options are aligned with the unified data type schema, including actions, objects, humans, and relations. The STAR includes 111 action predicates, 28 objects, and 24 relationships. The benchmark is split into training/validation/test sets with a ratio of about 6:1:1.

Evaluation Metrics

We consider five metrics: intersection over union (IoU), intersection over prediction (IoP), [email protected], [email protected], and Acc@GQA proposed in NExT-GQA [16]. IoU measures the overlap between ground truth and predicted groundings. IoP measures the portion of predicted grounding containing the ground truth, similar to precision. While these metrics focus on grounding, Acc@GQA focuses on both the correctness of the answers and correct grounding with IoP ≤ 0.5. Apart from these metrics, we consider the Acc@QA metric for QA performance.

Implementation Details

Experimental Setup. We evaluate DiffGround on the NExT-GQA test set (5,553 video-question pairs) and the STAR validation set (7,098 video-question pairs). Unless otherwise specified, GPT-4o [4] is used as the Multimodal Large Language Model (MLLM), while SigLIP2 serves as the Vision-Language Model (VLM). The temperature is set to 0.0 for Action Cue Prediction to ensure deterministic outputs and 0.2 for the final Video Question Answering inference.

Offline Semantic Segmentation. The only dataset-specific hyperparameter is the threshold coefficient α\alpha used in Algorithm 1. We set α=−0.25\alpha= -0.25 for NExT-GQA and α=0.25\alpha= 0.25 for STAR according to the average ground-truth answer duration of each dataset. Smaller values produce finer-grained action segments, whereas larger values generate coarser temporal partitions.

Implementation Details. All experiments are conducted on servers equipped with two NVIDIA RTX 4090 GPUs (24GB memory each). Additional implementation details, prompt templates, comparisons with alternative VLMs (CLIP and SigLIP2) and MLLMs (e.g., Qwen2.5-VL and Qwen3-VL), additional ablation studies, and the proposed Acc@GQA (IoU) evaluation metric are provided in the supplementary material.

Frame Retrieval Variants

To analyze the contribution of different retrieval queries, we compare four frame retrieval variants while keeping the VLM retrieval pipeline unchanged.

Uniform Sampling. Frames are sampled uniformly across the video and directly passed to the VQG prompt (Figure 4(b)) without Action Cue Prediction.

Prompt template design showing (a) a first prompt template for Action Cue Prediction and (b) a second prompt template for Final Prediction

Figure 4. Prompt Template Design. The prompt for our final VQG prediction, which is adapted from TStar [18].

Question Only. Frame relevance is computed using only the question embedding, QqQ_q. The selected frames are then processed using the VQG prompt (Figure 4(b)).

Action Cue Only. The Action Cue is first generated using the Action Cue Prediction prompt (Figure 4(a)). Frame relevance is computed using only the Action Cue embedding, QcQ_c, and the selected frames are fed into the VQG prompt (Figure 4(b)).

Object Cue Only. The Action Cue Prediction prompt (Figure 4(a)) is modified to output a single object noun (e.g., “boy”) instead of an action description. Frame relevance is computed using the corresponding object embedding, and the selected frames are processed using the VQG prompt (Figure 4(b)).

Detailed prompt templates for all variants are provided in the supplementary material.

Comparing to State-of-the Arts

NExT-GQA. Table 2 compares DiffGround with existing supervised and zero-shot approaches on NExT-GQA. Despite requiring only two MLLM calls and no additional training, DiffGround achieves the best overall performance across all evaluation metrics. Compared with the strongest zero-shot baseline, our method improves Acc@GQA by 4.5 percentage points while simultaneously achieving the highest mIoP and mIoU. These results demonstrate that Action Cue Prediction and Semantic Diff Refinement effectively improve both question answering and temporal localization without increasing computational complexity.

Table 2: Quantitative Results on NExT-GQA. Note: NN means the number of sampled frames with 1 fps. * indicates that the methods do not require any post-training. By employing a column-width resizing wrapper, all model names are preserved in full within the single-column layout.

MethodVenueModel# of framesAcc@GQA (↑)mIoP (↑)mIoU (↑)# of calls
IGV [5]CVPR’22-1610.221.414.0-
SeViLA [19]NIPS’23BLIP2416.629.521.7-
*LLoVi [21]EMNLP’24GPT-4 + CogAgent-26.839.421.5≈ NN
VideoStreaming [8]NIPS’24Phi-2 2.7B + Vicuna-7B1617.832.219.81
CRA [2]CVPR’25Temp[CLIP]3218.228.614.2-
CRA [2]CVPR’25FrozenBiLM3218.826.513.5-
Grounded-VideoLLM [12]EMNLP’25Phi3.59626.734.521.11
TOGA [3]ICCV’25-1624.640.524.6-
*Baseline (Uniform)-GPT-4o830.538.925.51
*Our-GPT-4o + SigLIP2835.042.725.52
Human---82.172.161.2-

Table 2.

STAR. Table 3 further validates the generalization of DiffGround on the STAR benchmark. Our framework consistently achieves the best performance in both question answering and grounding while using only two MLLM calls. Compared with uniform sampling, DiffGround significantly improves Acc@GQA and IoP@5 while also producing the highest IoU@5 among all compared methods. These results suggest that the proposed action-driven framework generalizes well across datasets with different video durations and reasoning styles.

Table 3: Quantitative Results on STAR. Note: * indicates that the methods do not require any post-training.

MethodVenueModel# of framesAcc@GQA (↑)Acc@QA (↑)IoP@5 (↑)IoU@5 (↑)# of calls
IGV [5]CVPR’22-1613.131.742.81.7-
SeViLA [19]NIPS’23BLIP2417.045.537.619.5-
CRA [2]CVPR’25Temp[CLIP]3226.858.644.55.5-
CRA [2]CVPR’25FrozenBiLM3227.560.543.15.1-
*Baseline (Uniform)-GPT-4o425.759.341.418.91
*Our-GPT-4o + SigLIP2433.361.151.920.12

Table 3.

Effectiveness of Action Cue Enhanced Key Frame Sampling

Table 4 and Table 5 analyze different frame retrieval strategies. Using the question alone substantially improves Acc@GQA over uniform sampling, confirming that semantic retrieval is considerably more effective than fixed sampling. However, this improvement comes at the expense of temporal grounding, as IoU consistently decreases on both datasets. This observation indicates that questions are optimized for reasoning rather than visual retrieval, often identifying a few highly discriminative frames while overlooking the complete temporal extent of the target action.

Method# of framesAcc@GQA (↑\uparrow)Acc@QA (↑\uparrow)mIoP (↑\uparrow)mIoU (↑\uparrow)# of calls
Baseline (Uniform)830.577.938.925.51
Question (Q)837.377.045.020.01
Object Cue (OC)834.077.341.418.72
Action Cue (AC)838.178.246.120.02
Our (Q + AC)838.5 (+8%)78.7 (+0.8%)46.1 (+7.2%)19.9 (-5.6%)2
Our (Q + AC + SD)835.0 (+4.5%)78.7 (+0.8%)42.7 (+3.8%)25.5 (+0%)2

Table 4. Ablation Study of Our Approach on NExT-GQA. Note: Percentage gains indicate relative improvements over the Baseline (Uniform) configuration. SD=Semantic Diff Refinement.

Method# of framesAcc@GQA (↑\uparrow)Acc@QA (↑\uparrow)IoP@5 (↑\uparrow)IoU@5 (↑\uparrow)# of calls
Baseline (Uniform)425.759.341.418.91
Question (Q)431.055.353.18.21
Object Cue (OC)430.760.747.27.42
Action Cue (AC)433.161.349.87.52
Our (Q + AC)433.6 (+7.9%)61.1 (+1.8%)52.2 (+11.8%)7.7 (-11.2%)2
Our (Q + AC + SD)433.3 (+7.6%)61.1 (+1.8%)51.9 (+11.5%)20.1 (+1.2%)2

Table 5. Ablation Study of Our Approach on STAR. Percentage gains indicate relative improvements over the Baseline (Uniform) configuration. SD=Semantic Diff Refinement.

We further compare our Action Cue with a simplified Object Cue that contains only the primary object noun. Although both cues improve retrieval over the baseline, the Action Cue consistently achieves higher Acc@GQA and IoP. This demonstrates that explicitly modeling actions and object interactions provides a more informative retrieval target than isolated object descriptions.

Finally, combining the original question with the predicted Action Cue achieves the strongest retrieval performance across both datasets. Nevertheless, despite producing the highest Acc@GQA, the retrieved keyframes still exhibit relatively low IoU, suggesting that frame retrieval alone is insufficient for accurate temporal localization. This motivates the Semantic Diff Refinement introduced in the next section.

Effectiveness of Semantic Diff Refinement

Table 4 and Table 5 evaluate the effectiveness of the proposed Semantic Diff Refinement. Comparing “Our (Question + Action Cue)” with the complete DiffGround framework shows that Semantic Diff consistently improves temporal grounding performance on both NExT-GQA and STAR. In particular, mIoU increases from 19.9% to 25.5% on NExT-GQA and from 7.7% to 20.1% on STAR, demonstrating that the proposed refinement effectively recovers more complete action boundaries from sparse keyframe predictions.

Interestingly, Semantic Diff Refinement slightly decreases Acc@GQA (IoP) and mIoP while substantially improving IoU. This behavior reflects the different optimization objectives of Video Question Answering and Video Question Grounding. Action Cue Prediction is designed to retrieve a small number of highly discriminative frames that maximize question answering accuracy. Such compact predictions often achieve high overlap precision but cover only part of the target action, leading to relatively low temporal IoU. In contrast, Semantic Diff Refinement expands these sparse predictions to semantically coherent action segments, producing predictions that more closely align with the complete temporal extent of the ground-truth action.

Consequently, the refined segments may include additional contextual frames that contribute little to frame-level retrieval metrics, resulting in a slight reduction in Acc@GQA (IoP) and mIoP. However, they provide substantially more accurate temporal localization, as reflected by the consistent improvements in mIoU. These results suggest that optimizing frame retrieval alone is insufficient for Video Question Grounding, and that explicit temporal refinement is necessary to recover complete action segments.

A complementary analysis using an IoU-based variant of Acc@GQA is provided in the supplementary material, further illustrating the trade-off between retrieval-oriented and localization-oriented evaluation.

Conclusions

We introduced DiffGround, a training-free, action-driven framework for zero-shot Video Question Grounding. Our key insight is that transforming reasoning-oriented questions into action-centric representations and refining sparse predictions using semantic transitions enables accurate and efficient temporal grounding. Extensive experiments on NExT-GQA and STAR demonstrate that DiffGround establishes a new state of the art while remaining computationally efficient.

Appendix: Ablation Study of Different VLMs

In Table 6, we analyze the impact of different Vision-Language Model (VLM) backbones on our framework. We compare the strongest variants from three prominent model families: CLIP (ViT-bigG-14) [9], SigLIP (ViT-SO400M-14) [11], and SigLIP2 (ViT-SO400M-16) [11]. The results demonstrate that SigLIP2 yields the most robust overall performance. Driven by its superior ImageNet accuracy (84.1%), it achieves the highest scores across the majority of tasks, securing 35.9% Acc@GQA and 42.7% mIoP. While CLIP remains competitive in mIoU, SigLIP2’s significant advantages in VQA accuracy and intersection-over-prediction make it the optimal choice for our unified query selection and semantic difference calculation.

VLMDataIN-1K# of framesAcc@GQAAcc@QAmIoPmIoU
CLIP [9]LAION-2B80.1834.278.141.525.7
SigLIP [20]WebLI83.1833.977.842.025.4
SigLIP2 [11]WebLI84.1835.978.742.725.5

Table 6. Comparison of different VLMs on NExT-GQA. “Train Data” denotes the pre-training dataset, and “IN-1K” denotes ImageNet-1K top-1 accuracy.

Appendix: A New Evaluation Metric for Video Question Grounding: Acc@GQA (IoU)

To provide a more rigorous evaluation of temporal precision, we introduce a complementary metric, Acc@GQA (IoU). The standard Acc@GQA metric [16] relies on Intersection over Prediction (IoP@5). While useful, this metric can be misleading: because it measures only precision (containment), an extremely short prediction—even a single timestamp—can achieve a perfect score as long as it falls within the ground truth. This fails to penalize models that capture only a fraction of the full event duration.

To measure how accurately the predicted segment aligns with the ground truth, we define Acc@GQA (IoU) using an Intersection over Union (IoU) threshold of 0.3. As shown in Table 7, this metric reveals the true impact of our pipeline. While adding the Semantic Difference Refinement (SD) causes a slight drop in the containment-focused Acc@GQA (IoP) (from 38.5% to 35.0%), it drives a substantial improvement in the alignment-focused Acc@GQA (IoU), boosting it from 24.5% to 32.0%. This confirms that our refinement module effectively expands the prediction to match the true action boundaries, trading "safe," partial containment for precise, full-event grounding.

Method# of framesAcc@GQA (IoP)Acc@GQA (IoU)Acc@QAmIoPmIoU
Q+AC838.524.578.746.119.9
Q+AC+SD835.032.078.742.725.5

Table 7. Complementary evaluation using Acc@GQA (IoU). SD=Semantic Diff Refinement.

Performance of Utilizing Open-source MLLMs

In Table 8, we evaluate the generalization capability of our framework by replacing GPT-4o with state-of-the-art open-source MLLMs: Qwen2.5-VL-Instruct-7B [1] and Qwen3VL-Instruct-8B [17]. While these models achieve reasonable VQA performance (e.g., 74.8% Acc@QA for Qwen2.5), they exhibit a significant drop in temporal grounding accuracy, with mIoU scores falling below 16%. Qualitative analysis reveals that unlike GPT-4o [4], these smaller open-source models struggle to follow the strict formatting constraints of our prompt. They frequently hallucinate repetitive or generic index ranges (e.g., consistently outputting "1, 2" or "4, 5") regardless of the video content, likely due to a lack of exposure to this specific grounding-via-indexing task during their instruction tuning. This highlights that while open-source MLLMs are closing the gap in visual reasoning, they still lag behind proprietary models in complex instruction following and structured output generation.

MLLM# of framesAcc@GQAAcc@QAmIoPmIoU
Qwen2.5-VL [1]823.774.831.714.8
Qwen3-VL [17]819.972.928.015.5
GPT-4o835.078.742.725.5

Table 8. Comparison of different MLLMs on NExT-GQA. All methods use the same retrieval pipeline; only the MLLM is changed.

Prompt Template Design for Object Cue Predictor

Figure 5 illustrate the prompt templates designed for our Object Cue Predictor. For a detailed quantitative analysis of these components, please refer to the ablation study in § 4.6.

Prompt Template for Object Cue Prediction. Here is a video: <image> <image> ... <image>. Here is a question about the video: Question: ... Options: (A) ... (B) ... (C) ... (D) ... (E) ... When answering this question about the video: 1. Identify key objects that can locate the answer (list key objects, separated by commas). 2. Identify cue objects that might be near the key objects and appear in the scenes (list cue objects, separated by commas). 3. Rank key objects according to the importance to the answer of the question (list ranked objects, separated by commas). Provide your answer in three lines, listing the key objects, cue objects, and key action descriptions separated by commas.

Figure 5. Prompt Template Design for Object Cue. The prompt for generating object cues, which is adapted from TStar [18].

Computational Cost

Each MLLM call takes approximately 3 secs. DiffGround requires two online MLLM calls plus approximately 1 sec for retrieval/refinement, resulting in about 7 secs/query online latency. Offline SigLIP2 features and Semantic Diff segments are computed once per video and reused across questions.

References

  1. [1]Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025) 18arxiv.org/abs/2502.13923
  2. [2]Chen, W., Liu, Y., Chen, B., Su, J., Zheng, Y., Lin, L.: Cross-modal causal relation alignment for video question grounding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 24087–24096 (2025) 4, 12arxiv.org/abs/2503.07635
  3. [3]Gupta, A., Roy, A., Chellappa, R., Bastian, N.D., Velasquez, A., Jha, S.: Toga: Temporally grounded open-ended video qa with weak supervision. arXiv preprint arXiv:2506.09445 (2025) 4, 12arxiv.org/abs/2506.09445
  4. [4]Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024) 11, 18arxiv.org/abs/2410.21276
  5. [5]Li, Y., Wang, X., Xiao, J., Ji, W., Chua, T.S.: Invariant grounding for video question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2928–2937 (2022) 12arxiv.org/abs/2206.02349
  6. [6]Liu, S., Zhao, C., Xu, T., Ghanem, B.: Bolt: Boost large vision-language model without training for long-form video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 3318–3327 (2025) 3
  7. [7]Min, J., Buch, S., Nagrani, A., Cho, M., Schmid, C.: Morevqa: Exploring modular reasoning models for video question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13235–13245 (2024) 3arxiv.org/abs/2404.06511
  8. [8]Qian, R., Dong, X., Zhang, P., Zang, Y., Ding, S., Lin, D., Wang, J.: Streaming long video understanding with large language models. Advances in Neural Information Processing Systems 37, 119336–119360 (2024) 4, 12arxiv.org/abs/2405.16009
  9. [9]Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021) 17arxiv.org/abs/2103.00020
  10. [10]Tang, X., Qiu, J., Xie, L., Tian, Y., Jiao, J., Ye, Q.: Adaptive keyframe sampling for long video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 29118–29128 (2025) 4arxiv.org/abs/2502.21271
  11. [11]Tschannen, M., Gritsenko, A., Wang, X., Naeem, M.F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al.: Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786 (2025) 5, 17arxiv.org/abs/2502.14786
  12. [12]Wang, H., Xu, Z., Cheng, Y., Diao, S., Zhou, Y., Cao, Y., Wang, Q., Ge, W., Huang, L.: Grounded-videollm: Sharpening fine-grained temporal grounding in video large language models. arXiv preprint arXiv:2410.03290 (2024) 4, 12arxiv.org/abs/2410.03290
  13. [13]Wang, X., Zhang, Y., Zohar, O., Yeung-Levy, S.: Videoagent: Long-form video understanding with large language model as agent. In: European Conference on Computer Vision. pp. 58–76. Springer (2024) 3arxiv.org/abs/2403.10517
  14. [14]Wang, Y., Jiang, X., Cheng, D., Li, D., Zhao, C.: Actprompt: In-domain feature adaptation via action cues for video temporal grounding. IEEE Transactions on Image Processing (2026) 4DOI
  15. [15]Wu, B., Yu, S., Chen, Z., Tenenbaum, J.B., Gan, C.: Star: A benchmark for situated reasoning in real-world videos. arXiv preprint arXiv:2405.09711 (2024) 1, 3, 9, 10arxiv.org/abs/2405.09711
  16. [16]Xiao, J., Yao, A., Li, Y., Chua, T.S.: Can I trust your answer? visually grounded video question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13204–13214 (2024) 1, 3, 9, 10, 17arxiv.org/abs/2309.01327
  17. [17]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025) 18arxiv.org/abs/2505.09388
  18. [18]Ye, J., Wang, Z., Sun, H., Chandrasegaran, K., Durante, Z., Eyzaguirre, C., Bisk, Y., Niebles, J.C., Adeli, E., Fei-Fei, L., Wu, J., Li, M.: Re-thinking temporal search for long-form video understanding (2025), https://arxiv.org/abs/2504.02259 2, 4, 11, 19
  19. [19]Yu, S., Cho, J., Yadav, P., Bansal, M.: Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems 36, 76749–76771 (2023) 4, 12arxiv.org/abs/2305.06988
  20. [20]Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975–11986 (2023) 17arxiv.org/abs/2303.15343
  21. [21]Zhang, C., Lu, T., Islam, M.M., Wang, Z., Yu, S., Bansal, M., Bertasius, G.: A simple llm framework for long-range video question-answering. arXiv preprint arXiv:2312.17235 (2023) 3, 12arxiv.org/abs/2312.17235
  22. [22]Zhang, S., Yang, J., Yin, J., Luo, Z., Luan, J.: Q-frame: Query-aware frame selection and multi-resolution adaptation for video-llms. arXiv preprint arXiv:2506.22139 (2025) 4arxiv.org/abs/2506.22139

Paper details

vlnc:W_zv75eey6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents