SAPO: Self-Supervised Attention Policy Optimization for Improving Temporal Video Grounding in Vision-Language Models

Introduction

Vision–language models (VLMs) have become a general interface for video understanding Bai et al. [1]. Beyond recognizing what happens, a capable video VLM must also determine precisely when an event occurs. Temporal video grounding (TVG) directly evaluates this ability by requiring a model to localize the start and end times of an event described by a natural-language query. Despite their strong general video understanding, current VLMs remain substantially less reliable when generating precise timestamps.

Recent studies attribute much of this weakness to insufficient alignment between visual features and absolute time. Existing methods therefore inject frame-level temporal information through interleaved textual timestamps, timestamp overlays, or time-aligned position embeddings [23, 8, 1] (Wu et al. 2025). Their improvements establish that VLMs predict time more accurately when the correspondence between visual evidence and temporal coordinates is made explicit.

Given the substantial gains from explicitly mapping visual inputs to temporal coordinates, we ask whether a VLM already contains internal evidence for temporal localization but loses precision when generating timestamps. To investigate this question, we compare the model’s explicitly generated predicted interval with an attention interval decoded from its temporal attention over video tokens. We evaluate both intervals by their IoU with the ground-truth interval and count the proportion of examples for which the attention interval is more accurate. Across four VLMs, attention achieves a higher IoU than prediction on an average of 66.0% of examples on Charades-STA and 65.8% on ActivityNet-Captions. We term this systematic discrepancy the temporal generation gap: the model’s attention can localize the queried event more accurately than its explicit timestamp prediction, as illustrated in Figure 1.

Although attention does not provide perfectly accurate boundaries, its relative advantage reveals a useful source of internal temporal supervision. Since annotating precise start and end timestamps for video–query pairs is highly costly, this finding motivates us to exploit unlabeled video–query data and improve temporal grounding directly through attention-based self-supervision.

Based on this insight, we propose Self-Supervised Attention Policy Optimization (SAPO), which uses internal attention to construct RL rewards for timestamp generation. Each selected video–query pair is processed by a trainable policy and a frozen reference model initialized from the same VLM. The policy samples a group of timestamp intervals, while the reference extracts a stationary temporal-attention distribution. To avoid the sparse supervision of directly using interval IoU as the reward, SAPO projects each rollout interval onto the video timeline with a Gaussian kernel and measures its Wasserstein-1 distance to the reference attention distribution, yielding a dense temporal reward. Group-relative advantage estimation then updates the policy, with the reference model also providing KL regularization. SAPO thus transfers internal temporal evidence into native timestamp prediction without ground-truth boundaries for training.

Not all video–query pairs, however, contain equally useful attention supervision. We therefore explore how to identify a higher proportion of valuable training examples without ground truth. Among the examined criteria, top-$k$ selection by Jensen–Shannon divergence between the prediction and attention distributions provides a lightweight solution and retrieves more high-value examples for RL than random sampling.

We evaluate SAPO on three benchmarks using both Qwen2.5-VL models and Time-R1-7B, a strong temporal grounding model, without temporal annotations during training. SAPO yields relative mIoU improvements of up to 33.7% on Qwen2.5-VL-3B and 5.4% on Time-R1-7B. Gains under cross-dataset training further demonstrate transfer across data domains, while the Video-MME accuracy of Qwen2.5-VL-3B remains close to the base model (58.0 vs. 57.4), indicating that SAPO largely preserves general video understanding. Further ablations verify the effects of high-value sample selection and the individual training objectives. Our contributions are summarized as follows:

• Across multiple VLMs and grounding datasets, we identify a temporal generation gap: internal temporal attention often localizes queried events more accurately than explicitly generated timestamps.

• We propose Self-Supervised Attention Policy Optimization (SAPO), which uses internal attention to improve temporal grounding through reinforcement learning without temporal-boundary annotations.

• Experiments across models, datasets, and training sources demonstrate consistent grounding improvements for both general-purpose VLMs and a strong temporal grounding model, together with cross-dataset transfer and largely preserved general video understanding.

Related Work

Vision–Language Models. Recent VLMs combine visual encoders with large language models to support general image and video understanding [1] (Li et al. 2024; Chen et al. 2024). Although these models exhibit strong semantic reasoning ability, precise temporal grounding remains challenging. Our work directly improves the temporal grounding ability of a VLM without changing its architecture or requiring temporal annotations.

Temporal Video Grounding. Temporal video grounding localizes the video segment described by a natural-language query. Conventional methods learn this task from annotated temporal intervals [24, 11]. Recent generative VLMs instead formulate grounding as native timestamp generation. They improve grounding through architectural designs, temporally grounded instruction data and supervised fine-tuning [8, 16, 19, 22, 6, 18, 5, 7], or more effective post-training algorithms such as reinforcement learning and on-policy distillation [20] (Li et al. 2026). Despite their different approaches to exploiting supervision, these methods rely on annotated temporal targets.

Training-free methods avoid temporal annotations by constructing external workflows that match frame-level information from frozen models with the query [10, 26] (Qu et al. 2024). Although they can produce temporal intervals without training, they do not improve the native grounding ability of the underlying VLM. SAPO differs from both supervised and training-free methods: it uses self-supervision from the model’s own attention to improve its native timestamp generation through policy optimization, without temporal annotations.

Model Mechanism Analysis. Prior studies inspect internal model states to understand how visual evidence is encoded before the final response [25, 14]. They show that model outputs may not fully reflect the information available in intermediate representations. Following this perspective, we analyze temporal attention in generative VLMs and identify a gap between internal localization and explicit timestamp generation.

Preliminary

Given an untrimmed video $V$ and a natural-language query $q$, temporal video grounding aims to localize the temporal interval corresponding to the queried event. We distinguish three intervals: the ground-truth interval $I_g$ (used only for evaluation and controlled analysis), the predicted interval $I_p$ generated by the model, and the attention interval $I_a$ decoded from its temporal attention. For each example, we determine which interval is more accurate by comparing $\operatorname{IoU}(I_a, I_g)$ with $\operatorname{IoU}(I_p, I_g)$.

We conduct this analysis with four VLMs on all 3,720 Charades-STA examples and the same fixed random sample of 4,000 examples from the 17,505 ActivityNet examples. We obtain $I_a$ from the last prompt token’s cross-modal attention to the video tokens. We average over attention heads, pool spatial tokens within each frame to obtain a normalized temporal profile $a(t)$, where $\sum_t a(t) = 1$. We decode this profile by finding the shortest contiguous frame window whose cumulative attention mass reaches a threshold $\tau$:

where $T$ is the number of sampled video frames and $\tau= 50\%$. We then map the frame indices $[i^*,j^*]$ to their corresponding timestamps to obtain $I_a$. The model-specific layers, sensitivity to the layer and threshold, and further extraction details are provided in the Appendix. Representing both attention and prediction as temporal intervals enables a direct comparison under the same IoU metric and avoids favoring attention merely because it is represented as a smooth distribution.

Table 1 provides quantitative evidence for a consistent temporal generation gap. Across all eight model–dataset combinations, the attention interval achieves higher mIoU than the predicted interval, and attention is more accurate on the majority of valid examples. The averaged intervals in Figure 2 exhibit the same pattern. This gap likely arises because temporal attention operates directly in video-token position space, whereas timestamp generation must further convert token positions into calibrated numerical seconds, introducing additional localization error. The availability of this effective internal supervision, together with the high cost of temporal boundary annotation, motivates us to improve timestamp generation through attention-based self-supervision without human-annotated boundaries.

Model	Data	Attn. wins{sup}†`	Attn. mIoU	Pred. mIoU
Qwen2.5-VL-3B	Charades	65.0	39.5	36.5
	ActivityNet	71.0	32.8	24.6
Qwen2.5-VL-7B	Charades	62.0	46.1	43.6
	ActivityNet	67.0	32.4	27.4
InternVL3-2B	Charades	79.0	39.2	15.2
	ActivityNet	64.0	23.9	16.9
Qwen3-VL-4B	Charades	58.0	46.9	44.7
	ActivityNet	61.0	43.5	41.3

Method

Building on the preceding finding, we propose Self-Supervised Attention Policy Optimization (SAPO), which uses internal temporal attention as self-supervision to improve the native grounding ability of a VLM. As illustrated in Figure 3, for each unlabeled video–query pair, a trainable policy samples timestamp intervals, while a frozen reference model provides a stationary temporal attention distribution. SAPO measures their alignment as the rollout reward, estimates group-relative advantages, and updates the policy. We next describe the complete optimization procedure and the selection of high-value samples without ground truth. Implementation details are provided in the Appendix.

Self-Supervised Attention Policy Optimization

Given an unlabeled sample $(V,q)$, the policy $\pi_\theta$ generates a group of rollouts $\{\hat{y}_k\}_{k=1}^{K}$, each containing a timestamp interval $\hat{I}_{\hat{y}_k}=[\hat{s}_k,\hat{e}_k]$. In parallel, the reference model $\pi_{\theta_0}$ produces a temporal attention profile over the video frames, which is corrected and normalized into $a \in\Delta^T$. Decoding $a$ with (1) gives the attention interval $I_a$. The frozen reference model therefore provides two related forms of temporal supervision: the continuous reference attention distribution $a$ and its decoded attention interval $I_a$. The temporal distribution reward operates directly on $a$, whereas the attention-interval IoU reward operates on $I_a$.

Directly using interval IoU as the reward produces sparse feedback: when rollouts do not overlap with $I_a$, they all receive zero reward regardless of their temporal distance from the attended region. SAPO therefore converts each rollout interval into a continuous distribution $g(\hat{I}_{\hat{y}})$ using a Gaussian kernel and compares it with the corrected attention distribution $a$ before threshold-based interval decoding. SAPO scores each rollout by

where $a$ provides attention-derived temporal supervision.

(i) Temporal distribution reward. The first term measures agreement between the rollout-induced distribution and the reference attention distribution using the Wasserstein-1 distance. Because $W_1$ increases with the temporal transport required to match $a$, it provides graded feedback even for non-overlapping rollouts. (ii) Attention-interval IoU reward. The second term measures temporal IoU between the generated interval and the interval $I_a$ decoded from the reference attention distribution, providing a sparse interval-level complement to the distribution reward. (iii) Format-validity reward. $r_{\mathrm{fmt}}$ rewards a well-formed interval ($0 \le s < e \le\mathrm{dur}$), preventing malformed outputs. (iv) Interval-width regularization. $p_{\mathrm{width}}$ penalizes near-full-video intervals to discourage a trivial duration-prior solution.

For each video–query pair, we standardize the rewards within its rollout group to obtain group-relative advantages and optimize the policy with GRPO (Shao et al. [17]). The update increases the likelihood of intervals that align more closely with the temporal attention while applying the standard KL regularization. Repeating this process transfers the reference model’s internal temporal localization into the policy’s explicit timestamp generation.

High-Value Sample Selection

The preceding objective can train on any unlabeled video–query pair, but not every sample provides equally useful attention supervision. If the base prediction already agrees with attention, there is little discrepancy for SAPO to correct; if prediction is more accurate than attention, the internal signal may not provide useful supervision. Determining which signal is more accurate normally requires ground-truth boundaries. We therefore seek a proxy that does not require ground truth and selects a higher proportion of samples on which attention provides the stronger localization signal. Given an unlabeled pool $\mathcal{D} = \{(V_i, q_i)\}_{i=1}^{M}$, we run the base model once to obtain the predicted interval $I_{p,i}$ and reference attention distribution $a_i$ for each sample. We convert $I_{p,i}$ into a temporal distribution $p_i = g(I_{p,i})$. Under a training budget of $k$ samples, we score each sample by Jensen–Shannon divergence and select the top-$k$ examples:

As shown in Figure 4, the attention lead rate, defined as the fraction of examples for which the attention interval is more accurate than the predicted interval, increases with JS divergence on both datasets. Unlike the conditional statistic in Table 1, this rate is computed over all samples: ties are retained

in the denominator and counted as non-leads. The consistent trend indicates that prediction–attention divergence can serve as a ground-truth-free proxy for high-value samples. Because $d_i$ requires no temporal boundaries, the same top-$k$ rule can be applied directly to a new unlabeled dataset. Further comparisons with alternative selection criteria that do not use ground truth are reported in the experiments.

Use of Ground Truth

Ground-truth boundaries are used only to validate the general diagnostic findings and to evaluate final grounding performance. Building on these findings, SAPO operates on training data with temporal boundaries removed; its sample selection, reward construction, and policy optimization require no ground truth.

Experiments

Experimental Setup

Base Models. We apply the same SAPO procedure to three base models: Qwen2.5-VL-3B and Qwen2.5-VL-7B [1], which represent general-purpose VLMs at two scales, and Time-R1-7B [20], a state-of-the-art model with strong temporal grounding performance.

Training Data. SAPO uses video–query pairs from the training splits of Charades-STA [4] and ActivityNet-Captions [9] after removing all temporal boundaries. Using the ground-truth-free criterion described in the Method, we select 2,000 high-value samples from one dataset for single-source training or 1,000 from each dataset for joint training, keeping the total budget fixed.

Benchmarks and Metrics. We evaluate temporal grounding on the held-out test splits of Charades-STA and ActivityNet-Captions under the standard train/test separation. To verify that temporal adaptation does not substantially impair general video understanding, we additionally report overall accuracy on Video-MME (Fu et al. 2024). Full training configurations, implementation details, and the Video-MME breakdown are provided in the Appendix.

Baselines. Our primary baselines are the zero-shot versions of the three base models, which enable controlled, paired comparisons under the same evaluation protocol. We also report ChatVTG (Qu et al. 2024), TFVTG [26], and TAG [10] as training-free numerical references because they share the no-annotation condition. They are not strictly comparable, however, because they improve inference through external workflows rather than update the underlying model. To our knowledge, no existing TVG method provides a directly matched setting that updates a model using video–query pairs without temporal annotations. Unless otherwise specified, post-training analyses and ablations use the joint-data setting.

Main Results

Table 2 shows that SAPO consistently improves both general-purpose VLMs and the already strong Time-R1 model. The gains are therefore not limited to correcting a weak base model, but extend across model scales and initial grounding abilities. Single-source training also transfers to the other dataset, while joint training remains effective across evaluation domains. These results indicate that SAPO learns a transferable improvement to native timestamp generation rather than fitting a dataset-specific temporal prior, supporting its use when adapting a VLM to previously unseen unlabeled data. Published training-free results are included as numerical references, but are not directly comparable because they improve inference through external workflows rather than updating the model itself.

Base	Method	train split	Charades-STA [email protected]	[email protected]	[email protected]	mIoU	$\Delta$	ActivityNet-Captions [email protected]	[email protected]	[email protected]	mIoU	$\Delta$
Qwen2.5-VL-3B	Zero-shot	–	0.553	0.346	0.175	0.365	–	0.342	0.216	0.119	0.246	–
	+ SAPO	Charades	0.664	0.392	0.158	0.415	+13.7%	0.409	0.241	0.126	0.281	+14.2%
	+ SAPO	ActivityNet	0.629	0.368	0.165	0.398	+9.0%	0.489	0.285	0.133	0.327	+32.9%
	+ SAPO	Charades+ActivityNet	0.656	0.384	0.156	0.410	+12.3%	0.493	0.287	0.130	0.329	+33.7%
Qwen2.5-VL-7B	Zero-shot	–	0.641	0.422	0.213	0.436	–	0.381	0.238	0.130	0.274	–
	+ SAPO	Charades	0.671	0.443	0.218	0.459	+5.3%	0.410	0.252	0.134	0.292	+6.6%
	+ SAPO	ActivityNet	0.658	0.434	0.215	0.451	+3.4%	0.522	0.303	0.142	0.348	+27.0%
	+ SAPO	Charades+ActivityNet	0.677	0.446	0.220	0.462	+6.0%	0.501	0.293	0.137	0.334	+21.9%
Time-R1-7B	Zero-shot	–	0.815	0.578	0.270	0.525	–	0.646	0.457	0.255	0.448	–
	+ SAPO	Charades	0.802	0.596	0.313	0.534	+1.7%	0.657	0.470	0.263	0.456	+1.8%
	+ SAPO	ActivityNet	0.805	0.588	0.301	0.530	+1.0%	0.677	0.486	0.276	0.472	+5.4%
	+ SAPO	Charades+ActivityNet	0.782	0.610	0.335	0.533	+1.5%	0.675	0.484	0.270	0.468	+4.5%
Training-Free Methods	ChatVTG		0.527	0.330	0.159	0.349	–	0.407	0.225	0.094	0.272	–
	TFVTG		0.670	0.500	0.243	0.445	–	0.493	0.270	0.134	0.341	–
	TAG		0.678	0.486	0.267	0.457	–	0.519	0.289	0.151	0.366	–

Table 3 further shows that SAPO preserves general video understanding while strengthening temporal grounding. Performance on Video-MME remains close to that of the corresponding base model across training sources.

Model	Video-MME Acc.	$\Delta$
Qwen2.5-VL-3B-Instruct	58.0	–
+ SAPO trained on Charades	57.0	-1.7%
+ SAPO trained on ActivityNet	57.4	-1.0%
+ SAPO trained on Charades+ActivityNet	57.4	-1.0%
Time-R1-7B	60.1	–
+ SAPO trained on Charades	60.4	+0.5%
+ SAPO trained on ActivityNet	60.3	+0.3%
+ SAPO trained on Charades+ActivityNet	60.3	+0.3%

Because SAPO requires neither ground-truth boundaries nor supervised targets, it also supports rapid adaptation directly on unlabeled target data. We construct a joint target pool by selecting 1,000 examples from each benchmark’s test split, apply SAPO without accessing their temporal annotations, and then evaluate the adapted model. Table 4 reports this explicitly transductive protocol alongside adaptation on the corresponding joint training pool. Target-set adaptation improves the model beyond standard training-split adaptation, showing that SAPO can be used in a practical test-time scenario: given a newly encountered unlabeled target set, the model first adapts to its video–query pairs and is then evaluated on that domain.

Setting	Charades-STA mIoU	$\Delta$	ActivityNet-Captions mIoU	$\Delta$
Base (zero-shot)	0.365	–	0.246	–
SAPO from train sets	0.410	+12.3%	0.329	+33.7%
SAPO from test sets	0.421	+15.3%	0.341	+38.6%

In-Depth Analysis

We examine whether SAPO transfers internal localization evidence into native timestamps, closes the temporal generation gap, and changes attention accuracy, while also comparing it with direct attention decoding.

Does SAPO close the temporal generation gap? As shown in Table 5, SAPO substantially improves native timestamp prediction on both datasets and reduces the prediction–attention gap to within one mIoU point. The trained prediction slightly exceeds attention on Charades-STA and nearly matches it on ActivityNet-Captions, indicating that the original generation gap is effectively closed. Figure 5 provides complementary visual evidence: compared with the base model, the SAPO prediction aligns more closely with both the attention interval and the ground-truth event region.

Variant	Predicted Interval: Charades-STA [email protected]	Predicted Interval: Charades-STA [email protected]	Predicted Interval: Charades-STA [email protected]	Predicted Interval: Charades-STA mIoU	Predicted Interval: ActivityNet [email protected]	Predicted Interval: ActivityNet [email protected]	Predicted Interval: ActivityNet [email protected]	Predicted Interval: ActivityNet mIoU	Attention Interval Char. mIoU	Attention Interval Act. mIoU
Base	0.553	0.346	0.175	0.365	0.342	0.216	0.119	0.246	<u>0.395</u>	<u>0.328</u>
Attention pseudo-label + SFT	0.647	<u>0.371</u>	0.149	0.395	0.480	0.247	0.102	0.315	0.394	0.322
SAPO (Full)	0.656	0.384	0.156	0.410	0.493	<u>0.287</u>	0.130	0.329	0.401	0.335
1 w/o attention-derived rewards	0.492	0.312	0.159	0.330	0.226	0.113	0.054	0.163	–	–
2 w/o temporal distribution reward	0.597	0.362	<u>0.161</u>	0.378	0.406	0.249	<u>0.122</u>	0.277	–	–
3 w/o attention-interval IoU reward	0.616	0.352	0.145	0.388	0.471	0.295	0.114	0.312	–	–
4 replace $W_1$ with JS divergence	<u>0.673</u>	0.362	0.141	<u>0.404</u>	<u>0.532</u>	0.272	0.119	0.323	–	–
5 w/o interval-width regularization	0.687	0.335	0.129	0.403	0.534	0.276	0.116	<u>0.325</u>	–	–

Attention accuracy remains largely stable after SAPO, with only minor increases on both datasets that may arise because prediction and attention share model parameters; we therefore treat this change as a secondary effect.

How does SAPO compare with direct attention decoding? As shown in Table 5, the native SAPO prediction achieves higher aggregate accuracy than directly decoding its attention. To explain this seemingly counter-intuitive result, we partition queries using the initial model: frozen attention has higher ground-truth IoU in the attention-better group, while the remaining queries form the prediction-better group. Each point in Figure 6 compares one query’s initial and post-SAPO prediction IoUs, with the diagonal denoting no change. SAPO improves 81%/86% of attention-better queries and, despite optimizing toward attention, also improves 38%/34% of prediction-better queries on Charades-STA/ActivityNet-Captions. Thus, gains on most attention-better queries outweigh regressions while many prediction-better cases are retained or further improved. Ground-truth-free selection further concentrates training on samples where attention is the stronger reference signal, allowing native predictions to exceed it in aggregate without doing so on every query.

SAPO also retains native timestamp generation at inference, unlike direct decoding, which requires white-box attention access, layer selection, correction, and temporal-window extraction for every test example.

Ablation Studies

We ablate two main design choices: the selection of high-value samples without ground truth and the reward for policy optimization. All experiments use the Qwen2.5-VL-3B base model and the joint training pool from Charades and ActivityNet.

Training Sample Selector. We first assess how top-$k$ selection based on prediction–attention JS divergence affects downstream grounding performance. As shown in Table 6, random selection already improves the base model because the original unlabeled pool contains many attention-better examples. Under the same sample budget, JS selection yields consistently stronger downstream performance by increasing the proportion of such high-value examples. We also report a ground-truth oracle that selects only attention-better samples as an upper-bound reference; this diagnostic upper bound is not part of standard SAPO, which never accesses ground truth.

Selector	Charades-STA [email protected]	Charades-STA mIoU	$\Delta$	ActivityNet-Captions [email protected]	ActivityNet-Captions mIoU	$\Delta$
Base (no training)	0.346	0.365	–	0.216	0.246	–
SAPO with different training-sample selectors
Random	0.369	0.396	+8.5%	0.271	0.312	+26.8%
JS divergence	0.384	0.410	+12.3%	0.287	0.329	+33.7%
$GT$ oracle†	0.386	0.423	+15.9%	0.292	0.343	+39.4%

Selection Metric. Beyond the random comparison, we evaluate alternative ranking metrics that do not use ground truth under the same selection budget. Figure 7 shows that prediction–attention JS divergence produces a higher attention lead rate, indicating that it retrieves a sample set with a larger proportion of high-value examples. Together with the downstream gains in Table 6, this result supports JS divergence as an effective criterion for constructing informative RL data without ground truth.

Reward Components. Table 5 compares Full SAPO with five controlled variants that modify one aspect of the reward while keeping the training data and optimization settings fixed. We report mIoU together with recall at multiple IoU thresholds to assess boundary precision more comprehensively. The final two columns additionally track whether training changes the attention interval itself. Variant 1 removes both attention-derived rewards and performs below the base model, confirming that the gains of SAPO arise from attention-based self-supervision rather than generic format or width constraints.

SAPO uses attention supervision at two complementary levels. The temporal distribution reward aligns each rollout distribution with the full reference attention distribution through $W_1$, providing dense feedback that reflects temporal distance. The attention-interval IoU reward instead aligns the generated interval with the decoded attention interval, providing sparse boundary-level supervision. Variants 2 and 3 show that both components are effective. The larger degradation in Variant 2 further establishes the importance of dense supervision: interval IoU cannot distinguish non-overlapping rollouts, whereas distribution alignment still reflects their distances from the attended region. Variant 4 replaces $W_1$ with JS. Although the two variants achieve similar mIoU, JS reduces recall at stricter IoU thresholds. Unlike JS, $W_1$ preserves temporal geometry and provides distance-sensitive feedback for separated distributions, leading to more precise boundaries. Variant 5 exposes the role of width regularization. Without it, the model expands its predicted intervals to obtain coarse overlap, keeping mIoU similar while substantially degrading high-precision localization. The pseudo-label SFT baseline is instead constrained by the accuracy of its hard attention targets: its predicted intervals approach but remain below Full SAPO, and its attention accuracy remains essentially unchanged. In contrast, SAPO improves both predicted and attention intervals, indicating a joint adaptation mechanism beyond direct imitation of hard attention targets.

Conclusion

We identify a temporal generation gap in video VLMs: attention intervals localize queried events more accurately than predicted intervals on approximately 66% of valid examples. This finding reveals internal temporal evidence that can supervise timestamp generation without human-annotated boundaries. Based on this insight, we propose SAPO, which uses temporal attention as self-supervision for group-relative policy optimization to improve native timestamp prediction. Across multiple models, datasets, and training sources, SAPO improves temporal grounding for both general-purpose VLMs and a strong temporal grounding model, transfers across datasets, and largely preserves general video understanding.

Attention Extraction Protocol

Evaluation data. The temporal generation gap in Table 1 is measured on all 3,720 Charades-STA examples and a fixed random sample of 4,000 examples from the 17,505 ActivityNet examples. The identical subsets are used for all four VLMs. Ground-truth intervals are accessed only to compute the diagnostic IoU values; they are not used to extract attention, choose examples, or train standard SAPO. The explicitly marked GT-oracle ablation is reported only as an upper bound.

Diagnostic attention extraction. We take the last prompt token’s cross-modal attention to video tokens from a fixed mid-to-late layer, average it over attention heads, and pool the spatial visual tokens belonging to each frame. After removing the first-bin attention sink, we renormalize the resulting temporal profile. Its attention interval is the shortest contiguous window covering 50% of the cumulative attention mass, as defined in Eq. (1), and is then mapped from frame positions to seconds. We use ground-truth boundaries only in this offline mechanism analysis to measure diagnostic IoU. Table 7 lists the layer used for each model; the same layer is used on both datasets and throughout all experiments. Figure 8 reports sensitivity across middle-to-late layers and attention-mass thresholds.

Model	Dataset	Layer
Qwen2.5-VL-3B	Charades-STA	L28
Qwen2.5-VL-3B	ActivityNet	L28
Qwen2.5-VL-7B	Charades-STA	L22
Qwen2.5-VL-7B	ActivityNet	L22
Qwen3-VL-4B	Charades-STA	L24
Qwen3-VL-4B	ActivityNet	L24
InternVL3-2B	Charades-STA	L26
InternVL3-2B	ActivityNet	L26

Reference attention distribution. The reference attention distribution used during training follows the same last-token extraction, head averaging, spatial pooling, and interval-decoding procedure. We additionally divide the temporal profile by a position prior estimated from the training pool without temporal boundaries to remove dataset-level temporal bias, and renormalize the result into a distribution $a \in\Delta^{T}$. The attention interval $I_a$ is the shortest contiguous window covering 50% of the corrected attention mass. The layer and threshold remain fixed throughout sample selection and SAPO adaptation. For each model architecture, we directly specify one representative middle-to-late layer without ranking candidate layers by ground-truth grounding accuracy. The same layer is then used across datasets; the sensitivity analysis below verifies that the diagnostic conclusion is stable across nearby middle-to-late layers.

Detailed Generation-Gap Analysis

Pre-SAPO group-wise attention–prediction analysis. To complement the aggregate results in Table 1, we partition each model–dataset pair according to whether temporal attention decoding (TAD) or predicted timestamp decoding (PTD) attains higher ground-truth IoU. Table 8 reports the TAD and PTD mIoU within the attention-better, prediction-better, and all-zero tie groups.

Post-SAPO TAD–PTD group analysis. We further partition evaluation queries according to whether the trained policy’s temporal attention decoding (TAD) or predicted timestamp decoding (PTD) attains the higher IoU with the ground truth. Charades-STA uses the full test set, while ActivityNet-Captions uses the same fixed 4,000-query analysis subset as the preceding attention experiments. Table 9 reports the number and proportion of examples in each group together with the within-group mIoU of TAD and PTD.

Dataset	Group	Samples	Share	TAD mIoU	PTD mIoU
Charades-STA	Attention better (TAD > PTD)	1,682	45.2%	0.565	0.352
Charades-STA	Prediction better (PTD > TAD)	1,745	46.9%	0.310	0.535
Charades-STA	Tie (TAD = PTD)	293	7.9%	0.000	0.000
ActivityNet-Captions	Attention better (TAD > PTD)	1,689	42.2%	0.454	0.250
ActivityNet-Captions	Prediction better (PTD > TAD)	1,957	48.9%	0.293	0.457
ActivityNet-Captions	Tie (TAD = PTD)	354	8.9%	0.000	0.000

Both attention-better and prediction-better examples remain substantial after SAPO. The large within-group margins show that the near-closed aggregate generation gap does not imply uniform agreement on every query; instead, the two signals retain complementary strengths across different examples.

SAPO Reward and Sample Selection

Distributions. A predicted interval $[s,e]$ is turned into a temporal distribution $g([s,e])$ by placing a Gaussian over the $T$ uniform frame bins, centered at $(s+e)/2$ with standard deviation $0.15\max(e-s,\Delta)$, where $\Delta$ is the bin width, and then normalizing. The temporal distribution reward uses the one-dimensional Wasserstein-1 distance $W_1(p,q)=\frac{1}{T}\sum_i\lvert\operatorname{CDF}_p(i)-\operatorname{CDF}_q(i)\rvert$, normalized to $[0,1]$.

Reward weights. We use $w_{\mathrm{dist}}=1.0$ for the temporal distribution reward, $w_{\mathrm{IoU}}=0.5$ for the attention-interval IoU reward, $\lambda_{\mathrm{fmt}}=0.1$ for the format-validity reward, and $\lambda_{\mathrm{width}}=0.5$ for interval-width regularization $p_{\mathrm{width}}=$ tbl:8: Pre-SAPO group-wise attention–prediction accuracy (mIoU). Queries are grouped by comparing the ground-truth IoU of TAD and PTD.

Model	Dataset	Attention Better		Prediction Better		Tie (Both Zero)
		TAD	PTD	TAD	PTD	TAD	PTD
Qwen2.5-VL-3B	Charades-STA	0.515	0.268	0.298	0.663	0.000	0.000
InternVL3-2B	Charades-STA	0.489	0.077	0.227	0.511	0.000	0.000
Qwen2.5-VL-7B	Charades-STA	0.587	0.346	0.368	0.689	0.000	0.000
Qwen3-VL-4B	Charades-STA	0.617	0.359	0.360	0.660	0.000	0.000
Qwen2.5-VL-3B	ActivityNet-Captions	0.412	0.124	0.270	0.656	0.000	0.000
InternVL3-2B	ActivityNet-Captions	0.374	0.087	0.171	0.437	0.000	0.000
Qwen2.5-VL-7B	ActivityNet-Captions	0.418	0.138	0.242	0.642	0.000	0.000
Qwen3-VL-4B	ActivityNet-Captions	0.600	0.300	0.275	0.682	0.000	0.000

max$(0,(e-s)/\operatorname{dur} - 0.5)$. Malformed or unparseable outputs receive a fixed penalty of $-1$. Ground-truth timestamps are used only for diagnostic analyses, the oracle upper bound, and final evaluation. They never enter sample selection, reward construction, optimization, or model selection in the standard SAPO setting.

Selection. We rank the training pool without temporal boundaries by prediction–attention disagreement, measured as the Jensen–Shannon divergence between $g(I_p)$ and the corrected attention distribution $a$; larger values indicate higher training value. The main pool is constructed according to the data source. For single-source adaptation, we select 2,000 examples from Charades-STA or 2,000 examples from ActivityNet-Captions. For joint adaptation, we select 1,000 examples from each dataset, resulting in the same total budget of 2,000 examples. We use these budgets consistently whether candidates are drawn from the training split or, in the transductive setting, from the test split.

Model	Video-MME Acc.
Qwen2.5-VL-3B + SAPO	58.0; 57.4 ± 0.3
Time-R1-7B + SAPO	60.1; 60.3 ± 0.2

Training Configuration

We train all base models with GRPO under the verl framework on five GPUs in a colocated actor/reference/rollout configuration, using sglang for rollout generation. The grounding prompt asks the model to output the start and end times of the queried event in seconds. Unless noted below, the three base models share the same optimizer settings: a learning rate of $1 \times10^{-6}$, KL coefficient $\beta= 0.03$ to the frozen reference model, entropy coefficient 0.01, a maximum prompt length of 4,096, and two epochs. We monitor the SAPO training reward and select a checkpoint after the reward has converged. No ground-truth or labeled-validation metric is used for checkpoint selection.

Model	Overall	Short	Medium	Long
Qwen2.5-VL-3B-Instruct	58.0	65.7	58.6	49.9
+ SAPO on Charades	57.0	65.2	57.2	48.7
+ SAPO on ActivityNet	57.4	65.7	57.8	48.7
+ SAPO on Charades+ActivityNet	57.4	66.2	57.2	48.7
Time-R1-7B	60.1	69.7	60.3	50.2
+ SAPO on Charades	60.4	69.9	60.4	51.0
+ SAPO on ActivityNet	60.3	70.0	60.4	50.6
+ SAPO on Charades+ActivityNet	60.3	70.0	60.4	50.3

For Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct, each step draws a training batch of 60 queries with a PPO mini-batch of 30 and samples eight rollouts per query at temperature 1.1, with a maximum response length of 48 tokens (the model emits a short [s, e] timestamp). The two scales share the same configuration and differ only in the frozen base model and its representative attention layer (Table 7).

For Time-R1-7B, which produces chain-of-thought before its answer, we use a training batch of 30 queries with a PPO mini-batch of 15, eight rollouts per query at temperature 1.0, and a maximum response length of 256 tokens to accommodate the longer reasoning trace. All other hyperparameters are identical to the shared settings above.

Run-to-Run Stability

We repeat the principal SAPO settings with three random seeds. Table 10 reports mIoU as mean and standard deviation. The main-model rows use joint training-set adaptation, the transductive row uses the joint test-set pool, and the selection rows use the matched mixed-data budget from Table 6.

Setting	Charades	ActivityNet
Qwen2.5-VL-3B + SAPO	40.9 ± 0.3	32.8 ± 0.4
Qwen2.5-VL-7B + SAPO	46.1 ± 0.2	33.3 ± 0.4
Time-R1-7B + SAPO	53.2 ± 0.2	46.7 ± 0.3
SAPO from test sets	42.0 ± 0.3	33.9 ± 0.4
Random selection	39.5 ± 0.3	31.1 ± 0.4
JS selection	40.0 ± 0.3	32.8 ± 0.4
Oracle selection	42.1 ± 0.2	34.1 ± 0.3

General Video Understanding

The following tables report run-to-run stability and results by video duration for the Video-MME evaluation in the main paper.