WorksComputer Science
FOCUS: Closed-Loop Attention Feedback for Efficient Vision-Language Understanding
Abstract
Processing every video frame through Vision-Language Models (VLMs) adds significant inference cost and limits their ability to comprehend real-time video. Existing approaches either sub-sample frames uniformly, ignoring content and missing important events, or prune visual tokens inside the VLM to cut per-call cost while still invoking the model on every sampled frame. We introduce FOCUS, a training-free three-layer framework that reduces the number of invocations and steers each one towards what matters. Temporal gating measures inter-frame dissimilarity in the patch-embedding space of a smaller VLM from the same family, triggering burst captures only when the scene changes in a semantically meaningful way. Patch-level dissimilarity is then accumulated across the burst and fused with a top-down attention prior into a spatial focus map that encodes where motion and semantic relevance coincide. That map calibrates the VLM's value cache during generation, progressively amplifying vision tokens in task-relevant regions and attenuating off-target ones, without removing any token from the context. The loop closes after each response: text-to-vision cross-attention is extracted from the decoder and fed back as the prior for the next burst, letting the system converge on semantically important regions over successive observations. We evaluate FOCUS on the 500-video EgoSchema subset in streaming mode, with a single Qwen3-VL-8B backbone shared by every method. It reaches 74.8% accuracy using 25.4 VLM calls per clip, against 74.6% with 55 calls for uniform sampling and 74.2% with 55 calls for BOLT. On egocentric video captioning across four domains, it matches the best baseline's quality while issuing 57% fewer VLM invocations than uniform sampling.
Introduction
Vision-Language Models (VLMs) can now describe scenes, answer visual questions, and reason spatially across images and videos [12, 22, 14, 28] (Li et al. 2025). Applying them to video remains expensive: a 30-fps video lasting one minute produces 1,800 frames, each adding hundreds of visual tokens to the model’s context. Where video arrives as a continuous stream, as from an egocentric camera on a robot or a wearable device narrating a user’s surroundings, the full clip is never available in advance and the system must decide what to process as frames arrive. Existing VLM pipelines are not designed for this setting: they either assume access to the complete video or allocate compute uniformly across frames regardless of content.
The most common mitigation is uniform temporal sub-sampling: selecting every -th frame under the assumption that intermediate frames carry little new information [14, 28]. It is blind to content, allocating the same compute budget to static, repetitive segments as to brief, information-dense events, and risking the loss of critical transitions. A second family reduces the cost of each VLM call rather than the number of calls: token pruning methods [3, 17] drop vision tokens that contribute little to the output, while temporal compression methods [20, 1] merge redundant tokens across consecutive frames, both lower
Code: https://github.com/akul-nv/focus ing per-call cost but still invoking the model on every sampled frame. A third line uses adaptive frame selection, where an agent or retrieval model identifies which frames to process ([23, 26, 2]); these can substantially reduce VLM calls, but typically require multiple selection rounds, an auxiliary retrieval model, or access to the full video, making them unsuitable for streaming.
Our work differs from all three categories in two ways. First, FOCUS reduces the total number of VLM calls through content-adaptive gating: it measures inter-frame dissimilarity in a VLM patch-embedding space rather than in pixel space, and triggers inference only when the scene changes meaningfully. Second, it closes the loop between perception and understanding: after each VLM call the decoder’s attention is fed back as a spatial prior for the next observation, so the system progressively converges on task-relevant regions without gradient updates or learned parameters (§ 3). A secondary effect is that the VLM generates more concise outputs once scene context is established, cutting per-call inference time by up to 41% on individual bursts (§ 4.4).
We evaluate FOCUS on two complementary benchmarks. The primary one is the 500-video EgoSchema ([15]) subset, where every method runs on one Qwen3-VL-8B backbone at a matched cadence and answers are graded against ground truth. FOCUS reaches 74.8% accuracy with 25.4 VLM calls per three-minute clip, against 74.6% with 55 calls for uniform sampling, 74.2% with 55 calls for BOLT ([13]) and 71.4% with 12 calls for DyCoke ([20]). It therefore matches the strongest baselines at under half their call budget.
The second benchmark is a fine-grained egocentric captioning study across four domains, 25 videos scored per criterion by GPT-5.2. Here FOCUS attains the best overall quality (3.92/5) while issuing 57% fewer VLM invocations than uniform sampling and 16% fewer than DyCoke and BOLT. The gaps between methods are small and the sample is limited, so we read this benchmark as a per-criterion diagnostic rather than as evidence of a quality improvement. The claim we make throughout is matched-or-better quality at substantially lower VLM cost, in a streaming setting applicable to robotic manipulation, accessibility narration and live event summarization.
Contributions.
Content-adaptive temporal gating in a VLM patch-embedding space, substantially reducing VLM invocations while maintaining quality.
A spatial focus pipeline that fuses bottom-up patch dissimilarity with top-down attention priors to steer each VLM call toward task-relevant regions.
Value-cache calibration, which scales vision-token values in the decoder’s KV cache using that prior, amplifying relevant regions without discarding context.
A closed-loop feedback architecture that feeds decoder attention back into the spatial prior, enabling progressive refinement across successive observations.
A controlled streaming evaluation in which all methods share one backbone and cadence, plus component, threshold and per-domain ablations separating the compute contribution of gating from the quality contribution of calibration.
Related Work
Frame selection for video VLMs. Uniform temporal sampling remains the default strategy for feeding video to VLMs, but it ignores content entirely and risks missing transient events. Content-aware alternatives have emerged along two axes. Query-guided methods such as VideoAgent ([23]) use an LLM to iteratively retrieve frames by caption similarity, achieving strong accuracy at the cost of multiple VLM and LLM calls per video. Training-based selectors such as Frame-Voyager ([26]) and the Flexible Frame Selector ([2]) learn to pick frame subsets instead, but require task-specific supervision. BOLT ([13]) avoids training altogether by scoring frames via CLIP similarity to the query, though it must encode and score all frames upfront and assumes a predefined query is available. All of these methods focus on improving which frames the VLM sees.
Figure 1. Architecture of the three-layer FOCUS system.
Visual token reduction. Rather than reducing the number of VLM invocations, a large body of work reduces the number of vision tokens within each invocation. FastV [3], SparseVLM [27], LLaVA-PruMerge [17] and PyramidDrop [25] prune or merge vision tokens at various stages of the model, and video-specific extensions such as PruneVid [9] and HoliTom [18] additionally exploit temporal redundancy across frames. These approaches typically remove a fraction of the tokens, yielding modest speedups while preserving most of the original accuracy.
KV cache optimization and generation steering. A related family operates directly on the decoder’s KV cache. DyCoke [20] combines temporal token merging at prefill with dynamic KV cache pruning during decoding, evicting less-attended vision-token entries, and VL-Cache [21] introduces modality-aware cache budget allocation exploiting VLM-specific sparsity; both reduce memory and compute by permanently discarding cache entries. SPARC [10] instead shows that selectively scaling value representations during decoding steers VLM generation without modifying the input, using EMA-smoothed attention ratios computed dynamically. That is the mechanism we build on in § 3.2; our departure is that the scaling factors come from an explicit spatial prior carried across bursts rather than from token rankings computed within a single generation.
To our knowledge, FOCUS is the first training-free framework to unify content-adaptive frame selection, spatial attention steering and generation-time value-cache calibration in one closed-loop system that applies naturally to streaming video. Our baselines span these axes under a single backbone: uniform sampling as the non-adaptive upper bound on calls, BOLT as a training-free query-aware selector, and DyCoke as a KV-cache and token-merging method (§ 4.1).
Method
FOCUS is a three-layer system for efficient, spatially focused VLM inference on streaming video. It asks: (1) should this frame be processed at all? (2) where in the frame should the VLM focus? (3) how can the VLM’s internal understanding improve future focus? These correspond to semantic frame gating (§ 3.1), value-cache calibration with spatial priors (§ 3.2) and closed-loop attention feedback (§ 3.3). Temporal gating and spatial attention have been explored independently; the central contribution of FOCUS is closing the loop between them, extracting the VLM’s own decoder attention after each invocation and feeding it back as the spatial prior for the next, steering calibration across bursts without gradient updates or learned parameters.
Appendix § 6 collects the notation in a single table and Appendix § 7 gives the full procedure as pseudocode.
Semantic Frame Gating
Dissimilarity in embedding space. Any streaming system must decide which frames deserve expensive processing. Traditional approaches make that decision from pixel-level cues such as histogram differences, optical flow or hand-crafted features [8, 19]. The problem is that these signals are decoupled from the model that will ultimately interpret the frame. FOCUS instead measures change in the kind of representation the VLM itself works in. A gating encoder maps each arriving frame to patch embeddings over a grid of patches, and we compare them against the previously processed frame , which is updated after every step, by cosine dissimilarity . Averaging over patches gives one scalar per frame, , which trades off sensitivity to localized against global change. Because these are learned semantic features rather than pixels, a meaningful change such as a hand entering the scene produces a large even when it occupies few pixels, while a visually dominant but semantically vacuous one such as camera shake produces a small one.1
Gating state machine. The purpose of gating is to reduce VLM computation by skipping frames that carry no new information. The logic is a two-state machine over CAPTURE and WATCH. In CAPTURE the system collects a burst of consecutive frames, which gives the VLM a short temporal window to reason about an action rather than a single snapshot. Once the burst is complete the system moves to WATCH and frames are skipped. It returns to CAPTURE when dissimilarity exceeds the threshold, , or when a safety timeout of consecutively skipped frames is reached, which ensures the VLM is never starved of input during a prolonged static scene. The frame that triggers the transition becomes the first frame of the new burst.
Efficiency metric. To measure the compute savings from gating we report , the ratio of total VLM inference time to video duration:
A ratio means the VLM keeps pace with the stream faster than real time. The metric covers VLM inference alone and excludes the gating encoder, which costs ms per frame regardless of what the frame contains or which gating policy is in force; leaving out a constant isolates the one quantity a method can actually change. That cost is still a real charge against a deployment, and FOCUS pays it on every arriving frame where uniform sampling and DyCoke do not. Adding it back is a single multiplication, , worth 0.77 on our captioning benchmark against for FOCUS. In a deployed system the two stages overlap, since neither waits on the other within a frame, so the cost is the larger of the two rather than their sum. Once encoding keeps pace with the stream it becomes a floor, and VLM inference is the only term left for any method to improve.
Value-Cache Calibration with Spatial Priors
Frame gating decides when to invoke the VLM. We observe that additionally modulating where the VLM attends within each invocation improves task performance across all settings (Section 4). The spatial prior that does this steering tends to concentrate on regions of activity, such as a robot arm or the object it manipulates (Figure 2), even though nothing in the mechanism supervises which region to pick.
Spatial prior construction. The per-patch dissimilarity vector is already a byproduct of gating, and it carries spatial information the gating decision throws away: it encodes where in the frame the change occurred. We reshape it onto the encoder’s grid and take the element-wise maximum over the frames of the burst, , where rescales to . Taking the maximum rather than the mean retains brief but significant motions at full strength even when they occupy only a few frames of the burst.
This map captures physical change, but not all change is task-relevant. A flickering monitor produces high dissimilarity while contributing nothing to the task, while a stationary object the VLM has been attending to, such as a tool about to be grasped, may show almost no dissimilarity yet be exactly what matters. Neither signal suffices alone, so we fuse this bottom-up map with a top-down attention prior from the previous burst (§ 3.3):
where (default 0.5) controls how sharply attention modulates dissimilarity and is a floor ensuring low-attention regions are down-weighted but never zeroed. The key property is that a region must score highly on both signals to receive a strong prior.
Value-cache scaling. SPARC [10] showed that selectively scaling internal representations steers VLM behaviour without modifying the input. We build on this but depart from it: where SPARC selects tokens dynamically via EMA-smoothed attention ratios, we apply the fixed spatial prior of Eq. (2), flattened and mapped directly onto the vision-token positions (resampled by nearest-neighbour interpolation when dynamic-resolution tiling makes the lengths differ), so calibration operates in spatial coordinates rather than token-ranking space. Each token receives a prior and a per-step scale
applied to the value cache of every calibrated layer at every autoregressive decoding step:
Here is the focus strength (default 1.05), the suppression strength (default 0.04), and the focus range (§ 4.1); is the cached value of position at decoder layer , and unless stated otherwise we calibrate the full stack of layers. Two properties of this update are worth stating plainly. The floor of 0.5 bounds the per-step factor only, not the compounded product, and at $\beta = 0.04 it never binds, since the smallest per-step factor is 0.96. And because the update multiplies a cache that persists across the whole response, the factors compound: after decode steps the effective multiplier is , so a per-step nudge of 1.05 becomes a scaling of by our 100-token cap while a fully off-prior token falls to . Appendix § 9 gives the remaining implementation details.
This works because value vectors determine how much each token contributes to the attention-weighted sum during generation. Scaling them amplifies the information flowing from the prior’s regions into every subsequently generated token. Keys and queries are never touched, so the attention weights are exactly those of the uncalibrated model and the next sub-layer’s normalisation absorbs any common scale; what survives is the relative re-weighting among vision tokens, and no token leaves the context. Prompt-level alternatives such as region-of-interest cropping, textual spatial hints and soft foveation all degraded performance in preliminary experiments, apparently because they discard context the model needs for coherent reasoning.
Closed-Loop Attention Feedback
Most frame selection systems are purely feed-forward: the selector never receives feedback from the model that consumes its output. FOCUS closes this loop by extracting the VLM’s internal attention after each burst and feeding it back as the top-down prior for the next.
Attention extraction. Once the VLM has answered for burst , we want a spatial map of where it actually looked while generating that answer. A single additional forward pass over the prompt and the generated response, with attention outputs enabled, yields the attention tensors for every decoder layer and head over the length- sequence. We aggregate the last layers (default 4), since later decoder layers carry more task-specific attention [4], by max-pooling across heads and averaging across layers. Max-pooling preserves the most spatially selective head rather than diluting it against heads that attend almost entirely to text:
From we keep only the text-to-vision block: how much the generated text attended to each vision token. Text positions before the vision block are dropped, because the causal mask gives them exactly zero attention to it. Collapsing that block to one score per vision token is not a plain average, since generated tokens differ widely in how much they look at the image at all; we weight each text token by the total attention it directs at the image, so tokens that are largely reasoning over text contribute little. The resulting scores are folded back onto the patch grid, averaged over the frames of the burst and normalised to to give . The extra pass has no autoregressive loop and is small relative to the generation it follows. Appendix § 8 states all six steps precisely.
Feedback loop lifecycle. After each burst the dissimilarity maps are cleared but is retained, so the VLM’s understanding from burst shapes the prior of burst . On the first burst no prior exists and the VLM processes the full scene without calibration, an orienting phase; from burst 2 onward the stored prior is fused with the new dissimilarity map (Eq. (2)) and the resulting calibrates the value cache (Eq. (4)), each burst producing a new attention map that replaces the previous one (Figure 2). Alongside it, a rolling window of the last responses (default ) carries forward as textual context, discouraging repeated descriptions across bursts; this window is enabled identically for every method we compare and every FOCUS ablation, so it is a controlled constant rather than a source of the differences we report.

Figure 2. Spatial prior evolution across bursts (robotic teleoperation). Columns are bursts; rows are a representative frame, the dissimilarity map, the decoder attention after generation, and the combined prior (absent at Burst 0, the orienting phase).
Experiments
Setup
Models and configuration. FOCUS uses the Qwen2-VL-2B-Instruct [22] vision encoder for gating and Qwen3-VL-8B-Instruct [16] as the inference
VLM; the same Qwen3-VL-8B backbone is used for every method in every table below. Experiments run in bfloat16 on NVIDIA A100 80 GB and RTX PRO 6000 GPUs, with all values measured on one RTX PRO 6000. Unless stated otherwise we use a single configuration for every FOCUS experiment: , , , , , greedy decoding and a 100-token cap. Every default was fixed once on a development set disjoint from all evaluation videos and never re-selected afterwards; Appendix § 9 gives that protocol and tabulates all twelve constants with their roles and sensitivity.
Benchmarks and baselines. Our primary benchmark is EgoSchema (Mangalam et al., [15]), whose publicly released 500-clip subset pairs three-minute egocentric clips with five-way multiple-choice questions and ground-truth answers, and is the split prior systems report on. We complement it with egocentric captioning over 25 videos from Ego4D (Grauman et al., [6]), EPIC-Kitchens (Damen et al., [5]), Ego-Exo4D (Grauman et al., [7]) and GR-1 teleoperation recordings (Wu et al., [24]) plus one in-house recording, treated as a per-criterion diagnostic: at with an LLM judge, gaps are within noise. Our baselines span the design axes of § 2 under one backbone: uniform sampling, the non-adaptive upper bound, firing a 32-frame burst on a fixed schedule; DyCoke (Tao et al., [20]); BOLT (Liu et al., [13]); and FOCUS (gating only), our gating layer with per-burst captioning but no spatial calibration or feedback, isolating gating from everything downstream. For captioning, GPT-5.2 rates each caption from 1 to 5 on Actor, Activity, Object and No Hallucination, with Overall their mean (Zheng et al., [29]); see Appendix § 13. Appendices E–G give the component, threshold, per-domain and burst-size ablations.
Main Results: EgoSchema under a Matched Backbone
Every method runs on the 500-video subset at a matched 3.3 s cadence with matched prompting, so only the selection and steering policy varies. The setting is streaming: frames arrive in order, with no future frames and no pre-specified query. FOCUS passes a cumulative narrative built from its burst captions, with the five answer choices, to the VLM in one final QA call; the baselines capture a 32-frame burst per interval and aggregate their outputs for that call.
FOCUS with feedback reaches 74.8% using 25.4 VLM calls per clip, against 74.6% for uniform sampling and 74.2% for BOLT, both of which spend 55. The accuracy differences here are not the result. One video is worth 0.2 points on a 500-question benchmark, so those two gaps come to one video and three, and five runs cannot resolve differences that small; we claim no accuracy win. What the table separates cleanly is the call budget, where the gap is large and deterministic. DyCoke is the informative exception: it issues the fewest calls but gives up 3.4 points, the price of compressing each call rather than deciding whether to make it. The 1.2 points between our own two rows sit inside the same band, so the case for feedback rests on the object-grounding effect isolated in Appendix § 10 rather than on this number.
That the two FOCUS rows issue identical calls is structural, not a coincidence. Gating (Layer 1) sits upstream of calibration and feedback (Layers 2–3) and is independent of them, so the burst count is fixed by and alone, measured at 25.4 per clip (sd 6.5 across clips).2 Feedback does not change how many calls are made, only what each one costs: it adds one attention-extraction pass per burst, moving from 0.57 to 0.59 on captioning (Table 2). One caveat: at 6 fps most arriving frames fall inside a burst, so gating here buys fewer, longer-gap bursts rather than frame-level skipping, leaving the call-count saving untouched since an invocation is billed per burst (Appendix § 12).
| Method | Actor | Activity | Object | No Halluc. | Overall | VLM Calls | |
| Uniform Sampling | 4.96 | 4.12 | 4.00 | 2.16 | 3.81 | 4.14 | 516 |
| DyCoke | 4.80 | 3.88 | 4.04 | 2.52 | 3.81 | 3.07 | 263 |
| BOLT | 4.76 | 4.12 | 4.08 | 2.68 | 3.91 | 0.87 | 263 |
| FOCUS (gating only) | 5.00 | 4.00 | 4.12 | 2.44 | 3.89 | 0.57 | 222 |
| FOCUS (w/ Feedback) | 5.00 | 3.96 | 4.12 | 2.60 | 3.92 | 0.59 | 222 |
Table 2. Captioning quality (GPT-5.2, 1–5) and efficiency on 25 egocentric videos. is the VLM compute ratio (Eq. (1)); lower is better. Same backbone throughout, but not decoding-matched: DyCoke and BOLT use sampled decoding and a 200-frame interval before their own selection, against greedy decoding and 100 for Uniform and FOCUS. A further reason to read the small quality gaps cautiously.
Prior systems on this subset use much larger proprietary backbones and non-streaming protocols, so we leave them out of Table 1: those differences would confound exactly the comparison it isolates. Appendices § 14 and § 15 add an offline sanity check and a per-answer breakdown.
Table 1. EgoSchema subset accuracy (500 videos, 5-way multiple choice) in the streaming setting. All methods share the Qwen3-VL-8B backbone at a matched 3.3 s cadence; accuracy is the mean over five runs, “VLM Calls” the mean invocations per three-minute clip. The two FOCUS rows issue identical calls by construction: gating alone determines how many bursts fire.
Egocentric Video Captioning
Table 2 is a statement about cost rather than quality. FOCUS with feedback issues 57% fewer VLM calls than uniform sampling (222 against 516) and 16% fewer than BOLT and DyCoke, and it runs at against BOLT’s 0.87 while satisfying a streaming constraint BOLT does not: BOLT needs every frame upfront and a predefined query. On quality the methods are tied. The 0.01 margin over BOLT gives under a paired Wilcoxon test on the 25 matched per-video scores, and the 0.11 over DyCoke gives , so we claim no quality gain (Appendix § 13). BOLT’s also excludes the CLIP encoding it runs over every frame upfront, so on that axis the comparison is conservative in its favour.
The per-criterion columns are what this benchmark is for. Uniform sampling ties for the best Activity score (4.12, with BOLT) yet ranks last on No Hallucination (2.16); seeing every frame does not reduce hallucination and appears to increase it, plausibly by flooding the model with redundant visual context. BOLT posts the best No Hallucination score (2.68), likely for the mirror-image reason: its CLIP-based selection discards the ambiguous frames that would prompt the model to invent detail. Feedback contributes +0.03 overall over gating alone, driven by +0.16 on No Hallucination, consistent with the prior suppressing irrelevant content in later bursts, though at this is directional rather than established.
Qualitative Analysis
What the closed loop does is easiest to see burst by burst. Figure 2 tracks the prior across a teleoperation clip: absent at the orienting burst, then contracting onto the arm and the object it manipulates. Two EPIC-Kitchens clips show the effect on the text.
Attention feedback corrects spatial hallucinations. In a 15-second clip of a person stirring soup and covering a pot (Figure 3), both variants select the same two bursts and give identical first-burst descriptions, since the orienting phase has no prior, but their second-burst descriptions diverge:

Figure 3. Frames from the soup-stirring clip (EPIC-Kitchens, 15s).
Without feedback: “placed the metal lid onto the right-hand burner of the stove, covering the empty pot.”
With feedback: “placed the metal lid onto the pot on the right side of the stove, covering it. The action has shifted from stirring or preparing to actively covering the pot, likely to let its contents simmer or cook.”
The prior from Burst 1, where the model attended heavily to the pot and its contents, persists into Burst 2 and anchors the lid to the correct object. The correction comes from the model’s own attention, not any outside signal, and the description now infers intent rather than only reporting position.
Feedback produces more focused descriptions. In a 25-second clip of stirring and transferring red sauce (Figure 4), the third-burst descriptions are:

Figure 4. Frames from the sauce-stirring clip (EPIC-Kitchens, 25s).
Without feedback (5.5s inference): “The person has now lifted the spoon . . . poised to transfer the sauce. The spoon is suspended above the bowl, indicating the action of pouring is about to begin. ... The sauce in the pot appears to be thick and reddish-brown, and the bowl is empty and ready to receive the contents.”
With feedback (3.2s inference): “The person has now lifted the spoon . . . poised to transfer the sauce.”
By the third burst the prior has converged on the spoon–bowl–pot region, so the model stops re-describing layout it has already covered and the burst is 41% faster. Uniform sampling here spends 15 bursts and 233s on 25 seconds of video (), 13 opening identically, against 3 bursts and 11s () for FOCUS.
Conclusion
We presented FOCUS, a training-free framework that reduces VLM video inference cost by gating frames in embedding space, steering each call with a spatial prior, and refining that prior through closed-loop feedback. Under a matched backbone it holds the accuracy of the strongest streaming baselines on EgoSchema at under half their VLM calls, and the quality of the best captioning baseline at two-thirds of its compute ratio. The gain is not better answers, but the same answers for substantially less computation. Appendix K gives limitations and next steps.
Notation
Table 3 collects every symbol used in § 3.
| Symbol | Meaning |
| index of a frame in the arriving stream | |
| ref | the reference frame for dissimilarity: the previously processed frame, updated every step |
| vision encoder used for gating (Qwen2-VL-2B ViT) | |
| index of a spatial patch within a frame, | |
| number of patches after merging, and the patch-grid height and width, | |
| patch-embedding width | |
| patch-embedding grid of frame and the embedding of its patch | |
| per-patch cosine dissimilarity and its mean over patches | |
| gating threshold on | |
| burst size (frames captured per trigger) | |
| safety timeout: forced capture after this many consecutively skipped frames | |
| burst index | |
| per-patch dissimilarity reshaped to the grid | |
| bottom-up dissimilarity map, max-pooled over the frames of a burst | |
| top-down attention map fed back from the previous burst | |
| fusion exponent and floor in Eq. (2) | |
| fused spatial focus prior and its value at vision token | |
| decoder input sequence length | |
| index of a position in the decoder sequence | |
| set of vision-token positions (input id equals the image-token id) | |
| set of text positions after the vision block, | |
| decoder layer index and number of layers ( for Qwen3-VL-8B) | |
| attention-head index | |
| value-cache entry of position at layer | |
| per-step value-cache scale factor for vision token (Eq. (3)) | |
| focus (amplification) strength and suppression strength | |
| focus range: priors inside are amplified, priors outside are suppressed | |
| first and last calibrated decoder layer | |
| number of tokens generated in a burst response | |
| raw attention tensor of layer , head , and its head/layer aggregate | |
| number of final decoder layers aggregated for attention extraction | |
| rolling window of previous responses carried forward as textual context | |
| VLM compute ratio (Eq. (1)) | |
| VLM inference time for burst ; per-frame gating-encoder time |
Table 3. Symbols used in Section 3.
Full Algorithm
Algorithm 1 states the complete streaming procedure, covering frame intake, the gating state machine, burst finalisation, spatial-prior fusion, value-cache calibration, attention extraction, and prior persistence.
Attention Extraction in Detail
EXTRACTATTENTION runs once per burst, immediately after the burst response has been generated, and returns the map that becomes the top-down prior for the next burst. The steps are:
One extra forward pass. Re-run the VLM over the concatenation of the burst prompt and the generated response with attention outputs enabled. This is a single teacher-forced pass with no autoregressive loop, so its cost is a small fraction of the generation it follows.
Head and layer aggregation. Stack the last layers, take the elementwise maximum over heads, then the mean over layers (Eq. (5)). Max-pooling over heads
Algorithm 1 FOCUS: streaming inference with closed-loop attention feedback
Input: frame stream $\{x^{(t)}\}$; encoder $g$; VLM $f$; parameters $\tau, B, N_{\max}, \alpha_f, \beta, r_{\mathrm{lo}}, r_{\mathrm{hi}}, \gamma, \varepsilon, N_a, W$
1: state $\leftarrow$ CAPTURE; $Q \leftarrow \varnothing$; $M_{\mathrm{attn}} \leftarrow \varnothing$; $\mathcal{H} \leftarrow \varnothing$; $k \leftarrow 0$
2: **for** each arriving frame $x^{(t)}$ **do**
3: $\mathbf{E}^{(t)} \leftarrow g(x^{(t)})$ ▷ gating encoder runs on every frame
4: $d_i^{(t)} \leftarrow 1 - \cos(e_i^{(t)}, e_i^{(\mathrm{ref})})$; $D^{(t)} \leftarrow \frac{1}{P}\sum_i d_i^{(t)}$; ref $\leftarrow t$
5: **if** state = WATCH **then**
6: $k \leftarrow k + 1$
7: **if** $D^{(t)} > \tau$ **or** $k \ge N_{\max}$ **then**
8: state $\leftarrow$ CAPTURE; $k \leftarrow 0$
9: **else**
10: **continue**
11: **end if**
12: **end if**
13: $Q \leftarrow Q \cup \{(x^{(t)}, d^{(t)})\}$
14: **if** $|Q| < B$ **then**
15: **continue**
16: — burst complete: run one VLM call —
17: $M_{\mathrm{dissim}} \leftarrow \operatorname{norm}(\max_{t\in Q} S^{(t)})$ ▷ {xref}`Section 3.2`
18: **if** $M_{\mathrm{attn}} \ne \varnothing$ **then**
19: $C \leftarrow \operatorname{norm}(M_{\mathrm{dissim}} \odot (\varepsilon + M_{\mathrm{attn}}^\gamma))$; install value-cache hooks with $s_j$ from {xref}`Eq. 3`
20: **else**
21: **no calibration** ▷ first burst: orienting phase
22: **end if**
23: $y \leftarrow f(Q, \text{prompt} \mathbin{\|} \text{last } W \text{ responses of } \mathcal{H})$; remove hooks; $\mathcal{H} \leftarrow \mathcal{H} \cup \{y\}$
24: $M_{\mathrm{attn}} \leftarrow \operatorname{EXTRACTATTENTION}(f,Q,y)$ ▷ {xref}`Eqs. 5–6`; {xref}`Appendix C`
25: $Q \leftarrow \varnothing$; state $\leftarrow$ WATCH; $k \leftarrow 0$
26: **end for**
27: **return** $\mathcal{H}$ ▷ burst captions; for QA, one final call over $\mathcal{H}$ and the answer choices
keeps the most spatially selective head instead of averaging it away with heads that attend almost entirely to text.
Span selection. Partition positions into vision positions (input id equal to the image-token id) and text positions, and let be the text positions lying strictly after the vision block, that is the response tokens and any trailing prompt tokens. Positions before the vision block are dropped because the causal mask gives them exactly zero attention to vision tokens, so retaining them would add rows of zeros and dilute the map. If no post-vision text position exists, fall back to all text positions.
Text-token reweighting. Write and collapse it over text tokens by row-sum weighting rather than a plain mean, so a token that directs more total attention at the image contributes proportionally more:
Row sums are clamped below at before normalisation.
Grid folding. Split the scores into their per-frame blocks using the processor’s
image_grid_thwmetadata, reshape each block into that frame’s own grid, and, when dynamic resolution makes those grids differ, bilinearly resize each to the common . Average over the frames of the burst.
Normalisation. Min–max rescale the result to ; if the map is constant, return zeros, which makes the next burst fall back to the dissimilarity signal alone.
Hyperparameters, Development Set, and Sensitivity
Development set and tuning protocol. Every default in Table 4 was fixed on a development set of egocentric clips drawn from the same source datasets as the captioning benchmark (Ego4D, EPIC-Kitchens, Ego-Exo4D and teleoperation recordings) but disjoint from it: no video used for tuning appears in the 25-clip captioning benchmark or in the 500-clip EgoSchema subset. Selection was done once, by inspecting caption quality and on the development clips, before any evaluation run, and no value was revised after seeing evaluation results. Two independent pieces of evidence support that the resulting operating point is stable rather than fit to a particular set. First, the threshold sweep in Table 6 is broadly flat in quality across a wide band of , so the operating point is not a sharp peak of the kind that leaks tuning signal. Second, the component ablation in Table 5, run on a cohort sampled from EgoSchema and never used for tuning, selects the same setting that the development set did.
| Symbol | Default | Swept | Role and sensitivity |
| 0.4 | yes | Gating threshold; trades compute against quality, flat over a wide band | |
| 32 | yes | Burst size; quality within 0.07 across 8–64, minimised at 32 | |
| 300 | no | Forced-capture timeout; liveness guard, inactive on all evaluated clips | |
| 1.05 | partly | Amplification strength; ablated on/off in Table 5 | |
| 0.04 | partly | Suppression strength; ablated on/off in Table 5 | |
| [0.75, 1.0] | no | Focus range; what fraction of tokens is amplified vs. suppressed | |
| floor | 0.5 | no | Lower bound on the per-step suppression factor; never binds at |
| 0.5 | no | Fusion exponent; how sharply attention modulates dissimilarity | |
| 0.01 | no | Fusion floor; keeps low-attention regions non-zero | |
| 0, | no | Calibrated layer range; full stack by default | |
| 4 | no | Final decoder layers aggregated for attention extraction | |
| 5 | no | Rolling response window; identical for all methods and ablations |
Table 4.
| Configuration | MCQ | Judge | Obj. | ||
| (a) Gating only | – | – | 76.0% | 3.81 | 3.76 |
| (b) + amplification | 1.05 | – | 76.0% | 3.84 | 3.96 |
| (c) + suppression | 1.05 | 0.04 | 78.0% | 3.81 | 3.92 |
Table 5. Component ablation, 50 EgoSchema videos. A dash means inactive. Row (a) is FOCUS (gating only) of Table 1, row (c) is FOCUS (w/ Feedback) as deployed, and (b) is an intermediate with amplification but no suppression.
| MCQ | Judge | VLM s/vid | ||
| 0.20 | 82.0% | 3.88 | 167.5 | 0.93 |
| 0.30 | 74.0% | 3.81 | 157.6 | 0.88 |
| 0.40⋆ | 78.0% | 3.81 | 131.7 | 0.73 |
| 0.45 | 76.0% | 3.81 | 96.7 | 0.54 |
| 0.50 | 70.0% | 3.81 | 54.7 | 0.30 |
Table 6. Gating threshold sweep, same 50 videos. VLM s/vid is mean VLM seconds per clip; is against the 180 s clip. ⋆ marks the default.
Hyperparameter inventory. The method carries a substantial number of hand-set constants, and we would rather state that cost than bury it. Table 4 lists every one of them, what it controls, and what we know about its sensitivity. Two, and , are swept over a range; and are toggled on and off in the component ablation; the remainder were set once and never tuned per domain.
Table 4: Complete hyperparameter inventory. “Swept” indicates whether the paper reports a sweep for that parameter.
Value-cache implementation notes. Three details of Eq. (4) matter for reproducing it. First, the update is applied by a forward pre-hook that fires only when the incoming hidden state has sequence length 1, that is on decode steps and not during prefill, so the values written by prefill are left alone and only their re-use during generation is biased. Second, the hook multiplies the persistent cache in place, which is why the per-step factors compound to over an -token response; the magnitudes this reaches are given in § 3.2. Third, is computed once per burst rather than once per token, so calibration costs one elementwise multiplication per calibrated layer per decode step, which is negligible beside generation itself.
On reducing the hyperparameter count. Three of these are, in principle, removable rather than merely tunable. is the most consequential and the most clearly replaceable: is a fixed threshold on a distribution ( over a video) that the system already observes online, so calibrating it to a running quantile of that distribution would make gating adaptive at no extra encoder cost. plays the same role in the spatial domain and could likewise be set from a quantile of rather than fixed. is a liveness guard whose value never binds in our experiments. The remaining constants are either soft (, the floor) or structural (, , the layer range). We flag adaptive thresholding as the highest-value follow-up in Appendix § 16.
Component and Threshold Ablations on EgoSchema
We ablate two axes: which components contribute what, and sensitivity to . Both use 50 videos randomly sampled from the 500-video subset, used only for ablations, and both sweeps are single-run and should be read directionally.
Gating versus calibration. Gating accounts for the entire reduction in calls; calibration supplies the quality steering at the cost of one extra forward pass per burst. Calibration is also not separable from feedback when streaming: the cache is steered by , which the closed-loop signal produces, so there is no “calibration without feedback” to ablate (Appendix § 14 has the one offline case where it exists). What varies instead is calibration strength, as Table 5 does. The clearest effect is on object grounding, rising from 3.76 to 3.96 once amplification is enabled (a → b) and then holding, consistent with the prior concentrating on the manipulated object; MCQ is flat from (a) to (b) and rises to 78.0% once suppression is added, while judge overall moves only 3.81 → 3.84 → 3.81. At these deltas are small relative to what the cohort can resolve, so the table supports that calibration improves object grounding without measurable degradation elsewhere, not that each sub-component gives an independent gain. The rolling text window is not ablated: it is enabled identically for every baseline and variant.
Threshold sensitivity. trades compute against marginal quality rather than acting as a quality-critical knob: judge quality is flat at 3.81–3.88 across Table 6 while falls threefold. We choose as the knee: it cuts VLM seconds by 21% against the accuracy maximum at (82.0%) while staying within 0.07 judge and 4 MCQ points of it, and is explicitly not that maximum. Larger values buy compute at a visible accuracy cost. The per-domain breakdown below finds no domain with a systematically different knee, though one whose dissimilarity distribution differs materially from ours would need a different .
Per-Domain Threshold Sensitivity
Table 7 breaks the threshold sweep of Table 6 down by activity domain. Domain labels are derived from the EgoSchema ground-truth caption summaries, since the subset carries no official scenario tags. Per-domain is small, so these are trends rather than precise per-cell estimates: at a single video is worth 7–17 percentage points.
| Domain | Best | @0.40 | ||||
| Cooking & food prep | 14 | 79% | 86% | 79% | 86% | 86% |
| Crafts, art & textiles | 11 | 64% | 64% | 55% | 73% | 64% |
| Construction & workshop | 9 | 89% | 89% | 89% | 100% | 89% |
| Housework & cleaning | 6 | 83% | 100% | 83% | 100% | 100% |
| Leisure, games & social | 10 | 60% | 60% | 50% | 70% | 60% |
| All (pooled) | 50 | 74% | 78% | 70% | 82% | 78% |
Table 7. Accuracy by domain across the gating threshold sweep (50 EgoSchema videos). “Best” is the highest accuracy that domain reaches at any in the full sweep, and “@0.40” repeats the accuracy at our default.
No domain exhibits a systematically different knee. lands at or near each domain’s best accuracy, and it is the best observed setting outright for cooking and for housework. Where it is not (crafts, leisure), the gap to that domain’s best is one or two videos wide at , which is within sampling noise for these cell sizes. We would still expect a domain whose inter-frame dissimilarity distribution differs materially from egocentric activity video, for instance static surveillance footage or fast-cut edited video, to need a different ; that is the case adaptive thresholding is meant to cover.
Ablations on the Captioning Benchmark
The sweeps in this section use the 25-video captioning benchmark with GPT-5.2 judging the VLM responses, and complement the EgoSchema-based sweeps in Appendix E. Because , they should be read as directional.
Capture ratio is rate-dependent, not threshold-dependent. At the same , gating retains 27% of processed frames on this benchmark but 74% on EgoSchema (§ 4.2), a gap worth stating plainly because it changes what “skipping redundant frames” means on each. The cause is the input rate. The captioning videos are ingested at fps on average, so consecutive processed frames are ms apart and most pairs are near duplicates whose embedding dissimilarity falls below . EgoSchema is decoded at 6 fps, so consecutive processed frames are 167 ms apart, much more of the scene has changed between them, and exceeds far more often; this is also why the liveness timeout never binds there. The consequence for our claims is limited but real: on EgoSchema the saving comes from issuing fewer, longer-gap bursts (25.4 per clip against 55 for uniform sampling at the same cadence) rather than from discarding frames, since a VLM invocation is billed per burst and not per frame. The frame-level reading of gating therefore holds at high ingest rates and weakens as the input is pre-decimated, and a deployment sampling at 6 fps should expect capture ratios closer to the EgoSchema figure.
Threshold. Quality peaks at (3.93 overall) and remains within 0.01 at (3.92), before declining at higher thresholds. The sharp drop in between and (2.11 0.59) reflects the distribution of inter-frame dissimilarity values in our videos: many frame-to-frame transitions produce dissimilarity scores in the 0.3 to 0.4 range, so a small threshold increase crosses a critical point where substantially fewer frames trigger bursts (capture drops from 65% to 27%). Setting too low degrades quality because the VLM is overwhelmed with redundant context; setting it too high (0.5+) skips important transitions. We adopt for its near-peak quality and the lowest in the sweep, consistent with the EgoSchema-based sweep in Table 6.
Burst size. Smaller bursts () return to the WATCH state sooner, triggering more VLM calls (17.6 and 12.3 on average). Each call costs roughly the same inference time (s), so total VLM compute increases despite processing fewer frames per call, yielding higher . The more frequent feedback iterations explain why achieves a higher Activity score (4.00 vs. 3.96 at ): the spatial prior converges faster with more updates. Larger bursts () give the VLM richer temporal context per call, yielding the highest overall quality (3.94) and the best No Hallucination score (2.72), at a moderate of 1.18. The default balances quality (3.92) with the lowest (0.59) in the sweep.

Figure 5. Quality and efficiency as a function of (a) gating threshold () and (b) burst size (). Circled markers denote the default configuration. The dotted grey line marks (real-time).
| Actor | Activity | Object | No Halluc. | Overall | Avg Calls | Capture | ||
| 0.2 | 4.76 | 3.68 | 4.04 | 2.56 | 3.76 | 2.31 | 20.4 | 95% |
| 0.3 | 4.80 | 3.92 | 4.40 | 2.60 | 3.93 | 2.11 | 17.0 | 65% |
| 0.4† | 5.00 | 3.96 | 4.12 | 2.60 | 3.92 | 0.59 | 8.9 | 27% |
| 0.5 | 4.76 | 3.80 | 4.04 | 2.52 | 3.78 | 0.63 | 5.2 | 12% |
| 0.6 | 4.84 | 3.88 | 4.08 | 2.56 | 3.84 | 0.81 | 5.2 | 12% |
Table 8. Gating threshold ablation (, spatial + feedback enabled) on the 25-video captioning benchmark. Capture ratio is the fraction of input frames included in bursts. † denotes the default configuration used in Table 2.
| Actor | Activity | Object | No Halluc. | Overall | Avg Calls | Capture | ||
| 8 | 4.96 | 3.88 | 4.12 | 2.52 | 3.87 | 2.26 | 17.6 | 15% |
| 16 | 4.96 | 4.00 | 4.04 | 2.64 | 3.91 | 1.64 | 12.3 | 20% |
| 32† | 5.00 | 3.96 | 4.12 | 2.60 | 3.92 | 0.59 | 8.9 | 27% |
| 64 | 5.00 | 3.92 | 4.12 | 2.72 | 3.94 | 1.18 | 7.2 | 36% |
Table 9. Burst size ablation (, spatial + feedback enabled) on the 25-video captioning benchmark. † denotes the default configuration used in Table 2.
LLM-as-Judge Protocol
GPT-5.2 scores each caption from 1 to 5 on four criteria: Actor Correct (is the agent correctly identified?), Activity Correct (is the main activity correct?), Object Correct (are the key objects correct?) and No Hallucination (is the caption factually grounded?), with Overall their arithmetic mean. To avoid penalizing stylistic variation rather than substance, the judge is given explicit equivalences within actor categories (“robotic arm” = “robot hand”) and among activity synonyms (“washing” = “rinsing”). The same prompt, criteria and equivalences are used for every method, so judge bias applies uniformly across the comparison.
We nonetheless treat judge-scored results as a per-criterion diagnostic only: our headline quantitative claims come from EgoSchema, which is scored against ground-truth answers.
Significance of the captioning margins. Because the quality differences in Table 2 are small on 25 videos, we test them rather than reading them as ordered. Each comparison is a paired two-sided Wilcoxon signed-rank test over the 25 matched per-video Overall scores; is the mean paired difference divided by the standard deviation of those differences. Against BOLT, FOCUS (w/ Feedback) gives with ; against DyCoke, with ; against uniform sampling, with ; and against its own gating-only variant, with . No planned comparison reaches , so correcting for multiplicity does not change any conclusion, and we claim no quality gain over the baselines. Intervals below are percentile bootstrap over 100,000 per-video resamples at seed 2026.
| Method | Overall | 95% CI |
| Uniform Sampling | 3.81 | [3.66, 3.97] |
| DyCoke | 3.81 | [3.60, 4.00] |
| BOLT | 3.91 | [3.74, 4.08] |
| FOCUS (gating only) | 3.89 | [3.69, 4.07] |
| FOCUS (w/ Feedback) | 3.92 | [3.73, 4.10] |
Table 11.
The intervals overlap heavily, which is the honest summary of a 25-video study: the efficiency separation is what this benchmark resolves, not the quality ordering.
Offline (Non-Streaming) Sanity Check
Our claims concern the streaming setting, in which frames arrive in order and no future frame or query is available. BOLT and DyCoke were originally proposed for the conventional offline setting, in which the entire clip is available upfront and all selected frames are passed to the VLM in a single call. To check that the gating-plus-calibration signal is not an artefact of the streaming protocol, we also ran both baselines in their native offline mode and built matching offline variants of our method: FOCUS (offline), which gates the clip and then passes the median frame of each retained burst to one QA call without calibration, and FOCUS with calibration (offline), which adds a dissimilarity-only spatial prior driving value-cache calibration. Note that in a single-pass offline setting there is no closed-loop feedback, because there is no subsequent burst to feed attention back into; the prior is the bottom-up dissimilarity map alone. This is the one configuration in which calibration exists without feedback, and it exists only offline.
In this regime our offline variants reach 74.4% on the 500-video subset against 74.0% for BOLT (offline), so gating plus calibration transfers out of streaming. That 74.4% is a single pooled figure: we did not retain the two offline variants’ scores separately, and the DyCoke (offline) run is not tabulated here. This appendix therefore supports only the coarse conclusion that the signal survives the change of protocol, and is not a like-for-like offline comparison. We report it only as a robustness check. It is not a streaming result and is not part of the claim made in § 4.2.
EgoSchema Per-Answer Breakdown
Table 10 gives the per-answer breakdown for a single representative run of FOCUS with feedback on the 500-video subset, the run that scored 73.2% (366/500). We report a single run here rather than the 5-run mean of Table 1 because the per-answer counts are only meaningful within one run.
| A | B | C | D | E | Overall | |
| GT count | 101 | 108 | 91 | 83 | 117 | 500 |
| Predicted | 99 | 108 | 94 | 93 | 106 | 500 |
| Correct | 72 | 79 | 70 | 67 | 78 | 366 |
| Accuracy | 71.3% | 73.1% | 76.9% | 80.7% | 66.7% | 73.2% |
Table 10. EgoSchema subset per-answer breakdown (500 videos, 5-way multiple choice), single run of FOCUS with feedback in streaming mode, , , 6 fps.
The prediction distribution closely mirrors the ground-truth distribution, indicating that the model does not exhibit strong answer-position bias. Accuracy is relatively uniform across answer choices (66.7%–80.7%), suggesting that performance is not dominated by surface-level shortcuts.
Limitations and Future Work
Gating is cheap only relative to the stream. Where a 2B-scale encoder cannot keep pace with arriving frames it, and not the VLM, becomes the bottleneck, and the encoder term of § 3.1 governs rather than . The method carries twelve hand-set constants (Appendix § 9); we sweep and and toggle and , leaving the rest unexamined. The captioning study is small () and LLM-judged, which is why we lead with EgoSchema. Most consequentially, gating is a decision not to look: an event whose inter-frame dissimilarity falls below never triggers a burst, and the timeout bounds how long the system can go unobserving without guaranteeing that any particular event is captured. This is a property of content-adaptive gating in general rather than an artefact of our thresholds, and it bounds where a system of this kind belongs: we would not recommend it as the sole perception path where a missed event carries safety consequences, without an independent always-on channel for the events that matter.
The natural technical response to the last point is to stop fixing in advance: it is a threshold on a distribution the system already observes online, and Appendix § 9 sets out how a running quantile would replace it at no extra encoder cost. Our other next steps are lighter gating encoders, which lower that floor, and a multi-prior memory that would let the system carry more than one region of interest across bursts.
LLM Usage Disclosure
Large language models (GPT-5.2 and Claude) were used during the preparation of this manuscript as assistive tools for: (1) brainstorming paper structure and experimental design, (2) generating matplotlib plotting scripts, (3) rephrasing and copy-editing draft text, and (4) converting plain-text drafts to LaTeX format. All scientific claims, experimental results, and technical contributions are the authors’ own. GPT-5.2 also serves as the LLM-as-judge evaluator in our captioning benchmark (§ 4.1), which is an experimental component of the work, not a writing aid.
References
- [1]Natan Bagrov, Eugene Khvedchenia, Borys Tymchenko, Shay Aharon, Lior Kadoch, Tomer Keren, Ofir Masad, Yonatan Geifman, Ran Zilberstein, Tuomas Rintamaki, Matthieu Le, and Andrew Tao. Efficient video sampling: Pruning temporally redundant tokens for faster VLM inference. arXiv preprint arXiv:2510.14624, 2025.arxiv.org/abs/2510.14624
- [2]Shyamal Buch, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. Flexible frame selection for efficient video reasoning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 29071–29082, 2025.DOI
- [3]Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision (ECCV), pp. 19–35, 2024.arxiv.org/abs/2403.06764
- [4]Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? An analysis of BERT’s attention. In ACL Workshop on BlackboxNLP, pp. 276–286, 2019.arxiv.org/abs/1906.04341
- [5]Dima Damen, Hazel Doughty, Giovanni Maria Farinella, et al. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100. International Journal of Computer Vision, 130:33–55, 2022.DOI
- [6]Kristen Grauman, Andrew Westbury, Eugene Byrne, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18973–18990, 2022.arxiv.org/abs/2110.07058
- [7]Kristen Grauman, Andrew Westbury, Lorenzo Torresani, et al. Ego-Exo4D: Understanding skilled human activity from first- and third-person perspectives. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19383–19400, 2024.
- [8]Berthold K.P. Horn and Brian G. Schunck. Determining optical flow. Artificial Intelligence, 17(1–3):185–203, 1981.
- [9]Xiaohu Huang, Hao Zhou, and Kai Han. PruneVid: Visual token pruning for efficient video large language models. In Findings of the Association for Computational Linguistics (ACL), 2025.arxiv.org/abs/2412.16117
- [10]Mingi Jung, Saehyung Lee, Eunji Kim, and Sungroh Yoon. Visual attention never fades: Selective progressive attention recalibration for detailed image captioning in multimodal large language models. In International Conference on Machine Learning (ICML), 2025.arxiv.org/abs/2502.01419
- [12]Haotian Liu, Chunyuан Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
- [13]Shuming Liu, Chen Zhao, Tianqi Xu, and Bernard Ghanem. BOLT: Boost large vision-language model without training for long-form video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3318–3327, 2025.arxiv.org/abs/2503.21483
- [14]Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 12585–12602, 2024.arxiv.org/abs/2306.05424
- [15]Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. EgoSchema: A diagnostic benchmark for very long-form video language understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
- [16]Qwen Team. Qwen3-VL technical report. arXiv preprint, 2025.
- [17]Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaVA-ProMerge: Adaptive token reduction for efficient large multimodal models. In International Conference on Computer Vision (ICCV), pp. 22857–22867, 2025.arxiv.org/abs/2403.15388
- [18]Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. HoliTom: Holistic token merging for fast video large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2505.21334
- [19]Michael J. Swain and Dana H. Ballard. Color indexing. International Journal of Computer Vision, 7(1):11–32, 1991.
- [20]Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. DyCoke: Dynamic compression of tokens for fast video large language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18992–19001, 2025.arxiv.org/abs/2411.15024
- [21]Dezhan Tu, Danylo Vashchilenko, Yuzhe Lu, and Panpan Xu. VL-Cache: Sparsity and modality-aware KV cache compression for vision-language model inference acceleration. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.23317
- [22]Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024a.arxiv.org/abs/2409.12191
- [23]Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. VideoAgent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision (ECCV), pp. 58–76, 2024b.arxiv.org/abs/2403.10517
- [24]Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2312.13139
- [25]Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zhang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and Dahua Lin. Conical visual concentration for efficient large vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14593–14603, 2025.DOI
- [26]Sicheng Yu, Chengkai Jin, Huanyu Wang, Zhenghao Chen, Sheng Jin, Zhongrong Zuo, Xiaolei Xu, Zhenbang Sun, Bingni Zhang, Jiawei Wu, Hao Zhang, and Qianru Sun. Frame-Voyager: Learning to query frames for video large language models. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.03226
- [27]Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A. Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. SparseVLM: Visual token sparsification for efficient vision-language model inference. In International Conference on Machine Learning (ICML), 2025a.arxiv.org/abs/2410.04417
- [28]Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. LLaVA-Video: Video instruction tuning with synthetic data. Transactions on Machine Learning Research, 2025b.arxiv.org/abs/2410.02713
- [29]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023.arxiv.org/abs/2306.05685
