betaFeedback

WorksComputer Science

NexSight: Verifiable Tasks for Fine-Grained Agentic Long-Video Understanding

Abstract

Fine-grained questions about long videos ask how many times an action occurs, in what order events unfold, or exactly when a state changes. Answering them can require finding brief moments scattered across hours of footage. Agentic models that decide when and where to look promise to find these moments, yet this ability is hard both to measure and to learn. Sparse supporting evidence may establish that an action occurred without establishing its total count; verifying timeline-wide claims requires checking for missed occurrences and double counting. Answer-only supervision, in turn, does not explicitly supervise the search for evidence. We present NexSight, which builds verifiable tasks to both evaluate and train agentic long-video understanding. Its agentic pipeline, SightForge, accepts a task only if a verifier blind to the draft answer can reconstruct it from evidence densely reacquired from the raw video and counterexample checks fail to refute it. This yields SightLine, 310 evidence-traceable tasks from 108 long videos, evaluated through SightLoop, a shared multiscale harness supporting temporal exploration, local inspection, evidence integration, and revisiting while recording returned observations. On disjoint videos, solver trajectories on verified tasks become step-level supervision for observation decisions. The strongest model reaches only 54.52% on SightLine, yet 79.35% when given the annotated evidence. With under 6K examples, Sight-35B nearly doubles its base model's interactive accuracy (21.29% to 42.26%), outperforms the 397B model of its family, begins using explicit spatial inspection after post-training, which its base model never uses, and gains over 8 points on two external long-video benchmarks.

Introduction

Long videos record how events unfold over minutes to hours, and many questions about them are fine-grained:

how many times an action repeats, which of two distant events came first, or exactly when a state changes. The answers hinge on a few seconds scattered across the timeline, which uniform sampling, spreading a fixed frame budget thinly over hours, easily skips. Agentic long-video understanding instead lets the model decide when to search the timeline, which details to inspect, and how to integrate evidence across observations [28, 29, 30, 34, 35], spending its budget where the question demands. Realizing this promise requires two things that remain open: measuring whether an agent acquires the right evidence and uses it correctly, and teaching it to do so.

Measuring requires tasks whose answers are correct and whose evidence suffices for anyone to reconstruct them. Fine-grained tasks make this hard in principle: many are global claims, such as a count or an order, and proving that an action occurred exactly three times requires negative evidence that no occurrence was missed or counted twice. Grounded benchmarks associate answers with supporting frames or intervals [4, 33], but sparse support alone may not suffice to verify timeline-wide claims, and a verifier that sees the draft answer tends to confirm rather than reconstruct it [7]. With unverified references, errors cannot be attributed to missed evidence, faulty reasoning, or flawed tasks, and agentic evaluations, which mostly score answers or milestones [1, 30, 36] rather than the frames actually decoded [15], further entangle acquisition with use.

Teaching this ability is harder still: agents must decide where to look next, what to zoom into, and when the evidence suffices, yet final-answer supervision does not specify these decisions. Tool access is not enough: the evaluated instruction-tuned 35B model scores lower when allowed to search (28.06%) than when simply given 64 uniform frames (33.23%). Learning these decisions requires tasks that genuinely demand evidence seeking, answers trustworthy enough to supervise, and records of the search itself; few prior works train on their constructed data [34] (Table 1).

Benchmark / systemVideo-grounded task checksAdaptive visual observationShared interactive evaluationObserved-frame evaluationPost-training utility
Video-MME [10]✓−−−−
LongVideoBench [31]✓−−−−
Active Video Perception [30]−✓✓−−
LongVidSearch [36]✓−✓−−
TRACE / VES-Bench [15]✓✓−✓−
AgentVidBench [1]✓✓✓−−
LongVT / VideoSIAH [34]✓✓−−✓
NexSight (ours)✓✓✓✓✓

Table 1. ✓ Reported; − not reported under the criterion. Task checks: candidate QA checked against source videos beyond initial generation/annotation. Adaptive visual observation: agent-selected frames/clips reach a visual solver or observer, excluding caption-only retrieval. Shared interactive evaluation: a common tool interface evaluated across model backbones on the same benchmark. Observed-frame evaluation: actual decoded frames compared with reference evidence, rather than predicted intervals or textual milestones. Post-training utility: training experiments using the work’s constructed supervision. NexSight includes SightForge, SightLoop, and SightLine.

Our key insight is that verification is itself an active observation problem. Confirming a global claim means returning to the raw video, gathering dense evidence where the claim lives, and searching where counterexamples could hide. These are the same observation decisions we want agents to learn. A pipeline that annotates from a fixed view cannot do this; an agentic one can. The resulting verified tasks serve as both yardstick and training ground: their evidence certifies references and measures what agents observe, and on held-out videos they demand the very evidence seeking we aim to teach.

We build NexSight on this insight (Figure 1). SightForge constructs verifiable tasks for two paths: SightLine supports fine-grained evaluation through SightLoop’s budgeted, multiscale observation interface; separately constructed training tasks provide QA supervision and support solver-trajectory collection. Solvers operate without reference answers or annotated evidence. Their interaction records complement QA supervision for post-training Sight-35B, connecting reliable task construction with both the evaluation and learning of active observation.

NexSight overview

Figure 1. NexSight overview. (1) Verifiable task construction (SightForge): dense raw-video evidence, answer-hidden verification, and counterexample checks establish reliable tasks. (2) Multiscale active evaluation (SightLine and SightLoop): a 310-task benchmark and interactive harness assess temporal exploration, local inspection, and evidence integration. (3) Agentic post-training (Sight-35B): QA supervision and separately collected solver trajectories improve interactive accuracy by 20.97 percentage points over Base and support Direct-64 transfer to Video-MME Long and LVBench (123-question subset).

Evaluation shows that SightLine is far from solved: the strongest configuration reaches only 54.52%, yet 79.35% when given the annotated evidence, making acquisition a key bottleneck. At an identical frame budget, agentic window selection doubles temporal evidence recall over fixed windows (19.84%→40.63%19.84\% \to40.63\%), yet cross-window counting and state change do not improve: finding the right moments is not enough without integrating them.

Training teaches this behavior. Sight-35B raises interactive accuracy from 21.29% to 42.26%, overtaking the instruction-tuned model by 14.19 points and every configuration of the 397B model of its family. Its largest gains come on cross-window counting (+37.50) and state change (+35.19), precisely where better window selection alone did not help, and it begins using explicit spatial inspection after post-training (0.72 calls per trajectory), which its base model never uses. The gains transfer: +8.22 points on Video-MME Long (+9.33 on training-disjoint videos) and +8.13 on an LVBench subset.

Our contributions are threefold:

  1. Verifiable task construction. SightForge separates task proposals from acceptance through dense raw-video evidence acquisition, answer-hidden verification, independent review, and counterexample checks, yielding SightLine, a six-capability benchmark with traceable visual evidence (Secs. [3] and [5]).

  1. Observation-level evaluation. SightLoop couples temporal exploration, local inspection, evidence integration, and revisiting through a shared raw-video interface under explicit budgets. This multiscale harness records observations, answers, and resource use for joint analysis of evidence acquisition and task outcomes (Secs. [4] and [6]).

  1. Learning to seek evidence. We combine QA supervision with step-level examples from solver trajectories collected separately on verified training tasks. The resulting Sight-35B improves interactive performance and transfers to external long-video evaluation sets (Secs. 5.4 and 6.4).

Related Work

Fixed-input long-video understanding. Video-MME, LongVideoBench, and LVBench evaluate diverse video content, long-context referring reasoning, and extremely long videos, respectively [10, 27, 31]. EgoSchema emphasizes extended temporal reasoning, HourVideo hour-long perception and reasoning, and MLVU diverse long-video tasks [3, 17, 40]. MVBench and Perception Test provide skill-level evaluation [14, 22], complemented by temporal-detail and cross-time diagnostics [2, 5, 16]. LongVA extends language context to accommodate more video frames [38].

Active and agentic video understanding. Wang et al.'s VideoAgent, VideoTree, and Active Video Perception develop iterative or query-adaptive video observation [28, 29, 30]. Fan et al.'s VideoAgent adds event/object memory [8]; T* targets efficient keyframe search [35]. LongVT couples global previews with native video cropping; VideoSIAH supports evidence-sparse QA and tool-integrated training [34]. Image-based methods offer complementary mechanisms: V* guides visual search, Visual CoT supplies regional supervision, and VLM-R³ trains interleaved region selection and reasoning [13, 25, 32]. ReXTime and VideoEspresso construct video-reasoning supervision [5, 11], while Video-R1 and VideoRFT combine supervised reasoning data with reinforcement learning [9, 26].

Agentic and evidence-grounded video evaluation. NExT-GQA and CG-Bench evaluate answer grounding [4, 33], while MINERVA diagnoses intermediate reasoning errors [18]. LongVidSearch evaluates multi-hop retrieval planning, while VideoGAIA studies tool-augmented video assistance [36, 37]. AgentVidBench evaluates multi-hop video reasoning with tool-agnostic evidence milestones [1]. TRACE / VES-Bench jointly audits decoded-frame coverage and answer correctness for ordering and counting [15]. NexSight connects verifiable construction, raw-visual interaction, and observation-level evaluation (Table 1).

The SightLine Benchmark

Task Definition and Information Boundaries

We operationalize agentic long-video understanding as task-conditioned observation selection: the model uses question and accumulated observations to choose subsequent temporal windows or local visual inspections, and decides when to answer within a specified budget BB.

Given a video V={(xt,t)}t∈[0,L]V = \{(x_t,t)\}_{t \in[0,L]} and a question qq, the model selects an observation action uku_k from its history hkh_k, receives visual observations oko_k, or outputs a final answer a^\hat{a} and stops:

uk∼πθ(⋅∣q,hk),ok=O(V,uk),(1)u_k \sim\pi_\theta(\cdot\mid q,h_k), \qquad o_k = \mathcal{O}(V,u_k), \tag*{(1)}

where O\mathcal{O} is the frozen video-access protocol. The goal is to acquire and integrate sufficient evidence to answer within budget BB.

Standard evaluation exposes only the question, permitted video metadata, and access interface; answers, annotated evidence locations, and construction summaries remain hidden. Answers must be grounded in video frames, including visible text, rather than webpages, video titles, external subtitles, or speech transcripts. Evidence Oracle is a separate diagnostic setting. Appendix § 11 specifies the sample and evidence schema.

Six Core Capabilities

SightLine covers six fine-grained capabilities (Figure 2). Search scope differs from evidence quantity: a single local window may suffice to verify an answer once found, yet locating it can require extensive temporal exploration. Formal identifiers appear in Appendix § 11.

Six core capabilities and task distribution of SightLine

Figure 2. Six core capabilities and task distribution of SightLine. The benchmark covers unlocalized fact retrieval without timestamp cues, ordering of separated events, counting across temporal windows without duplication, identity tracking across occlusions and shot changes, state/attribute change recognition, and temporal boundary localization. The donut chart shows the task count and percentage for each capability (310 tasks in total).

Scale and Composition

SightLine contains 310 verified tasks from 108 Video-MME Long videos, totaling 74.12 hours across five content domains, with 1,080 annotated evidence artifacts. Figure 2 and Appendix § 10 summarize its composition. Original Video-MME questions serve only source-question deduplication and leakage checks, not candidate generation. Appendix § 11 specifies source grouping.

SightLoop: Multiscale Active Observation

Task-directed human vision selectively acquires information according to the ongoing task [12]. Motivated by this principle, SightLoop couples two decisions for long-video reasoning: when to look along the timeline and what to inspect within an observed frame. Its observation loop connects global context, temporal exploration, and local visual inspection with evidence integration.

Temporal Exploration and Spatial Inspection

Global overview. A sparse full-video glance establishes coarse scene and event context. This temporal map guides subsequent exploration.

Temporal exploration. The video agent selects time windows for closer observation and can request earlier, later, or overlapping intervals. Returned frames expose local events that may be absent or ambiguous in the overview. Frame extraction makes selected visual evidence available for subsequent inspection.

Local Think-with-Image. The general-purpose harness provides an image-reasoning sub-agent that can crop, zoom, highlight, and compare regions from extracted frames. These operations let the agent examine answer-critical details such as identity cues, hand–object interactions, text, or state changes, following the image-assisted reasoning paradigm of DeepEyes [39]. The video agent selects the frame and inspection question; the image agent selects local operations. The image-reasoning sub-agent uses the same evaluated VLM and returns both transformed images and textual findings. It is invoked on demand, with its model calls, visual inputs, and token usage included in the evaluated system’s resource accounting.

Revisiting and Evidence Integration

Temporal exploration and spatial inspection form a revisitable observation loop. The agent uses accumulated observations to update its interpretation, identify missing evidence, and choose another window or visual region. For example, determining whether someone took an object before leaving can involve locating the approach and departure, inspecting the hand and tabletop, and revisiting an earlier view to compare object presence.

SightLoop records actual observations, decoded timestamps, and resource use. Optional structured memory (Ledger) retains observation summaries across turns. The agent may answer when it judges the evidence sufficient, or the run terminates at its configured budget. Tool availability and budgets are specified per configuration; § 6.2 isolates temporal selection using fixed window deliveries. Appendix § 11 gives execution and failure-handling details.

Answer, Evidence, and Efficiency Evaluation

We evaluate final answers using the frozen gpt-6-luna/high [21] semantic rubric and each question’s answer specification. Overall accuracy divides correct answers by 310; invalid, missing, and unresolved answers score zero. Temporal evidence coverage is measured by per-question frame-hit recall and its benchmark macro-average:

Rihit=1mi∑j=1mi1[Ti∩Iij≠∅],(2)R_i^{\mathrm{hit}} = \frac{1}{m_i}\sum_{j=1}^{m_i}\mathbb{1}\left[T_i \cap I_{ij} \ne\varnothing\right], \tag*{(2)}
Rhit=1N∑i=1NRihit.R^{\mathrm{hit}} = \frac{1}{N}\sum_{i=1}^{N} R_i^{\mathrm{hit}}.

Here TiT_i contains the decoded timestamps of actually returned frames, mim_i is the number of evidence annotations, and NN is the number of evaluated questions. Each IijI_{ij} is the annotated closed interval, with point evidence at tt evaluated over [t−2,t+2][t-2, t+2] seconds. Temporal hits do not imply sufficient visual evidence; answer accuracy and frame-hit recall jointly diagnose evidence-acquisition versus evidence-use failures. We additionally report model/tool calls, observed frames, and recorded token usage to characterize interaction resources. Full scoring and accounting rules appear in Appendix C.

SightForge: Evidence-Verifiable Data Construction

SightForge uses an agentic skill-based workflow to construct reliable tasks through full-video indexing, candidate generation, evidence acquisition, and verification. Figure 3 details how candidate tasks acquire raw-video support and pass independent checks before acceptance. The agent schedules these stages under shared skill protocols, yielding verified benchmark tasks or training data according to source-video membership. Construction is strictly separated from the evaluated VLM’s task-solving process.

SightForge verification workflow

Figure 3. SightForge: verification before acceptance. The verifier reconstructs an answer from the question and reacquired evidence, without seeing the draft answer or generation reasoning. Comparison, independent review, and counterexample checks gate acceptance; unresolved candidates return for additional evidence or revision, or are rejected. Solver trajectories are collected separately on training tasks.

Full-Video Indexing and Randomized Observation

Stratified randomized observations cover the full timeline so that candidate generation does not depend on fixed uniform sampling locations. An observation model extracts visible information from timestamped storyboards; deterministic temporal organization and redundancy compression produce a timeline summary without further free-form model rewriting. The summary guides proposals and search; final evidence is acquired from the raw video.

Task Generation and Dense Evidence Acquisition

Candidate generation proposes a question, draft answer, and evidence plan; verification requires returning to the raw video. The agent reacquires dense evidence from target windows and assembles chronological multi-frame Evidence Panels that preserve local changes and temporal order. Evidence acquisition is task-dependent: boundary tasks retain views before and after a transition; counting tasks cover the specified scope and counting units; identity tasks preserve cross-window identity anchors; and text-related tasks check readability across neighboring frames. These requirements guide dense reacquisition and independent review. The plan also identifies potential counterexamples; gaps or conflicts trigger further observations, revisions, or candidate rejection.

Answer-Hidden Verification and Counterexample Checks

Verification checks Answer Correctness against video facts, Evidence Sufficiency for supporting the answer, and

Question Validity: clarity, unambiguity, and answerability. A verifier independently answers from the question and evidence without seeing the draft answer or generation reasoning; its answer is then compared with the draft. Candidates then undergo independent review and counterexample checks.

Independent blind review, conducted by human reviewers or a separately instantiated Codex reviewer, then reconstructs the answer and submits supporting evidence. Counterexample checks inspect adjacent windows and broader task-required ranges; global claims such as counts and first or last occurrences require broader inspection with the search scope recorded. Only candidates passing all three verification dimensions, independent blind review, and counterexample checks enter the task pool for release assembly and validation. Sampling parameters, evidence layouts, and review rules appear in Appendix B.

Training Data Construction

Our construction protocol separates evaluation and training by original-source ID (Vtrain∩Vtest=∅V_{\mathrm{train}} \cap V_{\mathrm{test}} = \varnothing; Appendix § 11). Beyond static QA/evidence supervision, agentic supervision leverages actual solver interactions collected on training tasks. While construction trajectories record how a question is proposed and verified, solver trajectories record how an agent solves the task from scratch without access to the reference answer or annotated evidence.

Each retained solver call becomes a step-level training example, with its interaction history as context and the current assistant output as the target. Loss is computed only on the current target turn, whereas historical turns, visual inputs, and tool returns serve as context. Tool-call targets retain recorded assistant text, reasoning, and action arguments, while answer targets retain the solver’s original responses. Step-level conversion filters out malformed calls or execution errors rather than filtering trajectories by final-answer correctness, preserving diverse exploration behaviors.

Experiments

Experimental setup. All SightLine results and diagnostics use the same 310-task set. Direct-64 and Direct-128 sample uniform fixed frames. We evaluate two complementary interactive settings: the full multiscale SightLoop harness measures end-to-end agentic performance with configuration-specific interaction budgets and optional structured memory (Ledger On/Off); the matched-budget protocol in § 6.2 isolates temporal window selection. SightLine uses the frozen gpt-6-luna/high semantic rubric; external benchmarks use deterministic scoring. Invalid or missing outputs remain in the full denominator. Appendices B–C specify execution and scoring; Table 12 reports recorded full-harness budget profiles.

ProfileModel callsTool callswatch_clip callsTime (min)
B18161012
B210221418
B314362425
B414362460
B5143624120
B624603270
B7246032120
B8246032150
B9246032180
Model / deploymentLedgernnB1B2B3B4B5B6B7B8B9
GPT-5.6 SolOn31045315600017701
GPT-5.6 SolOff310453158000550121
DeepSeek V4 Flash VisionOn310453056155240000
DeepSeek V4 Flash VisionOff310433155153280000
Nex-N2.5-MiniOn30841204500020200
Nex-N2.5-MiniOff30939214000020900
Qwen3.5-397B-A17BOn303392952000158025
Qwen3.5-397B-A17BOff299423151000146029
Qwen3.5-35B-A3B-InstOn30338173300021500
Qwen3.5-35B-A3B-BaseOn31042315700017820
Sight-35BOn310443155006211800

Table 12. Full-SightLoop evaluation budget profiles. Top: configuration limits. Bottom: counts for one selected trajectory per matched question (nn), excluding unmatched questions and additional attempts. For 30 configuration–question entries, multiple traces match the prediction; their budget attribution depends on the selected trace. Accuracy retains all 310 questions. Training-trajectory collection and matched temporal selection use separate configurations (Appendix § 11).

Overall Performance and Passive Context Scaling

Frontier models remain challenged despite gains from active exploration. Figure 4 compares frontier and open-weights models under fixed-input Direct and SightLoop configurations across all 310 tasks. GPT-5.6 Sol achieves the highest overall accuracy at 169/310 (54.52%) with Ledger Off and 165/310 (53.23%) with Ledger On, followed by DeepSeek V4 Flash Vision at 154/310 (49.68%) with Ledger On. Under passive Direct evaluation, doubling uniform frame inputs from 64 to 128 consistently improves performance, raising GPT-5.6 Sol from 37.74% to 43.55% (+5.81 points), DeepSeek from 31.29% to 36.45% (+5.16), and Qwen3.5-397B from 33.55% to 35.81% (+2.26). For GPT-5.6 Sol and DeepSeek V4, interactive SightLoop configurations exceed Direct-128, whereas Qwen3.5-397B achieves its highest score under Direct-128.

Figure 4. Overall model performance on SightLine. Accuracy over 310 tasks across fixed-input Direct and interactive SightLoop configurations (Ledger On/Off). Each model has two overlaid bars: Direct-64/128 and Ledger On/Off. The top and rounded internal boundary mark scores, not their sum. Reference models are ordered by their highest score, followed by Sight-35B. Model sources: GPT-5.6 Sol [20], DeepSeek V4 Flash Vision [6], Nex-N2.5-Mini [19], Qwen3.5-397B-A17B [23], and Qwen3.5-35B-A3B-Base [24].

Matched-Budget Active Observation

Because the full SightLoop harness dynamically allocates variable tool calls and spatial inspections, we conduct a strictly controlled matched-budget study to isolate the effect of temporal window selection. Using GPT-5.6 Sol/high on all 310 questions, we compare Fixed-Window Interactive (uniformly spaced windows), Random-Window Interactive (three seeds averaged per question), and Agentic Window Selection. Every trajectory receives an identical budget: 32 overview frames followed by six 30-second windows of 16 frames each (128 delivered frame slots across seven logical calls), with spatial zooming disabled so that only temporal window coordinates vary.

Active search doubles temporal localization; answer gains are smaller. Figure 5 shows that Agentic selection raises temporal evidence recall (RhitR^{\mathrm{hit}}) from 19.84% (Fixed) and 19.09% (Random) to 40.63%. The paired Agentic–Fixed gain is +20.79 points (95% video-cluster bootstrap CI: [+16.04, +25.60]). Agentic accuracy is 40.97% (127/310), exceeding Random (32.69%) by +8.28 points (CI: [+3.81, +12.75]). The accuracy gain over Fixed (36.13%, 112/310) is 4.84 points (95% CI: [−0.96, +10.51]) and is not statistically significant. Capability-wise results examine how localization gains relate to answer accuracy (§ 6.5; Appendix § 15).

Figure 5. Temporal selection under matched budgets. Only window selection varies across a fixed 128-frame budget. Bars show 310-task accuracy and temporal evidence recall (RhitR^{\mathrm{hit}}); error bars denote 95% video-cluster bootstrap confidence intervals. Random averages three seeds per question. The paired Agentic–Fixed temporal-hit difference interval is positive (+20.79 points), while the accuracy-difference interval crosses zero (+4.84 points; Appendix § 15).

Evidence Acquisition and Evidence Use

To examine model performance when reference evidence is directly supplied, we evaluate an Evidence Oracle using annotated GT artifacts. GPT-5.6 Sol reaches 246/310 (79.35%; Figure 6), versus 53.23%–54.52% under SightLoop exploration. Oracle differs in reasoning setting (xhigh), single-call execution, and input format (GT evidence JPGs, including multi-frame panels); together, these conditions probe task performance when annotated visual support is directly available.

Figure 6. Evidence-availability diagnostic. GPT-5.6 Sol performance with actively selected observations versus supplied ground-truth (GT) evidence under the same semantic rubric over all 310 tasks. Evidence Oracle supplies annotated GT evidence artifacts in a single pass with xhigh reasoning. Agentic On/Off denotes SightLoop with Ledger On/Off.

Comparing per-question outcomes between standard SightLoop (Ledger On) and the Evidence Oracle yields 145 tasks solved by both, 101 solved only by the Oracle, 20 solved only under standard exploration, and 44 missed by both. These contrasting outcomes guide case-level analysis of observation and answering behavior. Detailed quadrant analyses and qualitative case studies are provided in Appendix § 14.

Training Utility and Cross-Benchmark Transfer

Agentic post-training substantially improves interactive problem solving. We post-train Qwen3.5-35B-A3B-Base [24] on 5,938 examples constructed via SightForge:

2,623 Direct QA examples and 3,315 step-level interaction examples derived from 837 solver trajectories (comprising 2,499 tool-call steps and 816 final-answer steps). Relative to Base, Sight-35B improves Direct-64 accuracy by 2.58 points and achieves 42.26% versus 21.29% (+20.97 points) with SightLoop (Ledger On) (Table 2).

Model / deploymentDirect-64Agentic
Qwen3.5-35B-A3B-Base80 / 25.8166 / 21.29
Qwen3.5-35B-A3B-Inst103 / 33.2387 / 28.06
Sight-35B (ours)88 / 28.39131 / 42.26

Table 2. Fixed-input and agentic performance of 35B models on SightLine. Entries report correct answers / accuracy (%) over 310 tasks; inference budgets vary across deployments.

Fixed-input versus agentic performance. The API-served Qwen3.5-35B-A3B-Instruct model leads Sight-35B in Direct-64 (33.23% vs. 28.39%), whereas Sight-35B leads by 14.19 points in interactive evaluation (42.26% vs. 28.06%; Table 2). This ranking reversal motivates evaluating passive QA and interactive solving separately.

NexSight supervision transfers to external fixed-input benchmarks. Under Direct-64, Sight-35B improves over Base by 8.22 percentage points on Video-MME Long (900 questions) and 8.13 points on the frozen 123-question LVBench subset (Table 3). See Appendix § 16 for details.

BenchmarkQuestions35B BaseSight-35BΔ\Delta
Video-MME Long900477 / 53.00551 / 61.22+8.22
LVBench subset12355 / 44.7265 / 52.85+8.13

Table 3. Direct-64 cross-benchmark transfer: correct answers / accuracy (%), with gains in percentage points.

Capability-Wise Analysis

Temporal selection vs. cross-window integration. The capability breakdown (Figure A7, Appendix § 15) shows higher RhitR^{\mathrm{hit}} under Agentic selection across all six capabilities. Accuracy increases for temporal boundary detection (6.00% →\rightarrow 36.00%) and unlocalized fact retrieval (34.78% →\rightarrow 44.93%). Cross-window counting, event ordering, and identity tracking instead show comparable or lower accuracy than Fixed, suggesting that temporal intersection alone does not guarantee sufficient evidence coverage or correct multi-window integration.

Figure A7. Matched-budget temporal observation diagnostics. (a,b) Answer accuracy and temporal evidence hit across six capabilities; both heatmaps use a 0–100% color scale and display one decimal place. Capability counts in row order are 50, 32, 69, 45, 60, and 54; full capability names follow Figure A4. Random averages three seeds. (c) Paired video-cluster bootstrap differences with 95% confidence intervals; A/F/R denote Agentic/Fixed/Random. The Agentic–Fixed accuracy interval includes zero. (d) Point-evidence tolerance sensitivity. Spatial tools are disabled to isolate temporal window selection.

Capability gains under the combined training recipe. Under SightLoop (Ledger On), Sight-35B shows substantial gains on cross-time tasks after combined QA and solver-trajectory post-training (Figure 7). Relative to 35B Base, cross-window counting rises from 15.62% to 53.12% (+37.50 points), state/attribute change from 18.52% to 53.70% (+35.19), identity tracking from 26.67% to 53.33% (+26.67), and event ordering from 8.33% to 28.33% (+20.00); unlocalized fact retrieval changes modestly from 40.58% to 42.03%. The combined recipe improves cross-time task performance and accompanies greater use of explicit spatial inspection: Sight-35B invokes these tools 0.72 times per recorded trajectory versus 0.00 for Base (Table 11).

Capability accuracy (%) across six capabilities for GPT-5.6 Sol, DeepSeek V4 Flash Vision, Qwen3.5 397B-A17B, Qwen3.5 35B-A3B Base, and Sight-35B A3B (ours)

Figure 7. SightLoop (Ledger On) capability accuracy (%) on the 310-task SightLine benchmark; row labels give task counts. Sight-35B gains most over Base in cross-window count and state change under combined post-training, complementing the temporal-localization diagnostic in § 6.2.

Table 11. Primary observation-tool calls per recorded SightLoop trajectory (nn). Spatial operations comprise crop, zoom, highlight, and region comparison. Totals in Table Table 8 also include memory, metadata, and file tools and use 310-task normalization. Image-subagent invocations are separate from child tool calls; these columns are not additive.

Model / deploymentLedgernnOverviewTemporal windowFrame extract.Explicit spatialImage subagentVideo / image OCR
GPT-5.6 SolOn3101.0411.691.500.250.130.44 / 0.27
GPT-5.6 SolOff3101.0412.331.490.160.100.29 / 0.23
DeepSeek V4 Flash VisionOn3101.165.811.550.490.150.70 / 0.28
DeepSeek V4 Flash VisionOff3101.136.911.680.500.170.79 / 0.26
Nex-N2.5-MiniOn3081.077.725.050.480.290.56 / 0.49
Nex-N2.5-MiniOff3091.057.764.100.320.230.61 / 0.52
Qwen3.5-397B-A17BOn3031.0010.760.730.010.070.18 / 0.09
Qwen3.5-397B-A17BOff2991.009.660.590.000.040.13 / 0.06
Qwen3.5-35B-A3B-InstOn3031.007.840.530.000.050.67 / 0.22
Qwen3.5-35B-A3B-BaseOn3101.009.370.390.000.040.53 / 0.06
Sight-35BOn3101.289.323.250.720.141.04 / 0.32

Table 11.

Model / checkpointLedgerAcc. (%)Model callsTool callsObserved framesRecorded tokens
GPT-5.6 SolOn53.2311.9919.72195.99331,446
GPT-5.6 SolOff54.529.6417.82211.20252,835
DeepSeek V4 Flash VisionOn49.6811.2614.63149.08268,386
DeepSeek V4 Flash VisionOff48.3910.1613.66159.14232,136
Qwen3.5-397B-A17BOn31.6116.6714.06274.19506,441
Qwen3.5-397B-A17BOff34.8415.0812.16248.36408,774
Qwen3.5-35B-A3B-BaseOn21.2914.4212.51180.42418,889
Sight-35B (ours)On42.2612.8522.82219.62409,360
Nex-N2.5-MiniOn41.9411.7819.44166.60276,343
Nex-N2.5-MiniOff42.9010.4516.81154.35224,801
Qwen3.5-35B-A3B-InstOn28.0612.3611.11188.99379,249

Table 8. SightLoop accuracy and mean recorded resources per task (310-task normalization).

Limitations

While NexSight advances verifiable evaluation and training for agentic long-video understanding, several limitations remain. First, SightLine focuses on visual evidence and on-screen text across 310 tasks from 108 videos; incorporating audio-visual cross-modal verification and scaling to larger video pools are promising directions for future work. Second, our temporal frame-hit metric (RhitR^{\mathrm{hit}}) measures whether decoded frames intersect annotated intervals, which diagnoses temporal localization but does not guarantee that fine-grained spatial details are resolved; combining temporal hit with spatial grounding metrics would further refine trajectory diagnosis. Finally, while our combined QA and trajectory post-training recipe yields substantial gains, future work should isolate the relative contributions of QA supervision and step-level tool imitation under matched training budgets, and investigate whether reinforcement learning provides further gains.

Conclusion

We presented NexSight for defining, evaluating, and improving fine-grained agentic long-video understanding. SightForge separates candidate generation from raw-video evidence reacquisition, answer-hidden verification, and counterexample checks. SightLine provides 310 verifiable tasks across six capabilities, while SightLoop supports temporal exploration and local inspection under explicit resource accounting. Controlled diagnostics show improved temporal evidence localization, while capability analyses characterize how observation and training gains vary across tasks. Sight-35B, post-trained with combined QA and separately collected solver-trajectory supervision, improves interactive accuracy and external-benchmark scores. Together, checkable task evidence, observation-level evaluation, and targeted supervision provide a foundation for studying and improving active long-video understanding.

LLM Usage Statement

We used gpt-5.6-sol/xhigh for SightForge observation, candidate generation, and answer-hidden verification. Independent Codex-assisted checks complemented the separately recorded human reviews. Training solver trajectories were collected with deepseek-v4-flash-vision-exp/xhigh as the main agent. SightLine answers were scored with gpt-6-luna/high using the frozen semantic rubric. These research uses, information boundaries, and collection settings are specified in Appendices § 11 and § 12.

OpenAI Codex assisted manuscript editing, LaTeX preparation, plotting code, and figure/table layout. The authors reviewed the resulting text, code, numerical results, and scientific claims. The author-provided overview and pipeline illustrations were not generated with generative AI; visual evidence frames are extracted from source videos.

Supplementary Distributions of SightLine

SightLine contains 310 tasks across 108 long videos, totaling 74.12 hours. Figure A1 and Table 4 summarize the content-domain distribution, while Table 5 summarizes overall scale and evidence statistics.

SightLine benchmark overview and distributions

Figure A1. SightLine benchmark overview and distributions. The benchmark contains 310 tasks from 108 long videos across six fine-grained capabilities. The figure summarizes benchmark statistics, task proportions by capability and content domain, and the distribution of reference evidence artifacts per task.

Content domainVideosQuestions
Art Performance2260
Film / TV2258
Knowledge2166
Life / Vlog2268
Sports Competition2158
Total108310

Table 4. Content-domain distribution of SightLine.

StatisticValue
Questions / videos310 / 108
Domains / capabilities5 / 6
Total video duration74.12 h
Evidence JPG references1,080

Table 5. Task scale and evidence statistics. Evidence JPG references denote distinct annotated artifacts (including multi-frame panels).

Implementation, Sample Schema, and Reproducibility

Sample and evidence schema. Each sample is defined as si=(Vi,qi,ai,σi,Ei,μi)s_i = (V_i, q_i, a_i, \sigma_i, E_i, \mu_i), where aia_i is the reference answer; σi\sigma_i specifies answer type, units, tolerance, and allowed aliases; EiE_i contains evidence intervals and visual artifacts; and μi\mu_i records content domain, capability, and construction provenance.

Samples record video ID and hash, question ID and text, core capability, content domain, answer specification, evidence intervals, and staged review records. Answer specifications contain kind, value, unit, tolerance, and aliases fixed before evaluation. Evidence items contain start, end, role, description, sampled timestamps, and source hash. Public evaluation inputs are stored separately from answers, evidence locations, and construction summaries.

Capability identifiers. The benchmark uses the six formal capability identifiers and task counts listed in Table 6.

CapabilityIdentifierN
Temporal boundarytemporal_boundary50
Cross-window countcross_window_count32
Unlocalized fact retrievalunlocalized_fact_retrieval69
Identity trackingidentity_tracking45
Event orderevent_order60
State/attribute changestate_attribute_change54

Table 6. Formal capability identifiers and task counts.

Construction configuration. The observation, generator, and verifier stages all use gpt-5.6-sol/xhigh. Full-video indexing samples 72 frames, arranged into six storyboards of 12 frames each, with each frame resized to a width of 320 pixels. Dense evidence is subsequently acquired according to task requirements. The verifier runs in a separate context with the question and reacquired evidence, while the draft answer and generation reasoning are withheld. Review records retain reconstructed answers, answer-hidden status, reviewer type, verification judgments, inspection scope, and rejection reasons.

Stage protocols and full-video indexing. Skill protocols specify each stage’s input schema, permitted operations, output schema, and acceptance criteria. Videos are selected by domain and subcategory with frozen source versions. Indexing divides the full timeline into 72 segments, selects one randomly jittered frame per segment using a fixed seed, and arranges them chronologically into six timestamped storyboards of 12 frames each. An observation model records visible people, actions, props, scene changes, and on-screen text for each storyboard. A deterministic program organizes observations by timestamp and compresses redundant information into a timeline summary without free-form model rewriting.

Candidate schema and evidence organization. Based on the timeline summary, the generator proposes a question, draft answer, task type, structured answer specification, and evidence/retrieval plan. The plan identifies events, time windows, and potential counterexamples to inspect. Evidence organization follows task requirements: boundary questions retain observations before and after a transition; counting questions cover the counting scope and specify units; identity questions retain cross-window identity anchors; and text questions check readability across neighboring frames. The benchmark references 1,080 evidence JPG artifacts. Multi-frame panels preserve local changes and temporal order, enabling reviewers to independently reconstruct answers rather than relying on a single isolated frame.

Independent review and counterexample records. Human or independent Codex reviewers reconstruct answers from an answer-hidden review queue and submit supporting evidence, producing judgments of supported, ambiguous, or rejected. Model, Codex, and human reviews are logged separately. Adjacent-window and broader-range checks target missed events, double counting, identity confusion, and shifted boundaries. Global claims such as counts and first or last occurrences require recording the corresponding search scope, combining local evidence support with global counterexample checks. Figure A2 summarizes the human review outcomes for the candidate task pool.

Human review outcomes for SightForge candidate tasks

Figure A2. Human review outcomes for SightForge candidate tasks. Of 370 candidates, 188 were accepted as is, 122 were retained after minor wording edits, and 60 were rejected. The resulting SightLine benchmark contains 310 tasks, corresponding to an 83.8% retention rate.

Source grouping and training trajectories. Source isolation is checked on the video pools and verified on the training records using case-sensitive original-source IDs rather than generated-task IDs. Video-MME numeric IDs are mapped to original video IDs in the selection metadata; LongVILA records are mapped through their source-task manifest. All observations and tasks sharing an original-source ID are assigned to the same source group. The training pool supplies verified QA pairs for static supervision, while agentic supervision collects actual solver interactions on training tasks (question, observation, reasoning, video action, new observation, and final answer). Unlike construction trajectories, solver trajectories operate without access to reference answers or annotated evidence.

Acceptance and packaging. Accepted samples pass question-validity, answer-specification, evidence-reconstruction, model blind-verification, independent-review, counterexample, and source-question leakage checks before release assembly. Strict validation covers unique IDs, normalized-question deduplication, video and evidence hashes, timestamp bounds, field validity, and review-record completeness. Release artifacts include the evaluation task file, accepted annotations, a video-selection list, a manifest, and validation records.

Resource budgets and execution rules. The resource budget is defined as

B=(Tmax⁡,Fmax⁡,Pmax⁡,Kmax⁡),(3)B = (T_{\max}, F_{\max}, P_{\max}, K_{\max}), \tag*{(3)}

constraining interaction turns, actually observed frames, visual pixel resolution, and the input-token limit, respectively. Both the initial overview and subsequent observations count toward the budget; multi-frame panels are charged by their number of source frames. Each interaction logs action parameters, actual returned observations, timestamps, and resource usage, yielding a replayable trajectory. Termination, invalid actions, budget overruns, and technical failures follow a fixed execution protocol. Technical retries cover only transient API failures and never re-run questions based on answer correctness. Accuracy uses the full 310-task denominator, counting invalid or missing answers as incorrect.

Full-SightLoop evaluation configuration. Base and Sight-35B use a shared main YAML configuration with the same main prompt and nine-tool interface, Ledger On, max_iterations=100, a 192k evaluation context, and MEDIA_RESOLUTION_HIGH. The YAML specifies limits of 24 model calls, 60 tool calls, 32 watch_clip calls, one subagent invocation, and 7,200 seconds per task. Actual per-trajectory budget profiles vary across runs and are reported in Table 12 alongside their configuration counts.

Controlled diagnostic configurations. Matched temporal selection uses gpt-5.6-sol/high: 32 overview frames plus six 16-frame local windows, 128 delivered frame slots, and seven logical calls, with spatial tools disabled. Evidence Oracle uses gpt-5.6-sol/xhigh with supplied GT artifacts.

Solver trajectory collection. Across the 837 retained training trajectories, the root/main-agent requests use deepseek-v4-flash-vision-exp with xhigh reasoning. Collection uses the SightLoop interface for Ledger, video overview, clip observation, frame extraction, OCR/visual experts, and an image subagent. Per-trajectory limits are 14 model calls, 36 tool calls, 24 watch_clip calls, one subagent invocation, and 1,800 seconds per trajectory, with a 600-second timeout per LLM request. Model-call counts follow the collection-log definition. Solvers receive neither reference answers nor annotated evidence.

Trajectory availability and conversion. We collect 837 usable solver trajectories on training tasks, yielding 3,315 step-level training examples (2,499 tool-call steps and 816 final-answer steps). We retain valid step-level interactions without filtering by final-answer correctness to preserve search and recovery behaviors. Together with 2,623 Direct QA examples, these form the 5,938-example training dataset.

Training-example construction. Each retained main-agent model call produces one example: interaction history provides context, and only the current assistant output contributes to the training loss. Tool-call targets retain tool names, arguments, and assistant reasoning; answer targets retain the solver’s original output. Step-level filters exclude malformed tool arguments and out-of-budget calls.

Training data and source separation. Sight-35B is trained on 5,938 examples across 789 original-source video groups: 2,623 Direct QA records from 433 groups and 3,315 trajectory step records from 356 disjoint groups, with zero duplicate lines. Source-ID checks confirm zero overlap between the 108 SightLine evaluation videos and these 789 training sources (Vtrain∩Vtest=∅V_{\mathrm{train}} \cap V_{\mathrm{test}} = \varnothing).

Sight-35B is fine-tuned from Qwen3.5-35B-A3B-Base [24] for 186 steps (2 epochs) using seed 1234, sequence length 262,144, global/micro batch sizes 64/1, AdamW with learning rate 2×10−52 \times10^{-5} decaying to 2×10−62 \times10^{-6} via cosine scheduling (10 warmup steps), and weight decay 10−410^{-4}. Tensor/pipeline/expert/context parallel sizes are 8/1/8/2. The language model backbone is trainable while the vision encoder and projector remain frozen, with image resolution bounds of 200,704–3,211,264 pixels.

Reproducibility manifests. Exact file manifests and SHA-256 checksums for evaluation splits, scoring rubrics, and training records accompany the reproducibility data.

Scoring and Metric Definitions

The semantic judge is gpt-6-luna/high [21], using a frozen rubric and per-question answer specifications. Judge inputs include the question, reference answer, answer specification, and model prediction; the judge is blind to model identity and does not view the video. Invalid, missing, or unresolved answers score zero. Overall accuracy is ∑ici/N\sum_i c_i/N for the evaluated NN tasks, and task-macro accuracy is 16∑t∑i∈tci/Nt\frac{1}{6}\sum_t\sum_{i\in t} c_i/N_t.

In the response-validity and Oracle tables, Valid counts predictions successfully processed by the semantic judge. In the matched-seed table, it counts trajectories with a valid answer, successful execution, and a complete protocol. Unresolved counts judgments flagged for review; it can overlap with Valid and is not its complement. Such judgments score zero.

Timestamp-answer tolerances follow the per-question specification (1/2/3/5 seconds). Under a stricter min(original tolerance, 2s) sensitivity check on the matched-budget study, Fixed and Agentic scores remain unchanged at 112/310 and 127/310, while Random changes by a single item (304/930 to 303/930 across three seeds). This answer tolerance is distinct from the ±2-second window used for point-estimate temporal hit (RhitR^{\mathrm{hit}}).

RhitR^{\mathrm{hit}} averages the per-question fraction of annotated intervals intersected by actual decoded-frame timestamps. Point annotations use [t−2,t+2][t-2,t+2] by default, with ±1s and ±5s evaluated in sensitivity analyses (Figure A7). We log logical frame deliveries, unique source frames, multi-turn history image presentations, and API-reported token usage separately. External benchmarks use their standard deterministic scoring rules over full denominators.

SightLoop Implementation Details and Resource Accounting

Figure A3 illustrates the interaction between the video agent, temporal access, and on-demand image inspection.

SightLoop multiscale active observation

Figure A3. SightLoop multiscale active observation. Given a question and video overview, the video agent selects temporal windows, invokes an image agent for local inspection, integrates returned evidence, and revisits observations as needed. The image agent uses the same evaluated VLM. Optional Ledger memory retains observations across turns; calls, frames, pixels, and tokens are governed by explicit budgets, while actions, observations, and timestamps are logged.

Table 7–Table 9 summarize model performance, validity rates, and per-question resource consumption across Direct and SightLoop configurations (with structured memory Ledger On/Off). All tables use the same 310-task set. Under Direct-64 evaluation, strict final-answer formatting is produced on 295/310 tasks for the 35B Base model and 224/310 for Sight-35B, with unformatted reasoning-only outputs scored as incorrect in the full denominator.

CheckpointDirect-64SightLoop Ledger On
Qwen3.5-35B-A3B-Base80 / 25.8166 / 21.29
Sight-35B (ours)88 / 28.39131 / 42.26

Table 7. 35B checkpoint scores (correct / accuracy %) over all 310 tasks.

ModelLedgerValid / 310Semantic correctFull-set Acc. (%)
GPT-5.6 SolOn310/31016553.23
GPT-5.6 SolOff310/31016954.52
DeepSeek V4 Flash VisionOn310/31015449.68
DeepSeek V4 Flash VisionOff310/31015048.39
Qwen3.5-397B-A17BOn303/3109831.61
Qwen3.5-397B-A17BOff299/31010834.84
Qwen3.5-35B-A3B-BaseOn310/3106621.29
Nex-N2.5-MiniOn308/31013041.94
Nex-N2.5-MiniOff309/31013342.90
Qwen3.5-35B-A3B-InstOn303/3108728.06
Sight-35B (ours)On310/31013142.26

Table 9. SightLoop response validity and accuracy across all 310 tasks.

Tool-use breakdown. Table 11 summarizes temporal observation, explicit spatial operations, and image-subagent use.

Table 10. SightLine leaderboard over 310 tasks. Agentic On/Off denotes Ledger On/Off.

ModelConfigurationCorrect / 310Accuracy (%)
GPT-5.6 SolAgentic Off16954.52
GPT-5.6 SolAgentic On16553.23
DeepSeek V4 Flash VisionAgentic On15449.68
DeepSeek V4 Flash VisionAgentic Off15048.39
GPT-5.6 SolDirect-12813543.55
Nex-N2.5-MiniAgentic Off13342.90
Sight-35BAgentic On13142.26
Nex-N2.5-MiniAgentic On13041.94
GPT-5.6 SolDirect-6411737.74
DeepSeek V4 Flash VisionDirect-12811336.45
Qwen3.5-397B-A17BDirect-12811135.81
Qwen3.5-397B-A17BAgentic Off10834.84
Qwen3.5-397B-A17BDirect-6410433.55
Qwen3.5-35B-A3B-InstDirect-6410333.23
Nex-N2.5-MiniDirect-6410132.58
Qwen3.5-397B-A17BAgentic On9831.61
DeepSeek V4 Flash VisionDirect-649731.29
Sight-35BDirect-648828.39
Qwen3.5-35B-A3B-InstAgentic On8728.06
Qwen3.5-35B-A3B-BaseDirect-648025.81
Qwen3.5-35B-A3B-BaseAgentic On6621.29

Table 10.

Full-SightLoop budget profiles. Table 12 summarizes inference-budget profiles in evaluation trajectories matched to the reported predictions by question ID and final answer. Budget allocations vary across models and configurations. Per-question bindings, alternative trace matches, and source-file hashes accompany the reproducibility records.

Full-Set Evidence-Availability Diagnostic

Figure A4 summarizes the Evidence Oracle diagnostic across all 310 tasks. All conditions are scored by gpt-6-luna/high under the same semantic rubric. The Evidence Oracle supplies the ground-truth evidence JPG artifacts (including multi-frame panels) in a single answering call with gpt-5.6-sol/xhigh.

Oracle accuracy is 79.35% (95% video-cluster CI: [74.68, 83.87]); its difference from Ledger On is +26.13 points (paired 95% CI: [+19.68, +32.41]). The Oracle solves 101 tasks missed by Ledger On, while both conditions miss 44 tasks. These outcome groups guide case review; assigning a failure cause requires examining the corresponding observations and answers.

The active conditions have 310 judge-processed predictions each, with one unresolved judgment each; Oracle has 309 judge-processed predictions and no unresolved judgments. The active accuracies are 165/310 (53.23%) for Ledger On, 169/310 (54.52%) for Ledger Off, and 127/310 (40.97%) for Matched Agentic. Oracle correct counts in panel (b), from top to bottom, are 41/50, 28/32, 62/69, 33/45, 43/60, and 39/54. Valid and unresolved counts are not complementary (Appendix § 12).

Qualitative Examples

Figures A5 and A6 illustrate representative qualitative failure and success modes. They show evidence-acquisition failure with Oracle recovery, answer-detail omission under supplied evidence, and successful active temporal localization.

Qualitative diagnosis of agentic long-video problem solving

Figure A5. Qualitative diagnosis of agentic long-video problem solving (GPT-5.6 Sol). Left: SightLoop timeline ticks indicate returned overview frames, and blue bars show the union of local observation windows. Neither GT evidence group (at ∼81\sim81 s and 172 s) is intersected by the explored windows, whereas supplying GT frames recovers the correct answer. Right: Complete GT evidence is supplied, yet the model response omits the required posture transition.

Qualitative comparison of fixed versus active evidence acquisition

Figure A6. Qualitative comparison of fixed vs. active evidence acquisition (GPT-5.6 Sol). Under an identical observation budget, Fixed-Window sampling misses both annotated evidence points and answers incorrectly. Active selection redirects subsequent windows toward the end-card region after observing the cooldown segment, capturing the target promotional URL (BODiPreviews.com) and answering correctly (Rihit=1/2R_i^{\mathrm{hit}} = 1/2).

Matched-Budget Active Observation: Protocol and Capability Analysis

Figure A7 reports capability-wise accuracy and temporal evidence hit, paired differences, and tolerance sensitivity. Oracle capability results are shown separately in Figure A4.

Figure A4. Evidence-availability diagnostic on 310 tasks. (a) Per-task outcome counts for each active condition versus the same Oracle. (b) Capability accuracy for Matched Agentic, SightLoop (Ledger On), and Oracle. Boundary, Count, Fact retrieval, Identity, and State change abbreviate temporal boundary, cross-window count, unlocalized fact retrieval, identity tracking, and state/attribute change. Oracle uses supplied GT JPGs and xhigh reasoning in a single pass; protocols and budgets differ, so these are diagnostic rather than controlled causal comparisons.

The matched-budget study evaluates 1,550 trajectories across all 310 tasks: 310 for Fixed-Window, 3 × 310 for Random-Window (three random seeds), and 310 for Agentic Window Selection. Every trajectory receives 32 overview frames followed by six 16-frame local observations from 30-second windows (128 delivered frame slots across seven logical calls). Multi-turn history re-presentation yields 560 logical image presentations per trajectory. Random-Window results are averaged across the three seeds per question prior to benchmark aggregation. Confidence intervals (95%) are computed via 10,000 paired video-cluster bootstrap resamples over the 108 videos.

ConditionNValidUnresolvedCorrectAcc. (%)Hit (%)
Fixed310309011236.1319.84
Random (seed 0)310307310232.9020.38
Random (seed 1)31031019731.2917.47
Random (seed 2)310310210533.8719.43
Agentic310310112740.9740.63

Table 13. Per-condition output coverage and seed-level statistics across all 310 questions.

The paired Agentic–Fixed differences are +4.84 percentage points for accuracy (95% CI: [−0.96, +10.51]) and +20.79 for temporal hit ([+16.04, +25.60]). Against Random, the corresponding differences are +8.28 ([+3.81, +12.75]) and +21.54 ([+17.36, +25.93]). Exact two-decimal capability and tolerance values accompany the supplementary data.

Exact-frame repeats are identified via video identity and source presentation timestamps (PTS), while local-window overlap is defined as one minus the union duration divided by the summed window duration (excluding overview frames). Recorded token means range from 155,326 to 156,160 per question, a spread of 0.53% relative to Fixed (Table 14).

ConditionPhysical callsUnique framesAPI imagesTokensRepeat (%)Overlap (%)
Fixed8.14127.95665.96156,1600.040.00
Random (mean)8.10127.91661.97155,9600.073.14
Agentic8.07126.81658.43155,3260.939.00

Table 14. Per-question mean resource accounting in the matched-budget study.

Cross-Benchmark Transfer Breakdown

This section supplements § 6.4 with detailed breakdowns on Video-MME Long (900 questions) and LVBench (123-question subset). Both benchmarks are evaluated under the Direct-64 setting using standard deterministic multiple-choice extraction over the full question denominators. On Video-MME Long, Sight-35B also exhibits substantially higher generation stability than the 35B Base model, producing valid non-empty final responses on 849/900 questions compared to 729/900 for Base (which experiences more frequent generation timeouts and length overruns). In addition to Direct-64, evaluating LVBench (123-question subset) under an interactive OCR-enabled harness yields a +7.32-point improvement for Sight-35B over the Base model (Table 15).

BenchmarkQuestions35B BaseSight-35BΔ
LVBench subset12337 / 30.0846 / 37.40+7.32

Table 15. LVBench (123-question subset) evaluation under the interactive harness: correct answers / accuracy (%).

Figure A8 reports per-category and per-domain accuracies under Direct-64 following each benchmark’s native taxonomy.

Figure A8. Direct-64 transfer by task and content. Gray and mint points show 35B Base and Sight-35B accuracy; orange segments indicate lower Sight-35B accuracy. Parentheses give category sample sizes. LVBench task categories can overlap; overall accuracy uses the 123-question denominator. Video-MME denotes Video-MME Long (900 questions), and LVBench denotes the 123-question subset. Exact category values and full category names accompany the supplementary data.

Across Video-MME Long, Sight-35B achieves consistent gains across major reasoning categories—including object reasoning (+7.08 points), action reasoning (+12.22), temporal reasoning (+12.09), and action recognition (+25.40)—with positive improvements across all six content domains. On LVBench (123-question subset), Sight-35B improves substantially on entity recognition (+8.62), event understanding (+11.90), and key information retrieval (+14.71).

Source-Video Overlap and Transfer

Figure A9 summarizes source-video overlap among training data, SightLine, and the external evaluation sets. Table 16 and Table 17 compare Base and Sight-35B on the same Direct-64 questions within each source group. Source overlap refers to video identity, not identical training and evaluation questions.

Source-video overlap across training and evaluation sets

Figure A9. Source-video overlap across training and evaluation sets. SightLine contains 310 new tasks from 108 training-disjoint videos, corresponding to 324 original Video-MME questions. Of 300 Video-MME Long videos, 100 overlap with the 789 training sources. The 67-video LVBench subset shares three videos (five questions) with SightLine and one video (two questions) with training. Areas are not to scale; question counts and results appear in Table 16 and Table 17.

Table 16. Video-MME Long Direct-64 performance by training-source overlap. Counts refer to correct answers on the same questions; percentage-point gains are computed from the count difference within each row.

Video sourcesQuestionsBase correctSight-35B correctNet gainΔ\Delta (pp)
Overlapping training sources300173191+18+6.00
Disjoint from training sources600304360+56+9.33
All900477551+74+8.22

Table 16.

Table 17. LVBench subset Direct-64 performance by training-source overlap. Both models are evaluated on the same questions within each row. Net gain is the difference in the number of correct answers.

Video sourcesQuestionsBase correctSight-35B correctNet gain
Overlapping training sources201+1
Disjoint from training sources1215564+9
All1235565+10

Table 17.

Sight-35B improves in both source groups (Table 16 and Table 17). Training-source-disjoint questions account for 56 of the 74 additional correct answers on Video-MME Long and nine of ten on LVBench.

Data, Ethics, and Research Transparency

Data availability. Release materials include annotations, evaluation code, evidence/provenance metadata, manifests, and SHA-256 hashes under applicable licenses. Source videos remain under their original platform and dataset licenses and are not redistributed.

Ethics and human verification. Public benchmark videos may depict people or copyrighted material. Annotations follow dataset terms without disclosing sensitive personal information. Human blind reviews and automated model/Codex checks are logged separately.

References

  1. [1]Seoyeon An, Hyeonseo Jang, Minsu Kim, Chanho Lee, Younghan Park, and Kangwook Lee. AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents. arXiv preprint arXiv:2609.21386, 2026. doi:10.48550/arXiv.2609.21386. 1, 2, 3DOI
  2. [2]Mu Cai, Reuben Tan, Jianrui Zhang, Boch​​eng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jae Lee, and Jianwei Yang. TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models. arXiv preprint arXiv:24010.10818, 2024. https://arxiv.org/abs/2410.10818. 2
  3. [3]Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hour Video: 1-Hour Video-Language Understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2024. https://doi.org/10.52202/079017-1684. 2
  4. [4]Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, and Limin Wang. CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding. In International Conference on Learning Representations, 2025. https://arxiv.org/abs/2412.12075. 1, 2
  5. [5]Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen-Chun Chen, and Yu-Chiang F. Wang. ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos. In Advances in Neural Information Processing Systems, 2024. https://doi.org/10.52202/079017-0900. 2
  6. [6]DeepSeek. DeepSeek-V4-Flash-Vision-Exp Release: Multimodal API Now Live. Official release announcement, 2026. Released August 21, 2026. https://api-docs.deepseek.com/news/news260821/. Accessed November 5, 2026. 7
  7. [7]Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-Verification Reduces Hallucination in Large Language Models. In Findings of ACL, 2024. https://doi.org/10.18653/v1/2024.findings-acl.212. 1
  8. [8]Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding. In Proceedings of ECCV, 2024. https://arxiv.org/abs/2403.11481. 2
  9. [9]Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Beny​​ou Wang, and Xiangyu Yue. Video-R1: Reinforcing Video Reasoning in MLLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2025. https://doi.org/10.52202/085713-3310. 2
  10. [10]Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-Modal LLMs in Video Analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24108–24118, 2025. CVPR 2025 Open Access. 2, 3arxiv.org/abs/2405.21075
  11. [11]Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xin Su, Shifeng Zhang, Xu Zhou, Xiaojun Qi, Yue Liao, and Si Liu. VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection. In Proceedings of CVPR, 2025. https://arxiv.org/abs/2411.14794. 2
  12. [12]Mary Hayhoe and Dana Ballard. Eye movements in natural behavior. Trends in Cognitive Sciences, 9(4):188–194, 2005. https://doi.org/10.1016/j.tics.2005.02.009. 4
  13. [13]Chaoya Jiang, Yongrui Heng, Wei Ye, Han Yang, Haiyang Xu, Ming Yan, Ji Zhang, Fei Huang, and Shikun Zhang. VLM-R3: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought. In Advances in Neural Information Processing Systems (NeurIPS), 2025. https://doi.org/10.52202/085713-2143. 2
  14. [14]Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. MVBench: A Comprehensive Multimodal Video Understanding Benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22195–22206, 2024. CVPR 2024 Open Access. 2arxiv.org/abs/2311.17005
  15. [15]Pengyang Liu, Junbo Niu, Xiaoyang Hu, Zhongyue Shi, Zitian Wang, Linjiang Huang, and Si Liu. TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understanding. arXiv preprint arXiv:2608.22516, 2026. doi:10.48550/arXiv.2608.22516. 1, 2, 3
  16. [16]Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Shisuo Chen, Xu Sun, and Lu Hou. TempCompass: Do Video LLMs Really Understand Videos? In Findings of ACL, 2024. https://doi.org/10.18653/v1/2024.findings-acl.517.2
  17. [17]Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2023. https://doi.org/10.52202/075280-2004.2
  18. [18]Arsha Nagrani, Sachit Menon, Ahmet Iscen, Shyamal Buch, Ramin Mehran, Nilpa Jha, Anja Hauth, Yukun Zhu, Carl Vondrick, Mikhail Sirotenko, Cordelia Schmid, and Tobias Weyand. MINERVA: Evaluating Complex Video Reasoning. In Proceedings of ICCV, 2025. https://arxiv.org/abs/2505.00681.2
  19. [19]Nex-AGI. Nex-N2.5-mini. Official model card, 2026. https://huggingface.co/nex-agi/Nex-N2.5-mini. Accessed November 5, 2026. 7
  20. [20]OpenAI. GPT-5.6 Sol. Official API model documentation, 2026. https://developers.openai.com/api/docs/models/gpt-5.6-sol. Accessed November 5, 2026. 7
  21. [21]OpenAI. GPT-6 Luna. Official API model documentation, 2026. https://developers.openai.com/api/docs/models/gpt-6-luna. Accessed November 5, 2026. 4, 15
  22. [22]Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovica, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception Test: A Diagnostic Benchmark for Multimodal Video Models. In Advances in Neural Information Processing Systems, 2023. https://arxiv.org/abs/2305.13786.2
  23. [23]Qwen Team. Qwen3.5-397B-A17B. Official model card, 2026. https://huggingface.co/Qwen/Qwen3.5-397B-A17B. Accessed November 5, 2026. 7
  24. [24]Qwen Team. Qwen3.5-35B-A3B-Base. Official model card, 2026. https://huggingface.co/Qwen/Qwen3.5-35B-A3B-Base. Accessed November 5, 2026. 6, 7, 15
  25. [25]Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2024. https://doi.org/10.52202/079101-0275.2
  26. [26]Qi Wang, Yanrui Yu, Ye Yuan, Rui Mao, and Tianfei Zhou. VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2025. https://doi.org/10.52202/085713-0155.2
  27. [27]Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiao-han Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. LVBench: An Extreme Long Video Understanding Benchmark. arXiv preprint arXiv:2406.08035, 2024. doi:10.48550/arXiv.2406.08035. 2arxiv.org/abs/2406.08035
  28. [28]Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. arXiv preprint arXiv:2403.10517, 2024. doi:10.48550/arXiv.2403.10517. 1, 2arxiv.org/abs/2403.10517
  29. [29]Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3272–3283, 2025. CVPR 2025 Open Access.arxiv.org/abs/2405.19209
  30. [30]Ziyang Wang, Honglu Zhou, Shijie Wang, Junnan Li, Caiming Xiong, Silvio Savarese, Mohit Bansal, Michael S. Ryoo, and Juan Carlos Niebles. Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding. arXiv preprint arXiv:2512.05774, 2025. Version 2, June 2026. doi:10.48550/arXiv.2512.05774. 1, 2, 3DOI
  31. [31]Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding. In Advances in Neural Information Processing Systems (NeurIPS), pages 28828–28857, 2024. doi:10.52202/079017-0907. 2, 3
  32. [32]Penghao Wu and Saining Xie. V* : Guided Visual Search as a Core Mechanism in Multimodal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13084–13094, 2024. CVPR 2024 Open Access. 2arxiv.org/abs/2312.14135
  33. [33]Junbin Xiao, Angela Yao, Yicong Li, and Tat Seng Chua. Can I Trust Your Answer? Visually Grounded Video Question Answering. In Proceedings of CVPR, 2024. https://arxiv.org/abs/2309.01327. 1, 2
  34. [34]Zuhao Yang, Sudong Wang, Kaichen Zhang, Keming Wu, Sicong Leng, Yifan Zhang, Bo Li, Chengwei Qin, Shijian Lu, Xingxuan Li, and Lidong Bing. LongVT: Incentivizing “Thinking with Long Videos” via Native Tool Calling. arXiv preprint arXiv:2511.20785, 2025. CVPR 2026. doi:10.48550/arXiv.2511.20785. 1, 2, 3
  35. [35]Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, and Manling Li. Re-thinking Temporal Search for Long-Form Video Understanding. In Proceedings of CVPR, 2025. https://arxiv.org/abs/2504.02259. 1, 2
  36. [36]Rongyi Yu, Chenyuаn Duan, and Wentao Zhang. LongVid-Search: An Agentic Benchmark for Multi-hop Evidence Retrieval Planning in Long Videos. arXiv preprint arXiv:2603.14468, 2026. doi:10.48550/arXiv.2603.14468. 1, 2, 3arxiv.org/abs/2603.14468
  37. [37]Fan Zhang, Guangming Yao, Jinyang Wu, Hao Wu, Zheng Lian, Xinyu Geng, Jingdong Chen, Yi Yuan, and Pheng-Ann Heng. VideoGAIA: A Benchmark for General AI As- sistants on Agentic Video Understanding. arXiv preprint arXiv:2608.14718, 2026. doi:10.48550/arXiv.2608.14718. 2DOI
  38. [38]Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangkao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long Context Transfer from Language to Vision. Transactions on Machine Learning Research, 2025. https://openreview.net/forum?id=3oRAWQVGIx. 2arxiv.org/abs/2406.16852
  39. [39]Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deep-Eyes: Incentivizing “Thinking with Images” via Reinforcement Learning. In International Conference on Learning Representations, 2026. https://arxiv.org/abs/2505.14362. 4
  40. [40]Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: Benchmarking Multi-task Long Video Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13691–13701, 2025. CVPR 2025 Open Access. 2arxiv.org/abs/2406.04264

Paper details

vlnc:W_ywu2tr3w
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex

View statement
Looking for related papers…
Contents