betaFeedback

WorksComputer Science

One Skill Does Not Fit All: Automatic Discovery and Taxonomy-Guided Routing of Frame-Selection Skills for Long-Video Question Answering

Abstract

Long-Video Question Answering (LVQA) requires locating decisive evidence in hour-scale videos under a limited frame budget. Most training-free methods apply the same frame-selection strategy to all questions, despite substantial variation in the evidence required by different question types. Our analysis shows that the relative effectiveness of frame-selection strategies varies across semantic categories and benchmarks, motivating adaptive evidence acquisition. In this paper, we introduce AutoSkill, a source-supervised framework for automatically discovering and routing executable frame-selection skills. Starting from a small labelled source pool, LLM agents iteratively propose, implement, evaluate, and refine candidate skills. For a target benchmark, AutoSkill uses only unlabelled question and option text to induce a shared semantic taxonomy, rewrite labelled source examples into the target style, and estimate a category-to-skill mapping. Neither target videos nor target answers are used in this process. At inference time, each question is assigned one skill, which selects the frames used in a single inference of the frozen video MLLM. Across five long-video benchmark splits, AutoSkill improves Qwen2.5-VL-7B and Qwen3.5-4B by 2.4% and 1.2%, respectively, demonstrating the effectiveness of our AutoSkill.

Introduction

Long-Video Question Answering (LVQA) requires answering questions by locating relevant visual evidence from videos spanning tens of minutes or even hours. Such videos arise in many real-world scenarios, including lectures, movies, and egocentric recordings ([?]; [?]; [11]; [?]). The evidence required to answer a question may appear briefly, recur across distant segments, or depend on the overall evolution of an event. Exhaustively processing all frames is impractical, as the resulting visual tokens quickly exceed the context capacity of current Multimodal Large Language Models (MLLMs) ([?]; [5]; [?]). Therefore, effective LVQA requires selecting a small set of informative frames under a limited visual token budget.

Frame selection is often treated as simple preprocessing. However, the selected frames define what the frozen MLLM can observe before it answers a question. Missing evidence cannot be inspected later, while irrelevant frames use the same limited visual budget.

same limited visual budget. Frame selection therefore acts as an evidence-acquisition policy in the reasoning pipeline. Different policies decide how to divide the frame budget.

Existing approaches address this challenge in two ways. They improve MLLMs through additional training or reduce visual inputs before inference. Training-based methods ([?]; [5]; [?]; [?]) extend context windows, compress visual tokens, or learn temporal representations. Although effective, they require substantial training resources and are often tied to specific model architectures. Training-free approaches ([?]; [?]; [?]; [?]; [12]; [?]) provide a more flexible alternative by selecting informative frames through mechanisms such as query-guided retrieval, temporal localization, coarse-to-fine search, and iterative evidence refinement. These methods can be directly applied to frozen MLLMs with low overhead. However, these approaches rely on a single manually designed frame-selection strategy, implicitly assuming that one selection rule can generalise across diverse LVQA questions and target domains.

This policy view leads to a deeper finding. LVQA is not one homogeneous retrieval problem. It contains different evidence-acquisition problems. For example, temporal reasoning questions may benefit from localized search, whereas counting or entity-related questions may require broader coverage or specialized sampling patterns. Figure 1 shows this heterogeneity at two levels. Within a benchmark, different question categories favour different skills. Across benchmarks, skill rankings also change. This means that evidence acquisition should adapt to the question and target distribution instead of using one fixed policy for every case.

Figure 1. (a) Per question type accuracy gains (Δ\Delta) over uniform sampling on LVBench. (b) Relative ranking of the four per-benchmark best sampling skills.

This finding raises a practical deployment challenge. The strongest fixed skill can only be identified by evaluating all candidates on labelled target examples, which are unavailable beforehand. A deployable system must therefore infer which skill to use from the observable target-query distribution alone. It must also avoid executing every skill for every question, as this would multiply video inference cost. The goal is thus to discover effective frame-selection skills, determine when each should be applied, and execute only one skill per question.

AutoSkill turns this idea into a practical method. AutoSkill represents each evidence-acquisition policy as an Figure 1: Frame-selection policies show two levels of heterogeneity. (a) Within LVBench, different policies are strong on different question types. (b) Across benchmarks, their relative rankings change. No evaluated fixed policy dominates every category and benchmark. These results motivate adaptive evidence acquisition.

executable frame-selection skill. It automatically discovers complementary frame-selection skills and performs target-adaptive, taxonomy-guided skill routing. AutoSkill consists of three stages, as illustrated in Figure 2. In Stage 1, AutoSkill uses an LLM agent to automatically discover a compact toolbox of frame-selection skills. Starting from a shared labelled source pool, the agent iteratively proposes, implements, executes, and evaluates candidate selection programs. Execution feedback guides subsequent iterations toward more effective and complementary skills, without requiring manual construction of the complete skill library. In Stage 2, AutoSkill adapts the discovered toolbox to a target LVQA distribution using only unlabelled target question and option text. It first induces a semantic taxonomy that captures the different forms of visual evidence required by target queries. Based on this taxonomy, AutoSkill constructs a target-style routing set by selecting semantically compatible labelled source examples and rewriting their questions and options to match the target query style, while preserving the original video grounding and answer labels. No target video or target answer annotation enters this adaptation process. Evaluating the discovered skills on the adapted routing set estimates their category-specific utility and produces a category-to-skill mapping. In Stage 3, AutoSkill applies this mapping during inference. Each test question is first assigned to a semantic category, after which only the corresponding frame-selection skill is executed before the frozen MLLM generates the final answer. Unlike post-hoc skill selection, AutoSkill does not require labelled target evaluation to identify a suitable skill. Unlike exhaustive skill execution, it retains approximately the inference cost of a single-skill method. AutoSkill does use labelled source examples as black-box feedback during skill search and routing-table construction. It is therefore source-supervised, training-free, and target-label-free. It does not update the video MLLM or auxiliary models used for frame-selection skills. Target adaptation uses question and option text only. It does not use paired target videos or target annotations. Our contributions are as follows:

Figure 2. Overview of AutoSkill. Stage 1 iteratively discovers a Top-KK skill toolbox on a capability-focused development set. Stage 2 uses unlabelled target question and option text and a target-style adaptation of the shared labelled pool to construct a category-to-skill routing table. Stage 3 predicts the semantic category of each test question, retrieves one associated skill, and answers with a single frozen MLLM pass, without using target answer annotations.

  1. We show that frame selection determines the evidence available to the downstream model. Different frame-selection skills excel on different question categories and benchmarks, indicating that no single policy is universally optimal. This motivates adaptive evidence acquisition.

  1. We present AutoSkill as a practical method based on this finding. It automatically discovers a diverse toolbox of executable frame-selection skills through LLM-agent-driven exploration and execution feedback. It further estimates category-specific skill utility from labelled source examples adapted to the unlabelled target-query distribution, without accessing target videos or answer annotations during routing-table construction.

  1. We introduce an efficient taxonomy-guided router that selects one automatically discovered frame-selection skill for each test question, avoiding exhaustive skill execution at inference. Across four long-video QA benchmarks comprising five evaluation splits, AutoSkill improves Qwen2.5-VL-7B and Qwen3.5-4B by up to 2.4% and 1.2%, respectively. Without using target answers, it matches or exceeds the post-hoc best fixed skill selected using target annotations.

Efficient Long-Video Understanding. Long-video understanding is constrained by temporal redundancy and limited visual tokens. Training-based methods address this through long-context adaptation, token compression, hierarchical representations, and external memory [?, 5, ?, ?], but they usually need video-language supervision and stay tied to one backbone. Training-free methods instead reduce the visual input before inference via query relevance [?, 22, ?], visual diversity [?, 24], event localization [?, ?, 12], or iterative search [?, 12, 22]. These approaches are strong and inexpensive, yet most optimize one selection mechanism and apply it to all questions. Query conditioning can change frame scores, but the evidence-acquisition strategy itself remains fixed. AutoSkill automatically discovers complementary frame-selection skills and learns when each skill should be applied.

Automatic Skill Discovery and Adaptive Routing. Adaptive computation appears in mixture of experts [?, 8], model cascades [3, 20], and

LLM tool use ([?], [25], [23]). Most systems assume an existing expert or tool library, and their routers often need supervised trajectories, learned retrievers, reward optimization, or multi-expert execution ([20], [?], [?]; [2]). Agent systems can request and refine tools during solving ([9]; [?]), but rarely build a reusable video skill toolbox from downstream evaluation. AutoSkill brings automatic skill discovery into long-video frame selection. An LLM agent discovers frame-selection skills on a capability-focused development set. AutoSkill then distills their execution history into a category-to-skill mapping without parameter updates or target benchmark answers.

Methodology

Frame Selection as Evidence Acquisition

Long-Video Question Answering (LVQA) requires selecting informative visual evidence from long videos before feeding them into a frozen MLLM. Given a question qq, candidate options O\mathcal{O}, and a video VV, a frame-selection skill ss selects a subset of frames:

Fs=s(V,q),∣Fs∣≤B,(1)\mathcal{F}_s = s(V,q), \qquad|\mathcal{F}_s| \le B, \tag*{(1)}

where BB denotes the frame budget. The frozen MLLM then produces the answer:

y^s=fθ(Fs,q,O).(2)\hat{y}_s = f_{\theta}(\mathcal{F}_s,q,\mathcal{O}). \tag*{(2)}

We use policy to mean a general strategy for acquiring visual evidence. We use skill to mean an executable program that implements such a policy. Thus, each frame-selection skill ss implements one evidence-acquisition policy and controls which visual evidence reaches the frozen MLLM. Existing training-free LVQA methods typically assume a universal frame selection strategy. However, different questions may require different evidence acquisition patterns, motivating a toolbox of complementary skills: S={s1,…,sL}\mathcal{S} = \{s_1,\ldots,s_L\}. For a target distribution Dt\mathcal{D}_t, the utility of a fixed skill is defined as:

Ut(s)=E(V,q,O,y)∼Dt[I(fθ(s(V,q),q,O)=y)].(3)U_t(s) = \mathbb{E}_{(V,q,\mathcal{O},y)\sim\mathcal{D}_t}\left[\mathbb{I}\left(f_{\theta}(s(V,q),q,\mathcal{O}) = y\right)\right]. \tag*{(3)}

Let Ut(s∣c)U_t(s \mid c) denote the utility of skill ss for semantic category cc. Evidence acquisition is heterogeneous when the skill with the highest utility changes across categories or target distributions. A universal selector uses one skill for every category. An adaptive selector instead uses a mapping from categories to skills. The strongest fixed skill is therefore:

st∗=arg⁡max⁡s∈S Ut(s),(4)s_t^* = \underset{s\in\mathcal{S}}{\arg\max}\ U_t(s), \tag*{(4)}

However, st∗s_t^* cannot be identified before deployment because its utility requires target videos and answer annotations for evaluation. Moreover, the optimal skill may vary across target distributions. Therefore, a practical LVQA system should select skills according to the target query distribution without exhaustively evaluating all skills. AutoSkill addresses this problem under a transductive target-adaptation setting. It is given a labelled source pool Ds={(Vis,qis,Ois,yis)}i=1Ns\mathcal{D}_s = \{(V_i^s,q_i^s,\mathcal{O}_i^s,y_i^s)\}_{i=1}^{N_s} and unlabelled target question and option pairs: Dt={(qit,Oit)}i=1Nt\mathcal{D}_t = \{(q_i^t,\mathcal{O}_i^t)\}_{i=1}^{N_t}. Only target query semantics are available during adaptation, while target videos, target answers, and target-domain skill performance are not used.

AutoSkill first discovers a toolbox S∗S^{*} from source data and then constructs a category-to-skill mapping

π(k)∈S∗,k∈{1,…,Kq},(5)\pi(k) \in S^{*}, \qquad k \in\{1,\ldots,K_q\}, \tag*{(5)}

where KqK_q denotes the number of semantic query categories. At inference time, only the skill assigned to the predicted category is executed.

This setting is source-supervised because source answer labels are used to score candidate programs and construct the routing table. It is training-free because these evaluations do not update any MLLM model or auxiliary models used in skills. It is also target-label-free because target annotations are not used. During target adaptation, AutoSkill uses target question and option text but not paired target videos or ground-truth answers.

Capability-Focused Development Set Construction

Automatic discovery of frame-selection skills requires an informative evaluation environment. Randomly sampled LVQA examples often provide limited feedback because they may be either trivially solved or consistently failed by the frozen MLLM, making it difficult to distinguish the benefits of different selection strategies.

We therefore construct a capability-focused development set from a shared labelled source pool. Specifically, we collect Ns=3,000N_s = 3{,}000 examples from VideoVista [17], ALLVB (Tan et al. 2025), and LongVideo-Reason [4]. The source pool is disjoint from all target evaluation benchmarks and is only used for skill discovery and routing construction. From this pool, we select intermediate-difficulty examples:

Ddev={(Vis,qis,Ois,yis)}i=1Ndev,(6)\mathcal{D}_{\mathrm{dev}} = \left\{(V_i^s,q_i^s,O_i^s,y_i^s)\right\}_{i=1}^{N_{\mathrm{dev}}}, \tag*{(6)}

where Ndev=300N_{\mathrm{dev}} = 300 and the frozen MLLM achieves approximately 60% accuracy. Such examples provide both successful and failed cases, enabling the agent to discover when alternative frame-selection mechanisms are beneficial. To maintain diversity, we uniformly stratify examples according to video duration and question categories. We use 300 examples to balance discovery quality and repeated evaluation cost, with the impact of development-set size analysed in Table 4(c).

Table 4. (a) Comparison with other training-free frame-selection methods under the 32-frame selection budget. (b) Computational cost versus RL post-training; data denote SFT+RL samples for the baselines and labelled source plus unlabelled target-text samples for AutoSkill. † uses training time reported by the official repository, and ‡ includes discovery and router built. (c) Development-set analysis across different capability-focused configurations; Base and Best denote uniform-sampling and best-discovered-skill accuracy, respectively. LongVB denotes LongVideoBench.

MethodFramesMLVU-devLongVB
Qwen2.5-VL3259.358.1
+ mDP3 ([?])3266.260.0
+ A.I.R. ([?])3267.561.4
+ AutoSkill3267.961.5
MethodUpdateDataCost
Video-R1 ([?])Yes165K+260K440 A100-h
LongVILA-R1 ([?])Yes36K+170K9.2–12.3K A100-h†
VideoZoomer ([?])Yes52K∼768 H100-h
AutoSkillNo3K+1.5K∼40-45 A100-h‡
NdevN_{\mathrm{dev}}BaseBestΔ\Delta
30060.063.0+3.0
45054.458.4+4.0
1,00049.952.9+3.0

Execution-Guided Skill Discovery

Without prior knowledge of effective frame-selection strategies, AutoSkill uses an LLM agent to discover executable skills rather than manually designing a fixed toolbox.

Starting from Ddev\mathcal{D}_{\mathrm{dev}}, the agent iteratively analyses previous execution results and failure cases, proposes new selection mechanisms, implements executable programs, and evaluates them using the frozen LVQA pipeline. The execution feedback guides subsequent exploration toward more effective and complementary evidence acquisition strategies. Each candidate skill specifies its selection hypothesis, input signals, auxiliary models if required, and frame allocation procedure. Before evaluation, a reviewer agent verifies program correctness, auxiliary model constraints, and compliance with the frame budget. All candidates are evaluated under the same frozen MLLM and inference configuration, ensuring that performance differences mainly reflect the frame-selection strategy. For each skill ss, we measure:

R(s,Ddev)=1Ndev∑i1[fθ(Vis,qis,Ois)=yis].(7)R(s,\mathcal{D}_{\mathrm{dev}})=\frac{1}{N_{\mathrm{dev}}}\sum_i \mathbb{1}\left[f_{\theta}(V_i^s,q_i^s,O_i^s)=y_i^s\right]. \tag*{(7)}

Besides overall accuracy, the agent receives stratified feedback by video duration, question category, and data source to identify where each skill is effective. After multiple exploration cycles, the Top-KsK_s remaining skills form the discovered toolbox, here we set Ks=5K_s=5:

S∗={s1,…,sKs}.(8)S^{*}=\{s_1,\ldots,s_{K_s}\}. \tag*{(8)}

The toolbox is then frozen before target-query adaptation.

Target-Adaptive Skill Routing

Although the discovered toolbox contains complementary skills, their utility on the target distribution is unknown. AutoSkill therefore estimates category-specific skill preference from target query semantics and source supervision.

Query Taxonomy Induction. We sample NqN_q unlabelled question–option pairs from each of the J=5J=5 target benchmark splits and merge them into a unified query set:

Qt=⋃j=1J{(qt,j,i,Ot,j,i)}i=1Nq.(9)\mathcal{Q}_t=\bigcup_{j=1}^{J}\left\{(q_{t,j,i},O_{t,j,i})\right\}_{i=1}^{N_q}. \tag*{(9)}

We set Nq=300N_q=300, with the effect of query-set size analysed in Table 6. An LLM agent jointly analyses the pooled queries and organizes them into a shared semantic taxonomy:

Table 6. Routing-set size ablation on Qwen2.5-VL-7B (mean over 5 random rewrites from selected target-style samples). LongVB denotes LongVideoBench.

SamplesMLVU devMLVU testLongVB valVideoMMELVBenchAvg./Δ\Delta
baseline66.047.761.264.642.556.4 / –
30068.150.861.765.246.258.4 / +2.0
40068.551.261.765.146.358.6 / +2.2
50068.651.161.965.246.458.6 / +2.2
60068.651.161.865.246.658.6 / +2.2
80068.751.261.865.246.558.7 / +2.3
100068.851.161.865.246.658.7 / +2.3
120068.951.261.865.146.758.7 / +2.3
150069.051.161.965.246.858.8 / +2.4
T={τ1,…,τKq},Kq=19.(10)\mathcal{T}=\{\tau_1,\ldots,\tau_{K_q}\}, \qquad K_q=19. \tag*{(10)}

The induced categories characterize the evidence required to answer each query, including temporal localization, event ordering, counting, and global understanding. Full taxonomy details are provided in the supplementary material.

Target-Style Routing Set Construction. The taxonomy reveals what evidence is required by target queries, but it does not reveal which discovered skill is suitable for each category. Therefore, we construct a source-grounded routing set. For each category τk\tau_k, we retrieve semantically compatible source examples and rewrite their questions and options with an LLM to match the target query style, while preserving the original video, evidence requirement, and answer label:

Droute={(Vns,q~n,O~n,yns,cn)}n=1Nroute,(11)\mathcal{D}_{\mathrm{route}}=\left\{(V_n^s,\widetilde{q}_n,\widetilde{O}_n,y_n^s,c_n)\right\}_{n=1}^{N_{\mathrm{route}}}, \tag*{(11)}

The routing set is constructed from source examples disjoint from Ddev\mathcal{D}_{\mathrm{dev}}:

Ddev∩Droute=∅,(12)\mathcal{D}_{\mathrm{dev}}\cap\mathcal{D}_{\mathrm{route}}=\varnothing, \tag*{(12)}

Therefore, Droute\mathcal{D}_{\mathrm{route}} provides a target-style proxy for comparing skills without accessing target videos or answers.

Category-to-Skill Assignment. For each category kk and skill ss, we compute:

Ak,sroute=1Nk∑n∣cn=k1[fθ(Vns,q~n,O~n)=yns],(13)A_{k,s}^{\mathrm{route}}=\frac{1}{N_k}\sum_{n\mid c_n=k}\mathbb{1}\left[f_{\theta}(V_n^s,\widetilde{q}_n,\widetilde{O}_n)=y_n^s\right], \tag*{(13)}

The router assigns each category to the skill with the highest estimated utility:

π(k)=arg⁡max⁡s∈S Ak,sroute.(14)\pi(k) = \underset{s \in\mathcal{S}}{\arg\max}\ A^{\mathrm{route}}_{k,s}. \tag*{(14)}

Importantly, this utility estimation relies only on source videos and labels. It serves as a proxy for category-specific skill preference rather than measured target performance. Several categories can share one skill. In the Qwen2.5-VL mapping, counting and anomaly detection select SigLIP-MMR, temporal ordering and causal reasoning select CLIP-Spatial, and timestamp-specific questions select SigLIP-OCR. The full category taxonomy, rewrite prompts, and mapping table are provided in the supplemental material.

Inference with Category-to-Skill Routing

At inference time, AutoSkill removes benchmark templates, answer choices, and output instructions from each query. Let qˉ\bar{q} denote the remaining question content. AutoSkill assigns qˉ\bar{q} to a semantic category and retrieves its associated skill:

c^=Acat(qˉ,T),s^=π(c^).(15)\hat{c} = A_{\mathrm{cat}}(\bar{q}, \mathcal{T}), \qquad\hat{s} = \pi(\hat{c}). \tag*{(15)}

The routed skill then selects a fixed-budget frame set F^=s^(V,q)\hat{\mathcal{F}} = \hat{s}(V,q), from which the frozen MLLM predicts

y^=fθ(F^,q,O).(16)\hat{y} = f_{\theta}(\hat{\mathcal{F}}, q, \mathcal{O}). \tag*{(16)}

This category-level routing matches frame selection to each query’s evidence while executing only one skill.

Experiments

Experimental Setup

Benchmarks. We evaluate on four long-video QA benchmarks covering five splits. They are MLVU dev and test (Zhou et al. 2025), LongVideoBench val (Wu et al. 2024), Video-MME without subtitles (Fu et al. 2025), and LVBench (Wang et al. 2025b). We use their official evaluation protocol and a unified frame budget of B=128B=128.

Implementation details. Routing uses J=5J=5 evaluation splits with M=300M=300 unlabelled question and option pairs from each split. This gives ∣Qtgt∣=1,500|\mathcal{Q}_{\mathrm{tgt}}|=1{,}500 target questions. Only their question and option text is used during routing-table construction. Paired target videos and annotations are not used.

The agent induces Kq=19K_q=19 categories, and builds a rewritten routing set of N=1,500N=1{,}500 samples from the P=3,000P=3{,}000 labelled source pool in Section []. Source answer labels are used to score skills, but no parameters are updated. Backbones are frozen Qwen2.5-VL-7B [1] and Qwen3.5-4B [26] and a direct MCQ prompt. Discovery is run by Sonnet 5 with Auto Research [?], with a GPT-5.5 reviewer who checks proposals before coding. Skills may call auxiliary models of at most 1B parameters, including CLIP, SigLIP, DINOv2, and Grounding-DINO. They may change only frame allocation. Following Table 3, we run T=6T=6 cycles with Mc=5M_c=5 evaluations per cycle. Cycle 1 proposes five skills. Later cycles use three refinements plus two explorations. Failed reviews are dropped before GPU runs. Runtime errors fall back to uniform sampling. Skills that regress against uniform are excluded from S∗S^*. No human filtering or editing is applied. Detailed prompts and the class-skill mapping are in the supplemental material.

VariantAvg. acc. (%)Δ\Delta
Baseline60.0–
Best skill after Cycle 161.0+1.0
Best skill after Cycle 462.7+2.7
Best skill after Cycle 663.0+3.0
Best skill after Cycle 863.0+3.0

Table 3. Skill discovery across successive feedback cycles on the development set. Δ\Delta is relative to uniform sampling. We use T=6T=6 in the main experiments after observing saturation.

Experimental Results

Figure 1 provides direct evidence for heterogeneous acquisition policies: skills exhibit distinct category-wise strengths on LVBench, and their rankings vary across benchmarks. This observation motivates AutoSkill. We next examine whether automatic discovery and routing can translate such complementarity into improved LVQA performance.

Results with Qwen2.5-VL-7B. Table 1 compares AutoSkill with the frozen backbone, individual discovered skills, and existing proprietary and open-source models. Under the unified 128-frame budget, the baseline averages 56.4% across five benchmark splits. All Top-5 skills improve the average by 1.4–2.2%, showing that discovery consistently produces effective strategies. Their strengths are complementary: CLIP-Spatial performs best on both MLVU splits and ties for the best LongVideoBench result among individual skills, while SigLIP-MMR performs best on LVBench and ties on VideoMME. By routing queries among these skills, AutoSkill achieves 69.0%, 51.1%, 61.9%, 65.2%, and 46.8% on MLVU-dev, MLVU-test, LongVideoBench, VideoMME, and LVBench, respectively. It obtains the best overall average among the Qwen2.5-VL variants, improving the baseline by 2.4% and every individual split. These results show that taxonomy-guided routing converts skill complementarity into consistent overall gains.

ModelSizeMLVU devMLVU testLongVideoBench valVideoMMELVBenchAvg/Δ\Delta
Proprietary Models
GPT-4o (Hurst et al. 2024)-64.654.966.771.948.9–
Gemini-2.5-Pro (Comanici et al. 2025)-81.2--84.378.7–
Open-Source VLMs
Video-LLaVA (Lin et al. 2024)7B36.230.737.639.9-–
LLaVA-OneVision (Li et al. 2024a)7B64.747.256.458.3-–
Video-XL (Shu et al. 2025)7B64.945.550.755.5-–
Kangaroo (Liu et al. 2026)8B61.0-54.856.039.4–
LongVU (Shen et al. 2024)7B65.4--60.6-–
LongVA (Zhang et al. 2024)7B56.341.1-52.6-–
LongVILA (Chen et al. 2025b)7B--57.160.1-–
Qwen2.5-VL (Bai et al. 2025)7B64.450.758.764.042.3–
LongVILA-R1 (Chen et al. 2026)7B--57.662.4-–
Video-R1 (Feng et al. 2026)7B65.049.252.061.138.7–
VideoZoomer (Ding et al. 2025)7B68.855.857.765.241.5–
VideoChat-R1† (Li et al. 2025)7B58.7-49.250.423.8–
ReWatch-R1† (Zhang et al. 2025a)7B60.1-53.658.838.5–
Video-Thinker† (Wang et al. 2025a)7B65.2-56.061.9-–
Conan† (Ouyang et al. 2026)7B59.2-54.555.538.2–
LongVT-RFT† (Yang et al. 2026a)7B59.4-54.759.537.9–
SAGE† (Jain et al. 2026)7B49.7-37.444.131.8–
Discovered Skill Toolbox (Ours, only selected Top-5 skills)‡
Qwen2.5-VL‡ (Bai et al. 2025)7B66.047.761.264.642.556.4/–
SigLIP-MMR7B68.351.561.665.346.258.6/+2.2
CLIP-Spatial7B68.852.162.064.045.658.5/+2.1
CLIP-Count7B67.750.662.064.145.257.9/+1.5
CLIP-MMR7B67.651.261.164.544.957.9/+1.5
SigLIP-OCR7B67.450.161.565.344.957.8/+1.4
AutoSkill (ours)‡7B69.051.161.965.246.858.8/+2.4

Table 1. Comparison on long video understanding benchmarks. † marks Qwen2.5-VL-7B post-training methods whose scores are taken from ParaVT (Yang et al. 2026b). ‡ denotes our reimplementation under the 128-frame budget. Each skill is a fixed training-free selector, and AutoSkill routes among them.

Results with Qwen3.5-4B. Table 2 repeats discovery with frozen Qwen3.5-4B, yielding a separate Top-5 toolbox rather than transferring skills from Qwen2.5-VL. The baseline averages 59.2%; individual skills improve it by 0.1–0.8%, while AutoSkill reaches 60.4% (+1.2%), showing that the discovery-and-routing protocol adapts to different models.

Method / SkillMLVULongVBVideoMMELVBenchAvg./Δ\Delta
devtestval
Qwen3.5†^{\dagger}69.955.161.367.342.359.2/–
QMSD-Zoom69.555.862.265.844.659.6/+0.4
Boundary-Evidence69.255.462.164.945.159.3/+0.1
Indexed-Evidence68.955.562.366.146.359.8/+0.6
Clip-Face-Expression69.955.562.366.046.160.0/+0.8
Count-Diversity-Nudge69.955.462.467.042.359.4/+0.2
AutoSkill69.457.562.467.145.460.4/+1.2

Table 2. Results of the Qwen3.5-4B baseline, Top-5 discovered skills, and AutoSkill. †\dagger denotes our reimplementation. LongVB denotes LongVideoBench.

Learned category-to-skill mapping. The routing table assigns the 19 categories based on category-specific accuracy on the routing set. SigLIP-MMR covers categories requiring diverse evidence, such as counting, event identification, and spatial location, whereas CLIP-Spatial handles categories involving co-occurring evidence, including causal reasoning and temporal ordering. SigLIP-OCR addresses action recognition and timestamp-specific questions, while narrative-plot questions retain uniform sampling for temporal coverage.

Ablation Studies

Execution feedback accumulates useful refinements. As shown in Table 3, the best skill after Cycle 1 improves Qwen2.5-VL-7B by 1.0%. Continued diagnosis and refinement raise the gain to 2.7% after Cycle 4 and 3.0% after Cycle 6. Extending the search to Cycle 8 yields no further improvement on the development set. Execution feedback therefore continues to help through early cycles and then saturates, so we set T=6T=6 for the main experiments.

Comparison with training-free frame selection. Table 4(a) compares AutoSkill with recent training-free frame-selection methods under the same Qwen2.5-VL-7B backbone and 32-frame budget. AutoSkill achieves 67.9% on MLVU-dev and 61.5% on LongVideoBench, outperforming mDP3^3 [?] and A.I.R. [?] on both benchmarks. Unlike these fixed frame-selection strategies, AutoSkill automatically discovers a toolbox of complementary evidence-acquisition policies and routes each query to an appropriate skill while retaining a single-skill inference budget.

Efficiency analysis. Table 4(b) compares AutoSkill with RL-based video post-training. AutoSkill requires only about 30 discovery evaluations and router built (roughly 340–45 A100-h), approximately 11×\times less training time than the 440 A100-h reported by Video-R1 [10]. Unlike RL methods, which rely on large-scale SFT and RL training data, AutoSkill uses only labelled source data and unlabelled target queries for skill discovery and routing. Under the matched protocol, it improves the frozen backbone by +2.4% without weight updates, while inference requires only one video-MLLM pass and a lightweight text-only routing call.

Development-set size. Table 4(c) compares discovery on capability-focused sets of 300, 450, and 1,000 samples. Absolute baseline accuracy decreases as the set grows harder, but the best discovered skill improves the baseline by +3.0%, +4.0%, and +3.0%, respectively. Enlarging the development set therefore does not yield a substantially larger discovery gain, while evaluation cost scales roughly with NdevN_{\mathrm{dev}}. We use Ndev=300N_{\mathrm{dev}}=300 in the main experiments.

Router comparison. Table 5 separates the gains from target-query adaptation and category-specific routing. Using the same target-style routing set, the target-adapted global fixed strategy selects one skill for all target queries and improves the uniform baseline by 1.5 points. AutoSkill instead routes each semantic category to its associated skill, reaching 58.8% and providing a further 0.9-point gain under the same supervision and inference budget. This shows that the improvement comes not only from identifying a stronger global skill, but also from adapting frame selection to different query categories. The oracle category-to-skill mapping reaches 60.4%, indicating additional headroom for category-guided routing within the current skill toolbox. AutoSkill also matches the 58.6% accuracy of the post-hoc best fixed skill, which requires evaluating every toolbox skill using target annotations. In contrast, AutoSkill learns the category-to-skill mapping without target videos or answers. The Discovery Cycle Router reaches 57.4%, while majority voting also attains 58.6% but requires five video-MLLM passes per query. AutoSkill achieves comparable performance with a single pass.

Table 5. Comparison of fixed selection, ensembling, and adaptive routing. The target-adapted global fixed strategy selects one skill for all queries using the same target-style routing set as AutoSkill. The post-hoc best fixed skill is selected using target annotations and is included only as an oracle reference.

StrategyTarget labelsAvg. acc.Δ\DeltaPasses
Uniform samplingNo56.4–1
Discovery Cycle RouterNo57.4+1.01
Target-adapted global fixedNo57.9+1.51
Majority votingNo58.6+2.25
Post-hoc best fixed skillYes58.6+2.21
Optimal category mappingYes60.4+4.01
AutoSkill routerNo58.8+2.41

Effect of routing set size. To measure how many samples are needed for routing table construction, we vary the routing-set size while keeping the discovered Top-5 toolbox and the inference configuration fixed. As shown in Table 6, AutoSkill is already effective with a small set. A set of 300 samples gives a +2.0% average gain over the baseline. Increasing the size further improves performance gradually. The gain reaches +2.2% around 400–600 samples and +2.3% at 800–1,200 samples. With 1,500 samples the average reaches +2.4%, and additional samples bring only marginal returns. We also randomly resample and rewrite the 1,500-example routing set five times and rebuild the table independently. The variation across seeds is negligible, which shows that the learned mapping is stable and not tied to one rewrite. We therefore use N=1,500N=1{,}500 in the main experiments as a trade-off between routing reliability and construction cost.

Why voting is not an effective router. Majority voting over the Top-5 skills reaches 58.6%, matching the best fixed skill but remaining below AutoSkill, while requiring five video-MLLM passes per query. For many instances, only one or two skills recover the critical evidence, making the correct answer a minority vote that is overridden by mismatched skills. AutoSkill instead routes each query to its specialised skill, preserving specialisation with only one pass.

Conclusion

We presented AutoSkill, which replaces a universal frame selector with an automatically discovered skill toolbox and taxonomy-guided routing. An LLM agent develops complementary skills on a capability-focused development set and adapts them to target domains without target answers or weight updates. Experiments show that AutoSkill exploits skill complementarity to outperform fixed-skill baselines, providing an effective training-free approach to LVQA.

References

  1. [1]Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2. 5-VL Technical Report. arXiv preprint arXiv:2502.13923.
  2. [2]Chen, J. C.-Y.; Yun, S.; Stengel-Eskin, E.; Chen, T.; and Bansal, M. 2025a. Symbolic mixture-of-experts: Adaptive skill-based routing for heterogeneous reasoning. arXiv preprint arXiv:2503.05641.arxiv.org/abs/2503.05641
  3. [3]Chen, L.; Zaharia, M.; and Zou, J. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.arxiv.org/abs/2305.05176
  4. [4]Chen, Y.; Huang, W.; Shi, B.; Hu, Q.; Ye, H.; Zhu, L.; Liu, Z.; Molchanov, P.; Kautz, J.; Qi, X.; et al. 2026. Scaling rl to long videos. Advances in Neural Information Processing Systems, 38: 172842–172870.
  5. [5]Chen, Y.; Xue, F.; Li, D.; Hu, Q.; Zhu, L.; Li, X.; Fang, Y.; Tang, H.; Yang, S.; Liu, Z.; et al. 2025b. Longvila: Scaling long-context visual language models for long videos. In International Conference on Learning Representations, volume 2025, 18227–18246.arxiv.org/abs/2408.10188
  6. [8]Fedus, W.; Zoph, B.; and Shazeer, N. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120): 1–39.arxiv.org/abs/2101.03961
  7. [9]Fei, X.; Zheng, X.; and Feng, H. 2025. Mcp-zero: Active tool discovery for autonomous llm agents. arXiv preprint arXiv:2506.01056.arxiv.org/abs/2506.01056
  8. [10]Feng, K.; Gong, K.; Li, B.; Guo, Z.; Wang, Y.; Peng, T.; Wu, J.; Zhang, X.; Wang, B.; and Yue, X. 2026. Video-rl: Reinforcing video reasoning in mlms. Advances in Neural Information Processing Systems, 38: 99114–99137.
  9. [11]Fu, C.; Dai, Y.; Luo, Y.; Li, L.; Ren, S.; Zhang, R.; Wang, Z.; Zhou, C.; Shen, Y.; Zhang, M.; et al. 2025. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 24108–24118.
  10. [12]Guo, W.; Chen, Z.; Wang, S.; He, J.; Xu, Y.; Ye, J.; Sun, Y.; and Xiong, H. 2026. Logic-in-frames: Dynamic keyframe search via visual semantic-logical verification for long video understanding. Advances in Neural Information Processing Systems, 38: 124389–124422.arxiv.org/abs/2503.13139
  11. [17]Li, Y.; Chen, X.; Hu, B.; Wang, L.; Shi, H.; and Zhang, M. 2024b. Videovista: A versatile benchmark for video understanding and reasoning. arXiv preprint arXiv:2406.11303.arxiv.org/abs/2406.11303
  12. [20]Ong, I.; Almahairi, A.; Wu, V.; Chiang, W.-L.; Wu, T.; Gonzalez, J. E.; Kadous, M. W.; and Stoica, I. 2024. RouteLlm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665.arxiv.org/abs/2406.18665
  13. [22]Park, J.; Ranasinghe, K.; Kahatapitiya, K.; Ryu, W.; Kim, D.; and Ryoo, M. S. 2026. Too many frames, not all useful: Efficient strategies for long-form video qa. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 3569–3588.
  14. [23]Patil, S. G.; Zhang, T.; Wang, X.; and Gonzalez, J. E. 2024. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37: 126544–126565.
  15. [24]Peng, J.; Song, B.; Li, J.; Li, H.; Zhou, Y.; Ji, R.; and Tian, Y. 2026. QCA: Query-and Content-Aware Keyframe Selection for Long Video Understanding. arXiv preprint arXiv:2607.00983.arxiv.org/abs/2607.00983
  16. [25]Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; et al. 2024. Toolllm: Facilitating large language models to master 16000+ real-world apis. In International Conference on Learning Representations, volume 2024, 9695–9717.
  17. [26]Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents.

Paper details

vlnc:W_dsqr9qdu
Version
v1 · latest
Published
Added
License
CC BY-SA 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents