betaFeedback

WorksComputer Science

Selecting or Solving? What Best-of-N Verification Delivers in Video Reasoning

Abstract

Answering a complex question about a long video is a grounding problem: the answer turns on a few seconds of evidence, and vision-language models remain well below human accuracy at finding them. The standard way to spend extra test-time compute on this kind of video reasoning is best-of-N selection: sample several chain-of-thought answers from a fixed model, then let a second model re-watch the video and pick one. Such pipelines are reported by a single headline number, their margin over a majority vote. That number conflates two effects with opposite deployment consequences: the benefit the second model adds by selecting among candidates, and the benefit it would have supplied by solving the question itself. We separate them, measuring the Selection Increment (SI) – selection accuracy minus the same verifier's accuracy with the candidates withheld – alongside the margin over self-consistency, across five long-video benchmarks. The Selection Increment is largest for the weakest verifiers: a verifier that solves poorly on its own leaves the candidates the most room to help. Beating the majority vote follows the opposite profile: where the vote already sits near its ceiling no verifier we test improves on it; below that ceiling only the strongest do. On Video-Holmes, the two come apart on a single pool of candidates – a weak verifier gains +19.3 points from the candidates yet lands 7.1 points below the vote, while the frontier verifier gains the least and beats the vote. The weakest verifier also shows a failure mode: on EgoSchema it picks by where an option sits in the list rather than by what the video shows. Read as a merit score, SI puts a verifier that fails to beat a majority vote at the top of its ranking; read as an audit, it says which of the two effects a headline number delivers. Code is available at https://github.com/facebookresearch/best-n-selection-video-reasoning.

Introduction

A question about a long video is rarely answered by a summary. What decides it is a few seconds of evidence buried somewhere in the footage, and a system is only right for the right reason—only faithful to the video—if its conclusion stays tied to that evidence. Vision–language models (VLMs) now reason over video rather than describe it, yet on benchmarks built to demand exactly this grounding, frontier models remain well below human accuracy [6, 35]. Video reasoning is therefore a revealing setting for asking how additional inference-time compute should be spent: a larger model is not the only lever. Our focus is the long-video form of the problem, long-video question answering (long-video QA): complex questions, answered with chain-of-thought, whose evidence is spread through long footage.

In video, that compute is most often spent gathering evidence. Agentic loops retrieve, caption, temporally ground, or zoom on demand ([44, 37, 18, 43]), spending extra passes to surface the few relevant seconds. We study a less explored alternative that uses the same compute budget on deliberation: sample several independent answers from a fixed model and choose among them—best-of-NN selection, the recipe behind much of the gain from test-time scaling in text reasoning ([7, 13, 47]).

Sampling alone shows how much such selection could recover. On Video-Holmes, eight independent chains of thought from Qwen3.5-27B contain a correct answer 78.8% of the time, while self-consistency—majority voting over those chains—recovers only 58.9%. Close to twenty points are already present in the model’s own output distribution and discarded by the vote, recoverable without retraining or a larger model. The natural way to claim them is a verifier: a second VLM that re-watches the video and picks one candidate (Figure 1).

Best-of-$N$ verifier pipeline

Figure 1. What a best-of-NN verifier pipeline measures. A base generator (Qwen3.5-27B) samples N=8N=8 chain-of-thought rollouts per question, each ending in a candidate letter. Three rules score that same pool: majority voting; the verifier selecting among the candidates after re-watching KK frames; and the same verifier with the candidates withheld. Their differences define our two axes, Δmaj\Delta_{\mathrm{maj}} and SI.

Gains reported for this recipe are read as evidence that the verifier selects well. They cannot be. A verifier that re-watches the video is unavoidably also a second solver, so a single margin over the vote conflates the benefit added by selecting among candidates with the benefit the verifier would have supplied by solving the question itself. Only the first adds something the base model could not produce; the second means the pipeline is a computationally expensive way to run a better model, and the right deployment differs in each case. Hence our question: when a second model appears to select well, is it selecting or merely solving—and which does a reported gain deliver?

Answering it cleanly requires two controls, because the naive comparison flatters the verifier twice over. First, a model asked to pick can ignore the candidates and answer directly, and when that answer is correct, standard scoring still credits a successful selection; we therefore also score under a strict policy that only credits picks drawn from the candidates. Second, a stronger verifier can beat self-consistency simply by answering well on its own. We therefore measure the Selection Increment (SI): the verifier’s accuracy when selecting among the candidates, minus its accuracy on the same questions with the candidates withheld. SI is the verifier’s contribution net of its own solving ability—exactly what a headline number leaves unresolved.

Findings and contributions. With both controls in place, each pipeline is scored on two questions—does the verifier add benefit beyond its own solving, and does it beat the majority vote?—and our central finding is that the answers are dissociable: neither implies the other.

  • An audit protocol. The two controls above—strict candidates-only scoring and the Selection Increment—separate selecting from solving; both are computationally cheap, and neither requires retraining (§ 3).

  • Two axes, two levers. Along a ladder of five verifiers over a fixed candidate pool, SI falls monotonically as the verifier’s own accuracy rises: the weakest verifier gains the most from the candidates. Beating the vote is governed instead by two conditions, checked in order—whether the vote already sits near its ceiling, and only then whether the verifier is strong enough. An intuitive “headroom” rule—that selection helps most where the pool leaves the most room above the vote—finds no support (§ 4).

  • Two failure modes of scaled selection. Verifiers below a capability threshold drift toward early answer positions rather than visual evidence, and letting the generator gather evidence on demand does not raise the ceiling that selection works against (§ 5, Appendix § 12).

  • What to report, and what to deploy. A cross-model verifier does beat self-consistency (+7.94 points on TempCompass); our strongest Video-Holmes margin, however, does not survive the strict scoring after multiplicity correction, so we do not claim it. The recommendation: report both axes, and the scoring policy behind each (§ 6).

Related Work

Best-of-NN verification and self-consistency. Test-time compute can substitute for parameters in text reasoning, through long chains of thought and sample-and-aggregate schemes [4, 20, 38, 8]. Self-consistency, the majority vote over sampled chains [38], is the strongest training-free member of that family and our point of comparison throughout: any verifier must beat it to justify the second model’s computational cost. Selecting one candidate with a verifier instead is equally well studied: outcome- and process-reward verifiers [7, 13], the scaling of best-of-NN with sample count [4], LLM-as-a-judge evaluation [47], and generative verifiers that reason before scoring [45]. Two findings from that literature shape our design: multi-model juries can outperform a single large judge [32], and judges prefer outputs from their own model family (Panickssery et al., 2024). Nearer our setting, Saluru [28] route between Gemini and Claude on disagreement for implicit-reasoning video QA, and concurrent work aligns multi-agent consensus to region-level visual evidence for image VQA [40]. What this literature reports is the verifier’s gain over majority voting or pass@1, a quantity that conflates selection benefit with the verifier’s own task skill. Our verifier is a judge applied listwise, making one holistic NN-way pick; we work on long video, ground selection in re-watched frames, and ask when selection adds benefit at all.

Long video: gathering evidence, and auditing whether it helps. In video the dominant inference-time recipe gathers evidence with agentic loops that retrieve, caption, temporally ground, or zoom, usually on small or medium splits [37, 44, 18, 43, 11]; Wang et al. [34] relate when such tool use helps to task difficulty. That line enriches what the model sees; the pool of answers a verifier must choose among goes unmeasured, and Appendix D finds it unmoved. A parallel line makes the input frames query-conditioned rather than uniform, ranking the most relevant frames per question at test time (Chen et al., 2026; Wang et al., 2026b; Zou et al., 2026; He et al., 2026); that axis is orthogonal to ours (it acts on the input, ours over sampled trajectories), so query-conditioned frames could feed both our generator and our verifier. We use uniform sampling throughout as a control. Running alongside both lines is a growing audit literature asking whether reported gains reflect added capability or format learning [9, 19] and showing that high accuracy is often attainable without genuine visual grounding [12]; closest to our task, Alavi [2] find implicit video QA largely perception-bound, with reasoning augmentation neutral to harmful. Our contribution belongs to this audit tradition rather than the method-proposing one: we take a recipe that is already standard and ask what its headline number is actually measuring.

Measurement Protocol

Problem setup. For a question qq with options Oq\mathcal{O}_q, gold answer a⋆a^{\star}, and video VV, a base generator π\pi samples NN independent zero-shot chain-of-thought trajectories c1,…,cN∼π(⋅∣V,q,Oq)c_1,\ldots,c_N \sim\pi(\cdot\mid V,q,\mathcal{O}_q), each contributing a candidate letter yi=letter⁡(ci)∈Oqy_i=\operatorname{letter}(c_i)\in\mathcal{O}_q; let C={y1,…,yN}\mathcal{C}=\{y_1,\ldots,y_N\} be the candidate multiset. A selection rule maps C\mathcal{C} to one answer. Our canonical setting is Qwen3.5-27B [25] reading 64 uniformly sampled frames (32 on EgoSchema) at temperature T=0.8T=0.8, with N=8N=8 for headline results and a sweep N∈{1,2,4,8,16}N\in\{1,2,4,8,16\} on Video-Holmes. The verifier J\mathcal{J} is a second VLM that re-watches its own KK uniformly sampled frames VKV_K and makes one holistic NN-way pick, y^=J(VK,q,C)\hat{y}=\mathcal{J}(V_K,q,\mathcal{C}) (Figure 1). The same verifier can also be run without candidates, answering from its own frames alone, y^=J(VK,q,Oq)\hat{y}=\mathcal{J}(V_K,q,\mathcal{O}_q); this is its standalone solver, and it is what makes the two axes separately measurable.

Two axes, and the headroom they live in. We compare the verifier against pass@1 (the expected accuracy of one chain drawn uniformly from CC), self-consistency y^=mode⁡(C)\hat{y} = \operatorname{mode}(C) (Wang et al. [38]), and the best-of-NN oracle (correct iff a⋆∈Ca^\star\in C), which upper-bounds what any selection rule can recover.2 Writing acc⁡(s)=Pr⁡[y^s=a⋆]\operatorname{acc}(s) = \Pr[\hat{y}_s = a^\star],

Δmaj=acc⁡(sel)−acc⁡(maj),SI=acc⁡(sel)−acc⁡(standalone),h=acc⁡(oracle)−acc⁡(maj).\Delta_{\mathrm{maj}} = \operatorname{acc}(\mathrm{sel}) - \operatorname{acc}(\mathrm{maj}), \qquad\mathrm{SI} = \operatorname{acc}(\mathrm{sel}) - \operatorname{acc}(\mathrm{standalone}), \qquad h = \operatorname{acc}(\mathrm{oracle}) - \operatorname{acc}(\mathrm{maj}).

Δmaj\Delta_{\mathrm{maj}} asks whether selection is worth its computational cost against the computationally cheap baseline; the Selection Increment SI asks whether the candidates help the verifier beyond its own solving; and the headroom hh bounds what any rule could recover. The three are distinct, and the distinction is the paper’s subject: a positive SI means the candidates helped, but a weak standalone solver inflates it (§ 4), so we report SI and Δmaj\Delta_{\mathrm{maj}} together throughout and never one alone.

Verifiers and benchmarks. To trace how selection depends on verifier strength we vary the verifier along model family and capability (Table 1), from a weak cross-family model, Phi-3.5-V (Abdin et al. [1]), through two mid-scale open ones, InternVL3-8B and -38B (Zhu et al. (2025)), and a same-family sibling 15×15\times the base’s size, Qwen3.5-397B (Qwen Team [26]), up to two frontier cross-model verifiers, a reasoning VLM from another family (Muse Spark 1.1 (Meta AI [17]), hereafter Muse) and the stronger closed GPT-5.6 Sol, a later release in the GPT-5 family (OpenAI [22]), over compute budgets K∈{16,24,32,48,64,96}K \in\{16, 24, 32, 48, 64, 96\}. Every headline result fixes Qwen3.5-27B as generator and varies the verifier; two base/verifier swaps that also vary the base are reported in § 4. We measure on five video multiple-choice QA benchmarks: our primary reasoning anchor Video-Holmes (Cheng et al. [6]) (1,837 six-way reasoning MCQs over 270 videos), with TempCompass (Liu et al. [14]), MLVU-Test (Zhou et al. [48]), VRBench (Yu et al. [42]), and EgoSchema (Mangalam et al. [16]). They are chosen so self-consistency accuracy spans a wide range, from about 59% (Video-Holmes) to about 91% (VRBench), so selection is tested in easy and hard regimes alike. All deltas are matched-sample, computed on the intersection of items with valid predictions from both compared methods; split sizes and per-cell denominators are in Appendix § 11.

RoleModel(s)Frames
Base generator (N=8N=8 CoT, T=0.8T=0.8)Qwen3.5-27B64 (32 on EgoSchema)
Verifier, cross-model (headline)Muse Spark 1.148
Verifier, cross-model (strong)GPT-5.6 Sol32
Verifier, same-familyQwen3.5-27B / -397B48 / 48, 96
Verifier, weak cross-familyPhi-3.5-V16, 24, 48, 64
Verifier, mid cross-familyInternVL3-8B / -38B48 / 24, 48

Table 1. Base and verifier roles. All headline best-of-NN results fix Qwen3.5-27B as the base generator and vary the verifier; two base-varying 2×22 \times2s (Video-Holmes, TempCompass; § 4) also vary the base. The standalone solver is the same verifier answering directly from its own KK frames.

Scoring: selecting versus solving. A verifier asked to select can solve instead: it may emit a letter outside the candidate set, and when that out-of-candidate (OOC) letter is correct, naive scoring credits a successful selection. We therefore score every cell under three policies: Solver counts OOC hits (the default in prior work), Selector counts a pick only if it is correct and drawn from CC, and Snap maps an OOC pick to its nearest candidate. Headline numbers use Solver so they remain comparable to published best-of-NN results; we report Selector as a labeled ablation and flag every disagreement, since the gap between the two is exactly the leakage a headline number over-credits. The standalone leg of SI is policy-invariant, because the solver never sees CC. Per-cell OOC rates, which run from under a percent to roughly 13% for a weak verifier, are in Appendix § 11.

Statistics. Every headline delta is matched-sample and tested with an exact two-sided paired McNemar test on the discordant pairs. We report per-comparison significance and flag non-significant deltas (n.s.) throughout, applying Holm–Bonferroni correction to the flagship beats-majority family where it is named and treating isolated significant results in the broader uncorrected grid accordingly. Permutation and Fisher tests in § 5 are one-sided against a predicted direction, a deliberate seam with the two-sided McNemar tests that we state rather than smooth over (Appendix § 11).

Two Axes, Two Levers

On our primary anchor, only a strong verifier beats self-consistency. Video-Holmes gives the cleanest reading, because there is a great deal to win: sampling N=8N=8 chains from the 27B base yields a self-consistency accuracy of 58.9% against a best-of-NN oracle of 78.8%, a ≈ 20-point selectable gap that widens with NN (at N=16N=16 the oracle reaches 82.5% while the vote stays flat at 59.0%; see Table 2). The strongest verifier we run, GPT-5.6 Sol, selects at 62.0%—+6.4 over pass@1 and +3.1 over self-consistency (exact McNemar p=7.0×10−4p=7.0 \times10^{-4})—while still leaving ≈ 17 points of oracle headroom on the table. The four weaker verifiers all fall below self-consistency. Selection helps here, but only with a verifier strong enough to be worth its own computational / inference cost: the paper’s thesis in miniature.3

Table 2: Video-Holmes (1,837 six-way MCQs), N=8N=8. Best verifier: GPT-5.6 Sol.

Ruleacc. (%)
pass@155.6
self-consistency58.9
best verifier62.0
oracle@878.8

Table 2.

Across the suite, the two axes come apart. Table 3 reports both axes for the two deployed cross-model verifiers on full splits. The margin over self-consistency shrinks as the candidate pool saturates: on TempCompass (n=1,574n=1,574) the cross-model verifier reaches 87.2% against 79.2%, a gain of 7.94 points and its largest anywhere; on MLVU-Test it only ties (+0.81, n.s.); and on VRBench, whose vote already reaches 91%, it loses (−3.26, p=5.6×10−6p=5.6 \times10^{-6}). The Selection Increment moves independently of that margin, and sometimes against it: SI is positive on MLVU-Test (+5.68) and VRBench (+3.67) precisely where the verifier fails to beat the vote, and negative on TempCompass (−1.08) where the verifier wins by almost eight points. There, Muse scores higher answering directly than selecting. A reader given only the margin would conclude selection works on TempCompass and not on VRBench; given only SI, the opposite—which is why we report the pair.

Table 3: The two axes come apart on full splits. Solver-policy (prior-work convention) accuracies for the Muse holistic-48 and GPT-5.6 Sol verifiers; majority is the shared 27B majority@8 baseline. The headroom h is a per-benchmark property of the candidate pool and predicts neither delta. VRBench cells use the first 1,000 questions of its split (118 of 960 videos); deltas are McNemar-significant at α=0.05\alpha=0.05 unless marked n.s.

BenchmarknnMajority (%)Selection (%)Δmaj\Delta_{\mathrm{maj}} (pt)SI (pt)hh (pt)
TempCompass (Muse)1,57479.287.2+7.94−1.08≈ 11
TempCompass (GPT-5.6 Sol)1,57079.188.5+9.43+1.15
MLVU-Test (Muse)49673.274.0+0.81 n.s.+5.68≈ 14
MLVU-Test (GPT-5.6 Sol)50272.970.9−1.99 n.s.+11.16
VRBench (Muse)98391.087.8−3.26+3.67≈ 6
VRBench (GPT-5.6 Sol)98990.989.3−1.62+3.96

Table 3.

The intuitive “headroom” rule finds no support. The natural predictor of selection benefit is the room the sampler leaves, h = oracle − majority. Individual benchmarks violate any monotone version of the rule in both directions: on TempCompass, at low headroom (≈ 11 points), the verifier beats the vote by +9.4 (p<10−23p<10^{-23}), whereas on MLVU-Test, at higher headroom (≈ 14 points), it underperforms the vote (−2.0, n.s.). Across a compressed ≈ 6–14-point range we claim no reliable correlation in either direction, only that there is no evidence for the rule. Headroom is not the level of the majority baseline: one is a gap, the other a position. Only the second carries signal below.

Lever one: the increment tracks standalone accuracy, inversely. Holding the candidate pool fixed and sweeping verifiers of increasing standalone accuracy drives SI monotonically toward zero. A five-verifier ladder on the Video-Holmes Qwen-base pool makes this explicit: SI falls from +28.7 (Qwen-27B self-verifier, standalone 27.7%) through +19.3 (Phi-3.5, 32.5%), +17.6 (InternVL3-8B, 39.7%) and +10.9 (InternVL3-38B, 47.7%) to +7.9 (GPT-5.6 Sol, 54.1%), and an MLVU-Test ladder reproduces the ordering (+26.3, +21.6, +20.8, +10.8 as standalone rises 38.6→46.7→51.1→62.0%38.6 \to46.7 \to51.1 \to62.0\%.4 A large increment therefore marks a weak solver that can nonetheless select. SI measures how much the candidates add to a given verifier, not how good that verifier is. The one swap that appears to run the other way confirms the rule: replacing Muse with GPT-5.6 Sol raises SI on every deployed benchmark, not because GPT selects better but because it under-solves those benchmarks by more, leaving the candidates more room (Appendix § 13).

The two axes dissociate inside a single candidate pool. On the Video-Holmes Qwen-base pool, base difficulty is held fixed and the verifier alone varies, which yields the paper’s sharpest contrast (Table 4). The weak Phi-3.5 verifier selects far better than it solves—SI +19.3 from a 32.5% standalone—yet its selected accuracy lands 7.1 points below the majority vote (51.8% vs. 58.9%; exact McNemar p<10−13p<10^{-13}), while the strong Muse verifier adds almost nothing over its own solving (+2.5) and significantly beats the vote (+2.8; p=2.3×10−3p=2.3 \times10^{-3}). Neither axis predicts the other, on one and the same pool. Three controls place the weak verifier’s deficit in its capability rather than in this pool or its protocol: it reproduces on MLVU-Test against a different solver pool and a different baseline (8.0 points below the vote, exact McNemar p=2.4×10−6p=2.4 \times10^{-6}); re-running the verifier at the generator’s own 64 frames leaves both deficits numerically larger rather than closing them, which falsifies the frame-handicap account without establishing what the real mechanism is; and swapping only verifier strength at a common 24-frame budget moves the outcome from that 8.0-point deficit to statistical parity with the vote (InternVL3-38B, Δmaj=−0.2\Delta_{\mathrm{maj}}=-0.2, exact McNemar p=1.000p=1.000). The pathology is therefore not confounded with frame budget, model family, or parameter count (Appendix § 13).5

Base poolQwen-27B verifier (weak, std. 27.7)Muse verifier (strong, std. 59.4)
Panel A: Selection Increment (SI = selection − standalone)
Qwen-base (split n=1,837n=1{,}837)+28.7+2.5
Muse-base (split n=1,837n=1{,}837)+31.2+0.9
Panel B: gain over self-consistency (Δmaj\Delta_{\mathrm{maj}} = selection − maj@8)
Qwen-base+0.05 (58.95 vs. 58.90)+2.8 (61.9 vs. 59.1)
Muse-base+0.00 (59.55 vs. 59.55)+0.8 (60.3 vs. 59.5)

Table 4. Base × verifier on Video-Holmes (full 2 × 2). Selection Increment (Panel A) and gain over self-consistency (Panel B, selection vs. majority@8), in points, for a weak and a strong verifier (Qwen-27B, standalone 27.7; Muse, 59.4) over a Qwen- and a Muse-generated candidate pool (each split n=1,837n=1{,}837; the Muse-verifier cells use matched n=1,808/1,807n=1{,}808/1{,}807). SI is a column effect; beating self-consistency is set by the verifier. Cell-level tests and provenance: Appendix § 13.

Lever two: the saturation gate first, then the strength gate. Ordering every base × verifier cell by the majority accuracy of its pool separates two conditions. Above roughly 87% the saturation gate decides and strength is irrelevant: on the Muse-base TempCompass pool (87.5%) and on VRBench (≈91%\approx91\%) no verifier beats the vote, and on VRBench both deployed verifiers significantly lose: frontier GPT-5.6 Sol (−1.62) no less than Muse (−3.26). Below the ceiling the strength gate decides: the two strongest verifiers win on the Qwen-base Video-Holmes pool (58.9%; +2.8 and +3.1) and every weaker one fails to. On the Qwen-base TempCompass pool (79.2%) the weak self-verifier only draws level (+0.9, exact McNemar p=0.082p=0.082)—a null with a mechanism: it agrees with the plurality on 95.1% of rows, rubber-stamping the modal candidate rather than exercising independent judgment. Because the TempCompass 2×22 \times2 holds the task fixed and varies the pool, it isolates saturation the way Video-Holmes isolates strength: the Muse verifier’s +7.94 win collapses to a null (−0.4) as majority rises from 79.2 to 87.5 (Table 6). The two gates are necessary rather than jointly sufficient: MLVU-Test sits in the permissive range (≈ 73%) with a capable verifier, and still neither deployed verifier beats the vote (+0.81 and −1.99, both n.s.)—a null we cannot presently explain and do not paper over. And the ceiling itself rests on two near-saturated pools, so we report an observed boundary rather than a calibrated threshold. Figure 2 places every deployed pair in these two coordinates.

Selection Increment versus gain over self-consistency

Figure 2. Selection Increment vs. gain over self-consistency. Each point is a (benchmark, verifier) pair; lines link the two verifiers on one benchmark. SI > 0 and beating self-consistency are dissociable. Solver policy; VRBench: first 1,000 questions (118 of 960 videos).

Base poolVerifierSIΔmaj\Delta_{\mathrm{maj}}
Qwen-base (n≈1,579n\approx1{,}579, maj 79.2)Qwen-27B+8.7+0.9
Qwen-baseMuse−1.1+7.9
Muse-base (n≈1,000n\approx1{,}000, maj 87.5)Qwen-27B+18.5−0.3
Muse-baseMuse+0.5−0.4

Table 6. Base × verifier on TempCompass (full 2×22 \times2). Selection Increment and gain over self-consistency (Δmaj\Delta_{\mathrm{maj}}), in points, for a weak (Qwen-27B) and a strong (Muse) verifier over each of two candidate pools. SI is a column effect (large for the weak verifier, near zero for the strong one on both bases); beating self-consistency is a row effect (attainable on the harder Qwen-base pool, out of reach on the near-saturated Muse-base pool). The two axes are governed by different factors. The three small Δmaj\Delta_{\mathrm{maj}} cells (+0.9, −0.3, −0.4) are effectively at parity; the row effect is clearest in the Muse column (+7.9 → −0.4). Provenance. SI is reported with both of its legs—selection and standalone—at the original 1,024-token generation cap, since re-measuring one leg alone would mix compute budgets within a single difference. Each SI cell is differenced on its verifier’s matched non-empty support. The two Qwen-27B Δmaj\Delta_{\mathrm{maj}} cells are re-measured at 8,192 tokens with a corrected answer extractor; the Muse cells never bound the cap but retain the original extractor (Appendix § 10). In both base-varying 2×22 \times2s the base axis moves the generator’s frame budget together with its identity—the Qwen-base pools read 64 frames, the Muse-base pools 32—so a row effect is a property of each pool as produced rather than of model identity alone.

Gathering more evidence does not raise the ceiling. One reading of these limits is that selection looks weak only because the sampled candidates are impoverished, and that letting the model gather more evidence would lift the ceiling a verifier works against. It does not. Augmenting candidate generation with an on-demand agentic loop that may zoom, crop, retrieve frames, or ground objects leaves the best-of-NN oracle on EgoSchema unchanged (Δoracle=0.000\Delta_{\mathrm{oracle}=0.000} at n=500n=500), and giving the same loop to a strong cross-model verifier leaves its accuracy within noise of its zero-shot self. Selection can only surface a correct answer the sampler already produced, which is what directs this study at selection rather than gathering (Appendix D).

A Failure Mode of Under-Capable Verifiers: Answer-Position Bias

Answer-position bias has as a threshold: below a capability floor a verifier drifts toward earlier options and away from the last; above that floor the early drift is absent. A verifier choosing among lettered options can appear to discriminate while favoring an early slot whenever its grounding is too weak to tell the candidates apart, in which case part of what we credit to selection is an artifact of letter position rather than of visual evidence. We quantify this two ways. A-lift is the rate at which a verifier picks “A” minus the rate at which “A” is gold, in points; primacy is the mean normalized position of the gold answer minus that of the pick, so a positive value means the verifier systematically chooses earlier options than gold warrants. Primacy captures the full early-to-late slope rather than one letter, and the two agree closely across our 22 verifier–benchmark cells (Spearman +0.942 with

Bias falls with standalone accuracy, but discontinuously. The rank correlation between accuracy and A-lift is ρ=−0.954\rho=-0.954 across the ten configurations (exact permutation p=4.5×10−5p=4.5 \times10^{-5} over all 10! label assignments), and sweeping the 27B self-verifier’s two ranks over the whole range consistent with a corrected, higher standalone accuracy leaves ρ\rho at −0.73 or stronger. The under-capable end carries that ordering entirely: dropping the weak Phi-3.5-Vision verifier moves the pooled within-benchmark test from p=0.013p=0.013 to p=0.34p=0.34, and among capable verifiers alone there is no accuracy-to-bias trend. Of the 22 cells, nine are individually significant by an exact McNemar test and five survive Holm–Bonferroni correction; all four positive survivors belong to Phi-3.5-Vision, while the fifth is the capable Qwen3.5-27B self-verifier sitting below gold on Video-Holmes (−2.8 points, p=0.0020p=0.0020, Solver policy)—a signed control rather than a bias of its own. The honest description is a threshold.

EgoSchema is the sharpest test, because it carries a biased arm and a control arm on the same questions. It is five-way with near-uniform gold whose mode is E (gold-“A” 20.2%, gold-“E” 23.4%, n=500n=500), so an early-option default lowers accuracy: any bias at reveal is a liability, not free accuracy. Only the genuinely weak Phi-3.5-Vision shows a clear default: under the Solver policy, A-lift +4.4 at 16 frames and +8.0 at 48 (Table 7), with the modal answer E correspondingly under-selected (13.4% picked vs. 23.4% gold). Every capable verifier stays within 1.4 points of gold. The failure mode is a graded positional slope rather than a first-option collapse, and recency avoidance—the deficit at the last option—persists at every capability level, including the frontier, where the first-option excess is absent (Appendix E).

VerifierClassA-lift (pp)
Phi-3.5-Vision @16fweak, cross-family+4.4
Phi-3.5-Vision @48fweak, cross-family+8.0
Muse (holistic-48)strong, cross-model+1.4
InternVL3-8B @48fmid, cross-family+0.6
InternVL3-38B @48fmid, cross-family+1.4
Qwen3.5-27B (self-verifier) @48fsame-family (self)+0.6
Qwen3.5-397B @48fsame-family (sibling)+0.2
Qwen3.5-397B @96fsame-family (sibling)+0.8

Table 7. Answer-position bias on EgoSchema is a weak-verifier property. Per-verifier A-lift (pick-“A” rate minus gold-“A” rate, in points, under the Solver scoring policy; gold-“A” = 20.2%, gold mode E, n=500n=500), ordered weak to strong. Only the weak cross-family verifier Phi-3.5-Vision lifts “A” above gold, at both of its compute budgets; every capable verifier stays within 1.4 points of gold.

The significant bias is confined to a degenerate response mode. Instrumenting the weak verifier’s raw output localizes the preference. Its responses fall into two disjoint modes, separated perfectly by output length: a marker mode returning the requested Answer: X form, and a terse mode returning a bare letter and nothing else. Neither truncates: every response terminates normally, and a bare letter is a complete, unambiguous verdict. What follows is therefore the verifier’s behavior rather than an artifact of how we parse it. The significant A-lift lives entirely in the terse mode: across ten verifier–benchmark arms, marker-mode responses show no significant A-lift anywhere (10 of 10), while terse-mode responses favor the first option in every arm where that mode occurs (8 of 8, +6.7 to +32.4 points, largest p=2.5×10−3p=2.5 \times10^{-3}; all eight survive Holm–Bonferroni over the full family of eighteen mode tests). Nor is it accuracy in disguise. On EgoSchema at 16 frames the two modes are equally accurate to within 0.11 points and the terse mode is still biased; on MLVU-Test and Video-Holmes the terse mode is the less accurate one, by 6.8 to 16.7 points, which cannot manufacture a preference for the first option either. The default is directional rather than a by-product of solving worse. Because that partition sits within one verifier at a fixed model and frame budget, what varies across it is the mode and not the model. A capable verifier that never enters the terse mode adds a further control—InternVL3-38B emits the marker on all 502 items of both arms and reaches significance in neither—but mode and strength are collinear there, so it rules out extraction failure and item difficulty rather than verifier strength (Appendix § 13). The mode structure revises the scope of the mechanism, not the deficit it explains: all twelve Δmaj\Delta_{\mathrm{maj}} cells are negative and significant under both scored fields, so a weak verifier falls below the vote whether or not it is in the mode that biases it.

Verifying versus Solving: What to Report, What to Deploy

Where the Selection Increment is not positive, selection provides only accuracy the verifier already gets by solving—at a higher computational cost. On TempCompass with the Muse verifier (SI=−1.08\mathrm{SI}=-1.08) the same verifier used for standalone solving is more accurate than the pipeline, and it avoids the computational cost of reading the 27B candidates, so verification there is the wrong way to spend the verifier’s compute. Where SI>0\mathrm{SI} > 0 (MLVU-Test with Muse, or TempCompass with the stronger GPT-5.6 Sol), selection does beat standalone solving and the pipeline earns its extra computational cost. A benchmark-matched token accounting is left to future work, so we make no blanket compute-efficiency claim for verification; Appendix § 14 sets out the reasoning in passes rather than tokens.

The sign of the increment is also what a reader needs in order to interpret someone else’s headline. A reported gain over majority voting is silent on which of the two systems of § 4 produced it, and our suite contains both: a weak verifier that selects far better than it solves and still falls below the vote, and a strong verifier that beats the vote while adding almost nothing to its own solving. Read as a merit score, SI’s top-ranked verifier is one that fails to beat a vote; read as an audit, it says which of the two effects a reported gain delivers, and that is how we intend to use it. The reading is deflationary—on our anchor, the verifier worth deploying is the one that adds the least selection benefit.

Both axes are computationally cheap enough to check before committing inference compute. On a labeled development slice, majority accuracy over the sampled candidates does not require additional compute once the candidates exist, so the saturation gate can be evaluated before any verifier is called; the increment requires one extra pass, the verifier answering with the candidates withheld. That makes the failure predictable in advance, and yields a three-step allocation policy. First, skip verification where the vote already sits near the observed ceiling: no verifier in our study beat it there. Second, below that ceiling, pair the base with a verifier clearly stronger than itself and drawn from a different model family, so that its errors are decorrelated from the base’s. Third, where even that verifier does not exceed the vote, do not allocate additional compute for selection: deploy whichever of the two computationally cheaper alternatives the protocol has already measured scores higher, the majority vote or the verifier answering alone. On TempCompass with Muse the standalone verifier wins that comparison; on MLVU-Test the vote does.

Conclusion

When a verifier beats the majority vote, the margin alone does not say why: the verifier may be selecting well, or simply solving the question better than the base model. Two cheap controls—strict candidates-only scoring and the Selection Increment—tell these apart, and they show that the two things a pipeline can do are separable. Across five verifiers on a fixed candidate pool, the weakest verifier gained the most from seeing candidates, while beating the vote depended on how close the vote already was to its ceiling and, only then, on verifier strength. Weak verifiers also latched onto answer position instead of the video, and letting the generator gather more evidence did not raise the ceiling. Reported on both axes, genuine selection gains do appear—a cross-verifier adds +7.94 points over self-consistency on TempCompass—and we see this stricter accounting as the way to find more of them.

Acknowledgments and Disclosure of Funding

This project is funded by Meta AI mentorship program. We appreciate the earlier discussions with Rongjie Huang and Jason Zheng, the great help Hongyu Gong provided for open-sourcing the code, and the help from Harris Warren and other members of C&L team in FAIR for getting this project approved.

Limitations

(1) Three benchmarks carry both deployed verifiers, over a compressed ≈\approx 6–14-point headroom range, so our headroom result is no evidence for the rule rather than a correlation in either direction. (2) Both strong verifiers are closed API models, though an open verifier does beat self-consistency on EgoSchema (Appendix § 13). (3) We select over final answer letters, not intermediate reasoning steps, and over uniformly sampled rather than query-conditioned frames ([5]). (4) Every benchmark we use labels its options alphabetically, so an option’s ordinal position and its letter rank are collinear; our position-bias statistics establish the drift but, as we report them, cannot separate the two (Appendix § 13). No caveat here touches the oracle-level tool null of Appendix § 12.

Implementation and scoring integrity

Beyond the models under study, the experimental pipeline itself (code, job orchestration, and result analysis) was implemented with the assistance of LLM-based autonomous agents under author supervision. To guard against errors from this process, every headline number in the paper was recomputed by two independently implemented scorers, which agree to within 0.1 points under a fixed convention that excludes empty trajectories from the vote. Where the harness retained the verifier’s full response the recomputation runs from it; the earlier verifier runs recorded the extracted answer and the gold label but not the response behind them, so for those legs the answer-extraction step cannot be re-audited after the fact. Where a quantity was re-measured at a larger generation compute budget, the re-run reproduces the original run’s per-row answers on 99.4–99.9% of rows when its output is truncated back to the original compute budget; the same-family self-verifier on MLVU-Test, at 95.2%, is the lone exception. On the positive-control verifier every disagreeing row is one whose response was shorter than the original cap, which attributes that residual to run-to-run generation nondeterminism rather than to the cap. Prediction and pick files were also checked directly rather than through logged summaries: gold labels were cross-verified, duplicates removed, and empty or malformed outputs handled explicitly.

Protocol and denominators

This appendix records in full the conventions that § 3 states in summary: the majority-vote tie-break, the three scoring policies and their out-of-candidate rates, the split sizes, and the per-cell denominators behind the anchor table. Every figure here is as measured.

Splits. Unless noted, results use each benchmark’s full test split (≈\approx 500–1,850 items). VRBench is the one exception: its cross-model (Muse / GPT-5.6 Sol) cells use a 1,000-question subset, while its Phi-3.5 and InternVL3 cells use a larger ≈\approx 1,500-item split. We flag the subset at each VRBench table and figure, and never compare across the two VRBench splits—every VRBench delta is computed within one of them.

Tie-breaking. Ties in the majority vote are broken by first appearance—the option whose earliest supporting chain has the lowest index—which we verified directly against the shipped implementation on all 1,837 Video-Holmes ballots at N=16N=16: the shipped choice matches first appearance on every row, including all 49 rows on which the vote is tied and the 26 on which an alphabetical rule would have chosen differently, so an alphabetical tie-break is falsified outright. The rule is also effectively letter-neutral. Against a letter-blind tie-break—one that resolves a tie uniformly at random among the tied options—it never moves the A-share by more than 0.6 points at any NN on any benchmark in our suite; the largest departure we measure is 0.57 points, on EgoSchema at N=8N=8.

Sensitivity of the majority deficit to the tie-break rule. Because the tie-break is an artifact of sampling order rather than a principled rule, we bounded its influence directly. Ties are not rare—they occur on 3.98% of MLVU-Test rows and 6.26% of Video-Holmes rows (empty ballots excluded throughout, per the convention of § 3), and the gold answer sits inside the tied set on well over half of them—so we recomputed Δmaj\Delta_{\mathrm{maj}} for all six Phi-3.5 cells under three gold-blind rules (first appearance; alphabetically first among the tied options; and alphabetically last), under both scored fields. All 36 resulting comparisons are negative and significant at α=0.05\alpha=0.05, the loosest at p=8.1×10−3p=8.1 \times10^{-3}, and all 36 pass the count identity of § 3: neither the direction nor the significance of the weak verifier’s majority deficit depends on the rule. The magnitude does. The gold-blind spread reaches 1.23 points on MLVU-Test, and an envelope that additionally admits gold-aware bounds reaches 4.30 points on Video-Holmes, so we do not rank cells by Δmaj\Delta_{\mathrm{maj}} at a resolution finer than about 1.2 points, and no claim in this paper rests on an exact Δmaj\Delta_{\mathrm{maj}} magnitude. The position-bias grid is untouched by this seam: A-lift compares the selector’s pick against gold and never consults the majority vote at all.

The flagship family, enumerated. The three-cell family over which we apply Holm–Bonferroni is the set of three Δmaj\Delta_{\mathrm{maj}} comparisons on Video-Holmes made by the two deployed cross-model verifiers: GPT-5.6 Sol on the Qwen-base pool (p=7.0×10−4p=7.0 \times10^{-4}), Muse on the Qwen-base pool (p=2.3×10−3p=2.3 \times10^{-3}), and Muse on the Muse-base pool (p=0.331p=0.331). The family-wise level is α=0.05\alpha=0.05. Both Qwen-base cells survive correction. The Muse-base cell survives at no family size and does not reach significance uncorrected, so it is a null we report rather than a survivor that drops out. Enlarging the family—for instance to the five cells the Solver policy makes available—leaves both survivors intact, but we do not offer that as independent confirmation, since it re-tests the same two comparisons against a larger family drawn from the same data. What the enlargement does establish is a robustness bound: under Holm, adding a comparison tightens an incumbent’s threshold by one rung whenever the added pp is at least the incumbent’s own, so the worst case for an incumbent at family size MM is the Bonferroni level α/M\alpha/M. Both survivors clear that level at every family size up to M=21M=21 (Sol up to M=70M=70), so no plausible enumeration of the comparison family overturns them.

Scoring policies. A verifier asked to select can solve instead: it may emit a letter outside the NN-candidate set, and if that out-of-candidate (OOC) letter happens to be correct, naive scoring credits it as a successful selection. OOC rates vary by benchmark and verifier. Across the six deployed cross-model cells they are 4.70% and 4.84% on TempCompass, 5.04% and 8.96% on MLVU-Test, and 1.73% and 0.71% on VRBench (Muse-48 and GPT-5.6 Sol respectively); the deployed Video-Holmes cells span 0.9–6.1%; and the highest rate we observe anywhere is ≈13%, on a weaker verifier. Verifier strength alone does not order them: the stronger verifier’s rate is nearly twice the weaker one’s on MLVU-Test and under half of it on VRBench. We therefore define three scoring policies. Solver counts OOC hits (the default in prior work). Selector counts a pick only if it is correct and belongs to the sampled candidate set; a correct pick outside the set is scored wrong, since it reflects solving rather than selection. Snap maps an OOC pick to the nearest candidate. Headline numbers below use the Solver policy, the default in prior work, so our accuracies are directly comparable to published best-of-NN results; we report the stricter Selector policy as a clearly-labeled ablation and flag where the two disagree. The gap between Solver and Selector scoring is exactly the leakage a headline number would over-credit: for the cross-model verifier, whose out-of-candidate rate stays under 9% on every deployed split, Solver scoring still inflates its apparent selection accuracy. Under Selector scoring the same six cells read 83.3% and 84.3% on TempCompass, 72.8% and 69.1% on MLVU-Test, and 87.7% and 89.3% on VRBench (Muse-48 and GPT-5.6 Sol respectively), against the Solver accuracies of Table 3; the ablation moves the headline by 3.88, 4.27, 1.21, 1.79, 0.10 and 0.00 points in that order. The out-of-candidate correction applies only to selection: the standalone solver never sees CC, so the standalone leg of the Selection Increment carries no out-of-candidate adjustment and is identical under all three policies.

Statistics. Every headline delta is matched-sample and tested with an exact two-sided paired McNemar test on the discordant pairs. Because we evaluate a grid of verifiers and benchmarks, we report per-comparison McNemar significance, flag non-significant deltas (n.s.) throughout, and treat isolated significant results cautiously given the number of comparisons. Implementation and scoring-integrity details are in Appendix § 10.

Denominators on the anchor. Majority@8, pass@1 and oracle@8 are computed on all 1,837 Video-Holmes questions. A chain that yields no extractable option letter (1.06% of the 14,696 chains; no question loses all NN) casts no ballot and is excluded from the pass@1 average, which we take per question and then average over questions, matching the majority@8 convention. The selector cells are computed on the items for which the verifier returned a parseable pick: 1,834 for GPT-5.6 Sol and 1,825/1,824 for the Muse verifier on the Qwen-/Muse-generated pools, following the matched-sample convention of § 3. The shortfall is entirely verifier-side: three video-decode failures inside the GPT verifier, and 12/13 Muse responses that completed but contained no extractable option letter. No item is lost at load time—every pick file contains all 1,837 question keys, with no duplicates and with gold labels identical to the released annotations, and a zero-shot NN=16 run on the same split scores all 1,837. Scoring the unanswered items as incorrect rather than excluding them leaves our conclusions unchanged: GPT-5.6 Sol +3.11 → +2.99 points over self-consistency (McNemar p=7.0×10−4→1.1×10−3p=7.0 \times10^{-4} \to1.1 \times10^{-3}) and the Muse verifier on the Qwen-base pool +2.85 → +2.40 (p=2.3×10−3→1.1×10−2p=2.3 \times10^{-3} \to1.1 \times10^{-2}), both still significant under the Solver policy after Holm correction across the three-cell family. Under the stricter Selector ablation it does move the GPT-5.6 Sol margin from +1.80 (p=0.043p=0.043) to +1.69 (p=0.059p=0.059); we report both and treat that margin as not established, consistent with its failure to survive Holm correction under either denominator.

Negative control: gathering more evidence does not raise the ceiling

On-demand tools add no selectable benefit, even at the oracle. A natural objection is that selection looks limited only because the sampled candidates are impoverished, and that letting the model gather more evidence would raise the ceiling a verifier works against. It does not. Augmenting candidate generation with an on-demand agentic loop that may zoom, crop, retrieve frames, or ground objects leaves the best-of-NN oracle on EgoSchema unchanged (Δoracle=0.000\Delta_{\mathrm{oracle}}=0.000 at n=500n=500). The reason is mechanical: selection can only surface a correct answer the sampler already produced, and the tools produce no new correct trajectory to surface, so even a hypothetical error-free verifier would gain nothing from them. Nor is the null a weak-model artifact—giving the same evidence loop to a strong cross-model verifier (Muse) leaves its accuracy within noise of its zero-shot self (−2.60 points at n=500n=500, a small, non-significant change). Tool-augmented inference does not move the ceiling that selection works against on either side of the pipeline, which is what directs this study at selection rather than gathering.

Extended results

Response-mode decomposition of the position bias

Table 5 gives the decomposition behind § 5. The weak verifier’s responses partition into two modes that output length separates without overlap—five tokens for the marker form Answer: X, three for a bare letter—and every response in both modes terminates normally, with no truncation at the token budget.6 The partition is therefore a property of the verifier’s output, not of our parser: the strict answer-marker parse is inert on this subset, and holding the row set fixed while swapping only the scored field changes no cell. What separates the two mode columns below is which rows are present, not how they are scored.

The first-option preference is confined to the terse mode in all eight arms in which that mode occurs, and no marker-mode arm reaches significance in any of the ten arms measured—a null we read as not distinguishable from zero at these per-arm nn, the closest cell missing α\alpha at p=0.0501p=0.0501. The split is not a restatement of accuracy: the terse mode is less accurate in seven of the eight arms, but on EgoSchema at KK=16 the two modes are accurate to within 0.11 points and the terse mode is still significantly biased, so the preference is a directional default rather than a by-product of solving worse.

That makes the mode contrast confounded with the item at the arm level, so we also test it within items. Restricting to the items whose mode changes across frame counts holds the question, its options, its gold answer and its difficulty fixed by construction, though not the frame count itself: the

Table 5: Position bias by response mode, weak verifier (Phi-3.5-Vision). Modes are disjoint and separated by output length. All tests are exact two-sided binomial McNemar on non-empty picks; A-lift is 100(bA−cA)/n100(b_A-c_A)/n. Marker-mode A-lift magnitudes are withheld pending independent verification: no marker-mode cell reaches significance in any of the ten arms as scored here, each on its own full row set, and every marker-mode ∣|A-lift∣| is below 4 points, though at these per-arm nn that is a failure to distinguish the marker mode from zero rather than a demonstration that it is unbiased—the closest cell sits just above α\alpha, at p=0.0501p=0.0501. Accuracies are under the Solver policy.

Verifier–benchmark armMarker mode (Answer: X)Terse mode (bare letter)
nnacc.ppnnacc.A-liftpp
MLVU-Test, K=24K=2430571.50.050119754.8+27.91.5×10−121.5 \times10^{-12}
MLVU-Test, K=48K=4827570.20.72022756.8+29.11.3×10−161.3 \times10^{-16}
MLVU-Test, K=64K=6432669.30.31117654.0+32.49.0×10−159.0 \times10^{-15}
Video-Holmes, K=24K=241,24153.80.72959647.0+13.42.3×10−102.3 \times10^{-10}
Video-Holmes, K=48K=481,22553.40.34661241.7+24.06.8×10−256.8 \times10^{-25}
Video-Holmes, K=64K=641,43252.31.00040438.6+26.75.4×10−195.4 \times10^{-19}
EgoSchema, K=16K=1618569.71.00031569.8+6.72.5×10−32.5 \times10^{-3}
EgoSchema, K=48K=4823267.70.70126863.4+13.87.5×10−77.5 \times10^{-7}

Table 5. Position bias by response mode, weak verifier (Phi-3.5-Vision).

terse and marker observations of such an item necessarily come from different frame budgets. Terse observations nevertheless favor the first option by twenty-five points more than marker observations of the same items on MLVU-Test, and by eighteen points more on Video-Holmes (exact two-sided sign tests, p=4.8×10−7p=4.8 \times10^{-7} and 1.9×10−171.9 \times10^{-17}; zero and four of 81 and 340 items respectively run the other way). The frame budget does not explain this. The contrast is positive in both switch directions and in every frame-count pair taken separately, so it cannot be a monotone effect of frame count; on the items whose terse and marker observations differ by only four frames it is larger, not smaller (+29.5 and +38.8 points); and the frame effect measured on items whose mode never changes—where the frame budget varies but the mode does not—is at most 2.4 points, an order of magnitude too small to produce it.

The capable InternVL3-38B control emits the marker on all 502 items of both its arms, produces no terse response at all, and reaches significance in neither (p=0.098p=0.098 and p=0.161p=0.161), which is the outcome we predicted before measuring it. Because that arm carries no terse rows, however, mode and model strength are collinear within it: it discharges extraction failure and item difficulty as explanations, but it cannot separate the response mode from verifier strength, and we do not read it as establishing that the strong verifier is unbiased.

What does separate mode from item is an item-matched stratification. Scoring InternVL3-38B on exactly the items where the weak verifier was terse—the same 502-item pool, and an identical gold map—leaves it at +1.02 points on the 197 items of the K=24K=24 arm (p=0.85p=0.85) and at 0.00 on the 227 items of the K=48K=48 arm (p=1.000p=1.000), against the +27.9 and +29.1 that the weak verifier posts on those very items (Table 5). The terse rows are therefore not merely A-heavy items. Two scoping notes attach to this. Under the same stratification InternVL3-38B’s marker cells do move to significance, in the direction of a deficit at the first option rather than an excess, so the tally we report scores each arm on its own full 502-item cell. And the two InternVL3-38B arms are near-duplicates—they agree on 486 of 502 picks—so they furnish nine effectively independent marker tests rather than ten.

Why the filtered view hides a real bias. The answer-marker partition acts entirely on the denominator, and not neutrally. It removes 22–63% of responses, and the removed rows are the biased ones: the rate at which the verifier picks “A” differs sharply between retained and removed rows (Fisher exact p<10−13p < 10^{-13} in all six MLVU-Test and Video-Holmes cells, as low as 1.3×10−351.3 \times10^{-35}; p=2.4×10−2p=2.4 \times10^{-2} and 6.7×10−16.7 \times10^{-1} in the two EgoSchema arms), while the rate at which “A is gold” is matched between them (exact two-sided Fisher, sum-of-tables convention, p=0.30p=0.30 to 0.74 across all eight arms). The removed rows are therefore not differently keyed. Neither are they materially harder: on a verifier-blind difficulty proxy—the majority@8 accuracy of the same generator pool—the two sets are statistically indistinguishable on Video-Holmes and EgoSchema (p=0.13p=0.13 to 1.00), and the one gap that does appear, on MLVU-Test (66.5% versus 77.0% at K=24K=24, p=0.010p=0.010 uncorrected, 0.062 after Holm correction over the six cells), is about half a task-type composition effect: the terse rows over-sample MLVU-Test’s hardest task types, and within task type the gap narrows to +5.1 points and is not significant (p=0.17p=0.17 to 0.260.26). What separates the two sets is overwhelmingly the verifier’s own letter preference—they are the rows on which this verifier misbehaves. Conditioning on the marker is thus a selection on the verifier’s own behavior, which is why a preference that is real in the full sample is not visible in the filtered one.

Provenance of the verifier ladders

(i) The two ladders are not independent: the Qwen-27B self-verifier rung appears on both, measured under a 1,024-token cap that bound its solving far harder than its picking, so its standalone is a lower bound and its increment an upper bound on both. Re-measured at 8,192 tokens it moves toward the strong-solver end of each ladder and its increment falls with it—each ordering survives, but MLVU-Test is not evidence independent of Video-Holmes. (ii) Phi-3.5 runs at 24 frames, against 48 for every other rung. Both its legs share that compute budget, so its SI is a matched difference; but a ladder sweeps standalone accuracy, so part of why this rung solves poorly is protocol, not capability. (iii) The GPT-5.6 Sol rung is differenced on its own matched non-empty support: three of its selector ballots and one of its standalone ballots are empty, and the two sets are disjoint, so that rung’s increment rests on n=1,833n=1{,}833 rather than the full 1,837 split. Scoring those four ballots wrong and differencing on all 1,837 instead moves the rung to +7.7 from a 54.2% standalone.

The verifier’s frame budget, rung by rung

The verifier’s frame budget is not a lever in either direction. Across six paired comparisons on these two benchmarks at 24, 48 and 64 frames, no rung improves selection and every point estimate is ≤0\le0; but the only significant degradations cross the 24 → 48 step, which also carries a forced doubling of the engine context length and a halving of the decode batch size, so we do not attribute them to frames alone. The single rung that isolates the frame budget under an otherwise identical engine configuration (48 → 64) is null on both benchmarks (MLVU-Test −0.20 points, p=1.00p=1.00; Video-Holmes −0.16, p=0.82p=0.82).

The weak Phi-3.5 verifier’s deficit against the vote also replicates on EgoSchema: Δmaj\Delta_{\mathrm{maj}} is −6.8 points at 16 frames (exact McNemar p=7.6×10−5p=7.6 \times10^{-5}) and −11.2 at 48 frames (p=3.1×10−9p=3.1 \times10^{-9}), against a majority@8 of 76.6% on the EgoSchema pool (n=500n=500). Both cells are deficits, stated with their sign; they sit outside the Holm-corrected flagship beats-majority family and we report them as a separate replication.

This appendix collects the tables and figures supporting § 4 and § 5 that the page limit keeps out of the main text. Every number is as measured; no cell is recomputed here.

Selection Increment by benchmark and verifier

Figure 3. Selection Increment by benchmark and verifier. SI (selection accuracy minus the same verifier answering directly, both under the Solver policy) for the Muse holistic-48 verifier (blue) and GPT-5.6 Sol (red). GPT-5.6 Sol yields positive SI on all three benchmarks while Muse splits (negative on TempCompass)—not because GPT is the stronger selector but because its standalone accuracy typically sits below Muse’s, leaving the candidates more room, consistent with SI tracking standalone accuracy inversely (§ 4). VRBench cells use the first 1,000 questions of its split (118 of 960 videos).

Verifier strength and the two axes, in full

The paragraphs below are the long-form analysis behind § 4.

Table 3 bears this out. On the full TempCompass split (n=1,574n=1{,}574) the cross-model verifier reaches 87.2% against 79.2% for self-consistency, a gain of 7.94 points—its largest anywhere. The edge closes, however, as the splits saturate: on MLVU-Test selection only ties self-consistency (+0.81, n.s., n=496n=496), and on VRBench, whose self-consistency already reaches 91%, it loses (−3.26, McNemar p=5.6×10−6p=5.6 \times10^{-6}). These accuracies use the Solver convention standard in prior work (§ 3); the candidates-only ablation, which removes any credit the verifier could earn by answering the question itself rather than selecting, moves them by 0.0–4.3 points and is reported in Appendix § 11. On VRBench with the GPT-5.6 Sol verifier the two policies coincide exactly, because none of the seven out-of-candidate picks there is correct.

The “headroom” rule does not predict selection benefit. A natural hypothesis—the “headroom” rule—holds that selection helps in proportion to the room the sampler leaves: the headroom h=h = oracle − majority, the gap between the oracle and the self-consistency (majority voting) baseline, with a break-even threshold below which selection should not beat self-consistency. The data give it no support. Across our benchmarks the verifier’s gain over self-consistency is only weakly and non-significantly correlated with headroom, and individual benchmarks violate any monotone version in both directions: TempCompass, at low headroom (≈ 11 points), beats self-consistency by +9.4 points (p<10−23p<10^{-23}), whereas MLVU-Test, at higher headroom (≈14\approx14 points), underperforms it (−2.0, n.s.). VRBench, at the lowest headroom of all (≈6\approx6 points), also loses (−1.6 to −3.3, McNemar-significant), consistent with the rule rather than against it. We report this as a negative result and caveat it honestly: with three benchmarks spanning a compressed ≈6\approx6–14-point range, we do not claim a reliable correlation in either direction, only that there is no evidence for a headroom rule. Headroom should not be confused with the level of the majority baseline, which does carry signal: the two are different quantities—one a gap, the other a position—and only the second survives below. What predicts the outcome is the verifier itself, once the baseline leaves it anything to win.

The Selection Increment falls as a verifier’s standalone accuracy rises. The cleanest read comes from holding the candidate pool fixed and swapping a weak verifier for a strong one. On Video-Holmes the weak Qwen-27B verifier (standalone 27.7%) posts a large increment (SI +28.7 and +31.2 on the two base pools), while the strong Muse verifier (standalone 59.4%) posts essentially none (−2.5 and +0.9); on TempCompass the same swap moves SI from +8.7 (weak Qwen) to −1.1 (strong Muse) on the Qwen-base pool (Tables 4, 6). The pattern is monotone: sweeping verifiers of increasing standalone accuracy over a fixed pool drives SI toward zero. A five-verifier ladder on the Video-Holmes Qwen-base pool makes this explicit—SI falls from +28.7 (Qwen-27B self-verifier, standalone 27.7%) through +19.3 (Phi-3.5, 32.5%), +17.6 (InternVL3-8B, 39.7%) and +10.9 (InternVL3-38B, 47.7%) to +7.9 (GPT-5.6 Sol, 54.1%)—and an MLVU-Test ladder gives the same ordering (+26.3, +21.6, +20.8, +10.8 as standalone rises 38.6 → 46.7 → 51.1 → 62.0%). A large SI therefore marks a weak solver that can nonetheless select, not a capability score. SI is an increment net of the verifier’s own solving—how much the candidates add to a given verifier—not a capability score.

Read this way, the one swap that appears to run the other direction is not a contradiction. Replacing Muse with the closed GPT-5.6 Sol raises SI on every deployed benchmark—to +1.15 from −1.08 (TempCompass), +11.16 from +5.68 (MLVU-Test) and +3.96 from +3.67 (VRBench) (Table 3, Figure 3)—not because GPT is the stronger selector but because it under-solves these benchmarks by more than Muse does: its standalone typically sits below Muse’s (on TempCompass 87.4% vs. 88.3%,7 partly because it reads fewer frames), so it has more room for the candidates to help. (Consistently, on Video-Holmes GPT is the strongest solver of the five verifiers above—and so posts the smallest increment there.) Both swaps thus obey one rule—SI grows as a verifier’s own standalone accuracy falls—rather than tracking raw verifier strength.

Video-Holmes isolates the verifier lever. The second axis—beating self-consistency—is cleanest on Video-Holmes, our primary reasoning anchor, because there the two candidate pools are near-equally hard: majority vote sits at 58.9% on the Qwen-base pool and 59.6% on the Muse-base pool. With base difficulty held nearly constant—though the Qwen-base pool reads 64 frames and the Muse-base pool 32—the verifier decides the outcome (Table 4). Under the Solver policy the strong Muse verifier significantly beats majority on the Qwen-base pool (+2.8; McNemar p=2.3×10−3p=2.3 \times10^{-3}, Holm-corrected over the three-cell flagship family) and numerically edges it on the near-tied Muse-base pool (+0.8; p=0.33p=0.33, n.s.), whereas the weak Qwen verifier fails to beat it on either (+0.05 and +0.00; exact McNemar p=1.000p=1.000 on both—exact draws, not marginal nulls); base has little say. Provenance for the 2×22 \times2: the two Δmaj\Delta_{\mathrm{maj}} cells in the Qwen-27B column are re-measured at 8,192 tokens with a corrected answer extractor; as originally measured at a 1,024-token cap that truncated the verifier’s own response they read −2.5 and −0.7, the second never significantly (p=0.26p=0.26), and both corrected cells are exact draws. Panel A of Table 4 is not re-measured, so for that column the two panels rest on different measurements of the same selection accuracy and are not additive (Appendix § 10). The standalone 27.7 that Panel A differences against was measured under that same 1,024-token cap, which bound solving far harder than picking, so this column is an upper bound on SI; the paper’s dissociation claim is anchored instead on the Phi-3.5 ladder rung, whose two legs share one token budget. The two-axis dissociation takes its sharpest form on this same Qwen-base pool. The weak Phi-3.5 verifier selects far better than it solves—SI +19.3 from a 32.5% standalone—yet its selected accuracy lands 7.1 points below the majority vote (51.8% vs. 58.9%; exact McNemar p<10−13p<10^{-13}), whereas the strong Muse verifier adds almost nothing over its own solving (+2.5) yet beats it. The same pattern runs the length of the five-verifier ladder over this pool: all four open verifiers post large increments and none beats the 58.9% majority; only the frontier GPT-5.0 clears it (62.0%, +3.1; McNemar p=7.0×10−4p=7.0 \times10^{-4}). Under the Solver policy both the Qwen-base Muse beat and this GPT beat survive Holm–Bonferroni correction across the three-cell flagship family; under the stricter candidates-only ablation neither does, and we report that margin as not established. Phi-3.5’s deficit is not confined to Video-Holmes: the same verifier reproduces it on MLVU-Test—8.0 points below the majority vote there (64.9% vs. 72.9%, n=502n=502; exact McNemar p=2.4×10−6p=2.4 \times10^{-6})—against a different solver pool and a different baseline, with neither cell truncating at the generation cap. Both cells do, however, share the verifier’s 24-frame budget—a confound that replication across benchmarks cannot discharge—so we re-ran the verifier on both at the generator’s own 64 frames. Parity does not close either deficit; it leaves it numerically larger, at 9.6 points below the vote on Video-Holmes (49.3% vs. 58.9%, n=1836n=1836; exact McNemar p=3.3×10−22p=3.3 \times10^{-22}) and 9.0 on MLVU-Test (63.9% vs. 72.9%, n=502n=502; p=1.0×10−6p=1.0 \times10^{-6}). A frame handicap predicts improvement as the verifier moves toward the generator’s compute budget; we measure degradation, which falsifies that account without establishing what the real mechanism is.

On TempCompass, base saturation takes over. TempCompass shows the second gate at work. Its full 2×22 \times2 (Table 6) toggles a Qwen-base pool (n≈1,579n\approx1,579, majority 79.2%) against a stronger Muse-base pool (n≈1,000n\approx1,000, majority 87.5%). The SI axis behaves exactly as before—a column effect, large for the weak Qwen verifier on both bases (+8.7, +18.5) and near zero for the strong Muse verifier (−1.1, +0.5). Beating self-consistency, however, becomes a row effect set by base saturation: on the Qwen-base pool every verifier matches or beats the 79.2% majority, and the strong verifier wins outright (+7.94), whereas on the near-saturated Muse-base pool (87.5% majority) neither verifier does. Because the task is held fixed and only the pool changes—the base model together with its frame budget—this contrast tracks saturation the way Video-Holmes isolates strength.

A positive increment need not beat self-consistency. Positive SI means the candidates help the verifier beyond its own solving; it does not guarantee beating computationally cheap self-consistency, above all when self-consistency is already near its ceiling. VRBench is the clearest case. At the suite’s lowest headroom both verifiers post positive SI (Muse +3.67, GPT +3.96) yet both significantly trail self-consistency (Δmaj=−3.26\Delta_{\mathrm{maj}}=-3.26, p=5.6×10−6p=5.6 \times10^{-6}; −1.62, p=3.7×10−3p=3.7 \times10^{-3}), because self-consistency there is near ceiling (≈91%\approx91\%), which leaves the verifier little to gain and easy ground to lose. MLVU-Test with the GPT verifier repeats the pattern with a much larger increment (SI = +11.16) and Δmaj=−1.99\Delta_{\mathrm{maj}}=-1.99 (n.s.). Figure 2 places all deployed benchmark–verifier pairs in these two coordinates: the stronger verifier raises SI everywhere, but whether a pair clears the self-consistency line is a separate matter—some rise into the useful quadrant, while VRBench and MLVU-Test stay below it despite large positive increments.

A mechanistic reading, and a benchmark-specific inversion. The lever behind SI is a verifier’s own standalone accuracy on the benchmark, measured independently of any selection outcome: SI is large when that accuracy is low and collapses as it rises. Because this is not raw capability—the same model can be a weak solver on one benchmark and a strong one on another—it predicts a benchmark-specific inversion inside the InternVL3 family. On EgoSchema the increment rises from the 8B to the ∼5×\sim5\times-larger 38B verifier (+0.4, n.s., →\to +5.6, p=0.01p=0.01), because there the 38B actually under-solves the 8B (standalone 72.4% vs. 79.2%—the 8B out-solves its larger sibling on this benchmark), leaving the larger verifier more room; the standalone ordering reverts on MLVU-Test (8B 51.1% vs. 38B 62.0%), where the increment falls with strength as usual. Consistent with the inverse rule, the weakest solver posts the largest increment on both benchmarks—Phi-3.5’s SI reaches +19.8 on VRBench and +25.2 on EgoSchema—and the decoupling again runs both ways: on EgoSchema InternVL3-8B beats self-consistency (+3.0) despite a negligible increment (SI= +0.4, n.s.). Two levers, then: a verifier’s independently measured standalone accuracy governs the size of the Selection Increment (inversely), while its strength—with base difficulty held fixed, as on Video-Holmes—governs whether selection is worth more than a majority vote.

Answer-position bias: mechanism, frontier behavior, and prior work

These paragraphs extend Section 5.

The mechanism is a graded positional slope, not a first-option collapse. On Video-Holmes (verifier K=24K=24) the weak Phi-3.5 verifier’s per-letter deviation from gold runs monotonically from +4.25 (A) through +3.81, +2.34, +1.52 down to −4.41 (E) and −7.51 (F)—a range of 11.76 points at a perfect rank order—and the last-option deficit (−7.51) exceeds the first-option excess (+4.25) in magnitude; EgoSchema (verifier K=48K=48) shows the same asymmetry (+8.0 at A to −10.0 at E, range 18.0). This is why primacy, not letter “A”, is the right headline: across benchmarks the weak verifier’s primacy is positive on all five (within-benchmark ρ\rho with accuracy = −0.900, Fisher p=0.0027p=0.0027), whereas its bare A-lift is positive on only four—on TempCompass the verifier still drifts toward earlier options yet its A-lift is flat (0.0), so a letter-“A” statistic alone would miss the bias that primacy detects. One caveat: because every benchmark here labels options alphabetically, an option’s position in the list and its letter rank move together, so neither primacy nor A-lift, as we report them, separates a preference for earlier positions from a preference for earlier letters; separating them cleanly would require re-running the verifier with the option labels permuted against fixed slots—merely reordering the options leaves “A” in the first slot and does not break the collinearity—which we leave to future work.

At the frontier the pattern narrows but does not reverse, and even the narrowing is partly a property of the candidate pool. Our two deployed verifiers (Muse and GPT-5.6 Sol) track gold on TempCompass, MLVU-Test, and VRBench (A-lift between −0.13 and −1.8). Where a strong verifier’s A-lift turns negative it is inheriting an A-poor candidate pool—the sampled candidates under-represent “A” relative to gold by 1.36–1.66 points on Video-Holmes, MLVU-Test, and TempCompass. Measured against the pool the verifier is actually offered, no frontier verifier is anti-“A” (Video-Holmes GPT-5.6 Sol −1.20 →\to +0.26, Muse −0.88 →\to +0.54), and the frontier verifiers’ own candidate-free A-lift is positive (GPT +1.74, Muse +1.15); the frontier is thus statistically indistinguishable from zero, not negative. Recency avoidance, by contrast, persists undiminished at every capability level: the last-option deficit is positive in 21 of 22 cells (sole exception MLVU-Test/InternVL3-38B, −0.80), tracks accuracy at within-benchmark ρ=−0.900\rho=-0.900 (Fisher p=0.0025p=0.0025), and for nine cells—all but one on EgoSchema—the deficit exceeds the first-option excess (EgoSchema Muse +4.60). The deficit persists even at the frontier, where the first-option excess is absent (Video-Holmes GPT +1.58, Muse +0.60). Position bias narrows at the frontier but does not disappear.

We report significance conservatively, and the statistics support a threshold rather than a law. Every permutation pp in this section is one-sided against the predicted negative direction; the ten-configuration test enumerates all 10!10! label assignments exactly, while the pooled within-benchmark tests are Monte-Carlo estimates over 2×1052\times10^{5} draws. Pooled across all cells with benchmark fixed effects, the accuracy–skew rank correlation is strong (ρ=−0.8313\rho=-0.8313, within-benchmark permutation p≈8×10−5p\approx8 \times10^{-5}), but the attenuation just described shows the under-capable end carrying it. Of the 22 cells, nine are individually significant by an exact McNemar test (α=0.05\alpha=0.05, uncorrected) and five survive Holm–Bonferroni correction. All four positive survivors belong to Phi-3.5-Vision; the fifth is the capable Qwen3.5-27B self-verifier on Video-Holmes, which clears correction with its sign reversed (p=0.0020p=0.0020, −2.8 points)—below gold, not above it. The bias that survives correction is thus entirely a weak-verifier effect, and the capable verifier supplies a signed control on it. Two qualifications keep the claim narrow. The mid-tier InternVL3 family is genuinely anti-“A” even against the offered pool—more so than the frontier closed models—so we claim capability gating and not an ordering of capability tiers by raw bias means. And VRBench, run on two verifiers, contributes only a sign, its k=2k=2 rank correlation having a floor exact pp of 0.5. The weak-verifier default is also benchmark-dependent (Table 8): Phi-3.5 lifts “A” by +8.6 on MLVU-Test and +3.0 on VRBench, while on TempCompass no verifier defaults at all. The capable self-verifier does not default on either cell we could re-measure without a truncating generation cap (MLVU-Test −2.2, Video-Holmes −2.8); its apparent default in the shipped measurement was an artifact of that cap.

BenchmarkPhi-3.5 (weak)Qwen3.5-27B (self)InternVL3-8BInternVL3-38B
TempCompass0.0−0.8−1.6−2.0
MLVU-Test+8.6−2.2−3.4−2.8
VRBench+3.0——−0.6
EgoSchema+8.0+0.6+0.6+1.4
Video-Holmes+4.2−2.8−1.6−1.9

Table 8. Answer-position bias is a pathology of under-capable verifiers. Per-verifier A-lift (pick-“A” rate minus gold-“A” rate, in points; under the Solver scoring policy) across benchmarks. Where a benchmark induces the bias, the weak verifier (Phi-3.5) lifts “A” above gold, whereas the capable same-family self-verifier sits below gold on both cells we re-measured without a truncating generation cap (MLVU-Test −2.2, Video-Holmes −2.8); no capable cross-family verifier lifts “A” above gold by more than 1.4 points (on Video-Holmes the frontier GPT-5.6 Sol is likewise robust, −1.2). TempCompass induces no default for any verifier. Compute budget caveat. The Qwen3.5-27B (self) column was first measured at a 1,024-token cap that truncated the verifier’s own response. Four cells were re-measured at 8,192 tokens with a corrected answer extractor. Both Qwen3.5-27B cells change sign—by −8.0 points on MLVU-Test and −5.0 on Video-Holmes—and raising the cap alone accounts for −6.4 and −4.4 of that, flipping both signs by itself; the extractor only deepens them. The other two are Phi-3.5 positive controls: its cap and extractor effects are exactly zero (its longest MLVU-Test response is five tokens, so the cap never bound), and re-measuring moved its Video-Holmes A-lift by only 0.22 points—from the +4.2 printed here to +4.0—through run-to-run sampling alone. That residual, against flips of 8.0 and 5.0 points, is what attributes the reversals to the cap rather than to the act of re-measuring. Phi-3.5 entries are the original run; the significance tests of Section 5 use the re-measured one, which changes no survivor. Every remaining entry, including the other two Qwen3.5-27B cells (−0.8, +0.6), is uncorrected: original extractor, 1,024-token cap. Phi-3.5 on EgoSchema is shown at 48 frames; dashes mark cells not run.

Answer-position bias is a well-documented failure mode of language-model selectors: an inherent behavioral bias over option-ID tokens rather than a prompting artifact [46, 27], it swings accuracy under option reordering [24], destabilizes LLM-as-a-judge evaluation [47, 33, 29], and has been measured in video- and vision-language multiple-choice models [15, 31, 3]. That capability attenuates it is likewise known: stronger vision-language models show reduced but nonzero selection bias [3], accuracy gates rather than removes length-driven position bias in reasoning models [36], and more capable judges hold up better as candidate lists grow [29]. On that axis our finding is a replication in a new setting. What has not been measured is position bias in the best-of-N verifier role—selecting among a model’s own sampled candidates rather than answering a multiple-choice item or ranking a pair. The skew belongs to the verifier and not to the surrounding pipeline: the candidate generator is B/C-skewed rather than A-skewed, and the majority-vote tie-break is exactly insertion-order rather than alphabetical, which rules out the argmax-tie mechanism identified for prior video-language selectors [15]. Our contribution is that setting and its controls, not a new phenomenon—frontier judges can be reliable yet position-biased [21], bias direction is model-specific [41], and near-ceiling accuracy can suppress measurable bias for reasons unrelated to robustness [30].

When a verifier pass is worth its compute

Section 6 compresses our computational cost reasoning into a three-step allocation policy. Because that policy is the part of this study a practitioner is most likely to act on, we set it out at length here, and separate what we measured from what we infer from the measurements.

What the second pass requires, and what it does not

The accounting is easiest to state in passes rather than tokens. Sampling NN candidates from the base is a requirement of admission for every method considered here, including the baseline: once those candidates exist, self-consistency is computationally cheap, because a majority vote reads the answer letters and never calls a model again. Verification adds one listwise pass in which the verifier re-watches the footage and reads all NN candidates, so it is the only step whose computational cost scales with both the video and the candidate set. The diagnostic we recommend adds one further pass—the verifier answering the same question with the candidates withheld—which is bounded by the computational cost of the verifier’s own solving and, unlike the selection pass, does not grow with NN.

We deliberately do not convert this into a token or wall-clock figure. Our runs were not instrumented for a benchmark-matched token accounting, both strong verifiers are reached by API under serving conditions we do not control, and frame budgets differ across rungs (Appendix E). Any efficiency ratio we printed would therefore report provisioning rather than a property of the method, so we make no compute-efficiency claim and treat computational cost qualitatively throughout.

Why the computational overhead can be justified

Diagnosis is computationally cheaper than the treatment. The two gates do not require the same amount of compute, and the computationally cheaper one is decisive wherever it applies. Majority accuracy over the sampled candidates requires nothing beyond the candidates themselves, so a practitioner can discover that verification is the wrong decision before choosing any of it: above roughly 87% no verifier in our study beat the vote, and on VRBench both deployed verifiers significantly lost to it (−1.62-1.62 and −3.26-3.26). Only once that computationally cheap gate is passed does the computationally expensive diagnostic become relevant—and it is the computationally expensive one, a single pass with the candidates withheld, that tells apart the two systems a headline gain conflates.

The increment separates two computational costs that one accuracy number merges. Read as an audit rather than a merit score, the sign of the increment answers a question about computational cost rather than about quality. Where it is negative—TempCompass with the Muse verifier, SI = −1.08-1.08—the verifier is more accurate answering directly than selecting, so the pipeline incurs computational cost for the candidate set and returns less than the verifier alone would. Where it is positive—MLVU-Test with that same verifier, +5.68+5.68—selection provides something standalone solving does not supply, and it does so even in cases where the result still fails to clear a computationally cheap majority vote. Neither fact is recoverable from the gain over the vote, which is why we report the two axes together.

A weak verifier can be a computationally expensive way to reproduce a computationally cheap baseline. The sharpest form of the computational cost argument never appears in an accuracy column. On the Qwen-base TempCompass pool the weak self-verifier agrees with the plurality candidate on 95.1% of rows, so its selection is close to a computationally expensive re-derivation of the vote it was meant to improve on: the pass is spent, the footage is re-watched, and the modal answer is returned. A delta near zero reads as harmless, but in computational cost terms it is the least attractive outcome available, because of the compute required for output the majority vote supplied for nothing.

Reliability, not only accuracy. Below a capability threshold the computational overhead also provides a failure mode. The weakest verifiers track where an option sits in the list—favoring earlier positions and avoiding the last—while every capable verifier we measured resists this (§ 5). That matters for deployment in a way an averaged accuracy hides: a positional preference is a systematic error correlated with option order, so it does not average away over a benchmark, and it will track whatever ordering convention a downstream system happens to adopt. A verifier that merely matches the vote, but reaches that accuracy partly through option order, is a weaker component than the arithmetic suggests.

Sampling widens the gap the vote cannot reach. One argument does favor the computational overhead of selection, and it concerns how the two quantities move with NN. On Video-Holmes, raising NN from 8 to 16 moves the best-of-NN oracle from 78.8% to 82.5% while self-consistency stays flat (58.9% to 59.0%; Table 2). Additional samples enlarge what a selector could in principle win without improving the computationally cheap baseline it has to beat, so the ceiling on selection is not pinned by the base model in the way the vote’s is. We report this as a property of our anchor rather than a scaling law: two values of NN establish a direction, not a rate, and the strongest verifier we ran still left roughly 17 points of that oracle headroom unclaimed.

Where the computational overhead is not justified

Three results bound the case for spending, and we state them as bounds rather than as discouragements. First, near-saturated pools: where the vote already sits at the observed ceiling, verification did not help and sometimes significantly hurt, so the allocation is to skip it. Second, a non-positive increment: there the verifier should be deployed as the solver directly, which is both more accurate and computationally cheaper, since it avoids reading the candidates at all. Third, the obvious alternative expenditure does not rescue the situation—augmenting candidate generation with an on-demand loop that may zoom, crop, retrieve frames, or ground objects left the best-of-NN oracle unchanged (Δoracle=0.000\Delta_{\mathrm{oracle}}=0.000; Appendix § 12), so a practitioner facing a pool that selection cannot exploit cannot repair it by gathering more evidence into that same pool. What remains, on our anchor, is the narrow case the body identifies: a pool below the saturation ceiling, paired with a verifier clearly stronger than the base and drawn from a different model family.

References

  1. [1]Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. Introduces the phi-3.5 series, including phi-3.5-Vision.arxiv.org/abs/2404.14219
  2. [2]Ali Alavi. Perception first: A frontier native-video model with self-consistency for implicit video question answering. arXiv preprint arXiv:2606.01485, 2026.arxiv.org/abs/2606.01485
  3. [3]Md. Atabuzzaman, Ali Asgarov, and Chris Thomas. Benchmarking and mitigating MCQA selection bias of large vision-language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025. arXiv:2509.16805.arxiv.org/abs/2509.16805
  4. [4]Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Re, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024.arxiv.org/abs/2407.21787
  5. [5]Wang Chen, Yu Chen, Xiang Wang, Shuai Li, Jinfa Huang, and Xiawu Zheng. One ranking, any budget: Matryoshka evidence-to-context frame selection for long-video understanding. arXiv preprint arXiv:2608.05707, 2026.arxiv.org/abs/2608.05707
  6. [6]Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-holmes: Can mllm think like holmes for complex video reasoning? arXiv preprint arXiv:2505.21374, 2025.arxiv.org/abs/2505.21374
  7. [7]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  8. [8]DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Nature, 645:633–638, 2025. arXiv:2501.12948.DOI
  9. [9]Garvin Guo, Donglei Yu, Yu Chen, Xiang Wang, Shuai Li, and Xinpei Zhao. Do multimodal agents really benefit from tool use? a systematic study of capability gains. arXiv preprint arXiv:2606.02357, 2026.
  10. [11]Muhammad Kamran Janjua, Hugo Silva, Di Niu, and Bahador Rashidi. Don’t show pixels, show cues: Unlocking visual tool reasoning in language models via perception programs. arXiv preprint arXiv:2604.12896, 2026.arxiv.org/abs/2604.12896
  11. [12]Jae Joong Lee. Accuracy without grounding: Diagnosing visual dependency dissociation in video llm benchmarks. arXiv preprint arXiv:2607.13305, 2026.arxiv.org/abs/2607.13305
  12. [13]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024. arXiv:2305.20050.arxiv.org/abs/2305.20050
  13. [14]Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? In Findings of the Association for Computational Linguistics ACL 2024, 2024. arXiv:2403.00476.arxiv.org/abs/2403.00476
  14. [15]Olga Loginova, Oleksandr Bezrukov, Ravi Shekhar, and Alexey Kravets. Addressing blind guessing: Calibration of selection bias in multiple-choice question answering by video language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025. arXiv:2410.14248.arxiv.org/abs/2410.14248
  15. [16]Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In Advances in Neural Information Processing Systems 36, 2023. arXiv:2308.09126.arxiv.org/abs/2308.09126
  16. [17]Meta AI. Introducing Muse Spark 1.1. https://ai.meta.com/blog/introducing-muse-spark-meta-model-api/, July 2026. Evaluation report: https://ai.meta.com/static-resource/muse-spark-1-1-evaluation-report/.
  17. [18]Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid. Morevqa: Exploring modular reasoning models for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. arXiv:2404.06511.arxiv.org/abs/2404.06511
  18. [19]Aseel Mohamed, Rama AlHamidi, Mohamed Rayan Barhdadi, et al. A cost-aware, paired protocol for auditing dynamic tool synthesis in agentic video question answering. arXiv preprint arXiv:2607.01469, 2026.
  19. [20]Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, and Tatsunori Hashimoto. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025. arXiv:2501.19393.arxiv.org/abs/2501.19393
  20. [21]Justin D. Norman, Michael U. Rivera, and D. Alex Hughes. Reliability without validity: A systematic, large-scale evaluation of LLM-as-a-judge models across agreement, consistency, and bias. arXiv preprint arXiv:2606.19544, 2026.arxiv.org/abs/2606.19544
  21. [22]OpenAI. OpenAI GPT-5 system card. arXiv preprint arXiv:2601.03267, 2025.
  22. [24]Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, 2024. arXiv:2308.11483.arxiv.org/abs/2308.11483
  23. [25]Qwen Team. Qwen3.5-27B. Hugging Face model card, https://huggingface.co/Qwen/Qwen3.5-27B, February 2026a.
  24. [26]Qwen Team. Qwen3.5-397B-A17B. Hugging Face model card, https://huggingface.co/Qwen/Qwen3.5-397B-A17B, February 2026b.
  25. [27]Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering. In International Conference on Learning Representations, 2023. arXiv:2210.12353.arxiv.org/abs/2210.12353
  26. [28]Durga Sandeep Saluru. Disagreement-based cross-model routing for implicit video question answering. arXiv preprint arXiv:2606.14723, 2026.arxiv.org/abs/2606.14723
  27. [29]Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic study of position bias in LLM-as-a-Judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, 2025. arXiv:2406.07791.arxiv.org/abs/2406.07791
  28. [30]Hiroki Tamba. Position bias is hidden behind ceiling effects: A permutation diagnostic for LLM benchmarks. arXiv preprint arXiv:2607.20864, 2026.arxiv.org/abs/2607.20864
  29. [31]Zhijie Tan, Xu Chu, Weiping Li, and Tong Mo. Order matters: Exploring order sensitivity in multimodal large language models. arXiv preprint arXiv:2410.16983, 2024.arxiv.org/abs/2410.16983
  30. [32]Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, et al. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796, 2024.arxiv.org/abs/2404.18796
  31. [33]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024a. arXiv:2305.17926.arxiv.org/abs/2305.17926
  32. [34]Qixun Wang, Yang Shi, Letian Cheng, Zhuoran Zhang, Yan He, and Yuqi Tang. Beacon: Knowing when and how to perform agentic visual reasoning. arXiv preprint arXiv:2607.28595, 2026a.arxiv.org/abs/2607.28595
  33. [35]Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. arXiv:2406.08035.arxiv.org/abs/2406.08035
  34. [36]Xiao Wang. More thinking, more bias: Length-driven position bias in reasoning models. arXiv preprint arXiv:2605.06672, 2026.arxiv.org/abs/2605.06672
  35. [37]Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision, 2024b. arXiv:2403.10517.arxiv.org/abs/2403.10517
  36. [38]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023. arXiv:2203.11171.arxiv.org/abs/2203.11171
  37. [40]Yuhan Wang, Shuochen Chang, Yalin Feng, Dongsheng Ma, Yuanzi Li, and Zhengren Wang. Seeing before agreeing: Aligning multi-agent consensus with visual evidence. arXiv preprint arXiv:2605.30698, 2026c.arxiv.org/abs/2605.30698
  38. [41]Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno, and Yoshitaka Ushiku. Am I more pointwise or pair-wise? revealing position bias in rubric-based LLM-as-a-Judge. arXiv preprint arXiv:2602.02219, 2026.arxiv.org/abs/2602.02219
  39. [42]Jiashuo Yu, Yue Wu, Meng Chu, Zhifei Ren, Zizheng Huang, Pei Chu, Ruijie Zhang, Yinan He, Qirui Li, Songze Li, Zhenxiang Li, Zhongying Tu, Conghui He, Yu Qiao, Yali Wang, Yi Wang, and Limin Wang. Vrbench: A benchmark for multi-step reasoning in long narrative videos. In ICCV, 2025.arxiv.org/abs/2506.10857
  40. [43]Zhang, Guo, Potamias, Deng, Xu, and Ma. Vtimecot: Thinking by drawing for video temporal grounding and reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025a. arXiv:2510.14672.arxiv.org/abs/2510.14672
  41. [44]Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024. arXiv:2312.17235.arxiv.org/abs/2312.17235
  42. [45]Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In International Conference on Learning Representations, 2025b. arXiv:2408.15240.arxiv.org/abs/2408.15240
  43. [46]Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2309.03882
  44. [47]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36 (Datasets and Benchmarks Track), 2023. arXiv:2306.05685.arxiv.org/abs/2306.05685
  45. [48]Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: Benchmarking multi-task long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. arXiv:2406.04264.arxiv.org/abs/2406.04264

Paper details

vlnc:W_g5a5g4ah
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents