WorksComputer Science
RAVEN-Bench: A Paired EO-IR Video QA Benchmark for Aerial Multimodal Understanding
Abstract
Multimodal large language models (MLLMs) have advanced rapidly in image and video understanding, yet their ability to reason over paired aerial electro-optical and infrared (EO-IR) videos remains poorly understood. Such video poses a harder test than standard RGB video QA: targets are often small, evidence may appear only briefly, viewpoints change with the platform, and useful cues may appear in only one stream. We introduce RAVEN-Bench (Reasoning over Aerial Videos with EO-IR aligNment), a paired EO-IR video QA benchmark for low-altitude aerial multimodal understanding. RAVEN-Bench contains 100 temporally aligned EO-IR video pairs and 1,200 human-authored four-way multiple-choice questions, each independently audited for answer keys, modality requirements, evidence intervals, and group validity. It supports EO-only, IR-only, and paired EO+IR evaluation, and uses hierarchical capability levels and structured question groups to diagnose paired-input gain beyond the stronger unimodal condition, unimodal reliance, target consistency, and evidence-chain coherence. We evaluate frontier, open-weight, and remote-sensing MLLMs, showing that aggregate accuracy can overstate evidence-supported EO-IR reasoning. RAVEN-Bench provides controlled modality conditions and group-level diagnostics for auditing whether progress on RGB video benchmarks transfers to aerial cross-spectral understanding. Code and data are available at https://github.com/yuhu990424/RAVEN-Bench and https://huggingface.co/datasets/TerraSenseAnalytics/RAVEN-Bench.
Introduction
Multimodal large language models (MLLMs) have made rapid progress in image and video understanding. Recent systems can describe dynamic scenes, follow visual instructions, and answer questions grounded in temporal evidence [1, 2, 3, 4]. Much of this progress has been measured on video benchmarks such as Video-MME [5], MLVU [6], and LVBench [7]. These benchmarks mainly use RGB natural videos, including web videos, movies, egocentric clips, instructional scenes, surveillance footage, and daily activities. Aerial electro-optical and infrared (EO–IR) video poses a different question, since the evidence can be small, brief, and tied to how each sensor responds to the scene.
Low-altitude EO–IR video often looks understandable at the scene level while remaining difficult at the target level. Aircraft, vessels, and vehicles may occupy only a small part of the frame, appear briefly under platform motion, or become ambiguous under occlusion, shadow, haze, low contrast, and background clutter. EO streams preserve shape, texture, color, and scene layout. IR streams expose thermal contrast that may become useful when visible-spectrum evidence is weak. For example, a vehicle may remain visually identifiable in EO across viewpoint changes, while its IR contrast varies as it moves between cooler background regions and warmer pavement. These conditions make aggregate video-QA accuracy a limited diagnostic signal. A model may answer an isolated question correctly while losing the same target in a related question. It may choose a plausible event-level answer without recognizing the intermediate visual evidence that supports it. It may also appear to benefit from EO+IR input while relying mainly on the stronger unimodal stream. Aerial EO–IR evaluation therefore needs controlled modality settings, temporally grounded questions, and group-level measures of target consistency and evidence-chain coherence. Before MLLMs are applied to non-RGB aerial video, researchers should be able to audit whether a model uses the sensor that carries the evidence, keeps the same target, and supports event-level answers with earlier evidence; RAVEN-Bench is built for this research audit, not for operational deployment.
Existing resources cover parts of this problem, but the paired EO–IR video reasoning setting remains largely untested. General video MLLM benchmarks evaluate temporal reasoning mainly in RGB natural videos. Remote-sensing vision-language benchmarks study captioning, grounding, referring, and VQA, with most evaluations centered on images [8, 9, 10]. EO–IR and multispectral aerial datasets show the value of cross-spectral sensing, with tasks usually framed as detection, tracking, segmentation, or sensor-fusion perception [11, 12, 13]. The missing setting is language-grounded reasoning over paired EO–IR aerial videos, where models must identify target-centric evidence, follow how it changes over time, and decide when EO, IR, or both streams are needed.
We introduce RAVEN-Bench (Reasoning over Aerial Videos with EO–IR aligNment), a curated diagnostic benchmark for paired EO–IR aerial video QA. RAVEN-Bench contains 100 EO–IR video pairs and 1,200 human-authored four-way multiple-choice QA items, with twelve questions per video pair. Every item is independently audited. Each sample centers on one target-centric aerial event and supports EO-only, IR-only, and paired EO+IR evaluation. As illustrated in Figure 1, a sample pairs synchronized EO and IR video observations with questions that move from local grounding to temporal-spatial reasoning and modality-aware event reasoning.
Figure 1. Overview of a RAVEN-Bench sample with synchronized EO–IR observations and representative L1–L3 four-way QA items. Labels indicate capability level and item-level modality requirement for analysis only, and are not provided as model input.
RAVEN-Bench focuses on language-grounded reasoning over paired EO–IR video evidence and provides no annotations or metrics for detection, tracking, segmentation, geolocation, targeting, or pixel-level registration; users with access could still create new annotations from the videos (Appendix § 13.3). The questions span three capability levels: grounding and cross-spectral evidence aggregation, temporal/spatial reasoning under modality shift, and modality-aware event reasoning. Related questions are grouped so that the same target or event is revisited across observations and ordered reasoning steps. This structure allows RAVEN-Bench to measure answer correctness together with unimodal reliance, paired-input gain beyond the stronger unimodal condition, stable target interpretation, and evidence-supported event reasoning.
Our contributions are summarized as follows:
We introduce RAVEN-Bench, a curated paired EO–IR aerial video QA benchmark with 100 EO–IR video pairs and 1,200 independently audited four-way QA items for target-centric aerial multimodal understanding.
We design an evidence-conditioned evaluation protocol with EO-only, IR-only, and paired EO+IR settings, enabling controlled evidence-condition slices and measurement of paired-input gain beyond the stronger unimodal input.
We develop a diagnostic structure with capability-balanced scoring, consistency groups, coherence groups, and cross-modal gain (, paired-input gain beyond the stronger unimodal condition), supported by audited annotations and evaluations of 26 MLLMs, including three remote-sensing VLMs.
Related Work
We compare prior resources by input format, sensor pairing, labels, correspondence controls, task, and scale (Table 1).
| Benchmark | Input | Sensor pairing | Labels | Corr. | Task | Scale |
| General video QA [14, 5, 15] | RGB video | – | QA, groupsa | – | MCQ | – |
| Long-video QA [22, 6, 7, 23] | RGB video | – | QA | – | VQA | – |
| RS VLMs [8, 9, 16, 17, 24, 25] | RS imageb | Multisensor | Cap., box, QA | – | Cap., VQA | – |
| EO–IR perception [13, 12] | EO–IR/MS | Paired | Box, mask, track | – | Det., seg., track | – |
| Traffic-VQA [18] | UAV image | Optical–thermal | QA | – | VQA | 8.2K / 1.3M |
| WildFireVQA [19] | UAV image | RGB–thermal | QA | – | MCQ | 6.1K / 207K |
| Thermo-VL-Bench [20] | Image | RGB–thermal | QA | – | Yes/no | 3.1K / 15K |
| RAVEN-Bench (ours) | EO–IR video | Synchronized | QA + metadatac | ✓ | MCQ, grouped | 100 / 1,200 |
Table 1. Comparison of RAVEN-Bench with related benchmark families and the closest optical–thermal QA resources.
Video understanding benchmarks. Benchmarks such as MVBench [14], Video-MME [5], MLVU [6], and LVBench [7] have been useful for testing temporal reasoning in RGB natural videos. Video-MME-v2 [15] additionally studies agreement across related questions and multistep reasoning chains. These benchmarks use RGB video only; RAVEN-Bench evaluates such agreement and reasoning chains in paired EO–IR aerial video, with controlled input conditions and correspondence tests.
Remote-sensing VLMs and benchmarks. GeoChat [8], EarthGPT [10], and VRSBench [9] broadened language-grounded aerial understanding through remote-sensing captioning, grounding, and VQA. SkyEyeGPT [16] unifies image- and region-level remote-sensing tasks and also covers aerial video captioning. EarthDial [17] handles multisensor, multitemporal, and multi-resolution Earth-observation imagery, including RGB, SAR, near-infrared, and infrared bands. None of them evaluates paired EO–IR video under controlled EO-only, IR-only, and EO+IR inputs.
Optical–thermal VQA. Traffic-VQA [18] pairs 8,180 aligned optical and thermal UAV images with about 1.3M QA pairs. WildFireVQA [19] provides 6,097 aerial RGB–thermal samples with radiometric thermal data and 207,298 multiple-choice questions. Thermo-VL-Bench [20] evaluates pixel-aligned RGB–thermal image pairs under RGB-only, thermal-only, and RGB+thermal inputs. These resources offer larger optical–thermal QA sets, calibrated thermal data, and paired-image evaluation, but all operate on still image pairs. RAVEN-Bench studies EO and IR evidence that changes within one event over time.
EO–IR and multispectral perception. Datasets such as TIV [11], KAIST [21], IndraEye [13], and MUST [12] cover EO–IR and multispectral sensing under difficult visibility conditions. They target perception tasks such as detection, tracking, segmentation, and sensor fusion. RAVEN-Bench uses no spatial or trajectory labels, and its Consistency score does not measure tracking.
Table 1 summarizes these differences: RAVEN-Bench combines paired EO–IR video with controlled inputs, correspondence controls, and grouped measures of target consistency and evidence-chain coherence.
RAVEN-Bench
RAVEN-Bench contains 100 paired EO–IR aerial observations and 1,200 four-way QA items designed for target-centric, temporal, and modality-aware reasoning. The benchmark task is language-grounded reasoning over paired EO–IR video evidence; RAVEN-Bench provides no annotations or metrics for Corr.: EO–IR correspondence controls (shuffled and temporally misaligned pairs). MS: multispectral. aVideo-MME-v2 only. bSome add multisensor or aerial video inputs. cModality requirement, evidence interval, and group labels; no spatial labels. Scale: image or video pairs / QA items, listed for the closest paired optical–thermal resources.
detection, tracking, segmentation, geolocation, targeting, or pixel-level registration. Section 4 defines the task design and diagnostic metrics.
Video Source and Curation
The videos come from an ongoing, weekly airborne collection program of TerraSense Analytics, which owns the footage and authorized its release for research evaluation; several authors are affiliated with it. The footage was captured with a WESCAM MX-10 EO/IR gimbal on a Vulcanair P68C aircraft in western North America. Exact dates, coordinates, flight paths, platform and sensor identifiers, and KLV and other operational metadata are withheld, and no third-party copyrighted video is included. The gated release includes evaluation code, Croissant metadata, a dataset card, and research-use terms (Appendix § 13.3).
Scene families. The retained events cover four families by primary target: ground transport and machinery (road vehicles, trains, industrial machinery, cyclists, and static vehicle displays), maritime vessels, aircraft (including seaplanes), and people. Each family stresses EO and IR evidence differently: maritime events contain glare, wakes, and similar-looking vessels; ground transport and machinery events contain small or partly hidden targets against warm pavement; aircraft events add large changes in viewpoint and scale. The people events contain no identifiable individuals (Appendix § 13.3). These families define evaluation scenes and carry no deployment recommendation.
We verify EO–IR pairing at the event level. Annotators check that the same target-centric event appears in the overlapping EO and IR observations, remove clips with severe temporal or field-of-view mismatch, and treat residual differences in viewpoint, scale, sensor response, and overlap as part of the benchmark difficulty. As a result, RAVEN-Bench evaluates cross-spectral reasoning under realistic non-identical viewpoints, with no assumption of registered EO–IR pixels.
We retain clips that contain a target-centric event, event-level EO–IR correspondence, and sufficient visual evidence for at least one evaluation setting. We remove clips with missing streams, severe blur or instability, ambiguous event boundaries, insufficient target evidence, sensitive metadata, or personally identifiable information. Retained clips are manually screened for identifiable people and readable identifiers; Appendix § 13.3 gives the screening and redaction rules. Section 4 formalizes the evidence conditions used for evaluation.
QA Construction and Quality Control
For each EO–IR pair, we construct twelve four-way multiple-choice QA items around the same target-centric event. This gives each event equal weight while probing target perception, temporal dynamics, modality selection, and event-level reasoning. Each item includes a ground-truth answer and diagnostic metadata used only for scoring, including modality requirement, capability level, reasoning tags, group family, and group step.
QA construction is human-authored, AI-assisted, and independently audited. Human annotators inspect the paired videos and write questions against the benchmark taxonomy. AI assistance is limited to wording refinement, option clarity, distractor review, and ambiguity flagging. It is not used to determine visual facts, answers, modality requirements, or capability/group labels.
Quality control filters out unsupported or ambiguous items, modality-label mismatches, language-prior shortcuts, and questions that depend on pixel-level registration, fine-grained geolocation, platform parameters, or unavailable sensor details. Grouped questions are kept only when they refer to the same target-centric event and support the diagnostics in Section 4. Screening categories, full guidelines, and metadata are provided in Appendix § 9.3 and Appendix § 9.2.
After these standard checks, a second, independent audit covers all 1,200 items: auditors answer each item without hidden annotations, then check the answer key, required sensor, evidence interval, distractors, shortcut risk, and group validity, with a third reviewer resolving disagreements. The audit yields 96.5% answer-key, 94.4% modality, and 93.6% group-validity agreement. All results use the audited annotations (Appendix § 9.4).
Dataset Statistics
Figure 2 summarizes the main dataset statistics. RAVEN-Bench contains 200 modality-specific videos organized into 100 EO–IR pairs. The total modality-stream duration is 17,939.0 seconds (about 5 hours), with 9,027.2 seconds from EO and 8,911.8 seconds from IR, giving a mean stream duration of 89.7 seconds over the 200 single-modality streams.

Figure 2. Dataset statistics for RAVEN-Bench. Panels show (a) scenario families over the 100 EO–IR pairs, (b) annotated evidence requirements, (c) capability levels, and (d) the 11 most frequent of 20 reasoning tags; (b)–(d) are computed over the 1,200 questions, and the mean duration over the 200 single-modality streams. Percentages in (d) need not sum to 100%.
By primary target, the 100 pairs comprise 59 ground transport and machinery, 27 maritime, 11 aircraft, and 3 people events. Together, they carry 94 distinct fine-grained scenario labels across land, water, and airfield settings. At the annotation level, the 1,200 four-way QA items are labeled by modality requirement, capability level, and reasoning tags for EO-only, IR-only, and paired EO+IR evaluation.
Evaluation Task Design
RAVEN-Bench tests whether MLLMs can reason over paired aerial EO–IR videos in ways that are target-centric, temporal, and sensitive to the sensing modality. The annotation metadata defined in Section 3 specifies the evidence conditions, capability levels, and group structure used by the diagnostic protocol. Raw question-level accuracy is a weak signal in this setting because the decisive evidence may be small, brief, modality-specific, or distributed across the EO and IR streams.
The evaluation focuses on three questions: whether the model keeps a stable interpretation of the same target, whether event-level answers are supported by earlier evidence, and whether paired EO+IR input improves over the stronger unimodal stream. Consistency groups, coherence groups, and modality-conditioned inputs provide these readouts, with capability levels and evidence-condition slices used for stratified analysis. Figure 3 summarizes this target–probe–metric design.

Figure 3. Diagnostic evaluation design. RAVEN-Bench organizes a paired EO–IR event into a shared QA set and reads out three diagnostic signals: target consistency from consistency groups, evidence-chain coherence from coherence groups, and paired-input gain from EO-only, IR-only, and EO+IR settings. Capability levels and evidence-condition slices are used for balanced scoring and slice-level analysis.
Task Formulation
Each RAVEN-Bench sample consists of a paired EO–IR video observation and twelve four-way multiple-choice QA items built around the same paired event. Given a visual input and a question , a model predicts one answer option:
The input is provided in one of three modality-conditioned settings: the EO stream, the IR stream, or the paired EO+IR streams. All three settings use the same exported question set. Question-level modality requirements are used only after inference to compute diagnostic slices; they do not change the question text or the answer options shown to the model.
Models receive only the visual input, the natural-language question, and the answer options. Capability labels, modality requirements, group identifiers, reasoning skill tags, and other annotations are reserved for scoring and analysis, and are not provided during inference. RAVEN-Bench operates at the video-QA level, without pixel-level registration or sensor-fusion supervision. The EO and IR videos are paired observations of the same aerial event, and models are evaluated on whether they can identify relevant evidence, track target dynamics, and reason over modality-dependent observations.
Diagnostic Structure
We next describe how the annotations and input settings instantiate these diagnostics.
Structured question groups. Beyond individual QA items, RAVEN-Bench uses two group types tied to the first two diagnostic targets. A consistency group is unordered and contains related questions about the same target or event, so it tests whether a model gives compatible answers across related observations. A coherence group is ordered as a short reasoning chain: earlier questions verify intermediate evidence, and later questions ask for conclusions that depend on that evidence. The two group types use different scores because stable target interpretation and evidence-supported event reasoning expose different failure patterns. In the retained benchmark, every QA item belongs to either a consistency group or a coherence group. Level-wise and main-score metrics are still computed over individual QA items, and the scorer also supports ungrouped items for future extensions.
Modality-conditioned inputs. To test paired-input benefit, each model is evaluated under EO-only, IR-only, and EO+IR settings using the same question set. In the EO-only setting, the model receives only the electro-optical stream. In the IR-only setting, it receives only the infrared stream. In the EO+IR setting, it receives both paired streams with modality labels. All settings use 16 synchronized event-relative timestamps per modality: EO-only and IR-only inputs contain 16 frames, and EO+IR inputs contain the matched EO and IR frames at each timestamp, up to 32 frames. Input settings specify which streams are provided to the model. Evidence conditions describe which modality contains the decisive evidence for a question or evaluation slice. We use modality requirement as the item-level annotation field and evidence condition as the evaluation slice induced by that field. This design allows us to compare visible-spectrum evidence use, thermal evidence use, and measurable benefit from paired EO+IR input under matched questions. Sampling, prompt templates, input formatting, and parsing details are in Section 5 and Appendix C.1.
Stratification metadata. Capability levels, evidence conditions, and reasoning tags serve as a balancing and analysis layer. Level 1 covers grounding and cross-spectral evidence aggregation, Level 2 covers temporal/spatial reasoning under modality shift, and Level 3 covers modality-aware event reasoning. These levels define the macro-averaged main score and the capability gap. Evidence conditions and reasoning tags support slice-level analyses without changing model prompts.
Metrics
All questions have four answer choices, which enables deterministic scoring. We use the diagnostics above for capability-balanced, group-level, and modality-conditioned analysis.
Main score. Let , , and denote the level-wise scores for the three capability levels. Each level score is computed as the average question accuracy over items assigned to that level. The main benchmark score is the macro-average:
This prevents a single level from dominating the aggregate score. We report it under EO-only, IR-only, and EO+IR inputs as , , and .
Consistency score. For a consistency group with questions and correct predictions:
The squared form penalizes fragmented correctness while preserving partial credit. We report as the macro-average over consistency groups.
Coherence score. For a coherence group , questions are ordered by reasoning step. We use coherence to mean prefix correctness over ordered evidence chains. Let be the number of consecutive correct answers from the first step before the first error. We define
Later correct answers do not compensate for earlier failures because the final conclusion lacks required intermediate support. We report as the macro-average over coherence groups.
Cross-modal gain. We measure paired EO+IR benefit beyond unimodal evidence as
Positive indicates paired-input improvement over the stronger unimodal condition. Near-zero or negative values are consistent with unimodal reliance, input-format limitations, or failure to use paired evidence. We report globally and on EO–IR evidence-condition slices. Because is an aggregate difference, it indicates paired-input gain and should not be read as proof of cross-modal integration.
Derived diagnostic gaps. We also report two diagnostic gaps. The capability gap
measures degradation from local grounding to modality-aware event reasoning. The coherence gap
measures how much paired-input main performance overestimates ordered evidence-chain coherence. Unless otherwise specified, these diagnostics are computed under paired EO+IR input. Appendix § 10 gives additional aggregation details.
Uncertainty. We treat small point-estimate differences descriptively and avoid significance claims based only on isolated score gaps. Appendix § 10.2 specifies the clustered bootstrap procedure used by the evaluation code. Question-level and level-wise metrics are resampled at the EO–IR video-pair level, while group metrics are resampled at the group level. This procedure avoids treating correlated QA items from the same paired observation as independent samples.
Experiments
Experimental Setup
We evaluate frontier API models, recent open-weight video-capable MLLMs, three remote-sensing VLMs (UniRS [26], SkyEyeGPT [16], and EarthDial [17]), and non-visual controls in a zero-shot four-way multiple-choice setting. Open-weight models are grouped by total parameter scale. For MoE models, we use total parameters rather than activated parameters.
All models use the same EO-only, IR-only, and EO+IR settings, question set, option order, prompt template, and temporal sampling protocol (16 synchronized event-relative timestamps per modality). Each model is asked to return one option from A–D. We parse the first valid option deterministically and count invalid outputs as incorrect. The constrained evidence budget is used to isolate diagnostic behavior; serving efficiency and native-video utilization are outside the comparison. Appendix § 11 gives prompts, sampling, formatting, parsing, and scoring details, while Appendix § 12 reports controls and human-reference results.
Main Results
Table 2 reports the main RAVEN-Bench results. The model groups are chosen to cover frontier API systems, open-weight checkpoints, hosted open-weight systems, and remote-sensing VLMs. We use these results to study paired-input gain, capability imbalance, and grouped consistency/coherence, rather than to produce a single leaderboard.
| Model | |||||||||||
| Frontier API models | |||||||||||
| GPT-5.4 [27] | 45.2 | 41.6 | 46.1 | 0.9 | 51.8 | 46.7 | 39.8 | 12.0 | 30.6 | 22.8 | 23.3 |
| Gemini 3.1 Pro [28] | 46.2 | 44.9 | 49.6 | 3.4 | 52.2 | 52.2 | <u>44.4</u> | 7.8 | <u>36.8</u> | <u>30.2</u> | 19.4 |
| Gemini 3 Flash [29] | 40.7 | 43.3 | 42.6 | -0.7 | 46.1 | 43.3 | 38.4 | 7.7 | 25.4 | 20.3 | 22.3 |
| Open-weight models, model size B | |||||||||||
| SmolVLM2-2.2B-Instruct [30] | 30.4 | 33.1 | 31.7 | -1.4 | 36.8 | 30.7 | 27.6 | 9.2 | 13.2 | 8.3 | 23.4 |
| Qwen3-VL-2B-Instruct [31] | 35.4 | 37.2 | 38.4 | 1.2 | 41.9 | 39.8 | 33.5 | 8.4 | 18.6 | 13.8 | 24.6 |
| VideoLLaMA3-2B [32] | 31.1 | 33.5 | 33.9 | 0.4 | 35.8 | 34.3 | 31.6 | 4.2 | 16.7 | 14.5 | 19.4 |
| Video-XL-Pro-3B [33] | 32.6 | 34.8 | 35.7 | 0.9 | 37.9 | 35.8 | 33.4 | 4.5 | 20.3 | 17.1 | 18.6 |
| Open-weight models, B model size B | |||||||||||
| Qwen3.5-9B-Instruct [34] | 39.7 | 39.4 | 43.3 | 3.6 | 45.3 | 46.4 | 38.2 | 7.1 | 23.5 | 18.2 | 25.1 |
| Qwen3-VL-8B-Instruct [31] | 41.9 | 38.7 | 43.3 | 1.4 | 46.6 | 44.5 | 38.8 | 7.8 | 24.7 | 19.7 | 23.6 |
| InternVL3.5-8B-Instruct [35] | 38.8 | 40.5 | 39.9 | -0.6 | 43.2 | 40.6 | 35.9 | 7.3 | 20.7 | 16.2 | 23.7 |
| MiniCPM-V 4.5 [36] | 37.8 | 37.5 | 41.7 | 3.9 | 44.4 | 41.8 | 38.9 | 5.5 | 24.5 | 19.5 | 22.2 |
| Molmo2-8B [37] | 35.1 | 41.8 | 41.2 | -0.6 | 47.9 | 40.1 | 35.6 | 12.3 | 21.9 | 15.1 | 26.1 |
| Keye-VL-1.5-8B [38] | 38.6 | 38.9 | 43.4 | 4.5 | 46.1 | 43.3 | 40.8 | 5.3 | 26.1 | 21.5 | 21.9 |
| MiMo-VL-7B-RL-2508 [39] | 40.2 | 40.8 | 41.9 | 1.1 | 44.7 | 43.5 | 37.5 | 7.2 | 23.8 | 17.8 | 24.1 |
| VideoLLaMA3-7B [32] | 39.7 | 37.8 | 42.4 | 2.7 | 44.8 | 41.6 | 40.8 | 4.0 | 25.2 | 21.9 | 20.5 |
| InternVideo2.5-Chat-8B [40] | 37.6 | 39.8 | 43.5 | 3.7 | 45.0 | 43.6 | 41.9 | <u>3.1</u> | 27.6 | 23.8 | 19.7 |
| mPLUG-Owl3-7B [41] | 37.1 | 36.6 | 37.7 | 0.6 | 41.7 | 40.1 | 31.3 | 10.4 | 17.9 | 12.5 | 25.2 |
| LLaVA-Video-7B-Qwen2 [4] | 29.3 | 27.6 | 29.8 | 0.5 | 31.8 | 31.5 | 26.1 | 5.7 | 8.4 | 6.9 | 22.9 |
| Open-weight models, B model size B | |||||||||||
| Kimi-VL-A3B-Instruct [42] | 41.7 | 40.2 | 42.1 | 0.4 | 48.8 | 41.0 | 36.5 | 12.3 | 23.2 | 15.5 | 26.6 |
| Qwen3-VL-30B-A3B-Instruct [31] | 41.6 | 43.3 | 45.9 | 2.6 | 48.5 | 47.9 | 41.3 | 7.2 | 28.9 | 23.4 | 22.5 |
| InternVL3.5-30B-A3B [35] | 40.4 | 42.9 | 43.8 | 0.9 | 46.8 | 44.9 | 39.7 | 7.1 | 27.3 | 22.1 | 21.7 |
| Hosted open-weight models, model size B | |||||||||||
| Qwen3-VL-235B-A22B-Instruct [31] | 44.2 | 44.1 | <u>47.8</u> | 3.6 | 50.0 | <u>50.3</u> | 43.1 | 6.9 | 33.4 | 27.9 | 19.9 |
| InternVL3.5-241B-A28B [35] | 43.0 | <u>44.8</u> | 45.5 | 0.7 | 47.9 | 46.8 | 41.8 | 6.1 | 32.7 | 27.1 | <u>18.4</u> |
| Remote-sensing VLMs | |||||||||||
| UniRS [26] | 43.2 | 38.6 | 47.0 | 3.8 | 46.2 | 50.2 | 44.6 | 1.6 | 37.2 | 33.4 | 13.6 |
| SkyEyeGPT [16] | 38.4 | 34.2 | 39.1 | 0.7 | 41.8 | 39.6 | 35.9 | 5.9 | 24.1 | 19.8 | 19.3 |
| EarthDial [17] | 40.5 | 37.9 | 42.3 | 1.8 | 44.3 | 43.1 | 39.5 | 4.8 | 28.4 | 23.6 | 18.7 |
Table 2. Main diagnostic results. Higher is better except for and ; bold and <u>underlined</u> mark best and second-best entries within this constrained evidence-budget protocol, not absolute model capability across serving interfaces.
Finding 1: aggregate performance overestimates grouped evidence reliability. Across the 26 evaluated models, is lower than by 16.9 points on average, and is lower by 21.8 points. Every model in Table 2 has a positive gap on both grouped diagnostics. Finding 3.1 Pro [28], for example, drops from 49.6% to 36.8% and 30.2% . UniRS [26] has the strongest grouped scores even though it does not have the highest aggregate accuracy. These results indicate that grouped reliability captures failures that isolated question-level accuracy can hide.
Finding 2: paired EO+IR gains require controlled interpretation. We interpret together with input-perturbation controls, since a positive paired-input gain does not by itself establish sensor fusion. varies substantially across models: Keye-VL-1.5-8B [38], MiniCPM-V 4.5 [36], and UniRS [26] show the largest positive point-estimate gains, while SmolVLM-2B [30], Molmo2-8B [37], and Gemini 3 Flash [29] show no positive gain. We treat positive values as directional paired-input improvements, with sensor-fusion claims left to the control analyses. In the control subset, aligned EO+IR reaches and . Duplicated EO+EO/IR+IR inputs yield smaller gains (+0.7, +0.3) and lower coherence (20.6, 21.1), while shuffled or temporally misaligned EO+IR reduces gain below the stronger unimodal baseline. These controls show that temporal and cross-spectral correspondence matter under this protocol.
Finding 3: local evidence performance is a weak proxy for event reasoning. The capability gap measures how much performance drops from local grounding to modality-aware event reasoning. Kimi-VL-A3B [42] and Molmo2-8B [37] have high Level 1 scores but . UniRS [26] has the smallest gap (1.6) despite not having the highest aggregate EO+IR score.
Evidence-Condition Diagnostics
Figure 4 analyzes representative models across EO-favorable, IR-favorable, and EO+IR-required evidence conditions. This slice-level view tests whether models respond to the modality that contains the decisive evidence and whether paired-input gains appear when both streams are needed.
In the EO-favorable and IR-favorable slices, the annotated favorable modality separates the two unimodal inputs: EO-only exceeds IR-only on EO-favorable items, while IR-only exceeds EO-only on IR-favorable items. Paired EO+IR stays within about one point of the favorable unimodal input, since it includes that stream plus additional context. In the EO+IR-required slice, paired EO+IR exceeds both unimodal inputs for all representative models, giving the clearest slice-level evidence for paired-input gain. UniRS [26] obtains the largest gain (), followed by Gemini 3.1 Pro [28] (6.8) and Qwen3-VL-235B [31] (5.4). LLaVA-Video-7B-Qwen2 [4] shows little benefit (0.3). These slices help distinguish unimodal skill, modality arbitration, and paired-input gain.
Validity and Robustness Controls
Appendix § 12 reports calibration, text-only, human-reference, input-perturbation, and frame-budget controls. The non-visual controls remain low: the best text-only setting reaches and , while the human-reference subset reaches 82.3 and 69.6 as an answerability calibration with retained EO–IR clips. The main paired-input control result follows the same pattern as the main evaluation. Aligned EO+IR gives the strongest control gain (); duplicated EO+EO/IR+IR inputs give much smaller gains (+0.7, +0.3); and shuffled or temporally misaligned EO+IR falls below the stronger unimodal baseline. Increasing the synchronized timestamp budget from to changes only from 43.5 to 44.1 and from 23.4 to 23.9. These controls support aligned EO–IR correspondence as the diagnostic condition while keeping claims about native-video utilization and sensor fusion bounded. A free-form variant with human-judged answers and evidence statements keeps the same ordering (Main 34.8, Consistency 24.6, Coherence 19.3, Answer+Evidence 27.1; Appendix § 12.4), so the grouped gaps do not depend on the multiple-choice format.
Observed Error Mechanisms
A manual audit identifies four recurring error mechanisms: aerial-platform ego-motion mistaken as target motion, tiny-target loss after resizing or sparse sampling, EO/IR distractors such as glare or hot backgrounds, and failed binding of complementary EO and IR evidence. The first two primarily disrupt temporal tracking and ordered coherence, while the latter two expose failures to select and bind the correct sensor evidence for the same target. Examples are in Appendix § 13.1.
Conclusion
RAVEN-Bench brings event-level EO–IR pairing, controlled input settings, consistency and coherence groups, and evidence-condition slices into a single video-QA evaluation. The results show that aggregate accuracy can mask unstable target interpretation, unsupported event-level answers, and limited paired-input benefit beyond the stronger unimodal stream. We recommend reporting grouped reliability, capability gaps, and modality-conditioned slices alongside aggregate accuracy. The gated release includes the evaluation code, Croissant metadata, and a dataset card; future versions will extend coverage while keeping the input conditions, correspondence controls, and grouped metrics fixed so that results remain comparable.
Limitations and responsible use. All videos come from one aircraft, one EO/IR gimbal, and one region in western North America, so the scores describe this sensing setup; condition subsets (weather, altitude, viewing geometry) remain too small for robustness comparisons. Paired EO–IR aerial video is dual-use. RAVEN-Bench provides no spatial, trajectory, geolocation, or flight-path data; access is gated, and the data release terms limit use to research and non-commercial purposes and prohibit re-identification and the surveillance, tracking, or profiling of specific individuals. These safeguards cannot prevent every misuse after download; Appendix § 13.3 details the remaining risk and the maintenance plan.
Acknowledgments and Disclosure of Funding
This work was supported by the Mitacs Accelerate Program (Grant No. IT42711), by TerraSense Analytics, and in part by grants from the US National Science Foundation (OAC-2118240, HDR Institute: Imageomics). We thank the University of British Columbia’s Survive and Thrive Applied Research (UBC STAR) initiative for its support of this work. This work uses data provided by TerraSense Analytics Ltd. (www.terrasense.ca).
References
Dataset Documentation and Annotation Protocol
This appendix provides the dataset-level information needed to audit RAVEN-Bench and reproduce its diagnostic evaluation. The benchmark consists of paired EO–IR aerial video observations and four-way multiple-choice QA items. Metadata are used for dataset organization, quality control, and scoring; they are not provided to models during inference.
Dataset Composition and Annotation-Derived Coverage
Table 3 summarizes the final benchmark composition, and Table 4 reports annotation-derived coverage over the retained evaluation set. RAVEN-Bench is intentionally compact and diagnostic rather than class-balanced at web scale: each retained EO–IR video pair contributes the same number of QA items, and the annotation metadata supports controlled slices by modality requirement, capability level, and group structure. The released annotation files document the retained benchmark; raw-pool rejection counts are therefore not reported because they require a separate screening log rather than the retained-set annotation package.
| Statistic | Value |
| EO–IR video pairs | 100 |
| Modality-specific video streams | 200 |
| QA items | 1,200 |
| Questions per video pair | 12 |
| Answer options per question | 4 |
| Total modality-stream duration | 17,939.0 s |
| EO stream duration | 9,027.2 s |
| IR stream duration | 8,911.8 s |
| Mean stream duration (200 streams) | 89.7 s |
| Scenario-family distribution | 59 ground transport and machinery / 27 maritime / 11 aircraft / 3 people |
| Fine-grained scenario labels (distinct type values) | 94 |
| Question-type tags | 20 tags / 1,963 assignments |
Table 3. Dataset composition of RAVEN-Bench. Durations sum over the 200 single-modality streams.
| Audit item | Annotation-derived count |
| Retained annotation files | 100 files |
| QA items with four answer options and one answer key | 1,200 QA items |
| Capability-level coverage | L1: 386; L2: 560; L3: 254 |
| Modality-requirement coverage | EO: 656; IR: 229; EO+IR: 315 |
| Grouped-question metadata | 1,200 QA items include group identifiers, group family, and group step |
| Consistency groups | 100 groups / 800 QA items / 8 QA items per group |
| Coherence groups | 100 groups / 400 QA items / 4 QA items per group |
| Temporal evidence references | 1,200 QA items include at least one EO or IR time reference (971 with EO, 544 with IR, 315 with both) |
Table 4. Annotation-derived retained-set coverage. Counts are computed from the 100 JSON annotation files in the retained benchmark.
Metadata Schema
RAVEN-Bench uses lightweight metadata for auditing and diagnostic scoring. Table 6 lists sample-level annotation fields, and Table 7 lists QA-level fields. Fields that expose event summaries, answer keys, option roles, rationales, or time references are used for auditing and scoring only; they are not provided to models during inference.
| Field | Type | Description |
sample_id | string | Unique identifier for the EO–IR video pair. |
type | categorical | Released fine-grained retained scenario label used for coverage analysis. |
main_event | text | Internal event descriptor used during annotation and auditing. |
event_description | text | Internal summary of the target-centric event. |
scenario_type | categorical | Legacy internal annotation descriptor used during QA construction; not used as an evidence-condition label or scoring slice. |
notes | object | Annotation warnings, design notes, and acceptance gates for prompt-leakage and ambiguity checks. |
qa | list | The twelve QA annotations associated with the video pair. |
Table 6. Sample-level annotation schema. These fields organize the retained EO–IR video pairs and are not model input.
| Field | Type | Description |
uid | string | Unique identifier for the QA item within a sample. |
group_id | string | Identifier for the consistency or coherence group. |
group_family | categorical | consistency or coherence. |
group_focus | text | Short internal descriptor of the group target or evidence pattern. |
group_step | integer | Ordered step index used for grouped scoring. |
capability_level | categorical | L1, L2, or L3. |
question | text | Natural-language question. |
options | object | Four answer options labeled A–D. |
answer | categorical | Ground-truth answer option. |
option_roles | object | Internal record of correct and distractor roles. |
question_type | list | Multi-label reasoning tags used for dataset analysis. |
time_reference_eo | text | Relevant EO temporal interval, when applicable. |
time_reference_ir | text | Relevant IR temporal interval, when applicable. |
modality_requirement | categorical | Item-level modality requirement, i.e., the required evidence modality: EO, IR, or EO+IR. |
Table 7. Question-level metadata schema. These fields support scoring, slicing, and annotation auditing; they are not provided to models during inference.
The field type is the released fine-grained scenario label used for retained-set coverage analysis. The field scenario_type is retained only as an internal annotation descriptor and should not be confused with item-level modality_requirement, which defines the evidence-condition slices used for scoring.
| Question-type tag | Count | QA items (%) |
spatial_reasoning | 350 | 29.17 |
temporal_reasoning | 348 | 29.00 |
entity_grounding | 322 | 26.83 |
cross_modal_phenomenon_explanation | 187 | 15.58 |
camera_motion_vs_target_motion | 119 | 9.92 |
thermal_evidence_interpretation | 115 | 9.58 |
evidence_sufficiency | 91 | 7.58 |
cross_modal_correspondence | 81 | 6.75 |
ir_contrast_grounding | 76 | 6.33 |
spatial_temporal_reasoning | 75 | 6.25 |
group_verdict | 60 | 5.00 |
trajectory_reasoning | 40 | 3.33 |
temporal_event_state_account | 38 | 3.17 |
scene_relation | 31 | 2.58 |
identity_continuity | 10 | 0.83 |
distractor_rejection | 7 | 0.58 |
scene_grounding | 5 | 0.42 |
causal_reasoning | 4 | 0.33 |
evidence_retrieval | 3 | 0.25 |
occlusion_reasoning | 1 | 0.08 |
Table 5. Question-type tag distribution over the 1,200 QA items (20 tags, 1,963 assignments). Percentages use the 1,200 items as the denominator; tags are multi-label, so percentages need not sum to 100%.
Annotation Protocol and Independent Audit
The annotation workflow is human-authored, AI-assisted, and independently audited. Human annotators inspect paired EO and IR videos, identify visual evidence, and write candidate questions and answer options. AI assistance is restricted to wording refinement, option clarity, distractor review, and ambiguity flagging; it is not used to determine visual facts, correct answers, modality requirements, capability labels, or group assignments. All accepted items are checked against the EO–IR video evidence before inclusion.
Annotation labor is organized by role rather than crowdsourced at scale. Retained QA construction was completed by three trained annotators with computer-vision or remote-sensing familiarity; before authoring, they reviewed the benchmark taxonomy and calibrated on pilot EO–IR clips covering EO-only, IR-only, and EO+IR-required evidence. The independent audit covers all items and is conducted by reviewers who did not author the audited items (Appendix § 9.4). The human-reference evaluation in Appendix § 12.1 uses a stratified 120-item subset, with separate instructions and scoring: audit reviewers check answer keys, modality requirements, and group validity, whereas human-reference annotators answer model-facing questions using retained EO–IR clips without hidden metadata. Annotation and audit labor was performed by authors or institutionally supervised research staff as part of paid research roles; no unpaid crowdsourcing was used.
Annotators follow five rules: each item must be visually answerable from the provided evidence; each item must have exactly one correct option; distractors must be plausible but visually incorrect; the relevant temporal evidence must remain identifiable despite platform motion and target-scale variation; and items must not require exact pixel-level registration, fine-grained geolocation, platform parameters, or unavailable sensor details.
Table 8 summarizes the screening categories used during annotation review. The retained annotation package records accepted items and required metadata, but the screening log was not maintained as a countable rejection ledger; we therefore report categories rather than rejection counts.
| Screening category | Rejection or revision rationale |
| Unsupported visual evidence | The proposed answer cannot be verified from the supplied EO or IR video evidence. |
| Multiple plausible answers | More than one option remains visually defensible after reviewing the relevant time interval. |
| Modality requirement mismatch | The annotated required modality does not match where the decisive evidence appears. |
| Language-only shortcut | The answer can be inferred from wording, option priors, or commonsense without visual evidence. |
| Pixel-registration requirement | The item requires exact EO–IR pixel alignment rather than event-level correspondence. |
| Unavailable sensor or platform metadata | The item depends on geolocation, sensor parameters, flight path, altitude, or platform details not released for evaluation. |
| Sensitive metadata risk | The item or rationale exposes operational, identifying, or unnecessary collection details. |
Table 8. Annotation screening categories. These categories were used to reject or revise candidate QA items before inclusion; counts are not reported because the retained release package is not a rejection log.
As summarized in Table 4, every QA item includes a capability level, modality requirement, group identifier, group family, group step, answer key, four answer options, question-type tags, and at least one EO or IR time reference. The notes field in each annotation file also records prompt-leakage safeguards: event descriptions, answer keys, option roles, rationales, and time-reference metadata are used for scoring and auditing but are not exported to model prompts.
Full Independent Audit and Capability Assignment
Full audit. Every one of the 1,200 items is independently audited after the standard checks in § 3.2. A reviewer first answers each item and locates its evidence without hidden annotations, then checks the answer key, required sensor, evidence interval, distractors, shortcut risk, and group validity. A third reviewer resolves disagreements. Table 9 reports the results.
| Audit result | Value |
| Coverage | 100/100 pairs; 1,200/1,200 items |
| Answer-key agreement | 96.5% |
| Modality agreement | 94.4% |
| Group-validity agreement | 93.6% |
| Wording or distractor revisions | 114 items (9.5%) |
| Answer-key corrections | 42 items (3.5%) |
| Modality corrections | 67 items (5.6%) |
| Evidence-interval corrections | 53 items (4.4%) |
| Group corrections | 77 items (6.4%) |
| Removed or replaced items | 19 items (1.6%) |
Table 9. Full independent audit of the 100 EO–IR pairs and 1,200 QA items.
Capability assignment. Human annotators assign every item to L1, L2, or L3 with a single item-level rubric, without per-level quotas. The resulting distribution is 386 L1, 560 L2, and 254 L3 items (Table 4).
Construction procedure. All pairs share the same question structure, annotation format, review procedure, metrics, and scoring code, and were constructed by the annotator team described in Appendix § 9.3. Event selection prioritized target categories and capture conditions that would otherwise be rare. Independent evidence review is the main time cost. A release of thousands of events would require a larger annotation team and tools for routine annotation checks, and human review of the visual evidence would remain part of the process.
Representative QA Examples
Table 10 gives representative examples from released or anonymized samples. The examples are included to clarify the intended distinction between EO-only, IR-only, EO+IR-required, consistency-group, and coherence-group questions. The question texts shown in the table are retained model-facing question texts rather than hidden-metadata summaries. Table columns such as type, level, evidence interval, and group label are explanatory appendix metadata only; models receive only the visual input, the question, and the four answer options. Wording such as “chain” or “unsafe” is used to ask for visually grounded temporal comparison or to avoid overclaiming an event interpretation, and does not expose capability labels, modality requirements, group membership, answer keys, option roles, or rationales.
| Type | Level | Question or group item | Correct option / evidence |
| EO-only | L1 | In the first close EO pass, which ground relation places the aircraft? | A; EO 00:17–00:29. |
| IR-only | L1 | In the close IR pass, where is the bright pavement relative to the aircraft body? | A; IR 00:28–00:52. |
| EO+IR-required | L3 | Which cross-modal chain avoids the paint-color shortcut? | A; EO 00:17–02:12 and IR 00:28–01:25. |
| Consistency group | L1/L2 | TerraSense_3 C1: eight grouped questions about orbit-induced screen-side changes and EO–IR placement cues. | Annotation answer keys for steps 1–8; EO and IR intervals vary by item. |
| Coherence group | L2/L3 | TerraSense_3 H1: four ordered questions from unsafe EO motion interpretation to cross-modal full-clip reading. | Annotation answer keys for steps 1–4; EO 00:17–02:12 and IR 00:28–01:25. |
Table 10. Representative QA examples from the retained annotations. Examples are drawn from TerraSense_3.json.
Group-Level Annotation Examples
Table 11 gives an aircraft consistency group, and Table 12 gives a maritime coherence group. The consistency group keeps the same target fixed while varying EO evidence, IR evidence, temporal scale changes, and cross-modal cue transfer. The coherence group is ordered: early steps verify supporting EO and IR evidence, and later steps ask for cross-modal and full-clip conclusions. These examples are used for compact illustration; analogous consistency and coherence groups are included for maritime, ground-vehicle, and aircraft samples in the released annotations.
| Step | Req. | Level | Question | Consistency role |
| 1 | EO | L1 | In the first close EO pass, which ground relation places the aircraft? | Grounds the target in EO using nearby layout. |
| 2 | IR | L1 | In the close IR pass, where is the bright pavement relative to the aircraft body? | Checks matching target placement in IR. |
| 3 | EO | L1 | In the later EO pass after the orbit, which screen-side layout is visible? | Rechecks placement after viewpoint change. |
| 4 | EO | L2 | Which EO sequence describes the pavement shifting around a grass-side aircraft? | Separates camera orbit from target travel. |
| 5 | IR | L2 | Which IR sequence tracks the aircraft through the scale change? | Tests IR persistence of the same target. |
| 6 | EO+IR | L2 | Which EO detail becomes a weak direct cue in IR? | Tests whether an EO cue transfers poorly to IR. |
| 7 | EO+IR | L3 | Which chain keeps the aircraft off pavement in both modalities? | Requires a stable cross-modal target interpretation. |
| 8 | EO+IR | L3 | Which cue argues against a taxi-along-pavement interpretation? | Tests the group-level event interpretation. |
Table 11. Complete consistency group example from TerraSense_3.json. The eight items all refer to the same aircraft-centered event and test whether a model preserves a stable target interpretation across local EO and IR cues, temporal changes, and cross-modal checks.
| Step | Req. | Level | Question | Coherence role |
| 1 | EO | L2 | When the view changes, which cue is checked against the nearby alternative? | Verifies EO tug-to-barge tow geometry against the cooler barge-pair alternative. |
| 2 | IR | L2 | Which IR cue pairing appears in this interval? | Verifies warm hull or disturbed-water contrast paired with the tow geometry. |
| 3 | EO+IR | L3 | Which combined EO/IR sequence account is visible? | Combines EO tow geometry with IR thermal/wake evidence. |
| 4 | EO+IR | L3 | For the full clip, which cue chain is visible? | Tests the final maritime event-level conclusion over the full paired observation. |
Table 12. Complete coherence group example from the maritime sample TerraSense_46.json. The questions are scored as an ordered evidence chain: failures in early EO or IR evidence steps limit the prefix coherence credit for later event-level answers.
Metric and Scoring Implementation
This appendix gives implementation details for the metrics defined in Section 4. All metrics are computed from parsed multiple-choice predictions. Unless otherwise specified, grouped metrics and derived gaps are computed under paired EO+IR input.
Metric Definitions
Question-level accuracy is the fraction of QA items whose parsed prediction matches the ground-truth option. Level-wise scores are computed by averaging question accuracy within each capability level, and the main score is the macro-average
For a consistency group with questions and correct predictions, we compute
and report as the macro-average over consistency groups. For an ordered coherence group, let be the number of consecutive correct answers from the first step before the first error. We compute
and report as the macro-average over coherence groups. Cross-modal gain is computed as
Clustered Bootstrap Procedure
The evaluation code supports nonparametric bootstrap resampling that preserves the clustered structure of the benchmark. For , , , level-wise scores, , , and , one bootstrap draw samples the 100 EO–IR video pairs with replacement and includes all twelve QA items associated with each selected pair. For , , and consistency-sensitivity readouts, we resample groups with replacement within the corresponding group family. In the retained benchmark, each EO–IR pair contributes one consistency group and one coherence group, so group-level resampling also preserves the event-level clustering used by the grouped diagnostics. Derived quantities such as , , and are recomputed inside each bootstrap draw rather than formed by subtracting marginal interval endpoints. When intervals are computed, the default setting uses 10,000 bootstrap draws with a fixed random seed, with the 2.5th and 97.5th percentiles as interval endpoints. This procedure treats questions from the same paired observation or the same group as correlated rather than independent.
Alternative Group-Level Consistency Readouts
The squared consistency score in § 4.3 is intended to penalize fragmented correctness while retaining partial credit. For robustness auditing, the scorer also defines two alternative group-level readouts:
The linear readout measures mean within-group correctness, the squared readout emphasizes fragmented correctness, and the all-correct readout measures strict group completion. The main paper reports the squared readout as the primary consistency metric, and the released scorer can compute the linear and all-correct variants from the same group-level predictions. Table 13 reports this sensitivity check on representative models using scores computed from the retained group-level predictions.
| Model | |||
| Gemini 3.1 Pro | 57.2 | 36.8 | 21.2 |
| Qwen3-VL-235B | 54.5 | 33.4 | 19.1 |
| UniRS | 57.9 | 37.2 | 23.4 |
| LLaVA-Video-7B-Qwen2 | 25.3 | 8.4 | 2.2 |
Table 13. Consistency-score sensitivity for representative models. Scores are percentages and are computed from the retained group-level predictions.
The same broad diagnostic ordering holds under a lenient linear readout, the primary squared readout, and a strict all-correct readout. This supports the conclusion that the grouped-reasoning contrast is not an artifact of the squared penalty alone.
Prefix Coherence Example
Prefix coherence assumes that an ordered group represents an evidence chain. Later correct answers therefore do not fully compensate for earlier failures, because the final answer may be correct for the wrong reason or by exploiting a shortcut rather than by preserving the required visual evidence. Table 14 shows a retained coherence group from TerraSense_3.json. The first two steps verify EO and IR evidence needed to avoid confusing camera orbit with aircraft motion; the final steps ask for cross-modal and full-clip conclusions built on that evidence.
| Step | Requirement | Level | Question | Evidence role |
| 1 | EO | L2 | Which early-to-late EO chain would be unsafe to treat as aircraft travel? | Distinguishes platform orbit from target travel in EO. |
| 2 | IR | L2 | In the later IR view, which paved-surface layout surrounds the aircraft? | Checks the IR spatial layout around the same target. |
| 3 | EO+IR | L3 | Which cross-modal chain avoids the paint-color shortcut? | Requires combining EO temporal evidence with IR evidence. |
| 4 | EO+IR | L3 | Which full-clip reading uses the orbit and IR evidence together? | Tests the final event-level interpretation over the full paired observation. |
Table 14. Example coherence chain from a retained sample. If a model fails step 1 or 2, a correct step-4 answer is not counted as full evidence-supported coherence because the intermediate EO or IR evidence needed by the final conclusion was not established.
Group Handling Rules
Groups are scored only when they contain at least two QA items with the same group_id. Consistency groups are unordered and use all questions assigned to the group. Coherence groups require valid group_step indices; if a step index is missing or duplicated, the group is excluded from and flagged for annotation review. In the retained benchmark, all QA items are assigned to either a consistency or coherence group; level-wise and main-score metrics are still computed over individual QA items. Future ungrouped items would remain included in question-level, level-wise, and main-score evaluation but would not contribute to or .
Experimental Protocol
This appendix documents the inference protocol used for all model evaluations. Models receive only the visual input, the question, and four answer options. Capability labels, modality requirements, group identifiers, skill tags, and ground-truth answers are used only for scoring and analysis.
Shared Inference Protocol
Table 15 summarizes the protocol-level settings shared across evaluated models. Table 16 and Table 17 summarize the serving records used for the evaluated systems, including API snapshots or checkpoint identifiers, interfaces, decoding settings, and hardware/runtime notes.
| Setting | Protocol |
| Task | Zero-shot four-way multiple-choice QA. |
| Input settings | EO-only, IR-only, and paired EO+IR. |
| Question set | The same retained 1,200 QA items and answer-option order are used across input settings. |
| Temporal budget | 16 synchronized event-relative timestamps per modality: 16 frames for EO-only and IR-only; the matched EO and IR frames at each timestamp, up to 32 modality-labeled frames, for EO+IR. |
| Visual annotations | No target boxes, crops, masks, capability labels, modality requirements, group identifiers, or answers are shown to models. |
| Prompt template | The input-specific templates in Appendix C.2. |
| Output format | One answer option from A, B, C, and D. |
| Prediction parsing | First valid option extracted deterministically; invalid outputs are counted incorrect. |
| Judging | No external LLM judge is used. |
Table 15. Protocol-level inference settings shared across evaluations.
| Model | Checkpoint or API snapshot | Interface | Decoding | Hardware/runtime notes |
| GPT-5.4 | OpenAI API snapshot gpt-5.4-2026-03-05 | Responses API /v1/responses; sampled image frames | Temperature 0; max output 16 | Provider-hosted; hardware not disclosed. |
| Gemini 3.1 Pro | gemini-3.1-pro-preview | Gemini API; multimodal input | Temperature 0; max output 16 | Provider-hosted; hardware not disclosed. |
| Gemini 3 Flash | gemini-3-flash-preview | Gemini API; multimodal input | Temperature 0; max output 16 | Provider-hosted; hardware not disclosed. |
| SmolVLM2-2.2B | HuggingFaceTB/SmolVLM2-2.2B-Instruct | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| Qwen3-VL-2B | Qwen/Qwen3-VL-2B-Instruct | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| VideoLLaMA3-2B | DAMO-NLP-SG/VideoLLaMA3-2B | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| Video-XL-Pro-3B | MINT-SJTU/Video-XL-Pro-3B | Official checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| Qwen3.5-9B | Qwen/Qwen3.5-9B | Transformers/vLLM checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| Qwen3-VL-8B | Qwen/Qwen3-VL-8B-Instruct | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| InternVL3.5-8B | OpenGVLab/InternVL3_5-8B | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| MiniCPM-V 4.5 | openbmb/MiniCPM-V-4_5 | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| Molmo2-8B | allenai/Molmo2-8B | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
Table 16. Model-specific serving records, part 1.
| Model | Checkpoint or API snapshot | Interface | Decoding | Hardware/runtime notes |
| Keye-VL-1.5-8B | Kwai-Keye/Keye-VL-1.5-8B | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| MiMo-VL-7B-RL-2508 | Xiaom iMiMo/MiMo-VL-7B-RL-2508 | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| VideoLLaMA3-7B | DAMO-NLP-SG/VideoLLaMA3-7B | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| InternVideo2.5-Chat-8B | OpenGVLab/InternVideo2_5_Chat_8B | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| mPLUG-Owl3-7B | mPLUG/mPLUG-Owl3-7B-241101 | Transformers/ModelScope; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| LLaVA-Video-7B-Qwen2 | lmms-lab/LLaVA-Video-7B-Qwen2 | LLaVA-NeXT checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| Kimi-VL-A3B | moonshotai/Kimi-VL-A3B-Instruct | Transformers checkpoint; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| Qwen3-VL-30B-A3B | Qwen/Qwen3-VL-30B-A3B-Instruct | Hosted/vLLM-compatible endpoint | Greedy decoding; max new tokens 16 | Hosted open-weight serving; hardware not disclosed. |
| InternVL3.5-30B-A3B | OpenGVLab/InternVL3_5-30B-A3B | Hosted checkpoint endpoint | Greedy decoding; max new tokens 16 | Hosted open-weight serving; hardware not disclosed. |
| Qwen3-VL-235B-A22B | Qwen/Qwen3-VL-235B-A22B-Instruct | Hosted/vLLM-compatible endpoint | Greedy decoding; max new tokens 16 | Hosted open-weight serving; hardware not disclosed. |
| InternVL3.5-241B-A28B | OpenGVLab/InternVL3_5-241B-A28B-HF | Hosted checkpoint endpoint | Greedy decoding; max new tokens 16 | Hosted open-weight serving; hardware not disclosed. |
| UniRS | IntelliSensing/UniRS release code and checkpoint | UniRS code; sampled frames | Greedy decoding; max new tokens 16 | Single NVIDIA A100 GPU; BF16 inference; single-GPU runtime. |
| SkyEyeGPT | ZhanYang-nwpu/SkyEyeGPT | Official code; 16 sampled frames per sensor with sensor and time labels | Greedy decoding; max new tokens 16 | — |
| EarthDial | akshaydu dhane/EarthDial_4B_MS | Official code; 16 sampled frames per sensor and time labels | Greedy decoding; max new tokens 16 | — |
Table 17. Model-specific serving records, continued.
Prompt Templates
All models use the same multiple-choice prompt structure. The input-specific prompts differ only in the visual-input description.
EO-only. You are given an electro-optical aerial video. Answer the following multiple-choice question based only on the provided video. Select one option from A, B, C, and D. Question: [QUESTION] Options: [A/B/C/D] Answer:
IR-only. You are given an infrared aerial video. Answer the following multiple-choice question based only on the provided video. Select one option from A, B, C, and D. Question: [QUESTION] Options: [A/B/C/D] Answer:
EO+IR. You are given paired electro-optical and infrared aerial observations of the same event. Answer the following multiple-choice question based on the provided EO and IR evidence. Select one option from A, B, C, and D. Question: [QUESTION] Options: [A/B/C/D] Answer:
Text-only. Answer the following multiple-choice question. No visual input is provided. Select one option from A, B, C, and D. Question: [QUESTION] Options: [A/B/C/D] Answer:
Input Formatting, Temporal Sampling, and Parsing
For the main evaluation, each video pair is sampled at 16 synchronized event-relative timestamps per modality. EO-only input contains sampled EO observations; IR-only input contains sampled IR observations; EO+IR input contains paired EO and IR observations from matched temporal positions, with modality labels. The shared inference settings are summarized in Table 15, and model-specific serving records are summarized in Table 16 and Table 17.
Temporal sampling is defined on normalized event time rather than fixed raw frame indices. For each retained EO–IR pair, we sample uniformly spaced normalized timestamps over the retained paired event interval. EO and IR frames are selected at the nearest available decoded timestamp after event-level alignment; this permits small EO/IR stream-duration differences while preserving the same event-relative temporal positions. EO-only and IR-only inputs contain sampled frames from the corresponding stream. EO+IR inputs contain paired timestamps, i.e., up to frames, ordered as , with explicit modality labels. All evaluated systems, including API and open-weight models, receive sampled frames rather than full native video files in the main protocol. Frames are passed through each model interface or processor with aspect ratio preserved when resizing is required; no target crops, bounding boxes, masks, tracking metadata, or other visual annotations are supplied.
Table 18 lists the exact frame count for each input setting. We match temporal positions across models and input settings rather than total image count: halving the timestamps per sensor in EO+IR would confound sensor availability with temporal coverage, and model-specific default interfaces would expose models to different frame counts and event times. Image and visual-token counts may therefore differ. The three remote-sensing VLMs receive the same 16 timestamps, and only the input formatting changes to match each released interface. For SkyEyeGPT and EarthDial, EO+IR input uses 32 frames ordered as with sensor and time labels.
Predictions are parsed deterministically by extracting the first valid answer option from the model response. Valid options are A, B, C, and D. If no valid option is found, the prediction is marked incorrect. API transport failures, timeout, rate-limit responses, or empty responses are retried up to three times with the same prompt, inputs, and decoding settings. Unresolved failures are recorded as invalid outputs and counted incorrect. Repeated calls are used only to recover transport or service failures; no retry-based majority voting, response-level self-consistency, or best-of- selection is used. No external LLM judge is used.
| Input setting | Timestamps | Frames per sensor | Total frames |
| EO-only | 16 | 16 EO | 16 |
| IR-only | 16 | 16 IR | 16 |
| EO+IR | 16 | 16 EO + 16 IR | up to 32 |
Table 18. Frames per input setting under the default protocol.
| Model output pattern | Parsed prediction |
| A | A |
| The answer is B. | B |
| Option C is correct. | C |
| No valid option found | Incorrect |
| Multiple valid options found | First valid option |
Table 19. Prediction parsing rules.
All locally executed open-weight models were evaluated on NVIDIA A100 GPUs. No model training or finetuning was performed. The reported local experiments consist of sampled-frame inference and deterministic scoring over the fixed RAVEN-Bench evaluation set. Including the main benchmark runs and the reported robustness, frame-budget, and free-form controls, local inference required approximately 250–300 A100 GPU-hours. Provider-hosted API models and hosted open-weight endpoints were evaluated through their serving interfaces; their underlying hardware is not disclosed by the providers.
Validity and Robustness Controls
This appendix reports controls used to test whether RAVEN-Bench is visually grounded and whether paired-input gains reflect aligned EO–IR evidence rather than answer priors, prompt artifacts, input-budget effects, or temporal-format artifacts.
Non-visual Controls and Human Reference
The random baseline samples uniformly from the four answer choices. The answer-prior control predicts from the marginal answer distribution and is used only as a diagnostic estimate of option-prior leakage, not as a deployable baseline. The text-only setting removes visual input and provides only the question and answer options. The human-reference row is used only as contextual answerability calibration, and the independent audit statistics in Appendix § 9.4 summarize annotation reliability.
The random and answer-prior rows in Table 20 are computed from fixed-seed sampled control prediction files on the exported evaluation prompts, not from analytic expectations. The random file samples one option per item uniformly from A–D; the answer-prior file samples one option per item from the marginal answer distribution of the exported prompt option order. , , and are then computed by the same scorer used for model predictions. Because and are nonlinear group metrics, we do not substitute expected question-level accuracy into the group-metric formulas.
| Setting | Visual input | |||
| Random | None | 24.8 | 10.4 | 8.5 |
| Answer-prior | None | 26.1 | 10.8 | 8.8 |
| Text-only (Qwen3.5-9B [34]) | None | 22.9 | 8.3 | 6.6 |
| Text-only (best over evaluated models) | None | 28.4 | 12.1 | 9.8 |
| Human subset (120 QA / 10 pairs) | EO+IR | 82.3 | 77.1 | 69.6 |
| Best MLLM on same subset | EO+IR | 50.4 | 38.6 | 31.8 |
Table 20. Validity controls. Scores are percentages. The text-only controls estimate language-prior leakage; the human-reference scores are computed on the stratified 120-item subset with retained EO–IR clips for answerability calibration.
The human-reference subset contains 120 QA items from 10 EO–IR video pairs, stratified by scenario family, modality requirement, and capability level. The full audit in Appendix § 9.4 is a separate procedure with different annotator roles, instructions, and scoring criteria. The human-reference subset is used as an answerability calibration rather than a model-matched evidence-budget comparison: human annotators may replay the retained EO–IR clips, while models are evaluated under the sampled-frame protocol in Appendix § 11.3. Annotators do not see capability labels, modality requirements, group identifiers, or answer keys. Majority vote determines the final response, with expert adjudication for ties or uncertainty.
The best-MLLM row in Table 20 is Gemini 3.1 Pro, selected by under EO+IR input on the same 120-item subset. The grouped scores in that row are from the same model, not metric-wise maxima over different models.
Cross-modal Robustness Controls
For a control condition , we define
For the aligned EO+IR condition, this equals . For duplicated, shuffled, or misaligned inputs, it is used only to test whether the aligned EO+IR improvement can be explained by input-budget or formatting artifacts.
Table 21 uses model-level aggregation. For each model in the representative subset, is computed using that model’s control score and that same model’s EO-only and IR-only scores on the same retained 1,200-item evaluation set. The table then reports the macro-average over models. The representative subset is the same subset used in Figure 4: Gemini 3.1 Pro, Qwen3-VL-235B, Qwen3.5-9B, InternVideo2.5-Chat-8B, UniRS, and LLaVA-Video-7B-Qwen2. Thus is not formed by subtracting averaged EO and IR baselines after aggregation.

Figure 4. Evidence-condition slices for representative models. Bars compare EO-only, IR-only, and EO+IR inputs. Numbers above EO+IR bars in the EO+IR-required panel show over the stronger unimodal input.
| Condition | Control purpose | ||||
| Aligned EO+IR | 43.5 | +3.1 | 45.7 | 23.4 | Main paired-input setting. |
| EO+EO | 41.1 | +0.7 | 42.8 | 20.6 | Controls for two-stream input budget using duplicated EO evidence. |
| IR+IR | 40.7 | +0.3 | 43.1 | 21.1 | Controls for two-stream input budget using duplicated IR evidence. |
| Modality-shuffled EO+IR | 39.6 | -0.8 | 41.2 | 18.3 | Tests whether true EO–IR correspondence is required. |
| Temporally misaligned EO+IR | 40.1 | -0.3 | 39.9 | 17.4 | Tests whether synchronized EO–IR timestamps are required. |
| Temporal-shuffle | 39.2 | -1.2 | 37.4 | 15.8 | Tests whether ordered temporal evidence is required. |
| EO–IR label-swap | 40.5 | +0.1 | 41.3 | 19.2 | Tests whether modality labels and sensor semantics are used correctly. |
| Side-by-side mosaic | 39.0 | -1.4 | 39.1 | 17.1 | Tests spatial concatenation and target-size reduction. |
Table 21. Robustness controls for cross-modal gain. Scores are averaged over a representative model subset.
The control inputs are constructed from the same sampled-frame protocol as Appendix § 11.3. EO+EO and IR+IR duplicate the corresponding stream into both two-stream input slots, preserving the paired-input frame budget while removing cross-spectral complementarity. Modality-shuffled EO+IR pairs an EO stream with an IR stream from a different retained event, preserving the two-modality format but breaking event correspondence. Temporally misaligned EO+IR preserves the same EO–IR event pair but shifts one modality’s timestamp sequence relative to the other. Temporal-shuffle keeps the same sampled frames but permutes their temporal order. EO–IR label-swap exchanges the modality labels while keeping the frames fixed, and side-by-side mosaic concatenates matched EO and IR frames into a single visual frame per timestamp. The aligned EO+IR setting yields the largest control gain and strongest coherence. Duplicated-stream controls preserve the two-stream input budget but produce smaller gains, indicating that paired-input improvement is not explained by additional visual tokens alone. Shuffled, misaligned, and temporal-order controls reduce main, temporal, and coherence scores, supporting the need for aligned cross-spectral and temporal evidence.
| 4 | 38.4 | +0.9 | 39.6 | 16.8 | 21.6 |
| 8 | 41.3 | +2.1 | 43.1 | 20.1 | 21.2 |
| 16 | 43.5 | +3.1 | 45.7 | 23.4 | 20.1 |
| 32 | 44.1 | +3.2 | 46.3 | 23.9 | 20.2 |
Table 22. Frame-budget sensitivity under paired EO+IR input. denotes the number of synchronized timestamps per modality.
| Evaluation set | MC Main | FF Main | Answer+Evidence | FF Cons. | FF Coh. |
| RAVEN-Bench (100 pairs) | 43.5 | 34.8 | 27.1 | 24.6 | 19.3 |
Table 23. Free-form evaluation on the 100-pair release, averaged over the six-model subset. MC: multiple-choice; FF: free-form.
Frame-budget Sensitivity
For each frame budget , EO-only, IR-only, and EO+IR inputs are all resampled at the same synchronized timestamps. is recomputed from the corresponding EO-only and IR-only baselines at that same , rather than using the default unimodal baselines. Scores average the representative six-model subset of Table 21.
Increasing the number of synchronized timestamps improves temporal and coherence scores up to a moderate budget, after which gains become smaller. This supports the default sampling budget while showing that the main diagnostic trends are not caused by a single arbitrary frame count. From to , Main changes by 0.6 points, by 0.1, by 0.6, and Coherence by 0.5, and the Main–Coherence gap is nearly unchanged (20.1 versus 20.2).
Free-form Evaluation
Multiple-choice questions can make an item easier because the options narrow the answer space; their advantage is deterministic scoring without a language-model judge. To test generation directly, we also ran an open-ended evaluation with a six-model subset (Gemini 3.1 Pro, Qwen3-VL-235B, Qwen3.5-9B, InternVideo2.5-Chat-8B, UniRS, and LLaVA-Video-7B-Qwen2) under the same protocol.
Every free-form item is derived from one benchmark item. Questions that depended on their options were rewritten to be self-contained, keeping the same target and evidence requirements, time interval, and answer detail. Each model returns a concise answer followed by a brief evidence statement that identifies the required EO and/or IR evidence, when it appears, and how it changes over time. Two annotators, blind to model identity, independently judge answer and evidence correctness with a question-specific rubric; a third annotator resolves disagreements. Agreement is 93.1% for answer correctness and 88.9% for evidence correctness (Cohen’s and 0.77).
Free-form Main and Answer+Evidence use the same balanced average over L1, L2, and L3 as the multiple-choice Main score. Answer+Evidence requires both judgments to be correct. Free-form Consistency and Coherence are computed from answer correctness only, using the original question groups.
Free-form Main is 34.8, compared with 43.5 under multiple choice. Question wording, output format, and scoring also change, so the gap reflects more than the removal of answer options. For all six models, free-form Main remains above both grouped scores, and Answer+Evidence is 7.7 points below free-form Main, so some correct answers still lack correct supporting evidence.
Qualitative Error Cases and Responsible Release
This appendix provides qualitative examples for the error mechanisms summarized in § 5.5 and documents the responsible-release policy for RAVEN-Bench.
Representative Error Cases
Figure 5 illustrates how visually plausible answers can fail to preserve the evidence needed for EO–IR reasoning. These examples are not intended as a model leaderboard; they are selected to make the observed failure mechanisms concrete. In row (a), foreground-tree occlusion and a nearby road vehicle create an identity-swap error: the model follows a salient nearby dark car instead of maintaining the road-aligned tow pair through the interruption. In row (b), glare and surface reflections reduce the visibility of a small target, causing the model to pair an unstable EO cue with the wrong IR evidence. In row (c), a hot background structure dominates the thermal scene and pulls the model away from the smaller vehicle cue. In row (d), crossing wakes and similar vessel appearances create a cross-modal binding error: the EO wake cue is not reliably associated with the same target in IR.

Figure 5. Representative EO–IR failure cases using real sampled frames and representative model outputs. Each row shows temporally aligned EO and IR evidence, the model-facing question, ground-truth answer, model prediction, and a manual diagnostic attribution. The rows illustrate identity swaps under partial occlusion, small-target evidence loss under glare, hot-background thermal distractors, and wake-induced cross-modal binding failures.
These cases support the diagnostic role of the grouped metrics. Identity swaps and small-target losses primarily reduce stable target interpretation and ordered evidence preservation, which are exposed by and . Hot-background and wake-based distractors affect whether paired EO+IR input is used to bind the correct evidence, which is reflected in and Level-3 performance. The examples therefore explain why aggregate multiple-choice accuracy can overstate evidence-supported EO–IR reasoning.
Error-Mechanism Taxonomy
Table 24 groups the qualitative cases into recurring error mechanisms tied to aerial EO–IR sensing conditions. The taxonomy connects observable conditions, model behavior, and the diagnostic readouts most directly affected by each mechanism. The categories are not mutually exclusive: a single failure may involve occlusion, target scale, thermal salience, and cross-modal ambiguity at the same time. The table is therefore intended as a diagnostic aid rather than an exhaustive causal classification.
| Error mechanism | Physical condition | Model behavior | Primary readouts |
| Occlusion-driven identity swap | Foreground trees, moving occluders, nearby vehicles, viewpoint changes, or scale changes interrupt the target track. | The model binds post-occlusion evidence to a visually similar nearby object instead of preserving the same target identity. | , |
| Tiny-target evidence loss | The target occupies only a few pixels after resizing, frame sampling, or video-token compression; glare, reflection, or surface texture further reduces usable evidence. | The model detects a local cue or salient artifact but fails to preserve the target across frames or modalities. | , |
| Sensor-salience distractor | EO glare, shadows, wakes, hot backgrounds, low thermal contrast, or thermal crossover dominates the visible or thermal salience field. | The model follows the visually or thermally dominant distractor rather than the smaller target cue. | , |
| Cross-modal binding failure | EO and IR provide asymmetric evidence: EO may preserve shape, color, texture, or wake geometry, while IR may emphasize thermal contrast or suppress visible-spectrum cues. | The model relies on one stream, switches between inconsistent candidates, or pairs an EO cue with the wrong IR evidence. | , |
Table 24. Remote-sensing error-mechanism taxonomy. Each mechanism links an EO–IR sensing condition to a typical model behavior and the diagnostic readouts most directly affected.
This taxonomy clarifies why the same aggregate accuracy can correspond to different evidence-use failures. exposes fragmented interpretation of the same target across related questions. exposes event-level answers that are not supported by the ordered intermediate evidence. measures paired-input gain beyond the stronger unimodal stream, and should be interpreted as a controlled diagnostic of EO+IR benefit rather than as direct proof of sensor fusion.
Responsible Release and Limitations
Intended use and license terms. RAVEN-Bench is intended for research evaluation and auditing of EO–IR video reasoning. It is not designed for operational deployment or for any setting in which model outputs would inform real-world observation, intervention, or enforcement decisions. The videos and annotations are released under the TerraSense Analytics Data Release Terms, which grant a non-exclusive license for research and non-commercial use and require attribution to TerraSense Analytics Ltd. The terms prohibit commercial use of the dataset; attempts to re-identify individuals; unlawful or harmful use, including surveillance, tracking, or profiling of specific individuals, generation of deceptive media, discriminatory profiling, and fully automated decisions that adversely affect legal rights; and use of the licensor’s name to suggest endorsement. Users who redistribute the dataset or a modified version must pass these terms on to downstream recipients and identify the modifications. The terms place no restrictions on models or other results derived from the data, and the license terminates if a user breaches it.
What the release contains and omits. RAVEN-Bench provides no annotations or metrics for detection, tracking, segmentation, geolocation, targeting, or pixel-level registration. Users with access could still create new annotations from the videos. The release contains sanitized EO and IR benchmark segments, QA annotations, modality requirements, evidence intervals, group metadata, and evaluation code. It contains no boxes, masks, crops, target tracks, trajectories, geolocation, flight paths, pixel-registration data, or KLV and other operational metadata. Temporal annotations mark evidence intervals only. The original collection files are not released.
Residual re-purposing risk. Framing RAVEN-Bench as a video-QA benchmark does not prevent other technical uses. Removing operational metadata lowers the risk of exposing collection details, but the visual and temporal content can still be reused: after receiving access, a user could create labels or pseudo-labels and apply the clips to detection, tracking, or other computer-vision tasks beyond the benchmark’s intended use. Misuse could include surveillance, targeting, enforcement, border-monitoring, or military applications. Because the EO and IR streams are paired in time but carry no spatial registration, boxes, masks, or correspondence labels, such reuse would require new annotation, and the size of the release limits its training value; it could still support small-scale evaluation or fine-tuning. The data release terms permit non-commercial derivative research and do not restrict derived models, so the remaining safeguards are the prohibitions above, the omitted metadata and spatial labels, and gated access. We considered releasing only a fixed set of sparse frames, which would reduce some re-purposing risk, especially for tracking and fine-tuning. It would also remove the temporal evidence needed to evaluate target continuity, motion interpretation, and evidence order, and to reproduce the sampling protocol, the frame-budget comparison, and the temporal-order and misalignment controls. We therefore release sanitized segments that preserve this temporal evidence.
Access process. The dataset is hosted on Hugging Face (https://huggingface.co/datasets/TerraSenseAnalytics/RAVEN-Bench) with gated access. The dataset card, with documentation and access instructions, is public. The data files become available after an authenticated user states an institutional affiliation and proposed research use, accepts the data release terms, and the maintainers approve the request. The evaluation code is released under the MIT License at https://github.com/yuhu990424/RAVEN-Bench. These controls cannot prevent every misuse after download. They prevent anonymous access, identify who received the files, and make the license terms a condition of access.
Data ownership and privacy screening. The footage was collected by and remains under the institutional control of TerraSense Analytics Ltd., which owns the dataset and authorized this release with management approval. Several authors are affiliated with TerraSense Analytics. Authorization from the data owner does not replace privacy review. During curation, the retained EO and IR clips were manually screened for identifiable people and readable identifiers, including names, license plates, vehicle or vessel registration numbers, and other unique markings. A localized identifier was blurred when this could be done without affecting the benchmark evidence; otherwise, the clip or the relevant segment was excluded. Before release, every retained clip was reviewed again and the final files were inspected after redaction. The release criterion is that no identifiable person or readable unique identifier remains.
Limitations. RAVEN-Bench is compact and diagnostic rather than class-balanced at web scale. All videos come from one aircraft, one EO/IR gimbal, and one region in western North America, so the reported scores describe this sensing setup. Weather, altitude, and viewing geometry are recorded per event, but several condition subsets remain too small for reliable robustness comparisons. The benchmark evaluates video-QA reasoning rather than pixel-level registration, detection, tracking, or segmentation. Its conclusions are tied to the retained low-altitude aerial EO–IR clips, the chosen temporal sampling protocol, and the evaluated model interfaces. Extending coverage to other regions, platforms, and sensors will require new data, potentially through additional partners.
Release package and maintenance. The release package includes versioned files, evaluation code, validated Croissant metadata with Responsible AI fields, a dataset card, access instructions, issue-reporting procedures, and documented maintenance responsibility. Collection continues weekly; new events pass the same release, privacy, and safety review before the EO–IR correspondence and evidence checks. Future versions will add harder questions that require both EO and IR evidence, longer temporal intervals, deeper evidence chains, condition-specific results once the subsets are large enough, and, once validated, the experimental SAR stream carried by the same aircraft. The EO-only, IR-only, and EO+IR input conditions, correspondence controls, and grouped metrics will remain fixed so that results stay comparable across versions.
Existing assets and terms. The experiments use existing API models, open-weight model checkpoints, and baseline code only for evaluation. The corresponding model papers, reports, checkpoints, or provider documentation are cited in the main text and Appendix § 11. We use each asset under its released license or provider terms of service and do not redistribute third-party model weights, API outputs beyond aggregate benchmark results, or third-party code as part of the dataset release. The RAVEN-Bench videos are released under authorization from the data owner for research evaluation, and no third-party copyrighted video sources are included.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction describe RAVEN-Bench as a paired EO–IR aerial video QA benchmark with modality-conditioned evaluation, hierarchical capability levels, structured question groups, and representative MLLM evaluation. The claims are limited to benchmark construction and diagnostic evaluation, and do not assert a new model, deployment-ready system, or general coverage beyond the low-altitude EO–IR setting studied in the paper.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 6 (Limitations and responsible use) and Appendix § 13.3 discuss the limitations of RAVEN-Bench, including its single-aircraft, single-sensor, single-region scope, the small size of condition subsets, its compact diagnostic scale, focus on retained low-altitude EO–IR clips, video-QA formulation rather than pixel-level registration, detection, tracking, or segmentation, and dependence on the chosen temporal sampling protocol and evaluated model interfaces.
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [N/A]
Justification: This paper does not include theoretical results.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix § 11 shows the implementation details and settings of our experiments.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example: (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code: https://github.com/yuhu990424/RAVEN-Bench. Data: https://huggingface.co/datasets/TerraSenseAnalytics/RAVEN-Bench (gated access; Appendix E.3).
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: Section 5, appendix C and D shows the implementation details and settings of our experiments.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper discusses statistical uncertainty in § 4.3 and avoids significance claims based on small isolated score gaps. Appendix § 10.2 specifies the clustered bootstrap procedure used by the evaluation code: question-level and level-wise metrics are resampled at the EO–IR video-pair level, group metrics are resampled at the group level, and derived quantities are recomputed within each bootstrap draw using 10,000 draws and percentile intervals.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Appendix § 11 reports the model interfaces, decoding settings, and hardware notes for the evaluated systems. The paper states that locally executed models were run on NVIDIA A100 GPUs, that no training or finetuning was performed, estimates the local inference cost at approximately 250–300 A100 GPU-hours, and notes that provider-hosted hardware is not disclosed.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: Nothing in this paper violates the NeurIPS Code of Ethics.
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 6 and Appendix E.3 discuss the intended positive use of RAVEN-Bench for research evaluation and auditing of EO–IR video reasoning, and also discuss negative societal risks from misuse in surveillance, targeting, enforcement, border monitoring, military, or critical-infrastructure settings. The paper further describes mitigations including metadata removal, exclusion of sensitive clips, gated research-evaluation access, research-use terms, dataset documentation, and maintenance procedures.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Appendix E.3 describes safeguards for the responsible release of RAVEN-Bench. The paper states that the dataset is intended only for research evaluation and auditing, states that it is not designed for operational deployment, licenses it for research and non-commercial use with restrictions on re-identification and harmful uses such as surveillance or tracking of specific individuals, removes or anonymizes sensitive metadata and operational details, excludes clips with personally identifiable information or unnecessary sensitive content, and releases the dataset through gated access under the TerraSense Analytics Data Release Terms with dataset documentation and issue-reporting procedures.
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites the evaluated API models, open-weight checkpoints, and baseline systems, and Appendix C reports their checkpoint or API identifiers and interfaces. Appendix E.3 states that these existing assets are used under their released licenses or provider terms, are not redistributed as part of RAVEN-Bench, and that the released videos are authorized by the data owner with no third-party copyrighted video sources.
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces RAVEN-Bench as a new benchmark asset, released under the TerraSense Analytics Data Release Terms (research and non-commercial use) with gated access to the video files and with evaluation code under the MIT License (Appendix E.3), and documents its composition, metadata schema, annotation protocol, independent audit, evaluation code, access constraints, limitations, and responsible-release process. The release package includes Croissant metadata, a dataset card, research-use terms, access instructions, issue-reporting procedures, and maintenance information. Code: https://github.com/yuhuu990424/RAVEN-Bench
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper describes the limited use of LLM-based assistance in the QA construction and annotation workflow. As stated in § 3.2 and Appendix § 9.3, LLM assistance was used only for wording refinement, option clarity, distractor review, and ambiguity flagging, and was not used to determine visual facts, answer keys, modality requirements, capability labels, or group assignments.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.
References
- [1]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.arxiv.org/abs/2502.13923
- [2]Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.arxiv.org/abs/2412.05271
- [3]Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023.arxiv.org/abs/2311.10122
- [4]Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava-video: Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024.arxiv.org/abs/2410.02713
- [5]Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024.arxiv.org/abs/2405.21075
- [6]Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, et al. Mlvu: Benchmarking multi-task long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.arxiv.org/abs/2406.04264
- [7]Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, and Jie Tang. Lvbench: An extreme long video understanding benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025.arxiv.org/abs/2406.08035
- [8]Kartik Kuckreja, Muhammad Shahzad Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fahad Shahbaz Khan. Geochat: Grounded large vision-language model for remote sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.arxiv.org/abs/2311.15826
- [9]Xiang Li, Jian Ding, and Mohamed Elhoseiny. Vrsbench: A versatile vision-language benchmark dataset for remote sensing image understanding. arXiv preprint arXiv:2406.12384, 2024.arxiv.org/abs/2406.12384
- [10]Wei Zhang, Miaoxin Cai, Tong Zhang, Yin Zhuang, and Xuerui Mao. Earthgpt: A universal multimodal large language model for multisensor image comprehension in remote sensing domain. IEEE Transactions on Geoscience and Remote Sensing, 62:1–20, 2024.arxiv.org/abs/2401.16822
- [11]Zheng Wu, Nathan Fuller, Diane Theriault, and Margrit Betke. A thermal infrared video benchmark for visual analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 201–208, 2014.DOI
- [12]Haolin Qin, Tingfa Xu, Tianhao Li, Zhenxiang Chen, Tao Feng, and Jianan Li. Must: The first dataset and unified framework for multispectral uav single object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
- [13]D. Manjunath, Prajwal Gurunath, Sumanth Udupa, Aditya Gandhamal, Shrikar Madhu, Aniruddh Sikdar, and Suresh Sundaram. Indrayee: Infrared electro-optical uav-based perception dataset for robust downstream tasks. arXiv preprint arXiv:2410.20953, 2024.arxiv.org/abs/2410.20953
- [14]Kunchang Li, Yali Wang, Yinan He, Yizhou Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024.arxiv.org/abs/2311.17005
- [15]Chaoyou Fu, Haozhi Yuan, Yuhao Dong, Yi-Fan Zhang, Yunhang Shen, Xiaoxing Hu, Xueying Li, Jinsen Su, Chengwu Long, Xiaoyao Xie, Yongkang Xie, Xiawu Zheng, Xue Yang, Haoyu Cao, Yunsheng Wu, Ziwei Liu, Xing Sun, Caifeng Shan, and Ran He. Video-mme-v2: Towards the next stage in benchmarks for comprehensive video understanding. arXiv preprint arXiv:2604.05015, 2026.arxiv.org/abs/2604.05015
- [16]Yang Zhan, Zhitong Xiong, and Yuan Yuan. Skyeegpt: Unifying remote sensing vision-language tasks via instruction tuning with large language model. arXiv preprint arXiv:2401.09712, 2024.
- [17]Sagar Soni, Akshay Dudhanе, Hiyam Debary, Mustansar Fiaz, Muhammad Akhtar Munir, Muhammad Sohail Danish, Paolo Fraccaro, Campbell D. Watson, Levente J. Klein, Fahad Shahbaz Khan, and Salman Khan. Earthdial: Turning multi-sensory earth observations to interactive dialogues. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14303–14313, 2025.arxiv.org/abs/2412.15190
- [18]Yu Zhang, Zhicheng Zhao, Ze Luo, Chenglong Li, and Jin Tang. Uav traffic scene understanding: A regulation embedded multi-modal network and a unified benchmark. arXiv preprint arXiv:2603.10722, 2026.arxiv.org/abs/2603.10722
- [19]Mobin Habibpour, Niloufar Alipour Talemi, John Spodnik, Camren J. Khoury, and Fatemeh Afghah. Wildfirevqa: A large-scale radiometric thermal vqa benchmark for aerial wildfire monitoring. arXiv preprint arXiv:2604.20190, 2026.arxiv.org/abs/2604.20190
- [20]Rusiru Thushara, Yasiru Ranasinghe, Jay Paranjape, and Vishal M. Patel. Thermo-vl: Extending vision-language models to thermal infrared perception. arXiv preprint arXiv:2605.21882, 2026.arxiv.org/abs/2605.21882
- [21]Soonmin Hwang, Jaesik Park, Namil Kim, Yukyung Choi, and In So Kweon. Multispectral pedestrian detection: Benchmark dataset and baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1037–1045, 2015.DOI
- [22]Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2407.15754
- [23]Xichen Tan, Yuanjing Luo, Yunfan Ye, Fang Liu, and Zhiping Cai. Allvbb: All-in-one long video understanding benchmark. arXiv preprint arXiv:2503.07298, 2025.arxiv.org/abs/2503.07298
- [24]Fengxiang Wang, Hongzhen Wang, Zonghao Guo, Di Wang, Yulin Wang, Mingshuo Chen, Qiang Ma, Long Lan, Wenjing Yang, Jing Zhang, Zhiyuan Liu, and Maosong Sun. Xlrs-bench: Could your multimodal llms understand extremely large ultra-high-resolution remote sensing imagery? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
- [25]Yunkai Dang, Meiyi Zhu, Donghao Wang, Yizhuo Zhang, Jiacheng Yang, Qi Fan, Yuekun Yang, Wenbin Li, Feng Miao, and Yang Gao. A benchmark for ultra-high-resolution remote sensing mlms. arXiv preprint arXiv:2512.17319, 2025.arxiv.org/abs/2512.17319
- [26]Yujie Li, Wenjia Xu, Guangzuo Li, Zijian Yu, Zhiwei Wei, Jiuniu Wang, and Mugen Peng. Unirs: Unifying multi-temporal remote sensing tasks through vision language models. arXiv preprint arXiv:2412.20742, 2024.arxiv.org/abs/2412.20742
- [27]OpenAI. Gpt-5.4 model. https://developers.openai.com/api/docs/models/gpt-5.4, 2026. Accessed: 2026-05-03.
- [28]Google DeepMind. Gemini 3.1 pro model card. https://deepmind.google/models/model-cards/gemini-3-1-pro/, 2026. Accessed: 2026-05-03.
- [29]Google. Gemini 3 flash preview. https://ai.google.dev/gemini-api/docs/models/gemini-3-flash-preview, 2026. Accessed: 2026-05-03.
- [30]Andres Marafioti, Orr Zohar, Miquel Farre, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, et al. Smolvlm: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299, 2025.arxiv.org/abs/2504.05299
- [31]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
- [32]Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025.arxiv.org/abs/2501.13106
- [33]Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, and Bo Zhao. Video-xl-pro: Reconstructive token compression for extremely long video understanding. arXiv preprint arXiv:2503.18478, 2025.arxiv.org/abs/2503.18478
- [34]Qwen Team. Qwen3.5-9b. https://huggingface.co/Qwen/Qwen3.5-9B, 2026. Accessed: 2026-05-03.
- [35]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025.
- [36]Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, Wenshuo Ma, Zhihui He, Tianchi Cai, Weize Chen, Yuxiang Huang, Yuanqian Zhao, et al. Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe. arXiv preprint arXiv:2509.18154, 2025.arxiv.org/abs/2509.18154
- [37]Allen Institute for AI. Molmo-2-8b. https://huggingface.co/allenai/Molmo-2-8B, 2026. Accessed: 2026-05-03.
- [38]Biao Yang, Bin Wen, Boyang Ding, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, et al. Kwei k eye-vl 1.5 technical report. arXiv preprint arXiv:2509.01563, 2025.
- [39]LLM-Core-Team Xiaomi. Mimo-vl technical report. arXiv preprint arXiv:2506.03569, 2025.
- [40]Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, Min Dou, Kai Chen, Wenhai Wang, Yu Qiao, Yali Wang, and Limin Wang. Internvideo2.5: Empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386, 2025.arxiv.org/abs/2501.12386
- [41]Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024.
- [42]Kimi Team. Kimi-v1 technical report. arXiv preprint arXiv:2504.07491, 2025.
