MarineLife-16K: An Attribute-Rich Video-Text Dataset for Motion-Aware Marine Video Understanding

Introduction

Marine video analysis presents unique challenges to computer vision, particularly marine life analysis, e.g., fish, whose complex behaviors are intertwined with functional purposes [14, 18]. For instance, schooling behavior serves critical ecological functions, including optimized orientation, hunting, predator confusion, and risk mitigation. Conversely, when individual fish detach from a school to navigate around reefs, their actions often signal a shift toward localized foraging, shelter-seeking, or resting. Capturing these behaviors requires changing from object-centric tracking to motion-centric video understanding [10, 20].

Current video-text benchmarks remain limited when applied to these real-world ecological contexts. Existing datasets focus on global video summarization [33]—describing actions at a broad level without localizing specific timestamps. In contrast, video Question Answering (video-QA) has emerged as a powerful paradigm to evaluate fine-grained reasoning across different temporal windows [31]. However, existing video-QA benchmarks suffer from a lack of motion diversity and fail to challenge models on complex, real-world temporal dynamics.

For QA, it requires datasets that are richly and carefully annotated across different aspects like fact-centric QAs [19], relationship and motion questions [4], focusing on fine-grained attributes. Furthermore, LLM captioning models are often redesigned for QAs, but struggle to perform grounding. To address those limitations, we introduce a novel video-text benchmark dataset for marine video analysis, evaluating dense video captioning and complex motion-aware QA.

Although video-language models (VLMs) are powerful at generating text, they still suffer from hallucinations Figure 2. Leveraging VLMs for captioning complex videos such as marine footage remains ineffective. In particular, MSC [23] introduces the two-stage video captioning pipeline, aiming to reduce object hallucinations, but fails to detect motion hallucinations in videos. To solve this, we propose a human-in-the-loop captioning engine that reduces motion hallucinations through manual, fine-grained attribute annotations. Compared with MSC [23], our dataset is 5× larger in scale and extracts motion-aware attributes. We compare our MarineLife-16K with existing video-text paired datasets in Table 1.

Dataset	Time (hours)	Domain	#Sentence	Caption Length	#Video-text Pairs	Cap Source	Object	Motion	Camera	Prompt Complexity
WebVid [2]	13K	open	-	12	2.5M	generated	✗	✗	✗	low
InternVid-VTT [26]	76K	open	1	10-20	7.1M	generated	✗	✗	✗	medium
MiraData [7]	16K	-	-	-	-	generated	✗	✗	✗	low
MovieBench [28]	69.2	movie	-	-	-	generated	✓	✓	✓	medium
MSR-VTT [30]	41.2	open	1	9.3	200K	manual	✗	✗	✗	low
EPIC-KITCHENS-100 [5]	100	cooking	1	3	19.8K	manual	✗	✗	✗	low
InternVid [26]	760K	open	1	17.6	234M	generated	✗	✓	✗	medium
MovieRecapsQA [19]	107	movie	-	-	60	generated	✓	✓	✗	high
MOMA-QA [4]	147	open	-	-	1412	manual	✓	✓	✗	high
MVK [22]	12.3	underwater	1	9.8	1378	generated	✗	✗	✗	low
MSC [23]	3.6	underwater	2.27	27.4	396	manual	✓	✓	✗	high
MARINELIFE-16K (Ours)	18.48	underwater	3.4	44	2000	manual	✓	✓	✓	high

In summary, we make the following contributions in our work.

• We propose a scalable caption engine to reduce hallucinations, using a human-in-the-loop workflow annotated by domain experts.

• We introduce a large-scale motion-driven video dataset with 2,000 video–text pairs, featuring 13 fine-grained attributes for the marine domain.

• We develop a benchmark for question answering (QA) tasks, with 16K question–answer pairs, to evaluate eight open-source models and four closed-source models. We observe that VLM models struggle with camera-motion understanding.

Related Works

Marine Video Understanding using LLMs. Marine video analysis has recently attracted considerable interest from the computer vision community. Recent works such as MVK [22] and MSC [23] leverage LLMs to understand marine scenarios. More broadly, models like OceanGPT [3], VideoRefer Suite [31], and Video-LLaMA3 [32] integrate spatio-temporal video features and language within a unified embedding space. Specifically, these frameworks map sequential visual tokens into an LLM text space to generate open-ended responses. However, existing LLM architectures still struggle with fine-grained temporal comprehension, which can cause hallucinations of complex camera motion and object motion dynamics in challenging underwater scenarios.

Video Question Answering. There has been a growing interest in multimodal VideoQA benchmarks like EgoSchema [12], where models must tightly integrate long-form visual and textual contexts to reason across modalities. Moving beyond generic video understanding, recent benchmarks introduce specialized dimensions of temporal and structural complexity. For instance, HD-EPIC [17] establishes fine-grained, multi-modal egocentric alignments to evaluate action and gaze anticipation over extended timelines, while MovieRecapsQA [19] shifts the focus toward long-form narrative comprehension by grounding open-ended QA pairs in atomic movie facts.

MarineLife-16K- A real-world video-text dataset with fine-grained annotations

Data engine.

We present a scalable, three-stage marine caption engine that leverages human-in-the-loop attributes and multi-agent alignment scoring to generate high-fidelity, factually grounded video descriptions, as illustrated in Figure 3. We use LLM, i.e., GPT-5.5 [15] for dataset construction steps.

• Step 1: Manual attributes. Three domain experts extract discrete marine attributes from raw videos.

• Step 2: Multi-agent-based synthetic caption. We leverage LLM agents to caption marine videos. Specifically, an LMM Captioner is used to draft text descriptions; an LMM Reviewer scores the semantic alignment between the manual attributes and the synthetic caption, yielding a matching score $(S_c)$, where $S_c \in[0,5]$.

• Step3: Final caption. If $S_c < \tau$, a Refiner incorporates human attributes into a synthetic caption to output the final caption; otherwise, it accepts the synthetic caption. We set $\tau$ to 3.5 in our work.

Collection and annotation

Overview. Here, we introduce a real-world dataset, including 2,000 video-text pairs in the marine domain. The example of our MarineLife-16K is illustrated in Figure 4.

Fine-grained attributes. We also provide fine-grained annotations, including central object and up to 3 additional objects, environment, lighting, video clarity, motion/behavior, and camera attributes, e.g., angle and position for the collected videos. The captioning annotation begins with manually watching each video entirely to identify a central object and any other relevant objects. For each central object, we further annotate its appearance attributes, including texture, size, and shape&color. We allow up to 4 objects in each video to be extracted and described in the video’s caption. It is important to provide insights of the underwater environment for future ecological studies. To do so, we further describe the environmental factors from the videos, including lighting conditions, video clarity, any movement or behavior observed, and the camera attributes (position, movement, and sharpness), which we describe in detail below. This captioning process takes around 10 minutes per video, excluding the time required to watch the content.

Multi-object. We bound the maximum object density at four objects per clip to maintain descriptive quality. The targeted taxonomy includes common marine entities, such as mobile fauna (fish, human divers), benthic flora (algae, seaweed, seagrass), and foundational substrate organisms (coral and encrusting sponges).

Object motion taxonomy and behavioral categorization. To comprehensively model temporal dynamics, we formalize a fine-grained taxonomy of marine object motions. We decompose behavioral patterns into active kinematics, metabolic functions, and survival strategies. Crucially, our data engine formalizes both the empirical movement speed/behavior and ecological context. We categorize these dynamic visual events into four primary behavioral dimensions: (1) Locomotion and kinematics: tracks fine-grained motion of fins, tails, and body musculature during swimming. (2) Behaviors: encompass fine-grained feeding actions (hunting, stalking, chewing, swallowing, or filter-feeding via suction, biting, and snapping). (3) Survival strategies: models resting/hiding states inside localized substrates (e.g., rocks, coral reefs, vegetation, caves) alongside active defenses such as morphological camouflage, color-changing dynamics. (4) Socio-ecological dynamics: captures coordinated group dynamics (e.g., collective schooling/shoaling) and reproductive events like live birth.

Camera motion dynamics. We describe the dynamic camera using three attributes: camera position, camera angle, and video sharpness. For camera position, we specify whether the camera captures the full scene, a single viewing angle, or only a specific part of the central object (e.g., a portion of the main object). For video sharpness, we categorize the footage as either clear or blurry. We also characterize the camera angle and movement. The camera may perform a horizontal pan (turning left/right), a vertical tilt (moving up/down), or a dolly/translation movement in which the camera shifts position while keeping the main object as the reference center.

Video data quality.

Increasing data diversity. We selected videos, recorded from 27 different locations and at different times, spanning 7 countries. We curated the videos from two datasets, MVK [22] and MSC [23] datasets.

Quality control. We performed quality control by having six human reviewers assess 2,000 text captions.

Fine-grained attributes with motion dynamics. Our dataset contains fine-grained attributes, aiming to reflect the primary challenges of real-world videos, outlined in Table 2.

Video Challenge	Visual Attribute
Object-Centric
Heterogeneous Object. Object regions have distinct colors.	Color
Shape Complexity. The object has complex boundaries, such as thin parts and holes.	Size/Shapes
Appearance Change. Noticeable appearance variation, due to illumination changes and relative camera-object rotation.	Lighting, Camera Motion, Size/Shape
Low Resolution. The ratio between the average object bounding-box area and the image area is smaller than 0.1.	Size/Shape
Object & Camera Pose
Fast-Motion. The average, per-frame object motion, computed as the Euclidean distance between centroids, is larger than 20 pixels.	Behavior
Camera-Shake. Footage displays non-negligible vibrations.	Camera Motion
Motion Blur. The object has fuzzy boundaries due to fast motion.	Lighting, Behavior
Out-of-view. Object is partially clipped by the image boundaries.	Object, Behavior
Occlusion. Object becomes partially or fully occluded.	Object
Scale-Variation. The relative area ratio between any two instances of the target object is restricted to less than 0.5.	Object Motion, Behavior
Interacting Objects. The target object is an ensemble of multiple, spatially-connected objects (e.g., school of fish).	Behavior
Dynamic Background Context
Dynamic Background. Background regions move or deform.	Background

Motion-aware QA Generation

For multiple-choice questions, we use fine-grained attributes as answers and use an LLM prompt to generate the remaining answer choices. For open-ended QA questions, we use video captions as facts to create question-answer pairs, evaluating the VLM’s ability to understand camera motions. Specifically, the video QA task is designed to evaluate a model’s capability to decode the functional relationship between camera movement and the corresponding object’s attributes, including motion, appearance, and geometry. We use GPT-5.5 [15] as LLM judge. Finally, we generate 16,080 video-question pairs, including 12080 multiple choice questions and 4,000 open-ended questions for the collection of 2,000 videos.

Statistics.

We conduct a statistical analysis of the frequency of text captions of MarineLife-16K. We show frequencies by instance object types (from object 1 to 4), and fine-grained attributes in Figure 5. In our MarineLife-16K, there are 16,080 questions, including 12080 multiple choice questions and 4,000 open-ended questions for the collection of 2,000 videos.

Dataset Split. We split the dataset into a training set with 1300 videos, a validation set with 200 videos and a testing set with 500 videos.

Evaluation Dimensions

To comprehensively evaluate model reasoning ability, we assess VLMs on multiple aspects, including how they summarize overall video content and how well they understand specific questions to test grounding and question-answering (QA) capabilities. We design two benchmarks focused on two key areas: multiple-choice marine-related QA and video captioning in Figure 7.

Video-QA

To comprehensively assess vision-language models in the marine domain, we systematically classify our evaluation protocol into five dimensions.

• Marine Biotic Communities (MBC) evaluates the model’s capacity for multi-object detection and counting. Specifically, we aim to measure the precise identification of the central object as MBC-so alongside up to three additional objects as MBC-mo to reconstruct the marine biotic community.

• Phenotypic/Morphological Traits (PMT) evaluates fine-grained feature extraction for taxonomic classification and biological health assessment. We target the accurate grounding and description of phenotypic traits like texture, size, shape, and color.

• Organism Behavioral Kinematics (OBK) assesses VLM’s capacity for spatiotemporal motion reasoning from marine video. We evaluate how effectively the model decodes complex motion/behavior patterns.

• Spatiotemporal Monitoring (SM) measures VLM models’ understanding the environmental context. We assess the integration of environmental attributes, specifically under shifting lighting conditions as SM-l and environments as SM-e.

• Perspective-Dependent Observations (PDO) evaluates the model’s viewpoint invariance against dynamic camera pose. Specifically, we test the consistency of marine resource identification across diverse camera angles (horizontal, vertical, centered) and spatial positions (full scene, partial view).

• Open-Ended with Motion Understanding (OEMU) assesses how VLMs captures camera viewpoint and objects. We target to object motion, central object and visible activity.

Video Captioning with Motion Understanding

Our captioning benchmark is used to rigorously measure the model’s resistance to hallucination by assigning a dimension score from 0 to 5 to each generated prediction across the following three dimensions.

• Biological Entity Hallucination (BEH) evaluates whether the objects referenced in the generated description correctly match the objects specified in the ground truth.

• Action Hallucination (AH) assesses whether the described object motion and camera pose are consistent with the actual movements.

• Environmental Context Fabrications (ECF) identifies discrepancies by checking whether the generated description

includes any facts or elements that are not present in reality.

Experiments

Our goal is to evaluate how state-of-the-art video-LLMs handle VideoQA tasks, focusing on reasoning capabilities that account for motion dynamics, across both open-ended and multiple-choice questions. This section describes our experimental setup and model performance across evaluation dimensions in marine scenarios.

Models

We evaluate a broad range of proprietary and open-source VLMs for video QA and captioning. Our proprietary lineup includes frontier close-source models, i.e., Claude-4.8-Opus [1], Grok-4.3 [29], Gemini-3.1-Pro [6], and Kimi K2.6 [13]. We further include open-source models, i.e., small VLMs (InternVL3-8B [25], LLaVA-OneVision-7B [8], Qwen2.5-VL-7B [27], LLaVA-NeXT-Video [9], Video-LLaMA3 [32]) and larger VLMs (Gemma4-31B [21], Qwen2.5-VL-72B [27], InternVL3-78B [25]), using official Huggingface releases ranging from 7B to 78B parameters. We uniformly sample 8 frames per video as an ordered image sequence input. Empirically, we find 8 frames effectively capture camera motion while balancing accuracy and token cost. All experiments for the open-source model are conducted using 6 NVIDIA GeForce RTX4090 D. We will release the prompts and hyperparameter configurations used for the public.

Human Study

To validate the LLM judge for open-ended QA, we conducted a human study with six domain experts, randomly selecting 100 questions across four open-ended QA types evaluating camera motion from visual attributes. Each question was rated by three experts, and scores were averaged. As shown in Figure 6, despite some gap between human and LLM (GPT-5.5) judges, human scoring is largely consistent with LLM scoring, confirming the effectiveness of using LLMs as judges for QA tasks. Small open-source models (7B-8B) struggle with AH.

Multiple-choice Questions with Motion Understanding

Observation #1: Proprietary models outperform open-source models across most downstream benchmarks. We find that closed-source models outperform open-source models across most evaluation dimensions like MBC-so, MBC-mo, PMT, OBK, SM-l and OEMU in Table 3 because closed-source models often train on larger and private datasets rather than open-source datasets.

Model	Backbone	MBC-so	MBC-mo	PMT	PDO	OBK	SM-e	SM-l	OEMU
Open-source VLMs (7B - 8B)
Video-LLaMA3-7B [32]	2D RoPE	96.6	66.6	76.6	63.3	90	73.3	35.3	2.2
LLaVa-NeXT-Video [9]	CLIP	80	56.7	40	60	83.3	23.3	11.8	2.14
LLaVA-OneVision-7B [8]	SigLIP/Qwen	90	63.3	60	53.3	80	83.3	29.4	2.11
Qwen2.5-VL-7B [27]	2D RoPE/Qwen	96.7	66.7	66.7	70	96.7	66.7	29.4	2.43
InternVL3-8B [25]	V2PE/Qwen	96.7	83.3	73.3	70	93.3	73.3	52.9	1.36
Average		92.00	67.32	63.32	63.32	88.66	63.98	31.76	2.04
Open-source VLMs (31B - 78B)
Gemma4-31B [21]	SigLIP/Qwen	89.3	36.6	53.3	43.3	86.6	60	47.05	2.52
Qwen2.5-VL-72B [27]	ViT/Qwen2.5	90.5	76.6	80	46.7	93.3	66.7	66.7	2.47
InternVL3-78B [25]	InternViT-6B/Qwen	91.2	63.3	70	46.6	83.4	53.3	29.4	1.98
Average		90.33	58.83	67.77	45.53	87.77	60	47.72	2.32
Close-source VLMs
Claude-Opus-4.8 [1]	-	93.33	73.3	83.3	60	96.6	60	70.5	2.42
Grok-4.3 [19]	-	93.33	70	76.6	53.3	93.3	60	70.5	2.58
Gemini-3.1-Pro [6]	-	93.3	76.6	90	66.6	96.6	66.6	35.3	2.1
Kimi K2.6 [13]	-	100	60	80	63.3	96.6	63.3	52.9	2.48
Average	-	94.99	69.98	82.46	60.8	95.78	62.48	57.3	2.40

Open-ended Questions

Model	STATIC CAMERA			MOVING CAMERA
	BEH	AH	ECF	BEH	AH	ECF
Open-source VLMs (7B - 8B)
Video-Llama3-7B [32]	3.13	2.1	2.6	3.13	2.1	2.43
LLaVa-NeXT-Video [9]	2.9	2.71	2.4	2.43	1.72	2.42
LLaVA-OneVision-7B [8]	3.1	2.29	3.14	3.1	1.85	3.14
Qwen2.5-VL-7B [27]	3.43	2.71	2.71	3.14	2.58	2.57
InternVL3-8B [25]	2.57	1.86	2.3	2.71	1.86	2.42
Average	3.03	2.33	2.63	2.9-0.13	2.02-0.31	2.6-0.03
Open-source VLMs (31B - 78B)
Gemma4-31B [21]	3.6	3.14	3.02	3.29	3.29	2.85
Qwen2.5-VL-72B [27]	3.57	2.86	2.85	3.71	3.43	3.43
InternVL3-78B [25]	3.29	2.57	2.57	3.57	3.01	3.1
Average	3.49	2.86	2.81	3.52+0.03	3.24+0.38	3.13+0.32
Close-source VLMs
Claude-4.8-Opus [1]	3.14	3	2.3	3.29	2.86	2.43
Grok-4.3 [29]	3.3	2.6	2.28	3.14	2.71	2.57
Gemini-3.1-Pro [6]	3.9	2.4	2.57	3.29	2.57	3.29
Kimi K2.6 [13]	3.6	3.11	2.43	3.14	3.1	2.28
Average	3.49	2.78	2.4	3.22-0.27	2.81+0.03	2.64+0.24

Conclusion

In this work, we introduced a sequential multi-trajectory expert annotation and refinement strategy, using a human-in-the-loop multi-agent data engine for curating high-quality, motion-centric video-text data for marine environments. Using this strategy, we built a benchmark of 2,000 marine videos with 16,080 fine-grained question-answer pairs, annotated across 13 attributes covering key marine video challenges. We further constructed a comprehensive evaluation suite spanning nine dimensions across open-ended and multiple-choice QA. Our results show both open-source and closed-source models struggle most with camera motion dynamics. We hope this benchmark advances underwater video understanding.

Acknowledgements

This project is funded by the Environment and Conservation Fund (Project No.: ECWW26IS07 235/2025) of the Hong Kong Special Administrative Region Government.

We thank all the volunteer students who helped do the annotations and validation.

Introduction

In this supplementary material, we present the copyright of our dataset in ‘Copyright’. The prompts for the caption engine are illustrated in ‘Data Engine’. We also present a prompt for QA generation in ‘Multiple-choice Generation’. Evaluation dimensions for open-ended QA with motion understanding, aimed at detecting hallucinations in predicted answers, are introduced in ‘Evaluation Dimensions of Open-ended’.

Copyright

This dataset is released for research purposes only. The dataset uses or is derived from data licensed from MVK [22] and MSC [23] datasets. Prior to releasing this dataset, we contacted the respective data owners of MVK and MSC and obtained their permission to use their datasets for research

use. In this submission, we release a small subset of the dataset, consisting of 500 videos. A full-scale version of the dataset will be publicly available after acceptance.

Dataset Details – Collection & Generation

In this section, we provide additional details on dataset construction, including the captioning engine, QA generation, and evaluation metrics.

Data Engine

We introduce a three-stage, human-in-the-loop caption engine that utilizes a multi-agent framework to generate high-quality video captions. First, domain experts will annotate attributes in correspondence with marine video. Second, we use a sequence of video frames as input to an LLM to generate captions. We use the prompt in Table 5 for the captioning agent. Then, we use a prompt in Table 6 with another LLM as a reviewer to score the semantic alignment between the synthetic caption and the manually annotated attributes. Third, an LLM refiner generates the final caption using the prompt in Table 7.

Instruction

You are given 8 frames sampled chronologically from a single underwater video. Treat all 8 frames as one video sequence, not as separate images. Write one complete English video-captioning sentence that contains fewer than 80 words, describes the central object’s identity, texture or pattern, color, size, and shape, includes up to three secondary objects, details the environment and lighting, mentions the video quality, describes the object motion, and defines the camera position and camera angle separately.

Instructions

You are an advanced intelligent chatbot specialized in evaluating the factual accuracy of generative outputs related to video descriptions. Your primary task is to meticulously compare a “Predicted Answer” with the provided “Ground Truth” (human-annotated attributes) and assess their factual consistency.

You will evaluate the alignment between the Predicted Answer and the Ground Truth across specific visual and contextual dimensions.

EVALUATION DIMENSIONS

Evaluate alignment across these 8 dimensions:

1. Objects (Presence & count of up to 4 main objects)

2. Object Attributes (Texture, size, shape, color of central objects)

3. Environment (Setting/location)

4. Lighting (Type/quality of illumination)

5. Video Clarity (Resolution/focus)

6. Motion/Behavior (Actions/movement)

7. Camera Angle (Perspective, e.g., low-angle, eye-level)

8. Camera Position (Framing/movement, e.g., close-up, panning)

SCORING RUBRIC (0-5 Integer Scale)

Please assign an average score for all these dimensions, using an integer from 0 to 5, where 5 indicates a perfect match and 0 signifies no relevance at all.

INPUT DATA

• Ground Truth (Human Annotation): [Insert Human Annotation/Attributes Here]

• Predicted Answer: [Insert Candidate Output Here]

Instructions

You are an expert video annotation editor. Your task is to correct a flawed “Predicted Caption” by incorporating the missing or incorrect details found in the provided “Ground Truth Attributes”.

INPUT DATA

• Original Predicted Caption: [Insert original caption here]

• Ground Truth Attributes: [Insert human annotations/attributes here]

• Lower-Scoring Caption: [List caption that their scores less than 3.5]

INSTRUCTIONS

1. Review the errors and omissions in the Predicted Caption based on the Ground Truth.

2. Rewrite the caption to seamlessly integrate the correct attributes (objects, textures, sizes, shapes, colors, environment, lighting, motion, and camera details).

3. Do not hallucinate. Ensure every detail in the corrected caption matches the Ground Truth perfectly.

4. Keep the caption concise, natural-sounding, and visually descriptive.

OUTPUT FORMAT

Corrected Caption: [Insert your newly edited, factually flawless caption here]

Multiple-choice Generation

For multiple-choice questions, we use fine-grained attributes as answer candidates. We then use an LLM prompt in Table 8 with rules in Table 9 to generate the final answer choices.

Instructions You are an expert dataset creator for Video Question-Answering in the marine biology domain. Your task is to transform a row of raw video annotation data (Ground Truth) into a dynamic list of high-quality QA pairs following a strict JSONL schema.

INPUT You will receive a JSON object representing the Ground Truth (GT) of a single video. It contains attributes such as: video_id, collection, central_object, color, size_shape, objects_present, camera_position, camera_angle, movement_behavior, environment, and lighting.

OUTPUT FORMAT You must output the results in JSON Lines (JSONL) format. This means you will output between 6 to 9 individual JSON objects (depending on the presence of optional attributes). Each object must be on its own single line, with no trailing commas between lines, and they MUST NOT be wrapped in a JSON array.

Instructions

QUESTION GENERATION RULES

A. Mandatory Questions (Always generate these 6):

1. type: ”central_object” (MCQ)

- Question: ”What is the central object in the video?”

- Correct Answer: The exact value of central_object.

2. type: ”central_object_appearance” (MCQ)

- Question: ”Which color and size or shape description best matches the [insert central object here]?”

- Correct Answer Format: ”color: [GT color]; size/shape: [GT size shape]”.

3. type: ”object_detect” (MCQ)

- Question: ”Which option lists only objects present in the video? An option is incorrect if it includes any object not shown.”

- Correct Answer: A comma-separated list combining the central_object AND all secondary objects (e.g., object_2, object_3). If there are NO secondary objects in the GT, the correct answer is simply the central_object by itself. Do NOT use any fallback text.

4. type: ”camera_motion” (MCQ)

- Question: ”Which camera position or camera movement best matches the video?”

- Correct Answer Format: ”camera position/view: [GT camera_position]; camera angle/movement: [GT camera_angle]”.

5. type: ”camera_object_open_ended” (open_ended)

- Question: ”How do the camera movement and viewpoint capture the [insert central_object here] and the visible object activity in the video?”

- Reference Answer: Construct a fluent sentence explicitly combining central_object, movement_behavior, camera_position, and camera_angle.

6. type: ”caption” (open_ended)

- Question: ”Write one complete English video-captioning sentence that contains fewer than 80 words, describes the central object’s identity, texture or pattern, color, size, and shape, includes up to three secondary objects, details the environment and lighting, mentions the video quality, describes the object motion, and defines the camera position and camera angle separately.”

- Reference Answer: Use video captions from the dataset GT

B. Conditional Questions (ONLY generate if the data exists in GT):

7. type: ”central_object_motion” (MCQ)

- CONDITION: Only generate this question if movement_behavior describes the movement or state of the central_object.

- Question: ”Which movement or state of the [central_object], best matches the video?”

- Correct Answer: The movement/state of the central_object only.

- IMPORTANT: All answer options must describe the same central_object. Do not use other objects or species as the subject of any option.

8. type: ”environment” (MCQ)

- CONDITION: ONLY generate this if environment is NOT empty or null.

- Question: ”How is the environment in the video?”

- Correct Answer: The exact value of environment.

9. type: ”lighting” (MCQ)

- CONDITION: ONLY generate this if lighting is NOT empty or null.

- Question: ”How is the lighting in the video?”

- Correct Answer: The exact value of lighting.

Here is the Ground Truth data for the current video. Please evaluate the conditional rules, generate the appropriate number of JSON objects with correct sequential indexing (q1, q2, etc. without skipping numbers), and compile them directly into a valid JSONL format exactly like a .jsonl file (one valid JSON object per line).

Evaluation Dimensions of Open-ended QA

Open-ended with motion understanding. We use video captions as facts to create question-answer pairs, evaluating the VLM’s ability to understand camera motions. We use a LLM judge with prompts in Table 10 to evaluate a model’s capability, aiming to decode the functional relationship bebetween camera movement and the corresponding object’s attributes.

Instructions

You are evaluating an open-ended Video Question Answering response.

Use only the structured ground-truth evidence supplied below. The reference answer is an example of a correct response, not a string that must be matched word-for-word. Accept accurate paraphrases and different sentence structures. Do not reward unsupported detail, verbosity, or stylistic similarity.

Score each criterion with an integer from 0 to 5:

1. Camera Motion and Viewpoint (weight 35%): Does it correctly describe the camera position, framing, angle, and movement contained in the GT?

2. Object Activity (weight 30%): Does it correctly describe the annotated movement, behavior, or stationary state of the central/supporting objects?

3. Camera-Object Relationship (weight 35%): Does it explicitly explain how the camera movement/viewpoint records, follows, reveals, or frames the object activity?

Scoring anchors:

– 5: fully correct and complete for that criterion.

– 4: correct main content with one minor omission.

– 3: partially correct but misses an important detail.

– 2: limited relevant content or includes a meaningful error.

– 1: minimally related.

– 0: absent, completely incorrect, or non-responsive.

Return only JSON matching the requested schema.

Question

{question}

Reference answer

{reference_answer}

Predicted answer

{predicted_answer}

Video captioning. We aim to measure the model’s resistance to hallucinations in Table 11. Specifically, we evaluate whether the objects referenced in the generated descriptions correctly match the objects specified in the ground truth. For action hallucination, we assess whether the described object motion in the predicted answer and reference answer is consistent with the actual movements, using this prompt. Finally, we also use the mentioning prompt to identify discrepancies by checking whether the generated description includes any facts or elements that are not present in reality.

Instructions

You are an advanced intelligent chatbot specialized in evaluating the factual accuracy of generative outputs related to object-based video descriptions.

Your primary task is to meticulously compare the predicted answer with the correct answer and assess their factual consistency. To accomplish this, evaluate the descriptions across three key dimensions:

1. Subject Correspondence: Assess whether the subject of the input description accurately matches the subject indicated in the ground truth description.

2. Temporal Description: Analyze whether the depiction of the object’s motion aligns with the actual movements described in the correct answer.

3. Hallucination Detection: Identify any discrepancies by determining if the description includes facts, actions, or elements that do not exist in reality, such as imaginative interpretations or incorrect inferences.

Assign an integer score from 0 to 5 for each dimension, where 5 indicates a perfect match and 0 signifies no relevance at all.

For Hallucination Detection, 5 means no hallucination and 0 means severe hallucination. Accompany each score with a brief explanation.

Return only JSON matching the requested schema.

Correct answer:

{correct answer}

Predicted answer:

{predicted answer}

Instructions You are an expert dataset creator for Video Question-Answering in the marine biology domain. Your task is to transform a row of raw video annotation data (Ground Truth) into a dynamic list of high-quality QA pairs following a strict JSONL schema.

INPUT You will receive a JSON object representing the Ground Truth (GT) of a single video. It contains attributes such as: video_id, collection, central_object, color, size_shape, objects_present, camera_position, camera_angle, movement_behavior, environment, and lighting.

OUTPUT FORMAT You must output the results in JSON Lines (JSONL) format. This means you will output between 6 to 9 individual JSON objects (depending on the presence of optional attributes). Each object must be on its own single line, with no trailing commas between lines, and they MUST NOT be wrapped in a JSON array.

Each JSON object MUST strictly follow this schema:

{
  "question_id": "{collection}--{video_file_name_without_ext}--q{index}",
  "id": "{collection}--{video_file_name_without_ext}",
  "task": "qa",
  "type": "<type_of_question>",
  "question_format": "multiple_choice" OR "open_ended",
  "collection": "{collection}",
  "video_id": "{video_id}",
  "video_file": "{video_file_name_with_ext}",
  "video_path": "{collection}/{video_file_name_with_ext}",
  "central_object": "{central_object}",
  "question": "<The standalone question>",
  "prompt": "<For MCQ: The question + options (A) (B) (C) (D) + ’Answer with the option letter only.’ | For open-ended: Just the question>",
  "options": [
    {"label": "A", "text": "<Option A text>"},
    {"label": "B", "text": "<Option B text>"},
    {"label": "C", "text": "<Option C text>"},
    {"label": "D", "text": "<Option D text>"}
  ] (Empty array [] for open-ended),
  "answer": "<A, B, C, or D> (Empty string for open-ended)",
  "answer_text": "<Text of the correct answer or reference answer>",
  "evidence_attributes": { "<key>": "<gt_value_used>" }
}

Instructions

QUESTION GENERATION RULES

A. Mandatory Questions (Always generate these 6):

1. type: ”central_object” (MCQ)

   • Question: ”What is the central object in the video?”

   • Correct Answer: The exact value of central_object.

2. type: ”central_object_appearance” (MCQ)

   • Question: ”Which color and size or shape description best matches the [insert central object here]?”

   • Correct Answer Format: ”color: [GT color]; size/shape: [GT size shape]”.

3. type: ”object_detect” (MCQ)

   • Question: ”Which option lists only objects present in the video? An option is incorrect if it includes any object not shown.”

   • Correct Answer: A comma-separated list combining the central_object AND all secondary objects (e.g., object_2, object_3). If there are NO secondary objects in the GT, the correct answer is simply the central_object by itself. Do NOT use any fallback text.

4. type: ”camera_motion” (MCQ)

   • Question: ”Which camera position or camera movement best matches the video?”

   • Correct Answer Format: ”camera position/view: [GT camera_position]; camera angle/movement: [GT camera_angle]”.

5. type: ”camera_object_open_ended” (open_ended)

   • Question: ”How do the camera movement and viewpoint capture the [insert central_object here] and the visible object activity in the video?”

   • Reference Answer: Construct a fluent sentence explicitly combining central_object, movement_behavior, camera_position, and camera_angle.

6. type: ”caption” (open_ended)

   • Question: ”Write one complete English video-captioning sentence that contains fewer than 80 words, describes the central object’s identity, texture or pattern, color, size, and shape, includes up to three secondary objects, details the environment and lighting, mentions the video quality, describes the object motion, and defines the camera position and camera angle separately.”

   • Reference Answer: Use video captions from the dataset GT

B. Conditional Questions (ONLY generate if the data exists in GT):

7. type: ”central_object_motion” (MCQ)

   • CONDITION: Only generate this question if movement_behavior describes the movement or state of the central_object.

   • Question: ”Which movement or state of the [central_object], best matches the video?”

   • Correct Answer: The movement/state of the central_object only.

   • IMPORTANT: All answer options must describe the same central_object. Do not use other objects or species as the subject of any option.

8. type: ”environment” (MCQ)

   • CONDITION: ONLY generate this if environment is NOT empty or null.

   • Question: ”How is the environment in the video?”

   • Correct Answer: The exact value of environment.

9. type: ”lighting” (MCQ)

   • CONDITION: ONLY generate this if lighting is NOT empty or null.

   • Question: ”How is the lighting in the video?”

   • Correct Answer: The exact value of lighting.

Here is the Ground Truth data for the current video. Please evaluate the conditional rules, generate the appropriate number of JSON objects with correct sequential indexing (q1, q2, etc. without skipping numbers), and compile them directly into a valid JSONL format exactly like a .jsonl file (one valid JSON object per line).