DramaVQA: Probing Detailed Understanding of Ultra-Long TV Dramas

Introduction

The rapid evolution of multimodal large language models (MLLMs) has greatly expanded the limits of video understanding. Recent state-of-the-art models, such as Gemini-3-Pro Team et al. [51] and the Qwen-VL series Bai et al. [4], now support very large context windows (e.g., 128K input tokens). This allows them to achieve remarkable accuracy on question-answering (QA) tasks for hour-long videos previously considered impossible. However, as model capacities grow, the community lacks a benchmark that pushes the frontier toward ultra-long contexts, specifically videos lasting tens of hours, where processing the entire video at once is no longer computationally practical.

To address this gap, we introduce DramaVQA, a large-scale benchmark built upon long-form TV dramas to encourage research in ultra-long video understanding. TV dramas serve as an ideal testbed because they feature continuous, detailed storylines driven by character growth and long-range cause-and-effect relationships. This structure allows us to create challenging questions that require finding key information across dozens of episodes. As a result, it rigorously tests MLLM capabilities across multiple dimensions, including visual recognition, temporal grounding, and high-level summarization.

The main hurdle in creating such a benchmark is the high cost of human annotation and the difficulty of ensuring quality control. To overcome this, we propose a scalable, LLM-assisted annotation pipeline with four key stages: (1) extracting structured metadata from the dramas (e.g., subtitles, facial identities, speaker identities) and using MLLMs to generate multi-level video captions; (2) using LLMs to generate high-quality candidate QA pairs from these captions; (3) employing human

caption: DramaVQA excels in QA number, average duration, and other attributes among recent long-form video benchmarks. MG: multi-granular (each QA supports different durations); VM: video metadata available (subtitles, faces, speaker data, hierarchical video captions); TG: temporal grounding provided (only for 0.2h+ videos).

:::

annotators to carefully filter and correct wrong or ambiguous QAs; and (4) using LLMs one last time to verify that the remaining questions are unique across the entire drama. This method allows DramaVQA to reach an unprecedented scale of approximately 20K QA pairs, an order of magnitude larger than existing long-video datasets like Video-MME Fu et al. [19] and LVBench Wang et al. [58]. Table 1 shows a detailed comparison between DramaVQA and recent benchmarks.

This construction process naturally gives DramaVQA a multi-level time structure. Each QA pair can be tested using different input video lengths, categorized into four distinct temporal tracks: Medium (averaging 7.6 minutes), Long (averaging 43.1 minutes), Extra-Long (averaging 9.3 hours), and Ultra-Long (the full drama, averaging 38.9 hours). Notably, the latter two tracks are much longer than existing benchmarks (e.g., the average durations of Video-MME-Long and LVBench are only 0.69 and 1.14 hours, respectively). Our extensive evaluation of state-of-the-art MLLMs shows a significant performance drop as the video length increases. While current models perform reasonably well on the Medium track, their accuracy drops significantly on the Long and track, and falls to near-random guessing levels on the Extra-Long and Ultra-Long tracks. This sharp drop shows a major weakness in current architectures: they rely too heavily on brute-force context scaling and cannot handle ultra-long videos.

Beyond providing a strict testbed for zero-shot video QA, DramaVQA pushes the field forward in another important way. First, we will release over 400K high-quality QA pairs. Although many were filtered from the final benchmark for failing the strict uniqueness test, all these QAs have been human-validated and include precise timestamps, offering a highly valuable, large-scale dataset for MLLMs. Unlike traditional benchmarks that provide only 3–20 questions per video, DramaVQA supplies an average of 1,600+ questions per drama. This high density of queries over massive time scales challenges the community to move beyond simple, single-pass processing. Instead, it motivates a necessary shift toward memory-efficient long video understanding, where models must organize video streams into compact, easily searchable memory formats to perform complex reasoning within practical computational limits.

Related Work

Multimodal large language models (MLLMs). The field of computer vision has greatly changed with the rise of MLLMs Hurst et al. [23]; Chen et al. [9]; Wang et al. [57]; Bai et al. [4]. By building on the strong abilities of basic LLMs Touvron et al. [56]; Achiam et al. [2]; Team et al. [51]; Guo et al. [20]; Yang et al. [64], these models convert visual signals into discrete tokens, easily combining them with text prompts Liu et al. [33]; Zhu et al. [71]; Kim et al. [25]; Tong et al. [55]. This step has moved the field past simple, closed-set tasks (e.g., basic detection or image classification) into open-ended video question answering Maaz et al. [35]; Lin et al. [32]; Zhang et al. [70]; Li et al. [29]. Furthermore, ongoing model improvements allow them to process much larger sequences of visual tokens, opening up new ways to understand longer videos.

Benchmarks for long-form video understanding. Since current models have largely mastered static images and short video clips, research has shifted toward longer video analysis. Recent large-scale datasets focused on hour-long videos and complex QA tasks (e.g., EgoSchema Mangalam et al. [36], LongVideoBench Wu et al. [62], Video-MME Fu et al. [19], LVBench Wang et al. [58], CG-Bench Chen et al. [6]) have created major challenges for modern vision systems. The main trend among these datasets is the steady increase of the input length (as shown in Table 1), which challenges MLLMs to integrate massive amounts of video information into their context windows. Meanwhile, some benchmarks Cheng et al. [10]; Liu et al. [34] use minute-level videos paired with complex logical questions to test the reasoning limits of these models. Although LongTimeScope Zohar et al. [72] and Ego-R1 Bench Tian et al. [54] cover much longer videos, they contain only 450 and 300 evaluation questions, respectively, with limited coverage of fine-grained understanding within individual videos. To address this, our research delivers the largest benchmark built to date for rigorous, ultra-long video reasoning, constructed specifically from long-form TV dramas. While datasets like MovieQA Tapaswi et al. [50], MovieChat Song et al. [46], DramaQA Choi et al. [11], and MoVQA Zhang et al. [69] also focus on video understanding using movie data, their video durations are still at the sub-hour level. By using multi-episode TV dramas, we make it possible to extend the video duration to the 10-hour level.

Advanced solutions for long-form video understanding. To mitigate inherent context limitations of MLLMs, existing methods fall into two categories: video compression that condenses visual content for efficient context utilization Li et al. [31]; Lee et al. [26]; Shen et al. [44]; Choudhury et al. [12], Zhang et al. [68], and chunk-based processing that segments long videos for step-wise inference Song et al. [47]; Tang et al. [49]; Zhang et al. [67]. Integrated with powerful reasoning models, chunking solutions have evolved into agentic frameworks, dominating mainstream benchmarks with rational reasoning and tool scheduling Hurst et al. [23]; Guo et al. [20]; Yang et al. [64]; Fan et al. [17]; Wang et al. [59, 61]; Feng et al. [18]; Tang et al. [48]; Qiu et al. [39]. Targeting extreme ultra-long video scenarios, DramaVQA further motivates robust agentic designs and memory-based mechanisms to address long-range video understanding challenges Wang et al. [60]; Diko et al. [16]; Jin et al. [24]; Zuo et al. [73].

The DramaVQA Benchmark

The overall pipeline for building the DramaVQA benchmark is shown in Figure 1. This process consists of two main stages: (1) data preparation, where we extract structured metadata from the TV dramas and use an MLLM to generate detailed video descriptions across different time levels; and (2) QA generation and validation, which combines LLM generation with strict human checking to ensure both the large scale and the high quality of the dataset.

Data Preparation

Data collection. Our dataset comprises 15 long-form TV series spanning multiple seasons, with a total runtime of 606 hours. Specifically, DramaVQA includes 12 totalling 466 hours, and the other three series are reserved for the training split. We extract metadata through a four-stage pipeline. We process each drama $\mathcal{V}$ individually; for simplicity, denote $T$ as its total duration, and omit the drama index for the rest of this work. Details of this part can be found in Appendix § 6.

1. Transcript and text extraction. We use PaddleOCR Cui et al. [14] followed by Qwen3-VL-32B Bai et al. [3] to extract and refine hardcoded subtitles and in-screen texts. This produces a sequence of utterances $\mathcal{U} = \{u_1, \ldots, u_N\}$, where each $u_n = (w_n, [s_n, e_n])$ contains the text $w_n$ and start/end timestamps $s_n, e_n \in[0, T]$, and a set of texts $\mathcal{T} = \{t_1, \ldots, t_M\}$, where each $t_m = (w_m, [s_m, e_m])$.

2. Character library construction. We build a cast roster $\mathcal{C}$ from credits and web data. We initialize a facial registry with official portraits and enrich it using 1-FPS video sampling to capture varied appearances. Then, a frame-by-frame processing is performed to detect all facial instances $\mathcal{F} = \{f_1, \ldots, f_R\}$. Here, each $f_r = (c_r, t_r, b_r)$ links a detected face to an identity $c_r \in\mathcal{C}$ at timestamp $t_r$ and bounding box $b_r$.

3. Speaker label annotation. We combine audio features within $[s_n, e_n]$ and visual instances $\mathcal{F}$ to assign initial speaker labels using a label propagation algorithm to find ‘who spoke when’. Human labelers then perform a sentence-by-sentence check following a well-established guideline, updating each utterance $u_n$ with a final speaker identity $c_n \in\mathcal{C}$.

4. Hierarchical video captioning. We use a multi-level temporal structure (L1–L6) to summarize the long video. First, the video is split into 10–15s base units (called L1 units) $h_k^{(1)} \in\mathcal{H}^{(1)}$ using the PySceneDetect algorithm Castellano [5] with CLIP similarity Radford et al. [41]. For these units, Qwen3-VL-32B Bai et al. [3] generates visual descriptions $d_k^{(1)}$. Next, we cluster these into L2 units and use Qwen3-32B Yang et al. [64] to summarize them. This repeats across six levels, $\mathcal{H}^{(l)}$ for $l \in\{1, \ldots, 6\}$. Each segment is formalized as $h_k^{(l)} = (d_k^{(l)}, [s_k^{(l)}, e_k^{(l)}])$, with start and end timestamps $s_k^{(l)}$ and $e_k^{(l)}$. This ends with a single L6 summary for the full drama $\mathcal{V}$.

QA Generation and Validation

We design a sophisticated pipeline to generate QA pairs from the multi-level captions, based on three main rules: (i) diversity, ensuring a wide coverage of question types; (ii) reducing language prior, making sure models cannot answer questions using only their internal knowledge (like drama scripts in their training data) without seeing the video; and (iii) global uniqueness, ensuring questions are not repeated across the entire drama to test ultra-long reasoning. To prevent the data from favoring any specific model, we use several different LLMs across the different stages. Details of all these steps are in Appendix § 7.

1. Initial QA generation. We use Gemini-2.5-Pro to generate initial QA pairs from L2 captions $h_k^{(2)}$. These segments, with an average length of 1.2 minutes, offer a good balance between visual details and the main storyline. Following previous methods (e.g., LLaVA-Video-178K Zhang et al. [70]), we ask for different question types (with demonstrations) in the prompt, including visual perception, spatial understanding, action & interaction, event understanding, temporal understanding, relation & reasoning, and counting. All entries are single-choice questions with 2–8 (average 6+) options.

2. Rule and LLM-based filtering. To reduce hallucination and internal knowledge prior, we first use an LLM (Qwen3-32B) to remove ‘self-answering’ questions where the answer is already in the question text, as well as invalid questions that rely on segment-specific time clues (e.g., ‘at the beginning of the clip’). In addition, two LLMs (GPT-5 and Gemini-2.5-Pro) are asked to answer the questions based only on the text options; any QA answered correctly by both models is removed.

3. Human checking and editing. We recruit human labelers to check the remaining questions. Labelers are assigned L2 segments and asked to watch the video $h_k^{(2)}$ to make sure each

QA meets three rules: (a) the question matches the video and is concise; (b) the choices are clear; (c) the answer is correct and unambiguous; and (d) the question is not repeated. We implement a two-stage labeling procedure, where the second pass rigorously double-checks and improves the quality of the first pass.

4. Language prior removal. After human editing, we give the QAs an even stricter text-only test using three models: GPT-5, Gemini-2.5-Pro, and Seed1.8 Pro. To further increase the difficulty, we only keep QA pairs that are answered incorrectly by all three LLMs. This high standard ensures a strong gap between text-only and vision-language results.

5. Global uniqueness check. We use a global search algorithm to make sure each QA is unique across the entire drama length $T$. We combine language embeddings from the bge-large-en/zh-v1.5 model Xiao et al. [63] and the BM25 algorithm Robertson and Zaragoza [42] to find similar L2 segments. Fo the top-ranked similar segments, an LLM extracts keywords from the QA; if another L2 segment contains these keywords, the QA is marked as not unique and removed. This ensures that the question can only be answered by referring to the specific target video segment, thereby improving the uniqueness and discriminative power of the QA pairs.

The dataset contains 25,404 L2 segments, of which 20,278 belong to DramaVQA. After the first four steps, the average numbers of QA pairs per L2 segment are 103.1, 27.3, 18.8, and 5.4, respectively. The training set retains all QA pairs (about 15k) generated after Step 4. These pairs are then processed in Step 5 to annotate the uniqueness level of each question. The benchmark set retains only QA pairs that are globally unique within their entire drama after Step 5, resulting in 1.7 QAs per L2 segment. We then sample the final 20,000 questions to ensure balanced distributions across question types and video timestamps.

Benchmark Statistics

We have established 20K QA pairs across 12 TV dramas, representing a scale one order of magnitude larger than existing long-video benchmarks. Furthermore, the filtered QAs (400K+), while not unique throughout the entire drama, can also be used as training data and served as a high-quality resource for supervised fine-tuning and advancing long-form video understanding. An illustration of seven basic categories and their breakdown into 28 sub-categories is shown in Figure 2. Detailed dataset statistics are provided in Appendix § 6 (Table 4).

Benchmark	#Vids	Dura.	#QAs	#Faces	#Spkrs	Sub Len.	Cap Len.	QA Len.
Video-MME-S	300	0.02h	3.0	N/A	N/A	198.6	N/A	28.7
Video-MME-M	300	0.14h	3.0	N/A	N/A	1425.6	N/A	32.8
Video-MME-L	300	0.69h	3.0	N/A	N/A	6515.6	N/A	45.6
LVBench	103	1.14h	15.0	N/A	N/A	N/A	N/A	45.4
DramaVQA-M	3,699	0.13h	5.4	14,594.7	126.3	829.0	19,098.5	80.5
DramaVQA-L	649	0.72h	30.8	83,183.1	720.0	4,724.7	109,286.1	80.5
DramaVQA-XL	50	9.32h	400.0	1.079,716.9	9,345.5	61,326.7	1,418,995.0	80.5
DramaVQA-UL	12	38.85h	1666.7	4,498,820.5	38,939.8	255,528.1	5,913,022.8	80.5

Because each preserved question is verified to be unique throughout the entire drama duration $T$, DramaVQA supports multi-granular evaluation. In our experiments, we use L3 through L6 segments to define four distinct evaluation tracks: Medium (averaging 7.6 minutes), Long (averaging 43.1 minutes), Extra-Long (averaging 9.3 hours), and Ultra-Long (the full drama, averaging 38.9 hours). Notably, our Medium and Long tracks are similar in length to the “long-form” subsets of existing benchmarks, while the Extra-Long and Ultra-Long tracks present an unprecedented challenge for modern MLLMs.

Another interesting feature of our benchmark is the high degree of inter-connectivity between QA pairs within a single drama. For example, the answer to one question may serve as a condition for another. These connections enable us to construct complex tasks that require multi-hop reasoning or retrieving and integrating multiple pieces of information, providing a rigorous assessment of MLLMs’ reasoning and multi-point information retrieval capabilities. We present a preliminary study of these complex reasoning tasks in § 4.5.

Experiments

Models and Evaluation Metrics

Models. We evaluate two main groups of models to build a strong baseline. The first group includes state-of-the-art MLLMs, such as Qwen3-VL-8B/32B Bai et al. [3], Qwen3.5-27B Team [52], Qwen3.6-27B Qwen Team (2026), GLM-4.6V Hong et al. [21] and GLM-5.3-Flash-FP8 Team [53]. These models uniformly sample a set number of frames (256 or 512) from the input video, tokenize them as visual context, and then perform inference. We strictly follow the default settings for these models; Thinking settings are specified in Table 2. The second group features recently released, agentic MLLMs, namely Conan Ouyang et al. [38] and Long Video-R1 Qiu et al. [39], which start with a basic video input and dynamically sample extra frames for question-answering.

Model	Fr.	Th.	Medium (L3) Mini [Full] (time)	Long (L4) Mini [Full] (time)	Extra-long (L5) Mini [Full] (time)	Ultra-long (L6) Mini [Full] (time)
Human	–	–	81.0 (215)	–	–	–
No-vision Models
Random Guess	0.0	✗	16.2 [16.2] (0.0)	16.2 [16.2] (0.0)	16.2 [16.2] (0.0)	16.2 [16.2] (0.0)
Language Only (Q3)	256	✗	21.5 [22.2] (0.1)	21.5 [22.2] (0.1)	21.5 [22.2] (0.1)	21.5 [22.2] (0.1)
Language Only (Q3.5)	256	✓	23.1 [23.3] (33.9)	23.1 [23.3] (33.9)	23.1 [23.3] (33.9)	23.1 [23.3] (33.9)
Proprietary Models
Seed2.0 Pro Seed (2026)	256	✓	73.2 (48.5)	49.2 (63.4)	32.2 (60.1)	30.7 (49.2)
GPT-5.5 OpenAI [37]	256	✗	67.2 (64.6)	45.2 (55.5)	29.9 (81.0)	27.6 (56.1)
Gemini-3.5-Flash Team et al. [51]	256	✗	77.5 (38.5)	61.2 (36.4)	43.9 (36.8)	37.1 (56.0)
Open-source Native MLLMs
GLM-4.6V-106B Hong et al. [21]	256	✗	51.0 (15.4)	34.5 (14.8)	27.4 (14.3)	26.5 (12.6)
GLM-4.6V-106B Hong et al. [21]	256	✓	57.1 (19.5)	36.6 (18.5)	27.1 (16.1)	25.4 (15.1)
GLM-5.3-Flash-FP8 Team [53]	256	✓	70.5 (31.7)	49.9 (14.4)	34.9 (14.3)	32.1 (14.2)
Qwen3-VL-32B Bai et al. [3]	256	✓	62.0 (20.5)	37.9 (18.3)	26.4 (14.9)	24.9 (16.8)
Qwen3.5-27B Team [52]	256	✗	69.4 (26.3)	47.1 (16.6)	35.7 (13.2)	34.4 (12.2)
Qwen3-VL-8B Bai et al. [3]	256	✗	40.9 [42.3] (17.9)	28.7 [30.2] (11.5)	24.8 [25.1] (13.0)	22.7 [24.1] (11.2)
Qwen3-VL-8B Bai et al. [3]	256	✓	54.2 [56.3] (19.2)	36.5 [37.7] (17.6)	24.6 [26.7] (16.3)	23.4 [24.9] (12.6)
Qwen3-VL-32B Bai et al. [3]	256	✗	51.0 [51.3] (22.6)	35.1 [35.0] (17.9)	27.8 [27.4] (18.2)	26.4 [25.9] (17.4)
Qwen3.6-27B Qwen Team (2026)	256	✗	69.9 [67.1] (50.5)	48.0 [49.5] (23.8)	35.2 [36.9] (22.5)	33.0 [34.6] (20.1)
Open-source Native MLLMs
GLM-4.6V-106B Hong et al. [21]	512	✗	55.0 (28.0)	37.0 (29.3)	27.4 (31.1)	26.6 (30.6)
GLM-4.6V-106B Hong et al. [21]	512	✓	60.8 (34.7)	40.1 (34.6)	25.6 (35.0)	24.6 (37.5)
GLM-5.3-Flash-FP8 Team [53]	512	✓	75.4 (28.2)	56.1 (33.5)	36.1 (31.3)	32.1 (32.6)
Qwen3-VL-32B Bai et al. [3]	512	✓	66.1 (28.6)	43.7 (33.3)	26.4 (30.7)	24.1 (21.3)
Qwen3.5-27B Team [52]	512	✗	72.2 (27.4)	51.0 (27.6)	37.1 (26.4)	33.9 (23.8)
Qwen3-VL-8B Bai et al. [3]	512	✗	41.7 [42.5] (23.3)	30.4 [31.1] (26.0)	24.1 [25.4] (26.8)	22.6 [24.2] (29.0)
Qwen3-VL-8B Bai et al. [3]	512	✓	57.2 [58.9] (29.8)	40.0 [41.0] (31.5)	26.9 [27.5] (32.2)	25.0 [25.2] (26.1)
Qwen3-VL-32B Bai et al. [3]	512	✗	54.8 [53.3] (33.3)	35.8 [37.2] (36.4)	28.2 [27.5] (37.6)	25.4 [26.2] (37.9)
Qwen3.6-27B Qwen Team (2026)	512	✗	72.5 [70.8] (41.4)	55.0 [54.9] (40.9)	37.9 [38.6] (36.8)	34.1 [35.1] (30.4)
Open-source Agentic MLLMs
Conan Ouyang et al. [38]	256	✓	48.8 [50.1] (63.3)	36.3 [37.1] (60.5)	29.7 [30.9] (59.7)	28.6 [29.9] (68.3)
Conan Ouyang et al. [38]	512	✓	49.0 [48.8] (87.5)	37.7 [37.9] (79.5)	29.7 [30.5] (79.6)	29.6 [29.5] (84.1)
RAG-BS (Q3)	var	✗	47.4 [47.1] (180.4)	33.5 [33.7] (372.1)	25.8 [25.7] (408.2)	24.8 [24.7] (406.5)
Naive-RAG (Q3)	var	✗	51.7 [50.4] (292.9)	46.6 [45.9] (1664.5)	29.7 [29.1] (1955.5)	26.3 [25.3] (1955.0)
RAG-BS (Q3.8)	var	✗	56.4 [56.0] (180.5)	40.4 [41.1] (372.2)	32.1 [32.7] (408.3)	30.9 [31.5] (406.5)
Naive-RAG (Q3.8)	var	✗	60.1 [59.8] (293.0)	56.7 [56.1] (1664.6)	36.9 [36.8] (1955.5)	31.5 [32.7] (1955.5)
LongVideo-R1 Qiu et al. [39]	var	✓	62.8 [63.7] (70.1)	54.8 [55.3] (90.1)	34.7 [34.0] (98.3)	27.2 [26.6] (93.2)
LongVideo-R1 (post-tuned or on data)	var	✓	66.1 [67.5] (61.7)	60.0 [61.4] (113.7)	40.0 [40.9] (163.8)	30.8 [30.5] (172.0)

Furthermore, we test a simple baseline named Naive-RAG that searches through the clip captions generated across the video. This method shares the same input structure as Long Video-R1 Qiu et al. [39]: for each input video, we build a 3-level tree structure, where each clip is evenly divided into $D = 6$ smaller clips at the next level. We extract a caption for every clip and use BM25 search to find the top-3 most relevant captions at each level. Finally, these 9 captions are combined and fed into an LLM to answer the question. We also develop a beam-search variant (RAG-BS) where, at the second and third levels, the search is strictly limited to the child nodes of the top-3 captions chosen in the previous level.

Data. We test the models on the entire set of 20K QA pairs created in the previous section. As explained earlier, each question is tested using different lengths of video input. These correspond to four distinct tracks: Medium, Long, Extra-Long, and Ultra-Long. To make it easier for the community to reproduce our results, we also build a ‘mini’ subset containing about 5% of the main test set. On this smaller set, we test three proprietary models, Seed2.0-Pro Seed (2026), GPT-5.5 OpenAI [37] and Gemini-3.5-Flash Team et al. [51], and compare them against the open-source models.

Evaluation Protocol.

For each sampled frame, original hardcoded subtitles are retained, forcing models to read text directly from images. To aid character recognition, we draw red bounding boxes for detected faces and add name labels. Models are required to output a single option (e.g., [A]). For each QA pair, we repeatedly prompt MLLMs for up to five attempts. If no valid answer is provided, we assign a random-guess score of $1/X$, where $X$ denotes the number of options. We report both QA accuracy and computational overhead³. Notably, the overhead of agentic methods, including RAG, covers video captioning costs. Full experimental settings are provided in Appendix C.1.

Main Results

Table 2 summarizes all test results. We first discuss the available results on the full-set that contains all 20,000 QA pairs. The key observations are listed below.

The foundational QA tasks remain challenging. Even within the Medium track, the tested models leave substantial room for improvement. Specifically, Qwen3.6-27B Qwen Team (2026) achieves the highest reported full-set accuracy at 70.8% with 512 frames, but its accuracy falls to 35.1% on the Ultra-Long track. This difficulty comes from three main factors: (i) our data preparation pipeline adds dense information into the video captions, allowing us to create difficult questions that require deep reasoning and an understanding of complex character relationships; (ii) most questions have 6–8 highly competitive (yet clear) options; and (iii) we removed simpler questions (those solvable via language priors) through our strict text-only filtering step.

Performance decreases as the input video duration increases. The average accuracy on the Long track is noticeably lower than that on the Medium track, and performance on the Extra-Long and Ultra-Long tracks drops close to the random guess baseline. This trend matches our expectation: as the video length grows, the chance of a model successfully sampling the key frames containing the answer becomes much lower. This also explains why all models show improved accuracy when the context window increases from 256 to 512 sampled frames. However, even with 512 frames, the performance remains close to the random baseline in the Ultra-Long track.

Stronger base MLLMs produce better results. Under the same 512-frame, non-thinking setting, Medium accuracy increases from 42.5% for Qwen3-VL-8B to 53.3% for Qwen3-VL-32B Bai et al. [3] and 70.8% for Qwen3.6-27B Qwen Team (2026). The corresponding Ultra-Long scores are 24.2%, 26.2%, and 35.1%, respectively. These results support DramaVQA as a useful testbed for evaluating foundation models. However, the smaller absolute gains on the longest track indicate that stronger base models still face substantial limitations when processing extended video contexts.

Thinking mechanisms help, but their benefits depend on video length. For Qwen3-VL-8B with 512 frames, thinking raises Medium accuracy from 42.5% to 58.9% and Long accuracy from 31.1% to 41.0%. However, the gain on the Ultra-Long track is smaller, from 24.2% to 25.2%. Similar trends hold with 256 frames, where thinking yields larger improvements on the shorter tracks. These results suggest that reasoning helps models interpret evidence, while offering limited benefits when sparse sampling fails to capture the information required to answer a question.

Agent-based and retrieval-augmented algorithms show better accuracy but may require higher computational costs. The basic RAG approach, which searches across all short-clip captions, helps reduce the information loss caused by uniform frame sampling. However, this comes at the cost of increased time complexity, which grows with the input video length. Smarter agentic models (e.g., LongVideo-R1 Qiu et al. [39]) offer a better approach by dynamically sampling a small number of video clips. Still, they cannot always guarantee finding the exact time segments needed for the correct answer. This shows a clear trade-off between accuracy and computational cost, which we discuss further in § 4.4.

Results on the mini-set. Its performance trends closely align with the full dataset, validating the mini-set as a reliable, cost-effective proxy for fast experiments such as parameter tuning and prompt design. Even proprietary models struggle with ultra-long contexts: while Gemini-3.1-Pro outperforms open-source native MLLMs, its accuracy still declines markedly on Long and Ultra-Long tracks, mainly due to severe information loss from sparse frame sampling across extended videos.

RL on our training set. To assess the effectiveness of our training data in improving information retrieval and localization across video scales, we adopt the agent-based LongVideo-R1 Qiu et al. [39] as our baseline. Starting from its original checkpoint, we perform reinforcement learning on our training set. The resulting model outperforms the original LongVideo-R1 by at least 3.6% across all four evaluation tracks, demonstrating the effectiveness of our training data. RL training increases caption retrieval and reasoning activity, with average medium-caption calls rising from 2.38 to 4.65, low-caption calls from 3.12 to 4.66, and reasoning steps from 7.41 to 10.94 per question.

Summary of Table 2. With extensive experiments, we show that ultra-long video understanding remains a major challenge, especially under limited computational budgets. By showing the current

caption: Evaluation on multi-point and multi-hop questions. Single-QA denotes the base question taken from the second hop of the paired Multi-Hop-QA sample, with identical video input. Parenthesized values are the total number of questions per task.

:::

limits of existing models, DramaVQA offers a valuable testbed to help guide the future development of state-of-the-art MLLMs and agentic video understanding algorithms.

On QA categories. Figure 3 shows the accuracy distribution across different question categories. Notably, certain question categories (e.g., temporal understanding and counting) are more difficult than others (e.g., visual perception and event understanding). This difference happens mainly because the harder categories require complex logical reasoning or collecting pieces of information scattered across the video. See Appendix § 8.§ 2 for further analysis.

Failure analysis. Our analysis reveals three main causes for these errors. First, many mistakes happen because sparse frame sampling often misses important keyframes. Second, current models struggle to understand the overall story logic and cause-and-effect relationships. Finally, the lack of audio inputs makes it hard for models to answer dialogue-based questions, such as identifying ‘who claimed what’. This clearly shows the need to develop advanced tri-modal (audio-visual-text) MLLMs in the future. More examples are provided in Appendix § 8.§ 3.

Accuracy-Efficiency Tradeoff

We compare native MLLMs and agentic algorithms from the perspective of the accuracy-efficiency trade-off. In brief, while agentic methods introduce higher preprocessing cost in case-wise evaluation, they become substantially more competitive in batch-wise settings by sharing captioning cost across many questions from the same long video context. We also observe a practical trade-off between stronger reasoning performance and additional inference overhead. Detailed quantitative results, the full comparison table, and time-cost estimation are provided in Appendix § 9.

Multi-Point and Multi-Hop Reasoning: A Preliminary Study

We extend DramaVQA with multi-point and multi-hop questions composed from basic QA pairs to evaluate cross-segment evidence integration and compositional reasoning.

Question construction. Multi-point questions cover Two-Three-Event Temporal Comparison (TTC-2/3) and Cross-Event Character Identity (CECI). We sample events from distinct L3 segments within a drama and use templates to query their temporal order or whether characters identified by two base questions are the same. An LLM refines the wording, and we discard questions answered correctly by at least two of three text-only LLMs. For multi-hop questions, we pair questions whose first answer (typically a character name) appears in the second question. An LLM replaces this answer in the second question with a description derived from the first; annotators verify correctness and the necessity of the intermediate answer. See Appendix § 10 for detailed construction procedures, illustrative examples, and supplementary experimental analyses.

Evaluation. We uniformly sample 256 frames from the concatenated L3 segments associated with each composed question. Table 3 shows that Qwen3-VL and GLM-4.6V perform near or below chance on TTC-2 and CECI, and only slightly above chance on TTC-3. Despite stronger temporal comparison by Qwen3.5/3.6, CECI accuracy peaks at 59.0%. All models perform worse on multi-hop questions than on their paired second-hop questions, highlighting reasoning challenges beyond atomic QA. Together, these results reveal limitations in cross-segment evidence integration and compositional reasoning that atomic QA evaluation alone may not fully capture.

Conclusions

In this paper, we introduced DramaVQA, a large-scale benchmark designed to advance ultra-long video understanding. Built from long TV drama series, DramaVQA contains about 20K high-quality QA pairs and evaluates models from Medium clips to Ultra-Long videos lasting tens of hours. Our results show that current MLLMs still struggle on Extra-Long and Ultra-Long tracks, revealing clear limitations in long-range video reasoning. We also explored the accuracy-efficiency trade-off, finding that agent-based algorithms offer a better balance when answering many questions over long videos. Moving forward, we believe future research must shift from simply expanding context windows toward developing smart memory systems and video-driven world models that can efficiently manage and reason over long-term temporal contexts.

Data Preparation Details

Video Pre-processing

Each drama in our dataset contains multiple episodes. To ensure the story flows naturally, we first remove the opening and closing parts (titles and credits) of each episode. The remaining video parts are combined into a single, continuous video stream for each drama⁴. This step provides a longer time context for conversations that happen near episode boundaries. It helps the DramaVQA keep character consistency and identify speakers based on the overall story.

Transcription Extraction

We extract dialogue text from on-screen subtitles using an OCR and refinement pipeline.

• Initial subtitle extraction. For each drama, we define a box around the subtitle area to reduce background noise. We then use PaddleOCR-v4 Cui et al. [14] for frame-by-frame recognition, producing a raw text sequence $\{t_n\}$.

• Grouping texts into utterances. To combine continuous frames into separate dialogue lines, we calculate the normalized Levenshtein similarity (also known as 1 minus edit distance) between adjacent frames:

Frames with a similarity score higher than 0.8 are grouped into a single utterance.

• MLLM refinement. For each segment, the middle frame is processed by Qwen3-VL-32B Bai et al. [3]. This step fixes OCR errors, separates overlapping subtitles, and makes the text format consistent.

• Quality control. We manually check about 1% of the data, focusing on cases where the text length does not match the video duration or where the logical flow seems broken.

Character Library Construction

To build high-quality facial reference data, we use a two-stage pipeline that combines external and in-situ visual data:

• Initial face library. Actor details and reference photos are collected from public databases (e.g., IMDB and TMDB for English dramas, Douban and BaiduBaike for Chinese dramas). We manually check the data to clearly map actors to their roles in the drama.

• In-situ library refinement. To handle changes in lighting, aging, and makeup within the drama, we scan the entire video at 1 FPS. Using the initial library as a reference, we extract all matching face images. To manage the size and quality of this data, we apply a step-by-step filtering process:

  • Multi-face exclusion. Samples containing multiple faces are removed to avoid confusion.

  • Deduplication. We remove duplicate samples using a cosine similarity threshold of 0.8.

  • Quality filtering. Low-resolution or blocked faces are removed based on a quality threshold of 0.45.

  • Quality control. From the remaining high-quality pool, we randomly sample 15 typical images per character to form the final reference library.

• Frame-wise scan. Finally, we use the InsightFace library Deng et al. [15] to perform frame-by-frame detection and recognition across the full length of each drama, providing the visual data needed to identify speakers.

Speaker Label Initialization and Annotation

Acoustic feature extraction. We use the ffmpeg library to extract raw audio from the videos at a sampling rate of 16kHz. For each dialogue segment $n$ in the set $S = \{(t_n, a_n, b_n)\}_{n=1}^{N}$, we cut out the matching audio clip using the pydub library. Voice features are created using the ERes2Net model Chen et al. [7] within the 3D-Speaker toolkit Chen et al. [8]. To make the similarity scores stable, all feature vectors are $\ell_2$-normalized. This maps them to a space where cosine similarity equals the dot product.

Seed cluster generation. Given the possible utterance sets $\{S_p\}$ linked to character $p$ based on nearby video frames, we start a strict clustering process. This is controlled by a similarity threshold $\theta_{\mathrm{seed}}$ (starting at 0.85, a high value to ensure that two very similar utterances belong to the same person) and a minimum cluster size $\eta_{\mathrm{seed}}$. We then use a two-step search method:

• Phase 1: Global high-confidence search. For all active characters, we build an audio graph. An edge connects two nodes (utterances) if their similarity is above $\theta_{\mathrm{seed}}$. We find the maximum connected component (MCC) for each character using a floodfill algorithm. We rank these groups by size. If the largest group is big enough (meets $\eta_{\mathrm{seed}}$) and is different enough from existing seeds (similarity difference is above $\Delta\theta_{\mathrm{seed}}$), we accept it. If no group is accepted, we gradually lower $\theta_{\mathrm{seed}}$ by 0.01 at a time until it reaches 0.70.

• Phase 2: Targeted character search. If any character $p$ still lacks a seed after Phase 1, we do a local search within $S_p$. We use the same rules to check against existing groups to avoid mixing up identities. We slowly lower the threshold $\theta_{\mathrm{seed}}$ until every active character has one seed voice set.

This process includes an optional human check to make sure the seed sets are correct. For a typical drama with about 50 characters, this guided process takes about 2 to 3 hours.

Iterative affinity propagation. After finding the seeds, we spread the labels to the whole dataset using a step-by-step process. Starting from a high threshold $\theta_{\mathrm{prop}}$, we repeat a two-pass search:

• Identity expansion. Each unlabeled utterance is compared to all existing character groups. If the highest similarity is above $\theta_{\mathrm{prop}}$, the utterance is assigned to that character and added to the reference set. We then update the center of the group.

• Latent identity discovery. Within a sliding window of 20 dialogue lines, we search for groups using a stricter threshold $(\theta_{\mathrm{prop}} + \Delta\theta_{\mathrm{prop}})$. Groups larger than $\eta_{\mathrm{prop}}$ become new clusters for minor characters⁵. This lets the model track consistent voices missed by the first visual checks.

We lower $\theta_{\mathrm{prop}}$ by 0.02 in each step. The process stops at $\theta_{\mathrm{prop}} = 0.45$. Any utterances still unlabeled at this stage get the [UNKNOWN] tag, waiting for a final check by the LRM.

Human annotation. Human labelers use a graphical user interface (GUI) that gives them three main types of information: (1) video streams with boxes drawn around detected faces, (2) initial speaker guesses from our algorithm, and (3) audio statistics (e.g., voice similarity scores). Labelers watch the videos in order and correct the initial labels where needed.

To ensure high-quality labels, especially for minor roles without early voice samples, we use a strict quality check process. After the first round of labeling, a computer script finds uncertain labels for a second human check. A label is marked for review if it meets any of these rules: (a) a main character is assigned to a voice that sounds very different from their normal voice, (b) a main character speaks while off-screen, (c) a new minor character appears, or (d) the tag is [UNKNOWN].

On average, the first labeling step takes 1.5 to 2.5 hours per video hour, while the targeted quality check takes an extra 10 to 15 minutes per video hour. This multi-step approach reduces mistakes in the final data, especially for the hard cases discussed in § 4.

Hierarchical Video Captioning

Clip segmentation via adaptive merging. We use the PySceneDetect library to cut the video into basic shots based on visual content. By using an adaptive threshold algorithm that checks changes in lighting and color, we divide the video into separate shots with an average length of 2 to 3 seconds.

To group these shots into meaningful clips, we use a multi-view feature method. For each shot, we extract the first, middle, and last frames and calculate their features using the CLIP ViT-L model Radford et al. [41]. Joining these three vectors creates a strong representation of the shot’s visual content. We then calculate the cosine similarity between neighboring shots. To handle different visual styles across dramas, we use a dynamic threshold: the merging threshold is set to the top $1/5$ value of all similarity scores within a specific drama. Neighboring shots with scores above this threshold are merged until they reach a target length of 8 to 15 seconds. This step makes sure the clips are long enough to show the story but short enough for the multimodal model to process easily.

Base clip captioning. Let the generated clip sequence be $\mathbf{V} = (C_1, \ldots, C_L)$. For each clip $C_l$, we sample frames at 1 FPS, focusing on frames with many faces. We represent each clip as a sequence of these sampled frames and use Qwen3-VL-32B Bai et al. [3] to generate captions. To improve the accuracy of these captions, we use two methods to link text and visuals: (a) Text adding: We add the dialogue text and current speaker guesses directly into the model’s input context (if the text comes from outside files, e.g., an SRT file, we draw the text onto the matching video frames), and (b)

Visual boxes: We draw red boxes around detected faces, labeled with the matching character ID. This produces detailed descriptions averaging 300 words per clip, ensuring that the visual story is closely tied to character identities. Figure 5 shows the prompts for video clip captioning.

Hierarchical Segment Summarization. To handle the high cost of processing millions of words per drama, we group clips into continuous sections $S_m$ based on meaning. We extract text features for each clip caption using the BAAI bge-large-zh-v1.5 model Xiao et al. [63], and calculate the cosine similarity between neighboring clips ($C_l$, $C_{l+1}$). Clips are merged into a section if their caption similarity is over 0.8, with a limit on the section size (around $8 \pm4$ clips).

For each created section $S_l$, we use Qwen3-32B Yang et al. [64] to combine the clip captions into a single, high-level segment summary $T_l$. This summary removes repeated visual details and focuses on main plot events and character interactions. Figure 6 shows the prompts for video segment summarization.

Dataset Statistics

Table 4 provides detailed statistics of DramaVQA across four evaluation granularities. As the temporal level increases from Medium to Ultra-Long, each video covers a substantially longer context and contains more QA pairs. The table also reports metadata scale, including facial bounding boxes, speaker labels, subtitle length, caption length, and average QA length, highlighting the dense multimodal annotations provided by our benchmark.

QA Generation and Validation: Details

Initial QA Generation

First, we define 28 detailed question categories. These 28 subcategories are then divided into three major groups for generation: low-level, high-level, and multi-point. For each subcategory, we provide a clear category definition and corresponding question examples. We show the definition and examples of a subcategory in Figure 8. All category definitions and examples are summarized in the qa_type_and_samples.json file.

Body Movement

Definition:

1. This task aims to assess the ability to capture the body movements and postures of individuals in the video. Therefore, questions in this dimension may involve asking about the type of actions performed by the person (such as standing, walking, waving, hugging), as well as the direction and magnitude of the movements (such as running to the left, raising arms high), etc.

Example 1:

• Caption: In the dance studio, the students are practicing fundamental skills, raising both hands above their heads in the ‘orchid fingers’ position, slightly bending their knees in a half squat, tilting their bodies to the left, turning their heads toward the right shoulder, and directing their gaze toward the upper diagonal.

• Question: What are the trainees’ physical movements like?

• Options: A. Arms hanging down, standing still B. Arms raised above the head with fingers in orchid position, half-squatting and leaning to the left C. Hands on hips, leaning to the right D. Arms crossed over chest, bending forward

• Answer: B

Example 2:

• Caption: On the train platform, the mother waves goodbye to her son; she stands on tiptoe, raises her right hand high to wave a red scarf, covers her mouth with her left hand, leans slightly forward, and her gaze follows the slowly departing train.

• Question: How does the mother wave goodbye?

• Options: A. Arms hanging down, standing still B. Standing on tiptoe, raising and waving a red scarf, body leaning forward C. Hands on hips, shaking head D. Arms crossed over chest, turning away Answer: B. Stand on tiptoe, raise the red scarf high and wave it, body leaning forward.

• Answer: B

Based on the predefined question categories and corresponding examples, we generate question-answer pairs using the prompts shown in Figure 9 and Figure 10.

Rule and LLM-based Filtering

Rule filtering. Through manual inspection of the generated QA pairs, we summarize a set of filtering rules to eliminate low-quality samples. The rules are as follows:

• Remove QA pairs whose question contains any of the following phrases: the beginning of the video, the start of the video, the opening of the video, the end of the video, the last part of the video, in the scene, the entire video, the overall narrative, the first half of the video, the second half of the video, the start of the scene, the end of the scene

QA Generation Prompts, Part I

Given a description of a certain clip in a film or television drama, generate ‘Question-Option-Answer’ triples based on the video description. These questions shall be applicable to the entire drama, and the understanding of video content shall be deepened through multi-dimensional Q&A. The question stem shall not be too brief and must consist of two parts: the first part introduces the main information of the description, and the second part raises the question.

Guidelines for Generating Question - Option - Answer Triples:

1. Core Requirements:

(a) Information Traceability: The answer shall clearly mark the corresponding key information basis in the video description (e.g., “According to the plot of ‘after the XX incident’ in the video. . . . . . ”), to ensure the traceability of reasoning basis.

(b) Content Fidelity: Strictly follow the core information such as scene settings, character relationships and event progression in the video description, and shall not add unmentioned details or inferences that deviate from the original content.

(c) Unambiguous Reference and Localization for Questioning: The question must precisely locate a unique scene in the video using time markers, location features, and event anchors. The referenced content should be clearly identifiable within the full video. Vague phrases such as ‘the current clip’, ‘this content’, ‘the beginning/opening/ending of the video’, and ‘video description’ are not allowed.

(d) Comprehensive Coverage: The generated triples shall evenly cover different stages, scenes and core events of the video description, avoiding concentration on a single plot to ensure the overall coverage of video content.

(e) Uniqueness of Answer: There is exactly one correct answer among the options, which is clear and objectively existing, and does not involve subjective evaluation or open-ended interpretation.

2. Option Design:

(a) Each question is set with 6-8 options to ensure a unique correct answer and a balanced distribution of options.

(b) Relevance: Distractors shall belong to the same logical category as the correct answer (e.g., homogeneous tools, similar scenes, related concepts)

(c) Discriminability: There shall be an essential difference from the correct answer to ensure it cannot be a reasonable answer.

(d) Rationality: Distractors themselves shall be reasonably possible options when separated from the video context.

(e) Misleading: Distractors shall have superficial credibility and can arouse reasonable doubt.

• Remove QA pairs whose question contains any of the following terms: subtitle, Subtitle, what is the function, what function

• Remove QA pairs that satisfy: Question category $\in\{counting, quantity comparison\}$ AND the question contains explicitly, dialogue, or mention

• Remove QA pairs whose question contains any of the following phrases: video description, explicitly mentioned, explicitly stated, described as, entire description

• Remove QA pairs if:

  • The answer contains not mentioned, not described, not specified, not indicated; OR

• The question category is counting and the answer contains words like several times or some people.

• Remove QA pairs for which a large language model (LLM) determines a high probability of multiple correct answers.

LLM-based filtering. To comprehensively evaluate the multi-modal abilities of models and prevent answers from being trivially deduced from the questions and options, we tested the direct response results of multiple language models. The prompts employed in the test are in Figure 11.

Human Checking and Editing

Annotation guidelines. Given a video of L2 segments and multiple single-choice ‘Question-Option-Answer’ triples related to the video, to help annotators understand the video content, the provided Prompts of LLM-based filtering

Task: Please answer the following multiple-choice question.

Instructions:

1. The question provides several options, but only one is correct. Please identify the correct option.

2. Note: Only return the letter of the correct answer (e.g., A, B, C, D, E, F, G, H). Do NOT include any reasoning or explanation.

video clip is marked with face boxes (red boxes + names) for key characters. Annotators are required to check and determine whether each question fully meets the requirements and can be ‘retained’ based on the video content. For data that does not meet the requirements, annotators should edit and correct it as much as possible to make it eligible for retention; for questions that are indeed unsuitable for correction, they should be marked as ‘not retained’.

Retention criteria. Those QA pairs are retained only if they meet three criteria: the question accurately describes the video content (events, actions, details, characters), exactly one correct option exists, and the provided answer is correct.

Editing criteria. The basic annotation principle is to edit and correct invalid triples as much as possible for retention, with detailed editing rules as follows: (1) Correct the answer if the question and options are valid but the answer is wrong; (2) Remove redundant options to retain only one correct one and confirm the answer if the question is valid but multiple correct options exist; (3) Revise or add the correct option if the question is valid but no correct option exists while a clear answer is available in the video; (4) Revise the question description and ensure answer correctness if the question is invalid but options are reasonable. Triples are marked as ‘not retained’ if: the question is invalid and options are invalid (no/multiple correct answers); the answer cannot be obtained from the video; duplicate questions (different wording, identical intent) appear under the same video (starting from the second duplicate); the question is ambiguous, ungrammatical, hard to revise, or requires modifying over 50% of the original text.

Language Prior Removal

For manually returned results, to further reduce the impact from language model priors and increase the difficulty of final evaluation results, we call language models again for evaluation without reference information. The prompt used is consistent with that in the second step ‘Rule and LLM-based Filtering’, as shown in Figure 11. Three powerful language models are adopted here: Gemini-2.5-pro, GPT-5, and Seed-1.8, and cases where all three models make wrong predictions are retained. Some filtered examples are shown in Figure 12.

Question 1: During the conversation in the servants’ hall, what items of clothing specifically identify Thomas and Jimmy as footmen in formal livery?

Options 1: A. White aprons and black boots B. Grey trousers and white shirts C. Black tailcoats and white bow ties D. Brown vests and tweed caps E. Blue jackets and matching hats F. Green breeches and buckled shoes

Answer 1: C

Question 2: When Charles Grigg arrives at Isobel Crawley’s house, from which part of the vintage Ford taxi does he exit?

Options 2: A. The back seat, driver’s side B. The back seat, passenger’s side C. The front seat D. The driver’s seat E. The luggage rack F. The running board

Answer 2: B

Global Uniqueness Check

To ensure the selected questions can be uniquely localized within the corresponding drama, we propose a multi-tool collaborative verification pipeline, which consists of the following three steps:

Step 1: candidate segment construction via dual text retrieval. We first adopt two independent retrieval strategies to obtain L2 segments highly relevant to the target question, and merge them into a unified candidate set:

• BM25 retrieval Robertson and Zaragoza [42]: For each drama, we build a retrieval index based on the captions of all L2 segments. We compute the relevance score between each question and the indexed caption texts, and retain the top 25 most relevant L2 segments.

• Embedding similarity retrieval: We use the pre-trained language model bge-large-en/zh-v1.5 Xiao et al. [63] to generate embedding vectors for all L2 segment captions, where the chunk size in the text splitter is set to 200. We then calculate the cosine similarity between the question embedding and all caption embeddings, and select the top 25 most similar L2 segments.

The two retrieved segment sets are merged. After excluding the segment where the question is originally annotated, we obtain a candidate set containing up to 50 segments.

Step 2: keyword extraction. We employ a large language model (LLM) to extract 3–5 key works from each question, which serve as the core semantic cues for uniqueness verification.

Step 3: fuzzy matching. We use an LLM (Qwen-3-32B) to examine the caption of each candidate L2 segment. The prompt used here is showed in Figure 13. If none of the candidate captions contain all the extracted key words, the question is regarded as uniquely locatable in the entire drama. Otherwise, its localization uniqueness is not confirmed.

Experimental Results

Setting

Uniform sampling. For each TV drama, we first calculate the ID for all frames. The starting and ending parts of each episode are removed beforehand. Therefore, each input video, no matter how long it is, is defined by a start and end frame ID. This makes it easy to sample frames evenly without needing to join all episodes into one single video file (note that L3 to L6 units can span multiple episodes).

Video input preparation. Each sampled frame is rendered with both the text and facial IDs. To avoid repeating information, please see Appendix § 6.§ 5 for details.

Hierarchical captioning for agentic MLLMs. This part is very similar to Appendix § 6.§ 5, but there are two differences. (1) The text transcript and speaker identities are not included in the prompt (only the rendered text on the frame is available); this ensures a fair comparison with the native MLLMs. (2) Following Qiu et al. [39], we define the time units from top to bottom (instead of the bottom to top method used earlier). This is because the earlier method assumes the base captions are already extracted, which is not true for this test.

In practice, starting from the main video, we evenly divide each part into $K$ smaller parts, and repeat this for 3 levels. This means there are $K^3$ parts at the lowest level. To use the pre-trained model from Qiu et al. [39], we set $K = \min\{6, \sqrt[3]{T/16\mathrm{s}}\}$, where $T$ is the length of the video in seconds. We generate captions by sampling 64, 32, and 16 frames evenly for the clips in the first (highest), second, and third (lowest) levels.

Configuration of Native MLLMs. We run all these models on a server with 8 NVIDIA A800 GPUs using the vLLM library. All models use the default settings and the same QA prompt, as shown in Figure 11. This QA prompt is the same one used for filtering easy questions, except (1) we add video tokens as context, and (2) we turn on the thinking feature for each specific model. During the test, we use different thread numbers for different models to reach the highest speed. However, to measure the time cost fairly, we set the thread number to 1 and report the average time across 100 samples.

Detailed Results

Category-wise. Figure 15 and Figure 16 show the performance comparison of different models on fine-grained subcategories over the entire Full dataset. In Figure 15, native MLLMs (e.g., the Qwen series) are evaluated with 256 sampled frames, whereas in Figure 16, they use 512 sampled frames. Each color corresponds to one model, and the color-to-model mapping is provided in the figure legend. The four adjacent histogram bars and their annotated performance values represent the results on Medium, Long, Extra-Long, and Ultra-Long sequences, respectively. The fine-grained categories are shown in abbreviated form, and their full names are listed in Table 5. This table also summarizes the hierarchical relationships between coarse-grained and fine-grained categories, as well as the number of QA pairs for each category and subcategory. For the seven major categories, we report the number of QA pairs on both the Full set and Mini set, allowing researchers to choose different evaluation sets according to their computational resources.

Major Category	QA Number (Full/Mini)	Subcategory	QA Number (Full)
Visual Perception	4966 / 475	Character Clothing (CCI)	1397
Visual Perception	4966 / 475	Character Hairstyle (CH)	306
Visual Perception	4966 / 475	Distinctive Features (DF)	399
Visual Perception	4966 / 475	Object Appearance (OA)	1168
Visual Perception	4966 / 475	Object Presence (OP)	437
Visual Perception	4966 / 475	Season and Weather (SAW)	155
Visual Perception	4966 / 475	Scene and Location (SAL)	498
Visual Perception	4966 / 475	Textual Content (TCo)	606
Spatial Understanding	1409 / 147	Spatial Location (SL)	704
Spatial Understanding	1409 / 147	Position Comparison (PC)	620
Spatial Understanding	1409 / 147	Shooting Perspective (SP)	85
Action and Interaction	5290 / 553	Body Movement (BM)	1044
Action and Interaction	5290 / 553	Action Claim Subject (ACS)	2852
Action and Interaction	5290 / 553	Action Recipient (AR)	1219
Action and Interaction	5290 / 553	Object Motion (OM)	175
Event Understanding	1981 / 195	Event Summary (ES)	309
Event Understanding	1981 / 195	Event Background (EB)	929
Event Understanding	1981 / 195	Event Outcome (EO)	361
Event Understanding	1981 / 195	Character Motivation (CM)	382
Temporal Understanding	2710 / 275	Character Change (CC)	230
Temporal Understanding	2710 / 275	State Transition (ST)	348
Temporal Understanding	2710 / 275	Temporal Sequence (TS)	1019
Temporal Understanding	2710 / 275	Temporal Comparison (TC)	1113
Relation & Logical Reasoning	1868 / 185	Human-Object Relation (HOR)	996
Relation & Logical Reasoning	1868 / 185	Character Relation (CR)	284
Relation & Logical Reasoning	1868 / 185	Causal Inference (CI)	588
Counting	1776 / 170	Object Counting (OC)	1325
Counting	1776 / 170	Quantity Comparison (QC)	451
Total	20000 / 2000	All Subcategories	20000

Input-Condition Ablation Study.

To investigate the effects of different prompting strategies and input conditions, we evaluate Qwen3-32B on Medium (L3) questions under different input conditions, as shown in Table C.2. For subtitles, we compare hard-rendering them onto video frames with directly providing the subtitle text. Providing subtitles as text slightly outperforms hard-rendering them onto video frames (51.0% vs. 50.0%). We therefore adopt direct subtitle text, which also aligns with the protocols of existing video benchmarks. For character-role matching, we compare rendered face bounding boxes with a character gallery containing one image per character. Rendering face bounding boxes performs better, suggesting that it provides a more effective representation of character identities. Incorporating subtitle text yields a 12.4% performance gain, demonstrating that subtitles are critical for video understanding, especially for narrative content. Speaker identity information further brings a 2.0% improvement. Such annotations are rarely available in other video understanding datasets and constitute a key contribution of our benchmark.

Face	Subtitle	Speaker	Acc.
bounding boxes	hard-rendered	✗	50.0
bounding boxes	text	✗	51.0
character gallery	text	✗	49.4
bounding boxes	✗	✗	38.6
bounding boxes	text	✓	53.0

Qualitative Studies

Figure 14 shows a typical test case. The experimental results in the text demonstrate that the reasoning model achieves higher accuracy when answering the same questions. Below is a concrete example: we use Qwen3-VL-8B Bai et al. [3] to answer the same question with and without the reasoning mode enabled. It can be clearly observed that the response quality is significantly higher under the reasoning mode.

• Question: After Rita leaves and the initial tense silence is broken, who does Robert identify as the person he is most disappointed in?

• Options: A. Himself B. Rita Bevan C. Mary D. Tony Gillingham E. His mother, Violet F. His wife, Cora

• Answer: D

• Response w/o reasoning: [A] Response w/ reasoning: In the provided video clip, the key scene occurs after the conversation between Robert Crawley and Rita Bevan. According to the subtitles and visual content, Robert explicitly expresses his disappointment in Tony Gillingham during his subsequent conversation with Lady Mary Crawley. Specifically, in the clip, Robert Crawley says: “I find I’m most disappointed in Tony Gillingham...” This statement directly identifies the person he is most disappointed in, as asked in the question. Therefore, the correct answer is: [D]

When the model is unable to answer the question based on the provided information, it will randomly select an option. Below is a concrete example: we use Qwen-VL-8B Bai et al. [3] to respond to a question. For demonstration purposes, only the answer under the reasoning mode is presented.

• Question: In the bathroom, Molesley briefly considers using another product to treat his hair. What is the described color of the liquid in the bottle he looks at?

• Options: A. Pink B. Blue C. Yellow D. Clear E. White F. Green

• Answer: C

• Response w/ reasoning: In the provided video frames, there is no scene relevant to the question. The question asks about the color of the liquid in the bottle that Molesley considers using for his hair while in the bathroom. However, the provided screenshots primarily depict various characters from Downton Abbey engaged in conversations and interactions in settings such as the kitchen, drawing room, bedroom, and hallway. None of these images show Molesley in a bathroom, nor do they depict him looking at or considering any hair treatment product. Therefore, based on the provided screenshots, it is impossible to answer this question. I will randomly select an answer: [D]

Details of Accuracy-Efficiency Tradeoff

This section compares native MLLMs and agentic algorithms regarding the accuracy-efficiency trade-off. Recall that the Naive-RAG methods and LongVideo-R1 in Table 2 use hierarchical video captions generated by Qwen3-VL-32B. Therefore, we mainly compare them against the direct QA results of that same model. Across all tracks, the Naive-RAG algorithms show a clear advantage over native MLLMs, but they take much longer to run (e.g., 293.0 seconds per Medium video, and 1,664.6 seconds per Ultra-Long video, which is 1–2 orders of magnitude longer than the execution time of native MLLMs). This is because they process more sampled frames and make more video captioning calls. Two methods can reduce this computational burden: using beam search to narrow the search scope, or adding a reasoning LLM (e.g., LongVideo-R1) for smart navigation. Both methods greatly reduce the time complexity, especially on the Ultra-Long track (by a factor of 6–8). LongVideo-R1 produces higher QA accuracy, but it adds noticeable costs for the LLM reasoning steps. We summarize the QA accuracy and inference times of these models in Table 7.

Model	TP	Medium Acc.	Medium time	Long Acc.	Long time	Extra-Long Acc.	Extra-Long time	Ultra-Long Acc.	Ultra-Long time
Open-source Native MLLMs
Qwen3-VL-32B†‡	C/B	51.0	22.6	35.8	36.4	28.2	37.6	34.1	30.4
Open-source Agentic MLLMs
Naive-RAG (Q3.8)	C	60.1	293.0	56.7	1,664.6	37.0	1,955.5	31.5	1,955.0
Naive-RAG (Q3.8)	B		34.9		34.1		4.9		1.4
RAG-BS (Q3.8)	C	56.4	180.5	40.4	372.2	32.1	408.3	30.9	406.5
RAG-BS (Q3.8)	B		26.7		15.3		4.1		1.3
LongVideo-R1	C	62.8	105.3	54.8	169.1	34.7	294.1	27.2	322.1
LongVideo-R1	B		36.1		41.2		41.8		33.5

Furthermore, unlike other long-video benchmarks, DramaVQA contains many questions for each input video, especially in the Extra-Long and Ultra-Long tracks. When processing multiple questions together, agentic algorithms (including RAG) can share the computation cost of video captioning across many queries. As shown in Table 7, this shifts the accuracy-efficiency trade-off. Naive-RAG and RAG-BS become as efficient as native MLLMs on the Medium and Long tracks, and even an order of magnitude faster on the Extra-Long and Ultra-Long tracks. Meanwhile, LongVideo-R1 reports comparable time costs while achieving higher QA accuracy. Overall, agentic algorithms become the better choice for batch-wise tasks, even though they require a heavy pre-processing stage to generate the hierarchical captions.

This analysis opens up several new questions: (1) Is there a better data structure than hierarchical video captions to further improve this trade-off? (2) How can we adapt agentic algorithms to achieve better performance for batch, streaming, and incremental QA tasks? (3) Can these learned abilities transfer across different TV dramas and to new video categories, such as egocentric videos? Studying these problems connects long-form video understanding with agentic memory mechanisms Yao et al. [65]; ); Hu et al. [22]; ); Abou Ali et al. [1], enlightening new research directions.

Time cost estimation. Below we explain how we estimate the time cost of different models in Table 7.

• The native MLLMs, e.g., Qwen3-VL-32B. We use the same estimation method stated before (setting the thread number to 1 and averaging over 100 samples). Native MLLMs cannot share computation among multiple test cases. We cannot feed multiple QAs into the native MLLMs at once (with the same video unit as context) because this may cause answer leaking: the answer to some QAs may be included in the description or options of other QAs.

• The agentic MLLMs, i.e., Naive-RAG, RAG-BS, and LongVideo-R1. They share the same process of hierarchical captioning for each input video unit. Table 2 reports the time cost as if each QA is executed individually, and Table 7 reports the amortized cost where

the captioning cost is averaged across all QAs in the input video unit. For the calculation, we estimate that the time used for Qwen3-VL-32B video captioning of the 1st, 2nd, and 3rd-level clips (with 64, 32, and 16 frames, respectively) is 14.18s, 11.16s, and 6.82s. These are tested on a server of 8 NVIDIA A100 GPUs. The time cost for building an inverted table and performing BM25 search is about 0.3-0.6s for each test case. LongVideo-R1 takes 30 to 40 seconds to navigate through the hierarchical caption (by calling the video_cap tool) and performs QA (by calling the video_qa tool) if necessary.

Details of Multi-Hop Composition

Overall objective. To make the problem more challenging, especially in evaluating the model’s ability to answer questions by capturing long-range clues, we generate riddles that require multi-step reasoning over multiple pieces of evidence, based on basic QA pairs. Given the vast amount of information conveyed by characters in TV shows, we leverage character entities as bridging elements in this preliminary exploration. As shown in the right diagram of Figure 17, the character Robert Crawley, Earl of Grantham, is the answer to question 1 ($Q_1$) and also appears in the premise of question 2 ($Q_2$). We remove Robert Crawley, Earl of Grantham, from $Q_2$ and introduce this entity through $Q_1$, thereby constructing a 2-hop question. To answer such questions, the model must first decompose the problem, answering the first sub-question by finding clues (from Season 3, Episode 3 of Downton Abbey) to obtain Robert Crawley, Earl of Grantham as an intermediate result, and then answer the second question (from Season 6, Episode 3) by finding clues, completing the multi-hop question.

The detailed pipeline. The entire question construction process is illustrated in the left diagram of Figure 17, consisting of two steps: Sampling Data Preparation and Multi-hop Question Generation. The purpose of Sampling Data Preparation is to establish the relationship graph between the bridging entities, questions, and answers. We first compile the correct answers (characters) from all basic QA pairs into an entity dictionary. Then, we iterate over all the questions in the basic QAs and establish the correspondence between the entities in the entity dictionary and these questions. This results in two data graphs: the Entity-Question Graph and the Entity-Answer Graph. In the Multi-hop Question Generation step, we randomly sample an entity and use these two graphs to find the corresponding QA pairs. Sampling from the Entity-Answer Graph yields an answer whose associated question forms the first question in the reasoning chain, while the question sampled from the Entity-Question Graph serves as the second question in the chain, with its corresponding answer being the final answer. However, the second question may contain some information leakage, where the model does not need to solve the first sub-question to obtain the bridging entity and can directly answer by only looking at the second sub-question (for example, “Who is the son of [MASK] and B.” In a TV show, the context is so limited that there is no need to know who [MASK] is to answer the question). To address this, we attempt to mask the bridging entity [MASK] and use LLMs to assess whether the question relies on the bridging entity to answer. If it doesn’t, it indicates that there is redundant information leakage, and we resample the question. This process is guided by the prompt shown in Figure 18 to instruct GPT-5. Afterward, we use the prompt in Figure 9 and Figure 10 to combine the two questions into a two-hop question.

Results. Using this pipeline, we generate 452 questions from the basic QA pairs for a preliminary evaluation. For evaluation, we use Qwen3-VL-32B with Medium track clips (approximately 20 minutes) provided as auxiliary context. The model achieves an accuracy of 24.8%, which increases to 27.8% when only the second-hop question is evaluated. This indicates that current models still struggle with multi-step reasoning over temporally distant clues. In future work, we plan to generate more challenging questions by leveraging richer video information, including annotations and other structured signals, enabling benchmarks that require integrating multiple clues across long temporal spans.

Prompt for Two-hop Riddle Generation: Part I

Please generate a two-hop reasoning question based on the following two related QA pairs, and return the result strictly in JSON format.

Generation Goal:

Construct a two-hop question that can only be answered by first resolving the prerequisite reasoning step (QA2). The question is designed for the full-series search space of a long-form TV drama (approximately 30 hours) and must satisfy both unique identifiability and a chained reasoning structure. The wording should be as natural as possible, avoiding awkward phrasing caused by excessively long stacked modifiers.

Requirements:

1. The new question must first use QA2 to identify a specific person or entity (the bridge entity, {bridge_ent}), and then use that person or entity to answer the question corresponding to QA1.

2. The new question must not explicitly mention “{bridge_ent}”. Instead, it must uniquely refer to this person or entity through a complete and specific description of the key scene, key utterance, key behavior, or key plot event in QA2.

3. Do not semantically compress or generalize any critical information from QA2 or QA1:

(a) Do not replace original key utterances with vague summaries.

(b) Do not remove plot details that contribute to unique identification.

(c) You must preserve the key scene, key action, key dialogue, key conflict, or key emotional state from the original questions.

(d) Reordering is allowed, but the information density must not be reduced and the meaning must not be blurred.

4. The new question must not introduce any temporal order that is not explicitly given in the original QA pairs:

(a) Do not use temporal markers such as “later,” “afterward,” “then,” “subsequently,” or similar expressions.

(b) If the temporal order is uncertain, use temporally neutral connections such as “this person” or “this character . . . when . . . ”.

5. Do not force the entire QA2 content into a single oversized premodifier phrase (e.g., avoid structures like “the person who . . . , when . . . , . . . ”). Instead, the question must adopt one of the following more natural forms:

(a) Narrative form: first describe the QA2 event in a complete sentence, then use “this person” or “this character” to introduce the QA1 query.

(b) Multi-clause form: connect the content using two or three shorter clauses or sentences, rather than one overly long sentence.

6. The new question must form a strict chained reasoning structure:

(a) Without the QA2-related information, the new question must not be uniquely locatable across the full series.

(b) QA2 alone must not directly answer the final question.

(c) QA1 alone must not allow the answer to be obtained without going through QA2.