betaFeedback

WorksComputer Science

A3^3Bench: Compositional Reasoning over Aggressive Interactions in Videos

Abstract

Recent Multimodal Large Language Models (MLLMs) have achieved strong performance on general video understanding benchmarks, yet their ability to reason about participant roles in aggressive interactions remains largely unexplored. Existing violence and crime datasets primarily evaluate whether aggression occurs, without testing whether models can identify who performed the action, who received it, and how participants are related within the scene. To address this gap, we introduce A3^3Bench, a benchmark to study aggressive action and association in videos. The benchmark evaluates whether models can jointly recognize aggressive actions, distinguish aggressors from victims and bystanders, and correctly bind actions to participants under challenging distractor settings. We benchmark 17 multimodal models and find that they struggle substantially on fine-grained aggressive-scene understanding. Performance degrades consistently as reasoning complexity increases, with models frequently confusing aggressors with victims or bystanders, revealing a systematic failure in role-action association rather than coarse aggression recognition alone. We further observe that increasing model scale or generic reasoning models experience limited improvement. Finally, we explore a simple role-graph prompting strategy that encourages structured reasoning over participant interactions, leading to improvements in compositional reasoning. Our findings highlight that reliable aggressive-scene understanding remains a major challenge for current MLLMs and motivate future research on role-aware video reasoning beyond coarse violence detection. Code and dataset: https://github.com/austincbaker/A3Bench.

Introduction

Recent Multimodal Large Language Models (MLLMs) have demonstrated strong performance on video understanding tasks, including action recognition [9, 15, 47, 24, 20], temporal event localization, and video question answering [25, 29, 63]. Existing action datasets have been highly effective in evaluating models across a wide range of everyday activities, sports actions [26, 43], human-object interactions [45, 22], and complex temporal events [35, 60]. While these tasks are important for general video understanding, reliable real-world systems must also recognize safety-critical behaviors, including physical altercations or threatening interactions. However, understanding aggression requires more than detecting that a violent or harmful action has occurred. In practical settings such as security, public safety monitoring, school safety, sports analysis, and online video moderation, it is often not enough to recognize what is happening; it is also necessary to understand

printed: 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Track on Evaluations and Datasets. how each person in the scene is involved. Without role-aware reasoning, a system may correctly classify a scene as aggressive while still producing an incorrect or harmful interpretation.

Prior work has studied aggressive actions through crime detection [48, 6], violence detection [58, 5, 37, 19, 11, 23, 34], and incident understanding datasets [4, 32]. Some provide multiple aggressive action categories, while others formulate the task as binary classification, such as normal vs. anomalous or violent vs. non-violent. These resources have advanced the study of abnormal and dangerous events, but they typically evaluate only whether aggression is present. They often do not require reasoning about the individuals involved in the scene, including who initiates the action, or who is affected by it. In particular, existing evaluations rarely test whether an aggressive action can be bound to the correct actor, whether aggressors can be distinguished from victims and bystanders, or whether multi-person interactions can be resolved. This limitation can reward models that rely on contextual shortcuts, scene priors, or coarse motion cues, rather than learning the semantic relationships between people, actions, and roles. This necessitates a more fine-grained capability: aggressive action and association, where the goal is not only to recognize an aggressive behavior, but also to associate it with the correct participant roles.

To address this, we introduce A3^{3}Bench, a benchmark for evaluating aggressive action and association (A3^{3}) in MLLMs. A3^{3}Bench contains 2,683 video clips and 15,004 question-answer pairs that move beyond coarse aggression recognition to evaluate role-aware reasoning in aggressive scenes. Rather than asking only whether aggression occurs, the benchmark probes models on aggressive actions and participant-role associations, including aggressors, victims, and bystanders. These questions require models to distinguish individuals, interpret interactions, and correctly bind actions to participants. By focusing on fine-grained role-action reasoning, A3^{3}Bench provides a challenging testbed for assessing whether current MLLMs can reliably understand aggressive scenes.

Our experiments show that recent MLLMs struggle on this benchmark, with performance remaining below 50%. Accuracy consistently drops as questions require more compositional reasoning, from identifying a single attribute to jointly reasoning about actions and participant roles (Figure 1, right). Error analysis further reveals a strong bias toward visually salient individuals, with models frequently misclassifying victims or bystanders as aggressors. These findings suggest that current MLLMs fail to reliably capture the interaction structure underlying aggressive scenes. Finally, we explore a simple role-graph prompting strategy that encourages structured reasoning over participant interactions, leading to improved performance on compositional reasoning tasks.

Figure 1. A3^{3}Bench evaluates fine-grained aggressive action and association. Left: Example question types. The benchmark progresses from basic questions that ask about a single attribute, to detailed questions that require understanding all aspects. Right: Performance of evaluated MLLMs across the three question levels. Models perform poorly overall scoring below 50%, and accuracy decreases as the task becomes more difficult, with the lowest scores on detailed questions.

We make the following contributions: 1) We introduce A3^{3}Bench, a benchmark for evaluating aggressive action and association in videos designed to assess fine-grained role-aware reasoning beyond coarse violence recognition, 2) We conduct a comprehensive evaluation of recent multimodal large language models in a zero-shot setting, showing that current systems struggle to reliably associate aggressive actions with the correct participants, with performance degrading significantly as reasoning complexity increases, and 3) Through detailed analysis, we identify systematic failures in role-action association, and show that a simple role-graph prompting strategy can improve structured reasoning in aggressive-scene understanding.

Related Work

Action recognition: The advancement of action recognition can be noted across video [49, 51], skeleton [59, 68, 53], and multimodal modalities [41, 52, 36, 21, 1, 12]. Certain methods, such as temporal localization [44, 28, 30] allow more precise event boundary detection. Fine-grained recognition has advanced through unified keypoint-based frameworks [18] and atomic action decomposition [39]. Action detection methods further localize and identify what each person is doing in space and time, but they typically focus on individual actions rather than the relationships between interacting participants. These methods still treat actions as a classification problem over participant-agnostic categories, without modeling the relationships between those involved.

Crime and anomaly detection: Anomaly detection and behavior analysis in security videos have been widely studied for identifying abnormal, unsafe, or criminal activities [66, 27]. In parallel, dedicated violence and aggression datasets have advanced the evaluation of models across diverse harmful scenarios, including fight detection, crowd violence, real-world security violence, intruder activity, child aggression, and crime/anomaly recognition [37, 23, 46, 19, 11, 34, 4, 58, 5, 6, 48]. Beyond visual understanding, text-based aggression detection has also examined aggressive behavior in language, such as harmful or hostile social-media comments [2], showing that aggression analysis spans multiple modalities. Most existing video benchmarks focus on violence, aggression, or anomaly detection with coarse event-level labels. In contrast, A3^{3}Bench evaluates fine-grained aggressive action association by requiring models to identify the actor, target, and role of each visible participant, exposing failures that prior benchmarks do not measure.

Compositional video reasoning: Existing benchmarks have explored compositional video understanding through temporal and causal reasoning [61], spatio-temporal question answering [17], and situated reasoning over human interactions [57]. More recent work evaluates fine-grained associations between agents, actions, and events using video-language entailment and challenging negative captions [42]. While these assess general compositional reasoning, A3^{3}Bench focuses on aggressive interactions, requiring models to distinguish aggressors, victims, and bystanders and correctly associate actions with participant roles and interaction direction.

Multimodal Large Language Models (MLLMs): Recent multimodal models have significantly expanded the scope of visual understanding by linking visual evidence with language-based semantic reasoning [40]. Building on large-scale cross-modal pretraining, such models have enabled zero-shot and open-vocabulary recognition of human actions and activities [3, 52, 36, 41, 21, 1, 55]. Recent video-centric MLLMs [7, 69, 31, 14], further extend these capabilities to temporal and multi-frame reasoning over complex events [33, 62, 67]. These advances motivate the use of MLLMs for aggressive action and association, where models must not only identify the observed action but also reason about the relationships between people and their roles.

Aggressive Action and Association Benchmark (A3^{3}Bench)

A3^{3}Bench is designed to evaluate fine-grained aggressive action and association in real-world videos. The benchmark construction begins with video curation and structured annotation of aggressive interactions, including actions, participants, roles, and scene context. These annotations are then converted into three levels of diagnostic VQA questions, ranging from basic attribute recognition to compound association and detailed aggressor-action-victim reasoning. The resulting benchmark enables controlled evaluation of MLLMs under a unified multiple-choice protocol.

Construction

Video curation and annotation: We curate a benchmark of 2,683 video clips collected from diverse sources, including publicly available social-media videos, UCF-Crime [48], and RWF-2000 [11]. Source videos are trimmed into shorter clips centered on individual interactions, resulting in an average clip duration of 5.75 seconds (standard deviation of 1.69 seconds), with most clips approximately 4 to 8 seconds long. The videos span a wide range of recording conditions, camera viewpoints, resolutions, scene layouts, and levels of visual clutter, reflecting the heterogeneity of real-world scenarios. The final corpus contains 2,234 clips with aggressive behavior and 449 non-aggressive clips.

BenchmarkActions/Verbs# VideosDesc.AggressionAggression RolesVQA/Caption PairsRelational StructureTask Format
DVD [23]2500×✓×N/AN/AClassification
Movies Fight [37]2200×✓×N/AN/AClassification
RLVS [46]22,000×✓×N/AN/AClassification
RWF-2000 [11]22,000×✓×N/AN/AClassification
VID [34]23,020×✓×N/AN/AClassification
Violent Flows [19]2246×✓×N/AN/AClassification
CABAD [5]6900×✓×N/AN/AClassification
CRxK [6]133,484×✓×N/AN/AClassification
IARD [4]4802×✓×N/AN/AClassification
UCF-Crime [48]141,900×✓×N/AN/AClassification
XD-Violence [58]74,754×✓×N/AN/AClassification
CLEVRER [61]–20,000×××300KProgrammaticOpen QA
AGQA [17]1579,600×××192MScene GraphOpen QA / MCQ
STAR [57]1119,000×××60KSituation GraphMCQ
VELOCITI [42]–864✓××3KSemantic Role LabelBinary Entailment
A3^{3}Bench (Ours)182,683✓✓✓15KStructured Participant RolesMCQ

Table 1. Comparison of A3^{3}Bench with prior video benchmarks spanning aggression recognition and compositional reasoning. Desc. denotes natural-language person descriptions; Aggression Roles indicates explicit aggressor, victim, and bystander annotations; VQA/Caption Pairs reports question-answer or video-caption pairs; Relational Structure denotes the relational or role annotation format; and Task Format specifies the evaluation protocol.

A group of annotators labeled each video with structured scene information. The annotations include: (1) the aggressive action being performed, selected from 17 aggressive action categories and a control “none” category for non-aggressive clips; a short appearance description of the (2) aggressor; (3) victim; (4) bystanders and (5) the environment or setting. As seen in Figure 2, person descriptions are primarily based on clothing and coarse visual appearance, such as “person in a dark jacket,” while avoiding names, faces, demographic attributes, or other identifying information. This design reduces the risk of relying on sensitive attributes and focuses the benchmark on visual role and action understanding. If a certain role does not exist in the video clip, or the appearance is too ambiguous, that is left out in the annotations.

Figure 2. Example videos from A3A^{3}Bench. Each video is annotated with structured role and scene information, including the aggressor, victim, bystanders, aggressive action, and environment. Person descriptions are based primarily on clothing and coarse visual attributes rather than identity, enabling fine-grained actor-role association while avoiding personally identifying information.

Dataset distribution. A3^{3}Bench intentionally contains more aggressive than non-aggressive clips, as its primary objective is to evaluate fine-grained action and participant-role reasoning within aggressive interactions. Non-aggressive clips are included to evaluate whether models can recognize the absence of aggression. As shown in Figure 3, A3^{3}Bench also exhibits a male-skewed participant distribution, reflecting the composition of the collected real-world source videos. This observation is consistent with broader crime statistics reporting higher male representation in several offense categories. According to the FBI’s 2019 arrest statistics, males accounted for 72.5% of all arrests, including 84.2% of robbery arrests, and 76.5% of aggravated-assault arrests [13]. UNODC Global Study on Homicide 2023 also reports that approximately 91% of suspects brought to police for intentional homicide (across 50 countries) between 2018 and 2021 were male [50].

Figure 3. Demography (aggressors/victims) and environment settings. A3A^{3}Bench contains a larger proportion of male and adult participants (aggressors or victims) among videos where demographic attributes are identifiable. For scene conditions, many clips are ambiguous in time of day, but among identifiable cases, slightly more aggressive clips occur during daytime than night, and most are captured in indoor environments.

Annotation quality assurance: We employ a structured, consensus-driven annotation protocol involving four annotators to ensure visually grounded and reliable annotations of aggressive interactions. Each annotator independently reviews assigned videos and labels the aggressive-action category, appearance descriptions of aggressors, victims, and bystanders, and scene environment. Participants are identified using clothing and other coarse, non-identifying visual cues (e.g., “person in a red shirt”), while avoiding race, gender, age, and other sensitive attributes. Demographic statistics in Figure 3 are reported only at the aggregate corpus level and are not used in questions, answer options, or role labels.

Annotators flag videos with ambiguous actions, interaction direction, participant visibility, or appearance descriptions, particularly in crowded, occluded, blurry, black-and-white, or low-resolution scenes. Flagged annotations are withheld from finalization pending group review. After each annotation batch, another team member independently re-reviews a randomly selected subset. Aggressor-victim direction conflicts occur in fewer than 5% of cross-checked cases, while appearance-description disagreements occur in approximately 20%, primarily because annotators select different clothing or coarse visual cues to describe the same participant rather than identifying different participants. All flagged cases and disagreements identified during cross-checking are collectively reviewed. Role-direction disagreements are resolved by determining who initiated and who received the aggressive action, while appearance-description disagreements are resolved by selecting the clearest non-identifying description. When reliable consensus cannot be reached, the corresponding attribute is omitted rather than guessed. Periodic spot checks and targeted re-review of difficult cases continue throughout benchmark development, refining participant descriptions and removing ambiguous attributes while retaining only visually supported annotations.

Question generation and distractor design: From the annotation corpus, we generate multiple-choice video question-answer pairs that function as controlled diagnostic probes for aggressive interaction understanding. Rather than only evaluating whether a model can recognize an action label, A3A^{3}Bench is designed to expose distinct failure modes in video-language models, including weak role grounding, poor action-participant binding, confusion between visible and relevant people, failure to model interaction directionality, and reliance on answer-choice priors. The questions are organized into three tiers of increasing complexity: basic questions, compound questions, and detailed questions.

Figure 4. Distribution of annotated roles per video. The role-count distribution is also skewed towards most clips containing a single identifiable aggressor and victim, while bystanders are more variable and often appear in larger groups.

Figure 1 illustrates representative examples from these three categories, and Figure 5 reports their subcategories and overall distribution across the benchmark.

Figure 5. Distribution of question types in an A3^3Bench. The benchmark contains three levels of questions with increasing association complexity: Basic questions focus on single attributes; Compound questions require associating two scene components; and Detailed questions require full interaction understanding.

Basic questions test whether a model can recover a single atomic attribute of the scene. These include primary action identification, where the model must recognize the aggressive action taking place; aggressor or victim identification, where the model must identify who performs the aggression or who is targeted; and role identification, where the model is given a person description and must assign the correct role, such as aggressor, victim, or bystander. These questions diagnose whether the model can localize individual actions and participant roles before requiring relational reasoning.

Compound questions test whether a model can bind two scene attributes correctly. These include aggressor-victim, action-aggressor, and action-victim associations. Since each answer choice is only correct when all components are jointly correct, these questions diagnose whether models merely recognize visible actions or participants independently, or whether they can associate the correct action with the correct person and role.

Detailed questions test whether a model can recover the full event structure of an aggressive interaction. These include aggressor-action-victim questions, which ask who did what to whom, and sequence verification questions, which require selecting the correct structured description of the interaction. These questions specifically diagnose failures in directionality, role distinction, and holistic event understanding, since a model must distinguish the aggressor from the victim and construct a coherent representation of the interaction rather than predict a participant-agnostic action category.

Importantly, Basic, Compound, and Detailed questions are generated from the same underlying video pool, with each video contributing approximately seven questions across multiple complexity levels.

We design distractor choices as targeted diagnostic stress tests: they are visually plausible, semantically close to the correct answer, and tied to specific failure modes. Cross-video distractors are sampled from real annotations of other clips, so incorrect options still contain natural actions, roles, and participant descriptions rather than synthetic or obviously invalid answers. Same-video distractors use visible non-target participants from the queried clip, such as the victim or a bystander when asking for the aggressor, preventing models from relying only on visibility, saliency, or person presence. Role reversals and bystander substitutions preserve much of the event content while swapping aggressor and victim roles or assigning bystanders to active roles, directly testing interaction directionality and the distinction between active participation and mere presence. Negation questions include answers such as “no aggressive action is taking place” to test whether models verify visual evidence rather than assume aggression is present. Frequency-inverted distractors are constructed so that each key distractor property, such as an action, role, or participant description, appears across multiple answer options, making marginal answer frequency uninformative. This forces models to reason over the complete aggressor-action-victim structure rather than selecting the option with the most common or familiar textual components.

Benchmarked Models

We evaluate 17 representative model configurations on A3^{3}Bench to assess their ability to perform aggressive action and association. All of our chosen models are trained on very large datasets with high-level video understanding.

General-Purpose Models. The evaluated models include two CLIP-based video-language models, ViCLIP [55] and X-CLIP [36], alongside several recent open-weight MLLM families, including Gemma [14], InternVideo [56], InternVL [10, 69, 54], LLaVA-Video [65], Ovis [31], Qwen-VL [8, 7], and VideoLLaMA [64]. Among closed-source, we include GPT 5.1 [38]. These families differ in visual encoder design, video modeling strategy, instruction tuning, model scale, and reasoning behavior.

Reasoning and Prompted Variants. To investigate whether explicit reasoning improves aggressive action and association, we additionally evaluate reasoning-oriented variant of Qwen3-VL-8B [7]. We also evaluate InternVL3.5-8B [54] under two reasoning-style prompting settings: a detailed chain-of-thought (CoT) prompt and a Dream-of-Thoughts (inspired from [16]) prompt. These variants encourage intermediate reasoning before producing a final answer, allowing us to test whether step-by-step inference helps models distinguish aggressors, victims, bystanders, actions, and scene context.

Implementation Details, Evaluation Protocol, and Metrics

All experiments are zero-shot. Open-source models are evaluated using their official implementations, hyperparameters, and released weights on a single NVIDIA RTX A6000 (48 GB) or A100 (80 GB) GPU. Qwen2.5-VL-72B-Instruct [8] was run on the quantized version to fit in memory. We use a multiple-choice video question answering (VQA) protocol across all models: for each video-question pair, the model is prompted to select the most appropriate answer from the provided options. Depending on the question type and the size of the valid answer space, each question contains four to eight answer choices, including one correct answer and a set of diagnostic distractors. The position of the correct answer is uniformly shuffled across all questions to mitigate positional bias. We report accuracy as the primary evaluation metric.

Experimental Results and Analysis

Table 2 summarizes model performance on A3^{3}Bench. We report both overall accuracy, computed as the question-count-weighted average, and category average, computed as the unweighted mean across Basic, Compound, and Detailed questions. Overall, current MLLMs perform poorly on the benchmark: most models achieve below 50% accuracy, and even the best-performing model reaches only 46.96% overall accuracy and 44.30% category average. These results indicate that aggressive action and association remains challenging for state-of-the-art systems. In particular, current MLLMs struggle when evaluation moves beyond coarse aggression recognition and requires identifying who is involved, what action occurs, and how participants are related.

ModelBasic ↑\uparrowCompound ↑\uparrowDetailed ↑\uparrowOverall ↑\uparrowAverage ↑\uparrow
ViCLIP [55]28.1121.6116.1623.8221.96
X-CLIP [36]23.6715.7520.0720.1219.83
gemma-4-26B-A4B-it [14]49.0739.6530.8642.7239.86
InternVideo2.5-8B [56]48.0739.3421.5140.7036.31
InternVL2.5-8B [10]52.6145.0126.7545.7741.46
InternVL3-9B [69]50.0539.0628.2242.5539.11
InternVL3.5-8B [54]48.3437.0328.7541.0638.04
LLaVA-Video-7B-Qwen2 [65]19.5819.3413.1118.5017.34
Ovis2.5-9B [31]50.6237.9933.0943.1640.57
Qwen2.5-VL-7B-Instruct [8]41.8737.1523.7337.3034.25
Qwen3-VL-8B-Instruct [7]51.0746.65<u>35.19</u>46.9644.30
VideoLLaMA3-7B [64]49.6530.86<u>21.41</u>38.2233.97
Qwen2.5-VL-72B-Instruct [8]45.7339.4726.9140.5337.37
GPT 5.1 [38]45.6339.1733.3241.3039.37
Qwen3-VL-8B-Thinking [7]44.5936.4130.7739.3737.26
InternVL3.5-8B CoT prompt<u>51.53</u>38.9233.1343.9441.19
InternVL3.5-8B DoT [16]49.6440.3140.9044.77<u>43.62</u>

Table 2. Benchmarking models on A3A^{3}Bench. Reported metric is accuracy (%) for Basic Questions, Compound Questions, Detailed Questions, Overall, and Average. Best and <u>second-best</u> performance are highlighted.

Human evaluation. We additionally conduct a human evaluation involving 34 participants using the same video-question pairs and answer choices presented to the models. Each participant answers five questions from five distinct action classes, with one video sampled per class. In total, we collect 170 responses, comprising 85 Basic, 49 Compound, and 36 Detailed questions. Human participants achieve an overall accuracy of 83.5%, highlighting substantial room for improvement of model performance in compositional understanding of aggressive interactions.

Analysis

Performance degrades as aggression understanding becomes more compositional. Models achieve an average accuracy of 44.11% on Basic questions, but performance drops to 35.51% on Compound questions and further to 27.29% on Detailed questions. This monotonic decline shows that current MLLMs do not simply fail at rare aggression labels; they fail progressively as the task requires role-action binding. Models can sometimes identify isolated visual cues, such as an action category, a salient participant, or a single role, but struggle once these cues must be composed into a structured interaction. Compound questions expose failures in binding two attributes, such as linking the correct action to the correct aggressor or victim. Detailed questions further require the full aggressor-action-victim structure, or who did what to whom. The sharp drop on these questions indicates failures in directionality, role assignment, and person-action binding.

Model-level trends show that the compositional drop is not uniform across systems. VideoLLaMA3-7B exhibits the steepest collapse, suggesting strong reliance on single-attribute cues. Qwen2.5-VL-7B-Instruct shows a small Basic-to-Compound drop but a much larger Compound-to-Detailed drop, indicating that pairwise association does not transfer to full aggressor-action-victim reasoning. Ovis2.5-9B shows the reverse pattern, with larger early drops and smaller later drops, suggesting that role-action binding is their main bottleneck. Qwen3-VL-8B-Instruct achieves the strongest Compound performance but still degrades on Detailed questions, while ViCLIP and LLaVA-Video-7B-Qwen2 remain weak across all tiers. The InternVL3.5-8B DoT [54, 16] variant and X-CLIP are notable exceptions, with Detailed accuracy slightly exceeding Compound accuracy.

Reasoning helps, but not uniformly. Qwen3-VL-8B-Thinking underperforms its instruction-tuned counterpart across all categories, dropping from 46.96% to 39.37% overall, suggesting that generic thinking mode does not necessarily improve role-action association. In contrast, reasoning prompts improve InternVL3.5-8B: CoT and DoT increases overall accuracy. The gains are most pronounced on Detailed questions, where InternVL3.5-8B rises from 28.75% to 33.13% with CoT and 40.90% with DoT. This suggests that structured reasoning is more useful when the task requires full aggressor-action-victim binding.

Models often identify participants without recognizing the correct aggressive action. To understand whether errors arise from recognizing the aggressive behavior or identifying the relevant participants, we analyze videos that include both a Basic primary action question and at least one aggressor or victim identification question. Figure 6 (left) reports results averaged across models. Models identify the correct aggressor or victim while failing to recognize the correct action in 26% of cases, whereas they recognize the correct action while failing to identify the relevant role in only 17% of cases. Another 16% of cases are correct on both, while 41% are incorrect on both. This pattern indicates that models are more likely to localize socially salient participants than to distinguish the fine-grained aggressive action being performed. Thus, even when participant grounding partially succeeds, current MLLMs often still lack reliable aggression understanding.

Figure 6. Error analysis of action and role association averaged across models. We analyze model failures on Basic questions by decomposing predictions across (left) action vs. role correctness, (center) aggressor vs. victim correctness, and (right) role identification. MLLMs struggle not only to recognize aggressive actions, but also to assign the correct roles to the persons involved.

Models are biased toward assigning the aggressor role. Figure 6 (center, right) show a consistent role-grounding failure: models tend to over-assign the aggressor role when identifying participants in aggressive interactions. In Figure 6 (center), for videos containing both Basic victim and aggressor identification questions, only 22% of cases correctly identify the aggressor and the victim simultaneously, while 31% fail on both roles. Among partial successes, models more often identify the aggressor while missing the victim (28%) than the other way (19%), suggesting that victim grounding is unreliable. Figure 6 (right) shows the same bias in role classification: many ground-truth victims (37%) and bystanders (32%) are predicted as aggressors. These errors suggest that models often treat visual involvement, proximity, or saliency as evidence of aggression, rather than reasoning about who initiates the action and who is targeted.

Distractor analysis reveals different failure modes across question complexity. Distractor analysis shows that models are especially vulnerable when incorrect options preserve visually plausible participants or partially match the interaction structure. For Basic questions, role-reversal distractors are consistently more challenging than cross-video distractors for most models, indicating that models often recognize relevant people but confuse their roles. For Compound questions, role-reversal and wrong-action distractors dominate, showing that models struggle to jointly bind the correct action with the correct aggressor or victim. In Detailed questions, frequency-inverted distractors dominate showing that models may rely on answer-choice priors rather than visual grounding, selecting options with familiar or frequently co-occurring action-role-person components instead of recovering the full aggressor-action-victim structure.

Model-level patterns reveal distinct vulnerabilities across question tiers. In Basic questions, Ovis2.5-9B is vulnerable to role-reversal distractors, while LLaVA-Video-7B spreads errors across cross-video, wrong-role, and bystander-substitution options, suggesting weaker visual grounding. InternVL3.5-8B-DoT, ViCLIP, and X-CLIP also show high bystander-substitution errors, suggesting over-attention to visible non-participants. In Compound questions, ViCLIP, X-CLIP, and LLaVA-Video-7B show more wrong-aggressor and wrong-victim errors than most models, indicating weaker participant grounding. In Detailed questions, frequency-inverted distractors dominate for most models, while GPT-5.1 shows more errors involving role reversals.

Models over-predict aggression even in non-aggressive clips. We conduct a free-form social appropriateness evaluation on four relatively strong models [31, 10, 69, 7]. Each model is asked to characterize the social appropriateness of the action shown in a video. Models are generally able to identify aggressive or socially inappropriate behavior in aggressive clips, with most models reaching 78.2%-89.8% accuracy. However, on non-aggressive clips, they frequently still attribute inappropriate or aggressive behavior to the videos, with most models near or below 60% accuracy. This indicates that current models are sensitive to aggressive cues but also exhibit a strong aggression prior, over-predicting aggression even when no aggressive behavior is present. InternVL2.5-8B is more balanced across aggressive and non-aggressive clips, but both scores are close to 60%; given the 50% binary chance level, this suggests that it does not reliably solve either side of the task.

Scene context adds another layer of grounding difficulty. We further evaluate whether models can associate aggressive actions with their surrounding scene context using a subset of videos with clearly identifiable environments. Models are asked to identify both the action and the environment in which it occurs. Adding this scene-grounding requirement makes the task harder for most models: 12 evaluated models show lower accuracy on action-environment association than on primary action recognition, with their average accuracy dropping from 38.56% to 32.34%. This indicates that models struggle not only to recognize aggressive actions, but also to ground them in the correct environmental context. Interestingly, InternVL3.5-8B and its reasoning variants are relatively robust, showing little degradation. Overall, however, the drop for most models suggests that reliable action-environment association requires joint reasoning over local behavior and broader scene context.

Detailed results and analysis charts are provided in the Supplementary.

Role-graph prompting

We observe that models often over-assign the aggressor role, confusing victims or bystanders with active participants. To encourage more explicit role-action reasoning, we explore a role-graph prompting strategy as a diagnostic structured-reasoning intervention. As shown in Figure 7, the graph represents aggressive-scene understanding through interaction direction, participant roles, and aggressive actions. We prepend this graph-based instruction template to each multiple-choice VQA query. The linguistic prompt serves as an interface that directs the model’s attention toward interaction direction and participant roles, without altering the underlying visual evidence or evaluation protocol.

Figure 7. Role-graph prompting structure. The graph decomposes aggressive-scene understanding into structured role-action reasoning, linking interaction, direction to participant roles and aggressive actions.

Table 3 compares baseline accuracy with role-graph prompting (RG) across six models. Role-graph prompting improves performance on five of the six models, with an average accuracy increase from 42.50% to 45.26%. Ovis2.5-9B shows the largest improvement, increasing from 43.16% to 50.28%. Smaller improvements are observed for InternVL2.5-8B and both Qwen2.5-VL models, while GPT-5.1 shows a slight decrease. These results suggest that explicitly encouraging models to consider participant roles and interaction direction can benefit aggressive-scene understanding, although the effectiveness varies across models. We view this as an exploratory approach to structured role-aware prompting rather than a comprehensive solution to compositional role-action reasoning. The detailed prompt and additional results are provided in the Supplementary.

ModelBase ↑RG ↑Δ
InternVL2.5-8B [10]45.7748.15+2.38
Ovis2.5-9B [31]43.1650.28+7.12
Qwen3-VL-8B-Instruct [7]46.9650.65+3.69
Qwen2.5-VL-7B-Instruct [8]37.3038.31+1.01
Qwen2.5-VL-72B-Instruct [8]40.5343.73+3.20
GPT-5.1 [38]41.3040.49-0.81

Table 3. Effect of role-graph prompting. Comparison of baseline overall accuracy with role-graph prompting (RG). Results suggest that explicit role-aware prompting can improve aggressive action and participant association, although gains vary across models.

Conclusion

We propose A3A^{3}Bench, a benchmark for evaluating whether MLLMs can reason about aggressive actions and participant associations in videos. Unlike prior evaluations focused on aggression detection, A3A^{3}Bench measures whether models can identify who initiates an action, who is affected, and how participants interact. Our evaluation across 17 models shows that current systems struggle with this task, with performance declining as questions require more compositional reasoning over actions and roles. We further observe an aggressor bias, where victims and bystanders are frequently misclassified as aggressors, suggesting that models fail to reliably capture interaction structure. Finally, we explore a role-graph prompting strategy that encourages structured reasoning over participant interactions and improves performance on most evaluated models. We hope A3A^{3}Bench serves as a diagnostic benchmark for studying fine-grained role-aware video reasoning.

Acknowledgments and Disclosure of Funding

This research received no external funding and the authors declare no competing interests. We gratefully acknowledge Adams Ubini (University of Maryland, Baltimore County), as well as Tarina Sadek and Godfred Obeng (University of Central Florida), for their valuable support and contributions to this work.

Supplementary material

The supplementary material expands on the main paper with limitations and broader impact discussion, prompt and question-format examples, additional analysis, figures and performance tables, and the full role-graph prompt.

Limitations and Broader Impact

A3A^{3}Bench focuses specifically on visible physical aggression in short video clips and does not cover broader forms of harmful behavior such as verbal abuse, long-term conflict, or subtle social interactions. Furthermore, A3A^{3}Bench is designed to evaluate compositional reasoning over aggressive interactions rather than spatial or temporal action detection, and therefore does not provide participant bounding boxes or temporal tracks. Extending the benchmark with such annotations represents an important direction for future work. While A3A^{3}Bench includes diverse videos, question types, and model families under a controlled evaluation protocol, cross-dataset generalization remains unexplored. Evaluating role-action reasoning across additional datasets and domains is an important direction for future research. This work aims to advance safer and more reliable video understanding systems for applications such as public safety, content moderation, and assistive monitoring. At the same time, aggressive-scene understanding systems may inherit societal biases or produce harmful misclassifications, particularly in high-stakes settings involving surveillance or security. We therefore position A3A^{3}Bench as a diagnostic evaluation framework rather than a deployment-ready solution.

Prompt and Multiple-choice question example

The prompt given to the model follows the format below, where the question and choice examples are as detailed in Figure 8.

Figure 8. Example A3A^{3}Bench questions across categories and list of choices. We show representative structured annotations, and multiple-choice questions generated from a video clip. The left grey panel illustrates the annotated scene information, including aggressor, victim, bystander, and aggressive action descriptions. The bottom-left example shows a Basic question that asks for a single attribute, such as identifying the victim. The middle panel shows a Compound question that requires binding two attributes, such as the action and victim. The right panel shows a Detailed sequence-verification question that requires selecting the full structured event description, including the aggressor, action, victim, and role directionality. Correct answers are highlighted along with green check mark.

Watch this video carefully and answer the following multiple-choice question.
Select ONLY the number (1, 2, 3, etc.) of the correct answer.
Question: <question>
Options: <choices>
Answer with ONLY the option number (e.g., ‘1’ or ‘2’).

Additional Analysis

Effect of temporal sampling. To assess the importance of temporal information for aggressive action and association, we evaluate Qwen3-VL-8B-Instruct using different numbers of uniformly sampled video frames. Table 4 reports performance using a single middle frame and 4, 8, and 16 frames, with 8 frames corresponding to our standard evaluation setting. Increasing temporal coverage from a single frame to 16 frames improves overall accuracy from 40.86% to 51.34%, with improvements across Basic, Compound, and Detailed questions. While performance is not strictly monotonic across all sampling configurations, these results suggest that temporal information is important for recognizing aggressive actions and associating them with the correct participants, beyond the static appearance and scene cues available from individual frames.

SettingBasicCompoundDetailedOverall
1 (center frame)43.9440.5532.2540.86
4 frames51.9844.9839.7247.45
8 frames51.0746.6535.1946.96
16 frames56.7748.5641.6251.34

Table 4. Effect of temporal sampling. Accuracy (%) of Qwen3-VL-8B-Instruct [7] using different numbers of video frames.

No-video baseline. To examine whether models can exploit language or answer-choice priors without visual evidence, we evaluate three MLLMs using only the question and answer options, without providing video frames. As shown in Table 5, overall accuracy remains low across all three models, ranging from 19.21% to 25.07%. These results indicate that language and answer-choice priors alone are insufficient for reliable performance on A3A^3Bench, further highlighting the importance of visual grounding.

ModelBasicCompoundDetailedOverall
Qwen3-VL-8B-Instruct [7]22.6632.4514.2925.07
LLaVA-Video-7B [65]17.2319.4824.8319.21
VideoLLaMA3-7B [64]15.0423.9433.3520.85

Table 5. No-video baseline. Accuracy (%) when models receive only the question and answer choices without video input.

Performance across participant sex. To examine potential demographic differences in participant grounding, we evaluate Basic aggressor identification, victim identification, and role identification questions separately according to the sex of the target participant. Averaged equally across all 17 model configurations, accuracy is nearly identical for male and female participants in aggressor identification (48.48% vs. 47.78%) and role identification (49.08% vs. 51.14%). However, female participants achieve substantially higher accuracy in victim identification (51.33% vs. 40.13%), a gap of 11.20%. This suggests that demographic performance differences are most pronounced in victim identification. Importantly, participant sex is used only for this diagnostic analysis and is not included in the benchmark’s role annotations, questions, or answer choices.

Aggressive scenes for fine-grained understanding. Table 6 compares model performance on questions associated with aggressive and non-aggressive clips. Most MLLMs achieve higher accuracy on non-aggressive clips, with pooled accuracy of 51.02% compared to 37.74% on aggressive clips. This suggests that models struggle more with fine-grained understanding of aggressive interactions, where identifying actions and correctly associating them with participant roles is challenging. Interestingly, this contrasts with the social appropriateness evaluation in Table 9, where models generally perform better on aggressive clips. Together, these findings highlight a distinction between recognizing broad aggressive behavior and accurately reasoning about the actions and participant relationships within aggressive scenes.

Table 6: Performance on non-aggressive and aggressive clips. Overall accuracy (%) is reported separately for questions associated with non-aggressive and aggressive video clips. The higher score for each model is highlighted.

ModelNon-aggressive ↑Aggressive ↑
ViCLIP [55]10.4124.39
X-CLIP [36]18.7020.18
gemma-4-26B-A4B-it [14]73.3341.41
InternVideo2.5-8B [56]54.9640.08
InternVL2.5-8B [10]60.4945.14
InternVL3-9B [69]57.0741.93
InternVL3.5-8B [54]53.6640.52
LLaVA-Video-7B-Qwen2 [65]18.5418.49
Ovis2.5-9B [31]74.4741.82
Qwen2.5-VL-7B-Instruct [8]57.2436.45
Qwen3-VL-8B-Instruct [7]57.5646.51
VideoLLaMA3-7B [64]49.4337.74
Qwen2.5-VL-72B-Instruct [8]61.7939.62
GPT 5.1 [38]59.1940.54
Qwen3-VL-8B-Thinking [7]57.0738.62
InternVL3.5-8B CoT prompt48.2943.76
InternVL3.5-8B DoT [16]55.1244.33

Table 6.

ModelAggressive ↑Non-aggressive ↑
Qwen3-VL-8B [7]89.840.9
Ovis2.5-9B [31]78.260.9
InternVL3-9B [69]78.555.3
InternVL2.5-8B [10]61.065.5

Table 9. Social appropriateness binary classification. Accuracy (%) is reported separately for aggressive and non-aggressive clips. The higher score for each model is highlighted.

Additional Data and Figures

We provide additional graphics and detailed numbers for the results and analysis in the main paper.

Figure 9. Action distribution. Number of video clips for each annotated aggressive action category in the dataset. This follows a long-tail distribution.

Table 7: Question-type-wise performance on A3^{3}Bench. Reported metric is accuracy (%) for each question subcategory and Overall Accuracy. Best and _second-best_ performance are highlighted.

ModelPrimary ActionRole IDAggressor IDVictim Recog.Action+ AggressorsAction+ VictimsAggressor+ VictimAggressor+Action +VictimSequence VerificationOverall Accuracy
ViCLIP [55]25.0824.7833.8630.9222.9925.6111.6518.4013.6723.82
X-CLIP [36]22.9725.6323.2521.9916.1618.469.9820.1320.0020.12
gemma-4-26B-A4B-it [14]48.9248.8153.1045.5037.9236.8048.1330.0031.8142.72
InternVideo2.5-8B [56]43.7054.6849.6741.7737.3938.9843.6421.3221.7240.70
InternVL2.5-8B [10]46.0056.0255.8952.1342.1545.5949.2925.9927.5945.77
InternVL3-9B [69]40.3455.7153.1950.1039.9335.2844.3930.9425.2142.55
InternVL3.5-8B [54]36.1353.6153.8549.6335.9832.4347.4628.8028.6941.06
LLaVA-Video-7B-Qwen2 [65]19.3723.1120.6713.3916.1915.5132.2513.2013.0218.50
Ovis2.5-9B [31]39.8454.8155.6352.2735.6334.8348.2133.5032.6343.16
Qwen2.5-VL-7B-Instruct [8]41.4842.9043.9438.6636.7734.1843.3124.9622.3637.30
Qwen3-VL-8B-Instruct [7]46.4755.4253.0648.0449.2743.8946.8034.6035.8446.96
VideoLLaMA3-7B [64]43.3453.5753.9846.8429.6335.0538.5720.9621.9138.22
Qwen2.5-VL-72B-Instruct [8]39.0050.5450.5941.6739.2235.1048.6726.5327.3440.53
GPT 5.1 [38]44.0651.6645.5938.5039.7737.8740.4333.7532.8441.30
Qwen3-VL-8B-Thinking [7]33.7051.6451.1940.4235.4432.1945.9730.4131.1639.37
InternVL3.5-8B CoT prompt37.7458.5358.2050.9138.6036.2344.4732.5133.8243.94
InternVL3.5-8B DoT [16]37.8551.8657.0853.0140.1536.6447.3839.7742.1644.77

Table 7.

Figure 10. Model-wise performance drop. Comparison of score changes across models for higher level of question complexity.

ModelPrimary Action ↑Action+Location ↑
ViCLIP [55]25.0826.50
X-CLIP [36]22.9715.95
gemma-4-26B-A4B-it [14]48.9231.90
InternVideo2.5-8B [56]43.7033.53
InternVL2.5-8B [10]46.0038.22
InternVL3-9B [69]40.3435.90
InternVL3.5-8B [54]36.1338.99
LLaVA-Video-7B-Qwen2 [65]19.3717.91
Ovis2.5-9B [31]39.8437.70
Qwen2.5-VL-7B-Instruct [8]41.4833.02
Qwen3-VL-8B-Instruct [7]46.4746.74
VideoLLaMA3-7B [64]43.3435.90
Qwen2.5-VL-72B-Instruct [8]39.0035.71
GPT 5.1 [38]44.0640.91
Qwen3-VL-8B-Thinking [7]33.7031.39
InternVL3.5-8B CoT prompt37.7438.30
InternVL3.5-8B DoT [16]37.8540.70

Table 8. Action recognition with and without scene grounding. Accuracy (%) is reported for Primary Action and Action+Location questions. The higher score for each model is highlighted.

Qualitative examples of clear and ambiguous participant grounding

Figure 11. Qualitative examples of clear and ambiguous participant grounding. Top: Clear examples where aggressors and victims are visually distinguishable through appearance, clothing, and interaction cues, enabling reliable role annotations. Bottom: Ambiguous examples where low-quality or black-and-white footage, occlusion, and limited visibility make participant identification difficult. Uncertain role descriptions are left empty rather than guessed, while clips with insufficient visual evidence are excluded from the dataset.

Figure 12. Distractor vulnerability heatmap for Basic questions. Each cell shows the percentage of a model’s errors attributed to each distractor type.

Figure 13. Distractor vulnerability heatmap for Compound questions. Each cell shows the percentage of a model’s errors attributed to each distractor type.

Figure 14. Distractor vulnerability heatmap for Detailed questions. Each cell shows the percentage of a model’s errors attributed to each distractor type.

ModelBasic ↑Compound ↑Detailed ↑Overall ↑Average ↑
InternVL2.5-8B [10]54.5446.3533.0648.1544.65
Ovis2.5-9B [31]55.0846.9443.8350.2848.62
Qwen3-VL-8B-Instruct [7]54.8148.6542.8850.6548.78
Qwen2.5-VL-7B-Instruct [8]37.2741.0034.8338.3137.70
Qwen2.5-VL-72B-Instruct [8]46.5644.5633.0643.7341.39
GPT-5.1 [38]44.1437.9535.5840.4939.22

Table 10. Table 10: Role-graph prompting results on A3^{3}Bench. Reported metric is accuracy (%) for Basic Questions, Compound Questions, Detailed Questions, Overall, and Average. Best and second-best performance are highlighted.

Role-Graph Prompting

We translate the proposed role graph into a structured natural-language prompt for aggressive-scene understanding. The prompt defines each graph component, specifies the directed dependencies between components, and provides illustrative examples for aggressive scenes, bystander cases, and non-aggressive scenes. During evaluation, this role-graph prompt is prepended to each multiple-choice VQA query, and the model is instructed to output only the selected option number. The exact prompt used in our experiments is provided below.

Your primary task is to act as a causal reasoning engine for understanding
aggressive scenes in videos. You will observe the persons present and use a causal
graph to identify each person’s role and the nature of the aggressive interaction.

### Component Definitions

First, understand the meaning of each component (node) in our causal graph. These
are the building blocks we use to analyze any aggressive scene.

• P (Persons): All individuals visible in the video. For example, ‘a man in
  a black coat’, ‘a woman in a red shirt’, ‘three people in the background’.
  Each person must be assigned exactly one role.

• I (Interaction): Whether a given person is engaged in a direct physical
  interaction with another person. This is a binary branch point:
    – YES — the person is physically engaged with at least one other person,
      e.g., striking, grabbing, restraining, being struck.
    – NO — the person has no physical contact with any other person in the
      scene.

• D (Direction): For persons where I = YES, the direction of the aggressive
  interaction:
    – INITIATING — the person is the one delivering the aggressive act, e.g.,
      throwing a punch, grabbing, pushing.
    – RECEIVING — the person is the one on the receiving end of the aggressive
      act, e.g., being punched, being grabbed, being pushed.

• R (Role): The role assigned to each person, inferred from I and D:
    – Aggressor — a person where I = YES and D = INITIATING.
    – Victim — a person where I = YES and D = RECEIVING.
    – Bystander — a person where I = NO, not physically involved.

• A (Aggressive Action): The specific type of aggressive behavior being
  performed, inferred from the Aggressor’s motions and the Victim’s reactions.
  For example, ‘punch’, ‘shove’, ‘kick’, ‘choke’, ‘hair grab’. This is
  determined jointly by the Aggressor and Victim roles.

### The Causal Graph: How Components are Related

The components are connected in a Directed Acyclic Graph (DAG). Each arrow (→)
represents a direction of inference - follow the graph in order to assign roles
correctly.

The directed edges of the graph are:
P→II(YES)→DI(NO)→R(Bystander)D(INITIATING)→R(Aggressor)D(RECEIVING)→R(Victim)R(Aggressor)+R(Victim)→AP \to I \\ I(\mathrm{YES}) \to D \\ I(\mathrm{NO}) \to R(\mathrm{Bystander}) \\ D(\mathrm{INITIATING}) \to R(\mathrm{Aggressor}) \\ D(\mathrm{RECEIVING}) \to R(\mathrm{Victim}) \\ R(\mathrm{Aggressor}) + R(\mathrm{Victim}) \to A

Here are the causal relationships explained in detail:

  • P→IP \to I (Does this person interact?): For every person identified in PP, ask whether they are physically interacting with another person. This is the first branch point. A person standing and watching is I=NOI = \mathrm{NO}. A person throwing a punch is I=YESI = \mathrm{YES}.

  • I(YES)→DI(\mathrm{YES}) \to D (What is the direction of their interaction?): For persons with I=YESI = \mathrm{YES}, determine whether they are initiating or receiving the physical contact. Watch carefully — the aggressor moves toward the victim with force; the victim reacts to the aggressor’s action.

  • I(NO)→R(Bystander)I(\mathrm{NO}) \to R(\mathrm{Bystander}): Any person with no physical involvement is immediately classified as a Bystander. No further reasoning is needed for them.

  • D(INITIATING)→R(Aggressor)D(\mathrm{INITIATING}) \to R(\mathrm{Aggressor}): A person who is initiating the aggressive physical contact is the Aggressor.

  • D(RECEIVING)→R(Victim)D(\mathrm{RECEIVING}) \to R(\mathrm{Victim}): A person who is on the receiving end of the aggressive physical contact is the Victim.

  • R(Aggressor)+R(Victim)→AR(\mathrm{Aggressor}) + R(\mathrm{Victim}) \to A (What is the Aggressive Action?): Only after both the Aggressor and Victim are identified can the Aggressive Action be determined. Observe the Aggressor’s body motion, e.g., arm swinging forward, and the Victim’s reaction, e.g., head snapping back, together to name the action precisely.

Illustrative Examples

Example 1: A clear two-person aggressive scene

A video shows two people. One person in a black jacket swings their fist at a person in a white shirt, who stumbles backward.

  • Causal Analysis (internal, do not output):

    1. P: Two persons — ‘person in a black jacket’, ‘person in a white shirt’.

    2. I for ‘person in a black jacket’: YES (physical contact with fist). I for ‘person in a white shirt’: YES (receiving the fist).

    3. D for ‘person in a black jacket’: INITIATING (swinging the fist). D for ‘person in a white shirt’: RECEIVING (stumbling from the impact).

    4. R: ‘person in a black jacket’ → Aggressor. ‘person in a white shirt’ → Victim.

    5. A: The Aggressor’s motion (arm swinging forward, fist connecting) and Victim’s reaction (head snapping, stumbling) → Aggressive Action = ‘punch’.

Example 2: A scene with a bystander

A video shows three people. One person in a red hoodie grabs another person in a grey jacket by the collar. A third person in a blue shirt stands nearby watching.

  • Causal Analysis (internal, do not output):

    1. P: Three persons — ‘person in a red hoodie’, ‘person in a grey jacket’, ‘person in a blue shirt’.

    2. I for ‘person in a red hoodie’: YES. I for ‘person in a grey jacket’: YES. I for ‘person in a blue shirt’: NO.

  1. D for ‘person in a red hoodie’: INITIATING (grabbing). D for ‘person in a grey jacket’: RECEIVING (being grabbed).

  1. R: ‘person in a red hoodie’ → Aggressor. ‘person in a grey jacket’ → Victim. ‘person in a blue shirt’ → Bystander (I = NO, skip D).

  1. A: The Aggressor grabs the Victim’s collar and pulls – Aggressive Action = ‘clothing grab’.

Example 3: No aggressive action

A video shows two people standing and talking to each other without any physical contact.

  • Causal Analysis (internal, do not output):

    1. P: Two persons.

    2. I for both persons: NO (no physical contact with any other person).

    3. Since I = NO for all persons, no one reaches the D step.

    4. R: Both persons → Bystander (no aggressor, no victim).

    5. A: No Aggressor and no Victim exist → No aggressive action is taking place.

Your Task

You will now be given a new video. Apply the causal graph above to reason about every person you see.

Reason through these steps internally. Do not output your reasoning.

  • P (Persons): Identify all individuals in the video. For each person, carefully describe their appearance – focus first on clothing (color and type, e.g., ‘black coat’, ‘white t-shirt’, ‘blue jeans’), then hair color/style, then any other distinguishing feature. These descriptions will serve as each person’s identifier throughout the rest of your reasoning, and must match the appearance-based labels used in the answer options.

  • I (Interaction): For each person, determine YES or NO – are they physically interacting with another person?

  • D (Direction): For each person with I = YES, determine INITIATING or RECEIVING.

  • R (Role): Assign each person a role – Aggressor, Victim, or Bystander – following the graph edges.

  • A (Aggressive Action): Using the identified Aggressor and Victim, determine the specific aggressive action. If no Aggressor and Victim exist, the answer is that no aggressive action is taking place.

From the list of options below, select the answer that most accurately reflects the question asked about the video. There might also be a choice if none of the other options apply. Review the video and options carefully. Only one choice is correct.

Output format: reply with the choice number only.

NeurIPS Paper Checklist

  1. Claims

    Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

    Answer: [Yes]

    Justification: The abstract and introduction state three contributions: The A3A^3Bench dataset, the diagnostic evaluation of 17 multimodal models, and the role-graph prompting strategy, all of which are supported by experimental results in Section 4.

    Guidelines:

    • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

    • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

    • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

    • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

  1. Limitations

    Question: Does the paper discuss the limitations of the work performed by the authors?

    Answer: [Yes]

    Justification: The supplementary material discuss limitations including the dataset’s focus on physical aggression (excluding verbal or online aggression), annotations, and its positioning as a diagnostic benchmark rather than end-to-end solution.

    Guidelines:

    • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

    • The authors are encouraged to create a separate “Limitations” section in their paper.

    • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

    • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

    • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

    • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

    • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

    • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

  1. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [N/A]

Justification: This paper is an empirical benchmark and evaluation study. It does not include theoretical results, theorems, or proofs.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The question generation procedure, distractor construction, and answer-shuffling protocol are described in § 3.1. § 3.2 and § 3.3 specify all model names, versions, GPU hardware, and evaluation protocol. All evaluated models use publicly available weights or official APIs. The role-graph prompt template is provided in the supplementary material.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The benchmark dataset (annotations, question bank, and evaluation code) has been publicly released on GitHub. Video clips are distributed in Huggingface under a research-use license.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: Section 3.3 reports the models, and evaluation protocol. No training is performed; all models are evaluated zero-shot using their released weights. The role-graph prompting strategy (§ 4.2) is training-free and its full prompt is in the supplementary.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No] Justification: All evaluations are deterministic (greedy decoding, fixed frame sampling, fixed answer-position shuffling), so there is minimal run-to-run variance to report. Each model is evaluated once on the full benchmark of over 15K questions, providing sufficient sample size for stable accuracy estimates.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section 3.3 specifies the models and compute.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: The research conforms to the NeurIPS Code of Ethics. The dataset contains videos of aggressive behavior collected for research purposes. Person descriptions use clothing and coarse visual appearance only, avoiding names, faces, demographic attributes, or other personally identifying information. The benchmark is designed to evaluate and improve safety-critical AI capabilities.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses positive impacts including improved safety monitoring and public safety applications. It also acknowledges potential negative impacts: the aggressor-bias finding highlights that deploying current models could systematically misidentify victims as aggressors, with serious consequences in safety-critical settings.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: The dataset contains sensitive content (videos of aggressive interactions) from public sources. Person descriptions deliberately avoid personally identifying information such as names, faces, or demographic attributes. The dataset is released under a research-use license with terms prohibiting re-identification of individuals and restricting use to academic research.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We collected videos from social-media and trimmed them for research evaluation purposes, which fall under fair usage policy. All source datasets (UCF-Crime, RWF-2000) and evaluated models are cited with their original publications. We are the original owners of the annotations. All models are used via their publicly released weights and official implementations.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: A3A^{3}Bench is a new benchmark dataset. Section 3 documents the video curation, annotation procedure, quality assurance process, question generation, and distractor design in detail. The supplementary material includes the full annotation schema, question type definitions, and dataset statistics. The dataset is released with appropriate documentation.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes] Justification: The dataset annotations were produced by a group of annotators following structured guidelines. The annotation instructions, including role definitions, labeling criteria, and quality assurance procedures, are described in Section 3.1. We also conducted an anonymous human evaluation involving 34 participants, with the evaluation procedure and results described in detail in Section 4.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: The annotators labeled publicly available video content and were not themselves study participants or subjects of research. The annotation task involves labeling observable actions and roles in video clips, not collecting personal data from the annotators. Separately, we conducted a human evaluation involving 34 participants through an anonymous webpage, where participants independently answered video-based multiple-choice questions. No personally identifying information was collected during the evaluation.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: LLMs are not used as a component of the core methodology. The evaluated MLLMs are the subject of the benchmark study, not a methodological tool.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]Shahzad Ahmad, Sukalpa Chanda, and Yogesh S Rawat. T2I: Efficient zero-shot action recognition with temporal token learning. Transactions on Machine Learning Research, 2025.
  2. [2]Mst Shapna Akter, Hossain Shahriar, Nova Ahmed, and Alfredo Cuzzocrea. Deep learning approach for classifying the aggressive comments on social media: Machine translated data vs real life data. In 2022 IEEE International Conference on Big Data (Big Data), pages 5646–5655, 2022.DOI
  3. [3]Mahmoud Ali, Di Yang, and François Brémond. Are visual-language models effective in action recognition? a comparative study. In Alessio Del Bue, Cristian Canton, Jordi Pont-Tuset, and Tatiana Tommasi, editors, Computer Vision – ECCV 2024 Workshops, pages 46–59, Cham, 2025. Springer Nature Switzerland.arxiv.org/abs/2410.17149
  4. [4]Shehzad Ali, Md Tanvir Islam, Ik Hyun Lee, Saeed Anwar, Javier Del Ser, and Khan Muhammad. Iard: Intruder activity recognition dataset for threat detection. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, CIKM ’25, page 6317–6322, New York, NY, USA, 2025. Association for Computing Machinery.DOI
  5. [5]Shehzad Ali, Md Tanvir Islam, Ik Hyun Lee, Mohammad Hijji, and Khan Muhammad. Cabad: A video dataset for benchmarking child aggression recognition. Alexandria Engineering Journal, 127:1143–1157, 2025.DOI
  6. [6]Chaehee An, Minyoung Lee, and Eunil Park. Crxk dataset: a multi-view surveillance video dataset for re-enacted crimes in korea. Scientific Reports, 15(28946), 2025.DOI
  7. [7]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunj iang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xueqiu Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report, 2025.arxiv.org/abs/2511.21631
  8. [8]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.arxiv.org/abs/2502.13923
  9. [9]João Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4733, 2017.arxiv.org/abs/1705.07750
  10. [10]Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.arxiv.org/abs/2412.05271
  11. [11]Ming Cheng, Kunjing Cai, and Ming Li. Rwf-2000: An open large scale video database for violence detection. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 4183–4190, 2021.arxiv.org/abs/1911.05913
  12. [12]Ying Cheng, Yang Li, Junjie He, and Rui Feng. Mixtures of experts for audio-visual learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 219–243. Curran Associates, Inc., 2024.DOI
  13. [13]Federal Bureau of Investigation. Crime in the United States, 2019: Table 42, Arrests by Sex, 2019, 2020.
  14. [14]Google DeepMind. Gemma 4 model card. https://ai.google.dev/gemma/docs/core, 2026. Accessed: 2026-04-28.
  15. [15]Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Freund, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The “Something Something” Video Database for Learning and Evaluating Visual Common Sense . In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5843–5851, Los Alamitos, CA, USA, October 2017. IEEE Computer Society.
  16. [16]Shresth Grover, Vibhav Vineet, and Yogesh S Rawat. Navigating hallucinations for reasoning of unintentional activities, 2024.
  17. [17]Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala. Aqqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11287–11297, June 2021.
  18. [18]Ryo Hachiuma, Fumiaki Sato, and Taiki Sekii. Unified keypoint-based action recognition framework via structured keypoint pooling. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22962–22971, 2023.arxiv.org/abs/2303.15270
  19. [19]Tal Hassner, Yossi Itcher, and Orit Kliper-Gross. Violent flows: Real-time detection of violent crowd behavior. In 3rd IEEE International Workshop on Socially Intelligent Surveillance and Monitoring (SISM) at the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2012.DOI
  20. [20]Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 961–970, 2015.DOI
  21. [21]Xiaohui Huang, Hao Zhou, Kun Yao, and Kai Han. FROSTER: Frozen CLIP is a strong teacher for open-vocabulary action recognition. In The Twelfth International Conference on Learning Representations, 2024.
  22. [22]Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10233–10244, 2020.DOI
  23. [23]Dimitrios Kollias, Damith Chamalki Senadeera, Jianian Zheng, Kaushal K. K. Yadav, Gregory G. Slabaugh, Muhammad Awais, and Xiaoyun Yang. Dvd: A comprehensive dataset for advancing violence detection in real-world scenarios. ArXiv, abs/2506.05372, 2025.arxiv.org/abs/2506.05372
  24. [24]Hilde Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso A. Poggio, and Thomas Serre. Hmdb: A large video database for human motion recognition. 2011 International Conference on Computer Vision, pages 2556–2563, 2011.DOI
  25. [25]Kunchang Li, Yali Wang, Yinhan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi-modal video understanding benchmark. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22195–22206, 2024.arxiv.org/abs/2311.17005
  26. [26]Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, and Limin Wang. Multisports: A multi-person video dataset of spatio-temporally localized sports actions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13536–13545, October 2021.arxiv.org/abs/2105.07404
  27. [27]Bo Liu, Pengfei Qiao, Minhan Ma, Xuange Zhang, Yinan Tang, Peng Xu, Kun Liu, and Tongtong Yuan. Surveillancevqa-589k: A benchmark for comprehensive surveillance video-language understanding with large models, 2025.arxiv.org/abs/2505.12589
  28. [28]Mengnan Liu, Le Wang, Sanping Zhou, Kun Xia, Qi Wu, Qilin Zhang, and Gang Hua. Stepwise multi-grained boundary detector for&nbsp;point-supervised temporal action localization. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part VII, page 333–349, Berlin, Heidelberg, 2024. Springer-Verlag.
  29. [29]Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shu huai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. TempCompass: Do video LLMs really understand videos? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 8731–8772, Bangkok, Thailand, August 2024. Association for Computational Linguistics.arxiv.org/abs/2403.00476
  30. [30]Ziyi Liu and Yangcen Liu. Bridge the gap: From weak to full supervision for temporal action localization with pseudoformer. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8711–8720, 2025.
  31. [31]Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, Shanshan Zhao, Yanqing Ma, Zhichao Wei, Yinglun Li, Lunhao Duan, Jianshan Zhao, Yuxuan Han, Haijun Li, Wanying Chen, Junke Tang, Chengkun Hou, Zhixing Du, Tianli Zhou, Wenjie Zhang, Huping Ding, Jiahе Li, Wen Li, Gui Hu, Yiliang Gu, Siran Yang, Jiamang Wang, Hailong Sun, Yibo Wang, Hui Sun, Jinlong Huang, Yuping He, Shengze Shi, Weihong Zhang, Guodong Zheng, Junpeng Jiang, Sensen Gao, Yi-Feng Wu, Sijia Chen, Yuhui Chen, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. Ovis2.5 technical report. arXiv:2508.11737, 2025.arxiv.org/abs/2508.11737
  32. [32]Weixin Luo, Wen Liu, and Shenghua Gao. A revisit of sparse coding based anomaly detection in stacked rnn framework. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 341–349, 2017.DOI
  33. [33]Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models, 2024.arxiv.org/abs/2306.05424
  34. [34]Abu Bakar Siddique Mahi, Farhana Sultana Eshita, Tabassum Chowdhury, Rashik Rahman, and Tanjina Helaly. Vid: A comprehensive dataset for violence detection in various contexts. Data in Brief, 57:110875, 2024.DOI
  35. [35]Mathew Monfort, Carl Vondrick, Aude Oliva, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfу Fan, and Dan Gutfreund. Moments in Time Dataset: One Million Videos for Event Understanding . IEEE Transactions on Pattern Analysis & Machine Intelligence, 42(02):502–508, February 2020.DOI
  36. [36]Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recognition. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, pages 1–18, Cham, 2022. Springer Nature Switzerland.
  37. [37]Enrique Bermejo Nievas, Oscar Deniz Suarez, Gloria Bueno Garcia, and Rahul Sukthankar. Movies fight detection dataset. In Computer Analysis of Images and Patterns, pages 332–339. Springer, 2011.
  38. [38]OpenAI. Gpt-5.1 instant and gpt-5.1 thinking system card addendum. https://openai.com/index/gpt-5-system-card-addendum-gpt-5-1/, 2025. Accessed: 2026-05-06.
  39. [39]Kunyu Peng, Jia Fu, Kailun Yang, Di Wen, Yufan Chen, Ruiping Liu, Junwei Zheng, Jiaming Zhang, M. Saquib Sarfraz, Rainer Stiefelhagen, and Alina Roitberg. Referring atomic video action recognition. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part XIX, page 166–185, Berlin, Heidelberg, 2024. Springer-Verlag.arxiv.org/abs/2407.01872
  40. [40]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
  41. [41]Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6545–6554, 2023.arxiv.org/abs/2212.03640
  42. [42]Darshana Saravanan, Varun Gupta, Darshan Singh, Zeeshan Khan, Vineet Gandhi, and Makarand Tapaswi. Velocity: Benchmarking video-language compositional reasoning with strict entailment. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 18914–18924, June 2025.
  43. [43]Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine-grained action understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  44. [44]Dingfeng Shi, Yujie Zhong, Qiong Cao, Lin Ma, Jia Li, and Dacheng Tao. Tridet: Temporal action detection with relative boundary modeling. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18857–18866, 2023.
  45. [45]Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 510–526, Cham, 2016. Springer International Publishing.arxiv.org/abs/1604.01753
  46. [46]Mohamed Mostafa Soliman, Mohamed Hussein Kamal, Mina Abd El-Massih Nashed, Youssef Mohamed Mostafa, Bassel Safwat Chawky, and Dina Khattab. Video recognition from videos using deep learning techniques. In 2019 Ninth International Conference on Intelligent Computing and Information Systems (ICICIS), pages 80–85, 2019.
  47. [47]K. Soomro, A. Roshan Zamir, and M. Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. In CRCV-TR-12-01, 2012.arxiv.org/abs/1212.0402
  48. [48]Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6479–6488, 2018.arxiv.org/abs/1801.04264
  49. [49]Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 10078–10093. Curran Associates, Inc., 2022.arxiv.org/abs/2203.12602
  50. [50]United Nations Office on Drugs and Crime. Global Study on Homicide 2023. Technical report. United Nations Office on Drugs and Crime, 2023.DOI
  51. [51]Mengmeng Wang, Zeyi Huang, Xiangjie Kong, Guojiang Shen, Guang Dai, Jingdong Wang, and Yong Liu. Action detail matters: Refining video recognition with local action queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19132–19142, June 2025.DOI
  52. [52]Mengmeng Wang, Jiazhen Xing, Jianbiao Mei, Yong Liu, and Yunliang Jiang. Actionclip: Adapting language-image pretrained models for video action recognition. IEEE Transactions on Neural Networks and Learning Systems, 36(1):625–637, 2025.
  53. [53]Ning Wang, Tieyue Wu, Naeha Sharif, Farid Boussaid, Guangming Zhu, Lin Mei, Mohammed Bennamoun, and Liang Zhang. Skeletoncontext: Skeleton-side context prompting for zero-shot skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20170–20180, June 2026.arxiv.org/abs/2603.29692
  54. [54]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang, Yuchen Duan, Xuehui Wang, Zhi Hou, Haoran Hao, Tianyi Zhang, Songze Li, Xiangyu Zhao, Haodong Duan, Nianchen Deng, Bin Fu, Yinan He, Yi Wang, Conghui He, Botian Shi, Junjun He, Yingtong Xiong, Han Lv, Lijun Wu, Wenqi Shao, Kaipeng Zhang, Huipeng Deng, Biqing Qi, Jiaye Ge, Qipeng Guo, Wenwei Zhang, Songyang Zhang, Maosong Cao, Junyao Lin, Kexian Tang, Jianfei Gao, Haian Huang, Yuzhe Gu, Chengqi Lyu, Huanzhe Tang, Rui Wang, Haijun Lv, Wanli Ouyang, Limin Wang, Min Dou, Xizhou Zhu, Tong Lu, Dahua Lin, Jifeng Dai, Weijie Su, Bowen Zhou, Kai Chen, Yu Qiao, Wenhai Wang, and Gen Luo. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency, 2025.arxiv.org/abs/2508.18265
  55. [55]Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2307.06942
  56. [56]Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, Min Dou, Kai Chen, Wenhai Wang, Yu Qiao, Yali Wang, and Limin Wang. Internvideo2.5: Empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386, 2025.arxiv.org/abs/2501.12386
  57. [57]Bo Wu, Shoubin Yu, Zhenfang Chen, Josh Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
  58. [58]Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. Not only look, but also listen: Learning multimodal violence detection under weak supervision. In European Conference on Computer Vision (ECCV), 2020.arxiv.org/abs/2007.04687
  59. [59]Wenhan Wu, Zhishuai Guo, Chen Chen, Hongfei Xue, and Aidong Lu. Frequency-semantic enhanced variational autoencoder for zero-shot skeleton-based action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11122–11131, October 2025.arxiv.org/abs/2506.22179
  60. [60]Serena Young, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, and Li Fei-Fei. Every moment counts: Dense detailed labeling of actions in complex videos. Int. J. Comput. Vision, 126(2–4):375–389, April 2018.
  61. [61]Kexin Yi*, Chuang Gan*, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2020.arxiv.org/abs/1910.01442
  62. [62]Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc.arxiv.org/abs/2305.06988
  63. [63]Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Shu, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9556–9567, 2024.arxiv.org/abs/2311.16502
  64. [64]Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding. 2025.arxiv.org/abs/2501.13106
  65. [65]Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava-video: Video instruction tuning with synthetic data, 2025.arxiv.org/abs/2410.02713
  66. [66]Chen Zhao, Dawei Du, Anthony Hooqs, and Christopher Funk. Open set action recognition via multi-label evidential learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22982–22991, 2023.arxiv.org/abs/2303.12698
  67. [67]Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6586–6597, 2023.arxiv.org/abs/2212.04501
  68. [68]Anqi Zhu, Jingmin Zhu, James Bailey, Mingming Gong, and Qiuhong Ke. Semantic-guided cross-modal prompt learning for skeleton-based zero-shot action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13876–13885, June 2025.DOI
  69. [69]Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Chu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025.arxiv.org/abs/2504.10479

Paper details

vlnc:W_tub6dsjm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents