betaFeedback

Reading v1View latest version

WorksComputer Science

TraCE: Trajectory-Aware Concepts for Explainable Video Understanding

Abstract

Video understanding models often rely on motion, yet their explanations can obscure which motion sources support a prediction. We identify motion-source under-specification as a limitation of existing video XAI: region-centric explanations can be source-ambiguous, while motion vocabularies restricted to single-person pose sequences are source-incomplete. We propose TraCE (Trajectory-Aware Concepts for Explainable Video Understanding), a concept-based framework that discovers source-flexible motion concepts by clustering optical-flow trajectories from training videos. These concepts represent dynamic evidence from people, objects, and interactions, while object and scene concepts account for static context. The resulting typed trajectory/object/scene interface supports class prediction for action recognition and question-conditioned concept matching for video question answering (VQA). We evaluate TraCE across five random seeds on six benchmarks: NTU RGB+D Mutual, Chiral SSv2, UCF-101, KTH, TOMATO, and STAR. Relative to non-interpretable baselines, TraCE maintains competitive task performance, with an average accuracy drop of 1.0 points for action recognition and a nominal gain of 0.4 points for VQA. Motion perturbation and concept-removal tests provide complementary evidence of directional semantic alignment and decision faithfulness, while sample-level interventions illustrate targeted model debugging. Our project page is available at https://leewooil.github.io/TraCE_project_page/.

Introduction

Motion is a central source of evidence in video understanding. Many events that share similar objects and scenes can differ primarily in how entities move over time [20, 51, 55]. However, modern video understanding models [6, 8, 32, 60, 63] remain largely opaque: their predictions do not reveal whether they rely on dynamic motion, static context, or spurious correlations [2, 14]. Reliable deployment therefore requires explainable video understanding methods that expose the evidence behind each prediction, including whether and how motion contributes. This need is not tied to a particular output format: whether a model predicts an action label or answers a language question, explanations should reveal the dynamic and static evidence behind the decision.

When motion contributes, explaining it as a single factor is insufficient. Motion in videos is not monolithic: it can arise from actors, manipulated objects, or interactions among entities, each carrying different semantics [39, 61]. For example, in a soccer play, a player’s leg swing, the ball trajectory, and a defender’s movement correspond to distinct sources of motion. An explanation that fails to specify these sources suffers from motion-source under-specification: it may indicate that “motion” matters, but not which motion supports the prediction. This failure can take two forms, as shown in Figure 1. An explanation may conflate multiple motion sources into an ambiguous visual cue (source-ambiguous), or omit important sources altogether (source-incomplete). In both cases, users cannot determine whether a model relies on action-relevant body dynamics, object trajectories, interactions, or static contextual shortcuts. Thus, explainable video understanding requires motion explanations that specify dynamic evidence by source and present it distinctly.

Figure 1. Trajectory concepts make motion sources flexible. Existing video XAI often underspecifies which motion source supports a prediction. (a) Heatmap-based explanations [7] can conflate actor motion, object motion, and static context (source-ambiguous). (b) Pose-based concepts [26] cover one person’s joints but omit object motion and interactions (source-incomplete). (c) In contrast, TraCE provides source-flexible motion explanations by representing dynamic evidence from people, objects, and interactions as trajectory concepts.

Despite this need, existing video XAI methods suffer from motion-source under-specification. Attribution-based and region-centric methods [19, 31, 59] explain predictions by producing spatio-temporal heatmaps, tubelets, or other localized visual evidence over input videos. While these outputs can indicate where influential evidence appears, they do not specify what type or source of evidence the model uses. A highlighted region may correspond to static appearance, object or scene context, actor motion, object motion, or their mixture. Thus, such explanations are source-ambiguous: they do not explicitly represent motion as interpretable concepts, let alone separate motion concepts according to their sources. For example, as shown in Figure 1 (a), highlighting a player and a ball cannot tell whether the model relies on the player’s kicking motion, the ball trajectory, or static contextual cues. This forces users to infer motion evidence from frame-to-frame changes in the visualization, making the explanation underspecified and ambiguous.

Concept-level explanations can reduce this ambiguity by exposing prediction evidence as interpretable concepts rather than raw spatio-temporal regions. However, in video understanding, their reliability critically depends on how motion concepts are defined. Language-derived concepts are effective for objects and scenes, but fine-grained motion is often tacit and difficult to express in words [45]. Recent pose-centric concept methods represent motion with human pose or skeleton sequences [26, 27], making single-person body motion explicit and interpretable. This pose-centric formulation, however, is source-incomplete: its motion vocabulary is defined over visible human joints and cannot directly represent dynamic evidence beyond single-person pose, such as object trajectories or multi-agent interactions. As a result, motion from interacting agents or manipulated objects may be omitted, or only indirectly reflected through human-pose or static-context concepts. This calls for motion concepts that are not tied to predefined human joints, but can instead trace dynamic evidence across diverse moving entities.

To address motion-source under-specification, we propose Trajectory-Aware Concepts for Explainable Video Understanding (TraCE), a concept-based framework that treats visual trajectories as first-class explanatory concepts. Derived from optical-flow trajectories, these concepts form a source-flexible vocabulary for dynamic evidence from people, manipulated objects, and interactions. As illustrated in Figure 1, TraCE explains predictions through these trajectory-aware motion concepts, while using object and scene concepts to account for static context. Because these concepts are defined at the level of video evidence rather than task-specific labels, the same typed trajectory/object/scene interface can support both class prediction and question-conditioned answer scoring. This shifts the explanatory bottleneck from describing single-person body poses to representing dynamic visual evidence at the trajectory level. To our knowledge, TraCE is the first concept-based video explanation framework that uses optical-flow trajectory concepts as a unified interface for explaining dynamic evidence across action recognition and VQA.

We instantiate TraCE on two representative video understanding tasks: action recognition and video question answering (VQA). Across six benchmarks—NTU RGB+D Mutual [52], Chiral SSv2 [5, 18], UCF-101 [57], KTH [49], TOMATO [53], and STAR [67]—we evaluate single-person body motion, multi-person interactions, object-centric dynamics, temporally directional cues, and language-grounded reasoning. For VQA, we use native VQA benchmarks together with multiple-choice VQA reformulations of action-recognition datasets, allowing us to test whether the same typed interface supports both class prediction and question-conditioned answer scoring. Spatial grounding analyses and user studies support sample-level alignment with motion sources and human-perceived interpretability, respectively. TraCE maintains competitive task performance, while its trajectory-only variant retains predictive value under object-category shift and shows modest cross-domain transfer. Motion perturbation tests and concept removal provide complementary evidence of directional semantic alignment and decision faithfulness, while sample-level interventions illustrate targeted model debugging.

To summarize, we make the following three contributions.

  • We identify motion-source under-specification as a limitation of existing video explanations and introduce trajectory concepts as a source-flexible representation of dynamic evidence from people, objects, and interactions.

  • We propose TraCE, a concept-based framework that automatically discovers motion concepts by clustering optical-flow trajectories and complements them with object and scene concepts, without requiring manual concept annotation. The resulting typed interface supports sparse classification for action recognition and question-conditioned concept matching for VQA.

  • Five-seed evaluations across six benchmarks show competitive task performance, with an average accuracy drop of 1.0 points for action recognition and a nominal gain of 0.4 points for VQA relative to non-interpretable baselines. Spatial grounding analyses and user studies support sample-level spatial alignment and interpretability, while perturbation and intervention tests provide complementary evidence of directional semantic alignment and decision faithfulness.

Related Work

Video understanding and dynamic evidence. Modern video understanding models span spatio-temporal action recognizers [8, 10, 17, 25, 28, 55, 60] and video-language models for language-grounded reasoning [6, 32, 63]. Yet their predictions can depend on entangled dynamic motion and static context [29]. Recent action-scene or spatial-temporal disentanglement methods address this issue for robustness or hallucination reduction [2, 3, 4], but do not expose dynamic evidence as concepts or specify which motion source supports an individual prediction. TraCE instead explains dynamic evidence with trajectory concepts and static context with object and scene concepts.

Region-centric and post-hoc concept explanations in video XAI. A first form of motion-source under-specification arises in region-centric video XAI. Saliency, perturbation, tube, and supervoxel methods localize where evidence appears [19, 31, 59], but remain source-ambiguous: a highlighted region may mix actor motion, object motion, static appearance, and scene context. Post-hoc concept methods [21, 24, 46] discover or probe recurring spatio-temporal concepts, but their concepts remain unt yped rather than reusable motion concepts. TraCE explains dynamic evidence at the trajectory level, avoiding source-ambiguous regions and generic spatio-temporal concepts.

Concept bottlenecks and pose-centric motion concepts. Concept bottleneck models explain predictions through human-interpretable concepts [23, 40, 44, 58, 69, 70, 71], but video CBMs depend critically on how motion concepts are defined. Language-derived concepts struggle with fine-grained motion, while pose-centric methods [26, 27] restrict motion concepts to human pose or skeleton sequences. Such vocabularies are source-incomplete: they can explain single-person body motion but cannot directly represent object trajectories or multi-agent interactions. TraCE addresses this limitation with optical-flow trajectory concepts, providing a source-flexible motion vocabulary.

Explainable video-language understanding. Beyond action recognition, video-language works improve interpretability through temporal or spatial grounding [1, 15, 30, 33, 68] and textual or structural reasoning traces [11, 12, 34, 47]. These signals localize evidence or organize reasoning, but do not provide concept-level explanations of what dynamic evidence the model uses. In particular, they do not expose reusable motion concepts or distinguish the motion source behind a prediction. TraCE represents dynamic evidence with trajectory concepts, yielding a task-general explanation interface for action recognition and VQA.

TraCE: A Trajectory-Aware Concept Interface

Design Overview

TraCE explains video understanding models through the trajectory-aware concept interface in Figure 2. The interface separates dynamic evidence from static context and exposes predictions through three typed concept groups: for a video ViV_i, zi=[zit;zio;zis]z_i = [z_i^t; z_i^o; z_i^s], where zitz_i^t denotes trajectory concept activations, and zio,zisz_i^o, z_i^s denote object and scene concept activations. The design follows three principles.

Figure 2. Overview of TraCE. TraCE represents videos with a typed trajectory/object/scene concept interface: trajectory concepts capture dynamic evidence, while object and scene concepts capture static context. Action recognition predicts directly from video-side concept activations, whereas VQA projects each question–answer pair into concept relevance and matches it with the video-side concepts. Thus, the same typed interface supports trajectory-based explanations for both class prediction and language-grounded reasoning.

D1. Visual trajectories as motion concepts. TraCE treats visual trajectories as first-class explanatory concepts. Unlike pose-centric motion concepts tied to predefined human joints, trajectory concepts are derived from optical-flow trajectories and capture visible entity-level dynamics from actors, manipulated objects, and interactions among multiple entities. This yields a source-flexible motion vocabulary without manual motion annotation or natural-language motion descriptions.

D2. A typed interface for dynamic and static evidence. As illustrated in Figure 2, trajectory concepts explain dynamic evidence, while object and scene concepts account for static context. The exact concept vocabularies are discovered separately from each downstream dataset’s training videos, but the typed interface—trajectory, object, and scene—remains fixed. This allows TraCE to adapt its concept vocabulary to each dataset while preserving a consistent explanation structure.

D3. Task-general consumption of the concept interface. The typed video-side concept interface supports different video understanding tasks. For action recognition, a sparse classifier predicts the action label directly from ziz_i. For VQA, each question–answer pair is projected into the same concept space as a question-conditioned concept relevance vector and matched with ziz_i. Thus, VQA prediction becomes concept-level: the model answers by matching concepts present in the video with concepts required by the candidate answer. Importantly, trajectory concepts do not require language names. This is crucial for VQA: although trajectory concepts are visual motion clusters, answer supervision can still teach the text branch which concepts are needed for a candidate answer. Object and scene relevance, in contrast, can be supervised using language concept names.

In the following subsections, we present task-agnostic concept discovery (§ 3.2), concept bottleneck learning (§ 3.3), and task instantiations (§ 3.4).

Task-Agnostic Concept Vocabulary Discovery

For each downstream dataset, TraCE discovers concept vocabularies from its training videos, independently of whether the final task is action recognition or VQA. These vocabularies consist of trajectory concepts Ct\mathcal{C}^{t} for entity-level dynamic evidence, and object and scene concepts Co,Cs\mathcal{C}^{o}, \mathcal{C}^{s} for static context. All concepts are discovered automatically without manual concept annotation.

Trajectory Concepts

Trajectory generation. Given a video Vi\mathbf{V}_i, we estimate dense optical flow between consecutive frames using a pretrained flow estimator [64]. We divide the video into NsN_s temporal segments and extract an LL-frame clip around each segment center. For one segment, let Δpr\Delta\mathbf{p}_r denote the optical flow from frame rr to r+1r+1. Starting from a seed pixel p0∈R2\mathbf{p}_0 \in\mathbb{R}^2, we trace a visual trajectory by accumulating flow as illustrated in Figure 3,

Trajectory concept discovery and labeling

Figure 3. Trajectory concept discovery and labeling. TraCE extracts optical-flow trajectories from training videos, filters weak or noisy motion, and clusters the remaining trajectories into recurring motion patterns. Each cluster defines a trajectory concept, and each video is labeled by the normalized frequency of its trajectory assignments. This yields a source-flexible motion vocabulary for people, objects, and interactions without pose annotations or language descriptions of motion.

Δpr=pr+1−pr,τ(p0)=[Δp0,…,ΔpL−2]∈R2(L−1).(1)\Delta\mathbf{p}_r = \mathbf{p}_{r+1} - \mathbf{p}_r,\qquad\tau(\mathbf{p}_0) = [\Delta\mathbf{p}_0,\ldots,\Delta\mathbf{p}_{L-2}] \in\mathbb{R}^{2(L-1)}. \tag*{(1)}

Applying this to all segments yields raw trajectories. We retain foreground trajectories with large motion magnitude, producing a filtered trajectory set Ti\mathcal{T}_i for video Vi\mathbf{V}_i. Details of interpolation, foreground masking, and trajectory filtering are provided in Appendix § 9.§ 4.1.

Trajectory concept discovery and labeling. We aggregate trajectories over the training set, T=⋃iTi\mathcal{T} = \bigcup_i \mathcal{T}_i, and cluster them with a clustering algorithm. TraCE is clustering-agnostic; we use FINCH [48] in our experiments. Let π:T→{1,…,Mt}\pi: \mathcal{T} \to\{1,\ldots,M_t\} denote the cluster assignment. The kk-th trajectory concept is

Ckt={τ∈T∣π(τ)=k},Ct={Ckt}k=1Mt.(2)C_k^t = \{\tau\in\mathcal{T} \mid\pi(\tau) = k\},\qquad\mathcal{C}^t = \{C_k^t\}_{k=1}^{M_t}. \tag*{(2)}

Each concept captures a recurring trajectory-level motion pattern shared across videos. For video Vi\mathbf{V}_i, we count cluster assignments hi,k=∑τ∈Ti1[π(τ)=k]h_{i,k} = \sum_{\tau\in\mathcal{T}_i} \mathbb{1}[\pi(\tau) = k] and form a soft trajectory presence label

cit=hi∥hi∥1∈[0,1]Mt.c_{i}^{t} = \frac{\mathbf{h}_i}{\lVert\mathbf{h}_i \rVert_1} \in[0,1]^{M_t}.

Thus, ci,ktc_{i,k}^{t} indicates the relative frequency of trajectory concept kk in Vi\mathbf{V}_i.

Object and Scene Concepts

To account for static context, we use object and scene concepts. Unlike prior class-name prompting [26], we query training videos with a Video-LLM [6], so the vocabularies reflect visible evidence rather than action-category priors. After filtering long or redundant phrases, we obtain Co\mathcal{C}^{o} and Cs\mathcal{C}^{s}. We assign soft pseudo-labels by video-text similarity using a pretrained vision-language encoder [63]. Additional details on prompts, filtering, and score normalization are provided in Appendix § 9.§ 4.2.

Concept Bottleneck Learning

Given a video Vi\mathbf{V}_i, a frozen video encoder fVf_V extracts xi=fV(Vi)∈Rd\mathbf{x}_i = f_V(\mathbf{V}_i) \in\mathbb{R}^d. We define a video-side concept layer gV:Rd→RMg_V : \mathbb{R}^d \to\mathbb{R}^M, where M=Mt+Mo+MsM = M_t + M_o + M_s is the total number of trajectory, object, and scene concepts. Then we compute raw concept scores ai=gV(xi)=[ait;aio;ais]\mathbf{a}_i = g_V(\mathbf{x}_i) = [a_i^t; a_i^o; a_i^s] and activations zi=[σ(ait);aio;ais]\mathbf{z}_i = [\sigma(a_i^t); a_i^o; a_i^s]. The sigmoid is applied only to trajectory concepts, which are trained with soft-label binary cross-entropy Lt\mathcal{L}_t using the soft trajectory label citc_i^t. Object and scene branches are trained with cosine-cubed losses Lo,Ls\mathcal{L}_o,\mathcal{L}_s using pseudo-labels cio,cisc_i^o,c_i^s, following label-free concept bottleneck models [26, 40]. The video-side concept learning objective is

Lconcept=λtLt+λoLo+λsLs.(3)\mathcal{L}_{\mathrm{concept}} = \lambda_t \mathcal{L}_t + \lambda_o \mathcal{L}_o + \lambda_s \mathcal{L}_s. \tag*{(3)}

This stage initializes the video-side concept interface. For action recognition, we freeze the learned video-side concept layer and train only the sparse classifier. For VQA, we freeze the video encoder but continue updating the video-side concept layer under concept supervision, while learning the text-side concept projector and answer-scoring parameters.

Task Instantiations

For each downstream dataset, TraCE discovers its own trajectory, object, and scene concept vocabularies from training videos. Thus, the exact concepts are task-specific, while the typed interface zi=[zit;zio;zis]\mathbf{z}_i = [\mathbf{z}_i^t;\mathbf{z}_i^o;\mathbf{z}_i^s] is shared, as shown in Figure 2. Action recognition consumes this interface directly with a sparse classifier, while VQA matches it with question-conditioned concept relevance.

Action Recognition

For action recognition, each sample is (Vi,yi)(V_i,y_i), where yi∈{1,…,Ncls}y_i \in\{1,\ldots,N_{\mathrm{cls}}\} is the action label. We freeze the video-side concept layer and train a sparse linear classifier on zi\mathbf{z}_i: y^i=softmax⁡(WARzi)\hat{y}_i = \operatorname{softmax}(W_{\mathrm{AR}}\mathbf{z}_i). We optimize cross-entropy with elastic-net regularization [66] on WARW_{\mathrm{AR}}, encouraging each prediction to rely on a compact set of trajectory, object, and scene concepts.

Video Question Answering via Concept Matching

For VQA, each sample is (Vi,Qi,Ai,yi)(V_i,Q_i,A_i,y_i), where QiQ_i is a question, Ai={ai1,…,aiJ}A_i = \{a_{i1},\ldots,a_{iJ}\} is the set of answer candidates, and yiy_i is the index of the correct answer.

Our VQA instantiation builds on the concept-space matching design of prior work [54], but adapts it to video understanding through a typed trajectory/object/scene interface and a name-free trajectory branch.

Text-side trajectory relevance without motion names. The key challenge is that trajectory concepts are visual motion clusters without natural-language names. Object and scene relevance can be supervised by concept-name similarity, but trajectory relevance cannot. We therefore learn text-side trajectory relevance through answer-supervised alignment with video-side trajectory evidence, rather than through text-concept similarity.

Question–answer projection. Let fTf_T be a pretrained text encoder and gT:RdT→RMg_T : \mathbb{R}^{d_T} \to\mathbb{R}^{M} a text-side concept projector. For each candidate answer, we compute question-conditioned concept relevance:

rij=gT(fT([Qi;aij]))=[rijt;rijo;rijs].(4)\mathbf{r}_{ij} = g_T\left(f_T([Q_i;a_{ij}])\right) = [\mathbf{r}_{ij}^t;\mathbf{r}_{ij}^o;\mathbf{r}_{ij}^s]. \tag*{(4)}

Here, zi\mathbf{z}_i represents video-side concepts, while rij\mathbf{r}_{ij} represents concepts needed to support answer aija_{ij}.

Concept-level answer scoring. We measure a concept-compatibility score of each answer as:

sij=∑g∈{t,o,s}exp⁡(γg)μKg(v(zig)⊙v(rijg)),(5)s_{ij} = \sum_{g\in\{t,o,s\}} \exp(\gamma_g)\mu_{K_g}\left(\mathbf{v}(\mathbf{z}_i^g)\odot\mathbf{v}(\mathbf{r}_{ij}^g)\right), \tag*{(5)}

where μKg\mu_{K_g} sums the top-KgK_g entries of v(z)⊙v(r)\mathbf{v}(\mathbf{z})\odot\mathbf{v}(\mathbf{r}) (⊙\odot denotes element-wise multiplication), following prior works [54, 58], v(x)=ReLU⁡(x)/(∥ReLU⁡(x)∥2+ϵ)\mathbf{v}(\mathbf{x}) = \operatorname{ReLU}(\mathbf{x})/(\lVert\operatorname{ReLU}(\mathbf{x})\rVert_2+\epsilon), and γg\gamma_g is a learnable weight of the concept group gg. The predicted answer is aij^a_{i\hat{j}}, where jipred=arg⁡max⁡jsijj^{\mathrm{pred}}_i = \arg\max_j s_{ij}.

VQA training objective. To preserve concept fidelity, we retain the video-side concept loss LconceptL_{\mathrm{concept}}, computed over videos in each mini-batch. The answer loss is the cross-entropy over candidate scores, denoted LQAL_{\mathrm{QA}}. For object and scene concepts, we supervise text-side relevance with a candidate-averaged cosine loss LtextgL_{\mathrm{text}}^g, g∈{o,s}g \in\{o,s\}, using text-concept similarity labels. For trajectory concepts, text-concept similarity is unavailable, so we align question-conditioned trajectory relevance with video-side trajectory evidence using an InfoNCE loss [41] LaligntL_{\mathrm{align}}^t, where the correct answer is the positive and other candidates are negatives. The overall VQA objective is

LVQA=λQALQA+λconceptLconcept+λtext(Ltexto+Ltexts)+λalignLalignt.(6)L_{\mathrm{VQA}} = \lambda_{\mathrm{QA}}L_{\mathrm{QA}} + \lambda_{\mathrm{concept}}L_{\mathrm{concept}} + \lambda_{\mathrm{text}}\left(L_{\mathrm{text}}^o + L_{\mathrm{text}}^s\right) + \lambda_{\mathrm{align}}L_{\mathrm{align}}^t. \tag*{(6)}

In summary, LconceptL_{\mathrm{concept}} preserves the video-side concept semantics during VQA training, LtextgL_{\mathrm{text}}^g trains text-side relevance for named object and scene concepts, and LaligntL_{\mathrm{align}}^t connects unnamed trajectory concepts to question-conditioned answer relevance.

Experimental Results

We assess explanation interpretability and sample-level grounding (§ 4.1), task performance and generalization (§ 4.2), and concept semantics and decision faithfulness (§ 5). Dataset and implementation details are provided in Appendix § 11 and Appendix § 9, respectively.

Source-Flexible Explanations with Trajectory Concepts

Concept contribution and visualization. For action recognition, the contribution of concept mm to the predicted class c^i=argmax⁡cy^i[c]\hat{c}_i = \operatorname*{argmax}_c \hat{y}_i[c] is zi,mWAR[c^i,m]z_{i,m}W_{\mathrm{AR}}[\hat{c}_i,m], the product of its activation and classifier weight. For VQA, we use the compatibility score sijs_{ij}: each concept contributes through top-KK matching between video-side evidence zig[m]z_i^{g}[m] and question-conditioned relevance rijg[m]r_{ij}^{g}[m]. We visualize trajectory concepts using representative cluster trajectories overlaid on top-activated clips with Grad-CAM [50], used only to localize responses for visualization.

TraCE provides source-flexible motion explanations. We quantify sample-level spatial grounding on object-disjoint Charades-SV2 using object bounding box annotations from Something-Else dataset [38]. For this post-hoc analysis, we associate the top-contributing concepts with input trajectories through cluster assignment. As shown in Table 1, TraCE achieves superior coverage of annotated hand and manipulated-object boxes than DANCE. These results support spatial alignment with both entity types without fixed entity labels for trajectory concepts. Protocol details and additional concept-selection results are provided in Appendix § 16.§ 4.

MethodHandObject
DANCE [26]42.37.6
TraCE64.563.5

Table 1. Motion-source grounding. Top-10 concept support within annotated boxes (%).

Complementary visualizations in Figure 4 show the top-3 contributing trajectory concepts for motion beyond single-person pose dynamics. For pushing other person, the selected concepts correspond to both the pushing motion of one person and the displacement of the other. For moving [something] and [something] closer to each other, they correspond to the objects’ converging movements even though no full human body is visible.

Sample-level explanation showing multi-agent interaction and object-centric dynamics

Figure 4. Sample-level explanation. We visualize the top-3 contributing concepts by overlaying them on top-activated video regions. (a) Interaction dynamics across agents. (b) Object-centric motion with partial human cues. Trajectory concepts capture motion from multiple sources.

Trajectory concepts and explanations are interpretable. We conduct two user studies on motion-focused subsets of NTU RGB+D [52] and Chiral SSv2 [5, 18] (Appendix § 13). First, participants compare TraCE explanations against DANCE [26], a language-based motion-concept CBM, and SALF-CBM [7]. As shown in Figure 5 (a), at least 93% of responses prefer TraCE over DANCE and the language-based baseline, and about 78% prefer TraCE over SALF-CBM. Second, participants rate how well a concept matches the motion shared by its top-4 activating clips. As shown in Figure 5 (b), trajectory concepts achieve the highest score (4.2/5), compared to DANCE pose concepts (2.5/5) and language-based concepts (2.2/5). These results indicate that users find trajectory concepts coherent and better aligned with the motion patterns in the clips.

Figure 5. Interpretability of TraCE. (a) Pairwise preference study comparing TraCE explanations with SALF-CBM [7], a language-based motion-concept CBM, and DANCE [26]. Bars show responses relative to TraCE. (b) Concept-level study comparing trajectory concepts (TraCE), pose-sequence concepts (DANCE), and language-derived motion concepts. Participants rate how well each concept matches the motion shared by its top-activating video clips on a five-point Likert scale.

Concept matching explains language-grounded predictions. Figure 6 shows how TraCE explains a VQA prediction by matching video-side concepts with question-conditioned concept relevance. The correct answer PullUps receives high scores across the top-3 trajectory concepts, which capture repeated vertical motion, while competing answers receive lower scores. To assess whether these concept-based explanations help users understand the model’s answers, we conduct a blinded pairwise study with 50 participants. We control for answer correctness by including only examples for which both compared models predict the same correct answer. Participants prefer TraCE explanations over those based on language-derived motion concepts and pose-sequence concepts in 70.4% and 85.6% of comparisons, respectively, indicating greater perceived usefulness in the VQA setting. The full protocol and response distributions are provided in Appendix § 13.§ 6.

Sample-level VQA explanation

Figure 6. Sample-level VQA explanation. Rows show the top-3 video-side concepts and columns show answer candidates, with cells indicating concept–answer compatibility.

Performance and Generalization of the Trajectory-Aware Interface

We next assess the predictive performance of the typed trajectory/object/scene interface across action recognition and VQA, including transfer under object-category and domain shifts.

TraCE preserves performance across video understanding tasks. In Table 2, we compare TraCE with non-interpretable baselines using the same frozen backbone within each task: Video-MAE [60] for action recognition and InternVideo2 [63] for VQA. Across five random seeds, TraCE remains close to the baseline on average, with a 1.0-point drop for action recognition and a nominal 0.4-point gain for VQA; given the reported variability, we do not interpret the VQA difference as a significant improvement. We also adapt SpLiCE [9] to question-conditioned answer scoring under the same frozen backbones. This single-run adaptation performs below the five-seed mean of TraCE on all five VQA benchmarks. Overall, the concept interface retains competitive task performance while providing concept-based explanations.

MethodNTU RGB+D ARNTU RGB+D VQAChiral SSv2 ARChiral SSv2 VQAUCF-101 ARUCF-101 VQAKTH ARKTH VQATOMATO VQASTAR ARSTAR VQAAvg. ARAvg. VQA
Baseline w/o interpretability99.7±0.297.4±0.186.6±0.282.6±0.289.8±0.199.8±0.091.1±0.335.9±2.254.5±0.491.874.0
LF-CBM [40] w/ language concepts96.9±0.3–77.6±0.5–88.3±0.2–89.1±0.7––88.0–
DANCE [26]97.1±0.3–81.7±0.3–89.0±0.2–91.3±0.4––89.8–
SALF-CBM [7]99.7±0.0–80.4±0.4–87.5±0.2–87.9±0.4––88.9–
SpLiCE [9] adaptation–53.0–57.3–86.7–22.449.2–53.7
TraCE97.4±0.197.1±0.384.9±0.482.6±0.689.0±0.399.7±0.191.8±0.837.0±1.755.4±1.290.874.4

Table 2. Performance comparison on action recognition and VQA benchmarks. We report mean ± sample standard deviation over five random seeds for Top-1 accuracy (%) using the same frozen backbone within each task, except for the single-run SpLiCE adaptation. A dash (–) indicates that the method is not applicable to the task. Task averages are computed over the corresponding dataset means.

Trajectory concepts remain predictive across diverse motion sources. We compare TraCE with LF-CBM [40] using language-derived motion concepts, DANCE using pose-sequence concepts [26], and SALF-CBM [7] on action recognition. Their original action-recognition formulations do not define question-conditioned answer scoring for VQA. Across benchmarks spanning single-person motion (KTH), multi-person interaction (NTU RGB+D Mutual), object-centric and direction-sensitive dynamics (Chiral SSv2), and general human actions (UCF-101), TraCE remains competitive among interpretable methods. It achieves the highest reported mean accuracy on KTH and Chiral SSv2, ties for the best on UCF-101, and remains competitive on the near-saturated NTU RGB+D Mutual split.

Trajectory concepts remain predictive under object-category and visual-domain shifts. We further evaluate whether the trajectory-only variant remains predictive under object-category and visual-domain shifts. Using Something-Else object annotations [38], we construct an object-disjoint Chiral SSv2 [5, 18] split where object categories do not overlap between training and test. As shown in Table 3, the trajectory-only variant of TraCE achieves 94.4%, outperforming the non-interpretable baseline [60] at 89.5%. We also train the concept pipeline on Chiral SSv2 and apply it to aligned TOMATO [53] examples without target-domain training. The trajectory-only variant achieves 43.1%, compared to 41.5% for the baseline. Together, these results suggest that trajectory concepts retain predictive motion cues under object category shift and show modest cross-domain transfer.

MethodSth.-Else disjoint-obj.SSv2→TOMATO
Baseline w/o interpretability89.541.5
TraCE w/o static concepts94.443.1

Table 3. Generalization under object-category and visual-domain shifts.

Faithfulness of TraCE

We examine the directional semantics of activated trajectory concepts. Interpretability alone does not establish that concept explanations reflect model behavior. We therefore examine whether activated concepts retain their directional meaning and whether the model’s predictions depend on them, before illustrating their use for debugging.

Trajectory concepts respond consistently to motion perturbations. We examine sensitivity to signed motion direction at both the prediction and concept levels on the directional-class subset of Chiral SSv2 [5, 18], whose labels specify motion directions. Temporal reversal swaps opposite directions, while a 90∘90^\circ rotation maps each direction to its rotated counterpart. As shown in Table 4, TraCE achieves 81.1% accuracy on original inputs, 79.2% consistency under temporal reversal, and 63.1% under rotation. To assess concept semantics directly, we assign each concept a fixed direction label from the net displacement of its training-cluster medoid. The highest-activated concept matches the ground-truth direction in 67.5% of held-out videos, compared with a 25% uniform four-way reference. Qualitative examples in Figure 7 show that activated trajectory concepts track directional changes under temporal reversal and rotation. Top-5 concept replacement and direction agreement are reported in Appendix § 16.§ 6. These analyses support motion sensitivity and directional semantic alignment; decision dependence is tested below.

Motion perturbation analysis

Figure 7. Motion perturbation analysis. Activated trajectory concepts track directional changes under reversal and rotation. Each panel overlays the first frame at low opacity with the last frame at full opacity.

PerturbationAcc.
Original81.1
Reverse79.2
Rotation63.1

Table 4. Prediction consistency under perturbations.

Contribution rankings identify decision-relevant concepts. Beyond semantic alignment, we test whether contribution rankings identify concepts used for prediction. We progressively deactivate trajectory concepts in descending, random, or ascending contribution order. In Figure 8, top-ranked removal causes sharper accuracy drops than random or bottom-ranked removal on NTU RGB+D Mutual, Chiral SSv2, and KTH; the separation is weaker on UCF-101. The corresponding native-VQA analysis shows that removing the top 5% of trajectory concepts reduces accuracy by

Figure 8. Concept intervention curves show faithfulness of TraCE. We deactivate concepts in descending (Top-kk), random, and ascending (Bottom-kk) order of contribution and report prediction accuracy as the removal ratio increases.

11.9 points on TOMATO and 40.5 points on STAR, compared with 4.3 and 1.3 points under random removal (Table 23). These comparisons assess contribution-ranking faithfulness rather than semantic correctness by themselves.

Removal ratioTOMATO TopTOMATO RandomTOMATO BottomSTAR TopSTAR RandomSTAR Bottom
0%38.038.038.053.453.453.4
5%26.133.738.012.952.153.4
10%18.533.738.014.250.953.4
20%16.333.738.011.748.853.4
40%16.327.238.010.644.953.4
60%19.630.438.011.540.353.4
80%17.426.138.014.833.253.4
100%23.923.923.924.724.724.7

Table 23. Trajectory concept removal on native VQA benchmarks. For each sample, we remove the indicated fraction of trajectory concepts in descending (Top), random, or ascending (Bottom) order of contribution and report answer accuracy (%).

Direction-targeted interventions link semantics to predictions. To connect directional semantics with decision dependence, we use the fixed direction labels above to remove equal numbers of ground-truth (GT)-direction, opposite-direction, or random non-GT concepts on the same directional class subset of Chiral SSv2 used in Table 4 and a direction-annotated subset of TOMATO. As shown in Table 5, GT-direction removal reduces AR accuracy by 24.2 points and flips 30.1% of initially correct predictions; neither control reduces accuracy, and both yield flip rates below 4%. For VQA, GT-direction removal produces the largest accuracy drop (22.5 points) and flip rate (70.5%). The VQA flip rate is descriptive because only 13 of the 28 samples are initially correct. These interventions provide model-internal evidence that direction-aligned trajectory concepts influence predictions in both tasks.

Removed conceptsChiral SSv2 (AR)TOMATO (VQA)
Acc. (%)Δ\DeltaAcc.Flip rate (%)Acc. (%)Δ\DeltaAcc.Flip rate (%)
None81.1––46.4––
GT direction56.9±0.156.9\pm0.1−24.230.1±0.130.1\pm0.123.9±0.423.9\pm0.4−22.570.5±0.570.5\pm0.5
Opposite direction82.2±0.182.2\pm0.1+1.13.6±0.13.6\pm0.154.0±0.254.0\pm0.2+7.630.8±0.030.8\pm0.0
Random non-GT82.1±0.282.1\pm0.2+1.03.7±0.23.7\pm0.245.4±1.345.4\pm1.3−1.021.3±2.021.3\pm2.0

Table 5. Count-matched trajectory concept interventions. We report accuracy, its change from the unablated model (Δ\DeltaAcc.), and the correct-to-incorrect flip rate on direction-annotated subsets. Mean ±\pm standard deviation is computed over 100 concept-subsampling draws. See Appendix § 16.6 for details.

Concept intervention supports sample-level debugging. These concept-level interventions also enable targeted correction of individual errors. In Figure 9 (a), the model misclassifies Moving [something] down as Folding [something], with a mismatched leftward-motion concept contributing to the prediction. Deactivating that concept corrects the prediction. In Figure 9 (b), the model predicts Opening bottle for a Pushing [something] from right to left video because the object concept “bottle cap” is highly influential. Removing this spurious object concept shifts the model from static context to motion evidence and corrects the prediction. TraCE exposes whether an error stems from misleading motion concepts or static context and enables targeted concept-level correction, illustrating how the concept interface can support model debugging.

Figure 9. Concept-level intervention. (a) Deactivating a mismatched trajectory concept fixes a motion error. (b) Deactivating the spurious object concept “bottle cap” shifts the model from static context to motion evidence. These examples illustrate actionable concept-level debugging with TraCE.

Conclusions

In this work, we address motion-source under-specification in explainable video understanding. We propose TraCE, a concept-based framework that derives source-flexible motion concepts from optical-flow trajectories to capture motion from actors, objects, and interactions while separating dynamic evidence from static object and scene context. Across action recognition and video question answering, TraCE maintains competitive task performance while producing explanations that users find interpretable and that intervention analyses show to be faithful to model behavior. These results suggest that trajectory concepts offer a practical typed interface for explaining motion evidence in video understanding.

Acknowledgments and Disclosure of Funding

This work was supported by the National Research Foundation of Korea (NRF) grants funded by the Korea government (MSIT) (RS-2025-02216217, 50%) and (RS-2026-25616405, 25%). Additionally, it was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea Government (MSIT) under grant RS-2024-00353131 (10%) and the Institute of Information & Communications Technology Planning & Evaluation (IITP)-ITRC (Information Technology Research Center) grant funded by the Korea government (MSIT) (IITP-2026-RS-2023-00259004, 10%). This research was also supported by the MSIT (Ministry of Science and ICT), Korea, under the National Program for Excellence in SW (2023-0-00042, 5%) supervised by the IITP (Institute of Information & Communications Technology Planning & Evaluation) in 2026.

Appendix

In this appendix, we provide additional details on the architecture, the optical flow / trajectory pipeline, datasets, baselines, and the user study, to complement the main paper. We organize the appendix as follows:

  • Architecture and implementation details (Appendix § 9)

  • Dataset details (Appendix § 11)

  • Baseline details (Appendix § 12)

  • Setup and protocol of the user study (Appendix § 13)

  • Additional experimental results and analyses (Appendix § 16)

  • Limitations (Appendix § 17)

  • Broader impacts (Appendix § 18)

Architecture and Implementation Details

In this section, we describe the backbones, the concept layers and task heads, the concept discovery pipeline, and the training details of TraCE.

Action Recognition Backbone

For action recognition, we use VideoMAE-B/16 [60] as the default video encoder fVf_V. Given an input video, fVf_V outputs a video-level feature xi∈Rd\mathbf{x}_i \in\mathbb{R}^{d}, and the video-side concept layer gVg_V takes this feature as its input. VideoMAE-B/16 inherits its layout from ViT-Base (12 transformer blocks, hidden dimension 768, 12 attention heads, tubelets of size 2×16×162 \times16 \times16), and we thus set d=768d = 768.

Data preprocessing. We sample T=16T = 16 frames per video as input to fVf_V. We use dense sampling for UCF-101 [57] and uniform sampling for KTH [49], NTU RGB+D Mutual [52], and Chiral SSv2 [5, 18]. After sampling, we resize each frame so that the shorter side is 224 pixels and apply random cropping to 224×224224 \times224. We do not apply any other data augmentation; in particular, we disable random erasing, mixup, and CutMix during fine-tuning.

Pretrained weights. For Chiral SSv2 and UCF-101, we directly use the publicly released VideoMAE-B/16 [60] fine-tuned weights from the official repository, without further fine-tuning. For KTH and NTU RGB+D Mutual, we initialize fVf_V from the VideoMAE-B/16 weights pretrained on Something-Something-V2 [18] and fine-tune on each target dataset. Something-Something-V2 is a large-scale, temporally fine-grained dataset, and the self-supervised VideoMAE weights pretrained on it are particularly effective for downstream datasets where temporal dynamics play a crucial role, such as KTH and NTU RGB+D Mutual.

Training. We fine-tune fVf_V on KTH and NTU RGB+D Mutual for 50 epochs with 5 warmup epochs and a cosine learning rate schedule [35]. We use the AdamW optimizer [36] (β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, weight decay 0.05), apply layer-wise learning rate decay of 0.75, and use a base learning rate of 5e−45\mathrm{e}{-4} for KTH and 1e−31\mathrm{e}{-3} for NTU RGB+D Mutual. We conduct backbone training on 4 NVIDIA GeForce RTX 3090 GPUs with a batch size of 10 per GPU. We linearly scale the base learning rate with respect to the total batch size using the rule lr=base learning rate×batch size/256lr = \text{base learning rate} \times\text{batch size}/256, following VideoMAE [60]. We list the per-dataset hyperparameters in Table 6. After fine-tuning, we freeze fVf_V and pre-extract its video-level features xi\mathbf{x}_i for every training and validation sample. This pre-extraction strategy decouples backbone training from concept-layer training and enables efficient and reproducible optimization of gVg_V and WAR\mathbf{W}_{AR}.

ConfigKTHNTU RGB+D Mutual
ModelViT-Base/16ViT-Base/16
Pretrained weightsVideoMAE-B/16 (SSv2)VideoMAE-B/16 (SSv2)
Num. frames1616
Input size224×224224 \times224224×224224 \times224
Short-side size224224
Frame samplinguniformuniform
OptimizerAdamWAdamW
Optimizer momentumβ1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999
Base learning rate5e−45\mathrm{e}{-4}1e−31\mathrm{e}{-3}
Weight decay0.050.05
Learning rate schedulecosine decaycosine decay
Warmup epochs55
Layer-wise lr decay0.750.75
Training epochs5050
GPUs4×4 \times RTX 30904×4 \times RTX 3090
Batch size per GPU1010
Augmentationnonenone
Test segments22
Test crops33

Table 6. Hyperparameters used for fine-tuning the action recognition backbone fVf_V on KTH and NTU RGB+D Mutual. For Chiral SSv2 and UCF-101, we directly use the publicly released VideoMAE-B/16 fine-tuned weights from the official repository.

Video Question Answering Backbone

For video question answering, we use the Stage-2 1B variant of InternVideo2 [63]—denoted as InternVideo2S2−1B_{S2-1B}—as the video encoder fVf_V, and we use its paired text tower as the text encoder fTf_T. We adopt InternVideo2S2−1B_{S2-1B} because Stage-2 aligns the video and text encoders into a shared embedding space through video–audio–speech–text contrastive learning. This alignment matches the video-side concept layer gVg_V and the text-side concept projector gTg_T to a shared concept space, rather than to two independently learned spaces, so that the matching operator in § 3.4.2 can compare the two outputs dimension-wise. The text tower is initialized from the first 19 layers of BERT-Large [16]; for each question–answer pair [Qi;aij][Q_i;a_{ij}], fTf_T outputs a text feature uij=fT([Qi;aij])∈RdTu_{ij}=f_T([Q_i;a_{ij}]) \in\mathbb{R}^{d_T}. We initialize both towers with the publicly released InternVideo2s2_{s2}-1B weights and freeze them throughout VQA training; we only update gVg_V, gTg_T, and the per-group weights γg\gamma_g used in the matching operator.

Data preprocessing. We sample T=4T=4 frames per video using uniform stride sampling: we partition the video into TT equal-length segments and select the first frame of each segment. We resize each frame so that the shorter side is 224 pixels and apply center cropping to 224×224224 \times224. For text inputs, we tokenize the question–answer pair [Qi;aij][Q_i;a_{ij}] with the BERT-Large [16] tokenizer paired with the InternVideo2 text tower, and we feed the tokenized sequence into the text tower without additional preprocessing. We do not apply any data augmentation during VQA training.

Non-interpretable VQA baseline. The non-interpretable VQA baseline used as the reference point for TraCE in Table 2 trains a single linear projection layer on top of the frozen video and text features to adapt them to the multiple-choice format. Specifically, we apply two separate linear projections, ϕV:Rd→Rdp\phi_V:\mathbb{R}^{d}\to\mathbb{R}^{d_p} and ϕT:RdT→Rdp\phi_T:\mathbb{R}^{d_T}\to\mathbb{R}^{d_p}, to the video feature xi=fV(Vi)\mathbf{x}_i=f_V(V_i) and the question–answer feature uij=fT([Qi;aij])u_{ij}=f_T([Q_i;a_{ij}]), with a shared projection dimension dp=512d_p=512. We ℓ2\ell_2-normalize the two projections and score each candidate answer with their dot product:

sijbase=exp⁡(γ)⋅ϕV(xi)∥ϕV(xi)∥2⋅ϕT(uij)∥ϕT(uij)∥2,(7)s_{ij}^{\mathrm{base}}=\exp(\gamma)\cdot\frac{\phi_V(\mathbf{x}_i)}{\lVert\phi_V(\mathbf{x}_i)\rVert_2}\cdot\frac{\phi_T(u_{ij})}{\lVert\phi_T(u_{ij})\rVert_2}, \tag*{(7)}

where γ\gamma is a learnable log-temperature that we initialize to 2.3, so that exp⁡(γ)≈10\exp(\gamma)\approx10 at the start of training. We optimize the parameters of ϕV\phi_V, ϕT\phi_T, and γ\gamma with the cross-entropy loss over the candidate scores {sijbase}j=1J\{s_{ij}^{\mathrm{base}}\}_{j=1}^{J} against the ground-truth answer index yiy_i. We use the same training configuration across all video question answering datasets (TOMATO [53], STAR [67], and the multiple-choice reformulations of NTU RGB+D Mutual [52], Chiral SSv2 [5, 18], and UCF-101 [57]); we list the hyperparameters in Table 7.

ConfigValue
Frozen backboneInternVideo2s2_{s2}-1B
Num. frames4
Input size224×224224 \times224
Frame samplinguniform stride
Projection dimension dpd_p512
Feature normalizationℓ2\ell_2
Temperature init (γ\gamma)2.3
OptimizerAdam [22]
Base learning rate1e−31\mathrm{e}{-3}
Weight decay0
Gradient clipping1.0
Batch size256
Training epochs100
Random seed42

Table 7. Hyperparameters for training the baseline projection layer. The video and text encoders (fVf_V and fTf_T) are kept frozen; only ϕV\phi_V, ϕT\phi_T, and the learnable temperature γ\gamma are updated. We use the same configuration across all video question answering datasets.

Concept Layers and Task Heads

Video-side concept layer. The video-side concept layer gV:Rd→RMg_V:\mathbb{R}^{d}\to\mathbb{R}^{M}, where M=MT+MO+MSM=M_T+M_O+M_S, is a group-wise bias-free linear projection with three disjoint weight matrices WV∈RMT×d\mathbf{W}_V\in\mathbb{R}^{M_T\times d}, $\mathbf{W}_V^O\in RMo×d\mathbb{R}^{M_o \times d}, and WVs∈RMs×dW_V^s \in\mathbb{R}^{M_s \times d}, giving raw scores aig=WVgxia_i^g = W_V^g x_i for g∈{t,o,s}g \in\{t,o,s\}. We apply an element-wise sigmoid only to the trajectory branch: zit=σ(ait)z_i^t = \sigma(a_i^t), zio=aioz_i^o = a_i^o, zis=aisz_i^s = a_i^s. We omit the bias term in all three branches, following prior label-free concept bottleneck models [40]. We train the trajectory branch with soft-label binary cross-entropy against the soft trajectory presence label cit∈[0,1]Mtc_i^t \in[0,1]^{M_t} defined in § 3.2.1:

Lt=−1Mt∑k=1Mt[ci,ktlog⁡zi,kt+(1−ci,kt)log⁡(1−zi,kt)].(8)L_t = -\frac{1}{M_t}\sum_{k=1}^{M_t}\left[c_{i,k}^t \log z_{i,k}^t + (1-c_{i,k}^t)\log(1-z_{i,k}^t)\right]. \tag*{(8)}

We train the object and scene branches with the cosine-cubed loss [40] against the pseudo-labels cigc_i^g defined in § 3.2.2:

Lg=−(zig)3⋅(cig)3∥(zig)3∥2∥(cig)3∥2,g∈{o,s}.(9)L_g = -\frac{(z_i^g)^3 \cdot(c_i^g)^3}{\lVert(z_i^g)^3\rVert_2\lVert(c_i^g)^3\rVert_2}, \quad g \in\{o,s\}. \tag*{(9)}

The object and scene branches keep their raw scores because this loss operates on directional alignment rather than per-dimension probabilities. We instantiate a separate gVg_V for each task: for action recognition, we freeze gVg_V after concept bottleneck learning and train only the sparse classifier WARW_{\mathrm{AR}}; for video question answering, we initialize gVg_V from the concept bottleneck learning stage and continue updating it under concept supervision.

Text concept layer. We use the text-side concept projector gT:RdT→RMg_T : \mathbb{R}^{d_T} \to\mathbb{R}^{M} only in video question answering. gTg_T shares the same group-wise bias-free design as gVg_V, with weight matrices WTtW_T^t, WToW_T^o, and WTsW_T^s, and produces the question-conditioned concept relevance rij=gT(uij)=[rijt;rijo;rijs]r_{ij} = g_T(u_{ij}) = [r_{ij}^t;r_{ij}^o;r_{ij}^s]. We supervise object and scene relevance by text-concept similarity to natural-language concept names with a candidate-averaged cosine loss:

Ltextg=−1J∑j=1Jrijg⋅dijg∥rijg∥2∥dijg∥2,g∈{o,s},(10)L_{\mathrm{text}}^g = -\frac{1}{J}\sum_{j=1}^{J}\frac{r_{ij}^g \cdot d_{ij}^g}{\lVert r_{ij}^g\rVert_2\lVert d_{ij}^g\rVert_2}, \quad g \in\{o,s\}, \tag*{(10)}

where dijgd_{ij}^g is the text-side pseudo-label obtained by comparing the embedding of [Qi;aij][Q_i;a_{ij}] with the concept-name embeddings in CgC_g, both encoded with the all-mpnet-base-v2 model [65]. Trajectory relevance, in contrast, has no language names and cannot be supervised by text-concept similarity. We therefore align question-conditioned trajectory relevance with video-side trajectory evidence through an InfoNCE loss [41], where the relevance vector of the correct answer is the positive and other candidates are negatives:

Lalignt=−log⁡exp⁡(sim⁡(zit,riyt)/τ)∑j=1Jexp⁡(sim⁡(zit,rijt)/τ),(11)L_{\mathrm{align}}^t = -\log\frac{\exp(\operatorname{sim}(z_i^t,r_{iy}^t)/\tau)}{\sum_{j=1}^{J}\exp(\operatorname{sim}(z_i^t,r_{ij}^t)/\tau)}, \tag*{(11)}

where sim⁡(⋅,⋅)\operatorname{sim}(\cdot,\cdot) is cosine similarity and τ\tau is a temperature.

Sparse linear classifier for action recognition. For action recognition, the sparse linear classifier WAR∈RK×M\mathbf{W}_{\mathrm{AR}} \in\mathbb{R}^{K \times M} predicts y^i=softmax⁡(WARzi)\hat{y}_i = \operatorname{softmax}(\mathbf{W}_{\mathrm{AR}} z_i), where KK is the number of action classes. We optimize cross-entropy with elastic-net regularization on WAR\mathbf{W}_{\mathrm{AR}} [66]:

LAR=−log⁡y^i[yi]+λ[(1−α)12∥WAR∥F2+α∥WAR∥1,1],(12)\mathcal{L}_{\mathrm{AR}} = -\log\hat{y}_i[y_i] + \lambda\left[(1-\alpha)\frac{1}{2}\lVert\mathbf{W}_{\mathrm{AR}} \rVert_F^2 + \alpha\lVert\mathbf{W}_{\mathrm{AR}} \rVert_{1,1}\right], \tag*{(12)}

where ∥⋅∥F\lVert\cdot\rVert_F is the Frobenius norm and ∥⋅∥1,1\lVert\cdot\rVert_{1,1} is the element-wise ℓ1\ell_1 norm. We optimize WAR\mathbf{W}_{\mathrm{AR}} with the GLM-SAGA solver [66], following prior label-free concept bottleneck models [26, 40].

Sparse matching operator for video question answering. For video question answering, we score each candidate answer with the sparse matching operator in § 3.4.2. We use Kt=Ko=Ks=5K_t = K_o = K_s = 5 across all video question answering experiments and set ε=10−8\varepsilon= 10^{-8} in the ReLU ℓ2\ell_2-normalization ν(⋅)\nu(\cdot). We initialize γg=0\gamma_g = 0 for all groups, which corresponds to exp⁡(γg)=1\exp(\gamma_g) = 1 at the start of training, so that all three concept groups contribute equally, and we update γg\gamma_g jointly with gVg_V and gTg_T.

Concept Discovery

Trajectory Concept

Optical flow estimation. We use WAFT [64] as the optical flow estimator across all experiments. WAFT is a pretrained dense flow estimator that produces a per-pixel displacement field between two consecutive frames. We use the official released WAFT weights and do not fine-tune the estimator on any target dataset. Keeping the flow estimator frozen makes the trajectory pipeline backbone-agnostic and reproducible across datasets. For each video, we sample frames at the original frame rate and compute forward flow between every pair of adjacent frames. We perform optical flow estimation once per dataset and cache the resulting flow fields and trajectories on disk, so that trajectory clustering and concept-layer training can both reuse the cached results without recomputation.

Trajectory generation. Given a video ViV_i, we partition it into NsN_s temporal segments of equal duration and extract an LL-frame clip centered at each segment center. Within a segment, let Fr:Ω→R2\mathbf{F}_r : \Omega\to\mathbb{R}^2 denote the forward flow field from frame rr to frame r+1r+1, where Ω={0,…,W−1}×{0,…,H−1}\Omega= \{0,\ldots,W-1\} \times\{0,\ldots,H-1\} is the discrete pixel grid. For each seed pixel p0∈Ω\mathbf{p}_0 \in\Omega, we trace a visual trajectory by recursively accumulating optical flow:

pr+1=pr+Fr(pr),r=0,…,L−2.(13)\mathbf{p}_{r+1} = \mathbf{p}_r + \mathbf{F}_r(\mathbf{p}_r), \qquad r = 0,\ldots,L-2. \tag*{(13)}

Although the seed p0\mathbf{p}_0 lies on the discrete grid, subsequent positions pr\mathbf{p}_r may take sub-pixel coordinates. We therefore evaluate Fr(pr)\mathbf{F}_r(\mathbf{p}_r) via bilinear interpolation. We represent each trajectory as the concatenated sequence of displacement vectors,

τ(p0)=[Δp0,…,ΔpL−2]∈R2(L−1),(14)\tau(\mathbf{p}_0) = [\Delta\mathbf{p}_0,\ldots,\Delta\mathbf{p}_{L-2}] \in\mathbb{R}^{2(L-1)}, \tag*{(14)}

where Δpr=pr+1−pr\Delta\mathbf{p}_r = \mathbf{p}_{r+1} - \mathbf{p}_r. Applying this procedure to every segment yields the raw trajectory set T~i\widetilde{\mathcal{T}}_i for ViV_i. The dataset-specific values of LL and NsN_s are listed in Appendix § 9.§ 6.

Trajectory filtering. Raw dense trajectories contain three failure modes: static background, weak or near-zero motion, and noisy flow predictions in low-texture regions. We address all three with a two-stage filter applied per segment. For notational simplicity, we describe the procedure for one segment and omit the segment index. Let P~\widetilde{\mathcal{P}} denote the set of seed pixels in the segment, and let M:Ω→{0,1}M : \Omega\to\{0,1\} denote the foreground saliency mask obtained by an off-the-shelf foreground estimator [42] on the segment’s center frame. Stage 1 (foreground gating). For each seed pixel p0∈P~\mathbf{p}_0 \in\widetilde{\mathcal{P}}, we retain its trajectory only if M(p0)=1M(\mathbf{p}_0) = 1. This focuses the trajectory pool on visible entity-level motion such as people, manipulated objects, and interacting agents. Stage 2 (magnitude ranking). Among the remaining foreground trajectories, we compute the motion magnitude

m(τ)=∑r=0L−2∥Δpr∥2,(15)m(\tau) = \sum_{r=0}^{L-2} \lVert\Delta\mathbf{p}_r \rVert_2, \tag*{(15)}

and retain the top-ρ%\rho\% trajectories ranked by m(τ)m(\tau). We use ρ=5\rho= 5 in all experiments. This stage suppresses low-amplitude flow noise and concentrates the trajectory vocabulary on salient entity-level motion. Applying both stages to every segment yields the filtered trajectory set Ti\mathcal{T}_i for ViV_i.

Trajectory clustering and labeling. We aggregate the filtered trajectories over the training set, T=⋃i=1NTi\mathcal{T} = \bigcup_{i=1}^{N} \mathcal{T}_i, and cluster them with FINCH [48]. FINCH produces a hierarchy of partitions at different levels of granularity without requiring a predefined number of clusters. We select a single partition level for each downstream dataset, and the resulting number of clusters MtM_t varies across datasets accordingly. We list the partition level and the resulting MtM_t per dataset in Appendix § 9.§ 6. Let π:T→{1,…,Mt}\pi: \mathcal{T} \to\{1,\ldots,M_t\} denote the cluster assignment of the selected partition. The kk-th trajectory concept is the set of trajectories assigned to cluster kk, Ckt={τ∈T∣π(τ)=k}\mathcal{C}_k^t = \{\tau\in\mathcal{T} \mid\pi(\tau) = k\}. Each concept captures a recurring trajectory-level motion pattern shared across videos. For each video ViV_i, we count how often its filtered trajectories are assigned to each concept,

hi,k=∑τ∈Ti1[π(τ)=k],(16)h_{i,k} = \sum_{\tau\in\mathcal{T}_i} \mathbb{1}[\pi(\tau) = k], \tag*{(16)}

and form a soft trajectory presence label

cit=hi∥hi∥1∈[0,1]Mt.(17)\mathbf{c}_i^t = \frac{\mathbf{h}_i}{\lVert\mathbf{h}_i \rVert_1} \in[0,1]^{M_t}. \tag*{(17)}

The entry ci,ktc_{i,k}^t is the relative frequency of trajectory concept kk in ViV_i, and serves as the soft target for the trajectory branch in Equation (8).

Concept visualization. To visualize a trajectory concept, we select the medoid trajectory of its cluster as the canonical example. The medoid is the trajectory with the smallest average distance to all other members of the cluster, measured in the same flattened displacement space used for clustering. For each visualized concept, we identify the most strongly activating training video and locate the region in that video where the concept activation comes from using Grad-CAM [50]. We then overlay the medoid trajectory on the Grad-CAM-localized region, so that the visualization shows both the canonical motion pattern and the spatial region in the video that triggers the concept. We use Grad-CAM only to localize concept responses for visualization purposes, not to compute the explanations themselves. We follow this same protocol for all qualitative figures involving trajectory concepts.

Object and Scene Concept

Object and scene concept candidate generation. We construct object and scene concept vocabularies by querying the training videos with the Video-LLM Qwen3-VL [6]. Unlike prior class-name prompting [26, 40], this video-grounded querying makes the candidate vocabulary reflect visible evidence rather than action-category priors. For each training video, we run two independent Video-LLM queries, one for object candidates and one for scene candidates, and we union the per-video outputs over the training set. The object prompt asks Qwen3-VL to list the main physical objects clearly visible in the video, with explicit instructions to exclude people, body parts, actions, and background locations, and to output a short bullet list of singular lowercase noun phrases. The scene prompt asks Qwen3-VL to list the background scenes or environment types clearly visible in the video, with explicit instructions to exclude people, actions, objects, and equipment. Both prompts cap the output length and provide in-context examples to enforce the bullet-list format. We provide the full prompt templates in Figure 10. Aggregating the per-video outputs yields the candidate sets C~o\widetilde{C}^{o} and C~s\widetilde{C}^{s}.

Figure 10. Concept candidate generation prompts. (a) Object and (b) scene prompts used to query Qwen3-VL [6].

Object and scene concept filtering. We refine C~o\widetilde{C}^{o} and C~s\widetilde{C}^{s} with four filters from LF-CBM [40]. (i) Length filter. We discard any candidate concept longer than 30 characters to keep concepts simple and to avoid unnecessarily long phrases. (ii) Redundancy filter. We remove duplicate and synonymous concepts using a weighted text-embedding similarity score that combines two encoders: MPNet [65] and CLIP ViT-B/16 [43]. (iii) Activation filter. For each remaining candidate, we compute its average video–text similarity score under the pretrained vision–language encoder [63] across the training set, and we discard concepts whose average score falls below an activation cutoff. This filter removes concepts that are not actually represented in the training data. (iv) Projection filter. After training the concept layer with the surviving candidates, we further remove concepts whose projected activations are consistently low across the training set, using a projection cutoff. This filter removes concepts that the model does not rely on after optimization. We do not apply the LF-CBM filter that removes concepts similar to target class names, because our object and scene candidates are generated from video evidence rather than action class names and rarely overlap with class labels. We tune the activation and projection cutoffs per dataset; the exact values and the resulting vocabulary sizes MoM_o and MsM_s are reported in Appendix § 9.§ 6.

Concept labeling. For each training video ViV_i and each concept group g∈{o,s}g \in\{o,s\}, we generate a soft pseudo-label using a pretrained vision-language encoder ET(⋅),EV(⋅)E_T(\cdot), E_V(\cdot). We use InternVideo2 [63] as the encoder to align the pseudo-labels with the VQA objective described in Appendix § 9.§ 2. Specifically, we encode every concept name in CgC^g with the text encoder, yielding the ℓ2\ell_2-normalized concept embedding matrix ET(Cg)∈RMg×dEE_T(C^g) \in\mathbb{R}^{M_g \times d_E}. We encode the video ViV_i with the video encoder, yielding EV(Vi)∈RdEE_V(V_i) \in\mathbb{R}^{d_E}. The soft pseudo-label is

cig=ET(Cg)EV(Vi)∈RMg,g∈{o,s}.(18)c_i^g = E_T(C^g) E_V(V_i) \in\mathbb{R}^{M_g}, \quad g \in\{o,s\}. \tag*{(18)}

We use the raw cosine similarity scores as soft pseudo-labels without further normalization. We use cioc_i^o and cisc_i^s as the soft targets for the object and scene branches in Equation (9).

TraCE Training Details

After training (or selecting) the backbone encoder, we freeze its parameters and pre-extract feature vectors for all training and validation samples. We conduct all concept-layer and classifier training on a single NVIDIA GeForce RTX 3090 GPU.

Video-side concept layer training. We train the video-side trajectory branch WVt\mathbf{W}_V^t separately from the object and scene branches WVo\mathbf{W}_V^o and WVs\mathbf{W}_V^s, following DANCE [26]. The trajectory branch optimizes LtL_t in (8) against the soft trajectory presence label citc_i^t obtained from FINCH cluster assignments (Appendix § 9.§ 4.1), and the object and scene branches optimize LgL_g in (9) against the InternVideo2 pseudo-labels cigc_i^g (Appendix § 9.§ 4.2). For all branches, we use the Adam optimizer [22] with a learning rate of 1e−31\mathrm{e}{-3} and a batch size of 256, and we train for 3,000 steps with early stopping based on validation loss. We use the same hyperparameters across all action recognition and video question answering datasets.

Text-side trajectory concept layer training. We train the text-side trajectory branch WTt\mathbf{W}_T^t jointly with the video-side trajectory branch WVt\mathbf{W}_V^t and the trajectory matching weight γ\gamma, since trajectory concepts have no natural-language names and the text branch must instead be supervised by the video-side trajectory evidence (§ 3.4.2). We retain the video-side trajectory concept loss LtL_t during VQA training and optimize it together with the trajectory alignment loss LaligntL_{\mathrm{align}}^t in (11) and the answer cross-entropy LQAL_{\mathrm{QA}}, following (6). We use the Adam optimizer with a learning rate of 1e−31\mathrm{e}{-3} and a batch size of 256, and we train for 100 epochs. We initialize WVtW^t_V from concept bottleneck learning, initialize WTtW^t_T from scratch, and set γt=0\gamma_t = 0.

Text-side object and scene concept layer training. During VQA training, we jointly update the text-side object and scene branches WToW^o_T, WTsW^s_T, the corresponding video-side branches, and their matching weights γo\gamma_o, γs\gamma_s. The objective combines the cosine-cubed loss on the video side (Lo+LsL_o + L_s in (9)), the candidate-averaged cosine loss on the text side (Ltexto+LtextsL^o_{\mathrm{text}} + L^s_{\mathrm{text}} in (10)), and the video question answering cross-entropy LQAL_{\mathrm{QA}}, with weights of 0.8 on the concept losses and 0.2 on LQAL_{\mathrm{QA}}. We use the Adam optimizer with a learning rate of 1e−31\mathrm{e}{-3} and a batch size of 256, and we train for 5,000 steps. We initialize WVoW^o_V and WVsW^s_V from concept bottleneck learning, use Xavier uniform initialization for WToW^o_T and WTsW^s_T, and set γo=γs=0\gamma_o = \gamma_s = 0.

Overall VQA objective. The overall VQA objective follows (6):

LVQA=λQALQA+λconceptLconcept+λtext(Ltexto+Ltexts)+λalignLalignt,(19)L_{\mathrm{VQA}} = \lambda_{\mathrm{QA}}L_{\mathrm{QA}} + \lambda_{\mathrm{concept}}L_{\mathrm{concept}} + \lambda_{\mathrm{text}}(L^o_{\mathrm{text}} + L^s_{\mathrm{text}}) + \lambda_{\mathrm{align}}L^t_{\mathrm{align}}, \tag*{(19)}

with λQA=0.2\lambda_{\mathrm{QA}} = 0.2, λconcept=0.8\lambda_{\mathrm{concept}} = 0.8, λtext=0.8\lambda_{\mathrm{text}} = 0.8, and λalign=1.0\lambda_{\mathrm{align}} = 1.0, shared across all video question answering datasets.

Interpretable classifier for action recognition training. We employ the GLM-SAGA solver [66] with α=0.99\alpha= 0.99, λ=0.05\lambda= 0.05, a step size of 5,000, and a batch size of 512, following DANCE [26]. Early stopping is applied based on validation loss to avoid overfitting. During this stage, the video encoder fVf_V and the video-side concept layer gVg_V are both frozen; only WARW_{\mathrm{AR}} is updated.

Concept configuration. We describe the dataset-specific concept configurations used across all experiments in Table 8. For each dataset, we list the trajectory length LL, the number of segments per video NsN_s, the resulting number of trajectory concepts MtM_t produced by FINCH [48], the activation and projection cutoffs used to filter object and scene candidates, and the number of object and scene concepts before and after filtering, MoM_o and MsM_s. The trajectory length LL is set to 16 for all datasets except STAR [67], where we increase it to 50 to capture longer-horizon motion required for situated reasoning. We do not apply the activation or projection cutoffs to TOMATO and STAR, since these video question answering datasets are evaluated with a fixed object and scene vocabulary that we keep unfiltered to preserve coverage over diverse question contents. Trajectory concepts have no candidate filtering stage; their count MtM_t is determined entirely by FINCH’s automatic partition selection on the filtered trajectory set T\mathcal{T} (Appendix § 9.§ 4.1).

DatasetLLNsN_sMtM_tActivation cutoffProjection cutoffMoM_o (after / before)MsM_s (after / before)
NTU RGB+D Mutual1651210.30.255 / 8028 / 28
Chiral SSv21631540.30.2404 / 1093286 / 519
UCF-1011631340.30.31103 / 1181578 / 615
KTH1661110.10.126 / 2733 / 33
TOMATO162370––182 / 18294 / 94
STAR5016274––513 / 513142 / 142

Table 8. Dataset-specific concept configurations used by TraCE. We report the trajectory length LL, the number of segments per video NsN_s, the number of trajectory concepts MtM_t after FINCH clustering, the activation and projection cutoffs used to filter object and scene candidates, and the number of object and scene concepts after / before filtering (MoM_o and MsM_s). Cutoffs marked “–” are not applied: TOMATO and STAR are video question answering datasets evaluated with the unfiltered object and scene vocabularies. Trajectory concepts have no candidate filtering stage.

Inference. During inference, we resize each frame so that the shorter side is 224 pixels and apply a center crop of 224×224224 \times224. We do not apply any test-time augmentation and report all results using a single-view setting (1 clip ×\times 1 crop). For action recognition, the predicted class is c^i=arg⁡max⁡c[WARzi]c\hat{c}_i = \arg\max_c [W_{\mathrm{AR}}z_i]_c. For video question answering, the predicted answer is a^i=aijipred\hat{a}_i = a_{i j_i^{\mathrm{pred}}}, where jipred=arg⁡max⁡jsijj_i^{\mathrm{pred}} = \arg\max_j s_{i j}.

Dataset Details

We evaluate TraCE on a suite of action recognition and video question answering datasets. The action recognition datasets test whether trajectory concepts capture dynamic evidence across diverse motion sources, while the video question answering datasets test whether the same typed concept interface supports language-grounded reasoning.

Action Recognition Datasets

NTU RGB+D. NTU RGB+D [52] is a large-scale indoor action recognition dataset captured with multiple camera views and Kinect depth sensors. We use two subsets: NTU RGB+D Mutual and NTU RGB+D Single. NTU RGB+D Mutual consists of 5 mutual-interaction classes where two people interact with each other: pushing other person, hugging other person, giving something to other person, walking towards each other, and walking apart from each other. NTU RGB+D Single consists of 12 single-person classes organized into 6 opposing pairs (e.g., drop vs. pickup, sitting down vs. standing up, wear jacket vs. take off jacket). We use the standard cross-subject split for both subsets. We include NTU RGB+D Mutual as the primary benchmark in Table 2 of the main paper because multi-person interactions directly test whether trajectory concepts can explain motion from multiple interacting agents beyond single-person pose. We use NTU RGB+D Single only in the concept-quality user study (Task 2 in Appendix § 13).

Chiral SSv2. Chiral SSv2 [5, 18] is a subset of Something-Something V2 [18], consisting of 32 classes where paired classes differ by signed motion direction (e.g., pulling [something] left to right vs. right to left). This design makes the dataset a strong test of whether trajectory concepts track motion direction rather than static appearance. We use the official Chiral SSv2 split.

UCF-101. UCF-101 [57] consists of 101 human action classes collected from YouTube. Many classes can be recognized from static scene or object cues, making it a useful test of whether trajectory concepts provide complementary dynamic evidence alongside object and scene concepts. We use the official split 1 for all experiments.

KTH. The KTH Action dataset [49] consists of 6 action classes performed by 25 subjects in 4 different scenarios. All actions involve single-person body motion (walking, jogging, running, boxing, hand waving, and hand clapping), making KTH a controlled test of basic trajectory concept quality on well-separated temporal patterns. We use the standard split.

Video Question Answering Datasets

TOMATO. TOMATO [53] is a temporal reasoning benchmark for multimodal foundation models that requires understanding of temporal order, duration, and frequency. The dataset provides human-centric, real-world, and simulated video scenarios. Since no official training split is provided, we use the human-centric and simulated scenarios as the training set and the real-world scenarios as the test set. Each question is a multiple-choice item with four answer candidates.

STAR. STAR [67] is a situated reasoning benchmark built upon real-world videos of human actions and interactions. The dataset includes four question types: interaction, sequence, prediction, and feasibility, each requiring structured situation comprehension and logical reasoning. We use the official train/val/test split. Each question is a multiple-choice item with four answer candidates.

Action Recognition-to-QA Reformulation. To test whether the same typed concept interface supports both class prediction and language-grounded reasoning, we reformulate three action recognition datasets (NTU RGB+D Mutual, Chiral SSv2, and UCF-101) into a multiple-choice video question answering format. Given a video, we pose the question “What is the action shown in this video?” and use all action class names as answer candidates. The ground-truth answer is the original action label. This reformulation lets us evaluate TraCE’s VQA instantiation on the same videos used for action recognition, enabling a direct comparison of the two task heads under the same typed concept interface.

Generalization Settings

Object-disjoint Split of Chiral SSv2. To evaluate whether trajectory concepts capture motion patterns rather than object appearance shortcuts, we construct an object-disjoint split of Chiral SSv2 using the Something-Else object annotations [38]. We partition the training and test sets so that the object categories in the two sets do not overlap. This forces the model to rely on motion evidence from trajectory concepts rather than recognizing familiar objects. We report results in Table 3 of the main paper.

Cross-domain. We evaluate cross-domain transfer on 65 TOMATO [53] videos labeled with leftward, rightward, upward, or downward motion. We directly apply the trajectory-only TraCE pipeline trained on Chiral SSv2 [5, 18], keeping the video encoder, concept vocabulary, concept layer, and classification head fixed. For each video, we select the top-1 prediction from the original 32 Chiral SSv2 classes. A prediction is correct if it matches a source action class corresponding to the video’s direction. For leftward motion, we accept Pushing [something] from right to left or Pulling [something] from right to left. For rightward motion, we accept Pushing [something] from left to right or Pulling [something] from left to right. For upward motion, we accept Moving [something] up. For downward motion, we accept Moving [something] down. All other predictions count as incorrect. We use the same evaluation criterion for the non-interpretable baseline.

Baseline Details

We provide implementation details for every baseline compared against TraCE in the main paper. For fair comparison, all baselines share the same backbone encoder as TraCE unless explicitly noted.

Saliency-based Explanations

GradCAM. Grad-CAM [50] generates class-discriminative saliency maps by computing the gradient of the predicted class score with respect to intermediate layer activations. We use Grad-CAM solely for visualization purposes: to localize which spatial region in a video activates a given trajectory concept (Appendix § 9.§ 4.1). For the VideoMAE-B/16 backbone [60], we extract Grad-CAM maps from blocks[9].norm; for the InternVideo2 video tower [63], we use blocks[38].norm. Grad-CAM is not used as a baseline explanation method in any quantitative comparison.

SALF-CBM. SALF-CBM [7] is a spatially-aware concept bottleneck model originally designed for image classification, generating visual explanations via spatially localized concept activations. We extend it to video by adding a temporal axis to the spatial concept maps while keeping the rest of the architecture unchanged. We use the same frozen VideoMAE-B/16 backbone and follow the official implementation for all other components. We compare against SALF-CBM in the user study (Figure 5 of the main paper) and in Table 2 for action recognition.

Concept-based Explanations

LF-CBM. Label-Free CBM [40] discovers concept vocabularies from language descriptions and trains a concept bottleneck model without manual concept annotation. We use the official LF-CBM codebase without modification, replacing only the backbone with the same frozen VideoMAE-B/16 features used by TraCE. LF-CBM uses language-derived concepts for all concept types, including motion, which limits its ability to represent fine-grained motion patterns that are difficult to express in words. We compare against LF-CBM in Table 2 for action recognition.

VLG-CBM. VLG-CBM [58] trains concept bottleneck models with vision-language guidance, using CLIP-based concept scoring to improve concept quality. We do not compare against VLG-CBM as a baseline in Table 2. Instead, we adopt the Number of Effective Concepts (NEC) metric proposed by VLG-CBM to evaluate concept quality and report results in the additional experimental results (Appendix § 16.§ 2).

DANCE. DANCE [26] is a concept-based video explanation method that represents motion using human pose sequences extracted by a pretrained pose estimator. We use the same frozen VideoMAEB/16 backbone and follow the official implementation. DANCE defines motion concepts over single-person pose sequences, which makes its motion vocabulary source-incomplete: it cannot directly represent object trajectories and multi-agent interactions. We compare against DANCE in Table 2 for action recognition and in the user study (Figure 5 of the main paper).

Language-based Explanations

GPT-5.4-derived concepts. We construct a language-based motion concept baseline by prompting GPT-5.4 [56] to generate motion-describing concepts for each action class. We then use these language-derived motion concepts in the same concept bottleneck structure as TraCE, replacing trajectory concepts with the generated language motion concepts while keeping the same object and scene concepts, concept layer architecture, and sparse classifier. This baseline tests whether language descriptions can substitute for visual trajectory concepts in capturing fine-grained motion evidence. We compare against this baseline in the user study (Figure 5 of the main paper) and in the concept intervention curves (Figure 8 of the main paper). GPT-5.4 is used only offline to generate the concept vocabulary, not as a video classifier. The unablated accuracies of this baseline at the zero-removal point of the curves in Figure 8 are 97.0%, 77.2%, 88.2%, and 90.2% on NTU RGB+D Mutual, Chiral SSv2, UCF-101, and KTH, respectively, compared with 97.5%, 84.2%, 88.6%, and 92.0% for TraCE; its stronger deletion effect on UCF-101 therefore does not result from a higher initial accuracy.

User Study Setup and Protocol

To assess the interpretability of TraCE, we conducted a user study to evaluate the quality of trajectory concepts and the effectiveness of trajectory-based explanations. The results of this study are summarized in Figure 5 of the main paper, and we provide further details in this section. Specifically, our study aimed to answer two key questions:

  • Does TraCE’s explanation better help users understand the model’s prediction compared to other explanation methods?

  • Do trajectory concepts effectively convey motion patterns in the video?

Recruitment

We collected responses from 50 adult participants recruited through the Prolific platform from English-speaking regions, regardless of gender. Each participant answered 57 questions, which included 2 sanity check questions to ensure engagement and attention. Participants were compensated £9 for the entire study. On average, participants completed the study in about 27 minutes. This study was reviewed and approved as exempt by the Institutional Review Board (IRB) at our institution. We did not collect any personal information from the participants, nor did we disclose any details about the model or research objectives during the study.

Study Design

Task 1: Quality of TraCE’s Explanation

Task 1 aims to address the first key question: Does TraCE’s explanation better help users understand the model’s prediction compared to other explanation methods? To evaluate the interpretability of TraCE, we compare its explanations with those of three baseline methods: DANCE [26], which represents motion using human pose sequences, a language-based CBM using GPT-5.4-derived motion concepts [56], and SALF-CBM [7], a saliency-based concept bottleneck model.

Experimental setting. We conduct pairwise comparisons on motion-focused subsets of NTU RGB+D Mutual [52] and Chiral SSv2 [5, 18]. For each comparison, participants are shown a video and two explanations (Explanation A and Explanation B), where one is from TraCE and the other is from a baseline. The assignment of TraCE to A or B is randomized. Each explanation shows the top-3 contributing concepts overlaid on the input video.

Evaluation protocol. Participants are asked: “Which explanation better helps you understand why the model predicted the action?” Responses are collected on a five-point scale: *A is much better, A is slightly better, About the same, B is slightly better, and B is much better. Each participant answers 18 pairwise comparison questions across the three baselines. We report the distribution of responses in Figure 5 (a) of the main paper.

Task 2: Quality of TraCE’s Trajectory Concept

Task 2 aims to address the second key question: Do trajectory concepts effectively convey motion patterns in the video? This task evaluates the interpretability of individual concepts by asking whether the concept matches the motion observed across its most strongly activating video clips.

Experimental setting. We compare three types of concepts: (i) Trajectory concepts from TraCE, (ii) Pose sequence concepts from DANCE [26], and (iii) Language-derived motion concepts generated by GPT-5.4 [56]. For each concept type, we randomly select one concept and visualize it with its top-4 most strongly activating video clips. This task is conducted on motion-focused subsets of NTU RGB+D Mutual, NTU RGB+D Single, and Chiral SSv2.

Evaluation protocol. Participants are asked: “How well does the given concept match the actions or motions shown in the video above?” Responses are collected on a five-point Likert scale (1: Strongly Disagree to 5: Strongly Agree). We independently evaluate each concept type without directly comparing them to each other. Each participant answers 37 concept evaluation questions across the three concept types. We report the average Likert scores in Figure 5 (b) of the main paper.

Interface and Instructions

We provide clear instructions to participants prior to each task, as illustrated in Figure 11. At the beginning of the study, participants receive an overview instruction page explaining that concepts are intuitive human-interpretable elements—such as body movements, objects, or motions—that help explain how the model perceives and understands the videos. We emphasize that the study does not aim to describe the videos themselves, but to evaluate how well the model interprets and understands the videos through concepts. Before each task, section-specific instructions describe the evaluation criteria and how to answer.

Instruction interfaces shown to participants

Figure 11. Instruction interfaces shown to participants. (a) We provide overview instruction at the beginning of the study. (b) We provide section-specific instructions before each task to guide participants throughout the study.

Task 1 interface. For each pairwise comparison question, we present a video with two explanations (Explanation A and Explanation B) side by side. Each explanation shows the top-3 contributing concepts overlaid on the input video. Participants are asked: “Which explanation better helps you understand why the model predicted the action?” Responses are collected on a five-point scale: A is much better, A is slightly better, About the same, B is slightly better, and B is much better. We show an example of the Task 1 evaluation interface in Figure 12.

Example question and user interface used in Task 1

Figure 12. Example question and user interface used in Task 1. Participants are asked to compare two explanations (A and B) for each sample or class. (a) DANCE [26], (b) Language-based [56], and (c) SALF-CBM [7]. In (a) and (b), we show a single video sample with top-3 sample-level concept contributions. In (c), SALF-CBM provides a heatmap-based spatial explanation alongside concept labels.

Task 2 interface. For each concept evaluation question, we present a single concept alongside its top-4 most strongly activating video clips. Participants are asked: “How well does the given concept match the actions or motions shown in the video above?” Responses are collected on a five-point Likert scale (1: Strongly Disagree to 5: Strongly Agree). Each interface corresponds to one of the three concept types: trajectory concepts (TraCE), pose sequence concepts (DANCE), or language-derived motion concepts (GPT-5.4). We show an example of the Task 2 evaluation interface in Figure 13.

Example question and user interface used in Task 2

Figure 13. Example question and user interface used in Task 2. We show how participants evaluate the interpretability of a single concept using several highly activated video clips. Each interface corresponds to one of the three concept types.

Additional Results

To further validate the user study results in Figure 5 (b) of the main paper, we report mean Likert scores under two conditions in Table 9. In the original condition, participants evaluate a concept alongside its top-4 most strongly activating video clips. In the random condition, we keep the same highly activated video clips but replace the displayed concept with a randomly selected concept from the same method. If a concept type is genuinely interpretable, the original condition should receive substantially higher ratings than the random condition, because the displayed concept should match the motion observed in the clips. TraCE achieves the highest mean score of 4.22 under the original condition, indicating that participants strongly agree that trajectory concepts match the observed motion. When the concept is replaced with a random one, the mean drops sharply to 2.67 (Δ=−1.55\Delta= -1.55), confirming that the high ratings reflect genuine concept–motion alignment rather than a response bias. In contrast, DANCE shows almost no difference between the original (2.52) and random (2.55) conditions (Δ=+0.03\Delta= +0.03). We attribute this to the nature of the evaluation datasets: NTU RGB+D Single contains many simple, repetitive single-person motions (e.g., sitting down, standing up), so different pose sequence concepts tend to look similar to participants, making it difficult to distinguish the correct concept from a random one. The language-based baseline receives the lowest original score (2.27), and its random condition score drops modestly (Δ=−0.41\Delta= -0.41), indicating that participants can sometimes distinguish correct language concepts from random ones, but the overall interpretability remains low. Together, these results support the conclusion in the main paper that trajectory concepts are more interpretable and better aligned with underlying motion patterns than both pose-based and language-based alternatives.

MethodOriginalRandomΔ\Delta
DANCE [26]2.522.550.03
TraCE4.222.67−1.55-1.55
Language-based [56]2.271.86−0.41-0.41

Table 9. Mean Likert scores for Task 2 (concept quality). We report the average score (1: Strongly Disagree to 5: Strongly Agree) over 300 responses per condition. The original condition shows each concept with its top-4 most strongly activating video clips; the random condition shows a randomly selected concept with the same clips. Δ\Delta denotes the change from original to random (random −- original).

Pairwise User Study in the VQA Setting

The user study in Figure 5 evaluates TraCE explanations for action recognition. We extend the evaluation to question-conditioned predictions through a blinded pairwise study with 50 adult participants recruited through Prolific. To isolate explanation quality from answer correctness, we include only examples for which both compared models produce the same correct answer. Each item presents an identical video, question, answer candidates, and model answer together with two concept-based explanations. Both explanations contain the same number of high-contribution concepts and use the same presentation format; the method identities are hidden. Participants answer, “Which explanation better helps you understand why the model chose its answer?”

Each participant evaluates five comparisons against the MM-CBM [54] variant with language-derived motion concepts and five against the MM-CBM variant with pose-sequence concepts, yielding 250 participant–item ratings per comparison. The examples span the native TOMATO and STAR benchmarks and multiple-choice VQA reformulations of NTU RGB+D Mutual and UCF-101.

As shown in Table 10, participants prefer TraCE in 70.4% of comparisons against explanations based on language-derived motion concepts and 85.6% against explanations based on pose-sequence concepts. Both models are correct on every evaluated item and display the same answer, so the preferences measure the perceived helpfulness of the explanations rather than differences in prediction correctness. Because the study contains both native and multiple-choice reformulated examples, we describe it as a VQA-setting study; native VQA faithfulness is evaluated separately in Appendix § 16.7.

ComparisonMuch betterSlightly betterAbout the sameSlightly worseMuch worsePrefer TraCE
Language-derived motion concepts123 (49.2%)53 (21.2%)25 (10.0%)34 (13.6%)15 (6.0%)70.4%
Pose-sequence concepts167 (66.8%)47 (18.8%)23 (9.2%)10 (4.0%)3 (1.2%)85.6%

Table 10. Pairwise explanation preference in the VQA setting. Each comparison contains 250 participant–item ratings from examples for which both models produce the same correct answer. The five response columns are ordered from TraCE being much better to the baseline being much better; the final column combines the first two responses.

Additional Experimental Results

For fair comparison, all experiments in this section use the same backbone, VideoMAE-B/16 [60] for action recognition and InternVideo2s2-1B [63] for video question answering unless otherwise noted. Unless specified otherwise, all ablation studies are conducted on Chiral SSv2 [5, 18] dataset.

Trajectory Concept Generation Ablation

Effect of number of concepts. We ablate the effect of the number of trajectory concepts MtM_t by varying the FINCH partition level. A higher partition level yields fewer, coarser clusters, while a lower level produces more, finer-grained clusters. As shown in Table 11a, the default configurations listed in Table 8 achieve accuracy within 3 percentage points of the finest partition across all datasets, while using substantially fewer concepts. On NTU RGB+D Mutual, accuracy is nearly invariant to the number of concepts (97.5% at both 681 and 121 concepts), reflecting the near-saturated nature of this benchmark. On KTH, the default partition (Mt=111M_t = 111) slightly outperforms the finest partition (Mt=589M_t = 589, 92.0% vs. 91.0%), suggesting that an excessive number of fine-grained concepts can introduce noise for datasets with simple, well-separated motion patterns. In contrast, Chiral SSv2 shows the largest sensitivity to concept count (86.8% → 80.3% from 870 to 29 concepts), which is expected because its paired directional classes require fine-grained trajectory discrimination.

(a) Number of trajectory concepts.
Chiral SSv2: # ConceptsChiral SSv2: Acc.NTU RGB+D Mutual: # ConceptsNTU RGB+D Mutual: Acc.UCF-101: # ConceptsUCF-101: Acc.KTH: # ConceptsKTH: Acc.
87086.868197.5412489.558991.0
15484.212197.575988.611192.0
2980.32596.913488.42190.4
(b) NsN_s.(c) LL.(d) Clustering.(e) Labeling.(f) Foreground mask.
NsN_sAcc.LLAcc.MethodAcc.MethodAcc.
183.2886.9FINCH [48]84.2Binary82.4
384.21684.2K-Means [37]84.6Soft84.2
582.63287.3Max normalized83.3
MaskingAcc.
On (TraCE)84.2
Off80.3

Table 11. Trajectory concept generation ablations. (a) is reported on all action recognition datasets; (b)–(f) are reported on Chiral SSv2. Default settings are TraCE’s configuration in Table 8.

Effect of the number of segments. As shown in Table 11b, accuracy is robust to the number of temporal segments NSN_S, with NS=3N_S = 3 achieving the best result (84.2%). Using a single segment (NS=1N_S = 1) slightly reduces accuracy (83.2%) because the entire video is summarized by one clip, losing temporal coverage. Increasing to NS=5N_S = 5 also decreases accuracy to 82.6%. We set NSN_S per dataset by dividing the average number of frames in the training set by the trajectory length LL, which provides a natural balance between temporal coverage and clip quality.

Effect of trajectory length LL. As shown in Table 11c, longer trajectories (L=32L = 32) achieve the highest accuracy (87.3%), while the default length (L=16L = 16) achieves 84.2% and shorter trajectories (L=8L = 8) achieve 86.9%. However, accuracy alone does not reflect concept interpretability. We visualize trajectory concepts discovered at each length in Figure 14. Short trajectories (L=8L = 8) capture only brief, fragmented motion that lacks sufficient temporal context to convey a meaningful motion pattern. Long trajectories (L=32L = 32) tend to encode the entire action class rather than reusable motion primitives shared across classes—analogous to classifying an image by the object itself rather than by its distinguishing attributes. We therefore adopt L=16L = 16 as the default, which captures representative motion fragments that are both discriminative and generalizable as an interpretable trajectory vocabulary.

Trajectory concepts: (a) $L=8$, (b) $L=16$, (c) $L=32$

Figure 14. Trajectory concepts discovered with different trajectory lengths. Short trajectories (L=8L = 8) capture only local and fragmented motion patterns with limited temporal structure. Long trajectories (L=32L = 32) often encode near-complete action dynamics, reducing concept reusability across classes. In contrast, the default trajectory length (L=16L = 16) captures representative motion fragments that remain both interpretable and reusable across different videos. These visualizations illustrate the trade-off between local motion specificity and action-level over-specialization when choosing the trajectory length.

Effect of clustering method. We compare FINCH [48] with K-Means [37] as the trajectory clustering algorithm. As shown in Table 11d, the two methods achieve comparable accuracy (84.2% vs. 84.6%), confirming that trajectory concept quality is largely agnostic to the choice of clustering algorithm. We adopt FINCH as the default because it automatically determines the number of clusters without requiring a predefined kk, making the pipeline easier to apply across datasets without per-dataset hyperparameter tuning.

Soft vs. hard concept labels. We compare three strategies for converting trajectory cluster assignments into video-level concept labels: (i) Binary, which assigns 1 if any trajectory in the video belongs to the cluster and 0 otherwise; (ii) Soft (default), which uses the normalized frequency cit=hi/∥hi∥1c_i^t = \mathbf{h}_i / \lVert\mathbf{h}_i \rVert_1; and (iii) Max normalized, which divides each count by the maximum count across all concepts for the video. As shown in Table 11e, the soft labeling strategy achieves the best accuracy (84.2%), outperforming both binary (82.4%) and max normalized (83.3%) alternatives. Binary labels discard frequency information and treat rare and dominant motion patterns equally, which reduces the model’s ability to distinguish fine-grained directional cues.

Effect of foreground masking. The trajectory filter in Appendix § 9.§ 4.1 retains a trajectory only when its seed pixel lies inside the estimated foreground saliency mask. On Chiral Ssv2, foreground masking removes approximately 81% of the raw trajectories. As shown in Table 11f, foreground masking improves validation accuracy from 80.3% to 84.2% when we keep the remaining pipeline unchanged, supporting its use to suppress background-seeded trajectories.

Analysis

Effect of Video-LLM. We compare five language models for generating object and scene concept candidates: GPT-5.4 [56], LLaVA-NeXT-Video [72], Video-LLaVA [32], VideoLLaMA2 [13], and Qwen3-VL [6]. As shown in Table 12, Video-LLaVA and VideoLLaMA2 achieve the highest accuracy (77.7% and 77.3%), while GPT-5.4 produces the fewest concepts (152) and the lowest accuracy (61.5%). Qwen3-VL achieves competitive accuracy (74.6%) despite generating the most candidates (1,683). However, accuracy alone does not capture concept quality. We qualitatively observe that many models produce concepts that violate the typed separation between object and scene groups: for example, scene-like terms appear in the object list, or action verbs are included among object candidates. We show representative examples of such violations in Figure 15. We adopt Qwen3-VL as the default because it produces the cleanest separation between object and scene concepts when guided by our structured prompts (Figure 10), even though other models may achieve slightly higher accuracy with noisier concept vocabularies.

Figure 15. Qualitative comparison of video-language models for text concept generation. We visualize representative object and scene concept candidates generated by different video-language models. (a) LLaVA-NeXT-Video [72] produces relatively clean scene-centric concepts. (b) Video-LLaVA [32] frequently mixes appearance descriptions and human-centric phrases into object concepts. (c) VideoLLaMA2 [13] sometimes generates action phrases or interaction descriptions instead of static scene concepts. These examples highlight that higher downstream accuracy does not necessarily imply cleaner object–scene concept separation.

Method# ConceptsAcc.
GPT-5.4 [56]15261.5
LLaVA-NeXT-Video [72]39371.8
Video-LLaVA [32]1,02677.7
VideoLLaMA2 [13]1,47877.3
Qwen3-VL [6]1,68374.6

Table 12. Effect of different object and scene concept discovery method.

Vision-language model ablation. We ablate the choice of the vision–language encoder used to compute the object and scene pseudo-labels (Appendix § 9.§ 4.2), while keeping the same concept candidates and backbone. As shown in Table 13, replacing InternVid [62] with InternVideo2 [63] as the labeling encoder improves accuracy from 83.4% to 84.2%. We attribute this improvement to the stronger video–text alignment provided by InternVideo2’s Stage-2 contrastive pretraining, which produces more accurate soft pseudo-labels for each concept.

Video encoderVLM encoderAcc.
VideoMAE [60]+ InternVid [62]83.4
VideoMAE [60]+ InternVideo2 [63]84.2

Table 13. Evaluation of VLM encoder for object and scene concept labeling

Disentanglement analysis. We analyze whether the three concept groups (trajectory, object, scene) use disentangled subspaces of the backbone feature xix_i. For this additional analysis, we ablate two orthogonality mechanisms: (1) SVD null-space projection, which removes previously used subspace dimensions from the backbone feature before training the next concept group, and (2) Wc_c row-orthogonality loss with weight λ\lambda, which penalizes cosine similarity between weight rows of different groups. We measure disentanglement via top-KK backbone-dimension overlap: for each group gg, we compute a per-dimension importance score imp⁡g[j]=mean⁡n∣xn[j]∣⋅mean⁡c∣Wcg[c,j]∣\operatorname{imp}_{g}[j] = \operatorname{mean}_{n}|x_n[j]| \cdot\operatorname{mean}_{c}|W^{g}_{c}[c,j]| over validation samples, select the top-KK dimensions, and count shared indices across group pairs. The random-chance baseline for K=50K=50 is K2/d=3.26K^2/d = 3.26 shared dimensions.

Group ordering sensitivity. Since concept groups are trained sequentially, we fix SVD + Ortho (λ=1.0\lambda= 1.0) and train all six permutations of {trajectory (T), object (O), scene (S)}. As shown in Table 15, the ordering induces a 1.2%p accuracy spread and a 1.7× variation in maximum pairwise top-50 overlap. Trajectory-first orders (TSO, TOS) achieve the highest average accuracy (86.85%), because the trajectory branch uses a structurally distinct BCE head and naturally occupies a separate subspace, imposing no representational cost on subsequent groups. Object-first orders (OTS, OST) yield the lowest accuracy (85.94% average), as the object group has the largest concept set (404 concepts) and consumes a large subspace first. SOT achieves the lowest maximum overlap (13) and lowest sum overlap (29), while TSO achieves the best accuracy (86.92%) with a moderate maximum overlap (20). We use TOS for the component ablation in Table 14.

VariantAcc. (%)traj ∩\cap objtraj ∩\cap sceneobj ∩\cap scene
Baseline84.20121034
SVD only (α=1.0\alpha=1.0)84.55101919
Ortho only (λ=1.0\lambda=1.0)86.928921
SVD + Ortho (λ=1.0\lambda=1.0)86.788922
SVD + Ortho (λ=2.0\lambda=2.0)87.838826

Table 14. Component ablation for concept group disentanglement. We report validation accuracy (%) and top-50 backbone-dimension overlap between group pairs. Random-chance overlap is 3.26. The object–scene pair (object ∩\cap scene) is the structurally hardest target because both groups are text-based visual concepts.

OrderAcc. (%)traj ∩\cap objtraj ∩\cap sceneobj ∩\cap scenesum
TSO86.921092039
TOS86.78892239
STO86.6412161240
SOT86.50137929
OTS86.151481234
OST85.7312131136

Table 15. Effect of concept group training order on accuracy and disentanglement. We report validation accuracy (%) and top-50 backbone-dimension overlap between group pairs for all six permutations of {trajectory (T), object (O), scene (S)}. SVD + Ortho (λ=1.0\lambda= 1.0) is fixed.

Component ablation. We compare four variants against the baseline (no orthogonality) in Table 14. The Wc_c orthogonality loss is the dominant driver of both accuracy and disentanglement. SVD alone contributes only +0.35%p in accuracy and becomes redundant at inference when the orthogonality loss is active: for all ortho-containing variants, removing the SVD projection at inference causes no accuracy change (Δ=0\Delta= 0). Combining SVD with orthogonality at λ=2.0\lambda= 2.0 achieves the best accuracy (87.83%). All variants reduce object–scene overlap relative to the baseline (34 →\rightarrow 19–26), confirming that orthogonality mechanisms successfully encourage the two text-based groups to use more distinct backbone dimensions. Increasing λ\lambda from 1.0 to 2.0 improves accuracy (+1.0%p) but slightly increases top-50 overlap (22 →\rightarrow 26), because row-orthogonality can be satisfied by using the same dimensions with opposite sign or scale.

Concept combination. We evaluate the contribution of each concept type by training TraCE with different subsets of the three concept groups. As shown in Table 16, using all three groups (trajectory + object + scene) achieves the best accuracy (84.2%). Trajectory concepts alone already capture the majority of the performance (80.7%), and adding object or scene concepts yields consistent gains (+1.7%p and +0.9%p, respectively). In contrast, removing trajectory concepts entirely (object + scene only) causes a substantial drop to 74.6%, a −9.6%p gap relative to the full model. This confirms that trajectory concepts are the most informative group, while object and scene concepts serve as useful but secondary complements.

Concept
TrajectoryObjectSceneAcc.
✓\checkmark×\times×\times80.7
✓\checkmark✓\checkmark×\times82.4
✓\checkmark×\times✓\checkmark81.6
×\times✓\checkmark✓\checkmark74.6
✓\checkmark✓\checkmark✓\checkmark84.2

Table 16. Effects of individual concept types on TraCE performance.

Concept sparsity. We evaluate whether TraCE maintains predictive performance with a compact set of concepts using the Number of Effective Concepts (NEC) metric proposed by VLG-CBM [58].

We vary the regularization strength λ\lambda of the GLM-SAGA solver to control the average number of non-zero weights per class in WARW_{\mathrm{AR}}. As shown in Table 17, at NEC=25\mathrm{NEC} = 25, accuracy reaches 83.8%, within 0.4 percentage points of the default model (84.2%). Even at NEC=5\mathrm{NEC} = 5, where each class uses only five concepts on average, accuracy remains at 67.7%.

NECSparsity (%)Acc. (%)
50.6367.7
101.2177.2
151.7981.3
202.4182.9
253.0383.8
303.5784.0

Table 17. Accuracy under controlled Number of Effective Concepts (NEC) [58]. NEC limits the average number of non-zero classifier weights per class. Lower NEC yields sparser, more interpretable predictions at the cost of accuracy. Sparsity denotes the fraction of non-zero entries in WARW_{\mathrm{AR}}.

Explanation Analysis

Motion perturbation analysis. We complement the perturbation analysis in § 5 with direction-specific results on the same directional-class subset of Chiral SSv2. We evaluate classes specifying left, right, up, or down motion under temporal reversal and 90° counter-clockwise rotation. As shown in Figure 7, TraCE adapts its top trajectory concepts to the induced motion transformation. For the original input in Figure 7 (a), the top concepts align with the left-to-right motion and support the prediction. When the video is temporally reversed in Figure 7 (b), the top concepts switch to the opposite direction and the prediction changes accordingly. Under a 90° rotation in Figure 7 (c), the activated concepts rotate consistently with the transformed motion direction. For each test sample, we apply the perturbation and check whether the predicted motion group follows the expected directional transformation: temporal reversal should swap LEFT ↔ RIGHT and UP ↔ DOWN, while 90° rotation should map LEFT → DOWN, RIGHT → UP, UP → LEFT, and DOWN → RIGHT. As shown in Table 18, TraCE achieves an overall reverse consistency of 79.2% and a rotation consistency of 63.1%. Reverse consistency is higher for LEFT (90.4%), RIGHT (84.0%), and DOWN (94.0%) than for UP (55.8%). Rotation consistency is also higher for LEFT (83.3%) and RIGHT (90.8%) than for UP (44.2%) and DOWN (45.2%). The prediction-level results support motion sensitivity and complement the concept-level directional analysis in Appendix § 16.§ 6.

Original groupReverse consistency (%)Rotation consistency (%)
Left90.483.3
Right84.090.8
Up55.844.2
Down94.045.2
Overall79.263.1

Table 18. Prediction consistency under perturbation on Chiral SSv2. We report the percentage of samples whose predicted motion group follows the expected transformation after perturbation. Expected transformations are: LEFT ↔\leftrightarrow RIGHT and UP ↔\leftrightarrow DOWN under temporal reverse, and LEFT →\to DOWN, RIGHT →\to UP, UP →\to LEFT, DOWN →\to RIGHT under 90∘90^\circ counter-clockwise rotation.

Faithfulness analysis. We extend the faithfulness evaluation in § 5 of the main paper by reporting per-dataset concept removal results in Table 19. For each sample, we deactivate either the top-KK trajectory concepts ranked by contribution (Top-KK) or KK randomly selected trajectory concepts (Random-KK), and measure Flip%, the percentage of samples whose predicted class changes. A faithful model should show high Flip% under Top-KK removal and low Flip% under Random-KK removal, indicating that the contribution ranking aligns with the model’s actual decision process. Across all datasets, Top-KK removal causes substantially more prediction flips than Random-KK removal at both K=5K = 5 and K=10K = 10. Chiral SSv2 shows the largest Top-KK effect (48.7% at K=5K = 5, 70.1% at K=10K = 10), consistent with the strong dependence of directional motion classes on a small number of discriminative trajectory concepts. NTU RGB+D Mutual shows the smallest effect (7.0% at K=5K = 5), reflecting the near-saturated nature of this benchmark where multiple trajectory concepts carry redundant evidence. UCF-101 shows a moderate Top-KK effect (15.9% at K=5K = 5), consistent with the observation that many UCF-101 classes can be supported by static scene or object cues alongside trajectory evidence. In all cases, Random-KK removal causes minimal prediction changes (≤5.4%\le5.4\%), confirming that random trajectory concepts do not carry decision-critical information.

DatasetTop-K ↑Random-K ↓Top-K ↑Random-K ↓
K=5K=10
Chiral SSv248.73.770.15.4
KTH35.12.146.02.8
UCF-10115.90.218.40.3
NTU RGB+D Mutual7.00.413.80.3
Average26.71.637.12.2

Table 19. Faithfulness of trajectory concepts via concept removal. We report Flip%, the percentage of samples whose prediction changes after deactivating trajectory concepts. We compare removing the top-K contributing concepts with removing K randomly selected concepts. Higher Top-K removal and lower Random-K removal indicate that the estimated concept importance aligns with the model’s actual decision process.

Sample-Level Motion-Source Grounding

The qualitative examples in Figure 4 show that trajectory concepts explain interactions and object dynamics, while Table 3 shows that the trajectory-only variant retains predictive motion cues under object category shift. We quantify whether high-contribution trajectory concepts are spatially grounded in the hands and manipulated objects in each video.

We use the same object-disjoint Chiral SSv2 split constructed from Something-Else annotations [38]. As described in Appendix § 9.§ 4, we discover trajectory concepts exclusively from filtered training trajectories and retain the medoid trajectory of each training cluster. For each validation video, we assign every cached trajectory to its nearest fixed medoid in the same flattened displacement space used for clustering. The cached validation assignments are used to compute the validation loss and select the checkpoint, but validation trajectories do not participate in vocabulary construction or gradient updates. We rank trajectory concepts by their contribution to the predicted class, select the trajectories assigned to the Top-10, Random-10, or Bottom-10 concepts, and recover their seed pixels in the input video. For Random-10, we sample one set of ten concepts uniformly without replacement for each validation video and aggregate coverage over the validation set. Because every assigned trajectory retains its seed pixel, this analysis directly recovers the spatial support of each selected concept without Grad-CAM.

Something-Else provides frame-level bounding boxes for hands and manipulated objects. A selected trajectory covers a box only when its seed pixel lies inside the box in the first frame of the corresponding LL-frame clip. Hand and Object measure the percentages of annotated hand and object boxes covered by at least one selected trajectory. Both measures the percentage of videos in which the selected trajectories cover at least one annotated hand and one manipulated object. We evaluate DANCE [26] analogously using the spatial support of its selected pose-sequence concepts.

As shown in Table 20, source coverage increases with concept contribution for TraCE. The Top-10 trajectory concepts cover 64.5% of annotated hand boxes and 63.5% of manipulated-object boxes, with both sources covered in 58.1% of videos. The joint coverage decreases to 25.1% for Random-10 concepts and 19.0% for Bottom-10 concepts. In contrast, DANCE’s Top-10 pose-sequence concepts cover 42.3% of annotated hand boxes but only 7.6% of manipulated-object boxes and both sources in 4.7% of videos. These results do not imply that each trajectory concept has a fixed entity label. Rather, they show that high-contribution trajectory concepts can be grounded at the sample level to motion from both people and manipulated objects, supporting their role as a source-flexible motion vocabulary.

SelectionDANCE [26] HandDANCE [26] ObjectDANCE [26] BothTraCE HandTraCE ObjectTraCE Both
Bottom-100.00.00.022.421.619.0
Random-104.10.70.529.128.525.1
Top-1042.37.64.764.563.558.1

Table 20. Sample-level motion-source grounding on object-disjoint Chiral SSv2. Top and Bottom select K=10K = 10 concepts according to their contribution to the predicted class; Random samples one set of K=10K = 10 concepts uniformly without replacement for each video. Hand and Object report the percentages of annotated boxes covered by the seed pixel of a selected trajectory in the first frame of its LL-frame clip; Both reports the percentage of videos covering at least one hand and one manipulated object.

Directional Semantic Validation

As shown in Table 4, TraCE’s predictions follow the expected directional transformations under temporal reversal and rotation. Prediction consistency alone, however, does not show whether the selected trajectory concepts carry the corresponding motion semantics. Because trajectory concepts are supervised by a soft distribution over multiple training-derived clusters, single-label precision or recall would discard the target structure. We instead evaluate signed motion direction on the Chiral SSv2 direction subset (n=572n = 572) [5, 18], whose actions specify left, right, up, or down motion. We assign each trajectory concept a fixed direction label from the net displacement of its medoid trajectory before evaluating the predicted concept activations. Across held-out videos, the highest-activated trajectory concept matches the ground-truth direction in 67.5% of cases, compared with the 25% four-way random reference.

For each original, temporally reversed, and 90°-rotated video, we measure whether the top-5 activated trajectory concepts follow the expected direction. Let Corig5C_{\mathrm{orig}}^{5} and Cpert5C_{\mathrm{pert}}^{5} denote the sets of top-5 concept indices for the original and perturbed videos. We measure concept replacement as 1−∣Corig5∩Cpert5∣/51 - |C_{\mathrm{orig}}^{5} \cap C_{\mathrm{pert}}^{5}|/5. For each transformed video, prediction correctness is determined using the expected transformed class.

As shown in Table 21, correctly predicted transformed videos replace 96.3% of the original top-5 concept identities under temporal reversal and 96.1% under rotation. The selected concepts achieve 80.1% direction agreement after reversal and 78.4% after rotation. Incorrect transformed predictions provide a control: they also exhibit substantial concept replacement, but their direction agreement decreases to 32.8% under reversal and 35.1% under rotation. Concept replacement alone therefore does not establish semantic consistency. Correct transformed predictions instead combine near-complete concept replacement with concepts whose fixed training-derived directions match the transformed motion at more than three times the 25% random reference. This analysis validates signed motion direction, where objective physical labels are available, rather than providing exhaustive semantic annotation or calibration for every fine-grained trajectory cluster.

PredictionInputDirection agreement (%)Replacement (%)
CorrectOriginal76.30.0
CorrectTemporal reversal80.196.3
Correct90∘90^\circ rotation78.496.1
IncorrectOriginal45.40.0
IncorrectTemporal reversal32.881.0
Incorrect90∘90^\circ rotation35.184.6
Random reference–25.096.8

Table 21. Directional semantics under controlled perturbations. Direction agreement is the fraction of top-5 activated trajectory concepts whose fixed direction labels, derived from their medoid trajectories, match the expected direction. For the original and perturbed top-5 concept-index sets Corig5C_{\mathrm{orig}}^5 and Cpert5C_{\mathrm{pert}}^5, replacement is 1−∣Corig5∩Cpert5∣/51-\lvert C_{\mathrm{orig}}^5 \cap C_{\mathrm{pert}}^5\rvert/5. Prediction correctness after perturbation is evaluated against the expected transformed class; the final row reports random-reference values.

Static-context stability. Using the same model and subset, we also test whether temporal reversal and rotation alter the semantics of the selected object and scene concepts. For each concept group, we average the embedding vectors of the top-kk activated concepts (k∈{1,3,5,10}k \in\{1,3,5,10\}) before and after perturbation and compute the cosine similarity between the two aggregate embeddings. The similarity ranges from 0.84 to 0.98 under temporal reversal and from 0.81 to 0.97 under rotation, showing that motion perturbations alter the selected directional content while largely preserving the aggregate semantics of static context.

Ground-Truth-Targeted Concept Interventions

The contribution-ranked removal results in Figure 8 and Table 19 show that TraCE relies on high-contribution trajectory concepts. However, contribution-ranked removal does not determine whether the influential concepts align with the ground-truth motion. We therefore perform direction-targeted interventions on the Chiral SSv2 direction subset (n=572n=572) and the TOMATO direction subset (n=28n=28), using the fixed directional concept labels defined in Appendix § 16.§ 6. For each sample, we remove concepts from the ground-truth-direction, opposite-direction, or random non-ground-truth pool. The random pool excludes concepts assigned to the ground-truth direction.

We consider two complementary interventions. In count-matched removal, for sample ii, let GiG_i denote the set of ground-truth-direction concepts and OiO_i the set of opposite-direction concepts. We set ki=min⁡(∣Gi∣,∣Oi∣)k_i=\min(\lvert G_i\rvert,\lvert O_i\rvert) and remove exactly kik_i concepts from the ground-truth-direction, opposite-direction, or non-ground-truth pool. The non-ground-truth pool contains only opposite or orthogonal directions. When a pool contains more than kik_i concepts, we randomly subsample kik_i concepts and average the result over 100 draws. Thus, all three conditions remove the same number of concepts for each sample, and the random control never includes ground-truth-direction concepts. When ∣Gi∣>ki\lvert G_i\rvert> k_i, some ground-truth-direction concepts remain active, making this a conservative test that controls intervention size without eliminating all same-direction redundancy.

Full-group removal instead deactivates every concept assigned to the ground-truth or opposite direction, testing the effect of removing potentially redundant concepts with the same directional semantics. Its random non-ground-truth control removes ∣Gi∣\lvert G_i\rvert concepts from the non-ground-truth pool and is averaged over 100 draws. The reported standard deviations therefore measure variability across concept-subsampling draws on the fixed evaluation set, not variability across training runs.

We report post-intervention accuracy and the correct-to-incorrect flip rate, the percentage of initially correct predictions that become incorrect after intervention.

As shown in Table 22, ground-truth-direction removal produces the largest degradation under both protocols. On Chiral SSv2, count-matched removal reduces accuracy by 24.2 points and changes 30.1% of originally correct predictions to errors, whereas opposite-direction and random non-ground-truth removal do not reduce accuracy and yield flip rates below 4%. Full-group removal produces the same ordering, reducing accuracy by 27.4 points and yielding a 34.1% correct-to-incorrect flip rate. On TOMATO, ground-truth-direction removal reduces accuracy by 22.5 points under count matching and 21.4 points under full-group removal, with corresponding flip rates of 70.5% and 69.2%. Only 13 TOMATO samples are predicted correctly before intervention, so these VQA flip rates should be interpreted descriptively. A positive net accuracy change under a control can coexist with a nonzero correct-to-incorrect flip rate because overall accuracy also includes initially incorrect predictions that become correct. Together with the directional analysis in Appendix § 16.§ 6, these results show that trajectory concepts aligned with the ground-truth motion contribute disproportionately to both action recognition and VQA predictions.

ProtocolTaskRemoved conceptsAcc. (%)Δ\DeltaAcc.Flip rate (%)
Count-matchedARNone (TraCE)81.1––
Count-matchedARGT direction56.9±0.1−24.230.1±0.1
Count-matchedAROpposite direction82.2±0.1+1.13.6±0.1
Count-matchedARRandom non-GT82.1±0.2+1.03.7±0.2
Count-matchedVQANone (TraCE)46.4––
Count-matchedVQAGT direction23.9±0.4−22.570.5±0.5
Count-matchedVQAOpposite direction54.0±0.2+7.630.8±0.0
Count-matchedVQARandom non-GT45.4±1.3−1.021.3±2.0
Full-groupARNone (TraCE)81.1––
Full-groupARGT direction53.7−27.434.1
Full-groupAROpposite direction81.3+0.24.5
Full-groupARRandom non-GT82.8±0.2+1.63.8±0.1
Full-groupVQANone (TraCE)46.4––
Full-groupVQAGT direction25.0−21.469.2
Full-groupVQAOpposite direction53.6+7.230.8
Full-groupVQARandom non-GT42.2±1.4−4.225.4±2.1

Table 22. Ground-truth-targeted trajectory concept interventions. We report post-intervention accuracy, its change from the unablated model, and the flip rate, i.e., the percentage of initially correct predictions that become incorrect. Count-matched results and the random non-ground-truth full-group results report mean ± standard deviation over 100 concept-subsampling draws on the fixed evaluation set.

Video Question Answering Analysis

The VQA results in Table 2 show that TraCE preserves competitive performance when predictions are routed through the typed trajectory/object/scene interface. We further evaluate whether predictions on the native TOMATO and STAR benchmarks depend on trajectory concepts that contribute to the predicted answer. For each sample, we rank trajectory concepts by their contribution to the concept-compatibility score for the predicted answer and set the selected trajectory concept activations to zero. We remove concepts in descending (Top), random, or ascending (Bottom) order of contribution.

As shown in Table 23, removing only the top-5% trajectory concepts reduces accuracy by 11.9 points on TOMATO and 40.5 points on STAR. The corresponding random removals reduce accuracy by 4.3 and 1.3 points, while Bottom-ranked removal has no effect through an 80% removal ratio. These results show that native VQA predictions depend on a sparse set of trajectory concepts with high contribution to the predicted answer rather than on arbitrary bottleneck dimensions. At 100% trajectory concept removal, object and scene concepts remain active, accounting for the nonzero residual accuracy.

The interpretable baselines evaluated in the main paper are designed for action recognition and do not directly define question-conditioned answer scoring. We therefore construct two controlled variants that combine the MM-CBM [54] concept-space matching design with language-derived motion concepts from LF-CBM [40] or pose-sequence concepts from DANCE [26]. Because neither pose-sequence nor trajectory concepts have language names, the pose-sequence variant learns text-side relevance using the same answer-supervised alignment as TraCE. We additionally adapt SpLiCE [9] to question-conditioned answer scoring. All methods use the same frozen video and text backbones, data splits, answer candidates, and training budget.

As shown in Table 24, TraCE achieves the highest mean accuracy across the five benchmarks among the concept-based methods (74.4%), compared with 73.5% for language-derived motion concepts and 72.8% for pose-sequence concepts. On the native VQA benchmarks, TraCE obtains the highest TOMATO accuracy (37.0%) and remains competitive on STAR (55.4%). We do not interpret the small numerical differences as significant improvements. Together with the deletion results, these comparisons show that trajectory concepts support native VQA explanations while maintaining competitive predictive performance under the same concept-space matching design.

MethodNTU MutualChiral SSv2UCF-101TOMATOSTARAvg.
Non-interpretable baseline97.4±0.197.4 \pm0.182.6±0.282.6 \pm0.299.8±0.099.8 \pm0.035.9±2.235.9 \pm2.254.5±0.454.5 \pm0.474.0
Language-derived motion concepts97.6±0.197.6 \pm0.182.0±0.482.0 \pm0.499.7±0.199.7 \pm0.131.7±3.631.7 \pm3.656.5±0.556.5 \pm0.573.5
Pose-sequence concepts97.6±0.197.6 \pm0.182.3±0.682.3 \pm0.699.7±0.199.7 \pm0.128.7±7.028.7 \pm7.055.9±0.755.9 \pm0.772.8
SpLiCE [9] adaptation53.057.386.722.449.253.7
TraCE97.1±0.397.1 \pm0.382.6±0.682.6 \pm0.699.7±0.199.7 \pm0.137.0±1.737.0 \pm1.755.4±1.255.4 \pm1.274.4

Table 24. Comparison with VQA-capable concept methods. We report answer accuracy (%) as mean ± sample standard deviation over five stochastic training seeds, except for the single-run SpLiCE adaptation. The controlled variants combine the MM-CBM [54] concept-space matching design with language-derived motion concepts from LF-CBM [40] or pose-sequence concepts from DANCE [26]. The first three datasets are multiple-choice VQA reformulations of action-recognition datasets; TOMATO and STAR are native VQA benchmarks.

Concept combinations on STAR. We evaluate combinations of trajectory, object, and scene concepts on STAR [67]. We report single-run results for both STAR ablations in Table 25 and Table 26. As shown in Table 25, trajectory concepts alone achieve 54.8% accuracy, compared with 54.4% for object concepts and 52.9% for scene concepts. Combining all three concept types yields 53.4%.

ConceptAcc.
TrajectoryObjectScene
✓✗✗54.8
✗✓✗54.4
✗✗✓52.9
✓✓✗54.5
✓✗✓52.9
✗✓✓53.4
✓✓✓53.4

Table 25. Effects of individual concept types on TraCE performance.

MethodAcc.
InfoNCE53.4
Cosine similarity52.0
KL divergence52.6
ℓ1\ell_1 regression52.1
ℓ2\ell_2 regression52.2

Table 26. Effect of text-side trajectory relevance learning objectives.

Adding static context does not improve the reported accuracy over trajectory concepts alone in this evaluation.

Text-side trajectory relevance learning on STAR. We compare loss functions for learning question-conditioned trajectory relevance on STAR [67]. As shown in Table 26, InfoNCE achieves the highest accuracy (53.4%), while the alternative objectives yield 52.0–52.6%. The comparison supports our choice of InfoNCE for answer-supervised trajectory alignment.

Limitations

While TraCE provides effective and interpretable explanations by disentangling trajectory, object, and scene concepts, it has several limitations.

First, trajectory concepts depend on the quality of the underlying optical flow estimator. Inaccurate flow predictions in challenging conditions—such as occlusion, fast motion, or low-texture regions—can produce noisy or misleading trajectories, which in turn degrade the quality of the discovered trajectory concepts. Moreover, optical-flow trajectories can mix foreground entity motion with global camera motion. In a diagnostic on 65 Chiral SSv2 videos from six camera motion classes, TraCE achieves 98.5% accuracy, showing that trajectory concepts can represent task-relevant camera motion; however, this does not establish invariance to camera motion when it acts as a nuisance, and foreground masking (Table 11f) is not equivalent to camera ego-motion compensation.

Second, due to the absence of explicit object and scene annotations in most video datasets, TraCE relies on a Video-LLM to generate concept candidates and on a pretrained vision–language encoder to produce soft pseudo-labels. Consequently, the resulting object and scene representations may inherit limitations of the underlying models, such as language priors and visual biases. Third, our concept visualizations rely on Grad-CAM [50] to localize where a trajectory concept is expressed, so the highlighted regions are approximate. Fourth, trajectory concepts are extracted from short clips (e.g., L=16L = 16 frames), which limits their ability to represent long-range temporal structure.

To mitigate sensitivity to optical flow quality and nuisance camera motion, future work could explore more robust flow estimators, video stabilization, global-motion subtraction, joint motion decomposition, or alternative trajectory representations that do not depend on dense flow fields. To improve visualization precision, learned spatial grounding or attention-based localization could replace Grad-CAM [50] for concept overlay. Extending TraCE to long-range temporal reasoning may involve generalizing trajectory concepts to broader visual dynamics, such as modeling temporal interactions among scene elements beyond pixel-level flow.

Broader Impacts

This work aims to advance explainability in video understanding by representing dynamic evidence as trajectory concepts and separating it from static object and scene context. By deriving motion concepts from optical-flow trajectories, TraCE enables source-flexible explanations that cover people, manipulated objects, and multi-agent interactions within a single framework. This broader coverage makes trajectory-based explanations applicable to a wider range of real-world scenarios, such as object-centric manipulation and multi-agent environments, and may extend to egocentric video. Furthermore, the same typed concept interface supports both action recognition and video question answering, demonstrating that concept-level explanations need not be redesigned for each task. By exposing whether a model relies on meaningful motion cues, static contextual shortcuts, or spurious correlations, TraCE can help practitioners diagnose and correct model errors through concept-level intervention, promoting trustworthy deployment in high-stakes domains such as healthcare, education, and human–AI interaction. We hope this line of research fosters safer, more interpretable, and human-aligned video AI systems. At the same time, explanations can be over-trusted: a plausible concept explanation does not by itself guarantee correct model behavior, so explanations should complement rather than replace rigorous evaluation. In addition, improved action recognition and interpretability tools could be misused for surveillance; we encourage their use in accordance with applicable privacy regulations and ethical guidelines.

References

  1. [1]Geo Ahn, Jiwoook Han, Youngrae Kim, Joonseok Lee, and Jinwoo Choi. EVIDENT: Routing MLLM adaptation through entity-grounded visual evidence for cross-domain video temporal grounding. In NeurIPS, 2026. 3arxiv.org/abs/2605.26104
  2. [2]Geo Ahn, Inwoong Lee, Taeoh Kim, Minho Shim, Dongyoon Wee, and Jinwoo Choi. Why can’t I open my drawer? Mitigating object-driven shortcuts in zero-shot compositional action recognition. In ECCV, 2026. 1, 3arxiv.org/abs/2601.16211
  3. [3]Kyungho Bae, Geo Ahn, Youngrae Kim, and Jinwoo Choi. DEVIAS: Learning disentangled video representations of action and scene for holistic video understanding. In ECCV, 2024. 3arxiv.org/abs/2312.00826
  4. [4]Kyungho Bae, Jinhyung Kim, Sihaeng Lee, Soonyoung Lee, Gunhee Lee, and Jinwoo Choi. MASH-VLM: Mitigating action-scene hallucination in Video-LLMs through disentangled spatial-temporal representations. In CVPR, 2025. 3arxiv.org/abs/2503.15871
  5. [5]Piyush Nitin Bagad and Andrew Zisserman. Chirality in action: Time-aware video representation learning by latent straightening. In NeurIPS, 2025. 3, 8, 9, 15, 16, 22, 23, 24, 28, 34
  6. [6]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. 1, 3, 5, 19, 30arxiv.org/abs/2511.21631
  7. [7]Itay Benou and Tammy Riklin Raviv. Show and tell: Visually explainable deep neural nets via spatially-aware concept bottleneck models. In CVPR, 2025. 2, 7, 8, 9, 23, 24, 26arxiv.org/abs/2502.20134
  8. [8]Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, 2021. 1, 3arxiv.org/abs/2102.05095
  9. [9]Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P. Calmon, and Himabindu Lakkaraju. Interpreting CLIP with sparse linear concept embeddings (SpLiCE). In NeurIPS, 2024. 8, 37arxiv.org/abs/2402.10376
  10. [10]Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017. 3arxiv.org/abs/1705.07750
  11. [11]Junwen Chen and Yu Kong. Explainable video entailment with grounded visual evidence. In ICCV, 2021. 3DOI
  12. [12]Tieyuan Chen, Huabin Liu, Tianyao He, Yihang Chen, Chaofan Gan, Xiao Ma, Cheng Zhong, Yang Zhang, Yingxue Wang, Hui Lin, et al. MECD: Unlocking multi-event causal discovery in video reasoning. In NeurIPS, 2024. 2arxiv.org/abs/2409.17647
  13. [13]Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. VideoLLaMA 2: Advancing spatial-temporal modeling and audio understanding in Video-LLMs. arXiv preprint arXiv:2406.07476, 2024. URL https://arxiv.org/abs/2406.07476. 30
  14. [14]Jinwoo Choi, Chen Gao, Joseph CE Messou, and Jia-Bin Huang. Why can’t I dance in the mall? learning to mitigate scene bias in action recognition. In NeurIPS, 2019. 1arxiv.org/abs/1912.05534
  15. [15]Andong Deng, Tongjia Chen, Shoubin Yu, Taojiannan Yang, Lincoln Spencer, Yapeng Tian, Ajmal Saeed Mian, Mohit Bansal, and Chen Chen. Motion-grounded video reasoning: Understanding and perceiving motion at pixel level. In CVPR, 2025. 3arxiv.org/abs/2411.09921
  16. [16]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. 16arxiv.org/abs/1810.04805
  17. [17]Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019. 3arxiv.org/abs/1812.03982
  18. [18]Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Freund, Peter Yianilos, Moritz Mueller-Freitag, et al. The “something something” video database for learning and evaluating visual common sense. In ICCV, 2017. 3, 8, 9, 15, 16, 22, 23, 24, 28, 34arxiv.org/abs/1706.04261
  19. [19]Thomas Hartley, Kirill Sidorov, Christopher Willis, and David Marshall. SWAG-V: explanations for video using superpixels weighted by average gradients. In WACV, 2022. 2, 3
  20. [20]De-An Huang, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, and Juan Carlos Niebles. What makes a video a video: Analyzing temporal information in video understanding models and datasets. In CVPR, 2018. 1DOI
  21. [21]Ying Ji, Yu Wang, and Jien Kato. Spatial-temporal concept based explanation of 3D ConvNets. In CVPR, 2023. 3arxiv.org/abs/2206.05275
  22. [22]Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 17, 20arxiv.org/abs/1412.6980
  23. [23]Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML, 2020. 3arxiv.org/abs/2007.04612
  24. [24]Matthew Kowal, Achal Dave, Rares Ambrus, Adrien Gaïdon, Konstantinos G Derpanis, and Pavel Tokmakov. Understanding video transformers via universal concept discovery. In CVPR, 2024. 3arxiv.org/abs/2401.10831
  25. [25]Dongho Lee, Jongseo Lee, and Jinwoo Choi. Cast: Cross-attention in space and time for video action recognition. In NeurIPS, 2023. 3arxiv.org/abs/2311.18825
  26. [26]Jongseo Lee, Wooil Lee, Gyeong-Moon Park, Seong Tae Kim, and Jinwoo Choi. Disentangled concepts speak louder than words: Explainable video action recognition. In NeurIPS, 2025. 2, 3, 5, 7, 8, 9, 18, 19, 20, 21, 23, 24, 25, 26, 27, 33, 34, 37arxiv.org/abs/2511.03725
  27. [27]Jongseo Lee, Wooil Lee, Gyeong-Moon Park, Seong Tae Kim, and Jinwoo Choi. PCBEAR: Pose concept bottleneck for explainable action recognition. In CVPRW, 2025. 2, 3arxiv.org/abs/2504.13140
  28. [28]Jongseo Lee, Joohyun Chang, Dongho Lee, and Jinwoo Choi. CA²ST: Cross-attention in audio, space, and time for holistic video recognition. TPAMI, 2026. 3arxiv.org/abs/2503.23447
  29. [29]Jongseo Lee, Hyuntak Lee, Sunghun Kim, Sooa Kim, Jihoon Chung, and Jinwoo Choi. Which way did it move? Diagnosing and overcoming directional motion blindness in Video-LLMs. In NeurIPS, 2026. 3arxiv.org/abs/2605.22823
  30. [30]Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. In CVPR, 2022. 3arxiv.org/abs/2206.02349
  31. [31]Zhenqiang Li, Weimin Wang, Zuoyue Li, Yifei Huang, and Yoichi Sato. Towards visually explaining video understanding networks with perturbation. In WACV, 2021. 2, 3arxiv.org/abs/2005.00375
  32. [32]Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaVA: Learning united visual representation by alignment before projection. In EMNLP, 2024. 1, 3, 30arxiv.org/abs/2311.10122
  33. [33]Huabin Liu, Xiao Ma, Cheng Zhong, Yang Zhang, and Weiyao Lin. TimeCraft: Navigate weakly-supervised temporal grounded video question answering via bi-directional reasoning. In ECCV, 2024. 3DOI
  34. [34]Huabin Liu, Filip Ilievski, and Cees GM Snoek. Commonsense video question answering through video-grounded entailment tree reasoning. In CVPR, 2025. 3arxiv.org/abs/2501.05069
  35. [35]Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017. 15arxiv.org/abs/1608.03983
  36. [36]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 15
  37. [37]James MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, pages 281–298. University of California press, 1967. 28, 29
  38. [38]Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-else: Compositional action recognition with spatial-temporal interaction networks. In CVPR, 2020. 7, 9, 23, 33arxiv.org/abs/1912.09930
  39. [39]Etienne Meunier and Patrick Bouthemy. Segmenting the motion components of a video: A long-term unsupervised model. TPAMI, 2025. 1DOI
  40. [40]Tuomas Oikarinen, Subhro Das, Lam Nguyen, and Lily Weng. Label-free concept bottleneck models. In ICLR, 2023. 3, 5, 8, 17, 18, 19, 20, 23, 37arxiv.org/abs/2304.06129
  41. [41]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 6, 17arxiv.org/abs/1807.03748
  42. [42]Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R Zaiane, and Martin Jagersand. U²-Net: Going deeper with nested U-structure for salient object detection. Pattern recognition, 106: 107404, 2020. 18DOI
  43. [43]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 20arxiv.org/abs/2103.00020
  44. [44]Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Discover-then-name: Task-agnostic concept bottlenecks via automated concept discovery. In ECCV, 2024. 3arxiv.org/abs/2407.14499
  45. [45]Arthur S Reber. Implicit learning and tacit knowledge. Journal of Experimental Psychology: General, 118(3):219, 1989. 2
  46. [46]Avinab Saha, Shashank Gupta, Sravan Kumar Ankedriddy, Karl Chahine, and Joydeep Ghosh. Exploring explainability in video action recognition. In CVPRW, 2024. 3arxiv.org/abs/2404.09067
  47. [47]Kate Sanders, Nathaniel Weir, and Benjamin Van Durme. TV-TREES: Multimodal entailment trees for neuro-symbolic video reasoning. In EMNLP, 2024. 3
  48. [48]Saquib Sarfraz, Vivek Sharma, and Rainer Stiefelhagen. Efficient parameter-free clustering using first neighbor relations. In CVPR, 2019. 5, 19, 21, 28, 29arxiv.org/abs/1902.11266
  49. [49]C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: a local SVM approach. In ICPR, 2004. 3, 15, 22
  50. [50]Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017. 7, 19, 23, 38
  51. [51]Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Only time can tell: Discovering temporal data for temporal modeling. In WACV, 2021. 1
  52. [52]Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. NTU RGB+D: A large scale dataset for 3D human activity analysis. In CVPR, 2016. 3, 8, 15, 16, 22, 24arxiv.org/abs/1604.02808
  53. [53]Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. TOMATO: Assessing visual temporal reasoning capabilities in multimodal foundation models. In ICLR, 2025. 3, 9, 16, 22, 23arxiv.org/abs/2410.23266
  54. [54]Tongqing Shi, Ge Yan, Tuomas Oikarinen, and Tsui-Wei Weng. Multimodal concept bottleneck models. In NeurIPSW, 2025. 6, 27, 37arxiv.org/abs/2606.19882
  55. [55]Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurIPS, 2014. 1, 3arxiv.org/abs/1406.2199
  56. [56]Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. OpenAI GPT-5 system card. arXiv preprint arXiv:2601.03267, 2026. 24, 25, 26, 27, 30arxiv.org/abs/2601.03267
  57. [57]Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 3, 15, 16, 22arxiv.org/abs/1212.0402
  58. [58]Divyansh Srivastava, Ge Yan, and Tsui-Wei Weng. VLG-CBM: Training concept bottleneck models with vision-language guidance. In NeurIPS, 2024. 3, 6, 23, 31, 33arxiv.org/abs/2408.01432
  59. [59]Alexandros Stergiou, Georgios Kapidis, Grigorios Kalliatakis, Christos Chrysoulas, Remco Veltkamp, and Ronald Poppe. Saliency tubes: Visual explanations for spatio-temporal convolutions. In ICIP, 2019. 2, 3
  60. [60]Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, 2022. 1, 3, 8, 9, 15, 23, 28, 31arxiv.org/abs/2203.12602
  61. [61]John YA Wang and Edward H Adelson. Representing moving images with layers. IEEE Transactions on Image Processing, 3(5):625–638, 1994. 1
  62. [62]Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. InternVid: A large-scale video-text dataset for multimodal understanding and generation. In ICLR, 2024. 30, 31arxiv.org/abs/2307.06942
  63. [63]Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. InternVideo2: Scaling foundation models for multimodal video understanding. In ECCV, 2024. 1, 3, 5, 8, 15, 20, 23, 28, 30, 31arxiv.org/abs/2403.15377
  64. [64]Yihan Wang and Jia Deng. WAFT: Warping-alone field transforms for optical flow. In ICLR, 2026. 5, 18arxiv.org/abs/2506.21526
  65. [65]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In EMNLP, 2020. 17, 20DOI
  66. [66]Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In ICML, 2021. 6, 18, 21arxiv.org/abs/2105.04857
  67. [67]Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. STAR: A benchmark for situated reasoning in real-world videos. In NeurIPS, 2021. 3, 16, 21, 22, 37, 38
  68. [68]Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can I trust your answer? Visually grounded video question answering. In CVPR, 2024. 3arxiv.org/abs/2309.01327
  69. [69]Xinyue Xu, Yi Qin, Lu Mi, Hao Wang, and Xiaomeng Li. Energy-based concept bottleneck models: Unifying prediction, concept intervention, and probabilistic interpretations. In ICLR, 2024. 3arxiv.org/abs/2401.14142
  70. [70]Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR, 2023. 3arxiv.org/abs/2211.11158
  71. [71]Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In ICLR, 2023. 3arxiv.org/abs/2205.15480
  72. [72]Yuanhan Zhang, Bo Li, Haotian Liu, Yong Jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. LLaVA-NeXT: A strong zero-shot video understanding model, April 2024. URL https://llava-vl.github.io/blog/2024-04-30-llava-next-video/. 30

Paper details

vlnc:W_fm844j8e
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 13

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents