VISTA: Vision-Language Integrated Semantic Triage Architecture for Online Face Clustering in Long Videos

main.pdf

CCS Concepts

• Computing methodologies → Video segmentation; Object identification; Tracking; Cluster analysis.

ACM Reference Format:

Jiashu Liao, Joanne Garde-Hansen, Sanjay Sharma, and Tanaya Guha. 2026. VISTA: Vision-Language Integrated Semantic Triage Architecture for Online Face Clustering in Long Videos. In Proceedings of the 34th ACM International Conference on Multimedia (MM ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3767308.3835422

Introduction

The explosive growth of long form video content demands robust algorithms to facilitate a number of human centric video understanding tasks [37], such as automated character identification and narrative analysis [3, 11, 31] and personalised video browsing [15]. At the core of these applications lies video face clustering i.e., grouping faces detected over a video stream based on identity, where each face cluster represents a unique person. The majority of existing video face clustering methods are offline methods i.e., they process all data points simultaneously through global optimization [3, 33, 35]; however, the scale of modern video streams demands online, single pass processing algorithms that can maintain low clustering latency and remain compatible with streaming.

Strictly online face clustering algorithms are scarce in the literature due to the constraints of single pass, low latency processing. To the best of our knowledge, Constrained Online Face Clustering (COFC) [16] remains the state of the art for online, unsupervised face clustering. It operates by generating them incrementally using geometric embeddings (e.g., FaceNet [25]) and spatio temporal must link/cannot link constraints. Despite the efficiency, such pure visual methods fall under extreme variations in illumination, thereby often confusing identities. Several works have attempted to address this via improved face representations [7] or graph based linkage technique [36], but the fundamental limitation is still relying solely on visual face similarity.

Recently, Vision Language Models (VLMs) like CLIP [22] have been successful in improving offline clustering [29] by extracting zero shot attributes (e.g., gender, age, clothing). However, applying VLM attributes directly as hard constraints introduces a critical prediction instability problem; i.e., in unconstrained videos, large variation in illumination causes VLM ‘hallucinations’ [19] (e.g., a red jacket identified as black in the dark) [29]. Furthermore, as characters in long videos frequently change costumes, hard constraints based on hair or clothing will force the algorithm to reject valid faces. Because continuous temporal sequences are broken when intermediate faces are rejected, this leads to the artificial fragmentation of face trajectories. This over segmentation artificially

This work is licensed under a Creative Commons Attribution 4.0 International License.

MM ’26, Rio de Janeiro, Brazil

© 2026 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-2213-4/2026/11

https://doi.org/10.1145/3767308.3835422 inflates Weighted Cluster Purity (WCP) while degrading global identification accuracy [9].

The basic idea of online face clustering is to group face tracks based on identity by processing short video segments (1 sec) as they come. If an identity has already been discovered, the new face track will be merged with that cluster, otherwise, a new cluster will be formed considering a ’new’ identity. To harness VLM reasoning within this paradigm (without falling into the over segmentation trap), we propose the Vision language Integrated Semantic Triage Architecture (VISTA). VISTA operates as a fully unsupervised, online clustering pipeline enriched by three semantic modules. (1) First, we introduce the idea of Symbolic Semantic Representation: Instead of comparing raw VLM predictions, SSR maps noisy coarse age attributes (e.g., oscillating age predictions like "20-29" vs. "30-39") to a compact set of stable discrete tokens (e.g., <age_adult>) via a predefined deterministic mapping table (which we term a symbol registry), acting as a temporal denoising layer. (2) We propose a three zone semantic triage mechanism to improve clustering accuracy for the difficult samples. By maintaining a sliding window of the most recent 150 geometric similarity scores between track cluster pairs, we dynamically fit a Gaussian Mixture Model (GMM) [23] to partition these track to cluster association candidates into three zones: high confidence (merge), reject, and ambiguity zone. The ambiguity zone requires VLM arbitration. (3) To resolve the ambiguity we introduce spatio temporal decoupled semantic arbitration. VISTA decouples attributes into persistent (gender, age) and transient (clothing, hair) attributes. By applying a low confidence semantic masking mechanism during poor illumination to filter out VLM hallucinations, and a temporal decay for transient attributes across long temporal gaps (bypassing costume changes), VISTA achieves more robust global identity tracking. To summarise, our main contributions are:

• We present VISTA, an online vision language framework that fuses geometric embeddings with zero shot semantic attributes. By designing the semantic integration to be lightweight, our core clustering module operates at over 1800 FPS (clustering module only; the end to end pipeline including detection and feature extraction runs at 7–8 FPS, § 2.3.6), closing the semantic gap between low level geometric features and human interpretable identity without compromising online efficiency.

• We introduce the idea of a Symbolic Semantic Representation and a Three Zone Semantic Triage scheme with GMM based dynamic thresholding to adapt to the high temporal variability in facial/personal attributes in long videos. Since COFC is already online and unsupervised, the novelty of VISTA lies specifically in this selective online semantic control layer.

• We identify and address the VLM induced over segmentation problem through a spatio temporal decoupled semantic arbitration strategy. Rather than naively enforcing semantic constraints, VISTA uses semantics selectively to reduce cross identity confusion while maintaining a favorable trade off between cluster purity and identity completeness across challenging long video benchmarks.

Related Work

Face and person clustering in videos. Video face clustering has traditionally been framed as an offline task. Standard approaches extract deep CNN features and perform global optimization such as graph linkage [38] or test time finetuning [33]. Beyond pure face features, person level representations incorporating body and clothing cues have been explored to disambiguate visually similar faces [28]. Sarfraz et al. proposed FINCH {citep}[24], a parameter free first nearest neighbour clustering method that achieves competitive results with pre trained features. Building on this, Kalogeiton and Zisserman introduced C1C {citep}[12], which combines first nearest neighbour clustering with spatio temporal cannot link constraints derived from the video structure, improving cluster purity. For streaming scenarios, Kulsreshtha et al. proposed COFC {citep}[16], utilizing temporal overlap within shots as a strict cannot link constraint ($Q$ matrix) to guide incremental updates; this online face clustering paradigm was later extended to construct dynamic character graphs for movie analysis {citep}[17]. Supervised methods such as Ball Cluster Learning (BCL) {citep}[33] learn to map features into pre defined metric balls; on the VPCD benchmark, BCL results are reported under the AT (unknown number of clusters) and OC (oracle number of clusters) protocols {citep}[3]`, and BCL is neither unsupervised nor streaming. Multi modal approaches like MuHPC [3] hierarchically fuse face, voice, and body features to bridge otherwise unmergeable clusters, achieving strong offline results on the VPCD benchmark but remaining inherently offline and requiring multiple modalities. More recently, VideoClusterNet [35] proposed self supervised refinement of face embeddings during inference, but requires multiple passes over the data. Because recent advancements have predominantly focused on offline or supervised paradigms, COFC [16] remains the most representative baseline for fully online, unsupervised settings. We note that existing offline methods cannot be trivially adapted to online settings: MuHPC requires global hierarchical merging across the full dataset, VideoClusterNet performs multi pass self supervised refinement, and C1C relies on a complete first-nearest-neighbour graph, all fundamentally requiring full data access. VISTA builds upon this online tracking paradigm but expands its inter shot capability by incorporating dynamic semantic triage while remaining fully unsupervised and single pass.

VLMs for long video understanding. The advent of VLMs like CLIP [22] has revolutionized zero shot video understanding. For face analysis, Shen et al. introduced CLIP-Cluster [29], showing that VLMs can extract semantic attributes to assist in face grouping. Subsequent large scale vision language models such as BLIP2 [18] and LLaVA [20] further improved zero shot recognition capabilities, enabling richer attribute extraction from video frames. However, VLM outputs are susceptible to motion blur and illumination drops in the wild. DeepFace [26] offers a complementary source of demographic predictions but similarly degrades under low image quality. Earlier attempts at naming TV characters also leveraged multimodal cues from subtitles and dialog [11], yet these require structured metadata not always available in general video streams. VISTA repurposes VLM predictions as a source of symbolic supervision, employing a spatio temporal decoupling mechanism to dynamically penalize or ignore semantic contradictions based on environmental context, thereby bridging the gap between powerful zero shot capabilities and the practical instability of real world deployment.

Methodology

The VISTA framework processes streaming video through three cascaded stages: (1) Feature Extraction, (2) Intra Shot Tracking, and (3) Inter Shot Semantic Association. Figure 1 illustrates the overall pipeline. Pseudocode for the complete online clustering procedure is provided in the Supplementary Material. Geometric embeddings and CLIP/DeepFace attributes are extracted up front for every face crop in Stage 1; triage controls only how often semantic arbitration fires (§ 2.2.3).

Problem Formulation

Given a continuous video stream $\mathcal{V} = \{f_1, f_2, \ldots\}$, we detect face instances and seek to assign each detection to an identity cluster in an online fashion, without revisiting past frames. Let $D_t = \{d_1^t, d_2^t, \ldots\}$ denote detected faces at frame $t$, each associated with a geometric embedding $v \in\mathbb{R}^{512}$ and a set of raw attribute predictions $\mathcal{A} = \{a_{\mathrm{gender}}, a_{\mathrm{age}}, a_{\mathrm{clothing}}, a_{\mathrm{hair}}\}$. The goal is to maintain a set of global identity clusters $\mathcal{C} = \{C_1, C_2, \ldots, C_K\}$ where $K$ is unknown and grows dynamically. Each incoming face track must be either merged into an existing cluster or instantiated as a new identity, using only information available up to the current time step.

Feature Extraction and Intra Shot Tracking

Faces are detected using RetinaFace [8] and represented via FaceNet [25] embeddings. While our architecture is modular and supports alternative backends (e.g., InsightFace [10]), we employ this specific setup by default to maintain a strict, fair comparison with classical online baselines like COFC. Within a camera shot, we construct short term face tracks $T$ based on bounding box IoU and $L_2$ feature distance. Specifically, a detection $d_i^t$ is appended to an existing track $T_j$ if $\operatorname{IoU}(d_i^t, d_j^{t-1}) > \theta_{\mathrm{iou}}$ and $\lVert v_i^t - \bar{v}_j \rVert_2 < \theta_{\mathrm{feat}}$, where $\bar{v}_j$ is the running average embedding of $T_j$. Each track accumulates up to $L_{\max}$ frames (default 24) to form a stable representation. To enforce physical realism, we construct a temporal constraint matrix $Q$. If two tracks co-occur in the same frame, $Q_{A,B} = 0$ (cannot link constraint); otherwise $Q_{A,B} = 1$.

Inter Shot Clustering: Semantic Association

Symbolic Semantic Representation (SSR). Applying raw VLM textual predictions (e.g., ‘Age: 20–29’) directly is fragile due to frame level noise. VISTA defines a Symbol Registry, consisting of deterministic mapping functions $\phi_d : L_d \to S_d$ that convert coarse age grouping labels into discrete symbols. For example, both ‘Age: 20–29’ and ‘Age: 30–39’ map to the symbol <age_adult>, absorbing boundary oscillations. We define four attribute dimensions $d \in\{\mathrm{gender}, \mathrm{age}, \mathrm{clothing}, \mathrm{hair}\}$, each with a compact symbol vocabulary $\lvert S_d \rvert\in\{2, 5, 7, 5\}$ respectively. The exact mapping schemes and an empirical comparison between different age symbol configurations are provided in the Supplementary Material. Overall, the coarse grouping used in our unified setting provides a robust compromise across datasets by reducing boundary oscillation while retaining more discriminative power than broader adult collapsing.

Over the duration of a track, VISTA accumulates these symbols. The track’s final SSR profile is generated via a temporal majority vote, acting as a robust denoising mechanism. The confidence $\rho(s)$ of a symbol $s$ in a track is defined as its frequency of occurrence across all valid frames in the track:

Only frames where the attribute prediction passes a minimum detection quality threshold contribute to the vote, filtering out corrupted observations.

Three zone triage with GMM dynamic thresholding. To evaluate a merge between a query track $T$ and a global cluster $C$, we compute the cosine similarity $\operatorname{sim}(v_T, v_C)$. Using thresholds $\tau_{\mathrm{high}}$ and $\tau_{\mathrm{low}}$, VISTA triages pairs into three zones, each activating a different mode of semantic arbitration: (1) High confidence zone ($\operatorname{sim} > \tau_{\mathrm{high}}$): merge is accepted; semantic cues contribute bonus only adjustments (no penalties) to reinforce the decision; (2) Ambiguity zone ($\tau_{\mathrm{low}} \le\operatorname{sim} \le\tau_{\mathrm{high}}$): full semantic arbitration with both rewards and penalties; (3) Reject zone ($\operatorname{sim} < \tau_{\mathrm{low}}$): merge is rejected by default, but a rescue mechanism can override the rejection when persistent semantic cues (gender + age) strongly agree.

Method	Online?	Unsup.?	Core Mechanism	TBBT	Buffy	Sherlock
Controlled comparison: online, unsupervised, identical features and protocol (WCP / NMI / ID_Acc)
COFC [16]	✓	✓	Constrained Incremental	92.13 / 86.97 / 89.74	64.60 / 66.28 / 64.31	76.22 / 50.56 / 71.27
VISTA (Full Framework)	✓	✓	Spatio Temporal Semantic Triage	96.97 / 91.25 / 93.94	87.39 / 79.30 / 82.82	80.45 / 51.19 / 68.10
Literature reference: SENet-50 features†, reported numbers (WCP / NMI)
C1C† [12]	✗	✓	Constrained 1NN	89.20 / 87.40	66.30 / 68.80	76.30 / 50.30
BCL† (AT) [3, 33]	✗	✗	Ball Cluster Learning	90.80 / 85.70	85.00 / 78.80	- / -
BCL† (OC) [3, 33]	✗	✗	Ball Cluster Learning	94.00 / 85.00	86.50 / 77.60	- / -
MuHPC† (Face) [3]	✗	✗	Global Hierarchical 1NN	99.40 / 97.80	96.10 / 92.80	86.70 / 60.30

Rather than relying exclusively on static boundaries, VISTA maintains a sliding window queue $Q$ of recent similarity scores and fits a bimodal Gaussian Mixture Model (GMM) online to capture the distribution of same identity and different identity pairs. During the initial warm up phase (i.e., before $Q$ is fully populated), the system relies on static initial thresholds $\tau_{\mathrm{high}}^{(0)}$ and $\tau_{\mathrm{low}}^{(0)}$. The sensitivity of these initial parameters is extensively evaluated in the Supplementary Material. The GMM is formalized as:

where the lower component ($\mu_1 < \mu_2$) models different identity similarities and the upper component models same identity similarities. We dynamically estimate the threshold proposals as

The 0.50 cap on $\tau_{\mathrm{low}}^{\star}$ is a deliberate design choice: it preserves a sufficiently wide ambiguity zone for semantic arbitration in visually compressed windows. A cap sweep over all 15 episodes (Supplementary Material) shows that relaxing it to 0.70 or 1.00 leaves all metrics unchanged after rounding, i.e., it is a stability safeguard rather than a tuned knob. To ensure temporal stability, we apply an exponential moving average update [34]:

where $\eta_{\mathrm{gmm}} = 0.3$ is the GMM threshold smoothing coefficient, distinct from the cluster representation momentum ema_rep (Section 4.2). This mechanism allows VISTA to adapt to evolving visual distributions across different scenes while preventing abrupt threshold oscillations. An empirical visualization confirming the bimodal structure of the similarity distributions across all 15 episodes is provided in the Supplementary Material.

Spatio temporal decoupled arbitration. For pairs in the ambiguity zone ($\tau_{\mathrm{low}} \leq\operatorname{sim} \leq\tau_{\mathrm{high}}$), VISTA compares their SSR profiles to compute an adjustment score $\operatorname{adj}$. We categorize attributes into persistent traits $\mathcal{P} = \{\text{gender}, \text{age}\}$ and transient traits $\mathcal{T} = \{\text{clothing}, \text{hair}\}$. We define a joint confidence weight $w = \min(p_{\mathcal{T}}(s), p_{\mathcal{C}}(s))$. The adjustment combines reward and penalty terms:

where $\gamma$ is a global penalty scaling factor. Reward terms include gender+age agreement ($\lambda_{\mathrm{ga}}$) and clothing consistency ($\lambda_{\mathrm{cloth}}$), while penalties include gender mismatch ($\lambda_{\mathrm{gm}}$), age mismatch ($\lambda_{\mathrm{am}}$), and hair conflict ($\lambda_{\mathrm{hc}}$). The final updated similarity is $\operatorname{sim}' = \operatorname{sim} + \operatorname{adj}$. A merge is accepted if $\operatorname{sim}' \geq\tau_{\mathrm{merge}}$. We set $\tau_{\mathrm{merge}} = 0.325$: it is the final acceptance gate for ambiguity zone pairs after arbitration, not a standalone visual threshold; a pair must be the best current assignment and its post arbitration score, after possible penalties, must pass the gate, whereas high confidence pairs are routed directly by $\tau_{\mathrm{high}}$. A sweep over $\{0.280\text{--}0.475\}$ (Supplementary Material) confirms the expected purity–continuity trade off. To prevent over segmentation, VISTA decouples the spatial and temporal dimensions as follows:

Low Confidence Semantic Masking. When the mean face detection confidence $\bar{c}_T$ of a track drops below a quality threshold $\theta_{\mathrm{cliff}}$ (indicating blur or darkness), VISTA continuously attenuates all semantic penalties via a linear scaling factor:

where $\theta_{\mathrm{floor}}$ is a lower bound below which attenuation saturates. This soft masking ensures that as visual quality degrades, the influence of potentially hallucinated VLM predictions is progressively suppressed rather than abruptly removed, avoiding discontinuous clustering behaviour at the quality boundary.

Temporal Decay for Transient Attributes. If the frame gap $\Delta f$ between a track and a cluster exceeds a long range threshold $\Delta_{\max}$, VISTA masks transient traits (e.g., clothing, hair) as unknown, bypassing costume driven conflicts:

This mechanism ensures that VISTA relies more on persistent traits for long range temporal association, while still leveraging transient cues for temporally adjacent merges where costume stability can be assumed.

Perplexity based confidence weighting. To further modulate the influence of semantic arbitration, VISTA maintains a short history window of $H$ recent per cluster entropy observations. When the perplexity of a cluster’s embedding distribution exceeds a threshold, indicating high intra cluster visual diversity, the semantic adjustment magnitude is down weighted by a factor

where $\beta$ controls the decay rate. This prevents aggressive semantic penalties for clusters that already exhibit high visual variance.

Experiments

Datasets and Evaluation Metrics

We evaluate on the VPCD benchmarks: TBVT (6 episodes, well lit sitcom), Sherlock (3 episodes, thriller with costume changes), and Buffy (6 episodes, dark action, which feature rich narrative and character arcs) [14]. We report Weighted Cluster Purity (WCP), Normalized Mutual Information (NMI), and Identification Accuracy (ID_Acc). WCP focuses on cluster homogeneity, while ID_Acc penalizes trajectory fragmentation. VISTA seeks a favorable balance across these metrics.

Implementation Details

VISTA is implemented in Python using PyTorch. As described in Section 3.2, we utilize RetinaFace [8] for face detection and FaceNet [25] for identity embedding. CLIP ViT-B/32 [22] performs zero shot attribute classification with text prompts for clothing and hair, and DeepFace [26] is used for age analysis. The complete CLIP text prompt templates, label vocabularies, and the full Symbol Registry mappings are provided in the Supplementary Material.

For intra shot tracking, we use $\theta_{\mathrm{iou}} = 0.3$ and $\theta_{\mathrm{feat}} = 1.0$. In the final full framework evaluation, we use a single unified configuration across all three datasets to avoid dataset specific tuning and demonstrate robust generalization: RetinaFace for detection, FaceNet for embedding, coarse age grouping age symbols, $\tau_{\mathrm{high}}^{(0)} = 0.85$, $\tau_{\mathrm{low}}^{(0)} = 0.25$, cluster representation smoothing momentum ema_rep = 0.95, GMM threshold smoothing $\eta_{\mathrm{gmm}} = 0.3$, arbitration gate $\tau_{\mathrm{merge}} = 0.325$, GMM queue size $|Q| = 150$, perplexity weighting enabled, and fixed length scene segmentation with length 24. Dynamic thresholding and semantic arbitration are enabled. Perplexity weighting uses a history window of $H = 20$ observations with $\omega_{\min} = 0.10$ and $\beta= 1.2$. For semantic arbitration (Eq. (5)), the adjustment weights are empirically set as $\lambda_{ga} = 0.08$, $\lambda_{cloth} = 0.04$, $\lambda_{gm} = 0.12$, $\lambda_{am} = 0.02$, and $\lambda_{hc} = 0.04$. A grid search over 243 $\lambda$ configurations (detailed in the Supplementary Material) shows that VISTA’s performance is stable across a wide range of settings: WCP standard deviation is only 0.31% on TBBT, 1.09% on Buffy, and 0.78% on Sherlock, confirming that the method is not driven by fragile hyperparameter tuning.

To study robustness, we additionally perform a parameter sweep over $\tau_{\mathrm{high}}^{(0)}$, $\tau_{\mathrm{low}}^{(0)}$, age symbol configuration, ema_rep momentum, and GMM queue size. The detailed dataset specific parameter effect tables are reported in the Supplementary Material and should be interpreted as marginal robustness studies rather than the source of the final unified full framework setting; key trends there: coarse age grouping and strong ema_rep smoothing yield consistently best results, and darker scenes (Buffy) favor larger GMM queues for stable threshold estimation. In addition, a $3 \times3$ grid bracketing the warm up initialisation over $\tau_{\mathrm{high}}^{(0)} \in\{0.75, 0.825, 0.90\}$ and $\tau_{\mathrm{low}}^{(0)} \in\{0.20, 0.275, 0.35\}$ (135 episode level evaluations, all else fixed) yields max−min variation of at most 0.69 pp across all metrics and datasets (Supplementary Material); $\tau_{\mathrm{high}}$ sensitivity is limited precisely because GMM dynamic thresholding overrides the static initialisation after warm up.

All experiments are conducted on a single NVIDIA RTX A6000 GPU. A detailed breakdown of the computational efficiency and throughput of our pipeline is discussed in § 2.3.6.

Quantitative Results

Table 1 reports a controlled comparison between VISTA and COFC under identical detector, embeddings, and protocol, together with literature reported results as reference points. We note that strictly online unsupervised face clustering methods are rare; COFC [16] remains the only published method in this category with results on VPCD. We additionally include offline unsupervised (C1C [12]), supervised (BCL [33]), and multi-modal (MuHPC [3]) methods as reference points. Methods marked with † use SENet-50 features, which are generally more discriminative than the FaceNet embeddings used by VISTA and COFC [7].

VISTA outperforms COFC across all three datasets under identical features and evaluation. On TBBT, WCP and NMI improve from 92.13%/86.97% to 96.97%/91.25%, and ID_Acc from 89.74% to 93.94%. On Buffy, VISTA raises WCP and NMI from 64.60%/66.28% to 87.39%/79.30%, with ID_Acc improving from 64.31% to 82.82% (+18.51 pp), also far surpassing the offline C1C (66.30%/68.80%) that uses stronger features. As literature reference points, VISTA’s TBBT results (96.97%/91.25%) are also higher than the reported supervised BCL (AT) numbers (90.80%/85.70%), and its Buffy NMI is comparable (79.30% vs. 78.80%), although feature and protocol differences preclude a strict comparison.

On Sherlock, VISTA improves over both COFC (76.22%/50.56%) and C1C (76.30%/50.30%), reaching 80.45%/51.19%. However, ID_Acc decreases from 71.27% (COFC) to 68.10% (−3.17 pp): under Sherlock’s heavy disguises and long gap appearance variation, VISTA prioritises cluster purity over identity continuity, yielding cleaner clusters at the cost of additional fragmentation. We regard this as a limitation rather than a universal all metric gain; § 2.3.4 and § 2.3.5 analyse this purity–continuity trade off in detail. Although the gains are more modest than on TBBT and Buffy, this confirms that selective semantic integration provides consistent WCP/NMI improvements even on the most volatile dataset, with room for future refinement.

Additional online baselines. To isolate the source of the gains, we evaluate two further online baselines under the same features and protocol (full tables in the Supplementary Material). (i) Naive VLM hard constraints: raw attributes applied as must link/cannot link rules (no SSR, no triage) over fragment severely, producing 4.0–11.3× more clusters than ground truth; the inflated cluster count artificially raises WCP while ID_Acc collapses to 63.95/60.64/40.65 on TBBT/Buffy/Sherlock, far below VISTA Full and below our weakest geometry driven ablation (Table 2). This confirms that VISTA’s improvements stem from selective semantic triage, not from VLM attributes alone. (ii) Sequential online HAC attains very high WCP (99.73/97.62/91.19) under severe fragmentation (#Pred/#GT = 4.5/9.5/39.8×) but substantially lower NMI/ID_Acc (84.93/81.29, 76.78/63.49, 49.28/40.56), illustrating the over segmentation trap that three zone triage is designed to avoid.

Configuration	TBBT	Buffy	Sherlock
VISTA (Full Framework)	96.97 / 91.25 / 93.94	87.39 / 79.30 / 82.82	80.45 / 51.19 / 68.10
w/o Semantic Arbitration	96.71 / 89.97 / 91.27	80.54 / 77.62 / 78.35	79.06 / 52.55 / 69.56
w/o SSR	94.83 / 89.29 / 91.60	74.75 / 74.47 / 73.31	78.36 / 52.62 / 71.43
w/o Three Zone Gating	96.78 / 90.80 / 92.32	80.17 / 77.03 / 76.58	80.27 / 51.49 / 64.63
w/o Temporal Smoothing	96.28 / 90.09 / 91.95	82.07 / 77.44 / 77.05	84.11 / 52.54 / 62.23

Ablation Studies

We ablate key semantic components in Table 2. Overall, the results confirm that the semantic modules affect the final precision recall trade off, with the clearest gains appearing on TBBT and Buffy.

Semantic Components. As shown in Table 2, disabling Semantic Arbitration or SSR degrades performance (especially on TBBT), confirming that geometric similarity alone struggles with appearance variations and that SSR is vital for stabilizing noisy predictions. Similarly, removing Three Zone Gating or Temporal Smoothing weakens the precision recall trade off, leading to brittle merges. Interestingly, on the volatile Sherlock dataset, some ablated variants achieve higher individual metric scores. In particular, removing Temporal Smoothing raises WCP to 84.11% (vs. 80.45%). We attribute this to Sherlock’s frequent costume changes and disguises: temporal smoothing carries stale transient attribute profiles into subsequent decisions, occasionally penalising valid same identity merges. Conversely, removing SSR yields a higher NMI (52.62%) and ID_Acc (71.43%) but at the cost of lower WCP (78.36%), reflecting the fundamental purity completeness trade off. This dataset specific tension motivates future work on adaptive smoothing decay rates.

Effect of Tracklet Length. We ablate intra shot tracklet length averaged over all 15 episodes. Increasing the length from 12 to 48 improves ID_Acc (77.88%→81.05%) and NMI (77.39%→78.72%),

but extending to 96 slightly degrades both metrics due to accumulated appearance noise. WCP peaks at length 24 (92.83%). We adopt $L_{\max} = 24$ as the unified default, balancing purity and the early decision requirement of online clustering. The full table is in the Supplementary Material.

Qualitative Analysis

Figure 3 and the crop level comparison in the Supplementary Material provide qualitative comparisons on Sherlock, the most challenging dataset. These visualizations reveal that the main failure mode of classical online clustering is not uniform: some characters primarily suffer from cross identity confusion, while others are more affected by trajectory fragmentation. Figure 3 shows that COFC’s identity errors extend beyond local boundary noise: predicted clusters intrude into other characters’ timelines for extended intervals. A representative case is COFC’s Lestrade timeline, which contains repeated intrusions from a Mycroft dominant cluster, indicating substantial cross identity confusion rather than mere short term fragmentation. VISTA preserves cleaner separation between these visually similar but semantically distinct characters, suggesting that semantic triage prevents overly aggressive merging.

The crop level comparison (Supplementary Material) complements this: under challenging pose, illumination, and blur conditions, COFC more frequently assigns samples to competing clusters, whereas VISTA keeps a larger portion within identity consistent clusters. Importantly, VISTA’s remaining errors on Sherlock stem from overly conservative semantic decisions that occasionally fragment the same identity into multiple segments, rather than cross identity confusion. This is precisely the mechanism behind the ID_Acc decrease relative to COFC on Sherlock (Table 1): purity is defended at the cost of identity continuity. The qualitative gain is thus better understood as a shift in error profile, from large scale identity mixing toward more structured, localized fragmentation, consistent with the quantitative trends in Table 2. This also explains why removing Temporal Smoothing can raise WCP on Sherlock: without stale attribute profiles, fewer valid merges are rejected, at the cost of reduced cross identity discrimination.

Computational Efficiency and Throughput

A critical requirement for online clustering is computational efficiency. To demonstrate that VISTA’s semantic triage and spatio-temporal arbitration do not introduce prohibitive overhead, we profile the throughput of our pipeline. As shown in Table 3, the performance exhibits a clear dichotomy between the heavy feature extraction front end and the efficient clustering back end.

Dataset	Frames	Total Time (s)	End to End FPS	Clustering FPS
TBBT (s01e01)	32,990	4,655	7.09	2122.45
Buffy (s05e01)	62,620	7,978	7.85	2288.42
Sherlock (e01)	132,150	16,393	8.06	1842.30

The end to end throughput (Total FPS) stabilizes at approximately 7 to 8 frames per second. Profiling reveals that over 99% of this time is consumed by Stage 1 (Feature Extraction), which involves decoding high resolution video frames and executing dense face detection alongside multi-modal feature encoding (FaceNet, CLIP, and DeepFace attribute extraction).

In contrast, VISTA’s online clustering module (Stages 2 and 3) operates at 1800 to 2300 FPS. This highlights a major advantage of our design: by abstracting raw VLM outputs into compact SSR and employing dynamic GMM thresholding, VISTA restricts semantic arbitration to ambiguous boundary cases. Consequently, the proposed vision language integration achieves robust performance gains while keeping the online clustering phase itself efficient; in our implementation, the dominant runtime bottleneck remains face detection and feature extraction and not the clustering.

The Spatio Temporal Semantic Triage acts as an effective decision filter (Table 4). Across 1.2M+ frames, semantic arbitration is invoked for only 20.50% of tracklets (covering 11.64% of total frames). TBBT requires the least intervention (9.19% of frames) due to cleaner visual conditions, while Buffy’s darker scenes trigger more ambiguity (13.54%). Since CLIP/DeepFace attributes are extracted up front for every crop in Stage 1 (always included in the end to end cost), Table 4 measures how often arbitration fires during association, not how many VLM forward passes are avoided; triage keeps the clustering stage lightweight by restricting arbitration to ambiguous boundary cases.

Dataset	Frame level		Tracklet level
	Arb. / Total	Rate	Arb. / Total	Rate
TBBT	29,389 / 319,646	9.19%	3,685 / 20,737	17.77%
Buffy	64,222 / 474,210	13.54%	8,198 / 35,079	23.37%
Sherlock	47,919 / 421,899	11.36%	5,797 / 30,448	19.04%
Overall	141,530 / 1,215,755	11.64%	17,680 / 86,264	20.50%

Analysis of Demographic Disparities

Although VISTA reduces global identity confusion, VLM arbitration is not invoked uniformly across predicted demographic groups. We therefore report both triage rate diagnostics (Figure 4) and subgroup clustering-quality gaps in the Supplementary Material.

Female characters show higher intervention and ambiguity-zone rejection rates than male characters (13.48% vs. 10.76%, and 2.38% vs. 1.62%), suggesting that more disambiguation burden is offloaded to semantic arbitration in unconstrained settings. This observation is consistent with known gender accuracy disparities in commercial vision models [4], although our analysis is diagnostic rather than causal.

Age Disparities: Figure 4b reveals distinct effects at age extremes. Children (0–9 years) trigger the highest VLM intervention rate (24.03%), while older adults (50–59 years) face the highest rejection rate (5.93%). Conversely, the core 30–39 demographic, which is common in face model training distributions [13], requires the least intervention (8.55%). These trends suggest that skewed training data and difficult capture conditions can jointly affect representation robustness for age extremes [32].

Subgroup Quality and Safeguards: Beyond triage rates, the Supplementary Material reports WCP/NMI/ID_Acc gaps on the IoU matched subset using the same VLM predicted gender/age groups. Because VPCD lacks ground truth demographics, these groups do not constitute a full fairness audit; nevertheless, the gap signs vary across datasets rather than showing one directional degradation. As an ethical safeguard, VISTA can disable gender/age cues and fall back to geometry plus transient attributes.

Discussion: These preliminary findings show clear algorithmic disparities, but demographic variables in videos often entangle with environmental factors. Screen industry tropes, visual ageism [5, 6, 21], and historical underrepresentation [2, 30] often dictate lighting, camera angles, and action sequences for specific groups. For instance, low illumination scenes common in action genres can disproportionately degrade geometric feature extraction for darker skin tones, compounding existing gender and age ambiguities. Thus, attributing these disparities solely to intrinsic model bias requires further disentanglement. Future work will systematically analyze these issues by cross referencing scene level factors (illumination, pose, focal presence) with face model training distributions, while extending to the intersectional impacts of digital racialization [1] [27]. Addressing these entangled biases is essential for deploying equitable face clustering systems in real world, unconstrained video streaming applications.

Conclusion

We presented VISTA, a vision language semantic triage architecture for fully online face clustering in long videos. By combining Symbolic Semantic Representation, dynamic GMM thresholding, and spatio temporal decoupled arbitration, VISTA mitigates over segmentation and improves the purity–continuity trade off on TBBT, Buffy, and Sherlock. Under identical detector, embeddings, and evaluation protocol, VISTA consistently outperforms the strongest matched online unsupervised baseline while preserving streaming compatible efficiency. Limitations and Future Work: VISTA still relies on fixed symbolic vocabularies, a bimodal GMM prior, and approximate confidence attenuation; cross domain validation beyond scripted TV remains open. On Sherlock, VISTA trades identity continuity for purity (ID_Acc −3.17 pp. vs. COFC), motivating adaptive smoothing, calibrated VLM uncertainty, and confounder aware fairness analysis.

Acknowledgments

This work was funded by the Leverhulme Trust (RPG-2023-091).

References

Supplementary Material: VISTA: Vision-Language Integrated Semantic Triage Architecture for Online Face Clustering in Long Videos

Jiashu Liao; Joanne Garde-Hansen; Sanjay Sharma; Tanaya Guha

Supplementary Video Demonstrations

We provide two supplementary video demonstrations, demo_mycroft.mp4 and demo_lestrade.mp4, to illustrate the temporal stability of our framework across different levels of scene complexity. Both videos have been compressed to meet the file size limit of the submission system, but still clearly show a side-by-side comparison between the COFC baseline and VISTA in Sherlock (Season 1, Episode 1).

Video Layout & Evaluation Metric. Both videos are rendered in a condensed highlight format, with the left panel showing COFC and the right panel showing VISTA. We color the bounding boxes dynamically based on a purity-based metric:

• Cyan Box (Correct): The predicted cluster’s majority identity matches the ground-truth target character, and the cluster purity exceeds our threshold ($\ge60\%$).

• Red Box (Wrong): The cluster is either misclassified (wrong majority identity) or heavily contaminated (purity $< 60\%$). A contaminated cluster is treated as a failure even if it contains the target character.

Demo 1: Resolving Identity Over-merging. This video (demo_mycroft.mp4) focuses on Mycroft in relatively stable conversational scenes.

• Baseline: COFC over-merges Mycroft and Lestrade into a single large cluster. Although the assigned ID appears temporally continuous, it is incorrect, producing a persistent red box throughout.

• VISTA: By leveraging vision-language semantic arbitration, VISTA distinguishes the two characters and isolates Mycroft into a separate high-purity cluster, as indicated by the consistent cyan box.

Demo 2: Purity vs. Continuity in Dynamic Scenes. This video (demo_lestrade.mp4) tracks Lestrade through scenes with frequent occlusions, motion blur, and illumination changes.

• Baseline: COFC merges Lestrade, Mycroft, and several background extras into a single contaminated cluster (purity $\sim55\%$), which fails our metric and remains red throughout.

• VISTA: To preserve cluster purity under these conditions, VISTA’s dynamic triage deliberately over-splits the track into a few separate but pure clusters (e.g., C22 and C26, both with purity $>60\%$). This trade-off ensures that downstream offline aggregation receives clean tracklets rather than contaminated ones. The cyan boxes confirm that each resulting cluster passes our threshold.

VISTA Algorithm Pseudocode

To provide a clearer understanding of the online clustering process, we detail the step by step execution of VISTA in Algorithm S1. The algorithm processes the video stream in a single pass. For each camera shot, it extracts dense visual features and zero shot VLM attributes. Short term temporal tracking is performed to construct reliable local tracklets. Subsequently, the inter shot association is guided by our dynamic Gaussian Mixture Model (GMM) thresholding and spatio temporal decoupled semantic arbitration. This pipeline ensures that low confidence semantic noise is filtered out while valuable temporal constraints and geometric similarities are preserved. We emphasise that geometric embeddings and CLIP/DeepFace attributes are extracted up front for every face crop at shot ingestion (line 3) and voted into tracklet level SSR profiles; the three zone triage controls only whether semantic arbitration is invoked during association, not whether VLM inference is performed. The final acceptance gate for ambiguity zone pairs is $\tau_{\mathrm{merge}} = 0.325$ (see Section 3.3 of the main paper).

Algorithm S1 VISTA Online Clustering Pipeline

Require: Video stream $V$, initial thresholds $\tau_{\mathrm{high}}^{(0)}$, $\tau_{\mathrm{low}}^{(0)}$, $\tau_{\mathrm{merge}} = 0.325$

Ensure: Global identity clusters $\mathcal{C}$

1. Initialize $\mathcal{C} \leftarrow\varnothing$, similarity queue $Q \leftarrow\varnothing$

2. for each shot $S$ in $V$ do

3. Extract faces, compute geometric embeddings $\boldsymbol{v}$ and VLM attributes $\mathcal{A}$ (for every face crop)

4. Form intra shot tracks $\{T_1,\ldots,T_m\}$; build temporal constraint $Q$-matrix

5. for each track $T_i$ do

6. Compute SSR profile via temporal majority vote

7. Find best matching cluster: $\mathcal{C}^{*} = \arg\max_{\mathcal{C}_j}\operatorname{sim}(\boldsymbol{v}_{T_i},\boldsymbol{v}_{\mathcal{C}_j})$ s.t. $Q_{T_i,\mathcal{C}_j}=1$

8. $s \leftarrow\operatorname{sim}(\boldsymbol{v}_{T_i},\boldsymbol{v}_{\mathcal{C}^{*}})$

9. if $s > \tau_{\mathrm{high}}$ then

10. Apply bonus only semantic adjustment; merge $T_i$ into $\mathcal{C}^{*}$

11. else if $s \ge\tau_{\mathrm{low}}$ then

12. Compute adjustment $\operatorname{adj}$ via Spatio Temporal Decoupled Arbitration

13. if $s+\operatorname{adj} \ge\tau_{\mathrm{merge}}$ then

14. Merge $T_i$ into $\mathcal{C}^{*}$ (Resolved ambiguity)

15. else

16. Create new cluster $\mathcal{C}_{\mathrm{new}} \leftarrow\{T_i\}$; $\mathcal{C} \leftarrow\mathcal{C} \cup\{\mathcal{C}_{\mathrm{new}}\}$

17. end if

18. else

19. if gender + age strongly agree (persistent trait rescue) then

20. Merge $T_i$ into $\mathcal{C}^{*}$ (Rescued from reject zone)

21. else

22. Create new cluster $\mathcal{C}_{\mathrm{new}} \leftarrow\{T_i\}$; $\mathcal{C} \leftarrow\mathcal{C} \cup\{\mathcal{C}_{\mathrm{new}}\}$

23. end if

24. end if

25. Push $s$ to $Q$; update GMM thresholds $(\tau_{\mathrm{high}},\tau_{\mathrm{low}})$ via EMA $(\eta_{\mathrm{gmm}}=0.3)$

26. end for

27. end for

28. return $\mathcal{C}$

Age Symbol Configurations

As discussed in Section 3.3 of the main paper, applying raw VLM textual predictions directly introduces severe frame level noise. To mitigate this, VISTA maps raw age strings to a discrete vocabulary. We tested three different age grouping strategies, detailed in Table S1.

The fine_grained strategy preserves decade level precision for adults, which can occasionally differentiate close identities but is highly susceptible to VLM oscillation (e.g., predicting “20-29” in one frame and “30-39” in the next). The broad_adult strategy collapses a massive 40-year span into a single token, which guarantees stability but loses discriminative power. The coarse_grouping strategy provides the most robust compromise for the unified setting by reducing age boundary oscillation without collapsing the entire adult range into a single token.

Raw age range	fine_grained	coarse_grouping	broad_adult
0–9	<age_child>	<age_child>	<age_child>
10–19	<age_teen>	<age_teen>	<age_teen>
20–29	<age_20s>	<age_young_adult>	<age_adult>
30–39	<age_30s>	<age_adult>	<age_adult>
40–49	<age_40s>	<age_adult>	<age_adult>
50–59	<age_50s>	<age_senior>	<age_adult>
60+	<age_senior>	<age_senior>	<age_senior>
Vocabulary size	7	5	4

CLIP Prompt Templates and Symbol Registry

VISTA uses CLIP ViT-B/32 for zero shot attribute classification. For each detected face crop, logits are computed against all prompt candidates simultaneously in a single forward pass. Table S2 lists the complete prompt templates and label vocabularies for each attribute dimension. DeepFace is used separately for age estimation (not via CLIP prompts) and its raw integer predictions are mapped through the age symbol configurations in Table S1.

Attribute	Prompt Template	Labels
Gender	“A face of a {label}”	Male, Female
Clothing	“A photo of a person wearing a {label}”	T-shirt, Jacket, Shirt, Sweater, Dress, Coat, Vest
Hair	“A close-up photo of a person with {label}”	Curly Hair, Straight Hair, Bald, Short Hair, Long Hair

Table S3 provides the complete Symbol Registry mappings that convert raw CLIP predictions into stable symbolic tokens (SSR). Gender and hair labels are mapped one to one, while clothing labels undergo a many to one compression (e.g., both “T-shirt” and “Shirt” map to distinct tokens, but the vocabulary is reduced from open ended text to a fixed set of 7 symbols). Age mappings are handled separately via the configurations in Table S1.

For age estimation, VISTA uses DeepFace [1] with detector_backend=’skip’ (since face crops are already provided by RetinaFace) and actions=[’age’]. The raw integer age prediction is binned into decade ranges (e.g., 25 → “20–29”) and then mapped through the Symbol Registry age configuration (Table S1). In the unified evaluation, we use the coarse_grouping preset, yielding a 5-symbol age vocabulary.

Dataset Specific Parameter Effects

While the main paper reports VISTA’s performance using a unified parameter setting for fair benchmarking, we present a marginal sensitivity analysis here to demonstrate the framework’s robustness across different visual environments. Tables S4, S5, and S6 display the marginal effects of individual hyperparameters on TBBT, Buffy, and Sherlock, respectively.

Attribute	Raw Label	Symbolic Token
Gender	Male	<gender_male>
	Female	<gender_female>
Clothing	T-shirt	<clothing_tshirt>
	Jacket	<clothing_jacket>
	Shirt	<clothing_shirt>
	Sweater	<clothing_sweater>
	Dress	<clothing_dress>
	Coat	<clothing_coat>
	Vest	<clothing_vest>
Hair	Curly Hair	<hair_curly>
	Straight Hair	<hair_straight>
	Bald	<hair_bald>
	Short Hair	<hair_short>
	Long Hair	<hair_long>

Parameter	Value	WCP / NMI / ID_Acc	Bal. Avg.
age	broad_adult	96.66 / 87.57 / 89.40	91.21
age	coarse_grouping	96.70 / 87.15 / 88.58	90.81
age	fine_grained	96.60 / 87.01 / 88.68	90.76
ema_rep	0.95	97.47 / 89.43 / 91.32	92.74
ema_rep	0.75	95.84 / 85.05 / 86.46	89.11
queue	300.0	96.97 / 87.65 / 89.17	91.26
queue	150.0	96.34 / 86.83 / 88.60	90.59
vishigh	0.75	96.65 / 87.24 / 88.89	90.93
vishigh	0.825	96.65 / 87.24 / 88.89	90.93
vishigh	0.9	96.65 / 87.24 / 88.89	90.93
vislow	0.275	96.64 / 87.30 / 88.97	90.97
vislow	0.2	96.65 / 87.21 / 88.88	90.91
vislow	0.35	96.67 / 87.20 / 88.82	90.90

Parameter	Value	WCP / NMI / ID_Acc	Bal. Avg.
age	coarse_grouping	84.85 / 75.24 / 76.98	79.02
age	broad_adult	84.18 / 74.50 / 76.02	78.23
age	fine_grained	81.62 / 72.17 / 73.49	75.76
ema_rep	0.95	85.18 / 77.22 / 79.68	80.69
ema_rep	0.75	81.87 / 70.67 / 71.24	74.59
queue	150.0	84.70 / 75.01 / 76.31	78.67
queue	300.0	82.39 / 72.93 / 74.67	76.66
vishigh	0.825	84.67 / 75.04 / 76.57	78.76
vishigh	0.9	84.67 / 75.04 / 76.57	78.76
vishigh	0.75	81.31 / 71.85 / 73.36	75.51
vislow	0.35	84.60 / 75.08 / 76.72	78.80
vislow	0.2	84.86 / 75.05 / 76.48	78.80
vislow	0.275	81.20 / 71.79 / 73.30	75.43

Overall, the standard deviation of metrics across these reasonable hyperparameter ranges remains small. We consistently observe that a higher cluster representation smoothing momentum (ema_rep = 0.95; named ema in an earlier version, distinct from the GMM threshold smoothing coefficient $\eta_{\mathrm{gmm}}$) is beneficial for preventing abrupt threshold oscillation in online scenarios. Furthermore, challenging datasets with darker scenes (e.g., Buffy) favor a larger GMM sliding window (queue = 300) to gather enough diverse samples for stable threshold estimation, whereas visually stable datasets (e.g., TBBT) can adapt faster with a smaller queue. Note that the $\tau_{\mathrm{high}}$ spread in these tables reflects simultaneous co-variation of ema_rep and queue size; these marginal tables are not a one factor test of $\tau_{\mathrm{high}}$ alone. A dedicated $3 \times3$ warm up threshold grid with all other settings fixed is reported in § 3.12.

Semantic Arbitration Invocation Rate Visualization

Figure S1 visually reinforces the high efficiency of VISTA, complementing the raw statistics reported in the main paper. Across all three datasets, the triage mechanism successfully limits semantic arbitration by routing the vast majority of frames (86.5%-90.8%) through the fast, visual only pipeline. As clarified in the main paper (§ 2.3.6), CLIP/DeepFace attributes are extracted up front for every face crop; these rates measure how often the arbitration step fires during association, not how many VLM forward passes are avoided. Notably, TBBT requires the least arbitration (9.2%), consistent with its well lit studio conditions. Conversely, Buffy triggers the highest rate (13.5%) due to frequent low illumination scenes and fast motion that push more track cluster pairs into the ambiguity zone.

Semantic Arbitration Weight Sensitivity

Sherlock falls between the two (11.4%) despite being the hardest dataset overall, suggesting that its difficulty stems more from extreme long range appearance variation (e.g., character disguises) than from persistent local visual ambiguity.

To systematically assess the robustness of the semantic arbitration function introduced in the main paper, we perform a comprehensive grid search over 243 configurations covering all five semantic weight dimensions:

Parameter	Value	WCP / NMI / ID_Acc	Bal. Avg.
age	broad_adult	80.16 / 46.69 / 58.51	61.79
age	fine_grained	80.23 / 46.68 / 58.08	61.66
age	coarse_grouping	80.38 / 46.67 / 55.58	60.87
ema_rep	0.95	81.88 / 49.36 / 61.54	64.26
ema_rep	0.75	78.63 / 44.00 / 53.24	58.62
queue	150.0	80.04 / 46.72 / 58.56	61.77
queue	300.0	80.47 / 46.64 / 56.22	61.11
vishigh	0.75	80.25 / 46.68 / 57.39	61.44
vishigh	0.825	80.25 / 46.68 / 57.39	61.44
vishigh	0.9	80.25 / 46.68 / 57.39	61.44
vislow	0.2	80.41 / 46.82 / 57.69	61.64
vislow	0.275	80.27 / 46.74 / 57.57	61.53
vislow	0.35	80.08 / 46.48 / 56.92	61.16

Overall stability. Table S7 reports the minimum, maximum, and standard deviation of WCP and NMI across all 243 configurations for each dataset. The remarkably low standard deviations (e.g., only 0.31 for WCP on TBBT) confirm that VISTA’s clustering gains are not the product of fragile weight tuning.

Dataset	Metric	Min	Max	Std
TBBT	WCP	96.08	97.59	0.31
	NMI	90.20	91.79	0.27
Buffy	WCP	84.71	90.34	1.09
	NMI	76.86	80.96	0.74
Sherlock	WCP	79.22	83.44	0.78
	NMI	48.90	53.37	0.78

Marginal effects. Table S8 isolates the marginal effect of each specific semantic trait. We observe that persistent traits (gender and age) heavily dictate performance. Specifically, the gender mismatch penalty $\lambda_{\mathrm{gm}}$ and gender+age reward $\lambda_{\mathrm{ga}}$ exhibit the largest influence on the Balanced Average. In contrast, transient cues like clothing

Weight	Value	TBBT	Buffy	Sherlock
$\lambda_{\mathrm{ga}}$	0.04	93.88	83.53	65.22
$\lambda_{\mathrm{ga}}$	0.08	93.81	82.42	65.70
$\lambda_{\mathrm{ga}}$	0.12	94.01	82.40	66.29
$\lambda_{\mathrm{cloth}}$	0.02	93.93	82.82	65.63
$\lambda_{\mathrm{cloth}}$	0.05	93.88	82.81	65.76
$\lambda_{\mathrm{cloth}}$	0.08	93.89	82.72	65.82
$\lambda_{\mathrm{gm}}$	0.08	93.95	82.74	65.64
$\lambda_{\mathrm{gm}}$	0.12	93.93	82.77	65.63
$\lambda_{\mathrm{gm}}$	0.16	93.82	82.84	65.95
$\lambda_{\mathrm{am}}$	0.01	93.94	82.68	65.77
$\lambda_{\mathrm{am}}$	0.03	93.88	82.76	65.72
$\lambda_{\mathrm{am}}$	0.05	93.88	82.90	65.73
$\lambda_{\mathrm{hc}}$	0.02	93.93	82.90	65.73
$\lambda_{\mathrm{hc}}$	0.04	93.88	82.73	65.72
$\lambda_{\mathrm{hc}}$	0.06	93.89	82.72	65.77

reward $\lambda_{\mathrm{cloth}}$ and hair conflict $\lambda_{\mathrm{hc}}$ are comparatively insensitive, validating VISTA’s decision to decouple transient and persistent traits.

Robustness over absolute peaks. It is worth noting that the default weights used in the main paper’s unified evaluation (e.g., $\lambda_{\mathrm{gm}} = 0.12, \lambda_{\mathrm{cloth}} = 0.04$) were deliberately selected as a robust global compromise rather than absolute dataset specific peaks. While Table S8 shows that tuning these weights per dataset could yield marginally higher scores (e.g., Buffy peaks at $\lambda_{\mathrm{ga}} = 0.04$ while Sherlock peaks at 0.12), the unified setting ensures strong out of the box generalization. The minimal performance variance across these settings ultimately proves that VISTA’s effectiveness stems from its core architectural design symbolic denoising and spatio temporal triage rather than fragile hyperparameter overfitting.

GMM Bimodal Fitting Visualization

To empirically validate the bimodal assumption underlying VISTA’s dynamic thresholding (§ 2.2.3 of the main paper), we visualize the GMM fit across all 15 episodes in Figure S2. The cosine similarity scores between track cluster pairs are aggregated per dataset and fitted with a two component GMM.

All three datasets exhibit clear bimodal structure, with the same identity component concentrated at $\mu\approx0.82$–$0.86$ and the different identity component at $\mu\approx0.58$–$0.60$. TBBT shows the tightest same identity distribution ($\sigma=0.06$), consistent with its controlled studio lighting. Buffy and Sherlock exhibit wider spread ($\sigma=0.08$), reflecting greater illumination and appearance variation. The estimated $\tau_{\mathrm{low}}$ and $\tau_{\mathrm{high}}$ thresholds consistently partition the score space into well separated reject, ambiguity, and merge zones. The raw GMM derived $\tau_{\mathrm{low}}$ values ($\mu_1+\sigma_1 \approx0.70$–$0.75$) are bounded via the online proposal rule $\tau_{\mathrm{low}}^{*}=\operatorname{clip}(\mu_1+\sigma_1,0.15,0.50)$, followed by EMA smoothing with $\eta_{\mathrm{gmm}}=0.3$ (§ 2.2.3 of the main paper). The 0.50 cap prevents the lower boundary from drifting too high in visually compressed windows, preserving an ambiguity zone in which semantic arbitration can operate; it is an intentional design choice rather than a fitting artifact. In an isolated online cap sweep over all 15 episodes, relaxing the cap to 0.70 or 1.00 yields identical WCP/NMI/ID_Acc after rounding to two decimals: once the ambiguity zone is preserved, downstream best assignment selection and the arbitration gate $\tau_{\mathrm{merge}}$ dominate the final accept/reject decisions. The cap therefore acts as a declared stability safeguard rather than a tuned performance knob.

Intra Shot Tracklet Length Ablation

Table S9 expands upon the tracklet length ablation discussed in § 2.3.4 of the main paper. As shown, increasing the maximum intra shot tracklet length from 12 to 48 frames improves Identity Accuracy (ID_Acc) and NMI by accumulating richer temporal context and producing more stable running average embeddings.

Length	WCP	NMI	ID_Acc	Precision	Recall	F1
12	92.27	77.39	77.88	81.16	96.19	87.84
24	92.83	77.92	79.14	81.16	96.19	87.84
48	91.89	78.72	81.05	81.16	96.19	87.84
96	91.23	78.50	80.79	81.16	96.19	87.84

However, over extending the window to 96 frames begins to degrade performance due to intra track appearance noise (e.g., characters turning completely away from the camera). Note that Precision, Recall, and F1 remain perfectly constant across these variations. This is expected, as the upstream face detection bounding boxes and initial track candidate generation are fixed; the tracklet length parameter only dictates downstream cluster aggregation and inter shot tracking logic. Based on this trade off, we adopted $L_{\max} = 24$ as our unified default to maintain high Weighted Cluster Purity (WCP) without sacrificing latency.

Table S9: Intra shot tracklet length ablation, averaged over all 15 episodes (WCP / NMI / ID_Acc / Precision / Recall / F1 %). Upstream detection metrics (Precision, Recall, F1) are naturally unaffected by downstream aggregation logic.

Additional Qualitative Results

Figure S3 provides the crop level qualitative comparison on Sherlock referenced in § 2.3.5 of the main paper (moved here for space). Under challenging pose, illumination, and motion blur variations, VISTA maintains higher local purity than COFC and minimizes identity confusion between visually similar characters.

Additional Online Baselines: Full Results

This section provides the full per dataset results for the two additional online baselines summarised in § 2.3.3 of the main paper. Both baselines use the same detector (RetinaFace), embeddings (FaceNet), tracks, and evaluation protocol as VISTA and COFC.

Naive VLM hard constraint baseline. Raw VLM attributes (no SSR, no triage) are applied as must link/cannot link rules. Table S10 shows severe over fragmentation: the number of predicted clusters exceeds ground truth by 4.0–11.3×. This inflates WCP while ID_Acc collapses far below VISTA Full (93.94/82.82/68.10 on TBBT/Buffy/Sherlock) and below the weakest geometry driven ablation in the main paper, confirming that VISTA’s gains stem from selective semantic triage rather than from VLM attributes alone.

Hard-VLM	WCP	NMI	ID_Acc	#Pred/ep	#GT/ep
TBBT	97.98	76.57	63.95	83.8	20.5
Buffy	91.01	72.99	60.64	108.5	27.0
Sherlock	84.67	44.98	40.65	136.0	12.0

Table S10: Naive VLM hard constraint baseline (%). #Pred/ep and #GT/ep are per episode averages by series.

Sequential online HAC baseline. A sequential online HAC variant on the same tracks reaches very high WCP, as expected under fragmentation, but substantially lower NMI and ID_Acc, with severe over fragmentation (Table S11).

Online HAC	WCP	NMI	ID_Acc	#Pred/#GT
TBBT	99.73	84.93	81.29	4.5×
Buffy	97.62	76.78	63.49	9.5×
Sherlock	91.19	49.28	40.56	39.8×

Table S11: Sequential online HAC baseline (%). The last column is the ratio of predicted clusters to ground truth clusters per episode.

Additional Threshold Sensitivity Analyses

Warm up threshold grid. Complementing the marginal tables in § 3.5, we run a dedicated $3 \times3$ grid bracketing the warm up initialisation over $\tau_{\mathrm{high}}^{(0)} \in\{0.75, 0.825, 0.90\}$ and $\tau_{\mathrm{low}}^{(0)} \in\{0.20, 0.275, 0.35\}$ with all other settings fixed (135 episode level evaluations). Table S12 reports the per dataset max–min variation, which remains below 0.7 pp for all metrics. $\tau_{\mathrm{high}}$ shows limited sensitivity in this grid precisely because GMM dynamic thresholding overrides the static initialisation after warm up.

Table S12: Max–min variation (pp) across the $3 \times3$ warm up threshold grid (all other settings fixed).

Metric	TBBT	Buffy	Sherlock
WCP	0.18	0.15	0.67
NMI	0.41	0.15	0.30
ID_Acc	0.67	0.37	0.69

$\tau_{\mathrm{merge}}$ sensitivity. We sweep the arbitration acceptance gate over $\tau_{\mathrm{merge}} \in\{0.280, 0.325, 0.375, 0.425, 0.475\}$. The sweep confirms the expected purity–continuity trade off: at the most conservative setting (0.475), Sherlock ID_Acc drops by 8.37 pp relative to the 0.325 default, reflecting conservative fragmentation rather than a standalone visual threshold failure.

$\tau_{\mathrm{low}}$ cap sweep. As reported in Section S8, an isolated online cap sweep over all 15 episodes shows that relaxing the 0.50 cap on $\tau_{\mathrm{low}}^{\star}$ to 0.70 or 1.00 yields identical WCP/NMI/ID_Acc after rounding to two decimals, confirming that the cap is a declared stability safeguard rather than a tuned performance knob.

Subgroup Clustering Quality

Complementing the triage rate analysis in the main paper (§ 2.3.7), Table S13 reports subgroup clustering quality gaps on the IoU matched face subset, using the same VLM predicted gender/age groups. VPCD provides no ground truth demographic annotations, so these figures should be read as a diagnostic rather than a full fairness audit. The signs of the gaps vary across datasets, indicating character and dataset dependent effects rather than a one directional degradation. As an ethical safeguard, our implementation provides an option to disable the gender/age attributes entirely, in which case arbitration falls back to transient cues and geometry only.

Gap	Dataset	WCP	NMI	ID_Acc
Gender (M−F)	TBBT	−0.5	+13.1	+1.7
Gender (M−F)	Buffy	+2.1	+14.7	+5.0
Gender (M−F)	Sherlock	−6.6	−19.8	−1.4
Age (adult−senior)	TBBT	+2.0	+12.0	+13.4
Age (adult−senior)	Buffy	−2.3	+4.7	−0.6
Age (adult−senior)	Sherlock	−6.6	+13.4	+1.1