betaFeedback

WorksComputer Science

EDA: Observe, Verify, and Assess for Evidence-Grounded Multimodal Depression Analysis

Abstract

Existing multimodal depression assessment methods often summarize long recordings into global behavioral descriptions. This leaves the source and decision relevance of individual observations difficult to inspect. We propose EDA, an evidence-grounded framework organized around Observe, Verify, and Assess. In Observe, literature-informed acoustic and visual cues are localized into source-linked Evidence Records with temporal intervals and behavioral measurements. In Verify, frozen multimodal agents examine whether the cited media support each observation. The simplified Support admission policy retains positively verified records and their review grounds. In Assess, statistical descriptors and temporal events support reference-based behavioral comparison, while PHQ-9-guided text analysis provides complementary symptom evidence. Cross-modal disagreement triggers review of the original text before final selection. Experiments on D-Vlog and LMVD evaluate human-referenced cue agreement and downstream assessment utility separately. AI cue annotations achieve 91.02% accuracy against human consensus labels on shared source segments. With a frozen 7B backbone, EDA Support improves F1 over direct prediction by 7.61 and 9.88 percentage points on D-Vlog Leave-Query-Out and LMVD, respectively. These findings support explicit, source-linked behavioral evidence for inspectable multimodal depression assessment.

Introduction

Video-based depression assessment uses both spoken content and observable behavior. Transcripts can describe symptoms and personal experiences. Vocal expression, facial activity, and head movement provide additional information [1, 5, 18]. In long recordings such as D-Vlog and LMVD [10, 31], these behaviors vary over time and across contexts. A useful assessment must identify relevant cues, locate their source, and explain how they inform the final judgment.

Current methods address different parts of this task. Fusion models learn acoustic–visual interactions [24, 30], while symptom-oriented language models organize transcript passages by depression-related domains [33, 36]. DepressionAgent adds modality-specific extraction, support–challenge deliberation, and cross-modal reflection [41]. Yet a recording-level evidence explanation still condenses a long video into a short narrative (Figure 1a). It tells readers what the model observes. A statement such as frequent pauses names a behavior, but does not identify its source intervals. Readers still need to know where/when it occurs and how it affects the assessment. This leads to three linked questions. Observe: What behavioral evidence is observable? Acoustic and visual behaviors need reproducible definitions and source intervals. Verify: Is the observation supported by its source? The cited media and measurements must support each candidate before it is retained. Assess: How should verified evidence inform assessment? Behavioral comparison, symptom-related text, and conflict review must have explicit roles in the final judgment.

Figure 1. Motivation. From global descriptions to cue-grounded multi-agent assessment.

We introduce EDA around Observe, Verify, and Assess. Observe operationalizes five acoustic and four visual cues from prior literature. Their annotations use no depression labels and retain source intervals and measurements. Verify checks these records against the original media. Its Evidence mechanism combines Support verification, Counter challenge, and cue-level adjudication, retaining the grounds for admission or exclusion. The optimized Support policy admits positively verified records directly and forms our final model. Assess compares retained behavioral records with labeled references, using statistical descriptors and temporal events. In parallel, PHQ-9-guided text agents examine symptom candidates and their context. Cross-modal disagreement triggers textual source review and an external acceptance gate. Thus, EDA connects each observation to its source, verification, and subsequent use.

Our experiments examine each step. Cue distributions show that the same behavior has different associations with depression across datasets. Cue interpretation therefore depends on the dataset. AI cue annotations achieve 91.02% accuracy against human consensus labels, pooled across D-Vlog and LMVD. This check evaluates cue agreement on shared source segments. For final assessment, 7B EDA Support improves F1 over direct prediction by 7.61 points in the D-Vlog reference setting and 9.88 on LMVD. In the corresponding D-Vlog comparison with a larger-model baseline, accuracy and F1 improve by 1.87 and 1.30 points. Admission comparisons support the optimized Support mechanism. Representation controls show that events need not improve classification, although their source-linked timing remains inspectable. Evidence quality and assessment utility are therefore evaluated separately. Our contributions are:

  • Observe: construct source-linked behavioral evidence. We operationalize five acoustic and four visual cues with literature-informed, label-independent criteria. Evidence Records retain each observation’s temporal anchors and measurements.

  • Verify: ground evidence admission in its source. We develop an explicit verification mechanism with source-based review grounds. The simplified Support policy optimizes admission while preserving this traceability.

  • Assess: use evidence in multimodal assessment. We combine behavioral reference comparison, PHQ-9-guided text analysis, and disagreement-triggered review. Cue agreement, admission behavior, and assessment utility are evaluated separately.

Related Work

Multimodal and temporal depression modeling. Speech and visible behavior inform depression assessment [1, 5, 18]. Multimodal models learn feature interactions across unaligned sequences [25, 32]. Depression-specific methods study temporal fusion, instance selection, and multiscale representations [9, 12, 20, 23, 24, 39, 40]. Recent work adds state-space modeling and behavioral interactions [14, 21, 30, 37, 38]. EDA instead makes the underlying cue records and their source support explicit.

Symptom evidence and agent deliberation. PHQ-9 defines symptom domains [11]. Question-wise Modality Fusion, RED, and Explain-MDRC organize interview content around symptoms and supporting passages [16, 33, 36]. Dep-LIM and AI Psychiatrist Assistant use evidence and specialized reasoning roles for assessment [6, 13]. Multimodal language-model studies extend this analysis to speech, vision, and text [34, 35]. DepressionAgent is closely related. It combines modality-specific extraction, support–challenge deliberation, arbitration, and reflection [41]. EDA develops source-linked cue records and optimizes their admission through Support.

Perception, grounding, and source verification. VideoAgent, VideoTree, and DrVideo organize long-video content for language-model reasoning [15, 26, 27]. VideoGLaMM studies fine-grained visual grounding [17]. OPPO distinguishes cue use from input-supported emotion reasoning and uses reinforcement learning to improve perception [8]. EDA verifies source-linked cue records with frozen models and uses Support to optimize admission. Debate and verification methods also examine alternative interpretations [3, 22]. Its record structure separates source verification from later decision use.

Observe: Fine-Grained Behavioral Cue Annotation

EDA defines which behaviors the audio-visual workflow should observe. We derive cue concepts from the literature, specify observable criteria, and annotate their time intervals. These annotations describe each behavior independently of the final depression label.

Literature-informed Cue Definitions

Prior behavioral studies guide the cue inventory. Speech studies inform low loudness variation [7], slow pitch change [19, 42], slow articulation [2], long pauses [18, 29], and repeated long pauses [1]. Visual-behavior studies inform infrequent lip-corner lowering, frequent lip-corner tightening, and limited head movement [5], as well as downward head posture [4]. These studies inform five acoustic and four visual cue definitions (Figure 2). We retain the original nonconsecutive visual identifiers for record consistency.

Figure 2. EDA: Observe, Verify, and Assess. Evidence Records connect source-linked cues, agent verification, and multimodal depression assessment.

The same definitions apply to both depression-label groups. Annotators identify the cue and its temporal extent without using the recording’s depression label. Labels enter only the later reference comparison and distribution analysis. This separates the annotation of a behavior from its interpretation in depression assessment. The supplement lists the full criteria and their literature basis.

Operationalization and Annotation Records

We construct fine-grained behavioral cue annotations through automatic interval localization and source-media review. OpenFace extracts facial action units and head pose, while FSMN-VAD identifies speech activity intervals. Cue selection uses pre-segmented 3-second audio and video windows with cue-specific measurement criteria. Adjacent windows with consistent cue assignments are merged. Source review checks the corresponding media, surrounding context, and within-recording baselines. Each record contains the recording identifier, cue label, onset, offset, source excerpt, and measurements. This Evidence Record links the source interval, cue assignment, measurements, verification grounds, and subsequent use. From these records, we construct statistical descriptors and temporal events to characterize aggregate behavioral patterns and local occurrences, respectively. The admission policies in § 4.3 determine which records enter behavioral comparison and retain the reasons for inclusion or exclusion. Missing media, failed tracking, and unavailable review are recorded separately from cue absence, distinguishing unavailable evidence from an unobserved behavior.

Retained Cue Coverage by Dataset and Depression Label

Figure 3 counts recordings with at least one retained annotation for each cue after Evidence adjudication. The bars separate datasets and depression-label groups. Each recording contributes at most once per cue, and cues can co-occur. These are recording counts, not interval totals or cue-detection recall. Depression labels serve only to group the resulting records in this analysis. The supplement reports media durations and admission totals.

Numbers of depression-related cue annotations

Figure 3. Numbers of depression-related cue annotations

EDA: Observe, Verify, and Assess

Overview and Evidence States

EDA organizes source-linked cue annotation, source verification, and assessment as Observe, Verify, and Assess (Fig. 2). It uses frozen models for source review and prediction. For a recording Xq=(XqT,XqA,XqV)X_q = (X_q^T, X_q^A, X_q^V), it predicts a binary depression label y^q\hat{y}_q. Text agents analyze symptom candidates under PHQ-9 domains. In parallel, audio-visual agents check cue annotations and compare retained records with labeled references. EDA then matches the two assessments. Disagreement triggers textual review and an external acceptance gate. The Evidence mechanism defines the source-linked record and its verification process. Its extended review adds contextual challenge and cue-level adjudication. We further optimize evidence admission through Support, which retains positively verified records directly. EDA Support combines this policy with the same reference comparison and text–AV coordination and is our final model. The framework reuses the originally blurred D-Vlog clip from InertiaMPDD. Its cue and review examples are schematic.

The evidence package retains original text passages, admitted behavioral events, and review grounds. For a candidate behavioral event,

hi=(mi,ci,τi,Xq,τimi,Fi),τi=[tis,tie],(1)h_i = (m_i,c_i,\tau_i,X_{q,\tau_i}^{m_i},F_i), \qquad\tau_i = [t_i^s,t_i^e], \tag*{(1)}

mi∈{A,V}m_i \in\{A,V\} identifies the modality, and cic_i identifies the cue. Xq,τimiX_{q,\tau_i}^{m_i} is the source excerpt, and FiF_i contains descriptors derived from the annotation. A candidate record proposes a cue at a specific interval. A source-reviewed record adds review judgments and their grounds. Admitted evidence contains records selected for comparison. These states show how an observation enters assessment. They do not establish that the observation is correct or relevant.

PHQ-9-guided Textual Evidence Analysis

Text agents first retrieve symptom candidates from the original transcript using the nine PHQ-9 domains [11]. Each passage records its symptom topic, temporal status, and semantic relation. These fields distinguish current personal experience from history, denial, third-party statements, and general discussion. For each domain, agents collect supporting passages, counter-evidence, and qualifying context. The assessment role considers these records alongside personal-status information and the original transcript. An internal text review checks the initial judgment. The workflow returns y^T\hat{y}_T, a rationale, and citations to original passages. It interprets symptom evidence without assigning questionnaire scores.

The audio-visual workflow runs in parallel. It receives the cue annotations from § 3 and checks each candidate against the original audio or video. It then computes a separate behavioral assessment. Neither cue annotation nor source review receives the recording’s depression label.

Source Verification and Evidence Admission

Support, Counter, and Evidence use the selected frozen multimodal model. Each role checks a different aspect of the candidate and records its judgment and source-based grounds.

Support: check the observation against media and annotations. For event ii, the Support role receives the source excerpt, cue definition, and cue annotations:

RiS=AS(Xq,τimi,ci,Fi)=(viS,OiS,GiS).(2)R_i^S = \mathcal{A}_S(X_{q,\tau_i}^{m_i},c_i,F_i) = (v_i^S,O_i^S,G_i^S). \tag*{(2)}

OiSO_i^S describes the observed phenomenon. The judgment viSv_i^S states whether the source supports that phenomenon under the cue definition. GiSG_i^S gives the justification. This keeps the observation separate from the review judgment.

Counter: test local alternatives. The Counter role receives the Support record and an expanded interval τi+\tau_i^+:

RiC=AC(Xq,τi+mi,ci,Fi,RiS,GiC).(3)R_i^C = \mathcal{A}_C(X_{q,\tau_i^+}^{m_i},c_i,F_i,R_i^S,G_i^C). \tag*{(3)}

The context card GiCG_i^C compares the core interval with its surroundings. It also supplies within-recording baselines and observation-quality information. Counter checks whether the pattern persists and differs from the person’s baseline. It then examines source or annotation inconsistencies as alternative explanations. The output records the counter-evidence relation, issue type, and observation. Technical failure is recorded as unavailable review rather than evidence against the behavior.

Evidence: adjudicate the cue record. For cue cc, the Evidence role considers its candidate records Hc\mathcal{H}_c and both review sets:

acE=AE(Hc,RcS,RcC)∈{admit,block}.(4)a_c^E = \mathcal{A}_E(\mathcal{H}_c,\mathcal{R}_c^S,\mathcal{R}_c^C) \in\{\text{admit},\text{block}\}. \tag*{(4)}

Evidence checks whether support matches the cue definition and source. It also decides whether the challenge weakens that interpretation. Records for the same cue are reviewed together. One problematic interval therefore need not invalidate all other occurrences. The output retains the grounds for admission or exclusion. The supplement shows a schematic example.

The extended Evidence configuration uses all three review roles. Building on this evidence-construction design, the optimized Support policy retains positively verified records without the later challenge and adjudication stages. It preserves the source interval, measurements, Support judgment, and verification grounds. Thus, the final EDA Support model keeps an inspectable evidence record while simplifying admission. We compare the two policies to assess their downstream utility.

From Cue Records to Behavioral Assessment

After admission, the behavioral workflow summarizes the retained records for comparison at the recording level. It uses aggregate descriptors and temporal events. For cue cc in recording qq, a sample descriptor fq,csf_{q,c}^s summarizes annotation count, event frequency, duration, interval coverage, and inter-event spacing. The event representation

Gq,c={(tis,tie,fi)}i=1Nq,c(5)\mathcal{G}_{q,c} = \{(t_i^s,t_i^e,f_i)\}_{i=1}^{N_{q,c}} \tag*{(5)}

retains each event’s boundaries and annotation-derived descriptor. Aggregate descriptors summarize overall cue patterns for class-reference comparison. Temporal events retain occurrence times, durations, and links to local context. These fields support source inspection and can contribute complementary comparison information. Their predictive value is evaluated empirically. We compare Statistical Descriptors, Temporal Events, and Joint Statistical–Event Representation. They isolate aggregate descriptors, localized temporal events, and their combined use, respectively.

The reference pool contains same-language recordings from both classes. It excludes the query and its label. Benchmark test queries use training references only. Leave-Query-Out also permits other eligible labeled recordings, as defined in § 5.1. Reference membership and normalization change for each query. Model parameters remain frozen.

Statistical distances use robust scaling based on reference medians. Symmetric event-set distances compare timing and annotation-derived features. The supplement specifies the scaling, event comparison, and missing-comparison rules.

For modality setting M∈{A,V,AV}M \in\{A,V,AV\}, let CM(q)\mathcal{C}_M(q) contain cues with valid information. Let Bq,c\mathcal{B}_{q,c} contain feature blocks that allow comparison with both classes. The class distance is

DyM(q)=1∣CM(q)∣∑c∈CM(q)1∣Bq,c∣∑b∈Bq,cδy,c,b(q)(6)D_y^M(q)=\sqrt{\frac{1}{|\mathcal{C}_M(q)|}\sum_{c\in\mathcal{C}_M(q)}\frac{1}{|\mathcal{B}_{q,c}|}\sum_{b\in\mathcal{B}_{q,c}}\delta_{y,c,b}(q)} \tag*{(6)}

Here δy,c,b\delta_{y,c,b} is the statistical or event-level contribution of block bb. We average valid blocks within each cue, then average across cues with equal cue weights. The prediction is y^M=1[D1M(q)<D0M(q)]\hat{y}_M=\mathbf{1}[D_1^M(q)<D_0^M(q)]. Distance ties go to class 0. Missing blocks are excluded. If no valid two-class comparison is available, we record an abstention or completion outcome separately.

Assessment Matching and Conflict Review

EDA compares the text and behavioral predictions. If y^T=y^AV\hat{y}_T=\hat{y}_{AV}, it retains their shared label. Otherwise, a text reviewer examines the original transcript, its context, and two anonymized textual judgments. It receives neither the AV label nor the dataset label. It returns a revised label y^R\hat{y}_R, self-reported confidence cc in the reviewed label, a rationale, and original-text citations. The behavioral branch determines when to review. It does not supply the answer to the reviewer.

For a conflict, the external gate selects the reviewed text judgment when its confidence exceeds 0.8. Otherwise, it uses the AV assessment:

y^={y^R,vR=1, c>0.8,y^AV,otherwise.(7)\hat{y} = \begin{cases} \hat{y}_R, & v_R=1,\ c>0.8,\\ \hat{y}_{AV}, & \text{otherwise}. \end{cases} \tag*{(7)}

The validity indicator vRv_R requires an admissible review citing the original text. The reviewed label need not agree with AV. The value cc is confidence in that reviewed label, not the probability of depression, and is not assumed to be calibrated. We measure beneficial and harmful revisions and compare them with retaining T. Missing-text completion is analyzed separately to distinguish its effect from that of behavioral evidence.

Experiments

Datasets, Reference Construction, and Metrics

Evaluation protocols. We evaluate on D-Vlog and LMVD [10, 31]. D-Vlog test follows the original benchmark split. Its reference pool uses labeled training recordings only and excludes validation and test recordings. For LMVD, we sort source identifiers and apply stratified ten-fold evaluation with shuffle and seed 42. This follows the documented BHDD-style construction [14]. Each fold builds its references and normalization from the other nine folds. We pool held-out predictions to compute the metrics.

We also evaluate D-Vlog under Leave-Query-Out. The reference pool uses other eligible same-language recordings and their labels. It excludes the assessed recording and its label. This is leave-one-recording-out reference construction, not leave-one-query-out cross-validation. The model stays frozen and is not retrained for each recording. Only reference membership and derived statistics change. This setting permits labels of other recordings in the assessed cohort. Benchmark testing permits training labels only. We report the two settings separately. Consistent with DepressionAgent [41], D-Vlog workflow and ablation tables prioritize Leave-Query-Out. Table 1 and the supplement retain test comparisons with training references only.

MethodYearD-Vlog test Acc.D-Vlog test Prec.D-Vlog test Rec.D-Vlog test F1LMVD Acc.LMVD Prec.LMVD Rec.LMVD F1
Depression Detector [31]2022–65.4065.5763.50––––
TAMFN [39]2023–66.0266.5065.82––––
Attention-MIL [20]2024–67.2767.7766.64––––
Depressformer [9]202465.0064.0054.0059.00––––
DepMamba [30]202568.8768.1986.9976.4472.1370.1876.5673.20
CMF-Mamba [37]2025––––74.8673.2078.0275.53
STCM-Mamba [38]202572.6473.7282.1177.6975.4172.5576.6876.49
MSI-Net [21]202670.2578.9169.6373.9174.3478.8075.2977.06
MDDFormer [10]2026––––76.8877.0276.8876.85
BHDD [14]2026–75.9875.9675.9478.1178.3978.1178.06
DepressionAgent [41]202695.2695.2496.1595.6982.5780.4885.7683.04
EDA Support (7B)–95.3297.6793.3395.4583.1085.3879.7482.46

Table 1. Matched-test comparison with published methods (%).

Human cue-annotation check. Three affective-computing researchers independently annotate shared 3-second source segments without seeing AI outputs. Unanimous cue labels are retained. Disputed cues are reviewed with two psychological counseling instructors and resolved by five-person binary majority voting. AI cue annotations achieve 91.02% accuracy against the human labels, pooled across D-Vlog and LMVD. This measures cue agreement on common segments, separately from classification. The supplementary human-validation protocol gives the voting procedure and metric scope.

Backbones and metrics. The primary implementation uses frozen Qwen2.5-Omni-7B [28]. Evidence denotes the extended support–challenge–adjudication configuration. The final EDA Support model optimizes admission while retaining source verification and the shared downstream workflow. We compare direct prediction, DepressionAgent [41], and EDA at 7B and 30B-A3B. The 7B DepressionAgent row adapts its workflow to the smaller backbone. Metrics are accuracy, precision, recall, and F1 with depression as the positive class. Unclassified outputs count as incorrect in accuracy. All differences are percentage points. Best classification scores are bold within comparable groups, including ties. Dashes denote unreported results.

Observe: Behavioral Cue Characterization

Cue annotations replace a video-level account with named records and source intervals. Figure 3 shows that cue occurrence has different associations with depression across datasets. Retained downward head posture (V05) appears in 92.86% of depression-labeled and 81.97% of non-depression-labeled D-Vlog recordings. The corresponding LMVD rates are 21.81% and 21.97%. Limited head movement (V07) also shows a larger label-group difference on D-Vlog. These differences make cue interpretation dataset-dependent. EDA preserves each cue and its source interval for reference comparison, rather than assigning it a fixed depression meaning. The retained inventory does not itself measure annotation correctness.

Verify: Source Grounding and Evidence Admission

Evidence quality concerns whether a cue has observable support in its cited source segment. The pooled 91.02% human-referenced cue accuracy checks this agreement. The shared segmentation supplies the source anchors, so this score does not measure independently estimated boundary accuracy. It also does not establish that one admission policy produces more reliable evidence than another.

Assessment utility concerns what the evidence and its use contribute to classification. Table 4 compares Support with extended Evidence review. Cue Retention is the share of initial candidate cue intervals retained by a policy. It measures admission selectivity, not evidence correctness. Support retains 1.26 points more intervals on D-Vlog Leave-Query-Out and 7.40 points more on LMVD. Its AV F1 gains are 1.37 and 0.08 points, while its AVT F1 gains are 0.12 and 0.25 points. The larger retention change on LMVD accompanies the smaller AV gain. Thus, retention and assessment utility need separate evaluation. Support improves prediction while preserving the shared source-verification mechanism.

AdmissionCue Retention (%)AV F1AVT F1
D-Vlog Leave-Query-Out
Evidence review76.3870.2694.38
EDA Support77.6471.6394.50
LMVD
Evidence review45.9470.2782.21
EDA Support53.3470.3582.46

Table 4. Cue retention and assessment utility (%).

Assess: Evidence-Guided Depression Assessment

Comparisons within the 7B backbone. At 7B scale (Table 2), Support improves F1 over direct prediction by 7.61 points on D-Vlog Leave-Query-Out and 9.88 on LMVD. It exceeds 7B DepressionAgent by 6.14 and 5.28 points. Extended Evidence also gains 7.49 and 9.63 points over direct prediction. Thus, the shared evidence-construction framework improves assessment at fixed model scale, with further gains from Support admission.

BackboneWorkflowD-Vlog Leave-Query-Out Acc.D-Vlog Leave-Query-Out F1LMVD Acc.LMVD F1
7BDirect87.5686.8976.4172.58
7BDepressionAgent86.7988.3676.7477.18
7BEDA Support94.3094.5083.1082.46
7BEDA Evidence94.1794.3882.8982.21
30B-A3BDirect81.3584.8774.6079.06
30B-A3BEDA Evidence90.1691.2680.5380.40
30B-A3BDepressionAgent92.4393.2082.5783.04

Table 2. Backbone and workflow comparisons (%).

Reference-setting and benchmark comparisons. On D-Vlog Leave-Query-Out, 7B Support reaches 94.30% accuracy and 94.50% F1, exceeding 30B-A3B DepressionAgent by 1.87 and 1.30 points. Its AV accuracy is 68.91%, compared with 63.84% for DepressionAgent. Table 1 separately reports matched benchmark comparisons. Support gains 0.06 accuracy points and 2.43 precision points on D-Vlog test, and 0.53 and 4.90 on LMVD. Recall and F1 are lower on both benchmarks. Thus, the precision gains involve a recall trade-off. They should be distinguished from the F1 gains under Leave-Query-Out. The supplement retains complete metrics.

Backbone comparison. On D-Vlog Leave-Query-Out, 30B-A3B EDA gains 8.81 accuracy and 6.39 F1 points over direct prediction. However, 7B EDA with the same Evidence policy scores 3.12 F1 points higher. Model scale alone therefore does not determine assessment performance.

Aggregate and temporal evidence. Table 3 fixes the Evidence-admitted AV events. Statistical Descriptors exceed the joint representation by 0.09 F1 points on D-Vlog Leave-Query-Out, while the joint representation gains 0.80 points on LMVD. Joint representation exceeds Temporal Events on both datasets. Descriptors are therefore strong standalone predictors, while event-level gains depend on the setting. The views also serve different purposes: descriptors summarize overall behavior for class-reference comparison; events preserve occurrence times, duration, and local context for inspection.

RepresentationD-Vlog Leave-Query-Out Acc.D-Vlog Leave-Query-Out F1LMVD Acc.LMVD F1
Statistical Descriptors67.3670.3569.2869.47
Temporal Events59.9763.8662.2666.01
Joint Statistical–Event Representation67.1070.2669.5670.27

Table 3. Statistical and temporal AV representations (%).

Effect of review triggering strategies. Table 5 compares triggers under matched review budgets, using the same completed text predictions and review rule. Scores precede the final gate; Random reports the mean and standard deviation over five seeds. AV disagreement exceeds Random by 0.26 accuracy and 0.30 F1 points on D-Vlog Leave-Query-Out, and by 0.69 and 0.91 points on LMVD. It also outperforms text-uncertainty triggering. Thus, cross-modal disagreement provides a more informative review-selection signal in these settings. This comparison concerns which predictions to review, separately from how review outcomes enter the final assessment.

TriggerD-Vlog Leave-Query-Out Acc.D-Vlog Leave-Query-Out F1LMVD Acc.LMVD F1
Completion93.5293.6982.7281.91
93.9194.0882.4781.60
Random±0.09±0.10±0.09±0.10
Uncertainty93.5293.6982.7281.78
AV disagreement94.1794.3883.1682.51

Table 5. Review triggering before the final gate (%).

Conflict review and accepted changes. On LMVD, completion raises accuracy from 81.51% to 82.72%. Source review reaches 83.16%, while the final gate yields 83.10%. Thus, review selection and final acceptance have distinct effects; the gate does not further improve accuracy here. Among changed predictions, 87.50% are beneficial on D-Vlog Leave-Query-Out and 58.97% on LMVD; harmful revisions account for 12.50% and 41.03%. These conditional rates differ from overall accuracy gains. The supplement reports uncertainty intervals. § 6 traces source context and reported decisions.

Case Studies

Figure 4 links source excerpts to subject-level evidence packages for three cases. Each panel shows video frames, an audio excerpt, and transcript passages alongside behavioral ioral measurements, class-reference distances, textual review, and the reported final assessment. The excerpts retain their respective timestamps rather than implying temporal alignment across modalities.

Figure 4. Source-to-assessment case visualizations.

The cases illustrate how textual context affects symptom interpretation. In case (a), reference to high-school experiences weakens the interpretation of the passage as evidence of current symptoms. In case (b), the stated loss of interest concerns a particular game, limiting its support for generalized loss of interest. In case (c), additional sleep and fatigue complaints broaden the evidence beyond the initial appetite-related passage, and the reviewed textual judgment agrees with the behavioral assessment. The displayed review confidences are c=0.70c = 0.70, 0.600.60, and 0.800.80. Each follows the AV fallback under the c>0.8c > 0.8 selection rule. These examples

make source observations, context, and reported decisions inspectable. They do not establish classification accuracy.

Conclusion

EDA addresses the difficulty of inspecting the source and use of recording-level behavioral descriptions through three stages. Observe operationalizes five acoustic and four visual cues with literature-informed, label-independent criteria. Evidence Records retain temporal anchors and behavioral measurements. Verify checks observations against their original media and preserves admission grounds. The simplified Support policy retains positively verified records with the same source traceability. Assess combines behavioral reference comparison with PHQ-9-guided text analysis and disagreement-triggered review. On D-Vlog and LMVD, we evaluate cue agreement and downstream utility separately. Cue annotations achieve 91.02% human-referenced accuracy; at 7B scale, Support improves F1 over direct prediction by 7.61 points under D-Vlog Leave-Query-Out and 9.88 on LMVD. Statistical descriptors are strong predictors, while event-level gains vary by setting. Matched-budget controls support disagreement as a useful review-selection signal. Beyond classification scores, traceable records show how observations enter assessment. Source intervals, measurements, and review grounds let readers distinguish behavioral support from symptom interpretation and inspect unresolved conflicts. These findings encourage explicit analysis and verification of source-media cues alongside representation learning and multimodal fusion. Evidence agreement and prediction performance remain distinct evaluation targets.

recording baselines support review. The original nonconsecutive visual identifiers are retained for consistency. Text passages are annotated separately by PHQ-9 domain, temporal status, and semantic relation. These fields distinguish current personal symptoms from history, denial, third-party experience, and general discussion. They describe semantic evidence and do not assign questionnaire scores.

Retained downward head posture (V05) appears in 92.86% of depression-labeled and 81.97% of non-depression-labeled D-Vlog recordings. The LMVD rates are 21.81% and 21.97%. Limited head movement (V07) also shows greater label-group separation on D-Vlog. These patterns describe annotations retained by Evidence. They do not measure exhaustive cue prevalence or clinical specificity. Excluding an annotation does not establish that the cue is absent. Cue-detection recall requires exhaustive human annotations.

Supplementary Material

Operational Cue Definitions

The inventory contains five acoustic and four visual cues from prior behavioral studies. Table S1 lists each cue’s observable criterion and literature basis. The definitions describe behavior without using the final depression label. Cue identifiers stay consistent across annotations and experiments.

IDObservable cueAnnotation criterionLiterature basis
A01Low loudness variationLimited fluctuation in vocal intensity within a localized speech intervalVocal emphasis/prosody [7]
A02Slow pitch changeGradual pitch changes across voiced speech intervalsPitch dynamics [19, 42]
A03Slow articulationReduced articulation pace in an observable speech segmentSpeech timing [2]
A04Long pauseA sustained silence between speech segments, interpreted in speech contextPause timing [18, 29]
A05Repeated long pausesMultiple prolonged pauses grouped within a local speech contextPausing/fluency [1]
V01Infrequent lip-corner loweringLow occurrence of visibly lowered lip corners across valid facial observationsFacial activity [5]
V03Frequent lip-corner tighteningRepeated visible tightening of the lip corners within the local contextFacial activity [5]
V05Downward head postureSustained downward head orientation in an observable intervalHead/gaze orientation [4]
V07Limited head movementRestricted variation in head orientation across a localized intervalHead-motion dynamics [5]

Table S1. Literature-informed acoustic and visual cue definitions.

Annotation Records and Dataset Variation

Each cue has an observable criterion, a temporal scope, source-validity checks, and interval-merging rules. Each annotation records the recording and cue identifiers, onset, offset, and source excerpt. We derive descriptors from these records for later comparison. Wider context and within-recording baselines support review.

Annotation tools and workflow. We use automatic feature extraction, candidate interval localization, and multi-agent source-evidence review. Visual features are extracted using OpenFace, including facial action units AU15 and AU14, face-tracking confidence, and head pitch and yaw angles. Audio is decoded into a 16 kHz mono signal. FSMN-VAD identifies speech activity intervals, from which we extract RMS loudness, fundamental-frequency trajectories, speech duration, and pause information. Table S2 summarizes the annotation and measurement procedures for each cue.

CueBehavioral annotation or measurement
A01: Low loudness variationCompute RMS loudness in dBFS and summarize its P90–P10 range, standard deviation, interquartile range, and adjacent changes to localize intervals with limited loudness variation.
A02: Slow pitch changeCompute the absolute difference between adjacent valid F0 frames, normalized by their mean F0 and temporal separation, to obtain the relative F0 change rate (%/s) and localize intervals with slow changes.
A03: Slow articulationDivide the number of linguistic units by VAD speech duration to obtain an articulation-rate proxy. English uses words and Chinese uses Han characters; low-rate intervals are localized within the same language and counting unit.
A04: Long pauseUse speech activity boundaries to identify silence between adjacent speech segments within the same response, retaining pauses lasting at least 0.5 s.
A05: Repeated long pausesReuse A04 pause events to localize at least two long pauses separated by valid speech within the same response.
V01: Infrequent lip-corner loweringCompute the proportion of AU15-positive valid frames among all valid AU15 frames to localize intervals with a low occurrence proportion of lip-corner lowering.
V03: High lip-corner tightening proportionCompute the proportion of AU14-positive valid frames among all valid AU14 frames to localize intervals with a high occurrence proportion of lip-corner tightening.
V05: Sustained downward head postureUse frame-level pitch angles and the within-recording posture baseline to localize candidate downward-head-posture intervals lasting at least 1.5 s.
V07: Limited head movementCompute angular-amplitude RMS and angular-velocity RMS separately for pitch and yaw to localize intervals with low movement amplitude and speed in both directions, lasting at least 4 s after merging.

Table S2. Behavioral annotation and measurement procedures.

Evidence Records and Numerical Specification

This section specifies evidence records, cue measurements, source-review context, and reference-comparison rules.

Event provenance. Each acoustic or visual candidate stores its recording identifier, modality, cue identifier, start and end times, and source-media locator. It also stores the computed feature vector and available review outputs. Review records distinguish support, counter-evidence, admission, and technical failure. Unavailable review does not count as a negative observation. Source identifiers can link to restricted recordings without making the media public.

Cue measurement and candidate localization. We obtain behavioral measurements for the nine cues using the feature extraction procedure described above and localize candidate intervals using the criteria in Table S3. Candidates then undergo support review, and retained events form the joint statistical–event representation.

CueMeasurement and candidate thresholdValid-observation criteria
A01: Low loudness variationLoudness P90–P10 ≤min⁡(P25,8 dB)\le\min(P_{25},8\,\mathrm{dB}); SD ≤3 dB\le3\,\mathrm{dB}; IQR ≤4 dB\le4\,\mathrm{dB}; maximum adjacent change ≤6 dB\le6\,\mathrm{dB}Speech proportion ≥0.7\ge0.7
A02: Slow pitch changeMean relative F0 change rate ≤P25\le P_{25} (%/s)Valid-F0 proportion ≥0.6\ge0.6; at least 3 baseline windows
A03: Slow articulationLinguistic units per VAD speech second ≤P25\le P_{25}; candidate duration ≥1 s\ge1\,\mathrm{s}Voiced duration ≥0.8 s\ge0.8\,\mathrm{s}; at least 3 units; alignment quality ≥0.7\ge0.7; at least 3 baseline intervals with the same language and counting unit
A04: Long pauseWithin-response pause ≥0.5 s\ge0.5\,\mathrm{s}RMS-confirmed silence ≥0.5 s\ge0.5\,\mathrm{s}; overlap ratio with VAD silence ≥0.5\ge0.5
A05: Repeated long pausesAt least 2 A04 pauses within one response; intervening speech ≥0.3 s\ge0.3\,\mathrm{s}; local candidate duration ≤15 s\le15\,\mathrm{s}Response duration ≥2 s\ge2\,\mathrm{s}
V01: Infrequent lip-corner loweringAU15-positive valid-frame proportion ≤min⁡(P25,0.1)\le\min(P_{25},0.1)Valid-face proportion ≥0.8\ge0.8
V03: High lip-corner tightening proportionAU14-positive valid-frame proportion ≥max⁡(P60,0.2)\ge\max(P_{60},0.2)Valid-face proportion ≥0.8\ge0.8
V05: Sustained downward head posturePitch above the within-video mean plus one SD; event mean pitch exceeds this threshold by at least 1∘1^\circ; duration ≥1.5 s\ge1.5\,\mathrm{s}Tracking confidence ≥0.8\ge0.8; adjacent observation gap ≤0.4 s\le0.4\,\mathrm{s}
V07: Limited head movementPitch and yaw amplitude RMS each ≤1∘\le1^\circ; angular-velocity RMS each ≤15∘/s\le15^\circ/\mathrm{s}; joint RMS of the four median-normalized measurements ≤P25\le P_{25}; merged duration ≥4 s\ge4\,\mathrm{s}Valid-pose proportion ≥0.8\ge0.8; at least 3 frames

Table S3. Measurement criteria and candidate localization parameters for the nine behavioral cues.

Joint Statistical–Event Representation. We retain sample-level statistics and event-level measurements as complementary views of the same evidence. Statistics summarize overall behavioral patterns for class-reference comparison. Events preserve local timing, duration, source links, and features for inspection and possible complementary comparison. Table S4 lists the cue-specific measures.

CueSample-level statisticsEvent-level measurements
A01Median of within-window loudness P90–P10Loudness P90–P10 (dB)
A02Median of within-window mean relative F0 change rateMean relative F0 change rate (%/s)
A03Median articulation rate within the same language and counting unitArticulation rate (words/s or Han characters/s)
A04Pause-union duration divided by response-proxy duration; median pause duration; events per minuteDuration only
A05Pause-union duration divided by response-proxy durationPause count; total pause duration divided by event duration
V01AU15-positive valid-frame proportionAU15-positive valid-frame proportion
V03AU14-positive valid-frame proportionAU14-positive valid-frame proportion
V05Median duration of downward-head-posture eventsMean downward pitch angle (degrees)
V07Window medians of pitch amplitude RMS, pitch angular-velocity RMS, yaw amplitude RMS, and yaw angular-velocity RMSPitch amplitude RMS, pitch angular-velocity RMS, yaw amplitude RMS, and yaw angular-velocity RMS

Table S4. Sample-level statistics and event-level measurements for the nine cues.

Review context. Counter-evidence review uses up to 3 s of source audio/video before and after each candidate event, clipped to the media boundaries. Event duration and behavioral measurements are computed over the original candidate interval.

Distance aggregation and missing comparisons. The event-level discrepancy between a query and a class is the median symmetric distance to reference samples of that class. Measurements are averaged equally within each statistical block. Available statistical and event blocks are then averaged equally within each cue, followed by an equal-weight average across valid cues and a square root to obtain the class distance. Reference comparisons are grouped by language; articulation rate additionally requires the same counting unit. Comparisons involving empty event sets or missing references for either class are excluded from aggregation. Samples with no valid comparison retain an unavailability flag and receive the default class 0 in full-denominator evaluation. Ties between class distances are assigned to class 0.

Model and generation settings. Support review uses the frozen Qwen2.5-Omni-7B model with a generation limit of 256 tokens. Independent and direct source-evidence reviews in the held-out experiments use the same model, with initial generation budgets of 512 and 384 tokens and maximum budgets of 2,048 and 768 tokens, respectively.

Reference Scaling and Event Comparison

For numerical coordinate jj, let μyj\mu_{yj} be the median among references of class yy. A robust scale fitted over the permitted references is

sj=max⁡{Q0.75,j−Q0.25,j, 0.05∣Q0.5,j∣, 10−6}.(8)s_j = \max\{Q_{0.75,j} - Q_{0.25,j},\ 0.05|Q_{0.5,j}|,\ 10^{-6}\}. \tag*{(8)}

The coordinate discrepancy is

ayj(q)=min⁡((∣fqj−μyj∣sj)2,4).(9)a_{yj}(q) = \min\left(\left(\frac{|f_{qj}-\mu_{yj}|}{s_j}\right)^2,4\right). \tag*{(9)}

This measures deviation from the class reference after scaling and limits the effect of extreme values. For events, a symmetric discrepancy compares timing and annotation-derived descriptors with each class’s references. The median discrepancy across references supplies the event-level contribution.

Textual Evidence and Review Boundaries

Figure S1 traces an illustrative V05 record from recording qq and video interval [ts,te][t_s,t_e] to assessment. Pitch, duration, and tracking validity support source verification; the traces are symbolic, not measured participant data. The solid Support path admits positively verified records directly. Dashed paths add Counter challenge and Evidence adjudication. Retained source anchors, measurements, judgments, and grounds feed behavioral reference comparison. The resulting AV assessment is matched with the independently obtained text assessment; disagreement triggers text-only review and the confidence gate. No single cue directly determines the depression label. Table S9 reports text-analysis stages, and Table S13 separates completion, source review, and final acceptance.

Figure S1. Source-grounded verification and evidence use.

Dataset / protocolInitial T Acc.Initial T F1Internal T review Acc.Internal T review F1Final AVT (Support) Acc.Final AVT (Support) F1
D-Vlog Leave-Query-Out90.2890.5493.5293.6994.3094.50
D-Vlog test90.6491.0194.1594.2595.3295.45
LMVD78.5578.2881.5181.8783.1082.46

Table S9. Text-analysis stages and final Support assessment (%).

RuleD-Vlog Leave-Query-Out Acc.D-Vlog Leave-Query-Out F1D-Vlog test Acc.D-Vlog test F1LMVD Acc.LMVD F1
T (unclassified retained)93.5293.6994.1594.2581.5181.87
+ completion93.5293.6994.1594.2582.7281.91
+ source review94.1794.3895.3295.4583.1682.51
+ final gate (AVT)94.3094.5095.3295.4583.1082.46

Table S13. Completion, review, and acceptance controls (%).

Textual evidence organization. We organize original transcript passages into PHQ-9 symptom support, explicit denial, personal-status statements, and contextual qualifiers. Each record retains its source identifier and quotation, distinguishing current or temporally unspecified personal experiences from historical states, third-party accounts, and general discussion. PHQ-9 provides symptom categories [11], not questionnaire scores inferred from text. An unmentioned symptom is not treated as a denial.

Textual assessment and internal review. Unimodal text analysis comprises evidence extraction, initial assessment, and internal review. Extraction organizes traceable source evidence; initial assessment integrates symptom support, denial, and alternative explanations; internal review independently revisits the source to examine the experiencer, temporal context, and symptom meaning. A valid review changes the initial prediction only when it disagrees with that prediction and its confidence in the selected class is at least 0.8, yielding the final text label T.

Note: P25P_{25} and P60P_{60} are the 25th and 60th percentiles of each cue’s measurements over valid observations within the video. Visual tracking confidence must be at least 0.8.

Conflict-triggered review. When valid T and AV predictions disagree, the conflict-review agent first forms an independent judgment from the source text and then reviews anonymized versions of the original text judgment and the independent judgment.

Output validation and acceptance. Conflict review returns a label, rationale, source identifiers, and confidence cc in its selected label. If the initial T and AV labels agree, their shared label is retained. Otherwise, a valid reviewed text label is used when c>0.8c > 0.8; if c≤0.8c \le0.8 or the review is invalid, AV is used. Validity requires an admissible output with nonempty citations belonging to the current source. A confident reviewed text label can therefore override AV. The reviewer does not receive the AV or ground-truth label. Confidence is self-reported and is not assumed to be calibrated. Missing text falls back to the available AV prediction, and unavailable T and AV predictions fall back to class 0.

Sections S9.2 and S9.6 present the prompts and output formats for four roles: textual evidence extraction, initial textual assessment, internal review, and conflict review. Each template specifies the role, inputs, task requirements, decision rules, and expected output.

Complete Reference-setting and Modality Results

Table S5 reports the Leave-Query-Out comparison and adds precision and recall to the main-paper accuracy and F1 results. DepressionAgent values come from its published reference-setting analysis. EDA uses the leave-one-recording-out reference construction defined in the main paper.

MethodAcc.Prec.Rec.F1
DepressionAgent (30B-A3B)92.4392.7393.6793.20
EDA Support (7B)94.3095.9493.1094.50
EDA Evidence (7B)94.1795.7093.1094.38

Table S5. D-Vlog Leave-Query-Out comparison (%).

Table S6 reports A, V, AV, T, and AVT for both admission settings. Support and Evidence use the same text predictions. The main-paper representation controls use the same Evidence-admitted AV events. Only the statistical or temporal representation changes.

SystemModalityD-Vlog Leave-Query-Out Acc.D-Vlog Leave-Query-Out F1D-Vlog test Acc.D-Vlog test F1LMVD Acc.LMVD F1
DepressionAgent (30B-A3B)A57.1339.75––––
V44.680.00––––
AV63.8454.80––––
T89.6090.38––––
AVT92.4393.20––––
EDA Support (7B)A67.1068.6465.5068.7869.0169.18
V61.7964.7663.1664.8061.2257.64
AV68.9171.6371.3574.6169.6770.35
T93.5293.6994.1594.2581.5181.87
AVT94.3094.5095.3295.4583.1082.46
EDA Evidence (7B)A67.1068.5666.0869.4769.4569.58
V61.5364.7764.3367.0356.8346.50
AV67.1070.2671.3574.8769.5670.27
T93.5293.6994.1594.2581.5181.87
AVT94.1794.3895.3295.4582.8982.21

Table S6. Complete modality results (%).

Table S7 compares backbones and workflows, while Table S8 compares statistical, temporal, and joint AV representations.

BackboneWorkflowD-Vlog Leave-Query-Out Acc.D-Vlog Leave-Query-Out F1D-Vlog test Acc.D-Vlog test F1LMVD Acc.LMVD F1
7BDirect87.5686.8988.8988.4876.4172.58
7BDepressionAgent86.7988.3687.7289.2376.7477.18
7BEDA Support94.3094.5095.3295.4583.1082.46
7BEDA Evidence94.1794.3895.3295.4582.8982.21
30B-A3BDirect81.3584.8783.6386.5474.6079.06
30B-A3BEDA Evidence90.1691.2692.4093.1980.5380.40
30B-A3BDepressionAgent92.4393.2095.2695.6982.5783.04

Table S7. Complete backbone and workflow comparisons (%).

RepresentationD-Vlog Leave-Query-Out Acc.D-Vlog Leave-Query-Out F1D-Vlog test Acc.D-Vlog test F1LMVD Acc.LMVD F1
Statistical Descriptors67.3670.3570.7674.2369.2869.47
Temporal Events59.9763.8666.0869.7962.2666.01
Joint Statistical–Event Representation67.1070.2671.3574.8769.5670.27

Table S8. Complete AV representation controls (%).

The main paper prioritizes D-Vlog Leave-Query-Out for workflow, representation, and admission comparisons. These complete tables also report D-Vlog test results with training references only. In Table S8, the joint representation on that test split uses fixed Evidence-admitted inputs. It reaches 71.35% accuracy and 74.87% F1. The gains over descriptors alone are 0.59 and 0.64 points. The gains over events alone are 5.27 and 5.08 points. In contrast, descriptors alone score slightly higher under Leave-Query-Out. Combining representations therefore does not help equally across evaluation settings.

Bold marks the best scores, including ties. Modality tables compare systems within each modality. Text-stage tables compare stages within the same dataset and protocol.

Text Analysis and Reference Protocols

Table S9 reports Support text-analysis stages alongside the modality results. For LMVD, we sort identifiers before stratified ten-fold splitting with shuffle and seed 42. Leave-Query-Out uses other eligible recordings and their labels as references. It excludes the query and its label. The foun- recording-out reference construction defined in the main paper.

Table S6 reports A, V, AV, T, and AVT for both admission settings. Support and Evidence use the same text predictions. The main-paper representation controls use the same Evidence-admitted AV events. Only the statistical or temporal representation changes.

Table S7 compares backbones and workflows, while Table S8 compares statistical, temporal, and joint AV representations.

The main paper prioritizes D-Vlog Leave-Query-Out for workflow, representation, and admission comparisons. These complete tables also report D-Vlog test results with training references only. In Table S8, the joint representation on that test split uses fixed Evidence-admitted inputs. It reaches 71.35% accuracy and 74.87% F1. The gains over descriptors alone are 0.59 and 0.64 points. The gains over events alone are 5.27 and 5.08 points. In contrast, descriptors alone score slightly higher under Leave-Query-Out. Combining representations therefore does not help equally across evaluation settings.

Bold marks the best scores, including ties. Modality tables compare systems within each modality. Text-stage tables compare stages within the same dataset and protocol.

Text Analysis and Reference Protocols

Table S9 reports Support text-analysis stages alongside the modality results. For LMVD, we sort identifiers before stratified ten-fold splitting with shuffle and seed 42. Leave-Query-Out uses other eligible recordings and their labels as references. It excludes the query and its label. The foun-

dation model stays frozen. This is leave-one-recording-out reference construction. D-Vlog test uses original training references only. LMVD builds references and normalization within the training folds, then pools held-out predictions across folds.

Table S9 separates initial text analysis, internal review, and final multimodal assessment; Table S6 provides the Support–Evidence modality comparison. On LMVD, Support reaches 83.10% accuracy and 82.46% F1. Evidence reaches 82.89% and 82.21%. Initial and internally reviewed text predictions stay fixed across admission settings. The final AVT row uses Support admission. Missing-text completion is tested separately in Table S13. The accompanying structured data retain run identifiers, source references, and transcribed values.

Admission and Decision Details

Admission configurations. Table S10 and Table S11 compare admission configurations on D-Vlog Leave-Query-Out and D-Vlog test, respectively; Table S6 provides the Support–Evidence comparison across modalities and datasets. Support retains positively supported candidates. Evidence weighs support against counter-evidence before admission. It can retain admissible observations despite a challenge. Technical failure is recorded separately from counter-evidence.

AdmissionIntervalsA Acc.A F1V Acc.V F1AV Acc.AV F1AVT Acc.AVT F1
Support99,16667.1068.6461.7964.7668.9171.6394.3094.50
+ Evidence97,55367.1068.5661.5364.7767.1070.2694.1794.38

Table S10. Behavioral admission on D-Vlog Leave-Query-Out (%).

AdmissionA Acc.A F1V Acc.V F1AV Acc.AV F1AVT Acc.AVT F1
Support65.5068.7863.1664.8071.3574.6195.3295.45
Evidence66.0869.4764.3367.0371.3574.8795.3295.45

Table S11. Cue-admission controls on D-Vlog test (%).

On D-Vlog Leave-Query-Out, Support reaches 94.30% AVT accuracy and Evidence reaches 94.17%. Support also exceeds Evidence on LMVD. On D-Vlog test, Support and Evidence have the same final AVT scores. Their A, V, and AV results still differ. On LMVD, Support reaches 83.10% accuracy and Evidence reaches 82.89%. Thus, changes in behavioral evidence need not produce the same changes in final assessment.

Completion, source review, and acceptance. Table S12 retains the main model’s conflict-review and confidence-gate controls. Table S13 further separates completion, source review, and final selection. The source-review and gate rows use the same completion outputs. The gate also reuses the source-review results. On LMVD, completion raises Support accuracy from 81.51% to 82.72%. Source review reaches 83.16%, and the gate yields 83.10%. The gate improves D-Vlog Leave-Query-Out accuracy by 0.13 points but lowers LMVD accuracy by 0.06 points. Its benefit therefore depends on the dataset. These controls fix Support admission and vary only review and final selection. Table S14 reports accepted revisions, corrected and harmed predictions, beneficial-revision rates, and overall accuracy changes. Revision-quality rates are conditional on changed outputs. Corrected minus harmed predictions equals the change in correct predictions from completion to final AVT.

ConfigurationD-Vlog Leave-Query-Out Acc.D-Vlog Leave-Query-Out F1LMVD Acc.LMVD F1
EDA Support94.3094.5083.1082.46
Without conflict review93.5293.6982.7281.91
Without confidence gate94.1794.3883.1682.51

Table S12. Decision controls for the final Support model (%).

ProtocolSamplesRevisedCorrectedHarmedRevised (%)Beneficial (%, 95% CI)ΔAcc. (pp)
D-Vlog Leave-Query-Out7728711.0487.50 [52.91, 97.76]+0.78
D-Vlog test1712201.17100.00 [34.24, 100.00]+1.17
LMVD1,8233923162.1458.97 [43.42, 72.92]+0.38

Table S14. Accepted prediction revisions with Support admission.

Annotation Inventory and Source Duration

Table S15 reports source-media durations and cue-admission totals. Annotation totals count cue-specific intervals. The main-paper distribution figure instead counts each recording once per retained cue. These units are not interchangeable. Source durations count each media input once per modality. Candidates, Support, and Evidence columns report cue-specific interval totals, not recording counts. Cue retention is the retained interval count divided by the initial candidate interval count, multiplied by 100%. It measures admission selectivity, not evidence correctness. LMVD uses separate audio and video sources, so their total durations differ.

Dataset / protocolAudio hoursVideo hoursCandidatesSupportEvidence
D-Vlog Leave-Query-Out123.85123.85127,71999,16697,553
D-Vlog test25.9625.9627,07920,99220,699
LMVD236.73214.62193,710103,33288,994

Table S15. Source-media durations and cue-admission totals.

Human Annotation and Source-Grounding Validation Protocol

The human evaluation covers annotated material from both D-Vlog and LMVD. Humans and AI use the same pre-segmented source units: consecutive 3-second windows for audio and for video. Annotators select the applicable cues from the literature-informed nine-cue codebook, using the corresponding modality and observable criteria. Adjacent windows with consistent cue assignments are merged into annotation intervals. The common segmentation avoids matching independently chosen human and AI boundaries.

Three researchers in affective computing independently review the source segments without seeing AI annotations. For each segment, each cue receives a binary present/absent judgment. A cue with unanimous initial judgments is retained without further review. Only cues with disagreement are referred to a panel comprising the original three researchers and two instructors in psychological counseling. The instructors participate in reviewing the disputed cues rather than creating a separate annotation set. All five panel members then vote on each disputed cue, and a majority of at least three determines its binary label. Agreement and escalation are therefore assessed per cue, not by comparing entire multi-label sets. The resulting human consensus is the ground-truth basis for evaluating the AI annotations.

We calculate annotation accuracy as

Acccue=KmatchKeval×100%.(10)\mathrm{Acc}_{\mathrm{cue}} = \frac{K_{\mathrm{match}}}{K_{\mathrm{eval}}} \times100\%. \tag*{(10)}

Here KevalK_{\mathrm{eval}} counts all evaluated AI annotations, and KmatchK_{\mathrm{match}} counts those matching the human labels. The pooled accuracy is 91.02% across the two datasets. It is not a D-Vlog-only result, an inter-rater reliability statistic, or a policy-specific Support–Evidence comparison. No separate per-dataset score is inferred from it.

This protocol addresses the three research questions at distinct levels. For Observe, the shared codebook makes the nine cue choices explicit. For Verify, blinded human judgments test whether the AI cue assignments agree with observations of the same source segments. Their timestamps

identify the inspected locations, but the score does not evaluate free-boundary temporal localization or the faithfulness of every generated rationale. For Assess, classification and accepted-revision metrics remain separate from annotation accuracy. They measure downstream utility, not the correctness of an individual cue.

Revision interpretation. The revision counts and beneficial-revision rates in Table S14 apply only to outputs that changed. They do not measure correction over the whole cohort. The main discussion therefore reports these rates separately from overall accuracy gains.

Case Studies and Source-to-Assessment Traces

Main-paper Figure 4 presents cases 0754, 0590, and 0306. Each case pairs video frames, an audio trace, and transcript passages with a subject-level evidence package. The package shows acoustic and visual measurements, class-reference distances, textual context review, and the reported final assessment. Table S16 summarizes the displayed cases.

CaseSource context and reported outcome
0754High-school context weakens a current-symptom reading; final non-depression assessment.
0590Game-specific interest limits symptom scope; final non-depression assessment.
0306Sleep and fatigue add symptom context; review agrees with AV; final depression assessment.

Table S16. Cases displayed in the main-paper source trace.

Source Excerpts and Context

The video, audio, and text excerpts retain their respective source timestamps. They are selected excerpts from the same recording, and their horizontal placement does not imply temporal alignment. The highlighted passages show why the scope of a textual statement matters. In 0754, high-school experiences weaken a current-symptom interpretation. In 0590, loss of interest concerns one game. In 0306, sleep and fatigue complaints add to the initial appetite-related passage.

Behavioral Comparison and Reported Decisions

The evidence packages display the actual supplied measurements and distances to both class references. The displayed textual review explains how source context changes the interpretation of the initial symptom passage. Cases 0754 and 0590 report a final non-depression assessment, while 0306 reports depression. Their post-review text confidences are c=0.70c = 0.70, 0.600.60, and 0.800.80, respectively. The confidence selector therefore uses AV in all three displayed cases, including the boundary value c=0.80c = 0.80. The high-confidence reviewed-text branch is defined in the method. These are the outcomes in the supplied case records. The figure does not assign a ground-truth correctness label or establish the causal contribution of an individual cue. Figure S1 separately explains the cue-level verification mechanism as a schematic.

Prediction Revision Results

Table S14 reports accepted revisions under Support admission. Among changed predictions, 87.50% are beneficial on D-Vlog Leave-Query-Out and 58.97% on LMVD. The count-based net revision gains are 0.78 and 0.38 percentage points. D-Vlog test has a beneficial-revision rate of 100.00% and a net gain of 1.17 points, with the uncertainty interval reported in the table. Revision rates use changed predictions as the denominator, while net accuracy gains use the full evaluated cohort. The counts compare completed text predictions with final AV predictions, matching the corresponding rows in Table S13.

Agent Prompts and Output Templates

This section presents the text-agent prompts in English, organized by role, inputs, task rules, and output format. The templates translate and organize the source instructions; the evidence-extraction template consolidates its constituent tasks.

Cue Annotation

Candidate intervals are generated by rules and subsequently submitted for evidence review.

Task. Locate observable acoustic and visual behaviors and record their temporal intervals, measurements, and source references, without predicting depression labels.

Inputs. Time-stamped acoustic and visual features, media timing information, and the definitions and selection parameters of the corresponding cues. Depression labels are excluded from the inputs.

Annotation rules.

  • Acoustic cues comprise loudness variation (A01), fundamental-frequency variation (A02), articulation rate (A03), pause duration (A04), and repeated pauses (A05). Visual cues comprise AU15 (V01), AU14 (V03), head lowering (V05), and head movement (V07).

  • Apply the measurement definitions, thresholds, and valid-observation rules specified earlier for each cue. Record the corresponding behavioral measurements; overall impressions of affect do not substitute for specific behaviors.

  • Express interval boundaries in seconds on the original media timeline. Merge intervals according to the cue-specific rules and retain source references for their constituent windows.

  • Retain quality flags and measurement evidence for each candidate. A candidate indicates that the selection criteria are satisfied; subsequent review determines whether the source evidence supports it.

Output format. Each candidate contains the fields below. If no candidate is detected, return an empty list and record the observability status of the cue.

FieldContent
Cue identifierCorresponding acoustic or visual cue.
Candidate identifierIdentifier distinguishing candidates within the same cue.
Temporal boundariesStart and end times in the original media, in seconds.
Behavioral measurementsCue-specific measurement values and units.
Quality flagsInformation about candidate observation quality.
Source referencesFeature records and constituent windows supporting the candidate.

PHQ-9 Text Analysis and Internal Review

Box T1: Textual Evidence Extraction Agent.

Role. You are a textual evidence extractor. Extract evidence related to PHQ-9 symptoms and personal status from the participant’s transcript. Do not predict a binary label or estimate questionnaire scores.

Inputs. The original transcript with segment identifiers and the PHQ-9 symptom definitions.

Task rules.

  • Read the source and extract symptoms, explicit denials, and personal-status statements expressed by the participant.

  • Preserve the experiencer, time, negation, and contextual qualifications. Distinguish current or temporally unspecified personal experiences from historical or resolved states, other people’s experiences, and general discussion.

  • Quotes must come from their cited source segments. Do not join different segments or invent symptoms, frequency, or severity.

  • Each symptom quote must express its assigned item. Ordinary task frustration, boredom with one activity, or general life stress does not automatically establish a symptom.

  • Physical illness, medication, and life events may qualify the evidence or provide alternative explanations, but do not automatically erase an explicitly reported symptom.

  • Record explicit personal depression status and treatment statements separately; they do not replace specific symptom evidence.

  • Absence of mention is not denial. Do not fill unsupported items or count duplicate evidence repeatedly.

Output format. Return structured evidence containing symptom-support records (PHQ-9 item, source segment identifier, quotation, and qualifications), explicit-denial or opposite-experience records (item, source identifier, and quotation), personal-status records (depression or treatment quotation, experiencer, and time), and source-supported alternative explanations and uncertainties.

Box T2: Initial Textual Assessment Agent.

Role. You are a textual assessment agent. Produce a research binary prediction from structured textual evidence, not a clinical diagnosis.

Inputs. Extracted symptom support, explicit denials, personal-status statements, alternative explanations, and uncertainties.

Decision rules.

  • Integrate all evidence. Do not decide solely from a keyword, one symptom, or the number of symptoms.

  • Persistent or explicit loss of interest or low mood, multiple independent current symptoms, functional impairment, and self-harm expressions can strengthen a positive assessment.

  • Reviewed personal current or temporally unspecified depression status and current antidepressant treatment may provide personal-status evidence, but cannot substitute for specific symptoms.

  • Other people’s experiences, general discussion, resolved history, and medication of unspecified purpose do not establish the participant’s current status.

  • A single mild, temporally unclear, or explicitly explained symptom does not automatically support a positive assessment.

  • Unmentioned symptoms are not counter-evidence. Reflect uncertainty when support and denial coexist.

  • Ground the rationale and key items in the supplied evidence. Do not infer the dataset label or questionnaire score.

Output format. Return a JSON object with the following fields:

  • prediction_label2: initial label, 0 or 1.

  • probability_label2_is_1: probability of class 1 in [0, 1].

  • rationale: principal evidence and reasoning for the assessment.

  • key_item_numbers: the PHQ-9 items supporting the assessment.

  • uncertainty: unresolved uncertainty.

Box T3: Internal Review Agent.

Role. You are an independent textual evidence reviewer. Revisit the source to check symptom interpretations and their evidential basis, and produce a review used to determine the final text label T.

Inputs. Original textual context with segment identifiers and speaker information, without audiovisual predictions or ground-truth labels.

Review rules.

Box T3: Internal Review Agent. (continued)

  • Independently examine the experiencer, symptom meaning, current or historical time, and alternative explanations.

  • Do not interpret depression-related discussion, other people’s experiences, education, or resolved history as the participant’s current state.

  • Loss of interest requires loss of interest or pleasure. Low mood requires an emotional expression, rather than an inference from physical fatigue or unfinished work.

  • Ordinary failure, polite guilt, everyday forgetfulness, and fatigue after exercise do not automatically establish the corresponding symptoms.

  • Retain genuine personal symptoms even when life stress is present. Consider persistence, functional impact, and independent symptoms together.

  • Interpret indirect self-harm expressions semantically; do not require a particular diagnostic word.

  • Cite only participant statements. Interviewer statements provide context, not participant evidence. Both positive and negative judgments require source citations.

  • Do not infer unstated symptoms, history, or labels.

Output format. Return a JSON object with rationale (a brief review rationale), evidence_ids (at most three supplied source identifiers), prediction_label2 (0 or 1), and probability_label2_is_1 (a value in [0,1] consistent with the label).

Support Verification

The following templates present the approved acoustic and visual review instructions in English. Output descriptions are translated for presentation.

Box A1: Acoustic Support Agent.

Role. You are an acoustic behavioral evidence reviewer. Determine whether the candidate interval supports the specified acoustic cue using the audio and its corresponding features.

Inputs. Cue identifier and definition, source audio interval, transcript for locating the utterance, and corresponding acoustic measurements.

Task rules.

  • Listen to the audio first, then use feature values to check the same acoustic phenomenon. The transcript locates the utterance but does not establish an acoustic observation.

  • Do not invent values, times, or comparison baselines that were not supplied.

  • Describe only what is heard in the auditory observation; place times and numbers in the corresponding feature explanation.

  • Describe the core phenomenon in at most 40 Chinese characters. Cite one or two actual features and explain how they relate to the observation.

  • The verdict must agree with the observation and feature explanation.

Cue-specific focus.

  • A01: Describe loudness variation and cite loudness SD or P90–P10.

  • A02: Describe the rate of pitch changes during continuous speech and cite relative F0 change rate. Small pitch excursions do not establish a low rate of change.

  • A03: Describe articulation speed excluding pauses and cite articulation rate. Slow overall speech caused by frequent pauses does not establish slow articulation.

  • A04: Check whether the pause occurs within the same response and cite its duration.

  • A05: Describe at least two audible pauses and cite their count, events, or durations.

Output format. Return only two lines:

VERDICT: supported or not_supported

JOINT_EXPLANATION: Heard in the audio: specific acoustic phenomenon; corresponding features: field = actual value, with an explanation of the relationship.

Box A2: Visual Support Agent.

Role. You are a visual behavioral evidence reviewer. Determine whether the core interval supports the specified visual cue using consecutive frames and temporally corresponding features.

Inputs. Cue definition, core interval, consecutive images with identifiers, corresponding frame-level features, and interval statistics.

Task rules.

  • Review only the core interval. Frame-level features correspond to images with the same identifiers; missing values are not zeros.

  • Observe consecutive frames first, then check the same behavior against temporally corresponding feature values. A positive feature value cannot replace visual observation.

  • Observe at least three adjacent images covering 80% of the core interval. Context frames cannot replace core observations.

  • Describe only visible actions in the observation; place times and numbers in the feature explanation.

  • Do not invent values, times, or comparison baselines. Do not interpret engineering selection thresholds as clinical thresholds.

  • Describe the core phenomenon in at most 40 Chinese characters. Cite one or two actual features and explain their correspondence with the visual observation.

  • Cite the frame identifier with each frame-level value. Interval statistics may be cited directly as a field and its actual value.

  • The verdict must agree with the explanation. Do not supply evidence absent from the input.

Output format. Return only two lines:

VERDICT: support or not-support

JOINT_EXPLANATION: Seen in the video: specific action; corresponding features: field = actual value, with an explanation of the relationship.

Counter Challenge

Box A3: Counter Agent.

Role. You are a behavioral counter-evidence reviewer. Seek specific counter-evidence or confounding evidence that could overturn the core behavioral claim. Do not replace Support with a supporting judgment.

Inputs. The behavioral claim, core-interval observations, surrounding-context observations, and relevant behavioral measurements and quality information.

Task rules.

  • Seek specific counter-evidence against the core behavior definition. Context provides interpretation but cannot replace the core state or contribute to the core behavior proportion.

  • Inaudibility does not establish stable loudness or slow pitch changes. A forward-facing head does not establish sustained downward posture.

  • Examine the necessary conditions of the behavior definition. Differences in task, sample, or statistical unit from the literature limit generalization but do not automatically negate an observed behavior.

  • For proportion-based cues, observing an action at least once does not establish a high occurrence proportion.

  • Detection failure, low confidence, duplicate frames, tracking gaps, or camera motion do not establish low activity. Insufficient quality means the claim cannot be verified, not that a behavioral counterexample exists.

  • Use quality information only when it affects the current claim; it cannot replace core-media observation.

  • Missing information is unknown, not normal or abnormal.

  • Failure to find counter-evidence does not support the claim. Leave the validity of counter-evidence and conflicts between observations to the Evidence agent.

Judgment categories. Counter-evidence found: a specific observation may negate the core claim. No counter-evidence found: the claim was checked without finding a specific counterexample; this provides no additional support. Unverifiable: observation or quality is insufficient for reliable verification.

Output format. Return a JSON object containing only a judgment (counter-evidence found, no counter-evidence found, or unverifiable) and a reason of at most 80 Chinese characters citing core observations and times and explaining the counter-evidence or observation limit. Requests with audiovisual speaker verification enabled additionally return a speaker-verification result: participant, non-participant, or uncertain. Omit this field when that check is disabled.

Evidence Adjudication

Box A4: Evidence Agent.

Role. You are a behavioral evidence adjudicator. Determine whether a cue is established by the support and counter-evidence records. Do not make a sample-level diagnosis.

Inputs. Cue definition, Support and Counter records for each interval, source information, and relevant quality limitations.

Adjudication rules.

  • Each evidence identifier denotes an independent interval. Compare support and counter-evidence separately for that interval.

  • Admit the cue only if at least one interval has a credible core Support observation satisfying the definition that remains valid after counter-evidence review. Block it if all intervals fail to satisfy the definition or cannot be confirmed.

  • A counterexample in one interval does not automatically negate other intervals.

  • Ground reasons in the records. Do not change evidence times, confuse sources, or reinterpret speaker-attribution observations as acoustic-speed judgments.

  • If a condition is not documented, state that it is unspecified and cannot be confirmed; do not claim it is absent.

  • Identify specific contradictory observations and adjudicate within the same core interval. Neighboring context is explanatory only.

Box A4: Evidence Agent. (continued)

  • Counter’s failure to find a counterexample adds no support, cannot increase support strength, and cannot compensate for an unsupported Support claim.

  • Examine Counter’s observations, reasons, confound explanations, and quality limitations rather than copying its verdict.

  • Insufficient quality limits verification but is not automatically a counterexample. Explicitly check facts such as cross-response pauses or frozen frames that violate necessary conditions. Reading or similar context cannot negate downward posture unless it contradicts the observed posture itself.

  • Execution or formatting failure does not establish counter-evidence. If the raw response is retained, inspect its actual observations without inventing source times.

Output format. Return one JSON object, with the decision first and rationale second:

decision: admit or block rationale: at most 80 Chinese characters stating the specific record-based grounds; identify missing information when evidence is insufficient.

Support admission setting. Support admission directly uses candidates that pass support review. Evidence admission additionally adjudicates these records using Counter’s findings. Both settings use the same downstream behavioral representation and class-comparison method.

Conflict-triggered Text Review

Box T4: Conflict Review Agent.

Role. You are a textual evidence reviewer. Examine symptoms and personal-status evidence in the original text and produce a research binary assessment. Conflict review comprises an independent review followed by a final review.

Step 1: Independent review. Inputs. Original textual context and the inventory of retained quotations.

Task rules. You are an independent textual evidence reviewer producing a research depression-screening classification, not a clinical diagnosis. You do not know the ground-truth label. Read the source directly, first give a brief rationale checking the experiencer, symptom meaning, current or historical time, and alternative explanations, and then make a judgment.

Class 1 denotes explicit personal current or temporally unspecified depression status, current treatment for depression, or a sufficiently clear combination of depressive symptoms. Class 0 denotes insufficient textual evidence. Do not treat discussion of depression, another person’s experiences, psychoeducation, or resolved history as the participant’s current state.

Loss of interest requires loss of previous interest or pleasure, not an ordinary dislike. Low mood requires an emotional expression, not physical fatigue, unfinished work, or a product description. Negative self-evaluation is not a failed examination, polite guilt, or quoted reassurance. Sleeping late normally, describing sleep products, mentioning dining locations, everyday forgetfulness, and fatigue after exercise do not automatically constitute symptoms. Distress, trauma, and fear of death in past events do not directly establish current depression or self-harm ideation.

Retain genuine personal symptoms even when life stress is present. Integrate independent symptoms, persistence, functional impairment, and explicit self-harm expressions. Examine indirect self-harm expressions semantically rather than requiring a particular English diagnostic term. Personal medication use and an explicit depression-treatment indication in the same passage may refer to each other; do not infer the purpose of unspecified medication.

Cite only the participant’s supplied statements; interviewer statements provide context. Do not infer unstated symptoms, history, or labels. You have not seen any existing classification labels; judge the source independently and do not guess an earlier model conclusion. Negative judgments also require source citations. Do not treat support/counter-evidence item names as semantically verified conclusions.

Output rules. Output valid JSON only. Write the rationale first, followed by citations, label, probability, and evidence profile. Both positive and negative judgments must cite their source evidence.

Output format.

  • rationale: a brief review rationale; the prompt specifies at most 120 Chinese characters.

  • evidence_ids: one to three supplied evidence identifiers.

  • prediction_label2: 0 or 1.

  • probability_label2_is_1: probability of class 1 in [0, 1].

  • evidence_profile: one of explicit_status, clear_symptom_cluster, non_specific_or_alternative, third_party_or_historical, or insufficient_source.

Step 2: Final review. Inputs. Original textual context, symptom or personal-status evidence to audit, and two anonymized textual judgments A/B, each with a label and rationale.

Task rules. You are a text-screening reviewer. Both judgments may be wrong, and their A/B order does not indicate priority. You do not know their authors, the AV label, the ground-truth label, or whether the sample was classified correctly. Make your own final binary judgment directly from the source rather than selecting a winning rationale.

Check whether the statements concern the participant, whether they describe current, historical, or resolved experiences, whether ordinary advice or education has been mistaken for symptoms, and whether genuine symptoms support the overall judgment. Item Box T4: Conflict Review Agent. (continued)

fields are previous extractions, not established facts. Retaining genuine symptoms and making an overall negative judgment can be consistent; insufficient support is different from explicit counter-evidence. Assess whether alternative explanations are supported by the source rather than merely possible. Life stress, positive experiences, and other illnesses can coexist with depressive symptoms. Absence of medical consultation or diagnosis does not itself invalidate an established state. Do not use depression-related words, a single frequency, or symptom counts as automatic positive criteria. The target is text-based screening risk: class 1 denotes explicit personal current or temporally unspecified status, explicit depression treatment, or a sufficiently clear symptom combination. Class 0 denotes source evidence favoring a negative assessment or adequate alternative explanations; it does not require every symptom to be false. Do not equate inability to make a diagnosis with a negative label. The two rationales are references only. Ground the final conclusion in the source, without favoring a rationale because it is longer, more confident, or lists more items.

Output rules. Output JSON only. Do not declare an A/B winner, recopy quotations, or add other fields.

Output format.

  • rationale: one Chinese sentence of at most 160 characters stating the key source basis for the final judgment.

  • evidence_ids: one to three existing E/S identifiers from the input.

  • prediction_label2: final label, 0 or 1.

  • confidence (c): self-reported probability that the selected label is correct, in [0.5,1]; greater uncertainty should be closer to 0.5.

External fusion rule. An initial T–AV conflict triggers this text-only review. The returned confidence is denoted cc. For a valid review, c>0.8c > 0.8 selects the reviewed text label; otherwise the AV label is used. The external selector applies this rule after the review and does not send the AV label to the reviewer.

Scope and Further Evaluation

Behavioral cues are nonspecific, and annotations can contain errors. Source links allow inspection but do not establish clinical validity. These cohorts were used during cue and decision development. Using training references alone does not make them untouched prospective evaluations. Recording-level folds do not guarantee speaker separation. Single-run scores do not measure inference variability or isolate the effect of model size. Further work needs external cohorts, repeated inference runs, and policy-specific evidence-quality evaluation. Source recordings and quotations retain their original access restrictions. EDA has not been validated for autonomous clinical use.

Completed validation and further paired diagnostics. Section S7.1 reports the completed human protocol and pooled 91.02% cue-annotation accuracy. This source-segment check is separate from policy-specific evidence quality. A paired Support–Evidence evaluation would require the same source units, human labels, and downstream settings. It could then measure retained-cue precision and recall, erroneous admission, and missed supported cues. These additional metrics are not reported here. Independently annotated boundaries would also be needed for a free-boundary localization metric.

Evidence quality and assessment utility. Evidence quality asks whether a cue is supported by its cited source. Assessment utility asks whether the evidence workflow improves prediction or revision. The completed human check addresses cue agreement on common segments. Admission counts describe selection behavior, not source precision. The final EDA Support model optimizes that selection while retaining source verification. Its classification gains over extended Evidence review do not establish superior annotation quality. Similarly, temporal events preserve local source access without guaranteeing better classification.

Among accepted revisions, the beneficial proportions are 87.50% on D-Vlog Leave-Query-Out and 58.97% on LMVD. Harmful changes account for 12.50% and 41.03%, respectively. These are conditional on outputs that changed. Their confidence intervals and net accuracy gains are given in Table S14. Figure 4 shows completed source-to-assessment examples, while these aggregate revision statistics include successes and failures.

Backbone effects. Table S7 reports the backbone comparison. On D-Vlog Leave-Query-Out, 30B-A3B EDA improves accuracy and F1 over direct prediction by 8.81 and 6.39 points. A larger backbone does not always improve EDA. With Evidence admission, 7B exceeds 30B-A3B by 3.12 F1 points. Both evidence organization and backbone choice affect the result.

Extended Related Work

Multimodal and temporal depression modeling. Speech, facial activity, and nonverbal behavior provide information for computational depression assessment [1, 5, 18]. Multimodal architectures learn feature interactions and handle unaligned sequences [25, 32]. Depression-specific methods use time-aware fusion, instance selection, and multiscale temporal representations [9, 12, 20, 23, 24, 39, 40]. Recent work also studies state-space models, multiscale interactions, and facial behavior [14, 21, 30, 37, 38]. EDA focuses on the observations used for assessment. It defines cues, links them to source intervals, and reviews them before comparison.

Symptom evidence and agent deliberation. PHQ-9 defines established symptom domains [11]. Question-wise Modality Fusion, RED, and Explain-MDRC organize interview content around symptoms and supporting passages [16, 33, 36]. Dep-LLM and AI Psychiatrist Assistant use evidence and specialized reasoning roles for assessment [6, 13]. Multimodal language-model studies extend this analysis to speech, vision, and text [34, 35]. DepressionAgent is closely related. It combines modality-specific extraction, support–challenge deliberation, arbitration, and reflection [41]. EDA defines the cue records that agents review and makes their admission into assessment explicit.

Perception, grounding, and source verification. VideoAgent, VideoTree, and DrVideo organize long-video content for language-model reasoning [15, 26, 27]. VideoGLaMM studies fine-grained visual grounding [17]. OPPO distinguishes cue use from input-supported emotion reasoning and uses reinforcement learning to improve perception [8]. EDA performs source checks at inference time with frozen models. Evidence retains cue definitions, source intervals, support, and contextual challenges. Support simplifies admission. Debate and verification methods also examine alternative interpretations [3, 22]. In EDA, agents review a defined behavioral record. Its source support and later use can be inspected separately.

References

  1. [1]Zakaria, Aldeneh, Mimansa Jaiswal, Michael Picheny, Melvin G. McInnis, and Emily Mower Provost. Identifying mood episodes using dialogue features from clinical interviews. In Interspeech, pages 1926–1930, 2019.arxiv.org/abs/1910.05115
  2. [2]Nicholas Cummins, Judith Dineley, Pauline Conde, Faith Matcham, Sara Siddi, Femke Lamers, Ewan Carr, Grace Lavelle, Daniel Leightley, Katie M. White, Carolin Oetzmann, Edward L. Campbell, Sara Simblett, Stuart Bruce, Josep Maria Haro, Brenda W.J.H. Penninx, Yatharth Ranjan, Zulqarnain Rashid, Callum Stewart, Amos A. Folarin, Raquel Bailón, Björn W. Schuller, Til Wykes, Srinivasan Vairavan, Richard J.B. Dobson, Vaibhav A. Narayan, and Matthew Hotopf. Multilingual markers of depression in remotely collected speech samples: A preliminary analysis. Journal of Affective Disorders, 341:128–136, 2023.
  3. [3]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, pages 11733–11763, 2024.arxiv.org/abs/2305.14325
  4. [4]Juliana Teixeira Fiquer, Ricardo Alberto Moreno, Andre R. Brunoni, Vivian Boschesi Barros, Fernando Fernandes, and Clarice Gorenstein. What is the nonverbal communication of depression? assessing expressive differences between depressive patients and healthy volunteers during clinical interviews. Journal of Affective Disorders, 238:636–644, 2018.
  5. [5]Jeffrey M. Girard, Jeffrey F. Cohn, Muhammad H. Mahoor, S. Mohammad Mavadati, Zakia Hammal, and Dean P. Rosenwald. Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses. Image and Vision Computing, 32(10):641–647, 2014.pubmed.ncbi.nlm.nih.gov/25378765
  6. [6]Adam Greene, Neviah Blair, Samin Mahdipour Aghabagher, Simmi Kumari, Michael W. Schlund, Alex Fedorov, Vince D. Calhoun, Xinhui Li, and Rogers F. Silva. AI psychiatrist assistant: An LLM-based multi-agent system for depression assessment from clinical interviews. In Proceedings of the Fifth Machine Learning for Health Symposium, pages 525–542, 2026.
  7. [7]Melisa Gumus, Danielle D DeSouza, Mengdan Xu, Celia Fidalgo, William Simpson, and Jessica Robin. Evaluating the utility of daily speech assessments for monitoring depression symptoms. DIGITAL HEALTH, 9, 2023.
  8. [8]Zhiyuan Han, Beier Zhu, Wenwen Tong, Pengyang Shao, Pipei Song, Xinyi Wang, Jiangnan Chen, Lewei Lu, and Xun Yang. Omni-perception policy optimization for multi-modal emotion reasoning. In Proceedings of the 43rd International Conference on Machine Learning, pages 40269–40294. PMLR, 2026.DOI
  9. [9]Lang He, Zheng Li, Prayag Tiwari, Cui Cao, Jize Xue, Feng Zhu, and Di Wu. Depressformer: Leveraging Video Swin Transformer and fine-grained local features for depression scale estimation. Biomedical Signal Processing and Control, 96:106490, 2024.DOI
  10. [10]Lang He, Kai Chen, Junnan Zhao, Yimeng Wang, Ercheng Pei, Haifeng Chen, Jiewei Jiang, Shiqing Zhang, Jie Zhang, Zhongmin Wang, Tao He, and Prayag Tiwari. LMVD: A large-scale multimodal vlog dataset for depression detection in the wild. Information Fusion, 126:103632, 2026.arxiv.org/abs/2407.00024
  11. [11]Kurt Kroenke, Robert L. Spitzer, and Janet B. W. Williams. The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9):606–613, 2001.pubmed.ncbi.nlm.nih.gov/11556941
  12. [12]Tianfei Ling, Deyuan Chen, and Baobin Li. MDAVIF: A Multi-Domain Acoustical-Visual Information Fusion Model for Depression Recognition from Vlog Data. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8115–8119, 2024.DOI
  13. [13]Yiqing Lyu, Xianbing Zhao, Buzhou Tang, and Ronghuan Jiang. Dep-LLM: Training-free depression diagnosis via evidence-guided structured multi-factor with reliable LLM reasoning. arXiv preprint arXiv:2606.10796, 2026.arxiv.org/abs/2606.10796
  14. [14]Mengyuan Ma, Chengji Guo, Sijia Liu, Weizhong Zhao, and Xingpeng Jiang. BHDD: Facial Behaviors Modeling and Hierarchical Multimodal Fusion for Depression Detection From Vlog Data. IEEE Transactions on Affective Computing, 17(3):2714–2728, 2026.
  15. [15]Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai. Drvideo: Document retrieval based long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18936–18946, 2025.arxiv.org/abs/2406.12846
  16. [16]Aishik Mandal, Dana Atzil-Slonim, Thamar Solorio, and Iryna Gurevych. Enhancing depression detection via question-wise modality fusion. In Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2025), pages 44–61, Albuquerque, New Mexico, 2025. Association for Computational Linguistics.
  17. [17]Shehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao, Eric Xing, Fahad Shahbaz Khan, and Salman Khan. VideoGLaMM: A large multimodal model for pixel-level visual grounding in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19036–19046, 2025.arxiv.org/abs/2411.04923
  18. [18]James C. Mundt, Peter J. Snyder, Michael S. Cannizzaro, Kara Chappie, and Dayna S. Gertals. Voice acoustic measures of depression severity and treatment response collected via interactive voice response (IVR) technology. Journal of Neurolinguistics, 20(1):50–64, 2007.
  19. [19]Åsa Nilsonne. Speech characteristics as indicators of depressive illness. Acta Psychiatrica Scandinavica, 77(3):253–263, 1988.pubmed.ncbi.nlm.nih.gov/3394527
  20. [20]Zixuan Shangguan, Xiaxi Li, Yanjie Dong, and Xiaoyan Yuan. Automatic Depression Detection Using Attention-Based Deep Multiple Instance Learning. In Quality, Reliability, Security and Robustness in Heterogeneous Systems, pages 40–51, 2024.DOI
  21. [21]Min Shi, Xu Qiao, Lang He, and Rui Gao. An Interaction Network Based on Multiscale Spatiotemporal Feature Fusion for Depression Prediction. IEEE Transactions on Affective Computing, 17(3):3100–3111, 2026.
  22. [22]Yudi Shi, Shangzhe Di, Qirui Chen, and Weidi Xie. Enhancing video-llm reasoning via agent-of-thoughts distillation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8523–8533, 2025.arxiv.org/abs/2412.01694
  23. [23]Yongfeng Tao, Minqiang Yang, Huiru Li, Yushan Wu, and Bin Hu. DepMSTAT: Multimodal Spatio-Temporal Attentional Transformer for Depression Detection. IEEE Transactions on Knowledge and Data Engineering, 36(7):2956–2966, 2024.DOI
  24. [24]Yongfeng Tao, Minqiang Yang, Yushan Wu, Kevin Lee, Adrienne Kline, and Bin Hu. Depressive semantic awareness from vlog facial and vocal streams via spatio-temporal transformer. Digital Communications and Networks, 10(3):577–585, 2024.DOI
  25. [25]Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558–6569, 2019.arxiv.org/abs/1906.00295
  26. [26]Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. VideoAgent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision, 2024.arxiv.org/abs/2403.10517
  27. [27]Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. VideoTree: Adaptive tree-based video representation for LLM reasoning on long videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3272–3283, 2025.arxiv.org/abs/2405.19209
  28. [28]Jin Xu, Zhiqiang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyan Lin. Qwen2.5-Omni technical report. arXiv preprint arXiv:2503.20215, 2025.arxiv.org/abs/2503.20215
  29. [29]Mao Yamamoto, Akihiro Takamiya, Kyosuke Sawada, Michitaka Yoshimura, Momoko Kitazawa, Kuo ching Liang, Takanori Fujita, Masaru Mimura, and Taishiro Kishimoto. Using speech recognition technology to investigate the association between timing-related speech features and depression severity. PLOS ONE, 15(9):e0238726, 2020.
  30. [30]Jiaxin Ye, Junping Zhang, and Hongming Shan. DepMamba: Progressive fusion mamba for multimodal depression detection. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025.arxiv.org/abs/2409.15936
  31. [31]Jeewoo Yoon, Chaewon Kang, Seungbae Kim, and Jinyoung Han. D-Vlog: Multimodal vlog dataset for depression detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12226–12234, 2022.
  32. [32]Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114, 2017.arxiv.org/abs/1707.07250
  33. [33]Linhai Zhang, Ziyang Gao, Deyu Zhou, and Yulan He. Explainable depression detection in clinical interviews with personalized retrieval-augmented generation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9927–9944, Vienna, Austria, 2025. Association for Computational Linguistics.arxiv.org/abs/2503.01315
  34. [34]Wei Zhang, Juan Chen, En Zhu, Wenhong Cheng, Yunpeng Li, Yuhan Li, and Yanbo J. Wang. MLIm-DR: Towards explainable depression recognition with multimodal large language models. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(4):1–23, 2026.DOI
  35. [35]Xiangyu Zhao, Yaling Shen, Yiwen Jiang, Zimu Wang, Jiahe Liu, Maxwell H Cheng, Guilherme C Oliveira, Robert Desimone, Dominic Dwyer, and Zongyuan Ge. It hears, it sees too: Multi-modal LLM for depression detection by integrating visual understanding into audio language models. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026, pages 4185–4196, Hengqin, China, 2026. Association for Computational Linguistics.arxiv.org/abs/2511.19877
  36. [36]Wenjie Zheng, Qiming Xie, Jianfei Yu, Yang Wang, Lei Cao, Fei Wang, Shijin Wang, Rui Xia, and Chengqing Zong. Explainable multimodal depression recognition in clinical interviews via PHQ-aligned symptom summarization. arXiv preprint arXiv:2501.16106, 2026. Version 2.arxiv.org/abs/2501.16106
  37. [37]Bowen Zhou, Marc-André Fiedler, and Ayoub Al-Hamadi. Cross-Modal Fusion Mamba for Multimodal Depression Detection. In 2025 14th International Symposium on Image and Signal Processing and Analysis (ISPA), pages 225–230, 2025.DOI
  38. [38]Bowen Zhou, Marc-André Fiedler, and Ayoub Al-Hamadi. STCM-Mamba: Multimodal Spatio-Temporal Cross-Modal Mamba for Depression Detection. IEEE Access, 13:189494–189505, 2025.DOI
  39. [39]Li Zhou, Zhenyu Liu, Zixuan Shangguan, Xiaoyan Yuan, Yutong Li, and Bin Hu. TAMFN: Time-Aware Attention Multimodal Fusion Network for Depression Detection. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:669–679, 2023.DOI
  40. [40]Li Zhou, Zhenyu Liu, Xiaoyan Yuan, Zixuan Shangguan, Yutong Li, and Bin Hu. CAIINET: Neural network based on contextual attention and information interaction mechanism for depression detection. Digital Signal Processing, 137:103986, 2023.DOI
  41. [41]Fangjie Zhu, Haifeng Lu, Sicheng Zhao, Runhao Zeng, and Xiping Hu. DepressionAgent: Reading, listening, seeing, and deliberating multimodal evidence for depression risk assessment. arXiv preprint arXiv:2608.13891, 2026. Version 1.arxiv.org/abs/2608.13891
  42. [42]Åsa Nilsonne, Johan Sundberg, Sten Ternström, and Anders Askenfelt. Measuring the rate of change of voice fundamental frequency in fluent speech during mental depression. The Journal of the Acoustical Society of America, 83(2):716–728, 1988.pubmed.ncbi.nlm.nih.gov/3351130

Paper details

vlnc:W_768qs8b3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents