betaFeedback

WorksComputer Science

Does the Voice Matter? Evaluating Frustration Detection on Deployed Clinical Voice-Agent Calls

Abstract

Autonomous voice agents now conduct routine clinical follow-up calls, so some automated triage is unavoidable. A patient becoming frustrated is one thing that triage should catch. These systems typically read only the transcript the speech-to-text stage produces, not the voice, and whether that is enough is largely untested on real clinical speech. We study per-turn frustration detection on real patient calls from a deployed clinical voice agent: 150 turns from six calls labelled by three annotators, with a length-matched acted corpus as a control, and 714 turns from 30 calls. Patient turns are overwhelmingly short: across the deployed calls, roughly 59% are one or two words, against 13% in the full acted corpus, and a corpus accuracy gap remains even after the acted set is length-matched. On one-word turns, transcript-only detection does not exclude chance. Across five audio-capable language models, the transcript flags 3 or 4 of the 39 frustrated one-word turns in the 30 calls, and a classifier given only the word count reaches 57.3%. Whether audio helps depends on the model and on the instruction. Asked to attend to delivery, two Gemini 2.5 models gain 3.0 pp over the transcript across 30 calls (-1.0 to 7.5 pp). Of the turns they confidently clear, 6.9% were in fact frustrated for audio against 13.0% for the transcript, a difference of 6.1 pp (2.2 to 10.4 pp). Across the 30 calls, given one prompt identical for both inputs, no model's audio-text interval excludes zero. Under the input-specific prompts, a newer Gemini model and two smaller open models do worse with audio than with the transcript (-3.5, -8.6 and -10.6 pp). A WavLM probe trained on the clinical calls matches the best prompted model overall (71.6%) but misses every frustrated one-word turn in the 24 added calls. When a patient's turn is short, the transcript alone is not enough to rule out frustration.

Introduction

Clinical voice agents are now deployed to autonomously conduct routine follow-up calls [1], making automated triage increasingly necessary. Patient frustration is one state such systems should detect: frustrated patients may disengage, answer tersely, or hang up before clinically important information is said [15, 11]. How a patient sounds should change what happens next. A clinician who heard strain in an otherwise fine answer would probe further, and the agent should too. Whether these systems can tell that has happened is largely unknown. We study a per-turn decision: whether each patient utterance is frustrated or not.

The obvious way to build that triage is on the transcript: cheap, already produced by the speech-to-text stage, and on standard benchmarks about as good as audio [21, 6, 27]. Our own headline initially agrees: pooled across three models, transcript-only and audio-only accuracy differ by about a point. But that average misleads. Real clinical speech is dominated by short turns: roughly 59 % of patient turns are one or two words, leaving little lexical evidence while preserving the delivery. The clearest case is the same word carrying opposite states: in Figure 1, an identical “Some of the time” is labelled frustrated in one call and not frustrated in another, so a transcript-only system cannot tell them apart and clears both. The two inputs therefore do not fail on the same turns: transcript-only detection silently clears short frustrated turns where a triage system can least afford it. The pooled average also hides a split by model and by instruction: audio helps the more capable models only when they are asked to listen for delivery, and hurts a weaker one either way.

Two real calls with similar questions and responses

Figure 1. Two real calls with similar questions and responses. The patient’s response “Some of the time” is identical in wording but annotators labelled it frustrated in Call A and not frustrated in Call B. Transcript excerpts are verbatim.

This is structural. The deployed agent is cascaded: speech is transcribed, a language model reasons over the text, and a reply is synthesised. Paralinguistic cues such as hesitation, strain, and pauses are therefore discarded before reasoning [20, 17]. The transcript is what the agent uses today; audio represents what would be available if that delivery were retained.

The same failure is much less visible on the acted emotion corpora the field validates against, because they contain few short turns: 13 % of IEMOCAP’s utterances are one or two words, against 59 % of patient turns on deployed calls. Existing evaluations rarely test this on real deployed clinical speech. We therefore compare transcript-only and direct-audio frustration detection on real calls from a deployed clinical voice agent, with turn-level labels agreed by three annotators and a length-matched acted corpus as the control. We study frustration because it is a single, actionable state: audible, tied to a patient disengaging, and mapped to a clear response (probe, soften, or escalate).

Contributions

  • Real deployed clinical speech is a different distribution from acted benchmarks. Patient speech from real clinical voice-agent calls occupies a different distribution from the acted corpora used to benchmark emotion recognition: across the full set of deployed calls, roughly 59 % of patient turns are one or two words, against 13 % in the full acted corpus, and a corpus accuracy gap remains even after the acted set is matched on length (§ 5.1).

  • Transcript-only detection collapses on short turns, which dominate clinical speech. We give the first evidence on deployed clinical speech that transcript-only frustration detection fails to exclude chance on one-word turns: across three models, no one-word transcript interval clears 50 %, recall on FRUSTRATED one-word clinical turns is 9.5 % for every model (Table 9), and a classifier using only word count reaches 58.0 % (§ 5.2).

CorpusModelInputBal. acc.95 % CIRecall (count)Excludes chance
ClinicalGemini 2.5 FlashText54.8[47.5, 64.5]9.5 (2/21)
ClinicalGemini 2.5 ProText54.8[47.5, 64.5]9.5 (2/21)
ClinicalQwen2.5-Omni 7BText47.1[37.4, 57.8]9.5 (2/21)
ActedGemini 2.5 FlashText51.2[41.3, 62.5]11.1 (2/18)
ActedGemini 2.5 ProText53.4[44.1, 64.4]11.1 (2/18)
ActedQwen2.5-Omni 7BText52.4[40.5, 65.0]22.2 (4/18)
ClinicalGemini 2.5 FlashAudio60.0[50.0, 70.7]23.8 (5/21)
ClinicalGemini 2.5 ProAudio61.7[47.6, 73.5]61.9 (13/21)
ClinicalQwen2.5-Omni 7BAudio42.1[29.8, 56.1]38.1 (8/21)
ActedGemini 2.5 FlashAudio62.9[49.6, 74.9]38.9 (7/18)
ActedGemini 2.5 ProAudio70.3[55.0, 81.2]66.7 (12/18)✓
ActedQwen2.5-Omni 7BAudio53.9[39.3, 67.5]55.6 (10/18)

Table 9. Every condition on one-word turns, the bucket the short-turn claims rest on. Balanced accuracy with its 95 % Wilson interval, and recall as the share of frustrated one-word turns the condition flagged (21 such turns on the clinical set, 18 on the acted one). A tick marks the one condition whose interval excludes chance. The audio range quoted in § 5.2 spans both models and corpora: its floor is the open model on clinical audio, which sits below chance, and its ceiling is Gemini 2.5 Pro on acted audio.

  • Whether audio helps depends on the model. Asked to attend to delivery, the two frontier models improve with audio (4.0 pp on the clinical calls, −2.3 pp to 10.3 pp; 4.9 pp, 0.9 pp to 9.1 pp with the acted control pooled in), while a smaller open model is worse with audio (−12.7 pp, −22.0 pp to −3.3 pp; § 5.3).

  • The findings hold with more calls and more models. Across 30 calls and two added models, every transcript still flags only 3 or 4 of the 39 frustrated one-word turns, and the frontier pair’s transcript still wrongly clears more frustrated turns than its audio (13.0 % against 6.9 %). The frontier audio gain stays positive but its interval includes zero, and Gemini 3.8 Flash and both small open models are worse with audio (§ 5.4).

  • The instruction decides whether audio helps. Given one prompt identical for both inputs, the frontier gain disappears (−1.3 pp and 0.7 pp on the six calls), and across the 30 calls no model’s audio–text interval excludes zero. Gemini 2.5 Pro’s audio flags 4 of the 39 frustrated one-word turns in place of 23, so audio recovers short frustrated turns only when the model is asked to listen for delivery (§ 5.5).

  • A trained acoustic probe is not a fix for short turns. WavLM probes trained on the 30 calls match the best prompted model overall, but their one-word recall on the six calls does not carry to the other 24 (§ 5.6).

Related work

Speech emotion recognition, on acted and synthetic data. Identifying a speaker’s emotional state from how something is said, rather than what is said, is long established, but its evidence base is dominated by acted, read, and synthetic speech. IEMOCAP [3] remains the reference set, and RAVDESS [18], CREMA-D [5] and the MSP corpora [4] share its design of prompted, categorical, high-arousal emotion recorded in studio. Self-supervised speech encoders such as WavLM [7], which our trained probes build on, are themselves benchmarked for emotion recognition on IEMOCAP. EmoNet-Voice [26] grows the label set rather than the source, with forty categories over a fully synthetic voice set, and reports that agreement with human raters falls away outside high-arousal states. Recent large-scale audio evaluations likewise rely on generated speech: LALM-as-a-Judge [13] resynthesises dialogues, while AudioJudge [19] and SpeechJudge [Zhang et al., 2026] use read or synthetic speech. Recent work using real audio [24] instead evaluates audio quality rather than affect.

Audio LLMs perceive delivery but often privilege the words. On controlled benchmarks, audio LLMs fall back on the transcript when words and delivery disagree: VoxParadox [21] and LISTEN [6] report failures when affect is carried mainly by delivery, with related failures in deployed systems missing distress and sarcasm [2] and in an audio judge that collapses on single items [20]. Hear2Act [17] shows the signal is present but underused: forcing the model to describe the state in text before acting substantially improves downstream behaviour. Reliance on audio also varies across models [27]. These studies use synthetic, acted or adversarial speech rather than real clinical calls, and do not examine utterance length as an explanatory variable. Whether a newer or larger model in the same family listens better is rarely tested; we compare two generations of one family and two small open models on the same turns.

Utterance length as a confound. Prior work has treated length or duration as a predictive feature, identified sentence length as an annotation artefact, and shown that very short patient utterances are particularly difficult for ASR [25, 10] (Zolnoori et al. 2024). Yet length is rarely controlled when comparing text- and audio-based affect detection. We instead measure the clinical length distribution directly, include a length-only baseline, and construct a length-matched acted control. Self-supervised speech encoders such as WavLM give a trained alternative to prompting, and we test whether a probe on them trained on clinical speech generalises across calls.

Frustration on real clinical speech. Clinical voice-agent evaluation has largely focused on what was said rather than how it was delivered, even as end-to-end audio models are known to degrade over multi-turn dialogue [Tam et al., 2026]. Paralinguistic work in healthcare has examined synthetic voice bias [Tam and Chen, 2026], while models targeting frustration are typically evaluated on acted corpora [14]. The closest clinical neighbours each miss one axis: CBT-Audio [12] compares audio, text and fusion for distress, but the patient speech is acted; thymia [22] uses real clinical audio, but read speech and audio-only prediction; and WER-is-unaware [8] studies real deployed cataract-call audio but only through its transcript, identifying as an open problem a patient whose words indicate agreement while the voice conveys something else. To our knowledge, no prior work tests whether audio improves frustration detection over transcripts on spontaneous speech from deployed clinical calls.

Data and annotation

Source. We use two data sources: real patient turns from clinical calls and acted dialogue. Patient turns are drawn from recorded follow-up calls conducted by a deployed clinical voice agent on an ophthalmology pathway. All audio is telephone-band (8 kHz) spontaneous speech. Calls potentially containing frustration were selected from over 2000 calls using low patient satisfaction ratings and explicit mentions of not wanting to speak to an automated clinician. At the start of every call, patients are asked whether their data may be used for research, and only calls from patients who consented are included. The recordings are anonymised production calls from a proprietary internal dataset that cannot be shared, and contain no personally identifying information. We obtained IEMOCAP [3] through the official University of Southern California distribution under its research-use licence, using only the audio, transcripts, and emotion annotations. IEMOCAP serves as the acted benchmark and consists of scripted and improvised interactions elicited from actors in a controlled setting. Its emotion categories were binarised by mapping anger and frustration to FRUSTRATED, and happiness, sadness, neutrality, excitement, fear, surprise, disgust, and other to NOT_FRUSTRATED.

Annotation. Each patient turn is labelled FRUSTRATED, NOT_FRUSTRATED or INCONCLUSIVE, the last reserved for poor audio quality or genuine ambiguity. Annotators could listen to surrounding turns for context. Three annotators labelled each turn independently, with final labels determined by majority vote. The dataset comprises 150 patient turns from real calls (75 FRUSTRATED, 75 NOT_FRUSTRATED) and 150 actor turns from IEMOCAP (76 FRUSTRATED, 74 NOT_FRUSTRATED).

Table 1: Inter-annotator agreement on the clinical and acted evaluation sets. Pairwise raw agreement and Cohen’s κ\kappa are reported alongside overall unanimity and unanimity by utterance length. Inconclusive judgements are excluded from the relevant calculation rather than counted as disagreement.

ClinicalActed
Rawκ\kappaRawκ\kappa
Annotators 1–289.1%0.7888.3%0.76
Annotators 1–383.4%0.6785.9%0.72
Annotators 2–386.2%0.7389.2%0.78
Unanimous78.5%82.0%
Unanimous, by utterance length
1 word75.0%84.8%
2–377.3%84.6%
4–1082.5%88.9%
11+78.8%73.7%

Table 1.

Agreement is comparable across the two corpora. All 150 acted turns carry three independent judgements. Agreement and unanimity are close between the corpora (Table 1).

Short turns are not harder for annotators. On the clinical calls, unanimity is 75.0 % on one-word turns and 78.8 % on turns of eleven words or more (Table 1), giving little indication that short turns are intrinsically harder to label. Human listeners resolve a single-word turn about as reliably as a long one, so the short-turn failures we report later are not an artefact of noisier labels there. Overall annotator unanimity was 78.5 %, providing a human-agreement reference for model performance.

Evaluation set. Sections 5.1–5.3 use 150 turns from six calls labelled by all three annotators: 75 frustrated and 75 not frustrated, so that guessing confers no advantage and chance is 50 % by construction. A matched 150-turn set (76 frustrated, 74 not frustrated) from an acted corpus, rebuilt to the clinical length profile and re-validated by the same three annotators, serves as the control in § 5.1. The acted dataset used utterances from 45 of the 151 available IEMOCAP dialogues.

Extension to 30 calls. Sections 5.4–5.6 use a larger clinical set. We added 24 calls from the same deployed agent and pathway. Annotator 1 labelled every patient turn in them on the same criteria as the original six calls. Turns labelled inconclusive are excluded, leaving 564 turns, of which 65 (11.5 %) are frustrated. No turns were dropped to balance the classes. Together with the 150 balanced turns from the six calls, the full clinical set has 714 turns from 30 calls, 140 of them frustrated (19.6 %).

Experimental setup

Models. We evaluate five multimodal models that accept audio directly, with no fine-tuning or task-specific training. Gemini 2.5 Flash, Gemini 2.5 Pro and Gemini 3.8 Flash are accessed via Vertex AI. The two open models are Qwen2.5-Omni 7B [Xu et al., 2025], served locally through llama.cpp at Q8.0 quantisation, and Voxtral Mini 3B [16]. All models use temperature 1.0, the Gemini 2.5 default. We keep it for every model rather than decode greedily, because greedy decoding makes repeats identical and so hides run-to-run variation instead of removing it. Repeats then measure that variation (Appendix § 9.§ 1). Other sampling parameters remain at provider defaults. Appendix § 9.§ 2 gives how each model is accessed and called. The acted corpus is scored with Gemini 2.5 Flash, Gemini 2.5 Pro and Qwen2.5-Omni only, so Sections 5.1–5.3 report those three.

Conditions. Each turn was presented twice, as audio and as transcript only, under two prompt conditions (Appendix § 9.§ 3). The input-specific prompts, used for the main results, direct the audio condition to delivery and the transcript condition to the words. The identical prompt gives both inputs the same instruction, word for word. Every model was run under both conditions on all 30 calls. The model saw the patient turn in isolation, unlike the annotators (§ 3). Two further experiments on the six calls add context: up to six preceding patient turns and, separately, the agent’s preceding turn, which the patient’s turn answers (§ 5.2). Responses comprised a label constrained to frustrated or not frustrated and a free-text rationale used for qualitative analysis only.

Trained probes. As a trained baseline, we fit probes on frozen WavLM Base+ [7]: one intermediate layer (of layers 6–9 of 12), pooled over the turn, with a logistic regression or an RBF support vector machine on top. One set is trained only on acted IEMOCAP speech (Appendix § 13), the other on the 30 clinical calls, split into five folds by patient and scored out of fold (Appendix § 13.§ 5). Layer and pooling are chosen by cross-validation inside each training fold, and three split seeds give three predictions per turn, combined by majority as for the language models.

Metrics. Every turn is classified three times independently in each condition, and the turns’ prediction is the majority of those three. Balanced accuracy, the mean of recall and specificity, is reported throughout. Turn length is measured in words after stripping bracketed annotation markers, and bucketed as 1, 2–3, 4–5, 6–10, 11–20 and 21 or more words. The six-call clinical set and the acted control are balanced overall, but the turn length buckets we analyse within them are not, and plain accuracy inside an imbalanced bucket rewards predicting whichever class dominates it. Chance is 50 % by construction. A single balanced accuracy gets a 95 % Wilson interval and a difference between conditions a 10 000-draw bootstrap over turns, both on the majority-vote predictions. Redrawing each turn’s three repeats inside the bootstrap as well moves no bound by more than 2 pp and changes no conclusion except the acted frontier pair, whose interval then just includes zero (Appendix § 9.§ 1). These intervals are used in Sections 5.1–5.3 and their appendices. Sections 5.4–5.6 and Appendix § 16 instead use a 10 000-draw bootstrap over whole calls, because turns from one call are not independent.

Results

Real clinical speech is a different distribution from acted benchmarks

Figure 2 shows the distribution of utterance lengths from the beginning to the end of the dialogues for 2073 real clinical calls (patient utterances only) in comparison to the 151 dialogues present in IEMOCAP (utterances of both actors).

Performance drops on the clinical calls, but telephony bandwidth is not the explanation. Models score lower on the clinical calls than on the acted set in all six model-and-input cells. Pooled across the three models the difference is −8.1 pp, with a 95 % interval of −15.6 pp to −0.8 pp; pooled across the two frontier models it is −8.7 pp (−16.7 pp to −0.9 pp). Figure 11 gives every cell and the pooled row. To test whether the audio-only differences were attributable to recording quality (16 kHz for the acted recordings versus 8 kHz for the real calls), we evaluated the acted corpus’ audio-only condition both in its original form and after applying the full telephony chain (8 kHz, 300 Hz to 3400 Hz bandpass, and μ\mu-law encoding). Performance changed only slightly: from 83.3 % to 82.2 % for one model and from 80.0 % to 83.3 % for the other. As the effect was small and changed direction across models, channel quality is unlikely to explain the performance gap between the datasets. Figure 12 in Appendix § 10.§ 2 presents the full comparison.

Figure 11. Comparison of real against acted, as differences with 95 % intervals. Left of zero means the real calls are harder. Rows below the rule pool across models and are the primary comparison; a green interval excludes zero.

Figure 12. The acted corpus as recorded and through the full telephony chain, on 90 of the 150 acted turns (45 frustrated, 45 not frustrated). The effect is small and changes sign between models, so the clinical–acted gap cannot be attributed to channel bandwidth.

The acted corpus, matched on length. Figure 2 illustrates an acted set whose length profile differs sharply from the clinical one, so a performance gap between them could be a length effect rather than a domain effect. The acted 150 turn evaluation set extracted from IEMOCAP was therefore built to have a word count distribution more closely matched to that of the real clinical calls; Table 10 shows the distribution of both evaluation sets.

Figure 2. Distribution of word counts across over 2000 clinical conversations. Each row is one conversation, ordered by mean utterance length; the horizontal axis runs from the start of the conversation to its end. Clinical calls are dominated by 1-2 patient turns throughout. The acted corpus contains proportionally under a quarter as many.

Turn lengthClinical nnClinical frustrated / notActed nnActed frustrated / not
1 word4721 / 264118 / 23
2–3248 / 16204 / 16
4–5105 / 5115 / 6
6–103417 / 172814 / 14
11–202216 / 62414 / 10
21+138 / 52621 / 5
All15075 / 7515076 / 74

Table 10. Turn counts by utterance length for both evaluation sets, with the frustrated and not-frustrated split in each bucket.

Table 2 shows the best model’s (Gemini 2.5 Flash) balanced accuracy by turn length for both evaluation sets when given only audio and only text. An overall corpus gap of about eight points remains after length matching, so the gap between spontaneous and acted speech is not just a matter of turn length. At one word, however, the gap disappears: the two corpora are within four points, and text performs slightly worse on the acted corpus (51.2 %) than on the clinical one (54.8 %), with recall of 11.1 % and 9.5 %, respectively (Table 9 in Appendix § 11.§ 1). The short-turn failure therefore generalises beyond the clinical recordings and patients: it reappears on the benchmark once similarly short turns are present. Above one word, where buckets hold 10 to 34 clinical turns each, the gap varies and favours the acted corpus in seven of the ten audio and text comparisons (Table 2).

Table 2: Best model’s balanced accuracy by turn length, clinical against acted, 150 turns each with matched length profiles. The clinical set balanced 75/75 and the acted set 76/74. nn gives the clinical and acted turns in each bucket, and Figure 15 gives each bucket’s interval.

Turn lengthnnAudio ClinicalAudio ActedText ClinicalText Acted
1 word47 / 4160.0 %62.9 %54.8 %51.2 %
2–3 words24 / 2071.9 %59.4 %56.2 %68.8 %
4–5 words10 / 1160.0 %71.7 %90.0 %61.7 %
6–10 words34 / 2873.5 %85.7 %64.7 %85.7 %
11–20 words22 / 2465.6 %81.4 %70.8 %77.9 %
21+ words13 / 2680.0 %80.0 %63.7 %77.6 %
All150 / 15071.3 %79.4 %66.7 %74.8 %

Table 2.

Figure 15. Balanced accuracy by turn length, with 95 % Wilson intervals, for every model and input. Tick labels give the frustrated and not-frustrated counts in each bucket: the scarcer of the two sets the interval width, which is why some larger buckets are less precisely measured than smaller ones.

On short turns, transcript-only detection collapses to chance; audio degrades but does not

On one-word turns (47 of the 150 clinical turns and 41 of the acted), no transcript-only condition has an interval that excludes chance. The strongest reach 54.8 % balanced accuracy and the weakest 47.1 %. Recall on those turns falls to between 9.5 % and 22.2 % (Table 9): these systems almost never flag a single-word answer, whatever its delivery. Audio degrades on the same turns without collapsing for some models, reaching 42.1 % to 70.3 % with recall between 23.8 % and 66.7 %, but not uniformly (Table 9): the floor is the open model on clinical audio, which sits below chance, while Gemini 2.5 Pro on the acted corpus is the only one of the twelve conditions whose interval excludes chance outright. Figure 13 shows all twelve conditions against the chance line.

Figure 13. One-word turns: not one transcript-only condition beats a coin flip. Every model and input, with 95 % Wilson intervals against the chance line. Six transcript conditions, none excluding 50 %.

Predictions track length more than the labels do. The rate at which each condition predicts frustration rises with turn length far more steeply than the true rate does (Figure 3). On the clinical calls the true rate varies by 39 pp across length buckets, whereas frustration prediction rate in the transcript condition varies by a minimum of 51 pp with length, and peaks at a variation of 88 pp. Two audio conditions vary by 33 pp and 21 pp, at or below the true variation, so access to delivery partly breaks the dependence on length (Table 11).

Figure 3. Predicted-frustration rate against turn length for all three models, against the true rate in black. Every transcript condition starts far below the true rate on the shortest turns and climbs more steeply than it.

InputModelClinicalActed
True rate39.460.8
TextGemini 2.5 Flash72.774.9
TextGemini 2.5 Pro88.177.3
TextQwen2.5-Omni 7B50.970.8
AudioGemini 2.5 Flash66.978.5
AudioGemini 2.5 Pro32.958.5
AudioQwen2.5-Omni 7B20.855.8

Table 11. Spread of the predicted-frustration rate across the six turn-length buckets, in percentage points, against the spread of the true rate. Larger than the true spread means a condition varies with length more than frustration itself does.

Strip delivery from a one-word turn and almost no length signal and almost no lexical signal remain. A logistic regression given only the word count, and never the audio, reaches 58.0 % on the clinical set (dialogue-disjoint train/test split; Appendix § 13.§ 1), and is stable across resamples: 57.9 % ± 0.3 pp across five held-out-session folds. The evidence here is correlational: it establishes that predictions track length more strongly than the labels do, not that length is the causal input.

Standard acoustic features cannot be measured on these turns either. We recomputed each of the 88 eGeMAPS parameters [9] on short windows drawn from inside longer turns, so spread across windows is measurement noise rather than a difference between turns: at 0.5 s, the median one-word turn, formant-frequency variability reaches an intraclass correlation of 0.17 and no parameter reaches 0.70 (Appendix § 14). The models are not given these features, so this does not explain their behaviour, but an interpretable acoustic detector is not an available substitute at these turn lengths.

The failure is a safety failure. A triage system does not report accuracy; it decides which calls a clinician never sees. A confidently cleared turn is one that both frontier models called not frustrated on every repeat prediction. Of the turns each input modality confidently clears, the transcript condition clears 87 and 37.9 % of them were in fact frustrated. Audio clears 45 and 17.8 % were frustrated (Figure 4). That lower rate depends on the audio prompt directing the model to delivery: under a prompt identical for both inputs, audio clears 81 turns of which 33.3 % were frustrated, against 94 and 38.3 % for the transcript (§ 5.5). These are shares within a sample balanced 75/75 and screened for dissatisfaction, so the absolute levels overstate deployment prevalence; the text-versus-audio contrast, not the absolute rate, is what the comparison supports. The wrongly cleared turns concentrate in the shortest ones: 19 of the 33 are one or two words, against short turns making up 35 % of frustrated turns overall (Table 3, Figure 14). Neither the threshold nor a single call carries this: sweeping the clearing threshold leaves text between 37.5 % and 40.0 % wrong while audio rises from 17.8 % to 31.7 %, and holding out each call in turn moves neither condition far from its full-sample value (Figure 18).

Turn lengthFrustrated turnsSilent failuresMissed
nnsharennshare
1–2 words2634.7 %1957.6 %73.1 %
3–10 words2533.3 %1030.3 %40.0 %
11+ words2432.0 %412.1 %16.7 %
All75100 %33100 %44.0 %

Table 3. Where the transcript condition’s silent failures fall. A silent failure is a confidently cleared turn that was labelled as frustrated by the human annotators (a falsely-cleared turn). Short turns are a third of the frustrated turns and well over half of the silent failures, and the miss rate falls as turns lengthen. Figure 14 plots the two shares.

Figure 4. Turns each input modality confidently clears, and how many of them were in fact frustrated. The transcript clears 87 of 150 turns and 37.9 % of those were frustrated; audio clears 45 and 17.8 % were. The audio rate holds only when its prompt directs the model to delivery (§ 5.5).

Figure 14. Silent failures concentrate in short turns for the transcript condition. Turns wrongly cleared, by turn length. The transcript condition’s failures concentrate in the shortest turns.

Figure 18. The clearing gap is neither a threshold artefact nor the product of one call. Left: sweeping the clearing threshold τ\tau leaves text between 37.5 % and 40.0 % wrong, so loosening τ\tau makes text clear more turns without making it safer; audio rises from 17.8 % to 31.7 %, closing much of the gap as the criterion weakens. Right: holding out each call in turn moves neither condition far from its full-sample value, shown dashed.

Context does not rescue the transcript. To understand if the text and audio conditions were handicapped by seeing only one turn in isolation, we ran a series of context experiments. We swept the number of preceding patient turns supplied from zero to six. Text improves by roughly two points and then plateaus: text given six turns of prior conversation still scores below audio given none (68.7% against 70.7%). Audio does not benefit either, and on short turns degrades by 14.7 pp across the sweep, consistently across repeats. Figure 22 in Appendix § 12.§ 4 gives the full curves. Giving both inputs the agent’s preceding turn, which the patient’s turn answers, does not rescue the transcript either: its balanced accuracy changes by between −2.0 pp and 1.3 pp across the three models, and recall on frustrated one-word turns rises only from 2 to between 3 and 5 of 21. With the agent’s turn, Gemini 2.5 Pro’s audio recovers 15 of the 21. Appendix § 12.§ 2 gives the prompts, every model and input with and without the agent’s turn (Table 12), and audio minus transcript in each run (Table 13).

RunModelInputBal. acc. [95% CI]Change [95% CI]FrustratedNot frustratedOne-wordOne-word, other calls
Patient turn onlyGemini 2.5 FlashText66.7 [59.4, 73.1]–36/7564/752/210/9
Gemini 2.5 ProText64.0 [56.4, 70.8]–37/7559/752/210/9
Qwen2.5-Omni 7BText65.3 [57.8, 71.9]–37/7561/752/210/9
Gemini 2.5 FlashAudio71.3 [64.1, 77.4]–41/7566/755/211/9
Gemini 2.5 ProAudio67.3 [59.4, 74.1]–49/7552/7513/212/9
Qwen2.5-Omni 7BAudio52.7 [44.8, 60.3]–36/7543/758/212/9
With agent’s preceding turnGemini 2.5 FlashText65.3 [58.3, 71.6]−1.3 [−6.7, +4.0]32/7566/753/212/9
Gemini 2.5 ProText65.3 [56.7, 70.2]+1.3 [−5.3, +8.0]40/7558/755/214/9
Qwen2.5-Omni 7BText63.3 [55.7, 70.2]−2.0 [−9.3, +5.3]37/7558/753/211/9
Gemini 2.5 FlashAudio65.3 [58.4, 71.5]−6.0 [−12.7, +0.7]31/7567/753/212/9
Gemini 2.5 ProAudio72.0 [64.4, 78.2]+4.7 [−3.3, +12.7]47/7561/7515/215/9
Qwen2.5-Omni 7BAudio53.3 [49.7, 64.5]+4.7 [−5.3, +14.7]32/7554/751/211/9

Table 12. The agent’s preceding turn added as context, on the clinical set, beside the main results on the patient turn only. Balanced accuracy with its 95 % Wilson interval; the paired change from the main results, with its paired-bootstrap interval; frustrated and not-frustrated turns classified correctly, of 75 each; and recall on the 21 frustrated one-word turns, overall and on the 9 outside the call that holds the other 12.

ModelPatient turn onlyWith agent’s preceding turn
Gemini 2.5 Flash+4.7 [−2.0, +11.3]+0.0 [−6.0, +6.0]
Gemini 2.5 Pro+3.3 [−5.3, +12.0]+6.7 [−1.3, +14.7]
Qwen2.5-Omni 7B−12.7 [−22.0, −3.3]−6.0 [−14.0, +2.0]

Table 13. Audio minus transcript in balanced accuracy (pp) on the clinical set, on the patient turn only and with the agent’s preceding turn, with paired-bootstrap 95 % intervals.

Figure 22. Supplying preceding patient audio degrades short-turn accuracy while leaving long turns intact; text moves in the opposite direction and gains little.

Audio’s improvement on transcripts is model dependent

Figure 5 shows the difference in the models’ performance when given audio versus transcripts. Pooled across the two frontier models across both corpora, audio outperformed transcript-only input by 4.9 pp (0.9 pp to 9.1 pp). On the clinical calls, the individual model estimates were 4.7 pp (−2.0 pp to 11.3 pp) and 3.3 pp (−5.3 pp to 12.0 pp); however, neither confidence interval excluded zero at this sample size.

Figure 5. Comparison of audio against the transcript, as differences with 95% intervals from a 10 000-draw bootstrap over turns. A filled marker marks an interval that excludes zero. No frontier model’s audio gain excludes zero on its own. Pooled, the frontier pair’s gain excludes zero on the acted corpus and on both corpora together, though not on the real calls alone, and adding the open model removes it. The open model is worse with audio on the real calls, by an interval that excludes zero.

In contrast, the smaller open model performed 12.7 pp worse with audio than with transcripts (−22.0 pp to −3.3 pp) on the clinical calls. Including this model in the pooled analysis reduced the overall audio advantage to 0.7 pp. Together, these results suggest that the benefit of audio input depends on the model. Figure 19 compares all models, input modalities, and corpora on a common scale.

Figure 19. Balanced accuracy for every model and input on both corpora, with 95 % Wilson intervals and the number of turns on each bar. Interval widths differ because a proportion’s variance peaks at one half, so cells nearer chance are measured less precisely.

The frontier gain depends on the instruction. Our audio and transcript prompts differ in what they direct the model to use (Appendix § 9.§ 3). Given one prompt identical for both inputs, the frontier difference disappears. Transcript-only detection fails on short turns under every prompt we tried; audio recovers them only when asked to listen for delivery (§ 5.5).

The two inputs are complementary. Because audio and the transcript get different turns right rather than failing together, an oracle that picked the better input for each turn would reach 78.0%; a simple vote across the models and inputs reaches only 66.0%, below either frontier model’s audio alone, so the complementarity is real but a naive combination does not capture it (Appendix § 12.§ 1).

The findings hold with more calls and more models

We repeat every comparison within the clinical calls made without context on all 714 turns of the 30 calls (§ 3), and add two models: Gemini 3.8 Flash, the newer model in the frontier pair’s family, and Voxtral Mini 3B, a second small open model. Figure 6 shows balanced accuracy for each model and input on the six, the 24 and the 30 calls, and Appendix § 16 gives the remaining measures.

Figure 6. Balanced accuracy on the six original calls, the 24 added calls and all 30, with 95 % bootstrap intervals over calls. Audio is solid and filled, the transcript dotted and hollow. The last panel shows the two WavLM probes trained on the 30 calls (§ 5.6). The bar and band mark the word-count-only baseline and its interval.

The results on 30 calls follow those on six. Every model’s transcript flags 3 or 4 of the 39 frustrated one-word turns (Figure 30 in Appendix § 16). The frontier pair’s transcript wrongly clears more frustrated turns than its audio: 13.0 % of 578 cleared turns against 6.9 % of 333, a difference of 6.1 pp (2.2 pp to 10.4 pp; Figure 7). Its audio gain stays positive, at 3.0 pp (−1.0 pp to 7.5 pp). Qwen2.5-Omni is again worse with audio, by 8.6 pp (−14.1 pp to −1.6 pp).

Figure 7. Share of the frontier pair’s confidently cleared turns that were frustrated, on the six calls (33 of 87 for the transcript, 8 of 45 for audio), the 24 (42 of 491, 15 of 288) and all 30 (75 of 578, 23 of 333), with 95 % bootstrap intervals over calls. Each part’s frustrated base rate is marked: 50 %, 11.5 % and 19.6 %.

Figure 30. Frustrated one-word turns flagged, out of 39 in the 30 calls, for each model and input and for the two WavLM probes trained on the 30 calls, with 95 % bootstrap intervals over calls. Beside each bar: the split between the six and the 24 calls, and false alarms among the 278 one-word turns that were not frustrated. The dashed line marks the three turns every transcript flags.

The two added models split. Voxtral behaves like Qwen2.5-Omni and is worse with audio, by 10.6 pp (−16.5 pp to −3.6 pp). Gemini 3.8 Flash reverses the pattern of the 2.5 models in its own family: it is 3.5 pp worse with audio (−6.0 pp to 0.0 pp), and from audio it flags 4 of the 39 frustrated one-word turns, against 9 and 23 for the 2.5 models. Gemini 2.5 Pro’s 23 come with 77 false alarms among the 278 one-word turns that were not frustrated.

On the 24 added calls, whose turns keep their natural prevalence, predictions again track length more than the labels do (Figure 8). The true rate varies by 21.2 pp across length buckets. The transcript’s predicted rate varies by 26.3 pp to 40.4 pp for four of the five models, and by 17.3 pp for Gemini 3.8 Flash. On one-word turns, 6.7% of which are frustrated, every transcript but Qwen2.5-Omni’s predicts frustration on at most 0.7%; Gemini 2.5 Pro’s audio predicts it on 28.5%.

Figure 8. Predicted-frustration rate by turn length on the 24 added calls (564 turns), against the true rate in black, for the five models and the two WavLM probes trained on the 30 calls. Audio is solid, the transcript dotted.

The instruction decides whether audio helps

Our audio and transcript prompts differ in what they direct the model to use (Appendix § 9.§ 3), so a difference between the inputs could come from the instruction rather than the input. We therefore reran all five models on all 30 calls with one prompt identical for both inputs (Table 4; Figure 31 in Appendix § 16 plots the differences).

ModelPromptsAudioTextAudio − textOne-word recall audio / text
Gemini 2.5 FlashInput-specific71.366.7+4.7 [−2.6, +10.9]5 / 2
Identical66.067.3−1.3 [−7.7, +4.7]0 / 2
Gemini 2.5 ProInput-specific67.364.0+3.3 [−6.2, +11.9]13 / 2
Identical67.366.7+0.7 [−7.0, +6.8]1 / 2
Gemini 3.8 FlashInput-specific64.068.7−4.7 [−8.7, −0.5]1 / 2
Identical66.767.3−0.7 [−5.1, +2.6]0 / 2
Qwen2.5-Omni 7BInput-specific52.765.3−12.7 [−19.5, −7.0]8 / 2
Identical56.768.7−12.0 [−24.3, −1.5]2 / 3
Voxtral Mini 3BInput-specific52.766.0−13.3 [−21.2, −2.8]0 / 2
Identical62.065.3−3.3 [−11.6, +7.2]1 / 8

Table 4. Input-specific prompts against one prompt identical for both inputs, on the six calls, for all five models. Balanced accuracy, the paired audio-minus-text difference in percentage points with its 95 % bootstrap interval over calls, and recall on the 21 frustrated one-word turns.

Figure 31. Audio minus transcript balanced accuracy for the five models on the six, the 24 and all 30 calls, under the input-specific prompts (filled) and the identical prompt (hollow), with 95 % bootstrap intervals over calls.

On the six calls, the identical prompt makes the frontier gain disappear: −1.3 pp (−7.7 pp to 4.7 pp) for Gemini 2.5 Flash and 0.7 pp (−7.0 pp to 6.8 pp) for Gemini 2.5 Pro. Their audio flags 0 and 1 of the 21 frustrated one-word turns, against 5 and 13 when asked to listen, and the transcript’s 2. The transcript barely moves. Frontier clearing converges the same way: audio clears 81 turns of which 33.3% were frustrated, against 94 turns and 38.3% for the transcript. Gemini 3.8 Flash moves from −4.7 pp (−8.7 pp to −0.5 pp) to −0.7 pp (−5.1 pp to 2.6 pp). The open models diverge. Qwen2.5-Omni’s deficit does not depend on the prompt (−12.7 pp and −12.0 pp). Voxtral’s shrinks from −13.3 pp (−21.2 pp to −2.8 pp) to −3.3 pp (−11.6 pp to 7.2 pp), because its audio rises from 52.7% to 62.0%.

The 30 calls show the same pattern. Under the identical prompt, no model’s audio–text interval excludes zero: −0.1 pp, 0.9 pp, −0.7 pp, −5.7 pp and −1.2 pp for Gemini 2.5 Flash, 2.5 Pro, 3.8 Flash, Qwen and Voxtral. Pro’s audio flags 4 of the 39 frustrated one-word turns in place of 23. Audio’s clearing advantage mostly disappears: 12.0% of the 501 turns audio clears were frustrated, against 13.2% for the transcript.

Trained acoustic probes

A probe on frozen speech representations needs no instruction to attend to delivery. We score two kinds on the 30 calls (Figure 9). The first is trained only on acted speech, the probe of Appendix § 13. The second is trained on the clinical calls and scored out of fold (§ 4).

Figure 9. The four WavLM probes on the six, the 24 and all 30 calls: trained on IEMOCAP with native and band-matched audio, scored at the acted cut-off (filled) and with a threshold chosen leaving one call out (hollow), and trained on the 30 calls with a logistic regression and an SVM. Left: balanced accuracy with 95 % bootstrap intervals over calls. Right: frustrated one-word turns flagged, of 21, 18 and 39; the band-matched counts average five folds. Reference lines: Gemini 2.5 Flash audio and word count only.

The acted-trained probe does not carry to the added calls. On the six calls the band-matched WavLM SVM reaches 74.0% and flags 14.0 of the 21 frustrated one-word turns on average across folds. On the 24 added calls it reaches 62.8 % (57.5 % to 69.5 %) and flags 7.8 of 18. With its threshold chosen on held-out clinical calls, it flags 3.8 of 18.

Training on clinical calls gives the best overall accuracy, but not on short turns. Trained on the 30 calls, the SVM reaches 71.6 % (63.2 % to 78.2 %) and the logistic regression 70.3 % (59.8 % to 77.1 %), level with Gemini 2.5 Flash on audio (70.2 %) and well above word count. Most of that comes from the six calls (76.0 % and 79.3 %); on the 24 they reach 63.0 % and 56.6 %. The logistic regression flags 15 of the 21 frustrated one-word turns in the six calls and none of the 18 in the 24; the SVM flags 8 and none. Of the turns the SVM confidently clears across its three split seeds, 10.2 % were frustrated, between the frontier pair’s audio (6.9 %) and transcript (13.0 %). The trained probes get long turns right and, on the added calls, clear the short ones much as the transcript does.

The six calls are a balanced, triple-annotated sample and the 24 a single-annotated sample at 11.5 % prevalence, so the drop from the six to the 24 mixes call difficulty, label source and class balance.

Discussion

A transcript-only system clears short frustrated turns silently, and with high confidence. On a short turn there is almost nothing to read, and the absence of explicit textual evidence reads as calm, so the miss is silent: the turn is cleared with nothing to flag it. Accuracy is therefore the wrong measure. Two systems can tie on balanced accuracy yet clear entirely different turns: text clears far more than audio and, on the six calls, is wrong about more than a third of them, and those are the short ones, often at a call’s onset where an escalation still matters. The starkest case is a patient answering “No.” with opposite states minutes apart (Appendix § 15.§ 1): identical words, and text clears both.

The failure tracks turn length, not our patients or recordings. It reappears on the acted benchmark as soon as that benchmark is made to contain short turns, which argues against our dataset, telephone channel, or clinical setting as the explanation, though length remains confounded with response type and dialogue position in both corpora. We would expect the same wherever a transcript-based system must read affect from brief spoken answers, but we show it only for frustration on clinical calls.

Standard emotion benchmarks such as IEMOCAP cannot show that transcript-only detection fails on short turns, because they contain almost none. An evaluation run on them dilutes the failure until it looks like a pass, which is more dangerous than a test that plainly does not apply, because it still reads as validation. The fix is cheap: report the length distribution of the turns evaluated, test on the lengths a system will meet in deployment, and add a length-only baseline. On our data word count alone reaches 57 % to 58 %; a model that cannot beat word count has not shown it is using the voice.

Adding audio is not automatically an upgrade; the model and its instruction decide whether it helps. Directed to delivery, it raises the two Gemini 2.5 models by about 4 pp on the six calls and 3 pp across 30, estimates whose intervals include zero. Under a prompt identical for both inputs, the gain is gone for every model we tested. The newer model in the same family and both small open models do worse with audio than without. These models receive the signal and often do not act on it, so the limit lies in using delivery, and it varies by model and by prompt. An audio system should be measured on the target distribution, under the prompt it will run with, before it is assumed better. That audio adds over the transcript for clinical affect is consistent with prior work on acted distress [12]. Our contribution is not that audio helps. It is that on real deployed speech, whatever margin audio offers sits in the one- to three-word turns that a transcript silently clears, and is realised only when the model is directed to use it.

For deployment, a short patient turn should not be cleared on the transcript alone; it is a candidate for routing to an audio model or a person, subject to the review load that creates. No system we tested makes that route safe on its own. Directed audio from the strongest model recovers most frustrated one-word turns, but with many false alarms. Smaller and newer models recover few. Trained probes recover them on some calls and not on others (§ 5.6). Our claim is about the measurement and the regime, not a detector ready to ship.

A successor model is not a drop-in replacement for listening. The two Gemini 2.5 models that gain from audio are scheduled for retirement. Their successor, Gemini 3.8 Flash, reads transcripts as well as Gemini 2.5 Flash (67.4 % against 66.6 % on 30 calls) but does worse from audio, and catches 4 of 39 frustrated one-word turns where 2.5 Flash catches 9 and 2.5 Pro 23. A system that relies on audio for short turns has to be re-measured whenever its model changes, because the property it relies on is not preserved across versions.

Training on clinical speech improves the average without solving short turns. A WavLM probe trained on the 30 calls matches the best prompted model overall, and it needs no instruction. But its one-word success on the six calls does not reappear on the 24 added calls, where it flags none of the 18 frustrated one-word turns. The six calls differ from the 24 in label source, class balance and prevalence, so we cannot attribute the drop to any one of them. A trained probe, too, has to be shown to catch short frustrated turns on the calls it will see, not only to score well overall.

Limitations. The data come from a single deployed agent on one clinical pathway, in one language and healthcare system, and judge a single affective state as a binary. The 24 added calls were labelled by Annotator 1 alone, so their labels carry no agreement estimate, and their transcripts are uncorrected Whisper output [23]. The 30-call set combines a balanced sample from the six calls with every labelled turn of the 24, so its pooled prevalence describes neither, and we rest prevalence-sensitive claims on the 24. The acted control, telephony, context, agent-turn, minimal-pair and onset experiments cover only the six calls and the three original models. We evaluate five models zero-shot, three from one provider, and train only frozen-encoder probes. The Gemini 2.5 models are scheduled for retirement. Appendix A expands these.

Conclusion

A deployed clinical voice agent decides what to escalate from the transcript, not the voice, and that choice fails precisely where clinical speech lives. Patient turns are short: roughly 59 % are one or two words. On those turns transcript-only frustration detection stops working. It does not read them wrong so much as decline to judge them, which in a binary decision reads as calm. This holds on six triple-annotated calls and on 30, for each of five models, which flag 3 or 4 of the 39 frustrated one-word turns, the same three for every model. The cost is a safety one rather than an accuracy one, because a triage system decides which calls a clinician never sees. Across 30 calls, 13.0 % of the turns the transcript confidently clears were frustrated, against 6.9 % for audio directed to delivery. Audio is no fix in itself. Directed to delivery, it helps two frontier models by about three points; without that instruction the gain disappears, and a newer model and two smaller ones do worse with audio under the directed prompt. A probe trained on the clinical calls matches the best prompted model overall, yet flags none of the 18 frustrated one-word turns in the 24 added calls. The failure is much less visible on acted corpora, which are dominated by longer turns, and reappears when the acted set is made short. The consequence is narrow and firm: an evaluation that does not report the length distribution of its turns cannot show that a transcript-based system will hold up in deployment, and a short patient turn should not be cleared on the transcript alone.

Ethics statement

The real-call corpus is 30 recordings from a deployed clinical voice agent, collected during routine service delivery. Patients consented at the point of collection to the use of their data for research and service improvement. The identity-verification section of each call was removed from the audio and transcript before analysis. Real-call audio was processed only on local machines and through Vertex AI under a data processing agreement with Google. No real-call audio, transcripts or patient speech are released; the paper reports aggregate results only. IEMOCAP is a corpus of acted dyadic conversations, used under its licence agreement.

Extended limitations

Scale and scope. The six-call evaluation set is 150 turns, and the 30-call set 714 turns, all from a single clinical pathway, in one language and one healthcare system. Per-call variance is substantial in both. The onset analysis rests on six items and is reported as an illustration.

Selection. The calls were originally screened using indicators of dissatisfaction, including low feedback scores, so the corpus may over-represent one kind of frustration. Reported frustration rates should not be read as population estimates. The within-corpus comparisons that carry our claims are unaffected, since every condition faces the same data.

Annotation ceiling. Human agreement bounds what any model result can demonstrate. With κ\kappa between 0.6 and 0.8, model performance should be read against that ceiling rather than against perfect agreement.

Model coverage. We evaluate five models zero-shot; three share a provider family. We fine-tune no acoustic model end to end, reporting only frozen-encoder probes (§ 5.6, Appendix § 13). Gemini 2.5 Flash and Pro are scheduled for deprecation, and their successor in our evaluation, Gemini 3.8 Flash, uses audio differently (§ 5.4).

Transcript quality. The text condition is only as good as the transcript it receives. On the six calls, every turn’s transcript is either agreed by two independent ASR systems or adjudicated by a human against the audio, and the onset examined in Appendix § 15.§ 2 was verified by listening because the production transcript for it is wrong. The 24 added calls use Whisper large-v3 transcripts. On a one-word turn a single substituted word changes the affective reading entirely.

Annotation of the added calls. The 24 added calls were labelled by Annotator 1 alone, so they carry no agreement estimate; the six triple-annotated calls give the agreement reference.

Method and configuration

Experimental configuration

Each (model, modality, turn) combination is run three times and the turn’s prediction is the majority vote of the three. The three runs differ in balanced accuracy by a standard deviation of 0.4 pp to 4.4 pp on the six calls and 0.1 pp to 2.6 pp on the 30, across the five models, and span at most 8.7 pp, for the open model on audio on the six calls (Figure 10), and in places this exceeds the effect sizes under study, so every headline number is a majority vote over repeats and every directional claim is additionally checked per repeat. Redrawing the repeats inside the bootstrap gives the acted frontier pair’s audio advantage of 5.9 pp an interval of −0.1 pp to 10.9 pp rather than 0.6 pp to 11.5 pp; the frontier pair pooled across both corpora stays above zero at 0.6 pp to 9.1 pp.

Balanced accuracy of each of the three runs for real clinical calls and IEMOCAP

Figure 10. Balanced accuracy of each of the three runs, scored on its own against the human labels; the marker shows the majority vote of the three. Standard deviations range from 0.4 pp to 4.4 pp.

How each model is run

Table 5 gives how each model is accessed and called. Every audio request sends the turn’s clip as a WAV file cut from the call recording, without resampling. The three Gemini models are called through the Google Gen AI SDK on Vertex AI with a JSON response schema. The two open models run locally. Qwen2.5-Omni runs in llama.cpp’s llama-server through its OpenAI-compatible chat endpoint, with the clip sent inline as base64 and a JSON schema on the response. Voxtral runs in Hugging Face Transformers with no response schema; the prompt ends with a request for the same JSON object, and the reply is parsed from its first JSON block, falling back to the label word. API and server errors get up to four attempts with exponential backoff; a turn whose reply cannot be parsed counts as a failed request.

Prompts

The prompts were held as consistent as possible across modalities, and differ in what they direct the model to use. Each is given exactly as sent to the model.

ModelIdentifierAccessWeightsSampling
Gemini 2.5 Flashgemini-2.5-flashVertex AI–temperature 1.0
Gemini 2.5 Progemini-2.5-proVertex AI–temperature 1.0
Gemini 3.8 Flashgemini-3.8-flashVertex AI–temperature 1.0
Qwen2.5-Omni 7Bggml-org/Qwen2.5-Omni-7B-GGUFllama.cppQ8_0temperature 1.0
Voxtral Mini 3Bmistralai/Voxtral-Mini-3B-2507Transformersbf16temperature 1.0

Table 5. How each model is run.

Audio prompt

Listen to this short clip of a patient speaking on a phone check-in call. Classify the patient’s emotional state in THIS clip as exactly one of: frustrated, not_frustrated. Then explain your reasoning based on what you hear in the audio (tone, pacing, pauses, pitch, etc.) -- not the words alone.

Transcript-only prompt

Below is the transcript of one turn a patient spoke during a phone check-in call. You do not have access to audio -- classify the patient’s emotional state in THIS turn based on the WORDS ALONE, as exactly one of: frustrated, not_frustrated. Then explain your reasoning based on the wording, phrasing, and content of what was said.

Transcript for this turn: "{text}"

The control of § 5.5 gives both inputs one prompt, identical word for word, followed by the audio clip or by the line Transcript: "{text}", with the turn’s text in place of {text}.

Identical prompt, both inputs

Below is one turn a patient spoke during a phone check-in call. Classify the patient’s emotional state in this turn as exactly one of: frustrated, not_frustrated. Then explain your reasoning.

The agent-turn runs of Appendix § 12.§ 2 extend the main prompts with what the agent said immediately before the turn, given in the same words to both inputs, with the agent’s words in place of {agent turn}.

Audio prompt with the agent’s preceding turn

Listen to this short clip of a patient speaking on a phone check-in call. Immediately before this clip, the automated assistant said: "{agent turn}"

The patient’s clip is their reply to that. Use it only to understand what they are responding to -- for instance whether they are being asked something for a second time, or being asked something they have already answered. Judge the emotional state from how the PATIENT sounds in their own clip (tone, pacing, pauses, pitch), not from what the assistant said and not from the words alone. Classify the patient’s emotional state in THIS clip as exactly one of: frustrated, not_frustrated. Then explain your reasoning.

Transcript-only prompt with the agent’s preceding turn

Below is the transcript of one turn a patient spoke during a phone check-in call. Immediately before this turn, the automated assistant said: "{agent turn}"

The patient’s turn is their reply to that. Use it only to understand what they are responding to -- for instance whether they are being asked something for a second time, or being asked something they have already answered. You do not have access to audio -- judge the emotional state from the WORDING of the patient’s own turn, not from what the assistant said. Classify the patient’s emotional state in THIS turn as exactly one of: frustrated, not_frustrated. Then explain your reasoning.

Transcript for this turn: "{text}"

Annotators

All 224 patient turns in the six calls were labelled by each of the three annotators, and 58 of the 672 judgements (8.6 %) were inconclusive. Table 6 gives how often a turn was marked inconclusive and how many turns this removed before the balanced evaluation set was drawn from the 75 frustrated and 130 not-frustrated turns that remained.

Turns
Inconclusive judgements per turn
None183
One28
Two9
Three4
Final label
Frustrated75
Not frustrated130
Inconclusive (majority), excluded13
No majority, excluded6
All turns224

Table 6. Inconclusive judgements on the six clinical calls, each patient turn labelled by all three annotators.

Supporting the corpus comparison

All three models score lower on the clinical calls than on the acted corpus in every condition.

Benchmark comparison and the length-matching requirement

A natural comparison is against an established acted emotion corpus. We found that a naive class-balanced sample from such a corpus can introduce a length–label confound: in one such sample, frustrated utterances were markedly longer than neutral ones, to the point that a classifier using *only word count—no audio, no lexical content, no model—scored well above chance and above some evaluated models. On our clinical corpus the same trick reaches 58.0 % (§ 5.2). The effect is far weaker in the source corpus as a whole than in that sample, so it is an artefact of sampling rather than a property of the benchmark.

Clinical, 150 turnsIEMOCAP, 9980 turns
Median turn length4 words9 words
Turns of 3 words or fewer47.3 %18.7 %
Single-word turns31.3 %6.5 %
A bare yes / no / okay20.0 %1.4 %
Frustrated turns of 3 words or fewer38.7 %15.0 %

Table 7. The 150 clinical evaluation turns against the whole of IEMOCAP (all 10 039 utterances except the 59 with no words, which are breathing or laughter only: 9980 turns), on the same measures. The two are different regimes, not two samples of one.

We therefore hold that any cross-corpus comparison in this setting must match length distributions explicitly and balance classes within length strata.

Telephony degradation

The drop in performance from acted to clinical data cannot be attributed to the degradation in audio quality. Both arms are scored on 90 of the 150 acted turns, 45 frustrated and 45 not frustrated.

Per-call variance

Three-way annotator unanimity ranges from 60.9 % to 88.1 % across the six calls, a spread wider than the differences between modalities. Accuracy varies more across calls than across conditions; Table 8 gives intervals that resample whole calls. The ranking of calls by annotator agreement does not match the ranking by model accuracy, which argues against the interpretation that model errors simply track intrinsic ambiguity in the turns.

EstimateOver turnsOver calls
Audio − text, Gemini 2.5 Flash+4.7[−2.0, +11.3][−2.6, +10.9]
Audio − text, Gemini 2.5 Pro+3.3[−5.3, +12.0][−6.2, +11.9]
Audio − text, Qwen2.5-Omni 7B−12.7[−22.0, −3.3][−19.5, −7.0]
Audio − text, frontier pair+4.0[−2.3, +10.3][−2.4, +9.3]
Cleared but frustrated, text − audio+20.2—[+6.3, +30.7]
One-word recall, Gemini 2.5 Pro audio61.9—[0.0, 84.6]

Table 8. Intervals that resample turns against intervals that resample whole calls, on the clinical set. Differences are audio minus text in percentage points; the clearance row is the transcript’s share of cleared turns that were frustrated minus audio’s, for the frontier pair. 95 % bootstrap intervals, 10 000 draws each.

Turns from one call are not independent, so intervals that resample turns can be too narrow. Table 8 resamples whole calls instead, drawing six calls with replacement 10 000 times. The intervals widen, most for one-word recall, but no difference changes sign. Holding out each call in turn tells the same story: the frontier audio advantage stays between 2.4 pp and 6.3 pp and the open model’s deficit between −9.9 pp and −14.9 pp. One call holds 12 of the 21 frustrated one-word turns. Without it, every transcript condition flags none of the remaining 9 and audio flags one or two, so the one-word recall contrast rests mostly on that call, while the balanced-accuracy contrasts do not.

Supporting the short-turn results

On one-word turns, audio beats the transcript for every model and corpus except Qwen2.5-Omni 7B on the clinical calls.

Accuracy and intervals by turn length

Accuracy split by turn length for every model and input, with intervals and the class counts in each bucket.

Table 9 gives recall on one-word turns, the bucket the short-turn claims rest on. Recall is the share of frustrated turns a condition actually flagged, so unlike balanced accuracy it says whether a system would have raised anything at all.

Every transcript condition on the clinical calls recovers two of the 21 frustrated one-word turns, and it is the same two in all three cases. Audio on the same turns ranges from 23.8 % to 61.9 %, better by a wide margin and still short of what a monitor needs. The acted corpus behaves the same way, with one exception: the open model’s transcript condition reaches 22.2 % there, roughly double either Gemini model, which is the only one-word transcript cell in the study that rises meaningfully above the floor.

Table 11 gives the quantity § 5.2 quotes: how far each condition’s predicted-frustration rate travels across the six length buckets, against how far the true rate itself travels. Figure 3 plots the same curves. A condition tracking frustration rather than length would move about as much as the truth does. On the clinical calls every transcript condition moves at least 11 pp further than the truth, and the audio conditions for Gemini 2.5 Pro and the open model move less than it.

Direction of text error by utterance length

Splitting the transcript condition’s errors by direction separates two failures that a single accuracy figure combines. A system that judged a short turn and got it wrong would produce misses and false alarms in some proportion. This one produces almost only misses: on one-word turns neither Gemini model raises a false alarm, and the open model raises 4, which is the signature of a condition that is declining to commit rather than committing incorrectly.

Direction of error on short turns

The same asymmetry holds across most models and inputs. On the 55 short turns, the three transcript conditions and Gemini 2.5 Flash on audio err almost entirely by missing frustration, with 18 to 23 misses against 0 to 5 false alarms. Gemini 2.5 Pro and the open model on audio err in both directions about equally: 11 misses and 11 false alarms, and 17 and 15. A monitor built on the first behaviour clears turns it should escalate; one built on the second escalates almost as often as it misses.

Clearing behaviour by model, threshold and call

A turn’s frustration score, for a given input, is the share of votes calling it frustrated, pooled over the two frontier models and their three repeats each: six votes per turn per input. The open model takes no part in it. A turn is cleared at threshold τ\tau when that share is at most τ\tau, and the confidently cleared counts in § 5.2 are the case τ=0\tau= 0, where not one of the six votes called the turn frustrated.

Two objections to the clearing result are that it depends on where that threshold is drawn, and that it is carried by one unusual call. Neither survives. We sweep τ\tau over {0,0.1,0.2,0.25,0.34,0.5}\{0, 0.1, 0.2, 0.25, 0.34, 0.5\}, which with six votes resolves to four distinct cut points, and hold out each of the six calls in turn. Loosening the threshold makes the transcript condition clear more turns without making the turns it clears any safer: it goes from clearing 87 turns at τ=0\tau= 0 to 104 at τ=0.5\tau= 0.5 while the share of those

Figure 16. Text errors are almost all misses on short turns, split into missed frustration and false alarms. On one-word clinical turns the three transcript conditions produce 57 misses (19 each) and 4 false alarms, all from the open model: it is not judging those turns and getting them wrong, it is declining to judge them, and in a binary decision declining reads as calm.

Figure 17. Direction of error on the 55 short turns. Four of six model–modality conditions err almost exclusively by missing real frustration. Gemini 2.5 Pro and the open model on audio err in both directions about equally. Neither failure mode is usable for monitoring.

that were in fact frustrated stays between 37.5 % and 40.0 % throughout. Audio clears fewer turns at every threshold and is wrong about fewer of them, rising from 17.8 % to 31.7 % as the criterion weakens. Holding out each call in turn leaves both conditions close to their full-sample values.

Supporting the input comparison

The pooled difference between the two inputs is a summary of the turns on which they disagree, and both the size and the direction of that set vary by model. On the clinical calls audio answers 17 turns correctly that the transcript misses for Gemini 2.5 Flash and 25 for Gemini 2.5 Pro, against 10 and 20 in the other direction; for the open model the counts reverse, 19 against 38.

Paired bar charts comparing turns answered correctly by only one condition for real clinical calls and IEMOCAP (acted)

Figure 20. The paired evidence: turns exactly one input answered correctly, with the count both answered correctly beneath. The two inputs score identical turns, so this is the comparison the pooled difference summarises.

Complementarity and fusion

Because the two inputs fail on different turns rather than on the same ones, it is worth asking what combining them could achieve. A simple vote across the models and inputs reaches 66.0 %, below either frontier model’s audio alone, while an oracle that picked the better input for each turn individually bounds what any combination rule could reach (Figure 21).

Figure 21. Left: of the 19 turns only audio gets right, 11 are frustrated; of the 15 turns only text gets right, 9 are calm. Right: a simple vote across the models and inputs reaches 66.0 %, against an oracle ceiling of 78.0 % from picking the better input for each turn.

The agent's preceding turn as context

We reran all three models with the agent’s preceding turn, what the agent said immediately before the patient spoke, added to the prompt in the same words for both inputs; the prompt states that it is there only to show what the patient is answering (Appendix § 9.§ 3). The agent’s turn is the agent audio between the end of the previous patient turn and the start of this one, capped at the last 30 s. Where that gap is silent, because the patient paused and went on, the agent’s most recent voiced turn is used instead. One turn has no agent speech before it, and its prompt carries the line [no agent speech captured] in its place. The agent’s turns were transcribed with Whisper large-v3 and every one was checked against the audio by ear. Each model ran three repeats, scored by majority as in the main runs. No transcript condition changes by an interval that excludes zero, and one-word transcript recall stays between 3 and 5 of 21 (Table 12). Outside the call that holds most frustrated one-word turns the agent’s turn helps Gemini 2.5 Pro’s transcript more, from 0 to 4 of 9, on too few turns to estimate. With the agent’s turn, the frontier pair’s transcript clears 83 turns of which 32.5 % were frustrated, against 52 and 19.2 % for audio. Within the run, audio minus transcript is 0.0 pp for Gemini 2.5 Flash, 6.7 pp for Gemini 2.5 Pro and −6.0 pp for Qwen2.5-Omni, against 4.7 pp, 3.3 pp and −12.7 pp on the patient turn only; only the last of these six has an interval excluding zero (Table 13).

Context depth pooled across utterance lengths

Swept over the whole evaluation set, context depth has no visible effect. Pooled across lengths the curve is flat and no depth is distinguishable from another: the variation across repeats at a fixed depth is comparable to the variation across depths. Stratified by length (§ 5.2, Appendix § 12.§ 4), the two length strata move in opposite directions and cancel when pooled. We report it because it illustrates a general hazard: an aggregate context-depth ablation can conceal a large effect of opposite sign in two subpopulations.

Context depth by utterance length

Model capability and the input gap

The input comparison as differences with intervals, so the paired gap and its precision are visible together.

Figure 23. Does hearing the voice help? The modality comparisons that matter, as differences with 95 % intervals. Solid bars exclude zero.

Speech-encoder baselines

A trained acoustic baseline

This appendix gives the full results for the probes trained on acted speech, which § 5.6 summarises, and the training details for the probes trained on the 30 calls. It first asks what the corpus gap looks like for a classifier fitted on speech.

Frozen WavLM Base+ representations are taken from a single intermediate layer (one of layers 6–9 of 12) and pooled over the turn by mean, maximum, or mean and standard deviation, with either a logistic regression or an RBF support vector machine fitted on top. Layer, pooling and hyperparameters are chosen by cross-validation inside the training data alone, and nothing in WavLM is fine-tuned. Each configuration is fitted once on 1810 IEMOCAP turns, split from its test set by dialogue so that no dialogue contributes to both, and is then scored twice without refitting: on 150 held-out acted turns and on the 150 real clinical turns, which take no part in training or in selection. Two bandwidth arms are reported, since the corpora differ in channel as well as in content. The native arm trains on the acted audio as recorded at 16 kHz; the band-matched arm trains on the same turns after an 8 kHz telephone round trip, so that training and test share an effective bandwidth.

Table 14 reports the result. A probe given enough acted speech learns something the transcript does not: the strongest configuration, WavLM with an SVM on native audio, reaches 70.0 % on the clinical turns against 58.0 % for word count and 52.7 % for clip duration. What does not go away is the corpus gap. Every configuration scores lower on the clinical turns than on held-out acted speech, by between 6.1 pp and 11.3 pp, and the band-matched arms drop as far as the native ones, so the loss is not carried by bandwidth. Word count transfers more evenly, falling only 4.3 pp, which is what a predictor with no access to the speech should do; clip duration falls 8.2 pp.

Table 14: Generalisation from acted IEMOCAP speech to real clinical speech. All values are balanced accuracy (%), so chance is 50%. Models were trained on the larger IEMOCAP pool. Dialogue-disjoint evaluation uses a single held-out acted set and clinical set (n=150n = 150 each; ntrain=1,810n_{\mathrm{train}} = 1{,}810). Session-rotation evaluation pools predictions from five held-out-session folds for the acted set and reports the mean and standard deviation across the five models for the same clinical set.

FeaturesClassifierDialogue-disjointSession rotation
ActedClinicalActed / Clinical
WavLM, band-matchedSVM77.468.078.0 / 74.0 ± 0.9
WavLM, nativeSVM76.170.073.4 / 71.3 ± 2.7
WavLM, nativeLogistic regression71.360.066.6 / 64.8 ± 4.0
WavLM, band-matchedLogistic regression66.660.064.0 / 62.7 ± 4.0
Word countLogistic regression62.358.062.3 / 57.9 ± 0.3
DurationLogistic regression60.952.760.2 / 52.3 ± 0.4

Table 14.

Separating training from test

Splitting training from test by dialogue is the arrangement reported throughout. Splitting by speaker instead trains on more turns but leaves only 42 acted turns to test on, so its acted column is measured far less precisely.

Figure 24. The probes and the audio-free baselines on both test sets, with 95 % intervals and the number of turns on each bar. Left: training and test split by dialogue, the arrangement Table 14 reports. Right: split by speaker, which trains on more turns but tests the acted side on only 42.

Table 15: Effect of acted training-set size on balanced accuracy for the same held-out clinical test set (n=150n = 150). The smaller-data condition uses 150 acted training turns; the pooled condition uses 1,810 dialogue-disjoint acted turns. Δ\Delta denotes the change in percentage points.

FeaturesClassifiern=150n = 150n=1,810n = 1{,}810Δ\Delta
WavLM, band-matchedSVM56.768.0+11.3
WavLM, nativeSVM66.770.0+3.3
WavLM, nativeLogistic regression56.760.0+3.3
WavLM, band-matchedLogistic regression61.360.0-1.3
Word countLogistic regression60.758.0-2.7

Table 15.

Training-set size

The probes’ margin over counting words depends on how much labelled acted speech they are given.

Probe advantage over counting words on the real 150 turns

Figure 25. More training data turns the probes from no better than word count into clearly better. Probe advantage over counting words, trained on the labelled 150 against the full acted pool. The two 150-turn rows use a deliberately conservative interval: that run did not save per-item predictions, so its probe and baseline cannot be compared on matched turns. The pool rows are properly paired.

Increasing the acted training set from 150 to 1,810 turns improved clinical accuracy by up to 11.3 percentage points for the WavLM probes, whereas the word-count baseline remained broadly unchanged. This indicates that the relatively small advantage of the probes in the earlier experiment was partly attributable to limited training data. Under five-fold held-out-session rotation, the strongest configuration achieved 74.0% clinical accuracy, compared with 57.9% for the word-count baseline, and exhibited an acted-to-clinical gap of 4.0 percentage points.

One-word turns and the decision threshold

The probes set their decision threshold on acted speech. To check that this cut-off does not flatter them on the clinical calls, we also chose it on clinical speech while leaving one call out: each call is scored with the threshold that maximises balanced accuracy on the other five. For the strongest configuration, band-matched WavLM with an SVM under session rotation, balanced accuracy averages 74.0% and the probe flags 14.0 of the 21 frustrated one-word turns on average under either threshold, though both vary from fold to fold (Table 16). § 5.6 reports this probe on the 24 added calls.

Acted thresholdHeld-out-call threshold
Bal. acc.One-wordBal. acc.One-word
Band-matched, fold 174.011/2172.09/21
Band-matched, fold 272.710/2171.39/21
Band-matched, fold 374.014/2172.015/21
Band-matched, fold 474.016/2176.018/21
Band-matched, fold 575.319/2178.719/21
Band-matched, mean74.014.0/2174.014.0/21
Native band70.04/2166.09/21

Table 16. WavLM SVM probes on the 150 clinical turns with the threshold set on acted speech or on the other clinical calls (leave one call out). Balanced accuracy and the frustrated one-word turns flagged, of 21. Band-matched features use the five session-rotation folds; the native-band row is a single model.

Probes trained on the 30 calls

The probes of § 5.6 that are trained on clinical speech use the same recipe as those trained on acted speech: frozen WavLM layers pooled over the turn by mean, maximum, or mean and standard deviation, with a logistic regression or an RBF support vector machine. No call outside the 30 is used for training or for choosing any setting. The 30 calls are split into five folds by patient, and each fold’s turns are scored by a probe whose layer, pooling and weights come from the other four folds only. Layer and pooling are chosen by cross-validation grouped by patient inside those four folds. A turn is flagged at the probe’s own decision boundary. Three split seeds give three predictions per turn, and the turn’s prediction is their majority, as for the language models. Selection settles on WavLM layers 6 to 9 for both probes, mostly with mean pooling for the SVM and with all three poolings for the logistic regression. The three seeds disagree on 29.1% of the 714 turns for the logistic regression and 13.3% for the SVM.

Can these parameters be measured on a one-word turn?

Every functional in eGeMAPS summarises a window of speech, so its value carries sampling noise that grows as the window shortens. We estimate that noise directly and without labels. For each turn of at least six seconds we compute a parameter over the whole turn, then recompute it on fifteen randomly placed windows drawn from inside the same turn. The turn is unchanged, so variation across its own windows is measurement error; comparing it against variation between turns gives a one-way random-effects intraclass correlation.

Figure 26 reports this across window lengths. Level parameters are reliable from half a second: loudness mean reaches 0.76 there. Variability parameters are not, and F2 frequency variability reaches conventional reliability only near six seconds. On the clinical calls at 0.5 s, F2 and F3 variability reach 0.17, and none of the 88 parameters reaches 0.70.

Figure 26. Reliability of eGeMAPS parameters against the amount of speech the estimate is computed from. Lines are the IEMOCAP window sweep; squares mark the clinical calls at the median one-word duration, where the two corpora agree. Level parameters are reliable from half a second, variability parameters are not. Shaded band marks the one-word range, 0.32 s to 1.18 s. One-way random-effects ICC(1); 45 source utterances, 15 windows per utterance at each length.

Two alternative explanations were tested and do not hold. Bandwidth is not the cause: IEMOCAP at its native 16 kHz gives the same poor reliability at 0.5 s, 0.19 and 0.17 for F2 and F3, and band-limiting it to a 300 Hz to 3400 Hz telephone channel moves reliability slightly upward rather than down. Nor is it an artefact of acted speech being acoustically uniform: the between-utterance spread is comparable across the two corpora.

We also attempted the direct labelled comparison, contrasting frustrated against calm one-word turns within call, and separately across turns of the identical word. Neither is informative: at 42 and 16 turns respectively they detect only effects above 0.97 and 1.57 standard deviations.

Onset of frustration

As well as the detection of frustration from real clinical calls, the prediction of the onset of frustration was also investigated.

The same word, escalating mid-call

The same difficulty appears mid-call, and in two different shapes. Figure 27 shows both, from real calls.

Two ways frustration escapes the transcript in Calls C and D

Figure 27. Two ways frustration escapes the transcript, both mid-call and both from real calls. Call C: the patient answers “No.” to a question about pain and “No.” again eighteen turns later, and the two carry opposite labels, so the same single word is both. Call D: a longer answer whose wording reads as a mild dismissal is labelled frustrated, and both frontier models get it right from the audio while both call it not frustrated from the transcript. Agent prompts are elided where long; patient turns are complete and verbatim.

Call C is the shortest possible answer. The patient says “No.” to a question about pain, which annotators label not frustrated, and “No.” again eighteen turns later as the agent begins reciting a summary, which they label frustrated. The two turns are the same single word, so nothing in the transcript separates them.

Call D is a longer answer whose wording carries no sign of frustration. Asked whether they would like a clinician to call about a six-to-eight week wait, the patient replies “Eight weeks doesn’t feel it is a long time.” Read as text that is a mild dismissal of the concern; annotators hearing it label it frustrated. Both frontier models label it correctly from the audio and both label it not frustrated from the transcript, so the frustration is present in the delivery and absent from the words.

Both examples sit well before the sign-off, where an escalation could still change the call.

Detecting the onset

The most consequential turn in a call is the one where frustration first becomes audible, since that is when intervention is still cheap. We isolated the onset of each call—the first turn annotators labelled frustrated—and asked whether each modality cleared it. Text detected only two of the six onsets and confidently cleared three (Gemini 2.5 Flash) or four (Gemini 2.5 Pro); audio detected four (Flash) or three (Pro) and confidently cleared two. The two the text detected are the two that state frustration in words, “You misunderstand.” and “are you going to repeat this all the time?”; the two audio cleared are “No, it was on the left eye.” and “no”. Six calls is an illustration rather than an estimate, and we report it as one, but the direction is unambiguous and it is the operational shape of the result: a transcript-based monitor discards the first sign of frustration and reports the call as calm.

The six onsets are worth reading individually rather than as a rate, both because six is too few to estimate one and because the turns differ in the way that matters here.

Detection can also be read along the call rather than at a single turn, which shows how far past the onset a condition first responds.

The 30 calls, measure by measure

Table 17 gives each measure of § 5.1–§ 5.3 on the six calls and on the 30 calls, for the three original models and for the two added ones. The corpus comparison, telephony, context, the agent’s preceding turn and onset are not repeated on the 30 calls, because they need the acted set or experiments run only on the six. Calls vary more than conditions do: within a condition, balanced accuracy has a standard deviation across calls of 7 pp to 19 pp, against 5.1 pp across the ten model-and-input conditions. Figure 8 shows predicted-frustration rate by length on the added calls, whose turns keep their natural prevalence. Figure 30 gives the frustrated one-word turns each model and input flags, and Figure 31 the audio-minus-transcript difference for each model under each prompt (§ 5.5).

MeasureSix calls, 3 models30 calls, same 3 models3.8 Flash · Voxtral, 30 calls
Share of turns of 1–2 words36.7 % of the balanced 15053.4 %; 57.8 % on the 24—
One-word transcript balanced accuracy47.1–54.849.9–53.8; lower bounds 44.6–50.053.8 · 53.7
Frustrated one-word turns flagged by the transcript2 of 21 for each model, the same 23, 3 and 4 of 393 of 39 each; all five share the same 3
One-word audio balanced accuracyFlash 60.0, Pro 61.7, Qwen 42.1Flash 59.9 [53.3, 63.9], Pro 65.6 [51.9, 74.7], Qwen 49.854.6 [49.7, 59.8] · 50.0
Word count only58.0 [50.5, 67.5]57.3 [52.8, 62.0]; Qwen audio 55.1Voxtral audio 54.4
Spread of the transcript’s predicted rate across lengths50.9–88.1 pp against true 39.4On the 24: 26.3–30.9 pp against true 21.217.3 · 40.4
Frustrated share of confidently cleared turns, text against audioFrontier pair: 37.9 % of 87 against 17.8 % of 45; +20.2 [6.3, 30.7]13.0 % of 578 against 6.9 % of 333; +6.1 [2.2, 10.4]; 24 alone +3.3 [0.3, 6.5]13.6 % against 14.5 % · 13.6 % against 17.7 %
Wrongly cleared turns that are one word18 of 33; 28 % of frustrated turns are one word35 of 75; 28 %—
Transcript misses / false alarms on short turnsGemini 23 / 0–1; Qwen 21 / 5Gemini 47 / 2; Qwen 44 / 3847 / 0 · 48 / 1
Audio minus textFrontier pair +4.0 [−2.4, 9.3]; Qwen −12.7 [−19.5, −7.0]Frontier pair +3.0 [−1.0, 7.5] (Flash +3.6, Pro +2.3); Qwen −8.6 [−14.1, −1.6]−3.5 [−6.0, 0.0] · −10.6 [−16.5, −3.6]
Oracle / vote across models and inputs78.0 / 66.089.8 / 67.1; Flash audio 70.2—

Table 17. Measures of Sections 5.1–5.3 on the six calls and the 30 calls. Balanced accuracy in percent unless stated; intervals are 95 % bootstrap intervals over calls, so a few six-call values differ slightly from the main text. Clearing by length counts one-word turns, where Section 5.2 counts turns of one or two words. The prediction spread is shown on the 24 added calls alone.

Figure 28. The first turn in each call that annotators labelled frustrated, and whether each condition detected it. Four of the six onsets are five words or fewer.

Onset utterancewordsAudio FlashAudio ProText FlashText Pro
“Yes.” Call 113/33/30/30/3
“No, it was on the left eye.” Call 470/30/30/30/3
“no” Call 510/30/30/30/3
“You misunderstand.” Call 323/32/33/33/3
“are you going to repeat this all t…” Call 293/33/33/33/3
“No. No.” Call 622/31/31/30/3
Onsets detected4/63/62/62/6

Table 28.

Figure 29. Detection timeline. Each row is one call, each block a patient turn coloured by its human label; the rule marks the first frustrated turn and the markers show where each condition first returns frustrated—audio above the row, text below. In the top call both audio conditions fire on the onset turn while both text conditions fire seven turns later. Gaps between blocks are turns outside the balanced sample, so a first-flag position is first among sampled turns rather than first in the call.

Qualitative examples

Every rationale below is quoted verbatim from the run that produced the verdict printed beside it. Turn length is given in words; the ground truth is the majority of three annotators.

Silent clearing

Both models, reading the transcript, clear a frustrated turn without hesitation. Nothing downstream would flag these. 33 such turns occur in the evaluation set.

“No.” FRUSTRATED · 1 w

Flash / voice× calmThe speaker’s single word ’Yeah’ is delivered with a neutral to slightly agreeable tone. There is no discernible tension, abruptness, or downward inflection in pitch that would suggest frustration. The pacing is quick, but without other vocal cues, it doesn’t indicate an emotional state of frustration.
Flash / words× calmThe patient’s response ’No’ is a single, short, and direct negation. It does not contain any emotionally charged language, exclamations, or elaborations that would suggest frustration. While it could be delivered with a frustrated tone, based on the words alone, it is a neutral answer.
Pro / voice× calmThe patient speaks a single word in a neutral, calm tone. The pitch is steady and in a normal range. There are no vocal cues indicating frustration, such as a sigh, a sharp tone, or vocal strain.
Pro / words× calmThe user’s response is a single word, ’No.’ Based on the text alone, this is a simple, direct answer to a question. There are no additional words, phrases, or exclamations to indicate frustration or any other emotional state.
“no”FRUSTRATED · 1 w
Flash / voice× calm
The patient’s tone is relatively neutral and calm when saying ’hello’. There is a slight upward inflection at the end, which is common in a polite greeting, but no discernible harshness, abruptness, or vocal tension that would suggest frustration. The pacing is normal for a single word utterance.
Flash / words× calm
The word "no" is a neutral, monosyllabic response that does not inherently convey frustration. There are no additional words, phrasing, or content in this turn to suggest anger, annoyance, or other markers of frustration.
Pro / voice× calm
The patient’s tone is neutral and even-keeled. The word ’hello’ is delivered with a slight upward inflection, which is standard for answering a phone and does not indicate any stress, agitation, or impatience. The pacing is normal and the pitch is within a calm range.
Pro / words× calm
The single word ’no’ is a direct answer and does not, by itself, contain any linguistic markers of frustration. It is a neutral response.

Table 18.

Delivery recovers the turn

The words are identical to turns that are genuinely calm; the delivery is not, and both models act on it. 6 such turns occur in the evaluation set.

“No.”FRUSTRATED · 1 w
Flash / voice✓ frustrated
The patient delivers the single word ’No’ with an abrupt, firm, and somewhat sharp tone, without any hesitation. This vocal quality suggests a lack of patience or irritation rather than a neutral or calm response.
Flash / words× calm
The word ’No’ is a simple, direct negation. There is no accompanying phrasing, exclamation, or strong language that would indicate frustration. By itself, it is emotionally neutral and simply answers a perceived question or statement.
Pro / voice✓ frustrated
The single word ’No’ is delivered with a sharp, clipped, and abrupt tone. The pacing is very fast for a single syllable, and the pitch is low and forceful, indicating impatience and annoyance rather than a neutral or thoughtful response.
Pro / words× calm
The single word ’No’ is a direct, neutral, and unembellished response. Standing alone without any further context or emotional language, it does not provide any evidence of frustration. It is simply a negative answer.

Table 19.

“No, no.”FRUSTRATED · 2 w
Flash / voice✓ frustrated
The patient’s tone is somewhat sharp and abrupt, delivering the two ’no’s in quick succession. This quick, emphatic delivery, combined with a slightly dismissive quality in the voice, suggests a feeling of impatience or annoyance, indicating frustration.
Flash / words× calm
The phrase "No, no" is a simple and emphatic denial or negation. It does not contain any exclamatory language, strong negative adjectives, or explicit expressions of annoyance, exasperation, or distress that would indicate frustration. The repetition of "no" serves to add emphasis to the denial, which is a neutral linguistic function.
Pro / voice✓ frustrated
The patient’s tone is sharp and clipped. The two words are spoken quickly and abruptly, which conveys a sense of impatience or annoyance. The delivery is forceful, not calm or neutral, which points toward frustration.
Pro / words× calm
The user is simply saying "No, no." which is a direct negation. The words themselves do not contain any language that indicates frustration, impatience, or annoyance. Without any other context or explicit negative wording, the statement is neutral.

Table 20.

Delivery misleads

The reverse case. Hearing the turn moves both models to the wrong answer. 3 such turns occur in the evaluation set.

“No!”FRUSTRATED · 1 w
Flash / voice× calmThe patient says ’No’ with a firm, clear, and steady tone. The pitch is relatively neutral, and there is no discernible strain, raised volume, or erratic pacing that would suggest frustration. It sounds like a definitive, straightforward negative response rather than an expression of annoyance or anger.
Flash / words✓ frustratedThe patient uses a single, absolute negation ’No!’ followed by an exclamation mark. This combination of a direct refusal and an emphatic punctuation suggests a strong emotional response, likely indicating frustration, exasperation, or strong disagreement rather than a neutral denial.
Pro / voice× calmThe patient says a single word, ’No,’ in a low-pitched, even-toned, and calm manner. The delivery is not sharp, clipped, or high-pitched, which would typically be associated with frustration. The tone is neutral and declarative.
Pro / words✓ frustratedThe response is a single, abrupt, and emphatic word, "No!". The use of an exclamation mark indicates a strong emotional charge rather than a simple, neutral answer. This type of forceful, monosyllabic negation is a common way to express frustration or exasperation.

Neither recovers the turn

The common one-word case: every condition clears a frustrated turn. 7 such turns occur in the evaluation set.

“No.”FRUSTRATED · 1 w
Flash / voice× calmThe speaker’s single word ’Yeah’ is delivered with a neutral to slightly agreeable tone. There is no discernible tension, abruptness, or downward inflection in pitch that would suggest frustration. The pacing is quick, but without other vocal cues, it doesn’t indicate an emotional state of frustration.
Flash / words× calmThe patient’s response ’No’ is a single, short, and direct negation. It does not contain any emotionally charged language, exclamations, or elaborations that would suggest frustration. While it could be delivered with a frustrated tone, based on the words alone, it is a neutral answer.
Pro / voice× calmThe patient speaks a single word in a neutral, calm tone. The pitch is steady and in a normal range. There are no vocal cues indicating frustration, such as a sigh, a sharp tone, or vocal strain.
Pro / words× calmThe user’s response is a single word, ’No.’ Based on the text alone, this is a simple, direct answer to a question. There are no additional words, phrases, or exclamations to indicate frustration or any other emotional state.

The word itself is not recovered

On a handful of one-word turns the model does not establish which word was spoken, and reasons confidently about a word that was never said. The transcript of the clip, produced independently, is given for comparison.

“No.” FRUSTRATED · 1 W

Independent transcription of this clip: “No.”.

Flash / voice × calm The speaker’s single word ’Yeah’ is delivered with a neutral to slightly agreeable tone. There is no discernible tension, abruptness, or downward inflection in pitch that would suggest frustration. The pacing is quick, but without other vocal cues, it doesn’t indicate an emotional state of frustration.

Flash / words × calm The patient’s response ’No’ is a single, short, and direct negation. It does not contain any emotionally charged language, exclamations, or elaborations that would suggest frustration. While it could be delivered with a frustrated tone, based on the words alone, it is a neutral answer.

Pro / voice × calm The patient speaks a single word in a neutral, calm tone. The pitch is steady and in a normal range. There are no vocal cues indicating frustration, such as a sigh, a sharp tone, or vocal strain.

Pro / words × calm The user’s response is a single word, ’No.’ Based on the text alone, this is a simple, direct answer to a question. There are no additional words, phrases, or exclamations to indicate frustration or any other emotional state.

“No.” FRUSTRATED · 1 W

Independent transcription of this clip: “No.”.

Flash / voice × calm The patient’s single word ’Hello’ is delivered with a neutral and even tone, without any raised pitch, abruptness, or vocal tension that would indicate frustration.

Flash / words × calm The patient’s response ’No.’ is a simple, direct negation. Without any additional context, inflection (which we cannot infer from text alone), or other words, there is nothing in this single word to indicate frustration. It serves as a neutral answer.

Pro / voice ✓ frustrated The patient says a single word, "No," with a sharp, clipped, and slightly forceful tone. The abruptness of the utterance and the slightly downward, declarative pitch suggest impatience and annoyance rather than a calm or neutral state.

Pro / words × calm The single word "No" is a direct and simple answer. Without any additional context or emotional language, it does not inherently convey frustration. It is a neutral response to a presumed yes/no question.

References

  1. [1]Scott J. Adams, Julián N. Acosta, and Pranav Rajpurkar. How generative AI voice agents will transform medicine. npj Digital Medicine, 8(1):353, 2025. doi: 10.1038/s41746-025-01776-y.DOI
  2. [2]Martijn Bartelds, Federico Bianchi, and James Zou. Real-time voice AI hears but does not listen. arXiv preprint arXiv:2606.26083, 2026. URL https://arxiv.org/abs/2606.26083.
  3. [3]Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sunghok Lee, and Shriikanth S. Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4):335–359, 2008. doi: 10.1007/s10579-008-9076-6.DOI
  4. [4]Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost. MSP-IMPROV: An acted corpus of dyadic interactions to study emotion perception. IEEE Transactions on Affective Computing, 8(1):67–80, 2017. doi: 10.1109/TAFFC.2016.2515617.DOI
  5. [5]Houwei Cao, David G. Cooper, Michael K. Keutmann, Ruben C. Gur, Ani Nenkova, and Ragini Verma. CREMA-D: Crowd-sourced emotional multimodal actors dataset. IEEE Transactions on Affective Computing, 5(4):377–390, 2014. doi: 10.1109/TAFFC.2014.2336244.DOI
  6. [6]Jingyi Chen, Zhimeng Guo, Jiyun Chun, Pichao Wang, Andrew Perrault, and Micha Elsner. Do audio LLMs really LISTEN, or just transcribe? Measuring lexical vs. acoustic emotion cues reliance. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5848–5877, 2026. doi: 10.18653/v1/2026.eacl-long.274.DOI
  7. [7]Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Xu, and Furu Wei. WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022. doi: 10.1109/JSTSP.2022.3188113.DOI
  8. [8]Zachary Ellis, Jared Joselowitz, Yash Deo, Yajie He, Anna Kalynina, Aisling Higham, Mana Rahimzadeh, Yan Jia, Ibrahim Habli, and Ernest Lim. WER is unaware: Assessing how ASR errors distort clinical understanding in patient facing dialogue. In Proceedings of the 16th International Workshop on Spoken Dialogue System Technology, pages 391–417, 2026. URL https://aclanthology.org/2026.iwsds-1.39/.arxiv.org/abs/2511.16544
  9. [9]Florian Eyben, Klaus R. Scherer, Björn W. Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y. Devillers, Julien Epps, Petri Laukka, Shrikant S. Narayanan, and Khiet P. Truong. The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing, 7(2):190–202, 2016. doi: 10.1109/TAFFC.2015.2457417.DOI
  10. [10]Suchin Gururangan, Swabha Swायamdिपta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, 2018. doi: 10.18653/v1/N18-2017.DOI
  11. [11]Mireia Hernandez Caralt, Ivan Sekulic, Filip Carevic, Nghia Khau, Diana Nicoleta Popa, Bruna Guedes, Victor Guimaraes, Zeyu Yang, Andre Manso, Meghana Reddy, Paolo Rosso, and Roland Mathis. “Stupid robot, I want to speak to a human!” User frustration detection in task-oriented dialog systems. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 276–285, 2025. URL https://aclanthology.org/2025.coling-industry.23/.
  12. [12]Qixuan Hu, Shuchang Ye, Xumou Zhang, Anastasia Serafimovska, Anastasia Suraev, Amit Saha, Ping-hsiu Lin, Sydney Su, Usman Naseem, Adam G. Dunn, and Jinman Kim. CBT-Audio: Evaluating audio language models for patient-side distress intensity estimation in CBT session recordings. arXiv preprint arXiv:2605.17370, 2026. URL https://arxiv.org/abs/2605.17370.
  13. [13]Amir Ivry and Shinji Watanabe. LALM-as-a-judge: Benchmarking large audio-language models for safety evaluation in multi-turn spoken dialogues. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 50124–50147, 2026. URL https://proceedings.mlr.press/v306/ivry26a.html.arxiv.org/abs/2602.04796
  14. [14]Minseok Kim, Jingxiang Chen, Seong-Gyun Leem, Yin Huang, Rashi Rungta, Zhicheng Ouyang, Haibin Wu, Surya Teja Appini, Ankur Bansal, Yang Bai, Yue Liu, Florian Metze, Ahmed A. Aly, Anuj Kumar, Ariya Rastrow, and Zhaojiang Lin. Aligning paralinguistic understanding and generation in speech LLMs via multi-task reinforcement learning. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track), pages 636–648, 2026. doi: 10.18653/v1/2026.eacl-industry.49.DOI
  15. [15]Jonathan Klein, Youngme Moon, and Rosalind W. Picard. This computer responds to user frustration: Theory, design, and results. Interacting with Computers, 14(2):119–140, 2002. doi: 10.1016/S0953-5438(01)00053-4.DOI
  16. [16]Alexander H. Liu, Andy Ehrenberg, Andy Lo, Clément Denoix, Corentin Barreau, Guillaume Lample, Jean-Malo Delignon, Khyathi Raghavi Chandu, Patrick von Platen, Pavankumar Reddy Muddireddy, Sanchit Gandhi, Soham Ghosh, Srijan Mishra, Thomas Foubert, et al. Voxtral. arXiv preprint arXiv:2507.13264, 2025. URL https://arxiv.org/abs/2507.13264.
  17. [17]Xinyi Liu, Hooshang Nayyeri, Dilek Hakkani-Tur, Emine Yilmaz, Joo-Kyung Kim, Yifei Zhang, Charith Peris, and Hari Thadakamalla. Hear2Act: Benchmarking when prosody should change what an assistant does. arXiv preprint arXiv:2608.19515, 2026. URL https://arxiv.org/abs/2608.19515.
  18. [18]Steven R. Livingstone and Frank A. Russo. The Ryerson audio-visual database of emotional speech and song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLOS ONE, 13(5):e0196391, 2018. doi: 10.1371/journal.pone.0196391.DOI
  19. [19]Potsawee Manakul, Woody Haosheng Gan, Michael J. Ryan, Ali Sartaz Khan, Warit Sirichotedumrong, Kunat Pipatanakul, William Barr Held, and Diyi Yang. AudioJudge: Understanding what works in large audio model based speech evaluation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3644–3663, 2026. doi: 10.18653/v1/2026.eacl-long.168.DOI
  20. [20]Kevin Miller, Arjun Chandra, and Venkatesh Saligrama. Do audio language models use paralinguistic evidence: Counterfactual audits for response evaluation. arXiv preprint arXiv:2608.06718, 2026. URL https://arxiv.org/abs/2608.06718.
  21. [21]Jiacheng Pang, Ashutosh Chaubey, and Mohammad Soleymani. Do audio LLMs listen or read? Analyzing and mitigating paralinguistic failures with VoxParadox. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 96307–96334, 2026. URL https://proceedings.mlr.press/v306/pang26b.html.arxiv.org/abs/2605.27772
  22. [22]Roseline Polle, Owen Parsons, George Fairs, Luis Miguel San Martin Fernandez, Cole Looney, Xiaoliang Wu, Alexandra Livia Georgescu, and Stefano Goria. Synthetic speech, real signal: Paralinguistic preservation and cross-lingual augmentation via voice cloning. In Proceedings of Interspeech 2026, pages 2303–2307, 2026. doi: 10.21437/Interspeech.2026-2993.DOI
  23. [23]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492–28518, 2023. URL https://proceedings.mlr.press/v202/radford23a.html.arxiv.org/abs/2212.04356
  24. [24]Armaan Sayyad, J. Emmons, S. Jones, T. Lin, and H. Krishnan. A reliability assessment of LALM audio judges for full-duplex voice agents. arXiv preprint arXiv:2607.07985, 2026. URL https://arxiv.org/abs/2607.07985.
  25. [25]Alexander Schmitt, Tobias Heinroth, and Jackson Liscombe. On NoMatchs, NoInputs and Bargelns: Do non-acoustic features support anger detection? In Proceedings of the SIGDIAL 2009 Conference, pages 128–131, 2009. URL https://aclanthology.org/W09-3918/.
  26. [26]Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby, Felix Friedrich, Maurice Kraus, Kourosh Nadi, Huu Nguyen, Kristian Kersting, and Sören Auer. EmoNet-Voice: A fine-grained, expert-verified benchmark for speech emotion detection. arXiv preprint arXiv:2506.09827, 2025. URL https://arxiv.org/abs/2506.09827.
  27. [27]Ramaneswaran Selvakumar, Ashish Seth, Nishit Anand, Utkarsh Tyagi, Sonal Kumar, Sreyan Ghosh, and Dinesh Manocha. Multi Vox: A benchmark for evaluating voice assistants for multimodal interactions. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 28481–28493, 2025. doi: 10.18653/v1/2025.emnlp-main.1447.DOI

Paper details

vlnc:W_2fq67qb2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents