WorksEngineering
How Much Prosody Does Your ASR Encoder Already Know
Abstract
End-of-turn (EoT) detection decides in real-time whether a speaker has finished. Traditional detectors rely on hand-built prosodic features (pitch, speaking rate, pauses), while modern detectors run on a neural ASR encoder that has processed the same audio. Because the encoder reads the signal those features are computed from, it may already hold much of the information they were meant to supply. We test this three ways, reading a streaming Zipformer encoder at four depths from raw input to the final layer. First, we measure how much prosody helps over the encoder alone. Second, we test whether prosody can be recovered from the encoder. Third, we ask whether any single feature still helps given the encoder. The three tests agree; prosody adds a large gain on raw audio, but that gain collapses once the signal passes through even the first encoder layer. The same pattern holds for two open-source encoders, wav2vec2 and Whisper, both of which are offline.
Introduction
A spoken dialogue system must work out when the user has finished speaking before it can respond, and it must make this end-of-turn (EoT) decision continuously and at low latency. The timing is demanding because human turn-taking is fast. In conversation the gap between speakers is usually only a few hundred milliseconds [1], [2], so a mistimed response is noticed at once. Responding too early cuts the user off, while responding too late makes the exchange feel sluggish [3]. The simplest approach is to wait for a fixed stretch of silence and treat it as the end of the turn. Silence is easy to measure, but on its own it is unreliable, because speakers also pause in the middle of a turn. A fixed silence threshold then has to choose between two failures, firing on these within-turn pauses or waiting too long after the turn has ended [4], [5].
To do better than silence alone, end-of-turn detection has long relied on prosody, the cues in how something is said rather than what is said, such as pitch, energy, speaking rate, and pause structure [3], [6]. Speakers mark the end of a turn with these cues, for instance by lowering their pitch and slowing down, a regularity documented across decades of work in conversation analysis and phonetics [7]–[9]. Early detectors used the cues directly. Combining pitch, vowel duration, and speaking rate, for example, detects the end of an utterance faster and more accurately than a fixed silence threshold [10], [11], and dialogue systems used similar timing cues to decide when to stop listening and respond [5]. Prosody thus became a turn-end signal in its own right, computed by hand straight from the audio, and it is still studied as a turn-taking cue in recent work [12], [13].
Modern detectors are built differently, yet they share one assumption, that prosody adds something the encoder lacks. They sit on top of a neural encoder, a network trained for automatic speech recognition (ASR) that turns the raw audio into the sequence of feature vectors a recogniser consumes [14]–[16]. Across two decades of such systems, prosody and the encoder’s features have stayed separate inputs, combined by design. Early neural detectors replaced the fixed silence rule with learned acoustic models that pick up cues such as filler sounds and speaking rate [17], and streaming variants made the decision online, updating the turn-end estimate at each new frame [18]. Later systems built endpointing into the recogniser itself, so that one model both transcribes and marks turn ends [19], [20], while others predicted from moment to moment whether the user is about to keep talking or stop [21]–[23]. The practice continues today, as current turn-taking models still pair a dedicated acoustic or prosodic stream with the learned representation, whether they predict turns from voice activity [13] or build on a large language model [24], and a recent survey still singles out prosody as a central turn-taking cue [3]. At the same time, the newest full-duplex speech LLMs fold turn-taking directly into a single end-to-end model with no separate prosodic front-end [25], [26], already hinting that turn dynamics can live inside a learned representation.
The encoder, though, reads the same audio the prosodic features are computed from, so it may already represent much of that prosody on its own [14], [15]. Properties such as pitch and stress can be recovered from a speech encoder’s internal representations by probing, a standard analysis technique with established safeguards against reading in correlations that are not really there [27]–[31]. A self-supervised turn-taking model has also been shown to change its predictions when the prosody of its input audio is altered [12], and a recent study probes how such turn-taking models rely on prosodic or lexical cues, finding either can suffice [32]. None of this answers the practical question. With a strong encoder already in the pipeline, does adding hand-built prosody still improve detection, or has the encoder made it redundant? If it is redundant, the field has been paying for feature extraction, latency, and per-speaker tuning for little benefit.
This paper answers the question directly, reading a streaming Zipformer ASR encoder at four depths, from the raw input features through to the final layer, and running three tests.
Does prosody still help? We train a turn-end detector on the encoder alone and again on the encoder together with prosody, and measure how much prosody adds at each depth (Sec. III-A).
If not, why not? We measure how accurately each prosodic feature can be reconstructed from the encoder’s own representations. A feature that is easy to reconstruct yet adds nothing when supplied separately is one the encoder already carries, which would explain why adding it again does not help (Sec. III-B).
Does any individual cue still help? We test whether any single prosodic feature carries turn-timing information the encoder lacks, including the pitch cues classical detectors relied on most (Sec. III-C).
A small measured gain could mislead us for two reasons, and we control for both. The first is that the detector might be too weak to use the prosody it is given, which would make prosody look useless even when it carries real information. We rule this out by repeating every measurement with a second, stronger detector and checking whether the conclusion holds. The second is that part of a feature’s apparent value can come from who the speaker is rather than from how the turn unfolds. Take a speaker’s mean pitch, which barely changes within a turn, so on its own it says little about when that turn ends, yet it can still help a detector if speakers with higher voices happen to take turns differently from speakers with lower voices. The detector then leans on the speaker’s identity as a shortcut instead of the turn-timing cue we care about [33, 34, 35]. We isolate and remove this speaker effect with a time-shift control that circularly shifts each prosodic feature off the turn boundary while leaving its per-speaker statistics intact (§ 2-D), so that each feature is credited only for genuine turn-timing information. We also repeat the main measurement on two further open-source encoders and on a public corpus, to test whether the pattern is specific to a single model or to our own data (§ 4-D). § 2 sets out each measurement, control, and data split in full.
Dataset and Methodology
Every measurement in this paper rests on one shared setup, namely a single dataset, two fixed detectors, and a time-shift control that separates genuinely turn-aligned timing from speaker-identity effects. We define that setup here and present the experiments and their results in § 4.
Task and labels
We work at the frame level, reading each utterance as a sequence of frames at 25 Hz, one every 40 ms, and at each frame the detector predicts whether the turn ends there. The ground truth is a single reference turn-end time per turn, the labelled instant at which the speaker actually finishes.
These reference times come from our annotation pipeline. Each call is two-party telephone speech with one speaker per channel, and marks the moment the speaker yields the floor. In a two-party call this is every turn boundary where the floor passes to the other speaker, together with the final turn of a call, where the speaker stops and no one takes over. Each cut is extended past by up to 1.3 s of trailing silence, checked to contain little speech, and labelled at 100 Hz, with frames before set to 0 and frames from onward through the extension set to 1.
Two frame rates are in play, since these reference labels are produced at 100 Hz (10 ms), the rate of the annotation pipeline, whereas the encoder and detector operate at 25 Hz (40 ms), a 4× downsampling. We map the 100 Hz labels onto the 25 Hz grid the detector reads and state every window below in 25 Hz frames. We do not train on the native 0/1 labelling directly, because the trailing silence is labelled entirely positive and no silence segment is labelled negative, so a detector can score well simply by firing on silence without locating the transition at . We therefore score on a collar around , where frames within of are positive, frames well before are negative, and a short guard band together with the trailing silence are dropped from training and evaluation alike (Figure 1). We use frames (±400 ms) as our main setting and also report . Because the collar keeps only the frames around the boundary, where turn-final prosodic cues are concentrated [6, 7], a finding that prosody is redundant under this label is conservative.
Figure 1. A turn end from the data. The top panel shows the waveform with normalized energy and pitch contours falling into the boundary before silence. The bottom panel shows the native reference label (0 before , 1 through the silence extension) against our collar label, whose positives sit in a window around while the guard band and trailing silence are dropped.
Data and representations
We evaluate on a fixed sample of 2,221 conversational cuts from 1,354 speakers, with 100 Hz reference EoT labels, drawn from an internal evaluation set of English conversational telephone speech that spans 16 application domains (distinct client call types, or verticals). The sample is balanced across these 16 domains, drawing a comparable number of cuts from each so that no single domain dominates the reported numbers, and splits are speaker-disjoint throughout (§ 2-E).
| family | features (motivating work) |
| pitch / F0 | f0_norm, f0_delta, f0_range, f0_slope, f0_mean_local, f0_std_local, pitch_contour_type, pitch_acceleration, f0_slope_300ms_abs, f0_slope_signed_500ms, f0_slope_at_voicing_offset, f0_range_1s, pitch_reset [6, 7], [9, 36, 37, 38] |
| energy | rms_energy_norm, energy_delta, energy_slope, energy_drop [6, 39] |
| timing / pause | speaking_rate_local, syllable_rate, speech_burst_duration, pause_after_speech, time_since_last_pause, silence_duration, silence_duration_norm, speech_flag [4, 5, 10, 11, 17, 40] |
| voice quality | voicing_strength, voicing_onset_rate_1s, hnr_ratio, jitter [3, 6, 41] |
| spectral | spectral_tilt, spectral_rolloff, spectral_bandwidth, spectral_flatness, zero_crossing_rate, spectral_contrast_0–3, mfcc_delta_1–3 [3] |
Table 1. The 41 prosodic features by cue family, with the turn-taking work motivating each as a turn-end cue. Spectral-shape descriptors are carried for completeness, with no specific turn-taking prior.
The prosodic side is a bank of 41 frame-level features spanning five families, each grounded in an established turn-taking or endpointing cue (Table 1), computed at 100 Hz and pooled to 25 Hz by taking the mean and maximum in each window.
The encoder side is a streaming Zipformer ASR encoder [15], read at four depths: the input log-mel features (80-dim), the output of the first encoder stack (zip-early, 192-dim), the third stack (zip-mid, 512-dim), and the final encoder output (zip-final, 768-dim). The encoder has six stacks in all, which we also read individually for a finer depth sweep (§ 4-A). We extract these activations from a single non-streaming forward pass, so that forward hooks fire on every stack, and verified that its final-layer output matches the deployed streaming causal encoder almost exactly (cosine 1.0 on cuts shorter than the encoder’s left context and on longer cuts). We therefore analyse offline activations that faithfully represent the streaming encoder.
Metrics
Every comparison is an increment, in that we never score prosody in isolation but ask how much a detector improves when prosody is added to a representation it already has. Let denote the encoder features and the prosodic features, and let be their concatenation. We train the same detector on and on and summarise the gain from with two paired measures of end-of-turn detection,
where is the frame’s turn-end label and the conditioning set denotes the concatenation of the two feature sets, not their sum. is the area under the ROC curve, the probability that a randomly chosen turn-end frame scores above a randomly chosen non-turn-end frame, so measures how much prosody improves the detector’s ranking of frames. is the held-out cross-entropy in bits per frame, so measures how much prosody sharpens the detector’s predicted probabilities. Both are defined so that 0 means prosody adds nothing once is present and a positive value means it helps.
has a clean information-theoretic reading. The held-out cross-entropy is the average number of bits needed to encode the true turn-end labels using the detector’s predicted probabilities, and it is small when the detector predicts those labels well and large when it predicts them poorly. is the reduction in this coding cost when prosody is added to the encoder features. Under a minimum-description-length view [31], that reduction is exactly the turn-end information prosody carries about the labels beyond what the encoder already supplies, so a larger means prosody contributes more information the encoder lacked. The quantity is detector-relative, in that it measures the information a given detector can exploit rather than a bound on the information actually present, so a stronger detector may extract more from the same features and move in either direction. We therefore rerun every measurement with a second detector (§ 2-D) and read the effect through the time-shift control rather than trusting any single value.
AUC captures ranking performance, the probability that a turn-end frame outscores a non-turn-end frame, and we report it because it is the standard detection metric. But once the encoder alone already ranks frames well its AUC sits near its ceiling, so any gain prosody adds is small and hard to read from alone. We therefore lead with , which still registers such gains, and report alongside it.
Our second experiment (§ 4-B) asks how much of each prosodic feature the encoder already represents, by trying to predict the feature from the encoder’s own activations, since a feature the encoder reconstructs well is one it already carries. We score this reconstruction with the held-out coefficient of determination,
where is the true feature value, the prediction, and the training-split mean. An of 1 is perfect reconstruction and 0 is no better than always predicting the training mean.
Detectors and controls
We use the same detector throughout, held fixed across conditions so that any difference in a metric reflects the representation and not the detector. Two detectors play this role, and we name them here and refer to them by name in the experiments.
Our main detector is the logistic-regression detector, an -regularised logistic regression solved with L-BFGS [42]. We standardise every input dimension using training-split statistics, which puts features of different scales on a common footing and lets the penalty act evenly across them. Turn-end frames are rare, so we fit on a class-balanced subsample with equal numbers of turn-end and non-turn-end frames, and the detector therefore cannot score well simply by predicting the majority class. We tune the regularisation strength separately for each representation, by a grid search over that maximises held-out AUC under four-fold cross-validation. The folds are speaker-grouped, keeping all frames from a speaker together, so tuning is never scored on a speaker it has already seen. Tuning per representation gives each its own best-fit penalty, so the paired -versus- comparison reflects the value of rather than a difference in how well happened to be regularised.
As a capacity check, a small neural network repeats every measurement. It uses a single hidden layer of 64 units, regularised with weight decay and early stopping. We recalibrate its outputs with Platt scaling so that its predicted probabilities, and with them the bits metric, stay meaningful [43]. This network is the stronger of the two detectors, so if an effect appears only with it, the logistic detector was too limited to use the information and we trust the neural answer.
The time-shift control (motivated in § 1) isolates genuinely turn-aligned timing. Within each utterance we circularly shift the prosodic features in time by a random, nonzero offset, so each feature is realigned against a different point in the same turn. This keeps the feature’s marginal distribution and its within-utterance autocorrelation intact while destroying its alignment with the turn boundary, and it leaves the encoder features, the labels, and the speaker grouping untouched. We average over several random offsets and report the corrected value as real minus time-shifted. Whatever value survives the shift cannot come from boundary timing, and we read it as a speaker-identity shortcut, in which a feature’s roughly fixed per-speaker level predicts boundaries through its correlation with how that speaker’s turns tend to be arranged.
Statistical significance and splits
We test significance with a bootstrap that resamples whole speakers rather than individual frames, since frames within a speaker are correlated, following standard practice for confidence intervals in speech-system evaluation [44]. We draw speaker blocks with replacement 1,000 times, recompute and on each resample, and report the 2.5 and 97.5 percentiles as a 95% confidence interval, treating an effect as present when the interval excludes zero and negligible when it sits close to zero on both sides. The data is partitioned by speaker into three disjoint splits, with about 50% of the frames for fitting the detector and tuning its regularisation, 25% for any feature or operating-point selection, and 25% held out for reporting. Groups are formed by speaker and assigned so that no group spans two splits, so a detector cannot do well by memorising a talker, and the reported numbers and intervals all come from speakers seen in neither fitting nor selection.
Experiments and Results
We take the three questions from the introduction in turn, then ask whether the answer generalises beyond one encoder and whether it survives our design choices. Two guards, both defined above, run under every measurement, and we report each one as we go. The neural detector reruns each test, so a result can never be blamed on a detector too weak to use the prosody it is given. The time-shift control recomputes each test with the prosodic features shifted off the turn boundary, so any value that survives the shift is a speaker-identity shortcut rather than real timing.
How much does prosody add, and where?
We compare the detector on the encoder alone against the same detector on the encoder plus prosody, at each depth, in bits per frame (bits) and AUC (AUC) Table II. Because the encoder alone already ranks frames well, AUC has little room to move, so we read the effect mainly from bits.
| representation | AUC() | bits | bits corr. | AUC | AUC corr. |
| fbank | 0.685 | +0.053 | +0.080 | +0.092 | +0.091 |
| zip-early | 0.874 | +0.021 | +0.016 | +0.014 | +0.007 |
| zip-mid | 0.908 | +0.015 | +0.008 | +0.006 | +0.003 |
| zip-final | 0.893 | +0.018 | +0.004 | +0.006 | +0.0015 |
| fbank | 0.686 | +0.181 | +0.184 | +0.153 | +0.135 |
| zip-early | 0.884 | +0.030 | +0.021 | +0.012 | +0.009 |
| zip-mid | 0.912 | +0.019 | +0.018 | +0.010 | +0.007 |
| zip-final | 0.901 | +0.024 | +0.014 | +0.006 | +0.006 |
Table II. Value prosody adds over the encoder at each depth (=10), in bits/frame (bits) and AUC (AUC). Corrected columns remove the speaker-identity shortcut via the time-shift control.
The value collapses in a single encoder stack. Prosody’s value does not fade gradually through the encoder. It collapses in one step, right at the boundary between the raw input and the first encoder stack Figure 2. On the raw filterbank features prosody adds a large bits=+0.053, but a single stack later most of that gain is gone, down to +0.021, and it stays small through the rest of the encoder Table II. The neural detector shows the same collapse, so a detector too weak to use prosody is not the explanation. A finer read across all six encoder stacks tells the same story, the added value staying well below its raw-input level throughout Figure 3. The final layer is also the deployment-relevant one. Real turn detectors read a recogniser’s learned outputs, its encoder or joint-network states [45], transcript [46], or a pretrained speech encoder [47], not a separate hand-built prosodic front-end. Prosody thus adds least exactly where a detector would use it.
Figure 2. Encoder-alone accuracy (AUC, green) and the extra accuracy from hand-built prosody, by depth. The prosody increment splits into genuine timing (orange, removed by the time-shift control) and a speaker-identity shortcut (yellow, which survives). Numbers above bars are the total AUC.
Figure 3. Value prosody adds (bits/frame) by depth, with solid lines for the real features and dashed lines for the time-shifted control, for both detectors. The inset shows a finer six-stack read (AUC, speaker-bootstrap CIs), and the circled point is the final layer.
What is left identifies the speaker, not the turn end. Figure 2 separates the two. Each bar splits the prosody gain into the part the time-shift control removes, which is genuine boundary timing (orange), and the part that survives the control, which is the speaker-identity shortcut (yellow). On the raw fbank input the bar is almost all orange, so there prosody really is reading turn timing. That orange part shrinks away inside the encoder, and by the final layer the genuine-timing gain is only AUC=+0.0015, nothing in practice.
What is left of the already tiny AUC=+0.006 total is mostly the yellow speaker-identity part. A speaker’s average pitch or rate barely changes within a turn, so it cannot mark when the turn ends, yet it still predicts boundaries because different speakers tend to end their turns in different ways. A second control confirms this, since replacing every frame with the utterance’s own average prosody, which leaves only the per-speaker level, recovers the same small gain, so the residue really is speaker identity. The neural detector leaves a slightly larger residual, but at AUC=+0.006 it too is negligible.
Prosody alone and silence alone both trail the encoder. Two doubts could weaken the result, and a baseline settles each one. The first doubt is that prosody might carry little turn-end information to begin with. To check this, we run a detector on prosody alone, and it reaches AUC 0.79. That is below every encoder depth but well above chance, so prosody does carry real turn-end information, and the encoder has picked up almost all of it. The second doubt is that the encoder might have learned only to spot silence rather than the prosodic cues we care about. To check this, we run a detector on voice activity alone, a speech flag and silence duration, and it reaches only AUC 0.63. If we tune it to catch 90% of real turn ends, it also goes off on 99.6% of the pauses in the middle of a turn, when the speaker has not actually stopped, so it fires almost all the time. Silence on its own cannot tell a turn end from a mid-turn pause [4], [5], so the encoder must be using far more than silence to reach its accuracy.
At an operating point, prosody barely lowers false alarms. The metrics so far are threshold-free, but a deployed endpointer runs at a fixed operating point, where what matters is how often it fires before the turn has actually ended. We therefore set each detector to 90% recall of true turn ends on the collar and measure its false-alarm rate, the share of mid-turn interior frames it wrongly flags, for and for Table III. The detection view agrees with the added-value curve. On the raw filterbank prosody cuts the false-alarm rate by 0.11 (from 0.71 to 0.60), a reduction the time-shift control confirms is genuine turn timing. That reduction collapses inside the encoder, and at the final layer prosody lowers false alarms by only 0.026 (0.3560.330), of which the control leaves just +0.011 as genuine. So even at a concrete operating point, a prosodic front-end buys almost no reduction in early firing at the deep encoder, and what little it buys is mostly the speaker-identity shortcut. This is the within-collar operating point, not a full streaming endpointer, which must also decide through the post-turn silence and is judged on latency (Sec. IV).
| representation | FA() | FA() | FA (CI) | FA corr. |
| fbank | 0.711 | 0.598 | +0.113 [.084,.144] | +0.121 |
| zip-early | 0.385 | 0.343 | +0.041 [.024,.055] | +0.020 |
| zip-mid | 0.291 | 0.271 | +0.019 [.009,.029] | +0.011 |
| zip-final | 0.356 | 0.330 | +0.026 [.012,.039] | +0.011 |
Table III. False-alarm rate at 90% recall on the collar (logistic, =10), for the encoder alone () and with prosody (). FA is the FA reduction from prosody, with a speaker-bootstrap 95% CI, and the corrected column removes the speaker-identity shortcut. A corrected value can slightly exceed the raw FA when time-shifted prosody raises the error rate.
Why does prosody become redundant?
Section III-A shows that prosody stops helping once it enters the encoder, but not why. The natural explanation is that the encoder already holds the prosody, so a second hand-built copy has nothing to add. We can test this directly by trying to read each prosodic feature back out of the encoder. At every depth we predict the feature from a short causal window of activations, the current frame and the four before it, about 200 ms in all, and report the held-out . A high means the encoder still represents the feature, so supplying it again is redundant. A low means the encoder has thrown it away. We read out with both a linear ridge regressor and a small neural network, so a low score cannot simply be the limit of a linear probe.
The encoder holds prosody most where prosody helps least. Reconstruction (Table IV) runs opposite to the added-value curve (Figure 3). As prosody’s added value falls through the early and middle stacks (0.053→0.021→0.015 bits), its reconstructability climbs the other way (=0.33 on raw features to 0.53 and 0.48 inside the encoder). The two trends move against each other because a hand-built copy adds the least exactly where the encoder already represents prosody best. The added-value curve and the reconstruction probe therefore agree that prosody becomes redundant once the encoder has absorbed it.
| depth | prosody (lin) | prosody (MLP) | energy (control) |
| fbank | 0.33 | 0.40 | 1.00 |
| zip-early | 0.53 | 0.51 | 0.98 |
| zip-mid | 0.48 | 0.42 | 0.91 |
| zip-final | 0.20 | 0.09 | 0.71 |
Table IV. Median reconstruction of prosody from a short causal encoder window, by depth (linear and neural reconstructors). Broadband energy is a present-signal control.
At the final layer it keeps only what the task needs. The rise does not last, and at the final layer reconstructability drops to =0.20 (ridge) and 0.09 (neural), so the encoder builds a prosodic representation in the middle and then strips it to the turn-relevant part. Even at its peak the probe recovers only about half the signal, so the encoder keeps no clean copy, only enough to make a hand-built one redundant. How it retains this prosody is partly lexical, since deep ASR layers carry word-level information [48], but this cannot be the whole story, because a self-supervised encoder trained with no word labels shows the same redundancy (wav2vec2, Sec. III-D).
The final-layer drop is lost information, not a limit of the probe. A low final-layer could instead mean the probe failed, so we rule out two ways it might. Does the probe work at that depth? Broadband energy, plainly in the audio, is recovered at every depth, slipping only to 0.71 at the final layer. Is prosody too rich to decode? The same probe reads it well at the middle stacks (0.53, 0.48), so difficulty is constant across depths. Widening the window from 5 to 11 frames does not recover it either (0.20 to 0.17), so the information is not merely spread across frames. The final-layer drop is therefore a genuine loss of prosodic detail, not a probe artefact.
Which features, if any, still matter?
Testing the whole bank at once can hide a standout, because a single feature that captures turn timing the encoder missed could be drowned out by the rest and leave the bank looking redundant. To catch such a case, we test each feature on its own, measuring its added value with and without the time-shift control and reading the drop as the share that comes from genuine turn timing.
The strongest-looking features are really the speaker shortcut. Before the control, the features with the most apparent value at the final layer are activity and voice-quality cues, the speech flag and normalized silence duration. The time-shift control removes almost all of it, which places them with the speaker-identity shortcut from Sec. III-A rather than with real timing. Once corrected, what remains is a scatter of pitch and spectral features worth about 0.001 bits each, with no clear winner.
No single feature clears our usefulness line. Figure 4 shows this directly, placing each feature by how well the encoder reconstructs it (x-axis) against how much real turn timing it adds (y-axis). A feature worth keeping would sit high and to the left, hard for the encoder to reconstruct yet still adding real timing, and that corner is empty. The best surviving feature adds only about 0.001 bits, roughly 50× less than the 0.053 bits prosody adds on raw input, so no reasonable threshold would keep any of them, including the pitch cues that classical end-of-turn detectors relied on most. Feature by feature, the result points the same way as the bank as a whole.
Figure 4. Each final-layer prosodic feature by encoder reconstructability (, x) and time-aligned added value (, y). The reference lines mark the median (0.20) and our usefulness threshold (0.0015).
Does this generalize to other encoders and data?
Every result so far comes from one encoder and one set of recordings, so the pattern might hold only for that encoder or data, so we check both.
The collapse is not specific to our encoder. We rerun the added-value measurement, with the time-shift control, on two open-source encoders. wav2vec2 [49] is self-supervised and never sees a recognition label, while Whisper [16] is trained for recognition on a different architecture. Both expect 16 kHz audio, so we upsample our 8 kHz telephone speech and run them off the shelf, which makes this supporting evidence rather than an exact replication. On Whisper the corrected value starts at +0.040 on the input and falls by an order of magnitude to +0.0035 within a few layers, the same one-step collapse as on our own encoder (Figure 5). On wav2vec2 it never rises, staying below 0.003 at every layer, because the encoder already reaches AUC 0.95 to 0.98 on its own. wav2vec2 is the telling case, since it is trained with no recognition labels and never learns the words, yet prosody is just as redundant on it, which rules out the reading § 4-B could not exclude, that the encoder absorbs prosody through lexical content alone.
Figure 5. Time-shift-corrected value prosody adds on two open-source encoders, by layer. Grey lines mark our encoder’s input and final-layer values for reference.
The result reproduces on fully public data. Both encoder checks still rely on our own recordings. To remove that dependence we repeat the measurement on the public CASPER corpus [50] of two-party English conversation, with the same feature bank, annotation pipeline, and time-shift control (Table V). Adding a standard 80-dim log-mel front-end gives a raw-input reference and reproduces the full raw-to-encoder collapse, not just the across-layer redundancy. The picture matches our main study, since raw log-mel prosody adds AUC=+0.085, of which +0.059 survives the control as genuine timing, whereas at any learned layer the corrected gain falls to at most 0.002 even though the encoders reach AUC 0.90 to 0.995. The speaker-identity split recurs too, with Whisper’s early layer showing a raw AUC=+0.046 that the control almost removes (+0.001). A public corpus, open-source encoders, and a public pipeline thus reproduce all three findings, so none depends on our own data or model. The log-mel front-end here is a standard 80-dimensional filterbank rather than our encoder’s exact input, so it serves as a comparable raw-input baseline rather than an identical one.
| representation | layer | AUC() | AUC | AUC corr. |
| raw input | log-mel | 0.763 | +0.085 | +0.059 |
| wav2vec2-base | early (0) | 0.994 | −0.001 | 0.000 |
| wav2vec2-base | deep (12) | 0.964 | +0.001 | +0.002 |
| whisper-small | early (0) | 0.897 | +0.046 | +0.001 |
| whisper-small | deep (12) | 0.995 | 0.000 | 0.000 |
Table V. PUBLIC-DATA REPLICATION ON CASPER USING LOGISTIC DETECTOR.
Does the result survive our design choices?
The redundancy survives the three design choices most likely to matter. Changing the label does nothing, since narrowing or widening the collar leaves the result unchanged and even a label that marks every post-boundary frame lets the best single feature add at most AUC=+0.002. The feature selector makes no difference either, because several standard selectors [51, 52, 53, 54, 55] return much the same set yet cannot separate genuine timing from the speaker shortcut, so our feature-level verdict rests on the time-shift control rather than on selection. Nor does the data split matter, since all our numbers already come from held-out speakers and retraining on the selected features and a fresh speaker group reproduces the full-bank value (AUC=+0.0047 against +0.006 at the final layer, Table III). Across all three, the redundancy holds, so it belongs to the encoder rather than to our setup.
Limitations and Future Work
Our study covers one language, telephone-band audio, and one main ASR encoder. The two-encoder and public-data checks push against this, but both open-source encoders were run off the shelf on upsampled audio, so a wider sweep across languages, channels, and architectures is still owed. Our 41-feature bank spans the major turn-taking cue families but is not exhaustive, so an omitted cue could behave differently.
The larger limitation is what we measure. The collar scores frames around the turn boundary, so we quantify the turn-end information prosody adds there, not how a deployed endpointer behaves. A real system decides at every frame and is judged on latency and false alarms across the whole stream, which our collar operating point (Table III) only partly captures. The added-value metric is also detector-relative, reporting what our two detectors extract rather than the information present in principle, and using both detectors together with the time-shift control makes a large hidden effect unlikely though not impossible. Two questions remain open, the first being whether the redundancy extends to other prosody-dependent tasks such as emotion recognition, and the second being the best read-out depth, since the encoder is slightly more accurate one stack earlier (0.91 at zip-mid against 0.89 at the final layer), in line with middle layers holding the richest acoustic detail [27, 28].
Conclusion
We investigated how much turn-relevant prosody a streaming ASR encoder already represents through three complementary analyses, and they agree. Prosody’s added value is large on raw audio but falls sharply after a single encoder stack, and a time-shift control shows that most of the residual reflects speaker identity rather than turn timing. Reconstruction explains this pattern, because the encoder represents prosody most strongly in its middle layers and keeps only the turn-relevant part at the final layer, so supplying it again adds little. The same pattern appears on two further encoders, including one trained without word labels, so it is not merely lexical. Prosody remains a genuine turn-taking signal, reaching AUC 0.79 on its own, but a trained encoder has already absorbed most of its turn-relevant part. For practitioners building on a trained streaming encoder, a separate hand-built prosodic front-end offers limited benefit for end-of-turn detection, even at the deep layer where a detector usually attaches, and no pared-down subset recovers it. Confirming this inside a full streaming endpointer, with latency and false-alarm costs, remains future work.
References
- [1]T. Stivers, N. J. Enfield, P. Brown, C. Englert, M. Hayashi, T. Heinemann, G. Hoymann, F. Rossano, J. P. de Ruiter, K.-E. Yoon, and S. C. Levinson, “Universals and cultural variation in turn-taking in conversation,” Proceedings of the National Academy of Sciences, vol. 106, no. 26, pp. 10 587–10 592, 2009.
- [2]S. C. Levinson and T. Torreira, “Timing in turn-taking and its implications for processing models of language,” Frontiers in Psychology, vol. 6, p. 731, 2015.
- [3]G. Skantze, “Turn-taking in conversational systems and human-robot interaction: A review,” Computer Speech & Language, vol. 67, p. 101178, 2021.DOI
- [4]M. Heldner and J. Edlund, “Pauses, gaps and overlaps in conversations,” Journal of Phonetics, vol. 38, no. 4, pp. 555–568, 2010.DOI
- [5]A. Raux and M. Eskanazi, “Optimizing endpointing thresholds using dialogue features in a spoken dialogue system,” in Proc. 9th SIGdial Workshop on Discourse and Dialogue. Association for Computational Linguistics, 2008, pp. 1–10.DOI
- [6]A. Gravano and J. Hirschberg, “Turn-taking cues in task-oriented dialogue,” Computer Speech & Language, vol. 25, no. 3, pp. 601–634, 2011.DOI
- [7]C. E. Ford and S. A. Thompson, “Interactional units in conversation: Syntactic, intonational, and pragmatic resources for the management of turns,” in Interaction and Grammar, E. Ochs, E. A. Schegloff, and S. A. Thompson, Eds. Cambridge: Cambridge University Press, 1996, pp. 134–184.DOI
- [9]S. Duncan, “Some signals and rules for taking speaking turns in conversations,” Journal of Personality and Social Psychology, vol. 23, no. 2, pp. 283–292, 1972.DOI
- [10]L. Ferrer, E. Shriberg, and A. Stolcke, “Is the speaker done yet? faster and more accurate end-of-utterance detection using prosody,” in Proc. 7th Int. Conf. on Spoken Language Processing (ICSLP), 2002, pp. 2061–2064.DOI
- [11]——, “A prosody-based approach to end-of-utterance detection that does not require speech recognition,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), vol. 1, 2003, pp. I–608–I–611.DOI
- [12]E. Ekstedt and G. Skantze, “How much does prosody help turn-taking? investigations using voice activity projection models,” in Proc. SIGDIAL, 2022, arXiv:2209.05161.arxiv.org/abs/2209.05161
- [13]K. Inoue, B. Jiang, E. Ekstedt, T. Kawahara, and G. Skantze, “Real-time and continuous turn-taking prediction using voice activity projection,” in Proc. Int. Workshop on Spoken Dialogue Systems Technology (IWSDS), 2024, arXiv:2401.04868.arxiv.org/abs/2401.04868
- [14]A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Proc. Interspeech, 2020, pp. 5036–5040, arXiv:2005.08100.arxiv.org/abs/2005.08100
- [15]Z. Yao, L. Guo, X. Yang, W. Kang, F. Kuang, Y. Yang, Z. Jin, L. Lin, and D. Povey, “Zipformer: A faster and better encoder for automatic speech recognition,” in Proc. Int. Conf. on Learning Representations (ICLR), 2024, arXiv:2310.11230.arxiv.org/abs/2310.11230
- [16]A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International Conference on Machine Learning (ICML), 2023, arXiv:2212.04356.arxiv.org/abs/2212.04356
- [17]M. Shannon, G. Simko, S.-Y. Chang, and C. Parada, “Improved end-of-query detection for streaming speech recognition,” in Proc. Interspeech, 2017, pp. 1909–1913.DOI
- [18]R. Masumura, T. Asami, H. Masataki, R. Ishii, and R. Higashinaka, “Online end-of-turn detection from speech based on stacked time-asynchronous sequential networks,” in Proc. Interspeech, 2017, pp. 1661–1665.DOI
- [19]R. Maas, A. Rastrow, C. Ma, K. Goehner, G. Tiwari, S. Joseph, and B. Hoffmeister, “Combining acoustic embeddings and decoding features for end-of-utterance detection in real-time far-field speech recognition systems,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 5544–5548.DOI
- [20]S.-Y. Chang, R. Prabhavalkar, Y. He, T. N. Sainath, and G. Simko, “Joint endpointing and decoding with end-to-end models,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 5626–5630.DOI
- [21]G. Skantze, “Towards a general, continuous model of turn-taking in spoken dialogue using LSTM recurrent neural networks,” in Proc. 18th Annual SIGdial Meeting on Discourse and Dialogue. Association for Computational Linguistics, 2017, pp. 220–230, aCL Anthology W17-5527.DOI
- [23]E. Ekstedt and G. Skantze, “Voice activity projection: Self-supervised learning of turn-taking events,” in Proc. Interspeech, 2022, arXiv:2205.09812.arxiv.org/abs/2205.09812
- [24]J. Wang, L. Chen, A. Khare, A. Raju, P. Dheram, D. He, M. Wu, A. Stolcke, and V. Ravichandran, “Turn-taking and backchannel prediction with acoustic and large language model fusion,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2024, arXiv:2401.14717.arxiv.org/abs/2401.14717
- [25]A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024.arxiv.org/abs/2410.00037
- [26]B. Veluri, B. N. Peloquin, B. Yu, H. Gong, and S. Gollakota, “Beyond turn-based interfaces: Synchronous LLMs as full-duplex dialogue agents,” in Proc. EMNLP, 2024, arXiv:2409.15594.arxiv.org/abs/2409.15594
- [27]A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self-supervised speech representation model,” in Proc. IEEE ASRU, 2021, arXiv:2107.04734.arxiv.org/abs/2107.04734
- [28]A. de la Fuente and D. Jurafsky, “A layer-wise analysis of mandarin and english suprasegmentals in SSL speech models,” in Proc. Interspeech, 2024, pp. 1290–1294, arXiv:2408.13678.arxiv.org/abs/2408.13678
- [31]E. Voita and I. Titov, “Information-theoretic probing with minimum description length,” in Proc. EMNLP, 2020, arXiv:2003.12298.arxiv.org/abs/2003.12298
- [32]S. O’Connor Russell, D. Charau, and N. Harte, “The role of prosodic and lexical cues in turn-taking with self-supervised speech representations,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2026, arXiv:2601.13835.
- [33]R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, no. 11, pp. 665–673, 2020, arXiv:2004.07780.DOI
- [34]S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller, “Unmasking clever hans predictors and assessing what machines really learn,” Nature Communications, vol. 10, p. 1096, 2019.
- [35]A. Torralba and A. A. Efros, “Unbiased look at dataset bias,” in Proc. IEEE CVPR, 2011, pp. 1521–1528.DOI
- [36]S. Bögels and F. Torreira, “Listeners use intonational phrase boundaries to project turn ends in spoken interaction,” Journal of Phonetics, vol. 52, pp. 46–57, 2015.DOI
- [37]A. Wennström and A. F. Siegel, “Keeping the floor in multiparty conversations: intonation, syntax, and pause,” Discourse Processes, vol. 36, no. 2, pp. 77–107, 2003.
- [38]C. De Looze, I. Yanushevskaya, J. Kane, and A. Ní Chasaide, “Pitch range declination and reset in turn-taking organisation,” in Speech Prosody 2014, 2014, pp. 1062–1066.DOI
- [39]J. K. Local, J. Kelly, and W. H. G. Wells, “Towards a phonology of conversation: turn-taking in Tyneside English,” Journal of Linguistics, vol. 22, no. 2, pp. 411–437, 1986.DOI
- [40]C. W. Wightman, S. Shattuck-Hufnagel, M. Ostendorf, and P. J. Price, “Segmental durations in the vicinity of prosodic phrase boundaries,” The Journal of the Acoustical Society of America, vol. 91, no. 3, pp. 1707–1717, 1992.pubmed.ncbi.nlm.nih.gov/1564206
- [41]R. Ogden, “Turn transition, creak and glottal stop in Finnish talk-in-interaction,” Journal of the International Phonetic Association, vol. 31, no. 1, pp. 139–152, 2001.
- [42]F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.arxiv.org/abs/1201.0490
- [43]C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. 34th Int. Conf. on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 70, 2017, pp. 1321–1330, arXiv:1706.04599.arxiv.org/abs/1706.04599
- [44]M. Bisani and H. Ney, “Bootstrap estimates for confidence intervals in ASR performance evaluation,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), vol. 1, 2004, pp. 409–412.DOI
- [45]S.-Y. Chang, B. Li, T. N. Sainath, C. Zhang, T. Strohman, Q. Liang, and Y. He, “Turn-taking prediction for natural conversational speech,” in Proc. Interspeech, 2022, pp. 1821–1825, arXiv:2208.13321.arxiv.org/abs/2208.13321
- [46]LiveKit, “Using a transformer to improve end of turn detection,” https://livekit.com/blog/using-a-transformer-to-improve-end-of-turn-detection, Dec. 2024, open-source end-of-turn utterance turn detector: a 135M transformer over the ASR transcript. Accessed 2026-07-02.
- [47]V. Varadarajan and J. Vytheswaran, “VoTurn-80M: State-of-the-art turn detection for voice agents,” https://blog.vogent.ai/posts/voturn-80m-state-of-the-art-turn-detection-for-voice-agents, 2025, multimodal turn detection model (Vogent-Turn-80M). Accessed 2026-07-02.
- [48]A. Pasad, C.-M. Chien, S. Settle, and K. Livescu, “What do self-supervised speech models know about words?” Transactions of the Association for Computational Linguistics, vol. 12, pp. 372–391, 2024, arXiv:2307.00162.arxiv.org/abs/2307.00162
- [49]A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in Neural Information Processing Systems (NeurIPS), 2020, arXiv:2006.11477.arxiv.org/abs/2006.11477
- [50]C. Xiao, R. Liang, X. Zhang, M. E. Tiryaki, V. Bae, L. Shankar, R. Yang, E. Poon, E. Dupoux, S. Khudanpur, and L. P. Garcia Perera, “CASPER: A large scale spontaneous speech dataset,” arXiv preprint arXiv:2506.00267, 2025.arxiv.org/abs/2506.00267
- [51]N. Meinshausen and P. Bühlmann, “Stability selection,” Journal of the Royal Statistical Society: Series B, vol. 72, no. 4, pp. 417–473, 2010, arXiv:0809.2932.arxiv.org/abs/0809.2932
- [52]E. Candès, Y. Fan, L. Janson, and J. Lv, “Panning for gold: Model-x knockoffs for high-dimensional controlled variable selection,” Journal of the Royal Statistical Society: Series B, vol. 80, no. 3, pp. 551–577, 2018.
- [53]M. Liu, E. Katsevich, L. Janson, and A. Ramdas, “Fast and powerful conditional randomization testing via distillation,” Biometrika, vol. 109, no. 2, pp. 277–293, 2022.DOI
- [54]Y. Benjamini and Y. Hochberg, “Controlling the false discovery rate: A practical and powerful approach to multiple testing,” Journal of the Royal Statistical Society: Series B (Methodological), vol. 57, no. 1, pp. 289–300, 1995.DOI
- [55]H. Peng, F. Long, and C. Ding, “Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 8, pp. 1226–1238, 2005.pubmed.ncbi.nlm.nih.gov/16119262




