betaFeedback

WorksComputer Science

Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors

Abstract

Speech recognition in real time in far-field conversational settings still incurs significant errors. Identifying potential error spans during streaming ASR permits a recourse mechanism where the speaker can be asked to repeat the affected content under cleaner acoustic conditions. We propose PARDON, a new framework for online error localization, that identifies spans likely to benefit from this intervention. Our supervision uses clean audio to estimate how the recognizer's output might change under improved acoustic conditions. We probabilistically align noisy and clean decodes using a pair-HMM to obtain soft supervision. These soft targets train a counterfactual gain scorer by combining a suite of acoustic and language features to produce token-level scores, that are further grouped into spans for targeted recourse. An LLM uses the ASR transcript of the repetition to revise only the selected span. At a precision of 70%, PARDON achieves the highest recall by identifying up to 52% more erroneous tokens compared to the strongest baseline. Targeted repetition yields relative WER reductions of up to 24% under a budget that accounts for both interruption frequency and the lengths of re-uttered spans.

Introduction

Streaming ASR predicts transcriptions in real time as the model processes incoming speech. However, ambient noise or interfering speakers could obscure parts of the speech signal, leading to localized perception errors. Retrospectively identifying these potential erroneous spans in an evolving transcript is our primary problem of interest. Addressing this problem also has practical value because the online setting naturally supports a recourse mechanism. Once the error spans are localized, a timely request to the speaker to repeat the affected segment under better conditions could improve transcription accuracy.

Error localization has been previously studied in the context of spoken dialogue systems (Skantze, [26]) and through confidence estimation modules for streaming ASR ([21, 20]). Supervision in these prior works typically comes from reference transcripts. However, reference-based supervision targets transcript correctness and not the potential benefit of re-utterance: some errors, such as those arising out of homophones, persist even after re-utterance under clean acoustic condition. This motivates using targets that directly reflect how the recognizer’s output might change under cleaner acoustic conditions.

In order to support targeted recourse, we propose a new framework for online error localization in streaming ASR called PARDON, short for Predicting Acoustic Re-utterance Desirability Over Noisy spans. PARDON derives supervision from a counterfactual estimate of what the recognizer would output if the same content were heard under cleaner acoustic conditions. We retrospectively compare these clean decodes with the original noisy decodes by marginalizing over multiple plausible alignments of the clean and noisy decodes using a pair hidden Markov model (pair-HMM). The pair-HMM assigns each token a soft label given by the posterior probability of a recognition error at that position. These soft labels supervise a counterfactual gain estimator that takes as input both language features and acoustic features extracted via a streaming posterior alignment with audio frames. This estimator runs alongside the ASR’s decodes producing token-level scores, which are grouped to identify spans that could benefit from re-utterance.

For recourse, the speaker is asked to repeat the identified span under cleaner acoustic conditions.1 The ASR model transcribes the repetition and an LLM uses the resulting transcript to revise only the selected span, after which decoding continues.

Our contributions can be summarized as follows: (1) We introduce PARDON, a new framework for online error localization in streaming ASR that isolates potentially erroneous spans through counterfactual supervision from clean audio and a pair-HMM that accounts for uncertainty in the error locations. To the best of our knowledge, the online error localization setting we consider is itself new. (2) We develop a streaming counterfactual gain scorer that combines acoustic and language features, with supervision from the pair-HMM, to identify spans for targeted recourse through repetition. (3) For error localization, we obtain the highest average precision across four challenging corpora of conversational speech, with the largest margins in the high-precision regime, where we recall up to 52% more of the erroneous tokens than the strongest baseline at a precision of 70%. (4) We harness these localized errors for targeted recourse and reduce the word error rate by up to 24% relative, and remove up to 30% more of the errors than recent prior work under the same budget on the number of speaker interruptions and re-uttered words.

Related Work

Localizing recognition errors. Word-level confidence estimation has a long history in speech recognition [11]. For end-to-end models, confidence is obtained either directly from the output distribution, where entropy-based measures detect incorrect words in CTC and RNN-T models considerably better than the token probability [13], or from a learned estimator over the decoder’s internal states [20]. Most of this work aims to detect recognition errors in general. We instead estimate the gain from re-uttering the audio under cleaner conditions. This shifts our solution in four ways: how the detector is supervised, which acoustic features it uses, how evidence is pooled over the transducer’s alignment lattice, and how it runs in a streaming setup alongside standard ASR decoding, where flagged token-level scores are converted to spans for online recourse.

Post-hoc correction. Recognition output can also be corrected after the fact, by sequence-to-sequence correctors [14] and, more recently, by large language models that rewrite the transcript from the recognizer’s NN-best hypotheses [16, 32, 4], optionally conditioned on the noise …

transcript from the recognizer’s NN-best hypotheses [16, 32, 4], optionally conditioned on the noise [10]. Such correction is complementary to clarification: it draws on linguistic context rather than new acoustic evidence, and could be applied on top of a clarified transcript. We include a few-shot prompted NN-best corrector as a zero-interruption reference point.

Humans in the loop. Prior work on human-assisted ASR correction has largely focused on spoken dialogue systems [2, 26]. These systems serve a single task, such as booking a room or navigation, and treat anything the user says unrelated to the task as an error. Furthermore, these systems often ask the user to repeat or rephrase the entire turn rather than localizing the error. More recent work involves users in correcting open-ended speech: He et al. [8] let meeting participants correct speaker attributions with brief feedback, and Peng et al. [19] diagnose the causes of recognition errors and resolve them through multi-turn clarification. Unlike Peng et al. [19], who seek clarification after each turn, PARDON runs synchronously with the streaming decoder and interrupts the speaker when it identifies a span where repetition is likely to help.

Our Approach

Figure 1 shows the pipeline of PARDON. Let X^\hat{X} denote an audio captured under noisy acoustic conditions, that is fed to a frozen streaming RNN-transducer Mθ\mathcal{M}_{\theta}, that produces a decode Y^\hat{Y} as explained in § 3.1. A centerpiece of PARDON is a CF-Gain Scorer that after each decoded token y^j\hat{y}_j, synchronously emits a counterfactual gain probability eje_j that a re-utterance of the corresponding audio would improve the transcript, together with two deletion estimates djd_j and ℓj\ell_j (§ 3.2). These scores are used by the recourse module to decide when to ask for a re-utterance; in our initial, simple interface, it pools them into spans as the transcript grows, and as soon as a noise-corrupted span is identified, asks the speaker to repeat the words in the selected span (§ 3.3.1). The repetition is decoded by the same transducer Mθ\mathcal{M}_{\theta}, and the resulting clarification transcript is passed, together with the noisy transcript Y^\hat{Y}, to an LLM corrector that produces the corrected transcript Y^′\hat{Y}', after which decoding continues (§ 3.3).

Figure 1. PARDON overview. As the speaker talks, the CF-Gain Scorer assigns every emitted token a score eje_j and two deletion estimates djd_j and ℓj\ell_j, and the scores are grouped into spans. Once a span closes, PARDON asks the speaker to repeat it, using an anchor-based question. An LLM corrector puts the repeated words into the transcript.

Background of RNN-T

An RNN-T has three components. The encoder maps X^\hat{X} to states h[1],…,h[T]h[1], \ldots, h[T], one per frame; in a streaming model, h[t]h[t] sees no audio beyond frame tt apart from a fixed look-ahead. The predictor encodes the tokens emitted so far using an RNN: g[j]g[j] summarizes y^1,…,y^j−1\hat{y}_1, \ldots, \hat{y}_{j-1}. The joint network combines the two into a distribution over the vocabulary V\mathcal{V} and the blank ϵ\epsilon:

P(⋅∣t,j)=softmax⁡(Joint⁡(h[t],g[j])).(1)P(\cdot\mid t,j)=\operatorname{softmax}(\operatorname{Joint}(h[t],g[j])). \tag*{(1)}

Greedy decoding scans the frames, emitting the most probable symbol at each step: it stays at frame tt after a token and moves to t+1t+1 after a blank. We write tjt_j for the frame at which y^j\hat{y}_j is emitted.

The CF-Gain Scorer

After each predicted token from the noisy audio, the CF-Gain scorer estimates the counterfactual probability that a re-utterance of the corresponding audio would lead to a gain in transcription quality. More precisely, we model pθ(Ej=1∣X^,y^1,…y^j)p_{\theta}(E_j=1\mid\hat{X},\hat{y}_1,\ldots\hat{y}_j) where EjE_j denotes an error between j−1j-1 and jj that can be eliminated with a clean re-utterance of the audio after the previous token y^j−1\hat{y}_{j-1}. The error EjE_j could be either because of missing some token between y^j−1\hat{y}_{j-1} or y^j\hat{y}_j, or because the token y^j\hat{y}_j is wrong. Furthermore, to estimate the amount of error that a re-utterance can recover, we also estimate the number of deletions in the gap between y^j−1\hat{y}_{j-1} and y^j\hat{y}_j, through djd_j, the probability that an error at y^j\hat{y}_j is a deletion, and ℓj\ell_j, the number of deleted words in the gap.

To supervise the training of this estimator, we assume a training dataset D={X^i,Yi,Xic:i=1…N}\mathcal{D}=\{\hat{X}_i,Y_i,X_i^c:i=1\ldots N\} denoting the same reference text YiY_i uttered under noisy settings X^i\hat{X}_i and clean settings XicX_i^c. Such datasets are increasingly available in the public domain, typically obtained by recording conversations with far-field and close-talking microphones, and we experiment with four such in § 5. If clean audio is not available, we show that even synthetically generated TTS audio suffices as distant supervision for our CF-gain scorer. We present the design of the CF-scorer in § 3.2.1 and describe its training in § 3.2.2.

Design of the CF-Gain scorer

We implement the CF-Gain scorer pθ(Ej=1∣X^,y^1,…y^j):=ejp_{\theta}(E_j=1\mid\hat{X},\hat{y}_1,\ldots\hat{y}_j):=e_j as another LSTM, much like the LM predictor, with two additional output heads for djd_j and ℓj\ell_j (Figure 2). The input to the LSTM is a set of features fj∈Rd\mathbf{f}_j\in\mathbb{R}^{d} extracted from X^,y^1,…y^j\hat{X},\hat{y}_1,\ldots\hat{y}_j. Instead of processing them afresh, we extract the feature set fj\mathbf{f}_j by tapping the current state of the RNN-T. The features in fj\mathbf{f}_j capture evidence from the audio frames, the LM state, and the joint token probabilities.

Figure 2. The CF-Gain Scorer (shaded) and the frozen transducer it reads (dashed). The emission and dwell posteriors qq and r~\tilde{r} of the streaming alignment lattice pool frame-level evidence to each token. The pooled vector hˉj\bar{h}_j is concatenated with the predictor state g[j]g[j] to form fjf_j. A unidirectional LSTM then scores each token as it is emitted via three heads: giving fixable error score eje_j and the deletion estimates djd_j and ℓj\ell_j (§ 3.2.2).

Features from audio. At each frame tt, the encoder reads the audio frames up to tt and outputs a hidden state h(l)[t]h^{(l)}[t] at each layer ll. We extract the outputs of the final encoder layer and of one intermediate layer. In addition we extract a voice-activity probability v[t]v[t] that frame tt contains speech. We pool these per frame to tokens through the alignment lattice of the RNN-T.

Alignment lattice. When decoding y^j\hat{y}_j at frame tjt_j, we create a lattice over all possible alignments of the tokens y^j′\hat{y}_{j'}, j′≤jj' \leq j, to the frames t≤tjt \leq t_j, by restricting the joint distribution P(⋅∣t,j′)P(\cdot\mid t,j') (Eq. (1)) to y^j′\hat{y}_{j'} and ϵ\epsilon. Furthermore, we limit the lattice to a band around the greedy emission path, which limits the alignments to those around the greedy decode (Appendix § 7). Forward–backward over this lattice gives the emission posterior q[t,j]q[t,j], that y^j\hat{y}_j is emitted at frame tt, and the dwell posterior r[t,j]r[t,j], that the decoder advances past frame tt while waiting to emit y^j\hat{y}_j.

We pool the frame-level audio features to each token under two weightings. The emission posterior q[⋅,j]q[\cdot,j] concentrates on the frames at which y^j\hat{y}_j was emitted. The normalized dwell posterior r~[t,j]=r[t,j]/∑t′r[t′,j]\tilde{r}[t,j] = r[t,j]/\sum_{t'}r[t',j] covers the frames the decoder passed over while waiting to emit y^j\hat{y}_j. For layer ll, the emission view hˉjemit,(l)=∑tq[t,j]h(l)[t]\bar{h}^{\mathrm{emit},(l)}_j = \sum_t q[t,j]h^{(l)}[t] describes the audio of the word that was output, and the dwell view hˉjdwell,(l)\bar{h}^{\mathrm{dwell},(l)}_j, the same sum weighted by r~\tilde{r}, describes the audio the decoder skipped, which is where a deleted word would have been. We similarly pool the voice activity score v[t]v[t].

Features from the joint network. We compute

a set of scalars from the distribution P(⋅∣t,j)P(\cdot\mid t,j) output by the joint network such as its entropy, the log-probability of the emitted token y^j\hat{y}_j, and the other-token mass, the probability of emitting a token that is neither y^j\hat{y}_j nor the blank. These scalars are defined at every lattice node (t,j)(t,j), so we pool them to tokens with qq and r~\tilde{r} as well. Pooling them in different ways gives 20 scalars, listed in Appendix § 8.

Features from LM. From the text branch of the transducer, we include g[j]g[j] as a feature that summarizes y^1,…,y^j−1\hat{y}_1,\ldots,\hat{y}_{j-1}. This feature is tied to the token, and does not require pooling.

Training the CF-Gain scorer

Figure 3 gives an overview of our method of training the gain scorer given dataset D={(X^i,Yi,Xic):i=1…N}\mathcal{D} = \{(\hat{X}_i,Y_i,X_i^c): i=1\ldots N\}. For each instance ii, we obtain the decode Y^i\hat{Y}_i from the noisy audio X^i\hat{X}_i, and the decode Yic=(y1c,…,yJc)Y_i^c=(y_1^c,\ldots,y_J^c) of the counterfactual clean rendering XicX_i^c, by the same frozen recognizer Mθ\mathcal{M}_{\theta}. Comparing these sequence-level decodes, we need to get token-level supervision at each jj. Since the sequences, can be differing in multiple positions, we marginalize over all possible alignments of the two sequences as follows.

Figure 3. Training supervision. The same frozen transducer decodes the noisy audio X^\hat{X} and the counterfactual audio XcX^c. A pair-HMM channel aligns the noisy decode Y^\hat{Y} to YcY^c and gives, for each token, the targets e~j\tilde{e}_j, d~j\tilde{d}_j and ℓ~j\tilde{\ell}_j of the scorer’s three heads; only the scorer receives gradients (§ 3.2.2).

For a fixed alignment AA between Y^i\hat{Y}_i and YicY_i^c, each noisy token gets three labels: (1) Ej(A)∈{0,1}E_j(A)\in\{0,1\} that denotes if, as per this alignment, the jj-th token in Y^i\hat{Y}_i is inserted, substituted, or a token before it is deleted, (2) Dj(A)∈{0,1}D_j(A)\in\{0,1\} set to 1 if words are deleted before y^j\hat{y}_j, and (3) Lj(A)L_j(A) representing the number of deleted words. Words deleted after the last token have no following token, so we append an end-of-sequence position Y^N+1\hat{Y}_{N+1} that takes their label. We marginalize these labels over all possible alignments efficiently using a pair hidden Markov model Durbin et al. [6].

A pair-HMM takes as input the two sequences Y^i\hat{Y}_i and YicY_i^c and gives the posterior probability P(A∣Yic,Y^i)P(A \mid Y_i^c, \hat{Y}_i) of each alignment. Using an efficient forward–backward algorithm over the pair HMM, we can calculate the marginal probability of the error at y^j\hat{y}_j as

e~ij=∑AP(A∣Yic,Y^i)Ej(A).(2)\tilde{e}_{ij} = \sum_A P(A \mid Y_i^c, \hat{Y}_i) E_j(A). \tag*{(2)}

We marginalize Dj(A)D_j(A) over the alignments in the same way and divide it by e~ij\tilde{e}_{ij}, which gives d~ij\tilde{d}_{ij}, the probability that the error at y^j\hat{y}_j is a deletion. The pair-HMM requires a set of hyper-parameters, and we fit these via a standard expectation maximization procedure as described in Appendix § 14.

Training Objective. To train the scorer, we use weighted binary cross entropy (BCE) losses for the two probabilities and a Poisson loss for the deletion count as follows:

min⁡θ∑i,j[Bw(e~ij,eij)+e~ijBwd(d~ij,dij)+1[ℓ~ij≥1]P(ℓ~ij,ℓij)](3)\min_{\theta} \sum_{i,j} \left[ B_w(\tilde{e}_{ij}, e_{ij}) + \tilde{e}_{ij} B_{w_d}(\tilde{d}_{ij}, d_{ij}) + 1[\tilde{\ell}_{ij} \ge1] \mathcal{P}(\tilde{\ell}_{ij}, \ell_{ij}) \right] \tag*{(3)}

Here Bw(y,p)B_w(y,p) is the weighted BCE loss of the prediction pp against the target yy, and P(ℓ~,ℓ)\mathcal{P}(\tilde{\ell},\ell) is the Poisson loss of ℓ~−1\tilde{\ell}-1 with mean ℓ−1\ell-1. The BCE weights ww and wdw_d are used to upweight the positive term in each BCE loss, respectively, and are estimated such that the balance of positive and negative supervision is equal. The deletion loss is weighted by e~ij\tilde{e}_{ij}, so that the deletion head focuses on distinguishing deletions from other error types.

Recourse

From tokens to spans

Noise-induced damage tends to extend over several consecutive tokens. It is also more natural to ask a speaker to repeat a stretch of words than to ask several questions about separate words. We therefore group the token scores into spans of words as the transcript is produced. Each token is first assigned a score mjm_j — the expected number of errors that a re-utterance would correct at y^j\hat{y}_j and in the gap before it:

mj=ej[(1−dj)+djℓj].(4)m_j = e_j \left[(1-d_j) + d_j \ell_j\right]. \tag*{(4)}

An error at y^j\hat{y}_j is either a substitution or insertion with probability 1−dj1-d_j, that costs one word, or a deletion with probability djd_j, that costs ℓj\ell_j words. Each word kk gets the highest mjm_j among its tokens, written as mˉk\bar{m}_k.

We assume that each interruption costs cIc_I and each re-uttered word costs cRc_R, where each word is re-uttered together with content in the gap before it. A word that is not selected for repetition costs mˉk\bar{m}_k, i.e., the expected number of errors left uncorrected at that word and in the preceding gap. We can use a cumulative sum (CUSUM; Page [18]) algorithm to compute the best spans in an online setting. More details of the method in Appendix § 9.§ 1.

Clarification and correction

The request to repeat a span is a fixed template that quotes the two words on either side of the span, its anchors, and asks the speaker to restate what they said in between, e.g. “I didn’t catch what you said between ‘L’ and ‘R’. Could you repeat it?”. Spans at the start or end of the utterance have only one anchor and use a shorter template (Appendix § 11).

For each span, the LLM corrector receives the transcript decoded up to the interruption, with the span marked and earlier spans already corrected, together with the request and the clarification transcript. It returns a replacement for the words of the span, and the corrected transcript Y^′\hat{Y}' leaves every other word unchanged, so the corrector can only alter the stretches the speaker was asked about.

Experimental Setup

ASR System. All experiments use Nemotron Speech Streaming (NVIDIA [17]), a 0.6B-parameter English transducer with a FastConformer (Rekesh et al. [23]) encoder of 24 layers, a 1024-dimensional encoder output at a frame rate of 80 ms, and a vocabulary of 1024 subword tokens plus the blank. We use its default attention context of 70 frames to the left and 13 to the right, so the encoder consumes audio in chunks of 1.12 s with at most 1.04 s of look-ahead. The transducer is frozen throughout.

Datasets. We use four corpora of conversational speech: two of them contain meeting conversations, AMI (Carletta et al. [3]) and NOTSOFAR-1 (Vinnikov et al. [29]), and two of them contain dinner party conversations, CHiME-6 (Watanabe et al. [31]) and DiPCo (Van Segbroeck et al. [28]). All four corpora contain both far-field channels and a close-talking recording of every speaker (a headset in AMI, a close-talking microphone in NOTSOFAR-1 and DiPCo, and a binaural microphone in CHiME-6). We use the official session-level train/development/evaluation splits [1, 5]. Each utterance is a speaker turn extracted from both far-field and close-talking recordings using reference timestamps. Very short turns and turns containing reference words marked as unintelligible are excluded from training; development and evaluation sets retain all turns. The development (dev) set is used for model selection and calibration, and all numbers are reported on the evaluation (eval) set.

To show how PARDON performs in the absence of a parallel close-talking recording, we also create a synthetic rendering of each training utterance by synthesizing its reference transcript with a text-to-speech (TTS) model, Kokoro-82M (hexgrad, 2025). Dev and eval labels come from the close-talking recordings. Appendix § 10 describes the corpora, how the turns are cut, speech synthesis details and the train/dev/eval set sizes after segmentation.

Implementation Details

We derive supervision by aligning normalized noisy and clean decodes by the same ASR system using a pair-HMM, and copying word-level labels to its subwords. Token features combine emission- and dwell-pooled encoder states from 13th and 24th blocks with the predictor state and scalar features of § 3.2.1; voice activity probabilities come from Silero VAD [25]. The CF-Gain Scorer uses a single-layer unidirectional LSTM with 128 hidden units and 3 prediction heads of § 3.2.2. Remaining implementation details are in Appendix § 20.

Baselines

Token-level scoring. We compare the CF-Gain Scorer with five baseline scorers: an entropy-based method and four learned confidence estimators. i) Laptev and Ginsburg [13]: Tsallis entropy of the recognizer’s output distribution at token emission, with α\alpha tuned on the development set. ii) Qiu et al. [20]: a Transformer over hypothesis tokens that attends to encoder frames and predicts token errors and deletions. iii) Wang et al. [30]: a Transformer over token embeddings and encoder frames at emission. iv) TruCLeS (Ravi et al., [22]: an LSTM predicting confidence based on lexical similarity to the reference. v) Peng et al. [19]: detects errors from the recognizer’s internal states with separate perception-error and deletion detectors, whose scores we combine. We omit their comprehension-error detector from our experiments as it targets errors not repairable by clean re-utterance. Every estimator is trained on re-utterance gains from close-talking decodes rather than reference-based labels, on the same data as the CF-Gain Scorer, and adapted to streaming, using only input available at token emission (Appendix § 13).

Recourse. We compare PARDON with two baselines: one without user clarification and the other using it exactly the same way as PARDON. i) NN-best correction: Following HyPoradise [4], an LLM corrects the transcript from a 5-best list: the greedy transcript and the four best distinct hypotheses from a width-32 beam search. We use its hypotheses-to-transcription (H2T) prompt with Gemini 3.8 Flash and 10 examples from the dev set (Appendix § 12.§ 2). We also report the oracle that picks the best hypothesis from the list. ii) Proactive ASR [19]: We use the same recourse pipeline as our system, with

Peng et al. [19] used for localizing the errors.

Evaluation metrics

Token-level scoring. Token scores eje_j are evaluated on the eval set of each corpus against the hard labels EjE_j from the close-talking decode, using average precision (AP) and recall at precisions of 0.5 and 0.7. We also report the share of erroneous tokens, which is the AP of a random scorer.

Recourse. Recourse is evaluated by the relative WER reduction over the far-field greedy transcript against gold references, under a re-utterance burden B=nI+βnWB = n_I + \beta n_W. Here, nIn_I and nWn_W count interruptions and re-uttered words per minute of audio, respectively; we use β∈{0.05,0.10}\beta\in\{0.05, 0.10\}. The speaker’s reply is simulated by the close-talking recording of the span or by TTS of its reference words. Further recourse details are in Appendix § 11, some recourse examples are in Appendix § 15, and tuning details for costs (cR,cI)(c_R, c_I) on dev set are in Appendix § 19.

Results and Analysis

Token-level scoring

Scoring accuracy. Table 1 compares the CF-Gain Scorer with the five scoring baselines on each corpus. The CF-Gain Scorer has the highest AP on all four corpora scoring 1.0 to 3.3 points above the strongest baseline, Qiu et al. [20]. The margin is larger at high precision: at a precision of 70%, it recalls 7.2, 13.5 and 5.0 points more of the erroneous tokens than the best baseline on AMI, NOTSOFAR-1 and CHiME-6.

ScorerAMINOTSOFAR-1CHiME-6DiPCo
APRecall at prec. 50Recall at prec. 70APRecall at prec. 50Recall at prec. 70APRecall at prec. 50Recall at prec. 70APRecall at prec. 50Recall at prec. 70
Entropy38.723.84.039.228.66.351.049.718.541.329.011.0
Qiu et al.55.158.525.356.561.625.965.780.044.442.133.02.5
Wang et al.52.956.318.350.551.113.762.475.536.135.49.20.0
TruCLeS47.741.913.145.535.28.858.667.230.032.31.9–
Peng et al.47.246.03.849.554.57.857.172.611.241.125.91.7
CF-Gain56.561.732.559.868.739.466.780.349.444.136.610.0

Table 1. Token-level scoring on the eval set of each corpus, against labels from the close-talking decode. Trained scorers are trained on the same corpus; the CF-Gain Scorer is evaluated by eje_j. Share of erroneous tokens (the AP of a random scorer): AMI 21.5, NOTSOFAR-1 16.4, CHiME-6 24.8, DiPCo 19.3. Best result for each metric within each corpus is in bold; – indicates that the target precision was never reached.

Out-of-domain generalization. To check how well the CF-Gain Scorer performs out of domain, we apply a scorer trained on one corpus unchanged to the other corpus of the same type: from AMI to

NOTSOFAR-1 for meetings, and from CHiME-6 to DiPCo for dinner parties (Table 2). On NOTSOFAR-1, the scorer trained on AMI exceeds the in-domain scorer on every metric by 3.4 AP points. On DiPCo, the scorer trained on CHiME-6 exceeds the in-domain scorer on every metric by 10.7 AP points. In both cases, the source corpus has far more training data than the target (62.9 vs. 6.2 hours and 28.5 vs. 0.7 hours; Table 5), which suggests that a scorer trained on a larger corpus of the same genre can replace one trained in-domain. With a subset of the source as large as the target’s training set, the transferred scorer still stays within 4 AP points of the in-domain one (Appendix § 16).

TestScorerTrained onAPRecall at prec. 50Recall at prec. 70
NOTSOFAR-1Entropy–39.228.66.3
CF-GainNOTSOFAR-159.868.739.4
CF-GainAMI63.271.739.7
DiPCoEntropy–41.329.011.0
CF-GainDiPCo44.136.610.0
CF-GainCHiME-654.858.024.4

Table 2. A CF-Gain Scorer trained on one corpus is applied unchanged to the other corpus of the same genre, and compared with a scorer trained on the target corpus along with the training-free entropy baseline. Best result per block in bold.

CorpusTrain Utt.Train HoursDev Utt.Dev HoursEval Utt.Eval Hours
AMI27,86162.99,1458.88,1288.5
NOTSOFAR-13,5336.29,4935.516,55711.0
CHiME-630,77828.56,9266.59,9405.9
DiPCo5260.72,2462.03,2793.2

Table 5. Utterances and hours of speech per set after segmentation. Training excludes short and partly unintelligible turns; development and evaluation keep every turn.

Synthetic supervision. We also train a scorer on each corpus with labels from TTS of the reference (Appendix § 10.§ 4) instead of the close-talking recording, and evaluate it against the close-talk labels (Table 3). On AMI, CHiME-6 and DiPCo, TTS loses at most 1.8 points of AP and 5.1 points of recall at 70% precision. On AMI and CHiME-6 it still recalls more at 70% precision than every baseline of Table 1, all trained with close-talk labels, and on DiPCo it still has a higher AP than all of them. NOTSOFAR-1 is the exception: there it loses 8.3 points of AP and 23.5 points of recall at 70% precision, and falls below Qiu et al. [20].

CorpusSupervisionAPRecall at prec. 50Recall at prec. 70
AMIClose-talk56.561.732.5
TTS54.759.127.4
NOTSOFAR-1Close-talk59.868.739.4
TTS51.554.815.9
CHiME-6Close-talk66.780.349.4
TTS65.381.044.7
DiPCoClose-talk44.136.610.0
TTS43.734.18.2

Table 3. CF-Gain Scorers trained with labels from the close-talking recording or from a synthesized (TTS) rendering of the reference, both evaluated against close-talk labels on the evaluation set. Best result per corpus in bold.

Ablations. Table 4 removes one group of features at a time from the CF-Gain Scorer, on AMI and CHiME-6. The intermediate encoder layer and the joint-network scalars both matter. Without layer 13, recall at 70% precision drops by 8.4 points on AMI and by 3.1 points on CHiME-6. Without the joint-network scalars, it drops by 4.3 and 8.4 points. Limiting frame level features to the emission frame based on greedy decoding has a smaller effect: AP drops by 0.8 points on AMI and 0.2 points on CHiME-6.

FeaturesAMICHiME-6
APRecall at prec. 50Recall at prec. 70APRecall at prec. 50Recall at prec. 70
CF-Gain (all features)56.561.732.566.780.349.4
Emission frame only55.760.533.466.581.446.9
w/o layer 1354.258.724.165.778.446.3
w/o joint scalars54.859.528.263.777.641.0

Table 4. Feature ablations of the CF-Gain Scorer on the evaluation sets of AMI and CHiME-6. Emission frame only: uses encoder features only from the greedy emission frame. w/o layer 13: only the final encoder layer is used. w/o joint scalars: the scalar features derived from the joint network are removed. All numbers are in percent; best per column in bold.

Recourse

Figure 4 shows the WER reduction of PARDON and of the same pipeline driven by Peng et al. [19] against the re-utterance burden, for spans of at most six words and β=0.05\beta= 0.05 on AMI and CHiME-6. On AMI, PARDON removes 23% of the word errors with synthesized replies at a burden of 6 per minute, where Peng et al. needs 8, and 16% with close-talking replies at 6 per minute, where Peng et al. needs 9. On CHiME-6, whose noisy decode has about twice the WER of AMI (64% vs. 30%), the same reductions take higher burdens: PARDON

Figure 4. Recourse on the AMI (a) and CHiME-6 (b) eval sets (β=0.05\beta= 0.05, spans of at most six words): relative WER reduction against the re-utterance burden with synthesized (solid) or close-talking (dashed) replies. PARDON uses the CF-Gain Scorer; Peng et al. uses the same recourse pipeline with the detectors of Peng et al. [19]. Horizontal lines: the zero-interruption NN-best correction of HyPoradise and the NN-best oracle. We also report WER reductions achieved through speech enhancement (which is quite poor) in Appendix § 17.

removes 18% and 13% at 7 per minute, where Peng et al. needs 8 and 9.

Exceeding the zero-interruption NN-best correction of HyPoradise (6% on AMI and 5% on CHiME-6) requires a burden of around 2 per minute on both corpora, with either kind of reply. At the highest burdens, PARDON with synthesized replies comes within 4 points of the NN-best oracle (28% on AMI and 27% on CHiME-6), which no zero-interruption method can exceed with these hypotheses. Synthesized replies remove more errors than close-talking ones, by up to 7 points, likely because the close-talking recording of a span also carries disfluencies and crosstalk from the other speakers and is decoded with more errors. The ordering of the two scorers is the same with β=0.10\beta= 0.10 and with spans of up to eight words (Appendix § 18).

Conclusion

We introduced PARDON, which improves streaming recognition under noise by asking the speaker to repeat the stretches it has likely misheard. Its CF-Gain Scorer reads its evidence from the transducer’s own alignment lattice, is trained on soft labels that marginalize over the alignment between the noisy and the counterfactual clean decode, and estimates how many words a repetition would recover. On four corpora, it ranks erroneous tokens above entropy-based and learned confidence estimators, and it transfers unchanged to another corpus of the same genre. With these scores, PARDON removes 23% of the word errors on far-field AMI meetings with synthesized replies, and 16% with close-talking replies, at a re-utterance burden of under 6 per minute, more than both an NN-best LLM corrector and the same pipeline driven by Peng et al. [19]. Evaluating clarification with real speakers instead of simulated replies is the natural next step.

Limitations

We study a single recourse mechanism, targeted repetition. In realistic interactions, users may respond in many other ways, for instance by paraphrasing, correcting only part of the span or adding context, which would make the LLM corrector’s task harder, and our replies are simulated rather than collected from real speakers. The scoring step is designed for one family of streaming recognizers, RNN transducers, whose alignment lattice it reads; other architectures would need a different source of token-level evidence. Finally, we focus on errors caused by far-field recording, because the parallel close-talking audio our supervision needs is readily available in such corpora; the same counterfactual supervision could be applied to other kinds of ASR errors wherever a cleaner rendering of the same speech can be obtained.

Ethical Considerations

All four corpora (AMI, NOTSOFAR-1, CHiME6 and DiPCo) are publicly released for research and were recorded with the consent of their participants; we use them under their licenses and do not redistribute any audio. No new data was collected from people: speaker replies are simulated with the existing close-talking recordings or with synthesized speech.

A deployed system raises two risks. First, recognition errors are more frequent for speakers whose accents or speaking styles are under-represented in training data, so such speakers may be interrupted more often and bear more of the burden of correction; we have not measured this, and it should be evaluated before deployment. Second, the LLM corrector can insert words the speaker did not say, which matters when transcripts serve as records, such as meeting minutes; a deployed system should mark corrected spans.

Transducer Alignment Lattice: Definitions

For an utterance X^\hat{X} of TT frames (x^1,…,x^T)(\hat{x}_1,\ldots,\hat{x}_T) and a transcript Y^\hat{Y} of NN tokens (y^1,…,y^N)(\hat{y}_1,\ldots,\hat{y}_N), an alignment is a monotonic path through a T×(N+1)T \times(N+1) grid whose node (t,j)(t,j) is the state in which the first tt frames x^1,…,x^t\hat{x}_1,\ldots,\hat{x}_t have been processed, y^1,…,y^j−1\hat{y}_1,\ldots,\hat{y}_{j-1} have been emitted, and the decoder waits to emit y^j\hat{y}_j; row N+1N+1 follows the last token. We consider only the paths that run from (1,1)(1,1) to (T,N+1)(T,N+1). They differ only in when each token is emitted, and the probability of the transcript is the total probability of all of them,

P(Y^∣X^)=∑paths (1,1)→(T,N+1)P(path∣X^).(5)P(\hat{Y}\mid\hat{X})=\sum_{\text{paths }(1,1)\to(T,N+1)}P(\text{path}\mid\hat{X}). \tag*{(5)}

At each node the model produces a distribution over V∪{ϵ}\mathcal{V}\cup\{\epsilon\}, where V\mathcal{V} is the vocabulary of the model and ϵ\epsilon the blank,

P(⋅∣t,j)=softmaxJoint⁡(h[t],g[j]),(6)P(\cdot\mid t,j)=\operatorname{softmaxJoint}(h[t],g[j]), \tag*{(6)}

where h[t]h[t] is the encoder state at frame tt, conditioned on the audio up to that frame, and g[j]g[j] the predictor state, conditioned on y^1,…,y^j−1\hat{y}_1,\ldots,\hat{y}_{j-1}.

Everything that follows is computed on a lattice restricted to the 1-best transcript Y^\hat{Y} obtained by greedy decoding. Although the model assigns probability to every element of V\mathcal{V} at every node, the restricted lattice admits only the token that continues Y^\hat{Y}. Only two transitions therefore leave (t,j)(t,j): emitting ϵ\epsilon, which advances the frame, (t,j)→(t+1,j)(t,j)\to(t+1,j), and emitting y^j\hat{y}_j, which advances the token index, (t,j)→(t,j+1)(t,j)\to(t,j+1), with probabilities

b[t,j]=P(ϵ∣t,j),e[t,j]=P(y^j∣t,j).(7)b[t,j]=P(\epsilon\mid t,j),\qquad e[t,j]=P(\hat{y}_j\mid t,j). \tag*{(7)}

The forward variable α[t,j]\alpha[t,j] is the probability of reaching (t,j)(t,j) and the backward variable β[t,j]\beta[t,j] the probability of completing the alignment from it:

α[1,1]=1,(8)\alpha[1,1]=1, \tag*{(8)}
α[t,j]=α[t−1,j]b[t−1,j]+α[t,j−1]e[t,j−1],(9)\alpha[t,j]=\alpha[t-1,j]b[t-1,j]+\alpha[t,j-1]e[t,j-1], \tag*{(9)}
β[T,N+1]=b[T,N+1],(10)\beta[T,N+1]=b[T,N+1], \tag*{(10)}
β[t,j]=β[t+1,j]b[t,j]+β[t,j+1]e[t,j],(11)\beta[t,j]=\beta[t+1,j]b[t,j]+\beta[t,j+1]e[t,j], \tag*{(11)}

with out-of-lattice terms taken as zero, giving Z=P(Y^∣X^)=α[T,N+1]b[T,N+1]=β[1,1]Z=P(\hat{Y}\mid\hat{X})=\alpha[T,N+1]b[T,N+1]=\beta[1,1].

Emission probability. q[t,j]q[t,j] is the posterior probability that token y^j\hat{y}_j is emitted at frame tt, given that Y^\hat{Y} is the final transcript; it is the occupancy of the label transition out of (t,j)(t,j):

q[t,j]=α[t,j]e[t,j]β[t,j+1]Z.(12)q[t,j]=\frac{\alpha[t,j]e[t,j]\beta[t,j+1]}{Z}. \tag*{(12)}

Dwell probability. r[t,j]r[t,j] is the posterior probability, given that Y^\hat{Y} is the final transcript, that the decoder advances from frame tt to frame t+1t+1 while waiting to emit y^j\hat{y}_j, i.e. in the gap between y^j−1\hat{y}_{j-1} and y^j\hat{y}_j (the frame at which y^j−1\hat{y}_{j-1} is emitted included); it is the occupancy of the blank transition out of (t,j)(t,j):

r[t,j]=α[t,j] b[t,j] β[t+1,j]Z.(13)r[t,j] = \frac{\alpha[t,j]\,b[t,j]\,\beta[t+1,j]}{Z}. \tag*{(13)}

Normalization. The two quantities normalize along different axes. Each token is emitted exactly once in any alignment, so ∑tq[t,j]=1\sum_t q[t,j] = 1 and q[⋅,j]q[\cdot,j] is a distribution over frames. In contrast, because each frame is advanced exactly once, ∑jr[t,j]=1\sum_j r[t,j] = 1: r[⋅,j]r[\cdot,j] instead sums to the expected number of blank frames between y^j−1\hat{y}_{j-1} and y^j\hat{y}_j, which may exceed one or approach zero. When pooling frame-level features to token level we therefore divide r[⋅,j]r[\cdot,j] by its sum over frames.

Streaming estimation. The posteriors above use the whole utterance. In PARDON the posteriors of token y^j\hat{y}_j are computed when the greedy decoder emits it, at frame tjt_j. Let NjN_j be the number of tokens the decoder has emitted by frame tjt_j (at least jj, more if further tokens are emitted at the same frame). The truncated backward variable βj\beta_j follows the recursion above on frames 1,…,tj1,\ldots,t_j and rows 1,…,Nj+11,\ldots,N_j+1, with the terminal condition

βj[tj+1,Nj+1]=1,(14)\beta_j[t_j+1,N_j+1] = 1, \tag*{(14)}
βj[tj+1,i]=0(i≤Nj),(15)\beta_j[t_j+1,i] = 0 \quad(i \leq N_j), \tag*{(15)}

so that every path has emitted the NjN_j known tokens by the cut, and Zj=βj[1,1]Z_j = \beta_j[1,1]. The forward variable needs no change, since it depends only on earlier frames. The posteriors of token jj are then

qj[t,j]=α[t,j] e[t,j] βj[t,j+1]Zj,(16)q_j[t,j] = \frac{\alpha[t,j]\,e[t,j]\,\beta_j[t,j+1]}{Z_j}, \tag*{(16)}
rj[t,j]=α[t,j] b[t,j] βj[t+1,j]Zj,(17)r_j[t,j] = \frac{\alpha[t,j]\,b[t,j]\,\beta_j[t+1,j]}{Z_j}, \tag*{(17)}

for t≤tjt \leq t_j, and zero for later frames. When tj=Tt_j = T this is the full lattice. The end-of-sequence position y^N+1\hat{y}_{N+1} is never emitted, so it has no emission frame; its posteriors are computed once the utterance has ended, on the full lattice, from row N+1N+1 (the frames after the last token).

Band. Both recursions are further restricted to the nodes (t,j)(t,j) with

tj−1−δ≤t≤tj+k,(18)t_{j-1} - \delta\leq t \leq t_{j+k}, \tag*{(18)}

where the lower bound is 1 for j=1j = 1 and the upper bound is TT when j+k>Nj + k > N; α\alpha and βj\beta_j are zero outside the band. Since node (t,j)(t,j) is reached only once y^j−1\hat{y}_{j-1} has been emitted, a path may thus emit each token at most δ\delta frames before the decoder did, and may fall at most kk tokens behind it. We use k=3k = 3 and δ=16\delta= 16 frames.

Token Features

The input fj\mathbf{f}_j of the CF-Gain Scorer concatenates four pooled encoder vectors (the emission and dwell views of § 3.2.1 at the intermediate and the final encoder layer), the predictor state g[j]\mathbf{g}[j], and the 27 scalars below: 20 computed from the joint network’s output distribution (1–20) and 7 from the voice-activity probability v[t]v[t] (21–27). Every feature is computed from the streaming posteriors of § 3.2.1, so it uses only frames up to the emission of y^j\hat{y}_j; features are standardized with training-set statistics.

Notation. Pt=P(⋅∣t,j)P_t = P(\cdot\mid t,j) is the output distribution over V∪{ϵ}\mathcal{V} \cup\{\epsilon\} at node (t,j)(t,j), and e[t,j]e[t,j] and b[t,j]b[t,j] are the probabilities of the emitted token and of the blank (Appendix § 7). Let o[t,j]=1−e[t,j]−b[t,j]o[t,j] = 1 - e[t,j] - b[t,j] be the probability of any other token. The emission frames of y^j\hat{y}_j are Tjemit={t:q[t,j]>0.01}\mathcal{T}^{\mathrm{emit}}_j = \{t : q[t,j] > 0.01\}, and its dwell frames are Tjgap={t:r[t,j]≥0.01max⁡t′r[t′,j]}\mathcal{T}^{\mathrm{gap}}_j = \{t : r[t,j] \geq0.01\max_{t'} r[t',j]\}, the frames of the gap before it; r~[t,j]\tilde{r}[t,j] is the normalized dwell posterior. The voice-activity probability v[t]v[t] is the unweighted mean of the Silero VAD [25] outputs of the three 32 ms chunks that overlap the 80 ms encoder frame, including a chunk that overlaps it only partly. For a distribution pp, H(p)=−∑kpklog⁡pkH(p) = -\sum_k p_k \log p_k is its Shannon entropy and Sα(p)=(1−∑kpkα)/(α−1)S_\alpha(p) = (1 - \sum_k p_k^\alpha)/(\alpha- 1) its Tsallis entropy. A statistic over an empty frame set is 0.

Uncertainty at omission (5). How spread out the model’s output is when it emits y^j\hat{y}_j, averaged over the frames at which it may have been emitted:

(1) ∑tq[t,j]H(Pt)\sum_t q[t,j] H(P_t)

(2–4) ∑tq[t,j]Sα(Pt)\sum_t q[t,j] S_\alpha(P_t) for α=0.5,2,4\alpha= 0.5, 2, 4

(5) ∑tq[t,j]log⁡e[t,j]\sum_t q[t,j] \log e[t,j]

The Shannon entropy weighs all alternatives; the Tsallis entropies, which Laptev and Ginsburg [13] found well suited to transducer confidence, shift the weight from the many low-probability alternatives (α<1\alpha< 1) to the few strongest competitors (α>1\alpha> 1). The expected log-probability (5) is the usual token confidence.

Probability of other tokens (3). The probability the model gives to tokens other than y^j\hat{y}_j and the blank, averaged under three weightings:

∑tq[t,j] o[t,j], at the emission(19)\sum_t q[t,j]\,o[t,j],\ \text{at the emission} \tag*{(19)}
∑tr~[t,j] o[t,j], over the gap(20)\sum_t \tilde{r}[t,j]\,o[t,j],\ \text{over the gap} \tag*{(20)}
∑tw[t,j] o[t,j] / ∑tw[t,j], over both,(21)\sum_t w[t,j]\,o[t,j]\ /\ \sum_t w[t,j],\ \text{over both,} \tag*{(21)}

with w=q[⋅,j]+r~[⋅,j]w=q[\cdot,j]+\tilde{r}[\cdot,j]

At the emission, mass on other tokens indicates that a different word may have been spoken (a substitution). In the gap, where the decoder emitted only blanks, it indicates that the model considered emitting a word there and did not, the trace a deleted word leaves.

Frame statistics (12). The weighted averages above can hide a single frame at which the model strongly preferred something else. We therefore also take, without weighting, the mean, maximum and 90th percentile of two per-frame values over each frame set:

log⁡e[t,j] over t∈Tjemit(22)\log e[t,j]\ \text{over }t\in\mathcal{T}^{\mathrm{emit}}_j \tag*{(22)}
o[t,j] over t∈Tjemit(23)o[t,j]\ \text{over }t\in\mathcal{T}^{\mathrm{emit}}_j \tag*{(23)}
log⁡e[t,j] over t∈Tjgap(24)\log e[t,j]\ \text{over }t\in\mathcal{T}^{\mathrm{gap}}_j \tag*{(24)}
o[t,j] over t∈Tjgap(25)o[t,j]\ \text{over }t\in\mathcal{T}^{\mathrm{gap}}_j \tag*{(25)}

Over the emission frames, they describe how confidently and how consistently y^j\hat{y}_j was preferred. Over the gap, (15–17) show whether y^j\hat{y}_j was already likely before it was emitted, and (18–20) whether some other token was, at any point in the gap.

Voice activity (7). Whether the frames contain speech at all, independently of the recognizer:

∑tr~[t,j] v[t], over the gap(26)\sum_t \tilde{r}[t,j]\,v[t],\ \text{over the gap} \tag*{(26)}
mean, maximum and 90th percentile of v[t] over t∈Tjemit(27)\text{mean, maximum and 90th percentile of }v[t]\text{ over }t\in\mathcal{T}^{\mathrm{emit}}_j \tag*{(27)}
mean, maximum and 90th percentile of v[t] over t∈Tjgap(28)\text{mean, maximum and 90th percentile of }v[t]\text{ over }t\in\mathcal{T}^{\mathrm{gap}}_j \tag*{(28)}

Speech in a gap through which the decoder emitted only blanks suggests that words were spoken there and lost; little speech around an emitted token suggests that it was produced from noise rather than from a word.

The end-of-sequence position y^N+1\hat{y}_{N+1} is treated as a token emitted at the last frame: its emission frame set is that frame, its dwell frames are those after y^N\hat{y}_N, and since only the blank can be emitted from row N+1N+1, the probability of the final blank takes the place of ee, with e[t,N+1]=b[t,N+1]e[t,N+1]=b[t,N+1] and o[t,N+1]=1−b[t,N+1]o[t,N+1]=1-b[t,N+1].

Pardon Algorithm

Algorithm 1 summarizes the training of the CF-Gain Scorer (§ 3.2) and the streaming inference with span selection and recourse (§ 3.3).

Online spans with the CUSUM algorithm

Re-uttering word kk yields a net gain of mˉk−cR\bar{m}_k-c_R, and a span of words warrants an interruption when these gains sum to at least cIc_I. We track the gains with a cumulative sum (CUSUM; [18]):

Ck=max⁡(0,Ck−1+mˉk−cR),C0=0.(29)C_k=\max(0,C_{k-1}+\bar{m}_k-c_R),\qquad C_0=0. \tag*{(29)}

CkC_k is the largest total gain of a span of words that ends at word kk and starts after the last interruption. Whenever the cumulative gain becomes positive, we track its current value CkC_k, its maximum PP during the positive run and the word where PP was reached. Once P−Ck≥cIP-C_k\geq c_I, we interrupt and select the span from the first word of the positive run to where PP was reached. Words after the peak are not part of the span; they only ensure that we do not interrupt while the speech remains degraded. After the interruption, the cumulative gain resets to zero. If CkC_k returns to zero before P−CkP-C_k reaches cIc_I, no span is formed. If the turn ends first, we interrupt if P≥cIP\geq c_I.

Data Preparation

Corpora

The AMI Meeting Corpus ([3]) contains about 100 hours of four-person meetings recorded by headset microphones and far-field arrays. NOTSOFAR-1 ([29]) contains office meetings recorded in many different conference rooms by several commercial devices at once, with close-talking microphones worn by the participants; we keep one device per meeting, so that no speech appears twice. CHiME-6 ([31]) contains four-person dinner parties recorded in private homes by distant microphone arrays and by binaural microphones worn by each speaker. DiPCo ([28]) contains ten dinner-party sessions of four participants, recorded by five distant seven-microphone devices and by a close-talking microphone per speaker. In all four, the noisy input is the first channel of the first far-field array. Speaking turns in the two dinner-party corpora are much shorter than in the meeting corpora.

Algorithm 1 PARDON end to end. TRAIN (offline) fits the CF-Gain Scorer pθp_\theta on a dataset D\mathcal{D} of noisy and clean renderings of the same utterances; INFER runs alongside the streaming decode of Mθ\mathcal{M}_\theta. The pair HMM has parameters Φ=(π,γ,η,ρ)\Phi= (\pi,\gamma,\eta,\rho): the probability that a noisy word is an error (π\pi), that a run of deleted words continues (γ\gamma), that the error after a deletion is its anchor (η\eta), and that a non-anchor error is an insertion rather than a substitution (ρ\rho) (Appendix § 14); AA is an alignment of Y^i\hat{Y}_i to YicY_i^c, and Ej(A)E_j(A), Dj(A)D_j(A) mark that y^j\hat{y}_j is an error and that words are deleted before it under AA (§ 3.2.2). tjt_j is the frame at which y^j\hat{y}_j is emitted, and sjs_j the LSTM state. In CUSUM, CC is the running sum of Eq. (29), aa the first word of the open span (⊥\perp if none), and PP, pp the peak of CC and its word.

  1. procedure TRAIN(D={(X^i,Yi,Xic)}i=1N,MθD = \{(\hat{X}_i,Y_i,X_i^c)\}_{i=1}^N,\mathcal{M}_\theta)

  2. &nbsp;&nbsp;for all i=1,…,Ni = 1,\ldots,N do

  3. &nbsp;&nbsp;&nbsp;&nbsp;Y^i,Yic←\hat{Y}_i,Y_i^c \leftarrow word decodes of X^i,Xic\hat{X}_i,X_i^c by Mθ\mathcal{M}_\theta

  4. &nbsp;&nbsp;&nbsp;&nbsp;Ai0←A_i^0 \leftarrow hard alignment (longest common runs)

  5. &nbsp;&nbsp;end for

  6. &nbsp;&nbsp;Φ←\Phi\leftarrow event rates under {Ai0}\{A_i^0\}, e.g. ρ←#ins#ins+#sub\rho\leftarrow\frac{\#\mathrm{ins}}{\#\mathrm{ins}+\#\mathrm{sub}}

  7. &nbsp;&nbsp;for 10 iterations do &nbsp;&nbsp;▷ EM on the training pairs

  8. &nbsp;&nbsp;&nbsp;&nbsp;E: nx←∑iEP(A∣Yic,Y^i;Φ)[#x in A]n_x \leftarrow\sum_i \mathbb{E}_{P(A\mid Y_i^c,\hat{Y}_i;\Phi)}[\#x\text{ in }A] by forward–backward, for each event xx: error or correct word, deletion continued or ended, anchor or not, insertion or substitution

  9. &nbsp;&nbsp;&nbsp;&nbsp;M: π←nerrnerr+ncor,γ←ncontncont+nend,η←nancnanc+nnon-anc,ρ←ninsnins+nsub\pi\leftarrow\frac{n_{\mathrm{err}}}{n_{\mathrm{err}}+n_{\mathrm{cor}}},\quad\gamma\leftarrow\frac{n_{\mathrm{cont}}}{n_{\mathrm{cont}}+n_{\mathrm{end}}},\quad\eta\leftarrow\frac{n_{\mathrm{anc}}}{n_{\mathrm{anc}}+n_{\mathrm{non\text{-}anc}}},\quad\rho\leftarrow\frac{n_{\mathrm{ins}}}{n_{\mathrm{ins}}+n_{\mathrm{sub}}}

  10. &nbsp;&nbsp;end for

  11. &nbsp;&nbsp;freeze Φ\Phi

  12. &nbsp;&nbsp;for all ii and tokens jj of Y^i\hat{Y}_i do

  13. &nbsp;&nbsp;&nbsp;&nbsp;e~ij←∑AP(A∣Yic,Y^i)Ej(A)\tilde{e}_{ij} \leftarrow\sum_A P(A\mid Y_i^c,\hat{Y}_i)E_j(A)

  14. &nbsp;&nbsp;&nbsp;&nbsp;d~ij←∑AP(A∣Yic,Y^i)Dj(A)/e~ij\tilde{d}_{ij} \leftarrow\sum_A P(A\mid Y_i^c,\hat{Y}_i)D_j(A)/\tilde{e}_{ij}

  15. &nbsp;&nbsp;&nbsp;&nbsp;ℓ~ij←\tilde{\ell}_{ij} \leftarrow number of words deleted before y^j\hat{y}_j in Ai0A_i^0

  16. &nbsp;&nbsp;&nbsp;&nbsp;fij←FEATURES⁡(j)f_{ij} \leftarrow\operatorname{FEATURES}(j)

  17. &nbsp;&nbsp;end for

  18. &nbsp;&nbsp;w,wd←w,w_d \leftarrow ratios of negative to positive labels

  19. &nbsp;&nbsp;train pθp_\theta on {(fij,e~ij,d~ij,ℓ~ij)}\{(f_{ij},\tilde{e}_{ij},\tilde{d}_{ij},\tilde{\ell}_{ij})\} with Eq. (3)

  20. &nbsp;&nbsp;Platt-scale eje_j on dev, and djd_j on its error tokens

  21. end procedure

  1. procedure FEATURES(jj) &nbsp;&nbsp;▷ frames t≤tjt\leq t_j only

  2. &nbsp;&nbsp;q[⋅,j],r[⋅,j]←q[\cdot,j],r[\cdot,j]\leftarrow banded forward–backward

  3. &nbsp;&nbsp;r~[t,j]←r[t,j]/∑t′r[t′,j]\tilde{r}[t,j]\leftarrow r[t,j]/\sum_{t'}r[t',j]

  4. &nbsp;&nbsp;pool h(13),h(24),vh^{(13)},h^{(24)},v, joint scalars under q[⋅,j],r~[⋅,j]q[\cdot,j],\tilde{r}[\cdot,j]

  5. &nbsp;&nbsp;return the pooled features and g[j]g[j]

  6. end procedure

  1. procedure INFER(stream X^\hat{X}; cR,cI,Wc_R,c_I,W)

  2. &nbsp;&nbsp;Y^′←[];C←0;a←⊥;s0←0\hat{Y}'\leftarrow[]; C\leftarrow0; a\leftarrow\perp; s_0\leftarrow0

  3. &nbsp;&nbsp;for all tokens y^j\hat{y}_j emitted by Mθ\mathcal{M}_\theta, at frame tjt_j do

  4. &nbsp;&nbsp;&nbsp;&nbsp;(ej,dj,ℓj,sj)←pθ(FEATURES⁡(j),sj−1)(e_j,d_j,\ell_j,s_j)\leftarrow p_\theta(\operatorname{FEATURES}(j),s_{j-1})

  5. &nbsp;&nbsp;&nbsp;&nbsp;mj←ej[(1−dj)+djℓj]m_j\leftarrow e_j[(1-d_j)+d_j\ell_j] &nbsp;&nbsp;▷ Eq. (4)

  6. &nbsp;&nbsp;&nbsp;&nbsp;if y^j\hat{y}_j completes word kk then

  7. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;append word kk to Y^′\hat{Y}'

  8. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CUSUM(k,max⁡y^i∈word kmi)(k,\max_{\hat{y}_i\in\mathrm{word}\ k}m_i)

  9. &nbsp;&nbsp;&nbsp;&nbsp;end if

  10. &nbsp;&nbsp;end for

  11. &nbsp;&nbsp;CUSUM(N+1,mN+1)(N+1,m_{N+1}) &nbsp;&nbsp;▷ end of sequence

  12. &nbsp;&nbsp;if a≠⊥a\ne\perp and P≥cIP\geq c_I then RECOURSE(a..p)(a..p)

  13. &nbsp;&nbsp;return Y^′\hat{Y}'

  14. end procedure

  1. procedure CUSUM(k,mˉk)(k,\bar{m}_k)

  2. &nbsp;&nbsp;if C=0C=0 and mˉk>cR\bar{m}_k>c_R then

  3. &nbsp;&nbsp;&nbsp;&nbsp;a←k;P←0a\leftarrow k; P\leftarrow0 &nbsp;&nbsp;▷ span opens

  4. &nbsp;&nbsp;end if

  5. &nbsp;&nbsp;C←max⁡(0,C+mˉk−cR)C\leftarrow\max(0,C+\bar{m}_k-c_R) &nbsp;&nbsp;▷ Eq. (29)

  6. &nbsp;&nbsp;if a≠⊥a\ne\perp and C>PC>P then P←CP\leftarrow C; p←kp\leftarrow k

  7. &nbsp;&nbsp;if a≠⊥a\ne\perp and (C=0(C=0 or C≤P−cI)C\leq P-c_I) then

  8. &nbsp;&nbsp;&nbsp;&nbsp;if P≥cIP\geq c_I then RECOURSE(a..p)(a..p)

  9. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;a←⊥a\leftarrow\perp; C←0C\leftarrow0

  10. &nbsp;&nbsp;&nbsp;&nbsp;end if

  11. end procedure

  1. procedure RECOURSE(span SS)

  2. &nbsp;&nbsp;if ∣S∣>W|S|>W then &nbsp;&nbsp;▷ length cap

  3. &nbsp;&nbsp;&nbsp;&nbsp;S←S\leftarrow its WW consecutive words with max⁡∑mˉk\max\sum\bar{m}_k

  4. &nbsp;&nbsp;end if

  5. &nbsp;&nbsp;oL,oR←o_L,o_R\leftarrow anchor words on either side of SS in Y^′\hat{Y}'

  6. &nbsp;&nbsp;ask Q(oL,oR)Q(o_L,o_R); record the reply audio XrX^r

  7. &nbsp;&nbsp;Yr←Y^r\leftarrow decode of XrX^r by Mθ\mathcal{M}_\theta

  8. &nbsp;&nbsp;Z←LIM⁡(Y^′ with S marked,Q,Yr)Z\leftarrow\operatorname{LIM}(\hat{Y}'\text{ with }S\text{ marked},Q,Y^r)

  9. &nbsp;&nbsp;replace the words of SS in Y^′\hat{Y}' by ZZ

  10. end procedure

The corpora are cut into utterances in two ways: AMI has a time stamp for every word, so its long turns can be split between two words, whereas the other three corpora only time each segment, so their long turns cannot be split and are left out of training. Across all of them, training excludes turns too short to carry reliable labels (3 s for the meeting corpora and 1.5 s for the dinner parties), while development and evaluation keep every turn. Table 5 gives the resulting sizes.

AMI

Segments. Speaker turns are taken from the only_words references of the AMI diarization setup of Landini et al. [12], which are derived from the AMI manual annotations (version 1.6.2): every annotated word is speech, consecutive words of the same speaker are merged into one turn, and turns are broken wherever the speaker pauses. The setup’s meeting lists follow the full-corpus-ASR partition (136 training, 18 development and 16 evaluation meetings). Two meetings in the training set (IS1003b, IS1007d) do not have far-field recording in the corpus and are excluded. Each speaker’s global identifier is mapped to their headset channel using the corpus metadata (meetings.xml). Length criteria. Turns longer than 15 s are split into pieces of at most 15 s, cutting only between two consecutive words of the speaker, at the mid-point of the pause between them. For each piece, the cut is placed at the widest pause whose mid-point lies between 7.5 s and 15 s from the start of the piece, falling back to pauses closer to the start, and avoiding cuts that would leave a short remainder; if no word boundary is available, the piece is cut at 15 s. In the training set, turns and pieces shorter than 3 s are discarded: 6.37% of turns are split, and 34.78% of training utterances come from split turns. The 3 s threshold removes 66% of turns but only 16% of the speech, and merging consecutive turns of the same speaker across pauses of up to 1 s recovers less than 1% of the speech, so we do not merge turns. The development and evaluation sets keep turns of every length; in them 4.29% and 6.67% of turns are split, giving 9.33% and 13.96% of their utterances.

Reference transcripts. The reference of each utterance consists of the words of the speaker’s manual transcript whose start time falls inside it. Words without a time stamp are included when the nearest time-stamped words on both sides are included. Non-word annotations (vocal sounds, gaps and disfluency markers) and tokens consisting only of punctuation are excluded.

Audio. The far-field and headset recordings are cut at the same boundaries, down-mixed to mono and resampled to 16 kHz.

NOTSOFAR-1, CHiME-6 and DiPCo

Sources. The three corpora are converted with the data-generation tools of the CHiME-8 distant ASR challenge (Cornell et al., [5]), which provide segment-level transcripts with a start time, an end time, a speaker and the words of each segment. For NOTSOFAR-1 we use the training (240825.1_train), development (240825.1_dev1) and evaluation (240629.1_eval_small_with_GT, 80 meetings) releases. Each NOTSOFAR-1 meeting appears once per recording device; we keep the sessions of a single device (device code 107), present in every meeting. The far-field input is the first channel of the first array (U01), and the clean recording is the speaker’s close-talking channel.

Segments. Consecutive segments of the same speaker are joined into one turn when the pause between them is at most 0.5 s, no other speaker’s segment overlaps the pause, no segment removed under the criteria below lies inside it, and the joined turn is at most 15 s long. The transcripts have no word-level time stamps, so a turn cannot be split at a word boundary as in AMI; in the training set, turns longer than 15 s are therefore discarded, as are turns shorter than 3 s (NOTSOFAR-1) or 1.5 s (CHiME-6, DiPCo). The development and evaluation sets keep every turn, with turns outside these limits kept whole.

Reference transcripts. Annotation tags (laughter, noise, and similar) are removed from the reference. Segments whose reference marks words as inaudible, unintelligible or redacted are excluded from training, since the recording contains words that their reference lacks; they are kept, with the tag removed, in the development and evaluation sets, where they make up a substantial share of DiPCo. Segments with no words after tag removal are excluded from training; in development and evaluation they are kept, and enter the evaluation whenever their close-talking decode is not empty.

Cross-talk. The close-talking microphones of these corpora, the binaural microphones of CHiME-6 in particular, also pick up nearby speakers. Words of another speaker that appear in the close-talking decode are then counted as words the far-field decode deleted, which adds noise to the labels of overlapping turns.

Speech Synthesis

The synthetic clean rendering of a training utterance is its reference transcript, read by Kokoro-82M (hexgrad, 2025). Punctuation-only tokens and the hyphens of partial words are removed, and fillers are kept. Each speaker has a fixed voice, drawn from five female and five male voices. In AMI, the voice matches the speaker’s gender, which the speaker identifier encodes; NOTSOFAR-1, CHiME-6 and DiPCo do not annotate gender, and their speakers get a voice from the pooled list. The audio is resampled to 16 kHz, padded with 0.25 s of silence on each side, and mixed with an excerpt of the free-sound noise of MUSAN (Snyder et al., [27]) at an SNR drawn uniformly from 20 to 30 dB. The synthesized replies of Appendix § 11 are rendered in the same way from the reference words of the span. Every rendering is seeded by the utterance, so it is reproducible.

Recourse Details

Span boundaries. Each word kk is scored by mˉk\bar{m}_k, the highest mjm_j among its tokens; tokens without a score (punctuation) are ignored. Spans are found by the CUSUM of § 3.3.1 (Algorithm 1): a span starts at the word where CC rises above zero and ends at the word where CC peaked, and the interruption is raised at the first word at which CC has fallen cIc_I below that peak. Spans are made of whole words, so a request never quotes part of a word. A span longer than WW words is cut to the WW consecutive words with the largest total score (W=6W = 6 unless stated otherwise). The two words before the span form the left anchor and the two words after it the right anchor.

Start and end of the utterance. Near either edge of the utterance, a span may have fewer than two words on that side, so its anchor is cut short: the left anchor never extends before the first word and the right anchor never past the last. A span that reaches an edge has no anchor on that side and uses the corresponding one-sided template below. At the end of the utterance, the end-of-sequence position y^N+1\hat{y}_{N+1}, which carries the deletions after the last word, is scored as one more word, so trailing deletions can also open or extend a span. A span still open at that point does not wait for CC to fall: it triggers an interruption if its peak gain PP is at least cIc_I and is dropped otherwise. We take the end of the utterance from the reference segmentation; a deployed system would need a turn-end detector instead.

Request templates. The template depends on which anchors exist, with LL the (up to) two words before the span, RR the (up to) two words after it, and SS the words of the span in the noisy decode:

  • both anchors: “I didn’t catch what you said between ‘L’ and ‘R’. Could you repeat it?”

  • left anchor only: “What did you say after ‘L’?”

  • right anchor only: “What did you say before ‘R’?”

  • no anchor, as when the span covers a whole short utterance: “Did you say ‘S’?”

Replies. The speaker’s reply is simulated in two ways: the speaker’s close-talking recording of the span, cut at its word times and widened by 400 ms on each side, or a synthesized rendering of its reference words (Appendix § 10.§ 4). Either reply is decoded on its own by the same recognizer, starting from an empty predictor state; an empty reference gives an empty reply.

Correction. Each span is corrected by Gemini 3.8 Flash ([7]) at temperature 0, as described in § 3.3.2. The prompt, and how the corrector’s output replaces the words of the span, are given in Appendix § 12.§ 1.

Prompts

Placeholders are shown in angle brackets. All calls use temperature 0.

Corrector

The corrector is called once per span, in the order in which the interruptions are raised. Its user message contains the transcript decoded up to the interruption, with earlier spans already replaced by their corrections and the current span marked [? . . . ?], followed by the request and the reply. Only the text inside <fix></fix> is used, and it replaces the words of the span; every other word of the transcript is left as decoded. When the reply is empty, no call is made and the span’s words are removed.

System

You correct a live speech-recognition transcript of a meeting. The recognizer flagged a stretch it probably got wrong; it is marked [? like this ?]. The system asked the speaker to repeat that stretch, and the speaker’s answer is given. The answer was also transcribed automatically, so it can contain recognition errors, and the speaker may have repeated a few words from just before or after the stretch.

Write the corrected text for the marked stretch only:

  • base it on the answer where the answer is plausible in context;

  • leave out words of the answer that belong to the transcript outside the marks;

  • keep the transcript’s style (case, punctuation);

  • if the marked words were not actually spoken, return nothing.

Reply with the corrected stretch inside <fix></fix> and nothing else.

User

Transcript so far:

⟨words before the span⟩ [?⟨span words⟩?] ⟨words after the span, up to the interruption⟩

System asked: ⟨request (Appendix § 11)⟩

Speaker answered: “⟨transcribed reply⟩”

N-best Correction

The NN-best baseline uses the instruction template of HyPoradise (Chen et al., [4]) in a single user turn, preceded by ten demonstrations in the same format. The best hypothesis is the greedy transcript, which every other method corrects; the other hypotheses are the four highest-scoring beam hypotheses that remain distinct from it and from each other after text normalization. The demonstrations are the longest development-set utterances of at most 40 words, after removing utterances whose greedy hypothesis is far from the reference (crosstalk in the reference). The answer is read after ### Response:; an empty or overly long answer is replaced by the greedy transcript. The NN-best oracle picks, for each utterance, the hypothesis of the same list closest to the reference.

System

You correct speech recognition transcripts. Follow the format of the examples exactly and reply with the corrected transcription only.

User

Below is a best-hypotheses that is transcribed from an automatic speech recognition system. Write a response to predict the true transcription using the tokens from other-hypotheses.

Example 1:

Best-hypothesis:

⟨greedy transcript⟩

Other-hypothesis:

⟨hypothesis 2⟩

. . .

⟨hypothesis 5⟩

Response:

⟨reference⟩

. . . (Examples 2–10)

Now the utterance to correct:

Best-hypothesis:

⟨greedy transcript⟩

Other-hypothesis:

⟨hypothesis 2⟩

. . .

⟨hypothesis 5⟩

Response:

Implementation of the Learned Confidence Estimators

All four learned estimators of § 4.2 read the same frozen transducer and are trained on the same corpus, partition and labels as the CF-Gain Scorer: the hard labels EjE_j from the close-talking decode, with early stopping on development-set average precision. Every subword token is scored, and a word error marks all of its tokens. Each estimator is made causal: token jj, emitted at frame tjt_j, is scored only from the tokens up to jj and the encoder frames up to tjt_j, as the CF-Gain Scorer is.

Qiu et al. [20]. Each token’s input is its embedding and position, the decoder activation, its log posterior and the top-KK log posteriors, read at its greedy emission point.

  • Self-attention is masked to tokens up to jj. Cross-attention covers only the 16 encoder frames ending at tjt_j, instead of the whole utterance.

  • The deletion head predicts our deletion anchor, the token after the gap, as an auxiliary task. The utterance-confidence head is an auxiliary task during training only. The token score is the output of the main head.

  • We omit the multi-hypothesis input, which needs complete beams.

Wang et al. [30].

  • The three frames around the emission, tj−1,tj,tj+1t_j - 1, t_j, t_j + 1, are replaced by the three frames ending at it, tj−2,tj−1,tjt_j - 2, t_j - 1, t_j, since tj+1t_j + 1 may fall in the next encoder chunk.

  • Self-attention covers the past 30 tokens only.

TruCLeS (Ravi et al., [22]).

  • The input is the same as for Wang et al. [30], and the LSTMs are unidirectional instead of bidirectional.

  • The confidence target is computed with the close-talking decode as the reference. The probability of the reference token is read under the noisy decode’s prefix, since the streaming recognizer never sees the true one. The error score is one minus the predicted confidence.

Peng et al. [19]. We reimplement the detectors from the description in the paper.

  • Detectors. We keep the perception-error detector, which scores each token from the input to the recognizer’s joint network, and the deletion detector, which scores encoder frames. We omit the comprehension-error detector: it is trained on errors against the reference transcript, so it would learn errors that a re-utterance does not correct. We also omit the

distortion-type detector, whose labels require synthetic distortion conditions and whose output is used only by their own dialogue policy.

  • Deletion targets. The paper does not specify where on the frame axis a deletion target is placed. We mark the frames the noisy decode spent waiting to emit the deletion’s anchor token, those whose dwell posterior is at least 1% of its maximum (the final row of the lattice for a deletion after the last token). Deletion scores are pooled to a token over the blank frames that precede it, matching our anchoring convention.

  • Combination. The scores of the two detectors are each mapped to percentiles fitted on the development set and combined by taking the maximum.

Pair-HMM Channel

The soft labels of §3.2.2 come from a pair hidden Markov model (Durbin et al., [6]) that reads the noisy decode Y^\hat{Y} token by token and writes the clean decode YcY^{c} (Figure 6). Figure 5 shows the case it is built for: a deletion that two alignments explain equally well.

Figure 5. An ambiguous deletion. Two alignments explain the noisy decode equally well: the second that was deleted and is anchors the deletion (solid), or the first was deleted and the surviving that anchors it (dashed). The hard alignment labels only one token and treats the other as clean; the soft label splits the mass between them, so neither is penalized, and a span covering both carries the same label mass either way.

Pair-HMM channel diagram at noisy token $\hat{y}_j$, with states C, D, I, S, and A and transition probabilities $1-\pi$, $\pi$, $\gamma$, $1-\gamma$, $\rho$, $1-\rho$, $(1-\eta)\rho$, $(1-\eta)(1-\rho)$, and $\eta$

Figure 6. The pair-HMM channel at noisy token y^j\hat{y}_j. A correct token (C) is written unchanged. An erroneous one is preceded by a run of deleted tokens (D, possibly empty) and is then an anchor (A, written unchanged; only after a deletion), an insertion (I, nothing written) or a substitution (S, a different token written). C has Ej=0E_j = 0; A, I and S have Ej=1E_j = 1.

Each noisy token carries a latent label Ej∈{0,1}E_j \in\{0,1\}, and four parameters govern the channel: the probability that a token is in error (π\pi), that a run of deleted tokens continues (γ\gamma), that an erroneous token is the anchor of a preceding deletion (η\eta), and that a non-anchor error is an insertion rather than a substitution (ρ\rho). They are fit by expectation maximization on the training pairs, initialized from the counts of the hard alignment, and frozen

after ten iterations (Algorithm 1). The forward–backward pass of the fitted model gives the posterior e~j=P(Ej=1∣Yc,Y^)\tilde{e}_j = P(E_j = 1 \mid Y^c,\hat{Y}) for every noisy token, including the end-of-sequence position, and in the same way the posterior that y^j\hat{y}_j is preceded by deleted tokens (the D state of Figure 6); dividing it by e~j\tilde{e}_j gives d~j\tilde{d}_j. The number of deleted words ℓ~j\tilde{\ell}_j is read from the hard alignment. Wherever the two decodes determine the alignment, e~j\tilde{e}_j equals the hard label EjE_j. The channel runs on normalized words, and its labels are mapped to subword tokens as described in §N.

Recourse Examples

Substitution repaired

Decode Yeah, [we get behind schedule yeah] pretty quick like you say do we want to set a date for interim prototype?

Request I didn’t catch what you said between ‘Yeah’ and ‘pretty quick’. Could you repeat it?

Reply It’ll get behind schedule, yeah.

Corrected Yeah, it’ll get behind schedule yeah pretty quick like you say do we want to set a date for interim prototype?

Reference Yeah it’ll get behind schedule yeah pretty quick Like you’re saying yeah Well actually do we wanna set a date for interim prototype

Substitution repaired

Decode [We have] the number buttons, I guess.

Request What did you say before ‘the number’?

Reply Leave out

Corrected Leave out the number buttons, I guess.

Reference leave out the number buttons I guess Deleted words restored

Decode Okay, is that how it works in the other ones in like the other windows like various like um [yeah topic] or

Request I didn’t catch what you said between ‘like um’ and ‘or’. Could you repeat it?

Reply If we’re in the topic

Corrected Okay, is that how it works in the other ones in like the other windows like various like um if we’re in the topic or

Reference Okay Is that how it works in the other ones In like the other windows like the various like um if we’re in the topic Or

Deleted words restored

Decode [This is not really] like for the group you were making this all really require a lot of marketing.

Request What did you say before ‘like for’? Could you repeat it?

Reply It’s pretty different, but this is not really

Corrected It’s pretty different, but this is not really like for the group you were making are remote, this will really require a lot of marketing.

Reference It’s pretty different But this is not really like for the group we were making our remote This will really require a lot of marketing

Inserted words removed

Decode Um, [actually this] first thing I just sort of did a really sort of brief um yeah version of what we have to do

Request I didn’t catch what you said between ‘Um’ and ‘first thing’. Could you repeat it?

Reply Actually

Corrected Um, actually first thing I just sort of did a really sort of brief um yeah version of what we have to do

Reference Um actually first thing I just sorta did a really sorta brief um yeah version of what we have to do Um actually I can just take that if you want

Partly repaired

Decode Nice and pink fluorescent banana color or whatever you know [make a banana chip on you] for example, you know, but the question is really who is our target group?

Request I didn’t catch what you said between ‘you know’ and ‘for example’. Could you repeat it?

Reply Yeah.

Corrected Nice and pink fluorescent banana color or whatever you know for example, you know, but the question is really who is our target group?

Reference nice and pink fluorescent banana colour or whatever you know yeah for example you know But the question is really who is our target group

Partly repaired

Decode [Is] there anything you would like to to ask me about the findings before I?

Request What did you say before ‘there anything’?

Reply I know. Okay, well I hope that’s a clear to the team is

Corrected I know. Okay, well I hope that’s clear to the team. Is there anything you would like to to ask me about the findings before I?

Reference I know Okay Well I hope that’s uh clear to the team Is there anything you would like to ask me about the findings before I

False alarm, transcript unchanged

Decode Maybe we should just have two, but yeah, if [he finds them then that’s cool] if he finds the people um.

Request I didn’t catch what you said between ‘yeah, if’ and ‘if he’. Could you repeat it?

Reply He finds them then that’s cool.

Corrected Maybe we should just have two, but yeah, if he finds them then that’s cool if he finds the people um.

Reference yeah Three or four Oh God Um but yeah if he finds them then that’s cool If he finds the people M um

False alarm, transcript unchanged

Decode [All these changes are not really] okay with me, but since

Request What did you say before ‘okay with’?

Reply All these changes are not really.

Corrected All these changes are not really okay with me, but since

Reference all these changes are not really okay with me But since

False alarm, error introduced

Decode Um like we have the right click menu [menu] right clicking but instead of like have a default double click so there’s so it’ll be a choice

Request I didn’t catch what you said between ‘click menu’ and ‘right clicking’. Could you repeat it?

Reply Menu.

Corrected Um like we have the right click menu right clicking but instead of like have a default double click so there’s so it’ll be a choice from the menu

Reference Um like we have the right-click menu You’ll have the menu right-click it but instead of like have a default double-click so if there’s a de so it’ll be a choice from the menu but yeah exactly Yeah

False alarm, error introduced

Decode [There something like that,] but that’s that sounds a lot cheaper to me.

Request What did you say before ‘but that’s’?

Reply There’s something like that.

Corrected There’s something like that, but that’s that sounds a lot cheaper to me.

Reference there something like that but that’s that sounds a lot cheaper to me

Failure: the reply was misrecognized

Decode Yeah [text yeah yeah yeah cause that’s] the like declares a variable isn’t and then yeah because

Request I didn’t catch what you said between ‘Yeah’ and ‘the like’. Could you repeat it?

Reply Yeah yeah yeah cause that that’s the

Corrected Yeah yeah yeah cause that the like declares a variable isn’t and then yeah because

Reference Yeah Yeah yeah ’cause that’s uh the declares a variable doesn’t it and then they keep Okay

Failure: the reply was misrecognized

Decode [The whistle] so we can financially voice recognition is not feasible we could go for a whistle.

Request What did you say before ‘so we’?

Reply For hell yeah, the whistle.

Corrected The whistle so we can financially voice recognition is not feasible we could go for a whistle.

Reference For l yeah the whistle So we can if financially voice recognition is not feasible we could go for a whistle

Failure: correct reply, wrong correction

Decode Right [attributes you’ve got lots of nom] something and then night something and then end something

Request I didn’t catch what you said between ‘Right’ and ‘something and’. Could you repeat it?

Reply Yeah, you’ve got lots of Nom

Corrected Right you’ve got lots of nom something and then night something and then end something

Reference Oh yeah You’ve got lots of nom something and then NITE something and then N_ something

Failure: correct reply, wrong correction

Decode [Very so we’ve] got twenty cents for our docking station.

Request What did you say before ‘got twenty’?

Reply Very subtitles buttons yeah seven yeah so we’ve

Corrected So we’ve got twenty cents for our docking station.

Reference Very Subtitles buttons Yeah Seven Yeah So we’ve got twenty cents for our docking station huh

Out-of-Domain Generalization with Matched Training Size

The source corpora of Table 2 have far more training data than their targets (62.9 vs. 6.2 hours and 28.5 vs. 0.7 hours; Table 5). To separate the effect of the corpus from that of the training size, we train a CF-Gain Scorer on a subset of the source’s training set with as many hours as the target’s training set. The subset is built from whole sessions: the sessions are visited in a fixed random order and their utterances are added until the target duration is reached, so the subset holds a few complete sessions and part of one, like a smaller corpus rather than a random sample of turns. This gives 11 complete AMI meetings and part of a twelfth, out of 134 training meetings, and part of one of the 16 CHiME-6 training sessions. The scorer is otherwise trained as in § 4.1, with model selection and calibration on the source’s development set, and applied unchanged to the target.

Table 6 shows the results. With matched training size, the transferred scorer falls below the in-domain scorer on both pairs: on NOTSOFAR-1 by 4.0 points of AP and 11.1 points of recall at 70% precision, and on DiPCo by 2.3 points of AP and 6.7 points of recall. The gains of Table 2 therefore

TestTrained onHoursAPRecall at prec. 50Recall at prec. 70
NOTSOFAR-1Entropy—39.228.66.3
NOTSOFAR-1NOTSOFAR-16.259.868.739.4
NOTSOFAR-1AMI62.963.271.739.7
NOTSOFAR-1AMI, subset6.255.860.928.3
DiPCoEntropy—41.329.011.0
DiPCoDiPCo0.744.136.610.0
DiPCoCHiME-628.554.858.024.4
DiPCoCHiME-6, subset0.741.831.43.3

Table 6. Out-of-domain generalization with matched training size. A CF-Gain Scorer trained on the other corpus of the same genre, on all of its training set or on a subset with as many hours as the target’s training set, is applied unchanged to the target and compared with a scorer trained on the target and with the training-free entropy baseline. All numbers except hours are in percent. Best result per block in bold.

come from the larger training set of the source corpus: at equal size, in-domain data is worth more, but a larger corpus of the same genre can replace it. The CHiME-6 subset comes from a single session, so its result likely depends on which session is drawn.

Speech Enhancement Baseline

The far-field recording is passed through DeepFilterNet3 ([24]), a causal noise-suppression model with about 40 ms of algorithmic latency that suits a streaming recognizer, and the enhanced audio is decoded by the same recognizer. The model runs at 48 kHz; the 16 kHz input is resampled up and the output resampled back. We cap its attenuation at a limit fitted on the development set. Enhancement increases the WER on both evaluation sets, from 30.0% to 32.4% on AMI (relative WER reduction −8.0%) and from 64.1% to 67.0% on CHiME-6 (−4.5%), so it is not shown in Figure 4.

Additional Recourse Results

Figure 7 repeats the recourse comparison of § 5.2 with a higher weight on re-entered words (β=0.10\beta= 0.10, spans of at most six words) and with longer spans (at most eight words, β=0.05\beta= 0.05). Operating points are selected on the development set as in §M.

Figure 7. Recourse on the AMI and CHiME-6 evaluation sets with a higher weight on re-uttered words (a, b) and with longer spans (c, d): relative WER reduction against the re-utterance burden, at the operating points selected on the development set, with synthesized or close-talking replies. Horizontal lines: the zero-interruption NN-best correction of HyPoradise and the NN-best oracle.

Tuning Cost Parameters

We find operating points on the development set by minimizing the burden for different budgets of recall, the share of erroneous words, counting each deleted word, that fall inside a span or in the gap before one of its words. Recall approximates the ceiling on the error reduction our system can achieve. It is measured against the close-talking decode, so no gold transcript is used: a substituted or inserted word counts when it lies inside a span, and the deleted words of a gap count when the gap lies inside a span or at its boundary. Utterances with an empty noisy decode stay in the denominator.

Operating points are chosen separately for each scorer, span cap WW and weight β\beta. We sweep cR∈{0.3,0.4,…,0.8}c_R \in\{0.3, 0.4, \ldots, 0.8\} and 64 values of cIc_I spaced geometrically from 0.02 to 20. The recall budgets are six values spaced evenly from 5% to 90% of the lower of the two scorers’ highest development recall, so that both scorers can reach every budget. For each budget, the pair (cR,cI)(c_R, c_I) with the lowest development burden among those that reach it is applied unchanged to the evaluation set. The zero-interruption baselines are reported as single points.

Training Details

Supervision. The close-talking recording is transcribed by the same ASR system as the far-field audio, and this transcript serves as the clean decode for supervision. Both noisy and clean decodes are normalized to words (lower-cased, with punctuation removed) before a pair HMM aligns them. Word-level substitution and insertion labels are copied to every subword token of the noisy word, and deletion labels, together with d~j\tilde{d}_j and ℓ~j\tilde{\ell}_j, only to its first token. The pair HMM is initialized from the hard-alignment counts, fit with ten EM iterations on the training utterances, and frozen before the CF-Gain Scorer is trained.

Features. Encoder states are pooled from the 13th and the final (24th) encoder blocks under the emission and dwell posteriors, giving four 1024dimensional vectors per token, and are concatenated with the predictor state and the scalar features of § 3.2.1. Voice activity probabilities come from Silero VAD [25].

Training. The CF-Gain Scorer is a single-layer unidirectional LSTM with 128 hidden units, followed by three heads, each a feed-forward network with one hidden layer of 64 units. It is trained with AdamW [15] at a learning rate of 10−310^{-3} and weight decay 10−410^{-4}, on batches of 64 utterances, for up to 60 epochs with early stopping on development-set AP of eje_j (patience 15). We then apply Platt scaling to eje_j on the development set, and to djd_j on its error tokens. A separate scorer is trained on each corpus.

Compute. All experiments run on a single NVIDIA L4 GPU. On AMI, the largest corpus, extracting the lattice features of the training set takes about 4 hours and training the CF-Gain Scorer about 30 minutes; all experiments in this paper, including the baselines, ablations, speech synthesis and recourse evaluation, took about 40 GPU hours in total. The LLM corrector and the NN-best baseline are queried through the Gemini API. All results are from a single training run per setting.

References

  1. [1]AMI Consortium. 2005. AMI corpus official partitions. https://groups.inf.ed.ac.uk/ami/corpus/datasets.shtml. Accessed 2026-10-01.
  2. [2]Dan Bohus and Alexander I. Rudnicky. 2005. Sorry, I didn’t catch that! — an investigation of non-understanding errors and recovery strategies. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, pages 128–143, Lisbon, Portugal. Special Interest Group on Discourse and Dialogue (SIGdial).
  3. [3]Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Welligner. 2006. The AMI meeting corpus: A pre-announcement. In Machine Learning for Multimodal Interaction (MLMI 2005), volume 3869 of Lecture Notes in Computer Science, pages 28–39, Berlin, Heidelberg. Springer.DOI
  4. [4]Chen Chen, Yuchen Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Pin-Yu Chen, and Eng Siong Chng. 2023. HyPoradise: An open baseline for generative speech recognition with large language models. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:2309.15701.arxiv.org/abs/2309.15701
  5. [5]Samuele Cornell, Taejin Park, He Huang, Christoph Boeddeker, Xuankai Chang, Matthew Maciejewski, Matthew Wiesner, Paola Garcia, and Shinji Watanabe. 2024. The CHiME-8 DASR challenge for generalizable and array agnostic distant automatic speech recognition and diarization. In Proc. 8th International Workshop on Speech Processing in Everyday Environments (CHiME 2024), pages 1–6.arxiv.org/abs/2407.16447
  6. [6]Richard Durbin, Sean R. Eddy, Anders Krogh, and Graeme Mitchison. 1998. Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge University Press.
  7. [7]Google DeepMind. 2026. Gemini 3.8 flash model card. https://deepmind.google/models/model-cards/gemini-3-8-flash/. Accessed October 2026.
  8. [8]Xinlu He, Yiwen Guan, Badrivishal Paurana, Pitipat Kongsomjit, Zilin Dai, and Jacob Whitehill. 2025. Interactive in-meeting speaker correction with human feedback. arXiv preprint arXiv:2509.18377.arxiv.org/abs/2509.18377
  9. [10]Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, and Eng Siong Chng. 2024. Large language models are efficient learners of noise-robust speech recognition. In International Conference on Learning Representations (ICLR). ArXiv:2401.10446.arxiv.org/abs/2401.10446
  10. [11]Hui Jiang. 2005. Confidence measures for speech recognition: A survey. Speech Communication, 45(4):455–470.DOI
  11. [12]Federico Landini, Ján Profant, Mireia Diez, and Lukáš Burget. 2022. Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: Theory, implementation and analysis on standard tasks. Computer Speech & Language, 71:101254.arxiv.org/abs/2012.14952
  12. [13]Aleksandr Laptev and Boris Ginsburg. 2022. Entropy-based methods of word-level confidence estimation for end-to-end automatic speech recognition. In IEEE Spoken Language Technology Workshop (SLT).DOI
  13. [14]Yichong Leng, Xu Tan, Linchen Zhu, Jin Xu, Renqian Luo, Linquan Liu, Tao Qin, Xiang-Yang Li, Ed Lin, and Tie-Yan Liu. 2021. FastCorrect: Fast error correction with edit alignment for automatic speech recognition. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2105.03842
  14. [15]Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR).
  15. [16]Rao Ma, Mengjie Qian, Potsawee Manakul, Mark Gales, and Kate Knill. 2023. Can generative large language models perform ASR error correction? arXiv preprint arXiv:2307.04172.arxiv.org/abs/2307.04172
  16. [17]NVIDIA. 2026. Nemotron speech streaming English 0.6B. https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b. Model card.
  17. [18]E. S. Page. 1954. Continuous inspection schemes. Biometrika, 41(1/2):100–115.DOI
  18. [19]Yizhou Peng, Ziyang Ma, Changsong Liu, Yi-Wen Chao, Xie Chen, and Eng Siong Chng. 2026. Proactive supportive clarification for spoken dialogue systems. Preprint, arXiv:2605.25404. Accepted to Findings of EMNLP 2026.
  19. [20]David Qiu, Yanzhang He, Qiujia Li, Yu Zhang, Liangliang Cao, and Ian McGraw. 2021. Multi-task learning for end-to-end ASR word and utterance confidence with deletion prediction. In Interspeech 2021, pages 4074–4078.arxiv.org/abs/2104.12870
  20. [21]Anton Ragni, Qiujia Li, Mark J. F. Gales, and Yu Wang. 2018. Confidence estimation and deletion prediction using bidirectional recurrent neural networks. In IEEE Spoken Language Technology Workshop (SLT).
  21. [22]Nagarathna Ravi, Thishyan Raj T, Ravi Teja Chaganti, and Vipul Arora. 2025. ASR confidence estimation using true class lexical similarity score. In Interspeech 2025, pages 3658–3662.
  22. [23]Dima Rekesh, Nithin Rao Koluguri, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, He Huang, Oleksii Hrinchuk, Krishna Puvvada, Ankur Kumar, Jagadeesh Balam, and Boris Ginsburg. 2023. Fast conformer with linearly scalable attention for efficient speech recognition. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).arxiv.org/abs/2305.05084
  23. [24]Hendrik Schröter, Alberto N. Escalante-B., Tobias Rosenkranz, and Andreas Maier. 2023. DeepFilterNet: Perceptually motivated real-time speech enhancement. In Interspeech 2023, pages 2008–2009.arxiv.org/abs/2305.08227
  24. [25]Silero Team. 2021. Silero VAD: Pre-trained enterprise-grade voice activity detector. https://github.com/snakers4/silero-vad.
  25. [26]Gabriel Skantze. 2007. Error Handling in Spoken Dialogue Systems: Managing Uncertainty, Grounding and Miscommunication. Ph.D. thesis, KTH Royal Institute of Technology, Stockholm, Sweden.
  26. [27]David Snyder, G uoguo Chen, and Daniel Povey. 2015. MUSAN: A music, speech, and noise corpus. ArXiv:1510.08484.arxiv.org/abs/1510.08484
  27. [28]Maarten Van Segbroeck, Ahmed Zaid, Ksenia Kutse nko, Cirenia Huerta, Tinh Nguyen, Xuewen Luo, Björn Hoffmeister, Jan Trmal, Maurizio Omologo, and Roland Maas. 2020. DiPCo – dinner party corpus. In Interspeech 2020, pages 434–436.arxiv.org/abs/1909.13447
  28. [29]Alon Vinnikov, Amir Ivry, Aviv Hurvitz, Igor Abramovski, Sharon Koubi, Ilya Gurvich, Shai Peer, Xiong Xiao, Benjamin Martinez Elizalde, Naoyuki Kanda, Xiaofei Wang, Shalev Shaer, Stav Yagev, Yossi Asher, Sunit Sivasankaran, Yifan Gong, Min Tang, Huaming Wang, and Eyal Krupka. 2024. NOTSOFAR-1 challenge: New datasets, baseline, and tasks for distant meeting transcription. In Interspeech 2024, pages 5003–5007.arxiv.org/abs/2401.08887
  29. [30]Mingqiu Wang, Hagen Soltau, Laurent El Shafey, and Izhak Shafran. 2021. Word-level confidence estimation for RNN transducers. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).arxiv.org/abs/2110.15222
  30. [31]Shinji Watanabe, Michael Mandel, Jon Barker, Emmanuel Vincent, Ashish Arora, Xuankai Chang, Sanjeev Khudanpur, Vimal Manohar, Daniel Povey, Desh Raj, David Snyder, Aswin Shanmugam Subramanian, Jan Trmal, Bar Ben Yair, Christoph Boeddeker, Zhaoheng Ni, Yusuke Fujita, Shota Horiguchi, Naoyuki Kanda, and 2 others. 2020. CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings. In Proc. 6th International Workshop on Speech Processing in Everyday Environments (CHiME 2020), pages 1–7.arxiv.org/abs/2004.09249
  31. [32]Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, and Andreas Stolcke. 2023. Generative speech recognition error correction with large language models and task-activating prompting. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).DOI

Paper details

vlnc:W_9dxesamr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents