betaFeedback

WorksEngineering

Pandemonium: Longform Target-Speaker Speech Recognition with Cross-Speaker Interactions

Abstract

Diarization-conditioned target-speaker ASR (TS-ASR), in particular Diarization-conditioned Whisper (DiCoW), is an accurate and data-efficient approach to multi-talker transcription. It has, however, three structural limitations: the model transcribes each speaker independently, operates on short (30 s) windows, and optimizes its diarization and TS-ASR components separately. We address the first two. We integrate the diarization conditioning into a longform Parakeet backbone (DiCoP) and add cross-speaker attention, which lets the model attend across the speaker dimension so that speakers are no longer decoded in isolation, and an STNO mask refiner that re-estimates the diarization-derived conditioning. As conversational data with genuine multi-speaker interaction is scarce, we scale training with a pseudo-labeling pipeline that mines 5.4k hours of in-the-wild multi-speaker audio from YODAS2. Combining these with synthetic and real data yields state-of-the-art results across several meeting benchmarks. We open-source the resulting models, DiCoP and Pandemonium, and the pseudo-labeled data.

Index Terms— target-speaker ASR, multi-talker ASR, speaker diarization, longform ASR, cross-speaker modeling

Introduction

Transcribing natural conversations remains challenging because of speaker overlap, diverse acoustic conditions, colloquial language, an unknown and varying number of participants, and recordings that span minutes rather than seconds. Multi-talker ASR (MT-ASR) transcribes all speakers with a single model. One line of work, serialized output training (SOT), serializes all speakers into a single output stream; it was introduced with attention-based encoder-decoders [1] and has recently been adopted by LLM-based systems [2, 3]. Others emit several streams under a permutation-invariant objective [4] or build on source separation [5]. In contrast, target-speaker ASR (TS-ASR) conditions a single-speaker recognizer on one speaker at

a time and emits only that speaker’s transcript. It is run once per detected speaker to reconstruct the full speaker-attributed transcript [6, 7]. Across backbones and data budgets, TS-ASR has proven accurate and data-efficient [8]. Our prior work, Diarization-conditioned Whisper (DiCoW) [9], is representative of TS-ASR: it conditions the model on per-speaker, per-frame activities derived from a diarizer. It has transferred to E-Branchformer [10], Whisper [11], and a spoken LLM [12], so it is largely backbone-agnostic.

Despite these strengths, DiCoW has three structural limitations. (i) It models each speaker independently: although other speakers are present in the input audio, the model has no access to how they were encoded or what was decoded for them, discarding the cross-speaker cues that help resolve overlaps, turn-taking, and backchannels. (ii) It is trained and decoded on short, fixed 30 s windows inherited from the Whisper backbone, which caps the context available at both training and inference time. (iii) It runs a multi-stage pipeline in which diarization and recognition are trained separately. In this work we address the first two limitations.

Recent longform ASR backbones [13]—in particular the Parakeet family of FastConformer models [14, 15]—combine aggressive convolutional subsampling with local window attention and global tokens, processing whole meetings [16, 17, 18, 19] on a single GPU without chunking. Meanwhile, systems that decode all speakers jointly (SOT [1] and its LLM-based successors [2, 3]) show that joint access to all speakers improves transcripts, yet they generalize poorly to unseen speaker counts and heavy overlap.

We integrate the diarization conditioning of DiCoW into Parakeet (DiCoP), using its local attention and a CTC loss to train faster and decode whole conversations in one pass at a very low real-time factor (§ 2.2 and § 2.3), and design a new interaction module for TS-ASR: cross-speaker attention (CSA), which lets the model attend across the speaker dimension, complemented by a mask refiner that sharpens the diarization-derived conditioning (§ 2.4). Since such interaction is scarce in curated corpora, we mine multi-speaker recordings from YODAS2 [20] and pseudo-label them with DiCoW, obtaining 5.4k hours of conversational data (§ 3.1.1). Training on synthetic, pseudo-labeled, and real data gives our final system, Pandemonium (Figure 1).1

Figure 1. Overview of Pandemonium. A diarizer yields per-speaker masks that condition (FDDT) a longform Parakeet encoder processing the SS speakers as a batch. Cross-speaker attention attends across speakers and a mask refiner sharpens the ambiguous STNO conditioning (blue boxes); a shared CTC head decodes each.

Contributions. We release:2 (i) DiCoP, a longform diarization-conditioned Parakeet TS-ASR in TDT and faster CTC variants; (ii) 5.4k h of pseudo-labeled multi-speaker YODAS2 conversations (§ 3.1.1); (iii) Pandemonium, DiCoP with CSA and an STNO mask refiner, with state-of-the-art results (Table 1).

SystemNSFAMIICSIMx6SBCDiPCoAvg
Oracle diarization
DiCoW v3.3 [22]15.514.514.111.415.027.116.3
SE-DiCoW [22]15.314.414.311.715.226.216.2
DiCoP (no CSA)17.314.514.411.916.026.116.7
Pandemonium14.113.413.311.113.822.414.7
+ MOVER10.812.611.99.713.819.413.0
Predicted diarization (pyannoteAI Precision-3) [23, 24]
DiCoW v3.3 [22]23.017.318.611.519.330.020.0
SE-DiCoW [22]22.817.318.211.819.629.319.8
DiCoP (no CSA)23.216.918.012.119.428.919.8
Pandemonium22.416.818.111.718.627.419.2
+ MOVER19.014.515.710.218.624.817.1
SOTA22.8[5]17.6[22]20.9[25]14.5[26]–34.0[26]–

Table 1. Main results, cpWER [%] ↓. DiCoP: longform Parakeet backbone (loc/ TDT of Fig. 2); Pandemonium adds CSA and the STNO refiner, trained synthetic → pseudo → real. + MOVER: all distant microphones decoded separately, merged with MOVER [21] (SBC: stereo, unchanged). SOTA: best published single-microphone result (Sep 2026). Best single-microphone result per block in bold; abbreviations in Section 3.1.

Method

Diarization-conditioned target-speaker ASR

We build on diarization-conditioned TS-ASR [9], using by default the DiCoW v3.3 baseline of Self-Enrolled DiCoW (SE-DiCoW) [22], to which we refer the reader for details. The diarizer provides the probability d(s,t)∈[0,1]d(s,t) \in[0,1] that speaker s∈{1,…,S}s \in\{1,\ldots,S\} is active in frame tt. For a target speaker sks_k, these activities define the probabilities of four mutually exclusive frame-level events—silence SS (nobody speaks), target TT (only sks_k), non-target NN (others, but not sks_k), and overlap OO (sks_k together with others):

pSt=∏s(1−d(s,t)),pTt=d(sk,t)∏s≠sk(1−d(s,t)),p^t_S = \prod_{s}(1-d(s,t)), \qquad p^t_T = d(s_k,t)\prod_{s\ne s_k}(1-d(s,t)),
pNt=1−pSt−d(sk,t),pOt=d(sk,t)−pTt.p^t_N = 1-p^t_S-d(s_k,t), \qquad p^t_O = d(s_k,t)-p^t_T.

Stacked into Mt=ϕsk(dt)=[pSt pTt pNt pOt]⊤\mathbf{M}^t = \phi_{s_k}(\mathbf{d}_t) = [p^t_S\ p^t_T\ p^t_N\ p^t_O]^\top, with dt=[d(s,t)]s=1S\mathbf{d}_t = [d(s,t)]_{s=1}^{S}, they form the STNO (silence/target/non-target/overlap) mask, whose size is independent of the number of speakers; the mapping ϕsk\phi_{s_k} is reused in § 2.4. This mask modulates the encoder through a frame-level diarization-dependent transformation (FDDT): each frame representation ztl\mathbf{z}^l_t at encoder layer ll is mapped by a convex combination of four class-specific affine transforms,

ztl′=∑c∈{S,T,N,O}(Wclztl+bcl)pct(1)\mathbf{z}^{l\prime}_t = \sum_{c\in\{S,T,N,O\}} (\mathbf{W}^{l}_{c}\mathbf{z}^{l}_t+\mathbf{b}^{l}_{c})p^t_c \tag*{(1)}

so that a single recognizer, run once per hypothesized speaker, transcribes only speaker sks_k while remaining acoustically aware of the others. The transforms Wcl\mathbf{W}^{l}_{c} and bcl\mathbf{b}^{l}_{c} are trained with the backbone, and in our prior work [9, 22] this lightweight modulation proved more data-efficient and robust than enrollment- or embedding-based conditioning [6, 7]. By default, each speaker is decoded in a separate forward pass, so the passes never exchange information, and the 30 s windows discard long-range context.

Longform diarization-conditioned Parakeet (DiCoP)

In this work, we integrate the conditioning into the Parakeet Fast-Conformer backbone [14, 15], which natively decodes long audio in a single pass, and call the result DiCoP (Figure 1 without the blue boxes). The integration keeps the diarization conditioning of § 2.1 intact: exactly the same FDDT mechanism (Eq. (1)) modulates the encoder, and the only adaptation is subsampling of the STNO mask Mt\mathbf{M}^t to match Parakeet’s coarser frame rate. Two obstacles arise when the training/decoding chunk grows from 30 s to several minutes. First, global self-attention is quadratic in length and exhausts GPU memory. We use the local (windowed) attention Parakeet supports, bounding memory at the cost of a limited attention span. Second, multilingual Parakeet v3 is trained with the token-and-duration transducer (TDT) loss [27], a transducer variant whose head predicts, along with each token, how many frames to skip; this head remains the more accurate choice at 30 s, but computing the transducer loss is too expensive for long chunks (§ 2.3).

From TDT to CTC training

The TDT loss scores the full T×UT \times U lattice of encoder frames and target tokens, so memory grows quadratically with chunk duration. Moreover, the CUDA kernels used to train Parakeet3 run one thread per target position, so the threads-per-block limit hard-caps the target length at 1024 tokens. Since each thread also holds many registers, the per-block register budget is exhausted well below this cap, and the kernel launch fails with a CUDA error already for shorter targets. A multi-­numpath chunk exceeds these limits. CTC [28] has neither issue. Parakeet is also released in RNN-T and CTC variants, but these are English-only and less accurate than the TDT models, so we start from the multilingual TDT checkpoint, keep its head frozen, and train with a CTC loss instead. The TDT joiner combines an encoder frame ft\mathbf{f}_t with a predictor state gu\mathbf{g}_u as

[zt,utok;zt,udur]=Woσ(Weft+Wpgu+b),(2)[\mathbf{z}^{\mathrm{tok}}_{t,u};\mathbf{z}^{\mathrm{dur}}_{t,u}] = \mathbf{W}_o \sigma(\mathbf{W}_e\mathbf{f}_t+\mathbf{W}_p\mathbf{g}_u+\mathbf{b}), \tag*{(2)}

where zt,utok\mathbf{z}^{\mathrm{tok}}_{t,u} are the V+1V+1 token-and-blank logits and zt,udur\mathbf{z}^{\mathrm{dur}}_{t,u} the duration logits. Zeroing the predictor contribution, Wpgu≡0\mathbf{W}_{\mathrm{p}}\mathbf{g}_{u} \equiv0, makes the joiner frame-synchronous,

P(k∣t)=softmax⁡(Wotok σ(Weft+b))k,(3)P(k \mid t) = \operatorname{softmax}\left(\mathbf{W}^{\mathrm{tok}}_{o}\,\sigma(\mathbf{W}_{\mathrm{e}}\mathbf{f}_{t}+\mathbf{b})\right)_{k}, \tag*{(3)}

where kk indexes the VV tokens and the blank, which is exactly the per-frame posterior CTC needs, while the duration logits are ignored. No parameters are added and TDT decoding remains possible; after CTC finetuning, the posteriors become spiky and blank-dominated, so greedy CTC decoding is much faster and slightly less accurate. The price is a small accuracy degradation relative to TDT training on short chunks, outweighed by the gains from the interaction modules, which need longform training (§ 4). All subsequent models build on this CTC variant, whereas the DiCoP baseline we compare against remains the stronger TDT-trained model.

Cross-speaker attention (CSA)

Lifting the per-speaker independence assumption is the main target of this work. Since conditioning already runs one pass per speaker, the SS streams of a conversation form a natural batch that we let interact inside the encoder (blue boxes in Figure 1). DiCoW [9] did so through a multichannel-style co-attention [29] over time and channels, which in later shortform experiments did not improve over DiCoW v3.3 alone while incurring a substantial GPU-memory overhead. Since our streams share one encoder and one time axis, we attend only across the speaker axis at each frame instead.

Cross-speaker attention. Let zs,tl∈RD\mathbf{z}^{l}_{s,t} \in\mathbb{R}^{D} denote the FDDT-modulated state (Eq. (1)) of speaker ss at layer ll and frame tt, and let Ztl=[z1,tl,…,zS,tl]⊤∈RS×D\mathbf{Z}^{l}_{t} = [\mathbf{z}^{l}_{1,t}, \ldots, \mathbf{z}^{l}_{S,t}]^{\top} \in\mathbb{R}^{S \times D} stack the conversation’s SS speakers at that frame. At selected layers we run multi-head attention along the speaker axis,

Utl=MHA⁡(LN⁡(Ztl)),(4)\mathbf{U}^{l}_{t} = \operatorname{MHA}\left(\operatorname{LN}(\mathbf{Z}^{l}_{t})\right), \tag*{(4)}

where query, key, and value are all LN⁡(Ztl)\operatorname{LN}(\mathbf{Z}^{l}_{t}), and add the result back through a plain residual,

zs,tl←zs,tl+us,tl.(5)\mathbf{z}^{l}_{s,t} \leftarrow\mathbf{z}^{l}_{s,t} + \mathbf{u}^{l}_{s,t}. \tag*{(5)}

The output projection of the attention is initialized close to zero (std 10−410^{-4}), so CSA is initially a near-identity and a warm-started DiCoP encoder is left almost unchanged. As SS acts as the sequence length without positional encoding, the number of speakers is not fixed.

STNO mask refiner. A limitation of the STNO conditioning is that speakers overlapping in a frame receive the same mask—an overlap frame is labeled identically for all of them—so the conditioning alone cannot tell them apart. Longform decoding should already help here: similarly to SE-DiCoW [22], per-speaker evidence can be aggregated across the whole recording into the global tokens. We additionally let the model re-estimate a soft per-speaker activity from the cross-speaker-informed states,

d^l(s,t)=σ(wl⊤LN⁡(zs,tl)+bl)∈[0,1],(6)\hat{d}^{l}(s,t) = \sigma\left(\mathbf{w}_{l}^{\top}\operatorname{LN}(\mathbf{z}^{l}_{s,t}) + b_{l}\right) \in[0,1], \tag*{(6)}

stacks the SS re-estimated activities into d^t=[d^l(s,t)]s=1S\hat{\mathbf{d}}_{t} = [\hat{d}^{l}(s,t)]_{s=1}^{S}, rebuilds a fresh STNO mask M^l,t=Φgl(d^t)\hat{\mathbf{M}}^{l,t} = \boldsymbol{\Phi}_{g_{l}}(\hat{\mathbf{d}}_{t}) with the mapping of Section 2.1, and blends it back through a sigmoid gate,

Mt←(1−βl)Mt+βlM^l,t,βl=σ(gl).(7)\mathbf{M}^{t} \leftarrow(1-\beta_{l})\mathbf{M}^{t} + \beta_{l}\hat{\mathbf{M}}^{l,t}, \qquad\beta_{l} = \sigma(g_{l}). \tag*{(7)}

Each refiner has its own parameters wl\mathbf{w}_{l}, blb_{l}, and glg_{l}. The logits glg_{l} start negative, so the input mask is unchanged at initialization.

Experimental Setup

Throughout this paper, we report the concatenated minimum-permutation word error rate (cpWER) [30], computed with MeetEval [31], with oracle diarization unless stated otherwise. Predicted diarization comes from pyannoteAI Precision-2 [23, 24].4 For consistency, all models use greedy decoding.

Datasets

We use four sources of training data. Real data is the genuine-meeting subset (115 h): the single-distant-microphone training sets of NOTSOFAR-1 [32] (35.6 h) and AMI [17] (79.4 h) only. DiCoW-orig. data (1.2k h) is the original SE-DiCoW training recipe [22]—AMI, NOTSOFAR-1, LibriMix [33], and LibriSpeechMix-like mixtures—and drives the controlled ablation (Figure 2). Synthetic data (4k h in total) serve as initialization: 2k h of FastMSS [34] mixtures of up to eight speakers from LibriSpeech [35], and 2k h built from Fisher [36] with Sdio [37] and forced alignment [38]. Pseudo-labels are filtered YODAS2 self-labels described below (5.4k h mined, of which 4.6k h train the Parakeet-based models). We evaluate on NOTSOFAR-1 (NSF), AMI, ICSI [39], AISHELL-4 (A4) [19], AliMeeting (Ali) [18], the Santa Barbara Corpus (SBC) [40], Mixer6 (Mx6) [41], and DiPco [42]. For SBC we use the conversation and task subsets, restricted to recordings with two to eight speakers. Tables and figures use these abbreviations.

Accuracy/speed trade-off per evaluation set

Figure 2. Accuracy/speed trade-off per evaluation set (oracle diarization) against real-time-factor speedup (RTFx, greedy decoding). Line color: evaluation set; marker: model, connected in order of speed; all decode full conversations. All systems but Pandemonium train on the DiCoW-orig. data (Section 3.1); DiCoP variants differ in attention (loc/glob: local/global) and training loss, without CSA.

Pseudo-labeling YODAS2

Since genuine multi-speaker interaction is scarce in curated data and unrealistic in synthetic data, we pseudo-label the YODAS2 corpus [20]. We diarize it with pyannoteAI Precision-2 [23, 24], and among the 10 min chunks that lie within the Whisper languages and contain at most eight speakers, select 5.4k h with the highest speaker overlap. Final pseudo-labels are produced by re-diarizing the selected data with an in-house eight-speaker streaming Sortformer [43] and transcribing with DiCoW v3.3 [22] conditioned on that diarization. We then clean the labels with three filters: a language-ID check4 with Kimi K3 [44] against the conditioning language, a regex detector of nn-gram repetitions (hallucinations), and a leakage filter that discards segments where two speakers share five or more consecutive words simultaneously. Of the 5.4k h mined, DiCoP and Pandemonium train on the 4.6k h that remain after dropping 0.8k h in languages Parakeet does not support (e.g., Mandarin). The DiCoW pseudo-label study of Table 2 instead uses the 3.4k h left after segmenting into 30 s windows and dropping single-speaker ones.

System / selectionHrsNSFAMIA4AliMx6SBCAvg
DiCoW v3.3 (orig.) [22]1.2k15.514.533.739.311.415.021.6
SE-DiCoW (orig.) [22]1.2k15.314.440.243.411.715.223.4
DiCoW (synthetic) [34]4.0k23.824.726.835.916.822.125.0
DiCoW — pseudo-labels only (no real data):
All pseudo-labels3.4k17.219.124.331.311.715.719.9
LID check (only)3.1k17.319.424.531.411.816.320.1
hallucination (only)3.2k17.118.024.431.411.215.519.6
leakage filter (only)3.3k17.018.924.631.614.216.020.4
all filters combined2.9k17.018.324.030.910.915.719.5
DiCoW — pseudo-labels + finetune on real:
FT: DiCoW-orig.4.1k15.514.428.137.111.415.120.3
FT: NSF+AMI+A4+Ali3.2k15.815.015.619.311.715.315.5

Table 2. Influence of pseudo-label quality on meeting benchmarks. Rows: DiCoW with each cleaning filter applied to the full pool; orig.: trained on the DiCoW-orig. data. cpWER [%] ↓\downarrow. Best per column in bold. Underlined: in-domain, i.e., the matched training set was used.

Model and training details

DiCoP warm-starts from Parakeet-TDT-0.6b-v3 [14] with local attention (context [256, 256]), decoded by the TDT head or the CTC head of § 2.3. Pandemonium inserts cross-speaker attention after encoder layers {8,12,16,20}\{8, 12, 16, 20\} and the STNO mask refiner after {12,16,20}\{12, 16, 20\}; training conversations are capped at S=8S = 8 speakers. The diarization mask is randomly corrupted in training so the refiner has errors to correct and gets a 10×10 \times learning rate. Configurations are released with the models; DiCoW baselines follow SE-DiCoW [22].

Results and Discussion

Table 1 presents our main results. Pandemonium outperforms the DiCoW v3.3 baseline [22] and the longform DiCoP backbone on every benchmark with oracle diarization, and on all but two with predicted diarization. With predicted diarization, it also surpasses the published single-microphone state-of-the-art. Without multichannel processing, decoding every distant microphone and merging the hypotheses with MOVER [21] (Pandemonium + MOVER) lowers cpWER by a further 0.8–3.4 absolute. Note that the DiCoP row is the stronger TDT-trained model, favoring the baseline.

A fast but weaker baseline. Figure 2 establishes the longform TS-ASR baseline: integrating the conditioning into Parakeet yields a model that decodes 44–8×8 \times faster than DiCoW but is up to 3.8 cpWER worse. Local attention decodes full conversations faster and better than global attention on every set, so we adopt it for longform. Switching to CTC training costs 0.6–1.5 cpWER but raises the RTFx by 1.5×1.5 \times (269 to 405). Longer training chunks (60 s, 10 min; not shown) do not close the gap, moving cpWER by at most 0.6 in either direction. Pandemonium keeps most of this speed (RTFx 330 vs. 405) and is the most accurate on every set.

Pseudo-labels as a source of interaction. Curated meeting corpora contain genuine multi-speaker interaction but are far too small to learn it from, so we mine more from YODAS2 (§ 3.1.1) and first check the quality of the resulting pseudo-labels by training DiCoW on them alone (Table 2). Even unfiltered, a DiCoW trained on this data is competitive with both the synthetic and the DiCoW v3.3 (orig.) reference, and reaches a better average cpWER (19.9 vs. 21.6). Thanks to the diverse YODAS2 languages, the pseudo-label-trained model is more generic than the English-skewed v3.3. SE-DiCoW, in turn, matches v3.3 in-domain but is 4–7 cpWER worse on the Mandarin sets, so self-enrollment does not transfer across languages. Adding the individual filters, only hallucination removal gives a clear improvement. We attribute the robustness to the remaining label noise to the diversity of the data, which, together with batch size 96 and gradient clipping, averages the noise out and skips the worst examples, as also observed in OWLS [45]. Finally, finetuning the pseudo-label-pretrained checkpoint on the DiCoW-orig. data improves NSF and AMI but degrades AISHELL-4 and AliMeeting. Finetuning on the real meeting sets alone, with the Mandarin AISHELL-4 and AliMeeting training sets added, instead brings both

to 15.6 and 19.3 cpWER, far below any English-only model, and gives the best average, 15.5.

Interactions turn the baseline around. Table 3 isolates each contribution of Pandemonium. The interaction modules of Section 2.4 help: over plain DiCoP (17.1 cpWER), CSA (Eq. (4)) gains 0.3 and the STNO mask refiner (Eq. (7)) a further 1.8 (15.0). They need longform context, though: the same architecture trained on 30 s windows falls back to 17.1, forfeiting the entire gain. On the data side, the picture is clear: the three-stage schedule synthetic →\rightarrow pseudo →\rightarrow real is best, with synthetic mixtures serving as a useful initialization and real data as a final refinement. Pseudo-labels alone trail synthetic-only (20.1 vs. 15.0). We attribute this to the English-skewed evaluation, which hides the cross-lingual gains of the pseudo-labels, as Parakeet does not support Mandarin. Together, they take DiCoP from 16.7 to 14.7 average cpWER in Table 1.

SystemNSFAMIMx6SBCAvg
In-domainOut-of-domain
Pandemonium (full)14.113.411.113.813.1
Training-data source:
w/o real (synth+pseudo)14.418.117.714.016.1
pseudo-labels only20.226.616.816.920.1
synthetic only16.717.610.915.015.0
Architecture (synthetic-only pretraining):
− refiner18.819.611.717.016.8
− refiner − CSA (DiCoP)19.520.011.817.117.1
full, 30 s windows19.320.011.917.117.1

Table 3. Ablation of Pandemonium. Top: training data. Bottom: architecture, removing the STNO refiner, then also CSA (plain DiCoP), and the full model trained on 30 s windows. cpWER [%] ↓\downarrow.

Conclusion

We addressed two limitations of diarization-conditioned TS-ASR: independent per-speaker decoding and the fixed 30 s window. Integrating the conditioning into a longform Parakeet backbone gives a fast but weaker baseline; cross-speaker attention and an STNO mask refiner, trained on 4.6k hours of pseudo-labeled YODAS2 conversations, make it state-of-the-art. Future work will fuse DiCoP with a conversation-pretrained language model, recover TDT training via a pruned transducer loss, and train diarization and recognition jointly.

References

References

  1. [1]N. Kanda et al., “Serialized output training for end-to-end overlapped speech recognition,” in Proc. Interspeech, 2020.
  2. [2]H. Shi et al., “Serialized output prompting for large language model-based multi-talker speech recognition,” in Proc. ASRU, 2025.arxiv.org/abs/2509.04488
  3. [3]Z. Peng et al., “VibeVoice-ASR technical report,” arXiv:2601.18184, 2026.arxiv.org/abs/2601.18184
  4. [4]D. Yu et al., “Recognizing multi-talker speech with permutation invariant training,” in Proc. Interspeech, 2017.arxiv.org/abs/1704.01985
  5. [5]S. Niu et al., “The USTC-NERCSLIP systems for the CHiME-8 NOTSOFAR-1 challenge,” in Proc. CHiME, 2024.arxiv.org/abs/2409.02041
  6. [6]N. Kanda et al., “Simultaneous speech recognition and speaker diarization for monaural dialogue recordings with target-speaker acoustic models,” in Proc. ASRU, 2019.arxiv.org/abs/1909.08103
  7. [7]M. Delcroix et al., “Single channel target speaker extraction and recognition with SpeakerBeam,” in Proc. ICASSP, 2018.DOI
  8. [8]T. Park et al., “Pushing the boundaries of streaming multi-speaker ASR: A systematic study of architectural trade-offs,” arXiv:2609.10265, accepted to Interspeech 2026.arxiv.org/abs/2609.10265
  9. [9]A. Polok et al., “DiCoW: Diarization-conditioned Whisper for target speaker automatic speech recognition,” Comput. Speech Lang., 2026.
  10. [10]K. Kim et al., “E-Branchformer: Branchformer with enhanced merging for speech recognition,” in Proc. SLT, 2023.arxiv.org/abs/2210.00077
  11. [11]A. Radford et al., “Robust speech recognition via large-scale weak supervision,” in Proc. ICML, 2023.arxiv.org/abs/2212.04356
  12. [12]A. Polok et al., “Grounding spoken LLMs in multi-speaker audio via diarization conditioning,” accepted to Interspeech 2026.arxiv.org/abs/2606.18134
  13. [13]X. Shi et al., “Qwen3-ASR technical report,” arXiv:2601.21337, 2026.arxiv.org/abs/2601.21337
  14. [14]M. Sekoyan et al., “Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and high-performance models for multilingual ASR and AST,” arXiv:2509.14128, 2025.arxiv.org/abs/2509.14128
  15. [15]D. Rekesh et al., “Fast Conformer with linearly scalable attention for efficient speech recognition,” in Proc. ASRU, 2023.arxiv.org/abs/2305.05084
  16. [16]A. Vinnikov et al., “NOTSOFAR-1 challenge: New datasets, baseline, and tasks for distant meeting transcription,” in Proc. Interspeech, 2024.arxiv.org/abs/2401.08887
  17. [17]I. McCowan et al., “The AMI meeting corpus,” Proc. Measuring Behavior, 2005.
  18. [18]F. Yu et al., “M2MeT: The ICASSP 2022 multi-channel multi-party meeting transcription challenge,” in Proc. ICASSP, 2022.arxiv.org/abs/2110.07393
  19. [19]Y. Fu et al., “AISHELL-4: An open source dataset for speech enhancement, recognition and speaker diarization in conference scenario,” in Proc. Interspeech, 2021.arxiv.org/abs/2104.03603
  20. [20]X. Li et al., “YODAS: YouTube-Oriented dataset for audio and speech,” in Proc. ASRU, 2023.arxiv.org/abs/2406.00899
  21. [21]N. Kamo et al., “MOVER: Combining multiple meeting recognition systems,” in Proc. Interspeech, 2025.
  22. [22]A. Polok et al., “SE-DiCoW: Self-enrolled diarization-conditioned Whisper,” in Proc. ICASSP, 2026.
  23. [23]A. Laurent et al., “Scaling self-supervised pretraining for speaker diarization,” in Proc. Odyssey, 2026.DOI
  24. [24]H. Bredin, “pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe,” in Proc. Interspeech, 2023.DOI
  25. [25]Trelis, “Tiron: Open-weights multi-speaker meeting transcription model,” https://huggingface.co/Trelis/tiron, 2026.
  26. [26]N. Tawara et al., “Who spoke what when? evaluating spoken language models for conversational ASR with semantic and overlap-aware metrics,” arXiv:2603.22709, accepted to Interspeech 2026.arxiv.org/abs/2603.22709
  27. [27]H. Xu et al., “Efficient sequence transduction by jointly predicting tokens and durations,” in Proc. ICML, 2023.arxiv.org/abs/2304.06795
  28. [28]A. Graves et al., “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proc. ICML, 2006.
  29. [29]S. Horiguchi et al., “Multi-channel end-to-end neural diarization with distributed microphones,” in Proc. ICASSP, 2022.arxiv.org/abs/2110.04694
  30. [30]S. Watanabe et al., “CHiME-6 challenge: Tackling multi-speaker speech recognition for unsegmented recordings,” in Proc. CHiME, 2020.DOI
  31. [31]T. von Neumann et al., “MeetEval: A toolkit for computation of word error rates for meeting transcription systems,” in Proc. CHiME, 2023.arxiv.org/abs/2307.11394
  32. [33]J. Cosentino et al., “LibriMix: An open-source dataset for generalizable speech separation,” arXiv:2005.11262, 2020.arxiv.org/abs/2005.11262
  33. [34]A. Polok et al., “Mind the gap: Impact of synthetic conversational data on multi-talker ASR and speaker diarization,” arXiv:2605.15442, accepted to Interspeech 2026.arxiv.org/abs/2605.15442
  34. [35]V. Panayotov et al., “LibriSpeech: An ASR corpus based on public domain audio books,” in Proc. ICASSP, 2015.DOI
  35. [36]C. Cieri et al., “The Fisher corpus: a resource for the next generations of speech-to-text,” in Proc. LREC, 2004.DOI
  36. [37]W. Nakata et al., “Sidon: Fast and robust open-source multilingual speech restoration for large-scale dataset cleansing,” in Proc. ICASSP, 2026.
  37. [38]M. McAuliffe et al., “Montreal Forced Aligner: Trainable text-speech alignment using Kaldi,” in Proc. Interspeech, 2017.DOI
  38. [39]A. Janin et al., “The ICSI meeting corpus,” in Proc. ICASSP, 2003.DOI
  39. [40]M. Maciejewski et al., “Evaluating the Santa Barbara Corpus: Challenges of the breadth of conversational spoken language,” in Proc. Interspeech, 2024.DOI
  40. [41]L. Brandschain et al., “Mixer 6,” in Proc. LREC, 2010.
  41. [42]M. Van Segbroeck et al., “DiPCo — Dinner Party Corpus,” in Proc. Interspeech, 2020.arxiv.org/abs/1909.13447
  42. [43]I. Medennikov et al., “Streaming Sortformer: Speaker cache-based online speaker diarization with arrival-time ordering,” in Proc. Interspeech, 2025.arxiv.org/abs/2507.18446
  43. [44]Kimi Team et al., “Kimi K3: Open frontier intelligence,” arXiv:2607.24653, 2026.
  44. [45]W. Chen et al., “OWLS: Scaling laws for multilingual speech recognition and translation models,” in Proc. ICML, 2025.arxiv.org/abs/2502.10373

Paper details

vlnc:W_pzdqft9b
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents