betaFeedback

WorksEngineering

Nemotron 3 Diarization: Advancing the Accuracy-Efficiency Frontier in Streaming Speaker Diarization

Abstract

Streaming Sortformer introduced an arrival-order speaker cache to maintain speaker identities across chunks in end-to-end streaming speaker diarization, with a single model supporting configurable latency profiles spanning offline operation to ultra-low-latency streaming. Nemotron 3 Diarization addresses the original model's four-speaker limitation by extending support to eight speakers while improving both accuracy and computational efficiency. The previous FastConformer-Transformer stack is streamlined into a pure Transformer encoder. A learned upsampling layer then maps the encoder's coarse 80-ms representations to 10-ms resolution, enabling fine-grained predictions. Training leverages large-scale simulated mixtures and adds activity and phantom-speaker losses to the standard Sortformer objectives. Despite doubling speaker capacity over its predecessor, the new architecture reduces training memory by a factor of 2.8, makes training 2.2×2.2\times faster, and delivers up to 6.4×6.4\times higher inference throughput. Nemotron 3 Diarization establishes a new state of the art in DER on DIHARD III and CALLHOME. When used to condition DiCoW, it also achieves state-of-the-art tcpWER on the single-microphone conditions of AMI and NOTSOFAR-1. Training code is open source, and model weights are released under the permissive OpenMDW-1.1 license.

Introduction

Speaker diarization remains a core capability for meeting transcription and multi-talker automatic speech recognition (ASR), even as speech large language models (LLMs) increasingly generate speaker-attributed transcripts directly. Recent systems and benchmarks explore unified prediction of lexical content, timestamps, and speaker labels [1, 2]. Yet these developments do not eliminate the need for explicit diarization: DM-ASR [3] uses diarization as an explicit structural prior for an LLM, while a recent comparative study [4] found unified models competitive in two-speaker conditions but less robust than modular pipelines as speaker count and overlap increase. Accurate diarization therefore remains valuable both as a standalone capability and as a component of systems built around speech LLMs.

End-to-end neural diarization (EEND) replaces multi-stage diarization pipelines with direct prediction of joint speaker activity and naturally represents overlapping speech [5]. Extensions based on encoder-decoder attractors accommodate an unknown number of speakers [6], while EEND with vector clustering combines overlap-aware local predictions with global embedding clustering to handle long recordings and arbitrary speaker inventories [7]. This local-to-global design underlies widely used systems such as Pyannote [8] and DiarizeN [9, 10]. By contrast, Sortformer [11] follows a fixed-capacity EEND approach, combining permutation-invariant loss (PIL) with arrival-time-sort (ATS) supervision to assign speakers to output channels according to their order of first arrival. Because densely annotated real conversations are scarce, EEND relies heavily on simulated mixtures. Large-scale pre-training improves performance [12], but its effectiveness depends on turn-taking statistics, source diversity, acoustic augmentation, and how simulated and real data are combined [13].

Most of this progress targets offline processing. Importantly, streaming diarization must preserve identities across chunks despite bounded right context, overlapping speech, and newly arriving speakers. Prior online systems address this problem through limited-context encoders, speaker-tracing buffers, or blockwise speaker detection and representation [14, 15, 16, 17]. Streaming Sortformer [18] adds an arrival-order speaker cache (AOSC) to maintain channel ordering across chunks, allowing one model to serve multiple latency profiles. Its released model, however, is limited to four speakers and produces predictions only every 80 ms. This leaves a gap for an open model that combines competitive accuracy in both offline and low-latency streaming settings, with support for more speakers, precise boundaries, and high computational efficiency.

These requirements are tightly coupled. Within the AOSC design, supporting more speakers requires a larger cache to retain sufficient evidence for each identity, which directly increases inference cost. Additional output channels also remain unused in typical recordings, increasing the risk of phantom-speaker activations. Meanwhile, running the encoder at a 10-ms frame step to improve boundary precision would greatly increase sequence length and attention cost. The model must also remain robust across latency profiles that provide different amounts of future context. These trade-offs motivate a design that supports more speakers and finer temporal resolution without sacrificing streaming efficiency.

This work introduces Nemotron 3 Diarization1, an open-weight model addressing these constraints. Our contributions are threefold. First, we redesign the architecture to efficiently increase speaker capacity and output temporal resolution. Second, we introduce an activity loss that models overall speech activity and a phantom loss that suppresses false activations in unused speaker channels. We pair these objectives with mixed-data and variable-context training to support both offline and streaming operation. Third, we analyze these choices through ablations of the backbone, initialization, temporal resolution, objectives, augmentations, and training data. We evaluate diarization accuracy across latencies and benchmark training and inference efficiency. We also assess downstream speaker-attributed ASR in offline and streaming settings.

Nemotron 3 Diarization

Training data

Nemotron 3 Diarization is trained on approximately 5,000 h each of human-annotated and pseudo-labeled real recordings, together with around 82,600 h of simulated multi-talker mixtures.

The real multi-domain training data include the following public corpora: AISHELL-4 [19], AliMeeting [20], AMI [21], CALLHOME Part 1 [22], DIHARD III Development [23], DiPCo [24], DISPLACE-2024 [25], DISPLACE-M [26], Fisher English [27], ICSI [28], NOTSOFAR-1 [29] (NSF1), and VoxConverse-v0.3 [30]. Additionally, a licensed David AI{^2} dataset of around 1,000 hours of 3–4-speaker English conversations is used. We also include a 5,000-hour multi-talker subset of YODAS-v2 {citep}[31] pseudo-labeled using an ensemble of nine Sortformers combined via DOVER-Lap {citep}[32]`.

The simulated mixtures (1–8 speakers, 180–200 s per mixture) were created using the FastMSS{^3} toolkit following the *Mind the Gap* protocol from {citep}[13]. The simulated data comprise 6,694 h from LibriSpeech {citep}[33], 6,743 h from AMI headsets, 6,745 h from AliMeeting headsets, 6,755 h from Fisher English, 36,458 h from around 22,000 h of licensed David AI English audio, and 19,216 h from around 2,000 h of licensed David AI multilingual audio in 21 languages. Notably, simulations use tight speaker-activity regions obtained with the Montreal Forced Aligner {citep}[34]`.

Architecture

Nemotron 3 Diarization builds on the Streaming Sortformer [18] paradigm with several important changes. First, the backbone is streamlined from a 17-layer FastConformer [35] followed by an 18-layer Transformer [36] to a single 31-layer Transformer with rotary positional embeddings (RoPE) [37]. This change is motivated by both higher computational efficiency and better scalability of a pure Transformer. This also makes the model more compact, reducing the total number of parameters from 117M to 99M. Input features are downsampled eightfold through feature stacking. The Transformer encoder therefore retains the FastConformer’s 80-ms frame step.

Second, we introduce a learned one-dimensional subpixel convolution on top of the encoder. This layer upsamples the encoder outputs by a factor of eight, enabling accurate detection of short turns and precise boundaries while preserving the efficient 80-ms encoder frame step. Specifically, a width-3 convolution maps each HH-dimensional encoder state to 8H8H channels, which are rearranged into eight consecutive HH-dimensional states. The encoder therefore remains at 80-ms resolution, while the output heads operate at an effective 10-ms resolution with almost no computational overhead.

Finally, AOSC capacity is increased from 188 to 264 frames to accommodate the twofold increase in supported speaker count. The mean silence embedding used to separate speakers in AOSC is replaced with a learnable embedding for greater robustness.

Training objectives

Alongside the standard Sortformer PIL and ATS losses, which use the best target-speaker permutation and arrival-time-ordered targets, respectively, Nemotron 3 Diarization introduces two auxiliary losses: the activity loss and the phantom loss.

The activity loss is a cross-entropy loss over three permutation-free frame activity classes: silence, single-speaker speech, and overlapping speech. A separate auxiliary head predicts these classes from the encoder representations. This auxiliary task regularizes training by supervising overall speech activity without requiring speaker assignment.

The phantom loss addresses speaker overcounting caused by high-confidence activity predictions (“phantom speakers”) in channels empty under both the PIL and ATS alignments. Let pbtsp_{bts} be the speech posterior and sg denote stop-gradient. For each empty channel, Ibs\mathcal{I}_{bs} contains valid frames satisfying sg⁡(pbts)>θ\operatorname{sg}(p_{bts}) > \theta; otherwise, Ibs=∅\mathcal{I}_{bs} = \varnothing. Each selected frame incurs the inactive-label negative log-likelihood ℓbts=−log⁡(1−pbts)\ell_{bts} = -\log(1-p_{bts}). These losses are pooled as

Lph=τBS∑b=1B∑s=1Slog⁡[1∣Ibs∣∑t∈Ibsexp⁡(ℓbtsτ)].\mathcal{L}_{\mathrm{ph}} = \frac{\tau}{BS}\sum_{b=1}^{B}\sum_{s=1}^{S}\log\left[\frac{1}{\lvert\mathcal{I}_{bs}\rvert}\sum_{t\in\mathcal{I}_{bs}}\exp\left(\frac{\ell_{bts}}{\tau}\right)\right].

We set θ=0.25\theta= 0.25, τ=0.5\tau= 0.5 and S=8S = 8, and define the contribution as zero when Ibs=∅\mathcal{I}_{bs} = \varnothing. This normalized log-mean-exp pooling emphasizes the largest false activations without scaling directly with the number of selected frames.

Model training and augmentations

Training is performed using the NVIDIA NeMo Speech Framework{^4} {citep}[38]` on 64 NVIDIA A100-SXM4-80GB GPUs in two stages.

First, offline Sortformer training [11] is performed on simulated mixtures only, with the encoder initialized from a self-supervised NEST [39] Transformer pretrained on a 900,000-hour subset of Granary-v2, an expanded version of the Granary corpus [40]. Training uses random 60-second chunks drawn on the fly from full recordings. The PIL, ATS, activity, and phantom loss weights are 0.5, 0.5, 0.5, and 0, respectively. The offline stage operates at the native 80-ms output resolution without the upsampling layer.

Second, the model is fine-tuned for streaming with AOSC [18] enabled, using both real and simulated data. The 105-second training samples are constructed on the fly, with two chunks from different parts of a source recording concatenated with probability 0.5; sampling favors examples containing more active speakers. The PIL, ATS, activity, and phantom loss weights are 0.75, 0.25, 0.5, and 0.1, respectively. The upsampling layer is introduced at this stage, initialized to repeat each encoder state across eight output positions, and jointly trained with the rest of the model using 10-ms targets.

Additionally, to prevent the model from relying on future context during streaming inference, each batch receives a block-causal tail with probability 0.5. This modifies the self-attention mask to contain a bidirectional prefix followed by a randomly sampled tail. The tail is divided into blocks of one to eight frames: attention is bidirectional within each block, while each block may attend to the prefix and earlier blocks but not to future blocks. Full-context batches and this family of masks jointly expose one checkpoint to offline and multiple streaming contexts.

Finally, both training stages apply SpecAugment [41] regularization with two frequency masks spanning up to 24 feature bins each. The detailed training parameters used are given in the publicly available offline{^5} and streaming{^6} configuration files in the NeMo Speech repository.

Table 1. DER (%) ↓ in offline and streaming conditions across evaluation sets. DH3, CH, Ali and NSF1 denote DIHARD III Eval, CALLHOME Part 2, AliMeeting and NOTSOFAR-1, respectively. All overlap is scored; collar tolerance is 0.25 s for CH and 0 s for the other sets. Best offline results are bold.

SystemLatencyDH3CHAli-NearAli-FarAMI-MHMAMI-SDMNSF1-MHMNSF1-SC
DiariZenoffline14.514.6816.7810.8*15.5613.9*14.2016.7*
Sortformer-v2.130.4 s19.0910.3211.5713.6915.8121.4221.7730.49
Nemotron 3 Diarization30.4 s12.739.106.4010.479.2511.146.7711.00
Sortformer-v2.11.04 s19.6011.3112.4715.5816.3621.7322.1231.81
Nemotron 3 Diarization1.04 s13.1810.296.5910.809.4812.807.7012.77
Sortformer-v2.10.32 s19.8512.6713.6816.8517.7723.8923.3732.95
Nemotron 3 Diarization0.32 s13.5511.327.1911.6010.0512.958.6514.53
SOTA 09/2026offline14.1 [42]10.08 [43]8.60* [12]10.6* [42]11.06* [12]13.7* [42]—16.5* [42]

Table 1.

*Reported using the original/default reference annotations rather than the force-aligned reference annotations used for the corresponding results in this work; these values are therefore not directly comparable with ours.

Results

Diarization accuracy evaluation

The evaluation contains the following datasets: DIHARD III evaluation [23] (259 recordings, 1–9 speakers per recording), CALLHOME Part 2 (250 recordings, 2–6 speakers), AliMeeting [20] near and far microphones (20 recordings, 2–4 speakers), AMI [21] Mix of Headset Microphones (MHM) and Single Distant Microphone (SDM) conditions (16 recordings, 3–4 speakers), and NOTSOFAR-1 [29] MHM and Single-Channel (SC) conditions (160 recordings, 3–7 speakers). All overlapped speech is scored. DER uses no forgiveness collar except for CALLHOME, where the conventional 0.25-s collar is used. It is important to note that we use forced-alignment-based reference labels for AMI, AliMeeting7 and NOTSOFAR-18 instead of the original segment-level annotations that label substantial within-segment silences as speech, thereby overestimating reference speaker activity [44].

We consider two open-source baselines: the previous four-speaker Streaming Sortformer-v2.19 [18] and DiariZen10 [9, 10], the strongest offline system. Table 1 reports DER at the three input-buffer latencies of 30.4, 1.04, and 0.32 s used in the original Streaming Sortformer study [18]. Reducing the buffer from 30.4 to 0.32 s increases DER by 0.79–3.53 percentage points across the evaluation sets, thus demonstrating graceful degradation. The same checkpoint can operate with input buffers as short as 0.08 s, although accuracy degrades further; we omit this regime to retain direct comparability with the previous work. At all three reported latencies, Nemotron 3 Diarization outperforms Sortformer-v2.1 across every evaluation set and the offline DiariZen baseline on every directly comparable condition. This establishes a new state of the art (SOTA) on DIHARD III and CALLHOME benchmarks. Although we believe our results on AMI, AliMeeting, and NOTSOFAR-1 are also SOTA, we do not claim this formally because the starred results use different ground-truth labels and are therefore not directly comparable with ours.

Table 2. Inference RTFx ↑ on an RTX PRO 5000 Blackwell GPU using bf16; “comp.” denotes dynamic torch.compile. The best result for each batch size and latency is bold.

ModelLatencyBatch 1 eagerBatch 1 comp.Batch 32 eagerBatch 32 comp.
DiariZenoffline20—44—
Sortformer-v2.130.4 s874146832042619
Nemotron 3 Diarization30.4 s134043851219615113
Sortformer-v2.11.04 s1642193136
Nemotron 3 Diarization1.04 s38164581865
Sortformer-v2.10.32 s82110176
Nemotron 3 Diarization0.32 s12.554199292

Table 2.

Training and inference efficiency

During training on 64 NVIDIA A100 GPUs in bf16-mixed precision, the streamlined Transformer reduces memory consumption by a factor of 2.8 and iteration time by a factor of 2.2 relative to an otherwise identical model with the legacy FastConformer encoder.

Table 2 compares inference speed on one NVIDIA RTX PRO 5000 Blackwell GPU in bf16 precision. Here, Real-Time Factor Speedup (RTFx) defined as audio duration divided by wall-clock processing time is calculated with batch sizes of 1 and 32 on a large set of around 900 s long recordings. Compiled measurements exclude the one-time graph compilation overhead and are collected after compilation and warm-up. It reveals massive inference speed improvements over Sortformer-v2.1 with legacy FastConformer [35] encoder, despite the twofold increase in speaker capacity. Notably, even with the ultra-low input buffer latency of 0.32 s, the model is fast enough to support multiple parallel sessions on a single GPU.

Ablations

To assess the contribution of each component, Table 3 reports DER and its absolute change relative to Nemotron 3 Diarization at offline (30.4 s) and ultra-low (0.32 s) input buffer latency. All recordings from the constituent evaluation sets are scored jointly as one compound evaluation set with a zero collar. Each ablation is represented by a single training run using the same optimization and eval-

System30.4 s DER30.4 s Δ\Delta0.32 s DER0.32 s Δ\Delta
Nemotron 3 Diarization10.42–12.06–
FastConformer enc.11.36+0.9413.33+1.27
no NEST initialization12.05+1.6314.28+2.22
downsampled (80 ms)11.09+0.6712.67+0.61
no upsampling (80 ms)12.69+2.2714.09+2.03
no phantom loss10.41-0.0112.15+0.09
no activity loss11.42+1.0013.03+0.97
no SpecAugment10.88+0.4612.47+0.41
no block-causal tail10.50+0.0812.36+0.30
no simulated data13.40+2.9815.21+3.15
no David AI data11.19+0.7712.83+0.77
no YODAS data10.81+0.3912.38+0.32

Table 3. Ablations on the compound evaluation set: globally pooled, zero-collar DER (%) ↓\downarrow at input-buffer latencies of 30.4 s and 0.32 s. Δ\Delta is the absolute DER change; positive values indicate degradation.

Table 1. DER (%) ↓ in offline and streaming conditions across evaluation sets. DH3, CH, Ali and NSF1 denote DIHARD III Eval, CALLHOME Part 2, AliMeeting and NOTSOFAR-1, respectively. All overlap is scored; collar tolerance is 0.25 s for CH and 0 s for the other sets. Best offline results are bold.

SystemLatencyDH3CHAli-NearAli-FarAMI-MHMAMI-SDMNSF1-MHMNSF1-SC
DiariZenoffline14.514.6816.7810.8*15.5613.9*14.2016.7*
Sortformer-v2.130.4 s19.0910.3211.5713.6915.8121.4221.7730.49
Nemotron 3 Diarization30.4 s12.739.106.4010.479.2511.146.7711.00
Sortformer-v2.11.04 s19.6011.3112.4715.5816.3621.7322.1231.81
Nemotron 3 Diarization1.04 s13.1810.296.5910.809.4812.807.7012.77
Sortformer-v2.10.32 s19.8512.6713.6816.8517.7723.8923.3732.95
Nemotron 3 Diarization0.32 s13.5511.327.1911.6010.0512.958.6514.53
SOTA 09/2026offline14.1 [42]10.08 [43]8.60* [12]10.6* [42]11.06* [12]13.7* [42]—16.5* [42]

Table 1 (PDF p. 3).

*Reported using the original/default reference annotations rather than the force-aligned reference annotations used for the corresponding results in this work; these values are therefore not directly comparable with ours.

Results

Diarization accuracy evaluation

The evaluation contains the following datasets: DIHARD III evaluation [23] (259 recordings, 1–9 speakers per recording), CALLHOME Part 2 (250 recordings, 2–6 speakers), AliMeeting [20] near and far microphones (20 recordings, 2–4 speakers), AMI [21] Mix of Headset Microphones (MHM) and Single Distant Microphone (SDM) conditions (16 recordings, 3–4 speakers), and NOTSOFAR-1 [29] MHM and Single-Channel (SC) conditions (160 recordings, 3–7 speakers). All overlapped speech is scored. DER uses no forgiveness collar except for CALLHOME, where the conventional 0.25-s collar is used. It is important to note that we use forced-alignment-based reference labels for AMI, AliMeeting7 and NOTSOFAR-18 instead of the original segment-level annotations that label substantial within-segment silences as speech, thereby overestimating reference speaker activity [44].

We consider two open-source baselines: the previous four-speaker Streaming Sortformer-v2.19 [18] and DiariZen10 [9, 10], the strongest offline system. Table 1 reports DER at the three input-buffer latencies of 30.4, 1.04, and 0.32 s used in the original Streaming Sortformer study [18]. Reducing the buffer from 30.4 to 0.32 s increases DER by 0.79–3.53 percentage points across the evaluation sets, thus demonstrating graceful degradation. The same checkpoint can operate with input buffers as short as 0.08 s, although accuracy degrades further; we omit this regime to retain direct comparability with the previous work. At all three reported latencies, Nemotron 3 Diarization outperforms Sortformer-v2.1 across every evaluation set and the offline DiariZen baseline on every directly comparable condition. This establishes a new state of the art (SOTA) on DIHARD III and CALLHOME benchmarks. Although we believe our results on AMI, AliMeeting, and NOTSOFAR-1 are also SOTA, we do not claim this formally because the starred results use different ground-truth labels and are therefore not directly comparable with ours.

Table 2. Inference RTFx ↑ on an RTX PRO 5000 Blackwell GPU using bf16; “comp.” denotes dynamic torch.compile. The best result for each batch size and latency is bold.

ModelLatencyBatch 1 eagerBatch 1 comp.Batch 32 eagerBatch 32 comp.
DiariZenoffline20—44—
Sortformer-v2.130.4 s874146832042619
Nemotron 3 Diarization30.4 s134043851219615113
Sortformer-v2.11.04 s1642193136
Nemotron 3 Diarization1.04 s38164581865
Sortformer-v2.10.32 s82110176
Nemotron 3 Diarization0.32 s12.554199292

Table 2 (PDF p. 3).

Training and inference efficiency

During training on 64 NVIDIA A100 GPUs in bf16-mixed precision, the streamlined Transformer reduces memory consumption by a factor of 2.8 and iteration time by a factor of 2.2 relative to an otherwise identical model with the legacy FastConformer encoder.

Table 2 compares inference speed on one NVIDIA RTX PRO 5000 Blackwell GPU in bf16 precision. Here, Real-Time Factor Speedup (RTFx) defined as audio duration divided by wall-clock processing time is calculated with batch sizes of 1 and 32 on a large set of around 900 s long recordings. Compiled measurements exclude the one-time graph compilation overhead and are collected after compilation and warm-up. It reveals massive inference speed improvements over Sortformer-v2.1 with legacy FastConformer [35] encoder, despite the twofold increase in speaker capacity. Notably, even with the ultra-low input buffer latency of 0.32 s, the model is fast enough to support multiple parallel sessions on a single GPU.

Ablations

To assess the contribution of each component, Table 3 reports DER and its absolute change relative to Nemotron 3 Diarization at offline (30.4 s) and ultra-low (0.32 s) input buffer latency. All recordings from the constituent evaluation sets are scored jointly as one compound evaluation set with a zero collar. Each ablation is represented by a single training run using the same optimization and evaluation settings. We therefore report observed effect sizes rather than statistical significance and interpret small differences cautiously. We emphasize effects that are consistent across both latency profiles or appear in the metric directly targeted by the component.

DiarizationAMI MHMAMI SDMNSF1 MHMNSF1 SC
Streaming: Multitalker-Parakeet-v1 (1.12 s latency)
Ground truth (oracle)12.6617.7916.8626.34
Softformer-v2.118.6434.1335.5951.35
Nemotron 3 Diarization (hard)13.8421.1818.6831.65
Offline: DiCoW-v3.3-large (greedy decoding)
Ground truth (oracle)11.7215.278.7815.45
DiarZen15.1719.3218.5725.63
Softformer-v2.118.1730.1130.1442.56
Nemotron 3 Diarization (hard)12.6517.1910.4719.23
Nemotron 3 Diarization (soft)12.3517.0610.3018.95
SOTA 09/202615.2 [47]18.5 [47]N/A23.63 [49]

Table 4. Downstream tcpWER (%) ↓\downarrow for streaming and offline Multi-Talker ASR. Hard and soft denote segment-level and posterior-based diarization conditioning, respectively. Bold indicates the best non-oracle result.

The pure Transformer backbone outperforms the legacy FastConformer. This may reflect better scaling with the available data or better compatibility with AOSC: cached speaker blocks do not form a continuous temporal sequence, so local convolutions can mix unrelated states across artificial block boundaries. The substantial degradation without NEST initialization highlights the importance of broad self-supervised pretraining, suggesting that the diarization data are insufficient to learn equally robust acoustic representations from scratch.

Removing high-resolution upsampling degrades DER considerably more than downsampling the 10-ms predictions to 80 ms before scoring. This indicates that learning with high-resolution outputs provides benefits beyond finer evaluation granularity.

Removing the phantom loss leaves DER nearly unchanged but increases speaker-count MAE from 0.157 to 0.222 offline and from 0.270 to 0.491 at 0.32-s latency, confirming its role in reducing overcounting. Removing the activity loss or SpecAugment also degrades DER at both latency profiles. The block-causal-tail augmentation has little effect at 30.4 s but provides a clearer benefit at 0.32 s.

Finally, training without simulated data produces the largest degradation, underscoring the importance of simulated mixtures. David AI and pseudo-labeled YODAS data also contribute, as excluding either degrades DER at both latency profiles.

Downstream multi-talker ASR performance

As already highlighted in subsection 3.1, DER is a highly annotation-style-dependent metric, which makes it hard to compare diarization systems trained on different labels directly. To overcome this limitation, we conduct a comparison of our Nemotron 3 Diarization with the baseline systems in a downstream multi-talker automatic speech recognition (MT-ASR) task. Unlike DER, time-constrained minimum-permutation Word Error Rate (tcpWER) operates on speaker-attributed ASR hypotheses rather than the diarization labels used to score frame-level errors. We calculate tcpWER using the meeteval package [45] with the default collar tolerance of 5 s and the chime8 text normalizer.

Table 4 therefore evaluates whether the diarization improvements transfer across reference-annotation protocols and two different MT-ASR models: a state-of-the-art offline DiCoW-v3.3-large11 [46, 47], and a streaming Multitalker-Parakeet-v112 [48] operating at default 1.12 s input buffer latency.

DiCoW conditioned on segment-level (hard) predictions from Nemotron 3 Diarization substantially outperforms both baselines, while posterior-based (soft) conditioning further reduces tcpWER in all four conditions. To our knowledge, this is the first consistent soft-over-hard gain with DiCoW. The resulting system also establishes new state-of-the-art tcpWER on both AMI and NOTSOFAR-1. In particular, on the NOTSOFAR-1 single-channel condition, the resulting system surpasses USTC-NERCSLIP sys2 [49] — the winner of the CHiME-8 NOTSOFAR-1 Challenge [29].13

With streaming Multitalker Parakeet, Nemotron 3 Diarization also substantially reduces tcpWER relative to the Softformer-v2.1 baseline. The consistent reductions in tcpWER across both ASR back-ends show that the diarization improvements transfer to downstream transcription rather than merely reflecting better agreement with the forced-aligned reference labels used for DER.

Conclusion

Nemotron 3 Diarization extends the Streaming Softformer framework from four to eight speakers while significantly improving both accuracy and throughput. Notably, it establishes a new state of the art on DIHARD III and CALLHOME in terms of DER, and on AMI and NOTSOFAR-1 in terms of tcpWER. Given that the model is released under the permissive OpenNMDW-1.1 license and is easily fine-tunable on target-domain data, we hope it sees wide adoption by both researchers and enterprises. Future work will further extend the supported speaker count, scale the model using expanded synthetic and pseudo-labeled corpora, and investigate direct integration into multimodal speech LLMs to enable seamless, end-to-end conversational reasoning.

References

References

  1. [1]B. Mu et al., “Summary on the multilingual conversational speech language model challenge: Datasets, tasks, baselines, and methods,” in Proc. ICASSP, 2026.arxiv.org/abs/2509.13785
  2. [2]Z. Peng et al., “VIBEVOICE-ASR technical report,” 2026.arxiv.org/abs/2601.18184
  3. [3]L. Li et al., “DM-ASR: Diarization-aware multi-speaker ASR with large language models,” 2026.arxiv.org/abs/2604.22467
  4. [4]N. Tawara et al., “Who spoke what when: evaluating spoken language models for conversational ASR with semantic and overlap-aware metrics,” in Proc. Interspeech, 2026.arxiv.org/abs/2603.22709
  5. [5]Y. Fujita et al., “End-to-end neural speaker diarization with permutation-free objectives,” in Proc. Interspeech, 2019.arxiv.org/abs/1909.05952
  6. [6]S. Horiguchi et al., “End-to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors,” in Proc. Interspeech, 2020.arxiv.org/abs/2005.09921
  7. [7]K. Kinoshita et al., “Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech,” in Proc. Interspeech, 2021.arxiv.org/abs/2105.09040
  8. [8]H. Bredin, “pyannote.audio 2.1 speaker diarization pipeline: Principle, benchmark, and recipe,” in Proc. Interspeech, 2023.DOI
  9. [9]J. Han et al., “Leveraging self-supervised learning for speaker diarization,” in Proc. ICASSP, arXiv:2409.09408.arxiv.org/abs/2409.09408
  10. [10]J. Han et al., “Efficient and generalizable speaker diarization via structured pruning of self-supervised models,” 2025.DOI
  11. [11]T. Park et al., “Sortformer: A novel approach for permutation-resolved speaker supervision in speech-to-text systems,” in Proc. ICML, 2025.arxiv.org/abs/2409.06656
  12. [12]S. J. Broughton et al., “Pushing the limits of end-to-end diarization,” in Proc. Interspeech, 2025.arxiv.org/abs/2509.14737
  13. [13]A. Polok et al., “Mind the gap: Impact of synthetic conversational data on multi-talker ASR and speaker diarization,” in Proc. Interspeech, 2026.arxiv.org/abs/2605.15442
  14. [14]E. Han et al., “BW-EDA-EEND: Streaming end-to-end neural speaker diarization for a variable number of speakers,” in Proc. ICASSP, 2021.DOI
  15. [15]Y. Xue et al., “Online streaming end-to-end neural diarization handling overlapping speech and flexible numbers of speakers,” in Proc. Interspeech, 2021.arxiv.org/abs/2101.08473
  16. [16]S. Horiguchi et al., “Online neural diarization of unlimited numbers of speakers using global and local attractors,” IEEE/ACM Trans. Audio, Speech, Lang. Process., 2022.DOI
  17. [17]M. Cheng et al., “Sequence-to-sequence neural diarization with automatic speaker detection and representation,” IEEE/ACM Trans. Audio, Speech, Lang. Process., 2025.arxiv.org/abs/2411.13849
  18. [18]I. Medennikov et al., “Streaming sortformer: Speaker cache-based online speaker diarization with arrival-time ordering,” in Proc. Interspeech, arXiv:2507.18446.arxiv.org/abs/2507.18446
  19. [19]Y. Fu et al., “AISHELL-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario,” in Proc. Interspeech, 2021.arxiv.org/abs/2104.03603
  20. [20]F. Yu et al., “Summary on the ICASSP 2022 multi-channel multi-party meeting transcription grand challenge,” in Proc. ICASSP, arXiv:2202.03647.arxiv.org/abs/2202.03647
  21. [21]J. Carletta et al., “The AMI meeting corpus: A pre-announcement,” in Proc. ICML, 2005.DOI
  22. [22]A. Canavan et al., “CALLHOME american english speech,” Linguistic Data Consortium, LDC97S42, 1997.DOI
  23. [23]N. Ryant et al., “The third DIHARD diarization challenge,” in Proc. Interspeech, 2021.arxiv.org/abs/2012.01477
  24. [24]M. V. Segbroeck et al., “DiPCo: Dinner party corpus,” 2019.arxiv.org/abs/1909.13447
  25. [25]S. B. Kalluri et al., “The second DISPLACE challenge: DI-arization of SPeaker and LAnguage in conversational environments,” in Proc. Interspeech, 2024.
  26. [26]Dhanya E et al., “Benchmarking speech systems for frontline health conversations: The DISPLACE-M challenge,” 2026.arxiv.org/abs/2603.02813
  27. [27]C. Cieri et al., “The fisher corpus: A resource for the next generations of speech-to-text,” in Proc. LREC, 2004.DOI
  28. [28]A. Janin et al., “The ICSI meeting corpus,” in Proc. ICASSP, 2003.DOI
  29. [29]A. Vinnikov et al., “NOTSOFAR-1 challenge: New datasets, baseline, and tasks for distant meeting transcription,” 2024.arxiv.org/abs/2401.08887
  30. [30]J. S. Chung et al., “Spot the conversation: Speaker diarisation in the wild,” in Proc. Interspeech, 2020.DOI
  31. [31]X. Li et al., “YODAS: YouTube-oriented dataset for audio and speech,” in Proc. ASRU, 2023.arxiv.org/abs/2406.00899
  32. [32]D. Raj et al., “COVER-Lap: A method for combining overlap-aware diarization outputs,” in Proc. SLT, 2021.arxiv.org/abs/2011.01997
  33. [33]V. Panayotov et al., “LibriSpeech: An ASR corpus based on public domain audio books,” in Proc. ICASSP, 2015.DOI
  34. [34]M. McAuliffe et al., “Montreal forced aligner: Trainable text-speech alignment using kaldi,” in Proc. Interspeech, 2017.DOI
  35. [35]D. Rekesh et al., “Fast conformer with linearly scalable attention for efficient speech recognition,” 2023.arxiv.org/abs/2305.05084
  36. [36]A. Vaswani et al., “Attention is all you need,” in Proc. NeurIPS, 2017.arxiv.org/abs/1706.03762
  37. [37]J. Su et al., “RoFormer: Enhanced transformer with rotary position embedding,” Neurocomputing, 2024.arxiv.org/abs/2104.09864
  38. [38]O. Kuchaiev et al., “NeMo: A toolkit for building AI applications using neural modules,” 2019.arxiv.org/abs/1909.09577
  39. [39]H. Huang et al., “NEST: Self-supervised fast conformer as all-purpose seasoning to speech processing tasks,” in Proc. ICASSP, arXiv:2408.13106.arxiv.org/abs/2408.13106
  40. [40]N. R. Koluguri et al., “Granary: Speech recognition and translation dataset in 25 european languages,” in Proc. Interspeech, 2025.arxiv.org/abs/2505.13404
  41. [41]D. S. Park et al., “Specaugment: A simple data augmentation method for automatic speech recognition,” in Proc. Interspeech, 2019.DOI
  42. [42]P. Pálka et al., “SphereVBx: Spherical variational bayes clustering for simplified EEND-VC diarization,” 2026.
  43. [43]Z. Chen et al., “Attention-based encoder-decoder end-to-end neural diarization with embedding enhancer,” IEEE/ACM Trans. Audio, Speech, Lang. Process., 2024.arxiv.org/abs/2309.06672
  44. [44]S. Horiguchi et al., “Can we really repurpose multi-speaker asr corpus for speaker diarization?,” in Proc. ASRU, 2025.arxiv.org/abs/2507.09226
  45. [45]T. von Neumann et al., “MeetEval: A toolkit for computation of word error rates for meeting transcription systems,” 2023.arxiv.org/abs/2307.11394
  46. [46]A. Polok et al., “DiCoW: Diarization-conditioned whisper for target speaker automatic speech recognition,” Computer speech & language, 2026.
  47. [47]A. Polok et al., “SE-DiCoW: Self-enrolled diarization-conditioned whisper,” in Proc. ICASSP, 2026.
  48. [48]W. Wang et al., “Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR,” in Proc. Interspeech, 2025.arxiv.org/abs/2506.22646
  49. [49]S. Niu et al., “The USTC-NERCSLIP systems for the CHiME-8 NOTSOFAR-1 challenge,” in Proc. CHiME, 2024.arxiv.org/abs/2409.02041

Paper details

vlnc:W_99bmakeh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents