WorksEngineering
Nemotron 3 Diarization: Advancing the Accuracy-Efficiency Frontier in Streaming Speaker Diarization
Abstract
Streaming Sortformer introduced an arrival-order speaker cache to maintain speaker identities across chunks in end-to-end streaming speaker diarization, with a single model supporting configurable latency profiles spanning offline operation to ultra-low-latency streaming. Nemotron 3 Diarization addresses the original model's four-speaker limitation by extending support to eight speakers while improving both accuracy and computational efficiency. The previous FastConformer-Transformer stack is streamlined into a pure Transformer encoder. A learned upsampling layer then maps the encoder's coarse 80-ms representations to 10-ms resolution, enabling fine-grained predictions. Training leverages large-scale simulated mixtures and adds activity and phantom-speaker losses to the standard Sortformer objectives. Despite doubling speaker capacity over its predecessor, the new architecture reduces training memory by a factor of 2.8, makes training faster, and delivers up to higher inference throughput. Nemotron 3 Diarization establishes a new state of the art in DER on DIHARD III and CALLHOME. When used to condition DiCoW, it also achieves state-of-the-art tcpWER on the single-microphone conditions of AMI and NOTSOFAR-1. Training code is open source, and model weights are released under the permissive OpenMDW-1.1 license.
Introduction
Speaker diarization remains a core capability for meeting transcription and multi-talker automatic speech recognition (ASR), even as speech large language models (LLMs) increasingly generate speaker-attributed transcripts directly. Recent systems and benchmarks explore unified prediction of lexical content, timestamps, and speaker labels [1, 2]. Yet these developments do not eliminate the need for explicit diarization: DM-ASR [3] uses diarization as an explicit structural prior for an LLM, while a recent comparative study [4] found unified models competitive in two-speaker conditions but less robust than modular pipelines as speaker count and overlap increase. Accurate diarization therefore remains valuable both as a standalone capability and as a component of systems built around speech LLMs.
End-to-end neural diarization (EEND) replaces multi-stage diarization pipelines with direct prediction of joint speaker activity and naturally represents overlapping speech [5]. Extensions based on encoder-decoder attractors accommodate an unknown number of speakers [6], while EEND with vector clustering combines overlap-aware local predictions with global embedding clustering to handle long recordings and arbitrary speaker inventories [7]. This local-to-global design underlies widely used systems such as Pyannote [8] and DiarizeN [9, 10]. By contrast, Sortformer [11] follows a fixed-capacity EEND approach, combining permutation-invariant loss (PIL) with arrival-time-sort (ATS) supervision to assign speakers to output channels according to their order of first arrival. Because densely annotated real conversations are scarce, EEND relies heavily on simulated mixtures. Large-scale pre-training improves performance [12], but its effectiveness depends on turn-taking statistics, source diversity, acoustic augmentation, and how simulated and real data are combined [13].
Most of this progress targets offline processing. Importantly, streaming diarization must preserve identities across chunks despite bounded right context, overlapping speech, and newly arriving speakers. Prior online systems address this problem through limited-context encoders, speaker-tracing buffers, or blockwise speaker detection and representation [14, 15, 16, 17]. Streaming Sortformer [18] adds an arrival-order speaker cache (AOSC) to maintain channel ordering across chunks, allowing one model to serve multiple latency profiles. Its released model, however, is limited to four speakers and produces predictions only every 80 ms. This leaves a gap for an open model that combines competitive accuracy in both offline and low-latency streaming settings, with support for more speakers, precise boundaries, and high computational efficiency.
These requirements are tightly coupled. Within the AOSC design, supporting more speakers requires a larger cache to retain sufficient evidence for each identity, which directly increases inference cost. Additional output channels also remain unused in typical recordings, increasing the risk of phantom-speaker activations. Meanwhile, running the encoder at a 10-ms frame step to improve boundary precision would greatly increase sequence length and attention cost. The model must also remain robust across latency profiles that provide different amounts of future context. These trade-offs motivate a design that supports more speakers and finer temporal resolution without sacrificing streaming efficiency.
This work introduces Nemotron 3 Diarization1, an open-weight model addressing these constraints. Our contributions are threefold. First, we redesign the architecture to efficiently increase speaker capacity and output temporal resolution. Second, we introduce an activity loss that models overall speech activity and a phantom loss that suppresses false activations in unused speaker channels. We pair these objectives with mixed-data and variable-context training to support both offline and streaming operation. Third, we analyze these choices through ablations of the backbone, initialization, temporal resolution, objectives, augmentations, and training data. We evaluate diarization accuracy across latencies and benchmark training and inference efficiency. We also assess downstream speaker-attributed ASR in offline and streaming settings.
Nemotron 3 Diarization
Training data
Nemotron 3 Diarization is trained on approximately 5,000 h each of human-annotated and pseudo-labeled real recordings, together with around 82,600 h of simulated multi-talker mixtures.
The real multi-domain training data include the following public corpora: AISHELL-4 [19], AliMeeting [20], AMI [21], CALLHOME Part 1 [22], DIHARD III Development [23], DiPCo [24], DISPLACE-2024 [25], DISPLACE-M [26], Fisher English [27], ICSI [28], NOTSOFAR-1 [29] (NSF1), and VoxConverse-v0.3 [30]. Additionally, a licensed David AI{^2} dataset of around 1,000 hours of 3–4-speaker English conversations is used. We also include a 5,000-hour multi-talker subset of YODAS-v2 {citep}[31] pseudo-labeled using an ensemble of nine Sortformers combined via DOVER-Lap {citep}[32]`.
The simulated mixtures (1–8 speakers, 180–200 s per mixture) were created using the FastMSS{^3} toolkit following the *Mind the Gap* protocol from {citep}[13]. The simulated data comprise 6,694 h from LibriSpeech {citep}[33], 6,743 h from AMI headsets, 6,745 h from AliMeeting headsets, 6,755 h from Fisher English, 36,458 h from around 22,000 h of licensed David AI English audio, and 19,216 h from around 2,000 h of licensed David AI multilingual audio in 21 languages. Notably, simulations use tight speaker-activity regions obtained with the Montreal Forced Aligner {citep}[34]`.
Architecture
Nemotron 3 Diarization builds on the Streaming Sortformer [18] paradigm with several important changes. First, the backbone is streamlined from a 17-layer FastConformer [35] followed by an 18-layer Transformer [36] to a single 31-layer Transformer with rotary positional embeddings (RoPE) [37]. This change is motivated by both higher computational efficiency and better scalability of a pure Transformer. This also makes the model more compact, reducing the total number of parameters from 117M to 99M. Input features are downsampled eightfold through feature stacking. The Transformer encoder therefore retains the FastConformer’s 80-ms frame step.
Second, we introduce a learned one-dimensional subpixel convolution on top of the encoder. This layer upsamples the encoder outputs by a factor of eight, enabling accurate detection of short turns and precise boundaries while preserving the efficient 80-ms encoder frame step. Specifically, a width-3 convolution maps each -dimensional encoder state to channels, which are rearranged into eight consecutive -dimensional states. The encoder therefore remains at 80-ms resolution, while the output heads operate at an effective 10-ms resolution with almost no computational overhead.
Finally, AOSC capacity is increased from 188 to 264 frames to accommodate the twofold increase in supported speaker count. The mean silence embedding used to separate speakers in AOSC is replaced with a learnable embedding for greater robustness.
Training objectives
Alongside the standard Sortformer PIL and ATS losses, which use the best target-speaker permutation and arrival-time-ordered targets, respectively, Nemotron 3 Diarization introduces two auxiliary losses: the activity loss and the phantom loss.
The activity loss is a cross-entropy loss over three permutation-free frame activity classes: silence, single-speaker speech, and overlapping speech. A separate auxiliary head predicts these classes from the encoder representations. This auxiliary task regularizes training by supervising overall speech activity without requiring speaker assignment.
The phantom loss addresses speaker overcounting caused by high-confidence activity predictions (“phantom speakers”) in channels empty under both the PIL and ATS alignments. Let be the speech posterior and sg denote stop-gradient. For each empty channel, contains valid frames satisfying ; otherwise, . Each selected frame incurs the inactive-label negative log-likelihood . These losses are pooled as
We set , and , and define the contribution as zero when . This normalized log-mean-exp pooling emphasizes the largest false activations without scaling directly with the number of selected frames.
Model training and augmentations
Training is performed using the NVIDIA NeMo Speech Framework{^4} {citep}[38]` on 64 NVIDIA A100-SXM4-80GB GPUs in two stages.
First, offline Sortformer training [11] is performed on simulated mixtures only, with the encoder initialized from a self-supervised NEST [39] Transformer pretrained on a 900,000-hour subset of Granary-v2, an expanded version of the Granary corpus [40]. Training uses random 60-second chunks drawn on the fly from full recordings. The PIL, ATS, activity, and phantom loss weights are 0.5, 0.5, 0.5, and 0, respectively. The offline stage operates at the native 80-ms output resolution without the upsampling layer.
Second, the model is fine-tuned for streaming with AOSC [18] enabled, using both real and simulated data. The 105-second training samples are constructed on the fly, with two chunks from different parts of a source recording concatenated with probability 0.5; sampling favors examples containing more active speakers. The PIL, ATS, activity, and phantom loss weights are 0.75, 0.25, 0.5, and 0.1, respectively. The upsampling layer is introduced at this stage, initialized to repeat each encoder state across eight output positions, and jointly trained with the rest of the model using 10-ms targets.
Additionally, to prevent the model from relying on future context during streaming inference, each batch receives a block-causal tail with probability 0.5. This modifies the self-attention mask to contain a bidirectional prefix followed by a randomly sampled tail. The tail is divided into blocks of one to eight frames: attention is bidirectional within each block, while each block may attend to the prefix and earlier blocks but not to future blocks. Full-context batches and this family of masks jointly expose one checkpoint to offline and multiple streaming contexts.
Finally, both training stages apply SpecAugment [41] regularization with two frequency masks spanning up to 24 feature bins each. The detailed training parameters used are given in the publicly available offline{^5} and streaming{^6} configuration files in the NeMo Speech repository.
Table 1. DER (%) ↓ in offline and streaming conditions across evaluation sets. DH3, CH, Ali and NSF1 denote DIHARD III Eval, CALLHOME Part 2, AliMeeting and NOTSOFAR-1, respectively. All overlap is scored; collar tolerance is 0.25 s for CH and 0 s for the other sets. Best offline results are bold.
| System | Latency | DH3 | CH | Ali-Near | Ali-Far | AMI-MHM | AMI-SDM | NSF1-MHM | NSF1-SC |
| DiariZen | offline | 14.5 | 14.68 | 16.78 | 10.8* | 15.56 | 13.9* | 14.20 | 16.7* |
| Sortformer-v2.1 | 30.4 s | 19.09 | 10.32 | 11.57 | 13.69 | 15.81 | 21.42 | 21.77 | 30.49 |
| Nemotron 3 Diarization | 30.4 s | 12.73 | 9.10 | 6.40 | 10.47 | 9.25 | 11.14 | 6.77 | 11.00 |
| Sortformer-v2.1 | 1.04 s | 19.60 | 11.31 | 12.47 | 15.58 | 16.36 | 21.73 | 22.12 | 31.81 |
| Nemotron 3 Diarization | 1.04 s | 13.18 | 10.29 | 6.59 | 10.80 | 9.48 | 12.80 | 7.70 | 12.77 |
| Sortformer-v2.1 | 0.32 s | 19.85 | 12.67 | 13.68 | 16.85 | 17.77 | 23.89 | 23.37 | 32.95 |
| Nemotron 3 Diarization | 0.32 s | 13.55 | 11.32 | 7.19 | 11.60 | 10.05 | 12.95 | 8.65 | 14.53 |
| SOTA 09/2026 | offline | 14.1 [42] | 10.08 [43] | 8.60* [12] | 10.6* [42] | 11.06* [12] | 13.7* [42] | — | 16.5* [42] |
Table 1.
*Reported using the original/default reference annotations rather than the force-aligned reference annotations used for the corresponding results in this work; these values are therefore not directly comparable with ours.
Results
Diarization accuracy evaluation
The evaluation contains the following datasets: DIHARD III evaluation [23] (259 recordings, 1–9 speakers per recording), CALLHOME Part 2 (250 recordings, 2–6 speakers), AliMeeting [20] near and far microphones (20 recordings, 2–4 speakers), AMI [21] Mix of Headset Microphones (MHM) and Single Distant Microphone (SDM) conditions (16 recordings, 3–4 speakers), and NOTSOFAR-1 [29] MHM and Single-Channel (SC) conditions (160 recordings, 3–7 speakers). All overlapped speech is scored. DER uses no forgiveness collar except for CALLHOME, where the conventional 0.25-s collar is used. It is important to note that we use forced-alignment-based reference labels for AMI, AliMeeting7 and NOTSOFAR-18 instead of the original segment-level annotations that label substantial within-segment silences as speech, thereby overestimating reference speaker activity [44].
We consider two open-source baselines: the previous four-speaker Streaming Sortformer-v2.19 [18] and DiariZen10 [9, 10], the strongest offline system. Table 1 reports DER at the three input-buffer latencies of 30.4, 1.04, and 0.32 s used in the original Streaming Sortformer study [18]. Reducing the buffer from 30.4 to 0.32 s increases DER by 0.79–3.53 percentage points across the evaluation sets, thus demonstrating graceful degradation. The same checkpoint can operate with input buffers as short as 0.08 s, although accuracy degrades further; we omit this regime to retain direct comparability with the previous work. At all three reported latencies, Nemotron 3 Diarization outperforms Sortformer-v2.1 across every evaluation set and the offline DiariZen baseline on every directly comparable condition. This establishes a new state of the art (SOTA) on DIHARD III and CALLHOME benchmarks. Although we believe our results on AMI, AliMeeting, and NOTSOFAR-1 are also SOTA, we do not claim this formally because the starred results use different ground-truth labels and are therefore not directly comparable with ours.
Table 2. Inference RTFx ↑ on an RTX PRO 5000 Blackwell GPU using bf16; “comp.” denotes dynamic torch.compile. The best result for each batch size and latency is bold.
| Model | Latency | Batch 1 eager | Batch 1 comp. | Batch 32 eager | Batch 32 comp. |
| DiariZen | offline | 20 | — | 44 | — |
| Sortformer-v2.1 | 30.4 s | 874 | 1468 | 3204 | 2619 |
| Nemotron 3 Diarization | 30.4 s | 1340 | 4385 | 12196 | 15113 |
| Sortformer-v2.1 | 1.04 s | 16 | 42 | 193 | 136 |
| Nemotron 3 Diarization | 1.04 s | 38 | 164 | 581 | 865 |
| Sortformer-v2.1 | 0.32 s | 8 | 21 | 101 | 76 |
| Nemotron 3 Diarization | 0.32 s | 12.5 | 54 | 199 | 292 |
Table 2.
Training and inference efficiency
During training on 64 NVIDIA A100 GPUs in bf16-mixed precision, the streamlined Transformer reduces memory consumption by a factor of 2.8 and iteration time by a factor of 2.2 relative to an otherwise identical model with the legacy FastConformer encoder.
Table 2 compares inference speed on one NVIDIA RTX PRO 5000 Blackwell GPU in bf16 precision. Here, Real-Time Factor Speedup (RTFx) defined as audio duration divided by wall-clock processing time is calculated with batch sizes of 1 and 32 on a large set of around 900 s long recordings. Compiled measurements exclude the one-time graph compilation overhead and are collected after compilation and warm-up. It reveals massive inference speed improvements over Sortformer-v2.1 with legacy FastConformer [35] encoder, despite the twofold increase in speaker capacity. Notably, even with the ultra-low input buffer latency of 0.32 s, the model is fast enough to support multiple parallel sessions on a single GPU.
Ablations
To assess the contribution of each component, Table 3 reports DER and its absolute change relative to Nemotron 3 Diarization at offline (30.4 s) and ultra-low (0.32 s) input buffer latency. All recordings from the constituent evaluation sets are scored jointly as one compound evaluation set with a zero collar. Each ablation is represented by a single training run using the same optimization and eval-
| System | 30.4 s DER | 30.4 s | 0.32 s DER | 0.32 s |
| Nemotron 3 Diarization | 10.42 | – | 12.06 | – |
| FastConformer enc. | 11.36 | +0.94 | 13.33 | +1.27 |
| no NEST initialization | 12.05 | +1.63 | 14.28 | +2.22 |
| downsampled (80 ms) | 11.09 | +0.67 | 12.67 | +0.61 |
| no upsampling (80 ms) | 12.69 | +2.27 | 14.09 | +2.03 |
| no phantom loss | 10.41 | -0.01 | 12.15 | +0.09 |
| no activity loss | 11.42 | +1.00 | 13.03 | +0.97 |
| no SpecAugment | 10.88 | +0.46 | 12.47 | +0.41 |
| no block-causal tail | 10.50 | +0.08 | 12.36 | +0.30 |
| no simulated data | 13.40 | +2.98 | 15.21 | +3.15 |
| no David AI data | 11.19 | +0.77 | 12.83 | +0.77 |
| no YODAS data | 10.81 | +0.39 | 12.38 | +0.32 |
Table 3. Ablations on the compound evaluation set: globally pooled, zero-collar DER (%) at input-buffer latencies of 30.4 s and 0.32 s. is the absolute DER change; positive values indicate degradation.
Table 1. DER (%) ↓ in offline and streaming conditions across evaluation sets. DH3, CH, Ali and NSF1 denote DIHARD III Eval, CALLHOME Part 2, AliMeeting and NOTSOFAR-1, respectively. All overlap is scored; collar tolerance is 0.25 s for CH and 0 s for the other sets. Best offline results are bold.
| System | Latency | DH3 | CH | Ali-Near | Ali-Far | AMI-MHM | AMI-SDM | NSF1-MHM | NSF1-SC |
| DiariZen | offline | 14.5 | 14.68 | 16.78 | 10.8* | 15.56 | 13.9* | 14.20 | 16.7* |
| Sortformer-v2.1 | 30.4 s | 19.09 | 10.32 | 11.57 | 13.69 | 15.81 | 21.42 | 21.77 | 30.49 |
| Nemotron 3 Diarization | 30.4 s | 12.73 | 9.10 | 6.40 | 10.47 | 9.25 | 11.14 | 6.77 | 11.00 |
| Sortformer-v2.1 | 1.04 s | 19.60 | 11.31 | 12.47 | 15.58 | 16.36 | 21.73 | 22.12 | 31.81 |
| Nemotron 3 Diarization | 1.04 s | 13.18 | 10.29 | 6.59 | 10.80 | 9.48 | 12.80 | 7.70 | 12.77 |
| Sortformer-v2.1 | 0.32 s | 19.85 | 12.67 | 13.68 | 16.85 | 17.77 | 23.89 | 23.37 | 32.95 |
| Nemotron 3 Diarization | 0.32 s | 13.55 | 11.32 | 7.19 | 11.60 | 10.05 | 12.95 | 8.65 | 14.53 |
| SOTA 09/2026 | offline | 14.1 [42] | 10.08 [43] | 8.60* [12] | 10.6* [42] | 11.06* [12] | 13.7* [42] | — | 16.5* [42] |
Table 1 (PDF p. 3).
*Reported using the original/default reference annotations rather than the force-aligned reference annotations used for the corresponding results in this work; these values are therefore not directly comparable with ours.
Results
Diarization accuracy evaluation
The evaluation contains the following datasets: DIHARD III evaluation [23] (259 recordings, 1–9 speakers per recording), CALLHOME Part 2 (250 recordings, 2–6 speakers), AliMeeting [20] near and far microphones (20 recordings, 2–4 speakers), AMI [21] Mix of Headset Microphones (MHM) and Single Distant Microphone (SDM) conditions (16 recordings, 3–4 speakers), and NOTSOFAR-1 [29] MHM and Single-Channel (SC) conditions (160 recordings, 3–7 speakers). All overlapped speech is scored. DER uses no forgiveness collar except for CALLHOME, where the conventional 0.25-s collar is used. It is important to note that we use forced-alignment-based reference labels for AMI, AliMeeting7 and NOTSOFAR-18 instead of the original segment-level annotations that label substantial within-segment silences as speech, thereby overestimating reference speaker activity [44].
We consider two open-source baselines: the previous four-speaker Streaming Sortformer-v2.19 [18] and DiariZen10 [9, 10], the strongest offline system. Table 1 reports DER at the three input-buffer latencies of 30.4, 1.04, and 0.32 s used in the original Streaming Sortformer study [18]. Reducing the buffer from 30.4 to 0.32 s increases DER by 0.79–3.53 percentage points across the evaluation sets, thus demonstrating graceful degradation. The same checkpoint can operate with input buffers as short as 0.08 s, although accuracy degrades further; we omit this regime to retain direct comparability with the previous work. At all three reported latencies, Nemotron 3 Diarization outperforms Sortformer-v2.1 across every evaluation set and the offline DiariZen baseline on every directly comparable condition. This establishes a new state of the art (SOTA) on DIHARD III and CALLHOME benchmarks. Although we believe our results on AMI, AliMeeting, and NOTSOFAR-1 are also SOTA, we do not claim this formally because the starred results use different ground-truth labels and are therefore not directly comparable with ours.
Table 2. Inference RTFx ↑ on an RTX PRO 5000 Blackwell GPU using bf16; “comp.” denotes dynamic torch.compile. The best result for each batch size and latency is bold.
| Model | Latency | Batch 1 eager | Batch 1 comp. | Batch 32 eager | Batch 32 comp. |
| DiariZen | offline | 20 | — | 44 | — |
| Sortformer-v2.1 | 30.4 s | 874 | 1468 | 3204 | 2619 |
| Nemotron 3 Diarization | 30.4 s | 1340 | 4385 | 12196 | 15113 |
| Sortformer-v2.1 | 1.04 s | 16 | 42 | 193 | 136 |
| Nemotron 3 Diarization | 1.04 s | 38 | 164 | 581 | 865 |
| Sortformer-v2.1 | 0.32 s | 8 | 21 | 101 | 76 |
| Nemotron 3 Diarization | 0.32 s | 12.5 | 54 | 199 | 292 |
Table 2 (PDF p. 3).
Training and inference efficiency
During training on 64 NVIDIA A100 GPUs in bf16-mixed precision, the streamlined Transformer reduces memory consumption by a factor of 2.8 and iteration time by a factor of 2.2 relative to an otherwise identical model with the legacy FastConformer encoder.
Table 2 compares inference speed on one NVIDIA RTX PRO 5000 Blackwell GPU in bf16 precision. Here, Real-Time Factor Speedup (RTFx) defined as audio duration divided by wall-clock processing time is calculated with batch sizes of 1 and 32 on a large set of around 900 s long recordings. Compiled measurements exclude the one-time graph compilation overhead and are collected after compilation and warm-up. It reveals massive inference speed improvements over Sortformer-v2.1 with legacy FastConformer [35] encoder, despite the twofold increase in speaker capacity. Notably, even with the ultra-low input buffer latency of 0.32 s, the model is fast enough to support multiple parallel sessions on a single GPU.
Ablations
To assess the contribution of each component, Table 3 reports DER and its absolute change relative to Nemotron 3 Diarization at offline (30.4 s) and ultra-low (0.32 s) input buffer latency. All recordings from the constituent evaluation sets are scored jointly as one compound evaluation set with a zero collar. Each ablation is represented by a single training run using the same optimization and evaluation settings. We therefore report observed effect sizes rather than statistical significance and interpret small differences cautiously. We emphasize effects that are consistent across both latency profiles or appear in the metric directly targeted by the component.
| Diarization | AMI MHM | AMI SDM | NSF1 MHM | NSF1 SC |
| Streaming: Multitalker-Parakeet-v1 (1.12 s latency) | ||||
| Ground truth (oracle) | 12.66 | 17.79 | 16.86 | 26.34 |
| Softformer-v2.1 | 18.64 | 34.13 | 35.59 | 51.35 |
| Nemotron 3 Diarization (hard) | 13.84 | 21.18 | 18.68 | 31.65 |
| Offline: DiCoW-v3.3-large (greedy decoding) | ||||
| Ground truth (oracle) | 11.72 | 15.27 | 8.78 | 15.45 |
| DiarZen | 15.17 | 19.32 | 18.57 | 25.63 |
| Softformer-v2.1 | 18.17 | 30.11 | 30.14 | 42.56 |
| Nemotron 3 Diarization (hard) | 12.65 | 17.19 | 10.47 | 19.23 |
| Nemotron 3 Diarization (soft) | 12.35 | 17.06 | 10.30 | 18.95 |
| SOTA 09/2026 | 15.2 [47] | 18.5 [47] | N/A | 23.63 [49] |
Table 4. Downstream tcpWER (%) for streaming and offline Multi-Talker ASR. Hard and soft denote segment-level and posterior-based diarization conditioning, respectively. Bold indicates the best non-oracle result.
The pure Transformer backbone outperforms the legacy FastConformer. This may reflect better scaling with the available data or better compatibility with AOSC: cached speaker blocks do not form a continuous temporal sequence, so local convolutions can mix unrelated states across artificial block boundaries. The substantial degradation without NEST initialization highlights the importance of broad self-supervised pretraining, suggesting that the diarization data are insufficient to learn equally robust acoustic representations from scratch.
Removing high-resolution upsampling degrades DER considerably more than downsampling the 10-ms predictions to 80 ms before scoring. This indicates that learning with high-resolution outputs provides benefits beyond finer evaluation granularity.
Removing the phantom loss leaves DER nearly unchanged but increases speaker-count MAE from 0.157 to 0.222 offline and from 0.270 to 0.491 at 0.32-s latency, confirming its role in reducing overcounting. Removing the activity loss or SpecAugment also degrades DER at both latency profiles. The block-causal-tail augmentation has little effect at 30.4 s but provides a clearer benefit at 0.32 s.
Finally, training without simulated data produces the largest degradation, underscoring the importance of simulated mixtures. David AI and pseudo-labeled YODAS data also contribute, as excluding either degrades DER at both latency profiles.
Downstream multi-talker ASR performance
As already highlighted in subsection 3.1, DER is a highly annotation-style-dependent metric, which makes it hard to compare diarization systems trained on different labels directly. To overcome this limitation, we conduct a comparison of our Nemotron 3 Diarization with the baseline systems in a downstream multi-talker automatic speech recognition (MT-ASR) task. Unlike DER, time-constrained minimum-permutation Word Error Rate (tcpWER) operates on speaker-attributed ASR hypotheses rather than the diarization labels used to score frame-level errors. We calculate tcpWER using the meeteval package [45] with the default collar tolerance of 5 s and the chime8 text normalizer.
Table 4 therefore evaluates whether the diarization improvements transfer across reference-annotation protocols and two different MT-ASR models: a state-of-the-art offline DiCoW-v3.3-large11 [46, 47], and a streaming Multitalker-Parakeet-v112 [48] operating at default 1.12 s input buffer latency.
DiCoW conditioned on segment-level (hard) predictions from Nemotron 3 Diarization substantially outperforms both baselines, while posterior-based (soft) conditioning further reduces tcpWER in all four conditions. To our knowledge, this is the first consistent soft-over-hard gain with DiCoW. The resulting system also establishes new state-of-the-art tcpWER on both AMI and NOTSOFAR-1. In particular, on the NOTSOFAR-1 single-channel condition, the resulting system surpasses USTC-NERCSLIP sys2 [49] — the winner of the CHiME-8 NOTSOFAR-1 Challenge [29].13
With streaming Multitalker Parakeet, Nemotron 3 Diarization also substantially reduces tcpWER relative to the Softformer-v2.1 baseline. The consistent reductions in tcpWER across both ASR back-ends show that the diarization improvements transfer to downstream transcription rather than merely reflecting better agreement with the forced-aligned reference labels used for DER.
Conclusion
Nemotron 3 Diarization extends the Streaming Softformer framework from four to eight speakers while significantly improving both accuracy and throughput. Notably, it establishes a new state of the art on DIHARD III and CALLHOME in terms of DER, and on AMI and NOTSOFAR-1 in terms of tcpWER. Given that the model is released under the permissive OpenNMDW-1.1 license and is easily fine-tunable on target-domain data, we hope it sees wide adoption by both researchers and enterprises. Future work will further extend the supported speaker count, scale the model using expanded synthetic and pseudo-labeled corpora, and investigate direct integration into multimodal speech LLMs to enable seamless, end-to-end conversational reasoning.
References
References
- [1]B. Mu et al., “Summary on the multilingual conversational speech language model challenge: Datasets, tasks, baselines, and methods,” in Proc. ICASSP, 2026.arxiv.org/abs/2509.13785
- [2]Z. Peng et al., “VIBEVOICE-ASR technical report,” 2026.arxiv.org/abs/2601.18184
- [3]L. Li et al., “DM-ASR: Diarization-aware multi-speaker ASR with large language models,” 2026.arxiv.org/abs/2604.22467
- [4]N. Tawara et al., “Who spoke what when: evaluating spoken language models for conversational ASR with semantic and overlap-aware metrics,” in Proc. Interspeech, 2026.arxiv.org/abs/2603.22709
- [5]Y. Fujita et al., “End-to-end neural speaker diarization with permutation-free objectives,” in Proc. Interspeech, 2019.arxiv.org/abs/1909.05952
- [6]S. Horiguchi et al., “End-to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors,” in Proc. Interspeech, 2020.arxiv.org/abs/2005.09921
- [7]K. Kinoshita et al., “Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech,” in Proc. Interspeech, 2021.arxiv.org/abs/2105.09040
- [8]H. Bredin, “pyannote.audio 2.1 speaker diarization pipeline: Principle, benchmark, and recipe,” in Proc. Interspeech, 2023.DOI
- [9]J. Han et al., “Leveraging self-supervised learning for speaker diarization,” in Proc. ICASSP, arXiv:2409.09408.arxiv.org/abs/2409.09408
- [10]J. Han et al., “Efficient and generalizable speaker diarization via structured pruning of self-supervised models,” 2025.DOI
- [11]T. Park et al., “Sortformer: A novel approach for permutation-resolved speaker supervision in speech-to-text systems,” in Proc. ICML, 2025.arxiv.org/abs/2409.06656
- [12]S. J. Broughton et al., “Pushing the limits of end-to-end diarization,” in Proc. Interspeech, 2025.arxiv.org/abs/2509.14737
- [13]A. Polok et al., “Mind the gap: Impact of synthetic conversational data on multi-talker ASR and speaker diarization,” in Proc. Interspeech, 2026.arxiv.org/abs/2605.15442
- [14]E. Han et al., “BW-EDA-EEND: Streaming end-to-end neural speaker diarization for a variable number of speakers,” in Proc. ICASSP, 2021.DOI
- [15]Y. Xue et al., “Online streaming end-to-end neural diarization handling overlapping speech and flexible numbers of speakers,” in Proc. Interspeech, 2021.arxiv.org/abs/2101.08473
- [16]S. Horiguchi et al., “Online neural diarization of unlimited numbers of speakers using global and local attractors,” IEEE/ACM Trans. Audio, Speech, Lang. Process., 2022.DOI
- [17]M. Cheng et al., “Sequence-to-sequence neural diarization with automatic speaker detection and representation,” IEEE/ACM Trans. Audio, Speech, Lang. Process., 2025.arxiv.org/abs/2411.13849
- [18]I. Medennikov et al., “Streaming sortformer: Speaker cache-based online speaker diarization with arrival-time ordering,” in Proc. Interspeech, arXiv:2507.18446.arxiv.org/abs/2507.18446
- [19]Y. Fu et al., “AISHELL-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario,” in Proc. Interspeech, 2021.arxiv.org/abs/2104.03603
- [20]F. Yu et al., “Summary on the ICASSP 2022 multi-channel multi-party meeting transcription grand challenge,” in Proc. ICASSP, arXiv:2202.03647.arxiv.org/abs/2202.03647
- [21]J. Carletta et al., “The AMI meeting corpus: A pre-announcement,” in Proc. ICML, 2005.DOI
- [22]A. Canavan et al., “CALLHOME american english speech,” Linguistic Data Consortium, LDC97S42, 1997.DOI
- [23]N. Ryant et al., “The third DIHARD diarization challenge,” in Proc. Interspeech, 2021.arxiv.org/abs/2012.01477
- [24]M. V. Segbroeck et al., “DiPCo: Dinner party corpus,” 2019.arxiv.org/abs/1909.13447
- [25]S. B. Kalluri et al., “The second DISPLACE challenge: DI-arization of SPeaker and LAnguage in conversational environments,” in Proc. Interspeech, 2024.
- [26]Dhanya E et al., “Benchmarking speech systems for frontline health conversations: The DISPLACE-M challenge,” 2026.arxiv.org/abs/2603.02813
- [27]C. Cieri et al., “The fisher corpus: A resource for the next generations of speech-to-text,” in Proc. LREC, 2004.DOI
- [28]A. Janin et al., “The ICSI meeting corpus,” in Proc. ICASSP, 2003.DOI
- [29]A. Vinnikov et al., “NOTSOFAR-1 challenge: New datasets, baseline, and tasks for distant meeting transcription,” 2024.arxiv.org/abs/2401.08887
- [30]J. S. Chung et al., “Spot the conversation: Speaker diarisation in the wild,” in Proc. Interspeech, 2020.DOI
- [31]X. Li et al., “YODAS: YouTube-oriented dataset for audio and speech,” in Proc. ASRU, 2023.arxiv.org/abs/2406.00899
- [32]D. Raj et al., “COVER-Lap: A method for combining overlap-aware diarization outputs,” in Proc. SLT, 2021.arxiv.org/abs/2011.01997
- [33]V. Panayotov et al., “LibriSpeech: An ASR corpus based on public domain audio books,” in Proc. ICASSP, 2015.DOI
- [34]M. McAuliffe et al., “Montreal forced aligner: Trainable text-speech alignment using kaldi,” in Proc. Interspeech, 2017.DOI
- [35]D. Rekesh et al., “Fast conformer with linearly scalable attention for efficient speech recognition,” 2023.arxiv.org/abs/2305.05084
- [36]A. Vaswani et al., “Attention is all you need,” in Proc. NeurIPS, 2017.arxiv.org/abs/1706.03762
- [37]J. Su et al., “RoFormer: Enhanced transformer with rotary position embedding,” Neurocomputing, 2024.arxiv.org/abs/2104.09864
- [38]O. Kuchaiev et al., “NeMo: A toolkit for building AI applications using neural modules,” 2019.arxiv.org/abs/1909.09577
- [39]H. Huang et al., “NEST: Self-supervised fast conformer as all-purpose seasoning to speech processing tasks,” in Proc. ICASSP, arXiv:2408.13106.arxiv.org/abs/2408.13106
- [40]N. R. Koluguri et al., “Granary: Speech recognition and translation dataset in 25 european languages,” in Proc. Interspeech, 2025.arxiv.org/abs/2505.13404
- [41]D. S. Park et al., “Specaugment: A simple data augmentation method for automatic speech recognition,” in Proc. Interspeech, 2019.DOI
- [42]P. Pálka et al., “SphereVBx: Spherical variational bayes clustering for simplified EEND-VC diarization,” 2026.
- [43]Z. Chen et al., “Attention-based encoder-decoder end-to-end neural diarization with embedding enhancer,” IEEE/ACM Trans. Audio, Speech, Lang. Process., 2024.arxiv.org/abs/2309.06672
- [44]S. Horiguchi et al., “Can we really repurpose multi-speaker asr corpus for speaker diarization?,” in Proc. ASRU, 2025.arxiv.org/abs/2507.09226
- [45]T. von Neumann et al., “MeetEval: A toolkit for computation of word error rates for meeting transcription systems,” 2023.arxiv.org/abs/2307.11394
- [46]A. Polok et al., “DiCoW: Diarization-conditioned whisper for target speaker automatic speech recognition,” Computer speech & language, 2026.
- [47]A. Polok et al., “SE-DiCoW: Self-enrolled diarization-conditioned whisper,” in Proc. ICASSP, 2026.
- [48]W. Wang et al., “Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR,” in Proc. Interspeech, 2025.arxiv.org/abs/2506.22646
- [49]S. Niu et al., “The USTC-NERCSLIP systems for the CHiME-8 NOTSOFAR-1 challenge,” in Proc. CHiME, 2024.arxiv.org/abs/2409.02041