WorksEngineering
Learning Speaker-Activity-Conditioned WavLM Speaker Embeddings with Real Conversational Data
Abstract
End-to-end neural diarization with vector clustering (EEND-VC) is a strong framework for long-form conversational diarization, where speaker embeddings are used to link local speaker decisions across windows. Conventional EEND-VC systems extract these embeddings by cutting out regions assigned to each local speaker, preferably using only non-overlapped speech. This works well when enough clean target-speaker speech is available, but it discards useful speaker evidence in overlapped regions and becomes unreliable when a local speaker appears mostly in overlap. We propose a speaker-activity-conditioned embedding extractor that adapts a pretrained WavLM-MHFA speaker verification model to extract target-speaker embeddings directly from full conversational windows. Inspired by the recently proposed diarization-conditioned Whisper (DiCoW) model for target-speaker ASR, frame-level diarization-dependent transformations inject local speaker-activity information into intermediate WavLM representations while preserving the pretrained initialization. Experiments on multiple conversational diarization benchmarks show that fine-tuning with limited real meeting data reduces speaker-confusion errors and improves EEND-VC diarization performance. The results further indicate that real conversational data provides an effective alternative to large-scale simulated-mixture training for diarization-aware speaker embedding extraction.
Introduction
Speaker diarization, the task of determining who spoke when, remains challenging for long-form conversational recordings with overlapping speech, short speaker turns, background noise, and an unknown number of speakers. A successful family of systems for such recordings is end-to-end neural diarization with vector clustering (EEND-VC) [1, 2, 3]. In this framework, a local EEND model estimates frame-level speaker activity in short, typically overlapping windows. The speaker identities predicted in each window are only local, so speaker embeddings are extracted for the local speakers and clustered to assign consistent global speaker identities across the whole recording. Recent progress in local EEND models [4, 5, 6] and clustering backends [7, 8, 9, 10] has improved EEND-VC diarization, but reliable speaker embeddings remain essential for linking local speakers across windows. Poor embeddings lead directly to speaker-confusion errors in the final diarization output.
Most speaker embedding extractors used in EEND-VC are trained for speaker verification on segments that typically contain speech from a single speaker [11, 12, 13, 14]. EEND-VC systems therefore usually construct the embedding-extractor input to satisfy this single-speaker assumption. For each local speaker, the system cuts out and concatenates subsegments assigned to that speaker by the local EEND output, preferably using only non-overlapped speech when available and falling back to overlapped regions when no clean speech exists. This is practical and effective when enough clean speech is available for each local speaker. However, in conversational recordings, some speakers may have little or no non-overlapped speech within a local window. In such cases, the system either ignores useful target-speaker evidence present in overlapped speech or must extract embeddings from speech mixed with interfering speakers.
Recent work has started to address this limitation by conditioning speaker embedding extraction on diarization information [15, 16, 17]. These approaches show that speaker activity can help extract more useful embeddings for EEND-VC, especially when clean single-speaker regions are short or unavailable. However, prior systems typically train dedicated embedding models on simulated multi-speaker mixtures generated from VoxCeleb-style data [18]. This raises two practical questions: whether a strong pretrained speaker embedding extractor can be adapted without training a new diarization-conditioned model from scratch, and whether such adaptation can be done using a limited amount of real conversational data. Training on real meetings may expose the model to conversational effects that are difficult to reproduce faithfully in simulation.
In this paper, we propose a speaker-activity-conditioned embedding extractor for EEND-VC. Like prior diarization-conditioned approaches, the goal is to extract embeddings from the full local window rather than relying only on cut-out clean speech regions. Our approach differs in that we start from a pretrained speaker verification extractor and adapt it with lightweight conditioning layers. For each selected local speaker, the local EEND output is converted into target-specific speaker-activity masks indicating silence, target-speaker activity, non-target activity, and overlap. These masks condition the embedding extractor frame by frame, allowing the model to exploit target-speaker evidence in both overlapped and non-overlapped regions while still producing a single embedding for the selected speaker.
We instantiate this idea with a pretrained WavLM-MHFA speaker embedding extractor [13], which combines a WavLM self-supervised speech encoder [19] with multi-head factorized attentive (MHFA) pooling for speaker embedding extraction. Inspired by diarization-conditioned Whisper (DiCoW) for target-speaker ASR [20], we inject speaker-activity information into intermediate WavLM representations using frame-level diarization-dependent transformations. The added conditioning layers are initialized to preserve the pretrained model behavior before adaptation, allowing us to start from a strong speaker verification model rather than training a diarization-conditioned extractor from scratch.
Our main contribution is a speaker-activity-conditioned WavLM-MHFA extractor, SAC-WavLM, that adapts a pretrained speaker verification model for target-speaker embedding extraction from full
The work was supported by Ministry of Education, Youth and Sports of the Czech Republic (MoE) through the OP JAK project (ID:CZ.02.01.01/00/23_020/0008518). Computing on IT4I supercomputer was supported by MoE through the e-INFRA CZ (ID:90254). local EEND windows. We show that fine-tuning this model on a limited amount of real conversational data substantially reduces speaker-confusion errors and improves EEND-VC diarization performance, providing an effective alternative to large-scale simulated-mixture training. We also evaluate SAC-WavLM together with several modern embedding extractors using both agglomerative hierarchical clustering (AHC) and SphereVBx [10] as clustering backends.
Related Work
Guided Speaker Embeddings. Horiguchi et al. [15] introduced guided speaker embeddings (GSE), which use target- and non-target-speaker activity as auxiliary information to extract a target-speaker embedding from multi-speaker audio. Speaker activity is provided to the embedding extractor together with the acoustic features and is also used to mask attentive pooling, allowing the model to use overlapped speech while suppressing regions where the target speaker is inactive. The model is trained on simulated three-speaker mixtures generated from VoxCeleb data using oracle speaker activities. A later extension [16] mitigates non-target-speaker bias by modifying modules that rely on global statistics, improving robustness in both low- and high-overlap conditions. These works are closest to ours in motivation, but they train dedicated guided embedding models on large-scale simulated mixtures, whereas we adapt a pretrained WavLM-MHFA speaker verification model using limited real conversational data.
Diarization-Guided Multi-Speaker Embeddings. Kalda et al. [17] extended guided embedding extraction to model multiple speakers jointly. Instead of extracting one target-speaker embedding per forward pass, their model uses information from a local segmentation model to produce embeddings for multiple active speakers at once, improving efficiency and matching the multi-speaker nature of diarization. This direction is complementary to ours: it addresses the efficiency and joint-modeling limitation of target-speaker embedding extraction, while our work focuses on adapting a strong pretrained extractor to conversational embedding extraction without relying on large-scale simulated-mixture training.
Diarization-Conditioned Foundation Models. Our conditioning mechanism is inspired by diarization-conditioned Whisper (DiCoW) [20], which injects diarization information into intermediate Transformer layers of Whisper [21] for target-speaker ASR. We apply the same general idea to speaker embedding extraction: rather than changing the audio input, we condition intermediate representations of a pretrained model on speaker activity. This allows us to preserve the pretrained WavLM-MHFA initialization while adapting it to extract target-speaker embeddings from full local EEND windows.
Proposed Method
Our goal is to adapt a pretrained speaker embedding extractor, originally trained on single-speaker speech segments, to extract an embedding for a selected speaker from a full local EEND window. The proposed speaker-activity-conditioned WavLM-MHFA model, denoted SAC-WavLM, keeps the architecture and initialization of this extractor, but adds speaker-activity conditioning through frame-level diarization-dependent transformation (FDDT) blocks inserted into the WavLM frontend. An overview of the model is shown in Figure 1. Although this work instantiates the approach with WavLM-MHFA, the conditioning principle is not specific to WavLM and can be applied to other deep embedding extractors with intermediate frame-level representations.
Figure 1. Block diagram of the proposed SAC-WavLM model. It shows extraction of a target embedding from a conversational audio chunk of the recording. STNO stands for Silence, Target, Non-target, Overlap.
Pretrained WavLM-MHFA Extractor
We build on the WavLM-MHFA speaker embedding extractor [13]. This extractor combines a self-supervised pretrained WavLM frontend with an MHFA pooling backend, and the complete system is trained for speaker classification on single-speaker speech segments using the standard additive angular margin (AAM) loss. In Figure 1, this baseline corresponds to the same WavLM-MHFA architecture without the STNO-mask input and without the FDDT blocks.
The extractor processes the raw waveform using the WavLM frontend, consisting of a convolutional feature encoder followed by a stack of Transformer layers. The multi-head factorized attentive pooling (MHFA) backend aggregates information across both time and WavLM layers to produce a speaker embedding. Let denote the frame-level representation at the input of the -th WavLM layer, where is the feature dimension and is the number of frames. MHFA first forms key and value representations by learned weighted combinations of WavLM layers:
where , and are learnable layer weights, and and are linear projections. The attention weights over frames are computed using a learnable query matrix , whose columns correspond to attention heads:
where . The value representations are pooled separately for each head and then concatenated to form the final embedding:
where denotes vectorization, i.e., concatenation of the head-specific pooled vectors. This layer-wise pooling motivates conditioning the model inside WavLM: transformations applied to intermediate WavLM representations can influence the representations that MHFA later selects and aggregates.
Target-Specific STNO Masks
Because SAC-WavLM processes the full local EEND window but must output an embedding for one selected local speaker, we convert the local speaker-activity labels into a target-specific STNO conditioning mask, following DiCoW [20]. For each selected target speaker, every frame is assigned to one of four mutually exclusive classes:
Silence: no speaker is active,
Target: only the selected target speaker is active,
Non-target: one or more non-target speakers are active and the target speaker is inactive,
Overlap: the target speaker and at least one non-target speaker are active.
Thus, regardless of how many local speakers are present in the EEND window, SAC-WavLM always receives four mask streams. Their values are target-specific: selecting a different local speaker produces a different STNO mask for the same audio window. Let , , , and denote the four STNO mask values at frame . In this work, we use binary STNO masks, so exactly one of these values is one at each frame and the others are zero.
Speaker-Activity-Conditioned WavLM
Rather than modifying the waveform or concatenating speaker-activity features at the input, we inject the STNO conditioning into intermediate WavLM representations. This follows the DiCoW FDDT mechanism [20] and allows us to preserve the pretrained WavLM-MHFA extractor while adapting its internal representation flow for target-speaker embedding extraction.
For each selected WavLM Transformer layer , an FDDT block applies one of four STNO-class-dependent affine transformations to each frame-level input representation. Let denote the original input to the -th Transformer layer at frame . The FDDT block replaces it with the transformed representation
where the four affine transformations correspond to silence, target-speaker speech, non-target speech, and overlap, respectively. Since we use binary mutually exclusive STNO masks, Eq. (4) selects exactly one affine transformation for each frame. Following DiCoW, the transformation matrices are constrained to be diagonal in our implementation, keeping the number of additional parameters small. The transformed representation is passed to the corresponding Transformer layer. After all WavLM layers have been processed, the MHFA backend aggregates the resulting layer representations and produces a single embedding for the selected target speaker. The same audio window can be processed repeatedly with different target-specific STNO masks to obtain embeddings for different local speakers.
Initialization
A key design goal is to adapt the pretrained WavLM-MHFA extractor without disrupting its already useful speaker representation space. We therefore initialize SAC-WavLM from the baseline WavLM-MHFA model: all WavLM and MHFA parameters are copied from the pretrained extractor, and only the newly introduced FDDT parameters are initialized separately.
If all FDDT transformations are initialized as identities with zero biases, inserting the FDDT blocks leaves the pretrained extractor unchanged. Following DiCoW [20], we use this identity initialization for all FDDT blocks except the first one.
| Statistics \ Train Dataset | Real Dataset | Simulated Dataset |
| Source | AMI | NSF | VoxCeleb 2 dev |
| Total duration (hrs) | 80 | 54 | 1600 |
| #Total Speakers | 151 | 15 | 5527 |
| Recording duration (mins) | 30-60 | 6-8 | 2-55 |
| #Speakers per recording | 3-5 | 4-8 | 2-4 |
Table 1. Training data statistics for real and simulated datasets.
The first FDDT block, denoted FDDT 0 in Figure 1, uses suppressive initialization. Its silence and non-target transformations are initialized as scaled-down identities, reducing the acoustic information passed from frames that do not contain the target speaker, while target and overlap transformations are initialized as identities. The block is placed after the WavLM convolutional feature encoder but before positional embeddings are added, so that suppression affects acoustic features without modifying positional information. This gives the model a reasonable target-speaker behavior at initialization, while still allowing all transformations to be adapted during fine-tuning.
Training Objective
Speaker embedding extractors are commonly trained with the additive angular margin (AAM) loss. Let be the embedding extracted from the -th training example, let be its speaker label, and let denote the learnable centroid corresponding to the speaker . For the -th training example, the AAM loss is defined as
where is the angle between and centroid , is the scale, and is the angular margin. In standard AAM training, the centroids are learned jointly with the embedding extractor.
This standard setting is appropriate when many training speakers are available, as in VoxCeleb-based speaker verification training or training on simulated mixtures generated from VoxCeleb. However, when fine-tuning SAC-WavLM on real conversational datasets, only a limited number of speakers is available. Learning new centroids in this setting could encourage the model to overfit to those speakers instead of learning how to use the speaker-activity conditioning.
We therefore use fixed centroids when fine-tuning SAC-WavLM on real conversational data. For each training speaker, we compute a centroid by averaging embeddings extracted from clean single-speaker regions using the already trained embedding extractor. These centroids are then kept fixed during fine-tuning. This anchors SAC-WavLM to an existing speaker embedding space and reduces the risk of overfitting to the small real-data speaker set: the model is encouraged to produce embeddings compatible with the original extractor, while learning to use the full local window and the target-specific STNO conditioning.
Experiments
We evaluate on standard conversational diarization benchmarks: AMI [22], AISHELL-4 (AIS) [23], AliMeeting (AliM) [24], NOTSOFAR-1 (NSF) [25], MSDWld (MSD) [26], DIHARD3 full (DH3) [27], and VoxConverse (VoxC) [28]. For AMI and AliMeeting, we use the first channel of the far-field microphone array, while for NOTSOFAR-1, we follow the single-channel evaluation condition using beamformed recordings. We exclude RAMC [29] because we observed label noise in its training and test sets.
System performance is evaluated using diarization error rate (DER) without a forgiveness collar and considering overlapped speech using the dscore toolkit1. Hyperparameters are selected by minimizing DER on a compound development set, including the threshold and minimum cluster size for agglomerative hierarchical clustering (AHC), and the and parameters for SphereVBx [10].
Baseline EEND-VC System
All experiments use the DiariZen framework2 as an implementation of the EEND-VC pipeline described in § 1: a local EEND model predicts speaker activity in overlapping windows, embeddings are extracted for the local speakers, and clustering maps local speaker identities to global speaker identities. We keep the local EEND fixed and vary the speaker embedding extractor and/or clustering backend.
The recording is processed using 16 s local windows with a 1.6 s shift. The local EEND model is configured for up to four local speakers, all of which may be active in the same frame. For each local speaker, embeddings are extracted preferably from non-overlapped regions; overlapped regions are used only when no non-overlapped speech is available.
The baseline embedding extractor is the pretrained WavLM-MHFA model3, which consists of a WavLM Base+ encoder with 12 Transformer layers followed by an MHFA pooling backend with 64 heads that produces 256-dimensional embeddings. The model is trained on the VoxCeleb2 development set as discussed in [13]. Although DiariZen uses a relatively lightweight ResNet34 [30] architecture with approximately 6.7M parameters, about 14× fewer than WavLM-MHFA, we adopt WavLM-MHFA because its Transformer-based architecture enables straightforward integration of speaker-activity conditioning while maintaining a comparable inference cost. In addition to this baseline, we evaluate a WavLM-MHFA variant fine-tuned on the same real conversational data but without STNO conditioning or FDDT blocks referred to as WavLM-MHFA (Real), to separate the effect of domain fine-tuning from the proposed speaker-activity conditioning.
For clustering, we mainly use SphereVBx-PF [10], the parameter-free variant of SphereVBx. SphereVBx was recently shown to simplify and improve the DiariZen clustering stage by replacing the PLDA-based VBx backend with a spherical model better matched to modern length-normalized speaker embeddings. We also report results with AHC to compare with a more traditional clustering backend.
SAC-WavLM Training Configuration
During training, STNO masks are derived from ground-truth diarization labels. At evaluation time, we consider both oracle segmentation and the output of the DiariZen local EEND model, as described in the Results section.
Training on Real Conversational Data
To fine-tune SAC-WavLM on limited labeled conversational data, we use the training partitions of AMI and NOTSOFAR-1, which provide speaker identities that are consistent across recordings. We do not use the other evaluation corpora for training because they do not provide reliable speaker labels across sessions to construct global speaker identities. Together, AMI and NOTSOFAR-1 provide 134 hours of training data and 166 speakers, as summarized in Table I.
For fixed-centroid AAM training described in Section II-E, we first compute one centroid for each training speaker. We extract 10 s embeddings from single-speaker regions using the pretrained WavLM-MHFA extractor and average embeddings belonging to the same speaker. These centroids are kept fixed during SAC-WavLM fine-tuning to reduce overfitting in the limited-speaker setting.
The SAC-WavLM model is initialized from the pretrained WavLM-MHFA extractor. All FDDT transformations are initialized as identities, except FDDT 0, where the silence and non-target transformations are scaled by 0.5 as discussed in Section III-D. Training is performed in two stages. First, only the FDDT and MHFA parameters are optimized for up to five epochs with a learning rate of . Then, the full model is fine-tuned for up to ten epochs using learning rates of for the MHFA/FDDT backend and for WavLM. Each mini-batch contains 10 audio segments. For each segment, we use up to the four most talkative speakers, producing up to 40 STNO masks and corresponding speaker embeddings. The AAM scale is set to , and the margin is set to zero based on validation performance.
Training on Simulated Mixtures
To compare fine-tuning on limited real conversational data with training on large-scale simulated speech, we also follow the simulation strategy of [31] to generate conversational mixtures from VoxCeleb2 development data. This gives approximately ten times more training audio and over thirty times more speakers than the real conversational training set; see Table I.
The simulated-mixture variant is initialized in the same way as SAC-WavLM trained on real conversational data, starting from the pretrained WavLM-MHFA extractor with newly inserted FDDT blocks. Unlike fine-tuning on the limited real conversational data, simulated-mixture training uses standard AAM with learnable centroids. We first train only the centroids on non-overlapped single-speaker regions, using margin , scale , and a learning rate of while keeping the rest of the model frozen. We then train on simulated multi-speaker mixtures: first, the FDDT and MHFA backend are optimized together with the centroids using a learning rate of ; then, the full model is trained with learning rates of for the backend and for WavLM.
Fine-Tuning After Simulated-Mixture Training
We also evaluate a variant that is first trained on simulated mixtures and then fine-tuned on the real conversational data. After simulated-mixture training, we compute fixed centroids for the AMI and NOTSOFAR-1 training speakers using embeddings extracted by the simulated-mixture-trained model. These centroids are then frozen, and the full model is fine-tuned on real conversational data using learning rates of for the MHFA/FDDT backend and for WavLM, with scale and zero margin.
Results
DER decomposes into missed speech, false alarm speech, and speaker-confusion (SC) errors. Since the local EEND output is fixed in our comparisons, changing the embedding extractor or clustering backend cannot recover missed speech or remove false alarms; it can only change speaker-confusion errors by altering how local speakers are linked across windows. We therefore first report SC to isolate the effect of the speaker embeddings and clustering. This is also important because missed-speech and false-alarm errors are strongly affected by annotation boundary conventions: conversational transcripts are often optimized for ASR and may contain loose speech boundaries [32]. In contrast, SC more directly measures whether speech from one speaker is assigned to another speaker.
tbl:II reports SC across datasets for the WavLM-MHFA baseline and SAC-WavLM variants using SphereVBx clustering. The first half of the table uses oracle segmentation, where local speaker activity is taken from ground-truth labels. In this condition, missed-speech and false-alarm errors are zero, so DER equals SC. The table also reports macro-averages across datasets and relative improvements with respect to the WavLM-MHFA baseline. The AMI and NSF columns are especially relevant because these are the real conversational datasets used for fine-tuning.
The second row, WavLM-MHFA (Real), corresponds to fine-tuning the baseline WavLM-MHFA extractor on real conversational data without STNO conditioning; it gives almost no macro-average improvement over WavLM-MHFA (Baseline). This indicates that the improvement obtained with SAC-WavLM is not simply due to domain fine-tuning. In contrast, SAC-WavLM (Real), i.e., SAC-WavLM fine-tuned only on the limited real conversational data, reduces the macro-average SC from 2.42% to 1.66%, corresponding to a 31.5% relative improvement. Training on simulated mixtures also improves over the baseline, and SAC-WavLM (Sim. + Real), which is trained on simulated mixtures and then fine-tuned on real data, gives the best oracle-segmentation result, with a 44.4% relative improvement. This suggests that simulated mixtures can provide useful additional training diversity when the conditioning masks are accurate.
The second half of Table II reports results using estimated DiariZen local speaker activity. In this condition, the DiariZen output is used both as the local EEND decision in the EEND-VC pipeline and, for SAC-WavLM, to derive the target-specific STNO masks. SAC-WavLM (Real) gives the best macro-average SC, reducing it from 3.65% to 3.08% and giving a 15.7% relative improvement. The simulated variants remain competitive, but they no longer improve over SAC-WavLM (Real). This suggests that fine-tuning directly on real conversational data better matches the estimated DiariZen outputs used at test time.
| Method | FC | Segmentation | NSF | AMI | AliM | MSDW | DH3 | AIS | VoxC | Macro average | Relative improv. |
| WavLM-MHFA (Baseline) | ✗ | Oracle | 4.42 | 2.03 | 1.11 | 2.08 | 2.74 | 2.65 | 1.91 | 2.42 | – |
| WavLM-MHFA (Real) | ✓ | Oracle | 4.97 | 1.38 | 1.06 | 2.48 | 2.95 | 2.24 | 1.77 | 2.41 | 0.5 |
| SAC-WavLM (Real) | ✓ | Oracle | 2.09 | 1.58 | 0.28 | 1.86 | 2.20 | 1.81 | 1.79 | 1.66 | 31.5 |
| SAC-WavLM (Sim.) | ✗ | Oracle | 1.62 | 0.99 | 0.09 | 1.93 | 2.17 | 2.38 | 1.58 | 1.54 | 36.5 |
| SAC-WavLM (Sim. + Real) | ✓ | Oracle | 1.44 | 0.78 | <u>0.13</u> | 1.58 | 2.11 | <u>1.82</u> | 1.56 | 1.35 | 44.4 |
| WavLM-MHFA (Baseline) | ✗ | DiariZen | 7.23 | 3.53 | 1.95 | 4.94 | 2.88 | 2.73 | 2.29 | 3.65 | – |
| WavLM-MHFA (Real) | ✓ | DiariZen | 7.74 | 2.71 | 2.06 | 5.11 | 2.93 | 2.56 | 2.18 | 3.61 | 1.0 |
| SAC-WavLM (Real) | ✓ | DiariZen | 5.83 | <u>2.20</u> | 1.34 | 5.04 | <u>2.76</u> | 2.29 | <u>2.08</u> | 3.08 | 15.7 |
| SAC-WavLM (Sim.) | ✗ | DiariZen | 5.52 | 2.22 | 1.21 | 5.49 | 2.67 | 2.63 | 2.15 | 3.13 | 14.3 |
| SAC-WavLM (Sim. + Real) | ✓ | DiariZen | 6.05 | 2.08 | 1.21 | <u>5.25</u> | 2.78 | <u>2.49</u> | 2.07 | 3.13 | 14.3 |
Table II. Comparison of diarization speaker-confusion error (SC, %) on benchmark datasets using SphereVBx clustering. FC denotes fixed centroids used during the final training stage. Relative improvements are computed with respect to the WavLM-MHFA baseline. Bold and <u>underlined</u> values indicate the best and second-best results in each column.
Table III compares full DER across different embedding extractors and clustering backends using DiariZen local speaker activity. The oracle-clustering row gives a lower bound for each dataset by mapping DiariZen local speakers directly to ground-truth speaker labels, thereby removing clustering errors. The remaining gap from oracle clustering reflects the contribution of speaker-linking errors. All systems in this table use the same DiariZen local EEND output and the same clustering/post-processing pipeline; the main component being compared is the speaker embedding extractor.
| Method | Diarization conditioning | Clustering | AIS | AliM | AMI | DH3 | MSDW | NSF | VoxC | Macro average | Gap from oracle |
| Oracle clustering | – | – | 8.5 | <u>9.9</u> | 12.3 | 12.4 | 13.4 | 13.6 | 7.6 | 11.1 | – |
| WavLM-MHFA | ✗ | AHC | 10.4 | 12.5 | 15.3 | 14.9 | 18.3 | 22.6 | 10.1 | 14.9 | 3.8 |
| ResNet34 [30] | ✗ | AHC | 9.9 | 12.1 | 14.6 | 15.6 | 18.7 | 21.3 | 10.1 | 14.6 | 3.5 |
| CAM++ [14] | ✗ | AHC | 10.6 | 12.1 | 15.4 | 14.8 | 18.1 | 20.2 | 10.0 | 14.4 | 3.3 |
| GSE (CAM++) [16] | ✓ | AHC | 10.3 | 11.1 | 14.9 | 14.6 | 17.3 | 18.1 | 10.0 | 13.8 | 2.7 |
| SAC-WavLM (Real) | ✓ | AHC | 10.2 | 11.6 | 14.7 | 14.7 | 17.2 | 18.8 | 9.8 | <u>13.9</u> | 2.8 |
| WavLM-MHFA | ✗ | SphereVBx | 10.4 | 10.8 | 14.5 | 14.1 | 15.9 | 17.3 | 9.3 | 13.2 | 2.1 |
| ResNet34 [30] | ✗ | SphereVBx | 9.8 | 10.7 | 13.5 | 14.1 | 15.4 | 16.8 | 9.3 | 12.8 | 1.7 |
| CAM++ [14] | ✗ | SphereVBx | 10.3 | 10.5 | 13.6 | 14.1 | 15.9 | 16.9 | 9.1 | 12.9 | 1.8 |
| GSE (CAM++) [16] | ✓ | SphereVBx | 9.6 | 10.1 | 13.1 | 14.0 | 15.3 | 15.4 | 9.0 | 12.4 | 1.3 |
| SAC-WavLM (Real) | ✓ | SphereVBx | 10.0 | 10.2 | 13.2 | 14.0 | 16.0 | 15.9 | 9.1 | <u>12.6</u> | 1.5 |
Table III. Comparison of DER (%) with AHC and SphereVBx clustering on benchmark datasets using DiariZen segmentation. Bold and <u>underlined</u> values indicate the best and second-best results in each column.
The table includes both conventional speaker verification embeddings and diarization-conditioned embeddings. WavLM-MHFA is the baseline extractor used in our system. ResNet34 [30] and CAM++ [14] are additional single-speaker embedding extractors trained on VoxCeleb-style speaker verification data. GSE (CAM++) [16] is included as a previously published diarization-guided embedding extractor built on top of CAM++. This makes CAM++ the natural non-conditioned baseline for GSE, while WavLM-MHFA is the non-conditioned baseline for SAC-WavLM.
With AHC clustering, GSE (CAM++) gives the best macro-average DER in this comparison, reducing DER from 14.4% for CAM++ to 13.8%. SAC-WavLM (Real) reduces DER from 14.9% for WavLM-MHFA to 13.9%. Thus, both diarization-conditioned embedding extractors provide clear gains over their corresponding non-conditioned baselines. SAC-WavLM obtains these gains by adapting a pretrained WavLM-MHFA extractor using only limited real conversational data, whereas GSE relies on large-scale simulated-mixture training.
With SphereVBx clustering, all embedding extractors improve compared with AHC, confirming that the clustering backend remains an important part of the EEND-VC pipeline. GSE (CAM++) gives the best macro-average DER in this comparison, while SAC-WavLM (Real) gives the second-best result. Compared with their corresponding non-conditioned baselines, GSE reduces DER from 12.9% to 12.4%, while SAC-WavLM reduces DER from 13.2% to 12.6%. SAC-WavLM also reduces the gap from oracle clustering from 2.1% to 1.5%. These results show that speaker-activity conditioning improves the embeddings used for linking local speakers, and that adapting a pretrained speaker verification model with limited real conversational data is an effective alternative to training a diarization-guided extractor on large simulated mixtures.
Analysis
To evaluate the discriminative quality of the extracted speaker embeddings independent of clustering, we compute the equal error rate (EER) using ground-truth speaker segmentation. Test-enrollment trials are formed within each recording to reflect the diarization setting. We consider two sets of verification trials.
Figure 2 reports the EER computed using test embeddings extracted from overlapped speech segments containing less than 1.6 s of non-overlapped target speech. Enrollment embeddings are extracted from segments with at least 1.6 s of non-overlapped target speech. This evaluation measures the ability of the embedding extractor to associate challenging overlapped speech with the corresponding clean speaker representation. As expected, speaker-activity conditioning provides the largest improvements on datasets with higher overlap ratios. However, these gains do not translate proportionally to DER, since the downstream EEND-VC pipeline is also affected by inaccuracies in the local DiariZen speaker-activity estimates.

Figure 2. EER performance across datasets for overlapped (< 1.6 s target single speaker segments) vs non-overlapped ( s target single speaker segments) speaker embeddings with ground-truth segmentation.
Figure 3 reports the EER using only non-overlapped speaker embeddings for both enrollment and test. The proposed method continues to

Figure 3. EER performance across datasets for pairs of non-overlapped (extracted from s target single speaker segments) speaker embeddings with ground-truth segmentation.
outperform the baseline, indicating that speaker-activity conditioning improves speaker representations even for clean speech. We attribute this improvement to training on both overlapped and non-overlapped conversational regions, whereas the baseline is trained using only single-speaker segments. This advantage is particularly evident on datasets with substantial overlap, such as NSF, AMI, and AliMeeting.
Figure 4 presents the detection error tradeoff (DET) curves on the NOTSOFAR-1 test set for overlapped versus non-overlapped speaker verification trials. Unlike EER, which summarizes performance at a single operating point, the DET curves illustrate the trade-off between miss and false-alarm rates across different decision thresholds. The plot is zoomed around the EER region for clarity. The proposed speaker-activity-conditioned models consistently outperform the baseline across the operating range, demonstrating that the observed improvements are not limited to a specific threshold. Furthermore, the SAC-WavLM variants achieve performance comparable to, and occasionally better than, the GSE approach.

Figure 4. Detection Error Tradeoff (DET) curve on NOTSOFAR-1 dataset.
Conclusion
We proposed SAC-WavLM, a speaker-activity-conditioned embedding extractor for EEND-VC diarization. The method adapts a pretrained WavLM-MHFA speaker verification model to extract target-speaker embeddings from full local EEND windows, including both non-overlapped and overlapped speech. Inspired by DiCoW, speaker-activity information is injected into intermediate WavLM representations through frame-level diarization-dependent transformations, allowing the model to preserve the pretrained initialization while learning to use target-specific diarization cues.
Experiments across multiple conversational diarization benchmarks showed that SAC-WavLM substantially reduces speaker-confusion errors and improves full EEND-VC diarization performance. Fine-tuning on a limited amount of real conversational data was more robust with estimated DiariZen outputs than training on large simulated mixtures alone, and provided clear gains over the corresponding WavLM-MHFA baseline. We further compared several modern embedding extractors under common AHC and SphereVBx clustering backends, showing that speaker-activity-conditioned embeddings consistently improve speaker linking and that SphereVBx further reduces the remaining gap to oracle clustering.
Generative AI Use Disclosure
Generative AI tools, such as ChatGPT, were used exclusively to refine language, paraphrasing, and improve manuscript readability. The scientific contributions, methodology, experiments, analysis, and conclusions were developed by the authors. The authors assume full responsibility for the accuracy, integrity and originality of the work and have approved its submission.
References
References
- [1]H. Bredin, “pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe,” in Proc. Interspeech, 2023, pp. 1983–1987.DOI
- [2]J. Han, F. Landini, J. Rohdin, A. Silnova, M. Diez, and L. Burget, “Leveraging self-supervised learning for speaker diarization,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5.arxiv.org/abs/2409.09408
- [3]K. Kinoshita, M. Delcroix, and N. Tawara, “Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 7198–7202.
- [4]J. Han, F. Landini, J. Rohdin, A. Silnova, M. Diez, J. Černocký, and L. Burget, “Fine-tune before structured pruning: Towards compact and accurate self-supervised models for speaker diarization,” in Proc. Interspeech, 2025, pp. 1583–1587.arxiv.org/abs/2505.24111
- [5]J. Han, P. Pálka, M. Delcroix, F. Landini, J. Rohdin, J. Černocký, and L. Burget, “Efficient and robust speaker diarization via structured pruning of self-supervised models,” IEEE Transactions on Audio, Speech and Language Processing, vol. 34, pp. 1903–1914, 2026.DOI
- [6]A. Plaquet, N. Tawara, M. Delcroix, S. Horiguchi, A. Ando, and S. Araki, “Mamba-based segmentation model for speaker diarization,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5.arxiv.org/abs/2410.06459
- [7]F. Landini, J. Profant, M. Diez, and L. Burget, “Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: Theory, implementation and analysis on standard tasks,” Computer Speech & Language, vol. 71, 2022.arxiv.org/abs/2012.14952
- [8]A. Pálka, J. Han, M. Delcroix, N. Tawara, and L. Burget, “Vbx for end-to-end neural and clustering-based diarization,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026, pp. 17472–17476.
- [9]M. Delcroix, N. Tawara, M. Diez, F. Landini, A. Silnova, A. Ogawa, T. Nakatani, L. Burget, and S. Araki, “Multi-stream extension of variational Bayesian HMM clustering (MS-VBx) for combined end-to-end and vector clustering-based diarization,” in Proc. Interspeech, 2023, pp. 3477–3481.arxiv.org/abs/2305.13580
- [10]P. Pálka, J. Han, P. Singh, M. Delcroix, N. Tawara, and L. Burget, “SphereVBx: Spherical variational Bayes clustering for simplified EEND-VC diarization,” in Proc. Interspeech, 2026.
- [11]H. Zeinali, S. Wang, A. Silnova, P. Matějka, and O. Plchot, “BUT system description for VoxCeleb speaker recognition challenge 2019,” arXiv preprint arXiv:1910.12592, 2019.arxiv.org/abs/1910.12592
- [12]B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” in Proc. Interspeech, 2020, pp. 3830–3834.DOI
- [13]J. Peng, O. Plchot, T. Stafylakis, L. Mösner, L. Burget, and J. Černocký, “An attention-based backend allowing efficient fine-tuning of transformer models for speaker verification,” in IEEE Spoken Language Technology Workshop (SLT), 2023, pp. 555–562.
- [14]H. Wang, S. Zheng, Y. Chen, L. Cheng, and Q. Chen, “CAM++: A fast and efficient network for speaker verification using context-aware masking,” in Proc. Interspeech, 2023, pp. 5301–5305.arxiv.org/abs/2303.00332
- [15]S. Horiguchi, T. Moriya, A. Ando, T. Ashihara, H. Sato, N. Tawara, and M. Delcroix, “Guided speaker embedding,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5.
- [16]S. Horiguchi, T. Ashihara, M. Delcroix, A. Ando, and N. Tawara, “Mitigating non-target speaker bias in guided speaker embedding,” in Proc. Interspeech, 2025, pp. 5208–5212.arxiv.org/abs/2506.12500
- [17]J. Kalda, C. Pagés, T. Alumäe, and H. Brédin, “Diarization-guided multi-speaker embeddings,” in Proc. Interspeech, 2025, pp. 5233–5237.DOI
- [18]A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “VoxCeleb: Large-scale speaker verification in the wild,” Computer Speech & Language, vol. 60, p. 101027, 2020.DOI
- [19]S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshii, X. Xiao et al., “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.DOI
- [20]A. Polok, D. Klement, M. Kocour, J. Han, F. Landini, B. Yusuf, M. Wiesner, S. Khudanpur, J. Černocký, and L. Burget, “DiCoW: Diarization-conditioned whisper for target speaker automatic speech recognition,” Computer Speech & Language, vol. 95, p. 101841, 2026.
- [21]A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeay, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning. PMLR, 2023, pp. 28492–28518.arxiv.org/abs/2212.04356
- [22]J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal et al., “The AMI meeting corpus: A pre-announcement,” in International workshop on machine learning for multimodal interaction. Springer, 2006, pp. 28–39.DOI
- [23]Y. Fu, L. Cheng, S. Lv, Y. J. Y. Kong, Z. Chen, Y. Hu, L. Xie, J. Wu, H. Bu, X. Xu, J. Du, and J. Chen, “AISHELL-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario,” in Proc. Interspeech, 2021, pp. 3665–3669.arxiv.org/abs/2104.03603
- [24]F. Yu, S. Zhang, Y. Fu, L. Xie, S. Zheng, Z. Du, W. Huang, P. Guo, Z. Yan, B. Ma et al., “M2MeT: The ICASSP 2022 multi-channel multi-party meeting transcription challenge,” in Proc. ICASSP, 2022, pp. 6167–6171.arxiv.org/abs/2110.07393
- [25]A. Vinnikov, A. Ivry, A. Hurwitz, I. Abramovski, S. Koubi, I. Gurvich, S. Peer, X. Xiao, B. M. Elizalde, N. Kanda et al., “NOTSOFAR-1 challenge: New datasets, baseline, and tasks for distant meeting transcription,” in Proc. Interspeech, 2024, pp. 5003–5007.arxiv.org/abs/2401.08887
- [26]T. Liu, S. Fan, X. Xiang, H. Song, S. Lin, J. Sun, T. Han, S. Chen, B. Yao, S. Liu, Y. Wu, Y. Qian, and K. Yu, “MSDWild: Multi-modal speaker diarization dataset in the wild,” in Proc. Interspeech, 2022, pp. 1476–1480.DOI
- [27]N. Ryant, P. Singh, V. Krishnamohan, R. Varma, K. Church, C. Cieri, J. Du, S. Ganapathy, and M. Liberman, “The third DIHARD diarization challenge,” in Proc. Interspeech, 2021, pp. 3570–3574.arxiv.org/abs/2012.01477
- [28]J. S. Chung, J. Huh, A. Nagrani, T. Afouras, and A. Zisserman, “Spot the conversation: Speaker diarisation in the wild,” in Proc. Interspeech, 2020, pp. 299–303.DOI
- [29]Z. Yang, Y. Chen, L. Luo, R. Yang, L. Ye, G. Cheng, J. Xu, Y. Jin, Q. Zhang, P. Zhang, L. Xie, and Y. Yan, “Open source MagicData-RAMC: A rich annotated mandarin conversational (RAMC) speech dataset,” in Proc. Interspeech, 2022, pp. 1736–1740.arxiv.org/abs/2203.16844
- [30]H. Wang, C. Liang, S. Wang, Z. Chen, B. Zhang, X. Xiang, Y. Deng, and Y. Qian, “Weakspeaker: A research and production oriented speaker embedding learning toolkit,” in Proc. ICASSP, 2023, pp. 1–5.arxiv.org/abs/2210.17016
- [31]F. Landini, A. Lozano-Diez, M. Diez, and L. Burget, “From simulated mixtures to simulated conversations as training data for end-to-end neural diarization,” in Proc. Interspeech, 2022, pp. 5095–5099.arxiv.org/abs/2204.00890
- [32]S. Horiguchi, M. Delcroix, N. Tawara, T. Ashihara, and A. Ando, “Tight boundary prediction in speaker diarization using causal-anticipausal consistency,” in Proc. Interspeech, 2026.
