betaFeedback

WorksEngineering

Sequence-to-Sequence Refinement for EEND-VC Diarization with a Shared Encoder

Abstract

End-to-end neural diarization with vector clustering (EEND-VC) separates local segmentation from global speaker assignment, leaving speaker confusion (SC) despite strong segmentation. We propose shared-encoder sequence-to-sequence (S2S) refinement, which reuses DiariZen's self-supervised learning (SSL) representations for speaker-conditioned decoding and prior-constrained assignment. We study frozen-feature refinement with WavLM and structurally pruned w2v-BERT 2.0, and extend the latter to joint training of local segmentation and refinement heads. The heads learn separate layer mixtures while their losses update one acoustic encoder. Across eight domains, frozen-feature refinement reduces macro diarization error rate (DER) from 12.95% to 12.40% with WavLM and from 12.35% to 12.08% with w2v-BERT 2.0, with relative SC reductions of 17.4% and 14.6%. It adds approximately 15% to system real-time factor. The joint system further lowers macro DER to 11.90%, improving seven of eight domains over non-joint refinement. Error decomposition separates SC-driven refinement gains from detection-driven joint-system gains.

Introduction

End-to-end neural diarization (EEND) uses permutation-invariant training (PIT) for frame-level speaker activity prediction [1]. EEND with vector clustering (EEND-VC) handles long recordings and varying speaker inventories through cross-window clustering [2, 3]. DiariZen strengthens local segmentation with self-supervised learning (SSL) representations, a Conformer backend, and power-set modeling [4, 5]. Structured pruning improves efficiency alongside its reported state-of-the-art performance across benchmarks [4, 6]. However, segmentation and global identity assignment are optimized separately: correct local activities and speaker counts can still receive incorrect recording-level identities. This mismatch motivates a refinement stage that can reconsider identities while retaining reliable local activity estimates.

Post-processing can revisit these assignments. Neural resegmentation revises local activities [7], while target-speaker voice activity detection (TS-VAD) uses enrollment embeddings [8]. Seq2Seq-TSVAD separates acoustic and speaker inputs [9]; neural speaker diarization using memory-aware multi-speaker embedding with sequence-to-sequence architecture (NSD-MS2S) adds memory-based representations [10]. Enrollment depends on initial diarization, motivating improved initialization and iterative refinement in USTC’s CHiME-8 system [11]. NTT CHiME-8 initializes NSD-MS2S with EEND-VC outputs but retains a separate acoustic encoder [12]. Chunk-level speaker queries also refine DiariZen outputs on DIHARD III [13].

Sharing an acoustic encoder can avoid repeated encoding and permit adaptation to both segmentation and refinement. DiariZen’s diarization output provides an activity prior for this stage, allowing speaker-conditioned predictions to be constrained by a strong local model. Yet lower diarization error rate (DER) need not imply better speaker assignment: detection-threshold changes can reduce detection errors [7]. We therefore distinguish speaker confusion (SC) from false alarms (FA) and missed speech (MISS) when evaluating refinement and joint training across domains.

We develop sequence-to-sequence (S2S) refinement by replacing NSD-MS2S’s acoustic encoder with DiariZen’s SSL representations while retaining its enrollment mechanism and a lightweight decoder. Prior-constrained inference revises identities and overlap counts. Each window undergoes one SSL pass, while independently learned layer mixtures let the two heads use different combinations of SSL representations. We study frozen-feature refinement and shared-encoder joint training of local segmentation and refinement; clustering remains outside the training graph. Eight-domain evaluation with WavLM and pruned w2v-BERT 2.0 shows SC-driven gains from frozen-feature refinement and further detection-driven DER gains from the joint system. Component analysis and identity-correction cases explain these effects.

Method

DiariZen and Shared SSL Features

DiariZen [4] combines local segmentation and cross-window clustering (Figure 1(a)). We share a pruned WavLM [14] or w2v-BERT 2.0 [15] encoder across both heads. Each window undergoes one SSL pass; the two heads independently mix and project its layer outputs:

Figure 1. DiariZen and shared SSL features (a), speaker-conditioned decoding and training (b), and prior-constrained inference (c). Gray-green dashed arrows denote conditioning; blue/red dashed arrows denote parameter updates.

E(j)=Pj(∑ℓaℓ(j)Hℓ),j∈{loc,ref}.(1)E^{(j)} = P_j\left(\sum_{\ell} a_{\ell}^{(j)} H_{\ell}\right), \quad j \in\{\mathrm{loc},\mathrm{ref}\}. \tag*{(1)}

Here HℓH_{\ell} denotes an SSL layer output, and aℓ(j)a_{\ell}^{(j)} are unconstrained learned weights. Each PjP_j applies linear projection and layer normalization (LN), followed by dropout for refinement. We denote the decoder input by E=E(ref)∈RT×dE = E^{(\mathrm{ref})} \in\mathbb{R}^{T \times d}.

We apply structured pruning [6, 16] with hard-concrete gates [17] to w2v-BERT 2.0’s dual feed-forward modules, attention heads, and convolution channels. Export removes inactive units and folds gate scales into weights; convolution statistics retain the original channel count.

The DiariZen branch maps E(loc)E^{(\mathrm{loc})} through a Conformer and power-set classifier to anonymous speaker activities. Its local loss Lloc\mathcal{L}_{\mathrm{loc}} uses power-set negative log-likelihood (NLL) with permutation-aligned targets. At inference, these activities guide speaker embedding extraction from audio; clustering across windows produces the global binary prior BB.

Speaker-Conditioned Decoder and Training

Figure 1(b) uses acoustic embeddings EE in place of the NSD-MS2S acoustic encoder, retaining its i-vector and memory-based enrollment mechanism [10]. A candidate’s non-overlapping prior segments provide an input mask MntM_{nt} and session i-vector vnv_n [18]. Masked averaging of EE over MntM_{nt} gives a query to fixed i-vector and x-vector center banks (i/x memory). Attention across candidates then mixes the retrieved representations. Concatenating the two outputs with vnv_n gives sn=[un;vn]s_n = [u_n; v_n], row nn of condition matrix SS.

The decoder initializes candidate states to zero. Each of LL layers injects sns_n into self-attention queries/keys and cross-attention queries, while EE supplies acoustic keys/values. A feed-forward network (FFN) completes the layer. Final states act as speaker attractors; layer-normalized frame and attractor projections ft,gnf_t, g_n produce

pnt=σ(τdftTgn+b),(2)p_{nt} = \sigma\left(\frac{\tau}{\sqrt{d}} f_t^{\mathsf{T}} g_n + b\right), \tag*{(2)}

with learned positive scale τ\tau and bias bb. Independent sigmoids predict all candidates’ activities in parallel, allowing overlap without autoregressive decoding.

Both training modes use binary cross-entropy (BCE) for real and sampled negative candidates. Each term averages valid frames; negative targets are zero:

Lref=BCE‾real+ηBCE‾neg.(3)\mathcal{L}_{\mathrm{ref}} = \overline{\mathrm{BCE}}_{\mathrm{real}} + \eta\overline{\mathrm{BCE}}_{\mathrm{neg}}. \tag*{(3)}

Non-joint training fixes the SSL encoder and local branch, updating the refinement branch, including its layer mixture and memory query/key projections. It uses paired predicted or oracle priors/profiles with oracle targets.

Joint training uses oracle priors/profiles and updates the encoder and both heads with Ljoint=Lloc+Lref\mathcal{L}_{\mathrm{joint}} = \mathcal{L}_{\mathrm{loc}} + \mathcal{L}_{\mathrm{ref}}. Unlike local PIT, refinement supervises enrolled speaker identities.

Both modes exclude overlap from enrollment but retain it in targets. Memory centers and the i-vector extractor stay fixed; clustering, profile extraction, and inference constraints are outside the training graph.

Prior-Constrained Inference

Both modes follow Figure 1(c). Non-overlapping regions of DiariZen’s prior BB supply MM and session i-vectors; candidates without usable profiles are excluded. During the local pass, the refinement branch caches its acoustic embeddings EE. Once global tracks and enrollment profiles are available, decoding reads this cache without re-encoding the waveform. Unlike MM, BB retains overlap. Identity-aligned window probabilities are equally averaged to obtain pˉnt\bar{p}_{nt}.

Prior-biased ranking combines pˉ\bar{p} and BB:

rnt=logit⁡(pˉnt)+βBnt,kt=∑nBnt.(4)r_{nt} = \operatorname{logit}(\bar{p}_{nt}) + \beta B_{nt}, \qquad k_t = \sum_n B_{nt}. \tag*{(4)}

We first select the top ktk_t identities. If kt>1k_t > 1 and the weakest selected probability is at most θdrop\theta_{\mathrm{drop}}, the count decreases by one. Otherwise, an unselected probability at least θadd\theta_{\mathrm{add}} increases a positive count by one. We then reselect identities at the adjusted count. An enrolled candidate inactive in BB at a frame can replace an incorrectly assigned speaker if its decoder evidence overcomes the prior bias. Ranking corrects identities at unchanged count; count adjustment can restore missed overlap or remove a spurious speaker. Silent frames remain silent and the last active speaker is retained. The profile-filtered prior’s speech support is preserved, while speaker activities and boundaries can change.

Experimental Setup

Data and Evaluation

We evaluate channel 0 of AISHELL-4 (AIS) [19], AliMeeting (AliM) [20], AMI [21], DIHARD III (DH3) [22], MSDWild (MSD) [23], NOTSOFAR-1 (NSF) [24], RAMC [25], and VoxConverse (VoxC) [26]: 1,209 recordings, including complete NSF evaluation. NIST md-eval scoring uses zero collar, overlap, and each set’s UEM. Macro scores equally weight datasets. Table 1 gives overlap/speech duration ratios within scored regions and reference speaker ranges per recording. RAMC has no annotated overlap.

SystemRAMCVoxCAISDH3MSDAMIAliMNSFFAMISSSCDER
Overlap (%)0.03.14.99.412.414.620.430.1
Speakers21–215–71–92–43–42–43–7
DiariZen (WavLM)10.999.11<u>9.99</u>14.4815.7713.9110.8118.503.875.643.4412.95
+ S2S Refinement11.058.919.5814.4215.4413.1610.1116.544.165.402.8412.40
DiariZen (w2v-BERT 2.0)<u>11.01</u>8.8810.6914.2214.4513.449.6916.433.925.452.9812.35
+ S2S Refinement11.17<u>8.70</u>10.1614.33<u>14.35</u><u>13.02</u><u>9.36</u><u>15.53</u>4.14<u>5.39</u>2.55<u>12.08</u>
Joint DiariZen (w2v-BERT 2.0)11.028.7311.0213.7814.4913.219.9115.913.745.542.9912.26
+ S2S Refinement11.088.6610.55<u>13.82</u>14.2712.749.3214.744.055.28<u>2.57</u>11.90

Table 1. DER by dataset and macro error components (%) under BF16 SSL inference. Datasets are ordered by overlap. DiariZen includes clustering; Joint uses shared-encoder training. Bold/underline mark lowest/second-lowest values per column. DER uses unrounded components.

Training and Inference

Pruned w2v-BERT 2.0 has 115.8M parameters (80.05% reduction) and 25 hidden-state outputs of width 1024. Local segmentation uses a 256-dimensional projection, four Conformer layers, and 16 power-set classes for four speakers. Refinement uses six 512-dimensional decoder layers with eight attention heads and 1024-dimensional FFNs. We reuse NSD-MS2S’s 100-dimensional i-vector extractor and two 128-center memory banks1 (i/x dimensions: 100/256). We pad training samples to eight candidates with negative profiles (η=0.25\eta= 0.25).

Both w2v-BERT 2.0 training modes use 16-s windows from eight corpora, 40% hard-window sampling, and 8× NVIDIA H20 GPUs. Non-joint training freezes the encoder and samples paired DiariZen/oracle priors and profiles with probabilities 0.7/0.3, using oracle targets. AdamW uses learning rate 2×10−42 \times10^{-4}, weight decay 0.01, 8% warmup, linear decay, and batch size 512. We average the five checkpoints with lowest oracle-prior development loss.

Joint training initializes the encoder from post-pruning distillation, reinitializes both heads, and uses oracle priors/profiles. Encoder/head AdamW learning rates are 2×10−6/10−42 \times10^{-6}/10^{-4}, with zero weight decay and no schedule. Eight samples per GPU and two accumulation steps give an effective batch of 128. Training stops at epoch 14 with patience five; we average the five checkpoints with lowest joint development loss. Local targets retain the four most active speakers without masking omitted-speaker frames.

Pooled development DER selects refinement parameters and, for joint training, chooses between the best epoch and the five-checkpoint average. Non-joint/joint systems use β=1/0.5\beta= 1/0.5 and add/drop thresholds 0.8/0.20.8/0.2, with no temporal smoothing in refinement. Inference uses 16-s windows, a 1.6-s hop, BF16 SSL computation, and FP32 heads. Table 2 reports frozen-feature real-time factor (RTF; processing time/audio duration).

Front endDiariZenRefine.TotalIncrease
WavLM0.01770.00270.020415.30%
w2v-BERT 2.00.01800.00270.020715.03%

Table 2. H20 RTF for frozen-feature refinement (batch 64). DiariZen includes clustering; Refine. is the full increment.

Results

Diarization Across Domains

Table 1 compares complete systems. Frozen-feature S2S refinement reduces macro DER from 12.95% to 12.40% with WavLM and from 12.35% to 12.08% with w2v-BERT 2.0, improving seven and six domains, respectively. Both improve all four meeting sets; high-overlap NSF shows the largest reductions, from 18.50% to 16.54% and 16.43% to 15.53%. Gains also extend to VoxC and AIS, with overlap below 5%. RAMC degrades slightly with both front ends, while DH3 improves only with WavLM.

Joint training yields 12.26% macro DER from the local output and 11.90% after refinement. The final system improves seven domains over non-joint refinement, chiefly NSF and DH3; AIS worsens from 10.16% to 10.55%. Refinement is not uniformly beneficial: on RAMC and DH3, the joint local output slightly outperforms its refined output.

Figure 2. Highlighted fixed-count identity corrections with non-joint w2v-BERT 2.0: (a) AliM and (b) AIS.

Speaker Confusion and Refinement

The macro components in Table 1 separate identity and detection errors. Frozen-feature refinement reduces SC by 17.4% with WavLM and 14.6% with w2v-BERT 2.0. Both reduce MISS but increase FA. For w2v-BERT 2.0, SC falls by 0.44 percentage points, exceeding the 0.16-point increase in FA plus MISS. SC decreases on seven domains; RAMC is the exception. On DH3, SC falls from 2.71% to 2.51% and MISS also decreases, but higher FA raises DER overall.

Figure 2 shows count-preserving identity correction: {S1,S2}→{S1,S3}\{S_1, S_2\} \to\{S_1, S_3\} in AliM overlap and S5→S4S_5 \to S_4 in AIS single-speaker speech. The highlighted intervals last 3.31 and 4.25 s, with zero FA and MISS for both systems. Both corrections select an enrolled speaker that was inactive in the prior at those frames, consistent with the ranking in Eq. (4). Figure labels report SC over the full 12-s crops. Both systems use identical full-recording identity mappings, fixed before cropping to exclude local rematching. Residual AIS errors illustrate incomplete correction within a favorable case.

Joint-System Gains and Error Trade-offs

Compared with non-joint refinement, the joint system lowers macro FA and MISS by 0.090 and 0.115 percentage points, while SC rises by 0.024 points. Local SC is also nearly unchanged (2.98% versus 2.99%). Detection therefore drives the incremental DER gain.

Within the joint system, refinement still reduces SC from 2.99% to 2.57% (13.9% relative), preserving its identity-correction benefit. Relative to the corresponding local output, the combined FA and MISS increase is 0.05 percentage points with joint refinement, versus 0.16 with non-joint refinement. Thus, despite a slightly smaller SC reduction, joint refinement retains more of this benefit in the final DER.

Joint training lowers final SC on NSF, RAMC, and VoxC but raises it elsewhere. NSF improves all three components; DH3 reduces FA and MISS despite higher SC, while AIS worsens in all three. Lower macro DER therefore does not ensure uniformly better speaker assignment.

With the original refinement parameters, the joint system achieves 11.902% macro DER, versus 12.077% for non-joint refinement; retuning changes DER by only 0.006 percentage points. The improvement thus persists without retuning the inference constraints. Initialization, prior sampling, and model selection also differ, so this system comparison does not isolate the joint objective.

Conclusion

Shared-encoder S2S refinement reuses DiariZen’s representations for identity correction without another refinement acoustic encoder. Across eight domains, frozen-feature refinement lowers SC and macro DER at approximately 15% additional RTF. Joint training with pruned w2v-BERT 2.0 yields the best macro DER, 11.90%, through complementary detection gains. Component analysis and fixed-count cases demonstrate the value of speaker-conditioned decoding and conservative prior constraints alongside local segmentation.

Acknowledgments

Generative AI use. OpenAI Codex (GPT-6) assisted with translating and polishing author-provided text and generating candidate visual elements for clearer presentation in figures and tables. The authors developed all scientific content, manually finalized the manuscript and visuals, verified numerical values, and take full responsibility for the work.

References

References

  1. [1]Y. Fujita, N. Kanda, S. Horiguchi, K. Nagamatsu, and S. Watanabe, “End-to-end neural speaker diarization with permutation-free objectives,” in Proc. Interspeech, 2019, pp. 4300–4304.arxiv.org/abs/1909.05952
  2. [2]K. Kinoshita, M. Delcroix, and N. Tawara, “Integrating end-to-end neural and clustering-based diarization: getting the best of both worlds,” in Proc. IEEE ICASSP, 2021, pp. 7198–7202.
  3. [3]P. Pálka, J. Han, P. Singh, M. Delcroix, N. Tawara, and L. Burget, “SphereVBx: spherical variational Bayes clustering for simplified EEND-VC diarization,” in Proc. Interspeech, 2026, pp. 4206–4211.arxiv.org/abs/2606.24528
  4. [4]J. Han, F. Landini, J. Rohdin, A. Silnova, M. Diez, and L. Burget, “Leveraging self-supervised learning for speaker diarization,” in Proc. IEEE ICASSP, 2025, pp. 1–5.arxiv.org/abs/2409.09408
  5. [5]A. Plaquet and H. Bredin, “Powerset multi-class cross entropy loss for neural speaker diarization,” in Proc. Interspeech, 2023, pp. 3222–3226.DOI
  6. [6]J. Han et al., “Fine-tune before structured pruning: towards compact and accurate self-supervised models for speaker diarization,” in Proc. Interspeech, 2025, pp. 1583–1587.arxiv.org/abs/2505.24111
  7. [7]H. Bredin and A. Laurent, “End-to-end speaker segmentation for overlap-aware resegmentation,” in Proc. Interspeech, 2021, pp. 3111–3115.arxiv.org/abs/2104.04045
  8. [8]I. Medennikov et al., “Target-speaker voice activity detection: a novel approach for multi-speaker diarization in a dinner party scenario,” in Proc. Interspeech, 2020, pp. 274–278.DOI
  9. [9]M. Cheng, W. Wang, Y. Zhang, X. Qin, and M. Li, “Target-speaker voice activity detection via sequence-to-sequence prediction,” in Proc. IEEE ICASSP, 2023, pp. 1–5.arxiv.org/abs/2210.16127
  10. [10]G. Yang et al., “Neural speaker diarization using memory-aware multi-speaker embedding with sequence-to-sequence architecture,” in Proc. IEEE ICASSP, 2024, pp. 11626–11630.arxiv.org/abs/2309.09180
  11. [11]S. Niu et al., “The USTC-NERCSLIP systems for the CHiME-8 NOTSOFAR-1 challenge,” in Proc. CHiME, 2024, pp. 31–36.arxiv.org/abs/2409.02041
  12. [12]N. Kamo et al., “Microphone array geometry-independent multi-talker distant ASR: NTT system for dASR task of the CHiME-8 challenge,” Computer Speech & Language, vol. 95, Art. no. 101820, 2026.
  13. [13]N. Tawara and S. Horiguchi, “Target-speaker voice activity detection with chunk-level speaker queries,” in Proc. IEEE ICASSP, 2026, pp. 22202–22206.DOI
  14. [14]S. Chen et al., “WavLM: large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.DOI
  15. [15]Seamless Communication et al., “Seamless: multilingual expressive and streaming speech translation,” arXiv:2312.05187, 2023.arxiv.org/abs/2312.05187
  16. [16]Z. Li, M. Cheng, and M. Li, “Enhancing speaker verification with w2v-BERT 2.0 and knowledge distillation guided structured pruning,” arXiv:2510.04213, 2025.arxiv.org/abs/2510.04213
  17. [17]C. Louizos, M. Welling, and D. P. Kingma, “Learning sparse neural networks through L₀ regularization,” arXiv:1712.01312, 2017.arxiv.org/abs/1712.01312
  18. [18]N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, May 2011.DOI
  19. [19]Y. Fu et al., “AISHELL-4: an open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario,” in Proc. Interspeech, 2021, pp. 3665–3669.arxiv.org/abs/2104.03603
  20. [20]F. Yu et al., “M2MeT: the ICASSP 2022 multi-channel multi-party meeting transcription challenge,” in Proc. IEEE ICASSP, 2022, pp. 6167–6171.arxiv.org/abs/2110.07393
  21. [21]J. Carletta et al., “The AMI meeting corpus: a pre-announcement,” in Machine Learning for Multimodal Interaction, pp. 28–39. Springer Berlin Heidelberg, 2006.DOI
  22. [22]N. Ryant et al., “The third DIHARD diarization challenge,” in Proc. Interspeech, 2021, pp. 3570–3574.arxiv.org/abs/2012.01477
  23. [23]T. Liu et al., “MSDWild: multi-modal speaker diarization dataset in the wild,” in Proc. Interspeech, 2022, pp. 1476–1480.DOI
  24. [24]A. Vinninkov et al., “NOTSOFAR-1 challenge: new datasets, baseline, and tasks for distant meeting transcription,” in Proc. Interspeech, 2024, pp. 5003–5007.
  25. [25]Z. Yang et al., “Open source MagicData-RAMC: a rich annotated mandarin conversational(RAMC) speech dataset,” in Proc. Interspeech, 2022, pp. 1736–1740.arxiv.org/abs/2203.16844
  26. [26]J. S. Chung, J. Huh, A. Nagrani, T. Afouras, and A. Zisserman, “Spot the conversation: speaker diarisation in the wild,” in Proc. Interspeech, 2020, pp. 299–303.DOI

Paper details

vlnc:W_ewmat7tr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Codex

View statement
Looking for related papers…
Contents