WorksComputer Science
MOUSSE: Multilingual Overlapped Speech Separation with Predictive Uncertainty Estimation
Abstract
Recent generative speech separation methods have shown promising performance, yet mixtures of overlapping speech in different languages remain underexplored. To this end, we first curate MuSSeD, an English-Korean benchmark comprising two- to four- speaker mixtures across varying overlap ratios. We further propose MOUSSE, a flow matching framework that incorporates boundary-aware training and frame-wise predictive uncertainty through a permutation-invariant Gaussian negative log-likelihood. The predicted scale is further used to adapt classifier-free guidance during sampling. MOUSSE achieves 13.18 dB SI-SDRi and 0.175 ESTOIi, outperforming discriminative and flow matching-based separation baselines, with a 1.58 dB SI-SDRi improvement over the strongest model.
Introduction
Speech separation, which aims to recover individual speakers’ signals from overlapping utterances, is central to the cocktail-party problem. Despite substantial progress, existing systems are developed predominantly in monolingual settings, especially English, with fixed speaker counts [1, 2]. Real-world conversations, by contrast, involve varying speaker counts, degrees of temporal overlap, and multiple languages, with cross-lingual differences in phonetic and intonational patterns introducing additional variability. A further challenge specific to multilingual settings is disentangling language identity from individual-speaker separation. Prior work on blind language separation (BLS) [3] emphasizes language-based disentanglement and generalization to unseen languages, but leaves open how reliably individual speakers are recovered as language composition varies. Therefore, systematic evaluations across languages, speaker counts, and overlap ratios are important in order to assess the robustness of multilingual speech separation.
Prior architectures have made significant progress in improving monolingual speech separation. Discriminative models such as Conv-TasNet [4] introduced an end-to-end time-domain approach, while RE-SepFormer [5] improves Transformer efficiency through compact representations. TIGER [6] and SR-CorrNet [7] further exploit time-frequency modeling and spatio-spectro-temporal correlations via an asymmetric encoder-decoder, respectively. Despite their effectiveness, these methods typically produce a single set of source estimates through direct inference, offering limited opportunity to iteratively refine uncertain source components.
Generative approaches offer a complementary approach, casting separation as an iterative reconstruction process. Diff-Sep [8] formulates separation via a score-matching diffusion process, while SpeechFlow [9] employs flow matching within a unified framework spanning separation, enhancement, and text-to-speech. UA-Flow [10] extends this line of work by jointly predicting a heteroscedastic uncertainty estimate alongside the velocity field in vision tasks. While classifier-free guidance (CFG) can strengthen conditional separation by amplifying the influence of the mixture during sampling, standard CFG [11] uses a shared guidance strength across sequence positions, without accounting for frame-wise variation in speaker activity and temporal overlap. This motivates adapting the guidance strength across frames.
To address these gaps, we propose MOUSSE, a flow matching framework for multilingual speech separation with predictive uncertainty. Our contributions are threefold: (1) we construct MuSSeD, an English–Korean speech separation benchmark spanning two to four speakers, English-only, Korean-only, and mixed-language compositions, and multiple overlap ratios, (2) we integrate a frame-wise predictive uncertainty estimator into a flow matching separator for uncertainty-aware training and sampling, and (3) we introduce boundary-aware weighting that accounts for speaker activity and assigns larger weights to frames near activity transitions. Extensive experiments show that our method outperforms discriminative and generative baselines on MuSSeD. The code and dataset will be made publicly available.12
Method
Speech Separation Dataset Curation
For bilingual speech separation, we curate the MuSSeD dataset by mixing English speech from LibriTTS-R [12] and Korean speech from the Multi-Speaker Speech Synthesis corpus [13]. The resulting dataset contains 37,800 mixtures from 514 speakers, with a male-to-female ratio of approximately 55:45. For each speaker count, the training, validation, and test sets contain 10,000, 1,300, and 1,300 mixtures, respectively. Speakers are mutually disjoint across the three splits to prevent speaker leakage between training and evaluation. Each mixture lasts 3–15 seconds and contains two to four speakers. We generate English-only, Korean-only, and mixed-language mixtures with overlap ratios of 0%, 25%, 50%, 75%, and 100%. Note that the overlap ratio is defined for each pair of adjacent speakers as the overlap duration relative to the duration of the shorter utterance. For mixtures with more than two speakers, speakers are placed sequentially such that the specified ratio is applied to each adjacent pair.
Uncertainty-Aware Training
We introduce boundary-aware weighting for the separation and uncertainty objectives with output slots (i.e., separated audio sources). Weighting focuses the objectives on frames, in which the target speaker is active, while assigning additional weight to frames near activity transitions. The separator follows the SpeechFlow backbone [9], which adapts a pre-trained flow matching model through task-specific conditioning for speech separation, and is trained as a single model on two- to four-speaker mixtures.
Formally, for target speaker at frame , we obtain a binary activity indicator from the target mel-spectrogram. Activity boundaries are identified from consecutive state changes, and a window of frames around each boundary is marked as the boundary region. Training weight is defined as:
where denotes the boundary indicator, set to 1 within the boundary neighborhood and 0 otherwise. Active frames near activity boundaries are therefore assigned twice the weight of steady active frames. Figure 1 visualizes the resulting boundary-aware weight map.

Figure 1. Boundary-aware weight maps for two- to four-speaker mixtures, which illustrate frames near activity boundaries.
Resulting boundary-aware weights are incorporated into the separation objective. The pairwise weighted mean squared error, using from (1), is given by
where denotes the predicted velocity for output slot at frame , and represents the target flow velocity for speaker .
is the total weight for speaker and is the number of mel dimensions. The velocity describes the flow from the noise distribution toward the target distribution. To account for speaker permutation ambiguity, permutation-invariant training (PIT) selects the assignment from the losses, resulting in the separation objective:
Furthermore, in order to identify regions where the model struggles to disentangle overlapping speech, we propose a predictive uncertainty estimator. The estimator takes the observed mixture , flow time , and current flow state . It uses sinusoidal and convolutional positional embedding followed by 12 Transformer blocks with hidden dimension 1024 and 8 attention heads. This uncertainty estimator then predicts log , a representation of the scale, at each frame and mel dimension for the output slot, providing a smooth parameterization of the positive scale required by the Gaussian likelihood. The corresponding scale is obtained by exponentiation, and the pairwise Gaussian negative log-likelihood is defined as:
where is the residual between the detached separator prediction and the target velocity , and is the corresponding predicted scale, both evaluated at each frame and mel dimension. PIT selects from the corresponding pairwise likelihood losses while keeping the separator prediction and predicted scale paired within each output slot. The uncertainty objective and the total training objective are
with . Note that the two objectives share the same boundary-aware weights, assigning larger weights to frames near speaker activity boundaries.

Figure 2. Overview of MOUSSE with boundary-aware training and adaptive uncertainty-guided sampling.
Adaptive Uncertainty-Guided Sampling
Recall that in Section §2.2, the uncertainty estimator predicts a scale for each position and mel dimension . To account for variations in the predicted scale across sequence positions, we adapt the classifier-free guidance (CFG) strength accordingly during sampling. Thus, the mean scale across mel dimensions defines the adaptive guidance strength:
where controls the uncertainty scaling, and bounds the guidance strength. During CFG, the separator produces conditional and unconditional velocity predictions at each frame. The conditional velocity is predicted using the observed mixture, while the unconditional velocity is obtained by replacing the mixture input with zeros. The guided velocity can then be defined as:
Moreover, guidance strength increases with predicted scale. Conditioned on the mixture , flow time , and current flow state , the uncertainty estimator updates at each frame and mel dimension for output slot . Note that sampling starts from Gaussian noise and proceeds with Euler updates using . The resulting sequence is split into separated mel-spectrogram outputs and subsequently converted to waveforms with the BigVSAN [14] vocoder.
Experimental Setup
MOUSSE is trained for 29000 iterations, and the scaling factor is set to as a unit scale so that the predicted uncertainty directly determines the guidance magnitude without additional rescaling. We use AdamW, batch size 4, on a single NVIDIA RTX A6000 GPU.
Table 1. Overall separation performance on MuSSeD.
To assess the proposed MOUSSE model, we juxtapose it against four different baseline models [4, 5, 7, 9] for reconstruction fidelity and speech intelligibility using signal-to-distortion ratio improvement (SDRi), scale-invariant signal-to-distortion ratio improvement (SI-SDRi), and extended short-time objective intelligibility improvement (ESTOIi). SDRi is sensitive to signal scaling, SI-SDRi is scale-invariant, and ESTOIi reflects speech intelligibility. For all metrics, improvement is computed relative to the input mixture, using the corresponding clean source as the reference.
All baseline models are retrained on MuSSeD with the official code and the same test set. Discriminative methods are evaluated directly, whereas predicted mel-spectrograms are decoded using a vocoder for flow matching models.
Results
Evaluation
Table 1 reports the overall performance on MuSSeD. Our model achieves 13.20 dB SDRi and 13.18 dB SI-SDRi, outperforming all baseline models. Among the baselines, SR-CorrNet [7] obtains the highest SI-SDRi of 11.60 dB, followed by Conv-TasNet [4] at 9.72 dB, ReSepformer [5] at 5.56 dB, and SpeechFlow [9] at 3.65 dB. MOUSSE therefore improves SI-SDRi by 1.58 dB over the strongest baseline. MOUSSE also achieves the highest intelligibility improvement, reaching 0.175 ESTOIi, while the baselines range from
| Configuration | SDRi | SI-SDRi | ESTOIi |
| Full model | 13.20 | 13.18 | 0.175 |
| – Boundary-aware weighting | 12.11 | 12.10 | 0.164 |
| – NLL loss () | 11.95 | 11.93 | 0.142 |
| Fixed guidance () | 13.00 | 12.98 | 0.172 |
| PIT MSE training | 10.53 | 10.52 | 0.138 |
caption: Ablation study of MOUSSE components.
:::
–0.072 to 0.092. Since ESTOIi is measured relative to the input mixture, values near zero indicate little improvement. These results show that MOUSSE improves both signal-level separation and intelligibility.
To evaluate the contribution of each component, we conduct ablation experiments as shown in Table 2. Removing boundary-aware weighting while retaining the uncertainty objective and adaptive guidance reduces SI-SDRi to 12.10 dB and ESTOIi to 0.164. Additionally, to evaluate the impact of uncertainty modeling, we remove the uncertainty NLL objective. Since adaptive guidance relies on the predicted uncertainty estimates, we use a fixed value with , which corresponds to the average adaptive guidance value of the full model. This configuration achieves 11.93 dB SI-SDRi and 0.142 ESTOIi, indicating the contribution of uncertainty-aware training with its use during sampling. To isolate the effect of adaptive guidance, we evaluate the fully trained model at inference with a fixed , which achieves 12.98 dB SI-SDRi and 0.172 ESTOIi. Lastly, training with the PIT MSE objective on target-active frames without boundary-aware weighting and uncertainty modeling achieves 10.52 dB SI-SDRi and 0.138 ESTOIi.
Uncertainty Analysis
We further analyze how the predicted scale varies across mixture conditions and whether it reflects model prediction error. Table 3 summarizes the mean conditional predicted scale , averaged over sampling steps and active frames, across language compositions and overlap ratios. As audio overlap increases from 0% to 100%, the predicted scale rises from 0.149 to 0.198, from 0.137 to 0.217, and from 0.131 to 0.221 for two- to four-speaker mixtures, respectively. This consistent increase indicates that the estimator assigns larger scales under higher overlap ratios. Across language compositions, English-only mixtures achieve the highest SI-SDRi, outperforming Korean-only mixtures by 3.98 dB. In contrast, multilingual mixtures demonstrate lower Si-SDRi, which suggests that cross-lingual overlap introduces additional challenges for speech separation.
| 2 spk | 3 spk | 4 spk | SI-SDRi | ||
| Lang. | Eng. | 0.166 | 0.168 | 0.172 | 14.79 |
| Lang. | Kor. | 0.193 | 0.193 | 0.181 | 10.81 |
| Lang. | Mixed | 0.168 | 0.171 | 0.170 | 13.89 |
| Overlap Ratio | 0% | 0.149 | 0.137 | 0.131 | 13.77 |
| Overlap Ratio | 25% | 0.166 | 0.152 | 0.142 | 13.95 |
| Overlap Ratio | 50% | 0.181 | 0.175 | 0.162 | 13.21 |
| Overlap Ratio | 75% | 0.197 | 0.200 | 0.190 | 11.54 |
| Overlap Ratio | 100% | 0.198 | 0.217 | 0.221 | 13.76 |
Table 3. Mean conditional predicted scale and SI-SDRi across language compositions and overlap ratios.
To further examine the frame-level relationship between predicted scale and prediction error, Figure 3a groups active frames by predicted-scale range and shows that the mean velocity residual increases consistently with predicted scale. A

Figure 3. Predictive uncertainty and adaptive guidance analysis.
Spearman correlation of , computed across active frames between predicted scale and velocity residual, supports this relationship, indicating that larger predicted scales are associated with larger flow-prediction errors. These results suggest that the predicted scale reflects local variations in flow-prediction error and can therefore serve as a control signal for adaptive guidance.
Moreover, guidance distribution shifts toward larger values as overlap increases, while substantial variation remains within each condition (Figure 3b). The concentration near low values reflects the large number of frames with relatively small predicted scales, while higher-overlap conditions contain more frames with larger guidance strengths. This variation thus validates using predicted scale to adjust guidance strength for each frame.
Conclusion
In this paper, we have presented MuSsED, an English–Korean speech separation benchmark that covers 2–4 speakers and multiple overlap conditions, and MOUSSE, a multilingual flow matching framework with predictive uncertainty. MOUSSE incorporates boundary-aware training and frame-wise predictive uncertainty, achieving 13.18 dB SI-SDRi and 0.175 ESTOIi. Ablation results show that boundary-aware weighting and uncertainty-aware training provide larger improvements, while adaptive guidance provides a relatively smaller improvement over fixed guidance. Future work will extend evaluation to additional languages and real conversational settings with background noise.
ACKNOWLEDGMENTS
This research was partly supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(RS-2022-NR070870), by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS-2019-III191906, Artificial Intelligence Graduate School Program(POSTECH)), and by the Gyeongsangbuk-do RISE(Regional Innovation System & Education) project(B0080527002599). This work was also supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada and in part by the AI2050 program at Schmidt Sciences. Finally, we are grateful for the support from IVADO and the Canada First Research Excellence Fund.
Compliance with Ethical Standards
This is a modeling study for which no ethical approval was required.
References
- [1]Marvin Borsdorf, Chenglin Xu, Haizhou Li, and Tanja Schultz, “Globalphone mix-to-separate out of 2: a multilingual 2000 speakers mixtures database for speech separation,” in Proc. Interspeech 2021, 2021, pp. 3905–3909.DOI
- [2]Fan-Lin Wang, Hung-Shin Lee, Yu Tsao, and Hsin-Min Wang, “Disentangling the impacts of language and channel variability on speech separation networks,” arXiv preprint arXiv:2203.16040, 2022.arxiv.org/abs/2203.16040
- [3]Marvin Borsdorf, Kevin Scheck, Haizhou Li, and Tanja Schultz, “Blind language separation: Disentangling multilingual cocktail party voices by language,” in INTERSPEECH, 2022, pp. 256–260.DOI
- [4]Yi Luo and Nima Mesgarani, “Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation,” IEEE/ACM transactions on audio, speech, and language processing, vol. 27, no. 8, pp. 1256–1266, 2019.DOI
- [5]Luca Della Libera, Cem Subakan, Mirco Ravanelli, Samuele Cornell, Frédéric Lepoutre, and François Grondin, “Resource-efficient separation transformer,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 761–765.arxiv.org/abs/2206.09507
- [6]Mohan Xu, Kai Li, Guo Chen, and Xiaolin Hu, “Tiger: Time-frequency interleaved gain extraction and reconstruction for efficient speech separation,” in International Conference on Learning Representations, 2025, vol. 2025, pp. 70205–70222.arxiv.org/abs/2410.01469
- [7]Ui-Hyeop Shin and Hyung-Min Park, “Asymmetric encoder-decoder based on time-frequency correlation for speech separation,” arXiv preprint arXiv:2603.29097, 2026.arxiv.org/abs/2603.29097
- [8]Robin Scheibler, Youna Ji, Soo-Whan Chung, Jaeuk Byun, Soyeon Choe, and Min-Seok Choi, “Diffusion-based generative speech source separation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.arxiv.org/abs/2210.17327
- [9]Alexander Liu, Matthew Le, Apoorv Vyas, Bowen Shi, Andros Tjandra, and Wei-Ning Hsu, “Generative pre-training for speech with flow matching,” in International Conference on Learning Representations, 2024, vol. 2024, pp. 9435–9454.arxiv.org/abs/2310.16338
- [10]Juyeop Han, Lukas Lao Beyer, and Sertac Karaman, “Flow matching with uncertainty quantification and guidance,” arXiv preprint arXiv:2602.10326, 2026.arxiv.org/abs/2602.10326
- [11]Jonathan Ho and Tim Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598, 2022.arxiv.org/abs/2207.12598
- [12]Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna, “Libritts-r: A restored multi-speaker text-to-speech corpus,” arXiv preprint arXiv:2305.18802, 2023.arxiv.org/abs/2305.18802
- [13]National Information Society Agency (NIA), “AI-Hub Multi-Speaker Speech Synthesis corpus,” https://www.aihub.or.kr/aihubdata/data/view.do?dataSetSn=542, 2021, Accessed: Sep. 07. 2026.
- [14]Takashi Shibuya, Yuhta Takida, and Yuki Mitsufuji, “Bigvsan: Enhancing gan-based neural vocoders with slicing adversarial network,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10121–10125.arxiv.org/abs/2309.02836