Valency Hub
alphaFeedback
PDF

WorksEngineering

XingChen-UniFE: A Unified Generative Speech Front-End via Complementary Acoustic-Semantic Modeling

Kelan Kuang, Qihan Hu, Jian Kang, Jie Li, Yongxiang Li

Published Fresh — added to the Hub within the last 30 daysv18 min read

AI contributions
Not recorded for this paper
View details

Abstract

This paper presents XingChen-UniFE, a unified large audio language model (LALM) for general speech restoration (GSR) and target speaker extraction (TSE). The proposed model employs complementary semantic and acoustic representations through a dual-stream encoder, where a self-supervised learning model provides hierarchical semantic features and a learnable acoustic branch captures fine-grained structures from complex spectra. A bidirectional cross-attention module further facilitates information exchange between the two representations. The fused representation conditions the LALM for discrete speech token generation with task and speaker information. Experiments demonstrate strong performance on both GSR and TSE and robust generalization to unseen degradation conditions.

INTRODUCTION

Real-world speech is often degraded by noise, reverberation, transmission distortion, and interfering speakers, which reduces speech quality and intelligibility. Existing systems typically use separate models for different speech restoration tasks and target speaker extraction, increasing training and deployment costs. This motivates general speech restoration with a single model.

Most general speech restoration (GSR) methods follow discriminative paradigms by predicting waveforms, spectrograms, or time-frequency masks [1, 2, 3]. Although effective for specific tasks, they usually rely on task-specific architectures and objectives, making heterogeneous restoration tasks difficult to handle within a unified framework. Severe degradation further requires modeling acoustic details, linguistic content, and speaker characteristics beyond signal reconstruction alone.

Recent advances in large audio language models (LALMs) and neural audio codecs enable speech processing to be formulated as conditional discrete token generation [4]. However, LALM-based restoration requires input representations that preserve robust semantic information and fine-grained acoustic cues. Self-supervised learning (SSL) models, such as wav2vec 2.0 [5], HuBERT [6], and WavLM [7], provide hierarchical speech representations but are not optimized for waveform reconstruction and may lose fine acoustic details. Recent SR studies therefore explore joint acoustic-semantic modeling to reduce linguistic and acoustic hallucinations. PASE [8] projects low-level SSL representations into the space of high-level phonetic representations and combines them by addition, while L3-SE [9] jointly distills acoustic and semantic representations from WavLM and uses them as continuous LM conditions. However, their acoustic representations remain largely derived from SSL features and thus lack explicit modeling of the frequency and phase structure of degraded speech. Complex spectrogram, by contrast, retains both a physically meaningful frequency axis and phase information, providing a natural representation for frequency-aware acoustic modeling. Moreover, existing acoustic and semantic representations are typically fused by simple addition or concatenation, without explicit interaction.

In this work, we propose XingChen-UniFE, a unified LALM-based model for GSR and TSE that integrates complementary acoustic and semantic representations. The proposed model employs a dual-stream encoder consisting of an SSL-based semantic branch and an acoustic branch, followed by a bidirectional cross-attention module to explicitly model the interaction between the two representations. The resulting fused representation provides the LALM with both robust linguistic cues and fine-grained acoustic details for discrete speech token generation. With task and speaker conditions, XingChen-UniFE can handle diverse speech restoration and target speaker extraction tasks in a single model.

The contributions of this work are summarized as follows: 1) We propose XingChen-UniFE, a unified LALM-based model that supports speech restoration and TSE under diverse degradation conditions. 2) We design a dual-stream acoustic-semantic encoder that combines SSL representations with complex spectral features to capture complementary linguistic and acoustic information. 3) We introduce a bidirectional cross-attention module to explicitly model interactions between acoustic and semantic representations.

Proposed Method

Overall Architecture

XingChen-UniFE formulates GSR and TSE as conditional discrete speech generation. Given degraded speech xx, a task control token, and enrollment speech rr for TSE, the model predicts acoustic and semantic tokens of the target speech.

As shown in Figure 1, the model comprises a complementary acoustic-semantic encoder, an LALM backbone, and a neural audio codec. The encoder produces the conditional representation, which guides the LALM to generate interleaved delayed tokens in an autoregressive manner. The generated tokens are then rearranged and decoded by the codec decoder.

Overall architecture of XingChen-UniFE

Figure 1. Overall architecture of XingChen-UniFE. Enrollment speech is used only for TSE.

Complementary Representation Learning

As shown in Figure 2, the complementary encoder consists of parallel semantic and acoustic branches followed by a bidirectional feature fusion module.

Overview of the complementary acoustic-semantic encoder

Figure 2. Overview of the complementary acoustic-semantic encoder. (a) Semantic and acoustic feature encoding; (b) bidirectional feature fusion.

Semantic and Acoustic Feature Encoding

The semantic branch employs a pre-trained SSL model to extract speech representations. Given the input waveform xx, we average the hidden representations from all layers of the SSL model:

Hsem=1Nw∑ℓ=1NwSSL⁡ℓ(x),(1)\mathbf{H}_{\mathrm{sem}} = \frac{1}{N_w} \sum_{\ell=1}^{N_w} \operatorname{SSL}_{\ell}(x), \tag*{(1)}

where NwN_w denotes the number of Transformer layers in the SSL model. To complement SSL features with fine-grained acoustic details, we introduce a learnable acoustic branch.

The input waveform is transformed into a complex STFT spectrogram X∈CT×321\mathbf{X} \in\mathbb{C}^{T \times321}, whose real and imaginary components are stacked and mapped through a sub-band projection. The resulting features are processed by three 2D convolutional blocks and three Conformer [10] blocks to obtain the acoustic representation Haco\mathbf{H}_{\mathrm{aco}}. The convolutional blocks reduce the frequency dimension while preserving temporal resolution.

Bidirectional Feature Fusion

To facilitate information exchange between the acoustic and semantic representations, we propose a two-layer bidirectional cross-attention fusion module. As shown in Figure 2(b), the semantic representation Hsem\mathbf{H}_{\mathrm{sem}} and acoustic representation Haco\mathbf{H}_{\mathrm{aco}} are first mapped into a shared embedding space through separate linear projections.

In each fusion layer, one branch serves as Query and the other as Key and Value, with the roles exchanged for bidirectional cross-attention. Each output is followed by a residual connection and Layer Normalization, and then by a shared FFN with another residual connection and Layer Normalization. After two fusion layers, the two outputs are concatenated and linearly projected to obtain Hfuse\mathbf{H}_{\mathrm{fuse}}.

Conditional Token Generation

During training, the target speech ss is encoded by a frozen Codec encoder into acoustic tokens A={At,k}\mathbf{A} = \{A_{t,k}\} and semantic tokens S={St,k}\mathbf{S} = \{S_{t,k}\}, where tt is the temporal index and k=0,…,K−1k = 0, \ldots, K - 1 is the codebook index. To jointly model the two token types, acoustic and semantic tokens at the same temporal position are first interleaved:

DenoiseTSETSE&Denoise
PESQSTOISBSLPSPESQSTOISBSLPSPESQSTOISBSLPS
LLaSE-G11.21874.23.835.8231.12766.81.774.6541.09161.04.708.456
UniSE1.68387.47.884.8691.66684.18.882.8141.37776.63.815.620
UniTok<u>2.100</u><u>90.14</u><u>.904</u><u>.894</u><u>2.015</u><u>83.71</u><u>.887</u><u>.797</u>1.64079.22<u>.845</u><u>.678</u>
Proposed2.13790.56.916.9302.09381.74.888.7861.703<u>78.55</u>.852.714

Table 1. Performance on Libri2Mix for Denoise, TSE, and TSE&Denoise using the mix-single, mix-clean, and mix-both subsets, respectively. Best and second-best results are bold and <u>underlined</u>.

C2t,k=At,k,C2t+1,k=St,k.(2)C_{2t,k} = A_{t,k}, \qquad C_{2t+1,k} = S_{t,k}. \tag*{(2)}

The interleaved sequence is then rearranged using the delayed pattern [11] to form the autoregressive target sequence:

C~u+k,k=Cu,k,k=0,…,K−1,(3)\widetilde{C}_{u+k,k} = C_{u,k}, \qquad k = 0, \ldots, K - 1, \tag*{(3)}

where uu denotes the position in the interleaved sequence. The fused representation Hfuse\mathbf{H}_{\mathrm{fuse}} is mapped to the LALM input space through a linear adapter to obtain the conditional representation HX\mathbf{H}_{X}. For TSE, the enrollment speech rr is processed by the same encoder and projection pipeline to obtain HR\mathbf{H}_{R}. Conditioned on HX\mathbf{H}_{X}, HR\mathbf{H}_{R}, the task control token SS, and previously generated tokens, the LALM backbone performs conditional next-token prediction,

Hu=fθ(C~<u,HX,HR,S),(4)\mathbf{H}_{u} = f_{\theta}\left(\widetilde{\mathbf{C}}_{<u}, \mathbf{H}_{X}, \mathbf{H}_{R}, S\right), \tag*{(4)}

where C~<u\widetilde{\mathbf{C}}_{<u} denotes the tokens preceding generation step uu. Based on Hu\mathbf{H}_{u}, KK parallel output heads predict the token distributions over the corresponding codebooks, with greedy decoding used during inference. The generated sequence is rearranged into time-aligned acoustic and semantic tokens and decoded by the codec decoder.

EXPERIMENTAL SETUP

Implementation Details

The proposed XingChen-UniFE adopts a decoder-only Transformer as the LALM backbone, which comprises 28 layers with a hidden dimension of 1024 and grouped-query attention [12] with 16 query heads and 8 key-value heads. The dual-stream encoder consists of a frozen semantic branch based on WavLM-Base-Plus [7] and a trainable acoustic branch, while H-Codec [13] serves as the speech codec. The LALM backbone is trained from scratch, while WavLM and H-Codec remain frozen, resulting in approximately 502 M trainable parameters. The model is optimized for 400k steps with a batch size of 64 using AdamW optimizer [14] and a Noam schedule with a 4k-step warm-up [15].

Training Data

Degraded speech is simulated on the fly. Each sample is randomly corrupted by one or more degradations, including additive noise, reverberation, interfering speech, clipping, packet loss, and bandwidth limitation. Target speech is sampled from DNS Challenge [16], Emilia [17], LibriSpeech [18], LibriHeavy [19], and AISHELL-3 [20]. Interfering speech is collected from WenetSpeech4TTS [21], VoxCeleb2 [22], and CN-Celeb [23], while noise samples are obtained from DEMAND [24], WHAM! [25], DNS Challenge, RealMAN [26], ESC-50 [27], and DESED [28]. RIRs are randomly sampled from SLR26 and SLR28 [29]. Utterances from Emilia and LibriHeavy with DNSMOS scores below 3.5 are discarded. The resulting training pool comprises approximately 10k hours of target speech, 405h hours of noise, and 4640 hours of interfering speech. All audio samples are resampled to 16 kHz and segmented into clips ranging from 3 to 15 seconds in duration.

RESULTS

We evaluate XingChen-UniFE and the baselines on GSR and TSE using PESQ [30], STOI [31], SpeechBERTScore (SBS) [32], WER, and Levenshtein Phoneme Similarity (LPS) [33]. Higher values are better except for WER.

Comparison Experiment

XingChen-UniFE is compared with three generative multi-task baselines: LLaSE-G1 [34], UniSE [35] and UniTok [36] for GSR and TSE. LLaSE-G1 and UniSE use the publicly released checkpoints provided by the original authors, whereas UniTok is reproduced by us and trained under the same conditions as the proposed model.

ModelNoiseRev.Interf.BWClipLossMixedAvg.
UniSE.754/21.8.833/11.1.884/7.1.846/10.2.873/6.4.814/13.4.335/89.1.763/22.7
UniTok.805/15.4.836/6.7.897/5.7.853/7.8.892/4.3.828/10.3.472/68.5.798/17.0
Proposed.840/10.8.859/5.1.903/5.2.881/4.8.899/3.5.852/8.8.548/55.4.826/13.4

Table 2. GSR subtask performance measured by LPS↑\uparrow / WER (%)↓\downarrow. Mixed denotes samples containing all listed distortion types.

PESQSTOISBSLPSWER
Proposed2.1490.6.916.9307.74
w/o Sem.2.0990.2.902.89213.01
w/o Aco.1.8489.1.911.9179.03

Table 4. Ablation of the acoustic and semantic branches on Libri2Mix mix-single.

Table 1 compares the models on Libri2Mix. XingChen-UniFE achieves the best results on all metrics for the denoising task, and leads in PESQ and SBS for TSE. Under the more challenging TSE&Denoise condition, it achieves the best PESQ, SBS, and LPS, demonstrating strong performance across different restoration tasks.

Table 2 further compares individual GSR subtasks. XingChen-UniFE consistently achieves the best LPS and WER, with a larger margin under the Mixed condition, indicating improved robustness to multiple degradations.

We further evaluate generalization to two unseen degradation types excluded from training, codec distortion and wind noise, as shown in Table 3. XingChen-UniFE achieves the best SBS and WER for codec distortion and the best LPS, SBS, and WER for wind noise, demonstrating strong generalization to unseen conditions.

CodecWind Noise
LPSSBSWERLPSSBSWER
LLaSE-G1.824.8558.70.834.8528.16
UniSE.900.9095.45.896.9033.91
UniTok.896.9373.29.880.9203.80
Proposed.896.9383.25.900.9273.20

Table 3. Performance under unseen degradation.

Ablation Study

Table 4 examines the contributions of the two branches. Removing the semantic branch causes larger degradation in SBS, LPS, and WER, whereas removing the acoustic branch mainly affects PESQ and STOI. As the former

metrics emphasize linguistic consistency and the latter reflect perceptual quality and intelligibility, these results demonstrate the complementary roles of the semantic and acoustic branches.

Figure 3 shows task-wise ablations of feature fusion and acoustic representation. Both simple concatenation (cat Fusion) and removal of cross-attention (w/o C.A.) degrade performance, validating explicit bidirectional interaction. Moreover, the acoustic-only variant using real and imaginary components (w/ RI) clearly outperforms its magnitude-only counterpart (w/ Mag), highlighting the importance of phase information, particularly under mixed degradations.

Task-wise ablation of the fusion and acoustic representations

Figure 3. Task-wise ablation of the fusion and acoustic representations. Bars and lines denote Δ\DeltaSBS and Δ\DeltaWER relative to XingChen-UniFE, respectively; downward displacement indicates performance degradation.

CONCLUSION

This paper presents XingChen-UniFE, a discrete speech generation model for GSR and TSE. The model integrates complementary acoustic and semantic representations through a dual-stream encoder and bidirectional cross-attention. This design jointly models fine-grained acoustic structure and semantic information. The results demonstrate the effectiveness of the proposed model and further validate the proposed acoustic-semantic modeling and fusion strategy.

References

References

  1. [1]C. Zheng et al., “Sixty years of frequency-domain monaural speech enhancement: From traditional to deep learning methods,” Trends in Hearing, vol. 27, p. 23312165231209913, 2023.DOI
  2. [2]D. Wang and J. Chen, “Supervised speech separation based on deep learning: An overview,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 26, no. 10, pp. 1702–1726, 2018.DOI
  3. [3]P. Ochieng, “Deep neural network techniques for monaural speech enhancement and separation: state of the art analysis: P. ochieng,” Artif. Intell. Rev., vol. 56, no. Suppl 3, pp. 3651–3703, 2023.
  4. [4]Z. Borsos et al., “AudioLM: a language modeling approach to audio generation,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 31, pp. 2523–2533, 2023.arxiv.org/abs/2209.03143
  5. [5]A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” vol. 33, 2020, pp. 12 449–12 460.
  6. [6]W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 29, pp. 3451–3460, 2021.arxiv.org/abs/2106.07447
  7. [7]S. Chen et al., “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE J. Sel. Topics Signal Process., vol. 16, no. 6, pp. 1505–1518, 2022.DOI
  8. [8]X. Rong, Q. Hu, M. Yesilbursa, K. Wojcicki, and J. Lu, “PASE: Leveraging the phonological prior of wavlm for low-hallucination generative speech enhancement,” in Proc. AAAI Conf. Artif. Intell., vol. 40, no. 39, pp. 32 826–32 834.
  9. [9]Z. Wang et al., “Reducing linguistic hallucination in lm-based speech enhancement via noise-invariant acoustic-semantic distillation,” arXiv preprint arXiv:2605.08608, 2026.arxiv.org/abs/2605.08608
  10. [10]A. Gulati et al., “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020.arxiv.org/abs/2005.08100
  11. [11]J. Copet et al., “Simple and controllable music generation,” Adv. Neural Inf. Process. Syst., vol. 36, pp. 47 704–47 720, 2023.arxiv.org/abs/2306.05284
  12. [12]J. Ainslie, J. Lee-Thorp, M. De Jong, Y. Zemlyanskiy, F. Lebron, and S. Sanghvi, “Gqa: Training generalized multi-query transformer models from multi-head checkpoints,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2023, pp. 4895–4901.
  13. [13]L. Chengwei et al., “Quarkaudio technical report,” arXiv preprint arXiv:2512.20151, 2025.arxiv.org/abs/2512.20151
  14. [14]I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. Int. Conf. Learn. Represent., 2019.
  15. [15]A. Vaswani et al., “Attention is all you need,” in Adv. Neural Inf. Process. Syst., vol. 30, 2017, pp. 5998–6008.arxiv.org/abs/1706.03762
  16. [16]H. Dubey et al., “ICASSP 2023 deep noise suppression challenge,” IEEE Open J. Signal Process., vol. 5, pp. 725–737, 2024.arxiv.org/abs/2303.11510
  17. [17]H. He et al., “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” in Proc. IEEE Spoken Lang. Technol. Workshop. IEEE, 2024, pp. 885–890.arxiv.org/abs/2407.05361
  18. [18]V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. IEEE, 2015, pp. 5206–5210.DOI
  19. [19]W. Kang et al., “Libriheavy: A 50,000 hours asr corpus with punctuation casing and context,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. IEEE, 2024, pp. 10991–10995.arxiv.org/abs/2309.08105
  20. [20]Y. Shi, H. Bu, X. Xu, S. Zhang, and M. Li, “Aishell-3: A multi-speaker mandarin tts corpus and the baselines,” arXiv preprint arXiv:2010.11567, 2020.arxiv.org/abs/2010.11567
  21. [21]L. Ma et al., “WeNetSpeech4TTS: A 12,800-hour mandarin tts corpus for large speech generation model benchmark,” arXiv preprint arXiv:2406.05763, 2024.arxiv.org/abs/2406.05763
  22. [22]J. S. Chung, A. Nagrani, and A. Zisserman, “Vox-Celeb2: Deep speaker recognition,” arXiv preprint arXiv:1806.05622, 2018.
  23. [23]Y. Fan et al., “Cn-Celeb: a challenging chinese speaker recognition dataset,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. IEEE, 2020, pp. 7604–7608.arxiv.org/abs/1911.01799
  24. [24]J. Thiemann, N. Ito, and E. Vincent, “The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings,” in Proc. Meet. Acoust., vol. 19, no. 1. Acoustical Society of America, 2013, p. 035081.
  25. [25]G. Wichern et al., “Wham!: Extending speech separation to noisy environments,” arXiv preprint arXiv:1907.01160, 2019.arxiv.org/abs/1907.01160
  26. [26]B. Yang et al., “Realman: A real-recorded and annotated microphone array dataset for dynamic speech enhancement and localization,” vol. 37, 2024, pp. 105 997–106 019.arxiv.org/abs/2406.19959
  27. [27]K. J. Piczak, “Esc: Dataset for environmental sound classification,” in Proc. ACM Int. Conf. Multimedia, 2015, pp. 1015–1018.
  28. [28]N. Turpault, R. Serizel, A. P. Shah, and J. Salamon, “Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,” in Proc. Detection Classification Acoust. Scenes Events Workshop, 2019.
  29. [29]T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2017, pp. 5220–5224.DOI
  30. [30]A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (PESQ)—a new method for speech quality assessment of telephone networks and codecs,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., vol. 2, 2001, pp. 749–752.
  31. [31]C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intelligibility prediction of time-frequency weighted noisy speech,” IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 7, pp. 2125–2136, 2011.DOI
  32. [32]T. Saeki, S. Maiti, S. Takamichi, S. Watanabe, and H. Saruwatari, “SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics,” in Proc. Interspeech 2024, 2024, pp. 4943–4947.arxiv.org/abs/2401.16812
  33. [33]J. Pirklbauer et al., “Evaluation metrics for generative speech enhancement methods: Issues and perspectives,” in Proc. 15th ITG Conf. Speech Commun., Aachen, Germany, Sep 2023, pp. 265–269.
  34. [34]B. Kang et al., “LLaSE-G1: Incentivizing generalization capability for llama-based speech enhancement,” in Proc. 63rd Annu. Meeting Assoc. Comput. Linguistics, 2025, pp. 13 292–13 305.arxiv.org/abs/2503.00493
  35. [35]H. Yan et al., “UniSE: A unified framework for decoder-only autoregressive lm-based speech enhancement,” arXiv preprint arXiv:2510.20441, 2025.arxiv.org/abs/2510.20441
  36. [36]C. Liu et al., “UniTok-Audio: A unified audio generation framework via generative modeling on discrete codec tokens,” arXiv preprint arXiv:2510.26372, 2025.arxiv.org/abs/2510.26372

Paper details

Publication

vlnc:W_d3h2q9uw
v1 · latestHistory (1)

published · Fresh — added to the Hub within the last 30 days

added ·

Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

CC BY 4.0

Original source

Taxonomy

  • Engineering

Authors

AI contributions

Not recorded for this paper · View statement →

Looking for related papers…
Contents