WorksComputer Science
Textraction: Unified Text-Guided Speaker Extraction and Enhancement for Single- and Multi-Channel Audio
Abstract
We present Textraction, a text-guided speaker extraction framework that enables the selective extraction of desired speakers or noise sources from an audio mixture using only natural-language descriptions. The proposed method operates within a single unified model that supports both single-channel and multi-channel inputs and it allows conditioning on either semantic textual descriptions of the target source or on its spatial description. Dedicated single- and multi-channel encoders feed a shared stack of time–frequency transformer blocks, which is conditioned on the prompt and on the recording configuration in every block through cross-attention and estimates the target directly from the mixture in one forward pass, without a separation stage or a source-selection stage. The system can extract an arbitrary subset of entities from a mixture, assuming up to two simultaneously active speakers and two noise sources, and interprets prompts specifying which entities to keep, which to remove, or both. In addition, we show that the framework naturally applies to a speech-enhancement setting, in which a single target speaker is present alongside background noise. To circumvent the application of computationally costly large language models used in prior work, we introduce a lightweight, task-adapted text encoder obtained by contrastively fine-tuning a compact text-retrieval model, which learns to be sensitive to instruction polarity (e.g., keep vs. remove) while remaining robust to paraphrasing. Experiments across attribute-based extraction, spatially guided extraction, and speech enhancement show that the proposed approach effectively isolates the desired entities under various acoustic and spatial conditions, and matches or surpasses dedicated spatial-extraction and enhancement baselines while offering substantially greater flexibility.
Introduction
Humans can effortlessly attend to a desired speaker in the presence of competing talkers and background noise, a capability often referred to as the cocktail-party effect. Reproducing this ability computationally is central to many speech technologies. Classical speech separation aims to decompose a mixture into all constituent sources, typically without specifying a preferred source, whereas speaker extraction focuses on isolating a particular speaker of interest while suppressing all other sources. Speaker extraction is especially appealing in user-driven scenarios (e.g., teleconferencing, hearing-assistive devices, and voice-controlled systems), where the user either implicitly or explicitly specifies which speaker to enhance.
The literature contains extensive work on both single-channel and multi-channel separation and extraction. Single-channel approaches have achieved remarkable performance with deep learning architectures that learn to disentangle sources from monaural mixtures, often under permutation-invariant training or related objectives [1, 2, 3, 4]. In parallel, multi-channel methods exploit spatial diversity offered by microphone arrays, incorporating spatial features or beamforming-inspired modules to improve robustness in reverberant and noisy environments [5, 6, 7, 8]. For target speaker extraction [9], a common paradigm is to condition the model on an auxiliary cue that identifies the target, such as a short enrollment utterance [10, 11, 12], visual information [13, 14, 15], or spatial cues derived from an array geometry [16, 17, 18, 19, 20, 21]. While effective, these cues are not always available in practice: enrollment audio may be missing, cameras may not exist, and multi-channel recordings are not guaranteed.
Recently, text-guided conditioning has emerged as a flexible and powerful paradigm for speech and audio processing, enabling semantic control over complex acoustic tasks. Instead of relying solely on acoustic references, text prompts provide a compact and expressive mechanism for specifying target attributes, speakers, or sound events.
Early works explored text guidance for sound extraction rather than explicit speaker extraction. For example, Yuan et al. [22] proposed extracting sounds using text prompts, where a frozen Flan-T5 model was employed to generate text embeddings. Jiang et al. [23] applied text prompts in a diarization setting; however, the method only indicates when a speaker is active and does not perform speech extraction.
Several studies have investigated textual cues tailored to specific application scenarios. In [24], dialogue history was used as a textual cue for human–AI question answering. In [25], slide or poster content was used for text enrollment without temporal alignment with the audio, targeting meetings, presentations, lectures, and poster sessions. Both studies adopted a pretrained Contrastive Language–Audio Pretraining (CLAP) model [26] as the text encoder. Similarly, Rahimi et al.
This work was supported by... A. Eisenberg is with Bar-Ilan University, Israel, and with OriginAI, Israel, e-mail: [email protected]; S. Gannot is with Bar-Ilan University, Israel, e-mail: [email protected]; and S. E. Chazan is with OriginAI, e-mail: [email protected]. al. [27] assumed that the exact linguistic content of the target speaker’s utterance is known a priori and used it as enrollment information.
More recently, several text-guided speaker extraction methods have been proposed for single-channel scenarios [28, 29, 30, 31, 32, 33]. Hao et al. [28] introduced a free-text-guided framework in which the desired speaker could be specified via spectral features, textual transcription, or an auxiliary waveform. However, its mixtures contain only two speakers, without environmental noise sources. Jiang et al. [32] considered mixtures of two speech and two non-speech sources, but did not explicitly simulate or evaluate controlled reverberant conditions. Dai et al. [31] proposed the use of relative textual references (e.g., “older,” “second,” “louder”) to identify the target speaker.
The aforementioned text-guided methods primarily operate in single-channel settings, relying on spectral or semantic cues for target speaker identification. A notable exception is the work of Zhu et al. [34], who proposed a multi-channel target speaker extraction framework guided by both semantic and spatial queries. In this framework, spatial queries specify pre-defined absolute regions in the microphone-array coordinate system, while semantic text queries, embedded with a frozen CLAP text encoder, describe speaker attributes and coarse spatial locations. However, the framework does not model relative spatial relations between speakers.
Despite these advances, several limitations remain. First, no existing text-guided speaker extraction model can seamlessly operate in both single-channel and multi-channel scenarios within a unified framework. Second, most prior approaches rely on large pretrained language models for text embedding, which are computationally expensive and typically require parameter-efficient fine-tuning strategies such as LoRA. Conversely, approaches using frozen text encoders lack flexibility in handling diverse prompt formulations, such as specifying entities to retain, remove, or both.
In this work, we propose a unified text-guided extraction framework for both single-channel and multi-channel scenarios. Natural-language prompts specify targets through spectral cues, spatial cues, or their combinations. The framework is robust to high reverberation and multiple noise sources and can extract any entity in the mixture, including noise components, under arbitrary permutations. Free-form prompts allow users to specify which entities to preserve, suppress, or both. We train a lightweight text encoder tailored to the relevant prompt types, improving efficiency and task alignment without relying on large language models. One network, conditioned on the prompt, estimates the target from the mixture in a single forward pass, and no separation or selection stage is used.
Our main contributions are summarized as follows: 1) We propose a unified model that supports both single-channel and multi-channel text-guided extraction, effectively leveraging spectral and spatial information within a single framework. SC and MC inputs enter through dedicated encoders and share all subsequent blocks; the prompt and an SC/MC embedding condition every block, and the network maps the mixture to the target in one pass, without a separate-then-select pipeline. 2) We introduce a lightweight text encoder based on text retrieval models, which is significantly more efficient than large language models while remaining expressive enough for flexible prompt-based control. 3) For spatial extraction, we incorporate relative position cues [35] expressed in free-form text, enabling the model to exploit relative spatial relationships between speakers.
Problem Formulation
A scenario with concurrently active speakers and directional noise sources, recorded by an array of microphones in a reverberant and noisy environment, is considered. The problem is formulated in the short-time Fourier transform (STFT) domain, where and denote the frequency-bin and time-frame indices, respectively, and and are the total numbers of frequency bins and time frames. The microphone-array observation are then given by:
where is the clean (anechoic) speech signal of the -th speaker, is the acoustic transfer function (ATF) vector between the -th speaker and the microphone array, is the -th noise source, and is the corresponding noise ATF vector.
We consider two main operating modes. For the extraction task, we focus on mixtures that contain two concurrently active speakers and two directional noise sources, i.e., . For the enhancement task, we consider a single target speaker, , in the presence of either one or two directional noise sources, i.e., . Both tasks are studied under single- and multi-microphone recording setups. Specifically, we consider a single-channel configuration with , as well as a two-microphone configuration with , in which spatial information provided by the microphone array is also exploited.
Given the observation and a natural-language prompt , the goal is to estimate the reference-microphone signal of the entity (or set of entities) described by , whether that entity is a speaker or a directional noise source.
Prompt Generation
Attribute-Based Extraction
For extraction based on semantic attributes, we randomly select the desired and interfering entities in each mixture. Each configuration is represented by a binary selection vector
where denotes an entity to be preserved and denotes an entity to be removed. For example, the vector corresponds to keeping the first speaker and the first noise source while removing the second speaker and the second noise source.
For each selection vector, we construct three types of base prompts: (i) prompts that contain only entities to be preserved, (ii) prompts that contain only entities to be removed, and (iii) prompts that contain both preserved and removed entities. Noise descriptions are taken directly from the dataset metadata. Noise sources whose descriptions include references to human speakers (e.g., person, people, speech) are excluded to avoid semantic ambiguity. Speaker descriptions are defined using their attributes, including gender, relative loudness in the mixture, or a combination of both.
When speaker separation is required (i.e., configurations of the form or ), the two speakers must differ in at least one attribute. If both speakers share the same gender, one of them is randomly assigned to be louder than the other. This attribute is stored and later enforced during signal generation to ensure consistency between the audio mixture and the textual prompt.
An example of a base prompt is:
“Remove the loud man, keep the weak woman, remove the barking dog, keep the sound of the engine.”
To increase linguistic variability, the order of the prompt components is randomly shuffled. Each base prompt is then expanded using a large language model (LLM). In this work, we employ Gemma 3 [36], which is prompted to generate five semantically equivalent paraphrases for each base prompt. Since this procedure is applied independently to each of the three base-prompt types, a total of 15 distinct prompts are generated for each mixture configuration.
For each signal, we additionally generate corresponding prompts describing the interfering entities using the same procedure. We refer to such a prompt as the complementary prompt: its target is the complementary set of entities, given by the selection vector . During training, the model alternates between conditioning on prompts describing the desired entities and prompts describing the interference.
Spatially Guided Extraction
For extraction based on spatial attributes, the model is trained to extract a single desired speaker from the mixture. Following [35], the spatial descriptions are limited to a finite set of predefined textual options, which are summarized in Table I. These descriptions express relative spatial relationships between the two speakers—for example, which one is closer or more to the right—rather than absolute coordinates in the array frame. To enable free-text conditioning, we employ a large language model to generate 1,000 paraphrased prompts for each spatial description. During training, one prompt is randomly selected from the corresponding set. Since the target in this task is a single speaker without the noises, the complementary prompt of a spatial scene describes the other speaker (e.g., the speaker on the left instead of the speaker on the right), not the complementary set of all four entities.
Speech Enhancement
For the speech enhancement task, where a single target speaker is present alongside background noise, we again use an LLM to generate textual prompts. Specifically, the following base sentences are used: “keep the talking speaker”, “remove the background noise”, and “keep the talking speaker and remove the background noise”. Each sentence is rephrased 1,000 times, and during training a single prompt is randomly selected, following the same procedure as in the spatially guided scenario. Complementary prompts are used only in the extraction tasks.
Proposed Model
Figure 1 presents an overview of the proposed TEXTRAC-TION framework. The network receives the mixture and the prompt and estimates, in one forward pass, the signal described by the prompt. It is built from TF-Locofromer blocks [37]. To enable unified operation in both single-channel (SC) and multi-channel (MC) settings, the input is processed by the corresponding SC or MC encoder, while the remaining blocks are shared between the two configurations. The prompt and a learned SC/MC embedding, which informs the shared network of the active recording configuration, condition the features after the encoder and after every shared block through cross-attention. The network has no separation stage and no source-selection stage; the prompt alone determines which signal is extracted. The following subsections describe the prompt representation and the network architecture in detail.
Prompt Embedding
To guide the model toward the desired output, we condition it on free-form textual prompts. Existing approaches often rely on large language models (LLMs) as text embedders; however, such models are computationally heavy and may be unnecessary for the limited semantic scope required in our task. Since the prompts mainly describe source attributes and extraction instructions, we hypothesize that a lightweight model capable of capturing linguistic meaning is sufficient.
Accordingly, we adopt the E5 text-retrieval model [38] as our text encoder. Given an input prompt, the E5 model produces a fixed-dimensional embedding vector. However, embeddings produced by a generic retrieval model are not guaranteed to reflect the semantic distinctions required for our task. For instance, prompts such as “remove the barking dog” and “keep the barking dog” may be mapped to nearby locations in the embedding space despite conveying opposite instructions.
To address this limitation, we fine-tune the E5 model using task-specific contrastive supervision. For each prompt in our dataset, we construct a triplet consisting of an anchor prompt, a positive prompt, and a negative prompt. These prompts are generated using the Gemma 3 LLM. Given an anchor prompt, the positive example is a paraphrase with equivalent semantic meaning, generated with as little lexical overlap as possible. The negative example conveys the opposite instruction while preserving similar wording to the anchor.
For example, given the anchor prompt “Extract the loud man and the audience’s applause, remove the woman and the running engine,” a positive example is “Focus on the louder male speaker and the sound of a crowd’s cheers, while filtering out the female presence and the motor noise,” and a negative example is “Remove the loud man and the audience’s applause, extract the woman and the running engine.”
| # | Base Spatial Prompts |
| 1 | The speaker positioned on the right |
| 2 | The speaker positioned on the left |
| 3 | The speaker located farther away |
| 4 | The speaker located closer |
| 5 | The speaker who is closer and oriented more to the left |
| 6 | The speaker who is closer and oriented more to the right |
| 7 | The speaker who is farther away and oriented more to the left |
| 8 | The speaker who is farther away and oriented more to the right |
Table I. Base spatial prompts describing speaker position and orientation. Each base prompt is expanded into 1,000 free-form paraphrases.

Figure 1. Overview of the proposed TEXTRACTION model. The mixture is represented in the STFT real–imaginary domain and routed, according to the recording configuration, to the SC encoder (reference microphone) or to the MC encoder (both microphones); each encoder is a convolutional layer followed by a TF-Locoformer block. Five shared TF-Locoformer blocks follow. After the encoder and after every shared block, a cross-attention layer conditions the features on the pooled prompt embedding and the token-level states of the fine-tuned lightweight text encoder, together with a learned SC/MC embedding. A decoder outputs the target estimate .
We fine-tune the E5 model using a triplet loss objective, encouraging high cosine similarity between the anchor and positive embeddings while pushing the anchor and negative embeddings farther apart. This training explicitly enforces sensitivity to instruction polarity (e.g., keep versus remove) while maintaining semantic robustness to paraphrasing. Table II presents the cosine similarity scores before and after fine-tuning. While the pre-trained E5 model assigns similarly high similarity scores to both positive and negative prompts, the fine-tuned model successfully distinguishes between them, demonstrating its suitability for text-guided extraction.
| Positive | Negative | |
| E5 (pre-trained) | 0.9565 | 0.9889 |
| E5 (fine-tuned) | 0.9911 |
Table II. Cosine similarity between anchor prompts and their positive and negative counterparts, before and after fine-tuning the E5 model. After fine-tuning, opposite-polarity prompts are pushed apart while paraphrases remain close.
The triplet-trained E5 model initializes the text encoder of TEXTRACTION, and it is fine-tuned jointly with the extraction network at a low learning rate. The extraction network receives two representations of the prompt. The first is the pooled E5 embedding , where denotes the embedding dimension: the L2-normalized average of the token states, which is the representation shaped by the triplet training. The second is the sequence of token-level hidden states of E5, which lets the network attend to individual words of the prompt.
Architecture
The proposed single-channel (SC)–multi-channel (MC) text-guided extraction network operates in the STFT domain using a real–imaginary (RI) representation. Given a mixture tensor , where is the batch size, is the number of input channels ( for the RI parts of the reference microphone and for the RI parts of both microphones), and are the frequency-bin and time-frame dimensions, the model outputs an estimate of the target in RI form, which is inverted by the inverse STFT to give the waveform estimate .
SC and MC encoders. To support both recording configurations within one model, we use two encoders of identical topology that differ only in their input dimensionality: an SC encoder operating on channels and an MC encoder operating on channels. Each encoder consists of a 2-D convolutional layer, which maps every time–frequency bin to an embedding of dimension , followed by group normalization and one TF-Locoformer block [37]. A binary indicator selects the encoder per example: every example passes only through the encoder of its own configuration, and the encoder outputs are merged back into one batch. The MC encoder can therefore exploit the inter-microphone phase and level differences, while SC examples are processed without padding the missing microphone.
Shared blocks. The encoder output is processed by five shared TF-Locoformer blocks. Each block alternates between modeling along the frequency axis and along the time axis, and it combines multi-head self-attention with convolutional feed-forward modules, which capture both local and global context. A transposed-convolution decoder maps the final features to the two RI channels of the target.
Conditioning. The conditioning sequence consists of the projected pooled prompt embedding, the E5 token states, and a learned SC/MC embedding that indicates whether the current example contains multi-channel spatial information; all its elements have dimension and are layer-normalized. The features are conditioned after the encoder and after each of the five shared blocks, at six locations in total. At each location, a cross-attention layer lets every time–frequency bin of the mixture feature map , i.e., the output of the encoder or of the preceding shared block, act as a query that attends to the conditioning sequence (the keys and values), and the result is added residually to . Rather than explicitly defining how the prompt and the recording configuration interact with the audio, the attention mechanism learns these relationships. Because the conditioning enters every block, the network decides at every depth which parts of the mixture to keep, and no explicit separation into sources is needed.
Initialization. The encoders, the shared TF-Locoformer blocks, and the decoder are initialized from a four-source TF-Locoformer separator that was trained with permutation-invariant training on the same data: the block of each encoder from the separator’s first block, and the five shared blocks from its remaining blocks. The separator is used only for initialization and is not a stage of the model. The decoder is initialized from the separator’s decoder by summing its weights over the four sources, the second-microphone input channels of the MC encoder are initialized to zero, and the output layers of all cross-attention layers are initialized to zero, so that at initialization the conditioning layers are the identity.
Objective Functions
We represent all signals by the real and imaginary (RI) parts of their STFT, which serve both as the network input and as the training target. The time-domain estimate is obtained by applying the inverse STFT to the network output. The training objective has three terms: an SI-SDR term, a log-magnitude term, and a Contrastive Language-Audio Pretraining (CLAP) term. The main criterion is the SI-SDR between the target signal and the estimate :
Here is the projection of onto , so the scale of does not change the measure. The loss is the negative SI-SDR improvement over the unprocessed mixture at the reference microphone:
The first term does not depend on the network, so it has no effect on the gradients. It only shifts the loss so that means no improvement over the mixture.
To encourage accurate spectral envelopes and stabilize optimization, we add an MSE loss on the log-magnitude spectra,
where .
In examples whose prompt keeps exactly one noise source, we additionally enforce consistency between the estimate and the textual description of that noise. We feed the estimated waveform , resampled to 48 kHz, to the audio encoder of a pretrained, frozen CLAP model and the dataset caption of the kept noise clip to its text encoder, obtaining the embeddings and , and maximize their cosine similarity:
The full training loss is
where equals 1 when the target is a single noise source and 0 otherwise. short-time objective intelligibility (STOI) is used only as an evaluation metric.
Prompt pairing. In the extraction tasks, every mixture appears in the batch twice, once with its prompt and once with its complementary prompt (Section III). The two examples have identical audio and different targets, which forces the network to rely on the text in order to reduce the loss.
Experimental Setup
Dataset
We generate a simulated text-guided extraction dataset composed of mixtures containing speech and environmental noise. For each mixture, we randomly select two speech utterances from LibriSpeech and two noise clips from VGGSound. To avoid semantic ambiguity between “noise” and “speech,” we exclude VGGSound samples whose captions indicate human speech activity (e.g., person talking, people speaking). Reverberant multi-microphone recordings are synthesized using a room-simulation framework based on the image method [39]. For each simulated room, the microphone-array position and its azimuth (with respect to the wall) are randomly selected, while enforcing a minimum distance of 0.5 m from the nearest wall. The reverberation time () is uniformly sampled in the range 0.3–0.8 s. We use a two-microphone array with 8 cm spacing. The two speakers and the selected noise sources are placed at random positions in the room; their distances (radii) from the array are uniformly sampled between 1 and 4 m, and the direction of arrival (DOA) of each source is measured relative to the array center and sampled on a grid over .
During training, mixtures and their corresponding targets are generated on the fly. For each mixture, we randomly sample signal-to-noise ratio (SNR) levels in the range 0–10 dB and signal-to-interference ratio (SIR) levels in the range 5–10 dB. When the textual prompt specifies a loudness difference between the two speakers, the noise signals are normalized with respect to the weaker speaker to preserve the intended loudness ordering in the mixture. In addition to the extraction setting with two speakers and two noises, we also generate an enhancement setting in which only a single randomly chosen speaker is present, mixed with background noise under the same simulation procedure. The dataset consisted of 70,000 training samples, 1,000 validation samples, and 1,000 test samples. During training, each example is an enhancement example with probability 0.1 and a two-microphone example with probability 0.5, and 75% of the two-microphone extraction examples use a spatial prompt.
Implementation Details
All audio signals are sampled at 8 kHz. The STFT is computed with 256-sample frames (rectangular window) and a hop size of 128 samples, giving frequency bins. Each encoder contains one TF-Locoformer block, followed by five shared blocks, all with embedding dimension ; the cross-attention layers use four heads. Text embeddings are generated using the e5-small-v2 encoder (), which is first contrastively fine-tuned using a triplet margin loss with a margin of 0.5 and then fine-tuned with the extraction network. The complete extraction network contains approximately 52 million parameters, 33 million of which belong to the text encoder. The model is trained using the AdamW optimizer with learning rates of for the acoustic layers, for the conditioning layers, and for the text encoder, all with a 1,000-step linear warmup, and with gradient clipping at 1. Each batch contains 30 mixtures, each used with two prompts (60 examples). Training uses bf16 mixed precision and gradient checkpointing, and an exponential moving average (EMA) of the weights with decay 0.999 is kept and used for evaluation. The model is trained on a single NVIDIA H200 GPU.
Baselines
We compare our method against established spatial-extraction systems and strong speech-enhancement models. For spatially guided extraction, we compare against SSF, a spatially selective nonlinear filter [16], and SpatialNet [6], a state-of-the-art multichannel network for joint separation, denoising, and dereverberation. SSF is conditioned on the oracle DOA of the target. SpatialNet receives no target cue; it is used unmodified as a two-output separator, trained from scratch on our two-microphone spatial-extraction mixtures with a permutation-invariant SI-SDR loss and with reverberant targets, as for Textraction. At test time, each prompt is assigned the SpatialNet output given by the oracle permutation, i.e., the assignment of the two outputs to the two reference speakers that maximizes SI-SDR. Both spatial baselines therefore use oracle information that Textraction does not have. For speech enhancement, we compare to HDDEMucs-TF [40] in the SC setting and to TF-GridNet [1], a state-of-the-art multi-channel time-frequency network, in the MC setting. TF-GridNet receives both microphones and is trained on our MC enhancement data from scratch, and it is evaluated on the same test items as our model. For a fair comparison, all models were trained or fine-tuned on our dataset.
Evaluation Metrics
We report four complementary metrics. SI-SDR (in dB) measures the fidelity of the estimated waveform to the reference and is invariant to overall scale. STOI estimates speech intelligibility on a scale. DNSMOS P.835 [41] is a non-intrusive estimate of perceived quality; we report its overall score (OVRL) on a 1–5 scale. The word error rate (WER, in %, lower is better) is obtained with the Whisper large-v3 recognizer [42] after Whisper’s English text normalization. The test items are crops of LibriSpeech utterances, so the WER of the mixture and of each output is computed against the Whisper transcript of the reference signal, and it is reported only for targets that contain exactly one speaker (with or without noise). All signals are sampled at 8 kHz and upsampled to 16 kHz for both measures, which lowers the absolute DNSMOS values of all signals, including the clean reference (OVRL of about 2.05). For each comparison, we also report the corresponding input (unprocessed mixture) values to explicitly indicate the improvement. Each test scenario of the extraction tasks is evaluated with both its prompt and its complementary prompt. Every item has a fixed random seed, so all results are computed on identical signals.
Experimental Results
We evaluate the proposed Textraction framework in five settings: (i) multi-channel (MC) attribute-based extraction, (ii) single-channel (SC) attribute-based extraction, (iii) spatially guided extraction, (iv) MC speech enhancement, and (v) SC speech enhancement. Table III summarizes the overall results, and Tables IV and V provide per-configuration breakdowns for MC and SC extraction. Table X breaks down attribute-based extraction by prompt formulation: All (retain and suppress), Keep (retain-only), and Remove (suppress-only).
| Task | Prompt | Items | SI-SDR ↑ | STOI ↑ | DNSMOS ↑ | WER ↓ |
| MC Extraction | All | 620 | 10.83 | 0.792 | 1.63 | 35.6 |
| MC Extraction | Keep | 708 | 11.11 | 0.800 | 1.61 | 32.3 |
| MC Extraction | Remove | 670 | 11.37 | 0.799 | 1.58 | 32.1 |
| SC Extraction | All | 667 | 10.63 | 0.780 | 1.61 | 36.7 |
| SC Extraction | Keep | 642 | 11.11 | 0.796 | 1.61 | 33.2 |
| SC Extraction | Remove | 689 | 10.77 | 0.792 | 1.60 | 33.4 |
Table X. Attribute-based extraction by prompt formulation. WER is computed over the items whose target contains a single speaker.
Attribute-Based Extraction
As shown in Table III, in MC extraction, SI-SDR improves from approximately 0 dB at the input to 11.11 dB, with STOI increasing from 0.575 to 0.797. In SC extraction, SI-SDR reaches 10.83 dB, demonstrating that semantic conditioning alone provides strong separation capabilities even in the absence of spatial information. The MC and SC configurations are within 0.3 dB of each other on attribute-based extraction. The semantic prompts in this task can be resolved from spectral cues alone, and the second microphone contributes mainly to the spatially guided task (Section VI-C). For targets that contain a single speaker, the WER drops from about 76% at the input to 33.3% (MC) and 34.4% (SC). The input WER is high because the recognizer also transcribes the competing speaker.
| Task | Input SI-SDR | Input STOI | Input DNSMOS | Input WER | TEXTRACTION SI-SDR | TEXTRACTION STOI | TEXTRACTION DNSMOS | TEXTRACTION WER |
| MC Extraction | -0.03 | 0.575 | 1.54 | 76.1 | 11.11 | 0.797 | 1.61 | 33.3 |
| SC Extraction | -0.03 | 0.575 | 1.53 | 75.8 | 10.83 | 0.789 | 1.61 | 34.4 |
| Spatial Extraction | -3.83 | 0.474 | 1.41 | 100.8 | 10.53 | 0.857 | 2.36 | 29.2 |
| MC Enhancement | 5.70 | 0.792 | 1.71 | 22.1 | 18.66 | 0.960 | 2.30 | 8.2 |
| SC Enhancement | 5.55 | 0.790 | 1.70 | 21.1 | 18.15 | 0.955 | 2.33 | 9.7 |
| Average | 1.47 | 0.641 | 1.58 | 59.2 | 13.86 | 0.872 | 2.04 | 22.9 |
Table 3. Overall results across the five settings on the test items. SI-SDR (dB), STOI, and DNSMOS OVRL: higher is better; WER (%): lower is better. In the attribute-based extraction tasks, WER is computed over the items whose target contains a single speaker.
| Action Category | Input SI-SDR | Input STOI | Input DNSMOS | Input WER | TEXTRACTION SI-SDR | TEXTRACTION STOI | TEXTRACTION DNSMOS | TEXTRACTION WER |
| Both Noises | -3.33 | 0.462 | 1.42 | – | 9.01 | 0.723 | 1.18 | – |
| Both Targets | 3.29 | 0.670 | 1.42 | – | 12.14 | 0.855 | 1.88 | – |
| Both Targets + Noise | 7.92 | 0.813 | 1.47 | – | 13.13 | 0.877 | 1.45 | – |
| Single Noise | -7.94 | 0.281 | 1.47 | – | 3.61 | 0.480 | 1.17 | – |
| Single Target | -3.88 | 0.497 | 1.61 | 81.5 | 11.23 | 0.843 | 2.27 | 36.3 |
| Target + Both Noises | 3.75 | 0.713 | 1.61 | 66.9 | 15.32 | 0.916 | 1.56 | 35.6 |
| Target + Noise | 0.00 | 0.579 | 1.57 | 77.3 | 11.24 | 0.825 | 1.66 | 32.7 |
Table 5. Per-configuration SC extraction results. SI-SDR (dB), STOI, and DNSMOS OVRL: higher is better; WER (%): lower is better, reported only when the target contains a single speaker.
The detailed breakdowns in Tables IV and V reveal how difficulty varies across target configurations. Mixed selections such as Target + Both Noises reach 15.27 dB (MC) and 15.32 dB (SC), and extracting both speakers reaches 12.93 dB (MC) and 12.14 dB (SC). In Single Target, one speaker must be taken out of a mixture with a second speaker and two noises; the model reaches 11.70 dB (MC) and 11.23 dB (SC), an improvement of about 15 dB over the input. As expected, extracting a single directional noise source from a heavily degraded input (input SI-SDR dB) remains the hardest case; nevertheless, the model still improves the input by more than 11 dB. DNSMOS rates any non-speech energy as a degradation, including noise that the prompt asks to keep. It is therefore informative mainly for speech-only targets: for Single Target it rises from 1.61 to 2.30 (MC) and 2.27 (SC), and for Both Targets from 1.45 to 1.92 (MC) and from 1.42 to 1.88 (SC), whereas outputs that correctly contain the requested noise receive low scores (1.15–1.20 for noise-only targets).
| Action Category | Input SI-SDR | Input STOI | Input DNSMOS | Input WER | TEXTRACTION SI-SDR | TEXTRACTION STOI | TEXTRACTION DNSMOS | TEXTRACTION WER |
| Both Noises | -3.32 | 0.464 | 1.45 | – | 9.40 | 0.730 | 1.20 | – |
| Both Targets | 3.28 | 0.668 | 1.45 | – | 12.93 | 0.874 | 1.92 | – |
| Both Targets + Noise | 7.93 | 0.813 | 1.49 | – | 13.27 | 0.877 | 1.43 | – |
| Single Noise | -7.95 | 0.282 | 1.49 | – | 3.99 | 0.491 | 1.15 | – |
| Single Target | -3.58 | 0.503 | 1.61 | 81.7 | 11.70 | 0.856 | 2.30 | 33.4 |
| Target + Both Noises | 3.48 | 0.709 | 1.61 | 67.6 | 15.27 | 0.917 | 1.53 | 34.8 |
| Target + Noise | -0.03 | 0.580 | 1.57 | 77.6 | 11.47 | 0.833 | 1.68 | 32.4 |
Table 4. Per-configuration MC extraction results. SI-SDR (dB), STOI, and DNSMOS OVRL: higher is better; WER (%): lower is better, reported only when the target contains a single speaker.
Semantic Fidelity of Single-Noise Extraction
For noise targets, SI-SDR is not an adequate measure: it rewards exact waveform reconstruction, whereas the identity of noise-like sound events lies in their spectro-temporal envelope, so perceptually correct outputs can still score low (about 3.8 dB here, from an input of about dB). To assess whether the extracted signal is nevertheless the requested sound event, we measure its semantic similarity in the joint text–audio embedding space of EmbeddingGemma 2 [43], a multimodal embedding model that maps text and audio into a single 768-dimensional space. This model was not used in training. We evaluate the 640 single-noise items (SC and MC extraction test sets, prompt and complementary prompt), and, in addition, the 244 items whose prompt asks for both noise sources (Both Noises). Each waveform is resampled to 16 kHz and embedded without a prefix, and each noise caption is embedded with the retrieval-query prefix. We report two measures: the cosine similarity between the output embedding and the embedding of the reference noise signal, and the cosine similarity between the output embedding and the caption of the requested noise. For Both Noises, the reference is the sum of the two noise signals, and the caption similarity is averaged over the two noise captions. Table VI summarizes the results, together with the unprocessed mixture and the clean reference noise.
Table VI. NOISE EXTRACTION ON THE SC AND MC TEST ITEMS (640 SINGLE-NOISE AND 244 BOTH-NOISE ITEMS): COSINE SIMILARITY IN THE EMBEDDINGGEMMA 2 SPACE BETWEEN THE OUTPUT AND THE REFERENCE NOISE AUDIO (REF.) AND THE REQUESTED NOISE CAPTIONS (CAP.). HIGHER IS BETTER. AUDIOSEP AND CLAPSEP USE THEIR RELEASED WEIGHTS AND RECEIVE THE REQUESTED NOISE CAPTION AS THE QUERY.
We compare with two off-the-shelf language-queried sound separation models, AudioSep [44] and CLAPSep [45], using their released pretrained weights without retraining. Both are single-channel models trained at 32 kHz on general sound-event data. They receive the first-microphone mixture resampled to 32 kHz, and their output is resampled back to 8 kHz before scoring. Because these models are trained with short sound-event descriptions rather than instructions, each receives the caption of the requested noise as its text query (for Both Noises, the two captions joined by “and”), whereas TEXTRACTION receives the dataset prompt. CLAPSep also accepts a negative query; using the removed sources as the negative query changed its cosine similarities by at most 0.02, so we report it with the positive query only.
The model moves the output far from the mixture toward the requested noise. In the audio embedding space, the similarity to the reference noise rises from 0.771 for the mixture to 0.950. The caption similarity rises from 0.629 for the mixture to 0.681, which is close to that of the clean reference noise (0.688). The same holds when both noises are requested: the similarity to the reference rises from 0.786 for the mixture to 0.967, and the caption similarity of the output (0.672) is close to that of the clean reference (0.676).
Both language-queried baselines also move the output toward the requested noise, but much less. AudioSep, the stronger of the two, reaches a similarity to the reference of 0.898 for a single noise and 0.908 for both noises, and CLAPSep reaches 0.854 and 0.861, compared with 0.950 and 0.967 for TEXTRACTION. Their caption similarity is also lower (0.677 for AudioSep and 0.660 for CLAPSep for a single noise, versus 0.681). This gap is consistent with a domain mismatch: the test mixtures are reverberant 8 kHz recordings that contain two competing speakers in addition to the two noise sources, a setting that general-purpose sound separators were not trained for.
Spatially Guided Extraction
Table VII presents a direct comparison of the proposed methodology with SSF and SpatialNet on the spatial extraction task. Although TEXTRACTION is designed as a general text-guided extraction framework rather than a spatially specialized separator, it consistently outperforms the evaluated spatial baselines in both SI-SDR and STOI across all reported spatial conditions.
| Method | Action Category | SI-SDR ↑ | STOI ↑ | DNSMOS ↑ | WER ↓ |
| Input | Both | -3.99 | 0.466 | 1.42 | 101.3 |
| Input | DOA gap | -3.75 | 0.475 | 1.41 | 104.3 |
| Input | Radius gap | -3.75 | 0.480 | 1.41 | 96.7 |
| SSF | Both | 5.00 | 0.689 | 1.75 | 66.2 |
| SSF | DOA gap | 3.87 | 0.650 | 1.79 | 57.4 |
| SpatialNet | Both | 5.70 | 0.71 | 1.98 | 62.4 |
| SpatialNet | DOA gap | 5.70 | 0.71 | 1.96 | 59.2 |
| SpatialNet | Radius gap | 3.80 | 0.69 | 1.96 | 63.9 |
| TEXTRACTION | Both | 10.67 | 0.856 | 2.36 | 31.6 |
| TEXTRACTION | DOA gap | 10.41 | 0.856 | 2.32 | 28.0 |
| TEXTRACTION | Radius gap | 10.52 | 0.857 | 2.41 | 28.1 |
Table VII. SPATIALLY GUIDED EXTRACTION RESULTS. SI-SDR (DB), STOI, AND DNSMOS OVRL: HIGHER IS BETTER; WER (%): LOWER IS BETTER. SSF IS CONDITIONED ON THE ORACLE DOA OF THE TARGET; IT HAS NO “RADIUS GAP” ROW BECAUSE A DOA CUE CANNOT DISTINGUISH SPEAKERS THAT DIFFER ONLY IN DISTANCE. SPATIALNET SEPARATES BOTH SPEAKERS WITHOUT A CUE, AND ITS OUTPUTS ARE ASSIGNED TO THE PROMPTS BY THE ORACLE PERMUTATION.
We evaluate spatially guided extraction under three spatial configurations, depending on the relative position of the target and interfering speakers. “Both” denotes scenarios in which the speakers differ in both DOA and radial distance, with gaps of at least and 0.5 m, respectively. “DOA gap” and “Radius gap” denote scenarios in which the speakers differ only in azimuth by at least or only in radial distance by at least 0.5 m, respectively. Figure 2 illustrates the three configurations. Under joint azimuth and distance discrimination (“Both”), TEXTRACTION reaches 10.67 dB, compared with 5.70 dB for SpatialNet and 5.00 dB for SSF. In the azimuth-only condition (“DOA gap”), it again achieves the strongest separation among the evaluated systems (10.41 dB). In the distance-only scenario (“Radius gap”), which is the most challenging condition for the baselines, TEXTRACTION reaches 10.52 dB, more than 6.5 dB above SpatialNet. SSF is not evaluated in this condition: it is conditioned on the DOA of the target, so when the two speakers share the same DOA and differ only in distance, its cue cannot identify the

Figure 2. The three spatial configurations of the spatially guided extraction test set. The target speaker (T) and the interfering speaker (I) differ in both DOA and distance (“Both”), only in DOA (“DOA gap”), or only in distance from the two-microphone array (“Radius gap”). Two directional noise sources are present in every scene.
target. The performance of TEXTRACTION is nearly uniform across the three conditions, so distance-based prompts are resolved about as well as azimuth-based prompts. In all three conditions, STOI reaches about 0.86, compared with 0.69–0.71 for SpatialNet and 0.65–0.69 for SSF. The output is also more intelligible and of higher perceived quality: its WER is 28.0–31.6%, compared with 59.2–63.9% for SpatialNet and 57.4–66.2% for SSF, and its DNSMOS is 2.32–2.41, compared with 1.96–1.98 and 1.75–1.79, respectively. The input WER exceeds 100% because the recognizer transcribes both speakers. Importantly, whereas SSF and SpatialNet are spatially specialized architectures that rely on oracle information (SSF on the target DOA, SpatialNet on the oracle assignment of its two outputs), so that neither has to identify the target itself, TEXTRACTION operates within a unified SC–MC framework, supports free-form natural-language spatial descriptions, and simultaneously enables semantic attribute-based conditioning. Thus, the proposed approach not only achieves superior spatial extraction performance compared with the other methods but also offers substantially greater flexibility.
Figure 3 breaks the spatial results down by how far apart the two speakers are. In DOA, TEXTRACTION reaches 9.99 dB SI-SDR for gaps of only 10°–20° and 10.8–11.3 dB for gaps beyond 60°, so speakers that are close in direction remain separable, and a wider angular separation adds only about 1 dB. In distance, TEXTRACTION is nearly flat at 10.3–10.8 dB over the whole range of radius gaps (0.5–1.0 m), so prompts that refer to a closer or farther speaker are resolved as reliably for a 0.5 m difference as for a 1 m one.

Figure 3. Spatially guided extraction SI-SDR as a function of the spatial separation between the two speakers. Left: DOA gap, over the scenes in which the speakers differ in DOA (“Both” and “DOA gap”); right: radius gap, over the scenes in which they differ in distance (“Both” and “Radius gap”). Each point is the mean over the test items (prompt and complementary prompt) in the bin.
Speech Enhancement
In the enhancement task (Table VIII and Table IX), TEXTRACTION achieves 18.66 dB SI-SDR / 0.960 STOI in MC and 18.15 dB SI-SDR / 0.955 STOI in SC. In the MC setting, the dedicated TF-GridNet reaches 18.15 dB SI-SDR / 0.959 STOI, so TEXTRACTION surpasses a state-of-the-art enhancement network trained for this task alone by 0.5 dB SI-SDR, with a similar STOI. In the SC setting, TEXTRACTION surpasses HDDEMUCS-TF (13.72 dB SI-SDR / 0.890 STOI) by about 4.4 dB in SI-SDR and 0.065 in STOI. In MC enhancement, the DNSMOS of TEXTRACTION is slightly higher than that of TF-GridNet (2.30 vs. 2.24), and the two reach a similar WER (8.2% vs. 7.9%), both far below the 22.1% of the input. In SC enhancement, TEXTRACTION reduces the WER from 21.1% to 9.7%, whereas the output of HDDEMUCS-TF has a higher WER than the input (25.9%), and TEXTRACTION also reaches a higher DNSMOS (2.33 vs. 2.24). This is notable because HDDEMUCS-TF and TF-GridNet are dedicated enhancement models, whereas TEXTRACTION performs enhancement as a special case of a more general text-guided extraction framework, with the same weights that also serve the four extraction tasks.
| Method | SI-SDR ↑ | STOI ↑ | DNSMOS ↑ | WER ↓ |
| Input | 5.70 | 0.792 | 1.71 | 22.1 |
| TF-GridNet | 18.15 | 0.959 | 2.24 | 7.9 |
| TEXTRACTION | 18.66 | 0.960 | 2.30 | 8.2 |
Table VIII. MC enhancement results on the same test items. WER in %.
| Method | SI-SDR ↑ | STOI ↑ | DNSMOS ↑ | WER ↓ |
| Input | 5.55 | 0.790 | 1.70 | 21.1 |
| HDDEMUCS-TF | 13.72 | 0.890 | 2.24 | 25.9 |
| TEXTRACTION | 18.15 | 0.955 | 2.33 | 9.7 |
Table IX. SC enhancement results. WER in %.
Robustness to Prompt Formulation
Table X groups the attribute-based extraction test items by the formulation of their prompt. The performance differences between “All,” “Keep,” and “Remove” prompts are small: SI-SDR differences are within 0.55 dB, STOI differences are within 0.016, and the WER ranges from 32.1% to 36.7%. “All” prompts, which contain both retain and suppress clauses, are slightly harder than the other two formulations. This confirms that the fine-tuned E5 encoder effectively captures instruction polarity, that the model is robust to linguistic variability and paraphrasing, and that both positive (retain) and negative (suppress) instructions are interpreted consistently.
Each extraction test mixture is evaluated with both its prompt and its complementary prompt, which describe different targets in the same mixture. To verify that the output is actually controlled by the text and not by an acoustic prior (e.g., “return the loudest speaker”), we swapped the prompts of each validation pair, so that every mixture was processed with the prompt of the complementary target. The SI-SDR with respect to the original target then dropped to −23.0 dB (SC extraction), −22.4 dB (MC extraction), and −33.5 dB (spatial extraction), which shows that the model switches its output to the other sound. Appendix A evaluates the robustness to typos in the prompt.
Discussion
The results demonstrate that TEXTRACTION consistently outperforms the evaluated spatial baselines while simultaneously supporting semantic and spatial text conditioning within a unified SC–MC architecture. In addition, it surpasses the dedicated SC enhancement baseline by a wide margin and matches or outperforms the dedicated MC enhancement network TF-GridNet. These results come from three design choices: a single network in which the prompt conditions every block, so that no separation or selection stage can fail; dedicated SC and MC encoders together with an SC/MC embedding, which let the shared blocks exploit spatial cues whenever a microphone array is available; and a strong time–frequency backbone initialized from a source separator. Together with its robustness to prompt formulation, these findings highlight text-guided conditioning as a flexible and competitive paradigm for speaker extraction and enhancement, capable of subsuming several specialized tasks in a single model.
Limitations and Future Work
Our study has several limitations that also point to promising directions for future work. First, all experiments are conducted on simulated reverberant mixtures; evaluating on real recordings captured by microphone arrays in reverberant acoustic environments is an important next step. Second, the extraction setting assumes at most two simultaneous speakers and two directional noise sources; scaling to more sources and to a variable, unknown number of entities remains open. Third, single-directional noise extraction from strongly degraded inputs remains challenging (about 3.8 dB SI-SDR), suggesting a need for stronger semantic grounding of noise events. Fourth, the spatial descriptions are drawn from a fixed set of relative relations; supporting richer, quantitative spatial language (e.g., approximate angles or distances) is a natural extension. Fifth, the TF-Locoformer backbone operates on every time–frequency bin and is relatively slow; a lighter backbone would reduce the inference cost. Finally, while our lightweight text encoder is efficient and polarity-aware, jointly modeling more compositional prompts (nested keep/remove clauses, counting, ordering) is an interesting future research avenue.
Conclusion
We introduced TEXTRACTION, a unified text-guided framework for speaker extraction and speech enhancement that operates across single- and multi-channel inputs within a single model. Dedicated SC and MC encoders feed shared TF-Locoformer blocks that are conditioned on the prompt and on the recording configuration in every block, and the network estimates the target in one forward pass, with no separation or selection stage. The framework accepts free-form natural-language prompts that specify targets by semantic attributes, by relative spatial relations, or by both, and that instruct the model to keep, remove, or jointly keep-and-remove arbitrary subsets of speakers and directional noise sources. Rather than relying on a large language model, we obtain conditioning embeddings from a lightweight text-retrieval encoder that is contrastively fine-tuned to be sensitive to instruction polarity while robust to paraphrasing. Across attribute-based extraction, spatially guided extraction, and speech enhancement, TEXTRACTION isolates the desired entities under diverse acoustic and spatial conditions, exploits spatial information when a microphone array is available, and matches or surpasses dedicated spatial-extraction and enhancement baselines—while being markedly more flexible. These results position text-guided conditioning as a practical and competitive interface for controllable source extraction, and motivate future work on real-array evaluation, larger and variable source counts, and richer compositional prompting.
Robustness to Typos
Prompts typed by users contain typos. To measure how TEXTRACTION handles them, we take 200 test scenes per task, each with two prompts (400 items per task, 2,000 in total), and misspell the dataset prompt in five ways. Swap exchanges two adjacent letters, Delete drops a letter, Insert adds a letter from a neighboring key, Keyboard replaces a letter with a neighboring key on a QWERTY keyboard, and Merge words removes the space to the next word. Mixed picks one of these five per word. Each type is applied to 10%, 25%, or 50% of the words of at least three letters. Two targeted conditions misspell every instruction word (e.g., keep, remove, extract, isolate) or every description word (all other words, which describe the sources), with one letter-level typo per word. Table XII shows one prompt under every condition. Because this evaluation uses a 200-scene subset of each task’s test set, the clean-prompt values in Table XI differ slightly from the full-test-set values in Table III; all conditions in this appendix are evaluated on the same subset and are directly comparable. Besides SI-SDR and STOI, we report for the three extraction tasks how often the output switched to the complementary sound: the share of outputs whose SI-SDR with respect to the target of the complementary prompt of the same mixture exceeds the SI-SDR with respect to their own target.
| Prompt | Extraction | Enhancement | ||||
| SC | MC | Spatial | SC | MC | Avg. | |
| SI-SDR (dB) | ||||||
| Clean prompt | 9.93 | 9.84 | 8.67 | 16.90 | 17.26 | 12.52 |
| Swap | 6.70 | 6.08 | 6.28 | 16.25 | 16.62 | 10.38 |
| Delete | 5.94 | 6.47 | 6.21 | 16.49 | 16.92 | 10.41 |
| Insert | 5.86 | 5.72 | 6.10 | 16.18 | 16.54 | 10.08 |
| Keyboard | 5.85 | 5.25 | 5.67 | 16.21 | 16.58 | 9.91 |
| Merge words | 7.32 | 6.87 | 7.31 | 16.74 | 17.11 | 11.07 |
| Mixed | 6.15 | 5.38 | 6.29 | 16.35 | 16.69 | 10.17 |
| Instruction words | 2.70 | 3.44 | 8.46 | 16.39 | 16.46 | 9.49 |
| Description words | -2.06 | -2.43 | -1.32 | 15.24 | 15.25 | 4.94 |
| STOI | ||||||
| Clean prompt | 0.764 | 0.767 | 0.799 | 0.944 | 0.945 | 0.844 |
| Swap | 0.688 | 0.680 | 0.748 | 0.936 | 0.936 | 0.798 |
| Delete | 0.677 | 0.692 | 0.747 | 0.937 | 0.940 | 0.799 |
| Insert | 0.679 | 0.672 | 0.743 | 0.935 | 0.935 | 0.793 |
| Keyboard | 0.676 | 0.667 | 0.734 | 0.935 | 0.937 | 0.790 |
| Merge words | 0.701 | 0.693 | 0.770 | 0.941 | 0.943 | 0.810 |
| Mixed | 0.685 | 0.671 | 0.746 | 0.936 | 0.937 | 0.795 |
| Instruction words | 0.601 | 0.621 | 0.795 | 0.935 | 0.933 | 0.777 |
| Description words | 0.488 | 0.480 | 0.576 | 0.921 | 0.920 | 0.677 |
Table XI. SI-SDR (dB) and STOI with 25% of the words misspelled, per typo type and task, and with every instruction word or every description word misspelled. Avg.: mean over the five tasks.
| Condition | Prompt |
| Clean prompt | Maintain the loud female speaker, include the bell ringing sound, keep the soft male speaker, and exclude the running motor vehicle engine. |
| Swap | Maitnain the loud female spaeker, include the bell rniging sound, keep the sotf male speaker, adn exclude the rnuning motor vehicle engine. |
| Delete | Mintain te loud fmale speaker, include the bell ringing sound, keep the soft male speaker, and exclude the running mtor vehcle engne. |
| Insert | Maintain thue loud femalpe speakoer, incluide the bell ringing sokund, keep the soft male speaker, and exzclude the running motor vehicle engine. |
| Keyboard | Maintain the loud female speaker, incluce the bell ribging sound, keep the soft male speqker, amd exclude tge running motor vdhicle engine. |
| Merge words | Maintainthe loud femalespeaker, includethe bell ringingsound, keep the soft male speaker, and exclude the running motor vehicle engine. |
| Mixed | Maintain the loud ffmale speaker, include tme bell ringing sound, keepthe soft mael speaker, and ecclude the running motor vehicle engine. |
| Instruction words | Mainain the loud female speaker, inlude the bell ringing sound, kefep the soft male speaker, and exlude the running motor vehicle engine. |
| Description words | Maintain thge lpud femle spaker, include teh belpl ringign soud, keep tge sft mawle sleaker, angd exclude tge runnign moitor vehikcle enginhe. |
Table XII. One MC-extraction test prompt under every typo condition (25% of the words for the six typo types).
Figure 4 shows SI-SDR as the share of misspelled words grows. Enhancement is almost unaffected up to 25% of the words (16.36–17.05 dB, compared with 17.08 dB for the clean prompt), because its prompts describe the same target in every scene. Extraction degrades gradually: from 9.48 dB for the clean prompt to 7.99–8.57 dB with 10% of the words misspelled and 5.59–7.17 dB with 25% (Table XI), and the share of switched outputs rises from 0.8% to 7.2–11.7% at 25%. With half of the words misspelled, extraction falls to 1.11–4.04 dB and 16.0–27.8% of the outputs switch. Merged words are the mildest typo, since both halves of a merged word remain intact, whereas Keyboard and Insert, which introduce letters that are not in the word, are the most harmful.

Figure 4. SI-SDR as a function of the share of misspelled words in the prompt, per typo type. Left: mean over SC, MC, and spatially guided extraction; right: mean over SC and MC enhancement. 0% is the clean dataset prompt.
The Instruction words and Description words rows of Table XI show which words matter. A typo in every instruction word (2.2 words per prompt on average) costs 7.23 dB in SC extraction and 6.40 dB in MC extraction, and 12.9% of the extraction outputs switch to the complementary sound, because the instruction words carry the polarity (keep versus remove) of attribute-based prompts. In spatially guided extraction and in enhancement, the instruction word does not change the target, and the same typos cost at most 0.79 dB. Typos in every description word (8.8 words per prompt) are the most harmful condition: extraction drops by 10.0–12.3 dB and 36.0% of the outputs switch, while enhancement loses about 2 dB. Misspelled descriptions remove the information that identifies the target, so that the model can no longer tell which source is requested.
References
References
- [1]Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, “Tf-gridnet: Integrating full-and sub-band modeling for speech separation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 3221–3236, 2023.arxiv.org/abs/2211.12433
- [2]K. Li, G. Chen, R. Yang, and X. Hu, “Spmamba: State-space model is all you need in speech separation,” arXiv preprint arXiv:2404.02063, 2024.arxiv.org/abs/2404.02063
- [3]K. Liu, Z. Du, X. Wan, and H. Zhou, “X-sepformer: End-to-end speaker extraction network with explicit optimization on speaker confusion,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.arxiv.org/abs/2303.05023
- [4]X. Jiang, C. Han, and N. Mesgarani, “Dual-path mamba: Short and long-term bidirectional selective structured state space models for speech separation,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5.arxiv.org/abs/2403.18257
- [5]A. Bohlender, A. Spriet, W. Tirry, and N. Madhu, “Spatially selective speaker separation using a dnn with a location dependent feature extraction,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 930–945, 2023.DOI
- [6]C. Quan and X. Li, “Spatialnet: Extensively learning spatial information for multi-channel joint speech separation, denoising and dereverberation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 1310–1323, 2024.
- [7]H. Taherian and D. Wang, “Multi-channel conversational speaker separation via neural diarization,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2467–2476, 2024.arxiv.org/abs/2311.08630
- [8]A. Cohen, D. Wong, J.-S. Lee, and S. Gannot, “Explainable DNN-based beamformer with postfilter,” vol. 33, pp. 3070–3084, 2025.arxiv.org/abs/2411.10854
- [9]K. Zmolikova, M. Delcroix, T. Ochiai, K. Kinoshita, J. Černocký, and D. Yu, “Neural target speech extraction: An overview,” IEEE Signal Processing Magazine, vol. 40, no. 3, pp. 8–29, 2023.
- [10]A. Eisenberg, S. Gannot, and S. E. Chazan, “A two-stage speaker extraction algorithm under adverse acoustic conditions using a single-microphone,” in 2023 31st European Signal Processing Conference (EUSIPCO). IEEE, 2023, pp. 266–270.arxiv.org/abs/2303.07072
- [11]T. Serre, M. Fontaine, E. Benhaim, and S. Essid, “Mtse: Multi-target speaker extraction for conversation scenarios,” in Proc. Interspeech, 2025, pp. 2970–2974.
- [12]A. Eisenberg, S. Gannot, and S. E. Chazan, “Single microphone speaker extraction using unified time-frequency Siamese-Unet,” in Proc. European Signal Processing Conf. (EUSIPCO), Aug. 2022, pp. 762–766.
- [13]Z. Pan, M. Ge, and H. Li, “Usev: Universal speaker extraction with visual cue,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 3032–3045, 2022.arxiv.org/abs/2109.14831
- [14]J. Lin, X. Cai, H. Dinkel, J. Chen, Z. Yan, Y. Wang, J. Zhang, Z. Wu, Y. Wang, and H. Meng, “Av-sepformer: Cross-attention sepformer for audio-visual target speaker extraction,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.arxiv.org/abs/2306.14170
- [15]R. Tao, X. Qian, Y. Jiang, J. Li, J. Wang, and H. Li, “Audio-visual target speaker extraction with selective auditory attention,” IEEE Transactions on Audio, Speech and Language Processing, 2025.DOI
- [16]K. Tesch and T. Gerkmann, “Multi-channel speech separation using spatially selective deep non-linear filters,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 542–553, 2023.DOI
- [17]M. Ge, C. Xu, L. Wang, E. S. Chng, J. Dang, and H. Li, “L-spex: Localized target speaker extraction,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2022, pp. 7287–7291.
- [18]C. Zorilă, M. Li, and R. Doddipatla, “An investigation into the multi-channel time domain speaker extraction network,” in IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 793–800.DOI
- [19]K. Jing, W. Zhang, and Y. Gao, “End-to-end dual-guided speech extraction in noisy multi-talker scenarios,” arXiv preprint arXiv:2507.20926, 2025.arxiv.org/abs/2507.20926
- [20]Y. Ellinson and S. Gannot, “Binaural target speaker extraction using HRTFs,” in Proc. European Signal Processing Conf. (EUSIPCO), 2026, arXiv preprint arXiv:2507.19369.arxiv.org/abs/2507.19369
- [21]——, “Hrtf-guided binaural target speaker extraction with real-world validation,” in Interspeech, 2026.arxiv.org/abs/2603.16668
- [22]Y. Yuan, X. Liu, H. Liu, M. D. Plumbley, and W. Wang, “Flowsep: Language-queried sound separation with rectified flow matching,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5.arxiv.org/abs/2409.07614
- [23]Y. Jiang, R. Tao, Z. Chen, Y. Qian, and H. Li, “Target speech diarization with multimodal prompts,” arXiv preprint arXiv:2406.07198, 2024.arxiv.org/abs/2406.07198
- [24]Z. Jiang, X. Chen, S. Wang, X. Qian, and H. Li, “Tpeech: Target speaker extraction and noise suppression with historical dialogue text cues,” IEEE Signal Processing Letters, vol. 33, pp. 351–355, 2025.
- [25]Z. Jiang, X. Qian, J. Lei, Z. Pan, W. Xue, and X.-c. Yin, “Ptse-t: Presentation target speaker extraction using unaligned text cues,” arXiv preprint arXiv:2411.03109, 2024.arxiv.org/abs/2411.03109
- [26]B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “Clap learning audio concepts from natural language supervision,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
- [27]A. Rahimi, T. Afouras, and A. Zisserman, “Reading to listen at the cocktail party: Multi-modal speech separation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10493–10502.arxiv.org/abs/2501.01518
- [28]X. Hao, J. Wu, Y. Ju, C. Xu, and K. C. Tan, “Typing to listen at the cocktail party: Text-guided target speaker extraction,” IEEE Transactions on Cognitive and Developmental Systems, 2025.arxiv.org/abs/2310.07284
- [29]M. Huo, A. Jain, C. P. Huynh, F. Kong, P. Wang, Z. Liu, and V. Bhat, “Beyond speaker identity: Text guided target speech extraction,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5.arxiv.org/abs/2501.09169
- [30]B. Shi, A. Tjandra, J. Hoffman, H. Wang, Y.-C. Wu, L. Gao, J. Richter, M. Le, A. Vyas, S. Chen et al., “Sam audio: Segment anything in audio,” arXiv preprint arXiv:2512.18099, 2025.arxiv.org/abs/2512.18099
- [31]W. Dai, A. Politis, and T. Virtanen, “Inter-speaker relative cues for text-guided target speech extraction,” arXiv preprint arXiv:2506.01483, 2025.arxiv.org/abs/2506.01483
- [32]X. Jiang, C. Han, Y. A. Li, and N. Mesgarani, “Listen, chat, and remix: Text-guided soundscape remixing for enhanced auditory experience,” arXiv preprint arXiv:2402.03710, 2024.arxiv.org/abs/2402.03710
- [33]A. Rahimi, T. Afouras, and A. Zisserman, “Say who you want to hear: Leveraging its style embeddings for text-guided speech extraction,” in Proc. Clarity 2025, 2025, pp. 53–57.DOI
- [34]X. Zhu, X. Qian, and D. Liang, “Ssdq: Target speaker extraction via semantic and spatial dual querying,” IEEE Signal Processing Letters, 2025.
- [35]A. Eisenberg, S. E. Chazan, and S. Gannot, “Spatially-guided speaker extraction based on relative position cues,” arXiv preprint arXiv:2611.01757, 2026. [Online]. Available: https://arxiv.org/abs/2611.01757
- [36]G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejcovia, A. Ramé, M. Rivière et al., “Gemma 3 technical report,” arXiv preprint arXiv:2503.19786, 2025.
- [37]K. Saijo, G. Wichern, F. G. Germain, Z. Pan, and J. Le Roux, “Tf-locosformer: Transformer with local modeling by convolution for speech separation and enhancement,” in 2024 18th International Workshop on Acoustic Signal Enhancement (IWAENC). IEEE, 2024, pp. 205–209.arxiv.org/abs/2408.03440
- [38]L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei, “Text embeddings by weakly-supervised contrastive pre-training,” arXiv preprint arXiv:2212.03533, 2022.arxiv.org/abs/2212.03533
- [39]E. A. Habets, “Room impulse response generator,” Technische Universiteit Eindhoven, Tech. Rep., vol. 2, no. 2.4, p. 1, 2006.
- [40]TODO: author list, “HDDEMUCS-TF: hybrid demucs for time-frequency speech enhancement,” 2024, TODO: fill in venue/arXiv id.
- [41]C. K. Reddy, V. Gopal, and R. Cutler, “Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,” in ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2022, pp. 886–890.arxiv.org/abs/2110.01763
- [42]A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 28 492–28 518.arxiv.org/abs/2212.04356
- [43]Google DeepMind, “Embeddinggemma 2: Model card,” https://ai.google.dev/gemma/docs/embeddinggemma/model_card_2, 2026, accessed: 2026-10-26.
- [44]X. Liu, Q. Kong, Y. Zhao, H. Liu, Y. Yuan, Y. Liu, R. Xia, Y. Wang, M. D. Plumbley, and W. Wang, “Separate anything you describe,” IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 458–471, 2025.arxiv.org/abs/2308.05037
- [45]H. Ma, Z. Peng, X. Li, M. Shao, X. Wu, and J. Liu, “CLAPSep: Leveraging contrastive pre-trained model for multi-modal query-conditioned target sound extraction,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 4945–4960, 2024.arxiv.org/abs/2402.17455