WorksComputer Science
Spatially-Guided Speaker Extraction Based on Relative Position Cues
Abstract
We present a multi-channel target speaker extraction (TSE) framework that is guided by relative spatial cues rather than explicit speaker-dependent references, such as clean enrollment utterances or precise directions of arrival (DOA). The proposed approach encodes the target speaker through discrete, interpretable relations between concurrent speakers, such as left and right or near and far, referred to as the Relative Position Cue (RPC). This relational conditioning effectively leverages spatial information while avoiding explicit DOA estimation, thereby enabling more robust operation under realistic conditions. We evaluate the proposed method across multiple datasets encompassing diverse acoustic conditions, including directional and diffuse babble noise, varying numbers of microphones, mixtures with different numbers of speakers, and dynamic scenarios involving moving speakers. Experimental results demonstrate that the RPC-based approach consistently achieves strong extraction performance and generalizes well across all tested configurations, establishing it as a practical and interpretable alternative for spatially informed speaker extraction.
INTRODUCTION
Speaker separation and extraction have been central challenges in speech processing for more than three decades. Both single-channel [1] and multichannel approaches [2] have been extensively studied. The recent adoption of deep learning has led to substantial progress in single-channel blind source separation (BSS) [3, 4, 5, 6, 7, 8].
While the primary objective of BSS is to recover all speakers present in a mixture, many practical applications require extracting only a single speaker of interest, a task commonly referred to as target speaker extraction (TSE). Existing TSE approaches typically condition the extraction model on explicit speaker-dependent references, such as clean enrollment utterances, or on geometric information, such as an estimated direction of arrival (DOA). However, these assumptions substantially limit their practical applicability. Enrollment recordings may be unavailable or acoustically mismatched to the target environment, while DOA estimates are often unreliable in reverberant, noisy, or dynamically changing acoustic scenes. As discussed in the comprehensive overview in [9], TSE systems mainly differ in the type of clue used to identify the target speaker, and each clue type entails a different trade-off between the information it carries and the effort required to obtain it. We next briefly review the main clue types.
A common approach is to condition the extraction network on a speaker embedding computed from a reference utterance of the target speaker. The embedding captures speaker-specific characteristics and guides the network to selectively extract the desired speech from the mixture [9, 10, 11, 12, 13]. More recently, spatial conditioning has emerged as an attractive alternative to speaker embeddings. In particular, Zhang et al. [14] presented a comprehensive comparison of spectral and spatial conditioning strategies, demonstrating that spatial information generally provides a more effective conditioning signal for target speaker extraction. Building on this observation, [15] proposed a systematic framework for adaptively leveraging spectral and spatial conditioning based on their inferred reliability.
A different paradigm is to perform blind multichannel speaker separation, recovering all speakers present in the mixture without requiring explicit conditioning. A prominent example is SpatialNet [16], which effectively exploits multichannel spatial information to separate all speakers in the mixture. An extension of SpatialNet for dynamic acoustic scenarios was subsequently proposed in [17].
Although blind separation methods recover all speakers in the mixture, they leave the target selection problem unresolved. Consequently, even when the user’s intention is specified spatially, an additional mechanism is required to identify the desired speaker among the separated outputs.
In multi-channel TSE, a central challenge is determining which speaker to extract. Several strategies have been proposed, differing in the type of auxiliary information used to identify the target speaker. Single-channel spectral information of the desired speaker has been used as a conditioning signal for multichannel separation models in [18, 19, 20]. However, such approaches rely on prior familiarity with the target speaker, an assumption that is often unrealistic in real-world scenarios. In [21], a joint diarization-separation was presented to overcome this limitation. First, an end-to-end diarization process is applied. Then, embeddings extracted from non-overlapping speech segments are used as reference guidance for the separation model, which estimates all speakers present in the mixture.
The most natural form of spatial enrollment is the DOA of the target speaker [22, 23]. Jing et al. [24] proposed an end-to-end framework that, given a DOA and a specified beamwidth, extracts all speech signals within the corresponding directional sector while suppressing signals outside it. Pandey et al. [25] presented a complementary approach in
This work was partially supported by the Israel Science Foundation (ISF) and the German Research Foundation (DFG) through the ISF–DFG Joint Research Program, Grant No. 1280/25.
A. Eisenberg is with Bar-Ilan University, Israel, and OriginAI, Israel, e-mail: [email protected]; S.E. Chazan is with OriginAI, Israel, email: [email protected]; and S. Gannot is with Bar-Ilan University, Israel, e-mail: [email protected]. which the model receives frame-wise DOA input, enabling rapid adaptation to moving speakers while maintaining low algorithmic latency, making it suitable for real-time applications. Another line of work uses audio samples captured at the target location as a spatial reference, either as raw recordings [26] or as estimated relative transfer functions (RTFs) [27].
It is worth noting that multimodal cues, such as text [28, 29] and visual information [30, 31, 32, 33], have also recently been used as reference enrollments to guide the extraction model. In this work, we focus on scenarios where only the audio modality is available.
In this paper, we introduce the Relative Position Cues (RPC) as a spatially defined auxiliary signal to guide the proposed separation model. Concretely, RPC encodes the relative spatial relationship between two speakers, assuming only a minimal spatial separation. This relational specification is both robust and intuitive, since the target speaker can be identified by their position relative to the interferer. We stress that the RPC specifies a relation between sources (e.g., “the target is to the right of the interferer”), rather than a region of the room or a sector relative to the array. The boundary that separates the target from the interferer is therefore not fixed; it depends on the positions of all the concurrent speakers and must be inferred by the network from the mixture itself. This is the main difference from region-based clues [34, 35], which fail when the target and the interferer reside in the same predefined region. The proposed design exploits spatial cues—widely shown to benefit speaker extraction—while avoiding explicit DOA/RTF estimation and reducing reliance on potentially unreliable enrollment utterances. Moreover, because enrollment is expressed in relative rather than absolute terms, the approach naturally generalizes to moving-speaker settings, provided the speakers’ relative configuration remains unchanged.
In real conversations, overlapped speech occupies only a fraction of the total speaking time, and practical systems must also handle overlap-free segments and a varying number of active speakers. In this work, we focus on the overlapped segments, in which the target must be selected among several concurrent speakers. The scope of this assumption and the integration of the proposed method into a complete system are discussed in § 7.
Our main contributions are summarized as follows: 1) We introduce a novel spatial enrollment strategy based on relative position cues, enabling target speaker extraction without requiring explicit DOA estimation or pre-recorded enrollment utterances; 2) We demonstrate that the proposed approach achieves strong performance across varying numbers of microphones and in mixtures containing multiple simultaneous speakers; and 3) We evaluate the proposed method under dynamic scenarios involving moving speakers, despite training exclusively on static mixtures, and demonstrate that it maintains robust extraction performance.
Related works
Beyond point-wise DOA conditioning, Gu et al. [34] introduce a more general region-based conditioning strategy.
Their model is trained to extract all speakers located within predefined spatial regions. Three types of spatial constraints are considered: angular sectors defined by DOA, circular regions defined by radius, and conical volumes that combine both DOA and radius. Directional features are derived from the similarity between inter-channel phase differences (IPD) and time-phase differences (TPD) within the query angle, while a learned embedding encodes the radius constraint. This formulation generalizes traditional DOA-based extraction by allowing the model to recover all sources in a queried region. However, an estimate of the DOA is still required.
Marković et al. proposed a similar approach in [35]. Rather than focusing on precise point locations, their approach partitions the acoustic scene into two predefined spatial regions relative to the microphone array, such as left versus right or near versus far with respect to the array center. The model is then trained to extract all speakers located within the selected region. This provides an alternative to DOA-based methods.
In contrast to these two works, the proposed approach does not rely on the absolute DOA of any speaker. Since the command is defined by the relative spatial configuration between speakers rather than their orientation with respect to the microphone array, explicit DOA estimation is unnecessary. Consequently, the proposed RPC enables separation even when all speakers are located on the same side of the microphone array, without requiring DOA information or prior knowledge of the speakers.
Region-based clues [34, 35, 36] are indeed simpler than precise source locations, but they share a common property: the region is defined with respect to the microphone array by fixed (or user-specified) angular and/or distance boundaries. Such clues are well suited for extracting all sources within a region of interest, but they cannot disambiguate two speakers that reside in the same region, e.g., two speakers that are both on the left of the array, or both closer than the distance threshold. The RPC, in contrast, has no fixed boundary: “left” is defined with respect to the interfering speaker. The two approaches also address different tasks; ReZero [34] requires the absolute query angle and/or distance of the region and returns the sum of the sources inside it, whereas the RPC returns a single speaker and requires no absolute geometric information. For this reason, we do not retrain ReZero as a baseline. Instead, in § 5-F, we quantify how often any fixed region boundary would fail to identify the target in our test data, and compare it with the behavior of the RPC-based models.
Problem Formulation
A scenario with concurrent speakers, captured by microphones in a reverberant, noisy environment, is considered. The problem is analyzed in the short-time Fourier transform (STFT) domain,
where and denote the frequency and time-frame indices, respectively. Here, and represent the total number of time-frames and frequency bands. Let represent the clean, anechoic speech signal of the -th speaker, where is a vector of the acoustic transfer functions (ATFs) relating the -th source and the microphones array, and is a vector of additive noise. This general formulation encompasses both directional and diffuse noise fields. For a single directional (point) noise source, , where is the ATFs relating the noise and the microphones array, and is the anechoic noise signal. In this case, the noise spatial covariance matrix is rank-1, and the mixture comprises point sources. For diffuse noise, e.g., babble noise generated by many talkers distributed around the array, is the sum of many reverberant contributions arriving from all directions, and its spatial covariance matrix is full-rank, such that it cannot be cancelled by spatial nulling alone. The reverberant target signal, as captured by the first microphone, is denoted . The goal is to extract the reverberant target speech signal, , from the mixture . This is achieved by utilizing spatial prior knowledge, denoted as the command , which indicates the desired speaker’s position relative to the interferer(s). Throughout this work, we assume that all speakers are concurrently active and that is known.
Proposed Model
In this section, we introduce a framework for speaker extraction that exploits relative spatial information and is compatible with a wide range of extraction architectures. We illustrate its application using two recently proposed TSE schemes. The relative spatial information is defined with respect to a reference microphone array, providing cues about both the target speaker’s angular position and distance from the array. We first introduce the RPCs for two- and three-speaker settings. We then describe how the RPCs can be integrated into existing TSE schemes.
Relative Position Cues (RPC): Two Speakers)
Speakers may differ in their lateral position (left–right) and in their distance from the array. This formulation is consistent with human perception: when two speakers are positioned in front of a listener, they can be readily distinguished by spatial cues such as horizontal separation and depth.
Assuming a two-concurrent-speaker scenario (), the spatial relationship yields eight RPCs, which can be used to identify the target speaker, as summarized in Table I. These eight options can be categorized into three groups: (1) vertical-only information, which reflects distance-related cues (#1–2, Figure 1(a)); (2) horizontal-only information, which corresponds to DOA cues (#3–4, Figure 1(b)); and (3) combined horizontal and vertical information, incorporating both directional and distance cues (#5–8, Figure 1(c)) and its mirrored and inverted versions). Here, “left” and “right” are defined from the viewpoint of the array.
Figure 1. In the two-speaker scenario, we consider three distinct spatial configurations of the speakers relative to the center of the array: (a) Both speakers share the same direction of arrival but are located at different distances from the center of the array; (b) Both speakers are equidistant from the center but differ in their directions of arrival; and (c) The speakers differ in both their distances from the array center and their directions of arrival.
A model can be trained using either the given DOA and distance, a textual prompt indicating the target speaker, or a prefix integer corresponding to each possible configuration. Given that the number of distinct cases is finite (eight in total), we opted for an index-based prompt rather than textual descriptions, as the latter were deemed unnecessarily complex for this setting.
To characterize a given scenario within the proposed categories, we define two key spatial metrics. The gap distance between the speakers is expressed as:
and the DOA distance is given by:
where denotes the distance of speaker from the center of the microphone array, and represents the DOA angle of speaker .
The speakers are considered to be radially separated if the gap distance satisfies , indicating that they are located sufficiently far from each other in terms of radial distance. Furthermore, a relative DOA separation is identified when it exceeds a threshold, i.e., , implying a significant angular displacement between the speakers. Throughout this work, we use m and .
How the two metrics are used. The metrics and are used only to assign the command to a training or test mixture; they are not given to the network. Exactly one command is assigned to each speaker in each mixture: if both and , a combined cue (#5–8) is used; if only , a distance-only cue (#1–2) is used; and if only , a direction-only cue (#3–4) is used. Note that the noise source is not treated as a source in the definition of the cue. The command is fed to the network as an integer index (Sec. IV-C); the textual descriptions in Table 1 are given for readability only.
| # | RPCs for two speakers scenario: |
| 1 | The target speaker is farther from the array than the interferer |
| 2 | The target speaker is closer to the array than the interferer |
| 3 | The target speaker is to the right of the interferer |
| 4 | The target speaker is to the left of the interferer |
| 5 | The target speaker is farther than and to the right of the interferer |
| 6 | The target speaker is farther than and to the left of the interferer |
| 7 | The target speaker is closer than and to the right of the interferer |
| 8 | The target speaker is farther than and to the left of the interferer |
Table 1. Relative speaker positions and orientations for the two-source configuration.
Limitations imposed by the thresholds. Configurations with and are not covered by the proposed cues, since the speakers cannot be reliably told apart by their
relative position. Two speakers can share the same radius (e.g., when positioned symmetrically with respect to the array broadside); such configurations, in which is arbitrarily small, do appear in our data, and in this case only a direction-only cue is used. Similarly, sources that share the same DOA are distinguished only by a distance-only cue. Finally, since a uniform linear array (ULA) cannot resolve front–back ambiguity, all sources are placed in the half-plane in front of the array, i.e., .
Relative Position Cues (RPC): Three Speakers
As the number of speakers increases, the complexity of the RPC also grows, since the spatial relationships among multiple speakers are inherently more intricate than in the case of only two speakers. In this scenario, we define the desired speaker as being isolated from the two interfering speakers either by radial distance, by DOA, or both. The two interfering speakers may be positioned very close to one another—potentially nearly indistinguishable in spatial terms—as long as the desired speaker is well separated from them in at least one of these two domains.
The spatial metrics are redefined as follows. The radial distance between speakers is expressed as:
and the DOA distance is given by:
where denotes the distance of the target speaker from the center of the microphone array, while and represent the distances of the interfering speakers, which are selected randomly. Similarly, corresponds to the DOA of the desired speaker, and and denote the DOA of the interfering speakers, which are also randomly assigned in correspondence with their respective radial distances.
As before, the same conditions must be satisfied for the desired speaker to be considered spatially separated from the interfering sources. In this setup, the command strategy is slightly modified to define the target speaker by specifying one of the nine spatial positions shown in Table II.
| # | Three speakers RPCs: |
| 1 | The target is the farthest and the left-most speaker |
| 2 | The target is the farthest speaker, between the other two |
| 3 | The target is the farthest and the right-most speaker |
| 4 | The target is the middle (in distance) and the left-most speaker |
| 5 | The target is the middle speaker in both distance and direction |
| 6 | The target is the middle (in distance) and the right-most speaker |
| 7 | The target is the closest and the left-most speaker |
| 8 | The target is the closest speaker, between the other two |
| 9 | The target is the closest and the right-most speaker |
Table II. Relative speaker positions and orientations for the three-source configuration.
These nine categories comprehensively cover the possible configurations for isolating the target speaker. An illustration of these scenarios can be found in Figure 2. In both the two- and three-speaker configurations, the user must accurately identify the desired speaker.
Figure 2. An illustration of the nine possible RPCs for the three-speaker scenario. The target speaker (in red) is defined by its relative position with respect to the two interfering sources, characterized by differences in radial distance and/or DOA: (a) The far speaker positioned on the left / center / right; (b) The middle speaker positioned on the left / center / right; (c) The near speaker positioned on the left / center / right.
Incorporating RPC Information into TSE Schemes
The proposed RPC framework is architecture-agnostic and can be integrated into a wide range of target speech extraction and speech separation systems. The key idea is to represent each RPC command using a learnable embedding and inject this representation into the network to guide the extraction process toward the desired speaker. To demonstrate the generality and effectiveness of the proposed approach, we incorporated the RPC framework into two previously published multichannel architectures: (i) our U-Net-based speaker extraction model [27], and (ii) SpatialNet [16]. The following subsections detail the modifications required to integrate RPC into each architecture.
Integration with a U-Net-Based Speaker Extraction Model
We first incorporated the proposed framework into our previously published multichannel speaker extraction model [27]. The model comprises a U-Net architecture with a self-attention mechanism in the bottleneck. The encoder consists of several Conv2D layers, each followed by batch normalization to improve training stability and generalization. A parametric rectified linear unit (PReLU) activation function is applied after each layer. Subsequently, the channel and frequency dimensions are combined, and a fully connected layer is used to reduce the resulting dimensionality. Finally, a self-attention layer is applied to model long-range dependencies in the latent representation.
The decoder employs transposed convolution layers whose dimensions match those of the corresponding encoder layers. This design enables skip connections between encoder and decoder stages, facilitating information flow and preserving fine-grained details throughout the reconstruction process.
To incorporate RPC information, each command is assigned a unique integer identifier. A learnable lookup table is jointly trained to generate a dedicated embedding for every command. The embedding corresponding to the selected command is then used to condition the mixture representation through feature-wise linear modulation (FiLM) [37]. Formally,
where denotes the mixture embedding, denotes the command embedding, and and are learned linear transformations. An illustration of the resulting architecture is presented in Figure 3a.

Figure 3. Block diagram of the proposed RPC-based target speaker extraction framework. Relative Position Cues (RPCs) are used to guide the network toward the desired (target) speaker. The lower panel shows an implementation based on SpatialNet, while the upper panel shows one based on a U-Net architecture. In the SpatialNet implementation, the command embedding is concatenated with the feature representations, whereas in the U-Net implementation it is incorporated using FiLM conditioning. These choices are not mandatory and are shown only as representative examples of how RPC information can be integrated into existing architectures.
Integration with SpatialNet
We further incorporated the proposed RPC framework into SpatialNet [16], a multichannel speech separation architecture composed of alternating narrow-band and cross-band processing blocks. The narrow-band blocks model temporal-spatial dependencies within each frequency band using a self-attention module followed by a convolution-augmented feed-forward module. The cross-band blocks operate across frequency bands and capture inter-band dependencies.
To guide the network toward the desired speaker, we associate each RPC command with a learnable embedding vector using the same embedding mechanism described above. The command embedding is concatenated with the frame-level representations before the self-attention module in each narrow-band block. Consequently, the self-attention mechanism jointly processes the acoustic representations and the command embedding, enabling the network to model dependencies between the RPC instruction and all time-frequency regions of the input signal. This allows the command information to guide the attention mechanism toward the desired speaker, as illustrated in Figure 3b.
Features and Objective Function
In this study, we employed the real and imaginary (RI) components of the STFT as both input features and output targets for the model. These components were concatenated along the channel dimension. To reconstruct the signal in the time domain, the inverse STFT was applied to the model’s output. We optimize the proposed extraction model using the time-domain scale-invariant scale-invariant signal-to-distortion ratio (SI-SDR) loss [38], a widely adopted objective for BSS. For each training mixture, both speakers are considered in turn as the target speaker, conditioned on their corresponding RPC sample. The SI-SDR between a reference signal and its estimate is defined as Eq. (7)
The resulting loss is given by:
where denotes the reverberant image of speaker at the reference microphone and is the network output when conditioned on the command of speaker . The loss is averaged over the three spatial variants of each mixture (Sec. V-A). In the three-speaker experiments, a single speaker per mixture serves as the target, and the sum in the loss reduces to a single term. Separate models are trained for each number of speakers ( and , with eight and nine commands, respectively) and for each array size (two and four microphones).
Experimental Study
This section presents the experimental evaluation of the proposed method. We begin by describing the datasets and training configuration, followed by the competing methods used as baselines. We then report results under different spatial configurations, microphone array sizes, and speaker counts, using both simulated and real room impulse responses.
Datasets
To generate training, validation, and test data, we synthetically created 20,000 training examples, 1,000 validation examples, and 1,000 test examples. The clean speech signals were sampled from the Librispeech dataset, and the noise samples were sampled from the DNS dataset. Training, validation, and test speakers are drawn from the disjoint LibriSpeech training, development, and test subsets, respectively, and the DNS noise files are split 80/10/10% among the three sets. The signal-to-noise ratio (SNR) was randomly chosen in the range of 5 to 20 dB. To simulate reverberant environments, we used a simulation framework based on the Image method [39], in which the microphone array’s location and azimuth (relative to the wall) were randomly assigned for each room. The reverberation time was randomly selected from the range 0.2 to 0.8 seconds. A ULA comprising four microphones, with an 8 cm spacing between adjacent microphones, was used. The minimum distance from the wall was set to 0.5 m. The room dimensions are drawn uniformly in m for each example, and the array is placed at a random position, such that all sources are at least 0.5 m from the walls. All sources are placed at a height of 1.5 m, in the half-plane in front of the array, on a DOA grid; the distances and DOAs are measured in the horizontal plane with respect to the array center. The two-microphone configuration uses the two middle microphones of the same array (8 cm spacing).
Each example comprises a primary speaker, a second, different speaker, and a directional background-noise source. The primary speaker is placed at a distance m from the array center and at a random DOA. The noise source is placed at a random distance and at a DOA. The same utterance of the second speaker is then rendered at three different positions, one per spatial configuration targeted by our model: (i) both a large radial gap, m, and a large DOA gap, ; (ii) a large radial gap, m, and a small DOA gap, , i.e., both speakers are located in practically the same direction; and (iii) a small radial gap, m, and a large DOA gap, . These three configurations define the three test sets (i)–(iii) used throughout this section. Since the absolute positions are random, the absolute location of a speaker does not determine its command; for example, the second speaker can be located between 0.5 and 3 m from the array, and can be either the closer or the farther speaker. During training, these spatial variants are used to generate three distinct mixtures and three corresponding command utterances per speaker, ensuring the model is exposed to all spatial dimensions of the task.
For the three-speaker dataset, a third speaker is added to each of the three spatial configurations in the same manner as the second speaker, and the primary speaker serves as the target. The commands follow Table II.
Algorithm Settings
To prepare the data, the speech and noise signals were first resampled to 8 kHz. The signals were then converted to the frequency domain by computing the STFT with a 256-sample frame size, a 50% overlap between frames, and a Hamming window. Only the first 129 frequency bins were processed due to the symmetry of the DFT. The neural network was trained on 4-second signal segments using the AdamW optimizer [40], with a learning rate of 0.0001 and a batch size of . Since each training batch includes all three spatial cases and each speaker is alternately used, the effective batch size in the two-speaker case is . This setup ensures comprehensive exposure to spatial diversity and balanced participation of each speaker across both separation and extraction objectives. In the three-speaker case, only the primary speaker serves as the target in each mixture, and the effective batch size is therefore .
Competing Methods
For comparison with existing methods, we considered FasNet [41] and SpatialNet [16], which are representative multi-channel speech separation frameworks. It is important to note that these methods address slightly different tasks: in separation, the model aims to recover all speakers from the mixture, whereas in extraction, it must also identify and recover only the target speaker. Nevertheless, since SpatialNet is a strong representative of spatial speech separation methods, we adapted it to the extraction setting by conditioning it on the target speaker embedding and retrained it on our dataset to serve as a competitive baseline. Since the separation models (FasNet and SpatialNet) do not select the target, their outputs are assigned to the speakers using the permutation that maximizes the SI-SDR, as in permutation invariant training (PIT). This oracle selection favors the separation baselines over the extraction methods, which must select the target on their own.
We also examine the spatially selective filter (SSF) framework [22], a microphone-array speaker-extraction method that uses the target speaker’s DOA as a reference. We adopt its architecture and replace the original objective with the SI-SDR loss, thereby aligning it with our proposed framework.
To assess the effectiveness of the proposed relative-reference approach, we compared our U-Net model with a variant that uses the target speaker’s absolute DOA for enrollment instead of a ratio-based command. To this end, we introduced a larger lookup table that learns a dedicated embedding for each DOA on the grid (90 entries). Aside from this change, the architecture remained identical to the original model. As for all other models, separate DOA-based models are trained for the two- and four-microphone arrays.
Finally, we compared our model with an oracle minimum variance distortion beamformer (MVDR) beamformer [42]. The MVDR beamformer is designed to estimate the target signal with minimal distortion while suppressing noise and interfering speakers. Let denote the MVDR beamformer at frequency bin , the estimated signal can be expressed as:
where denotes the Hermitian transpose. The MVDR weights are defined as
where and represent the spatial covariance matrices of the noise (including the interfering speaker) and the desired speaker, respectively. The vector is a one-hot vector specifying the reference channel. For the oracle setting considered here, we assume access to the noise and interfering sources in order to compute the MVDR weights.
Results
The scale-invariant signal-to-distortion ratio (SI-SDR) performance of the proposed models is reported in the following tables, together with the short-time objective intelligibility (STOI) [43] of the same outputs, computed at 8 kHz with respect to the same reference signal. To comprehensively evaluate our approach, we tested it under three different scenarios: (i) large gaps in both DOA and radius, (ii) small DOA gap with large radius gap, and (iii) small radius gap with large DOA gap, as defined in § 5-A. The same indices (i)–(iii) are used in all tables. The two-speaker results for four and two microphones are summarized in Table III. Entries marked by “x” are not reported: in test set (ii), the two speakers are located in practically the same direction (), so a DOA-based clue (SSF and U-Net DOA) cannot unambiguously identify the target.
| Model / Test Set | 4 Microphones (i) | 4 Microphones (ii) | 4 Microphones (iii) | 2 Microphones (i) | 2 Microphones (ii) | 2 Microphones (iii) |
| Unprocessed | -0.75/0.598 | -0.76/0.598 | -0.75/0.598 | -0.75/0.599 | -0.76/0.599 | -0.75/0.598 |
| Oracle MVDR | 8.80/0.839 | 8.07/0.822 | 8.80/0.840 | 5.44/0.737 | 4.57/0.714 | 5.47/0.737 |
| FasNet [41] | 7.47/0.733 | 6.90/0.678 | 7.60/0.735 | 6.80/0.744 | 5.40/0.711 | 6.90/0.745 |
| SpatialNet [16] | 13.20/0.922 | 12.20/0.912 | 13.20/0.922 | 12.39/0.875 | 11.80/0.866 | 12.43/0.877 |
| SpatialNet RPC | 14.20/0.911 | 13.40/0.901 | 14.10/0.909 | 10.71/0.848 | 9.00/0.811 | 10.61/0.846 |
| SSF [22] | 9.10/0.812 | x | 9.05/0.812 | 9.09/0.818 | x | 9.01/0.817 |
| Unet DOA | 9.90/0.847 | x | 10.29/0.849 | 9.69/0.827 | x | 9.85/0.832 |
| Unet RPC | 11.02/0.863 | 9.75/0.835 | 11.00/0.863 | 10.86/0.858 | 9.39/0.826 | 10.67/0.854 |
Table III.
We first discuss the four-microphone results in Table III. The oracle MVDR achieved similar SISDR scores in the large-DOA-gap scenarios, while exhibiting a modest degradation in the small-DOA-gap case. Among the separation methods, FasNet performed significantly worse than the other evaluated approaches. By contrast, SpatialNet consistently achieved strong results across all test conditions.
Among the extraction methods, the RPC-based SpatialNet model outperformed even its separation counterpart. This is particularly remarkable because speaker extraction is inherently more challenging than speaker separation, requiring both identification and isolation of a specific target speaker. Despite this added complexity, the extraction model achieved superior performance, demonstrating the effectiveness of the proposed RPC framework.
Finally, the RPC-based U-Net variants consistently outperformed their absolute DOA-based counterparts across all test sets (by 0.7–1.1 dB with four microphones and 0.8–1.2 dB with two microphones, Table III), demonstrating the benefit of using RPC commands in conjunction with the corresponding DOA information. Furthermore, DOA-based enrollment alone is insufficient to distinguish between speakers in the same direction, as it provides no information about their relative distances from the microphone array.
Since the mixtures also contain background noise, part of the improvement could, in principle, stem from noise suppression rather than from target extraction. This is not the case: an ideal noise suppressor that removes the noise but keeps both speakers yields an SI-SDR of only 0.0 dB, i.e., only 0.75 dB above the unprocessed mixture, whereas the RPC-based models reach 9.0–14.2 dB (Table III). Figure 4 illustrates this for a mixture from test set (ii), in which the two speakers are located in practically the same direction. Processing the same mixture with the commands “farther” and “closer” yields two different outputs, each containing only the commanded speaker (panels (e) and (f)), whereas noise suppression alone would retain both speakers (panel (b)).

Figure 4. Spectrograms (reference microphone) of a test mixture from test set (ii), processed by the four-microphone U-Net RPC model. Speaker A is at 1.68 m and 62°, speaker B at 0.70 m and 56°; s, SNR = 6 dB, signal-to-interference ratio (SIR) = −0.6 dB. (a) Noisy mixture. (b) Speech-only mixture (ideal noise suppression). (c), (d) Reverberant references of speakers A and B. (e) Output with the command “farther”: SI-SDR of 10.1 dB w.r.t. A and −26.5 dB w.r.t. B. (f) Output with the command “closer”: 10.2 dB w.r.t. B and −21.8 dB w.r.t. A.
We further evaluated our models in scenarios with a limited number of microphones to assess the impact on performance. In general, increasing the number of microphones provides richer spatial information that can be exploited to improve separation quality. Conversely, when only two microphones are available, models that rely heavily on spatial cues face a more challenging task.
As shown in Table III, the performance of the MVDR beamformer deteriorates when restricted to two microphones, owing to the reduced degrees of freedom in the minimization. Among the separation methods, SpatialNet continues to exhibit strong performance despite the limited spatial information. FasNet also remains relatively robust, although its performance
is slightly lower than that achieved with the four-microphone configuration.
For the extraction-based methods, our model exhibits only a small performance degradation of 0.2–0.4 dB relative to the four-microphone configuration, similar to that of the U-Net DOA model. Nevertheless, it consistently achieves the best results across all evaluated scenarios. It is also noteworthy that, unlike the four-microphone setting, the extraction variant of SpatialNet performs worse than its separation counterpart, despite sharing the same underlying architecture.
This contrasting behavior can be attributed to the fundamental differences between speaker extraction and speaker separation. Whereas separation models aim to recover all speakers present in the mixture and can leverage both spatial and spectral cues, the RPC-based extraction model is trained to primarily rely on spatial information to identify and isolate the target speaker. Consequently, its performance is closely tied to the quality of the available spatial cues. When the spatial information is rich and reliable, the model can effectively focus on and extract the desired speaker. However, with only a small number of microphones, the spatial information becomes less informative, leading to a more pronounced performance degradation. Despite this limitation, the proposed method remains superior to all competing extraction methods. The per-mixture analysis in Sec. V-G shows that the degradation of SpatialNet RPC is not caused by selecting the wrong speaker (the correct speaker is extracted in more than 98% of the cases), but by a lower extraction quality across all conditions, with the largest drop for small DOA gaps (Fig. 6). Since the U-Net-based RPC model degrades by only 0.2–0.4 dB, this behavior appears to be specific to the combination of the architecture and the conditioning mechanism rather than inherent to the RPC. A detailed study of the interplay between the conditioning mechanism, the architecture, and the array size is beyond the scope of this paper.
TABLE III: SI-SDRand STOI results for 2- and 4-microphone ULA. Test sets: (i) large DOA and radial gaps; (ii) small DOA gap, large radial gap; (iii) small radial gap, large DOA gap. x: not applicable. Each entry: SI-SDR [dB] / STOI.
The results for the three-speaker configuration, presented in Table IV, demonstrate that the proposed model maintains stable performance as the number of speakers increases. Notably, the model remains effective even when the number of speakers exceeds the number of microphones, i.e., when the available degrees of freedom are lower than the number of speakers in the mixture. When the number of microphones is reduced from four to two, the performance of the U-Net RPC model decreases by only about 0.5 dB, and that of SpatialNet RPC by only 0.3–0.4 dB, highlighting the robustness of the proposed approach.
| Model / Test Set | 4 Microphones (i) | 4 Microphones (ii) | 4 Microphones (iii) | 2 Microphones (i) | 2 Microphones (ii) | 2 Microphones (iii) |
| Unprocessed | -3.84/0.469 | -3.87/0.469 | -3.84/0.468 | -3.84/0.469 | -3.87/0.469 | -3.84/0.468 |
| Oracle MVDR | 5.70/0.747 | 4.99/0.725 | 5.65/0.746 | 1.55/0.614 | 1.03/0.595 | 1.48/0.612 |
| SpatialNet RPC | 6.70/0.768 | 5.70/0.736 | 6.07/0.751 | 4.60/0.772 | 5.40/0.744 | 5.70/0.754 |
| Unet DOA | 5.34/0.740 | x | 5.10/0.733 | 3.98/0.689 | x | 3.56/0.680 |
| Unet RPC | 6.31/0.768 | 4.77/0.719 | 5.90/0.759 | 5.87/0.751 | 4.20/0.699 | 5.42/0.740 |
Table IV.
In contrast, reducing the number of microphones from four to two results in larger performance degradation for the DOA-based model (1.4–1.5 dB) and an even more pronounced decline for the oracle MVDR beamformer (about 4 dB). These findings further demonstrate the robustness of the proposed relative-reference approach, particularly in challenging underdetermined scenarios where spatial information is limited and conventional spatial filtering methods become less effective.
TABLE IV: SI-SDRand STOI results for 3 speakers with 4 and 2 microphones. Test sets (i)–(iii) as in Table III; x: not applicable. Each entry: SI-SDR [dB] / STOI.
We also conducted experiments using real recorded room impulse responses (RIRs) obtained from [44]. The recordings were made at two possible source–microphone distances: 1 meter and 2 meters. The DOAs were sampled at 15° intervals between 0° and 180°. Here, scenarios with small DOA gaps mean the DOAs are identical, differing only in their radial distance. In [44], three reverberation times were provided: 160 ms, 360 ms, and 610 ms. Our evaluation focused on the 360 ms case. The results, presented in Table V, indicate that the proposed method achieves performance comparable to that obtained with simulated data across all tested architectures and microphone numbers.
| Model / Test Set | 4 Microphones (i) | 4 Microphones (ii) | 4 Microphones (iii) | 2 Microphones (i) | 2 Microphones (ii) | 2 Microphones (iii) |
| Unprocessed | -0.75/0.598 | -0.76/0.598 | -0.62/0.541 | -0.75/0.598 | -0.76/0.598 | -0.62/0.541 |
| SpatialNet RPC | 12.35/0.914 | 7.65/0.741 | 12.97/0.901 | 9.20/0.823 | 5.50/0.713 | 10.50/0.885 |
| Unet RPC | 9.39/0.829 | 5.24/0.736 | 9.70/0.853 | 8.80/0.804 | 4.70/0.721 | 8.44/0.766 |
Table V.
TABLE V: SI-SDR and STOI results for real data with 4 and 2 microphones.
Robustness to Diffuse Babble Noise
In all the experiments above, the background noise is a single directional source, i.e., in (1). To examine how the RPC-conditioned models behave with diffuse noise, we constructed a babble-noise version of the static two-speaker test set. For each test mixture, the directional noise was replaced by babble noise, while the target and interfering speech signals, the room, and the array were kept unchanged. The babble noise was generated by placing 20 different talkers on a circle centered at the microphone array, at the height of the speakers. The talkers were uniformly spaced over 360° (with a random rotation of the circle). The RIRs of the 20 talkers were simulated using the image method [39] in the same room, with the same reverberation time and microphone-array position as the corresponding test mixture, and the 20 reverberant talker signals were summed to form . The SNR between the two speakers and the babble noise was drawn uniformly from 10–20 dB. Neither model was retrained or fine-tuned, i.e., the models were never exposed to diffuse noise during training.
Fig. 5 depicts the magnitude-squared coherence (MSC) between two adjacent microphones ( cm), averaged over the test set. The coherence of the babble noise closely follows the theoretical coherence of a spherically isotropic noise field, , whereas the directional noise remains highly coherent across the entire frequency band. The babble noise is therefore a good approximation of a diffuse noise field, and, unlike the directional noise, it cannot be cancelled by spatial nulling.
The results are summarized in tbl:VI. All RPC-based models maintain their performance in the presence of diffuse babble noise. The U-Net RPC model degrades by at most 0.25 dB with both four and two microphones, and the two-microphone SpatialNet RPC degrades by at most 0.3 dB. The largest degradation, of approximately 0.9 dB, is observed for the four-microphone SpatialNet RPC, which nevertheless remains the best-performing model. The oracle MVDR beamformer exhibits similar trends, with a minor degradation for four microphones and a minor improvement for two microphones, where the directional noise source constitutes an additional point source in an already under-determined setting. Importantly, although the babble noise consists of speech, the models did not confuse the background talkers with the speakers to which the command refers, and the relative ordering of the three test scenarios is preserved. The RPC specifies the target with respect to the dominant, spatially coherent interfering speaker, whereas each babble talker is weak and the babble mixture lacks a distinct spatial signature. These results indicate that the proposed relational conditioning remains reliable when the directional-noise assumption is violated.
Figure 5. Magnitude-squared coherence between two adjacent microphones ( cm), averaged over the test set, for the directional (DNS) noise and the 20-talker babble noise, compared with the theoretical spherically and cylindrically isotropic (diffuse) noise fields.
| Model | Noise | 4 Microphones (i) | 4 Microphones (ii) | 4 Microphones (iii) | 2 Microphones (i) | 2 Microphones (ii) | 2 Microphones (iii) |
| Unprocessed | Dir. | -0.75/0.598 | -0.76/0.598 | -0.75/0.598 | -0.75/0.599 | -0.76/0.599 | -0.75/0.598 |
| Bab. | -0.39/0.610 | -0.40/0.610 | -0.39/0.610 | -0.39/0.610 | -0.39/0.610 | -0.39/0.610 | |
| Oracle MVDR | Dir. | 8.81/0.839 | 8.07/0.822 | 8.81/0.840 | 5.44/0.737 | 4.57/0.714 | 5.47/0.737 |
| Bab. | 8.67/0.824 | 7.95/0.807 | 8.67/0.825 | 5.63/0.737 | 4.78/0.715 | 5.66/0.738 | |
| SpatialNet RPC | Dir. | 14.20/0.911 | 13.40/0.901 | 14.10/0.909 | 10.71/0.848 | 9.00/0.811 | 10.61/0.846 |
| Bab. | 13.29/0.891 | 12.51/0.880 | 13.28/0.890 | 10.42/0.835 | 8.81/0.801 | 10.42/0.835 | |
| Unet RPC | Dir. | 11.02/0.863 | 9.75/0.835 | 11.00/0.863 | 10.86/0.858 | 9.39/0.826 | 10.67/0.854 |
| Bab. | 10.88/0.855 | 9.69/0.829 | 10.76/0.852 | 10.67/0.848 | 9.48/0.822 | 10.50/0.843 |
Table VI. SI-SDR and STOI results for directional (Dir.) and diffuse babble (Bab.) noise with 4- and 2-microphone ULA. The models were trained with directional noise only. Each entry: SI-SDR [dB] / STOI.
Relation versus Region
The concern. One could argue that the network does not learn the relation between the speakers, but only a fixed partition of the room. For example, it could interpret “right” as “on the right side of the array broadside” (), and “closer” as “closer than some fixed distance”. Such a fixed-region interpretation fails whenever both speakers are in the same region. If the target is at and the interferer at , both are on the left side of the array, yet the target is to the right of the interferer. Similarly, if the target is at m and the interferer at m, both are farther than m, yet the target is the closer speaker.
What the models do. We count an output as correct if it is closer, in SI-SDR, to the commanded speaker than to the other speaker. Over all two-speaker test mixtures, the four RPC-based models (two architectures, two array sizes) select the correct speaker in 98.4–100 % of the cases. Crucially, this rate remains just as high in the mixtures in which a fixed region would fail. When both speakers are on the same side of the broadside, the correct speaker is selected in 98.9–99.8 % of the cases, compared with 100 % when they are on opposite sides. For distance, the rate is 98.9–99.9 % both when the speakers are on the same side of a 1.75-m threshold and when they are on opposite sides, and the SI-SDR differs by at most 0.2 dB between the two groups. The only noticeable difference is a 0.6–1.4 dB lower SI-SDR when both speakers are on the same side of the broadside; it is explained by the smaller DOA gaps in these mixtures (Figure 6), not by selecting the wrong speaker. We conclude that the models resolve the relation between the speakers rather than a fixed partition of the room. A direct comparison with a model trained on region labels is left for future work.

Figure 6. SI-SDR of the RPC-based models (two-speaker test sets) as a function of the DOA gap (test sets (i) and (ii)), the radial gap (test sets (i) and (iii)), the reverberation time , the clarity index of the target, and the SIR. Each point is the mean over all outputs within the bin.
Sensitivity to Spatial Gaps, Reverberation, and SIR
Figure 6 presents the per-mixture SI-SDR of the RPC-based models in the two-speaker test sets as a function of the DOA gap (test sets (i) and (ii)), where m, the radial gap (test sets (i) and (iii)), where , the reverberation time, the clarity index of the target at the reference microphone, and the SIR. Note that the DOA grid resolution is , so all DOA gaps are integer multiples of the grid resolution, and the smallest gap is . The performance increases monotonically with , from 9.0–13.5 dB for to 11.4–14.4 dB for ; even when the speakers are in practically the same direction, the radial relation suffices for extraction. In contrast, the performance is almost independent of (variations within 0.5 dB), including m, where the speakers are practically equidistant and the angular relation is used. As expected, the performance degrades with increasing reverberation, by 2.9–4.4 dB between s and s, and increases with . Finally, SI-SDR increases by about 6 dB from SIR dB to SIR dB; however, the correct speaker is extracted in 97.5–100 % of the cases even when the target is more than 5 dB weaker than the interferer, indicating that the relative power of the speakers does not confuse the target selection within the tested range.
Moving Speakers
To assess the model’s robustness in more complex environments, we simulated dynamic scenarios where speaker positions change over time. These moving-source conditions represent a significant increase in task difficulty compared to static setups. Our goal is to determine whether the model can isolate the target speaker when both the radial separation (“gap distance”) and the angular separation (i.e., DOA difference) vary over time, despite the model being trained exclusively in static scenarios.
a) Dynamic scenarios.: Assuming a scenario with two concurrent speakers, we define the time-varying distance and DOA differences as
where and denote the instantaneous radial distance and DOA of speaker , respectively. Let and be the thresholds defined above.
To test this setting, we constructed dynamic test sets divided into three categories:
DOA-only information: and for all .
Distance-only information: and for all .
combined DOA and Distance information: and for all .
Note that while the individual DOAs and gap distances may vary over time, the relative distance between the two speakers remains valid. These configurations enable a systematic evaluation of the model’s robustness to time-varying speaker positions and orientations.
b) Data Generation.: To evaluate the performance of RPC-based extraction schemes in dynamic scenarios, we simulated speaker movements along linear, arc-shaped, and circular trajectories. These dynamic scenarios were excluded from the training data. Each speaker independently followed one of the three trajectory types, subject to the constraint that the RPC command remained constant throughout the utterance. In addition, each scenario included a stationary additive noise source that remained fixed throughout the simulation.
Linear movement: For each trajectory, a starting position within the room, a direction of motion, and a velocity (in meters per second) were sampled at random, with the only constraint being that the trajectory did not intersect the room walls.
Arc movement: An initial direction of arrival (DOA), a terminal DOA, and a radius were randomly selected. The velocity was then computed from the signal duration and the total arc length traveled.
Circular movement: A circular trajectory was randomly generated by selecting its radius and an initial DOA. The speaker then moved along the trajectory at a constant angular velocity. The angular velocity was computed from the signal duration and the total circumference traveled.
For all movement types, additional constraints were imposed to prevent trajectories from overlapping while preserving each category’s definition.
c) Results.: The tracking results are summarized in Table VII. We first evaluated the performance of the SpatialNet separation model under dynamic conditions. As expected, its performance degrades relative to the static scenarios. When a sufficient DOA separation exists between the speakers, SpatialNet maintains reasonable performance, achieving SISDR scores of 6–9 dB. However, as the angular separation decreases, its performance deteriorates substantially.
| model / test set | (i) | (ii) | (iii) |
| Unprocessed | -1.78/0.605 | -1.77/0.622 | -1.86/0.625 |
| FasNet [41] | 4.30/0.727 | 2.71/0.707 | 5.70/0.765 |
| SpatialNet [16] | 6.06/0.764 | 3.86/0.735 | 8.94/0.831 |
| SpatialNet RPC | 6.83/0.771 | 3.60/0.712 | 8.91/0.825 |
| SpatialNet RPC oracle window | 7.90/0.776 | 6.10/0.686 | 9.98/0.816 |
| Unet RPC | 10.72/0.846 | 8.34/0.802 | 11.90/0.867 |
| Unet RPC oracle window | 11.63/0.859 | 10.01/0.822 | 12.96/0.885 |
Table VII. SISDR and STOI results for two moving speakers for 2 microphones. Each entry: SI-SDR [dB] / STOI.
In contrast, the proposed method, which extracts the target speaker based on relative spatial cues, remains robust to speaker motion as long as the relative positions of the target and interfering speakers are preserved. This result highlights the advantage of exploiting relative spatial relationships rather than relying solely on absolute speaker locations.
Our model conditions the mixture embedding on the RPC and applies a global self-attention mechanism across the entire signal. Under speaker motion, however, long-range dependencies can be unnecessary, or even detrimental, because the relational label is defined instantaneously between the speakers at each time frame, rather than across their full trajectories. For example, if the task is to extract the speaker to the right, this relation is meaningful only with respect to the interferer’s position at the same time instant. A target position at one time may not be to the right of the interferer’s position at another, non-synchronous time instant. Therefore, aggregating evidence across distant frames may introduce spurious associations that are inconsistent with the instantaneous RPC. An illustration is provided in fig:7a. One speaker follows a straight path while the other moves along an arc; although the target remains to the right of the interferer at every time instant, this relation does not necessarily hold between points sampled from different times along their trajectories. Consequently, global attention may exploit misleading cross-temporal dependencies.

Figure 7. (a) Speaker trajectory. (b) Example of SI-SDRi as a function of the window size.
To improve performance under dynamic conditions, we incorporated an attention mask into the self-attention mechanism, restricting each frame’s receptive field to a local temporal window centered on that frame. We experimented with different window sizes and selected the optimal configuration for each speaker trajectory individually.
For instance, in the trajectory illustrated above, the first speaker achieved the best performance with a global (unmasked) attention window, whereas the second speaker reached the highest SISDR with a 160-frame window. On average, applying the attention mask led to approximately a 1 dB improvement in cases with large DOA gaps, and even larger gains for scenarios involving smaller angular separations (see Table VII). These results suggest that limiting temporal attention helps the model better handle frame-level variations in relative spatial positioning, thereby enhancing robustness in moving-speaker conditions.
Scope and Limitations
Overlapped speech and a known number of speakers. The proposed method assumes that all speakers are active throughout the processed segment and that is known, since the command vocabulary depends on . In real conversations, however, overlap-free speech occupies most of the speaking time [45], and the number of active speakers may change even within a single utterance. In single-speaker segments, the relative command is not defined, and in segments with a different number of speakers, the command must be respecified. A practical system should therefore combine the RPC-based extractor with a front-end that tracks the number of active speakers, e.g., a diarization or speaker-counting module, and apply the extractor only to overlapped segments, in the spirit of continuous speech separation [46]; overlap-free segments can be passed through a speech enhancement module.
Changing relative positions. The RPC is meaningful as long as the relation between the speakers holds. If the speakers swap their relative positions within an utterance, the command becomes ambiguous; whether the model then follows the spatial relation or implicitly tracks the speaker identity is an interesting question for future research.
Minimal spatial separation. Configurations in which the speakers are neither angularly () nor radially ( m) separated are not covered by the proposed cues.
Conclusions
We presented a spatially guided target speaker extraction (TSE) framework that uses RPC instead of explicit DOA estimates. These cues represent discrete, interpretable spatial relationships (e.g., left/right and near/far) between concurrent speakers. This relative command is intuitive and semantically meaningful, and it is typically easier and more reliable to infer than a precise DOA, especially in reverberant or noisy environments. Across diverse datasets, microphone configurations, and mixture complexities, the approach consistently demonstrated robustness, including in scenarios with dynamic speakers. By aligning the conditioning signal with frame-level relational cues, the method maintains stable extraction under motion, offering a practical and interpretable alternative for multi-channel speaker extraction. We further showed that the improvements stem from selecting the commanded speaker rather than from noise suppression, and that the models exploit the relative configuration of the speakers rather than fixed regions of the room, and that models trained with directional noise only remain robust to diffuse babble noise. Remaining challenges include handling overlap-free segments and a time-varying number of speakers, scaling the relative specification to more than three speakers, and robustness to nearby talkers.
References
- [1]D. Wang and J. Chen, “Supervised speech separation based on deep learning: An overview,” IEEE/ACM transactions on audio, speech, and language processing, vol. 26, no. 10, pp. 1702–1726, 2018.DOI
- [2]S. Gannot, E. Vincent, S. Markovich-Golan, and A. Ozerov, “A consolidated perspective on microphone speech enhancement and source separation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 4, pp. 692–730, 2017.DOI
- [3]C. Subakan, C. Ravanelli, S. Cornell, M. Bronzi, and J. Zhong, “Attention is all you need in speech separation,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 21–25.
- [4]Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, “Tf-gridnet: Integrating full-band sub-band modeling for speech separation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 3221–3236, 2023.arxiv.org/abs/2211.12433
- [5]T. H. Avenstrup, E. Lek, I. L. Madi, A. B. Schin, M. Mørup, B. S. Jensen, and K. Olsen, “Sepmamba: State-space models for speaker separation using mamba,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025.arxiv.org/abs/2410.20997
- [6]S. Lutati, E. Nachmani, and L. Wolf, “Separate and diffuse: Using a pretrained diffusion model for improving source separation,” arXiv preprint arXiv:2301.10752, 2023.arxiv.org/abs/2301.10752
- [7]S. Zhao, Y. Ma, C. Ni, C. Zhang, H. Wang, T. H. Nguyen, K. Zhou, J. Q. Yip, D. Ng, and B. Ma, “Mossformer2: Combining transformer and RNN-free recurrent network for enhanced time-domain monaural speech separation,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 10 356–10 360.arxiv.org/abs/2312.11825
- [8]M. Xu, K. Li, G. Chen, and X. Hu, “Tiger: Time-frequency interleaved gain extraction and reconstruction for efficient speech separation,” arXiv preprint arXiv:2410.01469, 2024.arxiv.org/abs/2410.01469
- [9]K. Žmolíková, M. Delcroix, T. Ochiai, K. Kinoshita, J. Černocký, and D. Yu, “Neural target speech extraction: An overview,” IEEE Signal Processing Magazine, vol. 40, no. 3, pp. 8–29, 2023.
- [10]A. Eisenberg, S. Gannot, and S. E. Chazan, “Single microphone speaker extraction using unified time-frequency Siamese-Unet,” in European Signal Processing Conference (EUSIPCO), 2022, pp. 762–766.arxiv.org/abs/2203.02941
- [11]——, “A two-stage speaker extraction algorithm under adverse acoustic conditions using a single-microphone,” in European Signal Processing Conference (EUSIPCO), 2023, pp. 266–270.arxiv.org/abs/2303.07072
- [12]M. Ge, C. Xu, L. Wang, E. S. Chng, J. Dang, and H. Li, “SpEx+: A complete time domain speaker extraction network,” in Interspeech, 2020, pp. 1406–1410.arxiv.org/abs/2005.04686
- [13]F. Hao, X. Li, and C. Zheng, “X-TF-GridNet: A time-frequency domain target speaker extraction network with adaptive speaker embedding fusion,” Information Fusion, vol. 112, p. 102550, 2024.DOI
- [14]S. Zhang, J. Zhang, Y. Wang, and H. Yan, “DOA or speaker embedding: Which is better for multi-microphone target speaker extraction,” IEEE Signal Processing Letters, 2025.DOI
- [15]A. Eisenberg, S. Gannot, and S. E. Chazan, “Spectral or spatial? Leveraging both for speaker extraction in challenging data conditions,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026, pp. 15 677–15 681.
- [16]C. Quan and X. Li, “SpatialNet: Extensively learning spatial information for multichannel joint speech separation, denoising and dereverberation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 1310–1323, 2024.arxiv.org/abs/2307.16516
- [17]——, “Multichannel long-term streaming neural speech enhancement for static and moving speakers,” IEEE Signal Processing Letters, vol. 31, pp. 2295–2299, 2024.arxiv.org/abs/2403.07675
- [18]G. Li, S. Liang, S. Nie, W. Liu, M. Yu, L. Chen, S. Peng, and C. Li, “Direction-aware speaker beam for multi-channel speaker extraction,” in Interspeech, 2019, pp. 2713–2717.DOI
- [19]M. Ge, C. Xu, L. Wang, E. S. Chng, J. Dang, and H. Li, “L-SpEx: Localized target speaker extraction,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 7287–7291.
- [20]J. Han, X. Zhou, Y. Long, and Y. Li, “Multi-channel target speech extraction with channel derelation and target-speaker adaptation,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 6094–6098.arxiv.org/abs/2010.09191
- [21]H. Taherian and D. Wang, “Multi-channel conversational speaker separation via neural diarization,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2467–2476, 2024.
- [22]K. Tesch and T. Gerkmann, “Spatially selective deep non-linear filters for speaker extraction,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.arxiv.org/abs/2211.02420
- [23]M. Elmnhshawi, S. R. Chetupalli, and E. A. Habets, “Beamformer-guided target speaker extraction,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5.
- [24]K. Jing, W. Zhang, and Y. Gao, “End-to-End DOA-guided speech extraction in noisy multi-talker scenarios,” arXiv preprint arXiv:2507.20926, 2025.arxiv.org/abs/2507.20926
- [25]A. Pandey, S. Lee, J. Azcarreta, D. Wong, and B. Xu, “All neural low-latency directional speech extraction,” arXiv preprint arXiv:2407.04879, 2024.arxiv.org/abs/2407.04879
- [26]J. M. Martín-Doñas, J. Heitkaemper, R. Haeb-Umbach, A. M. Gomez, and A. M. Peinado, “Multi-channel block-online source extraction based on utterance adaptation,” in Interspeech, Graz, Austria, 2019.DOI
- [27]A. Eisenberg, S. Gannot, and S. E. Chazan, “End-to-end multi-microphone speaker extraction using relative transfer functions,” arXiv preprint arXiv:2502.06285, 2025, accepted to Eusipco 2026.arxiv.org/abs/2502.06285
- [28]W. Dai, A. Politis, and T. Virtanen, “Inter-speaker relative cues for text-guided target speech extraction,” arXiv preprint arXiv:2506.01483, 2025.arxiv.org/abs/2506.01483
- [29]X. Hao, J. Wu, J. Yu, C. Xu, and K. C. Tan, “Typing to listen at the cocktail party: Text-guided target speaker extraction,” IEEE Transactions on Cognitive and Developmental Systems, vol. 18, no. 2, pp. 361–372, 2026.arxiv.org/abs/2310.07284
- [30]R. Gu, S.-X. Zhang, Y. Xu, L. Chen, Y. Zou, and D. Yu, “Multi-modal multi-channel target speech separation,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 3, pp. 530–541, 2020.DOI
- [31]Y. Xu, M. Yu, S.-X. Zhang, L. Chen, C. Weng, J. Liu, and D. Yu, “Neural spatio-temporal beamformer for target speech separation,” arXiv preprint arXiv:2005.03889, 2020.arxiv.org/abs/2005.03889
- [32]G. Li, J. Deng, M. Geng, Z. Jin, T. Wang, S. Hu, M. Cui, H. Meng, and X. Liu, “Audio-visual end-to-end multi-channel speech separation, dereverberation and recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.arxiv.org/abs/2307.02909
- [33]J. Xu, J. Cui, Y. Hao, and B. Xu, “Multi-cue guided semi-supervised learning toward target speaker separation in real environments,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
- [34]R. Gu and Y. Luo, “Rezero: Region-customizable sound extraction,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.arxiv.org/abs/2308.16892
- [35]D. Markovic, A. Defossez, and A. Richard, “Implicit neural spatial filtering for multichannel source separation in the waveform domain,” arXiv preprint arXiv:2206.15423, 2022.arxiv.org/abs/2206.15423
- [36]K. Patterson, K. Wilson, S. Wisdom, and J. R. Hershey, “Distance-based sound separation,” arXiv preprint arXiv:2207.00562, 2022.arxiv.org/abs/2207.00562
- [37]E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville, “FiLM: Visual reasoning with a general conditioning layer,” arXiv preprint arXiv:1709.07871, 2017.arxiv.org/abs/1709.07871
- [38]J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “Sdr–half-baked or well done?,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 626–630.arxiv.org/abs/1811.02508
- [39]E. A. Habets, “Room impulse response generator,” Technische Universiteit Eindhoven, Tech. Rep, vol. 2, no. 2.4, p. 1, 2006.
- [40]I. Loshchilov, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.arxiv.org/abs/1711.05101
- [41]Y. Luo, C. Han, N. Mesgarani, E. Ceolini, and S.-C. Liu, “FaSNet: Low-latency adaptive beamforming for multi-microphone audio processing,” in IEEE automatic speech recognition and understanding workshop (ASRU), 2019, pp. 260–267.arxiv.org/abs/1909.13387
- [42]M. Souden, J. Benesty, and S. Affes, “On optimal frequency-domain multichannel linear filtering for noise reduction,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 2, pp. 260–276, 2010.DOI
- [43]C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intelligibility prediction of time-frequency weighted noisy speech,” IEEE Transactions on audio, speech, and language processing, vol. 19, no. 7, pp. 2125–2136, 2011.DOI
- [44]E. Hadad, F. Heese, P. Vary, and S. Gannot, “Multichannel audio database in various acoustic environments,” in 14th international workshop on acoustic signal enhancement (IWAENC), 2014, pp. 313–317.DOI
- [45]Ö. Çetin and E. Shriberg, “Analysis of overlaps in meetings by dialog factors, hot spots, speakers, and collection site: Insights for automatic speech recognition,” in Proc. Interspeech 2006, 2006, pp. paper–1915.DOI
- [46]Z. Chen, T. Yoshioka, L. Lu, T. Zhou, Z. Meng, Y. Luo, J. Wu, X. Xiao, and J. Li, “Continuous speech separation: Dataset and analysis,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7284–7288.arxiv.org/abs/2001.11482


