WorksEngineering
Structured Speaker Conditioning for Diarization-Guided Target-Speaker ASR
Abstract
We propose structured speaker conditioning (SSC) for diarization-guided target-speaker recognition. SSC combines local activity structure with session-level acoustic identity to construct identity-consistent target conditions that control the acoustic encoder. Target-centric relational encoding (TCRE) retains per-speaker activity histories and factorized second- and third-order relations, capturing overlap and turn changes with aggregation cost linear in the number of tracks. Reference-based speaker conditioning (RSC) supplies session-derived identities that disambiguate local activity patterns within TCRE. Conditional low-rank modulation (CLRM) converts the joint condition into channel-wise affine modulation and a low-rank cross-channel residual with a condition-dependent core. On Qwen3-ASR-0.6B with oracle diarization, SSC uses 1.46M conditioning parameters and achieves a 6.23% mean relative reduction in concatenated minimum-permutation error rate (cpER) over four-state diagonal conditioning across four meeting test sets. SSC also transfers across audio foundation models. With predicted diarization and target-error group sequence policy optimization, the FireRedAudio text system achieves mean relative error reductions of 17.9% and 17.3% over DiCoW and MOSS-TD on the same four sets, and also outperforms both on three unseen domains. Scenario analyses identify gains under challenging overlap conditions, while fixed-checkpoint interventions support the contributions of session identity and the condition-dependent core. A timestamp variant jointly produces sentence boundaries and text. With predicted diarization, macro-average cpER across the four sets is 15.60% for text and 15.96% with timestamps.
Introduction
MEETING transcription should recover what each participant said while maintaining consistent speaker attribution across turn changes and overlapping speech. Target-speaker automatic speech recognition (ASR) addresses this problem by generating a transcript of a specified speaker’s speech from the mixture. When a long meeting is processed in windows, the condition should keep referring to the same session speaker as the local combination of active speakers changes. How can a target condition remain identity-consistent under changing local activity and ambiguous speaker cues while effectively controlling the acoustic encoder?
Meeting transcription systems combine diarization, separation, and recognition in modular pipelines [1], [2], or jointly predict speaker identities and transcripts [3]. Recent speech-language models also generate timestamps, as in MOSS-TD [4] and TagSpeech [5]. Diarization-guided target-speaker ASR directly supplies speaker-activity tracks to a recognition model. DiCoW [6], [7] demonstrates this approach by adapting Whisper [8] with frame-level diarization-dependent transformations (FDDT), and Dixtral [9] carries this conditioning into a spoken language model with a frozen language decoder. For each target, the activity tracks are converted into four states: silence, target-only, non-target-only, and overlap (STNO). Their probabilities control affine transformations of the acoustic features. This provides a basis for studying how diarization should be represented and how the condition should act on a pre-trained encoder.
The information retained in the condition determines which speaker configurations the encoder can distinguish. Four-state conditioning can map different interferer counts and hand-over histories to the same target condition. Encoding individual track histories before aggregation preserves these distinctions, including how the target and other speakers overlap and alternate over time.
Acoustic identity complements this temporal structure. Two speakers with identical activity throughout the available context receive the same activity-only condition, while speech elsewhere in the session can provide acoustic cues for distinguishing their voices. SE-DiCoW [10] introduces self-enrollment through encoder cross-attention, and SC-SOT [11] conditions the decoder on speaker embeddings and activity. Combining identity with track histories within a frame-level condition lets both sources determine the target–interferer relations used for recognition.
The modulation operator then translates these relations into acoustic adaptation. Diagonal FDDT scales and shifts channels, while self-speaker adaptation [12] generates speaker-specific encoder kernels from activity supervision. A low-rank cross-channel operator offers a compact way to vary channel combinations through a small generated core and shared read/write projections. Jointly designing the condition and this operator lets target identity and local speaker interactions control how acoustic features are combined.
We address this question with structured speaker conditioning (SSC). Target-centric relational encoding (TCRE) retains per-speaker activity histories and their overlap and turn-taking relations. Reference-based speaker conditioning (RSC) incorporates speaker identity information derived from session audio and diarization into these track representations, so TCRE can distinguish targets with similar local activity. Conditional low-rank modulation (CLRM) uses the joint condition to control channel-wise adaptation and cross-channel feature combinations in the acoustic encoder.
TCRE constructs permutation-equivariant target conditions using a shared causal temporal encoder and factorized second- and third-order relations. The representation captures activity history, interferer counts, and interactions with distinct interferer pairs. We relate its design to an analysis of STNO’s activity compression.
RSC associates session-derived acoustic identities with diarization tracks and augments TCRE features with this identity evidence. It supports target recognition when target speech is fully overlapped, target activity is missed, or speaker tracks are swapped.
CLRM combines feature-wise linear modulation (FiLM) [13] with a rank-8 cross-channel residual whose core varies with the target condition. The operator extends diagonal modulation with condition-dependent channel mixing. Operator comparisons and frozen-core interventions link this design to lower error.
SSC is trained using supervised fine-tuning (SFT). The FireRedAudio-based systems are subsequently optimized using group sequence policy optimization (GSPO) [14], with rewards based on target-speaker transcription error.
We evaluate SSC on Qwen3-ASR and FireRedAudio, using four in-domain meeting test sets (AliMeeting, AISHELL-4, AMI-SDM, and NOTSOFAR-1) and three out-of-domain test sets. Recognition performance is evaluated using concatenated minimum-permutation error rate (cpER), calculated at the character level for Chinese and the word level for English. On Qwen3-ASR-0.6B with oracle diarization, complete SSC uses 1.46M conditioning parameters and achieves a mean relative cpER reduction of 6.23% over STNO conditioning with the diagonal FDDT operator (STNO-FDDT). With predicted diarization, FireRedAudio + SSC + GSPO achieves mean relative cpER reductions of 17.9% and 17.3% over DiCoW and MOSS-TD, respectively, across the four in-domain sets. Scenario-specific analyses show larger gains in windows with at least three concurrent speakers and on fully overlapped target speech, while condition interventions show that session identity contributes most under target misses and track swaps. A timestamp-output variant jointly generates sentence boundaries and transcripts for recognition and timing evaluation.
Related Work
Meeting Transcription and Target-Speaker Recognition
Modular meeting transcription combines separation, recognition, and speaker attribution. Yoshioka et al. [1] integrate continuous speech separation into an online separate–recognize–diarize system. Permutation-invariant training [15] addresses the output-order ambiguity of speaker-independent separation, while TF-GridNet [16] models local and global time–frequency structure. Building on TF-GridNet separation, von Neumann et al. [2] combine ASR with transcription-supported diarization. These pipelines expose separated speech and speaker assignment as intermediate representations.
Joint recognition approaches learn to generate multiple speakers’ transcripts from the mixture. Serialized output training (SOT) [17] concatenates their transcriptions in a single decoder sequence. Speaker-attributed extensions [3] add a speaker inventory and jointly model speaker counting, transcription, and identification. Token-level SOT [18] interleaves recognition tokens and marks switches between virtual output channels for streaming recognition. Segment-based SOT [19] further addresses practical online and offline decoding, and LLM-based SOT [20] combines a pretrained speech encoder with a language-model decoder. Recent speech-language models extend structured generation to speaker labels and timestamps. MOSS-TD [4] and VibeVoice-ASR [21] jointly generate these outputs for long recordings, while DiaScriber [22] adapts a speech language model for joint diarization and transcription. TagSpeech [5] uses interleaved time anchors to synchronize semantic and speaker streams for temporal grounding. Target-speaker ASR supplies an explicit condition specifying whose speech to decode. SSC adopts this formulation and uses session-level diarization to associate window-level predictions with meeting speakers.
Activity and Acoustic Identity as Speaker Conditions
Speaker activity provides temporal information for target recognition. End-to-end neural diarization directly estimates overlapping speaker activities [23]; encoder–decoder attractors extend this formulation to an unknown number of speakers [24]. TS-VAD [25] instead conditions activity estimation on speaker representations. These diarization approaches supply activity estimates; target-speaker recognition uses such estimates to guide transcription. Whisper-based target-speaker ASR [6, 7] conditions the acoustic encoder through STNO-weighted feature transformations. In the diagonal configuration, these transformations map activity states to channel-wise gains and offsets. Dixtral [9] couples the DiCoW encoder with Voxtral, adapting the acoustic encoder and FDDT while freezing the language decoder and modality adapter.
Figure 1 illustrates the information compressed by this four-state representation. With binary tracks, one or two simultaneous interferers can yield identical STNO conditions. Different hand-over times between non-target speakers can also leave the STNO sequence unchanged despite distinct per-track histories. These cases motivate encoding track histories before symmetric aggregation and preserving relations between the target and individual interferers.
Figure 1. STNO activity compression. Different interferer counts (a) or hand-over histories (b) can yield the same four-state target condition despite distinct non-target track configurations.
Self-speaker adaptation [12] generates speaker-specific kernels from speaker-activity supervision and injects them into selected encoder layers. At the decoder, SC-SOT [11] combines speaker embeddings from a jointly trained diarization model with activity information. DM-ASR [26] converts speaker labels and segment boundaries into prompts for speaker- and time-conditioned dialogue turns. SSC combines explicit target–interferer relations with conditional low-rank modulation in the encoder.
Reference speech supplies acoustic identity complementary to activity. Speaker embedding models such as x-vectors [27] and ECAPA-TDNN [28] summarize utterances as identity representations. SpeakerBeam [29] learns a target representation from an adaptation utterance and uses it to condition speaker extraction. VoiceFilter [30] combines a speaker embedding with spectrogram masking for targeted separation. For recognition, Kanda et al. [31] alternate speaker-embedding estimation with target-speaker ASR, obtaining identity evidence from the dialogue itself. SQ-Whisper [32] uses trainable queries to derive enrollment-guided speaker prompts from the mixture, and supplies the prompts to Whisper’s encoder and decoder. SE-DiCoW [10] selects an enrollment segment from the conversation using diarization and introduces its encoded feature sequence through cross-attention at each encoder layer, while retaining the STNO path.
Deep Sets [33] uses symmetric aggregation over unordered inputs, while Relation Networks [34] aggregate learned pairwise relations. In SSC, RSC identities augment temporal track features before TCRE constructs target-centric second- and third-order relations. The target identity also enters the final condition directly, jointly controlling diagonal modulation and low-rank cross-channel mixing.
Method
Task Definition and Conditioning Pipeline
Let denote the acoustic features of a meeting window , and let contain its speaker-activity tracks on the same frame grid. Each is the fraction of a frame covered by diarization intervals for speaker . Track indices are associated with session speakers, and identifies the requested speaker. Given session audio and diarization , RSC derives a set of acoustic identities , where and indicates reference availability. The target transcript is
In Figure 2, RSC supplies the acoustic identity evidence . TCRE combines this evidence with the activity tracks to construct , and CLRM maps the aligned condition to encoder transformations. This computation is expressed as
Figure 2. SSC architecture. (a) RSC derives speaker identities from session audio and diarization. (b) TCRE combines these identities with local track histories into . (c) CLRM aligns the condition to encoder tokens and generates modulation parameters. The enlarged operator below combines diagonal FiLM with a low-rank read–mix–write branch on the same acoustic feature. Read/write bases are shared across conditions; the core varies with the condition. The pre-stack position uses FiLM only.
Here aligns frame-level conditions with the backbone’s acoustic tokens, indexes these tokens, and is the acoustic representation before modulation at position . CLRM specifies the matrix and offset . There is a pre-stack position and one position before each of the encoder blocks, . The resulting encoder features pass through the backbone’s native audio projection and decoder to generate .
Target-Centric Relational Encoding
TCRE determines which local activity information enters the target condition. It encodes each track’s history before aggregation, preserving overlap and alternation patterns as target-centric relations. RSC identity coordinates augment these track features, allowing the same relational construction to incorporate acoustic evidence about who is speaking.
From compressed states to track histories: Let be the set of valid non-target tracks and . Padded tracks are excluded from this set. The activity compression discussed in Section II motivates track-level encoding. In the STNO construction used by DiCoW [6, 7], summarizes the non-target tracks, giving the four coefficients
For binary activities, these coefficients indicate silence, target-only, non-target-only, and target-overlap states. Aggregating the non-target tracks in this way merges different interferer counts and hand-over histories. TCRE separately encodes tracks and computes symmetric relations around the target.
A shared causal temporal encoder maps each activity track to . With SiLU activations, the two convolutional layers are
The kernels have size five and dilations one and four, respectively; , , and both biases have dimension . We use . The receptive field covers 21 activity frames, or 210 ms on the 10 ms grid. Values outside the valid window are zero padded, and invalid intermediate positions are masked. Shared weights make the temporal encoding independent of track numbering.
The local feature and identity-augmented track feature are
The vector carries the reference identity and its validity, as defined in Section III-C. For a fixed target request, it remains constant over time. The leading constant in permits the factorized relations below to include constant and lower-order terms as well as interactions.
Factorized second- and third-order relations: TCRE
TCRE represents the target’s relation to individual interferers and to pairs of distinct interferers.
For , let be learned target-side and non-target-side projections, with relation width . Writing for the third-order non-target factors, we compute
The second-order term relates the target to one non-target track; the third-order term relates the target to two different non-target tracks. These orders count the participating tracks. Since the track features contain temporal history, a pairwise history interaction also describes hand-overs between interferers that are not currently active together.
The pair sum is evaluated without enumerating all pairs:
where and denote element-wise multiplication and squaring. This identity removes self-pairs, making aggregation linear in for fixed feature dimensions. The third-order term is zero with fewer than two valid non-target tracks.
Figure 3 summarizes the two branches, including the shared projections and self-pair removal in the third-order pool.
Figure 3. Factorized target-centric relations. (a) The second-order branch pools individual interferer factors. (b) The third-order branch combines the target with distinct interferer pairs. Subtracting from removes self-pairs, and the factor counts each unordered pair once. The grid illustrates this pair selection. Sums run over valid non-target tracks , giving pooling, with and element-wise multiplication .
Fixed-dimensional target condition and symmetry
The complete frame-level condition is
where is the current non-target activity mass. Thus describes current activity, whereas counts valid non-target tracks in the window. With and , and , independently of the number of speakers. The activity-only model omits the 33 identity coordinates, giving a 67-dimensional condition.
For binary activities, , so the retained coordinates determine the STNO coefficients before temporal alignment. The additional history and relation coordinates express the finer activity structures motivating TCRE: distinguishes one interferer from two, and the temporal relations describe which tracks interfere over time.
For any permutation of the speaker tracks, the condition satisfies
The temporal encoder and projection matrices are shared across tracks, and all non-target aggregations are symmetric sums over tracks or unordered pairs, following the shared-encoding and symmetric-aggregation principle of set representations [33]. Consequently, relabeling the tracks and requested target preserves the condition for the same physical speaker. The target conditions are thus permutation equivariant.
Reference-Based Speaker Conditioning
1) Diarization-guided session evidence: Activity histories describe temporal interactions, but two targets with identical activity throughout the available local history receive the same activity-only TCRE condition. RSC addresses this ambiguity by supplying session-level acoustic identity to the track features and the target coordinates of . This binds the requested track to a voice within the relational condition. Figure 4 illustrates how acoustic identities distinguish targets with identical local activity. Algorithm 1 summarizes reference selection and condition construction.
Figure 4. Session identity distinguishes locally ambiguous targets. (a) Identical local activity histories yield the same activity-only target condition. (b) RSC encodes speaker-exclusive references outside the current window . For valid, distinct identities (, ), the target identity coordinates distinguish the two conditions. Colored cells highlight these coordinates. The timeline elements are schematic.
For each speaker, the selection map
collects speaker-exclusive evidence from other intervals of the session. We exclude the current window with a temporal margin, subtract intervals assigned to other speakers, and trim candidate boundaries. Among segments meeting a minimum segment duration, we select up to , prioritizing longer segments and capping the total duration at . Distance to and start time resolve ties. A track has when at least of selected evidence is available and otherwise. At inference, input diarization supplies the selection intervals. Hyperparameter values are given in Section IV-A.
For online simulated mixtures, references are independent clean utterances of the same source identity. The selection excludes the utterances used in the current mixture and candidates with the same normalized text. This provides acoustic evidence for identity across different speech content.
2) Shared reference encoding and identity pooling: Reference audio passes through the shared acoustic encoder prefix with target-only activity conditioning. We use the first encoder blocks and pool their valid output tokens. Let denote these representations and . Using learned token weights as in attentive speaker pooling [35], we form a weighted mean:
Here , , and . The sums include only valid reference tokens. Reference encoding and pooling are differentiable under the target-transcription loss.
For a request targeting , the identity features are gated by target reference availability:
We set unavailable identities to zero. If , all are zero and the request uses activity-only conditioning. With a valid target reference, each available non-target identity participates in the relational features. The target identity also enters directly in (14). Identity therefore influences both the relations and the final condition supplied to the modulation generator.
3) Paired target binding: Training includes paired requests for the same highly overlapped mixture. Two targets A and B have matched speech durations satisfying . Both views share the mixture and augmentation randomness and are kept together within a device batch. Their supervised contribution is
Algorithm 1. RSC condition construction
Input: Session mixture and diarization ; local activity , current window , target , and reference hyperparameters .
Output: Frame-level target condition .
for do
Select speaker-exclusive evidence outside using (16).
Keep up to segments, totaling at most .
Set if total duration , else .
Compute by (18) if ; otherwise set .
end for
Use in for all (19).
; .
return
Training: Sample real / simulated / binding views with weights . For a binding view, use two targets from the same mixture with matched durations and minimize in (20).
Paired requests train target selection using acoustic identity when activity patterns provide little separation. Real, ordinary simulated, and paired-binding views use sampling weights (Section IV-A).
Conditional Low-Rank Modulation
The joint TCRE–RSC condition specifies the requested speaker and the surrounding activity structure. CLRM makes this information control recognition by generating both channel-wise gains and offsets and condition-dependent combinations of acoustic channels. CLRM combines feature-wise affine conditioning [13] with a compact cross-channel transform, using a hypernetwork to generate condition-dependent parameters [36]. Low-rank-plus-diagonal acoustic adaptation [37] decomposes speaker transforms into diagonal and low-rank terms; bottleneck adapters [38] and LoRA [39] use compact modules or weight updates for efficient adaptation. In CLRM, learned read/write bases are shared across conditions, while the diagonal parameters and a small mixing core vary with the target’s activity and identity at each encoder token.
Alignment with the acoustic token grid
Relations are constructed at the activity-frame rate and then aligned with the backbone’s convolutional subsampling. For stride , define the valid-normalized pooling operator
where marks valid input positions. Valid output lengths and masks are recomputed after each stage, separately for each acoustic chunk. For Qwen3-ASR, , mapping 10 ms frames to 80 ms tokens. FireRedAudio uses to match its native encoder grid. Validity normalization prevents padding from diluting the condition. The signed relation coordinates are averaged as features.
Shared generator and position-specific heads
All modulation positions use a shared hidden layer, with a learned position embedding :
We use and . Each position has an affine output head,
and each encoder-block position has a low-rank core head,
The head dimensions are and . Thus the same activity–identity condition controls channel-wise gains, offsets, and cross-channel mixing. The core rank is , distinct from the TCRE relation width .
Parallel acoustic modulation
Let be learned write and read bases shared across conditions at position . The operator in (5) is
The first term provides FiLM [13]; the second, referred to as the cross-channel low-rank (XLR) branch, reads the acoustic feature into an -dimensional space, mixes it with the generated core, and writes the result back. Both terms act in parallel on the same pre-modulation representation . The pre-stack position uses ; the cross-channel term is applied before each encoder block.
Writing and for columns of the two bases makes the conditional channel combinations explicit:
The coefficients of these read/write combinations vary with the target condition. The additional matrix has rank at most and costs to apply per token. Each block adds parameters for the bases and core head. Setting yields diagonal FiLM modulation.
The affine and core output heads are zero initialized, giving an identity transform at the start of fresh training. The read/write bases are orthogonally initialized. Condition construction, parameter generation, and the low-rank branch use FP32 arithmetic. The resulting acoustic updates are converted to match the numerical precision of the backbone’s hidden representations. For Qwen3-ASR-0.6B (, ), the cross-channel branches add 296,064 parameters. Complete SSC, including TCRE, RSC pooling, and CLRM, contains 1,464,312 trainable conditioning parameters.
Training and Target-Conditioned Inference
The supervised objective uses only the requested target’s transcript. For output tokens, it is
All SSC modules, including the reference pooling path, learn through this objective. Binding pairs contribute the two target-specific losses in (20).
After supervised fine-tuning, GSPO [14] optimizes target transcription error. For each target-conditioned input , the sampling policy generates a group of candidates. Candidate receives reward
where edit counts are computed after text normalization against the requested target’s window transcript, using characters for Chinese and words for English. Let denote the group-normalized advantage, where and are the group mean and population standard deviation. With candidate length , the sequence-level importance ratio and objective are
Here is the token-normalized supervised loss across the logical minibatch. The text-system recipe uses , , , , , and . Target identities and activity conditions accompany each candidate during sampling and optimization.
At inference, a long session is divided into windows. For each requested track, RSC obtains session evidence, TCRE constructs its local condition, and CLRM modulates the encoder used to decode that target. Window outputs are assembled using the session speaker identities. A timestamp-output variant represents as sentences with window-relative onset and offset times, with the same conditioning modules.
Experiments
Experimental Setup
Data and speaker priors
Table I summarizes the data. Data preparation builds on mt-asr-data-prep.3 Real meeting training uses the training partitions of AliMeeting [40], AISHELL-4 [41], AMI [42], and NOTSOFAR-1 [43], totaling approximately 331 h of single-channel recordings. Qwen3-ASR and FireRedAudio additionally use a 10,000-h TTS source pool to construct online 30-s mixtures with two to six speakers and 0–60% overlap, including room-response augmentation. Real and simulated target views are sampled at a 2:1 ratio. With RSC, the sampling weights for real, ordinary simulated, and paired-binding views are . Reference selection uses , s, and s, with a 0.5-s exclusion margin, 0.2-s boundary trimming, and a 0.5-s minimum segment duration.
The four in-domain test sets contain 20 AliMeeting recordings, 20 AISHELL-4 recordings, 16 AMI-SDM recordings, and 160 NOTSOFAR-1 device recordings from 80 meetings. All systems use far-field single-channel audio: channel 0 for AliMeeting and AISHELL-4, SDM with the Full-corpus-ASR partition for AMI, and the single-channel device tracks for NOTSOFAR-1. Out-of-domain evaluation uses six
| Dataset | Partition / channel | Lang. | Hours |
| Real meeting training data | |||
| AliMeeting | Train, far-field ch. 0 | zh | 104.75 |
| AISHELL-4 | Train, far-field ch. 0 | zh | 107.50 |
| AMI-SDM | Train, SDM | en | ≈81 |
| NOTSOFAR-1 | Train 1+2, SC | en | ≈38 |
| Auxiliary augmentation | |||
| TTS source pool | Online mixtures | zh/en | 10k |
| In-domain evaluation | |||
| AliMeeting | Test, far-field ch. 0 | zh | 10.00 |
| AISHELL-4 | Test, far-field ch. 0 | zh | 12.72 |
| AMI-SDM | Eval, SDM | en | ≈9 |
| NOTSOFAR-1 | Eval-small, SC | en | 16.00 |
| Out-of-domain evaluation | |||
| MISP-Meeting | Eval, far-field | zh | 3.00 |
| AISHELL-5 | Eval2, far-field | zh | 3.58 |
| DiPCo | Eval, far-field | en | ≈2.6 |
Table 1. Training and evaluation data. SDM: single distant microphone; SC: single-channel device recording.
MISP-Meeting recordings [44], 18 AISHELL-5 Eval2 recordings [45], and five DiPCo Eval recordings [46], with no training or adaptation on these evaluation sets.
Mechanism ablations primarily use oracle diarization to isolate the conditioning design from input diarization errors. Unless otherwise specified, predicted priors are obtained by Diarizen [47] with a W2V-BERT2 frontend, followed by sequence-to-sequence (S2S) refinement [48]. RSC selects its inference reference segments from this input prior. At a 0.25-s collar, input DERs for AliMeeting, AISHELL-4, AMI-SDM, and NOTSOFAR-1 are 3.62/4.89/8.65/5.10%.
Models and optimization
Our implementation extends TS-ASR-Whisper.4 Our backbones are Qwen3-ASR-0.6B [49] and the 9B FireRedAudio model [50]. Unless stated otherwise, the conditioning modules use the widths introduced in Section III (, , ). The reference path shares the first six encoder blocks with the recognition path. For both Qwen3-ASR and FireRedAudio, we jointly train the acoustic encoder, audio projector, and conditioning modules, with rank-8 attention LoRA [39] in the decoder (scaling 16, dropout 0.05). SFT uses eight GPUs, BF16 backbone computation, and FP32 conditioning modules. The base learning rate is ; conditioning parameters use a tenfold multiplier. Qwen3-ASR-0.6B uses a 15,000-step cosine schedule with 500 warmup steps; the activity-only variants stop after 10,000 steps. FireRedAudio uses 30,000 steps and 1,000 warmup steps. For SFT and GSPO, we select checkpoints by minimizing the equally weighted mean of autoregressive decoding error rates on the AliMeeting and AMI development sets.
Metrics and comparison protocol
We report concatenated minimum-permutation error rate (cpER), using characters for Chinese and words for English. After optimal speaker permutation within each session, corpus-level scores are
| Cond. | Operator | Par. (M) | AliMeeting | AISHELL-4 | AMI-SDM | NOTSOFAR-1 | All | 1 spk | 2 spk | 3 spk | No-solo target |
| STNO | FDDT | 0.14 | 17.92 | 13.20 | 18.35 | 21.36 | – | – | – | – | – |
| STNO | FiLM | 1.13 | 17.68 | 13.23 | 18.12 | 20.90 | 1.14 | 0.10 | 2.21 | 0.73 | 0.13 |
| FiLM | 1.13 | 18.03 | 13.07 | 18.31 | 20.97 | 0.62 | 2.33 | −0.15 | 0.19 | 0.97 | |
| FiLM | 1.13 | 17.18 | 12.95 | 18.15 | 20.47 | 2.84 | 5.70 | 1.03 | 3.42 | 2.65 | |
| FiLM | 2.23 | 17.76 | 13.40 | 18.14 | 20.68 | 0.92 | 0.03 | 0.19 | 1.97 | _3.47_ | |
| XLR | 0.31 | 17.57 | _12.86_ | 17.95 | 20.56 | 2.60 | _5.36_ | 1.67 | 2.83 | 1.25 | |
| CLRM | 1.43 | _16.96_ | 12.92 | _17.81_ | _20.27_ | 3.90 | 5.33 | _2.48_ | _4.49_ | 2.66 | |
| CLRM | 1.46 | 16.47 | 12.86 | 17.19 | 19.67 | 6.23 | 2.38 | 3.44 | 7.76 | 12.49 |
Table 2. Component ablations on Qwen3-ASR-0.6B with oracle diarization. uses activity relations through order , and adds RSC identities. Par. gives conditioning parameters in millions. RER is relative to STNO-FDDT. The 1/2/3-spk columns group windows by peak reference concurrency. Bold and underline mark the best and second-best values before rounding.
Here , , , and denote substitutions, deletions, insertions, and reference units in session . Relative error reduction (RER) is . Multi-dataset means weight each dataset equally. RSC interventions use fixed-target CER/WER, keeping the requested speaker assignment.
For timestamp outputs, MeetEval [51] computes time-constrained cpER (tcpER) with a 5-s collar using character-based pseudo-word timings. Diarization error rate (DER) uses a 0.25-s collar, includes overlapping speech, and pools errors over scored speaker time.
Paired comparisons share reference transcripts and text normalization, and complete-system comparisons use a common normalization. For corpus-level comparisons, 95% confidence intervals use 10,000 paired meeting-bootstrap samples [52], with NOTSOFAR-1 device pairs resampled together.
TCRE: Preserving Multi-Speaker Activity Relations
The configurations in Table II progress from four-state conditioning to TCRE, then to TCRE–CLRM, and finally to the full RSC recipe. We first vary the activity representation while fixing the FiLM operator and generator width. STNO-FiLM encodes the four state trajectories with causal convolutions and concatenates their temporal features with the state probabilities before FiLM parameter generation. It provides a 1.14% mean RER over STNO-FDDT. Second-order TCRE alone gives 0.62%, whereas adding the third-order term raises the reduction to 2.84%, with cpER 17.18/12.95/18.15/20.47% across the four sets. Thus, track-level conditioning is most effective when it includes distinct interferer-pair relations. Figure 5 shows that third-order TCRE improves over STNO-FiLM at a similar conditioning budget of about 1.13M parameters.
Figure 5. Parameter efficiency of the activity-only ablations in Table II. Points show conditioning-module size and mean RER over STNO-FDDT across the four test sets.
We examine this relation using the original 30-s windows, grouped by peak reference concurrency. The peak-1/2/3 groups contain 1,221/2,022/2,660 windows. Errors are attributed after full-session speaker matching and edit alignment: reference-unit midpoints locate substitutions and deletions, and decoder windows locate insertions. The third-order model improves over the second-order model by 3.22% in the peak-3 group (95% CI [1.85, 4.50]%). Peak concurrency of at least three supplies simultaneous non-target activity for the distinct-interferer-pair term in (12). The third-order model
also improves over the second-order model by 1.14% in the peak-2 group. In peak-1 windows, the third- and second-order models achieve 5.70% and 2.33% RER over STNO-FDDT, respectively (Table II).
CLRM: Conditional Cross-Channel Modulation
With the third-order TCRE condition fixed, we next vary how it controls the acoustic encoder. Rank-8 CLRM improves the mean RER over STNO-FDDT from FiLM’s 2.84% to 3.90% (Table II). Increasing the FiLM generator width from 32 to 64 instead gives 0.92% RER with 2.23M parameters, compared with CLRM’s 1.43M. The cross-channel-only variant is compact (0.31M parameters) and gives 2.60% RER; combining it with the diagonal branch produces the strongest activity-only result. These comparisons support combining diagonal adaptation with low-rank channel mixing.
We further intervene on the core of a frozen Qwen3-ASR-0.6B TCRE–CLRM checkpoint (Figure 6). Setting the core to zero increases meeting-macro cpER by 2.09 pp on AliMeeting-dev (95% CI [0.80, 3.51]) and 0.53 pp on AMI-SDM-dev ([0.19, 1.08]). Replacing it with a condition-independent calibration mean also increases error; with energy matching, the increases are 2.22 and 0.30 pp, respectively. The trained model therefore uses the varying core, beyond retaining a fixed cross-channel projection. These interventions support the role of the condition-dependent core described in Section III-D.
Figure 6. Recognition results from frozen-CLRM-core interventions. Values are changes in meeting-macro cpER from the intact model on eight AliMeeting-dev and four AMI-SDM-dev recordings, with 95% paired meeting-bootstrap CIs. The shared projections and all other weights remain fixed. The mean-core interventions replace the varying core with its calibration mean, optionally matching its energy.
RSC: Anchoring the Requested Speaker
The full RSC recipe adds session identities and paired target binding to TCRE–CLRM, yielding complete SSC. It achieves the best corpus scores in Table II, with cpERs of 16.47/12.86/17.19/19.67%, 1.46M conditioning parameters, and a 6.23% mean RER over STNO-FDDT. Complete SSC also achieves 3.44% RER over STNO-FDDT in the peak-2 group and 7.76% in the peak- group. The largest gain from the full RSC recipe occurs when the target has no solo speech, i.e., every target frame overlaps with another speaker. Relative to activity-only TCRE–CLRM, the full RSC recipe reduces error in this group by 10.09% (95% CI [7.58, 12.09]%); the reduction is 0.86% when more than half of the target’s speech is solo. Relative to STNO-FDDT, complete SSC reaches 12.49% RER in this fully overlapped group. The full RSC recipe is especially effective when local activity poorly distinguishes targets.
Table III measures the inference-time contribution of session identity at a fixed FireRedAudio timestamp checkpoint after GSPO (Section IV-F). With audio, target requests, reference candidates, and weights fixed, we supply or remove RSC identities under the same local activity. Starting from oracle tracks, we apply target misses, false alarms, and local track swaps to 0.5-s activity blocks at a 10% rate (seed 42). On clean activity, RSC reduces fixed-target CER from 12.60 to 12.14% on AliMeeting and WER from 14.23 to 13.59% on AMI-SDM. With target misses, the relative reductions increase to 17.81% and 14.51%; with track swaps, they reach 17.84% and 13.06%. Session identity therefore supports target recognition when local activity loses or exchanges target evidence. False-alarm interventions yield small increases of 0.02/0.36 pp on AliMeeting/AMI-SDM, indicating that the benefit centers on missing and exchanged target activity.
| Activity condition | AliMeeting (CER) | AMI-SDM (WER) | ||||
| w/o | w/ | RER | w/o | w/ | RER | |
| Clean activity | 12.60 | 12.14 | 3.67 | 14.23 | 13.59 | 4.50 |
| Target miss (10%) | 18.92 | 15.55 | 17.81 | 18.98 | 16.22 | 14.51 |
| Target-track swap (10%) | 18.01 | 14.80 | 17.84 | 18.50 | 16.09 | 13.06 |
| Target false alarm (10%) | 16.14 | 16.16 | −0.12 | 16.89 | 17.26 | −2.16 |
| AMI example. ES2004d / ME0015, 1650–1680 s | ||||||
| Reference | seems fine except voice recognition everything is yeah | |||||
| w/o RSC | seems fine except for iso admission everything looks yeah | |||||
| w/ RSC | seems fine except voice recognition everything is yeah |
Table 3. RSC under activity interventions. Fixed-target CER/WER (%) without (w/o) and with (w/) RSC, and relative error reduction (RER). Bold marks lower error. The example uses identical intervened activity at the same checkpoint.
| Prior | Set | cpER SSC + GSPO | cpER SSC | tcpER 5 s SSC + GSPO | |
| Oracle | AliMeeting | 12.56 | 12.11 | −0.45† | 12.65 12.18 |
| AISHELL-4 | 10.62 | 10.37 | −0.25† | 10.64 10.38 | |
| AMI-SDM | 14.01 | 13.60 | −0.41† | 14.28 13.85 | |
| NOTSOFAR-1 | 15.74 | 14.82 | −0.92† | 15.95 15.01 | |
| Predicted | AliMeeting | 14.05 | 13.59 | −0.46† | 14.35 13.84 |
| AISHELL-4 | 14.00 | 13.83 | −0.17† | 14.36 14.17 | |
| AMI-SDM | 16.79 | 16.07 | −0.71† | 17.60 16.82 | |
| NOTSOFAR-1 | 21.17 | 20.34 | −0.83† | 21.84 20.94 | |
| Mean RER (%) | 3.4 | 3.6 |
Table 4. Timestamp prediction with SSC and SSC + GSPO (%). : (SSC + GSPO) − SSC in cpER percentage points; †: 95% paired meeting-bootstrap confidence interval below zero.
The example in Table III illustrates how the same intervened activity can produce different target transcripts. In a 30-s AMI window, RSC preserves “voice recognition” and the complete reference phrase, while the activity-only condition yields “for iso admission” and an additional word substitution. The supplied activity is identical in both decodes and contains a 0.43-s target miss. This example shows how session identity changes recognition under the same input activity.
Transfer to FireRedAudio and GSPO Post-Training
On FireRedAudio, we compare FDDT, SSC, and SSC + GSPO (Figure 7). Under the shared 30,000-step SFT recipe, SSC achieves a 2.80% mean relative cpER reduction over FDDT across the two priors and four datasets. The SFT-to-GSPO comparison starts from the same SSC checkpoint and uses the target-error reward in (28). GSPO brings a further 3.67% mean RER; the combined reduction over FDDT is 6.34%. On NOTSOFAR-1, these reductions reach 13.8% with oracle priors and 9.2% with predicted priors.
Figure 7. FireRedAudio progression from FDDT to SSC and SSC + GSPO on the four datasets (a)–(d), with oracle priors and predicted S2S-refined priors. Endpoint labels give the relative cpER reduction over FDDT.
We group meetings into 0–10%, 10–30%, and > 30% overlap. With predicted priors, SSC + GSPO yields lower cpER than SSC in all 11 nonempty dataset–interval groups. Figure 8 shows the reductions relative to FDDT, with 5, 39, and 36 NOTSOFAR-1 meetings in the three intervals, respectively. These session-level comparisons complement the local-concurrency and fully overlapped-target analyses.
Figure 8. FireRedAudio results by full-session overlap ratio with predicted S2S refinement priors. (a) NOTSOFAR-1 absolute cpER reductions of SSC and SSC + GSPO from FDDT (pp), with meeting counts . (b) SSC + GSPO relative error reduction over FDDT for meetings with at least three meetings. Hatched cells have fewer than three meetings, and dashes mark empty bins.
The paired intervals in Figure 9 quantify the contribution of each stage. Five of eight SSC-versus-FDDT intervals lie entirely below zero. All eight GSPO-versus-SSC intervals lie below zero, showing consistent post-training gains across datasets and diarization priors.
Figure 9. Paired meeting-bootstrap changes in corpus-micro cpER for FireRedAudio: (a) SSC versus FDDT; (b) SSC + GSPO versus SSC. Squares and circles mark oracle and predicted priors. Open markers denote CIs spanning zero. Negative values favor the new system. Error bars show 95% CIs.
Sentence Timestamps and Joint Output Evaluation
The timestamp variant generates sentence onset/offset times and text from the same target-conditioned model. Table V compares the SFT checkpoint (SSC) with the checkpoint obtained from it through GSPO post-training (SSC + GSPO), using the same target-transcription-error reward. Across the eight dataset–prior comparisons, mean RER is 3.4% for cpER and 3.6% for tcpER. All eight CIs for the cpER differences
| Model | AliMeeting cpER | AliMeeting tcpER | AliMeeting DER | AISHELL-4 cpER | AISHELL-4 tcpER | AISHELL-4 DER | AMI-SDM cpER | AMI-SDM tcpER | AMI-SDM DER | NOTSOFAR-1 cpER | NOTSOFAR-1 tcpER | NOTSOFAR-1 DER | Macro cpER | Macro tcpER | Macro DER |
| DiCoW(zh/en) | 20.32 | 20.68 | 4.34 | 19.40 | 19.99 | 5.10 | 16.31 | 17.03 | 8.87 | 20.46 | 21.01 | 5.58 | 19.12 | 19.68 | 5.97 |
| MOSS-TD | 15.20 | 15.39 | 6.21 | 13.80 | 14.06 | 4.27 | 21.84 | 22.77 | 15.02 | 27.32 | 28.24 | 11.61 | 19.54 | 20.12 | 9.28 |
| Qwen3.8-Omni-Flash | 21.74 | 23.04 | 11.33 | 15.54 | 15.84 | 6.08 | 29.32 | 32.25 | 34.08 | 42.60 | 44.22 | 27.01 | 27.30 | 28.84 | 19.62 |
| Proposed (text) | 13.33 | – | – | 13.67 | – | – | 15.61 | – | – | 19.81 | – | – | 15.60 | – | – |
| Proposed (timestamp) | 13.57 | 13.83 | 3.91 | 13.84 | 14.19 | 5.02 | 16.07 | 16.82 | 9.48 | 20.34 | 20.94 | 5.52 | 15.96 | 16.44 | 5.98 |
| DiCoW(zh/en) | 23.33 | 24.08 | 5.37 | 45.68 | 46.17 | 16.20 | 29.24 | 31.25 | 22.02 | 32.75 | 33.83 | 14.53 | |||
| MOSS-TD | 17.62 | 17.96 | 4.82 | 30.91 | 31.24 | 15.35 | 27.60 | 28.80 | 22.59 | 25.37 | 26.00 | 14.26 | |||
| Qwen3.8-Omni-Flash | 21.51 | 21.93 | 10.76 | 35.26 | 35.80 | 21.13 | X | X | X | – | – | – | |||
| Proposed (text) | 14.04 | – | – | 28.77 | – | – | 26.73 | – | – | 23.18 | – | – | |||
| Proposed (timestamp) | 14.08 | 14.69 | 4.74 | 28.84 | 29.11 | 15.49 | 28.21 | 30.12 | 22.04 | 23.71 | 24.64 | 14.09 |
Table V. System Comparison in Our Evaluation on In-Domain and Out-of-Domain Test Sets (%, Lower Is Better). (a) In-Domain. (b) Out-of-Domain. Notes: Proposed is FireRedAudio + SSC + GSPO. Macro is the equal-weight mean over all datasets within each panel. Bold and underline mark the best and second-best per column. “–” marks an unavailable metric, and X marks Qwen3.8-Omni-Flash DiPCo scores left unavailable by one failed API recording. DiCoW and Proposed use S2S refinement priors, whereas MOSS-TD and Qwen3.8-Omni-Flash use no external prior. On MISP-Meeting, target-speaker systems use annotated UEM and MOSS-TD uses the full recording.
(SSC + GSPO minus SSC) lie below zero. With predicted priors, AMI-SDM tcpER falls from 17.60 to 16.82% and NOTSOFAR-1 cpER from 21.17 to 20.34%.
In the common-protocol system comparison (Table V), the timestamp model incurs an equal-dataset mean cpER cost of 0.43 pp relative to the text model across seven sets. Its tcpER ranks first among locally evaluated systems on five sets and second on AISHELL-4 and DiPCo. Output DER is 3.91% on AliMeeting and 9.48% on AMI-SDM.
Complete Systems and Generalization to Unseen Domains
Table V compares the complete FireRedAudio + SSC + GSPO systems with the Chinese–English DiCoW baseline [7], MOSS-TD [4], and Qwen3.8-Omni-Flash [53] across the seven test sets.
The proposed text system obtains the lowest cpER among locally evaluated systems on each of the seven sets. Across the four in-domain sets, its mean RER is 17.9% over DiCoW and 17.3% over MOSS-TD. Gains over DiCoW are largest on AliMeeting and AISHELL-4 (34.4% and 29.5% RER); gains over MOSS-TD are largest on AMI-SDM and NOTSOFAR-1 (28.6% and 27.5%). The AISHELL-4 difference from MOSS-TD is smaller, at 0.13 pp.
On the three unseen domains, text-system cpER is 14.04/28.77/26.73% for MISP-Meeting/AISHELL-5/DiPCo, compared with DiCoW’s 23.33/45.68/29.24% and MOSS-TD’s 17.62/30.91/27.60%. These gains extend to meeting, in-car, and dinner-party recordings from new corpora.
Conclusion
SSC connects local activity structure and session-level identity through a target condition that controls acoustic feature combinations. TCRE preserves per-speaker histories and target-centric activity relations. RSC grounds these relations in acoustic identity derived from session audio, and CLRM maps the joint condition to diagonal and rank-8 cross-channel modulation for target-speaker recognition. Complete SSC yields 6.23% mean RER over STNO-FDDT on Qwen3-ASR-0.6B with 1.46M conditioning parameters. Transfer to FireRedAudio, followed by target-error GSPO, produces a complete system with 17.9% and 17.3% mean in-domain RER over DiCoW and MOSS-TD, respectively, and lower cpER than DiCoW and MOSS-TD on three unseen domains. The timestamp variant retains competitive recognition while providing sentence-level text and timing in the same output.
Acknowledgment
Generative AI use. OpenAI Codex (GPT-6) assisted with translating, drafting, and polishing author-provided material. Claude Opus 5.5, accessed through CodeBuddy, assisted with manuscript review and the preparation of figures and tables. The authors developed all scientific content, reviewed the manuscript sentence by sentence, checked and corrected the mathematical derivations and their presentation, and verified numerical values. They manually adjusted the AI-drafted visual elements and finalized the manuscript, figures, and tables, taking full responsibility for the work.
References
- [1]T. Yoshioka, I. Abramovski, C. Akscoylar, Z. Chen, M. David, D. Dimitriadis, Y. Gong, I. Gurvich, X. Huang, Y. Huang, A. Hurvitz, L. Jiang, S. Koubi, E. Krupka, I. Leichter, C. Liu, P. Parthasarathy, A. Vinnikov, L. Wu, K. Xiao, W. Xiong, H. Wang, Z. Wang, J. Zhang, Y. Zhao, and T. Zhou, “Advances in online audio-visual meeting transcription,” in Proc. IEEE Autom. Speech Recognit. Understanding Workshop (ASRU), 2019, pp. 276–283.arxiv.org/abs/1912.04979
- [2]T. von Neumann, C. Boeddeker, T. Cord-Landwehr, M. Delcroix, and R. Haeb-Umbach, “Meeting recognition with continuous speech separation and transcription-supported diarization,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. Workshops (ICASSPW), 2024, pp. 775–779. [Online]. Available: https://doi.org/10.1109/icasspw62645.2024.10625894
- [3]N. Kanda, Y. Gaur, X. Wang, Z. Meng, Z. Chen, T. Zhou, and T. Yoshioka, “Joint speaker counting, speech recognition, and speaker identification for overlapped speech of any number of speakers,” in Proc. Interspeech, 2020, pp. 36–40. [Online]. Available: https://www.isca-archive.org/interspeech_2020/kanda20_interspeech.htmlarxiv.org/abs/2006.10930
- [4]MOSI.AI, D. Yu, Z. Lin, H. Chen, C. Yang, Y. Zhang, J. Chen, K. Chen, L. Fan, Y. Jiang, J. Zhu, M. Li, W. Wang, Y. Wang, Z. Xu, B. Jiang, Y. Gong, Y. Zhang, W. Zhang, S. Wang, Z. Wang, Z. Fei, Q. Cheng, S. Li, and X. Qiu, “MOSS Transcribe Diarize technical report,” 2026, arXiv:2601.01554, version 7. [Online]. Available: https://arxiv.org/abs/2601.01554v7
- [5]M. Huo, Y. Shao, and Y. Zhang, “TagSpeech: End-to-end multi-speaker ASR and diarization with fine-grained temporal grounding,” in Proc. Annu. Meeting Assoc. Comput. Linguistics (ACL), 2026, pp. 41 847–41 862. [Online]. Available: https://aclanthology.org/2026.acl-long.1938/
- [6]A. Polok, D. Klement, M. Wiesner, S. Khudanpur, J. Černocký, and L. Burget, “Target speaker ASR with Whisper,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2025, pp. 1–5.arxiv.org/abs/2409.09543
- [7]A. Polok, D. Klement, M. Kocour, J. Han, F. Landini, B. Yusuf, M. Wiesner, S. Khudanpur, J. Černocký, and L. Burget, “DiCoW: Diarization-conditioned Whisper for target speaker automatic speech recognition,” Comput. Speech Lang., vol. 95, 2026, art. no. 101841.
- [8]A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeay, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. Int. Conf. Mach. Learn. (ICML), ser. Proc. Mach. Learn. Res., A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202, PMLR, 23–29 Jul 2023, pp. 28 492–28 518. [Online]. Available: https://proceedings.mlr.press/v202/radford23a.htmlarxiv.org/abs/2212.04356
- [9]A. Polok, S. Cornell, S. Udupa, H. Černocký, S. Watanabe, and L. Burget, “Grounding spoken LLMs in multi-speaker audio via diarization conditioning,” in Proc. Interspeech, 2026, pp. 300–305. [Online]. Available: https://www.isca-archive.org/interspeech_2026/polok26b_interspeech.htmlarxiv.org/abs/2606.18134
- [10]A. Polok, D. Klement, S. Cornell, M. Wiesner, J. Černocký, S. Khudanpur, and L. Burget, “SE-DiCoW: Self-enrolled diarization-conditioned Whisper,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2026, pp. 16712–16716. [Online]. Available: https://doi.org/10.1109/ICASSP55912.2026.11461785
- [11]Y. Hirano and S. Sakti, “SC-SOT: Conditioning the decoder on diarized speaker information for end-to-end overlapped speech recognition,” in Proc. Interspeech, 2025, pp. 4923–4927. [Online]. Available: https://www.isca-archive.org/interspeech_2025/hirano25_interspeech.htmlarxiv.org/abs/2506.12672
- [12]W. Wang, T. Park, I. Medennikov, J. Wang, K. Dhawan, H. Huang, N. Rao Kologuri, J. Balam, and B. Ginsburg, “Speaker targeting via self-speaker adaptation for multi-talker ASR,” in Proc. Interspeech, 2025, pp. 5498–5502. [Online]. Available: https://www.isca-archive.org/interspeech_2025/wang25_interspeech.htmlarxiv.org/abs/2506.22646
- [13]E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville, “FiLM: Visual reasoning with a general conditioning layer,” Proc. AAAI Conf. Artif. Intell., vol. 32, no. 1, pp. 3942–3951, 2018.arxiv.org/abs/1709.07871
- [14]C. Zheng, S. Liu, M. Li, X.-H. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, J. Zhou, and J. Lin, “Group sequence policy optimization,” 2025, arXiv:2507.18071. [Online]. Available: https://arxiv.org/abs/2507.18071
- [15]D. Yu, M. Kolbaek, Z.-H. Tan, and J. Jensen, “Permutation invariant training of deep models for speaker-independent multi-talker speech separation,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2017, pp. 241–245.
- [16]Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, “TF-GridNet: Making time-frequency domain models great again for monaural speaker separation,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2023, pp. 1–5.arxiv.org/abs/2209.03952
- [17]N. Kanda, Y. Gaur, X. Wang, Z. Meng, and T. Yoshioka, “Serialized output training for end-to-end overlapped speech recognition,” in Proc. Interspeech, 2020, pp. 2797–2801. [Online]. Available: https://doi.org/10.21437/Interspeech.2020-999
- [18]N. Kanda, J. Wu, Y. Wu, X. Xiao, Z. Meng, X. Wang, Y. Gaur, Z. Chen, J. Li, and T. Yoshioka, “Streaming multi-talker ASR with token-level serialized output training,” in Proc. Interspeech, 2022, pp. 3774–3778. [Online]. Available: https://doi.org/10.21437/Interspeech.2022-7
- [19]A. S. Subramanian, A. Das, N. Kanda, J. Li, X. Wang, and Y. Gong, “Improving practical aspects of end-to-end multi-talker speech recognition for online and offline scenarios,” in Proc. Interspeech, 2025, pp. 5508–5512.arxiv.org/abs/2506.14204
- [20]M. Shi, Z. Jin, Y. Xu, Y. Xu, S.-X. Zhang, K. Wei, Y. Shao, C. Zhang, and D. Yu, “Advancing multi-talker ASR performance with large language models,” in Proc. IEEE Spoken Lang. Technol. Workshop (SLT), 2024, pp. 14–21.arxiv.org/abs/2408.17431
- [21]Z. Peng, J. Yu, Y. Chang, Z. Wang, L. Dong, Y. Hao, Y. Tu, C. Yang, W. Wang, S. Xu, Y. Sun, H. Bao, W. Xu, Y. Zhu, Z. Wang, T. Song, Y. Xia, Z. Chi, S. Huang, L. Wang, C. Ding, S. Wang, X. Chen, and F. Wei, “VIBEVOICE-ASR technical report,” 2026, arXiv:2601.18184. [Online]. Available: https://arxiv.org/abs/2601.18184
- [22]B. Mu, X. Shi, X. Wang, Z. Guo, T. He, X. Cheng, Y. Xi, J. Xu, and L. Xie, “DiaScriber: A speech LLM for joint diarization and transcription in multi-speaker scenarios,” 2026, arXiv:2608.22796. [Online]. Available: https://arxiv.org/abs/2608.22796
- [23]Y. Fujita, N. Kanda, S. Horiguchi, K. Nagamatsu, and S. Watanabe, “End-to-end neural speaker diarization with permutation-free objectives,” in Proc. Interspeech, 2019, pp. 4300–4304.arxiv.org/abs/1909.05952
- [24]S. Horiguchi, Y. Fujita, S. Watanabe, Y. Xue, and K. Nagamatsu, “End-to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors,” in Proc. Interspeech, 2020, pp. 269–273.arxiv.org/abs/2005.09921
- [25]I. Medennikov, M. Korenovsky, T. Prisyach, Y. Khokhlov, M. Kornevskaya, I. Sorokin, T. Timofeeva, A. Mitrofanov, A. Andrusenko, I. Podluzhny, A. Laptev, and A. Romanenko, “Target-speaker voice activity detection: A novel approach for multi-speaker diarization in a dinner party scenario,” in Proc. Interspeech, 2020, pp. 274–278.DOI
- [26]L. Li, M. Cheng, W. Zhu, Y. Wang, J. Liu, and M. Li, “DM-ASR: Diarization-aware multi-speaker ASR with large language models,” 2026, arXiv:2604.22467. [Online]. Available: https://arxiv.org/abs/2604.22467
- [27]D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust DNN embeddings for speaker recognition,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2018, pp. 5329–5333.DOI
- [28]B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” in Proc. Interspeech, 2020, pp. 3830–3834.DOI
- [29]K. Žmolíková, M. Delcroix, K. Kinoshita, T. Ochiai, T. Nakatani, L. Burget, and J. Černocký, “SpeakerBeam: Speaker aware neural network for target speaker extraction in speech mixtures,” IEEE J. Sel. Topics Signal Process., vol. 13, no. 4, pp. 800–814, 2019. [Online]. Available: https://doi.org/10.1109/JSTSP.2019.2922820
- [30]Q. Wang, H. Muckenhirn, K. Wilson, P. Sridhar, Z. Wu, J. R. Hershey, R. A. Saurous, R. J. Weiss, Y. Jia, and I. L. Moreno, “VoiceFilter: Targeted voice separation by speaker-conditioned spectrogram masking,” in Proc. Interspeech, 2019, pp. 2728–2732.arxiv.org/abs/1810.04826
- [31]N. Kanda, S. Horiguchi, Y. Fujita, X. Yue, K. Nagamatsu, and S. Watanabe, “Simultaneous speech recognition and speaker diarization for monaural dialogue recordings with target-speaker acoustic models,” in Proc. IEEE Autom. Speech Recognit. Understanding Workshop (ASRU), 2019, pp. 31–38.arxiv.org/abs/1909.08103
- [32]P. Guo, X. Chang, H. Lv, S. Watanabe, and L. Xie, “SO-Whisper: Speaker-querying based Whisper model for target-speaker ASR,” IEEE Trans. Audio, Speech, Lang. Process., vol. 33, pp. 175–185, 2025. [Online]. Available: https://doi.org/10.1109/taslp.2024.3513835
- [33]M. Zaheer, S. Kottur, S. Ravibankh, B. Poczos, R. Salakhutdinov, and A. J. Smola, “Deep Sets,” in Advances in Neural Information Processing Systems, vol. 30, 2017. [Online]. Available: https://proceedings.neurips.cc/paper/2017/hash/f22e4747daa1daa27e3638d640ff442fe-Abstract.html
- [34]A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap, “A simple neural network module for relational reasoning,” in Advances in Neural Information Processing Systems, vol. 30, 2017. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/e6acf4b0f69f6f6e60e9a815938aa1ff-Paper.pdfarxiv.org/abs/1706.01427
- [35]K. Okabe, T. Koshinaka, and K. Shinoda, “Attention statistics pooling for deep speaker embedding,” in Proc. Interspeech, 2018, pp. 2252–2256.DOI
- [36]D. Ha, A. Dai, and Q. V. Le, “HyperNetworks,” in Proc. Int. Conf. on Learning Representations (ICLR), 2017. [Online]. Available: https://arxiv.org/abs/1609.09106
- [37]Y. Zhao, J. Li, and Y. Gong, “Low-rank plus diagonal adaptation for deep neural networks,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2016, pp. 5005–5009. [Online]. Available: https://doi.org/10.1109/icassp.2016.7472630
- [38]N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attarian, and S. Gelly, “Parameter-efficient transfer learning for NLP,” in Proc. Int. Conf. Mach. Learn. (ICML), ser. Proc. Mach. Learn. Res., K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 2790–2799. [Online]. Available: https://proceedings.mlr.press/v97/houlsby19a.htmlarxiv.org/abs/1902.00751
- [39]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Proc. Int. Conf. on Learning Representations (ICLR), 2022. [Online]. Available: https://arxiv.org/abs/2106.09685
- [40]F. Yu, S. Zhang, Y. Fu, L. Xie, S. Zheng, Z. Du, W. Huang, P. Guo, Z. Yan, B. Ma, X. Xu, and H. Bu, “M2MeT: The ICASSP 2022 multi-channel multi-party meeting transcription challenge,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2022, pp. 6167–6171. [Online]. Available: https://doi.org/10.1109/ICASSP43922.2022.9746465
- [41]Y. Fu, L. Cheng, S. Lv, Y. Jv, Y. Kong, Z. Chen, Y. Hu, L. Xie, J. Wu, H. Bu, X. Xu, J. Du, and J. Chen, “AISHELL-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario,” in Proc. Interspeech, 2021, pp. 3665–3669.arxiv.org/abs/2104.03603
- [42]J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, I. McCowan, W. Post, D. Reidsma, and P. Wellner, “The AMI meeting corpus: A pre-announcement,” in Machine Learning for Multimodal Interaction. Springer Berlin Heidelberg, 2006, pp. 28–39. [Online]. Available: https://doi.org/10.1007/11677482_3
- [43]A. Vinnikov, A. Ivry, A. Hurvitz, I. Abramovski, S. Koubi, I. Gurvich, S. Peer, X. Xiao, B. M. Elizalde, N. Kanda, X. Wang, S. Shaer, S. Yagev, Y. Asher, S. Sivanskaran, Y. Gong, M. Tang, H. Wang, and E. Krupka, “NOTASOFAR-1 challenge: New datasets, baseline, and tasks for distant meeting transcription,” in Proc. Interspeech, 2024, pp. 5003–5007.arxiv.org/abs/2401.08887
- [44]M. Gao, S. Wu, H. Chen, J. Du, C.-H. Lee, S. Watanabe, J. Chen, S. M. Siniscalchi, and O. Scharenborg, “The multimodal information based speech processing (MISP) 2025 challenge: Audio-visual diarization and recognition,” in Proc. Interspeech, 2025, pp. 188–1892.
- [45]Y. Dai, H. Wang, X. Li, Z. Zhang, S. Wang, L. Xie, X. Xu, H. Guo, S. Zhang, H. Bu, and W. Chen, “AISHELL-5: The first open-source in-car multi-channel multi-speaker speech dataset for automatic speech diarization and recognition,” in Proc. Interspeech, 2025, pp. 5493–5497.arxiv.org/abs/2505.23036
- [46]M. Van Segbroeck, A. Zaid, K. Kutsenko, C. Huerta, T. Nguyen, X. Luo, B. Hoffmeister, J. Trmal, M. Omologo, and R. Maas, “DiPCo—dinner party corpus,” in Proc. Interspeech, 2020, pp. 434–436. [Online]. Available: https://www.isca-archive.org/interspeech_2020/segbroeck20_interspeech.html
- [47]J. Han, F. Landini, J. Rohdin, A. Silnova, M. Diez, and L. Burget, “Leveraging self-supervised learning for speaker diarization,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2025, pp. 1–5. [Online]. Available: https://doi.org/10.1109/ICASSP49660.2025.10889475
- [48]R. Wang, J. Du, W. Zhu, and Y. Wang, “Sequence-to-sequence refinement for EEND-VC diarization with a shared encoder,” 2026, arXiv:2611.02709. [Online]. Available: https://arxiv.org/abs/2611.02709
- [49]X. Shi, X. Wang, Z. Guo, Y. Wang, P. Zhang, X. Zhang, Z. Guo, H. Hao, Y. Xi, B. Yang, J. Xu, J. Zhou, and J. Lin, “Owen3-ASR technical report,” 2026, arXiv:2601.21337. [Online]. Available: https://arxiv.org/abs/2601.21337
- [50]F. Shen, F. Xie, J. Li, K. Xie, L. Xie, X. Tang, X. Geng, Y. Jia, Y. Hu, Y. Han, Y. Wu, Z. Dai, J. Chen, K. Huang, M. Wei, and Y. Li, “FireRedAudio: A general-purpose audio language model with decoupled continuous representations for understanding and generation,” 2026, arXiv:2608.24168. [Online]. Available: https://arxiv.org/abs/2608.24168
- [51]T. von Neumann, C. Boeddeker, M. Delcroix, and R. Haeb-Umbach, “MeetEval: A toolkit for computation of word error rates for meeting transcription systems,” in Proc. Int. Workshop Speech Process. Everyday Environ. (CHiME), 2023, pp. 27–32.arxiv.org/abs/2307.11394
- [52]M. Bisani and H. Ney, “Bootstrap estimates for confidence intervals in ASR performance evaluation,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), vol. 1, Montreal, Canada, May 2004, pp. 409–412. [Online]. Available: https://doi.org/10.1109/ICASSP.2004.1326009
- [53]Qwen Team, “Qwen3.8-Omni: Towards native omni-modal agents,” 2026, arXiv:2609.25611. [Online]. Available: https://arxiv.org/abs/2609.25611








