betaFeedback

WorksEngineering

Structured Speaker Conditioning for Diarization-Guided Target-Speaker ASR

Abstract

We propose structured speaker conditioning (SSC) for diarization-guided target-speaker recognition. SSC combines local activity structure with session-level acoustic identity to construct identity-consistent target conditions that control the acoustic encoder. Target-centric relational encoding (TCRE) retains per-speaker activity histories and factorized second- and third-order relations, capturing overlap and turn changes with aggregation cost linear in the number of tracks. Reference-based speaker conditioning (RSC) supplies session-derived identities that disambiguate local activity patterns within TCRE. Conditional low-rank modulation (CLRM) converts the joint condition into channel-wise affine modulation and a low-rank cross-channel residual with a condition-dependent core. On Qwen3-ASR-0.6B with oracle diarization, SSC uses 1.46M conditioning parameters and achieves a 6.23% mean relative reduction in concatenated minimum-permutation error rate (cpER) over four-state diagonal conditioning across four meeting test sets. SSC also transfers across audio foundation models. With predicted diarization and target-error group sequence policy optimization, the FireRedAudio text system achieves mean relative error reductions of 17.9% and 17.3% over DiCoW and MOSS-TD on the same four sets, and also outperforms both on three unseen domains. Scenario analyses identify gains under challenging overlap conditions, while fixed-checkpoint interventions support the contributions of session identity and the condition-dependent core. A timestamp variant jointly produces sentence boundaries and text. With predicted diarization, macro-average cpER across the four sets is 15.60% for text and 15.96% with timestamps.

Introduction

MEETING transcription should recover what each participant said while maintaining consistent speaker attribution across turn changes and overlapping speech. Target-speaker automatic speech recognition (ASR) addresses this problem by generating a transcript of a specified speaker’s speech from the mixture. When a long meeting is processed in windows, the condition should keep referring to the same session speaker as the local combination of active speakers changes. How can a target condition remain identity-consistent under changing local activity and ambiguous speaker cues while effectively controlling the acoustic encoder?

Meeting transcription systems combine diarization, separation, and recognition in modular pipelines [1], [2], or jointly predict speaker identities and transcripts [3]. Recent speech-language models also generate timestamps, as in MOSS-TD [4] and TagSpeech [5]. Diarization-guided target-speaker ASR directly supplies speaker-activity tracks to a recognition model. DiCoW [6], [7] demonstrates this approach by adapting Whisper [8] with frame-level diarization-dependent transformations (FDDT), and Dixtral [9] carries this conditioning into a spoken language model with a frozen language decoder. For each target, the activity tracks are converted into four states: silence, target-only, non-target-only, and overlap (STNO). Their probabilities control affine transformations of the acoustic features. This provides a basis for studying how diarization should be represented and how the condition should act on a pre-trained encoder.

The information retained in the condition determines which speaker configurations the encoder can distinguish. Four-state conditioning can map different interferer counts and hand-over histories to the same target condition. Encoding individual track histories before aggregation preserves these distinctions, including how the target and other speakers overlap and alternate over time.

Acoustic identity complements this temporal structure. Two speakers with identical activity throughout the available context receive the same activity-only condition, while speech elsewhere in the session can provide acoustic cues for distinguishing their voices. SE-DiCoW [10] introduces self-enrollment through encoder cross-attention, and SC-SOT [11] conditions the decoder on speaker embeddings and activity. Combining identity with track histories within a frame-level condition lets both sources determine the target–interferer relations used for recognition.

The modulation operator then translates these relations into acoustic adaptation. Diagonal FDDT scales and shifts channels, while self-speaker adaptation [12] generates speaker-specific encoder kernels from activity supervision. A low-rank cross-channel operator offers a compact way to vary channel combinations through a small generated core and shared read/write projections. Jointly designing the condition and this operator lets target identity and local speaker interactions control how acoustic features are combined.

We address this question with structured speaker conditioning (SSC). Target-centric relational encoding (TCRE) retains per-speaker activity histories and their overlap and turn-taking relations. Reference-based speaker conditioning (RSC) incorporates speaker identity information derived from session audio and diarization into these track representations, so TCRE can distinguish targets with similar local activity. Conditional low-rank modulation (CLRM) uses the joint condition to control channel-wise adaptation and cross-channel feature combinations in the acoustic encoder.

  • TCRE constructs permutation-equivariant target conditions using a shared causal temporal encoder and factorized second- and third-order relations. The representation captures activity history, interferer counts, and interactions with distinct interferer pairs. We relate its design to an analysis of STNO’s activity compression.

  • RSC associates session-derived acoustic identities with diarization tracks and augments TCRE features with this identity evidence. It supports target recognition when target speech is fully overlapped, target activity is missed, or speaker tracks are swapped.

  • CLRM combines feature-wise linear modulation (FiLM) [13] with a rank-8 cross-channel residual whose core varies with the target condition. The operator extends diagonal modulation with condition-dependent channel mixing. Operator comparisons and frozen-core interventions link this design to lower error.

SSC is trained using supervised fine-tuning (SFT). The FireRedAudio-based systems are subsequently optimized using group sequence policy optimization (GSPO) [14], with rewards based on target-speaker transcription error.

We evaluate SSC on Qwen3-ASR and FireRedAudio, using four in-domain meeting test sets (AliMeeting, AISHELL-4, AMI-SDM, and NOTSOFAR-1) and three out-of-domain test sets. Recognition performance is evaluated using concatenated minimum-permutation error rate (cpER), calculated at the character level for Chinese and the word level for English. On Qwen3-ASR-0.6B with oracle diarization, complete SSC uses 1.46M conditioning parameters and achieves a mean relative cpER reduction of 6.23% over STNO conditioning with the diagonal FDDT operator (STNO-FDDT). With predicted diarization, FireRedAudio + SSC + GSPO achieves mean relative cpER reductions of 17.9% and 17.3% over DiCoW and MOSS-TD, respectively, across the four in-domain sets. Scenario-specific analyses show larger gains in windows with at least three concurrent speakers and on fully overlapped target speech, while condition interventions show that session identity contributes most under target misses and track swaps. A timestamp-output variant jointly generates sentence boundaries and transcripts for recognition and timing evaluation.

Related Work

Meeting Transcription and Target-Speaker Recognition

Modular meeting transcription combines separation, recognition, and speaker attribution. Yoshioka et al. [1] integrate continuous speech separation into an online separate–recognize–diarize system. Permutation-invariant training [15] addresses the output-order ambiguity of speaker-independent separation, while TF-GridNet [16] models local and global time–frequency structure. Building on TF-GridNet separation, von Neumann et al. [2] combine ASR with transcription-supported diarization. These pipelines expose separated speech and speaker assignment as intermediate representations.

Joint recognition approaches learn to generate multiple speakers’ transcripts from the mixture. Serialized output training (SOT) [17] concatenates their transcriptions in a single decoder sequence. Speaker-attributed extensions [3] add a speaker inventory and jointly model speaker counting, transcription, and identification. Token-level SOT [18] interleaves recognition tokens and marks switches between virtual output channels for streaming recognition. Segment-based SOT [19] further addresses practical online and offline decoding, and LLM-based SOT [20] combines a pretrained speech encoder with a language-model decoder. Recent speech-language models extend structured generation to speaker labels and timestamps. MOSS-TD [4] and VibeVoice-ASR [21] jointly generate these outputs for long recordings, while DiaScriber [22] adapts a speech language model for joint diarization and transcription. TagSpeech [5] uses interleaved time anchors to synchronize semantic and speaker streams for temporal grounding. Target-speaker ASR supplies an explicit condition specifying whose speech to decode. SSC adopts this formulation and uses session-level diarization to associate window-level predictions with meeting speakers.

Activity and Acoustic Identity as Speaker Conditions

Speaker activity provides temporal information for target recognition. End-to-end neural diarization directly estimates overlapping speaker activities [23]; encoder–decoder attractors extend this formulation to an unknown number of speakers [24]. TS-VAD [25] instead conditions activity estimation on speaker representations. These diarization approaches supply activity estimates; target-speaker recognition uses such estimates to guide transcription. Whisper-based target-speaker ASR [6, 7] conditions the acoustic encoder through STNO-weighted feature transformations. In the diagonal configuration, these transformations map activity states to channel-wise gains and offsets. Dixtral [9] couples the DiCoW encoder with Voxtral, adapting the acoustic encoder and FDDT while freezing the language decoder and modality adapter.

Figure 1 illustrates the information compressed by this four-state representation. With binary tracks, one or two simultaneous interferers can yield identical STNO conditions. Different hand-over times between non-target speakers can also leave the STNO sequence unchanged despite distinct per-track histories. These cases motivate encoding track histories before symmetric aggregation and preserving relations between the target and individual interferers.

Figure 1. STNO activity compression. Different interferer counts (a) or hand-over histories (b) can yield the same four-state target condition despite distinct non-target track configurations.

Self-speaker adaptation [12] generates speaker-specific kernels from speaker-activity supervision and injects them into selected encoder layers. At the decoder, SC-SOT [11] combines speaker embeddings from a jointly trained diarization model with activity information. DM-ASR [26] converts speaker labels and segment boundaries into prompts for speaker- and time-conditioned dialogue turns. SSC combines explicit target–interferer relations with conditional low-rank modulation in the encoder.

Reference speech supplies acoustic identity complementary to activity. Speaker embedding models such as x-vectors [27] and ECAPA-TDNN [28] summarize utterances as identity representations. SpeakerBeam [29] learns a target representation from an adaptation utterance and uses it to condition speaker extraction. VoiceFilter [30] combines a speaker embedding with spectrogram masking for targeted separation. For recognition, Kanda et al. [31] alternate speaker-embedding estimation with target-speaker ASR, obtaining identity evidence from the dialogue itself. SQ-Whisper [32] uses trainable queries to derive enrollment-guided speaker prompts from the mixture, and supplies the prompts to Whisper’s encoder and decoder. SE-DiCoW [10] selects an enrollment segment from the conversation using diarization and introduces its encoded feature sequence through cross-attention at each encoder layer, while retaining the STNO path.

Deep Sets [33] uses symmetric aggregation over unordered inputs, while Relation Networks [34] aggregate learned pairwise relations. In SSC, RSC identities augment temporal track features before TCRE constructs target-centric second- and third-order relations. The target identity also enters the final condition directly, jointly controlling diagonal modulation and low-rank cross-channel mixing.

Method

Task Definition and Conditioning Pipeline

Let X∈RT×F\mathbf{X} \in\mathbb{R}^{T \times F} denote the acoustic features of a meeting window WW, and let A=[as(t)]∈[0,1]S×T\mathbf{A} = [a_s(t)] \in[0,1]^{S \times T} contain its speaker-activity tracks on the same frame grid. Each as(t)a_s(t) is the fraction of a frame covered by diarization intervals for speaker ss. Track indices are associated with session speakers, and q∈{1,…,S}q \in\{1,\ldots,S\} identifies the requested speaker. Given session audio Xsess\mathbf{X}^{\mathrm{sess}} and diarization Asess\mathbf{A}^{\mathrm{sess}}, RSC derives a set of acoustic identities EW={(es,vs)}s=1S\mathcal{E}_W = \{(e_s,v_s)\}_{s=1}^{S}, where es∈Rdee_s \in\mathbb{R}^{d_e} and vs∈{0,1}v_s \in\{0,1\} indicates reference availability. The target transcript is

y^q=arg⁡max⁡y pθ(y∣X,A,EW,q).(1)\hat{y}_q = \underset{y}{\arg\max}\ p_\theta(y \mid\mathbf{X},\mathbf{A},\mathcal{E}_W,q). \tag*{(1)}

In Figure 2, RSC supplies the acoustic identity evidence EW\mathcal{E}_W. TCRE combines this evidence with the activity tracks to construct cq(t)c_q(t), and CLRM maps the aligned condition to encoder transformations. This computation is expressed as

Figure 2. SSC architecture. (a) RSC derives speaker identities from session audio and diarization. (b) TCRE combines these identities with local track histories into cq(t)c_q(t). (c) CLRM aligns the condition to encoder tokens and generates modulation parameters. The enlarged operator below combines diagonal FiLM with a low-rank read–mix–write branch on the same acoustic feature. Read/write bases are shared across conditions; the core varies with the condition. The pre-stack position ℓ=0\ell= 0 uses FiLM only.

EW=RRSC(Xsess,Asess;W),(2)\mathcal{E}_W = \mathcal{R}_{\mathrm{RSC}}(\mathbf{X}^{\mathrm{sess}},\mathbf{A}^{\mathrm{sess}};W), \tag*{(2)}
cq(t)=ΦTCRE(A,EW;q)(t),(3)c_q(t) = \Phi_{\mathrm{TCRE}}(\mathbf{A},\mathcal{E}_W;q)(t), \tag*{(3)}
cˉq=Penc(cq),(4)\bar{c}_q = \mathcal{P}_{\mathrm{enc}}(c_q), \tag*{(4)}
h~ℓ,τ=Aℓ(cˉq(τ))hℓ,τ+βℓ(cˉq(τ)).(5)\tilde{\mathbf{h}}_{\ell,\tau} = \mathcal{A}_{\ell}(\bar{c}_q(\tau))\mathbf{h}_{\ell,\tau} + \boldsymbol{\beta}_{\ell}(\bar{c}_q(\tau)). \tag*{(5)}

Here Penc\mathcal{P}_{\mathrm{enc}} aligns frame-level conditions with the backbone’s acoustic tokens, τ\tau indexes these tokens, and hℓ,τ∈RD\mathbf{h}_{\ell,\tau} \in\mathbb{R}^{D} is the acoustic representation before modulation at position ℓ\ell. CLRM specifies the matrix Aℓ\mathcal{A}_{\ell} and offset βℓ\boldsymbol{\beta}_{\ell}. There is a pre-stack position ℓ=0\ell=0 and one position before each of the LL encoder blocks, ℓ=1,…,L\ell=1,\ldots,L. The resulting encoder features pass through the backbone’s native audio projection and decoder to generate yqy_q.

Target-Centric Relational Encoding

TCRE determines which local activity information enters the target condition. It encodes each track’s history before aggregation, preserving overlap and alternation patterns as target-centric relations. RSC identity coordinates augment these track features, allowing the same relational construction to incorporate acoustic evidence about who is speaking.

  1. From compressed states to track histories: Let QqQ_q be the set of valid non-target tracks and Nq=∣Qq∣N_q = |Q_q|. Padded tracks are excluded from this set. The activity compression discussed in Section II motivates track-level encoding. In the STNO construction used by DiCoW [6, 7], uq(t)=∏s∈Qq(1−as(t))u_q(t) = \prod_{s \in Q_q}(1-a_s(t)) summarizes the non-target tracks, giving the four coefficients

pq,S=(1−aq)uq,pq,T=aquq,p_{q,\mathrm{S}} = (1-a_q)u_q,\qquad p_{q,\mathrm{T}} = a_q u_q,
pq,N=(1−aq)(1−uq),pq,O=aq(1−uq).(6)p_{q,\mathrm{N}} = (1-a_q)(1-u_q),\qquad p_{q,\mathrm{O}} = a_q(1-u_q). \tag*{(6)}

For binary activities, these coefficients indicate silence, target-only, non-target-only, and target-overlap states. Aggregating the non-target tracks in this way merges different interferer counts and hand-over histories. TCRE separately encodes tracks and computes symmetric relations around the target.

A shared causal temporal encoder maps each activity track to ξs(t)∈Rd\boldsymbol{\xi}_s(t) \in\mathbb{R}^d. With SiLU activations, the two convolutional layers are

ζs(t)=SiLU⁡(∑j=04W1[j]as(t−j)+b1),(7)\boldsymbol{\zeta}_s(t) = \operatorname{SiLU}\left(\sum_{j=0}^{4}\mathbf{W}_1[j]a_s(t-j)+\mathbf{b}_1\right), \tag*{(7)}
ξs(t)=SiLU⁡(∑j=04W2[j]ζs(t−4j)+b2).(8)\boldsymbol{\xi}_s(t) = \operatorname{SiLU}\left(\sum_{j=0}^{4}\mathbf{W}_2[j]\boldsymbol{\zeta}_s(t-4j)+\mathbf{b}_2\right). \tag*{(8)}

The kernels have size five and dilations one and four, respectively; W1[j]∈Rd\mathbf{W}_1[j] \in\mathbb{R}^d, W2[j]∈Rd×d\mathbf{W}_2[j] \in\mathbb{R}^{d \times d}, and both biases have dimension dd. We use d=32d=32. The receptive field covers 21 activity frames, or 210 ms on the 10 ms grid. Values outside the valid window are zero padded, and invalid intermediate positions are masked. Shared weights make the temporal encoding independent of track numbering.

The local feature and identity-augmented track feature are

zs(t)=[as(t);ξs(t)]∈Rd+1,(9)\mathbf{z}_s(t) = [a_s(t); \xi_s(t)] \in\mathbb{R}^{d+1}, \tag*{(9)}
ϕs(t)=[1;zs(t);ηs]∈Rdϕ,dϕ=d+de+3.(10)\phi_s(t) = [1; \mathbf{z}_s(t); \eta_s] \in\mathbb{R}^{d_\phi}, \qquad d_\phi= d + d_e + 3. \tag*{(10)}

The vector ηs∈Rde+1\eta_s \in\mathbb{R}^{d_e+1} carries the reference identity and its validity, as defined in Section III-C. For a fixed target request, it remains constant over time. The leading constant in ϕs\phi_s permits the factorized relations below to include constant and lower-order terms as well as interactions.

Factorized second- and third-order relations: TCRE

TCRE represents the target’s relation to individual interferers and to pairs of distinct interferers.

For k∈{2,3}k \in\{2, 3\}, let Qk,Kk∈RR×dϕ\mathbf{Q}_k, \mathbf{K}_k \in\mathbb{R}^{R \times d_\phi} be learned target-side and non-target-side projections, with relation width R=16R = 16. Writing ks=K3ϕs(t)\mathbf{k}_s = \mathbf{K}_3\phi_s(t) for the third-order non-target factors, we compute

g2,q(t)=(Q2ϕq(t))⊙∑s∈OqK2ϕs(t),(11)\mathbf{g}_{2,q}(t) = (\mathbf{Q}_2\phi_q(t)) \odot\sum_{s\in O_q} \mathbf{K}_2\phi_s(t), \tag*{(11)}
g3,q(t)=(Q3ϕq(t))⊙∑i<ji,j∈Oqki⊙kj.(12)\mathbf{g}_{3,q}(t) = (\mathbf{Q}_3\phi_q(t)) \odot\sum_{\substack{i<j\\i,j\in O_q}} \mathbf{k}_i \odot\mathbf{k}_j. \tag*{(12)}

The second-order term relates the target to one non-target track; the third-order term relates the target to two different non-target tracks. These orders count the participating tracks. Since the track features contain temporal history, a pairwise history interaction also describes hand-overs between interferers that are not currently active together.

The pair sum is evaluated without enumerating all pairs:

∑i<ji,j∈Oqki⊙kj=12[(∑i∈Oqki)⊙2−∑i∈Oqki⊙2],(13)\sum_{\substack{i<j\\i,j\in O_q}} \mathbf{k}_i \odot\mathbf{k}_j = \frac{1}{2}\left[\left(\sum_{i\in O_q}\mathbf{k}_i\right)^{\odot2} - \sum_{i\in O_q}\mathbf{k}_i^{\odot2}\right], \tag*{(13)}

where ⊙\odot and (⋅)⊙2(\cdot)^{\odot2} denote element-wise multiplication and squaring. This identity removes self-pairs, making aggregation linear in NqN_q for fixed feature dimensions. The third-order term is zero with fewer than two valid non-target tracks.

Figure 3 summarizes the two branches, including the shared projections and self-pair removal in the third-order pool.

Figure 3. Factorized target-centric relations. (a) The second-order branch pools individual interferer factors. (b) The third-order branch combines the target with distinct interferer pairs. Subtracting P\mathbf{P} from S⊙2\mathbf{S}\odot^2 removes self-pairs, and the factor 1/21/2 counts each unordered pair once. The grid illustrates this pair selection. Sums run over valid non-target tracks Oq\mathcal{O}_q, giving O(Nq)O(N_q) pooling, with ki=K3ϕi(t)\mathbf{k}_i = K_3\boldsymbol{\phi}_i(t) and element-wise multiplication ⊙\odot.

Fixed-dimensional target condition and symmetry

The complete frame-level condition is

cq(t)=[zq(t);g2,q(t);g3,q(t);nq(t);Nq;ηq],(14)\mathbf{c}_q(t) = [\mathbf{z}_q(t); \mathbf{g}_{2,q}(t); \mathbf{g}_{3,q}(t); n_q(t); N_q; \eta_q], \tag*{(14)}

where nq(t)=∑s∈Oqas(t)n_q(t) = \sum_{s\in O_q} a_s(t) is the current non-target activity mass. Thus nq(t)n_q(t) describes current activity, whereas NqN_q counts valid non-target tracks in the window. With d=de=32d = d_e = 32 and R=16R = 16, dϕ=67d_\phi= 67 and cq(t)∈R100\mathbf{c}_q(t) \in\mathbb{R}^{100}, independently of the number of speakers. The activity-only model omits the 33 identity coordinates, giving a 67-dimensional condition.

For binary activities, νq(t)=1[nq(t)=0]\nu_q(t) = \mathbb{1}[n_q(t) = 0], so the retained coordinates (aq(t),nq(t))(a_q(t), n_q(t)) determine the STNO coefficients before temporal alignment. The additional history and relation coordinates express the finer activity structures motivating TCRE: nqn_q distinguishes one interferer from two, and the temporal relations describe which tracks interfere over time.

For any permutation σ\sigma of the speaker tracks, the condition satisfies

ΦTCRE(σA,σEW;σ(q))=ΦTCRE(A,EW;q).(15)\Phi_{\mathrm{TCRE}}(\sigma A,\sigma\mathcal{E}_W;\sigma(q))=\Phi_{\mathrm{TCRE}}(A,\mathcal{E}_W;q). \tag*{(15)}

The temporal encoder and projection matrices are shared across tracks, and all non-target aggregations are symmetric sums over tracks or unordered pairs, following the shared-encoding and symmetric-aggregation principle of set representations [33]. Consequently, relabeling the tracks and requested target preserves the condition for the same physical speaker. The target conditions are thus permutation equivariant.

Reference-Based Speaker Conditioning

1) Diarization-guided session evidence: Activity histories describe temporal interactions, but two targets with identical activity throughout the available local history receive the same activity-only TCRE condition. RSC addresses this ambiguity by supplying session-level acoustic identity to the track features and the target coordinates of cq(t)\mathbf{c}_q(t). This binds the requested track to a voice within the relational condition. Figure 4 illustrates how acoustic identities distinguish targets with identical local activity. Algorithm 1 summarizes reference selection and condition construction.

Figure 4. Session identity distinguishes locally ambiguous targets. (a) Identical local activity histories yield the same activity-only target condition. (b) RSC encodes speaker-exclusive references outside the current window WW. For valid, distinct identities (vA=vB=1v_A = v_B = 1, eA≠eB\mathbf{e}_A \ne\mathbf{e}_B), the target identity coordinates ηq\boldsymbol{\eta}_q distinguish the two conditions. Colored cells highlight these coordinates. The timeline elements are schematic.

For each speaker, the selection map

(Xsref,vs)=Ss(Xsess,Asess;W)(16)(\mathbf{X}^{\mathrm{ref}}_s,v_s)=\mathcal{S}_s(\mathbf{X}^{\mathrm{sess}},\mathbf{A}^{\mathrm{sess}};W) \tag*{(16)}

collects speaker-exclusive evidence from other intervals of the session. We exclude the current window with a temporal margin, subtract intervals assigned to other speakers, and trim candidate boundaries. Among segments meeting a minimum segment duration, we select up to KrefK_{\mathrm{ref}}, prioritizing longer segments and capping the total duration at Tmax⁡T_{\max}. Distance to WW and start time resolve ties. A track has vs=1v_s=1 when at least Tmin⁡T_{\min} of selected evidence is available and vs=0v_s=0 otherwise. At inference, input diarization supplies the selection intervals. Hyperparameter values are given in Section IV-A.

For online simulated mixtures, references are independent clean utterances of the same source identity. The selection excludes the utterances used in the current mixture and candidates with the same normalized text. This provides acoustic evidence for identity across different speech content.

2) Shared reference encoding and identity pooling: Reference audio passes through the shared acoustic encoder prefix with target-only activity conditioning. We use the first Le=6L_e=6 encoder blocks and pool their valid output tokens. Let hs,τref∈RD\mathbf{h}^{\mathrm{ref}}_{s,\tau}\in\mathbb{R}^{D} denote these representations and rs,τ=LN⁡(hs,τref)\mathbf{r}_{s,\tau}=\operatorname{LN}(\mathbf{h}^{\mathrm{ref}}_{s,\tau}). Using learned token weights as in attentive speaker pooling [35], we form a weighted mean:

αs,τ=exp⁡(wαTrs,τ)∑τ′exp⁡(wαTrs,τ′),(17)\alpha_{s,\tau}=\frac{\exp(\mathbf{w}_{\alpha}^{\mathsf{T}}\mathbf{r}_{s,\tau})}{\sum_{\tau'}\exp(\mathbf{w}_{\alpha}^{\mathsf{T}}\mathbf{r}_{s,\tau'})}, \tag*{(17)}
es=We∑ταs,τrs,τ+be∈Rde.(18)\mathbf{e}_s=\mathbf{W}_e\sum_{\tau}\alpha_{s,\tau}\mathbf{r}_{s,\tau}+\mathbf{b}_e\in\mathbb{R}^{d_e}. \tag*{(18)}

Here wα∈RD\mathbf{w}_{\alpha}\in\mathbb{R}^{D}, We∈Rde×D\mathbf{W}_e\in\mathbb{R}^{d_e\times D}, and de=32d_e=32. The sums include only valid reference tokens. Reference encoding and pooling are differentiable under the target-transcription loss.

For a request targeting qq, the identity features are gated by target reference availability:

v~s∣q=vsvq,ηs=[v~s∣qes;v~s∣q].(19)\widetilde{v}_{s|q}=v_sv_q,\qquad\boldsymbol{\eta}_s=[\widetilde{v}_{s|q}\mathbf{e}_s;\widetilde{v}_{s|q}]. \tag*{(19)}

We set unavailable identities to zero. If vq=0v_q=0, all ηs\boldsymbol{\eta}_s are zero and the request uses activity-only conditioning. With a valid target reference, each available non-target identity participates in the relational features. The target identity also enters cq\mathbf{c}_q directly in (14). Identity therefore influences both the relations and the final condition supplied to the modulation generator.

3) Paired target binding: Training includes paired requests for the same highly overlapped mixture. Two targets A and B have matched speech durations satisfying min⁡(δA,δB)/max⁡(δA,δB)≥0.95\min(\delta_A,\delta_B)/\max(\delta_A,\delta_B)\geq0.95. Both views share the mixture and augmentation randomness and are kept together within a device batch. Their supervised contribution is

Lbind=−∑s∈{A,B}log⁡pθ(ys∣X,cs).(20)\mathcal{L}_{\mathrm{bind}}=-\sum_{s\in\{\mathrm{A},\mathrm{B}\}}\log p_{\theta}(y_s\mid\mathbf{X},\mathbf{c}_s). \tag*{(20)}

Algorithm 1. RSC condition construction

Input: Session mixture Xsess\mathbf{X}^{\mathrm{sess}} and diarization Asess\mathbf{A}^{\mathrm{sess}}; local activity A\mathbf{A}, current window WW, target qq, and reference hyperparameters Kref,Tmin⁡,Tmax⁡K_{\mathrm{ref}}, T_{\min}, T_{\max}.

Output: Frame-level target condition cq(t)c_q(t).

  1. for s=1,…,Ss = 1, \ldots, S do

  2. Select speaker-exclusive evidence Xsref\mathbf{X}^{\mathrm{ref}}_s outside WW using (16).

  3. Keep up to KrefK_{\mathrm{ref}} segments, totaling at most Tmax⁡T_{\max}.

  4. Set vs←1v_s \leftarrow1 if total duration ≥Tmin⁡\ge T_{\min}, else vs←0v_s \leftarrow0.

  5. Compute es\mathbf{e}_s by (18) if vs=1v_s = 1; otherwise set es=0\mathbf{e}_s = 0.

  6. end for

  7. Use v~s∣q=vsvq\tilde{v}_{s|q} = v_s v_q in ηs\eta_s for all ss (19).

  8. EW←{(es,vs)}s=1S\mathcal{E}_W \leftarrow\{(\mathbf{e}_s,v_s)\}_{s=1}^{S}; cq(t)←ΦTCRE(A,EW;q)(t)c_q(t) \leftarrow\Phi_{\mathrm{TCRE}}(\mathbf{A},\mathcal{E}_W;q)(t).

  9. return cq(t)c_q(t)

Training: Sample real / simulated / binding views with weights ρ\boldsymbol{\rho}. For a binding view, use two targets from the same mixture with matched durations and minimize Lbind\mathcal{L}_{\mathrm{bind}} in (20).

Paired requests train target selection using acoustic identity when activity patterns provide little separation. Real, ordinary simulated, and paired-binding views use sampling weights ρ=(ρreal,ρsim,ρbind)\boldsymbol{\rho} = (\rho_{\mathrm{real}},\rho_{\mathrm{sim}},\rho_{\mathrm{bind}}) (Section IV-A).

Conditional Low-Rank Modulation

The joint TCRE–RSC condition specifies the requested speaker and the surrounding activity structure. CLRM makes this information control recognition by generating both channel-wise gains and offsets and condition-dependent combinations of acoustic channels. CLRM combines feature-wise affine conditioning [13] with a compact cross-channel transform, using a hypernetwork to generate condition-dependent parameters [36]. Low-rank-plus-diagonal acoustic adaptation [37] decomposes speaker transforms into diagonal and low-rank terms; bottleneck adapters [38] and LoRA [39] use compact modules or weight updates for efficient adaptation. In CLRM, learned read/write bases are shared across conditions, while the diagonal parameters and a small mixing core vary with the target’s activity and identity at each encoder token.

Alignment with the acoustic token grid

Relations are constructed at the activity-frame rate and then aligned with the backbone’s convolutional subsampling. For stride κ\kappa, define the valid-normalized pooling operator

Pκ(c)(τ)=∑j=−11m(κτ+j)c(κτ+j)max⁡(1,∑j=−11m(κτ+j)).(21)\mathcal{P}_{\kappa}(\mathbf{c})(\tau) = \frac{\sum_{j=-1}^{1}m(\kappa\tau+j)\mathbf{c}(\kappa\tau+j)} {\max\left(1,\sum_{j=-1}^{1}m(\kappa\tau+j)\right)}. \tag*{(21)}

where mm marks valid input positions. Valid output lengths and masks are recomputed after each stage, separately for each acoustic chunk. For Qwen3-ASR, Penc=P2∘P2∘P2\mathcal{P}_{\mathrm{enc}} = \mathcal{P}_2 \circ\mathcal{P}_2 \circ\mathcal{P}_2, mapping 10 ms frames to 80 ms tokens. FireRedAudio uses P2∘P1\mathcal{P}_2\circ\mathcal{P}_1 to match its native encoder grid. Validity normalization prevents padding from diluting the condition. The signed relation coordinates are averaged as features.

Shared generator and position-specific heads

All modulation positions use a shared hidden layer, with a learned position embedding ωℓ∈Rdω\boldsymbol{\omega}_{\ell} \in\mathbb{R}^{d_{\omega}}:

ψℓ,τ=SiLU⁡(Wg[cˉq(τ);ωℓ]+bg)∈Rdh.(22)\boldsymbol{\psi}_{\ell,\tau} = \operatorname{SiLU}\left(\mathbf{W}_{g}\left[\bar{\mathbf{c}}_{q}(\tau);\boldsymbol{\omega}_{\ell}\right]+\mathbf{b}_{g}\right) \in\mathbb{R}^{d_h}. \tag*{(22)}

We use dh=32d_h = 32 and dω=8d_{\omega} = 8. Each position has an affine output head,

[γℓ,τ;βℓ,τ]=Wℓfψℓ,τ+bℓf∈R2D,(23)[\boldsymbol{\gamma}_{\ell,\tau};\boldsymbol{\beta}_{\ell,\tau}] = \mathbf{W}^{\mathrm{f}}_{\ell}\boldsymbol{\psi}_{\ell,\tau}+\mathbf{b}^{\mathrm{f}}_{\ell} \in\mathbb{R}^{2D}, \tag*{(23)}

and each encoder-block position has a low-rank core head,

Mℓ,τ=1rreshape⁡r×r[tanh⁡(Wℓxψℓ,τ+bℓx)].(24)\mathbf{M}_{\ell,\tau} = \frac{1}{\sqrt{r}}\operatorname{reshape}_{r\times r} \left[\tanh\left(\mathbf{W}^{\mathrm{x}}_{\ell}\boldsymbol{\psi}_{\ell,\tau}+\mathbf{b}^{\mathrm{x}}_{\ell}\right)\right]. \tag*{(24)}

The head dimensions are Wℓf∈R2D×dh\mathbf{W}^{\mathrm{f}}_{\ell} \in\mathbb{R}^{2D\times d_h} and Wℓx∈Rr2×dh\mathbf{W}^{\mathrm{x}}_{\ell} \in\mathbb{R}^{r^2\times d_h}. Thus the same activity–identity condition controls channel-wise gains, offsets, and cross-channel mixing. The core rank is r=8r = 8, distinct from the TCRE relation width R=16R = 16.

Parallel acoustic modulation

Let Uℓ,Vℓ∈RD×r\mathbf{U}_{\ell},\mathbf{V}_{\ell} \in\mathbb{R}^{D\times r} be learned write and read bases shared across conditions at position ℓ\ell. The operator in (5) is

Aℓ(c)=Diag⁡(1+γℓ(c))+UℓMℓ(c)VℓT.(25)\mathcal{A}_{\ell}(\mathbf{c}) = \operatorname{Diag}(\mathbf{1}+\boldsymbol{\gamma}_{\ell}(\mathbf{c})) + \mathbf{U}_{\ell}\mathbf{M}_{\ell}(\mathbf{c})\mathbf{V}_{\ell}^{\mathsf{T}}. \tag*{(25)}

The first term provides FiLM [13]; the second, referred to as the cross-channel low-rank (XLR) branch, reads the acoustic feature into an rr-dimensional space, mixes it with the generated core, and writes the result back. Both terms act in parallel on the same pre-modulation representation hℓ,τ\mathbf{h}_{\ell,\tau}. The pre-stack position uses M0≡0\mathbf{M}_0 \equiv\mathbf{0}; the cross-channel term is applied before each encoder block.

Writing uℓ,i\mathbf{u}_{\ell,i} and vℓ,j\mathbf{v}_{\ell,j} for columns of the two bases makes the conditional channel combinations explicit:

UℓMℓ(c)VℓT=∑i,j=1r[Mℓ(c)]ijuℓ,ivℓ,jT.(26)\mathbf{U}_{\ell}\mathbf{M}_{\ell}(\mathbf{c})\mathbf{V}_{\ell}^{\mathsf{T}} = \sum_{i,j=1}^{r}[\mathbf{M}_{\ell}(\mathbf{c})]_{ij} \mathbf{u}_{\ell,i}\mathbf{v}_{\ell,j}^{\mathsf{T}}. \tag*{(26)}

The coefficients of these read/write combinations vary with the target condition. The additional matrix has rank at most rr and costs O(Dr+r2)\mathcal{O}(Dr+r^2) to apply per token. Each block adds 2Dr+(dh+1)r22Dr+(d_h+1)r^2 parameters for the bases and core head. Setting Mℓ=0\mathbf{M}_{\ell}=0 yields diagonal FiLM modulation.

The affine and core output heads are zero initialized, giving an identity transform at the start of fresh training. The read/write bases are orthogonally initialized. Condition construction, parameter generation, and the low-rank branch use FP32 arithmetic. The resulting acoustic updates are converted to match the numerical precision of the backbone’s hidden representations. For Qwen3-ASR-0.6B (D=896D = 896, L=18L = 18), the cross-channel branches add 296,064 parameters. Complete SSC, including TCRE, RSC pooling, and CLRM, contains 1,464,312 trainable conditioning parameters.

Training and Target-Conditioned Inference

The supervised objective uses only the requested target’s transcript. For KqK_q output tokens, it is

LSFT=−∑k=1Kqlog⁡pθ(yq,k∣yq,<k,X,cˉq).(27)\mathcal{L}_{\mathrm{SFT}} = -\sum_{k=1}^{K_q}\log p_{\theta}(y_{q,k}\mid y_{q,<k},\mathbf{X},\bar{\mathbf{c}}_q). \tag*{(27)}

All SSC modules, including the reference pooling path, learn through this objective. Binding pairs contribute the two target-specific losses in (20).

After supervised fine-tuning, GSPO [14] optimizes target transcription error. For each target-conditioned input Cq\mathcal{C}_q, the sampling policy πold\pi_{\mathrm{old}} generates a group of GG candidates. Candidate ii receives reward

Ri=−Si+Di+Iimax⁡(Nqref,1)−λcapI[candidate i is truncated],(28)R_i=-\frac{S_i+D_i+I_i}{\max(N_q^{\mathrm{ref}},1)}-\lambda_{\mathrm{cap}}\mathbb{I}[\text{candidate }i\text{ is truncated}], \tag*{(28)}

where edit counts are computed after text normalization against the requested target’s window transcript, using characters for Chinese and words for English. Let A^i=(Ri−Rˉ)/(σR+ϵA)\hat{A}_i=(R_i-\bar{R})/(\sigma_R+\epsilon_A) denote the group-normalized advantage, where Rˉ\bar{R} and σR\sigma_R are the group mean and population standard deviation. With candidate length KiK_i, the sequence-level importance ratio and objective are

wi(θ)=(πθ(y^(i)∣Cq)πold(y^(i)∣Cq))1/Ki,(29)w_i(\theta)=\left(\frac{\pi_\theta(\hat{y}^{(i)}\mid C_q)}{\pi_{\mathrm{old}}(\hat{y}^{(i)}\mid C_q)}\right)^{1/K_i}, \tag*{(29)}
LGSPO=−1G∑i=1Gmin⁡(wiA^i,clip⁡(wi,1−ϵ−,1+ϵ+)A^i)+λCELCE.(30)\mathcal{L}_{\mathrm{GSPO}}=-\frac{1}{G}\sum_{i=1}^{G}\min\left(w_i\hat{A}_i,\operatorname{clip}(w_i,1-\epsilon_-,1+\epsilon_+)\hat{A}_i\right)+\lambda_{\mathrm{CE}}\mathcal{L}_{\mathrm{CE}}. \tag*{(30)}

Here LCE\mathcal{L}_{\mathrm{CE}} is the token-normalized supervised loss across the logical minibatch. The text-system recipe uses G=8G=8, λcap=1\lambda_{\mathrm{cap}}=1, ϵA=10−4\epsilon_A=10^{-4}, ϵ−=3×10−4\epsilon_-=3\times10^{-4}, ϵ+=4×10−4\epsilon_+=4\times10^{-4}, and λCE=0.01\lambda_{\mathrm{CE}}=0.01. Target identities and activity conditions accompany each candidate during sampling and optimization.

At inference, a long session is divided into windows. For each requested track, RSC obtains session evidence, TCRE constructs its local condition, and CLRM modulates the encoder used to decode that target. Window outputs are assembled using the session speaker identities. A timestamp-output variant represents yqy_q as sentences with window-relative onset and offset times, with the same conditioning modules.

Experiments

Experimental Setup

Data and speaker priors

Table I summarizes the data. Data preparation builds on mt-asr-data-prep.3 Real meeting training uses the training partitions of AliMeeting [40], AISHELL-4 [41], AMI [42], and NOTSOFAR-1 [43], totaling approximately 331 h of single-channel recordings. Qwen3-ASR and FireRedAudio additionally use a 10,000-h TTS source pool to construct online 30-s mixtures with two to six speakers and 0–60% overlap, including room-response augmentation. Real and simulated target views are sampled at a 2:1 ratio. With RSC, the sampling weights for real, ordinary simulated, and paired-binding views are ρ=(20,9,1)\rho=(20,9,1). Reference selection uses Kref=4K_{\mathrm{ref}}=4, Tmin⁡=2T_{\min}=2 s, and Tmax⁡=8T_{\max}=8 s, with a 0.5-s exclusion margin, 0.2-s boundary trimming, and a 0.5-s minimum segment duration.

The four in-domain test sets contain 20 AliMeeting recordings, 20 AISHELL-4 recordings, 16 AMI-SDM recordings, and 160 NOTSOFAR-1 device recordings from 80 meetings. All systems use far-field single-channel audio: channel 0 for AliMeeting and AISHELL-4, SDM with the Full-corpus-ASR partition for AMI, and the single-channel device tracks for NOTSOFAR-1. Out-of-domain evaluation uses six

DatasetPartition / channelLang.Hours
Real meeting training data
AliMeetingTrain, far-field ch. 0zh104.75
AISHELL-4Train, far-field ch. 0zh107.50
AMI-SDMTrain, SDMen≈81
NOTSOFAR-1Train 1+2, SCen≈38
Auxiliary augmentation
TTS source poolOnline mixtureszh/en10k
In-domain evaluation
AliMeetingTest, far-field ch. 0zh10.00
AISHELL-4Test, far-field ch. 0zh12.72
AMI-SDMEval, SDMen≈9
NOTSOFAR-1Eval-small, SCen16.00
Out-of-domain evaluation
MISP-MeetingEval, far-fieldzh3.00
AISHELL-5Eval2, far-fieldzh3.58
DiPCoEval, far-fielden≈2.6

Table 1. Training and evaluation data. SDM: single distant microphone; SC: single-channel device recording.

MISP-Meeting recordings [44], 18 AISHELL-5 Eval2 recordings [45], and five DiPCo Eval recordings [46], with no training or adaptation on these evaluation sets.

Mechanism ablations primarily use oracle diarization to isolate the conditioning design from input diarization errors. Unless otherwise specified, predicted priors are obtained by Diarizen [47] with a W2V-BERT2 frontend, followed by sequence-to-sequence (S2S) refinement [48]. RSC selects its inference reference segments from this input prior. At a 0.25-s collar, input DERs for AliMeeting, AISHELL-4, AMI-SDM, and NOTSOFAR-1 are 3.62/4.89/8.65/5.10%.

Models and optimization

Our implementation extends TS-ASR-Whisper.4 Our backbones are Qwen3-ASR-0.6B [49] and the 9B FireRedAudio model [50]. Unless stated otherwise, the conditioning modules use the widths introduced in Section III (d=de=dh=32d=d_e=d_h=32, R=16R=16, r=8r=8). The reference path shares the first six encoder blocks with the recognition path. For both Qwen3-ASR and FireRedAudio, we jointly train the acoustic encoder, audio projector, and conditioning modules, with rank-8 attention LoRA [39] in the decoder (scaling 16, dropout 0.05). SFT uses eight GPUs, BF16 backbone computation, and FP32 conditioning modules. The base learning rate is 4×10−54\times10^{-5}; conditioning parameters use a tenfold multiplier. Qwen3-ASR-0.6B uses a 15,000-step cosine schedule with 500 warmup steps; the activity-only variants stop after 10,000 steps. FireRedAudio uses 30,000 steps and 1,000 warmup steps. For SFT and GSPO, we select checkpoints by minimizing the equally weighted mean of autoregressive decoding error rates on the AliMeeting and AMI development sets.

Metrics and comparison protocol

We report concatenated minimum-permutation error rate (cpER), using characters for Chinese and words for English. After optimal speaker permutation within each session, corpus-level scores are

cpER⁡=100∑m(Sm+Dm+Im)∑mNm.(31)\operatorname{cpER}=100\frac{\sum_m(S_m+D_m+I_m)}{\sum_m N_m}. \tag*{(31)}
Cond. ccOperatorPar. (M)AliMeetingAISHELL-4AMI-SDMNOTSOFAR-1All1 spk2 spk≥\ge3 spkNo-solo target
STNOFDDT0.1417.9213.2018.3521.36–––––
STNOFiLM1.1317.6813.2318.1220.901.140.102.210.730.13
c(≤2)c^{(\le2)}FiLM1.1318.0313.0718.3120.970.622.33−0.150.190.97
c(≤3)c^{(\le3)}FiLM1.1317.1812.9518.1520.472.845.701.033.422.65
c(≤3)c^{(\le3)}FiLM64_{64}2.2317.7613.4018.1420.680.920.030.191.97_3.47_
c(≤3)c^{(\le3)}XLR8_80.3117.57_12.86_17.9520.562.60_5.36_1.672.831.25
c(≤3)c^{(\le3)}CLRM8_81.43_16.96_12.92_17.81__20.27_3.905.33_2.48__4.49_2.66
c(≤3)(E)c^{(\le3)}(\mathcal{E})CLRM8_81.4616.4712.8617.1919.676.232.383.447.7612.49

Table 2. Component ablations on Qwen3-ASR-0.6B with oracle diarization. c(≤K)c^{(\le K)} uses activity relations through order KK, and E\mathcal{E} adds RSC identities. Par. gives conditioning parameters in millions. RER is relative to STNO-FDDT. The 1/2/≥\ge3-spk columns group windows by peak reference concurrency. Bold and underline mark the best and second-best values before rounding.

Here SmS_m, DmD_m, ImI_m, and NmN_m denote substitutions, deletions, insertions, and reference units in session mm. Relative error reduction (RER) is 100(ebase−enew)/ebase100(e_{\mathrm{base}}-e_{\mathrm{new}})/e_{\mathrm{base}}. Multi-dataset means weight each dataset equally. RSC interventions use fixed-target CER/WER, keeping the requested speaker assignment.

For timestamp outputs, MeetEval [51] computes time-constrained cpER (tcpER) with a 5-s collar using character-based pseudo-word timings. Diarization error rate (DER) uses a 0.25-s collar, includes overlapping speech, and pools errors over scored speaker time.

Paired comparisons share reference transcripts and text normalization, and complete-system comparisons use a common normalization. For corpus-level comparisons, 95% confidence intervals use 10,000 paired meeting-bootstrap samples [52], with NOTSOFAR-1 device pairs resampled together.

TCRE: Preserving Multi-Speaker Activity Relations

The configurations in Table II progress from four-state conditioning to TCRE, then to TCRE–CLRM, and finally to the full RSC recipe. We first vary the activity representation while fixing the FiLM operator and generator width. STNO-FiLM encodes the four state trajectories with causal convolutions and concatenates their temporal features with the state probabilities before FiLM parameter generation. It provides a 1.14% mean RER over STNO-FDDT. Second-order TCRE alone gives 0.62%, whereas adding the third-order term raises the reduction to 2.84%, with cpER 17.18/12.95/18.15/20.47% across the four sets. Thus, track-level conditioning is most effective when it includes distinct interferer-pair relations. Figure 5 shows that third-order TCRE improves over STNO-FiLM at a similar conditioning budget of about 1.13M parameters.

Figure 5. Parameter efficiency of the activity-only ablations in Table II. Points show conditioning-module size and mean RER over STNO-FDDT across the four test sets.

We examine this relation using the original 30-s windows, grouped by peak reference concurrency. The peak-1/2/≥\ge3 groups contain 1,221/2,022/2,660 windows. Errors are attributed after full-session speaker matching and edit alignment: reference-unit midpoints locate substitutions and deletions, and decoder windows locate insertions. The third-order model improves over the second-order model by 3.22% in the peak-≥\ge3 group (95% CI [1.85, 4.50]%). Peak concurrency of at least three supplies simultaneous non-target activity for the distinct-interferer-pair term in (12). The third-order model

also improves over the second-order model by 1.14% in the peak-2 group. In peak-1 windows, the third- and second-order models achieve 5.70% and 2.33% RER over STNO-FDDT, respectively (Table II).

CLRM: Conditional Cross-Channel Modulation

With the third-order TCRE condition fixed, we next vary how it controls the acoustic encoder. Rank-8 CLRM improves the mean RER over STNO-FDDT from FiLM’s 2.84% to 3.90% (Table II). Increasing the FiLM generator width from 32 to 64 instead gives 0.92% RER with 2.23M parameters, compared with CLRM’s 1.43M. The cross-channel-only variant is compact (0.31M parameters) and gives 2.60% RER; combining it with the diagonal branch produces the strongest activity-only result. These comparisons support combining diagonal adaptation with low-rank channel mixing.

We further intervene on the core of a frozen Qwen3-ASR-0.6B TCRE–CLRM checkpoint (Figure 6). Setting the core to zero increases meeting-macro cpER by 2.09 pp on AliMeeting-dev (95% CI [0.80, 3.51]) and 0.53 pp on AMI-SDM-dev ([0.19, 1.08]). Replacing it with a condition-independent calibration mean also increases error; with energy matching, the increases are 2.22 and 0.30 pp, respectively. The trained model therefore uses the varying core, beyond retaining a fixed cross-channel projection. These interventions support the role of the condition-dependent core described in Section III-D.

Figure 6. Recognition results from frozen-CLRM-core interventions. Values are changes in meeting-macro cpER from the intact model on eight AliMeeting-dev and four AMI-SDM-dev recordings, with 95% paired meeting-bootstrap CIs. The shared projections and all other weights remain fixed. The mean-core interventions replace the varying core with its calibration mean, optionally matching its energy.

RSC: Anchoring the Requested Speaker

The full RSC recipe adds session identities and paired target binding to TCRE–CLRM, yielding complete SSC. It achieves the best corpus scores in Table II, with cpERs of 16.47/12.86/17.19/19.67%, 1.46M conditioning parameters, and a 6.23% mean RER over STNO-FDDT. Complete SSC also achieves 3.44% RER over STNO-FDDT in the peak-2 group and 7.76% in the peak-≥3\ge3 group. The largest gain from the full RSC recipe occurs when the target has no solo speech, i.e., every target frame overlaps with another speaker. Relative to activity-only TCRE–CLRM, the full RSC recipe reduces error in this group by 10.09% (95% CI [7.58, 12.09]%); the reduction is 0.86% when more than half of the target’s speech is solo. Relative to STNO-FDDT, complete SSC reaches 12.49% RER in this fully overlapped group. The full RSC recipe is especially effective when local activity poorly distinguishes targets.

Table III measures the inference-time contribution of session identity at a fixed FireRedAudio timestamp checkpoint after GSPO (Section IV-F). With audio, target requests, reference candidates, and weights fixed, we supply or remove RSC identities under the same local activity. Starting from oracle tracks, we apply target misses, false alarms, and local track swaps to 0.5-s activity blocks at a 10% rate (seed 42). On clean activity, RSC reduces fixed-target CER from 12.60 to 12.14% on AliMeeting and WER from 14.23 to 13.59% on AMI-SDM. With target misses, the relative reductions increase to 17.81% and 14.51%; with track swaps, they reach 17.84% and 13.06%. Session identity therefore supports target recognition when local activity loses or exchanges target evidence. False-alarm interventions yield small increases of 0.02/0.36 pp on AliMeeting/AMI-SDM, indicating that the benefit centers on missing and exchanged target activity.

Activity conditionAliMeeting (CER)AMI-SDM (WER)
w/ow/RERw/ow/RER
Clean activity12.6012.143.6714.2313.594.50
Target miss (10%)18.9215.5517.8118.9816.2214.51
Target-track swap (10%)18.0114.8017.8418.5016.0913.06
Target false alarm (10%)16.1416.16−0.1216.8917.26−2.16
AMI example. ES2004d / ME0015, 1650–1680 s
Referenceseems fine except voice recognition everything is yeah
w/o RSCseems fine except for iso admission everything looks yeah
w/ RSCseems fine except voice recognition everything is yeah

Table 3. RSC under activity interventions. Fixed-target CER/WER (%) without (w/o) and with (w/) RSC, and relative error reduction (RER). Bold marks lower error. The example uses identical intervened activity at the same checkpoint.

PriorSetcpER SSC + GSPOcpER SSCΔ\DeltatcpER 5 s SSC + GSPO
OracleAliMeeting12.5612.11−0.45†12.65 12.18
AISHELL-410.6210.37−0.25†10.64 10.38
AMI-SDM14.0113.60−0.41†14.28 13.85
NOTSOFAR-115.7414.82−0.92†15.95 15.01
PredictedAliMeeting14.0513.59−0.46†14.35 13.84
AISHELL-414.0013.83−0.17†14.36 14.17
AMI-SDM16.7916.07−0.71†17.60 16.82
NOTSOFAR-121.1720.34−0.83†21.84 20.94
Mean RER (%)3.43.6

Table 4. Timestamp prediction with SSC and SSC + GSPO (%). Δ\Delta: (SSC + GSPO) − SSC in cpER percentage points; †: 95% paired meeting-bootstrap confidence interval below zero.

The example in Table III illustrates how the same intervened activity can produce different target transcripts. In a 30-s AMI window, RSC preserves “voice recognition” and the complete reference phrase, while the activity-only condition yields “for iso admission” and an additional word substitution. The supplied activity is identical in both decodes and contains a 0.43-s target miss. This example shows how session identity changes recognition under the same input activity.

Transfer to FireRedAudio and GSPO Post-Training

On FireRedAudio, we compare FDDT, SSC, and SSC + GSPO (Figure 7). Under the shared 30,000-step SFT recipe, SSC achieves a 2.80% mean relative cpER reduction over FDDT across the two priors and four datasets. The SFT-to-GSPO comparison starts from the same SSC checkpoint and uses the target-error reward in (28). GSPO brings a further 3.67% mean RER; the combined reduction over FDDT is 6.34%. On NOTSOFAR-1, these reductions reach 13.8% with oracle priors and 9.2% with predicted priors.

Figure 7. FireRedAudio progression from FDDT to SSC and SSC + GSPO on the four datasets (a)–(d), with oracle priors and predicted S2S-refined priors. Endpoint labels give the relative cpER reduction over FDDT.

We group meetings into 0–10%, 10–30%, and > 30% overlap. With predicted priors, SSC + GSPO yields lower cpER than SSC in all 11 nonempty dataset–interval groups. Figure 8 shows the reductions relative to FDDT, with 5, 39, and 36 NOTSOFAR-1 meetings in the three intervals, respectively. These session-level comparisons complement the local-concurrency and fully overlapped-target analyses.

Figure 8. FireRedAudio results by full-session overlap ratio with predicted S2S refinement priors. (a) NOTSOFAR-1 absolute cpER reductions of SSC and SSC + GSPO from FDDT (pp), with meeting counts nn. (b) SSC + GSPO relative error reduction over FDDT for meetings with at least three meetings. Hatched cells have fewer than three meetings, and dashes mark empty bins.

The paired intervals in Figure 9 quantify the contribution of each stage. Five of eight SSC-versus-FDDT intervals lie entirely below zero. All eight GSPO-versus-SSC intervals lie below zero, showing consistent post-training gains across datasets and diarization priors.

Figure 9. Paired meeting-bootstrap changes in corpus-micro cpER for FireRedAudio: (a) SSC versus FDDT; (b) SSC + GSPO versus SSC. Squares and circles mark oracle and predicted priors. Open markers denote CIs spanning zero. Negative values favor the new system. Error bars show 95% CIs.

Sentence Timestamps and Joint Output Evaluation

The timestamp variant generates sentence onset/offset times and text from the same target-conditioned model. Table V compares the SFT checkpoint (SSC) with the checkpoint obtained from it through GSPO post-training (SSC + GSPO), using the same target-transcription-error reward. Across the eight dataset–prior comparisons, mean RER is 3.4% for cpER and 3.6% for tcpER. All eight CIs for the cpER differences

ModelAliMeeting cpERAliMeeting tcpERAliMeeting DERAISHELL-4 cpERAISHELL-4 tcpERAISHELL-4 DERAMI-SDM cpERAMI-SDM tcpERAMI-SDM DERNOTSOFAR-1 cpERNOTSOFAR-1 tcpERNOTSOFAR-1 DERMacro cpERMacro tcpERMacro DER
DiCoW(zh/en)20.3220.684.3419.4019.995.1016.3117.038.8720.4621.015.5819.1219.685.97
MOSS-TD15.2015.396.2113.8014.064.2721.8422.7715.0227.3228.2411.6119.5420.129.28
Qwen3.8-Omni-Flash21.7423.0411.3315.5415.846.0829.3232.2534.0842.6044.2227.0127.3028.8419.62
Proposed (text)13.33––13.67––15.61––19.81––15.60––
Proposed (timestamp)13.5713.833.9113.8414.195.0216.0716.829.4820.3420.945.5215.9616.445.98
DiCoW(zh/en)23.3324.085.3745.6846.1716.2029.2431.2522.0232.7533.8314.53
MOSS-TD17.6217.964.8230.9131.2415.3527.6028.8022.5925.3726.0014.26
Qwen3.8-Omni-Flash21.5121.9310.7635.2635.8021.13XXX–––
Proposed (text)14.04––28.77––26.73––23.18––
Proposed (timestamp)14.0814.694.7428.8429.1115.4928.2130.1222.0423.7124.6414.09

Table V. System Comparison in Our Evaluation on In-Domain and Out-of-Domain Test Sets (%, Lower Is Better). (a) In-Domain. (b) Out-of-Domain. Notes: Proposed is FireRedAudio + SSC + GSPO. Macro is the equal-weight mean over all datasets within each panel. Bold and underline mark the best and second-best per column. “–” marks an unavailable metric, and X marks Qwen3.8-Omni-Flash DiPCo scores left unavailable by one failed API recording. DiCoW and Proposed use S2S refinement priors, whereas MOSS-TD and Qwen3.8-Omni-Flash use no external prior. On MISP-Meeting, target-speaker systems use annotated UEM and MOSS-TD uses the full recording.

(SSC + GSPO minus SSC) lie below zero. With predicted priors, AMI-SDM tcpER falls from 17.60 to 16.82% and NOTSOFAR-1 cpER from 21.17 to 20.34%.

In the common-protocol system comparison (Table V), the timestamp model incurs an equal-dataset mean cpER cost of 0.43 pp relative to the text model across seven sets. Its tcpER ranks first among locally evaluated systems on five sets and second on AISHELL-4 and DiPCo. Output DER is 3.91% on AliMeeting and 9.48% on AMI-SDM.

Complete Systems and Generalization to Unseen Domains

Table V compares the complete FireRedAudio + SSC + GSPO systems with the Chinese–English DiCoW baseline [7], MOSS-TD [4], and Qwen3.8-Omni-Flash [53] across the seven test sets.

The proposed text system obtains the lowest cpER among locally evaluated systems on each of the seven sets. Across the four in-domain sets, its mean RER is 17.9% over DiCoW and 17.3% over MOSS-TD. Gains over DiCoW are largest on AliMeeting and AISHELL-4 (34.4% and 29.5% RER); gains over MOSS-TD are largest on AMI-SDM and NOTSOFAR-1 (28.6% and 27.5%). The AISHELL-4 difference from MOSS-TD is smaller, at 0.13 pp.

On the three unseen domains, text-system cpER is 14.04/28.77/26.73% for MISP-Meeting/AISHELL-5/DiPCo, compared with DiCoW’s 23.33/45.68/29.24% and MOSS-TD’s 17.62/30.91/27.60%. These gains extend to meeting, in-car, and dinner-party recordings from new corpora.

Conclusion

SSC connects local activity structure and session-level identity through a target condition that controls acoustic feature combinations. TCRE preserves per-speaker histories and target-centric activity relations. RSC grounds these relations in acoustic identity derived from session audio, and CLRM maps the joint condition to diagonal and rank-8 cross-channel modulation for target-speaker recognition. Complete SSC yields 6.23% mean RER over STNO-FDDT on Qwen3-ASR-0.6B with 1.46M conditioning parameters. Transfer to FireRedAudio, followed by target-error GSPO, produces a complete system with 17.9% and 17.3% mean in-domain RER over DiCoW and MOSS-TD, respectively, and lower cpER than DiCoW and MOSS-TD on three unseen domains. The timestamp variant retains competitive recognition while providing sentence-level text and timing in the same output.

Acknowledgment

Generative AI use. OpenAI Codex (GPT-6) assisted with translating, drafting, and polishing author-provided material. Claude Opus 5.5, accessed through CodeBuddy, assisted with manuscript review and the preparation of figures and tables. The authors developed all scientific content, reviewed the manuscript sentence by sentence, checked and corrected the mathematical derivations and their presentation, and verified numerical values. They manually adjusted the AI-drafted visual elements and finalized the manuscript, figures, and tables, taking full responsibility for the work.

References

  1. [1]T. Yoshioka, I. Abramovski, C. Akscoylar, Z. Chen, M. David, D. Dimitriadis, Y. Gong, I. Gurvich, X. Huang, Y. Huang, A. Hurvitz, L. Jiang, S. Koubi, E. Krupka, I. Leichter, C. Liu, P. Parthasarathy, A. Vinnikov, L. Wu, K. Xiao, W. Xiong, H. Wang, Z. Wang, J. Zhang, Y. Zhao, and T. Zhou, “Advances in online audio-visual meeting transcription,” in Proc. IEEE Autom. Speech Recognit. Understanding Workshop (ASRU), 2019, pp. 276–283.arxiv.org/abs/1912.04979
  2. [2]T. von Neumann, C. Boeddeker, T. Cord-Landwehr, M. Delcroix, and R. Haeb-Umbach, “Meeting recognition with continuous speech separation and transcription-supported diarization,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. Workshops (ICASSPW), 2024, pp. 775–779. [Online]. Available: https://doi.org/10.1109/icasspw62645.2024.10625894
  3. [3]N. Kanda, Y. Gaur, X. Wang, Z. Meng, Z. Chen, T. Zhou, and T. Yoshioka, “Joint speaker counting, speech recognition, and speaker identification for overlapped speech of any number of speakers,” in Proc. Interspeech, 2020, pp. 36–40. [Online]. Available: https://www.isca-archive.org/interspeech_2020/kanda20_interspeech.htmlarxiv.org/abs/2006.10930
  4. [4]MOSI.AI, D. Yu, Z. Lin, H. Chen, C. Yang, Y. Zhang, J. Chen, K. Chen, L. Fan, Y. Jiang, J. Zhu, M. Li, W. Wang, Y. Wang, Z. Xu, B. Jiang, Y. Gong, Y. Zhang, W. Zhang, S. Wang, Z. Wang, Z. Fei, Q. Cheng, S. Li, and X. Qiu, “MOSS Transcribe Diarize technical report,” 2026, arXiv:2601.01554, version 7. [Online]. Available: https://arxiv.org/abs/2601.01554v7
  5. [5]M. Huo, Y. Shao, and Y. Zhang, “TagSpeech: End-to-end multi-speaker ASR and diarization with fine-grained temporal grounding,” in Proc. Annu. Meeting Assoc. Comput. Linguistics (ACL), 2026, pp. 41 847–41 862. [Online]. Available: https://aclanthology.org/2026.acl-long.1938/
  6. [6]A. Polok, D. Klement, M. Wiesner, S. Khudanpur, J. Černocký, and L. Burget, “Target speaker ASR with Whisper,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2025, pp. 1–5.arxiv.org/abs/2409.09543
  7. [7]A. Polok, D. Klement, M. Kocour, J. Han, F. Landini, B. Yusuf, M. Wiesner, S. Khudanpur, J. Černocký, and L. Burget, “DiCoW: Diarization-conditioned Whisper for target speaker automatic speech recognition,” Comput. Speech Lang., vol. 95, 2026, art. no. 101841.
  8. [8]A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeay, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. Int. Conf. Mach. Learn. (ICML), ser. Proc. Mach. Learn. Res., A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202, PMLR, 23–29 Jul 2023, pp. 28 492–28 518. [Online]. Available: https://proceedings.mlr.press/v202/radford23a.htmlarxiv.org/abs/2212.04356
  9. [9]A. Polok, S. Cornell, S. Udupa, H. Černocký, S. Watanabe, and L. Burget, “Grounding spoken LLMs in multi-speaker audio via diarization conditioning,” in Proc. Interspeech, 2026, pp. 300–305. [Online]. Available: https://www.isca-archive.org/interspeech_2026/polok26b_interspeech.htmlarxiv.org/abs/2606.18134
  10. [10]A. Polok, D. Klement, S. Cornell, M. Wiesner, J. Černocký, S. Khudanpur, and L. Burget, “SE-DiCoW: Self-enrolled diarization-conditioned Whisper,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2026, pp. 16712–16716. [Online]. Available: https://doi.org/10.1109/ICASSP55912.2026.11461785
  11. [11]Y. Hirano and S. Sakti, “SC-SOT: Conditioning the decoder on diarized speaker information for end-to-end overlapped speech recognition,” in Proc. Interspeech, 2025, pp. 4923–4927. [Online]. Available: https://www.isca-archive.org/interspeech_2025/hirano25_interspeech.htmlarxiv.org/abs/2506.12672
  12. [12]W. Wang, T. Park, I. Medennikov, J. Wang, K. Dhawan, H. Huang, N. Rao Kologuri, J. Balam, and B. Ginsburg, “Speaker targeting via self-speaker adaptation for multi-talker ASR,” in Proc. Interspeech, 2025, pp. 5498–5502. [Online]. Available: https://www.isca-archive.org/interspeech_2025/wang25_interspeech.htmlarxiv.org/abs/2506.22646
  13. [13]E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville, “FiLM: Visual reasoning with a general conditioning layer,” Proc. AAAI Conf. Artif. Intell., vol. 32, no. 1, pp. 3942–3951, 2018.arxiv.org/abs/1709.07871
  14. [14]C. Zheng, S. Liu, M. Li, X.-H. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, J. Zhou, and J. Lin, “Group sequence policy optimization,” 2025, arXiv:2507.18071. [Online]. Available: https://arxiv.org/abs/2507.18071
  15. [15]D. Yu, M. Kolbaek, Z.-H. Tan, and J. Jensen, “Permutation invariant training of deep models for speaker-independent multi-talker speech separation,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2017, pp. 241–245.
  16. [16]Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, “TF-GridNet: Making time-frequency domain models great again for monaural speaker separation,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2023, pp. 1–5.arxiv.org/abs/2209.03952
  17. [17]N. Kanda, Y. Gaur, X. Wang, Z. Meng, and T. Yoshioka, “Serialized output training for end-to-end overlapped speech recognition,” in Proc. Interspeech, 2020, pp. 2797–2801. [Online]. Available: https://doi.org/10.21437/Interspeech.2020-999
  18. [18]N. Kanda, J. Wu, Y. Wu, X. Xiao, Z. Meng, X. Wang, Y. Gaur, Z. Chen, J. Li, and T. Yoshioka, “Streaming multi-talker ASR with token-level serialized output training,” in Proc. Interspeech, 2022, pp. 3774–3778. [Online]. Available: https://doi.org/10.21437/Interspeech.2022-7
  19. [19]A. S. Subramanian, A. Das, N. Kanda, J. Li, X. Wang, and Y. Gong, “Improving practical aspects of end-to-end multi-talker speech recognition for online and offline scenarios,” in Proc. Interspeech, 2025, pp. 5508–5512.arxiv.org/abs/2506.14204
  20. [20]M. Shi, Z. Jin, Y. Xu, Y. Xu, S.-X. Zhang, K. Wei, Y. Shao, C. Zhang, and D. Yu, “Advancing multi-talker ASR performance with large language models,” in Proc. IEEE Spoken Lang. Technol. Workshop (SLT), 2024, pp. 14–21.arxiv.org/abs/2408.17431
  21. [21]Z. Peng, J. Yu, Y. Chang, Z. Wang, L. Dong, Y. Hao, Y. Tu, C. Yang, W. Wang, S. Xu, Y. Sun, H. Bao, W. Xu, Y. Zhu, Z. Wang, T. Song, Y. Xia, Z. Chi, S. Huang, L. Wang, C. Ding, S. Wang, X. Chen, and F. Wei, “VIBEVOICE-ASR technical report,” 2026, arXiv:2601.18184. [Online]. Available: https://arxiv.org/abs/2601.18184
  22. [22]B. Mu, X. Shi, X. Wang, Z. Guo, T. He, X. Cheng, Y. Xi, J. Xu, and L. Xie, “DiaScriber: A speech LLM for joint diarization and transcription in multi-speaker scenarios,” 2026, arXiv:2608.22796. [Online]. Available: https://arxiv.org/abs/2608.22796
  23. [23]Y. Fujita, N. Kanda, S. Horiguchi, K. Nagamatsu, and S. Watanabe, “End-to-end neural speaker diarization with permutation-free objectives,” in Proc. Interspeech, 2019, pp. 4300–4304.arxiv.org/abs/1909.05952
  24. [24]S. Horiguchi, Y. Fujita, S. Watanabe, Y. Xue, and K. Nagamatsu, “End-to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors,” in Proc. Interspeech, 2020, pp. 269–273.arxiv.org/abs/2005.09921
  25. [25]I. Medennikov, M. Korenovsky, T. Prisyach, Y. Khokhlov, M. Kornevskaya, I. Sorokin, T. Timofeeva, A. Mitrofanov, A. Andrusenko, I. Podluzhny, A. Laptev, and A. Romanenko, “Target-speaker voice activity detection: A novel approach for multi-speaker diarization in a dinner party scenario,” in Proc. Interspeech, 2020, pp. 274–278.DOI
  26. [26]L. Li, M. Cheng, W. Zhu, Y. Wang, J. Liu, and M. Li, “DM-ASR: Diarization-aware multi-speaker ASR with large language models,” 2026, arXiv:2604.22467. [Online]. Available: https://arxiv.org/abs/2604.22467
  27. [27]D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust DNN embeddings for speaker recognition,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2018, pp. 5329–5333.DOI
  28. [28]B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” in Proc. Interspeech, 2020, pp. 3830–3834.DOI
  29. [29]K. Žmolíková, M. Delcroix, K. Kinoshita, T. Ochiai, T. Nakatani, L. Burget, and J. Černocký, “SpeakerBeam: Speaker aware neural network for target speaker extraction in speech mixtures,” IEEE J. Sel. Topics Signal Process., vol. 13, no. 4, pp. 800–814, 2019. [Online]. Available: https://doi.org/10.1109/JSTSP.2019.2922820
  30. [30]Q. Wang, H. Muckenhirn, K. Wilson, P. Sridhar, Z. Wu, J. R. Hershey, R. A. Saurous, R. J. Weiss, Y. Jia, and I. L. Moreno, “VoiceFilter: Targeted voice separation by speaker-conditioned spectrogram masking,” in Proc. Interspeech, 2019, pp. 2728–2732.arxiv.org/abs/1810.04826
  31. [31]N. Kanda, S. Horiguchi, Y. Fujita, X. Yue, K. Nagamatsu, and S. Watanabe, “Simultaneous speech recognition and speaker diarization for monaural dialogue recordings with target-speaker acoustic models,” in Proc. IEEE Autom. Speech Recognit. Understanding Workshop (ASRU), 2019, pp. 31–38.arxiv.org/abs/1909.08103
  32. [32]P. Guo, X. Chang, H. Lv, S. Watanabe, and L. Xie, “SO-Whisper: Speaker-querying based Whisper model for target-speaker ASR,” IEEE Trans. Audio, Speech, Lang. Process., vol. 33, pp. 175–185, 2025. [Online]. Available: https://doi.org/10.1109/taslp.2024.3513835
  33. [33]M. Zaheer, S. Kottur, S. Ravibankh, B. Poczos, R. Salakhutdinov, and A. J. Smola, “Deep Sets,” in Advances in Neural Information Processing Systems, vol. 30, 2017. [Online]. Available: https://proceedings.neurips.cc/paper/2017/hash/f22e4747daa1daa27e3638d640ff442fe-Abstract.html
  34. [34]A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap, “A simple neural network module for relational reasoning,” in Advances in Neural Information Processing Systems, vol. 30, 2017. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/e6acf4b0f69f6f6e60e9a815938aa1ff-Paper.pdfarxiv.org/abs/1706.01427
  35. [35]K. Okabe, T. Koshinaka, and K. Shinoda, “Attention statistics pooling for deep speaker embedding,” in Proc. Interspeech, 2018, pp. 2252–2256.DOI
  36. [36]D. Ha, A. Dai, and Q. V. Le, “HyperNetworks,” in Proc. Int. Conf. on Learning Representations (ICLR), 2017. [Online]. Available: https://arxiv.org/abs/1609.09106
  37. [37]Y. Zhao, J. Li, and Y. Gong, “Low-rank plus diagonal adaptation for deep neural networks,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2016, pp. 5005–5009. [Online]. Available: https://doi.org/10.1109/icassp.2016.7472630
  38. [38]N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attarian, and S. Gelly, “Parameter-efficient transfer learning for NLP,” in Proc. Int. Conf. Mach. Learn. (ICML), ser. Proc. Mach. Learn. Res., K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 2790–2799. [Online]. Available: https://proceedings.mlr.press/v97/houlsby19a.htmlarxiv.org/abs/1902.00751
  39. [39]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Proc. Int. Conf. on Learning Representations (ICLR), 2022. [Online]. Available: https://arxiv.org/abs/2106.09685
  40. [40]F. Yu, S. Zhang, Y. Fu, L. Xie, S. Zheng, Z. Du, W. Huang, P. Guo, Z. Yan, B. Ma, X. Xu, and H. Bu, “M2MeT: The ICASSP 2022 multi-channel multi-party meeting transcription challenge,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2022, pp. 6167–6171. [Online]. Available: https://doi.org/10.1109/ICASSP43922.2022.9746465
  41. [41]Y. Fu, L. Cheng, S. Lv, Y. Jv, Y. Kong, Z. Chen, Y. Hu, L. Xie, J. Wu, H. Bu, X. Xu, J. Du, and J. Chen, “AISHELL-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario,” in Proc. Interspeech, 2021, pp. 3665–3669.arxiv.org/abs/2104.03603
  42. [42]J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, I. McCowan, W. Post, D. Reidsma, and P. Wellner, “The AMI meeting corpus: A pre-announcement,” in Machine Learning for Multimodal Interaction. Springer Berlin Heidelberg, 2006, pp. 28–39. [Online]. Available: https://doi.org/10.1007/11677482_3
  43. [43]A. Vinnikov, A. Ivry, A. Hurvitz, I. Abramovski, S. Koubi, I. Gurvich, S. Peer, X. Xiao, B. M. Elizalde, N. Kanda, X. Wang, S. Shaer, S. Yagev, Y. Asher, S. Sivanskaran, Y. Gong, M. Tang, H. Wang, and E. Krupka, “NOTASOFAR-1 challenge: New datasets, baseline, and tasks for distant meeting transcription,” in Proc. Interspeech, 2024, pp. 5003–5007.arxiv.org/abs/2401.08887
  44. [44]M. Gao, S. Wu, H. Chen, J. Du, C.-H. Lee, S. Watanabe, J. Chen, S. M. Siniscalchi, and O. Scharenborg, “The multimodal information based speech processing (MISP) 2025 challenge: Audio-visual diarization and recognition,” in Proc. Interspeech, 2025, pp. 188–1892.
  45. [45]Y. Dai, H. Wang, X. Li, Z. Zhang, S. Wang, L. Xie, X. Xu, H. Guo, S. Zhang, H. Bu, and W. Chen, “AISHELL-5: The first open-source in-car multi-channel multi-speaker speech dataset for automatic speech diarization and recognition,” in Proc. Interspeech, 2025, pp. 5493–5497.arxiv.org/abs/2505.23036
  46. [46]M. Van Segbroeck, A. Zaid, K. Kutsenko, C. Huerta, T. Nguyen, X. Luo, B. Hoffmeister, J. Trmal, M. Omologo, and R. Maas, “DiPCo—dinner party corpus,” in Proc. Interspeech, 2020, pp. 434–436. [Online]. Available: https://www.isca-archive.org/interspeech_2020/segbroeck20_interspeech.html
  47. [47]J. Han, F. Landini, J. Rohdin, A. Silnova, M. Diez, and L. Burget, “Leveraging self-supervised learning for speaker diarization,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2025, pp. 1–5. [Online]. Available: https://doi.org/10.1109/ICASSP49660.2025.10889475
  48. [48]R. Wang, J. Du, W. Zhu, and Y. Wang, “Sequence-to-sequence refinement for EEND-VC diarization with a shared encoder,” 2026, arXiv:2611.02709. [Online]. Available: https://arxiv.org/abs/2611.02709
  49. [49]X. Shi, X. Wang, Z. Guo, Y. Wang, P. Zhang, X. Zhang, Z. Guo, H. Hao, Y. Xi, B. Yang, J. Xu, J. Zhou, and J. Lin, “Owen3-ASR technical report,” 2026, arXiv:2601.21337. [Online]. Available: https://arxiv.org/abs/2601.21337
  50. [50]F. Shen, F. Xie, J. Li, K. Xie, L. Xie, X. Tang, X. Geng, Y. Jia, Y. Hu, Y. Han, Y. Wu, Z. Dai, J. Chen, K. Huang, M. Wei, and Y. Li, “FireRedAudio: A general-purpose audio language model with decoupled continuous representations for understanding and generation,” 2026, arXiv:2608.24168. [Online]. Available: https://arxiv.org/abs/2608.24168
  51. [51]T. von Neumann, C. Boeddeker, M. Delcroix, and R. Haeb-Umbach, “MeetEval: A toolkit for computation of word error rates for meeting transcription systems,” in Proc. Int. Workshop Speech Process. Everyday Environ. (CHiME), 2023, pp. 27–32.arxiv.org/abs/2307.11394
  52. [52]M. Bisani and H. Ney, “Bootstrap estimates for confidence intervals in ASR performance evaluation,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), vol. 1, Montreal, Canada, May 2004, pp. 409–412. [Online]. Available: https://doi.org/10.1109/ICASSP.2004.1326009
  53. [53]Qwen Team, “Qwen3.8-Omni: Towards native omni-modal agents,” 2026, arXiv:2609.25611. [Online]. Available: https://arxiv.org/abs/2609.25611

Paper details

vlnc:W_p2kkus3z
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Codex, Claude Opus 5.5

View statement
Looking for related papers…
Contents