WorksComputer Science
Low-Latency LLM-Based Streaming Multi-Talker ASR with Verifiable Rewards
Abstract
LLM-based architectures offer strong generalization and instruction-following capabilities for streaming automatic speech recognition (ASR), but multi-talker use introduces coupled transcription, attribution, and latency challenges. Our wearable setting requires real-time transcription of a conversation between a near-field device wearer and a far-field partner. Because partial hypotheses are rendered on persistent sides of a dialogue UI, speaker roles cannot be permuted and each speaker tag must precede its turn. These sparse tags are weakly optimized by supervised fine-tuning (SFT), while committing to a leading tag can delay the text that follows. We therefore post-train a streaming LLM-based ASR model with reinforcement learning with verifiable rewards (RLVR), combining a fixed-role Speaker-Attributed WER (SA-WER) grader with a turn-level onset-delay reward. On an eight-direction multi-talker cohort, SA-WER RL reduces SA-WER from for SFT to . Adding the delay objective reduces onset by (from to ) and by (from to ) relative to SA-WER-only RL, while achieving SA-WER. These results demonstrate that fixed-role recognition quality and streaming responsiveness can be improved jointly.
Introduction
Large language model (LLM)-based architectures are increasingly used for automatic speech recognition (ASR), motivated by their generalization, contextual modeling, and instruction-following capabilities [1, 2, 3, 4, 5]. Recent streaming speech LLMs have demonstrated competitive transcription and translation quality under real-time constraints [6, 7, 8]. Extending this paradigm to multi-talker ASR requires both recognizing overlapping or alternating speech and determining who spoke what. Prior streaming systems address these challenges with dedicated multi-channel or speech-unmixing architectures, including specialized Recurrent Neural Network Transducers (RNN-Ts) [9]. Meng et al. [10] demonstrates that speech LLM can understand overlapped human speech following versatile human instructions. Serialized Output Train-
ing (SOT) instead linearizes multiple speakers’ transcriptions into one sequence separated by speaker-change tokens [11], making it naturally compatible with autoregressive LLM decoders. Our wearable use case transcribes a real-time two-party conversation between the device wearer (near field) and a conversation partner (far field), while the UI deserializes the output stream into the two sides of a dialogue, as illustrated in Figure 1. Speaker identities must therefore be fixed semantic roles—<wearer> and <partner>—rather than labels that may be permuted during scoring. Moreover, each role tag must precede its turn so the UI can route partial words to the correct side as they arrive; a suffix tag would require buffering the complete turn before displaying it. Recent LLM-based multi-talker ASR combines neural beamforming with SOT supervision [12], but is not natively streaming and requires an additional streaming harness.

Figure 1. Wearable two-party streaming pipeline with fixed-role SOT, dialogue-side routing, and downstream bidirectional translation.
Supervised Fine-Tuning (SFT) can teach this SOT format, but token-level cross-entropy is inefficient for optimizing multi-talker recognition quality. It does not directly optimize sequence-level metrics such as word error rate (WER), and sparse speaker tags contribute little to the total loss. A model can therefore achieve low lexical loss while omitting the quieter speaker or assigning words to the wrong role. Reinforcement learning with verifiable rewards (RLVR) is a natural post-training alternative because ASR hypotheses admit automatic, quantitative scoring against references. Prior work has explored sequence-level reinforcement learning for ASR and reward-based post-training for speech models [13, 14]. We use group-relative reward normalization, following Group Relative Policy Optimization (GRPO) [15], to optimize multiple sampled transcripts per audio prompt.
These constraints motivate two rewards. Because cp-WER [16] can permute speakers to minimize error, we use fixed-role Speaker-Attributed WER (SA-WER) to score transcription and attribution. Since each turn must begin with a role tag, the model may delay all text while confirming the speaker; an onset-delay reward encourages earlier commitment without sacrificing recognition quality. Our contributions are: 1) Fixed-role quality alignment: We formulate an RLVR objective around a fixed-role SA-WER grader, directly optimizing transcription and speaker attribution for streaming SOT output. 2) Coverage-aware latency optimization: We introduce a bounded onset delay factor that rewards timely emission of aligned turns without allowing latency to dominate quality.
On our 2-speaker conversational evaluation sets, RLVR with SA-WER grader reduced the streaming SA-WER from 20.51% for SFT to 13.14%. Adding the delay objective reduces onset latency relative to SA-WER-only RL: decreases by 65%, from 0.918 s to 0.320 s, and by 44%, from 1.389 s to 0.780 s. The resulting model achieves 12.70% WER, demonstrating joint improvements in speaker-aware quality and streaming responsiveness.
Methodology
RLVR Framework
We initialize the policy from a supervised fine-tuned model and post-train it with reinforcement learning with verifiable rewards (RLVR). For each audio prompt, we sample multiple transcript rollouts and score them with the graders described below. Our objective is similar to GRPO [15], except that it directly weights each sampled transcript’s log-likelihood by its group-normalized advantage instead of using a clipped surrogate. We additionally apply full-distribution reverse KL regularization to the frozen SFT policy.
Fixed-Role Speaker-Attributed WER
Our SOT transcript begins each turn with a fixed-role tag for the wearer () or partner (). After normalization, each word is represented as . A joint edit-distance alignment compares the serialized hypothesis against separate wearer and partner reference streams without permitting role permutation. Besides insertion (), deletion (), and substitution (), it identifies , a correct word assigned to the wrong speaker, and , an error in both word and speaker. For reference words,
We also define the deletion rate to capture the overall deletion errors.
SA-WER Grader
The fixed-role alignment already includes attribution errors and deletion errors in SA-WER. To emphasize these product-critical failures during RL, our grader up-weights them through the combined error
The denominator keeps the combined quantity on the scale of an error rate. We convert it to a bounded quality reward using
A perfect transcript receives reward 1, while hypotheses with combined error at or above 1 receive reward 0. This sequence-level reward directly exposes lexical accuracy, fixed-role attribution, and transcription retention to RL optimization.
Turn-Level Onset-Delay Regularization
Our streaming decoder interleaves audio consumption and text generation. Each advancing audio marker, including empty-audio markers after stream exhaustion, increments the model-time clock by 80 ms. Predicted and reference turns are paired by aligning their fixed-role speaker-tag sequences and retaining only matching subsequences. Unmatched predicted or reference turns receive no latency credit and do not shift every subsequent pairing.
For a paired predicted turn and reference turn , let be the model-time at which the leading speaker tag is emitted and the oracle turn start. We define onset delay as
For an onset buffer and decay scale , the turn receives score
Emission within the buffer receives full credit, while excess delay decays exponentially. Negative delay is not penalized by (5), but is tracked separately as premature emission. Let be the set of matched turn pairs. The coverage-aware onset factor is
The denominator ensures that missing and hallucinated turns reduce the factor.
Finally, the onset factor acts as a bounded multiplicative regularizer on the quality reward:
Here, controls the latency-reward strength. The multiplier equals 1 at full onset credit and is bounded below by , making onset latency a secondary regularizer on the speaker-aware recognition objective.
Experiments
Experimental Setup
Base Model
We use a streaming mixture-of-experts (MoE) speech LLM initialized from a checkpoint pretrained for multilingual single-talker ASR. Its 32 transformer layers are all MoE layers, each containing 256 routed experts with top-8 routing. The model activates approximately 5.5B backbone parameters per token. The model consumes a 24-kHz waveform and represents each 1,920-sample (80 ms) audio block with one audio token. We continue SFT on multi-talker conversations while retaining the fixed-role SOT tags, and then apply RLVR. This setup isolates the effects of sequence-level quality and latency optimization from changes to the model architecture or acoustic front end.
Data
The multi-talker training corpus is generated through simulation. We sample two single-talker utterances from a multilingual ASR pool and randomly assign their speakers to the wearer (near-field) and partner (far-field) roles. To reproduce the role-dependent acoustics of the wearable microphone array, we convolve the utterances with corresponding near- and far-field room impulse responses. We then apply stochastic temporal offsets and overlap the two signals to form a simulated conversation. The simulator produces five-channel audio; we extract microphone channel 0 for both SFT and RLVR.
The evaluation corpus consists of real-world, scripted conversations recorded through the wearable device for live translation scenario. The corpus contains 1,181 five-channel recordings between strictly 2 speakers speaking different languages, across eight language pairs. They are German, Spanish, French, and Italian paired bidirectionally with English. The average duration per conversation is around 210 seconds.
Training Setup
SFT uses 64 H100 GPUs, a global batch size of 64, a maximum sequence length of 16,384, and a peak learning rate of with cosine decay and 20 warm-up steps. The SFT runs for 450 steps. RLVR initializes from the last checkpoint of the SFT job and also uses it as the frozen reverse-KL reference. RLVR uses 120 H100 GPUs, 512 rollouts per batch, a maximum sequence length of 4,096, the same learning rate, and up to 1,000. Each prompt produces eight samples at and top-. Group-normalized rewards optimize group-normalized policy-gradient objective with full-distribution reverse KL of weight 0.1; non-finite, effectively zero-variance, and truncated groups are removed. We set . Unless varied below, delay-controlled runs use a 700 ms onset buffer, and a 400 ms decay scale.
Evaluation Setup
All headline results use whole-conversation streaming recognition on the 1,181 recordings described above. The first channel is extracted for decoding, matching the training. Greedy decoding uses temperature 0. For the stochastic SFT study, we draw eight samples per recording using temperature 1.0, top-, and top-. Oracle@8 selects the minimum-SA-WER candidate for each recording and uses the same candidate for latency; it therefore measures latent model capability rather than deployable single-decode performance.
We report fixed-role SA-WER on untouched hypotheses using the official multi-talker scorer, together with wearer and partner WER where relevant. For latency, hypothesis and reference turns are paired through matching blocks of their fixed-role tag sequences. Onset delay is the speaker-tag emission time minus the oracle turn start time. We report pooled , , and over paired turns, the fraction with negative onset delay, and selected long-tail rates. Negative onset denotes premature emission rather than low latency. Missing, inserted, or misattributed turns remain unpaired, so latency percentiles are interpreted together with SA-WER and turn coverage.
Experiment Results
From SFT to SA-WER-RL
Table 1 decomposes the SFT and RL models’ performance into the error terms targeted by Eq. (2). SA-WER RL reduces overall error from 20.51% to 13.14%. More importantly, wearer/partner SAE falls from 3.2%/5.0% to 0.5%/0.3%, showing that RL suppresses fixed-role inversions. Wearer/partner deletion also decreases from 12.2%/14.4% to 9.5%/8.1%, which validates explicitly upweighting deletion errors.
Oracle@8 and pass@8 are complementary: oracle@8 reports corpus SA-WER after selecting the lowest-SA-WER candidate for each recording, whereas pass@8 is the probability that at least one of eight candidates meets a specified threshold. Sampling at does not improve the expected SFT output: candidate-corpus SA-WER rises to 24.96%. Nevertheless, oracle@8 reaches 14.61%, and Table 1 shows substantial pass@8 gains at the 5% and 10% thresholds. Thus SFT already assigns probability mass to better transcripts, but individual samples are worse on average and oracle selection is unavailable at inference. SA-WER RL converts this latent capability into greedy behavior, improving over greedy SFT by 7.37 absolute points (35.9% relative) and surpassing SFT oracle@8 by 1.47 points with one decode. However, when we measure the onset delay, its onset remains close to SFT (0.918 versus 0.906 s), confirming that this gain is only for the fixed-role quality objective and that latency requires an explicit reward.
| System | SA-WER (%) | SAE₀ (%) | SAE₁ (%) | D₀ (%) | D₁ (%) |
| SFT | 20.51 | 3.2 | 5.0 | 12.2 | 14.4 |
| SFT oracle@8 | 14.61 | – | – | – | – |
| SA-WER RL | 13.14 | 0.5 | 0.3 | 9.5 | 8.1 |
| SFT sampling statistic | Pass@1 (%) | Pass@8 (%) | |||
| Exact (0% SA-WER) | 0.00 | 0.00 | |||
| ≤ 5% SA-WER | 9.70 | 25.62 | |||
| ≤ 10% SA-WER | 33.55 | 58.09 |
Table 1. Quality decomposition. and are normalized by reference words for role ; Eq. (2) uses their reference-weighted SAE aggregate and . Speaker-level breakdowns were not retained for oracle@8.
Effect of Onset-Delay Regularization
Table 2 and Figure 2 separate recognition quality from latency for representative checkpoints. By Eq. (5), onset credit is constant for and decays exponentially beyond buffer with ms. The no-delay baseline has , so regardless of emission time. Under its observed s, the onset credit would be 1.00/0.27 for buffers of 1,300/400 ms. Smaller buffers therefore expose more of the baseline delay distribution to reward decay and provide stronger pressure for earlier emission. Quality remains near the SA-WER-only baseline across the stable runs. The 400 ms model obtains the lowest SA-WER and self/other deletion rates, while the diagnostic 1,000 ms run is degraded mainly by partner attribution error.
Figure 2. Pooled onset-delay percentiles for the same checkpoints. Smaller buffers monotonically improve and ; is less stable.
| Buffer (ms) | SA-WER (%) | SAE₀ (%) | SAE₁ (%) | D₀ (%) | D₁ (%) |
| None | 13.14 | 0.5 | 0.3 | 9.5 | 8.1 |
| 1,300 | 13.41 | 0.3 | 1.1 | 9.6 | 8.2 |
| 1,000 | 15.51 | 1.1 | 4.8 | 9.7 | 8.1 |
| 700 | 13.36 | 0.5 | 1.3 | 9.2 | 7.8 |
| 400 | 12.70 | 0.3 | 1.0 | 8.8 | 7.4 |
Table 2. Recognition-quality decomposition for the onset-buffer sweep ( except the no-delay baseline). The 1,000-ms run is diagnostic.
Figure 2 shows that reducing the buffer from 1,300 to 400 ms lowers pooled from 0.880 to 0.320 s and from 1.360 to 0.780 s. Relative to SA-WER-only RL, the 400 ms model reduces by 65%/44%/40%, but raises negative onset from 2.11% to 5.49%. Its pooled is 0.890 s. Thus, smaller buffers reliably accelerate typical turns with stronger regularization, while premature emission remains checkpoint-selection safeguards.
Delay-Reward Weight Ablation
Table 3 isolates reward strength at a fixed 700 ms onset buffer. Increasing from 0.05 to 0.2 lowers onset by 170 ms and both and by approximately 190 ms. SA-WER changes by only 0.19 absolute points, while the negative-onset rate remains nearly unchanged. The stronger bounded regularizer therefore supplies a materially clearer latency signal without a comparable degradation in recognition quality. Together with the buffer sweep, this motivates for the lowest-latency operating point while preserving SA-WER as the dominant objective through the multiplier’s 16.7% maximum reward reduction.
| SA-WER (%) | (s) | (s) | (s) | Neg. (%) | |
| 0.05 | 13.17 | 0.850 | 1.336 | 1.427 | 2.46 |
| 0.20 | 13.36 | 0.680 | 1.149 | 1.240 | 2.50 |
Table 3. Delay-reward weight ablation at a fixed 700 ms onset buffer.
Conclusion
We presented an RLVR approach for low-latency LLM-based streaming multi-talker ASR. A fixed-role SA-WER reward directly optimizes recognition and speaker attribution, while a bounded, coverage-aware onset-delay regularizer encourages timely turn emission without allowing latency to dominate quality. Our results show that sequence-level verifiable rewards can jointly improve fixed-role recognition quality and streaming responsiveness without modifying the underlying model architecture. Future work will extend evaluation to unscripted conversations and broader language coverage while improving robustness to premature emissions and direction-specific latency tails.
References
References
- [1]Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and Yu Wu, “On decoder-only architecture for speech-to-text and large language model integration,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). Dec. 2023, p. 1–8, IEEE.arxiv.org/abs/2307.03917
- [2]Chuanneng Sun, Zeeshan Ahmed, Yingyi Ma, Zhe Liu, Lucas Kabela, Yutong Pang, and Ozlem Kalinli, “Contextual biasing of named-entities with large language models,” in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Apr. 2024, p. 10151–10155, IEEE.arxiv.org/abs/2309.00723
- [3]Guanrou Yang, Ziyang Ma, Zhifu Gao, Shiliang Zhang, and Xie Chen, “Ctc-assisted llm-based contextual asr,” in 2024 IEEE Spoken Language Technology Workshop (SLT). Dec. 2024, p. 126–131, IEEE.arxiv.org/abs/2411.06437
- [4]Jing Pan, Jian Wu, Yashesh Gaur, Sunit Sivasankaran, Zhuo Chen, Shujie Liu, and Jinyu Li, “Cosmic: Data efficient instruction-tuning for speech in-context learning,” in Interspeech 2024. 2024, interspeech 2024, p. 4164–4168, ISCA.arxiv.org/abs/2311.02248
- [5]Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al., “Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras,” arXiv preprint arXiv:2503.01743, 2025.arxiv.org/abs/2503.01743
- [6]Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Basong Yang, Jin Xu, Jingren Zhou, and Junyang Lin, “Qwen3-asr technical report,” 2026.arxiv.org/abs/2601.21337
- [7]Ziyang Ma, Guanrou Yang, Wenxi Chen, Zhifu Gao, Yexing Du, Xiquan Li, Zhisheng Zheng, Haina Zhu, Jianheng Zhuo, Zheshu Song, Ruiyang Xu, Tiranrui Wang, Yifan Yang, Yanqiao Zhu, Zhikang Niu, Liumeng Xue, Yinghao Ma, Ruibin Yuan, Shiliang Zhang, Kai Yu, Eng Siong Chng, and Xie Chen, “Slam-llm: A modular, open-source multimodal large language model framework and best practice for speech, language, audio and music processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 20, no. 1, pp. 63–76, Jan. 2026.DOI
- [8]Alexander H Liu, Andy Ehrenberg, Andy Lo, Chen-Yo Sun, Guillaume Lample, Jean-Malo Deli gnon, Khyathi Raghavi Chandu, Patrick von Platen, Pavankumar Reddy Muddireddy, Rohin Arora, et al., “Voxtral realtime,” arXiv preprint arXiv:2602.11298, 2026.arxiv.org/abs/2602.11298
- [9]Liang Lu, Naoyuki Kanda, Jinyu Li, and Yifan Gong, “Streaming end-to-end multi-talker speech recognition,” IEEE Signal Processing Letters, vol. 28, pp. 803–807, 2021.DOI
- [10]Lingwei Meng, Shujie Hu, Jiawen Kang, Zhaoqing Li, Yuejiao Wang, Wenxuan Wu, Xixin Wu, Xunying Liu, and Helen Meng, “Large language model can transcribe speech in multi-talker scenarios with versatile instructions,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5.arxiv.org/abs/2409.08596
- [11]Naoyuki Kanda, Yashesh Gaur, Xiaofei Wang, Zhong Meng, and Takuya Yoshioka, “Serialized output training for end-to-end overlapped speech recognition,” arXiv preprint arXiv:2003.12687, 2020.arxiv.org/abs/2003.12687
- [12]Ju Lin, Jing Pan, Ruizhi Li, Ming Sun, Yuzong Liu, Alaa Hassan, Jing Zheng, and Florian Metze, “Equipping llm with directional multi-talker speech understanding capabilities,” arXiv preprint arXiv:2602.07211, 2026.arxiv.org/abs/2602.07211
- [13]Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura, “End-to-end speech recognition sequence training with reinforcement learning,” IEEE Access, vol. 7, pp. 79758–79769, 2019.DOI
- [14]Changfeng Gao, Yabin Li, Keyu An, Zhifu Gao, Zhihao Du, Han Zhao, and Xiangang Li, “Explore the reinforcement learning for the llm based asr and tts system,” arXiv preprint arXiv:2509.18569, 2025.arxiv.org/abs/2509.18569
- [15]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo, “Deepseek-math: Pushing the limits of mathematical reasoning in open language models,” 2024.
- [16]Thilo von Neumann, Christoph Boeddeker, Marc Delcroix, and Reinhold Haeb-Umbach, “Word error rate definitions and algorithms for long-form multi-talker speech recognition,” IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 3174–3188, 2025.DOI
