betaFeedback

WorksComputer Science

Think Before You Transcribe: Context-Steered Reasoning for Hotword-Aware Speech Recognition

Abstract

Contextual automatic speech recognition (ASR) improves rare-entity recognition by supplying candidate hotwords, but retrieved candidates may be incomplete or incorrect. A reliable system must therefore exploit a relevant candidate without allowing unsupported alternatives to override the acoustic evidence. We propose a context-steered reasoning framework that generates a short, audio-grounded caption before transcription. This task-specific representation lets the model assess candidate relevance before making a lexical decision. Supervised fine-tuning establishes the structured caption-to-transcription behavior, while targeted trajectory injection strengthens group relative policy optimization when difficult utterances yield only unsuccessful rollouts. Experiments across diverse context conditions show that the proposed method consistently improves both transcription quality and hotword recognition while remaining robust to irrelevant candidates.

Introduction

Modern ASR systems, such as Whisper [27], achieve high accuracy on general-domain speech, yet remain unreliable on rare names and specialized terminology in medical, legal, financial, and technical applications [1]. These expressions are sparsely represented in general training data, often acoustically confusable, and disproportionately important to the utterance. Because such errors can invalidate otherwise readable transcripts and downstream search, rare-entity recognition remains a persistent obstacle to deploying ASR in knowledge-intensive settings.

Contextual ASR mitigates this problem by supplying candidate hotwords as an additional prior. Conventional systems integrate this information through shallow fusion, contextual adapters, specialized biasing modules, or external memories [25] (Yu et al. 2024); recent audio-language models instead place candidate entities directly in the prompt [1, 36, 43] (Xu et al. 2025b). Both paradigms improve rare-entity recognition when the supplied context is accurate. In deployment, however, hotwords are retrieved from a large inventory rather than curated for each utterance. Deeper retrieval improves target coverage but also introduces more plausible distractors, so the list may omit a spoken entity or contain no relevant entity at all. Context is thus uncertain evidence rather than a trusted constraint.

This retrieval setting changes the problem from unconditional biasing to selective use of uncertain evidence. A model that discounts the list can still miss a relevant candidate, whereas a model that trusts it too readily can insert an unsupported entity. Yet most contextual ASR benchmarks emphasize curated ground-truth lists or a single domain. A controlled evaluation should instead vary candidate quality for the same utterance, separating gains from relevant context from degradation caused by unsupported context. Without such pairing, it is unclear whether an improvement reflects better speech recognition, stronger contextual integration, or merely an easier candidate list.

The same uncertainty exposes a limitation in direct transcription. Mapping audio and candidates immediately to the final text leaves no explicit stage at which the model can relate a plausible lexical form to the acoustic and semantic content before committing to it. Weak steering may miss an ambiguous entity, while excessive steering copies a plausible but unsupported candidate. Intermediate reasoning has proved useful in language and audio tasks [6] (Guo et al. 2025; Ma et al. 2025); for contextual ASR, it can instead serve the focused role of grounding supported candidates in a description of the utterance while leaving unrelated candidates unused.

Learning this selective behavior also creates an optimization problem. Relative policy objectives are informative when a rollout group contains outputs of different quality, but every rollout for a difficult utterance may miss a supplied target entity. GRPO can rank these failures, yet the group contains no trajectory that demonstrates how a successful caption should connect the audio, candidate, and final lexical choice.

We propose CAST (Contextual ASR with Steered Thinking), a framework that produces a natural-language caption before the transcript. The caption summarizes the perceived utterance and naturally mentions only candidates that the model finds supported before it commits to a lexical choice. We train this behavior with supervised fine-tuning (SFT) followed by GRPO [29]. During GRPO, targeted off-policy trajectory injection handles eligible all-failure groups by probabilistically replacing the lowest-reward roll- so the list may omit a spoken entity or contain no relevant entity at all. Context is thus uncertain evidence rather than a trusted constraint.

This retrieval setting changes the problem from unconditional biasing to selective use of uncertain evidence. A model that discounts the list can still miss a relevant candidate, whereas a model that trusts it too readily can insert an unsupported entity. Yet most contextual ASR benchmarks emphasize curated ground-truth lists or a single domain. A controlled evaluation should instead vary candidate quality for the same utterance, separating gains from relevant context from degradation caused by unsupported context. Without such pairing, it is unclear whether an improvement reflects better speech recognition, stronger contextual integration, or merely an easier candidate list.

The same uncertainty exposes a limitation in direct transcription. Mapping audio and candidates immediately to the final text leaves no explicit stage at which the model can relate a plausible lexical form to the acoustic and semantic content before committing to it. Weak steering may miss an ambiguous entity, while excessive steering copies a plausible but unsupported candidate. Intermediate reasoning has proved useful in language and audio tasks [6] (Guo et al. 2025; Ma et al. 2025); for contextual ASR, it can instead serve the focused role of grounding supported candidates in a description of the utterance while leaving unrelated candidates unused.

Learning this selective behavior also creates an optimization problem. Relative policy objectives are informative when a rollout group contains outputs of different quality, but every rollout for a difficult utterance may miss a supplied target entity. GRPO can rank these failures, yet the group contains no trajectory that demonstrates how a successful caption should connect the audio, candidate, and final lexical choice.

We propose CAST (Contextual ASR with Steered Thinking), a framework that produces a natural-language caption before the transcript. The caption summarizes the perceived utterance and naturally mentions only candidates that the model finds supported before it commits to a lexical choice. We train this behavior with supervised fine-tuning (SFT) followed by GRPO [29]. During GRPO, targeted off-policy trajectory injection handles eligible all-failure groups by probabilistically replacing the lowest-reward rollout with a reference-conditioned caption-to-transcription trajectory generated by the SFT checkpoint. This replacement supplies a concrete positive example while retaining the remaining sampled failures for relative comparison. The resulting mixed group is processed by the usual group-normalized loss. Because the injected item is not sampled under the original policy input and receives no importance correction, the update is a deliberately biased training heuristic rather than an on-policy GRPO estimate.

We introduce RACE-Bench (Robust ASR with Contextual Entities Benchmark) to evaluate the same model as context quality changes. The benchmark contains 9,136 real and synthetic English and Chinese utterances, all reserved for testing. Each utterance is paired with correct-only, contextless, retrieved, and all-noise candidate lists. This paired design measures not only whether a model can exploit a supplied entity, but also how its behavior changes as useful context disappears and distractors remain. Because the audio is fixed across conditions, the results separate effective use of available entities from resistance to unsupported candidates.

On RACE-Bench, caption-based reasoning outperforms direct-transcription variants trained on the same data, and targeted injection further improves an otherwise identical GRPO policy. CAST gains from larger retrieved candidate sets without becoming more susceptible to all-noise context. Comparisons with native reasoning models show that the benefit comes from the task-specific intermediate representation rather than reasoning length alone. Its performance also transfers to ContextASR-Bench, while its output remains much shorter than unrestricted audio reasoning.

The main contributions of this work are:

  1. We propose CAST, a caption-then-transcription framework that explicitly grounds candidate entities in the perceived audio content before the final lexical decision.

  1. We introduce targeted off-policy trajectory injection, supplying a positive trajectory when every GRPO rollout misses a target entity.

  1. We construct RACE-Bench, a 9,136-utterance bilingual benchmark that provides a controlled evaluation of contextual ASR under correct-only, contextless, retrieved, and all-noise context on the same audio.

Contextual ASR. Contextual ASR incorporates hotword lists, domain information, conversation history, or external memories to improve recognition of rare and domain-specific entities. Traditional systems use shallow fusion, contextual adapters, cross-attention, or explicit entity copying [1, 32, 25, 44, 43] (Yu et al. 2024). Audio-language models can instead accept candidate entities directly in the prompt [4 b] (Xu et al. 2025a). Retrieval-based approaches obtain relevant documents, speech exemplars, or acoustically similar entities before decoding [18, 24, 17, 36, 15] (Li et al. 2024), while LLM-based correction systems revise first-pass transcripts or filter retrieved candidates [2] (Hu et al. 2024; Im et al. 2025). Contextual ASR resources include SlideSpeech, ContextASR-Bench, and Contextual Earnings-22 [34, 33, 8].

Audio Reasoning Models. Chain-of-thought reasoning generates intermediate steps before a final answer and has been studied in general language models [35, 13] (Guo et al. 2025) and multimodal reasoning [28] (Xu et al. 2024). Large audio and omni language models align audio encoders with language-model backbones, enabling end-to-end speech recognition and audio understanding (Chu et al. 2024; Dinkel et al. 2025; Xu et al. 2025a,b; Yao et al. 2024; Li et al. 2025a,b). Recent work extends intermediate reasoning to audio understanding [6, 22] (Ma et al. 2025) and speech recognition, including contextual and target-speaker ASR [5, 42].

RACE-Bench

RACE-Bench contains 9,136 English and Chinese audio utterances spanning real and synthetic speech. It is designed to evaluate contextual ASR as candidate quality changes while the underlying audio remains fixed.

Multi-Stage Data Curation

For real speech, an LLM first extracts only explicit names, organizations, products, technical terms, and other proper entities from each reference, producing an ⟨audio, transcript, entities⟩ triplet. Qwen3-ASR-1.7B [30] then transcribes the audio without context. We retain samples with a nonzero named-entity false-negative rate (NE-FNR), more than ten transcript tokens, and a word error rate (WER) below 20%. These filters select audio that is intelligible at the utterance level but remains difficult at the entity level.

Quality control then uses Qwen3-Omni-30B-A3B-Instruct (abbreviated below as Qwen3-Omni-30B-Instruct) with correct-only context. An LLM judge reviews disagreements between the contextual output and the annotation, distinguishing likely annotation errors from normalization, disfluency, repetition, and pronunciation variation. Proposed corrections update only the disputed entity and its occurrence in the transcript while preserving the original fields for traceability. A 50-sample audit found approximately 90% of proposed corrections accurate; the main failure mode was replacing an audio-faithful pronunciation with a canonical spelling.

Synthetic speech follows four controlled stages. We first generate natural spoken sentences containing one to three difficult entities, including uncommon names, transliterations, alphanumeric models, and domain terms. We normalize numbers, dates, symbols, and abbreviations to match their spoken realizations, then require every annotated entity to be an exact substring of the normalized reference. A multi-speaker TTS model generates the audio. Finally, contextual and contextless ASR checks remove synthesis failures and retain samples that are acoustically valid but challenging at the entity level. This pipeline provides controlled English and Chinese hard cases while maintaining agreement between the transcript and audio.

Overview of the CAST framework

Figure 1. Overview of the CAST framework.

Context Construction

For each triplet with entity set EE, we derive four views: correct-only uses EE; contextless uses an empty list; RAG retrieves top-kk candidates for k∈{5,10,20,50}k \in\{5, 10, 20, 50\}; and all-noise removes every retrieved member of EE and backfills with the next-ranked candidates to preserve list length. Every utterance is evaluated in every condition. RAG averages are arithmetic means over the four values of kk.

We use GLCLAP [15] only as a fixed audio-to-text retrieval tool. Retrieval is closed-set over the deduplicated entity pool for the corresponding source and language. Its entity recall is 76.8/83.6/88.6/92.5% for Chinese and 69.8/79.6/86.7/92.4% for English at k=5/10/20/50k = 5/10/20/50, respectively. These imperfect recalls create realistic partial-context conditions, while rank-preserving backfill makes all-noise lists acoustically plausible rather than randomly unrelated.

Complete construction details, including subset composition, filtering and correction procedures, synthetic-speech generation, source-level entity-pool construction, and all-noise backfilling, are provided in the appendix.

Comparison with Existing Benchmarks

Existing resources provide multilingual, domain-specific, or naturally contextualized speech, but they do not pair correct-only, retrieved, and entirely incorrect candidate lists for the same utterances. As summarized in Table 1, RACE-Bench complements these benchmarks with a controlled comparison across candidate quality while holding the audio constant.

BenchmarkAudioMulti.OracleRet.Noise
CtxASR-B.TTS✓✓––
Earnings-22Real––––
SlideSpeechReal––––
RACE-Bench (ours)Mixed✓✓✓✓

Table 1. Coverage of contextual ASR benchmarks: ContextASR-Bench [33], Contextual Earnings-22 [8], SlideSpeech [34], and RACE-Bench. “Multi.” denotes multilingual coverage; “Oracle” denotes a correct-only entity list, “Ret.” retrieval-augmented lists, and “Noise” entirely incorrect lists.

The CAST Framework

CAST has two components. A caption before transcription records how the model relates the candidate list to the audio. Targeted trajectory injection supplies a successful example when ordinary GRPO sampling produces no entity-complete rollout.

Contextual ASR with Retrieved Candidates

In retrieval-based contextual ASR, a system receives a speech signal XX together with a candidate list C={c1,…,cN}C = \{c_1, \ldots, c_N\} and produces a transcription YY. Unlike a curated hotword list, CC is only a noisy prior: it may contain some, all, or none of the entities spoken in XX. The system must therefore preserve acoustic fidelity when the list is unhelpful while exploiting a candidate when it resolves genuine acoustic ambiguity. CAST is designed for this selective integration problem. At inference time it observes only XX and CC; references and entity annotations are confined to training objectives and offline trajectory construction.

Context-Grounded Reasoning

Direct contextual ASR estimates pθ(Y∣X,C)p_\theta(Y \mid X, C) and commits to each lexical decision while jointly interpreting the audio and candidate list. This implicit steering can be too weak to recover an acoustically ambiguous entity or too strong when an irrelevant candidate appears plausible. CAST introduces a caption ZZ before the transcription:

pθ(Z,Y∣X,C)=pθ(Z∣X,C) pθ(Y∣X,C,Z).(1)p_\theta(Z,Y \mid X,C)=p_\theta(Z \mid X,C)\,p_\theta(Y \mid X,C,Z). \tag*{(1)}

The caption is a natural-language description of the audio that mentions only candidates supported by the reference during target construction; supported entities are marked in the sentence rather than emitted as a separate selection list. The complete response has the form

⟨caption⟩Z⟨/caption⟩\langle\text{caption}\rangle Z\langle/\text{caption}\rangle
⟨transcription⟩Y⟨/transcription⟩.(2)\langle\text{transcription}\rangle Y\langle/\text{transcription}\rangle. \tag*{(2)}

The model therefore grounds useful candidates in an acoustic interpretation before producing the transcription. This intermediate step strengthens the influence of supported entities without requiring every retrieved candidate to determine the output. The caption receives no separate reward: its value is learned through the quality and entity coverage of the resulting transcription. The appendix details training-corpus construction, caption-target generation, model instructions, and output parsing.

Supervised Fine-Tuning

We initialize CAST from Qwen3-Omni-30B-Instruct (Xu et al. 2025b). Supervised fine-tuning provides a cold start for the two-part output format. If O⋆=(Z⋆,Y⋆)O^\star= (Z^\star,Y^\star) is the caption-transcription target, the objective is standard next-token prediction:

LSFT=−∑t=1∣O⋆∣log⁡pθ(ot⋆∣X,C,o<t⋆).(3)\mathcal{L}_{\mathrm{SFT}}=-\sum_{t=1}^{\lvert O^\star\rvert}\log p_\theta(o_t^\star\mid X,C,o_{<t}^\star). \tag*{(3)}

This stage teaches the model to produce a caption before the transcript, but does not directly optimize WER or entity recall.

Group Relative Policy Optimization

Starting from the SFT checkpoint, we apply GRPO (Shao et al. [29]; Shivakumar et al. [31]; Kong et al. [16]). For each input (X,C)(X,C), the policy samples a group of outputs Oi=(Zi,Yi)O_i=(Z_i,Y_i). Only the transcription YiY_i contributes to the reward:

Ri=λwordRword(Yi,Y⋆)+λentRent(Yi,E+),(4)R_i=\lambda_{\mathrm{word}}R_{\mathrm{word}}(Y_i,Y^\star)+\lambda_{\mathrm{ent}}R_{\mathrm{ent}}(Y_i,E^+), \tag*{(4)}

where RwordR_{\mathrm{word}} is clipped transcription accuracy derived from WER and RentR_{\mathrm{ent}} is normalization-aware recall over the ground-truth entities that are present in the candidate list, denoted by E+E^+. The word-level term preserves overall fidelity, while the entity term increases the learning signal on acoustically ambiguous targets. No separate format, caption, length, or hallucination reward is used.1 The appendix gives the exact normalization, multiplicity, empty-set, and malformed-output rules.

Following standard GRPO, we normalize the total rewards within each group to obtain token-level advantages, optimize the clipped policy-ratio objective, and regularize against the SFT reference policy with a Kullback–Leibler (KL) penalty. Implementation hyperparameters are reported in Experimental Setup.

Targeted Off-Policy Trajectory Injection

Standard GRPO assumes that relative differences within a sampled group provide a useful learning signal. Contextual ASR exposes a specific failure of this assumption: for a difficult input, every rollout may miss at least one supplied target entity. Although the failures can still be ranked, none demonstrates how the entity should be grounded in the caption and carried into a correct transcription. Targeted injection closes this optimization gap by placing one successful trajectory into precisely such a group.

For rollout ii, let qiq_i denote recall over the provided target entities E+E^+. We trigger injection when E+E^+ is nonempty and

max⁡i∈{1,…,G}qi<1.(5)\max_{i\in\{1,\ldots,G\}}q_i<1. \tag*{(5)}

Thus, even the best rollout in the group is incomplete. For an eligible group, we inject with probability p=0.5p=0.5; otherwise the group follows ordinary GRPO. Contexts without a provided target entity do not satisfy the eligibility condition.

After SFT and before GRPO, we construct one stored trajectory for each training example using the SFT checkpoint πSFT\pi_{\mathrm{SFT}}. A three-shot prompt demonstrates how an audio signal, candidate list, and reference transcript map to the required caption. For the current example, πSFT\pi_{\mathrm{SFT}} receives the correct audio XX, candidate list CC, and reference transcript Y⋆Y^\star, and generates only the caption:

Zoff∼πSFT(⋅∣X,C,Y⋆,D3shot).(6)Z_{\mathrm{off}}\sim\pi_{\mathrm{SFT}}(\cdot\mid X,C,Y^\star,\mathcal{D}_{\mathrm{3shot}}). \tag*{(6)}

We then assemble the generated caption with the reference transcription:

Ooff=(Zoff,Y⋆).(7)O^{\mathrm{off}}=(Z_{\mathrm{off}},Y^\star). \tag*{(7)}

The trajectory is off-policy because caption generation has access to the answer and its transcript is inserted from the reference. It is not sampled from the current policy under the original input (X,C)(X,C).

When injection is selected, we identify the rollout with the lowest combined reward in Eq. (4) and replace that single rollout with OoffO^{\mathrm{off}}. We then recompute both reward components for all eight outputs. The injected output participates in the same reward normalization, advantage computation, and clipped loss implementation as the other seven outputs, without importance weighting for the behavior-policy mismatch. The resulting gradient is therefore biased relative to on-policy GRPO. We use the procedure as an empirical group-repair heuristic: it introduces one entity-complete trajectory while retaining seven sampled failures, whereas non-injected eligible groups preserve ordinary exploration.

The appendix gives the prompt construction and complete injection procedure.

Experiments

Experimental Setup

We evaluate on RACE-Bench and ContextASR-Bench Wang et al. [33]. We report WER for overall transcription, named-entity word error rate (NE-WER) within aligned entity spans, and exact-match NE-FNR for missed ground-truth entities. Under irrelevant context, named-entity false-positive rate (NE-FPR) measures the proportion of output entities copied from the noise list. All values are percentages and lower is better; the appendix specifies normalization, alignment, denominator, and empty-context rules.

The baselines cover direct and reasoning-capable audio-language models at different scales: Qwen2-Audio Chu et al. [4], Qwen2.5-Omni-3B and 7B (Xu et al. 2025a), Qwen3-Omni-30B-Instruct and Qwen3-Omni-30B-Think (Xu et al. 2025b), Step-Audio Huang et al. [11], Valley3 Chen et al. [3], and Kimi-Audio Kimi Team et al. [14]. Table 2 includes every system evaluated on RACE-Bench, while Table 3 includes systems with corresponding ContextASR-Bench results.

ModelCorrect-onlyContextlessAll-noiseRAG avg.
Models without explicit reasoning
Qwen2-Audio22.8/19.0/21.323.3/50.8/59.526.7/51.4/58.828.9/50.0/34.9
Qwen2.5-Omni-3B4.2/14.0/14.88.2/45.2/52.88.8/42.8/50.55.3/22.2/24.7
Qwen2.5-Omni-7B3.2/11.4/13.27.2/40.0/46.46.9/37.1/43.24.4/20.5/22.6
Qwen3-Omni-30B-Instruct2.6/9.2/10.36.5/38.0/44.26.5/35.7/42.23.6/15.2/17.2
Step-Audio4.4/15.5/16.18.0/45.7/51.89.7/45.6/51.56.5/19.2/19.8
Kimi-Audio5.4/25.2/29.37.2/39.6/45.913.7/41.5/46.810.3/30.3/30.5
Instruct-SFT2.0/4.8/4.86.7/39.7/46.36.6/37.1/44.03.4/14.9/16.0
Instruct-GRPO2.0/4.5/4.26.7/39.8/46.26.5/36.6/43.23.4/14.2/15.8
Models with explicit reasoning
Qwen3-Omni-30B-Think3.3/11.7/13.17.6/43.4/51.78.4/43.3/51.24.5/18.1/20.2
Valley34.7/11.6/12.510.7/52.3/61.411.8/48.3/56.66.8/20.7/21.9
Think-SFT2.4/12.5/5.87.4/46.6/53.17.0/43.0/49.43.8/17.5/18.5
Think-GRPO2.4/5.4/5.47.5/46.8/53.47.1/46.3/49.43.8/16.6/18.2
CAST (ours)1.9/3.6/3.46.2/34.6/40.46.3/33.8/40.03.2/11.8/12.9

Table 2. Results on RACE-Bench across external audio-language models and controlled 30B variants. Systems are grouped by whether they produce an explicit intermediate reasoning output. All metrics are percentages and lower is better. Bold and underlining mark the best and second-best values in each metric column. RAG is the arithmetic mean over top-k∈{5,10,20,50}k \in\{5, 10, 20, 50\}.

ModelEnglish SpeechEnglish DialogueMandarin SpeechMandarin Dialogue
Models without explicit reasoning
Qwen2-Audio11.5/27.3/35.114.0/33.0/32.99.9/24.1/30.07.0/22.8/26.2
Qwen2.5-Omni-3B4.0/7.8/9.74.8/14.4/12.92.1/10.6/14.13.1/15.1/15.2
Qwen2.5-Omni-7B4.0/7.4/8.75.3/11.8/9.21.8/9.8/12.22.4/14.1/13.2
Kimi-Audio2.9/6.7/8.04.7/13.5/11.32.0/11.1/15.32.9/15.9/16.7
Qwen3-Omni-30B-Instruct1.1/1.7/0.34.4/21.3/1.20.8/5.4/0.41.6/13.4/0.7
Instruct-SFT1.1/1.2/0.02.0/4.0/0.30.4/4.3/0.10.7/7.6/0.3
Instruct-GRPO1.1/1.2/0.11.9/3.9/0.20.4/4.3/0.10.7/7.6/0.2
Models with explicit reasoning
Think-SFT1.4/1.2/0.12.4/4.7/0.30.5/4.2/0.20.8/7.5/0.3
Think-GRPO1.3/1.2/0.12.4/5.2/0.30.5/4.3/0.10.8/7.6/0.3
CAST (ours)1.1/1.1/0.11.8/4.4/0.20.5/4.7/0.10.8/7.0/0.2

Table 3. Results on ContextASR-Bench for systems with corresponding evaluations, grouped by whether they produce an explicit intermediate reasoning output. All metrics are percentages and lower is better; bold and underlining mark the best and second-best values in each metric column.

We construct controlled variants from the same 30B base model. Instruct-SFT and Instruct-GRPO transcribe directly without caption tags, whereas Think-SFT and Think-GRPO retain the base model’s native reasoning format. CAST-SFT uses the context-grounded caption without reinforcement learning. The no-injection baseline shares the CAST-SFT checkpoint, caption format, 2,298 GRPO prompts, group size, reward, and optimization settings with CAST, but uses ordinary rollout.

SFT uses 5,000 synthetic examples, and GRPO uses 298 post-SFT bad cases plus 2,000 new examples. The appendix reports the context-setting mixture, balanced retrieval sizes, and construction of these training records.

For SFT, we apply low-rank adaptation (LoRA) with rank 8 and scaling factor 32 to all linear layers while freezing the audio encoder and aligner. We train for three epochs with learning rate 5×10−55 \times10^{-5}, a 5% warmup ratio, minimum learning rate 10−610^{-6}, maximum sequence length 2,048, and global batch size 32. For GRPO, we retain the same LoRA configuration and train for three epochs with learning rate 10−510^{-5} and global batch size 128. We use the checkpoint at the end of this fixed training schedule rather than selecting one on an evaluation benchmark. We use G=8G = 8 generations per input, maximum input length 2,048, maximum completion length 1,024, temperature 1.0, top-pp 0.85, KL coefficient 0.001, and clipping parameter 0.2. Training uses eight GPUs with expert parallel size 8.

All supported models are deployed and evaluated with vLLM under matched settings. Models not supported by vLLM use the officially recommended inference implementation. Every context-capable system receives the same audio and candidate list, and evaluation uses the parsed transcript.

Main Results

Table 2 first demonstrates why correct-only evaluation is insufficient. Qwen3-Omni-30B-Instruct reaches an NE-FNR of 10.3 when every target entity is supplied, but the value rises by 6.9 points under retrieved context and by 33.9 points without context. The correct-only score therefore substantially overestimates performance in a deployment-like retrieval setting. Entirely incorrect context exposes a different failure mode: Kimi-Audio’s WER rises from 7.2 without context to 13.7 with an all-noise list. These two patterns are not interchangeable. Missing targets reveal dependence on oracle coverage, whereas the degradation from contextless to all-noise input reveals susceptibility to unsupported lexical cues.

CAST achieves the lowest errors in every RACE-Bench setting. The advantage is largest on the entity-sensitive metrics: its correct-only NE-FNR is 3.4, compared with 10.3 for Qwen3-Omni-30B-Instruct and 13.2 for Qwen2.5-Omni-7B. Under retrieved context, the corresponding values are 12.9, 17.2, and 22.6. This result is not obtained by indiscriminately following the list. CAST also has the lowest contextless and all-noise WER, and the two values differ by only 0.1 point. The main comparison therefore supports both sides of selective contextualization: exploiting relevant evidence and remaining stable when no supplied candidate is correct.

Table 3 reports the corresponding ContextASR-Bench results. CAST obtains NE-FNR values of 0.1, 0.2, 0.1, and 0.2 across its four subsets, improving over Qwen3-Omni-30B-Instruct in each case. On English Dialogue, it reduces NE-WER from 21.3 to 4.4 and WER from 4.4 to 1.8. Together with Table 2, these results show strong recognition with correct context as well as robustness when context is missing, retrieved, or incorrect.

Retrieval-Augmented Context

Increasing kk improves the probability that a target entity is retrieved, but it also enlarges the set of distractors that may bias decoding. Figure 2 shows that CAST remains strongest in both WER and NE-FNR across all retrieval sizes. Its transcription error stays stable while its missed-entity error decreases as retrieval coverage grows, followed by only mild saturation at the largest list. Qwen3-Omni-30B-Instruct also benefits from larger lists, but retains a consistent gap, while Valley3 and Step-Audio show weaker or less regular improvements. CAST therefore converts broader hotword coverage into recognition gains without allowing the additional candidates to dominate the acoustic evidence.

WER and NE-FNR as retrieval size increases on RACE-Bench

Figure 2. WER and NE-FNR as retrieval size increases on RACE-Bench.

Incorrect-Context Robustness

Figure 3 tests whether an entirely incorrect list distorts the transcript. CAST changes minimally from contextless to all-noise input, whereas Kimi-Audio, Step-Audio, and Valley3

WER under contextless and all-noise input on RACE-Bench

Figure 3. WER under contextless and all-noise input on RACE-Bench.

exhibit clear WER degradation. NE-FPR provides complementary evidence by measuring whether entities are copied directly from the noise list. Although Qwen2.5-Omni-7B attains a slightly lower NE-FPR, it has weaker entity recall when relevant context is available. CAST combines a low false-activation rate with the strongest all-noise NE-FNR among the displayed systems, indicating that its robustness does not come from ignoring contextual information altogether.

Ablations

Table 4 isolates trajectory injection against the matched GRPO baseline.

SettingInjectBaselineΔ\Delta
Correct-only1.9/3.6/3.42.5/7.6/8.3-0.6/-4.0/-4.9
Contextless6.2/34.6/40.46.9/39.8/46.3-0.7/-5.2/-6.0
All-noise6.3/33.8/40.06.8/36.9/43.7-0.5/-3.2/-3.7
RAG avg.3.2/11.8/12.93.7/14.7/16.5-0.5/-2.9/-3.6

Table 4. Direct comparison between trajectory injection and the otherwise matched GRPO baseline on RACE-Bench. All metrics are percentages and lower is better. Bold marks the better system; Δ\Delta is inject minus baseline, so negative values favor injection.

Injection improves all three metrics in every setting relative to the otherwise matched GRPO baseline. The improvements extend to contextless and all-noise evaluation, even though these inputs contain no supplied target entity and cannot trigger replacement during training. This suggests that repairing eligible all-failure groups changes the learned transcription policy rather than merely memorizing the injected examples. The complete ablations in the appendix separate direct transcription, caption supervision, GRPO, injection, and the two reward terms. Together, they show that the caption defines the intended intermediate behavior, while targeted injection is needed to turn that behavior into consistent entity-recognition gains without sacrificing transcript fidelity.

Discussion

The main results distinguish task-specific contextual reasoning from unrestricted chain-of-thought. The native Qwen3-Omni-30B-Think model and Valley3 both generate explicit reasoning, yet remain behind CAST on contextual entity recognition. Think-SFT and Think-GRPO use the same 30B base model and task data but still do not match CAST, showing that additional training of a generic reasoning format is insufficient. The Instruct-SFT and Instruct-GRPO variants are more competitive in several correct-context conditions, but their RACE-Bench results remain weaker when retrieval is incomplete or incorrect. The ContextASR-Bench results further show that the advantage of the context-grounded caption is not confined to the newly constructed benchmark. These comparisons concern transcription behavior rather than caption faithfulness, since the caption receives no independent correctness reward.

ModelAvg. tokensEst. decode (s)
Ground-truth transcript27.3–
Qwen2.5-Omni-3B27.80.2
Qwen2.5-Omni-7B27.70.3
Qwen3-Omni-30B-Instruct27.90.2
Qwen3-Omni-30B-Think892.46.5
Valley3245.73.0
CAST (ours)65.10.5

Table 5. Average output length and approximate decoding cost.

Output Length and Estimated Latency CAST averages 65.1 output tokens, compared with 892.4 for Qwen3-Omni-30B-Think and 245.7 for Valley3. To estimate decoding cost, we use the official batch-one BF16 speeds of Qwen3-4B, Qwen3-8B, and Qwen3-30B-A3B on one NVIDIA H20 as proxies for the 3B, 7–8B, and 30B-A3B systems, respectively [26]; the estimates exclude audio encoding and prefill. CAST is longer than direct transcription because it includes a caption, but remains much closer to the reference-transcript length than the unrestricted reasoning models. Its estimated decoding cost is also substantially lower, giving it a stronger accuracy–efficiency trade-off. These values are analytical estimates rather than end-to-end measurements, and actual latency depends on the serving stack, batching, and audio length.

Limitations

RACE-Bench currently covers only English and Chinese, so broader linguistic and acoustic variation remains untested. In addition, CAST is evaluated with a 30B-A3B base model; whether the same reasoning and injection strategy remains effective for substantially smaller models requires further study.

Conclusion

We introduce RACE-Bench, a 9,136-utterance benchmark that evaluates contextual ASR under correct-only, contextless, retrieved, and entirely incorrect candidate lists. We propose CAST, whose context-grounded caption provides a compact reasoning step for using supported hotwords while rejecting unsupported ones. We further introduce targeted off-policy trajectory injection to repair GRPO groups in which every rollout misses a supplied entity. Experiments on RACE-Bench and ContextASR-Bench show that CAST improves contextual entity recognition and retrieval robustness while remaining substantially more efficient than unrestricted audio reasoning.

References

  1. [1]Bai, Y.; Chen, J.; Chen, J.; Chen, W.; Chen, Z.; Ding, C.; Dong, L.; Dong, Q.; Du, Y.; Gao, K.; Gao, L.; Guo, Y.; Han, M.; et al. 2024. Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition. CoRR, abs/2407.04675.arxiv.org/abs/2407.04675
  2. [2]Chen, C.; Hu, Y.; Yang, C.-H. H.; Siniscalchi, S. M.; Chen, P.-Y.; and Chng, E. S. 2023. HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models. In Advances in Neural Information Processing Systems, volume 36.arxiv.org/abs/2309.15701
  3. [3]Chen, Z.; Zhou, G.; Yin, Q.; Zhao, Z.; Yao, H.; Xia, P.; Yang, M.; Chen, C.; and Qiu, M. 2026. Valley3: Scaling Omni Foundation Models for E-commerce. arXiv:2605.01278.arxiv.org/abs/2605.01278
  4. [4]Chu, Y.; Xu, J.; Yang, Q.; Wei, H.; Wei, X.; Guo, Z.; Leng, Y.; Lv, Y.; He, J.; Lin, J.; Zhou, C.; and Zhou, J. 2024. Qwen2-Audio Technical Report. CoRR, abs/2407.10759.arxiv.org/abs/2407.10759
  5. [5]Deng, K.; Fan, R.; Ren, B.; Wang, Y.; and Li, J. 2026. Speech LLMs are Contextual Reasoning Transcribers. arXiv:2604.00610.arxiv.org/abs/2604.00610
  6. [6]Diao, X.; Zhang, C.; Kong, K.; Wu, W.; Ma, C.; Ouyang, Z.; Qing, P.; Vosoughi, S.; and Gui, J. 2025. SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 528–540. Association for Computational Linguistics.arxiv.org/abs/2506.12935
  7. [8]Durmus, B.; Cen, C.; Pacheco, E.; Okan, A.; and Orhon, A. 2026. Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild. arXiv:2604.07354.arxiv.org/abs/2604.07354
  8. [11]Huang, A.; Wu, B.; Wang, B.; Yan, C.; Hu, C.; Feng, C.; et al. 2025. Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction. arXiv:2502.11946.arxiv.org/abs/2502.11946
  9. [13]Jaech, A.; Kalai, A.; Lerer, A.; Richardson, A.; El-Kishky, A.; Low, A.; Helyar, A.; Madry, A.; Beutel, A.; Carney, A.; et al. 2024. OpenAI o1 System Card. CoRR, abs/2412.16720.arxiv.org/abs/2412.16720
  10. [14]Kimi Team; Ding, D.; Ju, Z.; Leng, Y.; Liu, S.; Liu, T.; et al. 2025. Kimi-Audio Technical Report. arXiv:2504.18425.
  11. [15]Kong, Y.; Cui, F.; Guo, L.; Dinkel, H.; Fan, L.; Zhang, J.; and Luan, J. 2025a. GLCLAP: A Novel Contrastive Learning Pre-trained Model for Contextual Biasing in ASR. In Interspeech 2025, 5173–5177.DOI
  12. [16]Kong, Y.; Hou, J.; Tang, J.; Zhu, B.; Zhang, J.; and Xue, S. 2025b. Contextual Biasing for LLM-Based ASR with Hotword Retrieval and Reinforcement Learning. arXiv:2512.21828.arxiv.org/abs/2512.21828
  13. [17]Lei, Z.; Na, X.; Xu, M.; Pusatari, E.; Gysel, C. V.; Zhang, Y.; Han, S.; and Huang, Z. 2025. Contextualization of ASR with LLM Using Phonetic Retrieval-Based Augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, 1–5. IEEE.arxiv.org/abs/2409.15353
  14. [18]Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.-t.; Rocktäschel, T.; Riedel, S.; and Kiela, D. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, volume 33, 9459–9474.arxiv.org/abs/2005.11401
  15. [22]Lin, Z.; Gao, Y.; Zhao, X.; Yang, Y.; and Sang, J. 2025. Mind with Eyes: from Language Reasoning to Multimodal Reasoning. CoRR, abs/2503.18071.arxiv.org/abs/2503.18071
  16. [24]Mathur, P.; Liu, Z.; Li, K.; Ma, Y.; Karen, G.; Ahmed, Z.; Manocha, D.; and Zhang, X. 2024. DOC-RAG: ASR Language Model Personalization with Domain-Distributed Co-occurrence Retrieval Augmentation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, 5132–5139. ELRA and ICCL.DOI
  17. [25]Mittal, A.; Sarawagi, S.; Jyothi, P.; Saon, G.; and Kurata, G. 2023. Speech-enriched Memory for Inference-time Adaptation of ASR Models to Word Dictionaries. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 14820–14835. Singapore: Association for Computational Linguistics.DOI
  18. [26]Qwen Team. 2025. Speed Benchmark. https://qwen.readthedocs.io/en/stable/getting_started/speed_benchmark.html. Accessed: 2026-07-29.
  19. [27]Radford, A.; Kim, J. W.; Xu, T.; Brockman, G.; McLeavy, C.; and Sutskever, I. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In Krause, A.; Brunskill, E.; Cho, K.; Engelhardt, B.; Sabato, S.; and Scarlett, J., eds., International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, 28492–28518. PMLR.arxiv.org/abs/2212.04356
  20. [28]Shao, H.; Qian, S.; Xiao, H.; Song, G.; Zong, Z.; Wang, L.; Liu, Y.; and Li, H. 2024a. Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning. In Globerson, A.; Mackey, L.; Belgrave, D.; Fan, A.; Paquet, U.; Tomczak, J. M.; and Zhang, C., eds., Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.arxiv.org/abs/2403.16999
  21. [29]Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024b. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. CoRR, abs/2402.03300.arxiv.org/abs/2402.03300
  22. [30]Shi, X.; Wang, X.; Guo, Z.; Wang, Y.; Zhang, P.; Zhang, X.; Guo, Z.; Hao, H.; Xi, Y.; Yang, B.; Xu, J.; Zhou, J.; and Lin, J. 2026. Qwen3-ASR Technical Report. arXiv:2601.21337.arxiv.org/abs/2601.21337
  23. [31]Shivakumar, P. G.; Gu, Y.; Gandhe, A.; and Bulyko, I. 2025. Group Relative Policy Optimization for Speech Recognition. arXiv:2509.01939.arxiv.org/abs/2509.01939
  24. [32]Sudo, Y.; Fujita, Y.; Kojima, A.; Mizumoto, T.; and Liu, L. 2025. OWSM-Biasing: Contextualizing Open Whisper-Style Speech Models for Automatic Speech Recognition with Dynamic Vocabulary. arXiv:2506.09448.arxiv.org/abs/2506.09448
  25. [33]Wang, H.; Ma, L.; Guo, D.; Wang, X.; Xie, L.; Xu, J.; and Lin, J. 2025. ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark. CoRR, abs/2507.05727.arxiv.org/abs/2507.05727
  26. [34]Wang, H.; Yu, F.; Shi, X.; Wang, Y.; Zhang, S.; and Li, M. 2024. SlideSpeech: A Large Scale Slide-Enriched Audio-Visual Corpus. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024, 11074–11080. IEEE.arxiv.org/abs/2309.05396
  27. [35]Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E. H.; Le, Q. V.; and Zhou, D. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, volume 35.arxiv.org/abs/2201.11903
  28. [36]Xiao, C.; Hou, Z.; Garcia-Romero, D.; and Han, K. J. 2025. Contextual ASR with Retrieval Augmented Large Language Model. In 2025 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2025, Hyderabad, India, April 6-11, 2025, 1–5. IEEE.DOI
  29. [42]Zhang, Y.; Su, H.; Fan, L.; Luo, Z.; and Luan, J. 2025. Thinking in cocktail party: Chain-of-Thought and reinforcement learning for target speaker automatic speech recognition. arXiv:2509.15612.arxiv.org/abs/2509.15612
  30. [43]Zhou, S.; and Li, Z. 2025. Improving Contextual ASR via Multi-grained Fusion with Large Language Models. CoRR, abs/2507.12252.arxiv.org/abs/2507.12252
  31. [44]Zhou, S.; Li, Z.; Hong, Y.; Zhang, M.; Wang, Z.; and Huai, B. 2024. CopyNE: Better Contextual ASR by Copying Named Entities. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2675–2686. Association for Computational Linguistics.

Paper details

vlnc:W_b35uvxtf
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 19

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents