WorksComputer Science
Learning from Distribution: Rethinking One-Hot Supervision for ASR and QA in Speech Language Models
Abstract
Speech language models (SLMs) exhibit strong capabilities in automatic speech recognition (ASR) and general-purpose speech understanding, yet ASR-specific training often degrades spoken question answering (QA) performance. We revisit this conflict from the perspective of task-specific input–output mappings and one-hot supervision. Our findings reveal pronounced task-asymmetric effects: ASR-only training severely limits QA capability, whereas QA-only training retains reasonable ASR performance. We hypothesize that the relatively deterministic mapping of ASR, combined with single-reference supervision, biases models toward transcription-specific behavior. Motivated by this perspective, we advocate broadening ASR learning objectives while preserving recognition accuracy. We propose Learning from Distribution (LFD), which aligns speech-conditioned predictions with the conditional output distributions of a pretrained text LLM. Extensive experiments demonstrate that LFD improves both ASR and QA performance, exhibits superior data efficiency, and remains effective under full-parameter fine-tuning. CoT-ASR further supports the broader applicability of our proposed supervision principle.
Introduction
Recent advances in large language models (LLMs) have accelerated the development of speech language models (SLMs), improving both automatic speech recognition (ASR) and general-purpose speech understanding ([2, 26, 12, 28, 13]). Unlike conventional ASR systems, SLMs enable instruction-conditioned recognition, incorporating contextual information and domain-specific hotwords into transcription ([30, 24]). By leveraging pretrained LLM capabilities, SLMs offer a framework for unifying speech recognition and general-purpose language intelligence ([22 b]). Despite recent advances, general-purpose SLMs still lag behind specialized ASR systems in recognition accuracy and robustness, motivating ASR-specific fine-tuning ([20, 15]). However, ASR specialization may compromise general-purpose capabilities, particularly spoken question answering (QA), by biasing models toward transcription ([7, 8, 3, 5]). This raises a key challenge: improving ASR without sacrificing broader language capabilities. Existing studies have primarily attributed the degradation of general-purpose capabilities in SLMs to catastrophic forgetting and cross-modal misalignment ([8, 7, 3, 13, 21]). To mitigate these issues, various strategies have been explored, including experience replay, parameter-efficient adaptation, model merging, and cross-modal knowledge distillation ([8, 22, 4, 23, 5, 1]). Nonetheless, how task-specific data characteristics and supervision objectives contribute to capability degradation remains insufficiently understood. Addressing this question requires examining the relationship between task-specific mappings and training supervision. Although ASR and QA share the same generative formulation, ASR follows an approximately one-to-one mapping, whereas QA admits multiple valid responses. Despite these differences, both tasks commonly rely on one-hot cross-entropy (CE) supervision, which reinforces a single reference output per training example. While CE naturally aligns with the deterministic mapping of ASR, diverse QA responses can collectively approximate the underlying output distribution. We argue that these distinct mappings may lead to asymmetric learning behaviors under the same One-hot supervision objective.
We hypothesize that the interaction between task-specific input–output mappings and one-hot supervision contributes to the ASR–QA conflict. To address this, we advocate broadening ASR learning objectives beyond single-reference targets. We propose Learning from Distribution (LFD), which realizes this principle by aligning speech-conditioned predictions with the conditional output distributions of a pretrained text LLM. We further explore an alternative training strategy to demonstrate the broader applicability of this principle. Our main contributions are summarized as follows:
We investigate the ASR–QA conflict from a new perspective by examining the interaction between task-specific input–output mappings and one-hot supervision. Through systematic comparisons of ASR-only and QA-only training, we reveal pronounced task-asymmetric effects and provide a mapping-based explanation for this phenomenon.
We propose broadening ASR learning objectives based on the distinct input–output mappings of ASR and QA. We introduce Learning from Distribution (LFD) as one realization of this principle and further demonstrate its generality through Chain-of-Thought ASR.
Extensive experiments demonstrate that LFD improves both ASR and QA performance with superior data efficiency. Tail-aware top- supervision maintains competitive performance while enabling compact storage of teacher distributions and efficient offline supervision.
A Mapping Perspective on ASR–QA Conflict
ASR and QA can both be formulated as speech-conditioned text generation:
where is the speech input, is the task instruction, is the output sequence, and denotes the trainable adapter parameters.
Despite sharing the same formulation, ASR and QA exhibit distinct input–output mappings, as illustrated in Figure 1. ASR typically involves an approx-
Figure 1. Illustration of ASR’s approximately one-to-one mapping and QA’s one-to-many mapping with multiple valid responses.
imately one-to-one mapping from speech to transcription, whereas QA admits a one-to-many mapping with multiple semantically valid responses. These mapping differences are reflected in the entropy characteristics of their conditional output distributions. ASR generally exhibits lower conditional output entropy due to its relatively deterministic transcription targets, whereas QA tends to exhibit higher entropy owing to the diversity of valid responses. We investigate these task-dependent entropy characteristics in SLMs and text LLMs, with the main results presented in § 5.3 and additional analyses provided in Appendix § 10.
Conventional cross-entropy (CE) applies the same one-hot supervision to both tasks:
where denotes the reference token. For each training example, CE reinforces the conditional probability of a single reference output. For ASR, this supervision naturally aligns with its relatively deterministic input–output mapping, consistently reinforcing speech-to-transcription alignment. For QA, although each training example provides only one reference response, the inherent diversity of valid responses can be captured across sufficiently large training datasets, allowing CE to approximate the underlying conditional output distribution.
Nevertheless, the distinct mapping characteristics of ASR and QA may lead to different learning behaviors under the same supervision objective. In particular, repeated optimization toward deterministic ASR targets may bias speech–language alignment toward transcription-specific behavior, limiting access to the pretrained LLM’s broader generation capabilities. In contrast, diverse QA supervision may expose the model to richer generation patterns, resulting in different cross-task effects. We therefore hypothesize that the ASR–QA conflict is partly shaped by the interaction between task-specific input–output mappings and one-hot supervision, rather than being an inherent consequence of ASR data alone.
Method
Rethinking ASR Supervision
As discussed in § 2, the approximately one-to-one mapping of ASR naturally aligns with one-hot CE supervision. However, repeatedly optimizing deterministic transcription targets may bias speech–language alignment toward transcription-specific behavior, limiting the SLM’s ability to leverage the pretrained LLM’s broader generation capabilities. This suggests that the ASR–QA conflict may be mitigated by reconsidering how ASR data is used for supervision.
Motivated by this observation, we advocate broadening ASR learning objectives beyond single-reference supervision. Rather than relying solely on deterministic transcription targets, ASR training should incorporate richer supervision that preserves the pretrained LLM’s broader generation behavior.
This principle aims to maintain accurate speech recognition while reducing the risk of compromising general-purpose capabilities.
Learning from Distribution
Following this principle, we propose Learning from Distribution (LFD), which broadens the learning objective by aligning speech-conditioned predictions with the conditional output distributions of a pretrained text LLM. Instead of relying solely on one-hot reference tokens, LFD preserves the LLM’s probability allocation across possible predictions, providing richer supervision for ASR while aiming to retain its broader generation capabilities. As illustrated in Figure 2, a frozen text LLM teacher and an SLM student are conditioned on task-specific text and speech inputs, respectively.
Figure 2. Overview of LFD. The text LLM teacher generates responses and token distributions, which the Speech LLM student learns via forward KL divergence with a tail-aware top- approximation.
Given the target response , teacher forcing is applied to both models along the same response trajectory. Their token distributions are defined as
where denotes the task-conditioned textual context and the teacher distribution. LFD minimizes the forward KL divergence between them:
This objective transfers the LLM’s conditional generation behavior to speech, rather than supervising individual reference tokens alone.
Tail-Aware Top-k Distribution Learning
Directly storing full-vocabulary teacher distributions incurs substantial storage overhead [17, 29, 19]. To enable efficient distribution supervision, we retain only the top- tokens with the highest teacher probabilities [10]. Let denote the selected token set at position .
However, naively truncating the teacher distribution yields
which encourages the student to concentrate its probability mass on the selected tokens. In particular, when , this objective reduces to a scaled cross-entropy loss on the teacher’s most probable token, losing the intended benefits of distributional supervision.
Motivated by the importance of preserving tail probability information [29, 9, 14], we adopt a tail-aware KL objective that retains the aggregate probability mass outside the top- set. The teacher and student tail probabilities are defined as
The resulting objective is
By preserving residual probability mass, this objective constrains the student’s predictions within and beyond the top- set, enabling efficient distribution matching without storing full-vocabulary teacher distributions.
Experimental Setup
Model Architecture and Training Setup
Our SLM consists of a pretrained Whisper-large-v3 speech encoder [18], a modality adapter, and a Qwen3-4B LLM3 [27]. We employ Qwen3-4B as the frozen text teacher for distribution supervision.
We use 8,000 hours of speech from Emilia [6] to construct ASR and QA training data. ASR uses the original transcripts, while QA data is generated from transcripts using Qwen3-4B. We evaluate four settings: CE-ASR, CE-QA, LFD-ASR, and LFD-QA. Within each task, CE and LFD share identical data, architectures, and initializations, differing only in supervision objectives. LFD uses by default.
In the main experiments, we focus on speech–language alignment, where the pretrained speech encoder and LLM remain frozen, and only the modality adapter is optimized. All models are trained for one epoch with a learning rate of and a cosine learning rate scheduler.
Evaluation Benchmarks and Metrics
We evaluate ASR performance on the LibriSpeech test-clean and test-other sets [16] using word error rate (WER). For spoken QA, we adopt the TELEVAL benchmark [11], including LlamaQA-en, TriviaQA-en, and WebQ-en, and report answer accuracy (ACC). Detailed descriptions of the benchmarks and evaluation metrics are provided in Appendix § 11.
Results and Analysis
Task-Asymmetric Effects of One-Hot Supervision
Table 1 compares CE and LFD under ASR-only and QA-only training settings.
| System | Training Strategy | Training Data | ASR(Librispeech ↓) clean | ASR(Librispeech ↓) other | ASR(Librispeech ↓) Overall | QA(TELEVAL ↑) LlamaQA-en | QA(TELEVAL ↑) TriviaQA-en | QA(TELEVAL ↑) WebQ-en | QA(TELEVAL ↑) Overall |
| A1 | CE | ASR | 4.83 | 8.96 | 6.89 | 1.67 | 2.87 | 2.37 | 2.44 |
| A2 | LFD | ASR | 4.51 | 8.63 | 6.57 | 65.33 | 31.78 | 39.94 | 40.20 |
| B1 | CE | QA | 6.35 | 10.11 | 8.23 | 72.00 | 36.08 | 42.05 | 43.35 |
| B2 | LFD | QA | 5.18 | 9.23 | 7.20 | 73.33 | 36.56 | 42.57 | 43.93 |
Table 1. ASR and QA performance under CE and LFD supervision with ASR-only and QA-only training data. ASR performance is measured by WER(%), while QA performance is evaluated by ACC(%) on TELEVAL.
One-hot supervision exhibits task-asymmetric effects consistent with distinct input–output mappings. Comparing A1 and B1, ASR-only CE achieves strong recognition performance but severely limited QA capability. In contrast, QA-only CE maintains strong QA performance while retaining reasonable ASR capability. This pronounced asymmetry is consistent with our mapping perspective: the approximately one-to-one mapping of ASR, combined with one-hot supervision, repeatedly reinforces deterministic transcription targets, potentially biasing the adapter toward transcription-specific alignment. In contrast, the inherent diversity of QA responses may expose the model to richer generation patterns, resulting in different cross-task behaviors.
Broadening ASR supervision mitigates the degradation of general-purpose capabilities. Comparing A1 and A2, replacing CE with LFD under the same ASR-only training setting substantially improves QA capability while further enhancing ASR performance. Notably, these improvements are achieved without introducing any QA training data. By broadening the learning objective beyond individual reference tokens, LFD enables the model to better leverage the pretrained LLM’s broader generation capabilities. This finding suggests that the degradation associated with ASR specialization depends not only on the training data itself, but also on how the model learns from it.
Distributional supervision mitigates task-asymmetric learning effects. Comparing B1 and B2, LFD improves both ASR and QA under QA-only training. Comparing A2 and B2, LFD substantially narrows the cross-task gap between ASR-only and QA-only training, while each retains an advantage on its primary task. These findings suggest that learning richer conditional output distributions mitigates task-asymmetric effects, supporting our principle of broadening supervision beyond single-reference targets.
Scaling Behavior of CE and LFD
To investigate the scaling behavior of CE and LFD, we conduct ASR-only and QA-only training with k, k, k, and k hours of data, using a top- value of 256 for LFD. All experiments follow the same model architecture and training configurations described in § 4.1. Table 2 summarizes the overall ASR and QA performance, with detailed results provided in Appendix § 12.
| Training Data | Training Strategy | 2k | 4k | 6k | 8k |
| ASR Overall WER (%) ↓ | |||||
| ASR | CE | 113.89 | 9.73 | 7.00 | 6.89 |
| LFD | 9.79 | 7.32 | 6.83 | 6.57 | |
| QA | CE | 123.91 | 11.82 | 7.89 | 8.23 |
| LFD | 13.68 | 7.71 | 7.75 | 7.20 | |
| QA Overall Accuracy (%) ↑ | |||||
| ASR | CE | 0.52 | 2.44 | 2.99 | 2.44 |
| LFD | 35.90 | 40.04 | 40.13 | 40.20 | |
| QA | CE | 0.91 | 39.28 | 43.58 | 43.35 |
| LFD | 38.11 | 43.28 | 43.48 | 43.93 |
Table 2. Scaling results of CE and LFD under ASR-only and QA-only training settings with different amounts of training data.
LFD exhibits superior data efficiency. With limited data, LFD achieves substantially better ASR and QA performance than CE under both ASR-only and QA-only settings. While CE requires more data for effective speech–language alignment, LFD achieves strong cross-task performance at smaller scales. These results suggest that learning richer conditional output distributions provides more informative supervision, facilitating effective alignment with limited data.
Increasing data does not eliminate the task asymmetry of one-hot supervision. As training data increases, ASR-only CE substantially improves recognition performance but consistently fails to develop meaningful QA capability. In contrast, QA-only CE progressively improves both tasks, achieving strong QA performance while retaining reasonable ASR capability. This persistent asymmetry is consistent with our mapping perspective: the approximately one-to-one mapping of ASR, combined with one-hot supervision, repeatedly reinforces deterministic transcription targets, potentially favoring transcription-specific alignment. Importantly, increasing ASR data alone does not alleviate this limitation, suggesting that the observed capability degradation cannot be attributed solely to insufficient training data.
Broadening supervision changes cross-task scaling behavior. Unlike ASR-only CE, ASR-only LFD develops strong QA capability even with limited training data, while continuously improving recognition performance as data increases. QA-only LFD also maintains strong performance on both tasks, with QA accuracy gradually approaching saturation. Although QA-only LFD retains a modest advantage in QA performance, the gap between the two training settings remains substantially smaller than under CE. These findings further support our proposed principle that broadening the learning objective beyond single-reference targets can mitigate task-asymmetric alignment effects, rather than relying solely on increasing the amount of task-specific data.
Distributional Analysis of ASR and QA
To investigate the task-dependent distributional characteristics of ASR and QA, we analyze the token-level predictive entropy of SLMs trained with CE and LFD, as reported in Table 3.
| System | Training Strategy | Training Data | ASR ↓ | QA ↑ | ASR | QA |
| A1 | CE | ASR | 6.89 | 2.44 | 0.25 | 0.25 |
| B1 | QA | 8.23 | 43.35 | 0.17 | 0.39 | |
| A2 | LFD | ASR | 6.57 | 40.20 | 0.20 | 0.48 |
| B2 | QA | 7.20 | 43.93 | 0.15 | 0.60 |
Table 3. ASR and QA performance and token-level predictive entropy under CE and LFD supervision.
Predictive entropy reflects task-specific input–output mappings. Except for A1, all systems exhibit lower predictive entropy on ASR than on QA, consistent with the distinct mapping characteristics discussed in § 2. This difference reflects the relatively constrained output space of ASR and the diversity of valid responses in QA. In contrast, A1 exhibits similarly low entropy on both tasks and predominantly generates transcriptions even for QA inputs, suggesting a transcription-oriented bias induced by ASR-only CE supervision. Furthermore, as shown in Appendix § 10, LFD-trained SLMs exhibit task-dependent entropy patterns similar to those of the pretrained text LLM, suggesting that distributional supervision helps preserve its task-conditioned generation characteristics.
Effect of Distribution Granularity
To investigate the effect of distribution granularity, we vary top- over using 8,000 hours of training data under both ASR-only and QA-only settings. Figure 3 shows the overall performance, with details in Appendix § 13.
Figure 3. Effect of distribution granularity (top-) on ASR and QA performance under ASR-only and QA-only training settings. Dashed gray lines indicate the corresponding CE baselines.
LFD is robust to distribution granularity. Across both training settings, varying top- has a relatively limited impact on ASR and QA performance, with even achieving competitive results. For ASR-only training, this may be attributed to the concentrated teacher distributions induced by the repetition instruction. For QA-only training, diverse generation trajectories in the training data may provide rich supervision across different contexts, reducing the need to explicitly preserve large numbers of token probabilities at each position.
**Sparse distribution supervision offers practical advantages. The competitive performance of suggests that distributional supervision need not require fine-grained token probabilities. Notably, our tail-aware objective retains the teacher’s top-1 confidence and the remaining probability mass, rather than reducing supervision to a hard label. This finding suggests that sparse distribution supervision can substantially reduce the storage and data transfer overhead of teacher distributions, particularly in offline distillation.
Behavior under Full-Parameter Fine-Tuning
To investigate whether the task-asymmetric effects of CE persist beyond adapter-only alignment, we further conduct full-parameter fine-tuning experiments. For ASR-only training, both CE and LFD are initialized from the same ASR-only LFD model trained with top- during the alignment stage. Similarly, QA-only experiments are initialized from the corresponding QA-only LFD model. All training configurations are kept identical except for the supervision objectives and their associated settings, with top- used for LFD. Table 4 presents the results, including the performance of the initialization models.
| System | Training Data | Training Strategy | ASR (Librispeech ↓) clean | ASR (Librispeech ↓) other | ASR (Librispeech ↓) Overall | QA (TELEVAL ↑) LlamaQA-en | QA (TELEVAL ↑) TriviaQA-en | QA (TELEVAL ↑) WebQ-en | QA (TELEVAL ↑) Overall |
| C0(init) | LFD | 4.45 | 8.62 | 6.53 | 72.33 | 32.50 | 39.01 | 40.49 | |
| C1 | ASR | CE | 3.61 | 7.43 | 5.52 | 0.33 | 2.39 | 1.91 | 1.89 |
| C2 | LFD | 3.70 | 7.46 | 5.57 | 72.00 | 32.86 | 42.78 | 42.93 | |
| D0(init) | LFD | 5.24 | 9.27 | 7.25 | 73.33 | 36.68 | 42.11 | 43.67 | |
| D1 | QA | CE | 4.79 | 8.73 | 6.76 | 74.00 | 36.92 | 43.86 | 44.91 |
| D2 | LFD | 4.58 | 8.38 | 6.48 | 73.33 | 38.23 | 44.07 | 45.33 |
Table 4. ASR and QA performance and token-level predictive entropy under full-parameter fine-tuning with CE and LFD supervision. ASR performance is reported as WER (%), while QA performance is measured by accuracy (%). C0 and D0 denote the corresponding adapter-only LFD initialization models.
Task-asymmetric effects of one-hot supervision persist under full-parameter fine-tuning. From C0, ASR-only CE (C1) improves recognition performance but severely degrades QA capability. In contrast, QA-only CE (D1) improves both ASR and QA performance over its initialization (D0). This asymmetry is consistent with our mapping perspective: repeated optimization toward deterministic ASR targets under one-hot supervision may favor transcription-specific behavior, compromising previously acquired general-purpose capabilities. Importantly, this phenomenon persists even when
the model starts with strong QA capabilities, suggesting that the conflict is not limited to the initial speech–language alignment stage.
Broadening ASR supervision mitigates capability degradation during full-parameter fine-tuning. Compared with C0, ASR-only LFD (C2) improves ASR and overall QA performance. Notably, C2 achieves nearly identical ASR performance to C1 while substantially outperforming it on all QA benchmarks. Similarly, QA-only LFD (D2) improves both ASR and overall QA performance over D0 and outperforms CE (D1) on both overall metrics. These findings support our principle that broadening supervision beyond single-reference targets can mitigate task-asymmetric effects while maintaining strong task performance, even during full-parameter fine-tuning.
Beyond LFD: Rethinking ASR Supervision with CoT
The full-parameter fine-tuning results in Section 5.5 show that ASR-only CE training can severely degrade QA capabilities. To further examine our hypothesis regarding the interaction between ASR’s deterministic mapping and single-reference supervision, we investigate Chain-of-Thought ASR (CoT-ASR) as an alternative strategy. Unlike direct transcription supervision, CoT-ASR broadens the generation space through intermediate reasoning while preserving transcription as the primary objective. We first obtain a base SLM through adapter-only alignment on mixed CoT-ASR and CoT-QA data, then separately perform full-parameter fine-tuning with CoT-ASR and CoT-QA data from this shared initialization. As shown in Table 5, CoT-ASR improves recognition while largely preserving QA capability despite using only ASR data for full-parameter fine-tuning. CoT-QA similarly improves ASR while maintaining strong QA performance. These findings suggest that capability degradation is not inevitable under ASR-only training but may depend on the interaction between task mappings and supervision objectives. Thus, CoT-ASR provides an alternative realization of our principle of broadening ASR supervision beyond LFD.
| Model | Fine-Tuning Data | ASR WER (%) ↓ | QA ACC (%) ↑ |
| Base Model | – | 7.57 | 42.86 |
| CoT-ASR | ASR-only | 7.38 | 42.54 |
| CoT-QA | QA-only | 7.42 | 42.67 |
Table 5. ASR and QA performance of CoT-based supervision. The base model is obtained through adapter-only alignment with mixed CoT-ASR and CoT-QA data. CoT-ASR and CoT-QA are then separately full-parameter fine-tuned from the same initialization.
Discussion
Task Mappings and Supervision as Joint Factors
Our findings suggest that the ASR–QA conflict arises partly from the interaction between task-specific input–output mappings and supervision objectives. Although CE can learn conditional output distributions for both tasks, single-reference supervision may induce substantially different learning behaviors. In particular, ASR’s relatively deterministic mapping, combined with repeated transcription supervision, may bias speech–language alignment toward ASR-specific behavior. This interpretation is consistent with persistent QA degradation under ASR-only CE despite increasing training data, as well as severe capability loss during full-parameter fine-tuning. The distinct entropy characteristics of ASR and QA further support our proposed mapping perspective.
Our results also highlight broadening ASR supervision as a promising solution without changing the recognition objective. LFD realizes this principle by learning the pretrained LLM’s conditional output distributions beyond individual reference tokens, while CoT-ASR broadens the generation space through intermediate reasoning. Both strategies preserve ASR as the primary objective, suggesting that general-purpose capability degradation is not inevitable under ASR-only training but depends partly on how task-specific data is supervised during training.
Relation to Prior Distribution-based Alignment Methods
Prior studies have explored different perspectives to address the degradation of general-purpose capabilities in SLMs. DiVA (Held et al., [7]) and BLSP-KD (Wang et al., [22]) leverage cross-modal distillation to transfer pretrained LLM capabilities to speech, while SALAD (Cuervo et al., [3]) investigates capability degradation from the perspectives of catastrophic forgetting, cross-modal misalignment, and domain mismatch. Although these studies propose effective approaches to preserving general-purpose capabilities, they do not systematically examine why ASR-oriented training can lead to particularly severe capability degradation.
In contrast, our work introduces a mapping-based perspective to investigate the interaction between task-specific input–output mappings and one-hot CE supervision. We argue that the relatively deterministic mapping of ASR, combined with single-reference supervision, may bias models toward transcription-specific behavior, leading to asymmetric effects on ASR and QA capabilities. Based on this perspective, we advocate broadening the learning space of ASR while preserving its recognition objective. LFD realizes this principle by learning the conditional output distributions of pretrained LLMs, while CoT-ASR provides an alternative realization through richer generation trajectories. Our contribution lies not only in the method but also in a new perspective for understanding and mitigating the ASR–QA conflict.
Conclusion
In this work, we revisit the ASR–QA conflict in SLMs through the interaction between task-specific input–output mappings and supervision objectives. We reveal task-asymmetric effects of one-hot supervision and advocate broadening ASR learning objectives while preserving recognition accuracy. We realize this principle through Learning from Distribution (LFD), which leverages the conditional output distributions of pretrained LLMs to provide richer supervision. Experiments demonstrate that LFD improves both ASR and QA, mitigates task asymmetry, and preserves cross-task capabilities during full-parameter fine-tuning. CoT-ASR further supports the broader applicability of this principle, highlighting the importance of mapping-aware supervision for general-purpose SLMs.
Limitations
Our experiments are conducted on a single SLM architecture based on Whisper-large-v3 and Qwen3-4B. Although our mapping-based perspective focuses on task characteristics and supervision objectives rather than specific model architectures, its generalizability across different model families and scales remains to be explored. Furthermore, our analysis focuses on ASR and spoken QA, leaving its applicability to other speech-language tasks for future investigation. Extending this perspective to tasks with diverse input–output mappings may provide further insights into the interaction between task characteristics and supervision objectives.
Task-Conditioned Entropy Analysis
To further investigate the distributional characteristics of ASR and QA, we analyze the token-level predictive entropy of both text LLMs and SLMs. Given a model’s predictive distribution over vocabulary at generation step , the entropy is defined as
We report the average predictive entropy across response tokens for each evaluation benchmark. Text LLMs are conditioned on speech transcripts, whereas SLMs receive the corresponding speech inputs.
As shown in Table 6, both text LLMs and SLMs consistently exhibit substantially lower predictive entropy on ASR than on QA. This pattern is consistent with their distinct input–output mappings: ASR typically involves relatively deterministic transcription targets, whereas QA allows multiple semantically valid responses.
| Model | Type | ASR | QA | |||
| Qwen3-4B | LLM | 0.04 | 0.04 | 0.58 | 0.67 | 0.58 |
| LFD-ASR | SLM | 0.10 | 0.17 | 0.59 | 0.72 | 0.62 |
| LFD-QA | SLM | 0.09 | 0.13 | 0.54 | 0.66 | 0.56 |
Table 6. Token-level predictive entropy of text LLMs and SLMs on ASR and QA benchmarks. Lower values indicate more concentrated predictive distributions. LFD-ASR denotes the model obtained through full-parameter fine-tuning in § 5.5.
Notably, the pretrained Qwen3-4B exhibits the same task-dependent entropy pattern, suggesting that these distributional differences are not specific to speech–language alignment. Furthermore, the SLMs retain this distinction, exhibiting concentrated predictions on ASR and more diverse predictive distributions on QA. These observations provide additional evidence that the entropy characteristics of ASR and QA reflect their underlying task-specific mapping structures.
Evaluation Benchmarks and Metrics
ASR Evaluation
We evaluate ASR performance on LibriSpeech [16], a widely used English speech recognition benchmark derived from public-domain audiobooks. We use the test-clean and test-other subsets, which represent different recognition difficulty levels. Performance is measured by word error rate (WER), defined as , where , , and denote substitution, deletion, and insertion errors, respectively, and is the number of reference words. Overall WER is averaged across the two subsets.
Spoken QA Evaluation. We evaluate spoken QA using three English subsets of TELEVAL [11]: LlamaQA-en (300 samples), TriviaQA-en (837 samples), and WebQ-en (1,938 samples). These subsets assess spoken question answering across general knowledge and factoid QA tasks. Following TELEVAL’s evaluation protocol, responses are evaluated against multiple acceptable reference answers using string matching. We report answer accuracy (ACC) for each subset, with overall accuracy computed across all 3,075 samples.
Detailed Scaling Results
Table 7 presents detailed scaling results of CE and LFD under ASR-only and QA-only training settings. We evaluate both supervision objectives with 2,000, 4,000, 6,000, and 8,000 hours of training data to investigate their effectiveness across different data scales. ASR performance is reported as WER on LibriSpeech test-clean and test-other, while QA performance is measured by accuracy on three English TELEVAL subsets. Individual benchmark results and overall performance are provided for comprehensive comparison.
| Training Data | Training Strategy | Data Duration | ASR | QA (TELEVAL ) | |||||
| LibriSpeech-test | |||||||||
| clean | other | Overall | LlamaQA-en | TriviaQA-en | WebQ-en | Overall | |||
| ASR | CE | 2000 | 113.17 | 114.62 | 113.89 | 0.52 | 0.96 | 0.10 | 0.52 |
| 4000 | 7.31 | 12.16 | 9.73 | 2.33 | 2.75 | 2.32 | 2.44 | ||
| 6000 | 4.83 | 9.17 | 7.00 | 2.67 | 2.51 | 3.25 | 2.99 | ||
| 8000 | 4.83 | 8.96 | 6.89 | 1.67 | 2.87 | 2.37 | 2.44 | ||
| LFD | 2000 | 7.38 | 12.21 | 9.79 | 69.33 | 30.47 | 33.08 | 35.90 | |
| 4000 | 5.12 | 9.54 | 7.32 | 68.67 | 31.66 | 39.83 | 40.04 | ||
| 6000 | 4.71 | 8.95 | 6.83 | 68.33 | 32.62 | 39.01 | 40.13 | ||
| 8000 | 4.51 | 8.63 | 6.57 | 65.33 | 31.78 | 39.94 | 40.20 | ||
| QA | CE | 2000 | 121.52 | 126.31 | 123.91 | 1.00 | 2.03 | 0.41 | 0.91 |
| 4000 | 8.88 | 14.78 | 11.82 | 70.00 | 31.54 | 37.87 | 39.28 | ||
| 6000 | 5.63 | 10.15 | 7.89 | 74.33 | 35.01 | 42.52 | 43.58 | ||
| 8000 | 6.35 | 10.11 | 8.23 | 72.00 | 36.08 | 42.05 | 43.35 | ||
| LFD | 2000 | 10.63 | 16.75 | 13.68 | 69.00 | 29.27 | 37.15 | 38.11 | |
| 4000 | 5.56 | 9.88 | 7.71 | 73.67 | 37.04 | 41.28 | 43.28 | ||
| 6000 | 5.33 | 10.19 | 7.75 | 72.67 | 36.08 | 42.16 | 43.48 | ||
| 8000 | 5.18 | 9.23 | 7.20 | 73.33 | 36.56 | 42.57 | 43.93 |
Table 7. Detailed scaling results of CE and LFD under ASR-only and QA-only training settings with 2,000 to 8,000 hours of training data. ASR performance is reported as WER (%), while QA performance is measured by accuracy (%) on TELEVAL. Lower WER and higher accuracy indicate better performance.
The results show that LFD provides substantial advantages over CE at smaller data scales. As training data increases, ASR-only CE improves recognition performance but consistently exhibits limited QA capability. In contrast, LFD achieves stronger cross-task performance under both training settings. These observations further support the data efficiency and task-asymmetric learning effects discussed in § 5.2.
Detailed Distribution Granularity Results
Table 8 presents detailed results of LFD with different top- values under ASR-only and QA-only training settings. We investigate distribution granularity using , while keeping all other training configurations unchanged. For each setting, we report ASR performance on LibriSpeech test-clean and test-other, along with QA accuracy on three English TELEVAL subsets.
| Training Data | Top- | ASR (LibriSpeech-test ) | QA (TELEVAL ) | |||||
| clean | other | Overall | LlamaQA-en | TriviaQA-en | WebQ-en | Overall | ||
| ASR | 1 | 4.45 | 8.62 | 6.53 | 72.33 | 32.50 | 39.01 | 40.49 |
| 16 | 4.47 | 8.65 | 6.56 | 71.67 | 32.74 | 40.30 | 41.30 | |
| 64 | 4.51 | 8.56 | 6.53 | 69.00 | 32.86 | 38.70 | 40.07 | |
| 256 | 4.51 | 8.63 | 6.57 | 65.33 | 31.78 | 39.94 | 40.20 | |
| QA | 1 | 5.24 | 9.27 | 7.25 | 73.33 | 36.68 | 42.11 | 43.67 |
| 16 | 5.20 | 9.92 | 7.56 | 74.33 | 36.92 | 43.60 | 44.78 | |
| 64 | 5.17 | 9.90 | 7.53 | 74.00 | 37.28 | 43.09 | 44.52 | |
| 256 | 5.18 | 9.23 | 7.20 | 73.33 | 36.56 | 42.57 | 43.93 |
Table 8. Detailed results of LFD with different top- values under ASR-only and QA-only training settings. ASR performance is reported as WER (%) on LibriSpeech, while QA performance is measured by accuracy (%) on TELEVAL. Lower WER and higher accuracy indicate better performance.
Across different top- values, LFD maintains relatively stable performance on both ASR and QA tasks, suggesting limited sensitivity to distribution granularity. Notably, even achieves competitive results when the residual probability mass is preserved through tail-aware supervision. This finding highlights the practicality of compact teacher distribution storage for efficient offline training. Further analysis is provided in § 5.4.
References
- [1]Di Cao, Dongjie Fu, Hai Yu, Siqi Zheng, Xu Tan, and Tao Jin. 2026. X-opd: Cross-modal on-policy distillation for capability alignment in speech llms. arXiv preprint arXiv:2603.24596.arxiv.org/abs/2603.24596
- [2]Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.arxiv.org/abs/2407.10759
- [3]Santiago Cuervo, Skyler Seto, Maureen de Seyssel, Richard Bai, Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly, and Zakaria Aldeneh. 2026. Closing the gap between text and speech understanding in llms. In International Conference on Learning Representations, volume 2026, pages 146313–146335.arxiv.org/abs/2510.13632
- [4]Ruchao Fan, Bo Ren, Yuxuan Hu, Rui Zhao, Shujie Liu, and Jinyu Li. 2025. Alignformer: Modality matching can achieve better zero-shot instruction-following speech-llm. IEEE Journal of Selected Topics in Signal Processing, 19(7):1329–1337.arxiv.org/abs/2412.01145
- [5]Yujie Guo, Hongjie Chen, Jian Kang, Jie Li, Yongxiang Li, and Yong Qin. 2026. Ts-opd: Reconciling asr and qa in speech language models via task-specific on-policy distillation. arXiv preprint arXiv:2609.29464.arxiv.org/abs/2609.29464
- [6]Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. 2024. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 885–890. IEEE.arxiv.org/abs/2407.05361
- [7]William Held, Yanzhe Zhang, Minzhi Li, Weiyan Shi, Michael J Ryan, and Diyi Yang. 2025. Distilling an end-to-end voice assistant without instruction training data. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7876–7891.
- [8]Chi-Yuan Hsiao, Ke-Han Lu, Kai-Wei Chang, Chih-Kai Yang, Wei-Chih Chen, and Hung-yi Lee. 2025. Analyzing mitigation strategies for catastrophic forgetting in end-to-end training of spoken language models. arXiv preprint arXiv:2505.17496.
- [9]Huipeng Huang and Hongxin Wei. 2026. Tail-aware top-k on-policy distillation. arXiv preprint arXiv:2608.14728.arxiv.org/abs/2608.14728
- [10]Minchong Li, Feng Zhou, and Xiaohui Song. 2025a. Bild: Bi-directional logits difference loss for large language model distillation. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1168–1182.arxiv.org/abs/2406.13555
- [11]Zehan Li, Hongjie Chen, Qing Wang, Yuxin Zhang, Jing Zhou, Hang Lv, Mengjie Du, Yaodong Song, Jie Lian, Jian Kang, et al. 2025b. Televal: A dynamic benchmark designed for spoken language models in chinese interactive scenarios. arXiv preprint arXiv:2507.18061.arxiv.org/abs/2507.18061
- [12]Bin Lin, Bo Zhao, Boyong Wu, Chao Yan, Chen Wu, Cheng Yi, Chengyuan Yao, Daiqiao Liu, Fei Tian, Feng Tian, et al. 2026. Stepaudio 2.5 technical report. arXiv preprint arXiv:2605.23463.arxiv.org/abs/2605.23463
- [13]Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, and Hung-yi Lee. 2025. Developing instruction-following speech language model without speech instruction-tuning data. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
- [14]Linjian Meng, Siyuan Gan, YuHan Li, Xiran Wang, Ziyang Ding, Ditang Gou, Yiming Wu, and Zhen Zhao. 2026. Unbiased top-k estimation for on-policy distillation. arXiv preprint arXiv:2609.34447.arxiv.org/abs/2609.34447
- [15]Xuanfan Ni, Fei Yang, Fengping Tian, Qingjuan Li, Chenyang Lyu, Yichao Du, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2025. Marco-asr: A principled and metric-driven framework for fine-tuning large-scale asr models for domain adaptation. arXiv preprint arXiv:2512.22165.arxiv.org/abs/2512.22165
- [16]Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE.DOI
- [17]Hao Peng, Xin Lv, Yushi Bai, Zijun Yao, Jiajie Zhang, Lei Hou, and Ju anzi Li. 2025. Pre-training distillation for large language models: A design space exploration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3603–3618.arxiv.org/abs/2410.16215
- [18]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeav ey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
- [19]Bakbergen Ryskulov, Iker García-Ferrero, David Montero, David Jansen, Ali Hashemi, Jezabel R Garcia, Antonio Tiene, and Román Orús. 2026. Efficient knowledge distillation for llms: Offline top-k logits and a fused checked kl loss. arXiv preprint arXiv:2608.03796.arxiv.org/abs/2608.03796
- [20]Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, et al. 2026. Qwen3-asr technical report. arXiv preprint arXiv:2601.21337.arxiv.org/abs/2601.21337
- [21]Weiting Tan, Hirofumi Inaguma, Ning Dong, Paden D Tomasello, and Xutai Ma. 2025. Ssr: Alignment-aware modality connector for speech language models. In Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025), pages 56–75.arxiv.org/abs/2410.00168
- [22]Chen Wang, Minpeng Liao, Zhongqiang Huang, and Jiajun Zhang. 2024a. Bpls-kd: Bootstrapping language speech pre-training via knowledge distillation. arXiv preprint arXiv:2405.19041.arxiv.org/abs/2405.19041
- [23]Enzhi Wang, Qicheng Li, Zhiyuan Tang, and Yuhang Jia. 2026. Cross-modal knowledge distillation for speech large language models. In ICASSP 2026–2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 16112–16116. IEEE.arxiv.org/abs/2509.14930
- [24]He Wang, Linhan Ma, Dake Guo, Xiong Wang, Lei Xie, Jin Xu, and Junyang Lin. 2025. Contextasr-bench: A massive contextual speech recognition benchmark. arXiv preprint arXiv:2507.05727.arxiv.org/abs/2507.05727
- [26]Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yexuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. 2025. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765.arxiv.org/abs/2509.17765
- [27]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Cheng'en Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.arxiv.org/abs/2505.09388
- [28]Xiaoyu Yang, Xuenan Xu, Wenyi Yu, Siyin Wang, Changli Tang, Terumi Chiba, Siyuan Hou, Ziyang Zhang, Wen Wu, Baoxiang Li, et al. 2026. Salmonn-2: Advancing general-purpose hearing abilities with self-supervised representations. arXiv preprint arXiv:2607.17079.arxiv.org/abs/2607.17079
- [29]Mohd Abbas Zaidi, Akhil Kedia, Jinwoo Ahn, Taehwak Kwon, Kangwook Lee, Haejun Lee, Joohyung Lee, et al. 2025. Sparse logit sampling: Accelerating knowledge distillation in llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18085–18108.arxiv.org/abs/2503.16870
- [30]Jie Zheng, Gaofeng Cheng, Xuyang Wang, Qingwei Zhao, and Yonghong Yan. 2026. Multilevel contextual prompting for conversational asr: unifying conversation history and hotwords with speech llm. Speech Communication, page 103458.DOI


