WorksComputer Science
Induction Head Origins of SLAM-ASR Speech Recognition Models
Abstract
What makes a small trained map from encoder states to the embedding space of a frozen language model sufficient for competitive speech recognition? The premise of the SLAM-ASR architecture is that a frozen language model possesses linguistic knowledge rich enough to perform the recognition task. This assumption, while reasonable, provides little insight into the mechanisms underlying these models. We therefore seek the concrete internal structures on which they rely. This work shows that SLAM-ASR models rely on induction heads of the frozen language model: the same heads that copy repeated token sequences in text are also important for mapping speech embeddings to transcript tokens. Moreover, we find pairs of heads that behave as a constrained form of differential attention, cancelling attention noise and sharpening attention onto the previously generated word. Finally, we show how induction heads could be used to improve transcription quality under noisy audio.
Introduction
Integrating features representing the input audio with language models is at the core of automatic speech recognition (ASR) [1]. Recently, both of these constituents improved vastly, but separately, largely due to the widespread use of the Transformer architecture [2]. Speech encoder and language model fusion approaches were suggested in [3, 4, 5]; see [6] for details. An alternative method is to use a general-purpose large language model (LLM) to generate a corrected transcript for an ASR hypothesis [7, 8]. However, the most popular method in recent years, commonly referred to as speech LLM (SLLM), is to train an LLM to also accept acoustic information from an audio encoder as input, through a learnable speech projector.
Early works of this kind [9, 10, 11] trained the encoder and adapted the LLM through low-rank updates, together with the speech projector. SLAM-ASR [12] (Figure 1), however, trains only the projector, with both encoder and LLM being frozen, yet still achieves competitive recognition. The outputs of such a projector need not coincide with token embeddings, and its sequence length differs from the text counterpart, so linguistic modelling and alignment between two sequences are to be solved by the LLM. The premise of SLAM-ASR, that the frozen LLM suffices for the linguistic task, does not name the mechanisms on which it relies, in particular, what makes the frozen decoder capable of reading the projected speech embeddings.
Figure 1. SLAM-ASR architecture. Speech projector is the only component being trained, while LLM and speech encoder remain frozen.
Previous studies in internal analysis of SLLMs [13] found that in a spoken question answering context, speech and text states are similar in a broad band of middle layers. In [14], the authors applied mechanistic interpretability tools to ASR models to analyze encoder representations and error modes such as repetition hallucination, which they linked to a single attention head in Whisper [15]. In a frozen-decoder setting like ours, many decoder layers were found redundant for ASR [16], which suggests that recognition capability is concentrated, possibly at the head level. In this vein, text-to-speech in-context learning (ICL) work [17] considered the role of induction heads [18, 19], showing that, as in text, ablating them hurts ICL more than ablating random heads. In the audio LLM domain, [20] considered using text-backbone heads to steer speaker attribution. Another work [21] learned head masks to specify tasks, including ASR. For the frozen LLM decoder scenario as in SLAM-ASR, to the best of our knowledge, per-head impact on ASR has not been studied. Building upon the soft-prompting view [22, 23], in which the decoder is presented with a noisy prompt and redundant version of the embedded transcript, we consider the following research questions (RQs):
(RQ1) Does ASR quality depend on induction heads present in the frozen LLM?
(RQ2) Are there heads responsible for mitigating redundancy of speech embeddings?
(RQ3) Does scaling induction heads at inference help recover recognition quality under noisy audio?
We show that: (1) ASR in SLAM-ASR models depends on induction heads of the frozen LLM, (2) the decoder contains pairs of heads that act together as a constrained form of differential attention [24], sharpening attention on the speech embeddings of the previously generated word; and (3) scaling heads of (1) at inference reduces WER and insertions under noisy audio in 2 of 3 LLMs.
Background
This work connects SLAM-ASR speech recognition models with induction heads and differential attention. We therefore give a brief intuition behind these two notions; for a more complete picture, see [18] and [24], respectively.
Induction heads. On a sequence of repeated tokens . . . . . . , an induction head is an attention head that, at the second occurrence of , attends to the token that followed its first occurrence (prefix matching) and raises the logit of (copying), so that the model continues the sequence with [18, 19]. Mechanistically, a previous-token head in an earlier layer writes the token that precedes each position into it, so the key at carries “preceded by ” information, which the query at the second occurrence of matches. Such heads emerge during text pretraining of an LLM, together with ICL [19], and can be found in a trained model via the prefix-matching score (Sec. 3). In our context, the speech embeddings play the role of the first occurrence (Figure 2): the speech embeddings of each word approximate its generated token , e.g., , and are followed by those of the next word. At a generated token , the second occurrence, an induction head therefore attends to the speech embeddings approximating , which is the pattern shown in the bottom left of Figure 2.

Figure 2. Per-head contribution to generated sequence at token , which predicts . Bottom left: induction heads attend to the speech embeddings of . Bottom right: an attention noise cancelling pair jointly sharpens attention on the speech embeddings of .
Differential attention. Differential attention, proposed in [24], is an enhancement of the canonical Transformer, which mitigates attention noise, i.e., the non-negligible attention that softmax assigns to irrelevant context. Differential attention computes two softmax attention maps and subtracts the second, scaled by , from the first:
As in a differential amplifier, whose output is proportional to the difference of its two inputs, , so that the signal common to both is rejected [24], attention that both maps assign to the same positions cancels, and the resulting attention is sharper on the relevant context. In [24], the two maps and are built into the architecture and trained from scratch; a pretrained LLM has no such component. In our context, attention noise can be expected to grow with the redundancy of speech embeddings: each word spans several of them, so a head reading the speech sees more keys than for the same transcript as text, each receiving some attention. Two heads of the same layer can nevertheless act together as a constrained form of differential attention (Sec. 3); such a pair is shown in the bottom right of Figure 2, where its joint attention concentrates on the speech embeddings of the previously generated word, .
Methodology
Induction heads
Denote by head of layer , by and the query and key positions, by , , its query, key and value vectors, and by the set of positions of the speech embeddings, i.e., the outputs of the projector. The prefix-matching score of a head is computed on inputs with and drawn uniformly from the vocabulary, so that the query at is the second occurrence of :
where , averaged over 40 sequences. Let be the set of the top- heads by PM. As in [17, 25], we identify these as induction heads. The contribution of a set of heads to ASR is measured by setting their attention logits to for , so that each head’s attention is redistributed over the rest of the context, and by comparing the resulting increase in word error rate (WER) with that of random sets of heads drawn from the same layers, outside of .
Attention noise cancelling
Two heads of the same layer that attend to the same speech embeddings but project what they read in opposite directions cancel part of each other’s output. If the cancelled part is the attention that both heads assign to irrelevant positions, the pair as a whole attends more sharply to the relevant ones, as differential attention does (§ 2). Below, we state how such pairs are selected, the approximation under which a pair acts as one head whose attention is the difference of the two, and how we compare its sharpness with that of alone.
Without layer index, the output of head at position is , where is the head’s chunk of the output projection. Let be the head with the largest ASR importance, i.e., the largest increase in loss when only its logits for are set to (blocking), and the head of the same layer whose output is most anti-aligned with that of . We consider such pairs (Figure 2) if both heads put on average at least half of their attention on and is far more anti-aligned with than any other head of the layer, measured by the mean of over generated positions. In the pairs we find, the two heads share a key/value group of grouped-query attention (§ 4.1), and , so they read the same values and differ only in where they attend, through their queries, and in how they project the result, through their output chunks. Denote by the output with the sum restricted to . The joint output of the pair on the speech embeddings is then, exactly,
Anti-aligned outputs alone do not make this a subtraction of attention maps. Our reading rests on one approximation: projects each value in the direction opposite to , , with
where ranges over the positions in of all evaluation utterances; how well it holds is given by . Under it,
i.e., the pair acts as one head with effective attention : attention that and the -scaled assign alike to a position cancels. This is the form of differential attention [24], constrained in that the keys are shared, is determined by the output chunks, and there is no per-head normalisation; in particular, the effective attention can be negative and need not sum to one. To measure what the subtraction does, let be the set of query positions at which a token of a word other than the first is generated, and the positions aligned to the previously generated word. We call the attention on attention noise and measure the sharpness of an attention matrix by the share of its positive part that falls on ,
where , comparing the pair’s effective attention with that of alone, where .
Inference-time head scaling
The performance of SLAM-ASR degrades significantly under additive noise [26]. To make use of , we consider a common ASR hallucination mode under such noise: an increased insertion rate , the number of inserted words relative to the number of reference words. The decoder may then continue generating from its language prior, raising the output of heads that read the speech is expected to give the audio more weight against the prior. For each head , we scale its output at every position, , and compare the resulting WER and with those of the unscaled model, of equally many layer-matched random heads, and of all remaining heads of the same layers, each scaled by the same .
Experiments
Models
We use the instruction-tuned Llama 3.2-3B [27], Gemma 2-2B [28] and Ministral 3-3B [29] models as decoders; for Ministral, only the text decoder is used. All three decoders use grouped-query attention [30]: the heads of a layer are split into groups, and the heads of a group share keys and values. The SLAM-ASR models use the Whisper turbo encoder [15]. The projector design follows [12]: concatenate every 5 consecutive encoder states and pass through a two-layer MLP with hidden size 2048 and ReLU activation to the decoder’s embedding space; we append a trainable end-of-speech embedding, initialised from the decoder’s end-of-turn token. Encoder and decoder are frozen. The prompt is “Transcribe the speech:” in each decoder’s chat template.
Datasets
Each projector is trained on the train-clean-100 subset of LibriSpeech [31] and validated on 500 utterances from dev-other. Evaluation uses test-clean and test-other.
Training Details
Optimisation follows [12]: AdamW [32] with learning rate and no weight decay, 1,000 warm-up steps followed by a constant learning rate, and early stopping with patience 3, evaluated every 500 steps. Unlike [12], we use batch size 32 instead of 4 and train for at most 5 epochs instead of 100,000 steps. Each decoder is trained with three seeds.
Evaluation Protocol
Decoding is done greedily. is selected by ASR importance on 200 dev-clean utterances. Blocking and the head pairs are evaluated on 300 test-clean utterances with Montreal Forced Aligner alignments [33]: these give the word boundaries for and attention peaks. Each speech embedding covers 100 ms and is assigned to the word whose boundaries contain its centre. ASR importance, head-pair quantities and attention peaks use teacher forcing. Blocking uses in all decoders. Scaling uses and equal to 3% of the decoder’s heads (20 in Llama, 6 in Gemma, 25 in Ministral). Scaling is evaluated on 400 test-clean and 400 test-other utterances not used for blocking. We use five audio conditions: clean audio, and babble and white noise at 5 and 0 dB SNR.
Results
WER under attention head blocking is reported for each decoder in Table 1. Blocking increases WER in all three decoders. The increase is much larger in Gemma, where 10 heads are about 5 % of all heads, against 1.5 and 1.2 % in Llama and Ministral, and varies heavily between seeds. In Llama and Ministral it is milder, yet still above that of random sets of heads of the same size. Moreover, makes up 40, 33 and 40 % of the 3 % of heads with the largest ASR importance in Llama, Gemma and Ministral, against 4–9 % for random heads of the same layers. Within the speech embeddings, the attention of peaks on the word being generated for 64–76 % of tokens, against 45–56 % for equally many heads with the largest attention mass on . These results show that ASR in SLAM-ASR models is concentrated in induction heads of the frozen decoder.
| decoder | intact | blocked | random |
| Llama | 3.3–3.6 | 7.1–8.3 | 3.3–3.6 |
| Gemma | 3.8–4.0 | 23.2–53.5 | 4.0–4.2 |
| Ministral | 3.0–3.4 | 3.9–5.5 | 3.1–3.4 |
Table 1. WER (%) when the attention logits of on are set to ; random: mean WER over 10 sets of 10 heads from the same layers, blocked in the same way. Ranges are over 3 seeds throughout.
Head pairs that meet the conditions of an attention noise cancelling pair are reported for each decoder in Table 2. In all three decoders, the output for each speech embedding is approximately a negative multiple of the output, as the column shows, indicating that the two heads act as one head with differential attention [24]. The effective attention of the pairs also supports this view: as the eff. column shows, it is sharper on the speech embeddings of the previously generated word than the attention of alone in all decoders, though only slightly in Ministral. Blocking alone increases WER far more than blocking both heads, whereas blocking alone leaves it unchanged in Gemma and Ministral: the two heads act as a unit, and blocking one leaves the output of the other uncancelled. Blocked together, the pair costs little in Llama and Ministral and more in Gemma; its sharper attention describes how the pair reads the speech rather than how much recognition depends on it.
| decoder layer: | eff. | WER, blocked | WER, blocked both | |||
| Llama 8: H8, H6 | 0.72–0.74 | 0.78–0.82 | 0.52–0.57 | 0.64–0.67 | 45–56 | 3.8–7.2 |
| Gemma 8: H1, H0 | 0.66–0.67 | 0.92–0.94 | 0.74–0.76 | 0.80–0.81 | 58–149 | 15–39 |
| Ministral 14: H0, H2 | 0.78–0.79 | 0.73–0.74 | 0.78–0.80 | 0.80–0.81 | 5.1–13.8 | 3.3–4.2 |
Table 2. Attention noise cancelling head pairs, given as layer: , . The pair is the same in all three seeds. : negated median cosine between and , ; : scaling factor between the two; : share of positive attention on speech embeddings of the previously generated word, for alone and for the pair (eff.); WER (%) with attention to speech blocked for alone or both.
WER and insertion rate under scaling are reported for each decoder in Table 3. Under noisy audio, scaling results in lower WER and than scaling the same number of random heads (by 1.4–2.6 WER in the noisy mean) or the remaining heads of the same layers (by 4.6–10.9) in all three decoders, which indicates that the reduction is not an effect of scaling heads in these layers as such. Scaling instead equally many heads with the largest ASR importance reduces WER less than scaling in Llama (by 2.4, noisy mean), about as much in Ministral, and more in Gemma (by 4.5). Compared to the unscaled model, scaling reduces WER and in Llama and Ministral almost only under babble noise at 0 dB, where insertions are most frequent; under the other noise conditions WER changes by less than 1 point. In Gemma, scaling heads of these layers increases WER, and scaling increases it least. On clean audio, scaling increases WER by at most 0.2 points.
| decoder | scaled | clean | babble 0 dB WER | babble 0 dB | noisy (mean) WER | noisy (mean) |
| Llama | none | 4.6 | 89.0 | 40.4 | 41.9 | 15.0 |
| 4.7 | 81.8 | 32.1 | 39.9 | 12.4 | ||
| random | 4.5 | 89.9 | 41.3 | 42.4 | 15.5 | |
| same layers | 4.8 | 111.8 | 63.6 | 50.8 | 23.8 | |
| Gemma | none | 4.7 | 148.8 | 94.9 | 66.9 | 37.9 |
| 4.8 | 150.0 | 94.7 | 67.4 | 37.7 | ||
| random | 4.8 | 153.6 | 99.0 | 70.0 | 40.7 | |
| same layers | 5.2 | 156.2 | 101.5 | 72.0 | 42.6 | |
| Ministral | none | 4.2 | 86.0 | 39.3 | 39.6 | 14.1 |
| 4.4 | 81.8 | 34.4 | 38.5 | 12.6 | ||
| random | 4.3 | 86.0 | 39.6 | 39.9 | 14.4 | |
| same layers | 4.6 | 94.0 | 45.6 | 43.4 | 16.6 |
Table 3. WER and (%) with heads scaled by : none (unscaled model), ( % of heads), equally many random heads from the same layers (mean over 10 sets), or all remaining heads of the same layers. Clean: WER without added noise on 400 test-clean and 400 test-other utterances; noisy (mean): over babble and white noise at 5 and 0 dB SNR added to the same utterances. Means over 3 seeds; in each seed, gives lower noisy WER and than random heads. Bold: lowest per decoder under noise.
Discussion
The link between SLAM-ASR models and induction heads of the frozen LLM gives a concrete answer to the question we posed in Sec. 1: it suggests that a small map suffices because its outputs only need to be readable by the LLM’s existing context-reading circuits, which perform prefix matching on them. This raises the empirical question of how projector training should be done, especially when done jointly with the decoder. For instance, updating only the heads with the highest PM score, rather than the whole decoder, is a potentially promising approach.
The attention noise cancelling pairs we found show that text-pretrained LLMs contain heads that act together to cancel attention noise on redundant input, without being explicitly trained for it, while [34] has to add differential attention to a pretrained LLM through adaptation. Heads that read the same input and project in opposite directions are known in text LLMs as negative and copy-suppression heads [35, 36], which act on the logits of copied tokens; the pairs we found share keys and values, and their subtraction acts on attention. It is not clear how these heads emerge during text pretraining, which is a promising direction for mechanistic interpretability research.
Conclusion
In this work, we have shown that speech recognition with LLMs in models such as SLAM-ASR depends on induction heads of the frozen LLM. In the decoders we considered, pairs of attention heads act together as a constrained form of differential attention, cancelling attention noise and sharpening attention on the speech embeddings of the previously generated word. Induction heads also matter when doing recognition on noisy audio: scaling them at inference mitigates hallucinations in two of the three decoders we consider.
References
References
- [1]F. Jelinek, “Continuous speech recognition by statistical methods,” Proceedings of the IEEE, vol. 64, pp. 532–556, 1976.DOI
- [2]A. Vaswani et al., “Attention is all you need,” in Proc. NeurIPS, 2017.arxiv.org/abs/1706.03762
- [3]S. Karita, N. E. Y. Soplin, S. Watanabe, M. Delcroix, A. Ogawa, and T. Nakatani, “Improving Transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration,” in Proc. Interspeech, 2019, pp. 1408–1412.DOI
- [4]C. Gulcehre et al., “On using monolingual corpora in neural machine translation,” arXiv preprint arXiv:1503.03535, 2015.arxiv.org/abs/1503.03535
- [5]A. Sriram, H. Jun, S. Satheesh, and A. Coates, “Cold fusion: Training Seq2Seq models together with language models,” in Proc. Interspeech, 2018, pp. 387–391.arxiv.org/abs/1708.06426
- [6]R. Schmitt, A. Zeyer, M. Zeineldeen, R. Schlüter, and H. Ney, “LLMs and speech: Integration vs. combination,” arXiv preprint arXiv:2603.15045, 2026.arxiv.org/abs/2603.15045
- [7]C.-H. H. Yang, Y. Gu, Y.-C. Liu, S. Ghosh, I. Bulyko, and A. Stolcke, “Generative speech recognition error correction with large language models and task-activating prompting,” in Proc. IEEE ASRU, 2023, pp. 1–8.DOI
- [8]A. Fedorchenko and T. Alumäe, “Optimizing Estonian TV subtitles with semi-supervised learning and LLMs,” in Proc. NoDaLiDa/Baltic-HLT, 2025, pp. 136–141.arxiv.org/abs/2501.05234
- [9]J. Wu et al., “On decoder-only architecture for speech-to-text and large language model integration,” in Proc. IEEE ASRU, 2023, pp. 1–8.arxiv.org/abs/2307.03917
- [10]Y. Fathullah et al., “Prompting large language models with speech recognition abilities,” in Proc. IEEE ICASSP, 2024, pp. 13351–13355.arxiv.org/abs/2307.11795
- [11]Z. Chen et al., “SALM: Speech-augmented language model with in-context learning for speech recognition and translation,” in Proc. IEEE ICASSP, 2024, pp. 13521–13525.arxiv.org/abs/2310.09424
- [12]Z. Ma et al., “An embarrassingly simple approach for LLM with strong ASR capacity,” arXiv preprint arXiv:2402.08846, 2024.arxiv.org/abs/2402.08846
- [13]M.-H. Hsu, X. Tian, J. Zhang, L. Lu, Y. Wang, and Z. Wu, “Geometric alignment without functional equivalence: A layer-wise analysis of the speech-text modality gap,” in Proc. NeurIPS, 2026.arxiv.org/abs/2603.01502
- [14]N. Glazer et al., “Beyond transcription: Mechanistic interpretability in ASR,” in Proc. AAAI, 2026, pp. 37407–37416.
- [15]A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavy, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. ICML, 2023, vol. 202 of PMLR, pp. 28492–28518.arxiv.org/abs/2212.04356
- [16]A. Moumen, G. Sun, and P. C. Woodland, “Measuring the redundancy of decoder layers in SpeechLLMs,” in Proc. Interspeech, 2026, pp. 7286–7291.arxiv.org/abs/2603.05121
- [17]C. Pouw, H. MohebbI, A. Alishahi, and W. Zuidema, “In-context learning in speech language models: Analyzing the role of acoustic features, linguistic structure, and induction heads,” arXiv preprint arXiv:2604.06356, 2026.arxiv.org/abs/2604.06356
- [18]N. Elhage et al., “A mathematical framework for transformer circuits,” Transformer Circuits Thread, 2021, https://transformer-circuits.pub/2021/framework/index.html.
- [19]C. Olsson et al., “In-context learning and induction heads,” Transformer Circuits Thread, 2022, https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.arxiv.org/abs/2209.11895
- [20]B. Das, “Inherited heads: Audio language models track speakers with their text backbone’s attention, and an attention-mass ranking retrieves a different set,” arXiv preprint arXiv:2609.14174, 2026.arxiv.org/abs/2609.14174
- [21]Y. Guo et al., “AHAMask: Reliable task specification for large audio language models without instructions,” in Proc. AAAI, 2026, vol. 40, pp. 21513–21521.
- [22]B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” in Proc. EMNLP, 2021, pp. 3045–3059.arxiv.org/abs/2104.08691
- [23]X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” in Proc. ACL, 2021, pp. 4582–4597.
- [24]T. Ye et al., “Differential Transformer,” in Proc. ICLR, 2025.arxiv.org/abs/2410.05258
- [25]J. Crosbie and E. Shutova, “Induction heads as an essential mechanism for pattern matching in in-context learning,” in Findings of NAACL, 2025, pp. 5034–5096.arxiv.org/abs/2407.07011
- [26]S. Kumar et al., “Performance evaluation of SLAM-ASR: The good, the bad, the ugly, and the way forward,” in Proc. IEEE ICASSP Workshops, 2025.arxiv.org/abs/2411.03866
- [27]A. Graffatiori et al., “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024.
- [28]Gemma Team, “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118, 2024.
- [29]A. H. Liu et al., “Ministral 3,” arXiv preprint arXiv:2601.08584, 2026.arxiv.org/abs/2601.08584
- [30]J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, “GQA: Training generalized multi-query transformer models from multi-head checkpoints,” in Proc. EMNLP, 2023, pp. 4895–4901.
- [31]V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: An ASR corpus based on public domain audio books,” in Proc. IEEE ICASSP, 2015, pp. 5206–5210.DOI
- [32]I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. ICLR, 2019.
- [33]M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using Kaldi,” in Proc. Interspeech, 2017, pp. 498–502.DOI
- [34]C. Kong, J. Jang, and N. Kwak, “Understanding Differential Transformer unchains pretrained self-attentions,” in Proc. NeurIPS, 2025, vol. 38, pp. 87064–87092.arxiv.org/abs/2505.16333
- [35]K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt, “Interpretability in the wild: a circuit for indirect object identification in GPT-2 small,” in Proc. ICLR, 2023.arxiv.org/abs/2211.00593
- [36]C. S. McDougall, A. Conmy, C. Rushing, T. McGrath, and N. Nanda, “Copy suppression: Comprehensively understanding a motif in language model attention heads,” in Proc. Black-boxNLP Workshop, 2024, pp. 337–363.DOI
