betaFeedback

WorksComputer Science

Induction Head Origins of SLAM-ASR Speech Recognition Models

Abstract

What makes a small trained map from encoder states to the embedding space of a frozen language model sufficient for competitive speech recognition? The premise of the SLAM-ASR architecture is that a frozen language model possesses linguistic knowledge rich enough to perform the recognition task. This assumption, while reasonable, provides little insight into the mechanisms underlying these models. We therefore seek the concrete internal structures on which they rely. This work shows that SLAM-ASR models rely on induction heads of the frozen language model: the same heads that copy repeated token sequences in text are also important for mapping speech embeddings to transcript tokens. Moreover, we find pairs of heads that behave as a constrained form of differential attention, cancelling attention noise and sharpening attention onto the previously generated word. Finally, we show how induction heads could be used to improve transcription quality under noisy audio.

Introduction

Integrating features representing the input audio with language models is at the core of automatic speech recognition (ASR) [1]. Recently, both of these constituents improved vastly, but separately, largely due to the widespread use of the Transformer architecture [2]. Speech encoder and language model fusion approaches were suggested in [3, 4, 5]; see [6] for details. An alternative method is to use a general-purpose large language model (LLM) to generate a corrected transcript for an ASR hypothesis [7, 8]. However, the most popular method in recent years, commonly referred to as speech LLM (SLLM), is to train an LLM to also accept acoustic information from an audio encoder as input, through a learnable speech projector.

Early works of this kind [9, 10, 11] trained the encoder and adapted the LLM through low-rank updates, together with the speech projector. SLAM-ASR [12] (Figure 1), however, trains only the projector, with both encoder and LLM being frozen, yet still achieves competitive recognition. The outputs of such a projector need not coincide with token embeddings, and its sequence length differs from the text counterpart, so linguistic modelling and alignment between two sequences are to be solved by the LLM. The premise of SLAM-ASR, that the frozen LLM suffices for the linguistic task, does not name the mechanisms on which it relies, in particular, what makes the frozen decoder capable of reading the projected speech embeddings.

Figure 1. SLAM-ASR architecture. Speech projector is the only component being trained, while LLM and speech encoder remain frozen.

Previous studies in internal analysis of SLLMs [13] found that in a spoken question answering context, speech and text states are similar in a broad band of middle layers. In [14], the authors applied mechanistic interpretability tools to ASR models to analyze encoder representations and error modes such as repetition hallucination, which they linked to a single attention head in Whisper [15]. In a frozen-decoder setting like ours, many decoder layers were found redundant for ASR [16], which suggests that recognition capability is concentrated, possibly at the head level. In this vein, text-to-speech in-context learning (ICL) work [17] considered the role of induction heads [18, 19], showing that, as in text, ablating them hurts ICL more than ablating random heads. In the audio LLM domain, [20] considered using text-backbone heads to steer speaker attribution. Another work [21] learned head masks to specify tasks, including ASR. For the frozen LLM decoder scenario as in SLAM-ASR, to the best of our knowledge, per-head impact on ASR has not been studied. Building upon the soft-prompting view [22, 23], in which the decoder is presented with a noisy prompt and redundant version of the embedded transcript, we consider the following research questions (RQs):

(RQ1) Does ASR quality depend on induction heads present in the frozen LLM?

(RQ2) Are there heads responsible for mitigating redundancy of speech embeddings?

(RQ3) Does scaling induction heads at inference help recover recognition quality under noisy audio?

We show that: (1) ASR in SLAM-ASR models depends on induction heads of the frozen LLM, (2) the decoder contains pairs of heads that act together as a constrained form of differential attention [24], sharpening attention on the speech embeddings of the previously generated word; and (3) scaling heads of (1) at inference reduces WER and insertions under noisy audio in 2 of 3 LLMs.

Background

This work connects SLAM-ASR speech recognition models with induction heads and differential attention. We therefore give a brief intuition behind these two notions; for a more complete picture, see [18] and [24], respectively.

Induction heads. On a sequence of repeated tokens . . . ABA B . . . AA, an induction head is an attention head that, at the second occurrence of AA, attends to the token BB that followed its first occurrence (prefix matching) and raises the logit of BB (copying), so that the model continues the sequence with BB [18, 19]. Mechanistically, a previous-token head in an earlier layer writes the token that precedes each position into it, so the key K\mathbf{K} at BB carries “preceded by AA” information, which the query Q\mathbf{Q} at the second occurrence of AA matches. Such heads emerge during text pretraining of an LLM, together with ICL [19], and can be found in a trained model via the prefix-matching score (Sec. 3). In our context, the speech embeddings play the role of the first occurrence (Figure 2): the speech embeddings Vsj\mathcal{V}_{s_j} of each word approximate its generated token Vhi\mathcal{V}_{h_i}, e.g., Vs1,Vs2≈Vh1\mathcal{V}_{s_1}, \mathcal{V}_{s_2} \approx\mathcal{V}_{h_1}, and are followed by those of the next word. At a generated token Vhi\mathcal{V}_{h_i}, the second occurrence, an induction head therefore attends to the speech embeddings approximating Vhi+1\mathcal{V}_{h_{i+1}}, which is the pattern shown in the bottom left of Figure 2.

SLAM-ASR inference, induction head, and attention noise cancelling pair

Figure 2. Per-head contribution to generated sequence at token Vhi\mathcal{V}_{h_i}, which predicts Vhi+1\mathcal{V}_{h_{i+1}}. Bottom left: induction heads attend to the speech embeddings of Vhi+1\mathcal{V}_{h_{i+1}}. Bottom right: an attention noise cancelling pair jointly sharpens attention on the speech embeddings of Vhi\mathcal{V}_{h_i}.

Differential attention. Differential attention, proposed in [24], is an enhancement of the canonical Transformer, which mitigates attention noise, i.e., the non-negligible attention that softmax assigns to irrelevant context. Differential attention computes two softmax attention maps and subtracts the second, scaled by λ\lambda, from the first:

softmax⁡(Q1K1Td)−λsoftmax⁡(Q2K2Td)V.(1)\operatorname{softmax}\left(\frac{\mathbf{Q}_1\mathbf{K}_1^{\mathsf{T}}}{\sqrt{d}}\right)-\lambda\operatorname{softmax}\left(\frac{\mathbf{Q}_2\mathbf{K}_2^{\mathsf{T}}}{\sqrt{d}}\right)\mathbf{V}. \tag*{(1)}

As in a differential amplifier, whose output is proportional to the difference of its two inputs, G(v+−v−)G(v_+-v_-), so that the signal common to both is rejected [24], attention that both maps assign to the same positions cancels, and the resulting attention is sharper on the relevant context. In [24], the two maps and λ\lambda are built into the architecture and trained from scratch; a pretrained LLM has no such component. In our context, attention noise can be expected to grow with the redundancy of speech embeddings: each word spans several of them, so a head reading the speech sees more keys than for the same transcript as text, each receiving some attention. Two heads of the same layer can nevertheless act together as a constrained form of differential attention (Sec. 3); such a pair is shown in the bottom right of Figure 2, where its joint attention concentrates on the speech embeddings of the previously generated word, Vhi\mathcal{V}_{h_i}.

Methodology

Induction heads

Denote by (l,h)(l,h) head hh of layer ll, by tt and jj the query and key positions, by qtl,hq_t^{l,h}, kjl,hk_j^{l,h}, vjl,hv_j^{l,h} its query, key and value vectors, and by SS the set of positions of the speech embeddings, i.e., the outputs of the projector. The prefix-matching score of a head is computed on inputs x=(BOS,r1,…,rN,r1,…,rN)x=(\mathrm{BOS},r_1,\ldots,r_N,r_1,\ldots,r_N) with N=50N=50 and rir_i drawn uniformly from the vocabulary, so that the query at t=N+it=N+i is the second occurrence of rir_i:

PM⁡(l,h)=1N−1∑t=N+12N−1at,t−N+1l,h,(2)\operatorname{PM}(l,h)=\frac{1}{N-1}\sum_{t=N+1}^{2N-1}a_{t,t-N+1}^{l,h}, \tag*{(2)}

where at,jl,h=softmax⁡j≤t(qtl,hTkjl,h/d)a_{t,j}^{l,h}=\operatorname{softmax}_{j\leq t}\left(q_t^{l,h\mathsf{T}}k_j^{l,h}/\sqrt{d}\right), averaged over 40 sequences. Let CkC_k be the set of the top-kk heads by PM. As in [17, 25], we identify these as induction heads. The contribution of a set of heads to ASR is measured by setting their attention logits to −∞-\infty for j∈Sj \in S, so that each head’s attention is redistributed over the rest of the context, and by comparing the resulting increase in word error rate (WER) with that of random sets of kk heads drawn from the same layers, outside of CkC_k.

Attention noise cancelling

Two heads of the same layer that attend to the same speech embeddings but project what they read in opposite directions cancel part of each other’s output. If the cancelled part is the attention that both heads assign to irrelevant positions, the pair as a whole attends more sharply to the relevant ones, as differential attention does (§ 2). Below, we state how such pairs are selected, the approximation under which a pair acts as one head whose attention is the difference of the two, and how we compare its sharpness with that of h1h_1 alone.

Without layer index, the output of head hh at position tt is oth=∑j≤tat,jhvjWOho_t^h = \sum_{j \leq t} a_{t,j}^h v_j W_O^h, where WOhW_O^h is the head’s chunk of the output projection. Let h1h_1 be the head with the largest ASR importance, i.e., the largest increase in loss when only its logits for j∈Sj \in S are set to −∞-\infty (blocking), and h2h_2 the head of the same layer whose output is most anti-aligned with that of h1h_1. We consider such pairs (Figure 2) if both heads put on average at least half of their attention on SS and h2h_2 is far more anti-aligned with h1h_1 than any other head of the layer, measured by the mean of cos⁡(oth1,oth2)\cos(o_t^{h_1}, o_t^{h_2}) over generated positions. In the pairs we find, the two heads share a key/value group of grouped-query attention (§ 4.1), kjh1=kjh2k_j^{h_1}=k_j^{h_2} and vjh1=vjh2=vjv_j^{h_1}=v_j^{h_2}=v_j, so they read the same values and differ only in where they attend, through their queries, and in how they project the result, through their output chunks. Denote by oth,So_t^{h,S} the output otho_t^h with the sum restricted to j∈Sj \in S. The joint output of the pair on the speech embeddings is then, exactly,

oth1,S+oth2,S=∑j∈S, j≤t(at,jh1WOh1+at,jh2WOh2)vj.(3)o_t^{h_1,S}+o_t^{h_2,S}=\sum_{j\in S,\,j\leq t}\left(a_{t,j}^{h_1}W_O^{h_1}+a_{t,j}^{h_2}W_O^{h_2}\right)v_j . \tag*{(3)}

Anti-aligned outputs alone do not make this a subtraction of attention maps. Our reading rests on one approximation: h2h_2 projects each value in the direction opposite to h1h_1, WOh2vj≈−λWOh1vjW_O^{h_2}v_j\approx-\lambda W_O^{h_1}v_j, with

λ=arg⁡min⁡λ′∑j∥WOh2vj+λ′WOh1vj∥2,(4)\lambda=\underset{\lambda'}{\arg\min}\sum_j\left\|W_O^{h_2}v_j+\lambda'W_O^{h_1}v_j\right\|^2, \tag*{(4)}

where jj ranges over the positions in SS of all evaluation utterances; how well it holds is given by cos⁡(WOh1vj,WOh2vj)\cos(W_O^{h_1}v_j,W_O^{h_2}v_j). Under it,

oth1,S+oth2,S≈WOh1∑j∈S, j≤t(at,jh1−λat,jh2)vj,(5)o_t^{h_1,S}+o_t^{h_2,S}\approx W_O^{h_1}\sum_{j\in S,\,j\leq t}\left(a_{t,j}^{h_1}-\lambda a_{t,j}^{h_2}\right)v_j, \tag*{(5)}

i.e., the pair acts as one head with effective attention at,jh1−λat,jh2a_{t,j}^{h_1}-\lambda a_{t,j}^{h_2}: attention that h1h_1 and the λ\lambda-scaled h2h_2 assign alike to a position cancels. This is the form of differential attention [24], constrained in that the keys are shared, λ\lambda is determined by the output chunks, and there is no per-head normalisation; in particular, the effective attention can be negative and need not sum to one. To measure what the subtraction does, let TT be the set of query positions at which a token of a word other than the first is generated, and Ft⊆SF_t \subseteq S the positions aligned to the previously generated word. We call the attention on S∖FtS \setminus F_t attention noise and measure the sharpness of an attention matrix AA by the share of its positive part that falls on FtF_t,

ρ(A)=∑t∈T∑j∈Ft[At,j]+∑t∈T∑j∈S[At,j]+,(6)\rho(A)=\frac{\sum_{t\in T}\sum_{j\in F_t}[A_{t,j}]_+}{\sum_{t\in T}\sum_{j\in S}[A_{t,j}]_+}, \tag*{(6)}

where [x]+=max⁡(x,0)[x]_+=\max(x,0), comparing the pair’s effective attention Ah1−λAh2A^{h_1}-\lambda A^{h_2} with that of h1h_1 alone, where Ah=(at,jh)A^h=(a_{t,j}^h).

Inference-time head scaling

The performance of SLAM-ASR degrades significantly under additive noise [26]. To make use of CkC_k, we consider a common ASR hallucination mode under such noise: an increased insertion rate II, the number of inserted words relative to the number of reference words. The decoder may then continue generating from its language prior, raising the output of heads that read the speech is expected to give the audio more weight against the prior. For each head (l,h)∈Ck(l,h) \in C_k, we scale its output at every position, otl,h↦αotl,ho_t^{l,h}\mapsto\alpha o_t^{l,h}, and compare the resulting WER and II with those of the unscaled model, of equally many layer-matched random heads, and of all remaining heads of the same layers, each scaled by the same α\alpha.

Experiments

Models

We use the instruction-tuned Llama 3.2-3B [27], Gemma 2-2B [28] and Ministral 3-3B [29] models as decoders; for Ministral, only the text decoder is used. All three decoders use grouped-query attention [30]: the heads of a layer are split into groups, and the heads of a group share keys and values. The SLAM-ASR models use the Whisper turbo encoder [15]. The projector design follows [12]: concatenate every 5 consecutive encoder states and pass through a two-layer MLP with hidden size 2048 and ReLU activation to the decoder’s embedding space; we append a trainable end-of-speech embedding, initialised from the decoder’s end-of-turn token. Encoder and decoder are frozen. The prompt νp1,…,νpN\nu_{p_1},\ldots,\nu_{p_N} is “Transcribe the speech:” in each decoder’s chat template.

Datasets

Each projector is trained on the train-clean-100 subset of LibriSpeech [31] and validated on 500 utterances from dev-other. Evaluation uses test-clean and test-other.

Training Details

Optimisation follows [12]: AdamW [32] with learning rate 10−410^{-4} and no weight decay, 1,000 warm-up steps followed by a constant learning rate, and early stopping with patience 3, evaluated every 500 steps. Unlike [12], we use batch size 32 instead of 4 and train for at most 5 epochs instead of 100,000 steps. Each decoder is trained with three seeds.

Evaluation Protocol

Decoding is done greedily. h1h_1 is selected by ASR importance on 200 dev-clean utterances. Blocking and the head pairs are evaluated on 300 test-clean utterances with Montreal Forced Aligner alignments [33]: these give the word boundaries for FtF_t and attention peaks. Each speech embedding covers 100 ms and is assigned to the word whose boundaries contain its centre. ASR importance, head-pair quantities and attention peaks use teacher forcing. Blocking uses k=10k=10 in all decoders. Scaling uses α=1.5\alpha=1.5 and kk equal to 3% of the decoder’s heads (20 in Llama, 6 in Gemma, 25 in Ministral). Scaling is evaluated on 400 test-clean and 400 test-other utterances not used for blocking. We use five audio conditions: clean audio, and babble and white noise at 5 and 0 dB SNR.

Results

WER under attention head blocking is reported for each decoder in Table 1. Blocking C10C_{10} increases WER in all three decoders. The increase is much larger in Gemma, where 10 heads are about 5 % of all heads, against 1.5 and 1.2 % in Llama and Ministral, and varies heavily between seeds. In Llama and Ministral it is milder, yet still above that of random sets of heads of the same size. Moreover, CkC_k makes up 40, 33 and 40 % of the 3 % of heads with the largest ASR importance in Llama, Gemma and Ministral, against 4–9 % for random heads of the same layers. Within the speech embeddings, the attention of CkC_k peaks on the word being generated for 64–76 % of tokens, against 45–56 % for equally many heads with the largest attention mass on SS. These results show that ASR in SLAM-ASR models is concentrated in induction heads of the frozen decoder.

decoderintactC10C_{10} blockedrandom
Llama3.3–3.67.1–8.33.3–3.6
Gemma3.8–4.023.2–53.54.0–4.2
Ministral3.0–3.43.9–5.53.1–3.4

Table 1. WER (%) when the attention logits of C10C_{10} on SS are set to −∞-\infty; random: mean WER over 10 sets of 10 heads from the same layers, blocked in the same way. Ranges are over 3 seeds throughout.

Head pairs that meet the conditions of an attention noise cancelling pair are reported for each decoder in Table 2. In all three decoders, the h2h_2 output for each speech embedding is approximately a negative multiple of the h1h_1 output, as the −cos⁡-\cos column shows, indicating that the two heads act as one head with differential attention [24]. The effective attention of the pairs also supports this view: as the eff. column shows, it is sharper on the speech embeddings of the previously generated word than the attention of h1h_1 alone in all decoders, though only slightly in Ministral. Blocking h1h_1 alone increases WER far more than blocking both heads, whereas blocking h2h_2 alone leaves it unchanged in Gemma and Ministral: the two heads act as a unit, and blocking one leaves the output of the other uncancelled. Blocked together, the pair costs little in Llama and Ministral and more in Gemma; its sharper attention describes how the pair reads the speech rather than how much recognition depends on it.

decoder layer: h1,h2h_1,h_2−cos⁡-\cosλ\lambdaρ\rho h1h_1ρ\rho eff.WER, blocked h1h_1WER, blocked both
Llama 8: H8, H60.72–0.740.78–0.820.52–0.570.64–0.6745–563.8–7.2
Gemma 8: H1, H00.66–0.670.92–0.940.74–0.760.80–0.8158–14915–39
Ministral 14: H0, H20.78–0.790.73–0.740.78–0.800.80–0.815.1–13.83.3–4.2

Table 2. Attention noise cancelling head pairs, given as layer: h1h_1, h2h_2. The pair is the same in all three seeds. −cos⁡-\cos: negated median cosine between Wh1vjW^{h_1}v_j and Wh2vjW^{h_2}v_j, j∈Sj \in S; λ\lambda: scaling factor between the two; ρ\rho: share of positive attention on speech embeddings of the previously generated word, for h1h_1 alone and for the pair (eff.); WER (%) with attention to speech blocked for h1h_1 alone or both.

WER and insertion rate II under scaling are reported for each decoder in Table 3. Under noisy audio, scaling CkC_k results in lower WER and II than scaling the same number of random heads (by 1.4–2.6 WER in the noisy mean) or the remaining heads of the same layers (by 4.6–10.9) in all three decoders, which indicates that the reduction is not an effect of scaling heads in these layers as such. Scaling instead equally many heads with the largest ASR importance reduces WER less than scaling CkC_k in Llama (by 2.4, noisy mean), about as much in Ministral, and more in Gemma (by 4.5). Compared to the unscaled model, scaling CkC_k reduces WER and II in Llama and Ministral almost only under babble noise at 0 dB, where insertions are most frequent; under the other noise conditions WER changes by less than 1 point. In Gemma, scaling heads of these layers increases WER, and scaling CkC_k increases it least. On clean audio, scaling CkC_k increases WER by at most 0.2 points.

decoderscaledcleanbabble 0 dB WERbabble 0 dB IInoisy (mean) WERnoisy (mean) II
Llamanone4.689.040.441.915.0
CkC_k4.781.832.139.912.4
random4.589.941.342.415.5
same layers4.8111.863.650.823.8
Gemmanone4.7148.894.966.937.9
CkC_k4.8150.094.767.437.7
random4.8153.699.070.040.7
same layers5.2156.2101.572.042.6
Ministralnone4.286.039.339.614.1
CkC_k4.481.834.438.512.6
random4.386.039.639.914.4
same layers4.694.045.643.416.6

Table 3. WER and II (%) with heads scaled by α=1.5\alpha= 1.5: none (unscaled model), CkC_k (k=3k = 3 % of heads), equally many random heads from the same layers (mean over 10 sets), or all remaining heads of the same layers. Clean: WER without added noise on 400 test-clean and 400 test-other utterances; noisy (mean): over babble and white noise at 5 and 0 dB SNR added to the same utterances. Means over 3 seeds; in each seed, CkC_k gives lower noisy WER and II than random heads. Bold: lowest per decoder under noise.

Discussion

The link between SLAM-ASR models and induction heads of the frozen LLM gives a concrete answer to the question we posed in Sec. 1: it suggests that a small map suffices because its outputs only need to be readable by the LLM’s existing context-reading circuits, which perform prefix matching on them. This raises the empirical question of how projector training should be done, especially when done jointly with the decoder. For instance, updating only the heads with the highest PM score, rather than the whole decoder, is a potentially promising approach.

The attention noise cancelling pairs we found show that text-pretrained LLMs contain heads that act together to cancel attention noise on redundant input, without being explicitly trained for it, while [34] has to add differential attention to a pretrained LLM through adaptation. Heads that read the same input and project in opposite directions are known in text LLMs as negative and copy-suppression heads [35, 36], which act on the logits of copied tokens; the pairs we found share keys and values, and their subtraction acts on attention. It is not clear how these heads emerge during text pretraining, which is a promising direction for mechanistic interpretability research.

Conclusion

In this work, we have shown that speech recognition with LLMs in models such as SLAM-ASR depends on induction heads of the frozen LLM. In the decoders we considered, pairs of attention heads act together as a constrained form of differential attention, cancelling attention noise and sharpening attention on the speech embeddings of the previously generated word. Induction heads also matter when doing recognition on noisy audio: scaling them at inference mitigates hallucinations in two of the three decoders we consider.

References

References

  1. [1]F. Jelinek, “Continuous speech recognition by statistical methods,” Proceedings of the IEEE, vol. 64, pp. 532–556, 1976.DOI
  2. [2]A. Vaswani et al., “Attention is all you need,” in Proc. NeurIPS, 2017.arxiv.org/abs/1706.03762
  3. [3]S. Karita, N. E. Y. Soplin, S. Watanabe, M. Delcroix, A. Ogawa, and T. Nakatani, “Improving Transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration,” in Proc. Interspeech, 2019, pp. 1408–1412.DOI
  4. [4]C. Gulcehre et al., “On using monolingual corpora in neural machine translation,” arXiv preprint arXiv:1503.03535, 2015.arxiv.org/abs/1503.03535
  5. [5]A. Sriram, H. Jun, S. Satheesh, and A. Coates, “Cold fusion: Training Seq2Seq models together with language models,” in Proc. Interspeech, 2018, pp. 387–391.arxiv.org/abs/1708.06426
  6. [6]R. Schmitt, A. Zeyer, M. Zeineldeen, R. Schlüter, and H. Ney, “LLMs and speech: Integration vs. combination,” arXiv preprint arXiv:2603.15045, 2026.arxiv.org/abs/2603.15045
  7. [7]C.-H. H. Yang, Y. Gu, Y.-C. Liu, S. Ghosh, I. Bulyko, and A. Stolcke, “Generative speech recognition error correction with large language models and task-activating prompting,” in Proc. IEEE ASRU, 2023, pp. 1–8.DOI
  8. [8]A. Fedorchenko and T. Alumäe, “Optimizing Estonian TV subtitles with semi-supervised learning and LLMs,” in Proc. NoDaLiDa/Baltic-HLT, 2025, pp. 136–141.arxiv.org/abs/2501.05234
  9. [9]J. Wu et al., “On decoder-only architecture for speech-to-text and large language model integration,” in Proc. IEEE ASRU, 2023, pp. 1–8.arxiv.org/abs/2307.03917
  10. [10]Y. Fathullah et al., “Prompting large language models with speech recognition abilities,” in Proc. IEEE ICASSP, 2024, pp. 13351–13355.arxiv.org/abs/2307.11795
  11. [11]Z. Chen et al., “SALM: Speech-augmented language model with in-context learning for speech recognition and translation,” in Proc. IEEE ICASSP, 2024, pp. 13521–13525.arxiv.org/abs/2310.09424
  12. [12]Z. Ma et al., “An embarrassingly simple approach for LLM with strong ASR capacity,” arXiv preprint arXiv:2402.08846, 2024.arxiv.org/abs/2402.08846
  13. [13]M.-H. Hsu, X. Tian, J. Zhang, L. Lu, Y. Wang, and Z. Wu, “Geometric alignment without functional equivalence: A layer-wise analysis of the speech-text modality gap,” in Proc. NeurIPS, 2026.arxiv.org/abs/2603.01502
  14. [14]N. Glazer et al., “Beyond transcription: Mechanistic interpretability in ASR,” in Proc. AAAI, 2026, pp. 37407–37416.
  15. [15]A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavy, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. ICML, 2023, vol. 202 of PMLR, pp. 28492–28518.arxiv.org/abs/2212.04356
  16. [16]A. Moumen, G. Sun, and P. C. Woodland, “Measuring the redundancy of decoder layers in SpeechLLMs,” in Proc. Interspeech, 2026, pp. 7286–7291.arxiv.org/abs/2603.05121
  17. [17]C. Pouw, H. MohebbI, A. Alishahi, and W. Zuidema, “In-context learning in speech language models: Analyzing the role of acoustic features, linguistic structure, and induction heads,” arXiv preprint arXiv:2604.06356, 2026.arxiv.org/abs/2604.06356
  18. [18]N. Elhage et al., “A mathematical framework for transformer circuits,” Transformer Circuits Thread, 2021, https://transformer-circuits.pub/2021/framework/index.html.
  19. [19]C. Olsson et al., “In-context learning and induction heads,” Transformer Circuits Thread, 2022, https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.arxiv.org/abs/2209.11895
  20. [20]B. Das, “Inherited heads: Audio language models track speakers with their text backbone’s attention, and an attention-mass ranking retrieves a different set,” arXiv preprint arXiv:2609.14174, 2026.arxiv.org/abs/2609.14174
  21. [21]Y. Guo et al., “AHAMask: Reliable task specification for large audio language models without instructions,” in Proc. AAAI, 2026, vol. 40, pp. 21513–21521.
  22. [22]B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” in Proc. EMNLP, 2021, pp. 3045–3059.arxiv.org/abs/2104.08691
  23. [23]X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” in Proc. ACL, 2021, pp. 4582–4597.
  24. [24]T. Ye et al., “Differential Transformer,” in Proc. ICLR, 2025.arxiv.org/abs/2410.05258
  25. [25]J. Crosbie and E. Shutova, “Induction heads as an essential mechanism for pattern matching in in-context learning,” in Findings of NAACL, 2025, pp. 5034–5096.arxiv.org/abs/2407.07011
  26. [26]S. Kumar et al., “Performance evaluation of SLAM-ASR: The good, the bad, the ugly, and the way forward,” in Proc. IEEE ICASSP Workshops, 2025.arxiv.org/abs/2411.03866
  27. [27]A. Graffatiori et al., “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024.
  28. [28]Gemma Team, “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118, 2024.
  29. [29]A. H. Liu et al., “Ministral 3,” arXiv preprint arXiv:2601.08584, 2026.arxiv.org/abs/2601.08584
  30. [30]J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, “GQA: Training generalized multi-query transformer models from multi-head checkpoints,” in Proc. EMNLP, 2023, pp. 4895–4901.
  31. [31]V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: An ASR corpus based on public domain audio books,” in Proc. IEEE ICASSP, 2015, pp. 5206–5210.DOI
  32. [32]I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. ICLR, 2019.
  33. [33]M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using Kaldi,” in Proc. Interspeech, 2017, pp. 498–502.DOI
  34. [34]C. Kong, J. Jang, and N. Kwak, “Understanding Differential Transformer unchains pretrained self-attentions,” in Proc. NeurIPS, 2025, vol. 38, pp. 87064–87092.arxiv.org/abs/2505.16333
  35. [35]K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt, “Interpretability in the wild: a circuit for indirect object identification in GPT-2 small,” in Proc. ICLR, 2023.arxiv.org/abs/2211.00593
  36. [36]C. S. McDougall, A. Conmy, C. Rushing, T. McGrath, and N. Nanda, “Copy suppression: Comprehensively understanding a motif in language model attention heads,” in Proc. Black-boxNLP Workshop, 2024, pp. 337–363.DOI

Paper details

vlnc:W_h47gkdp5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents