WorksComputer Science
Who Asks, Who Answers: Linear Attention Builds the Query in Hybrid Language Models
Abstract
Hybrid language models interleave linear-attention layers with a small number of softmax-attention layers. Prior work largely attributes in-context retrieval to softmax attention, treating linear attention primarily as a mechanism for compressing past context. We revisit this division of labor and show that linear attention plays a direct role in retrieval by constructing the query used by softmax attention. Studying Qwen3.5 (Gated DeltaNet) and Granite4.0 (Mamba-2), we identify an Encode–Address circuit. An early Encode block preserves token identity, while an Address block constructs the final-position representation from which a softmax reader forms its attention query. Transplanting this representation between prompts with different queried keys redirects retrieval in 0.97–0.99 of cases. The Address block's recurrent state carries the retrieval target, and its computation transfers across architectures: replacing Granite's Address block with Qwen's restores retrieval accuracy to 0.92–0.94 without updating either model's weights. Across six Qwen3.5 models, Address computation becomes distributed at larger scales over a chain of blocks, each guiding a successive reader, as the query no longer passes from one reader to the next through attention. These findings reveal a division of labor in hybrid retrieval: linear attention constructs the query, while softmax attention performs the lookup. Code is available at https://github.com/yuang-chen/LinearQuery
Introduction
Hybrid language models combine linear-attention layers with a small number of softmax-attention layers. Linear attention, such as Gated DeltaNet [23] or Mamba-2 [4], compresses past context into a fixed-size state and scales linearly with sequence length, while softmax attention retains an explicit cache of past tokens. The combination aims to provide both the efficiency of recurrent state and the ability to retrieve exact information from earlier in the context, such as the value associated with a key introduced several paragraphs before. Recent open models, including Qwen3.5 and Granite-4.0-H, adopt this design with only one softmax layer for every few linear-attention layers [7] (Qwen Team 2026).
Prior work has shown that linear attention struggles with in-context retrieval [8, 1], while softmax attention plays a central role in retrieval within hybrid models [3]. However, identifying where retrieval occurs does not explain how the surrounding linear-attention layers contribute to it. We therefore ask: What do linear-attention layers compute to support retrieval by softmax attention?
We investigate this question through in-context key–value retrieval, where a model is given a dictionary (e.g., cherry=steel, mango=wood”) and must answer a question such as “What is the value of mango?” with “wood”. Although softmax attention can directly access the dictionary, we find that linear-attention layers are essential: removing the earliest linear-attention layer reduces retrieval accuracy to at most 0.05 across the models studied, even with all softmax layers intact.
To understand why, we first identify where retrieval emerges. Progressive layer pruning reveals a sharp increase in retrieval accuracy at a particular softmax layer, which we call the reader. We then use two causal interventions on linear-attention blocks: removal tests whether a block is necessary for retrieval, while transplantation tests whether its output can redirect retrieval between prompts with different queried keys.
These interventions reveal an Encode–Address circuit with two distinct roles (Figure 1a). The earliest linear-attention block, which we call Encode, preserves token identity needed for retrieval. Its contribution to this task can be reproduced by a simple token-wise linear projection. The Address block, immediately preceding the reader, constructs a representation at the final position that determines which dictionary entry the reader selects. Transplanting this output from one prompt into another redirects retrieval in 0.97–0.99 of cases in Qwen3.5-9B, even though the dictionary and question in the receiving prompt remain unchanged (Figure 1b).
Figure 1. Overview. (a) The Encode block re-encodes token identity, the Address block constructs the retrieval query, and softmax attention retrieves the matching value. (b) Transplanting the address from prompt A into prompt B makes the model answer A’s question. (c) Addresses transfer between Qwen3.5 (Gated DeltaNet) and Granite4.0 (Mamba-2)
We further investigate how the Address block performs this computation. Query–key interventions show that it primarily determines the reader’s attention query rather than the dictionary keys. Transplanting its recurrent state also redirects retrieval. These results indicate that the Address block carries the retrieval target in its recurrent state and uses it to construct the representation from which the softmax reader forms its query.
Strikingly, the Address block’s computation transfers across independently pretrained hybrid model families. Replacing the Address blocks of Granite-4.0-H (Mamba-2) with those from Qwen3.5-9B (Gated DeltaNet) restores retrieval accuracy to 0.94, without updating either model’s weights (Figure 1c). This suggests that the role of the Address block in guiding retrieval is a general property of hybrid language models, rather than one specific to a particular recurrent architecture. Across six Qwen3.5 models with growing sizes, we also find that Address computation becomes distributed at larger scales across a chain of linear-attention blocks, each guiding a successive softmax reader, because the query no longer passes from one reader to the next through attention.
Our contributions are:
The Encode–Address circuit. We identify two roles of linear attention in hybrid retrieval: an early Encode block supporting token identity and an Address block that determines which entry the softmax reader retrieves.
The Address mechanism. We show that the Address block uses its recurrent state to construct the representation that determines the reader’s attention query, with upstream blocks providing redundant contributions.
Cross-architecture transfer. We demonstrate that the Address block can transfer between Gated DeltaNet and Mamba-2 without updating model weights, suggesting a shared retrieval mechanism across hybrid architectures.
Scaling. We show that larger models distribute Address computation over a chain of blocks, each guiding a successive reader, and trace this to how the query reaches the later readers.
Related Work
Linear attention and recurrent sequence mixers. Transformers rely on softmax attention, which stores keys and values from previous tokens and attends over them explicitly at each step. While this provides direct access to the context, its key–value cache grows with sequence length. A broad class of efficient sequence models instead compresses past information into a fixed-size recurrent state, reducing the memory and computational cost of long-context processing. This includes linear-attention mechanisms such as fast weights and Gated DeltaNet [17, 23], as well as state-space models such as Mamba-2 [4]. Although their update rules differ, these architectures share the computational property most relevant to our study: past information is maintained in a fixed-size state that is updated recurrently and used to produce the current token representation. For simplicity, we use linear attention throughout the paper to refer to this broader class of fixed-state recurrent sequence mixers, including SSMs such as Mamba.
Hybrid architectures. Hybrid language models interleave linear-attention layers with a smaller number of softmax-attention layers [10, 16, 19]. This design is now used by many released models, including Nemotron-H, Kimi Linear, Qwen3.5, and Granite-4.0-H [12, 9, 15, 7]. Hybrids can also be obtained by distilling pretrained Transformers, retaining selected attention layers while converting the rest to recurrent layers [20]. Prior work mainly studies how many softmax layers to retain and where to place them. We instead ask what the surrounding linear-attention layers contribute to retrieval, which directly determines which of them a hybrid can remove, distill, or redesign without breaking retrieval.
Retrieval in recurrent and attention models. Recurrent models are known to struggle with copying and associative recall [8, 1, 21], a weakness commonly attributed to compression into a fixed-size state [2]. Hybrid models mitigate this limitation by restoring a small number of softmax-attention layers, and Waleffe et al. [ [19] show that even a small fraction of attention layers can substantially close the gap on recall-heavy tasks. Bick et al. [ [3] further trace this advantage to a small number of heads that implement a Gather-and-Aggregate mechanism, with hybrid models delegating aggregation to their attention layers. These works ask whether recurrent layers can perform retrieval. We instead study models in which retrieval already succeeds and ask how the computation is divided between recurrent and softmax layers. We find that linear attention is not bypassed: it constructs the query used by softmax attention.
Mechanisms of in-context retrieval. In Transformers, retrieval is often mediated by small sets of attention heads, including induction heads [13], heads that select answer letters in multiple-choice prompts [11], long-context retrieval heads [22], and Gather-and-Aggregate heads [3]. These studies explain how attention heads read from context, but largely leave open how their queries are constructed. In the Gather-and-Aggregate framework, our Encode block resembles a local gathering step and the reader performs aggregation, but the Address block, which determines which entry the reader should retrieve, has no direct counterpart.
Mechanistic studies of recurrent models have localized factual recall in Mamba [18, 5] and shown that several interpretability methods developed for Transformers transfer to recurrent architectures [14]. To our knowledge, prior work has not disentangled the roles of linear and softmax layers within a hybrid retrieval circuit. We show that linear-attention layers serve two distinct functions: an early, generic re-encoding of token identity (Encode) and a specific computation of the retrieval query (Address) consumed by softmax attention.
Case Study: In-Context Key–Value Retrieval
Task and Model
Retrieval task. We use a controlled in-context key–value retrieval task. Each prompt contains a dictionary of eight key–value pairs followed by a question about one key. The prompt is wrapped in each model’s chat template, and the assistant response is prefilled so that the value is predicted as the next token:
Dictionary: elder=iron, carrot=red, cherry=steel, mango=wood, ...
Question: What is the value of mango?
[assistant] The value of mango is **
Keys and values are single tokens, and each value has the same tokenization in the dictionary and in the answer. We call this format chat8. Two variants change the surface form while keeping the task, to test that our findings do not depend on one template. list8 writes the dictionary as a bulleted lookup table with one * key=value entry per line, and rewords the question and answer prefix (“Which value does mango map to?” and “The entry mango maps to **”). rev8 keeps the chat8 question but states each entry with the value first (“iron is the value of elder”). Each format contains 100 prompts. Accuracy is the fraction of prompts for which the queried value is the highest-logit token over the full vocabulary; all four models reach at least 0.99 on chat8.
We rely on this synthetic task rather than on standard benchmarks because it isolates retrieval from other capabilities. Benchmarks such as MMLU also depend on parametric knowledge, reasoning, and answer selection, which makes it hard to tell whether an intervention affects retrieval specifically. Our task fixes the relevant information inside the prompt, places the target at a known position, and lets us build matched prompts that differ only in the queried key, properties that the interventions of § 4 require.
Models. We study two families of hybrid language models with representative linear-attention mechanisms. The Qwen3.5 family includes Qwen3.5-0.8B (0.8B parameters, 24 layers) and Qwen3.5-9B (9B parameters, 32 layers), both using Gated DeltaNet with a 3:1 ratio of linear-attention to softmax-attention layers. The Granite-4.0-H family includes Granite4.0-1B (1B parameters) and Granite4.0-7B (7B total parameters, approximately 1B active per token), both using Mamba-2 with four softmax-attention layers among 40 layers. Granite4.0-7B is a mixture-of-experts model. § 7 extends our analysis to Qwen3.5-27B, which contains 48 linear-attention and 16 softmax-attention layers (3:1 ratio). We define a block as a maximal sequence of consecutive linear-attention layers. Main-text figures focus on Qwen3.5-9B, with results for the other models reported in Appendix § 11.
Retrieval Emerges Abruptly at a Softmax Layer
To locate where retrieval is completed, we follow the progressive-pruning analysis of [3]. We truncate the network after layer : the pruned model keeps layers , removes every later layer (both its token mixer and its MLP), and feeds the remaining hidden state directly into the final normalization and language-model (LM) head. As a contrast, we also evaluate knowledge-focused performance on six benchmarks (ARC-Challenge, ARC-Easy, PIQA, WinoGrande, OpenBookQA, and HellaSwag) using zero-shot evaluation with lm-evaluation-harness [6] and averaged over the six tasks. A truncated model rarely ranks the answer first over the full vocabulary, so we score retrieval in the same multiple-choice manner: the prediction is the dictionary value with the highest logit, and random guessing yields chance . We report normalized accuracy, , which maps random-guessing accuracy to 0 and the unpruned model’s accuracy to 1.
Figure 2 shows the results for Qwen3.5-9B and reveals three key observations:
Figure 2. Performance of Qwen3.5-9B truncated after layer . Retrieval appears abruptly when the softmax layer L19 is kept; knowledge grows gradually with depth.
Knowledge develops gradually, while retrieval emerges abruptly. Knowledge-focused performance im-
proves steadily as more layers are retained. Retrieval remains at the level of random guessing until layer 19, regardless of how many preceding linear-attention layers are retained.
A single softmax layer marks the onset of retrieval. Retaining the softmax layer L19 raises normalized retrieval accuracy from 0 to 0.67. Retrieval accuracy may fluctuate between softmax layers, possibly because intermediate representations are not decodable by the final LM head. This does not affect where retrieval first appears.
Retrieval requires only part of the network. Retaining 24 of 32 layers preserves 0.94 of the full model’s normalized retrieval accuracy, compared with only 0.47 of its knowledge-focused performance.
The Encode–Address Circuit
Progressive pruning localizes retrieval to a softmax layer but does not reveal what the preceding linear-attention layers contribute, since all layers below the reader remain intact. To retrieve the correct value, the reader must match its query against the stored keys and aggregate their values. We therefore ask how the preceding linear-attention layers enable this lookup.
We use two complementary interventions on linear-attention blocks, i.e., consecutive layers. Removal tests whether a block is necessary for retrieval; transplant tests whether its output is sufficient to determine what is retrieved. Unlike truncation, both interventions target only the linear-attention token mixers, leaving the MLPs intact, because our question concerns what linear attention computes.
We intervene at the block level for two reasons. First, pruning shows that retrieval changes primarily at softmax-layer boundaries, making blocks a natural unit of analysis. Second, single-layer interventions can miss effects that emerge across multiple layers: removing or transplanting individual linear-attention layers has little effect, except for removing the first layer (Appendix B). We examine individual layers when necessary, as in the Encode block.
Figure 3 presents the removal and transplant results of Qwen3.5-9B on chat8. It has eight blocks, to , of three GDN layers each; (layers 16–18) immediately precedes the reader layer L19. Additional models and prompt formats are reported in Appendix C. Together, these interventions reveal two distinct roles for linear attention: an Encode block that re-encodes token identity and an Address block that determines which entry the reader retrieves.
Figure 3. Removal and transplant of each linear-attention block of Qwen3.5-9B (chat8); comprises layers to . (a) Removing the Encode block destroys retrieval. Removing the Address block leaves accuracy at 1.00 but halves the logit gap and reduces the reader’s attention on the queried entry from 0.83 to 0.06. (b) Transplanting ’s output at the final position from a prompt about another key makes the model answer that key and moves the reader to the donor’s entry; no other block has an effect.
Removal: Identifying Critical Blocks
Removal sets a block’s token-mixer output to zero at every position, leaving its MLPs and all softmax layers intact. We evaluate its effect using three metrics. Accuracy measures whether the model predicts the correct value. The logit gap measures the model’s preference for the correct value over an alternative (e.g., “yellow” over “green” for “banana=yellow”). We swap the two values to create a paired prompt and average the correct-minus-alternative logit differences across both prompts. A larger positive gap indicates a stronger preference for the correct value. Reader attention measures the attention mass assigned by the reader, from the final position, to the queried dictionary entry. A higher value indicates stronger attention to the correct entry.
Figure 3a shows three patterns. First, removing the earliest block, , nearly eliminates retrieval (accuracy 0.01, normalized logit gap 0.01, reader attention 0.06). Removing its first layer alone reduces accuracy to 0.05. Second, removing , immediately before the reader, preserves accuracy at 1.00 but reduces the logit gap from +23.8 to +11.6 and reader attention from 0.83 to 0.06. Thus, although the model still predicts the correct answer, its preference for that answer weakens substantially, and the identified reader no longer attends reliably to the target. Third, removing intermediate blocks – partially weakens reader attention (0.53–0.73), while blocks after the reader have little effect.
The result for is particularly revealing. Its removal disrupts the reader at L19, but later softmax layers (L23 and L27) compensate and recover the answer, as discussed in § 7. In models without such redundancy, removing the block before the reader reduces accuracy to 0.13 in Qwen3.5-0.8B and 0.00 in Granite4.0-1B. These results establish that is essential for retrieval, while is critical to the reader but not necessarily to the final prediction. Removal alone therefore cannot establish which block determines what the model retrieves.
In summary, the earliest Encode block is essential for retrieval, while the Address block immediately before the reader is critical to its lookup but may be dispensable for the final prediction due to redundancy in later softmax layers.
Transplant: Redirecting Retrieval
Transplant tests whether a block’s output is sufficient to determine what is retrieved. We construct a donor prompt A and a host prompt B with the same dictionary but different queried keys (e.g., A asks about “banana” and B about “ginger”). Since their lengths and dictionary positions are identical, we can replace a block’s mixer output at the final position in B with the corresponding output from A. If the transplanted representation determines the retrieval target, B should now return A’s value instead of its own. We measure the success rate over 100 donor–host pairs.
Figure 3b shows that transplanting redirects the model to the donor’s entry in 0.99 of the pairs. The reader’s attention also shifts to the donor’s entry. In contrast, transplanting any other block, including , has no effect (0.00). Transplanting over the dictionary positions rather than the final position also fails. This result holds across prompt formats (0.97 on list8, 0.98 on rev8). Thus, the output of a single linear-attention block at the final position determines which entry the reader retrieves. We call this the Address block.
Removal and transplant reveal a striking distinction. Across all prompt formats, removing the Address block preserves accuracy at 0.76–1.00 but roughly halves the logit gap, whereas transplanting its output redirects retrieval in 0.97–0.99 of cases. Together, these results establish the central mechanism of the Encode–Address circuit: the Encode block enables retrieval, while the Address block determines which entry the softmax reader retrieves.
The Role of Encode
Removing destroys retrieval, yet transplanting its output at the final position has no effect. To understand its role, we perform two token-level interventions. In a value swap, we change a dictionary entry from “apple=red” to “apple=yellow,” then replace the Encode output for yellow with that recorded for red in the original prompt. The intervention succeeds if the model answers red” rather than yellow” when queried about “apple”. In a key swap, we similarly exchange the keys of two entries and restore their original Encode outputs. As a control, we also replace the input embeddings at the swapped positions with those of the original tokens. For all interventions, success means recovering the original key–value associations.
Patching the Encode outputs restores the original associations in 0.98 of cases for both value and key swaps (Table 1, left). Surprisingly, patching only the input token embeddings achieves 1.00, while patching any other block’s output has no effect. This suggests that the information needed to restore retrieval is already present in the input embeddings, rather than being uniquely computed by Encode.
| Patched at swapped tokens | Value swap | Key swap | Accuracy | Perplexity | |
| Encode block output | 0.98 | 0.98 | Intact | 1.00 | 10.6 |
| Input token embedding | 1.00 | 1.00 | Token-wise linear projection | 1.00 | 44.7 |
| Any other block’s output | 0.00 | 0.00 | Mixers zeroed (MLPs kept) | 0.00 | 1059 |
| No Patching | 0.00 | 0.00 | Removed | 0.00 |
Table 1. The Encode block of Qwen3.5-9B (chat8). Left: success rate of token-level interventions in restoring the original key–value associations. Right: retrieval accuracy and WikiText-2 perplexity under modifications of .
We further isolate the role of Encode by replacing with a token-wise linear projection fitted to its original inputs and outputs using ridge regression. This replacement preserves retrieval accuracy (1.00) but increases WikiText2 perplexity from 10.6 to 44.7. These results suggest that Encode’s role in retrieval can be fulfilled by a simple linear transformation, although the original block performs additional computations important for general language modeling. Having established Encode’s role in preserving token identity, we next investigate how Address constructs the representation that directs the softmax reader toward the queried key.
How Is the Address Computed?
§ 4.2 establishes that the Address block’s final-position output determines which dictionary entry the softmax reader retrieves. We now investigate how this output is constructed: whether it controls the reader’s query or keys, where the retrieval target is stored, and how other blocks contribute to its computation.
The Address Block Determines the Reader’s Query
A softmax head retrieves an entry by matching its final-position query against the dictionary keys. To determine whether the Address block controls the query or the keys, we run Qwen3.5-9B with and without , then recompute attention using (1) the query without and the original keys, or (2) the original query and the keys without (Figure 4a). We measure selectivity, the difference between the log attention mass on the queried entry and the mean log attention mass on other entries, and the top-1 rate, the fraction of prompts where the queried entry receives the most attention. Higher selectivity indicates a stronger preference for the queried entry, while a higher top-1 rate indicates more frequent correct selection.
Figure 4. How the Address block constructs the retrieval target in Qwen3.5-9B. (a) Reader selectivity when its query or keys are computed without the indicated block; numbers show top-1 rates. Solid, light, and hatched bars denote the three prompt formats. (b) Transplanting the Address block’s final-position output, or its recurrent state, redirects retrieval; the states of earlier blocks and the block’s input at the final position do not. (c) Address block outputs computed without any single upstream block still redirect retrieval, but those computed without all three do not.
Figure 4a shows that the Address block primarily controls the query. Using a query computed without reduces selectivity from 5.45 to 1.36, with the reader selecting the correct entry on only about half of the prompts. In contrast, using keys computed without leaves selectivity nearly unchanged (4.80) and the top-1 rate at 1.00. Removing the preceding block only mildly weakens query selectivity without changing the reader’s choice. Thus, these results indicate that the Address block primarily determines the reader’s attention query, rather than its keys.
The Recurrent State Determines the Retrieval Target
The preceding results establish that the Address block determines the reader’s attention query. We next investigate how its recurrent state contributes to this computation as plotted in Figure 4b.
As shown in § 4.2, transplanting the Address block’s final-position output redirects retrieval to the donor’s entry in nearly all cases. To determine whether its recurrent state carries the information needed to produce this output, we run donor and host prompts up to the penultimate token, copy the donor’s recurrent states in all three Address layers () into the host, and let the host process its own final token. This also redirects retrieval in 0.99 of cases. In contrast, transplanting the recurrent states of or has no effect (0.00).
As a further control, we replace the host’s input to at the final position with the donor’s, while keeping the host’s recurrent state unchanged. This also fails to redirect retrieval (0.00). Together, these results indicate that the Address block’s recurrent state carries the information determining the retrieval target, which the block uses to construct its final-position output. The softmax reader then uses this output to form the attention query that selects the corresponding dictionary entry.
Does the Address Transfer Across Architectures?
Qwen3.5 and Granite-4.0-H are trained independently and differ in their proportion of linear-attention layers, recurrent mechanisms (Gated DeltaNet vs. Mamba-2), and model sizes. Despite these architectural differences, can the Address block learned by one model support retrieval in the other? We investigate whether both its output and computation transfer across the two model families.
We test two pairs of similarly sized models: Qwen3.5-0.8B with Granite4.0-1B, and Qwen3.5-9B with Granite4.0-7B. Since their hidden representations differ, we fit linear alignment maps between models using ridge regression on paired activations without updating either model’s weights.
The Address Block’s Output Transfers
We first test whether one model can use the Address block output computed by another. Donor and receiver process the same dictionary but ask about different keys. We transplant the donor’s final-position Address block output into the receiver and measure how often the receiver answers the donor’s question. As a baseline, we also transplant outputs between two prompts processed by the same receiver model, reported in the Same-model column of Table 2.
| Donor → receiver | Output transplant: Cross-model | Output transplant: Same-model | Block replacement: Removed | Block replacement: Linear proj. | Block replacement: Zeroed mixer | Block replacement: Donor Address |
| Qwen3.5-0.8B → Granite4.0-1B | 0.98 | 1.00 | 0.01 | 0.24 | 0.28 | 0.92 |
| Granite4.0-1B → Qwen3.5-0.8B | 0.91 | 0.97 | 0.30 | 0.28 | 0.28 | 0.95 |
| Qwen3.5-9B → Granite4.0-7B1 | 0.99 | 1.00 | 0.00 | 0.14 | 0.18 | 0.94 |
| Granite4.0-7B → Qwen3.5-9B2 | 0.83 | 0.92 | 0.35 | 0.30 | 0.21 | 0.46 |
Table 2. Cross-architecture transfer on chat8. Output transplantation: success rate of redirecting retrieval using the donor Address block’s output, compared with a same-model baseline using two prompts processed by the receiver model. Block replacement: receiver accuracy after replacing its Address block with the donor block or a control. 1 Both Address blocks of Granite4.0-7B are replaced. 2 Qwen3.5-9B is evaluated after L19 (intact accuracy: 0.75), using both Address blocks of Granite4.0-7B as the donor.
For the smaller models, transplantation succeeds in both directions: 0.98 from Qwen3.5-0.8B to Granite4.0-1B and 0.91 in reverse on chat8, close to their same-model baselines of 1.00 and 0.97, respectively. Success rates range from 0.87 to 1.00 across formats, and the receiver’s softmax reader shifts its attention to the donor’s entry.
The larger pair introduces an additional complication. In Granite4.0-7B, retrieval depends on two reader layers, L25 and L35, preceded by Address blocks and , respectively. Transplanting alone redirects the first reader but rarely changes the final answer (0.03–0.06), whereas transplanting both blocks redirects both readers and the answer (1.00, 1.00, and 0.90 across formats). Thus, transplantation must account for both Address blocks in Granite4.0-7B.
Accordingly, transplanting the Address block output of Qwen3.5-9B into both blocks of Granite4.0-7B redirects retrieval in 0.99 of cases, matching the in-family result (1.00). Transplanting into only the first block rarely succeeds (0.04). In the reverse direction, the two Address block outputs of Granite4.0-7B redirect Qwen3.5-9B in 0.83 of cases, compared with 0.92 for in-family transplantation.
These results show that the Address block’s output can guide retrieval across independently trained architectures, with cross-model transplantation approaching the success rates of same-model transplantation.
The Address Block's Computation Transfers
Transplanting the Address block’s output shows that the receiver can use a retrieval target computed by another model. We next test whether the donor’s Address block can replace the receiver’s computation itself.
We remove the receiver’s Address block and insert the donor’s block in its place, using linear maps to connect their hidden representations. Unlike output transplantation, the donor block now processes the receiver’s sequence. We compare this replacement against three controls: (1) removing the receiver’s Address block without replacement, (2) replacing it with a token-wise linear projection, and (3) inserting the donor’s Address block but zeroing its linear-attention mixer outputs while retaining its MLPs. These controls test whether successful transfer requires the donor Address block’s recurrent computation rather than a simple linear transformation or its remaining components.
For the smaller models, block replacement succeeds in both directions (Table 2). Replacing Granite4.0-1B’s Address block with that of Qwen3.5-0.8B achieves 0.92 accuracy, compared with 0.01 when the receiver’s block is removed without replacement. The reverse replacement achieves 0.95, compared with 0.30 without replacement. In contrast, a token-wise linear projection or a donor block with zeroed mixer outputs achieves at most 0.28. These results indicate that successful transfer depends on the donor Address block’s recurrent token-mixing computation, rather than a simple linear transformation.
For the larger models, replacement is more challenging because Granite4.0-7B uses two Address blocks, whereas Qwen3.5-9B relies primarily on one (§ 6.1). We therefore replace both Address blocks of Granite4.0-7B and evaluate Qwen3.5-9B after L19 to prevent compensation by later readers. Replacing both blocks of Granite4.0-7B with Qwen3.5-9B’s Address block achieves 0.94 accuracy, compared with 0.00 without replacement, 0.14 with a token-wise linear projection, and 0.18 with zeroed donor mixers.
The reverse transfer is less effective. Replacing Qwen3.5-9B’s Address block with both Address blocks of Granite4.0-7B, run sequentially without their intervening softmax layer, achieves 0.46 accuracy. This exceeds the removal control (0.35) and either donor block alone (0.34–0.36), but remains below the intact receiver’s accuracy (0.75). This partial recovery suggests that computation distributed across multiple Address blocks plus a softmax layer is harder to transfer.
Together, these experiments show that the Address block’s computation can transfer between Gated DeltaNet and Mamba-2, particularly when donor and receiver have compatible Address block structures.
Scaling: From One Address Block to Several
We examine how Address computation changes across six Qwen3.5 models: five dense models from Qwen3.5-0.8B to Qwen3.5-27B, and the mixture-of-experts Qwen3.5-35B-A3B (3B active parameters). All six place one softmax layer after every three linear-attention layers. We investigate how many Address blocks are needed to redirect retrieval, which softmax readers they guide, and whether retrieval depends on them (Table 3).
| Model | Linear / softmax | Address block → guided readers | Removal | Transplant |
| Qwen3.5-0.8B | 18 / 6 | 0.06 | 0.99 | |
| Qwen3.5-2B | 18 / 6 | 0.41 | 1.00 | |
| Qwen3.5-4B | 24 / 8 | 0.94 | 0.70 | |
| Qwen3.5-9B | 24 / 8 | 0.92 | 0.98 | |
| Qwen3.5-27B | 48 / 16 | 1.00 | 0.98 | |
| Qwen3.5-35B-A3B | 30 / 10 | 0.34 | 0.75 |
Table 3. Address blocks and the readers they guide in Qwen3.5 models. Removal: accuracy after removing the Address blocks. Transplant: fraction of prompts redirected to the donor’s target when the Address blocks are transplanted jointly.
One Address block guides multiple readers up to 9B. The four smaller models share a common structure: a single Address block immediately before the first main reader guides all three readers. Transplanting this block’s final-position output redirects retrieval in 0.70–1.00 of cases for these models.
However, the necessity of the Address block differs across model sizes. Removing it reduces retrieval accuracy to 0.06 at 0.8B and 0.41 at 2B, but leaves accuracy at 0.94 and 0.92 at 4B and 9B. In the larger models with higher redundancy, the linear-attention blocks after the Address block also carry a weaker signal of which entry is queried. When is present, its stronger signal dominates and all readers follow it; when it is removed, the weaker signal is still enough for the later readers to retrieve the correct value.
Address computation becomes distributed in larger models. Multiple Address blocks emerge in larger models. In Qwen3.5-27B and Qwen3.5-35B-A3B, no single block can redirect retrieval. Instead, different Address blocks guide their own readers (Appendix § 12), and jointly transplanting these blocks redirects 0.98 and 0.75 of answers, respectively.
How Address blocks guide readers also changes with scale. In smaller models (up to 9B), the first reader influences subsequent readers, allowing a single Address block to guide multiple readers. In larger models (above 9B), each Address block guides a different set of readers, providing more independent pathways for retrieval. However, their necessity differs between Qwen3.5-27B and Qwen3.5-35B-A3B. In the dense 27B model, removing the Address blocks leaves retrieval accuracy at 1.00. In contrast, removing the Address blocks reduces accuracy to 0.34 in the MoE 35B-A3B model. We leave a systematic comparison of dense and MoE architectures for future work.
In summary, the Address computation becomes more distributed in larger models, with multiple blocks contributing to the final retrieval query. This distribution allows for redundancy and robustness in retrieval.
Conclusion
In hybrid language models, softmax attention performs retrieval, but linear attention helps determine what is retrieved. We identify an Encode–Address circuit in which the Address block carries the retrieval target in its recurrent state and constructs the final-position representation from which the softmax reader forms its attention query. This representation causally redirects retrieval even when removing the Address block leaves accuracy intact. Both the representation and its underlying computation transfer across Gated DeltaNet and Mamba-2, while larger models distribute Address computation across multiple blocks. Together, these findings reveal a division of labor: linear attention constructs the query, while softmax attention performs the lookup. Thus, linear attention serves not only as compressed memory but also as an active component in determining what the model retrieves.
Reproduction
All experiments run against frozen public checkpoints with eager attention. Interventions are forward hooks on the token-mixer outputs; no weights are modified. Per-model result files, logs and prompts accompany the code.
Task generation. Keys are drawn from 22 words and values from 20 colour/material words, all single tokens in context, sampled without replacement. For each item the corrupted twin swaps the queried value with a distractor’s. Token positions are recovered from character spans through the tokenizer’s offset mapping, and items are batched only when their length and all entry positions agree, so an intervention applies to identical positions across a batch. Items are rejected unless clean and corrupted prompts have equal token length, differ only inside the target and distractor entries, and both answers are single tokens.
Chat templates. Qwen prompts use the model’s template with an empty think block, so the answer is the next token. For other families the wrapper is derived from the tokenizer’s own chat template; Granite’s inserts a default system prompt, which is why its prompts are tokens longer for the same dictionary.
Single-Layer versus Block Interventions
Table 4 compares interventions on the whole block that precedes the reader with the same interventions on each of its layers (chat8, final-position transplant). In both Qwen models no single layer changes the answer when transplanted, while the whole block does; Granite4.0-1B concentrates most of the effect in L23. Removing layer 0 alone breaks retrieval in every model, so we report the Encode block at layer resolution.
| Model | Block | Transplant | Removal | Layer 0 | ||
| whole block | best single layer | whole block | worst single layer | removed | ||
| Qwen3.5-0.8B | L12–14 | 0.99 (0.84) | 0.00 (0.03, L14) | 0.13 | 0.90 (L14) | 0.00 |
| Qwen3.5-9B | L16–18 | 0.99 (0.78) | 0.00 (0.10, L18) | 1.00 | 0.99 (L17) | 0.05 |
| Granite4.0-1B | L16–24 | 1.00 (0.83) | 0.93 (0.70, L23) | 0.00 | 0.73 (L24) | 0.00 |
Table 4. Whole-block versus single-layer interventions on the block preceding the reader (chat8). Transplant: fraction of donor–host pairs that return the donor’s value, with the reader’s attention on the donor’s entry in parentheses (host baseline ); for single layers, the best layer of the block is shown. Removal: accuracy with the block, or the most damaging single layer of the block, removed. The last column gives accuracy with layer 0 alone removed.
Additional Models and Prompt Formats
Table 5 repeats the removal and transplant analysis of § 4 for Qwen3.5-0.8B, Granite4.0-1B and Granite4.0-7B, and for all three prompt formats. In every model, removing the Encode block destroys retrieval, transplanting the Address redirects retrieval in 0.90–1.00 of the pairs, and no other single block does. Granite4.0-7B reads the answer at two softmax layers, L25 and L35, preceded by (layers 16–24) and (layers 26–34). Transplanting alone moves the L25 reader to the donor’s entry (attention 0.80 on chat8) but changes the answer in only 0.03 / 0.06 / 0.05 of the pairs, because L35 still reads with the host’s query. We therefore treat and jointly as the Address of Granite4.0-7B: transplanting both redirects 1.00 / 1.00 / 0.90 of the pairs, the last matching the model’s own accuracy on rev8, and removing both reduces accuracy to at most 0.06.
| Model | Address | Removal | Transplant | ||
| Encode | Address | Address | best other block | ||
| Qwen3.5-0.8B | L12–14 | 0.00 / 0.00 / 0.00 | 0.13 / 0.03 / 0.01 | 0.99 / 1.00 / 0.97 | 0.00 / 0.00 / 0.00 |
| Qwen3.5-9B | L16–18 | 0.01 / 0.00 / 0.00 | 1.00 / 1.00 / 0.76 | 0.99 / 0.97 / 0.98 | 0.00 / 0.00 / 0.00 |
| Granite4.0-1B | L16–24 | 0.00 / 0.00 / 0.00 | 0.00 / 0.01 / 0.00 | 1.00 / 0.98 / 0.92 | 0.00 / 0.00 / 0.00 |
| Granite4.0-7B | L16–24, L26–34 | 0.00 / 0.00 / 0.00 | 0.01 / 0.06 / 0.00 | 1.00 / 1.00 / 0.90 | 0.00 / 0.00 / 0.00 |
Table 5. The circuit across models (chat8 / list8 / rev8). Removal: accuracy with the block removed (intact ). Transplant: fraction of pairs that return the donor’s value when the block’s output is transplanted at the final position. For Granite4.0-7B, the Address comprises two blocks, and , which are removed and transplanted jointly; transplanting alone gives 0.03 / 0.06 / 0.05.
Address Blocks and Readers in Larger Models
Figure 5 shows which readers each Address block guides in the two largest models: each block moves only the readers that follow it. At 35B-A3B, moves L27, moves L31 together with , adds L35, and additionally moves L19. At 27B, moves L39 and L43, and L51 and L59 follow once and are added.
Figure 5. Attention of the top head of each reader layer on the donor’s entry after transplanting single core blocks (above the line) or their combinations (below) at the final position, in Qwen3.5-27B (a) and Qwen3.5-35B-A3B (b). Intact host .
References
- [1]Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. In International Conference on Learning Representations (ICLR), 2024a.arxiv.org/abs/2312.04927
- [2]Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024b.arxiv.org/abs/2402.18668
- [3]Aviv Bick, Eric Xing, and Albert Gu. Understanding the skill gap in recurrent language models: The role of the gather-and-aggregate mechanism. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pages 4324–4344, 2025. Code: https://github.com/goombalab/Gather-and-Aggregate.arxiv.org/abs/2504.18574
- [4]Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2405.21060
- [5]Nir Endy, Idan Daniel Grosbard, Yuval Ran-Milo, Yonatan Slutzky, Itay Tshuva, and Raja Giryes. Mamba knock-out for unraveling factual information flow. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025.
- [6]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness. https://github.com/EleutherAI/lm-evaluation-harness, 2024.
- [7]IBM. IBM Granite 4.0: Hyper-efficient, high performance hybrid models for enterprise. https://www.ibm.com/new/announcements/ibm-granite-4-0-hyper-efficient-high-performance-hybrid-models, October 2025.
- [8]Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2402.01032
- [9]Kimi Team. Kimi Linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692, 2025.arxiv.org/abs/2510.26692
- [10]Barak Lenz, Opher Lieber, Alan Arazi, et al. Jamba: Hybrid transformer-Mamba language models. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2403.19887
- [11]Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in Chinchilla. arXiv preprint arXiv:2307.09458, 2023.arxiv.org/abs/2307.09458
- [12]NVIDIA. Nemotron-H: A family of accurate and efficient hybrid Mamba-transformer models. arXiv preprint arXiv:2504.03624, 2025.
- [13]Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. Transformer Circuits Thread, 2022. URL https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.arxiv.org/abs/2209.11895
- [14]Gonçalo Paulo, Thomas Marshall, and Nora Belrose. Do transformer interpretability methods transfer to RNNs? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27565–27572, 2025.DOI
- [15]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5.
- [16]Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2406.07522
- [17]Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2102.11174
- [18]Arnab Sen Sharma, David Atkinson, and David Bau. Locating and editing factual associations in Mamba. In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2404.03646
- [19]Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro. An empirical study of Mamba-based language models. arXiv preprint arXiv:2406.07887, 2024.arxiv.org/abs/2406.07887
- [20]Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, and Tri Dao. The Mamba in the Llama: Distilling and accelerating hybrid models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2408.15237
- [21]Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. RNNs are not transformers (yet): The key bottleneck on in-context retrieval. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2402.18510
- [22]Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2404.15574
- [23]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with delta rule. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2412.06464




