Multi-State Linear Attention with Generalized Second-Order Routing

Introduction

The self-attention mechanism is a core operation in long-sequence modeling [31]. However, its quadratic computational complexity and linearly growing key-value (KV) cache have become a key bottleneck in scaling compute and data [19]. Linear attention [18, 29, 24] addresses these limitations by reformulating attention to operate in sub-quadratic complexity. For example, DeltaNet [36] and Gated DeltaNet [37] remove the softmax in standard attention and replace the KV cache with a single fixed-size memory state. However, a single fixed-size memory state is inherently limited in memory capacity and may lead to information loss when compressing longer contexts [2, 17, 20]. As a result, models based on single-state linear attention still underperform standard attention in practice.

Recently, Multi-State Linear Attention (MSLA) mechanisms [4, 14, 32] have been introduced to improve the long-context capabilities of single-state linear attention. MSLA partitions the context into segments and compresses each segment into its own memory state. Each token extracts segment-level outputs from preceding memory states and then aggregates them to produce its final output representation (Fig. 1a). Operating in the multi-state regime therefore introduces two novel challenges: how to divide the context into segments (separation), and how to combine the segment-level outputs (aggregation).

We study aggregation, which is relatively underexplored compared to separation. Existing methods aggregate segment-level outputs via a weighted sum [4, 14, 32], where the weights represent each segment’s relevance to the query: closely aligned segments contribute more to the final output than others. However, prior works derive the weights either from position (Guo et al., [14]; Wang et al., [32]), assuming recency as the source of relevance, or from the mean (first-order) alignment between each segment’s keys and the query (Behrouz et al., [4]). They overlook the variance (second-order) of the alignment, which captures sharply aligned keys that the mean alone cannot distinguish from a segment of uniformly weak ones.

To analyze why the variance matters, we examine how an MSLA model assigns aggregation weights on a long-sequence reasoning task. Figure 1b shows that the segment containing the ground truth, which is the segment most relevant to the response, receives the highest aggregation weight when the weights are derived from both the mean and variance of the alignment, compared to the mean or position alone. This motivates an aggregation method that incorporates the variance of the alignment in deriving the weights.

We propose MERLIN (MomEnt Routing for multi-state LINear attention), a method that leverages the mean and variance of key-query alignment to improve aggregation in multi-state linear attention. MERLIN stores the first and second moments of each segment’s keys. Using the moments, a query recovers both the mean (first-order) and variance (second-order) of its alignment with each segment’s keys to derive the weights for the weighted sum of segment-level outputs. Our experiments show that MERLIN improves the long-context capabilities of multi-state linear attention models.

Our contributions are:

1. We identify segment-level output aggregation as a key bottleneck in multi-state linear attention: existing methods weight each segment’s output using only the mean alignment between the query and the segment’s keys (§ 2).

2. We propose MERLIN, a method that keeps the first and second moments of each segment’s keys. A query uses these moments to recover the mean and variance of its alignment with the segment (§ 3.1) and to derive the aggregation weights (§ 3.2).

3. We evaluate multi-state variants of Gated DeltaNet-1.3B and Mamba-2-780M (§ 4.1). On LongBench (Bai et al., [3]), MERLIN improves the average score over the strongest baseline by 13.5% on Gated DeltaNet and 12.0% on Mamba-2 (§ 5).

Background

Given an input sequence of $T$ tokens, the query, key, and value matrices $Q$, $K$, $V \in\mathbb{R}^{T \times d}$ are projected from the tokens’ hidden states $x_1,\ldots,x_T$. Standard full attention computes:

where $M \in\{-\infty,0\}^{T \times T}$ is a causal masking matrix. Softmax attention costs quadratic time, and requires a KV cache that grows linearly with the context. This motivates more efficient sub-quadratic attention formulations.

Linear Attention. Linear attention removes the softmax normalization and instead computes:

This formulation can be reorganized into a recurrence over a memory state $S_t\in\mathbb{R}^{d\times d}$.

where $k_t$, $v_t$, $q_t$ are the key, value, and query vectors of a token at position $t$. The memory state $S_t$ accumulates key-value outer products, acting as an associative memory that binds each key to its value. A query reads from the memory state via a matrix-vector product to get the output representation $o_t$. The recurrence now runs in linear time and constant memory. Recent works add gating and delta-rule correction to improve the memory capacity of the fixed-size state. For example, Gated DeltaNet (Yang et al. [37]) (GDN) uses the recurrence as follows:

where $\alpha_t\in(0,1)$ decays the state and $\beta_t$ sets the strength of the update.

Multi-State Linear Attention. Linear attention can be limiting when $T$ grows because it needs to compress the entire context into a single fixed-size state. Multi-state linear attention relaxes this bottleneck by partitioning the sequence into $L$ contiguous segments $C_1,\ldots,C_L$ and compressing each segment in its own state:

Intuitively, Eq. (3) can be viewed as splitting the first equation of Eq. (1) into $L$ parts, which we call separation. At step $t$, a query reads segment-level outputs from the last $\ell\le L$ memory states visible at step $t$. The intermediate outputs are then aggregated via a weighted sum to derive the final output representation:

where $\lambda_{t,i}$ is the aggregation weight that determines how much the output from segment $i$ contributes to the final output at position $t$. Eq. (4) can be viewed as the second equation of Eq. (1) applied to $\ell$ partitioned states, which we refer to as aggregation.

Unlike single-state linear attention, in multi-state linear attention, the number of memory states increases as the content length increases. Multi-state linear attention therefore interpolates between single-state linear attention (constant memory, linear time) and full attention (linear memory, quadratic time) ([4]). Table 1 summarizes the time and memory complexities of attention variants.

Mechanism	Compute (Prefill)	Memory
Full Attention ([31])	$O(T^{2}d)$	$O(Td)$
Linear Attention ([18])	$O(Td^{2})$	$O(d^{2})$
Multi-State Linear Attention
Log-Linear Attention ([14])	$O(Td^{2}\log T)$	$O(d^{2}\log T)$
Memory Caching ([4])	$O(TLd^{2})$	$O(Ld^{2})$
MERLIN (ours)	$O(TL(d^{2}+d_{r}^{2}))$	$O(L(d^{2}+d_{r}^{2}))$

Aggregation in Multi-State Linear Attention. The aggregation of segment-level outputs in Eq. (4) requires weights $\lambda_{t,i}$. These weights function as routers that increase the contribution of outputs from query-relevant segments and decrease the contribution of outputs from others. The simplest methods learn a position-based weight vector $\lambda_t$ (with entries $\lambda_{t,i}$) directly from the target token, projected from its query or hidden state ([14, 32]):

These weights do not consider what each segment stores and therefore cannot function as a content-aware router. Behrouz et al. [4] introduces content-awareness in aggregation weights as follows:

Behrouz et al. [4]:

where $u_t$ is a learned vector projected from hidden state $x_t$ and $\bar{k}_i$ is the first moment of keys in segment $i$ (i.e., the mean). However, this only captures the mean alignment between the query and a segment’s keys and does not capture the variance.

We revisit the analysis in Figure 1b and demonstrate that the variance of the alignment is necessary for more content-aware aggregation of segment-level readouts. We train multi-state models with different aggregation methods and evaluate on 30 randomly selected TriviaQA samples in HELMET (Yen et al. [38]) at 4K context length. This is a retrieval-augmented generation task where each sample consists of multiple passages and a prompt that requires retrieving information from one of them. Thus, the model should locate the relevant passage and extract the correct answer to the prompt.

We find that different aggregation methods assign varying amounts of aggregation weight to the segment containing the ground truth. Ideally, the weights assigned to the ground truth segment should be higher than others in order to increase the contribution of the information extracted from the segment in the final response. Figure 1b shows that mean- and variance-based aggregation assigns the highest weight to the ground truth segment compared to position-only and mean-only aggregation. This motivates our proposed method MERLIN, which leverages both orders of alignment to compute the weights for segment-level output aggregation.

MERLIN

Our goal is to capture the variance of the alignment between a query and each segment’s keys to derive more informative aggregation weights. We first introduce how the first and second moments of the keys are constructed (§3.1). Then, we discuss how the mean and variance of the alignment can be derived from the moments (§3.2). Finally, we elaborate on the relationship between multi-state linear attention equipped with MERLIN and softmax attention (§3.3).

First and Second Moments of Keys

For each token, we project new keys and queries $k'_t, q'_t \in\mathbb{R}^{d_r}$ from the hidden states. We use $k'_t, q'_t$ instead of $k_t, q_t$ to construct and read from the first and second moments of keys, primarily to keep moments compact by choosing $d_r$ that is smaller than $d$.

The first moment of keys is the mean-pooled vector of all keys ($\mathbf{k}'_j$) in the segment. The second moment is the average of the outer products $\mathbf{k}'_j\mathbf{k}'^{\mathsf{T}}_j$ over the segment. The first and second moments of keys can thus be formulated as:

The first and second moments are fixed-size, and their dimensions depend only on $d_r$. Together, they summarize segment $i$, where $\boldsymbol{\mu}_i$ captures the average direction of the keys and $\boldsymbol{\Sigma}_i$ captures how the keys are spread within the segment.

Aggregation Weight Derivation

Given a token at position $t$, we compute the following scalar scores for each visible segment $C_i$ (the segments leading up to the current one):

Here, $r^\mu_{t,i}$ captures the mean key-query alignment, measuring how well the segment’s keys broadly align with the query. Meanwhile, $r^\Sigma_{t,i}$ captures the variance, measuring the spread of the per-token alignment between the query and the keys in the segment. We mix the softmax-normalized $r^\mu_{t,i}$ and $r^\Sigma_{t,i}$ with a learned gate to get the final weight $\lambda_{t,i}$:

MERLIN (ours): $\lambda_{t,i} = \gamma_t\operatorname{softmax}_i(\tau_\mu r^\mu_{t,i}) + (1-\gamma_t)\operatorname{softmax}_i(\tau_\Sigma r^\Sigma_{t,i})$,

where $\operatorname{softmax}_i$ denotes the softmax operation over the segment index $i$, $\gamma_t$ is a scalar gate, and $\tau_\mu$ and $\tau_\Sigma$ are learnable temperatures that scale the two scores before normalization. The gate $\gamma_t$ is computed from the token’s hidden state $x_t$ through a learned projection. The temperatures $\tau_\mu,\tau_\Sigma$ calibrate the sharpness of each term. This shows how the first and second moments are used to derive the weights for Eq. (4).

Connection to Softmax Attention

Suppose we ran token-level softmax attention over the keys and then grouped its mass by segment. For a target token at position $t$ and a segment $C_i$, the key-query similarity can be expressed as:

We can define the length-normalized evidence of the segment as:

where $\delta$ is a temperature hyperparameter. Grouping the token-level softmax by segment, the total attention mass a full softmax layer would place on segment $i$ can be expressed as:

Because our segments are equal-sized, $\log\lvert C_i\rvert$ is the same for every segment and cancels inside the softmax. Expanding the remaining log-evidence in cumulants around $\delta=0$, we obtain:

Therefore, the relevance of a segment is governed by the first two terms: the mean $\eta_{t,i}$ and the variance $\sigma^2_{t,i}$. These are precisely the scores $r^\mu_{t,i}$ and $r^\Sigma_{t,i}$, which are computed from the first and second moments via matrix-vector products:

Rather than the fixed additive coefficients of Eq. (10) ($\delta$ on the mean, $\frac{\delta^2}{2}$ on the variance) inside a single softmax, MERLIN combines the two scores with a learned gated mixture of separately-normalized softmax terms (Eq. (9)), with learnable temperatures $\tau_\mu,\tau_\Sigma$ and gate $\gamma_t$. The gated mixture is also flexible to implement, since the two terms are normalized and scaled independently.

Experiments

We conduct experiments to quantify the effect of MERLIN on the long-context capabilities of multi-state linear attention models. We first explain the models, datasets, and baselines used for training and evaluation (§ 4.1). Then, we show the main evaluation results on the core benchmarks (§ 5). Third, we conduct analysis on positional biases and impact of segment length (§ 6). Finally, we conduct ablations on the core design choices in MERLIN (§ 7).

Setup

Models. We use the Gated DeltaNet ([37]) and Mamba-2 ([7]) architectures for the following experiments. Gated DeltaNet is a linear attention model that updates the memory state following the recurrence in Eq. (2). Mamba-2 is a state-space model that uses recurrence of structured semiseparable matrices. We show that MERLIN improves upon the multi-state variants of both architectures with distinct features to demonstrate its robustness and effectiveness. A more detailed pseudocode for each model can be found in Section C.5.

Training. We first pre-train the Gated DeltaNet-1.3B ([37]) model on 50B tokens in FineWeb-edu ([23]) following the official training configurations. For Mamba-2-780M, we use the officially released checkpoint¹ pre-trained on 300B tokens from Pile ([10]). These pre-trained models are still single-state: they require mid-training (or continued pre-training) to function as multi-state variants, i.e., training the pre-trained model with the same objective function as pre-training on the mid-training dataset.

In order to convert single-state models to multi-state variants, we introduce separation (Eq. (3)) and aggregation (Eq. (4)) mechanisms during mid-training. We fix the separation mechanism to uniform segmentation, where each segment has the same length of 256 tokens. At training context length of 16K tokens, the model learns to aggregate as many as 64 (16K / 256) segment-level outputs. We specify the full list of training hyperparameters and model configurations in Section C.1.

We mid-train the pre-trained models in two stages. Each stage is defined by the type of training data that was used. Stage 1 uses 2B tokens from Long-Data-Collections (LDC) ([30]), with 10% of the training samples (0.2B tokens) augmented with RULER ([15]) needle-in-a-haystack (NIAH)-like tasks. The augmented data contains artificial keys and values scattered throughout regular LDC sequences. The task query and ground truth answer are attached at the end of the context. This dataset is designed to train basic language modeling capability of a multi-state variant by activating its aggregation mechanism.

In stage 2, we train on a smaller dataset of 60M tokens. It consists of 50% regular LDC sequences, and 50% synthetically generated samples from RULER HotpotQA and SQuADQA. This dataset is designed to further strengthen the model’s aggregation mechanism under reasoning contexts. After stage 1 and stage 2, the pre-trained single-state models now function as multi-state models with long-context capabilities. The detailed mid-training hyperparameters can be found in Section C.2.

Benchmarks. We evaluate the models on various long-context benchmarks. First, we evaluate on the NIAH and QA tasks in RULER (Hsieh et al., [15]) at 16K context length. The single needle variants (S-NIAH-1/2/3) evaluate the model’s ability to retrieve a single key-value pair in different types of natural and synthetic text haystacks. The multi-needle variants (MK-NIAH-1/MQ-NIAH/MV-NIAH) evaluate the model’s ability to retrieve from multiple key-value pairs in different formats. The HotpotQA and SQuADQA tasks in RULER evaluate multi-hop and single-hop reasoning skills.

We also evaluate on the following subtasks in LongBench (Bai et al., [3]): NarrativeQA (NQ), Qasper (QR), MultiFieldQA-en (MF), HotpotQA (HP), 2WikiMultihopQA (2W), GovReport (GR), QMSum (QM), MultiNews (MN), SAMSum (SS), TREC (TC), TriviaQA (TQ). We filter for samples under 16K context length. These tasks largely evaluate single-document QA, multi-document QA, summarization, and few-shot learning capabilities.

Finally, we evaluate on the TriviaQA retrieval-augmented-generation task in HELMET (Yen et al., [38]) at 16K context length. We use this benchmark to analyze how the position of the ground truth in the sequence impacts the model’s performance. Section C.4 provides a detailed explanation of the evaluation protocol used in our experiments.

Baselines. We compare MERLIN with three other baselines: Vanilla, Position, and Meanpool. Vanilla is the unmodified single-state backbone, with neither segmentation nor aggregation mechanisms. The other two are alternative aggregation methods for multi-state variants. Position (Eq. (5)) derives aggregation weights from the hidden states alone, scaling segment-level outputs by a learned recency scalar (Guo et al., [14]). Meanpool (Eq. (6)) derives aggregation weights from first-order alignment between a query and each segment’s keys (Behrouz et al., [4]).

Main Results

We report results on RULER (Hsieh et al., [15]) NIAH and QA tasks. Table 2 shows that MERLIN outperforms baselines in most tasks for both GDN and Mamba-2 architectures. MERLIN applied to GDN leads to an average score of 62.7, while Meanpool has an average score of 57.9, followed by Position’s 54.6. In the case of Mamba-2, MERLIN shows a 7.0% relative improvement over Meanpool (76.2 vs. 71.2), and a 9.5% relative improvement over Position (vs. 69.6). This suggests that mid-training with MERLIN improves long-context retrieval.

	MidTr	S-NIAH-1	S-NIAH-2	S-NIAH-3	MK-NIAH-1	MQ-NIAH	MV-NIAH	HotpotQA	SQuADQA	Avg
Gated DeltaNet 1.3B	–
	–	18.6	25.2	4.2	16.8	5.1	15.8	10.0	8.3	13.0
	Vanilla	89.0	99.2	10.8	23.4	80.8	70.2	15.0	8.3	49.6
	Position	97.2	96.8	16.0	42.2	82.5	78.4	16.0	7.3	54.6
	Meanpool	99.6	99.4	21.2	48.4	86.6	85.0	12.0	10.7	57.9
	MERLIN	99.2	100.0	47.0	45.0	91.0	88.8	17.0	13.7	62.7
Mamba-2 780M
	–	0.0	0.0	0.0	0.0	0.0	0.0	2.0	0.0	0.3
	Vanilla	100	100	8.4	24.6	92.2	86.4	16	4.7	54.0
	Position	100.0	100.0	54.2	91.0	94.5	89.2	20.0	8.0	69.6
	Meanpool	100.0	100.0	66.2	89.4	93.7	91.8	20.0	8.1	71.2
	MERLIN	99.8	99.8	96.0	94.8	95.9	92.2	22.0	9.4	76.2

We also report results on LongBench (Bai et al., [3]) tasks filtered to 16K. Overall, MERLIN demonstrates strongest average performance, followed by Meanpool, Position, and finally Vanilla. Table 3 shows that MERLIN leads to robust gains particularly on single-doc and multi-doc QA. For GDN, MERLIN leads to a 13.5% improvement in average score compared to Meanpool (21.9 vs 19.3). For Mamba-2, MERLIN leads to a 12.0% (20.6 vs 18.4) improvement compared to Meanpool. This suggests that MERLIN strengthens general long-context capabilities.

		Single-doc QA			Multi-doc QA			Summarization			Few-shot Learning
	MidTr	NQ	QR	MF	HP	2W	GR	QM	MN	SS	TC	TQ	Avg
Gated DeltaNet 1.3B	-	2.4	5.0	13.9	4.5	7.9	9.1	16.1	9.2	33.0	22.3	19.5	13.0
	Vanilla	6.0	6.1	12.2	6.8	8.5	9.9	16.5	9.4	40.5	42.5	21.4	16.3
	Position	11.5	9.6	14.9	14.3	11.1	6.1	14.0	12.3	39.0	48.5	23.1	18.6
	Meanpool	8.8	15.2	18.0	14.6	9.8	8.7	15.3	10.1	34.5	54.8	22.4	19.3
	MERLIN	13.9	19.4	20.3	16.3	14.4	7.8	15.9	12.8	40.0	60.4	20.1	21.9
Mamba-2 780M	-	0.9	1.6	3.6	0.9	2.2	3.2	8.7	12.6	8.0	8.1	1.4	4.7
	Vanilla	1.0	3.2	7.6	4.2	5.2	10.2	12.3	14.7	36.5	54.5	5.7	14.1
	Position	1.2	5.1	10.5	5.7	6.8	15.5	15.0	17.4	42.0	65.1	12.5	17.9
	Meanpool	1.7	4.2	11.5	6.0	5.6	16.0	17.7	18.5	44.5	61.1	15.1	18.4
	MERLIN	1.9	6.7	12.6	7.8	9.1	20.4	16.1	19.2	46.0	72.7	14.2	20.6

Analysis

Positional Bias Analysis. We evaluate models on the TriviaQA task in HELMET (Yen et al., [38]), at 16K context length. This is a retrieval-augmented generation task, where the context consists of multiple passages and only one of them, which we refer to as the gold passage, is related to the final query of the prompt. The location of the gold passage is shuffled randomly within the context for each sample. Therefore, we can track the accuracy of the model’s responses depending on the location of the gold passage, and detect any positional biases that might exist. Figure 3 shows that the Vanilla baseline demonstrates a strong recency bias: it struggles unless the gold passage is located towards the end of the context. The Position baseline also demonstrates a weak recency bias, where the accuracy gradually increases as the gold passage moves closer to the end. Both the Meanpool baseline and MERLIN perform more evenly across gold passage positions, while MERLIN is better overall. This suggests that MERLIN enhances the model’s ability to extract information from important segments in the context by improving aggregation of segment-level readouts.

Segment Length Analysis. We examine whether MERLIN is robust to different segment lengths. Segment length, or the number of tokens per segment, determines the number of segments per context given that the entire length of the sequence is fixed ($L = T/m$). The shorter the segment length, the more states per sequence, and therefore the higher the memory consumption of the model. We experiment with $m \in\{64, 128, 256, 512\}$ for each aggregation method. At $m = 64$, the model consumes approximately 8 times more memory compared to $m = 512$. Figure 4 shows that for both GDN and Mamba-2, MERLIN outperforms baselines on LongBench [3] tasks across $m$. In particular, MERLIN at $m = 256$ (default) outperforms Meanpool and Position baselines at $m = 64$, which consume 4 times more memory. This suggests that MERLIN is more memory efficient compared to its baselines.

Memory Cost and Parameter Overhead. In Section D, we compare the memory cost of different multi-state linear attention strategies with MERLIN. We observe that the extra cost of our additional second moment features is limited to at most 12.6% for Gated DeltaNet and 4.2% for Mamba-2 at $d_r = 128$, while the memory cost decreases quadratically with $d_r$. We also compare the number of parameters that MERLIN adds to each backbone compared to baselines. We find that the MERLIN’s parameter overhead is on par with the Meanpool baseline at 4.6% for GDN and 2.2% for Mamba-2.

Qualitative Examples. In Appendix § 17, we provide examples of aggregation weight $(\lambda_{t,i})$ assignment on four HELMET (Yen et al., [38]) retrieval-augmented generation samples using MERLIN and baseline methods. As observed in the quantitative evaluations, MERLIN assigns a higher weight to the gold segment, and this supports that the variance of the alignment is critical for aggregation.

Ablations

Ablations on Dimension of Moments of Keys. We report on the impact of $d_r$, the dimension of the first and second moments of keys (§ 3.2), on the performance of MERLIN. We vary $d_r \in\{16, 32, 64, 128\}$ during training and evaluate on LongBench (Bai et al., [3]). Figure 5 shows that the average score on LongBench tasks gradually increases as $d_r$ increases. This suggests that the width of the moments of keys is critical in deriving informative aggregation weights. However, there is a trade-off in deciding $d_r$. The moments of keys can be more expressive when $d_r$ is larger. However, MERLIN consumes less memory compared to the Meanpool and Position baselines as $d_r$ gets smaller. More details on the memory consumption can be found in Table 9 of Section D.

(a) Gated DeltaNet-1.3B ($d_k \times d_v=256 \times512$)				(b) Mamba-2-780M ($P \times N=64 \times128$)
Method	512 (32)	256 (64)	128 (128)	64 (256)	Method	512 (32)	256 (64)	128 (128)	64 (256)
Vanilla	$1\times$	$1\times$	$1\times$	$1\times$	Vanilla	$1\times$	$1\times$	$1\times$	$1\times$
Position	$32.0\times$	$64.0\times$	$128.0\times$	$256.0\times$	Position	$32.0\times$	$64.0\times$	$128.0\times$	$256.0\times$
Meanpool	$32.0\times$	$64.0\times$	$128.0\times$	$256.0\times$	Meanpool	$32.0\times$	$64.0\times$	$128.0\times$	$256.0\times$
MERLIN ($d_r=16$)	$32.1\times$	$64.1\times$	$128.3\times$	$256.5\times$	MERLIN ($d_r=16$)	$32.0\times$	$64.0\times$	$128.1\times$	$256.2\times$
MERLIN ($d_r=32$)	$32.3\times$	$64.5\times$	$129.0\times$	$258.1\times$	MERLIN ($d_r=32$)	$32.1\times$	$64.2\times$	$128.3\times$	$256.7\times$
MERLIN ($d_r=64$)	$33.0\times$	$66.0\times$	$132.1\times$	$264.1\times$	MERLIN ($d_r=64$)	$32.3\times$	$64.7\times$	$129.4\times$	$258.7\times$
MERLIN ($d_r=128$)	$36.0\times$	$72.1\times$	$144.1\times$	$288.3\times$	MERLIN ($d_r=128$)	$33.3\times$	$66.7\times$	$133.4\times$	$266.7\times$

Ablations on Parameter Sharing. For each token, we project new keys and queries $(k'_t, q'_t)$ to construct the first and second moments, separate from the original keys and queries for the memory state $(k_t, q_t)$. We isolate the impact of decoupling the keys and queries by experimenting with a variant of MERLIN where $k'_t = k_t$ and $q'_t = q_t$ (shared-KQ rows in Table 4). We further experiment with a variant of MERLIN where we control the decay and update strength of the first and second moments of keys per token, similar to the decay and write-strength features of the memory state explained in Eq. (2) (write-forget rows in Table 4). We find that both ablations substantially degrade the performance of MERLIN on LongBench (Bai et al., [3]). This suggests that using a dedicated set of keys and queries from moments of keys is particularly important in deriving informative aggregation weights.

	Single	Multi	Summ	Few-shot	Overall
GDN + MERLIN	17.9	15.4	12.2	40.2	21.9
+ write-forget	14.9	9.2	11.6	35.3	18.5
+ shared-KQ	10.7	9.2	10.9	29.8	15.7
Mamba + MERLIN	7.1	8.5	18.6	44.3	20.6
+ write-forget	6.7	6.9	16.6	43.8	19.6
+ shared-KQ	5.1	6.1	15.1	38.3	17.1

Conclusion

We investigate the aggregation problem in multi-state linear attention, where segment-level outputs are combined via a weighted sum to produce the final output representation of each token. We propose MERLIN, an aggregation method that assigns weights to each segment based on the mean and variance of the alignment between a query and the keys of each segment. In order to compute the mean and variance of the alignment, MERLIN keeps compact first and second moments of the segment’s keys. We find that MERLIN is superior to existing aggregation methods in terms of performance and memory efficiency. MERLIN presents a generalizable and effective routing mechanism for the aggregation problem in multi-state linear attention models.

Acknowledgments

All experiments, data collection, and processing activities were conducted at Princeton University, and no Google DeepMind computing resources were used for this work. Sanghyuk Chun is supported by the Francis Robbins Upton Fellowship at Princeton University. Xindi Wu is supported by the Apple Fellowship.

AI Use Statement

In this work, we used generative AI tools to polish the writing, generate synthetic training datasets (§ 4.1), and write triton kernels for method implementation. We also used AI tools to plot figures and create a name for our method. We have not used generative AI tools to formulate and develop the conceptual framework of the method, or to identify relevant literature. We have reviewed all AI-assisted work. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Ethics Statement

Our work presents a method that improves general long-context capabilities of multi-state linear attention models. Although we do not see a clear ethical concern, we advise readers to be cautious with the datasets and tools they may use to extend our work beyond the scope of this paper.

Reproducibility Statement

We use publicly available model architectures and datasets for training and evaluation. We discuss the method details in § 3, and specifics of the training procedure in § 4.1. We also provide a full summary of model configurations and training hyperparameters in Appendix Section C. The evaluation details can be found in Section C.4.

Appendices

Notation

Symbol	Description
Sequences and softmax attention
$T$	Sequence / context length (number of tokens)
$t$	Position of the target token
$x_t$	Hidden state of token $t$
$d$	Key/value dimension of one head’s memory state (written $d$ for both, whereas the backbones use $d_k \times d_v$, see Table 9)
$Q, K, V \in\mathbb{R}^{T \times d}$	Query, key, and value matrices
$q_t, k_t, v_t$	Query, key, and value of token $t$
$W_q, W_k, W_v$	Backbone projections producing $q_t, k_t, v_t$ from $x_t$
$M$	Causal attention mask. In softmax attention it is additive with entries in $\{-\infty, 0\}$. In linear attention it is binary, with $M_{ij} = 1$ if $i \ge j$ and $0$ otherwise
$O, o_t$	Attention output, as a matrix and for token $t$
softmax	Row-wise softmax normalization
$\odot$	Elementwise (Hadamard) product
$I$	Identity matrix
Linear attention
$S_t \in\mathbb{R}^{d \times d}$	Recurrent memory state at step $t$
$\alpha_t \in(0, 1)$	State decay gate in Gated DeltaNet (Eq. (2))
Symbol	Description
$\beta_t$	Write strength of the delta term (Eq. (2))
Multi-state recurrent memory
$L$	Number of segments (memory states)
$C_i$	The $i$-th context segment
$i,\ \ell$	Segment indices ($1 \le i,\ell\le L$)
$j$	Index of a token within a segment ($j \in C_i$)
$m = \lvert C_i\rvert$	Number of tokens per segment
$S_{C_i} \in\mathbb{R}^{d \times d}$	Memory state of segment $i$. For the plain linear recurrence, $S_{C_i} = \sum_{j \in C_i} v_j k_j^{\mathsf{T}}$
$o_{t,i}$	Segment-level output at position $t$: readout of segment $i$’s state, $o_{t,i} = S_{C_i}q_t$
$\lambda_{t,i}$	Aggregation weight of segment $i$ at position $t$
$\lambda_t,\ W_{\lambda}$	Position-based weight vector (entries $\lambda_{t,i}$) and its projection
$W_u,\ u_t$	Target-token projection and vector used by mean-key routing
$\bar{k}_i$	Meanpooled key of segment $i$
MERLIN: key moments and aggregation weights
$d_r$	Moment dimension (hyperparameter)
$q'_t,\ k'_t \in\mathbb{R}^{d_r}$	New query and key of token $t$, projected from its hidden state
$W_{q'},\ W_{k'}$	Projections producing $q'_t$ and $k'_t$ from $x_t$
$\mu_i \in\mathbb{R}^{d_r}$	First moment of segment $i$’s keys (mean of $k'_t$ over $C_i$)
$\Sigma_i \in\mathbb{R}^{d_r \times d_r}$	Uncentered second moment of segment $i$’s keys (mean of $k'_t {k'_t}^{\mathsf{T}}$ over $C_i$)
$r^\mu_{t,i}$	Mean key–query alignment for segment $i$, $r^\mu_{t,i} = \mu_i^{\mathsf{T}}q'_t$
$r^\Sigma_{t,i}$	Variance of the key–query alignment over segment $i$, $r^\Sigma_{t,i} = {q'_t}^{\mathsf{T}}\Sigma_i q'_t - (r^\mu_{t,i})^2$
$\gamma_t$	Learned gate mixing the softmax-normalized mean and variance terms (Eq. (9)), distinct from the decay gate above
$\tau_\mu,\ \tau_\Sigma$	Learnable temperatures for the mean term ($\tau_\mu$) and the variance term ($\tau_\Sigma$)
Connection to softmax attention
$s_{t,i,j}$	Query-key similarity, $s_{t,i,j} = {q'_t}^{\mathsf{T}}k'_j$ for $j \in C_i$
$\delta$	Temperature in the grouped-softmax analysis (§ 3.3)
$Z_{t,i}(\delta)$	Length-normalized softmax evidence of segment $i$
$A_{t,i}$	Grouped softmax attention: total softmax mass on segment $i$
$\eta_{t,i}$	Mean of $s_{t,i,j}$ over $C_i$ (first cumulant)
$\sigma^2_{t,i}$	Variance of $s_{t,i,j}$ over $C_i$ (second cumulant)
Mamba-2 backbone (pseudocode)
$P,\ N$	Head dimension and SSM state size, with $S_{C_i} \in\mathbb{R}^{P \times N}$
$\Delta_t,\ A$	Step size at position $t$ and per-head scalar decay rate, so the state decays by $\exp(\Delta_t A)$
$B_t,\ D_t$	Input matrix (the write key) and output matrix (the read query, written $C$ in Mamba-2)
$W_B,\ W_D,\ W_\Delta$	Projections producing $B_t$, $D_t$, and $\Delta_t$ from $x_t$

Related Work

Linear Attention and Recurrent Memory Models. Linear attention replaces the softmax normalization in standard attention with a feature map, turning attention into a recurrence with a fixed-size memory state [18]. This state can be viewed as a fast-weight associative memory written by outer products [26, 16]. Recent works improve this recurrence with data-dependent gating [35], delta-rule updates that overwrite rather than accumulate [36], and their combination [37]. Selective state-space models [13, 7], and recurrent memory models such as RWKV [24] and RetNet [29] that are closely related to linear attention share the same fixed-size state form. These models compress the context into a single state, which may be limited in memory capacity due to information loss at longer contexts. Our work builds on these backbones to train more capable multi-state linear attention and recurrent memory models.

Multi-State and Hierarchical Memory. To improve the long-context capabilities of single-state linear attention and recurrent memory models, several methods partition the context into segments and compress each segment in its own memory state. As a result, these models maintain a bank of states over the context. Memory Caching [4] segments the context uniformly at fixed token length per segment. Log-linear attention organizes the recurrent states hierarchically by growing a logarithmic number of states along a Fenwick-tree partition [14] of the sequence. Dynamic linear attention determines state boundaries based on token-level information variation while bounding the cache by merging low-information states [32]. Segment-recurrent and compressive Transformers cache or compress past activations across chunks [6, 25, 5, 33]. These works primarily focus on how states are formed via content segmentation (i.e. separation). However, we identify how the information extracted from each state is merged (i.e. aggregation) as a key bottleneck in multi-state linear attention and propose a method that improves that derivation of aggregation weights.

Routing over Memories via Aggregation. Multi-state linear attention and recurrent memory models extract information from each state and aggregate them to generate the final output representation of a token. Routing occurs during aggregation by adjusting the contribution of each segment to the final output. Mixture-of-Memories routes each token to one of several linear-attention states [8]. Sparse State Expansion routes tokens into state partitions with top-$k$ hard selection [22]. Gated Slot Attention reads a bounded set of slots through a softmax [39]. Gated Residual Memory scores cached memory modules with a context-aware gate, with a sparse variant that selects the most relevant ones [4]. Such routers descend from mixture-of-experts gating [27, 9] and from content-based addressing of an external memory [12, 28, 34]. These aggregation methods assign weights based on position, the magnitude of its readout, or a single pooled key. However, MERLIN keeps first and second moments of each segment’s keys to derive more informative aggregation weights.

Associative Recall and Long-Context Retrieval. Retrieving specific information from a long sequence is a core capability of models that handle long-context. In particular, such tasks evaluate the memory capacity of linear attention and recurrent memory models that compress the entire context into a single memory state. Multi-query associative recall and the Zoology analysis attribute much of the quality gap between recurrent memory models and standard transformers to recall [1], and the recall-throughput tradeoff frames recall as bounded by recurrent state size [2]. Copying and factual-recall analyses make the capacity limit of the memory state more precise [17, 21]. Furthermore, studies show that long-context models demonstrate positional biases where they prefer information located in certain depth within the context, usually close to the end [15, 38]. We use these benchmarks to demonstrate that MERLIN mitigates such biases by improving aggregation of segment-level outputs.

Experiment Details

We describe our experimental setup in detail. Section C.1 describes the model and method configurations, Section C.2 the mid-training hyperparameters, Section C.3 the datasets and benchmarks, and Section C.4 the evaluation protocol. In Section C.5 we provide the pseudocode for MERLIN.

Model Configurations

Table 6 lists the two backbones and the MERLIN hyperparameters. Gated DeltaNet-1.3B (21 layers, hidden size 2048) was pre-trained on 50B tokens of FineWeb-edu, and Mamba-2-780M (48 layers, hidden size 1536) on 300B tokens of the Pile. For both, MERLIN uses a segment size of $m=256$ and a moment dimension of $d_r=128$.

	Gated DeltaNet-1.3B	Mamba-2-780M
Parameters	$\approx1.3\mathrm{B}$	$\approx780\mathrm{M}$
Pretraining corpus	FineWeb-edu	Pile
Pretraining tokens	50B	300B
Hidden size	2048	1536
Layers	21	48
Heads	6	48
Head dimension	256	64
SSM state size	–	128
Expansion factor	2	2
Short-conv kernel	4	4
MERLIN
Segment size $m$	256	256
Moment dimension $d_r$	128	128
Key/query normalization	$\ell_2$-normalized	$\ell_2$-normalized
Temperatures $\tau_\mu,\tau_\Sigma$, gate $\gamma_t$	learned	learned

Mid-Training Hyperparameters

Table 7 shows the mid-training recipe, which is shared by both backbones and by all $m$. We use AdamW with a weight decay of 0.1 and a peak learning rate of $1 \times10^{-4}$, which follows a cosine schedule after 305 warmup steps. Each run uses 16,384-token sequences and a global batch of 131,072 tokens for 15,259 steps, which amounts to 2.0B tokens in bfloat16 mixed precision.

Hyperparameter	Value
Optimizer	AdamW ($\epsilon= 10^{-15}$)
Weight decay	0.1
Peak learning rate	$1 \times10^{-4}$
Learning-rate schedule	cosine (min ratio 0.1)
Warmup steps	305
Gradient clipping (max norm)	1.0
Sequence length	16,384
Global batch size	131,072 tokens
Training steps	15,259
Total tokens	2.0B
Precision	bfloat16 (mixed)
Random seed	42
Data	Long-Data-Collections, augmented with RULER NIAH/QA

Datasets and Benchmarks

Table 8 summarizes the mid-training data and the evaluation benchmarks, together with the metric, context length, and number of samples for each. We evaluate long-context retrieval on RULER ([15]) and long-context understanding on LongBench ([3]), both at up to a 16K context.

Phase	Dataset / Task	Metric	Context	Size
Mid-training	Long-Data-Collections (+ RULER NIAH/QA)	–	16K	2.0B tokens
RULER (retrieval)	S-NIAH-1/2/3	recall	16K	500 each
RULER (retrieval)	MK-NIAH-1	recall	16K	500
RULER (retrieval)	MQ-NIAH	recall	16K	500
RULER (retrieval)	MV-NIAH	recall	16K	500
RULER (retrieval)	HotpotQA	recall	16K	500
RULER (retrieval)	SQuADQA	recall	16K	500
LongBench	NarrativeQA (NQ)	F1	≤16K	61
LongBench	Qasper (QR)	F1	≤16K	197
LongBench	MultiFieldQA (MF)	F1	≤16K	150
LongBench	HotpotQA (HP)	F1	≤16K	161
LongBench	2WikiMQA (2W)	F1	≤16K	198
LongBench	GovReport (GR)	ROUGE	≤16K	174
LongBench	QMSum (QM)	ROUGE	≤16K	116
LongBench	MultiNews (MN)	ROUGE	≤16K	200
LongBench	SAMSum (SS)	ROUGE	≤16K	172
LongBench	TREC (TC)	accuracy	≤16K	200
LongBench	TriviaQA (TQ)	F1	≤16K	142

Evaluation Protocol

All evaluations use a 16K context length. We evaluate on RULER ([15]) and LongBench ([3]) following the official metrics and configurations from Language Model Evaluation Harness ([11]). For RULER, we run the eight NIAH and QA tasks listed in Table 8 and report the score on each task together with the average across tasks. For LongBench, we run the eleven subtasks in Table 8, keeping only the samples of at most 16K tokens, and score each task with its standard metric (F1, ROUGE, or accuracy).

For HELMET ([38]), we use the TriviaQA retrieval-augmented generation task and follow the official evaluation setup. Each prompt contains several passages, and only one of them, the gold passage, contains the answer to the question. We randomly shuffle the position of the gold passage within the context and report accuracy as a function of that position, from the first sixth (1/6) to the last sixth (6/6) of the context.

Pseudocode

Algorithm 1 shows MERLIN on a Gated DeltaNet backbone, assuming equal-size segments of $m$ tokens each. The outer loop iterates over the $L$ segments, and the inner loop iterates over the $m$ tokens within each segment. For each token, the algorithm first updates the memory state of its segment together with the first and second moments of the segment’s keys, so both accumulate token by token. It then reads out from every visible segment, meaning all closed segments plus the current, partially built one. Finally, it aggregates these readouts using the moment-derived weights from Eq. (9).

Algorithm 2 shows MERLIN on a Mamba-2 backbone. The structure is identical to Algorithm 1, but differs only on the memory state recurrence. Each token’s hidden state is projected to the SSM

Algorithm 1 MERLIN (Gated DeltaNet backbone) with equal-size segments $C_1,\ldots,C_L$ of $m$ tokens, learnable temperatures $\tau_\mu,\tau_\Sigma$, and a learnable gate $\gamma_t$.

1. for $i=1,\ldots,L$ do

2. $S_{C_i},\mu_i,\Sigma_i\gets0,0,0$

3. for $t\in C_i$ do

4. $q_t,k_t,v_t\gets W_qx_t,W_kx_t,W_vx_t$

5. $q'_t,k'_t\gets W_{q'}x_t,W_{k'}x_t$

6. $S_{C_i}\gets S_{C_i}\left(\alpha_t(I-\beta_tk_tk_t^\top)\right)+\beta_tv_tk_t^\top$ ▷ GDN memory state, Eq. (2)

7. $\mu_i\gets\mu_i+\frac{1}{m}k'_t,\quad\Sigma_i\gets\Sigma_i+\frac{1}{m}k'_tk_t^{\prime\top}$ ▷ moments of keys

8. for $\ell=1,\ldots,i$ do

9. $o_{t,\ell}\gets S_{C_\ell}q_t,\quad r^\mu_{t,\ell}\gets\mu_\ell^\top q'_t,\quad r^\Sigma_{t,\ell}\gets q_t^{\prime\top}\Sigma_\ell q'_t-(r^\mu_{t,\ell})^2$

10. end for

11. $\lambda_{t,\ell}\gets\gamma_t\operatorname{softmax}_\ell(\tau_\mu r^\mu_{t,\ell})+(1-\gamma_t)\operatorname{softmax}_\ell(\tau_\Sigma r^\Sigma_{t,\ell})$ ▷ Eq. (9)

12. $o_t\gets\sum_{\ell\leq i}\lambda_{t,\ell}o_{t,\ell}$ ▷ aggregate readouts, Eq. (4)

13. end for

14. end for

15. return $o_1,\ldots,o_T$

Algorithm 2 MERLIN (Mamba-2 backbone) with equal-size segments $C_1,\ldots,C_L$ of $m$ tokens, learnable temperatures $\tau_\mu,\tau_\Sigma$, and a learnable gate $\gamma_t$. The state $S_{C_i}\in\mathbb{R}^{P\times N}$, and the cached moments are shared per group.

1. for $i=1,\ldots,L$ do

2. $S_{C_i},\mu_i,\Sigma_i\gets0,0,0$

3. for $t\in C_i$ do

4. $v_t,B_t,D_t,\Delta_t\gets W_vx_t,W_Bx_t,W_Dx_t,\operatorname{softplus}(W_\Delta x_t)$

5. $q'_t,k'_t\gets W_{q'}x_t,W_{k'}x_t$

6. $S_{C_i}\gets\exp(\Delta_tA)S_{C_i}+\Delta_tv_tB_t^\top$ ▷ Mamba-2 SSD state

7. $\mu_i\gets\mu_i+\frac{1}{m}k'_t,\quad\Sigma_i\gets\Sigma_i+\frac{1}{m}k'_tk_t^{\prime\top}$ ▷ moments of keys

8. for $\ell=1,\ldots,i$ do

9. $o_{t,\ell}\gets S_{C_\ell}D_t,\quad r^\mu_{t,\ell}\gets\mu_\ell^\top q'_t,\quad r^\Sigma_{t,\ell}\gets q_t^{\prime\top}\Sigma_\ell q'_t-(r^\mu_{t,\ell})^2$

10. end for

11. $\lambda_{t,\ell}\gets\gamma_t\operatorname{softmax}_\ell(\tau_\mu r^\mu_{t,\ell})+(1-\gamma_t)\operatorname{softmax}_\ell(\tau_\Sigma r^\Sigma_{t,\ell})$ ▷ Eq. (9)

12. $o_t\gets\sum_{\ell\leq i}\lambda_{t,\ell}o_{t,\ell}$ ▷ aggregate readouts, Eq. (4)

13. end for

14. end for

15. return $o_1,\ldots,o_T$

value $v_t$, input matrix $B_t$ (the write key), output matrix $D_t$ (the read query), and step size $\Delta_t$. The segment state $S_{C_i}\in\mathbb{R}^{P\times N}$ ($P$ the head dimension, $N$ the SSM state size) follows the Mamba-2 SSD recurrence with per-head scalar decay $\exp(\Delta_tA)$. In Mamba-2, the cached moments are shared per group. The aggregation step is identical to Eq. (9).

Additional Analysis

Memory Cost. Table 9 reports the recurrent memory retained to decode each token, relative to the single fixed-state (Vanilla) cache. All multi-state methods cache the $L=T/m$ segment states, which dominate the cost. Position adds no cached moments, and Meanpool adds a negligible first moment ($\leq0.1\%$), while MERLIN additionally caches a $d_r\times d_r$ second moment. This extra cost is controlled by $d_r$ and is at most 12.6% for Gated DeltaNet and 4.2% for Mamba-2 (whose moments are shared per group) at $d_r=128$, shrinking quadratically as $d_r$ decreases.

Parameter Overhead. We show the number of parameters that each aggregation method adds to the backbone under default model configurations ($m=256$, $d_r=128$). The parameter overhead of MERLIN roughly equals the Meanpool baseline (4.5% for Gated DeltaNet-1.3B and 2.2% for

Mamba-2-780M). The Position baseline is relatively lightweight, with only 1.2% and 0.6% overhead for Gated DeltaNet-1.3B and Mamba-2-780M respectively.

Backbone	Aggregation	Added	%
GatedDeltaNet-1.3B	Vanilla	0	0.0%
GatedDeltaNet-1.3B	POSITION	16.8M	1.2%
GatedDeltaNet-1.3B	MEANPOOL	66.1M	4.5%
GatedDeltaNet-1.3B	MERLIN	66.3M	4.6%
Mamba-2-780M	Vanilla	0	0.0%
Mamba-2-780M	POSITION	4.8M	0.6%
Mamba-2-780M	MEANPOOL	18.9M	2.2%
Mamba-2-780M	MERLIN	18.9M	2.2%

Qualitative Examples. Table 11 illustrates the aggregation weight assignment on individual segments given retrieval-augmented generation samples in HELMET Yen et al. [38] (two Natural Questions, two TriviaQA, all at 16K context). For each query we show the gold segment that contains the ground truth as well as four randomly sampled distractor (irrelevant) segments, and the aggregation weight $\lambda_{t,i}$ each receives in the last layer. MERLIN assigns highest weights on the gold segment, well above the recency-driven POSITION router and above MEANPOOL.

Index	Type	Candidate segment	POSITION	MEANPOOL	MERLIN
Question: “what is another name for the water cycle?” Answer: hydrological cycle
9	Distractor	. . . of the world’s water supply, about 321,000,000 cubic miles of it. . .	0.006	0.018	0.017
22	Distractor	. . . water vapour or aqueous vapor is the gaseous phase of water. . .	0.005	0.019	0.016
33	Ground Truth	. . . also known as the hydrologic cycle or the hydrological cycle, the continuous movement of water. . .	0.005	0.030	0.055
35	Distractor	. . . cultural eutrophication of lakes is primarily due to phosphorus. . .	0.005	0.008	0.005
54	Distractor	. . . loch or fen; each has a special meaning, though modern usage. . .	0.005	0.010	0.007
Question: “where are the Washington Redskins based out of?” Answer: the Washington metropolitan area
8	Distractor	. . . Franchise history: the Daniel Snyder ownership era (1999–present). . .	0.004	0.010	0.012
28	Distractor	. . . Daniel Snyder . . . W. Smith, the founder of Federal Express. . .	0.004	0.016	0.013
37	Ground Truth	. . . a professional football team based in the Washington metropolitan area; they compete in the NFL. . .	0.004	0.034	0.050
39	Distractor	. . . History. The Redskins relocated from Boston, Massachusetts. . .	0.004	0.018	0.031
50	Distractor	. . . a home for that team; its highest-ranking teams as of 2015. . .	0.004	0.030	0.027
Question: “in which Charles Dickens novel does the character Mr M Choakumchild appear?” Answer: Hard Times
16	Distractor	. . . the illiterate poor chipped in ha’pennies to have each new monthly episode. . .	0.004	0.010	0.024
26	Ground Truth	. . . where he wrote “Bleak House” (1852–53) and “Hard Times” (1854). . .	0.004	0.021	0.060
42	Distractor	. . . Mrs Nickleby is based on his mother, though she did not recognize herself. . .	0.004	0.012	0.010
45	Distractor	. . . the novel “Nicholas Nickleby”, and Mrs Micawber in “David Copperfield”. . .	0.004	0.023	0.010
53	Distractor	. . . Dickens in films and TV: Lennox Pawle (1935). . .	0.004	0.019	0.007
Question: “which chain of European retailers began near a crossroads in Annecy, France in 1958?” Answer: Carrefour
6	Distractor	. . . the family also holds a stake in 3 Susines, Norauto and others. . .	0.007	0.021	0.016
25	Ground Truth	. . . Carrefour S.A. is a French multinational retailer. . .	0.007	0.019	0.049
31	Distractor	. . . within its metropolitan boundaries; the Cagliari–Sarroch port system. . .	0.007	0.013	0.016
37	Distractor	. . . 17 September 2012 . . . Jacqueline de Jong; an internment camp. . .	0.007	0.008	0.006
42	Distractor	. . . some trades are also present along the avenue de Verdun. . .	0.007	0.011	0.007

Limitations

Due to compute constraints, our experiments cover two backbones (Gated DeltaNet-1.3B and Mamba-2-780M) at a 16K context length, and we mid-train multi-state models from single-state checkpoints. We compare MERLIN against Vanilla, Position, and Meanpool aggregation with uniform segmentation. Studying larger models, longer contexts, and a broader set of aggregation baselines is beyond our current compute budget, and we leave it to future work.